Have you tried examples of each .40 load being considered yourself, in your pistol?
I've been issued .40 S&W in both 165gr & 180gr, and have bought and tried various bullet weights as optional authorized ammunition (135gr, 155gr, 165gr & 180gr), over the years since I started buying .40's and having been issued a couple of them.
Personally, I don't have much concern whether I carry a 165gr or 180gr load, as long as it's made by one of the recognized major companies. Given my choice I'd pick one which uses one of the more modern bullet designs, but when I was carrying standard old-style JHP loads on & off-duty it wasn't keeping me awake at night.
I tend to prefer the heavier 180gr loads. The cartridge was designed around the desired performance characteristics of the reduced power 10mm 190gr & 180gr JHP's. The 180gr .40's are usually described by shooters as having the least amount of felt recoil effect (muzzle whip and torque).
Many folks of average shooting skills & experience may express feeling a bit more of the recoil effect when using 165gr loads, although a couple of the major companies have made some lower power 165gr loads for LE/Gov use over the years, as well. Also, I've often observed that as a shooter's skillset and experience increases, less attention (or awareness) seems to be paid to the difference between the 165gr & 180gr loads, recoil-wise. Skill & technique can mitigate the perceived slight difference between these 2 loads, it seems.
So, looking at the basics ...
How well does whatever
you choose feed, fire, extract & eject in
your particular pistol, when
you're shooting it?
How well can
you shoot
your pistol using the different ammunition? Controllability? Consistency of accuracy?
How available is the ammunition you've selected to you?
How affordable?
Can you get enough of it to not only test-fire it for function in your gun, but use it at least periodically for training & practice? (I like to test-fire randomly selected rounds from different boxes/cases of different production lots to check for any unexpected QC issues that might have occurred, myself, keeping the rest for carry and occasional practice.)
Ammunition selection is always a popular topic when you browse online firearms forums. It's something you can buy and look at, unlike the other critical considerations involved in the use of a firearm as a lawful defensive weapon.
Things like knowledge of the applicable laws involving the use of deadly force; familiarity with the function, operation & recommended maintenance of the firearm; skillset; training; practice; experience; and mindset.
Just my thoughts.
The specific ammunition has become a lower priority on my won list over the years I've worked in LE and as a firearms instructor. Sure, I want something that's good quality, and preferably something that has demonstrated good performance in both carefully controlled lab testing and actual shootings ... but it's still just ammunition, in the "ammunition" part of the whole equation.
How are you doing in respect to the rest of it?