So my question is, what do you use to judge whether or not you can comfortably handle a specific caliber.
I think you are confusing issues between the largest caliber you can handle and the largest caliber you can handle comfortably.
Assuming we are talking about concealed carry, you should try to carry the largest caliber possible that you can shoot well, on a platform you can fully manipulate, that can still be carried concealed without too much trouble that you end up not carrying the gun.
chris in va notes most shootings are within 20 feet. Assuming that is reflective of a situation you might be in, then I would suggest you train at .least 50% further away, say 10 yards. You want to be better than what you expect to have to handle, ahead of the curve, not behind.
Folks tend to fall apart when faced with shooting at distances great than at which they had trained. Take the North Hollywood Bank Robbery. While the LAPD did recognize the bandits had on body armor and called for making head shots, in the hundreds of rounds fired at the bandits before they moved out, none of the LAPD shooters were able to hit the bad guys in the head. The ranges at which most were shooting was 75 yards and more. Prior to that time, LAPD did not train street officers to be proficient beyond 25 yards with their pistols and as it turned out, they were unable to make head shots at 75 yards. Heck, I would have thought that eventually, with all the rounds being shot at the bandits, that at least one officer would get lucky, but it didn't happen. The one bandit shot in the head was shot after they moved from the bank and the shot was by an officer that seemed to be no more than 20 yards away from the bandit.
In regard to comfort, shooting the gun does not have to be comfortable. Of course if it isn't, you may not practice with it enough, but then again, if you can handle it well given how much you practice with it and it is a bigger caliber than might actually be comfortable to shoot, then stick with the larger caliber. While not a handgun example, shooting my 12 ga Rem 870 with Double Impact (2 oz.) loads is not a particular nice experience, but I can shoot the loads fairly well. They are the most powerful loads I can shoot from the shotgun, but I hate to shoot more than 5 or 6 during any one day.