Massad and others have offered their advice on this many times in the past.
When I'm carrying specifically for defensive reasons, I stick with high quality factory fodder.
The trouble is, I usually carry for purposes other than self-defense. Sure, that same factory stuff will work fine for general purposes, but at a buck a round it's expensive. My handloads are far cheaper to shoot at a ground squirrel, jackrabbit, or a 'coon, and are of equal quality.
In the end, I don't worry about it too much. In town, I carry factory stuff. out of town and around the home place, I carry whatever works.
Daryl