I have kids, I have neighbors, I have an active lifestyle that just does not allow me to carry 100% of the time at home. If I leave the house, I carry. If I am in the house I do not. Not to imply that some do, but I choose not to live in fear. I am prepared for the worst to happen, but I make choices and decide when it is worth it to carry and when it isn't.
If someone came into my backyard and started shooting, might I regret it? I would. But would I definitely regret not being able to tackle my 5 year old or get in a hose fight with my wife or any number of other scenarios that are simply not practical for carrying around the house? YES.
That said I do have means of self defense within a few yards most of the time. And a vicious labrador that will lick any intruder to the bone !
My choice.