I frequently do. One of my normal carry guns is a little LCP that disappears into my front pocket. I operate on the theory (and it may be a bad theory) that VERY few self-defense situations actually require serious, hard hits to the bad guy. In fact, a large percentage of defense situations in which the CCW carrier has actually needed to draw were resolved without having to fire (i.e. bad guy backs down and threat is removed). Another large percentage of situations are resolved when a shot is fired and the bad guy retreats either after a missed shot or after being hit with a round that doesn't immediately put him down. That leaves relatively few situations where the confrontation has ended due to the round's ability to incapacitate the bad guy. Therefore, I am taking a mathematically reasoned risk that my little .380 will protect me in the majority of situations I could be in.
Now, that may not be the right decision for everyone but it fits my needs.