If you DO choose to carry something big, more power to you.
I do. I carry a full-size 357Mag with a 4.68" barrel, unloaded weight of 42oz.
BUT I'm also 6'4" and almost 300lbs
and in the summer it's in a fanny pack. Any sort of jacket weather and it's on my belt in a "high and tight" outside-waistband holster broadly similar to a "pancake" type rig.
And trust me, I'm one of those "carry without fail" types.
So...it CAN be done. But if there's any chance you're not going to do so, then you'd have been better off with a smaller gun.
As to the five-seven itself:
It's actually not that crazy as a CCW piece. The ammo is relatively cheap in factory loadings, esp. if you can afford a big initial order. $700 gets you 2,000rds of good quality stuff:
http://www.fivesevenammo.com/shop/page2.html
That works out to less than $20 per box of 50 - WAY less per-shot than I can get 357Mag for and a bit less than most sources of 38Spl these days
. (To get enough practice in, I set up another gun the same way as my main one, as close as I could get it, in .22LR.)
The five-seven tend to be accurate, reliable and low-recoil. They're also less powerful of course! But...get good with one, drop half a dozen center-of-goblin in a very short space of time, you can solve quite a few problems with such a thing...probably better than a five-shot 38Spl snubbie in most cases.
I guess what I'm saying is, feel free to be somewhat weird
. "Normal" is overrated
. But for God's sake get good with what you carry - and CARRY.