Interesting thread.
I grew up in a rural midwest town of 600 people. Everyone owned guns, my friends and I all grew up shooting/hunting/fishing, I used to read Guns & Ammo magazines during study hall in Jr. High, and 1/2 my high school was "sick" the opening day of shotgun season.
Fast forward to today. I still live in the midwest but now live in a "suburban" subdivision on the outskirts of a midsize city. I'm a father of two young boys and have a white color career as a mid/upper manager in a corporate office environment. And, I own more guns than ever.
It's not a secret in our family, and my in-laws typically buy me things like grips, 15 round magazines, sights, targets, holsters, etc. for Christmas. My oldest son who is 6 knows I own guns but isn't really exposed to them yet in person outside of seeing me fawn over an o/u 12 gauge I bought last year.
With all of this said, I completely understand the OP's point of view. No, he's not ashamed. And he's not from another planet. His ideas are not "moronic," and his crackhead comment was an off-hand example, not a real life scenario. But he may be wanting to exercise some caution in terms of who does and doesn't know he owns and carries. My hobby is not a secret, but only a few people I work with are aware and I don't talk about it much... but I'll never deny it. I don't go out of my way to start debates with people over the issue, but if someone comes at me with argument I'll fire up and defend my firearm ownership until that person concedes their point or no longer wishes to speak with me on the topic. Most people are very surprised that I own firearms and are shocked that I grew up in a rural area because, outwardly, I look like a yuppie who has never touched anything dirtier than their Blackberry. My firearms ownership is no one else's business and I don't want anyone getting any ideas by overhearing from "so and so" that I'm on vacation and have guns in my house (they're in a safe anyway, but that's not the point). No, I don't associate with people that I think would steal from me but the point is that just because someone doesn't wish to shout it from the rooftops isn't a bad thing. It's a choice. I choose not to advertise my income, not to publish my phone number, and not to relay my hobbies via bumper stickers (I also don't have any Titleist or Callaway Golf stickers on my car... yet it's a hobby as well).
Now, my own PERSONAL thoughts on this matter are that my kids should understand their dad has firearms and what this means, but that some people may not understand the positive aspects of gun ownership so this isn't something we talk about with others unless they hear dad talking about it first. The world today requires discretion regarding some topics; that's just a fact.
Or, OP, here's how you can solve being concerned about someone talking about your concealed carry. Move to where I live (IL)... the last state in the US not to allow it.