I guess Im one of the lucky ones, as my everyday attire is pretty casual, Carhartts/Dickies, tee shirts with an untucked oxford shirts in the summer, fleece or sweatshirt in the winter.
I suppose having two sets of trousers is a personal choice. I prefer to wear my normal size, and while initially snug when you put the holster in, they quickly stretch to a perfect point, where they hold the gun and holster more securely, with no shifting around. They also look like they fit. The waist isnt the only thing "bigger" when you start going up in waist sizes. Bigger clothes look out of place and look like they dont fit.
I personally think anyone can do pretty much anything, if they "want" to do it. This really is all about "want" more than it is "cant". Im not big, nor do I wear over sized clothes, all my shirts, trousers, jackets, etc, are my normal size, and I dont dress out of line for my job or lifestyle, even as it might have changed over the years. Ive never had any troubles doing it. Then again, I want to do it. Im sure if I can do it, anyone else could, if they "wanted" to. "Trouble" I suppose, is something else that falls under perception.
All Im getting at here is, the OP seems to want to carry a full sized gun, for whatever his reasons are, and thats cool, if thats what he wants. He seems to have the "want" part figured out, so why tell him he "cant", because you choose to follow that line of thinking?
This "cant" thing seems to be something that intertwines its way through many of the threads anymore. It seems more of the various "problems" this or that gun has, is because someone "cant" deal with something about it they havent worked out. The trigger is bad, the grip angle is wrong, its to big, its to small, etc, etc. Henry Ford was right.