Theres no doubt, IWB takes a little getting used to at first. It initially doesnt feel right, and your body needs to adjust to the holster being there. Once you give it a little while, and find that "sweet spot", things are usually a lot better from there on out.
For me, keeping my normal pant size works a lot better than going up a size. Things are slightly snug when you first insert the holster, but the fabric soon "gives", and it feels good past that. The holster feels a lot more secure and theres a lot less movement this way, then when your pants are "to big".
Another downside to wearing one or two sizes larger is, your pants dont look like they fit, with or without the holster in place. The waist isnt the only part they make bigger, and your pants just dont look like they fit right.