Now we're cooking with fire . . . .
I mentioned a gunbelt earlier, and now that I know that you probably don't have one, let me add a few comments. I started carrying about 4 years ago, and didn't have a proper gunbelt when I started. IMHO, a proper gunbelt is a huge improvement. The belt is the "platform" on which the rest of carry rests, quite literally, so having one that doesn't sag or roll under the weight of the gun is important.
I'm in slacks and a jacket or a suit at least 5 days/week, so I need belts that don't scream "gunbelt" to the world. I use an Aker B21 when I need a brown belt and a DeSantis Econobelt when I need a black one. That means no heavy black leather with a basketweave pattern (like LEOs wear). No nylon instructor belts for me, either. Most of the big gunleather manufacturers (Aker, Galco, DeSantis, Bianchi, etc.) make belts, so that's a good starting place. However, depending on your budget, there are also plenty of smaller companies that are reputed to make an outstanding gunbelt. In particular, "The Beltman" comes to mind. I've never used one of his belts, but I've also never seen anybody complain about them, either.
IWB vs. OWB -- Over the last 4 years, I've carried guns ranging from a snubnose revolver to a full-sized steel semiauto. What and how I carry depends on the weather, wardrobe, where I expect to be and go, etc. These days, I carry a G19 in a Crossbreed Supertuck most of the time. While your XDM is slightly larger than my G19, the XDM is certainly a manageable size, ..... if you're willing to dress around the gun a little. The XDM 3.8 compact (which I've also been eyeballin') is very comparable in size to the G19 and I have to dress around my G19, too. So expect that if you change guns. At any rate, the CBST is exceptionally comfortable, as is my Theis holster. Tommy Theis makes kind of a CBST clone, being a leather or horsehide back, kydex shell, IWB tuckable. I also have a Galco OWB leather holster. A few years back, I was pretty much a dyed-in-the-wool OWB leather guy. Now? Not so much. While I do still carry the Galco once in a while I find myself more and more using the CBST. I like the "click" when the gun seats in the kydex, which (obviously) leather doesn't do. I also like the fact that, with the CBST, I'm not necessarily committed to wearing a jacket. For example, if I wear a jacket to work, but my wife and I go out to dinner after work, I can slip into the bathroom, tuck my shirt in over the CBST, and I'm good to go. With the Galco, I have to wear my jacket through dinner.
I'm in and out of "gun-free" locations all the time. City Hall, courthouses, etc. That means that there are lots of places I can't carry, and I don't really want the metal clips on the CBST to set off the metal detectors. That means that I'm also in and out of my holster pretty often. With the CBST, that's easy. With the Galco, I have to actually undo my belt to get it off. Not a huge headache, but slightly more trouble than the CBST. OTOH, the Galco wouldn't set off the metal detectors to begin with....
I also have a Fobus paddle holster that I rarely use. In theory, that should also be a pretty easy on-&-off. Unfortunately, it can be a little too easy. By that I mean that if I'm wearing my Aker belt, the holster comes off when I try to draw my gun. With my DeSantis belt, no such problem. I guess my point is that whatever you go with, be sure to try the holster with your belt for both carry and drawing. You'd really, really hate to discover at an inopportune moment that the holster came off of your belt, all the while keeping the trigger safely covered . . . .
If you're going to belt carry, I'd also suggest avoiding any holster that is designed to fit a "class" of pistols. IOW if it says "fits large semiautomatics," don't. You want a holster that is made for your gun, not guns in some generic range of sizes. It needs to say "Fits XDM 4.5-inch." I wouldn't necessarily say this about finding a pocket holster, but for belt carry, I do. The ones that fit a class of pistols tend to be cheap ones. Many folks who have the proverbial "Drawer of Unused Holsters" tried to save a few bucks by getting something like this. Those holsters wind up in the DUH (Drawer of Unused Holsters). Spend enough to get something actually made to fit your gun, and you're more likely to get years of wear out of it.
Good luck in your search and let us know what you find and how you like it.