litenite99, Thanks for the clear explanation. You should just make PF with your 4" XD. Of course, do your safety work-up.
Actually, I think your approach is right on target - pun intended. Power Factor requirements for the various shooting sports disciplines, favor heavy bullets, mostly due to the math. It's simply weight x velocity. Rather than actual muzzle energy, which is weight x velocity-squared. Muzzle energy more closely parallels felt recoil (over-simplified, yes, I know) than just straight PF. So yes, let the bullet's weight acquire as much of the factorization as possible. Hope that made sense.
And, you want to use a propellant that is as fast as you can safely make PF - because all else being equal, a slower propellant will yield more "thrust recoil" than a faster one.
So you're right on track. Well played.
Load safe.