It could be one of several things or a combination of them. They have all been talked about such as technique, hand/finger strength, and individual magazines.
My HK P-30 magazines take more pressure to load than say my Sig 226's. My PPQ's are fairly easy until the last round, which takes a lot of pressure. I don't seem to have any issues hand loading the magazines on any of my rifles, shotguns, or handguns. I do use an UpLULA and it makes loading way easier and faster. Try one, you'll likely end up with one.