I'd go with Gary's "fix". How much undersize is your rammer in comparison to the chamber? Are you getting a good "shear" on the ball when loaded or are you also getting a little swaging to the ball in the area between the O.D. of the rammer and the I.D. of the chamber?
If you can take the rammer end off - chuck it in a drill press and slightly bevel the outside of the cupped edge. That should solve the problem.
Also, check your chambers as mentioned. That ball ought to be tight enough in the chamber not to pull easily. Check the chamber mouth for burrs or inconsistency in concentricity. When it happens - somehow mark that chamber and see if it is doing it not he same one all the time.
The rammer doesn't need to "wrap" the ball with the concave end. It needs to conform to the shape of the top of the ball to seat it. It's no different than a seating stem on a reloading die. The seating stem is contoured to fit the round nose of the bullet and that's what it pushes on - the "nose". From your photo, your rammer stem seems to have a hefty rim to it - ease the outer edge and see what happens.