I had a problem generally with using speedloaders with my GP100. I had the rubber grips with wooden panel inserts first, and then I went a Hogue full rubber grip.
I know exactly what you mean, and I have seen a few modifications people have done that had the same problems. Some methods were a bit brutal with them basically destroying the rubber section that the speedloader came in contact with. Not a good look. Some 'file' the rubber down to make it a little easier to slot in.
I personally have done nothing to mine and don't ever intend to. I know the speedloader is not going to slot straight in perfectly, but who waits to get it in all the way when they need to do a quick reload? I get the speedloader in a position where the rounds at least have the projectiles inserted into the cylinder - push it down a little bit further and twist the knob. The rounds fall into place... close the cylinder and we're back in business.
I've also got those Safariland Comp III spring loaded speedloaders that are really good, but that technique doesn't always work, as you know it requires the speedloader 'button' to press against the ejector star to release the rounds into the cylinder. With this case I just push it as far as it goes to eject the rounds - even if the speedloader is not evenly centred against the cylinder.
The only thing I have against rubber grips something is when I palm strike the ejector rod to remove the spent rounds, and at least one round falls against the grip and won't fall to the gound unless you rotate the cylinder. Yeah its a petty little thing, but it can be frustrating at times when the clock is ticking.
Chris Cullen
