So, without being able to actually see what is happening, it sounds like you are saying the fastener is tight, but the item still wobbles. In that case, I'd say the wobble is due to a mismatch between the mating surfaces of the two parts.
Trouble is, you don't know there the trouble is - the grips could be nonplanar, or the grip frame, or both. So removing material may not work out.
I'd try adding some malleable material that will conform to the two surfaces under pressure and fill in any gaps. Assuming that the grip screws put pressure between the grips and grip frame, one suggestion would be to try some single layer grey cardboard, and slide that between the two of them - that might be 1/16" or thinner, and may be able to get rid of your wobble without any risk of making matters worse. Another material that might work would be a bicycle inner tube (just cut to a single layer) - it's about the same thickness, but will "flow" better than the cardboard under pressure.