Sorry I got no help for you with cleaning up the area. You used compounds with volatile solvents in them, and got a reaction. It might be the rough surface, trapping the light color in the "dips" and not allowing the remover to do as good a job as it could on a smooth surface.
It could be a chemical reaction to the finish, or the underlying metal or polymer. I just don't know. My guess would be tiny bits of pigment in the low spots of the rough finish, giving a lightened color.
I don't know the correct name (or where to get one) but many years ago a friend of mine had a color "pencil", (about the size of a carpenter's pencil) made up of a (fairly hard) waxy material and pigment. No volatile chemicals. He had Gold and white. Told me the gold one was expensive, because it was "real" gold flecks in it.
One just rubbed it over the lettering, and then rubbed off the excess. I had lovely gold lettering on one of my S&Ws for years with a single application. OF course, this was on a smooth surface, as well. A rough surface would likely make cleaning up the excess tougher.
Good luck!