Kydex is known for picking up grit and embedding it. This scratches finishes.
Yup. I quit using a kydex holster awhile back.
The one I was using with one of my polymer guns wore a groove in one side of the trigger guard and completely took the texture off one side of the frame up near the dustcover leaving it shiny and slick.
Since then, I've been using a quality leather holster with that gun long enough to have completely worn one holster out and started using a second (much longer than I used the kydex holster with that gun) and there's very little wear from the leather.
You still want to keep a leather holster clean, especially if it's in a gritty environment, but it doesn't seem to be nearly as bad for attracting and retaining grit and causing wear on guns as the kydex is.
You can polish those marks out, but if you've never done any metal polishing, you're as likely to make it worse as make it better, in my opinion.
Getting the marks out is one thing--getting the polished area to match the rest of the slide is harder. You might have to polish the whole slide to get it to match.
I'm not recommending the Scotch Brite method, but if you use it, start with the finest, least aggressive Scotch Brite and try it carefully--if it looks like it's not aggressive/coarse enough to do the job, got a little coarser and more aggressive with the Scotch Brite.
Keep the polishing strokes lined up with the grain that is already visible in the slide.