The sides don't really rub on anything.
The paddle, where the sear spring and disconnector ride on it, can sometimes do with some clean-up, there's usually an almost-invisible mold separation line right where the sear spring rides, and the area inside the window that bears on the sear pin might benefit from a little polishing.
I did quite a bit of prep on a Colt disconnector before installing it in a non-Colt, but it probably didn't really need it.