An ejector of the Remington type can put an uneven pressure on the cartridge case and I can see it might have a slight effect on consistency, which it the name of the game in shooting for pure accuracy. The problem is not due to "side pressure", though, but to the ejector pushing the cartridge case forward.
I am not sure the effect will be positive, though; it might turn out that removing the ejector will allow MORE case movement and hurt accuracy. Let us know the results.
Jim