Correct. If you had to shave more than a thousandth off to square it, I wouldn't do it.
TL,
The angle error is something Martin observed. It's not a fix because you then have out-of-square heads fireformed. We had a reloader at The Shooters Forum a couple of years back with a bolt rifle (forget what make) who found some of his reloads would chamber just fine, but others he had trouble closing the bolt on. And sometimes, if he retried one that didn't fit, it magically fit on the second go or at least wasn't quite as hard to close the bolt on. I suggested the off-axis chamber possibility and the trimmer check, and he was able to confirm this is what happened to him. No accuracy issue with new ammo though.
The idea is errors that repeat consistently move the bullet the same way consistently. You are correct to think there is a certain amount of random noise associated with them. That's true just from the fact cartridges don't create perfectly consistent velocities so the recoil moving the rifle around off-axis moments isn't perfectly consistent, so any source of added recoil-induced error adds to the overall noise that way. But the magnitude of those inconsistencies seems to be mainly in the sub-half moa range so that most shooters who aren't competing in benchrest matches won't notice it or think their gun lacks precision because of it. Harold Vaughn goes through a lot of detail using accelerometers to find off-axis moments of inertia in his book Rifle Accuracy Facts. He also goes to a lot of trouble to eliminate those moments and he gets groups to tighten. He built an integral machine-rest gun that reliably shot 6 PPC through one hole at 100 yards, and measuring and eliminating those moments was part of the build process. He goes into detail describing tuning a Remington sporter in 270 Win down to a 1/4 moa rife using a lot of the same ideas.
