Or vice versa.
What throws people off is that all the error elements preventing them from shooting bugholes have a normal distribution of some kind. A group is a bivariate normal distribution. When the groups are round, you can place the axes conveniently at horizontal and vertical. When the group is oblong, you place them perpendicular to one another along the narrow and long axes. Each axis then has its own bell curve.
Consider the round group to have the convenience of placing the perpendicular axes wherever we want them, and so make them horizontal and vertical to have the familiar Cartesian coordinates. Each hole location's distance from the center is the hypotenuse of a triangle whose right sides are the horizontal and vertical errors present for that shot. By Pythagoras, the length of that hypotenuse is the square root of the sum of the squares of the two right sides, or the horizontal and vertical errors in this case. Thus, you see the way two independent standard deviations add is as the square root of the sum of their squares. This turns out to be true even when the standard deviations being added are along the same axis, because there are times they will add and times they will subtract from one another, and, assuming normal distributions, they will tend add from close to the center more often than they add at extreme deviations.
So, suppose we play the SD addition game with independent sources of error. If you have two independent normally distributed random sources of error, each of which opens a bughole rifle's groups by one moa by itself, and you incur both sources of error at once, you wind up with a group size that is not 2 moa. You wind up with √(1²+1²) or √2 or a group 1.414 inches.
Now suppose you introduce another 1 moa error source. Now you have √(1²+1²+1²) or √(3) or a group of 1.732 inches. The first 1 moa source made a 1 moa group. Adding the second only grew the group 41.4%. Adding the third only grew it 22.5%, and fourth 1" source would add 15.5% to that, and, BTW, it has taken all four 1 moa error sources combined to double the group size. The same would apply if we worked with ½moa error sources, except it would take four of them to get to a 1 moa group. Whatever sources of random error you add up, their effect on group size diminishes as the number of other sources of random error are increased.
But most shooters come at this from the other direction. They have a big group that already is comprised of multiple different random error sources and many of them are not as big as others, so they affect the total group size even less than in the previous example when I add them to the pot. They could be anything from crown imperfections to inadvertent stock contact points, to uneven bolt lug contact, to out of square bolt faces and off-bore-axis chambers, to miscellaneous recoil moments due to asymmetries, to bad ammo with imbalanced bullets, to sights that have their adjustments jiggle around slightly under recoil, etc.
Example: Suppose all the random error sources in your gun combine to give you an average 1.5 moa group size. Now you remove enough average bullet runout from your handloads to eliminate what would be an average of 0.5 moa of error if it were the only error source you had. The effect on that 1.5 moa combined error source group will be to reduce it to:
√(1.5²-0.5²) = √(2.25-0.25 = √2 = 1.414 moa
So, you took 0.086 moa off the group size by removing a 0.5 moa independent error source. Given how much your group sizes already vary around the 1.5 moa average, could you tell the difference was real? You would probably have to shoot some inconveniently large groups to tell. You could use Student's T-test to work how how confident you could be that your 1.414 moa group was really smaller than your 1.5 moa group and not just a randomly smaller result due to normal group variation, but the confidence wouldn't be very high if the groups were only 5 rounds each.
Note that ammo is only one of the possible sources. And its own error will be a collection of sources, like primer seating errors, flash hole burrs, wall runout, axial runout of the loaded round, charge errors, bullet quality, etc. So, is it any wonder that when you remove just one of those sources of ammo error the effect is so small that you can't see it in your overall group size amid the variation they already have? Indeed, something as small as deburring a flash hole might not make a visible improvement until you had eliminated all the other error sources and had only that last 0.1 or 0.2 moa left to remove.
What I'm getting at with all this is, just because an accuracy step that eliminates a source of error makes no obvious improvement in group size, doesn't mean it didn't make an improvement. Indeed, you may well have to eliminate several error sources before the group starts shrinking to an obvious degree.
Instead, what many folks do is try eliminating different sources of error, then cease eliminating them when they move on to eliminate the next one, trying to keep the variables independent. They repeat until they remove one error source that makes an obvious improvement, then in the future continue to ignore everything they tried that didn't make an obvious improvement in the original group. But what they've actually done is try different things until they happen to fix one of the larger error sources in their shooting system, so the improvement effect was visible. Instead of blowing off all the other techniques, they would then need to go back and try them again to see if this time they make a visible improvement to the now-smaller group. The smaller the group gets, the more apparent the effect of removing small sources of error will be.