It happens, but most of the time the difference is so small than most rifles, and shooters aren't good enough for it to be a factor. Especially if you are talking about the level of accuracy needed for a hunting rifle fired in field positions.
To help eliminate as much possibility as I can I always zero my scope on the highest magnification. If I need to take a longish shot that is where my scope will be adjusted anyway. I then practice with the scope set at the lowest and highest settings just to know. I NEVER use any of the settings between highest and lowest.
On shorter range shots, out to 100-150 yards I do just fine on 2X or 3X and minor differences at those ranges aren't enough to cause a miss. On anything longer I go straight to 7X, 8X, or 9X depending on the scope where it was zeroed. While the differences at the mid range settings are likely to be small, or not at all, It is just one less variable to worry about.