The exact experience with the .30-06 was 1964 National Match ammunition (government magnum primer) with a charge of 46 grains of IMR4895. That filled about 81% of the space below its bullet. In one gun (production military barrel), tipping the muzzle down before bringing the gun level to shoot (putting the powder forward over the bullet), the average muzzle velocity was about 2530 fps. But tipping the muzzle up before bringing it level, putting the powder over the primer, it averaged about 2610 fps and the primers went from very round to pretty flat.
In another club gun with a very shot out chamber, it the velocity shift was 2495 fps and 2572 fps. A third gun was in between them. But the roughly 4% drop in velocity was consistent. Mind you, those are average velocities, so the extreme spread was higher, though I didn't record it for some reason. I can guess, though, that it was probably around 140 fps.
My point is simply to inquire if you are handling the ammunition to keep the powder position the same at each firing? If not, with 80-85% case fill, you could expect a significant difference.
Another thing to check is that your primers are all seated consistently. Ideally, you would press them in about 0.003" past the point where you can feel the anvil feet touch the bottom of the primer pocket. You could also try a different primer if you have some available. Occasionally, a magnum primer can make things worse.
For neck tension, you can test the effect of an increase in start pressure by loading with the bullet touching the lands. You should drop the powder charge 10% and work up doing that. It will increase start pressure a good deal more than increasing neck sizing will do.