It's the mechanics of the press.
Your case sizing station is more or less across from the bullet seating station,
The shell plate can and does wobble back & forth depending on if you are trying to compression size a case or not.
The shell plate is off center the ram, sticks out beyond the ram,
And there MUST be clearance for the shell plate to rotate freely.
When the sizing die is pushing a case into shape, the sizing die has leverage over the seating die, raising it up a little, getting deeper seated bullets.
When no case is in the sizing die, the seating/crimping stations have the most pressure on them,
The shell plate isn't being forced up into the seating die as far,
The result is shallower seated bullets.
Tightening up the shell plate bolt will remove just a little of what you are seeing,
But if you go too tight, you will have issues with cycling and the press 'Dragging',
The plastic indexer cam under the press plate won't live long.
A bearing on top the shell plate will help some (there is no room under the shell plate without machining a bearing well into the press head),
But they usually cause issues with the 'Eject' spring/wire that ejects finished rounds.
I have destroyed 3 press heads trying to support the sizing & seating dies on my XL650, and I've cleared a little of the problems up,
I finally broke down and built a stand alone case sizing press that produces as close to 'Perfect' cases as can be expected from once fired brass.
The problem you have found is only compounded as you feed the press larger/heavier cases to process.
As you get into larger cases, .308 & .30-06, especially the older/harder brass, the issue only gets worse.
Even the Dillon 1050 won't whip the older military .308 or .30-06 cases reliably,
And you can forget about belted magnums...
Every machine has its limits, I have to say the Dillon XL650 comes real close when compared to other machines, and it's smoothness of operation puts it on top the heap of production of ammo,
But like all machines, it has its short falls...
Feed that machine properly sized brass, and it will crank out as close to premium rounds as you can expect to get anywhere.
On par with a Rock Chucker in most cases...
Feed it brass that was fired through a blown out chamber, or the old surplus brass that's really hard, and it just isn't enough press to whip the brass.