First, I am assuming that you are using the same loads or factory ammo for each session and finding that the POI changes when everything is being kept constant.
I find that when testing different loads, the POI will change with velocity and seating depth changes even when using the same bullets and powder.
If I change powder and bullets, I see even more changes in POI with every load change even in the same session.
I you keep the conditions constant with respect to ammo - brass, bullet weight, powder charge, seating depth, and primers - then try these experiments and see if they cause changes in your POI.
Test 1 - Go through your regular drill to set up on your bags and set up on your point of aim.
Do whatever breathing and set up approach that you always do.
Then, with your cross-hairs on your POA, close your eyes for 10 seconds.
Open them and see if you are still on your POA.
If the new POA is to the right, shift your body slightly to the left and you'll see the POA move. Shifting your body in the opposite direction will move the POA the other way. Setting up for each session or ever for each group can cause you to aim by using your muscles instead of your set up. That will yield different POIs even though your POA appears to be on target.
If the POA has moved after you relaxed, your POA changed because you had adjusted your POA using your muscles. When your muscles relax, your POA finds your resting position - that is called your natural point of aim and that is where your bullet will impact even if your POA set up controlled by your muscles is slightly different. If your POA didn't move, you were at your natural point of am and muscling isn't the problem.
Test 2 - go through your normal set up.
Just get the shadow in the scope to go away and take your shot. See if the POI is at your POA.
Then move your head slightly closer to the scope and take the second shot and see what happens to the POI compared to the first.
It should be off the initial shot. If you always are exactly in the same position behind the scope - right on the center line, then scope positioning isn't your problem
Third test - press the stock tighter into your shoulder and see where the next shot goes. The POI should change again.
If you always can manage to keep the stock pressure on your shoulder identical from shot to shot and session to session, then this isn't your problem.
What could be happening is that your set up for each session is slightly different - closer to the scope, a bit more muscling of the rifle, or having the stock tighter into your shoulder. Your body position and what you 'feel' is right might be slightly different during successive sessions and it will cause your POA/POI to be different, even for the same ammo.
If you clamped the rifle in a test stand, you probably wouldn't see as much effect caused by positioning, muscle, and set up varitions, but the shooter is included in the rifle set up and unless the shooter can be sure that the set up is identical for every session, variations might be creating the conditions the would move the POI compared to your POA. Yes, the variations can also include temperature, humidity and other external conditions. I have long since given up expecting my set up to be identical for each session. For that matter, I find that when I start at the range I will set up slightly differently than when I am tired toward the end of a long range session.
The changes aren't on the order of inches in POI but they can cause variations of an inch in cases where multiple effects combine. I do not have the body control that really great shooters have but I don't do all that badly even though I know my own limitations.
I find the I have to regularly 'go back to basics' in my set up to get the best results even during a session.
I find that from session to session I have to find what eliminates most of the muscling that might be occurring, reestablish the right position behind the scope and get into my natural point of aim position if I expect to minimize variations in my POI for a given POA. Once I find the 'zero' for that given day or portion of a session, I get into the zone and, if I pay attention to the details, concentrate on my breath control and stay in the same position behind the scope, I get really tight groups and they hit where I was aiming.
If I get distracted, get lazy, or even start varying how tightly I press the rifle into my shoulder, I can get groups or even individual shots vary POI from my POA.