Several ways of dealing with the issue. You can clean the rifle, go to a benchrest, and see where the first shot hits, and then--natcherly--see where the next two or four shots hit.
If numbers two and onward are notably different from number one, and your primary purpose is hunting, I'd zero to the impact point from the cold, clean barrel.
Another way is fire a fouling shot from this cold, clean barrel, and then go off for a cuppa kawfee and then shoot a group. You then know what happens when the barrel is not-really dirty, but cold.
I've always preferred the second method for my hunting rifles. Unless I've hunted in rainy weather, I never clean a hunting rifle during the season. Maybe a dry patch, against dust...
This sort of possible behavior, of a cold-barrel shot versus warm/hot is important. My father built a Springfield sporter, not long after WW II. The first shot from a cold barrel was ALWAYS two inches higher than the next ten shots' fairly tight group. He twiddled, tweaked and rebedded, but it made no nevermind over the next 40 years.
Didn't keep him from collecting large numbers of Bambi, however. Know your rifle.
, Art