I shoot foulers. I have, and do, clearly see a point of impact change with my 22 lr match rifles. I will arrive on the line with a clean barrel and have tested this by not changing my sights. The first, second shots seldom land on each other.
I regularly blow a couple of foulers into the berm at a Bullseye Pistol Match. It does make a difference, the first shots out of a clean oiled barrel, hit a different point of impact.
For a centerfire rifle, this is tricky. You have to test and see if foulers are necessary. Before CMP Talladega, I shot in NRA matches, I shot the course of fire, and it always started at the 200 yard line in the standing position. With high mileage barrels, that I deep cleaned, removing all copper fouling, I could "see" a zero change as rounds went down range. But, considering I shot standing, and standing is a horrible position to use if you are trying to determine anything about accuracy, what I saw, could be bogus. But I saw it enough times, I think it was real. Therefore I do believe that the point of impact out of a new barrel is not as sensitive to barrel fouling as an old barrel.
However, since CMP Talladega, I have experimented with greased bullets, to see if my heavily greased bullets and cartridges have a stable point of impact, and I think they do, in the right rifle. This was a first shot down range, 300 yards, using established zero's with a known load. I made my best guess about the wind, and first shot was in the X. Yippie!
Point of impact did not change, I am going to claim, as the barrel fouled for a ten shot string.
This rifle is exceptionally accurate, and it may be non representative of the larger population of rifles.