You didn't say whether you have the plastic stock or not; I am rather underwhelmed by them.
Anyway, the average sporter-weight barrel in a plain-vanilla rifle will do quite well for such uses as deer hunting. "Tweaking" in various ways can improve accuracy or consistency, but not always.
Ideally, one should let a barrel return to ambient temperatures before firing the next shot of a string. Failing that, a rather slow rate of fire for a group is okay, but then let the barrel cool before firing another (for example) five-shot group.
If you DO let the rifle cool between groups, and the change in point of impact always occurs and always stays with you after the first group, I don't know why--that's new to me. As far as the enlargement of the size of the group, I suspect that your barrel might be sensitive to cleanliness. It might be a bit rougher than one would ideally hope for, and foul a bit more readily.
I suggest using a copper-solvent bore cleaner. During your next trip to the range, take your cleaning stuff with you and clean between groups just to see if it makes a difference.
FWIW, Art