long ago
When I was a kid, as the lever and pump gun era ended, my great uncle began to deer (whitetails) hunt with a Rem 760 in '06, (after shooting a .35 Rem pump for decades) and early on declared the "180 gr RN in the '06 the best killer in the woods". And many believed and loaded their own rifles just so, as well as .308's (that would be me, having retired a twice handed down .30 Rem pump) And .270 shooters tried to emulate and loaded 150 RN in an attempt to follow the declared best path.
And I quickly concluded that a .308/180 kicked a bunch in my M88 Winchester, and when shot for group at 100, did not group nearly as well as 150's (the M88 in .308 was twisted 1-12) and the 150's kicked less, important to a kid. So I deviated from the path....... shot light bullets, with less kick, more accurately. But ol'uncle Harry never gave me much credit after that.
All that to say that even a very big whitetail will not survive a .30 cal slug in the 150-165 range running 2800-2900 fps or more from the '06. Find a load/make that your rifle prefers and have at it. The '06 is much gun for the average whitetail, note not "too much" but really more than one likely needs. And a full house .30/180 in '06 begins to get in the range of recoil (in a decent sporter) that most begin to notice. No point in getting beat up in the process when you don't have to.
As far as getting beat up goes, I just spent the afternoon shooting an '06 bolt rifle, (Mark X Mannlicher) and a .308 bolt rifle Savage Hog) , both with 20" bbls. Both like 180's....alot. Heck, the Mark X likes 200's!!!!! Like sub MOA. So accurate with the heavies that I cannot turn my back on it. But after 30 rds or so, I was not enjoying the process, and wished like all get out that those two stubby blasters would shoot 150's!!!!!