Back in the 70s when the 25-06 was a newly "legitimized" wildcat, a good friend got a model 700 Varmint, and loaded 87gr HPs over a case full of IMR 4350. Saw him blow a woodchuck apart at a measured (later) 517 yds.
Decades later, the .25-06 that finally fell into my lap wasn't a pure varmint rig, but a converted 03 Springfield with a medium heavy barrel and a varmint style stock and a VERY light trigger.
I don't use it often but its a freakin laser beam flat shooter (or nearly so it seems). Did learn one thing, not all 87gr HPs are created equal. And in my area aren't common on the shelves, 100, 117 and 120s are everywhere but the pure varmint slugs, not so much. (yes pre-internet days).
the bullet I got (Hornady, if I remember right) had a warning in the box, not to exceed 3400fps. This is a couple hundred fps short of what the 20-06 can do. SO check CAREFULLY when choosing a varmint bullet. Right bore size and weight are only part of it.
The .25-06 was, for some time, the definition of an overbore cartridge. Slower powders work really well, and of course individual rifles vary so what performs best in yours is something you learn by testing different ones.