I've always dedicated certain screwdrivers to certain tasks. First off, I grind the end so the sides are parallel--getting rid of the wedge effect. ("Hollow ground" is needed whether or not you use a torque wrench.)
Small-handled screwdrivers for the small scope-mount screws. Say, no more than 3/4" diameter. Larger for the receiver screws, maybe 1" or 1-1/4". For the thumb-screws of a Weaver-type mount, a quarter coin does quite well, or the larger screwdriver.
I go by the feel of the screw as it nears its final tightening; you can feel the change in the resistance to turning. Odds are that as you get used to this, you'll be pretty close to torque limits or torque specs, anyway.
As near as I can tell, the most need for a torque wrench is for the Allen-head screws on a Remington 700; I understand the factory calls for a torque spec of 60 inch-pounds. (Is this correct?) Even with these, however, developing a sense of the "feel" of the screw as you near the tightness limit will suffice.
From what I've seen through the years, most folks tend to over-tighten. (Tire shops are the worst!
)
Just some thinking points...
Art