I'd like to drift the rear sight on my 2" BW .22mag by 1mm. I tried using a punch and a hammer but couldn't budge the dovetailed sight. I am concerned that I'll mark the finish and still not move the sight or that I'd warp the frame. Suggestions, please.