Front or rear sight? Its not unusual for a dovetail front sight to be pinned.
Padding like carpet is not your friend. A good solid vise is an important tool for gunsmith work. The vise needs to have smooth jaws. You can do pretty well with a couple of pieces of aluminum angle set over the jaws as covers.
Put some painters blue tape on the sides of the slide.Position the slide so the sight just barely sticks up above the vise jaws. You might be able to position the slide so your punch can rest on the top of the vise jaw.
Be very aware the slide is not solid. The inside is unsupported. You certainly can crush the slide with the vise.Its unnecessary to clamp hard.
You want to use the back solid jaw of the vise as your anvil. The moving jaw just holds the slide against the back jaw. There is no need for a lot of crush.
You just need the slide against a dead solid anvil.
A lighter weight,higher speed "whacking" well serve more to batter or peen the sight rather than move it. A slower,heavier hammer works better for drifting.
I generally prefer a brass punch,a rectangular bar is best,And I dress the end clean and square.
Before you start,its good to know how far you want to move the sight.
I figure my range to the target in terms of the sight radius, If the range to the target is 100 sight radii,the movement of the sight will be multiplied 100 times on the target.
With a depth mic or calipers you can measure the change.