The pin is meant to be a tight fit, since it's not a roll pin and there's no other method of retention.
They also like to collect a lot of crud and get a little corroded, from being at the bottom of the receiver where all the 'poop', oil, water, and lint ends up.
The pins usually come out easiest when driven in the conventional direction for firearm pins and dovetailed parts:
Left to right for removal.
Right to left for installation.
It may help to press the plunger in, but it shouldn't be necessary.
Try some penetrating oil. Let it sit for a couple days. Then revisit.
Use a decent size hammer, with solid hits. (No caveman brutality - just solid, meaningful hits.)
A light hammer with 'tappity-tap-tap' just flares the end of the pin and creates headaches.
And, of course, be certain the punch is small enough to fit through the hole.
