I've NEVER used, or heard of, any specific direction that straight, solid pins must be driven to. Tapered pins, are, of course another story. And, it doesn't matter whether rifle, shotgun or handgun, drive the straight pins in a direction that's convenient for you and your bench set-up.
Using the right punch is always a good idea when driving pins. Keep in mind that there are solid pins along with spring and slotted roll-pins used in many firearms, and those "hollow" pins need to be handled with a punch designed for that type of pin.