Its important to realize roll pins come in both inch and metric sizes.Trying to use inch rollpins in metric holes or the reverse will be trouble.
As I recall,AR pins are metric.For example,3mm is .118,and 1/8 in is .125.
Your local hardware store 1/8 pin will be a tough fit in a 3mm hole,even if they look about the same.
The pins do have some interference on the hole,and they are typically harder than the steel the hole is in. If they were not nicely chamfered,they would broach a chip as they are driven in.Think of that if you shorten a pin.
Often the pins are short and getting a good,square start is difficult.Anything less can be disaster.
Here is a tip.Take a fairly long (maybe 3 in) socket head cap screw or grade 8 bolt,hand grind the threaded end clean and square so a drill will start easily.
Simply drill a slip fit clearance hole for the pin maybe two diameters of the pin deep. A lathe works,but if you don't have one,use a drill press.Many drill press vises have a vee-groove in the jaw so the screw will be realtively vertical.
It makes no difference if the hole is off center,.
With that for a punch,its easy to get roll pins started square.Or you can but "roll pin starter punches" from Brownells/Midway.
A true roll pin punch has a little hemispherical bump on the tip that fits the end of the pin.That keeps the punch centered on the pin.Use one to finish driving the pin home.
As in many cases,using proper tools takes the frustration out of the job.
A finish nail for a punch and a crescent wrench for a hammer or a tweezers and a claw hammer does not always work out well.
I'm not a real advocate for roll pins,but they do compensate for some hole tolerance,they don't walk,they are tough,and frankly,where they are used is typically a place that the troop,LEO,operator,consumer ,is not "authorized" to monkey with.
Its someone at an "Armorer" level who should R+R roll pins,and they should have proper tools and spare pins.
The DIY peron ought to plan ahead,tool up,and get a pin kit.