"Sticks and stones may break my bones, but whips and chains exite me"
There is a lot of crap thrown about these days, mostly by those who are out to prove something either to themselves or those with them. Unfortunately if you don't feed into the ruse and attempt to walk away from it, you disappoint the immature bastard and his frustration leads him to want to escalate the situation.
As a former Navy SF member, black belt, and carrier of firearms, I am screwed any way I turn. I walk away. If you follow me, you'd better be prepared to face the consequences. I refuse to have a battle of wits with an unarmed man.
First thing to do... keep walking and ignore the twit. Whatever happens next in the situation keep moving ( a moving target is harder to hit) If you are attempting to distance yourself from the idiot during the entire confrontation and witnesses see this it goes better in court.
If the situation escalates and you feel there is no quick exit, either take out your cell phone or tell someone else to call for LEO backup ...make sure any witnesses hear you telling him that you don't want any trouble. If he still presses but is keeping a reasonable distance and you still feel a threat, inform him (without showing) that you are armed and give him room to exit. Next show him proof of arms, and last resort draw and be prepared to fire.
This is the worst case scenario, most of the time the idiot doesn't think you are worth the time because you didn't respond and will spend his time puffing himself up to his friends because you were "scared" of him. This is the "walk away knowing that if he only knew" scene.
The bottom line is, most punks are just looking to get a rise out of someone, (whether it is driving accross your lawn, yelling at you in the crosswalk or whatever) when it doesn't happen they move on. Chalk it up to adolescent overindulgence in testosterone, or alcohol (I havent figured out which is worse) and walk away knowing that you can handle the situation but you chose not to.
"Age and treachery will always overcome youth and skill."