Really? I haven't shot the M79, but I've shot the M203, and to me, that was about as bad as 3" magnum 12 gauge.
Not quite as bad as shooting off parachute flares or star clusters, though.
On the other hand, I could shoot AT4s all day with no apparent ill effects from the blast, and I know people who hate those.