Any time your ears ring, you cause yourself hearing damage. It is cumulative and non-reversible given today's medical technology. They're working on some amazing things to reverse it, but do you want to gamble that it's going to happen in your lifetime?
Like Brian said, I could see occasionally shooting a .22 bolt action rifle without hearing protection if you were doing a shot or two. The SPL of a .22 bolt would be somewhere around a loud car or movie theatre. My CZ 452 is the only firearm I shoot with just the ear plugs. If I shoot anything louder (like a 9mm handgun) I wear plugs and muffs.
You can lose hearing from doing things other than shooting. I play in a hard rock band and anytime I practice with live drums in the room (or an enthusiastic guitarist) I wear hearing protection. I've heard other musicians say that even some classic cars (Corvettes) require hearing protection to drive more than 20 minutes with the top down.
We're all going to lose a little bit in our lives but you don't want tinnitus or permanent hearing loss if you can avoid it at all.
This is incidentally one of the reasons I am dubious about using a 5.56 short barrel with a muzzle brake for home defense, but that's another topic altogether.