I've got a set of rubber ear plugs. I use the inner-ear headphones so I'm used to having something shoved in my ears anyway.
That being said, I've been out with my brother-in-law while he was shooting his S&W 40, 9mm and a pile of other including my 20ga. None of them bothered me. I did take one out while we were all reloading and I forgot to put it back in and I touched off my 22, that was loud by comparison.
Now, I'm not a fan of the foam ones. I found that if I had to take one of them out I could never get it back in right.