Barrel length, presence or absence of a barrel-cylinder gap, and bore diameter all affect sound amplitude considerably more than the velocity of the projectile does.
For example, a short-barreled .357 is several dB louder than a 16" barreled .223 Remington, even though the .223 bullet is pushing Mach 3.
I have felt nothing like the same ear-splitting crack when shooting next to others shooting 45s and even .44 mags. Maybe the .357 in a snubnose is a particularly bad combination for ear damage.
My experience is similar. The reason that it is SOOOO loud is (1) high pressure cartridge, (2) 35,000psi gas venting laterally from the barrel-cylinder gap, and (3) super-short barrel.
It's not the bullet velocity that makes it so loud---plenty of .22LR rifle loads exceed the velocity of a full-house 125gr .357 out of a 2" barrel, generally around 1250 fps---it's the pressure differential between the escaping gases and the air, which is arguably greater for a 2" .357 than almost any other firearm, at least until somebody makes a .44 magnum or .454 Casull snubby.
BTW, to get a feel for the barrel-cylinder-gap noise, if you have ever experienced the ears-plitting shriek/hiss of high pressure compressed air escaping at high flow rates through a valve, or a high-pressure gas cylinder (typically 1000-3000 psi) dumping pressure, imagine what it would sound like dumping
35,000 psi through an annular slit. That's why revolvers are louder than fixed-chamber firearms of similar caliber, because you're getting more than just the muzzle blast.
Here is an actual Schlieren photo of the blast waves produced by someone firing a revolver (I think a .357):
http://www.hightech-edge.com/explos...shockwave-images-high-speed-videography/1684/
Note that the initial shockwave bubble is centered on the barrel-cylinder gap, and that the shockwave from the supersonic bullet is much smaller and weaker, and is traveling away from the shooter so that it does not significantly affect the overpressure at the shooter's ears. Also note that the bullet is already outrunning the sound from the barrel-cylinder gap and the muzzle venting.
Just for comparison, here's an AK:
Note that the bullet's supersonic Mach wave is traveling away from the shooter and doesn't contribute much to the sound the shooter hears.
http://www.nytimes.com/slideshow/2008/10/27/science/102808-Cough_index.html