I started with 357 and most responses will be for 357, but I sold all of mine a few years ago and used only 44's for years. I did eventually buy another 357, but only because I ran across a great deal on a Smith 28.
Here is why. I owned both Smith 686's and Ruger GP-100's. They are good guns and the 357 is a good round. But after buying a 4" 629, the 357's just didn't make sense anymore. Because the 629 didn't have a full length lug, it was actually lighter. It is only about 1/8" thicker through the cylinder and would in fact fit in many of the same holsters. With proper grips, there was no difference in grip size.
You don't have to always shoot full power 44 loads. I firmly believe that hot 44 specials, or mid range 44 mags are MORE effective as a fight stopper, and they do it with less recoil, and muzzle blast than hot 357 loads. Not that much difference in ammo prices either. If you do need full power loads, the 44 takes it to another level 357 cannot begin to approach.
After doing a little shooting over a chronograph I've decided neither 44 or 357 makes any sense in barrels shorter than 4". And much longer is better, so the small snub 5 shot 357's with 2-3" barrels are out. If I need something small a comparable size 9mm semi will actually shoot faster than a short barreled magnum revolver. It will be more compact and hold up to 3X more ammo.