I'd think for Elk, you have to worry about 3 things. #1.. Bullet Placement.. it dosn't matter if you have a .50 BMG Barret semi auto, if you can't get a killing shot, the animal isn't going down. (on the other hand, I've seen a spike bull elk killed with a 22 Magnum, that had perfect bullet placement, not that I recommend it) #2, use a premium bullet.. John Nosler after shooting at a Moose with mud encrusted hide back in 1946 with a well placed bullet from his 300 H&H and failing to have it go down designed the Partition to handle tough shots.. and #3, kinetic/kill energy.. if you flinch from shooting a large Magnum rifle, such as the Ultra Magnums, large Weatherby Magnums etc, your going to fail on that #1 most of the time and won't get a good clean killing shot.
Honestly the average hunter can handle a 30-06 without flinching, with it's average of 22 ft lbs recoil, but above that most people will flinch from the recoil.
As for myself, back in 02 I bought a "Uber Magnum" a Remington 700 in 375 Ultra Magnum, and hand load for it.. since then, I've bought another one for my son, my brother bought one as well, and for longer range shots a 300 Ultra Magnum. We'll take them out and "plink" with them from 30 to 100 rounds in a day (yes they are handloads, can't afford to buy that much ammo, not at the prices they are) and when were hunting, use the same rifles. I've seen deer fall out of the scope before when they were shot, so I know that I don't flinch when I shoot them.. Modifications I've done are mercury tube and limb saver recoil pad.. makes mine MUCH easier to shoot, especally with a scope on it. I fired my son's right after I bought, it straight out of the box, and it's recoil was much harsher, so it will be getting the same treatment before the next hunting season.