The people are getting a little silly but they are exciting guns to shoot. First shot fear is the biggest hurdle, but it all boils down to rifle weight. I had a 505 Gibbs once but I only shot cast bullets around 1200 fps, and a 458 WM that didn't weigh much more than a regular rifle. With full house or factory loads, it definitely announced itself. The most powder I ever burned was 416 Rigby in the Ruger Magnum rifle, with many loads over 100 grs. The Ruger was over 11 lbs and could handle much more pressure than old British loads - 300 gr bullet at 3000 fps, 400 gr over 2700 - but it didn't really kick as much as sort of pushed you, not unpleasant at all. The rifle that hurt me the most was actually a light European sporter in (drumroll please) 308, or maybe the 7 X 61 S&H in again a fairly light rifle.
Something like this was posted somewhere a few years ago. I think it was in a ballistic lab, people shooting through a little window in an acoustic panel room. I think it was the 577 TR, but the rifle didn't look especially heavy. Several people dropped it, or had it fly out of their hands. One guy got bowled over, rolled back and shattered a plastic partition behind him.