You pays your money and takes your chances. My .308 Handi rifle is the most accurate center fire rifle in the house using 180 gr Remington Corelokt. If I can't shoot 3 shots touching at 100 yards it's my fault not the gun.
For those who aren't shooting good groups what loads are you using, I find heavier bullets better, I also rest the gun on my hand just in front of the trigger, I don't know if it makes a difference but I am out-shooting my Springfield's, Ruger's, Winchester, Stevens and Weatherby with that gun. I shoot it in Hunting class competition and had it challenged as being a match grade target rifle but it isn't. Those guys challenging me had more money in just their scope than I did scope and rifle combined.
Only thing I question is shooting S&W 50 in a Handi-rifle. I have seen a video of a gun opening up after each shot with that round. On the other hand my H&R 10 gauge has no problem and I shoot it a lot just because it's so much fun it should be illegal. I haven't compared recoil between the 10 gauge and the 50 but I suspect the 50 might be a little stouter and could be a problem.