I've got a Super Blackhawk Hunter in .41 Magnum, very good shooter and the .41 doesn't give up much to the .44 Magnum at all yet burns less powder. Recoil isn't bad at all for the performance it dishes out, so don't count out the .41 magnum!
I've also got a BFR in 454 Casull and a Super Redhawk Toklat 454. The BFR really is a very nice handgun, nicer than any Ruger single action and they come with a scope base. If you look into what it's capable of, the BFR is fantastically accurate. For the features you get, the BFR is a great buy.
As for cartridge, to me it's a balance between power and efficiency. The 500 is big and bad but it eats powder like nobodies business. The 475L and/or 480 Ruger is nice but the killer is bullet selection, the .500 has more bullet options than the 475/480. WIth the 475/480 it's going to be 95% cast bullets only, which isn't bad, but that's not good variety.
To me, for the best array of bullets from jacketed to cast, the 454 is the way to go. The BFR is a 5 shot and of course you can shoot .45 Colt in them too, and 5 shot .45 Colt level loads too! The BFR cylinder is longer than a SRH cylinder so you can play with even more powerful loads since you can load to a longer OAL.
You don't have to load the 454 to the max either, I've loaded up some real nice mid range 454's that shoot easy and hit hard, or you can load it on up if you feel the need for recoil.
I'd recommend the 454 if you're going to reload. I prefer single actions and would HIGHLY recommend the BFR but if you're more into double actions the SRH 454 is a good gun too, my Toklat 5" is very accurate.