First, I assume it's a Lyman reproduction rifle, rather than a vintage original of some kind?
Lyman's reloading data (Lyman Reloading Handbook) recommends 60 grains of FF with a 505 grain Minié, which is quite a light load actually. (Go with their recommendation, not mine!)
Here are some "standard" hunting loads for various cals using patched round balls.
.45 -- 70 grains FFF
.50 -- 90 grains FFF
.54 -- 110 grains FFF or FF
.58 -- 140 grains FF
I have an H&R Huntsman in .58 that H&R recommends no more than 100 grains with any projectile, but I've shot round balls with 140 FF in it a lot, as well as maxi's & Minié's at 100 grains. You don't gain much with that extra 40 grains. Lyman's own test data (Lyman Black Powder Handbook) using a Universal test barrel goes as high as 190 grains FF with a 460 grain Minié, but the deformation of the skirt at those pressures makes an already inaccurate bullet design even worse.