I haven't tried anything below 25 grns, and have only used more energetic powders such as Olde Eynsford and Triple 7, though I've also used Pyrodex, and have found, with a ball or my 170 and 195 grn bullets, that 30 grns is the more accurate load in my Pietta Remington, and 35 grns is the more accurate load in my Ruger.
As someone who intends to hunt with them anything less would be a waste of powder, lead, and caps, and could also put myself at risk of injury.
As I haven't tried lighter powder charges I cannot say whether or not accuracy would have returned, but it was certainly waning at the 25 grn mark.
I also tried cereal grains, but it's just a bit much to deal with in the field, especially on a windy day. I have been considering trying felt wads to fill the space to see if it helps enough to warrant doing.