I don't think they saw it as much, if any, handicap. Consider that while the infantry settled on the .45-70-500, the cavalry load became the .45-55-405.
A lighter bullet and only 55gr of powder vs the 70gr "infantry" round. SO, I doubt that a 300gr or a 325gr slug on top of 60gr powder was thought of as a handicap.
The infantry wanted full power and the 500gr bullet for maximum range and the soldiers in the infantry had to live with it. Cavalry wanted something ligher, so as not to knock troopers off their horses firing the cavalry carbine, and, they got it.
If you're not hunting buffalo, a .45-60 is plenty for deer or elk within its usable range.