The big Colt revolvers were military arms, primarily cavalry. The were designed for longer ranges. The SAA/.45 LC were sighted dead on for approximately 75 yards! The idea was you thinned out the opposition as you charge in and then finished them off with your saber when got amongst the survivors. I read where gunfighters would aim at your belt buckle to hit you in the chest. Seems to work.......shooting a silhouette target, I aim at the approximate waist and hit dead center at ranges up to 25 yards (max at the indoor range).
As far as shooting to one side or the other......I can only imagine its the physical layout of your particular sights. The notch rear may be cut more heavily to one side or the other, or... the front could be minutely bent/installed to one side of the barrel or another, or......considering the barrel of a cap and ball is not mounted as securely as a solid frame pistol, there could be some "whoops" factored in there!
In the army, the .45 autos we were issued would often shoot one way or the other.....not much we could do about it. Shooting the same piece enough times taught us the proper hold off, and after that it wasn't such a big deal...
Mike