When the cartridge fires and the bullet starts traveling down the barrel, that steel barrel whips like a garden hose.
This is the reason for trying to find what ammo your gun "likes". You are trying to find what bullet weight, at what muzzle velocity, makes the bullet exit the barrel at one of the nodes on this barrel whip so that it comes out flying straight.
Adding the muzzle break changes the weight out on the end of the barrel and, for that given cartridge, changes the angle at which the bullet comes out in relation to the scope.
This is the biggest reason for hand loading your own ammo. You get a chronograph and load 10 rds at each 100fps difference and see which makes a tight group.
If hand loading and wanting this muzzle break, you develope one load for with the break and one for without it. That should fix you right up!