I think you made a wise choice going with 180's over 165's. I have no doubt that 165's of the same construction would work. Even a good 150. But unless your rifle just won't shoot the 180's there isn't any real down side.
I used to stay with 150, or 165's in my 308's but after experimenting with 180's probably won't go back.
The better BC's of the heavier bullets make up for their slower muzzle velocity. My best hand loads show 165's leaving the muzzle about 100 fps faster than 180's. But at 400 yards the 180's are faster, only drop 1.5" more, and have more energy at all ranges.
If I can compensate for 31" of bullet drop with 165's, then 32.5" isn't enough to matter. The extra weight, velocity, and energy of the heavier bullet may, or may not help. But it sure won't hurt, and since there are no negatives, why not.