Need to know what kind of sights on the rifle, and how many groups have you fired at 25- and 100-yards that verify this quirk. The shooting results supporting your statement have to be statistically reliable.
However you didn't mention the amount of left/right movement. If it is consistent and very minimal then it could be "gyroscopic drift" (look here). I highly doubt this would be the case at such short distances, but it crossed my mind.