If you take yourself the gun, weather and everything else out of the picture so that it’s just your hand loads, there is so much that can affect accuracy that the list would be endless.
Personally I think bench resting the gun as Big JimP said is the best idea. If you see an improvement than depending on the amount of improvement you know where the fault is.
This is not to fault your shooting, I know that even though I shoot thousands of rounds each year, what I am capable of now, has become much worse that it was 30 years ago and its mostly the eyesight.
Since we don’t know your capability it’s hard to say how good 2.5 inches at 20 feet is. What are your stings like? Shot gun pattern or vertical or horizontal strings. That can say a lot a possible problem.
I know that with a similar gun that I picked up this year, 9mm new in the box S&W 3906, I would be very happy with 2.5 inches at 20 feet.
But at the same time I have a Beretta storm carbine in 9mm that off a bench will hold 5 inches at 100 yards with the same handloads, which is meaningless for the discussion.
Last, I would have someone else shoot the gun with your rounds. Some one that you know is as good as or better than you are.