Four to five inches off of point of aim is just unacceptable, How do they let something like that out of the factory??
I believe the factory test firing is a function check, and not done for accuracy.
And, while I'm not doubting your skills (I also have 40+ years handgunning), it still might be "you", and not the gun, exactly.
"You" in this case cover your aiming, your shooting, and the ammo used. We don't all look through the sights the same way. Your trigger pull might be throwing things off, but since you are an experienced shooter, I won't list that as a high probability.
The ammo used (and that particular gun) might be the entire culprit. You might send it back to Ruger, and they test fire it with their ammo and it might be spot on. Not saying that WILL happen, but it COULD happen.
A friend of mine had an SA "clone" (I forget the brand its been years now..) in .357Mag. The gun was a pure delight shooting 158gr .38 Spl (at 850fps). Perfect point of aim and small groups at 25yds. EVERY .357 load we put in it shot a FOOT low and RIGHT, in the hands of 4 different people.
This was an extreme case, I've never personally seen another one that much off, but I have seen numerous fixed sight guns that shoot one load (or three) well, and another load lands more than a little bit "off".
This is the reason I prefer adjustable sight guns. I almost never use them to adjust for distance or wind in the field, I only adjust them if the initial zero needs adjustment, and use hold off in the field. But, it is a great comfort to me knowing I have the ability to adjust the sights easily and quickly, IF I ever need to.
In your case, I would #1, try different loads (both bullet weights and manufacturers). If you find one (or three or whatever) that does shoot to point of aim, you really can't say its all the gun's fault, and sending it in MIGHT not fix the issue with the ammo that shoots badly off point of aim.
#2, have others shoot it, just as a verification that it isn't you.
If that is done, and nothing goes to point of aim for anybody, then the gun is definitely due for a trip to the shop.
Good Luck! and let us know how it goes.