I think most folks think their rifle's no longer accurate when it shoots proper test groups 50% larger than it did when new. For a benchrester, they may rebarrel when their 100-yard groups open up to 3/10 inch. A dangerous game hunter will rebarrel his 50 caliber double rifle when it shoots 2 inch groups at 50 yards instead of 1 inch ones when new. A deer hunter may rebarrel his .30-06 when it shoots 2 inches at 100 yards when it shot a little over an inch when new. It's a personal thing and we all have our standards.
Every rifle shooter has expectations of how his rifle should shoot. If it doesn't "live" up to expectations, it gets fixed or replaced.
All rifles (from one-hole benchrest to Kentucky black powder ones) have variables in their ammo as well as themselves. To say nothing about the variables in the way they're held; us humans are usually the biggest one.
When all those variables add up in the same direction, the group of X number of shots will be very big. When the variables are all zero or cancel each other out for several shots, all those bullets will go through the same tiny hole a few thousandths bigger than bullet diameter.
When you shoot several groups, if they all are not within 10% extreme spread of each other, there's not enough shots in the group to represent what the rifle's true accuracy is all the time. If you shoot ten 5-shot groups, about half will be close to average size, one-fourth will be smaller and one-fourth will be larger than average.
Most rifle shooters use the emotional method to judge accuracy, the smallest group(s) shot. To be statistically significant and see what the furthest your shot will miss the aiming point, use the biggest group shot. If four 5-shot groups are shot on four separate targets then overlayed to plot where all twenty went, that twenty-shot composite will be bigger than the largest single 5-shot group.
If a rifle and ammo (plus the shooter, of course) shoots five groups with one load measuring 2, 4, 5, 6 and 8 units and another five groups with another load measuring 3, 4, 5, 5 and 6 units, Which load's the most accurate? Which load will miss your point of aim the least when it shoots the worst round for accuracy.
I think the groups you're getting are very normal for any load to produce with several few-shot groups.
There's six rules that top level competors use to get best accuracy;
1. Use reloading tools and techniques that do not let the bullet be deformed upon entering the rifling.
2. Full length size all bottleneck cases reducing fired case dimensions only a thousandth or two.
3. Use bullets a few to several ten-thousandths larger than groove diameter.
4. Use extruded powder; ball powders are not popular among winners and record setters.
5. Make sure the rifle and yourself are very repeatable in all they do holding the rifle and shooting the round for every shot fired. The shooter is typically the least repeatable of everything.
6. Shoot enough shots per group testing each load to have it be statistically significant. One 30-shot group's about 10 times better than the average of ten 3-shot groups.