As Jim said, if you know you called the flier, that's one thing. But you want to verify that the particular called hole really did go out of the group. I've found the reliability of shot calling to be better some days than others. I remember calling a lot of 9's on an offhand target that I cleaned at Camp Perry one year. Match nerves just made every little wobble look worse than it really was.
Exaggerating called errors happens more often than you might suspect. We have a built-in bias to want our gun and ammo to be better than they actually are, so we can be quick to dismiss a flier as our fault because that doesn't detract from how well we can let ourselves believe our gun and ammo shoot, and it doesn't detract from what we see as our personal potential "when I do my part". Our egos actually want to fool ourselves in this regard because then some of our groups are potentially world class, and that means our personal potential is better than average. As Garrison Keillor recognized long ago, we want all the children to be above average, and I'll add, especially our own inner child.
So, what's an objective person to do? First, you need to recognize a statistical fact: randomness does produce shots outside the size of the average group. In fact, half of our groups have shots outside the average diameter. The further out a shot is, the rarer the occasions upon which it might appear, but it isn't never. This is why, the more shots you put into each group, the bigger they are on average. You are giving farther out and less common points of impact more chances to show up.
You see the same thing overlaying groups, and that's important to know. The hole most distant from the average center of all your groups (assuming same load and sight settings) is the one that had the lowest probability of appearing. So, let's suppose you have a nice round 15-shot group, except one hole opens it up the last 5% of its diameter. Now imagine that the 15-shot group is actually an overlay of five 3-shot groups. That most-distant hole had to be in one of those 3-shot groups. The odds are the other two shots in that group were closer to the center, so it might easily have doubled the size of the 3-shot group it appeared in, even though it only increased the 15-shot group by 5%. And, when you shot that 3-shot group and looked at it by itself you'd likely look at that hole and say "oh, I must have pulled it", or, "I knew the recoil felt like it slipped more on my shoulder than usual" or offer up any of a number of other rationalizations, when it doesn't look at all out of place in the 15-shot group. This is how we fool ourselves.
There are statistical ways of evaluationg these things that are beyond what I can put in this post and likely beyond what most shooters really want to deal with anyway. Nor do you have to. Your eyeballs may not be calibrated, but they can tell you a lot. The easiest thing is to do just what I described above. Overlap a series of groups, keeping the point of aim constant (assuming no sight setting changes).( Don't use the mean center (average hole location) of the individual groups for overlapping, as those center locations have their own radial standard deviation and wander around what the average for an infinitely large group's mean location would be.) Just line them up by that common aim point and look at the shape of the group and a lot of what you thought were fliers will turn out not to be when all holes are combined.
The few holes in the combined group that are really far from the rest are the ones that are candidates for exclusion. And the reason for excluding them becomes irrelevant in a way, because whether they are out there because the shooter messed up or because they are random outliers doesn't really matter as much as the fact they are few. This is because the fact they are few means they are caused by something that doesn't happen often enough to merit taking it into consideration when placing any single particular shot, be it on game or on a target. (An exception is in the case of end of barrel life described in the next paragraph)
As you overlay targets you can look for trends, like seeing your skill level change when the most recent 50 shots you overlay are better than an earlier sets of 50 shots. Indeed, if you want to see a trend in how your shooting is going, you can keep a running group average by eliminating the oldest target from the overlay every time you add a new one. This will not only inform you of trends, it makes developing problems easy to spot. If you have a problem with the gun walking on the target or starting to string diagonally because of changing stock contact points or a fault in the bedding, you will spot it in the overlays. If you spot outliers and track the number of shots between when they appear, you can tell when a barrel is starting to be shot out. That's how a shot-out barrel makes itself known, by fliers that appear more and more frequently, and not by gradual opening up of groups overall. Think of it as having frequent flier mileage on the barrel.
Exaggerating called errors happens more often than you might suspect. We have a built-in bias to want our gun and ammo to be better than they actually are, so we can be quick to dismiss a flier as our fault because that doesn't detract from how well we can let ourselves believe our gun and ammo shoot, and it doesn't detract from what we see as our personal potential "when I do my part". Our egos actually want to fool ourselves in this regard because then some of our groups are potentially world class, and that means our personal potential is better than average. As Garrison Keillor recognized long ago, we want all the children to be above average, and I'll add, especially our own inner child.
So, what's an objective person to do? First, you need to recognize a statistical fact: randomness does produce shots outside the size of the average group. In fact, half of our groups have shots outside the average diameter. The further out a shot is, the rarer the occasions upon which it might appear, but it isn't never. This is why, the more shots you put into each group, the bigger they are on average. You are giving farther out and less common points of impact more chances to show up.
You see the same thing overlaying groups, and that's important to know. The hole most distant from the average center of all your groups (assuming same load and sight settings) is the one that had the lowest probability of appearing. So, let's suppose you have a nice round 15-shot group, except one hole opens it up the last 5% of its diameter. Now imagine that the 15-shot group is actually an overlay of five 3-shot groups. That most-distant hole had to be in one of those 3-shot groups. The odds are the other two shots in that group were closer to the center, so it might easily have doubled the size of the 3-shot group it appeared in, even though it only increased the 15-shot group by 5%. And, when you shot that 3-shot group and looked at it by itself you'd likely look at that hole and say "oh, I must have pulled it", or, "I knew the recoil felt like it slipped more on my shoulder than usual" or offer up any of a number of other rationalizations, when it doesn't look at all out of place in the 15-shot group. This is how we fool ourselves.
There are statistical ways of evaluationg these things that are beyond what I can put in this post and likely beyond what most shooters really want to deal with anyway. Nor do you have to. Your eyeballs may not be calibrated, but they can tell you a lot. The easiest thing is to do just what I described above. Overlap a series of groups, keeping the point of aim constant (assuming no sight setting changes).( Don't use the mean center (average hole location) of the individual groups for overlapping, as those center locations have their own radial standard deviation and wander around what the average for an infinitely large group's mean location would be.) Just line them up by that common aim point and look at the shape of the group and a lot of what you thought were fliers will turn out not to be when all holes are combined.
The few holes in the combined group that are really far from the rest are the ones that are candidates for exclusion. And the reason for excluding them becomes irrelevant in a way, because whether they are out there because the shooter messed up or because they are random outliers doesn't really matter as much as the fact they are few. This is because the fact they are few means they are caused by something that doesn't happen often enough to merit taking it into consideration when placing any single particular shot, be it on game or on a target. (An exception is in the case of end of barrel life described in the next paragraph)
As you overlay targets you can look for trends, like seeing your skill level change when the most recent 50 shots you overlay are better than an earlier sets of 50 shots. Indeed, if you want to see a trend in how your shooting is going, you can keep a running group average by eliminating the oldest target from the overlay every time you add a new one. This will not only inform you of trends, it makes developing problems easy to spot. If you have a problem with the gun walking on the target or starting to string diagonally because of changing stock contact points or a fault in the bedding, you will spot it in the overlays. If you spot outliers and track the number of shots between when they appear, you can tell when a barrel is starting to be shot out. That's how a shot-out barrel makes itself known, by fliers that appear more and more frequently, and not by gradual opening up of groups overall. Think of it as having frequent flier mileage on the barrel.
