Welcome to the forum.
It starts as fliers. When you decide they have spread your groups out far enough depends on what standard you want to meet, as Jim suggested. For military battle rifles, once groups open beyond about 7" at 100 yards, they change them. If you are talking about a barrel for match target shooting accuracy, the criterion is a lot tighter.
When I shot the first barrel out of my M1A, I had begun posting fairly regular master class scores at the local match. I could usually clean the prone slow fire targets. Then one day I printed a slow fire prone target that had 19 10's and X's, but one 9 at 10:00 that I hadn't called. The next week I got another in roughly the same place. The week after that, I got two of them and that repeated a couple of times, to my frustration. Then I got four of them in one match and it finally dawned on me to check my scorebook to add up the shot history and it showed I had passed 3,000 rounds, which is pretty normal chrome-moly .308 Winchester barrel life expectancy for target shooting.
At that point, I ordered a new barrel. In effect, the gun and I combined in prone position had gone from about 1.0-1.5 moa to about 2-2.5 moa with the fliers. Since the error was always in the 10:00 to 11:00 location, I probably could have upped my score by bringing the sights a half moa down and right, giving up some Xs. But it needed a new barrel.
About 20 years ago Precision Shooting had an article in which Kevin Thomas, then a Sierra ballistics tech, had used their quality control test range to measure how long barrels lasted. Shooting groups from a machine rest in .308 Win barrels with a special lot of 168-grain MatchKings that had been set aside as a reference lot for having turned out extra precise, he would shoot until he got the first discernable flier and called the barrel life at that point. So, fliers, uncalled, are the key item to watch for.
Be aware that other chamberings have significantly different barrel shoot-out rates and how you clean and treat the barrel can affect it. They can range from under 1,000 rounds for some to over 20,000 for others.
But also be aware other things can cause fliers. Inadequate stock screw tightness is a common one, allowing the gun to shift. Scope mounts that are loose. A bad scope. A gun that has a copper ring built up in front of the throat that is constricting the bullets will do it. Carbon that has built up to the point the rifling is getting shallow will do it.