It depends.
The type of gun & the ammo being fired, type of powder & lube, volume of the powder charge, accuracy desired and the nature of the individual gun.
Shooting patched round balls is way more forgiving than shooting sabots or tight fitting conicals. With some guns, more swabbing delivers greater accuracy, while in other guns, a little powder fouling helps.
Inlines shooting sabots require more swabbing, unless shooting powerbelts with a plastic skirt which are looser fitting.
Some powders foul more, and using more lube keeps fouling soft and can make for easier loading.
Even the temperature & the amount of humidity matters at times.
More powder usually means more fouling, unless it's cleaner burning like Swiss or fffg or P.
If one is shooting in a target match vs. simply plinking, then more swabbing is in order to obtain better accuracy, unless it's the first shot.
If a gun has a loose bore, then a little more fouling may be in order.
There's lots of variables with BP guns, and patch thickness, smaller ball size, using less powder can save a lot of effort when it comes to loading or swabbing.
Shooting at a closer distance, or with a smaller caliber, means shooting with less powder and maybe less cleaning too..
If I use extra lube and less powder, I can sometimes shoot roundballs for hours and still obtain fairly good plinking accuracy. But the better target shooters usually swab much more often.
Even the depth or shape of the rifling can play a role with powder residue accumulation. Round bottom rifling may not accumulate as much residue.
Most people learn how often they need to swab through trial & error with their particular gun.