To prevent myself from doing this, I sat down and made a list of the calibers and actions that I wanted those calibers in. Made myself a promise that I wouldn't go beyond it.
Haven't done so, yet...
I would guess that "a sufficient number" would include one or two for every room in the house; plus one or two for each vehicle you own; plus one or two for each person living in the house or riding in those vehicles; plus a couple of spares at the homes of each of your immediate family, good friends, and loose associates; plus five or six for your place of business (subject to where you work, of course, because you may need eight or ten instead). Anyone have a calculator handy?
And don't forget two for CCW: your primary plus a backup. And your spouse's (of significant other's) CCW weapon plus backup. And a couple for just general range work/plinking.
And then at least one copy of each of the above, just in case something goes wrong with the primary.
THAT ought to cover it pretty well.