Indeed it is a good question.
Firearms cartridges come in the huge variety they do because in each case a person decided they wanted a cartridge like that, probably for certain performance reasons. Given that firearms cartridges have four basic components -- projectile (bullet), cartridge case, powder and primer -- and that there are miriad choices for projectile weight and caliber, powder burn rate and charge density, and case volume/strength, the number of possibilities is staggering.
Most of the cartridges out there have been developed to meet a specific requirement that the developer had in mind. There are many trade-offs in size, power, weight, etc... that play into the requirements the developer seeks to fulfill.
Taking your question about defense cartridges, for example, there are reasons some folks don't choose the "max size". Max size might mean 12 ga. shotshell, .50 BMG, or .50 AE among readily available cartridges. No reasonable conceal-carry arms are chambered in any of these, so if you need an arm to conceal these cartridges would be out of the running. Similarly, if you need a cartridge to stop a man, .50 BMG or .50 AE might introduce severe overpenetration into the equation and since you are responsible for each bullet fired this is clearly undesirable. Also, the simple size of the cartridge may mandate limited magazine size for handguns -- I don't think very many humans could wrap their hands around a hi-capacity double-stack .50AE magazine which is why no one makes them.