In a given caliber, ignoring things like jacket thickness and voids, higher SDs mean heavier and longer bullets. Heavier bullets mean higher pressures and, ultimately, chamber pressure is the limiting factor.
Longer bullets, if they are to fit in magazines, have to be seated more deeply in the case and reduce powder space. Handloaders can seat bullets further forward and reclaim some of that lost case volume, but they are limited by length of the chamber throat.
SD is only one design consideration; the bullet's flight and after-impact behavior have to be considered, too. I guess there's a magic diameter-to-length ratio for bullets, but I'd guess that what's best for flight might not have good terminal performance or may have too much barrel friction. (There is such a number for ships and boats, but they're only expected to push through water; they don't have to plow their way across the beach to get to the water and then come apart in an expected fashion when they hit something.)
I guess what we've got is an economically efficient combination of engineering, tradition, and regulation.