The size and power of handgun cartridges are governed by "laws". Some natural, some man made.
And, no matter what you do, somebody isn't happy with it. And that's a big part of the problem right there.
"Has to fit in an existing gun" (or frame size)
That's a biggie. And its the reason several of our cartridges are what they are.
The reason we have a .44 Magnum, and not a .45 magnum was that in the 50s when Elmer Keith was doing his loads to develop his desired power level, he had to use existing guns. What he discovered was his desired performance level created pressure too much for existing .45 caliber pistols. He blew up a couple doing it. (at least, some say 3)
He then switched to .44 caliber and was successful because the smaller caliber gun had more steel where it mattered most and was able to contain Keith's loads without failing.
Another one, in the opposite direction, sort of, is the .40 S&W. this time, it was not a matter of gun strength, but of user ability. And bureaucracy.
Because of the conclusions drawn from the Miami Shootout, the FBI wanted something "better" than the 9mm Luger. They chose the 10mm. However, both the round and the S&W pistol chosen were bigger and more difficult for enough of their agents to handle that qualification scores went down.
The FBI then went to a reduced power 10mm load, and while the "10mm lite" load had sufficient power to satisfy their requirements, the large 10mm guns were still a problem.
The solution came from S&W creating a short 10mm case, taking the "10mm lite" load and fitting in the existing 9mm frame size, which the majority of agents could handle well enough to meet qualification requirements.
remember that situation was an agency requirement, not an individual one. The problem could have been handled by allowing each agent to use whatever they chose and could qualify with. BUT, large govt agencies simply don't work that way. They require everyone to use the same thing and meet the same standards with it.
Other "laws" governing handgun cartridges are size, and fit in the hand. And the need for those to be within what average people can manage, in order to achieve commercial success.
The standard service rounds today, designed to fit in the grip magazine of semi autos are all approximately 1.3" (or less) loaded length. This works for the majority of people. Magnum revolver rounds (not originally designed to fit in the grip) run about 1.6" (+/-), and when those length rounds are put in the pistol grip they result in a grip that is too large for the MAJORITY of people. There are a lot of people who can manage the larger grip, but too many that simply are not physically big enough (hands) to do so. SO, the market for those guns (and cartridges) is limited by that.
Adding to that limitation is the LEGAL requirement (in many places) that the ammunition MUST BE in the pistol grip. Thanks to our politicians and various activists, today, pistols with magazines not in the grip are frequently classified as "assault weapons" and various legal restrictions, even complete bans in some places are the LAW, these days.
And another of the "laws" is the market. Popular opinion, along with physical characteristics drives the market. And popularity is a shifting sand, what was once popular no longer is, and some things that were never that popular now are. Very few people want a smaller, less effective round, though many want a smaller gun. Too much recoil, too big a gun, too heavy, in popular opintion, all restrict the market share. And this affects potential viability for any new cartrige (or gun) design.
Literally, there is no free lunch.
If you want it to be something other than a niche round, you have to provide something that is some level of significant improvement over what is available today. And that, isn't easy.
) and that WILL matter to many prospective buyers.