IIRC, the 38 Special was developed with the idea it would be a black powder cartridge but it was late enough to the game that no manufacturer ever produced black powder 38 Special ammunition.
You're pretty close. There was at least one black powder factory loaded .38 Special.
introduced with the 1902 S&W M&P model, the .38 Special was intended as a more powerful replacement for the .38 Long Colt. Essentially it fired the same weight bullet about 100fps faster. The cases are essentially the same head size (.38 colt rim is slightly smaller) the Special case is 0.13 longer than the Colt case.
Winchester loaded the .38 Special with smokeless and also had a black powder load listed in their 1916 catalogue. Seems like while they did make it, it wasn't "pushed" the way the smokeless loads were. In that catalogue, they have a picture of the round and list several smokeless loads under it. But, on a different page, listed under "Black Powder Pistol Rounds" (with no picture, just a list) they do list a black powder .38 Special load, 21gr of powder with a 158gr "ball" bullet.
Additionally in that era, smokeless powder for reloading wasn't very common and there were a number of people who would shoot factory (smokeless) and then reload with black powder, because that's what they had.
The large "empty" space in the case today is a result of better, newer powder not taking as much room to produce the desired velocity.
As to why they never went to a shorter case when newer powder removed the need for as much space, why bother??
My guess would be that the cost of creating a shorter "more efficient" case simply wasn't justified. What would you get? A tiny saving in brass cost for each individual case, vs the cost of a production line and distribution and marketing for the shorter case, AND the possibility that a shorter case would fit in .38 Long Colt guns where it didn't belong.
Simply put, the need for "efficiency" wasn't there in those days, and the .38 Special wasn't broke, so there was no need to "fix" it.
some cartridges were factory loaded with black powder all the way up to the start of WWII, but when civilian ammo production resumed after the war, everything was smokeless.