Like the OP, I had similar questions regarding my options for home defense. Consequently, over the years I've modified my choices more than once. And after talking with several professionals (shooting instructors, LEOs, defensive shooting instructors) I came to the conclusion that the answer is: it depends on what you've trained for and with what type of weapon you've trained.
My opinion on the subject is mine and mine alone, and probably shouldn't have much influence on what another decides is best for them.
However, in the interest of replying to the OP, my short answer is: first the AR, then the shotgun, then the handgun.
Now the long answer:
I have my mossberg 500 with the 18.5 inch barrel (standard stock; I gave away the pistol grip) loaded with 00 buckshot. When I lived in an apartment, I was concerned with over-penetration, and chose no. 4 buckshot. Now I live in a house, and being that I live in the PHX area, my house is surrounded on three sides by a 6 ft. tall cinder block wall (pretty common in PHX), so I don't worry about penetration at all. Two extra boxes of shells in the nightstand.
My reason for choosing the shotgun is that I'm very familiar with operating this particular gun in a hurry under some stress; it just comes with bird hunting. Birds appear out of nowhere and move quickly, so you get used to not only pointing quickly from odd body positions, but running the pump quickly as well for the follow-up shot. With all that practice, it's not difficult for me to snap off 3 shells in about one second.
As far as rounds, people seem to forget that in just one 00 buckshot shell there are 9 .33 inch (50 gr.?) lead balls, flying at approx. 1,200 fps. So I'm not shooting one round at a time, I'm shooting nine
projectiles per trigger pull. If I only get two rounds off per second (maybe I just woke up and I'm a bit slow) that's 18 pellets/bullets with two trigger pulls. Far more rounds than I can get off with either a rifle or a handgun in 1-2 seconds. If I have to fire all six rounds that's, let's see,
fifty-four .33 caliber balls; many more projectiles than either my AR-15 or my handgun can even hold in one of their magazines.
As far as pattern goes, well with that short barrel, at 15 ft. it's a little bigger than a baseball. So no, I can't sweep a whole room with a couple shots, but let's be honest, all I have to do is put the muzzle over the thoracic cavity and press, producing a baseball-sized pattern somewhere near the middle of the bad guy. (btw, from my own experience, if you hit a man-sized animal such as a 150-200 lb. porker in what amounts to the thoracic cavity area with one of these at a distance of less than 15 ft., they tend to go down immediately or very quickly).
But let's say for the sake of argument that the scenario is basically contact distance and the shot won't spread. The muzzle of my mossy is about .74 inches (.74 caliber). About an ounce of lead moving at 1,200 fps at the muzzle. Is anyone really going to tell me that wouldn't do serious damage to the bad guy? Is he made of iron? Kevlar?
At any rate, the caveats to my choice are- that I live alone (other than my niece and nephew a few times a year and the GF a few times a week) so I don't have a lot of friendlies running around the house to worry about. So if you're in my house and I didn't know about it, the probability is just about 100% that you're a bad guy. Of course, it's a moot scenario in my case, as there's no way around my two dogs that go into "danger, Will Robinson" mode if anyone gets within ten feet of the house. And Max is a Rhodesian Ridgeback (dog bred to hunt African lions, 150 lbs. with two-inch fangs and no pain receptors in his whole body). As a matter of fact, my best home defense weapons are probably my two very large, very protective dogs.
So, while I probably wouldn't tell one of my friends to make a pump shotgun their first choice in HD, especially if they didn't much experience with shotguns, for myself I'm perfectly comfortable using a shotgun to hold off the bad guys until the cavalry shows up.
For the OP, you must come to your own decision based on your own training and experience. Or get a big protective dog or two.
Little video for your viewing pleasure:
https://youtu.be/dbFLY9OIqPA