There are actually a lot of questions in this one post.
As far as a 5.56 rifle versus a 12ga shotgun, the shotgun, with Buck shot produces significantly more effective wound trauma.
Even taking into account bullet behavior at high velocities, a single .223 caliber projectile is not going to be as effective as nine to sixteen .33 to .30 caliber projectiles which penetrate 12 to 16”.
On the issue of what size shot, The Firearms Tactical Institute report recommends #1 Buck
For personal defense and law enforcement applications, the International Wound Ballistics Association advocates number 1 buckshot as being superior to all other buckshot sizes.
Number 1 buck is the smallest diameter shot that reliably and consistently penetrates more than 12 inches of standard ordnance gelatin when fired at typical shotgun engagement distances.
A standard 2 ¾-inch 12 gauge shotshell contains 16 pellets of #1 buck. The total combined cross sectional area of the 16 pellets is 1.13 square inches. Compared to the total combined cross sectional area of the nine pellets in a standard #00 (double-aught) buck shotshell (0.77 square inches), the # 1 buck shotshell has the capacity to produce over 30 percent more potentially effective wound trauma. In all shotshell loads, number 1 buckshot produces more potentially effective wound trauma than either #00 or #000 buck. In addition, number 1 buck is less likely to over-penetrate and exit an attacker's body.
http://www.firearmstactical.com/briefs10.htm
I’ve also seen the re-emergence of the “column of lead” argument in this thread. The column of lead argument puts forth the idea that at close ranges the shot doesn’t spread out enough to make a difference. What constitutes close, what exactly makes a difference versus what doesn’t make a difference and what constitutes significant or insignificant spread are usually nebulous terms in and of themselves. The column of lead argument can sometimes be used to argue that shot size doesn’t matter.
Unless the shooter is using something like Federal FLITECONTROL, the pattern is going to open up slightly even at 3 feet from the muzzle and each pellet is going to create it’s own wound channel in the assailant.
Even using Federal FLITECONTROL, when the pellets hit the body, although the entrance wound looks like a single projectile entry wound, the pellets do go their separate ways and create separate wound channels.
The same thing happens with a shotgun contact wound, just because the target is close does not mean that the shot column acts like a slug. Obviously there is going to be a difference in where the shot ends up in someone shot at point blank range with a shotgun versus someone shot at 20 feet with a shotgun. Obviously, with a close range shot, all the shot is going to end up in a smaller total volume (closer together), but that doesn’t mean that the pellets act like a slug and make a single wound.
The other thing that was mentioned in the OP was the use of a turkey gun. Most turkey guns have 24” to 28” barrels, and that does get a little unwieldy in the average home.
I don’t know your circumstances or how your home is laid out. I don’t advocate clearing a house, but sometimes people have to move with their firearm, to get kids or move to a safe room or whatever.
If you’re able to just hunker down, call 911 and do nothing else, then a turkey gun is probably fine. For people that have to move once they arm themselves, maybe a 43” to 44.5” long gun is little unwieldy.