Depends on the home to be defended, level of competence of those doing the defending, and what kind of yard is around the home.
Less trained people are more comfortable with shotguns - they're ultra reliable and easy to use in addition to thier great offensive qualities. Good fight stop power and theres that intimidating shotgun slide rack that has grown to its legendary status. So I give the shotgun the nod for the best all around home defense weapon.
Handguns are a backup for a longgun. That's about it excepting for some real scarey "tunnel rat" stuff that I'd not really call defense of the home. Their real value is concealability for sticking behind your leg when going to answer the door or carrying in a waistband when you've heard a bump in the night and don't really expect it to be more than the dog knocking over his water bowl again.
Short carbines in .223, wether AR 15, Mini 14, AK 74 or some other platform, are loud and will cause hearing loss when fired indoors. Other than that they do everything a homedefense long arm should do - quick incapacitation of a threat, smaller chance of overpenetration, multiple strike capability, great accuracy at short or longer ranges, etc...
So my recommend for folks choosing a home defense gun from scratch for the first time is get a shotgun. It'll do the job wonderfully. Then get a handgun for backup purposes and general carry when the shotgun gets left in the closet. And finally get a carbine in .223 when you've leared to be proficient with the other two and have identified areas in defense area that may need a carbine's unique abilities.
Anyone jumping uninvited into my saferoom at home will find my wife waiting with a 12 ga Mossberg 500 loaded with #4 buck and myself with an AR 15 loaded with 55 gr SP. I doubt they'll ever have the chance to find out what kind of handguns we each carry to supplement the main arms.