I would suggest that any home defense situation, as far as ammo goes, would be one where you have practiced a lot with and tested guns and ammo to meet your specific needs. In other words, if you have a .308 rifle, like an FAL, then you should have tested that rifle to see if it will meet your specific shooting needs and if that caliber of ammunition is what you really desire to use. If not, then change to something that you feel is more appropriate.
If you are asking, in general, which rifle is pretty good for home defense or which ammo is good for home defense, then consider a .223/5.56 M-4 type carbine or civilian legal clone using hollow point ammo or TAP ammo. Most of the experts seem to recommend this combination for overall effectiveness and good accuracy.
There is also the option of going with carbines that use pistol caliber ammunition. A good example of a renown firearm that has served Americans well is the old M-1 carbine. That carbine has been used in a number of wars and was actually used by some police SWAT teams when those teams were in their very early teething phases or start up phases. I know of one guy who served in the Korean War who swore by his .30 caliber carbine and actually used it one time to shoot up the car of a burglar he caught in the act. Strafed that car and never heard one word back about it either from the burglar or the police. In addition there are also lever action or saddle carbines that shoot pistol ammo. These carbines are like the ones that the cowboys used and are also very reliable.