The way you train is the way you'll react.
I'm surprised to hear you invoke the name of Ayoob.
Haven't heard that name in years.
As for the thread, I'm a former Marine, head of security (SOG) at Camp David.
CQB (Close Quarters Battle), as we called it, is about 75% of all we did.
Weapons used include Mossberg 590 with first 2 rds. loaded as slugs, MP-5 variants, M-9 Beretta, M-16A2 (a few variants as well) and then a slew of much heavier weapons, on up to .50 cals and MK-19 automatic 'nade launchers.
We trained with all the gear, i.e., ANPVS (night vision), gas masks, MOPP gear, body armor, massive ammo and flash bangs.
Our guys were some of the best shooters in the world (IMO), and we frequently brought in outsiders up to our "hill" (Camp David is located on top of a mountain) to train us, a different guy every week. Our pool of trainers came from the FBI HRT, Maryland State Police, D.C. SWAT, etc. In addition to that, all of us went to CQB school in Virginia, as well as Basic and Advanced Security Guard school.
I went through 400- 800 rounds a week (each) for two years.
It is the best training I could possibly receive.
What did I learn regarding headshots?
First the standard answer: To stop a target, you want to place a double-tap on them and follow up with shots of opportunity: two to the chest, one to the head. We trained to scan the target, looking for opportunity. If he was wearing body armor, we shot 2 to the head, then 2 to the hip. If his head was low, while wearing a helmet, we'd go 2 to the hip. We had paper targets that depicted targets in various states of armor and dress.
Laying aside the matters of the CNS and autonomic responses, the key to being able to confidently engage a target under a high stress situation such as the one outlined above is TRAINING. If you aren't training with your weapon of choice, you won't be able to employ it in the time of need. If you have put the hours in at the range, then when the time comes, you won't even think- you'll react. And reaction is always faster than thought. Thinking involves weighing consequences, balancing fear, analyzing data with the conscious mind, debating within yourself. Reaction is fast and silent. When the target of opportunity appears, your body automatically does what you've trained it to do thousands of times before, and you eliminate the target. No thought involved.
In the case you described, with a hostage, and the only target of opportunity is a head-shot, lets describe it for what it is:
A skull is a sphere (irregular at that), not a pancake or steel target. Because of this, there is a much smaller impact area that can have the desired effect- roughly 2" X 2". If you hit outside that impact zone, directly in the center of that sphere, chances of deflection increase past the point of acceptability. That isn't to say that you can't take down a target with a head-shot at imperfect angles. Its just saying chances of deflection are higher if your aim isn't what it should be. The further outside of that desired center-mass you impact, the greater the chance of deflection. Think of it like pool (billiards) with a really small, fast cue ball, and your target is the 8 ball.
Now that we've put this in perspective, lets remember that if you are ever called on for a headshot, and you have to do it with a pistol/handgun, you'd better be dead on, aim center-mass, and squeeze it off smoothly. And take the shot twice (double tap).
PRACTICE PRACTICE PRACTICE.