Jason Iowa,
The role of the shooter is dramatically different between private citizen carriers and those carrying in service roles. No question or disagreement there. Having said that, all of the guys I referenced above used their handguns inside realistic defensive ranges. One friend said his longest range using a handgun was about 40 feet. Most described situations ranging from 5-30 feet. Among those was a friend going Sig P220 vs AK inside of a room.
When they referenced using sights, it was always in a flash context, except the one LEO describing relatively deliberate aiming after instinctively firing the first three rounds. The sights were useful for them at defensive distances.
There is an asterisk. None of them used sights at contact distances. That makes sense, of course. They weren't at extension in 5 foot gunfights.
You may not have used your sights at close ranges. I believe you when you say that they weren't useful for you. I also believe these friends when they relate their experiences. These are no BS, upstanding guys, who have earned my trust.
After having spoken with them, experienced my own situation, and done a lot of sim work, I wouldn't want to go without sights. Beyond 7 feet or so, I prefer having them for flash reference. Inside that distance, I would almost assuredly not use them. The time it takes to be aware of them at realistic defensive distances is near zero, and hits are worth the investment.