A lot of it has to do with noticeing those ques that say, "Something isn't right."
Two examples:
I was with a group of friends one evening, after seeing a movie. As a group of young people, I was one of the two drivers for the group. When we dropped off one of my friends at her house, her new boyfriend showed up. She introduced us, and the group chatted for another 5-10 minutes before leaving. Within that time, myself and three of my male friends determined there was something "off" about "new boyfriend." It could be that he reminded me of someone who was a drug user. It could be the fact that he showed up at 11PM at his new girlfriend's house. It may have been his manner of dress (but that wasn't too odd, to be honest). None of that really screamed "wrong" to me. However, something was not right, and most of my friends realized it.
Fast forward a few years, and our suspicions were confirmed. "New boyfriend" can't hold a job. He tried to start a fight with one of my good friends (whom I don't think has ever caused trouble). He has been in trouble with the law a few times, for relatively "minor things." He fathered a child with my friend and can't provide adequate support...nor does he seem to be trying very hard...in essence a deadbeat 20-something father.
None of us knew this the first time we met him, yet the majority of males in the group felt the guy was "wrong."
story 2:
One night at the base theater, several drunks managed to get backstage and accidentally broke through the main screen. Of course, they immediately ran. One of the projector operators, another guest, and myself caught up with the three rather drunk suspects. We stopped them and called the police. The police took about 30 minutes to respond, so we waited inside a building. Two of the drunk guys had that "mommy caught me with my hand in the cookie jar" look. They stayed quiet, hands in their coats, looking down for 95% of the time. The third looked nervous. He was pacing and looking around. He occasionally chatted quietly with his buddies, who seemed to disagree with whatever he was suggesting. I was very worried that he would either attack myself or the projector operator and/or try to run. Fortunately, nothing eventful happened. The police showed up, took statements, and that was the last involvement I had. The next day, I ran into the guy who was operating the projector. He mentioned that he was really worried the one guy was going to attack us.
We had both felt something was wrong, and we both agreed that there was concern about the third guy attacking us. Neither of us had mentioned that, at the time of the incident.
Sometimes you just seem to know something is wrong, even though you cannot exactly determine what it is. If other people's experience teaches any lesson about this, it is to pay attention to those feelings.