It's the "talking smack" that gives licence to those who cross the line.
I disagree. For the most part, men understand that the line should not be crossed. They know that it might be OK to feel "that way" about gays or blacks, but its not OK to go out and mistreat them. The ones that do become subject to broader, overarching rules. That's why most guys don't put those words into action.
And the homophobic talk is a form of social control even if the threshold of violence isn't reached. Even that racist talk is a form of social control. It's an attempt to get those who hear it to tow the line. I know from experience that you pay a social price for resisting both of these pressures.
I know that I used to catch flak from the black kids in high school for having white friends. Whites were supposed to be the enemy. I learned about the social cost of bucking "the rules," and I decided that what I got out of my relationships with white kids was more important the scorn I got from the black kids. So I just kept doing what I wanted to do.
Peer pressure is not an insurmountable obstacle.
Getting back to "talking smack," guys don't do it to control their peers as much as they do it to blow off steam. And like so many other things that men do, blowing off steam often contains an element of aggression. So talking smack is often crude and abrasive, and can appear threatening to an outsider.