Facebook's leaked threat guidance for moderators makes more sense than people think

The instructions are relevant in a simplistic way, but certain aspects are concerning

The guidance is intended as training for inexpert editors rather than a coherent threat analysis, says Professor Tim Grant (REUTERS/Eduardo Munoz)

Facebook’s controversial guidance for moderators isn’t as ill-conceived as many people believe, according to a forensic linguist whose work on threat and language analysis helps the police convict criminals.

Leaked internal documents revealed that the site’s moderators are instructed to delete certain threats but leave others up, depending on how “credible” they are.

They were heavily criticised and widely considered to be confusing, with some people branding the guidance as counterintuitive.

“There was nothing [in the leaked slides] that struck me as being completely in the wrong direction,” Aston University’s Professor Tim Grant told The Independent.

“It all makes sense in a simplistic way. There’s nothing that makes you think ‘that’s completely wrong’. It’s not counterintuitive in that sense. It’s the idea that there can be simple criteria that is worrying. Threat analysis is a bit more complicated than this.”

According to the documents, some forms of abuse, such as “Someone shoot Trump” and “”#stab and become the fear of the Zionist”, should be deleted straight away.

Others, like “fuck off and die”, “kick a person with red hair”, “I hope someone kills you” and “to snap a bitch’s neck, make sure to apply all your pressure to the middle of her throat”, can be left on the site, as they apparently aren’t credible.

“I certainly have some concerns over that,” says Professor Grant. Focusing on that final example, he explains, “That seems to me to be quite a specific threat. It’s detailing an action that should lead to an outcome.

“We don’t know who the individuals are that have been trained, but from the material I have seen, it seems that it’s intended as training for inexpert editors rather than a coherent threat analysis. The impression you get is that there will be a lot of variation between individuals in the threat analysis.”

Threat analysis experts use the specificity of a threat as a measure of how likely someone is to carry through that threat, as Facebook’s moderators are instructed to do.

Certain measures, such as the amount of time, energy and effort someone puts into a threat, and whether or not it’s repeated, can be used as indicators of intent, but this is far from straightforward.

“I might threaten to shoot you but actually throw a brick through the window, or I might threaten to throw a brick through your window and actually shoot you. No simple criteria is going to capture that.

“If you look at school shooters’ Facebook pages, you get lots of disturbing threats and sometimes manifestos and other material that has been posted, which look obvious in retrospect. That the person was dangerous. But then you see that for every school shooter who goes out and performs an action, there might be 100 or 1,000 others who are posting similar material online.”

Facebook has promised to recruit 3,000 new members of staff to moderate the site, and also uses algorithms to help spot unacceptable material that has been posted by users.

Professor Grant, though, says threat analysis isn’t a job for a computer system.

“It’s about a broader profile of a threat rather than an algorithm that can be simplistically applied. There’s no wholly automated method that can do this, that I’m aware of. Social media analytics doesn’t really look at the linguistics, which is all about the context of language use.

“This is a really awkward halfway house between doing a proper job and doing no job at all.”

However, he recognises that only some of Facebook’s documents have been leaked, and the company’s internal processes could be “more nuanced” than they currently appear to be.

Unfortunately, Facebook isn't planning to share the details of its policies anytime soon, as the company believes that doing so could encourage people to work around them.

“It’s an interesting and difficult area to get right,” says Professor Grant. “Trying something is probably better than doing nothing at all.”