Lawrence Phillips: There's a high probability we'll end up uncertain

It was the job of the intelligence analysts to draw reasonable inferences from a mass of information

Many years ago, two intelligence analysts co-authored a report that included the statement "The cease-fire is holding but it could be broken within a week." Although they had agreed to express their uncertainty about the event with the phrase "could be broken", they later discovered it meant "a 30 per cent chance" to one of them and "an 80 per cent chance" to the other. Could this potential for misunderstanding reports about uncertain events explain why many feel the Government exaggerated the threat from Saddam Hussein?

Nobody knew for sure that Saddam Hussein possessed weapons of mass destruction. It was the job of the intelligence analysts to draw reasonable inferences from a mass of information of varying degrees of reliability. As no WMDs have been found, there could not have been any definitive piece of data before the war that would have led the intelligence community to give a definite yes or no, other than unreliable reports.

So how did they communicate their uncertainty to ministers? If they used words, another problem arises, as was demonstrated back in the 1970s by Nato intelligence analysts. They presented 22 military officers, who were familiar with reading intelligence, publications containing sentences like this: "It is highly likely that the Soviets will invade Czechoslovakia." The sentences varied only in the phrase used to indicate uncertainty: almost certainly, very good chance, likely, we doubt, unlikely, little chance, and so forth: 16 sentences altogether. Each officer was asked to indicate what probability, as a number between 0 and 100, he felt was intended by each message. The results showed great variation in the probabilities for nearly every phrase.

"Almost certainly" was thought to mean anything from 50 to 99 per cent and "very good chance" 50 to 90 per cent. The phrases "probable", "likely", "probably" and "we believe" elicited interpretations ranging from about 20 to 90 per cent. These wide ranges provide substantial scope for miscommunicating probability: carefully chosen words from the intelligence community might well be interpreted differently by ministers.

A further factor could make things even worse. Back in the days of the Central Electricity Generating Board, generating equipment was taken off line for repairs in the summer as demand for electricity dropped. When the generators did not come back on line at the scheduled times, which could lead to power shortages, investigation found that the verbal phrases describing the repairman's uncertainty about meeting the target date lost its uncertainty as it was communicated up the hierarchy.

The repairman's "probably", meaning to him 30 per cent, was heard by his first line manager as better than 50 per cent, so he reported "good chance" to his department manager, who passed on "highly likely". By the time it reached the top, the generator was "almost certain" to be ready in time.

To correct this bias, the CEGB then instituted two steps: a centralised information gathering and processing centre, and uncertainty always reported as probabilities. The problem disappeared. As verbal phrases of uncertainty are reported upward, the uncertainty decreases for events people want to happen, and increases for events they don't want to happen.

In short, verbal phrases that permit interpretations of both high and low uncertainty, combined with the uncertainty disappearing as it was communicated upward, might well have been at work in reporting the threat posed by Saddam Hussein.

What is the remedy? Simply to report uncertainty in the form of probabilities, not words. The intelligence communities in both the US and UK are aware of the advantages of working in probabilities, which make it possible to combine information, to take account of collateral information, and to deal with unreliable data. Probabilistic models typically provide more final certainty than is apparent from the uncertainty attending the inputs. To quote Francis Bacon, "If a man will begin with certainties he will end with doubts, but if he will be content to begin with doubts he shall end in certainties."

So was the threat posed by Iraq exaggerated? We won't know unless the inquiry looks into how intelligence information was processed, the words-versus-probabilities issue, and how the Government used the information it received. Failure to investigate any of these will leave us all as uncertain as we are now.

larry_phillips@msn.com

The writer is Visiting Professor of Decision Sciences at the LSE