Future imperfect: why do the financial forecasters keep on getting it wrong?

In March last year analysts at investment bank Credit Suisse First Boston lowered their 12-month share price target for Enron. Given what we now know, that would seem sensible, even prescient. In fact, the prediction was nonsensical: CSFB cut the target from $128 (£83) to $110 (£70). Nine months later Enron stock was worth precisely 26 cents.

Sadly, lousy forecasts are legion on the financial markets, be they of stock prices, interest rates or the overall economy. It's unfair to single out CSFB because it was not alone in failing to foresee the fiasco. Hard though it is to believe, late last year no fewer than 12 of the 17 Wall Street analysts covering Enron recommended clients should either hold or even buy the company's stock.

The conspiracy theory is attractive: analysts produce over-optimistic forecasts as they conspire with their investment banking colleagues to peddle potentially worthless stock to unsuspecting clients. But does this explain everything? In researching my book on the prediction business, Market Prophets, I learned never to underestimate the fallibility of forecasting. Most of the time forecasts go wrong because markets and economies, like life, are simply unpredictable. Which Market Prophet, however gifted, could have foreseen that suicide hijackers would snuff out 3,000 lives on 11 September 2001, let alone the economic and market consequences?

Not all shocks are external. Stock prices have plunged because they rose too far. During the late 1990s bubble many pundits forgot markets move in cycles and have a habit of returning to their long-term average growth rate, which is usually just a few per cent a year. So if prices rise 20 per cent a year, year after year, sooner or later they'll suffer a big drop. "In bubbles you get a kind of reverse Darwinism – survival of the stupidest," says Tony Dye, once a top pension fund manager who famously refused to run with the herd.

Forecasts sometimes fluctuate as wildly as the markets themselves. In a Reuters poll at the end of 2000, equity strategists predicted on average that the FTSE index would close 2002 at 7,600 points. By the end of 2001 they had slashed the forecast to 5,900 and this month they cut it again to 5,500. This year isn't over yet and markets can rise as well as fall, but the market has a long way to go to meet their latest prediction: the FTSE closed on Friday at 4,015.

To be fair, equity analysts usually get close to predicting company earnings (Enron and WorldCom apart), which tend to be less volatile than the share markets where investor psychology plays such a strong role. Graham Field, who runs AQ, a publication that "analyses the analysts", says forecasts typically come within 3 to 4 per cent of actual earnings per share figures – albeit often after nudges and winks from the companies that are anxious to avoid unpleasant surprises. Nevertheless many equity gurus prefer not to dwell on the past. "If you ask an analyst what was their track record, they often have no idea," says Mr Field.

Economists seem more inclined to reflect on their record. Again it's not great. Prakash Loungani, an IMF economist, has found that out of 72 recessions worldwide in the 1990s forecasters spotted only eight in the year before they happened.

Not everyone fails to spot downturns, be they of markets or economies, but timing remains an irritant. Mr Dye realised a crash was on the way but pulled out of US stocks several years too soon. "Like comedy, it's all about timing," says Peter Osler at the London futures house GNI. But the scientists' record is not uniformly dismal. They are often good at short-term predictions such as inflation figures. They're also good at foreseeing changes in interest rates, but don't forget central bankers often drop hints before they move.

Still there's no denying that accurate forecasts are more the exception than the rule. So why bother? Because decisions need to be made. "You obviously cannot construct macro-economic policy simply by relying on the seat of your pants and anecdotal evidence of people you talk to. It has to be done on a rational basis, and that involves having serious academic economic forecasts," says Kenneth Clarke, the former Chancellor of the Exchequer.

Peter Bernstein, a Wall Street veteran and author of the definitive history of risk, Against the Gods, is more gloomy: he has given up forecasting altogether. "The world is too complex. And there are too many shocks," he says. "We don't know the future and we should manage our affairs accordingly."

'Market Prophets' is published by Reuters and Pearson Education next month at £24.99. David Stamp is a reporter at Reuters, the international news agency.