Big data could help economists avoid any more embarrassing Michael Fish moments

Big data looks at the billions of decisions that people are making at the very moment that they make them, and then uses artificial intelligence to analyse the data to figure out what might be useful and what is not. No human could do this, for there is just too much stuff

There is hope that big data can help transform economic forecasting (Getty)

The “Michael Fish moment” may not mean much to younger readers, but for anyone over the age of 50 it has a nostalgic resonance: Michael Fish was the unlucky weather forecaster who assured TV viewers that there would not be a hurricane on Thursday 15 October 1987, the afternoon before the worst storm for 300 years hit Britain. That was, coincidentally, the weekend before the great stock market crash the following Monday.

The clip of that forecast has been played a few more thousand times over the past 24 hours, following Andy Haldane’s reference to it in the context of the failure of the economics profession to predict the great financial crash and subsequent recession of 2008-9. He also acknowledged the failure last year over the impact of Brexit, which contrary to the dire predictions of recession seems so far to have zero impact on economic growth.

He might, had he been even harsher, have added in not just the predictions that the UK economy would have a second recession in 2012, but the fact that according to initial statistics it actually had one. Those stats, we now know, were completely wrong. The economy was growing, albeit slowly, throughout.

Andy Haldane, who is chief economist for the Bank of England, was making a positive point. Weather forecasting, he said, had vastly improved over the past 30 years as more computer power and more data points had become available. His hope was that economic forecasting would similarly improve.

That is the fascinating question: will big data transform economic forecasting?

It is easy to make the argument that it won’t. There are a number of problems. One is that the way people respond to economic change – as consumers, business decision-makers and so on – are too subtle to be easily predicted. Another is that a lot of activity is mismeasured. For example the shift to people reading on screen instead of on paper shows up as a decline in GDP – no need to print or physically distribute the words – but actually there is an increase in economic welfare in terms of convenience. Still another is to point to the way econometrics has failed, or at least largely failed, to improve forecasting over the past 30 years. We have vastly more computer power but very little to show for it.

But I’m not sure this will remain the case. Remember that what has happened up to now is the application of more computer power to more data, but that data is still collected in conventional ways. Take the purchasing manager indices, regarded as the best predictors of economic growth for a few months ahead. They are simply the response to very crude questions as to whether companies think output, employment and so on will expand or contract.

Big data is quite different. You look at the billions of decisions that people are making at the very moment that they make them, and then you use artificial intelligence to analyse the data to figure out what might be useful and what is not. No human could do this, for there is just too much to consider, but AI eventually may work out how to do so.

It is quite hard to envisage the detail, but you can imagine the general picture. The gig economy is throwing off huge amounts of information that at the moment is hardly analysed from a general economic perspective. But maybe there are chunks of information that will give a good feeling for, say, overall employment and earnings in a year’s time. Maybe we will develop entire new models to explain consumer behaviour, and thereby make for a more efficient economic system.

In other words, you would not simply predict better – have a more accurate weather forecast – but actually improve overall economic welfare – have better weather. There are certainly lots of implications for economic policy: Ideology about what might work could be replaced by evidence of what does.

This will be a slow, stumbling, two-steps-forward, one-step-back march. But the prize is huge. Big data could be the driver of economic growth, the reason why the next generation will have higher living standards and – maybe more importantly – greater human welfare than the present one. Andy Haldane is one of the most thoughtful practical economists in the world right now, and we should take his concerns seriously. Prime among these is the uneven performance of the UK economy. The best is great, although there is a long tail of mediocre performance. But there was also an element of optimism about his remarks that did not come through in the press reports, and we need to cling to that.