David Blanchflower: The 'shy Tory' voters were missed by the pollsters, and that fooled us all

Prediction markets seem to predict stuff pretty well, while UK pollsters don’t seem to, because in a prediction or betting market, people back their opinions with real money

Protests after the election: will the Tories unleash more austerity? (Getty Images)

It’s 1992 all over again. There was a late swing to the Tories. The Lib Dems were wiped out and down to single digits. Labour were decimated in Scotland and lost seats. Miliband, Clegg and Farage were gone along, with my old mates Ed Balls and Vince Cable. Labour’s slab of limestone with six promises on it, that I didn’t understand, presumably is headed to the knacker’s yard. The markets responded positively; with a rise in both the pound and the FTSE by 2.3 per cent, not least because the potential uncertainties over the forming of a government over the next month or so disappeared. Markets dislike uncertainty.

The “shy Tory” voters were missed by the pollsters, and that fooled us all. The newspapers baying for blood, The Times, Telegraph, Sun, Mail, the FT, the Economist backing the Tories and no support at all from businesses sunk Labour. It looked like a stalemate in the polls, but there was always the potential of a late move to the incumbent. This is exactly what The Independent’s John Rentoul has predicted these many months.

The exit poll was a stunner that made Paddy Ashdown say he will eat his hat, marzipan or otherwise. At least we won’t have to watch Dan Hodges run naked down Whitehall, which he said he would do if Labour won. His many columns saying Labour would lose and Ed Miliband was hopeless were spot on, loath as I am to admit it. It seems the voters had little idea what Labour was for – Osborne lite was never going to cut it.

The polling guru Nate Silver noted that his pre-election forecast, along with everyone else’s, did project a narrow Conservative plurality, but not an outright majority. He admitted that “the forecast assigned too little of a chance to an outcome like this one, especially given that there have been significant polling errors in the UK before”.

Was it really a surprise? I started having my doubts on election day when I had an email from my Dartmouth colleague Eric Zitzewitz, an economist who is an expert on prediction markets*, warning me that it looked like the Tories might win outright.

It went as follows. “From what I can tell by comparing Betfair.co.uk and this site, http://electionforecast.co.uk/) betting markets seem to be expecting a 7-10 seat shift to the Tories relative to the website’s forecasts, which I’m assuming is close to a statistically optimal aggregation of current polling information, but who knows. The median forecast at Betfair.co.uk is 288 seats rather than 281 on the website, with a small probability of a huge shift with odds of 301 seats around 26 per cent, odds of 326 seats 6 per cent. It could be rational expectation of a rightward surprise (“shy Tories” and so on), rightward bias in markets, and/or an overpricing of small probabilities.”

Looks like he was right. He sent me a follow-up on Friday: “If exits hold up, I guess the markets were right to hold out a small probability of a big shift … Maybe the meta learning is that polling is harder in the UK than here …”. That looks right.

Prediction markets seem to predict stuff pretty well, while UK pollsters don’t seem to. In a prediction or betting market, people back their opinions with real money. They therefore have an incentive to take all the information available to them, like forecasts and polls, and make whatever adjustments they deem necessary. When markets work well, the prices reflect an average of many people’s private opinions, weighted by how confident they are. Particularly when you are dealing with a politically loaded subject, this can be more informative than an average of public opinion.

Despite much learned comment, including from the former governor of the Bank of England Mervyn King, who argued that Labour was not to blame for the recession, the public still seemed to blame the last Labour government for “overspending”. Even in the last hours before election day, Prime Minister Cameron was blaming the problem on “Labour’s recession” despite the fact it was a global recession that started in the US sub-prime housing market and spread to every advanced country at the same time. For whatever reason the Labour Party was never able to change that narrative. YouGov polls, for example, show a consistent pattern of support for spending cuts, right up to the last poll on 3-4 May.

This is apparent despite the fact that the majority have consistently seen spending cuts as being done “unfairly”, “too deep”, “too quickly”, and adversely impacting on their own lives. Despite all that, spending cuts continue to seen as “necessary” and “good for the economy”.

Mervyn King, former Governor of the Bank of England, argued that Labour was not to blame for the recession (Getty Images)

Of particular note is that there has been little or no change in the proportion of respondents to YouGov surveys who said that they blamed Labour versus the Con-Lib coalition. This was 20 per cent versus 46 per cent in 2010; 26 per cent to 36 per cent in 2011; 26 per cent to 35 per cent in each if the three years of 2012-2014. In the nine 2015 polls the averages were 30 per cent blaming the Con-Lib coalition and 35 per cent blaming Labour. The failure to address this narrative proved fatal.

The big question now is where is the economy going, and will George Osborne release another round of austerity on the economy. If so where will the spending cuts fall? Will there be another rise in VAT? What will he do if productivity growth doesn’t pick up, as seems likely? There is no evidence to suggest it will any time soon. What if investment slows and companies like HSBC quit the UK because of uncertainty over Brexit? What if living standards don’t pick up? Will we see a reversal of austerity, as we saw in 2012 through 2014?

It is time for me to congratulate the winners, eat humble pie, shake hands and move on. But when that is over, it’s back to business ripping chunks out of Tory economic policy on a regular basis.

*See Eric Zitzewitz and Justin Wolfers, “Prediction Markets”, Journal of Economic Perspectives, winter 2004, and with 17 co-authors, “The Promise of Prediction Markets”, Science, May 2008.