Why you should be very careful about believing polls on the EU referendum

Online and telephone polls are showing very different results

The referendum is due before the end of 2017 (PA)

Polls of support for leaving or staying in the European Union in the upcoming referendum vary dramatically depending on whether they are conducted online or over the telephone, according to a new experiment.

Researchers at the public opinion firm ComRes tested two identical polls asking the question set to be posed to voters in the EU referendum, which is due to be held before the end of 2017.

One of these polls was presented to a scientifically weighted sample online, while the other was presented to a scientifically weighted sample over the telephone.

The results varied quite dramatically – with the online poll showing a neck-and-neck race but the telephone poll showed a clear lead for the “in” campaign to remain in the bloc.

“Much of the polling on the EU referendum since May has been done online, and much of it points to a race which is neck and neck,” the pollsters Katharine Peacock, Tom Mludzinski, and Adam Ludlow explained in a write-up of the experiement.

“But telephone polling again suggests a different story: this time not showing a modest lead of a few points, but an unequivocal lead – for staying in the EU.”

“In order to understand this apparent difference between the methodologies we undertook our own experiment, asking the referendum question on our online and telephone surveys,” they said.

“The results were telling – both sets of data used the same question, were run within days of each other, had the same demographic weights applied and were past vote weighted in the same way – but despite this, the online polling showed the public split on the issue (as it does in polls conducted by other pollsters online), whereas the telephone polling showed a large lead for ‘Remain’.”

Though the pollsters cannot know for sure which poll is closer to the true figure, they said that the experience of May’s general election suggested that telephone polling produced more accurate figures.

Online polls are conducted from ramdomly selected voters drawn from online panels. Those panels are voluntary and opt-in - but supposed to be large enough to provide firms with a balanced sample.

Telephone polls are performed by randomly dialling UK numbers - with some firms including mobile phones and others only landlines. These polls have a very low response rate as most people refuse to take part.

The pollsters warned that online polls tended to reach people who were more politically engaged and exposed to strongly held beliefs, and that this could lead to misleading results.

Similar effects were seen in a split between firms which used online and telephone polling methods during the 2015 general election, they observed.

ComRes however said that within its own Westminster voting intention polls there was very little variation between its online and telephone polls, and that “the extent of the difference here seems to be instead related to the issue of the European referendum itself”.

The telephone poll in ComRes’s experiment found that 56 per cent of people wanted to remain in the EU while 41 per cent of people wanted to leave. The online poll however found 41 per cent supporting either side.

The number of people who said they did not know was eight per cent by telephone and 18 per cent online, however – a finding that is curious in light of the pollsters’ hypothesis that people responding to polls online are more politically engaged.

The experiment follows headlines based on polls suggesting that the race to stay in or leave the EU on a “knife edge”.

The referendum was promised in the Conservative manifesto and is expected to take place after David Cameron completes a renegotiation of the terms of membership of the bloc.