Polls Often Wrong But You're Stuck With Them, U.K. Inquiry Findsby
Probe concludes that `scope for radical change is limited'
Polling last year failed to predict Cameron's election victory
The inquiry into one of the worst disasters in British political polling has concluded that there are few measures companies can take to improve their results. Instead, it recommends those looking at the results of polls take a more skeptical view.
Polling companies took a reputational dent last May when they all forecast a tight result in Britain’s general election, in some cases showing the opposition Labour Party ahead. In the event, David Cameron’s Conservatives won by 38 percent to 31 percent, a gap much wider than in any published poll.
The inquiry’s final report, published Thursday, repeated its interim finding, that the main cause of the error was unrepresentative samples: Pollsters spoke to too many Labour supporters and too few Tories. But while it made recommendations around how to present polling reports, it had no simple solution to the problem.
“The scope for radical change is limited,” Patrick Sturgis, the inquiry chairman and a professor of research methods at Southampton University, told reporters at a briefing in London on Wednesday. “There’s not a hell of a lot we can do. Betting markets and Facebook analysis are nowhere near being able to do this, so the polls are all we have.”
Technical recommendations from the inquiry include requiring pollsters to publish confidence intervals for their results, rather than a simple number. That might mean results being given as a range, though details have yet to be worked out. They should also publish data on the statistical significance of changes in results and be more transparent in how they apply weightings and in any changes that they make.
While the inquiry found some evidence of pollsters “herding” toward a final forecast, Sturgis suggested this was likely to have been unconscious. John Curtice, president of the British Polling Council, pointed out that the herding had moved the results closer to the correct one, rather than further away. He said there was some evidence that pollsters themselves had doubts about whether Labour would really do as well in the election as their polls suggested.
Both Sturgis and Curtice urged those commissioning polls to reverse the recent trend toward more frequent surveys. “Do bigger surveys over a longer period of time,” Curtice said. “They may be more accurate.”
Pollsters face an immediate problem in the June 23 referendum on European Union membership. The volatility of the pound has risen as markets try to factor in the possibility of a so-called “Brexit.” But different polling methods give different results: While online polls show the two sides in the debate neck-and-neck, telephone polling has found a lead, albeit a shrinking one, for staying in. One analysis published Wednesday suggested neither was correct, but that the phone polls might be closer to reality.
Asked if he was suggesting that, because the polls were hard to fix, the consumers of polls should be fixed instead, Sturgis laughed.
“Stakeholders need to be more realistic about the risk that polls can be wrong,” he said. “The historical record shows that they are wrong quite frequently.”