- Panel finds pollsters struggled to reach the apathetic, young
- Predictions also fell short in Israel, Canada, Greece, Spain
An inquiry into the failure of Britain’s pollsters to predict last year’s election found they failed to contact the right kind of voters, raising questions about whether their counterparts in other countries may face similar problems.
Following seven months of analyzing the hundreds of polls published in the run-up to the U.K. election, a panel of academics and opinion experts concluded the fault lay in “sample error” -- the people the companies were speaking to weren’t representative of the general population. In particular, they struggled to get apathetic people to answer questions.
“There were too many Labour voters and too few Conservatives in their samples,” Patrick Sturgis, a professor of research methodology at Southampton University who chaired the inquiry, said in an interview. “It seems it’s likely to be related to having too many politically engaged people.”
The British investigation follows a run of polling failures from Israel through Europe to Canada that have called the industry’s methods into question as it struggles to adapt to shifting voter behavior in the Internet age.
When the U.K. inquiry began, many polling companies described their difficulties in getting certain demographic groups, especially young people, to respond. They cited the increasing unwillingness of the public to answer land-line telephones for fear of sales calls, and the fact that many young people don’t have landlines at all. This isn’t simply a British phenomenon: In the U.S., the Pew Research Center said on Jan. 5 it will now be making three-quarters of its calls to mobile phones.
The young people pollsters did find in the U.K. may have been unusually politically engaged, leading to an overestimate of support for the opposition Labour Party within the age group, according to the report.
James Morris, a partner at Greenberg Quinlan Rosner, who ran polls for the opposition Labour Party in the run-up to the election, said one issue with many published polls was that they were commissioned by news organizations with tight budgets and had a focus on one or two questions about the election result.
“Voting-intention questions are asking people to predict what they will do, which they aren’t good at,” Morris said in an interview. He preferred other ways into the question, such as asking people which party leader they like most. “Rating leaders is them expressing what they feel now, which they know.”
The most recent example of survey shortcomings came in Spain in December, when pollsters forecast the result of the two traditional parties fairly accurately but struggled to gauge support for two insurgent parties, Podemos and Ciudadanos, both of which were running in their first general election and attracting new voters. The upshot was a four-way split leading to political stalemate that continues to cripple the country.
Greek voters wrong-footed pollsters twice last year, by unequivocally stating ‘No’ in a referendum in July over whether to accept bailout conditions by the European Commission and the International Monetary Fund, then again in September when they re-elected Syriza leader Alexis Tsipras by a unforeseen margin.
The phenomenon is not confined to Europe. In Israel, polls made it seem that Prime Minister Benjamin Netanyahu was at risk of losing his job in March elections, when in fact he won comfortably. Then in Canada, Justin Trudeau swept into power at the head of a majority Liberal government in October when most polls forecast a minority for his party.
One tactic to ensure a balanced sample is to repeatedly attempt to reach randomly selected survey targets who don’t respond on the first attempt. According to Morris, in the U.S. it would be common for a polling company working for a candidate to make five attempts to re-contact. This is supposed to reveal the preferences of harder-to-reach groups. It also costs money.
Sturgis said those paying for polls needed to consider whether they should spend more for better results. “It’s a message for the commissioners and reporters of polls as much as the pollsters,” he said.
He also had a warning for those hoping for an easy solution. “I don’t think it can be fixed in any simple permanent way,” he said. “Polls and surveys are inherently prone to error. You can reduce and mitigate the risks, but there will not be a silver bullet which solves these problems for ever more.”