"The world may have a polling problem," stats guru Nate Silver declared at his news site FiveThirtyEight.com Thursday evening.
It sure looked like it. Forecasters — including Silver, who made his name with dead-on predictions for the New York Times about the 2008 and 2012 elections — had almost completely miscalled the day's U.K. election. They had basically agreed the Labour and Conservative parties would win about the same number of seats, and neither would achieve a majority. Instead, as Silver was channeling his profession's angst into a blog post, the Conservatives were on their way to winning 99 more seats than Labour did, handing Prime Minister David Cameron a 331-seat majority.
It wasn't the only high-profile failure, Silver pointed out: Polls had missed or understated the outcome of the Israeli elections of March, the U.S. midterm elections in November 2014, and the Scottish independence referendum two months before that. He linked to another FiveThirtyEight post that wondered if polling was "in crisis" as response rates plummet and technology makes it harder to reach some voters and easier to reach others in ways that are difficult to compensate for.
An informal poll of pollsters found most tend to agree that their world has changed.
"The future is complicated and worrisome," says J. Ann Selzer, president of Iowa-based Selzer & Co., who has conducted polls for Bloomberg. "There are all kinds of changes that make it difficult to end up with an accurate cross-section of your population of interest."
Cell phones are a big part of it. Less affluent voters, paying for each mobile minute, often do not want to take 20 minutes to finish answering a set of personal questions for a stranger, pollsters say. That means forecasters have fewer answers from a key demographic than if they had an equal-opportunity way to reach voters. In the past, landlines were everywhere and allowed pollsters to approximate a representative sample; now, challenges seem to be mounting.
Younger respondents are more likely to use email; older respondents are more likely to have landlines. Nobody really knows if what how much demographic groups – based on age, wealth or first language — tend to participate in polls. At a time when both U.S. political parties are battling to win votes with a number of demographics, those blind spots can make a big difference. Attempts to correct for this uneven coverage statistically can rely on bad assumptions that make the problem worse — an issue that famously dogged the Romney presidential campaign's internal numbers in 2012.
"It has definitely made it irrevocably more complicated," said Michael W. Link, the president of the American Association for Public Opinion Research.
Still, pollsters aren't ready to walk away from their profession.
Some say the British failure may be an anomaly that's unlikely to affect U.S. politics: John Curtice, who analyzed Thursday's result for the BBC, had warned Bloomberg in April that the strength of five parties in the election, which typically only sees two or three parties competing for a large number of seats, meant "disappointment" was "almost guaranteed." The same issues could have affected the multi-party Israeli election, according to Clifford Young, the head of global elections for Ipsos Public Affairs.
British pollsters may also have a less accurate record, FiveThirtyEight suggested.
Pollsters say accuracy is still achievable — with enough time and money.
"We still know how to do it," Zac McCrary, a partner in Anzalone Liszt Grove Research, a Democratic polling firm. "It just takes more expertise than it used to, and it takes more expense, more commitment... Campaigns can have a high degree of faith that they’re getting good advice."
And no one thinks campaigns are going to stop paying for polls. Certainly not the head of the American Association for Public Opinion Research.
"Information is needed about public attitudes," said Link. "The question is just how to we get that."