Failed Polls in 2016 Call Into Question a Profession’s Precepts

The polls blew it.

U.S. survey companies and media organizations that collectively presaged a close Hillary Clinton victory now face an autopsy on how they got it so wrong after a year suffused by polls, aggregates of polls and even real-time projections of the vote on Election Day. 

While the predictions gave some observers a soothing sense of certainty, actual voters still possessed the capacity to shock. Donald Trump’s commanding performance defied the final surveys of the American electorate, which broadly predicted a Clinton win of 2 to 4 percentage points. 

“It’s harder and harder to poll today, to get a sample that looks like the electorate,” said Karlyn Bowman, a public opinion analyst at the American Enterprise Institute in Washington. “We’ve seen epic fails.”

Tuesday’s results were just the latest high-profile predictive failure around the world, following on the heels of misleading surveys on the Colombia peace deal referendum this year and Greece’s bailout referendum in 2015. Surveys were rendered inaccurate by new forms of technology and communication and political questions unlike any seen in recent history. The inaccuracy called into question a basic gauge of sentiment used by politicians, citizens and financiers.

“The anger is stronger than any of us really expected," said Megan Greene, chief economist at Manulife Asset Management in Boston, which handles money for institutional investors such as pensions and foundations.

Missing Voters

In the U.S., questions linger about how to slice the electorate and how to weight under-represented demographics -- whether by ethnicity or location or political affiliation -- while Americans increasingly withdraw from survey participation and view pollsters themselves through a political lens.

Final tallies produced by CBS News, FiveThirtyEight, Fox News, Wall Street Journal-NBC News and Washington Post-ABC News all predicted a relatively safe 4-point win for Clinton. Only slightly less wrong were polls by Bloomberg Politics and New York Times’s Upshot, which estimated a Clinton victory by 3 points. Rasmussen Reports called for a 2-point Clinton triumph.

A few got it right: The Investor’s Business Daily-TechnoMetrica Market Intelligence poll and a tally by USC Dornsife-Los Angeles Times were among the rare outfits to call the election for Trump, by 2 and 3 points.

Peter Woolley, a professor of comparative politics at Fairleigh Dickinson University in Florham Park, New Jersey, said a key part of the difference between expectations and the results was that people simply expected surveys to be too precise. Woolley is a past director of the PublicMind polling institute at the university.

"Polling is a scientific method to arrive at an estimate," he said early Wednesday. "We tend to over-report the accuracy of the poll, and tend to forget very quickly that it’s an estimate within a range."

J. Ann Selzer, an Iowa pollster who conducts surveys for Bloomberg Politics, said her trade entered uncharted territory this year as it attempted to deal with the spread of wireless communication and a demographically volatile electorate.

"There was a lot of experimenting this year with the types of questions they were using, the types of methodologies they were using,” she said in an interview at Bloomberg News headquarters in New York. “There’s the continuing barrier of the lack of landlines, the erosion of landlines. In the old days, if we knew your landline phone number we knew where you lived and that was fantastic for pollsters. Now it’s very difficult."

Turning points in the race happened at poorly timed moments, she said.

"Day by day, things happened that would break the poll that was currently in the field,” she said. “Even since Sunday, when most polls were done, things changed."

Nobody Home

Pollsters have been dealing with a decline in participation for decades. The share of households that agreed to participate in a telephone survey by the Pew Research Center steadily dropped to 14 percent by 2012 from 43 percent in 1997.

Read more: The perils of polling -- a QuickTake on why pundits were wrong

When the phone is answered, many are unwilling to speak plainly.

“Let’s say you hold a controversial opinion,” said Greg Valliere, the Washington-based chief global strategist at Horizon Investments LLC, a Charlotte financial adviser. “Are you going to tell a complete stranger that calls you up at 8 in the evening, or will you keep it to yourself?”

Polls abound. For many media organizations, they serve not only as tools for their own use, but as calling cards and valuable franchises as the Internet erodes their authority and earning power. In recent years, statisticians such as Nate Silver, founder of ESPN’s FiveThirtyEight, have achieved a sort of nerdy celebrity.

Meanwhile, the grandfather of U.S. opinion polling, Washington-based Gallup Inc., has pulled back. Four years ago, Gallup endured its third polling defeat in four cycles and walked away from presidential horse-race polling altogether. An internal review found that four areas of the methodology contributed to the mistaken predictions: likely-voter estimates, regional representation, weighting of race and ethnicity, and outreach to landline-phone users.

Among high-profile failures this year were the Michigan Democratic primary and the Colombian peace deal.

In the March 8 primary, Clinton had been favored by about 20 points in surveys, only to lose to Vermont Senator Bernie Sanders by almost 20,000 votes. In Colombia, voters in October rejected a peace deal with rebels after several surveys in the prior week had predicted the referendum would pass by 60 percent or more.

At the same time, pollsters have gotten some relatively undeserved criticism. The U.K.’s June vote to leave the European Union, often called a surprise result, actually was largely deemed too close to call by opinion polls. While markets priced in a vote for "Remain," traditional tallies were much closer to the end result for a Brexit win.

Trump has been outspoken on polling results throughout the campaign, at times lauding those that project he has more support, while deriding others for shoddy methodology or bias. 

“I do think a lot of the polls are purposely wrong,” he said Tuesday on Fox News. “I think a lot of the polls are phony -- I don’t even think they interview people.”

Bowman of the American Enterprise Institute said every precept must be re-examined.

“I don’t think the business is particularly introspective, but it needs to be going forward,” she said. “This has been a business that’s told us so much about America. ... To lose that going forward would be a real problem.”

— With assistance by Ben Brody, Robert Hutton, and Patricia Laya