How One Pollster Saw Trump’s Win Coming
As Hillary Clinton's nightmarish election night unfolded, the narrative coalesced around one question: How did the pollsters not see this coming?
Not everyone missed it. While closely followed polling averages gave Clinton a 3.2-point edge and preelection forecasts showed her as a heavy favorite, a team at the University of Southern California led by professor Arie Kapteyn had managed to design a poll that proved to be one of the great contrarian forecasts in the modern history of U.S. elections. As of Tuesday morning, it showed Donald Trump leading by a little more than 3 percentage points.
For much of the year, the 2016 USC Dornsife/Los Angeles Times Presidential Election Poll—designed to track changes in voter opinions throughout the campaign, using what experts called a unique and more complex weighting model—behaved as a quirky outlier. It pointed all along to a Trump victory on the magnitude that came to pass Tuesday night.
Since then, Kapteyn and his team have been swamped with requests for more about their methods.
"I wouldn’t necessarily say that the polls were all very far off," Kapteyn said in an interview. "It is just that most were on the wrong side of the final result."
Kapteyn thinks some polling models probably misjudged the turnout in this election of people who did not vote in 2012. The candidates and their messages were so completely different this time, appealing to different sets of interests. And Kapteyn's data were showing that these 2012 nonvoters—if they did vote this time—were more inclined toward Trump than Clinton.
As the race came to an end, most polls had converged around a narrow consensus for Clinton, a phenomenon known in the trade as herding. But the USC poll didn't join the herd; it had been sticking out like an ugly wart for months. Week after week, the poll showed something remarkably different. Take the consensus polling numbers for Clinton and Trump, shift them 4 to 6 points in Trump's direction, and that's where the USC poll usually settled.
Partisans on both sides noticed. The poll became a constant Drudge Report headline throughout the election season when it would expand into a 4-point Trump lead. The Daily Kos, a progressive website, freaked out with a July 28 headline: "WTF is up with the USC Dornsife/LA Times tracking poll?"
As of Tuesday, the poll’s final forecast showed Trump leading by a little more than 3 points—46.8 percent to 43.6 percent. Its final accuracy relative to the outcome of the election is still being calculated, but it’s safe to say the poll outperformed the consensus by a wide margin.
How did the team at USC accomplish this?
The poll was set up differently than other major polls. Roughly 3,000 respondents were recruited into a panel that used an unusual method of "micro-weighting" to reflect the overall voter population. The poll was conducted by dipping back into this same pool of people each time. This may have created a more stable baseline from which to detect shifts in voter preference.
The poll design allowed respondents to assign themselves a probability, from zero to 100, of their voting for either candidate. This approach, rather than simply asking for a concrete voting preference, may have allowed the poll to be more precise in detecting shifts in sentiment. The USC poll’s results also were weighted based on how people said they voted in 2012—an approach that experts criticized on the basis that many people misstate or misremember how they voted in the past.
Many pollsters treated the outlier USC poll with a balance of respect and skepticism, but mostly skepticism. The consensus was voiced by Princeton's Sam Wang, editor of the Princeton Election Consortium, in a post on his website.
"I think [the USC poll] is one of the more interesting surveys this year," he wrote in October. "I just don’t think it means that Trump has ever been ahead in the general election, an idea that is contradicted by other polling evidence.”
Since nearly everyone else proved so wrong with their polls, perhaps no one will come in for special criticism. Just about the entire political polling industry missed this one. But bruised egos aside, there's a real chance for improvement. USC, in a rare display of transparency for political polling operations, has published its methodology for anyone to study.
Kapteyn believes it is critical for pollsters to cover every part of the population, which most online polls are unable to do, and, importantly, to have a good model of who is actually going to vote—something sorely missing ahead of Tuesday night.
Analysis of the data collected isn’t an exact science, either. Kapteyn recalls an infamous story in the polling world where four reputable firms using the same state polling data predicted four very different outcomes that varied by five points.
“On the same data!” he said. “So there’s always some room for better modeling."