The venerated Gallup Poll took a beating this year for being such an outlier in its predictions of who would win the presidency, and the New York Times’s Nate Silver has now served up further evidence that the firm has lost its mojo. Silver analyzed the results obtained by about two dozen top polling firms that ran multiple surveys in the last three weeks of the campaign and then ranked the pollsters based on their accuracy: Gallup came in dead last, with an average error of 7.2 points compared to the actual results.
What pollster nailed its predictions? A firm you probably haven’t heard of: TIPP Online, which conducts polls for Investor’s Business Daily. By Silver’s count, TIPP’s average error was just .9 points.
TIPP was founded in 1996. The firm publishes the Investor’s Business Daily/TIPP Economic Optimism Index, as well as consumer trend data. Political polling is just one part of the firm’s business, and yet its track record is pretty good. Internal data show a .04 percent margin of error for polling published the day before the 2004 presidential race and a zero margin of error for a similar poll in 2008. This year, TIPP President Raghavan Mayur says, the Ramsey (N.J.) firm faced a setback when Hurricane Sandy sapped its power right before the election. Because of that, TIPP sampled only about 712 people in the final three days of its tracking poll, down from 1,300 typically. Yet the pollster still outdid every other firm, according to Silver.
Mayur says his interviewers try to reach people at least five times, which corrects for the probability of oversampling the type of people (older, retired) who are likely to be sitting by the phone all day. One-third of the homes Mayur’s firm calls are cellphone-only households. These tend to be younger, minority, lower-income, and Democratic-leaning. Pollsters that merely call landline-only households run the risk of oversampling Republicans. After each campaign, Mayur “back-tests” his model—that is, he analyzes its performance and then makes tweaks to improve it. “Most of the work is done after the election, rather than before the election,” he says.
A good poll factors in not only how people will vote, but the likelihood that they will do so. Much of TIPP’s so-called back-testing focuses on how well the model predicts voter likelihood. (This is where Gallup’s tracking poll may have gone off the rails, Silver suggested.) To gauge enthusiasm, Mayur says TIPP asks people about their interest in the election at two different points during the telephone survey. At the end, the pollster asks respondents how probable it is that they will vote.
Mayur’s final tracking poll of likely voters, published on Election Day, included 38 percent Democrats, 31 percent Republicans, and 32 percent independents. (Gallup, on the other hand, figured Republican turnout would beat that of Democrats by one percentage point.) While the final results, including voters’ party affiliations, won’t be available from states until Dec. 20, exit polling from Nov. 6 suggests that Mayur’s model is working well when it comes to voter-likelihood: Democrats’ turnout was six percentage points greater than Republicans’.