Recent elections around the world have not been kind to pollsters. In contests in Israel and Britain, and in a referendum in Scotland, pre-election polls all showed tight races that ended up being, well, not so tight. Although American pollsters have not had to explain errors of such magnitude, there were some danger signs in the last two elections in the United States, with some pollsters doing much better than others. On average, pre-election polls underestimated the magnitude of Obama’s victory in 2012 and of Republican victories in the 2014 midterms. There’s a variety of possible culprits for these discrepancies. One clearly is tied to the challenge in correctly measuring both the size and the composition of the electorate. There’s some evidence that the polls that underestimated Obama’s margin in 2012 erred by underestimating the non-white and Democratic share of the electorate. The problem in 2014, on the other hand, was a kind of mirror image. Many polls erred by counting drop-off voters—individuals who voted in presidential elections, but not in midterms—who happen to be more Democratic.
In an article earlier in the week, I discussed whether Iowa and New Hampshire polls were getting the electorate right. Given the prominence and importance of national primary polls in the Republican contest for deciding who is in and who is out of the first GOP presidential debate, and also creating the self-fulfilling impression of momentum, are these national polls getting the “who votes” question right? Are they correctly gauging the size and composition of their supposed target population, the Republican primary electorate?
The first problem with trying to measure the attitudes of a target population of the national Republican primary electorate is that such an electorate does not actually exist in the quite the same way that other electorates do. The presidential nominating process is a dynamic one, with primaries and caucuses that could theoretically play out over a five-month period. The size and shape of the electorate in each state will be determined by whether the race is still competitive when it reaches a particular state and by the number of candidates still running. Put another way, probably only Republican caucus goers in Iowa and, perhaps, primary voters in New Hampshire, will have the chance to choose between the current crop of candidates. And that’s making the outlandish assumption that none of the candidates drop out in the next few months.
What percentage of registered votes or how many people do these surveys assume will take part in the GOP nominating process? Let’s look at the recent Fox News poll. The survey sampled 1,019 registered voters and was conducted by a Republican firm, Shaw and Company, and a Democratic one, Anderson Robinson Research, using high-quality methods such as live interviewers and a standard dual frame design including both landline and cell phones. The results on a variety of questions such as whether voters approve of President Obama’s job performance are perfectly in line with all the other public polls that have recently been released.
To its credit, the survey also asked questions to gauge how likely respondents were to take part in a presidential primary or caucus and in which party’s contest they were likely to participate. This screening question yielded 389 registered voters (or 38 percent of registered voters in the country) who said they were likely to vote or caucus in the battle for the Republican nomination. Among this group, the survey found Donald Trump ahead of the field by 3 percentage points over Scott Walker. More importantly, in the contest to be invited to the debate in Cleveland, John Kasich and Rick Santorum tied for 10th with each capturing 2 percent, besting Carly Fiorina and Rick Perry, both of whom polled at 1 percent, and just behind Chris Christie with 3 percent.
So, while it’s hard not to observe that a grand total of six poll respondents—that’s six actual human beings—separate the ninth-place candidate from the 13th, let’s look at the survey’s assumptions about the size of the Republican electorate.
From the Census Bureau’s Voter Supplement to the Current Population Study, we know that there are 150 million registered voters in the country. So, the 1,019 respondents to the Fox survey correspond to the opinions of 150 million Americans. Accordingly, 38 percent of registered voters (those who were asked their GOP nominee preference) means that these 389 respondents represent the preference of just over 57.2 million voters. Is that a plausible estimate of the number of people who will take part in the 2016 Republican nominating contest?
Well, actually, no. In 2012, 19 million Republicans took part in the nominating contest. In a more competitive race with a wider GOP field in 2008, 21 million took part in the nominating contest. On the other side in 2008, for some additional context, in the highest number of voters ever to take part in a nominating contest, just over 37 million took part in Democratic primaries and caucuses. (By the way, want to start a fight in Hillary Clinton’s Brooklyn HQ, ask old Obama and Clinton staffers who won the most votes and what should have been done with Michigan and Florida vote totals.) And, in the 2012 general election, Mitt Romney received 61 million votes. Put another way, the Fox News-survey, and most other surveys of Republican primary preferences are using samples that represent target populations that are nearly equal to the total number of votes that Mitt Romney received in the general election in 2012, about three times as large as recent primary electorates and almost twice as large as the record-shattering Democratic primary electorate in 2008. While there are lots of candidates and some enthusiasm on the GOP side, it is not likely to approach 60 million, as the Fox and other recent polls assume.
Even if GOP turnout more than doubles from 2012 and exceeds Democratic turnout in 2008, the attitudes of 20 million non-primary participants are being included in these polls. If their attitudes differ from the other 40 million, then the polls are not an accurate representation of who is winning—and certainly not an accurate representation of who is in 11th place, is such a thing is even possible.
Typically, polls don’t have a huge causal influence on the direction of the race. In other words, polls predict elections, they are not the election. But with the responsibility of determining who is going to be in the first Fox News debate the polls will have a clear causal effect on who is in the debate and who is out, and, thus on the possible direction of the race. Too, the polls supply media oxygen, never so crucial as this year, with Trump consuming so much of it.
Given the importance of getting the right target population, another important question arises: Do the samples that are being interviewed, in the surveys Fox News is using, map to the target population of the Republican primary electorate? In predicting election outcomes accurately, measuring who is going to vote is just as important as measuring whom people are going to vote for. In this case, the polls used to determine debate participants are creating a 60-million-strong electorate that will never exist.
Ken Goldstein is professor of politics at the University of San Francisco and is Bloomberg Politics' polling and political advertising analyst.