No matter the subject, for surveys to make sense, they have to be sampling the correct population. If you want to learn about car buyers, a researcher needs to interview people who buy cars. If you want to learn about diaper buyers, you need to interview people who buy diapers. If you want to learn about beer drinkers, you need to interview beer drinkers. And if you want to learn about likely Republican caucus-goers in Iowa or primary voters in New Hampshire, you should interview likely caucus-goers in Iowa and likely primary voters in New Hampshire.
So, with Sunday's NBC News/Marist University polls in Iowa and New Hampshire igniting a flood of news coverage highlighting Donald Trump’s “surge” (Politico) and “strong” showing (NBC and the New York Times), in these crucial early states, it’s worth examining who was actually interviewed. The poll's exact finding is that Trump leads the field in New Hampshire and comes in a close second to Wisconsin Governor Scott Walker in Iowa among what NBC News and Marist describe as “potential GOP voters.” While many polls use self-identified partisans as their primary sample, that’s a term not typically used by campaign pollsters, and is a much broader population than campaign pollsters would typically target.
In Iowa, for example, Marist pollsters interviewed 1,042 residents, of whom 919 reported that they were registered voters. The proportion of registered voters in the sample is absolutely in line with the nearly 90 percent of adult Iowans who are registered to vote (among the highest percentages in the country). From that sample of 919 registered voters, Marist pollsters qualified 342 respondents as the potential Republican electorate, defined as “all voters who prefer to participate in the Republican presidential caucus and registered Republicans without a caucus preference.” These are the people who are questioned as to who they support in the GOP caucuses.
These results mean that NBC and Marist are assuming a caucus turnout that is 37.2 percent of registered voters (342 divided by 919). According to the most recent figures from the Iowa secretary of state, there are 2,069,592 registered voters in the Hawkeye State. Multiplying that number by 37.2 percent yields a population of 769,888. The NBC/Marist poll release is careful to note that this is a potential electorate—but it represents a turnout that is 650,000 people greater than the largest turnout ever for the Iowa GOP caucuses and 600,000 more than the very highest estimates I've heard from political operatives on the likely size of the 2016 caucuses.
In New Hampshire, 1,037 adults were interviewed, of whom 910 (88 percent) reported being registered, which is very much in line with the over 80 percent of Granite State adults who are actually registered to vote. From this set of 910 respondents, 401 (44 percent of registered voters) were identified as members of the potential Republican electorate. Multiplying 44 percent by the 874,417 people registered to vote in New Hampshire yields a potential Republican electorate of 384,743—which is, again, significantly higher than in any previous election. The turnout in the 2012 Republican primary was 248,475 and the turnout in the 2008 Republican primary was 234,851.
Pollsters certainly want to keep their eye on surges of new voters and possible differences in the size and composition of the electorate. Barack Obama, after all, owes his victory in the Iowa caucuses and his presidency to the fact that he brought new voters into the process. Still, the population interviewed in the recent NBC/Marist polls encompasses a much broader number of people than are actually likely to caucus in Iowa or vote in the Republican primary in New Hampshire. If the nominee preferences of this larger pool of potential voters are different than the voters who actually show up, the polls could be painting an inaccurate picture of who is ahead and who is behind.
The poll paints an interesting picture of where the race stands with all possible Republican voters. What it's not, necessarily, is an accurate picture of who's actually ahead and behind in the crucial early states.
Ken Goldstein is professor of politics at the University of San Francisco and is Bloomberg Politics' polling and political advertising analyst.