The Polling Paradox: How to Read the Numbers

Two polls with contradictory results provide a teaching moment, and a veteran of the craft tells you how not to be a baby about reading them.

Republican Presidential candidate Jeb Bush greets supporters at the 4th of July Parade in Merrimack, New Hampshire.

Photo by Kayana Szymczak/Getty Images

Do you believe, as the ABC News/Washington Post poll showed this week, that Hillary Clinton is beating Jeb Bush nationally by a margin of 10 percentage points (51 percent to 41 percent)?

Do you believe, as the Quinnipiac University poll released Wednesday on a set of battleground states showed, that Bush as well as Marco Rubio and Scott Walker are besting Clinton by 5 percentage points in Colorado, 6 percentage points in Iowa, and 3 percentage points in Virginia?

Both can't be true.

There are basic questions I like to ask on whether poll results make sense. The recent ABC News/Washington Post and Quinnipiac polls provide a great opportunity to illustrate what factors we should consider.

I don’t know whether Clinton is ahead by 10 percentage points as the ABC News/Washington Post poll shows or is behind in key battleground states, as shown by Quinnipiac. Also, while I’m not sure that that any horserace numbers (accurate or not) mean much 15 months before the general election, what I do know this:

It is theoretically and mathematically possible that Clinton could be besting Bush nationally and losing in swing states, but not in practical terms. Yes, presidential elections are battles for Electoral College votes and it is the states that matter, but state results are not independent from national numbers. And, it would be phenomenally unlikely for a candidate to be up big nationally and be behind in swing states.

Also, looking forward to the next year when we will be flooded with state polls, state polling results can and should be compared to each other. Yes, there are 51 separate contests for electors, but those contests are not completely independent. There is a continuum of party performance of how states should line up in a presidential contest. No matter the year, Republicans will do better in Texas and Utah than California and Massachusetts. And, rarely do states leapfrog over one another in terms of their performance.

For example, Florida, North Carolina, and Ohio are all battleground states—and it’s hard to see how a Republican wins without winning all three. Still, while they all may swing in the same direction as they did for George W. Bush in 2004 or Barack Obama in 2008, they will not be decided by identical margin and, crucially, all three do not have the same probability of going for the eventual Republican or Democratic nominee.

A Republican may win, one, two, three, or none of the states, but a Republican will likely do better in North Carolina than Florida, and better in Florida than Ohio. And, on average, when looking at state polls, they should also reflect that logic or continuum.

Why? Fundamentally, the distribution of party identification and past voting trends tell us a lot about the basic political terrain and predispositions in a state. Predispositions in a state can change. West Virginia used to be reliably Democratic and Vermont reliably Republican, but there are rarely enormous jumps from year to year. There is some disagreement among pollsters about whether to weight to party identification, but an important factor to look at with any poll is the partisan distribution of the sample and how it compares to previous similar elections.

With a huge hat tip to Quinnipiac for being so transparent with their methodology, we can compare the partisan distributions in their state polls to previous years. Looking at exit poll results, which are generally but not completely comparable to phone poll findings, we see that the electorate in the Quinnipiac polls in all three states looks more Republican and more like the 2014 midterm electorate than the 2016 electorate. As the table below shows, Democrats went from a 5 percentage point advantage in Colorado in 2012 to a 4 percentage point deficit in 2014. The Quinnipiac survey has Republicans up by 3 percentage points now. While the general election electorate in 2016 may not be as Democratic as it was in 2012 (can the eventual Democratic nominee replicate the message and mobilizing muscle of the 2012 Obama campaign?) it is unlikely to look so similar to the Republican-leaning 2014 midterm electorate.


So, while we actually don’t know what truth is, except on the day after the election, there are some things that you can do to make sense of political polls. See if state results make sense when compared to other states and to national polls and look at the composition of the electorate. The nature of the times and muscular mobilization efforts can change the shape of electorates at the margin, but be wary of results that look fundamentally different from previous years.

Ken Goldstein is professor of politics at the University of San Francisco and is Bloomberg Politics' polling and political advertising analyst.