Under Attack, Pollsters Debate Their Methods

In June, Blanche Lincoln, the two-term moderate Democrat from Arkansas, faced a tough run-off primary against Bill Halter, the state's more liberal lieutenant governor. Polls showed he had pulled ahead by as much as 4 percentage points. Lincoln veered left, attacking Wall Street and rebuffing requests from the White House and Federal Reserve to weaken a proposal she put in the financial regulation bill to impose tough rules on derivatives trading.

Lincoln's victory in that primary has set off a war over polling and the way it has been transformed by the Internet and cell phones. Research 2000, the only polling firm that released data publicly in the weeks before Lincoln's June 8 runoff, has been accused of falsifying its numbers in a lawsuit filed by the liberal blog Daily Kos, which commissioned the polls. Other pollsters are also under attack. In September, the American Association for Public Opinion Research publicly rebuked Atlanta-based Strategic Vision for failing to disclose its methodology after being accused of falsifying polls it conducted before the 2008 Presidential primaries in New Hampshire and Wisconsin.

The two episodes have prompted a vigorous debate over the reliability of polls and how they drive politics and policy. In the spotlight is the tight-knit, largely unregulated community of pollsters, whose efforts to obtain scientific results have been made far more difficult because of the Internet and mobile technology. "Something has got to change," says Nate Silver, who ranks pollsters on his popular website, FiveThirtyEight.com. "These were two fairly prominent pollsters who were accused of just making data up out of thin air." Silver, whose blog will be hosted on The New York Times website beginning in August, is embroiled in a polling controversy of his own: He has called Zogby International the "worst pollster in the world," arguing that Zogby's Internet-based surveys rely on an unscientific sample of participants who volunteer on the Zogby website. Chief Executive Officer John Zogby says his results are accurate, and spokeswoman Leann Atkinson says the company is preparing an article questioning Silver's methodology for ranking pollsters.

Polls are attracting attention because they increasingly feed an Internet-driven appetite for 24/7 political news. Negative poll numbers can deliver a fatal blow to candidates or make it difficult to raise money and build grassroots momentum.

Daily Kos says it discovered flaws in the Arkansas polls after it was approached by three statistical experts. The website began an investigation and, on June 29, Kos founder Markos Moulitsas published his conclusion that the site was defrauded by the polling company. The following day, Daily Kos sued Research 2000 for more than $100,000 in damages for breach of contract, misrepresentation, and fraud. The pollster's president, Del Ali, has denied the allegations. His lawyers declined to comment. Lincoln spokeswoman Katie Laning Niebaum says the poll "was not a factor" in the senator's decision to push her derivatives proposal, which ultimately was included in modified form in the final law. Halter's aides, who touted the polls during the race, say they privately questioned the Research 2000 results.

The cost of conducting scientifically sound polling has increased. Scott Keeter, director of survey research for the Pew Research Center and incoming president of the American Association for Public Opinion Research, says only about 20 percent of people contacted agree to take part in political surveys. That reluctance has forced pollsters to try new methods to get a statistically sound sample size. More than 20 percent of all U.S. households now only use cell phones, according to government statistics released in May. Including those users in surveys drives up polling costs because lists of cell-phone users cost twice as much as standard lists of registered voters, says J. Ann Selzer, president of polling firm Selzer & Co., whose clients include Bloomberg News. "Every month, it's harder and harder to do this job and do it right," says Selzer.

Many research organizations are turning to the Internet, though that method is also fraught. To get a correct sample, every participant must have an equal chance of being contacted, says Selzer, the top-ranked pollster in Silver's 2008 rankings. A truly random sample is hard to achieve online, given that there's no national registry of e-mail addresses. "The Internet violates sampling 101," Selzer says.

The bottom line: Polling's influence in politics and policy is growing as the reliability of some polls is declining.

    Before it's here, it's on the Bloomberg Terminal.