Online Polls: How Good Are They?
Most political pollsters rely on the same tool they have used for decades: the phone. Business, typically, does not. In recent years big companies--or rather, their market research consultants--have moved to the Web. For the likes of Microsoft (MSFT), McDonald's (MCD), and General Mills (GIS), the advantages of online questionnaires are clear: They can poll multitudes and ask people lots of questions quickly and cheaply. Plus there are mounting concerns that traditional polling will become increasingly arduous as more people ditch their landlines for cell phones.
Not all online polls are created equal, however. The results aren't always accurate and sometimes miss by a mile. "Companies who buy this research are saying to themselves, 'The price is good, and the firm tells me it's representative,'" says Jon A. Krosnick, a Stanford political science professor who has spent the past eight years studying the efficacy of surveys. Drawing hard conclusions from online polls, he adds, can be "like making an automobile out of soft plastic." More and more companies think so, too. Procter & Gamble (PG), for one, is enforcing stricter guidelines when conducting Web polling.
Since the 1930s, when George Gallup went door-to-door trying to divine which candidate was most likely to move into the White House, companies have used in-person surveys and telephone polls to learn who their customers are and what products they want. Many surveys used random-digit dialing devices, which selected individuals from the phone company's database of all known numbers. The method required a sample size of only a few thousand people to get an accurate reflection of the greater population's opinions. But it took a huge staff to call or visit them.
About 10 years ago the likes of Harris Interactive (HPOL) and YouGov began promising to reduce polling costs by half or more by shifting more business and political surveys to the Web. Companies liked what they heard. "I'll always start with online unless there's a reason not to--like, say, a really elderly audience," says Benjamin Malbon, director of account planning at ad agency Bartle Bogle Hegarty, which has used Web surveys to get reactions to Unilever's Axe line of male grooming products, among other brands.
Online polling offers some clear benefits. Because respondents can choose when to take the survey and how much time to devote to each question, they are more likely to provide thoughtful and candid answers. Companies can ask more sensitive questions (how often do you bathe?) because respondents aren't being queried by a human being. Marketers also can use the Web's video capabilities to ask respondents what they think of a new ad campaign or logo.
But critics point to a central problem with many online surveys: the pools of respondents, though massive, rarely represent the larger population. That, argues Stanford's Krosnick, is because the respondents aren't selected randomly, violating a core requirement of probability-based research. Instead, research firms reward volunteer participants with gifts, certificates, or cash. That's far cheaper than hiring a staff to call thousands of people, but it skews the sample in favor of opinionated Internet users, and risks attracting people who rush through surveys for the prizes. "These are, in effect, poll-taking clubs," says Gary Langer, director of polling at ABC (DIS) News, which won't air results from nonrandom polling. Plus results can differ drastically from one online survey to the next. P&G cites one case in which an online survey firm said consumers found a concept attractive but in a poll conducted a week later found it "below average."
Market researchers argue that you don't always need to reflect the entire population. Often they are trying to understand a specific group or are looking for useful suggestions. "We rarely need to do totally representative surveys," says BBH's Malbon. "We always have a target audience in mind, like 19- to 25-year-old boys, or housewives, or people who like tennis."
The firms also say they correct for the absence of random sampling. Either they select a subset they believe best matches the overall population they wish to assess in a given survey, or they weight the results by counting the opinions of underrepresented groups multiple times, as telephone polls do. Both solutions, say critics, assume the pollsters can identify all of the problems peculiar to the target group and then correct for them.
Despite the issues, online surveys are here to stay and will become more reliable over time. Clients are demanding it. P&G now insists that firms demonstrate that respondents make up a broad, highly representative group, not enthusiasts. Joan Lewis, the company's head of research, says she'd prefer respondents be contacted randomly--and some market research firms are doing that. One is Knowledge Networks, which conducts studies for Novartis (NVS), ESPN (DIS), and Wal-Mart (WMT). To build a more representative sample, it calls people randomly and invites them to take part in regular online surveys. The method offers the best of both worlds: the statistical rigor of phone polling and the flexibility of the Web.