The Perils of Market ResearchSteve McKee
My colleagues and I conduct quite a bit of market research, both for our own company and on behalf of our clients. We've seen it used both for good and for ill. Here are a handful of thoughts that can keep your research from causing you pain.
1. There are things you can measure and things you can't. Don't mix them up.
How much do you love your wife? What's the value of poetry? What is a life worth? Ask most people these questions, and you'll get a funny look—or get into a metaphysical discussion. Some things just can't be quantified. Yet in business, we often act as if everything can.
In a 2008 Wall Street Journal story, Adrian Van Hooydonk, director of design at BMW (BMW:GR), explained how the carmaker evaluates vehicle prototypes: "We don't use customer clinics. They will be judging it based on the world today. Design needs to look good in eight years' time. You can't ask a customer whether he will like the design of the car in 2018." Van Hooydonk and his team must be onto something, because BMW has arguably been the most stylish and best-performing car company of the past two decades.
Research can't predict the kind of cars we will be interested in five years from now, how an ad concept will be received three months from now, or what the next hit movie or popular fashion trend will be. Yet we still hold onto hope that somehow statistics and spreadsheets will enable us to foresee the future. They won't. They can't.
Of course, it is possible to track events that have already happened, and that can provide valuable information. Such things as purchase patterns and visit frequency are historical, concrete events subject only to the laws of forgetting (I may not remember how I heard of your product) and deceit (I may not want you to know that I saw your ad in my wife's Glamour magazine). By and large they can be reliably tracked. But when we try to quantify attitudinal attributes—or assume that the past will accurately predict the future—we can get into trouble.
2. Just because you can't measure it doesn't mean it's not real.
When was the last time you responded to a TV ad—or any form of advertising, for that matter? That, of course, depends on the meaning of the word "responded." It's easy to think of "response" strictly as a measurable action, such as an inquiry or purchase. But there are many different ways consumers respond to ads, from the concrete (Web visits, phone inquiries, transactions) to the more abstract (preference, likability, identification). And the higher-involvement the purchase decision, the more likely a brand will need to generate a number of abstract responses before it sees anything concrete.
In 2003, Advertising Age ran a provocative story about advertising ROI—or the lack of it. The publication said that according to research, "Media advertising does the worst job of any marketing discipline in proving return on investment, and network TV is the worst of those media…."
The study was fielded among leading national advertisers—people who ought to know. But what, exactly, did it demonstrate? Not that network TV didn't generate ROI, but that it was the hardest medium with which to prove ROI. That's a significant distinction.
If network TV didn't work for advertisers, there would be no network TV. As business people, we know that TV works because it works on us as consumers. We may not be able to measure it with precision, but anyone who recognizes the Mac vs. PC guys, the Aflac Duck (AFL), the E-Trade Baby (ETFC), or the Budweiser Clydesdales (BUD) can't deny it. Passing judgment based only on what's easily measurable is myopic.
3. The best research is the real world.
Two words: New Coke. Possibly the most pretested new product launch ever, New Coke nevertheless failed in the marketplace. The story has been told many times, and there's no need to repeat it here, but basically what it came down to was that despite an unprecedented amount of market research, the brain trust at Coke (KO) didn't anticipate the monstrous emotional backlash they would engender by replacing their original formula. They couldn't know it until it happened.
Google (GOOG) once asked users how many research results they'd like to see on one screen. Since conventional wisdom says more is always better, people naturally said "more." When Google tripled the number of results, however, it found that traffic actually declined. Not only did the results take a fraction of a second longer to load, but having more options led people to click on links that were less relevant. The respondents in Google's research didn't intentionally lead researchers down the wrong path; they just didn't understand the real-world implications of their choices.
For any research to be scientifically reliable, every variable other than the one being tested must be controlled. But in most marketing research it's impossible to control all the variables, which means a certain amount of error is in every study. Where that error lies, and how significantly it affects the outcome, is always a mystery. That's what makes it so dangerous.
4. People lie.
"I will never have a cell phone." That's a direct quote from my wife, who also said she'd never use the Internet. Today she sent me a text message complaining that Facebook is acting funny again.
She didn't mean to lie. She's just not an early-adopter. And she can't predict how she will respond to new developments in the future. None of us can.
Harris Interactive (HPOL) recently conducted a survey about personal consumer behavior. Unlike most studies of this type, Harris split its panel between telephone and Internet respondents. The results demonstrated that online respondents tended to be more honest, whereas phone respondents were more likely to provide what they thought was the "correct" answer.
For example, more than twice as many people told telephone interviewers they were weekly churchgoers, while fewer admitted they gambled. People interviewed via phone were also more likely to say they give money to charity, exercise regularly, and brush their teeth twice a day. (What a wonderful world that would be.)
Sometimes people won't accurately describe their wants, needs, or behavior, and sometimes they simply can't. As David Lewis, chief designer at Bang & Olufsen (BO:DC), put it in a 2008 Wall Street Journal profile, "You can't go out and ask people what they need or want, because they don't know. The whole trick is to come out with a product and say, 'Have you thought of this?' and hear the consumer respond, 'Wow! No, I hadn't.' If you can do that, you're on."
5. If you blindly follow the research, you'll lose.
Swedish Vodka? Can't happen. Vodka is Russian.
Yep, research said that Absolut (PDRDY) would flop. Instead, it radically changed the spirits industry.
In a 2001 column, World magazine publisher Joel Belz called relying too much on research "the fallacy of false precision." Precision is what Ford (F) was seeking when it famously passed on launching the minivan. Hal Sperlich, who ended up taking the concept to Chrysler, recounted in a 1994 Fortune article that Ford balked because research couldn't prove there was a market for such an unprecedented vehicle. "In 10 years of developing the minivan we never once got a letter from a housewife asking us to invent one." Call it a hunch, call it intuition or insight, call it whatever—Sperlich and his team were correct, regardless of what the research said (or didn't say).
As is animated movie studio Pixar (DIS), time after time, as it churns out one hit after another. Andrew Stanton, director of Finding Nemo and WALL-E, admitted in a 2008 Wall Street Journal column, "We selfishly make movies for ourselves that happen to be juvenile enough that they cover the kids' interests.We've learned to trust our own instincts about what we like and not rely on, or trust, what the outside world tells us is going to work." Apple's (AAPL) Steve Jobs is cut from the same cloth."We do no market research," Jobs told Fortune in a 2008 interview. "We just want to make great products." I think he has proven his approach works.
The bottom line? Market research is a compass, not a map—it can give you a sense of where you are, but it can't tell you where to go. Measure to guide, don't measure to lead, and when you do talk to customers, remember you can't always go through the front door—sometimes you have to sneak in through a window to find out what they really think. Figuratively speaking, of course.
I'm a big fan of market research. But only research done right. Bang & Olufsen, BMW, and Apple are among the most innovative companies on earth, yet they refuse to be slaves to a spreadsheet. So should we all.