When executives of SportsLine USA Inc. sat down in March to review their marketing deal with America Online Inc., they didn't anticipate spending an hour poring over conflicting data on how many people visit their Web site. After an exasperating session, both sides decided to break off that part of the review, crunch the numbers from different rating companies, and come back later. "We weren't getting anywhere--we weren't even looking at the same measurements," says Kenneth Dotson, SportsLine's vice-president for marketing.
Such knock-down fights over measuring Internet traffic are all too common. While the Web has been trumpeted as a digital marketplace where advertisers could target precise demographic groups and gather reams of data on buying habits, the reality is altogether different. Today's technology just isn't up to snuff, and advertisers still can't aim their Web ads any more accurately than they can glossy magazine ads. Worse, even basic measurement techniques for sizing up the most popular Web sites are so varied that a list of the top 25 sites is instantly disputed.
DEVILISHLY DIFFICULT. The problem this creates is more than just data frustration. Reliable measurement methods are essential for the Net to become the advertising bonanza Web-site operators are banking on. By 2002, online ad revenues are expected to hit $9 billion--a ninefold jump from the $1 billion spent in 1997, says Jupiter Communications. Advertisers say they have been holding back, in part, because of differing tallies. "It affects ad spending when you see wide discrepancies in numbers," says Norman Lehoullier, managing director at Grey Interactive Worldwide, an ad agency with 71 clients, including Dell Computer, and Procter & Gamble.
Web sites measure their own popularity, largely by the number of "hits," or the times a page or parts of a page are called up. Sites then try to convert that into "unique visitors," so that one person calling up several pages is not counted more than once. That's an inexact science, and advertisers want precise--and impartial--data.
That's devilishly difficult. Figuring out how to zero in on specific groups of buyers is improving, though the Holy Grail of laser targeting by age, income, and past buying habits is generally still out of reach.
Even the easier target of measuring the number of people visiting a Web site is a tough slog. For the past couple of years, Media Metrix Inc., founded by market research firm NPD Group Inc., was the only game in town for tracking the popularity of sites. Now, the company is being challenged by a band of rating-service wannabes. Last fall, upstart RelevantKnowledge Inc. began nipping at the leader's heels. On Mar. 30, NetRatings Inc., a startup backed by Hitachi Ltd., launched a rival service. And this summer, PC Data Inc., which tracks hardware and software sales, and Nielsen Media Research, of TV research fame, are joining the fray.
The snag: Each uses different methods for monitoring usage, which produces conflicting results. Consider February's tallies: The lists of the top 25 Web sites put together by RelevantKnowledge and Media Metrix shared only 19 names. And while both ranked the search engine Yahoo! in the top three, they varied on the other two top slots. "I think no data is better than wrong data, especially when the industry is saying we can be an effective medium," says Marshall Cohen, president of Marshall Cohen Associates, an entertainment and Internet consultant.
The differences start with the survey groups. Each service has a group of people, or "panel," who agree to install software on their PCs to monitor their movement online. Media Metrix recruits its panel by buying population lists and conducting random mailings and phone calls. Some 45,000 people have agreed to take part in its surveys, but only 30,000 actually do. Of those, 9,400 were Web users in February. Media Metrix executives say that by enlisting PC users who are not all Netizens yet, they can track people as they start to come online.
NO-NOs. RelevantKnowledge, in contrast, relies on random phone calls to recruit people who already use the Web. While 11,000 have agreed to be part of RelevantKnowledge's survey group, only 4,000 usually participate.
The size of the sample can be crucial. While the Nielsen TV panel of 5,000 people is plenty for the 52 cable and network channels it monitors, small sample sizes for monitoring the Web make it difficult to track usage beyond the top 200 to 300 sites. Ad agencies and Web-site operators say that, ideally, they would like to see panel sizes of at least 10,000 people.
The latest company to join the measurement game, NetRatings, already is drawing criticism. Its sample group of 2,000 pales next to that of Media Metrix, although NetRatings expects within a year to have 25,000 participants. And some potential clients fret that its group is recruited off the Web--which can tilt results toward heavy users, rather than a cross-section. Self-selection for survey groups also is considered a no-no, because it means the process of selection isn't random. "Those of us who are conservative researchers wouldn't call something that's self-selected a sample," says Jim Alexander, vice-president of consumer and strategic research at the Weather Channel. NetRatings defends its practice of recruiting right off the Web, saying a savvy panel gives more useful results.
The varied sample groups are only one of the ingredients that contribute to conflicting data. Another issue is that none of the services adequately factors in people who use the Net at work. Tracking firms miss up to one-third of Web usage if they focus only on the home, says IntelliQuest Inc., a technology market-research firm. "The tough thing is that so much of what we do is business-to-business advertising, and none of the services do a good job giving information on that," complains Alan May, media director at Anderson & Lembke Inc., the agency that handles Microsoft Corp.'s Web ads. In March, RelevantKnowledge included about 700 Web users at work, while Media Metrix included 850. All three plan to expand work coverage, but admit difficulties.
The problem lies in convincing companies that it's O.K. for employees to install tracking software on their computers. This software is more sophisticated than cookies, the tiny software programs commonly used by individual sites to monitor usage. The fear is that the software can potentially trace identities, as well as confidential data, such as memos or sales information. "If people are concerned about cookies, imagine how they would feel about this technology," says Manish Bhatia, vice-president for interactive services at Nielsen, which won't audit work usage initially, although it's developing technical solutions aimed at making the software less intrusive.
International Web users also get short shrift. RelevantKnowledge began working with a Swedish partner in March, and the other services have plans in the next year to attack the problem. "That's important to us," says Sean Pfister, director of research and analysis at CNET Inc., whose technology news and search sites get between 8% and 15% of their traffic from international users.
The new ratings services may be muddying the waters for now, but the added competition is expected to lead to more reliable measurements. Media Metrix already has been forced to slice data in ways comparable to rivals. By June, the company plans to offer home and work data combined. That's something RelevantKnowledge already provides. NetRatings and RelevantKnowledge also are beefing up their ability to track America Online Inc.'s subscribers, who account for 40% of all Web usage from home. Today Media Metrix tracks all of AOL's subscribers, while its rivals have limited access. The reason: AOL executives say they are hesitant to give carte blanche to new tracking companies without details on their methodologies.
EARLY BETS. Other forces also are at work to standardize ratings. This month, the Advertising Research Foundation formed a committee of companies that use the ratings services. The objective: to look into how the services operate. "At the minimum, the idea is to understand what the companies are doing. It's unclear now," says James Spaeth, president of the foundation, a nonprofit group that encourages marketing and ad research. Media Metrix also has met with the Media Rating Council, which accredits ratings services that disclose their procedures and abide by them.
So, which method will win out? In the next 12 to 18 months, a front-runner is bound to emerge. For one, the sheer cost of using the services, which can range from $40,000 to hundreds of thousands of dollars annually, will force Web-site operators to be selective. "Our tactic this year is to buy all the data that looks valuable, get really knowledgeable about it, and then next year drop what we don't think is worth paying for," says Karen Edwards, Yahoo! Inc.'s vice-president for brand marketing.
Early bets are on Media Metrix, because it leads the pack, with 150 clients, and has the biggest survey group. And although Nielsen is late to the party, its brand name and resources have caught advertisers' attention. "Media Metrix is the front-runner, but nobody has got it right yet," says analyst Chris Charron of Forrester Research Inc. That means there will be more data angst before the Web measures up.