They're called spiders. Scores of these digital robots crawl through cyberspace, scouring the billions of pages in the World Wide Web. They catalog all the words in the pages, how many times they're viewed, and which other pages they link to. These spiders power the fast-growing search industry, led by Google, Yahoo! (YHOO ) and Microsoft's MSN. This is quickly becoming humanity's favorite tool for finding out just about anything, with 550 million Internet searches performed worldwide each day. Type in a couple words, say "sleeping" and "bag," and the engine, fueled by its spiders and scientific algorithms, promptly provide a list of Web sites, starting with the most relevant. Right?
Not so fast. In the last year, a host of search engines, including MSN and Lycos (TRLY ), have sprinkled growing numbers of paid corporate Web pages into the search results and accepting money each time one of these so-called paid-inclusion links is clicked. This practice is a booming business, one expected to double this year, to $200 million, and to reach $600 million by 2007, says U.S. Bancorp Piper Jaffray (USB ). But unlike the clearly marked advertised links that show up next to search results, these paid inclusions are virtually invisible to average Web surfers. Only specialists prepared to decipher Web addresses can begin to distinguish between the paid and unpaid pages. "These are not objective oracles of information," says Beau Brendler, director of Consumer WebWatch, a project of Consumers Union.
Controversy over paid inclusion is driving a wedge through the search industry. At the heart of the issue is whether the industry's content will be swayed by advertisers. Google, the leading search engine, steers clear of paid inclusions, saying that they undermine confidence in the objectivity of search results. Instead, Google relies entirely upon its technology for the search, and features paid ads only in clearly marked boxes near the search results. AOL uses Google technology and follows the same line. But others, including Yahoo, say that paid inclusion can provide users with better information. And they maintain that their search results are still displayed in order of relevance. That means that paid ads get no preferential treatment. "All listings are run through an algorithm that is blind to their source," says Tony Mamone, sales vice-president at LookSmart (LOOK ) Ltd., which feeds paid-inclusion results to portals like MSN.
TICKET TO THE TOP
Yet an investigation by BusinessWeek, based on more than 30 interviews and analysis of dozens of Internet searches, suggests that paying customers frequently fare better. Indeed, out of 20 advertisers and online marketing pros interviewed by BusinessWeek, 10 had experienced firsthand a boost in search-engine rankings when they signed up for paid inclusion. Says James Taylor, CEO of AICompany.com, a search-marketing company, "Paid inclusion dilutes the accuracy and relevance of a search engine."
Dennis Swanson learned the hard way. Earlier this year, the chief executive of Lamps Plus Inc., couldn't understand why his e-commerce offerings for light fixtures turned up so prominently in Google but were nearly out of sight on MSN. So he paid to have his site included with LookSmart, which feeds search results to MSN and shares with MSN revenues it gets from advertisers. Almost instantly, Lamps Plus pages soared to near the top of MSN's search results, and Microsoft's portal became Swanson's biggest traffic driver. When he temporarily stopped his LookSmart deal, MSN's traffic spigot shut off. "If we don't have a deal with LookSmart, we don't show up in MSN's results," he says. MSN, while declining to comment on this case, says it is currently reevaluating its entire Internet search offering, including paid-inclusion listings, to put forth the most relevant results.
HITS AND MISSES
What gives? For starters, the paid-inclusion pages are easily engineered for high placement. When submitting paid-inclusion data, companies typically fill out a spreadsheet with information on product details, along with the search words and phrases for which they'd like to appear. The result is a rich stream of data targeted precisely to what the search engine will deem relevant. Consequently, their ratings are primed to soar.
Anecdotal examples aren't tough to turn up. Plug in "green sleeping bag" to HotBot.com and the first six results, in a recent search, featured paid-inclusion listings from Inktomi, which is now owned by Yahoo. Some weren't even for sleeping bags. The fifth listing took users to a paid-inclusion product page for a white canvas bag. Meanwhile, numerous pure results displayed lower were highly relevant product pages for green sleeping bags. A Yahoo spokesperson responds that there are just as many examples of paid-inclusion listings improving search results.
Despite criticisms, the paid-inclusion business addresses some very real needs. One biggie is the sheer number of Web pages tucked away in corporate databases, where search-engine spiders have a difficult time navigating. By letting, say, Staples.com pipe its data directly to a search engine, a user seeking "inkjet cartridges" could get a fresh listing from Staples.com that might not otherwise show up. "We use [paid inclusion] only if it's improving the relevance of our results," says Tim Cadogan, Yahoo's vice-president for search.
Still, the growth of paid inclusion could tilt the search business toward deep-pocketed companies. For instance, travel sites that want to be in LookSmart's paid-inclusion program have to pay the company a flat rate of 30 cents for each click, according to the company's Web site. Jumbo chains like Marriott International (MAR ) Inc. or Hilton Hotels (HLT ) Corp. are certainly better equipped to cover such costs than, say, a bed-and-breakfast.
Just ask Rob Spooner. The 56-year-old runs a travel site called Online Highways with informational pages about various small towns throughout the U.S. Inktomi, a search engine that now belongs to Yahoo, contacted him late last year about becoming a paid-inclusion participant. The proposal: Spooner would pay 10 cents for every visitor Inktomi passed along. Spooner did the math, figured he would lose money on anything more than 3 cents a click, and declined the offer.
Then things went downhill. Spooner's Web pages soon plunged in Inktomi's search rankings and disappeared from key sites like MSN, where Inktomi feeds its listings. After he demanded to know what happened, Spooner learned from Inktomi that his site contained editorial flaws that hurt his ranking. And he would have to become a paid-inclusion customer to learn what these flaws were. All this, while his pages remained well ranked on Google. "I lost a quarter of my traffic," says Spooner.
Yahoo declined to comment on this incident. However, it and other search engines offering paid inclusion say it serves the customers' interest. Says Tom Wilde, general manager of search services at Terra Lycos: "If you deliver a poor user experience, people leave, and they leave rapidly."
And why not label the sponsored links? Many search companies and advertisers fret that marking them would scare the public away from relevant sites. Says Glenn McLaren, e-commerce manager at Friedrich Air Conditioning: "Part of what makes it attractive is that it doesn't look like advertising." The trouble is, if paid inclusions are hidden, readers could grow to dismiss Internet search as nothing but a pile of ads.
By Ben Elgin in San Mateo, Calif.