After BusinessWeek's cover story on click fraud used our experiences at MostChoice.com as the starting point for its investigations, I've received reactions that have run the gamut (see BusinessWeek.com, 10/2/06, "Click Fraud"). Some thanked me for risking reprisals to share our experience and challenge the giants. On the other side, some industry sages (and cynics) wondered how I could have been dumb enough to not opt out of some of the "recycled ad" traffic.
The reality is this: As an early partner with Yahoo! (YHOO) and Google (GOOG), we have worked behind the scenes for over three years on these issues with well-meaning people at both companies. Almost all the information we gave for the article had been sent to them previously, and they are not mean-spirited, so I don't expect reprisals. And trust me, from being in the business of generating high-quality, search-only leads for insurance agents, I know the ability to control search campaigns is a core competency.
The real issues here are how the ad partner networks have evolved in ways that can be gamed for profit by unscrupulous operators—or as I prefer to call them, thieves—and in ways that do not allow even the most sophisticated advertisers full freedom to avoid being gamed. I have always believed it important to speak truth to power to push needed reform, and given the opportunity I knew that BusinessWeek's powerful voice could expose the issues and accelerate needed change.
VIRUS-LIKE SPREAD. From my earliest days in business, I have always been a big believer in the power of incentives to drive social behavior. At its heart, click fraud is a prime example of how a well-conceived incentive to drive one behavior can get misapplied and lead to very different results. Let's look at how that developed here.
It was venerable Internet pioneer Amazon.com (AMZN) that plowed the fields for click fraud. In its earliest years, Amazon offered a commission of 10% on sales made from customers sent from other sites. It worked: Much of Amazon's early exponential growth came from the virus-like spread of thousands of affiliate sites. But customers had to actually buy things for affiliates to get paid.
Paid-search pioneer Goto.com saw this success in 1999 as they tried building their new system of syndicated paid-text search ads on their own site. So in 2000, Goto.com began enlisting other search engines and portals, paying 50% to 70% of the price bid per click on each search term, to their partners for displaying the ads.
THE BIRTH OF FRAUD. This worked fine when real portals such as Yahoo, MSN, and Excite were the main early recruits. The returns were great for advertisers, especially as early pricing was relatively low. Good results and VC-inflated advertising budgets quickly led to higher bids, which led to fast growth for Goto.
As Goto went public and changed its name to Overture, it kept adding hundreds and then thousands of much smaller search engines. Most of these partners had no business aside from passing off paid Overture listings as their own organic (free search) results.
It didn't take long for sharp operators to realize that early Overture "algorithms" to detect and filter multiple clicks were rudimentary at best. The more they clicked, the more they were paid, so the unscrupulous quickly found ways to boost traffic by using server programs and redirecting international sources.
So began Overture's cat-and-mouse game to detect spikes of bad clicks—similar to the campaign Google was then engaged in with search optimizers using bad techniques to get to the top of their free listings. Some junky non-U.S affiliates started cloning their sites onto hundreds or thousands of different domain names to spread clicks more widely and stay below the radar.
REMAINING WATCHFUL. Overture heard complaints from vigilant advertisers such as MostChoice and cut back a few of the most obviously nutty sites but stayed strong on one point: As an advertiser your ads appeared throughout the network; there was no picking and choosing. To do so would open a Pandora's box and imperil growth.
By contrast, when Google started its Adwords paid text ad program in early 2002, it not only employed a whole different bidding system incorporating ad click-thru rates (which inherently cut down incentives for competitors to click on each other) but was more careful in setting up its search network in two ways.
First, Google more selectively picked which search partners it allowed into its network—grabbing big portals like AOL (TWX), Earthlink, and AskJeeves. It didn't allow many search-only sites and soon kicked out bad ones, though a few problem places did flare up.
For example, in the Zapmeta case discussed in the BusinessWeek article, Google took fast action once I pushed them past the standard canned e-mail denial. It investigated, found a sophisticated new false-click algorithm, gave a refund, and thanked us for pointing it out.
THE LITTLE CLICKS. Second, given the high search volume Google was generating itself, it let advertisers simply opt out of showing their ads on the entire search partner network. While this is a blunt tool that we find seldom makes sense, it gives the advertisers the freedom to choose. Yahoo certainly could have tried this a year after acquiring Overture in mid-2003, but since much of its traffic was coming from non-Yahoo sites, it was loathe to do so.
When MSN broke away this last year and search affiliate traffic became more noticeable, Yahoo still didn't offer the option. Tellingly, MSN has not allowed other sites into its search program and commands high costs per clicks (CPCs), while smaller search syndicators with broad affiliate networks have CPC's 2% to 10% as high. We don't waste our time with them anyway.
Content ads—where an engine picks text ads to appear next to an affiliated site's content—came later, in 2003 or so, and allowed the spread of listings to even more and smaller affiliates. This is where most of the bad schemes are happening, including most of the domain parking and pay-to-read rings BusinessWeek explored.
SUDDEN FLARE-UPS. Knowing the content clicks would be less serious than search clicks at best, both Yahoo and Google kept content ad bidding separate and gave advertisers the option of playing this game. At MostChoice, after some initial testing we decided not to. But as with all paid search, it can work if you understand your metrics and watch results.
If click fraud were merely a consistent 10% to 15% tax on all ad dollars spent, it would just drive down costs per click uniformly, and the thieves would simply be siphoning off money that should be going to the sites sending good clicks. But the unpredictability of where click fraud will flare up across billions of combinations of search terms, domains, and time periods creates uncertainty that drives CPCs even lower and prompts some less sophisticated advertisers to not want to play the game at all.
The "dirty little secret" of a few years ago has been hitting the mainstream, and perhaps not surprisingly, given all the recent press, it seems that Yahoo and Google are getting better at responding to advertisers and taking action against some bad players. But the damage to the system still isn't being fixed nearly fast enough.
THREE-TIER SYSTEM. I told Yahoo years ago that while they were making more money in the short run, the fraudulent affiliates in their network would end up costing them in the long term if they eroded the integrity of the network and watered down CPCs. I think my prediction has been borne out with Yahoo's recent earnings warning for the third quarter. Yahoo blamed softness in the auto and financial search markets, two huge areas with expensive clicks particularly prone to fraud, as well as big advertisers who have become much more responsive.
So, what's the best path forward? Yahoo and Google should quickly change the incentives and allow advertisers more control over where their ads appear. They can accomplish this with better market pricing mechanisms.
For example, why not segment Yahoo's search network into three tiers? The top one would contain Yahoo and a few big full-line portals such as CNN, the second would be for smaller portals and well-established sites where search is only a small portion of revenues, and the third for search-only affiliate sites. Advertisers will quickly bid up the top tier and bid down or stay off the bottom tier, and sites will be rewarded for click quality much more accurately.
Advertisers like us want only for the system to be more honest and fairer for everyone. That is not too hard to accomplish if the will to make changes is there. Whether Yahoo will come around willingly or as a response to stronger counter-pressures, such as advertisers' new Click Quality Council, more lawsuits, or government intervention, is the next chapter in this drama.