Every once in a while in most Web surfers' lives, a suggestion pops up on the screen that leads them to wonder: How did they know that about me? The moment can seem magical, and a bit creepy.
Consider this one. A shopper at the retail site FigLeaves.com takes a close look at a silky pair of women's slippers. Next a recommendation appears for a man's bathrobe. This could seem terribly wrong—unless, of course, it turns out to be precisely what she wanted. This type of surprising connection will happen more often as e-marketers adopt a new generation of predictive technology. It's fueled by growing rivers of behavioral data, from mouse clicks to search queries—all crunched by ever more powerful computers.
Why the bathrobe? ATG (ARTG), a Cambridge (Mass.) e-commerce software company that crunches data for FigLeaves, has found that certain types of female shoppers at certain times of the week are likely to be shopping for men. Like all Web recommendations, this one will be wrong a good portion of the time. But as marketers scrutinize shoppers in greater detail, they're edging closer to their ultimate goal: teaching computers to blend data smarts with something close to the savvy of a flesh-and-blood sales clerk. "In the first five minutes in a store, the sales guy is observing a customer's body language and tone of voice," says Mark A. Nagaitis, CEO of 7 Billion People, an Austin (Tex.) startup that competes with ATG. "We have to teach machines to pick up those same insights from movements online."
This dissection of online shopping comes amid growing fears about invasions of privacy online. But unlike the most controversial advertising technology, which tracks Web surfers' wanderings from site to site, many of these "preference prediction" methods limit their scrutiny to behavior on a retailer's own Web page. Much of the analysis looks simply at the patterns of clicks, purchases, and other variables, without including personal information about the shopper. In most cases, personal details are incorporated only if customers register on sites such as Amazon.com (AMZN) and Walmart.com (WMT) and supply them.
In the early days of e-commerce, most of the analysis focused on simple buying patterns among shoppers. Amazon and others introduced so-called collaborative filtering in the late '90s. They found, to no one's surprise, that people who bought the same book were likely to share interests in other books.
Now the science is growing far more sophisticated. Three years ago, Netflix (NFLX), the movie rental powerhouse, dangled a $1 million prize before anyone who could plow through data from millions of anonymous users and improve Netflix predictions of what movies customers would like by 10%. Last month an international team of computer scientists reached that goal by introducing deeper analysis. The winning team factored loads of details into its algorithm. It attempts, for example, to compensate for the shifting sentiments of a movie watcher over time. If one reviewer pans a number of movies in a row, are they all really so terrible? The algorithm might allow for a stretch of the blues—and take those ratings with a mathematical grain of salt.
Introducing this level of complexity into a calculation requires computing muscle that until recently was beyond the range of most companies. "What we do today would be impossible on the machines we had in 1999," says Bruce D'Ambrosio, vice-president and chief architect at ATG. He says his company is crunching ever greater amounts of data as it attempts to map out the most likely path for each shopper on a site.
ATG quantifies the crazy quilt of relationships between every item in its customers' stores, whether Tommy Hilfiger or Body Shop International. It analyzes which type of customer buys each product, and what else they buy or look at. This adds up to hundreds of billions of relationships. But it's through such analysis that ATG finds connections such as the one between silky slippers and men's bathrobes. ATG also studies the shifts in Web surfers' behavior over time. Shoppers, D'Ambrosio says, tend to be in a big hurry when dropping in from work and have more leisure time on weekends. Ideally, the site adjusts to their rhythms, leading shoppers on a leisurely stroll on Saturday afternoon, and sending them hurtling toward checkout on Monday morning.
The algorithms used by 7 Billion People attempt to mimic the human feedback loop in a brick-and-mortar store. While a sales clerk can see a shopper is in a hurry, the Web site must pick up that insight from other signals, such as rapid-fire clicks of the mouse. The trick then, says CEO Nagaitis, is to fit the site to the customers. Those who dally over reviews and related products find themselves transported to Web pages with more features to explore. Shoppers who are more likely to be swayed by demonstrations, for instance, may find videos to click.
Such adjustments can prove very valuable, say customers. Doug Scott, who heads Web strategy at ASAP Ventures, a British e-commerce incubator, says he used to optimize sites he controlled to his own tastes, with lots of details and choices. But after running tests with 7 Billion People, Scott learned that only about one-third of its users shared his tastes. Others wanted to read testimonials or simply to hurry up. "We could fine-tune for one-third, but then we would upset the other two-thirds," he says. Adjusting for the different types, based on their behavior, has lifted the conversion of Web site visitors to buyers by 30% to 50%.
Another competitor of ATG, San Francisco's richrelevance, is dusting off theoretical algorithms from before the computer age to see if they can be used to predict customer behavior. "If someone's looking at a Dell (DELL) computer at Wal-Mart (WMT)," says CEO David Selinger, "is he more likely to be interested in a more expensive PC, a cheaper one, or a warranty?" Well, the answer depends on the individual, the time of day, and a few hundred other variables.
Business Exchange: Read, save, and add content on BW's new Web 2.0 topic network
The Netflix Prize
Three years ago, Netflix offered $1 million for an algorithm to improve movie recommendations by 10%. This unleashed a global race among mathematicians and computer scientists. The Netflix Prize site links to the various teams and explains their different approaches to predicting consumer tastes and opinions.
For more stories on predictive tech, go to bx.businessweek.com/data-mining/reference