Netflix Contest: Another Crowd Pleaser

In late 2006, Netflix (NFLX) essentially ran out of ideas. The company's movie-recommendation software was already renowned for suggesting rentals based on a customer's previous picks and those of others with similar tastes. But occasionally the predictive algorithms simply blew it, directing customers to films they ended up not liking. After several tries to improve the program internally fell short, Netflix went to the rest of the world for help.

The DVD-rental company offered $1 million as well as continued ownership of intellectual property to anyone who could boost the accuracy of its predictions by 10%.

Now, after evaluating 41,691 submissions from 4,670 contestants from 184 countries, Netflix is on the verge of naming a winner. On June 26, the company said a team known as BellKor's Pragmatic Chaos had written code that finally met the goal, with a program that was 10.05% more accurate. Contest rules give other contestants 30 days to top that achievement. If none can, the BellKor team will claim the Netflix Prize and split the $1 million.

Team BellKor, anchored by researchers at AT&T Labs (T), had been the front-runner; it won the progress prize from Netflix last December.

Netflix executives believe the $1 million will be money well spent. "We could not have hired a fraction of the people who worked on the Netflix prize," says Steve Swasey, a company spokesman. "And we're getting the best computer scientists in the world."

Racing for Cures and Solutions

Big-money contests have become popular since the X Prize Foundation promised in 2001 to pay $10 million to the first privately financed crew to fly into space. Since then, private companies, governments, and nonprofits have begun dozens of competitions for everything from curing disease to solving age-old math problems. There's also a company, InnoCentive, that offers cash to anyone who can solve scientific challenges it posts on its Web site on behalf of sponsors.

These winner-take-all races seem to fit in well with another 21st-century trend—crowdsourcing, or seeking help from others (often for no compensation) via the Internet. Indeed, many of the teams in the Netflix competition, including BellKor's Pragmatic Chaos, depended on such far-flung collaboration.

Pragmatic Chaos has seven members: Chris Volinsky and Bob Bell from AT&T's statistics research group in New Jersey; machine learning experts Andreas Toscher and Michael Jahrer from Commendo Research & Consulting in Austria; software engineers Martin Piotte and Martin Chabbert from Pragmatic Theory in Montreal; and Yehuda Koren from Yahoo! Research (YHOO) in Israel. Volinsky says he's never met his foreign teammates.

"One of the interesting themes about our success," notes Volinsky, "is that we worked independently and used the predictive power of different models combined at the end to meet success."

Just for the Recognition?

Despite their spread, prize competitions are sometimes dismissed as public relations stunts. The goals may be laudable, but unachievable. In those cases, sponsors might never pay out the big purse they've promised, but nevertheless can take credit for doing something beneficial for others. The Netflix contest had looked like it might fall into this category.

Netflix developed its own proprietary software known as Cinematch in 2000. The Los Gatos (Calif.)-based company collected ratings from customers so that when others wanted recommendations, it could steer them to DVDs they'd like. That, in turn, might get customers to rent more movies or TV programs and boost revenue. More than half of Netflix's 500 salaried employees worked on the software.

But by 2006, these software engineers were no longer able to improve the forecasting tool. So CEO Reed Hastings decided to enlist the best and brightest outside the company. To help contestants write better forecasting programs, Netflix supplied about 100 million ratings from 500,000 anonymous customers who rated 18,000 movies.

"There are movies you love and movies you hate," says Volinsky, who described himself as an indie film buff. "When we build models we can see if they make sense, and it helps to evaluate a subject area that everybody understands and relates to."

"More Than Their Money's Worth"

Contest participants can win even if they don't walk away with the top prize. The competition rules allow Netflix to license sofware, but the creators keep the rights. AT&T, for instance, is already using code developed by Team BellKor to recommend programs to viewers of its digital television service, U-verse.

"Netflix took a lot of risk releasing proprietary data that could have had privacy concerns, but they got well more than their money's worth in publicity," says Volinsky. "I think it was genius."

Netflix plans some "pomp and circumstance," in spokesman Swasey's words, when it declares the winner of the competition later this summer. The event could provide an opportunity for the members of BellKor's Pragmatic Chaos to finally meet in person.

Before it's here, it's on the Bloomberg Terminal.