Posted on Harvard Business Review: December 27, 2010 10:00 AM
While he was at Amazon from 1997-2002, Greg Linden prototyped a system for making personal recommendations for other products to customers as they checked out.
Problem was, a marketing Senior Vice President thought these would distract customers from completing their purchase and didn't think Linden had proof it would work. A business drama ensued; classic "gut versus data" stuff. "I heard the SVP was angry when he discovered I was pushing out a test," recalled Linden in his blog. "But, even for top executives, it was hard to block a test. Measurement is good. The only good argument against testing would be that the negative impact might be so severe that Amazon couldn't afford it. A difficult claim to make."
Linden's experiment ultimately showed that the customers really liked the feature, and recommendations have become a signature design feature for Amazon.
These days, the aspiring Greg Lindens of the world might expect a bit less resistance from business leaders when it comes to experimentation, but they may be surprised that it's only now, in 2010, that a majority report experimentation is the preferred approach. In our recent sample of 149 senior executives in the Babson Executive Education survey of global respondents, December 2010, 51% say that experimentation is now their organization's preferred approach to understanding and acting on potential opportunity. Interestingly, for the subset of executives leading the highest growth firms (more than 20% revenue growth last year), the percentage encouraging experimentation is 10 % higher than that (61%). The practice of experimentation is widespread, and seems to be linked to higher revenue performance as well, our data suggest.
Would-be experimenters have another advantage nowadays too: it's becoming easier. Experimentation tools are coming out of R&D labs into the mainstream, and can be made available for anyone's workspace or department. Below we describe three of the most accessible types of experimentation we have noted in our research:
Prototyping involves the process of trial-and-error, where numerous iterations of an idea are created and tested to see whether it fits the environment and provides the desired results. Prototyping used to be complicated, but now can be done inexpensively by using rapid prototyping methods (RP) to quickly convert computer graphics (CG) designs into 3-D prototypes, for example, or employing low-cost tools like Google's Sketchup.
As a result, hands-on modeling has been pushed up earlier in the product development cycle. For example, "Sony Ericsson has a policy where they don't want their designers to work more than a few hours before they build their very first physical model," said one respondent. The basic steps for prototyping are:
1. create a concept/prototype
2. document it
3. experiment with it, generate and analyze results
4. refine the concept or the experiment, if needed, and
5. repeat until satisfactory outcomes are reached.
Simulation of a product or service is another form of experimentation that has come a long way in the past few decades, resulting in more efficient team decision making and better productivity. For instance, early crash simulation models at BMW included about 3,000 finite elements, ran for nearly three months and were unable to have any significant impact on design decisions. More recently, though, the crash simulation model used in the development of the BMW X5 used roughly 700,000 finite elements, ran less than 30 hours, and drove important design decisions.
One of the advantages of simulation is that it allows organizations to engage diverse stakeholders into acts of collaborative problem solving — sometimes called "decision theaters" — to analyze of alternative actions or the unintended consequences from decisions using virtual reality environments. Local utility agencies, citizens, and businessesin Arizona have developed a decision theater to experiment with long-term sustainable water strategies, using computer simulations to understand the consequences of various population and urban development scenarios. Previously, stakeholders in that desert state had to rely on educated guesswork using reports and charts.
Test Groups and A/B Testing. Think of the problem of redesigning a web page. How would you know which of several competing designs is best? One way is to let a test group try out the new page, measure the gains from the visits, and compare them against the control group. Amazon often tests scenarios against outcomes in different "live" settings. For example, a portion of customers could see a website where coupons for purchases were presented on the first page. Another portion of customers could receive their coupons via email and be presented with click-through capabilities to link directly with the products that interest them. Amazon then determines which set of customers responded more favorably to making purchases, and tracked which customers visited the site and the duration of their stay.
Not all organizations can experiment with the ease of Amazon, but more and more employees can now conduct experiments effectively, and with their boss' approval. As a basic guideline, you should try different modes of experimentation: switch between prototyping, simulation, and group tests.
Have you gone through your own an 'experimental phase' at work? Which approaches worked best for you?