Why do you need experimentation? Can’t we just rely on our best judgment?
Just in case my prior blog didn't convince you, the answer is no, you can’t rely on your best judgment because our intuition is wrong 80% of the time. This Microsoft whitepaper offers the following proof:
- “QualPro, a consulting company specializing in [experimentation], tested 150,000 business improvement ideas over 22 years and reported that 75 percent of important business decisions and business improvement ideas either have no impact on performance or actually hurt performance.”
- "Avinash Kaushik, author of Web Analytics: An Hour A Day, wrote that ‘80% of the time you/we are wrong about what a customer wants.’”
- “In Do It Wrong Quickly, the author writes that Netflix considers 90% of what they try to be wrong.”
- “Regis Hadaris from Quicken Loans wrote that ‘I’ve been doing this for five years, and I can only ‘guess’ the outcome of a test about 33% of the time!’”
- “In an old classic, Scientific Advertising, the author writes that ‘[In selling goods by mail] false theories melt away like snowflakes in the sun… One quickly loses his conceit by learning how often his judgment errs – often nine times in ten.’”
- “we can report that Microsoft is no different. Evaluating well-designed and executed experiments that were designed to improve a key metric, only about one-third were successful at improving the key metric!”
Creating A Competitive Advantage
If you’ve warmed to the idea that 80% of business decisions actually hurt performance, the next logical question is ‘How can I avoid this behavior, and maybe create an advantage?’ After all, you could improve decision-making quality 5-fold, permit reallocation of resources to more profitable endeavors, and remove socio-political factors from the process. But if my word doesn’t carry enough weight I’ll defer to the following evangelists of CAvQM:
- “Being able to figure out quickly what works and what doesn’t can mean the difference between survival and extinction.” – Hal Varian, Google Chief Economist
- “There were three ways to get fired at Harrah’s: steal, harass women, or institute a program or policy without first running an experiment.” – Gary Loveman, quoted in Hard Facts.
Perhaps the most surprising fact is that applying experimental methods produces some of the largest ROI’s I’ve ever seen in the business world.
Quantification of Benefits – Experimentation ROI
Did I mention that experimentation is also lucrative? The process is inherently self-funding because bad decisions are expensive. How much would you pay to avoid bad decisions? Fortunately, Microsoft provided a number of project ROI in their white paper so we have reliable numbers.
- “In one case we ran an experiment for a site where the management was reluctant to run the test because they considered it a no-brainer’ that the Treatment would win. The Treatment had some unexpected and subtle negative aspects that would not have been detected had we not run the experiment. If the Treatment had been launched we estimate the annual loss to the site would have been in the millions of dollars. It only cost them $6,000 to run the experiment.”
- Experimentation ROI greater than 33,000%.
- Amazon Behavior Based Search: “In [another paper] we described the development of Behavior-Based Search at Amazon, a highly controversial idea. Early experiments by an intern showed the surprisingly strong value of the feature, which ultimately helped improve Amazon’s revenue by 3%, translating into hundreds of millions of dollars in incremental sales. Greg Linden at Amazon created a prototype to show personalized recommendations based on items in the shopping cart. Linden notes that ‘a marketing senior vice-president was dead set against it.’ Claiming it will distract people from checking out. Greg was ‘forbidden to work on this any further.’ Nonetheless, Greg ran a controlled experiment and the rest is history: the feature was highly beneficial. Multiple sites have copied cart recommendations.”
- Experimentation ROI greater than 1,000,000%.
- Microsoft tested the affect on user experience from adding 3 ads to the MSN home page, which would generate $1 million in advertising revenue annually. Unfortunately, experiments showed that the CTR (click-through rate) would decline by 0.35% if the ads were incorporated, “the estimated loss, had this feature been deployed, was millions of dollars per year."
- Experimentation ROI greater than 50,000%.
