Tuesday, October 18, 2011

Microsoft ExP and Experimentation ROI

Why do you need experimentation? Can’t we just rely on our best judgment?

Just in case my prior blog didn't convince you, the answer is no, you can’t rely on your best judgment because our intuition is wrong 80% of the time. This Microsoft whitepaper offers the following proof:
  • QualPro, a consulting company specializing in [experimentation], tested 150,000 business improvement ideas over 22 years and reported that 75 percent of important business decisions and business improvement ideas either have no impact on performance or actually hurt performance.”
  • "Avinash Kaushik, author of Web Analytics: An Hour A Day, wrote that ‘80% of the time you/we are wrong about what a customer wants.’”
  • “In Do It Wrong Quickly, the author writes that Netflix considers 90% of what they try to be wrong.”
  • “Regis Hadaris from Quicken Loans wrote that ‘I’ve been doing this for five years, and I can only ‘guess’ the outcome of a test about 33% of the time!’”
  • “In an old classic, Scientific Advertising, the author writes that ‘[In selling goods by mail] false theories melt away like snowflakes in the sun… One quickly loses his conceit by learning how often his judgment errs – often nine times in ten.’”
  • “we can report that Microsoft is no different. Evaluating well-designed and executed experiments that were designed to improve a key metric, only about one-third were successful at improving the key metric!”

Creating A Competitive Advantage

If you’ve warmed to the idea that 80% of business decisions actually hurt performance, the next logical question is ‘How can I avoid this behavior, and maybe create an advantage?’ After all, you could improve decision-making quality 5-fold, permit reallocation of resources to more profitable endeavors, and remove socio-political factors from the process. But if my word doesn’t carry enough weight I’ll defer to the following evangelists of CAvQM:
  • “Being able to figure out quickly what works and what doesn’t can mean the difference between survival and extinction.”Hal Varian, Google Chief Economist
  • “There were three ways to get fired at Harrah’s: steal, harass women, or institute a program or policy without first running an experiment.”Gary Loveman, quoted in Hard Facts.

Perhaps the most surprising fact is that applying experimental methods produces some of the largest ROI’s I’ve ever seen in the business world.

Quantification of Benefits – Experimentation ROI

Did I mention that experimentation is also lucrative? The process is inherently self-funding because bad decisions are expensive. How much would you pay to avoid bad decisions? Fortunately, Microsoft provided a number of project ROI in their white paper so we have reliable numbers.
  • “In one case we ran an experiment for a site where the management was reluctant to run the test because they considered it a no-brainer’ that the Treatment would win. The Treatment had some unexpected and subtle negative aspects that would not have been detected had we not run the experiment. If the Treatment had been launched we estimate the annual loss to the site would have been in the millions of dollars. It only cost them $6,000 to run the experiment.”
    • Experimentation ROI greater than 33,000%.
  • Amazon Behavior Based Search: “In [another paper] we described the development of Behavior-Based Search at Amazon, a highly controversial idea. Early experiments by an intern showed the surprisingly strong value of the feature, which ultimately helped improve Amazon’s revenue by 3%, translating into hundreds of millions of dollars in incremental sales. Greg Linden at Amazon created a prototype to show personalized recommendations based on items in the shopping cart. Linden notes that ‘a marketing senior vice-president was dead set against it.’ Claiming it will distract people from checking out. Greg was ‘forbidden to work on this any further.’ Nonetheless, Greg ran a controlled experiment and the rest is history: the feature was highly beneficial. Multiple sites have copied cart recommendations.”
    • Experimentation ROI greater than 1,000,000%.
  • Microsoft tested the affect on user experience from adding 3 ads to the MSN home page, which would generate $1 million in advertising revenue annually. Unfortunately, experiments showed that the CTR (click-through rate) would decline by 0.35% if the ads were incorporated, “the estimated loss, had this feature been deployed, was millions of dollars per year."
    • Experimentation ROI greater than 50,000%.

The Mechanics of Building A Competitive Advantage Around Experimentation

Hiring: Companies that build a competitive advantage around experimentation start by hiring developers and statisticians to provide the subject matter expertise, social credit, and coding ability. Early efforts focus on proof of concept experiments to build momentum, while socializing the benefits, and creating a data driven culture while constructing an experimentation platform.

Prove The Shortcomings of Intuition: People will not abandon intuitive decision making easily. They need to be shown the benefits of experimentation in contests. Offer $1,000 to any employee who can correctly pick the winner from 6 of 10 experiments (when Microsoft tried this, no one came close).

Why invest in an experimentation platform? “You have to kiss a lot of frogs to find a prince. So how can you find your prince faster? By finding more frogs and kissing them faster and faster.”  A robust experimentation platform is needed to “increase the rate of experimentation and lower the cost to run experiments. Also remember that 80% of the time, executives mistake princes for frogs so many decisions must be tested to avoid marrying frogs.

Test often, test early. Catching a poor decision, design, or idea early saves development expenses, organizational resources, brand value and customer goodwill. "It is well known that finding an error in requirements is 10 to 100 times cheaper than changing features in a finished product."

Just to reiterate the most important point, "applying experimental methods produces some of the largest ROI’s I’ve ever seen." Heck, for a 1,000,000% ROI I'll ignore the competitive advantage and just do it for the return!

"Online Experimentation at Microsoft." By Ronny Kohavi, Thomas Crook, and Roger Longbotham. June 28th, 2009.