Wednesday, May 9, 2012

Data-Driven Decision Making (DDDM) as an Intangible Asset

A fascinating New York Times article on Data-Driven Decision Making (DDDM) got my attention recently. Professors at MIT and Wharton concluded that companies using DDDM “achieved productivity that was 5 to 6 percent higher than could be explained by other factors” (Lohr). This prompted the question, “What caused this 6% productivity advantage?” Or perhaps more importantly, why is there so little information on data-driven decision making outside the field of education?

What is Data-Driven Decision Making?

The absence of internet content on the strategic implications of DDDM forced me to precisely define it. To the MIT professor who authored the research, DDDM is when ‘decisions [are] based mainly on ‘data and analysis’ [rather than] the traditional management arts of ‘experience and intuition.’” Or as this was phrased in my blog on experimentation platforms at Microsoft: “HiPPO vs DDDM”. It is important to emphasize though, that every company collects data and produces reports using it. The DDDM difference is made in the application of that data, when it is mined for actionable insights. DDDM “was defined not only by collecting data, but also by how it is used — or not — in making crucial decisions, like whether to create a new product or service.”
Now that we’ve defined DDDM for this discussion, it is time to revisit our original question, “What causes the 6% ROI advantage enjoyed by DDDM companies?”

Where does this ROI benefit come from?

The academic study did an admirable job to confirm that there is a causal relationship between DDDM and ROI rather than merely a correlation, so I will focus instead on micro examples that contribute to the ROI effect. The contributors I suspect are feedback loops, Experimentation ROI, and the avoidance of bad decisions.

Feedback Loops: Wired magazine published a fascinating article on feedback loops last year where they stated, feedback loops have been thoroughly researched and validated in psychology, epidemiology, military strategy, environmental studies, engineering, and economics.” More impressively though, they indicated that “feedback loops more typically improve outcomes by about 10 percent compared to traditional methods. That 10 percent figure is surprisingly persistent, it turns up in everything from home energy monitors to smoking cessation programs to speed signs.” It seems that just giving people the information they need to monitor their progress over time inspires them to improve that metric. Accordingly, we can expect companies that monitor performance over time and make that information accessible to employees would outperform companies that do not.

Experimentation ROI: As I mentioned in a prior blog, Microsoft earned unbelievable ROI (i.e. 1,000,000%) on many experimentation projects, saving millions of dollars in the process. The projects required minimal investment and technology, with only three headcount involved. Similarly, GoldCorp earned a 1,000,000% ROI on the GoldCorp Challenge, identifying billions of dollars in rich deposits through a $675,000 prize. With such ROI possible, experimentation alone could account for the productivity difference between DDDM and non-DDDM companies.

Bias Avoidance, Democratized Decision Making, Bad Decision Avoidance: Also mentioned in my Microsoft Experimentation blog is their conclusion that 80% of business decisions based on intuition are wrong. Given that experimentation, testing, and many other objective means of decision evaluation avoid this 80% of errors, I expect this ‘bad decision avoidance’ to greatly impact productivity metrics. Every bad decision avoided translates into greater employee productivity (just because an employee is busy doesn’t mean they are creating value), greater revenue (as more profitable projects are pursued), and resources are reallocated to more profitable endeavors.

Creating a DDDM Culture

I could find very little information on how to create a DDDM culture but I’ve collected a number of best practice ‘guesses’ below.

Analytical Executives: As suggested in Competing On Analytics (which I reiterated in a blog on Analytical CEOs),  analytical companies usually have a quant for a CEO. Given that most of these companies known as DDDM cultures (i.e. Google testing 43 shades of blue to decide the optimal shade for a webpage), I hypothesize that an analytical CEO helps to create a DDDM culture.

Worker Empowerment: The MIT research study cites other academic studies by Bresnahan 2002 and Galal 1998 that identified ‘decentralized decision making’  as a significant predictor of gains from DDDM. Avinash Kaushik also says that analysts must have the autonomy and freedom to research what they think is most important. He even goes so far as to suggest that analysts should have 20% of their time to research questions that no one has asked them.

DDDM as DDM (Democratic Decision Making): DDDM is also the most democratic decision making  approach because it implies that anyone can have the best idea, and the only arbiter is the data. This removes decision making as the province of the HiPPO (Highest Paid Person’s Opinion), and recasts decision making as the province of the customer. The data is the voice of the customer, whether in the form of surveys, experiments, or testing and that’s the only voice that counts. “Very few people, “HiPPO’s included, can argue with a customer’s voice.

Reporting is NOT Analysis:  Kaushik quips in another article, “If you need reporting, hire an intern.”  Analysts should be asked difficult, non-straightforward business questions so that they can learn the context of the question and answer it as thoroughly as their domain expertise allows.  “Reward analysis and not the number of emailed reports.”

Analytics Must Drive Outcomes: When creating a DDDM culture, focus your efforts on improving the outcomes that benefit your peers’ compensation because it will quickly win you the support of your colleagues. Much more important though, is that analytics should be in a department with revenue goals to which it can contribute. This is the only way to truly focus efforts on value adding activities. If analysts know that their ingenuity will affect their group’s compensation, and thereby their own compensation, they will search out opportunities like a bloodhound.

My Thoughts

This blog was a bit lengthy, so I only want to draw your attention to one sentence in the research paper: “Collectively, our results suggest that DDDM capabilities can be modeled as intangible assets which are valued by investors and which increase output and profitability.” The last time I heard of a new class of intangible assets, it was the Dot Com boom when every company was promoting their first CTO/CIO. Perhaps we’re in the midst of the big data boom and we’ll soon see a wave of Chief Data Officer promotions?


“When There’s No Such Thing As Too Much Information.” By Steve Lohr. The New York Times.  April 23rd, 2011.

“Strength In Numbers: How Does Data Driven Decisionmaking Affect Firm Performance?”  By Erik Brynjolfsson, Lorin Hitt, Heekyung Hellen Kim. April 22, 2011.

“Risks of Data Driven Decision Making.” By Arjun Moorthy. 08APR2012.

“Seven Steps To Creating a Data Driven Decision Making Culture.” By Avenash Kaushik. 23OCT2006.

“Rebel! Refuse Reporting Requests. Only Answer Business Questions, FTW. “ Avenash Kaushik.  04OCT2010.

“Harnessing The Power Of Feedback Loops.”  By Thomas Goetz. 19JUN2011. Wired. July 2011 Issue.