A fascinating New York Times article on Data-Driven Decision Making (DDDM) got my attention recently.
Professors at MIT and Wharton concluded that companies using DDDM
“achieved productivity that was 5 to 6
percent higher than could be explained by other factors” (Lohr). This
prompted the question, “What caused this 6% productivity advantage?” Or perhaps
more importantly, why is there so little information on data-driven decision
making outside the field of education?
What is Data-Driven Decision Making?
The absence of internet content
on the strategic implications of DDDM forced me to precisely define it. To the
MIT professor who authored the research, DDDM is when ‘decisions [are] based mainly on ‘data and analysis’ [rather than] the
traditional management arts of ‘experience and intuition.’” Or as this was
phrased in my blog on experimentation platforms at Microsoft: “HiPPO vs DDDM”. It is important to
emphasize though, that every company collects data and produces reports using
it. The DDDM difference is made in the application of that data, when it is
mined for actionable insights. DDDM “was
defined not only by collecting data, but also by how it is used — or not — in
making crucial decisions, like whether to create a new product or service.”
Now that we’ve defined DDDM
for this discussion, it is time to revisit our original question, “What causes
the 6% ROI advantage enjoyed by DDDM companies?”
Where does this ROI benefit come from?
The academic study did an admirable job
to confirm that there is a causal relationship between DDDM and ROI rather than
merely a correlation, so I will focus instead on micro examples that contribute
to the ROI effect. The contributors I suspect are feedback loops,
Experimentation ROI, and the avoidance of bad decisions.
Feedback
Loops: Wired magazine
published a fascinating article on feedback loops last year where they
stated, “feedback loops have been thoroughly researched and validated
in psychology, epidemiology, military strategy, environmental studies,
engineering, and economics.” More impressively though, they
indicated that “feedback loops more
typically improve outcomes by about 10 percent compared to traditional methods.
That 10 percent figure is surprisingly persistent, it turns up in everything
from home energy monitors to smoking cessation programs to speed signs.” It
seems that just giving people the information they need to monitor their
progress over time inspires them to improve that metric. Accordingly, we can
expect companies that monitor performance over time and make that information
accessible to employees would outperform companies that do not.
Experimentation
ROI: As I mentioned in a prior blog, Microsoft earned unbelievable ROI (i.e. 1,000,000%) on many
experimentation projects, saving millions of dollars in the process. The
projects required minimal investment and technology, with only three headcount
involved. Similarly, GoldCorp earned a 1,000,000% ROI on the GoldCorp
Challenge, identifying billions of dollars in rich deposits through a $675,000
prize. With such ROI possible, experimentation alone could account for the
productivity difference between DDDM and non-DDDM companies.
Bias
Avoidance, Democratized Decision Making, Bad Decision Avoidance: Also mentioned in my Microsoft
Experimentation blog is their conclusion that 80% of business decisions based
on intuition are wrong. Given that experimentation, testing, and many other
objective means of decision evaluation avoid this 80% of errors, I expect this
‘bad decision avoidance’ to greatly impact productivity metrics. Every bad
decision avoided translates into greater employee productivity (just because an
employee is busy doesn’t mean they are creating value), greater revenue (as
more profitable projects are pursued), and resources are reallocated to more
profitable endeavors.
Creating a DDDM Culture
I could find very
little information on how to create a DDDM culture but
I’ve collected a number of best practice ‘guesses’ below.
Analytical
Executives: As suggested in Competing On Analytics (which I
reiterated in a blog on Analytical CEOs), analytical companies usually have a quant for
a CEO. Given that most of these companies known as DDDM cultures (i.e. Google
testing 43 shades of blue to decide the optimal shade for a webpage), I
hypothesize that an analytical CEO helps to create a DDDM culture.
Worker Empowerment: The MIT research study cites other academic studies by Bresnahan 2002 and Galal 1998 that identified ‘decentralized decision making’ as a significant predictor of gains from DDDM. Avinash Kaushik also says that analysts must have the autonomy and freedom to research what they think is most important. He even goes so far as to suggest that analysts should have 20% of their time to research questions that no one has asked them.
DDDM as DDM (Democratic Decision Making):
DDDM is also the most democratic decision making approach
because it implies that anyone can have the best idea, and the only arbiter is
the data. This removes decision making as the province of the HiPPO (Highest
Paid Person’s Opinion), and recasts decision making as the province of the
customer. The data is the voice of the customer, whether in the form of surveys,
experiments, or testing and that’s the only voice that counts. “Very few people, “HiPPO’s included, can
argue with a customer’s voice.”
Reporting
is NOT Analysis: Kaushik quips in another article, “If
you need reporting, hire an intern.” Analysts
should be asked difficult, non-straightforward business questions so that they
can learn the context of the question and answer it as thoroughly as their
domain expertise allows. “Reward analysis and not the number of
emailed reports.”
Analytics Must Drive Outcomes: When creating a DDDM culture, focus your
efforts on improving the outcomes that benefit your peers’ compensation
because it will quickly win you the support of your colleagues. Much more
important though, is that analytics should be in a department with revenue
goals to which it can contribute. This is the only way to truly focus
efforts on value adding activities. If analysts know that their ingenuity will
affect their group’s compensation, and thereby their own compensation, they
will search out opportunities like a bloodhound.
My Thoughts
This blog
was a bit lengthy, so I only want to draw your attention to one sentence in the
research paper: “Collectively, our results suggest that
DDDM capabilities can be modeled as intangible assets which are valued by
investors and which increase output and profitability.” The last time I heard of a new class of
intangible assets, it was the Dot Com boom when every company was promoting
their first CTO/CIO. Perhaps we’re in the midst of the big data boom and we’ll soon see a
wave of Chief Data Officer promotions?
“When There’s No
Such Thing As Too Much Information.” By Steve Lohr. The New York Times. April 23rd, 2011.
“Strength In
Numbers: How Does Data Driven Decisionmaking Affect Firm Performance?” By Erik Brynjolfsson, Lorin Hitt, Heekyung
Hellen Kim. April 22, 2011.
“Risks of Data
Driven Decision Making.” By Arjun Moorthy. 08APR2012.
“Seven Steps To
Creating a Data Driven Decision Making Culture.” By Avenash Kaushik. 23OCT2006.
“Rebel! Refuse
Reporting Requests. Only Answer Business Questions, FTW. “ Avenash Kaushik. 04OCT2010.
“Harnessing The
Power Of Feedback Loops.” By Thomas
Goetz. 19JUN2011. Wired. July 2011 Issue.
