Normality Assumptions and Power Law Distributions

I was reading through What the Dog Saw by Malcolm Gladwell, and came across the following story, originally published by the New Yorker in 2006. The story offers three powerful examples demonstrating a fundamental misconception in problem solving (although, it is not described in such dry terms in the article).

Assuming Normality

Gladwell points to the human tendency to assume all problems are defined by a normal distribution. We've been taught that heights, IQ, stock market returns, diamond clarity, and SAT scores all obey this mean-centered behavior.  Even university students adopt this perspective due to a common misinterpretation of the Central Limit Theorem (CLT), that the distribution of values will approach the mean rather than the average of the values will approach the mean. As a result, "The bell-curve assumption has become so much a part of our mental architecture that we tend to use it to organize experience automatically."

Examples of Power Law Distributions in Everyday Life

The article starts with the story of 'Million Dollar Murray'. A lovable, destitute, Nevadan alcoholic who is regularly picked up by Reno police, and frequently by an ambulance for detoxification before he is handed over to the police. One officer has picked up Murray several times per week, for each of the 15 years he's been on the police force. He mentions that Murray ran up a $100,000 hospital bill over the past 6 months at the smaller of Reno's two hospitals, realizing that since taxpayers paid the cost of Murray's urgent care "It cost us one million dollars not to do something about Murray." Furthermore, Murray responded positively to probation and became a model citizen, but quickly regressed when his probation ended. With the hospital bills accumulating, Reno could save money by assigning Murray a chaperone to keep him out of the hospital. This begs the question though, "Can some our greatest social problems be solved with inordinate measures targeted at the few?"

Homelessness and the Power Law Distribution

The next example is as profound as it is politically contentious. Dennis Cullane studied homelessness for his doctoral thesis, where he stayed at a shelter for several weeks. He returned two months later and was shocked to not recognize anyone. This led to his first challenge to conventional wisdom, that "the most common length of homelessness is one day." He later created a homeless database so that he could scientifically study the issue, which led to the following insights.

• 80% of shelter visitors are homeless for one night "the most common length of homelessness is one day"
• 10% are episodic users, primarily young drug users who get off the street for a few weeks in winter
• 10% are chronic homeless, usually older, and many with mental illnesses who rack up exorbitant hospital bills

This has profound implications on solving homelessness because our conventional interpretation of 'homelessness' is really 'chronic homelessness.' Redefining the issue as a problem of the few (Power Law view of homelessness), rather than that all homelessness is identical (Normal distribution view of homelessness) we can target and eliminate the real issue.

Excessive Force in the LAPD and the Power Law Distribution

The LAPD has endured numerous accusations of excessive force over the years. The unwritten assumption has been that the majority of officers earned the same number of accusations and that they clustered around an average. In fact though, accusations of excessive force per officer obey a power law distribution:

• 6,700 officers had zero accusations of excessive force over a four year period
• 1,400 officers had one or two accusations over a four year period
• 183 officers had four or more complaints
• 44 officers had 6 or more complaints
• 16 officers had 8 or more complaints
• 1 officer had 16 complaints

Absent this data, "you'd propose solutions that would raise the performance of the middle—like better training or better hiring—when the middle didn't need help." The Christopher commission (responsible for investigating the systematic nature of the excessive force allegations) "gives the strong impression that if you fired those forty-four cops the L.A.P.D. would suddenly become a pretty well-functioning police department."


Think about the significance of this statement... firing 0.5% of the LAPD would instantly convert it into an exemplary law force whereas training programs would doubtfully produce concrete results. Isn't the immediate, concrete solution a powerful tool for improvement? Doesn't this tool arise directly because we've challenged the normality assumption?

Carbon Monoxide Emissions and the Power Law Distribution

Auto emissions are another example of power law distributions that offer a concrete solution to a broad social issue. The example begins with the statement that "A 2004 Subaru in good working order has an exhaust stream that's just .06 per cent carbon monoxide, which is negligible" and the assertion that 90% of vehicles don't need an emissions inspection. "But on almost any highway, for whatever reason—age, ill repair, deliberate tampering by the owner—a small number of cars can have carbon-monoxide levels in excess of ten per cent, which is almost two hundred times higher."  Heavily used vehicles such as taxi cabs epitomize the offenders, with one example of a cab that emitted more than its weight in annual carbon monoxide emissions. If "five per cent of the vehicles on the road produce fifty-five per cent of the automobile pollution" then shouldn't we focus more on those five percent, rather than inspect every car owner? Would Greenpeace achieve their goals faster by targeting cab companies or perhaps even by giving replacement cars to the worst offenders?

Moral Hazard and Second Order Effects

I find it interesting that some of the most seemingly intractable problems facing society could be solved immediately by targeting specific individuals. People expect these problems to be moon shots. Our intuition demands an enormous investment be made in resources and effort to solve these problems, because otherwise we wonder, "Why wasn't it solved before?" 

• Could we halve pollution next year if Greenpeace paid to replace the worst emitting vehicles? 
• Could we solve homelessness by giving the chronic homeless apartments? 
• Could the LAPD solve excessive force by firing 0.5% of their workforce? 

Even more shocking, could each of these actions be cheaper than maintaining the status quo? I've already mentioned the example of 'Million Dollar Murray' but what about the societal waste in the vehicle inspection process? Every vehicle is inspected biannually, wasting taxpayers' time, costing money, when more efficient solutions exist? How much is spent by LA on the legal defense of excessive force accusations?

Despite the benefits of immediate resolution and cost efficiency, they violate American standards of equal treatment and moral ideals of self-sufficiency. How can we give an apartment to a homeless person when there are many people who would like that help? For the sake of brevity I'll forego the moral implications, but instead rely on Gladwell's apt summarization of the problem, "Power-law problems leave us with an unpleasant choice. We can be true to our principles or we can fix the problem. We cannot do both."

Commentary

Understanding: I consider this article profound, and note how important it is to understand a problem before attempting to solve it. Similarly it points to our need to evaluate and recognize our assumptions in decision making. It reminds me of an important quote from Eliyahu Goldratt's The Goal, that people's arguments are usually correct but their assumptions are wrong. Or to loosely paraphrase Deming's Out of the Crisis, "Policies often outlive the circumstances they were instituted to address." These all point to the need to re-examine our understanding, and our assumptions to assure their validity.

Is employee value described by a power law distribution or a normal distribution?
I was struck by a recent Zuckerberg quote, “Someone who is exceptional in their role is not just a little better than someone who is pretty good,” he said. “They are 100 times better.” Even Paul Allen, the cofounder of Microsoft, wrote in his autobiography, "A great programmer can outproduce an average one by ten to one; with a genius, the ratio might be fifty to one." I would reinterpret this as a 'Power Law view of employee value'. I've argued for some time that employees with scalable skill sets are worth more than those with unscalable skill sets (an idea I picked up from Dr. Lester Thurow in Zero Sum Society), but I've never seen this reflected in salary information. What is the value of someone (Jeff Bezos) who builds a company (Amazon) of 33,700 employees that will soon surpass Wal-mart (572,000 employees) in revenues? Blockbuster laughed when Reed Hastings tried to sell them his patent, and instead created Netflix. What is the value of his insight and that patent now? I would argue that the abnormal returns that entrepreneurs can and have earned over the past three centuries are proof that employee value is a power law distribution. This in turn indicates that most companies seek abnormal returns by paying employees according to a normal distribution, and that Zuckerberg just wants to arbitrage the market inefficiency.


Evaluate Individuals Rather Than Populations
Capital One was the first credit card company to use microsegmentation and they were the fastest growing stock on the SP500. Progressive Insurance regularly targets low-risk individuals within unattractively risky populations such as elderly motorcycle owners vs the general motorcycle owner market. If microsegmentation has been fine tuned for customer application, then why have no governments applied it to social problems, or companies applied it to recruiting? It seems like HR would would be an ideal area of application, especially considering it's already been done in professional sports... so what's the hold up?

"Million Dollar Murray: Why problems like homelessness may be easier to solve than to manage." by Malcolm Gladwell. February 13, 2006. The New Yorker.

http://orionwell.files.wordpress.com/2007/06/random-vs-power-law-distribution-2.jpg

http://en.wikipedia.org/wiki/Eliyahu_M._Goldratt

http://mitpress.mit.edu/catalog/item/default.asp?ttype=2&tid=3261

"For Buyers of Web Startups, Quest to Corral Young Talent." By Miguel Helft. New York Times. May 17, 2011.

http://www.amazon.com/Zero-Sum-Society-Distribution-Possibilities-Economic/dp/0465085881

Idea Man: A Memoir by the Cofounder of Microsoft. by Paul Allen. Published by Penguin Group, 2011. New York, New York.