Beware the Dangers of Selection Bias

During World War II, American military personnel noticed that some parts of planes were hit by enemy fire more often than other parts. They analyzed the bullet holes in the returning planes and launched a program to have these areas reinforced so that they could withstand enemy fire better.

This course of action may seem natural enough but it also contains a fundamental error. It’s called selection bias.

Assume, for the sake of the argument, that planes got hit in lots of places. If the areas that formed vital parts of the machine were hit (call it part A), the airplane was unlikely to make it back to base. If the bullets hit the plane in parts that were not so vital (part B), the plane was much more likely to at least make it back home.

Military personnel inspecting those planes that made it home would conclude that it was part B that needed reinforcement. Of course, the military personnel were wrong in doing so. Planes got hit in part A just as often as in part B; it’s just that the first ones never made it back home. What’s worse, strengthening part B was exactly the wrong thing to do: those parts weren’t so vital; it is part A that needed strengthening!

This is why we call it “selection bias”; we only see a selection of the outcomes, and therefore draw false conclusions. And the world of business is full of it.

Consider, for example, the popular notion that innovation projects require diverse, cross-functional teams. This notion exists because if we analyze some path-breaking innovation projects, we see they were often staffed by such teams. However, it has also been suggested (see for instance the work of Professor Jerker Denrell from Stanford Business School) that diverse, cross-functional teams also often created the biggest failures of all. However, such failures never resulted in any products… Therefore, if we (only) examine the projects which actually resulted in successful innovations, it seems the diverse cross-functional teams did much better. Yet, on average, the homogeneous teams – although not responsible for the few really big inventions – might have done better; always producing a reliable, good set of results.

Similarly, we applaud CEOs who are bold and risk-taking, who use their intuition rather than careful analysis, such as Jack Welch. However, risk, by definition, leads some to succeed but it also leads quite a few of them to fail and slip into oblivion. Those CEOs we never consider; it is the risk-takers that happen to come out on top that we admire and aspire to. Yet, if we’d be able to see the full picture, of all CEOs, innovation teams, and fighter planes, we just might have reached a very different conclusion.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: