Professional Documents
Culture Documents
Survivorship Bias You Are Not So Smart PDF
Survivorship Bias You Are Not So Smart PDF
May
2013
23
About/Contact
All Posts
Books
Merch
Podcast
Survivorship Bias
TOP POSTS
Survivorship Bias
All Posts
22,203
These masters of math moved their families across the country, some
across an ocean, so they could work together. As they unpacked, the
theaters in their new hometowns replaced posters for Citizen Kane with
those for Casablanca, and the newspapers they unwrapped from photo
frames and plates featured stories still unravelling the events at Pearl
Harbor. Many still held positions at universities. Others left those sorts of
jobs to think deeply in one of the many groups that worked for the armed
forces, free of any other obligations aside from checking in on their
families at night and feeding their brains during the day. All paused their
careers and rushed to enlist so they could help crush Hitler, not with guns
and brawn, but with integers and exponents.
The official name for the people inside the apartment was the Statistical
Research Group, a cabal of geniuses assembled at the request of the
White House and made up of people who would go on to compete for and
win Nobel Prizes. The SRG was an extension of Columbia University, and
they dealt mainly with statistical analysis. The Philadelphia Computing
Section, made up entirely of women mathematicians, worked six days a
week at the University of Pennsylvania on ballistics tables. Other groups
with different specialties were tied to Harvard, Princeton, Brown and
others, 11 in all, each a leaf at the end of a new branch of the government
created to help defeat the Axis the Department of War Math.
Actuallyno. They were never officially known by such a deliciously sexy
title. They were instead called the Applied Mathematics Panel, but they
operated as if they were a department of war math.
The Department, ahem, the Panel, was created because the United States
needed help. A surge of new technology had flooded into daily life, and the
same wonders that years earlier drove ticket sales to the Worlds Fair were
now cracking open cities. Numbers and variables now massed into
scenarios far too complex to solve with maps and binoculars. The military
realized it faced problems that no soldier had ever confronted. No best
practices yet existed for things like rockets and radar stations and aircraft
carriers. The most advanced computational devices available were clunky
experiments made of telephone switches or vacuum tubes. A calculator still
looked like the mutant child of an old-fashioned cash register and a
mechanical typewriter. If you wanted solutions to the newly unfathomable
problems of modern combat you needed powerful number crunchers, and
in 1941 the worlds most powerful number crunchers ran on toast and
coffee.
Here is how it worked: Somewhere inside the vast machinery of war a
commander would stumble into a problem. That commander would then
send a request to the head of the Panel who would then assign the task to
the group he thought would best be able to resolve the issue. Scientists in
that group would then travel to Washington and meet with with top military
personnel and advisors and explain to them how they might go about
solving the problem. It was like calling technical support, except you called
a computational genius who then invented a new way of understanding the
world through math in an effort to win a global conflict for control of the
planet.
running shirtless across a football field swarming with angry hornets and
making it unharmed to the other side. You might make it across once, but if
you kept running back and forth, eventually your luck would run out. Any
advantage the mathematicians could provide, even a very small one, would
make a big difference day after day, mission after mission.
As with the torpedo problem, the top brass explained what they knew, and
the Panel presented the problem to Wald and his group. How, the Army Air
Force asked, could they improve the odds of a bomber making it home?
Military engineers explained to the statistician that they already knew the
allied bombers needed more armor, but the ground crews couldnt just
cover the planes like tanks, not if they wanted them to take off. The
operational commanders asked for help figuring out the best places to add
what little protection they could. It was here that Wald prevented the
military from falling prey to survivorship bias, an error in perception that
could have turned the tide of the war if left unnoticed and uncorrected. See
if you can spot it.
The military looked at the bombers that had returned from enemy territory.
They recorded where those planes had taken the most damage. Over and
over again, they saw the bullet holes tended to accumulate along the
wings, around the tail gunner, and down the center of the body. Wings.
Body. Tail gunner. Considering this information, where would you put the
extra armor? Naturally, the commanders wanted to put the thicker
protection where they could clearly see the most damage, where the holes
clustered. But Wald said no, that would be precisely the wrong decision.
Putting the armor there wouldnt improve their chances at all.
Do you understand why it was a foolish idea? The mistake, which Wald saw
instantly, was that the holes showed where the planes were strongest. The
holes showed where a bomber could be shot and still survive the flight
home, Wald explained. After all, here they were, holes and all. It was the
planes that werent there that needed extra protection, and they had
needed it in places that these planes had not. The holes in the surviving
planes actually revealed the locations that needed the least additional
armor. Look at where the survivors are unharmed, he said, and thats
where these bombers are most vulnerable; thats where the planes that
didnt make it back were hit.
Taking survivorship bias into account, Wald went ahead and worked out
how much damage each individual part of an airplane could take before it
was destroyed engine, ailerons, pilot, stabilizers, etc. and then through
a tangle of complicated equations he showed the commanders how likely it
was that the average plane would get shot in those places in any given
bombing run depending on the amount of resistance it faced. Those
calculations are still in use today.
environment are wildly successful because only the very best and the very
lucky can survive. All you are left with are super successes, and looking at
them day after day you might think its a great business to get into when
you are actually seeing evidence that you should avoid it.
Survivorship bias pulls you
toward bestselling diet gurus,
celebrity CEOs, and superstar
athletes. Its an unavoidable
tick, the desire to deconstruct
success like a thieving magpie
and pull away the shimmering
bits. You look to the
successful for clues about the
hidden, about how to better
live your life, about how you
too can survive similar forces
against which you too
struggle. Colleges and
conferences prefer speakers
who shine as examples of
making it through adversity, of
Googles Larry Page Source: Fortune
struggling against the odds
Magazine
and winning. The problem here
is that you rarely take away
from these inspirational figures advice on what not to do, on what you
should avoid, and thats because they dont know. Information like that is
lost along with the people who dont make it out of bad situations or who
dont make it on the cover of business magazines people who dont get
invited to speak at graduations and commencements and
inaugurations. The actors who traveled from Louisiana to Los Angeles only
to return to Louisiana after a few years dont get to sit next to James Lipton
and watch clips of their Oscar-winning performances as students eagerly
gobble up their crumbs of wisdom. In short, the advice business is a
monopoly run by survivors. As the psychologist Daniel Kahneman writes in
his book Thinking Fast and Slow, A stupid decision that works out well
becomes a brilliant decision in hindsight. The things a great company like
Microsoft or Google or Apple did right are like the planes with bullet holes
in the wings. The companies that burned all the way to the ground after
taking massive damage fade from memory. Before you emulate the history
of a famous company, Kahneman says, you should imagine going back in
time when that company was just getting by and ask yourself if the outcome
of its decisions were in any way predictable. If not, you are probably seeing
patterns in hindsight where there was only chaos in the moment. He sums it
up like so, If you group successes together and look for what makes them
similar, the only real answer will be luck.
If you see your struggle this way, as partly a game of chance, then as
Google Engineer Barnaby James writes on his blog, skill will allow you to
place more bets on the table, but its not a guarantee of success. Thus, he
warns, beware advice from the successful. Entrepreneur Jason Cohen, in
writing about survivorship bias, points out that since we cant go back in
time and start 20 identical Starbucks across the planet, we can never know
Over the course of 10 years, Wiseman followed the lives of 400 subjects of
all ages and professions. He found them after he placed ads in
newspapers asking for people who thought of themselves as very lucky or
very unlucky. He had them keep diaries and perform tests in addition to
checking in on their lives with interviews and observations. In one study, he
asked subjects to look through a newspaper and count the number of
photographs inside. The people who labeled themselves as generally
unlucky took about two minutes to complete the task. The people who
considered themselves as generally lucky took an average of a few
seconds. Wiseman had placed a block of text printed in giant, bold letters
on the second page of the newspaper that read, Stop counting. There are
43 photographs in this newspaper. Deeper inside, he placed a second
block of text just as big that read, Stop counting, tell the experimenter you
have seen this and win $250. The people who believed they were unlucky
usually missed both.
Wiseman speculated that what we call luck is actually a pattern of behaviors
that coincide with a style of understanding and interacting with the events
and people you encounter throughout life. Unlucky people are narrowly
focused, he observed. They crave security and tend to be more anxious,
and instead of wading into the sea of random chance open to what may
come, they remain fixated on controlling the situation, on seeking a specific
goal. As a result, they miss out on the thousands of opportunities that may
float by. Lucky people tend to constantly change routines and seek out new
experiences. Wiseman saw that the people who considered themselves
lucky, and who then did actually demonstrate luck was on their side over
the course of a decade, tended to place themselves into situations where
anything could happen more often and thus exposed themselves to more
random chance than did unlucky people. The lucky try more things, and fail
more often, but when they fail they shrug it off and try something else.
Occasionally, things work out.
Wiseman told Skeptical Inquirer magazine that he likened it to setting
loose two people inside an apple orchard, each tasked with filling up their
baskets as many times as possible. The unlucky person tends to go to the
same few spots over and over again, the basket holding fewer apples each
visit. The lucky person never visits the same spot twice, and that persons
basket is always full. Change those apples to experiences, and imagine a
small portion of those experiences lead to fame, fortune, riches, or some
other form of happiness material or otherwise, and you can see that chance
is not as terrifying as it first appears, you just need to learn how to
approach it.
The harder they looked, the less they saw. And so it is with
luck unlucky people miss chance opportunities because they
are too focused on looking for something else. They go to
parties intent on finding their perfect partner and so miss
opportunities to make good friends. They look through
newspapers determined to find certain type of job
advertisements and as a result miss other types of jobs. Lucky
people are more relaxed and open, and therefore see what is
there rather than just what they are looking for. Richard
Wiseman in an article written for Skeptical Inquirer
Survivorship bias also flash-freezes your brain into a state of ignorance
from which you believe success is more common than it truly is and
therefore you leap to the conclusion that it also must be easier to obtain.
You develop a completely inaccurate assessment of reality thanks to a
prejudice that grants the tiny number of survivors the privilege of
representing the much larger group to which they originally belonged.
Here is an easy example. Many people believe old things represent a
higher level of craftsmanship than do new things. Its sort of a they dont
make them like they used to kind of assumption. Youve owned cars that
only lasted a few years before you had to start replacing them piece by
piece, and, would you look at that, there goes another Volkswagon Beetle
buzzing along like it just rolled off an assembly line. Its survivorship bias at
work. The Beetle or the Mustang or the El Camino or the VW Minibus are
among a handful of models that survived in large enough numbers to
become iconic classics. The hundreds of shitty car designs and millions of
automobile corpses in junkyards around the world far outnumber the
popular, well-maintained, successful, beloved survivors. According to Josh
Clark at HowStuffWorks, most experts say that cars from the last two
decades are far more reliable and safer than the cars of the 1950s and
60s, but plenty of people believe otherwise because of a few high-profile
survivors. The examples that would disprove such assumptions are rusting
out of sight. Do you see how its the same as Walds bombers? The Beetle
survived, like the bombers that made it home, and it becomes a
representative of 1960s cars because it remains visible. All the other cars
that werent made in the millions and werent easy to maintain or were
poorly designed are left out of the analysis because they are now removed
from view, like the bombers that didnt return.
Similarly, photographer Mike Johnston explains on his blog that the
artwork that leaps from memory when someone mentions a decade like
the 1920s or a movement like Baroque is usually made up of things that do
not suck. Your sense of a past era tends to be informed by paintings and
literature and drama that are not crap, even though at any given moment
pop culture is filled with more crap than masterpieces. Why? It isnt
because people were better artists back in the day. It is because the good
stuff survives, and the bad stuff is forgotten. So over time, you end up with
skewed ideas of past eras. You think the artists of antiquity were amazing
in the same way you associate the music of past decades with the songs
that survived long enough to get into your ears. The movies about Vietnam
never seem include in their soundtracks the songs that sucked.
survivorship bias do the rest. Those always use the most positive claims,
the most outrageous examples to sell a product, Phil Plait, an astronomer
and leading voice in the skeptical movement, explained to me. When
these things dont work for the vast majority of customers, you never hear
about it, at least not from the seller. The people who use the diet, or the
product, or the pill, and fail to lose weight dont get trotted out for photo
shoots only the successes do. That same phenomenon has become a
problem in science publications, especially among the younger sciences
like psychology, but it is now under repair. For far too long, studies that
fizzled out or showed insignificant results have not been submitted for
publication at the same level as studies that end up with positive results, or
even worse, theyve been rejected by prominent journals. Left unchecked,
over time you end up with science journals that only present the survivors
of the journal process studies showing significance. Psychologists are
calling it the File Drawer Effect. The studies that disprove or weaken the
hypotheses of high-profile studies seem to get stuffed in the file drawer, so
to speak. Many scientists are pushing for the widespread publication of
replication, failure, and insignificance. Only then, they argue, will the
science journals and the journalism that reports on them accurately
describe the world being explored. Science above all will need to root out
survivorship, but it wont be easy. This particular bias is especially
pernicious, said Plait, because it is almost invisible by definition. The only
way you can spot it is to always ask: what am I missing? Is what Im seeing
all there is? What am I not seeing? Those are incredibly difficult questions
to answer, and not always answerable. But if you dont ask them, then by
definition you cant answer them. He added. Its a pain, but reality can be
a tough nut to crack.
Failure to look for what is missing is a common shortcoming, not just within
yourself but also within the institutions that surround you. A commenter at
an Internet watering hole for introverts called the INTJForum explained it
with this example: when a company performs a survey about job
satisfaction the only people who can fill out that survey are people who still
work at the company. Everyone who might have quit out of dissatisfaction is
no longer around to explain why. Such data mining fails to capture the only
thing it is designed to measure, but unless management is aware of
survivorship bias things will continue to seem peachy on paper. In finance,
this is a common pitfall. The economist Mark Klinedinst explained to me
that mutual funds, companies that offer stock portfolios, routinely prune out
underperforming investments. When a mutual fund tells you, The last five
years we had 10 percent on average return, well, the companies that didnt
have high returns folded or were taken over by companies that were more
lucky. The health of the companies they offer isnt an indication of the
mutual funds skill at picking stocks, said Klinedinst, because theyve
deleted failures from their offerings. All you ever see are the successes.
Thats true for many, many elements of life. Money experts who made
great guesses in the past are considered soothsayers because their
counterparts who made equally risky moves that failed nosedived into
obscurity and are now no longer playing the game. Whole nations left
standing after wars and economic struggles pump fists of nationalism
assuming that their good outcomes resulted from wise decisions, but they
can never know for sure.
crashed against the side of the Nilgiri mountains in India. Perhaps there is
an irony to that, something about airplanes and odds and chance and luck,
but it isnt the interesting part of Walds story. His contributions to science
are what survives his time on Earth and the parts of his tale that will endure.
In 1968, the National Academy of Sciences issued a report saying the
application of mathematics in World War II became recognized as an art,
and the lessons learned by the mathematicians were later applied to
business, science, industry, and management. They saved the world and
then rebuilt it using the same tools each time calculators and chalk.
In 1978, Allen Wallis, Director of SRG said of his team, This was surely the
most extraordinary group of statisticians ever organized. The bomber
problem was just a side story for them, a funny anecdote that only surfaced
in the 1980s as they all began to reminisce full time. When you think of how
fascinating the story is, it makes you wonder about the stories well never
hear about those numerical soldiers because they never made it out of the
war and into a journal, magazine, or book, and how thats true of so much
thats important in life. All we know of the past passes through a million,
million filters, and a great deal is never recorded or is tossed aside to
make room for something more interesting or beautiful or audacious. All
we will learn from history reaches us from the stories that, for whatever
reason, survived.
The new book from the author of You Are Not So Smart You Are
Now Less Dumb
Go deeper into understanding just how deluded you really are and
learn how you can use that knowledge to be more humble, better
connected, and less dumb in the sequel to the internationally
bestselling You Are Not So Smart. Preorder now at: Amazon | Barnes
& Noble | Books-a-Million | Indiebound | iTunes
Sources
Ask HN: Survivor Bias. Why dont we focus more on startups that fail? 21
November 2012. 15 January 2013 https://news.ycombinator.com/item?
id=4812480
Association, The American Statistical. Resolution in Honor of Abraham
Wald. The American Statistician February 1951: 19.
Bodanis, David. E=mc2: A Biography of the Worlds Most Famous
Equation. Penguin, 2004.
Rees, Mina. The Mathematical Sciences and World War II. The
American Mathematical Monthly October 1980: 607-621.
Samaniego, Marc Mangel and Francisco J. Walds Work on Aircraft
Survivability. Journal of the American Statistical Association 79.386
(1984): 259-267.
ScepticaTV. Derren Brown The System (Full). February 2011. 9 May
2013 https://www.youtube.com/watch?v=9R5OWh7luL4
Scherer, Sina. Bias in psychology: Bring in all significant results. 1 June
2012. 15 2013 January http://jeps.efpsa.org/blog/2012/06/01/falsificationof-previous-results/
Skinner, D. (2006). The Age of Female Computers. Retrieved May
2013, from http://www.thenewatlantis.com/publications/the-age-offemale-computers
Smith, M. D., Wiseman, R. & Harris, P. (2000). The relationship between
luck and psi. Journal of the American Society for Psychical Research,
94, 25-36.
Smith, M. D., Wiseman, R., Harris, P. & Joiner, R. (1996). On being lucky:
The psychology and parapsychology of luck. European Journal of
Parapsychology, 12, 35-43.
Tte, Annie. SciTech Tuesday: Abraham Wald, Seeing the Unseen. 13
November 2012. 15 January 2013
http://www.nww2m.com/2012/11/scitech-tuesday-abraham-wald-seeingthe-unseen/
Unknown. Our National Survivor Bias. 13 December 2010. 15 January
2013 http://spacetravelsacrime.blogspot.com/2010/12/our-nationalsurvivor-bias.html
Wallis, W. Allen. The Statistical Research Group, 1942-1945. Journal of
the American Statistical Association 75.370 (1980): 320-330.
. The Statistical Research Group, 1942-1945: Rejoinder. Journal of
the American Statistical Association 75.370 (1980): 334-335.
West, Walt. Examining Your Failures Can Lead to More Success: Beware
Survivor Bias Mistakes. 7 April 2011. 15 January 2013
http://www.practicemanagementacademy.com/2011/04/survivor-bias
Wilson, Kevin. Men of Air: The Doomed Youth of Bomber Command.
Orion, 2007.
Wiseman, Richard. The Luck Factor. The Magazine For Science And
Reason May
2003. http://richardwiseman.com/resources/The_Luck_Factor.pdf
Wolfowitz, J. Abraham Wald, 1902-150. The Annals of Mathematical
Statistics 23.1 (1952): 1-13.
Wolverton, M. (2011). Girl Computers. Retrieved May 2013, from
http://www.americanheritage.com/content/girl-computers
Share this:
Tw eet
1,510
38
Like
12k
1.4k
StumbleUpon
Tornado Damage
OLDER POST
S IT E M A P
About/Contact
All Posts
Books
Merch
Podcast
NEWER POST
YANSS Podcast
Episode Six
S O C IA L / U P D A T E S
BOOK S TU FF
Facebook
Twitter
Google+
RSS
Podcast RSS
Podcast at Soundcloud
Podcast at Stitcher
Email Updates
Pinterest