Professional Documents
Culture Documents
Book Report - Superforecasting The Art and Science of Prediction
Book Report - Superforecasting The Art and Science of Prediction
BOOK REPORT
Superforecasting: The Art and Science of Prediction
Authors: Philip E. Tetlock and Dan Gardner
INTRODUCTION:
Superforecasting The Art & Science of Prediction, is a book which teach us how
to be a good forecaster with real life examples, and approaches the events from a
different (but interesting) perspective. Each chapter describes different features’ of
superforecasters with striking examples and astonishes the reader. In first chapter the
author mentions about predictability and unpredictability. Also tells us there is no (weak)
relationship between intelligence, numeracy knowledge and being a good forecaster. The
point is how to think. In second chapter, we get information about our thinking system (1
& 2). And teaches us doubts’ importance in prediction. Chapter 3, the author talks about
importance of usage numeric probability rather than maybe, likely etc, clearly defined
terms and timelines. If we want to be a good forecaster we must satisfy these basic
requirements. In fourth chapter we see how luck and skill play role in forecasting and
observe that how an ordinary people beat experts in very serious game. In Chapter 5, we
sight, how to approach questions, and a path: Fermi-Ize-> Outside View-> Inside View-
>Synthesis. In sixth chapter Philip Tetlock mentions about 2 different ways to approach
the events: Fate and Probabilistic thinking. And shows the relationship between thinking
way and forecasting accuracy. In seventh chapter, updating the prediction is emphasized.
A superforecaster should update their prediction frequently and for not going wrong a
way, updating should be done bit by bit. In chapter eight, they emphasize that forecasts
should always be in perpetual beta. A superforecaster should always analysis and
improve their forecast. In chapter nine, they talk about the advantages and disadvantages
of a team for forecasters. In chapter ten, they talk about what should be the qualities of a
leader in the team. In chapter eleven, they analyzed how biases affects the forecasts. In
chapter twelve, they look at how human qualities of a forecaster can affect the forecast.
science but the lacks what makes it truly scientific. Feynman (Physician) observed,
“Doubt is not a fearful thing but a thing of very great value”.
In describing how we think and decide, modern psychologists often deploy a
dual-system model; system 1 and system 2. The numbering of the two systems is not
arbitrary. System 1 comes first. It is fast and constantly running in the background.
“System 1 is designed to jump conclusion from little evidence” which Kahneman said.
Kahneman also gave a memorable label to System 1: WYSIATI (What You See Is All
There Is). On the other hand, System 2 is charged with interrogating the System 1’s
answer. Another mistake which is done by forecasters is trying to solve a question which
requests system 2 with system 1. Formally, it is called attribute substation. The message
is not “System 2 is better than System 1”. System 1 is fast, automatic and effortless.
Makes it very good at pattern matching. The question is when should we trust it?
Consider a forecast Steve Ballmer made in 2007 when he was CEO of Microsoft:
“There’s no chance that the iPhone is going to get any significant market share. No
chance.”. This prediction was noted “The Ten Worst Tech Predictions of All Time” in
2013. But, how can we say that Steve Ballmer’s’ forecast was wrong? If we parse
forecast carefully the key term was there is “significant market share”. What qualifies
significant? And which market was Ballmer talking about? And market for what? There
were too many unanswered questions. Ballmer’s’ forecast is ambiguous. It looks wrong,
it feels wrong. But is it wrong beyond all reasonable doubt? The author says, “Judging
forecasts is much harder than often supposed, a lesson I learned the hard way—from
extensive and exasperating experience.”
Forecast without a time frame is absurd. This problem alone renders many
everyday forecasts untestable. Similarly, forecasts often rely on implicit understandings
of key terms rather than explicit definitions—like “significant market share” in Steve
Ballmer’s forecast. This sort of vague verbiage is more the rule than the exception. And it
too renders forecasts untestable. Forecasters use words like maybe, probably, likely, fair
chance etc. during making forecast, instead of numeric probabilities. These also make
prediction untestable. Sherman Kent (Academician) has legendary quote about this
situation: “I’d rather be a bookie than a goddamn poet”.
Shortly, we need to have forecasting questions with clearly defined terms and
timelines. We need to have lots of forecasts with numbers, and the math to calculate
scores (Brier). We must have squeezed out as much ambiguity as appears humanly
possible.
Chapter 4: Superforecasters
In October 2002 Intelligence Community(IC) reported that Iraq had continued its
weapons of mass destruction(WMD) programme and the threat was growing. The US
invaded Iraq in 2003 after this analysis. United States turned Iraq upside down looking
for WMD but found nothing. Which was one of the worst intelligence failure in modern
history. In 2006 the Intelligence Advanced Research Projects Activity (IARPA) was
IE 332 Book Report S. Kami Üçkardeş – Dilan Derin
created. to make the intelligence community smarter and more effective. They started
working on a project which was called Good Judgment Project (GJP). In the first year,
several thousand volunteers signed up, after first test this number decreased to 3200 and
these 3200-people started forecasting. Time passed. And the result was shocking. A few
hundred-ordinary people and some simple math can not only compete with professionals
supported by a multibillion-dollar apparatus but also beat them.
Most things in life involve skill and luck, in varying proportions. Regression to
the mean is an indispensable tool for testing the role of luck in performance: Mauboussin
(Managing director and head of Global Financial Strategies at Credit Suisse) notes that
slow regression is consistent with activities dominated by skill, while faster regression is
more associated with chance. Hopefully, we can conclude that the superforecasters were
not just lucky. Mostly, their results reflected skill.
Chapter 5: Supersmart?
Sandford Sillman was one of the greatest -actually the greatest- superforecaster
with a Brier score of 0,19. (In GJP). Sandy has a bachelor of arts degree, with a double
major in math and physics, from Brown University, plus a master of science degree from
MIT’s technology and policy program, along with a second master’s degree, in applied
mathematics, from Harvard—and a Ph.D. in applied physics from Harvard. What a mind
it is. Can we conclude that superforecasters are better simply because they are more
knowledgeable and intelligent than others? According to tests results; Regular forecasters
scored higher on intelligence and knowledge tests than about 70% of the total population.
This number was 80% for superforecasters. As we can see there is no big difference
between those numbers. So, intelligence and knowledge help but they add little beyond a
certain threshold.
How to approach questions? Common path is as follows: Fermi-Ize-> Outside
View-> Inside View->Synthesis. If you do not have any idea about the question, divide
the problem to sub-problems. Fermi-Ize; This method makes forecasting easier because it
reduces the number of random guesses and prevents bait and switch (If we face with a
difficult problem we automatically switch that with an easier one) which changes the
structure of the problem. Outside view; Before focusing on details of the problem, look at
the big picture. (If question asks a number of pianos in Roseland, first look the number of
pianos in Chicago etc.) If we start forecasting with outside view, we will begin with an
anchor (the number we start forecasting) which will be meaningful. And a better anchor
is a distinct advantage. Inside view; Extremely dangerous step of forecasting, if you
aimlessly examine one tree, then another, and another, you will quickly become lost in
the forest. It should be targeted and purposeful: it is an investigation, not an amble.
Firstly, we must construct a hypothesis, then start looking for evidence pro and con. This
looks like a detective work because it is methodical, slow and demanding. But it works
far better than wandering aimlessly in a forest. Another condition of being a successful
forecaster is to be able to approach events from different perspectives. How to obtain new
perspectives? Method 1: Wisdom of crowd (what other forecasters think about your
approach, what is their starting point). Method 2: Tweak wording (Imagine a question
like “Will the South African government grant the Dalai Lama a visa within six months?
IE 332 Book Report S. Kami Üçkardeş – Dilan Derin
“and replace grant with deny.) With this method, you can get opposite sides’ thought,
perspective.
For superforecasters, beliefs are the hypothesis to be tested, not treasures to be
guarded.
Chapter 6: Superquants?
Human beings have coped with uncertainty for as long as we have been
recognizably human. Both 0% and 100% weigh far more heavily in our mind than other
percentages. Because of this we always ignore small chances and use the two-setting dial
(Either true or false) as much as possible. This is the proof that why people who are so
sure about themselves are more popular on media. (Tv. Etc.)-In reality, they have no
ability except storytelling, talking heads-. Most people would identify science with
certainty. But that was a very nineteenth-century view of science. One of twentieth-
century science’s great accomplishments has been to show that uncertainty is an
ineradicable element of reality. “Uncertainty is real,” Byers (Canadian mathematician &
philosopher) writes. If nothing is certain, it follows that two settings mental dials totally
useless. The only setting that remains is “maybe” (The third one)-the one setting people
intuitively want to avoid-.
Let’s examine the relationship of forecasters with “certainty”. The average
forecaster who sticks with tens is less accurate than who uses five and still less accurate
than who uses one. Most people never attempt to be more precise. Which is a serious
mistake. As the legendary investor Charlie Munger sagely observed, “If you don’t get
this elementary, but mildly unnatural, mathematics of elementary probability into your
repertoire, then you go through a long life like a one-legged man in an ass-kicking
contest.”
We can approach the events we encounter in life in 2 different ways: Probabilistic
thinking and meant-to-happen thinking (fate). Experiments have shown that fate thinking
rate goes down from adult Americans to undergrad students and from undergrad students
to forecasters. There is also a relationship between thinking way and forecasting
accuracy. As the fate thinking rate decreases the accuracy increases.
Chapter 7: Supernewsjunkies?
irrelevant for our prediction and this can weaken our confidence. Solution for both
underreacting and overreacting is commitment and finding middle passage.
A superforecaster should update their prediction little by little. But like Bayes
theorem says, new prediction should be based on both past prediction and the new
information.
Chapter 9: Superteams
Building team from forecasters can be really effective. People tend to make
forecast with discussing other people even when they are not work in a team. Discussing
with other people can sharpen judgement and you can do things that you cannot do alone.
In teams, people share information between them; this lead them see other perspectives
for the issue. It is important for improving the accuracy.
But the danger is they can create illusions when they encounter with an
uncomfortable situation like in the Bay of Pigs disaster. Teams with collective
unconsciousness, tend to agree with each other. Since everyone agrees, they assume they
are in the right path. But if every team member is conscious about this and questioning
themselves if they’re in this delusion, the danger will be minimized. When
superforecasters encounter with an agreement, they question themselves and search for
the reasons why it is accurate. Tetlock’s research shows that in one year, teams were 23%
more accurate than individual forecasters. Individuals who are educated one year for
becoming a forecaster are placed in a team and after one year their accuracy is increased
by 50%.
Leaders have a crucial role in a team for avoiding the dangers of being a team.
They must lead the team because of that they need to be confident, decisive and have a
vision. But forecasting is fill with uncertainties. How can a leader lead the team in this
ocean of uncertainties?
In the 19th century, the Prussian army used uncertainty as a way to understand the
reality. All the circumstances can change very fast and because of that soldiers must
adapt these changing circumstances even without the orders of the commanders. They
need to be flexible. The commanders told the soldiers the goals, not step by step
instructions to achieve this goal. They communicated effectively and encouraged the
soldiers making decisions without waiting for the orders. The leader of the
superforecasters should also not give orders but the information, goals; leader should give
new perspectives. Uncertainty is important, superforecasters should doubt the ideas from
themselves and their teams’ but the uncertainty shouldn’t be about their own abilities.
The superforecasters are good because they work hard for their prediction, they
value the details and think every important aspect carefully. They use System 2 to catch
the mistakes that are from the System 1. But sustaining all this work for a long time is
hard even for a superforecaster. They are humans after all, there’ll be a moment where
they choose their biases for choosing the easier method. Like in Müller-Lyer Illusion;
even if you check with a ruler to see that the lines are identical, without the ruler again
you’ll see them in different lengths. But minimizing and resisting the biases is possible.
“Nassim Taleb’s Black Swan”: Let’s assume that we are in Europe four centuries
ago where all the swans are white. When a person asks you that think about a weird
swan, you’ll think an ugly swan whether a large size or different shape. But you won’t
think a swan with a different color, all the swans you think will be white. When an
Australian boat comes with a black swan, you’ll be stunned. Yes, some situations can be
unpredictable but consequences can be, at least you’ll be prepared for such situations.
Sometimes forecasters put ahead of their interests while they are forecasting. It
happens because as a human we tend to do that. Maybe you’re in politics and don’t want
to admit a truth or maybe you do this unconsciously. But for your forecast, accuracy will
fall inevitably. For controlling these obstacles, some countries introduced evidence-based
policies. Good forecasting effects many aspects of life. It can help us to see if an event is
success or failure. Even we can predict there will be a war or not. Accurate forecasting
will always be wanted.
For a superforecaster, asking the right questions is also really important. But there
is not one big question to answer. For finding accurate answer, one should ask many
questions. With enough information and considering that there are no certainties in life,
one can become a superforecaster.
IE 332 Book Report S. Kami Üçkardeş – Dilan Derin
TAKE-AWAYS:
*To be a skeptic and do not forget that that even small things lead great changes. As we
see in first chapter a Tunisian paddler changed the system of whole Arabic peninsula. We
need to think too much even in making small decisions.
*We have to measure our forecast after making it, and repeating following cycle:
Forecast, measure, revise. Without measure and revise forecasting means nothing.
Weather forecasters are good and getting better because they apply this cycle perfectly.
One of the important conditions of being a good forecaster is measuring and revising
forecast. We have to do it if we want to be one.
*We should not accept anything as it is in life (cargo cult) and try to understand it. We
must make the suspicion a part of our lives and approach the events with it. Do not forget
the Feynman’s’ speech, “Doubt is not a fearful thing but a thing of very great value”.
*If we want to make our prediction testable and understandable by everyone, we have to
make it with clearly defined terms and timelines, and also numeric probabilities than
maybe, likely etc. Because as Kent said “I’d rather be bookie than a goddamn poet.”
*IARPA’s Good Judgment Project proves that, there is really weak relation between
intelligence, numeracy knowledge and being a good forecaster. And shows that ordinary
people beats experts in a very serious game. The point is how to think.
*If we do not have any idea about a subject or a question, we can follow this path: Fermi-
Ize-> Outside View-> Inside View->Synthesis, and come up with an idea or an answer.
The first thing we have to do is looking at the big picture instead of a details of the
problem. If we do this we will begin with a good anchor. And a better anchor is distinct
advantage.
*If we want to be a good forecaster we have to know that, our beliefs are hypothesis to be
tested, not treasures to be guarded. In daily life we have to open to different opinions and
take them into account. The most important way to succeed is to be open to innovations
and different opinions.
* A good forecaster should update their prediction based on a new information that
comes. But he or she should be careful about not overreacting or underreacting the
information.