http://callingbullshit.org/index.

html

Calling Bullshit: Data Reasoning in a Digital World

The world is awash in bullshit. Politicians are unconstrained by facts. Science is conducted
by press release. Higher education rewards bullshit over analytic thought. Startup culture
elevates bullshit to high art. Advertisers wink conspiratorially and invite us to join them in
seeing through all the bullshit — and take advantage of our lowered guard to bombard us
with bullshit of the second order. The majority of administrative activity, whether in
private business or the public sphere, seems to be little more than a sophisticated exercise
in the combinatorial reassembly of bullshit.

We're sick of it. It's time to do something, and as educators, one constructive thing we
know how to do is to teach people. So, the aim of this course is to help students navigate
the bullshit-rich modern environment by identifying bullshit, seeing through it, and
combating it with effective analysis and argument.

What do we mean, exactly, by bullshit and calling bullshit? As a first approximation:

Bullshit involves language, statistical figures, data graphics, and other forms of
presentation intended to persuade by impressing and overwhelming a reader or listener,
with a blatant disregard for truth and logical coherence.

Calling bullshit is a performative utterance, a speech act in which one publicly repudiates
something objectionable. The scope of targets is broader than bullshit alone. You can call
bullshit on bullshit, but you can also call bullshit on lies, treachery, trickery, or injustice.

In this course we will teach you how to spot the former and effectively perform the latter.

While bullshit may reach its apogee in the political domain, this is not a course on political
bullshit. Instead, we will focus on bullshit that comes clad in the trappings of scholarly
discourse. Traditionally, such highbrow nonsense has come couched in big words and
fancy rhetoric, but more and more we see it presented instead in the guise of big data and
fancy algorithms — and these quantitative, statistical, and computational forms of bullshit
are those that we will be addressing in the present course.

Of course an advertisement is trying to sell you something, but do you know whether the
TED talk you watched last night is also bullshit — and if so, can you explain why? Can you
see the problem with the latest New York Times or Washington Post article fawning over
some startup's big data analytics? Can you tell when a clinical trial reported in the New
England Journal or JAMA is trustworthy, and when it is just a veiled press release for some
big pharma company?

Our aim in this course is to teach you how to think critically about the data and models
that constitute evidence in the social and natural sciences.

Carl T. Bergstrom and Jevin West

Seattle, WA.

Calling Bullshit has been developed by Carl Bergstrom and Jevin West to meet what we
see as a major need in higher education nationwide.

Disclaimer: This website is intended for personal educational use and should be employed
for informational purposes only. Accordingly, all warranties and forms of liability from
your use of this website are disclaimed to the extent applicable in your jurisdiction.
Nothing on this website constitutes guaranteed accuracy of any kind. Calls of bullshit
represent the opinions of the instructors and are not intended as definitive judgements of
fact. We are not liable for any loss of credulity you may suffer as a consequence of reading
the information herein. Viewer discretion advised. May cause drowsiness. Void where
prohibited. No animals were used during testing. May cause excitability. Not
recommended for children under the age of 12. Use only as directed. Any similarity to any
person living or dead is merely coincidental. Live, except on West Coast. Do not drive or
operate heavy machinery while using this website. Objects on this site may be closer than
they appear. Additional taxes may apply in some jurisdictions. Individual results may vary.
Not to be used with alcoholic beverages. I bet you think this website is about you, don't
you? Don't you?

Copyright © Calling Bullshit 2017

Syllabus

Calling Bullshit:
Data Reasoning in a Digital World

Logistics

Course: INFO 198 / BIOL 106B. University of Washington

To be offered: Autumn Quarter 2017
Credit: 3 credits, graded
Enrollment: 180 students
Instructors: Carl T. Bergstrom and Jevin West
Synopsis: Our world is saturated with bullshit. Learn to detect and defuse it.

Learning Objectives

Our learning objectives are straightforward. After taking the course, you should be able
to:

 Remain vigilant for bullshit contaminating your information diet.

 Recognize said bullshit whenever and wherever you encounter it.

 Figure out for yourself precisely why a particular bit of bullshit is bullshit.

 Provide a statistician or fellow scientist with a technical explanation of why a claim

is bullshit.

 Provide your crystals-and-homeopathy aunt or casually racist uncle with an

accessible and persuasive explanation of why a claim is bullshit.

We will be astonished if these skills do not turn out to be among the most useful and most
broadly applicable of those that you acquire during the course of your college education.

Schedule and readings

Each of the lectures will explore one specific facet of bullshit. For each week, a set of
required readings are assigned. For some weeks, supplementary readings are also
provided for those who wish to delve deeper.
Lectures

1. Introduction to bullshit

2. Spotting bullshit

3. The natural ecology of bullshit

4. Causality

5. Statistical traps

6. Visualization

7. Big data

8. Publication bias

9. Predatory publishing and scientific misconduct

10. The ethics of calling bullshit.

11. Fake news

12. Refuting bullshit

Week 1. Introduction to bullshit. What is bullshit? Concepts and categories of bullshit. The
art, science, and moral imperative of calling bullshit. Brandolini’s Bullshit Asymmetry
Principle.

 Harry Frankfurt (1986) On Bullshit. Raritan Quarterly Review 6(2)

Supplementary readings

 G. A. Cohen (2002) Deeper into Bullshit. Buss and Overton, eds., Contours of
Agency: Themes from the Philosophy of Harry Frankfurt Cambridge,
Massachusetts: MIT Press.

 Philip Eubanks and John D. Schaeffer (2008) A kind word for bullshit: The problem
of academic writing. College Composition and Communication 59(3): 372-388
  J. L. Austin Performative Utterance, in Austin, Urmson, and Warnock
(1979) Philosophical Papers. Clarendon.

Week 2. Spotting bullshit. Truth, like liberty, requires eternal vigilance. How do you spot
bullshit in the wild? Effect sizes, dimensions, Fermi estimation, and checks on plausibility.
Claims and the interests of those who make them. Forensic data
analysis: GRIM test, Newcomb-Benford law.

 Carl Sagan 1996 The Fine Art of Baloney Detection. Chapter 12 in Sagan (1996) The
Demon-Haunted World

 Case studies: Food stamp fraud, 99% caffeine-free

Week 3. The natural ecology of bullshit. Where do we find bullshit? Why news media
provide bullshit. TED talks and the marketplace for upscale bullshit. Why social media
provide ideal conditions for the growth and spread of bullshit.

 Gordon Pennycook et al. (2015) On the reception and detection of pseudo-

profound bullshit. Judgement and Decision Making 10:549-563

 Adrien Friggeri et al. (2014). Rumor Cascades. Proceedings of the Eighth

International AAAI Conference on Weblogs and Social Media

Week 4. Causality One common source of bullshit data analysis arises when people ignore,
deliberately or otherwise, the fact that correlation is not causation. The consequences can
be hilarious, but this confusion can also be used to mislead. Confusing causality with
necessity or sufficiency. Regression to the mean pitched as treatment effect. Milton
Friedman's thermostat. Selection masked as transformation.

 Robert Matthews (2000) Storks deliver babies (p=0.008). Teaching Statistics22:36-

38

 Case study: Traffic improvements

Supplementary reading
  Karl Pearson (1897) On a Form of Spurious Correlation which may arise when
Indices are used in the Measurement of Organs. Proceedings of the Royal Society
of London 60: 489–498. For context see also Aldrich (1995).

Week 5. Statistical traps and trickery. Bayes rule and conditional probabilities. Base-rate
fallacy / prosecutor's fallacy. Simpson's paradox. Data censoring. Will Rogers effect, lead-
time bias, and length time bias. Means versus medians. Importance of higher moments.

 Simpson’s paradox: an interactive data visualization from VUDlab at UC Berkeley.

 Alvan Feinstein et al. (1985) The Will Rogers Phenomenon — Stage Migration and
New Diagnostic Techniques as a Source of Misleading Statistics for Survival in
Cancer. New England Journal of Medicine 312:1604-1608.

 Case studies: Musicians and mortality, Track records

Week 6. Data visualization. Data graphics can be powerful tools for understanding
information, but they can also be powerful tools for misleading audiences. We explore the
many ways that data graphics can steer viewers toward misleading conclusions.

 Edward Tufte (1983) The Visual Display of Quantitative Information Chapters 2

(Graphical integrity) and 5 (Chartjunk: vibrations, grids, and ducks).

 Tools and tricks: Misleading axes

 Tools and tricks: Proportional Ink

Week 7. Big data. When does any old algorithm work given enough data, and when is it
garbage in, garbage out? Use and abuse of machine learning. Misleading metrics.
Goodhart's law.

 danah boyd and Kate Crawford (2011) Six Provocations for Big Data. A Decade in
Internet Time: Symposium on the Dynamics of the Internet and Society.

 David Lazer et al. (2014) The Parable of Google Flu: Traps in Big Data
Analysis. Science 343:1203-1205
  Alyin Caliskan et al. (2017) Semantics derived automatically from language corpora
contain human-like biases Science 356:183-186

 Jevin West (2014) How to improve the use of metrics: learn from game
theory. Nature 465:871-872

Supplementary reading

 Cathy O'Neil (2016) Weapons of Math Destruction Crown Press.

 Peter Lawrence (2014) The mismeasurement of science. Current Biology17:R583-

585

Week 8. Publication bias. Even a community of competent scientists all acting in good
faith can generate a misleading scholarly record when — as is the case in the current
publishing environment — journals prefer to publish positive results over negative ones.
In a provocative and hugely influential 2005 paper, epidemiologist John Ioannides went so
far as to argue that this publication biashas created a situation in which most published
scientific results are probably false. As a result, it’s not clear that one can safely rely on the
results of some random study reported in the scientific literature, let alone on Buzzfeed.
Once corporate funders with private agendas become involved, matters become all the
more complicated.

 John Ioannidis (2005) Why most published scientific results are false. PLOS
Medicine 2:e124.

 David Michaels and Celeste Monforton (2005) Manufacturing Uncertainty:

Contested Science and the Protection of the Public’s Health and
Environment. American Journal of Public Health 95:S39-S48

Supplementary Reading

 Erick Turner et al. (2008) Selective Publication of Antidepressant Trials and Its
Influence on Apparent Efficacy New England Journal of Medicine358:252-260

 Silas Nissen et al. (2016) Publication bias and the canonization of false
facts. eLife 5:e21451
Week 9. Predatory publishing and scientific misconduct. Predatory publishing. The list
formerly known as Beall's. Publishing economics. Pathologies of publish-or-perish culture.
Pursuit of PR instead of progress. Data dredging, p-hacking, and similar malfeasance.

 Fake academe looking much like the real thing.

New York Times Dec. 29, 2016.

 Adam Marcus and Ivan Oransky (2016) Why fake data when you can fake a
scientist? Nautilus November 24.

 Tools and tricks: How can you know if a paper is legit?

Supplementary Reading

 J. P. Simmons et al. (2011) False-Positive Psychology: Undisclosed Flexibility in Data

Collection and Analysis Allows Presenting Anything as Significant. Psychological
Science 22:1359–1366

Week 10. The ethics of calling bullshit. Where is the line between deserved criticism and
targeted harassment? Is it, as one prominent scholar argued, “methodological terrorism”
to call bullshit on a colleague's analysis? What if you use social media instead of a peer-
reviewed journal to do so? How about calling bullshit on a whole field that you know
almost nothing about? Pubpeer. Principles for the ethical calling of bullshit. The Dunning-
Kruger effect. Differences between being a hard-minded skeptic and being a domineering
jerk.

 Alan Sokal (1996) A physicist experiments with cultural studies. Lingua

Franca 6:62-64.

 Jennifer Ruark (2017) Anatomy of a hoax. Chronicle of Higher Education

 Robert Service (2014) Nano-Imaging Feud Sets Online Sites

Sizzling. Science343:358.

 Susan Fiske (2016) Mob Rule or Wisdom of Crowds? APS Observerpreliminary

draft. Also read commentaries [1] and [2].

 Michael Blatt (2016) Vigilante Science. Plant Physiology 169:907-909.

Week 11. Fake news.. Fifteen years ago, nascent social media platforms offered the
promise of a more democratic press through decentralized broadcasting and a decoupling
of publishing from advertising revenue. Instead, we get sectarian echo chambers and,
lately, a serious assault on the very notion of fact. Not only did fake news play a
substantive role in the November 2016 US elections, but recently a fake news story
actually provoked nuclear threats issued by twitter.

 Before 'Fake News' Came False Prophecy The Atlantic Monthly Dec. 27, 2016

 Factcheck.org: How to spot fake news

 Adrian Chen (2016) The Agency. New York Times Magazine

 Inside a fake news sausage-factory: 'It's all about income'

New York Times Nov. 25, 2016

 Donath, Judith (2016) Why fake news stories thrive online. CNN Opinion.

 Brian Feldman (2017) Google's dangerous identity crisis. New York Magazine

Week 12. Refuting bullshit. Refuting bullshit requires different approaches for different
audiences. What works for a quantitatively-skilled professional scientist won't always
convince your casually racist uncle on facebook, and vice versa.

 John Cook and Stephan Lewandowsky (2012) The Debunking Handbook.

 Craig Bennett et al. (2009) Neural correlates of interspecies perspective taking in

the post-mortem Atlantic Salmon: An argument for multiple comparisons
correction

 Case study: Gender gap in 100 m times

Exercises

Exercise 1: A bullshit inventory. How much bullshit are you dealing with, anyway? Keep
track of your encounters with bullshit over the course of a week, and come up with a way
to visualize your results.
Videos

Calling Bullshit, Spring 2017

We will record and post all ten lectures from our Spring 2017 course. Each will be presented here
as it becomes available. We have divided up every lecture into a set of a shorter segments; these
segments should more or less stand alone on their own merits. The full playlist of all course videos
is available on the UW Information School's YouTube channel.

Lecture 1: An Introduction to Bullshit

March 29, 2017

1.1 Introduction to Bullshit.

Bullshit is everywhere, and we've had enough. We want to teach people to detect and defuse
bullshit where ever it may arise.

1.2 Calling Bullshit on Ourselves.

Jevin uses data graphics to boast about explosive growth at our website callingbullshit.org — and
Carl calls bullshit. Old-school bullshit versus new-school bullshit.

1.3 Brandolini's Bullshit Asymmetry Principle.

Lecture 1.3 "The amount of effort necessary to refute bullshit is an order of magnitude bigger than
to produce it."

1.4 Classroom Discussion.

Students discuss: What is bullshit anyway?

1.5 The Philosophy of Bullshit.

How do we define bullshit? Does intention matter? Calling bullshit as a speech act.

Lecture 2: Spotting Bullshit

April 5, 2017

2.1 Spotting Bullshit.

Jevin discusses some ways to spot bullshit and challenges students to tell whether four nuggets of
wisdom from the internet are true or bullshit.

2.2 Sounds Too Good to be True.

If a claim seems too good — or too bad — to be true, it probably is. An example involving
recommendation letters, and the perils of confirmation bias.
2.3 Entertain Multiple Hypotheses.
The importance of generating and considering multiple alternative hypotheses. As an example, we
consider why men cite themselves more than women do.

2.4 Fermi Estimation.

Using Fermi estimation to check the plausibility of claims, with an example of food stamp fraud.
This example is treated in further detail in one of our case studies.

2.5 Unfair Comparisons.

In this segment on unfair comparisons, Carl explains why St. Louis and Detroit are not quite as bad
as clickbait "most dangerous cities" lists portray them to be, and looks at the silly arguments over
attendance at Trump's inauguration. Also: how to call bullshit on algorithms and statistics without
a PhD in machine learning or statistics.

2.6 Assignment: Bullshit Inventory.

In our first assignment, we ask students to take a week-long bullshit inventory of the bullshit they
encounter, create, and debunk.

Lecture 3: Correlation and Causation

April 12, 2017

3.1 Correlation and Causation

Correlations are often used to make claims about causation. Be careful about the direction in
which causality goes. For example: do food stamps cause poverty?

3.2 What are Correlations?

Jevin providers an informal introduction to linear correlations.

3.3 Spurious Correlations?

We look at Tyler Vigen’s silly examples of quantities appear to be correlated over time, and note
that scientific studies may accidentally pick up on similarly meaningless relationships.

3.4 Correlation Exercise

When is correlation all you need, and causation is beside the point? Can you figure out which way
causality goes for each of several correlations?

3.5 Common Causes

We explain how common causes can generate correlations between otherwise unrelated
variables, and look at the correlational evidence that storks bring babies. We look at the need to
think about multiple contributing causes. The fallacy of post hoc propter ergo hoc: the mistaken
belief that if two events happen sequentially, the first must have caused the second.

3.6 Manipulative Experiments

We look at how manipulative experiments can be used to work out the direction of causation in
correlated variables, and sum up the questions one should ask when presented with a correlation.
Lecture 4: Statistical Traps and Trickery

April 19, 2017

4.1 Right Censoring

We look at a graph of age at death for musicians in different genres, and use this to illustrate the
problem of right-censored data. We consider this issue in further detail in one of our case studies.

4.2 Means and Medians

Simple as it may sound, the difference between mean and median values offers fertile ground for
cooking up misleading statistics.

4.3 p-Values and the Prosecutor’s Fallacy

Carl presents what he thinks may be one of the most important segments in the whole course: a
discussion of the prosecutor’s fallacy. This logical fallacy is not limited to the courtroom: it
underlies a very common misinterpretation of the pvalues associated with scientific experiments.

4.4 The Will Rogers Effect

Will Rogers purportedly quipped that when the Okies left Oklahoma for California, they raised the
average intelligence in both states. The same phenomenon can arise in epidemiology and a host of
other areas.

4.5 Jevin's Turn

Jevin goes looking for bullshit and finds it — in Carl’s textbook. Jevin calls bullshit on Carl’s use of
track and field records by age to illustrate senescence, and Carl tries to explain himself. This
example is described further in another of our case studies.

Lecture 5: Big Data

April 26, 2017

5.1 Big Data Introduction

We briefly introduce big data and provide a few the cautionary tales surrounding this recent
phenomenon. Beware of those ponies…

5.2 Garbage In, Garbage Out

You don’t need a PhD in statistics or machine learning to call bullshit on big data. Simply by
focusing on the input data and the results is often sufficient to refute a claim.

5.3 Big Data Hubris

We discuss the Google Flu Trends project and how it moved from being a poster child for big data
to a providing an important precautionary tale.

5.4 Overfitting
We examine overfitting, the Achilles heel of machine learning. We illustrate overfitting visually,
and consider and what to look out for.
5.5 Criminal Machine Learning
A recent paper claims that machine learning can determine whether or not you are a criminal from
a photograph of your face. That's bullshit. This example is described further in one of our case
studies.

5.6 Algorithmic Ethics

We discuss gender and racial biases inherent to many of the machine learning algorithms and
recommender systems prevalent in today’s technology, and encourage others to call bullshit on
machine injustice.

Lecture 6: Data Visualization

May 3, 2017

6.1 Dataviz in the Popular Media

Until recently, the popular media made minimal use of sophisticated data visualization. People
have not necessarily had time to hone their bullshit detectors for application to data graphics.

6.2 Misleading Axes

One of the most common abuses of data visualization involves the inappropriate ranges on the
dependent variable (y) axis. Carl looks at a series of example, and explain why bar charts should
include zero whereas line graphs need not — and often should not — do so. This example is
treated in further detail in one of our callingbullshit.org articles.

6.3 Manipulating Bin Sizes

By binning data in different ways, bar charts can be made to tell very different stories. Here we
consider an example from the Wall Street Journal.

6.4 Dataviz Ducks

Edward Tufte uses the term “ducks” to refer to data graphics that put style ahead of substance.
We explain why, and explore a number of examples.

6.5 Glass Slippers

We propose the term “glass slipper” to describe to data visualizations in which the designer has
taken a beautiful data design intended for very specific situations, and tried to shoehorn entirely
inappropriate types of data into it. Carl considers examples including a periodic table of data
science, a subway map of corporate acquisitions, a phylogenetic tree of internet marketing, and
numerous Venn diagrams.

6.6 The Principle of Proportional Ink

Our principle of proportional ink states that when a shaded region is used to represent a numerical
value, the area of that shaded region should be directly proportional to the corresponding value.
We look at graphs that violate this principle and discuss how such violations can be misleading.
This example is treated in further detail in one of our callingbullshit.org articles.
Lecture 7: Publication bias

May 10, 2017

7.1 Duck hunting

For last week’s homework assignment, students searched for examples of “duck” and “glass
slipper” data visualizations. Carl and Jevin look at a few of the best finds.

7.2 Science is amazing, but…

Science is probably the greatest human invention of all time, but that doesn’t mean it doesn’t
come with its share of bullshit..

7.3 Reproducibility
Jevin discusses how spreadsheet errors reversed the conclusions of a high-profile paper that was
used to justify austerity measures.

7.4 A Replication Crisis

Scientists have difficulty reproducing a surprisingly large fraction of the published literature. What
is going on?

7.5 Publication Bias

Journals prefer to publish positive results and scientists prefer to submit successful experiments.
This can be misleading given that we typically can look only at the published literature.

7.6 Science is not Bullshit

The subject matter of today’s lecture notwithstanding, science generally works pretty darn well.
We can build airplanes and iPhones and save lives with antibiotics and vaccines, after all. Carl looks
at five reasons why this is true.

Lecture 8: Scholarly publishing and predatory publishers

May 17, 2017

8.1 What Motivates Scientists?

Scientists are not purely seeking knowledge; like anyone else, they are also pursuing fame and
fortune. If we can understand the incentives that scientists face, we can better understand why
they do what they do.

8.2 An Overview of Scholarly Publishing

Jevin talks about how scholarly publishing has become a big business, and describes the rise of the
open access publishing model.

8.3 Predatory Publishing

We introduce the world of so-called predatory or otherwise questionable scientific publishers, and
consider the reasons that authors publish in them anyway.
8.4 Reputable or Questionable?
We challenge our audience to distinguish papers published in reputable journals from papers
published in journals from so-called predatory publishers.

8.5 Journal Spam

Carl explores the scourge of journal spam and some of the humorous ways that academics have
fought back.

Lecture 9: Fake News

May 24, 2017

9.1 The Spreading of Fake News

We investigate some of the top fake news stories from the last year, and discuss how our digital
environments facilitate the spreading of this information.

9.2 Fake News Definitions and Examples

We present examples and definitions of fake news. We explore how this white noise is being
shared millions of times from both sides of the political aisle, making people lots of money, and
fooling our search engines, which can make it difficult for democracy to run effectively.

9.3 The Ecology of Fake News

Fake news is not a new thing. It has been around for a long time — but social media, bots, and
highly partisan environments are spreading fake news ever more readily.

9.4 Sharing as Social Signaling

We are all publishers. Often we share information as a signal of our group membership. Click-
driven publishing models are facilitating the spread of this fake news.

9.5 Stamping out Fake News

Resources exist for checking and stopping fake news. Remember to question a source of
information and do you best not to spread articles you haven’t checked. Consider supporting high
quality journalism with a subscription.

Lecture 10: Refuting Bullshit

May 24, 2017

10.1 Four Rules of Calling Bullshit

We consider four rules for calling bullshit: Be correct; Be charitable; Be clear; Admit fault.

10.2 Reductio ad Absurdum

The method of reductio ad absurdum, in which an argument’s methods are shown to lead to
ridiculous conclusions, is extremely powerful for refuting bullshit claims. We examine statistical
projections of gold medal 100 meter times and the cognitive-emotional responses of a dead
salmon.
10.3 Debunking Myths
Jevin introduces Cook and Lewandowsky’s Debunking Handbook, and suggests a number of rules
for how to successfully change opinions rather than reinforcing erroneous beliefs.

10.4 Deploying Null Models and Tracing the Origin of Falsehoods

First, models can sometimes be used to show that the evidence someone presents does not
require the process for which they use it as evidence. Second, people are more readily convinced
of a falsehood when they are shown where in the communication chain the falsehood arose.

10.5 Counterexamples and analogies

A single counterexample can demolish an elaborate argument, and a well-chosen analogy can
draw out the fallacious reasoning underlying an argument.

10.6 Walk away; Conclusions

Carl stresses the important of being willing to walk away, and offers a few parting thoughts.
Tools

In many of the course lectures we will discuss how you can spot bullshit, call bullshit, and avoid
becoming the victim of bullshit. Here we present a set of instructional essays on various aspects of
bullshit detection and refutation. Many of the examples we draw upon are classic examples that
others have brought to light in their articles, essays, blogs, and other sources.

Tools and Tricks

 Visualization: Spotting misleading axes. Data graphics tell stories. Fairly subtle choices on
the part of their creators can influence the stories they tell, sometimes in misleading
fashion. We look at how the ranges shown on axes can be misleading, and explore the
classic issue of when the y-axis of a graph needs to include zero.

 How do you know a paper is legit? Any scientific paper can be wrong, but you greatly
decrease the chances of being misled if you know how to distinguish legitimate articles
from untrustworthy ones. We discuss how to draw this distinction, and along the way
provide a brief overview of how the scientific publication process works.

 Visualization: Proportional ink. Many data graphics, including bar charts and pie charts,
use the sizes of shaded areas to represent data values. We describe what we call
the principle of proportional ink: in such charts, the amount of ink used to represent a
value should be directly proportional the value itself. Unfortunately, this principle is
commonly violated. We explore a number of examples.

Case Studies: Bullshit in the wild

Spotting bullshit in the wild it isn't something you have to let others do for you. To
illustrate this, we've provided a set of case studies based upon examples of bullshit in the
wild. We've spotted many of these ourselves; some have come via other channels. These
cases aren't the most egregious examples out there, but each illustrates one or more of
the principles and practices that we aim to teach in this course. We will be adding
additional case studies on a regular basis.
Case studies

Basic

These case studies require only clear thinking and occasionally a bit of arithmetic. As such,
they should be readily accessible to all of our readers.

 Food stamp fraud. In this example drawn from a Fox News story, we
demonstrate how Fermi estimation can cut through bullshit like a hot knife
through butter.

 Traffic improvements. Irrelevant facts can lead you to bullshit conclusions if

you approach them with an inaccurate model of how the world works. And if those
conclusions let you spin a trite story about terrible traffic and wasteful government
expenditures, what better clickbait?

 99.9% caffeine-free. In one section of his book The Demon-Haunted World,

Carl Sagan decried that way that advertisers try to dazzle us with irrelevant facts
and figures. He was mostly concerned with drug advertising; in this case study we
explore a more innocuous example.

 Criminal machine learning. Machine learning algorithms are sometimes

touted as generating results that unbiased and without prejudice. This is bullshit.
We explore an example in which two authors claim to have an algorithm that can
determine whether one is a criminal simply from an 80x80 facial image, and show
that this algorithm is actually doing something very different.

 Machine learning about sexual orientation? In this case study, we discuss a

controversial paper that claims a deep neural network can predict sexual
orientation from facial photographs. We illustrate how one can question the
interpretation of results without delving into the details of the machine learning
algorithm used to generate them.
Intermediate

These case studies introduce some basic concepts from statistics such as sample size and
extrapolation. However, they do not require any technical statistical knowledge to follow
and most readers should find these relatively accessible.

 Track and field records as examples of senescence. We lead off our series of
case studies by calling bullshit on a figure in one of our own publications. We
explore how differences in sample sizes can create misleading patterns in data,
and an example of how writing a simulation can be an effective method of calling
bullshit.

 A gender gap in 100 meter dash times. We examine a 2004 Nature paper
predicting that women sprinters will outrun men by the mid-22nd century. In
doing so, we see the danger of over-extrapolation, and we get to read a beautiful
example of reductio ad absurdum as a means of calling bullshit.

 Musicians and mortality. Here we consider what can go wrong as one goes
from scholarly article to popular science piece to social media meme. We explore
why a data graphic shared widely on social media gives a misleading impression,
explain the issue of right-censoring, and discuss how its effects can be seen in the
light of correlation analysis.

Advanced

These case studies make extensive use of calculus and/or mathematical statistics. They
may be of interest to readers without a strong background in those areas, but they will be
most accessible to readers who know some calculus and statistics.

 NIH's Rule of 21 - Part 1. The NIH wanted to restrict the number of grants
that a single investigator could hold, and tried to justify this policy using data
visualizations purported to illustrate decreasing marginal returns of investment in
 a given lab. Here we show that their graphs fail to demonstrate this and explain
where they went wrong.

 NIH's Rule of 21 - Part 2. It gets worse. The entire enterprise of looking at

researcher output as a function of funding received, and then attempting to
optimize the allocation of funding based on these data, is fundamentally flawed.
We explain why.
FAQ

Who are you guys?

We are both college professors at the University of Washington in Seattle.

Carl Bergstrom is a member of the Department of Biology, and Jevin West is a member of
the Information School.

Is this your idea of a joke?

No. This is the website that accompanies a college course entitled "Calling Bullshit".

We taught the course as a one-credit, once-a-week lecture at the University of

Washington during Spring Quarter 2017. Our course syllabus and readings, tools, and case
studies are all available on this website. We are recorded video of the lectures, and have
made the edited videos available here as well. We will be expanding the class to a full
three-credit course in Autumn 2017.

Why are you doing this?

As we explain on our home page, we feel that the world has become over-saturated with
bullshit and we're sick of it. However modest, this course is our attempt to fight back.

We have a civic motivation as well. It's not a matter of left- or right-wing ideology; both
sides of the aisle have proven themselves facile at creating and spreading bullshit. Rather
(and at the risk of grandiose language) adequate bullshit detection strikes us as essential
to the survival of liberal democracy. Democracy has always relied on a critically-thinking
electorate, but never has this been more important than in the current age of false news
and international interference in the electoral process via propaganda disseminated over
social media. Mark Galeotti's December 2016 editorial in The New York Timessummarized
America best defense against Russian "information warfare":

"Instead of trying to combat each leak directly, the United States government should teach
the public to tell when they are being manipulated. Via schools and nongovernmental
organizations and public service campaigns, Americans should be taught the basic skills
necessary to be savvy media consumers, from how to fact-check news articles to how
pictures can lie."

We could not agree more.

So is this some sort of swipe at the Trump administration?

No. We began developing this course in 2015 in response to our frustrations with the
credulity of the scientific and popular presses in reporting research results. While the
course may seem particularly timely today, we are not out to comment on the current
political situation in the United States and around the world. Rather, we feel that in a
democracy everyone will all be better off if people can see through the bullshit coming
from all sides. You may not agree with us about the optimal size of government or the
appropriate degree of US involvement in global affairs, and we're good with that. We
simply want to help people of all political perspectives resist bullshit, because we are
confident that together all of us can make better collective decisions if we know how to
evaluate the information that comes our way.

What exactly is bullshit anyway?

Surprising as it may seem, there has been considerable scholarly discussion about this
exact question. Unsurprisingly given that scholars like to discuss it, opinions differ.

As a first approximation, we subscribe to the following definition:

Bullshit is language, statistical figures, data graphics, and other forms of presentation
intended to persuade by impressing and overwhelming a reader or listener, with a blatant
disregard for truth and logical coherence.

It's an open question whether the term bullshit also refers to false claims that arise from
innocent mistakes. Whether or not that usage is appropriate, we feel that the verb
phrase calling bullshit definitely applies to falsehoods irrespective of the intentions of the
author or speaker. Some of the examples treated in our case studies fall into this domain.
Even if not bullshit sensu stricto, we can nonetheless call bullshit on them.

In this course, we focus on bullshit as it often appears in the natural and social sciences: in
the form of misleading models and data that drive erroneous conclusions.

I'm a UW student. How can I take this course?

The course will be offered in Autumn 2017 as a three-credit lecture, under the names
INFO 198 and BIOL 106B. The classroom's capacity is 180 students, and the class filled in
matter of minutes once registration opened to graduating seniors. If you didn't manage to
register this term, you can still do all the readings and watch the lecture clips on video. We
intend to teach the course again (hopely at the 200- or 300- level) credit course to a large
audience in Autumn 2018. For the latest information about the course, follow us
on twitter, on facebook, or by joining our mailing list.
I'm not a UW student. Will the course be offered online?

Informally, yes. Our full syllabus is already online. You can find almost all of the readings
on the internet and the few that are not online should be at your local library. We will be
adding course materials, including new case studies and tools-and-tricks articles, as they
become available. We have made video of the lectures freely available on youtube. For
the latest updates on new material, follow us on twitter, on facebook, or by joining
our mailing list.

In the longer-term we may develop an open online course (a MOOC). When and if we do
so, we will endeavor to keep enrollment costs to an absolute minimum.

Can you actually use the word "bullshit" in the title of a college course?

Apparently yes.

Do you really need to use profanity to make your point? Isn't that rather puerile?

For better or for worse, the term bullshit has few exact synonyms in the English
language. Horseshit is similar albeit with a somewhat more venomous connotation. In any
case, this term is no more family-friendly. The best alternative we can think of is the
shorter (and etymologically prior) bull.

One motivation for using the term bullshit is that this is the word employed when the
subject is discussed in the philosophy literature. But let's be honest: we like the fact that
the term is profane. After all, profane language can have a certain rhetorical force. "I wish
to express my reservations about your claim" doesn't have the same impact as "I call
bullshit!"

If you feel that the term bullsh*t is an impediment to your use of the website, we have
developed a "sanitized" version of the site at callingbull.org. There we use the term "bull"
instead of "bullsh*t" and avoid other profanity. Be aware, however, that some of the links
go to papers that use the word bullsh*t or worse.
How could you have omitted Darrell Huff's 1954 book How to Lie with Statistics from your
syllabus?

We acknowledge that Huff's book did a good job of providing a humorous and non-
technical introduction to the perils of statistical reasoning to a 1950s audience.
Unfortunately the casual racism and sexism of the illustrations make it virtually unusable
as a college text today.

Don't you guys know that "big data" is plural?

We disagree. In common usage, the term "big data" refers to the use of, or field of study
involving, very large data sets. Just as "Hydroponics has revolutionized the way we grow
weed", "Big data has revolutionized the way we sell bullshit."

Can you guys call bullshit on [this thing that I don't like]?

The purpose of this website is to teach people how to spot bullshit and refute it. We don't
intend to use it as a platform for calling bullshit on things that we don't like, and we
certainly don't intend to use it as a platform for calling bullshit on things you don't like.

Our case studies are not the most egregious examples of bullshit, nor the ones we most
wish to debunk. Rather, they are chosen to serve a pedagogical purpose, drawing out
particular pitfalls and highlighting appropriate strategies for responding. So read up, think
carefully, and call bullshit yourself.

I'm an instructor. Can I teach this course at my institution or use your materials in my
classroom?

Nothing would please us more. There are only so many students that we can reach first-
hand, so we would be delighted to see others take up the cause. If you use the syllabus or
materials, we have just two small requests.

1. Please acknowledge our efforts in your course materials. Mention our course and
what you have drawn from it. Provide a link to our webpage (or callingbull.org if
you prefer the sanitized version of the url). If you reproduce portions of our text,
indicate the source. Basically, we just ask that you follow appropriate norms of
academic attribution.

2. We would love to hear from you about how you are using these materials. Among
other things, this helps us justify the time and effort that we are putting into the
project. Any comments about what you find works well and what does not would
also be most welcome.
Please do not make copies of our case studies, articles, or other web pages on your own
web server. We view our course materials as works in progress and would like to keep a
single version of record on our server that we can update over time. After all, should we
ever discover that we've inadvertently spread bullshit, we want to be able to clean it up.

Doesn't your course just make matters worse by teaching people how to bullshit more
effectively?

It is true that if one knows how to detect subtle bullshit, one can also create effective
bullshit. As with biological weapons, there is no such thing as purely defensive bullshit
research. And that puts us in a slightly awkward positionBrandolini's Bullshit Asymmetry
Principle. Brandolini's principle dictates that refuting bullshit requires an order of
magnitude more effort than creating it. Unless one believes that good actors are an order
of magnitude more common than bad actors, it might seem that teaching people more
about the dark art of bullshit will only increase the amount of bullshit in the universe.

The problem with this line of reasoning is that it holds Brandolini's principle constant
while changing the bullshit detection and bullshit creation abilities of the populace. We
believe that as more people learn to detect and refute bullshit, Brandolini's ratio will
change. Bullshit is easier to spread and harder to eliminate when people are not expecting
it; it is also harder to eliminate when people don't know how to best refute it. This course
should help on both accounts. In our more optimistic moments, we can even imagine a
future in which the Second Law of Coprodynamics is violated.

What's with all the old art?

Bullshit is by no means a modern invention. Each page on this website features a famous
bullshit artist of yore.

Here on this page, a detail from Michelangelo's 1512 Expulsion from the Garden.
According to Christian theology, all of the pain and suffering (and mortality) that pervades
human life can be traced to the lies that the serpent fed Eve about the fruit of knowledge.

On our syllabus page, Theodoor Rombouts's early 17th century The Denial of Saint Peter.
At the Last Supper, Saint Peter assured Jesus that he would never deny him, but Jesus saw
right through that. By the next morn, Saint Peter had lied three times in denial of the
savior.

On the home page, Rafael's 1511 The School of Athens. Here Socrates is depicted as he
obliterates the arguments of the Sophists, a group of purported scholars who constructed
an entire philosophical school around talking bullshit. (Fortunately, the Sophists are long
gone and no other school of philosophy would venture to lay its foundations on the same
effluent base. )

On our contact page, Botticelli's 1494 Calumny of Apelles. King Midas looks down on a
man falsely accused by figures representing Slander, Fraud, Ignorance, Suspicion, and
Conspiracy.

On our about page, Nicholas Regnier's 1620 The Fortune Teller. One might imagine this
fortune-teller makes her living through her ability to deceive the willing.

On our exercises page, Nicholas Poussin's 1654 Death of Sapphira. According to Acts 5 of
the New Testament, Sapphira and her husband Ananias lied to Peter about holding back
some of their money, and were struck dead for this. Nevermind that Peter himself earned
a place on our syllabus page for denying Jesus three times before the cock crowed.

On our lecture videos page, Dosso Dossi's 1524 Jupiter, Mercury, and Virtue . Mercury was
the trickster of the Roman Dii Consentes.

Our case studies pages present many different renditions of a pioneering bullshit artist,
Odysseus, and the foes he defeated . Not only does Odysseus best multiple foes through
trickery and deceit; he is the original unreliable narrator. Here’s a leader who won a huge
war and sacked a wealthy city, yet somehow managed to return home a decade later,
impoverished and without any of his ships or men. The way he tells the tale, they were
lost to unspeakable danger while he alone survived through his bravery, cunning, and
heroism. Maybe the voyage really did go down the way he reports, but it sure is hard to
verify given that no one survived to question his claims.

The header images on our tools and tricks pages offer tribute to those philosophers and
scholars who over the centuries have sought to cut through the lies, ignorance, and
superstition bring the light of knowledge to the world.
Exercises

Assignment 1: A bullshit inventory

There's a lot of bullshit out there — but how much exactly, and of what form?

The purpose of our first assignment is for you explore this question by taking a "bullshit
inventory" of all of the bullshit you encounter of the course of one week. The idea is to
make note of each bit of bullshit that see or hear, and to record some information about
it. While the assignment is open-ended and we want you to be creative, you might
consider keeping track of:

1. Bullshit that you are exposed to

2. Bullshit that you produce yourself

3. Bullshit that you debunk or try to debunk

Many people try to be carbon-neutral in their daily living, and we admire that, but are you
bullshit neutral as well? That is, do you debunk as much bullshit as you produce? It's not
so easy to do. Are you at least bullshit-average, creating no more bullshit than you are
exposed to.

To do this assignment, you'll have to figure out a few things, including:

 What constitutes bullshit, in your mind? Does a deliberate lie count, for example?
Does the bullshitter have to be aware that he or she is bullshitting? And so on.

 How are you going to measure this bullshit? Are all instances of bullshit—a 10 page
term paper or a 140 character tweet—equivalent? Should we measure bullshit in
terms of the amount of time that one spends dealing with it? Are some kinds of
bullshit more significant than others?

 How are you going to track your daily encounters with bullshit? There's so much
bullshit out there that you may not want to sit down and try to remember it all
each evening. Maybe it would work better to have some way of recording your
observations as you go through your day.

 How much information do you feel comfortable providing? Personally, we'd rather
not write things down at the level of "9:15 AM. Ran into Dr. Smith in the elevator
and had to listen to him brag about his recent appointment as vice chair of the
 university's Interdepartmental Committee for the Reduction of Unnecessary
Administration." You never know when these things might get out (or when you
might choose to share them; see below). So how will you anonymize the data
enough that it won't get you in trouble on the down the road?

You can report or visualize the data however you see fit. Here's one example that Carl and
his daughter put together using made-up data. (You can tell it's made up, because Carl's
department chair would never bullshit him.)
For inspiration, you might wish to take a look at the Dear Data project. This site has a number of
cool examples of lo-fi visualizations of data about everyday life. While we like these, feel free to be
creative and display your data however you like, with an interactive applet, using data
visualization software, in powerpoint, with a stack of 3x5 cards, in song, or in any way that catches
your fancy.

Readers on the internet, please join us in doing this assignment. We'd love to see what you come
up with. You can send us your work by email to bullsht.course@gmail.com, or tweet it with the
hashtag #BullshitInventory to @callin_bull.

An extra incentive: If we get some really good designs from members of the public who are
following along, there is a chance of publishing them (with permission and attribution) in a very
high profile news venue.

Have fun!