Taking Aim at Biased Algorithms

Author(s): Rachel Levy

Source: Math Horizons, Vol. 25, No. 1 (September 2017), pp. 5-7
Published by: Mathematical Association of America
Stable URL: http://www.jstor.org/stable/10.4169/mathhorizons.25.1.5
Accessed: 09-11-2017 20:07 UTC

JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide
range of content in a trusted digital archive. We use information technology and tools to increase productivity and
facilitate new forms of scholarship. For more information about JSTOR, please contact support@jstor.org.

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at
http://about.jstor.org/terms

Mathematical Association of America is collaborating with JSTOR to digitize, preserve and

extend access to Math Horizons

This content downloaded from 128.122.230.132 on Thu, 09 Nov 2017 20:07:38 UTC
All use subject to http://about.jstor.org/terms
 Taking Aim
at Biased Algorithms
C
Rachel Levy
athy O’Neil is a mathematician, data
scientist, author, activist, and blogger (at
mathbabe.org). She has worked in higher
education, on Wall Street, and for various
start-up companies.
Her 2016 book Weapons of Math Destruction aims to
help the public better understand the impact that algo-
rithms have on everyone’s life. Many students trained in
the mathematical sciences take jobs directly addressed
in the book, such as in data science and finance. O’Neil
raises new issues about the impact of mathematicians’
work on society. As topics in the news heighten public
awareness and concern about the power and role of
Adam Morganstern
algorithms, mathematicians have an opportunity to Cathy O’Neil.
provide new tools to foster transparency, equity, and
benevolence. are somehow dealing with objective truth, but every
I chatted with Cathy O’Neil in January. This inter- step along the way requires human intervention. This
view has been edited for length and clarity. misconception that algorithms are somehow revealing
objective truth is the most important clarification that I
Rachel Levy: What are the main take-home messages of
want to make.
Weapons of Math Destruction?
In fact, it is the opposite. These algorithms are often
Cathy O’Neil: My book describes the way mathemat-
propagating historical biases and past mistakes. It is a
ics and the trust of mathematics is used against the
particular shame when the brand of mathematics is be-
public. The public trusts and fears mathematics. So
ing deployed to protect something that is fundamentally
when marketers, salespeople, or even scientists represent
immoral and corrupt.
algorithms as mathematical and when they represent
machine learning as sophisticated and opaque, people do RL: What are some examples of weapons of math de-
not question them. Therefore, those automated systems struction (WMDs)?
remain almost entirely unaccountable, and bad things CO: Weapons of math destruction are algorithms that
can happen. Algorithms are not mathematics. They are important, opaque, and destructive. There are ex-
have mathematical attributes, but they are ultimately amples all across normal everyday life for Americans.
working with human-curated data on a human-curated Examples include assessments for teachers that
agenda. are happening in many large cities. They are called
We—the developers—embed our biases in the algo- “value-added models” and have been addressed by the
rithms. Not to mention we have chosen what data to use American Statistical Society. Judges use predictive po-
to train our algorithms, and the data we have chosen is licing and recidivism-risk algorithms for parole, bail, and
a social construction. There’s a belief that algorithms sentencing. Political microtargeting algorithms inform

www.maa.org/mathhorizons : : Math Horizons : : September 2017 5

This content downloaded from 128.122.230.132 on Thu, 09 Nov 2017 20:07:38 UTC
All use subject to http://about.jstor.org/terms

mh-05-07-levy-final.indd 1 9/29/17 1:00 PM

 the kind of information we get from candidates. And
some kinds of tailored advertising, which is an area I
worked in until I realized that I wanted no part of it.
The same technology that shows me luxury yarn and
knows that I am very vulnerable to such advertisements
is used by predatory for-profit colleges and payday lend-
ers to target single black mothers who want a better life
for their children.
These algorithms don’t affect everyone equally. The
working class and working poor are affected more often
by these algorithms than highly educated and well-off
people. Well-off people can get through their lives fairly
unscathed by the kinds of corporate and government
surveillance that I worry about in the book. I urge
people to look at algorithms and weapons of math de-
struction through the lens of class and race.
The people building and deploying the algorithms
are often well intentioned but naive. They are technolo-
gists—sometimes mathematicians, computer scientists,
or statisticians—and they have an arm’s-length perspec-
tive on the targets of the algorithms that they build.
They do not acknowledge or understand the kind of
effects they are creating.
RL: Can you name some algorithms that are designed and
used in ways that are relatively fair and just? What quali-
ties make them so?
CO: One of my favorite examples is sports. The amount
of sports data is blossoming. The data comes from
the games, and the games are on public view, such as
national television. The top radio shows serve the pur-
pose of cleaning the data—they’ll talk endlessly about
whether a play should have been an error or a base hit.
We have transparency.
One of the most difficult aspects of building algorithms
fairly is to have a well-defined and agreed-upon defini-
tion of success. In sports, success is clearly defined: A
sports team wants to win games. You could argue they
want to make as much profit as possible, but there is at
least a correlation between the two. If you look under
the covers of many of the algorithms that are destruc-
tive or ineffective, you’ll see that their definition of suc-
cess is ambiguous, incorrect, or so hard to measure that
unintended consequences abound.
In the book I talk at length about the U.S. News &
World Report college-ranking model. For the last 30
years, colleges have been trying to get ranked higher,
and the definition of success in this model is an arbi-
trary set of attributes that do not include important
6 September 2017 : : Math Horizons : : www.maa.org/mathhorizons
This content downloaded from 128.122.230.132 on Thu, 09 Nov 2017 20:07:38 UTC
All use subject to http://about.jstor.org/terms
mh-05-07-levy-final.indd 2
“ Algorithms are not
mathematics. They have
mathematical attributes, but
they are ultimately working
with human-curated data on a
”
human-curated agenda.
factors such as tuition cost. The model has spun off an
industry of side effects, including growing administra-
tions and unnecessary expenditures, not to mention
rising tuitions. That’s just one example of many that
illustrates how important it is to have a good definition
of success.
The algorithms that are both important and nonde-
structive are the ones that help the people who are the
least lucky or are suffering the most. There are colleges
using algorithms to find struggling students, especially
struggling freshmen, and connecting them with advising
support.
It is important that the people who are struggling
the most and raising the largest number of red flags are
not being punished. There is an example of this that I
blogged about—Mount St. Mary’s College. They seemed
to be doing this identification and expelling students
before the U.S. News survey was due.
You asked for a good algorithm, and I came up with
one that is being used to do good but could easily be
used to do harm. Therein lies the conundrum. It really
depends on the usage. A good rule of thumb is to ask:
“Is it helping or punishing the people who are the worst
off ?”
Algorithms make moral decisions all the time, and
the people programming the algorithms should not be
the ones making those moral decisions. Right now, we
are conflating the job of building the algorithm with
answering the moral questions.
RL: What important things have happened since the
hardcover version of your book was published that we can
read about in the upcoming paperback version?
CO: The run-up to the election largely happened after
I finalized my book. So the biggest gaping hole is the
way propaganda, fake news, hoaxes, and the Facebook
algorithm have further destroyed our concept of truth
and our efforts at democracy. I talk about the Facebook
algorithm and political microtargeting in the hardcover
version, but I don’t come out and say that the Facebook
9/29/17 1:00 PM
 algorithm is a WMD because I didn’t have enough
evidence for the destruction part. I think we do now, so
I will update that.
I discuss political polling and predictive polling,
which are total wastes of time and destructive to the
kinds of information we should be interested in learn-
ing. They also change people’s voting patterns in bad
ways. For example, if you think that your candidate is
a shoo-in or has no chance, you might not vote. So it
can suppress votes on election day. In addition, be-
cause early polling of unknown candidates is so dis-
couraging and considered evidence that they will not
make it, we continue to have a very tightly controlled
two-party system.
RL: What are some of the main concerns that people
have raised about your book?
CO: I haven’t gotten much negative feedback, but I
have a thick skin about trollish comments on Amazon.
The biggest complaint is that the book is political. I’m
not a journalist. I’m more like an activist, so I don’t
take that badly. I don’t want it to be so vitriolic that
people who differ politically from me can’t read it. So
far, the consensus has been that it is readable, even to
people who disagree with me.
I have gotten overwhelmingly positive feedback, espe-
cially from young technical people considering jobs in
data science. They are emailing and offering to work for
my company because they want to do good with their
data science skills. It is very pleasant to see that there is
a burgeoning society of good nerds. There is not much
I’d rather do than hang out with people like that.
RL: What is the goal of your new company?
CO: My company is called O’Neil Risk Consulting and
Algorithmic Auditing (ORCAA). An organization or a
company like a police department that is currently using
an algorithm might worry that it is unfair. They may
not intend it to, but their algorithm might be racist,
sexist, or have disparate impact. We could audit that
algorithm for fairness, for legality, or even for meaning.
Our goal is to come up with a standard for evidence
in the field of algorithms. We were “told trust me, it is
a machine learning algorithm and it works.” That era
is over. We need to put the science into data science
and have a standard for what it means for an algo-
rithm to work well and how to measure bias.
Is a résumé-filtering algorithm filtering out people
from some groups unfairly? Regulators may not be on
top of this question, but private lawsuits might put
www.maa.org/mathhorizons : : Math Horizons : : September 2017 7
This content downloaded from 128.122.230.132 on Thu, 09 Nov 2017 20:07:38 UTC
All use subject to http://about.jstor.org/terms
mh-05-07-levy-final.indd 3
pressure on companies or or-
ganizations to deal with these
issues soon.
RL: My students are calling for
more focus on the impact of our
work on society. Are there ways
that your book could be used in
college courses?
CO: Hey, that’s great. It could be used for discussion—
hopefully also with case studies so there could be some
technical assignments and projects. A large obstacle is
that the WMD algorithms I discuss are proprietary and
not discoverable through the Freedom of Information
Act. So you would have to study some kind of replica of
the algorithms and see how the attributes come through
in those replicas. You could assign different groups the
same problem and see how different their answers are.
Modeling is a craft, not a science. It is almost never
proof-worthy.
It is challenging to get data sets that are anything like
the ones I talk about in my books, such as the data on
people who have applied for a job or committed a crime,
because with human data there are privacy concerns.
RL: What are you hoping to accomplish in the next five
years?
CO: I want there to be an academic field that deals with
algorithm accountability. It is interdisciplinary and in-
cludes the mathematical sciences, computer science, soci-
ology, information science, and law. Existing data science
institutes are not yet addressing basic questions of policy,
ethics, discrimination, fairness, meaning, interpretability,
transparency, and accountability in algorithms. I want
the field up-front and center. It could be a department
or at least tenure lines, journals, and conferences—a real
community of people tackling the question from different
angles.
Algorithms are not going away. Any part of our
culture that makes decisions in structured ways will be
replaced by algorithms, so this is extremely urgent. It
is the difference between being able to push back about
unfair decisions and living in an Orwellian machine
where decisions are made for us. n
Rachel Levy is a professor of mathematics and associate
dean for faculty development at Harvey Mudd College.
She serves as vice president for education for SIAM and
received the MAA’s Alder Award for teaching.
http://dx.doi.org/10.4169/mathhorizons.25.1.5
9/29/17 1:00 PM