Mariya Lincoln

March 2023

Algorithmic Bias
How Algorithms Can be Biased – and How to Make Them More Equitable

Bias: Implicit and Explicit

The word bias implies an unreasoned and unfair distortion of judgment, against (or in favor of) a person
or thing (Merriman-Webster, Incorporated), and has a generally negative connotation. In humans, bias
leads us to pre-judge someone or something. It shows a distorted view of reality and interferes with our
ability to understand the world around us. Because of this, biased decisions and actions can violate the
basic ethical principle of fairness. Here we examine how algorithms can have bias and why and how to
work on getting rid of this bias, in favor of objectivity, fairness, and impartiality.

When we think of bias or prejudice, often what comes to mind are blatant violations of fairness, such as
outright racism or sexism. Bias can be both explicit or implicit, however, and it is often the implicit kind
that is hardest to discover and weed out, because it is not purposeful, but automatic and involuntary. It
can go against what we profess to believe and is dangerous because it is generally invisible, both in
humans and in algorithms. The belief that data offers the impartial truth because “numbers don’t lie”
has largely been accepted to be false by the scientific community, but it is more difficult for us to judge
on a case-by-case basis when data or algorithms offer an incomplete or inaccurate picture of reality.

We can be misled by data due to factors such as small sample size, confounding correlation and
causation, selection bias, and choice of visualization techniques or scale (Kim, 2013), among others.
Perhaps we can see the error in relying on a study of only 10 people to test the effectiveness of a
cosmetic, or that having 10,000 social media followers doesn’t mean a lot because these can be
purchased for pennies. But when it comes to finding bias in algorithms, it becomes more murky.
Algorithmic bias can be looked as a distortion, and thus a less accurate portrayal of reality, which is
important in a scientific and business context. It can also be looked at through the human lens of
fairness, and this lens is important in a society where the value of equality is professed at the level of
individuals, society, and the legal system. However we choose to examine algorithmic bias, it exists, and I
believe it will serve us all to strive to find it, better understand it, and work to remove it.

Three Layers of Algorithmic Bias

There are many lenses through which to examine the biases in algorithms, and the three-layered
approach proposed here is simply a framework to help structure insights and motivate action. The lens
or framework is secondary to the discussion itself, as bringing the issue to light and to the conversation is
the first step in working to solve the problem of algorithmic bias. Though it comes from a 1980s cartoon,
the adage “[K]nowing is half the battle” is now part of the American ethos and can be helpful to keep in
mind. We can argue that this quote is a fallacy (Kristal & Santos, 2021), but knowledge is fundamental
and a nontrivial step in dealing with algorithmic bias. Bringing biases to light may be the hardest step in
the process, but once this step is done, bias moves from implicit to explicit and can therefore be
examined and dealt with appropriately.

The lens I propose here is that we can examine algorithmic bias in three layers. These layers are not
mutually exclusive, but they can be examined separately. Then we can discover how bias can creep in,
and how to work to remove it in each layer: that of the data itself, the algorithms built on the data, and
the humans who are involved in the process.

Data
Algorithms run on data. The saying “garbage in, garbage out” refers to the understanding that if we feed
inaccurate or misrepresentational data into our algorithms, they will dutifully spit out conclusions that
are likewise inaccurate.

How can data be biased if it has no malicious intent? While intent may be important in a legal context,
we can easily make an argument that data can be biased without intent, if we focus on the definition of
bias as meaning distorted or misrepresentative. Biased data can very often have the effect of disparate
impact, but at the level of data, we focus on how accurately the data captures and reflects reality. Data
can be biased by asking the wrong questions, or asking them in a leading way. Who we ask and how we
ask it also matters, which we can refer to as selection or sampling bias. These errors reflect implicit
human bias, but additional bias can be introduced in the data when steps such as tagging or categorizing
are done by humans (annotator bias). These are just a few of many ways that bias can exist in datasets. A
more comprehensive list can be found in the Appendix, and I will mention just one more important one
here, which is the introduction of bias by use of proxies.

Proxies serve an important role in data analysis because they offer a way to see and measure things that
cannot be measured directly, or when to measure something directly would be cost prohibitive. We use
the proxy of actions or professed beliefs any time we measure what people believe in, because true
beliefs cannot be directly measured. Collection of data about beliefs is further subject to implicit human
biases, such as being self-serving. In business we use statistics on views, searches, clicks, or purchases as
proxies for what people are truly interested in, and create algorithms based on these measurements that
decide who is shown particular content and advertisements. While proxies are useful and necessary in
data analysis, we need to exercise caution, as proxies are another way data can be inaccurate (and thus,
biased). Proxies are by definition substitutes, and can be poor ones.

Solving issues of bias in the data itself requires extra effort and input at every step of the data collection
and selection process. When collecting and examining data from a certain population, people from the
target population should be brought to the table for open-ended exploratory conversations about how
to best gather the information desired, and what questions to ask. This helps bring to light cultural
differences and leads to better-informed data collection. An example illustrating this process is outlined
later in this paper. Data generally looks backwards and captures past actions or decisions so it’s
especially important to examine critically what historical cultural biases may have made their way into
the data we use as the basis for algorithms.

Researchers have also proposed solutions such as transparency in the data used for training algorithms,
regular audits of this data (Heilwell, 2020), and deciding what questions they want to answer with the
data beforehand (Kearns & Roth, 2020). Legislative oversight such as the FTC’s pending legislation
requiring companies to check their AI systems for bias is something that can assist in making sure these
solutions are given the time and energy they need, and are regularly revisited as necessary when there is
a question of biased data.

Algorithms
Like data, algorithms have no intent of their own. They do not themselves seek to harm one group or
classification of people. Even when they do show bias it is often not due to the intent of a bad actor
programming such bias into the algorithm. Because algorithms have no intent of their own, US law
requires proving disparate treatment or disparate impact to show a violation of the law by algorithmic
decision. While seemingly simple, this violation can be difficult to prove conclusively, as there are
different ways to measure accuracy and fairness- and therefore, impact.

We return to the definition of bias here momentarily as a reminder that it “implies an unreasoned and
unfair distortion of judgment in favor of or against a person or thing (Merriam-Webster, Incorporated)”.
When we attempt to evaluate fairness, we find that what might seem to be something a child could
easily answer is again more complex, due to the varying definitions of fairness and the necessary
tradeoffs with accuracy that the pursuit of fairness entails.

There is more than one way to measure algorithmic fairness (Johndrow & Lum, 2019). One way put
forth to accomplish fairness is to simply exclude protected variables. If we do not want our algorithm to
show racial bias by this definition of fairness, for example, we do not provide it with any data on race.
One problem with measuring fairness in this way is that we often find that other variables have enough
correlation with the omitted ones that the algorithm still shows bias. In the race example, zip code often
is highly correlated with race and could be used as a proxy by the algorithm if not explicitly instructed
not to. Especially in combination with each other, these variables can act as a proxy discriminatory
variables in our algorithms, and when they act in combination it becomes more opaque which variables
are causing the biased results. It’s not unusual that excluding a protected variable can have the
unintended consequence of the algorithm becoming less accurate for the protected class of people
(Kearns & Roth, 2020), also, so this seemingly obvious solution may not be the best one.

Another way to measure algorithmic fairness is to measure predictive accuracy and ensure that it is
equal among all classes of a protected variable. We can measure each variable separately, but to be
truly fair we need to consider the variables in conjunction with each other to ensure there is no bias
against a particular subset of humanity. This gets complicated when we have more than one protected
variable and then have to compare each subset. With so many subsets, it’s more likely one will
experience disparate results, or won’t have a large enough sample size to ensure accurate predictive
results. If we try to measure fairness using predictive accuracy, we also need to agree on measure. It’s
not always easy to decide whether false negatives or false positives are more critical for equality, but we
can’t optimize an algorithm to be equal for both (Kearns & Roth, 2020).

I would argue that an effective measure of whether an algorithm is fair or biased is in its impact. What
matters are results in the real world and we can try to measure and correct for disparate impact as a
way to weed out bias from our algorithms. This may be simply a question of whether false negatives or
false positives have a greater detrimental impact, but asking the question in terms of impact brings a
human dimension as well as a legal one to our consideration and helps it bring the question from the
abstract to the concrete.

This is also a good place to bring up transparency and interpretability. Rebecca Heilweil (2020) argues
that transparency is a first step toward algorithmic accountability. But here again, what seems logical on
the surface becomes harder to implement in practice. Many of the most commonly utilized machine
learning algorithms are simple, in that they maximize a clearly stated objective, and are fed a dataset we
can examine and control. But the relationships a computer finds between variables may not be able to
be untangled or explained. When what goes in is clear and well-defined but how the algorithm arrives at
its conclusions is opaque, that doesn’t necessarily mean the algorithm fails in interpretability, though.
One way to increase the interpretability of these models is to adjust inputs one at a time and evaluate
the results to better understand how the algorithm arrives at its conclusions (Kearns & Roth, 2020).

When we argue for transparency (or taking this further, interpretability) in algorithms, we need to ask
the question of audience. We should ask who needs to be able to see and understand the algorithms,
and to what extent. The general construction of a loan approval algorithm that is clear to a data science
professional may not be understood by someone with limited education and income who is denied a
loan, for example. Kearns & Roth (2020) propose a structure for developing transparency and
interpretability that includes identifying the target population, whether it be those directly affected by
the algorithm or those in charge of applying and overseeing it, and then designing and refining the
algorithms through an iterative and consultative process with this audience. This takes more effort, time
and resources, but can help in building more transparent algorithms and ones that the target audience
and the general public can have greater trust in. The solutions to bias in algorithms lie largely in
stakeholders (including both those creating the algorithms and those most affected by them) coming to
agreement on transparency, accountability, and evaluation methods.

Princeton professor and author Ruha Benjamin recommends using “glitches”, or places where bias
shows up, not simply as factors to be tweaked and solved in the algorithm in question, but as a way to
illuminate underlying flaws in data collection and algorithmic treatment and to therefore advance our
collective learning more quickly. Benjamin also argues for the public (that means you, and me as well) to
demand improvements. Removing algorithmic bias is a social justice issue and history has shown us that
the way to create lasting change is to shine a light on injustice and continue to speak out until those in
political and legislative power take notice. This grassroots effort can be informed and reinforced by a
dedicated effort on the part of parents and our educational system to give children a solid foundation in
statistics from a young age, and teach them to always ask questions of the data- to treat it as a partner
but never a master. This leads us to consider the layer of human bias.

Human Bias
A discussion of human bias is unavoidable in discussing bias in data and algorithms, and deserves its own
treatment here. We should recognize that implicit bias is harder to discover and remove than explicit
bias, but both are still dangerous. Human bias comes from many places, including enculturation from our
families of origin and society at large, and from the media. I would argue that the underlying reason for a
bias or prejudice, particularly against a certain race or class of people, is largely due to ignorance.
Growing up without exposure to a diversity of humankind can lead to the kind of distortion of viewpoint
that comes from bias. Marilyn Cavicchia (2015) shares the insight of diversity expert Sharon E. Jones,
that the natural solution to this is contact. When we have regular and positive contact with a diversity of
people, we learn to consciously notice bias and decrease its power, and we learn empathy.

By striving to tackle our own biases, we improve our character and also strengthen our ability to both
notice and avoid bias in data and in algorithms. Avoiding bias by diversity of association can build a more
unified society, but in case we are not quite motivated by altruism, diversity is also good for business
(McKinsey & Company).

In “Disrupting the Gospel of Tech Solutionism to Build Tech Justice”, Byrum & Benjamin (2022) argue that
it’s critical to step back and consider the larger question of where and when algorithms are appropriate
and useful, and not default to the belief that technology will solve our problems. “Tech solutionism”
refers to the unquestioned belief that technology will make things better and solve intrenched problems
that humans have so far been unable to solve. They ask the provocative question, “What if, sometimes,
the answer isn’t technology?” and argue that we should always consider the possibility that technologies
may be simply ill-suited to the complex environments and problems they claim to address.

I would argue that many times the solution lies in the balance and interplay between technology and
humans, and offer this quote as what we should strive for:

“To create just, equitable, and self-determined tech futures that work for everyone, we need to center
and support voices from the communities most impacted by tech’s biases and harms. A more just tech
future requires deep investment in people to make space for visioning and creation, not simply tech
solutions (Byrum & Benjamin, 2022).”  

An Example
I would like to walk through a simple example to illustrate how an algorithm can be biased, and how to
make it more equitable, based on the points offered above. Here the three layers of data, algorithms,
and humans are all present and can be examined, but are interconnected and overlapping.

Let’s take a hypothetical research question of whether speeding tickets can be optimized by algorithm as
an example. For our experiment we obtain a basic dataset of one years’ worth of speeding violations
from a mid-sized town, and whether the tickets were issued to males or females. We first examine the
data to find whether males or females speed more, and can clearly see that males were issued twice as
many tickets as females. While it’s tempting to call the local newspaper with our story, we decide to
further examine the data for potential bias. To do this we gather a focus group that is representative of
the diverse elements of the local population and through it learn that due to various factors, it’s likely
that men simply spend more hours driving than women and this could explain part of the discrepancy.

We then go back to our data we plan to feed into the computer, and are able to factor an approximation
of hours driven into our data, which still shows men being ticketed proportionately more, but the
difference is smaller. We meet again with another focus group and learn that it’s rumored that women
more often get away with a warning while men are issued tickets. This causes us to question the use of
the proxy of speeding tickets ≈ true rate of speeding that underlies our whole analysis. We are able to
obtain data that shows that men are issued proportionately more speeding tickets in traffic stops. But by
this point we understand that traffic stops are not a good proxy for speeding, either, as this too could be
affected by human bias. We opt to use traffic camera data to measure the true rate of speeding.

Next, we look at the algorithm. Finally we are satisfied with the relative accuracy of our data and plug it
into our computer, which generates an algorithm to measure whether speeding tickets can be more
accurately issued. In order to ensure the algorithm improves on what was before an unfair distribution of
traffic tickets, we agree on a definition of what will be considered fair, as measured by parity of impact
among males and females.

Six months into our pilot program, we evaluate the data and see that ticketing does not match what we
understand to be true about speeding based on conversations with our focus groups and with police
officers. We find that although overall the gender makeup of traffic tickets more closely represents the
reality captured by self-reporting of speeding, and is thus overall more fair according to the definition we
agreed on at the outset, we still have one sub-group disproportionately affected- low income males. In
examining this discrepancy, we find another proxy issue. The traffic cameras issue tickets to the
registered car owners, and in in conversation with a focus group of low-income males, we find that
although many own their own cars, they often only have one car per household and share with their
spouse or girlfriend. The algorithm shows bias because it disproportionately affects low-income males in
a negative way.

Even more harmful, the bias of this algorithm can be amplified when the data on ticketing or the
algorithm it is based on is used as the basis for another algorithm, such as how much a car insurance
company charges, or the rate a person from this sub-group is offered on a used car loan. This is how
algorithmic bias is perpetuated and amplified into feedback loops that can cause even more harm.

In our hypothetical situation, we would again enter into discussions regarding how to rectify this
disparate impact, and the cycle would continue, with bias being removed from the data and our
algorithm at each step forward.

The reason for this example is to illustrate the iterative and consultative process that can be incorporated
at the levels of data and of algorithmic design to rid an algorithm of bias. Consideration of human bias is
given attention throughout this example, from the agreement on what constitutes success at the outset,
to the ongoing process of soliciting and incorporating input from members of the affected target group.

Final Thoughts
Tackling algorithmic bias is a huge and multifaceted undertaking. It’s not something we can make a
proclamation about, sign into law and be done with (though creating laws for oversight seems
appropriate and beneficial). It's a process that will continue for as long as the human race does. This
exercise of addressing algorithmic bias through iterative development is already happening at the
grassroots level, in legislation, and in education. Algorithms are here to stay. It is up to us to continue to
educate ourselves so that we can fight bias in ourselves, in our data, and in our algorithms- in order to
create a more just and equitable world, for all.
RESOURCES FOR FURTHER SELF-EDUCATION

For a brief introduction on the topic, here are three articles that touch on bias in data, algorithms and
humans:

DATA: 6 Ways Numbers Can Lie to Us, by Christopher Kim:

https://www.forbes.com/sites/sap/2013/08/08/6-ways-numbers-can-lie-to-us/?sh=6d8ec12c69dc

ALGORITHMS: Why Algorithms Can be Racist and Sexist, by Rebecca Heilweil:

https://www.vox.com/recode/2020/2/18/21121286/algorithms-bias-discrimination-facial-recognition-
transparency

HUMANS: How to Fight Implicit Bias? With Conscious Thought, by Marilyn Cavicchia:

https://www.americanbar.org/groups/bar_services/publications/bar_leader/2015-16/september-
october/how-fight-implicit-bias-conscious-thought-diversity-expert-tells-nabe/

If you want to dig into the topic further, I recommend these books for a general or business audience:

The Ethical Algorithm: The Science of Socially Aware Algorithm Design, by Michael Kearns and Aaron
Roth

Weapons of Math Destruction, by Kathy O’Neil

Race After Technology, by Ruha Benjamin (and resource list: https://www.ruhabenjamin.com/resources)

APPENDIX: TYPES OF BIAS

Here are some types of bias and ways it can creep into our data and algorithms. It’s not an
exhaustive list, but can be a starting place: selection bias, sampling bias, reporting bias,
automation bias, sampling bias, annotator bias, inherited bias, outlier bias, cognitive bias,
confirmation bias, status quo bias, analytics bias, wrong goals, underrepresented groups, faulty
interpretation, bias in assignment, omitted variables, self-serving bias, group attribution bias,
priming bias, selective perception, and experimenter expectations.

References
Benjamin, Ruha. Race After Technology: Abolitionist Tools for the New Jim Code. Medford, MA: Polity,
2019.
Byrum, Greta and Ruha Benjamin. "Disrupting the Gospel of Tech Solutionism to Build Tech Justice."
Stanford Social Innovation Review (2022). Web.

Cavicchia, Marilyn. "How to Fight Implicit Bias? With Conscious Thought, Diversity Expert Tells NABE."
Bar Leader 40.1 (2015). Web.

Heilweil, Rebecca. Why Algorithms Can be Racist and Sexist. 18 February 2020. 2023.
<https://www.vox.com/recode/2020/2/18/21121286/algorithms-bias-discrimination-facial-
recognition-transparency>.

Johndrow, James and Kristian Lum. "An Algorithm for Removing Sensitive Information: Application to
Race-Independent Recidivism Prediction." The Annals of Applied Statistics 13.1 (2019): 189-220.

Kearns, Michael and Aaron Roth. The Ethical Algorithm. New York: Oxford University Press, 2020.

Kim, Christopher. "6 Ways Numbers Can Lie to Us." Forbes August 2013. Web.

Kristal, Ariella and Laurie Santos. "G.I. Joe Phenomena: Understanding the Limits of Metacognitive
Awareness on Debiasing." working paper. Harvard Business School , 2021.

McKinsey & Company. "Diversity Wins: How Inclusion Matters." 2020.

<https://www.mckinsey.com/~/media/McKinsey/Featured%20Insights/Diversity%20and
%20Inclusion/Diversity%20wins%20How%20inclusion%20matters/Diversity-wins-How-inclusion-
matters-vF.pdf>.

Merriam-Webster, Incorporated. Bias. 2023. <https://www.merriam-webster.com/dictionary/bias>.

Momentive. 4 Leading Types of Bias in Research and How to Prevent Them From Impacting Your Survey.
n.d. March 2023. <https://www.surveymonkey.com/mp/dont-let-opinions-sneak-survey-4-ways-
avoid-researcher-bias/>.

Oxford University Press. Google Dictionary. 2023. <https://languages.oup.com/google-dictionary-en/>.

Rowe, Brian. Federal Trade Commission, 2009. web.

<http://www-personal.umich.edu/~rowebr/pdfs/rowe_policegenderbias_sep09.pdf>.