Professional Documents
Culture Documents
ALgorithmic Bias
ALgorithmic Bias
Above sections raise questions about the urgent need of a comprehensive regulatory framework
to tackle the menace of algorithmic bias. As algorithmic is currently involving in making
influential crucial decisions that significantly impact society, and algorithmic biases are
becoming challenges in the path of fair and equitable decision making. The question is how this
bias remaining unidentified. The answer lies in the mechanism of Black Box. So, what is this
Black Box? “In the AI development world, a black box is a solution where users know the inputs
and the final output, but they have no idea what the process is to make the decision.” 1 So it is
important to make each and every step of the process transparent and controllable which is
involved in making the decision. It is not possible to detect such biased decisions until and unless
a mistake is surfaced through its biased decision. Ultimately these biased decisions cause severe
threat to fairness by influencing decisions in ways that reinforce existing inequalities and
discriminations. Algorithmic biases are not only limited to black box but also contributed
through trained data bias i.e., biases which are originated from historical inequalities and
prejudices that were present during the data collection and reflected in the data used to train AI
models, Implicit Assumptions i.e., developers unintentionally incorporate their own prejudices or
presumptions into the AI system.
Before proposing a regulatory measures and frameworks, let’s discuss current regulations in
India, to address this threat. Currently in India there is no enactment which directly deals with
algorithmic bias. But there are regulatory frameworks that can be used indirectly to address cases
of algorithmic bias, like the Information Technology Act of 2000 and the Indian Penal Code,
especially those that deal with discrimination and privacy crimes. Along with, the Personal Data
Protection Bill, which intends to control how personal data is processed.
The absence of particular legislation to control and regulate the algorithmic bias in AI sector
draws significance attention to tackle the issue of algorithmic bias. To address this gap, there is
an urgent need to implement a regulatory framework to tackle the algorithmic bias. Before
implementing a regulatory framework, government should update its current legislation i.e., from
constitution to general statutes like Indian Penal Code and Civil Procedure Code. The
Government of India should update nondiscrimination and civil rights laws to address the unique
challenges posed by the digital space. Many of these laws predate the internet and need
clarification on how they apply to modern grievances encountered online. One such proposed
changes is update in the discriminatory law of India i.e., the protection of civil rights act, 1955
and Articles 14, 15, 16 of the Indian Constitution.
Proposed Regulatory Frameworks will include measures for identifying, mitigating, and
remedying bias in AI systems to ensure fairness, accountability, and transparency. Collaborative
efforts involving government bodies, industry stakeholders, researchers, and civil society
organizations are necessary for developing robust regulatory frameworks that effectively address
the challenges posed by algorithmic bias.
1
https://brighterion.com/explainable-ai-from-black-box-to-transparency/#:~:text=In%20the%20AI%20development
%20world,that%20problems%20start%20to%20surface.
So let’s discuss the proposed regulatory measure:
Regulatory measures
Tackling algorithmic bias is necessary to develop an equitable and fair society. We cannot leave
algorithm to process unregulated in a digital economy. India need regulatory measures that will
ensure benefits to whole society in a inclusivity, not just to select few. Government shall take
below measures to address the issue:
Transparent Data: The data being processed in the algorithm shall be mandatorily transparent,
and the processing of the data in the algorithmic decision-making shall also need to be
transparent and open to scrutiny. An Algorithm cannot process through black boxes with secrets
inside, which later can cause bias and discrimination. Knowing what data is processed in the
system will help us root out bias and ensure it operates fairly.
Independent Oversight: Companies should not regulate themselves through their internal
management. Instead, regulatory authority approved by the government should be established at
various places in India based on requirement levels. Such independent regulatory bodies are
needed to set standards, check for compliance, and provide the public with straightforward
explanations about how these systems work.
Diversity in Development: Teams responsible for developing and deploying the algorithms in
the system shall be mandatorily comprised of diverse backgrounds. Different viewpoints,
experiences, and backgrounds lead to better, more inclusive technology for everyone. The
government shall mandate diversity in the algorithm development team of an algorithm.
Impact Assessments: Serious and honest evaluations should be made before a significant
algorithmic system goes live. What might go wrong? Who will it help, and who might it
disadvantage? These are the questions authorities need to ask before it is too late. The
government shall establish a proper authority to verify and authorize the use of such an
algorithm. Such authority will measure the developed algorithm's impact and try to mitigate any
discrimination regarding decision outcomes.
After taking all these measures, below mentioned is the proposed regulatory framework to
mitigate and minimize the algorithmic biased outcome.
Regulatory Framework:
a. Preventive Stage:
From the title of the stage the purpose is obvious to prevent the existence of biased algorithms
and actions or decisions made based on them. Preventive Stage is oriented at the following:
Bias assessment: Before utilization, each algorithm shall have a proper assessment for the
negative consequences and sources. It shall be made at different levels including the way the
algorithm was developed, the data it was trained on and the practice environments. The training
data shall be diverse and should reflect all the categories that shall be used by the algorithm.
Regular audit of the sources of data for an algorithm shall be done to check their appropriateness
or diversity.
Review of the algorithm design: Algorithms shall be designed with fairness and shall have bias
attenuating methods in all the levels of the algorithmic decision-making.
Before using the algorithm for the intended purpose, it shall undergo rigorous testing before
deployment by simulation and other regulatory means to track any disparities.
b. Accountability Phase:
The second phase ‘– accountability ‘– focuses on ensuring that the involved entities are held
responsible for the impact of their algorithmic decisions. The key measures include: clear
documentation, which implies that the design, development process, deployment decisions, and
bias prevention activities be thoroughly documented and be available for further audits and
regulatory examinations, the latter one should be a part of required regulatory checks; regulatory
audits, implying audits from time to time by the special independent regulatory bodies to
measure compliance with bias prevention standards and the effectiveness of employed
algorithms; transparency requirements, which imply the need to disclose the entire process of
algorithmic decision-making to the affected stakeholders; legal accountability, meaning that the
involved organizations should be held responsible for biased decisions of their algorithms with
clear penalties and actions to be made to victims being required.
c. Mandatory Annual Review to Ensure Continuous Improvement:
Annual review promotes continual improvement and adaptation to new challenges and risks. It
involves the following Annual bias audits, requiring audits to be performed to check for new and
persistent biases in the system and any signs of deviation from the expected performance
standards. Update and retrain algorithms, regular updates based on new technologies, audit
findings or societal norms. The training also applies when the dataset used for training in the past
changes significantly. Structured feedback mechanisms, this ensures that users, consumers, or
affected groups have a proper channel through which perceived biases towards a particular end
may be reported. Public reporting on audit results, changes made to the system, and the
consequences realized as a result of such amendments are encouraged or required. Publicizing
promotes organizational accountability.
European Union (EU): The General Data Protection Regulation (GDPR) grants
individuals’ significant control over personal data.
United States (US): The US lacks a comprehensive regulatory framework, but individual
states have begun enacting laws to address specific concerns like bias in facial
recognition.
Singapore: The Personal Data Protection Commission (PDPC) has issued guidelines on
fairness and accountability in AI.
Notes:
Several countries are grappling with algorithmic bias and developing their own
regulatory frameworks. The EU's GDPR grants strong data protection rights to
individuals. The US has a patchwork of state-level regulations. Singapore has issued
guidelines promoting fairness and accountability in AI development. While India can
learn from these examples, it must tailor its regulations to address its unique social and
cultural context.
These biases may show up at various phases of the algorithm's creation and application. An
algorithm may pick up and reinforce biases if the data it was trained on is prejudiced or
represents social injustices and prejudices. Algorithmic bias can also arise from biased design
and implementation choices, which inadvertently introduce biases into the algorithm.
Moreover, the absence of diversity in the design team might impede the evaluation of the
possible effects of algorithms on various user segments. Biases may also surface at the
implementation stage, when discriminatory results may result from coding choices or setups.
Imagine a google review of a restaurant or that of a Zomato review on its platform. A raise in a
single star can increase the chance of the restaurant selling more of tis products and having
higher business. Similar to how a simple half-star improvement on a Yelp rating, for instance,
results in a 30-49% higher likelihood of selling out the seats for a restaurant (Anderson and
Magruder 2012). While rating algorithms are influential, little about how they work is made
public. These algorithms’ internal pro-cesses, and sometimes their inputs, are usually housed in
black boxes, both to protect intellectual property and to pre-vent reviewers from gaming business
ratings. Due to their influence added to their obscurity, they raise high concerns regarding
potential bias.
Australian Uber drivers accused the company of slowly decreasing their ratings to suspend them
and then charge higher com-missions to be re instated. The president of the Ride Share Drivers’
Association of Australia noted that “the lack of transparency makes it entirely possible for Uber
to manipulate the ratings” (Tucker 2016). Not only review algorithms but rather in many other
fields as well, such issues are arising. A tool developed by Amazon to streamline the hiring
process was found to be biased against female applicants and the facial recognition softwares of
major companies show less accurate results for persons of color than whites. Many such
examples will be further discussed for us to understand the growing concern with regards to
algorithmic bias.
The detriments of algorithmic bias and its threats are multifaceted and can have significant
negative impacts on individuals and society. Here are some key points based on the provided
sources:
2. Perception and Demand: The integration of AI in professions and services can influence
how people perceive and value them. People may have lower demand for services that
incorporate AI, impacting professionals' expertise and the demand for their services. This
can further perpetuate inequality, especially when biases intersect with marginalized
groups. To further understand this topic we may take an example of an hypothetical
situation where a certain group of people avoid visiting AI powered facilities as the
algorithm is not trained in accordance with their preferences and further isolate themself
from the AI.
3. Inequality Propagation: Algorithmic bias can perpetuate existing inequalities and even
create new disparities. The interplay of technological, supply-side, and demand-side
forces can create a cycle of inequality where certain groups are consistently
disadvantaged, leading to a feedback loop that reinforces disparities. This is in consistent
with the Garbage in Garbage out model, as to if there is inequality in the provided data
set then the algorithm shall propagate the same inequalities only.
Algorithmic bias has huge potential to exacerbate issues of diversity, inclusion, and
marginalization in the workplace and beyond.3 AI systems can amplify, perpetuate, or exacerbate
inequitable outcomes, leading to greater inequality and increased marginalization, especially for
workers in the Global South.
This is because algorithms can reflect and amplify the biases present in the data used to train
them. For example, an AI recruiting tool that favors resumes similar to those of a company's
current (predominantly male) employees will inherently discriminate against women and other
underrepresented groups.
2
Simon Friis and James Riley, 'Eliminating Algorithmic Bias Is Just the Beginning of Equitable AI' (Harvard Business Review,
29 September 2023) https://hbr.org/2023/09/eliminating-algorithmic-bias-is-just-the-beginning-of-equitable-ai accessed 28 April
2024
3
University of Cambridge Judge Business School. (2023). The Dark Side of AI: Algorithmic Bias and Global Inequality.
University of Cambridge Judge Business School. https://www.jbs.cam.ac.uk/2023/the-dark-side-of-ai-algorithmic-bias-and-
global-inequality/
Policymakers and organizations must proactively address these risks to ensure AI benefits
society equitably. Failing to do so could lead to the perpetuation of institutional racism and other
forms of discrimination, with far-reaching consequences.
Employment is one of the most prevalent places for bias to appear in modern life. Despite
advancements over the last few decades, women continue to be underrepresented in STEM fields
(science, technology, engineering and mathematics). According to the Global Gender Gap
Report, women accounted for less than thirty percent of STEM occupations across 146 nations in
2023.4
Not only can artificial intelligence reflect gender bias, but there are also several cases where
artificial intelligence displayed bias based on race. One such example is the Correctional
Offender Management Profiling for Alternative Sanctions (COMPAS), which attempted to
forecast the possibility of US convicts re-offending. In 2016, ProPublica studied COMPAS and
discovered that it had algorithmic bias. It was substantially more likely to classify black
offenders as at-risk of reoffending than white defendants.
4
‘Women and STEM: The Inexplicable Gap between Education and Workforce Participation’ (orfonline.org)
<https://www.orfonline.org/expert-speak/women-and-stem-the-inexplicable-gap-between-education-and-workforce-
participation> accessed 28 April 2024.
5
Carnegie Mellon University, ‘Amazon Scraps Secret AI Recruiting Engine That Showed Biases Against Women -
Machine Learning - CMU - Carnegie Mellon University’ (Machine Learning | Carnegie Mellon University,
$dateFormat) <https://www.ml.cmu.edu//news/news-archive/2016-2020/2018/october/amazon-scraps-secret-
artificial-intelligence-recruiting-engine-that-showed-biases-against-women.html> accessed 28 April 2024.
While COMPAS properly predicted reoffending rates of roughly 60% for both black and white
offenders, its results revealed biases. It misclassified nearly twice as many black defendants
(45%) as greater risk as white offenders (23%). Furthermore, it incorrectly classified more white
offenders as low risk, who later reoffended (48% of white defendants vs. 28% of black
defendants). Even after controlling for other characteristics like past offenses, age, and gender,
COMPAS assessed black defendants as higher risk, with a 77% higher likelihood than white
defendants.6
However, this assumption did not take into consideration the differences in how black and white
people pay for healthcare. A 2019 research in Science describes how black individuals are more
willing to pay for aggressive treatments such as emergency hospital visits, despite symptoms of
uncontrolled diseases.7
As a result, black patients got lower risk scores than their white counterparts and were classified
similarly to healthier white persons in terms of expenses. As a result, black patients were less
likely than white patients with identical requirements to qualify for additional care. The
algorithm's bias reinforced existing inequities in healthcare access and quality depending on race.
Microsoft's effort to display a chatbot on the network had sparked controversy. They launched
Tay in 2016 with the intention of learning from informal, casual exchanges with other app
users.8 Initially, Microsoft said that "relevant public data" will be "modelled, cleaned, and
filtered".
6
Jeff Larson Mattu Julia Angwin,Lauren Kirchner,Surya, ‘How We Analyzed the COMPAS Recidivism Algorithm’
(ProPublica) <https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm> accessed 28
April 2024.
7
Ziad Obermeyer and others, ‘Dissecting Racial Bias in an Algorithm Used to Manage the Health of Populations’
(2019) 366 Science 447.
8
‘Twitter Taught Microsoft’s AI Chatbot to Be a Racist Asshole in Less than a Day - The Verge’
<https://www.theverge.com/2016/3/24/11297050/tay-microsoft-chatbot-racist> accessed 28 April 2024.
However, within 24 hours, the
chatbot began spreading racist,
transphobic, and antisemitic tweets.
It learnt discriminating behaviour
through interactions with people,
many of whom sent it offensive
remarks. This instance demonstrates
how AI systems may swiftly adopt
and disseminate biases when trained
on raw user data, posing severe
ethical questions regarding the usage
of such technology.
9
Siladitya Ray, ‘Google CEO Says Gemini AI’s “Unacceptable” Responses Offended Users And Showed Bias’
(Forbes) <https://www.forbes.com/sites/siladityaray/2024/02/28/google-ceo-says-gemini-ais-unacceptable-
responses-offended-users-and-showed-bias/> accessed 28 April 2024.
10
Adam Hadhazy for the Office of Engineering Communications on April 18, 2017, and noon, ‘Biased Bots:
Artificial-Intelligence Systems Echo Human Prejudices’ (Princeton University)
<https://www.princeton.edu/news/2017/04/18/biased-bots-artificial-intelligence-systems-echo-human-prejudices>
accessed 28 April 2024.
in India.11 The report raised concerns about the website's algorithms, citing instances where a
Brahmin user's profile was not offered scheduled caste matches unless preferences were
adjusted, potentially violating equality laws. Shaadi.com denies any in-built bias, stating it's not
in violation of any act. This incident underscores the challenges of addressing caste-based
discrimination in online platforms and the importance of ensuring fairness and equality in
algorithmic decision-making processes.
When it comes to bias in AI, the examples all have one thing in common: data. AI learns bias
from the data it is trained on, therefore researchers must be extremely careful about how they
collect and use that data.
These real-life examples vividly illustrate the pervasive issue of algorithmic bias in various
domains, from employment to healthcare, and from social media platforms to matchmaking
websites. They underscore the critical need for vigilance and accountability in the design and
deployment of automated decision-making systems. Whether it's Amazon's biased recruiting
system, COMPAS's racial disparities in predicting reoffending rates, or the discriminatory
practices of digital safety applications and matchmaking platforms, algorithmic biases can
perpetuate inequality and reinforce societal prejudices. Addressing these biases requires a
concerted effort from both developers and policymakers to ensure that AI systems are fair,
transparent, and equitable for all.
11
‘Shaadi.Com under Fire for SC Discrimination: UK News Report’ (India Today, 3 February 2020)
<https://www.indiatoday.in/india/story/shaadi-dot-com-under-fire-alleged-sc-discrimination-uk-news-report-
1642759-2020-02-03> accessed 1 May 2024.
12
Yoshita Sood, ‘Addressing Algorithmic Bias in India: Ethical Implications and Pitfalls’ [2023] SSRN Electronic
Journal <https://www.ssrn.com/abstract=4466681> accessed 2 May 2024.
6. CONCLUSION
The ever-present issue of biased algorithms presents a significant challenge to fairness and
equity across various aspects of modern life, from employment practices to healthcare and
historical representation. Real-life examples have shown that algorithmic systems often
perpetuate and amplify existing biases, leading to discriminatory outcomes that
disproportionately impact marginalised communities.
The instances highlighted, such as Amazon's recruiting system that exhibited bias and the
COMPAS algorithm's racial disparities, underscore the pressing need for proactive measures to
address algorithmic bias. Furthermore, incidents involving Microsoft's Tay chatbot and Gemini's
image-generating tool demonstrate how AI systems can quickly adopt and propagate biases when
trained on flawed or incomplete data sets.
To effectively mitigate algorithmic bias, a multi-pronged approach is necessary. Firstly, there
must be increased transparency and accountability in algorithmic systems, ensuring that the
processes and data behind decision-making algorithms are accessible for scrutiny and auditing.
Additionally, ethical guidelines and industry standards should be developed and enforced to
promote fairness and inclusivity in AI development and deployment.
Furthermore, regulatory frameworks must be strengthened to address algorithmic bias
comprehensively. This involves implementing laws and regulations specifically targeting
algorithmic fairness and establishing robust oversight mechanisms to monitor algorithmic
systems and ensure compliance.
Moreover, engaging stakeholders is crucial in shaping effective regulatory policies and ensuring
that diverse perspectives are considered in algorithmic decision-making processes. Collaboration
between governments, industry, academia, and civil society can facilitate the development of
more inclusive and equitable AI technologies.
Looking ahead, policymakers, researchers, and practitioners must work together to tackle
algorithmic bias systematically and proactively. By adopting a holistic approach that combines
regulatory interventions, ethical guidelines, and stakeholder engagement, we can mitigate the
risks posed by algorithmic bias and foster a more equitable future for AI technology.
WAY FORWARD
1) Defining and Narrowing the Business Problem is a crucial initial step in addressing AI
bias. By clearly delineating the problem, organisations ensure that the AI model's purpose
aligns with its performance objectives. This clarity helps focus efforts on resolving the
specific issue, reducing the risk of unintended biases creeping into the model during
development and deployment.
2) Structuring Data Gathering plays a pivotal role in obtaining diverse perspectives and
opinions, fostering a more comprehensive understanding of the problem. By
incorporating a range of viewpoints, organisations can enhance the adaptability of their
AI models to different scenarios and perspectives. This flexibility mitigates the risk of
biases from narrow or homogeneous data sources.
4) Testing and Deploying with Feedback enables organisations to continuously evaluate and
refine their AI models based on end-user feedback. By incorporating feedback loops into
the deployment process, organisations ensure that AI systems maintain high-performance
levels and address emerging biases in real-world applications. This iterative approach
promotes continuous improvement and ensures that AI systems evolve to meet changing
needs and expectations.