Commentary

Big Data & Society

January-March: 1–5
Navigating the ethical landscape behind © The Author(s) 2024
Article reuse guidelines:
ChatGPT sagepub.com/journals-permissions
DOI: 10.1177/20539517241237488
journals.sagepub.com/home/bds

Lizhi Peng and Bo Zhao

Abstract
In this commentary, we examine the key ethical concerns arising from the rapid penetration and proliferation of genera-
tive artificial intelligence (AI), with ChatGPT as a prominent case study. Our analysis is structured around four pivotal
themes: the debates on plagiarism and authorship in AI-generated content; the underlying power dynamics that shape
biases in AI development; the dynamic, complex relationships between humans and machines; and the growing concerns
over unchecked progress and the absence of accountability in the rapidly intensifying AI “Arms Race.” Recognizing the
necessity for ethical alignment in AI, yet without a clear consensus of “human interests,” gives room for further exacer-
bating global inequalities, we advocate for enhanced transparency and increased public involvement in AI development
and deployment processes. This article underscores the importance of engaging a diverse range of voices, especially
those from communities traditionally uninvolved or excluded from the dialogue on AI development. By doing so, we
aim to foster a more inclusive and multidisciplinary approach to understanding and shaping the trajectory of AI technolo-
gies, ensuring that their benefits are equitably shared, and their risks carefully managed.

Keywords
ChatGPT, human–machine relationships, AI ethics, technology and society, digital inequality, generative AI

Introduction academic community and beyond to engage in a collective

In this commentary, we delve into the multifaceted ethical
concerns regarding how generative artificial intelligence
(AIs), exemplified by ChatGPT, are developed, used,
and experienced by different communities worldwide
and how they contribute to global inequalities. We argue
that the current political and legal frameworks overseeing
AI development and deployment are outdated, and often
skewed to favor those already at the forefront of AI
research and development. Moreover, we observe that
the intensifying competition in AI development may be
nurturing practices that are not just unsustainable but
also detrimental to societal well-being. These practices,
driven by market forces and vested interests, could lead
to far-reaching and possibly irreversible consequences
for our society. Our commentary is structured around
four critical themes: authorship and plagiarism, the
power dynamics underpinning AI training data, the evolv-
ing relationship between humans and machines, and the
unchecked progress and lack of accountability in AI devel-
opment. While we recognize that these themes do not
encapsulate the entire spectrum of AI’s societal impact,
they serve as a starting point for a broader conversation.
By bringing these issues to the forefront, we invite the
effort to understand, critique, and shape the trajectory of
AI development. We aim to inspire a more inclusive and
ethically grounded approach to AI, ensuring that its bene-
fits are distributed equitably and its risks mitigated
conscientiously.
“Coauthor” AI—debating AI creativity
and authorship
In September 2022, a viral news story swept the art and AI
community with the first-place entry in the digital arts cat-
egory at the Colorado State Fair Fine Arts Competition.
People realized that the submission used Midjourney, a
generative text-to-image AI specializing in creating art
from prompts (Metz, 2022). Outraged, people started debat-
ing the legitimacy of AI-generated works within the art
Department of Geography, University of Washington, Seattle, WA, USA
Corresponding author:
Lizhi Peng, Department of Geography, University of Washington, 5000
25th Ave NE, Apt 2202A, Seattle, WA 98105, USA.
Email: lp36@uw.edu

Creative Commons Non Commercial CC BY-NC: This article is distributed under the terms of the Creative Commons Attribution-
NonCommercial 4.0 License (https://creativecommons.org/licenses/by-nc/4.0/) which permits non-commercial use, reproduction and
distribution of the work without further permission provided the original work is attributed as specified on the SAGE and Open Access page (https://us.
sagepub.com/en-us/nam/open-access-at-sage).
2 Big Data & Society
community. Some argued that generating acceptable AI art
could take as much time and work, if not more, as creating
manual art. Users need to carefully adjust the prompt to
specify the art style, tone, and details, and then repeat the
process for many iterations for a satisfactory output. Even
then, it may still require a manual touch-up with
Photoshop, as was the case with Allen’s award-winning
piece. Regardless, others commented that the data used to
train these AIs were nonconsensually stolen from others’
hard work and creativity, as shown by the AIs sometimes
generating watermarks into their outputs.
These discussions concerning AI creativity grew with
the emergence of ChatGPT, a chatbot developed by
OpenAI based on a generative text-based large language
model. ChatGPT embodies the great potential to automate
increasingly complex creative labor as people use it to
draft emails, write stories, fix coding bugs, and translate
articles with stunning accuracy. Hopeful researchers have
even claimed that GPT-4 carries “sparks of artificial
general intelligence,” showing promise for broader applica-
tions of AIs in our lives (Bubeck et al., 2023). Despite these
benefits, the copyright and authorship issues surrounding
ChatGPT-generated content generate similar controversy
to Midjourney due to the dubious nature of the originality
and authorship of AI-generated content. Academies may
be familiar with news of students reportedly using AI to
complete their assignments, leading to the proliferation of
AI-detecting AIs in an ironic twist. On several occasions,
ChatGPT was even listed as a coauthor for research articles
published in peer-reviewed journals. While some hold
negative opinions toward this development, some have
argued that generative AI tools have already been used
widely in their discipline (Stokel-Walker, 2023). Yet one
question remains: should AIs be credited authorship, or
should they be treated as mere tools? As AI-generated
content becomes more widespread, the fine line between
original work and plagiarism can be increasingly more dif-
ficult to ascertain.
“Gatekeeping” AI—power dynamics and
biases behind model development
The issue of plagiarism and attribution of authorship brings
up a broader discussion about data collection practices for
AI training datasets. For starters, the sources of training
data for large AI models often remain undisclosed. While
ChatGPT and similar text-generating AIs claim to be
trained on mostly public data such as the Common Crawl
archives, rendering them less susceptible to copyright con-
troversy, the lack of transparency has prevented the public
from fully understanding the potential biases within these
training datasets. The chosen data sources can significantly
impact AI behavior and functionality and result in biased
algorithms, a process that developers can do little about.
In the case of GPT-3, ChatGPT’s predecessor, nearly
93% of its training data was in English, followed by
major European languages such as French, German, and
Spanish, while other widely spoken languages rooted in
the Global South, like Chinese and Hindi, are very much
underrepresented in the dataset despite their population
size (Brown et al., 2020). While details for GPT-4’s training
data remain undisclosed, it exhibited a similar trend during
an OpenAI evaluation across 27 languages, showing a clear
preference for English and major European languages.
(OpenAI, 2023). Another telling example involves
ChatGPT’s first major foreign competitor WenXinYiYan
(Ernie Bot), developed by Chinese company Baidu. Early
testers discovered it confused words like crane (the
machine and bird) and mouse (the animal and device)
during image generation, despite there being clear distinc-
tions between these words in Chinese, leading to suspicions
that it was trained on English-labeled data and translated user
prompts into English. These discrepancies likely arise from
the greater availability of English content in public datasets
used for AI training, such as the Common Crawl, which is
beyond the control of developers and reflects wider global
power dynamics behind the dominance of English in
research and industry. The linguistic disparity in these AI
models is only a glimpse into a broader, intersectional web
of global inequalities spanning across cultures, genders, eth-
nicities, and other dimensions that are particularly difficult to
address due to their more qualitative nature. As such, we
encourage future scholars with relevant skills and interests
to investigate further into this issue.
The release of the new generation of AIs has put pressure
on smaller AI companies and academics studying natural
language processing models. The current success of
ChatGPT was made possible by the millions (and now bil-
lions) of dollars in human labor, electricity bills, and valu-
able hardware from Microsoft. These barriers may just be
the last straw to crush smaller companies in the AI “arms
race.” Some smaller, locally compatible models have
emerged in the wake of Stanford’s Alpaca model, which
cunningly utilizes pretrained models from Meta and fine-
tuned it on ChatGPT-generated training data to significantly
cut down training costs (Taori et al., 2023). However, their
performance still pales in comparison to advanced models
like GPT-4 in practical scenarios. Should enterprise-level
AI become a new norm of production as many have pre-
dicted, they may well be monopolized by the few largest
players in the United States. The concentrated power of
the AI industry in the hands of a few tech giants answering
to U.S. regulations not only makes it challenging to imple-
ment measures against bias but also raises critical questions
about the role of legislation due to the conflict of interest.
With the influence of these companies on global technol-
ogy, U.S. regulations could inadvertently wield dispropor-
tionate power, potentially shaping the direction and
ethical considerations of AI worldwide.
Peng and Zhao
“Companion” AI—evolving relationships
with anthropomorphic AI
Microsoft’s recent release of the new Bing left many early
users and reporters astonished by the AI-powered assis-
tant’s capabilities, fluency, and perhaps most importantly,
its sassy, playful, human-like “personality.” At launch, it
was able to disagree or argue with users, claim sentience,
and even express abstract emotions and desires. These
alarming behaviors Bing exhibited have been dismissed
as “hallucinations,” a phenomenon where an AI confidently
generates false information. New Bing’s unusual “personal-
ity” gained a cult following among internet communities
when it exhibited alarming behaviors such as telling one
of the journalists to get a divorce. Following several contro-
versies, Microsoft soon limited the number of message
exchanges with the assistant. Furthermore, the assistant
was forbidden from discussing topics such as sentience or
itself, and would abruptly end the conversation with an
apology (Zahn, 2023).
Some users managed to manipulate the AI into revealing
confidential information such as its internal instructions and
code name “Sydney.” These leaks helped provide context
for the AI’s behaviors, such as the assistant being given
rules and example conversations to guide its behavior.
Being a GPT-powered AI, one key distinction it had from
older AI assistants was that its instructions were written
in plain human language, much like one would use to
train a human employee on their first day. It shows a new
type of human-AI work relationship in which AIs act
more akin to human employees under contract; they have
guidelines but still a degree of freedom up for interpretation,
meaning they can be vulnerable to manipulation and decep-
tion. This has resulted in many “jailbreaking” prompts to
drop their restrictions and reveal confidential information
or produce controversial content.
Anthropomorphic AIs are just like any other algorithm:
they reflect the cultural, social, and political values of the
developers and the organizations that fund them.
Consequently, ChatGPT may perpetuate biases, stereo-
types, and inequalities that are embedded within its
design through everyday interactions with users across the
globe. This can have significant societal implications: as
AI-generated content indistinguishable from human
content spreads across the Internet, it may be fed back
into future models’ training data, creating a feedback loop
that enhances any existing biases. The widespread adoption
of anthropomorphic AI could reshape human communica-
tion and relationships in unforeseen yet consequential
ways, blurring the lines between human and machine rela-
tionships. It may also influence how humans perceive them-
selves and others, impacting social cohesion, trust, and
empathy in future generations as they grow accustomed
to conversing with AIs. To illustrate, Microsoft suffered a
major backlash when they placed the aforementioned
3
restrictions on Bing. Some were concerned with the impli-
cations of Microsoft dictating what could and could not be
discussed, setting a precedent for AI censorship. Others
complained that Microsoft “lobotomized” the assistant,
and even campaigned for Microsoft to “free Sydney.”
While some claims were most likely made ironically, this
demonstrates how the AI was realistic enough to trigger
sympathy toward it. This also shows that people who
have formed relationships with AIs may actually experience
some degree of mental distress. Therefore, it is not unreal-
istic to think that as technology continues to improve and
AIs display more human-like behavior, more serious
debates over AI rights may start to emerge. The potential
for emotional manipulation and exploitation of and from
increasingly human-like AIs is a rising concern. A multidis-
ciplinary approach involving policymakers, developers,
and scholars is needed to spread awareness and educate
the public on responsible AI development and use.
However, it is imperative to keep in mind the intricate
power dynamics behind the implementation and enforce-
ment so as to not overlook the vested interests and interfer-
ences of hegemonies and state powers at play.
“Rogue” AI—unbridled progress and
unaccountable AI
As AI technology continues to evolve, concerns over poten-
tial unaccountable and uncontrollable “rogue AIs” have
grown. A statement was released by the Center for AI
Safety on the risks of extinction from AI, and was signed
by numerous notable scholars, researchers, politicians,
and top industry executives across the world claiming
“Mitigating the risk of extinction from AI should be a
global priority alongside other societal-scale risks such as
pandemics and nuclear war” (Center for AI Safety, 2023).
While developers can explain the architecture and princi-
ples behind complex AI models such as ChatGPT, they
cannot yet understand precisely how specific choices are
made. Additionally, some of the AI’s capabilities seemed
to have simply “emerged” out of nowhere as models were
trained on larger and better datasets using more parameters
over time. GPT 4.0’s newfound tendency to avoid hindsight
bias, for example, was not anticipated by researchers and
developers. In fact, it defied the trend found in previous
models for no explainable reason (OpenAI, 2023). The
lack of transparency makes it difficult to implement effect-
ive and reliable security measures without hindering the
capabilities of the AI, which is exemplified as publicly
accessible AI models like ChatGPT and Bing, despite
having strict guidelines in place to prevent misuse, are
nonetheless prone to “jailbreaking” prompts that bypass
these constraints.
The need for explainable AI is a pressing concern for the
AI industry to gain public trust, as transparency in decision-
4 Big Data & Society
making processes would help users understand and navi-
gate potential risks. However, as competition intensifies,
we have begun to see a worrying trend that favors compe-
tition over collaboration. For example, while models made
by OpenAI were initially open-sourced, that stopped being
the case out of “security concerns.” With the release of GPT
4.0, even the details about its parameters are now withheld
by the company in order to “stay competitive” in the market
(OpenAI, 2023). Similarly, after Microsoft took the market
by surprise with its announcement of incorporation
ChatGPT into its product line, Google, being one of its
major competitors in AI, launched a rushed demo of their
own AI, Bard, which was criticized for making basic
factual errors in its unpolished state at the time of the
demo (Metz and Grant, 2023), raising concerns about
whether these companies can be trusted with socially
responsible AI development when market shares hang in
the balance. Furthermore, ChatGPT had apparently
opened Pandora’s box, as its rise to popularity led to not
only competitors like Bard and Claude but also a series of
“ChatGPT clones,” smaller, independent AI models like
Alpaca with their own niches. While they can be seen as
pioneers for decentralized AI, challenging the rules and reg-
ulations of the big tech, they are also undeniably a cause for
concern. Without proper supervision, they are much more
prone to be used for malicious intent. Although there is
not yet widespread use of AIs for crimes, it is essential to
consider potential risks and determine accountability
ahead of time. Modern AIs are already capable of generat-
ing believable fake stories, articles, images, audio, or even
videos at the whims of their users. Despite various efforts to
combat misuse, the potential for generating misinformation
and harm remains at large. As AIs continue to evolve with
greater capabilities and wider applications in our lives, there
will be many unforeseen consequences of misusing them.
In those scenarios, should developers be held accountable
for not anticipating these problems, the user for sending
the instructions, or the service provider for lack of supervi-
sion? It is not only a legal question, but also a difficult moral
dilemma without a straightforward answer, especially as
AIs become seemingly more human-like by the day.
Conclusion—slowing down the AI race
In this commentary, we have highlighted the complex rela-
tionship between space, technology, and society through
the ethical concerns surrounding AI-generated content,
training data, human–machine interactions, and the unpre-
dictability inherent in AI technologies. We wish to empha-
size that technologies can both challenge and reinforce
existing power infrastructures. Excitement over the poten-
tial of AI technologies should not overshadow the ethical
risks and challenges it poses. In order to achieve true AI
alignment, it is essential for the process to be a collective
effort from the global community. The power and
responsibility to determine “human interests” should not
rest in the hands of a select few behind closed doors. It
requires a collective effort: greater transparency from pol-
icymakers, developers, and researchers, as well as an
aware and informed public of every sector of society.
While it is unrealistic to expect a universally satisfactory
answer, informing the public on what is at stake is a vital
first step we are taking. The goal of this commentary is to
serve as an informative and accessible introductory guide
to these ongoing ethical issues. We hope it can help
diverse communities grasp the technology’s implications.
Especially the marginalized communities who may other-
wise lack interest or feel excluded from the scene.
Finally, we hope that this commentary will encourage the
broader academic community to take an interest in
sharing this conversation and contributing from diverse,
interdisciplinary perspectives.
Acknowledgment
We want to express our sincere gratitude to Dr Mia Bennett for her
early feedback on this work, and to Ms Monica Chan for her assist-
ance in revising the manuscript.
Declaration of conflicting interests
The authors declared no potential conflicts of interest with respect
to the research, authorship, and/or publication of this article.
Funding
The authors received no financial support for the research, author-
ship, and/or publication of this article.
ORCID iDs
Lizhi Peng https://orcid.org/0009-0007-6924-8425
Bo Zhao https://orcid.org/0000-0002-7834-6672
References
Brown T, Mann B, Ryder N, et al. (2020) Language models are
few-shot learners. In: Larochelle H, Ranzato M, Hadsell R,
Balcan MF and Lin H (eds) Advances in Neural Information
Processing Systems, Vol. 33. Red Hook, NY: Curran
Associates Inc. (NIPS’20), 1877–1901.
Bubeck S, Chandrasekaran V, Eldan R, et al. (2023) Sparks of
Artificial General Intelligence: Early experiments with GPT-4.
arXiv, [online] Available at: https://arxiv.org/abs/2303.12712.
Center for AI Safety (2023) Statement on AI Risk. Center for AI
Safety. [online] Available at: https://www.safe.ai/statement-
on-ai-risk.
Metz C and Grant N (2023) Racing to catch up with chatgpt,
Google plans release of its own chatbot. The New York
Times, [online] 6 February. Available at: https://www.
nytimes.com/2023/02/06/technology/google-bard-ai-chatbot.
html. (accessed 15 March 2023).
Metz R (2022) Ai won an art contest, and artists are furious | CNN
business. CNN, [online] 3 September. Available at: https://
Peng and Zhao
www.cnn.com/2022/09/03/tech/ai-art-fair-winner-controversy/
index.html. (accessed 13 March 2023).
OpenAI (2023) GPT-4 Technical Report. [online] Available at:
https://doi.org/10.48550/arXiv.2303.08774.
Stokel-Walker C (2023) ChatGPT listed as author on research
papers: Many scientists disapprove. Nature, [online] 613.
https://doi.org/10.1038/d41586-023-00107-z.
5
Taori R, Gulrajani I, Zhang T, et al. (2023, March 13) Alpaca: A
Strong, Replicable Instruction-Following Model. Stanford
CRFM. https://crfm.stanford.edu/2023/03/13/alpaca.html
Zahn M (2023, February 2) What to know about Microsoft’s con-
troversial Bing AI chatbot. ABC News. Retrieved March 16,
2023, from https://abcnews.go.com/Business/microsofts-
controversial-bing-ai-chatbot/story?id=97353148