Frontiers
RESPONSIBLE HUMANS
How AI Skews Our Sense of Responsibility
Research shows how using an AI-augmented system may affect humans’ perception of their own agency
and responsibility.
By Ryad Titah
A
s artificial intelligence plays an ever-
larger role in automated systems and
decision-making processes, the ques-
tion of how it affects humans’ sense
of their own agency is becoming less theo-
retical — and more urgent. It’s no surprise
that humans often defer to automated deci-
sion recommendations, with exhortations
to “trust the AI!” spurring user adoption in
corporate settings. However, there’s grow-
ing evidence that AI diminishes users’ sense
of responsibility for the consequences of
those decisions.
This question is largely overlooked in
18 MIT Sloan Management Review summer 2024
current discussions about responsible AI.
In reality, such practices are intended to
manage legal and reputational risk — a
limited view of responsibility, if we draw
on German philosopher Hans Jonas’s use-
ful conceptualization. He defined three
types of responsibility, but AI practice
appears concerned with only two. The first
is legal responsibility, wherein an individ-
ual or corporate entity is held responsi-
ble for repairing damage or compensating
for losses, typically via civil law, and the
second is moral responsibility, wherein
individuals are held accountable via
punishment, as in criminal law.
What we’re most concerned about here
is the third type, what Jonas called the
sense of responsibility. It’s what we mean
when we speak admiringly of someone
“acting responsibly.” It entails critical think-
ing and predictive reflection on the purpose
and possible consequences of one’s actions,
not only for oneself but for others. It’s this
sense of responsibility that AI and auto-
mated systems can alter.
To gain insight into how AI affects users’
perceptions of their own responsibility and
agency, we conducted several studies. Two
Matt Chinworth / theispot.com
studies examined what influences a driver’s
decision to regain control of a self-driving
vehicle when the autonomous driving
system is activated. In the first study, we
found that the more individuals trust the
autonomous system, the less likely they
are to maintain situational awareness that
would enable them to regain control of the
vehicle in the event of a problem or inci-
dent. Even though respondents overall said
they accepted responsibility when operat-
ing an autonomous vehicle, their sense of
agency had no significant influence on their
intention to regain control of the vehicle in
the event of a problem or incident. On the
basis of these findings, we might expect to
find that a sizable proportion of users feel
encouraged, in the presence of an auto-
mated system, to shun responsibility to
intervene.
In the second study, conducted with
the Société de l’Assurance Automobile du
Québec, a government agency that admin-
isters the province’s public auto insur-
ance program, we were able to conduct
more-refined analyses. We surveyed 1,897
drivers (mostly of Tesla and Mercedes cars
with some level of autonomous driving
capabilities) to look at the separate effect
of each type of responsibility on the driver’s
intention to regain control of the vehicle
and found that only the sense of responsi-
bility had a significant effect. As in the first
study, the more trust respondents reported
having in the automated system, the lower
their intention to regain control behind the
wheel. It’s particularly notable that only the
proactive, individual sense of responsibility
motivated respondents to act, which indi-
cates that the threat of liability will be insuf-
ficient to prevent AI harm.
In another study aimed at understanding
the use of risk-prediction algorithms in the
context of criminal justice in the U.S., a sig-
nificant proportion of the 32 respondents
relied excessively on these tools to make
their decisions. We made a determination
of overuse based on respondents reporting
use of the tools to determine the length or
severity of a sentence, strictly abiding by
the tools’ results, and taking for granted the
results provided by the tool. Besides raising
fundamental legal and ethical questions
about the fairness, equity, and transparency
of such automated judicial decisions, this
result also points to an abdication of indi-
vidual responsibility in favor of algorithms.
Overall, these initial results confirm
what has been observed in similar con-
texts, namely that individuals tend to lose
their sense of agency in the presence of an
intelligent system. When individuals feel
less control — or that something else is in
control — their sense of responsibility is
likewise diminished.
In light of the above, we must ask
whether human in the loop, which is
increasingly understood as a best practice
for responsible AI use, is an adequate safe-
guard. Instead, the question becomes: How
do we encourage humans to accept they
have proactive responsibility and exercise it?
As noted at the outset of this article,
managers tend to exacerbate the prob-
lem by encouraging trust in AI in order to
increase adoption. This message is often in
terms that denigrate human cognition and
decision-making as limited and biased com-
pared with AI recommendations — despite
the fact that all AI systems necessarily reflect
human biases in data selection, specifica-
tion, and so forth. This position assumes that
every AI decision is correct and superior to
the human decision and invites humans to
disengage in favor of the AI.
To offset this tendency, we recommend
shifting emphasis in employee communica-
tions from trust the AI to understand the AI
to engender informed and conditional trust
in the outputs of AI systems and processes.
Managers need to educate users to under-
stand the automated processes, decision
points, and potential for errors or harm of
the AI. It’s also critical that users are aware
of and understand nuances of the edge cases
where AI systems can flounder and the risk
of a bad automated decision is greatest.
Managers also need to position and pre-
pare employees to own and exercise their
sense of responsibility. Toyota famously
models this by empowering anyone on the
factory floor to shut down the production
line if they see a problem. This will encour-
age employees to systematically question
AI systems and processes and therefore
maintain their sense of agency in order
to avoid harmful consequences for their
organizations.
Ultimately, a culture of responsibility —
versus a culture of avoiding culpability — is
always going to mean a healthier and more
ethically robust organization. It will be even
more important to foster such a culture in
the age of AI by leaving clear spaces of pos-
sibility for human intelligence.
Otherwise, Roderick Seidenberg’s pre-
diction about technologies far less powerful
than current AI could materialize:
The functioning of the system, aided
increasingly by automation, acts —
certainly by no malicious intent — as
a channel of intelligence, by which the
relatively minute individual contribu-
tions of engineers, inventors, and sci-
entists eventually create a reservoir of
established knowledge and procedure
no individual can possibly encompass.
Thus, man appears to have surpassed
himself, and indeed in a certain sense he
has. Paradoxically, however, in respect
to the system as a whole, each individ-
ual becomes perforce an unthinking
beneficiary — the mindless recipient
of a socially established milieu. Hence
we speak, not without justification, of a
push-button civilization — a system of
things devised by intelligence for the pro-
gressive elimination of intelligence! 2
Ryad Titah is associate professor and chair
of the Academic Department of Information
Technologies at HEC Montréal. The research in
progress described in this article is being con-
ducted with Zoubeir Tkiouat, Pierre-Majorique
Léger, Nicolas Saunier, Philippe Doyon-Poulin,
Sylvain Sénécal, and Chaïma Merbouh.
Reprint 65427. For ordering information, see page 4.
Copyright © Massachusetts Institute of Technology,
2024. All rights reserved.
summer 2024 sloanreview.mit.edu 19