3

Social Media Platforms as Public Health Arbiters

Global Ethical Considerations on Privacy, Legal, and Cultural
Issues Associated with Suicide Detection Algorithms

karen l. celedonia, michael lowery wilson,

and marcelo corrales compagnucci

3.1 Introduction
3.1.1 Suicide Prevention and the Use of Technology
Suicide is a growing, global public health problem.1 No corner of the
world remains untouched by the increasing incidence and prevalence of
suicidal behaviour across the lifespan. Every forty seconds, an individual
takes their own life somewhere in the world. Suicide is the second lead-
ing cause of death among adolescents and young adults aged 15–26,2 and
the suicide rates among this age group have been on the rise over the past
decade.3 Recently, there has also been an alarming trend of increased sui-
cidal behaviour and suicide among children as young as five years old.4

1 This is an extended version of an article published in the Journal of Law and the Biosciences
where we argued that Facebook’s practices in this area should be subject to well-established
protocols such as an ethical review process and the explicit approval of its users in the form
of informed consent. We also proposed a fiduciary framework with the assistance of a
panel of external impartial experts from various fields and diverse backgrounds. See KL
Celedonia and others, ‘Legal, Ethical, and Wider Implications of Suicide Risk Detection
Systems in Social Media Platforms’ (2021) 8(1) Journal of Law and the Biosciences, https://
doi.org/10.1093/jlb/lsab021, accessed 23 May 2021.
2 P Rodríguez Herrero, A de la Herrán Gascón and V de Miguel Yubero, ‘The Inclusion of
Death in the Curriculum of the Spanish Regions’ (2020) Compare: A Journal of Comparative
and International Education, 1–19.
3 MF Hogan, ‘Suicide Prevention: Rising Rates and New Evidence Shape Policy Options’ in
The Palgrave Handbook of American Mental Health Policy (Springer 2020) 229–57.
4 AH Sheftall and others, ‘Suicide in Elementary School-Aged Children and Early Adolescents’
(October 2016) 138(4) Pediatrics e20160436, https://doi.org/10.1542/peds.2016-0436, accessed
23 May 2021.

68

https://doi.org/10.1017/9781108921923.006 Published online by Cambridge University Press

 social media platforms as public health arbiters 69

Population-based research suggests that suicide rates are highest in

high-income counties (HICs), at 11.5 per 100,000 individuals.5 However,
globally available data suggest that low- and middle-income countries
(LMICs) may be disproportionately affected by suicide, with most of the
world’s suicides occurring in these countries. Of the at least 800,000 global
deaths by suicide in a given year, 76 per cent were from LMICs.6 Given
these ­concerning trends in suicide rates, early detection of suicide warn-
ing signs paired with appropriate intervention and treatment by mental
health professionals is imperative to saving the lives of individuals at risk
of suicide.
The mental health field has embraced the use of technology for timely
prevention and treatment of mental health disorders like suicidal behav-
iour. From tele-psychiatry to mobile health applications (apps), more
and more mental health providers are turning to technology to assist in
the provision of care to those in need. Transportation barriers and lack
of childcare are common impediments to individuals with mental illness
initially seeking mental health treatment or keeping appointments once
treatment is started.7 Tele-psychiatry and mobile health apps eliminate
the need for treatment sessions to be conducted in person, thereby allow-
ing individuals who may not be able to attend in-person sessions due to
economic constraints or competing demands to access the treatment they
need without having to leave home. Furthermore, given the potentially
crippling nature of mental illnesses like depression and anxiety, individu-
als experiencing acute symptomology of these disorders may be unable
to leave their homes to attend treatment sessions. These individuals
can benefit from tele-psychiatry and mobile health apps to manage and
improve their symptoms until they feel well enough to attend in-person
sessions. Technology-based services like tele-psychiatry have allowed
mental health care providers to reach clients at risk of suicide who oth-
erwise might not have received treatment due to service access barriers.8
Tele-psychiatry is an American Psychiatric Association (APA) approved
service delivery method that is recognised by insurance companies as a

5 World Health Organization, ‘Suicide in the World: Global Health Estimates’ (2019) World
Health Organization, https://apps.who.int/iris/handle/10665/326948, accessed 23 May 2021.
6 Ibid.
7 KM Shealy and others, ‘Delivering an Evidence-Based Mental Health Treatment to
Underserved Populations Using Telemedicine: The Case of a Trauma-Affected Adolescent
in a Rural Setting’ (2015) 22(3) Cognitive and Behavioral Practice, 331–44.
8 DM Hilty and others, ‘Telepsychiatry’ (August 2002) 16(8) Molecular Diagnosis and Therapy,
527–48, https://doi.org/10.2165/00023210-200216080-00003, accessed 23 May 2021.

https://doi.org/10.1017/9781108921923.006 Published online by Cambridge University Press

 70 karen l. celedonia et al.

billable service,9 and it is held to the same rigorous ethical standards of

client privacy and health information protection as treatment-as-usual,
in-person services. For example, US providers that offer tele-psychiatry
are expected to use Health Insurance Portability and Accountability Act
(HIPAA)-compliant video conferencing platforms when conducting vir-
tual treatment sessions.10
Beyond treatment sessions, mobile health apps can serve to supplement
psychiatric services and promote self-monitoring of symptoms among indi-
viduals with mental illness. Though research on the effectiveness of mobile
apps in treating symptoms of mental illnesses like depression and anxiety is
in its infancy, preliminary investigations suggest that mobile apps have the
potential to alleviate symptoms of depression and anxiety. Due to their ease
of access and seemingly comparable effectiveness, some health profession-
als have even suggested that mobile health apps could be a solution to the
global shortage of psychiatrists. However, experts warn that mobile health
apps currently suffer from inadequate regulation of quality and privacy.11
In an effort to leverage their ubiquitous global presence to help pre-
vent suicide, search engines and social media platforms have developed
algorithms to detect suicide risk among their users. Google, for example,
developed an algorithm to detect suicide risk trends in users’ search term
history.12 Depending on the level of risk detected, ads for suicide preven-
tion hotlines and mental health treatment may be displayed to those at
higher risk. By using key-word matching, Apple, Google Assistant, and
Amazon have also taken efforts to guide users of their digital assistant
devices (i.e. Siri, Echo, Alexa) who verbally express suicidal thoughts and
intentions to suicide prevention resources.13

9 JS Gardner and others, ‘Remote Telepsychiatry Workforce: A Solution to Psychiatry’s

Workforce Issues’ (January 2020) 22(2) Current Psychiatry Reports, 8–9, https://doi
.org/10.1007/s11920-020-1128-7, accessed 23 May 2021.
10 JH Wright and R Caudill, ‘Remote Treatment Delivery in Response to the COVID-19
Pandemic’ (2020) 89(3), Psychotherapy and Psychosomatics, 1, https://doi.org/10.1159/
000507376, accessed 23 May 2021.
11 P Chandrashekar, ‘Do Mental Health Mobile Apps Work: Evidence and Recommendations
for Designing High-Efficacy Mental Health Mobile Apps’ (2018) 4 mHealth, 6, https://doi
.org/10.21037/mhealth.2018.03.02, accessed 23 May 2021.
12 P Solano and others, ‘A Google-Based Approach for Monitoring Suicide Risk’ (2016) 246
Psychiatry Research, 581–86, www.sciencedirect.com/science/article/pii/S016517811630
1949, accessed 23 May 2021.
13 TW Bickmore and others, ‘Patient and Consumer Safety Risks When Using Conversational
Assistants for Medical Information: An Observational Study of Siri, Alexa, and Google
Assistant’ (2018) 20(9) Journal of Medical Internet Research e11510.

https://doi.org/10.1017/9781108921923.006 Published online by Cambridge University Press

 social media platforms as public health arbiters 71

Perhaps most controversial is Facebook’s suicide detection algorithm.

The algorithm works by scanning every post a Facebook user makes on
the platform and scoring the post on a scale of 0–1 for risk of imminent
harm. Facebook does not stop at a simple algorithm and displaying ads for
suicide prevention hotlines like Google: it takes suicide detection a step
further by relaying the risk information to law enforcement, who then
intervene in what are called ‘wellness checks’. Depending on the assess-
ment and the severity of the risk detected, users may then be transported
to psychiatric inpatient units for a thorough psychiatric evaluation by a
mental health professional.
While the use of technology to help prevent suicide seems congruous
with the changing landscape of the health care industry to include more
widespread implementation of eHealth and artificial intelligence to assist
with health care delivery, many of these technology-based suicide preven-
tion programmes have not been adequately researched before being made
available to the public. A review of 123 mobile health apps for suicide risk
detection and support found that none of the apps offered evidence-based
intervention for individuals at risk of suicide, with many apps actually
containing content potentially harmful to individuals in a vulnerable
mental state.14 In regard to suicide detection algorithms, concerns have
been raised about entities like Facebook, which are not health care provid-
ers, conducting health surveillance, providing health advice and interven-
tion without being held to the same ethical standards as legitimate health
care providers.15
Though mental health concerns like suicide are common on a global
scale, stigma persists surrounding individuals with mental health con-
cerns despite concerted efforts over the decades to normalise these com-
mon health conditions. In many LMICs, stigma around mental health
conditions has not improved,16 and the disclosure of mental illness can
have severe social consequences.17 Even in countries where mental

14 ME Larsen, J Nicholas and H Christensen, ‘A Systematic Assessment of Smartphone Tools

for Suicide Prevention’ (2016) 11(4) PLOS ONE e0152285.
15 A Pourmand and others, ‘Social Media and Suicide: A Review of Technology-Based
Epidemiology and Risk Assessment’ (October 2019) 25(10) Telemedicine and eHealth, 880–
88, https://doi.org/10.1089/tmj.2018.0203, accessed 23 May 2021.
16 E Heim and others, ‘Reducing Mental Health Related Stigma in Primary Health Care Settings
in Low- and Middle-Income Countries: A Systematic Review’ (2020) 29 Epidemiology and
Psychiatric Sciences, https://doi.org/10.1017/S2045796018000458, accessed 23 May 2021.
17 AC Krendl and BA Pescosolido, ‘Countries and Cultural Differences in the Stigma of
Mental Illness: The East–West Divide’ (February 2020) 51(2) Journal of Cross-Cultural
Psychology, 149–67, https://doi.org/10.1177/0022022119901297, accessed 23 May 2021.

https://doi.org/10.1017/9781108921923.006 Published online by Cambridge University Press

 72 karen l. celedonia et al.

illnesses are more accepted, stigma still exists, and as such, many individ-
uals with mental illness choose not to voluntarily disclose their disorders.
Non-health care entities like social media platforms revealing someone’s
mental state to public officials could therefore be construed as a violation
of an individual’s right to disability non-disclosure.
Another layer to the aforementioned issues with stigma and involun-
tary disclosure of mental illnesses is the possibility of the suicide detec-
tion algorithm resulting in false positives for suicide risk. As with the
mobile health apps for suicide prevention, these detection algorithms
were not (and have not been) properly researched before their implemen-
tation. Data on the accuracy of the algorithm are not publicly available,
and anecdotal accounts of the algorithms erroneously detecting suicide
risk when none was present are starting to surface. In countries where
mental illnesses are more accepted, these mistakes may mean little more
than extreme inconvenience and unnecessary resource expenditures, but
in countries where mental illnesses are still misunderstood and severely
stigmatised, such a mistake could ruin an individual’s life.
With over 2.6 billion users worldwide and an emerging trend of users
live-streaming suicide attempts, one might argue that social media plat-
forms like Facebook have a moral obligation to develop suicide detection
algorithms to protect their users. To refrain from doing anything to pre-
vent suicide among its users could itself be viewed as unethical. However,
given that research is revealing that mental health issues like depression,
which is a risk factor for suicide, are associated with social media use,18
one may wonder if suicide detection algorithms on social media platforms
are nothing more than these platforms trying to fix a problem that they
themselves created or had a role in exacerbating. In addition to possibly
contributing to mental illnesses that put individuals at risk for suicide,
the trend of live-streaming suicide attempts on social media creates the
possibility of behavioural contagion in which viewers of these videos may
mimic the behaviour they witness.19 Furthermore, studies show that there
is a significant positive, predictive association between social media use
and loneliness: the more time an individual spends using social media,

18 MG Hunt and others, ‘No More FOMO: Limiting Social Media Decreases Loneliness and
Depression’ (December 2018 Guilford Publications Inc), https://guilfordjournals.com/
doi/abs/10.1521/jscp.2018.37.10.751, accessed 23 May 2021.
19 M Westerlund, G Hadlaczky and D Wasserman, ‘Case Study of Posts before and after a
Suicide on a Swedish Internet Forum’ (December 2015) 207(6) British Journal of Psychiatry,
476–82, https://doi.org/10.1192/bjp.bp.114.154484, accessed 23 May 2021.

https://doi.org/10.1017/9781108921923.006 Published online by Cambridge University Press

 social media platforms as public health arbiters 73

the more lonely they report feeling;20 loneliness is often associated with
suicidal behaviour.21

3.1.2 The Ethics of Social Media Platforms

Acting As Health Care Providers
Providing any sort of health care to an individual is inherently rife with
ethical tension. Childress and Beauchamp developed the four prin-
ciples of health care ethics to guide the ethical provision of health and
­medical treatment. These four principles are autonomy, beneficence,
non-­maleficence, and justice.22 Autonomy refers to the right of the ­client
to retain control of their body; a health care provider may suggest a
­certain treatment course, but the decision to follow the suggested treat-
ment course is ultimately up to the client and must be made by the client
independently according to their personal beliefs and values, without
coercion from the health care provider. Beneficence dictates that health
care providers must do all they can to benefit the client in each situation,
with all recommendations being made solely within the best interest of
the client. This includes health care providers maintaining a high stan-
dard of skill and knowledge as evidenced by being trained in the most
up-to-date and best practices in health care provision. Additionally,
as part of the beneficence principle, health care providers must take
into consideration the unique circumstances of each individual client,
­recognising that what may be the best treatment course for one client
may not be the best for another. Non-maleficence states that health care
providers should strive to do no harm to their clients, other people, and
society at large. Justice suggests that fairness should be present in health
care ­provision and all treatment decisions. It also proclaims that health
care professionals should uphold any relevant laws and legislation when
making treatment decisions.23

20 M Savci and F Aysan, ‘Relationship between Impulsivity, Social Media Usage and
Loneliness’ (March 2016) 5(2) Educational Process: International Journal, 106–15, https://
doi.org/10.12973/edupij.2016.52.2, accessed 23 May 2021.
21 A Stravynski and R Boyer, ‘Loneliness in Relation to Suicide Ideation and Parasuicide: A
Population-Wide Study’ (June 2005 Guilford Publications Inc), https://guilfordjournals
.com/doi/abs/10.1521/suli.31.1.32.21312, accessed 23 May 2021.
22 JF Childress and TL Beauchamp, Principles of Biomedical Ethics (Oxford University Press
2001).
23 R Rawbone, ‘Principles of Biomedical Ethics’ (January 2015) 65(1) 7th Edition Occupational
Medicine (Lond) 88–89, https://doi.org/10.1093/occmed/kqu158, accessed 23 May 2021.

https://doi.org/10.1017/9781108921923.006 Published online by Cambridge University Press

 74 karen l. celedonia et al.

It has been suggested that social media platforms – particularly

Facebook with its suicide detection algorithm and corresponding inter-
vention protocol – are behaving as health care providers without being
held to the same guiding ethical principles as legitimate health care pro-
viders. Like health care providers, Facebook is collecting, analysing, and
acting upon personal health care information. However, unlike health
care providers, Facebook is not adhering to health information and data
protection laws that many countries have in regard to collecting and using
patient data. In what little has been described of the algorithm and inter-
vention thus far, it is evident that as a social media platform acting as a
health care provider, Facebook’s suicide detection algorithm and inter-
vention violates the four principles of health care ethics, not to mention
ignoring patients’ rights to privacy and knowledge of how their personal
health information may be used.
In this chapter, a more in-depth review and critical analysis of the
aforementioned ethical considerations inherent in social media platforms
interjecting their influence into the public health realm as it pertains to
suicide detection will be provided. These ethical considerations are clas-
sified into three categories: privacy and patient rights, legal, and cultural.
For the purposes of the discussion in this chapter, social media platforms
will be hypothetically held to the same ethical standards as legitimate
health care entities. In so doing, the four principles of health care ethics
will be used to frame the discussion in this chapter, providing guidance on
what constitutes an ethical violation.

3.2 Privacy and Patient Rights Considerations

The protection of an individual’s health information is not a new con-
cept in the health care field. Even in its nascent state, the medical profes-
sion sought to protect the privacy of patients’ health information through
the practice of confidentiality.24 However, it was not until recently that
the protection of health information was enforced through legislation
in the form of HIPAA. Under Title II of HIPAA, the Privacy Rule man-
dates that all health care providers adhere to a strict set of standards when
collecting, storing, and reporting on health information acquired on the
individuals in their care. In short, Protected Health Information (PHI)

24 GL Higgins, ‘The History of Confidentiality in Medicine’ (April 1989) 35 Canadian Family

Physician, 921, www.ncbi.nlm.nih.gov/pmc/articles/PMC2280818, accessed 23 May 2021.

https://doi.org/10.1017/9781108921923.006 Published online by Cambridge University Press

 social media platforms as public health arbiters 75

associated with an individual in care is not to be shared or made available

to unauthorised individuals or entities without the patient’s clear written
consent. In cases where it is deemed necessary to disclose PHI (e.g. legal
proceedings, threat to oneself or others), only the minimum amount of
information required to satisfy the request should be shared.
By using algorithms like Facebook’s suicide detection algorithm, social
media platforms are behaving like health care providers though not being
held to the same health information protection standards as health care
providers. HIPAA currently has no jurisdiction over the way Facebook
collects, stores, and uses the health information it collects on its users.
And while individuals receiving care from legitimate health care provid-
ers are aware that information is being collected and stored pertaining to
their health and that treatment suggestions will be formulated based on
this information, Facebook users are generally unaware that this type of
information is being collected on them. Furthermore, this health-related
information is being analysed and acted upon without the user’s consent
through the application of the suicide detection algorithm.
In one recent example of Facebook’s suicide detection algorithm erro-
neously identifying a suicide risk, a Facebook user was escorted to a psy-
chiatric hospital by law enforcement for a mental health evaluation despite
no previous history of mental illness or suicidal behaviour and her asser-
tion that she was not experiencing suicidal thoughts.25 Per Facebook’s
protocol for intervention once a suicide risk is detected, law enforcement
was sent to the user’s home and was obliged to follow through with taking
the woman to a psychiatric hospital for evaluation, despite her assertion
that such action was not necessary. As such, the woman’s right to choose
her own treatment course was precluded. As experts have argued that the
use of Facebook’s suicide prevention algorithm in conjunction with an
intervention that may lead to mandatory psychiatric evaluations consti-
tutes the practice of medicine,26 in the case of the aforementioned individ-
ual in which the algorithm produced a false positive, it can be suggested
that the health care ethics principle of autonomy was violated.
Returning to the guidelines delineated in HIPAA, there is also a Security
Rule as it pertains to PHI, and more specifically, electronic PHI (EPHI).

25 N Singer, ‘In Screening for Suicide Risk, Facebook Takes On Tricky Public Health Role’,
New York Times, June 2019, www.nytimes.com/2018/12/31/technology/facebook-suicide-
screening-algorithm.html, accessed 23 May 2021.
26 EH Morreim, ‘Playing Doctor: Corporate Medical Practice and Medical Malpractice’
(1998) 32 University of Michigan Journal of Law Reform, 939.

https://doi.org/10.1017/9781108921923.006 Published online by Cambridge University Press

 76 karen l. celedonia et al.

This rule outlines three areas of security standards regarding safeguarding

EPHI: administrative, physical, and technical. One particularly relevant
point from this rule as it relates to the present chapter is the dictate of
restricted access to EPHI to only those who need it to perform their job.
As there are currently no laws or regulations in the United States around
who has access to information collected by social media platforms like
Facebook, and very recently user data were made available to a third party
for its own private use,27 it can almost be certain that entities other than
social media platforms like Facebook are gaining access to user data.
Furthermore, as monetising information gathered from users is inte-
grated into the business models of social media platforms, it is likely they
are selling user data. Therefore, the privacy of individuals who wish to
keep their mental health status undisclosed or individuals who are falsely
identified as having a mental health concern like suicidal behaviour is
being disregarded.
Much as social media platforms like Facebook are acting like health care
providers when they collect and act upon individuals’ health informa-
tion, one could also claim that by collecting and analysing the same data
they are also engaged in a large-scale research project. Suicide detection
algorithms are not unique to social media platforms: a growing interest in
machine learning and predictive analytics in the health field has led to a
burgeoning corpus of research on predictive algorithms relating to a range
of health issues, including suicidal behaviour. But the distinction between
suicide detection algorithms developed by medical professionals and
Facebook’s suicide detection algorithm lies in the nature of the research
being conducted. Medical professionals and researchers are transparent in
their intentions with the algorithms and make findings publicly available
through venues like peer-reviewed journals. Additionally, medical profes-
sionals and researchers adhere to an ethical code of conduct when engag-
ing in their research, which includes having study protocols reviewed by
an objective institutional review board and obtaining participant consent.
No such review was conducted on Facebook’s suicide detection algorithm
before it was implemented, nor was informed consent obtained by users.
To continue with the research analogy, medical professionals research-
ing a novel pharmaceutical treatment, for example, would not disseminate
the drug to the wider public before the requisite clinical trials on its efficacy

27 CO Schneble, BS Elger and D Shaw, ‘The Cambridge Analytica Affair and Internet
Mediated Research’ (August 2018) 19(8) EMBO Reports, https://doi.org/10.15252/
embr.201846579, accessed 23 May 2021.

https://doi.org/10.1017/9781108921923.006 Published online by Cambridge University Press

 social media platforms as public health arbiters 77

and safety were completed. To do so would be a gross violation of research

ethics. By implementing its suicide detection algorithm before extensive
research was conducted on its effectiveness, rate of false positives, and
adverse consequences associated with false positives, social media plat-
forms like Facebook are in violation of research ethical standards.
Privacy violations which lead to involuntary disclosure of mental
health status are unacceptable, and even in countries where mental illness
is becoming less stigmatised, judgement and misunderstanding of mental
illness still exists. In certain areas of the world, such as the African region
and much of the Asian continent, mental illness is heavily stigmatised
and disclosing a mental health condition like suicidal thoughts or behav-
iour could have severe, often irreversible legal and cultural consequences.
These considerations will be discussed in the following two sections.

3.3 Legal Considerations

Privacy violations committed by the suicide detection algorithm which
lead to involuntary disclosure of mental health status can be distressing
to an individual with mental illness and potentially life-altering in many
areas of the world. But in certain areas of the world, disclosing suicidal
thoughts or behaviour could have legal ramifications. According to a
recent global review conducted in which criminal codes from 192 coun-
tries were obtained and reviewed to determine legal status of suicide and
suicide-related behaviours,28 attempting suicide is illegal in twenty-five
countries and therefore punishable by law. Even in countries where sui-
cide attempts are decriminalised, the law may still be used to punish indi-
viduals who attempt suicide. For example, in the US military, there have
been instances of military personnel who have attempted suicide being
tried and convicted in a court of law and sentenced to jail time.29
In the countries where suicide attempts are illegal, it is primarily because
suicide is seen as a morally or religiously reprehensible act. By making it
illegal to attempt suicide, those who attempt suicide are publicly shamed
through court hearings and sentences with the intent of deterring the
individual from engaging in future attempts. This approach, however, is

28 BL Mishara and DN Weisstub, ‘The Legal Status of Suicide: A Global Review’ (2016) 44
International Journal of Law and Psychiatry, 54–74.
29 A Freilich, ‘Fallen Soldier: Military (In)justice and the Criminalization of Attempted
Suicide After U.S. v. Caldwell’ (September 2014) 19 Berkeley Journal of Criminal Law, 74,
accessed 8 May 2020.

https://doi.org/10.1017/9781108921923.006 Published online by Cambridge University Press

 78 karen l. celedonia et al.

misguided and speaks to the lack of understanding of mental illness in these

countries, as punishments like jail time or other forms of social ostracisation
are anathema to individuals with mental illness and suicidal thoughts and
behaviour, serving to aggravate and intensify symptomology; social isolation
is one of the main contributing factors to suicide attempts.30 Additionally,
after these individuals serve their sentences and are reintegrated back into
society, they may be less likely to seek the treatment they need for their ill-
ness, assuming that appropriate treatment is available in their community.
If suicidal behaviour is therefore brought to the attention of public offi-
cials by a social media suicide detection algorithm, law enforcement will
arrest the individual rather than escort them to a psychiatric hospital for
the necessary treatment of their symptoms. This outcome will not only
harm the individual legally but could also amplify existing mental health
problems and actually increase the risk of the individual attempting sui-
cide rather than preventing it. Furthermore, in the instance where the
apprehended individual is an adolescent, such an early involvement with
the legal system could seriously and permanently alter the course of their
life. In this scenario, the principles of beneficence and non-maleficence
are violated by the implementation of the suicide detection algorithm.

3.4 Cultural Considerations

Mental illnesses and the individuals who experience them have a long his-
tory of being stigmatised. In recent years, HICs have committed to reduc-
ing mental illness stigma by using strategies such as targeted anti-stigma
campaigns.31 These large-scale public health interventions have helped
to decrease mental illness stigma in HICs, but even with these concerted
anti-stigma efforts, stigma still persists in HICs, albeit not as acutely as it
once did.32 Additionally, many employers in HICs include disabilities like

30 World Health Organization, ‘Preventing Suicide: A Global Imperative’ (May 2014), www
.who.int/mental_health/suicide-prevention/world_report_2014/en, accessed 8 May 2020.
31 ACH Szeto and KS Dobson, ‘Reducing the Stigma of Mental Disorders at Work: A
Review of Current Workplace Anti-Stigma Intervention Programs’ (June 2010) 14(1–4)
Applied and Preventive Psychology, 41–56, www.sciencedirect.com/science/article/pii/
S0962184911000047, accessed 23 May 2021, https://doi.org/10.1016/j.appsy.2011.11.002,
accessed 23 May 2021.
32 CC Young and SJ Calloway, ‘Assessing Mental Health Stigma: Nurse Practitioners’
Attitudes Regarding Managing Patients with Mental Health Disorders’ (March 2020)
Journal of the American Association of Nurse Practitioners, https://doi.org/10.1097/
JXX.0000000000000351, accessed 23 May 2021.

https://doi.org/10.1017/9781108921923.006 Published online by Cambridge University Press

 social media platforms as public health arbiters 79

serious mental illness (SMI) in company anti-discrimination policies, fur-

ther protecting the rights of an individual with mental illness should they
choose to disclose their illness.33 Even with this societal progress and legal
safeguards, an individual with a mental illness living in an HIC may still
rather choose not to disclose their diagnosis, for fear of judgement and
discrimination, personally and professionally.
The state of mental illness stigma in LMICs, however, remains in much
need of improvement. This is in large part due to lack of funding for men-
tal health services as well as general disinterest in mental health and indi-
viduals with mental illness.34 As a result, stigma continues to be a very
real, very painful reality for individuals with mental illness in LMICs who
either choose to disclose their mental illness or perhaps have it revealed
without their consent. In one study investigating the presence of stigma
towards individuals with mental illness in India (an LMIC), it was found
that the prevalence of stigma was close to 75 per cent.35 Because of the per-
vasiveness of stigma in LMICs, anti-stigma campaigns like those imple-
mented in HICs have not occurred.
Qualitative studies investigating the experience of stigma in LMICs
have found that individuals with mental illness describe frequent occur-
rences of perceived stigma. The behavioural manifestations of stigma can
range from rejection and derogatory language aimed at the individual
with mental illness to discrimination.36 The harmful effects of mental ill-
ness stigma do not remain isolated to the individual: relatives of individ-
uals with mental illness have reported experiencing stigma due to their
association with their ill relative and, in one study, more than a third of
relatives felt it necessary to conceal their relative’s mental illnesses.37 It is
within this context of mental illness stigma in LMICs that Facebook’s sui-
cide detection algorithm and accompanying intervention pose a serious

33 C Heginbotham, ‘UK Mental Health Policy Can Alter the Stigma of Mental Illness’ (1998)
352(9133) The Lancet, 1052–53.
34 F Mascayano, JE Armijo and LH Yang, ‘Addressing Stigma Relating to Mental Illness in Low-
and Middle-Income Countries’ (March 2015) 6 Front Psychiatry, https://doi.org/10.3389/
fpsyt.2015.00038, accessed 23 May 2021.
35 BT Venkatesh and others, ‘Perception of Stigma Toward Mental Illness in South India’
(2015) 4(3) Journal of Family Medicine and Primary Care, 449.
36 Mascayano, Armijo, and Yang, ‘Addressing Stigma Relating to Mental Illness in Low- and
Middle-Income Countries’.
37 T Shibre and others, ‘Perception of Stigma among Family Members of Individuals with
Schizophrenia and Major Affective Disorders in Rural Ethiopia’ (June 2001) 36(6) Social
Psychiatry and Psychiatric Epidemiology, 299–303, https://doi.org/10.1007/s001270170048,
accessed 23 May 2021.

https://doi.org/10.1017/9781108921923.006 Published online by Cambridge University Press

 80 karen l. celedonia et al.

ethical dilemma. Social media use is just as prevalent in LMICs as HICs,

and in fact, social media use may be even more prevalent in LMICs than
in HICs. Data as recent as October 2020 indicate that nine of the top
ten countries with the most Facebook users are LMICs.38 Depending on
whom the results of a positive suicide risk detected by the algorithm are
reported to, an individual with mental illness living in such settings could
be ostracised from their community and support systems. For example,
if a family member is made privy to the information, they may choose
to dissociate with their ill family member, perhaps even disowning them.
Being disconnected from family is a devastating, stressful situation for any
person, but for individuals with mental illness, such isolation and social
upheaval may exacerbate symptoms. Especially for individuals experienc-
ing suicidal thoughts, discord in interpersonal relationships can intensify
suicidal thoughts and often precipitate suicide attempts.39
At the community level in LMICs, for an individual with mental ill-
ness to have their illness disclosed could either impede their ability to
make a substantial living or result in a total loss of livelihood. In one
study of mental illness and income among South Africans, adults living
with depression or anxiety experienced a mean estimated lost income
of around $5,000 per year.40 It was suggested by the authors of the study
that stigma may contribute to lower earnings for individuals with mental
illness.41 In a qualitative study investigating experiences of stigma in the
Philippines, participants discussed the potential for stigma towards their
mental illness to economically destroy their entire family.42 In the same
study, participants also described how stigma towards their mental illness
reduced their social networks and opportunities, both of which play an
important role in procuring and maintaining gainful employment.43

38 Facebook users by country 2020 | Statista (December 2020), www.statista.com/statistics/

268136/top-15-countries-based-on-number-of-facebook-users, accessed 31 Dec. 2020.
39 BL Robustelli and others, ‘Marital Discord and Suicidal Outcomes in a National Sample of
Married Individuals’ (2015) 45(5) Suicide and Life-Threatening Behavior, 623–32.
40 C Lund and others, ‘Mental Health Services in South Africa: Taking Stock’ (2012) 15(6)
African Journal of Psychiatry, 402–5.
41 C Lund and others, ‘Mental Illness and Lost Income among Adult South Africans’ (2013)
48(5) Social Psychiatry and Psychiatric Epidemiology, 845–51.
42 C Tanaka and others, ‘A Qualitative Study on the Stigma Experienced by People with
Mental Health Problems and Epilepsy in the Philippines’ (October 2018) 18(1) BMC
Psychiatry, 325–13, https://doi.org/10.1186/s12888-018-1902-9, accessed 23 May 2021.
43 A Calvó-Armengol and MO Jackson, ‘The Effects of Social Networks on Employment
and Inequality’ (June 2004) 94(3) American Economic Review, 426–54, https://doi.org/
10.1257/0002828041464542, accessed 23 May 2021.

https://doi.org/10.1017/9781108921923.006 Published online by Cambridge University Press

 social media platforms as public health arbiters 81

To further compound the cultural ramifications of positive suicide risk

detection with the suicide detection algorithm, it is not understood how
well the algorithm performs across various cultures. Though the algo-
rithm is trained to detect key words and phrases related to suicide risk
in English, Spanish, Portuguese, and Arabic, experts in the medical field
question whether the algorithm works equally when applied to different
races, genders, and nationalities.44 Such unknowns leave room for the
possibility of false positives, which in LMICs can be much more detri-
mental to an individual’s life than in an HIC. A false positive in an HIC
may certainly cause inconvenience in the form of an unnecessary hospital
visit, but in an LMIC, a false positive can also result in some or all of the
aforementioned cultural consequences.
The negative cultural consequences of suicidal behaviour being
detected by a suicide detection algorithm like that of Facebook – accurate
or not – among individuals living in LMICs may be particularly damag-
ing to women in these countries. Suicidal ideation and suicidal behaviour
(i.e. attempts) are more prevalent among females compared with males.45
It is therefore likely that a suicide detection algorithm will identify suicide
risk more often in women than in men. In many LMICs, gender equal-
ity is lacking, as evidenced by low scores on the Gender Development
Index.46 Gender inequality creates unfavourable living conditions for
women as they are rendered powerless in male-dominated societies.
Furthermore, in countries where gender inequality exists, women are
more likely to be subjected to physical and sexual abuse,47 both of which
are risk factors for suicidal ideation and behaviour.48 Women in LMICs
already contend with subordinate social status due to gender inequality,
and if the stigma associated with mental illness is thrown into the mix,
women in LMICs with mental illness may find themselves completely

44 M Thielking, ‘Experts Raise Questions about Facebook’s Suicide Prevention Tools – STAT’
(February 2019), www.statnews.com/2019/02/11/facebook-suicide-prevention-tools-ethics-
privacy, accessed 8 May 2020.
45 EK Mościcki, ‘Gender Differences in Completed and Attempted Suicides’ (1994) 4(2)
Annals of Epidemiology, 152–58.
46 AG Dijkstra and LC Hanmer, ‘Measuring Socio-Economic Gender Inequality: Toward
an Alternative to the UNDP Gender-Related Development Index’ (2000) 6(2) Feminist
Economics, 41–75.
47 FT Alloh and others, ‘Mental Health in Low and Middle Income Countries (LMICs):
Going Beyond the Need for Funding’ (2018) 17(1) Health Prospect, 12–17.
48 Y Cohen and others, ‘Physical and Sexual Abuse and Their Relation to Psychiatric Disorder
and Suicidal Behavior among Adolescents Who Are Psychiatrically Hospitalized’ (1996)
37(8) Journal of Child Psychology and Psychiatry, 989–93.

https://doi.org/10.1017/9781108921923.006 Published online by Cambridge University Press

 82 karen l. celedonia et al.

disenfranchised citizens if their illness is disclosed. For women not yet

married, this could mean an end to any prospects of a decent life, as in
many LMICs being married and having children are core aspects of a
woman’s livelihood.
Given that social media use, in particular use of Facebook, is so high
among individuals living in LMICs, it would be prudent to ensure that the
aforementioned cultural considerations within many LMICs are taken
into account when implementing a suicide detection algorithm among this
population of social media users. If we return again to the four principles
of health care ethics and treat social media platforms engaging in public
health-like activities such as suicide detection and accompanying inter-
ventions like health care providers, to ignore the unique cultural context
of LMICs as it relates to mental illness and suicide would be violating the
principles of beneficence and non-maleficence. In these cultural contexts, a
one-size-fits-all approach to suicide detection and intervention is not best
practice and does not take into account what is best for the individual living
with mental illness in these environments that are at best intolerant of indi-
viduals with mental illness, and at worst hostile towards these individuals.

3.5 Conclusion
Suicide detection algorithms implemented on social media platforms
have the potential to prevent death by suicide. With billions of individu-
als across the globe using social media platforms daily, the reach of such
a public health intervention is unprecedented. However, the usefulness,
and indeed the appropriateness, of suicide detection algorithms on social
media platforms needs to be assessed within certain ethical considerations,
as well as the cultural context in which the algorithms are deployed. By
acting as a health care provider through the collection and use of personal
health information via the suicide detection algorithm, social media plat-
forms using this technology should be held to the same ethical standards
as legitimate health care providers.

Acknowledgements
The research for this chapter was supported by a Novo Nordisk Foundation
grant for a scientifically independent Collaborative Research Program in
Biomedical Innovation Law (grant agreement number NNF17SA0027784)
and by the Alexander von Humboldt-Stiftung, Bonn, Germany.

https://doi.org/10.1017/9781108921923.006 Published online by Cambridge University Press

 social media platforms as public health arbiters 83
Bibliography
Alloh FT and others, ‘Mental Health in Low and Middle Income Countries (LMICs):
Going Beyond the Need for Funding’ (2018) 17(1) Health Prospect, 12–17.
Bickmore TW and others, ‘Patient and Consumer Safety Risks When Using
Conversational Assistants for Medical Information: An Observational Study
of Siri, Alexa, and Google Assistant’ (2018) 20(9) Journal of Medical Internet
Research, e11510.
Calvó-Armengol A and Jackson MO, ‘The Effects of Social Networks on
Employment and Inequality’ (June 2004) 94(3) American Economic Review,
426–54, https://doi.org/10.1257/0002828041464542.
Celedonia KL and others, ‘Legal, Ethical, and Wider Implications of Suicide Risk
Detection Systems in Social Media Platforms’ (2021) 8(1) Journal of Law and the
Biosciences, https://doi.org/10.1093/jlb/lsab021.
Chandrashekar P, ‘Do Mental Health Mobile Apps Work: Evidence and
Recommendations for Designing High-Efficacy Mental Health Mobile Apps’
(2018) 4 mHealth, 6, https://doi.org/10.21037/mhealth.2018.03. 02.
Childress JF and Beauchamp TL, Principles of Biomedical Ethics (Oxford University
Press 2001).
Cohen Y and others, ‘Physical and Sexual Abuse and Their Relation to
Psychiatric Disorder and Suicidal Behaviour among Adolescents Who Are
Psychiatrically Hospitalized’ (1996) 37(8) Journal of Child Psychology and
Psychiatry, 989–93.
Dijkstra AG and Hanmer LC, ‘Measuring Socio-Economic Gender Inequality:
Toward an Alternative to the UNDP Gender-Related Development Index’
(2000) 6(2) Feminist Economics, 41–75.
Facebook users by country 2020 | Statista (December 2020), www.statista.com/
statistics/268136/top-15-countries-based-on-number-of-facebook-users.
Freilich A, ‘Fallen Soldier: Military (In)Justice and the Criminalization of Attempted
Suicide after U.S. v. Caldwell’ (September 2014) 19 Berkeley Journal of Criminal
Law, 74, www.bjcl.org/assets/files/19.1-Freilich.pdf, accessed 8 May 2020.
Gardner JS and others, ‘Remote Telepsychiatry Workforce: A Solution to
Psychiatry’s Workforce Issues’ (January 2020) 22(2) Current Psychiatry Reports,
8–9, https://doi.org/10.1007/s11920-020-1128-7.
Heginbotham C, ‘UK Mental Health Policy Can Alter the Stigma of Mental Illness’
(1998) 352(9133) The Lancet, 1052–53.
Heim E and others, ‘Reducing Mental Health Related Stigma in Primary Health
Care Settings in Low- and Middle-Income Countries: A Systematic Review’
(2020) 29 Epidemiology and Psychiatric Sciences, https://doi.org/10.1017/
S2045796018000458.
Higgins GL, ‘The History of Confidentiality in Medicine’ (April 1989) 35 Canadian
Family Physician, 921, www.ncbi.nlm.nih.gov/pmc/articles/PMC2280818.

https://doi.org/10.1017/9781108921923.006 Published online by Cambridge University Press

 84 karen l. celedonia et al.
Hilty DM and others, ‘Telepsychiatry’ (August 2002) 16(8) Molecular Diagnosis and
Therapy, 527–48, https://doi.org/10.2165/00023210-200216080-00003.
Hogan MF, ‘Suicide Prevention: Rising Rates and New Evidence Shape Policy
Options’ in Goldman HH, Frank RG, Morrissey JP (eds) The Palgrave Handbook
of American Mental Health Policy (Springer 2020), 229–57.
Hunt MG and others, ‘No More FOMO: Limiting Social Media Decreases
Loneliness and Depression’ (December 2018) Guilford Publications Inc.,
https://­guilfordjournals.com/doi/abs/10.1521/jscp.2018.37.10.751.
Krendl AC and Pescosolido BA, ‘Countries and Cultural Differences in the
Stigma of Mental Illness: The East–West Divide’ (February 2020) 51(2)
Journal of Cross-Cultural Psychology, 149–67, https://doi.org/10.1177/
0022022119901297.
Larsen ME, Nicholas J and Christensen H, ‘A Systematic Assessment of Smartphone
Tools for Suicide Prevention’ (2016) 11(4) PLOS One, e0152285.
Lund C and others, ‘Mental Illness and Lost Income among Adult South Africans’
(2013) 48(5) Social Psychiatry and Psychiatric Epidemiology, 845–51.
Lund C and others, ‘Mental Health Services in South Africa: Taking Stock’ (2012)
15(6) African Journal of Psychiatry, 402–05.
Mascayano F, Armijo JE and Yang LH, ‘Addressing Stigma Relating to Mental
Illness in Low- and Middle-Income Countries’ (March 2015) 6 Front Psychiatry,
https://doi.org/10.3389/fpsyt.2015.00038.
Mishara BL and Weisstub DN, ‘The Legal Status of Suicide: A Global Review’ (2016)
44 International Journal of Law and Psychiatry, 54–74.
Morreim EH, ‘Playing Doctor: Corporate Medical Practice and Medical
Malpractice’ (1998) 32 University of Michigan Journal of Law Reform, 939.
Mościcki EK, ‘Gender Differences in Completed and Attempted Suicides’ (1994)
4(2) Annals of Epidemiology, 152–58.
Pourmand A and others, ‘Social Media and Suicide: A Review of Technology-Based
Epidemiology and Risk Assessment’ (October 2019) 25(10) Telemedicine and
e-Health, 880–88, https://doi.org/10.1089/tmj.2018.0203.
Rawbone R, ‘Principles of Biomedical Ethics’ (January 2015) 65(1) 7th Edition
Occupational Medicine (Lond), 88–89, https://doi.org/10.1093/occmed/kqu158.
Robustelli BL and others, ‘Marital Discord and Suicidal Outcomes in a National
Sample of Married Individuals’ (2015) 45(5) Suicide and Life-Threatening
Behavior, 623–32.
Rodríguez Herrero P, de la Herrán Gascón A and de Miguel Yubero V, ‘The
Inclusion of Death in the Curriculum of the Spanish Regions’ (2020) 52(1)
Compare: A Journal of Comparative and International Education, 1–19.
Savci M and Aysan F, ‘Relationship between Impulsivity, Social Media Usage and
Loneliness’ (March 2016) 5(2) Educational Process: International Journal, 106–
15, https://doi.org/10.12973/edupij.2016.52.2.

https://doi.org/10.1017/9781108921923.006 Published online by Cambridge University Press

 social media platforms as public health arbiters 85
Schneble CO, Elger BS and Shaw D, ‘The Cambridge Analytica Affair and
Internet Mediated Research’ (August 2018) 19(8) EMBO Reports, https://doi
.org/10.15252/embr.201846579.
Shealy KM and others, ‘Delivering an Evidence-Based Mental Health Treatment to
Underserved Populations Using Telemedicine: The Case of a Trauma-Affected
Adolescent in a Rural Setting’ (2015) 22(3) Cognitive and Behavioral Practice,
331–44.
Sheftall AH and others, ‘Suicide in Elementary School-Aged Children and Early
Adolescents’ (October 2016) 138(4) Pediatrics e20160436, https://doi.org/10.1542/
peds.2016-0436.
Shibre T and others, ‘Perception of Stigma among Family Members of Individuals
with Schizophrenia and Major Affective Disorders in Rural Ethiopia’ (June
2001) 36(6) Social Psychiatry and Psychiatric Epidemiology, 299–303, https://doi
.org/10.1007/s001270170048.
Singer N, ‘In Screening for Suicide Risk, Facebook Takes on Tricky Public Health
Role’, New York Times, June 2019, www.nytimes.com/2018/12/31/technology/
facebook-suicide -screening-algorithm.html.
Solano P and others, ‘A Google-Based Approach for Monitoring Suicide Risk’
(2016) 246 Psychiatry Research, 581–86, www.sciencedirect.com/science/article/
pii/S0165178116301949.
Stravynski A and Boyer R, ‘Loneliness in Relation to Suicide Ideation and
Parasuicide: A Population-Wide Study’ (June 2005) Guilford Publications Inc.,
https://guilfordjournals.com/doi/abs/10.1521/suli.31.1.32.21312.
Szeto ACH and Dobson KS, ‘Reducing the Stigma of Mental Disorders at Work: A
Review of Current Workplace Anti-Stigma Intervention Programs’ (June 2010)
14(1–4) Applied and Preventive Psychology, 41–56, www.sciencedirect.com/
science/article/pii/ S0962184911000047.
Tanaka C and others, ‘A Qualitative Study on the Stigma Experienced by People
with Mental Health Problems and Epilepsy in the Philippines’ (October 2018)
18(1) BMC Psychiatry, 325–13, https://doi.org/10.1186/s12888-018-1902-9.
Thielking M, ‘Experts Raise Questions about Facebook’s Suicide Prevention
Tools – STAT’ (February 2019), www.statnews.com/2019/02/11/facebook-suicide-­
prevention-tools -ethics-privacy.
Venkatesh BT and others, ‘Perception of Stigma Toward Mental Illness in South
India’ (2015) 4(3) Journal of Family Medicine and Primary Care, 449.
Westerlund M, Hadlaczky G and Wasserman D, ‘Case Study of Posts before
and after a Suicide on a Swedish Internet Forum’ (December 2015) 207(6)
British Journal of Psychiatry, 476–82, https://doi.org/10.1192/bjp.bp.114
.154484.
World Health Organization, ‘Preventing Suicide: A Global Imperative’ (May 2014),
www.who.int/mental_health/suicide-prevention/world_report_2014/en.

https://doi.org/10.1017/9781108921923.006 Published online by Cambridge University Press

 86 karen l. celedonia et al.
World Health Organization, ‘Suicide in the World: Global Health Estimates’ (2019)
World Health Organization, https://apps.who.int/iris/handle/10665/326948.
Wright JH and Caudill R, ‘Remote Treatment Delivery in Response to the COVID-
19 Pandemic’ (2020) 89(3) Psychotherapy and Psychosomatics, 1, https://
doi.org/10.1159/000507376.
Young CC and Calloway SJ, ‘Assessing Mental Health Stigma: Nurse Practitioners’
Attitudes Regarding Managing Patients with Mental Health Disorders’ (March
2020) 33(4) Journal of the American Association of Nurse Practitioners, 278–82,
https://doi.org/10.1097/JXX.0000000000000351.

https://doi.org/10.1017/9781108921923.006 Published online by Cambridge University Press