Reducing long-term risks from malevolent

actors 234
by David_Althaus, Tobias_Baumann 37 min read 29th Apr 2020
64 comments
Dystopias & Totalitarianism Less-Discussed Causes Policy Change Forum Prize
Institutional Decision-Making Intelligence and Neuroscience EA Psychology Existential Risk
Frontpage + Add Tag

Summary
Dictators who exhibited highly narcissistic, psychopathic, or sadistic traits were
involved in some of the greatest catastrophes in human history. (More°)

Malevolent individuals in positions of power could negatively a ect humanity’s

long-term trajectory by, for example, exacerbating international con ict or other
broad risk factors. (More°)

Malevolent humans with access to advanced technology—such as whole brain

emulation or other forms of transformative AI—could cause serious existential
risks and su ering risks. (More°)

We therefore consider interventions to reduce the expected in uence of malevolent

humans on the long-term future.

The development of manipulation-proof measures of malevolence seems

valuable, since they could be used to screen for malevolent humans in high-
impact settings, such as heads of government or CEOs. (More°)

We also explore possible future technologies that may o er unprecedented

leverage to mitigate against malevolent traits. (More°)

Selecting against psychopathic and sadistic tendencies in genetically

enhanced, highly intelligent humans might be particularly important.
However, risks of unintended negative consequences must be handled with
extreme caution. (More°)

We argue that further work on reducing malevolence would be valuable from many
moral perspectives and constitutes a promising focus area for longtermist EAs.
(More°)
What do we mean by malevolence?
Before we make any claims about the causal e ects of malevolence, we rst need to
explain what we mean by the term. To this end, consider some of the arguably most evil
humans in history—Hitler, Mao, and Stalin—and the distinct personality traits they
seem to have shared.[1]

Stalin repeatedly turned against former comrades and friends (Hershman & Lieb, 1994,
ch. 15, ch. 18), gave detailed instructions on how to torture his victims, ordered their
loved ones to watch (Glad, 2002, p. 13), and deliberately killed millions through various
atrocities. Likewise, millions of people were tortured and murdered under Mao’s rule,
often according to his detailed instructions (Dikötter, 2011; 2016; Chang & Halliday, ch.
8, ch. 23, 2007). He also took pleasure in watching acts of torture and imitating in what
his victims went through (Chang & Halliday, ch. 48, 2007). Hitler was not only
responsible for the death of millions, he also engaged in personal sadism. On his speci c
instructions, the plotters of the 1944 assassination attempt were hung by piano wires
and their agonizing deaths were lmed (Glad, 2002). According to Albert Speer, “Hitler
loved the lm and had it shown over and over again” (Toland, 1976, p. 818). Hitler, Mao,
and Stalin—and most other dictators—also poured enormous resources into the
creation of personality cults, manifesting their colossal narcissism (Dikötter, 2019).
(The section Malevolent traits of Hitler, Mao, Stalin, and other dictators in Appendix B
provides more evidence.)

Many scienti c constructs of human malevolence could be used to summarize the

relevant psychological traits shared by Hitler, Mao, Stalin, and other malevolent
individuals in positions of power. We focus on the Dark Tetrad traits (Paulhus, 2014)
because they seem especially relevant and have been studied extensively by
psychologists. The Dark Tetrad comprises the following four traits—the more well-
known Dark Triad (Paulhus & Williams, 2002) refers to the rst three traits:

Machiavellianism is characterized by manipulating and deceiving others to further

one’s own interests, indi erence to morality, and obsession with achieving power
or wealth.

Narcissism involves an in ated sense of one’s importance and abilities, an

excessive need for admiration, a lack of empathy, and obsession with achieving
fame or power.

Psychopathy is characterized by boldness, callousness, impulsiveness, a lack of

empathy and guilt, and antisocial behavior, including violence and crime.

Sadism involves deriving pleasure from in icting su ering and pain on others.
There is considerable overlap between the Dark Tetrad traits. In general, almost all
plausible operationalizations of malevolence tend to positively correlate with each other
and negatively with “benevolent” traits such as altruism, humility or honesty. (See the
section Correlations between dark traits and other traits in Appendix B for more details.)

This suggests the existence of a general factor of human malevolence[2]: the Dark Factor
of Personality (Moshagen et al., 2018)—analogous to g, the general factor of intelligence
—characterized by egoism, lack of empathy[3] and guilt, Machiavellianism, moral
disengagement, narcissism, psychopathy, sadism, and spitefulness. Like most
personality traits (Johnson et al., 2008), malevolent traits seem relatively stable over the
lifespan (Obradović et al., 2007) and in uenced by genetic factors (Vernon et al., 2008),
but more on this below.[4]

Throughout this article, we will assume a dimensional—rather than categorical, “black-

or-white”—conception of malevolence. That is, we believe that malevolent traits exist
on a continuum—just like most other human traits such as extraversion or intelligence
(cf. Haslam et al., 2012; Plomin, 2019, ch. 5). Slight Machiavellian or sadistic tendencies,
for example, are common. Many humans seem to atter their superiors and enjoy seeing
(non-tragic) mishaps of their political opponents. But only a few individuals will derive
pleasure from witnessing human torture or will kill their former friends just to
consolidate their power.

It is this latter type of human—showing clear signs of at least some highly elevated Dark
Tetrad traits—who we have in mind when we use the term “malevolent”.

Malevolent humans in power pose serious

long-term risks
In this section we discuss why and how malevolent individuals in highly in uential
positions—such as political leaders or CEOs of notable companies—could negatively
a ect humanity’s long-term trajectory, ultimately increasing existential risks
(including extinction risks) and risks of astronomical su ering (s-risks).

Malevolent humans often rise to power

Malevolent humans are unlikely to substantially a ect the long-term future if they
cannot rise to power. But alas, they often do. The most salient examples are dictators
who clearly exhibited elevated malevolent traits: not only Hitler, Mao, and Stalin, but
also Saddam Hussein, Mussolini, Kim Il-sung, Kim Jong-il, Duvalier, Ceaușescu, and Pol
Pot, among many others.

In fact, people with increased malevolent traits might even be overrepresented among
business (Babiak et al., 2010; Boddy et al., 2010; Lilienfeld, 2014), military, and political
leaders (Post, 2003; Lilienfeld et al., 2012), perhaps because malevolent traits—
especially Machiavellianism and narcissism—often entail an obsession with gaining
power and fame (Kajonius et al., 2016; Lee et al., 2013; Southard & Zeigler-Hill, 2016)
and could even be advantageous in gaining power (Deluga, 2011; Taylor, 2019). Again,
Appendix B provides more details.

History suggests that malevolent leaders have caused

enormous harm
One reason for expecting malevolent humans in power to pose risks to the future is that
they seem to have caused great harm in the past.

Hitler, Mao, and Stalin were directly involved in several of the greatest atrocities in
history, such as World War II, the Holocaust, the Great Leap Forward, the Cultural
Revolution, and the Great Terror. There thus seems to be a correlation between the
malevolence of (autocratic) political leaders and the amount of harm that occurred
under their rule; at least according to our understanding of history.[5] If the past is any
guide to the future, individuals with highly elevated dark traits could again manage to
rise to positions of extreme power and cause extraordinary harm.

However, correlation does not imply causation. Even if we grant that this correlation
indeed exists[6], one could argue that there are better explanations for how these
atrocities came about. In particular, it seems plausible that other factors—such as
political instability or extremist ideologies—matter most. We discuss these issues in
more detail in this section° of Appendix A.

It’s also worth mentioning that individuals with malevolent personalities are more
likely to adopt dangerous ideologies. Dark Tetrad traits are associated with political
extremism generally, including supporting the use of violence to achieve political and
other ideological goals (Duspara & Greitemeyer, 2017; Međedović & Knežević, 2018;
Gøtzsche-Astrup, 2019; Jones, 2013).[7]

Thus, while we agree that history is largely shaped by economic, political, cultural,
institutional, ideological and other systemic forces, we believe that the personality traits
of individual leaders—at the very least in autocratic regimes—can plausibly make a
substantial di erence as well (see also Bertoli et al., 2019; Byman & Pollack, 2001;
especially p. 115-121; Gallagher & Allen, 2014; Jones & Olken, 2005). After all, there were
humans who rose to power within rather autocratic regimes but who nevertheless
enacted relatively bene cial policies. Examples include Juan Carlos I of Spain, Mustafa
Kemal Atatürk, Mikhail Gorbachev, Lee Kuan Yew, and Marcus Aurelius, who seemed to
exhibit more benevolent personality traits than the likes of Hitler, Mao, and Stalin.

Malevolent leaders have the potential to corrupt

humanity’s long-term future
One could question whether malevolent individuals can substantially in uence the long-
term trajectory of humanity for the worse, even from positions of extreme power. It is
possible that they only cause short-term harm, in which case reducing malevolence may
not be a priority from a longtermist° perspective.

However, we believe malevolent leaders plausibly have a signi cant detrimental e ect
on the long-term future. Hitler, Stalin, and Mao, for instance, seemed to have had a
profoundly negative in uence on global a airs and international cooperation, some of
which can arguably still be felt today, more than half a century after the atrocities they
perpetrated.[8] That said, it is di cult—if not impossible—to assess long-term impacts,
as we do not know what would have happened counterfactually.

Broad risk factors due to malevolent leaders

Beckstead (2013) asks whether there is “a common set of broad factors which, if we push
on them, systematically lead to better futures”. It seems plausible that malevolent
humans in power would push such factors in the wrong direction.[9]

Speci cally, we conjecture that malevolent humans in power would a ect the risk
factors below in the following ways:

Increase the spread of political extremism and other dangerous ideologies (see
again Appendix A°).

Exacerbate the risk of great power wars and international con ict (Byman &
Pollack, 2001, particularly p. 112, 134, 137-138; Gallagher & Allen, 2014)[10],
including the risk of nuclear war and arms races involving transformative AI.

Increase the likelihood of the formation of a global totalitarian regime, potentially

resulting in a permanent lock-in of harmful values and power structures.[11]
 Increase the likelihood of reckless behaviour, rather than careful re ection, in
high-stakes situations (for example, those resembling the Cuban Missile Crisis).

Dark Triad traits, psychopathy in particular, are associated with extreme risk-
taking (Hosker-Field et al., 2016; Visser et al., 2014).

Increase intranational con ict and undermine public institutions, social

coordination, collective decision making and general discourse[12], particularly by:

Exacerbating economic and social inequality.[13]

Increasing corruption (Bendahan et al., 2015, table 5), rent-seeking, and the
risk of nancial crises (Boddy, 2011).

Reducing access to information, e.g., through censorship and propaganda.[14]

Reducing trust in government and institutions (Bowler & Karp, 2004).[15]

Such trends would plausibly lead to worse futures in expectation. They also plausibly
increase existential risks (including extinction risks) and su ering risks (see the next
section). However, the evidence linking these risk factors to malevolent humans in
power is fairly weak, for various reasons. We are therefore only somewhat con dent in
these connections.

Existential and su ering risks due to malevolent leaders

In terms of more concrete scenarios, the most extreme risks to the long-term future
would arguably result from malevolent humans with access to highly advanced
technology, particularly transformative AI.

The following list outlines some (non-exhaustive) examples of how malevolent

individuals could increase existential and su ering risks:

As noted above, malevolent individuals tend to exhibit more risk-taking behaviour.

In the context of a project to develop and deploy transformative AI, they are
therefore more likely to ignore potential warning signs and omit precautionary
measures. This increases the risk of misaligned transformative AI.

Malevolent humans are likely less opposed to making threats than the average
human (Jonason et al., 2012; Ullrich et al., 2001)[16] and plausibly less motivated to
pursue peaceful bargaining strategies. Con icts involving malevolent humans are
therefore signi cantly more likely to escalate and result in catastrophic outcomes.
Also, it could be dangerous if AI systems inherit some of their values or heuristics,
 such as an increased willingness to make and carry out threats and/or a reduced
willingness to compromise.

Advanced technology might enable sadistic individuals in power to create su ering

on an unprecedented scale.

A malevolent individual, or a small group of such individuals—e.g., the inner circle

of an autocratic state—might manage to obtain control of Earth (cf. MacAskill,
2020)[17], and eventually the observable universe. For example, imagine Hitler or
Stalin had access to advanced technology—including aligned AGI and mind
uploading, enabling immortality. Such a lock-in of permanent rule by a (global)
malevolent dictator would clearly qualify as an existential risk, as it would thwart
any prospect of a more valuable future. It also constitutes a signi cant s-risk as
there would be nobody left to keep any sadistic tendencies of the dictator in check.

While speci c scenarios are necessarily speculative, it seems clear that malevolent
leaders pose a serious threat to humanity’s long-term future. Of course, malevolent
leaders are not the root of all evil, and many con icts, wars and atrocities would happen
without them. Nevertheless, we believe that preventing malevolent individuals from
rising to power is likely valuable and robustly positive, according to almost all moral
perspectives (compare also Beckstead, 2013; Tomasik, 2013a, 2013b).

Interventions to reduce the in uence of

malevolent actors
Advancing the science of malevolence
Further research into the construct of malevolence and its consequences would allow us
to make more rigorous statements about the links between malevolent leaders and bad
outcomes.

A more established science of malevolence would also help raise awareness of

malevolent personality traits and how to detect them among the general public,
in uencers, politicians, researchers, and academics. Generally, the more we know about
malevolence, the easier it is to accomplish many of the interventions discussed below.

Developing better constructs and measures of malevolence

It seems worthwhile to develop constructs capturing more precisely the constellation of
traits most worrisome from a longtermist perspective, as existing constructs will not
always do so.

For example, one of the most commonly used scales to measure psychopathy, the
Psychopathy Checklist-Revised by Hare et al. (1990), consists of 20 items, grouped into
two factors. Factor 1—characterized by cruelty, grandiosity, manipulativeness, and a
lack of guilt—arguably represents the core personality traits of psychopathy. However,
scoring highly on factor 2—characterized by impulsivity, reactive anger, and lack of
realistic goals—is less problematic from our perspective. In fact, humans scoring high
on factor 1 but low on factor 2 are probably more dangerous than humans scoring high
on both factors (more on this below). Generally, most measures of psychopathy include
items related to increased impulsivity (e.g., Cooke & Michie, 2001; Levenson et al., 1995;
Lilienfeld & Andrews, 1996).

The most dangerous individuals tend to go undiagnosed

Individuals o cially diagnosed as malevolent—e.g. those diagnosed with psychopathy,

antisocial or narcissistic personality disorder—are probably unrepresentative of the
most dangerous individuals. This is because an o cial diagnosis is only made when
somebody su ers from immediate and severe problems (relating to their malevolence)
or was forced to seek therapy, e.g., because they committed a crime.

In contrast, malevolent humans with good impulse-control and otherwise decent

mental health have no reason to seek out a therapist and will generally not be convicted
of crimes. The most dangerous malevolent humans will realize that not being unmasked
as malevolent is of the highest importance, and will have su cient motivation, cunning,
self-awareness, charisma, social skills, intelligence and impulse-control to avoid
detection (Perina et al., 2020).[18] Such individuals might even deliberately display
personality characteristics entirely at odds with their actual personality. In fact, many
dictators did precisely that and portrayed themselves—often successfully—as sel ess
visionaries, tirelessly working for the greater good (e.g., Dikötter, 2019). It may
therefore be very valuable to conduct more research on this hard-to-detect type of
conscientious, strategic malevolence (cf., e.g., Gao & Raine, 2010; Lilienfeld et al., 2015;
Mullins-Sweatt et al., 2010).

Manipulation-proof measures of malevolence

To prevent malevolent humans from reaching highly in uential positions, we need to be
able to reliably detect those traits.
Currently, most measures of dark traits take the form of either interviews or self-
completed questionnaires. Smart malevolent humans can easily manipulate these types
of instruments and evade detection by lying. It is key, therefore, that we develop
manipulation-proof measures of malevolence, i.e., measures that cannot (easily) be
gamed.

One possibility would be to ask peers and previous associates to evaluate the personality
traits of the person in question.[19] Of course, this raises several problems. Malevolent
humans could have charmed and fooled many of their (former) friends and colleagues.
They could also bribe or manipulate others to lie. So, while other-report measures (e.g.,
360 degree assessments) may be harder to manipulate than self-reported ones and are
therefore valuable, they are unlikely to completely solve the problem.[20]

Physiological or neurobiological measures based on methods like EEG or fMRI might be

particularly di cult to manipulate—though this would probably require substantial
technological and scienti c progress. Neuroimaging techniques might allow us to
identify abnormal brain structures or detect suspicious behavior, such as showing
neurological signs of pleasure and/or no distress when seeing other humans or animals
in pain. Therefore, more neurobiological research on the neurological signatures of
pleasure and displeasure (e.g., Berridge & Kringelbach, 2013), and on the neurobiology of
sadism and psychopathy, might be very valuable.[21] (Note that we have not investigated
this in detail, so it is probably best to start with a systematic literature review.) However,
such methods also raise ethical questions about judging people by brain scans rather
than their actual behavior.

Potential misuse and negative consequences

Manipulation-proof measures of malevolence could also be misused—like all

technology. For instance, governments might falsely brand political opponents as
psychopaths.

Another concern is that such tests may constitute an unfair form of discrimination
against humans with certain traits. This is because they measure innate characteristics
that are impossible to change, rather than exclusively considering the actual behaviour
of individuals. Also, even if this is deemed acceptable in the case of malevolence,
advocating for testing in this context might lead to the widespread adoption of
personality testing in general, which some believe could have negative consequences.
(On the other hand, existing selection procedures also implicitly or explicitly select
based on innate traits such as intelligence, and also include various kinds of tests.)
Lastly, unless tests of malevolence have perfect validity and reliability, there will be
measurement errors: Some people will be diagnosed as highly malevolent even though
they aren’t, and some truly malevolent people will escape detection.

How valuable would manipulation-proof measures of malevolence be in practice?

Given the potentially enormous bene ts, why has there been so little interest in the
development of manipulation-proof tests of malevolence? First, doing so is likely
di cult and, especially if it involves neuroscience research, expensive[22] (as an
example, MRI machines cost between $0.3M and $3M). Second, malevolent humans
might, in some cases, actually bene t individual companies or political parties: high
levels of psychopathy and narcissism could be useful for things like negotiating,
motivating employees, or winning public approval. Third, most people likely
overestimate their ability to discern malevolent traits in others, making them less
interested in such tests. Finally, it seems that tests in general are not used much in at
least some contexts; for example, most elected or appointed positions in government do
not require intelligence, knowledge, or personality tests.[23]

One might argue that it was obvious to most people that dictators such as Hitler, Mao,
and Stalin were malevolent even before they gained power, and that manipulation-proof
measures of malevolence would therefore have been useless. However, we are doubtful
that people can easily detect malevolence, at least in the most dangerous types of
individuals, as mentioned above. (See also the section How well can people detect
malevolent traits° in Appendix A for more details.)

Of course, many did suspect that Hitler, Mao, and Stalin were malevolent. However, this
was not common knowledge—and without objective evidence, calling an individual
malevolent can easily be dismissed as slander. Not to mention that anyone making such
accusations risks serious reprisals. So even if a majority had realized early on that Hitler,
Mao, and Stalin are malevolent, it might not have helped.

However, if manipulation-proof and valid measures of malevolence had existed—

alongside strong norms to use them to screen political leaders and widespread trust in
their accuracy—it could have been common knowledge that these individuals were
malevolent, which would have signi cantly reduced their chance of rising to power.
Essentially, manipulation-proof and valid measures of malevolence could serve as an
objective arbiter of good intentions, analogous to the scienti c use of experiments as
objective arbiters of truth. Their role in a hiring process could then be compared to
security clearances, for instance.
It is not clear whether even perfectly diagnostic measures of malevolence would ever
become widespread—for example, because of the abovementioned ethical concerns.
However, for highly in uential positions, people are most willing to make use of all
available evidence (and candidates for such positions have an incentive to provide
credible signals of trustworthiness). For instance, receiving a top-secret security
clearance involves extensive interviews with one’s (former) spouses, colleagues, friends
and neighbors, alongside reviews of medical and psychiatric records, and sometimes
even polygraph examinations. This elaborate and arguably privacy-violating process
would be unacceptable for a routine job, but is considered appropriate given the stakes at
hand. Last, it could already be valuable if only a few companies or government
departments started using manipulation-proof measures of malevolence; near-
universal adoption of such measures is by no means necessary.

Despite these caveats, we believe that work on manipulation-proof measures of

malevolence is promising. Subject to personal t, it may be worthwhile for some
e ective altruists to consider careers in psychology or neuroscience. This would allow
them to advance the science of malevolence, contribute to the development of
manipulation-proof measures of malevolence, and improve their chances to convince
decision-makers to take such measures seriously.

Political interventions
Many factors determine whether an individual can rise to a position of power, and it is
important to include (non-)malevolence as a criterion when selecting leaders. Ideally,
we should establish strong norms against allowing highly malevolent leaders to rise to
power—even in cases where elevated Dark Tetrad traits may be instrumental in
advancing the interests of a company or nation.

While the notion of Dark Tetrad traits is not foremost in most people’s minds, one could
argue that much political debate is about related concepts like the trustworthiness or
honesty of candidates, and voters do value those attributes.[24]

Perhaps the key issue, then, is not a lack of awareness; rather the non-availability of
reliable objective measures and the overestimation of people's ability to detect
malevolence. In fact, humans seem too eager to view their political opponents as
inherently malevolent and ill-intentioned. Conversely however, humans also tend to
view members of their own tribe as inherently good and overlook their misdeeds. (See
again Appendix A° for more details.)
The media also tends to depict impulsive psychopaths—say, ruthless serial killers with a
long history of violence or crime. These are relatively easy to detect, potentially leading
to a false sense of security (compare also Babiak et al., 2010, p.174-175). As mentioned
above°, it may therefore be valuable to raise awareness that at least some types of
malevolent humans are di cult to detect.

Alternatively, we could in uence political background factors that make malevolent

leaders more or less likely. It seems plausible that political instability, especially
outright revolutions, enable malevolent humans to rise to power (Colgan, 2013, p. 662-
665). Generally, democracies plausibly select for more trustworthy, predictable and
benevolent leaders (Byman & Pollack, 2001, p.139-140). Thus, interventions to promote
democracy and reduce political instability seem valuable—though this area seems rather
crowded.

Even within established democracies, we could try to identify measures that avoid
excessive polarization and instead reward cross-party cooperation and compromise.
Mitigating the often highly combative nature of politics would plausibly make it harder
for malevolent humans to rise to power.[25] (For example, e ective altruists have
discussed electoral reform° as a possible lever that could help achieve this.)

Since elevated Dark Tetrad traits are signi cantly more common among men (Paulhus &
Williams, 2002; Plou e et al., 2017), it also seems bene cial to advance gender equality
and increase the proportion of female leaders.

Other potential factors that might facilitate the rise of malevolent individuals include
social and economic inequality, poverty, ethnic, military or religious con icts, and a
“widespread sense of grievance or resentment” (Glad, 2002, p. 4). Thus, identifying
cost-e ective interventions to improve these factors (as well as identifying factors we
haven’t thought of) could be promising. A more thorough study of the history of
malevolent humans rising to power would also be valuable to better understand which
factors are most predictive.

Overall, it seems plausible that many promising political interventions to prevent

malevolent humans from rising to power have already been identi ed and implemented
—such as, e.g., checks and balances, the separation of powers, and democracy itself.
After all, much of political science and political philosophy is about preventing the
concentration of power in the wrong hands.[26] We nevertheless encourage interested
readers to further explore these topics.

Future technologies and malevolence

In this section, we explore how possible future technologies could be used to reduce the
in uence of malevolent actors.

Whole brain emulation

Whole brain emulation is the hypothetical process of scanning the structure of a brain
and replicating it on a computer. Hanson (2016) explores the possible implications of
this technology. In his scenario, brain emulations (“ems”) will shape future economic,
technological and political processes due to their competitive advantage over biological
minds.

One key question is: which human brains will be uploaded? We believe that it would be
crucial to screen potential ems for malevolence—particularly the rst individuals to be
uploaded. Considering the power that the rst ems would likely have to shape this new
“Age of Em”, it could be disastrous for humanity’s long-term future if a malevolent
individual forms the basis for (some of) the rst ems (cf. Bostrom, 2002, p. 12).
Conversely, by screening for malevolence, using manipulation-proof measures, we
could e ectively reduce malevolence among ems. This o ers an unprecedented
opportunity to ensure that malevolent forces have signi cantly less in uence over the
long-term future.[27]

Transformative AI
Many longtermist e ective altruists think that shaping transformative arti cial
intelligence, and in particular solving the alignment problem, is a particularly good lever
to improve the long-term future. Some concrete proposals for alignment—such as
Iterated distillation and ampli cation—involve a “human-in-the-loop” whose
feedback is used to align increasingly capable AI.

In these scenarios, the “human-in-the-loop” plausibly has enormous responsibility

and leverage over the long-term future. It is therefore extremely valuable to ensure that
the relevant individual or individuals—if e.g. a jury or parliament ful lls the role of
“human-in-the-loop”—do not exhibit malevolent traits. (Again, this requires or is at
least facilitated by the availability of manipulation-proof measures.)

Even without human involvement, arti cial agents may exhibit behaviour that
resembles malevolence (to the extent that this notion makes sense in non-human
contexts) if such heuristics prove useful in its training process. After all, the fact that
malevolent traits such as psychopathy or sadism evolved in some humans suggests that
those traits provided tness advantages, at least in certain contexts (Book et al. 2015;
McDonald et al., 2012; Nell, 2006; Jonason et al., 2015).
In particular, it is possible that domain-general capabilities will emerge via increasingly
complex multi-agent interactions (Baker et al., 2019). In this case, it is crucial that the
training environment is set up in a way that prevents the evolution of undesirable traits
like malevolence, and instead rewards cooperative and trustworthy behaviour.

To the extent that arti cial intelligence designs are inspired by the human brain
(“neuromorphic AI”), it seems important to understand the neuroscienti c basis of
malevolence in humans to reduce the risk of neuromorphic AIs also exhibiting
malevolent traits.

Genetic enhancement
A third class of relevant new technologies are those that make it possible to change the
genetic makeup of future humans. This would o er unprecedented leverage to change
personality traits and “human nature”, for better or for worse (cf. Genetic Enhancement
as a Cause Area°). In particular, selection against malevolent traits could signi cantly
reduce the in uence of malevolent individuals.

This is because most variance in adult personality is due to genetic in uences (~30–
50%) and nonshared environment e ects (~35–55%), leaving comparatively little room
for the shared environment (~5–25%)[28] (e.g., Knopik et al., 2018, ch. 16; Johnson et al.,
2008; Plomin, 2019; Vukasović & Bratko, 2015). (See the sections “Broad-sense
heritability estimates of dark traits” and “Is selecting for personality traits possible?” in
Appendix B for more details.) By contrast, nonshared environmental in uences—which
include measurement error, chance life events, and de novo mutations—seem to be
mostly unsystematic, idiosyncratic, and unstable, and therefore di cult to in uence
(Plomin, 2019, ch. 7).

Genetic enhancement technologies might also result in the creation of humans with
extraordinary intelligence (see, e.g., Shulman & Bostrom, 2014, p.2-3). Such humans, if
created, will likely be overrepresented in positions of enormous in uence and would
thus have an outsized impact on the long-term future. Reducing malevolence among
those individuals is therefore especially important.

Overview of genetic enhancement technologies

There are various technologies that would make it possible to modify the genetic
makeup of future humans. We think the following four are most relevant:

In vitro fertilization (IVF) is a process of fertilisation where an egg is combined

with sperm outside the body. In several Western countries, 2-8% of newborns are
 already conceived in this way. While this is primarily used to address infertility, it is
possible to create several fertilized eggs and select among those.

Gene editing (e.g., via CRISPR) is the insertion, deletion, modi cation or
replacement of DNA in the genome of an organism.

Iterated embryo selection (IES, Shulman & Bostrom, 2014; Sparrow, 2013) takes a
sample of embryos and repeats two steps: a) select embryos that are higher in
desired genetic characteristics; b) extract stem cells from those embryos, convert
them to sperm and ova, and cross those to produce new embryos.

Genome synthesis is the arti cial manufacturing of DNA, base pair by base pair.

Note that these methods can interact with each other and should thus not be viewed as
being completely separate. For more details, we highly recommend Gwern’s Embryo
selection for intelligence.

How far away are these technologies? Gwern writes that “IES is still distant and depends
on a large number of wet lab breakthroughs and netuned human-cell protocols.”
Nonetheless, he states that: “[...] it seems clear, at least, that it will certainly not happen
in the next decade, but after that…?”. He concludes that “IES has been badly under-
discussed to date.”

Regarding genome synthesis, Gwern writes that the “cost curve suggests that around
2035, whole human genomes reach well-resourced research project ranges of $10-30m”
and that it “is entirely possible that IES will develop too slowly and will be obsoleted by
genome synthesis in 10-20 years.”

Gwern gives the following summary:

“CRISPR & cloning are already available but will remain unimportant inde nitely for
various fundamental reasons; [...] massive multiple embryo selection is some ways o
but increasingly inevitable and the gains are large enough on both individual & societal
levels to result in a shock; IES will come sometime after massive multiple embryo
selection but it’s impossible to say when, although the consequences are potentially
global; genome synthesis is a similar level of seriousness, but is much more predictable
and can be looked for, very loosely, 2030-2040 (and possibly sooner).”

Dangers

Genetic enhancement is widely criticized. Numerous atrocities have been committed in

the quest to forge a new, “better” kind of human. We would like to emphasize that we do
not advocate for genetic enhancement per se. We only argue that if genetic enhancement
happens, it seems prima facie important to select against malevolent traits—
comparable to the rationale behind di erential intellectual progress and di erential
technological progress.

Still, we are treading dangerous waters. Even just bringing up the possibility of selection
for or against personality traits might inspire misuse of such methods. One particularly
worrisome scenario is selection against all forms of rebellion and independence,
branded as “antisocial tendencies”, which could enable extreme totalitarianism.
Generally, the currently dominant individuals and classes could abuse these powerful
technologies to cement their power.

It is also worth noting that very high levels of usually bene cial traits can be negative:
too much trust, for example, might result in naïvety and an increased likelihood of being
exploited. Similarly, completely eliminating usually harmful traits could back re as
well: for example, in certain situations, some degree of narcissism and
Machiavellianism may bene t entrepreneurs and politicians. Generally, di erent
personality traits are useful for di erent roles in society, so some diversity is bene cial.

For these and other reasons, it could be net negative to shift personality traits by more
than one or two standard deviations.[29] However, we are primarily interested in
shrinking the right tail of the distribution—e.g., by selecting against embryos with
polygenic scores for Dark Tetrad traits above, say, the 99th percentile. This could be
done while only marginally decreasing the mean. Generally, if we apply the (double)
reversal test, current rates of dark traits—particularly highly elevated ones—appear
very far from optimal.

Moreover, many dark traits appear to be genetically correlated with each other and
negatively genetically correlated with benevolent personality traits (Vernon et al., 2008).
Thus, selecting against one dark trait will tend to decrease other dark traits and increase
benevolent traits. This plausibly makes selection e orts more robust, though this could
also have some downsides.[30]

Lastly, we should arguably be especially cautious in scenarios that involve genetically

enhanced humans of extraordinary intelligence. Extremely intelligent sadists and
psychopaths would pose risks that outweigh any plausible bene ts.

Interventions

Funding or otherwise encouraging more research on the genetic basis of malevolent

traits would allow us to better select against these traits. Ideally, we would have a good
understanding of the genetic basis of malevolent traits before technologies such as
genome synthesis arrive. Thus, it is plausibly time-sensitive to do this research now,
even if powerful genetic enhancement technologies will not be developed for the next
several decades.

A particularly cost-e ective intervention might be to convince personal genomics

companies, such as 23andMe, to o er tests of Dark Tetrad traits. 23andMe has over 10
million customers, so even if only a small fraction of customers took these tests (e.g.,
out of curiosity), we would already achieve sample sizes surpassing those of large GWA
studies.[31] Improved psychological measures of malevolence with higher reliability and
validity, as discussed in previous sections, would also enable GWA studies to better
identify genetic variants associated with such traits.

In general, increasing the social acceptability of screening for Dark Tetrad traits
plausibly increases the probability that future projects involving more powerful
technology will also do so. The more established and well-known Dark Tetrad traits are,
and the less controversial their heritability, the easier it would be to accomplish many of
the interventions mentioned above. It might be valuable, for instance, to persuade
sperm banks or other institutions responsible for screening sperm (or egg) donors to
add measures of Dark Tetrad traits to their screening process and display the results
prominently to women choosing sperm donors.

As with non-genetic interventions, we could attempt to raise awareness of malevolent

traits, their heritability, and their dangers. Rather than trying to make changes to the
supply side, it might be easier to increase demand by popularizing Dark Tetrad traits.[32]
Most parents want their children to be responsible, empathic, and kind. If they are
willing to pay for screening for malevolent traits, then sperm banks or others will o er
such services.[33]

However, considering the signi cant dangers outlined above, we believe that public
advocacy of the idea of genetic selection against malevolence would likely be premature.
Indeed, more research on how to best avoid negative consequences—such as increased
inequality or dehumanization of (un)enhanced humans—of possible interventions in
this area would be important.

Subject to personal t, it may also be worthwhile for some e ective altruists to consider
careers in bioinformatics, social sciences relating to GWA studies, bioethics, or related
elds, to be in a good position to later in uence key players.

Concluding remarks
Many of the above interventions face serious technical challenges. It may be hard to
develop manipulation-proof measures of malevolence, and selection on personality
traits is probably di cult due to low additive heritability. In addition, many
interventions—especially those related to genetic enhancement technologies—entail
severe risks of misuse and unintended negative consequences.

However, some of the suggested interventions involve neither speculative future

technology nor controversial ideas about genetic enhancement. Overall, we recommend
a mix of di erent interventions, as well as further work aiming to nd new types of
interventions and checking the assumptions that underlie existing interventions.

Most parents, cultures, and religions feature some notion of "not being evil", so one
could argue that reducing malevolence, broadly construed, is already quite crowded.
However, we believe the interventions we have explored are more targeted, and are
potentially more far-reaching and more neglected than, say, cultural norms or
parenting.

Reducing the in uence of malevolent actors is not a panacea, of course. Many of the
world's biggest problems are not (primarily) due to malevolent intent per se, and
instead are mostly caused by incompetence, irrationality, indi erence, and our inability
to coordinate the escape from undesirable equilibria.

That being said, we believe that reducing the chances of malevolent individuals rising to
power would have substantially positive e ects under a broad range of scenarios and
value systems—whether they place primary importance on avoiding existential risks,
reducing su ering, or improving the quality of the long-term future.

Appendix A
How important are situational factors and ideologies
compared to personality traits?
In this section, we discuss the extent to which historical atrocities can be attributed to
the personality traits of individuals versus structural factors.

First, it seems plausible that background conditions that enable dictatorships in the rst
place—such as political instability and an absent rule of law—also make it more likely
that malevolent humans will rise to power. Individuals who are reluctant to engage in
murder and betrayal, for example, will be at a considerable disadvantage under such
conditions (also see Colgan 2013, especially p. 662-665).
Similarly, power tends to corrupt (e.g., Bendahan et al., 2015; Cislak et al., 2018) so it
could be argued that most individuals who rise to the top within autocratic regimes, will
become more malevolent. Generally, a wealth of social psychology research attests to
the importance of situational factors in explaining human behavior (Milgram, 1963;
Burger, 2009), though the understanding of modern psychology is that behavior
depends on both situational factors and individual personality traits (Bowers, 1973;
Endler & Magnusson, 1976).

One particularly relevant factor is the spread of extremist and fanatical ideologies such
as fascism, violent communism, and fundamentalist religion, which have undoubtedly
contributed to historical atrocities. In fact, such ideologies have plausibly had a much
bigger impact on history than the personality traits of individuals and could pose even
greater risks to the long-term future. So why focus on personality rather than ideology
or structural factors?

For one, tens of millions of people are already combating the dangerous ideologies
mentioned above, or work on ensuring political stability and rule of law. These e orts
are laudable, but also seem very crowded, which suggests that many of the most cost-
e ective interventions have already been identi ed and carried out.

As mentioned above, there is also ample evidence that individuals with malevolent
personalities are drawn to dangerous ideologies:[34] Dark Triad traits predict increased
intention to engage in political violence (Gøtzsche-Astrup, 2019). Narcissism and
psychopathy are associated with political extremism (Duspara & Greitemeyer, 2017).
Sadistic and psychopathic traits predict endorsing a militant extremist mind-set, in
particular the use of violence to achieve political and other ideological goals (Međedović
& Knežević, 2018). Machiavellianism and psychopathy predict racist attitudes, including
support for Neo-Nazis and the KKK (Jones, 2013). Dark Triad traits correlate with social-
dominance orientation (Jones & Figueredo, 2013; Jones, 2013), a measure of an
individual’s preference for economic and social inequality within and between groups
(Pratto et al., 1994; Dallago et al. 2008).[35]

Most ideologies also seem open for interpretation, leaving su cient room for the
idiosyncratic beliefs and personality traits of leaders to make a di erence. Khrushchev
and Gorbachev, for example, while broadly sharing Stalin’s Marxist-Leninist ideology,
have caused much less harm than Stalin. Conversely, as the examples of Stalinism,
Maoism, and Juche show, malevolent individuals can develop an existing ideology
further, making it even more harmful.
In the end, ideologies, belief systems, and personality traits appear inevitably
intertwined. Narcissism, for example, entails in ated beliefs about one’s abilities and
place in history, by de nition. Generally, malevolent individuals tend to hold beliefs that
serve as (un)conscious justi cations for their behavior, such as a sense of entitlement
and grandiosity, and seem more likely to endorse dangerous worldviews and “ideologies
that favor dominance (of individuals or groups)” (Moshagen, 2018, p. 659).

Finally, it is instructive to compare large-scale atrocities to small-scale atrocities like

murder or contract killing. While rates of violent crime surely depend on social
background factors and culturally transmitted norms, psychopathy is also considered a
strong predictor for homicide, including instrumental, calculated murder (Fox & DeLisi,
2019). If we accept that malevolent personality traits like psychopathy play a causal role
in violent crime, it stands to reason that such traits also play at least some causal role in
many large-scale atrocities.

How well can people detect malevolent traits?

Historical evidence suggests that even many of their political adversaries—at least for
some time—did not realize that Hitler, Mao, and Stalin were malevolent, even after they
were in power.

Chamberlain famously trusted Hitler’s sincerity for far too long. Churchill once
remarked that ‘‘Poor Chamberlain believed he could trust Hitler. He was wrong. But I
don’t think that I am wrong about Stalin’’ (Yergin, 1977, p. 65). Similarly, Truman
believed that Stalin ‘‘could be depended upon…I got the impression Stalin would stand
by his agreements’’ (Larson, 1988, p. 246).[36] At least until the 1940s, many Westerners
and Chinese seemed to have been enamored with Mao, potentially partly due to the
in uential book ‘Red Star over China’ (Snow, 1937) which painted him in an extremely
favorable light (Chang & Halliday, ch. 18, 2007).

Countless famous intellectuals—including G.B. Shaw, H.G. Wells, Sartre, Simone de

Beauvoir, Beatrice Webb, Sidney Webb, Susan Sontag, Oswald Spengler, Carl Jung,
Konrad Lorenz, and Martin Heidegger—praised authoritarian leaders like Mussolini,
Hitler, Stalin or Fidel Castro (Hollander, 2016; 2017). Even today, most Russians and
Chinese think highly of Stalin and Mao, respectively.

In summary, it seems that many humans fail to detect malevolent individuals,

particularly when ideological, patriotic or other biases a ect their judgment. Generally,
Hitler, Mao, and Stalin—like many narcissists—seem to have been quite polarizing;
some thought they were obviously malevolent, others viewed them as benevolent, nearly
messianic gures.

Appendix B
See Appendix B: Reducing long-term risks from malevolent actors for additional details.

Acknowledgements
Thanks to Jesse Clifton, Jonas Vollmer, Lukas Gloor, Stefan Torges, Chi Nguyen, Mojmír
Stehlík, Richard Ngo, Pablo Sta orini, Caspar Oesterheld, Lucius Caviola, Johannes
Treutlein, and Ewelina Tur for their valuable comments and feedback. Thanks to So a-
Davis Fogel for copy editing. All errors and views expressed in this document are our
own, not those of the commenters.

David’s work on this post was funded by the Center on Long-Term Risk.

References
Azizli, N., Atkinson, B. E., Baughman, H. M., Chin, K., Vernon, P. A., Harris, E., & Veselka,
L. (2016). Lies and crimes: Dark Triad, misconduct, and high-stakes deception.
Personality and Individual Di erences, 89, 34-39.

Babiak, P., Neumann, C. S., & Hare, R. D. (2010). Corporate psychopathy: Talking the
walk. Behavioral sciences & the law, 28(2), 174-193.

Baker, B., Kanitscheider, I., Markov, T., Wu, Y., Powell, G., McGrew, B., & Mordatch, I.
(2019). Emergent tool use from multi-agent autocurricula. arXiv preprint
arXiv:1909.07528.

Baron-Cohen, S. (2012). The science of evil: On empathy and the origins of cruelty. Basic
books.

Beckstead (2013). How to compare broad and targeted attempts to shape the far future.

Bendahan, S., Zehnder, C., Pralong, F. P., & Antonakis, J. (2015). Leader corruption
depends on power and testosterone. The Leadership Quarterly, 26(2), 101-122.

Berridge, K. C., & Kringelbach, M. L. (2013). Neuroscience of a ect: brain mechanisms of

pleasure and displeasure. Current opinion in neurobiology, 23(3), 294-303.
Bertoli, A., Dafoe, A., & Trager, R. (2019). Leader Age and International Con ict.

Boddy, C. R., Ladyshewsky, R., & Galvin, P. (2010). Leaders without ethics in global
business: Corporate psychopaths. Journal of Public A airs, 10(3), 121-138.

Boddy, C. R. (2011). The corporate psychopaths theory of the global nancial crisis.
Journal of Business Ethics, 102(2), 255-259.

Book, A., Visser, B. A., & Volk, A. A. (2015). Unpacking “evil”: Claiming the core of the
Dark Triad. Personality and Individual Di erences, 73, 29-38.

Bostrom, N., & Ord, T. (2006). The reversal test: eliminating status quo bias in applied
ethics. Ethics, 116(4), 656-679.

Bostrom, N. (2002). Existential risks: Analyzing human extinction scenarios and related
hazards. Journal of Evolution and Technology, 9.

Bowers, K. S. (1973). Situationism in psychology: An analysis and a critique.

Psychological review, 80(5), 307.

Bowler, S., & Karp, J. A. (2004). Politicians, scandals, and trust in government. Political
Behavior, 26(3), 271-287.

Buckels, E. E., Jones, D. N., & Paulhus, D. L. (2013). Behavioral con rmation of everyday
sadism. Psychological science, 24(11), 2201-2209.

Burger, J. M. (2009). Replicating Milgram: Would people still obey today?. American
Psychologist, 64(1), 1.

Burkle, F. M. (2016). Antisocial personality disorder and pathological narcissism in

prolonged con icts and wars of the 21st century. Disaster medicine and public health
preparedness, 10(1), 118-128.

Button, K. S., Ioannidis, J. P., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S., &
Munafò, M. R. (2013). Power failure: why small sample size undermines the reliability of
neuroscience. Nature Reviews Neuroscience, 14(5), 365-376.

Byman, D. L., & Pollack, K. M. (2001). Let us now praise great men: Bringing the
statesman back in. International Security, 25(4), 107-146.

Campbell, W. K., Goodie, A. S., & Foster, J. D. (2004). Narcissism, con dence, and risk
attitude. Journal of behavioral decision making, 17(4), 297-311.
Campbell, W. K., Rudich, E. A., & Sedikides, C. (2002). Narcissism, self-esteem, and the
positivity of self-views: Two portraits of self-love. Personality and Social Psychology
Bulletin, 28(3), 358-368.

Chang, J., & Halliday, J. (2007). Mao: The unknown story. Vintage.

Cislak, A., Cichocka, A., Wojcik, A. D., & Frankowska, N. (2018). Power corrupts, but
control does not: What stands behind the e ects of holding high positions. Personality
and Social Psychology Bulletin, 44(6), 944-957.

Colgan, J. D. (2013). Domestic revolutionary leaders and international con ict. World
Politics, 65(4), 656-690.

Cooke, D. J., & Michie, C. (2001). Re ning the construct of psychopathy: Towards a
hierarchical model. Psychological assessment, 13(2), 171.

Dallago, F., Cima, R., Roccato, M., Ricol , L., & Mirisola, A. (2008). The correlation
between right-wing authoritarianism and social dominance orientation: The
moderating e ects of political and religious identity. Basic and applied social
psychology, 30(4), 362-368.

Deluga, R. J. (2001). American presidential Machiavellianism: Implications for

charismatic leadership and rated performance. The Leadership Quarterly, 12(3), 339-
363.

Dere nko, K. J., & Lynam, D. R. (2006). Convergence and divergence among self-report
psychopathy measures: A personality-based approach. Journal of Personality Disorders,
20(3), 261-280.

Dikötter, F. (2011). Mao's great famine: The history of China's most devastating
catastrophe, 1958-1962. Bloomsbury Publishing.

Dikötter, F. (2016). The Cultural Revolution: A People's History, 1962—1976.

Bloomsbury Publishing.

Dikötter, F. (2019). Dictators: The Cult of Personality in the Twentieth Century.

Bloomsbury Publishing.

Duspara, B., & Greitemeyer, T. (2017). The impact of dark tetrad traits on political
orientation and extremism: an analysis in the course of a presidential election. Heliyon,
3(10), e00425.
Endler, N. S., & Magnusson, D. (1976). Toward an interactional psychology of
personality. Psychological bulletin, 83(5), 956.

Fox, B., & DeLisi, M. (2019). Psychopathic killers: a meta-analytic review of the
psychopathy-homicide nexus. Aggression and violent behavior, 44, 67-79.

Gallagher, M. E., & Allen, S. H. (2014). Presidential personality: Not just a nuisance.
Foreign Policy Analysis, 10(1), 1-21.

Gao, Y., & Raine, A. (2010). Successful and unsuccessful psychopaths: A neurobiological
model. Behavioral sciences & the law, 28(2), 194-210.

Glad, B. (2002). Why tyrants go too far: Malignant narcissism and absolute power.
Political Psychology, 23(1), 1-2.

Gluck, M., Heesacker, M., & Choi, H. D. (2020). How much of the dark triad is accounted
for by sexism? Personality and Individual Di erences, 154, 109728.

Gøtzsche-Astrup, O. (2019). Partisanship and violent intentions in the United States.

Hanson, R. (2016). The Age of Em: Work, Love, and Life when Robots Rule the Earth.
Oxford University Press.

Hare, R. D. (2003). Manual for the Hare Psychopathy Checklist-Revised. Toronto, ON,
Canada: Multi-Health Systems.

Hare, R D., et al. (1990). "The revised psychopathy checklist: reliability and factor
structure." Psychological Assessment: A Journal of Consulting and Clinical Psychology 2,
no. 3: 338.

Harris, G. T., & Rice, M. E. (2006). Treatment of psychopathy. Handbook of psychopathy,

555-572.

Haslam, N., Holland, E., & Kuppens, P. (2012). Categories versus dimensions in
personality and psychopathology: a quantitative review of taxometric research.
Psychological medicine, 42(5), 903-920.

Hershman, D. J., & Lieb, J. (1994). A brotherhood of tyrants: Manic depression and
absolute power. Amherst, NY: Prometheus

Hollander, P. (2016). From Benito Mussolini to Hugo Chavez. Cambridge University

Press.
Hollander, P. (2017). Political pilgrims: Western intellectuals in search of the good
society. Routledge.

Hosker-Field, A. M., Molnar, D. S., & Book, A. S. (2016). Psychopathy and risk taking:
Examining the role of risk perception. Personality and Individual Di erences, 91, 123-
132.

Jain, K., & Bearden, J. N. (2011). Machiavellianism and overcon dence.

Johnson, A. M., Vernon, P. A., & Feiler, A. R. (2008). Behavioral genetic studies of
personality: An introduction and review of the results of 50+ years of research. The SAGE
handbook of personality theory and assessment, 1, 145-173.

Jonason, P. K., Duineveld, J. J., & Middleton, J. P. (2015). Pathology, pseudopathology,

and the Dark Triad of personality. Personality and Individual Di erences, 78, 43-47.

Jonason, P. K., Slomski, S., & Partyka, J. (2012). The Dark Triad at work: How toxic
employees get their way. Personality and individual di erences, 52(3), 449-453.

Jones, B. F., & Olken, B. A. (2005). Do leaders matter? National leadership and growth
since World War II. The Quarterly Journal of Economics, 120(3), 835-864.

Jones, D. N., & Figueredo, A. J. (2013). The core of darkness: Uncovering the heart of the
Dark Triad. European Journal of Personality, 27(6), 521-531.

Jones, D. N. (2013). Psychopathy and machiavellianism predict di erences in racially

motivated attitudes and their a liations. Journal of Applied Social Psychology, 43,
E367–E378. doi:10.1111/jasp.12035

Kajonius, P. J., Persson, B. N., & Jonason, P. K. (2015). Hedonism, achievement, and
power: Universal values that characterize the dark triad. Personality and Individual
Di erences, 77, 173-178.

Kline, P., & Cooper, C. (1984). A construct validation of the Objective-Analytic Test
Battery (OATB). Personality and Individual Di erences, 5(3), 323-337.

Knopik, Neiderhiser, DeFries, Plomin (2018). Behavioral genetics. 7th edition. Macmillan
Learning. New York.

Koehn, M. A., Jonason, P. K., & Davis, M. D. (2019). A person-centered view of prejudice:
The Big Five, Dark Triad, and prejudice. Personality and Individual Di erences, 139, 313-
316.
Larson, D. W. (1988) Problems of Content Analysis in ForeignPolicy Research.
International Studies Quarterly 32 (2): 241–255.

Lee, K., Ashton, M. C., Wiltshire, J., Bourdage, J. S., Visser, B. A., & Gallucci, A. (2013). Sex,
power, and money: Prediction from the Dark Triad and Honesty–Humility. European
Journal of Personality, 27(2), 169-184.

Levenson, M. R., Kiehl, K. A., & Fitzpatrick, C. M. (1995). Assessing psychopathic

attributes in a noninstitutionalized population. Journal of personality and social
psychology, 68(1), 151.

Lilienfeld, S. O., & Andrews, B. P. (1996). Development and preliminary validation of a

self-report measure of psychopathic personality traits in a noncriminal population.
Journal of personality assessment, 66(3), 488-524.

Lilienfeld, S. O., Latzman, R. D., Watts, A. L., Smith, S. F., & Dutton, K. (2014). Correlates
of psychopathic personality traits in everyday life: Results from a large community
survey. Frontiers in psychology, 5, 740.

Lilienfeld, S. O., Waldman, I. D., Land eld, K., Watts, A. L., Rubenzer, S., &
Faschingbauer, T. R. (2012). Fearless dominance and the US presidency: Implications of
psychopathic personality traits for successful and unsuccessful political leadership.
Journal of personality and social psychology, 103(3), 489.

Lilienfeld, S. O., Watts, A. L., & Smith, S. F. (2015). Successful psychopathy: A scienti c
status report. Current Directions in Psychological Science, 24(4), 298-303.

MacAskill, W. (2020). Will MacAskill on the moral case against ever leaving the house,
whether now is the hinge of history, and the culture of e ective altruism. 80000 Hours
Podcast. Jan 24, 2020. https://80000hours.org/podcast/episodes/will-macaskill-
paralysis-and-hinge-of-history/

Macenczak, L. A., Campbell, S., Henley, A. B., & Campbell, W. K. (2016). Direct and
interactive e ects of narcissism and power on overcon dence. Personality and
Individual Di erences, 91, 113-122.

Marcus, D. K., Preszler, J., & Zeigler-Hill, V. (2018). A network of dark personality traits:
What lies at the heart of darkness?. Journal of Research in Personality, 73, 56-62.

Matthews, G., Reinerman-Jones, L. E., Burke, C. S., Teo, G. W., & Scribner, D. R. (2018).
Nationalism, personality, and decision-making. Personality and Individual Di erences,
127, 89-100.
McDonald, M. M., Donnellan, M. B., & Navarrete, C. D. (2012). A life history approach to
understanding the Dark Triad. Personality and Individual Di erences, 52(5), 601-605.

Međedović, J., & Knežević, G. (2018). Dark and Peculiar. Journal of Individual Di erences.

Milgram, S. (1963). Behavioral study of obedience. The Journal of abnormal and social
psychology, 67(4), 371.

Moshagen, M., Hilbig, B. E., & Zettler, I. (2018). The dark core of personality.
Psychological Review, 125(5), 656.

Mullins-Sweatt, S. N., Glover, N. G., Dere nko, K. J., Miller, J. D., & Widiger, T. A. (2010).
The search for the successful psychopath. Journal of Research in Personality, 44(4),
554-558.

Nai, A., & Toros, E. (2020). The peculiar personality of strongmen: comparing the Big
Five and Dark Triad traits of autocrats and non-autocrats. Political Research Exchange,
2(1), 1-24.

Nai, A. (2019). Disagreeable narcissists, extroverted psychopaths, and elections: A new

dataset to measure the personality of candidates worldwide. European Political Science,
18(2), 309-334.

Nell, V. (2006). Cruelty's rewards: The grati cations of perpetrators and spectators.
Behavioral and Brain Sciences, 29(3), 211-224.

O'Connell, D., & Marcus, D. K. (2016). Psychopathic personality traits predict positive
attitudes toward sexually predatory behaviors in college men and women. Personality
and Individual Di erences, 94, 372-376.

Obradović, J., Pardini, D. A., Long, J. D., & Loeber, R. (2007). Measuring interpersonal
callousness in boys from childhood to adolescence: An examination of longitudinal
invariance and temporal stability. Journal of Clinical Child and Adolescent Psychology,
36(3), 276-292.

Paulhus, D. L., & Williams, K. M. (2002). The dark triad of personality: Narcissism,
Machiavellianism, and psychopathy. Journal of research in personality, 36(6), 556-563.

Paulhus, D. L. (2014). Toward a taxonomy of dark personalities. Current Directions in

Psychological Science, 23(6), 421-426.

Perina, K., Hsu, S., Washington, C. (2020). Manifold Podcast.

https://manifoldlearning.com/episode-036/.
Plomin, R. (2019). Blueprint: How DNA makes us who we are. MIT Press.

Plou e, R. A., Saklofske, D. H., & Smith, M. M. (2017). The assessment of sadistic
personality: Preliminary psychometric evidence for a new measure. Personality and
individual di erences, 104, 166-171.

Post, J. M. (2003). Assessing leaders at a distance: The political personality pro le. The
psychological assessment of political leaders: With pro les of Saddam Hussein and Bill
Clinton, 69-104.

Pratto, F., Sidanius, J., Stallworth, L. M., & Malle, B. F. (1994). Social dominance
orientation: A personality variable predicting social and political attitudes. Journal of
personality and social psychology, 67(4), 741.

Preston, O. C., & Anestis, J. C. (2018). Psychopathic traits and politics: Examining
a liation, support of political issues, and the role of empathy. Personality and
individual di erences, 131, 142-148.

Ronson, J. (2012). The psychopath test: A journey through the madness industry.
Riverhead Books.

Shulman, C., & Bostrom, N. (2014). _Embryo Selection for Cognitive Enhancement:
Curiosity or Game-changer? Global Policy, 5(1), 85–92._doi:10.1111/1758-5899.12123.

Southard, A. C., & Zeigler-Hill, V. (2016). The dark triad traits and fame interest: Do dark
personalities desire stardom?. Current Psychology, 35(2), 255-267.

Sparrow, R. (2013). In vitro eugenics. Journal of Medical Ethics, 40(11), 725-731.

Taylor, S. (2019). Pathocracy. When people with personality disorders gain power.
Psychology Today.

Toland, J. (1976). Adolf Hitler. New York: Doubleday.

Tomasik, B. (2013a). Charity Cost-E ectiveness in an Uncertain World. Foundational

Research Institute.

Tomasik, B. (2013b). Di erential Intellectual Progress as a Positive-Sum Project.

Foundational Research Institute.

Ullrich, S., Borkenau, P., & Marneros, A. (2001). Personality disorders in o enders:
categorical versus dimensional approaches. Journal of Personality Disorders, 15(5), 442-
449.
Vernon, P. A., Villani, V. C., Vickers, L. C., & Harris, J. A. (2008). A behavioral genetic
investigation of the Dark Triad and the Big 5. Personality and individual Di erences,
44(2), 445-452.

Visser, B. A., Pozzebon, J. A., & Reina-Tamayo, A. M. (2014). Status-driven risk taking:
Another “dark” personality?. Canadian Journal of Behavioural Science, 46(4), 485.

Vukasović, T., & Bratko, D. (2015). Heritability of personality: a meta-analysis of

behavior genetic studies. Psychological bulletin, 141(4), 769.

Yergin, D. (1977). Shattered peace. Boston: Houghton Mi in Company.

1. Of course, assessing other people’s personality is always fraught with uncertainty, especially if they are
long dead. ↩

2. Other researchers have suggested similar constructs aimed to represent the common core of “evil” (e.g.,
Book et al., 2015; Jones & Figueredo, 2013; Marcus et al., 2018). ↩

3. Baron-Cohen (2012) argues that the de ning feature of human evil is “zero degrees of empathy.” However,
some psychopaths can read other people extremely well and would thus score highly on certain items of the
empathy questionnaires Baron-Cohen describes in his book. Furthermore, as Baron-Cohen acknowledges,
people on the autism spectrum tend to have less empathy—at least certain forms of it—but they are not
more malevolent than the population average. Therefore, reducing malevolence to “zero degrees of
empathy” could be problematic or at least crucially depends on how we de ne and operationalize empathy.
↩
4. As of now, there is no established treatment of malevolence. Harris and Rice (2006) review the empirical
ndings on the treatment of psychopathy but are quite pessimistic about their e ectiveness. ↩

5. A more rigorous analysis would be valuable, though it would also be methodologically challenging—
assessing the personality traits of historical gures, for example, is rather di cult. ↩

6. One reason to be hesitant here is that it seems plausible, for instance, that we, as well as journalists and
historians, see more signs of malevolent personality traits in leaders who have caused great harm, and will
tend to overlook malevolent personality traits in leaders who have done more good. ↩

7. Again, we refer to Appendix A for more details. ↩

8. For example, without Mao and Stalin the probability of a communist China is smaller. A non-communist
China may have better relations with the U.S., and the probability of great power wars and (AI) arms races
may be reduced. However, such claims are necessarily very speculative. For instance, one could also argue
that World War II may have increased longer-term stability by leading to the formation of the UN. ↩

9. Or, as Robert Hare, one of the most well-known researchers of psychopathy, puts it: “Serial killer
psychopaths ruin families. Corporate, political and religious psychopaths ruin economies [and] societies.”
(Ronson, 2012, p. 117). ↩

10. Gallagher & Allen (2014) found that U.S. presidents scoring higher on the Big Five facet “altruism” were
less likely to employ military force. ↩
11. Also compare MacAskill: “I still endorse the view of pushing resources into the future. The biggest caveat
actually I’d have is about the rise of fascism and Stalinism as the thing to push on [...] even though you
might not think that a particular ideology will last forever, well, if it lasts as long until you get like some
eternal lock-in event, then it lasts forever. [...] I kind of think the rise of fascism and Stalinism was a bigger
deal in the 20th century than the invention of nuclear weapons.” (MacAskill, 2020). ↩

12. Dark Triad traits in political candidates correlate with more negative campaigns and fear appeals (Nai,
2019). ↩

13. Since Dark Triad traits correlate with social dominance orientation (SDO, Jones & Figueredo, 2013),
malevolent leaders will, on average, exhibit higher SDO and prefer policies resulting in higher social and
economic inequality. ↩

14. Azizli et al. (2016) nd that psychopathy and Machiavellianism are associated with a greater propensity to
lie and engage in high-stakes deception. ↩

15. Bowler & Karp (2004) nd that scandals involving politicians tend to lower political trust. It seems
plausible that malevolent political leaders are more likely to be involved in scandals. ↩

16. According to Ulrich et al. (2001), the rate of antisocial personality disorder among criminal o enders, 45%
of whom were convicted of “robbery or extortion,” is more than 10 times higher than that of the control
sample. Jonason et al. (2012) also nd that Machiavellianism, narcissism, and psychopathy all correlate
with the use of “hard tactics” in the workplace, including “threats of punishment” (see Table 1, p. 451). ↩

17. MacAskill (2020, emphasis added): “[...] when you look at history of well what are the worst catastrophes
ever? They fall into three main camps: pandemics, war and totalitarianism. Also, totalitarianism or, well,
autocracy has been the default mode for almost everyone in history. And I get quite worried about that. So
even if you don’t think that AI is going to take over the world, well it still could be some individual. And if it
is a new growth mode, I do think that very signi cantly increases the chance of lock-in technology.” ↩

18. Kaja Perina on the Manifold podcast (Perina, 2020): “[M]ost of the studies on psychopaths [...] are done on
inmates. For that reason, we’re forced to conjecture about the really successful ones because I think the
more successful, the more they evade detection, perhaps, lifelong. So there is this disconnect wherein a lot
of them, the violent ones, the less intelligent ones, really end up in jail, and these are the ones who are
studied, but these are not the ones who are highly Machiavellian, necessarily, these are not the ones who
are brilliantly manipulative. These are the ones who are committing violent crimes and get caught.” ↩

19. Extensive background checks, for example with the help of private investigators, would be another
promising possibility. Intelligence agencies do this already for somewhat related purposes. Generally, the
competitive nature of the political process can often uncover past immoral behavior—though swaying
partisan views seems to require evidential strength that is di cult to achieve. Thanks to Mojmír Stehlík for
raising these points. ↩

20. Yet another possibility would be to use “objective” personality tests that don’t rely on self- or other-report
but use actual performance tests to evaluate personality traits (without the test-taker knowing which trait
is supposed to be measured). However, according to our cursory reading of the literature, the few
“objective” personality tests that exist seem to have low validity (e.g., Kline & Cooper, 1984). ↩

21. However, one needs examples to train such predictors in the rst place. One could start by looking for
di erences in the brains of normal people and, say, diagnosed psychopaths, but this metric will be biased
towards diagnosed psychopaths who are at least somewhat unrepresentative of non-diagnosed malevolent
humans (as explained above). One needs to correct for this ascertainment bias. ↩

22. Relatedly, neuroscience research is often underpowered, resulting in low reproducibility of the
accumulated ndings (Button et al., 2013). ↩
23. However, the education system also involves a lot of tests and grading; and is at least somewhat related to
career advancement. Such tests are also common for military entry and sometimes civil service. In the
context of elections, the key question is why voters do not generally demand such tests (including related
objective measures, such as tax returns). ↩

24. However, there exists the frightening possibility that some voters want their political leaders to be at least
moderately malevolent. Most Russians and Chinese, for example, seem to think highly of Stalin and Mao,
respectively—though this is likely at least partly due to propaganda. Generally, many voters seem to like
“strong men” like Putin and overlook or even appreciate elevated Dark Tetrad traits in their political
leaders. Also, according to the (potentially biased) assessment of “experts”, politicians with autocratic
tendencies—many of whom nonetheless received the majority of votes—score signi cantly higher on Dark
Triad traits than the average politician (Nai & Toros, 2020). ↩

25. It is also worth noting that in some forms of government, such as allocating political positions to randomly
selected individuals or hereditary monarchy, those in positions in power are exactly as likely to be
malevolent as the population at large. This may be better than erce competition for positions of power if
the latter advantages the most ruthless and malevolent individuals. On the other hand, good selection
procedures could also reduce malevolence in positions of power below the baseline; and of course this is
only one consideration among many when evaluating di erent forms of government. ↩

26. Thanks to Richard Ngo for making this point. ↩

27. However, initial distributions may change due to competitive pressures or other factors. Even if none of the
rst ems are malevolent, there is no guarantee that malevolence will remain absent in the long run. ↩

28. Extreme events like severe abuse or violence can make a huge di erence for the victims, but such events
are relatively rare and therefore do not explain much variance in the general population (Plomin, 2019). ↩

29. Also note that shifting personality traits by more than this would likely be very di cult even if one wanted
to do this. ↩

30. Some dark traits, such as Machiavellianism, can be bene cial under certain circumstances. It might be
better if one could single out a dark trait, such as sadism, and only select against it while leaving other dark
traits unchanged. ↩

31. However, GWA studies of personality are still fairly weak even at such scales. Even higher sample sizes
might be achieved by identifying proxy variables of malevolence, such as public records on crime. However,
this could easily back re and cause great harm in numerous ways, so one would have to be very careful. ↩

32. Currently, Dark Tetrad traits seem to be neglected in many relevant areas. Most sperm banks measure
traits such as height, attractiveness, physical and mental health but not Dark Tetrad traits. Services that
o er pre-implementation diagnostics screen for all sorts of genetic diseases and some even for IQ but not
for Dark Tetrad traits. Test batteries of enormous government projects like the UK Biobank measure
thousands of variables—including physical health, height, and preferred co ee and cereal type—but they
don’t measure Dark Tetrad traits or even most personality traits in any sort of rigorous manner. ↩

33. Generally, reducing the in uence of malevolent actors can be done in myriad (often mutually reinforcing)
ways. For instance, the more widespread belief systems are that put great value on non-malevolent traits
such as compassion and altruism, the more parents might demand screening for malevolent traits, and the
more future (government) projects will include measures of malevolence in their test batteries. ↩

34. Adopting certain ideologies could also make one more malevolent. However, we think it’s plausible that
most of the correlation is explained by causation from malevolent traits to dangerous ideologies, partly
because personality traits seem less amenable to change than beliefs. ↩
 35. The Dark Triad also predicts sexism (Gluck et al., 2020; O'Connell & Marcus, 2016), nationalism (Matthews
et al., 2018) as well as cognitive and a ective prejudice (Koehn et al., 2019). Psychopathic traits predict
opposition towards free speech and animal rights as well as support for using war as a tool for diplomacy
(Preston & Anestis, 2018, Table 3). Machiavellianism and narcissism also seem to correlate with
overcon dence (Campbell et al., 2002; 2004; Macenczak et al., 2016; Jain & Bearden, 2011), and thus
plausibly negatively correlate with epistemic humility which should serve a protective function against all
kinds of extremism and fanaticism. (Note that we don’t intend to convey that all these associations are
equally dangerous.) ↩

36. Many of these and similar quotes could have been made solely for political reasons, e.g., to strengthen
alliances. ↩

Dystopias & Totalitarianism 9 Less-Discussed Causes 7 Policy Change 3 Forum Prize 3

Institutional Decision-Making 3 Intelligence and Neuroscience 2 EA Psychology 2 Existential Risk 5
Frontpage + Add Tag

234

Pingbacks
241 Problem areas beyond 80,000 Hours' current priorities
221 The case of the missing cause prioritisation research
72 Some history topics it might be very valuable to investigate
59 Common ground for longtermists
57 Venn diagrams of existential, global, and su ering catastrophes
Load More

64 comments, sorted by top scoring Highlighting new comments since 08/03/2020

[-] weeatquince 5mo 41
Hi, interesting article. Thank you for writing.
I felt that this article could have said more about possible policy interventions and that it dismisses
policy and political interventions as crowded too quickly. Having thought a bit about this area in the
past I thought I would chip in.
MORE ON POLICY INTERVENTIONS
Even within established democracies, we could try to identify measures that avoid excessive
polarization and instead reward cross-party cooperation and compromise. ... (For example,
 e ective altruists have discussed electoral reform° as a possible lever that could help achieve
this.)
There are many things that could be done to prevent malevolent leaders within established
democracies. Reducing excessive polarization (or electoral reform) are two minor ones. Other ideas
you do not discuss include:
Better mechanisms for judging individuals. Eg ensuring 360 feedback mechanisms are used
routinely to guide hiring and promotion decisions as people climb political ladders. (I may do
work on this in the not too distant future)
Less power to individuals. Eg having elections for parties rather than leaders. (The Conservative
MPs in the UK could at any time decide that Boris Johnson is no longer t to be a leader and
replace him with someone else, Republicans cannot do this with Trump, Labour MPs in the UK
cannot do this with a Labour leader to the same extent).
Reduce the extent to which corruption / malevolence is bene cial for success. There are many
ways to do this. In particular removing the extent to which individuals raising money is a key
factor for their political success (in the UK most political fundraising is for parties not for
individuals). Also removing the extent to which dishonesty pays, for example with better fact-
checking services.
More checks and balances on power. A second house. A constitution. More independent
government institutions (central banks, regulators, etc – I may do some work in this space soon
too). More transparency of political decision making. Better complaint and whistle-blowing
mechanisms. Limits on use of emergency powers. Etc.
ARE POLITICAL INTERVENTIONS CROWDED?
Alternatively, we could in uence political background factors that make malevolent leaders
more or less likely... interventions to promote democracy and reduce political instability seem
valuable—though this area seems rather crowded.
You might be correct, but this feels a bit like saying the AI safety space is crowded because lots of
groups are trying to develop AI. However it may not be the case that those groups are focusing as
much on safety as you would like. Although there are many groups (especially nation states) that want
to promote democracy there may be very speci c interventions that prevent malevolent leaders that
are signi cantly under-discussed, such as elections for parties rather than leaders, or other points
listed above. It seems plausible that academics and practitioners in this space may be able to make
valuable shifts in the way edgling democracies are developing that are not otherwise being
considered.
And as someone in the improving government institutions space in the UK is is not evident to me that
there is much focus on the kinds of interventions that would limit malevolent leaders.
[-] David_Althaus 5mo 16
Thank you, excellent points!
I will probably add some of your intervention ideas to the article (I'll let you know in that case).
I felt that this article could have said more about possible policy interventions and that it
dismisses policy and political interventions as crowded too quickly.
Sorry about that. It certainly wasn’t our intention to dismiss political interventions out of hand. The
main reason for not writing more was our lack of knowledge in this space; which is why our
discussion ends with “We nevertheless encourage interested readers to further explore these
topics”. In fact, a comment like yours—containing novel intervention ideas written by someone
with experience in policy—is pretty much what we were hoping to see when writing that sentence.
Better mechanisms for judging individuals. Eg ensuring 360 feedback mechanisms are used
routinely to guide hiring and promotion decisions as people climb political ladders. (I may do
work on this in the not too distant future)
Very cool! This is partly what we had in mind when discussing manipulation-proof measures to
prevent malevolent humans from rising to power (where we also brie y mention 360 degree
assessments).
For what it's worth, Babiak et al. (2010) seemed to have some success with using 360 degree
assessments to measure psychopathic traits in a corporate setting. See also Mathieu et al. (2013).
[-] weeatquince 2mo 4
Relevant policy report from the UK Parliament on enforcing the Ministerial Code of good
behaviour, (from 2006):
https://publications.parliament.uk/pa/cm200506/cmselect/cmpubadm/1457/1457.pdf
(I wasn't sure what to do with this when I found it, I might add other policy reports I nd to this
thread too until I have the capacity to actually work on this in any detail)
Less directly relevant but somewhat interesting too:
https://publications.parliament.uk/pa/cm200607/cmselect/cmpubadm/121/121i.pdf

[-] MichaelA 5mo 8

Thanks for sharing these additional ideas and insights!
Also removing the extent to which dishonesty pays, for example with better fact-checking
services.
I ran (and published a paper on) an experiment on fact-checking for my Psychology Honours in 2017,
so I have a smidge of knowledge here, though it's a bit rusty. Some brief thoughts:
I do suspect "better fact-checking services", in the sense of more accuracy or checking more
facts, would be somewhat bene cial
But I think there's some reason for pessimism about just how much people really "care" about
the results of fact-checks, even when they see those fact-checks. Here's my study's abstract:
In the ‘post-truth era’, political fact-checking has become an issue of considerable
signi cance. A recent study in the context of the 2016 US election found that fact-checks of
statements by Donald Trump changed participants' beliefs about those statements—
regardless of whether participants supported Trump—but not their feelings towards Trump
or voting intentions. However, the study balanced corrections of inaccurate statements with
an equal number of a rmations of accurate statements. Therefore, the null e ect of fact-
 checks on participants’ voting intentions and feelings may have arisen because of this
arti cially created balance. Moreover, Trump's statements were not contrasted with
statements from an opposing politician, and Trump's perceived veracity was not measured.
The present study (N = 370) examined the issue further, manipulating the ratio of
corrections to a rmations, and using Australian politicians (and Australian participants)
from both sides of the political spectrum. We hypothesized that fact-checks would correct
beliefs and that fact-checks would a ect voters’ support (i.e. voting intentions, feelings and
perceptions of veracity), but only when corrections outnumbered a rmations. Both
hypotheses were supported, suggesting that a politician's veracity does sometimes matter
to voters. The e ects of fact-checking were similar on both sides of the political spectrum,
suggesting little motivated reasoning in the processing of fact-checks.
This is also relevant to the following sentence from the original post: "While the notion
of Dark Tetrad traits is not foremost in most people’s minds, one could argue that much
political debate is about related concepts like the trustworthiness or honesty of
candidates, and voters do value those attributes." I think this is true, but probably less
true than many might think (or at least than they would've thought pre-2016).
And then there's also the matter of whether people come to encounter fact-checks in the rst
place. In my study's conclusion, I wrote that "participants were unable to avoid fact-checks or
to select which ones they received. In reality, some people may not encounter any fact-checks
at all [9], and the sample of fact-checks which others encounter is often in uenced by
selective exposure and selective sharing [65,66]." (I feel weird about quoting myself, but 2017
Michael knew more about this than 2020 Michael does!)
So I'd tentatively see more value in making fact-checking services "better" in the sense of
being clearer, more attention-grabbing, better publicised, or things like that (as long as this
doesn't cost too much accuracy, nuance, etc.), rather than in e.g. making more or more
accurate fact-checks.
And there may be still more value in somehow "shifting norms" towards valuing truth
more highly, or things like that, though I don't know how one would actually do that.
(I'm guessing this post is relevant, but I haven't read it yet.)

[-] Jc_Mourrat 5mo 5

I've just read the results of an interesting new study on the e ect of red- agging some
information on social media, with ags such as "Multiple fact-checking journalists dispute the
credibility of this news", and variations with "Multiple fact-checking journalists" replaced by,
alternatively, "Major news outlets", "A majority of Americans", or "Computer algorithms using
AI". The researchers tested the e ect this had on the propensity of people to share the content.
The e ect of the "fact-checking" phrasing was the most pronounced, and very signi cant (a
reduction of about 40% of the probability to share content; which jumps to 60% for people who
identify as Democrats). Overall the e ect of the "AI" phrasing was also very signi cant, but quite
counterintuitively it has the e ect of increasing the probability of sharing content for people who
identify as Republicans! (By about 8%; it decreases that same probability by 40% for people who
identify as Democrats.)
https://engineering.nyu.edu/news/researchers- nd-red- agging-misinformation-could-slow-
spread-fake-news-social-media
[-] weeatquince 5mo 5
 Thank you for the insight. I really have no strong view on how useful each / any of the ideas I
suggested were. They were just ideas.
I would add on this point that narcissistic politicians I have encountered worried about appearance
and bad press. I am pretty sure that transparency and fact checking etc discouraged them from
making harmful decisions. Not every narcissistic leader is like Trump.
[-] MichaelA 5mo 3
Yeah, that sounds right to me. And reminds me of a paper I read when working on that
experiment, the abstract of which was:
Does external monitoring improve democratic performance? Fact-checking has come to
play an increasingly important role in political coverage in the United States, but some
research suggests it may be ine ective at reducing public misperceptions about
controversial issues. However, fact-checking might instead help improve political
discourse by increasing the reputational costs or risks of spreading misinformation for
political elites. To evaluate this deterrent hypothesis, we conducted a eld experiment on
a diverse group of state legislators from nine U.S. states in the months before the
November 2012 election. In the experiment, a randomly assigned subset of state
legislators was sent a series of letters about the risks to their reputation and electoral
security if they were caught making questionable statements. The legislators who were
sent these letters were substantially less likely to receive a negative fact-checking rating
or to have their accuracy questioned publicly, suggesting that fact-checking can reduce
inaccuracy when it poses a salient threat.
Relatedly, it could be that "more" or "better" fact-checking would lead to better actions by or
discourse from politicians, even if voters "don't really care much" about fact-checks or never
really see them, due to politicians overestimating what impact fact-checks would have on voters'
perceptions.
(To be clear, I do think fact-checks probably have at least some impact via the more obvious route
too; I wonder mostly about the magnitude of the e ect, not whether it exists.)

[-] Ramiro 5mo 17

Personally, I think this is gold.
On the other hand, I’m not so sure that sadism is particularly worse for the long-term future than the
dark triad traits. Yes, sadistic leaders may pose a special S-risk, but I am not sure they increase other
x-risks so much – e.g., Hitler was kind of cautious concerning the risk of nuclear weapons igniting the
atmosphere, and Stalin was partially useful in WW II and avoided WW III.
Contrast:
Some dark traits, such as Machiavellianism, can be bene cial under certain circumstances. It
might be better if one could single out a dark trait, such as sadism, and only select against it
while leaving other dark traits unchanged.
with these parts of the Appendix A:
Dark Triad traits predict increased intention to engage in political violence (Gøtzsche-Astrup,
2019)…
Historical evidence suggests that even many of their political adversaries—at least for some
time—did not realize that Hitler, Mao, and Stalin were malevolent
Actually, open sadism often raises opposition; and I think the big 3 above weren't salient because they
were sadistic – would they have killed less people if they were just functional psychopaths?
It might be hard to spot dark tetrad individuals, but it’s not so hard to realize an individual is
narcissistic or manipulative. I don’t think people contend that Stalin and Mao were so – they may
argue that they were bene cial, or that they were excused because they were “great man”.
So why do such guys acquire power? Why do people support it? We often dislike acquaintances that
exhibit one of the dark traits; then why do people tolerate it when it comes from the alpha male boss?
My point is that the problem might be precisely that we often claim things like “Some dark traits, such
as Machiavellianism, can be bene cial under certain circumstances. It might be better if one could
single out a dark trait, such as sadism, and only select against it while leaving other dark traits
unchanged.” If we could just acknowledge that dark triad traits individuals are very dangerous, even if
they’re sometimes useful (like a necessary evil), then perhaps we could avoid (or at least be
particularly cautious with) malevolent leaders.
[-] David_Althaus 5mo 18
Thank you, good points.
I agree that it’s not clear whether sadism is really worse than the Dark Triad traits.
e.g., Hitler was kind of cautious concerning the risk of nuclear weapons igniting the
atmosphere,
I’m not sure whether Hitler and Stalin were more sadistic than they were Machiavellian or
narcissistic. If I had to decide, I’d say that their Machiavellianism and narcissism were more
pronounced than their sadism.
Regarding Hitler being “kind of cautious”: It seems plausible to me that Hitler was less cautious than
a less malevolent counterfactual German leader.
What is more, it seems not too unlikely that Hitler would have used nuclear weapons in 1945 if he
had had access to them. For instance, see this quote from Glad (2002, p. 32):
As the Allies closed in on Berlin, [Hitler] wanted churches, schools, hospitals, livestock,
marriage records, and almost anything else that occurred to him to be destroyed. In April
1945 he wanted the entire leadership of the Luftwa e to be summarily hanged. He
considered bringing about the destruction of German cities by announcing the execution of
all Royal Air Force war prisoners, so that there would be massive bombing in reprisal. He may
also have given orders for all wounded German soldiers to be killed. His aim became the
destruction of Germany in the greatest Gotterdaemmerung of history. As Albert Speer said
 about Hitler's obsession with architecture: "Long before the end I knew that Hitler was not
destroying to build, he was building to destroy.
You write:
and Stalin was partially useful in WW II and avoided WW III.
I'm not sure about that. Stalin was kind of Hitler’s ally until Hitler invaded Russia. In other words, if it
weren’t for Hitler (and his overcon dence), Stalin might have even helped Hitler to win WW II
(though Hitler might have lost WW II even with Stalin's support, I don't know). I’m no historian but
Stalin’s response to Hitler’s invasion also seems to have been rather disastrous (see also Glad, 2002,
p.8) and might have ended up in Russia losing WW II.
Resources like Khlevniuk (2015) also suggest that Stalin, if anything, was detrimental to Russia’
military success, at least for the rst years of the war.
It might be hard to spot dark tetrad individuals, but it’s not so hard to realize an individual is
narcissistic or manipulative.
I generally agree but there also seem to exist exceptions. The most strategic individuals know that
narcissism and manipulativeness can be very o -putting so they will actively try to hide these traits.
Stalin, for example, often pretended to be humble and modest, in public as well as in his private life,
at least to some success. My impression from reading the various books listed in the references is
that Stalin and Mao managed to rise to the top, partly because they successfully deceived others into
believing that they are more modest and less capable, ruthless, and strategic than they actually are.
Also, it’s often not clear where justi ed con dence ends and narcissism begins—certainly not to the
admirers of the leader in question. Similarly, what some would see as benign and even necessary
instances of being realistic and strategic, others describe as unprincipled and malicious forms of
manipulativeness. (The Sanders supporters who don't want to vote for Biden come to mind here
even though this example is only tangentially related and confounded by other factors.)
So why do such guys acquire power? Why do people support it? We often dislike
acquaintances that exhibit one of the dark traits; then why do people tolerate it when it
comes from the alpha male boss?
Good questions. Indeed, some people do not only seem to tolerate these traits in leaders, they seem
to be actively drawn to such "strong men" who display certain Dark Tetrad characteristics. I have
some theories for why this is but I don't fully understand the phenomenon.
If we could just acknowledge that dark triad traits individuals are very dangerous, even if
they’re sometimes useful (like a necessary evil), then perhaps we could avoid (or at least be
particularly cautious with) malevolent leaders.
I’m de nitely sympathetic to this perspective.
One important worry here is that di erent movements/political parties seem to be in a social
dilemma. Let’s assume—which seems relatively plausible—that leaders with a healthy dose of
narcissism and Machiavellianism are, on average, really better at things like motivating members of
their movement to work harder, inspiring onlookers to join their movement, creating and taking
 advantage of high-value opportunities, and so on. If one movement decides to exclude members
who show even small signs of Dark Triad traits, it seems plausible that this movement will be at a
disadvantage compared to other movements that are more tolerant regarding such traits.
Conversely, movements that tolerate (or even value) Dark Triad traits might be even more likely to
rise to power. It seems very important to avoid such outcomes.
I’m also somewhat worried about "witch hunts" fueled by simplistic conceptions of malevolence. I
know of several individuals who exhibit at least some level of Machiavellian and narcissistic traits—
many even say so themselves—but whom I’d still love to give more in uence because I believe that
they are “good people” who are aware of these traits and who channel these traits towards the
greater good. (Admittedly, a supporter of, say, Mao might have said the same thing back in the days.)
[-] Ramiro 5mo 7
Thanks for the remarks concerning Hitler and Stalin.
I think it might be quite valuable, for the project as a whole, to better understand why people are
drawn to leaders with features they would not tolerate in peers, such as dark traits.
For one, it’s very plausible that, as you mentioned, the explanation is (a) dark traits are very useful
– these individuals are (almost) the only ones with incentives enough to take risks, get things
done, innovate, etc. Particularly, if we do need things like strategies of mutually assured
destruction, then we need someone credibly capable of “playing hawk”, and it's arguably hard to
believe nice people would do that. This hypothesis really lowers my credence in us decreasing x-
risks by screening for dark traits; malevolent people would be analogous to nukes, and it’s hard to
unilaterally get rid of them.
A competing explanation is that (b) they’re not that useful, they’re parasitical. Dark traits are
uncorrelated with achievement, they just make someone better at outcompeting useful pro-social
people, by, e.g., occupying their corresponding niches, or getting more publicity - and so making
people think (due to representative bias) that bad guys are more useful than they are. That's
plausible, too; for instance, almost no one outside EA and LW communities knows about Arkhipov
and Petrov. If that’s the case, then a group could indeed unilaterally bene t from getting rid of
malevolent / dark trait leaders.
(Maybe I should make clear that I don't have anything against dark triad traits individuals per se,
and I'm as afraid of the possibility of witch hunts and other abuses as everyone else. And even if
dark traits were uncorrelated with capacity for achievement, a group might deprise itself from a
scarce resource by selecting against very useful dark trait individuals, like scientists and
entrepreneurs)

[-] sbehmer 5mo 15

Thanks, it's a very nice article on an important topic. If you're interested, there's a small literature in
political economy called "political selection" (here's an older survey article) . As far as I know they
don't focus speci cally on the extreme lower tail of bad leaders, but they do discuss how di erent
institutional features can lead to di erent types of people gaining power.
[-] David_Althaus 5mo 1
Thanks for the pointer!
[-] Emanuele_Ascani 5mo 15
This is one of the best posts I've read here, wow.
One of the main things that concern me is that malevolent people could appropriate the concept of
malevolence itself and start a witch hunt for people who have nothing to do with malevolence. This
was passingly mentioned when acknowledging that political leaders could brand their opponents as
malevolent. Overall I think this post makes a good job of outlining the pros and cons, but I just wanted
to write this consideration in a comment because it has been somewhat prominent in my mind.
[-] Tobias_Baumann 5mo 8
Thanks for the comment!
I would guess that having better tests of malevolence, or even just a better understanding of it, may
help with this problem. Perhaps a takeaway is that we should not just raise awareness (which can
back re via “witch hunts”), but instead try to improve our scienti c understanding and
communicate that to the public, which hopefully makes it harder to falsely accuse people.
In general, I don’t know what can be done about people using any means necessary to smear political
opponents. It seems that the way to address this is to have good norms favoring “clean” political
discourse, and good processes to nd out whether allegations are true; but it’s not clear what can be
done to establish such norms.

[-] Ben_West 5mo 14

Thanks for this interesting article. Regarding malevolence among business leaders: my impression is
that corporations have rewarded malevolence less over time.
E.g. in the early 1900s you had Frederick Taylor (arguably the most in uential manager of the 20th
century) describing his employees like:
one of the very rst requirements for a man who is t to handle pig iron as a regular
occupation is that he shall be so stupid and so phlegmatic that he more nearly resembles in
his mental make-up the ox than any other type.
Modern executives would never say this about their sta , and no doubt this is partly because what's
said in the boardroom is di erent from what's said in public, but there is a serious sense in which
credibly signaling prosocial behaviors towards your employees is useful. E.g. 80 years later you have
Paul O'Neill, in almost exactly the same industry as Taylor, putting worker safety as his key metric,
because he felt that people would work harder if they felt taken care of by the company.
My guess is that corporations which rely on highly skilled workers bene t more from prosocial
executives, and that it's hard to pretend to be prosocial over a decades-long career, though certainly
not impossible. So possibly one hard-to-fake measure of malevolence is whether you repeatedly
succeed in a corporation where success requires prosociality.
[-] David_Althaus 5mo 4
 Thanks, that's a good example.
my impression is that corporations have rewarded malevolence less over time.
Yeah, I think that's probably true.
Just to push back a little bit, the pessimistic take would be that corporate executives simply have
become better at signalling and public relations. Maybe also partly because the downsides of having
bad PR are worse today compared to, say, the 1920s—back then, people were poorer and consumers
didn't have the luxury to boycott companies whose bosses said something egregious; workers often
didn't have the option to look for another job if they hated their boss, et cetera. Generally, it seems
plausible to me that "humans seem to have evolved to emphasize signaling more in good times than
in bad." (Hanson, 2009).
I wonder if one could nd more credible signals of things like "caring for your employers", ideally in
statistical form. Money invested in worker safety might be one such metric. Salary discrepancies
between employees and corporate executives might be another one (which seems to have gotten
way worse since at least the 1970s) though there are obviously many confounders here.
The decline in child labor might be another example of how corporations have rewarded malevolence
less over time. In the 19th century, when child labor was common, some amount of malevolence (or
at least indi erence) was arguably bene cial if you wanted to run a pro table company. Companies
run by people who refused to employ children for ethical reasons presumably went bankrupt more
often given that they could not compete with companies that used such cheap labor. (On the other
hand, it's not super clear what an altruistic company owner should have done. Many children also
needed jobs in order to be able to buy various necessities—I don't know.)
Maybe this is simply an example of a more general pattern: Periods of history marked by poverty,
scarcity, instability, con ict, and inadequate norms & laws will tend to reward or even require more
malicious behavior, and the least ruthless will tend to be outcompeted (compare again Hanson's
"This is the Dream Time", especially point 4).
[-] Ben_West 5mo 4
I wonder if one could nd more credible signals of things like "caring for your employers",
ideally in statistical form. Money invested in worker safety might be one such metric.
That seems reasonable. Another possibility is looking at bene ts, which have grown rapidly
(though there are also many confounders here).
Something which I can't easily measure but seems more robust is the fraction of "iterated games".
E.g. I would expect enterprise salespeople to be less malevolent than B2C ones (at least towards
their customers), because successful enterprise sales relies on building relationships over years or
decades. Similarly managers are often recruited and paid well because they have a loyal team who
will go with them, and so screwing over that team is not in their self-interest.

[-] Kaj_Sotala 5mo 13

 Malevolent humans with access to advanced technology—such as whole brain emulation or
other forms of transformative AI—could cause serious existential risks and su ering risks.
Possibly relevant: Machiavellians Approve of Mind Upload Technology Directly and Through
Utilitarianism (Laakasuo et al. 2020), though it mainly tested whether machiavellians express moral
condemnation of mind uploading, rather than their interest directly.
In this preregistered study, we have two novel ndings: 1) Utilitarian moral preferences are
strongly and psychopathy is mildly associated with positive approval of MindUpload; and 2)
that Machiavellianism – essentially a calculative self-interest related trait – is strongly
associated with positive approval of Mind Upload, even after controlling for Utilitarianism
and the previously known predictor of Sexual Disgust (and conservatism). In our
preregistration, we had assumed that the e ect would be dependent on Psychopathy
(another Dark Triad personality dimension), rather than Machiavellianism. However, given
how closely related Machiavellianism and Psychopathy are, we argue that the results match
our hypothesis closely. Our results suggest that the perceived risk of callous and sel sh
individuals preferring Mind Upload should be taken seriously, as previously speculated by
Sotala & Yampolskiy (2015)

[-] caspar42 2mo 12

Probably you're already aware of this, but the APA's Goldwater rule seems relevant. It states:
On occasion psychiatrists are asked for an opinion about an individual who is in the light of
public attention or who has disclosed information about himself/herself through public
media. In such circumstances, a psychiatrist may share with the public his or her expertise
about psychiatric issues in general. However, it is unethical for a psychiatrist to o er a
professional opinion unless he or she has conducted an examination and has been granted
proper authorization for such a statement.
From the perspective of this article, this rule is problematic when applied to politicians and harmful
traits. (This is similar to how the right to con dentiality has the Duty to Warn exception.) A quick
Google Scholar search gives a couple of articles since 2016 that basically make this point. For
example, see Lilienfeld et al. (2018): The Goldwater Rule: Perspectives From, and Implications for,
Psychological Science.
Of course, the other important (more empirical than ethical) question regarding the Goldwater rule is
whether "conducting an examination" is a necessary prerequisite for gaining insight into a person's
alleged pathology. Lilienfeld et al. also address this issue at length.

[-] Pivocajs 5mo 12

Some thoughts that occured to me while reading:
1) Research suggestion: From afar, malevolence-detection techniques seem like a better version of
the already-existing tool of top-secret security clearance (or tests similar to it). I am not con dent
about this, but it already seems that if top-secret security clearance was a requirement for holding
important posts, a lot of grief would be avoided (at least where I am from). Yet we generally do not use
this tool. Why is this? I suspect that whatever the answer is, it will apply to malevolence-detection
techniques as well.
2) Potential bottleneck: Suppose you succeed and develop 100% accurate malevolence-detection
technique. I think that, by default, you would have trouble convincing people to use it. ("I mean, what
if I score high on it? You know, I am keeping my dark side in check and I don't plan to become too
in uential either, so my malevolence doesn't really hurt anybody. But the other people don't know
that! If I get branded as malevolent, nobody will talk to me ever, or hire me, or anything!") I conjecture
that the impact of this agenda will be bottlenecked on guring out how to leave the malevolent
people a line of retreat; making sure that if you score high on this, the implications aren't that bad. I
see three reasons for this:
a) non-malevolent people might not know they are non-malevolent, and hence be afraid of this,
b) malevolent-and-know-it people might have enough power to hinder this,
c) reasonable general concerns about any test like this getting out of hand.
3) Relatedly to (2), would it make sense to consider some alternative branding that more accurately
suggests what you intend to do with the concept (and doesn't suggest other things)? Unwieldly
suggestion, to illustrate what I mean: Being publicly known as "potentially too risky to be in a position
of great power" indicates that you shouldn't be a president, but you might still have friends, a spouse,
and a prestigeous job. Being publicly known as "malevolent", however, ... . (Also, it seems plausible
that there are people who are malevolent, but do not endorse being so, similarly to how, I think, there
are paedophiles who wish they weren't so.)
(Also, it might not be obvious from my nitpicking, but I really like the post, thanks for it :-).)
[-] David_Althaus 5mo 10
(Also, it might not be obvious from my nitpicking, but I really like the post, thanks for it :-).)
Thank you. :) No worries, I didn't think you were nitpicking. I agree with many of your points.
[...] if top-secret security clearance was a requirement for holding important posts, a lot of
grief would be avoided (at least where I am from). Yet we generally do not use this tool. Why
is this? I suspect that whatever the answer is, it will apply to malevolence-detection
techniques as well.
One worry with security clearances is that they tend to mostly screen for impulsive behaviors such as
crime and drug use (at least, according to my limited understanding of how these security clearances
work) and would thus often fail to detect more strategic malevolent individuals°.
Also, your claim that “we generally do not use this tool [i.e., security clearances]” feels too strong.
For example, 5.1 million Americans seem to have a security clearance. Sounds like a lot to me. (Maybe
you had a di erent country in mind.)
I conjecture that the impact of this agenda will be bottlenecked on guring out how to leave
the malevolent people a line of retreat; making sure that if you score high on this, the
implications aren't that bad.
 Good point. I guess we weren’t su ciently clear in the post about how we envision the usage of
manipulation-proof measures of malevolence. My view is that their results should, as a general rule,
not be made public and that individuals who are diagnosed as malevolent should not be publicly
branded as such. (Similarly, my understanding is that if someone doesn’t get a top level security
clearance because they, for instance, have a serious psychiatric disorder, they only don't get the job
requiring the security clearance—it's not like the government makes their mental health problems
public knowledge.)
My sense is that malevolent individuals should only be prevented from reaching highly in uential
positions like becoming senators, members of Congress, majors, CEOs of leading AGI companies, et
cetera. In other words, the great majority of jobs would still be open to them.
Of course, my views on this issue are by no means set in stone and still evolving. I’m happy to
elaborate on my reasons for preferring this more modest usage if you are interested.
[-] Pivocajs 5mo 1
Of course, my views on this issue are by no means set in stone and still evolving. I’m happy
to elaborate on my reasons for preferring this more modest usage if you are interested.
I think the more modest usage is reasonable choice.
Maybe you had a di erent country in mind. [regarding top-secret security clearance]
I am Czech. We do have the institute, and use it. But, as far as I know, our president doesn't have it,
and a bunch of other people don't have it. (I.e., it seems that people who need secret information
on a daily basis have it. But you don't need it for many other positions from which you could put
pressure on people who have the clearance.)

[-] JustinShovelain 5mo 9

Nice post!
Here are a couple additional posts that I think are worth checking out by Gwern:
https://www.lesswrong.com/posts/ktr39MFWpTqmzuKxQ/notes-on-psychopathy
https://www.lesswrong.com/posts/Ft2Cm9tWtcLNFLrMw/notes-on-the-psychology-of-power
[-] David_Althaus 5mo 7
Thank you!
Agree, these posts are excellent. For what it's worth, I share Gwern's pessimistic conclusion about
the treatment of psychopathy. Other Dark Tetrad traits—especially if they are less pronounced—
might be more amenable to treatment though I'm not especially optimistic.
However, even if e ective treatment options existed, the problem remains that the most dangerous
individuals are unlikely to ever be motivated to seek treatment (or be forced to do so).
[-] MichaelPlant 5mo 8
Thanks for this write-up, I thought it was really interesting and not something I'd ever considered -
kudos!
I'll now hone in on the bit of this I think needs most attention. :)
It seems you think that one of the essential things is developing and using manipulation-proof
measures of malevolence. If you were very con dent we couldn't do this, how much of an issue would
that be? I raise this because it's not clear to me how such measures could be created or deployed. It
seems you have (1) self-reports, (2) other-reports, (3) objective metrics, e.g. brain scans. If I were
really sneaky, I would just lie or not take the test. If I were really sneaky, I would be able to con others,
at least for a long-time - perhaps until I was in power. Regarding objective measures, there will be
'Minority Report' style objections to actually using them in advance, even if they have high predictive
power (which might be tricky as it relies on collecting good data, which seems to require the consent
of the malevolent).
The area where I see this sort of stu working best is in large organisations, such as civil services,
where the organisations have control over who gets promoted. I'm less optimistic this could work for
the most important cases, political elections, where there is not a system that can enforce the use of
such measures. But it's not clear to me how much of an innovation malevolence tests are over the
normal feedback processes used in large organisations. Even if they could be introduced in politics
somehow, it's unclear how much of an innovation this would be: the public already try to assess
politicians for these negative traits.
It might be worth adding that the reason the Myers-Brigg style personality tests are, so I hear, more
popular in large organisations than the (more predictive) "Big 5" personality test is that Myers-
Briggs has no ostensibly negative dimensions. If you pass round a Big-5 test, people might score
highly on neuroticism or low on openness and get annoyed. If this is the case, which seems likely, I nd
it hard e.g. Google will insist that sta take a test they know will assess them on their malevolence!
As a test for the plausibility of introducing and using malevolence tests, notice that we could already
test for psychopathy but we don't. That suggests there are strong barriers to overcome.
[-] David_Althaus 5mo 5
Thank you, great points.
It seems you think that one of the essential things is developing and using manipulation-
proof measures of malevolence. If you were very con dent we couldn't do this, how much of
an issue would that be?
I wouldn't say it's "essential"—in uencing genetic enhancement° would still be feasible—though
it would certainly be a big problem.
Regarding objective measures, there will be 'Minority Report' style objections to actually
using them in advance, even if they have high predictive power
Yes, that would be an issue.
The area where I see this sort of stu working best is in large organisations, such as civil
services, where the organisations have control over who gets promoted. I'm less optimistic
 this could work for the most important cases, political elections, where there is not a system
that can enforce the use of such measures.
That seems true though it seems at least conceivable that voters will demand such measures in the
future. (As an aside, you mention large organisations but it seems such measures could also be
valuable when used in smaller (non-pro t) organizations?)
But it's not clear to me how much of an innovation malevolence tests are over the normal
feedback processes used in large organisations.
Yeah, true. I guess it's also a matter of how much (negative) weight you put on malevolent traits,
how much of an e ort you make to detect them, and how attentive you are to potential signs of
malevolence—most people seem to overestimate their ability to detect (strategic) malevolence° (at
least I did so before reality taught me a lesson).
It might be worth adding that the reason the Myers-Brigg style personality tests are, so I
hear, more popular in large organisations than the (more predictive) "Big 5" personality test
is that Myers-Briggs has no ostensibly negative dimensions.
Interesting, that seems plausible! I've always been somewhat bewildered by its popularity.
If this is the case, which seems likely, I nd it hard e.g. Google will insist that sta take a test
they know will assess them on their malevolence!
True. I guess measures of malevolence would work best as part of the hiring process (i.e., before one
has formed close relationships).
As a test for the plausibility of introducing and using malevolence tests, notice that we could
already test for psychopathy but we don't. That suggests there are strong barriers to
overcome.
I agree that there are probably substantial barriers to be overcome. On the other hand, it seems that
many companies are using "integrity tests" which go in a similar direction. According to Sacket and
Harris (1984), at least 5,000 companies used "honesty tests" in 1984. Companies were also often
using polygraph examinations—in 1985, for example, about 1.7 million such tests were
administered to (prospective) employees (Dalton & Metzger, 1993, p. 149)—until they became
illegal in 1988. And this even though polygraph tests and integrity tests (as well as psychopathy
tests) can be gamed (rather easily).
I could thus imagine that at least some companies and organizations would start using
manipulation-proof measures of malevolence (which is somewhat similar to the inverse of
integrity) if it was common knowledge that such tests actually had high predictive validity and could
not be gamed.
[-] Milan_Gri es 5mo 2
Regarding objective measures, there will be 'Minority Report' style objections to actually
using them in advance, even if they have high predictive power (which might be tricky as it
relies on collecting good data, which seems to require the consent of the malevolent).
 You could imagine scenarios where we apply objective measures to children, embryos, and/or
couples who are considering having children.
This would avoid some of the free-choice problem around who takes the test, though it doesn't
really get around the Minority Report problem. Also there'd still be selection at the parent level
(some would-be parents would decide to not get tested).
Interesting that we don't do anything like this for psychopathy currently, as far as I know.
(Psychopathy appears to be somewhat genetic.)

[-] MichaelA 5mo 8

Great post, thanks for writing it!
Baron-Cohen (2012) argues that the de ning feature of human evil is “zero degrees of
empathy.” However, some psychopaths can read other people extremely well and would thus
score highly on certain items of the empathy questionnaires Baron-Cohen describes in his
book. Furthermore, as Baron-Cohen acknowledges, people on the autism spectrum tend to
have less empathy—at least certain forms of it—but they are not more malevolent than the
population average. Therefore, reducing malevolence to “zero degrees of empathy” could be
problematic or at least crucially depends on how we de ne and operationalize empathy.
Wikipedia notes that "Empathy is generally divided into two major components":
"A ective empathy, also called emotional empathy:[28] the capacity to respond with an
appropriate emotion to another's mental states.[27] Our ability to empathize emotionally is
based on emotional contagion:[28] being a ected by another's emotional or arousal state."
"Cognitive empathy: the capacity to understand another's perspective or mental state."
And that article further notes:
Although science has not yet agreed upon a precise de nition of these constructs, there is
consensus about this distinction.[36][37] A ective and cognitive empathy are also
independent from one another; someone who strongly empathizes emotionally is not
necessarily good in understanding another's perspective.
And I remember being told during my psych undergrad that "psychopaths" have low levels of
a ective empathy but roughly average levels of cognitive empathy, while people on the autism
spectrum have low levels of cognitive empathy but roughly average levels of a ective empathy. (I
haven't fact-checked this, though, or at least not for years.)
In the context of this post's arguments, I'd guess it's low a ective empathy that is especially
concerning, and perhaps especially when paired with high cognitive empathy.
(I assume you already know this stu , but thought it might be useful for other readers.)
[-] David_Althaus 5mo 6
Thank you! I agree that the distinction between a ective and cognitive empathy is relevant, and that
low a ective empathy (especially combined with high cognitive empathy) seems particularly
concerning. I should have mentioned this, at least in the footnote you quote.
And I remember being told during my psych undergrad that "psychopaths" have low levels
of a ective empathy but roughly average levels of cognitive empathy, while people on the
autism spectrum have low levels of cognitive empathy but roughly average levels of
a ective empathy. (I haven't fact-checked this, though, or at least not for years.)
That sounds right. According to my cursory reading of the literature, psychopathy and all other Dark
Tetrad traits are characterized by low a ective empathy. While all Dark Tetrad traits except for
narcissism also seem to correlate with low cognitive empathy, the correlation with diminished
a ective empathy seems substantially more pronounced (Pajevicc et al., 2017, Table 1; Wai &
Tiliopoulos, 2012, Table 1).[1] As you write, people on the autism spectrum basically show the
opposite pattern (normal a ective empathy, lower cognitive empathy (Rogers et al., 2006; Rueda et
al., 2015).
We focused on the Dark Tetrad traits because they overall seem to better capture the personality
characteristics we nd most worrisome. Low a ective empathy seems a bit too broad of a category
as there are several other psychiatric disorders which don’t seem to pose any substantial dangers to
others but which apparently involve lower a ective empathy: schizophrenia (Bon ls et al., 2016),
schizotypal personality disorder (Henry et al., 2007, Table 2), and ADHD (Groen et al., 2018, Table 2).
[2]

Of course, the simplicity of a unidimensional construct has its advantages. My tentative conclusion
is that the D-factor (Moshagen et al., 2018) captures the most dangerous personalities a bit better
than low a ective empathy—though this probably depends on the precise operationalizations of
these constructs. In any case, more research on (diminished) a ective empathy seems de nitely
valuable as well.

1. Though Jonason and Krause (2013) found that narcissism actually correlates with lower cognitive empathy and showed no
correlation with a ective empathy. ↩
2. This list is not necessarily exhaustive. ↩

[-] MichaelA 5mo 2

Interesting!
Reading your comment, I realised one statement of mine may have been unclear. I wrote:
In the context of this post's arguments, I'd guess it's low a ective empathy that is
especially concerning, and perhaps especially when paired with high cognitive empathy
By that I just meant that, in terms of those two components of empathy, that might be the most
concerning pattern. I didn't meant to imply that that's the key thing we should be concerned about
in relation to personality traits more broadly - the idea that Dark Tetrad traits are a better focus
seems reasonable to me (though I don't bring much independent knowledge to the table on that
point).
[-] Milan_Gri es 5mo 8
In this section, we explore how possible future technologies could be used to reduce the
in uence of malevolent actors.

Psychedelics probably help with this as well (when carefully administered).

Anecdotally, a lot of Western contemplative teachers got started on that path because of psychedelic
experiences (Zen, Tibetan Vajrayana, Vipassana, Advaita Vedanta, Kashmiri Shaivism). These
traditions are extremely prosocial & anti-malevolent.
Less anecdotally, Hendricks et al. 2018 found lifetime psychedelic use correlated with reduced
criminality (survey with n = 480,000). Lyons & Carhart-Harris 2018 found psilocybin decreased
authoritarian political beliefs (small, open-label, prospective study).
There hasn't been a proper RCT yet though.
[-] David_Althaus 5mo 35
There hasn't been a proper RCT yet though.
Pokorny et al. (2017) seems like a relevant RCT. They found that psilocybin signi cantly increased
empathy.
However, even such results don’t make me very optimistic about the use of psychedelics for reducing
malevolence.
The kind of individuals that seem most dangerous (people with highly elevated Dark Tetrad traits
who are also ambitious, productive and strategic) seem less likely to be interested in taking
psychedelics—such folks don't seem interested in increasing their empathy, becoming less
judgmental or having spiritual experiences. In contrast, the participants of the Pokorny et al. study—
like most participants in current psychedelics studies (I think)—wanted to take psychedelics which
is why they signed up for the study.
Moreover, my sense is that psychedelics are most likely to increase openness and compassion in
those who already started out with some modicum of these traits and who would like to increase
them further. I’m somewhat pessimistic that giving psychedelics to highly malevolent individuals
would make them substantially more compassionate. That being said, I'm certainly not con dent in
that assessment.
My intuition is partly based on personal experience and anecdotes but also more objective evidence
like the somewhat disappointing results of the Concord Prison Experiment. However, due to various
methodological aws, I'd be hesitant to draw strong conclusions from this experiment.
Overall, I’d nevertheless welcome further research into psychedelics and MDMA. It would still be
valuable if these pharmaceutical agents “only" increase empathy in individuals who are already
somewhat empathic.
 Anecdotally, a lot of Western contemplative teachers got started on that path because of
psychedelic experiences (Zen, Tibetan Vajrayana, Vipassana, Advaita Vedanta, Kashmiri
Shaivism). These traditions are extremely prosocial & anti-malevolent.
My guess is that most Western contemplative teachers who, as a result of taking psychedelics, got
interested in Buddhism and meditation (broadly de ned) were, on average, already considerably
more compassionate, idealistic, and interested in spiritual questions than the type of practically
oriented, ambitious, malevolent people I worry about.
As an aside, I'm much more optimistic about the use of psychedelics, empathogens, and
entactogens for treating other issues such as depression or PTSD. For example, the early results on
using MDMA for treatment-resistant PTSD seem extremely encouraging (and Doblin's work in
general seems promising).
Aside from the obvious dangers relating to bad trips, psychosis, neurotoxicity (which seems only
relevant for MDMA), et cetera[1], my main worry is that psychedelics sometimes seem to decrease
people’s epistemic and instrumental rationality. I also observed that they sometimes seem to have
shifted people’s interests towards more esoteric matters and led to “spiritual navel-gazing”—of
course, this can be bene cial for people whose life goals are comparatively uninformed.

1. Though my impression is that these risks can be reduced to tolerable levels by taking psychedelics only in appropriate
settings and with the right safety mechanisms in place. ↩

[-] Milan_Gri es 5mo 8

Thanks for this great comment!
I had forgotten about Pokorny et al. 2017 – that's de nitely relevant work.

The kind of individuals that seem most dangerous (people with highly elevated Dark Tetrad
traits who are also ambitious, productive and strategic) seem less likely to be interested in
taking psychedelics—such folks don't seem interested in increasing their empathy,
becoming less judgmental or having spiritual experiences.
A lot of Nazis were interested in the occult, and Mao wrote poetry.

I’m somewhat pessimistic that giving psychedelics to highly malevolent individuals would
make them substantially more compassionate. That being said, I'm certainly not con dent
in that assessment.
More research needed here.
As far as I know, there hasn't been any work done on how psychedelics a ect individuals with high
Dark Tetrad scores.
 ... the Concord Prison Experiment. However, due to various methodological aws, I'd be
hesitant to draw strong conclusions from this experiment.
Yeah, the Concord Prison Experiment is pretty bad. Hendricks et al. 2014 is better work on this,
though it's just a survey.

my main worry is that psychedelics sometimes seem to decrease people’s epistemic and
instrumental rationality.
Do you know where this worry comes from?
Gabay et al. 2019 found that MDMA boosted people's cooperation with trustworthy players in an
iterated prisoner's dilemma, but not with untrustworthy players. I take that as some evidence that
MDMA doesn't acutely harm one's rationality. I don't think there's been similar work done for
other psychedelics.

I also observed that they sometimes seem to have shifted people’s interests towards more
esoteric matters and led to “spiritual navel-gazing”—of course, this can be bene cial for
people whose life goals are comparatively uninformed.
I think spiritual bypassing is a real problem, and unwise psychedelic use can de nitely facilitate it.
Context & integration seems very important.
[-] David_Althaus 5mo 4
A lot of Nazis were interested in the occult, and Mao wrote poetry.
Good point, my comment was worded too strongly. I’d still guess that malevolent individuals are,
on average, less interested in things like Buddhism, meditation, or psychedelics.
Do you know where this worry [that psychedelics sometimes seem to decrease people’s
epistemic and instrumental rationality] comes from?
Gabay et al. 2019 found that MDMA boosted people's cooperation with trustworthy
players in an iterated prisoner's dilemma, but not with untrustworthy players. I take that
as some evidence that MDMA doesn't acutely harm one's rationality.
Interesting paper! Though I didn’t have MDMA in mind; with “psychedelics” I meant substances
like LSD, DMT, and psilocybin. I also had long-term e ects in mind, not immediate e ects. Sorry
about the misunderstanding.
One reason for my worry is that people who take psychedelics seem more likely to believe in
paranormal phenomena (Luke, 2008, p. 79-82). Of course, correlation is not causation. However,
it seems plausible that at least some of this correlation is due to the fact that consuming
psychedelics occasionally induces paranormal experiences (Luke, 2008, p. 82 .) which
presumably makes one more likely to believe in the paranormal. This would also be in line with my
personal experience.
 Coming back to MDMA. I agree that the immediate, short-term e ects of MDMA are usually
extremely positive—potentially enormous increases in compassion, empathy, and self-
re ection. However, MDMA’s long-term e ects on those variables seem much weaker, though
potentially still positive (see Carlyle et al. (2019, p. 15).
Overall, my sense is that MDMA and psychedelics might have a chance to substantially decrease
malevolent traits if these substances are taken with the right intentions and in a good setting—
ideally in a therapeutic setting with an experienced guide. The biggest problem I see is that most
malevolent people likely won’t be interested in taking MDMA and psychedelics in this way.
[-] Milan_Gri es 4mo 3
Overall, my sense is that MDMA and psychedelics might have a chance to substantially
decrease malevolent traits if these substances are taken with the right intentions and in
a good setting—ideally in a therapeutic setting with an experienced guide. The biggest
problem I see is that most malevolent people likely won’t be interested in taking MDMA
and psychedelics in this way.

Our estimates of the likelihood of malevolent people being interested probably hinge on our
theory of where malevolence comes from.
e.g. if we think malevolence mostly arises as a maladaptive coping response to early trauma, you
could imagine interventions that resolve the trauma and replace the maladaptive response with
a more prosocial & equally t response (and malevolent people being interested in those
interventions).
But if we think malevolence is mostly a genetically-mediated trait, it's probably harder to
change.
I haven't poked the literature on this yet.

[-] alexherwix 4mo 6

Thanks for writing this post, very interesting! I haven't read all of the comments but wanted to share
one point that came to me over and over again while reading the post. Apologies if it has already been
mentioned in another comment.
It seems like you assume a strong (and relatively simple) causal relationship from genetics to
malevolent traits to bad behavior. I think this view might make the problem seem more tractable than
it actually might be. Humans are complex systems that are nested in other complex systems and
everything is driven by complex and interacting feedback loops. Thus, to me it seems very di cult to
untangle causality here. To me it would be much more intuitive to think about malevolence as a
dynamic phenomenon that is emergent based on a history of interactions rather than a static
personality trait. If you accept this characterization as plausible the task of screening for malevolence
in a valid and reliable way seems much more di cult than just designing a better personality test. I
think the main di erence between those two perspectives is that in the simple case you have a lot of
corner cases to keep in mind (e.g., what is if people have malevolent traits but actually want to be
good people?) whereas the complex case is more holistic but also much more, well, complex and likely
less tractable.
Nevertheless, I agree with the general premise of the post that mental health is an important aspect
in the context of X/S-risk related activities. I would go even further than this post and argue that
mental health in the context of X/S-risk related activities in general is a very pressing cause area that
would score quite well in terms of ITN-Analysis. Thus, I would really love to see an organization or
network being set up dedicated to the serious exploration of this area because existing e orts in the
mental health space seem only to be focus on happiness in the context of global development. If some
interested in this topic reads this, don't hesitate to reach out, I would love to support such e orts.

[-] peter_janicki 5mo 6

applause. thx for this highly interesting (and important) article.
while reading i thought about a lot of commenting, but you already considered most of these things…
still some minor comments:
as to narcissism (maybe (?) the least important "dark tetrad trait"): as for narcistic personality
disorder, there is a reason why (some) of these people are trying to gain power. a huge lack of self-
esteem etc. and some narcistic people are trying to ll this lack with a successful career etc., still it
does not gets lled this way, so (some of them) are trying to become even more successful... a vicious
circle. but maybe even more important for narcissism usually has it´s origin in childhood/ growing up. i
don´t believe, that genetic (epigenetic?) factors attribute a lot to this trait. (I only found that one here,
and I don’t know, whether it is good or representative:
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3973692/ - still, it seems, that those twins grew up
with the same parents …).
as for scanning for dark tetrad-traits: smart malevolent people might deceive tests/scanning
procedures, once they are smart enough, once they know what they want. they usually don´t do so,
when still in kindergarten, in elementary school. And it is sure (unprecise) to scan for example for
those kids, who torture animals (or peers) etc. i am not saying: “let´s do this”, only saying: “looking at
kids might makes sense”.
as for proposing to cut f. e. only the 1% with the highest polygenetic score for dark tetrad-traits: it be
interesting, if this would include persons like hitler, stalin, mao, … they sure had some luck with
timing, being at the right place, living in instable times, … maybe they would only be in the top 10th
percentile? still, it sure be a bad idea, to make big cuts without more reasoning and knowledge.
and: actually, even though (or because) i just read some things about dictators and sadism... i am not
convinced. most journalists argue, that being responsible for the deaths of millions makes one a
sadist. or because one makes up some torture-routines. but from my understanding, sadism is to gain
direct bene t through direct torturing, humuliating (and seeing/getting feedback on their reaction)
of humans (or animals). if someone lets millions of people getting killed or tortured - i guess this is for
other reasons (economic pro t, consolidating power - even if through terror or because someone
simply does not care). having a lack of empathy, it might easily sound like a reasonable idea to send
millions to deaths, if there are other things to gain. actually my understanding of sadism is not
important. but if those dictators haven´t been sadists, then it might be a good idea, to exclude sadism
out of this topic.
still: up until now in this comment, there are 3 lines of praise for this article, and some other 20 lines...
It should be the other way round. thx again.
[-] Tobias_Baumann 5mo 2
Thanks for commenting!
I agree that early detection in children is an interesting idea. If certain childhood behaviours can be
shown to reliably predict malevolence, then this could be part of a manipulation-proof test.
However, as you say, there are many pitfalls to be avoided.
I am not well versed in the literature but my impression is that things like torturing animals,
bullying, general violence, or callous-unemotional personality traits (as assessed by others) are
somewhat predictive of malevolence. But the problem is that you'll probably also get many false
positives from those indicators.
Regarding environmental or developmental interventions, we write this in Appendix B:
Malevolent personality traits are plausibly exacerbated by adverse (childhood)
environments—e.g. ones rife with abuse, bullying, violence or poverty (cf. Walsh & Wu,
2008). Thus, research to identify interventions to improve such environmental factors
could be valuable. (However, the relevant areas appear to be very crowded. Also, the shared
environment appears to have a rather small e ect on personality, including personality
disorders (Knopik et al., 2018, ch. 16; Johnson et al., 2008; Plomin, 2019; Torgersen, 2009).)
Perhaps improving parenting standards and childhood environments could actually be a fairly
promising EA cause. For instance, early advocacy against hitting children may have been a pretty
e ective lever to make society more civilised and less violent in general.

[-] EdoArad 5mo 6

Thanks for a very thorough and interesting report!
It seem plausible that institutional mechanisms that prevent malevolent use of power may work well
today in democracies. I think that the comparison is very important for understanding the value of the
suggested interventions. You have brie y touched this -
Overall, it seems plausible that many promising political interventions to prevent malevolent
humans from rising to power have already been identi ed and implemented—such as, e.g.,
checks and balances, the separation of powers, and democracy itself. After all, much of
political science and political philosophy is about preventing the concentration of power in
the wrong hands.[26]° We nevertheless encourage interested readers to further explore these
topics.
If these mechanisms are actually working quite well today, this somewhat lowers the importance of
the suggested interventions. The analysis given above is mostly for non-modern institutions, but
perhaps the court system, democracy and transparency has evolved so that malevolent actors can not
really do much harm (or that it will be harder for them to get in power).
Also, the major alternative to reducing the in uence of malevolent actors may be in the institutional
decision making itself, or some structural interventions. AI Governance as a eld seems to mostly go
in that route, for example.
That said, I think that e orts going into your suggested interventions are largely orthogonal to these
alternatives (and might actually be supportive of one another). Also, I intuitively nd your arguments
quite compelling.
[-] David_Althaus 5mo 10
It seem plausible that institutional mechanisms that prevent malevolent use of power may
work well today in democracies.
I agree that they probably work well but there still seems to be room for improvement. For example,
Trump doesn't seem like a beacon of kindness and humility, to put it mildly. Nevertheless, he got
elected President. On top of that, he wasn't even required to release his tax returns—one of the
more basic ways to detect malevolence.
Of course, I agree that stable and well-functioning democracies with good cultural norms would
bene t substantially less from many of our suggested interventions.
Also, the major alternative to reducing the in uence of malevolent actors may be in the
institutional decision making itself, or some structural interventions. AI Governance as a
eld seems to mostly go in that route, for example.
Just to be clear, I'm very much in favor of such "structural interventions". In fact, they overall seem
more promising to me. However, it might not be everyone's comparative advantage to contribute to
them which is why I thought it valuable to explore potentially more neglected alternatives where
lower-hanging fruits are still to be picked.
That said, I think that e orts going into your suggested interventions are largely orthogonal
to these alternatives (and might actually be supportive of one another).
Yes, my sense is that they should be mutually supportive—I don't see why they shouldn't. I'm glad
you share this impression (at least to some extent)!

[-] edcon 5mo 6

Thanks for the post! Really well reasoned on the broad impact of malovelence.
1. It seems that any research on manipulation proof measures for detection for malevolence, would
help the development of tools that would be useful for a totalitarian state.
2. I'm sceptical of further research on malevolence being helpful in stopping these people being in
positions of power. At rst glance I don't think a really well developed literature on malevolence,
would of changed leaders coming to power in 20th century.
3. In terms of Public engagement, I am also sceptical (eg. jon ronson the psychopath test) as I suspect
that making people want more altruistic people in charge, is hard to move needle on. (Interesting
piece on malevelonce in leaders https://theconversation.com/narcissists-and-psychopaths-how-
some-societies-ensure-these-dangerous-people-never-wield-power-118854)
4. In terms of development of genetically engineered individuals, I think it really matters more who is
in charge. I doubt bio-ethicists saying dark triad traits are bad will have much of an e ect. In terms of
further GWAS studies, I suspect by the time this becomes feasible more GWAS on desirable
personality traits will have been undertaken. And it seems that work done ahead of time will be less
useful, due to earlier studies being superseded by studies with more advanced technology .
[-] David_Althaus 5mo 7
Thanks, these are valid concerns.

1. It seems that any research on manipulation proof measures for detection for
malevolence, would help the development of tools that would be useful for a
totalitarian state.

My guess is that not all research on manipulation-proof measures of malevolence would pose such
dangers but it’s certainly a risk to be aware of, I agree.

2. I'm sceptical of further research on malevolence being helpful in stopping these

people being in positions of power. At rst glance I don't think a really well developed
literature on malevolence, would of changed leaders coming to power in 20th century.

In itself, a better scienti c understanding of malevolence would not have helped, agreed. However,
more reliable and objective ways to detect malevolence might have helped i there also had existed
relevant norms to use such measures and place at least some weight on them.
I think it really matters more who is in charge. I doubt bio-ethicists saying dark triad traits
are bad will have much of an e ect.
Bioethicists sometimes in uence policy though I generally agree with your sentiment. This is also
why we have emphasized the value of acquiring career capital in elds like bioinformatics.
In terms of further GWAS studies, I suspect by the time this becomes feasible more GWAS on
desirable personality traits will have been undertaken.
I agree that this is plausible—though also far from certain. I’d also like to note that (very
rudimentary) forms of embryo selection are already feasible, so the issue might be a bit time-
sensitive (especially if you take into account that it might take decades to acquire the necessary
expertise and career capital to in uence the relevant decision makers).

[-] Neil_Dullaghan 1mo 5

Thanks for this!
I see you referenced Matthews et al. (2018), which I haven't read, and wondered if you had also seen
the Authoritarian Ruling Elites Database, compiled by Matthews (2019): “a collection of biographical
and professional information on the individuals who constitute the top elite of authoritarian
regimes.” Each of the project’s 18 datasets focuses on a particular regime, such as the military
dictatorship that ruled Chile from 1973 to 1990. The biographical data-points include gender,
occupation, dates of birth and death, tenure among the elite, and more. (This came to me via the
Data Is Plural mailing list fwiw). Apologies if you mentioned it and I missed it.

[-] Timothy_Liptrot 4mo 5

I am skeptical of this line of reasoning because I see little reason to believe that malevolence
determined the policies in question. Game theory political scientists argue that di erent institutional
structures make it rational or irrational for leaders to distribute public goods or targeted goods,
practice repression, allow political parties. For a more in depth treatment, see the Dictator's
Handbook by Bruce Bueno De Meqsuita and Alistair Smith. Their core argument is that because
dictators must appease a very small group of powerful interest leaders (generals for Mussolini,
members of the centcom for Stalin, tribal and military leaders for the Abdullah II) they can protect
their power by rewarding only that small group at the expense of the masses.
Here is an illustrative example of political phenomena that is di cult to explain from the leaders
personality. Torture is more common in multi-party autocracies than in one-party states. If the
leaders narcissism strongly in uences policies and narcissism and sadism are strongly correlated,
then we would expect torture to be more common in states that ban dissent. Suppose that torture is
not about satisfying the personal desire of the dictator and is instead about policing dissent. Now it
makes sense that if some dissent (like resistance to a new "non-security" policy) is allowed, there
must be some boundary into banned dissent. Then the occurrence of torture in multi-party states
makes more sense and the rarity of torture in the most severe autocracies makes sense.
Opposing personality-of-dictator explanations to ideological explanations surprised me because it
ignored the strongest explanations in institutional structures of states and in political cultures.
Possibly you emphasized ideology because your samples are older. While the early modernist
dictators were authentically ideological, most modern autocrats espouse a bland, centrist, syncretic
corporatism. Dictators like Chavez and Castro are the exception today (although their ideology does
in uence behavior). Here is an article which argues most dictatorships are interest driven, not
ideologically driven https://sci-hub.tw/10.1080/13510347.2017.1307823
[-] MichaelA 1mo 7
I'd de nitely agree that leaders' personalities are not the sole factor in determining a countries'
policies, levels of repression, etc., even in autocracies. And I'd tentatively guess that leaders'
personalities tend not to be a larger factor than other in uences combined. And it's very plausible
to me that institutional structures tend to be a larger factor.
But it seems to me that leaders personalities are very likely at least sometimes a substantial factor.
And, from memory, I don't think this post makes or requires stronger claims than that. I.e., I don't
recall it making con dent claims that other factors like institutional structures are usually smaller
factors than personality. Are there passages you thought were overly strong?
I think this post was largely arguing that, on the margin, it could be quite valuable to think about
reducing risks from malevolence - not that other factors like institutional structures don't also
matter.
 Torture is more common in multi-party autocracies than in one-party states.
This claim seems plausible to me, but I've never seen the claim before or seen speci c evidence on
it. Could you cite a source for that?
I also know of some data points that seem to push against that claim. For example, I believe there
was a lot of torture during the one-party states of Hitler, Stalin, and Mao. (Here are some sources
that seem to support this, though I only read the titles, as I was already pretty sure this was true: 1, 2,
3.)
And a handful of historical accounts I've read make it seem quite plausible that the personality of
the leaders played a role in this. In particular, those accounts suggest these leaders (or at least Hitler
and Stalin; I know less about Mao) took pleasure in things like ordering tortures or gruesome deaths,
that they regularly personally ordered these things, and that they sometimes ordered these things
at times when it seems it wasn't useful or logical to do so. Here's one paper that supports those
claims (though I'm mildly skeptical of the paper in some ways): Why Tyrants Go Too Far: Malignant
Narcissism and Absolute Power.
(I've just started doing research following on from this post, and collaborating somewhat with its
lead author, but all of the above opinions are my own, as always.)
[-] Timothy_Liptrot 1mo 3
Source:
https://edisciplinas.usp.br/plugin le.php/103226/mod_resource/content/1/James%20Vreelan
d%202008.PDF
Concerns with causation:
I worry about the underlying assumption that democracies don't encourage malevolent traits. So
we observe less mass killings and rival killings in democracies than in dictatorships. One
explanation is that democracies are selecting for anti-killing leaders. Another explanation is that
malevolent leaders in democracies see little gain from killing while malevolent leaders in certain
types of dictatorships see much gain from killing. For example, Abraham Lincoln and Richard
Nixon seem to have a lot of malevolent traits, but they mostly refrained from political killings. So
the fourth section suggests too much causation on the personalities of leaders.
But it seems to me that leaders personalities are very likely at least sometimes a
substantial factor.
Fair - even if most of the di erence in political killing is from institutional incentives, preventing
malevolent actors from becoming dictators or presidents is big gains.
Secondly, this research agenda must recognize that authoritarian cultures may accept or
encourage violence against non-comformists or "disloyal" people. It's a deeply sad fact, but
important to understand. If torturing dissidents is an expected and approved behavior it is weaker
evidence of a malevolent personality (fundamental attribution error).
Targeting your interventions:
You should pay more attention to how autocrats are actually selected. There are a few good
models, my favorite is selectorate theory which (in dictatorships) emphasizes trust between top
lieutenants. To continue with your preferred 40's references, Herman Goring, Himler, Keitel and
Borman all hate and distrust one another. But they each trust Hitler because Hitler selected them
to receive stolen wealth in exchange for support. So as long as the autocrat is alive the alliance is
"stable".
In this model they care most about malevolence when the ruling coalition/launching org selects
the next dictator. This famously happened after Stalin died, when Beria (head of secret police) got
overthrown and replaced with Khrushchev who promised to stop political killings. So target
regimes in or near transitions.
For example, the politburo of Cuba might be interested in this research. Saudi Arabia, now that
MBS has solidi ed control, will be less fruitful.
[-] MichaelA 1mo 7
https://edisciplinas.usp.br/plugin le.php/103226/mod_resource/content/1/James%
20Vreeland%202008.PDF
Thanks for sharing that source. I tried to skim through to nd the part most relevant to your
claim, and found this:
What is observed regarding patterns of torture under dictatorship? The Hathaway
torture data include 967 country-year observations of from 1985 to 1996 covering 109
separate dictatorships. The mean rate of torture is 3.0 with a standard deviation of 1.1;
the median is 3. My principal explanatory variable is Gandhi's "parties," a dummy
variable coded 1 if more than one party exists legally and 0 otherwise. For the Hathaway
data, nearly 52 percent of the observations have multiple political parties. Average rates
of torture under dictatorships without political parties is 2.8 (standard deviation 1.1),
but rates of torture under dictatorships with political parties is 3.1 (standard deviation
1.0). A t-test indicates that the di erence in average rates of torture is statistically
signi cant beyond the 0.01 con dence level (t = -4.44). Rates of torture are ironically
higher under open dictatorships that allow multiple political parties
So I do think this source supports your claim. But I'd note that:
That data just covers 1985-1996. Hitler, Stalin, and Mao seem the most salient examples
of potentially very malevolent individuals wielding great power and causing great harm,
and each would be excluded from that data set.
I'm not saying that the study should've focused on an earlier period, or that there's
some reason our predictions about the future would be better informed by an earlier
11 year period.
But I'd be more con dent about extrapolations from the data set if it spanned a
larger period of time.
And I think it would make sense for our predictions to draw on both that data and
the observation of extreme harms under the one-party states of Hitler, Stalin, and
Mao. (The latter is just a sample of 3, but I believe it accounted for a large portion of
all deaths in mass murders, famines, etc.)
The di erence in averages doesn't look hugely substantial - ~30% of 1 standard deviation
- even if it was statistically signi cant.
 This seems like a further reason to also pay substantial attention to other
observations, like harms under Hitler, Stalin, and Mao.
(Again, this doesn't mean your claim was false, just that its evidence base and implications may
be more limited than one might have thought.)
[-] Timothy_Liptrot 1mo 3
I'm compelled by this. That the di erence is only 30% of 1 standard deviation means that lots
of variation could be explained by other factors. Personality of the dictator could still explain
lots of variation, even a majority. There could also be a relationship between dictator
personality and allowances for dissent. Thanks for explaining that!
Aside, you would be more compelling if you talked about autocrats other than Hitler, Stalin
and Mao.
[-] MichaelA 1mo 3
Aside, you would be more compelling if you talked about autocrats other than
Hitler, Stalin and Mao.
Do you mean talking about other autocrats as well, or instead of, talking about Hitler, Stalin,
and Mao?
If you mean "as well", I'd agree. I've already started looking at some others (as did David
and Tobias, the authors of this post), and will hopefully do more of that. The reason for
focusing mostly on those three so far is just that it takes time to learn about more, and those
three were (I'd argue) huge factors in a large portion of all harms from political atrocities in
the 20th century.
If you mean "instead of", could you explain further why you say that?
[-] Timothy_Liptrot 1mo 4
Hmmm, that is a good question. Let me dig in more. Here are reasons to talk about others
than Hitler Stalin and Mao
Coup Proo ng is a common practice of dictators for political
survival
Some behaviors of Hitler, Stalin and Mao have compelling institutional explanations that
have become repeated behavior of long-ruling dictators. I'm thinking of coup proo ng in
particular. Coup proo ng is a set of policies dictators enact to prevent a single small group
from seizing power; rotatring or purging o cers (Tukachevsky/Rommel) splitting the
army into multiple factions/militias (NKVD/SS/Revolutionary Guard). We've since
observed that lots of dictators (and coup-threatened democracies) practice coup-
proo ng. So I would be careful about attributing the *intra-elite* violence by particular
regimes to the personalities of the leaders. Coup-proo ng cannot explain violence against
non-elites by those regimes.
Change in ideological motivation between early dictators and
modern dictators
Secondly, Hitler, Stalin and Mao were much more ideological than most modern dictators.
The Mussolini model of a "moderate" or "synarchic" authoritarianism spread more after
WWII. By moderate I mean without a narrow, extreme vision of state-society interaction,
not that dissent or economic freedom were allowed. Particularly today ideology structures
the behavior of dictators much less. So one could argue that both Hitler and Stalin
faithfully followed the visions laid out in their (terrible and warped) ideologies. If you buy
that argument you would update downward on future dictators committing similar
violence against non-elites.
For example, Mohammad Bin Salman has killed and tortured elite rivals, dissenters and
starved many thousands of Yemenis to death. But he does not seem interested in any state
project that would involve violence on the scale of Hitler/Stalin/Pol Pot.
All that said, I personally do not put much weight on my ideology argument. Firstly, the
ideological explanation of Hitler and Stalin's behavior is not that strong. Secondly just
because most dictators, like MBS and the Ethiopians, have "moderate" visions of state-
society relations does not prevent future radical dictators from taking power (tail risk).
Note that ISIS is a deathcult that took over half of two middle income countries, and that
even in the 20's and 30's Mein Kampf was such ridiculous nonsense that Hitler should not
have risen to power.
But on a rst read this is a real hole in your argument. Almost all modern dictators do not
look like Hitler, Stalin and Mao due to their weak ideoligical commitments. If you include a
few more modern examples you can evade this argument. O the top of my head I would
suggest:
Khomeini and Ahmedinajad - they did not murder lots of people but their stubborn refusal
to compromise with the world order has impoverished Iran without causing a regime
change (they changed a bit recently but 30 years of bad decisions before).
Pol Pot - 1980's, only stopped by Vietnamese intervention
Modern Burma
[-] MichaelA 1mo 7
Your comments about coup-proo ng seem interesting and useful.
We've since observed that lots of dictators (and coup-threatened democracies)
practice coup-proo ng. So I would be careful about attributing the *intra-elite*
violence by particular regimes to the personalities of the leaders.
I think the fact that more leaders engaged in violent coup-proo ng (rather than it just
being Hitler, Stalin, and Mao) should indeed provide weak evidence against the theory
that the unusual (compared to the population as a whole) personalities of leaders plays a
key role in whether violent coop-proo ng occurs. This is because that theory would now
need to claim that a larger number of leaders have personalities that are unusual in the
relevant way, or that a lack of such unusual personalities was "made up for" by other
factors in some cases.
But I think that this fact would only serve as weak evidence, because it doesn't seem
very implausible to claim that a fairly large number of dictators, or leaders of coup-
threatened democracies, have personalities that are unusual in the relevant way. These
are people in unusual positions which are arguably easier to get into if one is ruthlessly
self-interested, so it wouldn't seem surprising (prima facie) if their average levels of
ruthlessness-relevant traits was notably above population averages.
Additionally, it seems worth distinguishing violent coup-proo ng from coup-proo ng
as a whole. In terms of how well they might evidence malevolent traits, "rotating [...]
o cers" and "splitting the army into multiple factions/militias" seem quite di erent
from the sorts of violent purges engaged in by e.g. Stalin. (It may well be that violent
coup-proo ng is very common as well; I'm just agging that the distinction seems
relevant for my purposes.)
You also seem to imply that (a) these coup-proo ng behaviours may have been rational
things for a self-interested leader to do in those situations, and (b) this is reason to be
careful in assuming that this is about personality. I think (a) is a good point. And I think
there's some merit to (b), in the sense that this pushes against thinking something like
"These leaders are just crazy and evil."
But overall, I don't think this is a question of personality vs incentive structures. I think
neither determines a person's behaviour by itself. And I think whether or not one would
do even horri c things if it's in one's self-interest is partly a matter of personality. E.g.,
I'd be willing to bet that a very large portion of people wouldn't engage in violent coup-
proo ng, even if they were in a situation where doing so would help them keep power.
(Another framing of this is that a person's personality and values helps determine their
incentives, in the sense that it in uences what we like and dislike, including how much
we dislike harming people.)
Coup-proo ng cannot explain violence against non-elites by those regimes.
I think this is key; I think the massive-scale violence against non-elites is probably a
larger part of why Hitler, Stalin, and Mao seem like interesting case studies of potential
harm from malevolence than is the potentially "rational" coup-proo ng.
That said, it's good to independently consider the merits of a even non-central
arguments/points, and to update incrementally, so your arguments against making
naive inferences from coup-proo ng behaviours are still useful.
[-] Timothy_Liptrot 1mo 6
I have one criticism of the argument that coup-proo ng prevalence is evidence for
personality factors. Suppose that if people observe a game being played multiple
times they are more likely to set aside their personal preferences and "play to win". So
if I were the rst dictator of Iraq I might say "no I'm not going to kill generals who
come from di erent towns, that would be evil". And then get killed for it. And maybe
the second dictator says the same thing. But by the time the third or fourth dictator
rises to power he'll either be selected for willingness to use violence or he will decide
his preference for living is stronger than his preference for not killing. While I agree
that many people would not commit inter-elite violence as the rst leader, I suspect a
much larger number would as the 5th leader. So an argument for point B.
Saddam Hussein was the 5th Iraqi leader to take power by coup within 21 years.
 But on the other hand, there are lots of leaders that just stepped down when they lost
the support of their ruling coalitions. And those heroes do not become famous. This is
strong evidence of the importance of personality.
[-] MichaelA 1mo 3
I have one criticism of the argument that coup-proo ng prevalence is
evidence for personality factors.
To be clear, my argument was more like "coup-proo ng prevalence doesn't seem
like strong evidence against personality playing an important role". I.e., I don't think
that it should reduce our belief that personality plays an important role.
It is true that I think I'd see these behaviours as evidence for personality playing an
important role. But I'm not sure, and I'm not seeing it as key evidence.
While I agree that many people would not commit inter-elite violence as the
rst leader, I suspect a much larger number would as the 5th leader. So an
argument for point B.
I'd agree that a much larger number would as the 5th leader than as the 1st leader, in
the scenario you describe. And I think this is a valuable point.
But, in line with your nal paragraph, I'd still bet that many people wouldn't; I think
many people would simply step down, ee, or accept radical changes to the nature of
their regime.
And perhaps more importantly, I think personality in uences whether someone tries
to become a leader in the rst place, and whether they succeed in that. So I expect a
lot of people to not want to "do horrible things", recognise that pursuing this
leadership position may require them to "do horrible things" along the way or to
stay in power, and thus just not pursue those positions.
(That said, I did say "I'd be willing to bet that a very large portion of people wouldn't
engage in violent coup-proo ng, even if they were in a situation where doing so
would help them keep power." So there's a valid reason why you focused on how
people would behave if they somehow landed in the leadership position, rather than
how likely they are to enter those positions to begin with.)

[-] MichaelA 1mo 3

Thanks for this comment.
Your points (and counterpoints!) about changes in ideological motivation are very
interesting. And I think it'd probably be good for me to spend some time engaging with
evidence/arguments about how much ideology in uenced Hitler, Stalin, and Mao's
most "extreme" behaviours and whether/how much the in uence of ideology has
waned.
 And it does seem wise to think about that, and about more modern examples, if one is
planning to communicate with the public, policymakers, or academics about this topic
in a way that leans substantially on historical examples of dictators. (I'm not sure if
anyone will actually do such communications, or emphasise those cases when doing so.
It may, for example, make more sense to just focus on the psychological studies, or on
examples from business.)

[-] MichaelA 1mo 3

Another explanation is that malevolent leaders in democracies see little gain from
killing while malevolent leaders in certain types of dictatorships see much gain from
killing. For example, Abraham Lincoln and Richard Nixon seem to have a lot of
malevolent traits, but they mostly refrained from political killings. So the fourth section
suggests too much causation on the personalities of leaders.
I did recently think it might be interesting to look into Nixon as a case study in how, and how
well, democratic institutions can mitigate the harm caused by leaders with high levels of dark
tetrad traits. (I just think it might be such a case study, because I haven't yet really looked into
evidence on Nixon's personality - this is just a guess so far.) Thanks for highlighting Lincoln too
- I wouldn't have guessed he had high levels of dark traits, but I'll look into it.
I'd de nitely guess that the reasons there is less harm from malevolent leaders in democracies
are both that democracies select for malevolence less, and that democracies don't
allow/incentivise malevolent behaviours as much.
In my head, I currently break intervention options in this cause area into:
Reducing how malevolent people are (via, e.g., very cautious and well thought-out and
not-rushed genetic engineering)
Reducing the chances of malevolent people getting into positions where those traits
create major risks (via, e.g., electoral reform, reducing instability and con ict)
Reducing the risks created when malevolent people get into those positions (via, e.g.,
checks and balances, maybe reducing centralisation of power)
I had felt like this post implied all three of those categories, not just the rst two. But now that I
re-skim the Political interventions° section, I see that that might not have been made very
explicit. So that critique of yours may be valid. (And I de nitely agree with your point, separate
from how it relates to potential oversights of this post.)
this research agenda must recognize that authoritarian cultures may accept or
encourage violence against non-comformists or "disloyal" people. It's a deeply sad
fact, but important to understand. If torturing dissidents is an expected and approved
behavior it is weaker evidence of a malevolent personality (fundamental attribution
error).
I de nitely agree that:
it's important to consider multiple explanations of the various horri c or troubling
behaviours
 it can be easy to psychoanalyse/diagnose from a distance in a foolish way
the fundamental attribution error is worth keeping in mind here
I'm hoping to look a bit into how much we can trust speculative psychological pro les in general,
maybe "best practices" for that shaky endeavour, and maybe how this relates to people like
Hitler in particular. (Before getting into EA, I might've dismissed such psychological pro les.
But I've moved towards thinking that, for many questions, it's hard to do better than weak
evidence, and it can be best to just gather diverse kinds of weak evidence, update incrementally,
and remain aware of how uncertain we are.)
(Also, by the way, I read The Dictator's Handbook last year and found it interesting. I agree that
the models/ideas in that book seem relevant here.)

[-] Milan_Gri es 5mo 5

Somewhat related – US college students have been growing more narcissistic (or maybe just gaining
higher self-esteem) since the 1980s: https:// ightfromperfection.com/college-students-more-
narcissistic.html

[-] aaronb50 5mo 3

Another type of intervention that could plausibly reduce the in uence of malevolent actors is to
decrease intergenerational transfer of wealth and power. If competent malevolence both (i) increases
one's capacity to gain wealth and/or power and (ii) is heritable, then we should expect malevolent
families amass increasing wealth and power. This could be one reason that the global shift away from
hereditary monarchies is associated with global peace (I sense that both of these things are correct,
but am not positive).
For example, North Korea's Kim family is almost certainly malevolent in the way that this post
describes, and the country's political structure enables this family to continually keep power.
On a broader scale, larger estate taxes and other economic policies to decrease wealth transfer might
have a similar e ect.

[-] maxh94 2mo 2

Here is an interesting paper related to using machine learning to identify language patterns in
psychopaths if people are interested. https://egrove.olemiss.edu/cgi/viewcontent.cgi?
article=1445&context=etd

[-] Ben_West 5mo 2

A minor copyediting suggestion (adding the words in bold):
Factor 1—characterized by cruelty, grandiosity, manipulativeness, and a lack of guilt—
arguably represents the core personality traits of psychopathy. However, scoring highly on
factor 2—characterized by impulsivity, reactive anger, and lack of realistic goals—is less
 problematic from our perspective. In fact, humans scoring high on factor 1 but low on factor 2
are probably more dangerous than humans scoring high on both factors (more on this below).
It's not a big deal, but it took me a minute to understand why you were saying it's both less
problematic and more dangerous.
[-] David_Althaus 5mo 3
Thanks! Added your suggestion.

[-] brb243 5mo 1

Thank you very much for the breathtaking analysis. I think that malevolent humans will best be
prevented from gaining in uence if benevolent humans outcompete them at in uential posts.
[-] David_Althaus 5mo 4
Thank you.
The section “What about traits other than malevolence?” in Appendix B brie y discusses this.
[-] brb243 5mo 1
Thank you. I see. Then, I wish you the very best of luck in making an e ective impact with your
actions. I hope that you will succeed in preventing many decision-makers from promoting
malevolent leaders.

[-] MikeJohnson 5mo 0

A core 'hole' here is metrics for malevolence (and related traits) visible to present-day or near-future
neuroimaging.
Brie y -- Qualia Research Institute's work around connectome-speci c harmonic waves (CSHW)
suggests a couple angles:
(1) proxying malevolence via the degree to which the consonance/harmony in your brain is correlated
with the dissonance in nearby brains;
(2) proxying empathy (lack of psychopathy) by the degree to which your CSHWs show
integration/coupling with the CSHWs around you.
Both of these analyses could be done today, given su cient resource investment. We have all the
algorithms and in-house expertise.
Background about the paradigm: https://opentheory.net/2018/08/a-future-for-neuroscience/