Article

Party Politics
1–11
The integrity of the 2016 US Presidential ª The Author(s) 2019
Article reuse guidelines:
Election: Exploring the possible impact sagepub.com/journals-permissions
DOI: 10.1177/1354068818809524
journals.sagepub.com/home/ppq
of ideology on experts’ judgments

Luigi Curini
Università degli Studi di Milano, Italy

Abstract
Using new comparative expert data about the 2016 US Presidential Election, this article explores the effects of experts’
ideological preferences on their evaluations of electoral integrity. Without contesting the claim that the 2016 election
faced challenges of integrity, our analyses reveal a substantial association between negative evaluation of election integrity
and experts’ ideological orientation. Our results also suggest that this ideological effect is stronger in States that “swinged”
from Democratic to Republican in the 2016 election.

Keywords
expert surveys, electoral integrity, ideological heterogeneity, US 2016 Presidential Election

Introduction appears that while many states have performed quite well
overall, severe problems seem to exist in other cases (Nor-
Elections in the Unites States are rarely a smooth ride.
ris et al., 2017); experts, for instance, were, critical about
Among many other issues, decentralized voting adminis-
gerrymandering in North Carolina and Wisconsin, pointed
trations, obsolete rules for electoral registration, opaque
to serious problems with the way media covered the cam-
campaign finance regulations, extreme polarization driving
paign in Nevada, Arkansas, and Oregon and questioned
partisan redistricting procedures, and the sheer huge num-
regulations of campaign finance in Wisconsin and South
ber of voters to account for all participate to create a situ-
Dakota (Norris et al., 2017). Overall, these trends suggest
ation where the final quality of the election can—at the
that some systemic problems exist in the way US elections
very least—be normatively questioned. The 2016 Presiden-
are administered (Norris, 2017) and that 2016 was no
tial Election was not an exception in this sense, facing
exception.
resolute challenges from both sides of the ideological
Confronting winners to the quality of elections that put
divide. As early as during the first primaries, Trump
them in power is essential to ensure democratic govern-
claimed that the election was marred by voting fraud; in
ance. In electoral democracies, popular support directly
the aftermath of election day, requests for recounts in three
equates with political legitimacy, and empirical initiatives
states were filed based on (partially misunderstood) foren-
that independently assess the quality of electoral pro-
sic evidence1 suggesting some form of tampering of voting
cesses—such as postelection forensics (Myagkov et al.,
machines; almost a year after the events, Hillary Clinton
2009), surveys of poll workers (Clark and James, 2017),
suggested she might after all consider challenging the
election auditing (Alvarez et al., 2012; Goggin et al., 2012),
legitimacy of the election2; and, of course, the possibility
or expert evaluations—have much to contribute to this
of foreign influence aiming at swinging the results remains
an open question. The integrity of the election was also
questioned in the academic debate. In the weeks following Paper submitted 2 October 2017; accepted for publication 7 October
the election, a new data set was released that contains mea- 2018
sures of integrity of the contest at the state level (PEI_US3);
Corresponding author:
more than 700 experts were asked to evaluate the integrity Luigi Curini, Department of Social and Political Sciences, Università degli
of the election in their state through batteries of questions Studi di Milano, Via Conservatorio, 7 - 20122 Milano, Italy.
covering the whole election cycle. Through the data, it Email: luigi.curini@unimi.it
2 Party Politics XX(X)
cause. The importance of the issue calls however for cau-
tion. When it comes to expert judgments, several questions
have been advanced with respect to their “measurement
validity,” that is, the degree to which the measurements
estimated via an expert survey actually capture the phe-
nomenon of interest (Adcock and Collier, 2001). For exam-
ple, how can one elicit valid responses from experts about
the object of the survey (such as party policy positions, or,
as in our case, election integrity)? What is the basis of the
judgments that experts offer? Do experts use the same cri-
teria when judging? Do experts rely on their expertise or
also provide their personal views? (see Budge, 2000; Mair,
2001; Steenbergen and Marks, 2007). The latter two ques-
tions, in particular, raise the problem of expert heterogene-
ity and the possible impact of that on measurement validity.
If the considerations that come into play were to vary this
much, it would not be at all clear what expert judgments
actually measured (Martı́nez i Coma and van Ham, 2015).
In this article, we focus on the impact of a particular type of
expert heterogeneity, that is, ideological heterogeneity.
With that we mean the potential influence of experts’
ideological preferences on their judgments. Experts’ ideo-
logical preferences (e.g. as measured on a standard left to
right scale), as well as their ‘closeness’ or ‘sympathy’ to
parties (see Benoit and Laver, 2006; Laver and Hunt,
1992), have been shown in the past to systematically affect
in some given instances their opinions and evaluations
(Curini, 2010). The aim of an expert survey is not of course
to make any inferences about the attributes of the whole
population of experts from the sample of respondents
selected but rather to establish an accurate estimation of
some properties using the said experts’ knowledge. In this
sense, it is perfectly reasonable to have a highly unrepre-
sentative sample from the ideological point of view
(because experts can be more liberal, or conservative, than
the whole population) that does not show any evidence of
an impact of experts’ ideology on their reported judgments
or vice versa to have an ideologically representative sample
that presents such evidence. The former would produce
useful estimates of the phenomenon under investigation
(because experts’ opinions are not colored by their ideolo-
gical orientations), whereas the latter would not.
The aim of this article is to explore how ideological
preferences of experts possibly contribute to shape their
opinions. We do so by looking at the Perceptions of Elec-
toral Integrity (PEI) data set about the 2016 Presidential
Election in the United States, which is ideal for this goal.
First, the data set replicates the protocol of previous studies
assessing the quality of elections worldwide through expert
surveys (e.g. Martı́nez i Coma and van Ham, 2015; Norris
et al., 2013), thus relying on a consistent scholarship and
agreed upon empirical definition of electoral integrity. Sec-
ond, the data were gathered just a few weeks after the
election, which represents a perfect setting to test the
assumption that experts’ opinions are affected by their
personal stance about the election results. Third, the data
set asks the experts to evaluate the integrity of the election
in their state (i.e. the state where they are currently work-
ing); as a result, heterogeneity at the state level allows us to
test for systematic differences among experts also across
and within different contexts. Beyond the data itself, we
test here for the structuring effects of experts’ ideological
positions on a different aspect than the one usually studied
(party positioning, e.g. Castles and Mair, 1984; Hooghe
et al., 2010).
The remaining of this article is structured as follows.
The next section provides a brief overview about recent
studies addressing the issue of electoral integrity in the
United States and suggests that independent evidence
pointing to the presence of severe flaws is now ubiquitous.
The third section focuses on the PEI data set and introduces
our main argument. The fourth and fifth sections present
the main empirical results, and the last section concludes.
A conclusion follows.
Election integrity in the United States
In an important article, Dennis Thompson (2004) discusses
three temporal issues that create normative challenges for
elections in the United States and with very concrete impli-
cations for electoral integrity. Elections in the United States
take place at regular intervals (“periodicity”), the over-
whelming majority of voters cast their ballot on the same
day (“simultaneity”), and the outcome of the electoral pro-
cedure is irrevocable until the next election (“finality”). On
the onset, these seem legitimate objectives: periodicity
rhymes with regularity and stability, which is undoubtedly
a virtue of modern democracies; simultaneity ensures that
all voters are treated as equal, very much in the same way
as their different voices are equally valid and deserving
representation; finality, then, guarantees the legitimacy of
the results and invests the winners with a clear mandate.
And, yet, these three issues are not without concrete
challenges:
periodicity requires that the control of redistricting procedures
be independent of legislatures. Simultaneity opposes publish-
ing the results of exit polls, expanding the use of early voting,
and granting legislatures the authority to select presidential
electors on their own schedule. Finality justifies more stringent
regulations of campaign practices, such as electioneering ads.
(Thompson, 2004: 51)
These are just but a few of the important challenges that
elections in the United States face, and in the recent
decades, a nourished literature developed to question and
assess the integrity of US elections (Alvarez et al., 2012;
Alvarez and Grofman, 2014; Burden and Stewart, 2014;
Cain et al., 2008; Donovan and Bowler, 2004; Fife, 2010;
Norris, 2017).
Curini
It is not our goal to provide here a complete overview of
this literature nor to discuss the normative claims and pos-
sible solutions which are advanced in those studies. Rather,
we wish to make a simple point: Assessing the quality of
elections in the United States is not a trivial endeavor, nor a
“niche” one. Whether focussed on the electoral conse-
quences of partisan gerrymandering (Friedman and Hol-
den, 2009; McCarty et al., 2009), opaque campaign
finance regulations that drive polarization and conflict
(Brooks and Murov, 2012; Norris and Van Es, 2016), race
inequalities in electoral registration and access to voting
booths (Leighley and Nagler, 2013; Piven et al., 2009), or
covering all aspects of the election process (Norris, 2017),
much evidence exists that elections in the United States are
lacklustre at best, with potential dire consequences for
democratic legitimacy and public confidence (Birch,
2008; Norris, 2014).
The perceptions of electoral integrity and
experts’ ideological positions
Among the multitude of concepts used to assess the quality
of electoral contests—electoral fraud, manipulations, vote
rigging, and the whole idea of “free and fair” elections—
the notion of “electoral integrity” (Norris, 2014), developed
within the framework of studies assessing the quality of
elections in a holistic and systematic way (e.g. Elklit and
Reynolds, 2005), is undoubtedly the one that has received
the highest attention and has quickly imposed itself as the
standard in the comparative empirical literature. Electoral
integrity is seen as a more encompassing approach to elec-
tion quality, one that tackles issues during the election itself
but also before election day and in its aftermath. Thus,
issues at the onset of the election cycle—such as the com-
position of election management bodies (Garnett and
Zavadskaya, 2017) or the definition of electoral boundaries
(McCarty et al., 2009)—or during the campaign such as the
quality of political journalism covering the election (Nai,
2017) or campaign finance (Norris and Van Es, 2016), are
an integral part of the definition. Within this framework,
empirical measures of integrity have to incorporate the idea
of elections as multistage processes and their multiple
dimensions and components. This is the idea behind the
“Perceptions of Electoral Integrity” score (henceforth, PEI
score; Norris et al., 2013) developed by the Electoral Integ-
rity Project (EIP).4 Launched in 2012, the EIP is an inde-
pendent and nonprofit research project hosted at the
Department of Government and International Relations
(University of Sydney) and Harvard’s JFK School of Gov-
ernment. The EIP aims at providing independent and non-
partisan systematic evidence about the quality of elections
across the world and does so via a multi-awarded large-
scale comparative expert survey.5 Since 2012, the project
has monitored all elections that happened worldwide
(excluding very small states and “façade” elections without
3
any real competition) and has published since a bi-annual
report and data set illustrating the results and global trends;
the latest version of the data set,6 released in early 2018,
covers cumulatively 285 national elections that happened
in 164 countries worldwide between July 2012 and Decem-
ber 2017, based on answers provided by more than 3000
experts (Norris et al., 2018).
A key element within the released data monitoring the
quality of elections is the PEI score. For each election, the
PEI score is obtained by combining the experts’ answers to
a series of 49 items, organized in 11 macrosections with
equal weight and covering all stages of the election cycle:
electoral laws, electoral procedures, boundaries for voting
districts, voter registration, party and candidate registra-
tion, media coverage, campaign finance, voting process,
vote count, result index, and electoral authorities. The
answers provided by experts to the 49 items are added into
a 0–100 scale, which is then aggregated to the desired
geographical unit (country, state) to measure the overall
integrity of the election. This measure of electoral integrity
has been shown to perform well in terms of reliability and
validity (Martı́nez i Coma and van Ham, 2015; Norris et al.,
2013).7
We do, however, believe that one element deserves
additional attention, that is, the potential effect of experts’
ideology on their judgments as a possible source of error in
data collection. In this respect, we advance three main
hypotheses.
First, it has been shown that a left-wing ideology is
associated with lower trust in political institutions due to
the fact that progressivism fosters independent thought,
critical thinking, and openness to change—whereas conser-
vatism promotes preservation and stability (Devos et al.,
2002); furthermore, people on the left-hand side of the
spectrum have been shown to be more sensitive to inequal-
ity and injustices (Anderson and Singer, 2008), and thus
potentially more tuned toward pinpointing malpractices—
whereas conservatives are more likely to score high on
system justification (Jost et al., 2008). From both perspec-
tives, we deduct that a left-wing ideology is likely to foster
a more critical and pessimistic view of electoral integrity,
ceteris paribus. The addition of the ceteris paribus clause is
of course important here, given the amount of factors that
could affect the PEI score, as we will discuss below.
H1: The left-right ideological positions of experts is
positively related to the PEI score.
The possible impact of ideology could, however, be
nonlinear. Several scholars noted as long as half a century
ago that right- and left-wing extremists share a number of
commonalities (Eysenck, 1954; Rokeach, 1960). Possibly
relevant for the present study, McClosky and Chong (1985:
343) pointed out that extremists on both ideological ends
are characterized by resentment and pessimism toward
4 Party Politics XX(X)

Table 1. Explaining the left-right self-placement impact on PEI Index.

(1) (2) (3) (4) CEM (5)

Left-right self-placement 1.174** (0.348) 3.051** (1.048) 1.211** (0.397) 2.221** (0.198) 0.942* (0.382)
Left-right self-placement squared 0.218þ (0.115)
Role in the election (0–1)a 1.155 (2.760)
Domestic (0–1)b 0.946 (2.291)
Years lived in this state 1.088þ (0.607)
How familiar with these elections? (1–10) 1.176** (0.309)
Loser–winner dummyc (1 ¼ loser; 0.598 (2.493)
0 ¼ winner)
Age group (by decade) 0.711* (0.331)
Flipped states 9.815** (2.549)
Left-Right Self-Placement  Swung States 1.315* (0.517)
Constant 58.559** (1.763) 55.232** (2.373) 50.047** (5.062) 55.841** (0.838) 60.361** (1.852)
Observations 715 715 587 302 715
BIC 5450.523 5453.255 4507.388 1003.622 5432.143
Note: CEM: coarsened exact matching; BIC: Bayesian information criterion. The covariates included in the matching procedure in model 4 are all the
control variables used in model 3. Cluster standard errors on states are given in parentheses.
a
The variable takes the value 1 if the expert had an active role in the election (as a representative, candidate, activist, election monitor, NGO monitor, or
election official), and the value 0 in all other cases.
b
The variable takes the value 1 if the respondent has an expertise in domestic or comparative politics (including elections, American politics, state
politics, public administration, public policy, and so forth), and 0 for an expertise in international politics only.
c
The variable takes the value 1 if the respondent declared to have voter for a “losing” candidate at the Presidential Election; 0 otherwise.
*p < 0.05; **p < 0.01.

institutions, as well as toward mainstream politicians and
policies (on the relationship between ideology and trust
toward political institutions: (Newton and Norris 2000;
Gershtenson et al., 2006)). Once again, this pessimism can
play a role with respect to the assigned PEI score.
H2: The left-right ideological positions of experts is
nonlinearly related to the PEI score.
Note that the impact of ideological heterogeneity on
the ability of an expert survey to recover a valid measure
of electoral integrity has been discussed also in Martı́nez i
Coma and van Ham (2015). In this latter article, however,
the authors control via a cross-national analysis if there is
any systematic impact of a set of variables (including
experts’ ideological position) on the variance of experts’
answers (i.e. the deviation of the expert placement from
the mean placement by all experts in each election) rather
than on their level, as we do here. Moving from variance
to level allows us to identify, as we will see, also the
direction, and not only the existence, of the impact of
ideological heterogeneity on the PEI score retrieved by
the expert survey.
Finally, it could also be argued that the effect of ideol-
ogy on experts’ evaluations is magnified in specific con-
textual situations, in particular in those instances wherein
the electoral result appears to be surprising (compared to
the expectations). Research in consumer psychology shows
that surprise is likely to alter judgments about perceived
satisfaction regarding the object of the evaluation by shar-
pening the focus of the attention and thus providing more
critical assessments (Vanhamme and Snelderts, 2001). This
applies in particular when a specific form of surprise is at
play: disappointment. This specific affective experience
exists when a surprising situation acquires a negative
valence if matched with individual predispositions and
expectations. And much evidence exists that disappoint-
ment has specific behavioral components (Zeelenberg and
Pieters, 2004), including incentives for complaining (in our
case, against the quality of the election) that are higher
compared to what would have been under different
circumstances.
H3: The effect of experts’ ideology on their evaluations
is stronger in surprising contexts.
Empirical results
To control for our hypotheses, we have followed a variety
of statistical approaches. Table 1 reports the result of the
models we have fitted. In each model, we estimated a linear
model with standard errors clustered at the state level,
where the experts’ PEI score is regressed on their self-
reported left-right placement.8
Model 1 brings a strong empirical corroboration for
hypothesis 1. Figure 1 (left panel) gives a visual representa-
tion of such result (in this and in the following figures we
also superimpose the distribution for the experts’ ideological
position on the right-hand side of the graph). Instead of being
two independent phenomena, moving from left to right on
the self-reported ideological scale significantly increases the
PEI score: For each increase of one left-right unit, the overall
Curini
75
Expected value of PEI
70
65
60
55
0 2 4 6
Left-Right Ideological Dimension
Figure 1. Expected PEI value as left-right experts’ placement changes (left panel: linear model; right panel: nonlinear model). Source:
model 1 and Table 1 (left panel); model 2 and Table 1 (right panel). PEI: Perceptions of Electoral Integrity.
evaluation of electoral integrity (PEI score) increases by
approximately 1.2 points.
Given that the ideological distribution of the experts in
the data set is skewed to the left in a remarkable way (see
Figure 1), the overall PEI score for the 2016 US Presiden-
tial Elections is thus possibly underestimated. In other
terms, it might be possible that when aggregating the
experts’ answers at the state level, a more pessimistic pic-
ture emerges. This claim does not hold if we assume that
“leftist” (vs “centrist” or “rightist”) experts have better
knowledge about the functioning of the electoral institu-
tions. In this eventuality, resulting aggregation of experts’
opinions would not be skewed but rather increase in its
predictive quality: If left-leaning experts are more knowl-
edgeable, having a lot of them should improve the data. The
problem here is that we do not really know the “true” PEI
value, that is, we do now have an objective benchmark of
the integrity of the US election, so we cannot estimate the
existence of a systematic deviation from such true value
according to experts’ ideological position. What we can,
however, say is that, using a jargon derived from psycho-
metrics (Steenbergen, 2000), the items (i.e. the perceived
PEI scores) appear to behave differently across expert’s
groups according to their ideological positions. As a result,
care should be taken with aggregating expert scores.9
Model 2 tests hypothesis 2, with less satisfactory results:
as can be seen, the evidence of a nonlinear relationship
between left-right self-placement of experts and the PEI
value is a weak one (and statistically significant only at the
90%). This is also confirmed by the Bayesian information
criterion of the models estimated so far, which clearly
advantages model 1 over model 2.10
Models 3 and 4 control the robustness of what found in
model 1. In model 3, we added several control variables at
the individual level that allows us to check for the impact of
different aspects of experts’ heterogeneity other than
experts’ ideological position. First, experts may have been
5
75
.3
.3
70
Expected value of PEI
kdensity leftrightscale
kdensity leftrightscale
.2
.2
65
.1
.1
60
55
0
0
8 10 0 2 4 6 8 10
Left-Right Ideological Dimension
involved in the electoral process more or less directly—for
example, as an election monitor or activist. To measure
this, we added a dummy that takes the value of 1 if the
expert had an active role in the election (as a representative,
candidate, activist, election monitor, Non-Governmental
Organization (NGO) monitor, or election official) and 0
otherwise. Similarly, we added a dummy that takes the
value of 1 if the respondent has an expertise in domestic
or comparative politics (including elections, American pol-
itics, state politics, public administration, public policy,
and so forth) and 0 for an expertise in international politics
only. The idea here is that domestic experts may come to
different conclusions about election integrity (e.g. being
more demanding) than international experts. Third, it could
be also important to understand if one has first-hand and
on-the-ground experience of an American state’ elections.
To understand this, we control for the number of years an
expert lived in a given state, assuming that the longer the
time spent in a given state, the better her experience about
its electoral procedures. Finally, to account for the possible
impact of the heterogeneity in expertise and knowledge
about the electoral process among experts, we also use a
continuous measure of self-reported expertise (How famil-
iar are you with elections in this country? 1 ¼ little; 10 ¼
very much).
Moreover, it could be also argued that, assuming that the
ideological position of American experts is closely tied to
their voting attitude, the left-right variable is just a proxy of
the electoral winner–loser phenomenon (Anderson and
Guillory, 1997; see also Flesken and Hartl, 2017), and it
is, in reality, this latter variable that explains the negative
relationship found between the ideological position of
experts and the PEI score. This would imply that experts
on the left side of the political spectrum are significantly
more likely to have a pessimistic opinion about the quality
of the election not because a left-wing ideological stance
decreases trust in political institutions as hypothesis 1
6 Party Politics XX(X)
suggests (Devos et al., 2002), but as the result of a loser
effect due to the disappointing outcome of the election. Of
course, this would still be potentially problematic in a sit-
uation wherein, within the subset of respondents that actu-
ally declared which candidate they supported, 98%
declared supporting the “losing” side (either Hillary Clin-
ton, in the 93% of the cases, or some other minor candi-
dates—such as Jill Stein or Gary Johnson).11 As a final
control variable, we also include the age of the respondent.
As can be seen from model 3, the existence of a signif-
icant impact of experts’ ideology on the PEI score is con-
firmed also after adding such a list of covariate variables.
This is reassuring for our analysis, especially once noting
how the coefficient magnitude of experts’ ideology remains
pretty stable when we move from model 1 to model 3.
Among the control variables, it is interesting to note how
augmenting experts’ familiarity with elections increases
the value of PEI as well. Note, however, that experts’ famil-
iarity with elections does not appear to mediate the impact
of experts’ ideology on the PEI score.12
In a further attempt to control for the internal validity of
our estimation, model 4 reports the result obtained by
employing a matching method via the coarsened exact
matching (CEM) algorithm (Iacus et al., 2012). CEM is
an algorithm to preprocess the data so that the estimation
of the sample average treatment effect on the treated
(SATT) based on the matched data set is less model depen-
dent, with lower bias and increased efficiency, than the one
based on the original full data set (Ho et al., 2007). More-
over, CEM works both with binary and multicategory treat-
ment variables.
The idea behind CEM is to drop observations from the
original data set so that the empirical distributions of the
covariates in the “treated” and “control” groups are similar.
When observations are exactly balanced (i.e. when the dis-
tribution of the covariates between the treated and the con-
trol groups is identical), the simple difference in means
between the outcome in the “treated” and “control” groups
provides a fully nonparametric estimator of the SATT.13 In
our case, the key causal (“treatment”) variable is the left-
right ideological placement of experts, while the covariates
included in the matching procedure are all the control vari-
ables used in model 3. Note, moreover, that given the
extremely low number of experts that place themselves
on the right of the ideological spectrum in the PEI data set,
we have collapsed all left-right ideological scores higher
than 7 to 6. This is necessary for increasing the number of
matched observations (otherwise we would have some
matched ideological cells completely empty). This also
implies that the estimated magnitude of the coefficient of
the left-right ideological scale in model 4 cannot be directly
compared with the one obtained in the previous models.14
Having said that, also in this further stringent environ-
ment, the systematic impact of our “treatment” variable
remains highly significant and substantial, suggesting again
that experts on the left-side of the political spectrum are
more likely to have a pessimistic view about integrity of the
election.
Finally, note that if we replicate the analysis reported
above by replacing the PEI score with the 11 macrosections
scores constituting the overall index of electoral integrity
for the US 2016 case, the impact of experts’ ideology on
such scores will remain strong and significant in most cases
(7 of 11; see Table 2).
The “surprising” states
According to several observers, the US 2016 Presidential
Campaign was one of the most surprising in the history of
American politics (Jacobson, 2017). Almost none of the
analytical models developed during the campaign was
effectively able to predict Donald Trump’s victory in the
Electoral College. Although in general polls were quite
accurate in predicting the vote share for the two candidates
at the national level, estimations in several states were
understandably less precise and failed to capture Trump’s
narrow margins of victory in some key battlegrounds
(Campbell et al., 2017; Terhanian, 2017). More specifi-
cally, Trump was effectively able to outpace the polls in
several states that in the last 8 years (or more) voted con-
sistently for the Democratic candidate. Regardless of the
performance of preelection polling estimates (which is not
our main concern here), the fact remains that estimates
were off especially in such key States that “swung” from
blue to red after two Presidential terms, that is, Florida,
Iowa, Michigan, Ohio, Pennsylvania, and Wisconsin. In
these six states, the mean average errors of surveys were
4.27, compared to 2.42 in the remaining swing states.15 We
believe that this specific set of circumstances is likely to
reinforce the effect of ideology on experts’ judgments as
highlighted in hypothesis 3. To control this, we test an
interaction term between expert left-right self-placement
and a dummy taking the value of 1 if the expert was judging
the electoral integrity of one of the six “swung” states.
Interestingly, if we take a closer look at the states where
Trump won, we see that they have, on average, a substan-
tially lower PEI score than the states where Clinton won
(respectively, 67 for Clinton states vs. 61 for Trump states).
The average PEI score is, however, even lower (59) for the
“swung” states.
The relatively low score for such “swung” states is
somehow puzzling. If we look at the previous two elections
(2012 and 2014) through the lens provided by an alternative
data set (the Pew Research Center (PEW) Elections Perfor-
mance Index16), we can observe that the electoral perfor-
mance of those states is rather high: Their PEW score is
higher than any other state won by Trump in 2016, but also
higher than the average of the states won by Clinton, con-
trary to what their PEI score would suggest as noted above.
As a result, the correlation between the 2016 PEI score and
 Table 2. The impact of the expert left-right self-placement on the 11 macrosections constituting the overall PEI score.
(1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (6)

Party and
Electoral Voting district Voter candidate Campaign Electoral
Electoral laws procedures boundaries registration registration Media coverage finance index Voting process Vote count Results authorities
index (0–100) index (0–100) index (0–100) index (0–100) index (0–100) index (0–100) (0–100) index (0–100) index (0–100) index (0–100) index (0–100)

Left-right self- 2.059** (0.689) 0.921 (0.583) 1.496* (0.623) 0.545 (0.410) 1.897** (0.561) 1.049* (0.413) 2.433** (0.524) 0.363 (0.372) 0.427 (0.361) 2.216** (0.293) 1.005þ (0.526)
placement
Constant 37.978** (3.923) 75.508** (2.738) 23.986** (4.963) 57.730** (1.971) 66.750** (2.217) 48.573** (1.867) 42.964** (1.982) 66.928** (1.651) 76.838** (1.557) 54.344** (1.395) 69.966** (2.857)
Observations 715 715 715 715 715 715 715 715 715 715 715
BIC 6648.910 6204.970 6630.117 6023.425 6124.571 6089.997 6127.921 5537.709 5631.915 5547.460 6228.504

Note: PEI: Perceptions of Electoral Integrity; BIC: Bayesian information criterion. Cluster standard errors on states are given in parentheses.
þ
p < 0.10; *p < 0.05; **p < 0.01.

7
8 Party Politics XX(X)

10

.3

3
Marginal Effect of Swung State on PEI

20
Marginal Effect of Left-Right on PEI
5

kdensity leftrightscale

15
.2

2
0

Density
10
-5

1
.1

5
-15 -10

0
0
0 2 4 6 8 10 0 .2 .4 .6 .8 1
Left-Right Ideological Dimension Swung State (0=NO; 1=YES)

Figure 2. Marginal impact of being in a swung state on PEI as left-right experts’ placement changes (left panel); marginal impact of left-
right experts’ placement on PEI as swung state changes. Source: model 5, Table 1 (left and right panels). PEI: Perceptions of Electoral
Integrity.

the 2012 and 2014 PEW score for the “swung” states is a hypothetical expert placed at the extreme left position
negative (0.27 and 0.34, if we focus, respectively, on (i.e. a left-right score equals to 1) is 53 (49.8–55.7); for
the 2012 or 2014 elections), contrary to what happens with an expert that places herself in a centrist position (i.e. a left-
the other states (þ0.32 for the 2012 elections; þ0.24 for the right score equals to 5.5) is 63 (61.1–64.8).
2014 elections).
Of course, it is always delicate to compare different data Conclusion
sets, which in this case also relate to different elections (a
difficulty that is of course magnified in this case where we Expert judgments are an increasingly common source in
have compared different data sets covering differing tem- social science research. They have been used in the past to
poral elections).17 When performing comparisons across study a wide palette of issues, from party positioning
(Hooghe et al., 2010) to the state of democracy worldwide
different instruments, the expectation is not one of perfect
(Knutsen et al., 2016) and the personality reputation of com-
convergence (Steenbergen and Marks, 2007), but substan-
peting candidates (Nai and Maier, 2018). Beyond being rel-
tial divergences should nonetheless be questioned.
atively simple to implement, expert judgments are often a
In this respect, model 5 shows that, coherently with
reliable and accurate alternative to the coding of primary
hypothesis 3, the experts’ ideology could explain at least
data. No need to train research assistants, spend a large
partly this apparent divergence. As can be seen, the inter-
chunk of the research grant to gather and code the primary
action term between left-right self-placement and the
data, and worry about the results of intercoder reliability
dummy for the swung states is significant and positive. This
tests: just ask the experts and aggregate their answers.
can be better appreciated by looking at Figure 2 that plots
Of course, expert judgments come with a few caveats
the marginal impact of, respectively, the swung state (Budge, 2000; Mair, 2001; Steenbergen and Marks, 2007).
dummy (left panel) and the left-right position of experts In particular, as we have shown here, experts’ ideological
(right panel) on the PEI score. heterogeneity might have a far from negligible impact on
The left-hand panel of Figure 2 shows that the negative measurement validity. In the US 2016 case here analyzed,
impact on the integrity score of being an expert in a swung for example, the skewness of the experts toward the left
state is very strong for an expert that places herself to the side of the spectrum is likely to yield a more pessimistic
extreme-left side of the ideological spectrum. Such nega- overall PEI score.
tive effect, however, rapidly decreases as experts move to Moreover, the effect of ideological considerations on
the right, becoming then insignificant. The right-hand panel experts’ judgments can be magnified in particular contexts,
of Figures 2 illustrates the complementary story of what we wherein a surprising and, possibly (according to one’s ideo-
just saw: The marginal impact of increasing by one unit on logical self-placement), unwelcome outcome risks of being
the ideological left-right scale is stronger in a swung state rationalized ex post by blaming the same electoral institu-
than in a non-swung one. As before, this trend is robust to tions that yielded such outcome.
alternative specification of the statistical model estimated The 2016 US election is a very specific election context,
and to control variables. Moreover, this trend remains that is, one of high ideological polarization (Jacobson,
rather substantial. For example, according to the estimates 2017), which may exacerbate the ideological effects we
of model 5, the expected value of PEI for a swung state for found in our data. In the already quoted Martı́nez i Coma
Curini
and van Ham (2015), the authors found that the impact of
experts’ positions on the left-right spectrum on the variance
of their evaluations related to electoral integrity (i.e. their
disagreement about election integrity) is stronger in highly
polarized contexts, such as the 2015 elections in Vene-
zuela. A similar story could be advanced also for the
2016 American case. In this respect, to what extent our
findings related to the impact of experts’ ideological het-
erogeneity on measurement validity can be generalizable to
other less exceptional and, possibly, more ideologically
convergent, contexts is an empirical question that warrants
more attention.
The validity of experts’ answers, that is, their ability to
reflect a valid empirical reality, should never however be
simply assumed. This also points at the need of always
including some relevant questions related to the ideological
positions of the respondents in any expert survey, as well as
the importance of controlling for it and eventually devel-
oping models to correct for the presence of potentially
skewed judgments.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with
respect to the research, authorship, and/or publication of this
article.
Funding
The author(s) disclosed receipt of the following financial support
for the research, authorship, and/or publication of this article:
Author’s work was supported by the Italian Ministry for Research
and Higher Education, Prin 2015 -prot. 2015P7RCL5_001.
ORCID iD
Luigi Curini https://orcid.org/0000-0003-1974-9815
Notes
1. “Experts Urge Clinton Campaign to Challenge Election
Results in 3 Swing States,” by G. Sherman, New York Mag-
azine, November 22, 2016.
2. “Clinton opens door to questioning legitimacy of 2016
election,” by D. Merica, CNN, September 19, 2017. http://
edition.cnn.com/2017/09/18/politics/hillary-clinton-russia-
2016-election/index.html
3. Data available for download at https://dataverse.harvard.edu/
dataverse/PEI
4. www.electoralintegrityproject.com.
5. According to Electoral Integrity Project, an “expert” is
defined as a political scientist (or social scientist in a related
discipline) who has published or who has otherwise demon-
strated knowledge of the electoral process in a particular
country (see Martı́nez i Coma and Frank, 2014).
6. Available at https://www.electoralintegrityproject.com/the-
year-in-elections-2017/
7. The advantage of employing an aggregate score as the Per-
ceptions of Electoral Integrity one is that it allows to consider
9
a variety of electoral integrity dimensions within itself. This
approach, however, assumes that all dimensions have an
effect of the same magnitude (and direction), an assumption
that in some cases could fail to be sustained (see Frank and
Martı́nez I Coma, 2017, on this point). As a robustness check,
and following Frank and Martı́nez i Coma (2017), we have
used the expert’s rating of a state election’s overall integrity
on a 10-point scale from 1 (very low electoral integrity) to 10
(very high electoral integrity) as an alternative dependent
variable for our statistical analysis. This latter variable is also
contained in the Electoral Integrity Project data set for the
2016 US Elections. The results reported below hold equally
well also under this further scenario. Data available upon
request.
8. One potential criticism to our empirical strategy is that
cluster-robust standard errors are asymptotic in the number
of clusters. In our sample, we have 51 clusters and precisely
for that we feel reasonably safe (see Arcenaux and Nickerson,
2009). Note also that all our results hold when running alter-
native models (fixed-effects model with dummies for each
state; multilevel model; fractional logit given that the Percep-
tions of Electoral Integrity score ranges between 0 and 1).
9. The number of experts with a left-right placement larger than
7 is rather low (less than 0.5%). The skewedness of the data
does not appear, however, to be problematic in our case. To
control that, we followed two different strategies. First, we
have collapsed all left-right ideological scores higher to 7 to
6. Second, we have left out the experts above point 7 on the
scale. In both scenarios, the relationship between ideology
and the Perceptions of Electoral Integrity score reported in
Table 1 remains unaffected. This is also true for all the other
models discussed in this article.
10. This happens always with all the different models estimated
in Table 1.
11. Considering as “losers” just the experts that declared to have
voted for Hillary Clinton (while dropping from the analysis
the “independents”) does not change any of the reported
results.
12. To further control this point, we have also explored the pos-
sibility to introduce an explicit interaction between experts’
ideology and experts’ familiarity with elections in model 3.
The interaction term, however, turns to be insignificant.
13. The strategy followed by a matching method can be seen as a
large N application of Mill’s method of difference (Sekhon,
2010).
14. After running coarsened exact matching, the £1 statistic,
which is a comprehensive measure of global imbalance in
the data ranging from 0 (perfect global balance) to 1 (com-
plete global imbalance) (see Iacus et al., 2012), decreases to
less than 0.1, producing basically a case of exactly balanced
data.
15. See the latest data published on Real Clear Politics before the
2016 Elections (https://www.realclearpolitics.com/epolls/
2016/president/2016_elections_electoral_college_map.
html). Note that also the Maine’s second congressional
10
district could be strictly speaking considered as a “swung”
state given that in the last 28 years voted consistently for the
Democratic candidate. Including also the Main as a swung
state does not alter any of the results reported below. The
same happens if we consider as “surprising states” only the
Rust Belt states (i.e. Ohio, Michigan, Wisconsin, and
Pennsylvania).
16. See http://www.pewtrusts.org/en/multimedia/data-visualiza
tions/2014/elections-performance-index. The PEW Elections
Performance Index is an objective measure, based on 17 indi-
cators, created to comprehensively assess how election
administration functions in each American state. See http://
www.pewtrusts.org/*/media/assets/2016/08/epi_methodol
ogy.pdf.
17. Comparing similar measures for a specific set of observations
is fundamental to assess the external “convergent” validity of
the data, and the PEI national scores usually perform quite
well in this regard, for example, when compared with other
measures such as the electoral integrity measure in the
NELDA data set. See Norris (2014: 101).
References
Adcock R and Collier D (2001) Measurement validity: a shared
standard for qualitative and quantitative research. American
Political Science Review 95(3): 529–546.
Alvarez RM and Grofman B (eds.) (2014) Election Administration
in the United States. New York: Cambridge University Press.
Alvarez RM, Atkeson LR and Hall TE (eds.) (2012) Confirming
Elections: Creating Confidence and Integrity Through Elec-
tion Auditing. New York: Palgrave Macmillan.
Anderson CJ and Singer MM (2008) The sensitive left and the
impervious right: multilevel models and the politics of
inequality, ideology, and legitimacy in Europe. Comparative
Political Studies 41(4–5): 564–599.
Anderson CJ and Guillory CA (1997) Political institutions and
satisfaction with democracy: a cross-national analysis of con-
sensus and majoritarian systems. American Political Science
Review 91: 66–81.
Arcenaux K and Nickerson DW (2009) Modeling certainty with
clustered data: a comparison of methods. Political Analysis
17(2): 177–190.
Benoit K and Laver M (2006) Party Policy in Modern Democra-
cies. London: Routledge.
Birch S (2008) Electoral institutions and popular confidence in
electoral processes: a cross-national analysis. Electoral Stud-
ies 27(2): 305–320.
Brooks DJ and Murov M (2012) Assessing accountability in a
post-Citizens United era: the effects of attack ad sponsorship
by unknown independent groups. American Politics Research
40(3): 383–418.
Budge I (2000) Expert judgements of party policy positions: uses
and limitations in political research. European Journal of
Political Research 37(1): 103–113.
Burden BC and Stewart III C (eds.) (2014) The Measure of
American Elections. New York: Cambridge University Press.
Party Politics XX(X)
Cain BE, Donovan T and Tolbert CJ (2008) Democracy in the
States: Experimentation in Election Reform. Washington, DC:
Brookings.
Campbell JE, Norpoth H, Abramowitz AI, et al. (2017) A recap of
the 2016 election forecasts. PS: Political Science & Politics
50(2): 331–338.
Castles FG and Mair P (1984) Left–right political scales: some
‘expert’ judgments. European Journal of Political Research
12(1): 73–88.
Clark A and James TS (2017) “Poll Workers”. In: Pippa Norris
and Alessandro Nai (eds.), Election Watchdogs. Transparency,
Accountability and Integrity. New York, NY: Oxford Univer-
sity Press, pp. 144–164.
Curini L (2010) Experts’ political preferences and their impact on
ideological bias: an unfolding analysis based on a Benoit-
Laver expert survey. Party Politics 16(3): 299–321.
Devos T, Spini D and Schwartz SH (2002) Conflicts among
human values and trust in institutions. British Journal of Social
Psychology 41(4): 481–494.
Donovan T and Bowler S (2004) Reforming the Republic: Dem-
ocratic Institutions for the New America. Upper Saddle River:
Prentice Hall.
Elklit J and Reynolds A (2005) A framework for the systematic
study of election quality. Democratization 12(2): 147–162.
Eysenck HJ (1954) The Psychology of Politics. London: Routle-
dge and Kegan Paul.
Fife BL (2010) Reforming the Electoral Process in America:
Toward More Democracy in the 21st Century. Santa Barbara:
Praeger.
Flesken A and Hartl J (2017) Party support, values, and percep-
tions of electoral integrity. Political Psychology 39: 707–724.
DOI: 10.1111/pops.12431.
Frank RW and Martı́nez i Coma F (2017) How election dynamics
shape perceptions of electoral integrity. Electoral Studies 48:
153–165.
Friedman JN and Holden RT (2009) The rising incumbent reelec-
tion rate: What’s gerrymandering got to do with it? The Jour-
nal of Politics 71(2): 593–611.
Garnett HA and Zavadskaya M (2017) Introduction. In Garnett
HA and Zavadskaya M (eds.), Electoral Integrity and Political
Regimes. New York: Routledge, pp. 1–17.
Gershtenson J, Ladewig J and Plane DL (2006) Parties, institu-
tional control, and trust in government. Social Science Quar-
terly 8: 882–901.
Goggin SN, Byrne MD and Gilbert JE (2012) Post-election audit-
ing: effects of procedure and ballot type on manual counting
accuracy, efficiency, and auditor satisfaction and confidence.
Election Law Journal: Rules, Politics, and Policy 11(1): 36–
51.
Ho DE, Imai K, King G, et al. (2007) Matching as nonparametric
preprocessing for reducing model dependence in parametric
causal inference. Political Analysis 15(3): 199–236.
Hooghe L, Bakker R, Brigevich A, et al. (2010) Reliability and
validity of the 2002 and 2006 Chapel Hill expert surveys on
Curini
party positioning. European Journal of Political Research
49(5): 687–703.
Iacus SM, King G and Porro G (2012) Causal inference without
balance checking: coarsened exact matching. Political Analy-
sis 20(1): 1–24.
Jacobson GC (2017) The triumph of polarized partisanship in
2016: Donald Trump’s improbable victory. Political Science
Quarterly 132(1): 9–41.
Jost JT, Nosek BA and Gosling SD (2008) Ideology: its resur-
gence in social, personality, and political psychology. Per-
spectives on Psychological Science 3(2): 126–136.
Knutsen CH, Møller J and Skaaning SE (2016) Going historical:
measuring democraticness before the age of mass democracy.
International Political Science Review 37(5): 679–689.
Laver M and Hunt WB (1992) Policy and Party Competition.
London: Routledge.
Leighley JE and Nagler J (2013) Who Votes Now?: Demo-
graphics, Issues, Inequality, and Turnout in the United States.
Princeton: Princeton University Press.
Mair P (2001) Searching for the positions of political actors: a
review of approaches and a critical evaluation of expert sur-
veys. In: Laver Michael (ed.) Estimating the Policy Position of
Political Actors. New York: Routledge, pp. 10–30.
Martı́nez i Coma F and Frank RW (2014) Expert judgments. In:
Norris P, Frank RW and Martı́nez i Coma F (eds.), Advancing
electoral integrity. New York: Oxford University Press, pp.
51–72.
Martı́nez i Coma F and van Ham C (2015) Can experts judge
elections? Testing the validity of expert judgments for measur-
ing election integrity. European Journal of Political Research
54(2): 305–325.
McCarty N, Poole KT and Rosenthal H (2009) Does gerryman-
dering cause polarization? American Journal of Political Sci-
ence 53(3): 666–680.
McClosky H and Chong D (1985) Similarities and differences
between left-wing and right-wing radicals. British Journal of
Political Science 15(3): 329–363.
Myagkov M, Ordeshook PC and Shakin D (2009) The Forensics
of Election Fraud: Russia and Ukraine. New York: Cambridge
University Press.
Nai A (2017) “The Fourth Estate”. In: Norris P and Nai A (eds.),
Election Watchdogs. Transparency, Accountability and Integ-
rity. New York: Oxford University Press, pp. 191–210.
Nai A and Maier J (2018) Perceived personality and campaign
style of Hillary Clinton and Donald Trump. Personality and
Individual Differences 121: 80–93.
Newton K and Norris P (2000) Confidence in public institutions:
faith, culture or performance? In: Pharr S and Putnam R (eds.),
Disaffected Democracies: What’s Troubling the Trilateral
Countries? Princeton, NJ: Princeton University Press, pp.
54–73.
Norris P (2014) Why Electoral Integrity Matters. New York:
Cambridge University Press.
Norris P (2017) Why American Elections Are Flawed (And How to
Fix Them). Ithaca: Cornell University Press.
11
Norris P and Van Es AA (eds.) (2016) Checkbook Elections?:
Political Finance in Comparative Perspective. Oxford:
Oxford University Press.
Norris P, Frank RW and Martı́nez i Coma F (2013) Assessing the
quality of elections. PS: Political Science & Politics 47(4):
789–798.
Norris P, Garnett H and Grömping M (2017) The 2016 American
Presidential Election– Perceptions of Electoral Integrity, US
2016 subnational study, Release 1.0, (PEI_US_1.0). Sydney:
The Electoral Integrity Project, University of Sydney. Avail-
able at: https://thedata.harvard.edu/dataverse/PEI (accessed 10
July 2018).
Norris P, Wynter T and Cameron S (2018) Corruption and
Coercion: The Year in Elections 2017. Sydney: The Electoral
Integrity Project. Available at: https://www.electoralintegrity
project.com/the-year-in-elections-2017/ (accessed 10 July
2018).
Piven FF, Minnite LC and Groarke M (2009) Keeping down the
Black vote: Race and the Demobilization of American Voters.
New York: New Press.
Rokeach M (1960) The Open and Closed Mind. New York: Basic
Books.
Sekhon J (2010) The Neyman-Rubin model of causal inference
and estimation via matching methods. In: Box-Steff Ensmeier
JM, Brady HE and Collier D (eds.), The Oxford Handbook of
Political Methdology. New York: Oxford University Press, pp.
271–299.
Steenbergen MR (2000) Item similarity in scale analysis. Political
Analysis 8(3): 261–283.
Steenbergen MR and Marks G (2007) Evaluating expert judg-
ments. European Journal of Political Research 46: 347–366.
Terhanian G (2017) What survey researchers can learn from the
2016 US pre-election polls. Journal of Advertising Research
57(2): 182–189.
Thompson DF (2004) Election time: normative implications of
temporal properties of the electoral process in the United
States. American Political Science Review 98(1): 51–63.
Vanhamme J and Snelders D (2001) The role of surprise in satis-
faction judgements. Journal of Consumer Satisfaction, Dissa-
tisfaction and Complaining Behavior 14: 27–45.
Zeelenberg M and Pieters R (2004) Beyond valence in customer
dissatisfaction: a review and new findings on behavioral
responses to regret and disappointment in failed services.
Journal of Business Research 57(4): 445–455.
Author biography
Luigi Curini is a full professor in political science at the Univer-
sity of Milan, Italy, and visiting professor at Waseda University,
Tokyo. His recent publications include “Politics and Big Data:
Nowcasting and Forecasting Elections with Social Media” (Rou-
tledge, 2017) and “Corruption, Ideology, and Populism. The Rise
of Valence Political Campaigning” (Palgrave/MacMillan, 2018).
His main research interests focus on spatial theory of voting,
electoral competition, legislative behavior, quantitative methods,
and social media analysis.