Asmus Leth Olsen

University of Copenhagen, Denmark

Citizen (Dis)satisfaction: An Experimental

Equivalence Framing Study

Abstract: This article introduces the importance of equivalence framing for understanding how satisfaction measures Asmus Leth Olsen is assistant profes-
sor in the Department of Political Science at
affect citizens’ evaluation of public services. Does a 90 percent satisfaction rate have a different effect than a logically the University of Copenhagen. His research
equivalent 10 percent dissatisfaction rate? Two experiments were conducted on citizens’ evaluations of hospital services focuses on the effects of performance
in a large, nationally representative sample of Danish citizens. Both experiments found that exposing citizens to a information, political and administrative
psychology, behavioral public administra-
patient dissatisfaction measure led to more negative views of public service than exposing them to a logically equivalent tion, and the application of experimental
satisfaction metric. There is some support for part of the shift in evaluations being caused by a negativity bias: dissatis- methods in public administration. His
faction has a larger negative impact than satisfaction has a positive impact. Both professional experience at a hospital work has appeared in journals such as
Political Behavior, Public Choice, and
and prior exposure to satisfaction rates reduced the negative response to dissatisfaction rates. The results call for further Judgment and Decision Making.
study of equivalence framing of performance information. E-mail: ajlo@ifs.ku.dk

Practitioner Points
• The valence (positive/negative) of performance information can have substantial effects on citizens’
perception of public services—even if the underlying performance is exactly the same.
• Presenting citizens with a dissatisfaction rate of 10 percent induces a much more negative evaluation of public
services than presenting them with a logically equivalent satisfaction rate of 90 percent.
• Policy makers must carefully consider how minor equivalent changes in the presentation of performance
information can induce large shifts in citizens’ perceptions of public service performance.

C
itizen and user satisfaction surveys have
become a widespread performance informa-
tion metric across countries, services, and
levels of government (Bouckaert, Van de Walle, and
Kampen 2005; Stipak 1980). Citizen satisfaction is
seen as a key way of overcoming the many difficul-
ties of measuring actual outcomes in the public sector
by applying more subjective user-centered measures
rather than objective quality or output metrics
(Bouckaert and Van de Walle 2003; Folz 1996;
Holzer and Yang 2004). At the same time, there has
been a focus on potential biases in citizen satisfaction
as an indicator of service outcomes (Kravitz 1998;
Stipak 1979; Van de Walle and Van Ryzin 2011;
Van Ryzin 2013; Van Ryzin et al. 2004; Williams
1994). However, there has been little to no focus on
the potential biases that publicly available satisfac-
tion measures induce in the attitudes and behaviors
of citizens and policy makers. This article turns its
focus to how malleable citizens’ evaluations of public
services are when citizens are confronted with satisfac-
tion measures of different valence but with equivalent
information content. That is, how does a positive
versus a negative framing of the exact same level of
satisfaction affect citizens’ perceptions about a service?
The article advances this research agenda along two
lines. First, it introduces the importance of equiva-
lence framing for our understanding of how satisfac-
tion measures affect citizens’ evaluations of public
services (Druckman 2004; Levin, Schneider, and
Gaeth 1998). Specifically, it points out that logi-
cally equivalent changes to the valence of satisfaction
measures can have large effects on citizens’ subsequent
evaluations of public services. The trivial substitu-
tion of a 90 percent satisfaction rate for a 10 percent
dissatisfaction rate may transform positive associations
and memories into negative ones for the exact same
underlying information, which, in turn, shifts citizens’
evaluations of the data. The article tests this hypoth-
esis by employing two experimental studies using a
large-scale, nationally representative sample of Danish
citizens (n = 3,443). It will highlight just how sensi-
tive the effect of performance measures on citizens’
perceptions of public services is to minor changes in
the descriptive valence.
Second, the article offers an explanation of the
potency of equivalency frames from the basis of Public Administration Review,
Vol. 75, Iss. 3, pp. 469–478. © 2015 by
a negativity bias. The negativity bias implies that The American Society for Public Administration.
“negative events are more salient, potent, dominant DOI: 10.1111/puar.12337.

Citizen (Dis)satisfaction: An Experimental Equivalence Framing Study 469

in combinations, and generally efficacious information as an attempt to assign num-
than positive events” (Rozin and Royzman The negativity bias implies that bers to the inputs, outputs, and outcomes of
2001, 297). In the study of performance citizens are asymmetrical in public services (Behn 2003; James 2011b;
information, the negativity bias implies that their responses to good and bad Moynihan 2008). Many such measures can
citizens are asymmetrical in their responses
performance, reacting mostly to be presented with varying valence of the label
to good and bad performance, reacting describing the underlying numerical attribute.
mostly to the latter. The bias has found some the latter. Examples of this would be “death rates” versus
support in observational studies (James and “survival rates,” “unemployment rates” versus
John 2007; Boyne et al. 2009) and in experimental studies (James “employment rates,” or, for our focus here, “satisfaction rates” versus
2011a; Olsen 2013a). As James has argued, “More investigation “dissatisfaction rates.” With the help of numbers, these labels can
of possible difference in magnitude of effect between informa- be shifted while holding the numerical value of the performance
tion about good and bad performance is merited” (2011b, 414). information logically equivalent. For instance, we can choose to
However, the bias is difficult to detect if there is no neutral point report a 90 percent satisfaction rate or a 10 percent dissatisfaction rate.
of comparison for the asymmetrical effects of “good” versus “bad” These are two objectively equivalent ways of presenting the exact
performance. same information while varying the valence of the wording of the
performance measure.
Another challenge is the extent to which “good” and “bad” perfor-
mance are in fact qualitatively different. Maybe being asymmetrical However, changing the valence of performance information is
in our response is sometimes warranted if bad performance is inher- important and not trivial because information is often encoded
ently different. The experiments applied here overcome the latter according to its descriptive valence (Levin and Gaeth 1988;
problem by comparing logically equivalent pieces of negative- and Quattrone and Tversky 1988). The encoding evokes our associative
positive-valence information. The former challenge is approached memory by making associations of similar valence more accessible: a
by testing how the framing effect is moderated by individual dif- positive-valence description of an attribute leads to positive associa-
ferences in alternative sources of information about performance tions in our memory, while a negative valence directs our memory
and prior (experimental) exposure to information of opposite toward negative associations. For example, a dissatisfaction rate
valence (Chingos, Henderson, and West 2012; Druckman 2004; makes negative associations about a public service more accessible.
Johnsen 2012). We can think of this as associations with a valence reflecting dis-
satisfaction moving to the foreground: displeasing experiences, long
In summary, the article adds to the existing body of research in waiting times, low-quality service, unresponsive employees, and
two major ways: (1) to show how simple variations in equivalent so on. On the other hand, when we are presented with a satisfac-
presentations of performance information affect citizens’ percep- tion rate, associations consistent with being satisfied become more
tion of public services and (2) to test how part of this effect is accessible. The result is a “valence-consistent shift” in our judgment
driven by a negativity bias in citizens’ responses. In addition, of the attribute (Druckman 2004; Levin, Schneider, and Gaeth
the more general implications of the study are threefold. First, 1998). That is, positive-valence information will lead to more posi-
the potency of equivalence frames for citizen satisfaction rates tive evaluations and negative-valence information to more negative
in particular—and performance information in general—poses evaluations—even if the underlying numerical information is logi-
fundamental questions about the fragility of these measures. cally equivalent.
Second, the article highlights the importance of framing research
to our understanding of how performance information can serve This leads to the first expectation of the study: logically equivalent
an external accountability role. Third, the article informs our ways of framing citizen satisfaction can shift citizens’ evaluations of
understanding of how reactions to performance information are public services in either a negative or positive direction. Specifically,
affected by a negativity bias and what this implies for blame- exposure to a satisfaction rate should induce more positive evalu-
avoiding behavior among policy makers. It also has implications ations of public services, whereas exposure to a dissatisfaction rate
for our understanding of the potential for manipulating perfor- should result in more negative evaluations.
mance measures by those responsible for collecting and reporting
them. These implications are further addressed in the concluding Equivalence Framing Effects and the Negativity Bias
section. At this point, we have stated the simple expectation that attribute
equivalence frames can cause valence-consistent shifts. Here we
Equivalence Framing and the Valence of Citizen (Dis) introduce the idea that valence-consistent shifts can be asymmetri-
satisfaction cal. Specifically, the negativity bias implies that negative information
An equivalence framing effect occurs when “two logically equivalent has a stronger impact than positive information of the same magni-
(but not transparently equivalent) statements of a problem lead tude. Metareviews of the negativity bias in psychology have found
decision makers to choose different options” (Rabin 1998, 36). consistent support for this asymmetry across human perception,
With the notion of equivalence framing in mind, we can easily see memory, decision making, and behavior (Baumeister et al. 2001;
that many performance information metrics can be presented as Rozin and Royzman 2001). We tend to pay more attention to,
multiple variations of different “objectively equivalent descriptions and direct more cognitive capacities toward, negative information
of the same problem” (Levin, Schneider, and Gaeth 1998, 150). than positive. Stronger reactions to negative information are likely
For the study of performance information, equivalence frames to exist for evolutionary reasons: negative events (illness, combat,
have a very immediate relevance. Most definitions see performance hunger, etc.) could be life threatening, while positive events usually
470 Public Administration Review • May | June 2015
did not have the same immediacy (Hibbing, Smith, and Alford asymmetry of a negativity bias in this case would employ a neutral
2014, 303). reference point in the middle of satisfaction and dissatisfaction.
Here we confront this challenge in two ways.
This idea has also found its way into political psychology, which,
for a relatively long time, has focused on the asymmetrical effects of The first approach to unlock the dilemma is to consider how
“positive” or “negative” information (Lau 1982, 1985). In politics, individual differences in alternative sources of information affect
the negativity bias means that blame for bad performance is assigned responses to different-valence information. Here we draw on
to a much greater extent than is credit for good performance of a Johnsen, who speculated whether the negativity bias works differ-
similar magnitude. Retrospective voting studies have found that ently for “public services where people in general have less direct
a worsening economy damages the incumbent to a greater extent experience” (2012, 139). Along the same lines, James has argued
than an improving economy helps (Bloom and Price 1975; Kinder that the negativity bias may depend on how consistent the perfor-
and Kiewiet 1979; Mueller 1973). In recent years, this research has mance information is with “personal experience or word of mouth”
spread beyond the traditional measures of economic performance (2011b, 414). Consistent with this, Chingos, Henderson, and West
indicators. Boyne et al. (2009) found evidence of a negativity bias in (2012) found that mostly citizens with few alternative sources of
the effect of municipal performance information on electoral sup- information about school performance had the strongest response
port among English local governments. James and John (2007), also to accountability ratings of school performance. Citizens can draw
in an English local government setting, found that voters primarily on multiple informal sources for performance information about
punished poor performance and did not reward good performance the public sector. These include media reports, personal experience,
to the same extent. Soroka (2006) found that negative economic advice from family and friends, political debates, or inference from
performance was covered more intensely in the media than positive visible traits of a particular organization, such as its current users,
economic performance of a similar magnitude. facilities, staff, or manager. As laid out by James, “Citizens may not
have much of an idea about the overall performance of a local pub-
Recently, there has also been a set of experimental studies focusing lic body only interacting with it on a case-by-case basis for a subset
on a negativity bias in citizens’ responses to good and bad perfor- of services” (2011b, 402).
mance information. James (2011a) found both experimental and
observational support for a negativity bias. These alternative sources may alter how
Using a survey, he found that poor prior performance information is encoded. Direct
performance was punished more than excel-
If citizens generally respond experience can be an important source of
lent prior performance was praised. Further, more strongly to negative information about services, which can affect
he also found a negativity bias in an experi- information, we can expect that how formal performance information is used.
ment on citizens’ service expectations, as negatively framed performance If citizens generally respond more strongly
poor performance affected expectations more information will affect mostly to negative information, we can expect that
than excellent performance. However, James citizens with limited prior negatively framed performance information
(2011b) found no support for a negativity will affect mostly citizens with limited prior
bias in a laboratory setting in which partici-
information. information. Therefore, we should expect that
pants were exposed to “good” and “bad” per- alternative information sources will dimin-
formance information, although a possible explanation for this may ish the negativity bias of performance information. This would be
be a discrepancy between the information provided and the actual consistent with the more general finding that individuals with high
performance of the jurisdiction in question. Another recent experi- personal involvement or strongly held attitudes are less suscep-
ment on voters’ prospective performance preferences found support tible to framing effects (Druckman 2011; Levin, Schneider, and
for a negativity bias in relative performance evaluations, as voters Gaeth 1998).
showed a strong preference for “not falling behind” other countries
while showing very little interest in “getting in front” (Hansen, The second approach states that the asymmetrical effects of a
Olsen, and Bech 2014). James and Moseley (2014) found reduc- satisfaction rate and a dissatisfaction rate can be teased out by
tions in satisfaction for citizens exposed to low performance and no exposing citizens to a sequence of both. The idea is that we can
detectable increase in satisfaction for cases of high performance. detect asymmetrical responses to positive and negative informa-
tion by looking at how evaluations change when citizens become
This leads to our second expectation: that the valence-consistent aware of the equivalency. This can be done by comparing the
shifts will be overly influenced by the negative valence. That is, the evaluations of citizens exposed to different sequences of conflict-
difference we may observe between a citizen satisfaction rate and ing valence (e.g., negative to positive or positive to negative) with
a citizen dissatisfaction rate is attributable to the larger negative those of citizens exposed to performance information of the same
impact of the latter and less so to the positive impact of the former. valence (e.g., positive/positive or negative/negative). Generally,
exposing individuals to mixed types of valence frames should
Negativity Bias or Positivity Bias? reduce the framing effect as they become aware of the equivalence
A major challenge to the foregoing hypothesis is the lack of a proper (Druckman 2004, 2011). For a negativity bias to be present, we
counterfactual for comparison. If we are able to show that a satisfac- expect that prior exposure to a satisfaction frame should reduce the
tion rate affects citizens’ evaluations of public services differently valence-consistent shift in subsequent evaluations when exposed to
from a dissatisfaction rate, how can we know which one of the two a dissatisfaction frame—but this should not be the case the other
frames has a greater impact? Ideally, identifying the underlying way around.
Citizen (Dis)satisfaction: An Experimental Equivalence Framing Study 471
Design: Two Experimental Studies in a Large collected during October 15–22, 2012. For participating in the
Representative Sample survey, participants received a number of points, which they could
The expectations formulated earlier will be examined through two use in a shop administered by YouGov. The study was restricted
separate experiments conducted in a large, nationally representa- to citizens between the ages of 18 and 74 and was prestratified on
tive sample of Danish citizens. The context for both experiments is gender, region, age, and political party choice in order to achieve a
Danish hospital services. The Danish health care system is a corner- near-representative sample of the Danish population. Table 1 shows
stone of the modern Danish welfare state. In the Danish health care descriptive statistics for the sample and for the Danish population
system, 85 percent of the costs are financed through taxes. Health in the specified age range. It shows a sample that, overall, reflects the
care services are administered at three different political levels, general population on these general sociodemographic characteris-
namely, the central government, five regions, and 98 municipali- tics. Furthermore, the sample is highly diverse and largely represent-
ties. Here the focus is on hospitals, which are under the political ative in terms of political party choice and education. In summary,
authority of the five regions. Reporting various measures of patient the sample provides strong external validity of the findings across
satisfaction with hospital care is common practice today across most citizens with very different background characteristics.
developed countries (Kravitz 1998; Mannion, Davies, and Marshall
2005; Pope 2009; Williams 1994). There are already some stud- Experimental Design and Procedure
ies on how performance reporting affects health professionals and The experiment was a between-subjects design, as outlined in
patients (Hibbard, Stockard, and Tusler 2003, 2005). table 2. The premise of the experiment was that respondents were
asked to evaluate a hospital given a single piece of performance
Study 1: Experiment on Hospital Satisfaction/ information. The experiment took the following form: All respond-
Dissatisfaction ents were provided a short factual note about the Danish Health
The purpose of the first experimental study was twofold: (1) to test and Medicines Authority. Following this, they were asked to evalu-
whether valence-consistent shifts in evaluations of hospital services ate the performance of an unnamed hospital. Here the experiment
happened in response to exposing citizens to either satisfaction randomized the performance information provided to respondents
rates or dissatisfaction rates, and (2) to test whether the effect was at two levels. Overall, the two levels of treatment constituted an
moderated by citizens’ alternative sources of information about equivalence framing experiment in which respondents were assigned
hospital services in order to identify a negativity bias as an underly- logically equivalent pieces of information (Levin, Schneider, and
ing mechanism. Gaeth 1998; Tversky and Kahneman 1981). At the first level of
randomization, respondents were randomly assigned to two differ-
Participants ent conditions. The two conditions differed in how the hospital’s
The study relied on a large, nationally representative sample performance was framed in terms of either patient satisfaction or
recruited through YouGov’s Danish online panel (n = 3,443). patient dissatisfaction. Clearly, the satisfaction frame constituted
Through the panel, YouGov made contact with 8,204 respond- the positive-valence framing of performance information, while
ents, which means the response rate was 42 percent. The data were the dissatisfaction frame stressed the negative-valence aspects of the
performance information.
Table 1 Descriptive Statistics
Variable Sample Population
In order for the treatments to be logically equivalent, a second
level of random assignment was introduced in which respondents
Gender (male) 49.8% 49.1%
Age were randomly assigned various percentages of satisfaction or dis-
18–29 12.3% 16.6% satisfaction. For the two different frames, the numerical content
30–44 22.1% 27.8% was drawn from two different uniform distributions. In a uniform
45–59 29.3% 26.6%
distribution, the values within an interval have an equal probability
60–74 37.6% 29.3%
Geographic region of being drawn. In the satisfaction frame, the uniform distribution
Capital area 24.7% 28.3% ranged from 75.0 percent to 95.0 percent satisfied patients (pre-
Zealand 15.3% 14.5% sented with one decimal). In the dissatisfied frame, this interval was
Southern Denmark 23.9% 23.3%
inverted, providing a range of treatment values from 25.0 percent
Middle Jutland 23.8% 22.4%
Northern Jutland 12.3% 11.7% to 5.0 percent dissatisfied patients. For instance, some respondents
were given the following treatment: “At the hospital, 10 percent
Note: CAWI (computer-assisted web interviewing) survey in the Danish YouGov
panel. of the patients are dissatisfied with their treatment.” At the same
N = 3,443. time, other respondents were provided with the logical equivalent:

Table 2 Experimental Design in Study 1

Baseline Question (all participants) Treatment Frame Treatment Wording (randomly assigned) Numerical Treatment (randomly assigned)
Danish Health and Medicines Authority consistently A: Satisfaction At the hospital, X% of the patients are X ∈ U (75.0, 95.0)
records how patients experienced their treatment N = 1,716 satisfied with their treatment.
at Danish hospitals. How do you think the fol- B: Dissatisfaction At the hospital, X% of the patients are X ∈ U (5.0, 25.0)
lowing hospital is doing? N = 1,727 dissatisfied with their treatment.
Notes: Outline of experiment conducted with YouGov’s Danish online panel (n = 3,443). Participants were randomly assigned to one of two conditions (satisfaction/
dissatisfaction) and randomly assigned a percentage level of satisfaction/dissatisfaction.

472 Public Administration Review • May | June 2015

“At the hospital, 90 percent of the patients are satisfied with their
treatment.” That is, for each set of respondents asked to evaluate a
given rate of hospital satisfaction, the data contain a similar set of
respondents evaluating logically identical metrics of hospital dissat-
isfaction. The random assignment of percentages provided respond-
ents with around 100 different treatment values under each frame.
The assignment of a large range of different values allows for testing
whether the framing of performance is dependent on the numerical
content of the frame. Numerical content has been found to affect
both citizens’ attitudes and the behaviors of policy makers in unex-
pected ways (Ansolabehere, Meredith, and Snowberg 2013; Olsen
2013b, 2013c). By randomizing the numerical content, the results
cannot be driven by idiosyncratic artifacts of the numerical values
that respondents were given. Therefore, it enhances the robustness
of the findings if they hold for a large range of numerical values. For
the outcome measure, the respondents were asked to provide their
evaluation of the hospital. Their response was given on a 101-point
sliding scale ranging from “very bad” (0) to “very good” (100).
Respondents could not choose to not respond or to provide a “don’t
know” response. Across all treatments, the average response was
55.6 (SD = 24.7) and the median response 57.
It is important to note that the experiment did not allow us to
directly assess the question of a negativity bias. As outlined in the
theory section, if we found a difference in hospital evaluations
between the two frames, we would not be able to directly attribute
this difference solely to one of them specifically. Ideally, we would
like to have some neutral point of satisfaction or dissatisfaction
to which we could compare the two frames in order to measure
which one of them has an asymmetrical impact on evaluations.
Unfortunately, no such neutral category is easy to come up with for
satisfaction measures.
Instead, as argued in the theory section, we used alternative
information sources as a moderator that could shed some light on
which frame was driving a potential effect. In terms of alternative
information sources, we relied on two different indicators. The first
was a dummy variable indicating whether the respondent had been
to a hospital within the last year (17.8 percent). This indicated a
personal experience with hospital services, which can be seen as
an important alternative source of performance information. The
second indicator was a dummy variable indicating whether the
respondent either currently worked or previously had worked at a
hospital (10.6 percent). This indicator captured any type of profes-
sional work-related experience in a hospital setting. In summary,
both indicators captured the extent to which respondents had
alternative sources of hospital performance other than the perfor-
mance information they were provided in the experiment. We also
included some controls in some specifications in order to make sure
that these two indicators did not simply capture some other factor
correlated with alternative information sources and evaluations.
The controls included age, gender, and region. We also included a
dummy for respondents with private sector employment (32.6 per-
cent) to capture differences between them and those in the public
sector or out of work. Finally, a dummy captured respondents with
an intent to vote for one of the four parties that were either in gov-
ernment or that supported the government at the time of the study
(36.5 percent).
Citizen (Dis)satisfaction: An Experimental Equivalence Framing Study 473
0.03
Satisfied frame
0.02
Dissatisfied frame
Density
0.01
0
0 25 50 75 100
Citizens’ Evaluation of Hospital Performance
Notes: The x-axis represents the dependent variable of citizens’ evaluations of
hospital performance. Higher scores indicate a better evaluation.
N = 3,443.
Figure 1 Density Plot of the Distribution of Responses under
the Two Frames
Empirical Results
The distributions of responses under the satisfied and dissatisfied
frames are reported in the density plot in figure 1. Under the satis-
fied frame, citizens gave the unnamed hospital an average score of
65.9 (SD = 20.7). However, for the dissatisfied frame, the average
score was only 45.4 (SD = 24.1). On average, citizens evaluated
hospitals under the satisfied frame as significantly better, with an
average difference of 20.5 points (p < .01). The effect is substantially
similar or even larger if medians or trimmed means are calculated.1
The effect is substantial: the mean difference corresponds to about
a one-standard-deviation change in the dependent variable. This
strongly indicates a valence-consistent shift in evaluations induced
by framing performance information as either satisfaction or
dissatisfaction.
In figure 2, the framing effect is shown across the numerical treat-
ments. Table 3 reports further tests of the framing effect. From the
figure and table, we can see that citizens responded positively to both
higher satisfaction and lower dissatisfaction rates. On average, evalu-
ation improved significantly, by around 0.9 point for each percent-
age point improvement in satisfaction/dissatisfaction (see model B).
We can now compare the frame treatment and numerical treatment
in magnitude. Doing so tells us that an approximately 22-point
improvement in the percentage of satisfied/dissatisfied has the same
effect as changing the overall framing from dissatisfaction to satisfac-
tion. This strongly indicates the potency of the valence framing effect.
We can compare differences in responses to changes in satisfac-
tion and dissatisfaction rates. In table 3, model C, this is done by
interacting the treatment frame with the numerical frame. The
interaction term is positive, which indicates a stronger response to
numerical changes in the dissatisfaction frame. However, the effect
is not significant. This implies that a difference in the numerical
magnitude of positive- and negative-valence information does not
assert any influence. The only thing that matters is the valence of
the information, not its magnitude.
The next step was to understand the underlying mechanisms
for the large difference in evaluations for the satisfaction and
 Treatment B: Pct. Dissatisfied with Hospital Services
25% 20% 15% 10% 5%

100

Citizens’ Evaluation of Hospital Performance

75 Satisfied

Dissatisfied
50

25

75% 80% 85% 90% 95%

Treatment A: Pct. Satisfied with Hospital Services

Notes: Ordinary least squares estimated slopes for the satisfied (black line and dots) and dissatisfied (gray line and dots) frames. The lower horizontal axis shows the
treatment percent received under the satisfied frame. The upper horizontal axis shows the treatment percent assigned under the dissatisfied frame.
N = 3,443.

Figure 2 Effect of Satisfied Percentage and the Dissatisfied Percentage

Table 3 Ordinary Least Squares Results from Experiment in Study 1

Model A Model B Model C Model D Model E Model F
Treatment frame (1 = dissatisfaction) −20.51** −20.68** −30.25** −21.46** −21.27** −22.06**
(.77) (.74) (10.89) (.78) (.82) (.86)
Treatment percent .94** 0.88** .93** .94** .94**
(.06) (.10) (.06) (.06) (.06)
Frame * Treatment percent .11
(.13)
Hospital work experience −1.68 −1.81
(1.72) (1.74)
Patient experience .97 0.79
(1.37) (1.38)
Work experience * Treatment frame 7.39** 7.31**
(2.41) (2.41)
Patient * Treatment frame 3.32 3.16
(1.94) (1.94)
Intercept 65.87** −13.73** −9.00 −13.26** −14.35** −14.28**
(.54) (5.45) (7.66) (5.44) (5.45) (5.74)
Adjusted R2 .17 .22 .22 .22 .22 .24
F-statistic 717.2 488.6 325.9 248.1 247.2 59.2
N 3,443 3,443 3,443 3,443 3,443 3,413
Notes: Ordinary least squares estimates with standard errors in parentheses. Significance levels denote *p < .05 and **p < .01. Model F includes the following control
variables: gender dummy, age, government party supporter dummy, dummy for private sector employment, five dummies for educational level, and five regional
dummies. Sample size in model F is reduced due to missing values on the education measure.

dissatisfaction frame. Here we studied heterogeneous responses for
respondents with alternative information sources about hospital
services. Table 3, models D–F report these results. In figure 3,
coefficients with 95 percent confidence intervals are shown for (1)
whether the respondent had been hospitalized within the last year
or (2) whether the respondent currently worked or had worked at
a hospital. The mean differences are based on the coefficients in
table 1, model F.
For respondents with hospital work, the interaction effect is
positive and significant at 7.4 points. That is, in the dissatisfied
frame, respondents with current or prior work experience at a
hospital gave higher evaluations than those with no experience.
474 Public Administration Review • May | June 2015
Interestingly, the main effect of work experience is negative and
insignificant, which suggests that there is no difference under the
satisfied frame. In other words, alternative sources of informa-
tion matter for the effect of negative-valence information but do
not alter the effect of positive-valence information. This result
is substantially the same if we add a set of control variables (cf.
model F). The finding is also substantially the same if we inter-
act all control variables with the treatment frame in the same
model.2
In figure 3, we can directly compare the effects for the two frames.
The mean difference between the two frames amounts to a statisti-
cally significant effect of 9.1 points (p < .01) between those with and
 12
10
Mean Difference in Evaluation
8
−2
−4
Satisfaction
Hospital work experience vs. none
Note: Mean differences in hospital evaluations with 95% confidence intervals.
N = 3,443.
Figure 3 Alternative Information Sources and Frame Effect
without hospital work experience. The other indicator of alternative
sources of information compared respondents that had been to a
hospital in the past year with everybody else. The interaction effect
is also positive but not significant. We do not observe the same
effect for this group. The mean difference-in-difference between the
two frames is 2.4 (p = .31).
In summary, the distance between evaluations under the positive
and negative frame is reduced by having alternative sources of infor-
mation, and this effect is driven solely by a less negative response
under the dissatisfied frame. Importantly, if the results were driven
by a generally higher degree of satisfaction with hospital services
among these groups, then we would expect them to respond more
positively under both frames. However, this is not what the findings
indicate. The fact that alternative information sources affect only
evaluations under the dissatisfaction frame indicates that alterna-
tive information sources are likely to play a role in diminishing the
negativity bias.3
Study 2: Experiment with Multiple Performance Cues
The second study served two purposes: (1) we were able to assess
whether the results of study 1 would hold up if the performance
information on satisfaction and dissatisfaction was presented in a
context of alternative sources of information, and (2) the sequence
of valence stimuli between study 1 and study 2 allowed us to obtain
a better understanding of the underlying mechanism. Specifically,
the experiment in study 2 occurred after study 1, which allowed
us to analyze the effects in study 2 conditional on the treatment
received in study 1. Study 2 relied on the exact same participants as
study 1.
Experimental Design and Procedure
Study 2 had a between-subjects design. All respondents were con-
fronted with the following hypothetical scenario: “Imagine that you
have become sick and that you therefore need to have a nonemer-
gency operation made at a hospital. About the hospital, you know
Dissatisfaction Satisfaction Dissatisfaction
Patient experience vs. none
Table 4 Experimental Design in Study 2
Information Source Experimental Variation
Media coverage of the hospital is: Positive/negative
The proportion of former patients who were 90% satisfied/50% satisfied/
satisfied/dissatisfied with their treatment: 10% dissatisfied/50% dissatisfied
Your neighbors view of the hospital is: Positive/negative
Notes: Respondents were assigned to one of 16 possible combinations of values
for the three stimuli variables. The experiment was conducted with YouGov’s
Danish online panel (n = 3,443).
that . . .” Below the introduction was a box with three types of
information about the hospital that varied in terms of the valence
of the information provided. The three sources of information were
shown simultaneously. Information sources and the experimental
variation are reported in table 4. The main source of information
echoed that of study 1: participants were informed about satisfac-
tion/dissatisfaction performance measures for former patients of the
hospital.
As in study 1, the percentage of satisfied/dissatisfied was var-
ied, here with only four equivalent quantities. The two alterna-
tive sources of information were media coverage and neighbors’
opinions. These could vary only in terms of being either positive
or negative. Importantly, these were not equivalents of different
valence but stated actual substantive differences. Combining treat-
ment variations across the treatment types of information sources
generated 16 different treatment combinations (2 x 2 x 4 full facto-
rial design). This gave us approximately 215 respondents in each
condition.
Below the information box, the respondents were asked to report
how likely they were to undergo the operation at the aforemen-
tioned hospital. The response was provided on a scale similar to that
used in study 1, a 101-point sliding scale ranging from “not at all
likely” (0) to “very likely” (100). Across all treatments, the average
response was 49.0 (SD = 26.9), with a median of 51.
Citizen (Dis)satisfaction: An Experimental Equivalence Framing Study 475
Table 5 Ordinary Least Squares Results from Study 2 However, if respondents received a dissatisfaction frame in study
Satisfaction/dissatisfaction (ref. = 10% dissatisfied) (ref.) 2, their response depended in part on their treatment frame in
90% satisfied 15.62** study 1. Specifically, those who received a satisfaction frame in
(1.04) study 1 evaluated the dissatisfaction frame significantly better, by
50% satisfied −11.67** 4.7 points, in study 2 than those who had dissatisfaction frames in
(1.03)
50% dissatisfied −19.24**
both studies (p < .01). This did not, however, work in reverse. Those
(1.04) first exposed to a dissatisfaction frame did not rate the satisfaction
Media coverage (1 = positive) 12.97** frame any differently from those first exposed to a satisfaction frame
(0.73) (p = .83). The results are substantially the same or stronger if median
Neighbor opinion (1 = positive) 13.28**
(0.73) values are calculated.4 In other words, exposure to positive-valence
Adjusted R2 0.36 information diminished the (negative) response to subsequent
F-statistic 386.3 negative-valence information—but not the other way around. One
N 3443 interpretation is that respondents first exposed to the satisfaction
Notes: Ordinary least squares estimates with standard errors in parentheses. frame became aware of the equivalence and therefore viewed the dis-
Dependent variable: Likelihood of undergoing the operation at that hospital,
“not at all likely” (0) to “very likely” (100). Significance levels denote *p < .05
satisfaction frame more mildly (Druckman 2004). This “correction”
and **p < .01. then did not happen the other way around, which could indicate
that the satisfaction frame was viewed as the baseline or “regular”
Table 6 Mean Responses in Study 2 Conditional on Treatment in Study 1 metric. This interpretation supports the negativity bias in the sense
Study 2 that exposure to both forms leads to a calibration of judgment con-
Study 1 Satisfaction Frame Dissatisfaction Frame cerning the negative information and not the positive information.
Satisfaction frame 54.7 45.6
[52.9–56.5] [43.9–47.3] Conclusion
(n = 868) (n = 848) Citizens’ immediate responses to satisfaction metrics can be highly
Dissatisfaction frame 55.0 40.9
[53.2–56.8] [39.2–42.6] contingent on an arbitrary choice of a positive or negative label
(n = 855) (n = 872) description. This is not a trivial fact in a world in which citizens’ sat-
Notes: Dependent variable: Likelihood of undergoing the operation at that isfaction rates play a key role in informing the public about public
hospital, “not at all likely” (0) to “very likely” (100). Mean responses with 95% sector performance. This article has shown how equivalence framing
confidence intervals in brackets and sample sizes in parentheses. of citizen satisfaction measures can assert a huge impact on how citi-
zens evaluate public services. Two experiments in a large, nationally
Empirical Results representative sample of Danish citizens highlighted the following
The main results are reported in table 5. First, we can note that major findings on the potency of equivalence framing.
respondents valued positive media coverage and neighbors’ opinions
at the same order of magnitude. Changing Study 1 found that framing hospital perfor-
media coverage or neighbors’ opinions from Citizens’ immediate responses mance in terms of dissatisfaction instead of
negative to positive caused about a 13-point to satisfaction metrics can be satisfaction had a substantial negative impact
increase in the likelihood of choosing the hos- on citizens’ evaluations of hospital services.
pital for a nonemergency operation. Turning to
highly contingent on an arbi-
The effect was constant along a large inter-
the satisfaction/dissatisfaction frames, we find a trary choice of a positive or val of different numerical treatments for the
similarly large and significant effect as in study negative label description. percentage of patients being either satisfied
1. Shifting from a hospital with 90 percent or dissatisfied. Study 2 replicated the fram-
being satisfied to one with 10 percent being dis- ing effect in an experiment that also offered
satisfied changes the likelihood of participants choosing that hospital alternative sources of information with both negative and positive
by around 15 percentage points. For the lower level of 50 percent valence. Importantly, the change in effect attributable to the equiva-
satisfaction/dissatisfaction, the difference is around 7.5 points ( p < lence frame was comparable to valence changes for media coverage
.01). This highlights that study 1 may have induced an awareness of and neighbor opinions, which both were nonequivalence frames.
the equivalence between satisfaction and dissatisfaction that reduced
the overall framing effect in study 2 (Druckman 2011). Theoretically, we also aimed to interpret part of this effect to reflect
a negativity bias. That is, differences in responses to satisfaction and
This being said, it speaks to the power of the equivalency frame of dissatisfaction measures are driven by a disproportionate impact
satisfaction/dissatisfaction that the effects are similar for those found of negative-valence performance information—in this instance, a
for logically equivalent changes in valence compared with logically dis- dissatisfaction rate. Some additional findings support this inter-
similar changes in valence (e.g., positive versus negative media cover- pretation. First, respondents with current or prior work experi-
age). Finally, we combined the treatments in study 1 and study 2 ence at a hospital were less affected by the dissatisfaction frame
in order to test whether switching frame types changes the effect of but responded like everyone else to the satisfaction frame. We take
exposure to satisfaction and dissatisfaction percentages. The results this as an indication that alternative sources of information about
are reported in table 6. performance diminish the negativity bias. Second, by combining
treatments across studies, we were able to show that those exposed
For those who received the satisfaction frame in study 2, the to satisfaction in study 1 and dissatisfaction in study 2 were less
frame received in study 1 did not affect their mean evaluations. negative in their assessment than those exposed to other treatment
476 Public Administration Review • May | June 2015
combinations. We again take this to support a negativity bias
interpretation of the main results because it highlights that prior
exposure to positive-valence information weakens the response to
negative-valence information—but not the other way around.
A minor cautionary note on the limits of the findings should be
made as a result of the hypothetical, low-incentive setting of the
experimental design. Performance information is often presented in
a more data-rich context of news reports, government Web pages,
or official publications. This being said, the analysis points to the
importance of equivalence framing and, in part, the negativity bias
for citizens’ response to satisfaction measures, and the findings were
very substantial and identified in a large representative sample. In
addition, we could argue that the identified framing effect and nega-
tivity bias could be even larger in a real-world setting, where citizens
have to make sense of a greater number of cues about performance.
This could leave them even more vulnerable to the effects found
across the two studies.
The study offers three broader implications for our understanding
of citizens’ satisfaction measures and performance information more
generally: The first implication relates to our use and understanding
of the effects of citizen satisfaction measures. Today, both govern-
ments and researchers are focused on citizen satisfaction. We now
know that changing the discussion to dissatisfaction has the potential
to fundamentally affect citizens’ evaluations. This should remind
both practitioners and researchers about the fragile nature of the
performance measures we apply and how they affect the greater
public. Small, arbitrary changes in the reporting of performance
information have the potential to significantly shift the perception
of public services among those receiving the information in both
negative and positive directions.
The second implication is a need to integrate our study of the
effects of performance information into the broader framing
research agenda (James 2011b; Moynihan 2008). There are end-
less ways to present performance information to citizens, and
shifting the valence of a metric’s descriptive label is just a very
basic example. Other examples include logically equivalent ways
to present performance information on different scales and with
the aid of various reference points (Olsen 2013a). This opens up
a broader research agenda on the informational cues that affect
citizens’ attitudinal and behavioral response to performance
information.
The final implications of an equivalence framing effect and the
underlying mechanism of a negativity bias point at downstream
effects for the broader political-administrative system. The impor-
tance of performance information valence adds another example
to the list of ways in which individuals’ perceptions of blame are
potentially malleable (Marvel 2014). Maybe the apparent wide-
spread use of positive-valence performance information is indicative
of policy makers’ fear of backlash. Some have linked the negativity
bias to the general blame avoidance observed at the political level
(Carpenter and Krause 2012; Mortensen 2013; Weaver 1986). As
Hood notes, a “key test of political power can be said to be the
ability to overcome or counteract negativity bias” (2007, 199). It
is therefore highly important that future studies aim to understand
how the informational valence of satisfaction measures can evoke an
Citizen (Dis)satisfaction: An Experimental Equivalence Framing Study 477
either negative or positive response among citizens and, from there
on, the rest of the political-administrative system.
Acknowledgments
The research was made possible by Grant No. 0602-02577B from the
Danish Council for Independent Research. An earlier version of the
paper was presented at the 11th Public Management Research Confer-
ence, Madison, Wisconsin, June 20–23, 2013. I would like to thank
the conference participants for valuable comments. The manuscript
has also benefited greatly from the feedback from three anonymous
reviewers and the editors of PAR. Any remaining errors are my own.
Notes
1. The median responses are 72 for the satisfaction frame and 45 for the dissatisfac-
tion frame. This gives a median difference of 27 points. If we apply trimmed
means, where the 10 percent most extreme observations in each tail are cut out,
the results are 67.8 for the satisfaction frame and 44.9 for the dissatisfaction
frame. This gives a mean difference of 22.9 points (rounded).
2. The interaction effect of work experience is 6.4 (p < .05), and the main effect of
work experience is −1.0 (p = .45).
3. Psychological research has found slower reaction times for processing nega-
tive information compared with positive information (Baumeister et al. 2001,
342). Some researchers have attributed this to the way that negative informa-
tion initiates more complex and deeper processing (Rozin and Royzman 2001).
Respondents provided with the dissatisfied frame were about 4.7 seconds slower
in their response time than respondents under the satisfied frame (dissatisfied
frame, M = 25.1 seconds; satisfied frame, M = 20.4 seconds). If median response
times are used, the difference is cut to about 1.4 seconds.
4. Median values: dissatisfaction to satisfaction, 53; satisfaction to satisfaction, 52;
satisfaction to dissatisfaction, 49; dissatisfaction to dissatisfaction, 42.5.
References
Ansolabehere, Stephen, Marc Meredith, and Erik Snowberg. 2013. Asking about
Numbers: Why and How. Political Analysis 21(1): 48–69.
Baumeister, Roy F., Ellen Bratslavsky, Catrin Finkenauer, and Kathleen D. Vohs.
2001. Bad Is Stronger than Good. Review of General Psychology 5(4): 323–70.
Behn, Robert D. 2003. Why Measure Performance? Different Purposes Require
Different Measures. Public Administration Review 63(5): 586–606.
Bloom, Howard S., and H. Douglas Price. 1975. Voter Response to Short-Run
Economic Conditions: The Asymmetric Effect of Prosperity and Recession.
American Political Science Review 69(4): 1240–54.
Bouckaert, Geert, and Steven Van de Walle. 2003. Comparing Measures of Citizen
Trust and User Satisfaction as Indicators of “Good Governance”: Difficulties in
Linking Trust and Satisfaction Indicators. International Review of Administrative
Sciences 69(3): 329–43.
Bouckaert, Geert, Steven Van de Walle, and Jarl K. Kampen. 2005. Potential for
Comparative Public Opinion Research in Public Administration. International
Review of Administrative Sciences 71(2): 229–40.
Boyne, George A., Oliver James, Peter John, and Nicolai Petrovsky. 2009.
Democracy and Government Performance: Holding Incumbents Accountable in
English Local Governments. Journal of Politics 71(4): 1273–84.
Carpenter, Daniel P., and George A. Krause. 2012. Reputation and Public
Administration. Public Administration Review 72(1): 26–32.
Chingos, Matthew M., Michael Henderson, and Martin R. West. 2012. Citizen
Perceptions of Government Service Quality: Evidence from Public Schools.
Quarterly Journal of Political Science 7(4): 411–45.
Druckman, James N. 2004. Political Preference Formation: Competition,
Deliberation, and the (Ir)relevance of Framing Effects. American Political Science
Review 98(4): 671–86.
———. 2011. What’s It All About? Framing in Political Science. Perspectives on
Framing. In Perspectives on Framing, edited by Gideon Keren, 297–302. New
York: Psychology Press.
Folz, David H. 1996. Survey Research for Public Administration. Thousand Oaks, CA:
Sage Publications.
Hansen, Kasper M., Asmus L. Olsen, and Mickael Bech. 2014. Cross-National Yardstick
Comparisons: Choice Experiment on a Forgotten Voter Heuristic. Political Behavior.
Published electronically on September 11. doi:10.1007/s11109-014-9288-y.
Hibbard, Judith H., Jean Stockard, and Martin Tusler. 2003. Does Publicizing
Hospital Performance Stimulate Quality Improvement Efforts? Health Affairs
22(2): 84–94.
———. 2005. Hospital Performance Reports: Impact on Quality, Market Share, and
Reputation. Health Affairs 24(4): 1150–60.
Hibbing, John R., Kevin B. Smith, John R. Alford. 2014. Differences in Negativity
Bias Underlie Variations in Political Ideology. Behavioral and Brain Science 37(3):
297–350.
Holzer, Marc, and Kaifeng Yang. 2004. Performance Measurement and
Improvement: An Assessment of the State of the Art. International Review of
Administrative Sciences 70(1): 15–31.
Hood, Christopher. 2007. What Happens When Transparency Meets Blame-
Avoidance? Public Management Review 9(2): 191–210.
James, Oliver. 2011a. Managing Citizens’ Expectations of Public Service
Performance: Evidence from Observation and Experimentation in Local
Government. Public Administration 89(4): 1419–35.
———. 2011b. Performance Measures and Democracy: Information Effects on
Citizens in Field and Laboratory Experiments. Journal of Public Administration
Research and Theory 21(3): 399–418.
James, Oliver, and Peter John. 2007. Public Management at the Ballot Box:
Performance Information and Electoral Support for Incumbent English Local
Governments. Journal of Public Administration Research and Theory 17(4): 567–80.
James, Oliver, and Alice Moseley. 2014. Does Performance Information about
Public Services Affect Citizens’ Perceptions, Satisfaction, and Voice Behaviour?
Field Experiments with Absolute and Relative Performance Information. Public
Administration 92(2): 493–511.
Johnsen, Åge. 2012. Why Does Poor Performance Get So Much Attention in Public
Policy? Financial Accountability and Management 28(2): 121–42.
Kinder, Donald R., and D. Roderick Kiewiet. 1979. Economic Discontent and
Political Behavior: The Role of Personal Grievances and Collective Economic
Judgments in Congressional Voting. American Journal of Political Science 23(3):
495–527.
Kravitz, Richard. 1998. Patient Satisfaction with Health Care: Critical Outcome or
Trivial Pursuit? Journal of General Internal Medicine 13(4): 280–82.
Lau, Richard R. 1982. Negativity in Political Perception. Political Behavior 4(4): 353–77.
———. 1985. Two Explanations for Negativity Effects in Political Behavior.
American Journal of Political Science 29(1): 119–38.
Levin, Irwin P., and Gary J. Gaeth. 1988. How Consumers Are Affected by the
Framing of Attribute Information Before and After Consuming the Product.
Journal of Consumer Research 15(3): 374–78.
Levin, Irwin P., Sandra L. Schneider, and Gary J. Gaeth. 1998. All Frames Are
Not Created Equal: A Typology and Critical Analysis of Framing Effects.
Organizational Behavior and Human Decision Processes 76(2): 149–88.
478 Public Administration Review • May | June 2015
Mannion, Russell, Huw Davies, and Martin Marshall. 2005. Impact of Star
Performance Ratings in English Acute Hospital Trusts. Journal of Health Services
Research and Policy 10(1): 18–24.
Marvel, John D. 2014. The Boston Marathon Bombings: Who’s to Blame and Why
It Matters for Public Administration. Public Administration Review. 75(6):
713–25.
Mortensen, Peter B. 2013. Public Sector Reform and Blame Avoidance Effects.
Journal of Public Policy 33(2): 229–53.
Moynihan, Donald P. 2008. The Dynamics of Performance Management: Constructing
Information and Reform. Washington, DC: Georgetown University Press.
Mueller, John E. 1973. War, Presidents, and Public Opinion. New York: Wiley.
Olsen, Asmus Leth. 2013a. Compared to What? Experimental Evidence on Social
and Historical Reference Points in Performance Evaluation. Paper presented
at the 11th Public Management Research Conference, Madison, WI, June
20–23.
———. 2013b. Leftmost-Digit-Bias in an Enumerated Public Sector? An
Experiment on Citizens’ Judgment of Performance Information. Judgment and
Decision Making 8(3): 365–71.
———. 2013c. The Politics of Digits: Evidence of Odd Taxation. Public Choice
154(1–2): 59–73.
Pope, Devin G. 2009. Reacting to Rankings: Evidence from America’s Best Hospitals.
Journal of Health Economics 28(6): 1154–65.
Quattrone, George A., and Amos Tversky. 1988. Contrasting Rational and
Psychological Analyses of Political Choice. American Political Science Review
82(3): 719–36.
Rabin, Matthew. 1998. Psychology and Economics. Journal of Economic Literature
36(1): 11–46.
Rozin, Paul, and Edward B. Royzman. 2001. Negativity Bias, Negativity Dominance,
and Contagion. Personality and Social Psychology Review 5(4): 296–320.
Soroka, Stuart N. 2006. Good News and Bad News: Asymmetric Responses to
Economic Information. Journal of Politics 68(2): 372–85.
Stipak, Brian. 1979. Citizen Satisfaction with Urban Services: Potential Misuse as a
Performance Indicator. Public Administration Review 39(1): 46–52.
———. 1980. Local Governments’ Use of Citizen Surveys. Public Administration
Review 40(5): 521–25.
Tversky, Amos, and Daniel Kahneman. 1981. The Framing of Decisions and the
Psychology of Choice. Science 211(4481): 453–58.
Van de Walle, Steven, and Greg G. Van Ryzin. 2011. The Order of Questions in a
Survey on Citizen Satisfaction with Public Services: Lessons from a Split-Ballot
Experiment. Public Administration 89(4): 1436–50.
Van Ryzin, Greg G. 2013. An Experimental Test of the Expectancy-Disconfirmation
Theory of Citizen Satisfaction. Journal of Policy Analysis and Management 32(3):
597–614.
Van Ryzin, Greg G., Douglas Muzzio, Stephen Immerwahr, Lisa Gulick, and
Eve Martinez. 2004. Drivers and Consequences of Citizen Satisfaction: An
Application of the American Customer Satisfaction Index Model to New York
City. Public Administration Review 64(3): 331–41.
Weaver, R. Kent. 1986. The Politics of Blame Avoidance. Journal of Public Policy
6(4): 371–98.
Williams, Brian. 1994. Patient Satisfaction: A Valid Concept? Social Science and
Medicine 38(4): 509–16.