Public management and multi-level

governance
Course 8 - Policy evaluation and actors’ mapping
Policy evaluation
 Policy evaluation: Moment when an overall (substantive and/or formal) assessment of the policy
is provided
 Key moments: in understanding what went wrong and why
 It is not necessarily something that only happens at the end, may happen in policy formulation
and policy implementation
 To better understand how policies work out, governments as well as other members of the
relevant policy subsystem conduct informal or formal assessments of policy outputs and
outcomes of varying intensity and sophistication.
 Value judgment—whether a policy is effective or not—is critical to evaluation
 “Evaluation refers to the process of determining the worth or significance of an activity, policy,
or program.”
 Key actors: bureaucracies (historically, they tend to evaluate themselves, which is biased and
problematic, because of this stakeholders often demand 3rd parties to conduct the evaluations) ,
experts, think tanks, judges, voters
 It is very different from a political evaluation (elections)
 But a political evaluation sometimes may be a policy evaluation
 It is also very important for the linear reading the policy process, this is when policy learning
occurs
 There are some policy games played overtime, between these games you may have evaluations
 There is a constant tension between politicians and the ones who evaluate the policies in the
formulation phase and it continues after the decision making as well
 The more detailed the policy is the better for analysts, the worse for politicians

Types of evaluations
 Technical evaluation:
o Analysis trying to understand the relationship between expected and obtained outcomes
(key actors- bureaucrats, experts, think tanks)
o The goal of politicians and bureaucrats is to have the least different outcome from
expectations
 Judiciary evaluation:
o Law-based assessment (key actors- judges)
o This is a great opportunity for those who lost in the implementation battle
 Political evaluation:
o Consensus-based assessment (key actors- voters)
o The “mother of all evaluations” informal
 o The least accurate evaluation, it’s very biased
o Sometimes it can be done by party members
o Political evaluation of policies is undertaken by just about everyone with any interest in
politics
o Their initial objective in undertaking an evaluation is rarely to improve a government’s
policy, but rather to either support or challenge it
o A more common type of political policy evaluation involves consulting with members of
relevant policy subsystems
o In many countries, political evaluation of government action is built into the system, in
the form, for example, of congressional or parliamentary oversight committees
o

Perspectives on Evaluation: Positivist and Post-Positivist

 Policy evaluation has encouraged its practitioners to view such assessment methods and
orientations as neutral, technical exercises in determining the success (or failure) of government
efforts to deal with policy problems
 “the objective systematic, empirical examination of the effects ongoing policies and public
programs have on their targets in terms of the goals they are meant to achieve.”
 Evaluation is at times employed deliberately to show a policy in a better or worse light than
justified by objective evidence, depending on the intention of those commissioning or conducting
the evaluation
 This is accomplished through framing the terms of evaluation in such a way as to lead to
conclusions that suit the sponsor’s motives
 We must be aware that relying solely on formal evaluation for drawing conclusions about a
policy’s relative success or failure will undoubtedly yield unduly limited insights into policy
outcomes and their assessment

Policy Evaluation as Policy Learning

 Combines elements of both the positivist and post-positivist perspectives
 We can look at policy evaluation as part of a policy learning
 Policy evaluation stimulates educational dynamics and learning among policy-makers
 Policy learning has a broader meaning that includes better understanding both the intended and
unintended consequences of policy-making activities, as well as both the “positive” and
“negative” implications of existing policies and their alternatives on the status quo and efforts to
alter it
 It is also important to note that the number of actors—both governmental and non-governmental
—involved in policy evaluation expands toward the size of the policy universe existing during
agenda-setting
 Some lessons are likely to concern practical suggestions about specific aspects of the policy
cycle, based on the actual experience with the policy on the part of policy implementers and
target groups

Types of Policy Evaluation

 Inputs – resources (funding, staffing, equipment, etc.) that go into implementing a policy, while
activities is what is done with these inputs (provide, facilitate, etc.)
  Outputs – the observable products and services produced by the inputs and activities
 Outcomes – the changes that result from the outputs
 Impacts – the intended and unintended medium- and long-term effects of the outcomes
 Process evaluation (formative)
o Concentrate on the inputs and activities devoted to producing outputs
o The intent is not only to generate information on the linkages between inputs and
activities in regards to outputs and outcomes, but also to draw conclusions about how to
improve them
o Beyond assessing the performance of a policy intervention, process evaluations explore
the reasons underlying it, paying special attention to contextual and organizational factors
o Process evaluations are often conducted within the rubric of the logic framework, also
called theory-based evaluation, these focus on “unpacking the theoretical or logical
sequence by which a policy intervention is expected to bring about its desired effects . . . .
[They] attempt to identify the mechanisms by which policies and/or programmes might
produce their effects”
 Impact evaluation (summative)
o Concentrates on the effects of the outputs
o Its purpose is to comprehensively understand the effects caused by the program, it also
seeks to pass judgement on the program’s performance
o A more feasible and commonly used approach to impact evaluation is to estimate only
the direct costs of inputs assessed against observable first-order effects
o It is arguable that it is insufficient to merely assess all effects resulting from a policy and
it is necessary to specify the causality—that is, what effect was caused by which specific
program component—for evaluation to be rigorous and definitive
o The fundamental problem that all evaluations face is how to attribute the effects or
outputs of a policy intervention to the inputs that were launched following a positive
decision to initiate policy. Even when a policy achieves its goals, it cannot be concluded
that the achievement was the effect of the policy in question because other factors—
including other policies and broader contextual conditions— may have played an
influential role

Mapping policy actors

 The range of actors involved in policy evaluation is much broader than often presented in the
administrative and managerial literature on evaluation, which tends to concentrate
overwhelmingly on internal evaluation by agencies them selves as well as those done at their
behest by external consultants or by think tanks.
 The range of actors also includes the judiciary, interest groups and in deed, as discussed above,
the public at large
 Policy actor type
 Policy actors’ resources
 Policy actors’ role
 Policy actors’ links:
o Policy network (interest)
o Policy community (knowledge)
o Policy coalitions (beliefs)
Internal Evaluators
 Internal evaluators are usually paid and trained professionals who routinely apply formal
techniques such as cost–benefit or budgetary analysis
 They can affect the “framing” and assessment of policy success and failure by how they develop
and apply various techniques, measures, and benchmarks to the program
 At other times, they may serve as “brokers” linking policy-makers to implementers, or to those
outside the formal institutions of government who are generating new knowledge on social
problems and the techniques for resolving these problems
 They have better access to information and officials who may supply it, but also because of this
closeness they may be too attached to the program to ask hard questions and conduct independent
evaluation

External Evaluators
 External evaluators include a variety of actors directly or indirectly involved in assessing and
otherwise passing formal or informal judgment on a policy’s performance and its impacts
 They include concerned interest groups that conduct their own, less formal, reviews of policy
performance. They also include paid consultants who have been playing an increasingly
important role in evaluation
 Think tanks, on the other hand, once played an important role in independently evaluating
policies in many countries, but their proliferation and the trend toward their identification with
specific partisan positions have undermined their ability to affect policy discourses and directions
through their evaluative activities, despite the exponential increase in the number of the latter
 Judges are able to review legislative and administrative actions to determine the extent to which
policies match up to larger, often constitutionally established principles of social justice and
conduct
 The judiciary is entitled to review government actions either on its own initiative or when asked
to do so by an individual or organization filing a case against a government agency in a court of
law
 In parliamentary systems judicial courts do not review the facts specific to the case, but tend to
restrict their evaluation to procedural issues
 Courts in republican systems with constitutionally entrenched divisions of powers, as in the US,
courts enjoy more authority to question legislative and executive actions
 Members of the public can be said to have the ultimate say on a government’s policy record when
they vote at election or comment to the media or pollsters about it

Policy conflict
 Its not only linked to advocacy coalitions
 Relatively stable parameters
 External events
 Long term coalition opportunity structures
 

Policy actors’ roles

 Policy initiators: the motor of a policy process, sometimes they can be policy brokers, they have
to have resources (cognitive, economic, political), it can be a “synonym” to policy entrepreneur
 Policy supporter: actors(s) providing support to the policy initiator- under the form of policy
network or policy community
 Policy broker: particular role in setting links among actors that otherwise would not cooperate –
may ”slice up” the policy in order to broaden support, redefining the goals
 Policy opponent: actor(s) contrasting the policy initiator/supporter; ambivalent relationship with
a possible broker, they may not oppose, but also initiate something else, so they may be in favor
of other settings
 All of them may be lobbyists (institutional and non-institutional)

Policy actors and policy phases

 Policy actors may intervene in all policy phases or only in a limited number of them
 You have to match preferences and actors
 Mapping policy actors means also mapping them in each policy phase, in order to unveil their
preferences and select the most appropriate strategies for
o Organizing supporting coalitions
o Predict “enemies’” moves, but you first have to define the “enemy”
o Reach out for external support (or legitimation)
 You can mobilize during agenda setting or analysis
Policy Success and Failure
 Policies can succeed or fail in numerous ways
 Sometimes an entire policy regime can fail, while more often specific programs within a policy
field may be deemed as successful or unsuccessful
 Policies can fail in substantive terms (not delivering the goods) and also in procedural terms
(being illegitimate or unfair)
 Failures can occur at any stage of the policy process
 Failure can also arise from a mismatch between goals and policy tools in the formulation stage, or
it can result from the consequences of lapses or misjudgments at the decision-making stage
 Implementation failures – the aims of decision-makers fail to be properly or accurately translated
into practice, lack of effective oversight by decision-makers over those who implement policy, or
governments and policy-makers not effectively evaluating policy processes and learning useful
lessons from past experiences

Assessing the Results of Policy Evaluation

 Efficiency
o Allocating resources to achieve policy goals in an economical manner
o It measures the relationship between input and output with the intention of maximizing
outputs for a given input or minimizing costs for a given output
o Efficiency in the policy context includes not only the material costs but also the
transaction and other costs associated with rules and operating procedures
o It relies heavily on cost-effectiveness analysis
 Effectiveness
o The extent to which policy inputs and activities achieve the desired outcomes
o A policy that is ineffective in achieving its primary goals is a useless policy, regardless of
its performance with regard to efficiency equity, and other criteria
o A policy intervention is considered effective if it has made a demonstrable contribution to
the achievement of the set objectives

Conclusion
 Mapping actors mapping preferences…
 …in each phase of the policy process
 Once you found and analyzed a coalition you have to make sure it’s the right one
 Coalitions though do not change easily from one policy phase to the other…
 …but expansionary strategies are possible
 European multilevel settings makes coalition building particularly important (ex umbrella
associations)
 Policy recommendations are “calls for action”

Study Questions
1. What are the potentials and limitations of different evaluation techniques?
2. What are the respective advantages and disadvantages of evaluation by internal and external
experts?
3. What capacity is required to carry out effective evaluation? How can this be developed?
 4. To what extent is it possible to engage the public in policy evaluation?
5. What is policy learning, and how would you promote it?

Article
David Glover

What Makes a Good “Policy Paper”? Ten Examples

 Some shortcomings in writing a research report

o Drawing implications that are not based on the data
o Misinterpreting data to draw unwarranted conclusions
o Providing generic recommendations that could pertain to almost any problem
o Burying worthwhile conclusions here and there within the report, rather than distilling
them in a concluding section
o Drawing vague conclusions or not drawing policy implications at all

Sidebar: Policy recommendations vs. policy implications

 A policy recommendation is a statement that makes a specific proposal for action.

 Policy implications also interpret data in ways that are useful to policymakers, but without
specifying precisely what should be done.
 There is a gray area between the two, when a policy implication contains a very vague
recommendation
Ten Examples
1. Hans Binswanger, “Brazilian Policies that Encourage Deforestation in the Amazon”. World
Development, V. 19, # 7, 821-829, 1991 (8 p.)
 Before this, many people had claimed or assumed that deforestation in Brazil was caused by
logging for the timber trade or that people were clearing land to raise cattle and exporting the beef
to North America
 Market failure? – Policy failures?
 Macroeconomic conditions and policies had indirect and unintended effects on land use\
 Brazil had a long history of hyperinflation
 One of the few safe places to put your money was land, which was sure to increase in value
 Income from agriculture and livestock was exempt from income tax
 Binswanger reviews half a dozen policies that unintentionally encourage forest clearing
 It generated a lot of interest in looking at economy-wide policies in other countries
 A couple of years later, many of these subsidies and incentives for land clearing had been
 removed.
 The strongest feature of the article is its clear and concise analysis of the impact of government
policies -> it’s very easy for him to draw policy recommendations
 The weak point of the paper is that, because it it’s so brief (eight pages) there’s no description of
his research methodology
2. Mike Rock and Jean Aden, “Initiating Environmental Behavior in Manufacturing Plants in
Indonesia”. Journal of Environmental and Development, V. 8 # 4, 357-375, Dec. 1999 (18 p.)
 The purpose of this study was to find out what factors lead Indonesian firms to undertake
pollution abatement and other environmentally-friendly behavior.
 The study identified plant-level environmental behaviors and measured the extent of exposure of
plants to regulatory, community, and market pressures and government incentives designed to get
plants to install pollution control equipment.
 One is that firms carry out some preliminary/preparatory measures for pollution control, in
response to regulatory, community and market pressures
 But hey will not spend substantial amounts of money to reduce pollution unless they have to.
 The paper provides a clear and complete, step-by-step report on the research process, from initial
hypothesis to conclusions - all in 18 pages.
3. Randall Bluffstone, “Reducing Degradation of Forests in Poor Countries When Permanent
Solutions Elude us: What Instruments Do We Really Have?”. Environment and Development
Economics, 295-317, 1998. (22 p.)
 This paper acknowledges that the basic/long-term causes of deforestation in developing countries
are difficult to address in the short run, then describes some of those causes and the difficulties
associated with them.
 It clearly establishes its policy relevance early on, it promises something practical.
 Because it assesses something hypothetical, it has to use models. But it uses real data, not just
algebra, so it’s more persuasive.
 The Results section uses a few key numbers and presents them well.
 It shows a clear difference in cost-effectiveness between two policies and explains why (labor re-
allocation vs. technical progress).
 It explains why a subsidy is costly and inefficient
 The main weakness of the paper is that it does not explain why such an obviously efficient
strategy has not been adopted.
4. Paul Ferraro and David Simpson, “The Cost-effectiveness of Conservation Payments”. Land
Economics V. 78 # 3, 2002 (18 p.)
 This paper introduces the hypothesis that it is more cost-effective to make direct payments to
individuals or groups that protect ecosystems than it is to sponsor development projects that
produce commercial outputs and ecosystem protection as joint products.
 It finishes with an empirical example – a beekeeping project – that plugs real data into the mode
 When the model is run using real costs of labour, capital, honey and so forth, it is found that the
subsidy approach costs about 12 times more than the direct payment approach.
 The paper then discusses some of the implementation problems of both approaches and finds that
they do not differ very much.
 Finally, it asks why donors favor the direct approach, even though it costs so much more. The
reasons seem to be things like the belief that it promotes development; that projects are more
visible than payments; that they provide demonstration effects; and so on.
 The strongest point of this paper is that its hypothesis and conclusions go against current
conventional wisdom. So it is likely to attract attention – it is more likely to be published in
journals and attract readers than a paper that argues the same thing as a lot of other papers
 Another strong point is the final section, which analyzes the motives of donors for favoring the
more costly approach and provides counterarguments to it.
 The weakest point in the paper is that, although the model is not complex, neither is it easy for a
non-economist to understand.
5. W. Magat & Kip Viscusi, “Effectiveness of the EPA’s Regulatory Enforcement: The Case of
Industrial and Effluent Standards”. Journal of Law & Economics, V. XXXIII, 331-359
October, 1990 (30 p.)
 The purpose of this study was to find out how effective the US Environmental Protection
Agency’s regulations about water pollution have been
 The author decided compare the effectiveness of the EPA’s regulations to that of health and
safety regulations.
 The author’s findings were:
o Discharges of pollution are inversely related to inspection rates, each inspection results in
roughly a 20% percent decrease in discharges.
o There is no “rebound” effect - discharges don’t increase again after the inspection
 Next they do an “exploratory” cost-benefit analysis using benefit transfer and find that benefits
greatly exceeded costs
 It provides a clear and complete description of the purpose, methods and conclusions
 It could serve as a good model for authors.
 It also makes very good use of literature review
 This paper is an example of (ex post) policy evaluation – assessing a policy after it has been
operating for some time to see how effective it has been, and comparing it with other policies that
operate differently.
6. S. Dasgupta, et al, “Water Pollution Abatement by Chinese Industry: Cost Estimates and
Policy Implications”. World Bank Working Paper, May 1996 (11 p. plus tables)
 This paper begins with a very short problem statement (3 sentences), which mentions that a new
plant-level database provides information about the cost of abating industrial water pollution in
China.
 The paper makes use of this database to estimate abatement cost functions; assess the economic
efficiency of current regulations; and simulate the impact of an emissions charge system.
 Cost variations between firms are large (up to 30:1), so potential cost savings from market-based
instruments are large.
 Tightening standards would create significant costs. A benefit estimate would be needed to make
the decision, but a priori it seems that switching to the highest standard is not justified
 Emission charges as low as $1 per ton would induce significant abatement.
 Switching from the current levy system (which has a maximum allowable emission level plus a
fine for excess emissions) to a conventional emissions charge would reduce abatement costs by
73%.
 The main strengths of the paper are its brevity and clarity
 The authors use considerable ingenuity to extract very sharp policy conclusions from a large
amount of data. Each of the four bullet-point conclusions includes one or two numbers, expressed
in terms that are easy to understand.
 If the paper has a weakness, it may be in the problem statement. Because it is so brief.
7. Mateen Thobani, “Water Markets: Why, When, and How to Introduce Tradable Water
Rights”. World Bank Research Observer, V. 12 # 2, 161-179, August 1997 (18 p.)
 Thobani first describes the problem of water scarcity and discusses the political feasibility
problems of introducing market-based instruments to deal with it.
 The main strength of the paper is that it is practical, non-technical and easy to understand.
 It emphasizes political and administrative obstacles to policy reform, and how to overcome them
in order to achieve the advantages identified by economic theory.
8. K. Choe, D. Whittington, and D. Lauria, “The Economic Benefits of Surface Water Quality
Improvements in Developing Countries: A Case Study of Davao, the Philippines”. Land
Economics V. 72 # 4, 519-37, Nov. 1996 (18 p.)
 This study used the contingent valuation method and travel cost model to estimate the economic
value people place on improving the water quality of the rivers and sea near their community.
The two valuation methods provided very similar results: both found that the willingness to pay
for improved water quality was very low, both in absolute terms and as a percentage of household
income.
 The use of two valuation methods that provided very similar estimates increases the credibility of
the study.
 The contingent valuation method, properly applied, involves close contact between researchers
and the community and provide some insights into attitudes and motivations
 The study’s conclusion – that willingness to pay is too low to provide a basis for project financing
– is a valuable one
9. Ken Chomitz, “Evaluating Carbon Offsets from Forestry and Energy Projects: How Do They
Compare?” World Bank Discussion Paper, n.d (2000?),Kchomitz@worldbank.org. (24 p.)
 This paper discusses the merits of using forestry or energy projects in developing countries as
ways to offset increases in carbon emissions. Both kinds of projects have been discussed in
negotiations about carbon trading as a way to reduce the costs of avoiding climate change.
 This paper is likely to attract attention because it disputes conventional wisdom.
 It does not provide specific policy recommendations but it does have clear policy implications.
 The implication is that forest projects do offer possibilities for carbon trading and these
possibilities should not be neglected.
10. EEPSEA/WWF, Indonesia’s Fires and Haze: The Cost of Catastrophe. ISEAS/IDRC, 1999
(book:145 p.; summary: 6 p.; press release: 1 p.)
 This study estimated the total value to humans and ecosystems from the 1997 fires and haze.
 One of the strengths of the study is the disaggregation of impacts. It showed that the largest
damages by far from the haze were to people’s health, not to business as previous media reports
had implied.
 The main damage estimates were summarized in two simple tables. Numbers were rounded off
where possible. The main figures were accompanied by readily understandable comparisons
 Because the study did not examine the causes of the forest fires, the author could not cite original
research as a basis for policy recommendations.
 However, anticipating that the study would draw many questions about what should be done, the
authors surveyed existing literature about the causes of the fires and recommendations that had
been made about better forest management.
 A summary of this literature, with appropriate citations, was included in the book. The authors
could therefore legitimately respond to questions about what should be done – and base their
response on solid research.