Economic Assessment of Public Policies

B.A. Political and Administration Science

Lecturer: Álvaro Choi
The objectives of the EAPP vary substantially depending on the type of evaluation.

Problem  Resources  Activity  Product  Impact

a b c d e

I.I. Needs assessment: analyses whether the problem justifies the existence of a
program. It also studies the fit of the program to the type of problem to be tackled. (a, b)

I.II. Evaluation of policy design: analysis of the different hypothesis on which the theory
of
the intervention based. In other words: is it rational to intervene in a certain way?
(a,b,c,d,e)

I.III. Process evaluation: studies if the program is being applied as originally

planned. It allows to correct issues in the implementation of the program. (b, c, d)

I.IV. Impact evaluation: it analyses the effectiveness of the program to tackle a specific
problem. (c, d)

2
I.V. Efficiency evaluation: analysis of the effectiveness of a program, putting in touch
outcomes and expenditure. (b, d, e)

3
I.I. Needs assessment

� Improve the targeting of available services

Detection of deviations in the use of the service that reduce the impact of the program.

� Improve the territorial allocation of resources

Identifying inequalities in the territorial distribution of resources.

� Identifying the most effective services and encouraging their use

Identifying good practices.

� Anticipating service needs

Analysis of future needs and the services which would be needed to cover them.

� Measuring the size of the problem and estimating the volume of resources needed
for a new policy

� Quantifying (roughly) the success of the policy.

Comparing the evolution of needs across time.

4
Steps for the evaluation of needs

1. Defining the social issue, its size, the environment and causes.
2. Analysis of the need for social benefits: analysis of previous studies; expert groups;
comparison with other jurisdictions.
3. Analysis of needs, demand and use of social benefits.

1. Supply of unnecessary benefits withoutD

2. Supply of necessary benefits without D.
(non take-up)
3. Entitled individuals who claim and
receive a social benefit.
4. D of social benefits by people who are
not entitled to claim them… but get them.
5. D for unnecessary social benefits which
are not provided
6. Entitled individuals who claim for social
benefits but do not get them (queue)
7. People with needs who do not claim
for services (which are not available).

5
I.II. Evaluation of policy design

Problem  Resources  Activity  Product  Impact

a b c d e
The evaluation of policy design is, above all, an evaluation the different
hypotheses on which the theory of the intervention is based.

Steps for evaluating the design of a policy

1. Assessing the design of the program taking into account the needs
This means that, somehow, a previous evaluation of needs should have been made…

2. Assessing the rationale and plausibility of the design of the program

Are there rational reasons for believing the activities will lead to the desired product and
that the product will have the desired impact.

3. Comparing the program to similar ones.

4. Following the program during the initial stages of its implementation

6
I.III. Process evaluation: is the actual program similar to that initially
designed? Is it achieving the expected results?

Thus, it analyses the outputs which are generated as the program is

implemented. It studies the mismatches between the program’s design and the
final program.

The assessment namely answers to the following two questions:

1. Is the target population receiving the benefits?

Some groups may make a more (less) intense use of the services; self-selection; over-
coverage (ex: flu-CAP-emergency services); physical barriers; subjective perception on
access to social benefits (ex: public health); social stigmas; etc.

2. Is the internal organization of the program adequate? Is it operating

properly?
Analysis of the financial and human resources; production and provision processes of the
services; process; organizational structure.

7
I.IV. Impact evaluation: analysis of the effectiveness of the program. Does it
tackle the problem? It includes incidence analysis and assessing externalities. It
is usually an ex post type of assessment.

The main objective of impact evaluation is establishing causal relationships.

Consequently, it is not only interested in comparing the situation before and
after the implementation of a specific policy (sheer associations…), but
comparing the situation after the implementation of the policy with the
situation that would have existed had the policy not been implemented . This
hypothetical scenario is the counterfactual.

As you may imagine, constructing the counterfactual is a difficult task.

Impact = Y1 – Y0

The main challenge for impact evaluation techniques (experimental and quasi-
experimental) is creating a credible counterfactual.

8
Impact evaluation allows to identify not only if the program works, but also for
who it works.

If we aim at evaluating the impact of a policy, the latter should have the
following characteristics:

� It should be stable.
� It should have a coherent theory of change.
� The implementation process should be known in detail.
� Enough time must have passed since the implementation of the program so
that the impact may be visible.
� Micro-data should be available.

Lessons 5 and 6 present different experimental and quasi-experimental

techniques.

9
I.V. Efficiency evaluation: analysis of the effectiveness of a program,
considering the costs

It provides information for guiding policymakers on what policies are more

efficient (i.e. how to spend money better). It may be ex ante or ex post.

Common elements to all economic evaluations:

� The research question

� An alternative policy (for comparison)
� Perspective of the evaluation
� Identification and measure of costs
Direct; externalities; opportunity cost; welfare; etc. Always in monetary units.
� Identification and measure of results
In monetary units; in physical units; in utility/welfare scales.

Different types of efficiency evaluation will be presented in lesson 4.

10
Different as the various types of evaluation might be, they share a similar
planning process.

In other words, the questions that evaluators must ask themselves before
starting the evaluation –for planning and designing the evaluation- are similar.

These questions are presented in section 2.2.

Students interested in going into detail with any of the types of evaluation presented throughout this
lesson may find very useful the guides elaborated by Ivalua, which are publicly available at:
http://www.ivalua.cat/

11
Planning an EAPP requires describing thoroughly the tasks to be performed at
each stage of the evaluation. This implies the following steps:

1. Preliminary analysis of the program

2. Describing the institutional setting

3. Understanding the purpose of the evaluation

4. Identifying the intended audience of the evaluation

5. Inventory of resources for the evaluation

6. Methodology (section 2.3)

12
1. Preliminary analysis of the program

1.A. Describing the theory of change

� Analysis of the different activities in which the program consists.
� Identifying the theoretical impact the program aims to have.
� Analysis of the activity product impact sequency

1.B. Identifying the stage of the program

Planning; initial stages of implementation; mature policy; concluded.

1.C. Description of changes in the program

1.D. Identifying previous evaluations and analyses

13
2. Describing the institutional setting

Identifying the individuals and institutions which have participated in the

program. What role did they play?

3. Understanding the purpose of the evaluation

Why did you receive the evaluation request? The answer will determine the
approach. You will have to work and write thinking about who will read the
assessment.

� Policymaking aims: strategic; corrective; selecting alternative policies;

modifying a policy design; etc.
� Understanding how the organisation works.
� Using the evaluation as a management tool.
� Accountability.
� Understanding the internal operations of a program
� Fulfillment of formal requirements.

14
4. Identifying the intended audience of the evaluation

The evaluation must be adapted to the interests of the recipients of their

results.

Possible recipients:

� High level decision makers

� Program managers
� Staff applying the program
� Financing organizations
� Direct beneficiaries of the program
� Social scientists
� Social representation groups
� Citizens

The recipient of the evaluation is usually linked to the reason why the
evaluation was requested  the type of information to be provided will differ.

15
5. Inventory of resources for the evaluation

The available resources limit the possibilities of evaluation.

� Financial
� Human: quantity & quality
� Cooperation of different individuals and institutions for performing the
evaluation
� Information and databases
� Time for completing the evaluation

Having completed the former five steps, the evaluator should ask the key
question…

16
… ¿is the evaluation viable? The identification of strong limitations in any of the
previous points threatens the possibility of performing an EPP.

Consequently, an evaluation is unfeasible if (factors that make the evaluation

impossible):

1. If the intervention cannot be clearly identified.

2. If the objectives of the program are unclear (thus, the expected impacts will
be vague)

3. If the theory of change (the hypotheses on which the intervention is based)

is uncoherent or unrealistic.

4. If the available resources (human, financial, material, data) are inadequate

for answering the main research questions.

5. If the time elapsed between the implementation of the policy and the
evaluation is too short in order to identify the impact of the policy.

17
Therefore, when designing an evaluation, (at least) the following essential
questions will have to be answered:

1. Preliminary analysis of the intervention

� Preliminary analysis of the intervention What is the purpose of the program?

Is it reasonable? The reform of Diagonal and transform it into a large space for walking and leisure. It is reasonable
because of the number of cars existing in that zone and the limited space available for walking.

� What are the causes of the issue (raison d’être)? What are the circumstances/
environment surrounding the problem? The high quantity of cars in Diagonal were a pollution problem.
The space was limited. There’s a street with 10 lanes for cars, no bicycle lane and bicycles and walkers share the same space
making the leisure less comfortable for everybody. The lack space for people means that there are not enough activities.
In short, the issues were the accidents, the lack space, pollution and traffic jams.

� What activities does the program include? Transform the Diagonal Avenue into a large space for
walking and leisure, reduce the number of lanes of cars (from 10 to 8), creation of bike lanes in both directions, construction of
motorcycle and loading zones, construction of electric car charging stations, the installation of wireless networks for internet
access and a fibre optic network, as well as LED lighting. 

18
� What is the expected impact of the program? What is the product(s) of the
program? To reduce the pollution and to make a safer space for walkers. A 24,078 square meters of sidewalk,
motorcycle parking and loading zones, electric car charging stations, wireless networks, and a reduction of the lanes for cars
from 10 to 8. 

� Is there a clear relationship between the activity, product, and impact? In

other words: are the hypotheses behind the intervention rational? Yes. the impact is to
have more space and to reduce pollution, with this amplification of the meters there is this space improvement and with the kind
of pavement and the motivation of using electric cars there is the reduced pollution.

� At what stage of implementation is the program? Is finished. Once the program has been applied.

� Has the program undergone changes or reforms since it was implemented?

Has it remained unchanged? It has remained unchanged.

� Are there any previous evaluations of the program (or similar ones)? We don’t have
information about that, but we assumed that probably not.

19
2. Describing the institutional setting
� Who designed and implemented the program? Who is who in the program?
The city council.

3. Identifying the purpose of the evaluation

� Why does the public body request the evaluation? What use will it make of the
evaluation?
Because the city council is considering extending the reforms from Plaça Juan Carlos I to the Plaça de les Glòries.

4. Identifying the audience of the evaluation

� Who is the intended audience of the evaluation?
Policy makers. Probably an internal evaluation.

5. Inventory of resources for the evaluation

� What financial and human resources are available? We can’t answer this question with the information of the case.
� Will the different stakeholders cooperate for the evaluation? Probably yes.
� Are the required information and databases available? In this case we don’t know, it also depends on the program.
� When should I deliver the results of the evaluation? Deadlines. 5-6 months

The previous questions allow us to, finally, answer the following question:
� Are insurmountable problems identified in any of the previous answers? In other words: Is the
evaluation viable? It is viable
The previous questions are at the core of all evaluations. Answering them also help us to select the
appropriate methodological strategy (2.3)

20
Selecting the appropriate methodology depends on a set of elements:

1. Aim of the evaluation: what questions should it answer to and at what stage
is currently the program?

2. Availability of resources: human and physical

3. Availability of data

In general, it is often difficult to carry out methodologically complex ex post

evaluations if their implementation was not considered ex ante, that is, before
the policy was applied  importance of understanding evaluation as an intrinsic
element to programs, and not just as an "add-on".

Example: analysis of the school meals and transportation grants in the Catalan
shires (comarques).

21
Evaluation methodologies

- Direct observation - Participant observation - Delphi method

Qualitative - Focus groups - Documentary analysis - SWAT analysis
- Qualitative interviews - Policy analysis

- Specific indicators
- Surveys
Methodology Descriptive - Structured interviews
- Archive data on users

Quantitative Macroeconomic - Cost-effectiveness analysis

- Cost-benefit analysis
Analytical
- Data envelopment analysis (DEA)
Non-parametric - Experiments
- Quasi-experiments: matching; diff in diff;
regression discontinuities
Microeconomic
- Instrumental variables
Parametric - Panel models
- Microsimulation models

22
A. Qualitative methodologies( lesson 3)
Observation; focus groups; qualitative interviews; participant observation;
policy analysis; etc.

B. Quantitative methodologies

B.1. Descriptive (lesson 3): specific indicators; surveys; structured interviews;

archive data on users; etc. Bivariate analysis.

B.2. Analytical (non descriptive)

� B.2.1. Macroeconomic methods (lesson 4)
� B.2.2. Microeconomic methods (lessons 5 and 6)
B.2.2.1. Non-parametric methods (lesson 5)
B.2.2.2. Parametric methods (lesson 6)

23
You have received an evaluation request by a public administration. For the
moment, this is the only information you have and next week you have a
meeting with the person who sent you the offer. Prepare the meeting.

In order to do so:

a) Plan the evaluation -without specifying the methodology (but describing the
chosen type of methodology, according to the different classifications)-
answering the fundamental questions exposed throughout the lesson.

b) Is the program evaluable? What does it depend on? What points do you think
may make evaluation more difficult in this particular case?

24