CHAPTER-FIVE

EVALUATION:SOME BASICS OF IMPACT

EVALUATION METHODOLOGIES

1
5.1. Introduction
 Impact assessment is the process of identifying the anticipated or actual
impacts of an intervention, on those social, economic and
environmental factors in which the intervention is designed to affect.

 Impact evaluation provides a framework sufficient to understand

whether the beneficiaries are truly benefiting from the program and
not from other factors.

 The main question of impact evaluation is one of attribution isolating

the effect of the program from other factors and potential selection
bias.

2
5.2. Qualitative vs Quantitative Impact Assessment
Methodologies
 Impact evaluation could be qualitative and quantitative methods.

Qualitative method of analysis

 It seeks to measure potential impacts that the program may generate,
the mechanisms of such impacts, and the extent of benefits to
recipients from in-depth and group-based interviews.

 It is not generalizable and generate information that may be critical for

understanding through which the program helps beneficiaries.

 These methods generally require high levels of skill, and may be

relatively costly.

3
Cont…
 Some degree of qualitative interpretation may be necessary in all
impact
assessments, in order to evaluate the causes of impacts which have
been observed.

 It can't assess outcomes against relevant alternatives or counterfactual

outcomes.

 That is, it doesn’t really indicate what might happen in the absence of
the program.

Quantitative method of analysis

 It involving baseline studies, the precise identification of baseline
conditions, definition of objectives, target setting, rigorous
performance evaluation and outcome measurement.
4
Cont…
 This methods is costly, limited in the types of impacts which can be
accurately measured, and may pose difficulties for inference of cause
and effect.

 Basically, there are two types of quantitative impact evaluation

methodologies: thus are, ex post and ex ante

Ex ante impact evaluation:

 It attempts to measure the intended impacts of future programs and
policies, given a potentially targeted area’s current situation.
Ex post evaluations:
 In contrast, measure actual impacts obtained by the beneficiaries that
are attributable to program intervention.

 Ex post evaluations have immediate benefits and ref lect reality.

5
Cont…
 It can also be much more costly than ex ante evaluations because they
require collecting data on actual outcomes for participant and
nonparticipant groups.

6
5.3. Methodologies in impact evaluation
 A number of different methods can be used in impact evaluation theory
to address the fundamental question of the missing counterfactual.

 Counterfactual refers to the state of beneficiaries without project

intervention.

 Each of these methods carries its own assumptions about the nature of
potential selection bias in program targeting and participation, and the
assumptions are crucial in developing the appropriate model to
determine program impacts.

7
Cont…
 The most popular methods of impact evaluation are:

 Randomized evaluations

 Matching methods, specifically propensity score matching (PSM)

 Double-difference (DD) methods

 Instrumental variable (IV) methods

 These methods vary by their underlying assumptions regarding how to

resolve selection bias in estimating the program treatment effect.

8
Cont…
Randomized evaluations
 Randomized evaluations involve a randomly allocated initiative across a
sample of subjects (communities or individuals, for example); the
progress of treatment and control subjects exhibiting similar
preprogram characteristics is then tracked over time.

 Randomization could be conducted purely randomly (where treated

and control units have the same expected outcome in absence of the
program)

 This method requires ensuring external and internal validity of the

targeting design.

9
Cont…
 Careful selection of control areas (or the counterfactual) is also
important in ensuring comparability with participant areas and
ultimately calculating the treatment effect (or difference in outcomes)
between the two groups

Propensity score matching (PSM)

 PSM constructs a statistical comparison group that is based on a model
of the probability of participating in the project, using observed
characteristics.

 Participants are then matched on the basis of this probability, or

propensity score, to nonparticipants.

10
Cont…
 PSM methods therefore assume that selection bias is based only on
observed characteristics; they cannot account for unobserved factors
affecting participation.

Double-difference (DD) methods

 DD methods assume that unobserved selection is present and that it is
time invariant;

 The treatment effect is determined by taking the difference in

outcomes across treatment and control units before and after the
program intervention.

 It can be used in both experimental and non-experimental settings.

11
Cont…

IV methods
 IV models can be used with cross-section or panel data and in the latter
case allow for selection bias on unobserved characteristics to vary with
time.

 we may be interested in the effect of time-invariant variables (like

gender, region……./ but FE can’t

 In the IV approach, selection bias on unobserved characteristics is

corrected by finding a variable (instrument) that is correlated with
participation but not correlated with unobserved characteristics
affecting the outcome;

 This instrument is used to predict participation.

12