4/10/2021

3.1 Introduction
Impact evaluation is the way of checking whether an
intervention is responsible for the change in the outcome variable
or not.
The objective of impact evaluation is quantifying the impacts
of programs and projects on the beneficiaries such as
3.1 Introduction individuals, households and the community (Khandker, Koolwal, &
3.2 Propensity Score Matching Model Samad, 2010).
3.3 Difference in Difference Model Impact evaluation is aimed to examine whether the implemented
3.4 Switch Regression Model program resulted in the needed impact on institutions,
households and individuals and whether the impacts are attributed
to the intervention (Baker, 2000).
Impact evaluation helps policy makers to make informed
decision about the effectiveness of programs (Gertler et al., 2011).

3.1 Introduction 3.1 Introduction

Impact Evaluation questions Impact Evaluation questions

With only observational data, how to estimate the true impact
of the intervention?
 What is the “treatment effect on the treated”?
The impact of the intervention on those who actually
participated in the program (i.e., received the treatment).
What is the counterfactual?
What would have been the outcome for the participants had
they not participated?
How to find a proper counterfactual (i.e., participant as “non-
participant”)?
A proper counterfactual can be found by matching a
participant to a non-participant with similar pre-intervention
characteristics (X)
For the observation units with matched characteristics, each
has an equal chance of being a participant or an a
nonparticipant.

1

3.1 Introduction
Even though various techniques of impact evaluation methods
are found, choosing a specific evaluation method needs careful
investigation.
According to Baker (2000), given various choices, selecting an
appropriate technique is not an easy task since the results may be
varied based on the methods used.
The choice of appropriate evaluation method rely on;
 the type of information available to the researcher,
 the underlying model and
 the parameter of interest.
Propensity score matching (PSM), difference in differences
(DD) and Switch Regression Models are the most commonly
applied impact evaluation techniques.
Steps in PSM
Conduct sample surveys of eligible non-participants and participants
Pool the two samples and estimate the probability of participation
using a logit / probit model with observable individual characteristics
that are likely to determine participation (age, gender, income,
education, etc.).
Get predicted probability of participation (propensity score) for every
sampled participant and non- participant.
For each individual in the sample of participants, find a small n (e.g.,
5) observations in the non-participant sample with the closest
propensity scores.
Note that, however, there are various matching algorithms including Nearest-
Neighbor Matching , Kernel-Based Matching , Radius Matching and
Stratification Matching.
4/10/2021
3.2 Propensity score matching (PSM)
A direct comparison of the treated and the control group is misleading
because the differences between them may not be resulted solely from
the treatment but due to other socio-economic factors too.
Hence, the basic task here is to establish methods which can help to
identify the true effect of the intervention.
PSM constructs a statistical comparison group that is based on a model
of the probability of participating in the treatment, using observed
characteristics.
PSM compares each observation of the treated group with the control
group having similar observed characteristics where the mean
outcome difference between the two groups yields the average
treatment effect of the intervention (Becker & Ichino, 2002).
Steps in PSM
Calculate the mean of the outcome for the chosen n observations.
The difference between that mean and the actual outcome of the
participant is an estimate of the program impact for that participant.
Calculate the mean of these individual program impacts to obtain the
average overall program impact.
It is known as the average treatment effect for the treated (ATT) and
can be shown mathematically as;
2
 4/10/2021

Application of PSM Application of PSM

Estimation of Propensity Scores
Logit model was used to estimate the propensity scores of fertilizer adoption
Example: Estimating the impact of fertilizer adoption
on farmers income.
Data Set:
Primary data has been collected from 400 farmers.
211 fertilizer adopters (Treatment group) and 189 non-
adopters (Control group).

Application of PSM Application of PSM

Region of common support
Once the propensity score estimation of the covariates has been estimated, the
common support region should be imposed on the propensity score
distributions of both sample groups
The area of common support is those propensity scores within the range of the
lowest and highest estimated values for households in the treatment group.
The results of the estimated propensity scores for the case of fertilizer
suggested the region of common support of [0.14108334, 0.99943489] where
only 16 (4%) out of 400 observations were out of the common support
Perform balancing tests: The participants and nonparticipants
should have balanced covariates.
Comparison of standardized bias (difference in means ÷ standard
deviation) before & after matching: Lower after matching
Joint significance (likelihood ratio tests): insignificant after matching.
Psuedo-R2: lower after matching

0 .2 .4 .6 .8 1
Propensity Score

Untreated: Off support Untreated: On support

Treated

3
 4/10/2021

Application of PSM Advantages of PSM

Estimate the average treatment effect of the treated
Adoption of fertilizer resulted in an increment in total income of
households’ range from Birr 8369 to 10710 based on different PSM technique is able to control for observed characteristics.
matching algorithms. It does not require data prior to the intervention/treatement.

Limitations of PSM 3.3 Difference-in-differences (DD)

Assumes that all relevant variables are included in X (observed
covariates), there are no “unobserved” variables that influenced
participation or outcome. It cannot control for unobservables
Often need large data set to identify appropriate comparisons
since it relies on finding similar comparison units
In order to do PSM well, you will need to find many
comparisons, which will require a very large sample size .
This is very expensive.

4
 4/10/2021

3.3 Difference-in-differences (DD) Steps in DD

It estimates the difference in the outcome during the post
intervention period between a treatment group and
comparison group relative to the outcomes observed during a
pre intervention baseline survey (Khandker et al., 2010).
In order to employ DD technique, therefore, at least one
period pre-intervention and post-intervention observational
data should be available.

3.3 Difference-in-differences (DD) 3.3 Difference-in-differences (DD)

Example: Assume a certain project is implemented to reduce
the transmission of HIV/AIDS (2008-2012) among youths
through peer education. Teen pregnancy rate was considered as
a proxy for unprotected sex. The results are presented below

5
 4/10/2021

3.3 Difference-in-differences (DD) 3.3 Difference-in-differences (DD)

3.3 Difference-in-differences (DD) Advantages of DD

Conduct a pre-intervention baseline survey of both
participants and non-participants.

6
 4/10/2021

Limitations of DD 3.4 Switch Regression Model

The underlying assumption is that the treatment and comparison groups Switch Regression Model can account for selection bias.
would behave the same in the absence of the project. That is, the
comparison is the control group. In reality, this is not true –> estimation bias. A limitation of PSM is that unobservable variables that may
In other words, DD method does not take into account the influence of the affect both the outcome variables and choice of technology are
variables other than the treatment. not accounted.
Let us say that you are trying to estimate the impact fertilizer on maize However, the presence of unobserved characteristics in the
production and are doing this by measuring the maize production of fertilizer
adopter farmers (treatment) farmers and non-adopter farmers (comparison)
propensity score estimation can create mismatching and biased
before and after. If in year 1 the adopter farmers are affected by drought, estimators.
whereas the non-adopter farmers are not, then the difference-in-differences To address this problem, switching regression model is found
estimate will produce an invalid estimate of the impact of subsidizing fertilizer.
to be important which assumes selection on unobservables.
In general, any factor that affects only the treatment group has the potential to
invalidate or bias the estimate of the impact of the program.
DD assumes that no such factor is present.

3.4 Switch Regression Model 3.4 Switch Regression Model

Consider the following model, which describes the welfare Switch Regression Model can be used to compare the expected outcome
outcome of households with two regression equations, and a variables of
criterion function D* that determines which regime the  the treated group (a)
household faces:  non-treated group (b)
 the counterfactual hypothetical cases that the treated had they decided
not to treat(c),
 and that the non-treated group had they decided to treat (d).
The conditional expectations for our outcome variables in the four cases
where D* is the unobservable or latent variable for treatment, technology are defined as follows:
adoption, D is its observable counterpart, X are non-stochastic vectors of
observed farm and non-farm characteristics determining technology
adoption, Y is the outcome variable in regimes 1 (treated) and 2 (untreated ),
J represents a vector of exogenous variables thought to influence the
outcome variable.

7

3.4 Switch Regression Model
The effect of the treatment ‘to adopt’ on the treated (TT) can be
calculated as the difference between (a) and (c).
which represents the effect of treatment on the outcome variables that
actually participated in the treatment.
Similarly, the effect of the treatment of the untreated (TU) that actually
did not treated can be calculated as the deference between (d) and (b),
4/10/2021
3.4 Switch Regression Model
Example: A certain specific study is conducted to evaluate the impact of
adoption of improved maize varieties of farmers consumption
expenditure and the following results were observed.
(a) and (b) represent the expected welfare outcome observed in the
sample while (c) and (d) represent the counterfactual scenarios.
The results of the treatment effects indicate that adoption significantly
increases consumption expenditure by about 31% compared to the
non-adopters.
For non-adopters, the mean per capita expenditure would have been
increased by 29% had they adopted improved maize varieties.