Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Meeting 3: Randomized Assignment

Gumilang Aryo Sahadewo

Universitas Gadjah Mada

August 30, 2021

Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Outline of the Talk

1 Introduction to Randomized Evaluations

Why randomize?

2 How to do a randomized evaluation

Random selection and random assignment
External and Internal Validity

3 Basic overview of randomization methods

Simple Lottery
Phase in
Encouragement Design

4 References
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Outline of the Talk

1 Introduction to Randomized Evaluations

Why randomize?

2 How to do a randomized evaluation

Random selection and random assignment
External and Internal Validity

3 Basic overview of randomization methods

Simple Lottery
Phase in
Encouragement Design

4 References
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Outline of the Talk

1 Introduction to Randomized Evaluations

Why randomize?

2 How to do a randomized evaluation

Random selection and random assignment
External and Internal Validity

3 Basic overview of randomization methods

Simple Lottery
Phase in
Encouragement Design

4 References
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Outline of the Talk

1 Introduction to Randomized Evaluations

Why randomize?

2 How to do a randomized evaluation

Random selection and random assignment
External and Internal Validity

3 Basic overview of randomization methods

Simple Lottery
Phase in
Encouragement Design

4 References
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Why randomize?

1 Introduction to Randomized Evaluations

Why randomize?

2 How to do a randomized evaluation

Random selection and random assignment
External and Internal Validity

3 Basic overview of randomization methods

Simple Lottery
Phase in
Encouragement Design

4 References
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Why randomize?

Measuring impact

We aim to evaluate the impact of a particular program,

intervention, or policy
Impact is a comparison between:
the outcome after the program has been introduced
the counterfactual.
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Why randomize?

Measuring impact

We aim to evaluate the impact of a particular program,

intervention, or policy
Impact is a comparison between:
the outcome after the program has been introduced
the counterfactual.
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Why randomize?

Measuring impact

We aim to evaluate the impact of a particular program,

intervention, or policy
Impact is a comparison between:
the outcome after the program has been introduced
the counterfactual.
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Why randomize?

Measuring impact

We aim to evaluate the impact of a particular program,

intervention, or policy
Impact is a comparison between:
the outcome after the program has been introduced
the counterfactual.
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Why randomize?

What is a Counterfactual?

What is a counterfactual?
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Why randomize?

What is a Counterfactual?

Counterfactual is the outcome in the absence of treatment.

What is the issue?
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Why randomize?

The problems of counterfactual

Counterfactual is unobservable as there is no parallel world!

For the same individual, we can only observe either:
The beneficiary receiving business training or
The beneficiary not receiving business training
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Why randomize?

Potential outcome framework

Specifically:

E [Yi | Di = 1] − E [Yi | Di = 0]
= E [Y1i | Di = 1] − E [Y0i | Di = 1]
+E [Y0i | Di = 1] − E [Y0i | Di = 0]
= ATE + E [Y0i | Di = 1] − E [Y0i | Di = 0]
= ATE + Bias
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Why randomize?

Potential outcome framework

Specifically:

E [Yi | Di = 1] − E [Yi | Di = 0]
= ATE + Bias

Comparing individuals who received a treatment to those who

didn’t receive a treatment can be problematic.
These two groups can be different even before the treatment
was implemented: selection bias
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Why randomize?

Random assignment as a solution

The golden standard to address this problem is random

assignment
Basically, we (randomly) assign units to either:
treatment group
control group
Units can be individuals, households, students, schools, firms,
which depends on the context of the evaluation.
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Why randomize?

Average Treatment Effect

In other words, the treatment Di is independent to the

potential outcomes.

E [Yi | Di = 1] − E [Yi | Di = 0]
= E [Y1i | Di = 1] − E [Y0i | Di = 0] .

That is,
E [Y0i | Di = 1] = E [Y0i | Di = 0] .
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Why randomize?

Average Treatment Effect

Random assignment ensures that individuals in the treatment

group do not differ systematically to those in the control
group before the implementation of the program
In other words:
outcomes had individuals not received the treatment,
E [Y0i | Di = 1],
can be proxied by a randomly selected control group,
E [Y0i | Di = 0].
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Why randomize?

Average Treatment Effect

We can then express:

E [Yi | Di = 1] − E [Yi | Di = 0]
= E [Y1i | Di = 1] − E [Y0i | Di = 0]

using the sample analog:

E [Yi | Di = 1] − E [Yi | Di = 0] = E [Y1i − Y0i ] .

The difference in outcome between the two groups can be

attributed to the program but not other factors.
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Why randomize?

Why does randomized assignment produce a valid

counterfactual?

Randomization produces two groups with a high probability of

being statistically identical.
Note that we need enough number of units for the
randomization.
Given enough units, the randomized assignment process will
produce groups that have statistically equivalent averages for
all of their characteristics.
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Why randomize?

An illustration (Gertler et al., 2016)

Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Why randomize?

Each unit has equal probability of being selected for

treatment

Randomized assignment ensures that units in the treatment

and control groups have the same observed characteristics of
the target population.
More importantly, randomized assignment ensures that the
treatment and control groups have the same unobserved
characteristics of the target population.
Note that the unobserved characteristics may differ from one
context to the others.
They may include motivation, preferences, knowledge, etc.
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Why randomize?

How can we test that the treatment and control groups are
equivalent?

We can use the baseline data to conduct balance test between

the treatment and the control groups.
With a high probability, the balance test would show that the
treatment and control groups have the same characteristics on
average.
Is it mandatory to have baseline data for a randomized
assignment?
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Why randomize?

“Women as policymakers” at baseline by Chattopadhyay

and Duflo (2004)

Variables Treatment Control Difference

Female literacy rate 0.35 0.34 0.01
(0.01)
Number of public health facilities 0.06 0.08 -0.02
(0.02)
Tap water 0.05 0.03 0.02
(0.02)
Number of primary schools 0.95 0.91 0.04
(0.08)
Number of high schools 0.09 0.10 -0.01
(0.02)
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Why randomize?

Question #1

What is the difference between impact evaluation and

randomize evaluation?
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Why randomize?

Questions #2

What are the reasons we would want to choose to do a

randomized evaluation?
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Why randomize?

Why randomize?

Randomized evaluations provide the most credible method to

estimate the impact of a program
Individuals in the treatment and those in the control group do
not differ systematically before the start of the program
Thus, we can attribute differences in outcomes after the
implementation of the program to:
the program itself
but not to other factors
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Why randomize?

Opportunities for Randomization (Glennerster, 2009)

Limited budget for full coverage

Random assignment is fair and transparent
Limited implementation capacity
Phase-in
No evidence on which alternative is best
Random assignment to alternatives with equal ex ante chance
of success
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Why randomize?

Opportunities for Randomization (Glennerster, 2009)

Take up of existing program is not complete

Provide information or incentive for some to sign up
Pilot a new program
Good opportunity to test design before scaling up
Operational changes to ongoing programs
Good opportunity to test changes before scaling them up
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Steps in conduction randomized evaluations

Design the study carefully

Randomly assign people to treatment or control
Collect baseline data
Verify that the treatment and control groups do not differ
systematically
Process evaluation and monitoring to protect the quality of
the intervention
Collect follow-up data (midline, or endline) for both groups
Estimate the impact of the program by comparing mean
outcomes of treatment and of control group.
Assess whether the impacts are statistically significant
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Random selection and random assignment

1 Introduction to Randomized Evaluations

Why randomize?

2 How to do a randomized evaluation

Random selection and random assignment
External and Internal Validity

3 Basic overview of randomization methods

Simple Lottery
Phase in
Encouragement Design

4 References
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Random selection and random assignment

Random selection and random assignment

It is important to distinguish between random selection and

random assignment
Random selection chooses units (individuals, households,
firms) randomly from the population
This is also called (random) sampling
The goal is to obtain a representative sample from the
population
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Random selection and random assignment

Random selection and random assignment

It is important to distinguish between random selection and

random assignment
Random selection chooses units (individuals, households,
firms) randomly from the population
This is also called (random) sampling
The goal is to obtain a representative sample from the
population
Random assignment assigns units randomly into the
treatment and control group
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Random selection and random assignment

Random selection
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Random selection and random assignment

Random selection
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Random selection and random assignment

Random selection
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Random selection and random assignment

Random selection
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Random selection and random assignment

Random assignment: balanced check

Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Random selection and random assignment

What is problematic in this figure?

Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Random selection and random assignment

Different randomization units

Individual
Household
Group:
School level
Village or kecamatan level
etc.
Why do we want to randomize at the group level?
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Random selection and random assignment

Choosing the units of randomization

The lower the level of randomization, the higher the

probability that the comparison group will be inadvertantly
affected by the program.
Risks:
Spillovers
Imperfect compliance
Spillovers occur when the treatment group affects the
comparison group or the other way around.
Deworming medicine to children (Kremer and Miguel, 2004)
Imperfect compliance implies that some individuals comply to
the treatment assignment while some don’t
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

External and Internal Validity

1 Introduction to Randomized Evaluations

Why randomize?

2 How to do a randomized evaluation

Random selection and random assignment
External and Internal Validity

3 Basic overview of randomization methods

Simple Lottery
Phase in
Encouragement Design

4 References
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

External and Internal Validity

External and Internal Validity

Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

External and Internal Validity

Different randomization units

External validity
Key: sample is representative of the target population
The results of the evaluation can be generalized to the target
population
Internal validity
Key: the estimated effect of the intervention/program on the
evaluated population reflects the real impact on that
population
Randomized assignment ensures internal validity because it
produces a valid counterfactual
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Simple Lottery

1 Introduction to Randomized Evaluations

Why randomize?

2 How to do a randomized evaluation

Random selection and random assignment
External and Internal Validity

3 Basic overview of randomization methods

Simple Lottery
Phase in
Encouragement Design

4 References
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Simple Lottery

Basic Lottery

This method is most useful when the program was

oversubscribed
Advantages:
easy to implement
Disadvantages
control group may not cooperate
differential attrition
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Simple Lottery

Slum relocation program (Barnhardt, Field, Pande, 2015)

There were 497 eligible women who applied

110 women were assigned to the treatment group
387 women were assigned to the control group
The aim of the study is to evaluate the impact of the
relocation program on general wellbeing, tenure security, and
children’s educational attainment.
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Phase in

1 Introduction to Randomized Evaluations

Why randomize?

2 How to do a randomized evaluation

Random selection and random assignment
External and Internal Validity

3 Basic overview of randomization methods

Simple Lottery
Phase in
Encouragement Design

4 References
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Phase in

Phase in

Phase in takes advantage of programme expansion

Everyone gets programme eventually
This method is sensible when there are resource constraints
from expanding the program
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Phase in

Illustration: the goal is to give everyone textbooks, but...

Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Phase in

Illustration
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Phase in

Illustration
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Phase in

Illustration
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Phase in

Illustration
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Phase in

Phase in

This method is most useful when the program is:

expanding over time
everyone must receive treatment eventually
Advantages:
control group complies because they would receive the benefit
later on
Disadvantages
anticipation of treatment
difficult to measure long-term impact
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Phase in

Primary School Deworming in Kenya (Edward and Kremer,

2004)

This study evaluates the impact of providing deworming drugs

and worm prevention education on educational outcomes
Why phase in?
Logistical and resource constraints
Sample frame: 75 schools in Southern Busia District
Year 1: Group 1 (25 schools) as a treatment, Group 2 and 3 as
controls
Year 2: Group 1 and Group 2 as treatments, Group 3 as control
Year 3: All treatments
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Encouragement Design

1 Introduction to Randomized Evaluations

Why randomize?

2 How to do a randomized evaluation

Random selection and random assignment
External and Internal Validity

3 Basic overview of randomization methods

Simple Lottery
Phase in
Encouragement Design

4 References
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Encouragement Design

Encouragement Design

Consider undersubscribed programs such as savings programs

among poor households, JKN enrollment among informal
workers.
Instead of randomizing access, we can randomize
encouragement to receive treatment
We would expect higher take-up among the encouraged
groups
which eventual induces treatment differential
Examples of encouragement:
SMS messages, information campaigns, visit from agents
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Encouragement Design

Encouragement Design

This method is most useful when the program is:

open to all eligible
but the take-up is low and can be improved with a nudge
Advantages:
Can randomize at individual level even when the program is
not administered at that level
Disadvantages:
Measure impact on specific group: those who responded to the
incentives
Need large inducement to improve take up
Direct effect of encouragement
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Encouragement Design

Increasing Access to JKN Mandiri (J-PAL SEA)

What are the policies to improve enrollment and financial

sustainability of the JKN program?
Both treatment and control groups are eligible to obtain the
JKN
Treatment group:
Subsidies to lower premium cost
Assisted internet registration
Provided additional information
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

Random assignment
Introduction to Randomized Evaluations How to do a randomized evaluation Basic overview of randomization methods References

References for this lecture

Gertler, Paul J., S. Martinez, P. Premand, L. B. Rawlings,

CMJ Vermeesch, (2011). Impact Evaluation in Practice,
IBRD/The World Bank, Washington DC.
“Randomized Evaluation Method,” The Abdul Latif Jameel
Poverty Action Lab
https://www.povertyactionlab.org/research-
resources/introduction-evaluations
Miguel, Edward, and Michael Kremer. 2004. "Worms:
Identifying Impacts on Education and Health in the Presence
of Treatment Externalities." Econometrica 72(1): 159-217.
Barnhardt, Sharon, Erica Field, and Rohini Pande. “Moving
to Opportunity or Isolation? Network Effects of a Randomized
Housing Lottery in Urban India.” NBER Working Paper No.
21419, July 2015.