Module 2.

Lecture 2: The Experimental Ideal

Anirban Mukherjee

November 17, 2021

Anirban Mukherjee Module 2.Lecture 2: The Experimental IdealNovember 17, 2021 1 / 16

The Selection Problem

The most powerful research designs use random assignment.

But we often don’t have data generated from Randomized control
trials (RCT). In RCT the assignment of treatment is random. But
for survey data it is not.
In reality, the assignment of the treatment may be governed by
some mechanism thereby creating selection bias.
Suppose you are interested in a “if-then” type question (Remember
that this is the type of predictions you get from positive theories. Also
remember that for a question to be interesting there has to be some
ambiguity in the theoretical prediction. Empirical estimation brings a
closure to this ambiguity).

Anirban Mukherjee Module 2.Lecture 2: The Experimental IdealNovember 17, 2021 2 / 16

To be concrete, consider a simple example: Do hospitals make
people healthier? Contemplate the possible answers to this
question.
To make the case interesting imagine we are studying a poor
elderly population that uses hospital emergency rooms for primary
care.
This sort of care is expensive, crowds hospital facilities, and is,
perhaps, not very effective. In fact, people may pick secondary
infection from staying at the hospital.
But on the other hand, hospitals provide many valuable services
thereby tilting the answer towards an “yes”.
How do we look at the data to find answer to these questions?

Anirban Mukherjee Module 2.Lecture 2: The Experimental IdealNovember 17, 2021 3 / 16

The first approach to find answer to this problem would be to
compare the health status of those who were admitted to the
hospital with the health of those who were not.
Consider the following data from National Health Service, UK. In
this data people responses to two questions are summarized.
– During the past 12 months, was the respondent a patient in a
hospital overnight?
–Would you say your health in general is excellent (1), very
good(2), good (3), fair (4), poor (5)?
The difference in the means is 0.71, a large and highly significant
contrast in favor of the non-hospitalized, with a t-statistic of 58.9.

Anirban Mukherjee Module 2.Lecture 2: The Experimental IdealNovember 17, 2021 4 / 16

 Table: Hospitalization and Health Status in UK

Group Sample Size Mean Health status Standard Error

Hospital 7774 2.79 0.014
No Hospital 90049 2.07 0.003

Anirban Mukherjee Module 2.Lecture 2: The Experimental IdealNovember 17, 2021 5 / 16

At the face value the results suggests that hospital makes people
sicker. This may not be wrong as people get secondary infection
from hospitals.
However, it is easier to see why the comparison should not be
taken at the face value – people who go to the hospital are
probably less healthy to begin with.
To describe this problem more precisely, think about hospital
treatment as described by a binary random variable, Di = 0, 1. The
outcome of interest, a measure of health status, is denoted by Yi .
The question is whether Yi is affected by hospital care.

Anirban Mukherjee Module 2.Lecture 2: The Experimental IdealNovember 17, 2021 6 / 16

For any individual there are two potential health variables:

Y1i if Di =1
potential outcome =
Y0i if Di =0
In other words,Y0i is the health status of an individual had he not
gone to the hospital, irrespective of whether he actually went,
while Y1i is the individual’s health status if he goes.
The observed outcome, Yi , can be written in terms of potential
outcomes as 
Y1i if Di =1
Yi =
Y0i if Di =0 (1)
= Y0i + (Y1i − Y0i )Di
This notation is useful because (Y1i − Y0i ) is the causal effect of
hospitalization for an individual.

Anirban Mukherjee Module 2.Lecture 2: The Experimental IdealNovember 17, 2021 7 / 16

In general, there is likely to be a distribution of both Y1i and Y0i
in the population, so the treatment effect can be different for
different people.
But we never see the both potential outcome for any one person,
we must learn about the effects of hospitalization by comparing
the average health of those who were and were not hospitalized.

Anirban Mukherjee Module 2.Lecture 2: The Experimental IdealNovember 17, 2021 8 / 16

The comparison of average health conditional on hospitalization
status is formally linked to the average causal effect by the
equation below

E [Y |D = 1] − E [Yi |Di = 0] = E [Y1i |Di = 1] − E [Y0i |Di = 1]

| i i {z } | {z }
Observed difference in average health Average treatment effect on the treated

+ E [Y0i |Di = 1] − E [Y0i |Di = 0]

| {z }
Selection bias

Note that E [Y1i |Di = 1] − E [Y0i |Di = 1] = E [Y1i − Y0i |Di = 1] is

the average causal effect of hospitalization on those who
were hospitalized.

Anirban Mukherjee Module 2.Lecture 2: The Experimental IdealNovember 17, 2021 9 / 16

In the data we can only observe the LHS term which is the
observed difference.
But that term includes selection bias which is the difference in
average Y0i between those who were and were not hospitalized.
Because the sick are more likely than the healthy to seel
treatment, those who were hospitalized have worse Y0i making the
selection bias negative in the example.

Anirban Mukherjee Module 2.Lecture 2: The Experimental Ideal

November 17, 2021 10 / 16
Random assignment solves the selection problem

Random assignment of Di solves the selection problem because

random assignmment makes Di independent of potential
outcomes. Note that

E [Yi |Di = 1] − E [Yi |Di = 0] = E [Y1i |Di = 1] − E [Y0i |Di = 0]

(2)
= E [Y1i |Di = 1] − E [Y0i |Di = 1]

where the independence of Y0i and Di allows us to swap

E [Y0i |Di = 1] for E [Y0i |Di = 0]

Anirban Mukherjee Module 2.Lecture 2: The Experimental Ideal

November 17, 2021 11 / 16
In fact, given random assignment, this simplifies further to

E [Y1i |Di = 1] − E [Y0i |Di = 1] = E [Y1i − Y0i |Di = 1]

(3)
= E [Y1i − Y0i ]
Randomized trials have their own problems. But experiments
often reveal things that are not what they seem on the basis of
naive comparisons alone.
An icononic example from our own field of labour economics is the
evaluation of government-subsidized training programs in UK.
These are programs that provide a combination of classroom
instruction and on-the-job training for groups of disadvantaged
workers such as the long-term unemployed, drug addicts, and
ex-offenders to increase their employment and earnings.

Anirban Mukherjee Module 2.Lecture 2: The Experimental Ideal

November 17, 2021 12 / 16
Paradoxically, studies based on non-experimental comparisons of
participants and and non-participants often show that after
training, the trainees earn less than plausible comparison groups
(see e.g. Ashenfelter, 1978, Ashenfelter and Card, 1985, Lalonde,
1995)
Here too, selection bias is a natural concern since subsidized
training programs are meant to serve men and women with low
earnings potential.
In contrast, evidence from randomized evaluations of training
programs generate mostly positive effects (see, e.g., Lalonde, 1986;
Orr, et al, 1996).

Anirban Mukherjee Module 2.Lecture 2: The Experimental Ideal

November 17, 2021 13 / 16
Regression Analysis of Experiments

Suppose that the treatment effect is the same for everyone, say
Y1i − Y0i = ρ, a constant. With constant treatment effects, we can
rewrite equation (1) as

Yi = |{z}
α + ρ Di + ηi (4)
|{z} |{z}
= E [Y0i ] = (Y1i − Y0i ) = Y0i − E (Y0i )

Hence, we have

E [Yi |Di = 1] = α + ρ + E [ηi |Di = 1]

E [Yi |Di = 0] = α + E [ηi |Di = 0]

This means that we have E [Yi |Di = 1] − E [Yi |Di = 0] =
ρ(the treatment effect)+E[ηi |Di = 1] − E [ηi |Di = 0](selection bias).

Anirban Mukherjee Module 2.Lecture 2: The Experimental Ideal

November 17, 2021 14 / 16
Thus, selection bias amounts to correlation between the regression
error term, ηi , and the regressor, Di .
Since,

E [ηi |Di = 1] − E [ηi |Di = 0] = E [Y0i |Di = 1] − E [Y0i |Di = 0]

this correlation reflects the difference in (no-treatment) potential

outcomes between those who got treated and those who don’t.
In an experimental set up where Di is randomly assigned, the
selection term disappears, and a regression of Yi on Di estimates
the causal effect of interest, ρ.

Anirban Mukherjee Module 2.Lecture 2: The Experimental Ideal

November 17, 2021 15 / 16
Usually, in a regression other controls variables are attached and
the long regression looks like this:

Yi = α + ρDi + Xi0 γ + ηi (5)

If the controls Xi0 are uncorrelated with the treatment Di , then

they will not affect the estimate of ρ
Inclusion of these controls may get more precise estimates of the
causal effect of interest.
Although the controls variables are uncorrelated with the
treatment, they have substantial explanatory power.
Including these control variables therefore reduces the residual
variance, which in turn lowers the standard error of the regression
estimates.

Anirban Mukherjee Module 2.Lecture 2: The Experimental Ideal

November 17, 2021 16 / 16