Methods of Comparison:

Controlled Experiments and

Observational Studies

Math 1680

Overview

Introduction
Controlled Experiments
Minimizing Bias
Observational Studies
Association, not Causation
Comparing Rates
Salk Vaccine Trial
Breast Cancer Screening
Summary

Introduction

Suppose a company wanted to market a

weight-loss pill
After design is complete, the company still
needs to answer one basic question

Does the pill actually work?

How can the company determine this?

Introduction

One method would be to give the pill to everyone

who wants to lose weight and see if they actually do

Whats wrong with this approach?

First of all, the cost of giving everyone the pill is

prohibitive, and there may be dangerous side effects
for the untested pill.
Secondly, people who want to lose weight may be
exercising or eating more healthily, which could
overshadow the effects of the pill (This is an example
of a confounding factor.)

Controlled Experiments

To test the pill, two groups are created from a

sample of people

Control group is given a placebo

Treatment group is given the pill

Controlled Experiments

Why is it important that the people dont

know which group they are in?
It has been shown repeatedly that a patient's
symptoms can be alleviated by an otherwise
ineffective treatment if the individual expects or
believes that it will work. Conversely, members
of a control group knowingly receiving an inert
substance tend to report a worsening of
symptoms. This phenomenon is called the
placebo effect.

Controlled Experiments

The most ideal form of experiment is one

where neither the subjects nor the evaluators
know who received the treatment

This is a double-blind experiment

This design provides the least opportunity for

biased results

Controlled Experiments

After a period of time, the company checks to

see how much weight people in each group
have lost

If there is a significant difference between the

groups, assume the pill was the reason
If there is no significant difference between the
groups, assume the pill had no effect

Minimizing Bias

What if the control group and the treatment group

differ in some way other than the treatment?

Then we have confounding factors in the study, and the

results will be biased and/or invalid

Confounding factors can include

Age
Race
Gender
Socio-economic status (SES)
Location
Political/religious orientation

Minimizing Bias

In the pill example, suppose all of the men were

placed into the treatment group and all of the women
were placed into the control group

What is wrong with this design?

The pill may interact differently with people

based on gender. For example, suppose
that high levels of testosterone diminish the
effect of the pill. Then the results would be
biased against the pills effectiveness.

Minimizing Bias

In order to minimize the risk of bias,

researchers assign people to each group
randomly from the pool of subjects

If enough people are in the pool, the groups will

look similar in terms of age, race, gender, or any
other demographic which could be a confounding
element

Observational Studies

How would you design a controlled

experiment to test if second-hand smoke
causes lung cancer?
Answer: You cant, really. The time scale is too large and
it would be virtually impossible to ensure that your
control group never got second-hand smoke exposure.

Observational Studies

In many studies, designing a controlled experiment

is simply not possible

In such cases, researchers must settle for observational

studies
Observational studies let the subjects select which group to
join

Observational studies should be placed under great

scrutiny because the control and treatment groups
may be quite different in many ways

These differences may introduce confounding influences on

the study results

Observational Studies

A TV commercial observes that volunteers

who switched to having to Cheerios for
breakfast reduced their cholesterol level by
4%. Does this show that Cheerios cereal is
effective in reducing your cholesterol level?
Not necessarily. People who
volunteer for such a study may
already be interested in improving
their health and may be taking steps
such as exercising daily. This could
lower their cholesterol level.

Observational Studies

One form of observational study involves

using historical controls to compare a
modern treatment with

The difference in conditions between past and

present can itself be a confounding factor
For example, in testing a polio vaccine, one would
not want to compare infection rates in the 1950s
with infection rates in the 1900s because of the
large differences in standard of living and health
care quality in the two periods

Association, not Causation

An important limitation of all studies is that they can only show

the strength of association between two variables

Association may point to causation

For example, if exposure to a virus causes a disease, then people
who are exposed should be sicker than similar people who are not
exposed

However, a study alone cannot show causation

Human inference is needed to connect causal links

Generally, a casual link is accepted if

A scientific theory is derived which explains the association

Enough studies controlling all potential confounding factors are
performed, and all of them show the same association

Association, not Causation

Cervical cancer is more common among

women who have been exposed to the
herpes virus, according to many
observational studies. Is it fair to conclude
that the virus causes cervical cancer?
No. Both herpes and cervical cancer have been shown
to be sexually transmitted diseases. Women who have
many partners are at higher risk for both diseases than
women with few partners, so the number and type of
partners is a confounding influence which explains the
association.

Association, not Causation

Some studies find an association between liver

cancer and smoking. Does this mean smoking
causes liver cancer, or is there a confounding
influence at play?

People who smoke also tend to drink, so drinking is

associated with smoking. It has been shown
repeatedly that over consumption of alcohol can
cause liver cancer.

Comparing Rates

Sometimes the groups in comparison are of

different sizes

Rather than comparing the actual numbers, use

rates or proportions such as percents
Adjusting to rates allows for a more direct
comparison

Comparing Rates

In the U.S. in 1990, there were 2.1 million deaths

from all causes compared to 1.7 million in 1960

According to Census data, there were about 177 million

Americans in 1960 and about 250 million in 1990

True or false: The publics health got worse over the

period 1960-1990.
False. Convert the death counts to percentage
rates. The death rate in 1960 was 0.96%, while
the death rate in 1990 was 0.84%. If anything,
public health improved.

Salk Vaccine Trial

In 1954, researchers tested the effectiveness of a

vaccine for polio
Approximately 400,000 children were selected

All children in the study had parental consent to be

vaccinated (Why?)
Children randomly divided into equal sized treatment and
control groups

Control group was given a placebo

Is this a controlled experiment or an observational

study?

Salk Vaccine Trial

Results
Study Group

Population

Polio Cases

Rate per
100,000

Vaccinated

200,745

57

28.4

Placebo

201,229

142

70.6

Was the vaccine effective?

Breast Cancer Screening

In 1963, an insurance company used 62,000

women (all insured by the plan) as subjects

Women ranged in age from 40 to 64

31,000 of the women were used as controls
The others were encouraged to undergo annual
screening

Roughly 20,200 women from the treatment group came

for annual screenings, the other 10,800 refused

After 5 years, the death rates were checked

Breast Cancer Screening

Is this a controlled experiment or an

observational study?
It depends on what questions you are trying to answer.
If you are only interested in the difference between the
treatment and control groups, then you are using the
results of a controlled experiment. However, when
you start drawing inferences between the screened
and refused groups, you are looking at an after-thefact observational study.

Breast Cancer Screening

Deaths (breast
cancer)

Deaths (all other

causes)

Number

Rate per
1,000

Number

Rate per
1,000

31,000

39

1.3

837

27

Examined

20,200

23

1.1

428

21

Refused

10,800

16

1.5

409

38

31,000

63

879

28

Treatment

Control

Researchers noted three things:

1. Screening had little impact on diseases other than breast cancer.
2. Poorer women were less likely to accept screening than richer ones.
3. Most diseases fall more heavily on the poor than on the rich.

Breast Cancer Screening

Deaths (breast
cancer)

Deaths (all other

causes)

Number

Rate per
1,000

Number

Rate per
1,000

31,000

39

1.3

837

27

Examined

20,200

23

1.1

428

21

Refused

10,800

16

1.5

409

38

31,000

63

879

28

Treatment

Control

Does screening save lives? Which numbers prove

your point?

Breast Cancer Screening

Deaths (breast
cancer)

Deaths (all other

causes)

Number

Rate per
1,000

Number

Rate per
1,000

31,000

39

1.3

837

27

Examined

20,200

23

1.1

428

21

Refused

10,800

16

1.5

409

38

31,000

63

879

28

Treatment

Control

Why is the death rate from all other causes in the whole
treatment group about the same as the rate for the control
group?

Breast Cancer Screening

Deaths (breast
cancer)

Deaths (all other

causes)

Number

Rate per
1,000

Number

Rate per
1,000

31,000

39

1.3

837

27

Examined

20,200

23

1.1

428

21

Refused

10,800

16

1.5

409

38

31,000

63

879

28

Treatment

Control

Why is the death rate from all other causes higher for the
refused group than the examined group?

Breast Cancer Screening

Deaths (breast
cancer)

Deaths (all other

causes)

Number

Rate per
1,000

Number

Rate per
1,000

31,000

39

1.3

837

27

Examined

20,200

23

1.1

428

21

Refused

10,800

16

1.5

409

38

31,000

63

879

28

Treatment

Control

Unlike most diseases, breast cancer afflicts the rich more

than the poor. Which numbers in the table confirm this
association?

Case Study: Child care and

behavior

Question: If parents choose to use child care,

are they more likely to see undesirable
behaviors in their children?
Question: How should one find out? By use
of controlled experiments, or by
observational study?

Case Study: Child care and

behavior

Answer: In 1991, a study commenced on 1364

infants (subjects) and followed them through their
sixth year in school. 12 years later, an article was
published. the more time children spent in child care
from birth to age four-and-a-half, the more adults
tended to rate them, both at age four-and-a-half and
at kindergarten, as less likely to get along with
others, as more assertive, as disobedient, and as
aggressive.

Case Study: Child care and

behavior

A summary of the study noted, The study

authors noted that their study was not
designed to prove a cause and effect
relationship. (that more child care time
causes more aggressive behavior)

Possible confounding factors?

Perhaps perhaps perhaps

Case Study: Child care and

behavior

Question: How about a controlled experiment

instead? (i.e. the researchers select children
to place in child care)

What are the advantages?

What problems might arise?

Summary: Basic Principles of

Statistical Experimental Design

Raise question to answer: Explanatory

variable(s) & Response variable(s).
Leading principle: Comparison.
Other principles: randomize, repeat (or
involve many subjects to reduce chance
variation)
Why:

Summary: Principles of Statistical

Experimental Design

Limitations: Statistical analysis of an experiment

cannot tell us how far the results will generalize to
other settings. (Experiment run in Massachusetts;
how about in the entire USA?)

However, the randomized comparative experiment,

because of its ability to give convincing evidence for
causation, is one of the most important ideas in
statistics.

More Elaborate Statistical Designs

Matched Pairs Design: Pair subjects with the

same age, sex, income, etc. and compare
their responses
Example: Compare two advertisements for
the same product
Common variation of MPD: impose both treatments
on the same subjects, so that each subject serves as
his or her own control. (See HW2, Q9)

More Elaborate Statistical Designs

Question: How does randomization work here?

Answer: Which one of a matched pair sees the first
ad is decided at random. Or, which treatment a
person gets first is decided at random if he or she
also serves as his or her own control.
Additional Example: Matched Pairs for the cell phone
experiment. (Details: trained in using simulator, order
in which a subject drives with and without the phone
be random, two drives be on separate days)

Summary

Statisticians use the method of comparison

To determine the effectiveness of a treatment, they usually

compare the responses of a treatment group with a control
group

If the control group is comparable to the treatment

group (apart from the treatment), then a significant
difference in the responses of the two groups is likely
to be due to the effect of the treatment

Summary

If the treatment group is different from the

control group with respect to other factors,
the effects of these other factors are likely to
be confounded with the effect of the
treatment
To make sure the treatment group is like the
control group, investigators put subjects into
either group at random

Summary

Whenever possible, the control group is given a

placebo

The response should be to the treatment itself rather than to

the idea of the treatment

In a double-blind experiment, the subjects do not

know whether they are in treatment or in control, and
neither do those who evaluate the responses

Guards against bias, both in the responses and in the

evaluations

Summary

In an observational study, the investigators

do not assign the subjects into treatment or
control groups

Subjects which have the condition are the

treatment group and the ones which do not are
the control

Observational studies can establish

association, which is not necessarily
causation

Summary

Observational studies are particularly susceptible to

confounding factors

This can make them very misleading about cause and effect
relationships

With observational studies, try to find out how the

subjects came to be in treatment or control

Are the groups comparable?

What factors are confounded with the treatment?
What adjustments (if any) did the investigators make to
control for these factors?