Data Collection Methods

Lesson 5: Producing data - Experiments

1 / 30
2 / 30
How to Experiment Well

3 / 30
Well-Designed Experiments

Experiments vs. Observational Studies

Principles of Valid Experiments

1. Comparison
2. Randomization
3. Replication
4. (Double-Blinding)

4 / 30
 Principles of Experimental Design
Comparison: compare side effects of treatments by including at least
two groups; use of a control group can control effect of
lurking variables and be used to measure placebo effect
Randomization: neutralize effects of lurking variables by assigning
subjects to treatments randomly
• Note: random assignment is key, not random
selection of subjects; volunteers are commonly
used as subjects in experiments
Replication: assign more than one subject to each treatment group
to detect important effects
Double Blinding: neither the subjects nor the people who evaluate them
know which treatment each subject is receiving; used to
prevent experimenter effect

• Note: An experiment can still be valid without

incorporating double-blinding 5 / 30
Is this a valid experiment?

Because of concerns about employee stress, a large company

is conducting a study to compare two programs, Tai chi and
Yoga, and their effectiveness in reducing employee stress
levels. The company has assembled a group of volunteer
employees to participate in the study during the first half of their
lunch hour each day for a 10-week period. Each volunteer will
be assigned at random to one of the two programs. Volunteers
will have their stress levels measured just before beginning the
program and 10 weeks later at the completion of it.

Does this study satisfy the 4 principles of valid experiments?

Why or why not?

6 / 30
Class Discussion of Tai chi and Yoga Experiment

A group of volunteers who work together ask to be assigned to

the same program so that they can participate in that program
together. Give an example of a problem that might arise if this
is permitted.

How can this problem be avoided?

7 / 30
Class Discussion of Tai chi and Yoga experiment

Someone proposes that a control group be included in the

design as well. The stress level would be measured for each
volunteer assigned to the control group at the start of the study
and again 10 weeks later. What additional information, if any,
would this provide about the effectiveness of the two programs?

Is it reasonable to generalize the findings of this study to all

employees of this company?

Random selection vs. random assignment

8 / 30
Observational Studies vs. Experiments

Note that observational studies can also have comparison

groups and replication but they don’t have randomization. Only
experiments randomly assign subjects to different treatment
groups.

9 / 30
Even if You Use a Well-Designed Experiment . . .

10 / 30
Pitfalls in Experimentation

Randomized comparative experiments may still have problems:

• placebo effect
• diagnostic bias
• lack of realism
• Hawthorne effect
• noncompliance

11 / 30
Placebo Effect

• A placebo is a dummy treatment that is as similar to the

treatment as possible but contains ”no active ingredient”
• Placebo effect is the favorable response of a human
subject to a placebo or a treatment with no therapeutic
value because of trust in the medical provider or belief that
the treatment will work
• Consequences of placebo effect: ineffective treatment
appears effective relative to untreated subjects

12 / 30
13 / 30
Placebo Effect

Solution:
1. use dummy treatment (saline, sugar pill, etc.) rather than
“no treatment” as comparison treatment
2. blind subjects as to which treatment they are receiving

14 / 30
15 / 30
Diagnostic Bias

Diagnosis of subjects biased by preconceived notions about

effectiveness of treatment
• preconception is confounded lurking variable

16 / 30
Diagnostic Bias

Solution:
1. blind diagnosticians (doctors)
2. studies in which both subjects and diagnosticians are
blinded called double blinded
• e.g., Salk vaccine trial

17 / 30
Lack of Realism

• realism is often compromised by controlled study

conditions, choice of homogeneous subjects, application of
treatments

18 / 30
Lack of Realism

Example 1:
Patients participating in medical trials get better care than most
other patients. Their doctors are specialists doing research on
their specific ailment. They are watched more carefully than
other patients. They are more likely to take their pills regularly
because they are constantly reminded to do so.

Example 2:
Using students in a psychology class in place of workers in the
real world.

19 / 30
Lack of Realism

Solution:
1. awareness of hidden bias
2. admit limitations of experiments

20 / 30
Hawthorne Effect

Hawthorne Effect: phenomenon where people in an

experiment behave differently from how they would
normally behave; attention/observation bias
Example:
• Nielsen TV ratings and TV watching behavior
• Keeping a food journal of snacking habits
Consequence: inaccurate reporting

21 / 30
Non-Compliance

Non-Compliance:
• failure to submit to the assigned treatment
• refusal to follow the protocol of the experiment

Consequence: invalid results

Noncompliance may undermine an experiment. A volunteer

sample might solve this problem

22 / 30
Self-check

In a famous randomized vitamin C study, most patients could

tell from taste whether they were receiving vitamin C pills or
placebo pills. The rate of cold/flu was lower in the vitamin C
group. What do you conclude?
(a) vitamin C reduces the cold/flu rate
(b) nothing – the difference could be due to vitamin C or a
placebo effect

23 / 30
Self-check

In a famous randomized vitamin C study, most patients could

tell from taste whether they were receiving vitamin C pills or
placebo pills. The rate of cold/flu was lower in the vitamin C
group. What do you conclude?
(a) vitamin C reduces the cold/flu rate
(b) nothing – the difference could be due to vitamin C or a
placebo effect

23 / 30
Data Ethics

24 / 30
Data Ethics

25 / 30
Principles of Data Ethics

• Tuskegee experiment
https://www.cdc.gov/tuskegee/timeline.htm
• safety and well-being of the subjects must be protected
• all individuals must give their informed consent before data
are collected
• individual data must be kept confidential

26 / 30
Data Ethics

Was the Salk vaccine trial ethical?

• sufficient belief in the vaccine’s potential to justify exposing
subjects to it?
• enough doubt about the vaccine’s potential to justify
withholding it from subjects?

27 / 30
28 / 30
Self-check

Subjects in an experiment knew that they were being observed,

so they behaved better than they usually did. This is an
example of:
(a) diagnostic bias
(b) Hawthorne effect
(c) placebo effect
(d) Lack of realism

29 / 30
Self-check

Subjects in an experiment knew that they were being observed,

so they behaved better than they usually did. This is an
example of:
(a) diagnostic bias
(b) Hawthorne effect
(c) placebo effect
(d) Lack of realism

29 / 30
Vocabulary

• randomized controlled experiment

• control
• placebo
• randomization
• replication
• response variable
• subject
• treatment
• informed consent
• confidentiality
• non-compliance

30 / 30