Lesson 6: PRE-EXPERIMENTAL DESIGNS

In general, a research design is like a blueprint for the research.  A

research design is a plan that guides the decision as to:
1. when and how often to collect data
2. what data to gather and from whom
3. how to analyze the data

More specifically, a research design refers to the type of study that will be
conducted, whether it will be pre-experimental, quasi-experimental, or true
experimental.

Pre-experimental designs include:

1. case study design
2. one group pre-test/post-test design
3. static group comparison design (cross-sectional study)
Quasi-experimental designs include:
1. time series design (may include panel design)
2. equivalent time samples design
3. equivalent materials design
4. nonequivalent control group (comparison group) design
5. counterbalanced design
6. separate sample pre-test/post-test design
7. separate sample pre-test/post-test control group design
8. multiple time-series design
9. recurrent institutional cycle design
10. regression/discontinuity analysis
True experimental designs include:
1. pre-test/post-test control group design
2. Solomon four-group design
3. post-test only control group design

Research Methodology concerns how the design is implemented, how

the research is carried out.  The methodology employed often determines the
quality of the data set produced.

Methodology is concerned with:

1. when and how often to collect data
2. construction of data collection measures
3. identification of the sample or test population
4. choice of strategy for contacting subjects
5. selection of statistical tools
6. presentation of the findings
Pre-Experimental Designs for Description

Descriptive research can provide data for monitoring and evaluating

policies and programs.  These designs are concerned with how to answer
questions such as:
1. How many?
2. How much?
3. How efficient?
4. How effective?
5. How adequate?

Cross-Sectional Design

A cross-sectional design is used for research that collects data on

relevant variables one time only from a variety of people, subjects, or
phenomena.  The data are collected all at the same time (or within a short time
frame).

A cross-sectional design provides a snapshot of the variables included in

the study, at one particular point in time.  It may reveal how those variables
are represented in a cross-section of a population.  Cross-sectional designs
generally use survey techniques to gather data, for example, the U.S. Census.

 Advantages and Disadvantages of Cross-Sectional Designs

Advantages  Disadvantages
data on many variables
data from a large number of subjects
data from dispersed subjects
data on attitudes and behaviors
answers questions on who, what, when,
where
good for exploratory research
generates hypotheses for future research
data useful to many different researchers
increased chances of error
increased cost with more subjects
increased cost with each location
cannot measure change
cannot establish cause and effect
no control of independent variable
difficult to rule out rival
hypotheses
static, time bound

 
Longitudinal Designs

A longitudinal design collects data over long periods of time. 

Measurements are taken on each variable over two or more distinct time
periods.  This allows the researcher to measure change in variables over time. 
There are two different types of longitudinal designs:  time series and panel.
 A Time Series Design collects data on the same variable at regular
intervals (weeks, months, years, etc.) in the form of aggregate measures of a
population.  For example, the Consumer Price Index (CPI), the FBI Uniform
Crime Rate, unemployment rates, poverty rates, etc.

Time series designs are useful for:

1. establishing a baseline measure
2. describing changes over time
3. keeping track of trends
4. forecasting future (short term) trends

Time series data are nearly always presented in the form of a chart or
graph.  The horizontal (or x) axis is divided into time intervals, and the vertical
(y) axis shows the values of the dependent variable as they fluctuate over time.
 
Researchers inspect a time series graph to look for four types of patterns:
1. long term trends (increases or decreases over the whole time span);
2. cyclical variations (short-term, valley-to-valley or peak-to-peak
cycles);
3. seasonal variations (due holidays or weather);
4. irregular fluctuations (none of the above).

 Advantages and Disadvantages of Longitudinal Designs

Advantages   Disadvantages
data easy to collect
easy to present in graphs
easy to interpret 
can forecast short term
trends
 data collection method may change over time
difficult to show more than one variable at a
time
 needs qualitative research to explain
fluctuations
assumes present trends will continue
unchanged

Panel Designs collect repeated measurements from the same people or

subjects over time.  Panel studies reveal changes at the individual level, for
example, when a particular person was employed or unemployed, or when they
were on or off of welfare. Panel data can show different patterns from time
series data.  For example, about 5% of the elderly are institutionalized at any
one time, but it is not always the same people.  So elderly people have a 20%
chance of being institutionalized at some point.
 
 Advantages and Disadvantages of Panel Designs

Advantages Disadvantages
reveals individual level changes
 establishes time order of
variables
can show how relationships
emerge
difficult to obtain initial sample of subjects
difficult to keep the same subjects over time
 repeated measures may influence subjects
behavior

 
  Case Study Design

Case studies examine some phenomenon in depth, e.g., people,

programs, policies, decisions, organizations, etc.  Case studies are useful for
learning about:
1. policies or programs with remarkable successes
2. policies or programs with ambiguous or unexpected outcomes
3. situations where actors have discretionary behavior (e.g., street-
level bureaucrats)

A case study weave together data from documents, archives, interviews,

participation, observation, artifacts, etc.  It attempts to document not only the
"what" but also the "why."
 
Advantages and Disadvantages of Case Study Design
Advantages  Disadvantages
includes data from multiple perspectives limited to contemporary phenomena
combines data from different sources  need direct access to subjects
need diverse sources of information
need skills in many techniques
can be an intense experience
difficult to replicate findings
insiders can be biased
outsiders can be naive
difficult to draw boundaries
 

Focus Groups

Focus groups are a method of group interviewing for obtaining qualitative

data.  It is not so much a research design as a data collection method.  More
will be said about focus groups in the section on data collection.
Meta-Analysis

A meta-analysis is a quantitative analysis of a sample of existing

research studies on a particular topic.  It is used to draw conclusions about
the topic from a range of studies, for example, identify aspects of a program
associated with program success.  A meta-analysis may also generate new
hypotheses for future research.

Problems with doing a meta-analysis are:

1. locating suitable studies
2. studies may not all have the same dependent variable
3. only findings of statistical significance are usually published
4. different studies may have many dissimilar aspects making them
difficult to compare

Control Over Sources of Invalidity in Designs for Description 

Source Case study Cross-section Panel Time-series

Internal Validity
History weak weak weak weak
Maturation weak weak strong strong
Testing strong strong
Instrumentation weak ? ?
Regression weak strong strong
Selection weak weak weak strong
Mortality weak weak strong
Design Contamination ? strong
External Validity
Testing weak weak
Selection weak weak weak ?
Experimental Arrangements ? ?
Multiple Interactions
 
 

How to Improve the Validity of Designs for Description

1.  Case study design

 
X O2
 
2.  One-group Pre-Test/Post-Test Design
 

O1 X O2

3.  Non-randomized comparison group Pre-Test/Post-Test Design

 

O1 X O2
O1 O2

4.  One-group Time Series Design

 

O1 O2 O3 X O4 O5 O6

 
 

Another way to improve the validity of designs for description:

1.  Case study design

 

X O2

2.  Randomized Control Group Post-Test only design

 

X O2
O2

3.  Randomized Control Group Pre-Test/Post-test design

 

O1 X O2
 O1 O2

4.  Control group Time Series Design

 

O1 O2 O3 X O4 O5 O6
O1 O2 O3 O4 O5 O6