Research Design: Sampling and

Method of Data Collection

Chapter 4
How do you identify the research design in a
quantitative study?
– Usually named in the Methods section
– May be under a sub-heading:
• Research Design
• Study Design
• Methodology
– May be named in the Abstract
– Includes references to literature that
discusses this design
– Explains why the particular design was
selected
– If the design is not named, clues may be
found in the Methods section
What characteristics distinguish the different quantitative research designs?

• There are two basic categories:

– Experimental research tests the effect of
an intervention by manipulating the
conditions experienced by participants
– Non-experimental research describes
variables without manipulating the
conditions experienced by participants
What is experimental quantitative research?
– Used to determine whether an independent
treatment variable caused an effect in a dependent
variable
– Conditions are carefully controlled to ensure any
measured effect was the result of treatment, and
not some other confounding variable
– Conditions are manipulated and the dependent
outcome variable is measured to determine if an
effect occurred
– Also referred to as intervention research because
when conditions are manipulated, they are
intervening in participants’ lives in some way
 What is non-experimental quantitative
research?
– Describes how variables are related to each other
or describes the trends for variables in a large
population
– Measures variables of interest, but does not
manipulate the conditions experienced by
participants
– Also referred to as descriptive or observational
research
– Carefully selects representative participants
– Makes strong claims that the results generalize to
that population
– Cannot conclude cause/effect relationship
What are the five common quantitative research designs?

–True experiment design

–Quasi-experiment design
–Single-subject research design
–Correlational research design
–Survey research design
What is a true experiment research design?

– Used to be certain a treatment causes an effect

– Sometimes called the “gold standard” for
conducting quantitative research
– Manipulates the conditions experienced by
participants
– Uses random assignment to the different levels of
the treatment variable
• Controls for confounding variables that might affect the
outcome
– Frequently used in medical research
 What is a quasi-experiment research
design?
– Used to determine the effect of a treatment for
intact groups
– Cannot randomly assign individuals to the different
treatment groups
– Uses groups that exist naturally or are already
formed
– Assigns existing groups to the different conditions,
but does not randomly assign individuals because
groups cannot be artificially created for the
experiment
– Only able to provide moderate to weak conclusions
about the treatment causing a measured effect
 What is a single-subject research design?

– Uses single-subject designs when they want to

impact an individual
– Involves the study of single individuals
– Administration of an intervention
– Careful monitoring of the individuals’ behaviors
before, during, and after the intervention to
determine whether the treatment affects the
behavior
– The individual becomes his/her own “control” in a
single-subject experiment by the researcher
collecting many measurements of his/her behavior
over time
 What is a correlational research design?

– Used to describe the relationships among variables

without manipulating what individuals are
experiencing
– Indicates the extent to which the two variables are
or are not related or the extent that one can predict
another
– Usually studies a single group of individuals
– Does not attempt to control or manipulate variables
– Can conclude that two variables are related to each
other
– Cannot conclude cause and effect
What is a survey research design?
– Used to describe trends in a population
– Administers a survey questionnaire to a smaller
group of people (called the sample) in order to
identify trends in attitudes, opinions, behaviors, or
characteristics of a larger group of people (called
the population)
– Carefully selects people to study to ensure that they
are representative of a larger population that is of
interest to the researcher
– Able to make conclusions about trends in the
population as a whole based on the results from the
sample
 How do you recognize the research design in a quantitative research report?

• Look to see if the author named the design in the

title or abstract of the report
• Examine the purpose statement to see if it names
or suggests the study’s research design
• Read the beginning paragraphs of the Method
section and look for a statement that identifies
the design
• Examine the procedures described within the
Method and Results sections
Characteristics of quantitative research designs

Did the researcher What was the focus How were participants Most likely research
manipulate conditions? of data analysis? selected/assigned? design is…
No
One group
Who did the Describing trends
researcher study? Random selection of participants
Survey Research

Relationships among variables

Conveniently available individuals
Correlational

Yes
One individual
(or a of
Inspecting graphs few)
individual data
Single Subject

Two or more groups

Comparing groups
Random assignment
True Experiment

Assign intact groups to condition

Quasi-experiment
 How do you evaluate the research design in
a quantitative study?
Indicators of Indicators of
Quality criteria
higher quality lower quality
The choice of  Design fits study’s  Mismatch among
the research intent study’s intent and
design is  A convincing research design
appropriate and explanation for why the  Explanation for why
justified research design was the research design
selected is provided was selected is
unconvincing
Good  Randomly assign  Procedures for
quantitative individuals to assigning or selecting
procedures are conditions or select participants to
used to select individuals to conditions are
and assign participate when inappropriate
participants possible
 How do you evaluate the research design in
a quantitative study?
Indicators of Indicators of
Quality criteria
higher quality lower quality
Good data  Rigorous quantitative  Poor quantitative data
collection data collection collection procedures
procedures are procedures are used are used
used
Good data  Rigorous quantitative  Poor quantitative data
analysis data analysis analysis procedures
procedures are procedures are used are used
used
Good results  Reported results are  Reported results are
and conclusions appropriate and inappropriate or
are reported complete incomplete
 Claims made are  Inappropriate claims
appropriate for design are made
How do you evaluate the research design in
a quantitative study?
Indicators of Indicators of
Quality criteria
higher quality lower quality
The study  All elements of the  There are
used a study from problem to inconsistencies in how
rigorous purpose to methods to the study’s problem,
research results to conclusions purpose, methods,
design fit together in a logical, results, and
coherent way conclusions fit together
The use of the  Results and  Results and
quantitative conclusions provide a conclusions provide a
research rigorous explanation haphazard examination
design that fulfills the study’s of the study’s intent and
addressed the intent and answers the do not adequately
study’s study’s research answer the study’s
purpose questions research questions
How do you identify the participants in a
quantitative study?
– Typically appears as one of the first
topics discussed in the method section
– Often under a subheading such as
participants or sample
– Identifies the population and sample
– Determines the sampling strategy used
• Probability sampling
• Nonprobability sampling
 What is probability sampling?
– Random process to select individuals (or units,
such as schools) from the population so that each
individual has a known chance (or probability) of
being selected
– Most rigorous form of sampling because the
investigator can make the strongest claims that the
sample is representative of the population
– Requires that the researcher obtain a list of every
person in the population
– Results found in the study may likely be generalized
to the larger population
What is nonprobability sampling?

– Select individuals to study because they

are available, convenient, and meet
some criteria
– Easier for researchers to implement
than probability sampling
– Limits the conclusions that can be
drawn about the results from a study
 How are the different sampling methods characterized?
Category Type of sampling strategy Characteristics
Probability Random sampling Randomly select individuals from
sampling a list of all members of the
population of interest

Systematic random Randomly pick a starting point

sampling within a list of all members and
then selects every nth individual
on the list

Stratified random Divide the population into

sampling subgroups and randomly selects
participants from each group
Nonprobability Convenience Select participants who are
sampling sampling available and accessible
Purposive sampling Selects participants “on purpose”
because they are considered to
be most appropriate for the study
 What is an adequate number of
participants?
– A minimum of 15 participants in each group in
a true experiment or quasi-experiment
– One participant (or a few) in a single-subject
study (with data collected over a large
number of points in time)
– A minimum of 30 participants for a
correlational study that relates variables
– Approximately 350 individuals (or more) for a
survey study, but this size varies
How do you identify the data collection in a
quantitative study?

– Look in the Method section of research

reports
– May be under a heading “data
collection”
– Note the general procedures for
collecting data
– Identify the instruments used to gather
quantitative data
How do you understand the instruments used to gather quantitative data?

– Identify how the variables were specified

– Identify the type of instrument used to gather
information
• Demographic forms
• Performance measures
• Attitudinal measures
• Behavioral observation checklists
• Factual information documents
– Assess the evidence that the researcher used a
good instrument
• The scores need to be reliable (i.e., consistent)
• The scores need to valid (i.e., accurate)
How do assess if the scores are reliable
and valid?
– Citation to the literature indicating that the
instrument was previously developed and
used for research purposes
– Text of the items is included so you can
judge whether they are clear and
unambiguous
– Experts assessed that the items on the
instrument cover the appropriate content
(called content or face validity)
How do you understand the procedures researchers use to collect data?

– Note indicators that the procedures were

ethical
– Expect the data collection procedures to be
standardized
– Identify how the researchers reduced threats
to the studies’ conclusions
• Independent variables really caused effects in
the dependent variables
• Findings from the sample generalize to a larger
population
What procedures can increase a study’s
internal validity?
– Manipulating the conditions so they are as similar
as possible between groups and across time
except for the treatment related to the independent
variable
– Randomly assigning individuals to treatment
groups so existing differences among the
individual participants are spread across the
groups
– Measuring other variables that need to be
controlled, such as giving a pretest or assessing
individuals for attitudes or behaviors that may be
relevant to how they respond to a treatment
What procedures can increase a study’s external
validity?
– Randomly select individuals from a list of all members
of the population to participate in the study
– Use procedures that encourage as many participants
as possible to respond to the study’s measurement
instruments
• If the response rate is low (e.g., 60% or lower) then the
results may not generalize since the people who chose not
to respond may be different from those who did respond
– Examine the demographic information obtained from
the participants to determine whether the individuals
who participated are similar to the larger group
 How do you evaluate participants and data collection
in a quantitative study?

Indicators of Indicators of
Quality criteria
higher quality lower quality
The sampling  Probability sampling  Nonprobability
strategy is strategy is used to sampling strategy is
appropriate and select sites and used to select sites
justified participants and participants
 Procedures are fully  Procedures are
described unclear
The sample  Sample size is large,  Sample size is small,
size is reducing the amount of introducing the
appropriate and sampling error likelihood of sampling
justified  Sample size is justified error
using sample size  Weak rationale is
formula provided to justify
sample size
 How do you evaluate participants and data collection
in a quantitative study?

Indicators of Indicators of
Quality criteria
higher quality lower quality
High quality  Appropriate for  Unclear how
instruments are variables instruments align
used to gather  Strong evidence of  Weak evidence of
data reliability and validity reliability and validity
The data are  Appropriate approvals  Approvals were not
gathered using were secured secured
ethical  Adequate steps were  Procedures protecting
procedures taken to prevent harm participants are
to participants questionable
The data are  Procedures are fully  Procedures are unclear
gathered using described and and poorly explained
standardized explained
procedures
How do you evaluate participants and data collection
in a quantitative study?
Indicators of Indicators of
Quality criteria
higher quality lower quality
The study  All elements of the  Inconsistencies in how
used a study fit together in a the study’s problem,
rigorous logical, coherent way purpose, methods,
research results, and
design conclusions fit together
The use of the  Results and  Results and
quantitative conclusions from conclusions from the
research research design research provide a
design provide a rigorous haphazard examination
addressed the explanation that fulfills of the study’s intent and
study’s the study’s intent and do not adequately
purpose answers the study’s answer the study’s
research questions research questions
What are the four common qualitative research
designs?

– Narrative
– Case study
– Ethnography
– Grounded theory
What characteristics distinguish the different qualitative research designs?

– Central phenomena of interest (e.g., a

group’s culture, an individual’s story)
– Intents (e.g., to describe and interpret,
to develop a theory)
– Procedures (e.g., how data are
collected, analyzed, and interpreted;
how results are reported)
 What are the identifying the characteristics of narrative research?

– Used terms such as narrative research, narrative

inquiry, story, biography, or autobiography
– Explored individual experiences as told through stories
– Collected field texts (personal conversations, journals,
letters, etc.) which document the individual’s story
– “Re-storied” the data by identifying story elements
(e.g., characters, plot, setting) and organizing them
– Analyzed the data for themes, tensions within the
story, and larger contexts that shape the story
– Results and Conclusion sections focused on telling
and interpreting the individual’s story
 What are the identifying the characteristics of case study research?

– Uses terms such as case study, collective case

study or multiple case study
– Research problem calls for in-depth exploration of
a case, a bounded system of people
– Selects case because it is unusual and has merit
in and of itself or illustrates a specific issue
– Collects interviews, observations, pictures,
documents, videotapes, and/or e-mails
– Analyzes data for description and themes
– Results included description of the case(s),
themes, and lessons learned
What are the identifying the characteristics of ethnographic research?

– Uses terms such as ethnography or ethnographic

– Describes a culture-sharing group
– Conducts extensive fieldwork to understand the
language, rituals, communication styles, and
economic and political structures that develop over
time
– Collects interviews, documents, and other artifacts
– Analyzes the data for shared patterns of behaviors,
beliefs, and language
– Results included a cultural portrait
– Researcher reflects on his or her role
 What are the identifying the characteristics of grounded theory research?

– Uses the term grounded theory

– Research problem calls for a theory that explains a
process, action, or interaction
– Collects interviews from multiple people to provide
the most complete data
– Analyzes the data using multiple stages of coding
• Open coding develops categories from the collected data
• Axial coding identifies one category that is at the heart of
the process and its relation to the other categories
• Selective coding develops propositions or hypotheses
– Results include a model of the theory and
propositions from the theory
How do you recognize the research design in a
qualitative research report?
• Check if the author named the design in the title or
abstract of the report
• Examine the purpose statement to see if it names
or suggests the study’s research design
• Read the beginning paragraphs of the Method
section and look for a statement that identifies the
design
• Examine the Method and Results sections to
identify how the author collected, analyzed, and
reported the data
How do you evaluate the research design in
a qualitative study?
Indicators of Indicators of
Quality criteria
higher quality lower quality
A research  Used an overall plan,  Used generic
design guides such as narrative qualitative approach or
the conduct of research, case study, no clear plan for the
the qualitative ethnography, or study
study grounded theory  Incorrect use of terms
 Correct use of terms or lack of supporting
and citing up-to-date literature
literature
How do you evaluate the research design in
a qualitative study?
Indicators of Indicators of
Quality criteria
higher quality lower quality
The choice of the  Design fits central  Mismatch among the
research design is phenomenon: study’s intent, central
appropriate and  Narrative: tell the story of phenomenon, and
an individual’s life
justified  Case study: explore one or research design
more cases  Explanation for why
 Ethnography: describe a
group’s cultural patterns
the specific research
 Grounded theory: generate design was selected
a theory about a process
is unconvincing or
 Convincing unclear
explanation for why
specific research
design was selected
How do you evaluate the research design in
a qualitative study?
Indicators of Indicators of
Quality criteria
higher quality lower quality
Good data  Procedures are  Procedures are
collection appropriate inappropriate
procedures are  Rigorous qualitative  Poor qualitative
used procedures are used procedures are used
Good  Procedures for  Procedures for
qualitative data analyzing data are analyzing the data are
analysis appropriate inappropriate
procedures are  Rigorous qualitative  Poor or limited
used procedures that go qualitative procedures
beyond identifying are used
themes are used
How do you evaluate the research design in
a qualitative study?
Indicators of Indicators of
Quality criteria
higher quality lower quality
Good qualitative  Results are  Results are
results and appropriate and inappropriate or
interpretations complete incomplete
are reported  Rigorous qualitative  Poor qualitative
procedures are used procedures are used
Study used a  All elements of the  There are
rigorous research study fit together in a inconsistencies in how
design logical, coherent way the study fits together
Research design  Findings provide a rich  Findings provide a
addressed exploration that fulfills superficial
study’s purpose the study’s intent examination of the
study’s intent
How do you identify the participants and data collection in a qualitative study?

– Typically found in the Method section

– Look for information about the sites and
participants
– Identify the types of qualitative data
gathered
– Discern the procedures used to gather
the data
– Note the issues related to collecting
data
How are the sites and participants selected in a qualitative study?

– Sites and participants are purposefully selected

• Called purposeful sampling (or purposive sampling)
– Researcher is able to select the individuals that are
most appropriate for learning about the central
phenomenon
– Specific strategies guide the purposeful sampling
• Maximal variation sampling used to describe diverse
perspectives
• Homogenous sampling used to describe a subgroup
• Snowball sampling used to identify individuals not known
to the researcher
How many sites or participants are selected for a qualitative study?

– May be a single individual or a single site

– May range from 2 – 3 individuals or sites to 30 or 40
– May use saturation (no longer learning new
information) to determine sample size
– General guidelines for the four common designs:
• Narrative: one or two individuals
• Case Study: one case
• Multiple Case Study: two to five cases
• Ethnography: one entire group
• Grounded Theory: Approximately 20 to 30 individuals (or
until saturation is reached)
What types of qualitative data do researchers collect?

– Interviews
– Observations
– Documents
– Audiovisual materials
 What are the procedures for qualitative interviews?

– A researcher asks one or more participants open-

ended questions and records their answers
– An open-ended question is a question stated so
that it allows the participant to create his/her own
options for responding
– Pay attention to:
• The reason why the researcher decided to use interviews
• The type of interviews conducted
• What questions were asked and whether probes were
used
• How the interviewer recorded the conversation
 What are the different types of qualitative interviews?

– Focus group interviews

– One-on-one interviews
– Telephone interviews
– E-mail interviews
– Open-ended questions as part of a
questionnaire
 What does the interview protocol contain?

– Instructions for the process of the interview

– A place to record basic information
• Who the interviewer and interviewee(s) were
• Where the interview took place
• How long the interview lasted
– An “ice breaker” general question to help the
participants become at ease
– A few general “semi-structured” questions, which
are stated in a broad, open-ended fashion
– Space to take notes about the responses from the
interviewee
 How do researchers record data during the interview?

• Field notes
– Note about the interview conversation
during the interview and immediately
after the interview
• Audio recordings
– Standard or digital
What is important to note about the procedures for qualitative observations?

– Why the researcher decided to gather observations

– The researcher’s role during the observation
• Nonparticipant observational role
• Participant observational role
• Changing observational role
– How the observer recorded the data
• Descriptive and reflective field notes
– How many observations were gathered
• Single observation or multiple observations over time
– What specific types of documents (public and
private records) were collected
What is important to note regarding the collection of audiovisual materials?

– Audiovisual materials consist of photographs,

videos, digital images, paintings, sounds, or
objects that help researchers understand the
central phenomenon under study
– Were the forms created for the purposes of
the research or some other purpose
– How the forms provide insight into the central
phenomenon
How do researchers use photos in collecting
qualitative data?

– Photo elicitation
• The researchers show participants pictures (their
own personal photographs, photos taken by the
researcher, or historical photographs) and ask
them to discuss the contents of the images
– PhotoVoice
• Participants take and discuss photos as a means
to self-understanding and empowerment
What ethical issues are reported about qualitative data collection?

– Institutional Review Boards (IRBs)

– Ethical codes of conduct
• American Educational Research Association
– Ethical Standards of the American Educational
Research Association, Strike et al., 2002
• American Psychological Association
– Ethical Principles of Psychologists and Code of
Conduct, 2003
– Procedures that respected participants’ rights
– Steps to protect participants’ anonymity
What are the challenges in gaining access
and gathering data?
– Often need to seek and obtain permissions from
individuals and sites at many levels
– May use a gatekeeper, a person who
• Assists in the identification of a place to study
• Has an official or unofficial role at the site
• Provides entrance to a site
• Helps researchers locate people at the site
– Successfully recruiting participants
– Engaging participants with the study’s central
phenomenon at an in-depth level
 How do you evaluate participants and data
collection in a qualitative study?
Indicators of Indicators of
Quality criteria
higher quality lower quality
The sampling  Purposeful sampling  Sampling strategy is
strategy is used convenient or random
appropriate and  Sampling procedures  Sampling procedures
justified are fully described are unclear
 Strong reasons justify  Weak reasons do not
sampling strategy fully justify strategy
The sample  Sufficiently small to  Large sample size
size is capture rich detail limits detail
appropriate and  Consistent with  Not consistent with
justified research design research design
 Sound rationale  Weak rationale does
justifies sample size not justify sample size
 How do you evaluate participants and data
collection in a qualitative study?
Indicators of Indicators of
Quality criteria
higher quality lower quality
The data types  Data types are suited  Data types are poorly
are appropriate for learning about the suited for learning
central phenomenon about the central
phenomenon
The data are  Open-ended  Closed-ended
gathered using procedures gain procedures lead to
rigorous thorough superficial
qualitative understanding understanding
procedures  Accurate record such  No accurate, detailed
as an audiorecording record, only
or detailed field notes summaries of
information
 How do you evaluate participants and data
collection in a qualitative study?
Indicators of Indicators of
Quality criteria
higher quality lower quality
Data collection  Appropriate  Permissions were not
issues are permissions were secured
handled secured  Procedures
ethically and  Respectful treatment safeguarding
thoughtfully of participants is participants are
demonstrated questionable
 Quality interactions  Poor interactions
between researchers between researchers
and participants and the participants
 How do you evaluate participants and data
collection in a qualitative study?
Indicators of Indicators of
Quality criteria
higher quality lower quality
The selected  Sites and participants  Sites and participants
participants are fit study’s purpose do not fit study’s
information rich  Participants are purpose
clearly able to provide  Participants provide
rich information about limited information
central phenomenon about central
phenomenon
The database  Database is extensive  Database is limited
provides  Includes detailed  Includes superficial
extensive and information related to information which may
credible central phenomenon not relate to central
information phenomenon
The End