Critically Evaluating the Strengths and Weaknesses of a Study

FUNDAMENTALS
1. Describe the main reason for the study.
Normally the topic sentence

2. What type of study is it? (pick one)

a. Experiment (assigned treatments)
b. Observational (observed, no treatments)

3. Identify the Explanatory and Response Variables

a. Label each as quantitative (numerical) or qualitative (categorical)

numerical = a number (anything that can have decimals or it makes sense to take an average of)

categorical = groups (groupings that do not have an order to them (colors, majors, etc…))

Explanatory:
Treatments or the thing they think is responsible for the change
Response:
What is being measured in the study

SUBJECTS
4. Describe the population (who can the results be extended out to?)

Look for inclusion and exclusion criteria.

5. What type of sampling was used?

A lot of volunteer sampling occurs, especially with medical studies. Occasionally there is more than one
scheme in a study.

6. Are there any potential sources of bias in the sample?

Think about groups that were excluded either accidentally or on purpose. Hint: Volunteer samples are
ALWAYS biased.

DATA COLLECTION
7. Was the study prospective or retrospective?
Prospective = future – got the participants and then measured the data (record what you’ll eat for
breakfast tomorrow)
Retrospective = past – data point already occurred then put the participant in the study (what did
you eat for breakfast yesterday)

8. Were the subjects randomized to treatments? If so, how?

Only applies if it is an experiment. Look for key words like blocks or simple random sample

9. What results are presented? (only list a few for each category)
a. Descriptive Statistics
Summaries of the data including: means, medians, standard deviations, ranges, proportions

b. Inferential Statistics

Hypothesis tests, “p-value”, confidence intervals, chi-square, z-test, proportion test, ANOVA.

10. Are there any Confounding or lurking variables that could impact the results?
Confounding = measured in the study. Different level for treatment groups, impacts results
Lurking variable = not measured in the study. Different level for treatment groups, impacts results.
Favorite example is socioeconomic status. Impacts where you live, what resources you have
available, and general lifestyle.

11. What were the final conclusions?

Based on the explanatory and response variables, what are the findings? Did it works, under which
conditions?

REFLECTION
12. What aspects of the study could be improved or included for next time?

Opinion question. Often discussed by the author in the limitations section of the paper.