Chapter 5

Critically Appraising Quantitative

Evidence for Clinical Decision
Making

Copyright © 2014 Wolters Kluwer Health | Lippincott Williams & Wilkins

 Four Phases to Critical Appraisal of
Quantitative Studies
1. Rapid Critical Appraisal (RCA)
Goal: identify “keeper studies” (studies that
should be retained)
2. Evaluation
Goal: extract data from individual studies to
establish agreement across studies
3. Synthesis
Goal: pull data together from studies within the
body of evidence to paint the picture of what
the body of evidence tells us
4. Recommendation
Goal: “next steps” for practice, based on the
evidence, are clearly articulated
Copyright © 2019 Wolters Kluwer • All Rights Reserved
 Getting Started: General Appraisal
Overview (GAO)
General Appraisal Overview (GAO) is a working
form that allows clinicians the opportunity to
document their thinking about a study
A GAO provides an overview to get appraisers to
start thinking about a study to be appraised
A GAO is not Rapid Critical Appraisal (RCA) ; it
is a preliminary step to RCA
Rapid Critical Appraisal includes:
o Levels of evidence
o Detailed questions that are good markers of
quality research design

Copyright © 2019 Wolters Kluwer • All Rights Reserved

 General Appraisal Overview (GAO)
Elements and Questions #1
Overview of the study
1. Why was the study done (purpose)?
2. What was the study design?
3. Was appropriate ethics review reported?
General description of the study
1. Research question or hypothesis
2. Sampling
• What was the sampling technique?
• What was the sample size?
Copyright © 2019 Wolters Kluwer • All Rights Reserved
 General Appraisal Overview (GAO)
Elements and Questions #2

Major variables studied

1. What were the dependent and independent
variables?
2. Are the measurements of the major variables
valid and reliable?
3. How were the data analyzed? Were the
statistics appropriate?
4. Were there any untoward events during the
conduct of the study?
5. How do the results fit with previous research?
6. What does this mean for clinical practice?
Copyright © 2019 Wolters Kluwer • All Rights Reserved
 Question #1

The statement that best describes GAO is:

A. GAO stands for General Appraisal Ordering
B. GAO and critical appraisal are different names for
the same thing
C. GAO is a working form that allows clinicians the
opportunity to document their thinking about a
study
D. Appraisers do a GAO after they are done doing
their critical appraisal

Copyright © 2019 Wolters Kluwer • All Rights Reserved

 Answer to Question #1

C. GAO is a working form that allows clinicians the

opportunity to document their thinking about a
study
Rationale: GAO, or General Appraisal Overview, is a
working form that allows clinicians the opportunity to
document their thinking. It is different from critical
appraisal, where a study’s level of evidence is
identified and detailed questions that are good
markers of quality are explored. GAO is done prior to
critical appraisal.

Copyright © 2019 Wolters Kluwer • All Rights Reserved

 Rapid Critical Appraisal of Quantitative
Studies
 Critical appraisal of quantitative studies encompasses
identifying a study’s place on levels of evidence hierarchy and
answers three broad questions:
o Are the results of the study valid? (Validity)
o What are the results? (Reliability)
o Will the results help me in caring for my patients?
(Applicability)
 Levels of Evidence hierarchies provide guidance about the
types of research studies, if well done, that are more likely to
provide reliable answers to clinical questions.
 There are a variety of different hierarchies of evidence

Copyright © 2019 Wolters Kluwer • All Rights Reserved

 Hierarchy of Evidence for Intervention
Questions

 Figure 5.2: Hierarchy of evidence for intervention questions. EBP,

evidence-based practice; QI, quality improvement.

Copyright © 2019 Wolters Kluwer • All Rights Reserved

Critical Appraisal of Quantitative Studies:
Validity, Reliability, and Applicability #1

Validity is whether or not results of a study were

obtained by sound scientific methods.
Key questions:
o Can the results of the study be “believed”?
(Internal validity)
o Can the results of the study be generalized to
other settings/patient populations? (External
validity)
o Study bias and/or confounding study results
can impact study validity.

Copyright © 2019 Wolters Kluwer • All Rights Reserved

 Study Validity vs. Instrument Validity

Study validity refers to whether the study results

can be “believed” (internal validity) and whether the
results can be generalized to other settings/patient
populations (external validity).
Instrument validity refers to whether the
instruments are measuring what they are supposed
to be measuring.

Copyright © 2019 Wolters Kluwer • All Rights Reserved

 Validity: Types of Study Bias

1. Selection bias
2. Knowledge of who receives the intervention
3. Gatekeeping
4. Measurement bias
5. Recall bias
6. Information bias
7. Loss to follow up
8. Contamination

Copyright © 2019 Wolters Kluwer • All Rights Reserved

 Validity: Confounding Study Results

 Confounding variables are those factors that

interfere with the relationship between the
independent and dependent variables.
 Most often encountered in studies about lifestyle
and health
 Example: In a study examining the association
between nurses’ intake of caffeine and fasting-
related headaches, shift work would be a
possible confounding variable
 In longitudinal studies, may be a result of
history or maturation

Copyright © 2019 Wolters Kluwer • All Rights Reserved

 Question #2

The question that does not address study validity is:

A. How big is the magnitude of the effect?
B. Were the study results obtained by sound scientific
methods?
C. Was there any biases that influenced study results?
D. Were there any confounding factors that influenced
study results?

Copyright © 2019 Wolters Kluwer • All Rights Reserved

 Answer to Question #2

 A. How big is the magnitude of the effect?

Rationale: Magnitude of effect is one question asked
to determine reliability of a study, not its validity.
Validity is whether or not the study results obtained
by sound scientific methods. Bias and confounding
factors influence a study’s validity.

Copyright © 2019 Wolters Kluwer • All Rights Reserved

Critical Appraisal of Quantitative Studies:
Validity, Reliability, and Applicability #2
Reliability is whether or not the effects of a study
have sufficient influence on practice, clinically and
statistically; that is, the results can be counted on to
make a difference when clinicians apply them to
their practice.
Key questions:
o Do the numbers add up?
o What is the magnitude of the effect?
o How precise was the effect estimated?
o What is the strength of the association?

Copyright © 2019 Wolters Kluwer • All Rights Reserved

Study Reliability vs. Instrument Reliability

Study reliability refers to the “repeatability” of the

study—that is, if the study were repeated under the
same circumstances, would the results be the same?
Instrument reliability refers to the consistency of the
instruments—will they yield the same results under
the same conditions?

Copyright © 2019 Wolters Kluwer • All Rights Reserved

Reliability: Key Questions and their
Associated Statistics of Interest #1
 Do the numbers add up?
o Total number (N)
 What is the magnitude of the effect?
o The effect is the rate of occurrence in each of the
groups for the outcome of interest.
o Magnitude of the effect is the important
differences between groups.
 How precise was the effect estimated?
o p values (statistical significance)
o Clinical significance
o Confidence intervals
Copyright © 2019 Wolters Kluwer • All Rights Reserved
 Reliability: Key Questions and the
Associated Statistics of Interest #2

What is the strength of the association?

o Absolute Risk Reduction (ARR)
o Relative Risk Reduction(RRR)
o Odds Ratio (OR)
o Number needed to treat (NNT)
o Number needed to harm (NNH)

Copyright © 2019 Wolters Kluwer • All Rights Reserved

 Statistical vs. Clinical Significance
Statistical significance is the result of statistical
analysis of data are unlikely to have been caused by
chance, at a predetermined level of probability.
o Represented by p value
o Statistical significance is when p < .05
o Statistical significance is influenced by sample
size
Clinical significance is defined as study findings
that will directly influence clinical practice, whether
they are statistically significant or not
o Not influenced by sample size
Copyright © 2019 Wolters Kluwer • All Rights Reserved
 Question #3

The question that best addresses study reliability is:

A. How big is the magnitude of the effect?
B. Did the investigator(s) applying the intervention
know to which groups subjects were assigned?
C. Were the instruments calibrated according to
manufacturer specifications?
D. How many subjects were lost to follow-up and
what groups were they in?

Copyright © 2019 Wolters Kluwer • All Rights Reserved

 Answer to Question #3

A. How big is the magnitude of the effect?

Rationale: Magnitude of effect allows for
determination of the certitude of results in similar
patient populations (generalizability). Answers “B” and
“D” are questions related to the validity of the study.
Answer “c” is related to instrument reliability, not
study reliability.

Copyright © 2019 Wolters Kluwer • All Rights Reserved

Critical Appraisal of Quantitative Studies,
Validity, Reliability, and Applicability #3

Applicability refers to whether or not the results

from a study are appropriate for a particular patient
population.
This is the third key question in critical appraisal.

Copyright © 2019 Wolters Kluwer • All Rights Reserved

 Types of Quantitative Studies
Systematic reviews
Randomized controlled trials
Cohort studies
Case studies
Case control studies
NOTE: Each type of study has its own unique
nuances when critically appraising them.

Copyright © 2019 Wolters Kluwer • All Rights Reserved

Evaluation and Synthesis of Quantitative
Studies #1

 Evaluation is an evaluation of worth. This is a table that

helps one organize key points from each piece of
literature included in the body of evidence.
o Goal: capture the data from each piece of literature
in order to:
Ensure there is no need to search through the
volume of articles in a body of evidence more
than once
Establish patterns across studies
 Contains information about each individual study
 Each column on the evaluation table holds a different key
piece of literature about the articles included.
Copyright © 2019 Wolters Kluwer • All Rights Reserved
 Sample Topics on Evaluation Tables
 Citation
 Purpose of study
 Conceptual framework
 Study design and method
 Sample and settings
 Major variables studied and definitions
 Measurement of major variables
 Data analysis
 Study findings
 Worth to practice
Copyright © 2019 Wolters Kluwer • All Rights Reserved
Evaluation and Synthesis of Quantitative
Studies #2
Synthesis is the process of putting together parts
to make a whole (i.e. putting together the results of
several studies to tell the story of a body of
evidence)
o Goal: To formulate and communicate the
information that is essential for
comparison across studies
Information from synthesis is often organized on a
synthesis table, creating a visual depiction of
comparisons across studies.
Synthesis tables contain only those aspects of the
individual studies that are common or unique across
studies (i.e. levels of evidence and outcomes).
Copyright © 2019 Wolters Kluwer • All Rights Reserved
 Question #4

Is the following statement true or false?

The main difference between an evaluation table
and a synthesis table is that evaluation tables are
used to compare and contrast studies that are
included in a systematic review.

Copyright © 2019 Wolters Kluwer • All Rights Reserved

 Answer to Question #4

False
Rationale: Evaluation tables are tools which serve to
organize the body of evidence and identify patterns
within the studies. Synthesis tables put together the
results of several studies to tell the story of a body of
evidence. They contain only those aspects of the
individual studies that are common or unique across
studies (i.e. levels of evidence and outcomes).

Copyright © 2019 Wolters Kluwer • All Rights Reserved