[ evidence in practice ]

STEVEN J. KAMPER, PhD1

Risk of Bias and Study Quality

Assessment: Linking Evidence to Practice
Downloaded from www.jospt.org at University of Sydney on May 3, 2020. For personal use only. No other uses without permission.

J Orthop Sports Phys Ther 2020;50(5):277-279. doi:10.2519/jospt.2020.0702

E
vidence-based practice requires use of the best available evidence, ment for confounders are critical for prog-
which implies that some evidence is better than other evidence. nostic studies, but not for randomized
Better evidence comes from research findings that are at lower clinical trials (RCTs). Intention-to-treat
analysis is an important feature of RCTs,
risk of bias. Recalling an earlier Evidence in Practice article,2 bias
but is not relevant to diagnostic studies.
means that the results of the study do not reflect the true effect of an The tool used to measure study quality or
intervention. A biased study can overes- • Checklist: a set of criteria that iden- risk of bias must match the study design.
Copyright © 2020 Journal of Orthopaedic & Sports Physical Therapy®. All rights reserved.

timate (or underestimate) the size of an
effect or association.
Sorting the good from the bad when
it comes to research evidence requires
assessment of the methods and results
of the study. There are several tools
available to help the reader assess study
quality or risk of bias. Study quality and
risk of bias are overlapping but subtly
Journal of Orthopaedic & Sports Physical Therapy®
tify study features that indicate study
quality or risk of bias. Checklists
should not be summed to produce a
total score.
• Scale: a set of criteria that are summed
to produce a total score.
• Domain-based tool: a list of types of
bias (domains); assessors make judg-
ments about the risk of each type of
Measurement of study quality is best
developed for treatment-effectiveness
studies. The most common tools are the
Physiotherapy Evidence Database (PE-
Dro) scale and the Cochrane risk of bias
tool. The former is commonly used as a
scale and the latter is a domain-based
tool. The PEDro scale includes a list of
study features that are marked as pres-

different concepts. Risk of bias is how
likely it is that the study findings are bi-
ased, whereas study quality is a broader
concept that includes risk of bias among
other features, such as generalizability4
and appropriate reporting. The items
included in risk of bias assessment tools
are more narrowly focused, whereas
those in study quality tools cover more
ground.
Study quality and risk of bias assess-
ment tools both aim to help the reader
assess how likely it is that the results of a
study are true, and, therefore, the extent
to which the study should inform clinical
decisions.
Measuring Risk of Bias and Study Quality
Researchers have developed checklists,
scales, and domain-based tools that iden-
tify indicators (criteria) of study quality
and potential bias.
bias.
The principles underlying measure-
ment of study quality and risk of bias
are no different from those measuring
clinical outcomes.3,5 When using any tool
to assess study quality or risk of bias,
we are most concerned about reliability
(whether different people have the same
scores on the scale/checklist) and validity
(whether the tool actually measures risk
of bias). Not all tools have been tested for
reliability and validity. Place more faith
in established and well-known tools to
guide your assessment.
Assessment Tools
Different study quality and risk of bias
tools apply to different study designs,
because certain types of bias may or may
not be relevant depending on the research
question. For example, identification of,
measurement of, and statistical adjust-
ent, absent, or unclear, and the number
of “present” items is summed to represent
the study quality score. The Cochrane risk
of bias tool includes a list of types of bias
(domains), and the rater judges whether
the study is at risk (high, low, or unclear)
of bias. The PEDro scale and Cochrane
risk of bias tool identify many of the same
study features: both include items related
to randomization, concealed allocation,
blinding, and appropriate reporting of
results. While both tools were developed
using robust methods,1,6,7 there is no
strong evidence that either is more reli-
able and valid than the other.
While numerous assessment tools are
available for most study types, they gen-
erally include similar items, and there
is typically no consensus as to which is
preferred. This is because the most im-
portant features that denote study qual-
ity and lead to bias are well recognized.

School of Public Health, University of Sydney, Camperdown, Australia; Centre for Pain, Health and Lifestyle, Australia. t Copyright ©2020 Journal of Orthopaedic & Sports Physical
1

Therapy®

journal of orthopaedic & sports physical therapy | volume 50 | number 5 | may 2020 | 277

Some well-accepted tools for study types
include the Quality In Prognostic Studies
(QUIPS) tool for observational studies in-
vestigating prognostic factors, the Qual-
ity Assessment of Diagnostic Accuracy
Studies-revised (QUADAS-2) for diagno-
sis studies, and A MeaSurement Tool to
Assess systematic Reviews-revised (AM-
STAR-2) for systematic reviews (TABLE).
Downloaded from www.jospt.org at University of Sydney on May 3, 2020. For personal use only. No other uses without permission.
Reporting Guidelines
Cousins of study quality assessment
tools are reporting guidelines, although
TABLE
Study Type Cochrane Risk of Bias Tool
Copyright © 2020 Journal of Orthopaedic & Sports Physical Therapy®. All rights reserved.
RCTs • Selection bias
• Attrition bias
• Performance bias
• Detection bias
• Reporting bias
• Other bias
Journal of Orthopaedic & Sports Physical Therapy®
Prognosis
Diagnostic
tests
Systematic
reviews
Abbreviations: AMSTAR-2, A MeaSurement Tool to Assess systematic Reviews-revised; PEDro, Physiotherapy Evidence Database; QUADAS-2, Quality Assess-
ment of Diagnostic Accuracy Studies-revised; QUIPS, Quality In Prognostic Studies; RCT, randomized clinical trial.
278 | may 2020 | volume 50 | number 5 | journal of orthopaedic & sports physical therapy
[ evidence in practice ]
they are not designed to assess risk of There are specific guidelines for dif-
bias. Reporting guidelines help authors ferent study designs. The best known
ensure that their articles include suffi- include the CONsolidated Standards
cient information for readers to judge Of Reporting Trials (CONSORT) state-
the quality and applicability of their ment for RCTs, the STrengthening the
research. The items often overlap with Reporting of OBservational studies in
those in study quality checklists, but Epidemiology (STROBE) statement
they serve a different purpose. Because for observational studies, and the Pre-
reporting guidelines focus on what ap- ferred Reporting Items for Systematic
pears in the manuscript (rather than Reviews and Meta-Analyses (PRISMA)
the design of the study), they are often statement for systematic reviews. The
also used by journal editors to improve Enhancing the QUAlity and Transpar-
research articles during peer review. ency Of health Research (EQUATOR)
Items in Study Quality Tools
PEDro Scale QUIPS QUADAS-2 AMSTAR-2
• Eligibility
• Randomization
• Concealed allocation
• Group comparability
• Participant blinding
• Therapist blinding
• Assessor blinding
• Intention-to-treat analysis
• Loss to follow-up
• Statistical comparisons
• Point and variability measures
• Participation
• Attrition
• Prognostic factor measurement
• Confounder measurement
• Outcome measurement
• Analysis and reporting
• Patient selection
• Index test
• Reference standard
• Flow and timing
• Research question
• A priori protocol
• Inclusion criteria
• Comprehensive search
• Justification of exclusions
• Duplicate study selection
• Duplicate data extraction
• Included study description
• Risk of bias assessment
• Impact of bias on interpretation
• Impact of bias on synthesis
• Appropriate meta-analysis
• Funding sources
• Heterogeneity
• Publication bias
• Conflicts of interest
 network (https://www.equator-network.
org/) keeps an up-to-date repository of
reporting guidelines.
Application
Sifting out the poor-quality research
and paying attention to the good is a re-
quirement of evidence-based practice.
The key to assessing quality is under-
Downloaded from www.jospt.org at University of Sydney on May 3, 2020. For personal use only. No other uses without permission.
of bias or study quality assessments of
the included studies.
Study quality and risk of bias assess-
ment tools can help the reader under-
stand how much confidence one should
place in the findings of a study. However,
the reader must select the right tool for
the job, and proper application requires
an understanding of the principles that
https://doi.org/10.2519/jospt.2018.0703
3. Kamper SJ. Fundamentals of measurement:
linking evidence to practice. J Orthop Sports Phys
Ther. 2019;49:114-115. https://doi.org/10.2519/
jospt.2019.0701
4. Kamper SJ. Generalizability: linking evidence to
practice. J Orthop Sports Phys Ther. 2020;50:45-
46. https://doi.org/10.2519/jospt.2020.0701
5. Kamper SJ. Reliability and validity: linking
evidence to practice. J Orthop Sports Phys
Ther. 2019;49:286-287. https://doi.org/10.2519/

standing how and to what extent bias
may impact study findings. Recognize
which types of bias are most relevant
to the study in question, judge the ex-
tent to which bias should impact confi-
dence in the study findings, and decide
when the risk of bias is so great that the
results should be ignored altogether.
Systematic reviews can be a useful re-
source, because most reviews report risk
Copyright © 2020 Journal of Orthopaedic & Sports Physical Therapy®. All rights reserved.
Journal of Orthopaedic & Sports Physical Therapy®
underpin the items in the tool.
REFERENCES
1. Higgins JP, Altman DG, Gøtzsche PC, et al. The
Cochrane Collaboration’s tool for assessing risk of
bias in randomised trials. BMJ. 2011;343:d5928.
https://doi.org/10.1136/bmj.d5928
2. Kamper SJ. Bias: linking evidence with practice.
J Orthop Sports Phys Ther. 2018;48:667-668.
jospt.2019.0702
6. Maher CG, Sherrington C, Herbert RD, Moseley
AM, Elkins M. Reliability of the PEDro scale
for rating quality of randomized controlled
trials. Phys Ther. 2003;83:713-721. https://doi.
org/10.1093/ptj/83.8.713
7. Verhagen AP, de Vet HC, de Bie RA, et al. The
Delphi list: a criteria list for quality assessment
of randomized clinical trials for conducting
systematic reviews developed by Delphi
consensus. J Clin Epidemiol. 1998;51:1235-1241.
https://doi.org/10.1016/s0895-4356(98)00131-0

BROWSE Collections of Articles on JOSPT’s Website

JOSPTs website (www.jospt.org) offers readers the opportunity to browse
published articles by Previous Issues with accompanying volume and issue
numbers, date of publication, and page range; the table of contents of the
Upcoming Issue; a list of available accepted Ahead of Print articles; and
a listing of Categories and their associated article collections by type
of article (Research Report, Case Report, etc).

Features further curates 3 primary JOSPT article collections:

Musculoskeletal Imaging, Clinical Practice Guidelines, and Perspectives
for Patients, and provides a directory of Special Reports published
by JOSPT.

journal of orthopaedic & sports physical therapy | volume 50 | number 5 | may 2020 | 279