Clinical Microbiology and Infection 24 (2018) 1115e1116
Contents lists available at ScienceDirect
Clinical Microbiology and Infection
journal homepage: www.clinicalmicrobiologyandinfection.com
Editorial note
Systematic reviews of diagnostic test accuracy in CMI
Over the years, systematic reviews have become increasingly
important as they provide physicians with an up-to-date overview
of evidence on specific topics, identify knowledge gaps, and are
regarded as the highest level of evidence in clinical guidelines.
Diagnostic systematic reviews are a comprehensive summary of
the literature about the accuracy of a diagnostic test, including
outcome measures such as sensitivity, specificity, predictive values,
receiver operating characteristic (ROC) curves or likelihood ratios.
In the field of microbiology and infection, these reviews may
include a wide variety of tests, often applied close to the bedside
and with short turnaround times. The reference standard is often
absent or imperfect, because ensuring that there is no agent at all
is nearly impossible. Furthermore, infectious diseases pose the
highest burden in countries with the least resources. This requires
attention to less resource intensive techniques and to population-
specific effects.
Clinical Microbiology and Infection (CMI) aims to publish diag-
nostic accuracy systematic reviews that are well-performed and
that address a clinically relevant question, regardless of the
outcome [1,2]. However, not all reviews meet the current quality
standards: they lack critical steps in the review process or do not
report on important items. An important omission is the lack of a
clear explanation of the potential consequences of testing on subse-
quent patient managementdinformation that is required to under-
stand the potential impact of testing for the patient.
An important requirement for publication is completeness of
reporting. An extension of the PRISMA statement has recently
been developed and published to address the specific study charac-
teristics of diagnostic accuracy reviews: PRISMA-DTA [3], and
PRISMA-DTA for abstracts (both accessible on http://www.
prisma-statement.org). The PRISMA-DTA checklist provides an
overview of all required items to be reported in the final manu-
script, but it is also a useful tool in the early preparation stage of
the review as it may guide its design and execution. The PRISMA-
DTA checklist will be required as a Supplementary table for system-
atic reviews of diagnostic accuracy. Preferably, to establish the val-
idity of the review and ensure reproducibility, the protocol should
be registered at PROSPERO (https://www.crd.york.ac.uk/prospero),
the online database of registered systematic reviews.
This Editorial Note explains the expectations of CMI with regard
to systematic reviews of diagnostic test accuracy and encourages
the use of the Cochrane Handbook for systematic reviews on diag-
nostic accuracy (http://methods.cochrane.org/sdt/handbook-dta-
reviews) and the use of the Preferred Reporting Items for System-
atic Reviews and Meta-Analyses (PRISMA) of Diagnostic Test Accu-
racy (DTA) guideline for reporting.
https://doi.org/10.1016/j.cmi.2018.05.022
1198-743X/© 2018 European Society of Clinical Microbiology and Infectious Diseases. Published by Elsevier Ltd. All rights reserved.
Steps in a systematic review of diagnostic test accuracy
Similar to any other systematic review, a systematic review of
diagnostic test accuracy starts with formulating a relevant and
clearly defined research question. This should be a clinically rele-
vant question, including the participants, index tests and target
condition. Too often we see systematic reviews submitted that
only consider one particular biomarker, without any further infor-
mation of potential alternative tests or the (potential) role of the
test in the clinical pathway. We would like to ask authors to
consider explicitly comparing multiple tests in the same review,
providing that these tests are each other's competitors in clinical
practice.
In the case of a comparative question, the ideal study design to
include would be an accuracy study comparing the accuracy of the
tests of interest. This would mean that all participants in the study
undergo (all) the index tests and the reference standard. After iden-
tification of citations, two authors should independently perform
screening of titles, abstracts and subsequently full-text articles to
minimize the number of incorrect inclusions and exclusions. Simi-
larly, data extraction should be performed in duplicate and inde-
pendently using piloted forms.
An important step in performing a systematic review is risk of
bias assessment, as the overall judgement of the quality of the
included studies influences the interpretation of the results and
contributes to formulating a well-balanced final conclusion. The
Quality Assessment of Diagnostic Accuracy Studies 2 (QUADAS-2)
is the recommended tool for quality assessment and evaluates
four domains: (1) participant selection, (2) index test, (3) reference
standard and (4) flow and timing [4]. Risk of bias, which refers to
limitations or flaws in the design or conduct of a study, is assessed
for all domains, and concerns regarding applicability, referring to
differences between the clinical features of the included study
and the review question, are assessed for the first three domains.
Signalling questions can be adjusted or omitted as appropriate,
but these adjustments should be reported. Quality assessment of
the included studies should preferably be performed by at least
two independent authors and should be reported as ‘low’ or
‘high’ risk of bias, and as ‘low concerns regarding applicability’ or
as having ‘concerns regarding applicability’. Quality scores are not
advised, as there is no direct link between the amount of bias
and any quality score [3]. Sensitivity analyses exploring whether
diagnostic accuracy estimates vary in studies regarded as having
low or high quality are highly recommended.
The data synthesis should be clearly described, including the
handling of multiple thresholds of test positivity, or multiple test
1116 Editorial note / Clinical Microbiology and Infection 24 (2018) 1115e1116
readers or reference standards, and the handling of indeterminate
test results and comparison of different tests. Systematic reviews
may or may not contain one or more meta-analyses. Several statis-
tical packages and models are available for performing a meta-
analysis, each one more advanced and often more accurate than
the other. For example, many authors use the software package
META-DISC, which is a useful package for data description, but for
the meta-analysis it depends on an outdated method, which does
not appropriately take into account the correlation and covariance
between sensitivity and specificity. The most widely advocated
methods are the so-called bivariate model, or Reitsma model, and
the Hierarchical summary Receiver Operating Characteristic
(HSROC) model, or Rutter and Gatsonis model [5]. The first one is
often easier to understand and to implement, for example in STATA
software, and it provides a summary estimate of sensitivity and
specificity. The HSROC model provides summary estimates for the
parameters of the summary ROC curve and assumes an inherent
threshold effect, meaning that the test has different thresholds
and that sensitivity and specificity change according to this
threshold. Bayesian methods and methods accommodating multi-
ple tests and imperfect reference standards are also available [6],
but require high levels of statistical knowledge and experience.
For each included study we expect description of the following
items: key characteristics of the study, including participant char-
acteristics, target condition, and index and reference test used;
the evaluation of risk of bias and concerns regarding applicability;
for each analysis in each study, 2  2 data should be reported (true
positives, false positives, true negatives and false negatives), and
estimates of diagnostic accuracy and confidence intervals, ideally
with a corresponding ROC or forest plot. If a meta-analysis was per-
formed, the summary estimates should be presented with a corre-
sponding confidence interval, ideally with a corresponding
summary ROC plot or point estimate plus confidence region.
One of the greatest challenges of systematic reviews is the inter-
pretation of the results in the discussion and conclusion. Although
systematic reviews of diagnostic test accuracy may provide infor-
mation about the average percentage of false-negative and false-
positive results, it is important to realize that they do not provide
direct evidence for the usefulness or effectiveness of the test. For
that, we need additional information about the subsequent man-
agement steps and potential consequences of those steps, included
in the Discussion section. Furthermore, the methodological quality
of the included studies and the limitations of the review process
should be explained.
Conclusion
Designing and writing a systematic diagnostic accuracy review
requires an explanation of how the accuracy of a test will inform
relevant clinical decision making. As test accuracy varies between
health-care settings and tested populations, a clear description of
the target population is necessary. Furthermore, authors are
advised to use PRISMA-DTA as a writing guideline, to be used
from the start of the study, rather than as a checklist afterwards.
We look forward to publishing the resulting high-quality system-
atic reviews of diagnostic test accuracy.
Editorial note: not peer-reviewed.
References
[1] Huang H-S, Tsai C-L, Chang J, Hsu T-C, Lin S, Lee C-C. Multiplex PCR system for
the rapid diagnosis of respiratory virus infection: systematic review and meta-
analysis. Clin Microbiol Infect 2017. https://doi.org/10.1016/j.cmi.2017.11.018
[epub ahead of print].
[2] Eguchi H, Horita N, Ushio R, Kato I, Nakajima Y, Ota E, et al. Diagnostic test
accuracy of antigenaemia assay for PCR-proven cytomegalovirus infection-
systematic review and meta-analysis. Clin Microbiol Infect 2017;23:
907e15.
[3] McInnes MDF, Moher D, Thombs BD, McGrath TA, Bossuyt PM, Clifford T, et al.
Preferred reporting items for a systematic review and meta-analysis of diag-
nostic test accuracy studies. JAMA 2018;319:388e96.
[4] Whiting PF, Rutjes AWS, Westwood ME, Mallett S, Deeks JJ, Reitsma JB, et al.
QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy
studies. Ann Intern Med 2011;155:529e36.
[5] Macaskill P, Gatsonis C, Deeks J, Harbord R, Takwoingi Y. Chapter 10 Analy-
sing and presenting results. In: Deeks JJ, Bossuyt PM, Gatsonis C, editors.
Cochrane handbook for systematic reviews of diagnostic test accuracy. The
Cochrane Collaboration; 2010. Available from: Version 1.0. http://srdta.
cochrane.org/.
[6] Menten J, Boelaert M, Lesaffre E. Bayesian meta-analysis of diagnostic tests
allowing for imperfect reference standards. Stat Med 2013;32:5398e413.
M.M.G. Leeflang*
Department of Clinical Epidemiology, Biostatistics and Bioinformatics,
Amsterdam Public Health, Academic Medical Centre,
University of Amsterdam, Amsterdam, The Netherlands
N. Kraaijpoel
Department of Vascular Medicine, Academic Medical Centre,
Amsterdam, The Netherlands
E-mail address: n.kraaijpoel@amc.nl.
*
Corresponding author. M.M.G. Leeflang.
E-mail address: m.m.leeflang@amc.uva.nl (M.M.G. Leeflang).
29 May 2018
Available online 7 June 2018
Editor: L Leibovici