IJID Regions 8 (2023) 157–163

Contents lists available at ScienceDirect

IJID Regions
journal homepage: www.elsevier.com/locate/ijregi

Comparison of statistical methods for the early detection of disease

outbreaks in small population settings
Adam T. Craig a,b,∗, Robert Neil F. Leong b, Mark W. Donoghoec, David Muscatello b,
Vio Jianu C. Mojica d, Christine Joy M. Octavo d
a
School of Public Health, The University of Queensland, Herston, Australia
b
School of Population Health, University of New South Wales, Sydney, Kensington, Australia
c
Mark Wainwright Analytical Centre, University of New South Wales, Sydney, Kensington, Australia
d
Department of Physical Sciences and Mathematics, University of the Philippines, Manila, Philippines

a r t i c l e i n f o a b s t r a c t

Keywords: Objectives: This study examines the performance of 6 aberration detection algorithms for the early detection of
Outbreak disease outbreaks in small population settings using syndrome-based early warning surveillance data collected by
Syndromic surveillance the Pacific Syndromic Surveillance System (PSSS). Although previous studies have proposed statistical methods
Performance
for detecting aberrations in larger datasets, there is limited knowledge about how these perform in the presence
Evaluation
of small numbers of background cases.
Simulation
Modelling Methods: To address this gap a simulation model was developed to test and compare the performance of the 6
Pacific Islands algorithms in detecting outbreaks of different magnitudes, durations, and case distributions.
Results: The study found that while the Early Aberration Reporting System–C1 algorithm developed by Hut-
wagner et al. outperformed others, no single approach provided reliable monitoring across all outbreak types.
Furthermore, aberration detection approaches could only detect very large and acute outbreaks with any relia-
bility.
Conclusion: The findings of this study suggest that algorithm-based approaches to outbreak signal detection
perform poorly when applied to settings with small numbers of background cases and should not be relied upon
in these contexts. This highlights the need for alternative approaches for accurate and timely outbreak detection
in small population settings, particularly those that are resource-constrained.

Introduction of cases of an event of interest during a surveillance reporting or obser-

Public health surveillance is the ongoing, systematic collection, anal-
ysis, and dissemination of health-related data to provide intelligence
that can be used to monitor and improve the health of populations.
Used correctly, surveillance outputs can raise risk awareness and guide
evidence-based decision-making [1]. One objective of surveillance for
communicable diseases is the rapid identification of outbreaks so that
control measures can be quickly implemented and the impact can be
minimized [1,2].
A commonly used method for the early detection of outbreaks is
syndromic surveillance [3]. Syndromic surveillance can be defined as a
public health surveillance approach where health staff monitor the signs
and symptoms of patients to identify patterns that may signal a poten-
tial outbreak [1]. Syndromic surveillance for outbreak detection has 2
essential statistical elements: (i) the estimation of an expected number
vation interval (e.g., latest day or week) and geographical location; and
(ii) the comparison of the expected estimate with the observed value
during the most recent reporting interval. A statistical method is used
to detect or signal an observed recent incidence of illness that is higher
than expected (an ’aberration’) and are epidemiologically important.
The main differences among statistical methods used lie in how the
expected value is calculated and how the statistical significance of the
difference between the expected and observed value is determined [4].
Several aberration detection methods have been proposed including sta-
tistical process control, change-point detection, and spatial statistics
methods [5–9]. Most research has been conducted on datasets drawn
from large populations. The literature concludes that no single algo-
rithm outperforms others in all contexts and that selecting a syndromic
algorithm requires the weighing up of complex factors – including sys-
tem objective, system design, data quality, and data stability.

∗
Corresponding author: Tel.: +61 (0)7 3346 4922.
E-mail address: adam.craig@uq.edu.au (A.T. Craig).

https://doi.org/10.1016/j.ijregi.2023.08.007
Received 29 May 2023; Received in revised form 10 August 2023; Accepted 12 August 2023
2772-7076/© 2023 The Author(s). Published by Elsevier Ltd on behalf of International Society for Infectious Diseases. This is an open access article under the CC
BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/)
A.T. Craig, R.N.F. Leong, M.W. Donoghoe et al.
With many small population states (including small island develop-
ing states) dependent on syndromic surveillance for the early detection
of outbreaks [3,10,11] there remains a need for performance evaluation
based on these contexts, especially in settings where background cases
are scarce to begin with. To address this, we performed a simulation
study that overlayed authentic baseline data with synthesized outbreak
scenarios. We then compared the performance of 6 aberration detection
algorithms. Simulation-based analysis methods offer the advantage of
allowing the performance of multiple algorithms to be tested under a
range of scenarios and with greater control over outbreak scenario pa-
rameters [4].
Methods
Setting
We conducted our analysis using syndromic surveillance data col-
lected from 8 small population Pacific Island States and Territories
(PICTs). The median estimated 2023 population of these PICTs was
163,400 (interquartile range: 93,950-401,650) [12]). Supplementary
file S1 provides population and syndromic surveillance system-related
information for each.
The PICTs cover one-third of the earth and are home to approxi-
mately 13 million people of which 9.5 million reside in Papua New
Guinea, with the remainder dispersed over thousands of islands and
atolls that make up the other 21 PICTs [13]. Rates of infectious diseases
are high in the PICTs, as are the number of new and re-emerging disease
threats [14]. The ability to detect outbreaks is often hampered by poor
health infrastructure, insufficient human resources, lack of advanced
diagnostic capacities, geographic isolation, and inadequate communi-
cation infrastructure. Recognizing the challenge of conducting early
warning outbreak surveillance in low-resource small population island
settings and heeding a call to strengthen outbreak early warning sys-
tems as a requirement of the International Health Regulations (2005),
in 2010 PICTs designed and implemented a region-wide early warning
syndromic surveillance system known as the Pacific Syndromic Surveil-
lance System (PSSS) [10]. The PSSS collects data on a weekly basis for
a limited number of syndromes from sentinel sites located across the
PICTs. Data are collected, reported, and analyzed weekly by national
surveillance officers using a simple threshold-based approach for signal
generation.
Algorithms tested
The algorithms tested include the approach used by the PSSS and 5
commonly applied options cited in the literature. These were as follows:
(1) The simple threshold approach used by the PSSS. The algorithm
compares a weekly case count value with the 90th percentile
of historical weekly values and generates a signal if is exceeded
[15]. We call this method the ’90th percentile approach.’
(2) A modified Poisson CUmulative SUMmation (CUSUM) method
proposed by Rogerson and Yamada [16] and designed for syn-
dromic surveillance systems with small counts. The test statistic
is calculated as St = max(0,St–1 + z – k) where k is a reference
value, t is the surveillance week, and z is the standardized score
based on the variance of the observed values. A signal is gener-
ated if St > h, where h is a threshold parameter. We set h and k to
the default values of 1 and 11, respectively.
(3–5) Three variations of the Early Aberration Reporting Systems
(EARS) – C1, C2, and C3 developed by Hutwagner et al [16] The
EARS-C1 algorithm signals if the weekly observed value exceeds
the mean of the previous 7 weekly reporting periods plus 3 SDs.
The EARS-C2 algorithm differs from C1 in that it uses a temporal
guard band of 2 reporting periods between the baseline and the
reporting period being evaluated. The EARS-C3 algorithm also
158
IJID Regions 8 (2023) 157–163
uses a 2-reporting period guard band; however, it calculates a
partial sum for the last 3 reporting periods of the positive devia-
tion of the current value from the mean.
(6) An exponentially weighted moving average (EWMA) approach
[17] using a moving 28-reporting period baseline (t), a two-
reporting period guard band and a smoothing constant (𝑎) of be-
tween 0 and 1. The test statistic is calculated as St = axt + (1-
a)St-1 . A signal is generated if St > h, where h is pre-defined false
positive rate close to 5% (i.e., Prob[St > h | no outbreak] = 0.05).
Reference datasets
We extracted national level aggregated data from the PSSS database
for the period January 08, 2011 to September 17, 2018. Data extracted
were for 4 core syndrome categories – diarrhea (DIARR), influenza-like
illness (ILI), acute fever and rash (AFR), and prolonged fever (PF). Across
all data, the mean and 95% confidence interval (CI) weekly counts for
ILI, DIARR, PF, and AFR syndromes were: ILI: 172.7 (CI: 165.0-180.4),
DIARR: 73.7 (CI: 70.6-76.9), PF: 17.7 (CI: 16.2-19.1), and AFR: 7.0 (CI:
5.8-8.2), respectively.
To address gaps in the reference data, we replaced reporting periods
where no data were recorded with the mean of the 2 preceding and 2
following periods. As we were not able to determine if, and if so when,
true outbreaks were reflected in the reference data (given the lack of
a ’gold standard’ record of outbreaks against which to compare), we
applied a robust filtering method devised by Fried et al. (2014) to pro-
duce what we term ’trimmed reference datasets’ [18]. Robust filtering
removed peaks in data while retaining data authenticity.
Simulations
We used R (The R Project for Statistical Computing v3.5.1,
https://www.r-project.org/) and the R packages surveillance [19], im-
puteTS [20], robfilter [18], purrr [21], dplyr [22], qcc [23], and forcats
[24] to create 16 unique scenarios representing ways that outbreaks
in the community may alter patient presentation counts to health fa-
cilities. Two parameters were varied to generate the scenarios: (i) the
total number of additional case presentations (magnitude) and (ii) the
temporal distribution of additional case presentations. Four arbitrary
magnitudes (× 1.5, × 2.0, × 2.5, and × 3.0) of increase in case presen-
tations above baselines and 4 signal duration/distribution pairs were
combined to produce outbreak epidemic curves resembling acute point
source, acute point source with tail (as is common for gastrointestinal
disease outbreaks), person-to-person outbreak (as is common for respi-
ratory disease outbreaks), sustained disease transmission patterns (as is
common for endemic diseases transmission events) (Supplementary file
S2). Syndrome-specific baselines for each PICT were estimated by cal-
culating the mean of the 52 reporting periods preceding the start of the
simulated outbreak. Supplementary file S3 describes what each of the
abovementioned R packages contributed to the model.
Implementing the model
We systematically superimposed each scenario onto each trimmed
reference dataset at 500 randomly selected start points. This pro-
duced 256,000 unique test datasets (4 syndrome categories × 8 coun-
tries × 16 scenarios × 500 runs) for analysis. We then applied the panel
of aberration detection algorithms and calculated performance metrics.
Performance metrics
To measure the performance of each algorithm, we computed the
following metrics:
• Probability of detection (POD): defined as the proportion of out-
breaks detected at any point during the outbreak’s duration. This
metric provides a measure of overall sensitivity.
A.T. Craig, R.N.F. Leong, M.W. Donoghoe et al.
• POD in the first week (POD1 ): defined as the proportion of outbreaks
detected during the first reporting period (regardless of an outbreak’s
duration).
• Positive predictive value (PPV): defined as the probability that a
surveillance signal truly indicates the presence of a simulated out-
break.
• False-positive rate (FPR): defined as the proportion of time an algo-
rithm generated a signal in the absence of a simulated outbreak. One
minus FPR provides a measure of the algorithm’s specificity.
• Timeliness (T-measure): defined as the proportion of outbreaks last-
ing more than one reporting period that were detected in the first
reporting period.
To allow comparability of algorithms, parameter thresholds for al-
gorithms 2-6 were calibrated to generate FPRs that were not signifi-
cantly different from the FPR of the 90th percentile approach, which
was around 5.4% (range: 5.1-5.8%). To derive 95% CIs, we used the Wil-
son score method [25] for POD, PPV, and POD1 , and the delta method
for T-measure [26].
Further, to assess the performance of the 90th percentile approach,
we computed 2 versions of extended receiver operating characteristic
(ROC) curves that account for timeliness in addition to analyzing sen-
sitivity and FPR, as proposed by Kleinman and Abrams (2006) [27].
The first version was a weighted ROC (WROC) that adapts a conven-
tional two-dimensional ROC to include T-measure. The summary area
under the WROC curve (AUC) was calculated using a linear trapezoidal
method from the DescTools package in R [28]. The second version was
a three-dimensional ROC curve, having the T-measure as an added di-
mension. The summary volume under the three-dimensional ROC curve
(VUC) was calculated using the Delaunay triangulation from the geom-
etry package in R [29].
Results
General
Across all PICTs, syndromes, and scenarios, EARS-C1 was the most
sensitive with a POD of 46.5% (CI: 46.3-46.7), a POD1 of 38.0% (CI:
37.8-38.2), and a PPV of 99.2% (CI: 99.2-99.3). The 90th percentile
approach and EWMA were the next most sensitive with POD, POD1,
and PPV of 24.4%, 19.5%, and 98.6% for EWMA and 29.7%, 24.1%,
and 98.9% for the 90th percentile approach, respectively. The CUSUM,
EARS-C2 and EARS-C3 all had poorer sensitivity measures (Figure 1).
All algorithms had high PPV results, indicating that surveillance signals
represent ’true’ outbreaks (Figure 1).
159
IJID Regions 8 (2023) 157–163
Figure 1. Comparative performance of 6 aber-
ration detection algorithms. CUSUM = cu-
mulative summation; EARS = early alert
and response system; EWMA = exponentially
weighted moving average; POD = probability
of detection; POD1 = POD in the first report-
ing period; PPV = positive predictive value;
T-measure = proportion of all detected multi-
reporting period outbreaks detected in the first
reporting period. Results are available in tabu-
lar form in Supplementary file S4.
Background case count
In general, we found that the probability of detecting an outbreak
was higher for the more common syndromes (Figure 2). For example,
the median POD across all algorithms tested for DIARR, which had a
mean weekly baseline count of 73.7 (CI: 70.6-76.9) cases per PICT, was
47.2% (CI: 46.7-47.6%), compared to only 11.7% (CI: 11.4-11.9) for PF,
which had a mean weekly baseline count of 17.7 (CI: 16.2-19.1).
Size of outbreak
We found that the probability of detecting an outbreak increased as
the size of the outbreak increased. For example, the POD for EARS-C1
ranged from as high as 46.7% (CI: 46.3-47.1%) for ’very large’ outbreaks
(i.e., those with 3 times the number of case presentations as baseline) to
as low as 28.4% (CI: 28.0-28.8%) when the outbreak was ‘small’ (i.e., an
additional half the number of case presentations as the baseline). While
there were large differences in performance between the methods tested,
overall detection was mediocre (Figure 2).
Timeliness
As the timeliness of outbreak detection is a key measure of perfor-
mance, we calculated the proportion of outbreaks detected in the first re-
porting period (the POD1 ), and the proportion of all detected outbreaks
that lasted for more than one reporting period that were detected in the
first reporting period (T-measure).
By the POD1 measure, we found that EARS-C1 performed best with
an overall detection rate of 38.0% (CI: 37.8-38.2) and the 90th per-
centile approach was second best with an overall detection rate 24.1%
(CI: 23.9-24.2). We found that the POD1 varied greatly across syndrome
categories; for instance, using EARS-C1, POD1 was 65.3% (CI: 64.9-
65.7) for AFR but only 21.1% (CI: 20.8-21.4) for PF (Figure 3).
By the T-measure, we found CUSUM performed best with 87.8% (CI:
85.8-89.8) of multi-reporting period outbreaks detected by the algo-
rithm in the first reporting period. The next best-performing algorithms
by this measure were the EARS-C1 at 75.1% (CI: 74.4-75.8) and the 90th
percentile approach at 73.1% (CI: 72.0-74.1%). The performance of the
other EARS group of algorithms was notably poorer (Figure 4).
Receiver operating characteristic curves analysis on the 90th percentile
approach
For the weighted ROC, AUC was computed to be 0.7687 and for the
three-dimensional ROC, VUC was computed to be 0.8034 (Figure 5).
A.T. Craig, R.N.F. Leong, M.W. Donoghoe et al. IJID Regions 8 (2023) 157–163

Figure 2. Mean proportion of outbreaks de-

tected by 6 algorithms in the first reporting pe-
riod, by the size of the outbreak. CUSUM = cu-
mulative summation; EARS = early alert
and response system; EWMA = exponentially
weighted moving average; ∗ ’ × ’ indicates the
factor by which the estimated baseline was
multiplied to generate the outbreaks at differ-
ent magnitudes.

Figure 3. Proportion of outbreaks detected

within the first reporting period. AFR = acute
fever and rash; CUSUM = cumulative summa-
tion; EARS = early alert and response system;
EWMA = exponentially weighted moving aver-
age; ILI = influenza-like illness; PF = prolonged
fever.

Figure 4. Proportion of multi-reporting

period outbreaks detected within the first
reporting period. AFR = acute fever and
rash; CUSUM = cumulative summation;
EARS = early alert and response system;
EWMA = exponentially weighted moving
average; ILI = influenza-like illness; PF = pro-
longed fever; POD1 = probability of detection
in the first reporting period.

160
A.T. Craig, R.N.F. Leong, M.W. Donoghoe et al.
Figure 5. Extended ROC curves applied to the 90th percentile approach. (Left) weighted ROC curve; (right) three-dimensional ROC curve. ROC = receiver operating
characteristic.
Both versions led to extended ROC curves with relatively high prediction
performance, albeit against a backdrop of very high false positive rates.
Discussion
We present a comparative assessment of the performance of 6 out-
break detection algorithms using simulation scenarios based on param-
eters derived from national-level data collected from 8 PICTs as part of
the PSSS. A simulation-based evaluation method was chosen as there is
no reference standard of ’true’ outbreaks that could be used for com-
parison, nor was it feasible to enhance existing surveillance activities
and prospectively collect data that might fulfill the requirements of a
reference standard.
Our analysis comparing algorithms’ performance found that no sin-
gle approach outperformed others across all dimensions tested (i.e.,
sensitivity, specificity, and timeliness), suggesting that decision-makers
need to consider trade-offs between these parameters when deciding
which surveillance signal generation algorithm (or algorithms) to use.
Decision-makers must consider both performance needs and contextual
realities, including the health sector’s tolerance for false alarms and
costs associated with investigating excessive numbers of false signals.
As an example, exploration of the 90th percentile approach’s perfor-
mance using the extended ROC curve (Figure 5) suggests that the 90th
percentile approach can reliably provide time-sensitive outbreak pre-
dictions if a high FPR (of around 50%, or one-in-two signals generated
are false alarm) is tolerable. If a lower FPR is required (say 5% or one-
in-twenty signals generated being false), time-sensitive outbreak predic-
tions fall to levels that have limited outbreak intelligence-related public
health value. Further, our analysis was executed using aggregated na-
tional data. As analysis moves toward the sub-national/sentinel site lev-
els, with the primary aim to identifying outbreaks locally and quickly,
the volume (and stability) of data will dwindle resulting in a higher
probability of encountering false positives. This jeopardizes the integrity
and credibility of surveillance efforts, as frequent false alarms will both
waste precious resources and erode trust in the system, risking alerts
being inconsequential in driving public health actions. These findings
reinforce the need for future work to explore if, and if so how, routinely
collected data in small population settings may best be used for reliable
outbreak signal generation.
Beyond these general observations, we make 4 specific comments.
First, while overall the EARS-C1 performed the best of the algorithms
tested, no algorithm was found to perform particularly well as a reliable
and timely means to detect all types of outbreaks. Interestingly, overall,
there was a four-fold difference in sensitivity across the algorithm tested
161
IJID Regions 8 (2023) 157–163
(i.e., 11.1% for CUSUM to 46.5% for EARS-C1), when the FPR was kept
relatively constant (between 5.7-6.1%). This is of particular relevance
in resource-limited settings where excessive false positives have the po-
tential to overwhelm limited public health capacities, draw human and
financial resources away from other surveillance activities and, in doing
so, undermine the acceptability of surveillance-based intelligence. Fur-
ther, we found that algorithms that performed best –90th percentile ap-
proach, EARS-C1 and EARS-C2 – could only detect very large and acute
outbreaks of syndromes with small baselines (notably AFR and PF) with
any degree of reliability. Outbreaks with these dynamics are likely to be
obvious and detected and reported by clinical staff or the public through
event-based reporting mechanisms. Algorithm-based outbreak detection
adds the most value if able to detect smaller and less obvious outbreaks
(i.e., those that result in low-to-moderate attack rates or manifest with
patients presenting over extended periods).
Second, we observed a direct correlation between the size of a syn-
drome’s baseline and algorithm performance. Except for EARS-C’s detec-
tion of simulated AFP outbreaks, detection performance was markedly
poorer when the baseline was small, likely because of high variability
in the time series due to few case counts. This finding raised questions
about the appropriateness of using algorithmic approaches when the
background data they rely on are inadequate to provide a reliable in-
dication of the ’normal’ pattern of disease for a given time and place.
This finding may inspire further research into developing early warning
surveillance approaches specifically designed for use when count data
is small.
Third, and consistent with existing literature, across all algorithms
tested, we found the ability to detect outbreaks was directly propor-
tional to the magnitude of an outbreak and how temporally clustered
cases were, suggesting that syndromic surveillance is likely most ap-
propriately used to detect large and explosive events, and perhaps not
suitable for the early detection of outbreaks with low reproduction rates.
Again, our analysis found that EARS-C1 outperformed other algorithms
by this measure.
Finally, POD and FPR represent the detection performance of algo-
rithms from 2 perspectives, which may confuse the interpretation of re-
sults and challenge comparative analysis. To overcome this, we fixed the
FPR and computed the resulting PODs. We acknowledge that the balance
between sensitivity and specificity needs to consider factors such as re-
source availability and the implications of too high a false alarm rate. To
address these, future analysis may consider including a modifying factor
to adjust the relative emphasis placed on sensitivity or specificity com-
ponents to better meet local conditions and system tolerances. Further-
more, given the importance of prompt outbreak detection, researchers
A.T. Craig, R.N.F. Leong, M.W. Donoghoe et al.
may consider how a timeliness dimension can be incorporated into a
modified measure. We have demonstrated how this may be done using
2 modified ROC curves (Figure 5); however, computational efficiency
must be further optimized.
The following should be considered to improve algorithm-based out-
break detection performance. The health information systems, the pro-
cess of capturing and transferring data, and the quality of data exhibit
significant variation across the PICT, often characterized by fragmen-
tation [30]. Top priority must be given to improving the quality (i.e.,
the completeness, stability, and timeliness) of data that is collected and
used in analysis. This will require exploring what motivates staff re-
sponsible for data collection and entry (typically nurses) to perform
surveillance-related tasks with rigor. Data quality may be improved
through surveillance nurse training, mentoring, and supervision; follow-
up of non-reporting sites; implementation of data quality monitoring;
and recognition and rewards to encourage good data collection and re-
porting practices. Importantly, syndromic case definitions need to be
clinically meaningful and distinct. In addition, because some datasets
will be too small for meaningful analysis in a small-population context,
policymakers should consider whether case definitions used are too re-
strictive to be useful (from a data analysis point of view) and hence re-
quire modification. To boost the volume of data for statistical analysis,
and being pragmatic, aggregation of data collected by multiple sentinel
sites in close proximity to each other (i.e., within a mile or 2) before
analysis may be considered as a strategy to increase the power of anal-
ysis that is conducted.
As mentioned earlier, our analysis revealed that no single algorithm
showed significant superiority over others tested; therefore, the deci-
sion regarding which algorithm to use is not straightforward. It is es-
sential for decision-makers to acknowledge the inherent limitations of
syndrome-based early warning surveillance, especially when applied to
small populations. Decision-makers should be counseled not to rely en-
tirely on a syndromic strategy to meet all national health protection
needs. Instead, syndromic surveillance should be viewed as a comple-
mentary strategy that supports other more robust outbreak detection
methods, such as laboratory-based and formalized event-based surveil-
lance mechanisms. We re-emphasize the need for conducting further
analysis using localized data to better understand how algorithms per-
form when applied at the community level.
The results presented in this paper should be interpreted with cau-
tion. Most notably, we assume that robust filtering produced outbreak-
free reference baselines; some outbreaks may have been missed. Sec-
ond, we analyzed national-level data, which may have led to missing
smaller outbreaks occurring at individual sentinel sites. Third, simu-
lated outbreaks are theoretical and may not represent how outbreaks
evolve. Fourth, we tested a limited number of algorithms and set the
modifying variables to the default levels recommended by their authors.
Despite these limitations, we could retain contextual reality within the
model by drawing on authentic baseline data. Our analysis has gener-
ated novel policy-relevant insights into the performance of syndromic
surveillance when applied to nationally aggregated data in small popu-
lation contexts and may help shape early warning surveillance system
design within these settings. We emphasize that outbreaks need to be
detected locally and hence suggest the application of this tool using sen-
tinel site data is appropriate. The modelling tool offers potential utility
for the prospective measurement (and monitoring) of surveillance sys-
tems’ performance at sentinel site and aggregated district or national
levels. The methods presented in this paper are broadly transferable and
may be used to measure (and monitor) surveillance system performance
in other settings.
Conclusion
While having notable limitations in detecting outbreaks, the EARS-
C1 approach outperformed all other algorithms tested. Our findings sug-
gest that algorithm-based outbreak signal detection methods perform
162
IJID Regions 8 (2023) 157–163
poorly in the presence of small numbers of background cases and hence,
in these settings, should not be relied upon but rather seen as supplemen-
tary to more robust surveillance approaches, including hospital-based,
point-of-care-test-based and event-based surveillance strategies. Efforts
to increase the volume and stability of the data on which aberration
detection relies will likely improve performance.
Declarations of competing interest
The authors have no competing interests to declare.
Funding
Adam Craig is supported by the University of Queensland Health Re-
search Accelerator Initiative and David Muscatello is supported by an
Australian National Health and Medical Research Council (NHMRC) In-
vestigator Grant (APP1194109). The contents of the published material
are solely the responsibility of the Administering Institution, a Partici-
pating Institution or individual authors and do not reflect the views of
the NHMRC.
Ethical approval
Given the nature of the research conducted and data used, ethical
approval was not required.
Author contributions
ATC, RNFL, and MWD were involved in the manuscript\220s con-
ceptualisation, analysis, interpretation of data, writing, revision, and re-
view. ATC, RNFL, MWD, VJCM, and CJMO were involved in analysis,
interpretation of data, writing, revision, and review. DM was invovled
in the interpretation of data, revision, and review.
Supplementary materials
Supplementary material associated with this article can be found, in
the online version, at doi:10.1016/j.ijregi.2023.08.007.
References
[1] Buehler JW, Hopkins RS, Overhage JM, Sosin DM, Tong V, Working Group CDC.
Framework for evaluating public health surveillance systems for early detection of
outbreaks: recommendations from the CDC Working Group. MMWR Recomm Rep
2004;53:1–11.
[2] Wagner M, Tsui F, Cooper G, Espino JU, Harkema H, Levander J, et al. Probabilis-
tic, decision-theoretic disease surveillance and control. Online J Public Health Inform
2011;3 ojphi.v3i.3798. doi:10.5210/ojphi.v3i3.3798.
[3] May L, Chretien JP, Pavlin JA. Beyond traditional surveillance: applying syndromic
surveillance to developing settings–opportunities and challenges. BMC Public Health
2009;9:242. doi:10.1186/1471-2458-9-242.
[4] Bédubourg G, Le Strat Y. Evaluation and comparison of statistical methods
for early temporal detection of outbreaks: a simulation-based study. PLoS One
2017;12:e0181227. doi:10.1371/journal.pone.0181227.
[5] Shmuei G, Fienberg SE. Current and potential statistical methods for monitoring
multiple data streams for biosurveillance. In: Wilson AG, Wilson GD, Olwell DH,
editors. Statistical methods for counterterrorism. New York: Springer; 2006. p. 109–40.
[6] Noufaily A, Morbey RA, Colón-González FJ, Elliot AJ, Smith GE, Lake IR, et al.
Comparison of statistical algorithms for daily syndromic surveillance aberration de-
tection. Bioinformatics 2019;35:3110–18. doi:10.1093/bioinformatics/bty997.
[7] Yuan M, Boston-Fisher N, Luo Y, Verma A, Buckeridge DL. A systematic review of
aberration detection algorithms used in public health surveillance. J Biomed Inform
2019;94:103181. doi:10.1016/j.jbi.2019.103181.
[8] Alsentzer E, Ballard SB, Neyra J, Vera DM, Osorio VB, Quispe J, et al. As-
sessing 3 outbreak detection algorithms in an electronic syndromic surveil-
lance system in a resource-limited setting. Emerg Infect Dis 2020;26:2196–200.
doi:10.3201/eid2609.191315.
[9] Yeng PK, Woldaregay AZ, Solvoll T, Hartvigsen G. Cluster detection mechanisms
for syndromic surveillance systems: systematic review and framework development.
JMIR Public Health Surveill 2020;6:e11512. doi:10.2196/11512.
[10] Craig AT, Kama M, Samo M, Vaai S, Matanaicake J, Joshua C, et al. Early
warning epidemic surveillance in the Pacific island nations: an evaluation of the
Pacific syndromic Surveillance System. Trop Med Int Health 2016;21:917–27.
doi:10.1111/tmi.12711.
A.T. Craig, R.N.F. Leong, M.W. Donoghoe et al.
[11] Vilain P, Maillard O, Raslan-Loubatie J, Ahmed Abdou M, Lernout T, Filleul L. Use-
fulness of syndromic surveillance for early outbreak detection in Small Islands: the
case of Mayotte. Online J Public Health Inform 2013;5. doi:10.5210/ojphi.v5i1.4503.
[12] Pacific Community. Pacific datahub: population projections, https://sdd.spc.int/
dataset/df_pop_proj; 2023 [accessed 27 July 2023].
[13] Craig AT, Kool J, Nilles EJ. The Pacific experience: supporting small island coun-
tries and territories to meet their 2012 International Health Regulations (2005)
commitments. Western Pac Surveill Response J 2013;4:14–18. doi:10.5365/WP-
SAR.2012.3.4.007.
[14] Roth A, Mercier A, Lepers C, Hoy D, Duituturaga S, Benyon E, et al. Concurrent
outbreaks of dengue, chikungunya and Zika virus infections - an unprecedented
epidemic wave of mosquito-borne viruses in the Pacific 2012–2014. Euro Surveill
2014;19:20929. doi:10.2807/1560-7917.es2014.19.41.20929.
[15] World Health Organization, Pacific Community A practical guide to implementing syn-
dromic surveillance in Pacific island countries and territories. Geneva: World Health
Organization; 2010.
[16] Rogerson PA, Yamada I. Approaches to syndromic surveillance when data consist of
small regional counts. MMWR 2004;53:79–85 Suppl.
[17] Salmon M, Schumacher D, Höhle M. Monitoring Count Time Series in R:
Aberration Detection in Public Health Surveillance. J Stat Soft 2016;70:1–35.
doi:10.18637/jss.v070.i10.
[18] Fried R. Robust filtering of time series with trends. J Nonparametric Stat
2004;16:313–28. doi:10.1080/10485250410001656444.
[19] Meyer S, Held L, Höhle M. Spatio-Temporal Analysis of Epidemic Phenomena Using
the R package surveillance. J Stat Soft 2017;77. doi:10.18637/jss.v077.i11.
[20] Moritz S, Bartz-Beielstein T. ImputeTS: time series missing value imputation in R. R
J 2017;9:207–18. doi:10.32614/RJ-2017-009.
163
IJID Regions 8 (2023) 157–163
[21] Wickham H, Henry L. purrr: functional Programming Tools. R Foundation for Statis-
tical Computing, https://purrr.tidyverse.org/; 2019 [acccessed 13 February 2023].
[22] Wickham H, Francois R, Henry L, Muller K, Vaughan D. dplyr: A Grammar of Data.
R Foundation for Statistical Computing, https://dplyr.tidyverse.org; 2023 [accessed
13 February 2023].
[23] Scrucca L, Snow G, Bloomfield P. qcc: an R package for quality control charting and
statistical process control. R News 2004;4:11–17.
[24] Wickham H. forcats: Tools for Working with Categorical Variables (Factors). R Foun-
dation for Statistical Computing, https://forcats.tidyverse.org/ https://github.com/
tidyverse/forcats; 2022 [acccessed 21 March 2023].
[25] Fagerland MW, Lydersen S, Laake P. Recommended confidence intervals for
two independent binomial proportions. Stat Methods Med Res 2015;24:224–54.
doi:10.1177/0962280211415469.
[26] Franz VH. Ratios: A short guide to confidence limits and proper use. Arxiv 10 Oc-
tober 2007 [accessed 21 March 2023].
[27] Kleinman KP, Abrams AM. Assessing surveillance using sensitivity,
specificity and timeliness. Stat Methods Med Res 2006;15:445–64.
doi:10.1177/0962280206071641.
[28] Signorell A, Aho K, Alfons A, Anderegg N, Aragon T, et al. DescTools: Tools for de-
scriptive statistics, https://cran.r-project.org/web/packages/DescTools/index.html;
2023 [acccessed 21 March 2023].
[29] Roussel R-L, Barber CB, Habel K, Grasman R, Gramacy R, Mozharovskyi P,
et al. Geometry: mesh generation and surface tessellation, https://davidcsterratt.
github.io/geometry/; 2023 [acccessed 21 March 2023].
[30] World Health Organization Manila, Phillipines. Pacific Health Information Network Re-
gional Meeting on Strengthening Health Information Systems and Digital Health. Geneva:
World Health Organization; 2023.