Journal of Dentistry 110 (2021) 103664

Contents lists available at ScienceDirect

Journal of Dentistry
journal homepage: www.elsevier.com/locate/jdent

Review article

Assessment of predictive performance of caries risk assessment models

based on a systematic review and meta-analysis
Naichuan Su a, *, Maxim D. Lagerweij b, Geert J.M.G. van der Heijden a
a
Department of Social Dentistry, Academic Centre for Dentistry Amsterdam (ACTA), University of Amasterdam and VU University, the Netherlands
b
Department of Cariology, Endodontology and Pedodontology, Academic Centre for Dentistry Amsterdam (ACTA), University of Amasterdam and VU University, the
Netherlands

A R T I C L E I N F O A B S T R A C T

Keywords: Objectives: To assess the predictive performance of caries risk assessment (CRA) models for prediction of caries
Dental caries increment for individuals based on a systematic review and meta-analyses.
Caries risk assessment Data/Sources: We included external validation studies assessing the predictive performance of CRA models for
External validation
prediction of caries increment for individuals, using discrimination and calibration as the outcome parameters.
Discrimination
PubMed, EMBASE, and CINAHL were searched electronically on 10th September 2020 to identify prediction
Calibration
Systematic review modeling studies on external validation of CRA models. The risk of bias of the included studies was assessed
using the Prediction model Risk Of Bias ASsessment Tool (PROBAST).
Study selection: A total of 22 studies with seven different CRA models were included. As for full Cariogram, the
pooled area under the receiver operating characteristic curve (AUC) was 0.78 (95 %CI: 0.68; 0.85) based on eight
studies regardless of the risk of bias levels, and 0.82 (95 %CI: 0.58; 0.93) based on four studies with low risk of
bias only. The pooled observed: expected ratio (O:E ratio) of full Cariogram was 0.91 (95 %CI: 0.72; 1.14) based
on 12 studies regardless of the risk of bias levels, and 0.89 (95 %CI: 0.71; 1.12) based on five studies with low
risk of bias only. As for reduced Cariogram, the pooled AUC was 0.72 (95 %CI: 0.67; 0.77) based on six studies
regardless of the risk of bias levels, and 0.74 (95 %CI: 0.45; 0.91) based on two studies with low risk of bias only.
The pooled O:E ratio of reduced Cariogram was 0.84 (95 %CI: 0.59; 1.18) based on six studies regardless of the
risk of bias levels, and 1.05 (95 %CI: 0.43; 2.59) based on two studies with low risk of bias only. Based on an
insufficient number of studies for the other CRA models, the pooled AUCs ranged from 0.50 to 0.88, while the
pooled O:E ratio ranged from 0.38 to 1.00.
Conclusion: The average predictive performance of both full and reduced Cariogram seems to be acceptable.
However, the evidence from research does not allow a firm conclusion on the performance of the other included
CRA models, due to the insufficient number of high-quality studies.
Clinical significance: Both full and reduced Cariogram were found to be reliable CRA models for prediction of
caries increment in clinical practices for dental patients and communities for general populations. The reduced
Cariogram showed better predictive performance and less burden in terms of time and resources to individuals
than the full Cariogram. Therefore, the reduced Cariogram could be more recommended than the full Cariogram.

1. Introduction prevalence of dental caries of permanent teeth is 35 %, which is the

Caries is a chronic progressive disease caused by a multifactorial
dynamic interplay of microbial, behavioral, and social aspects [1]. If
caries is not treated in time, it may affect deeper layers of teeth and may
result in a severe toothache, infection, and tooth loss. Caries is a major
oral health problem in most industrialized countries. The global
highest among the 50 most common diseases, while the global preva­
lence of dental caries of primary teeth is 9 % in the year 2010 [2]. In the
year 2017, around 2.3 billion people globally had caries of permanent
teeth, and 0.5 billion children had caries of primary teeth [3]. However,
in recent decades, a paradigm shift in caries management is seen from
restorative treatments towards non-operative caries risk management

* Corresponding author at: 5F, Department of Social Dentistry, Academic Centre for Dentistry Amsterdam (ACTA), University of Amsterdam and Vrije Universiteit
Amsterdam, Gustav Mahlerlaan 3004, 1081, LA, Amsterdam, the Netherlands.
E-mail address: n.su@acta.nl (N. Su).

https://doi.org/10.1016/j.jdent.2021.103664
Received 23 November 2020; Received in revised form 5 March 2021; Accepted 9 April 2021
Available online 10 May 2021
0300-5712/© 2021 The Author(s). Published by Elsevier Ltd. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).
N. Su et al.
(CRM) policies including early preventive strategies [4]. Nowadays,
CRM is an acknowledged integral part of contemporary dental practice.
This paradigm shift has stimulated the development of multivariable
caries risk scores to support caries risk assessment (CRA) [4].
CRA refers to an approach to establish the probability of a future
(new or incident) enamel or dentine lesion, i.e. predicting caries after
some period of follow-up. CRA primarily aims at the identification of
individuals with an increased risk of the occurrence or progression of
caries over a specified period of follow-up [5]. It anticipates detection of
early-stage caries and allows for tailoring CRM to individual risk levels.
Hence, CRA may assist dentists in decision-making on diagnostic pro­
cedures, treatment, and recall appointments. It is considered a corner­
stone in patient-centered CRM [6].
Several CRA models have been developed [7–12]. They differ in their
content and approach to assess the caries risk. Their predictive perfor­
mance, i.e. correspondence between predicted and observed caries at
follow-up, may depend on such differences, but also on the quality of
study methods and the characteristics of the study population. The
discrimination and calibration are the most commonly used parameters
for evaluating the predictive performance of a prediction model [13,14].
In absence of assessment of such evaluation, one should be cautious
using CRA models in practice [14]. That is, poor predictive performance
could eventually result in inadequate or untoward decisions in caries
risk management, and thereby impact dental health of individuals.
Therefore, it is essential to systematically appraise the predictive per­
formance of CRA models and the external validation thereof across
different study populations, settings, and locations. Such evaluation may
reveal limitations of CRA models and may provide clues for adjustments
or further improvements [13]. So far, reviews have attempted to assess
the most important risk factors for caries increment, reported qualitative
analyses of CRA models or their sensitivity, specificity and predictive
values only [1,6,15,16]. Hence, to date, an overview of the systematic
quantitative assessment of discrimination and calibration of CRA models
is lacking.
While inexpensive and user-friendly approaches to CRA are expected
to fulfill the promise of cost-effective deployment of CRM policies,
external validation of the performance of CRA models is imperative for
the effectiveness of CRM policies. It should be noted that reports on the
discrimination and calibration of CRA models resulting from pop­
ulations in which the models are derived do not provide a credible basis
for external validation. Evidence on external validity of CRA models
should come from studies that evaluate the predictive performance in
subsequent studies in other populations. Hence, to quantitatively assess
the predictive performance of CRA models, notably discrimination and
calibration, the external validation studies that assessed the perfor­
mance of CRA models for prediction of caries increment for individuals
were systematically reviewed.
2. Materials and methods
This systematic review and meta-analysis were carried out based on
the Preferred Reporting Items for Systematic Reviews and Meta-
analysis: The PRISMA Statement [17].
2.1. Search strategy
Relevant publications were searched in electronic bibliographic
sources, including PUBMED, EMBASE, and CINAHL, without language
restriction, up to 10th September 2020. A combination of free text words
and systematic vocabulary (Medical Subject Headings for PUBMED and
EMTREE terms for EMBASE) relevant to “dental caries”, “prognosis”,
and “cohort studies” was used in the search strategies. The search
strategy is presented in Appendix Table 1.
To improve the comprehensiveness of the search and identify
potentially eligible studies that were not retrieved from the databases,
both forward and backward citation searching were used [18]. That is,
2
Journal of Dentistry 110 (2021) 103664
all the references cited in the studies eligible for inclusion were
inspected (backward citation searching) and all the articles which have
cited the studies eligible for inclusion were also inspected (forward
citation searching). Besides, the top 20 related/similar articles to the
studies eligible for inclusion are inspected. The forward citation
searching, backward citation searching, and related articles searching
were all performed via PubMed on 26th September 2020.
2.2. Selection criteria
The development of a CRA model requires a derivation cohort.
Before the model is implemented in practice the predictive performance
of the model (discrimination and calibration) should be established and
this requires a validation cohort. Since this systematic review is about
the empirical evidence of such validation studies, we excluded the
studies on development of CRA models in derivation cohorts. If a study
included both derivation cohort and validation cohort, only the vali­
dation cohort was included for analysis in the present review.
The selection criteria were framed based on the PICOTS (Population,
Intervention, Comparison, Outcome, Timing, and Setting) system in the
Critical Appraisal and Data Extraction for Systematic Reviews of Pre­
diction Modelling Studies (CHARMS) checklist [19]. Only publications
reporting studies satisfying the following criteria were selected:
(1) Including participants at risk of incident caries, irrespective of
demographic background (e.g. gender, age, race, or socioeco­
nomic status).
(2) Externally validating a CRA model. Studies in which CRA models
were developed were excluded. For studies reporting data from
both a derivation cohort and a validation cohort, only the vali­
dation cohort was included.
(3) Reporting data on either or both discrimination (e.g. area under
the receiver operating characteristic curve [AUC] or c-statistic)
and calibration (e.g. observed: expected ratio [O:E ratio]), or data
sufficient to (re)calculate these. Discrimination is defined as the
ability to differentiate between those with and those without the
caries increment in the follow-up [20]. Calibration is defined as
the agreement between the predicted risk and observed risk of
caries increment in the follow-up [20]. The O:E ratio is defined as
the ratio between total number of observed and total number of
expected events (O:E ratio) [21].
(4) Both quantitative or qualitative prediction models used to
perform risk stratification in the assessment of increment of caries
at follow-up in clinical practice or communities. Quantitative
models are defined as the models in which the individuals` pre­
dicted risks are calculated through algorithms and expressed
quantitatively as percentages. Qualitative models are defined as
the models in which the individuals` predicted risks are qualita­
tively classified into multiple ordinal predicted risk groups
through manual charting using a checklist of important factors/
indicators [22].
Publications which met the following criteria were excluded:
(1) Case reports, conference papers, case-control studies, or cross-
sectional studies;
(2) Duplicate publications;
(3) Languages other than English.
Two reviewers (NS and ML) independently assessed the titles and
abstracts of all identified studies from the electronic searches. Full texts
were obtained for studies that met the inclusion and exclusion criteria,
or where a clear decision could not be made from the title and abstract
alone. In the latter case, the selection was based on full-text reading.
Reviewers resolved initial disagreements by consensus discussion.
N. Su et al.
2.3. Risk of bias assessment
The Prediction model Risk Of Bias ASsessment Tool (PROBAST) was
used to assess the risk of bias of the included studies [14]. The tool in­
cludes four domains, including participants, predictors, outcome, and
analysis. Each domain was assessed in terms of risk of bias, while the
first three domains were also assessed in terms of applicability concerns.
A “risk of bias” judgment (“high”, “low”, and “unclear”) was made for
each domain. If the answers to all signaling questions within a domain
were judged as “yes” or “probably yes” (indicating a low risk of bias for
each question), the domain was judged to be at low risk of bias. If any
signaling question was judged as “no information”, or any signaling
question was judged as “no” or “probably no” (indicating a high risk of
bias), the risk of bias of the domain was first discussed by the consensus
discussion and then was judged as “low”, “unclear” or “high” risk of bias
[14]. The overall risk of bias of a primary study was regarded as “low”
when the risk of bias was low in each domain, while the overall risk of
bias was “high” when the risk of bias was high in any domain. Other­
wise, the overall risk of bias was regarded as “unclear”. This was fol­
lowed by a judgment about concerns regarding the applicability that
considers the extent to which the included studies match the systematic
review question in the aspect of participants, predictors, and outcome
domains. Concerns about applicability were rated as “low”, “high”, and
“unclear”. High concern was rated when the primary study did not
match the review question in any domain, while low concern was rated
when the study matched the review question in each domain. Unclear
domain was rated only when insufficient data were reported.
Two independent raters (NS and ML), blinded for each other`s rat­
ings, performed the PROBAST assessment. Reviewers resolved initial
disagreements by consensus discussion.
2.4. Data extraction
The data extraction of the primary studies was guided based on the
CHARMS checklist [19]. The information on source of data (study type),
participants (country and settings, number of participants, sex, age, and
country), outcomes (follow-up time, definition of outcome event,
criteria for diagnosis of caries, use of radiography, and percentage of
event), names of the validated prediction models, and model perfor­
mance (AUC and O:E ratio) were extracted using a standardized form.
The data extraction was performed by one reviewer (NS) and cross-
checked by the other reviewer (ML). Reviewers resolved initial dis­
agreements by consensus discussion.
2.5. Design of meta-analysis
To assess the discrimination of the CRA models, the meta-analyses
were performed by computing AUC extracted from the included
studies using the random-effect model. For quantitative models, only the
AUCs that were produced with the endpoints versus the quantitative
predicted risks were included. For qualitative models, the AUCs that
were produced with the endpoints versus the qualitative risk categories
were included. If the variance of the AUC, including standard error (SE)
or 95 % confidence interval (95 %CI), was not reported, it was calcu­
lated based on the total number of observed events, total number of
sample size, and AUC values [13]. An pooled AUC of 0.70 to 0.80 in­
dicates the discrimination of the model is acceptable, while an pooled
AUC ≥ 0.80 indicates that the discrimination of the model is excellent to
outstanding [23].
To assess the calibration of the CRA models, the meta-analyses were
performed by computing the O:E ratio extracted from the included
studies using the random-effect model. If neither the total number of
expected events nor the expected risk of the population was directly
reported, the O:E ratio was estimated based on the reported mean pre­
dicted risks across different risk groups in calibration plots or calibration
tables or reported sensitivity and specificity. For quantitative models,
3
Journal of Dentistry 110 (2021) 103664
the O:E ratio based on the calibration plots or calibration tables was
prioritized. For qualitative models, the O:E ratio based on the sensitivity
and specificity was prioritized. If the variance of O:E ratio was not re­
ported, it was calculated with total number of observed events and total
sample size [13]. An pooled O:E ratio of 1 indicates that the observed
risk equals the expected risk and hence the model is optimally cali­
brated. An pooled O:E ratio <1 indicates the overestimation of the
model, while an pooled O:E >1 indicates the underestimation of the
model. An pooled O:E ratio between 0.8 and 1.2 indicates that the
calibration of the model is acceptable [13].
For pooling the AUC and O:E ratio, a logit-transformation was used
because both were not normally distributed. The meta-analyses were
performed for the different CRA models separately. If the number of
included studies was not big enough for the meta-analysis, a qualitative
description was performed instead. The subgroup analysis was per­
formed based on the risk of bias of included studies for each meta-
analysis. The meta-analysis was conducted using R 3.6.1 (R Develop­
ment Core Team, Vienna, Austria).
3. Results
3.1. Results of search and selection
The initial search identified a total of 3,506 records from the elec­
tronic databases. During screening the titles and abstracts, 482 of them
satisfied the inclusion criteria. Another 16 records were screened after
reading the titles and abstracts based on forward citation searching,
backward citation searching and related articles searching. 476 studies
eventually did not satisfy the inclusion criteria after carefully reading
the full-text publications. Therefore, a total of 22 studies were included
in the present review [10,22,24–43] (Fig. 1).
3.2. Characteristics of included studies
The 22 included studies concerned three quantitative models (Car­
iogram [10,22,24–34], the National University of Singapore Caries Risk
Assessment (NUS-CRA) model [22], and Dentoprog-Method model [42])
and four qualitative models (Caries Management By Risk Assessment
(CAMBRA) model [22,35–39], the Caries-risk Assessment Tool (CAT)
[22,39,40], the Public Dental Service (PDS) guidelines [41], and the
University of Michigan School of Dentistry Caries Risk Assessment
(UMSD-CRA) model [43]). Among the seven CRA models, Cariogram,
CAMBRA, NUS-CRA, and CAT had both full and reduced versions. The
full model was defined as the model with a complete set of predictors
including biological predictors, such as saliva tests with microbiological
cultivations. A reduced model was defined as one excluding such bio­
logical predictors. 12 studies reported about the full Cariogram [10,22,
24–33], six studies about the reduced Cariogram [22,25,26,28,33,34],
six studies about the full CAMBRA [22,35–39], one study about the
reduced CAMBRA [22], one study about the full NUS-CRA [22], one
study about the reduced NUS-CRA [22], two studies about the full CAT
[22,39], two studies about the reduced CAT [22,40], one study about
the Public Dental Service (PDS) guidelines [41], one study about the
Dentoprog-Method model [42], and one study about UMSD-CRA [43].
The essential features of the seven CRA models are presented in Ap­
pendix Table 2. The main characteristics of the included studies are
presented in Table 1.
3.3. Risk of bias assessment
The overall risk of bias was considered low in only five studies [10,
22,25,29,31] and high in the remaining 17 studies (Table 2). All the five
studies with low risk of bias [10,22,25,29,31] concerned the assessment
of the performance of the full Cariogram, two studies [22,25] with low
risk of bias concerned the performance of the reduced Cariogram, and
one study with low risk of bias [22] concerned the performance of both
N. Su et al.
Fig. 1. Flow diagram of study inclusion (CAMBRA: Caries Management By Risk Assessment; NUS-CRA: the National University of Singapore Caries Risk Assessment;
CAT: Caries-risk Assessment Tool; PDS: Public Dental Service; UMSD-CRA: the University of Michigan School of Dentistry Caries Risk Assessment model.
full and reduced NUS-CRA, both full and reduced CAMBRA, and both
full and reduced CAT (Fig. 1).
With respect to the participants domain, all the included studies were
considered low risk of bias. With respect to the predictors domain, we
considered the risk of bias high in one study [42] because the predictors
were not defined and assessed in a similar way for all the included
participants. No study was considered unclear risk of bias in this
domain. With respect to the outcome domain, we considered the risk of
bias high in one study [42] because the outcome was not determined in a
similar way for all participants. We considered the risk of bias unclear in
another study [40] because the definition of the caries was not reported.
With respect to the analysis domain, the risk of bias in 16 studies [24,
26–28,30,32–41,43] was considered high. The reasons were that (1) the
number of participants with the events was too small, which did not
meet the criteria of >100 participants with outcome events based on the
widely used thumb of rules [14]; (2) the loss to follow-up of the par­
ticipants >20 %; and (3) a complete case analysis was used to handle the
missing data when the percentages of missing values were large and the
missing data mechanism was not checked. One study [42] was consid­
ered to have an unclear risk of bias because the information on loss to
follow-up of the participants and the handling of the missing data were
not reported.
All the 22 studies were considered to have low concerns regarding
the applicability of primary studies to the review question in the aspect
of participants, predictors, and outcome domains (Table 2).
4
Journal of Dentistry 110 (2021) 103664
3.4. Results of meta-analysis
3.4.1. Full version of Cariogram
Eight studies [10,22,25,26,28–30,33] were pooled for discrimina­
tion, irrespective of the risk of bias, and the pooled AUC was 0.78 (95 %
CI: 0.68; 0.85; I2 = 94 %), which indicated that the discrimination of the
model was acceptable. When the studies with low and high risk of bias
were pooled separately, it indicated that the discrimination of the model
based on the studies with low risk of bias and high risk of bias was both
acceptable (Table 3).
Twelve studies [10,22,24–33] were pooled for calibration, irre­
spective of the risk of bias, and the pooled O:E was 0.91 (95 %CI: 0.72;
1.14; I2 = 97 %), which indicated that the calibration of the model was
acceptable. When the studies with low and high risk of bias were pooled
separately, it indicated that calibration of the model based on the studies
with low risk of bias and high risk of bias was both acceptable (Table 3).
3.4.2. Reduced version of Cariogram
Six studies [22,25,26,28,33,34] were pooled for discrimination,
irrespective of the risk of bias, and the pooled AUC was 0.72 (95 %CI:
0.67; 0.77; I2 = 44 %), which indicated that the discrimination of the
model was acceptable. When the studies with low and high risk of bias
were pooled separately, it indicated that the discrimination of the model
based on the studies with low risk of bias and high risk of bias was both
acceptable (Table 3).
 N. Su et al.
Table 1
Characteristics of the included studies.
Source of date Participants Outcomes Models Outcomes

Studies Study Type Participants No. of No. of Age at Outcome event % of Location Criteria for Radiography Follow-up Models Discrimination O:E ratio
description (country participants participants baseline definition events of caries diagnosis of used for time validated (AUC)
and settings) at baseline at follow-up caries diagnosis of
(F/M) (F/M) caries

Gao 2010 [10] Prospective Children from Grade- 1782 (893/ 1576 3− 6 years Δdmft>0 44 % Coronal WHO criteria No 1 year Cariogramc 0.73 (0.71 0.76) 689:789
cohort 1 government 889) (unknown) (689/
kindergartens in 1576)
Singapore
NUS-CRAa 0.88 (0.84 0.91) 178:184
NUS-CRAb 0.85 (0.81 0.89) 178:178
Cariogramc 0.78 (0.74 0.82) 178:181
Children in grade-1 in
Prospective 544 (262/ 485 (224/ 37 % Cariogramd 178:180
Gao 2013 [22] kindergartens in Hong 3 years Δdmft >0 Coronal WHO criteria No 1 year
cohort 282) 261) (178/485) CAMBRAe 178:263
Kong, China
CAMBRAf 0.76 (0.72 0.81) 178:340
CATg 178:474
CATh 178:467
Patients registered in
Prospective public dental clinics in 1295 (676/ 982 (454/ 35 %
Petersson 2015 [24] 19 years ΔDFS>0 Coronal WHO criteria Yes 3 yeras Cariogramc 344:368
cohort the Skåne region, 619) 528) (344/982)
Sweden
Retrospective Children from schools 438 (208/ 392 10− 11 31 % Cariogramc 0.75 (0.70 0.80) 122:107
Petersson 2010 [25] ΔDMFS>0 Coronal Other Yes 2 years
cohort in Halmstad, Sweden 230) (unknown) years (122/392) Cariogramd 0.72 (0.67 0.78) 122:107
The elderly in Cariogramc 0.77 (0.70 0.83) 70:187
Prospective 334 280 25 % (70/
Hayes 2017 [26] communities in Cork, ≥65 years ΔDFS>0 Root ICDAS Unknown 2 years
cohort (unknown) (unknown) 280) Cariogramd 0.79 (0.72 0.85) 70:123
Ireland
5

The elderly in 55, 65,

Retrospective 208 148 74 % Root and
Petersson 2003 [27] communities in and 75 ΔDMFS>0 WHO criteria Yes 5 years Cariogramc 109:85
cohort (unknown) (unknown) (109/148) coronal
Sweden years
Patients visiting the Cariogramc 0.71 (0.65 0.78) 76:94
Department of
Conservative
Dentistry and
Endodontics,
Prospective 215 192 23.3 ± 3.1 40 % (76/
Dou 2018 [28] Stomatological ΔDMFS>0 Coronal WHO criteria Yes 2 years
cohort (unknown) (unknown) years 192) Cariogramd 0.70 (0.63 0.77) 76:103
Hospital of Chongqing
Medical University,
China, for caries or
pulpitis/pulp necrosis
caused by caries
Prospective Children from schools 957 (485/ 861 (440/ 36 %
Campus 2012 [29] 7− 9 years ΔDFS>0 Coronal WHO criteria No 2 years Cariogramc 0.93 (0.91 0.95) 312:368
cohort in Sardinia, Italy 472) 421) (312/861)
Children residing in
social welfare schools

Journal of Dentistry 110 (2021) 103664

Prospective in principle of 36 36 12.9 ± 0.7 36% (13/
Sudhir 2017 [30] ΔDFS>0 Coronal ICDAS Unknown 1.5 years Cariogramc 0.79 (0.63 0.95) 13:15
cohort Vatsalaya (unknown) (unknown) years 36)
vidhayasharam
Nellore, India
Children
5 ± 0.5
Children from schools aged 5
Prospective 520 499 years, 12 ΔDMFT/
Garg 2018 [31] in Paonta Sahib, years: Coronal WHO criteria Unknown 1 year Cariogramc 161:233
cohort (unknown) (unknown) ± 0.5 dmft>0
District Sirmour, India 37 % (92/
years
250);
(continued on next page)
 N. Su et al.
Table 1 (continued )
Source of date Participants Outcomes Models Outcomes

Studies Study Type Participants No. of No. of Age at Outcome event % of Location Criteria for Radiography Follow-up Models Discrimination O:E ratio
description (country participants participants baseline definition events of caries diagnosis of used for time validated (AUC)
and settings) at baseline at follow-up caries diagnosis of
(F/M) (F/M) caries

Children
aged 12
years:
28 % (69/
249)
Dolic 2020 [32] Retrospective Pregnant women from 96 (96/0) 80 (80/0) 20− 42 ΔDMFT>0 68% (54/ Coronal WHO criteria No 4 years Cariogramc 54:32
cohort communities in Banja years 80)
Luka, Bosnia and (mean:
Herzegovina 27.4 ± 7.2
years)
Children from public 2− 5 years Cariogramc 0.65 (0.56 0.74) 74:77
Retrospective preschools in the 175 140 (mean: 3.3 53 % (74/ WHO criteria
Birpou 2019 [33] ΔECC>0 Coronal No 2 years
cohort greater area of Athens, (unknown) (unknown) ± 0.8 140) and ICDAS Cariogramd 0.65 (0.56 0.74) 74:65
Greece years)
Patients referred to
12− 20
the Specialist Clinic
years
Prospective for Orthodontic, 270 255 (165/ 13 % (34/
Enerbäck 2020 [34] (mean: ΔDFT>0 Coronal WHO criteria Yes Unknown Cariogramd 0.69 (0.59 0.79) 34:60
cohort Public Dental Service, (unknown) 90) 255)
15.4
Mölndal Hospital,
years)
Sweden
Children residing in a
social welfare school
6

Prospective in principle of 36 % (26/

Sudhir 2016 [35] 72 (24/48) 72 (24/48) 13 years ΔDFS>0 Coronal ICDAS unknown 2 years CAMBRAe 0.64 (0.50 0.77) 26:21
cohort Vatsalaya 72)
vidhayasharam
Nellore, India
Patients from the
University of
54 %
Retrospective California San 3810 (1772/ 1315 (624/ 0.5− 6 4− 24
Chaffee 2016 [36] Δdft>0 (713/ Coronal Unknown Unknown CAMBRAe 713:867
cohort Francisco Pediatric 2029) 687) years months
1315)
Dentistry clinic, the
USA
Patients from the
University of
58 %
Retrospective California San 18,004 4468 (2238/ 539 ± 257
Chaffee 2015 [37] >18 years ΔDFT>0 (2597/ Coronal Unknown Unknown CAMBRAe 2597:2298
cohort Francisco Pediatric (9560/8444) 2230) days
4468)
Dentistry clinic, the
USA
Children from Close to 0.50
Retrospective communities in the 466 214 35 % (75/
CAMBRAe

Journal of Dentistry 110 (2021) 103664

Christian 2020 [38] 1.5 years ΔCL>0 Coronal ICDAS-II No 4 years 75:142
cohort western corridor of (unknown) (unknown) 214) 0.50 (0.42 0.58)
Victoria, Australia
Children from public 2− 5 years CAMBRAe 0.73 (0.63 0.83)
Agouropoulos 2019 Prospective preschools in the 175 140 (mean: 3.3 53 % (74/ WHO criteria
ΔECC>0 Coronal No 2 years g
[39] cohort greater area of Athens, (unknown) (unknown) ± 0.8 140) and ICDAS CAT 0.50 (0.46 0.54)
Greece years)
Children from
Prospective 68 4.9 ± 0.7 51 % (35/
Kuru 2020 [40] kindergartens in 90 (45/45) ΔCL>0 Coronal Unknown Unknown 3 years CATh 35:51
cohort (unknown) years 68)
Izmir, Turkey
Prospective Patients registered at 1295 (676/ 982 (454/ 35 %
Petersson 2013 [41] 19 years ΔDFS>0 Coronal WHO criteria Yes 3 years PDS 0.71 (0.68 0.74) 344:560
cohort public dental clinics in 619) 528) (344/982)
(continued on next page)
 N. Su et al.
Table 1 (continued )
Source of date Participants Outcomes Models Outcomes

Studies Study Type Participants No. of No. of Age at Outcome event % of Location Criteria for Radiography Follow-up Models Discrimination O:E ratio
description (country participants participants baseline definition events of caries diagnosis of used for time validated (AUC)
and settings) at baseline at follow-up caries diagnosis of
(F/M) (F/M) caries

the Skåne region,

Sweden
Dutch
children
Dutch aged 7.5 Dutch children
population: years: 10 aged 7.5 years:
% (27/
270);
Dutch
children
270 aged 9.5
0.83 (0.75 0.91)
(unknown) years: 15
Children in
% (36/
Van Palenstein communities in Tiel 7.5, 9.5, Dentoprog-
Retrospective Unknown 235);
Helderman 2001 and Culemborg, the and 10.5 ΔD3S>3 Coronal Unknown Yes 4 years Method
cohort (unknown) Swiss
[42] Netherland, and in the years model
children
Canton of Zurich Swiss Dutch children
aged 7.5
population: aged 9.5 years:
years: 5 %
(8/154);
0.78 (0.70 0.86)
Swiss
Swiss children
children
aged 7.5 years:
aged 10.5
7

194 0.77 (0.60 0.94)

years: 19
Swiss children
% (36/
aged 10.5 years:
194)
0.77 (0.58 0.96)
Δnew carious Using dental
lesion>1 (the procedures,
Patients from a
number of new procedures 0.5− 4
vertically integrated
dental with an years
Brons-Piche 2019 Retrospective adult clinics of the 2449 447 (236/ 56.6 ± 55 %
procedures Coronal associated Yes (Mean: UMSD-CRA 247:272
[43] cohort University of (unknown) 211) 18.5years (247/447)
treatment with diagnosis of 2.2 ± 0.8
Michigan School of
an associated caries as a years)
Dentistry, the USA
diagnosis of surrogate for
caries) caries lesions
a
Full version of NUS-CRA.
b
Reduced version of NUS-CRA (mutans streptococci level and lactobacilli level were excluded).
c
Full version of Cariogram.
d
Reduced version of Cariogram (mutans streptococci level, saliva secretion, and saliva buffering capacity were excluded).

Journal of Dentistry 110 (2021) 103664

e
Full version of CAMBRA.
f
Reduced version of CAMBRA (saliva flow rate, mutans streptococci level and lactobacilli level were excluded).
g
Full version of CAT.
h
Reduced version of CAT (saliva flow rate and mutans streptococci level were excluded); F, female; M, male; AUC, Area under the receiver operating characteristic curve; O:E, total number of observed: total number of
expected events; DMFT/dmft, the sum of the number of decayed, missing due to caries, and filled teeth; DMFS/dmfs, the sum of the number of decay, missing due to caries, and filled surfaces of teeth; DFT/dft, the sum of
the number of decayed and filled teeth; DFS/dfs, the sum of the number of decayed and filled surfaces of teeth; D3S, the sum of the number of decayed surfaces of teeth with a discontinuity of integrity or clearly visible
undermined enamel in fissures and pits and on buccal and lingual smooth surfaces and approximal surfaces; CL, the sum of the number of caries lesions; ECC, the number of surfaces that changed from “sound” to “decay”;
WHO, World Health Organization; ICDAS, International Caries Detection and Assessment System; NUS-CRA, the National University of Singapore Caries Risk Assessment; CAMBRA, Caries Management By Risk
Assessment; CAT, Caries-risk Assessment Tool; PDS, Public Dental Service; UMSD-CRA, the University of Michigan School of Dentistry Caries Risk Assessment model.
N. Su et al.
Six studies [22,25,26,28,33,34] were pooled for calibration, irre­
spective of the risk of bias, and the pooled O:E was 0.84 (95 %CI: 0.59;
1.18; I2 = 93 %), which indicated that the calibration of the model was
acceptable (Table 3). When the studies with low and high risk of bias
were pooled separately, it indicated that the calibration of the model
based on the studies with low risk of bias was acceptable while that
based on the studies with high risk of bias was moderately overestimated
(Table 3).
3.4.3. Full version of Cariogram vs. reduced version of Cariogram
Five studies [22,25,26,28,33] assessed the predictive performance of
both full and reduced Cariogram. The pooled AUCs for the five studies
were 0.74 (95 %CI: 0.68; 0.79; I2 = 52 %) for full Cariogram and 0.73
(95 %CI: 0.66; 0.79; I2 = 52 %) for reduced Cariogram, respectively. The
pooled O:E for the five studies were 0.80 (95 %CI: 0.47; 1.38; I2 = 97 %)
for full Cariogram and 0.89 (95 %CI: 0.61; 1.31; I2 = 93 %) for reduced
Cariogram. Therefore, the discrimination between full and reduced
Cariogram is similar, while reduced Cariogram seems to have better
calibration than full Cariogram.
3.4.4. Full version of CAMBRA
The discrimination of CAMBRA was assessed in three studies [35,38,
39] and the reported AUC was 0.62 (95 %CI: 0.32; 0.85; I2 = 79 %)
(Table 3), which indicated that the discrimination of the model was low.
All the three studies were considered high risk of bias.
Five studies [22,35–38] were pooled for calibration, and the pooled
O:E was 0.83 (95 %CI: 0.54; 1.28; I2 = 99 %), which indicated that the
calibration of the model was acceptable. When the studies with low and
high risk of bias were pooled separately, it indicated that the calibration
of the model based on the study with low risk of bias was moderately
overestimated while that based on the studies with high risk of bias was
acceptable (Table 3).
3.4.5. Reduced version of CAMBRA
No study assessed the discrimination of reduced CAMBRA. The O:E
ratio based on one study [22] carrying a low risk of bias was 0.52 (95 %
CI: 0.47; 0.59) (Table 3), which indicates that the calibration of the
model was poor (Table 3).
3.4.6. Other externally validated CRA models
The NUS-CRA model was externally validated in only one study [22]
carrying a low risk of bias. The discrimination and calibration of both
full version and reduced version of NUS-CRA was acceptable (Table 3).
The full version of the CAT model was externally validated in two
studies [22,39]. One study [39] carrying a high risk of bias assessed the
discrimination of the model and showed that the discrimination of the
model was low (Table 3). The other study [22] carrying a low risk of bias
assessed the calibration of the model and showed that the calibration of
the model was poor (Table 3).
The reduced version of the CAT model was externally validated in
two studies [22,40]. Both of the two studies did not assess the
discrimination of the model. The calibration based on the two studies
was poor (Table 3). The PDS model was externally validated in one study
[41] carrying a high risk of bias. It showed that the discrimination of the
model was acceptable but the calibration was poor (Table 3).
The discrimination of the Dentoprog model was externally validated
in one study [42] carrying a high risk of bias. In this study, 7.5- and
9.5-year-old Dutch children were recruited for the validation of the
model. This study also used historical data collected from 7.5- and
10.5-year-old Swiss children for the validation. The pooled AUC based
on both Dutch and Swiss children in this study showed that the
discrimination of the model was acceptable (Table 3). The O:E ratio of
Dentoprog-Method model cannot be obtained from any included study.
The UMSD-CRA model was externally validated in one study [43]
carrying a high risk of bias. The study did not assess the discrimination
of the model. The O:E ratio was acceptable (Table 3).
8
Journal of Dentistry 110 (2021) 103664
4. Discussion
In this systematic review, we included seven CRA models which have
been externally validated in at least one population: Cariogram, CAM­
BRA, Dentoprog, NUS-CRA, CAT, PDS, and UMSD-CRA models. Gener­
ally, the AUCs of the CRA models included in the present review ranged
from 0.50 to 0.88, showing that most of the CRA models had acceptable
to good discrimination except for full CAMBRA and full CAT (Table 3).
The O:E ratio of the CRA models ranged from 0.38 to 1.00, indicating
that most of the included models tended to overestimate the prediction
of caries. This is also a common problem for most clinical prediction
models in general [44]. However, due to the limited number of included
studies, in particular, the studies with low risk of bias, the evidence on
the predictive performance of the included CRA models except for
Cariogram was limited.
So far, several systematic reviews on the predictive performance of
the CRA models have been published [1,6,45]. However, those reviews
were either qualitative or only reported sensitivity, specificity, and
predictive values of CRA models. Most of the systematic reviews [1,6]
reported that Cariogram had limited predictive performance based on
the sensitivity and specificity. These conventional diagnostic classifica­
tion measures require one or more thresholds, and therefore are known
to lead to loss of information because the entire range of predicted
probabilities of the model is not fully used [14]. Besides, the use of
classification measures may cause substantial bias because the thresh­
olds can be data-driven instead of pre-specified on clinical grounds [14].
The present study clearly goes beyond these conventional diagnostic
classification measures and makes use of the entire range of predicted
probabilities and showed the external validated predictive performance
of the Cariogram, more specifically discrimination and calibration, to be
acceptable. The evidence on predictive performance of other CRA
models was scarce in the previous reviews due to an insufficient number
of validation studies. To our knowledge, this is the first quantitative
systematic review on the predictive performance of the existing CRA
models using discrimination and calibration rather than the classifica­
tion measures. This is also the first systematic review that compared the
predictive performance between the full Cariogram and the reduced
Cariogram.
In interpreting the findings of the present systematic review and
meta-analysis, some limitations should be taken into consideration.
Firstly, the number of the included studies for the CRA models, other
than the Cariogram, was small. Second, there is large variation in the
design of the included studies, notably with respect to demographics of
study populations with in particular the age groups, the definition and
measures for outcome events, and follow-up period, which may cause
various baseline risks of incident caries in different study populations.
The performance of a prediction model may vary considerably with the
baseline risk [46] and this may have caused the observed rather large
heterogeneity in the meta-analyses. For example, some included studies
predicted the incident caries of primary teeth for children, while other
studies predicted the permanent teeth for adults or the elderly. Primary
teeth are more vulnerable to caries than permanent teeth because
compared with permanent enamel, primary enamel has faster lesion
progression due to its higher degree of porosity, smaller thickness, and
greater susceptibility to acid-induced demineralization due to its lower
degree of mineralization and remarkable mechanical and dynamic
characteristics [47–49]. This may indicate that the baseline risk of
incident caries of the subjects is various with ages across included
studies. Besides, the follow-up time varies across the included studies,
ranging from 4 months to 5 years. The subjects with longer follow-up
time may have higher risk to have incident caries than those with
shorter follow-up time [50]. Therefore, the baseline risk varies with
different follow-up time. Some studies regarded the increment of
DMFS/dmfs or DMFT/dmft as the outcome while other studies regarded
the increment of DFS/dfs or DFT/dft as the outcome. The subjects may
have higher risk to have incident caries when the missing teeth were
N. Su et al. Journal of Dentistry 110 (2021) 103664

Table 2
Summary of risk of bias assessment of included studies based on PROBAST.
RoB Applicability Overall
Study
Participants Predictors Outcome Analysis Participants Predictors Outcome RoB Applicability

Validation studies
Gao 2010 [10] + + + + + + + + +
Gao 2013 [22] + + + + + + + + +
Petersson 2015 [24] + + + – + + + – +
Petersson 2010 [25] + + + + + + + + +
Hayes 2017 [26] + + + – + + + – +
Petersson 2003 [27] + + + – + + + – +
Dou 2018 [28] + + + – + + + – +
Campus 2012 [29] + + + + + + + + +
Sudhir 2017 [30] + + + – + + + – +
Garg 2018 [31] + + + + + + + + +
Dolic 2020 [32] + + + – + + + – +
Birpou 2019 [33] + + + – + + + – +
Enerbäck 2020 [34] + + + – + + + – +
Sudhir 2016 [35] + + + – + + + – +
Chaffee 2016 [36] + + + – + + + – +
Chaffee 2015 [37] + + + – + + + – +
Christian 2020 [38] + + + – + + + – +
Agouropoulos 2019 [39] + + + – + + + – +
Kuru 2020 [40] + + ? – + + + – +
Petersson 2013 [41] + + + – + + + – +
Van Palenstein Helderman 2001 [42] + – – ? + + + – +
Brons-Piche 2019 [43] + + + – + + + – +

RoB, risk of bias; +, low risk of bias; -, high risk of bias;?, unclear risk of bias.

considered a component of the outcome. Therefore, the baseline risk
may vary with different outcome measures. The I2 of the meta-analyses
for Cariogram and CAMBRA ranged between 44 % and 99 %. However,
validation studies typically differ in design, execution, and thus
case-mix. So, the variation between the results of the validation studies
is unlikely to occur by chance only, but this does not necessarily imply
that the reported pooled estimates are biased [51,52]. Because of this,
the meta-analysis for the validation studies can allow for the presence of
heterogeneity and produce a summary result that quantifies the average
performance across studies [13]. The meta-analysis with random effect
models is therefore recommended to pool the validation studies [13].
That is why meta-analyses were still performed even though the het­
erogeneity across included studies was relatively large in the present
review. Third, most of the included studies had a high risk of bias based
on the PROBAST. The high risk of bias was due to the insufficient sample
size, large loss to follow-up, and inappropriate handling of missing data
of the included studies. The subgroup meta-analyses showed that the
risk of bias may have some impact on the predictive performance of the
models because small to moderate difference in the AUC and O:E ratio
was found between the studies with low risk of bias and high risk of bias
in Cariogram and CAMBRA (Table 3). For instance, the calibration of the
reduced Cariogram was acceptable based on the studies with low risk of
bias [22,25] with O:E ratio of 1.05, while the calibration was moderately
overestimated based on the studies with high risk of bias [26,28,33,34]
with O:E ratio of 0.74. The main reason for this discrepancy may be that
all the studies with high risk of bias included insufficient number of
events (<100), which may lead to exaggerated and misleading perfor­
mance of the prediction models [53]. The inaccurate estimation of the
true performance is more likely in the validation studies with a low
number of events [54]. Overestimation of the calibration is more likely
to be captured when the number of events is lower than 100 based on a
simulation study [54]. Furthermore, the calibration of full CAMBRA was
found to be moderately overestimated with O:E ratio of 0.68 based on
one study with low risk of bias [22], while the calibration was accept­
able with O:E ratio of 0.88 based on the four studies with high risk of
bias [35–38]. This indicated that the calibration based on the studies
with high risk of bias seemed to be better than the calibration based on
the study with low risk of bias. The main reason for this may be that
three of the four studies with high risk of bias [36,36,37,38] had a very
high rate of loss to follow-up ranging from 54 % to 75 %. Omitting those
participants from the analysis may cause biased predictor-outcome as­
sociations and biased predictive performance of validated models if the
participants were not missing at random [14]. For example, if the par­
ticipants whose predictor values were unclear are excluded from the
analysis, it is likely to yield a study sample with participants in the ex­
tremes of the predictor range [55]. This can make the performance of the
models biased and better than the actual performance. However, most of
the I2 values of the subgroup meta-analyses did not significantly
decrease compared with the I2 values of the overall meta-analyses
including all the studies. For example, as for the discrimination of full
Cariogram, the I2 value of the overall meta-analysis was 94 % while that
of the subgroup meta-analysis for low risk of bias was 98 %. This indi­
cated that risk of bias may not be the main reason for the large het­
erogeneity in the meta-analyses.
This review shows that, to date, for all the available CRA models
except for Cariogram, the evidence on the performance in prediction of
caries increment at follow-up is scant due to the limited number of
studies. The overall predictive performance for both full and reduced
Cariogram is acceptable when all the studies were included regardless of
the risk of bias. Besides, the predictive performance is still acceptable
when only studies with low risk of bias or high risk of bias were
considered separately. This indicated that both full and reduced Cario­
gram may be reliable CRA models for prediction of caries increment at
follow-up in clinical practices for dental patients and communities for
general populations and can help healthcare providers make decisions
on whether individuals need more dental check-ups, more frequent
recall appointments, or more comprehensive preventive means for
caries.
When the predictive performance of full Cariogram was compared
with that of reduced Cariogram, reduced Cariogram had slightly better
calibration than full Cariogram while the discrimination of the full and
reduced Cariogram was very similar. This indicated that the biological
predictors including saliva secretion rate, saliva buffering capacity,
lactobacilli (LB) count, and mutans streptococci (MS) count do not have
added values to the predictive performance of Cariogram. Hence,
considering that the reduced Cariogram has better predictive perfor­
mance and poses less burdens in terms of time and resources to in­
dividuals, it may be well placed when used in clinical practice or

9
N. Su et al. Journal of Dentistry 110 (2021) 103664

Table 3
The overall results and the results for the subgroups based on meta-analyses.
Overall Low risk of bias High risk of bias
2 2
CRA models No. of No. of Pooled AUC I No. of No. of Pooled AUC I No. of No. of Pooled AUC I2
studies subjects (95 %CI) studies subjects (95 %CI) studies subjects (95 %CI)

Discrimination
Full Cariogram 8 3962 0.78 (0.68 94 4 3314 0.82 (0.58 98 4 648 0.72 (0.62 39
0.85) % 0.93) % 0.80) %
Reduced 6 1744 0.72 (0.67 44 2 877 0.74 (0.45 3% 4 867 0.71 (0.60 53
Cariogram 0.77) % 0.91) 0.80) %
Full CAMBRA 3 426 0.62 (0.32 79 0 NA NA NA 3 426 0.62 (0.32 79
0.85) % 0.85) %
Reduced 0 NA NA NA 0 NA NA NA 0 NA NA NA
CAMBRA
Dentoprog 1 853 0.79 (0.74 NA 0 NA NA NA 1 853 0.79 (0.74 NA
model 0.84) 0.84)
Full NUS-CRA 1 485 0.88 (0.84 NA 1 485 0.88 (0.84 NA 0 NA NA NA
0.91) 0.91)
Reduced NUS- 1 485 0.85 (0.81 NA 1 485 0.85 (0.81 NA 0 NA NA NA
CRA 0.88) 0.88)
Full CAT 1 140 0.50 (0.46 NA 0 NA NA NA 1 140 0.50 (0.46 NA
0.54) 0.54)
Reduced CAT 0 NA NA NA 0 NA NA NA 0 NA NA NA
PDS 1 982 0.71 (0.68 NA 0 NA NA NA 1 982 0.71 (0.68 NA
0.74) 0.74)
UMSD-CRA 0 NA NA NA 0 NA NA NA 0 NA NA NA
Calibration
Full Cariogram 12 5671 0.91 (0.72 97 5 3813 0.89 (0.71 92 7 1858 0.91 (0.59 98
1.14) % 1.12) % 1.41) %
Reduced 6 1744 0.84 (0.59 93 2 877 1.05 (0.43 55 4 867 0.74 (0.43 92
Cariogram 1.18) % 2.59) % 1.27) %
Full CAMBRA 5 6554 0.83 (0.54 99 1 485 0.68 (0.60 NA 4 6069 0.88 (0.48 99
1.28) % 0.76) 1.61) %
Reduced 1 485 0.52 (0.47 NA 1 485 0.52 (0.47 NA 0 NA NA NA
CAMBRA 0.59) 0.59)
Dentoprog 0 NA NA NA 0 NA NA NA 0 NA NA NA
model
Full NUS-CRA 1 485 0.97 (0.86 NA 1 485 0.97 (0.86 NA 0 NA NA NA
1.09) 1.09)
Reduced NUS- 1 485 1.00 (0.89 NA 1 485 1.00 (0.89 NA 0 NA NA NA
CRA 1.12) 1.12)
Full CAT 1 485 0.38 (0.33 NA 1 485 0.38 (0.33 NA 0 NA NA NA
0.42) 0.42)
Reduced CAT 2 553 0.51 (0.01 95 1 485 0.38 (0.34 NA 1 68 0.69 (0.54 NA
21.22) % 0.43) 0.86)
PDS 1 982 0.61 (0.56 NA 0 NA NA NA 1 982 0.61 (0.56 NA
0.67) 0.67)
UMSD-CRA 1 447 0.91 (0.84 NA 0 NA NA NA 1 447 0.91 (0.84 NA
0.99) 0.99)

NA, not applicable; NUS-CRA, the National University of Singapore Caries Risk Assessment; CAMBRA, Caries Management By Risk Assessment; CAT, Caries-risk
Assessment Tool; PDS, Public Dental Service; UMSD-CRA, the University of Michigan School of Dentistry Caries Risk Assessment model; AUC, Area under the
receiver operating characteristic curve; O:E, total number of observed: total number of expected events.

population studies.
The present review indicated that Cariogram is a reliable CRA tool
that healthcare providers can rely on to classify populations or patients
into different levels of caries risk in clinical practice or communities and
to develop different prevention plans for caries for individuals with
different risk levels. Eventually, this may contribute to achieving good
dental health outcomes of individuals. Furthermore, the present review
can provide policymakers with the information that Cariogram is the
CRA model that can be relied on when they make public policies in
targeting people at moderate or high risk and justifying the reallocation
of resources from people at low risk to those at moderate and high risk.
This may help to reduce potential redundant provision of services for
those at low risk and enhance the efficiency of oral health care system.
However, due to the limited number of high-quality validation
studies so far, high-quality studies with low risk of bias are needed to
further provide evidence on the external validity of the other CRA
models before they are routinely applied in CRA management. To reduce
the risk of bias of the future validation studies on CRA models, re­
searchers are suggested to include a sufficient number of outcome events
(>100), minimize the risk of loss to follow-up by optimizing the study
designs, and use appropriate techniques like multiple imputation to
handle the missing values when analyzing the data. Besides, it is rec­
ommended to further assess the added values of biological predictors to
the predictive performance of other CRA models, such as CAMBRA, CAT,
and NUS-CRA.
5. Conclusions
Both full and reduced Cariogram are found to be reasonably reliable
tools for prediction of caries in individuals in clinical practices and
communities. The reduced Cariogram shows slightly better performance
and poses less burden in terms of time and resources to individuals than
the full Cariogram. Therefore, the reduced Cariogram could be more
recommended to use for caries prediction for individuals than the full
Cariogram. However, the evidence on the predictive performance of the
other CRA models, notably CAMBRA, NUS-CRA, CAT, Dentoprog, PDS,
and UMSD-CRA, is limited so far due to the insufficient number of high-
quality studies included. Therefore, more high-quality research with low

10
N. Su et al.
risk of bias on external validation of CRA models is required to further
assess their predictive performance in the future.
Funding
This research did not receive any specific grant from funding
agencies in the public, commercial, or not-for-profit sectors.
Declaration of Competing Interest
The authors declare that they have no known competing financial
interests or personal relationships that could have appeared to influence
the work reported in this paper.
Acknowledgements
None.
Appendix A. Supplementary data
Supplementary material related to this article can be found, in the
online version, at doi:https://doi.org/10.1016/j.jdent.2021.103664.
References
[1] M.G. Cagetti, G. Bontà, F. Cocco, P. Lingstrom, L. Strohmenger, G. Campus, Are
standardized caries risk assessment models effective in assessing actual caries
status and future caries increment? A systematic review, BMC Oral Health 18
(2018) 123, https://doi.org/10.1186/s12903-018-0585-4.
[2] T. Vos, A.D. Flaxman, M. Naghavi, R. Lozano, C. Michaud, M. Ezzati, et al., Years
lived with disability (YLDs) for 1160 sequelae of 289 diseases and injuries
1990–2010: a systematic analysis for the Global Burden of Disease Study 2010,
Lancet 380 (2012) 2163–2196, https://doi.org/10.1016/S0140-6736(12)61729-2.
[3] GBD 2017 Disease and Injury Incidence and Prevalence Collaborators, Global,
regional, and national incidence, prevalence, and years lived with disability for
354 diseases and injuries for 195 countries and territories, 1990–2017: a systematic
analysis for the Global Burden of Disease Study 2017, Lancet 392 (2018)
1789–1858, https://doi.org/10.1016/S0140-6736(18)32279-7.
[4] M. Fontana, D.T. Zero, Assessing patients` caries risk, J. Am. Dent. Assoc. 137
(2006) 1231–1239, https://doi.org/10.14219/jada.archive.2006.0380.
[5] S. Twetman, M. Fontana, Patient caries risk assessment, Monogr. Oral Sci. 21
(2009) 91–101, https://doi.org/10.1159/000224214.
[6] M. Tellez, J. Gomez, I. Pretty, R. Ellwood, A.I. Ismail, Evidence on existing caries
risk assessment systems: are they predictive of future caries? Community Dent.
Oral Epidemiol. 41 (2013) 67–78, https://doi.org/10.1111/cdoe.12003.
[7] D. Bratthall, G. Hänsel Petersson, Cariogram–a multifactorial risk assessment
model for a multifactorial disease, Community Dent. Oral Epidemiol. 33 (2005)
256–264, https://doi.org/10.1111/j.1600-0528.2005.00233.x.
[8] J.D.B. Featherstone, B.W. Chaffee, The evidence for caries management by risk
assessment (CAMBRA®), Adv. Dent. Res. 29 (2018) 9–14, https://doi.org/
10.1177/0022034517736500.
[9] American Academy of Pediatric Dentistry Council on Clinical Affairs, Policy on use
of a caries-risk assessment tool (CAT) for infants, children, and adolescents,
Pediatr. Dent. 30 (2008-2009) 29–33.
[10] X.L. Gao, C.Y. Hsu, Y. Xu, H.B.T. Loh, D. Koh, Building caries risk assessment
models for children, J. Dent. Res. 89 (2010) 637–643, https://doi.org/10.1177/
0022034510364489.
[11] G. Hänsel Petersson, E. Ericson, P.E. Isberg, S. Twetman, Caries risk assessment in
young adults using Public Dental Service guidelines and the Cariogram–a
comparative study, Acta Odontol. Scand. 71 (2013) 534–540, https://doi.org/
10.3109/00016357.2013.788734.
[12] T.N. Imfeld, M. Steiner, G.D. Menghini, T.M. Marthaler, Prediction of future caries
increments for children in a school dental service and in private practice, J. Dent.
Educ. 59 (1995) 941–944.
[13] T.P. Debray, J.A. Damen, K.I. Snell, J. Ensor, L. Hooft, J.B. Reitsma, R.D. Riley, K.
G. Moons, A guide to systematic review and meta-analysis of prediction model
performance, BMJ 356 (2017) i6460, https://doi.org/10.1136/bmj.i6460.
[14] K.G. Moons, R.F. Wolff, R.D. Riley, P.F. Whiting, M. Westwood, G.S. Collins, J.
B. Reitsma, J. Kleijnen, S. Mallett, PROBAST: A tool to assess risk of bias and
applicability of prediction model studies: explanation and elaboration, Ann. Intern.
Med. 170 (2019) W1–W33, https://doi.org/10.7326/m18-1377.
[15] I. Mejàre, S. Axelsson, G. Dahlén, I. Espelid, A. Norlund, S. Tranæus, S. Twetman,
Caries risk assessment. A systematic review, Acta Odontol. Scand. 72 (2014)
81–91, https://doi.org/10.3109/00016357.2013.822548.
[16] A. Senneby, I. Mejàre, N.E. Sahlin, G. Svensäter, M. Rohlin, Diagnostic accuracy of
different caries risk assessment methods. A systematic review, J. Dent. 43 (2015)
1385–1393, https://doi.org/10.1016/j.jdent.2015.10.011.
11
Journal of Dentistry 110 (2021) 103664
[17] D. Moher, A. Liberati, J. Tetzlaff, D.G. Altman, PRISMA Group, Preferred reporting
items for systematic reviews and meta-analyses: the PRISMA statement, PLoS Med.
6 (2009), e1000097, https://doi.org/10.1371/journal.pmed.1000097.
[18] S. Briscoe, A. Bethel, M. Rogers, Conduct and reporting of citation searching in
Cochrane systematic reviews: a cross-sectional study, Res. Synth. Methods 11
(2020) 169–180, https://doi.org/10.1002/jrsm.1355.
[19] K.G. Moons, J.A.H. de Groot, W. Bouwmeester, Y. Vergouwe, S. Mallett, D.
G. Altman, J.B. Reitsma, G.S. Collins, Critical appraisal and data extraction for
systematic reviews of prediction modelling studies: the CHARMS checklist, PLoS
Med. 11 (2014), e1001744, https://doi.org/10.1371/journal.pmed.1001744.
[20] E.W. Steyerberg, A.J. Vickers, N.R. Cook, T. Gerds, M. Gonen, N. Obuchowski, M.
J. Pencina, M.W. Kattan, Assessing the performance of prediction models: a
framework for traditional and novel measures, Epidemiology 21 (2010) 128–138,
https://doi.org/10.1097/ede.0b013e3181c30fb2.
[21] S. Siregar, R.H. Groenwold, F. de Heer, M.L. Bots, Y. van der Graaf, L.A. van
Herwerden, Performance of the original EuroSCORE, Eur. J. Cardiothorac. Surg. 41
(2012) 746–754, https://doi.org/10.1093/ejcts/ezr285.
[22] X. Gao, I. Di Wu, E.C. Lo, C.H. Chu, C.Y. Hsu, M.C. Wong, Validity of caries risk
assessment programmes in preschool children, J. Dent. 41 (2013) 787–795,
https://doi.org/10.1016/j.jdent.2013.06.005.
[23] D.W. Hosmer, S. Lemeshow, Applied Logistic Regression, 2nd ed., John Wiley &
Sons, New York, 2000.
[24] G.H. Petersson, S. Twetman, Caries risk assessment in young adults: a 3 year
validation of the Cariogram model, BMC Oral Health 15 (2015) 17, https://doi.
org/10.1186/1472-6831-15-17.
[25] G.H. Petersson, P.E. Isberg, S. Twetman, Caries risk assessment in school children
using a reduced Cariogram model without saliva tests, BMC Oral Health 10 (2010)
5, https://doi.org/10.1186/1472-6831-10-5.
[26] M. Hayes, C. Da Mata, G. McKenna, F.M. Burke, P.F. Allen, Evaluation of the
Cariogram for root caries prediction, J. Dent. 62 (2017) 25–30, https://doi.org/
10.1016/j.jdent.2017.04.010.
[27] G. Hänsel Petersson, S. Fure, D. Bratthall, Evaluation of a computer-based caries
risk assessment program in an elderly group of individuals, Acta Odontol. Scand.
61 (2003) 164–171, https://doi.org/10.1080/00016350310002261.
[28] L. Dou, J. Luo, X. Fu, Y. Tang, J. Gao, D. Yang, The validity of caries risk assessment
in young adults with past caries experience using a screening Cariogram model
without saliva tests, Int. Dent. J. 68 (2018) 221–226, https://doi.org/10.1111/
idj.12378.
[29] G. Campus, M.G. Cagetti, S. Sale, G. Carta, P. Lingström, Cariogram validity in
schoolchildren: a two-year follow-up study, Caries Res. 46 (2012) 16–22, https://
doi.org/10.1159/000334932.
[30] K.K. Kanupuru, K.M. Sudhir, N. Fareed, E. Srikanth, N.T. Chaitra, Validation of
Cariogram as a tool for caries risk prediction among 12-year-old institutionalized
children - a longitudinal follow-up study, OHDM 16 (2017) 1–8.
[31] A. Garg, M. Madan, P. Dua, S. Saini, R. Mangla, P. Singhal, A. Dupper, Validating
the usage of Cariogram in 5- and 12-year-old school-going children in Paonta
Sahib, Himachal Pradesh, India: a 12-month prospective study, Int. J. Clin. Pediatr.
Dent. 11 (2018) 110–115, https://doi.org/10.5005/jp-journals-10005-1495.
[32] O. Dolic, M. Obradovic, Z. Kojic, N. Trtic, S. Sukara, N. Knezevic, V. Veselinovic,
Validation of Cariogram in caries prediction in women and their children 4 years
after pregnancy-Longitudinal study, Risk Manag. Healthc. Policy 13 (2020)
549–557, https://doi.org/10.2147/rmhp.s243907.
[33] E. Birpou, A. Agouropoulos, S. Twetman, K. Kavvadia, Validation of different
Cariogram settings and factor combinations in preschool children from areas with
high caries risk, Int. J. Paediatr. Dent. 29 (2019) 448–455, https://doi.org/
10.1111/ipd.12476.
[34] H. Enerbäck, P. Lingström, M. Möller, C. Nylén, C.Ö. Bresin, I.Ö. Ros,
A. Westerlund, Validation of aries risk assessment methods in orthodontic patients,
Am. J. Orthod. Dentofacial Orthop. 158 (2020) 92–101, https://doi.org/10.1016/j.
ajodo.2019.07.017.
[35] K.M. Sudhir, K.K. Kanupuru, N. Fareed, P. Mahesh, K. Vandana, N.T. Chaitra,
CAMBRA as a tool for caries risk prediction among 12- to 13-year-old
institutionalised children - a longitudinal follow-up study, Oral Health Prev. Dent.
14 (2016) 355–362, https://doi.org/10.3290/j.ohpd.a35621.
[36] B.W. Chaffee, J.D. Featherstone, S.A. Gansky, J. Cheng, L. Zhan, Caries risk
assessment item importance: risk designation and caries status in children under
age 6, JDR Clin. Trans. Res. 1 (2016) 131–142, https://doi.org/10.1177/
2380084416648932.
[37] B.W. Chaffee, J. Cheng, J.D. Featherstone, Baseline caries risk assessment as a
predictor of caries incidence, J. Dent. 43 (2015) 518–524, https://doi.org/
10.1016/j.jdent.2015.02.013.
[38] B. Christian, H. Calache, G. Adams, M. Hall, S. Dashper, L. Gibbs, M. Gussy,
A methodological study to assess the measurement properties (reliability and
validity) of a caries risk assessment tool for young children, J. Dent. 95 (2020),
103324, https://doi.org/10.1016/j.jdent.2020.103324.
[39] A. Agouropoulos, E. Birpou, S. Twetman, K. Kavvadia, Validation of three caries
risk assessment tools for preschool children from areas with high caries prevalence,
Paediatr. Dent. 15 (2019) 391–399.
[40] E. Kuru, E. Eden, Success of two caries risk assessment tools in children: a pilot
study within a 3-year follow-up, Int. Q. Community Health Educ. 40 (2020)
317–320, https://doi.org/10.1177/0272684x19892356.
[41] G. Hänsel Petersson, E. Ericson, P.E. Isberg, S. Twetman, Caries risk assessment in
young adults: a 3-year validation of clinical guidelines used in Public Dental
Service, Acta Odontol. Scand. 71 (2013) 1645–1650, https://doi.org/10.3109/
00016357.2013.788734.
N. Su et al.
[42] W.H. Helderman, J. Mulder, M.A. van’T Hof, G.J. Truin, Validation of a Swiss
method of caries prediction in Dutch children, Community Dent. Oral Epidemiol.
29 (2001) 341–345, https://doi.org/10.1034/j.1600-0528.2001.290503.x.
[43] E. Brons-Piche, G.J. Eckert, M. Fontana, Predictive validity of a caries risk
assessment model at a dental school, J. Dent. Educ. 83 (2019) 144–150, https://
doi.org/10.21815/jde.019.017.
[44] C.D. van Steenbeek, M.C. van Maaren, S. Siesling, A. Witteveen, X.A.A.M. Verbeek,
H. Koffijberg, Facilitating validation of prediction models: a comparison of manual
and semi-automated validation using registry-based data of breast cancer patients
in the Netherlands, BMC Med. Res. Methodol. 19 (2019) 117, https://doi.org/
10.1186/s12874-019-0761-5.
[45] M.R. Jørgensen, S. Twetman, A systematic review of risk assessment tools for early
childhood caries: is there evidence? Eur. Arch. Paediatr. Dent. 21 (2020) 179–184,
https://doi.org/10.1007/s40368-019-00480-2.
[46] I. Ahmed, T.P. Debray, K.G. Moons, R.D. Riley, Developing and validating risk
prediction models in an individual participant data meta-analysis, BMC Med. Res.
Methodol. 14 (2014) 3, https://doi.org/10.1186/1471-2288-14-3.
[47] R.J.M. Lynch, The primary and mixed dentition, post-eruptive enamel maturation
and dental caries: a review, Int. Dent. J. 63 (2013) 3–13, https://doi.org/10.1111/
idj.12076.
[48] L.J. Wang, R. Tang, T. Bonstein, P. Bush, G.H. Nancollas, Enamel demineralization
in primary and permanent teeth, J. Dent. Res. 85 (2006) 359–363, https://doi.org/
10.1177/154405910608500415.
[49] A.B. Sønju Clasen, B. Ogaard, H. Duschner, J. Ruben, J. Arends, T. Sönju,
Permanent enamel in situ examined by microradiography and confocal laser
12
Journal of Dentistry 110 (2021) 103664
scanning microscopy, Adv. Dent. Res. 11 (1997) 442–447, https://doi.org/
10.1177/08959374970110041001.
[50] J.M. Broadbent, L.A. Foster Page, W.M. Thomson, R. Poulton, Permanent dentition
caries through the first half of life, Br. Dent. J. 215 (2013) E12, https://doi.org/
10.1038/sj.bdj.2013.991.
[51] T.P. Debray, Y. Vergouwe, H. Koffijberg, D. Nieboer, E.W. Steyerberg, K.G. Moons,
A new framework to enhance the interpretation of external validation studies of
clinical prediction models, J. Clin. Epidemiol. 68 (2015) 279–289, https://doi.org/
10.1016/j.jclinepi.2014.06.018.
[52] Y. Vergouwe, K.G. Moons, E.W. Steyerberg, External validity of risk models: use of
benchmark values to disentangle a case-mix effect from incorrect coefficients, Am.
J. Epidemiol. 172 (2010) 971–980, https://doi.org/10.1093/aje/kwq223.
[53] G.S. Collins, J.A. de Groot, S. Dutton, O. Omar, M. Shanyinde, A. Tajar, M. Voysey,
R. Wharton, L.M. Yu, K.G. Moons, D.G. Altman, External validation of multivariate
prediction models: a systematic review of methodological conduct and reporting,
BMC Med. Res. Methodol. 14 (2014) 40, https://doi.org/10.1186/1471-2288-14-
40.
[54] G.S. Collins, E.O. Ogundimu, D.G. Altman, Sample size considerations for the
external validation of a multivariate prognostic model: a resampling study, Stat.
Med. 35 (2016) 214–226, https://doi.org/10.1002/sim.6787.
[55] B. Shinkins, M. Thompson, S. Mallett, R. Perera, Diagnostic accuracy studies: how
to report and analyse inconclusive test results, BMJ 346 (2013) f2778, https://doi.
org/10.1136/bmj.f2778.