Ultrasound Obstet Gynecol 2022; 59: 668–676

Published online 8 April 2022 in Wiley Online Library (wileyonlinelibrary.com). DOI: 10.1002/uog.24777

Performance of IOTA Simple Rules, Simple Rules risk

assessment, ADNEX model and O-RADS in differentiating
between benign and malignant adnexal lesions in North
American women
A. K. HIETT1 , J. D. SONEK1 , M. GUY2 and T. J. REID2
1
Boonshoft School of Medicine, Wright State University, Division of Maternal–Fetal Medicine, Fetal Medicine Foundation, Dayton, OH,
USA; 2 University of Cincinnati, Department of Obstetrics and Gynecology, Division of Oncology and Advanced Pelvic Surgery, Cincinnati,
OH, USA

K E Y W O R D S: ADNEX; IOTA; malignancy; O-RADS; ovary; Simple Rules; ultrasound

CONTRIBUTION Methods This was a single-center diagnostic accuracy

What are the novel findings of this work?
This is the first diagnostic accuracy study comparing
the International Ovarian Tumor Analysis (IOTA)
models and Ovarian-Adnexal Reporting and Data
System (O-RADS) in patients from a single gynecologic
ultrasound center in North America. It compared their
performance in differentiating preoperatively between
benign and malignant adnexal lesions in the same
cohort of patients. Both the IOTA models and O-RADS
performed well, with similar sensitivity. The three IOTA
models had superior specificity compared to O-RADS in
differentiating between benign and malignant lesions.
What are the clinical implications of this work?
When the IOTA models are used to differentiate
preoperatively between benign and malignant adnexal
lesions, the use of magnetic resonance imaging, as
recommended for O-RADS Categories 3 and 4, may be
unnecessary.
ABSTRACT
Objectives To apply the International Ovarian Tumor
Analysis (IOTA) Simple Rules (SR), the IOTA Simple
Rules risk assessment (SRR), the IOTA Assessment of
Different NEoplasias in the adneXa (ADNEX) model
and the Ovarian-Adnexal Reporting and Data System
(O-RADS) in the same cohort of North American
patients and to compare their performance in preoperative
discrimination between benign and malignant adnexal
lesions.
study, performed between March 2018 and February
2021, which included 150 women with an adnexal
lesion. Using the ADNEX model, lesions were classified
prospectively, whereas the SR, SRR assessment and
O-RADS were applied retrospectively. Surgery with
histological analysis was performed within 6 months of
the ultrasound exam. Sensitivity and specificity were
determined for each testing modality and the performance
of the different modalities was compared.
Results Of the 150 women, 110 (73.3%) had a benign
ovarian tumor and 40 (26.7%) had a malignant
tumor. The mean risk of malignancy generated by
the ADNEX model without CA 125 was significantly
higher in malignant vs benign lesions (63.3% vs 11.8%)
and the area under the receiver-operating-characteristics
curve (AUC) of the ADNEX model for differentiating
between benign and malignant adnexal masses at the
time of ultrasound examination was 0.937. The mean
risk of malignancy generated by SRR assessment was
also significantly higher in malignant vs benign lesions
(74.1% vs 15.9%) and the AUC was 0.941. To compare
the ADNEX model, SRR assessment and O-RADS, the
malignancy risk threshold was set at ≥ 10%. This cut-off
differentiates O-RADS low-risk categories (Category ≤ 3)
from intermediate-to-high-risk categories (Categories 4
and 5). At this cut-off, the sensitivity of the ADNEX model
was 97.5% (95% CI, 85.3%–99.9%) and the specificity
was 63.6% (95% CI, 53.9%–72.4%), and, for the SRR
model, the sensitivity was 100% (95% CI, 89.1%–100%)
and the specificity was 51.8% (95% CI, 42.1%–61.4%).
In the 113 cases to which the SR could be applied, the

Correspondence to: Dr A. K. Hiett, 1 Wyoming Street, Maternal–Fetal Medicine, Berry Building Ground, Dayton, OH 45409, USA (e-mail:
akhiett@mac.com)
Accepted: 6 September 2021

© 2022 International Society of Ultrasound in Obstetrics and Gynecology. ORIGINAL PAPER

IOTA models and O-RADS in preoperative evaluation of adnexal lesions
sensitivity was 100% (95% CI, 81.5%–100%) and the
specificity was 95.6% (95% CI, 88.5%–98.6%). If the
remaining 37 cases, which were inconclusive under SR,
were designated ‘malignant’, the sensitivity remained
at 100% but the specificity was reduced to 79.1%
(95% CI, 70.1%–86.0%). The 150 cases fell into the
following O-RADS categories: 17 (11.3%) lesions in
Category 2, 34 (22.7%) in Category 3, 66 (44.0%) in
Category 4 and 33 (22.0%) in Category 5. There were
no histologically proven malignant lesions in Category 2
or 3. There were 14 malignant lesions in Category 4 and
26 in Category 5. The sensitivity of O-RADS using a
malignancy risk threshold of ≥ 10% was 100% (95% CI,
89.1%–100.0%) and the specificity was 46.4% (95% CI,
36.9%–56.1%).
Conclusions When IOTA terms and techniques are used,
the performance of IOTA models in a North American
patient population is in line with published IOTA results
in other populations. The IOTA SR, SRR assessment and
ADNEX model and O-RADS have similar sensitivity in
the preoperative discrimination of malignant from benign
pelvic tumors; however, the IOTA models have higher
specificity and the algorithm does not require the use of
magnetic resonance imaging.© 2022 International Society
of Ultrasound in Obstetrics and Gynecology.
INTRODUCTION
Optimal treatment of women with a pelvic mass requires
knowledge of the specific nature of the mass. Some pelvic
masses may be managed expectantly, while others are best
treated by surgery. With respect to surgical management,
many benign tumors can be removed laparoscopically
by a gynecologist. Conversely, malignant lesions are best
managed by a specialized team, including a gynecologic
oncologist, in a specialized oncology center; this results in
improved survival1 .
Since its introduction in the mid 1980s, transvaginal
ultrasound has become the primary imaging modality for
evaluation of adnexal pathology2 . However, it has failed
to reach its full potential, due to a lack of standardization
of both the required elements of a transvaginal ultrasound
examination and the terms used to describe the findings.
In response to these concerns, the International Ovarian
Tumor Analysis (IOTA) group developed and validated
ultrasound-based rules and models to improve the
preoperative characterization of ovarian pathology3,4 .
Three of these models are the Simple Rules (SR), Simple
Rules risk (SRR) assessment and Assessment of Different
NEoplasias in the adneXa (ADNEX) models. SR is a
dichotomous model that predicts whether a pelvic mass
is likely to be benign or malignant, with approximately
25% of cases falling into an ‘inconclusive’ category when
the lesion cannot be classified5 . SRR assessment provides
an estimated risk of malignancy for pelvic lesions6 . The
ADNEX model is a polytomous model that provides
a risk assessment for the various pelvic malignancy
subtypes7 .
© 2022 International Society of Ultrasound in Obstetrics and Gynecology.
669
The Ovarian-Adnexal Reporting and Data System
(O-RADS) ultrasound risk stratification and management
tool is modeled on the IOTA rules. Both approaches
share the goals of improving sonographic interpretation,
decreasing the ambiguity of ultrasound reports and
providing management recommendations for ovarian
lesions in each O-RADS risk category8 .
The adoption of IOTA rules and models into clinical
practice in North America has been somewhat limited9 .
One of the likely contributors to the low level of
acceptance is the lack of validation of these models in
the USA. The aim of this study was to evaluate and
compare the performance of the ADNEX model, SR, SRR
assessment and O-RADS in preoperative differentiation
between benign and malignant adnexal lesions in a North
American population.
METHODS
Study design and patients
This was a single-center, diagnostic accuracy study con-
ducted by the Boonshoft School of Medicine, Department
of Obstetrics and Gynecology at Miami Valley Hospital in
Dayton, OH, USA. This is a tertiary referral gynecologic
oncology center. Enrolled consecutively into the study
between March 2018 and February 2021 were 150
patients with an ultrasound diagnosis of an adnexal mass.
The inclusion criteria were as follows: (1) presence of at
least one adnexal lesion detected by transvaginal and/or
transabdominal ultrasonography; (2) lesion removed
surgically and evaluated histopathologically; and (3) time
interval between ultrasound detection and surgery not
exceeding 180 days. The study was approved by the
Institutional Review Board of Wright State University.
Transvaginal ultrasonography was the primary
scanning modality. Transabdominal ultrasound was
performed when the lesion was too large to be evaluated
adequately with transvaginal ultrasonography alone
or when the patient could not tolerate the transvagi-
nal approach. Two expert IOTA-certified sonologists
(A.K.H., J.D.S.), each with at least 25 years of experience
in gynecological ultrasound, assessed prospectively the
sonographic tumor morphology using nomenclature and
methodology proposed by the IOTA Group3 . All variables
used in the IOTA models were collected at the time of
the ultrasound examination by the two sonologists. The
ADNEX risk calculation was then performed prospec-
tively and provided to the attending physician. SR results,
SRR assessment and O-RADS scores were obtained
retrospectively by reviewing the recorded IOTA variables
and each ultrasound examination. The SR results, SRR
assessment and O-RADS scores were not provided to
the attending physician. The ultrasound machine used
was either a Voluson E8 (GE Healthcare, Zipf, Austria),
Siemens S2000 (Siemens Medical Solutions, Malvern, PA,
USA) or EPIC 7 (Philips Health Systems, Bothell, WA,
USA). When multiple adnexal lesions were detected in the
same patient, the lesion with the highest risk of malignancy
based on the ADNEX model was included in the study.
Ultrasound Obstet Gynecol 2022; 59: 668–676.
670
Test methods
The histopathological diagnosis after surgical removal of
the lesion served as the reference standard. All excised
tissues were examined histologically in our surgical
pathology department, following the guidelines of the
World Health Organization10 . Malignant tumors were
staged using International Federation of Gynecology and
Obstetrics (FIGO) criteria11,12 .
The ADNEX model, which is available on the IOTA
website (http://www.iotagroup.org/adnexmodel), was
applied prospectively for each lesion. This model uses
nine predictors, including three clinical and six ultrasound
variables: patient age (years), serum CA-125 level (U/mL),
type of center (oncology referral center vs non-oncology
center), maximum diameter of the lesion (mm), maximum
diameter of the largest solid component of the lesion
(mm), more than 10 cyst locules (yes or no), number
of papillary projections (0, 1, 2, 3 or > 3), presence of
acoustic shadows (yes or no) and presence of ascites (yes
or no). The six ultrasound variables must conform to
the approved IOTA nomenclature13 . ‘Oncology referral
center’ was defined as a tertiary referral center with a
specific gynecologic oncology unit14 . Results are provided
as percentages and in graphical form to present the overall
risk of malignancy for a lesion. We also calculated the risk
for each malignancy subtype: borderline ovarian tumor
(BOT), Stage-I ovarian cancer, Stages-II–IV ovarian
cancer and metastasis to the ovary7 . For the purposes of
this study, the overall risk of malignancy was used.
The ADNEX model can be applied with or without
serum CA-125 level. The omission of CA 125 has
limited impact on the ADNEX model in differentiating
between benign and malignant tumors overall; however,
inclusion of CA-125 level in risk calculations improves the
differentiation between Stage-II–IV ovarian cancer and
the other malignancy subtypes13 . Since a CA-125 level
was not available for every patient in our study, we did
not use this in the risk calculation for the ADNEX model.
SR and SRR assessment were applied retrospectively
to each lesion by reviewing the ultrasound images and
using the recorded IOTA morphologic features. SR were
developed by the IOTA group as an ultrasound triage
test to predict a malignant or benign lesion prior to
surgery. It includes five ultrasound rules that are classified
as benign and five that are considered malignant. If
one or more benign rules is present in the absence
of any malignant rules, the lesion is predicted to be
benign. If one or more malignant rules is present in the
absence of any benign rules, the lesion is predicted to
be malignant. If both malignant and benign rules are
present, or if none of the rules is present, SR classify
the lesion as ‘inconclusive’5 . SR have been shown to be
informative in about 75% of adnexal lesions. In cases
in which the results are informative, the performance
of SR has been shown to be as good as subjective
assessment by an experienced ultrasound examiner using
pattern recognition for differentiating between benign and
malignant lesions15 .
© 2022 International Society of Ultrasound in Obstetrics and Gynecology.
Hiett et al.
The SRR assessment was developed in 2016 by the
IOTA group, partly to address inconclusive results and to
provide a numeric risk of malignancy for pelvic lesions6 .
The SRR assessment utilizes the same five benign rules
and five malignant rules that are used in SR. The presence
or absence of each rule is entered into the ‘Simple
Rules Risk calculator’, which is available online (https://
homes.esat.kuleuven.be/∼sistawww/biomed/ssrisk/). The
calculator provides a numeric risk of malignancy based
on the ultrasound-generated benign and malignant
characteristics of the lesion and type of center (oncology
referral center vs non-oncology center), thus eliminating
the inconclusive classification.
O-RADS was applied retrospectively to each lesion by
reviewing the ultrasound images. O-RADS was developed
by an international committee sponsored by the American
College of Radiology8 . This approach relies on the
sonographic nomenclature developed by the IOTA group.
This nomenclature is used to characterize each pelvic
lesion, which is then placed into one of six categories
(O-RADS 0–5). The O-RADS working group defined
the risk classification for each of these categories8 based
on expert opinion and retrospective analysis of data
from IOTA phases 1–3. The categories incorporate
probabilities ranging from normal/benign to high risk
of malignancy. Category 0 is assigned to incomplete
evaluations. Category 1 represents normal sonographic
ovarian morphology. Categories 2–5 include sonographic
findings of ovarian lesions, with increasing levels of risk
of malignancy: Category 2 is associated with lesions
that are almost certainly benign (risk of malignancy
< 1%), Category 3 includes lesions that have low risk
(1 to < 10%) of malignancy, Category 4 includes those
with intermediate risk (10 to < 50%), and Category 5
includes those with high risk (≥ 50%) of malignancy.
O-RADS includes guidelines for management of lesions
in the various risk categories: in Table 1, the risk of
malignancy based on the ADNEX model is stratified
to correspond with the malignancy risk associated
with each of the O-RADS categories and management
algorithms8 .
Statistical analysis
Statistical analysis was performed using JMP version
15.2.0 (SAS Institute Inc., Cary, NC, USA) software. For
statistical purposes, BOTs were considered malignant.
The area under the receiver-operating-characteristics
curve (AUC) for basic differentiation between benign and
malignant tumors was calculated for the ADNEX model
and SRR assessment. The AUC for the ADNEX model
was calculated using the overall malignancy risk without
CA-125 level. Various malignancy risk thresholds have
been proposed for the ADNEX and SSR models to achieve
the optimal test sensitivity and specificity, based on avail-
ability of specialists, different types of health system and
local protocols or guidelines13 . Sensitivity and specificity
with their corresponding 95% CIs for the ADNEX model
were calculated at cut-off values of 5%, 10%, 15%
Ultrasound Obstet Gynecol 2022; 59: 668–676.
IOTA models and O-RADS in preoperative evaluation of adnexal lesions 671

and 20% overall risk of malignancy (i.e. the sum of the
estimated risks for the four subtypes of malignancy). Sen-
sitivity and specificity with their corresponding 95% CIs
for the SRR assessment were also calculated at cut-off
values of 5%, 10%, 15% and 20% risk of malignancy.
Diagnostic performance for SR was assessed by calculat-
ing the sensitivity and specificity with their corresponding
95% CIs.
For the purposes of comparison, the sensitivity and
specificity for O-RADS was calculated for a malignancy
risk threshold of ≥ 10%. This O-RADS threshold was
selected because risk below this cut-off includes lesions
considered to be at low risk for malignancy (Category
≤ 3) and risk at or above the cut-off includes lesions with
intermediate-to-high risk for malignancy (Categories 4
and 5)8 . The ADNEX model and SRR assessment lend
themselves to setting the same threshold, being easily
adjustable to any threshold.
The Z-score for two population proportions
(McNemar’s test) was evaluated to determine if
the agreement between the models and final histology
differed. The sonographic characteristics, patients’
epidemiological features and malignancy risks were
compared using Pearson’s chi-square test for categorical
data and Student’s t-test or the Mann–Whitney U-test
for continuous data. Means with standard error (SE) are
presented for continuous data. Statistical significance was
assumed at P < 0.05 for all comparisons.
RESULTS
Participants
Between March 2018 and February 2021, a total of
704 patients underwent ultrasound examination and
ADNEX model assessment at our center. Of those, 150
underwent surgery within 180 days after the ultrasound
examination, thus meeting the study inclusion criteria. In
each case, a full description of the ultrasound findings and
pathological examination of the lesion were available for
review.
Of the 150 women, 110 (73.3%) had a benign
ovarian tumor and 40 (26.7%) had a malignant tumor.
Demographic, clinical and ultrasound characteristics
of these groups are presented in Table 2. Patients
with malignant lesions, including BOT, were similar
in age to women with benign lesions (mean ± SE age,
47.5 ± 3.1 vs 48.2 ± 1.7 years, P = 0.78). There were
21 malignant lesions and 63 benign lesions (25.0%
malignancy prevalence) in the premenopausal cohort

Table 1 Summary of the Ovarian-Adnexal Reporting and Data System (O-RADS) ultrasound risk stratification and management system

O-RADS ADNEX
category risk category Premenopausal management Postmenopausal management

0 Incomplete Repeat study Repeat study

evaluation
1 Normal ovary None Not applicable

2 < 1%
3 1% to < 10%
4 10% to < 50%
5 ≥ 50%
Ranging from none to repeat imaging in 8–12 weeks
or evaluation by an ultrasound specialist in
concerning cases
MRI or ultrasound imaging by an expert,
management by gynecologist
MRI or ultrasound imaging by an expert,
management by gynecologist with gynecologic
oncologist, or solely by gynecologic oncologist
Management by gynecologic oncologist
Ranging from none to repeat imaging in 1 year or
with evaluation by an ultrasound specialist or
MRI in concerning cases
MRI or ultrasound imaging by an expert,
management by gynecologist
MRI or ultrasound imaging by an expert,
management by gynecologist with gynecologic
oncologist, or solely by gynecologic oncologist
Management by gynecologic oncologist

ADNEX, Assessment of Different NEoplasias in the adneXa; MRI, magnetic resonance imaging.

Table 2 Demographic, clinical and sonographic characteristics in 150 women with an adnexal lesion

Characteristic Benign (n = 110) Malignant (n = 40) P*

Age (years) 48.2 ± 1.7 47.5 ± 3.1 0.78

Premenopausal 63 (57.3) 21 (52.5)
Postmenopausal 47 (42.7) 19 (47.5)
Presence of solid tissue in lesion 42 (38.2) 38 (95.0) < 0.001
Maximum dimension of lesion (mm) 83.7 ± 5.2 101.4 ± 7.7 0.013
Maximum dimension of solid tissue (mm) 34.4 ± 4.8 63.6 ± 6.5 < 0.001
Three or more papillations 6 (5.5) 10 (25.0) 0.001
More than 10 locules 23 (20.9) 9 (22.5) 0.833
Presence of acoustic shadowing 70 (63.6) 13 (32.5) < 0.001
Ascites 0 (0.0) 12 (30.0) < 0.001
Color score of 1 49 (44.5) 3 (7.5) < 0.001
Color score of 4 1 (0.9) 15 (37.5) < 0.001

Data are given as mean ± standard error or n (%). *Groups compared using Student’s t-test or Mann–Whitney U-test, as appropriate.

© 2022 International Society of Ultrasound in Obstetrics and Gynecology. Ultrasound Obstet Gynecol 2022; 59: 668–676.
672 Hiett et al.

and 19 malignant lesions and 47 benign lesions (28.8%
malignancy prevalence) in the postmenopausal cohort.
On ultrasound, malignant lesions were significantly
larger than benign lesions with respect to the maximum
dimension of the lesion (mean ± SE, 101.4 ± 7.7 mm
vs 83.7 ± 5.2 mm) and malignant lesions were more
likely to contain solid tissue (95.0% vs 38.2%). In
addition, malignant lesions were more likely to have
more than three papillations, and they were more likely
to demonstrate high vascular flow on color Doppler,
with 37.5% vs 0.9% having a color score of 4. Acoustic
shadowing was more common in benign lesions.
Histological findings of the 150 women were as
follows. Among the 110 benign ovarian tumors, those
diagnosed most frequently were benign cystic teratomas
(n = 25) and mucinous cystadenomas (n = 23), followed
by serous cystadenomas (n = 15), cystadenofibromas
(n = 15), endometriomas (n = 9), fibromas (n = 8), Fal-
lopian tube pathology (n = 3), hemorrhagic cysts (n = 2),
benign simple cysts (n = 2), struma ovarii (n = 2) and one
case each of Leydig cell tumor, luteal cyst, functional cyst,
benign retroperitoneal Müllerian cyst, uterine fibroid
and inclusion cyst. The 40 malignant lesions included
12 (8.0%) cases of BOT, 19 (12.7%) of invasive ovarian
cancer, seven (4.7%) of metastatic tumor to the ovary
and one (0.7%) of ovarian carcinoma in situ. There was
one (0.7%) additional rare pelvic malignant tumor, a
case of epithelioid malignant mesothelioma
The mean risk of malignancy generated by SRR assess-
ment was significantly higher in malignant than in benign
lesions (74.1 ± 0.04% vs 15.9 ± 0.02%, P < 0.001). The
performance of SRR assessment is presented in Figure 2.
The AUC of SRR assessment for differentiating between
benign and malignant adnexal masses at the time of
ultrasound examination was 0.941. The performance of
SRR assessment in differentiating between malignant and
benign adnexal tumors at different threshold values for
risk of malignancy (5%, 10%, 15% and 20%) is shown
in Table 3. The sensitivity was 100% for risk cut-offs
of both ≥ 5% and ≥ 10%, but the specificity at these
thresholds increased from 40.9% to 51.8%. At a risk
cut-off of ≥ 15%, the sensitivity decreased to 97.5%,
with an increase in specificity to 60.0%.
SR were informative in 113 (75.3%) cases and the
results were inconclusive in 37 (24.7%) cases. In the 113
cases in which SR could be applied, the sensitivity was
100% (95% CI, 81.5%–100%) and the specificity was
1.0
0.9
0.8
0.7
0.6
Sensitivity

Test results 0.5

The mean overall risk of malignancy generated by the 0.4

ADNEX model was significantly higher in malignant
than in benign lesions (63.3 ± 4.5% vs 11.8 ± 1.6%, 0.3
P < 0.001). The performance of the ADNEX model,
based on all 150 cases, is presented in Figure 1. The AUC 0.2
of the ADNEX model for differentiating between benign 0.1
and malignant adnexal masses at the time of ultrasound
examination was 0.937. The performance of the ADNEX 0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
model in differentiating between malignant and benign
1 – Specificity
adnexal tumors for all 150 cases at different threshold
values for risk of malignancy (5%, 10%, 15% and 20%)
is shown in Table 3. The sensitivity was 97.5% for risk Figure 1 Receiver-operating-characteristics curve for performance
of IOTA ADNEX model without CA-125 level in differentiating
cut-offs of both ≥ 5% and ≥ 10%, but the specificity at
between benign and malignant adnexal masses (n = 150). Area
these thresholds increased from 50.1% to 63.6%. At a under the curve = 0.937. ADNEX, Assessment of Different
risk cut-off of ≥ 15%, the sensitivity decreased to 95.0%, NEoplasias in the adneXa; IOTA, International Ovarian Tumor
with an increase in specificity to 72.7%. Analysis.

Table 3 Diagnostic performance of the IOTA ADNEX model without CA-125 level and Simple Rules risk assessment in differentiating
between benign and malignant lesions at different thresholds for overall risk of malignancy (n = 150)

ADNEX model (without CA 125) Simple Rules risk assessment

Malignancy risk
threshold Sensitivity (%) Specificity (%) Sensitivity (%) Specificity (%)

≥ 5% 97.5 (85.3–99.9) 50.1 (41.3–60.5) 100 (89.1–100) 40.9 (31.8–50.7)

≥ 10% 97.5 (85.3–99.9) 63.6 (53.9–72.4) 100 (89.1–100) 51.8 (42.1–61.4)
≥ 15% 95.0 (81.8–99.1) 72.7 (63.2–80.1) 97.5 (85.3–99.9) 60.0 (50.2–69.1)
≥ 20% 90.0 (75.4–96.7) 78.2 (69.1–85.3) 97.5 (85.3–99.9) 80.9 (72.1–87.5)

Values in parentheses are 95% CI. ADNEX, Assessment of Different NEoplasias in the adneXa; IOTA, International Ovarian Tumor
Analysis.

© 2022 International Society of Ultrasound in Obstetrics and Gynecology. Ultrasound Obstet Gynecol 2022; 59: 668–676.
IOTA models and O-RADS in preoperative evaluation of adnexal lesions 673

95.6% (95% CI, 88.5%–98.6%). If the 37 inconclusive according to the corresponding O-RADS categories. The
cases were instead designated ‘malignant’, the overall most common benign lesions in Categories 2 and 3 were
sensitivity remained at 100% but the specificity was benign cystic teratomas and serous cystadenomas. The
reduced to 79.1% (95% CI, 70.1%–86.0%). most common benign lesions in Categories 4 and 5 were
Histological diagnosis of the 37 inconclusive cases is mucinous cystadenomas and cystadenofibromas. All inva-
shown in Table 4. Based on histology, these 37 cases com- sive ovarian cancers, BOTs and metastatic tumors to the
prised 19 (51.4%) benign lesions and 18 (48.6%) malig- ovary were assigned to Category 4 or 5.
nant lesions. The most common malignant lesions were For comparison of the O-RADS, the ADNEX model
invasive ovarian cancers (n = 8) and BOTs (n = 6). Of the and SRR assessment, the risk threshold was set at 10%.
invasive ovarian cancers, four were FIGO Stage 1, three According to the O-RADS working group, risk below this
were Stage 2 and one was Stage 3. When applied to just cut-off is considered ‘low risk for malignancy’ and risk at
the inconclusive group, neither the ADNEX model, SRR or above this cut-off includes lesions that are considered
assessment nor O-RADS was helpful in distinguishing at either ‘increased risk for malignancy’ or ‘high risk
between malignant and benign lesions. In these histologi- for malignancy’. Using this threshold, for O-RADS, the
cally proven 18 malignant and 19 benign lesions, the over- sensitivity was 100% (95% CI, 89.1%–100%) and the
all risk of malignancy based on the ADNEX model was specificity was 46.4% (95% CI, 36.9%–56.1%), for
41.5% and 26.7%, respectively (P = 0.13) and the risk the ADNEX model, the sensitivity was 97.5% (95% CI,
of malignancy based on the SRR assessment was 52.9% 85.3%–99.9%) and the specificity was 63.6% (95% CI,
and 44.8% (P = 0.30), respectively. Using O-RADS clas- 53.9%–72.4%), and for SRR assessment, the sensitivity
sification, all 37 inconclusive cases were assigned to either
Category 4 or Category 5 with risk of malignancy ≥ 10%. Table 4 Histological findings in 37 women with an inconclusive
The distribution of cases based on the O-RADS adnexal lesion based on the IOTA Simple Rules, according to
categories is presented in Table 5. None of our cases benign (n = 19) or malignant (n = 18) findings on histology
was assigned to O-RADS Category 0 or 1 (defined as
Histology Cases (n)
‘incomplete evaluation’ and ‘normal ovary’, respec-
tively). Seventeen (11.3%) lesions were in Category 2, Benign
34 (22.7%) were in Category 3, 66 (44.0%) were in Mucinous cystadenoma 4
Cystadenofibroma 4
Category 4 and 33 (22.0%) were in Category 5. There
Serous cystadenoma 3
were no histologically proven malignant lesions in Fibroma 2
either Category 2 or Category 3. The malignancy rate in Benign cystic teratoma 2
Categories 4 and 5 were 21.2% and 78.8%, respectively. Uterine fibroid 1
Table 6 presents the distribution of the 110 histo- Endometrioma 1
logically benign and 40 histologically malignant lesions Leydig cell tumor 1
Luteal cyst 1
Malignant
Invasive ovarian cancer 8
1.0
Borderline ovarian tumor 6
Metastatic to the ovary 2
0.9
Ovarian carcinoma in situ 1
Epithelioid malignant mesothelioma 1
0.8
IOTA, International Ovarian Tumor Analysis.
0.7

0.6 Table 5 Distribution of 150 women with an adnexal lesion within

Sensitivity

the six O-RADS categories, with corresponding malignancy risk

0.5 according to the IOTA ADNEX model, and rate of histologically
confirmed malignancy
0.4
Rate of
0.3 ADNEX risk histologically
O-RADS Cases of malignancy confirmed
0.2 category (n (%)) (% (mean ± SE)) malignancy (%)

0.1 0 N/A N/A N/A

1 N/A N/A N/A
0 2 17 (11.3) 1.0 ± 0.3 0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 3 34 (22.7) 3.6 ± 0.6 0
1 – Specificity 4 66 (44.0) 25.5 ± 3.2 21.2
5 33 (22.0) 60.8 ± 5.1 78.8

Figure 2 Receiver-operating-characteristics curve for performance ADNEX, Assessment of Different NEoplasias in the adneXa;
of IOTA Simple Rules risk assessment in differentiating between IOTA, International Ovarian Tumor Analysis; N/A, not applicable;
benign and malignant adnexal masses (n = 150). Area under the O-RADS, Ovarian-Adnexal Reporting and Data System; SE,
curve = 0.941. IOTA, International Ovarian Tumor Analysis. standard error.

© 2022 International Society of Ultrasound in Obstetrics and Gynecology. Ultrasound Obstet Gynecol 2022; 59: 668–676.
674 Hiett et al.

was 100% (95% CI, 89.1%–100%) and the specificity
was 51.8% (95% CI, 42.1%–61.4%).
Agreement between each model’s prediction (benign or
malignant) and the final lesion histology was calculated.
When the SR were informative (excluding inconclusive
cases), the predicted outcome (benign vs malignant)
agreed with the final histology in 96.5% of cases, and
if the inconclusive cases were considered malignant and
combined with those cases, the predicted outcome (benign
vs malignant) agreed with the final histology in 84.7%
of cases. Using a malignancy risk threshold of ≥ 10% for
the ADNEX model and SRR assessment, the predicted
outcome (benign vs malignant) agreed with the final his-
tology in 72.7% and 64.7% of cases, respectively. Using
a malignancy risk threshold of ≥ 10% for the O-RADS
model, the predicted outcome (benign vs malignant)
agreed with the final histology in 60.7% of cases.
Z-scores for two population proportions were used
to determine if, at the threshold of 10%, the agree-
ment between the models and final histology differed
significantly. SR, when informative, demonstrated a
significantly higher rate of agreement compared to the
ADNEX model, SRR assessment and O-RADS: Z = 4.58,
P < 0.001; Z = 5.57, P < 0.001; and Z = 6.08, P < 0.001,
respectively. When inconclusive cases were considered
malignant and combined with the malignant cases, SR
again demonstrated a significantly higher rate of agree-
ment with histology than did the ADNEX model, SRR
assessment and O-RADS: Z = 3.53, P = 0.011; Z = 5.30,
P < 0.001; and Z = 5.84, P < 0.001, respectively. The
ADNEX model demonstrated a higher rate of agreement
with the final pathology when compared with the SRR
and O-RADS models: Z = 2.27, P = 0.023; and Z = 3.53,
P < 0.001, respectively. Finally, there was no difference in
agreement between SRR and O-RADS models; Z = 1.417,
P = 0.157.
O-RADS recommendations for additional testing for
Categories 3 and 4 includes the option of magnetic
resonance imaging (MRI). To evaluate if there may be
another option to this expensive modality, we applied
the ADNEX model, SRR assessment and SR to the 100
(66.7%) cases that fell into these two categories. Out
of this group of patients, 86 (86.0%) had a benign
lesion, and 14 (14.0%) had a malignant lesion (Table 6).
The most common benign lesions were mucinous
cystadenomas (n = 23 (26.7%)) and serous cystadenomas
(n = 15 (17.4%)). The 14 malignant lesions included six
BOTs, six invasive ovarian cancers, and two metastatic
tumors to the ovary. In these 100 patients, the mean
ADNEX risk was significantly higher for histologically
confirmed malignant lesions compared with benign
lesions (mean ± SE, 52.2 ± 0.09% vs 12.5 ± 0.02%,
P < 0.001). Using the ADNEX model with a risk
threshold of ≥ 10%, 46 lesions were predicted to be
malignant and 54 were predicted to be benign. Thirteen
(28.3%) of the 46 lesions predicted to be malignant were
proven to be malignant histologically and 53 of the 54
(98.1%) lesions predicted to be benign were proven to be
benign. The sensitivity and specificity of ADNEX at a risk
threshold of ≥ 10% for lesions in O-RADS Categories 3
and 4 were 92.9% (95% CI, 64.2%–99.6%) and 61.6%
(95% CI, 50.5%–71.7%), respectively.
The mean SRR assessment risk of malignancy was
also significantly higher for histologically confirmed
malignant compared to benign lesions (63.9 ± 0.06% vs
16.3 ± 0.02%, P < 0.001). Using SRR assessment with

Table 6 O-RADS categories of histologically benign (n = 110) and histologically malignant (n = 40) adnexal lesions

Histology O-RADS 2 (n = 17) O-RADS 3 (n = 34) O-RADS 4 (n = 66) O-RADS 5 (n = 33)

Benign 17 34 52 7
Benign cystic teratoma 12 7 6 0
Benign retroperitoneal Müllerian cyst 0 1 0 0
Benign simple cyst 1 1 0 0
Cystadenofibroma 0 3 10 2
Endometrioma 3 3 3 0
Fallopian tube pathology 1 0 1 1
Fibroma 0 3 4 1
Functional cyst 0 0 0 1
Hemorrhagic cyst 0 1 1 0
Inclusion cyst 0 0 1 0
Leydig cell tumor 0 0 1 0
Luteal cyst 0 0 1 0
Mucinous cystadenoma 0 6 17 0
Serous cystadenoma 0 9 6 0
Struma ovarii 0 0 1 1
Uterine fibroid 0 0 0 1
Malignant 14 26
Invasive ovarian cancer — — 6 13
Borderline ovarian tumor — — 6 6
Metastatic — — 2 5
Ovarian carcinoma in situ — — 0 1
Epithelioid malignant mesothelioma — — 0 1

Data are given as n. O-RADS, Ovarian-Adnexal Reporting and Data System.

© 2022 International Society of Ultrasound in Obstetrics and Gynecology. Ultrasound Obstet Gynecol 2022; 59: 668–676.
IOTA models and O-RADS in preoperative evaluation of adnexal lesions
a risk threshold of ≥ 10%, 61 lesions were predicted
to be malignant and 39 were predicted to be benign.
Fourteen (23.0%) of the 61 lesions predicted to be
malignant were proven to be malignant histologically
and 39 of the 39 (100%) lesions predicted to be benign
were proven to be benign. The sensitivity and specificity
of SRR assessment at a threshold of ≥ 10% for lesions
in O-RADS Categories 3 and 4 were 100% (95% CI,
73.3%–100%) and 45.3% (95% CI, 34.7%–56.4%),
respectively.
The SR were informative, i.e. not ‘inconclusive’, in 75
(75.0%) of these lesions. Sixty-eight were predicted to
be benign by the SR and all of these were histologically
benign. Of the seven lesions that were predicted to be
malignant, five were confirmed to be malignant histolog-
ically. Thus, when the SR results were informative, the
sensitivity and specificity in O-RADS Categories 3 and
4 were 100% (95% CI, 46.3%–100.0%) and 97.1%
(95% CI, 89.1%–99.5%), respectively. Out of the 25
lesions that were inconclusive based on the SR, 16 were
found to be benign and nine were found to be malignant
histologically. If the lesions considered inconclusive by
the SR were instead treated as being malignant, the
specificity fell to 79.1% (95% CI, 68.7%–86.8%).
DISCUSSION
To our knowledge, this is the first study that validates
the diagnostic accuracy of the IOTA SR, SRR assessment
and ADNEX model and O-RADS in a North American
population assessed at a single center. All four models per-
formed well in identifying lesions likely to be malignant,
with sensitivities exceeding 95%; however, the specifici-
ties varied. SR had a specificity of 95.6% when these rules
could be applied; however, the specificity decreased to
79.1% when the ‘inconclusive’ lesions were reclassified as
‘malignant’. The specificity of SRR assessment at a malig-
nancy risk threshold of ≥ 10% was 51.8%. The specificity
for the ADNEX model at the same risk threshold of
≥ 10% (63.6%) was higher than that for SRR assessment,
and also than that for the O-RADS system (46.4%). The
SR, with or without considering the inconclusive lesions
as malignant, demonstrated the highest rate of agreement
with the final pathology, when compared with the other
three models using the 10% malignancy risk threshold.
Using the same 10% risk threshold, the ADNEX model
demonstrated higher agreement with final pathology
when compared with O-RADS.
Accurate preoperative assessment of pelvic lesions is
essential in planning clinical management strategies16 .
There are currently no approved screening programs for
ovarian cancer and, unfortunately, the majority of ovarian
cancers are detected only at advanced stages17 . This high-
lights the need for clinical tools that predict accurately
which pelvic lesions are likely malignant. This knowledge
would help in the appropriate referral of patients to a
gynecologic oncology center. It is clear that ovarian cancer
management by gynecologic oncologists in high-volume
oncology centers improves overall survival18–20 .
© 2022 International Society of Ultrasound in Obstetrics and Gynecology.
675
Subjective assessment of pelvic ultrasound images using
pattern recognition, by a clinician with considerable
experience in gynecologic ultrasound, has been shown to
differentiate between benign and malignant pelvic lesions
with a high degree of accuracy21,22 . In fact, subjective
assessment appears to be the best method to predict
the likelihood of a pelvic malignancy23 . Unfortunately,
clinicians with this level of expertise are not universally
available. With this in mind, IOTA have developed and
validated ultrasound-based rules and models to assist
clinicians with different levels of training and expertise in
characterizing adnexal pathology14,24–26 .
The test performance of the IOTA models in our popu-
lation is in line with the performance reported in previous
studies. In a prospective validation study of 1938 women
with an adnexal lesion15 , SR yielded a conclusive result in
1501 (77%) cases. The resulting sensitivity and specificity
were 92% (95% CI, 89%–94%) and 96% (95% CI,
94%–97%), respectively. In another study7 , the ADNEX
model was developed and validated in 5909 cases and
had an AUC of 0.94 when CA 125 was not included in
the risk calculation. The omission of CA 125 appears to
have little impact on the performance of the ADNEX
model in differentiating between benign and malignant
lesions. However, it appears to be useful in differentiating
between Stage-I and Stage-II–IV ovarian cancers13,14 .
SSR assessment was developed to estimate the risk of
malignancy based on the presence or absence of each of
the 10 SR features and the type of center6 ; the AUC for
the original external validation dataset (n = 2403) from
oncology centers was 0.917. Our results are also in agree-
ment with those of smaller studies validating the ADNEX
model. Araujo et al.27 reported on the performance of the
ADNEX model in 131 women and Chen et al.28 in 278
women, with AUCs of 0.92 and 0.94, respectively.
The more recent development of O-RADS is another
attempt to improve the overall ultrasound evaluation,
reporting and management of women presenting with
a pelvic lesion, but this new system has not been
prospectively validated previously. The major difference
between the approaches designed by IOTA and the
O-RADS system is that, for certain categories, O-RADS
includes recommendations for additional evaluations. It
suggests the use of MRI in lesions that are assigned
to O-RADS Categories 3 and 4, in order to better
characterize their nature. However, our study suggests
that, if either the ADNEX model or SSR assessment, each
set at a malignancy threshold of 10%, is applied to these
two O-RADS categories, the sensitivity exceeds 90%, and
using just the SR yields the same result. The IOTA model
with the highest specificity in these lesions was SR with
inconclusive lesions treated as malignant, with a specificity
of 79.1%. The robust performance of these models for
lesions assigned to O-RADS Categories 3 and 4 calls into
question the utility of MRI to improve characterization
of these lesions, especially considering the expense of this
procedure.
The main strength of our study is the prospective
collection of IOTA-specific ultrasound features. The
Ultrasound Obstet Gynecol 2022; 59: 668–676.
676
ADNEX model was applied at the time of the ultrasound
examination, prior to histologic diagnosis. Even though
risk assessments using the SR, SRR and O-RADS were
done retrospectively, the ultrasound parameters used in
these models conformed to IOTA terminology and were
obtained at the time of each ultrasound examination. All
cases had surgery and histologic diagnosis performed
within 180 days of the ultrasound examination. The
second strength is that the sonologists interpreting the
ultrasound studies were experienced and IOTA-certified.
Limitations of our study are the relatively small sample
size and the fact that SR, SRR and O-RADS evaluations
were applied retrospectively. In addition, our data were
collected in a single gynecological ultrasound center,
which may limit broad application of the findings.
Furthermore, not all patients who underwent ADNEX
model evaluation during the study period were included,
due to the lack of surgery within 180 days. This almost
certainly increased the prevalence of malignant lesions in
our population.
In conclusion, when IOTA terms and techniques are
used, the IOTA models and O-RADS perform well
in a North American population. The four models
demonstrate similar sensitivities in the preoperative
differentiation of malignant from benign pelvic lesions;
however, the IOTA models have higher specificity. In
addition, when either SR, SRR assessment or the ADNEX
model is used, the additional use of MRI in the evaluation
of malignancy risk may be unnecessary. Larger multicen-
ter trials are needed to further evaluate the performance
of these three testing modalities in North America.
REFERENCES
1. Fung-Kee-Fung M, Kennedy EB, Biagi J, Colgan T, D’Souza D, Elit LM, Hunter A,
Irish J, McLeod R, Rosen B. The optimal organization of gynecologic oncology
services: a systematic review. Curr Oncol 2015; 22: e282.
2. Higgins RV, van Nagell Jr JR, Woods CH, Thompson EA, Kryscio RJ. Interobserver
variation in ovarian measurements using transvaginal sonography. Gynecol Oncol
1990; 39: 69–71.
3. Timmerman D, Valentin L, Bourne TH, Collins WP, Verrelst H, Vergote I, IOTA
Group. Terms, definitions, and measurements to describe the sonographic features of
adnexal tumors: a consensus opinion from the International Ovarian Tumor Analysis
(IOTA) Group. Ultrasound Obstet Gynecol 2000; 16: 500–505.
4. Kaijser J, Bourne T, Valentin L, Sayasneh A, Van Holsbeke C, Vergote I, Testa A,
Franchi D, Van Calster B, Timmerman D. Improving strategies for diagnosing ovarian
cancer: a summary of the International Ovarian Tumor Analysis (IOTA) studies.
Ultrasound Obstet Gynecol 2013; 41: 9–20.
5. Timmerman D, Testa AC, Bourne T, Ameye L, Jurkovic D, Van Holsbeke C,
Paladini D, Van Calster B, Vergote I, Van Huffel S, Valentin L. Simple
ultrasound-based rules for the diagnosis of ovarian cancer. Ultrasound Obstet
Gynecol 2008; 31: 681–690.
6. Timmerman D, Van Calster B, Testa A, Savelli L, Fischerova D, Froyman W,
Wynants L, Van Holsbeke C, Epstein E, Franchi D, Kaijser J, Czekierdowski A,
Guerriero S, Fruscio R, Leone FPG, Rossi A, Landolfo C, Vergote I, Bourne T,
Valentin L. Predicting the risk of malignancy in adnexal masses based on the Simple
Rules from the International Ovarian Tumor Analysis group. Am J Obstet Gynecol
2016; 214: 424–437.
7. Van Calster B, Van Hoorde K, Valentin L, Testa AC, Fischerova D, Van Holsbeke C,
Savelli L, Franchi D, Epstein E, Kaijser J, Van Belle V, Czekierdowski A, Guerriero S,
Fruscio R, Lanzani C, Scala F, Bourne T, Timmerman D. Evaluating the risk of
ovarian cancer before surgery using the ADNEX model to differentiate between
© 2022 International Society of Ultrasound in Obstetrics and Gynecology.
Hiett et al.
benign, borderline, early and advanced stage invasive and secondary metastatic
tumours: prospective multicentre diagnostic study. BMJ 2014; 349: g5920.
8. Andreotti RF, Timmerman D, Strachowski LM, Froyman W, Benacerraf BR, Bennett
GL, Bourne T, Brown DL, Coleman BG, Frates MC, Goldstein SR, Hamper UM,
Horrow MM, Hernanz-Schulman M, Reinhold C, Rose SL, Whitcomb BP, Wolfman
WL, Glanc P. O-RADS US Risk Stratification and Management System: A Consensus
Guideline from the ACR Ovarian-Adnexal Reporting and Data System Committee.
Radiology 2020; 294: 168–185.
9. Abramowicz JS, Timmerman D. Ovarian mass-differentiating benign from malignant:
the value of the International Ovarian Tumor Analysis ultrasound rules. Am J Obstet
Gynecol 2017; 217: 652–660.
10. International Agency For Research On Cancer. WHO Classification of Tumours of
Female Reproductive Organs, Kurman RJ, Carcangiu ML, Herrington CS, Young
RH (eds). World Health Organization: Geneva, 2014; 307.
11. Berek JS, Kehoe ST, Kumar L, Friedlander M. Cancer of the ovary, fallopian tube,
and peritoneum. Int J Gynaecol Obstet 2018; 143 (Suppl 2): 59–78.
12. Bhatla N, Denny L. FIGO Cancer Report 2018. Int J Gynecol Obstet 2018; 143:
2–3.
13. Van Calster B, Van Hoorde K, Froyman W, Kaijser J, Wynants L, Landolfo C,
Anthoulakis C, Vergote I, Bourne T, Timmerman D. Practical guidance for applying
the ADNEX model from the IOTA group to discriminate between different subtypes
of adnexal tumors. Facts Views Vis Obgyn 2015; 7: 32–41.
14. Sayasneh A, Ferrara L, De Cock B, Saso S, Al-Memar M, Johnson S, Kaijser J,
Carvalho J, Husicka R, Smith A, Stalder C, Blanco MC, Ettore G, Van Calster B,
Timmerman D, Bourne T. Evaluating the risk of ovarian cancer before surgery using
the ADNEX model: a multicentre external validation study. Br J Cancer 2016; 115:
542–548.
15. Timmerman D, Ameye L, Fischerova D, Epstein E, Melis GB, Guerriero S, Van
Holsbeke C, Savelli L, Fruscio R, Lissoni AA, Testa AC, Veldman J, Vergote I,
Van Huffel S, Bourne T, Valentin L. Simple ultrasound rules to distinguish between
benign and malignant adnexal masses before surgery: prospective validation by IOTA
group. BMJ 2010; 341: c6839.
16. Timmerman D, Planchamp F, Bourne T, Landolfo C, du Bois A, Chiva L,
Cibula D, Concin N, Fischerova D, Froyman W, Gallardo G, Lemley B, Loft A,
Mereu L, Morice P, Querleu D, Testa AC, Vergote I, Vandecaveye V, Scambia G,
Fotopoulou C. ESGO/ISUOG/IOTA/ESGE Consensus Statement on preoperative
diagnosis of ovarian tumors. Ultrasound Obstet Gynecol 2021; 58: 148–168.
17. Robertson SE, Peipert JF. Ultrasound Screening for Ovarian Cancer: Are We There
Yet? Obstet Gynecol 2018; 132: 1089–1090.
18. Bristow RE, Zahurak ML, Diaz-Montes TP, Giuntoli RL, Armstrong DK. Impact of
surgeon and hospital ovarian cancer surgical case volume on in-hospital mortality
and related short-term outcomes. Gynecol Oncol 2009; 115: 334–338.
19. Woo YL, Kyrgiou M, Bryant A, Everett T, Dickinson HO. Centralisation of services
for gynaecological cancer. Cochrane Database Syst Rev 2012 (3): CD007945.
20. Cliby WA, Powell MA, Al-Hammadi N, Chen L, Philip Miller J, Roland PY, Mutch
DG, Bristow RE. Ovarian cancer in the United States: contemporary patterns of care
associated with improved survival. Gynecol Oncol 2015; 136: 11–17.
21. Timmerman D, Schwärzler P, Collins WP, Claerhout F, Coenen M, Amant F,
Vergote I, Bourne TH. Subjective assessment of adnexal masses with the use of
ultrasonography: an analysis of interobserver variability and experience. Ultrasound
Obstet Gynecol 1999; 13: 11–16.
22. Valentin L. Prospective cross-validation of Doppler ultrasound examination and
gray-scale ultrasound imaging for discrimination of benign and malignant pelvic
masses. Ultrasound Obstet Gynecol 1999; 14: 273–283.
23. Meys EM, Kaijser J, Kruitwagen RF, Slangen BF, Van Calster B, Aertgeerts B,
Verbakel JY, Timmerman D, Van Gorp T. Subjective assessment versus ultrasound
models to diagnose ovarian cancer: A systematic review and meta-analysis. Eur
J Cancer 2016; 58: 17–29.
24. Sayasneh A, Wynants L, Preisler J, Kaijser J, Johnson S, Stalder C, Husicka R,
Abdallah Y, Raslan F, Drought A, Smith AA, Ghaem-Maghami S, Epstein E, Van
Calster B, Timmerman D, Bourne T. Multicentre external validation of IOTA
prediction models and RMI by operators with varied training. Br J Cancer 2013;
108: 2448–2454.
25. Sayasneh A, Kaijser J, Preisler J, Johnson S, Stalder C, Husicka R, Guha S, Naji O,
Abdallah Y, Raslan F, Drought A, Smith AA, Fotopoulou C, Ghaem-Maghami S, Van
Calster B, Timmerman D, Bourne T. A multicenter prospective external validation
of the diagnostic performance of IOTA simple descriptors and rules to characterize
ovarian masses. Gynecol Oncol 2013; 130: 140–146.
26. Ruiz de Gauna B, Sanchez P, Pineda L, Utrilla-Layna J, Juez L, Alcázar JL.
Interobserver agreement in describing adnexal masses using the International Ovarian
Tumor Analysis simple rules in a real-time setting and using three-dimensional
ultrasound volumes and digital clips. Ultrasound Obstet Gynecol 2014; 44: 95–99.
27. Araujo KG, Jales RM, Pereira PN, Yoshida A, de Angelo Andrade L, Sarian LO,
Derchain S. Performance of the IOTA ADNEX model in preoperative discrimination
of adnexal masses in a gynecological oncology center. Ultrasound Obstet Gynecol
2017; 49: 778–783.
28. Chen H, Qian L, Jiang M, Du Q, Yuan F, Feng W. Performance of IOTA ADNEX
model in evaluating adnexal masses in a gynecological oncology center in China.
Ultrasound Obstet Gynecol 2019; 54: 815–822.
Ultrasound Obstet Gynecol 2022; 59: 668–676.