ARTICLE IN PRESS

Current Problems in Diagnostic Radiology 000 (2020) 1 6

Current Problems in Diagnostic Radiology

journal homepage: www.cpdrjournal.com

Original Articles

Machine Learning and Improved Quality Metrics in Acute Intracranial

Hemorrhage by Noncontrast Computed Tomography
Melissa A. Davis, MD, MBA, Balaji Rao, MBBS, FRCR, Paul A. Cedeno, MD, Atin Saha, MD,
Vahe M. Zohrabian, MD*
a
Department of Radiology and Biomedical Imaging ,Yale School of Medicine, Yale University, New Haven, CT 06520.

Objective: The timely reporting of critical results in radiology is paramount to improved patient outcomes. Artificial intelligence has the ability to improve qual-
ity by optimizing clinical radiology workflows. We sought to determine the impact of a United States Food and Drug Administration-approved machine learning
(ML) algorithm, meant to mark computed tomography (CT) head examinations pending interpretation as higher probability for intracranial hemorrhage (ICH),
on metrics across our healthcare system. We hypothesized that ML is associated with a reduction in report turnaround time (RTAT) and length of stay (LOS) in
emergency department (ED) and inpatient populations.
Materials and Methods: An ML algorithm was incorporated across CT scanners at imaging sites in January 2018. RTAT and LOS were derived for reports and
patients between July 2017 and December 2017 prior to implementation of ML and compared to those between January 2018 and June 2018 after implementa-
tion of ML. A total of 25,658 and 24,996 ED and inpatient cases were evaluated across the entire healthcare system before and after ML, respectively.
Results: RTAT decreased from 75 to 69 minutes (P <0.001) at all facilities in the healthcare system. At the level 1 trauma center specifically, RTAT decreased from
67 to 59 minutes (P <0.001). ED LOS decreased from 471 to 425 minutes (P <0.001) for patients without ICH, and from 527 to 491 minutes for those with ICH
(P = 0.456). Inpatient LOS decreased from 18.4 to 15.8 days for those without ICH (P = 0.001) and 18.1 to 15.8 days for those with ICH (P = 0.02).
Conclusion: We demonstrated that utilization of ML was associated with a statistically significant decrease in RTAT. There was also a significant decrease in LOS
for ED patients without ICH, but not for ED patients with ICH. Further evaluation of the impact of such tools on patient care and outcomes is needed.
© 2020 Elsevier Inc. All rights reserved.

Introduction
Intracranial hemorrhage (ICH) is a devastating disease, with an
overall incidence of spontaneous ICH worldwide of 24.6 per 100,000
person-years and approximately 40,000-67,000 cases per year in the
United States.1-3 The 30-day mortality rate is high, ranging from 35%
to 52%, with approximately half of this mortality occurring within the
first 24 hours.4 Numerous prior studies have demonstrated a clinical
benefit to the early detection of ICH.5-8 The Joint Commission man-
dates that the timely reporting of critical test results and diagnostic
procedures is of paramount importance in improving treatment out-
comes and preventing comorbidities.9 Given that computed tomog-
raphy (CT) imaging is such a frequent component of the diagnostic
process in suspected ICH, radiologists are essential in the implemen-
tation of this major national patient safety goal.
A previous study in neuroradiology found that the average time
from study acquisition to critical finding discovery represented the larg-
est component of delay in the radiological workup.10 Prioritization of
radiology worklists, or “priority queuing,” that is, the interpretation of
Abbreviations: ICH, intracranial hemorrhage; AI, artificial intelligence; ML, machine
learning; RTAT, report turnaround time; ED, emergency department; LOS, length of
stay
The authors declare no conflict of interest.
Funding: None.
*Reprint requests: Vahe M. Zohrabian, MD, North Shore University Hospital/North-
well Health, 300 Community Drive, Manhasset, NY 11030.
E-mail address: vahe.zohrabian@gmail.com (V.M. Zohrabian).
medical images with higher priority first, as opposed to the standard
“first in first out” model, has been shown to significantly improve report
turnaround time of both significant and non-significant findings.11-12 As
artificial intelligence (AI) algorithms continue to gain traction across the
radiology landscape, attention is focused on the direct impact of these
processes on routine radiology operations. Large-scale studies evaluat-
ing the influence of AI and deep learning tools on radiology workflow
optimization are limited.13-15 For example, a previously published study
of re-prioritization of head CTs from “routine” to “stat” based on AI
detection of ICH demonstrated a significant reduction in median time to
diagnosis from 512 to 19 minutes16 Aidoc, an FDA approved machine
learning (ML) algorithm based on a convolutional neural network that
detects hyperdense abnormalities on head CTs, has been shown to be
highly accurate in the detection of ICH.17 Moreover, Aidoc’s prioritiza-
tion algorithm has been shown via a mathematical model using a
Monte-Carlo simulation to decrease report turnaround times (RTAT) of
positive findings.18
Here, we use the Aidoc ML algorithm in real-time at a large, ter-
tiary academic medical center and its smaller satellite hospitals to
determine its impact on RTAT and hospital (emergency department
[ED] and inpatient) length of stay (LOS), 2 key metrics of quality in
healthcare.
Materials and Methods
Institutional review board approval was obtained for this study.
Aidoc, an FDA approved ML algorithm based on a convolutional neural

https://doi.org/10.1067/j.cpradiol.2020.10.007
0363-0188/© 2020 Elsevier Inc. All rights reserved.
Etiam venenatis non libero at lacinia. Sed vitae lacinia justo. Vestibulum rutrum porta lacus, nec vestibulum enim fringilla
id. Curabitur nunc risus, egestas vel nunc in, vestibulum scelerisque enim. Cras aliquet euismod magna. Maecenas viverra
neque quam, vitae fringilla augue dignissim sed. Phasellus nisl sapien, iaculis sed lobortis vitae, tristique tincidunt neque.
Aliquam ac blandit risus.

Sed dignissim placerat enim et aliquet. Aenean consectetur interdum massa at cursus. Maecenas aliquam, tortor a eleifend
aliquam, tellus orci fringilla arcu, id dictum lacus risus id nunc. Curabitur quis dolor a ex cursus iaculis. Nulla bibendum
hendrerit orci, nec aliquam ipsum mollis nec. Vestibulum aliquet, urna nec ullamcorper gravida, tortor est porta leo, ut
aliquet mi enim non turpis. Vestibulum ante ipsum primis in faucibus orci luctus

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aliquam tempor aliquet risus, nec auctor ante ultrices vel. Nam
rutrum justo quis magna ornare, non lobortis purus faucibus. In hac habitasse platea dictumst. Vivamus imperdiet est vel
massa fringilla fringilla. Curabitur eros nulla, ornare quis sollicitudin ac, accumsan in tellus. Mauris tempus efficitur mi et
scelerisque. Duis diam arcu, facilisis vel eleifend vitae, congue eu nulla

Donec pretium egestas tellus ac rutrum. Vivamus cursus malesuada ipsum. Proin ornare eleifend eros at ornare. Duis
semper convallis nunc ut dignissim. Praesent nec nulla sit amet libero dignissim elementum at id velit. Vestibulum tempor
mollis porttitor. Cras viverra quam a ex efficitur, ac laoreet libero pulvinar. Etiam condimentum suscipit odio sed laoreet.
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Phasellus ac tellus tempus,

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aliquam tempor aliquet risus, nec auctor ante ultrices vel. Nam
rutrum justo quis magna ornare, non lobortis purus faucibus. In hac habitasse platea dictumst. Vivamus imperdiet est vel
massa fringilla fringilla. Curabitur eros nulla, ornare quis sollicitudin ac, accumsan in tellus. Mauris tempus efficitur mi et
scelerisque. Duis diam arcu, facilisis vel eleifend vitae, congue eu nulla

Current Problems in Diagnostic Radiology, Corrected proof. doi:10.1067/j.cpradiol.2020.10.007

Ut id mauris at velit placerat dignissim vel eu tellus. Donec erat ligula, blandit sed mauris in, laoreet lacinia felis. Ut et
risus purus. Praesent eget gravida nunc. Nunc id nisl commodo mauris maximus ultricies. Etiam congue aliquam elit, a
sollicitudin velit tincidunt sed. Integer consequat erat at turpis fringilla elementum. Phasellus scelerisque dui nec molestie
scelerisque. Phasellus ut mattis metus. Duis lectus ipsum,

Sed nunc nisi, egestas sit amet commodo ac, scelerisque at justo. Curabitur ultricies ligula nec massa tempus, vitae
placerat ante hendrerit. Duis orci augue, rutrum eget libero nec, porta feugiat libero. In lacinia venenatis finibus. Nunc
eleifend nisi faucibus dui maximus, id sagittis lacus laoreet. Aliquam imperdiet arcu id turpis molestie tempor. Phasellus
orci urna, vestibulum non mattis vitae, cursus at velit. Mauris consequat tellus ac mi volutpat laoreet.

At vero eos et accusamus et iusto odio dignissimos ducimus qui blanditiis praesentium voluptatum deleniti atque corrupti
quos dolores et quas molestias excepturi sint occaecati cupiditate non provident, similique sunt in culpa qui officia
deserunt mollitia animi, id est laborum et dolorum fuga. Et harum quidem rerum facilis est et expedita distinctio. Nam
libero tempore, cum soluta nobis est eligendi optio cumque nihil impedit quo minus id quod

Maecenas sodales semper euismod. Nam non suscipit enim. Donec id lorem at est suscipit posuere. Etiam tincidunt nisi
eget leo faucibus porta. Sed congue magna in dignissim tristique. Morbi sodales, enim quis scelerisque congue, risus nisi
venenatis dolor, ut finibus magna augue vel elit. Integer nec mauris nec ligula elementum dictum. Nullam semper luctus
justo, eget tincidunt metus eleifend id. Nunc semper non leo et eleifend

Donec pretium egestas tellus ac rutrum. Vivamus cursus malesuada ipsum. Proin ornare eleifend eros at ornare. Duis
semper convallis nunc ut dignissim. Praesent nec nulla sit amet libero dignissim elementum at id velit. Vestibulum tempor
mollis porttitor. Cras viverra quam a ex efficitur, ac laoreet libero pulvinar. Etiam condimentum suscipit odio sed laoreet.
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Phasellus ac tellus tempus,

Current Problems in Diagnostic Radiology, Corrected proof. doi:10.1067/j.cpradiol.2020.10.007

Nullam quis elementum velit. Proin mollis ornare mi id tempus. Nam faucibus mauris sapien, vitae scelerisque erat
lobortis sed. Vestibulum vel lectus a diam pharetra egestas. Etiam pulvinar ut metus vitae pretium. Etiam eu nisi sed urna
sollicitudin porta at eu enim. Nullam eu facilisis ligula. Cras magna ligula, porttitor in tellus at, gravida commodo est.
Nullam non odio sed mauris vestibulum rutrum in in nulla.

Cras metus lectus, pellentesque ullamcorper bibendum a, interdum non ligula. In ultrices sodales iaculis. Ut ipsum felis,
eleifend non nunc ut, pharetra molestie ex. In volutpat ante ex. Suspendisse bibendum tortor a lectus consectetur
scelerisque. Ut ut lacus dui. Ut non molestie velit. Sed efficitur enim in venenatis tristique. Mauris pulvinar libero quis
faucibus consectetur.

Sed dignissim placerat enim et aliquet. Aenean consectetur interdum massa at cursus. Maecenas aliquam, tortor a eleifend
aliquam, tellus orci fringilla arcu, id dictum lacus risus id nunc. Curabitur quis dolor a ex cursus iaculis. Nulla bibendum
hendrerit orci, nec aliquam ipsum mollis nec. Vestibulum aliquet, urna nec ullamcorper gravida, tortor est porta leo, ut
aliquet mi enim non turpis. Vestibulum ante ipsum primis in faucibus orci luctus

At vero eos et accusamus et iusto odio dignissimos ducimus qui blanditiis praesentium voluptatum deleniti atque corrupti
quos dolores et quas molestias excepturi sint occaecati cupiditate non provident, similique sunt in culpa qui officia
deserunt mollitia animi, id est laborum et dolorum fuga. Et harum quidem rerum facilis est et expedita distinctio. Nam
libero tempore, cum soluta nobis est eligendi optio cumque nihil impedit quo minus id quod

Quisque et risus posuere, hendrerit libero in, vulputate massa. Vestibulum accumsan felis a odio rhoncus consequat.
Morbi consectetur eu ante sit amet molestie. Integer eu massa eget orci placerat volutpat vitae non nulla. Maecenas
dapibus lacus vitae vehicula tristique. Morbi a venenatis nisi. Ut imperdiet felis non neque feugiat consectetur nec id nisi.
Cras ut elementum nulla. Maecenas et posuere diam.

Current Problems in Diagnostic Radiology, Corrected proof. doi:10.1067/j.cpradiol.2020.10.007

Curabitur varius est id magna facilisis, in suscipit tellus auctor. Donec suscipit sed ligula id condimentum. Cras ultrices
nisl non nisl interdum, at tincidunt ante tempor. Sed vitae elit quam. Mauris eget vehicula ligula, id sollicitudin ante.
Nullam semper venenatis sollicitudin. Etiam sem enim, rhoncus at lobortis vel, mattis et mauris. Ut posuere nisi ac lacus
pulvinar mollis.

Sed dignissim placerat enim et aliquet. Aenean consectetur interdum massa at cursus. Maecenas aliquam, tortor a eleifend
aliquam, tellus orci fringilla arcu, id dictum lacus risus id nunc. Curabitur quis dolor a ex cursus iaculis. Nulla bibendum
hendrerit orci, nec aliquam ipsum mollis nec. Vestibulum aliquet, urna nec ullamcorper gravida, tortor est porta leo, ut
aliquet mi enim non turpis. Vestibulum ante ipsum primis in faucibus orci luctus

At vero eos et accusamus et iusto odio dignissimos ducimus qui blanditiis praesentium voluptatum deleniti atque corrupti
quos dolores et quas molestias excepturi sint occaecati cupiditate non provident, similique sunt in culpa qui officia
deserunt mollitia animi, id est laborum et dolorum fuga. Et harum quidem rerum facilis est et expedita distinctio. Nam
libero tempore, cum soluta nobis est eligendi optio cumque nihil impedit quo minus id quod

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Ut vulputate augue eget odio placerat, in consectetur orci
tincidunt. In hac habitasse platea dictumst. Aliquam efficitur metus ligula, sit amet vehicula ligula rutrum ut. Duis mollis
nibh fringilla lacus maximus, vitae lobortis metus volutpat. Suspendisse ut sapien erat. Pellentesque commodo nulla vitae
facilisis dignissim. Vivamus at orci mi. Donec tristique rhoncus elit ac dapibus.

Ut id mauris at velit placerat dignissim vel eu tellus. Donec erat ligula, blandit sed mauris in, laoreet lacinia felis. Ut et
risus purus. Praesent eget gravida nunc. Nunc id nisl commodo mauris maximus ultricies. Etiam congue aliquam elit, a
sollicitudin velit tincidunt sed. Integer consequat erat at turpis fringilla elementum. Phasellus scelerisque dui nec molestie
scelerisque. Phasellus ut mattis metus. Duis lectus ipsum,

Current Problems in Diagnostic Radiology, Corrected proof. doi:10.1067/j.cpradiol.2020.10.007

At vero eos et accusamus et iusto odio dignissimos ducimus qui blanditiis praesentium voluptatum deleniti atque corrupti
quos dolores et quas molestias excepturi sint occaecati cupiditate non provident, similique sunt in culpa qui officia
deserunt mollitia animi, id est laborum et dolorum fuga. Et harum quidem rerum facilis est et expedita distinctio. Nam
libero tempore, cum soluta nobis est eligendi optio cumque nihil impedit quo minus id quod

Curabitur varius est id magna facilisis, in suscipit tellus auctor. Donec suscipit sed ligula id condimentum. Cras ultrices
nisl non nisl interdum, at tincidunt ante tempor. Sed vitae elit quam. Mauris eget vehicula ligula, id sollicitudin ante.
Nullam semper venenatis sollicitudin. Etiam sem enim, rhoncus at lobortis vel, mattis et mauris. Ut posuere nisi ac lacus
pulvinar mollis.

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Ut vulputate augue eget odio placerat, in consectetur orci
tincidunt. In hac habitasse platea dictumst. Aliquam efficitur metus ligula, sit amet vehicula ligula rutrum ut. Duis mollis
nibh fringilla lacus maximus, vitae lobortis metus volutpat. Suspendisse ut sapien erat. Pellentesque commodo nulla vitae
facilisis dignissim. Vivamus at orci mi. Donec tristique rhoncus elit ac dapibus.

Cras metus lectus, pellentesque ullamcorper bibendum a, interdum non ligula. In ultrices sodales iaculis. Ut ipsum felis,
eleifend non nunc ut, pharetra molestie ex. In volutpat ante ex. Suspendisse bibendum tortor a lectus consectetur
scelerisque. Ut ut lacus dui. Ut non molestie velit. Sed efficitur enim in venenatis tristique. Mauris pulvinar libero quis
faucibus consectetur.

At vero eos et accusamus et iusto odio dignissimos ducimus qui blanditiis praesentium voluptatum deleniti atque corrupti
quos dolores et quas molestias excepturi sint occaecati cupiditate non provident, similique sunt in culpa qui officia
deserunt mollitia animi, id est laborum et dolorum fuga. Et harum quidem rerum facilis est et expedita distinctio. Nam
libero tempore, cum soluta nobis est eligendi optio cumque nihil impedit quo minus id quod

Current Problems in Diagnostic Radiology, Corrected proof. doi:10.1067/j.cpradiol.2020.10.007