Editorials
and mean GCS (propensity matching not performed). Multi-
variable regression determined that β-blockers were associated
with lower mortality (adjusted odds ratio, 0.35; p < 0.001), and
propranolol was the superior agent (adjusted odds ratio, 0.51;
p = 0.010). A Cox regression model using a time-dependent
variable confirmed the mortality benefit.
There are several questions that still need to be addressed in
the design of such a trial. Besides decreasing HR, β-adrenergic
blockade can also decrease cardiac contractility, alter metabo-
lism, lead to decreased arterial resistance, and increase venous
vascular resistance (12). These effects could be deleterious to
a significant subgroup of patients with TBI. How would the
treatment be monitored and what surrogates (for efficacy and
safety) are appropriate? HR? Blood pressure? RPP? Serum
catecholamines? Echocardiographic markers? Would there be
monitoring of advanced hemodynamics and/or multimodal-
ity neuromonitoring? Which agent(s) should be tested and at
what dosing? This could be a highly anticipated trial due to
relative procedural simplicity of the intervention, in conjunc-
tion with biologic plausibility, robust observational data, and
low cost; as Derek Parfit said, “it is not irrational to have high
hopes.”
REFERENCES
1. Maas AIR, Menon DK, Adelson PD, et al; InTBIR Participants and
Investigators: Traumatic brain injury: Integrated approaches to
improve prevention, clinical care, and research. Lancet Neurol 2017;
16:987–1048
Mortality Prediction Gets a “Boost”*
David M. Maslove, MD, MS, FRCPC
Department of Critical Care Medicine
Queen’s University and Kingston Health Sciences Centre
Kingston, ON, Canada
T
he last few decades have seen the creation and sequen-
tial refinement of a host of ICU mortality prediction
models. Familiar models such as the Acute Physiology
and Chronic Health Evaluation, Simplified Acute Physiology
Score, and Mortality Probability Model use common clini-
cal features and regression analyses to predict ICU outcomes,
with the latest iteration of each of these displaying excellent
overall performance (1). Other composite scores, such as the
Sequential Organ Failure Assessment, can also be used to
stratify patients by mortality risk (2, 3), and even some simple
*See also p. e481.
Key Words: critical care; machine learning; medical informatics; predictive
value of tests; severity of illness index
Dr. Maslove has disclosed that he does not have any potential conflicts
of interest.
Copyright © 2018 by the Society of Critical Care Medicine and Wolters
Kluwer Health, Inc. All Rights Reserved.
DOI: 10.1097/CCM.0000000000003037
1024 www.ccmjournal.org June 2018 • Volume 46 • Number 6
Copyright © 2018 by the Society of Critical Care Medicine and Wolters Kluwer Health, Inc. All Rights Reserved.
2. Lazaridis C, Robertson CS: The role of multimodal invasive moni-
toring in acute traumatic brain injury. Neurosurg Clin N Am 2016;
27:509–517
3. Chesnut RM, Marshall LF, Klauber MR, et al: The role of secondary
brain injury in determining outcome from severe head injury. J Trauma
1993; 34:216–222
4. Carney N, Totten AM, O’Reilly C, et al: Guidelines for the manage-
ment of severe traumatic brain injury, fourth edition. Neurosurgery
2017; 80:6–15
5. Simard JM, Bellefleur M: Systemic arterial hypertension in head
trauma. Am J Cardiol 1989; 63:32C–35C
6. Meyfroidt G, Baguley IJ, Menon DK: Paroxysmal sympathetic hyper-
activity: The storm after acute brain injury. Lancet Neurol 2017;
16:721–729
7. Krishnamoorthy V, Vavilala MS, Chaikittisilpa N, et al: Association of
Early Myocardial Workload and Mortality Following Severe Traumatic
Brain Injury. Crit Care Med 2018; 46:965–971
8. Barmparas G, Liou DZ, Lamb AW, et al: Prehospital hypertension is
predictive of traumatic brain injury and is associated with higher mor-
tality. J Trauma Acute Care Surg 2014; 77:592–598
9. Alali AS, Mukherjee K, McCredie VA, et al: Beta-blockers and trau-
matic brain injury: A systematic review, meta-analysis, and eastern
association for the surgery of trauma guideline. Ann Surg 2017;
266:952–961
10. Grände PO: Critical evaluation of the lund concept for treatment of
severe traumatic head injury, 25 years after its introduction. Front
Neurol 2017; 8:315
11. Ley EJ, Leonard SD, Barmparas G, et al; Beta Blockers TBI Study
Group Collaborators: Beta blockers in critically ill patients with trau-
matic brain injury: Results from a multicenter, prospective, observa-
tional American Association for the Surgery of Trauma study. J Trauma
Acute Care Surg 2018; 84:234–244
12. Magder SA: The ups and downs of heart rate. Crit Care Med 2012;
40:239–245
univariate models have been shown to be predictive of ICU
mortality (4). All of which begs the question: In 2018, what
remains to be done in the field of ICU mortality prediction?
Enter the study by Delahanty et al (5), published in this issue of
Critical Care Medicine, which describes the development of the Risk
of Inpatient Death (RIPD) score, a 17-feature model showing best-
in-class performance in predicting in-hospital mortality among
patients admitted to the ICU. The RIPD score was developed from
a large dataset of more than 237,000 patients cared for in a diver-
sity of ICUs within a 53-hospital network. Like its predecessors, this
new construct relies on an assembly of patient indicators to gener-
ate a quantitative prediction. But the RIPD score differs from earlier
models in important ways, providing a glimpse into where ICU
mortality prediction may be heading in the era of big data analytics.
Four notable innovations show how the RIPD score uses data
science to modernize mortality prediction. First, the use of machine
learning algorithms enabled the authors to evaluate a much larger
number of clinical and administrative features for inclusion in
their final model. Whereas previous scores have looked only at a
small collection of hand-curated elements—including vital sign
measurements, laboratory values, and diagnostic details—the
algorithm used by the RIPD developers was able to sift through
a much larger selection of potentially predictive features. These

included not only conventional clinical features but also novel con-
structs, such as the mean, median, or most recent value of a given
indicator or even a change in that indicator’s value over time. The
capacity to select from a larger pool of features opens the door to
“engineered features” or “meta-features” that may show predictive
utility beyond any obvious clinical utility.
Second, while conventional mortality prediction models
rely on regression analysis, the RIPD score was derived using a
machine learning technique known as gradient boosting. This
form of “ensemble learning” relies on combining a large number
of simple prediction models to generate a single, more accurate
model. Ensemble learning has garnered significant attention
among data scientists for the striking prediction accuracy it has
achieved in a variety of settings. The XGBoost algorithm used in
developing the RIPD score is a form of gradient boosting that
has been deployed to great effect in a number of machine learn-
ing competitions. Gradient boosting shows promise for use with
big biomedical data (6, 7) and has recently been used to derive
ICU mortality prediction models from the popular Medical
Information Mart for Intensive Care database as well (8).
Third is the RIPD score’s emphasis on usability. While some
existing scores can be calculated automatically, the viability of
this approach may be constrained by the cost of acquiring soft-
ware or of ready access to emergency medical record (EMR) data.
Manual approaches require trained personnel abstracting data
from other sources, which may also incur substantial expense
and opportunity costs. Ostensibly, the RIPD score is open source,
which will increase its accessibility for some users. It also makes
use of data that could be recovered from some EMR systems.
Exceptions to this are the All Patient Refined Diagnosis Related
Group (APR-DRG) elements that were the first and third most
influential features in the model. The reliance on these adminis-
trative features undercuts the RIPD’s usability, as APR-DRG ele-
ments might also require software licenses and access to accurate
postdischarge diagnostic coding. Acknowledging this potential
limitation, the authors also developed the RIPD_reduced, a ver-
sion of the model that functions without APR-DRG codes and
that nonetheless maintains excellent performance.
Fourth, The RIPD achieves notable improvements over pre-
vious mortality predictors in terms of discrimination—how
well the model distinguishes patients who survive from those
who die. Breaking a potentially important (albeit arbitrary)
barrier, the model surpasses an area under the receiver oper-
ating characteristics curve of 0.9. This is a notable milestone
because even incremental gains are hard to come by at higher
levels of model performance, a phenomenon evidenced by the
Netflix Prize (9), in which the video streaming service awarded
$1 million to a team that was able to improve the predictive
performance of their algorithm by a mere 10%.
Further comparisons of the RIPD with its predecessor scores
may prove challenging. Previous mortality prediction models have
used the Hosmer-Lemeshow test to assess model calibration—how
well the model’s predictions compare with actual outcomes across
the entire range of mortality risk (10). But, because this test may
be overly pessimistic with larger cohorts (11), the authors used a
modified Brier score for this purpose, making direct comparison
with previous models difficult. They did, however, make assessing
Critical Care Medicine www.ccmjournal.org 1025
Copyright © 2018 by the Society of Critical Care Medicine and Wolters Kluwer Health, Inc. All Rights Reserved.
Editorials
calibration easier and more intuitive with another innovative fea-
ture, an online data exploration tool that allows readers to evaluate
model performance on customizable subsets of the overall cohort
(http://shiny.cac.queensu.ca/CritCareMed/RIPD/). Like the online
visualization tool published alongside, another recent assessment
of mortality prediction models by Badawi et al (12), this platform
allows the reader to parse the RIPD dataset based on diagnosis,
length of stay, and other variables, and directly assess model per-
formance for a more homogeneous group of patients at a similar
level of mortality risk.
There are a few limitations to consider. The RIPD score still
requires validation by different groups using different datasets,
and although an increase in usability is touted as one of the main
strengths of the RIPD score, it is possible that greater ease of use
is on the horizon for legacy ICU mortality predictors as well,
simply by virtue of the wider availability of EMR data. The RIPD
might also face barriers to deployment in resource-limited set-
tings or in ICUs that lack robust EMR systems. The RIPD score
does, however, illustrate how the advances in data science yield-
ing better predictions in other industries might be brought to
bear on the ever-expanding troves of clinical and administrative
data that characterize modern ICU care. Integrating these tools
with existing health information systems stands to increase the
uptake of benchmarking in critical care and make it easier for
researchers and administrators to compare ICU cohorts.
REFERENCES
1. Vincent JL, Moreno R: Clinical review: Scoring systems in the critically
ill. Crit Care 2010; 14:207
2. Timsit JF, Fosse JP, Troché G, et al; OUTCOMEREA Study Group,
France: Calibration and discrimination by daily Logistic Organ
Dysfunction scoring comparatively with daily Sequential Organ
Failure Assessment scoring for predicting hospital mortality in criti-
cally ill patients. Crit Care Med 2002; 30:2003–2013
3. Pettilä V, Pettilä M, Sarna S, et al: Comparison of multiple organ dys-
function scores in the prediction of hospital mortality in the critically
ill. Crit Care Med 2002; 30:1705–1711
4. Bazick HS, Chang D, Mahadevappa K, et al: Red cell distribution
width and all-cause mortality in critically ill patients. Crit Care Med
2011; 39:1913–1921
5. Delahanty RJ, Kaufman D, Jones SS: Development and Evaluation of
an Automated Machine Learning Algorithm for In-Hospital Mortality
Risk Adjustment Among Critical Care Patients.Crit Care Med 2018;
46:e481–e488
6. Torlay L, Perrone-Bertolotti M, Thomas E, et al: Machine learning-
XGBoost analysis of language networks to classify patients with epi-
lepsy. Brain Inform 2017; 4:159–169
7. Cobb AN, Daungjaiboon W, Brownlee SA, et al: Seeing the forest
beyond the trees: Predicting survival in burn patients with machine
learning. Am J Surg 2017 Nov 7. [Epub ahead of print]
8. Awad A, Bader-El-Den M, McNicholas J, et al: Early hospital mortality
prediction of intensive care unit patients using an ensemble learning
approach. Int J Med Inform 2017; 108:185–195
9. Netflix Prize. Available at: https://en.wikipedia.org/wiki/Netflix_Prize.
Accessed January 16, 2018
10. Keegan MT, Gajic O, Afessa B: Severity of illness scoring systems in
the intensive care unit. Crit Care Med 2011; 39:163–169
11. Kramer AA, Zimmerman JE: Assessing the calibration of mortality
benchmarks in critical care: The Hosmer-Lemeshow test revisited.
Crit Care Med 2007; 35:2052–2056
12. Badawi O, Liu X, Hassan E, et al: Evaluation of ICU risk models
adapted for use as continuous markers of severity of illness through-
out the ICU stay [Abstract 14]. Crit Care Med 2018; 46(Suppl 1):S7