Deep Recurrent Neural Networks for

Mortality Prediction in Intensive

Care using Clinical Time Series at
Multiple Resolutions
Suparna Ghanvatkar, Vaibhav Rajan
National University of Singapore
Short Paper
 Motivation for ICU Mortality
Models

In-hospital mortality models used for:

• Estimating severity of patients
• Identifying high risk patients
• Defining hospital policies
• Determining value of novel treatments and
interventions
 Data Sources - Heterogeneity

Electronic Medical Records Bedside Monitors

Demographics High frequency (125 Hz) – e.g. ECG

Laboratory and vitals Per sec/min – e.g. Heart Rate

Medications

Clinical Notes
Demographics
- Static

Medications –
sparse, hourly

Laboratory
– sparse,
irregular

ECG –
continuous,
125 Hz
 Previous Works

Electronic Medical Records Bedside Monitors

Demographics ECG

Laboratory and vitals

Medications

Benchmarking of tasks in ICU:

Harutyunyan, H., Khachatrian, H., Kale, D. C., Ver Steeg, G., and Galstyan, A. 2019. “Multitask Learning
and Benchmarking with Clinical Time Series Data,” Scientific Data (6:1), p. 96
 Previous Works

Electronic Medical Records Bedside Monitors

Demographics ECG

Laboratory and vitals

Medications

MGP-RNN
Futoma, J., Hariharan, S., and Heller, K. 2017. “Learning to detect sepsis with a multitask Gaussian process RNN classifier” in Proceedings of the 34th International
Conference on Machine Learning (70), pp. 1174-1182.
 Potential advantages

Electronic Medical Records Bedside Monitors

Demographics ECG

Laboratory and vitals

Medications

Provides complementing information about patient

Bedside monitor – not biased individual recordings
 • Given 24 hrs of ICU stay data of patient, predict the
Problem in-hospital mortality.
Definition • In-hospital mortality posed as a classification
problem.
 • Given 24 hrs of ICU stay data of patient, predict the
Problem in-hospital mortality.
Definition • In-hospital mortality posed as a classification
problem.
Sparse Time-Series – MGP-RNN

Futoma, J., Hariharan, S., and Heller, K. 2017. “Learning to detect sepsis with a multitask Gaussian process RNN classifier” in Proceedings of the 34th International
Conference on Machine Learning (70), pp. 1174-1182.
Sparse Time-Series – MGP-RNN

• Missing value imputation

• Uncertainty in imputed value
Our Model – Multi Time Scale RNN (MTS RNN)
At hourly input to RNN

Static – copied every hour

Hourly input of medication

given

Latent function at the hour

Statistical
aggregate
to every of RNN at the hour
second
Model – t = y sec t = (y+1) sec

MTS RNN 𝑟𝑡 …
𝐸𝑡

ECG
Model – t = y sec t = (y+1) sec

MTS RNN 𝑧𝑇 𝑟𝑡 …
𝐸𝑡

MGP

Glucose
ECG
Cholesterol
Sodium
 𝑟𝑡
Model – t = y sec t = (y+1) sec

MTS RNN 𝑧𝑇 𝑟𝑡 …
𝐸𝑡

MGP

Glucose
ECG
Cholesterol
Sodium
 𝑜

𝑟𝑡
Model – t = y sec t = (y+1) sec

MTS RNN 𝑧𝑇 𝑟𝑡 …
𝐸𝑡

MGP

Glucose
ECG
Cholesterol
Sodium
 TIME

T=x hrs T=(x+1) hrs 𝑜

ℎ𝑇 …

𝑟𝑡
Model – t = y sec t = (y+1) sec

MTS RNN 𝑧𝑇 𝑟𝑡 …
𝐸𝑡

MGP

Glucose
ECG
Cholesterol
Sodium
 Total ICU #stays with #stays
Data Stays in-hospital without in-
• MIMIC III Dataset + MIMIC mortality hospital
III Waveform Dataset mortality
• 10,282 matched patients
• Pre-processing steps as in 7317 788 6529
Harutyunyan et al. + filter
only patients with ECG

5 fold CV and training undersampling

(Condensed Nearest Neighbour)

Harutyunyan, H., Khachatrian, H., Kale, D. C., Ver Steeg, G., and Galstyan, A. 2019. “Multitask Learning and Benchmarking with Clinical Time Series Data,” Scientific Data (6:1), p. 96
 Evaluation results: Averages over 5-fold CV (Standard Deviation)
AUROC AUPR Sensitivity Specificity Precision F1 Score
SVM 0.5 (0) 0.1076 0 (0) 1 (0) 0 (0) 0 (0)
(0.0075)
KNN 0.512676 0.112754 0.0531 0.9724 0.2057 0.08210
(0.0075) (0.0112) (0.0162) (0.0135) (0.0666) (0.0229)

Evaluation MLP 0.51154

(0.0193)
0.11048
(0.0107)
0.3949
(0.5185)
0.6282
(0.4993)
0.08262
(0.0805)
0.0984
(0.1043)
LSTM 0.68026 0.2121 0.2812 0.89347 0.2423 0.25096
(0.0177) (0.0269) (0.1221) (0.0476) (0.0312) (0.0569)
MGP-RNN 0.7096 0.2341 0.4865 0.2186 0.0698 0.1220
(0.0306) (0.0465) (0.0324) (0.0159) (0.0049) (0.0081)
MTS-RNN 0.7094 0.24216 0.4946 0.7876 0.21903 0.3031
(0.0193) (0.0122) (0.0435) (0.0129) (0.01412) (0.0167)

• Performs significantly better on F1-score with p-value 0.001

 Evaluation results: Averages over 5-fold CV (Standard Deviation)
AUROC AUPR Sensitivity Specificity Precision F1 Score
SVM 0.5 (0) 0.1076 0 (0) 1 (0) 0 (0) 0 (0)
(0.0075)
KNN 0.512676 0.112754 0.0531 0.9724 0.2057 0.08210
(0.0075) (0.0112) (0.0162) (0.0135) (0.0666) (0.0229)

Evaluation MLP 0.51154

(0.0193)
0.11048
(0.0107)
0.3949
(0.5185)
0.6282
(0.4993)
0.08262
(0.0805)
0.0984
(0.1043)
LSTM 0.68026 0.2121 0.2812 0.89347 0.2423 0.25096
(0.0177) (0.0269) (0.1221) (0.0476) (0.0312) (0.0569)
MGP-RNN 0.7096 0.2341 0.4865 0.2186 0.0698 0.1220
(0.0306) (0.0465) (0.0324) (0.0159) (0.0049) (0.0081)
MTS-RNN 0.7094 0.24216 0.4946 0.7876 0.21903 0.3031
(0.0193) (0.0122) (0.0435) (0.0129) (0.01412) (0.0167)

Do not model temporal data – poor prediction

 Evaluation results: Averages over 5-fold CV (Standard Deviation)
AUROC AUPR Sensitivity Specificity Precision F1 Score
SVM 0.5 (0) 0.1076 0 (0) 1 (0) 0 (0) 0 (0)
(0.0075)
KNN 0.512676 0.112754 0.0531 0.9724 0.2057 0.08210
(0.0075) (0.0112) (0.0162) (0.0135) (0.0666) (0.0229)

Evaluation MLP 0.51154

(0.0193)
0.11048
(0.0107)
0.3949
(0.5185)
0.6282
(0.4993)
0.08262
(0.0805)
0.0984
(0.1043)
LSTM 0.68026 0.2121 0.2812 0.89347 0.2423 0.25096
(0.0177) (0.0269) (0.1221) (0.0476) (0.0312) (0.0569)
MGP-RNN 0.7096 0.2341 0.4865 0.2186 0.0698 0.1220
(0.0306) (0.0465) (0.0324) (0.0159) (0.0049) (0.0081)
MTS-RNN 0.7094 0.24216 0.4946 0.7876 0.21903 0.3031
(0.0193) (0.0122) (0.0435) (0.0129) (0.01412) (0.0167)

Sparse time-series only modelling: MGP based imputation has better

AUROC, AUPR and sensitivity
 Evaluation results: Averages over 5-fold CV (Standard Deviation)
AUROC AUPR Sensitivity Specificity Precision F1 Score
SVM 0.5 (0) 0.1076 0 (0) 1 (0) 0 (0) 0 (0)
(0.0075)
KNN 0.512676 0.112754 0.0531 0.9724 0.2057 0.08210
(0.0075) (0.0112) (0.0162) (0.0135) (0.0666) (0.0229)

Evaluation MLP 0.51154

(0.0193)
0.11048
(0.0107)
0.3949
(0.5185)
0.6282
(0.4993)
0.08262
(0.0805)
0.0984
(0.1043)
LSTM 0.68026 0.2121 0.2812 0.89347 0.2423 0.25096
(0.0177) (0.0269) (0.1221) (0.0476) (0.0312) (0.0569)
MGP-RNN 0.7096 0.2341 0.4865 0.2186 0.0698 0.1220
(0.0306) (0.0465) (0.0324) (0.0159) (0.0049) (0.0081)
MTS-RNN 0.7094 0.24216 0.4946 0.7876 0.21903 0.3031
(0.0193) (0.0122) (0.0435) (0.0129) (0.01412) (0.0167)

MTS-RNN has multi-time scale architecture which uses dense time-series.

Improved prediction performance
 Initial Contribution

Combination of both data sources can improve

prediction models

Use of multiple modalities as inputs, our model

can potentially counter the biases in individual
data inputs
 Future Work
Additional clinical variables include the use of more dense time
series variables as well as other modalities such as clinical notes and
imaging data
Model varying temporal gaps in the time-series.

Interpretability
Questions?