Article 1

Charlson Comorbidity Indices collected from Anesthesiolo- 2

gists, Patients and Hospital Administrative Data agree fairly 3

but predict Postoperative In-Hospital Mortality equally well: A 4

Prospective Observational Study 5

Eike J Röhrig 1, Henning Schenkat 2, Nadine Hochhausen 1 , Anna B Röhl 1 , Matthias Derwall 1 , Rolf Rossaint 1 and 6
Felix Kork 1,* 7

1 Department of Anesthesiology, Medical Faculty, RWTH Aachen University; eike.roehrig@web.de; nhoch- 8

hausen@ukaachen.de; anna.roehl@rheinmaasklinikum.de; matthias.derwall@joho-dortmund.de; rross- 9
aint@ukaachen.de; fkork@ukaachen.de 10
2 Deanery of Studies, Medical Faculty, RWTH Aachen University; hschenkat@ukaachen.de 11
* Correspondence: fkork@ukaachen.de; Tel.: +49 241 80 37783 12

Abstract: Patients’ comorbidities play an immanent role in perioperative risk assessment. It is un- 13
known how Charlson Comorbidity Indices (CCIs) from different sources compare. In this prospec- 14
tive observational study, we compared the CCIs of patients derived from patients’ self-report and 15
from physicians’ assessment with hospital administrative data. The data of 1,007 patients was ana- 16
lyzed. Agreement between the CCI from patients’ self-report compared to administrative data was 17
fair (kappa0.24;[95%CI 0.2–0.28]) and patients overestimated their comorbidities by half a point 18
(mean difference 0.39;limits of agreement -3.70–4.48). Agreement between physicians’ assessment 19
and the administrative data was also fair (kappa0.28[95%CI 0.25–0.31]), physicians overestimated 20
the CCI by one point (mean difference0.74; limits of agreement -3.19–4.68). Physicians’ assessment 21
and patients’ self-report had the best agreement (kappa0.33[95%CI 0.30–0.37]) and physicians over- 22
estimated comorbidities compared to patients by less than half a point (mean difference 0.34;limits 23
of agreement -3.25–3.95). The CCI calculated from the administrative data showed the best predict- 24
ability for in-hospital mortality (AUROC0.86[95%CI 0.68–0.91]), followed by equally good predic- 25
tion from physicians’ assessment (AUROC0.80[95%CI 0.65–0.94]) and patients’ self-report (AU- 26
ROC0.80[95%CI 0.75–0.97]). CCIs derived from patient’s self-report, physicians’ assessment and ad- 27
ministrative data perform equally well in predicting postoperative in-hospital mortality. 28
Citation: To be added by editorial
staff during production.
Keywords: outcome, frailty, perioperative care 29
30
Academic Editor: Firstname Last-
name

Received: date 1. Introduction 31

Revised: date
According to the World Health Organization, in 2012 approximately 312.9 million 32
Accepted: date
surgeries were performed worldwide [1], which is a 33.6 percent increase from its previ- 33
Published: date
ous approximation of 234.2 million operations in 2005 [2]. A recently published study re- 34
ported that in 2015 266 million operations were performed worldwide. The lower number 35
compared to the data of 2012 were explained, according to the authors, due to the exclu- 36
sion of data from before 2010 and extrapolation from subnational data [3]. The European 37
Copyright: © 2024 by the authors.
Submitted for possible open access
Surgical Outcome Study found that one in 25 of non-cardiac surgery patients in Europe 38

publication under the terms and

conditions of the Creative Commons
Attribution (CC BY) license
(https://creativecommons.org/licenses
/by/4.0/).
dies before discharge [4]. Although this number has later been revised downwards, a re- 39
cent estimate suggests that at least 4.2 million people worldwide die within 30 days of 40
surgery [5], making it the third greatest cause of death with 7.7% of all deaths globally [6]. 41
Besides the immanent risk of the surgical procedure, patients’ comorbidities are one 42
of the most relevant risk factors for perioperative mortality. Different scores exist to 43

J. Clin. Med. 2021, 10, x. https://doi.org/10.3390/xxxxx www.mdpi.com/journal/jcm

J. Clin. Med. 2021, 10, x FOR PEER REVIEW 2 of 14

evaluate the risk of perioperative morbidity and mortality like the American Society of 44
Anesthesiologists (ASA) physical status. [7] or the Elixhauser Comorbidity Score [8-10]. A 45
good compromise due to the poor interrater reliability or complexity of the scores could 46
be the Charlson Comorbidity Index (CCI). The CCI was originally designed by Charlson 47
et al. in 1987 to predict the mortality of hospitalized patients [11]. While initially designed 48
for the general hospital population, the CCI has been frequently used in surgical contexts 49
[12-18]. Although sometimes the score has been calculated through a self-report question- 50
naire [19,20], it has more commonly been calculated from patients’ administrative data 51
[8,9] using one of the existing administrative data adaptations [21,22]. When calculated 52
from administrative data, the CCI is often used in retrospective analyses to account for 53
patients’ comorbidities. However, very little scientific data exist on the validity of the CCI 54
in the perioperative setting in general, but moreover, whether the CCI calculated from 55
administrative data is a suitable instrument to describe and control for comorbidities ret- 56
rospective analyses. 57
We therefore conducted a prospective observational study comparing (1) the pro- 58
spective CCI from patients’ self-report before and (2) anesthesiologists’ assessment after 59
the consultation with the anesthesiologist in our pre-procedural evaluation center to (3) 60
the CCI calculated from our hospital’s administrative data after patients’ discharge. Sec- 61
ondary aim of our study was to compare the predictive capability for perioperative mor- 62
tality of these different CCIs. 63
64

2. Materials and Methods 65

2.1. Ethics and consent 66
The data safety monitor of the University Hospital RWTH Aachen and the Ethics 67
Committee of the Medical Faculty, RWTH Aachen University, approved of the Study (EK 68
056/17) on 1 March 2017. Patients were included after informed written consent. The study 69
has been reported in line with the STROCSS criteria [23]. 70
71
2.2. Patients and setting 72
In our tertiary care hospital, every patient undergoing elective surgery with anesthe- 73
sia is scheduled for an appointment in our pre-procedural evaluation center. Besides pre- 74
anesthesiologic evaluation screening, the attending anesthesiologist discusses possible an- 75
esthesia plans with the patient and documents informed consent to the proposed anesthe- 76
sia procedures. All patients aged ≥18 years between March 7th and April 13th, 2017, were 77
invited to participate in this prospective study. All invites took place before consulting 78
with the anesthesiologist. Patients were not invited when they did not speak German or 79
were unable to fill out the questionnaire. There was no sample size calculation, we aimed 80
to include 1,000 patients. 81
82
2.3. Charlson comorbidity index from two questionnaires 83
While waiting for the anesthesiologic consultation, participating patients filled out a 84
paper-based questionnaire (Table S1). Patients were asked to estimate their state of health 85
on a scale of 0 to 100, and to acknowledge the existence of comorbidity items of the CCI. 86
The questionnaires were returned before consulting with the anesthesiologist. The con- 87
sulting physician was asked to fill out a similar paper-based questionnaire (Table S2). An- 88
esthesiologists had to state their work experience and answer the items of the CCI for the 89
respective patient. 90
91
2.4. Charlson Comorbidity Index and in-hospital Mortality from Hospital Administrative 92
Data 93
In Germany, hospitals are legally bound by section 21 of the Hospital Reimbursement 94
Act (§21 Krankenhausentgeldgesetz, KHEntgG) to collect specific data from all their pa- 95
tients: all diagnoses in international statistical classification of diseases and related health 96
J. Clin. Med. 2021, 10, x FOR PEER REVIEW 3 of 14

problems (ICD) codes, patients` procedures in operation and procedure (Operationen- 97

und Prozeduren-Schlüssel, OPS) codes, as well as data on length of stay and type of dis- 98
charge. These data are forwarded to the Institute for Reimbursement in Hospitals (Institut 99
für das Entgeldsystem im Krankenhaus, InEK) which calculates case-based compensa- 100
tions. 101
From these locally archived data, we abstracted the ICD-10 diagnoses, hospital length 102
of stay, whether the patient underwent out-patient or in-patient surgery, and whether the 103
patient died during the hospital stay. CCIs were calculated from the ICD-10 diagnoses 104
from the hospital data management system using the ‘icd’ package for R which is based 105
on the algorithm introduced by Quan [22]. 106
107
2.5. Statistical analyses 108
We prespecified the aims of this study prior obtaining the ethics committee’s ap- 109
proval of our study. We planned to analyze our data using the descriptive measures Co- 110
hen’s Kappa and Bland-Altman-plots. As these measures are merely descriptive and do 111
not test a hypothesis, a sample size calculation was not conducted, but we aimed at in- 112
cluding 1,000 patients. For these reasons, all analyses must be considered exploratory and 113
hypothesis tests were conducted post-hoc. Cases with missing data in the questionnaires 114
or administrative data were excluded from the analysis. Although central limit theorem 115
would apply, we used non-parametric testing and reporting because all three methods to 116
calculate the CCI resulted in heavily skewed distributions. Continuous variables were 117
therefore reported as median and interquartile range, frequencies as absolutes and per- 118
centages. Agreement between continuous variables was assessed calculating both un- 119
weighted and weighted Cohen’s kappa with the ‘psych’ package for R, comparisons be- 120
tween agreements were done post-hoc and likewise conducted the ‘psych’ package. In 121
addition, agreement was graphically assessed using Bland-Altman plots from the ‘Blan- 122
dAltmanLeh’ package for R. Prediction of binary outcomes from continuous variables 123
were compared by plotting Receiver Operating Characteristics (ROC) using the ‘pROC’ 124
package for R, as were the post-hoc pairwise comparisons of ROC curves. All analyses 125
were conducted using R for MacOs (www.r-project.org) and figures were created using 126
GraphPad Prism 8 for MacOs. The probability of a type I error <0.05 was considered stat- 127
ically significant. Correlations were described as following: <=0 no correlation, 0.01-0.20 128
poor correlation, 0.21-0.40 fair correlation, 0.41-0.60 moderate correlation, 0.61-0.80 good 129
correlation and 0.81-1.0 perfect correlation [24]. 130
131
132

3. Results 133

3.1 Patients 134

During the study period, a total of 1,580 patients 18 years and older visited our pre- 135
procedural evaluation centre, a total of 1,114 were approached for participation in our 136
study, 63 declined to participate, and 1,051 were successfully recruited for inclusion in 137
our study. After the exclusion of 44 incomplete datasets, the data of 1,007 patients were 138
analyzed (Figure S1). 139

Patients were 52.1% (n = 525) female and the had a median age of 57 [IQR 39 to 58; Table 140
1]. 141

142

143
J. Clin. Med. 2021, 10, x FOR PEER REVIEW 4 of 14

Table 1. Characteristics of the study population. Patients (n = 1,007) were assessed in our pre-anesthetic evaluation 144
clinic before elective surgery. 145

Participants (n = 1,007)
Characteristics
n, median (%), [IQR]

Age, years 57 [39 – 68]

Sex, female 525 (52.14)

Type of Surgery

Inpatients 581 (57.7)

Outpatients 426 (42.3)

Charlson Comorbidity Indexa 1 [0–3]

Malignancy without metastases 182 (18.07)

Congestive heart failure 65 (6.45)
Peripheral vascular disease 65 (6.45)
Chronic pulmonary disease 58 (5.76)
Diabetes without complication 55 (5.46)
Renal disease 51 (5.06)
Metastatic solid tumor 49 (4.87)
Cerebrovascular disease 40 (3.97)
Mild liver disease 26 (2.58)
Myocardial Infarction 19 (1.89)
Hemiplegia 11 (1.09)
Diabetes with chronic complication 10 (0.99)
Rheumatic disease 10 (0.99)
Moderate or severe liver disease 4 (0.40)
Peptic Ulcer Disease 2 (0.20)
Dementia 1 (0.10)
AIDS 0 (0.00)

Outcome

In-hospital mortalityb 5 (0.5)

Hospital length of stayb, days 5.20 [3.15–10.03]
anumerical Charlson Comorbidity Index and items from administrative data; b of 581 inpatients. 146
AIDS: Acquired immunodeficiency syndrome. 147

148

149

150

151

152
J. Clin. Med. 2021, 10, x FOR PEER REVIEW 5 of 14

42.3% (n = 426) of the study population underwent outpatient surgery while more than 153
half of the patients (54.1%; n = 581) underwent inpatient surgery. In-hospital mortality 154
was 0.9% (n = 5) in the inpatient population. According to the CCI derived from the 155
administrative data, “Malignancy without metastases” was the most common 156
comorbidity with 18.1% (n = 182) in our study cohort, followed by “Congestive heart 157
failure” and “Peripheral vascular disease” with 6.5% (n = 65) each. The least common 158
comorbidities were “Moderate or severe liver disease” with 0.4% (n = 4), “Peptic ulcer 159
disease” with 0.2% (n = 2), and “Dementia” with 0.1% (n = 1). No patient with “HIV or 160
AIDS” was included in our study (Table 1). 161

162

3.2 Numerical score of the Charlson Comorbidity Index 163

First, the numerical scores of the CCI obtained by the anesthesiologists’ assessment 164
(CCIphysician), the patients’ self-report (CCIpatients) and the calculations from the 165
administrative data (CCIadministrative) were compared. Correlation between the 166
anesthesiologists’ assessment and the administrative data was moderate ‘r = 0.59 (95% 167
CI 0.55 to 0.63); p < 0.001; Figure 1A)’ and the agreement was fair ‘kappa 0.28 0.25 to 168
0.31)’. The Bland-Altman-Analyses revealed that anesthesiologists overestimated the 169
CCI score by approximately one point ‘mean difference 0.74 (limits of agreement -3.19 to 170
4.68); Figure 1B)’. Similarly, patients’ self-report correlated moderately with the 171
administrative data ‘r = 0.52 (95% CI 0.48 to 0.57); p < 0.001; Figure 1C’, the agreement 172
was fair ‘kappa 0.24 (95%CI 0.21 to 0.28)’ and patients overestimated their comorbidities 173
half a point compared to the administrative data ‘mean difference 0.39 (limits of 174
agreement -3.70 to 4.48); Figure 1D’. Anesthesiologists’ assessment and patients’ self 175
report correlated best ‘r = 0.67 (95%CI 0.63 to 0.70); p = 0.001; Figure 1E’ had the best 176
agreement ‘kappa 0.33 (95%CI 0.30 to 0.37)’ and anesthesiologists overestimated 177
comorbidities compared to patients by less than half a point ‘mean difference 0.34 (limits 178
of agreement -3.25 to 3.95)’; Figure 1F, see also Table S3). 179

180
J. Clin. Med. 2021, 10, x FOR PEER REVIEW 6 of 14

181

Figure 1. Pairwise comparison of Charlson Comorbidities Indices (CCIs) from anesthesiologists’ assessment (CCIphysician), pa- 182
tients’ self-report (CCIpatient) and administrative data (CCIadministrative) in 1,007 surgical patients. On the left, heat maps of cor- 183
relation and on the right, heatmap Bland-Altman plots of CCIs derived from (A, B) anesthesiologists’ assessment and administra- 184
tive data, (C, D) patients’ self-report and administrative data, and (E, F) anesthesiologists’ assessment and patients’ self-report. 185

186

187

3.3 Individual Items of the Charlson Comorbidity Index 188

Frequency of the individual CCI items from anesthesiologists’ assessment and 189
patients’ self-report are displayed in Table 2, from the administrative data in Table 1. 190
The agreement of these individual CCI items between anesthesiologists’ assessment, 191
self-report and administrative data are shown in Table 2. Overall, most items only show 192
a fair to moderate agreement. 193

194
J. Clin. Med. 2021, 10, x FOR PEER REVIEW 7 of 14

Table 2. Frequency of Charlson Comorbidity Index items in anesthesiologists’ and patients’ questionnaires in 1,007 pre-anesthesio- 195
logic consultations. 196

Charlson Comorbidities Anesthesiologists Patients

Malignancy without metastases 210 (20.85%) 151 (15.00%)

Congestive heart failure 106 (10.53%) 68 (6.75%)
Peripheral vascular disease 113 (11.22%) 116 (11.52%)
Chronic pulmonary disease 136 (13.51%) 95 (9.43%)
Diabetes without complication 92 (9.14%) 88 (8.74%)
Renal disease 107 (10.63%) 89 (8.84%)
Metastatic solid tumor 60 (5.96%) 57 (5.66%)
Cerebrovascular disease 63 (6.26%) 13 (1.29%)
Mild liver disease 28 (2.78%) 31 (3.08%)
Myocardial Infarction 89 (8.84%) 68 (6.75%)
Hemiplegia 13 (1.29%) 10 (0.99%)
Diabetes with chronic complication 24 (2.38%) 16 (1.59%)
Rheumatic disease 47 (4.67%) 59 (5.86%)
Moderate or severe liver disease 33 (3.28%) 28 (2.78%)
Peptic Ulcer Disease 39 (3.87%) 23 (2.28%)
Dementia 7 (0.70%) 6 (0.60%)
AIDS 1 (0.10%) 2 (0.20%)
AIDS: Aquired immunodeficiency syndrome. 197

198

When comparing physician’s assessment with the administrative data, agreement was 199
only moderate: “Congestive Heart Failure” ‘kappa 0.54 (95%CI 0.44 to 0.53)’, “Metastatic 200
solid tumor” ‘Cohen’s kappa 0.54 (95%CI 0.43 to 0.66)’ and “Malignancy without 201
metastases” ‘kappa 0.51 (95%CI 0.44 to 0.57)’ showed the best agreement. In contrast, 202
“Peptic Ulcer Disease”, “Dementia” and “AIDS” showed no agreement (Table 3). 203

204

Table 3. Agreement for physicians’ assessment, patients’ self-report and administrative data for 205
each item of the Charlson Comorbidity Index (Cohen’s kappa with 95%CI) in 1,007 surgical pa- 206
tients. 207

Physicians’ assessment Patients’ self-report Physicians’ assessment

vs. vs. vs.
administrative data administrative Data patients’ self-report

Malignancy without metastases 0.51 (0.44 – 0.57) 0.46 (0.38 – 0.53) 0.50 (0.43 – 0.57)

Congestive heart failure 0.54 (0.44 – 0.63) 0.46 (0.21 – 0.42) 0.50 (0.40 – 0.59)

Peripheral vascular disease 0.34 (0.24 – 0.43) 0.46 (0.09 – 0.26) 0.32 (0.23 – 0.40)
J. Clin. Med. 2021, 10, x FOR PEER REVIEW 8 of 14

Chronic pulmonary disease 0.37 (0.28 – 0.46) 0.39 (0.29 – 0.49) 0.49 (0.41 – 0.57)

Diabetes without complication 0.41 (0.30 – 0.51) 0.42 (0.32 – 0.53) 0.63 (0.55 – 0.72)

Renal disease 0.46 (0.36 – 0.55) 0.40 (0.30 – 0.51) 0.62 (0.53 – 0.70)

Metastatic solid tumor 0.54 (0.43 – 0.66) 0.46 (0.34 – 0.58) 0.63 (0.52 – 0.73)

Cerebrovascular disease 0.28 (0.16 – 0.39) 0.10 (-0.02 – 0.21) 0.11 (0.01 – 0.21)

Mild liver disease 0.16 (0.02 – 0.30) 0.12 (-0.01 – 0.24) 0.28 (0.13 – 0.44)

Myocardial Infarction 0.29 (0.18 – 0.40) 0.32 (0.20 – 0.45) 0.70 (0.62 – 0.79)

Hemiplegia 0.24 (0.01 – 0.47) 0.09 (-0.09 – 0.26) 0.25 (0.01 – 0.49)

Diabetes with chronic complication 0.40 (0.19 – 0.61) 0.22 (0.00 – 0.44) 0.34 (0.15 – 0.53)

Rheumatic disease 0.27 (0.12 – 0.42) 0.16 (0.04 – 0.28) 0.30 (0.18 – 0.42)

Moderate or severe liver disease 0.10 (-0.03 – 0.24) 0.18 (0.00 – 0.36) 0.44 (0.28 – 0.60)

Peptic Ulcer Disease 0.00 (-0.01 – 0.00) 0.00 (-0.01 – 0.00) 0.30 (0.15 – 0.46)

Dementia 0.00 (0.00 – 0.00) 0.28 (-0.15 – 0.72) 0.15 (-0.12 – 0.42)
AIDS 0.00 (0.00 – 0.00) 0.00 (0.00 – 0.00) 0.00 (0.00 – 0.00)
AIDS: acquired immunodeficiency syndrome; CI: Confidence interval 208

209

The comparison between the patients’ self-report with the administrative data revealed 210
poorer agreement: Similar to the comparison of the physician’s assessment compared 211
with the administrative data, the best agreement was also found for the two items 212
“Malignancy without metastases” ‘kappa 0.46 (95%CI 0.38 to 0.53)’ and “Metastatic solid 213
tumor” ‘kappa 0.46 (95%CI 0.34 to 0.58)’, followed by “Diabetes without complication” 214
‘kappa of 0.42 (95%CI 0.32 to 0.53)’ and “Renal Disease” ‘kappa of 0.40 (95%CI 0.30 to 215
0.51)’ with moderate agreement. No agreement was found for the items “Peptic Ulcer 216
Disease” and “AIDS” (Table 3). 217

The best agreements were found when comparing the anesthesiologists’ assessment 218
with the patients’ self-report. Substantial agreement was found for the item “Myocardial 219
infarction” ‘kappa of 0.70 (95%CI 0.62 to 0.79)’, “Diabetes without complication” ‘kappa 220
0.63 (95%CI 0.55 to 0.72)’, “Metastatic solid tumor” ‘kappa 0.63 (95%CI 0.55 to 0.72)’ and 221
“Renal Disease” ‘kappa 0.63 (95%CI 0.52 to 0.73)’. Again, there was no agreement for the 222
item “AIDS/HIV” (Table 3). 223

224

3.4 Prediction of in-hospital Mortality 225

Figure 2 shows the predictive capabilities of the numerical CCIs for in-hospital 226
mortality in the subgroup of 581 surgical inpatients. The CCI calculated from the 227
administrative data showed the best predictability for in-hospital mortality ‘AUROC 228
0.80 (95%CI 0.66 to 0.94)’. The predictive capabilities by the CCIs from the 229
anesthesiologists’ assessment and the patients’ self-report were equally good in 230
predicting in-hospital death ‘CCIphysician AUROC 0.75 (95%CI 0.62 to 0.88) and 231
CCIpatient AUROC 0.77 (95%CI 0.61 to 0.93), p = 0.88’, but both were inferior compared 232
J. Clin. Med. 2021, 10, x FOR PEER REVIEW 9 of 14

to the predictive capabilities of the administrative data, although not reaching statistical 233
significance (p > 0.58). 234

235

236
237

238

Figure 2. Prediction of surgical in-hospital mortality by three differently derived Charlson Comor- 239
bidity Indices (CCIs) in 581 surgical in-patients. The CCI calculated from the administrative data 240
(CCIadministrative) showed the best (AUROC 0.86) predictability for in-hospital mortality. Pre- 241
dictions by the CCIs from the anesthesiologists’ assessment (CCIphysician AUROC 0.80) and the 242
patients’ self-report of comorbidities (CCIpatient AUROC 0.80) equally good but inferior to the 243
CCI calculated from the administrative data. Pairwise comparison of the ROC curves was not sta- 244
tistically significant (p >0.18). 245

246

247

4. Discussion 248
In this prospective observational study, we compared three differently collected ver- 249
sions of the CCI in the perioperative setting: one from anesthesiologists’ during pre-pro- 250
cedural assessment, one from patients’ self-report before pre-procedural assessment, and 251
one calculated from the hospitals’ administrative data after patients’ discharge. Regarding 252
the numerical CCI, we found that there was only fair agreement between the anesthesiol- 253
ogists’ assessment and the patients’ self-report when compared with the administrative 254
J. Clin. Med. 2021, 10, x FOR PEER REVIEW 10 of 14

data. Both the anesthesiologists and the patients overestimated the CCI by 1 and 0.5 255
points, respectively. Best agreement was found between anesthesiologists’ assessment 256
and patients’ self-report. When comparing the individual items, “tumor without metasta- 257
ses”, “metastatic solid tumor” and “congestive heart failure” showed the best agreements 258
in all pair-wise comparisons. Surprisingly, and despite the poor agreement of the three 259
CCIs, all three presented themselves to be equally good predictors of in-hospital mortal- 260
ity. 261
One strength of our prospective observational study is the sample size of 1,007 with 262
a fairly even split between in- and outpatients (58% vs. 42%) as well as females and males 263
(52% vs. 48%). We explicitly designed the data acquisition in a way that filling out the 264
questionnaires by patients and anesthesiologists did not coincide. However, we cannot 265
rule out that patients filling out the questionnaire while waiting for consultation with the 266
anesthesiologist influenced patients’ behavior and reporting in their subsequent 267
pre-anesthesiologic consultation. 268
A high rate of 90% of the approached patients were included in the study, but we 269
could only approach 71% of eligible patients. This has most likely introduced bias in our 270
study. In addition, we only approached patients who were scheduled for elective surgery 271
who were able to come to our evaluation center either by foot or in a wheelchair. This fact 272
most likely skewed our study population, especially the inpatient subgroup, towards 273
younger and/or physically fitter patients resulting in lower CCI scores and a lower mor- 274
tality rate in in-patients. Additionally, as the questionnaires were in German, non-Ger- 275
man-speakers were not included. It is possible that a different set of comorbidities may be 276
more prevalent in this group and could lead to different agreements. In addition, gener- 277
alizability of these results may be limited because a higher proportion of comorbid pa- 278
tients are treated in German tertiary care university hospitals compared to hospitals of 279
lower care. 280
Misclassification bias for comorbid conditions may have happened because of inac- 281
curate diagnosis coding. Therefore, administrative data would rather underestimate pa- 282
tients’ comorbidities than misclassified. This goes in line with the fact, that anesthesiolo- 283
gists and patients underestimated the numerical CCI compared to administrative data. 284
Altogether, we therefore assume high data quality. 285
Our study has two unique aspects compared to other studies. First, we compared 286
three differently collected CCIs while most other studies compared either patients’ self- 287
report with administrative data [25-28] or anesthesiologists’ assessment with administra- 288
tive data [29]. Furthermore, only few studies compared anesthesiologists’ assessment with 289
patients’ self-report [29]. We are not aware of any study comparing three different CCIs 290
in a diverse surgical patient population, especially not in a pre-procedural anesthesia eval- 291
uation center. Subject of most comparable studies were population-based analyses [27,30], 292
emergency patients [25], or patients with arthritis [29]. 293
We found that anesthesiologists’ assessment overestimated the CCI from adminis- 294
trative data by approximately one point. This fact may be explained by differing patient 295
perception of different CCI sources: While the paper-based questionnaire was filled out 296
by anesthesiologists, the ICD diagnosis codes were added to the administrative data on 297
the surgical wards. It is likely that anesthesiologists rather focus on the cardiopulmonary 298
fitness, whereas surgeons rather focus on the surgical reason for admission. This conjec- 299
ture is supported by the fact that the agreement of particular items of the CCI differ. While 300
agreement between the anesthesiologists’ assessment with administrative data on surgical 301
aspects such malignancies with or without metastases were rather high, agreement was 302
poorer on comorbidities (Table 3) that are well known to be associated with poorer cardi- 303
opulmonary fitness and therefore with adverse perioperative outcomes, i.e. diabetes 304
[31,32], renal disease [33,34], and liver disease [13,35]. 305
A similar effect could explain the CCI overestimation of patients’ self-report when 306
compared to administrative data – or undercoding in the administrative data compared 307
to patients’ self-report. In fact, other studies have demonstrated similar results. For 308
J. Clin. Med. 2021, 10, x FOR PEER REVIEW 11 of 14

example, one study found fair to poor agreement when comparing patients’ self-reported 309
CCI with the CCI calculated from administrative data in 520 emergency room patients 310
[25]. The authors explain their findings solely by the high age in their study population 311
and the unreliable report of comorbidities by the elderly. The median age in our study 312
was indeed lower but most likely had a similar effect to the one described above: The 313
predominantly surgically driven diagnoses coding for the administrative data could have 314
led to ‘undercoding’ of certain ‘non-surgical’ comorbidities while in contrast, patients and 315
anesthesiologists were more meticulous when reporting or detecting these comorbidities. 316
Two facts support this notion: First, single “non-surgical” items with great impact on the 317
numerical CCI score had poor agreement, e.g. diabetes with complications (2 points) or 318
moderate to severe liver disease. Second, anesthesiologists and patients came closest 319
when assessing these comorbidities especially in contrast to the comparison of either as- 320
sessment with the administrative data. As a matter of fact, while only 55 patients had 321
diabetes with complications (2 points as item of the CCI) coded in the administrative data, 322
88 patients reported this comorbidity and anesthesiologists detected it in 92 patients (Ta- 323
ble 1 and Table 2). The magnitude of this effect is even more pronounced for moderate to 324
severe liver disease (3 points as item of the CCI): Only four patients had diabetes with 325
complications coded in the administrative data, while 28 patients reported it, and anes- 326
thesiologists detected it in 33 patients. In addition, these two examples also demonstrate 327
why patients’ self-report and anesthesiologists’ assessment had the best agreement. 328
The CCI, established in 1987 by Charlson and colleagues as a tool for classifying 329
comorbid conditions [11], has since been validated to assess and adjust for comorbidities 330
in various patient populations [8,19,36]. With the possibility to calculate the CCI automat- 331
ically from administrative data, it became an easy-to-use popular tool used in a multitude 332
of studies [9]. However, the CCI has not been validated for all types of patients. To our 333
knowledge, there is no validation for surgical patients. In this manner, we have found that 334
adjusting the weights of the CCI in a sample of 180,000 surgical patients largely improves 335
the predictability of in-hospital mortality in a general surgical population [13]. The ques- 336
tion remained, whether anesthesiologists’ assessment or patients self-report of CCI items 337
were better compared to administrative data in surgical patients. The results of this study 338
are limited by the small number of events and the in part very poor agreement on single 339
items between the different CCI sources. However, the predictability for in-hospital mor- 340
tality was surprisingly good for all three CCI scores (AUROC ≥0.80). This is well in line 341
with results from a few other studies. A just recently published systematic review found 342
26 studies that predicted mortality from the CCI with an AUROC from 0.66 to 0.90 [37]. 343
Notably, none of the included studies were conducted in surgical patients, most studies 344
investigated patients on internal medicine wards and acute geriatric wards. Despite the 345
limitations in our surgical patient sample, the CCI from patients’ self-report, anesthesiol- 346
ogists’ assessment or hospital administrative data appears to be equally well suited for 347
the prediction of in-hospital mortality and can therefore be used equally well when ad- 348
justing for comorbidities. 349
In addition to facilitating research in perioperative patient population, our findings 350
may also have an impact on clinical practice in the future. An automated comorbidity 351
assessment by self-report or from administrative data may improve and expedite the pre- 352
operative assessment processes. Anesthesiologists could already be presented with a de- 353
tailed comorbidity profile of the patient at first consultation, allowing them to faster plan 354
a tailored anesthesiologic and perioperative treatment. Moreover, together with infor- 355
mation on the planned surgery and a self-learning model form retrospective hospital data, 356
anesthesiologists and surgeons could be presented with detailed risk estimates for indi- 357
vidual patients allowing them to take individually tailored pre-, intra-, and postoperative 358
interventions or safety measures to improve patient’s outcome. 359

5. Conclusions 360
J. Clin. Med. 2021, 10, x FOR PEER REVIEW 12 of 14

Agreement of the CCI of anaesthesiologists’ assessment with administrative data, 361

patient’s self-report with administrative data, and patients’ self-report with 362

anaesthesiologists’ assessment is only fair. In pair-wise comparisons, patients’ self-report 363

and anaesthesiologists’ assessment reached the best agreement. When assessing the 364

predictive capacity of these three CCI, all performed equally well in predicting in-hospital 365

mortality with a slight advantage when calculated from administrative data. Self-report, 366

anaesthesiologists’ assessment as well as administrative data can therefore be used equally 367

well to assess or control for comorbidities in surgical patient populations when researching 368

perioperative outcomes and could also expedite assessment processes and improve patient 369

care in the future when automated and integrated reasonably in routine patient care. 370

Supplementary Materials: The following supporting information can be downloaded at: 371
www.mdpi.com/xxx/s1, Table S1: Patients questionnaire; Table S2: Anesthesiologists questionnaire; 372
Table S3: Pairwise comparison of the Charlson Comorbidity Index (CCI), Figure S1: Flow chart of 373
patient inclusion. 374

Author Contributions: “Conceptualization, F.K.; methodology, H.S., N.H., A.B.R. and F.K..; formal 375
analysis, E.J.R. and F.K.; investigation, E.J.R..; resources, H.S. and M.D.; writing—original draft 376
preparation, E.J.R.; writing—review and editing, N.H., A.B.R., M.D., R.R. and F.K.; visualization, 377
E.J.R. and F.K.; supervision, R.R..; project administration, F.K. All authors have read and agreed to 378
the published version of the manuscript.” 379

Funding: This research received no external funding. 380

Institutional Review Board Statement: The study was conducted in accordance with the Declara- 381
tion of Helsinki and approved by the Ethics Committee of the Medical Faculty, RWTH Aachen Uni- 382
versity (EK 056/17 on 1 March 2017). 383

Informed Consent Statement: Informed consent was obtained from all subjects involved in the 384
study. 385
386
Data Availability Statement: Data is contained within the article or supplementary material. 387

Acknowledgments: The authors thank Adrijana Kolar for her patience and logistic support in our 388
pre-procedural evaluation center. 389

Conflicts of Interest: The authors declare no conflicts of interest. 390

References 391
392
1. Weiser T.G.; Haynes A.B.; Molina G.; Lipsitz S.R.; Esquivel M.M.; Uribe-Leitz T.; Fu R.; Azad T.; Chao T.E.; Berry W.R; Gawande 393
A.A. Size and distribution of the global volume of surgery in 2012. Bulletin of the World Health Organization 2016, 94:201-209f. 394
2. Weiser T.G; Regenbogen S.E; Thompson K.D.; Haynes A.B.; Lipsitz S.R.; Berry W.R.; Gawande A.A. An estimation of the global 395
volume of surgery: a modelling strategy based on available data. Lancet (London; England) 2008, 372, 139-144. 396
3. Holmer H.; Bekele A.; Hagander L.; Harrison E.M.; Kamali P.; Ng-Kamstra J.S.; Khan M.A.; Knowlton L.; Leather A.J.M.; Marks 397
I.H.; Meara J.G.; Shrime M.G.; Smith M.; Søreide K.; Weiser T.G.; Davies J. Evaluating the collection; comparability and findings 398
of six global surgery indicators. British Journal of Surgery 2019, 106 (2), e138–e150. 399
4. Pearse R.M.; Moreno R.P.; Bauer P.; Pelosi P.; Metnitz P.; Spies C.; Vallet B.; Vincent J.L.; Hoeft A.; Rhodes A. Mortality after 400
surgery in Europe: a 7 day cohort study. Lancet (London; England) 2012, 380, 1059-1065. 401
5. Nepogodiev D.; Martin J.; Biccard B.; Makupe A.; Bhangu A. National Institute for Health Research Global Health Research 402
Unit on Global S. Global burden of postoperative death. Lancet (London; England) 2019, 393, 401. 403
6. Collaborators GBDCoD; Global; regional; and national age-sex specific mortality for 264 causes of death. 1980-2016, a systematic 404
analysis for the Global Burden of Disease Study 2016. Lancet 2017, 390, 1151-1210. 405
7. Sankar A.; Johnson S.R.; Beattie W.S.; Tait G.; Wijeysundera D.N. Reliability of the American Society of Anesthesiologists phys- 406
ical status scale in clinical practice. British journal of anaesthesia 2014, 113, 424-432. 407
8. Sharabiani M.T.; Aylin P.; Bottle A. Systematic review of comorbidity indices for administrative data. Med Care 2012, 50, 1109- 408
1118. 409
J. Clin. Med. 2021, 10, x FOR PEER REVIEW 13 of 14

9. Yurkovich M.; Avina-Zubieta J.A.; Thomas J.; Gorenchtein M.; Lacaille D. A systematic review identifies valid comorbidity 410
indices derived from administrative health data. J Clin Epidemiol 2015, 68, 3-14. 411
10. Elixhauser A.; Steiner C.; Harris D.R.; Coffey R.M. Comorbidity measures for use with administrative data. Med Care 1998, 36, 412
8-27. 413
11. Charlson M.E.; Pompei P.; Ales K.L.; MacKenzie C.R. A new method of classifying prognostic comorbidity in longitudinal 414
studies, development and validation. J Chronic Dis 1987, 40, 373-383. 415
12. Dias-Santos D.; Ferrone C.R.; Zheng H.; Lillemoe K.D.; Fernandez-Del Castillo C. The Charlson age comorbidity index predicts 416
early mortality after surgery for pancreatic cancer. Surgery 2015, 157, 881-887. 417
13. Kork F.; Balzer F.; Krannich A.; Weiss B.; Wernecke K.D.; Spies C. Association of comorbidities with postoperative in-hospital 418
mortality, a retrospective cohort study. Medicine (Baltimore) 2015, 94, e576. 419
14. Marventano S.; Grosso G.; Mistretta A.; Bogusz-Czerniewicz M.; Ferranti R.; Nolfo F.; Giorgianni G.; Rametta S.; Drago F.; Basile 420
F.; Biondi A. Evaluation of four comorbidity indices and Charlson comorbidity index adjustment for colorectal cancer patients. 421
International journal of colorectal disease 2014, 29, 1159-1169. 422
15. Menendez M.E.; Neuhaus V.; van Dijk C.N.; Ring D. The Elixhauser comorbidity method outperforms the Charlson index in 423
predicting inpatient death after orthopaedic surgery. Clin Orthop Relat Res 2014, 47, 2878-2886. 424
16. St-Louis E.; Iqbal S.; Feldman L.S.; Sudarshan M.; Deckelbaum D.L.; Razek T.S.; Khwaja K. Using the age-adjusted Charlson 425
comorbidity index to predict outcomes in emergency general surgery. The journal of trauma and acute care surgery 2015, 78, 318- 426
323. 427
17. Suzuki H.; Hanai N.; Nishikawa D.; Fukuda Y.; Koide Y.; Kodaira T.; Tachibana H.; Tomita N.; Makita C.; Hasegawa. The 428
Charlson comorbidity index is a prognostic factor in sinonasal tract squamous cell carcinoma. Japanese journal of clinical oncology 429
2016, 46, 646-651. 430
18. Whitmore R.G.; Stephen J.H.; Vernick C.; Campbell P.G.; Yadla S.; Ghobrial G.M.; Maltenfort M.G.; Ratliff J.K.. ASA grade and 431
Charlson Comorbidity Index of spinal surgery patients, correlation with complications and societal costs. The spine journal, 432
official journal of the North American Spine Society 2014, 14, 31-38. 433
19. Frenkel W.J.; Jongerius E.J.; Mandjes-van Uitert M.J.; van Munster B.C.; de Rooij S.E. Validation of the Charlson Comorbidity 434
Index in acutely hospitalized elderly adults, a prospective cohort study. Journal of the American Geriatrics Society 2014, 62, 342- 435
346. 436
20. Habbous S.; Chu K.P.; Harland L.T.; La Delfa A.; Fadhel E.; Sun B.; Xu W.; Wong A.; Howell D.; Ringash J.; Waldron J.; O'Sulli- 437
van B.; Goldstein D.; Huang S.H.; Liu G. Validation of a one-page patient-reported Charlson comorbidity index questionnaire 438
for upper aerodigestive tract cancer patients. Oral oncology 2013, 49, 407-412. 439
21. Deyo R.A.; Cherkin D.C.; Ciol M.A. Adapting a clinical comorbidity index for use with ICD-9-CM administrative databases. J 440
Clin Epidemiol 1992, 45, 613-619. 441
22. Quan H.; Sundararajan V.; Halfon P.; Fong A.; Burnand B.; Luthi J.C.; Saunders L.D.; Beck C.A.; Feasby T.E.; Ghali W.A. Coding 442
algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data. Med Care 2005, 43, 1130-1139. 443
23. Agha R.; Abdall-Razak A.; Crossley E.; Dowlut N.; Iosifidis C.; Mathew G.; for the STROCSS Group. The STROCSS 2019 Guide- 444
line, Strengthening the Reporting of Cohort Studies in Surgery. International Journal of Surgery 2019, 72, 156-165. 445
24. McHugh M.L. Interrater reliability, the kappa statistic. Biochem Med (Zagreb) 2012, 22(3), 276-82. 446
25. Susser S.R.; McCusker J.; Belzile E. Comorbidity information in older patients at an emergency visit, self-report vs. administra- 447
tive data had poor agreement but similar predictive validity. J Clin Epidemiol 2008, 61, 511-515. 448
26. Hoffmann J.; Haastert B.; Brune M.; Kaltheuner M.; Begun A.; Chernyak N.; Icks A. How do patients with diabetes report their 449
comorbidities? Comparison with administrative data. Clin Epidemiol 2018, 10, 499-509. 450
27. Lujic S.; Simpson J.M.; Zwar N.; Hosseinzadeh H.; Jorm L. Multimorbidity in Australia, Comparing estimates derived using 451
administrative data sources and survey data. PLoS One 2017, 12,e0183817. 452
28. Chaudhry S.; Jin L.; Meltzer D. Use of a self-report-generated Charlson Comorbidity Index for predicting mortality. Med Care 453
2005, 43, 607-615. 454
29. Singh J.A. Discordance between self-report of physician diagnosis and administrative database diagnosis of arthritis and its 455
predictors. J Rheumatol 2009, 36,2000-2008. 456
30. Quail J.M.; Lix L.M.; Osman B.A.; Teare G.F. Comparing comorbidity measures for predicting mortality and hospitalization in 457
three population-based cohorts. BMC Health Serv Res 2011, 11, 146. 458
31. Frisch A.; Chandra P.; Smiley D.; Peng L.; Rizzo M.; Gatcliffe C.; Hudson M.; Mendoza J.; Johnson R.; Lin E.; Umpierrez G.E. 459
Prevalence and clinical outcome of hyperglycemia in the perioperative period in noncardiac surgery. Diabetes Care 2010, 33, 460
1783-1788. 461
32. Sato H.; Carvalho G.; Sato T.; Lattermann R.; Matsukawa T.; Schricker T. The association of preoperative glycemic control; 462
intraoperative insulin sensitivity; and outcomes after cardiac surgery. J Clin Endocrinol Metab 2010, 95, 4338-4344. 463
33. Carlo J.O.; Phisitkul P.; Phisitkul K.; Reddy S.; Amendola A. Perioperative Implications of End-stage Renal Disease in Ortho- 464
paedic Surgery. J Am Acad Orthop Surg 2015, 23, 107-118. 465
34. Meersch M.; Schmidt C.; Zarbock A. Patient with chronic renal failure undergoing surgery. Curr Opin Anaesthesiol 2016, 29, 413- 466
420. 467
J. Clin. Med. 2021, 10, x FOR PEER REVIEW 14 of 14

35. Goel N.J.; Abdullah K.G.; Choudhri O.A.; Kung D.K.; Lucas T.H.; Chen H.I. The Effect of Underlying Liver Disease on Periop- 468
erative Outcomes Following Craniotomy for Tumor, An American College of Surgeons National Quality Improvement Program 469
Analysis. World Neurosurg 2018, 115, e85-e96. 470
36. Birim O.; Maat A.P.; Kappetein A.P.; van Meerbeeck J.P.; Damhuis R.A.; Bogers A.J. Validation of the Charlson comorbidity 471
index in patients with operated primary non-small cell lung cancer. Eur J Cardiothorac Surg 2003, 23,30-34. 472
37. Soh C.H.; Ul Hassan S.W.; Sacre J.; Maier A.B. Morbidity Measures Predicting Mortality in Inpatients, A Systematic Review. J 473
Am Med Dir Assoc 2020; 21, 462-468 e7. 474
475
476
477
478
479

480