Articles

Application of the Third International Consensus Definitions

for Sepsis (Sepsis-3) Classification: a retrospective
population-based cohort study
John P Donnelly, Monika M Safford , Nathan I Shapiro, John W Baddley, Henry E Wang

Summary
Background The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3) present clinical Lancet Infect Dis 2017;
criteria for the classification of patients with sepsis. We investigated incidence and long-term outcomes of patients 17: 66170

diagnosed with these classifications, which are currently unknown. Published Online
March 3, 2017
http://dx.doi.org/10.1016/
Methods We did a retrospective analysis using data from 30239 participants from the USA who were aged at least S1473-3099(17)30117-2
45 years and enrolled in the Reasons for Geographic and Racial Differences in Stroke (REGARDS) cohort. Patients were See Comment page 573
enrolled between Jan 25, 2003, and Oct 30, 2007, and we identified hospital admissions from Feb 5, 2003, to Dec 31, 2012, Department of Emergency
and applied three classifications: infection and systemic inflammatory response syndrome (SIRS) criteria, elevated Medicine, School of Medicine
sepsis-related organ failure assessment (SOFA) score from Sepsis-3, and elevated quick SOFA (qSOFA) score from (J P Donnelly MSPH,
Sepsis-3. We estimated incidence during the study period, in-hospital mortality, and 1-year mortality. Prof H E Wang MD), Department
of Epidemiology, School of
Public Health, (J P Donnelly),
Findings Of 2593 first infection events, 1526 met SIRS criteria, 1080 met SOFA criteria, and 378 met qSOFA criteria. and Division of Infectious
Incidence was 82 events (95% CI 7887) per 1000 person-years for SIRS, 58 events (5461) per 1000 person- Diseases (Prof J W Baddley MD),
years for SOFA, and 20 events (1822) per 1000 person-years for qSOFA. In-hospital mortality was higher for Department of Medicine
(Prof M M Safford MD),
patients with an elevated qSOFA score (67 [23%] of 295 patients died) than for those with an elevated SOFA score University of Alabama at
(125 [13%] of 960 patients died) or who met SIRS criteria (128 [9%] of 1392 patients died). Mortality at 1 year after Birmingham, Birmingham, AL,
discharge was also highest for patients with an elevated qSOFA score (294 deaths [95% CI 223387] per 100 person- USA; Department of Medicine,
Weill Cornell Medical College,
years) compared with those with an elevated SOFA score (226 deaths [192266] per 100 person-years) or those who
New York, NY, USA
met SIRS criteria (147 deaths [125172] per 100 person-years). (Prof M M Safford); Harvard
Medical School, Boston, MA,
Interpretation SIRS, SOFA, and qSOFA classifications identified different incidences and mortality. Our findings USA (N I Shapiro MD); and
Department of Emergency
support the use of the SOFA and qSOFA classifications to identify patients with infection who are at elevated risk of
Medicine and Center for
poor outcomes. These classifications could be used in future epidemiological assessments and studies of patients Vascular Biology Research,
with infection. Beth Israel Deaconess Medical
Center, Boston, MA, USA
(N I Shapiro)
Funding National Institute for Nursing Research, National Center for Research Resources, and National Institute of
Neurological Disorders and Stroke. Correspondence to:
Prof Henry E Wang, Department
of Emergency Medicine, School
Introduction the ability of three classifications to identify patients with of Medicine, University of
Sepsislife-threatening organ dysfunction due to a infection who were at high risk of mortality: high risk was Alabama at Birmingham,
Birmingham, AL 35249, USA
dysregulated response to infectionis a major public defined as two or more systemic inflammatory response
hwang@uabmc.edu
health problem worldwide.14 Although mortality due to syndrome (SIRS) criteria (the previous standard for
sepsis has improved, sepsis remains a leading cause of definition of sepsis), two or more points on the sepsis-
death, with in-hospital mortality ranging from 12% to related organ failure assessment (SOFA) score, or two or
26%.37 Advances in clinical care and research for sepsis more points on the empirically derived quick SOFA
have been hampered by disparate terminology for (qSOFA) score. An elevated SOFA score was the best
and approaches to the definition of sepsis and its discriminator of patients with infection who had a high
components.8 risk of death while in intensive care. By contrast, an
The European Society of Intensive Care Medicine and elevated qSOFA score (based on respiratory rate, altered
Society for Critical Care Medicine proposed the Third mentation, and blood pressure) was best able to identify
International Consensus Definitions for Sepsis and Septic patients with a high risk of death in hospital but not in
Shock (Sepsis-3) to standardise sepsis terminology.811 intensive care. These analyses were unable to establish the
Previous iterations of the international consensus incidence of elevated SOFA and qSOFA scores or SIRS
definitions for sepsis focused on the systemic criteria, and were unable to assess how SIRS criteria and
inflammatory response to infection.12,13 For Sepsis-3, the the revised Sepsis-3 classifications (based on SOFA and
task force considered several candidate criteria with less qSOFA) performed with respect to long-term outcomes; a
focus on systemic inflammation.10 Using several large population-based cohort is the best design to formulate
hospital databases, Seymour and colleagues9 examined such estimates.

www.thelancet.com/infection Vol 17 June 2017 661

 Articles
Research in context
Evidence before this study
The Third International Consensus Definitions for Sepsis and
Septic Shock (Sepsis-3) task force released revised
classifications in February, 2016. As part of this initiative,
several manuscripts were published that described the
conclusions of the task force, assessed the validity of candidate
clinical criteria with respect to inpatient outcomes, and
presented a revised framework for evaluating sepsis
definitions. We searched PubMed up to Nov 9, 2016, for papers
related to the identification of sepsis. We searched for the
terms sepsis and validation, sepsis and definition, or (qSOFA) criteria had the highest in-hospital mortality, 28-day
sepsis and identification. We found a wide range of studies
pertinent to the definition of sepsis, but few assessments of
the revised Sepsis-3 classifications. Two single-centre studies
have examined the Sepsis-3 classifications in patients admitted
with sepsis in intensive care units in Brazil (n=957) and the
USA (n=214), with results showing increased mortality in
patients who met the revised criteria compared with those
identified by the old criteria. The study in Brazil found that the
revised criteria had greater accuracy for identification of
patients at high risk of mortality, whereas the US study found
similar prognostic value for the revised and old criteria. These
studies had limited sample sizes and lacked robust data on
individual patients. All previous validation efforts to date have
also not incorporated outcome and baseline data outside of
the inpatient setting.
Using data from one of the largest contemporary
population-based longitudinal cohorts in the USA, we
compared incidence, short-term mortality, and long-
term mortality for patients with infection who met the
revised Sepsis-3 classifications and the previously
established SIRS criteria. We postulated that these
classifications would identify different populations,
show different trends in incidence and outcomes, and be
useful for identifying patients with infection at high risk
of mortality.
Methods
Study design
We did a retrospective analysis of data collected in the
Reasons for Geographic and Racial Differences in Stroke
(REGARDS) study .14 REGARDS was a longitudinal cohort
study that included 30239 community-dwelling adults
aged at least 45 years from the 48 contiguous states of the
USA and the District of Columbia. Briefly, the sampling
frame covered roughly 95% of adults in the USA and the
study oversampled black people and people living in the
southeastern USA, with 21% of the cohort originating
from the coastal plains of North Carolina, South Carolina,
and Georgia (the so-called stroke buckle), and 35% of the
remainder of North Carolina, South Carolina, and
Georgia, as well as Tennessee, Mississippi, Alabama,
Louisiana, and Arkansas (the stroke belt).14
662 www.thelancet.com/infection Vol 17 June 2017
Added value of this study
Because previous analyses were restricted to inpatients, they could
not provide data on incidence or post-discharge outcomes for the
revised classifications. We used data from one of the largest
contemporary population-based cohorts in the USA. The revised
classifications identified distinct populations with different
incidence, and mortality at 28 days and 1 year. Compared with
patients with infection who met systemic inflammatory response
syndrome (SIRS) criteria or who had elevated sepsis-related organ
failure assessment (SOFA) scores, those who met quick SOFA
mortality, and 1-year mortality after discharge. Models that
included variables for SOFA and qSOFA classifications showed the
greatest improvements in discrimination and reclassification for
both in-hospital and 1-year mortality.
Implications of all the available evidence
The revised sepsis classifications derived by the Sepsis-3 task
force are useful for identifying patients at increased risk of poor
outcomes during hospital stays. In addition to serving as an
in-hospital screening tool, the revised classifications might also
be useful for the characterisation and identification of patients
with infection who are at increased risk of poor outcomes after
discharge. Further study is needed to establish whether
widespread use of the revised classifications would lead to
improved outcomes.
Patients were enrolled in REGARDS between Jan 25,
2003, and Oct 30, 2007, and baseline data were collected
via telephone interview and an in-person assessment.
Information collected at baseline included medical
history, functional status, health behaviours, physical
characteristics, physiological measures, blood and urine
specimens, and an inventory of medications. The
REGARDS team contacted study participants at 6-month
intervals by telephone, identifying the date, location, and
attributed reason for hospital admissions. The REGARDS
study was approved by the institutional review boards of
participating institutions, and all participants provided
verbal consent before the telephone interview and written
informed consent before the in-home study visit.
Infections and sepsis
We identified reported hospital admissions attributed to
a serious infection. Our definition of serious infection
events was admission to the emergency department or
hospital for infection, which was based on the criteria of
Angus and colleagues.1 Two trained members of the
abstraction team independently reviewed all retrieved
medical records to identify characteristics of the hospital
admission and confirm the presence of infection as a
reason for admission.1 All discordances were resolved via
adjudication by a third physician-reviewer (HEW). For
clinical criteria, we used the worst physiological and

laboratory measurements reported during the initial 28 h
of admission; we chose this timeframe because we
focused on community-acquired infection and sepsis.
Because measurement was limited to the first 28 h of
hospital admission, we did not restrict our analyses on
the basis of admission destination. Thus, we assessed
each classification in all patients admitted to hospital
with serious infection.
To classify infection events, we identified admitted
patients who met the SIRS criteria, which served as
a comparator group, and the revised Sepsis-3
classifications (SOFA and qSOFA). First, we identified
hospital admissions for an infection with at least two
SIRS criteria: abnormal body temperature (>38C or
<36C), heart rate (>90 beats per min), respiratory rate
(>20 breaths per min or partial CO2 <32 mmHg), and
white blood cell count (>12000 per L, <4000 per L, or
>10% immature bands).13 We also identified admitted
patients with at least two SOFA score points, in
accordance with established criteria.15 Last, we identified
infection events with at least two qSOFA criteria,
which consisted of altered mentation (Glasgow coma
score <14 or deemed as non-alert on the alert, voice,
pain, unresponsive scale), systolic blood pressure of
100 mmHg or lower, or respiratory rate of at least
22 breaths per min.9 Among patients with sepsis, we
also identified those who had systolic blood pressure of
100 mmHg or lower and lactate of 2 mmol/L or higher.16
Consistent with previous studies, we deemed events
with missing values for sepsis criteria factors to be
normal.9
We included reported events from Feb 5, 2003, to
Dec 31, 2012, and all-cause mortality up to the end of
available follow-up (Dec 31, 2013).
Participant characteristics
Baseline participant characteristics were established at
enrolment into REGARDS. A description of the
definitions of baseline variables is provided in the
appendix (p 2). Demographic characteristics included
age, gender, race, region of residence, and self-reported
annual household income and education (years of
school). We also collected information on smoking, body-
mass index (BMI),17 and chronic medical disorders
including chronic lung disease, diabetes, dyslipidaemia,
hypertension, myocardial infarction, stroke, and chronic
kidney disease.18
We collected data on the destination and length of stay
for events resulting in hospital admission (intensive care
unit, floor, or other). We identified infection type on the
basis of the available clinical documentation in medical
records. We also collected information about whether
participants who had events were admitted from or
discharged to a nursing, assisted care, or rehabilitation
facility. We recorded the availability of lactate measurement
in the first 28 h and whether or not each patient with
sepsis had hypotension with abnormal lactate.
www.thelancet.com/infection Vol 17 June 2017 663
Articles
Outcomes
For all infection events, we defined in-hospital mortality
on the basis of participant vital status at hospital
discharge, as recorded in the medical record. For analyses
of 28-day mortality (relative to the date of hospital
admission or emergency department visit) and 1-year
mortality (relative to discharge), we calculated all-cause
mortality on the basis of information collected as part of
REGARDS. If the participant died, researchers from the
REGARDS study reviewed death certificates, autopsy
reports, and medical records, and interviewed proxies (ie,
family members or other knowledgeable contacts) to
ascertain the circumstances of the participants death.
Statistical analysis
Our classifications of hospital admissions for infection
(by SIRS, SOFA, and qSOFA) were not mutually
exclusive, so to compare characteristics, we reported
means and SDs for continuous variables and percentages
with 95% CIs estimated by the logit transformation
method for categorical variables. We calculated the
incidence of infection and events meeting classifications
per 1000 person-years during the study period, censoring
at loss to follow-up, death, or on Dec 31, 2012. To compare See Online for appendix
30 239 REGARDS participants
547 missing follow-up
29 692 included in analytic cohort
3413 adjudicated serious infection events
2593 first events
Any sepsis Non-sepsis infection
2260 total events 1153 total events
1673 first events 1031 first events
Met SIRS criteria High SOFA score Met qSOFA criteria
1845 total events 1332 total events 423 total events
1526 first events 1080 first events 378 first events
Figure 1: Study population
Dotted lines show that events can belong to multiple groups. Non-sepsis infections were defined as infections not
meeting any criteria for sepsis and were mutually exclusive from any sepsis events. Patients who met SIRS criteria
had at least two criteria (based on respiratory rate [breaths per min], white blood cell count [10 cells per L],
bands [%], heart rate [beats per min], temperature [C], and arterial CO2 [mm Hg]); high SOFA score was defined as
an infection with at least two SOFA points (based on PaO2/FiO2 ratio, altered mentation, mean arterial pressure,
vasopressor administered, serum creatinine [mg/dL], bilirubin [mg/dL], and platelet count [10 cells per L]); and
meeting qSOFA criteria was defined as an infection and meeting at least two qSOFA criteria (based on respiratory
rate [breaths per min], altered mentation, and systolic blood pressure [mm Hg]). First events represent the
first event for a given participant (each participant is represented once). Total events includes recurrent events
(each participant can be included multiple times). REGARDS=Reasons for Geographic and Racial Differences in
Stroke study. SIRS=systemic inflammatory response syndrome. SOFA=sepsis-related organ failure assessment.
qSOFA=quick sepsis-related organ failure assessment.
 Articles

No infection (n=27099) Non-sepsis infection Classification

(n=1031)
SIRS (n=1526) SOFA (n=1080) qSOFA (n=378)
Age, years 646 (94) 667 (90) 681 (93) 697 (91) 702 (91)
Gender
Men 446% (440452) 402% (372432) 509% (484534) 592% (562621) 569% (518618)
Women 554% (548560) 598% (568628) 491% (466516) 408% (379438) 431% (382482)
Race
White 582% (576587) 678% (649706) 665% (641688) 640% (611668) 669% (620715)
Black 418% (413424) 322% (294351) 335% (312359) 360% (332389) 331% (285380)
Education*
Less than high school 123% (119127) 127% (108149) 163% (145182) 166% (145190) 170% (135211)
High school 258% (253263) 256% (231284) 273% (251296) 284% (258312) 260% (218307)
Some college 266% (261271) 297% (270326) 286% (264309) 272% (246299) 326% (281375)
College or more 354% (348359) 319% (292349) 278% (256301) 278% (252306) 244% (203290)
Income, US$
<$20000 175% (171180) 228% (203255) 239% (218261) 245% (221272) 249% (208295)
$2000034000 239% (234244) 243% (218271) 286% (264309) 281% (255308) 312% (267361)
$3500074000 299% (293304) 279% (253308) 266% (244289) 278% (252305) 249% (208295)
$75000 163% (158167) 127% (108149) 103% (89119) 89% (73107) 90% (65123)
Not available 125% (121128) 122% (104144) 106% (92123) 107% (90127) 101% (74135)
Region
Stroke buckle 209% (204213) 226% (201253) 210% (191232) 213% (190238) 193% (156236)
Stroke belt 344% (338350) 362% (333392) 379% (355403) 372% (344402) 405% (356455)
Not belt or buckle 447% (441453) 412% (382443) 411% (386436) 415% (386445) 402% (354453)
Smoking
Current 144% (140148) 129% (110151) 173% (155193) 162% (141185) 199% (162243)
Past 397% (391403) 449% (419480) 466% (441491) 493% (463523) 493% (443544)
Never 459% (453465) 422% (392452) 361% (337386) 345% (317374) 308% (263356)
BMI class
Normal or underweight 249% (244254) 228% (203254) 233% (212255) 212% (189238) 222% (183268)
Overweight 371% (365377) 349% (321379) 339% (316363) 352% (324381) 371% (323422)
Obese 380% (374386) 423% (393454) 428% (403453) 436% (406466) 407% (357458)
Chronic disorders
Lung disease 85% (8288) 143% (123165) 187% (169208) 174% (153198) 188% (151231)
Chronic kidney disease 102% (99106) 127% (108149) 201% (181221) 315% (288343) 241% (200287)
Stroke 60% (5863) 92% (76111) 104% (89120) 126% (107147) 135% (104173)
Myocardial infarction 119% (115123) 185% (163210) 198% (179219) 244% (219270) 246% (205292)
Hypertension 583% (577589) 632% (602661) 683% (660706) 745% (718771) 712% (664755)
Dyslipidaemia 566% (560572) 614% (584643) 630% (606654) 673% (645701) 638% (588685)
Diabetes 239% (234244) 322% (294351) 345% (321369) 418% (388447) 378% (331429)

Data are mean (SD) or % (95% CI). Only first events were included (n=29692). Sepsis classifications were not mutually exclusive and events can belong to multiple groups.
Non-sepsis infections were defined as infections not meeting any criteria for sepsis (mutually exclusive from sepsis criteria). CIs were estimated with the logit transformation
method. The stroke buckle consists of North Carolina, South Carolina, and Georgia, and the stroke belt consists of North Carolina, South Carolina, Georgia, Tennessee,
Mississippi, Alabama, Louisiana, and Arkansas. SIRS=systemic inflammatory response syndrome. SOFA=sepsis-related organ failure assessment. qSOFA=quick sepsis-related
organ failure assessment. BMI=body-mass index. *23 missing. 114 missing. 214 missing. Chronic kidney disease was defined on the basis of creatinine values and the
Chronic Kidney Disease Epidemiology Collaboration (CKD-EPI) equation.

Table 1: Baseline participant characteristics by sepsis classification

temporal trends in 2-year intervals, we estimated
standardised incidence ratios with 200304 as the
reference, using direct standardisation to the overall
cohort age distribution to account for age differences.
To assess infection and sepsis outcomes, we included
only the first identified infection for each participant. We
calculated in-hospital mortality and 28-day mortality
relative to the date of hospital admission or emergency
department visit, as well as all-cause 1-year mortality
relative to the discharge date. We constructed Kaplan-
Meier curves to assess differences in time to death
between classifications. For outcomes at 28 days or 1 year,

664 www.thelancet.com/infection Vol 17 June 2017

 Articles

we estimated mortality rates per 100 person-years during

A
the study period, censoring at loss to follow-up. To 100 All infections 20
establish whether mortality differences could be SIRS
SOFA
explained by differences in SOFA criteria, we also did a qSOFA 15
sensitivity analysis of events that met the SIRS and 80
qSOFA criteria, with stratification by SOFA score. 10
We also assessed the ability of the classifications to

Proportion with event (%)

identify patients at high risk of mortality. We constructed
60 5
logistic regression models for in-hospital mortality adjusted
for demographic characteristics and comorbidities, adding
0
a dichotomous variable for each classification individually. 0 10
40
We reported odds ratios (ORs) with 95% CIs, discrimination
in the form of areas-under-the-curve (AUCs), the change in
AUC, net reclassification improvement (NRI), and
integrated discrimination improvement (IDI) indices.19,20 20

We estimated changes in AUC, NRIs, and IDIs with

95% CIs using bootstrap resampling with 1000 replications.
We assessed model calibration using Hosmer-Lemeshow 0
goodness of fit tests.21 For 1-year mortality, we fitted Cox Incidence per 0 2 4 6 8 10
1000 person- Time since enrolment (years)
proportional hazards models with time to mortality as the years (95% CI)
outcome. We reported hazards ratios (HRs), Harrells All infections 126 (117136) 138 (128149) 154 (143167) 166 (151183) 154 (125190)
C indices (transformed Dxy), and extensions of IDI and SIRS 75 (6882) 78 (7186) 89 (8098) 94 (83106) 97 (75125)
SOFA 44 (3949) 55 (4962) 66 (5874) 75 (6586) 83 (63110)
NRI to survival data as described by Pencina and colleagues19 qSOFA 14 (1117) 19 (1623) 25 (2130) 25 (1931) 30 (1948)
and Uno and colleagues.2224 We estimated C indices using Number at risk
bootstrap resampling with 150 replicates and IDIs and (start of time
period)
NRIs using 200 resampling perturbations. We assessed All infections 29 692 27 290 24 024 17 604 8265
model calibration with Groennesby and Borgan score SIRS 29 692 27 561 24 533 18 162 8641
tests.25 We verified the proportional hazards assumption SOFA 29 692 27 726 24 787 18 453 8806
qSOFA 29 692 27 873 25 032 18 711 8975
with Schoenfeld residuals and interactions with the
logarithm of follow-up time. B
20
We used Stata 13.1 and R 3.1.1 for all analyses.

Role of the funding source

Age-adjusted incidence (per 1000 person-years)

16
The funder of the study had no role in study design, data
collection, data analysis, data interpretation, or writing of
the report. The corresponding author had full access to
12
all the data in the study and had final responsibility for
the decision to submit for publication.
8
Results
Of 30239 participants enrolled in REGARDS between
2003 and 2007, we included 29692 individuals with follow- 4
up information available. In this population, we identified
3413 serious infections (2593 first events) between 2003
and 2012. With the exception of Glasgow coma scale score, 0
200304 200506 200708 200910 201112
measures used in the sepsis classifications were available Standardised
Time period
for a large proportion of patients with an infection incidence ratio
(95% CI)
All infections 1 (Ref) 114 (093138) 124 (102150) 153 (126184) 145 (119176)
SIRS 1 (Ref) 103 (079132) 118 (092150) 143 (112183) 132 (102170)
Figure 2: Incidence of events by sepsis classification SOFA 1 (Ref) 135 (093192) 155 (109218) 213 (150299) 204 (142288)
(A) Kaplan-Meier failure curves for time to event after study enrolment. (B) Age- qSOFA 1 (Ref) 198 (085437) 251 (109544) 375 (164806) 299 (129653)
standardised event incidence by time period. Includes first events only from the Number at risk
total cohort of 29692 individuals. All infections refers to sepsis events as well as (participants
non-sepsis events. Sepsis classifications were not mutually exclusive and events contributing
can belong to multiple groups. Incidence is reported as per 1000 person-years of person-time)
All infections 15 173 25 643 27 467 24 059 20 245
follow-up. Error bars show 95% CIs. Age-adjusted rates were for each time period,
SIRS 15 173 25 693 27 717 24 515 20 865
directly standardised to the overall cohort age distribution. SIRS=systemic SOFA 15 173 25 734 27 849 24 752 21 171
inflammatory response syndrome. SOFA=sepsis-related organ failure assessment. qSOFA 15 173 25 766 27 976 24 980 21 443
qSOFA=quick sepsis-related organ failure assessment.

www.thelancet.com/infection Vol 17 June 2017 665

 Articles

In-hospital mortality 28-day mortality (time from hospital admission or 1-year mortality (time from discharge)
emergency department visit)*
Total Deaths Deaths Total Mortality per 100 Deaths Total exposure Mortality per 100
infection exposure person-years (95% CI) (person-years) person-years (95% CI)
events (person-
years)
All Infections 2593 163 (6%) 167 (7%) 184 907 (7791056) 238 (10%) 2058 116 (102131)
Non-sepsis infection 916 19 (2%) 21 (2%) 68 310 (202476) 38 (4%) 796 48 (3566)
Met SIRS criteria 1392 128 (9%) 128 (10%) 96 1332 (11201584) 154 (13%) 1050 147 (125172)
High SOFA score 960 125 (13%) 122 (13%) 64 1910 (16002281) 147 (18%) 650 226 (192266)
Met qSOFA criteria 295 67 (23%) 62 (23%) 17 3504 (27324495) 51 (23%) 173 294 (223387)

Only first events were included. All infections included those defined as sepsis events as well as non-sepsis infection. Non-sepsis infections were defined as infections not meeting any criteria for sepsis and were
mutually exclusive from any sepsis events. Sepsis classifications were not mutually exclusive and events could belong to multiple groups. Mortality per 100 person years of follow-up is relative to the date of
encounter or discharge. SIRS=systemic inflammatory response syndrome. SOFA=sepsis-related organ failure assessment. qSOFA=quick sepsis-related organ failure assessment. *Excludes patients with no
follow-up after the date of encounter and patients who died on the day of the encounter, leaving 2519 events. Excludes patients who died in hospital or on the day of discharge and those with no follow-up
after discharge, leaving 2386 events.

Table 2: Mortality by classification

(appendix p 5). Most infections were classified as meeting
SIRS criteria (1845 [54%] of 3413), with fewer having
elevated SOFA scores (1332 [39%]), non-sepsis infection
(1153 [34%]), and meeting qSOFA criteria (423 [12%];
figure 1). Of the 2593 first infection events, 678 (26%) both
met SIRS criteria and had elevated SOFA scores, 238 (9%)
both had high SOFA score and met qSOFA criteria, and
282 (11%) met SIRS and qSOFA criteria (appendix p 7).
Internal review of 1349 hospital records suggested excellent
inter-rater agreement for the presence of infection
(=092). Table 1 shows demographic characteristics of
patients with sepsis according to each classification.
Incidence was 82 events (95% CI 7887) per
1000 person-years for SIRS, 58 events (5461) per
1000 person-years for SOFA, and 20 events (1822) per
1000 person-years for qSOFA. Age-adjusted incidence of
having elevated SOFA score or meeting qSOFA criteria
increased over time (figure 2).
Compared with patients who met SIRS criteria, more
patients with an elevated SOFA score or who met qSOFA
criteria were admitted to the ICU, had sepsis recorded as
an infection type, and were admitted from or discharged
to a nursing facility (appendix p 9). More patients who
met qSOFA criteria had lactate measurements available
and had hypotension with an abnormal lactate than did
patients meeting other classifications (appendix p 9). In-
hospital mortality was highest for patients who met
qSOFA criteria and lowest for those meeting SIRS
criteria, but mortality differences were restricted to
events with at least two SOFA points (table 2; appendix
p 11). Mortality at 28 days and 1 year were also higher for
patients with high SOFA scores and those who met
qSOFA criteria than for those who met SIRS criteria
(figure 3), including after adjustment for demographics
and comorbidities (figure 4).
For in-hospital mortality, AUCs with dichotomous
classification variables included in logistic regression
models were substantially higher for patients with
elevated SOFA scores and those who met qSOFA criteria
than for those who met SIRS criteria or the baseline
model (table 3; appendix p 13). Relative to the baseline
model, SOFA provided the largest change in AUC and
NRI but qSOFA had a larger IDI. In analyses of mortality
at 1 year after discharge, the NRI relative to the baseline
model for elevated SOFA score was larger than for that
meeting SIRS or qSOFA criteria. All models for in-
hospital mortality and 1-year mortality showed good
calibration (goodness of fit p>005; appendix pp 15, 17).
Discussion
Our results showed different overlapping groups, with
elevated SOFA scores and qSOFA criteria occurring
disproportionately in older participants with several
comorbidities. Our analyses of mortality during hospital
stay and at 1 year after discharge support the use of SOFA
and qSOFA as screening tools, showing that these
classifications can identify patients with infection who
are at high risk of poor outcomes.
Our analyses extend studies of administrative
databases.9 Seymour and colleagues9 retrospectively
analysed patients presenting to a geographically diverse
pool of health systems and established the associations
between the SIRS, SOFA, and qSOFA classifications and
in-hospital mortality. Our results support those of
previous work: we noted similar discrimination,
calibration, and measures of association for models
including classification variables as well as demographics
and comorbidities. However, we also showed that
classifications based on SOFA and qSOFA criteria were
able to identify patients at high risk of mortality within
1 year after discharge. Taken together, these findings
show the robustness of the proposed classifications, given
that we arrived at similar conclusions despite using an
independent sample with a different study design, a
different method of event detection, and examination of
post-discharge outcomes.

666 www.thelancet.com/infection Vol 17 June 2017

 Articles

Our results also underscore a well recognised challenge

in the characterisation of sepsisthat different A
100 qSOFA 30
classification methods will provide different results. SOFA
Several previous studies have assessed methods for SIRS 25
All infections
identification of patients with high-risk infection in 20
80
various settings.4,6,2631 Iwashyna and colleagues26 used
15
previous international consensus criteria to assess the
validity of claims-based algorithms to identify sepsis 10
60
relative to events defined clinically, finding high

Mortality (%)
5
specificity and positive predictive values with modest
0
sensitivity. Gaieski and colleagues6 examined several 0 28
methods for identifying sepsis in administrative data and 40

reported noticeable differences in the total burden of

sepsis and temporal trends in incidence depending on
the algorithm. Our results are broadly consistent with 20
these findings, showing large differences in populations
of patients, incidence, and outcomes depending on the
classification used. However, our study differs in that we 0
applied the Sepsis-3 classifications to a large population- 0 7 14 21 28
Mortality per Time since event (days)
based cohort. Our results emphasise the variations 100 person-
years (95% CI)
associated with different methods of infection qSOFA 7309 (527210133) 3390 (20445624) 1648 (7863458) 977 (3672602)
classification and show how these classifications might SOFA 3294 (25354281) 2053 (14602888) 1089 (6771752) 1053 (6451719)
be used in practice, systems planning, and research. SIRS 2425 (18873116) 1449 (10402018) 847 (5461312) 518 (294913)
All infections 1640 (13142047) 908 (6711229) 593 (407865) 447 (288692)
These criteria have been proposed for use as in-hospital Number at risk
screening tools to identify patients at high risk of (start of time
mortality. Additionally, they could be used to assess the period)
qSOFA 275 241 225 218
total burden on health systems and potentially to SOFA 913 862 825 807
categorise patients with infection in future studies. SIRS 1340 1285 1248 1221
All infections 2519 2443 2397 2357
However, SOFA and qSOFA do not define sepsis, but
instead might serve as indicators of an increased risk of B
100 30
death among patients with infection.
Our study builds on previous work in that we assessed 25
several approaches for applying sepsis classifications that 20
80
do not rely on administrative claims.29 For example, the
15
incidence of having high SOFA scores and meeting
qSOFA criteria increased more over time than did the 10
incidence of patients meeting SIRS criteria or any 60
Mortality (%)

5
infection. Because of the manner in which events were
ascertained in REGARDS, it is difficult to draw firm 0
0 360
conclusions about temporal trends in our study because 40
observed patterns could be affected by event detection.
However, Rhee and colleagues29,32 reported similar
patterns of increases in incidence during a 10-year period 20
for identification methods that depended on coding
practices and steadily increasing use of acute organ
0
0 90 180 270 360
Figure 3: Mortality after event by sepsis classification Mortality per Time since discharge (days)
(A) 28-day mortality relative to the date of hospital admission or emergency 100 person-
department visit. (B) 1-year mortality relative to discharge date. Includes first years (95% CI)
qSOFA 623 (435891) 206 (107397) 147 (66328) 146 (66326)
infection events only. All infections refers to sepsis events as well as non-sepsis
SOFA 464 (375573) 152 (103225) 137 (90211) 107 (65174)
events. Sepsis-related classifications were not mutually exclusive and events can
SIRS 304 (247376) 110 (76158) 76 (49119) 76 (48119)
belong to multiple groups. 28-day mortality includes in-hospital mortality and All infections 230 (193273) 87 (65116) 67 (4895) 65 (4692)
excludes events with no follow-up after the date of encounter and deaths
Number at risk
occurring on the first day of admission (zero follow-up time). 1-year mortality (start of time
excludes in-hospital mortality and events with no follow-up after the date of period)
discharge and deaths occurring on the first day of discharge (zero follow-up qSOFA 221 183 172 160
time). Mortality is per 100 person-years. SIRS=systemic inflammatory response SOFA 812 696 642 595
syndrome. SOFA=sepsis-related organ failure assessment. qSOFA=quick sepsis- SIRS 1236 1109 1041 986
related organ failure assessment. All infections 2386 2164 2050 1937

www.thelancet.com/infection Vol 17 June 2017 667

 Articles

10
Measure of association

OR 620
OR 571 (438878)
(8861843)
HR 169
(130221)
OR 326
(222479) HR 243 HR 259
(184321) (190354)

1
SIRS SOFA qSOFA SIRS SOFA qSOFA

In-hospital mortality 1 year mortality

Figure 4: Association between sepsis classification and outcome

Includes first infection events only (n=2593). Classifications were not mutually exclusive and events can belong to multiple groups. All models include the events not
meeting a given criterion as the reference group and are adjusted for age, gender, race, hypertension, dyslipidaemias, chronic kidney disease, history of stroke, history
of myocardial infarction, and diabetes. Error bars represent 95% CI limits. Numbers in parentheses are 95% CI. Analysis of 1-year mortality included 2386 first
infection events after exclusion of events with in-hospital mortality, no follow-up after the date of the encounter, and deaths occurring on the first day of discharge
(zero follow-up time). SIRS=systemic inflammatory response syndrome. SOFA=sepsis-related organ failure assessment. qSOFA=quick sepsis-related organ failure
assessment. OR=odds ratio. HR=hazard ratio.

In-hospital mortality 1-year mortality*

AUC Change in AUC (95% CI) Net reclassification Integrated discrimination C index Net reclassification Integrated discrimination
improvement (95% CI) improvement (95% CI) improvement (95% CI) improvement (95% CI)
Baseline model 0664 Reference Reference Reference 0738 Reference Reference
Baseline plus SIRS criteria 0721 0057 (00290085) 0530 (03960664) 0017 (00060028) 0748 0139 (00670199) 0007 (00020019)
Baseline plus high SOFA score 0765 0101 (00670136) 0846 (07140979) 0036 (00200053) 0763 0314 (02450380) 0023 (00090040)
Baseline plus qSOFA criteria 0759 0095 (00580132) 0634 (04800789) 0056 (00260085) 0749 0130 (00700182) 0022 (00080041)

Only first events were included (n=2593). Classifications were not mutually exclusive and events could belong to multiple groups. C index represents Dxy transformed to Harrells C for survival data. SIRS=systemic
inflammatory response syndrome. SOFA=sepsis-related organ failure assessment. qSOFA=quick sepsis-related organ failure assessment. AUC=area under the curve. *Excludes events with in-hospital mortality,
no follow-up after the date of discharge, and deaths occurring on the first day of discharge (zero follow-up time), leaving 2386 events. Estimated with bootstrap sampling with 1000 replications. Extension of
net reclassification improvement and integrated discrimination improvement to survival data19 estimated with 200 resampling perturbations. Includes age, gender, race, hypertension, dyslipidaemia, chronic
kidney disease, history of stroke, history of myocardial infarction, and diabetes.

Table 3: Relative discrimination and reclassification improvement by classification model

dysfunction codes with lower clinical thresholds. Further
research is needed to establish how estimates could be
affected by vital sign and laboratory value measurement.
The main strengths of our study were its large sample
size and the detailed health information among enrolled
participants, complete mortality follow-up for about
10 years, and systematic abstraction with adjudication for
all hospital admissions for infection. Despite these
strengths, our findings must be interpreted in light of
several limitations. We were unable to identify infections
that occurred during the course of hospital stays and
instead focused on events for which infection was a main
reason for admission. Given our focus on community-
acquired infection, we also included only vital sign and
laboratory measurements within the first 28 h of admission.
Because participants with a first hospital admission for
infection might have a different risk profile after the event,
we restricted outcome analyses to the first infection event
on record. Since hospital admissions for infection were
reported by the participants, we could have missed some
events. However, any bias from these missed admissions
would act systematically, and we used standard methods
of event ascertainment, chart review, and adjudication.
Reporting of laboratory values and vital sign measurement
could vary by hospital. However, the REGARDS cohort
contains participants from across the USA and includes a
diverse sample of hospitals, so we would expect the effect
of this limitation to be small. Because of the detection
method used and the population studied, our results
might not be readily generalisable to other settings.
Our results do not address the feasibility of
incorporating the Sepsis-3 classifications into routine
clinical care or the potential effect on patient outcomes.
Rather, we show that these classifications might be useful
to characterise and identify patients with high-risk
infection using routine laboratory and vital sign
measurements, with fewer components than the SIRS
criteria. On the basis of the event rates we recorded, in

668 www.thelancet.com/infection Vol 17 June 2017


American black and white adults aged 45 years or older,
we would expect more than 400000 patients admitted to
hospital with infection to have high SOFA scores and
100000 to meet qSOFA criteria each year in a 10-year
period. These figures would correspond to at least
50000 deaths in patients with high SOFA scores and
30000 deaths in patients meeting qSOFA criteria each
year. These classifications might also help to identify
high-risk subgroups in future clinical trials of sepsis.
Future studies must assess the feasibility and usefulness
of the Sepsis-3 classifications in these contexts.
Contributors
HEW and JPD conceived the study. HEW and MMS organised and
oversaw the data collection. HEW and JPD conducted the analysis, and
all authors contributed to the review of the results. JPD produced an
initial draft of the manuscript and all authors contributed to its editorial
review and revision. JPD and HEW assume responsibility for the work
as a whole.
Declaration of interests
JPD reports a predoctoral training fellowship that included a stipend and
tuition from the Agency for Healthcare Research and Quality. MMS
reports an investigator-initiated research grant from Amgen and grants
from the US National Institutes of Health (NIH) and Patient-Centered
Outcomes Research Institute (PCORI). NIS reports research funding
from Siemens, Rapid Pathogen Screening, ThermoFisher, and the NIH,
in addition to consulting for Cheetah Medical and Cyon. All other
authors declare no competing interests.
Acknowledgments
This study was supported by award R01-NR012726 from the National
Institute for Nursing Research, UL1-RR025777 from the National Center
for Research Resources, and grants from the Center for Clinical and
Translational Science and the Lister Hill Center for Health Policy of the
University of Alabama at Birmingham. The parent REGARDS study was
supported by cooperative agreement U01-NS041588 from the National
Institute of Neurological Disorders and Stroke, National Institutes of
Health. JPD was supported by award T32-HS013852 from the Agency for
Healthcare Research and Quality. MMS was supported by award
K24-HL111154 from the National Heart, Lung and Blood Institute. The
content is solely the responsibility of the authors and does not
necessarily represent the official views of the funding agencies.
Representatives of the funding agencies have been involved in the
review of the manuscript but not directly involved in the collection,
management, analysis or interpretation of the data. We thank the other
investigators, the staff, and the participants of the REGARDS study for
their valuable contributions. A full list of participating REGARDS
investigators and institutions can be found online.
References
1 Angus DC, Linde-Zwirble WT, Lidicker J, Clermont G, Carcillo J,
Pinsky MR. Epidemiology of severe sepsis in the United States:
analysis of incidence, outcome, and associated costs of care.
Crit Care Med 2001; 29: 130310.
2 Wang HE, Shapiro NI, Angus DC, Yealy DM. National estimates of
severe sepsis in United States emergency departments.
Crit Care Med 2007; 35: 192836.
3 Kaukonen KM, Bailey M, Suzuki S, Pilcher D, Bellomo R.
Mortality related to severe sepsis and septic shock among critically
ill patients in Australia and New Zealand, 20002012. JAMA 2014;
311: 130816.
4 Martin GS, Mannino DM, Eaton S, Moss M. The epidemiology of
sepsis in the United States from 1979 through 2000. N Engl J Med
2003; 348: 154654.
5 Iwashyna TJ, Angus DC. Declining case fatality rates for severe
sepsis: good data bring good news with ambiguous implications.
JAMA 2014; 311: 12957.
6 Gaieski DF, Edwards JM, Kallan MJ, Carr BG. Benchmarking the
incidence and mortality of severe sepsis in the United States.
Crit Care Med 2013; 41: 116774.
www.thelancet.com/infection Vol 17 June 2017 669
Articles
7 Dellinger RP, Carlet JM, Masur H, et al. Surviving Sepsis Campaign
guidelines for management of severe sepsis and septic shock.
Intensive Care Med 2004; 30: 53655.
8 Singer M, Deutschman CS, Seymour CW, et al. The third
international consensus definitions for sepsis and septic shock
(sepsis-3). JAMA 2016; 315: 80110.
9 Seymour CW, Liu VX, Iwashyna TJ, et al. Assessment of clinical
criteria for sepsis: for the third international consensus
definitions for sepsis and septic shock (sepsis-3). JAMA 2016;
315: 76274.
10 Angus DC, Seymour CW, Coopersmith CM, et al. A framework for
the development and interpretation of different sepsis definitions
and clinical criteria. Crit Care Med 2016; 44: e11321.
11 Seymour CW, Coopersmith CM, Deutschman CS, et al.
Application of a framework to assess the usefulness of alternative
sepsis criteria. Crit Care Med 2016; 44: e12230.
12 American College of Chest Physicians/Society of Critical Care
Medicine Consensus Conference: definitions for sepsis and organ
failure and guidelines for the use of innovative therapies in sepsis.
Crit Care Med 1992; 20: 86474.
13 Levy MM, Fink MP, Marshall JC, et al. 2001 SCCM/ESICM/ACCP/
ATS/SIS International Sepsis Definitions Conference.
Intensive Care Med 2003; 29: 53038.
14 Howard VJ, Cushman M, Pulley L, et al. The reasons for geographic
and racial differences in stroke study: objectives and design.
Neuroepidemiology 2005; 25: 13543.
15 Vincent JL, Moreno R, Takala J, et al. The SOFA (Sepsis-related
Organ Failure Assessment) score to describe organ dysfunction/
failure. On behalf of the Working Group on Sepsis-Related
Problems of the European Society of Intensive Care Medicine.
Intensive Care Med 1996; 22: 70710.
16 Shankar-Hari M, Phillips GS, Levy ML, et al. Developing a new
definition and assessing new clinical criteria for septic shock: for
the Third International Consensus Definitions for Sepsis and Septic
Shock (Sepsis-3). JAMA 2016; 315: 77587.
17 Janssen I, Katzmarzyk PT, Ross R. Body mass index, waist
circumference, and health risk: evidence in support of current
National Institutes of Health guidelines. Arch Intern Med 2002;
162: 207479.
18 James MT, Hemmelgarn BR, Wiebe N, et al. Glomerular filtration
rate, proteinuria, and the incidence and consequences of acute
kidney injury: a cohort study. Lancet 2010; 376: 2096103.
19 Pencina MJ, DAgostino RB Sr, Steyerberg EW. Extensions of net
reclassification improvement calculations to measure usefulness of
new biomarkers. Stat Med 2011; 30: 1121.
20 Gu W, Pepe M. Measures to summarize and compare the predictive
capacity of markers. Int J Biostat 2009; 5: article 27.
21 Lemeshow S, Hosmer DW Jr. A review of goodness of fit statistics
for use in the development of logistic regression models.
Am J Epidemiol 1982; 115: 92106.
22 Harrell FE. Regression modeling strategies: with applications to For the REGARDS investigators
linear models, logistic regression, and survival analysis. New York, see http://www.regardssepsis.org
NY: Springer, 2001.
23 Uno H, Tian L, Cai T, Kohane IS, Wei LJ. A unified inference
procedure for a class of measures to assess improvement in risk
prediction systems with survival data. Stat Med 2013; 32: 243042.
24 Chambless LE, Cummiskey CP, Cui G. Several methods to assess
improvement in risk prediction models: extension to survival
analysis. Stat Med 2011; 30: 2238.
25 Hosmer DW, Lemeshow S, May S. Applied survival analysis:
regression modeling of time-to-event data, 2nd edn. Hoboken, NJ:
Wiley-Interscience, 2008.
26 Iwashyna TJ, Odden A, Rohde J, et al. Identifying patients with
severe sepsis using administrative claims: patient-level validation of
the Angus implementation of the international consensus
conference definition of severe sepsis. Med Care 2014; 52: e3943.
27 Whittaker SA, Mikkelsen ME, Gaieski DF, Koshy S, Kean C,
Fuchs BD. Severe sepsis cohorts derived from claims-based
strategies appear to be biased toward a more severely ill patient
population. Crit Care Med 2013; 41: 94553.
28 Jolley RJ, Sawka KJ, Yergens DW, Quan H, Jette N, Doig CJ.
Validity of administrative data in recording sepsis: a systematic
review. Crit Care 2015; 19: 139.
 Articles
29 Rhee C, Murphy MV, Li L, et al. Comparison of trends in sepsis
incidence and coding using administrative claims versus objective
clinical data. Clin Infect Dis 2015; 60: 8895.
30 April MD, Aguirre J, Tannenbaum LI, et al. Sepsis clinical criteria
in emergency department patients admitted to an intensive care
unit: an external validation study of quick sequential organ failure
assessment. J Emerg Med 2016; published online Nov 4.
DOI:10.1016/j.jemermed.2016.10.01.
670 www.thelancet.com/infection Vol 17 June 2017
31 Besen BA, Romano TG, Nassar AP, Jr., et al. Sepsis-3 definitions
predict ICU mortality in a low-middle-income country.
Ann Intensive Care 2016; 6: 107.
32 Rhee C, Murphy MV, Li L, et al. Improving documentation and
coding for acute organ dysfunction biases estimates of changing
sepsis severity and burden: a retrospective study. Crit Care 2015;
19: 338.
 Reproduced with permission of copyright owner.
Further reproduction prohibited without permission.