Sustained Improvement in Tracheal Intubation

Safety Across a 15-Center Quality-Improvement

Collaborative: An Interventional Study From
the National Emergency Airway Registry for
Children Investigators*
Akira Nishisaki, MD, MSCE1
OBJECTIVES: To evaluate the effect of a tracheal intubation safety bundle Anthony Lee, MD2
on adverse tracheal intubation-associated events across 15 PICUs. Simon Li, MD, MPH3
DESIGN: Multicenter time-series study. Ronald C. Sanders Jr, MD, MS4
Calvin A. Brown III, MD5
SETTING: PICUs in the United States. Kyle J. Rehder, MD, CPPS, FCCM,
FCCP6
PATIENTS: All patients received tracheal intubations in ICUs.
Natalie Napolitano, MPH,
INTERVENTIONS: We implemented a tracheal intubation safety bundle RRT-NPS, FAARC7
as a quality-improvement intervention that includes: 1) quarterly site Vicki L. Montgomery, MD8
benchmark performance report and 2) airway safety checklists (prepro- Michelle Adu-Darko, MD, FAAP9
cedure risk factor, approach, and role planning, preprocedure bedside G. Kris Bysani, MD, FAAP, FCCM10
“time-out,” and immediate postprocedure debriefing). We define each Ilana Harwayne-Gidansky, MD,
FAAP, CHSE11
quality-improvement phase as baseline (–24 to –12 mo before checklist Joy D. Howell, MD, FAAP, FCCM12
implementation), benchmark performance reporting only (–12 to 0 mo be- Sholeen Nett, MD, PhD13
fore checklist implementation), implementation (checklist implementation Alberto Orioles, MD, FAAP14
start to time achieving > 80% bundle adherence), early bundle adherence Matthew Pinto, MD3
(0–12 mo), and sustained (late) bundle adherence (12–24 mo). Bundle Asha Shenoi, MD, Dch, FAAP15
adherence was defined a priori as greater than 80% of checklist use for David Tellez, MD16
tracheal intubations for 3 consecutive months. Serena P. Kelly, MS, CPNP-AC,
FNP-BC, CCRN, CPEN17
MEASUREMENTS AND MAIN RESULTS: The primary outcome was
Melinda Register, MS, RRT, CPFT,
the adverse tracheal intubation-associated event, and secondary outcomes RRT-NPS18
included severe tracheal intubation-associated events, multiple tracheal in- Keiko Tarquinio, MD, FAAP19
tubation attempts, and hypoxemia less than 80%. Dennis Simon, MD, PhD20
From January 2013 to December 2015, out of 19 participating PICUs, 15 Conrad Krawiec, MD21
ICUs (79%) achieved bundle adherence. Among the 15 ICUs, the adverse Justine Shults, PhD22
tracheal intubation-associated event rates were baseline phase: 217/1,241 Vinay Nadkarni, MD, MS1
(17.5%), benchmark reporting only phase: 257/1,750 (14.7%), early 0–12 for National Emergency
month complete bundle compliance phase: 247/1,591 (15.5%), and Airway Registry for Children
(NEAR4KIDS) and Pediatric
late 12–24 month complete bundle compliance phase: 137/1,002 (13.7%). Acute Lung Injury and Sepsis
After adjusting for patient characteristics and clustering by site, the adverse Investigators (PALISI)
tracheal intubation-associated event rate significantly decreased com-
pared with baseline: benchmark: odds ratio, 0.83 (0.72–0.97; p = 0.016);
early bundle: odds ratio, 0.80 (0.63–1.02; p = 0.074); and late bundle *See also p. 358.
odds ratio, 0.63 (0.47–0.83; p = 0.001). Copyright © 2020 by the Society of
CONCLUSIONS: Effective implementation of a quality-improvement Critical Care Medicine and Wolters
bundle was associated with a decrease in the adverse tracheal intubation- Kluwer Health, Inc. All Rights Reserved.
associated event that was sustained for 24 months. DOI: 10.1097/CCM.0000000000004725

250      www.ccmjournal.org February 2021 • Volume 49 • Number 2

Copyright © 2020 by the Society of Critical Care Medicine and Wolters Kluwer Health, Inc. All Rights Reserved.

KEY WORDS: adverse event; bundle; checklist;
child; intensive care unit; intubation; pediatric;
safety; tracheal intubation
T
racheal intubation (TI) is a life-saving, yet
hazard-prone procedure for critically ill chil-
dren (1). In contrast to children intubated by
anesthesiologists in the operating suites, critically ill
children are more likely to have a rapid oxygen de-
saturation, hemodynamic instability, and difficult
airway anatomy. These pediatric-specific features
increase risks and adverse events associated with TI
procedures (1–4).
To improve the TI safety for critically ill children,
a multicenter quality-improvement (QI) collabora-
tive network: National Emergency Airway Registry
for Children (NEAR4KIDS) was developed. The
collaborative reported 20% of all TIs are associated
with adverse tracheal intubation-associated events
(TIAEs), and 3% are associated with severe events
such as cardiac arrests (1). Nineteen percent of TIs
are also associated with hypoxemia with pulse ox-
imetry Spo2 < 80% (3). Identified risks for adverse
TIAEs were respiratory failure, hemodynamic in-
stability, and nonanesthesia-resident participation
as primary laryngoscopists (2–5). The occurrence
of either adverse TIAEs or severe hypoxemia dur-
ing TI was associated with a longer duration of me-
chanical ventilation, and the occurrence of severe
adverse events was associated with increased ICU
mortality (6).
Leveraging our knowledge in the risk factors as-
sociated with adverse TIAEs and hypoxemia events,
we have developed a bundle of QI interventions for
diverse PICUs (7). This bundle includes: 1) quarterly
benchmark performance data and debriefing feed-
back to ICUs and 2) implementation of airway safety
checklists, which consists of the following three com-
ponents: a) TI procedure planning with risk factor re-
view by bedside clinicians, b) time-out immediately
before the TI to review team member roles and ex-
pected behaviors, and c) bedside team debriefing im-
mediately after the TI procedure.
Our objective was to evaluate the clinical effective-
ness of the bundled QI interventions to reduce adverse
TIAEs after each ICU met and sustained the prede-
fined QI bundle adherence criteria.
Critical Care Medicine www.ccmjournal.org      251
Copyright © 2020 by the Society of Critical Care Medicine and Wolters Kluwer Health, Inc. All Rights Reserved.
Clinical Investigations
MATERIALS AND METHODS
Study Design, Setting, and Participants
This study was conducted as a multicenter QI inter-
vention to improve the safety of TIs in critically ill chil-
dren in PICUs in the United States. Each participating
ICU received an endorsement from the local ICU
committee or leadership. Participating sites were the
part of the NEAR4KIDS Network, an international QI
collaborative within the Pediatric Acute Lung Injury
and Sepsis Investigators Network (8).
All consecutive TIs in the participating ICU during
the study period were entered into the NEAR4KIDS
registry database. We censored TIs that occurred during
the “run-in implementation phase” before achievement
of 80% compliance (defined as the time from check-
list introduction in the ICU to time achieving > 80%
bundle adherence for 3 mo) in the original analysis.
We excluded the ICUs that did not achieve bundle ad-
herence, because our goal was to evaluate the clinical
effectiveness of the bundled QI interventions after each
ICU met and sustained the predefined QI bundle ad-
herence criteria. Therefore, only the TI data from the
ICUs that achieved bundle adherence were included in
the main analyses. The TI data from the ICUs that did
not reach bundle adherence were analyzed separately.
Definitions and Interventions
We defined each QI phase as baseline phase (–24 to
–12 mo before checklist implementation), benchmark
performance reporting only phase (–12 to 0 mo before
checklists introduced), run-in implementation phase
(time from checklist introduction to time achieving
> 80% bundle adherence for 3 consecutive months),
early bundle adherence phase (0–12 mo after > 80%
bundle adherence), and late (sustained) bundle adher-
ence phase (12–24 months after the > 80% bundle ad-
herence). Participating ICUs received their quarterly
benchmark performance reports after 12 months of
NEAR4KIDS participation (baseline). These bench-
mark reports included the number of TIs, patient and
provider demographics, practice characteristics, ad-
verse TIAEs, and hypoxemia rates for their ICU (site),
and overall in the NEAR4KIDS collaborative. Since the
majority of sites were already participating in the reg-
istry before the study phase, some site leaders received
their benchmark report during the baseline phase.
However, for the sites whose leaders were already
Nishisaki et al
receiving reports, an emphasis on quarterly debriefing
with the site team on safety outcomes as well as pro-
cess variances (e.g., multiple attempts) was made dur-
ing the benchmark performance reporting only phase
(–12 to 0 mo before checklist implementation).
The airway safety checklist was developed using the
observational data from the NEAR4KIDS database to
address patient, provider, and practice risk factors asso-
ciated with the adverse TI outcomes (7). The checklist
was designed to align with clinicians’ bedside work-
flow: preparation (risk identification and planning for
provider, approach, and backup plan), preprocedure
timeout, and postprocedural debriefing. In the rare sit-
uation of emergency, the preparation section might be
omitted (by indicating as an emergent procedure such
as pericardiac arrest) but the preprocedure timeout
and postprocedural debriefing were still required. The
checklist implementation process used the following
five elements: 1) strong encouragement of site lead-
ers to develop a multidisciplinary team for QI bundle
implementation, 2) best practice videos available to
all sites (www.youtube.com/watch?v=YCGPOl6E1rc&
feature=youtu.be), 3) data feedback with a run chart
with checklist compliance rate, 4) quarterly teleconfer-
ences to address issues related to QI implementation,
and 5) semiannual (twice a year) face-to-face meetings
for peer learning and exchange of barriers and facili-
tators. Detailed data regarding the QI implementation
process, barriers, and facilitators were previously pub-
lished (9). Full bundle adherence was a priori defined
as greater than 80% compliance with a checklist use for
3 consecutive months for each ICU.
Data Collection
The TI data collection was approved by the Institutional
Review Board (IRB) or review was waived as a QI ac-
tivity at each NEAR4KIDS participating site. Each
IRB granted a waiver to obtain an individual informed
consent from patients, guardians, or providers for data
collection.
The NEAR4KIDS dataset includes patient, provider,
and practice characteristics as well as the outcomes
(i.e., attempt success, the occurrence of predefined ad-
verse TIAEs, and hypoxemia with Spo2 < 80%) and
process variances (i.e., multiple attempts). Consensus
data definitions for data elements and outcomes are
previously published (1, 5). Briefly, the adverse TIAEs
include both severe and nonsevere TIAEs. The severe
252      www.ccmjournal.org February 2021 • Volume 49 • Number 2
Copyright © 2020 by the Society of Critical Care Medicine and Wolters Kluwer Health, Inc. All Rights Reserved.
TIAEs include cardiac arrests, esophageal intubation
with delayed recognition, emesis with aspiration, hy-
potension, laryngospasm, and pneumothorax/pneu-
momediastinum. The nonsevere TIAEs include main
stem bronchial intubation, esophageal intubation with
immediate recognition, emesis without aspiration,
hypertension requiring treatment, epistaxis, dental/
lip trauma, medication error, dysrhythmia, and pain/
agitation delaying the procedure. Data were initially
collected by the bedside clinicians after each TI using
a standard data collection form, followed by a data-
verification process by each site project team following
a site-specific compliance plan. The compliance plan
required a capture rate of greater than 95% of TIs at
each site. The data-coordinating center crosschecked
the data entry and communicated with the site leader
and coordinator for clarification of the data when
needed. The operational definitions were used consist-
ently throughout the data collection.
Outcomes
The primary outcome was the occurrence of any ad-
verse TIAE. The secondary outcomes included severe
TIAEs, multiple TI attempts (defined as three or more
attempts), and hypoxemia (3, 10). Hypoxemia (oxygen
desaturation) was defined as the lowest oxygen pulse
oximetry during TI procedure less than 80% in a TI en-
counter despite initial pulse oximetry level was greater
than 90% after preoxygenation (3, 6). Therefore, those
infants who did not achieve Spo2 > 90% after preoxy-
genation (i.e., unrepaired cyanotic heart disease) were
not considered to have hypoxemia even when they ex-
perienced a drop in Spo2 below 80% during intubation.
Statistical Analysis
Sample size calculation was not performed due to the
nature of QI interventions and initial uncertainties in
the number of participating sites and sites achieving
adherence criteria. We planned to recruit at least 15
sites with 1,200 TIs during the baseline phase to be
able to detect a 4.5% reduction in adverse outcomes
during the checklist adherence phase. To evaluate each
component of the QI bundle in reducing the adverse
outcomes, we a priori decided to use the baseline phase
(12 mo) as a reference throughout the analyses.
Categorical variables were described as number and
percentages, and nonnormally distributed continuous

variables as median and interquartile ranges (IQRs). For
the estimated odds ratio (OR), a 95% CI was presented.
To evaluate the effectiveness of QI bundle implemen-
tation, the following steps were taken. First, a univariate
analysis was performed to evaluate each phase of the QI
intervention on primary and secondary outcomes with
chi-square tests. Second, a multivariable analysis was per-
formed with patient factors associated with the QI inter-
vention phase as covariates. The run-in implementation
phase (time from checklist introduction to time achiev-
ing > 80% bundle adherence for 3 consecutive months)
was excluded from the main analysis, since the goal of
this project is to evaluate the TI safety outcomes after
each ICU met and sustained the predefined QI bundle
adherence criteria (Supplemental Fig. A, Supplemental
Digital Content 1, http://links.lww.com/CCM/F957).
Pediatric Index of Mortality 2 (PIM2), as a patient se-
verity score, was missing in 14% of the TI encounters.
Therefore, the PIM2 score was imputed with missing at
random assumption. This method is more robust than
simply eliminating the subjects with missing values (11).
Note that in our multivariable analysis, we adjusted for
patient factors (e.g., age, severity of illness, difficult airway
features, and indications) and clustering by site, but not
for provider (e.g., experience level of laryngoscopist)
or measured practice factors (e.g., equipment use and,
medications used), because the bundle checklists were
specifically intended to have clinicians match provider
skill sets and the approach selections to the anticipated TI
risks and difficulties. We did adjust for clustering by site
to account for unmeasured variables in patient popula-
tions, provider skill level, or practice patterns that might
contribute to the TIAEs that were site-specific.
A sensitivity analysis was conducted to include the
TIs during the run-in implementation phase, which
was excluded in the original analysis. Multivariable
analyses were repeated after the missing PIM2 values
were imputed with this cohort. We also evaluated the
potential secular trend by analyzing the TI safety out-
comes across the four sites that did not meet the criteria
of bundle adherence. All data analyses were conducted
using STATA 14.0 (StataCorp, College Station, TX).
RESULTS
ICU Characteristics
From January 2013 to December 2015, a total of
19 ICUs that completed 24 months of baseline data
Critical Care Medicine www.ccmjournal.org      253
Copyright © 2020 by the Society of Critical Care Medicine and Wolters Kluwer Health, Inc. All Rights Reserved.
Clinical Investigations
collection (i.e., baseline phase and benchmark perfor-
mance reporting only phase) implemented QI bundles
with checklists. Fifteen ICUs (79%) achieved greater
than 80% QI bundle compliance for 3 consecutive
months over a median of 395 days (IQR, 119–752 d)
(Supplemental Table B, Supplemental Digital Content
2, http://links.lww.com/CCM/F958). Pediatric Critical
Care Medicine Fellowship Programs were present in
66% of the participating ICUs. There was a system for
24-hour in-hospital ICU attending coverage in 73%
of sites. Checklist compliance increased over time
(Supplemental Fig. C, Supplemental Digital Content
3, http://links.lww.com/CCM/F959).
Patient, Provider, and Practice Characteristics
Table 1 shows patient characteristics during baseline
and benchmark performance feedback phases (24 mo,
before checklist implementation) versus after greater
than 80% QI checklist adherence was achieved. Fewer
intubations for procedural indication were reported after
checklist implementation. Patient age was older and
PIM2 was lower in TIs after the QI checklist adherence
was achieved (these differences are controlled for in the
multivariable analysis). Table 2 describes provider and
practice characteristics. TIs by attending physicians and
residents decreased after achieving bundle adherence.
Effect of the Bundled Quality-Improvement
Intervention
The rate of adverse TIAEs decreased from 17.5% (95%
CI, 15.4–19.7%) in the baseline phase to 13.7% (95%
CI, 11.6–16.0%) in the 24 months after greater than
80% checklist adherence was achieved (Table 3). After
adjusting for patient age, PIM2 predicted mortality,
history of difficult airway, difficult airway features, TI
indications, and site-level clustering, the QI phases
were associated with decreased prospectively de-
fined adverse TIAEs: benchmark performance phase
OR 0.84 (95% CI, 0.73–0.97; p = 0.022), early (0–12
mo) checklist bundle greater than 80% compliance
phase OR 0.80 (95% CI, 0.63–1.02; p = 0.074), and late
(12–24 mo) checklist bundle greater than 80% com-
pliance phase OR 0.63 (95% CI, 0.47–0.83; p = 0.001)
(Table 4). Figure 1 shows the probability of adverse
TIAE rates for each quarter (3 mo) before checklist
implementation, and after greater than 80% checklist,
bundle adherence was achieved.
Nishisaki et al

TABLE 1.
Patient Characteristics Before Checklist Implementation and After Bundle Checklist
Adherence
Before Checklist After Checklist
Implementation Adherence
Phases (n = 2,991) (n = 2,593) p

Age
  Age, yr (median [IQR]) 1 (0–6) 1 (0–8) 0.0011
 Infant, n (%) 1,331 (44) 1,040 (40)
  Child (1–7 yr), n (%) 981 (33) 904 (35)
  Older child (8-17 yr), n (%) 563 (19) 533 (21)
  Adult (18 yr or above), n (%) 116 (4) 116 (4)
Pediatric Index of Mortality 2 (%, median [IQR])
a
2.9 (0.9–6.9) 2.1 (0.9–4.9) < 0.001
Diagnosis
  Lower respiratory, n (%) 943 (33) 864 (33) < 0.001
  Upper respiratory, n (%) 248 (9) 200 (8)
 Neurologic, n (%) 506 (18) 623 (24)
 Cardiac-surgical, n (%) 317 (11) 178 (7)
 Cardiac-medical, n (%) 135 (5) 138 (5)
 Sepsis/shock, n (%) 171 (6) 186 (7)
 Trauma, n (%) 92 (3) 65 (3)
 Other, n (%) 439 (15) 339 (13)
Indication for tracheal intubationb
 Respiratory, n (%) 1,585 (53) 1,418 (55) 0.206
 Shock, n (%) 310 (10) 304 (12) 0.105
 Procedural, n (%) 618 (21) 477 (18) 0.033
 Neurologic , n (%)
c 281 (9) 324 (13) < 0.001
Difficult airway features
  History of difficult airway, n (%) 406 (14) 425 (16) 0.003
  Any difficult airway feature, n (%) 1,092 (37) 881 (34) 0.048
IQR = interquartile range.
Pediatric Index of Mortality 2 was missing in 785 cases: 362 cases (12%) before checklist implementation and 423 cases (16%) after
a

checklist adherence.
Patients may have more than one indication for intubation.
b

Neurologic indication includes therapeutic hyperventilation, airway protection, and neuromuscular weakness.
c

The occurrence of multiple attempts and hypoxemia
with Spo2 < 80% did not significantly decrease after the
sites achieved bundle adherence.
Sensitivity analysis including all TIs during the
run-in implementation phase revealed a similar re-
sult (Supplemental Table D, Supplemental Digital
Content 4, http://links.lww.com/CCM/F960).
Tracheal Intubation Safety From the Sites That
Did Not Achieve Bundle Adherence
The rate of adverse TIAEs, multiple attempts, and hypox-
emia did not change across QI phases among the four
ICUs that did not achieve bundle adherence: baseline
phase (15.1%), benchmark performance reporting only

254      www.ccmjournal.org February 2021 • Volume 49 • Number 2

Copyright © 2020 by the Society of Critical Care Medicine and Wolters Kluwer Health, Inc. All Rights Reserved.
 Clinical Investigations

TABLE 2.
Provider and Practice Characteristics Before and After Bundle Checklist Adherence
Before Checklist After Checklist
Implementation Adherence
Phases (n = 2,991) (n = 2,593) p

Provider, n (%)
  PCCM/EM attending 877 (30) 505 (19) < 0.001
  PCCM/EM Fellow 853 (29) 1,164 (45)
  Peds/EM resident 718 (24) 356 (14)
  Nurse practitioner 216 (7) 280 (11)
 Hospitalist 5 (0) 14 (1)
  Respiratory therapist 6 (0) 8 (0)
 Subspecialist 158 (5) 159 (6)
 Other 158 (5) 107 (4)
Devicea, n (%)
  Direct laryngoscopy 2,578 (87) 2,225 (88) 0.357b
  Video laryngoscopy 318 (11) 297 (12)
 Other 44 (1) 19 (1)
Method of intubation, n (%)
  Initial intubation 2,612 (88) 2,307 (89) 0.144
  Tube change 366 (12) 286 (11)
Medication, n (%)
 Atropine 675 (23) 600 (23) 0.612
 Glycopyrrolate 172 (6) 272 (10) < 0.001
 Fentanyl 1,617 (54) 1,660 (64) < 0.001
 Midazolam 1,555 (52) 1,334 (51) 0.685
 Ketamine 837 (28) 590 (23) < 0.001
 Propofol 492 (16) 496 (19) 0.009
 Etomidate 53 (2) 34 (1) 0.166
  Any paralytic 2,557 (85) 2,223 (86) 0.798
 Rocuronium 1,891 (63) 1,924 (74) < 0.001
 Vecuronium 403 (13) 281 (11) 0.003
 Cisatracurium 267 (9) 22 (1) < 0.001
 Succinylcholine 15 (1) 19 (1) 0.268
Endotracheal tube type, n (%)
 Cuffed 2,808 (94) 2,479 (96) 0.003
 Uncuffed 166 (6) 93 (3)
 Other ≠
15 (0) 21 (1)
EM = Emergency Medicine, PCCM = Pediatric Critical Care Medicine.
Device data were missing in eight cases (0.2%).
a

p value was calculated for tracheal intubations with direct versus video laryngoscopy.
b

Critical Care Medicine www.ccmjournal.org      255

Copyright © 2020 by the Society of Critical Care Medicine and Wolters Kluwer Health, Inc. All Rights Reserved.
Nishisaki et al

TABLE 3.
Univariate Analysis: Primary and Secondary Outcomes Before and After Bundle Checklist
Adherence
Checklist Checklist
Baseline Benchmark Adherence Adherence
(Year 1), (Year 2), (Year 1), (Year 2), Total,
Phase n = 1,241 n = 1,750 n = 1,591 n = 1,002 n = 5,584 p

Primary outcome, n (%)

  Any adverse TIAEs 217 (17.5) 257 (14.7) 247 (15.5) 137 (13.7) 858 (15.4) 0.067
Secondary outcome, n (%)
  Severe TIAEs 67 (5.4) 87 (5.0) 76 (4.8) 54 (5.4) 284 (5.1) 0.848
  Nonsevere TIAEs 167 (13.5) 193 (11.0) 187 (11.8) 99 (9.9) 646 (11.6) 0.053
  Multiple attempts 130 (10.5) 222 (12.7) 195 (12.3) 133 (13.3) 680 (12.2) 0.178
  Oxygen desaturation 209 (16.8) 301 (17.2) 282 (17.7) 188 (18.8) 980 (17.6) 0.652
TIAEs = tracheal intubation-associated events.
Baseline phase consists of –24 to –12 mo before the checklist implementation. During this baseline phase, some sites received benchmark


feedback data.
Benchmark phase consists of –12 mo to the date when the bundle checklist was officially implemented. See the Materials and Methods


section for details.

Checklist adherence phase year 1 is defined as the first 12 months from the month when the adherence criteria were first met. For the
adherence criteria, refer to the Materials and Methods section of the article.
p value is calculated by χ2 analysis.
This table does not include the data during bundle implementation (from implementation date to checklist adherence > 80%).

phase (13.7%), and run-in implementation phase (13.4%)
(p = 0.672) (Supplemental Table E, Supplemental Digital
Content 5, http://links.lww.com/CCM/F961).
DISCUSSION
Our study demonstrated that TI procedural safety
was enhanced across 15 PICUs after successful imple-
mentation and sustained maintenance of QI bundles
with checklists, performance reports, and individual
postprocedural peer debriefing. The vast majority of
the ICUs were able to achieve QI bundle adherence
defined a priori as greater than 80% use of QI bundle
checklists. The rate of adverse TIAEs decreased signif-
icantly over time across the successfully implementing
sites, after adjusting for important patient-level factors
(e.g., age, severity of illness, difficult airway features,
and indication). These effects were sustained for at
least 24 months.
Our QI intervention incorporated best practice QI
approaches, including benchmarking, evidence-based
checklist development and implementation, leveraging
ICU leadership buy-in, and involving multidisciplinary
clinicians (12). The checklists thoroughly addressed
three critical components of safe TI procedure: initial
patient risk factor assessment and planning, preproce-
dural time out, and scripted postprocedure debriefs.
The effect of benchmark performance review and
checklist implementation was significant, and the im-
pact was most evident in the sustaining maintenance
phase. Specifically, the adverse TIAE rate was signif-
icantly reduced initially with the implementation of
the benchmark performance review dashboards. It
continued to decrease during the first 12 months of
compliant checklist implementation (early), achiev-
ing statistical significance during the 12–24-month
sustaining maintenance (late) phase. This suggests that
impact may also be influenced by the duration of im-
plementation (sustained penetration). Once bundle
adherence of greater than 80% was achieved, adverse
TI event rates steadily decreased over time. This finding
contrasts with the unchanged rates of adverse TIAEs in
the four ICUs that did not achieve bundle adherence.
Substantial variance was observed in the reduction of

256      www.ccmjournal.org February 2021 • Volume 49 • Number 2

Copyright © 2020 by the Society of Critical Care Medicine and Wolters Kluwer Health, Inc. All Rights Reserved.
 Clinical Investigations

TABLE 4.
Multivariable Analysis for Any Adverse Tracheal Intubation-Associated Events and Quality-
Improvement Implementation Phase, Adjusting for Patient Factors and Site-Level Clustering
Variable OR (95% CI) p

Quality-improvement phase
 Baseline Reference
  Benchmark performance 0.84 (0.73–0.97) 0.022
  Early checklist adherence (year 1) 0.80 (0.63–1.02) 0.074
  Late checklist adherence (year 2) 0.63 (0.47–0.83) 0.001
Patient factors
 Age
   Infant (< 12 mo) Reference
   Young child (1–7 yr) 0.95 (0.79–1.14) 0.591
   Older child (8–17 yr) 0.94 (0.72–1.21) 0.618
   Adult (18 yr or above) 1.14 (0.83–1.57) 0.424
Pediatric Index of Mortality 2 scorea 1.00 (1.00–1.01) 0.331
Indication for tracheal intubation
 Respiratory 1.33 (1.07–1.65) 0.011
 Shock 1.58 (1.33–1.88) < 0.005
 Procedural 1.06 (0.85–1.32) 0.612
 Neurologic 1.10 (0.91–1.32) 0.334
Difficult airway features
  History of difficult airway 1.23 (0.98–1.53) 0.072
  Any difficult airway feature 1.10 (0.93–1.31) 0.276
OR = odds ratio.
Tracheal intubation-associated events denotes tracheal intubation-associated events.
Baseline phase consists of –24 to –12 mo before the checklist implementation.
Benchmark performance consists of –12 mo to the date when the bundle checklist was officially implemented.
Multiple imputation was used to impute Pediatric Index of Mortality 2 in 785 cases. No other variables were imputed.
a

Multivariable logistic regression using generalized estimate equation with an independent covariate structure and binomial linkage.
This table does not include the data during bundle implementation (from implementation date to checklist adherence > 80%).

adverse TIAEs across the ICUs (13). Future studies are
needed to identify the organizational-level factors that
may be responsible for the variance.
It is notable that none of our secondary outcomes,
including severe TIAE, multiple TI attempts, and hy-
poxemia with Spo2 < 80%, were substantially decreased
during the QI implementation despite the lower PIM2
predicted mortality during this period. One would
expect several components of the bundle checklist (risk
identification and planning in provider, approach, and
backup plan) would reduce the multiple attempts and
oxygen desaturation rates. This lack of decrease in mul-
tiple attempts and hypoxemia may be a limitation of
our checklist-based intervention, which was designed
to reduce TIAEs primarily focused on the skills and
behaviors during laryngoscopy and endotracheal tube

Critical Care Medicine www.ccmjournal.org      257

Copyright © 2020 by the Society of Critical Care Medicine and Wolters Kluwer Health, Inc. All Rights Reserved.
Nishisaki et al

the main analysis, pre-

suming the impact of the
full QI bundle would not
be achieved unless check-
list compliance was greater
than 80% for 3 months at
each site. We addressed
this limitation with the
sensitivity analysis, which
demonstrated a similar
clinical effectiveness of QI
bundle implementation
with the run-in implemen-
tation phase included. We
excluded four ICUs that
did not achieve the bundle
adherence in our analyses.
This limits the generaliza-
bility of our study findings.
Figure 1. Estimated probability for adverse tracheal intubation-associated event. Quarter indicates In other words, those ICUs
3 mo on x-axis. The vertical line indicates the time at which each ICU met the criteria for bundle that achieved the adher-
compliance (> 80% checklist use). TI = tracheal intubation. ence may be different from
those that did not in their
placement, and did not focus on bag-mask ventilation ICU structure, safety culture, and TI processes. These
effectiveness, specific devices, or alternative airway four ICUs that did not achieve adherence were large
management skills. Our QI bundle incorporated the academic ICUs with residency and fellowship pro-
current best practice with specific attention to a shared grams. The more complex system in a large academic
mental model, anticipation, and planning. However, ICU may require a well-structured interdisciplinary
this might not be sufficient to reduce the rate of severe QI implementation leadership and a robust commu-
TIAEs, multiple attempt rates, or the occurrence of hy- nication with frontline clinicians as identified in our
poxemia during the TI procedure, which may be more previous mixed-method study (9). Although reporting
related to the severity of illness and trajectory of di- bias could occur, each site developed and followed a
sease than procedural intervention quality. Specific QI rigorous site-specific compliance plan to ensure sys-
interventions, such as: 1) optimizing hemodynamics tematic capture of all TI events and accurate data entry.
to prevent cardiac arrest during TI by early utilization The time-series study design could be prone to
of epinephrine infusion and correction of metabolic changes in secular trends and other unmeasured factors
acidosis (2), 2) use of passive oxygen flow during lar- affecting safety practices. Since each site implemented
yngoscopy (i.e., apneic oxygenation) (14–18), and 3) the QI bundle at various times of the year and achieved
routine use of video laryngoscopy to reduce multiple greater than 80 % adherence at different times, making
attempts and esophageal intubations (19), may poten- seasonal trend biases from patient risk factors (e.g., more
tially result in safer TI practice with lower severe event respiratory failure patients during winter time) or pro-
rates, fewer multiple attempts, and fewer hypoxemia vider expertise (e.g., trainee expertise) less likely. Our
events during TI. Several PICUs have recently imple- analyses accounted for clustering by sites to mitigate po-
mented apneic oxygenation and/or video laryngos- tential unmeasured factors specific to a site. Our analysis
copy as the next set of QI interventions (14). of TI data from four ICUs that did not achieve bundle
Our study has limitations inherent to time-series adherence did not show significant improvement in TI
study design and the nature of QI activity. First, we a safety outcomes despite their baseline lower TIAE rates.
priori eliminated the QI implementation phase from This result suggests that the significant and sustained

258      www.ccmjournal.org February 2021 • Volume 49 • Number 2

Copyright © 2020 by the Society of Critical Care Medicine and Wolters Kluwer Health, Inc. All Rights Reserved.

improvement requires the bundle adherence, which was
most visible after adjusting for patient factors. It is also
possible that our intervention was only able to reduce the
adverse TIAE rates to a certain level; therefore, the non-
adherence sites failed to show an improvement due to
their baseline lower TIAE rates. However, we are not able
to eliminate or adjust for the indirect effect of other spe-
cific concurrent QI activities, the Hawthorne effect, and
general changes in medical practice affecting multiple
sites. Finally, our study did not evaluate the long-term
outcomes such as the duration of mechanical ventilation
or ICU length of stay, since our QI intervention focused
on the intubation procedure, that is, the beginning of in-
vasive mechanical ventilation. Future QI interventions
should address the ventilation-weaning process with the
length of mechanical ventilation as the targeted outcome.
CONCLUSIONS
Effective implementation of a QI bundle in 15 collab-
orative sites was associated with a significant decrease
in the adverse event rate that was sustained for 24
months. These effects persisted after adjusting for age,
patient factors, mortality risk, and clustering by site.
ACKNOWLEDGMENT
We thank all National Emergency Airway Registry for
Children sites participating in airway safety quality-
improvement bundles for their dedication and hard
work. We also thank Hayley Buffman, MPH, and Cassie
Simpson-Dukes for their administrative support.
1 Department of Anesthesiology and Critical Care Medicine,
Children’s Hospital of Philadelphia, Philadelphia, PA.
2 Division of Critical Care Medicine, Department of Pediatrics,
Nationwide Children’s Hospital, Ohio State University,
Columbus, OH.
3 Department of Pediatrics, Division of Pediatric Critical Care
Medicine, Maria Fareri Children’s Hospital, Valhalla, NY.
4 Section of Critical Care, Department of Pediatrics, Arkansas
Children’s Hospital, Little Rock, AR.
5 Department of Emergency Medicine, Brigham and Women’s
Hospital, Harvard Medical School, Boston, MA.
6 Division of Pediatric Critical Care, Department of Pediatrics,
Duke Children’s Hospital, Durham, NC.
7 Department of Respiratory Therapy, Children’s Hospital of
Philadelphia, Philadelphia, PA.
8 Division of Pediatric Critical Care, University of Louisville
and Norton Children’s Hospital, Louisville, KY.
Critical Care Medicine www.ccmjournal.org      259
Copyright © 2020 by the Society of Critical Care Medicine and Wolters Kluwer Health, Inc. All Rights Reserved.
Clinical Investigations
9 Division of Critical Care, Children’s Hospital of the University
of Virginia, Charlottesville, VA.
10 Pediatric Critical Care Medicine, Pediatric Acute Care
Associates of North Texas PLLC, Medical City Children’s
Hospital, Dallas, TX.
11 Division of Critical Care, Stony Brook Children’s Hospital,
Stony Brook, NY.
12 Department of Pediatrics, New York-Presbyterian Weill
Cornell Medical Center, New York, NY.
13 Division of Pediatric Critical Care, Department of Pediatrics,
Dartmouth-Hitchcock Medical Center, Lebanon, NH.
14 Division of Critical Care, Children’s Hospital and Clinics of
Minnesota, Minneapolis, MN.
15 Department of Pediatrics, Division of Pediatric Critical Care,
Kentucky Children’s Hospital, University of Kentucky School
of Medicine, Lexington, KY.
16 Pediatric Critical Care Medicine, Department of Pediatrics,
Phoenix Children’s Hospital, Phoenix, AZ.
17 Division of Pediatric Critical Care Medicine, Doernbecher
Children’s Hospital, Portland, OR.
18 Department of Respiratory Therapy, Children’s Healthcare of
Atlanta, Atlanta, GA.
19 Division of Pediatric Critical Care Medicine, Department of
Pediatrics, Emory University School of Medicine, Atlanta
GA.
20 Department of Anesthesiology and Critical Care Medicine,
University of Pittsburgh School of Medicine, Pittsburgh, PA.
21 Division of Pediatric Critical Care Medicine, Department of
Pediatrics, Penn State Hershey Children’s Hospital, Pennsylvania
State University College of Medicine, Hershey, PA.
22 Department of Biostatistics, Perelman School of Medicine at
the University of Pennsylvania, Philadelphia, PA.
Supplemental digital content is available for this article. Direct
URL citations appear in the printed text and are provided in the
HTML and PDF versions of this article on the journal’s website
(http://journals.lww.com/ccmjournal).
Drs. Nishisaki, Napolitano, Shults, and Nadkarni are supported
by the Agency for Healthcare Research and Quality (AHRQ
R03HS021583, R18HS022464, and R18HS024511). Dr.
Nadkarni is supported by the Endowed Chair in Critical Care
Medicine at Children’s Hospital of Philadelphia. Dr. Napolitano’s
institution received funding from AHRQ, Draeger, Aerogen,
Philips/Respironics, Smiths Medical, and VERO-Biotech. Dr.
Howell received funding from UptoDate. Dr. Nadkarni’s institu-
tion received funding from AHRQ R18. The remaining authors
have disclosed that they do not have any conflicts of interest.
Successful implementation of a tracheal intubation patient safety
bundle intervention improves outcomes and is sustained in across
PICUs
For information regarding this article, E-mail: nishisaki@email.
chop.edu
Nishisaki et al
REFERENCES
1. Nishisaki A, Turner DA, Brown CA 3rd, et al; National
Emergency Airway Registry for Children (NEAR4KIDS);
Pediatric Acute Lung Injury and Sepsis Investigators (PALISI)
Network: A National emergency airway registry for children:
Landscape of tracheal intubation in 15 PICUs. Crit Care Med
2013; 41:874–885
2. Shiima Y, Berg RA, Bogner HR, et al; National Emergency
Airway Registry for Children Investigators: Cardiac arrests as-
sociated with tracheal intubations in PICUs: A multicenter co-
12. Klugman D, Melton K, Maynord PO, et al: Assessment of
hort study. Crit Care Med 2016; 44:1675–1682
3. Li S, Hsieh TC, Rehder KJ, et al; for National Emergency
Airway Registry for Children (NEAR4KIDS) and Pediatric
Acute Lung Injury and Sepsis Investigators (PALISI) Network:
Frequency of desaturation and association with hemodynamic
adverse events during tracheal intubations in PICUs. Pediatr
Crit Care Med 2018; 19:e41–e50
4. Graciano AL, Tamburro R, Thompson AE, et al: Incidence
and associated factors of difficult tracheal intubations in
pediatric ICUs: A report from National Emergency Airway
Registry for Children: NEAR4KIDS. Intensive Care Med 2014;
40:1659–1669
5. Sanders RC Jr, Giuliano JS Jr, Sullivan JE, et al; National
Emergency Airway Registry for Children Investigators and
Pediatric Acute Lung Injury and Sepsis Investigators Network:
Level of trainee and tracheal intubation outcomes. Pediatrics
2013; 131:e821–e828
6. Parker MM, Nuthall G, Brown C 3rd, et al; Pediatric Acute Lung
Injury and Sepsis Investigators (PALISI) Network: Relationship
between adverse tracheal intubation associated events and
PICU outcomes. Pediatr Crit Care Med 2017; 18:310–318
7. Li S, Rehder K, Giuliano J, et al; for the National Emergency
Airway Registry for Children (NEAR4KIDS) and Pediatric
Acute Lung Injury and Sepsis Investigators (PALISI) Network:
Development of a quality improvement bundle to reduce tra-
cheal intubation-associated event in pediatric ICUs. Am J Med
Qual 2016; 31:47–55
8. Randolph AG, Meert KL, O’Neil ME, et al; Pediatric Acute Lung
Injury and Sepsis Investigators Network: The feasibility of con-
ducting clinical trials in infants and children with acute respira-
tory failure. Am J Respir Crit Care Med 2003; 167:1334–1340
9. Davis KF, Napolitano N, Li S, et al; National Airway Emergency
for Children (NEAR4KIDS) and Pediatric Acute Lung Injury
and Sepsis Investigators (PALISI) Network: Promoters and
barriers to implementation of tracheal intubation airway safety
260      www.ccmjournal.org February 2021 • Volume 49 • Number 2
Copyright © 2020 by the Society of Critical Care Medicine and Wolters Kluwer Health, Inc. All Rights Reserved.
bundle: A mixed-method analysis. Pediatr Crit Care Med 2017;
18:965–972
10. Lee JH, Turner DA, Kamat P, et al; Pediatric Acute Lung Injury
and Sepsis Investigators (PALISI); National Emergency Airway
Registry for Children (NEAR4KIDS): The number of tracheal
intubation attempts matters! A prospective multi-institutional
pediatric observational study. BMC Pediatr 2016; 16:58
11. Janssen KJ, Donders AR, Harrell FE Jr, et al: Missing covariate
data in medical research: To impute is better than to ignore. J
Clin Epidemiol 2010; 63:721–727
an unplanned extubation bundle to reduce unplanned extu-
bations in critically ill neonates, infants, and children. JAMA
Pediatr 2020; 174:e200268
13. Nett S, Emeriaud G, Jarvis JD, et al; NEAR4KIDS Investigators
and Pediatric Acute Lung Injury and Sepsis Investigators
(PALISI) Network: Site-level variance for adverse tracheal
intubation-associated events across 15 North American
PICUs: A report from the national emergency airway registry
for children. Pediatr Crit Care Med 2014; 15:306–313
14. Napolitano N, Laverriere EK, Craig N, et al; National Emergency
Airway Registry for Children (NEAR4KIDS) and Pediatric
Acute Lung Injury and Sepsis Investigators (PALISI): Apneic
oxygenation as a quality improvement intervention in an aca-
demic PICU. Pediatr Crit Care Med 2019; 20:e531–e537
15. Chaudhuri D, Granton D, Wang DX, et al: Moderate certainty
evidence suggests the use of high-flow nasal cannula does not
decrease hypoxia when compared with conventional oxygen
therapy in the peri-intubation period: Results of a systematic
review and meta-analysis. Crit Care Med 2020; 48:571–578
16. Binks MJ, Holyoak RS, Melhuish TM, et al: Apneic oxygenation
during intubation in the emergency department and during re-
trieval: A systematic review and meta-analysis. Am J Emerg
Med 2017; 35:1542–1546
17. Binks MJ, Holyoak RS, Melhuish TM, et al: Apnoeic oxygena-
tion during intubation in the intensive care unit: A systematic
review and meta-analysis. Heart Lung 2017; 46:452–457
18. Soneru CN, Hurt HF, Petersen TR, et al: Apneic nasal oxy-
genation and safe apnea time during pediatric intubations by
learners. Paediatr Anaesth 2019; 29:628–634
19. Grunwell JR, Kamat PP, Miksa M, et al; National Emergency
Airway Registry for Children (NEAR4KIDS) and the Pediatric
Acute Lung Injury and Sepsis (PALISI) Network: Trend and
outcomes of video laryngoscope use across PICUs. Pediatr
Crit Care Med 2017; 18:741–749