LWW/JNCQ JNCQ-D-15-00118 February 9, 2016 1:24

J Nurs Care Qual

Vol. 31, No. 2, pp. 161–166
Copyright c 2016 Wolters Kluwer Health, Inc. All rights reserved.

Feasibility and Reliability of

Pediatric Early Warning Score
in the Emergency Department
Xun Niu, PhD; Bradley Tilford, MD; Elizabeth Duffy, MA;
Hitomi Kobayashi, PhD, RN; Kelley Ryan, MSN, RN;
Mindi Johnson, MSN, RN, CPN; Bethany Page, MSN, RN,
CPN; Claire Martin, MSN, RN; Rhonda Caldwell, BSN, RN,
CPN; Prashant Mahajan, MD, MPH, MBA
Pediatric early warning scores in an emergency department may be used in routine patient eval-
uation of illness severity and resource allocation, thereby positively impacting quality and safety
in pediatric care. This prospective nursing study assessed the feasibility and reliability of pediatric
early warning scores in a busy, inner-city, level 1 trauma center pediatric emergency department.
The pediatric early warning scores demonstrated high interrater reliability (degree of agreement
among scorers) (intraclass correlation coefficient = 0.91) and intrarater reliability (multiple rep-
etitions by a single scorer) (intraclass correlation coefficient = 0.90). Key words: emergency
department, interrater reliability, intrarater reliability, nurses, pediatric early warning score
(PEWS), pediatrics

T HE PEDIATRIC EARLY WARNING SCORE

Author Affiliations: Division of Critical Care,
Department of Pediatrics (Dr Tilford) and
Department of Pediatrics (Dr Niu) and Nursing (Dr
Kobayashi and Mss Ryan, Johnson, Page, Martin,
and Caldwell), Children’s Hospital of Michigan,
Detroit; Department of Pediatrics, Wayne State
University, Detroit, Michigan (Ms Duffy); and
Division of Emergency Medicine, Department of
Pediatrics, Wayne State University, Children’s
Hospital of Michigan, Detroit (Dr Mahajan).
This project was supported by award R2-2014-AB from
the Children’s Hospital of Michigan Foundation and the
Center for Quality and Innovation, Children’s Hospital
of Michigan.
The authors acknowledge contributions of Eklund
Fisher and William Rumao to this project.
The authors have no conflicts of interest to report.
Supplemental digital content is available for this article.
Direct URL citations appear in the printed text and are
provided in the HTML and PDF versions of this article
on the journal’s Web site (www.jncqjournal.com).
Correspondence: Prashant Mahajan, MD, MPH,
MBA, Emergency Department, Children’s Hospital of
(PEWS), a scoring system based on age-
dependent vital signs, clinical appearance,
and level of consciousness, was initially de-
signed to assess a patient’s clinical status
to identify risk for clinical deterioration and
need of concentrated care in a pediatric in-
tensive care unit (ICU).1 The initial scor-
ing assessment evaluated patient’s behavior,
cardiovascular status, and respiratory status.1
Since 2005, there have been several studies
on alternately designed PEWS for the inpa-
tient and more recently, emergency depart-
ment (ED) settings.2-6 While recent ED-based
studies have examined the PEWS in assessing
need for admissions,7-9 identifying patients at
Michigan, 3901 Beaubien, Detroit, MI 48201 (pmaha-
jan@dmc.org).
Accepted for publication: September 6, 2015
Published ahead of print: February 5, 2016
DOI: 10.1097/NCQ.0000000000000162

161

Copyright © 2016 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
LWW/JNCQ JNCQ-D-15-00118 February 9, 2016 1:24
162 JOURNAL OF NURSING CARE QUALITY/APRIL-JUNE 2016
high-risk for deterioration at the time of ED
presentation,10 and validating PEWS in the
ED,11 few have established the feasibility and
reliability in this setting.4,12-14
The PEWS could potentially be added to the
clinician’s resources of objective tools to help
quantify clinical status at ED presentation and
monitor progress of illness. However, it is im-
portant to demonstrate that (a) it is feasible to
perform PEWS in an ED setting where nurses
face substantial time constraints and (b) the
score is reliable. Therefore, the objectives of
this study are to assess the nursing time re-
quired to reliably administer a modified PEWS
and the inter- and intrarater reliability of the
PEWS in the pediatric ED setting.
MATERIALS AND METHODS
Location
The study was conducted at an urban, ter-
tiary care hospital with a level 1 pediatric
ED, which has an annual census of approx-
imately 90 000 visits. We held focus groups
with nurses from the inpatient and ED units to
modify the initial PEWS1 to be in line with clin-
ical nursing care prior to the education, train-
ing, and implementation of the study. The
study investigators and the ED nurse educa-
tor designed and documented training materi-
als for 56 ED nurses, including PowerPoint
presentations and in-person meetings with
question and answer sessions prior to study
start-up.
Study design
The PEWS was scored on ED patients aged
18 years or younger, presenting May 1, 2013,
to April 30, 2014. The modified PEWSs range
from 0 to 3 in each of the following domains:
behavior, cardiovascular, and respiratory, and
the total score ranges from 0 to 9.6 In addi-
tion to the modified PEWS evaluation, abnor-
mal age-related vital signs, and white blood
cell count cutoffs15 were used in the PEWS
evaluation. Nurses were provided a 1-page
handout and laminated cards that could be at-
tached to their ID badges. Patients with acute
Copyright © 2016 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
life-threatening emergencies and children re-
quiring cardiopulmonary resuscitation were
excluded. The study received institutional re-
view board approval and a waiver of informed
consent and assent. Although this study was
not designed to test validity of the modified
PEWS, we collected clinical outcomes includ-
ing disposition, final diagnosis on Interna-
tional Classification of Diseases, Ninth Re-
vision (ICD-9) codes, and data on any clinical
interventions and therapies.
Nurses independently evaluated patients in
ED rooms. Nurses were directed to withhold
their score so as to not create undue influ-
ence on other nurses scoring the modified
PEWS for interrater reliability, and they were
instructed to not review their own scores
during the intrarater reliability investigation.
The nurses performing the modified PEWS
may or may not have been actively involved
with bedside care of patients scored. Nurs-
ing education level and years of nursing ex-
perience were collected. The start and end
times for each PEWS evaluation were also
recorded.
We performed 2 separate tests to evaluate
reliability: (1) interrater reliability to assess
degree of agreement among those adminis-
tering the score and (2) intrarater reliability
to assess the agreement among multiple
repetitions by a single scorer. For both the
inter- and intrarater reliability investigations,
the time spent on each PEWS evaluation
was computed as the difference between
ending and starting time recorded on grading
sheet.
To measure interrater reliability, 3 nurses
performed PEWS on an individual patient 3
minutes apart. A total of 47 patients were
scored for interrater reliability investigation.
For the intrarater reliability assessment, 53 in-
dividual nurses performed modified PEWS on
53 different patients. Each patient was scored
by the same nurse 3 times at 0, 30, and 60
minutes. Intrarater reliability was defined as
the degree of agreement among multiple rep-
etitions of PEWS performed by a single nurse.
The study procedure training for intrarater re-
liability included instructions to not review
LWW/JNCQ JNCQ-D-15-00118 February 9, 2016 1:24
Feasibility and Reliability of Pediatric Early Warning Score in the Emergency Department
the previous score administered at 30 and 60
minutes.
In addition, we queried the electronic med-
ical record to collect data on patient clinical
outcomes for patients scored in both the inter-
and intrarater reliability investigations, such
as admission to the hospital, transfer to ICU,
antibiotics, intravenous saline bolus, and labo-
ratory tests (urinalysis, urine culture analysis,
and blood culture analysis).
We used intraclass correlation coefficient
(ICC, absolute agreement, single measure) in
SPSS 22.0 (SPSS Inc., Chicago, Illinois) to mea-
sure the intrarater reliability (1-way random
ICC) and interrater reliability (2-way random
ICC) of PEWS numeric scores. Nurses (n =
56) were educated on the modified PEWS and
administered the PEWS on a total of 100 pa-
tients (47 for interrater reliability and 53 for
intrarater reliability) to achieve an 80% sta-
tistical power for the ICC analysis. We de-
fined excellent reliability as ICC of 0.90 or
greater, very good reliability as ICC of 0.85
greater, and good as ICC of 0.75 or greater for
analysis. Pearson χ 2 test was used to analyze
2 × 2 contingency table for PEWS and patient
clinical outcomes if the minimum expected
number was at least 5 in any cell, otherwise
Fisher-Irwin test was used for the analysis. Sig-
nificance level was set at 0.05.
RESULTS
Among the 56 nurses who participated in
the study, 40% had an associate, 48% had
a bachelor’s, and 10% had master’s degree
in nursing. Their nursing experience ranged
from 2 months to 38 years (median: 4.0 years,
range: 1-11 years). More than half (58.9%) of
the nurses had prior experience in perform-
ing PEWS. Recorded PEWS ranged from 0 to 7,
and their distributions were 62.3% (0-2, mild
illness severity); 33% (3-5, moderate severity);
and 4.7% (6-9, severe). Because of the low per-
centage of PEWS in the category of 6 to 9, 2
categories of 3 to 5 and 6 to 9 were combined
for the purpose of analysis; it has been shown
that patients with a PEWS of 3 or greater have
a higher chance to be transferred to intensive
Copyright © 2016 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
163
care level.4,6,16 Patient demographics and the
PEWS for all patients (n = 100) are reported in
Supplemental Digital Content Table, available
at http://links.lww.com/JNCQ/A230. Patients
between 1 month and 5 years of age had a
higher chance to be in category 3 to 9 than
in category 0 to 2, while patients older than
6 years had higher chance to be in category 0
to 2 than 3 to 9.
In the interrater tests, 83.0% of patients
(39/47) were given exactly the same score by
the 3 nurses; 10.6% of patients (5/47) received
the same scores from only 2 nurses, and 6.4%
of patients (3/47) received scores that were in
complete disagreement. The differences be-
tween 3 PEWSs and the 3-score average on
each patient were calculated; the standard de-
viation of the pooled difference was found
to be 0.36. The maximum difference among
scores for each patient was also computed.
It was found the largest score difference was
4, which occurred in only 1 patient (see Sup-
plemental Digital Content Table, available at
http://links.lww.com/JNCQ/A230); the mean
of the maximum differences was 0.3. Two
patients received mixed scores across cate-
gories 0 to 2 (mild illness severity) and 3 to
5 (moderate severity); another 2 patients re-
ceived mixed scores across categories 3 to 5
and 6 to 9 (severe); no patient got PEWS be-
tween group 0 to 2 and 6 to 9. The interrater
reliability of PEWS was found to be excellent:
ICC = 0.91 (0.87, 0.95).
In the intrarater tests, 52.8% of patients
(28/53) received the same score in 3 evalu-
ations; 45.3% of patients (24/53) received the
same score from only 2 scoring episodes, and
1 patient (1.9%) was given different scores in
3 evaluations. The standard deviation of the
difference between the PEWS and 3-score av-
erage on each patient was 0.47. It was shown
that the largest maximum score difference of
each patient was 3; the mean of the differ-
ences was 0.6. Four patients received mixed
scores across categories 0 to 2 and 3 to 5; 5 pa-
tients received mixed scores over categories 3
to 5 and 6 to 9; and no patient received a PEWS
between group 0 to 2 and 6 to 9. The intrarater
reliability was also found to be excellent: ICC
LWW/JNCQ JNCQ-D-15-00118 February 9, 2016 1:24

164 JOURNAL OF NURSING CARE QUALITY/APRIL-JUNE 2016

= 0.90 (0.85, 0.94). We did not observe a sys-
tematic change of PEWS across raters in inter-
rater evaluation or over time in intrarater eval-
uation (Table 1). The average time spent on
PEWS scoring on inter- and intrarater reliabil-
ity was 58 ± 4 seconds and 79 ± 15 seconds,
respectively.
It was found that patients with a high PEWS
(≥3) had a greater chance of being admitted
to the hospital (odds ratio [OR] = 4.1, P <
.001), being transferred to ICU (P = .026),
and receiving laboratory tests (OR = 4.8,
P = .001). It also was shown that patients
with a high PEWS were more likely to re-
ceive antibiotics and intravenous saline bolus
(OR = 2.1, and OR = 1.8); however, these
differences did not achieve statistical signifi-
cance (P = .176 and P = .272, respectively)
(Table 2).
DISCUSSION
The results show that the modified PEWS
demonstrates significant inter- and intrarater
reliability. The PEWS can be performed
rapidly in the busy ED setting and could be
easily integrated in routine patient evaluation.
Potentially, PEWS can be used for prompt
assessment of illness severity to guide ap-
propriate resource allocation and subsequent
management.
A number of early scoring systems, includ-
ing PEWS, have been widely used to detect
clinical signs of early deterioration and iden-
tify need of pediatric intensive care for vari-
ous morbidities.1,2,5,6 Recent data have sug-
gested a potential role for PEWS in identifi-
cation of early sepsis and is currently being
incorporated in a standard sepsis algorithm at
pediatrics hospitals to trigger rapid response
teams.4,13,17,18 While there have been inves-
tigations of the use of PEWS in the pediatric
emergency setting,3,7-12,16 knowledge about
the effect of PEWS on ED assessment of a pa-
tient’s clinical status is limited and requires
further investigation. To successfully imple-
ment PEWS into an ED-based clinical pathway
for early sepsis identification and subsequent
resource allocation, it is essential to evaluate
its reliability as an assessment instrument.
Reliability is a necessary condition for valid-
ity and refers to the consistency with which
the same information is processed repeatedly,
across different raters, over time, and with mi-
nor variations in the administration and appli-
cation of PEWS. While the goal is to achieve
consistent performance of PEWS on the same
patient at different time points regardless of
provider, this may not always be feasible in
clinical practice. Similar to the findings re-
garding high interrater reliability from Tucker
et al,19 our study found that PEWS has

Table 1. Inter- and Intrarater reliability of PEWS and Differences Among Scores

Interrater, N = 47 (%) Intrarater, N = 53 (%)

ICC (95% CI) 0.91 (0.87-0.95) 0.90 (0.85-0.94)

Maximum difference among 0 39 (83.0) 28 (52.8)

scores for each patient
1 4 (8.5) 20 (37.7)
2 3 (6.4) 2 (3.8)
3 0 (0) 3 (5.7)
4 1 (2.1) 0 (0)
Patients who received scores 0 39 (83.0) 28 (52.8)
in complete disagreement
1 5 (10.6) 24 (45.3)
2 3 (6.4) 1 (1.9)

Abbreviations: CI, confidence interval; ICC, intraclass correlation coefficient.

Copyright © 2016 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
LWW/JNCQ JNCQ-D-15-00118 February 9, 2016 1:24

Feasibility and Reliability of Pediatric Early Warning Score in the Emergency Department 165

Table 2. Relationship Between PEWS and Patients’ Disposition, Treatments, and Tests

PEWS

≤2, N = 59 (%) ≥3, N = 41 (%) OR (95% CI) P

Admitted 13 (22.0) 22 (53.7) 4.1 (1.7-9.8) <.001

Intensive care unit 0 (0) 4 (9.8) N/A .026a
Antibiotics 7 (11.9) 9 (22.0) 2.1 (0.7-6.2) .176
Normal saline bolus 8 (13.6) 9 (22.0) 1.8 (0.6-5.1) .272
At least 1 laboratory test 9 (15.3) 19 (46.3) 4.8 (1.9-12.3) .001
performedb

Abbreviations: N/A, not applicable; CI, confidence interval; OR, odds ratio; PEWS, pediatric early warning score.
a Denotes use of the Fisher-Irwin test.
b Among urinalysis, urine culture, and blood culture.

significant intrarater and interrater reliability.
Our findings provide clinical and statistical ev-
idence that PEWS can be a reliable clinical tool
to identify pediatric patient deterioration. The
variation caused by raters will not affect the
objectivity of the PEWS scoring system.
Furthermore, time spent in PEWS evalua-
tion has not previously been reported when
performed in the pediatric ED settings. Our
findings of 58 and 79 seconds for intra- and
interrater evaluations in the ED are 2 to 3
times higher than the 30 seconds reported by
Monaghan1 in a pediatric ICU. However, in
terms of clinical care in a busy pediatric ED,
less than 90 seconds to perform the modified
PEWS is mitigated by the potential to identify
patients in early sepsis and provide necessary
clinical resources to prevent patients’ deterio-
ration. Furthermore, it is likely that the speed
of performance of PEWS will improve with
experience. Finally, although the time taken
to perform PEWS is longer than previously re-
ported, the prior study was in an ICU while
ours is in the ED, which in itself may explain
the difference in time for PEWS performance.
As a part of the investigation of successful inte-
gration of PEWS in the pediatric ED, time for
nurses is important to understand the com-
plexity versus ease of performing PEWS and
accordingly plan pediatric care in the ED. Our
study results reveal that PEWS can be per-
formed in the ED setting by nurses in an ex-
peditious manner and may potentially be in-
corporated in algorithms designed to identify
severity of illness.
Previous studies have shown that patients
with PEWS of 3 or greater have a higher
chance to be transferred to intensive care
level.4,6,16 Although not powered for this anal-
ysis, patients with high PEWS (≥3) in our co-
hort were more likely to receive laboratory
tests, be admitted to the hospital, and be trans-
ferred to the ICU, suggesting that the PEWS
may be a valid score for severity of illness.
In summary, our study introduces a general
framework for reliability estimation in PEWS
grading. Its high interrater and intrarater re-
liability, ease of evaluation, and relationship
with clinical outcomes provide evidence that
the PEWS can potentially be used in the ED
as an objective and effective tool to identify
children in high risk of deterioration.
Limitations
It is possible that PEWSs vary with progres-
sion of illness during the time spent in the
ED. The scores on intrarater reliability might
be affected by a change in the patient’s clin-
ical condition at different time points when
the PEWS was scored for a single patient. We
tried to limit the impact from illness progres-
sion by requiring nurses to perform repeated
scores within 15 to 30 minutes of proceed-
ing PEWS evaluation. Furthermore, patients

Copyright © 2016 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
LWW/JNCQ JNCQ-D-15-00118 February 9, 2016 1:24
166 JOURNAL OF NURSING CARE QUALITY/APRIL-JUNE 2016
with acute life-threatening illness were not
included. Therefore, the number of patients
with a high PEWS (6-9) was relatively small
and integrated with PEWS category 3 to 5 for
the purpose of analysis on clinical outcomes.
In addition, the statistical power to examine
the relationship among PEWS and patients’
clinical outcomes was low. More studies with
a larger sample size will be required to reveal
that relationship.
REFERENCES
1. Monaghan A. Detecting and managing deterioration
in children. Paediatr Nurs. 2005;17(1):32-35.
2. Duncan H, Hutchison J, Parshuram CS. The Pediatric
Early Warning System score: a severity of illness score
to predict urgent medical need in hospitalized chil-
dren. J Crit Care. 2006;21(3):271-278.
3. Egdell P, Finlay L, Pedley DK. The PAWS score: valida-
tion of an early warning scoring system for the initial
assessment of children in the emergency department.
Emerg Med J. 2008;25(11):745-749.
4. Akre M, Finkelstein M, Erickson M, Liu M, Vanderbilt
L, Billman G. Sensitivity of the pediatric early warn-
ing score to identify patient deterioration. Pediatrics.
2010;125(4):e763-e769.
5. Parshuram CS, Bayliss A, Reimer J, Middaugh K, Blan-
chard N. Implementing the Bedside Paediatric Early
Warning System in a community hospital: a prospec-
tive observational study. Paediatr Child Health.
2011;16(3):e18-e22.
6. Skaletzky SM, Raszynski A, Totapally BR. Validation of
a modified pediatric early warning system score: a ret-
rospective case-control study. Clin Pediatr (Phila).
2012;51(5):431-435.
7. Sefton G, McGrath C, Tume L, Lane S, Lisboa PJ, Carrol
ED. What impact did a Paediatric Early Warning sys-
tem have on emergency admissions to the paediatric
intensive care unit? An observational cohort study.
Intensive Crit Care Nurs. 2015;31(2):91-99.
8. Breslin K, Marx J, Hoffman H, McBeth R, Pavuluri P.
Pediatric early warning score at time of emergency de-
partment disposition is associated with level of care.
Pediatr Emerg Care. 2014;30(2):97-103.
9. Gold DL, Mihalov LK, Cohen DM. Evaluating
the Pediatric Early Warning Score (PEWS) sys-
tem for admitted patients in the pediatric emer-
gency department. Acad Emerg Med. 2014;21(11):
1249-1256.
10. Nielsen KR, Migita R, Batra M, Gennaro JL, Roberts JS,
Weiss NS. Identifying high-risk children in the emer-
Copyright © 2016 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
CONCLUSIONS
The PEWS demonstrates high inter- and in-
trarater reliability. It can be performed rapidly
(>90 seconds) in a busy ED setting. If inte-
grated with routine clinical evaluation in the
pediatric ED, it may potentially improve pa-
tients’ outcomes through the early identifi-
cation and initiation of appropriate interven-
tions to prevent patients’ deterioration.
gency department [February 10, 2015]. J Intensive
Care Med doi: 10.1177/0885066615571893.
11. Chaiyakulsil C, Pandee U. Validation of pediatric early
warning score in pediatric emergency department.
Pediatr Int. 2015;57(4):694-698.
12. Seiger N, Maconochie I, Oostenbrink R, Moll
HA. Validity of different pediatric early warning
scores in the emergency department. Pediatrics.
2013;132(4):e841-e850.
13. Bonafide CP, Localio AR, Roberts KE, Nadkarni VM,
Weirich CM, Keren R. Impact of rapid response sys-
tem implementation on critical deterioration events
in children. JAMA Pediatr. 2014;168(1):25-33.
14. Watson A, Skipper C, Steury R, Walsh H, Levin A. Inpa-
tient nursing care and early warning scores: a work-
flow mismatch. J Nurs Care Qual. 2014;29(3):215-
222.
15. Jaimes F, Garcés J, Cuervo J, et al. The systemic inflam-
matory response syndrome (SIRS) to identify infected
patients in the emergency room. Intensive Care Med.
2003;29(8):1368-1371.
16. Solevåg AL, Eggen EH, Schröder J, Nakstad B. Use
of a modified pediatric early warning score in a de-
partment of pediatric and adolescent medicine. PLoS
One. 2013;8(8):e72534.
17. Zhai H, Brady P, Li Q, et al. Developing and eval-
uating a machine learning based algorithm to pre-
dict the need of pediatric intensive care unit trans-
fer for newly hospitalized children. Resuscitation.
2014;85(8):1065-1071.
18. Panesar R, Polikoff LA, Harris D, Mills B, Messina C,
Parker MM. Characteristics and outcomes of pediatric
rapid response teams before and after mandatory trig-
gering by an elevated Pediatric Early Warning System
(PEWS) score. Hosp Pediatr. 2014;4(3):135-140.
19. Tucker KM, Brewer TL, Baker RB, Demeritt B, Voss-
meyer MT. Prospective evaluation of a pediatric in-
patient early warning scoring system. J Spec Pediatr
Nurs. 2009;14(2):79-85.