Research Report
Using Resident-Sensitive Quality Measures
Derived From Electronic Health Record Data
to Assess Residents’ Performance in Pediatric
Emergency Medicine
Alina Smirnova, MD, PhD, Saad Chahine, PhD, MEd, Christina Milani, MSc,
Abigail Schuh, MD, MMHPE, Stefanie S. Sebok-Syer, PhD, Jordan L. Swartz, MD, MA,
Jeffrey A. Wilhite, MPH, Adina Kalet, MD, MPH, Steven J. Durning, MD, PhD,
Downloaded from http://journals.lww.com/academicmedicine by BhDMf5ePHKav1zEoum1tQfN4a+kJLhEZgbsIHo4X
Kiki M.J.M.H. Lombarts, PhD, MSc, Cees P.M. van der Vleuten, PhD, and
Daniel J. Schumacher, MD, PhD, MEd
Mi0hCywCX1AWnYQp/IlQrHD3i3D0OdRyi7TvSFl4Cf3VC4/OAVpDDa8K2+Ya6H515kE= on 04/18/2024
Abstract
Purpose
Traditional quality metrics do not
adequately represent the clinical work
done by residents and, thus, cannot
be used to link residency training to
health care quality. This study aimed to
determine whether electronic health
record (EHR) data can be used to
meaningfully assess residents’ clinical
performance in pediatric emergency
medicine using resident-sensitive quality
measures (RSQMs).
Method
EHR data for asthma and bronchiolitis
RSQMs from Cincinnati Children’s
Hospital Medical Center, a quaternary
children’s hospital, between July 1, 2017,
and June 30, 2019, were analyzed by
ranking residents based on composite
scores calculated using raw, unadjusted,
A ligning learner and patient
outcomes is a fundamental
component of competency-based
medical education (CBME). Without
Please see the end of this article for information
about the authors.
Correspondence should be addressed to
Alina Smirnova, Office of Health and Medical
Education Scholarship, University of Calgary,
Calgary, AB T2R 0X7, Canada; email:
alina.smirnova1@ucalgary.ca.
Written work prepared by employees of the Federal
Government as part of their official duties is, under
the U.S. Copyright Act, a “work of the United States
Government” for which copyright protection under
Title 17 of the United States Code is not available. As
such, copyright does not extend to the contributions
of employees of the Federal Government.
Acad Med. 2023;98:367–375.
First published online November 8, 2022
doi: 10.1097/ACM.0000000000005084
Supplemental digital content for this article is
available at http://links.lww.com/ACADMED/B356.
Academic Medicine, Vol. 98, No. 3 / March 2023
and case-mix adjusted latent score
models, with lower percentiles indicating
a lower quality of care and performance.
Reliability and associations between
the scores produced by the 3 scoring
models were compared. Resident and
patient characteristics associated with
performance in the highest and lowest
tertiles and changes in residents’ rank
after case-mix adjustments were also
identified.
Results
274 residents and 1,891 individual
encounters of bronchiolitis patients
aged 0–1 as well as 270 residents
and 1,752 individual encounters of
asthmatic patients aged 2–21 were
included in the analysis. The minimum
reliability requirement to create a
composite score was met for asthma
understanding how residency training
is linked to quality of care, CBME
cannot achieve its mandate of ensuring
that training prepares graduates to
provide the best possible care to
populations of patients. 1 While several
large studies have demonstrated
that where a physician completes
residency training predicts future
clinical performance 2–8 and others
have highlighted the critical role that
residents play in ensuring patient
care quality during training, 9,10 the
use of clinical performance metrics in
graduate medical education (GME)
remains limited. Some have proposed
a “big data” approach to elucidate the
relationship between education and
health care processes and outcomes,
which uses electronic health record
(EHR) data for quality metrics. 11,12
EHR data have previously been used
data (α = 0.77), but not bronchiolitis
(α = 0.17). The asthma composite
scores showed high correlations (r =
0.90–0.99) between raw, latent, and
adjusted composite scores. After case-
mix adjustments, residents’ absolute
percentile rank shifted on average 10
percentiles. Residents who dropped by
10 or more percentiles were likely to be
more junior, saw fewer patients, cared
for less acute and younger patients, or
had patients with a longer emergency
department stay.
Conclusions
For some clinical areas, it is possible
to use EHR data, adjusted for patient
complexity, to meaningfully assess
residents’ clinical performance and
identify opportunities for quality
improvement.
to determine resident experience (e.g.,
conditions seen), 13 and obtaining
resident performance quality metrics
would be a logical next step. The
success of this approach depends
on the availability of metrics with
reliability and validity evidence. 14
Traditional quality metrics do not
adequately represent the clinical work
performed by residents, and, therefore,
cannot be used to link residency
training to health care quality. 14–20
For instance, mortality rates are often
multifactorial and usually cannot be
attributed to a single clinician. Resident-
sensitive quality measures (RSQMs)
attempt to address this gap. RSQMs
are clinical care measures that are both
important to providing care for an
illness of interest, and likely completed
by a resident rather than another
367
 Research Report
member of the team or by the team
collectively. 19,21 RSQMs in pediatric
emergency medicine, developed in
consultation with supervisors and
residents, 19,21 have demonstrated a wide
range of resident performance on both
individual and composite measures of
asthma, bronchiolitis, and closed head
injury, 22 in relation to other variables
Downloaded from http://journals.lww.com/academicmedicine by BhDMf5ePHKav1zEoum1tQfN4a+kJLhEZgbsIHo4X
such as patient complexity and acuity, 23
Mi0hCywCX1AWnYQp/IlQrHD3i3D0OdRyi7TvSFl4Cf3VC4/OAVpDDa8K2+Ya6H515kE= on 04/18/2024
and for potential use in summative
assessments. 24 For RSQMs to fulfill their
potential for widespread use in GME,
they must not only be relatively easily
extractable from the EHR, but they must
also demonstrate reliability and provide
validity evidence.
The aim of this study was to
examine whether RSQMs that are
easily extractable from the EHR
can meaningfully assess residents’
performance. In this proof-of-concept
study, we assessed the feasibility of
systematically collecting resident-
specific performance measures from
the EHR and evaluated the validity
of these measures using the first 3
steps in Kane and Messick’s validity
frameworks: scoring (supported
by Messick’s response process),
generalization (supported by Messick’s
internal structure and content), and
extrapolation (supported by Messick’s
relationship to other variables). 25
We did not examine the use of these
measures for decision supports, but
rather to highlight the ability to develop
a metric of performance through
the use of automatically extractable
clinical performance measures, without
the need for chart review, which is
essential for applying these measure
on a larger scale. We tested whether
EHR data could be used to create
RSQM composite scores that reliably
discriminate high- and low-performing
residents in management of 2 common
respiratory diseases, bronchiolitis and
asthma, in the pediatric emergency
department (PED). Additionally, we
aimed to understand which resident,
faculty, and patient characteristics were
associated with residents’ performance
on these RSQMs. Providing evidence of
validity and reliability for RSQMs that
are easily extractable from the
EHR can help provide a baseline
for future replication studies and
comparison between sites and,
ultimately, support their wider
application in CBME.
368
Method
Setting and participants
Bronchiolitis and asthma are 2 of the
most common reasons for visits to PEDs,
where residents have a high degree
of autonomy in ordering treatments
before the attending sees the patient. 26
Their frequency and evidenced-based
standardized care protocol make these
chief complaints ideal for examining
what and how residents do when caring
for patients with these illnesses. We did
not include closed head injury RSQMs
because they were less automatically
extractable from the EHR. Any PED
encounters with billing diagnoses of
bronchiolitis or asthma, where a resident
was assigned to the patient and completed
the clinical note, were extracted from the
EHR (Epic Systems, Verona, Wisconsin)
at Cincinnati Children’s Hospital Medical
Center (CCHMC) between July 1, 2017,
and June 30, 2019. The deidentified
datasets contained information about
patient and resident characteristics,
as well as the supervising faculty of
record for the encounter. The full data
dictionary is available in Supplemental
Digital Appendix 1, at http://links.lww.
com/ACADMED/B356. CCHMC is
a quaternary children’s hospital with
approximately 60,000 PED visits annually.
Between 2 and 7 residents usually
staff the department at any given time.
The majority are categorical pediatric
residents, who spend 4 months in the
PED during their 3-year residency, and
emergency medicine residents, who
complete 6 months during a 4-year
residency. Residents from pediatric
combined training programs such as
medicine–pediatrics (4–5 years) and
family medicine (3 years) also rotate
through the PED. We obtained CCHMC
institutional review board approval
before data extraction and analysis.
The study findings were reported using
the CONSORT extension to pilot and
feasibility trials. 27
Data and measures
The analysis focused on easily extractable
RSQMs for each illness of interest.
Using composite scores based on several
individual measures compensates for
the potential lack of variability in single
performance measures. 28–31 While
RSQM composite scores based on 19–23
individual measures previously have
shown good variability, 22 only 5 RSQMs
were readily extractable from the EHR for
Academic Medicine, Vol. 98, No. 3 / March 2023
each condition we studied, which were
ultimately included in the analysis. We
focused on RSQMs only available from
the EHR since the aim of the study was
to develop a set of measures that could be
realistically replicated at other institutions
(scalability).
Patients aged 0–1 year were included
in the bronchiolitis dataset. Patients
aged 2–21 years were included in the
asthma dataset. The patient age ranges
were chosen because bronchiolitis
becomes less common over the age of
1 year and a formal asthma diagnosis
is less common under 2 years of age.
Additionally, we included several relevant
patient, resident, and supervising faculty
characteristics, described below.
Methodological approach
To provide evidence of scoring, we used
a psychometric approach to determine
appropriateness of individual items for
the composite score for each condition.
First, we calculated the proportion
of patients who received appropriate
care at the item level, with ideal values
being ~0.50 and a range of 0.30–0.70
deemed generally acceptable. 32 We also
calculated the internal consistency of
each composite score, with a range
of 0.7–1 indicating that items may be
grouped together to provide an overall
competence in the care of bronchiolitis or
asthma evaluation. 32
Once individual items passed the initial
screening, we used 3 different models
to create the composite scores: a raw
score model (model 1), an unadjusted
latent score model (model 2), and an
adjusted latent score model (model 3).
In model 1, each indicator was given
equal weight, and each contributed to
the overall score uniformly. In models 2
and 3, we used hierarchical generalized
linear models to provide differential
weights for each indicator. 33 In addition
to valuing more difficult items compared
with easier items, this approach can take
into account various patient and resident
characteristics. Model 2 was unadjusted,
while model 3 tested several resident
and patient characteristics as covariates.
For model 3, the following statistically
significant (P < .05) characteristics were
included: patient age, PED length of
stay, initial placement in the medical
resuscitation bay, and year presented
in the PED (see Supplemental Digital
 Research Report

Appendix 2, at http://links.lww.com/
ACADMED/B356). In all models,
indicators were nested within patients and
patients within residents. We standardized
the scores to have a mean of 50 and
standard deviation of 10 to facilitate
interpretation and comparability.
In models 2 and 3, we also assessed the
Downloaded from http://journals.lww.com/academicmedicine by BhDMf5ePHKav1zEoum1tQfN4a+kJLhEZgbsIHo4X
percentiles indicate a lower quality
of care and, therefore, lower level of
performance. Shifts in rankings were
then evaluated using correlations
and comparative statistical testing.
Additionally, we examined which
residents experienced the greatest shift
in ranking by investigating which patient
encounter characteristics were associated
own performance on the relevant RSQM
calculated based on patient encounters
seen by faculty within the same study
period without a resident and mean
patient panel size.
For descriptive and comparative analysis,
we used SPSS statistical software, version
26 (SPSS, Inc., Chicago, Illinois), and

difficulty of each item comprising the with at least a 10-point increase or SAS statistical software, version 9.4 (SAS
composite score as a final check to ensure
Mi0hCywCX1AWnYQp/IlQrHD3i3D0OdRyi7TvSFl4Cf3VC4/OAVpDDa8K2+Ya6H515kE= on 04/18/2024

decrease in RSQM percentile scores. Institute, Cary, North Carolina) for

each item significantly contributed to the
latent composite scores. Item difficulty
was judged based on the assumption that
we were indirectly measuring a “latent”
ability estimate (theta) of a person who
would be considered average (50%
of residents) in providing the correct
treatment. The estimates range from −3
to +3. An item that would be equally
difficult or easy for an average ability
person would have a value of 0. Negative
values indicate more difficult items.
To provide evidence of generalization
within each model, we used composite
scores to rank residents, where lower
To provide evidence of extrapolation, we
explored the relationships of residents’
performance in the top and bottom
tertiles with several patient, resident,
and faculty-of-record characteristics
using t tests and chi-square tests. Patient
characteristics were mean age, PED
length of stay, whether they presented
to a medical resuscitation bay compared
with regular PED room initially, and
time of presentation to the PED. Resident
characteristics were postgraduate training
year, sex, program type, and mean
patient panel size. Supervising faculty-
of-record characteristics were faculty’s
multilevel modeling. Models were then
confirmed in HLM software, version 8.0
(Scientific Software International, Inc.,
Chapel Hill, North Carolina) and latent
scores (models 2 and 3) were generated.
There were no missing data.
Results
Asthma RSQMs
Of the 349 total residents, 270 (77%)
treated at least one patient with an
asthma exacerbation in the PED during
the study period, amounting to 1,752
encounters (Table 1). Overall resident

Table 1
Demographic Characteristics of Residents and Patients, From a Study of
Bronchiolitis and Asthma RSQMs for Pediatric Emergency Department Care,
Cincinnati Children’s Hospital Medical Center, Cincinnati, Ohio, July 2017–June 2019

Bronchiolitis dataset Asthma dataset

Characteristic No. Metric (95% CI) No. Metric (95% CI)
Resident
Mean composite score percentile 274 82.64 (81.6, 83.6) 270 50.0 (46.6, 53.4)
% PGY-1 120 43.8 (38.1, 49.7) 124 45.9 (39.9, 52.1)
% PGY-2 71 25.9 (21.1, 31.4) 69 69 (20.5, 31.2)
% PGY-3 61 22.3 (17.7, 27.6) 56 20.7 (16.1, 26.1)
% PGY-4 22 8.0 (5.4, 11.9) 20 7.4 (4.6, 11.2)
% Female gender 161 58.8 (52.9, 64.4) 158 58.5 (52.4, 64.5)
% Pediatric training program 197 71.9 (66.3, 76.9) 195 72.2 (66.5, 77.5)
% Emergency medicine training program 67 24.5 (19.7, 29.9) 66 24.4 (19.4, 30.0)
Mean patient panel size 274 6.9 (6.3, 7.5) 270 6.49 (6.0, 7.0)
Supervising faculty
Mean composite score percentile 51 82.5 (80.1, 84.9) 52 50.0 (46.6, 53.4)
Mean patient panel size 55 34.4 (28.5, 40.3) 52 48.7 (45.8, 51.6)
Patient
Mean age 1,891 0.3 (0.3, 0.3) 1,752 7.7 (7.4, 8.0)
Mean length of PED stay 1,891 145.3 (133.8, 156.8) 1,752 156.2 (149.7, 162.7)
% Presented to medical resuscitation bay initially 1,891 19.3 (17.6, 21.1) 1,752 15.4 (13.2, 16.5)
% Admitted 1,891 63.6 (61.4, 65.8) 1,752 41.1 (37.8, 42.7)
% Patients seen during the day (08:00–15:59) 773 40.9 (38.7, 43.1) 529 30.2 (28.1, 32.4)
% Patients seen during the evening (16:00–23:59) 719 38.0 (35.9, 40.2) 688 39.3 (37.0, 41.6)
% Patients seen overnight (0:00–7:59) 399 21.1 (19.3, 23.0) 535 30.5 (28.4, 32.7)
Abbreviations: RSQM, resident-sensitive quality measure; CI, confidence interval; PGY, postgraduate year; PED,
pediatric emergency department.

Academic Medicine, Vol. 98, No. 3 / March 2023 369

 Research Report

performance on asthma RSQMs is
reported in Table 2. Internal consistency
of treatment of asthma was good (α =
0.77). Table 2 reports the proportion of
patients receiving the correct treatment
for each item in addition to item difficulty
for models 2 and 3. Based on these
results, it was psychometrically justifiable
to combine the individual asthma
Downloaded from http://journals.lww.com/academicmedicine by BhDMf5ePHKav1zEoum1tQfN4a+kJLhEZgbsIHo4X
type of residency program, patient panel
size, or other patient characteristics. The
unadjusted latent score model (model
2) produced almost identical results to
model 1, with correlation values of r =
0.99 for the standardized scores. The
tertile comparison between ranking
of residents also showed very little
difference between the 2 models.
scores were in their second, third, or
fourth postgraduate year in July 2017,
while residents in their first training year
decreased in their ranking.
Bronchiolitis RSQMs
Of the 349 total residents, 274 residents
(79%) treated patients with bronchiolitis,
amounting to 1,891 encounters over

indicators into a composite score. 2 years (Table 1). Overall resident

Mi0hCywCX1AWnYQp/IlQrHD3i3D0OdRyi7TvSFl4Cf3VC4/OAVpDDa8K2+Ya6H515kE= on 04/18/2024

Given the almost negligible difference performance on bronchiolitis RSQMs

The unstandardized raw score model
(model 1) produced a normal distribution
(n = 270; m = 0.47; 95% confidence
interval [CI] = 0.45, 0.49). Table 3 shows
characteristics associated with ranking in
the highest or lowest tertile, with the last
column indicating significant differences.
For example, residents ranking in the
top tertile were significantly more likely
to be in their first year of training or to
be supervised by faculty who themselves
scored higher on asthma RSQMs.
Conversely, residents in the bottom
tertile were significantly more likely to
be in their second year of training or
have had more encounters with patients
who initially presented to the medical
resuscitation bay. No differences were
found in ranking between third and
fourth postgraduate years, resident sex,
and the ease of calculating RSQM
composite scores in model 1, we opted
to compare model 1 with the adjusted
latent model (model 3). Although the
correlation between models 1 and 3
was extremely high (r = 0.92), there
was a noticeable difference in the
tertile groupings between the 2 models
(Table 4). Compared with model 1, model
3 resulted in higher RSQM percentile
scores for residents treating more acute
patients, characterized by the initial
presentation to the medical resuscitation
bay, and penalized those who were
treating fewer acute patients (Table 5).
On average, residents shifted 10.08
percentiles after patient characteristics
were taken into account. As highlighted
in Table 5, residents who increased at
least 10 points on their RSQM percentile
is reported in Table 2. The internal
consistency of bronchiolitis treatment
was very poor (α = 0.17). From a
psychometric perspective, the items
are considered too easy (i.e., the
care was uniformly good), with an
average performance of the individual
bronchiolitis RSQMs of 82.4%. Only
half of patients had “nasal bulb suction
teaching ordered,” making this the only
indicator able to distinguish among
residents. Based on these results, a
composite bronchiolitis RSQM score
could not reliably differentiate higher-
skilled residents from lower-skilled ones.
Discussion
This is the first study to provide
evidence for scoring, generalizability,

Table 2
Item Scoring of RSQMs and Characteristics of Individual RSQMs Included in
Composite Scores, From a Study of Bronchiolitis and Asthma RSQMs for Pediatric
Emergency Department Care, Cincinnati Children’s Hospital Medical Center,
Cincinnati, Ohio, July 2017–June 2019

Proportion (95% CI) of Item difficulty Item difficulty

patients receiving (model 2, (model 3,
appropriate treatment unadjusted unadjusted
Condition No. provided by residents latent) latent)
Bronchiolitis (maximum score = 5) 1,891
No antibiotics ordered (1 point) 1,773 93.8 (92.6, 94.8) — —
No albuterol ordered (1 point) 1,677 88.7 (87.2, 90.0) — —
No steroid ordered (1 point) 1,818 96.1 (95.2, 97.9) — —
No chest X-ray ordered (1 point) 1,560 82.5 (80.7, 84.2) — —
 Nasal bulb suction teaching ordered (1 962 50.8 (48.6, 53.1) — —
point)
Average score ~1,559 82.4 (80.6, 84.1) — —
Asthma (maximum score = 5) 1,752
Albuterol ordered (1 point) 890 50.8 (48.5, 53.1) 1.59a 1.20a
Correct dose of albuterol ordered (1 point) 982 56.1 (53.7, 58.4) 1.88 a
1.82a
Dexamethasone steroid ordered (1 point) 834 47.6 (45.3, 49.9) 1.42 a
1.52a
 Correct dose of dexamethasone ordered 731 41.7 (39.4, 44.0) 1.10 a
1.05a
(1 point)
Asthma order set used (1 point) 628 35.8 (33.6, 38.1) −0.77a −0.64a
Average score ~781 44.6 (42.3, 46.9) — —
Abbreviations: RSQM, resident-sensitive quality measure; CI, confidence interval.
a
Items that significantly contribute to the latent composite score (P < .001).

370 Academic Medicine, Vol. 98, No. 3 / March 2023

 Research Report

Table 3
Resident, Faculty, and Patient Characteristics Associated With Residents Being
Ranked in the Top or Bottom Tertile Based on Asthma Raw Composite Score,
From a Study of Bronchiolitis and Asthma RSQMs for Pediatric Emergency
Department Care, Cincinnati Children’s Hospital Medical Center, Cincinnati, Ohio,
July 2017–June 2019

Asthma RSQM raw composite score

Downloaded from http://journals.lww.com/academicmedicine by BhDMf5ePHKav1zEoum1tQfN4a+kJLhEZgbsIHo4X

Top tertile Bottom tertile

% or Mean % or Mean
Mi0hCywCX1AWnYQp/IlQrHD3i3D0OdRyi7TvSFl4Cf3VC4/OAVpDDa8K2+Ya6H515kE= on 04/18/2024

Characteristic No. (95% CI) No. (95% CI) P value

Resident characteristics (n = 270)
Mean composite score percentilea 89 83.0 (46.6, 53.4) 83 16.0 (14.1, 17.9) < .001
% PGY1b 46 51.7 (41.5, 61.8) 27 32.5 (23.4, 43.2) .011
% PGY2b 18 20.2 (13.2, 29.7) 30 36.1 (26.6, 46.9) .02
% PGY3b 17 19.1 (12.3, 28.5) 18 21.7 (14.2, 31.7) .674
% PGY4b 7 7.9 (3.9, 15.4) 8 9.6 (5.0, 17.9) .68
% Female sexb 50 56.2 (45.8, 66.0) 47 56.6 (45.9, 66.8) .953
% Pediatric training programb 62 69.7 (59.5, 78.2) 60 72.2 (61.8, 80.8) .705
% Emergency medicine training programb 24 27.0 (18.8, 37.0) 20 24.1 (16.2, 34.3) .666
Mean patient panel sizec — 6.2 (5.3, 7.1) — 6.7 (5.7, 7.7) .453
Supervising faculty characteristics (n = 52)
Mean composite score percentilea — 62.3 (56.8, 67.8) — 35.6 (30.0, 41.2) < .001
Mean patient panel sizea — 47.7 (42.6, 52.8) — 50.8 (45.7, 55.9) .403
Patient characteristics (n = 1,752)
Mean agea — 7.3 (6.9, 7.8) — 7.8 (7.2, 8.5) .229
Mean PED length of staya — 165.3 (148.7, 181.8) — 150.4 (141.6, 159.2) .129
% Presented to medical resuscitation bay initiallyc — 7.1 (4.9, 9.4) — 23.8 (18.7, 28.9) < .001
% Admittedc — 37.7 (31.7, 43.6) — 42.9 (37.0, 48.8) .162
% Patients seen during the day (08:00–15:59)c — 31.0 (25.2, 36.7) — 30.8 (24.8, 36.8) .850
% Patients seen during the evening (16:00–23:59)c — 39.5 (33.3, 45.7) — 37.4 (32.3, 42.6) .739
% Patients seen overnight (24:00–7:59)c — 29.5 (23.8, 35.3) — 31.8 (26.2, 37.3) .457
Abbreviations: RSQM, resident-sensitive quality measure; CI, confidence interval; PGY, postgraduate year; PED,
pediatric emergency department.
a
t test.
b
Chi-square test.
c
Mann-Whitney U test.

care that is attributable to the actions of

Table 4
Number of Residents Grouped in Tertiles Using Adjusted Asthma Composite Score
Compared With Raw Composite Score, From a Study of Bronchiolitis and Asthma
RSQMs for Pediatric Emergency Department Care, Cincinnati Children’s Hospital
Medical Center, Cincinnati, Ohio, July 2017–June 2019
Tertile based on adjusted
composite asthma RSQM score,
Tertile based on raw composite RSQM
no. (%) residents
score 1 (n = 90) 2 (n = 90) 3 (n = 90)
1 (n = 89) 73 (81.1) 16 (17.8)
2 (n = 98) 15 (16.7) 61 (67.8) 22 (24.4)
3 (n = 83) 2 (2.2) 13 (14.4) 68 (75.6)
Abbreviation: RSQM, resident-sensitive quality measure.
and extrapolation of available RSQMs not bronchiolitis. Overall, the 5-item
from the EHR for 2 common pediatric asthma composite RSQM scores
conditions, to our knowledge. Scoring could discriminate between high and
and generalizability evidence favors low performers, suggesting there is a
asthma RSQM composite scores, but proportion of the variation in patient
residents making decisions about care.
This discrimination improves when data
are adjusted for patient acuity, age, and
academic year, favoring the adjusted
model. Extrapolation evidence for the
asthma composite RSQM scores indicates
associations with resident postgraduate
year, faculty performance, and patient
acuity where first-year residents are
0 overrepresented in the top tertile and
second-year residents overrepresented
in the bottom tertile; better faculty
performance is associated with better
resident performance; and poor resident
performance is associated with higher
patient acuity. On the other hand, the
5-item bronchiolitis composite RSQM
scores showed low variability with
consistently high resident performance.
Thus, from a psychometric perspective,

Academic Medicine, Vol. 98, No. 3 / March 2023 371

 Research Report

Table 5
Resident, Faculty, and Patient Characteristics Associated With an Absolute Rank
Difference of at Least 10 Percentiles After Adjustment, From a Study of
Bronchiolitis and Asthma RSQMs for Pediatric Emergency Department Care,
Cincinnati Children’s Hospital Medical Center, Cincinnati, Ohio, July 2017–June
2019

Based on minimum 10 percentile change from raw to

adjusted score
Downloaded from http://journals.lww.com/academicmedicine by BhDMf5ePHKav1zEoum1tQfN4a+kJLhEZgbsIHo4X

>10 Percentile increase >10 Percentile decrease

Mi0hCywCX1AWnYQp/IlQrHD3i3D0OdRyi7TvSFl4Cf3VC4/OAVpDDa8K2+Ya6H515kE= on 04/18/2024

Characteristic No. % or mean (95% CI) No. % or mean (95% CI) P value
Resident characteristics (n = 270)
Mean composite score percentilea 55 39.9 (34.2, 45.5) 56 57.4 (52.5, 62.3) < .001
Mean adjusted latent score percentilea 55 59.8 (54.4, 65.3) 56 39.5 (34.4, 44.6) < .001
% PGY1b 11 20.0 (11.5, 32.4) 40 71.4 (58.5, 81.6) < .001
% PGY2b 23 41.8 (29.7, 55.0) 10 17.9 (10.0, 29.8) .006
% PGY3b 13 23.6 (14.4, 36.4) 6 10.7 (5.0, 21.5) .043
% PGY4 b
7 12.7(6.3, 24.0) 0 0 (0.0, 0.1)c
.006
% Female sexb 30 54.6 (41.5, 67.0) 32 57.1 (44.1, 69.2) .783
% Pediatric training programb 36 65.5 (52.3, 76.6) 43 76.8 (64.2, 85.9) .118
% Emergency medicine training programb 18 32.7 (21.8, 45.9) 11 19.6 (11.3, 31.8) .666
Mean patient panel sized 55 7.7 (6.4, 9.1) 56 5.4 (4.5, 6.3) .007
Supervising faculty characteristics (n = 52)
Mean composite score percentilesa — 50.5 (43.4, 57.5) — 50.2 (43.0, 57.4) .96
Mean patient panel sizea — 47.5 (40.6, 54.3) — 48.2 (41.6, 54.9) .876
Patient characteristics (n = 1,752)
Mean agea — 8.1 (7.5, 8.8) — 7.0 (6.6, 7.5) .009
Mean length of PED staya — 142.6 (134.4, 150.8) — 170.3 (160.2, 180.5) < .001
% Presented to the medical resuscitation bay on initial arrivald — 37.0 (32.9, 41.0) — 1.1 (0.1, 2.1) < .001
% Admittedd — 46.6 (40.4, 52.8) — 38.4 (31.0, 45.8) .075
% Patients seen during the day (08:00–15:59) (n = 529)d — 30.8 (24.2, 36.4) — 28.0 (21.7, 34.2) .401
% Patients seen during the evening (16:00–23:59) (n = 688)d — 40.0 (34.3, 45.5) — 42.6 (35.6, 49.5) .724
% Patients seen overnight (24:00–7:59) (n = 525) d
— 29.3 (23.4, 35.2) — 29.5 (22.8, 63.2) .762
Abbreviations: RSQM, resident-sensitive quality measure; CI, confidence interval; PED, pediatric emergency
department.
a
t test.
b
Chi-square test.
c
97.5% confidence interval.
d
Mann-Whitney U test.

it poorly discriminates amongst
residents’ performance. By providing
baseline scoring, generalizability, and
extrapolation evidence for future
replication studies, this study facilitates
the wider application of EHR data to
provide individualized, performance-
based information for residents and
programs. 16
In contrast to asthma care, residents
consistently provided high-quality care
to patients with bronchiolitis on the
chosen measures. Although there was
no QI initiative around bronchiolitis,
it is possible that residents performed
better on bronchiolitis measures because
the standard of care for bronchiolitis
is far simpler and less dependent on
clinical context compared with asthma
exacerbations. Although this finding may
not be desirable from a psychometric
perspective, it is important from a
mastery perspective. While consistently
high performance on bronchiolitis
RSQMs does not allow for creation
of composite scores or performance
ranking, such measures can be used to
identify outliers. It would be meaningful
for training programs to know if a
resident underperforms compared
with their peers since, from a patient
care perspective, any substandard
performance by a resident is, by
definition, a compromise of patient
care. Such measures can be considered
a baseline clinical benchmark and
significant signal, for all residents should
consistently provide high-quality patient
care based on the selected measures. On
the other hand, asthma RSQM composite
scores showed more variability in resident
performance, and therefore, may be more
useful for differentiating between the
performance of individual residents. This
finding raises an important question,
whether (or which) RSQMs could be used
as both criterion-based and performance
differentiating measures in the context of
a CBME program of assessment.
The finding that higher-performing
residents were more likely to be
supervised by higher-performing faculty

372 Academic Medicine, Vol. 98, No. 3 / March 2023


may represent the interdependence
between residents and faculty,
especially earlier in residency training. 34
Interestingly, this relationship did
not hold once patient characteristics
were taken into account. Thus, patient
complexity may explain the performance
of both faculty and residents. While
this study cannot prove causation, this
Downloaded from http://journals.lww.com/academicmedicine by BhDMf5ePHKav1zEoum1tQfN4a+kJLhEZgbsIHo4X
finding may be a further indication that
Mi0hCywCX1AWnYQp/IlQrHD3i3D0OdRyi7TvSFl4Cf3VC4/OAVpDDa8K2+Ya6H515kE= on 04/18/2024
individual practice style may be shaped
during residency training in a manner
that imprints future practice patterns for
decades to come. 2,5
We found that residents in the beginning
of their training were more likely to
score higher on asthma RSQMs, which
goes against the logic that performance
improves with experience. 25 This finding
could be explained by more junior
residents being more likely to follow
standards of care provided to them
because they do not yet have their own
care style and preferences. It could also be
due to greater interdependence between
faculty and first-year residents, in which
less experienced residents are more
likely to review orders with their faculty
before placing them in the EHR. 35 As
residents advance in their training and
gain more independence, they are also
given more responsibility and harder
tasks, leading to a lower scores because
the task (i.e., the requisite patient care)
is harder. 36 In this study, the apparent
overrepresentation of first-year residents
in the top tertile seems to be accounted
for by seeing fewer patients overall,
seeing fewer acute patients, or potentially
spending more time on patient care.
In addition, residents in the second
year of their training performed worse
on asthma RSQMs, but their ranking
improved on average when higher
patient acuity and larger patient panel
sizes were taken into account. This may
have implications for the number and
acuity of patients whom residents should
be allowed to manage to ensure they
are positioned to provide high-quality
care. In an earlier study, Schumacher
and colleagues found higher patient
acuity and complexity to be associated
with a higher RSQM composite score
for both asthma and bronchiolitis after
controlling for postgraduate year. 23 Our
current study builds upon these findings
using a more objective measure of
acuity (i.e., presentation to the medical
resuscitation bay on arrival rather than
Academic Medicine, Vol. 98, No. 3 / March 2023
faculty report). This also resonates with a
previous study of primary care physicians
that found adjustment for patient panel
characteristics affected physician ranking
on quality measures by an average of 7.6
percentiles, with more than a third of
physicians studied being reclassified into
a different tertile. 37
While adjustment for case-mix
characteristics avoids unfairly penalizing
residents who tend to see sicker
patients, 38 from a patient perspective,
these findings suggest that sicker patients
may not consistently receive high-quality
care in this setting. Alternatively, sicker
patients may prompt faculty and residents
to engage with the system in a different
way that is not captured by this EHR
data. 39 Hence, case-mix adjustment may
be reasonable from an assessment point
of view, but one should not overlook the
opportunities for improvement in patient
care from a care quality perspective.
Implications for practice
RSQM composite measures, easily
extractable from the EHR, can provide
meaningful information about residents’
clinical performance and point out
opportunities for improving health care
delivery. Although our application in this
study is just one example, RSQMs are
generalizable to any training program
that uses an EHR. In the future, RSQM
use can help set a standard among
programs for resident feedback, in
addition to setting educational goals
when residents are missing clinical
experiences. A similar procedure of
defining and testing RSQMs can be
applied to other clinical settings, although
future replication studies are needed
to examine the feasibility of extracting
RSQMs from EHRs in different contexts,
such as using different EHR systems.
When calculating RSQM composite
scores, however, some programs might
prefer calculating raw scores, but
residents should be compared within
their own level of experience/cohort.
EHR data can also provide direct
information for program directors about
the levels of exposure of individual
residents to various clinical scenarios.
In this study, roughly 20% of residents
did not get the experience of caring
for patients with asthma exacerbation
or bronchiolitis in the PED. While
residents may have cared for patients
Research Report
with these illnesses in other settings,
this information about the PED may be
used by the program to target residents
who did not get this experience
to ensure competency is achieved.
Regarding patient care, low use of
the asthma order set could prompt
the program to emphasize the use of
clinical tools for residents before the
start of the rotation. Residents may also
benefit from closer clinical supervision
when caring for patients with acute
exacerbation of asthma, rather than
bronchiolitis, especially in their first 2
years of training and with higher acuity
patients.
Strengths and limitations
Using EHR data to assess residents has
several strengths. We were able to obtain
data for all patients with bronchiolitis and
asthma exacerbation seen by a majority
of residents rotating through the PED
within the study period. Using RSQMs
strengthens the assessment process by
providing resident-level as well as patient-
level data. It is possible to objectively
identify underperforming residents,
while providing feedback to residents
that is performance-specific, allowing
them to track and work to improve their
performance. This method provides
unique opportunities for identifying
needed quality improvement initiatives,
thereby responding to the goal of GME to
improve quality of care while supporting
the goals of CBME.
This study has several limitations. First,
this was a single-institution study. While
this controls for environmental factors
when comparing between residents at
the same institution, it is necessary to
determine whether these findings are
generalizable to other institutions. More
work is needed to provide residents and
programs with a full dashboard of RSQMs,
as performing well on one measure does
not translate to good performance on all
measures.28,31 Second, EHR data may not
provide sufficient information necessary
for risk adjustment.29 Variations in
electronic charting systems may limit the
ability of RSQMs to be replicated at other
sites. This work is currently underway.
Finally, the weights of individual items in
the composite score may not reflect their
true proportional contribution to patient
outcome.29 Health care is ultimately
provided by teams in a collaborative way,
and a single RSQM, or even a composite of
373
 Research Report
them, cannot fully encompass the quality
of care delivered by a single resident.
Future research
Future studies are needed to replicate these
findings at other sites as well as in programs
located in smaller and rural locations. This
research should focus on understanding
the relationships between faculty and
Downloaded from http://journals.lww.com/academicmedicine by BhDMf5ePHKav1zEoum1tQfN4a+kJLhEZgbsIHo4X
resident performance as well as the effects
Mi0hCywCX1AWnYQp/IlQrHD3i3D0OdRyi7TvSFl4Cf3VC4/OAVpDDa8K2+Ya6H515kE= on 04/18/2024
of patient case-mix and other resident
characteristics on clinical performance.17
Further research should examine the extent
of interdependence between residents and
faculty in decision making around orders
at different stages of training to better
interpret the extrapolation evidence.36
Studying the relationship between RSQMs
and other traditional workplace-based
assessment approaches would provide
additional extrapolation evidence for
this novel method of assessment. Further
research could also focus on studying
cross-classification effects stemming
from the interdependent nature of team-
based patient care39 while developing
the methodological ingenuity needed to
capture both nesting and cross-effects when
measuring such performance.35,40
Conclusions
This study shows that EHR data for 2
specific clinical conditions can be used
to meaningfully assess residents’ clinical
performance in the context of a CBME
program of assessment, and to identify
opportunities for improving health
care delivery. Our findings suggest that,
in the context of a CBME program of
assessment, meaningful RSQMs should
include performance differentiating
measures that are criterion based, valid,
and reliable to capture a wider range of
resident performance in practice.
Funding/Support: The authors would like to
thank the Edward J. Stemmler, MD Medical
Education Research Fund of the National Board
of Medical Examiners for funding the work of
this collaborative group.
Other disclosures: None reported.
Ethical approval: Cincinnati Children’s Hospital
Medical Center (CCHMC) institutional review
board approval was obtained before data
extraction and analysis.
Disclaimers: The views expressed herein are those
of the authors and not necessarily those of the
U.S. Department of Defense or other federal
agencies.
Previous presentations: Preliminary results of
this study were presented at the Office of Health
374
and Medical Education Scholarship Symposium
(OHMES), February 8, 2021, in Calgary, Canada
(virtual conference). This study was presented
for oral presentation as a research paper at the
AMEE conference in Lyon, France, August
29–31, 2022.
Data: Only data from CCHMC were used for
this study. CCHMC institutional review board
approval was obtained before data extraction and
analysis.
A. Smirnova is clinical assistant professor,
Department of Family Medicine, University of
Calgary, Calgary, Alberta, Canada, and adjunct
assistant professor, Kern Institute for the
Transformation of Medical Education, Medical
College of Wisconsin, Milwaukee, Wisconsin; ORCID:
https://orcid.org/0000-0003-4491-3007.
S. Chahine is associate professor of measurement
and assessment, Faculty of Education, Queen’s
University, Kingston, Ontario, Canada; ORCID:
https://orcid.org/0000-0003-0488-773X.
C. Milani is clinical research assistant, Bruyère
Research Institute, Ottawa, Ontario, Canada.
A. Schuh is associate professor of pediatrics,
Division of Emergency Medicine, Medical College
of Wisconsin, Milwaukee, Wisconsin; ORCID: http://
orcid.org/0000-0002-6422-2361.
S.S. Sebok-Syer is assistant professor, Department
of Emergency Medicine, Stanford University School
of Medicine, Palo Alto, California; ORCID: http://
orcid.org/0000-0002-3572-5971.
J.L. Swartz is clinical associate professor, Ronald
O. Perelman Department of Emergency Medicine,
NYU Grossman School of Medicine, and director
of clinical informatics, Department of Emergency
Medicine, NYU Langone Health, New York,
New York.
J.A. Wilhite is senior research coordinator,
Department of Medicine, NYU Langone Health,
New York, New York; ORCID: https://orcid.
org/0000-0003-4096-8473
A. Kalet is professor and Steven and Shelagh Roell
Chair, Robert D. and Patricia P. Kern Institute for
the Transformation of Medical Education, Medical
College of Wisconsin, Milwaukee, Wisconsin; ORCID:
http://orcid.org/0000-0003-4855-0223.
S.J. Durning is professor and vice chair,
Department of Medicine, and director, Center
for Health Professions Education, Uniformed
Services University of the Health Sciences,
Bethesda, Maryland; ORCID: http://orcid.
org/0000-0001-5223-1597.
K.M.J.M.H. Lombarts is professor of professional
performance, Department of Medical Psychology,
Amsterdam University Medical Centers, University of
Amsterdam, and Amsterdam Public Health research
institute, Amsterdam, The Netherlands; ORCID:
https://orcid.org/0000-0001-6167-0620.
C.P.M. van der Vleuten is professor
of education, Department of Educational
Development and Research. Faculty of Health,
Medicine and Life Sciences, Maastricht University,
Maastricht, The Netherlands; ORCID: http://orcid.
org/0000-0001-6802-3119.
D.J. Schumacher is professor of pediatrics,
Division of Emergency Medicine, Cincinnati
Children’s Hospital Medical Center/University of
Cincinnati College of Medicine, Cincinnati, Ohio;
ORCID: http://orcid.org/0000-0001-5507-8452.
Academic Medicine, Vol. 98, No. 3 / March 2023
References
1 Frenk J, Chen L, Bhutta ZA, et al.
Health professionals for a new century:
Transforming education to strengthen health
systems in an interdependent world. Lancet.
2010;376:1923–1958.
2 Asch DA, Nicholson S, Srinivas S, Herrin J,
Epstein AJ. Evaluating obstetrical residency
programs using patient outcomes. JAMA.
2009;302:1277–1283.
3 Epstein AJ, Srinivas SK, Nicholson S, Herrin
J, Asch DA. Association between physicians’
experience after training and maternal
obstetrical outcomes: Cohort study. BMJ.
2013;346:f1596.
4 Asch DA, Nicholson S, Srinivas SK,
Herrin J, Epstein AJ. How do you deliver
a good obstetrician? Outcome-based
evaluation of medical education. Acad Med.
2014;89:24–26.
5 Bansal N, Simmons KD, Epstein AJ, Morris
JB, Kelz RR. Using patient outcomes
to evaluate general surgery residency
program performance. JAMA Surg.
2016;151:111–119.
6 Chen C, Petterson S, Phillips R, Bazemore
A, Mullan F. Spending patterns in region
of residency training and subsequent
expenditures for care provided by practicing
physicians for Medicare beneficiaries. JAMA.
2014;312:2385–2393.
7 Sirovich BE, Lipner RS, Johnston M,
Holmboe ES. The association between
residency training and internists’ ability to
practice conservatively. JAMA Intern Med.
2014;174:1640–1648.
8 Phillips RL, Petterson SM, Bazemore AW,
Wingrove P, Puffer JC. The effects of training
institution practice costs, quality, and other
characteristics on future practice. Ann Fam
Med. 2017;15:140–148.
9 Denson JL, McCarty M, Fang Y, Uppal A,
Evans L. Increased mortality rates during
resident handoff periods and the effect of
ACGME duty hour regulations. Am J Med.
2015;128:994–1000.
10 Denson JL, Jensen A, Saag HS, et al.
Association between end-of-rotation
resident transition in care and mortality
among hospitalized patients. JAMA.
2016;316:2204–2213.
11 Chahine S, Kulasegaram K, Wright S, et al.
A call to investigate the relationship between
education and health outcomes using big
data. Acad Med. 2018;93:829–832.
12 Arora VM. Harnessing the power of big data
to improve graduate medical education: Big
idea or bust? Acad Med. 2018;93:833–834.
13 Levin JC, Hron J. Automated reporting of
trainee metrics using electronic clinical
systems. J Grad Med Educ. 2017;9:361–365.
14 Smirnova A, Sebok-Syer SS, Chahine S, et al.
Defining and adopting clinical performance
measures in graduate medical education:
Where are we now and where are we going?
Acad Med. 2019;94:671–677.
15 Kalet AL, Gillespie CC, Schwartz MD, et
al. New measures to establish the evidence
base for medical education: Identifying
educationally sensitive patient outcomes.
Acad Med. 2010;85:844–851.
16 Simpson D, Sullivan GM, Artino AR,
Jr, Deiorio NM, Yarris LM. Envisioning
graduate medical education in 2030. J Grad
Med Educ. 2020;12:235–240.

17 Schumacher DJ, van der Vleuten CP,
Carraccio CL. The future of high-quality
care depends on better assessment of
physician performance. JAMA Pediatr.
2016;170:1131–1132.
18 Wyer PC. Assessing resident performance:
Do we know what we are evaluating? Ann
Emerg Med. 2019;74:679–681.
19 Schumacher DJ, Holmboe ES, van der
Vleuten C, Busari JO, Carraccio C.
Developing resident-sensitive quality
Downloaded from http://journals.lww.com/academicmedicine by BhDMf5ePHKav1zEoum1tQfN4a+kJLhEZgbsIHo4X
measures: A model from pediatric emergency
medicine. Acad Med. 2018;93:1071–1078.
Mi0hCywCX1AWnYQp/IlQrHD3i3D0OdRyi7TvSFl4Cf3VC4/OAVpDDa8K2+Ya6H515kE= on 04/18/2024
20 Kinnear B, Kelleher M, Sall D, et al.
Development of resident-sensitive
quality measures for inpatient general
internal medicine. J Gen Intern Med.
2021;36:1271–1278.
21 Schumacher DJ, Martini A, Holmboe E, et
al. Developing resident-sensitive quality
measures: Engaging stakeholders to inform
next steps. Acad Pediatr. 2019;19:177–185.
22 Schumacher DJ, Martini A, Holmboe E, et al.
Initial implementation of resident-sensitive
quality measures in the pediatric emergency
department: A wide range of performance.
Acad Med. 2020;95:1248–1255.
23 Schumacher DJ, Holmboe E, Carraccio C, et
al. Resident-sensitive quality measures in the
pediatric emergency department: Exploring
relationships with supervisor entrustment
and patient acuity and complexity. Acad
Med. 2020;95:1256–1264.
24 Schumacher DJ, Martini A, Sobolewski B,
et al. Use of resident-sensitive quality
measure data in entrustment decision
making: A qualitative study of clinical
competency committee members at
Academic Medicine, Vol. 98, No. 3 / March 2023
one pediatric residency. Acad Med.
2020;95:1726–1735.
25 Cook DA, Brydges R, Ginsburg S, Hatala
R. A contemporary approach to validity
arguments: A practical guide to Kane’s
framework. Med Educ. 2015;49:560–575.
26 Mittiga MR, Schwartz HP, Iyer SB, Gonzalez
del Rey JA. Pediatric emergency medicine
residency experience: Requirements
versus reality. J Grad Med Educ.
2010;2:571–576.
27 Lancaster GA, Thabane L. Guidelines
for reporting non-randomised pilot and
feasibility studies. Pilot Feasibility Stud.
2019;5:114.
28 Smith KA, Sussman JB, Bernstein SJ,
Hayward RA. Improving the reliability
of physician “report cards.” Med Care.
2013;51:266–274.
29 Shwartz M, Restuccia JD, Rosen AK.
Composite measures of health care provider
performance: A description of approaches.
Milbank Q. 2015;93:788–825.
30 Scholle SH, Roski J, Adams JL, et al.
Benchmarking physician performance:
Reliability of individual and composite
measures. Am J Manag Care.
2008;14:833–838.
31 Parkerton PH, Smith DG, Belin TR, Feldbau
GA. Physician performance assessment:
Nonequivalence of primary care measures.
Med Care. 2003;41:1034–1047.
32 Yudkowsky R, Park YS, Downing SM, eds.
Assessment in Health Professions Education.
New York, NY: Routledge; 2019.
33 Kamata A. Item analysis by the hierarchical
generalized linear model. J Educ Meas.
2001;38:79–93.
Research Report
34 Sebok-Syer SS, Chahine S, Watling CJ,
Goldszmidt M, Cristancho S, Lingard
L. Considering the interdependence of
clinical performance: Implications for
assessment and entrustment. Med Educ.
2018;52:970–980.
35 Sebok-Syer SS, Shepherd L, McConnell
A, Dukelow AM, Sedran R, Lingard L.
“EMERGing” electronic health record
data metrics: Insights and implications for
assessing residents’ clinical performance
in emergency medicine. AEM Educ Train.
2021;5:e10501.
36 Sebok-Syer SS, Goldszmidt M, Watling CJ,
Chahine S, Venance SL, Lingard L. Using
electronic health record data to assess
residents’ clinical performance in the
workplace: The good, the bad, and the
unthinkable. Acad Med. 2019;94:853–860.
37 Hong CS, Atlas SJ, Chang Y, et al.
Relationship between patient panel
characteristics and primary care physician
clinical performance rankings. JAMA.
2010;304:1107–1113.
38 Gebauer S, Steele E. Questions program
directors need to answer before using
resident clinical performance data. J Grad
Med Educ. 2016;8:507–509.
39 Sebok-Syer SS, Pack R, Shepherd L, et al.
Elucidating system-level interdependence in
electronic health record data: What are the
ramifications for trainee assessment? Med
Educ. 2020;54:738–747.
40 Sebok-Syer SS, Shaw JM, Asghar F, Panza
M, Syer MD, Lingard L. A scoping review of
approaches for measuring “interdependent”
collaborative performances. Med Educ.
2021;55:1123–1130.
375