FEATURE ARTICLE
Changes in Efficiency and Quality of Nursing Electronic
Health Record Documentation After Implementation of
an Admission Patient History Essential Data Set
Eva L. Karp, DHA, MBA, RN-C, FACHE, SVP, Rebecca Freeman, PhD, RN, Kit N. Simpson, DrPH, Annie N. Simpson, PhD
Downloaded from https://journals.lww.com/cinjournal by BhDMf5ePHKav1zEoum1tQfN4a+kJLhEZgbsIHo4XMi0hCywCX1AWnYQp/IlQrHD3/pCPTECtaMRyg4SBPyana1WzHubb2oMfp1PB5bf7c/Dec1V0O/bf8Q== on 09/24/2019
The objective of this quality improvement study was to eval-
uate whether electronic health record system timers and
event logs can measure the efficiency and quality of a clini-
cal process in an electronic health record. Using an experi-
mental pre– and post–nonrandomized prospective cohort
design, the researchers introduced a newly defined admis-
sion patient history essential data set and examined the
electronic health record event files and timers to analyze
the nursing experience from an efficiency and quality per-
spective. The researchers evaluated efficiency by measur-
ing the time and clicks required to complete an admission
history. The average active time spent documenting the
admission patient history decreased by 72% from the pre-
intervention measure (mean = 9.30 minutes) to the postin-
tervention measure (mean = 2.55 minutes). The number of
clicks decreased by 76% from the preintervention number
of clicks (mean = 151.5) to the postintervention number
of clicks (mean = 35.93). The quality of documentation
was measured as the proportion of completed essential
items and the frequency of completing an assessment in
one sequence. The capture of essential data elements im-
proved by almost 6%, and admission patient history data
completed in one sequence increased by 24%. These study
results demonstrate that system timers and event logs can
measure the preintervention and postintervention changes
in efficiency and quality of a defined clinical workflow into an
electronic health record.
KEY WORDS: EHR event logs, Electronic health record,
Nursing documentation, Nursing efficiency
Author Affiliations: Cerner Corporation, Kansas City, MO (Dr Karp); and Information Services,
University of Vermont Health System, Meggett (Dr Freeman); and Department of Healthcare
Leadership and Management, Medical University of South Carolina, Charleston (Drs K.
Simpson and A. N. Simpson).
The authors have disclosed that they have no significant relationships with, or financial interest
in, any commercial companies pertaining to this article.
Corresponding author: Eva L. Karp, DHA, MBA, RN-C, FACHE, SVP, Cerner Corporation, 2800
Rockcreek Pkwy, Kansas City, MO 64117 (ekarp@cerner.com).
This is an open-access article distributed under the terms of the Creative Commons Attribution-Non
Commercial-No Derivatives License 4.0 (CCBY-NC-ND), where it is permissible to download and
share the work provided it is properly cited. The work cannot be changed in any way or used
commercially without permission from the journal.
Copyright © 2019 The Authors. Published by Wolters Kluwer Health, Inc.
DOI: 10.1097/CIN.0000000000000516
260 CIN: Computers, Informatics, Nursing
OBJECTIVE
Electronic health record (EHR) nursing documentation of-
fers a method to record the patient's health status, individual
needs, and responses to care, and to support clinical reason-
ing regarding the patient's future care.1 Additionally, the EHR
has the potential to improve patient care, care team communica-
tion, and patient experiences through well-designed workflows
and clinical content relevant to the role, venue, and patient's clin-
ical status. The objective of this quality improvement study was
to evaluate whether EHR system timers and event logs can mea-
sure the efficiency and quality of a clinical process in an EHR.
BACKGROUND AND SIGNIFICANCE
Over the last few decades, nurses have increasingly been bur-
dened with documentation to meet regulatory and quality
reporting requirements. This reporting requires nurses to col-
lect data not intrinsically linked with knowledge and context
of the patient's story and potential risk factors.2,3 Nurses play
a crucial role in patient care and performance improvement
in the healthcare system. This role requires documenting
and managing patient information through coordinating care
and communicating with other healthcare professionals.4
The nursing profession is striving to standardize care processes
with clinically relevant content while considering the complexity
of care, conditions in the environment, market and regulatory
changes, and clinician preferences.5,6 These factors increase
the need for health information technology (IT) professionals
to have a mechanism to measure the effectiveness, efficiency,
and quality of clinical processes supported within the EHR.
Today's health information systems capture significant data
in event and timer logs. Combined with EHR data, these logs
allow examination of temporal patterns of care, including min-
ing, modeling, and measuring the clinician's experiences within
the EHR. Event logs are extracts from EHRs that comprised lists
of time-stamped process-steps created as a by-product of operations.7
Researchers and informatics professionals can use EHR
logs to evaluate historical content, workflow design, and uti-
lization patterns, including time and clicks required for
an EHR workflow. Wu et al8 completed a study that used
EHR audit logs, and found that these logs are a valid source
for workflow analysis and can provide an objective view of
May 2019
clinician behaviors. Mining process information directly
extracted from these logs can provide insight into how health-
care processes are executed.9 For the event logs to be useful,
investigators must understand which events and timers are
relevant to measure processes related to a clinician's role,
venue, quality, and sometimes the patient's clinical condition.
Chen et al10 found that existing methods were insufficient,
resource-intensive, and time-consuming in measuring the
influences of EHRs regarding end-user adoption, efficiency,
and utilization of workflows. Chen and colleagues introduced
a framework to infer clinical workflows through the utilization
of an EHR. This framework provides a method to generate
workflows at multiple levels of granularity through data min-
ing of EHR event logs, allowing healthcare systems and health
informatics organizations to evaluate and refine inefficient
and unstable workflows. The researchers categorized workflows
into four classes based on their duration and variance:
1. stable efficient blocks: a short average duration with
small variance;
2. unstable efficient blocks: a short average duration with
large variance;
3. stable inefficient blocks: a long average duration with
small variance; and
4. unstable inefficient blocks: a long average duration with
large variance.10
The researchers selected the nursing admission patient
history (APH) for this study because it meets the classification
of a stable and efficient process. Prior to this quantitative
quality improvement study, a qualitative research team
consulted an expert panel of 20 chief nursing officers and
chief nursing informatics officers, which included a doctor
of nursing practice and PhD to establish an adult APH essen-
tial clinical data set (APH ECD). The panel represented a
cross section of 12 healthcare organizations, which accounted
for approximately 46 000 acute care beds and 192 hospitals,
including a mix of academic medical centers, integrated deliv-
ery networks, and community hospitals.
The research team used a modified Delphi approach to
analyze the historical form design and 3 months of utilization
patterns for adult APHs. The research team pulled timer
and discrete data for APHs of 12 organizations and analyzed
each organization's admission history form and the average
utilization patterns for the nursing documentation of patient
histories. Based on Weiskopf and Weng's11 dimensions of qual-
ity as well as federal regulatory requirements, the research
team defined an essential standardized data set of 40 elements
for the adult APH that met the intended use and were of suffi-
cient quality for a patient history assessment on admission.
This study was intended to measure the quantitative impact
of implementing the reduced number of essential data elements
utilizing EHR timers and event logs. The authors had two ob-
jectives for this study: (1) to determine whether system timers
Volume 37 | Number 5
and event logs can measure the preintervention and postin-
tervention status of a defined clinical workflow, and (2) to
determine whether introducing a standardized APH ECD
improves nursing efficiency and quality of documentation.
We hypothesize that reducing this data set to essential
data elements will (1) increase the proportion of essential items
completed, (2) decrease the documentation time required to
complete the APH, and (3) reduce the number of mouse clicks
to complete the APH.
MATERIALS AND METHODS
The research setting was a 600-bed academic medical center
in the United States. The study included medical-surgical,
ICU, emergency department, step-down, and telemetry clin-
ical documentation for adult APHs performed by RNs, with a
focus on US-based practices at the federal level. The research
site's nursing leadership and informatics team compared their
baseline APH 215 data elements to the APH ECD 40 data
elements and adjusted for local and state requirements. This
review resulted in the study site reducing its APH from 215
data elements to 58 data elements, reflecting a 73% reduc-
tion in its APH ECD.
The quality improvement research site integrated the
APH ECD into their EHR for acute adult inpatient units.
We collected 1 month of data preintervention and postinter-
vention to measure nursing efficiency and quality measures
to evaluate the impact of the APH ECD. We utilized timers
and event logs embedded in the software to capture duration
and the number of data elements documented by each RN.
Data sets were extracted from the EHR event files that in-
cluded timer data. The timers systematically captured the
beginning and end of the APH documentation. The system
timer data were validated by an electronic video recording
of a sample of nurses completing the forms and comparing
the system timers to the video recording timers. Event files
were also utilized to measure the data elements captured.
For the preintervention data set, the investigators ana-
lyzed baseline measures for patient admission histories com-
pleted over 30 days prior to implementation of the ECD in
September 2017. The timer data were captured for each ep-
isode of care, which was the unit of analysis. The data set in-
cluded all data elements in the APH form, the number of
data elements documented, and time and number of mouse
clicks required to complete the APH for each encounter.
The APH ECD was placed in production (ie, viewable to
clinicians) on November 1, 2017. The postimplementation
period included 30 days of APHs completed from November
20, 2017, to December 17, 2017. Data from November 1
to November 19 were excluded to allow nurses to adjust to
the APH ECD.
To assess quality, the difference in the proportion of each of
the 58 essential data elements captured before and after introducing
CIN: Computers, Informatics, Nursing 261
FEATURE ARTICLE

the APH ECD was evaluated. This measure was based on the
premise that if there are fewer total elements in the patient his-
tory, the most important elements were likely documented.
The researchers included all assessments completed in pre-
intervention (n = 904) and postintervention periods (n = 805).
To assess efficiency, we measured (1) the time it took, in mi-
nutes; and (2) the number of mouse clicks for nurses to complete
an APH before and after implementation of the APH ECD.
For the efficiency measures, the researchers included all
assessments completed in one-sequence preintervention (n =
536) and postintervention periods (n = 640), due to limitations
of the timer to measure time and clicks for assessments only
partially completed initially and then completed later.
Descriptive statistics were used to evaluate the differences
preintervention and postintervention on baseline character-
istics, the percentage of essential data elements captured,
and the number of clicks and time to complete the APH.
The significance of the difference between means was deter-
mined using an independent-samples t test. Descriptive and
statistical calculations were performed using SAS version
9.4 (SAS Institute, Cary, NC). This quality improvement
study was deemed nonhuman research by the research site's
institutional review board (IRB); no IRB approval was required.
RESULTS
Table 1 reflects the descriptive statistics for preintervention
and postintervention quality and efficiency measures. Quality
is measured by the number and percentage of essential vari-
ables captured as well as the proportion of APH completed
in one sequence. These quality measures included all APHs
during the study period. Efficiency measures included the
mean number of clicks and the time spent completing an
APH, which was performed on the subset of APHs that were
completed in one sequence.
Quality Measures
To assess quality, the difference in the number of the 58 es-
sential data elements captured for the APH after introducing
the APH ECD was evaluated. Figure 1 shows a statistically
significant increase in the number of essential data elements
captured from the preintervention measure (mean = 48%
[SD = 0.10]) to the postintervention measure (mean = 54%
[SD = 0.11]; t = −10.83, P = <.0001, two-tailed). The percent-
age of capture of data elements considered essential in-
creased from 48% to 54%. In addition, the number of APHs
completed in one sequence, without saving and signing or
modifying later, increased by 24% from the preintervention
measure (mean = 62% [SD = 0.48]) to the postintervention
measure (mean = 86% [SD = 0.35]).
Efficiency Measures
We analyzed the nurses' total time and clicks from start to
completion of an APH for all patients admitted within the
preintervention and postintervention periods. Only patient
histories completed in a single sequence were included in this
analysis. If a patient history was partially completed initially
and then completed later, the history was excluded from the
analysis due to timer data limitations. The time in minutes
and the number of clicks to complete the APH were assessed
using descriptive statistics.
The average active time spent documenting the APH de-
creased by 72%. Figure 2 shows the decrease in active time
from the preintervention measure (mean = 9.30 [SD = 4.66]
minutes) to the postintervention measure (mean = 2.55 [SD =
1.65] minutes), resulting in a difference of 6.76 minutes
(t = 31.95, P = <.0001, two-tailed).
The average number of clicks to document the APH de-
creased by 76%. We found a statistically significant decrease
in the number of clicks from the preintervention number of
clicks (mean = 151.51 [SD = 37.35]) to the postintervention
number of clicks (mean = 35.93 [SD = 15.01]; t = 67.40,
P = <.0001, two-tailed). The mean decrease in the number
of clicks was 115.6 with a 95% confidence interval and with
a range of 112.2 to 118.9, P < .0001. The mean differences of both
the time and clicks represent statistically significant changes that
may have a noticeable impact on clinical workflows and efficiency.
DISCUSSION
The automation of clinical workflows provides new opportu-
nities to measure efficiency and quality. Historically, researchers

Table 1. Preintervention and Postintervention Measures

Time Period n Quality Outcomes Mean Median SD
Pre 904 Essential elements captured 48% 48% 0.10
904 Completed in one sequence 62% 0.48
Post 805 Essential elements captured 54% 52% 0.11
805 Completed in one sequence 86% 0.35
Time Period n Efficiency Outcomes Mean Median SD
Pre 536 Active time (min) 9.30 8.33 4.66
640 No. of clicks 151.51 147 37.35
Post 536 Active time (min) 2.55 2.21 1.65
640 No. of clicks 35.93 33.00 15.01

262 CIN: Computers, Informatics, Nursing May 2019

FIGURE 1. Preintervention and postintervention efficiency measures.
have used time and motion studies, observations, surveys, retro-
spective chart reviews, and structured and semistructured inter-
views to measure the effect of technology on the nursing process,
documentation quality, and workflow.12–15 This study supports
Wu and colleagues'8 and Chen and colleagues'10 findings that
today's health information systems have significant data cap-
tured in event and timer logs that could research the influ-
ences of EHRs regarding end-user adoption, efficiency, and
utilization of workflows.
This study demonstrates that, by analyzing EHR event
files, health systems could discover whether automated clinical
processes are meeting the intended objectives from a quality
and efficiency perspective.10 This study also demonstrates that
EHR event files are an additional source to support an objec-
tive analysis of workflow, clinicians' behaviors related to EHR
use, and efficiency measures. This information can support
quality and efficiency initiatives and increase knowledge and
understanding of the clinician's experience.
This approach has not been used to evaluate the quality
and efficiency of nursing documentation, so we cannot com-
pare it to previous studies. However, we found it very effective
to measure the impact of quality and efficiency of changing
FIGURE 2. Preintervention and postintervention percentage capture of the essential data elements.
Volume 37 | Number 5
content within an EHR nursing process. Health systems
and researchers are striving to standardize processes while con-
sidering the complexity of care, conditions of the environ-
ment, and clinician preferences.
This study demonstrates that an analytical framework
maximizing the use of EHR event files can provide practice-
based insights into clinical workflows and interventions, with
added insight into the context of data collected during clinical
care and the impact on clinical decision making. This frame-
work can support a health system's ability to use EHR data for
discoveries, creating data-driven contextual intelligence to de-
velop strategies to optimize and advance knowledge related
to clinical practice in an EHR environment.
Future studies could advance knowledge for healthcare in-
formatics professionals and researchers about the evolution
of EHR event logs and timers to measure end-user adoption,
process, and outcome measures. Additional factors that could
be measured include clinician experience as it relates to commu-
nication between nursing and other care providers, measurable
secondary use of the data, and the impact on patient outcomes.
Further research should include the downstream use of
clinical documentation by other members of the care team
CIN: Computers, Informatics, Nursing 263
FEATURE ARTICLE
to support team-wide clinical reasoning and decision making
and to improve patient outcomes. In future studies, we will
include physical assessment documentation and how the
care team utilizes the information. Documentation is a com-
munication method with care team members; a mixed-
methods study would add additional information related to
the value the care team assigns to documentation as an effec-
tive method to support communication, collaboration, and
coordination of care.
LIMITATIONS
The results of this study may not be generalizable, especially
if a health system has no EHR with timers and event
logs to measure quality and efficiency. However, this work
will inform future research to determine the impact and
feasibility of an optimized approach to documentation,
using system timers and event logs for preintervention and
postintervention measurements.
This study occurred in an organization that used an EHR
with embedded timers and event logs capable of discrete
data capture. All EHRs may not have the same features em-
bedded into the clinical workflows. Additionally, if the event
logs exist, they may have limitations. For example, for this re-
search project, the event logs did not allow the researchers to
measure time and clicks if the provider documented, saved
the form, then returned later to complete/sign the form.
The internal timers could not collect the actual duration of
intermittent documentation episodes. Due to this technical
limitation, only uninterrupted documentation instances were
included in the analysis of efficiency. Future research should
examine whether timers can be adapted to capture breaks
in time and ensure storage of that data.
Conditional logic needs to be considered when measuring
quality. The essential data elements in an EHR event file
might be relevant for a particular patient and not others.
This functionality is called conditional logic, and it stream-
lines documentation by not displaying or requiring docu-
mentation of data elements unless relevant. For example,
when nurses document patient belongings, a patient may
have glasses and hearing aid, but not a cane. The nurse
would select only the belongings relevant to the patient.
Therefore, 100% capture of essential data elements could
not be expected, because some data elements are not rele-
vant for all patients.
This study did not assess how the content captured in the
APH was utilized by other clinicians to support clinical reasoning
or to meet regulatory requirements. A method that could pro-
vide insight into nursing documentation used by other team
members would be to study the downstream use of the informa-
tion captured in workflows, such as patient summaries, in other
care team workflows and reporting or during clinical deci-
sion support. Those key components support care team
264 CIN: Computers, Informatics, Nursing
communications and collaboration and warrant further
study. Future studies should address the multiple factors that
affect nursing efficiency so hospital systems can maximize
the benefits of EHRs.
CONCLUSION
Both the literature and anecdotal evidence from nurses indi-
cate that EHR systems must be revised to reduce the time
spent on documentation so nurses can spend more time on
patient care. The first step is to assess the characteristics of
EHR systems and redesign them to optimize nursing pro-
cesses by including only essential documentation to support
clinical practice.11 Automation of clinical workflows in EHRs
has created new opportunities for clinical informaticists to
measure and evaluate the impact of health IT on clinical
workflow effectiveness, efficiency, utilization, and quality of
workflows within an EHR.
Changing an EHR clinical workflow is a multifactorial
process. The outcome could be influenced by staff training,
years of experience of staff, the acuity of the patients, the
physical environment of the study, and the organizational
culture of the study site. While patterns and trends can be
measured, proving cause and effect may be influenced by more
than the changes in content. Clinical informatics professionals
should consider the use of EHR event files and timers to gain
insight into process and workflow changes. The use of system
data can substantiate the transformational value of informat-
ics practice and inform future optimization efforts. The re-
sults may be the first step toward achieving the early promise
regarding the value of EHRs from more than 20 years ago.
Acknowledgments
The authors thank Amy Peters and Cathy McFarland from the study site
for their total support and belief that this study will make a difference to
clinicians and patients. Also, they thank Darinda Sutton, Cameron Johnson,
Carly Evans, and Lindsey Jarrett for their partnership on this study.
References
1. Akhu-Zaheya L, Al-Maaitah R, Bany Hani S. Quality of nursing documentation:
paper-based health records versus electronic-based health records. Journal
of Clinical Nursing. 2018;27(3–4): e578–e589.
2. Okaisu EM, Kalikwani F, Wanyana G, Coetzee M. Improving the quality of
nursing documentation: an action research project: original research.
Curationis. 2014;37(2): 1–11.
3. Kruse CS, Kothman K, Anerobi K, Abanaka L. Adoption factors of the
electronic health record: a systematic review. JMIR Medical Informatics.
2016;4(2): e19.
4. Kahn MG, Callahan TJ, Barnard J, et al. A harmonized data quality
assessment terminology and framework for the secondary use of electronic
health record data. EGEMS (Washington, DC) 2016;4(1): 1244.
5. Gocsik T, Barton AJ. Workflow optimization: essential in the electronic health
record journey. Clinical Nurse Specialist. 2014;28(3): 132–134.
6. Ghattas J, Soffer P, Peleg M. A formal model for process context learning. In:
International Conference on Business Process Management. Berlin,
Heidelberg, Germany: Springer; 2009:140–157.
May 2019
 7. Fox F, Aggarwal VR, Whelton H, Johnson O. A Data Quality Framework
for Process Mining of Electronic Health Record Data. Paper presented
at Healthcare Informatics (ICHI), 2018 I.E. International Conference;
2018.
8. Wu DTY, Smart N, Ciemins EL, Lanham HJ, Lindberg C, Zheng K.
Using EHR audit trail logs to analyze clinical workflow: a case study from
community-based ambulatory clinics. American Medical Informatics
Association Annual Symposium Proceedings. 2018;2017: 1820–1827.
9. Mans RS, van der Aalst WM, Vanwersch RJ, Moleman AJ. Process mining in
healthcare: data challenges when answering frequently posed questions. In:
Process Support and Knowledge Representation in Health Care. Berlin,
Heidelberg, Germany: Springer; 2013: 140–153.
10. Chen Y, Xie W, Gunter CA, et al. Inferring clinical workflow efficiency via
electronic medical record utilization. AMIA Annual Symposium
Proceedings/AMIA Symposium. 2015;2015: 416–425.
Volume 37 | Number 5
11. Weiskopf NG, Weng C. Methods and dimensions of electronic health record
data quality assessment: enabling reuse for clinical research. Journal of the
American Medical Informatics Association. 2013;20(1): 144–151.
12. Hendrich A, Chow MP, Skierczynski BA, Lu Z. A 36-hospital time and motion
study: how do medical-surgical nurses spend their time? The Permanente
Journal. 2008;12(3): 25–34.
13. Collins SA, Fred M, Wilcox L, Vawdrey DK. Workarounds used by nurses to
overcome design constraints of electronic health records. NI 2012 (2012)
2012;2012: 93. eCollection 2012.
14. Banner L, Olney CM. Automated clinical documentation: does it allow nurses more
time for patient care? CIN: Computers, Informatics, Nursing. 2009;27(2): 75–81.
15. Larrabee JH, Boldreghini S, Elder-Sorrells K, et al. Evaluation of
documentation before and after implementation of a nursing information
system in an acute care hospital - improving patient record data validity.
Computers in Nursing. 2001;19(2): 56.
CIN: Computers, Informatics, Nursing 265