See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/275595129

Patterns in Patient Safety with Computerized Consult Management and Clinical

Documentation

Article  in  Proceedings of the International Symposium on Human Factors and Ergonomics in Health Care · July 2014
DOI: 10.1177/2327857914031022

CITATIONS READS
0 128

4 authors:

Jason Saleem Richard Frankel

University of Louisville Indiana University School of Medicine
101 PUBLICATIONS   2,808 CITATIONS    254 PUBLICATIONS   15,165 CITATIONS   

SEE PROFILE SEE PROFILE

Bradley N Doebbeling Emily S Patterson

Arizona State University The Ohio State University
281 PUBLICATIONS   11,116 CITATIONS    276 PUBLICATIONS   5,910 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Interview reference text View project

Patient Safety with HIT View project

All content following this page was uploaded by Emily S Patterson on 29 August 2015.

The user has requested enhancement of the downloaded file.

 2014 International Symposium on Human Factors and Ergonomics in Health Care: Advancing the Cause 134

Patterns in Patient Safety with Computerized Consult Management

and Clinical Documentation
Jason J. Saleem1-2, Richard M. Frankel2-4, Bradley N. Doebbeling5, and Emily S. Patterson6
1
Human Factors Engineering, Office of Informatics and Analytics, Veterans Health Administration,
Louisville, KY, USA; 2VA HSR&D Center on Implementing Evidence-Based Practice, Roudebush
VA Medical Center, Indianapolis, IN, USA; 3Regenstrief Institute, Inc., Indianapolis, IN, USA;
4
Department of Medicine, Indiana University (IU) School of Medicine, Indianapolis, IN, USA;
5
Department of BioHealth Informatics, Indiana University (IU) School of Informatics and Computing,
Indianapolis, IN, USA; 6Division of Health Information Management and Systems, School of Health
and Rehabilitation Sciences, College of Medicine, The Ohio State University, Columbus, OH, USA

Patient safety issues continue to be prevalent in delivering care with the electronic health record (EHR) and
other health information technology (HIT) tools. Defining the unit of analysis for the study of clinical
information systems is important based on the focus of the research. This study applies a conceptual
framework with three levels of units of analysis for human-computer interaction (HCI) to identify and guide
investigations at each level. Ethnographic observations and semi-structured, key-informant interviews were
conducted with 40 healthcare workers across five primary care clinics, nine specialty clinics, and one
inpatient ward at a tertiary-care Veterans Affairs Medical Center (VAMC). Two researchers recorded the
interactions and interview responses of 40 healthcare workers related to their work with computerized
consultations, clinical documentation, and the EHR in general. Patient safety issues from these data were
categorized using three different units of analysis: EHR interface-, team coordination / workflow-, and
organizational-levels. A total of 30 patient safety issues were identified; 17 emerged from the observation
and interview data on computerized consults, 10 from the data on clinical documentation, and 3 were
related to the EHR in general. Patient safety issues were organized by level of unit of analysis and mapped
to specific methods that could be used for further investigation. Relevant concepts are discussed to help
guide investigations at each level.

INTRODUCTION

Patient safety issues arise in all healthcare settings, but
continue to be prevalent in delivering care with the electronic
health record (EHR) and other health information technology
(HIT) tools (Ash, Berg, & Coiera, 2004; Cheung et al., 2013).
Hospitals and health care systems are implementing EHRs
quickly following the American Recovery and Reinvestment
Act of 2009. EHRs can help improve patient safety, but there
Not subject to U.S. copyright restrictions. DOI 10.1177/2327857914031022
(Hutchins, 1995; Saleem et al., 2009) expand the unit of
analysis beyond the traditional computer system and user to
include workflow and the organization at large. An EHR is an
organizational tool and can be studied, depending on the
specific research questions, at each level: computer interface,
team coordination / workflow, and organizational levels
(Figure 1). This conceptual framework was employed in a

is a need to consider patient safety goals in their
implementation (Jha & Classen, 2011; Sittig & Singh, 2012).
There are well-established classification schemes for safety
incidents, including the World Health Organization’s
International Classification for Patient Safety (ICPS-WHO;
Mikkelsen, Thommesen, & Andersen, 2013; World Alliance
for Patient Safety, 2009) and the Human Factors Analysis
Classification System (HFACS; Elbardissi, Wiegmann,
Dearani, Daly, & Sundt, III, 2007; Wiegmann & Shappell,
2003). Additionally, Sittig and Singh have proposed a three-
phase framework for the development of EHR-specific patient-
safety goals (Sittig & Singh, 2012). However, we are aware of
no corresponding consensus framework for identifying patient
safety issues with HIT.
Previous research has distinguished different units of
analysis for human-computer interaction (HCI). Usability
assessment techniques often focus on a user interacting with a
single computer interface. Frameworks such as cognitive
engineering (Woods & Roth, 1988) and distributed cognition
research study of barriers and facilitators to effective use of
computerized clinical reminders (Saleem et al., 2005); a
similar framework was presented for conducting “cognitive-
socio-technical analyses” in health informatics (Borycki &
Kushniruk, 2010). The objective of this research is to identify
how potential patient safety issues revealed by a study of an
EHR can be characterized by different units of analysis from
an HCI perspective, which in turn can guide study design for
further research. Patient safety issues related to computerized
consults (Saleem et al., 2011) and clinical documentation
(Saleem, Adams, Frankel, Doebbeling, & Patterson, 2013) in
the US Department of Veterans Affairs’ (VA’s) EHR, the
Computerized Patient Record System (CPRS), are analyzed
for this purpose. Consultation and clinical documentation are
the focus for data collection because these applications are part
of a larger research agenda for identifying patient safety risks
associated with workarounds to the EHR (Saleem, 2009).

Downloaded from hcs.sagepub.com by guest on August 26, 2015

2014 International Symposium on Human Factors and Ergonomics in Health Care: Advancing the Cause 135

TABLE 1: Participants’ occupation/role and clinic/service

N Occupation/Role Clinic or Service

1 Administrator Ambulatory Care
2 Administrator Medicine
3 CAC Systems Redesign
4 Licensed Practical Nurse Primary Care
5 Registered Nurse Primary Care
6 Nurse Practitioner Renal
7 Physician Endocrinology
8 Physician Primary Care
9 Physician Primary Care
10 Physician Primary Care
11 Physician Primary Care
12 Physician Ambulatory Care
13 Clinical Pharmacist Primary Care
Figure 1. Patient safety issues according to three levels of analysis: 14 Clinical Pharmacist Primary Care
EHR-interface, team coordination / workflow, and organizational. 15 Clinical Psychologist Polytrauma Unit
Human-computer interaction with the EHR expands beyond the 16 Medical Support Assistant Sleep Study
computer interface level to include team coordination / workflow and 17 Medical Support Assistant Endocrinology
larger organizational issues in a sociotechnical system (Saleem et al., 18 Registered Dietician Primary Care
2005; Saleem et al., 2013). 19 Registered Nurse Primary Care
20 Registered Nurse Dialysis
METHODS 21 Registered Nurse Cardiology
22 Registered Nurse Pain clinic
23 Nurse Practitioner Infectious Disease
This study received approval from the Indiana University 24 Nurse Practitioner Orthopedics
(IU) Institutional Review Board (IRB; study # 0905-53). 25 Nurse Practitioner Primary Care
Ethnographic observations and semi-structured, key-informant 26 Nurse Practitioner Primary Care
interviews were conducted with 40 healthcare workers at a 27 Nurse Practitioner Primary Care
28 Nurse Practitioner Primary Care
single tertiary-care Veterans Affairs Medical Center (VAMC), 29 Physician Assistant Primary Care
which has full laboratory, pharmacy, and radiology services. 30 Physician Primary Care
Ethnographic field observation is an approach that allows 31 Physician Primary Care
32 Physician Primary Care
study of a complex system, such as in situ use of a clinical 33 Physician Renal
information system, from its sociotechnical influences in the 34 Physician Cardiology
organization at large down through design flaws at the 35 Physician Inpatient ward
computer interface level (Hutchins, 1995; Saleem et al., 2005). 36 Physician Nutrition
37 Physician Primary Care
Rapid ethnographic observation was employed to be minimally 38 Physician Pulmonary
intrusive to busy clinicians, while helping capture the context 39 Physician GI clinic
surrounding the EHR tools being studied (McMullen et al., 40 Physician Oncology
2011). Opportunistic interviews were also conducted with the Note. CAC = Clinical Application Coordinator
28 participants who were directly observed during periods of
time when they were less busy. This allowed for follow-up on Data Collection
issues that arose during the observation, assessment of the
thinking processes that were employed while using the Participants were directly observed as they interacted with
software and/or related paper tools, and elicitation of direct computerized consults (8/40) and clinical documentation tasks
feedback on computerized consults and clinical (18/40). Two participants were observed during their
documentation. interaction with the EHR to pilot the data collection techniques
prior to focusing data collection specifically on consults and
Participant Characteristics documentation (2/40). These data were included in our
analysis of patient safety issues. Observations were recorded
The sample of 40 participants included healthcare workers via handwritten notes on an observation form during
from five primary care clinics, nine specialty clinics, and one participant interaction with the EHR and related paper
inpatient ward. In order to have a diverse sample to increase artifacts. The observation form facilitated the use of time
the likelihood of detection of multiple levels of patient safety stamps with corresponding content. For observations related
issues, participants were purposefully recruited from a large to clinical documentation, we observed participants, before,
variety of primary and specialty care clinics and from a variety during, and after patient encounters. We also conducted semi-
of clinical roles. Table 1 shows the professional backgrounds structured interviews with participants for consultation
of each participant and their clinic or service affiliation. management (8/40) and clinical documentation (4/40).
Participants 1-12 served as key informants for the semi- Participants were interviewed if they were key-informants
structured interviews and 13-40 were shadowed as part of the (e.g., high-level administrators) or in a position where
ethnographic observation. interview, rather than observation, was a preferred method.
For example, observation was not used in the primary care

Downloaded from hcs.sagepub.com by guest on August 26, 2015

2014 International Symposium on Human Factors and Ergonomics in Health Care: Advancing the Cause 136

clinics to collect data on computerized consults as the relative
time a primary care provider spends entering consults at the
study site is low (about three consults per half-day), making
interviewing a more efficient method for primary care. All
interviews were digitally audio-recorded and transcribed by
the lead author. A detailed breakdown of participants by the
data collection focus of EHR application (computerized
consults or clinical documentation) exists elsewhere (Saleem
et al., 2011; Saleem, Adams, Frankel, Doebbeling, &
Patterson, 2013).
Analysis
Data analysis followed a process of abstraction, in which
details that are specific to the context of a setting are replaced
by the underlying strategies that are relevant across settings
(Patterson, Cook, & Render, 2002; Xiao & Vicente, 2000).
Two analysts independently coded the observation notes and
interview transcripts for strategies relating to use of the
computerized consults and completion of the EHR progress
notes. We integrated findings across the participants into
meaningful patterns and abstracted the data into emerging
themes (Roth E.M. & Patterson E.S., 2005). These overall
findings are reported elsewhere (Saleem et al., 2011; Saleem et
al., 2013). For the present analysis, only patient safety issues
were considered. Patient safety issues were independently
coded by two analysts for the 16 observations and interview
transcripts related to computerized consultation. The analysts
resolved all differences in each of the 16 independently coded
observation notes and interview transcripts by consensus.
After coding consistency was established using the data from
the first 16 observation and interview transcripts, the
remaining 24 transcripts were coded for patient safety issues
by the lead author.
RESULTS
A total of 30 patient safety issues were coded and
organized by unit of analysis. Of the 30 patient safety issues
identified, 17 emerged from the observation and interview data
on computerized consults, 10 from the data on clinical
documentation, and 3 were related to the EHR in general.
Rather than outlining all 30 identified patient safety issues,
Tables 2-4 present a sample of five patient safety issues each
at the EHR interface (Table 2), team coordination / workflow
(Table 3), and organizational levels (Table 4). In Tables 2-4,
each patient safety issues has a corresponding short,
descriptive label. Several patient safety issues related to
interface usability and conflicting information in the EHR
were classified at the EHR-level (Table 2). Examples of team
coordination and workflow level patient safety issues included
lack of communication between primary and specialty care, as
well as delays in documentation (Table 3). Organization level
examples of patient safety issues included patient volume
exceeding available resources and coordination of care with
other facilities (Table 4).

TABLE 2: Example patient safety issues at the EHR-interface level

Participant type Label Patient Safety Issue

Registered Ambiguous “Patients with the last 4 numbers of social security number. In CPRS, if 2 patients have
Nurse information the same last 4 of the social security number, it makes you choose one of them. But RMS
[another software package] does not make you choose. This is a patient safety issue.
This happened to me recently - I tried to schedule a patient recently and the wrong patient
got scheduled [in RMS].”

Registered Difficult to “Accurate phone numbers for patients is essential. They are not captured at the front
Nurse change desk. And to change one [a phone number] you have to do it in DHCP [another software
information package] - I don't know how to do that. I do most things myself but that I don't know how
to do - it's not easy. You have to have a clerk do it. Right now I can't get a hold of one of
my patients who has suspected leukemia to get test. I can't reach the patient to explain
how important it is to follow up.”

Clinical Information “I use the med list template to pull in meds for my [progress] note. It only pulls in the
Pharmacist hidden in a active meds. It would be good if it pulled in the expired [meds] as well. So I go back to
secondary see what's fallen off this list [as expired] because they still might be taking it.”
display

Registered Information alert “I print all of the consults to review with Dr. [X]. Once I click on it [the alert], it's gone. So if
Nurse is not retained I don't print it, it's gone. It's happened [where she's clicked on it and cannot retrieve it in
after viewed the EHR] and I have to call the clerk and tell her 'I clicked off someone - look out for this'
and she'll say 'this is the one I think you missed'.”

Physician Conflicting The physician looks at the previous progress note from a different physician. Physician to
information observer: “Patient said can't he take Lisinopril…not sure why.” Physician to patient: “Do
you want to take it anyway? I can't find that you have an allergic reaction to it.” [patient
agrees]. Physician enters alert override justification in the order check that appears when
signing the note: ‘Patient states he is not allergic.’

Note. CPRS = Computerized Patient Record System; RMS: Resource Management System; DHCP = Decentralized Hospital Computer Program

Downloaded from hcs.sagepub.com by guest on August 26, 2015

2014 International Symposium on Human Factors and Ergonomics in Health Care: Advancing the Cause 137

TABLE 3: Example patient safety issues at the team coordination / workflow level

Participant type Label Patient Safety Issue

Administrator Delay at “…this is when the organizational skills of the nurses really have to come into play. The
bottleneck in physician may have not yet even seen it, because they know nothing is going to get
workflow decided or even considered until I get A, B, and C. So that patient’s life is literally left with
this consult, on this nurse's desk. Are there opportunities to strengthen that or add an
additional safety element to it? Maybe. They may be adding comments to it [consult], but
you don't know that's there until you open up the actual document.”

Physician Outdated role The physician and resident explain that the he was added as a co-signer for an order
initiated by a nurse but that he is no longer on that service (he rotated off). So the resident
does not sign and types, "no longer seeing this patient". Physician: “If they want
something ordered, they [the nurse] should be paging somebody [not just relying on the
physician to get an view alert].”

Nurse Reliance on “Hopefully if it was urgent, that provider [the provider who initiated the consult] would
Practitioner communication contact you or call you and say, 'hey, I got to get this guy in'. Unfortunately sometimes
via EHR for high that necessarily doesn't always happen. But it can be like 6 days before somebody, you
stakes task know, can triage that actually knows what's going on.”

Administrator Hanging task “Probably the patient safety risk I would think having with consults is those patients who
with ambiguous had their original consult scheduled and then either got cancelled and they no-showed or
responsibility something, and then they didn't get re-scheduled. … And you still see those - they are out
there. They go from a 'scheduled' status to 'inactive' if the appointment gets cancelled and
not immediately rescheduled. So it's not like the consult goes away, it's still there. It's just
how timely people are at addressing those.”

Physician Delayed “So I mean there's really no standard aside that we do say that all progress notes need to
completion of be entered I believe it's within 48 hours - there's a policy on that. Because then it's a
progress notes patient safety issue if there's no documentation after a long time - then if the patient
presents you don't know what went on in the last day or two.”

TABLE 4: Example patient safety issues at the organizational level

Participant type Label Patient Safety Issue

Registered Scheduling “In the VA system, outlying hospitals are allowed to just not have GI [capacity] and they're
Nurse delays due to sending every [GI] patient to us. We're 6 months [out in terms of scheduling] and we used
workload to be able to schedule within a week. When I tell them [the remote request] to fee base it
demands / out, it could take months - this is a patient safety issue. And we're supposed to do it within
patient volume 30 days?

Physician Scheduling Physician: “The number of patients routinely outpaces the available number of
delays due to appointments, further necessitating a triaging of the severity and necessity of each
workload consult. Patients coming from other out of town facilities are given priority, as they have
demands / likely experienced long wait times and transportation issues in getting to [this hospital] and
patient volume this may have inadvertently caused their cancers to progress through no fault of their own.
If there is any area where there is an increased risk to patient safety due to the consult
management procedures, it is likely here.”

Physician Information Physician comments that the patient gets things outside of the VA pharmacy and he
sharing gaps cannot keep track of the medications and gets them confused.
across
organizations

Physician Information Physician states that because the patient received care somewhere else and then came
sharing gaps back he is out of touch with the prescriptions that the patient is on.
across
organizations

Physician Information “Currently the wait times to get documentation from other government facilities [are]
sharing gaps unnecessarily long and complicated which can delay treatment.”
across
organizations

Note. GI = Gastroenterology

Downloaded from hcs.sagepub.com by guest on August 26, 2015

2014 International Symposium on Human Factors and Ergonomics in Health Care: Advancing the Cause
DISCUSSION
The intent of this paper is to show how potential patient
safety issues revealed by this analysis relate to different units
of analysis from an HCI perspective. We found that the
patient safety issues identified varied, depending upon whether
they primarily related to issues at the organizational (e.g.,
information system, support, training, database design),
practice (e.g., team coordination/workflow level,
communication, scope of practice), or EHR interface level
(e.g., graphical user interface, usability, information
organization). A similar conceptual framework of HIT
Management suggests that the ‘Operational Healthcare
Environment’ (organizational resources, technological
availability, financial and governmental factors) and
‘Healthcare Entities’ (e.g., patients, providers, processes,
populations, and organizations) influence ‘HIT Infrastructure
and Strategy’ to achieve ‘Quality Outcomes’, including patient
safety (Doebbeling, Chou, & Tierney, 2006).
Our results demonstrate how the classification of patient
safety issues in the EHR at the level of analysis can help
inform further investigation of patient safety issues. Figure 2
maps methods to the levels of unit of analysis for HCI-focused
studies. Most of these methods were outlined in a literature
review of workflow-related research (Unertl, Novak, Johnson,
& Lorenzi, 2010), and many of them are grounded in human
factors, HCI, cognitive science, and other related fields. The
methods listed in Figure 2 are not meant to be exhaustive, but
rather represent many of those that are commonly used (Unertl
et al., 2010). These methods are organized by two main
groupings: (1) applicable to investigation of patient safety
issues at the EHR-interface level and (2) applicable to the team
coordination / workflow and organizational levels. It is
important to note that the organization of methods is simply a
guide; for example, observation (structured or ethnographic),
surveys, and other methods grouped together in Figure 2 can
also be used to learn about patient safety issues at the EHR-
interface level. However, based on our experience, these
methods are often advantageous when investigating issues at
the team coordination / workflow and organizational levels. A
patient safety issue from each of the three units of analysis is
selected to illustrate this framework and to suggest specific
methods and design for further investigation.
Downloaded from hcs.sagepub.com by guest on August 26, 2015
138
Figure 2. Specific human factors methods that can be employed to
further investigate patient safety issues classified at different units of
analysis. Specific descriptions of these methods are available in the
literature (Kushniruk & Patel, 2004; Unertl et al., 2010).
EHR-Interface Level
A patient safety issued identified at the EHR-level
involved a Registered Nurse reviewing alerts related to consult
requests in the VA’s EHR (Table 2). The primary care
provider initiates a consult request and enters the consult using
the consult package in the EHR. The specialist or other staff
member (the Registered Nurse in this case) is notified of the
new consult request with a computerized alert immediately
after logging into the EHR. She noted: “I print all of the
consults to review with Dr. [X]. Once I click on it [alert], it's
gone. So if I don't print it, it's gone. It's happened [where
she's clicked on it and cannot retrieve it in the EHR] and I
have to call the clerk and tell her 'I clicked off someone - look
out for this' and she'll say 'this is the one I think you missed'.”
This is a potential patient safety issue because the consult
request of a patient who is being referred could get lost.
Usability testing would be a key method to employ to further
investigate this issue (Figure 2). After prototyping one or
more redesign ideas, a comparative usability test of the
redesign(s) and current design could be performed to ensure
that sufficient tracking of pending consults is provided.
Usability testing of clinical information systems can range in
formality from informal qualitative studies with a few
participants for exploratory purposes to a controlled laboratory
simulation with a large number of participants to test for
statistical significance across usability design metrics (Saleem
et al., 2007). In this case, a qualitative usability test with a
small number of participants, using Think Aloud technique,
would likely be sufficient to uncover the majority of usability
issues (Neilsen, 1994), including the issue of appropriately
tracking alerts related to consult requests. After verification of
resolved usability issues, the redesigned could be implemented
in a live-clinic setting.
2014 International Symposium on Human Factors and Ergonomics in Health Care: Advancing the Cause
Team Coordination / Workflow Level
Delay in a specialist seeing a patient being referred is a
potential patient safety issue at the team coordination /
workflow level (Table 3). An administrator noted, “…this is
when the organizational skills of the nurses really have to
come into play. The physician may have not yet even seen it,
because they know nothing is going to get decided or even
considered until I get A, B, and C. So that patient’s life is
literally left with this consult, on this nurse's desk.”
Recommended methods to further explore this issue could
include a combination of ethnographic observation in the
specialty clinic and interviews with the specialist and
supporting staff, as well as primary care physicians who
initiate the consult requests. Ethnographic field observation is
an approach that allows study of a complex system, such as in
situ use of an EHR and its sociotechnical influences in the
organization at large (Hutchins, 1995). This type of long-term
observation in the specialty clinic would allow for a real-time
understanding of how requests for consults are triaged and
factors that may lead to delay in getting a patient scheduled to
be seen by the specialist. Conducting semi-structured
interviews (Patterson et al., 2005) with the various clinicians
involved in the referral process, including the primary care
physicians and the patients themselves, would be quite
complementary to the observational data as it provides
flexibility and gives interviewees an opportunity to identify
and explain important coordination and workflow issues that
may not have surfaced otherwise. Observation and interview
methods were used in this study to reveal the potential patient
safety issues initially. They key difference is scope of the
observation and interviews: broad and exploratory vs. targeted
to a particular application or problem.
Organizational Level
Methods illustrated in Figure 2 associated with team
coordination / workflow are also useful for studying patient
safety issues at the organizational level. For example,
coordination of care with other facilitates may create patient
safety issues (Table 4). One physician specialist noted, “The
number of patients routinely outpaces the available number of
appointments, further necessitating a triaging of the severity
and necessity of each consult. Patients coming from other out
of town facilities are given priority, as they have likely
experienced long wait times and transportation issues in
getting to [this hospital] and this may have inadvertently
caused their cancers to progress through no fault of their own.
If there is any area where there is an increased risk to patient
safety due to the consult management procedures, it is likely
here.” Here, a combination of survey and key-informant
interviews with administrators and clinical leaders at each site
could be used to investigate these organizational-level issues.
A survey could be constructed to assess respondents’
perceptions about organizational factors that impact inter-
facility referral. Key-informant interviews with the same
survey respondents could subsequently explore potential
Downloaded from hcs.sagepub.com by guest on August 26, 2015
139
solutions to lessen the risk of delays in care associated with
inter-facility consultations.
Integration of Levels
Categorization of the patient safety issues we identified in
Table 2-4 were done according to single best fit. However,
many patient safety issues have multiple causative factors that
exist at more than one level depicted in Figure 1.
Furthermore, these levels integrate with each other. Solving a
patient safety issues may require investigation at multiple
levels and necessitate solutions that integrate across the EHR
interface-, team coordination / workflow-, and organizational-
levels. The units of analysis depicted in Figure 1 are meant to
serve as general guides, not hard lines of demarcation between
levels.
Relevant Frameworks
There exist many frameworks that would be helpful for
investigating patient safety issues at each unit of analysis,
including the following examples. Classic usability heuristics
for interface design (Nielsen & Molich, 1990; Nielsen, 1994),
for example, could be considered for EHR-interface-level
issues. Frameworks such as cognitive engineering (Patterson,
Bozzette, Nguyen, Gomes, & Asch, 2003; Woods & Roth,
1988) and distributed cognition (Hutchins, 1995; Saleem et al.,
2009) expand the unit of analysis beyond the traditional
computer system and user to include workflow and team
coordination. At the organizational level, the sociotechnical
framework can be used to distinguish three integrated parts of
a work system: social, technical, and environmental
subsystems (including internal and external; Kleiner, 2008;
Saleem et al., 2011). Choosing an appropriate framework, in
addition to specific methods, can help guide the investigation
of potential patient safety issues at each unit of analysis
illustrated in Figure 1. The study of human social behavior
and its interface with technology is itself a social construction
and sometimes an artifact of the frameworks and methods we
use to study such phenomena. Sir Arthur Eddington, an early
20th century philosopher put it well when he asserted:
“Suppose that an ichthyologist trawls the seas using
a fish net of two inch mesh: then fish less than two
inches in length will escape him and he will find
when he pulls up the net only fishes two inches long
or more. This…may tempt him to conclude that the
world contains no fish of smaller size; He may
generalize and announce, ‘All fish are two inches
long or more,’ and until he has the sense to examine
his own methods of fish-catching, he may fail to
realize that these methods, not the ichthyological
facts are what have led him to the conclusion.” (Sir
Arthur Eddington, 1939).
2014 International Symposium on Human Factors and Ergonomics in Health Care: Advancing the Cause
Limitations
Data was collected from a relatively small convenience
sample of diverse users of computerized consults and clinical
documentation from a single governmental organization with a
unique informatics infrastructure and consultation and
documentation functionalities. Therefore, it is not clear that the
frequency and examples will generalize to other settings or
EHRs. Nevertheless, multiple issues were identified at all three
levels of the conceptual framework which are based on
knowledge obtained from a variety of high consequence
settings beyond healthcare, and thus suggest that it is likely
that all levels will also exist in other related healthcare
settings. Some patient safety issues identified and categorized
at the team coordination / workflow and organizational levels
may not explicitly relate to EHR function and usability.
However, we argue that by taking a holistic system
perspective, these issues impact EHR interface design. The
sociotechnical components of the organization at large inform
the design of the EHR at the interface level.
Conclusion
A variety of options are available to consider when
selecting the unit of analysis for the study of clinical
information systems and related patient safety issues. There is
general consensus in the scientific community that for
optimum results, the question should always determine the
methods of study and not the reverse. Using an HCI
perspective to study user behavior, the unit of analysis can
narrowly focus on the interaction between a clinical user and
computer tool. When studying a clinical information system in
its naturalistic setting, it is often necessary to expand the unit
of analysis to include related paper-based and technological
artifacts, communicative pathways and coordination
mechanisms with other clinicians, and the greater organization
influences. This study demonstrated that potential patient
safety issues can be uniquely categorized by these three levels
of analysis. Recommendations were made for each level of
analysis for further investigation to better understand the issues
and determine whether solutions ameliorate the frequency of
negative outcomes.
ACKNOWLEDGEMENTS
The authors thank Stephanie Adams, Paul Blades, Brian
Foresman, Adam Neddo, and Alissa Russ for their various
contributions and advice related to this work. This material is
based upon work supported by the Department of Veterans
Affairs, Veterans Health Administration, Office of Research
and Development, Health Services Research and Development
(HSR&D) Service [grant numbers CDA 09-024-1, HFP 04-
148]. The views expressed in this article are those of the
authors and do not necessarily reflect the position or policy of
the Department of Veterans Affairs or the United States
government.
Downloaded from hcs.sagepub.com by guest on August 26, 2015
140
REFERENCES
Ash, J. S., Berg, M., & Coiera, E. (2004). Some unintended
consequences of information technology in health care: the
nature of patient care information system-related errors.
J.Am.Med.Inform.Assoc., 11, 104-112.
Borycki, E. M. & Kushniruk, A. W. (2010). Towards an integrative
cognitive-socio-technical approach in health informatics:
analyzing technology-induced error involving health information
systems to improve patient safety. Open.Med.Inform.J, 4, 181-
187.
Cheung, K. C., van, d., V, Bouvy, M. L., Wensing, M., van den Bemt,
P. M., & de Smet, P. A. (2013). Classification of medication
incidents associated with information technology.
J.Am.Med.Inform.Assoc., 21 (e1), e63-e70.
Doebbeling, B. N., Chou, A. F., & Tierney, W. M. (2006). Priorities and
strategies for the implementation of integrated informatics and
communications technology to improve evidence-based practice.
J.Gen.Intern.Med., 21 Suppl 2, S50-S57.
Elbardissi, A. W., Wiegmann, D. A., Dearani, J. A., Daly, R. C., &
Sundt, T. M., III (2007). Application of the human factors analysis
and classification system methodology to the cardiovascular
surgery operating room. Ann.Thorac.Surg., 83, 1412-1418.
Hutchins, E. (1995). Cognition in the Wild. Cambridge, MA: MIT Press.
Jha, A. K. & Classen, D. C. (2011). Getting moving on patient safety--
harnessing electronic data for safer care. N.Engl.J Med., 365,
1756-1758.
Kleiner, B. M. (2008). Macroegonomics: work system analysis and
design. Hum.Factors, 50, 461-467.
Kushniruk, A. W. & Patel, V. L. (2004). Cognitive and usability
engineering methods for the evaluation of clinical information
systems. J Biomed.Inform., 37, 56-76.
McMullen, C. K., Ash, J. S., Sittig, D. F., Bunce, A., Guappone, K.,
Dykstra, R. et al. (2011). Rapid assessment of clinical information
systems in the healthcare setting: an efficient method for time-
pressed evaluation. Methods Inf.Med., 50, 299-307.
Mikkelsen, K. L., Thommesen, J., & Andersen, H. B. (2013). Validating
the Danish adaptation of the World Health Organization's
International Classification for Patient Safety classification of
patient safety incident types. Int.J Qual Health Care., 25, 132-
140.
Neilsen, J. (1994). Estimating the number of subjects needed for a
thinking aloud test. Int J Hum Comput Interact, 41, 385-397.
Nielsen, J. (1994). Usability Engineering. San Diego: Academic Press.
Nielsen, J. & Molich, R. (1990). Heuristic evaluation of user interfaces.
Proceedings from ACM CHI'90 Conference, 249-256.
Patterson, E. S., Bozzette, S. A., Nguyen, A. D., Gomes, J. O., &
Asch, S. M. (2003). Comparing Findings from Cognitive
Engineering Evaluations. Proceedings of the Human Factors and
Ergonomics Society Annual Meeting, 47, 483-487.
Patterson, E. S., Cook, R. I., & Render, M. L. (2002). Improving patient
safety by identifying side effects from introducing bar coding in
medication administration. J Am.Med.Inform.Assoc., 9, 540-553.
Patterson, E. S., Doebbeling, B. N., Fung, C. H., Militello, L., Anders,
S., & Asch, S. M. (2005). Identifying barriers to the effective use
of clinical reminders: bootstrapping multiple methods.
J.Biomed.Inform., 38, 189-199.
Roth E.M. & Patterson E.S. (2005). Using Observational Study as a
Tool for Discovery: Uncovering Cognitive Demands and Adaptive
Strategies. In H.Montgomery, R. Lipshitz, & B. Brehmer (Eds.),
How Professionals Make Decisions (pp. 379-393). Mahwah, New
Jersey: Lawrence Erlbaum Associates, Inc.
Saleem, J. J. (2009). Circumventing Health IT: Identifying Patient
Safety Risks. Oct 2009 - Sept 2014. VA HSR&D. Grant #: CDA
09024-1.
Saleem, J. J., Adams, S., Frankel, R. M., Doebbeling, B. N., &
Patterson, E. S. (2013). Efficiency strategies for facilitating
computerized clinical documentation in ambulatory care.
Stud.Health Technol.Inform., 192, 13-17.
Saleem, J. J., Flanagan, M. E., Russ, A. L., McMullen, C. K., Elli, L.,
Russell, S. A. et al. (2013). You and me and the computer makes
three: variations in exam room use of the electronic health record.
J.Am.Med.Inform.Assoc., 21 (e1), e147-e151.
2014 International Symposium on Human Factors and Ergonomics in Health Care: Advancing the Cause 141

Saleem, J. J., Patterson, E. S., Militello, L., Anders, S., Falciglia, M.,
Wissman, J. A. et al. (2007). Impact of clinical reminder redesign
on learnability, efficiency, usability, and workload for ambulatory
clinic nurses. J.Am.Med.Inform.Assoc., 14, 632-640.
Saleem, J. J., Patterson, E. S., Militello, L., Render, M. L., Orshansky,
G., & Asch, S. M. (2005). Exploring barriers and facilitators to the
use of computerized clinical reminders. J.Am.Med.Inform.Assoc.,
12, 438-447.
Saleem, J. J., Russ, A. L., Neddo, A., Blades, P. T., Doebbeling, B.
N., & Foresman, B. H. (2011). Paper persistence, workarounds,
and communication breakdowns in computerized consultation
management. Int.J Med.Inform., 80, 466-479.
Saleem, J. J., Russ, A. L., Sanderson, P., Johnson, T. R., Zhang, J., &
Sittig, D. F. (2009). Current challenges and opportunities for
better integration of human factors research with development of
clinical information systems. Yearb.Med.Inform., 48-58.
Sir Arthur Eddington (1939). The Philosophy of Physical Science,
Cited in Stephen Toulmin (1960), The Philosophy Science: An
Introduction. New York, p. 125: Harper Torchbooks.
Sittig, D. F. & Singh, H. (2012). Electronic health records and national
patient-safety goals. N.Engl.J Med., 367, 1854-1860.
Unertl, K. M., Novak, L. L., Johnson, K. B., & Lorenzi, N. M. (2010).
Traversing the many paths of workflow research: developing a
conceptual framework of workflow terminology through a
systematic literature review. J Am.Med.Inform.Assoc., 17, 265-
273.
Wiegmann, D. A. & Shappell, S. A. (2003). A human error approach to
aviation accident analysis. The human factors analysis and
classification system. Burlington, VT: Ashgate.
Woods, D. D. & Roth, E. M. (1988). Cognitive engineering: Human
problem solving with tools. Human Factors, 30, 415-430.
World Alliance for Patient Safety (2009). The conceptual framework for
the International Classification for Patient Safety (v1.1) Final
technical report.
http://www.who.int/patientsafety/taxonomy/icps_full_report.pdf.
Xiao, Y. & Vicente, K. J. (2000). A framework for epistemological
analysis in empirical (laboratory and field) studies. Hum.Factors,
42, 87-101.

Downloaded from hcs.sagepub.com by guest on August 26, 2015

View publication stats