WAGNER, HOGAN, Accuracy of Medication Data

Research Paper

The Accuracy of Medication

Data in an Outpatient
Electronic Medical Record

Downloaded from https://academic.oup.com/jamia/article/3/3/234/732087 by guest on 24 February 2022

MICHAEI., M. WAGNER: MD, PhD: WILLIAM R. HOGAN, MD

Abstract
Objective: To measure the accuracy-of medication records stored in the electronic medical record
(EMR) of an outpatient geriatric center. The authors analyzed accuracy from the perspective of a
clinician using the data and the perspective of a computer-based medical decision-support
system (MDSS).
Design: Prospective cohort study.
Methods: The EMR at the geriatric center captures medication data both directly from clinicians
and indirectly using encounter forms and data-entry clerks. During a scheduled office visit for
medical care, the treating clinician determined whether the medication records for the patient
were an accurate representation of the medications that the patient was actually taking. Using
the available sources of information (the patient, the patient’s vials, any caregivers, and the
medical chart), the clinician determined whether the recorded data were correct, whether any
data were missing, and the type and cause for each discrepancy found.
Results: At the geriatric center, 83% of medication records represented correctly the compound,
dose, and schedule of a current medication; 91% represented correctly the compound. 0.37
current medications were missing per patient. The principal cause of errors was the patient
(36.1% of errors), who misreported a medication at a previous visit or changed (stopped, started,
or dose-adjusted) a medication between visits. The second most frequent cause of errors was
failure to capture changes to medications made by outside clinicians, accounting for 25.9% of
errors. Transcription errors were a relatively ucommon cause (8.2% of errors). When the accuracy
of records from the center was analyzed from the perspective of a MDSS, 90% were correct for
compound identity and 1.38 medications were missing or uncoded per patient. The cause of the
additional errors of omission was a free-text “comments” field-which it is assumed would be
unreadable by current MDSS applications-that was used by clinicians in 18% of records to
record the identity of the medication.
Conclusions: Medication records in an outpatient EMR may have significant levels of data error.
Based on an analysis of correctable causes of error, the authors conclude that the most effective
extension to the EMR studied would be to expand its scope to include all clinicians who can
potentially change medications. Even with EMR extensions, however, ineradicable error due to
patients and data entry will remain. Several implications of ineradicable error for MDSSs are
discussed. The provision of a free-text “comments” field increased the accuracy of medication
lists for clinician users at the expense of accuracy for a MDSS.
n JAMIA. 1996;3:234-244.

Affiliations of the authors: Section of Medical Informatics Correspondence and reprints: Michael Wagner, MD, PhD, Sec-
(MMW) and Department of Medicine (WRH), University of tion of Medical Informatics, B50A Lothrop Hall, 190 Lothrop
Pittsburgh School of Medicine, Pittsburgh, PA. Street, Pittsburgh, PA 15261. e-mail: mmw@med.pitt.edu

Presented at the American Medical Informatics Association Received for publication: 10/12/95; accepted for publication:
Spring Congress, Cambridge, MA, June 1995. l/10/96.
Journal of the American Medical Informatics Association
Data collected by electronic medical records (EMRs)
are used by clinicians, researchers, managers, process-
improvement teams, and decision-support systems.
Although it is self-evident that the accuracy of such
data is important to the activities of these users, there
have been only sporadic reports about the accuracy
of data in EMRs.‘-~ These studies suggest that, in gen-
eral, data accuracy is a significant problem.
We are interested in the accuracy of patient medica-
tion data because these data are important clinically
(and managerially), yet their accuracy has not been
the subject of study. The importance of medication
data is related to the contribution of medications to
the cost of health care, and the frequency of adverse
events related to medications.6,7 Clinicians need ac-
curate medication data to avoid drug interactions, to
monitor patient compliance and therapeutic response,
to diagnose potential side effects, and to respond to
requests for refills. Decision-support systems use
medication data to check for drug-drug, drug-al-
lergy, and drug-problem interactions, for diagnosis,
and to encourage cost-effective prescribing. In this re-
search, we developed methodologies for the analysis
of medication data accuracy and used them to char-
acterize the accuracy of medication data in an opera-
tional outpatient EMR.
Methodologic Issues
The standard methodologic approach to the measure
of data accuracy, which we follow, characterizes data
accuracy along two dimensions that are termed ge-
nerically “correctness” and “comp1eteness.“‘~*~” Cor-
rectness is defined by Wiederholt and Perrault 10 as the
accuracy with which events are recorded and entered.
Completeness is a measure of the prevalence of missing
events or observations. The reason researchers use
dual measures is that incomplete data and incorrect
data may result in different clinical errors.
Researchers have employed a variety of “gold stan-
dards” against which to measure correctness and
completeness, partly for pragmatic reasons and partly
because the appropriate gold standard is often data-
dependent. For example, a gold standard for birth
date might be a birth certificate, whereas a gold stan-
dard for childhood vaccinations might be a parental
interview or direct observation of vaccination.’ The
standard adopted by most researchers has been un-
blinded chart review.1,3,5,9,11-13
A new methodologic issue in the investigation of
medication data accuracy results from the fact that a
medication record is a compound data type; that is, it
Volume 3 Number 3 May / Jun 1996 235
comprises fields for storing the identity of the medi-
cation, the dose, the schedule of administration, the
route of administration, the number of refills, and the
name of the prescriber (e.g., Table 1). Each field can
be accurate or inaccurate, some fields may be op-
tional, and some fields may be used only by certain
applications and not by others. The methodologic
question is which subset of fields to include in the
definition of a correct record. If we were to require
Downloaded from https://academic.oup.com/jamia/article/3/3/234/732087 by guest on 24 February 2022
that all fields be correct, we would overestimate the
error experienced by any particular application. On
the other hand, if we were to use a single application-
specific definition, our results would apply only to
that application. These observations suggested that
we should use multiple definitions of data accu-
racy-corresponding to a distinct subset of fields for
each application of interest.
A second methodologic issue is the meaning of a med-
ication record in an outpatient EMR. We have ob-
served informally that physicians use medication lists
in outpatient charts both prescriptively (i.e., to record
what they instructed the patient to do) and descrip-
tively (i.e., to record what the patient is actually tak-
ing). For example, the entry phenobarbital 60 mg 1 tab
po q8k could correspond to a situation in which 1) the
patient is not taking this medication in this dose and
on this schedule, but was instructed to do so; 2) the
patient is taking this medication in this dose and on
this schedule, but was instructed otherwise; or 3) the
patient is taking the medication exactly as instructed.
Clinicians can sometimes resolve this ambiguity using
additional sources of information such as progress
notes, their memories, or congruence with how they
prescribe medications. To study accuracy, however,
we must select one of these perspectives. For this
study, we viewed medication data descriptively be-
cause a representation of the true state of a patient is
important for reasoning about side effects and dose-
response effects (e.g., warfarin-INR). A second reason
why we elected to view medication data descriptively
is that we wanted to analyze accuracy from the per-
spective of a decision-support system, which often re-
quires a representation of the state of the patient,
rather than what the physician prescribed.
Methods
Setting
Benedum Geriatric Center (BGC) is a multidiscipli-
nary geriatrics clinic at a university medical center
serving a patient population of approximately 2,000.
The clinic staff includes geriatricians, social workers,
236 WAGNER,
psychiatrists, and nurse practitioners (approximately
30 clinicians total). The BGC EMR is a locally de-
signed and developed system running as a Windows
application on 22 workstations in the clinic. The EMR
reproduces the functionality of the paper record and
of the appointment schedule with electronic forms
that resemble a face sheet, an appointment status
board, and visit records. The EMR captures patient
data by direct entry of data by clinicians at computer
terminals at the time of a visit (representing 12% of
medication data capture),* direct entry of data at com-
puter terminals by clinicians during phone contacts
with patients or providers (14%), and data entry by
licensed nurse practitioners from structured encounter
forms (75%). Regardless of who enters medication
data, the procedure is the same. Working in a form,
the user enters the first four letters of the generic or
the brand name of a medication, and is then presented
with a pick list of matching formulary items. From
that list, the user selects the desired formulary item
(e.g., ampicillin, 250-mg tablet) and enters the sched-
ule of administration and comments. There are field-
level validation procedures for each field.
The BGC EMR had been in operation for 18 months
at the. time of the study. Although clinicians were
changing from indirect to direct data entry during this
period, at the time of the study no clinician was
changing modality. At the time of the study, four of
six nurses and nurse practitioners were entering data
directly. Three of ten geriatricians whose patients
were included in the study were entering data di-
rectly. The study included patients whether or not
their providers entered data directly into the EMR.
Inclusion Criteria
We included any patient with an appointment for ei-
ther a medical nurse practitioner or a geriatrician dur-
ing a three-week study period in December 1994. One
author is a geriatrician in the clinic and his patients
were eligible to be included in the study
Exclusion Criteria
We excluded new patients by excluding all patients
with empty medication lists. This exclusion criterion
also excluded patients who had been seen previously,
but not since the inception of the EMR. It was appro-
*We measured the proportion of changes in medication data
(medication starts, medication stops, and medication adjust-
ments) captured by different routes for a sample of 1,114 pa-
tients. We did this for the month preceding this study. We found
a total of 629 modifications of the medication data for those
patients in that month, a rate of 0.57 modifications per patient
per month.
HOGAN, Accuracy of Medication Data
priate to exclude both types of patients because there
had been no opportunity to record their medications
in the EMR. We used this criterion because it could
be implemented readily on a computer. This exclusion
criterion could inappropriately exclude patients who
were taking medications, but for whom the clinic staff
had failed to record any medications, and patients
who were taking no medication. Additionally, we ex-
cluded subsequent visits of those patients who had
Downloaded from https://academic.oup.com/jamia/article/3/3/234/732087 by guest on 24 February 2022
already been seen during the study period, because
the study may have influenced the accuracy of their
lists.
Data Collection
On the day before a scheduled visit, the EMR system
determined automatically the set of patients who sat-
isfied the inclusion and exclusion criteria for the
study, and printed a study form (Appendix) for each
eligible patient. The form listed the patient’s current
medications as recorded in the EMR with check boxes
and coding boxes for the recording of errors, and their
types and causes. It provided space for a clinician to
record any missing medications. It also contained
printed instructions, and a legend for the types and
causes of errors. One investigator met individually
with each clinician and explained the purpose of the
study, reviewed the instructions, gave an example of
an error corresponding to each type and cause, and
reviewed the use of the form. The clinicians were told
that the purpose of the study was to determine the
accuracy of medication data in the EMR. We in-
structed clinicians that any discrepancy between the
medication list and what the patient was actually tak-
ing should be noted. During the visit, the clinician
marked each listed medication as correct or, if incor-
rect, coded both the type and the cause of the error
according to the legend on the form. The clinician
wrote in missing medications and coded the cause of
the error.
Main Outcome Measures
A correct medication record from the perspective of a
clinician was determined as follows. A study form
presented the same data from the EMR that are shown
to clinicians during routine clinical care, namely, the
formulation, sig, and comments fields from the EMR
records (Table 1, rows 14-20). The clinician deter-
mined whether the patient was actually taking the
medication as described. We refer to this definition of
error as clinician perspective, medication, schedule, and
dose. The clinician had available, when making this
determination, some or all of the following sources of
information: his or her knowledge of the patient, the
Journal of the American Medical Informatics Association Volume 3 Number 3 May / Jun 1996 237

Table 1 n

Medication-record Schema in the Benedum Geriatric Center Electronic Medical Record

Field Name Type Description

1 medication-id Number (Long) Primary Key

2 pid Number (Long) Foreign key for the patient record
3 med-ancestor Number (Long) Medication-id of the record from which this record was created due to
a dose change or correction
4 created-date Date/Time Date record was created
5 created-who Text Name of provider who created this record

Downloaded from https://academic.oup.com/jamia/article/3/3/234/732087 by guest on 24 February 2022

6 creator-id Number (Long) Foreign key to provider table
7 created-reason Text Whether the event was a dose change or correction
8 created-encounterID Number (Long) Foreign key to the encounter table
9 archived-date Date/Time Date record became inactive
10 archived-reason Text Reason why record made inactive
11 archived-who Text Name of provider who made record inactive
12 archived-encounterID Number (Long) Foreign key to encounter table
13 archiver-id Number (Long) Foreign key to provider table
14 formulary-id Number (Long) Foreign key to formulary
15 sig-value Text Number of units of medication
16 sig-units Text Units (e.g., tablet)
17 sig-route Text Route of administration
18 sig-interval Text Interval of administration
19 sig-pm Text Whether medication is pm
20 sig-comments Text Free-text comments
21 who Text Name of provider
22 provider-id Number (Long) Foreign key to provider table
23 disp-number Number (Int) Quantity dispensed
24 disp-units Text Units dispensed (e.g., tablets)
25 disp-refills Number (Int) Number of refills
26 disp-date Date/Time Date of last dispensing
27 disp-encounterID Number (Long) Foreign key to encounter table
28 disp-who Text Name of provider who last dispensed
29 dispenser-id Number (Long) Foreign key to provider table
30 disp-p Yes/No Whether the medication requires a prescription
31 verifieddate Date/Time Date the medication was last checked with the patient
32 verified-who Text Name of provider who checked medication
33 verified-encounterID Number (Long) Foreign key to encounter table
34 verifier-id Number (Long) Foreign key to provider table

patient, a caregiver, the patient’s prescription
and the patient’s medical records.
We defined six categories for types of errors, and seven
categories for causes of errors. We developed these cat-
egories from an analysis of a sample of patient charts,
and refined them in a one-week pilot study. We de-
fined these categories to be mutually exclusive and
exhaustive, and to correspond to correctable causes of
errors. We collected data describing the severity of er-
rors in the pilot study, but there was significant vari-
ability in clinician compliance and reliability with this
metric and we did not include it in the final study.
The categories for types of errors were: 1) a recorded
medication that the patient was not. taking (Extra
Listed), 2) an error in the name of the formulation,
such as quinine instead of quinidine (Med), 3) a dif-
ference in the strengths of the formulation recorded
bottles, and that being taken by the patient (Dose), 4) a differ-
ence in the sigs, or schedule of administration for a
medication (Schedule), 5) duplicate entries (Duplicates),
and 6) an unrecorded medication that the patient was
taking (Not Listed). These category labels, and those
listed in the next paragraph, correspond to the codes
provided on the study form (there was no code for
the Not Listed error type because clinicians wrote in
the names of missing medications).
The categories for causes of errors were: 1) uncaptured
changes made by hand in the paper chart (Handwrit-
ten), 2) data-entry errors (Data entry), 3) patient-initi-
ated changes in medications (Patient), 4) unrecorded
changes made by a clinic physician (Clinic MD), 5)
changes made by medical center clinicians outside of
the clinic (UPMC MD), 6) changes made by non-
medical center clinicians (Outside MD), and 7) Un-
238
Table 2 n and MDSS, medication only). To characterize complete-
Results Using clinician perspective, medication,
schedule, and dose Definition of Accuracy
Number of patients in study 117
Number of medication records 663
Average number of medications/patient 5.67
Correctly listed medications 549
Number of correct medications/patient 4.69
Number of missing medication records 43
Correctness 0.83 (549/663)
Completeness 0.37 (43/117)
known. There was no code on the study form for the
Unknown cause; we added this type during data anal-
ysis to encode a response written in by clinicians
when they did not know the cause.
As a secondary analysis, we removed Dose and Sched-
ule errors from our original analysis. This adjustment
in the error rate gave us a measure of accuracy for the
identity of the medication only We refer to this defi-
nition as clinician, medication only. To obtain an esti-
mate of data error from a decision-support perspec-
tive, we took the clinician, medication only analysis and
reclassified as error any medication record where the
identity of the medication was only available in a free-
text “comments” field. Information in such a free-text
field, although usable from a clinician’s perspective,
would be unintelligible to a medical decision-support
system (MDSS). We refer to this third definition as
MDSS, medication only. This definition of accuracy
based on encoding of the identity of the compound
alone uses only field 14 in Table 1.
Statistical Methods
To obtain a measure of correctness, we computed the
proportion of all medication records that were correct.
This measure would be expected to be informative to
a clinician who needs to know the probability that a
patient is actually taking a recorded medication. We
computed this proportion using our main and sec-
ondary definitions of correctness (clinician perspective,
medication, schedule, and dose; clinician, medication only;
Table 3 n
Indirect versus Direct Data Entry
Direct
Correctness (clinician, medication, schedule, dose)
Correctness (clinician, medication only).
Completeness
WAGNER, HOGAN, Accuracy of Medication Data
ness of medication lists, we computed the mean num-
ber of medications per patient for which no medica-
tion record existed. This measure would inform a
clinician of the number of medications likely to be
missing from a medication list. Although this quantity
technically measures the incompleteness of medication
lists, we refer to it in this paper as completeness, mean
because completeness is a conventional term and this
Downloaded from https://academic.oup.com/jamia/article/3/3/234/732087 by guest on 24 February 2022
measure cannot be converted into a measure of com-
pleteness.
We analyzed the data to test whether direct clinician
entry was associated with a different level of error
than that for encounter form data entry. We compared
the accuracy of the medication records of those pa-
tients whose primary clinician entered medications di-
rectly with the accuracy of the records for those pa-
tients whose primary clinician used encounter forms.
We also tested whether the correctness of prescription
medications was equal to that of nonprescription
medications. For the comparisons of correctnesses, we
calculated a 95% confidence interval (CI) for the dif-
ferences between the error rates using a test for dif-
ferences between two proportions.14 We compared the
completenesses of direct and indirect data entry using
a two-sample t-test for independent samples with
equal variances.15 We tested for significant differences
among the causes of errors by computing 95% CIs and
testing for overlap.
To facilitate comparison of our results with those of
published studies about the accuracy of medication
lists in paper-based medical records, we computed the
proportion of patient medication lists that were both
complete and correct (i.e., all of a patient’s current
medications were recorded correctly). This is the sta-
tistic most commonly reported in that literature. We
refer to it as the proportion of error-free lists.
Results
In the three-week study period at BGC, 208 of 399
scheduled visits satisfied the inclusion criteria. Of
these visits, we excluded 28 because the patient had
Physician Data-entry
Entry Personnel Difference 95% CI
0.83 (106/127) 0.83 (443/536) 0.00 -0.07, 0.07
0.91 (115/127) 0.92 (494/536) 0.01 -0.04, 0.06
0.30 (6/20) 0.38 (37/97) 0.08 -0.26, 0.42
Journal of the American Medical Informatics Association Volume 3 Number 3 May / Jun 1996 239

Table 4 n

Prescription versus Nonprescription Medications

Prescription Nonprescription
Medications Medications Difference 9.5% CI
Correctness (clinician perspective, medication, 0.82 (392/479) 0.85 (156/184) 0.03 -0.03, 0.09
schedule, and dose)
Correctness (clinician, medication only) 0.95 (453/479) 0.90 (166/l&1) 0.05 0.00, 0.09

Downloaded from https://academic.oup.com/jamia/article/3/3/234/732087 by guest on 24 February 2022

an empty medication list (21 patients were new pa-
tients, seven patients were not taking medications,
and three patients were taking medications but had
erroneously empty lists). We excluded an additional
44 visits because the patient did not keep the appoint-
ment. We distributed 136 study forms to physicians,
of which they completed and returned 119 (response
rate 89.3%). We excluded two of these visits because
the patient had been seen already during the study
period. There were on average 5.67 medications re-
corded per patient. The proportion of medication rec-
ords that were correct was 0.83. There were 0.37 un-
recorded medications per patient (Table 2).
Of the 117 patients, 20 (17%) saw the three physicians
who used the computer directly to record visit notes.
The proportion of medication records that were cor-
rect for this group of patients (83%) was not signifi-
cantly different from the proportion for the other
patients (83%). The completenesses were also not sig-
nificantly different (Table 3). Analysis of prescription
versus nonprescription medications did not show a sta-
tistically significant difference in correctness (Table 4).
Figure 1 cross-tabulates the types and causes of errors.
In Figure 1, we aggregate the dose and schedule types
of errors because clinicians seemed to confuse the two
categories of errors (based on an informal comparison
of study forms with the patient records). The most
common cause of discrepancies between data in the
EMR and what the patient was actually taking was
the patients themselves. Because patients stop, start,
and adjust medications on their own between office
visits, and occasionally misreport medications, the
medication records can diverge from the actual med-
ication usage of the patient. The second most common
cause was medication changes made by clinicians in
the medical center, but outside of the clinic (e.g., sub-
specialists).
To test whether accuracy varied as a function of time
from the last clinic visit, we plotted the proportion of
medications that were correct for each of the 117 pa-
tients against the number of days since the last clinic
visit (Fig. 2). The correlation coefficient for this com-
parison was -0.1, not significantly different from zero
(p > 0.2), suggesting that either time does not influ-
ence accuracy (unlikely), the intervisit intervals were
too short for significant effect on accuracy to emerge,
or that the relationship between visit intervals and
medication accuracy is complex.
Table 5 reports the results of our secondary analyses.
When we required only that the identity of the com-

'OT T
60
.$ 50
t 40
5 30
‘4 20
10
0
Patient UPMC MD Handwritten Clinic MD Data Entry Unknown Outside MD
TOTALS
29 (18.4%) 8 (5.1%) 3 (1.9%) 2 (1.3%) 1 (0.6%) 0 (0.0%) 1
7 (4.4%) 19 (12.0%) 7 (4.4%) 4 (2.5%) 0 (0.0%) 3 (1.9%) 3
(8.9%) 6 (3.8%) 11 (7.0%) 7 (4.4%) 7 (4.4%) 8 (5.1%) 2
7 (4.4%) 1 (0.6%) 0 (0.0%) 1 (0.6%) 5 (3.2%) 1 (0.60/j 1
T?EEq 57 (36.1%) 34 (21.5%) 21 (13.3%) 14 (8.9%) 13 (8.2%) 12 (7.6%) 7 (4.4%)1

Fi gure 1 Causes and types of medication list errors (error bars show 95% confidence intervals).
240 WAGNER, HOGAN, Accuracy of Medication Data

pound be correct, correctness increased to 0.92. When suggest that access to the EMR by those who are al-
analyzed from the perspective of a MDSS, correctness lowed to make changes must be improved (e.g., from
was 0.90. Completeness for the clinician, medication home, car). Such an expansion would require solu-
only definition was 0.37. From the MDSS perspective, tions to technical problems such as how to capture
whenever a medication was not coded, it was an ad- changes made by clinicians when they are at home or
ditional error, hence, the number of incompleteness in their cars using cellular telephones.
errors per patient increased to 1.38. Table 5 also shows
the proportion of error-free lists. We compare this rate
with the rates described in the literature for paper- In this study, we used the determinations of the treat-

Downloaded from https://academic.oup.com/jamia/article/3/3/234/732087 by guest on 24 February 2022

based medical records in the Discussion. ing clinicians as a gold standard. An ideal gold stan-
dard for this type of study would be an independent,
blinded observer using multiple sources of informa-
Discussion tion such as pharmacy records, patient interviews,
and chart reviews. Although our method used mul-
Error Rates and Causes of Error tiple sources of data (the clinicians had access to the
patient, often the patient’s prescription bottles, care-
The principal result of this study is the demonstration givers, and the paper chart), a potential limitation of
of moderately high levels of error in the medication our design is that clinicians may have undercounted
records of an EMR. A second result is that the main their own errors. However, in this center, more than
causes of error are problems in the capture of medi- one clinician is responsible, for the maintenance of any
cation data, not problems with the accurate entry of one patient’s medication list; therefore, by identifying
data. Many of the potentially remediable errors that an error, a clinician did not have to implicate himself
we observed at the geriatric center were caused by the or herself as the source. Also, many study forms were
limited scope of the EMR (only practitioners from the completed by medical residents working in the clinic,
clinic had accounts) and limited access (access to the who were not present when data they were judging
EMR was available only from workstations within the were collected, and therefore were presumably im-
clinic). An implication of this study for single-clinic partial. If present, this bias would be in the direction
EMRs is that to improve data accuracy, the orga- of minimizing observed data error and therefore
nizational scope of the EMR must grow as large as would not change our principal conclusion that data
possible, ideally communitywide (to include all prac- error is present in outpatient EMRs. This type of bias
titioners who potentially make changes in medica- may disproportionately affect the physician category
tions). This observation suggests the need for citywide of causes of errors; however, this influence would not
or regional data networks. Additionally, our results change our conclusion that limited scope is an im-

<cc<<
‘.. c<..-<.<<...
. ..-.-.-.
T

II.... ...... .
l
S?* ,; .
. . . .
‘. .
.
. <. .
. .

l . . . .

I .
l
.

0 +- b-r-.-..

0 50 100 150 200 250 300 350

Days since previous visit

F I gure 2 Proportion of correct medication records for each of 117 patients as a function of the time since the patient
was last seen in clinic.
Journal of the American Medical Informatics Association
portant source of medication data error in this EMR.
A second potential limitation is that clinicians may not
have completed the forms accurately due to lack of
interest or time. During the pilot phase of this study,
we informally correlated the physician responses with
a chart review and found that the clinicians did not
make obvious mistakes. A third potential source of
bias is the inclusion of patients of one of the authors
in. the study; however, only five such patients were
included. Our exclusion criteria excluded three pa-
tients who had empty medication lists in error. These
patients were taking a total of four medications,
which would not have changed our completeness
.measures significantly.
We did not estimate the severity of errors. We at-
tempted to elicit such judgments during the pilot pe-
riod from clinicians, but found that our clinicians
found such a question distracting when they were see-
ing patients (whereas determining what medications
a patient i s t aki ng is their job). Our breakdown of er-
rors into types (e.g., dose, omitted, schedule, medi-
cation) could be used by those interested in forming
a summative judgment of whether the data accuracy
was good or bad.
Generality of Results
It is likely that other single-clinic outpatient EMRs
will have levels of error at least as high as that which
we found. The geriatric center had many characteris-
tics that may have enhanced accuracy. Clinicians share
responsibility for different aspects of a patient’s care.
Therefore, they are careful to record medications.
Moreover, geriatricians by training are attuned to the
risk of medication errors in the elderly. Finally, one of
the authors (MMW) developed this EMR specifically
to address a problem of medication data error. The
fact that we measured error at a maximum time from
the last visit may inflate the error levels; however, our
analysis showed that this effect was small.
We can use our categorization of errors to obtain es-
timates of medication data error for EMRs with
broader scopes and more effective data capture mech-
anisms. For example, by removing errors of types
UPMC MD and Outside MD, we can estimate that 0.86
of listed medications would be correct and 0.18 med-
ications would be missing per list in an enterprise-
wide system. By further adjusting for errors due to
access to the EMR (Clinic MD, Handwritten), we expect
that correctness would increase to 0.89 of listed med-
ications and incompleteness would decrease to 0.09
per list. We can also use our categorization of errors
to estimate the effect of any particular intervention
Volume 3 Number 3 May / Jun 1996 241
Table 5 n
Medication List Accuracy According to Alternate
Definitions
Correctness (clinician perspective, medication, 83% (549/663)
schedule, and dose)
Correctness (clinician, medication only) 92% (609/663)
Correctness (MDSS, medication only) 90% (595/663)
Completeness (clinician, medication only)’ 0.37 (43/117)
Completeness (MDSS, medication only) 1.38 (162/117)
Correctness (paper-based definition)t 43.5% (51/117)
Downloaded from https://academic.oup.com/jamia/article/3/3/234/732087 by guest on 24 February 2022
*There is no difference in completeness for the clinician perspec-
tive, medication, schedule, and dose and clinician, medication only
definitions.
t% of lists with no errors.
designed to improve data accuracy. For example, one
approach would be to use additional sources of da’ta
about medications such as prescription records from
pharmacies. Such a source of data will mainly” im-
prove completeness by capturing prescriptions writ-
ten, but not recorded, by physicians (corresponding
to error type Not listed and cause types .UPMC. MD,
Handwritten, Clinic MD, and Outside MD .in Figure 1).
We estimate that this approach, currently under in-
vestigation by others,16 would potentially reduce
missing medication records to 009 per list in our set-
ting.
After removing potentially remediable errors, we are
left with causes of errors (e.g., Patients, Data Entry)
that would not be expected to improve with EMR ex-
tensions. We estimate that ineradicable sources of er-
ror would limit maximum correctness to approxi-
mately 0.90 and incompleteness to 0.06 medications
per patient. Using the clinician, medication only defi-
nition of accuracy, the adjusted correctness would be
0.93 if scope were improved, 0.94 if both scope and
access were improved, and 0.94 maximum achievable
accuracy. The number of missing medications per pa-
tient would be the same. We discuss the implications
of ineradicable error for MDSS systems below.
Data-entry Errors
Data-entry errors were a smaller component of the
overall’ error than we expected. Research on data ac-
curacy in EMRs has focused on errors introduced by
transcription from data-entry forms into computers,
giving the false impression that transcription is the
main source of error in EMRs. These studies also sug-
gest that direct entry of data by clinicians may pro-
duce improvements in data accuracy. Considering our
comparison of accuracy levels with direct and indirect
entry, we would not expect that switching to direct
entry of data by physicians would have a large effect
242
on overall accuracy. It may be that direct entry of data
is associated with other types of errors (e.g., omitting
medications because direct entry takes longer) that
eliminate any gains achieved by the elimination of the
transcription step. We are cautious, however, in over-
interpreting this comparison because it is a small n
study and we did not randomize clinicians into direct-
entry and encounter-form groups. The biggest reason
to have physicians enter medications is to provide de-
cision support to decision makers; hence, the lack of
improvement in data accuracy should not lead to the
conclusion that indirect entry of medication data is
desirable.
Are Computer-based Medication Lists More
Accurate than Paper-based Lists?
EMRs have the potential to achieve higher data ac-
curacy than paper records through the use of validity
checks at the point of data entry, and other mecha-
nisms such as the continual improvement of current
medication lists by editing rather than rewriting (or
redictating). However, this advantage has never been
demonstrated experimentally, to our knowledge.
Studies of paper-based records differ methodologi-
cally from the present study in gold standard; hence,
we must be cautious in drawing conclusions from
comparisons. Price et al.” obtained current medica-
tion lists from English general practitioners for 46 pa-
tients and compared them with medication lists ob-
tained from patients at an interview. They found
omissions or errors in dose for 70% of the patients,
46% of the patients were taking medications that were
not reported by their general practitioners. Monson
and Bond” compared the medical records of 355 pa-
tients sampled from several outpatient clinics at the
Madison VA Hospital with pharmacy files. They
found omissions or errors in dose for 70% of the pa-
tients; 21% of the prescriptions on file were not re-
corded in the paper record. The EMR that we studied
had an error rate, when computed similarly, of 44%.
These results provide support for the conjecture that
a benefit of EMRs is improvement in (medication)
data accuracy.
Implications of Data Error for MDSSs
To examine the effect of data error on MDSSs, let us
consider the effect of incorrect and incomplete medi-
cation data on the performance of one simple type of
MDSS-a rule-based reminder system. Using medi-
cation-specific data from the BGC EMR for furose-
mide, we would expect to see the following four types
of problems in the advice produced by such a system.
WAGNER, HOGAN, Accuracy of Medication Data
First, for rules that require the presence of a single
medication in the left-hand side (e.g, if the patient is
taking furosemide and has not had serum potassium
checked within 2 year), the medication data would be
incorrect at a rate of 1 - 0.93 = 0.07. Hence, 7% of
such reminders sent to physicians would be inappro-
priate due to medication data error. Second, 1 - 0.98
= 0.02 of patients who truly were taking furosemide
would not have this fact recorded accurately in the
Downloaded from https://academic.oup.com/jamia/article/3/3/234/732087 by guest on 24 February 2022
EMR. Thus, 2% of opportunities to send this reminder
appropriately would be missed. Third, for rules that
refer to the absence of a single medication in the left-
hand side (e.g., if the patient is in pulmonary edema and
is not receiving furosemide), we would expect situa-
tions in which erroneous advice would be offered be-
cause the fact that the patient was taking the medi-
cation was not recorded, and, fourth, situations in
which the advice would have been appropriate, but it
would not have been sent. The above simple analysis
assumes only one data element per rule, and that the
accurate identity of the compound alone was required
(not a correct calculation of dosage). In more complex
cases, the level of erroneous advice and missed op-
portunity can be expected to be higher.
To simplify the above discussion, we deferred consid-
eration of an additional source of error in medication
data that may affect MDSSs. In the Introduction, we
discussed the ambiguity that we found in medication
lists in outpatient records. Although we could resolve
the ambiguity in these records, we did not think that
the ambiguity could be resolved automatically by a
MDSS. Thus, if the distinction between prescription
and description were important in the rules of a
MDSS (e.g., the rule if the patient is taking ampicillin
and has a rash, then suggest that the rash is caused by
ampicillin is different from the rule if ampicillin is pre-
scribed and the patient has a penicillin allergy, then sug-
gest discontinuation of ampicillin), additional false
alarms or missed reminders would occur. In our ex-
perience, outpatient EMRs do not support the above
distinctions. The addition of fields to allow clinicians
to record how and whether a medication is pre-
scribed, and how it is being taken would rectify this
deficiency (inpatient EMRs often maintain this dis-
tinction because they have separate order and medi-
cation-administration-record tables). This is an im-
portant observation for designers of EMRs.
Our study revealed one other issue relevant to
MDSSs. We observed that the provision of a free-text
“comments” field increased the completeness of med-
ication lists for clinician users at the expense of com-
pleteness for a MDSS. These free-text fields allowed
physicians to record partial data about a medication
Journal of the American Medical Informatics Association Volume 3 Number 3 May / Jun 1996 243

(e.g., the name of the medication, but not the formu- Referencesn
lation). However, because they do not force the phy- 1. Barrie JL, Marsh DR. Quality of data in the Manchester Or-
sician to encode the medication, they contribute to in- thopaedic Database. BMJ. 1992;304:159-62.
completeness errors for a MDSS system. Thus, we 2. Block B, Brennan JA. Reliability of morbidity data in a com-
note that there is a tradeoff in providing free-text puterized medical record system. In: Hammond WE (ed).
Proceedings of the AAMSI Eighth Annual National Con-
fields. Providing auxiliary free-text fields is a common gress, 1989:21-30.
practice in EMRs (e.g., to allow elaboration of physical 3. Jelovsek F, Hammond W. Formal error rate in a computer-
findings). Our data suggest that they can increase the ized obstetric medical record. Methods Inf Med. 1978;17:
error rate experienced by a MDSS as much as 18%. 151-7.

Downloaded from https://academic.oup.com/jamia/article/3/3/234/732087 by guest on 24 February 2022

4. Maresh M, Dawson AM, Beard RW. Assessment of an on-
line computerized perinatal data collection and information
The Study of Medication Record Errors system. Br J Obstet Gynaecol. 1986;93:1239-45.
5. Payne T, Kanvik S, Seward R, et al. Development and val-
Our study investigated only the accuracy of medica- idation of an immunization tracking system in a large
tion data. Medications are an important class of data health maintenance organization. Am J Prev Med. 1993;9:
96-100.
elements for many applications, especially decision-
6. Bates D, Cullen D, Laird N, et al. Incidence of adverse drug
support applications. Thus, the accuracy of medica- events and potential adverse drug events: implications for
tion data is an important attribute of a clinical infor- prevention. JAMA. 1995;274:29-34.
mation system. Furthermore, if there is a problem in 7. Leape L, Bates D, Cullen D, et al. Systems analysis of ad-
data accuracy, there is likely to be a problem in the verse drug events. JAMA. 1995;274:35-43.
8. Kuhn K, Gaus W, Wechsler JG, et al. Structured reporting
EMR system that is worthy of improvement, suggest- of medical findings: evaluation of a system in gastroenter-
ing that medication-data accuracy is an important ology. Methods Inf Med. 1992;31:268-74.
process variable for quality improvement efforts. 9. Ricketts D, Patterson M, Newey M, Hitchin D, Fowler S.
Markers of data quality in computer audit: the Manchester
Orthopaedic Database. Ann R Co11 Surg Engl. 1993;75:393-
Conclusions 6.
10. Wiederhold G, Perreault L. Clinical research systems. In:
Shortliffe E, Perrault L, Wiederhold G, Fagan L (eds). Med-
Medication records in outpatient EMRs may have sig- ical Informatics: Computer Applications in Health Care.
nificant levels of data error. Based on an analysis of Reading, MA: Addison-Wesley Publishing, 1990.
correctable causes of error, the most effective exten- 11 Dambro MR, Weiss BD. Assessing the quality of data entry
in a computerized medical records system. J Med Syst. 1988;
sion to the EMR studied would be to expand its scope
12:181-7.
to include all clinicians who can potentially change 12 Scott EA. Low immunization
13 rates: Fact or fiction? Public
medications. Even with EMR extensions, however, in- Health. 1990;104:275-8.
eradicable error due to patients and data entry will 13. Wilton R, Pennisi AJ. Evaluating the accuracy of transcribed
remain. clinical data. SCAMC Proc. 1993:279-83.
14. Kirkwood B. Essentials of Medical Statistics. Oxford: Black-
well Scientific Publications, 1988.
15. Rosner BA. Fundamentals of Biostatistics, Third Edition.
The authors are grateful to the clinicians at Benedum Geriatric Boston: PWS-Kent Publishing, 1989.
Center for their participation, and to Claudia Matus for statis- 16. Overhage JM, McDonald CJ, Tierney WM. Design and im-
tical advice. plementation of the Indianapolis network for patient care
and research. MLA Bull. 1995;83:48-56.
17. Price D, Cooke J, Singleton S, Feely M. Doctors’ unaware-
ness of the drugs their patients are taking: a major cause of
overprescribing? BMJ. 1986;292:99-100.
18. Monson RA, Bond CA. The accuracy of the medical record
as an index of outpatient drug therapy. JAMA. 1978;240:
2182-4.

(The Appendix appears on the next page.)

 Downloaded from https://academic.oup.com/jamia/article/3/3/234/732087 by guest on 24 February 2022
244