Creating Healthly Connection

CREATING HEALTHY CONNECTIONS
CANADA HEALTH INFOWAY

BENEFITS EVALUATION INDICATORS
TECHNICAL REPORT
Version 1.0
September 2006
CANADA HEALTH INFOWAY | Benefits Evaluation Indicators Technical Report
CONTRIBUTORS
Consulting team who managed the BE Plan development process and compiled report:
Praxia Information Intelligence

100 Broadview Ave, Suite 319
Toronto, Ontario, Canada M4M 3H3
Tel: (416) 778 7335 Fax: (416) 778-8296
support@praxia-info.com
HayGroup Canada
121 King Street West, Suite 700
Toronto, Ontario, Canada M5H 3X7
Tel: (416) 868 1371 Fax: (416) 868-6871
Subject matter experts who developed the program specific plans:
Diagnostic Imaging Program........................................... Don MacDonald

Drug Information Systems Program ................................ Dr. Robyn Tamblyn
Interoperable Electronic Health Record Program............ Dr. David Bates, Dr. Eduardo Ortiz
Lab Information System Program.................................... Dr. Nikki Shaw, Dr. Michael McNeely
Public Health Surveillance Program................................ Dr. David Buckeridge
Telehealth Program.......................................................... Dr. Richard Scott
Expert Advisory Panel who provided guidance in the development of the BE Plan:
Anne McFarlane .............................................................. CIHI

Michael Wolfson ............................................................. Statistics Canada
Doreen Neville................................................................. Memorial University
David Bates ..................................................................... Partners Healthcare
Robyn Tamblyn ............................................................... McGill University
Nikki Shaw ...................................................................... University of Alberta
Richard Scott ................................................................... University of Calgary
Francis Lau ...................................................................... University of Victoria
ii
TABLE OF CONTENTS
1 INTRODUCTION TO CANADA HEALTH INFOWAY BENEFITS EVALUATION..............................5
1.1 PURPOSE OF THIS DOCUMENT .....................................................................................................................5
1.2 PROCESS FOR DEVELOPING THE BENEFITS EVALUATION FRAMEWORK ......................................................5
1.3 INDICATOR DOMAIN EXPERTS .....................................................................................................................6
1.4 INFOWAY BENEFITS EVALUATION FRAMEWORK ..........................................................................................6
1.5 CRITERIA FOR DEVELOPING INFOWAY BENEFITS EVALUATION INDICATORS .............................................10
2 DIAGNOSTIC IMAGING PROGRAM ........................................................................................................11
2.1 DIAGNOSTIC IMAGING PROGRAM INDICATORS SUMMARY ........................................................................11
2.2 TIMELINESS OF SERVICE DELIVERY ..........................................................................................................13
2.3 TIMELINESS OF ACCESS TO TEST RESULTS ................................................................................................16
2.4 CHANGE IN PATIENT TRANSFERS FOR DI SERVICES ..................................................................................22
2.5 TECHNOLOGIST EFFICIENCY ......................................................................................................................25
2.6 RADIOLOGIST EFFICIENCY ........................................................................................................................29
2.7 AVOIDANCE OF DUPLICATE TESTS ............................................................................................................32
3 DRUG INFORMATION SYSTEM PROGRAM ..........................................................................................35
3.1 DRUG INFORMATION SYSTEM PROGRAM INDICATORS SUMMARY ............................................................35
3.2 DRUG INTERACTIONS AND THERAPY DUPLICATIONS WITH A COMPLETE MEDICATION PROFILE ..............37
3.3 COMPLETENESS AND LEGIBILITY OF SCRIPTS WITH E-PRESCRIBING .........................................................41
3.4 ADVERSE DRUG EVENTS ...........................................................................................................................43
3.5 PHYSICIAN RESPONSE TO TARGET ALERTS AT TIME OF E-PRESCRIBING...................................................45
3.6 TIME TO TAKE MEDICATION HISTORY/PATIENT ASSESSMENT ..................................................................48
3.7 PHARMACIST EFFICIENCY .........................................................................................................................51
3.8 NUMBER OF PHARMACIST TO PHYSICIAN CALLBACKS ..............................................................................54
4 LABORATORY INFORMATION SYSTEM PROGRAM .........................................................................56
4.1 LABORATORY INFORMATION SYSTEM PROGRAM INDICATORS SUMMARY ................................................56
4.2 TIMELINESS OF ACCESS TO TEST RESULTS ................................................................................................57
4.3 IMPACT OF LIS ON PATIENT SAFETY .........................................................................................................60
4.4 COMPLETENESS OF LAB PROFILE ..............................................................................................................62
4.5 LAB TECHNICIAN CALLBACKS ..................................................................................................................64
4.6 CLINICIAN WORKFLOW .............................................................................................................................66
4.7 CHANGE IN DUPLICATE LAB TESTS ...........................................................................................................68
4.8 DATA REQUIREMENTS – ALL MEASURES ..................................................................................................72
5 PUBLIC HEALTH SYSTEM PROGRAM ...................................................................................................74
5.1 PUBLIC HEALTH SURVEILLANCE PROGRAM INDICATORS SUMMARY ........................................................74
5.2 VACCINCATION RATE................................................................................................................................75
5.3 OUTBREAK DETECTION AND INTERVENTION .............................................................................................78
5.4 TIME SPENT MANAGING OUTBREAKS .......................................................................................................81
5.5 AVOIDANCE OF VACCINE WASTAGE AND UNNECESSARY VACCINATIONS ................................................83
6 INTEROPERABLE ELECTRONIC HEALTH RECORD PROGRAM ...................................................85
6.1 INTEROPERABLE ELECTRONIC HEALTH RECORD PROGRAM INDICATORS SUMMARY ................................85
6.2 READMISSION RATES ................................................................................................................................86
6.3 CHRONIC DISEASE MANAGEMENT ............................................................................................................88
iii
6.4 PROVIDER EFFICIENCY AND EFFECTIVENESS IN EMERGENCY DEPARTMENTS ...........................................92

7 TELEHEALTH PROGRAM..........................................................................................................................96
7.1 TELEHEALTH PROGRAM INDICATORS SUMMARY ......................................................................................96
7.2 EFFICIENCY OF RECOVERY WITH TELEHOMECARE ..................................................................................100
7.3 ACCESS TO PREVIOUSLY UNAVAILABLE SERVICES WITH TELEMEDICINE ................................................103
8 SYSTEM AND USE INDICATORS ............................................................................................................106
8.1 DIAGNOSTIC IMAGING PROGRAM ............................................................................................................106
8.2 DRUG INFORMATION SYSTEM PROGRAM ................................................................................................107
8.2.1 System Response Time for Tasks ......................................................................................................110
8.2.2 System Response/Failure for Connection and Downloading.............................................................113
8.2.3 Functionalities....................................................................................................................................116
8.2.4 Standardized Tasks ............................................................................................................................118
8.2.5 User Satisfaction................................................................................................................................121
8.2.6 Technology Acceptance.....................................................................................................................123
8.2.7 System Use ........................................................................................................................................125
8.2.8 Physician Response to Alerts.............................................................................................................128
8.3 LABORATORY INFORMATION SYSTEM PROGRAM ....................................................................................131
8.3.1 Clinician Adoption and Sharing Results............................................................................................133
8.3.2 Laboratory Capacity ..........................................................................................................................136
8.4 PUBLIC HEALTH SYSTEM PROGRAM .......................................................................................................138
8.5 INTEROPERABLE ELECTRONIC HEALTH RECORD PROGRAM ....................................................................139
8.6 TELEHEALTH PROGRAM ..........................................................................................................................141
APPENDIX A: ADDITIONAL PROGRAM INDICATORS ..............................................................................143
A.1 ADDITIONAL LABORATORY INFORMATION SYSTEM PROGRAM INDICATORS ..................................................143
A.2 ADDITIONAL PUBLIC HEALTH SYSTEM PROGRAM INDICATORS ......................................................................144
A.3 ADDITIONAL INTEROPERABLE ELECTRONIC HEALTH RECORD PROGRAM INDICATORS ..................................145
A.4 ADDITIONAL TELEHEALTH PROGRAM INDICATORS ........................................................................................146
APPENDIX B: PROGRAM SAMPLE TOOLS ...................................................................................................170
B.1 DIAGNOSTIC IMAGING PROGRAM SAMPLE TOOLS ..........................................................................................170
B.2 DRUG INFORMATION SYSTEM PROGRAM ........................................................................................................192
B.3 TELEHEALTH PROGRAM ..................................................................................................................................233
APPENDIX C: EXAMPLES OF INDICATORS/MEASURES, DESIGNS AND INSTRUMENTS ...............236
C.1 SUGGESTED EVALUATION MEASURES FOR QUALITY AND SYSTEM USE DIMENSIONS ....................................236
C.2 SUGGESTED EVALUATION MEASURES FOR NET BENEFITS DIMENSIONS – QUALITY .......................................238
C.3 SUGGESTED EVALUATION MEASURES FOR NET BENEFITS DIMENSION – ACCESS AND PRODUCTIVITY ..........240
APPENDIX D: OTHER KEY CONTRIBUTORS ...............................................................................................244
iv
1 INTRODUCTION TO CANADA HEALTH INFOWAY

BENEFITS EVALUATION
Canada Health Infoway's mission is to foster and accelerate the development and adoption of
electronic health information systems with compatible standards and communications
technologies on a pan-Canadian basis, with tangible benefits to Canadians. The goal is to have an
interoperable electronic health record in place for 50% of Canada’s population by the end of
2009. “Measure Benefits and Adjust” is one of Infoway’s business strategies. Infoway is
committed to continually measuring benefits achieved against those planned, and making
necessary adjustments.
1.1 PURPOSE OF THIS DOCUMENT
The purpose of this document is to:
• Describe the Infoway Benefits Evaluation Framework

• Describe indicators for six of the Infoway investment programs: Laboratory, Drugs,
Diagnostic Imaging, Public Health, Telehealth, and Interoperable EHR
• Describe System and Use indicators that apply generally to all programs or specifically to
certain programs
• Detail measurement tools and methods
• Provide common definitions
The intended audience for this document is project staff, Infoway staff or evaluators who wish to
evaluate and Infoway funded project.
1.2 PROCESS FOR DEVELOPING THE BENEFITS EVALUATION FRAMEWORK
Measurement and Benefits Evaluation is one of Infoway’s key business strategies. To assess the
return on Infoway’s EHR investments, the impact and benefits must be evaluated. Infoway
wishes to develop a Benefits Evaluation plan that will allow Infoway to move forward with
evaluating projects and programs and demonstrating the value of Infoway investments.During
October – November 2005, Infoway retained HayGroup/Praxia to produce a roadmap for
developing the Infoway Benefits Evaluation plan. The consultants:
• Developed an approach to Infoway EHR Impact Assessment

• Gathered input from Infoway Senior Executive stakeholders
• Held a consensus building workshop with expert advisors to review the approach
5
A measurement framework and template for documenting indicators was developed. Subject
matter experts (SME) were used to recommend and define indicators for each of the individual
Infoway investment programs (diagnostic imaging, laboratory information systems, drug
information systems, telehealth, public health surveillance and interoperable EHR).
The input from each of the SMEs was collated and reviewed for feasibility, consistency and
alignment with the Infoway benefits evaluation framework. The measurement framework was
completed with the proposed indicators. A technical document was prepare from the raw input
with all of the program measures, collated methodologies and guidelines.
Based on the comprehensive measurement framework, an implementation plan was defined

which includes:
• High level implementation plan

• Budget estimates
• Resource options
• Recommended IT projects to be evaluated
1.3 INDICATOR DOMAIN EXPERTS
Subject matter experts were used to develop the program specific indicator plans:
Diagnostic Imaging Program........................................... Don MacDonald

Drug Information Systems Program ................................ Dr. Robyn Tamblyn
Interoperable Electronic Health Record Program............ Dr. David Bates, Dr. Eduardo Ortiz
Lab Information System Program.................................... Dr. Nikki Shaw, Dr. Michael McNeely
Public Health Surveillance Program................................ Dr. David Buckridge
Telehealth Program.......................................................... Dr. Richard Scott
1.4 INFOWAY BENEFITS EVALUATION FRAMEWORK
The aim of this Infoway Benefits Evaluation framework is to provide a high-level coherent
framework that is evidence-based to guide subsequent field evaluation studies by the respective
jurisdictions and investment programs. Individual Infoway investment programs (diagnostic
imaging, laboratory information systems, drug information systems, telehealth, public health
surveillance and interoperable EHR) will have selected evaluation measures, designs and
instruments that are specific and appropriate for their particular programs and settings.
6
This framework is an adaptation of the 2003 DeLone and McLean model of information systems
(IS) success tailored specifically for the Canadian healthcare setting1.
The goal of the framework, and the evaluation projects which flow from it, is to evaluate the
benefits of Infoway investments in terms of healthcare quality, productivity and access. To
assess whether the conditions for benefits realization are met, the framework identifies key
functionality and use metrics.
This framework has the potential to be expanded to become a more comprehensive health
information systems success framework. Key organizational and context factors such as
strategy, culture and business process are not included in this framework. The framework is also
selective in the attributes included, focusing only on the most significant factors and those which
are practical and measurable.
7
The Infoway success model has six conceptual dimensions. There are three quality dimensions –
information, system and service:
• Information quality includes such criteria as completeness, ease of understanding and

relevance.
• System quality includes adaptability, availability and response time.
• Service quality includes assurance and responsiveness.
• The quality of the information, system and service can affect the extent of system use,
intention to use, and user satisfaction.
• In turn, system usage and satisfaction can lead to positive and negative impacts at the
individual and organizational levels, which are collectively viewed as net benefits.
Dimension Category Subcategories and/or Definition

System Functionality Type of features and level of decision support (types of DS as discussed in
Hayward2 and Randolph3 include: report/view, reference, reminder, alert,
assist and guide)
Performance Accessibility (distance, availability), reliability (up/downtime) and system
response time
Security Type of features
Information Content Completeness, accuracy, relevance, comprehension
Availability Timeliness, reliability and consistency of information when and where
needed
Service Responsiveness User training, ongoing technical support, availability of support

Use Use Frequency, duration, location, type/nature and flexibility of actual usage
behavior/pattern
Self reported use Frequency, duration, location, type/nature and flexibility of perceived
usage
Intention to use Proportion of and factors for current non-users to become users
Satisfaction Competency Knowledge, skills and experience
User satisfaction Perceived expectations, value, information/system/service quality and use
of the system (including provider/patient interaction, preference, comfort
and experience)
Ease of use User-friendliness and learn-ability
8
Net Quality Patient Safety
Benefits • Preventable Adverse events - near misses and near errors
• Surveillance - post marketing and public health (communicable
disease surveillance)
• Reduction in patient risks and safety related reportable drug and
device events.
Appropriateness / effectiveness
• Adherence /Compliance with benchmark, policy or practice standards
/ guidelines
• Self reported practice or captured in the system
• Immunization /testing and other relevant rates
• Continuity of care
o Information, relational and management continuity
o by individuals or multidisciplinary teams or geographically
dispersed teams
o Access to information, GP specialist referral effectiveness
Health outcomes
• Clinical outcomes
• Change in Health status attributable to the e-Health intervention
Access Ability of patients/providers to access services
• Availability of services
• Diversity of services
• Consolidation of services
• Timeliness, geographic, financial and cultural/linguistic – removal of
inequitable barriers (including affordability, acceptability,
accommodation)
• E.g. Telehomecare
Patient and caregiver participation
• Patient self management
o Patient access to their information
Productivity Efficiency
• Provider resource utilization
• Short term improvement in outputs vs. inputs and longer term
improvements in care continuity
• Improved health system management capability
• Improved patient efficiency eg more efficient scheduling of pre-op
testing.
• Non monetary effects
Care coordination
• Care provision by team
• Continuity of care across continuum
Net cost
• Monetary avoidance
• Monetary reductions, savings
9
1.5 CRITERIA FOR DEVELOPING INFOWAY BENEFITS EVALUATION INDICATORS
Indicator selection criteria are a key tool to ensure that the set of indicators are appropriately
selected. The criteria below were used as a guideline for selecting key measures.
• Importance: The indicator reflects aspects of health system functioning that matter to
users and are linked to a Strategy Map priority area.
• Relevance: The indicator provides information that advances the understanding of
population health and the health system, and can be used to monitor and measure health
system performance over an extended period of time.
• Feasibility: Data required for the indicator are readily available for the areas and time
periods indicated, and there should be no unreasonable obstacles or constraints on access
to the information collected, nor restrictions on its use.
• Reliability: The indicator produces consistent results in repeated measurements of the
same condition or event.
• Validity: There is consensus on the part of users and experts that the indicator is related
to the dimension it is supposed to assess (face validity), covers the whole dimension it is
supposed to assess (content validity), is related to other indicators measuring the same
dimension (construct validity), and has predictive power (criterion validity).
References
DeLone WH, McLean ER. (2003) The DeLone and McLean model of information systems
success: a ten-year update. Journal of management Information Systems 19(4):9-30.
Hayward R. Clinical Decision Support Tools – Do They Support Clinicians? Presentation at the
University of Victoria health informatics seminar series on Dec 9/2005.
Randolph AG, Haynes B, Wyatt JC, et al. (2000) How to Use an Article Evaluating the Clinical
Impact of a Computer-based Clinical Decision Support System. JAMA 28(1):67-74. URL:
http://www.cche.net/usersguides/computer.asp; accessed July 31/2005.
10
2 DIAGNOSTIC IMAGING PROGRAM
2.1 DIAGNOSTIC IMAGING PROGRAM INDICATORS SUMMARY
Category & Study Question Indicator Measures

Sub-category
Quality
Change in Does PACS support more Timeliness of Referring physician capacity to make
provider timely delivery of patient care service clinical care decisions in a timely
effectiveness/ by referring physicians? delivery manner.
appropriateness Subjective Measure; Perceived
of care Benefits (Survey – Appendix B)
Change in Does PACS support more Timeliness of Time elapsed from the point of the exam
provider timely access to information by access to test completion to the availability of the
effectiveness/ the referring physicians? results radiologist report to the referring
appropriateness physician
of care Objective measure; Report TAT
Subjective Measure; Perceived
Benefits (Survey – Appendix B)
Time spent by the referring physician
retrieving images and reports.
Subjective Measure; Perceived Benefits
(Survey, Appendix B)
Change in Health Is patient travel (i.e., transfers) Change in Patient travel required to access DI
Outcomes for radiology services reduced patient services
as a result of the transfers for Objective measure; Rate of patient
implementation of PACS? DI services transfers for DI services pre and post
PACS
Productivity
Change in With the implementation of Technologist Time elapsed from patient registration
provider PACS, is there a decrease in the Efficiency to exam available to radiologist for
efficiency time it takes a technologist to interpretation
create an exam and make it Objective measure; Exam Turn Around
available to the radiologist, Time (TAT)
compared to the time taken in
the film environment?
Change in With the implementation of Radiologist Time required by the radiologist to

provider PACS, is there a perceived Efficiency access an exam and generate the report
efficiency decrease in time it takes a Subjective measure; Perceived
radiologist to access an exam Benefits (Survey – Appendix B)
and generate the report,
11
Category & Study Question Indicator Measures
Sub-category
compared to the time taken in
the film environment?
Change in net Did a reduction in the ordering Avoidance of Number of redundant exams ordered
cost of redundant exams occur as a duplicate test Objective measure; Number of exams
result of PACS? re-ordered Pre-PACS because original
was lost or missing
12
2.2 TIMELINESS OF SERVICE DELIVERY
Indicator Description:
One measure of timeliness of patient care delivery is the referring physicians’ capacity to make
clinical care decisions in a timely manner.
Study Question:
Does PACS support more timely delivery of patient care by referring physicians?
Indicator Category and Sub Category:

Quality, Change in provider effectiveness/ appropriateness of care
Indicator Rationale:
With the introduction of PACS, delays in treatment due to image and/or report unavailability
(common in the film environment) will be eliminated. Referring physicians will be able to view
the images and reports from multiple sites (e.g., hospital, office or home) at any time during the
day or night, and use this information to make timely and appropriate treatment decisions.
Indicator Setting and Population:

• The setting for this indicator will be a defined catchment area (e.g. a health region)
• The population will include all referring physicians within a catchment area. Referring
physicians would include all physicians who would normally refer patients for diagnostic
imaging services.
Measures:
Survey questions can be employed to capture the perceived improvements in the timely delivery
of patient care by referring physicians.
The survey questions, primarily utilizing Likert scales, would focus on:
• Patient Care – timeliness of treatment
• Patient Care – appropriateness of treatment (continuity arising from access to previous
exams)
Exclusion Criteria:
Physicians who normally would not refer patients for diagnostic imaging (e.g. psychiatrists).
Study Design:
It is recommended that a survey questionnaire (mailed or web-based) be administered 3-months
pre-PACS implementation and 6 and/or 12-months post PACS implementation. Time series
analyses are used to investigate changes in perceived benefit over time.
13
Study Assumptions:
• Questionnaire is validated;
• The referring physician environment will remain stable throughout study;
• survey items will remain unchanged for each data collection period, however additional
items could be added;
• A representative sample of responses is achieved; and
• Consideration is given to the issue of socially desirable responses.
Key informant interviews of a sample of referring physicians (approx. 5) following the analysis
of the last post PACS survey are also recommended. The purpose of the interviews would be to
solicit more detailed information on the benefits of PACS to referring physicians.
Data Sources and instrument:

Examples of mail-out and web-based surveys for referring physicians can be found in
Appendix B.
Indicator Analysis:
Percentage of respondents per item; statistical tests to assess differences over time and between
appropriate items (e.g., type of physician, location of physician (urban/rural).
Associations:
Referring physicians’ access to current and historical information through PACS would enhance
the delivery of appropriate care.
Comments:
The core of the referring physician survey should be adapted from the questionnaire developed
by Canada Health Infoway. A stakeholder consultation is recommended at pre-evaluation to
present the survey and provide the opportunity for key stakeholders to identify additional
questions specific to the local environment. Where possible, findings should be collated with
previously conducted PACS surveys (e.g., British Columbia, Newfoundland and Labrador and
Nova Scotia).
References:
Bryan, S., Weatherburn, G., Buxton, M., Watkins, J., Keen, J. and Muris, N. (1999). Evaluation
of a hospital picture archiving and communication system. Journal of Health Services Research
and Policy; 4(4), 204-9.
Bryan, S, Weatherburn, GC, Watkins, JR and Buxton, MJ. (1999). The benefits of hospital-wide
picture archiving and communication systems: a survey of clinical users of radiology services.
The British Journal of Radiology; 72, 469-78.
14
Reiner, B., Siegel, E., Hooper, F., Pomerantz, S., Protopapas, Z., Pickar, E. and Killewich, L.
(1996). Picture archiving and communication systems and vascular surgery: clinical
impressions and suggestions for improvement. Journal of Digital Imaging; 9(4), 167-71.
Pilling, J. (2003). Picture archiving and communication systems: the users’ view. The British
Journal of Radiology; 76, 519-24.
Wadley, B., Hayward, U., Trambert, M., Kywi, A and Hartzman, S. (2002). Are referring
doctors ready for enterprise and community wide immediate image and report access? Journal
of Digital Imaging; 15(Suppl 1), 140-43.
Watkins, J. (1999). A hospital-wide picture archiving and communication system (PACS): the
views of users and providers of the radiology service at Hammersmith Hospital. European
Watkins, J., Weatherburn, G., Bryan, S. (2000). The impact of a picture archiving and
communication system upon an intensive care unit. European Journal of Radiology; 34, 3-8.
Worthy, S., Rounds, K. and Soloway, C. Strengthening your ties to referring physicians through
RIS/PACS integration. Radiology Management, March/April 2003.
15
2.3 TIMELINESS OF ACCESS TO TEST RESULTS
Indicator Description: Measure #1

One measure of timely access to information for the referring physician is the report turn around
time (TAT); the time the radiology exam is completed to the time that the radiologist’s report
(verified or unverified) is available to the referring physician.
Study Question:
Does PACS support more timely access to information by referring physicians?

With the introduction of PACS, referring physicians will be able to access the radiologist report
from multiple sites (e.g., hospital, office or home) at any time. Therefore, the time required by
referring physicians to access radiology reports should be reduced.

• The setting for this indicator will be a defined catchment area (e.g., health region, enterprise,
hospital).
• The population will be all imaging modalities and patient exams within scope; sampling to
be used where appropriate.
Measures:
Report turn around time (TAT) measured as the average time (hours) by modality, from when
the technologist completes the exam to when the report (verified or unverified) is available to the
referring physician.
Exclusion Criteria:
None
Study Design:
It is recommended that data for the Report TAT indicator be collected using two separate study
designs; recorded time checks and a time motion study.
Study Design #1: Report TAT determined through recorded time checks, pre and post PACS
In the film environment, time is recorded from the time of exam completion to time of report
completion (stamped on film package).
16
In the PACS environment, time is recorded from the time the exam is posted on PACS to the
time the report (verified or unverified) is posted on PACS.
Study Design #2: Report TAT determined through a Time Motion Study (TMS), pre and post
PACS
A time motion study will identify and measure the time taken for specific tasks performed by
radiologist in creating the report. The study will measure the Report TAT by collecting start and
end times for specific tasks, as well as the total time from exam completion to report availability.
In order to enhance the reliability and validity of the study findings, it is recommended that:
• Both study designs be utilized;
• Studies be conducted in at least 2 sites;
• Sites should be selected based on the volume of exams (e.g., high and low) and the DI
services provided (i.e., relevant modalities); and
• Study results to be collated with those obtained from other PACS BE projects (e.g., Interior
Health Authority in BC).
Study Assumptions:
• All clocks/timers in hospital are synchronized;
• All appropriate modalities are included;
• External impacts on Report TAT are considered (e.g., staffing, internal policies).
PACS
• Study is designed to capture results that will be indicative of overall TAT for the DI
department;
• Consideration is given to how many weeks the TMS will be carried out, the timing of the
data collection with respect to: (a) time of day data; (b) days of the week; and (c) season of
the year;
• TMS collection tool is validated.
Data Sources and Instrument:

• Hospital Registration Information Systems
• Time stamp on film package
• PACS
PACS
17
• TMS Collection Sheet (Interior Health Authority, BC Sample: Appendix B)
Indicator Analysis:
A = ∑ report available time– exam available time (film environment)
B = ∑ report available time– exam available time (PACS environment)
C = Total number of exams in film environment included in study
D = Total number of exams in PACS environment included in study
%∆ = (B/D – A/C) x 100
Note: Given that some modalities have a small number of events, consideration should be given
to calculating both mean and median values for specific tasks (TMS) and for total TAT.
Differences in TAT between pre and post PACS are to be assessed using appropriate statistical
tests (e.g. t-test and Mann-Whitney U test).
Comments:
The report TAT could also be used as another measure for Indicator #2 “radiologist efficiency”.
It was included in this section given its potential to impact on the timeliness of information
availability for the referring (treating) physician.
Indicator Description: Measure #2

A second measure of timely access to information for the referring physician is time spent by the
referring physician retrieving images and reports.
Study Question:
Does PACS support more timely access to information by referring physicians?

With the introduction of PACS, delays in treatment due to image and/or report unavailability
(common in the film environment) will be eliminated. Physicians will able to view the images
and reports from multiple sites (e.g., hospital, office or home) at any time during the day or
night. PACS will therefore reduce the time required by referring physicians to access radiology
images and reports.

• The setting for this indicator will be a defined catchment area (e.g. a health region).
18
• The population will include all referring physicians within a catchment area. Referring
physicians would include all physicians who would normally refer patients for diagnostic
imaging services.
Measures:
Survey questions can be employed to measure the perceived improvements in the time required
by referring physicians to access radiology images and reports.
The survey, primarily utilizing Likert scales, would focus on:

• Accessibility – ability to access images and reports
• Efficiency and Time Management – time spent retrieving images and reports
• Patient Care – more timely treatment; access to previous exams
Exclusion Criteria:
Physicians who normally would not refer patients for DI (e.g. psychiatrists).
Study Design:
Study Assumptions:
• The referring physician environment will remain stable throughout study;
• Survey items will remain unchanged for each data collection period, however additional
Key informant interviews of a sample of referring physicians (approx. 5) following the analysis
of the 12-month post PACS survey are also recommended. The purpose of the interviews would
be to solicit more detailed information on the benefits of PACS to referring physicians.

Examples of mail-out and web-based surveys for referring physicians can be found in Appendix
B.
Indicator Analysis:
appropriate items (e.g., type of physician, location of physician (urban/rural)).
19
Associations:
Referring physician access to information through PACS would enhance the timely delivery of
patient care.
Comments:
The core of the referring physician survey should be adapted from the questionnaire developed
by Canada Health Infoway. A stakeholder consultation is recommended at pre-evaluation to
present the survey and provide the opportunity for key stakeholders to identify additional
questions specific to the local environment. Where possible, findings should be compared with
previously conducted PACS surveys (e.g., British Columbia, Newfoundland and Labrador and
Nova Scotia).
References:
Bryan, S., Weatherburn, G., Buxton, M., Watkins, J., Keen, J. and Muris, N. (1999). Evaluation
of a hospital picture archiving and communication system. Journal of Health Services Research
and Policy; 4(4), 204-9.
Bryan, S, Weatherburn, GC, Watkins, JR and Buxton, MJ. (1999). The benefits of hospital-wide
picture archiving and communication systems: a survey of clinical users of radiology services.
The British Journal of Radiology; 72, 469-78.
Reiner, B., Siegel, E., Hooper, F., Pomerantz, S., Protopapas, Z., Pickar, E. and Killewich, L.
(1996). Picture archiving and communication systems and vascular surgery: clinical
impressions and suggestions for improvement. Journal of Digital Imaging; 9(4), 167-71.
Pilling, J. (2003). Picture archiving and communication systems: the users’ view. The British
Wadley, B., Hayward, U., Trambert, M., Kywi, A and Hartzman, S. (2002). Are referring
doctors ready for enterprise and community wide immediate image and report access? Journal
of Digital Imaging; 15(Suppl 1), 140-43.
Watkins, J. (1999). A hospital-wide picture archiving and communication system (PACS): the
views of users and providers of the radiology service at Hammersmith Hospital. European
Watkins, J., Weatherburn, G., Bryan, S. (2000). The impact of a picture archiving and
communication system upon an intensive care unit. European Journal of Radiology; 34, 3-8.
20
Worthy, S., Rounds, K. and Soloway, C. Strengthening your ties to referring physicians through
RIS/PACS integration. Radiology Management, March/April 2003.
21
2.4 CHANGE IN PATIENT TRANSFERS FOR DI SERVICES
One measure of availability of DI services in the patient’s location is patient travel required to
access DI services.
Study Question:
Is patient travel (i.e., transfers) for radiology services reduced as a result of implementing
PACS?

Quality, Change in Health Outcomes
A patient requiring diagnostic imaging services could be transferred from one site to another if
the originating site does not have the ability to generate the required exam, or there is no
radiologist or specialist available to interpret the exam. With the introduction of PACS, the
number of patient transfers between facilities due to the inability to share images and consult
should be reduced.

• The setting for this indicator will be selected sites that provide diagnostic imaging services;
• The population will be all patients transferred for any reason to these sites prior to the
implementation of PACS.
Measures:
Total average number (annually) of patient transfers for DI services five(5) years Pre- PACS as a
proportion of all patient transfers for the same period, compared to the total number of patient
transfers for DI services one (1) year Post- PACS as a proportion of all patient transfers for the
same period.
Exclusion Criteria:
There will be no exclusions – a random sample of all transfers may be selected both pre and
post PACS.
Study Design:
Study design is a pre/post comparative analysis using a retrospective chart review as the data
collection method.
22
Study Assumptions:
• A provincial hospital information system in place to identify all patient transfers for five (5)
years pre, and one (1) post PACS for all selected sites;
• The reason for patient transfer is available from medical chart documentation;
• The pre-PACS originating site will not have had the capabilities to generate soft copy exams,
nor have access to a radiologist on site;
• Post-PACS originating site will have capabilities to produce soft copy exams, but not have
access to a radiologist on site.
Data Sources and Instruments:

• Provincial Hospital Discharge Information System
• Patient Medical Chart
• Reason for Patient Transfer Data Collection Sheet
Indicator Analysis:
A = Number of Patient Transfers for DI Pre-PACS
B = Total Patient Transfers Pre-PACS
C = Number of Patient Transfers for DI Post-PACS
D = Total Patient Transfers Post-PACS
%∆ = (A/B – C/D) x 100
Associations:
Improved patient access to DI services and the avoidance of unnecessary travel may contribute to
improved quality of care.
Comments:
Pilot study in one PACS site recommended.
References:
Daucourt V., Petitjean ME., Chateil, JF., Philippe M. (2005) Evaluation of the benefits for the
patient arising from an inter-hospital teleradiology network in a French administrative area.
Journal of Telemedicine and Telecare. Vol. 11 No, 4, June, pp 178-184(7)
Daucourt V., Sicotte C., Pelletier-Fleury N., Petitjean ME., Chateil JF., Michel P. (2005). Cost-
minimization analysis of a wide-area teleradiology network in a French region. International
Journal for Quality Health Care (Sept) pp 1-7
Park J., Ruess L., O’Connor S., Hussain F., Oshiro D., Person D. (2004). Internet Consultations
from a Remote Pacific Island: Impact of Digitized Radiologic Images on Referral Decisions.
Journal of Digital Imaging Vol. 17, No. 4 (December) pp 253-257
23
Stromo A., Sollid S., Stormer J., Ingebrigtsen T. (2004) Neurosurgical teleconsultations in
northern Norway. Journal of Telemedicine and Telecare. Vol. 10 No, 3, June, pp 135-139(5)
Jithoo R., Govender P., Corr P., Nathoo N. (2003). Telemedicine and neurosurgery: experience
of a regional unit based in South Africa. Journal of Telemedicine and Telecare, Vol 9, No. 2,
April, pp 63-66)4)
Servadei F., Antonelli V., Mastrilli A., Cultrera F., Giuffrida M., Staffa G. (2002). Integration of
image transmission into a protocol for head injury management: a preliminary report. British
Journal of Nuerosurgery 16(1):36-42
Heautot J., Gibaud B., Catroux B., Thoreux P., Cordonnier E., Scarabin J., Carsin M., Gandon.
(1999). Influence of the teleradiology technology (N-ISDN and ATM) on the inter-hospital
management of neurosurgical patients. Medical Informatics Vol. 24, No. 2, pp122-134
Bailes J., Poole C., Hutchinson W., Maroon J., Fukushima T. (1997). Utilization and cost
savings of a wide-area network for neurosurgical consultation. Journal of Telemedicine
3(2);135-139
24
2.5 TECHNOLOGIST EFFICIENCY
One indicator of technologist efficiency is the time elapsed from: (a) the time the patient
registers (or checks in) at the Radiology Department to (b) the time the exam is available to the
radiologist for interpretation. This time, from registration to exam availability, for the purpose of
this framework, is referred to as the exam turn around time (TAT).
Study Question:
With the implementation of PACS, is there a decrease in the time it takes a technologist to
generate an exam and make it available to the radiologist for interpretation, compared to the time
taken in the film environment?

Productivity, Change in provider efficiency
Decreasing the time to exam availability through the elimination of steps involved in film
processing and handling is an important benefit of PACS. Increased technologist efficiency in
exam processing may also have positive impacts on workflow optimization that would improve
the quality of service and ultimately quality of patient care.

• The setting for this indicator will be select hospital departments (e.g. outpatient, inpatient,
ER, etc) and select modalities (e.g., radiology, ultrasound).
• The population will be all patient exams within scope; sampling to be used where
appropriate.
Measures:
Exam turn around time (TAT), defined as the time from patient check-in or registration in
diagnostic imaging department to the availability of exam for interpretation by the radiologist.
Exclusion Criteria:
Excludes modalities that created soft copy images prior to the implementation of PACS (e.g.,
CT).
Study Design:
It is recommended that data for the Exam TAT indicator be collected using two separate study
designs; recorded time checks and a time motion study.
25
Study Design #1: Exam TAT determined through recorded time checks, pre and post PACS
In the film environment, time is recorded from registration (or check-in) to exam completion,
with time stamped on the film package.
In the PACS environment, time is recorded from registration (or check-in) to time the exam is
posted on PACS.
Study Design #2: TAT determined through a Time Motion Study (TMS)
A time motion study will measure the time taken for specific tasks performed by the technologist
in generating the exam. The study will measure start and end times for specific tasks, as well as
overall exam turn around time from patient registration to exam availability.
In order to enhance the reliability and validity of the study findings, it is recommended that:
• Both study designs be utilized;
• Studies be conducted in at least 2 sites;
• Sites should be selected based on the volume of exams (e.g., high and low) and the DI
services provided (i.e., relevant modalities); and
• Study results to be collated with those obtained from other PACS BE projects (e.g., Interior
Health Authority in BC).
Study Assumptions:
Study Design #1: TAT Determined Through Recorded Time Checks, Pre and Post PACS
• All clocks/timers in hospital are synchronized
• Different registration procedures for inpatient/outpatient services are considered
• Sites selected to reflect both urban and rural environments
• All relevant modalities to be considered
• External impacts on turn around time are considered (e.g., staffing, internal policies)
Study Design #2: TAT determined through a Time Motion Study (TMS), pre and post PACS
• Study is designed to capture results that will be indicative of the overall exam turn around
time for the DI department;
• Consideration is given to how many weeks the TMS will be carried out and the timing of the
data collection with respect to: (a) the time of day; (b) the days of the week; and (c) the
season of the year;
• TMS collection tool is validated
Data Sources and Instrument:

Study Design #1: Exam TAT Determined Through Recorded Time Checks, Pre and Post PACS
• Hospital Registration Information System
• Time stamped on film package
26
• PACS
Study Design #2: Exam TAT determined through a Time Motion Study (TMS), pre and post
PACS
• TMS Collection Sheet (Interior Health Authority, BC Sample: Appendix B)
Indicator Analysis:
A = ∑ exam available time– registration time (film environment)
B = ∑ exam available time– registration time (PACS environment)
C = Total number of exams in film environment included in study
D = Total number of exams in PACS environment included in study
%∆ = (B/D – A/C) x 100
Note: Given that some modalities may have a small number of events, consideration should be
given to calculating both mean and median values for specific tasks, and for total exam TAT.
Differences in exam TAT between pre and post PACS are to be assessed using appropriate
statistical tests (e.g. t-test and Mann-Whitney U test).
Associations:
Gains in efficiency (exam turn around time) can impact productivity of the radiology department
and improve the quality (timeliness) of patient care.
Comments:
Considerable work on technologists’ productivity has been completed in the Interior Health
Authority (British Columbia). It is recommended that this study be thoroughly documented and
disseminated so that it can serve as the foundation for those sites which will undertake further
PACS benefit evaluation.
References:
Andriole, K. Luth, D and Gould, R. (2002). Workflow assessment of digital versus computed
radiography and screen-film in the outpatient environment. Journal of Digital Imaging; 15
(Suppl 1), 124-26.
Langlosi, SLP, Vytialingam, RC and Aziz, NA. (1999). A time-motion study of digital radiology
at implementation. Australian Radiology; 43, 201-5.
Redfern, RO, Horri, Sc, Feingold, E., Kundel, HL. (2000). Radiology workflow and patient
volume: effect of picture archiving and communication systems on technologists and
radiologists. Journal of Digital Imaging; 13 (2 Suppl 1), 97-100.
27
Reiner, B., Siegel, E. and Carrino, J. (2002). Workflow optimization: Current trends and future
directions. Journal of Digital Imaging; 15 (3), 141-52.
Reiner, B., Siegel, E. and Scanlon, M. (2002). Changes in technologist productivity with
implementation of an enterprise PACS. Journal of Digital Imaging, 15(1), 22-6.
Reiner, B., Siegel, E., Carrino, J. and Goldburg, M. (2002). SCAR Radiologic technologic
survey: analysis of the impact of digital technologies on productivity. Journal of Digital Imaging;
15(3), 132-40.
28
2.6 RADIOLOGIST EFFICIENCY
One measure of Radiologist Efficiency is the time it takes a radiologist to access the exam and
generate a report in the PACS environment, compared to the time taken in the film environment.
Study Question:
With the implementation of PACS, is there a perceived decrease in time it takes a Radiologist to
access an exam and generate the report, compared to the time taken in the film environment?

Productivity; Change in provider efficiency
PACS will provide quicker access to the image by the Radiologist, thus reducing the time
required to interpret the image and generate the report for the referring physician. Delays in
reporting due to image unavailability (common in the film environment) will be eliminated.
Radiologists will be able to access and view the image and post reports from multiple sites (e.g.,
hospital, office or home) at any time during the day or night.

• The setting for this indicator will be a defined catchment area (e.g., health region, enterprise,
hospital)
• The population for this indicator will be all Radiologists within the defined catchment area
Measures:
Survey questions can be employed to measure the perceived improvements in radiologist
efficiency resulting from the implementation of PACS.
The survey, primarily utilizing Likert scales, will focus on:

• Accessibility – access to images
• Efficiency and Time Management – time spent retrieving images
• Quality Care – continuity of care (access to previous exams)
Key informant interviews with a sample of radiologists following the analysis of the last post
PACS survey are also recommended. The interviews will solicit more detailed information on
the benefit of PACS to radiologists.
Exclusion Criteria:
None
29
Study Design:
Study Assumptions:
• The radiologist environment will remain stable throughout study;
• Survey items will remain unchanged for each data collection period, however additional

Examples of mail-out and web-based surveys for radiologists can be found in Appendix B. The
web-based survey was used as the post-PACS collection tool in British Columbia; the paper-
based survey was used as the pre-PACS collection tool in Newfoundland and Labrador. Format
and wording differ slightly between the two surveys.
Indicator Analysis:
selected items (e.g., years practicing).
Associations:
Gains in efficiency (radiologist turn around time) can impact productivity of the radiology
department and improve the quality (timeliness) of patient care.
Comments:
The core of the survey should be adapted from the questionnaire developed by Canada Health
Infoway. A stakeholder consultation is recommended at pre-evaluation to present the survey and
provide the opportunity for key stakeholders to identify additional questions specific to the local
environment. Where possible, findings should be collated with those of previously conducted
PACS surveys (e.g., in British Columbia, Newfoundland and Labrador and Nova Scotia).
References:
Toby M. (2004). Paperless Medical Records: Measuring Success. Radiology Management; Sept-
Oct;26(5):16-20,22
30
Mehta A., Dreyer K., Boland G., Frank M. (2000). Do Picture Archiving and Communication
Systems Improve Report Turnaround Times? Journal of Digital Imaging. May;13(2 Suppl
1):105-7
Redfern R., Kundel H., Horri S. (2000). Radiology Workflow and Patient Volumes: Effect of
PACS on Technologists and Radiologists. Journal of Digital Imaging. 13:97-100.
Redfern R., Kundel H., Polansky. (2000). A Picture Archiving and Communication System
Shortens Delays in Obtaining Radiographic Information in a Medical Intensive Care Unit.
Critical Care Medicine. 28:1006-1013.
Siegel E., Reiner B. (2002). Work Flow Resign: The Key to Success when Using PACS.
American Journal of Roentgenology. 178, March.
Redfern R., Horri S., Kundel H. (1999). Experiences with Radiology Workflow and PACS:
Effects on Technologists and Radiologists Task Times. Proceedings of SPIE Medical Imaging.
3662:42.
Andriole K., Storto M., Gamsu G., Arenson R. (1996). ICU Display Station Impact and
Utilization Studies. Symposium for Computer Assisted Radiology. Reston, VA: Society for
Computer Applications in Radiology: 387-392.
Reiner B., Seigel E., Siddiqui K. (2003). Evolution of the Digital Revolution: A Radiologists
Perspective. (2003). Journal of Digital Imaging, Vol 16, No 4 (December), pp 324-330
Krupinski E., McNeill K., Haber K., Ovitt T. (2003). High-Volume Teleradiology Service: Focus
on Radiologist Satisfaction. Journal of Digital Imaging, Vol 16, No 2 (June): pp 203-209
31
2.7 AVOIDANCE OF DUPLICATE TESTS
Avoidance of both cost and unnecessary medical interventions (such as repeat DI exams) can be
measured by examining the number of duplicate (i.e., redundant) DI exams ordered due to lack
of exam availability when required.
Study Question:
Did a reduction in the ordering of duplicate exams occur as a result of PACS?

Productivity, Change in net cost
If a patient’s original DI exam is unable to be found for any reason (e.g., lost, misfiled, etc.) for
patient care, a duplicate (i.e., redundant) exam would be re-ordered by the physician. The
introduction of PACS should eliminate/significantly reduce the potential for missing exams.

• The setting will be a selected catchment area (e.g., a regional health board).
• The population will include all duplicate DI exams ordered within a pre-determined period
(e.g., 3 days) as identified by the hospital’s DI order entry system.
Measures:
Average number (annual) of redundant DI exams five(5) years pre- PACS as a proportion of all
duplicate exams, compared to total number of redundant DI exams one (1) year Post- PACS as a
proportion of all duplicate exams.
Exclusion Criteria:
There will be no exclusions – random sample of all duplicate exams for select sites will be
selected both pre and post PACS.
Study Design:
The study design is a pre-post comparative analysis using retrospective chart review.
Study Assumptions:
• Hospital DI order entry system is able to identify all duplicate DI exams (as per agreed upon
case definition) five-year post PACS and one-year pre PACS for selected sites.
• The reason for duplicate exam order is available from medical chart documentation, or
legitimacy of duplicate exam can be ascertained from other sources (e.g., patient has
pneumonia).
32
Data Sources and instruments:
• Hospital DI Order Entry System
• Patient Medical Chart
• Reason for Duplicate Exam Data Collection Sheet
Indicator Analysis:
A = Number of Redundant DI Exams Pre-PACS
B = Total number of Duplicate DI Exams Pre-PACS
C = Number of Redundant DI Exams Post-PACS
D = Total number of Duplicate DI Exams Post-PACS
%∆ = (A/B-C/D) x 100%
Associations:
A reduction in redundant exams can lead to: (a) cost savings in the institution; and (b) improved
quality of care for the patient due to avoidance of unnecessary testing and the availability of
previous exams (which can enhance continuity of patient care delivery).
Comments:
This indicator is intended to estimate the reduction of redundant exams resulting from the
introduction of PACS. There can be many legitimate reasons why a patient will have the same
DI exam generated over a short period of time (e.g., 3 days). In carrying out the study, it will be
important to remove all exams having a moderate to high probability of being legitimate. A high
proportion (e.g., 80%) of those duplicate exams left (e.g., having a low probability of being a
legitimate) may be assumed to be a redundant exam.
Another option that may be considered is to simply measure the reduction in the total number of
duplicate exams following the implementation of PACS. While easy to calculate, this indicator is
high on sensitivity and low on specificity. As well, it will not take into account any external
factors that might impact on the number of exams produced in a DI Department over time.
A pilot study in one high DI volume site is recommended.
References:
Cox B., Dawe N. (2002). Evaluation of the Impact of a PACS System on an Intensive Care Unit.
Journal of Management in Medicine Vol 16, No 2/3, pp199-205.
Naul L., Sinclear S. (2001). Radiology Goes Filmless. Postgraduate Medicine Online. Vo109/No
6/June
33
Siegel E., Reiner B. (2002). Filmless Radiology at the Baltimore VA Medical Centre: a 9-Year
Retrospective. Computerized Medical Imaging and Graphics 27, pp 101-109
Strickland N. (2000). PACS (Picture Archiving and Communications Systems): Filmless

Radiology. Archives of Disease in Childhood 83;82-86
Becker S., Arenson R. (1994). Cost and Benefits of Picture Archiving and Communication
Systems. Journal of the American Medical Informatics Association. Vol 1 No 5, Sept/Oct
34
3 DRUG INFORMATION SYSTEM PROGRAM
3.1 DRUG INFORMATION SYSTEM PROGRAM INDICATORS SUMMARY
Category & Sub- Study Question Indicator Measures

category
Quality
Change in patient safety Does access to the Drug # drugs (%) & # drugs/daily doses
complete drug profile interactions and (%) documented in chart that exist in
increase the therapy drug profile (gold standard)
completeness of the duplications
documented drug profile with a complete % detected drug interactions/
and identification of medication therapy duplications identified with
drug interactions, and profile chart documentation alone vs drug
therapy duplications? profile information
Change in patient safety Does the use of Completeness % of missing fields (e.g. quantity,
computer-generated and and legibility of directives) per prescription in hand-
printed prescriptions: scripts with e- written versus computer-generated
a) increase the prescribing printed prescriptions
completeness of
prescriptions and % of prescriptions where any
b) reduce the time required field for a legal prescription
required to process is judged illegible
prescriptions including
call-backs from Time (in secs.) for prescription
pharmacists to processing and call-backs to MDs
physicians? for clarification of the prescription
Change in patient safety Does a DIS system Adverse drug Chart-abstracted assessment and
reduce adverse drug- events identification of preventable adverse
related events? drug related events
Database assessment of hospital-

related discharge diagnoses that may
be attributable to prescribing errors
Change in patient safety Do alerts for prescribing Physician MD response to alerts (see system
errors at the time of response to indicators)
electronic prescribing target alerts at
reduce potentially time of e- Pharmacist response to alerts (? In
serious prescribing prescribing scope)
errors?
Incidence and prevalence of new
prescribing problems generated by
e-rx physicians with and without
decision-support among all eligible
patients (all on medication and all
high risk)
35
category
Productivity
Change in provider Does access to the Time to take Time (in seconds) for: 1) medication
efficiency complete drug profile medication history-taking, 2) patient assessment,
reduce the time required history/patient 3) pharmacy calls for information
for taking a medication assessment per patient by nurse, MD,
history? receptionist
Change in provider Does the use of Pharmacist Time to read, retrieve, enter, etc. the
efficiency computer generated and efficiency prescription for e-Rx
printed prescriptions
reduce the time required Efficiency rating of prescriptions for
to process prescriptions, new/repeat single/multiple
including call-backs medication
from pharmacists to
physicians? Percentage of e-Rx, computer
printed, and hand written
prescriptions requiring a change or
addition
Change in provider Does the availability of Number of Physician and pharmacist self-
efficiency 2-way communication pharmacist to reported assessment of time-saving
for prescription refill physician call and call-back frequency.
reminders reduce the backs
number of calls for Time-motion analysis of call-backs
prescription refills for written and electronic Rx and
between pharmacists time per script
and physicians?
36
3.2 DRUG INTERACTIONS AND THERAPY DUPLICATIONS WITH A COMPLETE
MEDICATION PROFILE
The Completeness of Drug Profile indicator quantifies the degree to which a DIS differs from
written patient charts in terms of completeness of drug lists.
Study Question:
Does access to the complete drug profile increase the completeness of the documented drug
profile and identification of drug interactions or therapy duplications?

Quality, Change in patient safety
Incomplete documentation of all drugs a patient is taking hampers the physician’s ability to
identify and avoid prescription errors that could lead to adverse drug events. Literature has
shown that both in the inpatient and outpatient setting, patient charts have incomplete listings of
the active drugs (Lau 2000, West 1994, Chrischilles 1988, Monson 1978). However, the rate at
which the drug lists were found to be incomplete varies widely, with West (1994) finding that
89% of the target drugs were documented, while Lau (2000) identified 61% of patients having
one or more drugs not registered on their chart. Therefore, an accurate assessment of the
completeness of the patient chart must be determined within the target group of DIS users to
realistically reflect the benefits to be obtained by adoption of the DIS system. The assessment of
completeness is the first step to assessing the possible safety benefits to be derived by reducing
the frequency of adverse drug events due to unrecognized drug interactions or therapeutic
duplications. An outline of the adverse drug event/drug interaction literature is provided in
Appendix B. By minimizing drug interactions and duplications, leading to adverse drug events,
substantial morbidity (and in some cases mortality) may be avoided.

• Emergency rooms, general practices
• Analysis for all patients, and also a separate analysis for vulnerable patients taking 5 or more
medications.
Measures:
• Chart completeness: The number drugs/daily doses found in written chart as a proportion of #
of drugs/daily doses in drug profile for all patients with one or more active prescriptions, and
for the subgroup of patients with five or more active prescriptions
37
• Undetected drug interactions or therapeutic duplications (chart vs. drug profile): Number of
clinically relevant drug interactions or therapeutic duplications found through chart review,
from list of clinically relevant drug interactions/duplications (see Appendix B) as a
proportion of the number of drug interactions or therapeutic duplications found through drug
profile review, for all patients with one or more active prescriptions, and for the subgroup of
• patients with five or more active prescriptions
• Undetected drug interactions or therapeutic duplications (chart vs. alert system): Number of
clinically relevant drug interactions or therapeutic duplications found through chart review
from list of clinically relevant drug interactions/duplications (see Appendix B) / Number of
alerts for clinically relevant drug interactions or therapeutic duplications from list of
clinically relevant drug interactions/duplications (see Appendix B) using automated alerting
system for all patients with one or more active prescriptions, and for the subgroup of patients
with five or more active prescriptions.
Exclusion Criteria:
• Prescriptions with stop orders or dose changes that indicate that the interaction or duplication
has been recognized and acted upon.
• Interactions/duplications that are justified in the chart (indicating that they have been
recognized)
• Interactions/duplications not included on the clinically relevant list.
Study Design:
Intervention and contemporaneous control group within the same province ideally randomized to
early or late intervention with the primary contrast being the difference in the pre-post
implementation comparison in the intervention and control groups. The difference in pre and
post comparisons estimates differences due to other co-interventions, secular trends and seasonal
variation in the control group and changes due to co-interventions, secular trends and seasonal
variation + changes due to the DIS intervention in the DIS intervention group. Thus, by using the
difference in the pre-post measures of prescribing problems and adverse events in the control and
intervention groups, one can isolate the independent, unique contribution of the DIS system. In
both groups and time periods (pre and post), chart review and drug profile review will be
conducted using a list of clinically relevant drug interactions/duplications identified in the
literature (see Appendix B).
Study Assumptions:
• Access and ability to print the complete drug profile by nurses and physicians, with < 10
• secs. of patient consent + patient ID data entry and retrieval of patient information within 5
• seconds
• Optional alerts for prescribing problems
38
• Identified prescribing problems substantially increase the risk of serious adverse drug-related
events in many high risk patients.

Patient charts, DIS drug profiles, DIS alert output files.
Indicator Analysis:
• Chart completeness: count of drugs/daily doses found in written chart / count of drugs/daily
doses in drug profile for all patients with one or more active prescriptions, and for the
subgroup of patients with five or more active prescriptions
• Undetected drug interactions or therapeutic duplications (chart vs. drug profile): count of
clinically relevant drug interactions or therapeutic duplications from target list found through
chart review / count of clinically relevant drug interactions or therapeutic duplications from
target list found through drug profile review, for all patients with one or more active
prescriptions, and for the subgroup of patients with five or more active prescriptions
• Undetected drug interactions or therapeutic duplications (chart vs. alert system): count of
clinically relevant drug interactions or therapeutic duplications from target list found through
chart review / count of alerts for drug interactions or therapeutic duplications from target list
using automated alerting system for all patients with one or more active prescriptions, and for
the subgroup of patients with five or more active prescriptions.
Associations:
If the chart vs. alert system analysis is undertaken, this indicator could also be used for Use: Use
behaviour/pattern, Information: Content, Satisfaction: User satisfaction, Ease of Use, and Net
Benefits: Quality dimensions.
Comments:
The ‘gold standard’ drug profile in the control group can be assessed by creating a drug list from
the same sources that would be available for the DIS drug profile.
References:
Chrischilles EA et al. (1988). Documentation and appropriateness of prescribing for Veterans
Administration ambulatory-care patients. Am.J.Hosp.Pharm. 45:2345-51.
Lau HS et al. (2000) The completeness of medication histories in hospital medical records of
patients admitted to general internal medicine wards. Br.J.Clin.Pharmacol. 49:597-603.
Monson R, Bond CA, Schuna A. (1981) Role of the clinical pharmacist in improving drug
therapy. Clinical pharmacists in outpatient therapy. Arch Intern Med. 141:1441-4
39
West SL et al. (1994) Completeness of prescription recording in outpatient medical records from
a health maintenance organization. J.Clin.Epidemiol. 47:165-71.
40
3.3 COMPLETENESS AND LEGIBILITY OF SCRIPTS WITH E-PRESCRIBING
The Completeness and Legibility of Prescription indicator will assess the impact of DIS on the
proportion of handwritten vs. computer-generated printed prescriptions received at community
pharmacies that are incomplete or illegible.
Study Question:
To what extent are computer-generated electronic printed prescriptions more complete and
legible than handwritten prescriptions?

In an in-hospital study of completeness and legibility of medication orders, Winslow et al (1997)
found that 20% of medication orders were illegible, and 24% were incomplete. The rate of
illegibility and incompleteness in prescriptions received in community pharmacies and the
magnitude of the improvement that can be produced by computer-generated printed prescriptions
needs to be quantified to determine the impact that DIS, specifically the benefits that mandatory
fields for prescriptions and a computer printed prescription order, could have for efficiency and
safety.

• Computer-generated printed and handwritten prescriptions received in community pharmacies
• Hand-written and computer-generated printed prescriptions from physicians in the area that are
and are not using the order-entry, e-Rx component of the system.
Measures:
• Number (%) of handwritten and computer-generated printed prescriptions received with
incomplete information on drug name, strength, quantity, and directions
• Number (%) of handwritten and computer-generated printed prescriptions rated as
• completely illegible or only partially legible on a 3-point scale.
Exclusion Criteria:
Verbal prescription orders.
41
Study Design:
Survey of handwritten and electronic prescriptions, rated for completeness of drug name,
strength,
quantity, directions, and legibility.
Study Assumptions:
• Electronic prescribing system with mandatory requirement to complete quantity and
• directions, menu-driven (structured) drug and dose selection from available products in
Canada, automatic abort on refills for narcotics, speed of completing electronic prescription
equal to or better than hand-written, and capacity to print within 10 seconds.

Sample of handwritten prescriptions from pharmacies, reporting sheets with 3-point rating scale
for legibility (as used in Winslow et al, 1997), and checklist for completeness of drug name,
strength, quantity, and directions.
Indicator Analysis:
Analysis of the proportion of computer-generated printed and handwritten prescriptions with:
• Incomplete drug name
• [Count of prescriptions with incomplete drug name / total number of prescriptions]
• Incomplete drug strength
• [Count of prescriptions with incomplete strength / total number of prescriptions]
• Incomplete quantity
• [Count of prescriptions with incomplete quantity / total number of prescriptions]
• Incomplete directives
• [Count of prescriptions with incomplete directions / total number of prescriptions]
• Illegible or partially illegible
• [Count of prescriptions rated illegible/partially illegible / total number of prescriptions]
Associations:
This indicator could also be used to inform the domain of Net Benefits:
Quality: Appropriateness/effectiveness.
Comments:
As there is some judgment required in assessing the legibility and completeness, the raters
should be tested for agreement to ensure reliability of results. Sampling should be structured to
ensure a wide variety of prescriptions and prescribing physicians.
References:
Winslow EH et al. (1997) Legibility and completeness of physicians’ handwritten medication
orders. Heart Lung. 26:158-64.
42
3.4 ADVERSE DRUG EVENTS
The Incidence and Prevalence of Prescribing Problems and Adverse Drug-related Event indicator
is designed to quantify the frequency at which drugs with potential for serious adverse events are
prescribed, both by physicians with alerting systems, and those without.
Study Question:
Do alerts for prescribing errors at the time of electronic prescribing reduce potentially serious
prescribing errors, and adverse drug related events?

To assess the impact of DIS on improving patient safety, the expected reduction in the incidence
and prevalence of prescribing problems and potentially preventable adverse drug-related events
(e.g. stroke), must be determined. The magnitude of this reduction will be an important
determinant of the cost-benefit of the DIS.
Setting and Population:

• High risk patients in general practice for physicians with and without decision support
• High risk patients with a higher probability of new drug treatment starts (i.e. incident
exposure: 50 – 70 years)
• Emergency and ambulatory-based primary care and specialty clinics.
Measures:
• Number of high risk patients with one or more of the target prescribing problems (see
Appendix B) / Number of high risk patients in practice, for physicians with and without
decision support
• Number of adverse drug-related events for high risk patients / all high risk patients, for
physicians with and without decision support.
Exclusion Criteria:
Patients with no medication.
Study Design:
Intervention and contemporaneous control group within the same province, ideally randomized
to early or late intervention with the primary contrast being the difference in the pre-post
implementation comparison in the intervention and control groups. The difference in pre and
43
post comparisons is due to other co-interventions, secular trends and seasonal variation in the
control group and changes due to co-interventions, secular trends and seasonal variation +
changes due to the DIS intervention in the DIS intervention group. Thus, by using the difference
in the pre-post measures of prescribing problems and adverse events in the control and
intervention groups, one can isolate the independent, unique contribution of the DIS system.
An alternate approach would be to use an interrupted time-series (pre-post) with adoption rate as
a time-varying covariate.
Study Assumptions:
• DIS used for > 50% of all patients and >80% of high risk patients,
• drug profile accessed for >80% with multiple prescribing physicians,
• response to alerts >50% to recommended alternate therapy for serious problems.

Electronic prescribing and alerts records from DIS, provincial and private insurer dispensed
prescription database; medical services (ER) visit +hospitalization + mortality database; random
sample of medical charts for adverse drug-related event (ADR) algorithm validation.
Indicator Analysis:
• Proportion of high risk patients seen in the previous year with one or more of the target
prescribing problems grouped by physicians with and without decision support
• Proportion of patients with an adverse drug-related event (any ADR, targeted ADRs where
there were prescribing problem alerts) grouped by physicians with and without decision
support.
Associations:
This indicator will help to inform the Net Benefits: Quality: Health Outcomes domain.
Comments:
The time examined may be varied as needed.
References:
N/A
44
3.5 PHYSICIAN RESPONSE TO TARGET ALERTS AT TIME OF E-PRESCRIBING
The Physician Response to Target Alerts indicator assesses whether being alerted to a serious
drug interaction when writing a prescription will reduce the number of prescriptions written that
could potentially lead to serious adverse drug events.
Study Question:
Do alerts for prescribing errors at the time of electronic prescribing reduce potentially serious
prescribing errors?

Physicians have complained that alerting systems have a very high rate of clinically irrelevant
alerts, and this may lead to ignoring even more important alerts (Glassman et al 2002, Magnus et
al 2002).

• Community-based primary care physicians and specialists
• High risk patients in general practice (those with >3 co-morbidities, >1 score on the Charlson
Index, and/or > 4 active drugs).
• Pre- and post-implementation assessments should be undertaken within the same province to
serve as a control population.
• To increase physician acceptability, there may be a staggered implementation (one location
received a system with lab information system, one receives a basic drug information system)
to allow benefits to all physicians, with different benefits at different times.
Measures:
• Percent of alerts in high risk patients that are over-ridden, overall, and broken down by
• reason for over-ride, for drug interactions/duplications/excess dose from the target list
• Percent of alerts in high risk patients from target list that result in a change of prescription
• Number of prescriptions written for drug combinations from the target list, pre-
implementation
• (no alerts available), for high risk patients.
Exclusion Criteria:
DIS with no alert over-ride function.
45
Study Design:
Intervention and contemporaneous control group within the same province, ideally randomized
to early or late intervention with comparison of pre-post differences in the two groups.
Audit trail analysis of physician response to alerts, and control group prescribing extracted from
prescribing system and prescription dispensed provincial databases.
Study Assumptions:
• Automated surveillance for serious prescribing errors,
• Recommended alternatives for patient-specific drug profiles,
• Ratio of alerts for serious to minor problems no greater than 1: 3 or capacity to select level of
screening.

Audit trails from DIS system, prescribing records from public and private insurers.
Indicator Analysis:
• Number of alerts from target list over-ridden in high risk patients / number of alerts from
• target list generated and seen for high risk patients
• Number of alerts from target list resulting in a change of prescription in high risk patients /
number of alerts from target list generated and seen for high risk patients
• Number of alerts from target list over-ridden for each over-ride reason in high risk patients /
Number of alerts from target list over-ridden for high risk patients
• Number of each of targeted alerts (including unseen due to settings change) in high risk
patients / Total number of alerts from target list (including unseen due to settings change) for
high risk patients
• Number of prescriptions written for drug combinations from target list pre-implementation in
high risk patients / total number of prescriptions for high risk patients
Associations:
This indicator is a subset of the Physician Response to Alerts indicator.
Comments:
Since a drug interaction and duplication requires two drugs to be prescribed, the definition of the
number of prescriptions for the pre-implementation rate of prescribing drugs resulting in
interactions and duplications will have to be defined.
References:
Glassman PA et al. (2002). Improving recognition of drug interactions: benefits and barriers to
using automated drug alerts. Med.Care. 40:1161-71.
46
Magnus D, Rodgers S, Avery AJ. (2002). GPs’ views on computerized drug interaction alerts:
questionnaire survey. J.Clin.Pharm.Ther. 27:377-82.
47
3.6 TIME TO TAKE MEDICATION HISTORY/PATIENT ASSESSMENT
The Consultation Time indicator assesses the time taken for different stages of a patient
consultation to show where time savings may be created by using DIS.
Study Question:
Does access to the complete drug profile reduce the time required to take a medication history?

In their 2003 paper, Lipton et al state that physicians were quite clear that using electronic
prescribing systems must save them time or money, or they would not be acceptable as an
alternative to current practice. To determine where time savings are available using DIS, a
consultation must be broken into its constituent parts and the average time for each determined,
and the impact of using DIS on these times quantified.

• Emergency rooms, general practices
• Pre- and post-implementation assessments should be undertaken within the same province to
serve as a control population. To increase physician acceptability, there may be a staggered
implementation (one location received a lab information system, one receives a drug
information system) to allow benefits to all physicians, with different benefits at different
times
• Analysis for all patients, and also a separate analysis for vulnerable patients taking 5 or more
medications.
Measures:
In general practice:
• Total time for consultation (ER nurse + MD components)
• Subcomponent times for:
o Introduction
o history taking
o consent (for use of DIS) **
o examining drug profiler (using DIS) **
o history of current medications including calls to pharmacies **
o physical examination
48
o prescription writing**
o test ordering
o determining plan of action
o closing of patient consultation
o other activities – not patient related
o refills requests for other patient by telephone/ by receptionist (not the patient under
examination) **
** activities that will likely be directly affected by the DIS
In each of these domains, time is further divided into file writing, file reading, and verbal
interaction with the patient. Where DIS is used in a manner different from the workflow without
DIS (for example, under examining profile or obtaining consent), a separate heading is provided
to distinguish between the time spent using the DIS for these activities, and the file reading, file
writing, or patient interaction activities.
Exclusion Criteria:
Patients who do not consent to the presence of the researcher-time-motion analyst observer
during their consultation.
Study Design:
• Time and motion study of a patient consultation, both for physicians with DIS, and a control
group of physicians with no DIS, as well as pre- and post- DIS implementation periods for
each physician
• Intervention and contemporaneous control group within the same province ideally
randomized to early or late intervention
• Pre-post implementation comparison.
Study Assumptions:
• Access and ability to print the complete drug profile by nurses and physicians, with < 10
• secs. of patient consent + patient ID data entry and retrieval of patient information within 5
• seconds
• Consultations for patients who do not consent to allow the researcher to observe their
appointment do not differ significantly from those patients who do grant consent.

Physician appointments with patients, in their practice. A touch-screen automatic timing entry
tool with an underlying database allows exact timing of each stage of the consultation.
49
Indicator Analysis:
• Difference in total time in seconds for file reading, file writing, and patient verbal interaction
with and without DIS access for physicians, nurses, and patients in the ER for:
o Introduction
o history taking
o consent (for use of DIS) **
o examining drug profiler (using DIS) **
o history of current medications including calls to pharmacies**
o physical examination
o prescription writing**
o test ordering
o determining plan of action
o closing of patient consultation
o other activities – not patient related
o refills requests for other patient by telephone/ by receptionist (not the patient under
examination) **
** activities that will likely be directly affected by the DIS
• Cumulative total reduction in time spent (in minutes/shift/month) and estimated costs in
personnel time saved
Associations:
This indicator may also be used to address the question ‘does the availability of 2-way
communication for prescription refill reminders reduce the number of calls for prescription refills
between pharmacists and physicians’. Results of the study may also inform the System:
Performance and Satisfaction dimensions.
Comments:
While it cannot be expected that using DIS will have an impact on every portion of the
consultation, knowing the average time for each part of the consultation will allow a breakdown
of where efficiency can be increased.
References:
Lipton HL, Miller RH, Wimbush JJ. (2003) Electronic prescribing: ready for prime time? J.
Healthcare Inf. Management 17(4): 72-79.
50
3.7 PHARMACIST EFFICIENCY
The Pharmacist Assessment of Benefits indicator determines the increase in efficiency of using
DIS by assessing the time spent by community pharmacists on processing handwritten
prescriptions, computer printed prescriptions, and electronically received prescriptions.
Study Question:
Does the use of computer generated and printed prescriptions reduce the time required to process
prescriptions, including call-backs from pharmacists to physicians?

Success of DIS systems will partly depend on their acceptability to pharmacists as well as
physicians. A decrease in the time to process prescriptions, and a decrease in the number of
callbacks to physicians to clarify prescriptions that are illegible or incomplete will increase
pharmacist efficiency.

Community pharmacists receiving handwritten, computer printed, and electronic prescriptions.
Measures:
• Time to read the prescription for e-Rx, printed Rx, and handwritten Rx
• Time to retrieve patient profile for e-Rx, printed Rx, and handwritten Rx
• Time for entry or checking of prescription information (for submission to adjudication) for e-
Rx, printed Rx, and handwritten Rx
• Time for checking process prescription for accuracy for e-Rx, printed Rx, and handwritten
Rx
• Time for resolving illegible prescriptions for e-Rx, printed Rx, and handwritten Rx
• Time for resolving incomplete prescriptions for e-Rx, printed Rx, and handwritten Rx
• Efficiency rating of prescriptions for new single medication for e-Rx, printed Rx, and
handwritten Rx
• Efficiency rating of prescriptions for multiple new medications for e-Rx, printed Rx, and
handwritten Rx
• Efficiency rating of prescriptions for repeat single medication for e-Rx, printed Rx, and
handwritten Rx
51
• Efficiency rating of prescriptions for repeat multiple medications for e-Rx, printed Rx, and
handwritten Rx
• Percentage of e-Rx, computer printed, and handwritten prescriptions requiring callback to
physician to clarify prescription
• Percentage of e-Rx, computer printed, and handwritten prescriptions requiring change in
posology based on professional judgment
• Percentage of e-Rx, computer printed, and handwritten prescriptions requiring addition of
supplemental information based on professional judgment
• Percentage of e-Rx, computer printed, and handwritten prescriptions requiring change
in DIN
Exclusion Criteria:
Community pharmacies who have never received a printed prescription (survey may be used for
those who have not downloaded electronic prescriptions).
Study Design:
Survey of community-based pharmacists with access to both electronic and hand-written
prescriptions.
Study Assumptions:
Pharmacists have received enough printed prescriptions, or where applicable, electronic
prescriptions, to accurately assess the average time or average percentage of prescriptions
needing
changes for the measures.

Pharmacist survey (see Appendix B).
Indicator Analysis:
• Average of estimated time in minutes to read the prescription for e-Rx, printed Rx, and
handwritten Rx.
• Average of estimated time to retrieve patient profile for e-Rx, printed Rx, and handwritten
• Rx.
• Average of estimated time for entry or checking of prescription information (for submission
• to adjudication) for e-Rx, printed Rx, and handwritten Rx.
• Average of estimated time for checking process prescription for accuracy for e-Rx, printed
• Rx, and handwritten Rx.
• Average of estimated time for resolving illegible prescriptions for e-Rx, printed Rx, and
handwritten Rx
• Average of estimated time for resolving incomplete prescriptions for e-Rx, printed Rx, and
handwritten Rx
52
• Number of pharmacists that rated e-Rx, computer printed, handwritten prescriptions most
efficient, or no difference among forms for new single medication prescriptions
efficient, or no difference among forms for new multiple medication prescriptions
efficient, or no difference among forms for repeat single medication prescriptions
efficient, or no difference among forms for repeat multiple medication prescriptions
• Average percentage of e-Rx, computer printed, and handwritten prescriptions requiring
callback to physician to clarify prescription
• change in posology based on professional judgment
addition of supplemental information based on professional judgment
• change in DIN
Associations:
Portions of this survey instrument may be used to assess the question ‘does the availability of 2-
way communication for prescription refill reminders reduce the number of calls for prescription
refills between pharmacists and physicians?’
Comments:
Many pharmacists may not be technically able to receive fully electronic prescriptions. However,
the survey may be used if they have received printed prescriptions only.
References:
N/A
53
3.8 NUMBER OF PHARMACIST TO PHYSICIAN CALLBACKS
The Time Saving and Call-back Frequency indicator will quantify the degree to which a refill
reminder system may save time for physicians and pharmacists, allowing them to see more
patients.
Study Question:
Does the availability of 2-way communication for prescription refill reminders reduce the
number of calls for prescription refills between physicians and pharmacists?

Call-backs to physicians for prescription refills take time and do not add any benefit to the
patient. If the time used to deal with refill calls could be reduced, physicians and pharmacists
would have more time with patients.
Setting and Population:

General practices and community pharmacies.
Measures:
• Number of calls per day from pharmacists to physicians for refills
• Time spent per day by pharmacists to call physicians for refills
• Number of calls per day received by physicians for refills
• Time spent per day by physicians responding to pharmacist calls for refills
Exclusion Criteria:
Pharmacist calls to physicians for other purposes (clarification of prescription, illegible
prescription).
Study Design:
Pre- and post-implementation assessments should be undertaken in a DIS intervention and
control group within the same province and time period. To increase physician acceptability of
being in the control group, there may be a staggered implementation (one location received a
system with refill reminders, one receives a basic drug information system) to allow benefits to
all physicians, with different benefits at different times.
54
Study Assumptions:
The DIS system used has an automated refill reminder functionality.

Survey of call-back frequency and time for physicians and pharmacists.
Indicator Analysis:
• Average of count of calls per day made by pharmacists to physicians for the purpose of refills
• Average time in minutes spent by pharmacists calling physicians for refills, including time
spent on hold
• Average of count of calls per day received by physicians for refills
• Average time in minutes spent by physicians taking calls from pharmacists for refills
• Analysis should be undertaken pre- and post-implementation for intervention and control
groups
Associations:
Information from the pharmacist survey and the physician time and motion study may be used in
assessing this indicator.
Comments:
Since most calls will be routed through a secretary, the count of calls to the physician may be
undertaken by the secretary.
References:
Higgins TJ, Diener JC, Grunsfeld T, Wogen S. (2002) Aim for perfection. Evaluate e-prescribing
technology using six-sigma techniques. Mgma Connexion Med Group Manage Assoc. 2:358.
55
4 LABORATORY INFORMATION SYSTEM PROGRAM
4.1 LABORATORY INFORMATION SYSTEM PROGRAM INDICATORS SUMMARY

category
Quality
Change in provider Does the JLIS Timeliness of Pre&Post-implementation:
effectiveness/ implementation improve access to test • Average turnaround times by
appropriateness of care turnaround time for results lab, region and sector for
results dissemination? specified tests.
Patient Safety Does the JLIS result in Impact of LIS on Post Implementation
anecdotal evidence of patient safety • Case Study required designed to
improvement in patient answer the question “Did the
safety? JLIS result in increased patient
safety?”
Patient safety Does the JLIS result in a Completeness of Pre&Post-implementation
more complete LIS lab profile • Number of Lab tests for an
profile for patients? individual patient within a
specified time period as
documented in their composite
of charts
• Number of Lab results available
within JLIS for an individual
patient within the same
specified time period
Productivity
Change in provider Does the JLIS result in Lab technician Pre&Post-implementation
efficiency reduced callbacks from callbacks • Number of call-backs from
Lab Technicians to ordering provider to lab
ordering provider? technologist
Change in provider What impact does the Clinician Pre&Post-implementation
efficiency JLIS have on provider workflow • Observational study required
workload & workflow, designed to address the question
& patient care? “What actually happens?”
Change in net costs Does the JLIS Change in Pre&Post-implementation:
implementation duplicate lab • Number of times a best practice
decrease utilization and tests duplicate test rule was invoked
unnecessary duplicate by month and action taken
testing? stratified by JLIS user
practitioners and non JLIS user
practitioners
56
4.2 TIMELINESS OF ACCESS TO TEST RESULTS
Focus Area: System Effect – Increase the quality and effectiveness of the healthcare system.
Objective: Improve patient safety by reducing the length of time required to return test results
(test turnaround time).
Study Question:
Does the JLIS implementation improve turnaround time for results dissemination?

Quality, Change in provider effectiveness/appropriateness of care
This effectiveness indicator assesses the rapidity of result reporting by measuring test turnaround
time from the time the specimen is collected at the Specimen Collection Centre (or the
equivalent) to the time when the test results is available and reported to the practitioner. Factors
affecting this indicator include the location where the test is collected and tested as well as the
type of test that was requested.
The premise behind this indicator is that the JLIS will improve the care physicians are able to
give patients by ensuring rapid result reporting.
Accurate and timely information is an important factor in the diagnosis and treatment of disease.
This particular segment of the testing and results generation has been chosen because it is the
only turnaround time segment that JLIS has complete control over. In effect, outside the context
of an emergency or critical care setting, controlling when the patient will go to the SCC to have
the specimen collected or when the practitioner will retrieve the test results is not feasible.

This indicator will track a representative bundle of pre-selected tests (similar to the Consumer
Price Index) as detailed below which feature the following criteria:
• High volume, and shorter testing time
• High volume and longer testing time
• Complex, yet critical tests.
This bundle of tests will be used to track turn around time (TAT).
Serum Sodium: Should represent the fastest possible turn around time and should be performed
by 98% of labs. The new Laboratory Information System is unlikely to have a significant impact
on TAT for this test as the TAT will already be optimized. For settings where electronic return of
results is not available, the LIS will shorten the TAT as long as “push” is employed.
57
Throat Culture: The fastest and least complicated of all microbiology tests. This is similar to
Serum Sodium but will be one day later. Similarly, the new Laboratory Information System is
unlikely to have a significant impact on TAT for this test as the TAT will already be optimized.
For settings where electronic return of results is not available, the LIS will shorten the TAT as
long as “push” is employed.
Serum Testosterone: A test that is performed by most large laboratories but is seldom done by
smaller labs. Thus, many of these assays will be collected and sent to a central lab within a
network. The new Lab Information System will not speed up the collection/transport/assay
components but should speed up the return of results. In settings where there is no electronic
reporting of results it will demonstrate the improvement over sending results back from a central
lab to a related but remote lab for distribution or to a remote practitioner.
HIV Testing: In most provinces, this is done by a central laboratory that is not part of the
organizational structure of the collecting laboratory. The collection, sending and testing phase of
the test will not be improved by the Lab Information System but the return of results should be
faster as many of the current results are returned by mail or require two steps (return to the
sending lab and then distribution to the requesting physician).
Triple Marker Test: Similar to HIV testing.
Measures:
Pre&Post-implementation:
Average (Mean, median & mode) turnaround times by lab, region and sector for specified tests.
Turnaround Time (TAT) is defined as the time between when a sample is taken from a patient to
the time the result is available to the ordering provider (whether viewed at that time or not).
Exclusion Criteria:
None.
Study Design:
Average turnaround time through manual means (from the time the specimen is collected to the
time the test is reported) in the given period using calculated data. Report monthly after L+3 for
a six month period. Continuous data collection might not be necessary past a certain period. A
checkpoint should also be established at L+12, L+18, L+24, L+30, & L+36 months.
If the information is available, which we are aware it is not in many jurisdictions; a baseline
should also be reported as the average for the month, 3 months prior to the Go Live date (L-3).
58
Study Assumptions:
• Criteria/Norms and parameters will be supplied for comparative reporting (e.g. the
turnaround time for a specific test).
• Turnaround time will differ from test to test as well as from lab to lab.
• We assume that no data is presently available that accurately measures manual turnaround
times.
• A proxy average turnaround time will be used for the bundle of tests.
• Test turnaround time will be reduced by the introduction of JLIS.
• The same bundle of tests used as the baseline will be collected during implementation.
o The delta between the two should demonstrate a reduction in test turnaround time.
• The true benefits of this indicator will be in the community labs where test turnaround time
pre-JLIS is not optimized.

Criteria Norms from MOHLTC Lab Branch will establish baseline for manual calculation.
Indicator Analysis:
TAT (in hours and minutes) for the selected bundle of tests in non-emergency or critical care
setting (e.g. Community Labs) stratified by laboratory, region and sector.
Associations:
None.
Comments:
Tests have been chosen for the Time indicators that specifically look at a variety of
recommended turnaround times (i.e. Minutes, hours, days, weeks).
References:
N/A
59
4.3 IMPACT OF LIS ON PATIENT SAFETY
Focus Area: System Effect – Increase safety of the healthcare system.
Objective: Impact of the JLIS on patient safety.
Study Question:
Does the JLIS result in anecdotal evidence of improvement in patient safety?

Quality, Patient Safety
Whilst difficult to measure, the impact of a JLIS on patient safety is vitally important to the
success of such systems. Therefore, this qualitative indicator is included. It is not anticipated that
all jurisdictions would be able to undertake this measure but that a jurisdiction would be able to
act as an exemplar in this for the others.

JLIS leads in each jurisdiction will be asked to identify an example of where they believe the
JLIS has improved patient safety. These will then need to be considered for which is likely to
demonstrate the greatest improvement and interviews and observations conducted in that area.
For example, it may be felt that the use of JLIS for cancer patients has specifically increased
patient safety as it would be known in the ER that the patients are immune suppressed when
presented with non cancer related symptoms.
Measures:
Case Study required designed to answer the question “Did the JLIS result in increased patient
safety?”
Exclusion Criteria:
None.
Study Design:
Qualitative case study conducted using semi-structured interviews & observation.
Study Assumptions:
• It is assumed that JLIS leads can identify an area of increased patient safety and that
providers and patients will be willing to be interviewed and observed regarding this.
60
Video or field notes from periods of ethnographic observation and audio transcription of
interviews.
Indicator Analysis:
Grounded theory analysis of qualitative materials.
Associations:
None.
Comments:
None.
References:
N/A
61
4.4 COMPLETENESS OF LAB PROFILE
Objective: Impact of the JLIS on the completeness of the patient record.
Study Question:
Does the JLIS result in a more complete LIS profile for patients?

Quality, Patient safety
Whilst difficult to measure the impact of a JLIS on completeness of a patient’s record is vitally
important to the success of such systems. Therefore, this indicator is included. It is not
anticipated that all jurisdictions would be able to undertake this measure but that a jurisdiction
would be able to act as an exemplar in this for the others.

This indicator will use a predetermined population that are known to require multiple laboratory
tests, of various kinds, within a specified time period.
Measures:
Pre&post implementation
• Number of Lab tests for an individual patient within a specified time period as documented
in their composite of charts
• Number of Lab tests billed for an individual patient within a specified time period
• Number of Lab results available within JLIS for an individual patient within the same
specified time period
Exclusion Criteria:
None.
Study Design:
Having selected a cohort of patients that meet the population requirements chart review will be
undertaken retrospectively for a pre-specified time period (of at least three months). A review of
JLIS records for this same cohort of patients, for the same time period will also be
retrospectively identified; as will a review of Billing data for this same cohort of patients, for the
same time period. A comparison of the three sets of data will lead to a measure of completeness.
62
Study Assumptions:
• This measure will require access to multiple charts across healthcare boundaries. i.e.
inpatient, ambulatory care and general practice. It is assumed that access to these multiple
charts is both possible, and permitted.
• It is assumed that all lab tests undertaken for a patient will be billed and available as a
billable data item in provincial data sources.

• Patient charts
• Provincial Billing Data
• By querying the repository generated from the JLIS
Indicator Analysis:
• Simple percentage calculations of completeness of:
o Number of tests identified in patient charts/Number of tests billed for patient
o Number of tests identified in patients charts/Number of test results identified in JLIS
For a subset of patients a side-by-side comparison should also be undertaken to ensure that there
is a correlation at the individual test level for each patient. That is that the number of tests, match
at the individual test level and not just at the composite number. (For example, that if a patient
has had 3 HbA1C tests documented in the charts and there are 3 test results in the JLIS that these
test results are for the 3 HbA1C tests ordered and not 3 other tests).
Associations:
To be associated with Drug Information Systems Indicator 1.
Comments:
None.
References:
N/A
63
4.5 LAB TECHNICIAN CALLBACKS
Objective: To improve capacity and quality of working life for lab techs & providers.
Study Question:
Does the JLIS implementation result in reduced callbacks from lab technologist to ordering
provider?

The goal for this indicator is to track the number of callbacks required between ordering provider
and lab technologists or call-desk staff. It is anticipated that the number should decrease due to
the implementation of the JLIS.

All lab technologists & call-desk staff working in JLIS participating laboratories.
Measures:
• Number of call-backs made from lab tech to ordering provider by provider role.
Exclusion Criteria:
None.
Study Design:
Reports will be generated monthly from 12 months preceding “go-live” for the duration of the
implementation period (L-12 months - L+36 months). A simple tally will be kept by the lab
technician of call-backs made by date.
Study Assumptions:
• Lab Techs will take part in the study and keep accurate tally sheets.

Tally sheets to be kept by lab technicians.
64
Indicator Analysis:
Simple count of number of call-backs by lab technician and laboratory, stratified by provider role
(Family Medicine, Specialist type etc).
Associations:
None.
Comments:
None.
References:
N/A
65
4.6 CLINICIAN WORKFLOW
Objective: Impact of the JLIS on provider workload & workflow.
Study Question:
What impact does the JLIS implementation have on provider workload & workflow?

Whilst difficult to measure the impact of a JLIS on provider workload and workflow it is vitally
important to the success of such systems. Therefore, this qualitative indicator is included. It is
not anticipated that all jurisdictions would be able to undertake this measure but that a
jurisdiction would be able to act as an exemplar in this for the others.

All providers in a pre-selected Emergency Room.
Measures:
Pre&post implementation
• Observations of how Lab results are handled, used and acted upon within an ER setting.
Exclusion Criteria:
None.
Study Design:
Qualitative observation study conducted using videos or field observers for prolonged periods of
time (at least 24 hour consecutive hours) at quarterly intervals from L-12 to L+36.
Study Assumptions:
• Access to an ER for the study could be facilitated and all providers would consent to being
observed for the period of study.

Video or field notes from periods of ethnographic observation.
66
Indicator Analysis:
Grounded theory analysis of ethnographic materials.
Associations:
To be associated with indicator 5.
To be associated with iEHR indicator 9.
Comments:
None.
References:
N/A
67
4.7 CHANGE IN DUPLICATE LAB TESTS
Focus Area: System Effect – Increase patient safety and creating economic efficiencies.
Objective: Reduction of unnecessary duplicate testing across jurisdiction.
Study Question:
Does the JLIS implementation decrease utilization and unnecessary duplicate testing?

Productivity, Change in net costs
This indicator assesses how successful JLIS will be in avoiding unnecessary duplicate testing
over a period of time by measuring the % reduction of successful avoidance of repeat testing by
either accepting the system rule described below, or by abandoning the order once the rule has
been generated.
Repeat testing is often necessary in the treatment or monitoring of disease. For example, blood
chemistry of an HIV patient is regularly monitored to determine the effectiveness of drug
therapy. Similarly, glucose levels in a Type II diabetic are monitored to determine the optimum
dosage and frequency of insulin injections.
Unnecessary duplication occurs when a patient is re-tested even though a clinically valid result is
available. The cost of unnecessary duplication goes well beyond the cost of the test itself. Often
it requires a patient to make an additional visit to the practitioner as well as requiring the
collection of an additional specimen. In more serious cases, it may affect patient safety by
delaying critical treatment. In all cases, it adds to the workload of busy practitioners and
providers. Unnecessary duplication of laboratory tests can happen for many reasons including:
• A practitioner misplacing or misfiling a report
• A referring practitioner not forwarding a copy of the test result to a specialist or the report
does not arrive in a timely manner
• A patient seeing a new doctor
• A patient arriving unexpectedly at a hospital
• A result, which is normal for the patient but is usually considered abnormal
When a practitioner orders a test for which a clinically valid result exists, JLIS will inform the
practitioner of its presence. The practitioner may then retrieve the result, or may reorder it if
he/she determines it clinically necessary.
68
Once the system has both been deployed and has begun to be used by the targeted adopters, it
will be important to measure how much the system actually improves the safety and efficiency of
Lab systems. One way this may be quantified is by tracking whether the duplicate test rule
decreases the number of duplicate tests carried out.

Any, or all, of the following tests should be used for this measure:
Hemoglobin A1c
• Longer term nature
• Ordered by many different practitioner groups on a regular basis
• Good for about 120 days (has a clinical validity period of about 90 days)
• normal 1x in 90 days
Alkaline Phosphatase
• normal 2x in one week
Sodium
• normal 2x in one week – out-patient only
Hemoglobin
• normal 2x in one week – out-patient only
Cholesterol
• 2x over 2 weeks – in-patient or out-patient
TSH
• normal 2x within 4 weeks
Measures:
• Number of times a best practice duplicate test rule was not followed.
(Gen3+ only) - Number of times a best practice duplicate test rule was invoked by month and
action taken stratified by JLIS user practitioners and non JLIS user practitioners.
Exclusion Criteria:
None.
69
Study Design:
Reports will be generated quarterly from 12 months prior to the Go-Live date to the end of the
implementation period (L-12 - L+24). An annual trends report will be submitted Reports will be
generated semi-annually from the end of the implementation data (L+24 months) to the end of
the agreed upon evaluation period (no later than L+36 months).
Hemoglobin A1c
1. Filter pseudonymous repository for all Hemoglobin A1C tests conducted in time period
(quarterly)
2. Search all resulting tests for duplicate patient IDs
3. Identify any duplicate tests conducted within 90 days of each other over the same set of
results for the previous period.
Alkaline Phosphatase
1. Filter pseudonymous repository for all Alkaline Phosphatase tests conducted in time period
(quarterly)
Sodium
1. Filter pseudonymous repository for all Sodium tests conducted in time period (quarterly)
Hemoglobin
1. Filter pseudonymous repository for all Hemoglobin tests conducted in time period (quarterly)
Cholesterol
1. Filter pseudonymous repository for all Cholesterol tests conducted in time period (quarterly)
TSH
1. Filter pseudonymous repository for all TSH tests conducted in time period (quarterly)
70
Study Assumptions:
• No real-time systems in place to avoid or monitor duplicate testing.
• Only Gen3+ systems will have the functionality so that an Order will allow user to accept
recommendations of rule (e.g. button) in a way that can be tracked/captured by the system.
Synchronization with roll-out waves.
• Using the transaction log as a place to sift the data would require a very long processing
window for the query - possibly taking more than 24 hours, since none of the information
would be keyed data.
• Context will be defined clearly. That is, it will be known whether the tests are being
conducted on an inpatient or out patient setting.
• Data coming from Hospital Emergency or Critical Care settings will be filtered out.
• Total tests can be segregated between those ordered through JLIS and those that have order
information attached to the result.
• Patients have unique identifiers within the JLIS and pre JLIS systems.

• By querying the pseudonymous repository generated from the JLIS
• Provincial billing data.
Indicator Analysis:
Number of duplicate tests (as per criteria stated above) within time quarter stratified by test, Lab
type & contextual setting (in-patient versus out-patient) or provider making the order.
Associations:
To be associated with iEHR indicator 2.
Comments:
Specific tests will be chosen to be measured re. the duplicate test rule that will specifically
address out-patient ordering as primary care orders may not be part of the JLIS at this time.
References:
N/A
71
4.8 DATA REQUIREMENTS – ALL MEASURES
L-12 L-6 L=0 L+6 L+12 L+18 L+24 L+30 L+36

1. Capacity
Number of test results by Monthly
user role, ordering facility,
region & facility type
2. Duplicates
Number of times a best Quarterly Semi-Annual
practice rule was not
followed
3. Time
Average TAT by lab, Monthly Semi-
region & sector Annual
4. Call-Backs
Number of call-backs from Monthly
lab tech to ordering
provider by provider role
5. Adoption
Number of participating Monthly
users by geographical
region & user role with a
unique user account with
JLIS
Number of active users by Monthly
geographical region & user
role that have ordered tests
role that have viewed lab
results either via JLIS or
pulled from the JLIS by
their EMR
role that have viewed lab
results either via JLIS or
pulled from the JLIS by
their EMR the same lab
results for an individual
patient within a defined
time period
6. Impact
Observations of how lab 24hr periods of observations quarterly
results are handled, used
and acted upon within an
ER setting
7. Completeness
72
Number of lab tests for an Monthly

individual patient within a
specified time period as
documented in their
composite of charts
Number of lab tests billed Monthly
for an individual patient
within a specified time
period
Number of lab results Monthly
available for an individual
patient within the specified
time period
8. Safety
Interviews & observations
re. perceived increase in
patient safety
73
5 PUBLIC HEALTH SYSTEM PROGRAM
5.1 PUBLIC HEALTH SURVEILLANCE PROGRAM INDICATORS SUMMARY

category
Quality
Change in provider Does an immunization Vaccination rate Eligible individuals with full
effectiveness/ registry increase vaccine coverage.
appropriateness of care immunization rates? Vaccinated individuals that received
vaccinations according to schedule.
Change in provider Does a PHS system Outbreak Number of outbreaks detected.
effectiveness/ enhance outbreak detection and Average time until an outbreak is
appropriateness of care detection and limit intervention detected.
disease spread? Average number of secondary
infections per case.
Productivity
Change in care co- Does case-management Time spent Average time to share case data
ordination software enhance managing with patient’s region of residence.
coordination of care? outbreaks Average time per case for managing
outbreaks that span regions
Change in net cost Does an immunization Avoidance of Cost of vaccines wasted
registry reduce vaccine wastage Cost of unnecessary vaccinations
unnecessary vaccine and unnecessary
costs? vaccinations
74
5.2 VACCINCATION RATE
The proportion of the target population up-to-date for recommended immunizations.
Study Question:
Does establishment of an immunization registry with the necessary functionality result in an
increase in immunization rates for children and eligible adults?

A population-based immunization registry gathers information about immunizations for all
individuals residing within a geographic area. The availability of this information in a standard
format enables functions or interventions (1) to identify patients who are eligible for
immunization and (2) to notify eligible patients and their providers about the need for
immunization. Notification of patients (Jacobson 2005) and providers (Garg 2005) of the need
for immunization are both effective strategies for increasing vaccine coverage. Absolute
improvement in coverage rates with patient notification should be in the range of 1% to 20%
(Jacobson 2005). Establishment of an immunization registry and the functionality to identify and
notify patients eligible for immunization should, therefore, increase immunization coverage in
the population. As immunization prevents infection, establishment of a registry with the
necessary functionality should also ultimately result in a decrease in infection due to vaccine-
preventable diseases.

• Geographic areas with plans to establish an immunization registry
• Children ≤ 24 months of age
• Adults eligible for pneumococcal vaccination.
Measures:
• Proportion of children ≤ 24 months of age that are up-to-date for routine vaccinations.
• Proportion of adults eligible for pneumococcal vaccine that are immunized.
• Both measures should be calculated for all individuals, that is, individuals enrolled in the
registry, and individuals not enrolled in the registry.
Exclusion Criteria:
• Individuals not eligible for immunization (e.g., medical contraindication, religious objection)
should be excluded from the denominator when calculating immunization rates.
75
Study Design:
• A before-and-after design, ideally with multiple follow-up assessments and across multiple
regions
• For children, a series of cross-sectional surveys using cluster probability sampling of small-
areas (e.g., postal codes). See the study by Rosenthal for an example of a suitable approach
(Rosenthal 2004). At a minimum, surveys should be conducted twice within the same region,
once before implementation of the registry and again at some time interval during or
following implementation. Performing multiple follow-up assessments and/ or performing
the same study in different regions would strengthen the design.
• For adults, the survey design for children (or even the same survey) could be used.
Depending on the setting, an alternative design would be to use billing records to estimate
pneumococcal vaccination and the population eligible for vaccination. See the study by
Schopflocher as an example (Schopflocher 2003).
Study Assumptions:
• The magnitude of the benefit will be influenced by the proportion of the population enrolled
in the registry, the completeness of information in the registry, and the accuracy of
information in the registry (Boyd 2002). Concurrent studies to assess these issues (see
Section 8.4) should be undertaken to provide context for the results of this study. These
factors should be assessed at each sampling interval to allow estimation of the effect of these
factors on the outcomes of interest.
• Automated algorithms for identifying individuals eligible for vaccination should be
developed and applied routinely to the data held in the registry. In addition, automated
systems for notifying patients and / or providers about necessary vaccinations should be used.
Without these identification and notification functions, a registry is not likely to have a large
impact on vaccination rates.

• Surveys
• Enrollment records in registry
• Billing records
Indicator Analysis:
• From the survey results, calculate at each time point the proportion of children up-to-date for
routine vaccinations (and eligible adults that have received pneumococcal vaccine). Use the
eligible, due, and overdue guidelines for immunization registries (Boulianne 2004) to
determine up-to-date and eligibility.
• Stratify calculation of coverage by whether the individual is enrolled in the registry.
76
• Calculate the change in coverage over time for all individuals as the difference in rates
between survey points. If multiple regions (i.e., multiple registries) are included in the study,
calculation of change over time should be stratified by region.
• For time-points where the registry is established, calculate the difference in coverage
between individuals who are and who are not enrolled in the registry.
Comments:
• The possibility of conducting the survey as an add-on to an existing survey, such as the
Canadian Community Health Survey, should be explored.
• Design would be stronger if multiple locations could be studied, ideally with implementation
occurring at different times across the locations.
• Enrollment in the registry should occur quickly over time once the registry is established in
order to minimize the influence of other effects.
References:
Boulianne, N., Y. A. Hemon, et al. (2004). National eligible, due, and overdue guidelines for
immunization registries: draft recommendations from the Canadian Immunization Registry
Network, Data Standards Task Group. Can Commun Dis Rep 30(6): 53-9.
Boyd, T. D., R. W. Linkins, et al. (2002). Assessing immunization registry data completeness in
Bexar County, Texas. Am J Prev Med 22(3): 184-7.
Garg, A. X., N. K. Adhikari, et al. (2005). Effects of computerized clinical decision support
systems on practitioner performance and patient outcomes: a systematic review. JAMA 293(10):
1223-38.
Jacobson, V. J. and P. Szilagyi (2005). Patient reminder and patient recall systems to improve
immunization rates. Cochrane Database Syst Rev(3): CD003941.
Rosenthal, J., L. Rodewald, et al. (2004). Immunization coverage levels among 19- to 35-month-
old children in 4 diverse, medically underserved areas of the United States. Pediatrics 113(4):
e296-302.
Schopflocher, D. P. and M. L. Russell (2003). Method for ascertaining denominators for

evaluation of population coverage with pneumococcal vaccine. Can J Public Health 94(5): 377-
80.
77
5.3 OUTBREAK DETECTION AND INTERVENTION
Outbreak detection and intervention.
Study Question:
Does a public health surveillance system enhance the detection of outbreaks and limit the spread
of disease?

One of the main reasons for conducting reportable disease surveillance is to identify outbreaks of
disease rapidly. Prompt identification of outbreaks enables interventions aimed at preventing the
spread of disease. If the spread of disease can be prevented, this should mitigate the morbidity,
mortality and costs attributable to an outbreak. This mitigation is one the main reasons given for
implementing a public health surveillance system (Walker 2003). One approach to evaluating
these benefits is to assess the incremental impact of a system on a health department’s ability to
detect outbreaks and to intervene successfully.

• Entire population in a region
• Cases of selected reportable diseases associated with outbreaks
Measures:
• Number of outbreaks detected.
• Time until detection of an outbreak. The method used to calculate this measure will vary
according to the type of outbreak and the causative organism. Details and examples are given
under ‘Indicator Analysis’ on the next page.
• Number of secondary cases attributable to each case.
Exclusion Criteria:
• Certain reportable diseases may not be well suited to this type of evaluation for a variety of
reasons. Reportable disease with a sufficient incidence of cases and outbreaks should be
selected for this study.
78
Study Design:
• A before-and-after design, ideally across multiple locations. The unit of analysis should be a
health department (as opposed to a province) or the lowest level of regionalization at which
outbreak detection and control are performed.
• Measures should be assessed over a baseline interval prior to implementation of the PHS.
This assessment may be prospective, if time permits, or retrospective, if data permit.
• Measures should be assessed again at least once after the baseline assessment over an interval
following the implementation of the PHS.
• Over the course of each measurement interval, data should also be collected about the
methods used (1) to analyze data for outbreaks, (2) to investigate potential outbreaks, and (3)
to manage cases of disease.
Study Assumptions:
• Public health departments must have a protocol for recording outbreaks prior to
implementation of the PHS or it must be possible to determine the existence of outbreaks
retrospectively from records.
• Once the PHS is implemented, public health departments must perform some type of
automated analysis to detect disease outbreaks. As analytic software and specific analysis
approaches are outside of the core PHS implementation, the potential impact of different
analytic protocols between departments must be taken into account. Ideally, public health
departments would use standard protocols for surveillance analysis.

• Records of communicable disease outbreaks prior to implementation of PHS and individual
case reports for individuals involved in outbreaks.
• Records within the PHS for outbreaks and individual cases involved in outbreaks.
Indicator Analysis:
• The number of disease outbreaks detected over a time interval is the count of identified
outbreaks divided by the duration of the interval. This and all other measures should be
calculated separately for all diseases of interest.
• The time until detection of a disease outbreak is the difference between the time that an
outbreak was detected and either (1) the time of symptom onset for the earliest known case
or (2) the time of initial common exposure, if known and relevant. The type of disease and
the nature of the outbreak will determine the appropriate method for computing time to
detection. For example, in the case of an outbreak of campylobacter resulting from a
common source exposure (e.g., contaminated meat at a large gathering), the time of the initial
common exposure is clear and this time should be used. As another example, in the case of
an outbreak of Hepatitis A where the mechanism of transition is person-to-person spread, the
79
time of symptom onset for the first case linked epidemiologically to the outbreak should
be used.
• The number of secondary cases attributable to each disease case is calculated as the total
number of other cases known or suspected to be due to exposure to the index case. The
information on the certainty of the attribution (e.g., suspected or know) should be retained for
use in the analysis.
Comments:
• Pilot work with selected health departments should be performed (1) to establish the
feasibility of identifying outbreaks and associated cases retrospectively and (2) to identify the
most appropriate disease for inclusion in the study including an assessment of whether the
numbers of outbreak and cases are sufficient to detect a change in the indicators.
• Simulation may also be used to evaluate the performance of the PHS for outbreak detection
in different settings (Buckeridge 2005, Kleinman 2005), but simulation is not able to
establish a baseline for performance prior to implementation of PHS unless it is possible to
develop a model of the analytic process prior to the implementation.
• Another possible design is one where experts review retrospectively time-series of
aggregated cases and specify the locations of outbreaks, in their opinion. Both pre-
implementation and post-implementation outbreak detection performance can then be
assessed against the identified outbreaks (Siegrist 2004, Kleinman 2006).
References:
Buckeridge, D. L., P. Switzer, et al. (2005). An evaluation model for syndromic surveillance:
assessing the performance of a temporal algorithm. MMWR Morb Mortal Wkly Rep 54 Suppl:
109-15.
Kleinman, K., A. Abrams, et al. (2006). Evaluating spatial surveillance: detection of known
outbreaks in real data. Stat Med 25(5): 755-69.
Kleinman, K. P., A. Abrams, et al. (2005). Simulation for assessing statistical methods of
biologic terrorism surveillance. MMWR Morb Mortal Wkly Rep 54 Suppl: 101-8.
Siegrist, D. and J. Pavlin (2004). Bio-ALIRT biosurveillance detection algorithm evaluation.

MMWR Morb Mortal Wkly Rep 53 Suppl: 152-8.
Walker, H. and M. Anderson (2003). Assessment of the Benefits and Costs of Adoption of i-
PHIS, Walker Economics.
80
5.4 TIME SPENT MANAGING OUTBREAKS
The time spent managing cases of reportable diseases.
Study Question:
Does a PHS change the amount of time spent managing cases within and across regions?

Productivity, Change in care co-ordination
Productivity gains are cited as a main reason for implementing a PHS. Hypothesized areas for
benefit include faster data entry and reduced duplication (Walker 2003). Evidence from the use
of electronic health records in hospital settings suggests, however, that data entry time may
decrease or increase following introduction of automated systems (Poissant 2005). Similarly,
automated transmission of communicable disease reports may decrease the delay between
diagnosis and reporting, but it may also increase the number of duplicate reports received by
public health (Effler 1999). This indicator will summarize the overall change in the time spent
managing cases of communicable diseases, from when a report is received until the case is
closed. In doing so, the indicator will summarize the total impact of a PHS on a process that is
central to most public health departments.

• Reportable disease unit within a public health department.
• Personnel involved in receiving reports, opening case files, and managing cases.
• Selected reportable diseases.
Measures:
• Time used to receive a report of a disease and file the report.
• Time used to open a case file for a reportable disease.
• Time used to manage a case of a reportable disease.
Exclusion Criteria:
• Certain reportable diseases may not be well suited to this type of evaluation for a variety of
reasons. Reportable disease with a sufficient incidence of cases should be selected for this
study.
81
Study Design:
• Time and motion study, performed prior to implementation of the PHS and at least once
following implementation.
• The study should be conducted in at least two neighboring public health units to allow
assessment of the time spent managing cases that involve more than one health department.
• Calculation of average times should be stratified by disease and whether cases were managed
wholly by one health department or not.
Study Assumptions:
• Experience with information system implementation in other settings suggests that
productivity will vary over time following implementation. We assume the same result will
occur in a public health setting.
• Users should be trained to perform the functions evaluated in this study.

• Time and motion study results.
Indicator Analysis:
• The difference in each time measure should be calculated between the baseline interval and
the post-implementation interval(s).
• The difference should also be calculated for the sum of all time measures, with summation
over individuals.
• The indicator should be calculated separately for different reportable diseases.
Comments:
• The quality of training and the timing of the second evaluation will influence the study
results.
• This indicator can be linked with a productivity indicator for laboratory information systems
to determine the total benefit of automated case reporting from labs to public health.
References:
Effler, P., M. Ching-Lee, et al. (1999). Statewide system of electronic notifiable disease
reporting from clinical laboratories: comparing automated reporting with conventional methods.
JAMA 282(19): 1845-50.
Poissant, L., J. Pereira, et al. (2005). The impact of electronic health records on time efficiency of
physicians and nurses: a systematic review. J Am Med Inform Assoc 12(5): 505-16.
Walker, H. and M. Anderson (2003). Assessment of the Benefits and Costs of Adoption of i-
PHIS, Walker Economics.
82
5.5 AVOIDANCE OF VACCINE WASTAGE AND UNNECESSARY VACCINATIONS
The cost of vaccines wasted due to sub-optimal logistics of vaccine management and
unnecessary duplicate vaccination.
Study Question:
Does an immunization registry reduce unnecessary vaccine costs?

Productivity, Change in net cost
Vaccines are expensive medications which often have special requirements for handling and
storage. The estimated cost of vaccines for Ontario’s children to meet NACI recommendations
for only four diseases is over $460 million (Reynolds 2003). Surveys suggest that somewhere
between 1% and 5% of vaccine doses are wasted, mainly due to refrigeration lapses and
expiration (Setia 2002). Another source of vaccine cost is the administration of invalid doses,
which must be repeated (Stokley 2004). Invalid doses are doses that are administered at the
incorrect time according to an individual’s immunization schedule and that must be administered
again. Taken together, wasted and invalid vaccine doses likely result in millions of dollars of
unnecessary costs. An immunization registry and a vaccine management system with the
appropriate functionality may decrease these costs. Dosing decision-support and patient or
physician reminder systems may reduce the occurrence of invalid doses and software to support
the logistics of vaccine distribution may reduce the occurrence of wasted doses.

• Geographic region with plans to implement an immunization registry.
Measures:
• Number and cost of wasted vaccine doses by vaccine type.
• Number and cost of invalid doses by vaccine type.
Exclusion Criteria:
None.
Study Design:
• A review of expense records should allow calculation of the number and cost of wasted
vaccine doses.
• The number and cost of invalid doses could be assessed using the survey described for
assessment of vaccine coverage rates.
83
• In both cases, the indicator should be calculated over time and the total number of vaccines
administered.
Study Assumptions:
• Baseline data on wasted and invalid doses must be available and it must be possible to access
these data prospectively through the PHS.

• Survey results.
• Public health unit and ministry records of wastage and vaccine costs.
Indicator Analysis:
• The difference in the number and the cost of wasted and invalid doses by vaccine should be
calculated between the baseline and post-implementation intervals.
• The difference for wasted doses should also be calculated by delivery settings (e.g., public
versus private clinics) and for components of the supply chain.
Comments:
• The potential reduction in invalid doses will be influenced strongly by the coverage of the
immunization registry, the accuracy of the algorithms used to define immunization
schedules, and the ability to recall individuals to be immunized.
References:
Reynolds, D. (2003). Estimates of Cost of Vaccine Coverage, Ontario, Association of Local
Public Health Agencies.
Setia, S., H. Mainzer, et al. (2002). Frequency and causes of vaccine wastage. Vaccine 20(7-8):
1148-56.
Stokley, S., E. Maurice, et al. (2004). Evaluation of invalid vaccine doses. Am J Prev Med 26(1):
34-40.
84
6 INTEROPERABLE ELECTRONIC HEALTH RECORD PROGRAM
6.1 INTEROPERABLE ELECTRONIC HEALTH RECORD PROGRAM

INDICATORS SUMMARY
Category & Sub - Study Question Indicator Measures

category
Quality
Change in health system Does use of the iEHR Readmission rates - Hospital readmission rates within 30
outcomes reduce hospital days after discharge using the
readmission rates after Discharge Abstract Database
discharge?
Change in health Does the iEHR improve Chronic disease - Chronic disease performance
outcomes chronic disease management measures in patients with asthma,
management? diabetes, heart failure, and hypertension
- Qualitative data from focus group
sessions
- Survey data
Productivity
Change in provider Does the iEHR improve Provider - Time-motion studies with direct
efficiency the productivity and efficiency and observation of providers in the ED
efficiency of the effectiveness in - Timers and/or other recording devices
emergency department emergency given to patients to track time in ED
(ED)? departments from time they arrive in the ED to time
they are discharged
- Qualitative data from focus group
sessions with providers
- Number of lab and imaging tests
ordered in the ED
- Costs of lab and imaging tests ordered
in the ED
85
6.2 READMISSION RATES
Hospital readmission rate is the number of times a patient is readmitted within 30 days after
being discharged from the hospital.
Study Question:
Do hospital readmission rates decrease after implementation of the iEHR?

Quality, Change in health outcomes
Use of the iEHR will provide better communication between providers when patients are
discharged from the hospital. Providers caring for patients after discharge will now have access
to patients’ laboratory, medication, imaging information and discharge summaries from their
hospital stay, as well as important clinical information from other outpatient providers (e.g., a
patient’s primary care physician, cardiologist, and rheumatologist) who are seeing the patient
after discharge. This type of coordinated care should result in improved quality of care and
therefore lead to a decrease in readmission rates to the hospital, since about 25% of all
readmissions to the hospital are estimated to be preventable.

• Hospitals with iEHR implemented
• All patients in a given study population who have been admitted to the hospital, discharged,
and re-admitted within 30 days.
Measures:
The number of patients readmitted to acute inpatient hospitals within 30 days of being
discharged from an acute inpatient hospital stay will be measured.
Exclusion Criteria:
None.
Study Design:
Patient readmission rates will be measured in the 12 months prior to implementation of the iEHR
and then again in the 12 months after implementation of the iEHR and compared. An additional
study design that Infoway may wish to consider is to randomize a certain number of sites (e.g., 4
or 6 sites) with similar demographic characteristics. At half of the sites, the iEHR will be
implemented. At the other sites, it will not be implemented. Data on hospital readmission rates at
86
these sites can then be collected 12 months prior to and 12 months after implementation of the
iEHR. Comparisons can then be made within each site using pre- and post- implementation data
on readmission rates. Inter-site comparisons can also be made comparing hospital readmission
rates between those sites that implemented the iEHR program and those that did not.
Study Assumptions:
• Data on admissions and discharges from the hospitals is readily available
• The data includes information that would allow identification of patients, since this is
necessary to determine if a patient is readmitted within 30 days of his /her most recent
hospital discharge
• If it is not available, this measure would require chart review, which would be labor intensive
and costly
• The assumption is that use of the iEHR would decrease the rate of hospital readmissions.

• Hospital admission and discharge data
• Inpatient chart review can also be utilized as a source of data if desired.
Indicator Analysis:
Count of number of patients readmitted within 30 days of hospital discharge.
Associations:
None.
Comments:
None.
References:
N/A
87
6.3 CHRONIC DISEASE MANAGEMENT
For the purposes of this study, chronic disease management refers to the appropriateness of care
that is provided to patients with the following conditions: asthma, diabetes, heart failure, and
hypertension. Appropriateness of care will be determined by adherence to commonly accepted
practice guidelines and avoidance of preventable complications (e.g., hospitalization for asthma
exacerbation).
Study Question:
• Does use of the iEHR improve chronic disease management, specifically in asthma, diabetes,
heart failure, and hypertension?
• Beyond these four specific conditions, does use of the iEHR help providers make better
diagnostic and treatment decisions?

Use of the iEHR can potentially improve chronic disease management. Allowing clinicians to
have access to lab, imaging, medication information, hospital discharge summaries, ED visit
summaries, allergies, vaccinations, and blood type will provide them with important information
that will enable them to make better and more informed clinical decisions. It will also facilitate
coordination of care across providers and sites of care.

• Physician practices with iEHR implemented
• Hospitals with iEHR implemented
• Sample of patients with asthma, diabetes, heart failure, or hypertension receiving care by
providers at sites that have implemented iEHR.
Measures:
For patients with asthma:
• Number of patients admitted to the hospital related to asthma exacerbation / uncontrolled
asthma
• Number of patients seen in the ED related to asthma exacerbation / uncontrolled asthma
• Of those who qualify for controller therapy (e.g., inhaled corticosteroid therapy) based on
guideline criteria, the number and percentage of patients who are prescribed a daily controller
agent
88
• Of those who qualify for PRN rescue medication therapy (e.g., inhaled beta-2 agonist
therapy) based on guideline criteria, the number and percentage of patients who are
prescribed rescue medication
For patients with diabetes:

• Number and percentage of patients with a hemoglobin A1C measure performed and
documented within the previous 12 months
• Number and percentage of patients with the most recent hemoglobin A1C ≤ 8.0
• Number and percentage of patients who are admitted to the hospital related to uncontrolled
diabetes
• Number and percentage of patients seen in the ED related to uncontrolled diabetes
• Number and percentage of patients (without a contraindication) on a daily aspirin
• Number and percentage of patients (without a contraindication) on an ACE-inhibitor or
angiotensin receptor blocker
• Number and percentage of patients whose LDL cholesterol is 100
For patients with heart failure from systolic dysfunction:

• Number and percentage of patients of patients with documented ventricular function (e.g., by
echocardigram)
• Number and percentage of patients (without a contraindication) on an ACE-inhibitor
• Number and percentage of patients (without a contraindication) on a beta-blocker
• Number and percentage of patients who are admitted to the hospital related to heart failure
exacerbation
• Number and percentage of patients who are seen in the ED related to heart failure
exacerbation
For patients with hypertension

• Number and percentage of patients of patients with blood pressure ≤ 140/80
• Number and percentage of patients with documented high blood pressure who are on
antihypertensive medications
For overall assessment of the impact of the iEHR on physicians’ clinical decision-making and
quality of care of patients with chronic diseases, qualitative data from focus group sessions and
responses from survey instruments distributed to participating physicians.
Exclusion Criteria:
None.
89
Study Design:
Several different study designs can be used to assess this study question. Infoway may choose to
use all or only a select number of the following, depending on the resources available to them:
• To assess the impact of the iEHR on the quality of care provided to patients with the four
conditions described above, chart reviews would be conducted on patients identified through
ICD-9 codes as having any one of the following conditions: asthma, diabetes, heart failure,
and hypertension. Ambulatory care charts from primary care practices will be reviewed
looking at the performance measures described above for each of the four conditions. A
random sample of charts would be reviewed during the 6-month period prior to iEHR
implementation and then again 12 months after iEHR implementation. The number of charts
selected will depend on resources available and how much it costs for each chart pull, but it
would be desirable to review at least 100 charts for each condition.
• To assess whether the iEHR helps providers make better diagnostic and treatment decisions,
focus groups would also be conducted with primary care physicians in ambulatory care
practices before and after iEHR implementation. Investigators with experience in qualitative
study design would conduct 3-5 sessions with groups of 8-12 physicians and discuss issues
relating to how use of the iEHR has improved their clinical decision-making and the quality
of care they deliver, including anecdotes where use of the iEHR made a difference in how
they managed patients with a chronic diseases. The focus groups would also address issues
relating to physician satisfaction or dissatisfaction with the iEHR, why they do or do not use
the iEHR, and what they would recommend to make the iEHR better.
• A survey would also be sent to physicians where the iEHR has been implemented. The
number of physicians selected to participate in the survey will be determined later based on
the resources available. Some possibilities for physician selection include: all physicians in a
province where the iEHR was implemented; physicians from a select group of communities
(e.g., 2-6 communities) where the iEHR was implemented, etc. The goal would be to survey
a representative sample of physicians.
Study Assumptions:
• Use of the iEHR would improve clinical decision-making and chronic disease management
• Physicians taking care of patients whose charts are pulled for the study have access to the
iEHR and are actually using it
• Other interventions are not being undertaken during the time of this study that will affect
these particular performance measures and result in improved chronic disease management
• Physicians will fill out and return the surveys.
• Infoway will provide physicians with an incentive to fill out and return the surveys
• An appropriate survey instrument will be developed by personnel with expertise in
informatics and survey design.
90

• Chart pulls for chart reviews
• Investigators with qualitative study design expertise to collect and analyze focus group data
• Survey instrument.
Indicator Analysis:
• Performance measures for each of the four conditions as described above under “Measures”
• Qualitative data from focus group discussions assessing the following: how use of the iEHR
has improved their clinical decision-making; how use of the iEHR has improved their
management of patients with asthma, diabetes, heart failure, and hypertension; how use of
the iEHR has improved their management of patients with other chronic conditions; how use
of the iEHR has improved the overall quality of care they deliver; physician satisfaction or
dissatisfaction with the iEHR; why they do or do not use the iEHR; and what they would
recommend to make the iEHR better.
• Responses from survey instruments distributed to participating physicians. Survey questions
will center around use of the iEHR, effects on clinical decision-making and quality of care in
patients with chronic diseases, and satisfaction with the iEHR.
Associations:
None.
Comments:
None.
References:
N/A
91
6.4 PROVIDER EFFICIENCY AND EFFECTIVENESS IN EMERGENCY DEPARTMENTS
Productivity and efficiency of ED staff is a measure that takes into account how much time
physicians and nurses spend tracking down prior medication, laboratory, and diagnostic imaging
information when a patient is seen in the ED and how this affects workflow and the amount of
time spent on the care of a patient in the ED.
Study Question:
Does the iEHR improve the productivity and efficiency of the doctors and nurses providing care
for patients in the emergency department (ED)?

When patients are seen in the ED, physicians and nurses often need to access outside medical
records in order to provide appropriate care. If this information is not available, the ED staff has
to spend time trying to track down and contact the patient’s providers and facilities where they
have received care in order to obtain this information. This is usually accomplished via telephone
and fax. A lot of time is wasted by virtue of the fact that ED personnel have to spend time trying
to contact these providers and facilities to obtain copies of this information. ED staff may also
not always be able to reach the appropriate persons or facilities that have this information,
resulting in repeated calls and other attempts to make contact, which results in even more
inefficiency. It also delays care, which, in more serious cases, may adversely affect patient safety
by delaying treatment. In addition, because medical records are usually not available to the ED
and they are often unable to obtain this information from outside sources in a timely manner (if
at all), laboratory and diagnostic imaging tests that have already been completed are often
repeated, resulting in redundant and unnecessary testing that is only done because the results of
the previous tests are not readily available. This results in additional monetary costs to the health
care system. It can also result in additional burdens on the patient, who has to undergo additional
testing, and on the staff because resources need to be made available to perform all these tests
(e.g., need for phlebotomists or nurses to draw blood, laboratory technicians to run the lab tests,
radiology technicians to do the x-rays, radiologists to read the x-rays, etc.). An iEHR, which
would allow ED staff to access lab, medication, and diagnostic imaging information that was
performed by outside providers and facilities prior to the ED visit, would improve the
productivity and efficiency of the ED.
92
• Emergency departments that have implemented the iEHR

• Patients being seen in the ED during the specified study period.
Measures:
• Waiting time of patients seen in the ED from the time they arrive in the ED to the time they
are discharged from the ED. Discharge can mean the patient was discharged home, the
patient was admitted to an acute inpatient unit, etc. Discharge time would be measured as the
time when the patient is physically discharged from the care of the ED.
• Time spent by ED physicians, nurses, and other staff tracking down and obtaining lab results,
diagnostic imaging results, and medication information.
• Number of lab and diagnostic imaging tests ordered by the ED
• Costs of lab and diagnostic imaging tests ordered by the ED
Exclusion Criteria:
None.
Study Design:
Several different study designs can be used to assess this study question. Infoway may choose to
use all or only a select number of the following, depending on the resources available to them:
• A sample of ED departments that have implemented the iEHR will be studied. The number
of EDs studied will depend on the resources available to Infoway. Selection of more than one
ED is desirable, and, if possible, EDs selected should exhibit some variability in
characteristics, such as size, how many patients are seen in a given year, case mix of patients
seen, severity level of patients seen, community and population served, academic vs non-
academic setting, urban vs rural, level of trauma service, etc. This would allow a better
assessment of the impact of the iEHR on the different types of EDs in Canada and would also
improve the generalizability of the findings. We recommend that Infoway include at least two
EDs in the study but no more than ten, striving for a study that involves 4-6 different EDs if
possible. Data would be collected from these EDs during the 12 months pre- and post- iEHR
implementation and compared.
• In order to assess the time patients spend in the ED, a sample of patients would be given
timers and/or other types of recording devices (e.g., paper-based recording forms or audio
recording devices like microcassette recorders) on their arrival and check-in to the ED and
asked to track their time and specific aspects of their care while in the ED.
• In order to assess the time providers spend trying to track down lab, imaging and medication
information, time-motion studies with direct observation of physicians, nurses, and other ED
93
staff would occur. Investigators would be assigned to follow specific providers throughout
the ED during their shifts, and they would document what the providers were doing and how
much time was spent on different activities, focusing on time spent trying to track down lab,
imaging and medication information.
• Focus groups would also be conducted with ED personnel before and after iEHR
implementation. Investigators with experience in qualitative study design would interview
ED personnel in focus group sessions and discuss issues relating to perceived time and effort
spent trying to track down lab, imaging and medication information in the ED prior to and
after iEHR implementation; perceived effects of the iEHR on time and resource utilization;
perceived effects of the iEHR on safety and quality of care; effects of the iEHR on provider
satisfaction; overall assessment of the iEHR and its overall impact on the ED; why providers
in the ED do or do not use the iEHR; and what ED staff would do to make the iEHR better.
• In order to assess the impact of the iEHR on the ordering of lab and imaging tests in the ED,
the number and types of lab and imaging tests ordered in the 12 months pre- and post- iEHR
implementation will be measured, as well as the costs of lab and imaging tests ordered in the
ED during this same time period.
Study Assumptions:
• Use of the iEHR will improve productivity and efficiency of ED providers.
• The iEHR will be implemented in the EDs being studied.
• Lab, imaging, and medication information will be captured in an electronic format from
hospitals, clinics and other clinical facilities in the same geographic location as the ED and
be available to the ED via the iEHR.
• Data is available on the number and types of lab and imaging tests that are ordered in the ED.
• Cost data is available for lab and imaging tests that are ordered in the ED.
Data sources and instruments:

• ED Data on the number and types of lab and imaging tests that are ordered
• Recording devices for tracking the time patients spend in the ED
• Investigators with qualitative study design expertise to collect and analyze focus group data
• Direct observers to collect provider data for time motion studies
Indicator Analysis:
• Total time (in minutes) that patients selected for the study spent in the ED from the time of
arrival and check in to the time they are formally discharged from the ED.
• Total time (in minutes) that providers spend tracking down outside lab results, imaging
results, and medications.
94
• Qualitative data from focus group discussions assessing the following: perceived time and
effort spent trying to track down lab, imaging and medication information in the ED prior to
and after iEHR implementation; perceived effects of the iEHR on time and resource
utilization; perceived effects of the iEHR on safety and quality of care; effects of the iEHR
on provider satisfaction; overall assessment of the iEHR and its overall impact on the ED;
why providers in the ED do or do not use the iEHR; and what ED staff would do to make the
iEHR better.
• Total number of lab tests ordered (per month) in the ED during the 12 months pre- and post-
implementation of the iEHR. Numbers would be tallied on a monthly basis and also added up
to reflect cumulative results up to the most recent month for which data exists.
• Total number of imaging tests ordered (per month) in the ED during the 12 months pre- and
post- implementation of the iEHR. Numbers would be tallied on a monthly basis and also
added up to reflect cumulative results up to the most recent month for which data exists.
• Total costs of lab tests ordered (per month) in the ED during the 12 months pre- and post-
implementation of the iEHR. Numbers would be tallied on a monthly basis and also added up
to reflect cumulative results up to the most recent month for which data exists.
• Total costs of imaging tests ordered (per month) in the ED during the 12 months pre- and
post- implementation of the iEHR. Numbers would be tallied on a monthly basis and also
added up to reflect cumulative results up to the most recent month for which data exists.
Associations:
None.
Comments:
None.
References:
N/A
95
7 TELEHEALTH PROGRAM
7.1 TELEHEALTH PROGRAM INDICATORS SUMMARY
This section presents key Telehealth program study questions and associated indicators that have
been identified. Indicators in Appendix A are proffered to provide broader perspective of the
issues that could be addressed in more focused assessments of telehealth initiatives. This aligns
with the realization that additional indicators will be used within each telehealth initiative to
evaluate other outcomes addressing more specific and focused needs of other stakeholders.
Refer to Appendix B for descriptions of consistent terminology used within this section.

category
Quality
Change in health Does telehealth improve Efficiency of a) Time to return to functional status
outcomes client recovery or recovery with (reduced)
health? telehomecare b) Time to return to work (reduced)
c) ‘Client’ readmissions to hospital
(reduced)
d) ‘Client’ visits to family
physicians (reduced)
e) ‘User’ time and travel costs
(reduced)
f) Generic physical and mental
health well-being survey /
questionnaire (improved)
g) Symptom or disease specific
survey / questionnaire (improved)
Productivity
Change in care co- Do care providers share Provider sharing a) Knowledge, action, and attitudes
ordination information with other of patient of care providers to inter-
healthcare professionals information with disciplinary practice
to take shared decisions telemedicine b) Maturity of care provider inter-
about client needs? personal relations
c) Academic detailing
Change in care co- Does telehealth improve Change in work a) Efficiency pre and post telehealth
ordination current working practices with intervention
practices? telehomecare b) Audit
c) Academic detailing
Access
Change in access to Does Telehealth Access to Clinical Use:
services improve access to previously a) Patient wait time for initial
services for ‘users’? unavailable appointment (reduced)
96

category
services with b) Patient wait time for specialist
telemedicine referral (reduced)
c) Patient wait time for diagnosis
and treatment (reduced)
d) Patient access to clinical or
laboratory test results (increased)
e) Clinician access to clinical or
laboratory test results (increased)
f) Number of clinical ‘telehealth
sites’ within a health region, health
authority, or LHIN per year -
include the ‘forward sortation area’
element of the postal code
(increased)
g) Number of clinical ‘telehealth
services’ 1 within a health region,
health authority, or LHIN per year
– include the ‘forward sortation
area’ element of the postal code
(increased)
h) Number of clinical ‘telehealth
programs’ within a health region,
health authority, or LHIN per year
– include the ‘forward sortation
(increased)
References:
Andrews G, Peters L, Teesson M. Measurement of Consumer Outcome. Government of
Australia Publication. http://www.health.gov.au/internet/wcms/Publishing.nsf/Content/mental-
pubs/$FILE/measurecomh.pdf. (Last accessed May 2006)
Bloom BS. (2005) Effects of continuing medical education on improving physician clinical care
and patient health: a review of systematic reviews. Int J Technol Assess Health Care. 21(3):380-
5.
CADTH. Guidelines for the economic evaluation of health technologies: Canada [3rd Edition].
Ottawa: Canadian Agency for Drugs and Technologies in Health; 2006.
Cole KD, Waite MS, Nichols LO. (2003) Organizational structure, team process, and future
directions of interprofessional health care teams. Gerontol Geriatr Edu. 24(2):35-49
empirica. Review of state-of-the-art in eHealth economic evaluation - focusing on method and

measurement tools to be developed within the project context. ‘e-Health IMPACT’ Project -
97
Study on Economic Impact of e-Health: Developing an evidence-based context-adaptive method

of evaluation for e-health. Confidential Draft Document, December 2005.
healthimpact@empirica.com. (Last accessed April 2006).
Fahy PJ. (2005) Online and Face-to-Face Group Interaction Processes Compared Using Bales'
Interaction Process Analysis (IPA). European Journal of Open, Distance, and e-Learning.
Available at http://www.eurodl.org/materials/contrib/2006/Patrick_J_Fahy.htm (Last accessed
April 2006).
Garber AM, Phelps CE. (1997) Economic foundations of cost-effectiveness analysis. J Health
Econ 16(1):1-31.
McIntosh E, Cairns J. (1997) A Framework for Economic Evaluation of Telemedicine. Journal

Telemed Telecare. 3(3):132-139.
Scott RE, McCarthy GF, Jennett PA, Perverseff T, Palacios M, Rush B. National Telehealth
Outcome Indicators Project [NTOIP] - Project Information Document and a Synthesis of
Telehealth Outcomes Literature. Health Telematics Unit, University of Calgary. March 2003.
SF-6D - http://www.shef.ac.uk/scharr/sections/heds/mvh/sf-6d (last accessed May 2006)
SF-8 Health Survey. http://www.sf-36.org/tools/sf8.shtml (last accessed May 2006)
Sloan JA, Aaronson N, Cappelleri JC, Fairclough DL, Varricchio C, and the Clinical
Significance Consensus Meeting Group. (2002) Assessing the Clinical Significance of Single
Items Relative to Summated Scores. Mayo Clin Proc. 77:479-487.
Sohn W, Ismail AL, Telles M. (2004) Efficacy of educational interventions targeting primary
care providers' practice behaviors: an overview of published systematic reviews. J Public Health
Dent. 64(3):164-72.
Hadjistavropoulos HD, Sagan M, Bierlein C, Lawson K. (2003) Development of a Case

Management Quality Questionnaire. Care Management Journal. 4(1):8-17.
Wolters R, Wensing M, Klomp M, Lagro-Jansen T, Weel C, Grol R. (2005) Effects of distance

learning on clinical management of LUTS in primary care: a randomised trial. Patient Educ.
Couns. 59(2):212-8. (Epub 2004 Dec 30)
Whittaker SL, Adkins S, Phillips R, Jones J, Horsley MA, Kelley G. (2004) Success factors in
the long-term sustainability of a telediabetes programme. J Telemed Telecare. 10(2):84-88.
98
Yeager S. (2005) Interdisciplinary Collaboration: The Heart and Soul of Health Care. Critical
Care Nursing Clinics of North America. 17(2):143-148
99
7.2 EFFICIENCY OF RECOVERY WITH TELEHOMECARE
The ‘Impact on Health’ indicator will measure change in physical and mental well-being of
individuals over time.
Study Question:
Does telehealth improve client recovery or health?

The ultimate desirable outcome of any telehealth intervention is a positive impact on the health
or recovery of ‘clients’. In order to establish a ‘net benefit’ for any telehealth intervention, it is
essential that an impact on health be demonstrated.

• Evaluation of any clinical telehealth solution
• Telehealth Net Benefit assessment.
Measures:
The measure identified below refers only to clinical telehealth solutions, and only to those
recommended for use. The complete list of Clinical, Administrative, Research, and Educational
indicators can be found in Appendix A.
1) Generic physical and mental health well-being survey / questionnaire (improved)
Exclusion Criteria:
None identified.
Study Design:
See Appendix B, Section 9.4.4 for comments relating to Study Design.
Study Assumptions:
It will be possible to administer the ‘quality of life’ survey to all client participants at least 2
times approximately 4 weeks apart.
100

It is recommended that the SF-12v2 (SF-12 version 2) Health Survey be used (see ‘Comments’
below). The SF-12v2™ Health Survey is a 12-item subset of the SF-36v2™ that measures eight
domains of health, and generates eight dimension scores and two summary scores for physical
and mental health. It is a brief (2-3 minutes), reliable measure of overall health status.
Indicator Analysis:
Software and guidelines are available to support analysis. The SF-12v2 is available in a standard
form that uses a four-week recall period, and an acute form that uses a one-week recall period.
Associations:
The SF-12v2 data can be used together with another tool, the SF-6D (see Appendix B), for
economic evaluation of telehealth initiatives.
Comments:
There was discussion about the use of a Canadian survey tool, however it is recommended that
the SF-12v2 tool be used as the measure for the indicator Impact on Health. This is for two
reasons: the SF-12v2 tool is simpler to administer; and it can be used together with another tool
(the SF-6D) to provide a means of economic evaluation for telehealth initiatives (the SF-6D,
provides a means for using the SF-12 in economic evaluation by estimating a preference-based
single index measure for health from these data using general population values).
QualityMetric SF™ Generic Health Surveys capture reliable and scientifically valid patient-
reported health outcomes information and have been adapted for use in more than 60 languages.
Used successfully in more than 600 randomized clinical trials reported in over 240 scientific and
medical journals, these surveys are proven responsive in 44 disease conditions and are accepted
by the Food and Drug Administration as proof of benefit for improved functioning and other
patient-reported outcomes (PROs).
Version 2 of the SF-12 Health Surveys has been shown to produce substantial improvements.
The SF-12v2 Health Survey was developed by QualityMetric to improve on the SF-12® in
scoring, item construction, and survey layout. Adapted for use in more than 36 countries, it
yields physical and mental component summary measure scores that are comparable to those
from the SF-36v2 in both general and disease-specific populations. The tool can be self-
administered or given by a trained interviewer in person or by telephone to persons ages 14 and
older and usually takes two to three minutes to complete. Most of the items in the survey ask
respondents to consider a specific period of time, or recall period, when responding which can be
as short as one week. The SF-12v2 has been adopted as the standard of measurement by key
government agencies in the US, including the Agency for Healthcare Research and Quality
(AHRQ), which includes the SF-12v2 in the National Medical Expenditure Panel Survey.
101
References:
See Section 7.1.
102
7.3 ACCESS TO PREVIOUSLY UNAVAILABLE SERVICES WITH TELEMEDICINE
The ‘Access to Services’ indicator assesses whether discrete telehealth sites, services, or
programs are being made available to users to access clinical services. These services will
typically have been previously available without telehealth intervention, but innovative
applications of telehealth may result in introduction of services previously unavailable in a given
setting. Note that access does not speak to whether these telehealth sites, services, or programs
are being used, only whether they are available for use. Use is reflected in the indicator
‘Utilisation’.
Study Question:
Does telehealth improve access to services for ‘users’?

Access, Change in access to services
Improved access to health and healthcare services is a stated policy goal of Federal, Provincial,
and Territorial governments. If telehealth is to be supported on a long-term basis, it must be
adequately demonstrated that access to health and healthcare services is enhanced by telehealth.
Access is defined as the ‘ability to obtain, make use of, or participate in something’ (Merriam-
Webster's Dictionary of Law, © 1996 Merriam-Webster, Inc.). In the realm of clinical telehealth,
before you can ‘obtain, make use of, or participate’, a telehealth solution must first simply be
available.
A high-level indicator is required that can reflect this availability. One that is sufficiently generic
to be applicable regardless of the type of clinical telehealth initiative being evaluated. A simple
count of the number of clinical telehealth sites available within a defined area each year would
be evidence of access, with a growing number demonstrating growing access. In addition,
evidence of a change in sophistication of the available telehealth initiatives would demonstrate
that access is available for appropriate solutions. Here, a simple count of the number of
‘telehealth services’ and ‘telehealth programs’ (as defined; see Appendix B, Section 9.4.4)
available within a defined area each year would demonstrate this. Again, a growing number
would demonstrate growing access to appropriate solutions.
Perhaps more importantly, if also linked with some form of geographic locator (such as the
‘forward sortation area’ element of the postal code recommended here), then a clear and
103
quantitative picture of the distribution and growth in telehealth sites, telehealth services, and
telehealth programs could be developed, and shown to change with investment by Infoway.

Measures:
The measures identified below refer only to clinical telehealth solutions, and only to those
indicators can be found above.
1. Number of clinical ‘telehealth sites’ within a health region, health authority, or LHIN per
year - include the ‘forward sortation area’ element of the postal code (increased)
2. Number of clinical ‘telehealth services’ within a health region, health authority, or LHIN per
year – include the ‘forward sortation area’ element of the postal code (increased)
3. Number of clinical ‘telehealth programs’ within a health region, health authority, or LHIN
per year – include the ‘forward sortation area’ element of the postal code (increased)
Exclusion Criteria:
None identified.
Study Design:
Study Assumptions:
a) Scheduling systems are available and have the appropriate capability.
b) Utilisation logs can be simply and effectively implemented, and can accumulate accurate
data
c) Delivery versus receipt of a telehealth session can be simply differentiated (see Appendix B,
Section 9.4.4)
d) Capture of the ‘forward sortation area’ of the postal code can be achieved simply (e.g.
through minor modification of scheduling systems or utilisation logs).

Query tools of administrative databases and scheduling databases, or manual logs.
Indicator Analysis:
i) For each desired health region, health authority, or LHIN, document the number of
‘telehealth sites’ that are present each year as of 31 December; calculate the percentage
104
change (increase) in number of ‘telehealth sites’ year over year; map the number and / or
percentage change to visually demonstrate increasing access in these areas.
ii) For specific geographic regions (e.g. North of 60; rural areas), communities (e.g. First
Nations), or localized minority groups, map out the absolute number and / or percentage
change within those specific locales to visually demonstrate increasing access in these
specific areas.
iii) Repeat i) and ii) for ‘telehealth services’ data.
iv) Repeat i) and ii) for ‘telehealth programs’ data
Associations:
Co-analysis and co-presentation of data from the indicator ‘access to services’ with that from the
indicator ‘utilisation’ will provide a clear picture of growth in availability and utility of telehealth
services across the country and in specific regions of Canada.
Comments:
The desire to increase access to healthcare services remains a longstanding and oft stated goal of
efforts to modify healthcare systems.
Linking access data to geographic data provides Infoway with a valuable means to demonstrate
and illustrate impact and net benefit in a profound and highly visual way.
It may be prudent for Infoway’s Telehealth Program to either develop in-house expertise, or
develop relationships with relevant experts, to gain experience and practical application of
geographic data (including the more sophisticated capabilities of GIS analysis).
References:
See Section 7.1.
105
8 SYSTEM AND USE INDICATORS
Below is a set of additional key program specific indicators that fall into the System and Use
dimensions. The purpose of these indicators is to identify key issues in projects which might be
barriers to the realization of the desired net impact.
The purpose of these indicators is to identify key issues in projects which might be barriers to the
realization of the desired net impact indicators. The list of system and use indicators does not
need to be comprehensive but should align to the impacts recommended for evaluation.
8.1 DIAGNOSTIC IMAGING PROGRAM
Category / Indicator Measures Study Design Data Sources /

Sub- Instruments
Category
User Satisfaction User Descriptive post Post PACS Survey to
Satisfaction with PACS Friendliness/Usability PACS; repeated Radiologists,
functionality Productivity measures Technologist, and
Referring physicians
Quantitative
Interviews/Focus groups
Qualitative
System Performance of Unscheduled Descriptive, post Post PACS Survey to
Quality PACS downtime PACS; repeated Radiologists,
functionality Speed measures Technologist, and
Image Quality Referring Physicians
Image Availability Quantitative
Accessibility Interviews/focus groups
Problem Solving Qualitative
Support
Change Change in workflow Time Motion Study Workflow Collection
Management for technologists and Sheet
Success radiologists
Interviews/focus groups
PACS System Use Adoption Post PACS repeated Total Unique Clinician
Adoption measures at 6, 12 and Accounts
24 months post
implementation Total Remote (VPN)
Clinician Accounts
Quantitative
Number of CD burns
requested to the IT
Department
106
8.2 DRUG INFORMATION SYSTEM PROGRAM
Category / Study Question Indicator Measures Study Design Data Sources /

Sub-Category Instruments
Technical What is the System response 1.Time to login Response time DIS, survey,
Performance technical time for tasks 2.Time to search patient on-site survey timing sheet
performance 3.Time to search drug
time of the 4.Time to complete
system (re: its prescription (including a
basic functions default posology and
and impact of therapeutic indication
security 5.Time to generate alerts
features)? 6. Time to save and print
prescription
System What is the System response In addition to the prior Response time DIS, survey,
Performance reliability of the time for tasks (as list, on-site survey timing sheet
system and how above) 1.Number of
variable is it in functionalities that Technical audit Technical audit
high and low System require communication trail analysis trails
volume periods, success/failure with a central server or
and by for connection system outside the DIS
geographic and downloading itself, or require the
access? loading of a new
webpage
2.Proportion of attempts
to connect to central
server/outside data
source/load a webpage
that result in failure for
each functionality that
requires such a
connection
3. Time to
download/connect to
central server or outside
data source or to load a
webpage
System What is the Survey of Number of Survey Survey
Performance functionality functionalities functionalities on system distributed to
included in the that correspond to the DIS developers
system list of functionalities or DIS
identified in expert companies
consensus processes (see
attached list)
System How effective Standardized task 1. Time to complete Timed tasks DIS, reporting
Functionality was the training + user standardized tasks sheets
on the system satisfaction immediately following
survey, overall training and two weeks
107

and by past after training
computer 2. User ratings of
experience usefulness of
functionalities Rating survey Survey
Service + Ease What was the TAM+ user 1. Behavioral intention Survey of users Modified
of Use perceived value satisfaction 2. Perceived ease of use after training Technology
of the system survey 3. Perceived usefulness Acceptance
and its 4. Social influence on Model
functions, the use Questionnaire
ease of use and 5. Relative advantage
intention to use 6. Computer self-
efficacy
7. Clinical computer
fluency
8. Clinical satisfaction
Use +Intent to What was the Audit trials + 1. Proportion of patients Analysis of Audit trails for
use actual use of the denominator = seen in practice who audit trails all
system and its visits scheduled, consent to use of DIS functionalities,
functions dispensed Rx or 2. Proportion of patients health system
billings seen for whom physician billing records
used DIS for all patients
3. Proportion of patients seen (for
seen for whom physician physicians billing
prescribed using DIS, by the act), or
out of all patients for practice records
whom physician wrote a for salaried
prescription physicians
4. Physician use of
Problem list
5. Physician use of Drug
Profiler (or drug list)
6. Physician use of Drug
Details function
7. Physician use of Re-
prescribing function
8. Physician use of
monographs
9. Physician use of stop
function
10. Physician use of
dose change function
11. Physician
editing/correction of E-
Rx before completing
prescribing process
Use-actual use What was the Technology 1. Behavioral intention Survey of users Modified
108

user satisfaction Acceptance re: continued use 3 months after Technology
with the system Model (TAM)+ 2. Perceived ease of use implementation Acceptance
and its user satisfaction 3. Perceived usefulness; Model
functionalities survey overall and of specific Questionnaire
functions
Satisfaction- What was user’s Audit trails + 1. Percent of alerts that Analysis of Audit trails for
user response to reasons (desirable are over-ridden, overall, audit trails all aspects of
alerts for drug not essential) and broken down by alert
safety reason for over-ride functionalities
problems? 2. Percent of alerts that
result in a change of
prescription
3. Most common alerts
and the drugs, allergies,
or diagnoses that are
involved
4. Most common type of
alerts
5. Physician use of
changes of settings to
suppress alerts (if
available)
Use (no
category yet) or
in quality
109
8.2.1 System Response Time for Tasks
The System Response Time for Tasks indicator determines the speed at which core tasks may
be completed within a DIS in a way that can be compared across systems.
Study Question:
What is the technical performance time of the system (re: its basic functions and impact of
security features)?

System: Performance
Speed of processing in electronic prescribing has been cited as a barrier to use (Bates et al 2003,
Lipton et al 2003, Tamblyn et al 2005). The time taken to execute core functions such as signing
into the system, accessing a patient, and completing a prescription may help to determine the
acceptability of the system overall.

• Physician practices with DIS implemented
• Hospital departments with DIS implemented.
Measures:
• Time to login
• Time to execute patient search based on minimum required characters
• Time to complete search for selected drugs
• Time to complete prescription, including default directives and therapeutic intent
• Time to generate alerts associated with prescription
• Time to save and print completed prescription.
Exclusion Criteria:
• Where functionality does not exist on DIS system, omit test for that function
Study Design:
• Site survey using physicians’ DIS in actual use setting
Study Assumptions:
• Physicians will consent to use of their system and patient database
110
• There may be differences geographically, by patient database size, connection type, and
patient drug list size
• Functions being timed exist on DIS system.

• Response time survey results
• DIS system logs to determine patient list size and drug list size
• Physician reports of connection type.
Indicator Analysis:
• Login time: ΔTime from entering the login number/password/identity to time at opening
screen for system appearing
• Patient search time: ΔTime from completion of entering minimum characters for patient
search to appearance of patient list containing the characters chosen
• Drug search time: ΔTime from entering characters for drug search to appearance of drug list
• Time to complete prescription: ΔTime from entering posology to posology appearing on
prescription, ΔTime from entering therapeutic intent on prescription to therapeutic intent
appearing on prescription
• Time to generate alerts: ΔTime from activating alert function to time alerts appear
• Time to save and print prescription: ΔTime from activating save and print function to
completion of save and prescription printed (may be separated if the two functions do not
take place simultaneously).
Associations:
• This indicator relates to Use, Ease of Use, and Satisfaction
• This survey may also be used to answer the question ‘What is the reliability of the system
and how variable is it in high and low volume periods, and by geographic access?’
Comments:
Survey format may need to be edited for inpatient testing or for DIS with workflow incompatible
with the flow used.
References:
Bates DW, Kuperman GJ, Wang S, Gandhi T, Kittler A, Volk L et al. (2003) Ten commandments
for effective clinical decision support: making the practice of evidence-based medicine a reality.
J Am Med Inform Assoc 10(6):523-530.
Lipton HL, Miller R.H., Wimbush J.J. (2003) Electronic Prescribing: Ready for Prime Time?
Journal of Healthcare Information Management 17[4], 72-79.
111
Tamblyn R, Huang A, Kawasumi Y, Bartlett G, Grad R, Jacques A, Dawes M, Abrahamowicz

M, Perreault R, Taylor L, Winslad N, Poissant L, Pinsonneault A. (2006) The development and
evaluation of an integrated electronic prescribing and drug management system for primary
care. J Am Med Inform Assoc 13(2):148-159.
112
8.2.2 System Response/Failure for Connection and Downloading
The System Response/Failure for Connection and Downloading indicator measures the
connection speed and success for functions which require processing time, connection to a
central server, loading of new webpages, or other connections to sources outside the DIS.
Study Question:
What is the reliability of the system and how variable is it in high and low volume periods, and
by geographic access?

System: Performance
Speed of processing in electronic prescribing has been cited as a barrier to use (Bates et al 2003,
Lipton et al 2003, Tamblyn et al 2005). The time taken to execute core functions such as signing
into the system, accessing a patient, and completing a prescription may help to determine the
acceptability of the system overall. For any functionality that requires downloading, loading of
multiple webpages, or connection to a central server, if that connection is not fast and reliable,
physician acceptance may be compromised.

• Physician practices with DIS implemented
• Hospital departments with DIS implemented
• Technical audit trails for all users
Measures:
• Time to login
• Time to execute patient search based on minimum required characters
• Time to complete search for selected drugs
• Time to complete prescription, including default directives and therapeutic intent
• Time to generate alerts associated with prescription
• Time to save and print completed prescription
• Number of functionalities that require communication with a central server or system outside
the DIS itself, or require the loading of a new webpage
• Proportion of attempts to connect to central server/outside data source/load a webpage that
result in failure for each functionality that requires such a connection/download/new
webpage
113
• Time to download/connect to central server or outside data source or to load a webpage for
each functionality that requires such a connection/download/new webpage.
Exclusion Criteria:
• DIS systems directly connected to a central server (such as in hospital systems) for measures
that assume the need for a communication connection outside the system.
Study Design:
• Site survey using physicians’ DIS in actual use setting
• Analysis of technical audit trails.
Study Assumptions:
• Physicians will consent to use of their system and patient database
• There may be differences geographically, by patient database size, connection type, and
patient drug list size
• Functions being timed exist on DIS system
• Audit trails of connections successes and failures and timings for downloading, page loading,
or connecting exist.

• Response time survey results
• DIS system logs to determine patient list size and drug list size
Indicator Analysis:
• Login time: ΔTime from entering the login number/password/identity to time at opening
screen for system appearing
• Patient search time: ΔTime from completion of entering minimum characters for patient
search to appearance of patient list containing the characters chosen
• Drug search time: ΔTime from entering characters for drug search to appearance of drug list
• Time to complete prescription: ΔTime from entering posology to posology appearing on
prescription, ΔTime from entering therapeutic intent on prescription to therapeutic intent
appearing on prescription
• Time to generate alerts: ΔTime from activating alert function to time alerts appear
• Time to save and print prescription: ΔTime from activating save and print function to
completion of save and prescription printed (may be separated if the two functions do not
take place simultaneously)
• Number of functionalities that require outside connection/download/webpage loading: Count
of each functionality with these requirements
114
• Proportion of failed attempts to connect/download/load page: Count of failed attempts to

connect/download/load page / total number of attempts to connect/download/load page, by
functionality
• Time to connect/download/load page: ΔTime from accessing functionality that requires
connection/download/page load to connection/download/page load complete.
Associations:
• This indicator relates to Use, Ease of Use, and Satisfaction
• This survey may also be used to answer the question ‘What is the reliability of the system
and how variable is it in high and low volume periods, and by geographic access?’
Comments:
Survey format may need to be edited for inpatient testing or for DIS with workflow incompatible
with the flow used.
References:
Bates DW, Kuperman GJ, Wang S, Gandhi T, Kittler A, Volk L et al. (2003) Ten commandments
for effective clinical decision support: making the practice of evidence-based medicine a reality.
J Am Med Inform Assoc 10(6):523-530.
Lipton HL, Miller R.H., Wimbush J.J. (2003) Electronic Prescribing: Ready for Prime Time?
Journal of Healthcare Information Management 17[4], 72-79.

M, Perreault R, Taylor L, Winslade N, Poissant L, Pinsonneault A. (2006) The development and
115
8.2.3 Functionalities
The Functionality indicator determines the number of functionalities available on a DIS, and the
proportion of these functionalities that correspond to those identified by expert panels as the
most important to include for patient safety, cost management, and clinician acceptance.
Study Question:
What is the functionality included in the system?

System: Functionality
Though there are many existing drug information systems, there has been limited analysis to
determine the key functionalities required to include in the system for optimal patient safety and
health outcomes, cost management, clinician acceptance, and patient privacy. Two groups have
attempted to define the most important functionalities through expert consensus processes: one
group headed by Douglas Bell in California and one group headed by Anthony Avery in the UK.
These lists can serve as a basis for determining whether a DIS has sufficient functionality to
ensure patient safety and privacy, clinician acceptability, and ability to help manage costs.

• Surveys may be distributed to commercial companies and research groups who have
developed or are developing DIS.
Measures:
• Proportion of key functionalities defined by Bell and Avery that exist on system, by patient
safety, clinician acceptance, and cost management domains
• Number of key functionalities as defined by Bell and Avery that exist on the system
• Number of other functionalities that exist on the system, by patient safety, clinician
acceptance, and cost management domain.
Exclusion Criteria:
All DIS eligible.
Study Design:
Survey with yes/no choice for key functionalities, as well as write in ‘other’ category to allow for
consideration of new functionalities.
116
Study Assumptions:
• Some of the Bell and Avery functionalities are defined in a general way, and may be attained
by very different designs of interface on the system
• Bell and Avery have captured the full range of key functionalities required within the
domains of patient safety and privacy, clinician acceptance, and cost management
• Patient safety and privacy, clinician acceptance, and cost management are the key areas that
should be considered when assessing DIS functionality
• Some of the functionalities identified by Bell were stated to not be possible within the next
three years (from 2004, when the paper was published). As technology is constantly
changing, some of these functionalities may be available at this time and in use.

• Survey results
• DIS research groups or private companies
Indicator Analysis:
• Number and Proportion of the identified key functionalities that exist in the DIS being
considered. There are 74 functionalities in total. Each functionality listed may belong to one
or more subgroups: patient safety (66 functionalities), clinician acceptance (54
functionalities), cost management (21 functionalities), or patient privacy (4 functionalities).
• Number of functionalities on the DIS under consideration outside the group of identified key
functionalities.
Associations:
The measures used for patient privacy in this indicator may also be used in System: Security.
Comments:
Some of the identified key functionalities may not be possible technically at this time. These
functionalities have been marked on the accompanying table. Proportions could be calculated
that exclude these, if the functionalities are still considered beyond the reach of current
technology.
References:
Bell DS, Marken RS, Meili RC, Wang CJ, Rosen M, Brook RH. (2004) Recommendations for
comparing electronic prescribing systems: results of an expert consensus process. Health
Affairs Web Exclusive: 305-317.
Avery JA, Savelyich BSP, Sheikh A, Cantrill J, Morris CJ, Fernando B, Bainbridge M, Horsfield
P, Teasdale S. Identifying and establishing consensus on the most important safety features of
GP computer systems: e-Delphi study
117
8.2.4 Standardized Tasks
The standardized task indicator tests physicians using DIS on a set number of identical actions
that will be undertaken with a typical patient. It also incorporates a written prescription task,
which serves to compare the time for writing a prescription to that of using DIS. The
standardized tasks should be administered three times for each physician, to show initial
response to training, and mastery of the system after short and longer term use in a practice
environment.
Study Question:
How effective was the training on the system?

Use: Competency
Physicians who find the DIS difficult to use, or who receive training that does not adequately
teach them the functionalities of the DIS are less likely to use the system. With good training,
physicians who are initially uncomfortable with computerized systems, and therefore test badly
after initial training, should improve with practice. In a study by Tamblyn et al (2006),
physicians were found to reduce their time to complete a prescription by 50%, after two weeks of
system use post-training. By testing the physician’s mastery of the system directly after training,
then after set periods of use, the ease of use of the system can be assessed.

• Physicians who have been trained on DIS system, immediately following training, two weeks
post-training, and three months post-training
• First test to take place at the training location, second and third tests to take place in the
practice setting.
Measures:
• Time to login to system (warm-up exercise)
• Time to search for patient name (warm-up exercise)
• Time to confirm a diagnosis
• Time to enter an allergy
• Time to write a handwritten prescription for four medications (medications provided by tester
– this test is only used for the initial testing, and can be skipped for tests 2 and 3)
• Time to complete prescription on DIS for the same four medications, including sending the
prescription
• Time to stop a medication and prescribe a substitute (medications provided by tester)
118
• Time to change the dose of a medication provided by tester

• Time to handwrite a renewal for 8 given medications (this test is only used for the initial
testing, and can be skipped for tests 2 and 3)
• Time to enter and complete electronic renewal for same 8 medications.
Exclusion Criteria:
None. All physicians using DIS should be required to do the testing.
Study Design:
Timed response to tasks given.
Study Assumptions:
• Physicians will be trained adequately to perform all the tasks given after training
• Physicians will improve their mastery of the system with use
• The time to handwrite a prescription will not change between test 1 and test 2
• The functionalities being tested exist on the system

Test sheet, tester card with directions, physicians, DIS system. Please note, the sample survey
provided will have to be adapted for each system to show the appropriate icons or menu items.
Indicator Analysis:
• Δ Average time to complete all tasks between test 1 and test 2, test 2 and test 3, and test 1
and test 3
• Average time to complete all tasks at test 1, average time to complete all tasks at test 2,
average time to complete all tasks at test 3
• Time to complete electronic prescriptions at test 1, test 2, and test 3 / Time to complete
handwritten prescription.
Associations:
This indicator may also be used for questions pertaining to system performance and ease of use,
if comparisons are made between different systems. This indicator may also be used for the
service: response dimension and category.
Comments:
The survey included would have to be adapted for each system, as the appropriate workflow
would differ. However, the tasks should remain the same. The procedure for administering the
survey is included.
119
References:
120
8.2.5 User Satisfaction
The User Satisfaction indicator quantifies the perceived usefulness of the key functionalities on
the DIS. The survey is constructed as a Likert scale, asking the physicians to rate their
perception of the usefulness of each of the functionalities.
Study Question:
What was the perceived value of the system and its functions, the ease of use and intention to use
the system?

Satisfaction: User satisfaction
Determining the users’ satisfaction can be used to both compare different DIS systems in terms
of the way physicians perceive the functionalities through the workflow, and to validate the
functionality list against the preferences of actual users.

Physicians who have used the DIS in their practice.
Measures:
Perceived value using a five-point Likert scale from Not beneficial at all to Extremely beneficial
for key functionalities.
Exclusion Criteria:
Physicians who have not used the DIS in practice.
Study Design:
Survey with five-point likert scale.
Study Assumptions:
• Physicians need to use the DIS in their practice to adequately decide the usefulness of the
functionalities
• Perception of usefulness is an estimate for intention to use or actual use
• Functionalities that are used solely for security or other non-patient care functionalities do
not need to be assessed for this indicator.

Survey (see Appendix B), physicians and DIS.
121
Indicator Analysis
• Average score by functionality
• Percentage of respondents who considered the functionality ‘extremely beneficial’
Associations:
The measures for this indicator may also be used for Use: intention to use. The survey may also
be used to answer the questions: ‘What was the perceived value of the system and its functions,
the ease of use and intention to use?’ and ‘What was the user satisfaction with the system and its
functionalities?’
Comments:
The sample survey in Appendix B can be adapted for different systems, but should include the
same functionalities, with agreed upon terms to describe them, to allow comparability between
systems.
References:
122
8.2.6 Technology Acceptance
The Technology Acceptance questionnaire measures physicians’ acceptance of the DIS through
their intention to use the system, perceived ease of use, perceived usefulness, social influence on
use, relative advantage of use, computer self-efficacy, clinical computer fluency, and clinical
satisfaction. The Technology Acceptance indicator uses a modified version of the Technology
Acceptance Model (Kawasumi 2004), a validated measure created to predict user acceptance of
computers (Davis 1989).
Study Question:
What was the perceived value of the system and its functions, the ease of use and intention to use
the system?

Use: Intention to Use
The Technology Acceptance Model was initially developed to assess acceptance of information
technology (Davis 1989). The model was validated, and found to be both reliable and valid as a
predictor of future use of the technology tested. However, for extension to the health care
industry, modifications had to be made. These modifications included adding questions to
determine the perceived clinical relative advantage, clinical computer fluency, and clinical
satisfaction (Kawasumi 2004).

Users of DIS.
Measures:
• Intention to use the system
• Expectations of usefulness and ease of use
• Expectations for impact of DIS on practice in terms of patient care, professional satisfaction,
communication with colleagues, and continuity of care
• Current level of computer use both within practice and without
• Current level of satisfaction in practice
• Physician typology
Exclusion Criteria:
None.
123
Study Design:
Baseline survey with five point likert scale, followed by follow-up survey.
Study Assumptions:
• Intention to use is related to actual use
• Physician typology is related to adopt new technologies

Survey, users of DIS.
Indicator Analysis:
• Average rating for intention to use
• Average rating for expectations of DIS impact on practice
• Average rating for current computer use
• Average rating of professional satisfaction
• Change in rating for impact on practice
• Change in rating for computer usage.
Associations:
This indicator may also be used for Satisfaction: User Satisfaction, and Ease of Use dimensions.
Comments:
None.
References:
Davis FD. (1989) Perceived Usefulness, Perceived Ease of Use, and User Acceptance of
Information Technology. MIS Quarterly 13(3):319-340.
Kawasumi Y, Tamblyn R. (2004) Physician predictors of utilization of an electronic drug

management system in primary care. Medinfo. 2004(CD):1678. PMID: 15360507
124
8.2.7 System Use
The System Use indicator tracks the functionalities the physicians use in their navigation
through the system.
Study Question:
What was the actual use of the system and its functions?

Use: Use behaviour/pattern
The proportional use of all functionalities of a DIS, and physicians’ acceptance of drug
information systems, as measured by the proportion of their patients for whom they use the
system (in outpatient settings) has not been widely studied (Tamblyn). In Schectman’s 2005
study, only 44% of physicians indicated that the system they were using had the capabilities they
were expecting. Lipton (2003) identified the necessity of balancing ease of use and available
functionalities, as physicians expressed a preference for learning the systems incrementally.
However, there may be significant differences between expressed preferences and actual use
which can help identify barriers to adoption of systems. The System Use indicator quantifies
the use of the system, allowing comparisons both between systems and within systems. This
indicator will help to identify the relative value placed on functionalities, as determined by how
often they are accessed, and barriers to use in cases where functionalities are designed in such a
way that they do not match workflow.

Audit trails for all implemented DIS systems.
Measures:
• Proportion of patients seen in practice who consent to use of DIS
• Proportion of patients seen for whom physician used DIS
• Proportion of patients seen for whom physician prescribed using DIS, out of all patients for
whom physician wrote a prescription
• Physician use of Problem list
• Physician use of Drug Profiler (or drug list)
• Physician use of Drug Details function
• Physician use of Re-prescribing function
• Physician use of monographs
• Physician use of stop function
• Physician use of dose change function
125
• Physician editing/correction of E-Rx before completing prescribing process

• Physician over-ride of alert message
• Physician acceptance of alert message
Exclusion Criteria:
Practices where denominators cannot be determined (salaried physicians who do not provide
appointments records).
Study Design:
Database analysis using audit trails and billing records or practice appointment records.
Study Assumptions:
• Functionalities exist on the system
• Audit trails exist for all functionalities
• Functionalities are comparable across systems
• Physicians can provide adequate denominators in terms of patients seen or patients with
prescriptions for the year being analysed
• Actual use of functionalities (or lack of use) can indicate barriers to use

Technical audit trails, billing records from the ministries of health, practice records for
physicians who are salaried.
Indicator Analysis
• Count of distinct patients (identifier) for whom a service has been billed in the practice and
who give consent for use of DIS/Count of all patients for whom a service has been billed in
the practice by year
• Count of patient identifier accessed on DIS (NAM for Quebec)/Total number of patients for
whom physician billed in the same practice by day/week/month (Since the physician may use
the E-Rx system to check diagnoses and other drugs on profile, they do not necessarily have
to write a prescription for this indicator)
• Count of patient identifier with at least one drug prescribed on visit (can use Rx number,
which groups all the drugs in a single prescription)/Total number of patients for whom at
least one drug prescribed through billing records, in same practice, by day/week/month
• Number of confirmed diagnoses entered on the DIS system per patient/Number of patients
for whom the DIS was used
• Number of sessions (patient appointment where DIS used) where drug profiler/drug list
consulted / Number of distinct sessions of DIS use by day/week/month
• Count of times drug details accessed / Count of distinct sessions of DIS use by
day/week/month
126
• Number of sessions where re-prescribing function accessed / number of distinct sessions of

DIS use by day/week/month
• Count of times monographs accessed /number of distinct sessions
• Count of times stop function accessed / count of drug identification numbers electronically
prescribed by week/month OR Count of sessions where stop function accessed / count of
distinct sessions of DIS use
• Count of times dose change function accessed / count of drug identification numbers
electronically prescribed by week/month OR Count of sessions where dose change function
accessed / count of distinct sessions of DIS use
• Count of times edit function is accessed in Rx preview, before completing prescription /
count of drug identification numbers electronically prescribed
• Count of alerts over-ridden by physician / Count of all alerts generated and seen by physician
• Count of all prescription changes resulting from an alert that prevent the same alert from
reappearing / Count of all alerts generated.
Associations:
The System Use indicator can also be used, in addition to other measures, in the Use: self-
reported use, and intention to use, and Satisfaction: User satisfaction, User competency, and ease
of use dimensions.
Comments:
Different systems will have different configurations for their audit trails and different ways of
achieving the same functionalities. However, audit trails that are sufficient to calculate the
indicators should be required. Calculation of indicators that require billing records may not be
possible for salaried physicians, and may differ within a hospital environment.
References:
Lipton HL, Miller RH, Wimbush JJ. (2003) Electronic prescribing: ready for prime time? J.
Healthcare Inf. Management 17(4): 72-79.
Schectman JM, Schorling JB, Nadkarni MM, Voss JD. (2005) Determinants of physician use of
an ambulatory prescription expert system. Int. J. Med. Inf. 74:711-717.

127
8.2.8 Physician Response to Alerts
The Physician Response to Alerts indicator determines the quality of the alerting system
provided by the DIS through how often the physician acts upon the alerts, how often the alerts
are over-ridden, and the reasons the alerts are over-ridden.
Study Question:
What was user’s response to alerts for drug safety problems?

Use: Use behaviour/pattern
Alerts for inappropriate prescribing are one of the functionalities most often investigated in the
literature, and the functionality that has presented the greatest frustration for physicians (Bates
1999, Feldstein 2004, Hsieh 2004, Krall 2001, Kuperman 1997, Magnus 2002, Payne 2002,
Spina 2005, Tamblyn 2003, Taylor 2004, Weingart 2003, Van der Sijs et al 2006). Alerts have
the potential to reduce adverse drug events by warning physicians of contraindications based on
allergies, diagnoses, and other drugs, among other factors. Alert systems have been hampered by
insufficient sensitivity and specificity in their alerts, leading to a high rate of over-riding the
alert, and loss of the potential benefit to patients. Assessing the response to alerts will determine
the value placed on the alerting system, and can lead into an assessment of the adverse events
avoided by use of the alerting system.

Users of DIS that include alert functionalities in any setting.
Measures:
• Percent of alerts that are over-ridden, overall, and broken down by reason for over-ride
• Percent of alerts that result in a change of prescription
• Most common alerts and the drugs, allergies, or diagnoses that are involved
• Most common type of alerts
• Physician use of changes of settings to suppress alerts (if available).
Exclusion Criteria:
• DIS with no alert over-ride function
Study Design:
Audit trail analysis of physician response to alerts
128
Study Assumptions:
• Alerts in the system can be over-ridden
• Over-riding alerts requires a reason

Audit trails from DIS system.
Indicator Analysis:
• Number of alerts over-ridden / number of alerts generated and seen
• Number of alerts by type (drug-drug interaction, drug-diagnosis interaction, drug allergy
interaction, inappropriate dose, drug/age interaction, drug/pregnancy interaction, therapeutic
duplication) that are over-ridden / number of alerts by type generated and seen
• Number of alerts resulting in a change of prescription / number of alerts generated and seen
• Number of alerts over-ridden for each over-ride reason / Number of alerts over-ridden
• Number of alerts by drug (chemical entity) (including unseen due to settings changes) / Total
number of alerts (including unseen due to settings changes)
• Number of alerts by diagnosis (including unseen due to settings change) / Total number of
alerts (including unseen due to settings change)
• Number of alerts by type (including unseen due to settings change) / Total number of alerts
(including unseen due to settings change)
• Proportion of all physician users with system at each possible alert level setting / Total
number of physician users
• Proportion of all physician users who change the alert level setting, broken down by up or
down, and how many levels changed / Total number of physicians users.
Associations:
This indicator may also be used for the Information: Content, Satisfaction: User satisfaction,
Ease of Use, and Net Benefits: Quality dimensions.
Comments:
Analysis of the alerts system will depend on the types of alerts available, and functionalities to
allow users to change the level of alerts seen. To compare across systems, reasons for over-
riding alerts may have to be compared to group them into similar groups. To analyse diagnosis
alerts, if ICD codes are used, they should be grouped to represent a disease, rather than broken
down by single codes. Similarly, most of the analysis for drugs should be by chemical entity
rather than DIN. This will prevent an over-estimation of counts.
References:
Bates DW. (1999) Frequency, consequences and prevention of adverse drug events.
J.Qual.Clin.Pract. 19:13-7.
129
Feldstein A et al. How to design computerized alerts to safe prescribing practices. Jt.Comm
J.Qual.Saf. 30:602-13.
Hsieh TC et al. (2004) Characteristics and consequences of drug allergy alert overrides in a
computerized physician order entry system. J.Am.Med.Inform.Assoc. 11:482-91.
Krall MA, Sittig DF. (2001) Subjective assessment of usefulness and appropriate presentation
mode of alerts and reminders in the outpatient setting. Proc.AMIA.Symp. 334-8.:334-8.
Kuperman GJ, Hiltz FL, Teich JM. (1997) Advanced alerting features: displaying new relevant
data and retracting alerts. Proc.AMIA.Annu.Fall.Symp. 243-7.:243-7.
Magnus D, Rodgers S, Avery AJ. (2002) GPs' views on computerized drug interaction alerts:
questionnaire survey. J.Clin.Pharm.Ther. 27:377-82.
Payne TH et al. (2002) Characteristics and override rates of order checks in a practitioner order
entry system. Proc.AMIA.Symp. 602-6.:602-6.
Spina JR et al. (2005) Clinical relevance of automated drug alerts from the perspective of
medical providers. Am.J.Med.Qual. 20:7-14.
Tamblyn R et al. (2003) The medical office of the 21st century (MOXXI): effectiveness of
computerized decision-making support in reducing inappropriate prescribing in primary care.
CMAJ. 169:549-56.
Taylor LK, Tamblyn R. (2004) Reasons for physician non-adherence to electronic drug alerts.
Medinfo. 11:1101-5.
Weingart SN et al. (2003) Physicians' decisions to override computerized drug alerts in primary
care. Arch.Intern.Med. 163:2625-31.
Van der Sijs H, Aarts J, Berg M. (2006) Overriding of drug safety alerts in computerized
physician order entry. J.Am.Med.Inform.Assoc. 13(2): 138-147.
130
8.3 LABORATORY INFORMATION SYSTEM PROGRAM

Category
Access Does the JLIS Deployment Pre-implementation:
implementation result in Total number of labs
increased automation of Number of labs providing lab
lab results? results electronically
Number of paper reports produced
# of calls for verbal results
Post-implementation:
Total number of labs
Number of labs providing lab
results electronically integrated with
JLIS
Number of paper reports produced
# of calls for verbal results
Access, Quality & Does the JLIS Adoption Post-implementation:
Productivity implementation result in Number of participating users by
improved access to lab geographical region & user role
results? (GP, Nurse, Patient, Type of
specialist etc) registered with the
JLIS.
Number of active users by
geographical region & user role
(GP, Nurse, Patient, Type of
specialist etc) that have ordered
tests.
Number of active users by
geographical region & user role
(GP, Nurse, Patient, Type of
specialist etc) that have viewed lab
results either via the JLIS or
provided by the JLIS to their EMR.
Does the JLIS Adoption and Post-implementation:
implementation result in Sharing Results Number of active users by
improved coordination? geographical region & user role
(GP, Nurse, Patient etc) that have
viewed (either via the JLIS or
provided by the JLIS to their EMR)
the same lab results for an
individual patient within a defined
time period.
Does the JLIS Laboratory capacity Pre&Post-implementation:
implementation result in Number of test results by user role,
increased service ordering facility, region & facility
provision? type
131

Category
Access Is the JLIS available for all Availability Post-implementation:
users 24/7? Hours & Minutes of unplanned
downtime
Hours & Minutes of planned
downtime
Number of user complaints/queries
related to access problems
Quality Does the JLIS meet Privacy Post-implementation
privacy requirements? Ability to generate audit log that
details ‘Who has seen my records,
in what circumstances and for what
reason?”
Ability for patients to determine
who can see their records in what
circumstances
Ability for patients to both view &
receive their own results.
Quality Does the JLIS improve Audit Post-Implementation
audit capability? Ability to conduct complex searches
for anonymous data for audit
purposes
Ability to re-link anonymous data to
identifiable data through secure
mechanism in event of public health
need.
132
8.3.1 Clinician Adoption and Sharing Results
Focus Area: System Use/Adoption – System Usage
Objective: Increase system adoption – Increase practitioner usage of the JLIS system.
Study Question:
Does the JLIS implementation result in improved access to lab results?

Productivity, Change in coordination of care
The goal for this indicator is to track the increase in users becoming eligible to order and retrieve
test information through JLIS.

This indicator will include all lab providers, practitioners and patients eligible and authorized to
use the JLIS system.
Measures:
Post-implementation:
• Number of participating users by geographical region & user role (GP, Nurse, Patient etc)
with a unique user account with the JLIS.
• Number of active users by geographical region & user role (GP, Nurse, Patient etc) that have
ordered tests.
viewed lab results either via the JLIS or pulled from the JLIS by their EMR.
viewed lab results either via the JLIS or pulled from the JLIS by their EMR the same lab
results for an individual patient within a defined time period.
Exclusion Criteria:
None.
Study Design:
Reports will be generated monthly from the go-live date for the duration of the implementation
period (L+0 - L + 36 months).
Study Assumptions:
• Baseline will be zero.
133
• This measurement will require the assistance and co-operation of the JLIS design team.
While the User repository has the capacity to measure this indicator, the specific function
that allows the action to be performed must be custom built into the system.
• The adoption threshold will limit the pool of practitioners that will be given valid IDs: At
given points in the Benefits Evaluation process, the pool of potential users (the number of
valid IDs) will limit the potential total number of log-ins over that period of time. It is likely
that the number of valid IDs will never match the total practitioner population. Currently, it
may be assumed that a maximum of 80% of practitioners will have valid user IDs by the end
of the funded period.
• While it is assumed that the majority of user IDs will be created during the deployment
period of JLIS, user IDs will continue to be created as needed after L+36 months.
• Type of specialist will be identifiable.

Enrolment from the User Repository (which needs to be built in) or Smart Systems for Health
(SSHA) RMS (Registration Management System) interfaced with the CPDB (Corporate Provider
Database). This is a distinct count from the Audit Log.
Indicator Analysis:
• Number of participating users stratified by geographical region & user role (GP, Nurse,
Patient etc) with a unique user account with the JLIS.
• Number of active users stratified by geographical region & user role (GP, Nurse, Patient etc)
that have ordered tests.
that have viewed lab results either via the JLIS or pulled from the JLIS by their EMR.
that have viewed lab results either via the JLIS or pulled from the JLIS by their EMR the
same lab results for an individual patient within a defined time period.
• % of users in each category stratified by ordering versus non-ordering provider.
(Gen3+ Time between Order and test result lookup).
Associations:
To be associated with indicator 6.
Comments:
• We very specifically included ‘patients’ within our example of users. Whilst direct patient
access may be a Gen3+ requirement we know that in several jurisdictions this is already
happening. (For example, in BC c.6, 000 INR results are sent directly to patients).
134
• With regards to adoption we intend that an increasing variety of users over the 36 month
period post implementation could be a surrogate measure for assessing ‘increased continuity
of care’ and ‘care coordination’.
References:
N/A
135
8.3.2 Laboratory Capacity
Focus Area: System Use/Adoption – System Usage
Objective: Increase system adoption – Increase usage of, & hence capacity within, the JLIS
system.
Study Question:
Does the JLIS implementation result in increased service provision?

The goal for this indicator is to track the increase in use & capacity of the JLIS system.

This indicator will include all test results pre and post JLIS implementation.
Measures:
• Number of test results by user role, ordering facility, region & facility type
Exclusion Criteria:
None.
Study Design:
Reports will be generated monthly from 12 months prior to “go-live” for the duration of the
implementation (L-12 months to L + 36 months).
Study Assumptions:
• This measurement will require the assistance and co-operation of the JLIS design team.
While the User Repository has the capacity to measure this indicator, the specific function
that allows the action to be performed must be custom built into the system.
• A result is assumed to be each discrete element reported for each test.
• Many tests result in one simple element (e.g. a number or qualifier such as positive or
negative). For other tests, groups of results are reported (e.g. a group of results for
microbiology tests may include the organism (s), antimicrobial susceptibility, and other
attributes).
136
• There are defined retention requirements defined for a Gen2 repository. Results may not be
flushed, or cleared unless there is a process created and implemented for capturing the data
required to measure this indicator.

• Enrolment from the User Repository (which needs to be built in) interfaced with the Provider
Database. This is a distinct count from the Audit Log.
• Audit Logs generating raw data.
• Provincial Billing Data.
Indicator Analysis:
Simple counts of number of test results, per month, stratified by user role, ordering facility,
region & facility type.
Associations:
To be analyzed in conjunction with the Duplicates Indicator. It is being inferred that a reduction
in duplicates, with an increase in number of tests during the same time period, indicates that
capacity for required tests has been increased. It is assumed that demand will exceed capacity.
Comments:
None.
References:
N/A
137
8.4 PUBLIC HEALTH SYSTEM PROGRAM
Category / Sub- Indicator Study Design Measures Data Sources

Category / Instruments
System: Accuracy of up- Comparison of results Accuracy of algorithms in Log files,
Functionality to-date from application of immunization registry for chart review,
algorithms algorithms to registry with determining if people are patient
information obtained from up-to-date with interview.
chart review and other immunization schedules
sources.
Information : Immunization Age-specific cohort Proportion of eligible Registry
Content registry analysis of registry people and vaccinations demographic
completeness patients compared to that are captured in the data, vital
eligible population. registry. statistics data,
Similar analysis census data.
comparing all billings for Provincial
vaccination to records in billing data.
registry. Survey.
Information: Immunization Comparison of Accuracy of Log files,
Content registry information (e.g., number immunization information chart review,
accuracy and timing of contained in the registry. patient
immunizations) within the interview.
registry with information
from other sources.
Information : Automated Cross-sectional or Laboratories that report Survey.
Availability Reporting rate longitudinal survey of notifiable diseases
laboratories. automatically
Use: Use Immunization Cross-sectional survey of Physicians that attempt to Survey,
behavior / registry physicians and / or access records in registry registry log
pattern utilization rate analysis of system log for patients. files.
files
Use: Use Utilization of Survey and participant- Proportion of cases and Survey,
behavior / case and observer analysis to outbreaks managed using observation
pattern outbreak identify when and how the software. notes.
management the software is used in
software. practice.
Use: Intention to Immunization Cross-sectional survey of Providers with access to Survey.
use registry physicians registry in their office.
accessibility
138
8.5 INTEROPERABLE ELECTRONIC HEALTH RECORD PROGRAM

Category
Use behavior / pattern: Do providers use the - Frequency of use of - Count of iEHR viewer
Frequency, duration, iEHR viewer? Are lab, iEHR by providers “hit” rate. How many
location, type/nature, of radiology, and - Type of use of iEHR by times do providers log on
actual usage. medication data actually providers to the system?
Self reported use being used by providers - Repeat “hit” rate for
Productivity: in the management of providers who have used
Care coordination. their patients? How is it it at least once before.
Availability: being used? Does it - Mean frequency of use
Timeliness, reliability, actually affect their by providers.
consistency, accuracy, decision-making? - Direct observation of
completeness of providers
information when and - Provider survey and
where it is needed. focus groups
- Chart review. Is there
indication in the note that
they used the remote
viewer and saw the
patient’s lab, radiology,
or medication data from
another source?
User satisfaction: Why do clinicians use the - Perceived value of the - Focus groups with
Expectations, value, iEHR viewer? For those iEHR to a clinician providers
information / system / who don’t use it, why - Provider surveys
service quality don’t they use it and what
Use Availability: would get them to use it?
Timeliness, reliability, If they used it but
consistency, accuracy, stopped using it, why did
completeness of they stop?
information when and
where it is needed
Functionality: Are lab, radiology, and - Frequency and rapidity - System audit. Is remote
Type of features medication data (and data of being able to access data available at the
available to providers from the shared record if remote lab, medication, point-of-care in
Availability: implemented) available radiology, and shared participating EDs,
Timeliness, reliability, to providers participating record data at the point of outpatient clinics, and
consistency, accuracy, in the program at the care (when and where it inpatient hospital
completeness of point-of-care via the is needed). settings?
information when and iEHR viewer? - Random and periodic
where it is needed functional testing of the
system
- Provider survey
- Provider focus groups
139

Category
- Direct observation
Availability: What is the reliability of - System downtime - Random and periodic
Reliability the system? functional testing of the
system
- Provider survey
- Provider focus groups
User satisfaction: What is the level of - Provider satisfaction - Provider survey
Perceived expectations, provider satisfaction with - Accessibility of data - Provider focus groups
value, information / the iEHR? when and where it is
system / service quality, needed
preference, comfort, - Ease of use
experience
Ease of use:
User friendliness,
learnability
User satisfaction: What is the level of - Patient satisfaction - Patient survey
Perceived expectations, patient satisfaction with - Patient focus groups
value, service quality, the iEHR?
preference, comfort,
experience
140
8.6 TELEHEALTH PROGRAM
Category / Indicator Rationale Measures Data Sources /

Quality: Telehealth When linked with other Number of Implementation will
Appropriateness Setting indicators (e.g. access, ‘telehealth require consistent
/ effectiveness utilisation) the indicator sessions’ provided application across all
‘telehealth setting’ / received per data collection
Access: tracks in sufficient detail month at each mechanisms (e.g.
Ability of the diffusion of ‘telehealth site’ scheduling systems,
Patients / telehealth within and connecting with: utilisation logs). This
Providers to outside the public health 1. Hospital in turn will require
Access Services system. When linked to 2. Community consensus on the
geographic information, Health Centre terminology and
Productivity: it will also allow later 3. Community definitions of each
Efficiency; additional analysis, e.g. Health Facility element.
Care optimal distribution of (e.g. LTC facility;
Coordination; telehealth sites, services, residential care
Net Cost and programs. setting)
4. General practice
5. Specialist
Practice
6. Place of
Employment
7. Home
8. Other
Quality: Clinical Service When linked with other Type of clinical
Appropriateness indicators (e.g. access, service provided
/ effectiveness utilisation) the indicator during a
‘clinical service’ tracks ‘telehealth
Access: in detail the growth and session’:
Ability of diffusion of telehealth 1 alcohol/drugs
2 allergy
Patients / amongst clinical areas. 3 audiology
Providers to 4 burns
Access Services When linked to 5 cardiology
geographic information, 6 dentistry
7 dermatology
Productivity: it will also allow later 8 diabetes
Efficiency; additional analysis, e.g. 9 ear, nose and throat
Care optimal distribution of 10 emergency medicine
Coordination; telehealth sites, services, 11 endocrinology
12 gastroenterology
Net Cost and programs. 13 general medicine
14 genetics
‘Clinical Service’ is 15 geriatric medicine
based upon the clinic 16 gynaecology
17 haematology
classification from 18 immunology
Cleary et al. (see 19 infectious diseases
references). 20 intensive care
141
Category / Indicator Rationale Measures Data Sources /

21 mental health
22 neonatology
23 nephrology
24 neurology
25 neurosurgery
26 nutrition/dietetics
27 obstetrics
28 occupational
therapy
29 oncology
30 ophthalmology
31 optometry
32 orthopaedics
33 orthotics
34 paediatrics
35 palliative care
36 pathology
37 physiotherapy
38 plastic surgery
39 podiatry
40 prosthetics
41 psychiatry
42 psychology
43 radiology
44 rehabilitation
45 respiratory medicine
46 rheumatology
47 social work
48 speech pathology
49 spinal
50 surgery
51 urology
98 other
142
APPENDIX A: ADDITIONAL PROGRAM INDICATORS
A.1 ADDITIONAL LABORATORY INFORMATION SYSTEM PROGRAM INDICATORS

category
Quality Does the JLIS Efficiency Pre&Post-implementation:
implementation result in • For a group of tests that
improved efficiencies clinically could be undertaken at
for the patient? one time (E.g. Hematology
Profile, Sodium, Alkaline
phosphatase, Total protein,
TSH) – the number of times that
all the tests were conducted on
the same day and not spread out
over multiple days
Quality Does the JLIS Audit Post-implementation:
implementation result in • Number of audit related
an improved ability to searches conducted and
conduct clinical audit & actioned
research?
Quality Does the JLIS Patients Post-implementation:
implementation allow • Number of patient initiated data
patients to upload their uploads to data repository by
own test results into the region, result type, age & sex of
data repository for patient, disease
viewing/access by their
care providers?
Quality Does the JLIS Alerts Post-implementation:
implementation result in • Number of alerts triggered and
increased patient safety? acted upon
Access Does the JLIS result in a Cost Pre&Post-implementation
decrease in the average • Cost per test, for a set of
cost per test predetermined tests.
143
A.2 ADDITIONAL PUBLIC HEALTH SYSTEM PROGRAM INDICATORS

category
Quality: Health Does an immunization Infection rate Infection with vaccine-preventable
Outcomes registry decrease diseases.
infection rates?
Quality: Health Does a CD and outbreak Infection rate Infection with communicable
Outcomes management system diseases.
decrease infection?
Access: Ability of Does a health alert Notification rate Providers reached by a health alert.
Patients / Providers to network improve access Time until provider is reached by a
Access Services to providers? health alert.
Productivity: Efficiency Does case management Time to manage Average time to investigate a case,
software improve a case by type of communicable disease.
productivity?
Productivity: Net Cost Is an immunization Cost- The incremental cost-effectiveness
registry cost-effective? effectiveness of of an immunization registry as
registry compared to the approach used
currently in a jurisdiction.
144
A.3 ADDITIONAL INTEROPERABLE ELECTRONIC HEALTH RECORD PROGRAM

INDICATORS

category
Productivity: Does the iEHR, with - Decrease in - Count of redundant laboratory tests
- Efficiency views to remote labs, number of pre- and post- iEHR implementation
- Net costs imaging, and redundant lab - Overall count of laboratory tests
medications, decrease tests pre- and post- iEHR implementation
redundant ordering of - Decrease in - Count of redundant imaging tests
lab and imaging tests number of pre- and post- iEHR implementation
more than the LIS or redundant - Overall count of imaging tests pre-
DI programs alone? imaging tests and post- iEHR implementation
Quality: Does the iEHR, with - Decrease in - Number of contraindicated
- Patient Safety views to remote labs, number of medications prescribed and dispensed
imaging, and contraindicated using chart reviews and pharmacy
medications, decrease medications prescription reviews
the use of
contraindicated
medications more than
the DIS alone?
Quality: Does the iEHR, with - Decrease in the - Number of medication errors using
- Patient Safety views to remote labs, number of chart reviews, pharmacy prescription
imaging, and medication errors reviews, data from an error reporting
medications, decrease database (if available)
the number of - May also want to consider provider
medication errors more and patient interviews/surveys
than the DIS alone?
Quality: Does the iEHR, with - Decrease in - Number of potential and actual
- Patient Safety views to remote labs, potential and preventable ADEs using chart
imaging, and actual reviews and computerized monitoring
medications, decrease preventable for ADEs
the number of potential ADEs - May also want to consider provider
and actual preventable and patient interviews/surveys
adverse drug events
more than the DIS
alone?
145
A.4 ADDITIONAL TELEHEALTH PROGRAM INDICATORS
Category & Sub-Category Study Question Indicator Measures

(desired change / net benefit)
Information Quality: Are ‘clients’ better ‘Client’ Clinical or Education Use:
Content, Availability informed about the awareness and a) Awareness of critical
Access: ‘User’ participation disease or condition compliance knowledge; e.g. about disease
Productivity: Care through Telehealth? or condition, its management or
coordination treatment, prevention of relapse,
and desirable outcomes
(increased)
b) Monitoring of key social and
physiological parameters; e.g.
prescribed drug use, prescribed
exercise regimen, HgbA1c, vital
lung capacity (improved).
Access: Are ‘clients’ better able Interaction with a) Opinion of ‘clients’ and
‘User’ participation to participate as a healthcare healthcare providers
partner with healthcare providers immediately after a client-care
professionals? provider encounter (improved)
Quality: Do ‘clients’ achieve a Health related a) Mortality (reduced)
Health Outcomes longer life of better quality of life b) Morbidity - burden of illness
objective quality? of ‘client’
c) QALY (increased)
Quality: Do ‘clients’ report Self reported a) Generic physical and mental
Health Outcomes better health quality of life health well-being survey /
User Satisfaction: experienced questionnaire (improved)
User Satisfaction subjectively?
Quality: Does telehealth improve Rate of recovery a) Time to return to functional
Health outcomes client recovery or status (reduced)
health? b) Time to return to work
(reduced)
Efficiency of a) ‘Client’ readmissions to
recovery hospital (reduced)
b) ‘Client’ visits to family
physicians (reduced)
c) ‘User’ time and travel costs
(reduced)
Productivity: Do care providers know a) Opinion of ‘client’ and care
Care Coordination more about their provider assessed immediately
Quality: Appropriateness / ‘clients’ needs after a client-care provider
effectiveness encounter (improved)
Productivity: Efficiency Do healthcare providers Change in a) Average number of ‘clients’
Quality: Appropriateness / manage demand and healthcare handled per week (increased)
effectiveness become more provider b) Average clinician time spent
146

productive (with the productivity per ‘client’ (reduced)
same quality of c) Average clinician
healthcare provision) administrative staff time spent
per ‘client’ (reduced)
d) Average time spent with
‘client’ (reduced)
e) Average time for review of
‘client’ past medical history
(reduced)
f) Number of return visits by
‘client’ to clinician for same
complaint (reduced)
Information Quality: Does telehealth assist Change in a) Small area variation
Content, Availability healthcare providers to healthcare (reduced)
Quality: apply evidence based provider practice b) Availability of Clinical
Patient safety, standards in designing Practice Guidelines (CPG’s)
Appropriateness / and providing health and other evidence-based
effectiveness, information and guidelines (increased)
Health outcomes healthcare? c) Application of CPG’s pre and
Productivity: post telehealth (increased)
Care Coordination d) Clinical management of
disease / condition (improved)
e) Inter-practice comparison
(telehealth vs non-telehealth)
Productivity Does telehealth Economic value a) Cost Utility Analysis
Efficiency, Net cost demonstrate value for b) Cost Effectiveness Analysis
money? c) Cost Minimisation Analysis
d) Cost Benefit Analysis
e) Cost Consequence Analysis
f) Resource utilisation review
g) Change in absenteeism
(decreased)
h) Change in ‘client’ travel
costs (decreased)
i) Change in air ambulance
service utilization (decreased)
j) Change in length of stay for
specific conditions (decreased)
Access Is the telehealth solution Utilisation Clinical Use:
Productivity sustainable? a) Total number of clinical
Net cost ‘Telehealth Sessions’ delivered
1
per month by each ‘telehealth
site’ identified by the ‘forward
1
For clinical telehealth activities, the ‘delivering site’ will be viewed as that site at which the specialist or referred
clinician is located
147

sortation area’ element of the
postal code (increase).
b) Total number of clinicians
and staff participating at the
delivery site.
c) Total number of clinical
‘Telehealth Sessions’ received
2
per month by each ‘telehealth
site’ identified by the ‘forward
sortation area’ element of the
postal code
d) Total number of clinicians
and staff participating at the
receiving site.
e) Percentage of clinicians
within a health region, health
authority, or LHIN using
telehealth per year (increased).
f) Number of ‘other sites
connected’ per ‘Telehealth
Session’
g) Number of emergent,
urgent, and routine ‘Telehealth
Sessions’ per month.
h) Number calls to “call
centres” per 100,000
population.
Administrative Use:
a) Number of out-of-facility
administrative meetings
delivered virtually within a
health region, health authority,
or LHIN per month - include
‘forward sortation area’
(increased)
b) Total number of personnel
participating.
Research Use:
a) Number of telehealth studies
/ evaluations performed within
2
For clinical telehealth activities, the ‘receiving site’ will be viewed as that site at which the referring clinician and
/ or patient is located.
148

a health region, health
authority, or LHIN, with
virtual access to common
research study / evaluation
related materials [e.g. study /
evaluation portal] - include
‘forward sortation area’
(increased)
Education Use:
a) Number of out-of-facility
seminars / rounds / CPD
activities delivered virtually
within a health region, health
authority, or LHIN per month -
include ‘forward sortation
area’ element of the postal
code (increased)
b) Total number of clinicians
and other staff participating at
the delivering site.
c) Total number of clinicians
and other staff participating at
the receiving site.
d) Total number of clinicians,
by profession, involved in
continuous learning through
distance learning.
Integration a) Number of ‘Telehealth
Services’ interoperably and
functionally connected with and
using EHR’s, EMR’s, or EPR’s
- include ‘forward sortation
(increased).
b) Number of healthcare
providers using automated
referrals.
c) Number of transactions on a
shared record within a clinical
setting (e.g. practice group,
hospital).
d) Volume of record data
transferred within or between
clinical settings (e.g. practice
group, hospital).
Longevity a) Number of months the
149

‘Telehealth Service’ has been
active
b) Number of months the
‘Telehealth Program’ has been
active
Standardisation a) Degree of adoption of
approved Pan-Canadian
standards for communication
protocols, data and messaging
formats, and EHRS integration
Long-term a) Intra- and inter-jurisdictional
commitment of telehealth policy in place and
Senior enforced
Administration b) CCHSA accreditation
achieved or sought
Long-term a) Application of approved
commitment of CPG’s
Clinicians b) Uptake of technology tools
c) Number of clinicians using
telehealth
150
Indicator: Economic Value
The ‘Economic Value’ indicator will use Cost Utility Analysis (CUA) to “identify, measure,
value and compare the costs and consequences of alternatives being considered” and thereby to
inform “value for money” judgments about a telehealth intervention or program.a
Study Question:
Does telehealth demonstrate value for money?

Productivity, Economic value
Given the continued and significant investment in e-health solutions, including telehealth
solutions, it is highly desirable to produce credible and standardized economic information that is
relevant and useful to decision makers in Canada’s publicly funded health care system.
Identifying and consistently applying a single approach to economic evaluation of telehealth
solutions would allow this. Five types of economic evaluations exist (i.e., Cost Utility Analysis
(CUA), Cost Effectiveness Analysis, Cost Minimisation Analysis, Cost Benefit Analysis, and
Cost Containment Analysis). The selection of the appropriate type of evaluation depends on
several issues, including the availability of data on outcomes. Based on information recently
presented by the Canadian Agency for Drugs and Technologies in Health, it is recommended that
Cost Utility Analysis (CUA) be adopted.

Measures:
Cost Utility Analysis is a complex and sophisticated undertaking. It is anticipated it will be
necessary to provide significant guidance to groups undertaking a CUA for their telehealth
initiatives. As a result, it is recommended the Infoway Telehealth Program develop an in-house
team to perform consistent CUA of telehealth solutions.
The complete list of Clinical, Administrative, Research, and Educational indicators can be found
above.
Exclusion Criteria:
Non noted.
151
Study Design:
See Appendix B, Section 9.4.4 for generic comments relating to Study Design.
Study Assumptions:
a) Cost Utility Analysis is agreed to be the most appropriate mode of economic analysis for
telehealth initiatives.
b) The Infoway Telehealth Program can either: i) provide adequate support and consultation to
groups performing CUA of their telehealth solution, or b) develop an in-house team to
perform consistent CUA of telehealth solutions. The latter is recommended to ensure
consistent and comparable CUA’s are performed that can build upon each other.

a) SF-12v2. The SF-12v2™ Health Survey is a 12-item subset of the SF-36v2™ that measures
eight domains of health, and generates eight dimension scores and two summary scores for
physical and mental health. It is a brief (2-3 minute), reliable measure of overall health status.
b) SF-6D. This is a classification for describing health derived from a selection of SF-36 items. It
is composed of six multi-level dimensions. Any patient who completes the SF-12 can be
uniquely classified according to the SF-6D. The SF-6D describes 18,000 health states in all.
Additional tools will be required to collect the necessary data, but cannot be specified at this
time.
Indicator Analysis:
Software and guidelines are available to support analysis for both SF-12v2 and SF-6D, together
with economic analysis guidelines from CADTH (formerly CCOHTA).
Associations:
The SF-6D data can be used together with another tool, the SF-12 (see Section 7.2), for
economic evaluation of telehealth initiatives.
Comments:
A CUA should be used where a meaningful but generic outcome measure (e.g. Health Related
Quality of Life (HRQL)) exists that shows differences between the intervention and alternatives.
A CUA uses a generic outcome measure that permits decision makers to make broad
comparisons across different conditions and interventions, it also measures outcomes as health-
related preferences, which are most often expressed as QALYs (quality-adjusted life-years)
gained. It is anticipated that the SF-12 will provide the generic outcome measure (impact on
health), and that together with the SF-6D tool it will be possible to derive QALY’s to facilitate a
CUA and demonstrate economic value.
152
The SF-12 generates eight dimension scores and two summary scores for physical and mental
health. These scores will provide an excellent means for judging the effectiveness of telehealth
interventions, but they have only limited application in economic evaluation. The SF-6D
provides a means for using the SF-12 in economic evaluation by estimating a preference-based
single index measure for health from these data using general population values. The SF-6D
allows the analyst to obtain quality adjusted life years (QALYs) from the SF-12 for use in cost
utility analysis.
To ensure consistency and to provide maximum benefit, it may be desirable for Infoway to
provide significant support to jurisdictions when performing economic analyses. This could
include provision of detailed tools and consultation. Alternatively, the Infoway Telehealth
Program might develop an in-house team to perform consistent CUA of telehealth solutions. The
latter route is recommended to ensure consistent and comparable CUA’s are performed that can
build upon each other.
References:
Guidelines for the economic evaluation of health technologies: Canada [3rd Edition]. Ottawa:
Canadian Agency for Drugs and Technologies in Health; 2006.
Drummond MF, O'Brien BJ, Stoddart GL, Torrance GW. Methods for the economic evaluation
of health care programmes [Oxford medical publications]. 2nd ed. New York: Oxford University
Press; 1997.
Contandriopoulos AP, Champagne F, Denis JL, Avargues MC. (2000) L'évaluation dans le
domaine de la santé: concepts et méthodes [Evaluation in the health sector: concepts and
methods]. Rev Epidemiol Sante Publique 48(6):517-39.
Ohinmaa A, Scott RE. Costing Model for Videoconferencing in Alberta. (J Telemed Telecare, in
press)
Ohinmaa A, Scott RE. Modeling of total costs of videoconferencing in two different tele-
network sites in Alberta (Int J Med Inform., in press)
153
Indicator: Utilization
The ‘Utilisation’ indicator assesses the volume of usage made of any implemented telehealth
intervention, and provides a measure of understanding of which solutions are inherently viewed
as successful in the eyes of ‘clients’ and ‘users’ and thereby most likely to be sustainable. It is an
indicator that can provide application specific perspective (specific telehealth programs should
have measurable and realistic adoption targets), as well as to provide a more global perspective
that reflects sustainability (see ‘Indicator Rationale’).
Study Question:
Is the telehealth solution sustainable?

Quality, Effectiveness
It is argued that by a process of ‘natural selection’ those telehealth solutions that are found by
‘clients’ or ‘users’ to be of clear value will show a solid increase in utilisation over time. By
monitoring the number of ‘Telehealth Sessions’ this steady growth will be visible, and those
showing little or no growth in utilisation (or very low steady-state levels of utilisation) can be
reviewed and re-assessed. Collecting distinct data for providing and receiving sessions, and
relating this to geographic data, is anticipated to reveal additional valuable relationships.
In addition, the first two indicators can be used as a tool to monitor whether individual telehealth
initiatives achieve their stated ‘adoption targets’.
Infoway has recognized the importance of bringing onside key healthcare personnel (specifically
physicians, nurses, and pharmacists) and is launching an awareness strategy. It makes sense to
document any change over time in the number of such clinicians actually using telehealth.
Data for the three recommended measures should be easily accessible, and provide the desired
perspective.

154
Measures:
The measures identified below refer only to clinical telehealth solutions, and only to those
indicators can be found above.
1. Total number of clinical ‘Telehealth Sessions’ delivered per month by each ‘telehealth site’
identified by the ‘forward sortation area’ element of the postal code (increased).
2. Total number of clinical ‘Telehealth Sessions’ received per month by each ‘telehealth site’
identified by ‘forward sortation area’ element of the postal code (increased).
3. Percentage of clinicians (physicians, nurses, pharmacists) within a health region, health
authority, or LHIN using telehealth per year (increased).
Exclusion Criteria:
None identified.
Study Design:
See Appendix B, Supplementary Telehealth Reference Material section for comments relating to
Study Design.
Use existing scheduling systems, or develop a manual or automated log system, to collect the
required data.
Study Assumptions:
a) Scheduling systems are available and have the appropriate capability.
b) Utilisation logs can be simply and effectively implemented, and can accumulate accurate
data
c) Delivery versus receipt of a telehealth session can be simply differentiated (see Appendix B,
Supplementary Telehealth Reference Material section)
d) Capture of the ‘forward sortation area’ of the postal code can be achieved simply (e.g.
through minor modification of scheduling systems or utilisation logs).

a) Administrative databases
b) Scheduling databases
c) Utilization logs
d) Query tools
Indicator Analysis:
a) For each desired health region, health authority, or LHIN, document the number of
‘telehealth sessions’ that are delivered each calendar month; calculate the percentage change
(increase) in number of ‘telehealth sessions’ per month; map the number and / or percentage
change to visually demonstrate increasing access in these areas.
155
b) For specific geographic regions (e.g. North of 60; rural areas), communities (e.g. First
change within those specific locales to visually demonstrate increasing access in these
specific areas.
c) Repeat a) and b) for ‘telehealth sessions’ received data.
d) Calculate rate of change in utilisation (month to month) - a valuable parameter that can be
used to summarise growth in utilization.
e) Compare observed utilisation rates to adoption targets for rate of change and steady-state
levels of utilisation provided by proponents of the telehealth solution.
f) Repeat a) and b) for ‘clinician user’ data; present as a percentage of the total clinicians within
the geographic region.
g) Collate, and graphically display utilisation data.
Associations:
Co-analysis of this indicator with geographic information (as recommended above) will allow
improvements in utilisation rates in remote sites (historically low) to be clearly visualized and
monitored.
Utilisation figures may also be of benefit in costing activities.
Comments:
None.
References:
See Section 7.1.
156
Indicator: Integration
The indicator ‘Integration’ assesses whether telehealth solutions are being melded with
electronic record initiatives (either retroactively or ideally as they are being implemented), which
is essential for telehealth solutions to be sustainable. This indicator represents an estimation of
functional integration with electronic record systems, but does not assess practical or policy
integration with healthcare systems.
Study Question:
Is the telehealth solution sustainable?

Access, Productivity, Net Cost
It is accepted that to achieve maximum benefit, telehealth solutions must become fully integrated
into daily practice. The term ‘integration’ has diverse interpretation. Thus practical and policy
integration with existing healthcare processes is essential, and will probably be addressed
through change management and policy analysis initiatives. Similarly functional integration with
the growing number of electronic records is essential so that patient and clinical information and
data created during telehealth, or other, encounters can be shared. It is this latter form of
integration that is the focus of this indicator. Of note is that the Telehealth Investment strategy
places high emphasis on the need to ensure functional integration of Telehealth and Health
Informatics initiatives (e.g. Electronic Health Records, Electronic Medical Records, Electronic
Patients Records, and even Personal Health Records).

Measures:
The measure identified below refers only to clinical telehealth solutions, and only to those
measures recommended for use. The complete list of Clinical, Administrative, Research, and
Educational indicators can be found above.
a) Number of ‘Telehealth Services’ interoperably and functionally connected with and using
EHR’s, EMR’s, or EPR’s - include ‘forward sortation area’ element of the postal code
(increased).
157
Exclusion Criteria:
It is possible that some innovative telehealth applications that do not utilize or collect clinical
data may not require integration (e.g. purely educational / informational web portal telehealth
applications).
Study Design:
Study Assumptions:
Suitable tools can be developed to allow subjective or objective measure of the degree of
interoperable and functional connectivity.

a) Subjective assessment of the degree of interoperable and functional connectivity of a
telehealth application or service with identified electronic records.
b) Objective assessment of the degree of interoperable and functional connectivity of a
telehealth application or service with identified electronic records using a standardized ‘test
patient record’ and ‘test patient clinical assessment’ tool.
Indicator Analysis:
a) Document the number of ‘telehealth applications’ or ‘telehealth services’ found to satisfy
either of the subjective or objective assessments.
b) For specific geographic regions (e.g. North of 60; rural areas), communities (e.g. First
change within those specific locales to visually demonstrate increasing integration in these
specific locales.
Associations:
None noted.
Comments:
The indicator ‘Integration’ perhaps most directly addresses the sub-category ‘care coordination’,
but can also be viewed as an indirect indicator for the sub-category ‘Client participation’, and
perhaps the sub-category ‘patient safety’ (greater exchange of patient information between
clinicians would improve clinical decisions and thereby patient safety).
References:
See Section 7.1.
158
Supplementary Telehealth Reference Material
1. Terminology
When describing indicators and measures it is essential that a common understanding exist of
what is meant by any particular term. At this time there is no single or accepted source available
that describes or defines common terms applied within the telehealth environment. In the
absence of such material, the following descriptions have been extracted or adapted from various
telehealth related documents or presentations. Their use is recommended to ensure consistency
as parameters are measured.
i) Functional Infrastructure
Telehealth Unit: The related group of elements (hardware and software, including peripheral
devices) that comprises a distinct and functioning apparatus that can be used to perform a
specific Telehealth Activity, Application, or Service [see definitions below]. A Telehealth Unit
may be static, mobile, or handheld, and includes units for off-site use.
Telehealth Facility: A discrete and identifiable physical location (e.g. dedicated room, or
dedicated space within a room) from which Clinical, Research, Education, Administration, or
Mixed; (CREAM) telehealth related pursuits are provided or received. A Telehealth Site may
have more than one Telehealth Facility.
Telehealth Site: A discrete and identifiable geographic location (e.g. healthcare facility, clinic,
campus) from which one or more Telehealth Activities, Applications, or Services are provided or
received. This will include ‘client’ homes and other locations as home telehealth activities
expand.
ii) Receiving and Delivering Site:

• For clinical telehealth activities, the ‘delivering site’ is that site at which the specialist or
referred clinician is located, and the ‘receiving site’ is that site at which the referring
clinician and / or patient is located.
• For administrative meetings facilitated via telehealth, no distinction is made between any
sites as delivering or receiving.
• For educational telehealth activities, the ‘delivering site’ is that site at which the primary
presenter is located, and the ‘receiving sites’ are those sites at which the learners are
located.
159
iii) Functional Relationships
Telehealth Activity: A telehealth mediated pursuit, at the experimental, pilot, or formative

evaluation stage.
Telehealth Application: A traditional or novel healthcare related pursuit (clinical, administrative,

research, or educational; CARE) at the summative evaluation stage or demonstrated to be
effectively facilitated through the use of telehealth.
Telehealth Service: A specific and proven Telehealth Application offered routinely between
Telehealth Sites typically within a Telehealth Program [e.g. Forensic Telemental Health
Assessment; Pre-catheterisation Teleassessment; Home Telemonitoring].
Telehealth Program: A distinct, appropriately conceived, designed, staffed, managed, funded,

and accredited set of Telehealth Services orchestrated under a common theme and common
administrative structure [e.g. Telemental Health Program; Telecardiology Program; Home
Telehealth Program].
Telehealth Setting: Type of facility at which a telehealth session is performed. (for example:
hospital, community health centre, community health facility (LTC facility / residential care
facility). General practice, specialist practice, home, or other).
Telehealth Policy: A set of statements, directives, regulations, laws, and judicial interpretations
that direct and manage the life cycle of e-health.
iv) Miscellaneous
Client: A generic term intended to encompass any individual who is the recipient or beneficiary
of a telehealth session (thereby interacting in some form with the healthcare system). For
example; a patient, a patients’ family member, a well person, or a clinician. It is NOT intended to
imply any monetary transaction or obligation.
User: A generic term intended to describe any individual who applies telehealth during the
course of their day. A ‘user’ may or may not be a ‘client’.
Other Sites Connected: The count of all other sites that were involved in a ‘telehealth session’
while a site was connected to a session. Recorded by each telehealth site for all other sites, this
describes the size of a multipoint connection, It is accepted that there will be some anomalies in
the number of sites recorded by each site due to sessions in which some participants drop in and
out of the session.
160
Emergent: Refers to a situation where a service cannot be delayed for more than 24 hours from
the time of referral.
Urgent: Refers to a situation where a service is required within 7 days of referral.
Routine: Refers to a situations that are neither emergent or urgent.
Telehealth Session: A period of time set aside or used for a telehealth-related activity that
involves the use of information and communications technology (ICT)* to link two or more
facilities. The activity might include clinical interactions (delivering services, providing inter-
professional consults), administrative interactions (management meetings between facilities or
jurisdictions), research interactions (assessing ‘telehealth activities’), or educational interactions
(seminars, rounds, CPD, on-line CBT).
* For the purpose of current data collection, use of ICT will refer to activities such as
videoconferencing, image transfer modalities, and wireless data transfer, and exclude
the use of telephone, facsimile, and e-mail for data transfer.
A single ‘telehealth session’ may be made up of one or several activities, which may or may not
be recorded individually (e.g. one chronic disease management clinic may provide services to
several patients individually or collectively during one session). A single ‘telehealth session’
may be between: a ‘client’ and a healthcare provider; a client, a healthcare provider, and another
person such as an interpreter, another healthcare provider, or a family member or carer; two or
more healthcare providers; or individuals involved in administrative, research, or educational
interactions, and which may or may not include healthcare providers.
Telehealth Session Identifier: This is a unique identifier allocated to each single ‘telehealth
session’. Use of such an identifier links activity from multiple sites involved in a single
telehealth session, and removes ambiguity or inflation of data when quantifying telehealth
activity. A ‘telehealth session identifier’ may vary by jurisdiction, and could be assigned through
a centralised scheduling / booking system.
Telehealth Session Purpose: The principal purpose for conducting a telehealth session. This
may be identified through the acronym CARE: 1. Clinical (including health promotion and / or
pubic health), 2. Administrative (including management meetings), 3. Research (including
evaluation activities), and 4. Educational.
2. Proposed use of ‘Forward Sortation Area’ of Postal Code
Objectives of the Telehealth Investment strategy include geographic and cultural targets. To
assess achievement of these targets requires a simple means by which to assess coverage. The
161
Postal Code represents a well known and universally applied concept across Canada with which
people are familiar. The first component of the Postal Code – the ‘Forward Sortation Area ‘ -
could be applied to accurately understand the true penetration and distribution of many aspects
of interest within the Telehealth portfolio, including the precise location of Telehealth Sites,
Telehealth Services, clinicians utilising telehealth capabilities, and distinction between urban and
rural (or more precise) use. Because the second component of the postal code is not used, there
should be no privacy concerns. The level of geographic detail possible using the ‘Forward
Sortation Area’ is described below (modified from the Canada Post website at
http://www.canadapost.ca/personal/tools/pg/manual/b03-E.asp#41091).
i) Forward Sortation Area – The First Segment of the Postal Code

The “forward sortation area” or “FSA” represents a specific area within a major geographic
region or province. The first character of the forward sortation area segment identifies one of
the 18 major geographic areas, provinces or districts:
The second character of the forward sortation area is an important component as it identifies
either:
• an urban postal code: numerals 1 to 9 (E2J). Urban postal codes are generally serviced by
Letter Carrier or community mailboxes; or
162
• a rural postal code: numeral 0 (zero) (A0A). Rural postal codes are serviced by rural
route drivers and/or postal outlets.
The third character of the forward sortation area segment (E2J) in conjunction with the first
two characters, describes an exact area of a city or town or other geographic area.
3. Study Design
A generic evaluation ‘framework’ has recently been developed 3, illustrated below (Figure 1). This
framework offers the evidence-based, structured, and systematic approach desired for any study.
Within the framework, specific recommendations for outcomes and measures can be incorporated
based upon the existing objectives of Infoway, with additional outcomes and measures included to
satisfy local needs.
In the diagram below, it is implied that growth of telehealth initiatives proceeds in a generally
sequential manner through several ‘stages’ – the pre-telehealth stage, the telehealth development
stage (introduction of ‘telehealth activities’), the telehealth implementation stage (introduction of
‘telehealth applications’), the telehealth integration stage (establishing ‘telehealth services’), and
3
NT Consulting: Evaluation Workshop – ‘Doing It Right’ © 2005
163
ultimately the sustained operation stage (introduction of ‘telehealth programs’) where telehealth
is regarded as just another healthcare process. The stages of development, implementation, and
integration are disruptive and transforming processes that take current routine clinical care to an
environment where telehealth is no longer disruptive, but simply routine.
Based on this perspective, the recommended study design is related to the stage of the telehealth
initiative and the goal of the evaluation. For example:
• If the telehealth initiative is in the design or pre-deployment phase, and the goal is to
understand the best design or deployment options, then performance of a comprehensive,
focussed (not systematic) literature review is recommended.
• If the telehealth initiative is being actively developed, implemented, or integrated, and the
desire is to understand the many change processes that occur at these stages, then it may
be appropriate to look at a case study approach for the evaluation.
• If the telehealth initiative is late stage development, and being implemented (i.e. a
telehealth ‘activity’ transitioning to a telehealth ‘application’, or a telehealth ‘application’
transitioning to a telehealth ‘service’), and the goal is to understand the impact (on
process or health), then a pre-post or comparison group design would be appropriate.
(See definitions of ‘activity’, ‘application’, and ‘service’ in footnote above).
• If the telehealth initiative is undergoing implementation, integration, or even sustained
operation, and the goal is to understand the outcomes (health or economic), then a
rigorous comparison group evaluation design would be recommended.
164
STAGE Pre-
Pre-Telehealth Development Implementation Integration Sustained Operation
DESCRIPTION Routine Care Disruption / Transformation Routine Telehealth

Design / Deploy
Literature Review
Change Process
Case Study
EVALUATION
APPROACHES Impact
Pre and Post

Outcome
Comparison Group
EVALUATION CQI
Lessons Learned
OUTPUT Process Outcomes
Cost-
Cost-Benefit
Health Outcomes
BENEFITS REALISATION
‘Go / No Go’
Go’ ‘Go / No Go’
Go’ Confirmation
Recommended Evaluation Framework

(Taken from: NT Consulting: Evaluation Workshop – ‘Doing It Right’ © 2005)
Given the anticipated focus of Infoway on more mature telehealth solutions, it is recommended that the
study design required of evaluators be either a pre and post or comparison group study design.
165
Synopsis of Telehealth/Telemedicine Systematic Reviews
A synopsis of 23 systematic/literature reviews of evaluation studies for different types of

telehealth and/or telemedicine applications is enclosed. The synopsis shows the topic areas
reviewed, the study designs involved, and the types of indicators/measures used. In some cases
only the abstract of the publication is on-hand at the time of our review; so the data provided is
incomplete and will be updated when the full article becomes available.
Authors Topic Design Indicators/Measures

Azarmina 05 Telephone and videoconference 9 studies – 7 Interval between consultation, client/doctor
interpretation phone, 1 VC, 1 satisfaction, videoconferencing cost
RCT
Bonacina 05 Telecardiology, 1992-2004 61 studies - 21 Feasibility studies, pilot projects, short term
>1yr outcomes
Eadie 03 Telemedicine in surgery 2 telesurgery Patient outcomes, operating times
systems in clinical
use
Farmer 05 Telemedicine to support glucose 32 papers - 12 Telemedicine for diabetes care
self-monitoring in diabetes RCT, 10 PC, 3 feasible/acceptable; improving
XT, 1 GT HbA1c/management and reducing cost not
strong,
Hailey 02 Telemedicine assessment vs. 66 studies Administrative changes, patient outcomes,
non-telemedicine alternative economic assessment
1966-2000
Hailey 03 Success/failure papers in 89 papers – 10 on Equipment reliability, vendor reliability,
telehealth conferences concepts, 40 political/budgetary issues, perceived telehealth
primary studies, 26 need, stakeholder involvement, provider
programs opinion, management structure stability, staff
turnover, organization cooperation, health
outcomes
Hailey 04 Telemedicine assessment vs. 48 papers – 42 administrative changes, patient outcomes,
non-telemedicine alternative programs, 46 economic assessment
1966-2000 studies
Heinzelmann Telemedicine/telehealth ? Care process – diagnosis, clinical
05 management/outcomes
Hersh 01 Efficacy of telemedicine home, 25 articles Clinical outcomes (CDM, HT, AIDS), home
office/hospital based glucose monitoring, comparison with face-to-
face in ER, surgical and neonatal ICU, patient
transfer in neurosurgery
Hersh 02 Telemedicine efficacy in 58 articles Diagnostic/management decision efficacy –
diagnostic and management medical history, physical exams for sensitivity
decisions in home, and specificity, compared to face-to-face care
office/hospital, store-n-forward
Jaatinen 02 Teleconsultations – 128 articles Data protection, patient confidentiality,
patient/doctor, patient/nurse, synchronous vs. real time, flexibility, cost-
nurse/doctor effectiveness
Jennett 03 Telehealth socio-economic 306 sources Access to health services, cost-effectiveness,
impact education, health outcomes, quality of care,
quality of life, social support
Lamminen 03 Telemedicine in ophthalmlology ? Educational benefits, patient satisfaction;
166
Authors Topic Design Indicators/Measures

questions on cost-efficiency,
organizational/operational issues
Louis 03 Telemonitoring of heart failure 20 studies – 18 Hospital bed occupancy, patient
1966-2002 OS, 6 RCT acceptance/compliance, early deterioration
detection, readmission rates/charges, LOS,
mortality at 6 months, hospital care cost vs.
videoconsult cost
Mair 00 Patient satisfaction in 32 studies – 26 Patient satisfaction, patient perception,
telemedicine surveys,1 RCT provider/patient interaction
only
Monnier 03 Telepsychiatry, 1970-2000 68 studies Novel clinical demonstration, reliability of
clinical assessment, clinical outcomes,
patient/provider satisfaction, cost and cost-
effectiveness, legal/regulatory/ethical issues
NIFTE 03 Telehealth framework of Environment scan, Clinical standards/outcomes, human resources,
guidelines literature review, organizational readiness, organizational
survey, interview leadership, technology/equipment
NTOIP 03 Telehealth outcome indicators Environment scan Quality, access, acceptability, cost indicators
OHIH 00 Telemedicine/telehealth 90 papers – 33 as IOM – quality, accessibility, cost,
evaluation key references, 15 acceptability
as key for Bashshur – accessibility, cost, quality; client,
presentation provider, society
Huston – quality of patient care, legal issue,
cost-effectiveness
Perednia – safety, efficacy, clinical utility,
cost-effectiveness
Burghgraeve – accessibility/first contact,
continuity of care, comprehensiveness of care;
structure, process, outcome
Lobley – structured economic analysis;
cost/benefits
McIntosh – cost-consequence matrix
Reardon 05 Telemedicine cost ? Methods on statistical inference, critical
economic concepts, contextual definition for
costs/benefits
Roine 01 Telemedicine assessment studies 50 studies Administrative changes, patient outcomes,
economic assessments
Whitten 02 Cost benefit of telemedicine 24/55 studies Cost comparison, cost utility, level of
utilization, sensitivity analysis
Williams 01 Telehealthcare patient 93 studies – 19 CT Satisfaction – provider/patient interaction,
satisfaction patient feeling about consultation, technical
aspects
Legend: RCT-randomized control trial, PC-prospective cohort, XT-crossover trial, GT-group trial, OS-observational studies, CT-controlled
trial, VC-videoconference
Systematic/Literature Review Sources
Azarmina P, Wallace P. Remote interpretation in medical encounters: a systematic review.

Journal of Telemedicine and Telecare 2005; 11(3):140-145.
167
Bonacina S, Draghi L, Masseroli M, Pinciroli F. Understanding telecardiology success and

pitfalls by a systematic review. Studies in Health Technology and Informatics 2005;
116:373-8.
Eadie LH, Seifalian AM, Davidson BR. Telemedicine in surgery. British Journal of Surgery
2004; 90:647-658.
Farmer A, Gibson OJ, Tarassenko L, Neil A. A systematic review of telemedicine interventions

to support blood glucose self-monitoring in diabetes. Diabetic Medicine 2005;22(10):1372-
8.
Hailey D, Roine R, Ohinmaa A. Systematic review of evidence for the benefits of telemedicine.
Journal of Telemedicine and Telecare 2002; 8(Suppl.1): S1:1-7.
Hailey D, Crowe B. A profile of success and failure in telehealth – evidence and opinion from
the successes and failures in telehealth conferences. Journal of Telemedicine and Telecare
2003; 9(Suppl. 2): S2:22-24.
Hailey D, Ohinmaa A, Roine R. Study quality and evidence of benefit in recent assessments of
telemedicine. Journal of Telemedicine and Telecare 2004; 10(6):318-324.
Heinzelmann PJ, Williams CM, Lugn NE, Kvedar JC. Clinical outcomes associated with
telemedicine/telehealth. Telemedicine Journal and E-health 2005; 11(3):329-47.
Hersh WR, Helfand M, Wallace J, Kraemer D, Patterson P, Shapiro S, Greenlick M. Clinical

outcomes resulting from telemedicine interventions: a systematic review. BMC Medical
Informatics and Decision Making 2001; 1:5.
Hersh W, Helfand M, Wallace J, Kraemer D, Patterson P, Shapiro S, Greenlick M. A systematic

review of the efficacy of telemedicine for making diagnostic and management decisions.
Journal of Telemedicine and Telecare 2002; 8(4):197-209.
Jaatinen PT, Forsstrom J, Loula P. Teleconsultations: who uses them and how? Journal of
Telemedicine and Telecare 2002; 8(6):319-324.
Jennett P, Hall LA, Hailey D, Ohinmaa A, Anderson C, Thomas R, Young B, Lorenzetti D, Scott
RE. The socio-economic impact of telehealth: a systematic review. Journal of Telemedicine
and Telecare 2003; 9(6):311-320.
168
Lamminen H, Voipio V, Ruohonen K, Uusitalo H. Telemedicine in ophthalmology. Acta

ophthalmologica Scandinavica 2003; 81(2):105-9.
Louis AA, Turner T, Gretton M, Baksh A, Gleland JGF. A systematic review of telemonitoring
for the management of heart failure. The European Journal of Heart Failure 2003;5:583-
590
Mair F, Whitten P. Systematic review of studies of patient satisfaction with telemedicine. BMJ
2000; 320:1517-20.
Monnier J, Knapp RG, Frueh BC. Recent advances in telepsychiatry: an updated review.
Psychiatric Services 2003;54(12):1604-1609.
National Initiative for Telehealth. NIFTE Framework of Guidelines. September 2003.
Scott RE, McCarthy F, Jennett P, Perverseff T, Palacios MF, Rush B. National Telehealth
Outcome Indicators Project. A Canadian Consensus Approach to Identification and
Definition of Outcome Indicators for Evaluation of Telehealth. May 2003.
Office of Health and the Information Highway. Evaluating Telehealth Solutions: A review and
Synthesis of the Telehealth Evaluation Literature. Health Canada. March 2000. URL:
http://www.hc-sc.gc.ca/ohih-bsi/pubs/2000_tele/tele_e.html . Last accessed Dec17/2002.
Reardon T. Research findings and strategies for assessing telemedicine costs. Telemedicine
Journal and E-health 2005; 11(3):348-69.
Roine R, Ohinmaa A, Hailey D. Assessing telemedicine: a systematic review of the literature.

Canadian Medical Association Journal 2001;165(6):765-71.
Whitten PS, Mair FS, Haycox A, May CR, Williams TL, Hellmich S. Systematic review of cost
effectiveness studies of telemedicine interventions. BMJ 2002;324:1434-7.
Williams TL, May CR, Esmail A. Limitations of patient satisfaction studies in telehealthcare: a
systematic review of the literature. Telemedicine Journal and E-health 2001; 7(4):293-316.
169
APPENDIX B: PROGRAM SAMPLE TOOLS
B.1 DIAGNOSTIC IMAGING PROGRAM SAMPLE TOOLS
1. Referring Physician Questionnaire (Pre-PACS) (Paper Based), Newfoundland and

Labrador Example
PACS OPINION SURVEY: REFERRING PHYSICIANS

Thank you for agreeing to complete this questionnaire. As noted in the cover letter, the purpose of this study is
to determine the benefits of implementing/enhancing the Picture Archiving and Communications System in
Newfoundland and Labrador. This survey looks at your current film environment (Sections I and II), your
perceived benefits and potential challenges to PACS prior to implementation (Sections III and IV) and
demographics (Section V). Your responses are anonymous; no personal identifiers are attached to this
questionnaire.
Section I: Current Use of Film
Please respond to statement 1 through 8 by circling one of the following responses:
1 Never (N)
2 Rarely
3 Sometimes
4 Frequently
5 Always (A)
6 Not Applicable (N/A)
How often you use film in the following ways:
(N) (A) N/A

1) Clinical assessment 1 2 3 4 5 6
2) Clinical diagnosis 1 2 3 4 5 6
3) Clinical treatment 1 2 3 4 5 6
4) Professional education 1 2 3 4 5 6
5) Rounds 1 2 3 4 5 6
6) Patient education 1 2 3 4 5 6
7) Health services research 1 2 3 4 5 6
8) Other (specify) 1 2 3 4 5 6
170
Secition II: Current Enviroment with Respect to Locating Film/Reports
Please respond to statement 9 and 10 by circling one of the following responses:
1 Strongly Disagree (D)

2 Moderately Disagree
3 Moderately Agree
4 Strongly Agree (A)
(D) (A) N/A

9) I can always find film when I need it 1 2 3 4 5
10) I can always find a report when I need it 1 2 3 4 5
11) What is the average time per day you spend looking for film? Minutes____ Hours _____ N/A_____
12) What is the average time per day you spend looking for reports? Minutes____ Hours _____ N/A_____
13) What is the average time per day you spend managing/handling film? Minutes____ Hours _____ N/A _____
14) How often is your clinical schedule delayed because of a delay in obtaining prior exams?
Never Rarely Sometimes Very Often Always N/A
15) How satisfied are you with the amount of time it takes to retrieve or access each of the following?
Very Very
Dissatisfied Dissatisfied Satisfied Satisfied N/A
Film
Reports
16) How important are the following in managing patient care (e.g. rendering a diagnosis, treatment
planning)?
Not At All Not Very Somewhat Very

Important Important Important Important N/A
Film
Reports
171
17) How often do you look at prior film and/or reports?
Film
Reports
18) After how much time is a film no longer referred to in the patient care process?
< 3 Months 3-6 Months 6-12 Months 12-18 Months >18 Months
Depends on Clinical Context N/A
19) How many hospital sites do you work in? ____
20) Please estimate the number of hours per week you spend travelling between hospital sites
(If less than one hour, please write 0)
Hours ____ NA_____
21) Where do you currently access film/reports? (Please check all that apply)
Film Reports
In medical imaging In medical imaging

Private office Private office
Home office Home office
22) What do you access most frequently?

Exams
Reports
Both
Section III: Perceived Benefits of PACS Pre-Implementation
In your opinion, what might be the benefits of having PACS? Please indicate the extent to which you agree
or disagree with the following statements.
172
Please respond to statement 23 through 33 by circling one of the following

responses:
3 Moderately Agree
(D) (A) N/A

23) PACS will reduce the time I must wait to review an 1 2 3 4 5
exam (images).
24) I will access exams more frequently with PACS than I do 1 2 3 4 5
with film.
25) I believe that report turnaround time will improve 1 2 3 4 5
with the implementation of PACS.
26) I believe that having access to PACS tools and 1 2 3 4 5
functionality will improve the quality of the report.
27) PACS will facilitate consultation between myself, 1 2 3 4 5
other clinicians and/or radiologists at other health
care locations.
28) My efficiency will improve because of PACS. 1 2 3 4 5
29) PACS will improve my ability to make decisions 1 2 3 4 5
regarding patient care.
30) PACS will lead to a reduction in my patients' length of 1 2 3 4 5
stay in hospital.
31) PACS will reduce the number of patient transfers between 1 2 3 4 5
facilities due to the ability to share images and consult
remotely.
32) PACS will reduce the number of exams reordered because 1 2 3 4 5
the exams were not available (lost or located elsewhere)
when I need them.
33) PACS will enhance patient care and service delivery in 1 2 3 4 5
rural Newfoundland and Labrador.
Section IV: Potential Challenges of PACS Pre-Implementation

In your opinion, what might be the potential challenges to using PACS? Please indicate the extent to
which you agree or disagree with the following statements.
173

3 Moderately Agree
(D) (A) N/A

34) PACS will produce inadequate image quality on the Web 1 2 3 4 5
(e.g. from home)
35) PACS will produce inadequate image quality on the 1 2 3 4 5
workstation
36) I will have difficulty finding images when needed 1 2 3 4 5
37) I will experience inadequate Web performance (speed) 1 2 3 4 5
38) I will experience inadequate workstation performance (speed) 1 2 3 4 5
39) I will have inadequate access to PACS viewing stations
(PCs with Web or Workstations). 1 2 3 4 5
40) I will have difficulty logging on to the system 1 2 3 4 5
41) PACS downtime will be higher than acceptable 1 2 3 4 5
42) I will receive insufficient training in the new technology 1 2 3 4 5
43) I will be unable to view images at the patient's bedside 1 2 3 4 5
44) I will experience a lack of availability of system support 1 2 3 4 5
Section V: Demographics
45) Please indicate your gender
Male
Female
46) Years in practice
under 2 years
2 to 5
6 to 10
11 to 15
16 to 20
21 to 25
over 25
174
47) Have you had experience with PACS prior to this implementation project?
Yes
No
48) Please indicate your speciality ___________________________
49) What hospital site do you normally work in?
General Hospital
St. Clares
Western Memorial
Captain William Jackman
Charles S. Curtis Memorial
Other (Specify) __________________________________________
50) Comments
Please use this space to write any other comments you may have about the PACS system.
Thank you for taking the time to complete this questionnaire!!!
175
2. Referring Physician Questionnaire (Post-PACS) (Web Based), British Columbia Example

(Interior Health Authority)
PACS OPINION SURVEY: REFERRING PHYSICIANS
As a user of PACS, your opinion on PACS would be greatly appreciated in the PACS
Opinion survey. This survey has been designed by Canada Health Infoway with the input of
radiology experts.
The purpose of this survey is to gather your opinion on the impact PACS has had on image
access, patient care, patient transfers, duplicate tests, productivity, report turnaround time,
etc. The results from many project sites across Canada will be used to develop a national
view of the state of the diagnostic imaging environment since the implementation of PACS.
Information that is obtained during this survey will be kept confidential to the full extent
permitted by the law. You will not be identified by name in any reports of the completed
survey; your responses will only be analyzed in aggregate by Canada Health Infoway or a
third party that has been contracted for analysis. All identifiers in the records of data will be
removed. Records of the data, identifying participants by code number only, will be stored on
a fire-wall protected server and protected by password. Only authorized users will have
access.
Your opinion on the impact of this technology is important as it will assist in the
measurement of the “actual” benefits created by technologies that are implemented to
improve timely access to patient results and to establish the shared electronic patient record.
INSTRUCTIONS
The survey should take about 5-10 minutes to complete. Please return the survey in the enclosed
envelope by XXXXXXXX
If you have any questions about the survey please contact XXXXXXXX
176
1. Please select your project.
  Fraser Health Authority PACS Project  Other, please specify _____________________
2. Please select your site.

FRASER NORTH FRASER SOUTH FRASER EAST
Royal Columbian Surrey Memorial Abbotsford MSA
Hospital Hospital Hospital
Eagle Ridge Hospital Peace Arch Hospital Chilliwack General
Hospital
Ridge Meadow Hospital Delta Hospital Fraser Canyon Hospital
Burnaby Hospital Langley Memorial Mission Memorial
Hospital Hospital
USAGE AND BENEFITS
3. Where do you access the PACS system? (Please check all that apply)
Diagnostic Imaging Department Clinics/Units/Patient care floors Personal Office
4. Do you access PACS most frequently for: (Please check one response)
Exams Reports Both
5. Please indicate the extent to which you agree or disagree with the following statements:
(Please select one response per row)
Strongly Moderately Moderately Strongly Not

Disagree Disagree Agree Agree Applicable
a) PACS has reduced the time I must wait to review an
exam/image.
b) I access exams more frequently with PACS than I did
with film.
c) I believe that report turnaround time has improved since
the implementation of PACS.
d) I believe that PACS tools and functionality improve the
quality of the report.
e) PACS has facilitated consultation between myself,
other clinicians and/or radiologists at other health
care locations.
Please indicate the extent to which you agree or disagree with the following statements.
6a. My efficiency has improved because of PACS. (Please check one response)
Strongly  Moderately  Moderately  Strongly Not

177
6b. Please comment on the degree of agreement/disagreement with the above statement.
7a. PACS has improved my ability to make decisions regarding patient care. (Please check one response)

8a. PACS has led to a reduction in my patients' length of stay in hospital. (Please check one response)

9a. PACS has reduced the number of patient transfers between facilities due to the ability to share images and
consult remotely. (Please check one response)

9b. If you agreed with the statement above please estimate the number of transfers avoided per month since the
Estimated number of transfers avoided per month in my practice:
10a. PACS has reduced the number of exams reordered because the exams are not available (lost or located
elsewhere) when I need them. (Please check one response)
178

10b. If you agreed with the statement above please estimate how many reordered exams are prevented per
month since PACS was implemented.
Estimated number of reorders (duplicates) avoided per month:
11a. PACS has reduced the amount time it takes me to access exams/reports. (Please check one response)

11b. If you agreed with the statement above please estimate, on average, how much time do you save per week
by being able to access exams/reports on PACS? Please specify your response in minutes (e.g. 3 hours should
be entered as 180 minutes). If not applicable, enter '0'.
Minutes:
SATISFACTION WITH PACS
12. Please indicate the extent to which you agree or disagree with the following statements. (Please select one
response per row)

a) PACS image quality on the Web (e.g. from home or
office) is adequate.
b) PACS image quality on the workstation in the
Hospital is adequate.
c) PACS functionality on the remote Web is
adequate.
d) PACS functionality on the workstation in the
Hospital is adequate.
e) Finding images when needed is easy.
f) Remote web performance (speed) is adequate.
g) Hospital workstation performance (speed) is
adequate.
h) Access to PACS viewing stations (PCs with
Web or Workstations) is adequate.
i) Logging on to the system is easy.
j) PACS downtime is acceptable.
k) Training in the new technology was sufficient.
l) Ability to view images at the patient's bedside is
adequate.
m) The Implementation/Installation and transition
from film was well managed.
179
SUMMARY AND COMMENTS
13. Please use this space to write any other comments you may have on the PACS system.
DEMOGRAPHICS
14. Years in practice:
under 2 years 2 to 5 6 to 10 11 to 15 16 to 20 21 to 25 over 25
15. Have you had experience with PACS prior to this implementation project?
Yes No
16. Please specify your specialty:
Cardiology Internal Medicine Obstetrics/ Paediatrics Thoracic Surgery

Gynaecology
Emergency Nephrology Oncology Surgery Other, please specify

Medicine
______________________
Family Practitioner / Neurology Orthopaedics Cardiac
General Practitioner Surgery
Gastroenterology Neurosurgery Orthopaedic Vascular

Surgery Surgery
Thank you for taking the time to complete this survey.
Please return the survey in the enclosed envelope by *************
180
3. Radiologist Questionnaire (Pre-PACS) (Paper Based), Newfoundland and Labrador

Example
PACS OPINION SURVEY:

RADIOLOGIST/TECHNICIANS/TECHNOLOGISTS
Thank you for agreeing to complete this questionnaire. As noted in the cover letter, the purpose of this study is
to determine the benefits of implementing/enhancing the Picture Archiving and Communications System in
Newfoundland and Labrador. This survey looks at your current film environment (Sections I and II), your
perceived benefits and potential challenges to PACS prior to implementation (Sections III and IV), and
demographics (Section V). Your responses are anonymous; no personal identifiers are attached to this
questionnaire.
Section I: Current Use of Film
1 Never (N)
2 Rarely
3 Sometimes
4 Frequently
5 Always (A)
How often you use film in the following ways:

(N) (A) N/A
1) Clinical assessment 1 2 3 4 5 6
2) Clinical diagnosis 1 2 3 4 5 6
3) Clinical treatment 1 2 3 4 5 6
4) Professional education 1 2 3 4 5 6
5) Rounds 1 2 3 4 5 6
6) Patient education 1 2 3 4 5 6
7) Health services research 1 2 3 4 5 6
8) Other (specify) 1 2 3 4 5 6
181
Section II: Current Enviroment with Respect to Locating Film/Reports
.
Please respond to statement 9 and 10 by circling one of the following responses:

3 Moderately Agree
(D) (A) N/A

9) I can always find film when I need it 1 2 3 4 5
10) I can always find a report when I need it 1 2 3 4 5
11) What is the average time per day you spend looking for film? Minutes____ Hours _____ N/A____
12) What is the average time per day you spend looking for reports? Minutes____ Hours _____ N/A____
13) What is the average time per day you spend managing/handling films? Minutes____ Hours _____ N/A ____
14) How often is your clinical schedule delayed because of a delay in obtaining prior exams?
21) How satisfied are you with the amount of time it takes to retrieve or access each of the following?
Very Very
Dissatisfied Dissatisfied Satisfied Satisfied N/A
Film
Reports
22) How important are the following in managing patient care (e.g. rendering a diagnosis, treatment planning, etc.)?
Not At All Not Very Somewhat Very

Important Important Important Important N/A
Film
Reports
182
23) How often do you look at prior film and/or reports?

Film
Reports
24) After how much time is a film no longer referred to in the patient care process?
< 3 Months 3-6 Months 6-12 Months 12-18 Months >18 Months
Depends on Clinical Context N/A
25) How many hospital sites do you work in? ____
26) Please estimate the number of hours per week you spend travelling between hospital sites
(If less than one hour, please write 0)
Hours ___ N/A ____
21) Where do you currently access film and/or reports? (Please check all that apply.)
Film Reports
In medical imaging In medical imaging
Private office Private office
Home office Home office
22) What do you access most frequently?
Exams
Reports
Both
Section III: Perceived Benefits of PACS Pre-Implementation
Please consider the current film-based environment when indicating the extent to which you agree or
disagree with the following statements.
183
Please respond to statement 23 through 35 by

circling one of the following responses:
3 Moderately Agree
5 Not Applicable
(D) (A) N/A

23) PAC will reduce the time I spend locating 1 2 3 4 5
exams for review.
24) I will access prior exams more frequently with 1 2 3 4 5
PACS than I did with film.
25) I believe report turnaround time will improve because 1 2 3 4 5
of PACS (i.e. time to report dictated or time to
preliminary report available).
26) I believe that PACS tools and functionality will improve 1 2 3 4 5
the quality of my report.
27) PACS will improve the quality and number of patient 1 2 3 4 5
management rounds that I participate in.
28) PACS will increase the number of face to face 1 2 3 4 5
consultations I have with physicians and other
radiologists.
29) PACS will increase the number of phone (or other) 1 2 3 4 5
consultations I have with physicians and other
radiologists.
30) PACS will reduce my professional 1 2 3 4 5
travel time.
31) PACS will improve medical student/radiology 1 2 3 4 5
resident teaching.
(D) (A) N/A
32) With the implementation of PACS, I will report remotely 1 2 3 4 5
for sites to which I previously traveled.
33) With the implementation of PACS, I will report remotely 1 2 3 4 5
for new sites.
34) PACS will improve my reporting and consultation 1 2 3 4 5
efficiency.
35) PACS will enhance patient care and service delivery in 1 2 3 4 5
rural Newfoundland and Labrador.
184
Section IV: Potential Challenges of PACS Pre-Implementation
In your opinion, what might be the potential challenges to using PACS? Please indicate the extent to
which you agree or disagree with the following statements.
Please respond to statement 36 through 47 by circling one of

the following responses:

3 Moderately Agree
5 Not Applicable
(D) (A) N/A

remote Web (e.g. from home).
workstation
38) PACS will provide inadequate functionality on the 1 2 3 4 5
remote Web
39) PACS will produce inadequate functionality on the 1 2 3 4 5
workstation
40) I will have difficulty finding images in PACS when 1 2 3 4 5
I need them.
41) I will experience inadequate remote Web performance 1 2 3 4 5
(speed)
42) I will experience inadequate Workstation performance 1 2 3 4 5
(speed)
43) I will experience inadequate access to PACS viewing 1 2 3 4 5
stations.
44) I will have difficulty logging on to the 1 2 3 4 5
system
45) PACS downtime will be higher than 1 2 3 4 5
acceptable.
46) I will receive insufficient training in the new 1 2 3 4 5
technology.
47) I will experience a lack of availability of system 1 2 3 4 5
support.
Section V: Demographics
48) Please indicate your gender
Male
185
Female
49) Years in practice
under 2 years
2 to 5
6 to 10
11 to 15
16 to 20
21 to 25
over 25
50) Have you had experience with PACS prior to this implementation project?
Yes
No
51) Please indicate your profession
Radiologist Physician
Radiology Technologist
Radiology Technician
Other (specify) __________________________________________
52) What hospital site do you normally work in?
Labrador Health Centre (Goose Bay)

Captain William Jackman Memorial Hospital
Charles S. Curtis Memorial Hospital
Other (Specify) _______________________________________
53) Comments
Please use this space to write any other comments you may have about the PACS system.
Thank you for taking the time to complete this questionnaire!!!
186
4. Radiologist Questionnaire (Post-PACS) (Web Based), British Columbia Example (Interior

Health Authority)
PACS OPINION SURVEY: RADIOLOGISTS
As a user of PACS, your opinion on PACS would be greatly appreciated in the PACS
Opinion survey. This survey has been designed by Canada Health Infoway with the input
of radiology experts.
The purpose of this survey is to gather your opinion on the impact PACS has had on
image access, patient care, patient transfers, duplicate tests, productivity, report
turnaround time, etc. The results from many project sites across Canada will be used to
develop a national view of the state of the diagnostic imaging environment since the
Information that is obtained during this survey will be kept confidential to the full extent
permitted by the law. You will not be identified by name in any reports of the completed
survey; your responses will only be analyzed in aggregate by Canada Health Infoway or a
third party that has been contracted for analysis. All identifiers in the records of data will
be removed. Records of the data, identifying participants by code number only, will be
stored on a fire-wall protected server and protected by password. Only authorized users
will have access.
Your opinion on the impact of this technology is important as it will assist in the
measurement of the “actual” benefits created by technologies that are implemented to
improve timely access to patient results and to establish the shared electronic patient
record.
INSTRUCTIONS
The survey should take about 5-10 minutes to complete. Please return the survey in the
enclosed envelope by XXXXXXX
If you have any questions about the survey please contact XXXXXXXX
187
1. Please select your project.
 Fraser Health Authority PACS Project  Other, please specify _____________________
2. Please select your site.
FRASER NORTH FRASER SOUTH FRASER EAST

Royal Columbian Hospital Surrey Memorial Hospital Abbotsford MSA Hospital
Eagle Ridge Hospital Peace Arch Hospital Chilliwack General Hospital
Ridge Meadow Hospital Delta Hospital Fraser Canyon Hospital
Burnaby Hospital Langley Memorial Mission Memorial Hospital
Hospital
USAGE AND BENEFITS
3. Where do you access the PACS system? (Please check all that apply)
Diagnostic Imaging Department Private office Home office
4. Please compare to a film-based environment when indicating the extent to which you agree or disagree
with the following statements. (Please select one response per row)

a) PACS has reduced the time I spend locating
exams for review.
b) I access prior exams more frequently with PACS
than I did with film.
c) I believe that report turnaround time has improved
because of PACS (i.e. time to report dictated or
time to preliminary report available).
d) I believe that PACS tools and functionality
improve the quality of my report.
e) PACS has improved the quality of patient
f) Since the implementation of PACS the number of
face to face consultations I have with physicians
And other radiologists has decreased.
g) Since the implementation of PACS the number of
phone (or other) consultations have with physicians
and other radiologists has increased.
h) PACS has reduced my professional travel time.
i) PACS has improved medical student/radiology
resident teaching.
188
5. Since the implementation of PACS, I report remotely for sites to which I previously travelled.
Yes No Not Applicable
6. Since the implementation of PACS, I report (will report) remotely for new sites.
Yes No
7a. Overall, PACS has improved my reporting and consultation efficiency: (Please check one response)
Increased (Specify the % increase) Decreased (Specify the % decrease)
Stayed the same
7b. Please comment on how PACS has increased or decreased your reporting and consultation efficiency.
SATISFACTION WITH PACS
8. Please indicate the extent to which you agree or disagree with the following statements. (Please select
one response per row)

a) PACS has reduced the time I spend locating
exams for review.
b) I access prior exams more frequently with PACS
than I did with film.
c) I believe that report turnaround time has improved
because of PACS (i.e. time to report dictated or time
to preliminary report available).
d) I believe that PACS tools and functionality improve
the quality of my report.
e) PACS has improved the quality of patient
f) Since the implementation of PACS the number of face to
face consultations I have with physicians and other
radiologists has decreased.
189
g) Since the implementation of PACS the number of phone

(or other) consultations have with physicians and other
radiologists has increased.
h) PACS has reduced my professional travel time.
i) PACS has improved medical student/radiology
resident teaching.
SUMMARY AND COMMENTS
9. Please use this space to write any other comments you may have on the PACS system.
DEMOGRAPHICS
10. Years in practice:
under 2 years 2 to 5 6 to 10 11 to 15 16 to 20 21 to 25
over 25
11. Have you had experience with PACS prior to this implementation project?
Yes
No
Thank you for taking the time to complete this survey.
Please return the survey in the enclosed envelope by XXXXXXX
190
4. Time Motion Study Collection Sheet, British Columbia Example

We are trying to understand the time it takes for a patient to flow through your area. This is not an evaluation of
your performance. The data will give us a “before picture” so we can evaluate whether PACS speeds up or slows
down patient flow. Please fill in the start and end times as noted (if possible, please record the from the right hand
bottom corner of your computer monitor) and your name.
Front Desk
Subject PHN:
Requesting Physician: (RP) ER OR
IP External
Procedure: General CT
Angio US
Mammo SP
Patient Arrival Date: Time: Your Name:
Time patient registration in ITS
is complete:
Time exam requested by RP
(e.g. ER) __:__ am / pm
Time of scheduled appointment:
N/A __:__ am / pm
Technologists Date: Time: Your Name:
Time first image taken: __:__ am / pm
Time of checkpoint with
radiologist: N/A __:__ am / pm
Time last image complete: __:__ am / pm
Time all images printed: __:__ am / pm
Time exam available for
radiologist to interpret: __:__ am / pm
Radiologists
Were patient’s prior exams available at the time of interpretation? Y N
Time interpretation (reading)
started: __:__ am / pm
Time interpretation (dictation)
finished: __:__ am / pm
Please attach this sheet to transcription requisition.
Transcriptionist
Time draft report is available: __:__ am / pm
Thank you!!
191
B.2 DRUG INFORMATION SYSTEM PROGRAM
1. Important Prescibing Problems Identified by Consensus Panel, Hospital Based Studies

and Outpatient Studies
192
193
194
195
196
197
198
199
Drug Interactions Identified in Hospital Based Studies

Literature examining adverse drug events (ADE) in the general hospital population had a variety
of measurements for the adverse events. Nebeker et al (2005) found a rate of 52 ADE per 100
admissions, with 26% of admissions having an ADE. In Baker et al’s 2004 study, 7.5 per 100
admissions were found to have an adverse event, however, only 23.6% of these were drug
related. Overall, 36.9% of the adverse events were considered preventable. In Buajordet’s
(2001) study of fatal adverse events, of the 732 deaths examined, 18% were considered to be
drug events. 50% of the fatal adverse drug events involved inappropriate prescriptions.
Hardmeier’s team (2004) distinguished between ADEs that were present on admission, and those
that took place during hospitalisation, as well as distinguishing between adverse drug reactions,
which could not be anticipated, and error related adverse drug events, which could be prevented.
They found that 1.2% of admissions were the result of ADEs due to error, and 7.5% of patients
experience an ADE in hospital, of which 6% were due to error, and were therefore preventable.
Forster et al (2004) similarly distinguished between adverse events in hospital and those before
hospitalisation, and found that 12.7% of patients had an adverse event, and 4.8% of patients had
a preventable adverse event. Only 50% of these events were due to drugs. Slightly more of the
in-hospital events (40%) were considered preventable than those that happened prior to
hospitalisation (32%). Bates et al (1999) reported a rate of 247 adverse events in 4031
admissions, of which 28% were preventable. Blix et al (2004) and Viktil et al (2004), examining
the same patient population, found that 81% of patients had a drug related problem, and only 8%
of these were adverse drug reactions. Other problems involved poor choice of medication,
unnecessary drugs, and improper dosages.
In more specialised studies that examined particular problems, drug groups, or patient
populations, high rates of ADE’s or inappropriate prescribing were found. Davies (2004) found
19% of psychiatric patients with combinations of medications that led to clinically significant
interactions. Guglielmo (1999), examining antibiotic prescribing, found that 1.6% of antibiotic
orders required an intervention to prevent an adverse drug reaction. Lesar (2002), in his study
examining dosage and form errors in prescriptions, found a rate of 1.23 errors per 100
admissions during the duration of the study.
Where specified, the drug groups most commonly contributing to adverse drug events across the
hospital literature were narcotics (Nebeker 2005, Lesar 2002, Bates 1999, Blix 2004, Viktil
200
2004, Buajordet 2001), cardiovascular or renal (Nebeker 2005, Buajordet 2001, Hardmeier 2004,
Lesar 2002, Blix 2004, Viktil 2004), and antithrombotic agents (Buajordet 2001, Hardmeier
2004, Blix 2004, Viktil 2004), Sedatives and NSAIDs were listed as among the top drug groups
for adverse drug events in three studies (Hardmeier 2004, Bates 1999, Viktil 2004, and
Hardmeier 2004, Blix 2004, Viktil 2004, respectively). Antibiotics (where not the focus of the
study) were mentioned in two studies (Bates 1999, Hardmeier 2004). Diuretics (Nebeker 2005),
bronchdilatory agents (Buajordet 2001), antidiabetic agents (Lesar 2002), and psychiatric
medications were mentioned in one study each.
Table 2: Examples of drugs leading to adverse drug events in hospital based studies
Primary drug(s) Interacting drug, Possible adverse event Reference
involved in population, or (where given)
interaction condition
bupivacain ephedrine Cardiac arrest Buajordet et al
2001*
Glibenlamid ampicillin Haemolytic anaemia Buajordet et al
2001*
potassium cephalexin Oesophageal ulcer and Buajordet et al
pseudomembranous 2001*
colitis
prednisone Gastric bleeding with Buajordet et al
aspiration and CNS 2001*
depression
diazepam sedation Buajordet et al
2001*
furosemide Aggravation of Buajordet et al
dehydration 2001*
Glipzide, insulin metformin hypoglycaemia Buajordet et al
2001*
Enalapril, diltiazem furosemide hypotension Buajordet et al
2001*
furosemide hypomagnaesaemia Buajordet et al
2001*
diazepam Respiratory arrest Buajordet et al
2001*
Enalapril, Glycerol trinitrate hypotension Buajordet et al
furosemide 2001*
Fluorouracil leucovorin Neutropenia and sepsis Buajordet et al
2001*
verapamil Aggravation of Buajordet et al
hypotension 2001*
Antineoplastic agent Bone marrow depression Buajordet et al
and sepsis 2001*
201
tobramycin netilmicin Acute renal failure Buajordet et al

2001*
Antineoplastic agent Dexamethasone Bone marrow depression, Buajordet et al
immunosuppression and 2001*
pneumonia
nitrates Hypotension, myocardial Buajordet et al
ischaemia 2001*
diuretics Hypotension, Buajordet et al
dehydration/electrolytic 2001*
disorders
Beta blockers Hypotension, Buajordet et al
cardiodepression, 2001*
dysrhythmia, bronchial
obstruction
Calcium channel Hypotension, Buajordet et al
blockers cardiodepression, 2001*
dysrhythmia
ACE inhibitors Hypotension, renal Buajordet et al
failure 2001*
digitoxin Dysrhythmia, digitoxin Buajordet et al
overdose 2001*
antidisrhythmics dysrhythmia Buajordet et al
2001*
etilephrine Myocardial ischaemia Buajordet et al
2001*
amiodarone alveolitis Buajordet et al
2001*
phenprocoumon multiple risk factors Muscle haematoma, Hardmeier 2004
for falls
iloprost Hypotension Hardmeier 2004
Nasal decongestant Rhinitis medicamentosa Hardmeier 2004
Amoxicillin, Known allergy rash Hardmeier 2004
clavulanic acid
Co-trimoxazole Known allergy rash Hardmeier 2004
ASA History of ulcer GI bleeding Hardmeier 2004
diclofenac History of ulcer GI bleeding Hardmeier 2004
ceftriaxone Previous known thrombopenia Hardmeier 2004
interaction
flurbiprofen Known renal Aggravation of renal Hardmeier 2004
insufficiency insufficiency
midazolam Respiratory insufficiency, Hardmeier 2004
hepatic insufficiency
lorazepam Somnolence, apnea, Hardmeier 2004
multimorbidity
202
digoxin Renal insufficiency Hardmeier 2004

metoprolol Bradycardia, hypotension Hardmeier 2004
paracetamol Toxic liver injury Hardmeier 2004
Lidocaine Bradycardia, asystole Nebeker 2005
NSAID Heparin analogues Upper gastrointestinal
bleeding, myocardial
infarction
Narcotic analgesics Apnea, stupor, vomiting
and aspiration
*
This study examined fatal adverse drug events only
Drug Interactions Identified in Outpatient Based Studies

Because ascertainment of adverse drug effects in primary care is more difficult, there are fewer
studies have attempted to quantify the adverse drug events in the outpatient population. Some of
the hospital based studies above differentiated between those patients who arrived with an
adverse event related to drugs and those who experienced the adverse event while in hospital
(Forster et al 2004, Hardmeier et al 2004). Howard et al (2003) attempted to determine the rate
of preventable ADE admissions by having patients examined by a pharmacist on admission. In
this study, 6.5% of admissions were found to be drug-related, and 67% of those were
preventable.
A small number of studies examined ADE in primary care, without the requirement of
hospitalisation. Gurwitz et al (2004) identified ADEs in an elderly outpatient population. In
their study, 1523 ADEs were identified, 28% of which were preventable. Gandhi et al (2003) in
a survey study found that 25% of the patients who responded had an adverse drug event, with a
rate of 27 events per 100 patients. 11% of the events were considered preventable. Honigman et
al (2001), identifying ADEs through a computer programme, found a rate of 5.5 ADEs per 100
patients who received care, and 3.4 hospital admissions for ADE per 1000 patients. Miller et al
(2006) in their study in Australia found that 10.4% of general practice patients had experienced
an ADE, of which 10% were categorized as serious. 23.2% of the ADEs were considered
preventable.
Several papers tried to determine the incidence of inappropriate prescribing, rather than ADEs.
Bjerrum et al (2003) did not examine adverse events, but studied pharmacy records to identify
the prevalence of prescribing potentially interacting drugs. They found that 15% of patients had
potentially interacting drugs prescribed to them, though they state that only a small percentage
would be expected to go on to experience an adverse drug event. Bjorkman et al (2002)
undertook a similar analysis of prescribing to elderly people in 6 European countries, and found
that 46% of patients had a potential drug-drug interaction, though 90% of these involved
combinations where an adverse event could be avoided by dose adjustments. Fialova et al
(2005) surveyed prescribing in the elderly receiving home care in 8 European countries, and
203
found an average of 19.8% had inappropriate prescriptions. Adverse events resulting from the
inappropriate prescribing were not considered. The most common drugs involved in
inappropriate prescribing were pentoxifylline, diazepam, amiodarone, amitriptylline, ticlodipine,
unopposed estrogens in women >= 75 years, doxazosine, fluoxetine (daily), piroxicam, short-
acting dipyridamole, short-acting nidedipine, short-acting oxybuynin, and chlordazepoxide.
There were significant differences across countries, suggesting that generalising from one
country to another may not be valid. Guedon-Moreau undertook a similar study, but examined
prescriptions for the entire population, and limited the area to part of Northern France. In this
study, 0.28% of prescriptions were found to have a contraindication. Of the contraindicated
prescriptions, 26% were an absolute contraindication, and 74% were relative.
The drug groups most often implicated in ADE in outpatients (where specified) varied more by
study than in those found in the hospital based studies. Only diuretics were named as a top drug
contributing to ADEs in more than two studies (Gurwitz et al 2003, Honigman et al 2001,
Howard et al 2003), and beta blockers (Gandhi et al 2003, Howard et al 2003), ACE inhibitors
(Gandhi et al 2003, Honigman et al 2001), NSAIDs (Gandhi et al 2003, Howard et al 2003) and
hypoglycaemics (Gurwitz et al 2003, Howard et al 2003), and anticoagulants (Gurwitz et al
2003, Howard et al 2003) were listed as drug groups most involved in adverse drug events in two
studies each. Howard (2003) also listed antiepileptic medication, inhaled steroids, and cardiac
glycosides as important drugs for adverse events, Gurwitz (2003) included nonopioid analgesics,
Gandhi (2003) had SSRIs, and Honigman (2001) included antihypertensives and antibiotics.
Table 3: Examples of drugs leading to adverse drug events in outpatient based studies
Primary drug(s) Interacting drug, Possible adverse Reference
involved in population, or event (where given)
interaction condition
amiloride Potassium, lithium Bjerrum et al 2003*
spironolactone potassium Bjerrum et al 2003*
bendroflumethiazide lithium Bjerrum et al 2003*
indapamide lithium Bjerrum et al 2003*
Warfarin ASA, thyroid Bjerrum et al 2003*
hormones,
cimetidine,
amiodarone,
Phenobarbital,
metronidazole,
carbamazepine,
disulfiram,
testosterone
phenprocoumon ASA, thyroid Bjerrum et al 2003*
hormones,
amiodarone,
Phenobarbital,
204
carbamazepine,
gemfibrozile,
testosterone,
diflunisal,
bezafibrate
ASA Warfarin, Bjerrum et al 2003*
phenprocoumon,
methotrexate
lithium Thiazides, amiloride, Bjerrum et al 2003*
indapamide,
metropolol
chinidine digoxin Bjerrum et al 2003*
phenobarbital Phenprocoumon, Bjerrum et al 2003*
warfarin
carbamazepin Erythromycin, Bjerrum et al 2003*
phenprocoumon,
warfarin
metropolol lithium Bjerrum et al 2003*
digoxin chinidine Bjerrum et al 2003*
cimetidine warfarin Bjerrum et al 2003*
phetidine Selegiline, Bjerrum et al 2003*
isocarboxazid
tramadole isocarboxazid Bjerrum et al 2003*
terfenadine erythromycin Bjerrum et al 2003*
erythromycin Carbamazepine, Bjerrum et al 2003*
terfenadine,
ergotamine,
ciclosporine
metronidazole warfarin Bjerrum et al 2003*
tetracycline Aluminium Bjerrum et al 2003*
hydroxide, iron
preparations, calcium
ergotamine Roxithromycin, Bjerrum et al 2003*
erythromycin
citalopram Xylocain-adrenaline Bjerrum et al 2003*
fluoxetine moclopamide Bjerrum et al 2003*
nortriptyline Xylocain-adrenaline Bjerrum et al 2003*
selegiline pethidine Bjerrum et al 2003*
testosterone Warfarin, Bjerrum et al 2003*
phenprocoumon
disulfiram warfarin Bjerrum et al 2003*
methotrexate ASA, diflunisal Bjerrum et al 2003*
Ipratropium bromide Beta-2 agonists Bjorkman et al
2002*
205
Potassium chloride Potassium chloride Bjorkman et al

sparing agents 2002*
Antithrombotic agents NSAID, ASA, Bjorkman et al
propafenone, 2002*
amiodarone,
cimetidine
codeine Antipsychotic Bjorkman et al
medications 2002*
methotrexate ASA, NSAID Bjorkman et al
2002*
verapamil Beta blockers Bjorkman et al
2002*
Dopaminergic Dopamine receptor Reciprocal Guedon-Moreau et al
antiparkinson agents antagonists antagonism of 2004*
prescribed as medications
antipsychotic agent
or antiemitics
Ergot alkaloids triptans Hypertension, Guedon-Moreau et al
coronary 2004*
vasoconstriction
cisapride macrolides Ventricular Guedon-Moreau et al
arrhythmia 2004*
floctafenin Beta blockers If anaphylactic Guedon-Moreau et al
shock induced by 2004*
floctafenin, beta
blockers decrease
compensatory
mechanism
Ergotamine macrolides Ergotism with risk of Guedon-Moreau et al
/dihydroergotamine extremity necrosis 2004*
Disopyramide and sotalol Ventricular Guedon-Moreau et al
quinidines arrhythmias 2004*
amiodarone sotalol Ventricular Guedon-Moreau et al
arrhythmias 2004*
Opiod agonist Agonist/antagonist Decrease of Guedon-Moreau et al
analgesics opiod analgesics analgesic effect, risk 2004*
of withdrawal
cisapride Imidazole Ventricular Guedon-Moreau et al
antifungals arrhythmias 2004*
*Potential interactions only
206
2. Pharmacist Satisfaction Survey
207
208
209
210
3. Response Time Testing
Protocol for Testing Response Times
• The testing will be done in the field at a variety of sites
• The evaluator will be constant.
• The evaluator will borrow a DIS from one physician by patient list size and connection type (ie
one physician with 1st quartile list size and DSL, one physician with median list size and DSL
etc.) at the site.
• Physicians will be sampled to represent the full range of patient list sizes. Physicians will also be
sampled to represent the full range of connection types, if applicable, including air cards, DSL,
and dial-up.
• The physician will provide a PIN/password/login for DIS access. In addition, the physician will
provide the names of two patients with the following criteria: a) one patient who has 1- 5
medications in his/her profile b) a second patient who has 10-15 medications in his/her drug
profile. Note: the names of the 2 patients having these criteria may be retrieved from a central
server or other data repository prior to testing so that physicians do not have to provide this
information.
• At each site, testing should be done for both patients. Testing should be undertaken at a variety of
times to capture times of greater or lower demand for connectivity by service providers.
• Before beginning the test, please make sure the DIS is set-up as follows:
i) DIS is connected to any necessary central server or has a web connection, as appropriate
ii) Full synchronization has already occurred, if necessary, or other full update of patient
information has been completed, as applicable
iii) Preferences should be set with the alert levels allowing medium level alerts to appear, but
suppressing lowest level alerts
• For #4: Prior to starting the time recording, choose Ventolin 0.4 mg/ml oral liquid; 5 mg QID
• For #6: Prior to generating alerts, prescribe Ventolin 0.4 mg/ml oral liquid; 5 mg QID, Nr,
Ind=Asthma and then complete the Rx Pad with PMS Atenolol, 100mg TAB, 30 days, Nr, Ind =
Anxiety.
• Once testing is completed, evaluator should delete all prescriptions made during testing and if
necessary, reset preferences.
211
RESPONSE TIME RECORDING CHART
Date: ______________________________ Time:_________ Location:

_____________________________
Dr: ____________________________ Memory Free Storage ______ Free Program ______ Moxxi
______
Task Start time End time Time taken for task

1. Logging in a) Press enter on icon for DIS Login/ password screen
appears ___ mins___ secs ____
system
b) Enter pin or password DIS entry screen appears ___ mins___ secs ____
2. Calling up a a) Enter minimum number of a) Patient List with selected

patient’s file characters necessary for Patient characters appears
Search and enter ___ mins___ secs ____
b) Select the patient’s name b) Patient Information

from the list Screen or other patient file ___ mins___ secs ____
entry screen appears
3. Accessing a i. Select prescribing screen from i. Rx-Pad screen appears
drug to menu or icon ___ mins___ secs ____
prescribe
(drug ii. a)Select Search after typing ii. Drug screen appears
recognition) ‘apo’ (generic) in the Drug field ___ mins___ secs ____
on the Rx-Pad
b) Select Search after typing

‘ati’ (trademark) in the Drug ___ mins___ secs ____
field on the Rx-Pad
c) Select Search after typing

‘ven’ (trademark) in the Drug ___ mins___ secs ____
field on the Rx-Pad
d) Select Search after typing

‘vita’ (generic) in the Drug field ___ mins___ secs ____
on the Rx-Pad
e) Select Search after typing
212
‘benzo’ (generic) in the Drug ___ mins___ secs ____

field on the Rx-Pad
f) Select Search after typing ___ mins___ secs ____

‘vent’ (trademark) in the Drug
field on the Rx-Pad
4. Entering a Choose the posology indicated The Rx-Pad screen appears.
posology in the directions from menu list ___ mins___ secs ____
(default or other default listing, if
dosing) applicable
5. Entering an If DIS has this function, access Indication appears in the
indication indication function and enter prescription ___ mins___ secs ____
asthma for indication on
prescription
6. Prescription From the completed prescription Alerts detected screen
Alert from select or otherwise activate alert appears ___ mins___ secs ____
Rx-Pad system
7. Save & print a Select function required to save Patient Information (or
prescription and print the prescription. other default) Screen appears ___ mins___ secs ____
213
4. Standardized Assessment of User Ability
Standardized Tasks
Materials for evaluators:

- 1 stop watch per evaluator
- Computer with training data base installed on DIS for each physician
- 1 Procedure Sheet per Evaluator
- 1 set of numbered index cards per evaluator (Warm-up & Index Cards 1-5) with Patient Name &
Health card#; diagnosis to be changed from potential to confirmed; allergy to be added; eRx with
4 medications with Rx details; eRx Stop with details; eRx Change with details; and eR-Rx with 8
renewal medications
- 1 Recording Chart per physician to be evaluated
- Tape
- 1 Pen per evaluator; 1 pen per physician
- personalized prescription paper of each physician (5 per physician being evaluated)
General Tasks:
1. Log on to DIS
2. Search for a patient
3. Write a prescription for four medications and electronically send it
4. Re-prescribe eight medications and electronically send the prescription
5. a. stop and change a medication; b. change a medication dose
6. Change a patient’s problem status
7. Add one allergy
Specifics for each task:
Warm-up
Name of patient: XXXXXX
RAMQ#: XXXXXX (adjust as necessary for other provincial health systems)
1. 4 Medications to be prescribed and sent electronically:
1) Senokot 8.6mg tabs, 2 tabs HS for 3o days, no repeat; indication: constipation
2) Codeine 30 mg tabs, one tab q4h prn #20, no repeat; indication: cough
3) Serax 10 mg tabs, one tab HS prn #30, no repeat; indication: insomnia
4) Ventolin HFA 100 mcg met. Inh, 2 puffs qid prn for 30 days, no repeat; indication: asthma
2. 8 Medications to be re-prescribed are those already on the system and include:
i. Coumadin 5 mg 1 TABLET once a day for 30 days, no repeats
214
ii. Sotalol 80mg 1 TABLET twice a day for 30 days, no repeats

iii. Prinivil 10mg 1 TABLET once a day for 30 days, no repeats
iv. Metformin 500mg 1 TABLET twice a day for 30 days, no repeats
v. Avandia 4 mg 1 TABLET once a day for 30 days, no repeats
vi. Oxeze Turbuhaler 12mcg 1 twice a day for 30 days, no repeats
vii. Centrum select 1 TABLET once a day, for 60 days, no repeats
viii. Zocor 10mg 1 TABLET once a day for 30 days, no repeats
3a. Stop: Serax 10 mg tabs, one tab HS prn #30, no repeat

Reason: Intolerance to drug or side effect(s)
Prescribe: Restoril 15 mg capsule, 1 capsule qHS prn #30, no repeat; indication: insomnia
3b. Change: Codeine 30 mg tabs(from 3b)

Reason: Error in prescription
To: Codeine 15 mg tables, 1 tab q4h prn, #20, no repeat; indication: cough
4. Confirm diagnosis of “OBESITY”
5. Allergy to be added: Celebrex – medication intolerance
Setting:
The standardized tasks should be performed in a non-distracting environment.
NOTE: It is important to follow the instructions as they are written here. Do not provide any assistance or
clues during the tasks. If necessary, you can reassure the physician that although the case used for the
standardized tasks is a complicated one, he/she only has to concentrate on doing the requested tasks.
Schedule for administering the standardized tasks:
The standardized tasks should be performed by the physicians during the following times:
1) At the training session

2) At the end of the first week of DIS usage in clinic
3) At the end of the first month of DIS usage in clinic
4) At the end of the third month of DIS usage in clinic
NOTE: the paper comparison tasks are to be performed only at the training session and not thereafter.
215
Procedure for the Evaluator:
1) The physician sits in front of a table with DIS in front.
2) The Evaluator sits next to the physician with a stopwatch in one hand and a pen in the other hand;
the recording chart is taped directly in front of the evaluator on the table; the index cards are
stacked in order (from #1 to #5) and are placed on the table between the evaluator and the
physician.
3) The stopwatch is set at zero. The DIS is turned on with the start/home page on the screen.
4) The following instructions are given to the physician before beginning the tasks:
“This is a standardized procedure that is designed to determine the length of time it takes to
perform routine tasks on the drug information system. You will be asked to use the drug
information system to complete five* tasks that include: searching for a patient’s record;
electronically prescribing 4 different medications; electronically re-prescribing 8 medications.
For a comparison, you will also be asked to handwrite the same 4 medications that are to be
prescribed and the same 8 medications that are to be re-prescribed on a prescription pad. You
will also be asked to change the status of a problem listed for the patient; and add an allergy*
During this procedure I will be seated here and timing you. Each task is timed separately.
Before I begin timing you for each task, I will give you verbal instructions. The details of the
task will be written on an index card that I will place in front of you [point to spot on table].
Although this is a timed exercise, it is important that you be able to complete each task as
instructed so please proceed at comfortable pace. Please complete only the tasks that have
been described to you.
I will not be able to answer questions or help you, while you are doing the task.
Do you have any questions? [ answer questions – record questions on form ]
When you are ready to start please let me know and we will begin with the first task.”
*
Note: The number of tasks may vary for systems without the optional functionalities for tasks 3, 4, and 5
5) The following are instructions to give when beginning the standardized tasks:
*NOTE: Tasks need to be done accurately so the evaluator should keep timing the task until the
physician completes it accurately.
216
*NOTE: If the physician quits a task before completely it accurately, ask, “Are you sure you
want to stop?” If the response is “yes” then stop the stopwatch and record the time with a note
that the task was not completed.
i) Ok. I am going to describe the warm-up task now.
Log on to the drug information system using the PIN, login, or password and search
for this patient.
Place the index card in front of the physician
Are you ready to start the warm-up task?

if no – answer questions then repeat question until response is “yes”
if yes say,
Please start now.

Start recording time when physician inputs the PIN, login, or password and hits the
appropriate enter key
Finish recording time when physician: selects correct patient on result list.
Record the time on the Recording Chart.

Reset the stopwatch to zero.
Place the index card #1 in front of the physician
ii) Ok. I am going to describe the first task now.
Task #1) Electronic Prescription: Write a prescription for these 4 medications and
electronically send it.
Place the index card #1 in front of the physician
Are you ready to start the first part of the first task?
if yes say:
Please start now.
Start recording time when physician: Accesses electronic RX-PAD .

Finish recording time when physician: completes the save and send process after entering
the complete information on the Rx-pad for the fourth medication.

217
Okay, now I am going to describe the second part to this task.
Task #1b) Handwritten Prescription: Write the same prescription for this patient using
this prescription paper.
Place the physician’s prescription paper, a pen and index card #3 directly in front of the
physician. Keep index card #3 in view of the physician.
Are you ready to start?

if yes say:
Please start now.
Start recording time when physician: Begins writing

Finish recording time when physician: Completes the prescription (4 medications) with a
signature, date, & patient’s name.

Remove the prescription paper and pen from table.
Ok. I am going to describe the second task now.
Task #2a) Electronic Renewal: renew these 8 medications and electronically send the
prescription.
Place the index card #2 in front of the physician.
Are you ready to start the first part of the second task?
if yes say:
Please start now.
Start recording time when physician: accesses the function for re-prescribing.
Finish recording time when physician: completes the save and send process after
completing the renewal for all 8 medications.

218
Okay, now I am going to describe the second part to this task.

Task #2b): Handwritten Renewal: Re-prescribe the same medications to this patient
using this prescription paper.
Place the physician’s prescription paper, a pen, and index card # directly in front of the
physician.
Are you ready to start?

if yes say:
Please start now.
Start recording time when physician: Begins writing

Finish recording time when physician: Completes the prescription (8 medications) with a
signature, date, & patient’s name.

Ok. I am going to describe the third task now
Task #3a) Electronic Stop: Stop the medication, prescribe this other one and
electronically send the prescription.
Place the index #3 in front of the physician.
Are you ready to start the first part of the third task?
if yes say:
Please start now.
Start recording time when physician: Accesses electronic Rx-Pad

Finish recording time when physician: completes the save and send process after stopping
the medication

Ok. I am going to describe the second part of the third task now.
219
Task #3b) Electronic Change: Change the medication dose from 30 to 15 mg, and
electronically send the prescription.
Are you ready to start the second part of the third task?
if yes say:
Please start now.
Start recording time when physician: accesses electronic Rx-Pad

Finish recording time when physician: completes the save and send process after
changing the dose of the medication

Ok. I am going to describe the fourth task now.
Task #4. Add a diagnosis and confirm.
Place the index card# in front of the physician
Are you ready to start the fourth task?

if yes say:
Please start now.
Start recording time when physician: Function for adding diagnosis is accessed
Finish recording time when physician: Saves the confirmed diagnosis

Task #5: Add this allergy to the problem list.
Place the index card# in front of the physician
Are you ready to start the fifth task?

if yes say:
Please start now.
220
Start recording time when physician: Accesses function that adds an Allergy.
Finish recording time when physician: Completes the addition of the allergy and saves

6) At the end of the final task, thank the physician for his/her participation.
7) Remove the diagnosis of “obesity” or change from confirmed to potential on the problem list so
that the DIS is ready for the next standardized task procedure.
8) Remove “Celebrex” from the allergy list so that the DIS is ready for the next standardized task
procedure.
9) Delete prescriptions made today from the Prescription List.
10) Return to the home page.
221
WARM-UP PERIOD
1. Log on to the DIS
2. Search for this patient
Patient’s name: XXXXXX
RAMQ# : XXXXXXXX (or other as per provincial health system)
INDEX CARD #1
Electronic Prescription
«Task #1. Prescribe these 4 medications and electronically

send the prescription»
a) Senokot 8.6 mg tabs

2 tabs qHS
for 30 days
no repeat;
indication: constipation
b) Codeine 30 mg tabs
1 tab q4h prn
#20,
no repeat;
indication : cough
c) Serax 10 mg tabs
1 tab qHS prn
#30
no repeat;
indication : insomnia
d) Ventolin HFA 100 mcg met. inh.

2 dose qid prn
for 30 days
222
no repeat;
indication : asthma
INDEX CARD #3
Electronic Stop & Change
«Task #3. a. Stop this medication, prescribe this other one and electronically send the prescription »
STOP:
Serax 10 mg tab
1 tab qHS prn
#30
no repeat
Reason:
Intolerance to drug or side-effect(s)
Prescribe:
Restoril 15 mg capsule
1 capsule qHS prn
#30
no repeat;
indication : insomnia
«Task #3. b. Change only this medication’s dose from 30 to 15 mg, and electronically send the prescription »
Codeine 30 mg tabs (from task 3.b)

Reason: Error in prescription
To: Codeine 15 mg tabs
1 tab q4h prn
#20,
no repeat;
indication : cough
223
INDEX CARD #4 INDEX CARD #5
«Task #4.Confirm the diagnosis» «Task #5.Add this allergy to the Allergy List»
Problem : OBESITY Celebrex – Medication Intolerance
Physician’s Name:__________________________ Date:___________________
Instructions and procedure before beginning Time taken for

Start time End time
task task
“Log on to the DIS.” Access screen with input code, Entry screen appears after
PIN, or password input code, PIN or WARM-UP
password entered PERIOD
”Search for this patient.” Begins input of first characters Tap/click correct patient WARM-UP
in Last name, First name or on result list. PERIOD
Health Card number entry
field.
Task #1: Access Rx pad on DIS Prescription saved and

a. Handwritten Prescription: send acknowledged as
“Write a handwritten prescription for these 4 complete after entering a. _____ min_____
medications.” the complete information s
on the Rx-pad for the
b. Electronic Prescription: fourth medication.
“Write a prescription for these 4 medications b. _____
and electronically send it.” min_____ s
Task #2: Enter the system function Prescription saved and

a. Handwritten Renewal: “Write a handwritten where renewals can be made send acknowledged as a. _____ min_____
renewal for these 8 medications.” complete after stopping s
and renewing all
b. Electronic Renewal: medications.
“Re-prescribe these 8 medications and b. _____
electronically send the prescription. min_____ s
Task #3: Enter the system function Prescription saved and

a. Stop&prescribe another: where stops can be made send acknowledged as a. __ min
“Stop that medication, prescribe this complete after stopping _____s___
224
medication and electronically send the and changing the

prescription.” Enter the system function medication.
b. change dose: where dose changes can be
“change only the dose of this medication and made b. __ min____
electronically send the medication”. s___
Task #4: Enter the system function Diagnosis acknowledged

“Confirm this diagnosis.” where diagnoses are confirmed. as confirmed by system
If necessary, enter the
diagnosis ___ min_____
s_____
Task #5: Enter the system function Allergy acknowledged as

“Add this allergy to the Allergy List.” where allergies added, enter the confirmed by system
allergy ___ min_____
s_____
225
5. Assessment of DIS Functionalities
Table 1: Key Functionalities as identified in Bell and Avery

Functionality Domains where the functionality has a positive
impact
Patient Identification
A minimal set of patient identifying information Patient safety and health outcomes/clinician
(name, gender, and date of birth or age) visible in acceptance
the user interface throughout the process of
creating a prescription. (Bell 2004)
System capability of importing patient Patient safety and health outcomes/clinician

identification and demographic data from an acceptance
electronic medical record (EMR) or practice
management system (PMS) used by the health
care organization. (Bell 2004)
Method for manual entry of patient identification Patient safety and health outcomes/clinician
and demographic data when importing this acceptance
information from an EMR or PMS is not possible.
(Bell 2004)
Ability to merge or treat as one patient records ***Rated by Bell as not achievable within three
created under separate identities for the same years
patient. (Bell 2004)
patient safety and health outcomes/clinician
acceptance
Access to Patient Historical Data
Ability to extract patient data for decision support ***Rated by Bell as not achievable within three
from external sources including pharmacy, years
hospital, laboratory, and EMR systems. (Bell
2004) patient safety and health outcomes/ helping patients
manage costs/ clinician acceptance
Indicator for when an external Patient safety and health outcomes/clinician
interface that provides data for decision support is acceptance
not operational. (Bell 2004)
Complete current medication list, based on open Patient safety and health outcomes/helping patients
prescriptions from all other clinicians. (Bell manage costs/clinician acceptance
2004)
List of all nonprescription medications the patient ***Rated by Bell as not achievable within three
is currently taking, including over-the-counter and years
alternative medications (Bell 2004)
Patient safety and health outcomes/helping patients
manage costs/clinician acceptance
Means for entering medications the patient is Patient safety and health outcomes/helping patients
currently taking that have not been prescribed manage costs/clinician acceptance
through the system and are not available through
226
external interfaces. (Bell 2004)
Summary by drug class of the patient’s ***Rated by Bell as not achievable within three
medication history including drug names, years
dosages, start and end dates, and adherence gaps.
(Bell 2004) Patient safety and health outcomes/helping patients
Details of individual past prescriptions including ***Rated by Bell as not achievable within three
dosages, prescribing dates, and dispensing dates. years
(Bell 2004) Patient safety and health outcomes/helping patients
Orders to discontinue currently open prescriptions ***Rated by Bell as not achievable within three
with a message sent to notify the original years
prescriber of the discontinuation. (Bell 2004)
Patient safety and health outcomes/helping patients
Medication Selection
List of medications appropriate to the diagnosis Patient safety and health outcomes/helping patients
provided when a diagnosis is entered. (Bell 2004) manage costs/clinician acceptance
Prescribing without the entry of a diagnosis and Patient safety and health outcomes/helping patients
with the entry of speculative or tentative manage costs/clinician acceptance
diagnoses. (Bell 2004)
Method for prescribers to create customized Clinician acceptance

menus of medication options. (Bell 2004)
The display of medication options not influenced Patient safety and health outcomes/helping patients
by promotional considerations. manage costs/clinician acceptance
The meaning of any symbols or special fonts Patient safety and health outcomes/helping patients
available during the prescribing process. (Bell manage costs/clinician acceptance
2004)
Access within the system to the rationale for any Patient safety and health outcomes/helping patients
medication choice that the system displays as manage costs/clinician acceptance
being recommended or
preferred for the current patient. (Bell 2004)
Omit from suggested medication menus options Patient safety and health outcomes/clinician
that would be medically contraindicated for the acceptance
patient. (Bell 2004)
Prescribing by name search (generic or Patient safety and health outcomes/clinician

trademark) from a complete list of medications, acceptance
bypassing any restricted medication menus. (Bell
2004)
System provides mechanisms to prevent Patient safety and health outcomes
227
prescribing a drug of a similar name to the one

intended (eg penicillin and penicillamine) (Avery
2005)
System enables providers to determine the ***Rated by Bell as not achievable within three
accurate formulary status and the actual cost to years
the patient for each medication option based on
the patient's prescription insurance coverage. Helping patients manage costs/clinician acceptance
(Bell 2004)
System provides access to the current amount ***Rated by Bell as not achievable within three
remaining on the patient’s prescription drug years
benefit cap, if one exists. (Bell 2004)
Helping patients manage costs
System provides brief summaries of medication ***Rated by Bell as not achievable within three
effectiveness and potential harms based on the years
evidence used for regulatory actions and on
evidence from peer reviewed randomized clinical patient safety and health outcomes/clinician
trials or peer reviewed meta-analyses. (Bell 2004) acceptance
(Avery 2005)
Selection from the dosages and forms that are patient safety and health outcomes/clinician
available and appropriate for a given medication. acceptance
(Bell 2004)
Access to assistance with dosing calculations patient safety and health outcomes/clinician
based on body size and age, when such acceptance
adjustments are indicated. (Bell 2004)
Alerts and other messages to prescribers

Alert generated when a medication is selected that Patient safety and health outcomes/clinician
has a contraindication or significant precaution acceptance
based on the patient's allergies, current
medications, medical conditions, and laboratory
findings. (Bell 2004, Avery 2005)
Alert generated when a medication is selected that Patient safety and health outcomes
has a contraindication or significant precaution
based on the patient’s age, pregnancy,
breastfeeding, because appropriate monitoring
requirements have not been met (eg full blood
count in the previous 12 weeks for patient taking
methotrexate), the length of treatment course
supplied may be dangerous without review, or
medication is being prescribed with an unlicensed
indication (Avery 2005)
Alerts generated are displayed clearly on the Patient safety and health outcomes
screen (Avery 2005)
228
System should alert the user when a diagnosis Patient safety and health outcomes
code entered cannot be used to trigger an alert
(Avery 2005)
Reminder when a prescription for a new ***Rated by Bell as not achievable within three
medication might be indicated based on patient years
data (e.g., medical history, labs) in conjunction
with rules from practice criteria that are based on patient safety and health outcomes/clinician
thorough analyses of the peer-reviewed literature. acceptance
(Bell 2004)
For every message, an explanation of its rationale, patient safety and health outcomes/clinician
including disclosure of all criteria and financial acceptance
support used in its development. (Bell 2004)
(Avery 2005)
Distinguish alerts and messages based on patient patient safety and health outcomes/clinician
safety and health outcome concerns from those acceptance
based on formulary adherence and other
considerations. (Bell 2004)
Prioritize safety alerts based on clinical patient safety and health outcomes/clinician
importance (e.g., the frequency, severity, and acceptance
certainty of the possible adverse consequences).
(Bell 2004)
System allows low-priority safety patient safety and health outcomes/clinician

alerts to be suppressed either by the prescriber or acceptance
at the time of installation. (Bell 2004)
System allows the prescriber to suppress alerts Clinician acceptance

and messages that are not based on patient safety.
(Bell 2004)
Sponsorship of a message to prescribers Helping patients manage costs/clinician acceptance

indicated in the message. (Bell 2004)
Prescribers able to proceed with their intended Clinician acceptance

order, overriding any alert, with clinical
justification required for the most severe potential
adverse events. (Bell 2004) (Avery 2005)
It should be difficult to override serious hazard Patient safety and health outcomes
alerts (Avery 2005)
Alerts and messages display the date that the Patient safety and health outcomes/clinician
underlying decision support rules were last acceptance
updated. (Bell 2004)
229
Prescribers have access to patient data ***Rated by Bell as not achievable within three
that was used to trigger alerts or tailor years
medication menus. (Bell 2004) patient safety and health outcomes/clinician
acceptance
Prescribers able to correct or flag patient safety and health outcomes/clinician
patient information that they believe to be acceptance
erroneous. (Bell 2004)
Alerts coded to the most precise level that results Patient safety and health outcomes
in genuine hazard (eg interaction between
verapamil and beta blockers, rather than calcium
channel blockers and beta blockers) (Avery 2005)
Patient Education
System provides information for Patient safety and health outcomes/helping patients
patients on how to take the medications manage costs/clinician acceptance
prescribed and why they should be taken. (Bell
2004)
System able to print a complete current Patient safety and health outcomes/helping patients
medication list for patients.(Bell 2004) manage costs/clinician acceptance
Data Transmission and Storage

Transmission of prescription data between Patient safety and health outcomes/clinician
systems conforms to the most recent acceptance
versions of criteria from Health Level 7 (HL7);
transmission of other clinical data conforms to the
most recent version of HL7. (Bell 2004)
The system uses National Provider Patient safety and health outcomes
Identifiers when they become available. (Bell
2004)
System uses universal patient ***Rated by Bell as not achievable within three
identifiers (or health system identification years
number) (Bell 2004)
Patient safety and health outcomes/clinician
acceptance
System discloses any promotional Patient safety and health outcomes/helping patients
considerations that have influenced the display of manage costs/clinician acceptance
pharmacy fulfillment options.(Bell 2004)
Prescribers notified of transmission Patient safety and health outcomes/clinician

failure of a prescription to a pharmacy. (Bell acceptance
2004)
System able to receive and store notification from ***Rated by Bell as not achievable within three
the pharmacy when each prescription is delivered years
to the patient.(Bell 2004)
230

acceptance
Monitoring and Renewals
System notifies the prescriber when a prescription ***Rated by Bell as not achievable within three
or prescription refill is not dispensed and years
delivered to the patient within a time interval
specified by the provider or when a prescription is Patient safety and health outcomes/clinician
refilled too frequently, indicating overuse by acceptance
patient. (Bell 2004) (Avery 2005)
System indicates whether a medication requested Patient safety and health outcomes
as a repeat by a patient is authorized for repeats
(Avery 2005)
System reminds the clinician to place orders for ***Rated by Bell as not achievable within three
follow up laboratory tests years
recommended by the manufacturer for
monitoring. (Bell 2004) (Avery 2005) Patient safety and health outcomes/clinician
acceptance
System indicates when a repeat prescription has Patient safety and health outcomes
exceeded its review date (Avery 2005)
System requires entry of a review date for repeat Patient safety and health outcomes
prescriptions (Avery 2005)
System does not allow review dates to exceed one Patient safety and health outcomes
year or sets maximum repeats allowed at 13
(Avery 2005)
System requires entry of an indication for repeat Patient safety and health outcomes
prescriptions (Avery 2005)
System alerts the prescriber when the results of ***Rated by Bell as not achievable within three
laboratory monitoring tests require action. (Bell years
2004) (Avery 2005)
acceptance
System provides access to the results of ***Rated by Bell as not achievable within three
laboratory monitoring tests. (Bell 2004) years

acceptance
Prescription renewals entered by non-prescribing Patient safety and health outcomes/clinician
office staff clearly attributable to both an acceptance
individual staff person and an authorizing
prescriber. (Bell 2004)
System does not allow unqualified staff to put Patient safety and health outcomes
drugs onto a repeat prescriptions (Avery 2005)
231
System does not allow unqualified staff to re- Patient safety and health outcomes
authorize repeat prescriptions (Avery 2005)
Transparency and Accountability

System displays notification of Patient safety and health outcomes/helping patients
corporate sponsorships and critical business manage costs/clinician acceptance
relationships that could represent conflicts of
interest, and vendors completely and
transparently disclose details of these
relationships in publicly available documents.
(Bell 2004)
Prescribing system vendors provide Patient safety and health outcomes/helping patients
access to complete and transparent disclosure of manage costs/clinician acceptance
the sources and methods used to develop clinical
decision support rules, including the triggers for
alerts and other messages. (Bell 2004)
Prescriber-level Feedback
Prescribers able to review profiles of their own Patient safety and health outcomes/helping patients
prescribing patterns. (Bell 2004) manage costs/clinician acceptance
System able to profile prescribers' history of Patient safety and health outcomes
overriding alerts.(Bell 2004) (Avery 2005)
Security and Confidentiality
System supports compliance with the most current Patient privacy/clinician acceptance
legislative criteria for privacy and security.(Bell
2004)
User activities recorded in a reliable audit trail Patient privacy
that is accessible only to authorized personnel
responsible for enforcing data privacy and
security. (Bell 2004)
Each user individually identified in the system Patient safety and health outcomes/patient
with role-based access privileges. (Bell 2004) privacy/clinician acceptance
System supports a method for Patient safety and health outcomes/patient
checking the integrity of stored or transmitted privacy/clinician acceptance
data. (Bell 2004)
Reporting and Clinical Audit
System records intended referrals and reports Patient safety and health outcomes
whether patient has been referred or not (Avery
2005)
System is able to run reports on patients who have Patient safety and health outcomes
received potentially hazardous drug/drug
combinations (Avery 2005)
System able to run reports on patients at risk from Patient safety and health outcomes
medications (eg patients on long-term high-dose
oral corticosteroids who are not receiving
prophylaxis for osteoporosis) (Avery 2005)
232
B.3 TELEHEALTH PROGRAM
About the SF-6D
A brief overview:
The SF-36 has become the most widely used measure of general health in clinical studies
throughout the world. It currently generates eight dimension scores and two summary scores for
physical and mental health. Whilst such scores provide an excellent means for judging the
effectiveness of health care interventions, they have only a limited application in economic
evaluation because they are not based on preferences.
The SF-6D provides a means for using the SF-36 and SF-12 in economic evaluation by
estimating a preference-based single index measure for health from these data using general
population values. The SF-6D allows the analyst to obtain quality adjusted life years (QALYs)
from the SF-36 for use in cost utility analysis.
What is the SF-6D?
The SF-6D is a classification for describing health derived from a selection of SF-36 items. It is
composed of six multi-level dimensions. Any patient who completes the SF-36 or the SF-12 can
be uniquely classified according to the SF-6D. The SF-6D describes 18,000 health states in all.
How does the SF-6D generate preference scores?
The SF-6D comes with a set of preference weights obtained from a sample of the general
population using the recognised valuation technique of standard gamble. Members of the general
population were asked to value a selection of health states from which a model has been
estimated to predict all the health states described by the SF-6D.
Reference:
SF-6D - http://www.shef.ac.uk/scharr/sections/heds/mvh/sf-6d (last accessed July 2006)
233
The SF-8™ Health Survey
The SF-8™ Health Survey represents a major advance in the application of SF technology for
purposes of achieving both brevity and comprehensiveness in population health surveys. It has
only eight questionnaire items. By relying on a single item to measure each of the eight domains
of health in the SF-36® Health Survey, the SF-8™ represents the ultimate phase in the evolution
of SF assessments - single-item scales.
SF-8™ is also the first SF survey to be constructed on the basis of empirical studies linking each
questionnaire item to a comprehensive "pool" of widely used questionnaire items, including but
not limited to the SF-36®, proven to measure the same health concept. Because the item "pool"
for each of the eight health concepts was calibrated on the same metric as the corresponding SF-
36® Health Survey scale, each SF-8™ single-item scale and the SF-8™ summary measures can
be scored on the same norm-based metrics as the SF-36® scales and summary measures. Thus,
the SF-8™ is an 8-item version of the SF-36® that yields a comparable 8-dimension health
profile and comparable estimates of summary scores for the physical and mental components of
health.
The SF-8™ provides the long awaited "missing link" - a practical tool for directly linking the
norms from large population surveys with the results from more focused clinical trials, outcomes
research studies, and monitoring efforts in everyday clinical practice. The SF-8™ was
constructed to replace the SF-36® and SF-12® in population health surveys in the U.S. and
internationally. Accordingly, it has been translated and linguistically validated for use in more
than 30 countries and languages using IQOLA Project methods. It has been adopted by federal
agencies (e.g., the DOD), leading polling organizations (e.g., the Roper-Starch Worldwide
Health Report), and industry sponsors of clinical trials and effectiveness research (e.g., Glaxo
Smith Kline, Johnson & Johnson, and Searle).
Finally, the SF-8™ is being used to determine which questions from widely used "static" surveys
will be most informative when assessments are computerized and items are administered
dynamically to match them to each respondents level of health. For example, SF-8™ items are
the first questions asked by QualityMetric Incorporated's Dynamic Health Assessment
(DynHA™) "engine," which uses computerized testing logic to achieve very brief and accurate
assessments of SF-36® scales and summary measures. QualityMetrics' new dynamic version of
the SF-36® is demonstrated on "www.amIhealthy.com"
The development, validation and norming of the new SF-8™ Health Survey forms, including
standard (4-week recall) acute (1-week recall) and 24-hour recall versions is documented in a
220-manual, "How to Score and Interpret Single-Item Health Status Measures: A Manual for
User of the SF-8™ Health Survey (With a Supplement on the SF-6™ Health Survey."
234
Reference:
SF-8 Health Survey. http://www.sf-36.org/tools/sf8.shtml (last accessed July 2006)
235
APPENDIX C: EXAMPLES OF INDICATORS/MEASURES, DESIGNS AND
INSTRUMENTS
In order to encourage an evidence-based approach to evaluation by building on earlier research

and lessons learned from these studies, a set of suggested evaluation indicators/measures, designs
and instruments have been included as part of the Infoway Benefits Evaluation Framework (see
Section 1.4). The intention is to provide the jurisdictions and programs with a set of sufficiently
high quality literature as the initial evidence base they can draw on to guide evaluation efforts.
The examples cited in the framework are enclosed under Citations in terms of the publication
sources.
C.1 SUGGESTED EVALUATION MEASURES FOR QUALITY AND SYSTEM USE

DIMENSIONS
Dimension Category Examples of Indicators/Measures Examples of Designs/Instruments

System Functionality Ammenwerth 2005 – type and level of Hailey 2003 – success/failure
HIS assessment criteria
Garg 2005 – type and level of CDSS NIFTE 2003 telehealth framework of
Hayward 2005 – level of CDSS guidelines
Kawamoto 2005 – type and level of CDSS
Montgomery 1998 – HIS interventions
and role of computer
Randolph 2000) – level of CDSS
van der Loo 1995 – type and level of HIS
Performance Abenstein 1992 – HIS interface, monitors Abenstein 1992 – survey of system
Balas 1998 – network data transmission in users
diabetic patient self-monitoring
Kawamoto 2005 – general system features
(computer speed)
Urschitz 1998 – data export capability Urschitz 1998 – survey of system
van der Loo 1995 – time consumption users
process/logistics, system performance NTOIP 2003 Telehealth outcome
indicators
Information Content Abenstein 1992 – user profile, Abenstein 1992 – survey of system
satisfaction, and attitude users
Allan 2000 - nursing care plan consistency Allan 2000 – content analysis of
Kawamoto 2005 – comprehension and plans
integration of CDSS and patient
information Larrabee 2001 – retrospective chart
Larrabee 2001 - nursing documentation review
completeness for assessments and
outcomes
Montgomery 1998 – recording of key Nielsen 2000 – comparison of 10%
patient information sample of paper and electronic chart
Nielsen 2000 – comparison of electronic Urschitz 1998 – survey of system
236

and paper chart on accuracy and users
completeness NTOIP 2003 Telehealth outcome
Urschitz 1998 – patient care indicators
documentation format/procedure,
accuracy, flexibility
Availability Allan 2000 – nursing documentation flow Allan 2000 – survey of system users
Balas 1996 – medical record and info
access
Kawamoto 2005 – communication and
availability of CDSS and patient
information Urschitz 1998 – survey of system
van der Loo 1995 – availability and users
timeliness of patient data (e.g. imaging) NTOIP 2003 Telehealth outcome
Urschitz 1998 – access to patient care indicators
documentation
Service Responsiveness Mitchell 2001 – user training
Use Use behavior/pattern Kawamoto 2005 – Data entry and prompt Lee 1996 – survey of system users
for action by users NTOIP 2003 Telehealth outcome
Lee 1996 – user satisfaction indicators
van der Loo 1995 – system use pattern
Self reported use Lee 1996 – self reported usage Lee 1996 – survey of system users
Intention to use
Satisfaction Competency Montgomery 1998 – physician knowledge Urschitz 1998 – survey of system
Urschitz 1998 - patient data management users
system: knowledge of the system
User satisfaction Abenstein 1992 – user profile, Abenstein 1992 – survey of system
satisfaction, and attitude users
Allan 2000 – nursing satisfaction of HIS Allan 2000 – survey of system users
Delpierre 2004 – practitioner satisfaction Murphy 1994 - survey of system
Mitchell 2001 – practitioner attitude of users (tool developed and refined)
HIS Urschitz 1998 – survey of system
Murphy 1994 – nursing attitudes pre/post user
HIS NTOIP 2003 Telehealth outcome
Urschitz 1998 - patient data management indicators
system: satisfaction with the system
van der Loo 1995 – practitioner
satisfaction
Ease of use Abenstein 1992 - user interface, time to Abenstein 1992 – survey of system
produce report users
Mitchell 2001 – time to learn/use HIS NTOIP 2003 Telehealth outcome
indicators
237
C.2 SUGGESTED EVALUATION MEASURES FOR NET BENEFITS DIMENSIONS –

QUALITY

Net Benefits Quality - Balas 1996 – performance in screening
Appropriateness – compliance and vaccination rates,
practitioner monitoring rates for medications and
performance and blood pressure, use of lab tests
patient outcome Bates 1999 – medication error rates Bates 1999 - prospective time series
adherence to policy, Delpierre 2004 – performance in with 4 periods
guidelines, preventive care, guideline compliance,
standards and treatment prescriptions
benchmarks Dennis 1993 - nursing documentation Dennis 1993 – Chart audit
adherence to standards for
documentation and patient care Hailey 2003 – success/failure
Garg 2005 – performance in rates of assessment criteria
outpatient screening and monitoring, Hailey 2004 – administrative changes,
medication use, identification of at-risk patient outcomes, economic assessments
behaviors (table 2); performance and NIFTE 2003 telehealth framework of
outcomes in disease management and guidelines
health conditions (tables 3,4,5); NTOIP 2003 Telehealth outcome
performance and outcomes in policy indicators
adherence in resource utilization and
drug dosing (tables 6,7,8)
Kawamoto 2005 – performance in
compliance with protocols, reminders,
preventive care, screening
Mitchell 2001 – performance in
immunization rates, preventive tasks,
disease management and prescribing
Montgomery 1998 – blood pressure
control and prescribing
Effectiveness – Balas 1996 – outcomes as patient Hailey 2003 – success/failure

change in knowledge, attitudes, morbidity, and assessment criteria
practitioner physical and psychological measures Hailey 2004 – administrative changes,
performance and Balas 1998 – outcomes as clinical patient outcomes, economic assessments
patient outcomes endpoints in self-monitoring (e.g. NIFTE 2003 telehealth framework of
HbA1C) guidelines
Delpierre 2004 – performance in NTOIP 2003 Telehealth outcome
consultation pattern; outcomes in indicators
cardiovascular risk, survival,
hospitalization, disease management
(e.g. blood pressure, asthma, angina)
238

Kawamoto 2005 – performance in
change in workflow, decision making,
follow-up visits, test reduction
Mitchell 2001 – performance in pattern
of consultation; outcome as clinical
endpoints
van der Loo 1995 – performance in
practice pattern; outcomes in morbidity,
mortality
Patient safety – Allan 2000 - nursing care Allan 2000 - audit of implementation of
adverse events, documentation for risk management policies
surveillance and Kaushal 2003 – medication safety i.e.
risk management adverse drug events, medication errors,
corollary orders, drug-dosing and
prescribing practice
Kawamoto 2005 – adverse event
response
239
C.3 SUGGESTED EVALUATION MEASURES FOR NET BENEFITS DIMENSION –

ACCESS AND PRODUCTIVITY

Access Availability of services – NIFTE 2003 telehealth framework of
guidelines
NTOIP 2003 Telehealth outcome
indicators
Access to services - NIFTE 2003 telehealth framework of
guidelines
indicators
Patient/caregiver participation – NTOIP 2003 Telehealth outcome
Balas 1998 – self management of indicators
diabetes with clinical measurements
Productivity Efficiency
Allan 2000 - nursing charting time Allan 2000 - survey of users and
Balas 1996 – use of lab tests, managers
medications, hospital/ER admissions,
outpatient visits
Burkle 1999 - nursing documentation Burkle 1999 - observational time
time analysis
Maraovic 1997 - clinical information Marasovic 1997 - random activity
system: workflow pattern and activity sampling
frequency
Market 1997 - transmission of clinical
data Market 1997 - computerized chart
Mitchell 2001 – resource utilization review
Montgomery 1998 – follow-up visits
Ostbye 1997 - laboratory information
system with order entry: workflow and Ostbye 1997 - multiple methods
productivity changes including telephone log review, system
van der Loo 1995 – healthcare log review, interviews, test volume
consumption, time to review and survey
decision/intervention, patient time
Wang 1995 - computerized anesthesia Wang 1995 - time motion analysis
charting, user workflow and time saving
Weiner 1999 - inpatient POE system: Weiner 1999 - survey of system users
nurse and physician workflow and time Hailey 2004 – administrative changes,
savings patient outcomes, economic assessments
indicators
Care coordination Ammenwerth 2001 – survey, interviews,
Ammenwerth 2001 -nursing time measurement versus control group
documentation cooperation and planning Hailey 2004 – administrative changes,
time among team members patient outcomes, economic assessments
Mitchell 2001 – referral pattern, rates of NTOIP 2003 Telehealth outcome
visits and recalls indicators
Cost
Allan 2000 – incremental overtime Allan 2000 – incremental overtime
Balas 2001 – healthcare cost analysis
240

van der Loo 1995 – costs Hailey 2004 – administrative changes,
patient outcomes, economic assessments
indicators
Publication Sources for Examples Cited in Framework
Abenstein J, DeVos C, Abel M, Tarhan S. Eight year’s experience with automated anesthesia
record keeping: Lessons learned—New directions taken. Int J Clin Monit Comput 1992;
9:117–29.
Allan J, Englebright J. Patient-centered documentation: An effective and efficient use of clinical

information systems. J Nurs Admin 2000;30:90–95.
Ammenwerth E, Eichstadter R, Haux T, Pohl U, Rebel S, Ziegler S. Arandomized evaluation of

a computer-based nursing documentation system. Meth Inform Med 2001;40:61–-68.
Ammenwerth E, de Keizer N. An inventory of evaluation studies of information technology in

health care. Meth Inform Med 2004;44:44-56.
Balas AE, Boren SA, Griffing G. Computerized management of diabetes: a synthesis of

controlled trials. Proceedings of AMIA 1998.
Balas AE, Austin SM, Mitchell JA, Ewigman BG, Bopp KD, Brown GD. The clinical value of
computerized information services. Arch Fam Med 1996;5:271-278.
Bates D, Teich J, Lee J, Seger D, et al. The impact of computerized physician order entry on
medication error prevention. Obstet Gynecol 1999;6:313-21.
Bürkle T, Kuch R, Prokosch H-U, Dudeck J. Stepwise evaluation of information systems in an

university hospital. Meth Inform Med 1999;38:9–15.
Delpierre C, Cuzin L, Fillaux J, Alvarez M, Massip P, Lang T. A systematic review of computer-

based patient record systems and quality of care: more randomized clinical trials or a
broader approach? Intl J Qual Health Care 2004;15(5): 407-416.
Dennis K, Sweeney P, Macdonald L, Morse N. Point of care technology: impact on people and
paperwork. Nurs Econ 1993;11:229–37.
Garg AX et al. Effects of Computer-based clinical decision support systems on practitioner

performance and patient outcomes: A systematic review. JAMA 2005; 293(10):1223-1238.
241
Hayward RH. Clinical Decision Support Tools – Do They Support Clinicians? Presentation at
the University of Victoria health informatics seminar series on Dec 9/2005.
Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry and clinical
decision support systems on medication safety. Arch Intern Med 2003; 163:1409-1416.
Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical
decision support systems: A systematic review of trials to identify features critical to
success. BMJ 2005;330;765.
Larrabee J-H, Boldreghini S, Elder-Sorrells K, Turner Z-M, et al. Evaluation of documentation

before and after implementation of a nursing information system in an acute care hospital.
Comput Nurs 2001;19:56–65.
Lee F, Teich JM, Spurr CD, Bates DW. Implementation of physician order entry: User
satisfaction and self-reported usage patterns. JAMIA 1996;3:42–55.
Marasovic C, Kenney C, Elliott D, Sindhusake D. A comparison of nursing activities associated

with manual and automated documentation in an Australian intensive care unit. Comput
Nurs 1997; 15:205–11.
Markert DJ, Haney PJ, Allman RM. Effect of computerized requisition of radiology
examinations on the transmission of clinical information. Acad Radiol 1997;4:154–56.
Mitchell E, Sullivan F. A descriptive feast but an evaluative famine: systematic review of

published articles on primary care computing during 1980-97. BMJ 2001;322:279-282.
Montgomery AA and Fahey T. A systematic review of the use of computers in the management
of hypertension. J Epidemiol Community Health 1998;52:520-525.
Murphy CA, Maynard M, Morgan G. Pretest and post-test attitudes of nursing personnel toward
a patient care information system. Comput Nurs 1994;12:239–44.
Nielsen PE, Thomson BA, Jackson RB, Kosman K, Kiley KC. Standard obstetric record charting
system: evaluation of a new electronic medical record. Obstet Gynecol 2000;96:1003-1008.
Ostbye T, Moen A, Erikssen G, Hurlen P. Introducing a module for laboratory test order entry
and reporting of results at a hospital ward: an evaluation study using a multi-method
approach. J Med Sys 1997;21:107–17.
242
van der Loo RP, Gennip EMSJ, Bakker AR, Hasman A, Rutten FFH. Evaluation of automated
information systems in healthcare: An approach to classify evaluative studies. Comput
Methods Programs Biomed 1995;48:45–52.
Urschitz M, Lorenz S, Unterasinger L, Metnitz P, Preyer K, Popow C. Three years experience

with a patient data management system at a neonatal intensive care unit. J Clin Monit
1998;14:119–25.
Wang X, Gardner RM, Seager PR. Integrating computerized anesthesia charting into a hospital
information system. Int J Clin Monit Comput 1995;12:61–70.
Weiner M, Gress T, Thiemann DR, Jenckes M, et al. Contrasting views of physicians and nurses
about an inpatient computer-based provider order-entry system. JAMIA 1999; 6:234–44.
243
APPENDIX D: OTHER KEY CONTRIBUTORS
Expert Advisory Panel
Provided regular input into process validation and validated proposed individual program
evaluation plans.
Anne McFarlane................................................ CIHI

Michael Wolfson............................................... Statistics Canada
Doreen Neville .................................................. Memorial University
David Bates....................................................... Partners Healthcare
Robyn Tamblyn ................................................ McGill University
Nikki Shaw........................................................ University of Alberta
Richard Scott..................................................... University of Calgary
Francis Lau........................................................ University of Victoria
Infoway BE Working Group
Project management of BE plan development, coordinate activities with internal and external
stakeholders and develop implementation plan
Dr. Sarah Muttitt ............................................... Acting VP, Innovation and Adoption, Infoway
Simon Hagens ................................................... Manager, Benefits Strategy and Analysis,
Infoway
Mark Nenadovic................................................ Director, Change and Evaluation Services,
Infoway
Nancy Kraetschmer........................................... Infoway
Dan Gordon....................................................... Consultant, Praxia
Adam Topp ....................................................... Consultant, HayGroup
Deana Leicht ..................................................... Consultant, Praxia
244

Creating Healthly Connection

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Creating Healthly Connection

Uploaded by

Copyright:

Available Formats

CREATING HEALTHY CONNECTIONS

CANADA HEALTH INFOWAY

Praxia Information Intelligence

Subject matter experts who developed the program specific plans:

Diagnostic Imaging Program........................................... Don MacDonald

Anne McFarlane .............................................................. CIHI

6.4 PROVIDER EFFICIENCY AND EFFECTIVENESS IN EMERGENCY DEPARTMENTS ...........................................92

1 INTRODUCTION TO CANADA HEALTH INFOWAY

1.1 PURPOSE OF THIS DOCUMENT

The purpose of this document is to:

• Describe the Infoway Benefits Evaluation Framework

1.2 PROCESS FOR DEVELOPING THE BENEFITS EVALUATION FRAMEWORK

• Developed an approach to Infoway EHR Impact Assessment

Based on the comprehensive measurement framework, an implementation plan was defined

• High level implementation plan

1.3 INDICATOR DOMAIN EXPERTS

Diagnostic Imaging Program........................................... Don MacDonald

1.4 INFOWAY BENEFITS EVALUATION FRAMEWORK

• Information quality includes such criteria as completeness, ease of understanding and

Dimension Category Subcategories and/or Definition

Dimension Category Subcategories and/or Definition

2.1 DIAGNOSTIC IMAGING PROGRAM INDICATORS SUMMARY

Category & Study Question Indicator Measures

Change in With the implementation of Radiologist Time required by the radiologist to

Indicator Category and Sub Category:

Indicator Setting and Population:

Data Sources and instrument:

Indicator Description: Measure #1

Indicator Category and Sub Category:

Indicator Setting and Population:

Data Sources and Instrument:

%∆ = (B/D – A/C) x 100

Indicator Description: Measure #2

Indicator Category and Sub Category:

Indicator Setting and Population:

The survey, primarily utilizing Likert scales, would focus on:

Data Sources and instrument:

Indicator Category and Sub Category:

Indicator Setting and Population:

Data Sources and Instruments:

Indicator Category and Sub Category:

Indicator Setting and Population:

Data Sources and Instrument:

%∆ = (B/D – A/C) x 100

Indicator Category and Sub Category:

Indicator Setting and Population:

The survey, primarily utilizing Likert scales, will focus on:

Data Sources and instrument:

Indicator Category and Sub Category:

Indicator Setting and Population:

A pilot study in one high DI volume site is recommended.

Strickland N. (2000). PACS (Picture Archiving and Communications Systems): Filmless

3.1 DRUG INFORMATION SYSTEM PROGRAM INDICATORS SUMMARY

Category & Sub- Study Question Indicator Measures

Database assessment of hospital-

Indicator Category and Sub Category:

Indicator Setting and Population:

Data Sources and instruments:

Indicator Category and Sub Category:

Indicator Setting and Population:

Data Sources and instruments:

Indicator Category and Sub Category: