International Journal of Medical Informatics 152 (2021) 104459

Contents lists available at ScienceDirect

International Journal of Medical Informatics

journal homepage: www.elsevier.com/locate/ijmedinf

Usability evaluation of a nursing information system by applying cognitive

walkthrough method
Mehrdad Farzandipour a, b, Ehsan Nabovati a, b, Hamidreza Tadayon b, c,
Monireh Sadeqi Jabali b, *
a
Research Centre for Health Information Management, Kashan University of Medical Sciences, Kashan, Islamic Republic of Iran
b
Department of Health Information Management & Technology, Kashan University of Medical Sciences, Kashan, Islamic Republic of Iran
c
Department of Health Information Technology, Neyshabur University of Medical Sciences, Neyshabur, Islamic Republic of Iran

A R T I C L E I N F O A B S T R A C T

Keywords: Background and objective: The user interface usability of the nursing information system (NIS) should be such that
Hospital information system the nurses can learn and interact with it easily and quickly. Therefore, it is necessary to identify and solve the
Nursing information system usability problems of these systems. The present study aimed to evaluate the usability of a NIS using the cognitive
Usability
walkthrough (CW) evaluation method.
Cognitive walkthrough
Methods: Based on five selected scenarios, five evaluators evaluated the NIS in Shafa Hospital Information Sys­
User interface
tem. After identifying the problems, the evaluators assigned each problem to one of the usability attributes. The
severity of each identified problem was determined by the evaluators and five real users of the system.
Results: In total, 24 unique problems were identified. The average severity of the problems was determined by the
evaluators (2.77) and the actual users (2.82) in the "major problem" category. The highest number of problems
were assigned to the scenarios 3 and 2 with 15 and 14 problems, respectively. The highest average severity in
terms of evaluations and actual users was related to the scenario 5 (3.06 and 2.94, respectively), which was in the
"major problem" category. The highest number of problems were associated with learnability (8 problems) and
efficiency (6 problems).
Conclusion: Since most of the nurses do not have enough time for learning the system, and given that the selected
scenarios for evaluation were based on the most frequent daily tasks that nurses performed using the NIS; The
correction of usability problems in the scenarios 3 and 2 that had the highest number of problems and the
scenario 5 that had the highest average severity and execution time; it can reduce the nurses’ cognitive load and
the learning time of the system and increase the efficiency of nurses.

1. Introduction
Today, in many medical centers, nursing information systems (NISs)
have been implemented as one of the key elements of a hospital infor­
mation system (HIS) [1,2]. This system, which combines computer sci­
ence, information science and nursing science, has been designed to
support nurses’ duties and nursing care [3]. Nurses are required to use
these systems in doing their daily tasks [4]. Thus, they play an important
role in the acceptance of the system [5,6]. Therefore, it is necessary to
understand the factors that affect nurses’ use of NIS [7].
Usability is one of the factors affecting the acceptance of health in­
formation systems by users [8–10], and it is a critical component for safe
and effective use of information technology [11]. In a study by Topaz
et al. [12], it was also shown that almost one-third of nurses were
concerned about the system’s usability problems. Usability is defined as
the degree to which users can accurately and efficiently perform tasks
using the system [13]. The International Organization for Standardiza­
tion (ISO) defined usability as efficiency, effectiveness, and satisfaction
[14]. Also, Nielsen described attributes of learnability, efficiency,
memorability, error prevention, and satisfaction as usability attributes

Abbreviations: NIS, nursing information system; HIS, hospital information system; CW, cognitive walkthrough; EMR, electronic medical record; ISO, International
Organization for Standardization; HER, electronic health record; ICDL, International Computer Driving License.
* Corresponding author at: Qutbe ravandi blvd, Kashan University of Medical Sciences, Department of Health Information Management & Technology, Islamic
Republic of Iran.
E-mail address: msadeqi2005@gmail.com (M. Sadeqi Jabali).

https://doi.org/10.1016/j.ijmedinf.2021.104459
Received 11 August 2020; Received in revised form 5 March 2021; Accepted 6 April 2021
Available online 10 April 2021
1386-5056/© 2021 Published by Elsevier B.V.
M. Farzandipour et al.
[15].
Usability evaluations can detect problems and flaws in user interface
design in the early phase and also throughout the cycle of system
development [16]. There are several methods to evaluate system us­
ability [17], among which Cognitive Walkthrough (CW) is an
expert-based one in which user interface is evaluated by analyzing the
mental processes users are required to perform [18,19]. This method can
identify important usability problems in an interface design easily,
quickly and at low cost [20] and also identify issues existing in system
learnability for novice users [21].
Nurses, as the main users of NISs, do not have enough time to learn
how to use the system and are required to learn how to work with the
system while doing their duties [22]. In a study conducted by Berhe et al.
[23], 37 % of nurses were not satisfied with the ease of learning elec­
tronic medical record (EMR), more than half of whom stated that they
had not received training on using the system. The results of CW eval­
uation are more applicable in identifying learnability problems and are
useful for systems developers.
Shafa HIS, which is a product of Tirajeh Rayaneh company, has been
implemented in more than 200 hospitals in a developing country (Iran).
Therefore, this study was conducted to evaluate the usability of the NIS
as a subsystem that has the largest group of users in this HIS through CW
method. The identified usability problems were also categorized based
on a combination of ISO and Nielsen usability attributes.
2. Materials and methods
Usability evaluation of the NIS of Shafa HIS was performed in the
Health Information Technology Laboratory at Kashan University of
Medical Sciences using CW evaluation method.
2.1. Cognitive walkthrough evaluation
CW is a task-based, structured evaluation method that follows a
linear path [20]. Focusing on system learnability for novice users, this
method is based on theories of exploratory learning developed by Lewis
et al. [24]. In this method, one evaluator or a group of evaluators sim­
ulates users’ cognitive processes when following the sequence of actions
performed to accomplish certain tasks based on users’ background
knowledge. Therefore, the quality of results heavily depends on how
well the evaluators were able to put themselves in users’ position [20].
The evaluator analyzes the user interface and examines the ease of each
step for novice users [24,25].
2.2. System
The NIS of Shafa HIS, which is a product of Tirajeh Rayaneh com­
pany, was selected for evaluation in this study. At the time of conducting
the present study, Shafa HIS was implemented in more than 200 hos­
pitals in Iran and was one of the most widely used HISs in the country.
2.3. Target users
NIS users are nurses and secretaries in hospital wards. They include
technicians having bachelor’s and master’s degrees and having
completed International Computer Driving License (ICDL) courses.
2.4. Identification of scenarios
Five scenarios were selected based on the most important and
frequent daily tasks of nurses and secretaries of inpatient wards where
the NIS was used. These scenarios were identified based on the opinion
of three nurses working in the inpatient ward and were then approved
by the head nurse, including admitting patients and allocating bed and
room for patients in the inpatient ward; recording a request for surgery;
recording a request for paraclinical services and monitoring the results;
2
International Journal of Medical Informatics 152 (2021) 104459
recording patient transfer to another ward and recording patient
discharge information from the inpatient ward. For each scenario, the
goals and sub goals of users, the sequence of actions to perform specific
tasks and system responses were all defined. Tables 1 and 2 show the
number of tasks and actions of each scenario and an example of a
scenario.
2.5. Evaluators
Since 3–5 evaluators are adequate to carry out the CW evaluation
[26], this study was conducted with the participation of five evaluators
who were selected randomly and based on aim. The evaluators included
three Ph.D students in health information management and two M.Sc
students in health information technology who had previous experience
in performing the CW evaluation, each of whom, due to the nature of
their field of study, had work experience or was familiar with various
health care information systems; however, unlike the nurses, they were
not the NIS users.
2.6. Data collection and analysis method
In order to perform the evaluation, evaluators independently per­
formed the sequence of actions of each scenario using the NIS user
interface. Thus, evaluators independently put themselves in real users’
position based on users’ background knowledge. Evaluators were able to
report back if a problem arose afterwards to accomplish a task based on
users’ point of view. The researcher, acting as an observer, along with
evaluators, was present at the time of evaluation and took some notes on
data collection forms regarding evaluators’ comments, questions, and
ambiguities, detailed explanations of usability problems recognized by
evaluators and the exact location of usability problems in the system.
Once an evaluation process was completed, each evaluator reviewed the
list of mentioned usability problems and added or corrected a comment
if required. Then, the five lists obtained from evaluators were compared
in a meeting and all the recognized problems were added to the list of
main problems while repeated problems were discarded. Finally, eval­
uators were asked to independently determine the severity of each
problem on a scale of 0–4, based on the frequency of problem occur­
rence, the potential impact of the problem on users, and the persistence
of the problems. Problem severity was graded as zero: no problem, 1:
cosmetic problem, 2: minor problem, 3: major problem and 4: catas­
trophe problem [27].
Furthermore, in this study, five nurses, who were the real users of the
system, determined the severity of identified problems. For this purpose,
the problem determination form including the identified problems along
with a picture of the problem location in the system was provided to the
nurses. Then, these nurses were asked to independently determine the
severity of each problem as requested by evaluators on a scale of 0–4.
In order to present the type of identified problems based on usability
attributes of ISO and Nielsen [14,15], each evaluator independently
identified usability problems using the form of determining usability
Table 1
Number of tasks and actions of each scenario in the NIS.
Scenario Scenario description Number of Number of
number tasks actions
1 Admitting patients and allocating bed 3 9
and room for patients in the inpatient
ward
2 Recording surgery request 3 15
3 Recording a request for paraclinical 3 14
services and monitoring the results
4 Recording patient transfer to another 2 8
ward
5 Recording patient discharge 4 23
information from the inpatient ward
M. Farzandipour et al. International Journal of Medical Informatics 152 (2021) 104459

Table 2 which 24 unique problems remained after combining the problems and
An example of a scenario and the associated tasks and actions. eliminating the repeated ones. The list of unique identified problems
Scenario Tasks Actions based on usability attributes is shown in Table 3.
As shown in Table 4, the total average severity from the perspective
Click on ‘orthopedic ward’.
Icons and patients’ names in of evaluators and real users was 2.77 and 2.82, respectively. Moreover,
orthopedic ward are displayed. the Mann-Whitney U test indicated no significant difference according
Selecting a
patient
Right click on the desired to these figures (Z=–0.437, P = 0.662). The highest number of problems
patient’s icon. was related to the scenarios 3 and 2 with 15 and 14 problems, respec­
The drop-down menu next to
the patient icon is displayed.
tively. The highest average severity from the perspective of evaluators
Click on "Request Services" and real users was that of scenario 5 with an average severity of 2.94 and
option from the drop-down 3.06, respectively, which was in the category of major problems. This
menu. scenario also had the highest average execution time (i.e., 3:07 min). As
The service request window is
shown in Fig. 1, the highest average severity determined by evaluators
displayed for the desired
patient. and real users based on usability attributes was related to error (3.1 and
Select the laboratory option in 3) and system learnability (2.93 and 3.1), respectively.
Requesting a CBC test for a
the "Submit Request" field.
patient in the orthopedic
Entering a Select CBC from the test name in 4. Discussion
ward
request for a CBC drop-down menu.
test Click on the check mark to add
the CBC test to the list of patient The aim of this study was to describe the results of usability evalu­
tests. ation in one NIS by CW method. In this study, 24 usability problems
The CBC test is displayed in the were identified in the most important daily tasks of nurses performed via
list of requested tests for the
the NIS. No significant difference was observed between the average
patient.
Click on the "Save" button. severity of identified problems by evaluators and real system users and
The “Patient Paraclinical they both categorized the average severity of these problems as major
Services” window is displayed. problems. It showed that evaluators had the right view of users and were
Click on "Return" to previous able to put themselves in real users’ position. Moreover, the classifica­
Return to
window.
inpatient ward
Inpatient ward window is
tion of identified problems based on ISO and Nielsen usability attributes
window showed that the highest number of identified problems was associated
displayed.
with system learnability.
Based on the findings of this study, the highest number of identified
attributes. The definitions of usability attributes of ISO and Nielsen are problems was observed in scenarios 3 and 2 (recording a request for
as follows: paraclinical services and monitoring the results; recording a surgery
request), which were in the major problem category. Compared to other
- Effectiveness: The extent to which users reach the goals set through scenarios, scenario 5 (recording patient discharge information from the
the system. inpatient ward) had the highest average execution time. The problems
- Efficiency: How many resources and how much of each is required so that were jointly identified in most scenarios are discussed in the
that the pre-set goals can be achieved by users? following paragraphs.
- Satisfaction: How satisfied are users when they use the system?
- Learnability: To what extent is it convenient for users to fulfill 4.1. Items are not grouped together logically
fundamental tasks during the first occasion on which they use the
system? This problem, which was jointly reported in scenarios 3, 2 and 5, was
- Memorability: How well can users remember how to use the system assigned to efficiency attribute. Options and items that are logically
after a while when they have not used it? connected should be grouped together on the screen effectively [29]. For
- Errors: The number of errors made their severity and the ease of this purpose, white space between collections of items or boxes around
recovery from errors. them can be used to group them together [30]. Adherence to this prin­
ciple in the NIS user interface leads to a decrease in nurses’ cognitive
The session for determining the severity and type of problems based load and prevents them from wasting time in searching for options and
on usability attributes by the evaluators lasted for 5 h. items on the screen. Grouping of items and options on the screen needs
The collected data were analyzed using descriptive and inferential to be understandable and meaningful and based on their order and
statistical methods by SPSS Statistics for Windows, version 20.0 (SPSS priority.
Inc., Chicago, Ill., USA). The average severity of the identified problems
was determined. Each problem was then classified based on its average 4.2. There is no proper feedback for completing required fields
severity in one of the five following categories: 0− 0.5: no problem,
0.6–1.5: cosmetic problem; 1.6–2.5: minor problem, 2.6–3.5: major This problem, which was jointly shown in scenarios 3, 2 and 5, was
problem and 3.6− 4: catastrophe problem [28]. Furthermore, each assigned to efficiency attribute. Required fields to complete in system
problem was attributed to the usability attribute in which it had the user interface must be clearly visible at first sight of the optional fields,
highest frequency. Provided that an identified problem had equal fre­ which can be marked by a background color or using an asterisk (*) in
quency in more than one usability attribute, the problem was assigned to the fields and options. Properly highlight required fields reduce form
the most appropriate usability attribute based on experts’ opinion. The completion time [31] and play an important role in efficiency and pre­
average severity of the problems identified by evaluators and real users vention of users’ errors [32]. However, one of the problems in system
(nurses) was compared using the Mann-Whitney U test user interface is lack of an indicator to identify mandatory fields or
feedback if such fields are not completed [33]. Graham et al. [34] stated
3. Results that, if there is no feedback provided to users that a form is incomplete,
in most cases, users will not be able to detect them due to workload. In
In this study, five evaluators analyzed NIS usability using CW relation to the NIS, due to working conditions of nurses, system user
method and five scenarios. A total of 46 problems were identified, of interface should be designed in such a way that required fields can be

3
M. Farzandipour et al. International Journal of Medical Informatics 152 (2021) 104459

Table 3 Table 3 (continued )

The identified problems based on usability attributes. Usability Identified problems Evaluators’ comments Scenario
Usability Identified problems Evaluators’ comments Scenario attributes number
attributes number
similar throughout the
Some items are not system.
Group logically related
grouped together 2,3,5 Some message Design message windows 2,4
items together.
logically. windows are displayed in the same place.
There are too many in unexpected places.
steps that must be Combine windows and Some icons are Design icons in expected 3,5
2,5
taken to complete the delete a number of steps. displayed in location for users (for
task. unexpected locations. example, save icon in the
Screens are crowded bottom left corner of the
Delete or disable the
and there are duplicate screen).
options which do not 2,5
and non-applicable Data can be entered in Disable the apparently 3
need to be completed.
fields. apparently inactive inactive fields.
There is no back Include the back button fields.
Efficiency
button in some in all windows in the 2,3 The symbol used for Use a clear and
windows. same way. patients awaiting recognizable symbol to
5
There is no icon to discharge is not show patients waiting
Design the minimize and
minimize the recognizable. discharge.
maximize icons in all 3 Error
paraclinical service Provide feedback to the
windows. The values outside the
window. user if values outside the
normal range can be 3
1. Provide clear feedback normal range are
recorded in the fields.
There is no proper to guide the user to fill entered.
feedback for incomplete fields.
2,3,5
completing the 2. Highlight the
required fields. incomplete fields on the
screen.
Items in both Persian Use a single language for Table 4
and English are used in all items of drop-down 2,3 Number and average severity of problems and average time based on scenarios.
Effectiveness the drop-down menus. menus.
Overlapping screens Specify the active Scenarios Average Number Average Average
2,3
are not appropriate. window for users. time to of severity - severity -
There is empty space perform problems problem problem
Fill the empty spaces by
on the inpatient 1,3 each category category
useful options for users.
screen. scenario determined by determined by
Only the first letter of (minute) evaluators real users
Satisfaction Capital letters are used
drug names and
in the list of Admitting 01:58 5 2.72 (Major) 2.68 (Major)
paraclinical services need 3
paraclinical services patients and
to be displayed in capital
(drug names). allocating bed
letters.
and room for
Some icons do not
Use icons with patients in the
match their functions
understandable 1,3,5 inpatient ward
(Users are not familiar
meanings for users. Recording a 02:31 14 2.71 (Major) 2.77 (Major)
with icons).
surgery
The method of
request
allocating room and Use the expected method
Recording a 01:10 15 2.73 (Major) 2.8 (Major)
bed for patients is to allocate room and bed 1
request for
unexpected for the for patients.
paraclinical
user (dragging).
services and
Service code, service
monitoring
name, and service
Use names for all screen the results
number fields in 3
fields. Recording 01:02 2 2.4 (Minor) 2.5 (Minor)
recording paraclinical
patient
services are unnamed.
transfer to
The names and Use understandable
another ward
functions of some terminology and
2,5 Recording 03:07 10 2.94 (Major) 3.06 (Major)
Learnability menu items are expressions in menu
patient
unclear for users. items.
discharge
Some graphic symbols
Pair unfamiliar icons information
and icons on the screen 1,2,3,5
with labels. from the
are not labelled.
inpatient ward
Use colors according to
Colors do not match Total 01:57 46 2.77 (Major) 2.82 (Major)
commonly known rules
common expectations 2
(for example, red color
about color codes.
for errors).
The same language is
marked from optional fields. It is obligatory for them to observe proper
Use the same language feedback on incomplete fields, which, while preventing confusion of the
not used for button 2
for all labels.
labels. nurses, helps them to correct errors more quickly and effectively.
Screens have no titles
or their titles are
Place titles in the right
placed in an 1,2,4,5 4.3. Icons do not match their functions
place for all screens.
unexpected place for
the user. This problem, which was jointly identified in scenarios 1, 3 and 5,
Design of buttons
Memorability throughout the system
Design the buttons in
2,3
relates to system learnability. The effectiveness of an icon in relation to
such a way that they are its meaning depends on the degree of compatibility between physical
is not uniform.
shape of that icon and its function; it must convey the meaning of the
function it represents, without the use of text [35]. Ideally, icons should

4
M. Farzandipour et al.
Fig. 1. The number and severity of problems based on the usability attributes.
activate appropriate mental models in users because how the user in­
terprets the icon will depend on user’s mental models [36]. Therefore,
system designers can minimize user’s memory load by using meaningful
and easy-to-understand icons. In the study conducted by Darbyshire
[37], users cited lack of intuitive icons in the electronic health record
(EHR) as one of the reasons of system’s time-consuming use. However,
poor icon design leads to misinterpretation of them by users. Over time,
as users gain experience with a system, the performance difference be­
tween familiar and unfamiliar icons diminishes significantly [38]. In
designing NISs with user interfaces having familiar icons, learning the
system becomes easy for nurses; however, poor design of icons and se­
mantic distance between the physical shape of icons and their meaning
diminishes once nurses have gained considerable experience with icons.
4.4. Graphic symbols and icons are not labelled
This problem, which was jointly reported in scenarios 1, 2, 3 and 5,
was attributed to system learnability. In the studies performed by Farrhi
et al. [39] and Guo et al. [40], lack of clear and proper labels in user
interface items was reported as a usability problem. In another study
done by Atashi et al. [41], using clear labels was proposed as a strategy
to solve system usability problems. Well-designed labels are 90 % more
efficient in finding items and require 25 % fewer clicks to complete tasks
[42]. In addition to the need for labels in icons, buttons and graphic
symbols, user interface designers are advised not to use ambiguous
terminology in labels. Familiar terminology used in labels increases
users’ knowledge of performance of icons [14,43]. In this study, due to
lack of icons familiar for users, system designers are advised to add
meaningful and appropriate text labels in order to reduce ambiguity and
nurses’ memory load and facilitate system learnability.
4.5. Screens have no titles or their titles are shown in an unexpected place
This problem, which was reported jointly in scenarios 1, 2, 4 and 5, is
related to system learnability. According to the rules of user interface
design, all system windows, except message windows, should have a
title that clearly and concisely indicates the purpose of the window [42].
In the study done by Wiklund et al. [44], large and meaningful screen
titles were expressed as one of EHR user interface aspects in order to
improve user satisfaction. However, untitled screens should also be
understandable to nurses and nurses may not pay attention to the title
screen over time and gain experience working with the system. For
novice nurses, the title indicates displayed window performance and is
helpful in system learnability.
According to the obtained results, the highest number of identified
problems was associated with system learnability and efficiency based
on usability attributes. In the study conducted by Khajoui et al. [45]
5
International Journal of Medical Informatics 152 (2021) 104459
performing CW method in an evaluation of a medical office management
system, the largest number of problems was associated with system
learnability and memorability. Given that CW method focuses on iden­
tifying learning problems, it is convincing that most of the identified
problems in these two studies will be attributed to system learnability. In
the present study, the second largest number of problems belonged to
efficiency. This finding was not in line with that of the study done by
Khajouei et al. [45], showing that the second highest number of prob­
lems belonged to memorability. This inconsistency might be caused by
the difference in the nature of problems identified by the systems
evaluated in these two studies.
According to the results, the highest average time was observed in
scenario 5. This can be explained by the higher number of the actions
performed in this scenario compared to other scenarios. To perform this
scenario, the nurses must complete patient discharge form with 29 data
elements. Some data elements in this form such as weight, height, blood
group, allergies, safety index, patient complaint, infections and burn
must be completed at patient’s admission time. Therefore, presence of
all these elements in patient discharge form does not seem to be
appropriate and is time-consuming for users. It is recommended that the
data elements of this form as well as other data entry forms be reviewed
in terms of necessity of data elements and their correct placement on
these screens.
5. Strengths and weaknesses
This study was conducted with participation of five evaluators,
which is the maximum number of evaluators in a CW evaluation. Also, in
this study, considering that CW focuses on system learnability, the
severity of problems was determined by real users of NIS, which is one of
the strengths of this study. Another point of strength in the present study
was its implementation on the NIS of one of the most widely used
implemented HISs in a developing country. A significant number of
nurses in hospitals are service commitment staff, who do not receive
training by the NIS vendor. They are essentially required to learn how to
work with the system through their colleagues and while doing their
duties. Therefore, learnability plays an important role for them and
system designers can use the results of this study to improve user
interface usability and also increase its learnability for novice nurses.
On the other hand, the study has its limitations. It was conducted on
a single NIS, in an HIS. Since several vendor companies in the country
offer HISs, the results of the present study cannot be fully generalized to
all these systems. Hence, it is recommended that an NIS be evaluated in
other HISs implemented in the country by applying CW method in order
to identify usability problems of these systems.
M. Farzandipour et al.
6. Conclusion
Since the training by companies supplying NISs is provided to nurses
only in the system implementation phase, for many nurses who will be
the future users of the system in the maintenance phase, there is not
enough time to do learn how to use the system properly. These nurses
are required to learn how to work with the system while doing their
duties. Therefore, it will also be useful to perform CW method on a fully
implemented system.
In this study, the selected scenarios for evaluation were based on the
most frequent daily tasks that nurses performed using the NIS. Thus,
correction of usability problems in two scenarios – “recording a request
for paraclinical services and monitoring the results” and “recording a
surgery request” - which had the highest number of problems, and also
the scenario of "recording patient discharge information from the
inpatient ward", which had the highest average severity and execution
time, are recommended to be considered by designers and developers of
NISs.
In general, most of the learnability problems identified by evaluators
and real users (nurses) were in the major problem category. Therefore,
based on the identified problems, redesigning the system user interface
using familiar icons and graphic symbols to users, performing an action
according to user’s expectations, defining an appropriate name for all
data entry fields, using clear and concise terminology in menu options,
adding labels to the icons unfamiliar to users, using colors according to
commonly known rules, using the same language for all labels, and
defining an appropriate title for screens and data entry forms are among
the improvements that can reduce nurses’ memory load and system
learning time and increase nurses’ efficiency.
Authors’ contributions
MF made substantial contributions to the conception, drafting and
design as well as acquisition of funding. HT and MSJ participated in data
collection and performed the statistical analysis. EN and MSJ contrib­
uted to manuscript drafting, revision and approval and MF was in charge
of general supervision of the research group.
Data statement
The datasets used and analysed during the current study are avail­
able from the corresponding author on reasonable request.
Funding
The present research was supported by the Research Council of
Kashan University of Medical Sciences [grant number: 97031] and did
not receive any grants from nonprofit organizations and funding
agencies in the public and commercial sectors.
Summary Points
What was already known on the topic:
- The daily work of nurses with the NIS can be demonstrated using a
number of scenarios, tasks and actions.
- Usability problems can be classified using usability attributes.
- Currently, it is unknown what usability attributes are related to the
identified problems by CW in an NIS.
What this study added to our knowledge:
- By using CW, usability problems related to learnability and effec­
tiveness attributes can be better identified.
6
International Journal of Medical Informatics 152 (2021) 104459
- More than half of the identified problems are reported in more than
one scenario. Therefore, using the CW, similar problems can be
identified in different scenarios.
- The scenario with the highest number of actions had the highest
average severity and execution time.
- CW is an appropriate method in identifying problems with major
severity.
Declaration of Competing Interest
The authors report no declarations of interest.
Acknowledgments
The authors would like to express their gratitude to the esteemed
Vice-Chancellor of Kashan University of Medical Sciences and the
management of Health Information Technology Laboratory at Kashan
University of Medical Sciences and the evaluators and nurses who
participated in usability evaluation.
Appendix A. Supplementary data
Supplementary material related to this article can be found, in the
online version, at doi:https://doi.org/10.1016/j.ijmedinf.2021.104459.
References
[1] M. Farzandipour, Z. Meidani, H. Riazi, Ms Jabali, Nursing information systems
requirements: a milestone for patient outcome and patient safety improvement,
Comput. Inform. Nurs. 34 (12) (2016) 601–612, https://doi.org/10.1097/
CIN.0000000000000260.
[2] M. Malliarou, S. Zyga, Advantages of information systems in health services, Sport
Manage. Int. J. 5 (2) (2009) 43–54.
[3] N. Staggers, C.B. Thompson, The evolution of definitions for nursing informatics: a
critical analysis and revised definition, J. Am. Med. Inform. Assoc. 9 (3) (2002)
255–261, https://doi.org/10.1197/jamia.m0946.
[4] A. Tubaishat, Perceived usefulness and perceived ease of use of electronic health
records among nurses: application of technology acceptance model, Inform. Health
Soc. Care 43 (4) (2018) 379–389, https://doi.org/10.1080/
17538157.2017.1363761.
[5] H. Barzekar, F. Ebrahimzadeh, J. Luo, M. Karami, Z. Robati, P. Goodarzi, Adoption
of hospital information system among nurses: a technology acceptance model
approach, Acta Inform. Med. 27 (5) (2019) 305–310, https://doi.org/10.5455/
aim.2019.27.305-310.
[6] K. Kimiafar, F. Sadoughi, A. Sheikhtaheri, M. Sarbaz, Prioritizing factors
influencing nurses’ satisfaction with hospital information systems: a fuzzy analytic
hierarchy process approach, Comput. Inform. Nurs. 32 (4) (2014) 174–181,
https://doi.org/10.1097/CIN.0000000000000031.
[7] H.-C. Lin, Nurses’ satisfaction with using nursing information systems from
technology acceptance model and information systems success model perspectives:
a reductionist approach, Comput. Inform. Nurs. 35 (2) (2017) 91–99, https://doi.
org/10.1097/CIN.0000000000000293.
[8] M. Farzandipour, Z. Meidani, H. Riazi, M. Sadeqi Jabali, Task-specific usability
requirements of electronic medical records systems: lessons learned from a national
survey of end-users, Inform. Health Soc. Care 43 (3) (2018) 280–299, https://doi.
org/10.1080/17538157.2017.1290639.
[9] M. Farzandipour, H. Riazi, M.S. Jabali, Proposing electronic health record usability
requirements based on enriched ISO 9241 metric usability model, Acta Inform.
Med. 26 (1) (2018) 29–34, https://doi.org/10.5455/aim.2018.26.29-34.
[10] T.T. Lee, T.Y. Lee, K.C. Lin, P.C. Chang, Factors affecting the use of nursing
information systems in Taiwan, J. Adv. Nurs. 50 (2) (2005) 170–178, https://doi.
org/10.1111/j.1365-2648.2005.03376.x.
[11] I. Cho, E. Kim, W.H. Choi, N. Staggers, Comparing usability testing outcomes and
functions of six electronic nursing record systems, Int. J. Med. Inform. 88 (2016)
78–85, https://doi.org/10.1016/j.ijmedinf.2016.01.007.
[12] M. Topaz, C. Ronquillo, L.-M. Peltonen, L. Pruinelli, R.F. Sarmiento, M.K. Badger,
S. Ali, A. Lewis, M. Georgsson, E. Jeon, Nurse informaticians report low satisfaction
and multi-level concerns with electronic health records: results from an
international survey, AMIA Annu. Symp. Proc. 2016 (2017) 2016–2025.
[13] C. Johnson, D. Johnston, P. Crowle, EHR Usability Toolkit: A Background Report
on Usability and Electronic Health Records, Agency for Healthcare Research and
Quality, Rockville, MD, 2011.
[14] International Organization For Standardization, ISO 9241-11: Ergonomic
Reqirements for Office Work With Visual Display Terminals (VDTs): Guidance on
Usability Specifications and Measures, 1998.
[15] J. Nielsen, Usability 101: Introduction to Usability, Nielsen Norman Group, 2012.
[16] N. Staggers, M. Troseth, G. Alexander, C. Parker, P. Rogers, K. Smith, D. Tyler,
Designing Usable Clinical Information Systems: Recommendations From the TIGER
M. Farzandipour et al.
Usability and Clinical Application Design Collaborative Team, Technology
Informatics Guiding Education Reform (TIGER), 2009.
[17] Y. Rogers, H. Sharp, J. Preece, Interaction Design: Beyond Human-computer
Interaction, 3rd ed., John Wiley & Sons, New York, 2011.
[18] C. Lewis, C. Wharton, Cognitive walkthroughs. Handbook of Human-computer
Interaction, 2nd ed., Elsevier, 1997.
[19] B.C. Zapata, A.H. Niñirola, A. Idri, J.L. Fernández-Alemán, A. Toval, Mobile PHRs
compliance with android and iOS usability guidelines, J. Med. Syst. 38 (8) (2014)
81, https://doi.org/10.1007/s10916-014-0081-6.
[20] L.-O. Bligård, A.-L. Osvalder, Enhanced cognitive walkthrough: development of the
cognitive walkthrough method to better predict, identify, and present usability
problems, Adv. Hum. Interact. 2013 (2013), https://doi.org/10.1155/2013/
931698.
[21] M.W. Jaspers, A comparison of usability methods for testing interactive health
technologies: methodological aspects and empirical evidence, Int. J. Med. Inform.
78 (5) (2009) 340–353, https://doi.org/10.1016/j.ijmedinf.2008.10.002.
[22] M. Berge, M. Fossum, A. Fruhling, A cognitive walkthrough and focus group study
of nursing personnel to improve EHRs used in nursing homes, in: Scandinavian
Conference on Health Informatics, Linköping, Sweden, Linköping University
Electronic Press, 2012, pp. 13–18. October 2-3 2012.
[23] M. Berhe, K. Tadesse, G. Berhe, T. Gebretsadik, Evaluation of electronic medical
record implementation from user’s perspectives in Ayder Referral Hospital
Ethiopia, J. Health Med. Inform. 8 (249) (2017) 2, https://doi.org/10.4172/2157-
7420.1000249.
[24] C. Lewis, P.G. Polson, C. Wharton, J. Rieman, Testing a walkthrough methodology
for theory-based design of walk-up-and-use interfaces, in: Proceedings of the
SIGCHI Conference on Human Factors in Computing Systems, ACM, 1990,
pp. 235–242, https://doi.org/10.1145/97243.97279.
[25] C. Wharton, J. Rieman, C. Lewis, P. Polson, The cognitive walkthrough method: a
practitioner’s guide, in: J. Nielsen, R.L. Make (Eds.), Usability Inspection Methods,
John Wiley & Sons, New York, 1994, pp. 105–140.
[26] R. Khajouei, S.H. Gohari, M. Mirzaee, Comparison of two heuristic evaluation
methods for evaluating the usability of health information systems, J. Biomed.
Inform. 80 (2018) 37–42, https://doi.org/10.1016/j.jbi.2018.02.016.
[27] J. Nielsen, Severity ratings for usability problems, Pap. Essays 54 (1995) 1–2.
[28] L. Ahmadian, F. Salehi, A. Abedinzadeh, F. Khatibi, Usability evaluation of a
radiology information system, J. Health Adm. 20 (69) (2017) 67–78.
[29] M. Assante, P. Pagano, L. Candela, F. De Faveri, L. Lelii, An approach to virtual
research environment user interfaces dynamic construction, in: International
Conference on Theory and Practice of Digital Libraries, Springer, 2011,
pp. 101–109.
[30] Y.J. Kumar, M.C. Rao, Unauthorized data access detection by insertion of duplicate
data records, Int. J. Res. Comput. Commun. Technol. 1 (7) (2012) 538–542.
[31] S.L. Pauwels, C. Hübscher, S. Leuthold, J.A. Bargas-Avila, K. Opwis, Error
prevention in online forms: use color instead of asterisks to mark required-fields,
International Journal of Medical Informatics 152 (2021) 104459
Interact. Comput. 21 (4) (2009) 257–262, https://doi.org/10.1016/j.
intcom.2009.05.007.
[32] J. Nielsen, Usability Engineering, 2nd ed., Academic Press, San Diego, 1993.
[33] C. Wilson, User Interface Inspection Methods: A User-centered Design Method,
Elsevier Inc, USA, 2014.
[34] T.A. Graham, A.W. Kushniruk, M.J. Bullard, B.R. Holroyd, D.P. Meurer, B.H. Rowe,
How usability of a web-based clinical decision support system has the potential to
contribute to adverse medical events, AMIA Annual Symposium Proceedings,
American Medical Informatics Association (2008), 06 Nov 2008: p 257.
[35] C. Gatsou, A. Politis, D. Zevgolis, The importance of mobile interface icons on user
interaction, Int. J. Comput. Sci. Appl. 9 (3) (2012) 92–107.
[36] S. Isherwood, Graphics and semantics: the relationship between what is seen and
what is meant in icon design, in: International Conference on Engineering
Psychology and Cognitive Ergonomics, Springer, 2009, pp. 197–205.
[37] P. Darbyshire, ‘Rage against the machine?’: nurses’ and midwives’ experiences of
using computerized patient information systems for clinical information, J. Clin.
Nurs. 13 (1) (2004) 17–25, https://doi.org/10.1046/j.1365-2702.2003.00823.x.
[38] Z. Shen, C. Xue, H. Wang, Effects of users’ familiarity with the objects depicted in
icons on the cognitive performance of icon identification, Iperception 9 (3) (2018),
2041669518780807, https://doi.org/10.1177/2041669518780807.
[39] R. Farrahi, F.R. Jeddi, E. Nabovati, M. Sadeqi Jabali, R. Khajouei, The relationship
between user interface problems of an admission, discharge and transfer module
and usability features: a usability testing method, BMC Med. Inform. Decis. Mak.
19 (1) (2019) 172, https://doi.org/10.1186/s12911-019-0893-x.
[40] J. Guo, S. Iribarren, S. Kapsandoy, S. Perri, N. Staggers, Usability evaluation of an
electronic medication administration record (eMAR) application, Appl. Clin.
Inform. 2 (02) (2011) 202–224, https://doi.org/10.4338/ACI-2011-01-RA-0004.
Print 2011.
[41] A. Atashi, R. Khajouei, A. Azizi, A. Dadashi, User Interface problems of a
nationwide inpatient information system: a heuristic evaluation, Appl. Clin.
Inform. 7 (01) (2016) 89–100, https://doi.org/10.4338/ACI-2015-07-RA-0086.
[42] W.O. Galitz, The Essential Guide to User Interface Design: an Introduction to GUI
Design Principles and Techniques, John Wiley & Sons, New York, 2007.
[43] R. Khajouei, Usability Evaluation of Healthcare Information Systems: Comparison
of Methods and Classification of Usability Problem. Dissertation, University of
Amsterdam, Faculty of Medicine, 2011.
[44] M.E. Wiklund, J. Kendler, L. Hochberg, M.B. Weinger, Technical Basis for User
Interface Design of Health IT. No. Grant/Contract Reports (NISTGCR)-15-996,
2015.
[45] R. Khajouei, M. Zahiri Esfahani, Y. Jahani, Comparison of heuristic and cognitive
walkthrough usability evaluation methods for evaluating health information
systems, J. Am. Med. Inform. Assoc. 24 (e1) (2016) e55–e60, https://doi.org/
10.1093/jamia/ocw100.