International Journal of Human–Computer Interaction

ISSN: (Print) (Online) Journal homepage: https://www.tandfonline.com/loi/hihc20

A Usability Evaluation Instrument for Pain

Management Mobile Applications: An Elderly’s
Perspective

Umm e Mariya Shah, Thiam Kian Chiew & Yasir Mehmood

To cite this article: Umm e Mariya Shah, Thiam Kian Chiew & Yasir Mehmood (2022):
A Usability Evaluation Instrument for Pain Management Mobile Applications: An
Elderly’s Perspective, International Journal of Human–Computer Interaction, DOI:
10.1080/10447318.2022.2158517

To link to this article: https://doi.org/10.1080/10447318.2022.2158517

View supplementary material

Published online: 26 Dec 2022.

Submit your article to this journal

Article views: 62

View related articles

View Crossmark data

Full Terms & Conditions of access and use can be found at

https://www.tandfonline.com/action/journalInformation?journalCode=hihc20
INTERNATIONAL JOURNAL OF HUMAN–COMPUTER INTERACTION
https://doi.org/10.1080/10447318.2022.2158517

A Usability Evaluation Instrument for Pain Management Mobile Applications: An

Elderly’s Perspective
Umm e Mariya Shah, Thiam Kian Chiew, and Yasir Mehmood
Department of Software Engineering, Faculty of Computer Science and Information Technology, Universiti Malaya, Kuala Lumpur, Malaysia

ABSTRACT
Technology offers new opportunities in healthcare to manage chronic conditions cost-effectively.
In the adoption of mHealth services, one of the main barriers is usability, especially for users hav-
ing limitations or special needs, such as the elderly. Unlike mHealth apps specifically designed for
the elderly to monitor and manage chronic conditions such as cardio-vascular health, diabetes,
and hypertension, there is a lack of research in pain management, including usability evaluation.
Moreover, there is a lack of comprehensive usability evaluation tool. This research focuses on the
formulation and assessment of the usability evaluation instrument for the mHealth apps, targeting
elderly users. An instrument comprising 8 usability constructs and 58 distinct items was formu-
lated. The content of the instrument was verified by five experts. Face validity was performed with
16 elderly pain sufferers on pre-selected pain apps, it also aids in validating the reliability of the
instrument. The research also discusses the mapping of the instrument’s items with the usability
measures and elderly barriers influencing technology, with graphical visualization and scores of
various artefacts. The graphical details would be helpful in identifying the areas for usability
improvement from the elderly’s perspective and the scores would be helpful in comparing usabil-
ity of different applications.

1. Introduction adoption of any innovation, whereas the adoption rate

According to WHO (World Health Organization), elderly,
i.e., 65 years old and above people are rapidly becoming a
majority of the worldwide populace (Chuna & Patterson,
2012). There are several requirements to sustain the elderly’s
living, which include health monitoring, information seek-
ing, social interaction, relaxation activities, security, and pro-
tection (Hassan & Md Nasir, 2008; Plaza et al., 2011). In
this context, ICT (Information and Communication
Technology) plays a significant role. A new domain named
Gerontechnology has been evolving which refers to the tech-
nology that facilitates the aging society as per their needs. It
means incorporating research, development, and design in
the engineering disciplines keeping in view the peculiarity of
the aging phenomenon. Among the areas of research cov-
ered by Gerontechnology, it specifically includes areas of
independent living, caregiver support, and healthcare (Plaza
et al., 2011).
However, the elderly being resistant to the adoption of
new technologies are categorized as late majority or laggards
according to Rogers Innovation Diffusion theory (Wong,
2011). According to Rogers (1995), adoption of any innov-
ation is influenced by five factors: relative advantage, com-
patibility, trialability, observability, and complexity. The first
four factors have a positive correlation with the rate of
declines with increased complexity.
In recent years, studies have shown the elderly’s growing
interest in mobile health technology, proving the myth of the
elderly’s less diversion towards the technology to be less legit-
imate (Huguet et al., 2015). Besides, mobility and independent
living mobile technology could be a promising tool in the
self-management of chronic diseases, thus enhancing the qual-
ity of living of the elderly (Holzinger et al., 2007; Lindeman,
2015; Plaza et al., 2011). Nevertheless, certain barriers
described in the next section put hurdles in the acceptance of
mobile technology by the elderly. It is necessary to understand
the complexities, uncertainties, and difficulties encountered by
the elderly to give them user-friendly solutions and increase
motivation and acceptance in adopting the solution.
This research is confined to provide a usability assess-
ment tool that covers major aspects of mobile app’s usability
evaluation for the elderly. It will be significant for the pro-
fessionals to involve users in a comprehensive usability
evaluation of healthcare mobile applications concerning the
elderly. The instrument also contributes to the quantification
of usability measures as well as the elderly barriers of the
mHealth (mobile health) application. The visualization of
assessment results will help to identify the areas of improve-
ment. Researchers and designers could use this instrument

CONTACT Thiam Kian Chiew tkchiew@um.edu.my Department of Software Engineering, Faculty of Computer Science and Information Technology,
Universiti Malaya, 50603 Lembah Pantai, Kuala Lumpur, Malaysia
Supplemental data for this article can be accessed on the publisher’s website.
ß 2022 Taylor & Francis Group, LLC
2 U. E. M. SHAH ET AL.
to modify or develop the app in accordance with the tool’s
highlighted areas of improvement. The solution could also
be helpful for the clinicians to choose among the existing
pain management apps and recommend the most suitable
app for elderly patients as a self-management regime.
2. Background
This section reviews literature related to the elderly’s barriers
to technology use, usability of mHealth technology from the
elderly’s perspective, and pain management mobile applica-
tions and the elderly.
2.1. Elderly’s barriers to technology use
There are several specific technological issues for the elderly,
which include, cognition, motivation, physical and percep-
tion issues (Czaja et al., 2013; Holzinger et al., 2007; W. A.
Rogers & Fisk, 2010).
Cognitive issues are related to a reduced capacity of
working, prospective, semantic, and procedural memory,
attention, processing of time-restricted discrete information,
ability to recall, increased time to learn new skills, and
decline in numeracy and representational fluency
(Wildenbos et al., 2018). As cognitive performance slows
down with age so for the complex applications, it is recom-
mended to simulate user interaction and alter the design of
the applications upon feedback to produce a simplified
design. A simple application can help to match and enhance
the performance of elders, as implicit learning ability does
not reduce by aging (Holzinger et al., 2007).
The motivational issues are comprised of poor under-
standing of benefits, problems with beliefs, attitudes, anxiety,
fear, computer literacy, and acceptability (Holzinger et al.,
2007; Wildenbos et al., 2015). Most elders have poor or less
knowledge of the benefits of using a mobile application.
They also feel uncomfortable learning new technology and
lose concentration easily and become bored while learning.
Sometimes they don’t have trust in their ability to learn or
capability to operate certain devices (Holzinger et al., 2007).
The physical issues are problems with the physical body
parts of elders that results in reduced flexibility and motor
skills (Holzinger et al., 2007). Physical barriers also affect
slower movement and reflexes, stiffer muscles and joints,
tremors (in hands), and a diminished balance or difficulty
in holding a device (Wildenbos et al., 2018). Also, basic ICT
usage might result in a challenge for most elders, since they
might have a difficult time locating items on the screen, and
will require substantial effort to have the hand-eye coordin-
ation required to use mobile devices (Mendez & Mendoza,
2013). Such issues lessen the confidence of elders to learn
new things and absorb knowledge (Holzinger et al., 2007).
The perception issues include barriers to vision and audi-
tion (Holzinger et al., 2007). The abilities that decrease with
age include visual acuity, visual accommodation, color
vision, contrast detection, dark adaptation, glare, motion
perception, and peripheral vision (Holzinger et al., 2007).
The factors where the necessity increases with age involved
demand for illumination increases with age, and a signifi-
cant reduction of auditory facilities with age (Holzinger
et al., 2007).
The aforementioned barriers offer numerous rational rea-
sons for why mHealth applications are not adopted by the
elderly. Longer learning times and low visual acuity were
the two most often mentioned causes among those found. It
is possible to correlate various conditions that an older adult
might experience and understand which aspects of usability
might impede by this condition (Paiva et al., 2020).
2.2. Usability of mobile health technology from elderly
perspective
Some of the foundational studies (Darroch et al., 2005;
Goodman et al., 2005; Siek et al., 2005; Sterns, 2005)
addressed the usability of handheld technology by the eld-
erly and showed that portable technology can be helpful in
providing mobility and independence for the elderly.
However, if the objectives and needs are not fulfilled and
the solution is not user-friendly and does not provide ease
to use, it will hinder the elderly’s acceptance of technology
(Nimmanterdwong et al., 2022). Thus, usability is essential
to technology adoption and is a significant field of research
(Caboral-Stevens et al., 2015).
A few previous studies investigated the usability evalu-
ation of mobile technology with respect to the elderly.
Becker and Webbe (2006), proposed a framework based on
Schneiderman’s model, comprising four main components:
user profile, environmental factors, usability quality factors,
and technology objective; to uncover the potential barriers
associated with the interface design. The framework could
be used as a code of conduct to execute and report a usabil-
ity study involving PPC (Pocket PC) and elderly caregivers;
and define measurable outcomes for a usability study involv-
ing the elderly.
Mendez and Mendoza (2013) proposed a conceptual
framework based on the principles of ISO/IEC 2000
(International Organization for Standardization/International
Electrotechnical Commission), abstraction hierarchy com-
monly found in the cognitive systems (Rasmussen, 1986)
and Ham’s hierarchical usability model (Heo et al., 2009).
This framework identifies a relationship of the elderly bar-
riers with the usability criteria and usability properties. As
this model presents a complex multi relation scheme, it
would be difficult to implement this model and a compre-
hensive checklist is required for an elaborative evalu-
ation scheme.
Another study (Wildenbos et al., 2015) suggested a
framework for the evaluation of mHealth tools for elderly
patients. It is a theoretical framework that explains the age-
related barriers impacting mHealth’s ease of use. Based on
the literature, the researchers identified the age-related bar-
riers and developed an app for the elderly. The usability
issues were analyzed via a cognitive walkthrough. The issues
were then categorized based on the mHIMSS (mobile
Healthcare Information and Management Systems Society)
themes and severity ranking was performed as per Neilson’s
 INTERNATIONAL JOURNAL OF HUMAN–COMPUTER INTERACTION 3

classification of severity. The usability issues were further
mapped with the potential underlying feature of the elderly
that could be the reason behind that usability issue. This
study concluded that mHIMMS design guidelines are not
sufficient for developing highly usable apps and this frame-
work could give insight that can help in the (re)design of
the app.
Wildenbos et al.’s work was further extended by the
researchers where a framework MOLD-US was presented to
summarize the evidence of the effect of aging barriers on
the use of mHealth (Wildenbos et al., 2018). A scoping
review was carried out to formulise the physical and func-
tional age-related barriers and their causes, due to some
common medical conditions, in accordance with the usabil-
ity issues of the elderly in using mHealth apps. The usability
aspects in this framework are based upon Neilson’s quality
components: errors, efficiency, learnability, memorability,
and satisfaction (Nielsen, 2012). The correlation between
aging barriers and medical conditions is visualised in the
framework. However, it is not necessary that a specific dis-
ease only leads to that particular barrier as specified in the
framework. The elderly could have multiple morbidities and
a mHealth intervention should address multiple aging bar-
riers. Table 1 summarises the usability evaluation measures
of the aforementioned studies.
It has been observed that all the aforementioned usability
frameworks do not include the same set of attributes.
Learnability, efficiency, and satisfaction are the common
attributes among all the frameworks; whereas, retention
overtime/memorability, error rate, effectiveness, functional
support as per user needs, and control to operate are not
taken into account in some of the frameworks. This high-
lights the need for a common and comprehensive usability
evaluation framework. In the aforementioned studies, there
is also a lack of involvement of elderly patients using health-
care mobile applications in the validation of the proposed
frameworks. Studies on the realistic use by the elderly are
required to understand how age obstacles affect elderly
patients’ use of mHealth applications.
The frameworks also provide theoretical knowledge of
usability evaluation without a comprehensive assessment
tool for mapping usability evaluation with the age-related
barriers. Such assessment tool will be helpful in analyzing
the areas of improvement and (re)designing the mHealth
applications with respect to the elderly population. Despite
few available guidelines (Aging & Medicine, 2001; Silva
et al., 2015) for performing usability evaluation with respect
to the elderly, little research has been conducted towards the
application of the usability assessment tool with respect to
the elderly barriers to technology use. There is a gap in this
area of research.
2.3. Pain management mobile applications and elderly
Self-management of chronic diseases could be made effective
by giving easy communication, accessible information, and
simple data storage and retrieval techniques to the patients,
caretakers, and clinical experts. Mobile health technology
could prove beneficial in this context and capable of reform-
ing patients’ conduct towards their wellbeing. Similar to the
youth population, elders may be benefited from mobile
health technology (Bakshi et al., 2015). According to WHO,

Table 1. Summary of the usability evaluation measures in the usability frameworks.

Becker and Webbe (2006)
Time to Learn
“The time it takes for a typical user of
the targeted group to learn how to
use the device in order to learn
how to use the actions relevant to
a set of tasks.”
Speed of Performance
“The time it takes to complete the
benchmarked tasks.”
Error Rate
“The number, type, and severity of
errors made by a typical user when
completing the benchmarked tasks”
Retention over time
“The knowledge that is maintained
after a period of non-use.”
Subjective Satisfaction
“The user’s feedback on how well he
or she liked various aspects of the
user interface”
– Effectiveness
– Customization
– –
Mendez and Mendoza (2013)
Learnability
“Support of cognitive interaction”
Efficiency
“Support of efficient interaction”
– –
– –
Satisfaction
“Ergonomic support”
“Visual support of task goals”
“Functional support of user needs”
Caboral-Stevens et al. (2015)
Learnability
“Easiness of learning the system and
retrieving information from
the system”
Efficiency
“Amount of effort required to use the
system and usefulness in meeting
the user’s needs and goals.”
Perceived User Experience
“Level of pleasantness in using the
system and the quality of the
system’s design.”
– –
– –
Perceived Control
“Amount of control users have to
choose and decide how
to proceed.”
Wildenbos et al. (2018)
Learnability
“How easy is it for users to accomplish
basic tasks the first time they
encounter the design?”
Efficiency
“Once users have learned the design,
how quickly can they
perform tasks?”
Errors
“How many errors do users make, how
severe are these errors, and how
easily can they recover from
the errors?”
Memorability
“When users return to the design after
a period of not using it, how easily
can they re-establish proficiency?”
Satisfaction
“How pleasant is it to use the design?”
–
4 U. E. M. SHAH ET AL.
the elderly, are rapidly becoming a majority of the world-
wide populace as well as the population group suffering
from the chronic diseases than young adults (Chuna &
Patterson, 2012), having three diseases on average (Mira
et al., 2014) where the pain is their common illness.
As identified from a literature review of usability issues in
healthcare mobile applications for the elderly (Shah & Chiew,
2017), numerous elderly-specific healthcare mobile applica-
tions exist for certain diseases like cardiovascular health, dia-
betes, fall detection and prevention, and Alzheimer’s disease.
However, far too little attention has been paid to the area of
mHealth and pain management in relevance to the elderly
age group, although chronic pain ranks six among the major
chronic conditions (Gereau et al., 2014).
An integrative review was conducted in a study
(Bhattarai & Phillips, 2017) that evaluates the role of digital
health technologies in the management of pain in the eld-
erly. The review revealed that most studies involved com-
puter-based interventions. Only one study (Parker et al.,
2013), which was qualitative (focus group discussion),
involved smartphone-based intervention in which the bar-
riers and facilitators for the usage of mHealth for pain man-
agement by the elderly were identified. This review also
affirms a research gap in the area of testing and evaluation
of the exponentially growing mobile apps for pain manage-
ment, involving the elderly population.
Another integrative review was conducted in a study
(Dunham et al., 2021) that objectively analyzed the quantita-
tive and qualitative research that is currently available
regarding the advantages of smartphone applications for the
treatment of chronic pain in the elderly. The review’s out-
comes highlight the general dearth of studies supporting
older adults who have long-term diseases like chronic pain.
Figure 1. Process flow of the formulation of usability evaluation instrument for the elderly.
Large-scale research on the unique requirements of elderly
adults with chronic pain is lacking.
Hence it is required to elicit the elder specific utility and
usability requirements to develop an optimized solution for
pain management using mobile technology and integrate the
apps into the clinical practice (Bhattarai & Phillips, 2017).
Inter-professional collaboration alongside real end users is
important in this perspective as well (Nimmanterdwong
et al., 2022).
Thus, in the purview of the aforementioned research gaps
this research endeavors to: formulate and validate a compre-
hensive tool for the usability evaluation, in particular for the
elderly; mapping of the usability constructs with the usabil-
ity measures and elderly barriers influencing usability; fol-
lowed by usability evaluation of pain management mobile
applications involving the elderly population
3. Methodology
This study is exploratory and experimental in nature.
Taking into consideration various parameters of this
research, mixed method approach was adopted. Mixed
method approach involves a blend of quantitative and quali-
tative techniques which integrate the facts based on both
approaches (Collins et al., 2007; Creswell & Creswell, 2017).
A systematic literature review-based research approach was
adopted to formulate the usability evaluation instrument.
The instrument was validated using qualitative and quantita-
tive approaches, i.e., via a questionnaire survey and usability
evaluation study (statistical method). The reliability of the
instrument was then analysed quantitatively (questionnaire).
The approach followed by this study is shown in
Figure 1. The research first explores the elderly’s barriers to

technology use, the proposed frameworks of usability evalu-
ation, the interface design guidelines for the elderly, and the
elderly’s usability issues in healthcare mobile applications.
In the second phase, the method for formulating the
usability evaluation instrument was determined.
Questionnaire was chosen as a mean of usability evaluation
instrument for the elderly.
The third phase involved the formulation of the usability
evaluation questionnaire. In this context, the usability evalu-
ation attributes, identified via systematic literature review of
the pain management mobile applications (Shah & Chiew,
2019) and the systematic literature review of the elderly and
healthcare mobile applications (Shah & Chiew, 2017), were
grouped into key themes in accordance with their similar-
ities and proximity, termed as usability constructs. The
items (questionnaire statements) of the usability constructs
were formulated then by taken into account the respective
usability constructs, and the usability evaluation attributes
and findings, or usability recommendations identified from
the SLRs. Moreover, QUIS (Questionnaire for User Interface
Satisfaction) (Shneiderman & Norman, 1992) and general
guidelines for the interface design specific for the elderly
(Aging & Medicine, 2001; Silva et al., 2015) were also con-
sidered. The measurement method was designed in the form
of closed-end questions with a range of choices using either
Likert or semantic differential scale, as it would be simpler
for respondents to record their opinion/perception towards
the usability of mHealth applications (Brinkman, 2009; de
Vet et al., 2011).
In the fourth phase, the proposed usability evaluation
questionnaire was validated using quantitative approaches,
i.e., via a questionnaire survey (content validity with domain
experts and face validity with non-domain experts and eld-
erly participants), and involved elderly pain sufferers to
evaluate the selected commercially available pain manage-
ment mobile applications (statistical method). The cumula-
tive responses of the domain experts, non-domain experts,
and the elderly participants provided the assessment of the
proposed usability evaluation instrument, reflected their
agreement on the comprehensiveness, appropriateness, and
relevancy of the items to the constructs. The suggestions of
experts are incorporated in the finalized version of the
usability evaluation instrument
The fifth phase involved mapping of the usability con-
structs (based on the concept and definition) with the
usability measures of ISO/IEC 25010 and the elderly barriers
that influence usability hinder the elderly’s technology use.
Each individual item under all the usability constructs was
treated separately in the mapping.
In the sixth phase, the proposed usability evaluation
instrument was implemented based on the mapping criteria
to calculate numeric values for the usability measures and
the barriers (cognitive load, motivational load, perceptual
load, and physical load) in technology use, for healthcare
mobile applications with respect to the elderly population.
Lastly, in the seventh phase, usability evaluation was per-
formed involving the elderly pain sufferers. The elderly par-
ticipants evaluated pain management mobile applications via
INTERNATIONAL JOURNAL OF HUMAN–COMPUTER INTERACTION 5
the usability evaluation instrument. Scores were assigned to
each item of the questionnaire as per the participant’s
response. The mean values for the respective entities were
then calculated and the evaluation results visualised. Post
evaluation session face validity of the proposed usability
evaluation instrument was performed with the participants
and a quantitative evaluation of reliability was also per-
formed using Cronbach’s alpha coefficient, a.
4. Formatting the elderly’s usability
evaluation instrument
Evaluation in this research used a questionnaire, which is a
common tool for evaluating a product by HCI (Human
Computer Interaction) practitioners (Bark et al., 2006).
4.1. Conceptualization
In this phase, the constructs intended to be measured are
identified and elaborated as items of the questionnaire (de
Vet et al., 2011). The usability evaluation attributes used in
the studies, identified via a systematic literature review of
the pain management mobile applications and the systematic
literature review of the elderly and healthcare mobile appli-
cations (Shah & Chiew, 2017, 2019) were grouped as per
similarity and proximity in their meaning under main
themes, termed as the usability constructs in this research.
Eight usability constructs are proposed: readable design/typ-
ography, text writing style, layout, navigation, learn and use,
memory load and mental effort, user guidance and error
handling, and cues, instructions, and user control. The defi-
nitions of these constructs and grouping of the usability
evaluation attributes identified from the literature are shown
in Table 2.
For this research, closed-end questions were selected with
a set of options as it will be easier for the elderly respond-
ents to respond while assessing the usability of mHealth
applications. The items of the usability evaluation instru-
ment were developed by keeping in view the usability con-
structs and usability evaluation attributes shown in Table 1.
Findings or usability recommendations from the SLRs
(Systematic Review of Literature) (Shah & Chiew, 2017,
2019) were also taken into account to formulate the items of
the usability evaluation instrument. QUIS (Questionnaire for
User Interface Satisfaction) (Shneiderman & Norman, 1992)
and general guidelines for the interface design specific for
the elderly (National Institute on Aging & National Library
of Medicine, 2001; Silva et al., 2015) were also considered.
A point to note is that no comprehensive guidelines exist
in literature about learning, memory load, user guidance,
error handling, cues, instructions, and user control as per
the elderly population. The proposed comprehensive elderly-
specific usability evaluation instrument is either helpful for
the mHealth app designers and developers, or for the usabil-
ity experts to perform a heuristic evaluation. Moreover, it is
simple enough to be assessed by the elderly in the usabil-
ity studies.
6 U. E. M. SHAH ET AL.

Table 2. Usability constructs and grouping of the usability evaluation attributes identified from literature.
Usability
No. constructs Definition
1. Readable design/ It is the composition of the text for clarity in
Typography appearance and readability in terms of font
face, size, alignment, spacing and format
(Foraker Labs, 2012).
2. Text writing style It is the representation of the text in a clear
and simple way to easily comprehend the
information (Foraker Labs, 2012).
3. Layout It is the way in which the elements of the
screen (text, images, menu or any active
component) are arranged/organized
(Foraker Labs, 2012).
4. Navigation It is the way to proceed within large or
complex information space, i.e., to move
within the elements of a screen, forward
and back, and from one screen to another
(Sikorski, 2002).
5. Learn and use It is a measure of how quickly and effectively
an application can be understood to use by
the user without prior knowledge or at the
first time of use (Foraker Labs, 2012).
6. Memory load and The amount of effort involved with thinking
mental effort and reasoning, that might overlap with
other thought processes, i.e., how much a
user has to keep in mind while performing
a task (Foraker Labs, 2012).
7. User guidance and It is the way to decrease the possibility of
error handling occurrence of mistakes by assisting users in
providing them ease to complete the task;
or by guiding the users for the
management of error situation by
presenting some output in response to
users’ input (Sikorski, 2002).
8. Cues, instructions It is the degree to which an application
and user enables the users to acquire additional
control guidance via information available within
the application regarding various features
and their step-by-step usage so that the
disabled users or the users with the widest
possible range of skills and attributes could
make better control and use of the
application (Association, 2005).
Usability evaluation attributes identified
from literature
Font size, colour scheme/colour contrast
Clarity of content, clarification of the wording
of questions, language, simplicity of
content, easy to understand, paraphrasing,
comprehension (meaning and
understanding of the question),
comprehensiveness of the queries and
response set/completeness, re-reading of
the instructions, participant’s uncertainty
about the task instructions, difficulty
comprehending the instructions
Visual appearance, layout, consistency/
inconsistency, aesthetics, clear visual display,
representation, look and feel
Navigation, user interaction, controllable, easy
to use/user-friendliness/ease of data input/
cumbersome to use, functionalities (submit,
clear, cursor movement), issues with the
interface, self-explanatory menu structures,
menu labelling, recognition of touch
sensitive areas, large size of operating
elements, display size
Learnability, knowledge, ease of learning, quick
learn to use, need to learn, ability to use
the application without prior knowledge,
fast achievement of a first feeling of
success, requirement of technical support
for system usage, intervention required
Memory retrieval (ability to accurately recall
the answer), complexity, workload demand,
minimalist display, simple comprehensibility
and interpretability of displayed images and
depictions, recognisability
Error/error prevention/system protection
against making errors, responsiveness of the
screens to touch, helpfulness or usefulness
of information, help, help documentation
and support, communication and
support, feedback
Alarm and notifications setting, controllable,
additional notes, information to support
device familiarity (e.g., dragging finger
instead of tapping and holding
device surface)

4.2. Measurement method
In this research, the approach of using attitude scales (de Vet
et al., 2011) has been adopted to evaluate the opinion/percep-
tion of the elderly participants towards the usability of the
pain management mobile applications. Each item was pro-
vided with a set of fixed choice answers using either Likert or
semantic differential scale. The complete questionnaire (after
validation by the experts) is shown in Appendix A.
5. Validation of the usability evaluation instrument
Five domain experts and three non-domain experts were
invited by email to take part in the expert review. Three
domain experts (usability) and two non-domain experts (lin-
guistics and medical) accepted the invitation. Generally,
responses from two or three experts are acceptable for valid-
ation purpose (Lam et al., 2018). The domain experts eval-
uated the comprehensiveness of the identified constructs
and items, the relevancy of the items to the constructs, and
the appropriateness of the usability evaluation instrument
with respect to elderly people.
A 5-point Likert scale was used to obtain the experts’
opinion, i.e., strongly agree, agree, neither agree nor dis-
agree, disagree, and strongly disagree. The experts were also
required to identify areas of weakness by giving recommen-
dations on ways of improving the instrument to ensure
transparency and comprehensibility (Lynn, 1986). The

questionnaire for the assessment of the proposed usability
evaluation instrument for elderly people is given in
Appendix B. The relevancy of the items to the constructs
was validated on a 4-point ordinal scale, i.e., not relevant,
somewhat relevant, quite relevant, and highly relevant
(Chiwaridzo et al., 2017; Davis, 1992; Lam et al., 2018;
Vrbnjak et al., 2017). Moreover, the items were also vali-
dated for their clear meaning and understandability. The
experts were also provided with the option to share their
suggestions for improvement, addition, or deletion of
any items.
Furthermore, the subjective assessment was performed as
per Oluwatayo (2012) and Wynd and Schaefer (2002) to
evaluate the overall comprehensiveness, appropriateness, and
understandability of the proposed usability evaluation
instrument with the non-domain experts (medical doctor
and linguistic) and the elderly people (participated in the
usability evaluation). They were also asked to make sugges-
tions for improving the instrument.
The validation process is depicted in Figure 2. The ques-
tionnaire is attached in Appendix C. For the inter-rater
agreement, the simplest degree of measurement, i.e., per-
centage as specified in Bowling (2014) has been used.
The demographic details of the experts (Appendix D),
socio-demographic details of the elderly participants
(Appendix E), validation result of the comprehensiveness
and appropriateness of the usability constructs and items of
the usability evaluation instrument, and relevancy of the
items to the constructs are attached in Appendices F to G
respectively. Result of subjective assessment of the proposed
usability evaluation instrument by non-domain experts and
elderly participants is shown in Appendix H.
The accumulative responses of the domain experts, non-
domain experts, and the elderly participants provided the
assessment of the proposed usability evaluation instrument,
Figure 2. Validation process.
INTERNATIONAL JOURNAL OF HUMAN–COMPUTER INTERACTION 7
reflected their agreement on the comprehensiveness, appro-
priateness, and relevancy of the items to the constructs. The
suggestions of experts are incorporated in the finalized ver-
sion of the usability evaluation instrument. The accessibility
aspects (as highlighted by one of the experts) were also con-
sidered while the mapping of the usability constructs with
the usability measures and elderly barriers influenc-
ing usability.
5.1. Reliability of the usability evaluation instrument
Usability evaluation of existing pain management mobile
applications was also performed by involving real-time eld-
erly pain app users to evaluate the reliability of the proposed
usability evaluation instrument. The quantitative evaluation
of reliability was performed using a famous reliability statis-
tic Cronbach’s alpha coefficient, a (Cronbach, 1951). For
this purpose, sixteen elderly participants 65 years old or
older, diagnosed with chronic pain, understand English and
have literacy in smartphone usage were involved to perform
the usability evaluation of existing pain management mobile
applications. Participants were excluded if they have active
mental health problems, e.g., depression or anxiety or have a
cognitive and visual impairment.
For elderly participants’ recruitment, ethical approval was
granted by the Medical Research Ethics Committee of
UMMC (University of Malaya Medical Centre) (MREC ID
NO. 201881-6552). The medical doctors of the Department
of Primary Care Medicine, rehabilitation physician of the
Advanced Pain Care and Rehabilitation center, Department
of Rehabilitation Medicine, and consultant neurosurgeon,
Pediatric Neurosurgery and Interventional Pain
Management, Department of Surgery were approached to
seek their permission and assistance to recruit patients.
8 U. E. M. SHAH ET AL.
Patients were also provided with the participant information
sheet and the consent form to proceed with the usability
evaluation study.
On the other hand, commercially available pain man-
agement mobile applications were screened and selected
from Google Play Store and Apple’s App Store based on
a specified time frame, keywords, and inclusion/exclusion
criteria. The search was conducted in May 2017 that was
revised and updated in January 2019. The keywords
involved in the electronic search were: pain, pain manage,
pain management, pain track, pain log, pain diary, pain
record, pain scale, pain severity, and pain journal. From
the search results, an app was excluded if the app was:
(a) paid; (b) outdated i.e., not updated for 2 years or
more; (c) for general health record; (d) for any specific
pain type (such as headache, migraine, back pain, fibro-
myalgia); (e) specifically, for kids; (f) designed for the
health care professional only; (g) for pain management
program only, i.e., describing any techniques for pain
management, e.g., guided image therapy, pain-relieving
breath, hypnotherapy, exercise package; (h) only based on
pain information or based on pain management reference
resource only; (i) for the introduction of any clinic or
institute or academy; (j)for a private study only.
Twenty-one apps were identified from Google Play Store;
the apps were sorted in descending order of the number of
downloads. After removing the outdated, paid and duplicate
versions of the apps, 11 unique apps were identified. 34
apps were identified from iTunes Store; the apps were sorted
in descending order of customer rating. After removing the
outdated, paid and duplicate versions of the apps, 16 unique
apps were identified for further analysis.
As our study was intended to perform usability evalu-
ation via the proposed instrument and also a comparative
study among the four selected commercially available pain
management mobile applications (i.e., two apps from
iTunes Store and two apps from Google Play Store), the
criteria to recruit 16 ± 4 participants were adopted to serve
the purpose, as per literature (Alroobaea & Mayhew,
2014; Six & Macefield, 2016). There were 12 participants
for the Android apps and 4 participants for the
iPhone apps.
To narrow down the scope, the top five apps were first
selected from each platform’s unique app list, based on the
number of downloads and customer rating/stars. The apps
were further screened with respect to their functionality as
per the conceptual utility model proposed by Shah & Chiew
(2019b) and pain data input methods discussed in Shah &
Chiew (2019a) to select two apps from each platform. In
this context, Manage My Pain Lite and Pain Companion
were identified from Google Play Store; and Pain Diary and
Community CatchMyPain and Chronic Pain Tracker Lite
were identified from App Store, for further assessment with
the proposed usability evaluation instrument.
The usability evaluation study was preceded by a demo
session where participants were presented with the selected
pain apps and provided with hands-on training on the use
of these apps. The mobile applications were preinstalled on
an Android or iPhone devices of the researcher and each
participant evaluated both apps depending on the platform
(operating system) of the participant’s personal smartphone.
As the quantitative evaluation of reliability was performed
using Cronbach’s alpha coefficient, a few items with zero
variance were removed from the scale in the reliability tests.
The test result shows that the coefficient values were greater
than 0.7 (0.948, 0.898, 0.875, 0.906) for each pain manage-
ment mobile application respectively. It indicates a high
level of internal consistency and ensures that the evaluation
results are reliable with an acceptable value of
Cronbach’s alpha.
The internal consistency of the usability constructs was
also determined via the accumulative evaluation results of
the four pain management mobile applications used in the
usability evaluation. Results indicated an acceptable value
of Cronbach’s alpha coefficient for each usability con-
struct, i.e., readable design (0.945), test writing style
(0.908), layout (0.941), navigation (0.658), learn and use
(0.847), memory load or mental effort (0.844), user guid-
ance and error handling (0.610) and cues, instructions
and user control (0.550). The a values of five usability
constructs are above 0.7, while three usability constructs
are below 0.7, nevertheless, the a value of 0.5 and above
is sufficiently good to measure attitude. Thus, the a values
of all usability constructs are acceptable (Tuckman &
Harper, 2012). All the items also seemed retainable, as
deleting any result substantially decreases the a (indicating
the importance of those items), or a slight increase in a
or has no effect on the value of a.
6. Mapping of the usability constructs with the
usability measures and elderly barriers
influencing usability
Usability constructs of the proposed usability evaluation
instrument are mapped with the six ISO/IEC 25010
usability measures (appropriateness recognisability, user
interface aesthetics, operability, learnability, user error pro-
tection, accessibility) and the elderly barriers (cognition,
motivation, perception, and physical) that impact usability
and hinder the use of technology by the elderly. The
mapping was performed based on the definition and con-
cept of the entities. Since WCAG (Web Content
Accessibility Guidelines) is known as a mobile accessibility
assessment standard, its guidelines were also used to map
the items with the usability measure “accessibility.” Each
individual item of the proposed tool is independently
handled in the mapping.
Barriers are the restricted capabilities or proficiency that
could affect usability measures with respect to the con-
structs. This mapping would help to observe the usability
measures impacted by the elderly barriers. In relation to
text-writing (usability construct), for instance, motivation
(barrier) can impact learnability (usability measure).
Quantifying the items of each construct will help to calculate
the numerical values of cognitive load, motivational load,

perceptual load, and physical load corresponding to the use
of an application.
Mapping of the usability constructs and items with the
ISO/IEC 25010 usability measures and the elderly barriers
influencing usability is shown in Table 3.
7. Implementation of the elderly’s usability
evaluation instrument
The elderly’s usability evaluation instrument was imple-
mented based on the scenario as shown in Figure 3.
The usability evaluation instrument is comprised of
eight usability constructs. There are multiple items “i” in
each usability construct “UC.” Table 4 shows the usability
constructs and the related items “q.” Each item has a
fixed set of options and the options were allocated with a
weightage for computation. Table 5 displays the various
response options and their numerical values.
A score was given to each usability construct based on
the mean weighing value of all the related items. Thus, the
score of a usability construct “UCx” is defined as:
X
q
UCx ¼ in =q (1)
n¼1
Each item was mapped with one or multiple usability
measures and the elderly barriers to calculate numeric values
of the six usability measures and the four elderly barriers
in technology.
The proposed usability assessment instrument comprises
a total of 58 items, which can be represented as a set I ¼ i1,
i2, i3, i4, … … … … .i58.
The usability measures could likewise be described as
set UM ¼ UM1, UM2, UM3, UM4, UM5, UM6, where
UM1 ¼ Appropriateness recognisability, UM2 ¼ User
interface aesthetics, UM3 ¼ Operability, UM4 ¼
Learnability, UM5 ¼ User Error Protection and UM6 ¼
Accessibility.
Thus, UMx will be defined as:
UMx ¼ ½Rðweightage for each mapped inÞ=
½total number of mapped in
For instance, appropriateness recognisability UM1 will be
calculated as follows:
UM1 ¼ ði6 þ i7 þ i8 þ i9 þ i10 þ i11 þ i12 þ i13 þ i14
þ i15 þ i16 þ i17 þ i18 þ i19 þ i20 þ i22 þ i23
þ i24 þ i30 þ i35 þ i41 þ i42 þ i57 Þ=ð23Þ
Similarly, the elderly barriers in technology use could
also be represented as set EB ¼ EB1, EB2, EB3, EB4, where
EB1 ¼ Cognition, EB2 ¼ Motivation, EB3 Physical, EB4
¼ Perception.
Therefore, EBx will be defined as:
EBx ¼ ½R ðweightage for each mapped inÞ=
½total number of mapped in
INTERNATIONAL JOURNAL OF HUMAN–COMPUTER INTERACTION 9
For example, cognition EB1 will be calculated as follows:
EB1 ¼ ði13 þ i14 þ i15 þ i16 þ i17 þ i18 þ i19 þ i21
þ i22 þ i23 þ i24 þ i25 þ i26 þ i27 þ i28 þ i29
þ i30 þ i33 þ i34 þ i36 þ i37 þ i38 þ i39
þ i40 þ i42 þ i43 þ i44 þ i45 þ i46 þ i47 þ i48
þ i51 þ i52 þ i53 þ i56 þ i57 Þ=ð36Þ
The mean score s calculated for each UCx, UMx and EBx
were interpreted as very poor (1.0 < ¼ s < ¼ 1.8), poor
(1.8 < s < ¼ 2.6), average (2.6 < s < ¼ 3.4), good (3.4 < s < ¼
4.2) and excellent (4.2 < s < ¼ 5.0). The overall usability score
U of the mobile application is calculated via the mean UMx
scores of the six usability measures and the overall usability
level will be assigned as per the defined range of scores.
The assessment results of the proposed usability evalu-
ation tool are visualized as graphs and tables. A box plot
graph first presents an overview, followed by a tabular view
that shows the overall mean scores and corresponding level
of each entity. A sample of graphical and tabular representa-
tion is shown in Figures 4 and 5, respectively.
A detailed report concerning a specific entity from the
tabular view could be observed by clicking on the link given
in the last column. A graphical summary is represented in
the form of a bar graph.
Figure 6 shows a sample bar chart of the entity
“cognition.” Additional information on the scores and level
of the corresponding usability constructs and items could
also be seen in tabular form. Figure 7 shows the table repre-
senting one of the barriers "cognitive load.”
8. Usability evaluation results of the pain
management mobile applications
Usability evaluation was conducted to assess the selected
pain management mobile applications involving elderly
patients. The quantified scores of the pain management
mobile applications, evaluated using the usability evaluation
instrument, are summarized in Tables 6–8 in relevance to
the elderly barriers influencing usability, usability measures,
and the usability constructs respectively.
The mean scores in Table 6 represent less cognitive,
motivational, physical, and perceptual load for the elderly as
almost all scores were interpreted as “Good.” The scores
indicate the elderly’s barrier in using the apps. The higher
score represents a reduced respective load. Thus, among the
Android apps, Manage My Pain Lite shows slightly less cog-
nitive and motivational load compared to Pain Companion.
In the meantime, the physical load appears to be the same
in both apps and the perceptual load appears to be less in
Pain Companion than in the other app. On the other hand,
among the iPhone apps, Chronic Pain Tracker Lite has a
relatively reduced cognitive, motivational, and perceptual
load than Pain Diary and Community CatchMyPain. On the
contrary, Pain Diary and Community CatchMyPain are
10 U. E. M. SHAH ET AL.

Table 3. Mapping of the usability constructs with the usability measures and the elderly barriers in the use of technology.
Usability constructs and items Usability measures Elderly barriers influencing usability
Readable design/Typography
Font appearance User Interface Aesthetics and Accessibility Perception
Text spacing User Interface Aesthetics and Accessibility Perception
Text alignment User Interface Aesthetics and Accessibility Perception
Use of font colour User Interface Aesthetics and Accessibility Perception
Background contrast User Interface Aesthetics and Accessibility Perception
Text writing style
Positive statements (Sentences do not contain Appropriateness Recognisability and Accessibility Motivation (computer literacy)
negative words like not, no, never etc.)
Sentences with proper description Appropriateness Recognisability and Accessibility Motivation (computer literacy)
Simple language Appropriateness Recognisability and Accessibility Motivation (computer literacy)
Clear meaning Appropriateness Recognisability and Accessibility Motivation (computer literacy)
Sentences in active voice (where subject performs Appropriateness Recognisability and Accessibility Motivation (computer literacy)
the action)
Bold keywords or terminologies Appropriateness Recognisability, User Interface Motivation (computer literacy) and Perception
Aesthetics and Accessibility (visual acuity)
Availability of definition for terminologies Appropriateness Recognisability, Learnability and Motivation (computer literacy)
Accessibility
Layout
Organization of Information User Interface Aesthetics, Appropriateness Cognition (design complexity – simple design may
Recognisability, Learnability and Accessibility enhance the performance)
Standard page design on each screen User Interface Aesthetics and Appropriateness Cognition (design complexity – simple design may
Recognisability enhance the performance)
Sequence of screens/Flow of tasks User Interface Aesthetics, Appropriateness Cognition (design complexity – simple design may
Recognisability and Learnability enhance the performance)
Use of same symbols and icons on each screen Appropriateness Recognisability, User Interface Cognition (Memory and Recall) and Motivation
Aesthetics and Accessibility (computer literacy)
Same position of display messages on each screen User Interface Aesthetics, Appropriateness Cognition (design complexity – simple design may
Recognisability and Accessibility enhance the performance)
Navigation buttons at the same place on User Interface Aesthetics, Appropriateness Cognition (design complexity – simple design and
each screen Recognisability and Accessibility easy navigation may enhance the performance)
Labels at the same place on each screen User Interface Aesthetics, Appropriateness Cognition (design complexity – simple design may
Recognisability and Accessibility enhance the performance)
Navigation
Spacious click sensitive areas Appropriateness Recognisability, Operability and Perception (Peripheral vision) and Physical (ease in
Accessibility locating items on the screen)
Step by step easy to follow navigation Operability, Learnability and Accessibility Cognition (design complexity – simple design and
easy navigation may enhance the performance)
Clearly visible links and large buttons or icons Appropriateness Recognisability, Operability and Perception (visual acuity), Cognition (easy to locate
Accessibility and interact with the application) and Physical
(Motor skills)
Labelled buttons or icons (simple and meaningful) Appropriateness Recognisability and Accessibility Cognition
Underlined hyperlinks Operability, Appropriateness Recognisability and Cognition (design complexity – simple design and
Accessibility easy navigation may enhance the performance)
No pull down menus Operability Cognition (easy navigation may enhance
performance)
No automatic scrolling Operability and Accessibility Cognition (easy navigation may enhance
performance), Perception (motion perception)
and Physical (Hand-eye coordination)
Buttons for previous page and next page Operability Cognition (easy navigation may enhance
performance)
Flexible control Operability Cognition (easy navigation may enhance
performance)
Learn and use
Need the support of a technical person to be able Learnability Motivation (belief and acceptability)
to use the application
Need to learn many things before proper use of Learnability Motivation (Computer Literacy, belief and
this application acceptability)
Easy to operate the system Operability Motivation (belief) and Cognition (simple
interaction may enhance performance)
Performing task is straightforward Operability (user interaction) Motivation (belief) and Cognition (simple
interaction may enhance performance)
Feeling of success in the first attempt Appropriateness Recognisability Motivation (Attitude, Acceptability)
Learn to use the app very quickly Learnability Motivation (belief) and Cognition (time taken to
learn something new)
Memory load and mental effort
Easily focus on one task Operability Cognition (Attention)
Application provides ample time to read Operability (efficiency) and Accessibility Cognition (decreased processing of time restricted
and respond discrete information) and Physical (Speed of
Performance)
Instruction to be appeared on the top of the Learnability and Accessibility Cognition (memory) and Motivation
screen always (computer literacy)
Clear indication of current position Operability and Accessibility Cognition (memory) and Motivation (anxiety
and fear)
(continued)

Table 3. Continued.
Usability constructs and items
No animation or fast moving object
User guidance and error handling
Fast response of the application to the
entered data
Hint of data input is available
Alphabetically sorted answers
Frequently selected options are highlighted
Option to skip the inapplicable query
Dynamic query set based on the user responses
(e.g., in case of “No” response to a question,
application automatically skips the
related questions)
Clear and helpful error messages
Easy recovery from an error
Availability of help and documentation
UNDO option to reverse the action
Cues, instructions and user control
Availability of textual information along with the
colour codes
Multiple modes of feedback (visual, demonstrative
like vibration and audio)
Audio and visual instructions
Multiple ways to access the information (e.g., table
of contents, list of links to all the related sub-
pages, search function etc.)
Control to avoid over alerting or repeated
notifications
Adjustable text size
Figure 3. Implementation model of the usability evaluation instrument for the elderly.
INTERNATIONAL JOURNAL OF HUMAN–COMPUTER INTERACTION
Usability measures
User Interface Aesthetics, Appropriateness
Recognisability and Accessibility
Operability
User error protection, Accessibility
User error protection
User error protection
Operability
Operability
User error protection, Accessibility
User error protection, Accessibility
User error protection and learnability and
operability, Accessibility
Operability, Accessibility
Accessibility and Learnability
User Error Protection and Accessibility
User Error Protection, Learnability, Operability and
Accessibility
Accessibility and Operability
Appropriateness Recognisability, Operability and
Accessibility
Accessibility
11
Elderly barriers influencing usability
Perception (motion perception) and
Cognition (attention)
Cognition and Motivation
Motivation (technology anxiety and computer
literacy) and Cognition (less memory load)
Cognition (less memory load)
Cognition (less memory load)
Cognition (flexible interaction to enhance
performance)
Cognition (less memory load)
Motivation (technology anxiety and acceptability)
Motivation (technology anxiety and acceptability)
Motivation (computer literacy) and
Cognition (recall)
Motivation (anxiety) and Cognition (flexible control
for easy interaction)
Motivation (computer literacy) and Cognition (less
memory load)
Motivation (Technology anxiety and
computer literacy)
Motivation (computer literacy and acceptability)
and Perception (visual and auditory acuity)
Motivation (computer literacy and acceptability)
and Cognition (flexible control for easy
interaction)
Cognition (recall) and Motivation (anxiety)
Perception (visual acuity)
12 U. E. M. SHAH ET AL.

slightly better in terms of the physical load than Chronic
Pain Tracker Lite. Similarly, comparison in terms of usabil-
ity measures and usability constructs between different apps
can be done based on the respective scores.
Further breakdown of the evaluation results will help to
identify the areas of improvement in relevance to the elderly
Table 4. Usability constructs and number of corresponding items.
UC ¼ Usability constructs q ¼ no of corresponding items (i)
UC1 ¼ Readable design/Typography
UC2 5 Text writing style
UC3 5 Layout
UC4 5 Navigation
UC5 5 Learn and use
UC6 5 Memory load and mental effort
UC7 5 User guidance and error handling
UC8 5 Clues, instructions and user control
population. The detailed results as shown in Figure 7 help
to identify specific items that are well-accepted by the users,
or need further improvement despite the overall usability
barrier, measure, or construct scores being excellent or
good. In this way, the implementation of the usability evalu-
ation instrument enables viewing the detailed report for
each entity under the elderly barriers, usability measures,
and usability constructs, with respect to a particular app and
identifying the areas for improvement.
5
7
7
11 9. Discussion
6
6 The proposed instrument is comprised of usability con-
10 structs and items from the elderly’s perspective for the
6
usability evaluation of healthcare mobile applications, in

Table 5. Response options with respective weightage.

Options Very Poor Below average Average Above average Excellent
Weightage 1 2 3 4 5
Options Very confusing Below average Average Above average Very clear
Weightage 1 2 3 4 5
Options Inconsistent Below average Average Above average Consistent
Weightage 1 2 3 4 5
Options Strongly disagree Disagree Neither Agree nor Disagree Agree Strongly Agree
Weightage 1 2 3 4 5

Figure 4. Elderly barriers influencing usability: overview.

Figure 5. Elderly barriers influencing usability: tabular view.

 INTERNATIONAL JOURNAL OF HUMAN–COMPUTER INTERACTION 13

Figure 6. Detailed report of cognitive load for Manage My Pain Lite – overview in form of bar chart.

Figure 7. Detailed report of cognitive load for Manage My Pain Lite – tabular view.

particular pain management mobile applications. In the con-
text of medical information and in relevance to elderly
users, text display or presentation (i.e., the typography and
layout of text), simplicity and intuitiveness of the text to be
grasped easily by the individual, and the text’s effectiveness
are three important areas of consideration (Hartley, 1999).
To overcome the issues that arise with age-related transi-
tions, there are effective means of presenting the text
(Czaja & Sharit, 1998). Thus, healthcare mobile applica-
tions should be more accessible by adopting the
14 U. E. M. SHAH ET AL.

Table 6. Mean scores and corresponding levels of the elderly barriers influencing usability.
Elderly barriers Manage My Pain Lite Pain Companion Chronic Pain Tracker Lite Pain Diary and Community
influencing usability (Android app) (Android app) (iPhone app) CatchMyPain (iPhone app)
Cognitive load 3.78 (Good) 3.7 (Good) 3.72 (Good) 3.64 (Good)
Motivational load 3.75 (Good) 3.57 (Good) 3.64 (Good) 3.58 (Good)
Physical load 3.96 (Good) 3.96 (Good) 4.06 (Good) 4.25 (Excellent)
Perceptual load 3.54 (Good) 3.83 (Good) 3.98 (Good) 3.54 (Good)

Table 7. Mean scores and corresponding levels of the usability measures.

Manage My Pain Lite Pain Companion Chronic Pain Tracker Lite Pain Diary and Community
Usability measures (Android app) (Android app) (iPhone app) CatchMyPain (iPhone app)
Appropriateness recognisability 3.98 (Good) 3.8 (Good) 4 (Good) 3.83 (Good)
User interface aesthetics 3.83 (Good) 3.94 (Good) 4.32 (Excellent) 3.82 (Good)
Operability 3.69 (Good) 3.55 (Good) 3.55 (Good) 3.65 (Good)
Learnability 3.54 (Good) 3.48 (Good) 3.63 (Good) 3.7 (Good)
User error protection 3.25 (Average) 3.1 (Average) 2.97 (Average) 2.97 (Average)
Accessibility 3.77 (Good) 3.72 (Good) 3.85 (Good) 3.58 (Good)
Overall mean usability score 3.67 3.59 3.72 3.59

Table 8. Mean scores and corresponding levels of the usability constructs.

Manage My Pain Lite Pain Companion Chronic Pain Tracker Lite Pain Diary and Community
Usability constructs (Android app) (Android app) (iPhone app) CatchMyPain (iPhone app)
Readable design/Typography 3.6 (Good) 4.07 (Good) 4.45 (Excellent) 3.25 (Average)
Text writing style 4.1 (Good) 3.88 (Good) 4.32 (Excellent) 3.97 (Good)
Layout 3.91 (Good) 3.89 (Good) 4.29 (Excellent) 4.18 (Good)
Navigation 3.78 (Good) 3.71 (Good) 3.57 (Good) 3.78 (Good)
Learn and use 3.56 (Good) 3.54 (Good) 3.46 (Good) 3.54 (Good)
Memory load and mental effort 4.14 (Good) 4.08 (Good) 3.96 (Good) 4.08 (Good)
User guidance and error handling 3.48 (Good) 3.38 (Average) 3.3 (Average) 3.16 (Average)
Cues, instructions and user control 3.12 (Average) 3.13 (Average) 2.8 (Average) 2.65 (Average)

instructions and recommendations specific to the elderly
(Hartley, 1999).
Similarly, simple and understandable design and layout
and increasing the ease of navigation are also crucial to pro-
vide a usable application to the elderly (Arnhold et al., 2014).
Accessibility, which evaluates the application with respect to
the individuals with reduced skills (Billi et al., 2010), is gener-
ally a less examined usability characteristic (Petrie & Kheir,
2007). Nonetheless, certain accessibility concerns may impact
non-disabled users as well (Petrie & Kheir, 2007). Although
evaluation of usability does not imply that everyone’s issues
are addressed (Billi et al., 2010), yet making technological sol-
utions for healthcare needs accessible to the elderly is still
crucial in an aging society where the elderly are the majority
of pain sufferers (Holzinger et al., 2011).
Furthermore, learning innovative approaches and strat-
egies is another challenge owing to cognitive limitations in
the elderly. Providing easy-to-learn pain self-management
mobile applications reduces cognitive stress and increases
technology acceptance among elderly consumers. User guid-
ance and error prevention is also essential aspect of usability
concerning elderly users. For instance, notifying consumers
before they make any deletion, allows them to ensure their
action and contributes to a low error rate and increased
usability (Norman, 2013).
The proposed instrument was also validated by the
domain experts from the perspective of the elderly. The
experts agreed on the comprehensiveness of the identified
constructs and items, the relevancy of the items to the
constructs, and the appropriateness of the usability evalu-
ation instrument. The effectiveness of the instrument’s text
was also evaluated by the medical doctor, language expert,
and elderly participants. The validation indicates the applic-
ability of the proposed instrument for the effective assess-
ment of the usability of healthcare mobile applications from
the elderly’s perspective.
This usability evaluation instrument can be used by the
elderly working with the usability experts or mobile applica-
tion developers to produce or enhance the mHealth apps in
accordance with their needs. Thus, this instrument is a rem-
edy for the lack of the elderly population’s involvement in the
design and development of pain management mobile applica-
tions although pain occurs mostly in the elderly. The partici-
pation of the elderly population in the usability evaluation
studies can help to improve the elderly’s motivation and
acceptance of pain management mobile applications. The reli-
ability test also indicates that the proposed instrument is reli-
able for usability evaluation in the elderly’s context.
From the evaluation results with the proposed instrument
and real users, it can be concluded that the common usabil-
ity issues faced by the elderly users with pain management
mobile apps are the lack of guidance, instructions, and cues
to guide the users in using the apps. This in turn has
resulted in issues in learning and operating the apps.
Researchers and app designers can leverage on the proposed
instrument to eliminate these potential usability issues from
the app. Clinical practitioners can rely on the instrument to
filter apps fulfilling desirable usability measures before

recommending the apps to their patients. Moreover, meas-
urement of the items, via usability evaluation instrument,
also contributes to the quantification of usability measures
as well as the elderly barriers of the mHealth application.
The visualization of assessment results helps to identify the
areas of improvement.
Though efforts have been made to maintain the rigor of
the study, yet there were certain shortcomings to acknow-
ledge. One of the possible limitations is that the formulation
of instrument was guided by the SLRs with scope restricted
to five databases (although relevant literature was identified
from snowball sampling) and only patient-oriented pain
management mobile applications were addressed. Moreover,
the researchers faced difficulty in getting cooperation from
the elderly to participate in the study. It affected the sample
size. Thus, the sample size for the usability evaluation study
was limited to only sixteen old age participants.
Nevertheless, valuable data was derived from the usability
evaluation study.
The percentage of less literate participants in the usability
evaluation study was low. Thus, their opinions in using pain
management apps might be under represented. Also, partici-
pants evaluated the pain apps relevant to the platform as per
their owned smartphones. The four selected apps were plat-
form-specific and not cross-platform. Hence, the users’ per-
spective may be influenced by their smartphone platforms
and may introduce bias.
There are some potential future research goals for over-
coming aforementioned limitations. Applications that sup-
port healthcare providers in their clinical decisions could
also be considered for review in future to obtain more
insights about pain management, as pain management is a
two-way process between patients and healthcare providers.
Future research may also replicate the study protocol by
including more old age respondents specifically with lower
educational level or illiterate elderly and report the results of
the usefulness and practicality of the proposed usability
evaluation instrument. Also, usability evaluation study may
generate more insightful results by involving user evaluating
cross-platform apps.
Although the formulation of instrument was guided by
the SLRs that were focused on pain management mobile
applications, the proposed instrument could be helpful in
the usability evaluation of other mHealth apps from the eld-
erly’s perspective. Future research is required to evaluate
various mHealth apps other than pain management from
the elderly’s perspective via the proposed usability evaluation
instrument.
10. Conclusion
In this research, a usability evaluation instrument that maps
the usability measures with the elderly’s barriers is proposed.
The content of the instrument was verified by five experts
(usability, linguistics, and medical). A usability evaluation
session, followed by face validity, was performed with six-
teen elderly participants on the pre-selected pain manage-
ment mobile applications. In this research, the reliability of
INTERNATIONAL JOURNAL OF HUMAN–COMPUTER INTERACTION 15
the proposed usability evaluation instrument was tested and
the identification of usability issues of the pre-selected appli-
cations was demonstrated. The instrument also provides
graphical visualization of the mapping of usability measures
with the elderly’s barriers. The graphical details help in
identifying the areas of usability improvement and the
scores helps in comparing usability of different applications.
The usability assessment instrument serves as a compre-
hensive basis for evaluating healthcare mobile applications
with respect to the elderly population. It will be significant
for the professionals to involve real users in the usability
evaluation of pain management mobile applications and
modify or develop the app in accordance with the instru-
ment’s highlighted areas of improvement, or to assure
usability of the app. The instrument also helps clinicians to
choose among the existing pain management apps and rec-
ommend the most usable app to elderly patients as a self-
management regime.
Acknowledgments
We would like to sincerely acknowledge Professor Dr. Ng Chirk Jenn,
Dr. Anwar Suhaimi, Mr. Shanmuganathan Velu, and, Ms. Anne
Nirmala Titus for their support.
Disclosure statement
No potential conflict of interest was reported by the author(s).
Funding
This research was supported by the Universiti Malaya, Malaysia under
Grand Challenge—HTM (Wellness) Grant [GC003B—14HTM] (ICT
for Healthcare Connectedness) and Interdisciplinary Impactful
Research Grant [IIRG025B-HWB] (A Data Analytic-enabled Intelligent
Mobile App To Support Information, Communication, Exercise And
Monitoring Needs Of Oral Cancer Patients And Survivors).
References
Alroobaea, R., & Mayhew, P. J. (2014). How many participants are
really enough for usability studies? [Paper presentation]. 2014
Science and Information Conference.
Arnhold, M., Quade, M., & Kirch, W. (2014). Mobile applications for
diabetics: A systematic review and expert-based usability evaluation
considering the special requirements of diabetes patients age 50
years or older. Journal of Medical Internet Research, 16(4), e104.
https://doi.org/10.2196/jmir.2968
Association, U. E. P. (2005). Usability body of knowledge. In Principles
for usable design. http://www.usabilitybok.org/principles-for-usable-
design.
Bakshi, N., Stinson, J. N., Ross, D., Lukombo, I., Mittal, N., Joshi,
S. V., Belfer, I., & Krishnamurti, L. (2015). Development, content
validity, and user review of a web-based multidimensional pain
diary for adolescent and young adults with sickle cell disease. The
Clinical Journal of Pain, 31(6), 580–590. https://doi.org/10.1097/AJP.
0000000000000195
Bark, I., Følstad, A., & Gulliksen, J. (2006). Use and usefulness of HCI
methods: Results from an exploratory study among Nordic HCI
practitioners. In T. McEwan, J. Gulliksen, & D. Benyon (Eds.),
People and computers XIX—The bigger picture (pp. 201–217).
Springer.
16 U. E. M. SHAH ET AL.
Becker, S. A., & Webbe, F. M. (2006). Designing for older adult users of
handheld technology [Paper presentation]. 2006 International
Conference of the IEEE Engineering in Medicine and Biology
Society.
Bhattarai, P., & Phillips, J. L. (2017). The role of digital health technol-
ogies in management of pain in older people: An integrative review.
Archives of Gerontology and Geriatrics, 68(January–February), 14–24.
https://doi.org/10.1016/j.archger.2016.08.008
Billi, M., Burzagli, L., Catarci, T., Santucci, G., Bertini, E., Gabbanini,
F., & Palchetti, E. (2010). A unified methodology for the evaluation
of accessibility and usability of mobile applications. Universal Access
in the Information Society, 9(4), 337–356. https://doi.org/10.1007/
s10209-009-0180-1
Bowling, A. (2014). Research methods in health: Investigating health
and health services. McGraw-Hill Education (UK).
Brinkman, W.-P. (2009). Design of a questionnaire instrument. In
Handbook of mobile technology research methods (pp. 31–57). Nova
Publisher.
Caboral-Stevens, M., Whetsell, M. V., Evangelista, L. S., Cypress, B., &
Nickitas, D. (2015). USABILITY framework for older adults.
Research in Gerontological Nursing, 8(6), 300–306. https://doi.org/10.
3928/19404921-20150522-02
Chiwaridzo, M., Chikasha, T. N., Naidoo, N., Dambi, J. M.,
Tadyanemhandu, C., Munambah, N., & Chizanga, P. T. (2017).
Content validity and test-retest reliability of a low back pain ques-
tionnaire in Zimbabwean adolescents. Archives of Physiotherapy,
7(1), 3. https://doi.org/10.1186/s40945-017-0031-y
Chuna, Y. J., & Patterson, P. E. (2012). A suggestion for future research
on interface design of an Internet-based telemedicine system for the
elderly. Work, 41(Suppl 1), 353–356. https://doi.org/10.3233/WOR-
2012-0181-353
Collins, K. M., Onwuegbuzie, A. J., & Jiao, Q. G. (2007). A mixed
methods investigation of mixed methods sampling designs in social
and health science research. Journal of Mixed Methods Research,
1(3), 267–294. https://doi.org/10.1177/1558689807299526
Creswell, J. W., & Creswell, J. D. (2017). Research design: Qualitative,
quantitative, and mixed methods approaches. Sage publications.
Cronbach, L. J. (1951). Coefficient alpha and the internal structure of
tests. Psychometrika, 16(3), 297–334. https://doi.org/10.1007/
BF02310555
Czaja, S. J., & Sharit, J. (1998). Ability–performance relationships as a
function of age and task experience for a data entry task. Journal of
Experimental Psychology: Applied, 4(4), 332. https://doi.org/10.1037/
1076-898X.4.4.332
Czaja, S. J., Sharit, J., Lee, C. C., Nair, S. N., Hernandez, M. A., Arana,
N., & Fu, S. H. (2013). Factors influencing use of an e-health web-
site in a community sample of older adults. Journal of the American
Medical Informatics Association, 20(2), 277–284. https://doi.org/10.
1136/amiajnl-2012-000876
Darroch, I., Goodman, J., Brewster, S., & Gray, P. (2005). The effect of
age and font size on reading text on handheld computers [Paper pres-
entation]. IFIP Conference on Human-Computer Interaction.
Davis, L. L. (1992). Instrument review: Getting the most from a panel
of experts. Applied Nursing Research, 5(4), 194–197. https://doi.org/
10.1016/S0897-1897(05)80008-4
de Vet, H. C., Terwee, C. B., Mokkink, L. B., & Knol, D. L. (2011).
Measurement in medicine: A practical guide. Cambridge University
Press.
Dunham, M., Bonacaro, A., Schofield, P., Bacon, L., Spyridonis, F., &
Mehrpouya, H. (2021). Smartphone applications designed to
improve older people’s chronic pain management: An integrated
systematic review. Geriatrics, 6(2), 40. https://doi.org/10.3390/
geriatrics6020040
Foraker Labs. (2012). Glossary j Usability first. http://www.usabilityfirst.
com/glossary.
Gereau, R. W., Sluka, K. A., Maixner, W., Savage, S. R., Price, T. J.,
Murinson, B. B., Sullivan, M. D., & Fillingim, R. B. (2014). A pain
research agenda for the 21st century. The Journal of Pain, 15(12),
1203–1214. https://doi.org/10.1016/j.jpain.2014.09.004
Goodman, J., Brewster, S. A., & Gray, P. (2005). How can we best use
landmarks to support older people in navigation? Behaviour &
Information Technology, 24(1), 3–20. https://doi.org/10.1080/
01449290512331319021
Hartley, J. (1999). What does it say? Text design, medical information,
and older readers. In R. J. Hartley (Eds.), Processing of medical infor-
mation in aging patients: Cognitive and human factors perspectives
(pp. 233–248). Psychology Press.
Hassan, H., & Md Nasir, M. H. N. (2008). The use of mobile phones
by older adults: A Malaysian study. ACM SIGACCESS Accessibility
and Computing, 92(92), 11–16. https://doi.org/10.1145/1452562.
1452564
Heo, J., Ham, D.-H., Park, S., Song, C., & Yoon, W. C. (2009). A frame-
work for evaluating the usability of mobile phones based on multi-level,
hierarchical model of usability factors. Interacting with Computers,
21(4), 263–275. https://doi.org/10.1016/j.intcom.2009.05.006
Holzinger, A., Searle, G., & Nischelwitzer, A. (2007). On some aspects
of improving mobile applications for the elderly (Universal Access in
Human Computer Interaction). In Coping with diversity (pp.
923–932). Springer.
Holzinger, A., Searle, G., & Wernbacher, M. (2011). The effect of pre-
vious exposure to technology on acceptance and its importance in
usability and accessibility engineering. Universal Access in the
Information Society, 10(3), 245–260. https://doi.org/10.1007/s10209-
010-0212-x
Huguet, A., McGrath, P. J., Wheaton, M., Mackinnon, S. P., Rozario,
S., Tougas, M. E., Stinson, J. N., & MacLean, C. (2015). Testing the
feasibility and psychometric properties of a mobile diary (myWHI)
in adolescents and young adults with headaches. JMIR mHealth and
uHealth, 3(2), e39. https://doi.org/10.2196/mhealth.3879
Lam, K. W., Hassan, A., Sulaiman, T., & Kamarudin, N. (2018).
Evaluating the face and content validity of an instructional technol-
ogy competency instrument for university lecturers in Malaysia.
International Journal of Academic Research in Business and Social
Sciences, 8(5), 367–385. https://doi.org/10.6007/IJARBSS/v8-i5/4108
Lindeman, D. (2015). mHealth technologies: Applications to benefit older
adults. Center for Technology and Aging.
Lynn, M. R. (1986). Determination and quantification of content
validity. Nursing Research, 35(6), 382–386. https://doi.org/10.
1097/00006199-198611000-00017
Mendez, J., & Mendoza, A. (2013). A conceptual framework to evaluate
usability in mobile aged care applications: A health care initiative.
CONF-IRM.
Mira, J. J., Navarro, I., Botella, F., Borras, F., Nuno-Solinis, R., Orozco, D.,
Iglesias-Alonso, F., Perez-Perez, P., Lorenzo, S., & Toro, N. (2014). A
Spanish pillbox app for elderly patients taking multiple medications:
Randomized controlled trial. Journal of Medical Internet Research,
16(4), e99. https://doi.org/10.2196/jmir.3269
National Institute on Aging & National Library of Medicine. (2001).
Making your web site senior friendly: A checklist. US Government
Printing Office.
Nielsen, J. (2012). Usability 101: Introduction to usability (2012). http://
www.nngroup.com/articles/usability-101-introduction-to-usability/.
Nimmanterdwong, Z., Boonviriya, S., & Tangkijvanich, P. (2022).
Human-centered design of mobile health apps for older adults:
Systematic review and narrative synthesis. JMIR mHealth and
uHealth, 10(1), e29512. https://doi.org/10.2196/29512
Norman, D. (2013). The design of everyday things: Revised and
expanded edition. Constellation.
Oluwatayo, J. A. (2012). Validity and reliability issues in educational
research. Journal of Educational and Social Research, 2(2), 391–400.
https://www.richtmann.org/journal/index.php/jesr/article/view/11851
Paiva, J. O., Andrade, R. M., de Oliveira, P. A. M., Duarte, P., Santos,
I. S., Evangelista, A. L. d P., Theophilo, R. L., de Andrade, L. O. M.,
& Barreto, I. C. d H. (2020). Mobile applications for elderly health-
care: A systematic mapping. PloS One, 15(7), e0236091.
Parker, S. J., Jessel, S., Richardson, J. E., & Reid, M. C. (2013). Older
adults are mobile too! Identifying the barriers and facilitators to
older adults’ use of mHealth for pain management. BMC Geriatrics,
13(1), 43.

Petrie, H., & Kheir, O. (2007). The relationship between accessibility
and usability of websites [Paper presentation]. Proceedings of the
SIGCHI Conference on Human Factors in Computing Systems.
https://doi.org/10.1145/1240624.1240688
Plaza, I., MartıN, L., Martin, S., & Medrano, C. (2011). Mobile applica-
tions in an aging society: Status and trends. Journal of Systems and
Software, 84(11), 1977–1988. https://doi.org/10.1016/j.jss.2011.05.035
Rasmussen, J. (1986). Information processing and human-machine inter-
action. An approach to cognitive engineering. Elsevier Science Inc.
Rogers, E. (1995). Diffusions of innovation (4th ed.). The Free Press.
Rogers, W. A., & Fisk, A. D. (2010). Toward a psychological science of
advanced technology design for older adults. Journals of Gerontology
Series B: Psychological Sciences and Social Sciences, 65(6), 645–653.
https://doi.org/10.1093/geronb/gbq065
Shah, U. M., & Chiew, T. K. (2017). Usability issues in healthcare
mobile applications for the elderly – A review of the literature [Paper
presentation]. The 6th International Conference on Computer
Science and Computational Mathematics (ICCSCM).
Shah, U. M., & Chiew, T. K. (2019). A systematic literature review of
the design approach and usability evaluation of the pain manage-
ment mobile applications. Symmetry, 11(3), 400. https://doi.org/10.
3390/sym11030400
Shneiderman, B., & Norman, K. (1992). Questionnaire for User Interface
Satisfaction (QUIS). [Paper presentation]. Designing the User Interface,
Strategies for Effective Human-Computer Interaction.
Siek, K. A., Rogers, Y., & Connelly, K. H. (2005). Fat finger worries:
How older and younger users physically interact with PDAs [Paper
presentation]. IFIP Conference on Human-Computer Interaction.
Sikorski, M. (2002). A framework for developing the on-line HCI gloss-
ary: Technical report. Technical University of Gdansk.
Silva, P. A., Holden, K., & Jordan, P. (2015). Towards a list of heuristics
to evaluate smartphone apps targeted at older adults: A study with
apps that aim at promoting health and well-being [Paper presentation].
2015 48th Hawaii International Conference on System Sciences.
Six, J. M., & Macefield, R. (2016). How to determine the right number
of participants for usability studies. https://www.uxmatters.com/mt
Sterns, A. A. (2005). Curriculum design and program to train older
adults to use personal digital assistants. The Gerontologist, 45(6),
828–834. https://doi.org/10.1093/geront/45.6.828
Tuckman, B. W., & Harper, B. E. (2012). Conducting educational
research. Rowman & Littlefield Publishers.
INTERNATIONAL JOURNAL OF HUMAN–COMPUTER INTERACTION 17
Vrbnjak, D., Pahor, D., Nelson, J. W., & Pajnkihar, M. (2017). Content
validity, face validity and internal consistency of the Slovene version
of Caring Factor Survey for care providers, caring for co-workers
and caring of managers. Scandinavian Journal of Caring Sciences,
31(2), 395–404. https://doi.org/10.1111/scs.12338
Wildenbos, G. A., Peute, L., & Jaspers, M. (2018). Aging barriers influenc-
ing mobile health usability for older adults: A literature based frame-
work (MOLD-US). International Journal of Medical Informatics,
114(June), 66–75. https://doi.org/10.1016/j.ijmedinf.2018.03.012
Wildenbos, G. A., Peute, L. W., & Jaspers, M. W. (2015). A framework
for evaluating mHealth tools for older patients on usability. MIE.
Wong, C. Y. (2011). Exploring the relationship between mobile phone
and senior citizens: A Malaysian perspective. The International Journal
of Recent Trends in Human Computer Interaction (IJHCI), 2(2), 65–77.
https://www.cscjournals.org/library/manuscriptinfo.php?mc=IJHCI-27
Wynd, C. A., & Schaefer, M. A. (2002). The osteoporosis risk assess-
ment tool: Establishing content validity through a panel of experts.
Applied Nursing Research, 15(3), 184–188. https://doi.org/10.1053/
apnr.2002.34243
About the authors
Umm e Mariya Shah received the PhD degree in Computer Science
from Universiti Malaya, Malaysia in 2021. She remained associated
with the software industry and also served in the academic sector as
a Visiting Lecturer and as a Research Assistant. Her research inter-
est includes usability engineering, human-computer interaction
and mHealth.
Thiam Kian Chiew is an associate professor in Software Engineering.
He heads the Research and Innovation in Software Engineering (RISE)
Research Group in the Faculty and was the Deputy Dean
(Postgraduate) in 2016–2019. His research areas include usability, per-
formance and interoperability of web-based systems, software engineer-
ing in e-health and m-health.
Yasir Mehmood is technical project manager at Syslab Technologies
Sdn Bhd, Selangor, Malaysia. He received the Master’s degree in
Computer Science (research mode) from Universiti Malaya, Malaysia
in 2018. His research interest includes Opinion Mining, Sentiment
Analysis and Social Media.