Health and Technology (2020) 10:1045–1061

https://doi.org/10.1007/s12553-020-00451-4

REVIEW PAPER

A systematic review into the assessment of medical apps:

motivations, challenges, recommendations
and methodological aspect
A. H. Alamoodi 1 & Salem Garfan 1 & B. B. Zaidan 1 & A. A. Zaidan 1 & Moceheb Lazam Shuwandy 2 & Mussab Alaa 1 &
M. A. Alsalem 3 & Ali Mohammed 4 & A. M. Aleesa 5 & O. S. Albahri 1 & Ward Ahmed Al-Hussein 6 & O. R. Alobaidi 4

Received: 27 April 2020 / Accepted: 11 June 2020 / Published online: 23 June 2020
# IUPESM and Springer-Verlag GmbH Germany, part of Springer Nature 2020

Abstract
Recent years have shown significantly pervasive interest in mobile applications (hereinafter “apps”). The number and popularity
of these apps are dramatically increasing. Even though mobile apps are diverse, countless ones are available through many
platforms. Some of these apps are not useful nor do they possess rich content, which benefits end users as expected, especially in
medical-related cases. This research aims to review and analyze articles associated with medical app assessment across different
platforms. This research also aimed to provide the best practices and identify the academic challenges, motivations and recom-
mendations related with quality assessments. In addition, a methodological approach followed in previous research in this
domain was also discussed to give some insights for future comers with what to expect. We systematically searched articles
on topics related to medical app assessment. The search was conducted on five major databases, namely, Science Direct,
Springer, Web of Science, IEEE Xplore and PubMed from 2009 to September 2019. These indices were considered sufficiently
extensive and reliable to cover our scope of the literature. Articles were selected on the basis of our inclusion and exclusion
criteria (n = 72). Medical app assessment is considered a major topic which warrants attention. This study emphasizes the current
standpoint and opportunities for research in this area and boosts additional efforts towards the understanding of this research field.

Keywords Apps assessment . Mobile apps . Medical apps . Assessment . Evaluation

Electronic supplementary material The online version of this article
(https://doi.org/10.1007/s12553-020-00451-4) contains supplementary
material, which is available to authorized users.
* A. H. Alamoodi
Alamoodi@fskik.upsi.edu.my
1
Department of Computing, Sultan Idris University of Education
(UPSI), Tanjong Malim, Malaysia
2
Computer Science Department, College of Computer Science and
Mathematics, Tikrit University (TU), Tikrit, Iraq
3
Department of Management Information System, College of
Administration and Economics, University of Mosul, Mosul, Iraq
4
Faculty of Engineering and Built Environment, Universiti
Kebangaan Malaysia (UKM), Bangi, Selangor, Malaysia
5
Department of Engineering Technology, Universiti Tun Hussein
Onn (UTHM), Batu Pahat, Malaysia
6
Faculty of Computer Science and Information Technology,
University of Malaya (UM), Kuala Lumpur, Malaysia
1 Introduction
Recent years have shown significantly pervasive interest in
mobile applications [1]. Their daily utilisation has been in-
grained in the daily lives of people worldwide, and statistics
suggest that apps users reached approximately 2.5 billion in
2019 [2]. Mobile apps are identified as apps designed to be
executed over mobile devices [3, 4]. The number and popu-
larity of these apps are dramatically increasing, owing to the
cadence adoption of smartphones [5] and due to other bene-
fits, including their effectiveness [6], accessibility [7] and con-
venience [8] not only for end users, but also for developers
who are encouraged to create apps rather than web-based ser-
vices [5]. Owing to this popularity, apps have shown tremen-
dous diversity in their content [5] ranging from commerce [9],
education [10], medicine [11], gaming [12], sports [13] and
more. Countless mobile apps run on different mobile devices
(e.g. iPhone, BlackBerry and Windows) [14] and across dif-
ferent operating systems, such as Google’s Android,
Microsoft’s Windows, Symbian’s Horizon [15], Apple’s iOS
1046
[16] and BlackBerry OS [17]. Majority of these apps are free
of charge, and the market size was notably more than $90
billion in 2018 [18] and is expected to continue to grow rap-
idly by 2022 [19]. Even though mobile apps are diverse and
several of them are available on many platforms; an end user
would not need to use or install them all for the intended
purpose. Some of these apps are not useful nor do they possess
rich content, which benefits the end user as expected, espe-
cially in medical-related cases. For example, patients or nor-
mal users tend not to focus on the reliability and authenticity
of the knowledge in these apps, yet they utilize them as sub-
stantial source of medical guidance, in spite of the fact that
such apps can contain falsified or inaccurate information.
Such thing can worsen the health condition of their users
leaving them with health degradation or even worse condi-
tions. Therefore, it comes with no surprise that assessing such
medical applications for their content warrants considerable
attention for different people including those in the medical
domains and their patients, let alone enthusiasts like software
developers and researchers from different scientific disciplines
who are interested in such studies.
Systematic assessment or quality assessment is neces-
sary to evaluate app content [20]. Such assessments re-
quire far beyond knowledge and experience and are often
time consuming and technically demanding [11]. This pro-
cess is not unified and it sometimes requires consistent
subjectivity [21]. Other challenges hindering app quality
assessments include the slow-paced scientific method
aimed for app evaluation. [22], lack of features that could
help identify potentially useful apps [23] and scarcity of
available apps that have been empirically evaluated [24].
Pursuing this domain of assessment was encouraged for
different reasons, including quality of information in an
app [11, 25] or information for an app [26], which would
also aid in creating a quality-based app [2] and consider
ways of assuring app’s quality [27]. Literature also calls
for mechanisms of quality control and validation of apps
that need to be addressed [28], framework development
for the evaluation of apps [29], comprehensive audit of
their quality [30, 31] and a new tool of multiplatform
app quality assessment [32, 33] were also needed.
For such reasons, smartphone applications should undergo
suitable assessment and evaluation procedures. Different re-
searchers from different scientific domains have established
and performed various approaches for assessing medical apps
for the purpose of showing their adherence and suitability.
Still, there are no unified guidelines or a clear checklist a
developer of these apps must follow prior to development,
especially if such apps are aimed for different scientific pur-
poses. Therefore, this study aims to review this domain by
using a systematic literature approach to understand its major
components and discuss certain steps. The following Fig. 1
illustrates the main highlight of this paper.
Health Technol. (2020) 10:1045–1061
2 Systematic review protocol
This study is based on systematic review approach or systematic
literature review (SLR). This type of review enables a compre-
hensive understanding of a certain topic of interest or phenome-
non, which provides details and important insights for policies
and future studies [34]. SLR is also a structured approach to
research synthesis, following a number of pre-determined steps
to evaluate the state of the literature on a specific topic [35].
Moreover, SLR employs a methodological and systematic pro-
cess to identify, choose and critically appraise relevant pieces of
research to generate collective insights from past research [36].
Moreover, SLR has various merits over conventional approaches
as it can synthesise the literature in a systematic, transparent and
reproducible manner [37]. Systematic review approach has been
long recognised for its importance and its ability to accommodate
various types of research methods, not only for researchers across
scientific disciplines, but also for post-graduate students who
consider systematic review an integral objective of their study
or even as their entire dissertation [38]. The process of a system-
atic review includes several stages, such as the identification of
research area, search procedure, criteria for study selection, data
extraction process and data synthesis [39]. In its core, a system-
atic literature review aims at summarising the topic being studied
and identifying gaps in current research to position new research
activities [40]. The individual studies used in the systematic re-
view are called primary studies. The systematic review is often
referred to as secondary study.
2.1 Information source
The applied search strategy in this review was based on pre-
ferred reporting items for systematic reviews and meta-analyses
statement [34], which is summarised in Fig. 2. Five scientific
digital databases were chosen for article search, download, fil-
tration, extraction and drafting of this review; (1) Science Direct
database offers extensive access to scientific papers across dif-
ferent academic disciplines, (2) Scopus database also offers
various publication content related to various domains of sci-
ence. (3) IEEE Xplore manifests scientific soundness and ex-
pertise on technology and engineering publications on different
technological domains. (4) Springer Link database features dif-
ferent publications from multidisciplinary sciences. (5) Web of
Science has a wide range of publications across different do-
mains, such as social sciences, arts and the humanities. The
utilized databases are noteworthy in the academic field and
present various scientific publications on high-impact journals;
therefore, they were selected and used in this review.
2.2 Search strategy
The search was initiated at the end of September 2019 using
the advanced search boxes of the five scientific databases (i.e.
Health Technol. (2020) 10:1045–1061
Fig. 1 Main highlights
Science Direct, Scopus, IEEE Xplore, Springer Link and Web
of Science). Boolean operator was used for the search (i.e.
AND, OR), and three groups of keywords (i.e. queries) were
used in the process as presented on Fig. 2. During the process
of searching and filtration, content was only selected on the
basis of research articles and review articles. This option was
deemed most suitable to circle the latest and most related
content relevant to the designated topic of this review.
1047
2.3 Study selection
The study selection process started with an initial
search, which resulted in (n = 4331) articles. These arti-
cles underwent the following processes: (1) duplication
screening for articles from selected databases; (2) title
and abstract scanning to determine initial relevance of
articles to the topic; (3) and full text reading and data
1048
Fig. 2 Systematic review protocol
extraction of each article matching the inclusion criteria
of the protocol. Several author notes and comments
were taken during the process, which later on turned
into good insights that helped shape the final form of
this review. Different elements were selected and ex-
tracted into Microsoft Excel.
Health Technol. (2020) 10:1045–1061
2.4 Inclusion and exclusion criteria
Several criteria were enforced for the selection of stud-
ies in the midst of the screening process. Date of pub-
lication was set between 2009 and September 2019.
Only review and research papers written in the English
Health Technol. (2020) 10:1045–1061
language were included. Other remaining criteria were
concerned with topics on app assessment and evaluation
(i.e. mobile app) for apps downloaded from mobile apps
stores (i.e. Google Play, ITunes, etc.). Some articles
reviewed mobile apps on previous studies, and no app
stores existed as main source of apps to be assessed.
This latter type was excluded as we are only targeting
app assessment from app stores. Figure 2 presents the
inclusion and exclusion criteria and the queries and
search results.
3 Motivations
Researchers and academics are drawn to their respected fields
for various scientific reasons. Some are encouraged by the
significance of topics like hot trends research topics, latest
topics and specials issues in scientific journals, while some
are merely enthusiast with the benefits the topics can present,
like curing a disease or raising awareness or any sort of con-
tributions. For systematic app assessment over mobile apps,
different significances and motivations show researchers so-
licitude in this area. These motivations are summarized and
grouped in the following Fig. 3.
3.1 Related to strategies
In the world of mobile apps and quality assessment, strategies
are recognised as a very valuable assist resulting from such
technologies. Strategies also encourage interaction between
an app and a user [41]. In the world of academia, researchers
are motivated for different reasons, and amongst them are the
beneficial strategies related to the medical domain. These
strategies benefit decision making cases, like disease screen-
ing [41] and tracking [42], medication dosage suggestions
[43] and adherence [23, 44–46]. In addition, other benefits
Fig. 3 Motivations overview
1049
related to management include future interventions planning
[21] and assistance for clinicians [31] and patients [46–49].
3.2 Related to information
Information is important in mobile apps to meet the needs of
users [50]. For the literature on mobile app assessment, differ-
ent benefits are presented with regard to information, includ-
ing details on step-by-step instructions [51], availability [52]
and content suitability [33, 53] which in turn can be beneficial
to different people [53] and provide guidance [53, 54] and
compliance [55].
3.3 Related to mobile apps
When it comes to mobile apps and quality assessments, stud-
ies that are linked to using apps are warranted. Based on the
literature analysed, majority of mobile apps benefits fall in one
of two major grouping: (1) APP functions; where all aspects
within app development and features are discussed or (2) APP
usages; where it’s various usages are highlighted. The first
category’s benefits include real-time connection [56] and sup-
port [57], tele monitoring [58, 59], customisability [22], sim-
plicity [60], remote instructions [61], continuous development
[43], wide array of functions [62] and accessibility [22]. The
benefits associated with using mobile apps include usage con-
venience [63], accessibility [45, 64–66] availability [67], pa-
tient empowerment [68] connecting people together [69] and
replacing traditional approaches [70] due to their wide adop-
tion [5, 71].
3.4 Related to outcome
In mobile app quality assessment, different benefits related to
the outcome are presented. Some of these benefits include
better clinical and health outcomes [58, 72] for surgeries
1050
[56], disease management [31, 58], better delivered interven-
tions [20, 27, 69, 73] healthy knowledge [25, 26] and healthy
practices [25]. In addition, other benefits include promoting
self-management [2, 28, 50] and behavioural changes [20, 45,
46, 64, 74, 75], ensuring continuity of care [57] and assessing
quality of service delivery [76].
4 Challenges
Issues and challenges are amongst the most common academ-
ic dilemma a researcher may face. Whether or not directly
related to a researcher’s area of interest or have an indirect
impact, such dilemma require further efforts to address them
and advance the science in that designated domain. For sys-
tematic assessment of mobile apps, challenges are scattered
over different categories, and they are grouped on the basis of
common characteristics to ease their understanding for future
researchers and studies. (See Fig. 4).
4.1 Related to quality ASSESMENT
It comes with no surprise that quality assessments are one of
the most intriguing challenges when it comes to mobile apps
[29, 64], and that is not only due to the reliability and accuracy
of their content, also because the impact will affect others like
users of such apps. This instance indicates that even the crea-
tion of high-quality apps is challenging [2]. Most of the chal-
lenges in this domain revolve around Apps Information’s
quality and content, where literature suggested that no exten-
sive assessments have been performed [25, 66] which, in turn,
makes quality of the information in these apps contested as-
pects [41] and variables [2]. Some authors relate this notion to
different reasons, including the lack of extensive evaluation
[20], difficulty of assessment [11], slow development [57] and
assessment [31], lack of data and information [22, 26, 45,
52] or auditing [30]. Others suggest that quality assessments
are still lacking [2] due to a scarcity of comparative
assessment studies [20, 31, 66] and tools [25].
Fig. 4 Overview of challenges
Health Technol. (2020) 10:1045–1061
4.2 Related to guidelines
In app assessment, guidelines indicate whether strategies in
mobile apps are followed or not. These guidelines will also
contribute towards having ‘beneficial’ or ‘not beneficial’ evi-
dence to make a specific recommendation [28]. Having this
major impact over quality app assessment, three main deriva-
tives existed, including, (1) Official Guidelines, (2) Approval
Authorities and (3) Individuals. For the Official Guidelines,
literature was scattered with different views on why such thing
has an impact over the quality assessment of apps. Some au-
thors relate to the absence of guidelines [29], whether validated
guidelines [57], scientific evidence guidelines [23, 28, 61, 65,
77] or app development guidelines [57, 78]. Others found dis-
crepancies between information provided in an app and official
guidelines [70], which in turn, made it difficult to guarantee
quality and conformity with guidelines [61] or even proper
adherence to guidelines [79, 80]. The second phase of this
challenge concerned with Approval Authorities suggests that
a few apps are approved by organisations with solid guidelines,
like the National Health Service (NHS), or regulated by orga-
nisations like the Food and Drug Administration (FDA) [53].
For the last components of this challenge, which are related to
Individuals’ involvement in quality assessments of mobile
apps, various researchers across academic literature present dif-
ferent perspectives. Researchers suggest that lack of individual
involvement, like a physician, could result in vague guidelines
[57] or absence of input or endorsement by medical and health
professionals [29, 30, 78, 80, 81], which may result in contra-
vening evidence-based guidelines.
4.3 Related to content
The content of apps plays a significant part in their assessment
because evaluating mobile apps on the basis of their content
have appraised quality of apps [20], which, in turn, can con-
tribute to better user experience [5]. Moreover, the content of
mobile apps is often more important as it is directed at users
who may not be expected to have a sophisticated knowledge
[53]. Hence, app content should be cautiously used [55]. In the
Health Technol. (2020) 10:1045–1061
quality assessment of mobile apps, content has a tremendous
significance; however, such importance is jeopardised by
some elements. These elements include (1) unavailability,
(2) inaccuracy, (3) control, (4) selection and (5) restrictions.
Scientific literature attributes the unavailability of contents
related to mobile apps to lack of content, whether educational
[43], comprehensive experience [57] or related to the analysis
[71]. Others in this domain were more concerned with the
investigation of content sources [41], evaluating content using
a comprehensive checklist [50] or incorporating content that
adhere to evidence-based practices [44, 52, 54, 56, 57, 70].
The second element of content challenges relates to the inac-
curacy of content. Researchers faced by this challenge
discussed major effectors, which are due to the incorrect con-
tent [51], [11, 30, 31, 54, 64], limited content [11, 47] and
description [57]. The following class of challenges relates to
Control of app content. Different effectors and causes existed
including the lack of regulatory control over content [2, 53] and
due to content provided by unspecialised organisations [82],
which later brings concerns about the validity of app content
[57, 77]. The next element of content challenges is concerned
with Selection of mobile apps for quality assessment. No unified
approach was presented in this regard, and it is considered a great
challenge [23]. Some other challenges associated with this area
include the ambiguity of app selection to undergo assessment
[53], which makes it unclear to identify the best app [61] or the
most useful one [23] whether from the perspective of the app
itself or its content [50]. The final element linked to content
challenges revolves around Restrictions in quality assessments
of mobile apps. Across the academic literature, these restrictions
were related to some causes including the language of apps.
Some content of apps are only found in specific languages like
English [21, 23, 32, 61, 71, 83] and German [11]. Other apps
were only assessed from specific country stores, like Canada,
France [74], New Zealand [50], Australia [27], United States,
[24] and Singapore [23]. In addition, few authors in this regard
discussed only downloading apps from multiple stores [23, 27],
single one [11] or insufficient number of apps [24, 79]. The
next group of restrictions was concerned with targeted popula-
tion for mobile app content [30, 58]. This case shows the lack of
literature for applications aimed at the general population [56]
or different groups of end users [53, 84]. Remaining restrictions
discussed the limitation of only evaluating publicly available
apps [42, 57, 83], apps with trial period [48] or apps that are not
available to the general public [57].
4.4 Related to evaluation
Evaluating mobile apps is meant to ensure sufficient content
quality, instructional value, conformity with current guide-
lines and availability of a minimal set of features [71]. It is
worth mentioning from the latter that quality assessment of
mobile applications are with considerable importance.
1051
Therefore, there have been various approaches aimed for
them, such approaches includes author-created rating systems,
rating scales and user testing [71]. However, no widely unified
standards of quality evaluation for apps is accepted [20],
which could be due to the limited regulatory oversight around
their quality [20]. This area is no different from other compo-
nents related to quality assessments, and it is prone to numer-
ous challenges. Some of these challenges include the apparent
complexity of evaluation [32, 33] lack of evaluation [2, 20,
57, 74] for evaluated scales [29, 32], proper evaluation re-
views [59, 71], empirical evaluation [70] and lack of system-
atic evaluation [61, 64]. In addition, other evaluation chal-
lenges include scarcity of mobile apps subjected to research
evaluation [27], scientific evaluation [46, 85] or app evalua-
tion based only on their descriptions [70, 71, 74].
4.5 Related to features
Mobile applications features are with no doubt some of the most
important elements that guarantee whether an app will be a suc-
cess or failure in term of number of downloads, usage or even
quality assessment evaluation. Nevertheless, they might be faced
with different issues, which serves as a barrier in the face of
researchers. The challenges in the features domain are
categorised on the basis of four parts, including, (1)
Confidentiality, (2) Security, (3) Functionality and (4)
Usability. The confidentiality challenge is engulfed around
breaches of confidentiality [30, 57] as their main challenge.
However, data was the main issue in security [29, 82]. For the
third part linked to challenges related to apps features, according
to literature, in consideration of different views, many are
summarised according to issues like the technical demand and
time consumption [11, 22], design weaknesses [65, 80, 82] soft-
ware glitches [30], overlapping functions [31], built-in mecha-
nism [42], functioning as advertised [57], complicated user in-
terface [78], alignment with best practices [44, 86] and function-
ality differences [22], not to mention the identification of poten-
tially useful functionalities [48, 87]. The last set of challenges
linked to features is summarised according to usability issues.
App usability raises a major concern [41], and a few follow
recent design guidelines with acceptable usability for non-
expert users [61]. Furthermore, usability is faced with issues like
the lack of usability scoring for apps [31], let alone the quality
control of content usability presented in mobile apps [61] .
4.6 Related to research
In academic literature, researchers are interested to pursue qual-
ity assessment of mobile apps. However, they find themselves
handicapped by the scarcity of research in this domain. They
consider it a challenge to explore such interesting topic with
insufficient literature [5]. Scarcity includes lack of systematic
1052
reviews [53], comparison studies [58, 77], cost-effectiveness
evaluation studies [58] and research evidence studies [64].
5 Recommendations
At the end of the research journey, a researcher’s perspective on
a given topic is dramatically changed in the end. A large amount
of knowledge was obtained, and a researcher’s experience ex-
pands. He/she, at this stage, appreciates the findings and ac-
knowledges the shortcomings encountered. Moreover, mitigat-
ing such issues in the future and what sort of academic message
is supposed to be handed to fellow researchers in that domain are
some of the major goals for academicians and researchers.
Therefore, this section aims to summarise researchers’ efforts
by recommending future studies in mobile app assessment.
The sub section acknowledges three major points, including,
Content, Research and Development, and assigns their
respected recommendations accordingly. (See Fig. 5).
5.1 Related to contents
In the domain of systematic assessment of mobile apps, con-
tent is considered a significant influencer of users’ behavioural
changes [88]. Academic literature has numerous recommen-
dations associated with content. These recommendations
Fig. 5 Overview of recommendations
Health Technol. (2020) 10:1045–1061
represent different views of authors engaged in this area, in-
cluding the Accuracy of content, which was suggested by [30,
51, 81, 89] for better decision making. Aside from the accu-
racy of content, its consistency with Evidence was another
suggestion by researchers [22, 41, 70, 90] and they urge for
more evidence-based apps [33, 45, 52, 68]. More recommen-
dations suggested that content of apps should be Beneficial
[21], provide Guidance for users [56], meet their Needs [53]
and base their content on centralised Resources [64].
5.2 Related to app development
Professional Development of apps hold great importance
and it has a key role when looking at successful apps.
While considering some of the most key factors in this
regard from the literature, it is clear that many recom-
mendations are dedicated towards usability of apps
which always should be in an enhancement cycle [43],
sufficient [51] and taken into consideration [5, 41].
More Cooperation and involvement with different
parties, including users [2, 25, 49, 78], technology com-
panies [71], academicians [25, 71], industry [49], health
professionals [25, 49, 50, 79, 91] and government [25,
91] were also recommended. Aside from cooperation,
developers should aim to develop their apps on the ba-
sis of Best Practices [76, 82, 86] to assure content
quality [28, 33]. Developers should also guarantee that
Features will be beneficial for end users [23] and are
based on their needs [25, 41, 64] to provide best app
experience [5, 86]. Other recommendations targeted for
developers suggest that apps should be more Interactive
[41] and based on improved and collaborative Design
methods [20, 41, 42, 64], analysis [92] and guidelines
[70, 80].
5.3 Related to research
This section discussed previous researchers’ hopes and rec-
ommendations for the future of this domain. Majority of them
elaborated on the importance of more studies in different set-
tings, including interdisciplinary studies [61], validation stud-
ies [89, 93] assessment studies [59, 69], exploratory studies
[22, 44, 58, 88], evaluation studies [2, 22, 66] and develop-
ment studies [25, 64]. Other research recommendations were
concerned with conducting further research on different
topics, including clinical practice and mobile apps [57, 94],
analysing features to motivate end users to use these apps [61],
making apps effective [28, 66] testing the functionality and
usability of the app [33, 70, 81] including more experts [46,
61] investigating applications compliance with clinical set-
tings [73] and more reliable comprehensive quality assess-
ments tools [23, 68, 95].
Health Technol. (2020) 10:1045–1061
6 Methodological aspects
This section discuses methodological aspect of previous re-
searches with relation to mobile apps and their assessment.
Various points are discussed and identified to inform future
researchers in this area, whose settings and approaches have
been utilized before.
6.1 Health apps assessed
In the literature review, different medical, health and fitness
apps were systematically reviewed and evaluated. Different
diseases were reviewed and assessed in the app stores as on-
cology [55], prostate cancer [41, 63], cardiac arrest [51], dia-
betes [43, 44, 58, 77, 87], arthritis [57], bipolar disorder [27],
cancer [73, 74], asthma [28, 42], eczema [31], inflammatory
bowel [88], Parkinson’s [89] and gout [46]. Several studies
assessed self-management pain apps as in [50, 78]. Other
studies systematically reviewed surgery apps, such as hip re-
placement and knee replacement surgery [2], neurosurgery
[30] and bariatric surgery [29]. Moreover, some studies
assessed apps related to physical injuries, such as sports con-
cussion apps [53], rib hump angle apps [95], pelvic floor mus-
cle [56], arthritic pain [49] and burns [94]. Different mental
health apps were assessed, such as intellectual disability [33],
children and adolescent anxiety [59], social anxiety [70],
nomophobia [75] and suicide prevention [90]. One study
had already assessed different mental and physical mobile
health applications, such as stroke and spinal cord injury,
brain injury, psychoactive substance abuse and addiction, de-
pression, stress and anxiety apps [81].
Different health apps were assessed in the literature. These
apps include medical applications [84], medication adherence
[23, 45, 71, 83] and drug-drug interaction [21], accredited
health and wellness apps [82], fitness apps for people with
cancer [47] and cardiorespiratory fitness [24], dietary
weight-loss, nutrition apps [20, 48, 64, 79], and blood dona-
tion apps [5]. Apps related to cardiopulmonary resuscitation
(CPR) [61], stroke survivors and caregivers, [62], emergency
preparedness and disaster response [92], caring of elder peo-
ple [66], cycle tracking [54], pregnancy prevention, [69, 86],
children’s feedings [25, 52], palliative care [26, 67], special
education [32], and insulin dose calculators were also assessed
[91].
6.2 Apps stores used
On the basis of the literature review, most studies have
assessed apps from Apple and Google Play stores.
Nevertheless, only six studies have used Apple [11, 32, 43,
54, 62, 92], and only two studies have used Google Play [55,
64]. Several studies have used other apps stores, such as
Windows Mobile store [2, 5, 28, 53, 63, 67, 70, 89],
1053
Blackberry App World [2, 5, 26, 28, 45, 53, 63, 67, 74],
Nokia Ovi [2, 53, 74] and Samsung apps [53]. However,
one study did not mention the app stores used for app assess-
ment [80]. Table 1 provides a summary of the app stores used
in the literature.
Several other studies utilized different sources to download
their apps, these sources include Oppo Software Store, VIVO
App Store, Huawei App Market and MIUI App Store and
Sogou Mobile Assistant [31]. Another study used different
sources like MyFirstApp and Grupo Promedia [33] with
Apple and Google Play. Another study implied Chrome with
Apple and Google Play [83]. Another study used the NHS
Health Apps Library to find accredited health and wellness
apps [82].
6.3 Evaluation measurements
In the literature, different aspects of evaluations were mea-
sured. Many studies focused their evaluation on usability,
and some studies evaluated app content. Moreover, several
studies measured app quality and functionality. However,
most studies agreed that currently, no standards of evaluation
method for apps are widely accepted.
6.3.1 Usability evaluation measurement
Different usability evaluation measurements were used in the
literature. A study used a usability heuristics evaluation meth-
od questionnaire developed by an expert on the basis of
Nielsen’s 10 usability heuristics [41]. Other studies used the
system usability scale (SUS) [20, 51, 61, 84]. Another study
used a questionnaire on usability, which was developed in a
previous study, by taking the app repository guidelines into
consideration [5]. Another study used a previous study usabil-
ity criteria, which were rated by means of a 5-point Likert
scale or a dichotomous scale (0–1) [77]. Another study used
the total number of users for the app and the application rating
as indicators of app usability [29].
6.3.2 Content and app quality evaluation measurement
Different evaluation measurements were used for different
health apps. The Mobile App Rating Scale (MARS) was used
in the literature to assess the quality of apps [2, 21, 24, 26, 32,
33, 47, 50, 83]. Another study assessed the quality of prostate
cancer apps on the basis of the validated user version of
MARS called uMARS [63]. Another study used an extended
version of MARS for adult cancer survivors’ app auditing
[73]. The Quality Component Scoring System tool was used
to measure the quality of apps [52]. Moreover, a tool was
developed on the basis of the Health-Related Website
Evaluation Form, and previous studies aimed to assess the
quality of apps’ components [52]. The 9-point Silberg scale
1054 Health Technol. (2020) 10:1045–1061

Table 1 Summary of the app stores

Reference

Apple Google Play Windows Mobile Blackberry App World Nokia OVI Others

[55] x
[41] x x
[63] x x x x
[51] x x
[58] x x
[43] x
[44] x x
[77] x x
[87] x x
[57] x x
[27] x x
[74] x x x x
[73] x x
[28] x x x x
[42] x x
[31] x x
[88] x x
[89] x x x
[46] x x
[78] x x
[50] x x
[2] x x x x x
[30] x x
[53] x x x x x x
[95] x x
[56] x x
[49] x x
[94] x x
[33] x x x
[59] x x
[70] x x x
[75] x x
[90] x x
[81] x x
[84] x x
[71] x x
[83] x x x
[23] x x
[45] x x x
[21] x x
[47] x x
[24] x x
[20] x x
[79] x x
[64] x
Health Technol. (2020) 10:1045–1061 1055

Table 1 (continued)

Reference

Apple Google Play Windows Mobile Blackberry App World Nokia OVI Others

[5] x x x x
[61] x x
[62] x
[92] x
[66] x x
[54] x
[69] x x
[86] x x
[25] x x
[52] x x
[67] x x x x
[26] x x x
[32] x
[91] x x
[11] x x
[65] x x
[76] x x
[48] x x
[29] x x
[82] x

was utilised to assess the information of bariatric surgery apps
[29]. One study used an adapted scoring system, which was
developed in a previous study called the “Applications
Scoring System” and aimed to evaluate menstrual cycle track-
ing apps [54]. A tool was developed for quality assessment of
adherence apps called Medication Adherence App Quality
assessment on the basis of a systematic literature review
[23]. Behavioural content of fitness apps for people with can-
cer was evaluated by using the Behaviour Change Techniques
Taxonomy [47]. One study assessed the content of prostate
cancer apps by using an evaluation codebook derived from the
2016 ACS Prostate Cancer Prevention and Early Detection
Guidelines [41]. Another study evaluated the content quality
of cardiac arrest apps on the basis of adherence to the 2015
International Consensus on Cardiopulmonary Resuscitation
and Emergency Cardiovascular Care Science, the American
Heart Association Guidelines Update for Cardiopulmonary
Resuscitation and Emergency Cardiovascular Care 2015 and
the European Resuscitation Council Guidelines for
Resuscitation 2015, with treatment recommendations guide-
lines of the International Liaison Committee on Resuscitation
[51]. Another study assessed the quality of bipolar disorder
information apps on the basis of recent treatment guidelines
and prominent meta-analyses and reviews of evidence-based
treatment for the acute phases of the disorder. This study also
used statements derived from Colom and Vieta’s Psycho
Educational Manual for Bipolar Disorder to evaluate the com-
prehensiveness of bipolar disorder psychoeducational infor-
mation. [27]. Moreover, one study used the Sport
Concussion Assessment Tool 2 (SCAT2) and Pocket
SCAT2 for content assessment of sports concussion apps by
using a custom scoring scheme to generate a percentage com-
pliance statistic [53]. Another study developed a quality ap-
praisal for pain self-management apps on the basis of five
components of each app: interface design, clinical content,
product description, ease of use and development team [78].
Another study evaluated the content of pain self-management
apps by using a customised 14-item self-management support
checklist [50]. Furthermore, the quality of asthma apps were
assessed on the basis of a health information website devel-
oped by the Health on the Net foundation (HON) [28].
Another study evaluated the quality of asthma apps on the
basis of a set of criteria derived from the UK, US and interna-
tional asthma guidelines [42]. In addition, a study developed
an assessment criteria for diabetes apps on the basis of inter-
national medication management and diabetes guidelines
1056
[44]. Another study assessed diabetes apps on the basis of
recommendations from Andalusian Health Service, an orga-
nisation that develops quality, safety strategy and certifica-
tions for AppSaludable [77]. Furthermore, apps related to ec-
zema were assessed on the basis of conformance with inter-
national eczema guidelines. The guidelines used were from
The National Institute for Health and Care Excellence
(2007) and American Academy of Dermatology. The guide-
lines were intended for the management of atopic dermatitis in
China, the management of atopic dermatitis in Singapore, and
the Argentinean National Atopic Dermatitis Consensus 2013
[31]. The quality of neurosurgery apps were assessed on the
basis of the Healthcare Smartphone App Evaluation Tool, and
the scientific content quality of the apps were assessed by
reviewing app content and cross-checking the information
using established medical references [30]. The content of
CPR apps were assessed by a study on the basis of the Basic
Life Support guidelines [61]. The content of apps on inflam-
matory bowel disease were evaluated using the international
guidelines for inflammatory bowel disease (IBD) from the
European Crohn’s and Colitis Organization, the
Gastroenterological Society of Australia and the American
College of Gastroenterology [88]. One study assessed the
quality of arthritis apps by evaluating quality measurement
apps for chronic diseases found in scientific medical databases
(PubMed) [57]. Cognitive–behavioural intervention mobile
applications (CBAs) were evaluated on the basis of feasibility
and efficacy of smartphone-based CBAs found in scientific
articles [65]. A framework was adopted to evaluate the quality
and usability of pain apps on the basis of three levels: the
World Health Organization Innovative Care for Chronic
Conditions, Core Self-Management Skills and Stanford
Arthritis Self-Management Program [49].
Different studies evaluated the quality of apps on the basis of
related topics found in the scientific literature review, such as
caring for older people [66], suicide prevention [90], social
anxiety [70] and Parkinson’s disease [89]. Different guidelines
were followed in a study aimed to assess dietary weight loss
apps. The practice guidelines for the treatment and management
of being overweight and obesity in adults were released by the
National Health and Medical Research Council (NHMRC) and
the Dietitians Association of Australia (DAA). The healthy eat-
ing app features were extracted from the Nutrient Reference
Values (NRVs) and NHMRC Australian Dietary Guidelines.
The nutrient analysis software package FoodWorks was used
with the Australian Food and Nutrient Database 2007 to ana-
lyse the weighed food record [20]. One study developed a qual-
ity assessment tool in four domains: accountability, advertising
policy, scientific basis and functionality using the Silberg scale,
HON Foundation code (HONcode) principles, Journal of the
American Medical Association benchmarks and DISCERN rat-
ing tool [25]. Smoking cessation apps were assessed on the
basis of two sets of evidence-based guidelines: the FIVE A’s
Health Technol. (2020) 10:1045–1061
framework for behavioural change and the National Institute of
Care and Excellence guidelines for smoking cessation self-help
materials [79].
6.3.3 Other evaluation measurement
Some studies evaluated the functionality of apps assessed. One
study adapted HONcode principles to evaluate the functionality
of eczema apps in terms of information reliability on the basis
of citations, claims’ justifications, authority information, ethical
standards of transparency, advertising policies and privacy [31].
Another study assessed medication adherence apps by
conducting a qualitative content analysis of users’ review
[71]. A grade-level readability was evaluated in a study that
assessed prostate cancer apps using the software readability.io,
and culture sensitivity was also evaluated using the Cultural
Sensitivity Checklist [41]. Another study evaluated the content
and functionality of adherence apps using a developed list of
desirable attributes [45]. Another study evaluated adolescent
anxiety apps on the basis of descriptive characteristics, app
functionality and adherence to evidence-based treatment prin-
ciples [59]. The functionality of inflammatory bowel diary apps
was assessed on the basis of the comprehensiveness of four
parameters, namely, bowel habit, abdominal pain, dietary in-
take and medication [88]. A standardised tool was developed to
evaluate features of cardiorespiratory fitness apps on the basis
of results of the related scientific literature review [24].
App popularity was assessed in a study using App Annie to
evaluate market visibility [92]. One study developed an assess-
ment tool on the basis of research results of the scientific data-
bases, promising best practices for Mobile Criteria for
Adolescent Pregnancy Prevention (mCAPP), which was aimed
to extract data from evidence [69]. Another study used evidence-
based clinical practice guidelines for unintended pregnancy pre-
vention. These guidelines were extracted from the scientific lit-
erature and federal and nongovernmental programmes [86].
6.4 Inclusion criteria
In the literature, some inclusion criteria were commonly used
in most of the studies. These criteria are:
& Inclusion of a certain language
& Inclusion of the availability of apps in the apps store in one
country or more
& Inclusion of paid apps or the free versions only
& Inclusion of specific features related to the study case
6.4.1 Inclusive to a certain language
In the literature, most studies set their inclusion criteria to one
language only such as English [2, 20, 23, 27–29, 41, 42, 44,
Health Technol. (2020) 10:1045–1061
46, 47, 49, 50, 52, 53, 55, 69, 70, 74, 78, 86, 91, 94], Chinese
[25], German [11] or Spanish [32, 33, 77, 83]. Moreover,
some studies expanded their inclusion criteria to more than
one language. One study included two different languages:
English and German [56] and English and Spanish [89].
Other studies included three different languages: English,
Chinese or Spanish [31] and English, Italian or French [76].
6.4.2 Limited app availability in app stores in one country
or more
Most studies in the literature restricted their app assessments
to a specific app store of a country. For instance, a search
setting was restricted to Germany in one study [51] or using
the Apple app store in Germany only [11]. Other studies used
the Apple app store in different countries, such as in Australia
[20, 27], Canada [21], New Zealand [50], Spain [77], UK [79]
and US [30, 62]. Another study used the Apple app store in
Australia, Canada, New Zealand, UK and US [53]. Another
study used app stores in Canada and France [74]. Another
study included apps developed only in China [25].
6.4.3 Inclusion of paid apps or the free versions only
Some studies in the literature included paid and free apps,
such as in [28, 31, 42, 44, 91]. One study limited the inclusion
criteria of paid apps if the app cost does not exceed $1.99 [86].
However, many studies limited their inclusion criteria to free
apps only, such as in [2, 5, 26, 32, 33, 47, 48, 69, 77, 83].
6.4.4 Inclusion of specific features related to the study CASE
Different studies set their app assessments on the basis of app
descriptions of the apps stores, such as in [11, 41, 57] or on the
basis of the main category of the app, such as in [11, 32, 33,
76, 83]. Other studies set their inclusion criteria on the basis of
target users of the app, such as patients [2, 28, 31, 42, 43, 46,
50, 62, 74, 77, 78, 91], professionals [53], caregivers [27, 62],
teenagers [69], adults [47], parents [25, 52], pregnant women
[86] or the general population [41].
Some studies set their inclusion criteria on the basis of content
and features of app relatedness to the study case, such as infor-
mation about prostate cancer [41], asthma diagnosis [28, 42],
eczema diagnosis [31], Parkinson’s disease [89], gout self-
management [46], exercises for the pelvic floor [56], total hip
replacement or total knee replacement [2], social anxiety [70],
medication self-management features [44] or related to medica-
tion [94]. These criteria focused on behaviours related to physical
activities [47], provided strategies for pain management [50],
including components related to pregnancy prevention [69,
86], relevant to palliative care [26, 67], contained issues of a
health management unit [76], contained information on stroke
risk factors and assisted coordination among caregivers [62].
1057
These criteria were based on the main functionality of an app
[5, 20, 24, 25, 27, 43, 49, 53, 78, 91, 94], specific features that
should be in the app [25, 26] and development or updates in a
specific year and onwards [25, 49]. One study included apps if
their rating was 4 or higher [64]. Another study included avail-
able apps in the NHS health apps library that are compatible with
Android and IOS systems only [82]. Another study included an
app if it was available in the iPhone operating system and
Android platform [23].
6.5 Exclusion criteria
Most of the exclusion criteria in the literature were the opposite
of the inclusion criteria. For instance, apps that were not in [2,
41, 43, 54, 57, 62, 66, 69, 73, 75, 84, 88, 90], German [11],
Spanish [32, 83], English [51], Spanish [24], Arabic, nor
French [81] were excluded. Another exclusion criterion was
duplicated apps [11, 29, 32, 41, 43, 44, 47, 51, 54, 56, 57, 64,
70, 75, 83]. Moreover, apps that did not work, outdated or
incompatible with smartphone operating systems were exclud-
ed [5, 24, 28, 31, 32, 42, 44, 47, 52, 57, 73, 77, 82, 91].
Unavailable apps in a particular app store were excluded [32,
42, 73], and if an app was meant for advertisement purposes
only [2, 50, 62, 71].
Some studies set their exclusion criteria on the basis of target
users, such as excluding an app if it targeted professional users
[2, 41, 43, 62, 77], providers [31, 54, 57], adults [69], kids [30,
73] or if an app was not targeting patients [88]. Some studies
excluded an app if it was categorised as a game or an unrelated
category [2, 11, 20, 57, 66, 69, 84, 92] or if an app was devel-
oped for an event or conference [69, 86]. Other studies exclud-
ed apps on the basis of app functionality or if its features failed
to meet the study inclusion criteria [2, 30, 31, 41, 44, 45, 50, 53,
54, 69, 78, 83]. Moreover, some studies excluded some app
stores [51]. Multiple studies excluded the paid version of apps
[2, 25, 32, 47, 50, 51, 54, 66, 83], if an app has ongoing sub-
scription fees [73], or if the app cost is more than the amount set
for the inclusion criteria [82]. One study excluded apps that do
not provide step-by-step support [51]. Apps requiring wearable
devices were also excluded [47].
After reviewing and identifying the best practices from the
literature, systematic steps were performed by other re-
searchers in the area of app assessment (See Appendix) .
6.6 Apps assessment like
In the literature, some studies did not conduct a full app as-
sessment. These studies conducted a systematic review of sci-
entific articles to extract apps related to their study case. Then,
these studies assessed those particular apps or searched for
some them in these app stores to check these apps’ availability
[22, 60, 68, 72, 85, 93]. Different applications were aimed for
different goals for app assessment, and different researchers
1058
set different objectives for themselves. This subsection re-
flects their experience and summarises their respected works.
Numerous discussions focus on applications related to
medical conditions. In cancer prevention apps, prostate cancer
apps provided thoroughly covered PrCA content, but had the
lowest readability. Best Prostate Cancer Treatment and
Oncotip apps also provided thoroughly covered PrCA content
relative to American Cancer Society (ACS) Prevention and
Detection guidelines. Procee app is highly interactive, educa-
tional and sensitive to African Americans [41]. Another study
related to prostate cancer ranked Rotterdam Prostate Risk the
best [63]. Cardiac arrest is a major health concern affecting
primarily elders worldwide. Thus, the study in [51] concluded
that HELP Notfall is the only app that scored above average
and was highly rated by emergency physicians. Medscape,
Skyscape Medical Library and MyNAG had the highest scor-
ing [66] in the apps dedicated to caring for older people.
Elders may also suffer from progressive Parkinson’s disease,
and REMPARK app was designed to motivate them through
an exercise-based treatment programme [89]. Asthma patients
suffer from breathing difficulty, coughing and shortness of
breath among other things, and using Asthma Consultant for
Blackberry, Truth About Asthma for Android and Asthma for
iPhone is recommended because they address the widest range
of topics and provide guidance that are consistent with the US
guidelines [28]. Rheumatoid diseases are basically inflamma-
tion that affects the connecting structure of the human body.
TRACK + REACT, My Rheumatoid Arthritis Manager and
MyRA were recommended for such diseases [57]. The pelvic
floor is a group of muscles in the floor of the pelvic area.
Although many apps are available for training those muscles,
[56] iPelvis app was cited as ideal, interactive, dynamic and
real time. In the surgical department, and specifically for pa-
tients undergoing total hip replacement and total knee replace-
ment surgery, THR and TKR apps were shown to be of mod-
erate quality. Although some apps provide information, they
do not customise features for their users [2]. Apple apps were
superior, well-calibrated and unbiased compared with other
platforms with regard to scoliometer measurements [95].
There were also studies in the fitness department, including
weight loss and dietary apps, which concluded that Noom
Weight Loss Coach app ranked the highest in overall evalua-
tion [20]. In a second study [72] related to fitness and health
apps, MyMealMate was highly praised, and MyFitnessPal is
consistently the highest rated app on the market. In a third
study [24], HRV4training app was hailed as the most useful
tool. Bipolar disorder, or manic depression, is a mental health
condition that causes extreme mood swings, and [27] the app
that correctly addressed the most treatment guideline criteria
was Your MD’s Symptom Checker. For blood donation apps,
Android apps, such as Time Blood, GiveBloodApp, Yadonor
and Ahyeeha bi damik, achieved the highest score with
Blackberry apps coming last [5]. In medical sports-related
Health Technol. (2020) 10:1045–1061
apps, sports concussions are critical to an athlete’s career,
which is why [53] performed sport concussion apps assess-
ment stated that SCAT2 and the SCAT2 were recommended.
Reanimatie, a highly recommended app, is related to CPR
training and real incident support [61]. Many people may
choose to self-manage pain through online apps; [78] re-
searchers stated that Pain Diary app had the highest overall
score. Among apps on self-management of pain, those with
the most average high scores were free [50]. Pain comes in
many shapes and forms, and burn victims are no exception.
Burn Journey is designed specifically for kids who suffer from
burns [94]. Another form of pain is arthritic pain, and
WebMed Pain app had the highest overall score in usability
and quality [49]. Diabetic patients who aim to self-manage
their condition may use DiabetesManager, which has FDA
clearance, and MyCareConnect, which has HIPAA compli-
ance [43]. In another study related to diabetes, researchers in
[77] suggested the use of OneTouch RevealTM, Social
DiabetesTM, mySugr:App Diario de diabetesTM, Diabetes
menúTM, Tactio SALUD and Diabetes:MTM due to their
high usability and quality features. The findings in [30] sug-
gest that the three most popular neurosurgical apps
(Neurosurgery Survival Guide, Neuromind, and the Journal
of Neurosurgery App) are reliable for clinical use. Smoking is
a global health concern, which is why smoking cessation apps
were assessed in some studies. [22] SamrtQuit was listed
among the top apps and was supported by evidence. Care
for terminally ill is crucial in palliative-care related apps.
Symptom Management Guides was statically the most
downloaded app, whereas PalliAGED was more broadly ap-
plicable [67]. In this study, Orthodose was the only app that is
freely available on Android and iOS platforms. In another
research related to the assessment of apps that aim to calm,
relax and mindfully intervene for paediatric palliative care,
Kindoma, Calm and Minshift were appraised for certain fea-
tures [26]. Infants must be fed regularly and, in that regard, the
study [25] listed Murumamashouce app with the overall
highest score in infant feeding apps. Several Red Cross,
CDC and FEMA apps were recommended for disaster and
emergency response in [92]. For a healthy sexual life style,
pregnancy prevention is important, and Safe Sex Tips app
included most of the evaluated practices as stated in [69].
Another study found that YA SRH apps had the best pregnan-
cy prevention practices [86]. For medicine adherence apps,
MyMedSchedule, MyMeds and RxmindMe were ranked the
highest [45]. In hospitality management domain, apps such as
Practice & Hospital Mgmt, OPD MANAGEMENT and Smart
Hospital were mentioned to cover all aspects of patient assis-
tance [76]. For people in need of special education, the study
in [32] stated that Eureka 1 had the highest score on MARS
evaluation. Some researchers performed app assessment stud-
ies with relation to certain criteria, even though those studies
did not specifically state which apps performed better than
Health Technol. (2020) 10:1045–1061
others on certain conditions nor which apps fell behind. [74]
Examined 295 apps on four different smartphone platforms
but did not conclude which apps on certain platforms were
better, nor did they specify which platform is more suitable for
use than others. Instead, these studies opted to draw conclu-
sions on the basis of their overall assessment of the apps.
Similar research pattern was observed in [23, 31, 33, 44, 54,
58, 60, 62, 68, 70, 84, 85, 93]. Another procedure, but also
similar to the ones stated above, was done in [88], which
stated the apps of several platforms during their assessment,
but also did not state the recommendations regarding which
apps to use or which ones to shy away from, as their goal was
to provide an exploration of IBD apps and shed light on their
strengths and limitations. Another example of a study that did,
in fact, compare specific apps to different platforms was [71].
This study used statements, such as ‘AndoidApp #4’ and
‘AppleApp #3,’ in their assessments, but also, did not draw
particular recommendations that would favour one platform
over the other or one app over the other. Variety of the
discussed applications were designated for various medical
purposes. Researchers discussed them differently in aspects
like method of assessment and their feedback and experience.
Judging by the positive feedback and recommendations ob-
served within the literature, most of the researchers clearly had
positive experience. Some of them relate this positive experi-
ence with specific app names, whereas others praised apps for
specific medical cases without precisely choosing an app over
others, which indicated their good experience with the apps
they assessed”.
7 Conclusion
The number of research associated with mobile apps con-
tinues to increase. However, despite the significance of the
previous discussion, it still needs more assessment measures.
Mobile app assessment is considered an emerging and impor-
tant topic. The main contributions of this study are a compre-
hensive survey and classification of works related to mobile
app assessment and the methods used. Some patterns are ob-
served in literature, and identified articles are grouped into two
categories. The first category is associated with discussion
including challenges, motivations and recommendations.
The second category is linked to methodological aspects
followed in the assessment of apps. Such contributions would
allow better understanding and advancement of this topic.
Then, highlighted the different applications were aimed for
different goals for app assessment, and different researchers
set different objectives for themselves. This subsection re-
flects their experience and summarises their respected works.
Funding information this research is supported by Universiti Pendidikan
Sultan Idris under University Research Grant (2017–0310–107-01).
1059
Compliance with ethical standards
Disclosure of potential conflicts of interest The authors declare that they
have no conflict of interest.
Research involving human participants and/or animals This article
does not contain any studies with human participants or animals per-
formed by any of the authors.
Conflict of interest The authors declare that they have no competing of
interests.
References
1. Holl K, Elberzhager F. Mobile application quality assurance. Adv
Comput. 2019;112:1–77 Elsevier.
2. Bahadori S, Wainwright TW, Ahmed OH. Smartphone apps for
total hip replacement and total knee replacement surgery patients:
a systematic review. Disabil Rehabil. 2018:1–6.
3. Orsini G, Bade D, Lamersdorf W. Context-aware computation
offloading for mobile cloud computing: requirements analysis, sur-
vey and design guideline. Procedia Comput Sci. 2015;56:10–7.
4. Beam L, Burrows B, Dobey Z, Poser R, Sopko M. RecycMe: the
Ohio State University recycling phone application. 2013.
5. Ouhbi S, Fernández-Alemán JL, Pozo JR, El Bajta M, Toval A, Idri
A. Compliance of blood donation apps with mobile OS usability
guidelines. J Med Syst. 2015;39(6):63.
6. Kim MS, Park JH, Park K-Y. Development and effectiveness of a
drug dosage calculation training program using cognitive loading
theory based on smartphone application. J Korean Acad Nurs.
2012;42(5):689–98.
7. Nayebi F, Desharnais J-M, Abran A. The state of the art of mobile
application usability evaluation. In 2012 25th IEEE Canadian
Conference on Electrical and Computer Engineering (CCECE).
IEEE; 2012, p. 1–4.
8. Chen Z-S, Li R, Chen X, Xu H. A survey study on consumer
perception of mobile-commerce applications. Procedia Environ
Sci. 2011;11:118–24.
9. Tang AK. A systematic literature review and analysis on mobile
apps in m-commerce: implications for future research. Electron
Commer Res Appl. 2019;37:100885.
10. Darras KE, van Merriënboer JJG, Toom M, Roberson ND, de Bruin
ABH, Nicolaou S, et al. Developing the evidence base for M-
learning in undergraduate radiology education: identifying learner
preferences for Mobile apps. Can Assoc Radiol J. 2019;70(3):320–
6.
11. Albrecht U-V, Hillebrand U, von Jan U. Relevance of trust marks
and CE labels in German-language store descriptions of health
apps: analysis. JMIR mHealth and uHealth. 2018;6(4):e10394.
12. Anderson N, Steele J, O'Neill L-A, Harden LA. Pokémon Go: mo-
bile app user guides. Br J Sports Med. 2016;2016:096762.
13. Ahtinen A, Isomursu M, Huhtala Y, Kaasinen J, Salminen J,
Häkkilä J. Tracking outdoor sports–user experience perspective.
In: Aarts E, Crowley JL, Ruyter B, Gerhäuser H, Pflaum A,
Schmidt J, Wichert R, editors. European Conference on Ambient
Intelligence. Berlin: Springer; 2008. p. 192–209.
14. Lior LN. Writing for interaction: crafting the information experi-
ence for web and software apps. Newnes. 2013, Writing Text for
Interaction.
15. Cortimiglia MN, Ghezzi A, Renga F. Mobile applications and their
delivery platforms. IT Prof. 2011;13(5):51–6.
16. Beimborn D, Palitza M. Enterprise app stores for mobile
applications-development of a benefits framework. 2013.
1060
17. Harman M, Jia Y, Zhang Y. App store mining and analysis: MSR
for app stores. In Proceedings of the 9th IEEE Working Conference
on Mining Software Repositories. IEEE Press; 2012, p. 108–111.
18. Freier A. App revenue reaches $92.1 billion in 2018 driven by
mobile gaming apps. Business Apps. 2018;13(09).
19. Cheney S, Thompson E. The 2017–2022 app economy forecast: 6
billion devices, $157 billion in spend & more. App Annie. 2018.
20. Chen J, Cade JE, Allman-Farinelli M. The most popular
smartphone apps for weight loss: a quality assessment. JMIR
mHealth uHealth. 2015;3(4):e104.
21. Kim BY, Sharafoddini A, Tran N, Wen EY, Lee J. Consumer
mobile apps for potential drug-drug interaction check: systematic
review and content analysis using the mobile app rating scale
(MARS). JMIR mHealth uHealth. 2018;6(3):e74.
22. Haskins BL, Lesperance D, Gibbons P, Boudreaux ED. A system-
atic review of smartphone applications for smoking cessation.
Transl Behav Med. 2017;7(2):292–9.
23. Ali EE, Teo AKS, Goh SXL, Chew L, Yap KY-L. MedAd-AppQ: a
quality assessment tool for medication adherence apps on iOS and
android platforms. Res Soc Adm Pharm. 2018;14(12):1125–33.
24. Muntaner-Mas A, Martinez-Nicolas A, Lavie CJ, Blair SN, Ross R,
Arena R, et al. A systematic review of fitness apps and their poten-
tial clinical and sports utility for objective and remote assessment of
cardiorespiratory fitness. Sports Med. 2019;49(4):587–600.
25. Zhao J, Freeman B, Li M. How do infant feeding apps in China
measure up? A Content Quality Assessment. JMIR mHealth
uHealth. 2017;5(12):e186.
26. Weekly T, Walker N, Beck J, Akers S, Weaver M. A review of apps
for calming, relaxation, and mindfulness interventions for pediatric
palliative care patients. Children. 2018;5(2):16.
27. Nicholas J, Larsen ME, Proudfoot J, Christensen H. Mobile apps
for bipolar disorder: a systematic review of features and content
quality. J Med Internet Res. 2015;17(8):e198.
28. Huckvale K, Car M, Morrison C, Car J. Apps for asthma self-man-
agement: a systematic assessment of content and tools. BMC Med.
2012;10(1):144.
29. Zhang MW, Ho RC, Hawa R, Sockalingam S. Analysis of the
information quality of bariatric surgery smartphone applications
using the silberg scale. Obes Surg. 2016;26(1):163–8.
30. Bergeron D, et al. Mobile applications in neurosurgery: a systematic
review, quality audit, and survey of Canadian neurosurgery resi-
dents. World neurosurg. 2019.
31. van Galen L, Xu X, Koh M, Thng S, Car J. Eczema apps confor-
mance with clinical guidelines: a systematic assessment of func-
tions, tools and content. Br J Dermatol. 2019.
32. Larco A, Enríquez F, Luján-Mora S. Review and evaluation of special
education iOS Apps using MARS. In 2018 IEEE World Engineering
Education Conference (EDUNINE). IEEE; 2018, p. 1–6.
33. Larco A, Montenegro C, Luján-Mora S, Quality improvement
criteria of apps in Spanish for people with disabilities. In 2018 4th
International Conference on Information Management (ICIM).
IEEE; 2018, p. 260–264.
34. Alamoodi A, et al. A review of data analysis for early-childhood
period: taxonomy, motivations, challenges, recommendation, and
methodological aspects. IEEE Access. 2019;7:51069–103.
35. Burgers C, Brugman BC, Boeynaems A. Systematic literature re-
views: four applications for interdisciplinary research. J Pragmat.
2019;145:102–9.
36. Loureiro SMC, Romero J, Bilro RG. Stakeholder engagement in
co-creation processes for innovation: a systematic literature review
and case stud. J Bus Res. 2019.
37. Kushwah S, Dhir A, Sagar M, Gupta B. Determinants of organic
food consumption. A systematic literature review on motives and
barriers. Appetite. 2019;143:104402.
38. Daigneault P-M, Jacob S, Ouimet M. Using systematic review
methods within a Ph. D. Dissertation in political science: challenges
Health Technol. (2020) 10:1045–1061
and lessons learned from practice. Int J Soc Res Methodol.
2014;17(3):267–83.
39. Ain N, Vaia G, DeLone WH, Waheed M. Two decades of research
on business intelligence system adoption, utilization and success – a
systematic literature review. Decis Support Syst. 2019;125:113113.
40. Dani VS, Freitas CMDS, Thom LH. Ten years of visualization of
business process models: a systematic literature review. Comput
Stand Inter. 2019.
41. Owens OL, Beer JM, Reyes LI, Thomas TL. Systematic review of
commercially available mobile phone applications for prostate can-
cer education. Am J Men’s Health. 2019;13(1):
1557988318816912.
42. Huckvale K, Morrison C, Ouyang J, Ghaghda A, Car J. The evo-
lution of mobile apps for asthma: an updated systematic assessment
of content and tools. BMC Med. 2015;13(1):58.
43. El-Gayar O, Timsina P, Nawar N, Eid W. Mobile applications for
diabetes self-management: status and potential. J Diabetes Sci
Technol. 2013;7(1):247–62.
44. Huang Z, Lum E, Jimenez G, Semwal M, Sloot P, Car J.
Medication management support in diabetes: a systematic assess-
ment of diabetes self-management apps. BMC Med. 2019;17(1):
127.
45. Dayer L, Heldenbrand S, Anderson P, Gubbins PO, Martin BC.
Smartphone medication adherence apps: potential benefits to pa-
tients and providers. J Am Pharm Assoc. 2013;53(2):172–81.
46. Nguyen AD, Baysari MT, Kannangara DRW, Tariq A, Lau AYS,
Westbrook JI, et al. Mobile applications to enhance self-
management of gout. Int J Med Inform. 2016;94:67–74.
47. Payo RM, Harris J, Armes J. Prescribing fitness apps for people
with cancer: a preliminary assessment of content and quality of
commercially available apps. J Cancer Surviv. 2019;13:1–9.
48. Rusin M, Årsand E, Hartvigsen G. Functionalities and input
methods for recording food intake: a systematic review. Int J Med
Inform. 2013;82(8):653–64.
49. Bhattarai P, Newton-John T, Phillips JL. Quality and usability of
arthritic pain self-management apps for older adults: a systematic
review. Pain Med. 2017;19(3):471–84.
50. Devan H, Farmery D, Peebles L, Grainger R. Evaluation of self-
management support functions in apps for people with persistent
pain: systematic review. JMIR mHealth uHealth. 2019;7(2):
e13080.
51. Metelmann B, Metelmann C, Schuffert L, Hahnenkamp K,
Brinkrolf P. Medical correctness and user friendliness of available
apps for cardiopulmonary resuscitation: systematic search com-
bined with guideline adherence and usability evaluation. JMIR
mHealth uHealth. 2018;6(11):e190.
52. Taki S, Campbell KJ, Russell CG, Elliott R, Laws R, Denney-
Wilson E. Infant feeding websites and apps: a systematic assess-
ment of quality and content. Interact J Med Res. 2015;4(3):e18.
53. Lee H, Sullivan SJ, Schneiders AG, Ahmed OH, Balasundaram
AP, Williams D, et al. Smartphone and tablet apps for concussion
road warriors (team clinicians): a systematic review for practical
users. Br J Sports Med. 2015;49(8):499–505.
54. Moglia ML, Nguyen HV, Chyjek K, Chen KT, Castaño PM.
Evaluation of smartphone menstrual cycle tracking applications
using an adapted APPLICATIONS scoring system. Obstet
Gynecol. 2016;127(6):1153–60.
55. Brouard B, Bardo P, Bonnet C, Mounier N, Vignot M, Vignot S.
Mobile applications in oncology: is it possible for patients and
healthcare professionals to easily identify relevant tools? Ann
Med. 2016;48(7):509–15.
56. Latorre GF, de Fraga R, Seleme MR, Mueller CV, Berghmans B.
An ideal e-health system for pelvic floor muscle training adherence:
systematic review. Neurourol Urodyn. 2019;38(1):63–80.
Health Technol. (2020) 10:1045–1061
57. Luo D, Wang P, Lu F, Elias J, Sparks JA, Lee YC. Mobile apps for
individuals with rheumatoid arthritis: a systematic review. JCR: J
Clin Rheumatol. 2019;25(3):133–41.
58. Sun C, Malcolm JC, Wong B, Shorr R, Doyle M-A. Improving
glycemic control in adults and children with type 1 diabetes with
the use of smartphone-based mobile applications: a systematic re-
view. Can J Diabetes. 2019;43(1):51–8. e3.
59. Bry LJ, Chou T, Miguel E, Comer JS. Consumer smartphone apps
marketed for child and adolescent anxiety: a systematic review and
content analysis. Behav Ther. 2018;49(2):249–61.
60. Milani P, Coccetta CA, Rabini A, Sciarra T, Massazza G, Ferriero
GJP. Mobile smartphone applications for body position measure-
ment in rehabilitation: a review of goniometric tools. PM&R.
2014;6(11):1038–43.
61. Kalz M, et al. Smartphone apps for cardiopulmonary resuscitation
training and real incident support: a mixed-methods evaluation
study. J Med Internet Res. 2014;16(3):e89.
62. Piran P, et al. Medical mobile applications for stroke survivors and
caregivers. J Stroke Cerebrovasc Dis. 2019;28(11):104318.
63. Adam A, Hellig JC, Perera M, Bolton D, Lawrentschuk N. ‘Prostate
Cancer risk Calculator’ mobile applications (apps): a systematic review
and scoring using the validated user version of the Mobile application
rating scale (uMARS). World J Urol. 2018;36(4):565–73.
64. Schumer H, Amadi C, Joshi A. Evaluating the dietary and nutri-
tional apps in the Google play store. Healthcare Inform Res.
2018;24(1):38–45.
65. Torous J, Levin ME, Ahern DK, Oser ML. Cognitive behavioral
mobile applications: clinical studies, marketplace overview, and
research agenda. Cogn Behav Pract. 2017;24(2):215–25.
66. Anthony Berauk VL, Murugiah MK, Soh YC, Chuan Sheng Y, Wong
TW, Ming LC. Mobile health applications for caring of older people:
review and comparison. Ther Innov Regul Sci. 2018;52(3):374–82.
67. Meghani SH, MacKenzie MA, Morgan B, Kang Y, Wasim A,
Sayani S. Clinician-targeted mobile apps in palliative care: a sys-
tematic review. J Palliat Med. 2017;20(10):1139–47.
68. Rincon E, Monteiro-Guerra F, Rivera-Romero O, Dorronzoro-
Zubiete E, Sanchez-Bocanegra CL, Gabarron E. Mobile phone
apps for quality of life and well-being assessment in breast and
prostate cancer patients: systematic review. JMIR mHealth
uHealth. 2017;5(12):e187.
69. Chen E, Mangone ER. A systematic review of apps using mobile
criteria for adolescent pregnancy prevention (mCAPP). JMIR
mHealth uHealth. 2016;4(4):e122.
70. Alyami M, Giri B, Alyami H, Sundram F. Social anxiety apps: a
systematic review and assessment of app descriptors across mobile
store platforms. Evid-Based Ment Health. 2017;20(3):65–70.
71. Park JYE, Li J, Howren A, Tsao NW, De Vera M. Mobile phone apps
targeting medication adherence: quality assessment and content analy-
sis of user reviews. JMIR mHealth uHealth. 2019;7(1):e11919.
72. Byambasuren O, Sanders S, Beller E, Glasziou P. Prescribable
mHealth apps identified from an overview of systematic reviews.
Digit Med. 2018;1(1):12.
73. Short CE, Finlay A, Sanders I, Maher C. Development and pilot
evaluation of a clinic-based mHealth app referral service to support
adult cancer survivors increase their participation in physical activ-
ity using publicly available mobile apps. BMC Health Serv Res.
2018;18(1):27.
74. Bender JL, Yue RYK, To MJ, Deacken L, Jadad AR. A lot of
action, but not in the right direction: systematic review and content
analysis of smartphone applications for the prevention, detection,
and management of cancer. J Med Internet Res. 2013;15(12):e287.
75. Bychkov D, Young SD. Facing up to nomophobia: a systematic
review of mobile phone apps that reduce Smartphone usage. In Big
data in engineering applications. Springer; 2018, p. 161–171.
76. Olivero E, Bert F, Thomas R, Scarmozzino A, Raciti IM, Gualano
MR, et al. E-tools for hospital management: an overview of
1061
smartphone applications for health professionals. Int J Med
Inform. 2019;124:58–67.
77. Rodríguez AQ, Wägner AM. Mobile phone applications for diabe-
tes management: a systematic review. Endocrinol Diab Nutr.
2019;66(5):330–7.
78. Reynoldson C, Stones C, Allsop M, Gardner P, Bennett MI, Closs
SJ, et al. Assessing the quality and usability of smartphone apps for
pain self-management. Pain Med. 2014;15(6):898–909.
79. Rajani NB, Weth D, Mastellos N, Filippidis FT. Adherence of
popular smoking cessation mobile applications to evidence-based
guidelines. BMC Public Health. 2019;19(1):743.
80. Xiao Q, Wang Y, Sun L, Lu S, Wu Y. Current status and quality
assessment of cardiovascular diseases related smartphone apps in
China. Nurs Inform. 2016;225:1030–1.
81. Sedrati H, Nejjari C, Chaqsare S, Ghazal H. Mental and physical
mobile health apps. Procedia Comput Sci. 2016;100:900–6.
82. Huckvale K, Prieto JT, Tilney M, Benghozi P-J, Car J. Unaddressed
privacy risks in accredited health and wellness apps: a cross-
sectional systematic assessment. BMC Med. 2015;13(1):214.
83. Larco A, Yanez C, Almendáriz V, Luján-Mora S. Thinking about
inclusion: Assessment of multiplatform apps for people with dis-
ability. In 2018 IEEE Global Engineering Education Conference
(EDUCON). IEEE; 2018, pp. 350–354.
84. Seabrook HJ, Stromer JN, Shevkenek C, Bharwani A, de Grood J,
Ghali WA. Medical applications: a database and characterization of
apps in Apple iOS and Android platforms. BMC Res Notes.
2014;7(1):573.
85. Buechi R, et al. Evidence assessing the diagnostic performance of
medical smartphone apps: a systematic review and exploratory me-
ta-analysis. BMJ Open. 2017;7(12):e018280.
86. Mangone ER, Lebrun V, Muessig KE. Mobile phone apps for the
prevention of unintended pregnancy: a systematic review and con-
tent analysis. JMIR mHealth uHealth. 2016;4(1):e6.
87. Williams JP, Schroeder D. Popular glucose tracking apps and use of
mHealth by Latinos with diabetes. JMIR mHealth uHealth.
2015;3(3):e84.
88. Con D, De Cruz P. Mobile phone apps for inflammatory bowel
disease self-management: a systematic assessment of content and
tools. JMIR mHealth uHealth. 2016;4(1):e13.
89. Linares-Del Rey M, Vela-Desojo L, Cano-de la Cuerda R. Mobile
phone applications in Parkinson’s disease: a systematic review.
Neurología (English Edition). 2018.
90. Larsen ME, Nicholas J, Christensen H. A systematic assessment of
smartphone tools for suicide prevention. PloS One. 2016;11(4):
e0152285.
91. Huckvale K, Adomaviciute S, Prieto JT, Leow MK-S, Car J.
Smartphone apps for calculating insulin dose: a systematic assess-
ment. BMC Med. 2015;13(1):106.
92. Bachmann DJ, Jamison NK, Martin A, Delgado J, Kman NE.
Emergency preparedness and disaster response: there’s an app for
that. Prehosp Dis Med. 2015;30(5):486–90.
93. Bakker D, Kazantzis N, Rickwood D, Rickard N. Mental health
smartphone apps: review and evidence-based recommendations
for future developments. JMIR Ment Health. 2016;3(1):e7.
94. Wurzer P, Parvizi D, Lumenta DB, Giretzlehner M, Branski LK,
Finnerty CC, et al. Smartphone applications in burns. Burns.
2015;41(5):977–89.
95. Hayes W, Naziri Q, De Tolla JE, Akamnonu CP, Merola AA,
Paulino C. A systematic review of all smart phone applications
specifically aimed for use as a scoliosis screening tool. Spine J.
2013;13(9):S38.
Publisher’s note Springer Nature remains neutral with regard to jurisdic-
tional claims in published maps and institutional affiliations.