Received September 26, 2019, accepted November 8, 2019, date of publication December 4, 2019,

date of current version December 18, 2019.

Digital Object Identifier 10.1109/ACCESS.2019.2957341

A 5-Step Framework for Measuring

the Quality of Experience (QoE) of
Internet of Things (IoT) Services
MUHAMMAD SURYANEGARA 1 , (Senior Member, IEEE), DIMAS AGUNG PRASETYO1,2 ,
FERY ANDRIYANTO2 , AND NUR HAYATI 1,3 , (Member, IEEE)
1 Graduate Program in Telecommunications Management, Department of Electrical Engineering, Universitas Indonesia, Depok 16424, Indonesia
2 Ministry of Communications and Information Technology, Jakarta Pusat 10110, Indonesia
3 Department of Electrical Engineering, Universitas Muhammadiyah Yogyakarta, Yogyakarta 55183, Indonesia
Corresponding author: Muhammad Suryanegara (suryanegara@gmail.com)
This work was supported by the Hibah PTUPT 2019, Ministry of Research, Technology, and Higher Education, Indonesia.

ABSTRACT The ultimate benefit of the Internet of Things (IoT) is to improve quality of life by supporting
the automation and interconnection of various services; this interconnection includes public facilities in
platforms such as smart cities. IoT implementation is believed to improve services and yield additional
benefits. However, assessment of how the IoT brings such an experience to users is still a challenge that
is not yet standardized. Given that little research has discussed this issue, this paper aims to formulate a
framework for measuring the quality of experience (QoE) regarding IoT services on the basis of the Absolute
Category Rating with Hidden Reference (ACR-HR) scale, which measures the perception of experience by
having the IoT users provide scores for their experiences before and after IoT implementation. The proposed
framework consists of 5 steps: setting up the focus of the IoT services and the QoE parameters being assessed,
defining users of IoT services, conducting a mean opinion score (MOS) survey of the users, calculating the
DMOS (differential MOS) on the basis of the ACR-HR quantitative scale, and finally providing the strategic
implications to those who implement the service. To test the proposed framework, we have conducted
a proof of concept by measuring the QoE for the IoT services in the Jakarta Smart City platform. Six
institutional users operate the IoT for their public services, ranging from fire and rescue brigades to public
bus transportation. The results imply that the IoT-based services provide a good experience to institutional
users when compared to the state of services without IoT. A strategic implication is that some features of
Jakarta Smart City must be evaluated and improved to increase the level of satisfaction of users. To conclude,
our framework is sufficiently reliable in measuring the QoE for IoT-based services and can be recommended
for use for any other IoT-type service.

INDEX TERMS IoT, quality of experience, smart city, ACR-HR, Jakarta, Indonesia.

I. INTRODUCTION technology is applied? Answering these questions is a chal-

Similar to other technological innovations, the ultimate
advantage of the Internet of Things (IoT) is its ability to
bring solutions to everyday aspects of life so that people can
improve their quality of life [1]. However, does the IoT truly
bring a satisfactory solution experience to its users? What are
the perceptions of users after the IoT has been implemented,
and how do these perceptions compare to those before such
The associate editor coordinating the review of this manuscript and
approving it for publication was Sara Pizzi . scale. ACR-HR is an assessment procedure in which the order
lenge because there is no standardized step-by-step platform
for measuring users’ quality perceptions, i.e., their quality
of experience (QoE), for IoT-based services. By definition,
QoE is defined as the overall acceptance of an application
or service that is perceived subjectively by users and can be
influenced by expectations and user context [2].
This paper reports our work on proposing a generic frame-
work for measuring the QoE of IoT services on the basis of the
Absolute Category Rating with Hidden Reference (ACR-HR)

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see http://creativecommons.org/licenses/by/4.0/
VOLUME 7, 2019 175779

is presented one by one and is assessed independently on cat-
egory scales following the International Telecommunications
Union (ITU) ITU-T P.910 standard [3]. The concept has been
used with variations, such as in our previous work on mea-
suring the QoE of 4G mobile technological implementation
in Indonesia [4]. Previous QoE measurement research by [5]
has developed a quality model to evaluate IoT applications by
mapping the characteristics of IoT to quality attributes under
the scope of ISO 9126.
Our framework works on the basis of five steps: setting up
the focus of the IoT services to formulate the QoE parame-
ters, judging the institutional users who are the users of IoT
services, conducting a mean opinion score (MOS) survey of
IoT service users, calculating the differential MOS as the
ACR-HR quantitative scale, and finally providing the strate-
gic implications to those responsible for implementation. As
defined by the ITU-T P.910 standard [3], the ACR-HR scale
translates qualitative perceptions of experience into quantita-
tive scores, in which the users judge a service by assigning
a certain score. Thus, the uniqueness of our framework is
the QoE parameters that are specifically developed for the
services being assessed.
To validate our proposal, we test the framework using
the case of smart city implementation in Jakarta, the cap-
ital city of Indonesia. A smart city is a perfect example
because IoT technology is a prominent tool for enabling
the creation of a smart city. We focus on six public ser-
vices that are operated by government institutional offices.
The services range from fire and rescue services to city
bus transportation. Before IoT implementation, these ser-
vices were operational, and therefore, those six institutions’
users can provide perceptions regarding two conditions, i.e.,
before and after IoT implementation. We further utilize the
difference between the two scores to exploit the lessons
learned and inform the government as to how it should
undertake strategic actions to improve quality. Some pre-
vious studies have addressed the case of Indonesian smart
cities [6]–[8], and our previous work also presents cases
of the IoT and relevant technological implementation in
Indonesia [9], [10].
This work contributes to research on measuring the QoE
of a technological implementation. The important benefit of
our work is that the framework provides a comprehensive
sequential process for measuring experience in any IoT-based
service. The framework utilizes QoE measurement via the
ACR-HR method (as recommended by ITU [3]); however,
the framework adds to the ACR-HR results by developing
strategic implications to improve the services. Moreover, the
framework can be adapted and modified to measure other
IoT-based services.
The remainder of the paper is organized into sections
addressing the theories of the IoT and of QoE (Section II),
the framework that we are proposing (Section III), a discus-
sion of the proof of concept (Section IV) and conclusions of
this study (Section V).
175780
M. Suryanegara et al.: 5-Step Framework for Measuring the QoE of IoT Services
II. THEORIES
A. THE IoT AND SMART CITIES
The IoT can be interpreted as the use of systems and intelli-
gent devices that are interconnected to utilize data collected
by sensors and actuators [11]. The IoT supports solutions
for the improvement of the decision-making process and
productivity in manufacturing, retail, agriculture and other
sectors [1], [12]–[14]. A prominent example of the IoT in
the urban planning sector is its prominent role in the creation
of smart cities [15]. Together with telecommunications net-
works and artificial intelligence, the IoT is one of the three
core information and communications technologies (ICTs)
that are essential for a smart city [16].
The economic value of the IoT is expected to increase the
number of Internet-connected devices to 34 billion in 2024,
with an estimated 22.3 billion of those devices expected to
have IoT connections originating from wide-area IoT, cellular
IoT and short-range IoT [17]. The economic potential for the
largest application of IoT is in the industrial sector, which
is estimated to have an economic impact of 1.2 to 3.7 trillion
dollars by 2025; next is the urban sector, which is estimated to
have an economic impact of 930 billion to 1.7 trillion dollars
by 2025 [18].
The drive to maximize the application of the IoT in the
urban sector is even more widespread with the effort to
create smart cities in various cities in the world. The smart
city concept involves a vision of escalating the collaborative
agility of the city, realized by interconnecting multiple infras-
tructures, namely, business, social, information technology
and physical infrastructure [19]. Meanwhile, ITU defines a
smart sustainable city as a city that ensures the demands of
the current and next generations in environmental, social and
economic aspects by utilizing ICTs and other instruments to
increase the quality of life of society, competitiveness, and
provision of professional urban services and operations [20].
Therefore, the IoT acts as the fundamental enabler for con-
nection and sensor-based data collection, integrated among
public city facilities.
B. QoE
QoE is defined as the overall acceptance of an applica-
tion or service as perceived subjectively by users, which
can be influenced by expectations and user context [21].
The European Telecommunications Standard Institute (ETSI)
Technical Report (TR) 102 643 also defines QoE as a method
for measuring performance according to users based on sub-
jective and objective psychological measurement, for the use
of product or service ICT. Subjective psychological measure-
ment depends on user opinion, while objective psychological
measurement is influenced by technical parameters. The QoE
concept is being developed and is attracting attention because
it is related to user experience. In ITU-T recommendation
G.1011, QoE measurement is unable to be performed using
only one reference method [2]. It may include subjective QoE
measurement and objective QoE measurement.
VOLUME 7, 2019
M. Suryanegara et al.: 5-Step Framework for Measuring the QoE of IoT Services
The Absolute Category Rating (ACR) is one of the impor-
tant methods for measuring QoE [22]. It measures the MOS in
the form of a scale so that quality assessment can be translated
into numerical values from the average of all subjects and
other experimental factors. Some variations include ACR-HR
as described in the ITU-T P.910 standard [3]. It is a scoring
procedure in which the order of assessment is presented one
by one and assessed independently on a category scale. This
assessment procedure must include a reference version of
each order of evaluation as a stimulus or a hidden reference,
which in this study is a service before using the IoT.
Some relevant works regarding QoE in the IoT techno-
logical sector include reference [23], which examines the
relationship between consumer experiences and the quality
perception of the IoT. It also signifies the importance of
QoE, as the IoT industry should move beyond monitoring
QoS and expand that focus to QoE [23]. In a more specific
sector, we note reference [24], which proposed a conceptual
model for QoE in the case of the Internet of Medical Things
(IoMT); reference [25], which proposed a user experience
model in the case of an augmented reality (AR) game; and
reference [26], which proposed a QoE layered management
framework for the case of multimedia IoT (MIoT) services.
Reference [27] has proposed a framework for QoE modeling
of explosively increasing applications in the IoT. The last
work was motivated by the fact that IoT applications have
a tendency to grow very rapidly, and thus, the industry needs
a model to address such a phenomenon.
III. THE PROPOSED GENERIC FRAMEWORK FOR
MEASURING THE QoE OF IoT-BASED SERVICES
The theory of technological innovation management suggests
that any innovation should apply a diagnostic tool to define a
strategic plan for change improvement [28]. In this context,
analytical results from QoE measurement can be regarded as
part of the technological innovation diagnostic tool. Further,
the values of QoE, both quantitative and qualitative, can be
translated into relevant strategic action to improve the ser-
vice’s performance, following the perspective of technology
adoption, as discussed in [29], [30]. Our proposed generic
framework consists of five steps as follows.
A. STEP 1: DEFINING THE IoT SERVICES AND
FORMULATING THE QoE PARAMETERS
This is the crucial step because formulating the QoE param-
eters will be different case by case. The main motivation of
this step is to specify the characteristics of the service because
details of the technical formulation are a crucial ingredient of
quality measurement. There is no single parameter that can
be applied to all services, but they should be adapted to the
case. The parameters will be derived from the IoT platform’s
objective as well as the vision of the one who implements the
IoT. Thus, relevant theories and other works can be variously
combined to formulate the QoE parameters matching the
case being assessed. In our proof of concept (Section IV),
VOLUME 7, 2019
the defining case is the IoT-based services implemented in
Jakarta Smart City.
B. STEP 2: DEFINING THE USERS WHO IMPLEMENT IoT
SERVICES
The users are defined as the implementers of IoT services
who are expecting the most advantage after using the IoT
for their activities. They can be institutional users or the
end user (common people) who uses the technology. In the
case of a smart city, institutional users refer to governmental
offices or agencies that run the public services and are the
IoT implementers. They are the ones who will judge their
experiences before and after implementing the IoT for their
services.
The basic technical work of the IoT is the use of sensors for
collecting and managing the data of the devices. Meanwhile,
evaluating the quality of the IoT depends on the type of
public service. For example, the office that runs the public
service of the fire and rescue brigade may relate the benefit
of the IoT to managing the data relative to its time-critical
service performance. One simple expectation is that the IoT
can shorten the response time of the fire brigade when a fire
incident occurs. Alternately, the office that runs the public
service of street lighting may not relate the benefit of the IoT
with time-critical performance. That office may raise a simple
expectation of the IoT fostering energy consumption savings.
C. STEP 3: MOS SURVEY TO INDICATE THE LEVEL OF THE
ACR-HR SCORE
A basic conception is the use of a survey as a basic tool to
examine user perception. In this research, we measure the
QoE by using the MOS method combined with the ACR-HR
scale. According to ITU-T P.800.2, the MOS is obtained
in subjective testing, where subjects are asked to give their
opinions on a scale [22]. The survey always has a pair of
statements, questioning the perceived quality before and after
implementing the IoT. The survey asks IoT users to fill out the
score from 1 to 5 as follows:
D. STEP 4: CALCULATING DMOS AS THE ACR-HR VALUE
SCALE
As implied by its name, in the ACR-HR, the measurement is
calculated on the basis of comparing a tested QoE value and
hidden reference QoE value. In this research, the main propo-
sition is that the tested QoE value is the condition of services
after implementing the IoT, while the hidden reference values
are the previous condition of services before implementing
175781

TABLE 1. DMOS (ACR-HR value comparison) [3], [4].
the IoT. Therefore, the ACR-HR scale is simply obtained by
calculating the DMOS (differential MOS) using the following
formula [3]:
DMOS = A − B
where
DMOS = Differential MOS or ACR − HR value scale
A = The MOS for the condition after implementing
the IoT
B = The MOS for the condition before implementing
the IoT
(1)
Subsequently, we translate the DMOS into qualitative com-
parison values, as presented in Table 1. The principle value
of such a comparison is taken from [4].
For example, if users perceive that the service before
implementing the IoT is good (MOS = 4) and the service
after implementing the IoT is very good (MOS = 5), then
the DMOS = 1. This suggests that the respondent perceived
the experience of service using the IoT as somewhat ‘‘better’’
than when not using the IoT. In general, the ACR-HR scores
are such that any value close to zero indicates a perception
that the service before and after implementing the IoT is not
175782
M. Suryanegara et al.: 5-Step Framework for Measuring the QoE of IoT Services
TABLE 2. Guidance matrix to develop strategic implications.
significantly different. A greater positive or negative DMOS
indicates that the difference between the two services is per-
ceived as more significant [4]
E. STEP 5: DEVELOPING STRATEGIC IMPLICATIONS
Based on the results of Step 4, the lessons learned should be
developed, followed by discussion to build a relevant strategy
for improving the system. The method builds a matrix of
the lowest DMOS value, the highest DMOS value and the
average DMOS value. Table 2 reflects the general guidance
matrix to develop such relevant strategic implications. This
guidance matrix adapts the theory of [29], [30] that signifies
the importance of performance expectancy as the strongest
determinant of satisfaction of a technology adoption. In this
case, DMOS actually represents users’ satisfaction with an
IoT-based service performance. Consequently, the strategy
includes increasing parameters that have already been per-
ceived as satisfactory, while the services that are less satis-
factory must be evaluated and restored.
IV. PROOF OF CONCEPT
This section implements the five steps of the proposed frame-
work in the case of Jakarta Smart City, the capital city of
Indonesia, as illustrated in Figure 1.
As the most densely populated city in Indonesia [31],
Jakarta decided to implement smart city technology as an
effort to increase competitiveness and develop public services
for societal welfare [32]. A smart city is realized by maxi-
mizing the use of ICT technology to know, understand, and
control various resources more effectively and efficiently to
maximize public services, provide solutions to problems, and
support sustainable development. Jakarta Smart City has six
indicators in its implementations, namely, the environment,
economy, people, governance, living and mobility [32].
VOLUME 7, 2019
M. Suryanegara et al.: 5-Step Framework for Measuring the QoE of IoT Services
FIGURE 1. Location of Jakarta, the capital city of Indonesia, and some
strategic indicators.
The technical implementation of Jakarta Smart City is car-
ried out by the Jakarta Provincial Government’s Office. Some
of the main program directions include displaying informa-
tion that can be accessed by the public, closed-circuit televi-
sion (CCTV) operation covering the city’s public areas, data
analysis activities to obtain recommendations or basic pol-
icy analysis for the government, providing digital space for
the community to participate in the city’s development, and
implementing IoT technology for the city’s public facilities.
Similar to another smart city objective, Jakarta Smart City
is generally proclaimed to have the goal of improving peo-
ple’s living standards and overcoming problems that occur
in a city with 10.37 million people as well as increasing
competitiveness and developing public services to support
public welfare [32].
The proof of concept of our proposed framework focuses
on six public services that have been operated before and after
the services implemented the IoT. The public services are fire
& rescue services, the operation of water level recording in
relation to the flood mitigation system, emergency ambulance
operation, and the operation of garbage trucks, street lighting,
and public bus transportation; see Table 3, Column (1) and
Column (2).
We use the basic definition for the use of the IoT in Jakarta
Smart City: the connection of existing operational devices
using an Internet connection in an effort to monitor or control
public services [32], [33]. Table 3 Column (3) defines the
general conditions of public services before the IoT imple-
mentation. Table 3 Column (4) defines the condition of each
of these public services after the IoT implementation. They
operate the IoT hardware (and the relevant sensors) embedded
in their main operational devices or vehicles with the goal of
improving their service functionalities.
VOLUME 7, 2019
A detailed step-by-step description of the proof of concept
is presented as follows:
A. STEP 1: THE FORMULATION OF THE QoE PARAMETERS
OF JAKARTA SMART CITY
This step aims to build QoE parameters that are applied
specifically in the case of measuring the QoE of Jakarta Smart
City.
Reference [5] discusses the quality models for evaluating
IoT applications. This research divides the quality factors of
the IoT application into functionality, reliability, efficiency,
and portability based on quality attributes in ISO / IEC
9216. However, the standard reference used, ISO 9216, is no
longer valid and has been revised with ISO / IEC 25010,
so adjustments need to be made. Referring to ISO / IEC
25010, the aforementioned attributes change to functionality
suitability, reliability, performance efficiency, and portability.
Reference [34] discusses QoE in the context of a smart city.
It states that nine QoE factors that are estimated to have an
impact on a smart city are usability, personalization, use-
fulness, transparency, accessibility, effectiveness, efficiency,
learnability, and findability. Meanwhile, the Y.2066 ITU-T
standard [35] describes the general requirements of the IoT,
which are divided into functional and nonfunctional require-
ments. Functional requirements relate to actors of the IoT,
while nonfunctional requirements relate to the implementa-
tion and operation of the IoT and are not related to actors of
the IoT. Nonfunctional requirements of the IoT are interop-
erability, scalability, reliability, high availability, adaptability,
and manageability.
This proof of concept utilizes the studies of [5] and [34],
combined with ITU-T Y.2066 standard [35], to formulate the
QoE parameters to measure IoT services in Jakarta Smart
City. We end up with five core parameters, as listed in Table 4.
The first is to measure the user perception of the consistency
of the IoT service, including the aspects of accuracy and secu-
rity of data. The second is the perception of the convenience
of services, such as the ease of accessing the data from
massive devices. The third is about service efficiency, in
relation to the resource’s utilization. The fourth is to measure
the user perception of the integration of the IoT services
with another technological platform. The fifth concerns the
functional suitability, meaning the perception of whether the
service has truly met its function. These five parameters
are varied according to the type of service. Finally, each of
these five main parameters is extracted into detailed QoE
parameters, as reported in Table 4, Column 3. These detailed
QoE parameters are measured in the case of Jakarta Smart
City.
B. STEP 2: DEFINING THE USERS OF THE IoT IN JAKARTA
SMART CITY
This step is to define the users who value their perception
experience after implementing the IoT in Jakarta Smart City.
We choose six public-service offices, so-called institutional
175783
 M. Suryanegara et al.: 5-Step Framework for Measuring the QoE of IoT Services

TABLE 3. The six institutional users of the IoT in Jakarta Smart City.

175784 VOLUME 7, 2019

M. Suryanegara et al.: 5-Step Framework for Measuring the QoE of IoT Services
TABLE 4. The formulation of QoE parameters for the case of Jakarta
Smart City.
users, who implement the IoT in their public services,
as already listed in Table 3 Column (1).
C. STEP 3: MOS SURVEY OF THE INSTITUTIONAL USERS
For this step, we have conducted MOS surveys of the six
institutional users from 24 May 2019 to 19 June 2019. The
respondents are officers who run the services daily and have
already operated the services under two conditions, before
the institution implemented the IoT and after implementing
VOLUME 7, 2019
TABLE 5. Examples of questions on the MOS survey.
it. These criteria are expected to help respondents understand
the specific differences of each service and thus express their
experience perceptions objectively. In Table 3 Column (5), we
list the number of officers who are working on the six public
services and the number of respondents for the MOS survey.
Statistically, we have sampled 60% of total officers (31 out
of 52 persons).
Derived from the QoE parameters in Table 4 Column
(3), we developed a questionnaire consisting of eight state-
ments regarding nonfunctional suitability and seven state-
ments regarding functional suitability QoE parameters. The
statements regarding nonfunctional suitability are similar for
all six institutional IoT users, while those regarding func-
tional suitability are varied depending on type of service user.
For each statement, respondents were asked to give a score
representing their subjective evaluations. The scale used was
an ACR of 5 points, graded as follows: 5 = very good, 4 =
good, 3 = normal/ordinary, 2 = poor, and 1 = very poor.
Table 5 lists several examples of statements included in the
questionnaire.
D. STEP 4: CALCULATING THE AVERAGE DMOS OF THE
ACR-HR VALUE SCALE
In this step, we compute the DMOS on the basis of the QoE
values provided by the users in Step 3.
175785

FIGURE 2. Chart indicating the survey results of DMOS values of QoE
nonfunctional parameters by all six institutional users of Jakarta Smart
City.
1) ANALYSIS OF NONFUNCTIONAL QoE PARAMETERS OF
ALL PUBLIC SERVICES BY ALL SIX INSTITUTIONAL USERS
The QoE survey results of nonfunctional parameters for
all IoT-based services in Jakarta Smart City are shown
in Figure 2. We present graphs of survey results in the form
of the average DMOS value, indicating a score between the
conditions after and before implementing IoT. There are eight
statements of QoE parameters. Overall, the average DMOS
values obtained ranged from 1.58 to 2.06. This means that
the IoT-based service implementation provided a ‘‘better-
to-much better’’ experience for users compared to previous
service without implementation of IoT.
It can be observed from the graph that the most significant
value corresponded to the aspect of the ease of data access
(DMOS = 2.06). This was perceived by all six institutional
users as providing a better experience. All users could more
easily access a majority of existing services anytime and any-
where, especially for the purpose of asset management. The
second-highest value was for the aspect of the data accuracy
level (DMOS = 1.97). This proves that data accuracy is very
important for all IoT-based services on the Jakarta Smart City
platform. However, it appears that data security has the lowest
value (DMOS = 1.58). Even though it was in the ‘‘better’’
category, all six institutional users perceived a less significant
experience for this aspect. This can be caused by vulnerability
issues due to the large number of IoT devices connected to
the Internet. Other QoE parameters are in the range of the
175786
M. Suryanegara et al.: 5-Step Framework for Measuring the QoE of IoT Services
‘‘better’’ category, which signifies a good level of experience
by the users.
We argue that the use of average DMOS for nonfunc-
tional parameters is relevant enough to reflect the entire
performance of the six public services. When seeing the
entire performance, the single DMOS value of nonfunctional
parameters for each public service is not that essential to
be identified. Such a paradigm is developed because those
public services are actually working under a single umbrella,
i.e., the Jakarta Smart City platform. According to the the-
ory of system engineering [36], a complex system is actu-
ally structured by its subsystem or the system’s elements.
We make an analogy that a city is a complex system, and
public services are the city’s elements. Thus, the overall
perceived quality of a city system is actually a collective
effect of the quality perception of each public service in the
city.
However, this is not a locked-in perspective because we
may investigate details of the single DMOS values for each
nonfunctional parameter of the six public services. This is
to evaluate what services were impacted more or less in
terms of each nonfunctional parameter by the IoT imple-
mentation. When focusing on such a specific public ser-
vice, an evaluation of single DMOS values is still useful
to notice what services may need more effort regarding
IoT implementation for some specific nonfunctional
parameters.
2) ANALYSIS OF THE FUNCTIONAL QoE PARAMETERS OF
SERVICES BY THE OFFICE OF FIRE AND RESCUE SERVICE
The DMOS results for the Office of Fire and Rescue service
are shown in Figure 3. This office operates fire trucks for any
related incident. The graph shows that the average DMOS
values range from 0.6 to 1.4, which means that, generally,
the IoT-based services provide ‘‘just the same’’ or ‘‘some-
how better’’ experiences in supporting the operation of fire
truck services. The most significant value (DMOS = 1.4) is
the ease of accessing data regarding the availability of fire
trucks because by using the IoT, users could easily obtain
more information about the location and status of the fire
trucks. This information is very useful due to the Office of
Fire and Rescue’s asset relocation function, in which duty
officers should update conditions regarding vehicles and their
drivers in certain locations every 12 hours. It is an anticipatory
method for facing emergency situations, where the vehicle
closest to a fire location can be immediately deployed to
provide assistance so that an officer can reach a specified
service response time.
The functional parameters that have the lowest value
(DMOS = 0.6) are in two QoE parameters, namely, the esti-
mated maintenance time of fire trucks and the speed of fire
truck arrival. This condition occurs because the maintenance
function based on mileage has not been used. Meanwhile,
the speed of arrival is less significant because, with or without
implementation of the IoT, fire trucks have to quickly arrive
at the fire location. Thus, only a slight change is felt by users.
VOLUME 7, 2019
M. Suryanegara et al.: 5-Step Framework for Measuring the QoE of IoT Services
FIGURE 3. Chart indicating the survey results of the DMOS values of QoE
functional parameters by the Office of Fire & Rescue Service.
3) ANALYSIS OF THE FUNCTIONAL QoE PARAMETERS OF
SERVICES FROM THE OFFICE OF WATER RESOURCES
Figure 4 presents the charts of the DMOS functional parame-
ters of the IoT-based service at the Office of Water Resources.
The main service of this office is the operation of the auto-
matic water level recorder (AWLR) in accordance with the
flood mitigation system. It can be observed from the graph
that the average DMOS values range from 0.8 to 1.8, which
means that the IoT-based services generally provide ‘‘just the
same’’ to ‘‘somehow better’’ service. Functional parameters
that have the most significant values are in anticipation of the
possibility of flooding, with an average value of DMOS =
1.8. This is the main duty and function of the office, so it can
be interpreted that the use of the IoT can function as expected,
i.e., it is functionally suitable. Moreover, it is already well
known that Jakarta faces flood problems every year. The QoE
parameters that have the least-significant value of 0.8 are
applications related to the trend of flooding occurrence, along
with the accuracy of operating a water pump based on water
level. This is because the use of the IoT to measure the river
water level height is not optimized further by connecting it
to pump operations water. One reason is that the use of the
IoT remains linked to the operational manual data collection
from operators at the field, as well as the operation of the
water pump, which is still manual. This condition is unfor-
tunate because the IoT actually supports automation, and the
VOLUME 7, 2019
FIGURE 4. Chart indicating the survey results of the DMOS values of the
QoE functional parameters for the Office of Water Resources.
comprehensive use of technology for automation is part of the
smart city concept.
4) ANALYSIS OF THE FUNCTIONAL QoE PARAMETERS OF
SERVICES FROM THE OFFICE OF HEALTH SERVICE
Figure 5 shows a chart indicating the results of DMOS of
services from the Office of Health Services. The average
values of such QoE parameters range from 1.14 to 2.43,
which means that in general, the IoT implementation pro-
vides a ‘‘much better’’ experience. The highest value of func-
tional parameters is in terms of emergency ambulance service
response time, with an average value of 2.43. The IoT con-
tributes to providing information about location management
and the availability of ambulances that helps to increase the
response time of emergency ambulance services. Response
time to service plays an important role in the medical world
since it affects patients’ conditions. It can be summarized that
the IoT-based service implementation could work properly
as expected. Meanwhile, a functional parameter that has a
less significant value is anticipating ambulance vacancies at
service times, which has an average value of 1.14. This is
because even though data regarding ambulance deficiencies
at certain times of service can be known, the addition of the
necessary number of ambulances cannot be easily achieved.
When there is a lack of ambulances, the destinations of
ambulances can be shifted away from those with less critical
175787

FIGURE 5. Chart indicating the survey results of the DMOS values of the
QoE functional parameters for the Office of Health Service.
health conditions, such as those delivering patients to a
hospital.
5) ANALYSIS OF THE FUNCTIONAL QoE PARAMETERS OF
SERVICES FROM THE OFFICE OF INDUSTRY AND ENERGY
DMOS results reflecting the QoE of services deployed by
the Office of Industry and Energy are shown in Figure 6.
The main public service using the IoT is the operation of
street lighting. The graph shows that the average values range
from 1.0 to 2.75, which means that generally, IoT-based
services provide a ‘‘better’’ to ‘‘much better’’ experience.
The best value of functional parameters concerns anticipation
of inactive public street lighting, which reaches DMOS =
2.75. Anticipation of nonfunctioning public street lighting
is related to lamp maintenance response times, which can
affect the road conditions or the environment when the public
street lighting is not functioning. With better anticipation,
the Office of Industry and Energy can carry out its func-
tions better, which also means that the use of the IoT is in
accordance with the expected functions. Alternately, the least
optimal value (DMOS = 1.0) of functional parameters is
estimating locations that require street lighting.
As seen in the chart, in general, the parameters have
reached DMOS > 1. It is linear, with the fact that the gov-
ernment successfully saves approximately IDR 310 billion to
pay electricity bills used for public street lighting [37]. This
savings is achieved by combining LED lights, which are more
175788
M. Suryanegara et al.: 5-Step Framework for Measuring the QoE of IoT Services
FIGURE 6. Chart indicating the survey results of the DMOS values of the
QoE functional parameters for the Office of Industry and Energy.
energy efficient, and IoT sensors for the switching operation.
The switch is set to control the light intensity based on the
conditions of the surrounding environment.
6) ANALYSIS OF THE FUNCTIONAL QoE PARAMETERS FOR
SERVICES FROM THE OFFICE OF ENVIRONMENTAL SERVICES
Figure 7 reflects the QoE results of the IoT-based services
at the Office of Environmental Services, giving a very good
perception of its functional suitability. The main service
measured is the operation of garbage trucks. The average
DMOS values range from 1.0 to 2.2, implying a perception
of ‘‘better’’ to ‘‘much better’’ service compared to when the
office had not yet implemented the IoT.
The most prominent parameters (DMOS = 2.2) are
achieved by the capability of the IoT to provide information
related to the accuracy of garbage truck mileage as well as
the availability and status of the garbage trucks. Along with
achievement of these two parameters, the Office of Envi-
ronmental Services could minimize unwanted situations that
have previously occurred, such as vehicle loss or transporta-
tion of garbage to the wrong destination.
The least prominent value still exhibits a good response
of DMOS = 1.0. It concerns the QoE for the parameter of
VOLUME 7, 2019
M. Suryanegara et al.: 5-Step Framework for Measuring the QoE of IoT Services
FIGURE 7. Chart indicating the survey results of the DMOS values of the
QoE functional parameters for the Office of Environmental Services.
selection of the fastest route for garbage collection. This cir-
cumstance occurs because the feature has not yet been imple-
mented, and it is a plan for future development. A concern
for future development is the fact that the waste volume at a
disposal site is calculated manually based on the estimation
of waste input from a field officer without a scale or weight
sensor.
7) ANALYSIS OF THE FUNCTIONAL QoE PARAMETERS OF
SERVICES FROM THE OFFICE OF TRANSJAKARTA
The main service provided by this office is operation of
the city bus transportation, the so-called Transjakarta. The
results of IoT-based service at PT Transjakarta have been
perceived as a very significant experience, as can be observed
from Figure 8. The average value of the survey is in the range
of 1.8 to 2.4. The highest score is achieved for the feature
of determining the opening or closing of bus routes. The
decision to open or close a route is performed by utilizing the
data obtained from the IoT-based service. The combination
of information related to the number of active buses on a
particular route and the number of users at the existing bus
stops supports the determination of whether to open new
routes for locations experiencing congestion or close old
routes that have fewer passengers. The lowest parameter is
VOLUME 7, 2019
FIGURE 8. Chart indicating the survey results of the DMOS values of the
QoE functional parameters for the Transjakarta company.
achieved for the feature of the accuracy of bus mileage. This
result confirms that the lowest value for IoT implementation
in this sector is fairly good. Thus, it can be concluded that the
IoT works properly to support this field.
E. STEP 5: DEVELOPING STRATEGIC IMPLICATIONS
BASED ON THE RESULTS OF QoE MEASUREMENT
The results can be used to define how to innovate and which
features should be developed/innovated to benefit society in
Jakarta, as can be observed from Table 6. We summarize
the elements with the highest, lowest and average DMOS,
reflecting the facts of the lessons learned. Based on those
three facts, we can list brief strategic implications, as reported
in Table 2.
In principle, the highest DMOS indicates the features that
significantly meet users’ expectations. The lowest are ones
that the government should pay special attention to and
develop specific strategies for to restore the parameters to
a level that meets users’ expectations. Finally, the average
DMOS can be inferred and linked with the overall perception.
The overall results indicate that five institutional users have
a definite perception that after implementing the IoT, services
175789
 M. Suryanegara et al.: 5-Step Framework for Measuring the QoE of IoT Services

TABLE 6. Results of the QoE measurement of the IoT-based services in
Jakarta Smart City.
perform ‘‘better-to-much better’’ than before the implemen-
tation of the IoT. There is one institutional user that has a
slightly lower-value experience, in which the system is con-
sidered to be ‘‘somehow just the same’’ as under the previous
condition. As a result, the Jakarta Smart City management
should pay more attention to the IoT-based services regarding
the operation of fire trucks.
By looking at the nonfunctional parameters, the highest
DMOS is for the parameter of ‘‘Ease of data access.’’ This
may be followed up by taking strategic action to further pro-
cess data for various purposes. This is a strategic opportunity
for future improvements in the application of advanced ser-
vices that shall satisfy the actual definition of the IoT, i.e., to
utilize data collected by sensors and actuators embedded in
machines and other physical objects that are interconnected
for advanced services that can provide solutions to increase
efficiency and productivity and assist in decision-making [1].
Strategic implications related to the operational service
of fire trucks, ambulances, garbage trucks and city bus
transportation imply a strategy in which vehicle location
monitoring services can be increased by connecting to prox-
imity sensors to find the distances of vehicles that have been
recorded. Subsequently, the data are processed to determine
the maintenance times of the vehicles. With proper vehicle
maintenance, it will certainly be possible to improve fuel
efficiency and reduce long-term maintenance costs.
For firefighting services, vehicle location sensors can be
further increased for the process of relocating fire engines to
a service, service or postal service that lacks or emptiness
of fire engines at service time. By utilizing existing asset
data, the distance from each agency, the official department,
and the post that has the relocation of these assets can be
minimized automatically through the existing system.
For garbage transportation services, besides proximity
sensors, a height sensor for the volume of waste that is
being transported can also be added such that the garbage
truck can optimize the number of polling stations visited
on its route according to the volume of waste available.
In addition, because the garbage disposal route runs out
of town, a passable route can be added, with notification
to an authorized official if the vehicle exits the specified
route.
In line with the flood mitigation system, for water level
monitoring services, advanced services that can be improved
are related to the use of water level altitude sensors to help
make decisions regarding the operation of water pumps. This
is because the process is still performed manually. Indeed,
not only is the water level the basis of the operation of the
water pumps but also if other requirements can be identified
correctly, automation of decision-making is also certainly
possible.
Finally, the relevant strategy for street lighting operation
is about expanding the services to support other lighting
platform systems, such as electrical billboards. It is expected

175790 VOLUME 7, 2019

M. Suryanegara et al.: 5-Step Framework for Measuring the QoE of IoT Services
that the eventual benefit will be increased energy efficiency
of the city’s infrastructure.
V. FUTURE IMPLICATIONS AND LIMITATIONS OF THIS
STUDY
The practical implications of this study include two con-
cerns. First, the framework can be sufficiently flexible to
be further combined with other strategic methods to enrich
Step 5 (developing strategic implications). As mentioned in
Section III E, the final step of this framework is to build a
lesson learned leading to relevant improvement strategy for
the IoT service. Instead of presenting a guidance matrix to
develop strategic implications (Table 2), our current frame-
work does not specify what method should be used to con-
struct such a strategy. Therefore, this first concern actually
has opened up an opportunity for a future research agenda.
Second, the framework works on the basis of the DMOS
value, which reflects the users’ perceptions that arise after
a technology has been applied, compared to those percep-
tions before it has been implemented. Therefore, the value
of DMOS can also be used to identify the status of a techno-
logical acceptance by users. Further, we can also relate this
concern to the policy-related issue. The action to implement
a new technology is actually a policy made by manage-
ment or government. Thus, DMOS can be used to determine
whether a policy is considered successful compared to the
status quo policy.
Finally, the limitation of this study is that to obtain valid
results, the framework is very dependent on Step 1, which
is formulation of the QoE parameters. If we fail to generate
proper QoE parameters, then it may reduce the validity of
the assessment results. As already mentioned in Section III
A, Step 1 is the crucial step since the QoE parameters will
be different case by case. Hence, at some point, it can be
perceived as a limitation that there is not a generic param-
eter matching all cases, but the parameters can be uniquely
derived or variously combined from other relevant theories.
VI. CONCLUSION
We have formulated a framework to measure the QoE of IoT
services on the basis of the ACR-HR scale. Our proposed
framework consists of five steps: setting up the focus of
the IoT services to formulate its QoE parameters, defin-
ing the users who operate the IoT services, conducting an
MOS survey of the users, calculating the DMOS using the
ACR-HR quantitative scale, and finally developing strategic
implications for the service implementer. A proof of concept
has been performed by taking the case of Jakarta Smart
City. The overall framework in which six institutional users
expressed their experience of utilizing IoT technology in their
relevant services has been tested. They are the offices that
manage public transportation, garbage trucks, ambulances,
the fire and rescue brigade, street lighting, and water level
measurement. The overall results indicate that five users have
a definite perception that services after implementing IoT
perform better than before the implementation. There is one
VOLUME 7, 2019
institutional user that had a slightly lower-value experience,
in which the system was considered ‘‘just the same’’ as under
the previous condition. As a result, the Jakarta Smart City
management should pay more attention to this service. To
conclude, our framework is sufficiently reliable for measur-
ing QoE in IoT-based services and can be recommended for
use in any other IoT-type services.
REFERENCES
[1] Understanding the IoT, GSM Association, London, U.K., 2014.
[2] Reference Guide to Quality of Experience Assessment Methodologies,
document ITU-T G.1011, Geneva, Switzerland, 2016. [Online]. Available:
https://www.itu.int
[3] Subjective Video Quality Assessment Methods for Multimedia Applica-
tions, document ITU-T P.910, Geneva, Switzerland, 2008. [Online]. Avail-
able: https://www.itu.int
[4] M. Suryanegara, F. Andriyanto, and A. S. Arifin, ‘‘Lessons learned from
the quality of experience (QoE) assessment of 4G mobile technology in
Indonesia,’’ Indonesian J. Elect. Eng. Comput. Sci., vol. 10, no. 3, p. 1203,
Jun. 2018.
[5] M. Kim, ‘‘A quality model for evaluating IoT applications,’’ Int. J. Comput.
Elect. Eng., vol. 8, no. 1, pp. 1–11, 2016, doi: 10.17706/ijcee.2016.8.1.66-
76.
[6] H. S. Firmansyah, S. H. Supangkat, A. A. Arman, and P. J. Giabbanelli,
‘‘Identifying the components and interrelationships of smart cities in
Indonesia: Supporting policymaking via fuzzy cognitive systems,’’ IEEE
Access, vol. 7, pp. 46136–46151, 2019.
[7] S. H. Supangkat, A. A. Arman, Y. A. Fatimah, R. A. Nugraha, and
H. S. Firmansyah, ‘‘The role of living labs in developing smart cities in
Indonesia,’’ in Data-Driven Multivalence in the Built Environment. Cham,
Switzerland: Springer, 2020, pp. 223–241.
[8] D. Kusumawati, D. Setiawan, and M. Suryanegara, ‘‘Spectrum require-
ment for IoT services: A case of Jakarta smart city,’’ in Proc. IEEE Int.
Conf. Commun., Netw. Satell. (Comnetsat), Oct. 2017, pp. 21–25.
[9] M. Suryanegara, A. S. Arifin, M. Asvial, K. Ramli, M. I. Nashiruddin, and
N. Hayati, ‘‘What are the indonesian concerns about the Internet of Things
(IoT)? Portraying the profile of the prospective market,’’ IEEE Access,
vol. 7, pp. 2957–2968, 2019.
[10] M. Suryanegara, F. Andriyanto, and B. Winarko, ‘‘What changes after
switching to 4G-LTE? Findings from the Indonesian market,’’ IEEE
Access, vol. 5, pp. 17070–17076, 2017.
[11] R. Want, B. N. Schilit, and S. Jenson, ‘‘Enabling the Internet of Things,’’
Computer, vol. 48, no. 1, pp. 28–35, 2015, doi: 10.1109/MC.2015.12.
[12] Y. J. Kwon and D. H. Kim, ‘‘IoT-based defect predictive manufacturing
systems,’’ in Proc. Int. Conf. Inf. Commun. Technol. Converg. (ICTC),
Oct. 2017, pp. 1067–1069, doi: 10.1109/ICTC.2017.8190856.
[13] N. N. Dlamini and K. Johnston, ‘‘The use, benefits and challenges
of using the Internet of Things (IoT) in retail businesses: A literature
review,’’ in Proc. Int. Conf. Adv. Comput. Commun. Eng. (ICACCE),
Durban, South Africa, Nov. 2016, pp. 430–436, doi: 10.1109/ICACCE.
2016.8073787.
[14] N. Ahmed, D. De, and I. Hussain, ‘‘Internet of Things (IoT) for smart
precision agriculture and farming in rural areas,’’ IEEE Internet Things J.,
vol. 5, no. 6, pp. 4890–4899, Dec. 2018, doi: 10.1109/JIOT.2018.2879579.
[15] G. Solmaz, F.-J. Wu, F. Cirillo, E. Kovacs, J. R. Santana, L. Sanchez,
P. Sotres, and L. Munoz, ‘‘Toward understanding crowd mobility in smart
cities through the Internet of Things,’’ IEEE Commun. Mag., vol. 57, no. 4,
pp. 40–46, Apr. 2019, doi: 10.1109/MCOM.2019.1800611.
[16] W. Tu, ‘‘Data-driven QoS and QoE management in smart cities: A tutorial
study,’’ IEEE Commun. Mag., vol. 56, no. 12, pp. 126–133, Dec. 2018, doi:
10.1109/MCOM.2018.1700870.
[17] Ericsson Mobility Report, Special Edition World Economic Forum, Erics-
son, Stockholm, Sweden, 2019.
[18] J. Manyika, M. Chui, P. Bisson, J. Woetzel, R. Dobbs, J. Bughin, and
D. Aharon, ‘‘The Internet of things: Mapping the value beyond the
hype,’’ McKinsey Global Inst., New York, NY, USA, Tech. Rep., 2015.
[Online]. Available: https://omegalambdatec.com/fileadmin/user_upload/
PDFs/Blog/OLT_McKinsey_Unlocking_the_potential_of_the_internet_
of_things_full_report.pdf
175791

[19] C. Harrison, B. Eckman, R. Hamilton, P. Hartswick, J. Kalagnanam,
J. Paraszczak, and P. Williams, ‘‘Foundations for smarter cities,’’ IBM J.
Res. Develop., vol. 54, no. 4, pp. 1–16, Jul./Aug. 2010.
[20] Smart Sustainable Cities: An Analysis of Definitions, document ITU-T FG-
SSC-10/2014, Focus Group Technical Report, Geneva, Switzerland, 2014.
[21] Human Factors (HF)—Quality of Experience (QoE) Requirement for
Real-Time Communication Services, Standard ETSI TR 102 643, Sophia
Antipolis, France. 2010.
[22] Mean Opinion Score Interpretation and Reporting, document ITU-T
P.800.2, Geneva, Switzerland, 2013.
[23] D.-H. Shin, ‘‘Conceptualizing and measuring quality of experience of the
Internet of Things: Exploring how quality is perceived by users,’’ Inf. Man-
age., vol. 54, no. 8, pp. 998–1011, 2017, doi: 10.1016/j.im.2017.02.006.
[24] D. Shin and Y. Hwang, ‘‘Integrated acceptance and sustainability evalu-
ation of Internet of medical things: A dual-level analysis,’’ Internet Res.,
vol. 27, no. 5, pp. 1227–1254, 2018, doi: 10.1108/IntR-07-2016-0200.
[25] D. Shin, ‘‘How does immersion work in augmented reality games? A user-
centric view of immersion and engagement,’’ Inf., Commun. Soc., vol. 22,
no. 9, pp. 1212–1229, 2019, doi: 10.1080/1369118X.2017.1411519.
[26] A. Floris and L. Atzori, ‘‘Managing the quality of experience in the mul-
timedia Internet of Things: A layered-based approach,’’ Sensors, vol. 16,
no. 12, p. 2057, 2016.
[27] Y. Ikeda, S. Kouno, A. Shiozu, and K. Noritake, ‘‘A framework of scal-
able QoE modeling for application explosion in the Internet of Things,’’
in Proc. IEEE 3rd World Forum Internet Things (WF-IoT), Dec. 2016,
pp. 425–429.
[28] R. B. Bagno, T. L. Leiva, and L. G. H. de Oliveira, ‘‘Innovation manage-
ment: Lessons learned from innovation diagnostic tools,’’ Product Manage.
Develop., vol. 14, no. 1, pp. 12–21, 2016, doi: 10.4322/pmd.2016.005.
[29] F. K. Y. Chan, J. Y. L. Thong, V. Venkatesh, S. A. Brown, P. J. H. Hu, and
K. Y. Tam, ‘‘Modeling citizen satisfaction with mandatory adop-
tion of an E-government technology,’’ J. Assoc. Inf. Syst., vol. 11,
no. 10, pp. 519–549, 2010. [Online]. Available: https://ssrn.com/
abstract=1976951
[30] V. Venkatesh, M. G. Morris, B. Gordon, and F. D. Davis, ‘‘User acceptance
of information technology: Toward a unified view,’’ MIS Quart., vol. 27,
no. 3, pp. 425–478, Sep. 2003.
[31] Largest Cities in Asia. Accessed: Jul. 1, 2019. [Online]. Available:
https://www.worldatlas.com/articles/the-largest-cities-in-asia.html
[32] Setiaji. (Nov. 28, 2018). Smart City Development in the Capital of Indone-
sia. [Online]. Available: https://files.acci.or.id/files/presentation/jsc-
smartcity-development-in-the-capital-of-indonesia.pdf
[33] D. F. Murad, B. S. Abbas, A. Trisetyarso, W. Suparta and C. Kang,
‘‘Development of smart public transportation system in Jakarta city based
on integrated IoT platform,’’ in Proc. Int. Conf. Inf. Commun. Tech-
nol. (ICOIACT), Yogyakarta, Indonesia, Mar. 2018, pp. 872–878, doi:
10.1109/ICOIACT.2018.8350812.
[34] L. G. M. Ballesteros, O. Álvarez, and J. Markendahl, ‘‘Quality of Expe-
rience (QoE) in the smart cities context: An initial analysis,’’ in Proc.
IEEE 1st Int. Smart Cities Conf. (ISC), Guadalajara, Mexico, Oct. 2015,
pp. 1–7.
[35] Common Requirements of the Internet of Things, document ITU-T Y.2066,
Geneva, Switzerland. 2014. [Online]. Available: https://www.itu.int
[36] J. C. Boarder, ‘‘The system engineering process,’’ Proc. Oper.
Res. Manage. Sci., pp. 293–298, Jun. 1995. [Online]. Available:
https://ieeexplore.ieee.org/xpl/conhome/3996/proceeding
[37] Kompas. Ganti Lampu Jalan Jadi LED, DKI Hemat Tagihan Listrik
Hingga Rp 300 Miliar. Jul. 9, 2019. [Online]. Available: https://
megapolitan.kompas.com/read/2019/03/29/21240391/ganti-lampu-jalan-
jadi-led-dki-hemat-tagihan-listrik-hingga-rp-300-miliar
175792
M. Suryanegara et al.: 5-Step Framework for Measuring the QoE of IoT Services
MUHAMMAD SURYANEGARA (M’06–SM’16)
received the bachelor’s degree in electrical engi-
neering from Universitas Indonesia, in 2003,
the master’s degree from University College Lon-
don, U.K., in 2004, and the Ph.D. degree from the
Tokyo Institute of Technology, Japan, in 2011. He
is currently an Associate Professor in telecommu-
nications management with Universitas Indonesia,
where he is also a Lecturer with the Graduate
Program in Telecommunications Management,
Department of Electrical Engineering. He has published more than 60 aca-
demic articles as a main author or coauthor. He is the Principal Investigator on
the research area of ICT policy and technology management, the IoT, 4G/5G
and wireless communication technology, concerning both technical research
and regulatory aspects. Since 2013, he has been involved in international
regulatory activities as drafting group chairman for the Asia Pacific Telecom-
munity (APT) Preparatory Group for ITU’s World Radiocommunications
Conference (WRC-15) and WRC-19.
DIMAS AGUNG PRASETYO received the bach-
elor’s degree in electronics and instrumentation
from Universitas Gadjah Mada, in 2009, and the
master’s degree in telecommunications manage-
ment from Universitas Indonesia, in 2019. He is
currently a State Employee with the Directorate of
Standardization, Ministry of Communications and
Information Technology, Indonesia.
FERY ANDRIYANTO received the bachelor’s
degree in industrial engineering from Universitas
Gadjah Mada, in 2010, and the master’s degree
in telecommunications management from Univer-
sitas Indonesia, in 2017. He is currently a State
Employee with the Directorate of Standardiza-
tion, Ministry of Communications and Informa-
tion Technology, Indonesia.
NUR HAYATI received the bachelor’s degree
in applied science from the State Electronic-
Polytechnique of Surabaya, in 2010, and the
master’s degree in computer engineering from
Universitas Indonesia, in 2014, where she is cur-
rently pursuing the Ph.D. degree in computer
engineering with the Department of Electrical
Engineering. She is also a Lecturer with Univer-
sitas Muhammadiyah Yogyakarta. Her research
interests include embedded systems and security in
the IoT.
VOLUME 7, 2019