2013 IEEE 20th International Conference on Web Services
Evaluating Quality of Web Services: a Short Survey
Olga Kondratyeva1, Ana Cavalli
Department of Software Networks
TELECOM SudParis
Evry, France
{olga.kondratyeva, ana.cavalli}@it-sudparis.eu
Abstract — This paper presents a short survey on the quality
evaluation of web services. The most popular metrics for
estimating such quality and user perception of web services are
Quality of Service (QoS) and Quality of Experience (QoE),
which represent objective and subjective assessments
correspondingly. For different types of web services, the values
of QoS and QoE are measured in different ways. In this paper,
we consider various definitions of QoS based on web service
parameters and describe several methods for evaluating QoS
and QoE. We start with experimental evaluation of QoS based
on network traffic analysis and further turn to model based
methods for QoS estimating. Existing relationships between
different kinds of service quality evaluation are also discussed.
Keywords - web service; quality of service; trace models;
attribute/parameter; composite service quality.
I. INTRODUCTION
Nowadays web services are widely used in various
applications such as online multimedia services, booking
systems, online stores, etc. [1]. Most of web services are
distributed for free via Internet and the quality of such
services especially involved in critical systems [2] has to be
studied thoroughly. Since web services use the so-called
service-oriented architecture (SOA), which is highly
heterogeneous and platform-independent [3], the estimation
of their quality is rather complicated against ordinary web
applications. The functionality of web services is
scrupulously tested by other methods [4] and for this reason,
the functionality of web services is not considered when
estimating their quality.
In order to track the growing popularity of the topic we
studied a number of digital libraries including IEEE XPlore,
ACM Portal, Elsevier Online Library and Springer Online
Library. The obtained results for one of such libraries (the
IEEE Xplore [5]) are shown in Fig. 1. One may notice that
first, the popularity of the topic in 2010 has been increased
almost 40 times compared with 2000 and secondly, the
number of research devoted to the Quality of Experience
(QoE) is less than that devoted to the Quality of Service
(QoS). Other metrics for evaluating the quality of web
services are not taken into account, since QoS and QoE still
remain prevailing metrics by now [6].
Additionally, the importance of the topic is confirmed by
a number of existing and developing repositories of web
services where the QoS is somehow calculated [see, for
example, 7-10]. There exist several publications where the
___________________________________________________
1
– On leave from Tomsk State University
978-0-7695-5025-1/13 $26.00 © 2013 IEEE
DOI 10.1109/ICWS.2013.83
Natalia Kushik, Nina Yevtushenko
Department of Radiophysics
Tomsk State University
Tomsk, Russia
ngkushik@gmail.com, ninayevtushenko@yahoo.com
authors have collected the information about existing web
services and their QoS [see, for example, 1]. A special data
set that could be used by people from QoS evaluating
community for verifying their methods and techniques
against counterexamples (benchmarks) is presented in the
database [7]. A number of tools [see, for example, 11-14]
were developed for the QoS evalution.
201
Number of publications
151
QoS
101
QoE
51
1
1996 1998 2000 2002 2004 2006 2008 2010
Year
Fig. 1. Number of publications related to QoS and QoE of web services
In most papers the quality of service is defined on a set of
attributes (or parameters) [6, 7, 15, 16] where these attributes
mostly come from the traffic analysis. As it is mentioned in
[15], the major attributes to define the QoS are the response
delay, the package loss percentage, the service access facility
(the percentage when service is available), reliability (the
number of successful operations per certain time period), etc.
For estimating the quality of web services based on the
above parameters heuristic methods are usually proposed
[6, 17]. A usual way when evaluating the QoS based on
traffic analysis is to use linear combinations of weighted
network parameters [16]. Sometimes coefficients of the
formula are not given as they can be a know-how of a
company that evaluates the service quality for its purposes.
For many services that are located in repositories the QoS is
evaluated based on parameters mostly related to traffic
analysis. However, the parameters used in traffic analysis are
more concerned about the transport level than about service
issues; thus, new parameters appear that have to be
considered when evaluating the service quality. Such
parameters may be the amount of money to be spent, service
reputation, the comfort of a solution proposed by the service,
etc. A single utility function that maps the values of all
parameters into a single resulting value still is used for QoS
587
evaluation [18] but usually it is difficult to represent all these
parameters by means of a single value. Correspondingly, the
service quality is represented as a set or a vector of
heterogeneous attributes [19, 20]. The parameters which are
related to the traffic analysis are usually evaluated based on
the traffic monitoring but such parameters as the comfort of a
proposed solution, the number of money to be spent etc. can
hardly be estimated applying traffic based methods and thus,
there is a need for more complex models to be involved.
Taking into account sequences of various user requests,
cancellations etc., web services are often described by the
sets of permissible sequences of actions [1, 3] and for this
reason, researchers turn their attention to trace models such
as flow graphs, Markov chains, weighted and probabilistic
automata, Petri nets, fuzzy logic [21-27], etc.
We also mention that in spite of the fact that QoE and
QoS are known to be most popular metrics for evaluating the
quality of web services, still somehow it may happen that
QoE and QoS are not enough to correctly specify the real
quality [28]. Thus, people from Computer Science industry
propose to use other metrics to evaluate the quality of web
services. For example, A. Moorsel in his paper [28]
discusses the so-called Quality of Business (QoBiz) that is
evaluated in terms of money being received during executed
transactions. Another specific metrics is Quality of Design
(QoD) [29] that might be interpreted as the quality of
interaction between an end-user and a client application.
This paper contains a short survey of existing methods
and tools for evaluating quality of web services. In
particular, we also discuss the quality of a web service
composition for which a number of basic patterns is
considered. We further discuss the relationship between
various metrics and how QoS can be used for the QoE
evaluation.
The rest of the paper is organized as follows. Section II
contains preliminaries. Section III is devoted to methods for
QoS evaluation. Methods are divided into those which rely
on network traffic analysis and those that use trace models in
order to describe web service and to estimate its quality.
Section III also contains the state of the art for evaluating
QoS of web service composition while in Section IV, we
briefly discuss the relationship between some known
metrics/functions which are used for estimating the quality of
web service. Section V concludes the paper.
II. PRELIMINARIES
Web service can be defined as a composition of web
applications where a server (client) in one application can be
turned into a client (server) in another application. A similar
definition is given, for example, in [30], where web service
is defined as “a software system designed to support
interoperable machine-to-machine interaction over a
network”. The communication of simple component services
may be organized via a remote procedure call (RPC
technology), though, the simple object access protocol
(SOAP) is more often used. In order to support main features
of the heterogeneous service oriented architecture (SOA), the
interaction with services is processed through public
588
interfaces that are defined and described using XML-based
languages. The service description should contain the
description of service semantics and a machine-processable
description of the messages that are processed by the service
[30]. One of the basic standard languages is the web service
description language (WSDL). A number of service
depositories publish and provide the automatic analysis of
WSDL service files; for example, in [10], a ‘WSDL
Analyzer’ extracts the list of supported operations and a
required transport protocol that usually is HTTP. Complex
services may be derived as composition of simpler ones, and
special languages for the composite service description are
developed. Workflows which support the execution logic of
composite complex services can be described using the
business process execution language (BPEL) that for each
component service, defines which messages it gets from and
sends to other components. The sequences of message
exchange occurred in the system are usually described using
the web service choreography description language (WS-
CDL) [31]. Canonical descriptions of services allow an
automatic search for a service with the required
functionality, though, the information on the quality of a
selected service is usually not presented.
The QoS (Quality of Service) can be defined as a vector
with components which are values of given attributes
(parameters), such as delay time, number of packages being
lost, reputation, corresponding to a given web service. Such
vector can be mapped into a single value using an
appropriate computable function [32] and the result of this
function can be an integer, rational, (fuzzy) logic constant,
etc. The QoE is supposed to represent human satisfaction
with the service and thus, is highly subjective. Still, there is a
lot of research on automatic QoE computing based on the
values of QoS attributes [19].
Almost all existing methods for QoS evaluation can be
implicitly divided into two groups. Methods of the first
group mostly use passive and/or active network traffic
analysis to deduce some statistics about “good” and “bad”
packages, packages being lost or being delivered to a wrong
host, etc., and are well established. Such statistics allows
drawing a conclusion not only about network quality but
about the user satisfaction with service as well [see, for
example, 6]. Methods of the second group use more complex
formal models that describe the service properties based on
parameters essential for the quality evaluation, and these
parameters can be different from those utilized in traffic
analysis. In this paper, we briefly discuss the above types of
quality evaluation and classify them w.r.t. to
models/techniques they are based on.
Most services can be considered as composite services
containing simpler services as components and thus, two
questions arise: how to evaluate the quality of such
composite service (composition analysis) and which
components (with the same functionality) should be selected
in order to reach the best quality of their joint work
(decomposition/selection problem). In Section III, we briefly
discuss the problem of the composite service QoS
evaluation.
 III. METHODS AND TOOLS FOR QOS EVALUATIONS
A. QoS evaluation based on traffic analysis
For basic (non-composite) web services heuristic
methods are very popular when evaluating the quality [see,
for example, 15]. Most of these methods are based on
network traffic analysis. One of the reasons can be that
historically the quality was first estimated for networks; the
results then were used for the multimedia quality evaluation.
Correspondingly the same methods were inherited for the
evaluation of the quality of web services. Based on passive
and active traffic analysis results some statistics are collected
like the number of packages being lost, the maximal time
delay between transactions, the probability of losing a
package, bandwidth etc. [6] and how these parameters
influence the traffic QoS evaluation. As an example of such
monitor, one may turn to the paper [6] where the authors
focus on two network parameters. Those are network
delivery speed (bandwidth) and latency and the authors have
experimentally shown that end-user experience would be
better only if the response latency will be below 50mSec
while effective network bandwidth plays a crucial role in the
end-user satisfaction.
Traffic network parameters are weighted very precisely
for audio and video transmitting systems, since they are
known to be closely related to the end user satisfaction. As
an example of a metric for estimating the best route for
delivering a message, one may turn to Cisco protocol IGRP
[33]. The best route is selected based on integrated metrics
§ K ⋅B · § K5 ·
m = ¨ K1 ⋅ Be + 2 e + K 3 ⋅ Dc ¸ ⋅ ¨¨ ¸¸ ,
© 256 − L ¹ © r + K4 ¹
where K1, K2, K3, K4, K5 are coefficients which have values
between 0 and 1 and are defined by a network administrator,
Be is a channel bandwidth calculated as the ratio of the
integer 107 to minimal bandwidth (Kb/sec), L is a traffic
load with values 1, …, 255, Dc is one tenth of overall delay
measured in microseconds, r is a relative reliability, e.g., the
percentage of successfully transmitted packets via this route.
If K5 = 0 then the reliability is not taken into account. By
default, K1 = K2 = 1, K3 = K4 = K5 = 0, and Cisco
recommends not to change these parameters.
Similar formulae for the QoS evaluation are used for
multimedia services [see for example, 34]. We note that
coefficients of the formula may not be given in the literature
and sometimes they are considered as know-how of the
company that evaluates quality of its services. Still, there are
some statistical data available that may help developers and
testers to evaluate QoS of services they are using.
An obvious advantage on this approach is that the quality
estimation based on the traffic analysis is well established
and is widely used, especially in the industry. However,
these approaches which work rather well for evaluating
audio and video transmitting systems become almost useless
when the parameters of interest are related to the user
satisfaction such as the comfort of proposed solutions, the
amount of money to be spent etc. In order to illustrate this
fact we consider a simple example.
589
Consider the finite state system in Fig. 2 that describes a
vacation planner service. This automaton with slightly
changes is taken from [35, 36]. As usual, we use “?” for
input actions while using “!” for output actions.
!date
?date ?fstat
q1 q2
?date
!not reserved
q0 ?exit q4 q3
!reserved
!date
q6 ?hstat
!not reserved q5
Fig. 2. Automaton for vacation planner web service
At the initial state q0 the vacation planner “asks” a user to
define preferable dates of traveling and replies at state q1
with the dates available for plane tickets. The user enters
his/her opinion (agree/disagree) about the ticket and the
service moves to state q3 where two options are available. If
the travel conditions do not satisfy the user then the system
moves to state q4 where the user can change the travel dates
and continue or can quit the service via state q7 with the
corresponding output exit. If the user accepts the ticket
conditions provided by the service at state q3 then the service
proposes the dates for booking a hotel room, moves to state
q5 and “asks” the user if he/she accepts these hotel
conditions. If the conditions are accepted (not accepted) the
output is reserved (not_reserved). In both cases, the system
moves to the final state that coincided with the initial state in
our example. Taken apart traffic parameters we can evaluate
the quality of this service using a vector with two
components. The first component is responsible for the
execution time while the second is concerned about the
service popularity, i.e., the value of this component that is
between 0 and 1 corresponds to the percentage of users who
succeeded in planning their vacations via this service. Thus,
QoS = ¢t, r² where t is a rational corresponded to the
maximum time (in seconds) needed for the query execution
and r is popularity. In other words, if QoS = ¢35, 0.5² then
each query is executed in at most 35 seconds and on average,
50% of travellers who use this service will successfully plan
their vacations. Parameters related to the traffic analysis can
hardly help to evaluate the above values and for this reason,
the researchers are now switching to trace model based
evaluation of the quality of web services.
B. Trace models for evaluating QoS of web services
Different formal models are considered when describing
web service or/and evaluating its quality. Taking traffic
analysis apart, these models can be considered as
semidiscrete (semicontinuous) trace models, since usually
when describing a web service [4, 21, 25] a state transition
graph is used where the edges of the graph are weighted,
possibly, with probabilistic values. We note that trace models
are efficiently applied for verification, testing, or automated
design of web services [35-37], and therefore, integrating
quality parameters into such models for evaluating the
quality seems to be perspective. Lately, a number of methods
for the automatic translation of the web service description in
one of the XML-based languages (WSDL, BPEL, WS-CDL,
etc.) into corresponding formal models are developed
[31, 38-40]. In [38], for example, the authors develop a tool
for the two-way automatic translation between extended
versions of BPEL specifications and timed coloured Petri
net. In other papers [for example, 31, 40] finite state models
are considered. In [40] a push-down automaton is derived for
XML parsing and its application for web services is
considered. Automata augmented with timed parameters are
in the focus of [31], where it is proposed how to derive a
timed automata from the WS-CDL description of a service
composition. In [39], proper grammar rules for web service
interface verification are generated from WSDL description.
Despite the fact that in these papers the service quality issues
are not explicitly considered, the automatic translation of
service descriptions into formal models allows to use these
formal models for the quality evaluation process.
When evaluating such attributes of a web service as
reliability, reputation, popularity etc. probabilistic models
such as Hidden Markov Models [16], probabilistic automata
[26], fuzzy logic [27] etc. can be effectively used. The use of
most of these models provide an opportunity to conclude that
a given web service is good with a probability p or rather
good than bad and those parameters are also very important
as it is illustrated by the above example. The probability can
be treated as the ratio of the number of successful events to
the overall number of events (statistical definition), and thus,
may be of a good help when calculating relative parameters
for a service like popularity, reliability etc.
Consider a graph in Fig. 2. In order to estimate the time
that is needed to process a query we can accordingly weight
edges corresponded to each task and apply the methods used
for analyzing a weighted automaton [24]. On the other hand,
if we would like to estimate the popularity of the service then
transitions should be labeled with corresponding
probabilities that usually are derived due to known statistics.
In this case, the methods for analyzing a probabilistic
automaton [25] will be involved for the quality evaluation.
We note that almost for all models that take probabilities
into account the main problem is how the probabilities are
distributed, and thus, some companies/institutions keep those
probabilistic parameters as know-how.
C. QoS for web service composition
When a new service is derived by composing already
existing web services, possibly developed by a third-party,
the question arises whether existing components can provide
a required composition quality or some component services
should be re-designed. As before, we consider the QoS as a
vector of values of selected quality attributes such as cost,
execution time, reliability, etc., and when evaluating the
quality of service composition, it is usually supposed that the
values of quality attributes for components are given by
service providers or so-called service brokers [e.g. 41].
Given a set {S1, S2, … , Sn} of component services, their
QoS vectors {Q1, Q2,…Qn}, and the composition structure
590
CS, the question is what is the quality Q of the composition
of component services. As mentioned above, in this paper,
we do not consider the functioning of component services
and messages they exchange. Since we are only interested in
task invocations between components, the QoS of a
composite service can be evaluated using a corresponding
workflow, and in Fig. 3 we consider basic composition
patterns in which the workflow can be decomposed. When a
component service is invoked, it executes some task
(according to the composition requirements), and after
completing the task, the service either produces the result if
it is the final task, or invokes other components to execute
further composition tasks. The same component service can
be used for executing different composition tasks. To avoid
any ambiguity we further refer to the component service
when some task is executed by this service.
The simplest workflow compositional pattern is
sequential (Fig. 3a) where the composite service is organized
as follows: when service S0 completes a task service S1 is
invoked. In a conditional pattern (Fig. 3b), also referred to as
XOR-split, S0 invokes one and only one of services S1 … Sk
depending on the results of the task execution. When
probabilities are involved for each possible invocation the
k
equality ¦ pi = 1 must be held. When service S0 invokes
i =0
several services S1, …, Sk parallel pattern (Fig. 3c), or AND-
split pattern, is considered. Services executing tasks in
parallel can be further merged in order to execute a required
later task. If the next service is invoked only when all
preceding services have completed their tasks, the
synchronizing, or AND-joint, pattern (Fig. 3d) is considered.
Otherwise, if the next service is invoked after the first
service completes its task, the concurrent pattern (Fig. 3e), or
XOR-joint, is at hand. When some tasks should be repeated,
the loop pattern (Fig. 3f) is considered, and the number of
repetitions may be either known a priori or can be calculated
during the task execution. Without loss of generality, we
consider loops with a single starting point (service S1 in Fig.
3f) and any number of exits (service Si in Fig. 3f). For each
exit point probabilities of continuing the loop or of going out
may be specified.
S1 S1
p1
. .
||
S0 S1 S0 … . S0 … .
. .
pk
Sk Sk
a b c
S1 S1 …
Si
. . |
. … . … S1
Sk+1 Sk+1
. . …
Sk Sk
Sk
d e f
Fig. 3. Basic compositional patterns: (a) sequential, (b) conditional, (c)
parallel, (d) synchronizing, (e) concurrent, (f) loop
 TABLE I. AGGREGATION FUNCTIONS FOR QOS ATTRIBUTES

Compositional pattern
QoS Attribute Reference
Sequential XOR-split Parallel AND-joint Parallel XOR-joint Loop
k k k k k
Cost (c) ¦ ci ¦ ci ¦ ci ¦ ci n ¦ ci [20, 41]
i =1 i =1 i =1 i =1 i =1
n
k k k k
1 − ∏ (1 − ai ) § k ·
Availability (a) ∏ ai ∏ ai ∏ ai ¨ ∏ ai ¸ [18, 41]
i =1 i =1 i =1 i =1 © i =1 ¹
k k
Execution time, response time (t) ¦ ti max(ti ) max(ti ) min(ti ) n ¦ ti [20, 41]
i =1 i =1
n
k k k k
§ k ·
Reliability, success rate (r) ∏ ri ∏ ri ∏ ri ∏ ri ¨ ∏ ri ¸
[18, 21,
41]
i =1 i =1 i =1 i =1 © i =1 ¹
1 k 1 k 1 k 1 k 1 k
Reputation (q) ¦ qi ¦ qi ¦ qi ¦ qi ¦ qi [18, 21]
k i =1 k i =1 k i =1 k i =1 k i =1

For all basic patterns and their combinations the question
of overall QoS evaluation has been studied properly and for a
number of QoS attributes so-called aggregation functions
have been elaborated. The result of each aggregation
function is the value of a corresponding attribute calculated
for a proper pattern. Some of these functions without
considering probabilities for XOR-splits are summarized in
Table I. In Table II, probabilities for XOR-splits and loops
are taken into account when constructing aggregation
functions. In these tables, the integer k denotes the number of
involved services while the integer n is used for the number
of loop iterations, and pi is the probability of invocating the
service Si. The last column contains the references where
such aggregation functions are taken from.
In fact, the set of functions in Table I is incomplete, since
more attributes can be considered such as popularity,
especially for social networks [42]. When involving some
logic models, such as fuzzy logic, k-value logic etc., or
modular arithmetic models, aggregation functions which use
only sum and multiplication operators do not seem to be
sufficient and thus, novel corresponding aggregation
functions have to be elaborated. Moreover, some functions in
the table are rather questionable. For example, it is a bit
questionable that the reputation of the sequential
composition is just the average of component reputations. In
our opinion, this function should depend on service
objectives and for some services be rather the minimum of
reputations of component services.
Table II contains no aggregation functions for sequential
and parallel patterns since they coincide with those in
Table I; for a conditional pattern the average quality
evaluation is calculated, though the quality often is computed
[41] for each path separately, which allows to assess the
worst case, the best case, or the quality along the most
probable execution path. When probabilities are given, the
quality of a loop pattern is calculated for an arbitrary number
of iterations.

TABLE II. AGGREGATION FUNCTIONS FOR QOS ATTRIBUTES WHEN PROBABILITIES ARE INVOLVED

Compositional pattern
QoS Attribute Reference
XOR-split Loop

k k (∏ j −1
)(1 − p )(¦ c + ∏
i =0 p i j
j
i =1 i
k
i =1 pi ¦i = j +1 ci )
k

¦ p i ci ¦
(1 − ∏ p )
Cost (c) 2
[20, 41]
i =1 j =1 k
i =1 i

k
¦ pi a i ¦
k(∏ p )(1 − p )∏
j −1
i =0 i j
j
i =1 a i
Availability (a)
i =1 j =1 (1 − ∏ p a ) k
i =1 i i
[20]

k k (∏ j −1
p )(1 − p )(¦ t + ∏
i =0 i j
j
i =1 i
k
i =1 pi ¦i = j +1 ti )
k

¦ pi t i ¦
(1 − ∏ p )
Execution time, response time (t) 2
[20, 41]
i =1 j =1 k
i =1 i

591
 A vacation planner in Fig. 2 can be represented as the
composition of two services: Flight Booking (FB) and Hotel
Booking (HB) services. In the workflow in Fig. 4, nodes S
and F are initial and final nodes correspondingly.
After asking for a flight ticket a user has three options: to
ask for a hotel room when flight dates are set; to change
flight dates or to quit the service when the flight dates cannot
be changed for some reason. In the case of booking a hotel
room a request is processed only once. In both cases when a
hotel room is booked or not booked the service is quitted.
Correspondingly, using the above tables we can calculate the
QoS of the vacation planner when knowing the QoS of FB
and HB services.
S FB F divided into five groups.
HB
Fig. 4. Workflow for the vacation planner service
We also note that sometimes workflows, which are often
used for calculating the composition QoS, are insufficient for
the precise evaluation of the quality of the composite service.
Thus, more complex trace models such as finite transition
systems are needed to accurately evaluate the quality of the
composition when service component quality is given.
Another interesting issue devoted to the problem of a
quality-aware component selection is not considered in this
paper. The latter is often reduced to specifying a relevant
utility function and optimizing its value using classical local
and global optimization algorithms [43-48]. In the next
subsection, we present some existing tools for deriving a
quality-aware composition.
D. Tools for deriving a quality-aware composition
A number of tools and automated methods for deriving a
quality-aware composition based on service selection have
been developed [11-14]. Most approaches are developed
under the assumption that the compositional workflow (or a
work plan) is given, and problems of deriving service
composition and quality-aware service selection are
separated. However, in the ‘Qsynth’ tool [12] the authors
combine these two sub-problems and prove that it allows
increasing the scalability and enhancing the properties of the
QoS-aware composition. In order to handle QoS information,
special monitors are implemented, which dynamically collect
the QoS information, track history statistics and predict QoS
parameters in the real time using some simulation
techniques. The composition is derived to assure both
functional and quality constraints and the authors claim that
a proposed algorithm can provide the global optimal
solution. In [13], another approach that combines the
derivation of a quality-aware service composition and the
quality-aware component selection is proposed based on
classification techniques. Since the main objective of a
service is to satisfy an end-user, the authors mention that a
better selection can be performed when the history of user
592
requests is available. Experimental results are provided in
order to illustrate the effectiveness of a proposed service
filtering and ranking algorithm. In the ‘Selector’ tool [14]
functional and non-functional requirements for a composed
service are explicitly separated, and the tool takes into
account both quality and context parameters.
IV. THE RELATIONSHIP BETWEEN DIFFERENT METRICS
FOR QUALITY EVALUATING
The main purpose of any web service is to satisfy an
end-user and there still is no confidence that a user is
satisfied even when the QoS is rather high. For this reason,
another metric QoE for web services [28] is very popular
and there are many research papers on automatic estimation
of the QoE value and these methods can be implicitly
In many cases, the QoE is calculated manually based on
the expert evaluation. This approach is somehow inherited
from multimedia services, for example, when transmitting
video information after a number of experiments the experts
can evaluate the QoE (as an integer between 1 and 10, for
instance) while varying the number of lost packages or
response delays, and after many experiments the
dependency of QoE on QoS parameters is extrapolating.
The same experiments may be done for web services [see,
for example, 19]. More complex functions are considered
for automatic calculating QoE based on the known QoS
value.
The experimental evaluation of user satisfaction also can
be done based on passive and active traffic analysis as
proposed in [6]. End-user activity is being monitored, and
influence of service quality parameters (like latency,
bandwidth, etc.) on frequency of cancellation and reload
requests is assessed.
Some approaches are rather concerned on determining a
‘good’ continuous approximation function based on QoS
attribute values referring to psychophysics and cognitive
studies. If such function can be approximated using
exponential/logarithmic dependencies between QoS and
QoE [17] then the QoS evaluation is considered as
satisfactory. This dependency is similar to the dependency
of human perception on physiological stimulus magnitude
known as logarithmic Weber-Fechner law when interpreting
the values of QoS parameters as stimulus and the QoE value
as perception.
Methods of another group are based on artificial
intelligence techniques when automatic calculating the QoE
value, such as fuzzy logic, neural networks etc. Fuzzy logic
is used to calculate the correlation between measured
network parameters and to predict the value of QoE [see, for
example 27, 34]. Since this correlation differs from service
to service and is usually significantly non-linear, it is
essential to introduce adaptive algorithms to derive function
coefficients automatically for given services. In [27], for
example, neural network learning algorithms are used to
develop the fuzzy logic based tool to determine values and
grades (excellent, good, average, or poor) for services. As
mentioned above, the quality of novel tools may be verified
with statistical data published in datasets [7].
The last group of methods deals with new metrics rather
different from QoS and QoE. The Quality of Business
(QoBiz) [20] and Quality of Design (QoD) [29] can serve as
example of such metrics. In the paper [20], the QoBiz is
evaluated in terms of money being received during executed
transactions. Another specific metric QoD can be
interpreted as the quality of interaction between end-user
and client application. Below we show a picture (Fig. 5)
how the sets of attributes for some metrics are related.
The set of QoS The set of
parameters QoBiz Automatic evaluation
parameters
The set of QoD
parameters
Fig. 5. The relationship between sets of attributes used for different metrics
when evaluating the quality of web services
As it is shown in Fig. 5, the main purpose still remains
to automatically calculate the user satisfaction (QoE) based
on the known values of attributes. Based on the results of
this paper we expect that trace models could be very helpful
for such evaluation
V. CONCLUSION
In this paper, we have discussed an approach for
automatic QoS calculating that remains one of most general
metrics when evaluating the web service quality. The
methods and techniques for QoS evaluation implicitly fall
into two groups. The first group of methods is based on
experimental evaluation of QoS and QoE. Such experimental
evaluation might be performed based on passive or active
network traffic analysis and for these techniques linear
combinations are used as a formal model. The second group
of methods for evaluating QoS of web services contains
methods which rely on more complex formal models such as
trace models. We have also discussed how the quality of a
composite web service can be evaluated by the use of such
models.
At the same time, the main purpose of any web service is
to satisfy an end-user and despite of the fact that QoS
contains a number of new attributes which are far away from
traffic analysis and can be automatically calculated using
formal models, there is no confidence that a user is satisfied
even when the QoS is rather high. For this reason, we briefly
mentioned other metrics such as QoD, QoBiz, etc. for web
services and have described the correspondence between
them. More research is needed in order to establish a good
correlation between different metrics/functions used for
evaluating the quality of a web service, in particular,
between automatically estimated quality attributes and the
QoE, which is related to the quality of service that satisfies
an end-user. There is a number of research work on how to
automatically evaluate QoE based on the QoS results, but
this problem still remains unsolved. As a perspective, we are
planning to try our hand in developing a method for
automatic QoE evaluation based on trace models.
ACKNOWLEDGMENT
The work is partly supported by Futur et Ruptures
scholarship from Fondation Telecom (France) for Olga
Kondratyeva, and by Russian Ministry of High Education
and Science (contract  14.37.21.0622, Russia).
REFERENCES
[1] E. Al-Masri, and Q.H. Mahmoud, “Investigating web services on the
world wide web,” Proc. 17th international conference on World Wide
Web (WWW 08), ACM New York, NY, USA, 2008, pp. 795-804,
doi:10.1145/1367497.1367605.
The QoE value [2] C.C. Chang, and L. Hsueh-Ming, “Integration of heterogeneous
medical decision support systems based on web services,” Proc.
Ninth IEEE International Conference on Bioinformatics and
BioEngineering (BIBE 09), IEEE Press, June 2009, pp.415-422,
doi:10.1109/BIBE.2009.59.
[3] F. Curbera, et al., “Unraveling the web services web: an introduction
to SOAP, WSDL, and UDDI,” IEEE Internet Computing, vol. 6, no.
2, March-April 2002, pp. 86-93, doi: 10.1109/4236.991449.
[4] L. Bentakouk, P. Poizat, and F. Zaïdi, “Checking the behaviroal
conformance of web services with symbolic testing and an SMT
solver,” Proc. Test and Proof conference, (TAP 11), Zurich, 2011,
pp. 33-50.
[5] IEEE Xplore digital library, url: http://ieeexplore.ieee.org.
[6] S. Khirman, and P. Henriksen, “Relationship between Quality-of-
Service and Quality-of-Experience for public Internet service,” Proc.
Passive and Active Measurement Workshop 2002 (PAM 2002), Fort
Collins, Colorado, USA, March 2002.
[7] E. Al-Masri, and Q.H. Mahmoud, “Quality of web services dataset,”
url: http://www.uoguelph.ca/~qmahmoud/qws/index.html.
[8] Directory of Public SOAP Web Services, url: http://www.service-
repository.com.
[9] Online community for the Universal Description, Discovery, and
Integration OASIS Standard, http://uddi.xml.org.
[10] Web service depository XMethods, url: http://www.xmethods.net:
[11] J. Rao and X. Su, “A survey of automated Web service composition
methods,” in Proceedings of the first International workshop on
Semantic Web Services and Web Process Composition, San Diego,
California, USA, 2004, pp. 43–54.
[12] W: Jiang, C: Zhang, Z: Huang, M: Chen, S: Hu1, “QSynth : A Tool
for QoS-Aware Automatic Service Composition”, in Proceedings of
IEEE International Conference on Web Services (ICWS), 2010, pp.
42-49.
[13] W. Nagy, H. M. O. Mokhtar, Ali El-Bastawissy, “A Flexible Tool for
Web Service Selection in Service Oriented Architecture”, in
International Journal of Advanced Computer Science and
Application; Dec 2011, vol. 2 issue 12, pp. 191-201.
[14] A. Alti, A. Boukerram, P. Roose, “Selector: A tool for dynamic
service selectionand management”, in Journal of Computing, vol. 4,
issue 4, 2012, pp. 23-32.
[15] Hyun Jong Kim, Dong Hyeon Lee, Jong Min Lee, Kyoung Hee Lee,
Won Lyu, Seong Gon Choi, “The QoE Evaluation Method through
the QoS-QoE Correlation Model,” Proc. 4th Int. Conf. on Networked
Computing and Advanced Information Management (NCM 08),
Sept. 2008, vol. 2, pp. 719-725, doi: 10.1109/NCM.2008.202
[16] A. Morais, A. Cavalli, “Deliverable D2.1 – State of the art of
SQM/CEM technology, tools, and standartization”, IPNQSYS project,
2012, url: http://projects.celtic-initiative.org/ipnqsis/IPNQSIS-D21.pdf
593
[17] P. Reichl, S. Egger, R. Schatz, and A. D'Alconzo, “The Logarithmic
Nature of QoE and the Role of the Weber-Fechner Law in QoE
Assessment,” Proc. IEEE International Conference on
Communications (ICC 2010), IEEE Press, May 2010, pp.1-5,
doi:10.1109/ICC.2010.5501894
[18] S.X. Sun, and J. Zhao, “A decomposition-based approach for service
composition with global QoS guarantees,” Information Sciences, vol.
199, Sept. 2012, pp.138–153, doi:10.1016/j.ins.2012.02.061.
[19] F. Lalanne, A. Cavalli, and S. Maag, “Quality of experience as a
selection criteria for web services,” Proc. 8th Int. Conf. on Signal
Image Technology and Internet Based Systems (SITIS 2012), IEEE
Press, Nov. 2012, pp.519-526, doi: 10.1109/SITIS.2012.81
[20] J. Cardoso, A. Sheth, J. Miller, J. Arnold, and K. Kochut, “Quality of
Service for Workflows and Web Service Processes,” Journal of Web
Semantics, 2004, pp.281-308
[21] J. El Hadad, M. Manouvrier, and M. Rukoz, “TQoS: Transactional
and QoS-aware selection algorithm for automatic web service
composition,” IEEE Transactions on Services Computing, vol. 3,
issue: 1, 2010, pp. 73-85, doi:10.1109/TSC.2010.5.
[22] S. Dustdar, and W. Schreiner, “A survey on web services
composition,” International Journal of Web and Grid Services, vol. 1
iss. 1, August 2005, pp. 1-30.
[23] M.C. Jaeger, G. Rojec-Goldmann, and G. Muehl, “QoS aggregation
for web service composition using workflow patterns,” Proc.
Enterprise Distributed Object Computing Conf. (EDOC 2004), 2004,
pp.149 – 159.
[24] Proceedings of the annual IFIP conference TestCom (International
conference on Testing Communication Systems) (formely IWPT
(International Workshop on Protocol Testing) and ICTSS
(International conference on testing communicating systems and
software), 1988-2012.
[25] P.-C. Héam, O. Kouchnarenko, and J. Voinot, “Component
simulation-based substitutivity managing QoS aspects.” Electr. Notes
Theor. Comput. Sci. 260, 2010, pp. 109-123.
[26] B.A. Trakhtenbrot, and Y.M. Barzdin', “Finite automata: behaviour
and synthesis (fundamental studies in computer science),” Elsevier
Science Publishing Co Inc., U.S, 1973. 328 p.
[27] B. Pernici, “Evaluating web service QoS: A neural fuzzy approach,”
Proc. IEEE Int. Conf. Service-Oriented Computing and Applications
(SOCA), 2011, pp. 1-6.
[28] A. van Moorsel “Metrics for the Internet age: Quality of Experience
and Quality of Business,” Technical Report HPL-2001-179, 2001.
[29] P. Reichl, “From ‘Quality-of-Service’ and ‘Quality-of-Design’ to
‘Quality-of-Experience’: A holistic view on future interactive
telecommunication services,” Proc. 15th Int. Conf. on Software,
Telecommunications and Computer Networks (SoftCOM 2007), Sept.
2007, pp.1-6.
[30] D. Booth, H. Haas, F. McCabe, E. Newcomer, M. Champion,
C. Ferris, and D. Orchard, “Web services architecture,” W3C
Working Group Note, W3C Technical Reports and Publications, Feb.
2004, url:http://www.w3.org/TR/ws-arch/.
[31] G. Diaz, J.-J. Pardo, M.-E. Cambronero, V. Valero, and F. Cuartero,
“Automatic Translation of WS-CDL Choreographies to Timed
Automata”, in Formal Techniques for Computer Systems and
Business Processes, Lecture Notes in Computer Science, vol. 3670,
2005, pp 230-242.
[32] J.E. Hopcroft, R. Motwani, and J.D. Ullman, “Introduction to
Automata Theory, Languages, and Computation,” 2nd ed., Addison-
Wesley Publishing Company, 2001.
594
[33] Charles L. Hedrick Rutgers. “An Introduction to IGRP”, url:
http://www.cisco.com/en/US/tech/tk365/technologies white paper091
86a008000c8ael.shtml
[34] J. Pokherel, B. Wehbi, A, Morais, A. Cavalli, and E. Allitaire,
“Estimation of QoE of video traffic using a fuzzy expert systems,”
Proc. IEEE 10th Consumer communication & networking conf.
(CCNC 2013), January 2013, pp. 224-229, doi:
10.1109/CCNC.2013.6488450
[35] J.P. Escobedo Del Cid, Symbolic test case generation for testing
orchestrators in context, PhD thesis, Telecom SudParis, 2011.
[36] C. Gaston and P. Le Gall, “About incremental model-based testing of
web service orchestrations,” Slides from TAROT’2012,
url:http://tarot2012.univ-fcomte.fr/?talks.
[37] I: Rodriguez, G: Diaz, P: Rabanal, J:A. Mateo, “A centralized and a
decentralized method to automatically derive choreography-
conforming web service systems”, in The Journal of Logic and
Algebraic Programming, vol. 81, issue 2, Feb. 2012, pp. 127–159
[38] M. Diaz, V. Valero, H. Macia, J:A: Mateo, G: Diaz, “BPEL-RF Tool:
An Automatic Translation fromWS-BPEL/WSRF Specifications to
Petri Nets”; in Proceedings of The Seventh International Conference
on Software Engineering Advances (ICSEA), 2012, pp. 325-330.
[39] S. Halle, G. Hughes, T. Bultan, and M. Alkhalaf, “Generating
Interface Grammars from WSDL for Automated Verification of Web
Services”, in Proceedings of 7th International Joint Conference,
ICSOC-ServiceWave, Nov. 2009, pp. 516-530.
[40] W. Zhang, “An Adaptive XML Parser for Developing High-
Performance Web Services”, in Proceedings of IEEE Fourth
International Conference on in eScience, Dec. 2008, pp. 672-679.
[41] H. Zheng, W. Zhao, J. Yang, and A. Bouguettaya, “QoS analysis for
web service compositions with complex structures,” IEEE
Transactions on service computing, in press,
doi:10.1109/TSC.2012.7.
[42] E. Billionniere, D. Greiman, and K. Gosha, “A comparison of social
service selection techniques,” Proc. 8th IEEE Int. Conf. Dependable,
Autonomic and Secure Computing (DASC 09), Dec. 2009, pp.260-
265, doi: 10.1109/DASC.2009.24.
[43] Y. Yang, M. Dumas, L. García-Banuelos, A. Polyvyanyy, and
L. Zhang, “Generalized aggregate Quality of Service computation for
composite services,” The Journal of Systems and Software, vol. 85,
2012, pp. 1818– 1830.
[44] Tao Yu, and Kwer-Jay Lin, “Service selection algorithms for web
services with end-to-end QoS constraints,” Proc. IEEE Int. Conf. on
e-Commerce Technology (CEC 04), 2004, pp. 129-136.
[45] X. Deng, and C. Xing, “A QoS-oriented optimization model for web
service group,” Proc. 8th IEEE/ACIS Int. Conf. on Computer and
Information Science, 2009, pp.903-909.
[46] Yuan Yu Qian, and Hu Xiao Hui, “A web service selection approach
based on the authenticity of QoS data and the confidence of users,”,
Proc. Int. Symp. on Computer Network and Multimedia Technology
(CNMT 2009), Jan. 2009, pp. 1-5.
[47] L. Zeng, B. Benatallah, M. Dumas, J. Kalagnam, and Q.Z. Sheng,
“Quality driven web services composition,” Proc. 12th Int. Conf. on
World Wide Web (WWW 03), 2003, pp. 411-421,
doi:10.1145/775152.775211.
[48] Wu Du, and Hong Fan, “An automatic service composition algorithm
for constructing the global optimal service tree based on QoS,” Proc.
IEEE International Geoscience and Remote Sensing Symposium
(IGARSS), 2010, pp. 3976-3979.