See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/316883982

Quality assurance of web services: A systematic literature review

Conference Paper · October 2016

DOI: 10.1109/CompComm.2016.7924932

CITATIONS READS
2 225

5 authors, including:

Gulshan Saleem Farooque Azam

COMSATS University Islamabad National University of Sciences and Technology
7 PUBLICATIONS   24 CITATIONS    140 PUBLICATIONS   582 CITATIONS   

SEE PROFILE SEE PROFILE

Mm Muhammad Muhammad Nisar Ahmed

Kabul University University of Engineering and Technology, Lahore
1 PUBLICATION   2 CITATIONS    28 PUBLICATIONS   54 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Scheduling Algorithms View project

Software Project Management View project

All content following this page was uploaded by Mm Muhammad Muhammad on 01 November 2017.

The user has requested enhancement of the downloaded file.

 2016 2nd IEEE International Conference on Computer and Communications
Quality Assurance of Web Services: A Systematic Literature Review
Gulshan Saleem, Farooque Azam
Department of Computer Engineering, College of EME
National University of Sciences and Technology (NUST)
Islamabad, Pakistan
e-mail: Gulshan.saleem14@ce.ceme.edu.pk;
farooqu@ce.ceme.edu .pk
Nisar Ahmed
Department of Computer Engineering
University of Engineering and Technology
Lahore, Pakistan
e-mail: nisarahmedrana@yahoo.com
Abstract-The web has evolved and is being extended
continuously. Its services are becoming more and more
complex. Nevertheless, one of the principal benefits and
challenges in services of the web is for those services to be
offered in a useful, flexible, effective and reliable way. The
quality of web services is as much challenging as it is important
to meet. Testing of Web services is important due to their
wider application areas. Due to their benefits in every field,
business builders is trying to adopt this new style to provide
their service so need greater attention to meet the best quality
practices. This study is based on an investigation of testing
approaches for web services, assessment of their working
based on defined criteria and research questions which are
eventually helpful in addressing what type of parameters are
useful to provide the best quality web services. This provides
new research areas for the researcher to achieve expertise in
these identified areas too. This survey is based on such
techniques which have a practical implementation in real life
scenarios. The main purpose of this work is to analyze what
parameters are essential to provide quality assurance and how
the quality can be further improved. Moreover, this study also
provides help for the software quality assurance team to
understand the existing approaches, their advantages and
limitations and hence helps in applying and considering
suitable plans to achieve quality assurance of web services
depending on different scenarios.
Keywords-quality; testing; web services
I. INTRODUCTION
The World Wide Web has advanced into a channel
presenting a broad range of e-commerce and business
oriented services. An SOA (Service Oriented Architecture)
application is a collection of independent services, working
together to deliver a specific functionality.
Web services are software systems, used for integration
of different web applications using particular standards. Web
services have now turned out to be the most practical and
low-cost solution for communicating data that is distributed
978-1-4673-9026-2/ 16/$31 .00 ©20 16 IEEE
Muhammad U sman Y ounus
School of Electronic and Information
Northwestern Poly technical University
Xian, China
e-mail: usmanl644@gmail.com
Li Yong
School of Electronic and Information
Northwestern Poly technical University
Xian, China
e-mail: ruikel@nwpu.edu.cn
between applications running on different operating systems,
languages, and platforms. This was previously impossible.
The advantages of SOA and web services arrive at the cost
of adding more layers of complex ity to the environment
where they are used. Testing of these web services is a
critical job as each service requires different authentication
procedures and observes different security policies.
Web services can be categorized into two types: web
services used on intranet and web services used on the
Internet. Web services provided by intranet are used only
within the concerned organization and are not available for
use to the general public. Internet web services, on the other
hand, are open to using for all. Testing these two different
types of web services is a pole apart problem. An intranet
web service is secure as the organizations have control over
who accesses its web service. Since it is only accessible to
internal users, you are on the safe side. On the other hand,
with an internet web service, security becomes a dominant
issue and there are prominent scalability, security and
privacy concerns.
Another major challenge of web services testing is that
they have no User Interface (UI). Hence there is no UI to be
tested. They are hard to be tested manually but are ideal for
automated testing. As a consequence, testers need
programming skills and specific tools or algorithms to test
them. In short, testing web services is not a simple job to be
done and need special skills, expertise, and technology. As
adoption of web services have increased, more testing needs
to be done. Quality assurance job has become less
straightforward and more of an intellectual challenge due to
asynchronous nature of the business services.
Many tools have been put into practice to verify web
services and few approaches have been proposed to study the
quality of these tools. This study is based on two basic
testing methodologies: black box and white box testing for
web services. This work aims to address this major challenge
of quality assurance with respect to web services. A
comprehensive survey of currently proposed techniques for
1391
testing of web services and generated a comparison among
them is performed. This comparison provides the basis to
figure out the issues with these testing techniques. Finally,
this can address those areas which are not effectively
handled in these testing methods. The results of comparison
will help to specify more efficient testing techniques. Note
that it is possible that one technique will be the best in testing
one criterion and the worst in testing another one.
II. METHOD
To conduct fruitful analysis on testing of web services,
following steps are followed: by specifying research
questions and analyzing the selected data based on the
specified questions. Before conducting the analysis, previous
SLRs are also reviewed to use better guidelines. This work
has used guidelines from A. Fink [1], K. Petersen et al [2]
and M. I. Ladan [3]. The rest part of this section includes
details of these steps and the methods being used.
A. Research Question
We have formulated two main research questions to
assess the defined quality and research evaluation parameters.
RQ 1. How quality of web services is maintained?
RQ2. What kind of testing is done to assure quality?
RQ3. How quality of web services can be further
improved? Research question RQ1 is relevant to quality
assurance of web services which is being done through
present techniques. Furthermore, it includes some parameters
which are essential for quality assurance. This study is
mainly focused on the analysis of these parameters to answer
first research question. The selected parameters are
Reliability, Compatibility, Efficiency Interoperability, Code
Maintainability, Effectiveness, Complexity Measures,
Performance Measures and Traceability. Main purpose is to
make a detailed review of selected studies for the following
parameters. Below is the description of these parameters
which further elaborate the need for selected parameters.
Reliability: Reliability refers to the ability of the testing
tool to perform its operation, under specified conditions and
time duration, without failure. This parameter is intended to
evaluate how reliable the technique is in testing the web
services. Ideally, the tool should ensure 100 percent failure-
free performance. This parameter is evaluated based on
author's own view about the reliability of the technique, the
test results and operation description of the proposed tool.
Compatibility: Compatibility deals with the ability of
the tool to be operated on specified platforms, operating
systems or CPU architectures. This parameter evaluates if
the tool is able to run on multiple platforms and operating
systems. Being compatible with different platforms and CPU
architectures will increase the tool 's portability and therefore
will add to the overall quality of the tool.
Efficiency: Efficiency, in simple terms, is the ability of
software to perform its task by utilizing a minimum amount
of resources. This parameter will evaluate if the testing
software makes the most optimized use of system resources
like processor capacity, RAM, bandwidth etc. A tool
consuming too much disk space or memory indicates a low
efficiency. Similarly, if a testing tool or technique takes too
1392
much time to generate the output, it can be regarded as
inefficient.
Interoperability: This attribute refers to the ability of the
tool to successfully communicate with other systems if
required. Many times a system need to perform information
exchange through various mechanisms to interact with other
systems. For testing web services, this is a crucial factor, as
the tool must be able to interact and coordinate with external
components.
Code Maintainability: This parameter evaluates if the
code of tool can be maintained to correct defects, modify or
evolve in future. Code maintainability can be achieved by
following good coding standards, doing proper
documentation of code and maintaining consistency in terms
of programming language and coding standards. Code
modularization and Object Oriented Programming (OOP)
can also help in code maintainability in future.
Effectiveness: Effectiveness refers to the ability of the
testing tool to achieve the desired outcome. In the current
context, it refers to the ability of testing software to detect
and report defects of web services. If the proposed technique
or algorithm successfully satisfies its predefined
requirements of testing, it can be considered as effective.
Complexity Measures: This parameter refers to the
ability of the tool to measure the code complexity of web
service. Cyclomatic complexity is a quantitative measure of
code complexity which gives the number of linearly
independent paths in a program code.
Performance Measures: This parameter indicates the
ability of the tool to do performance testing. This deals with
testing the responsiveness and performance of the web
service under test, under specific workload and conditions.
Stress and load testing are two important types of
performance testing. Stress testing is aimed at testing the
performance of a system under extreme load, increasing its
maximum capacity. Load testing is done by executing the
system under specific predefined load to understand its
behavior.
Traceability: Requirement traceability is an essential
requirement management factor. It means tracing the life of a
requirement from its specification to development,
deployment, use and refinement in both forward and
backward direction. This includes questions like who
specified the requirement, why the requirement exists and
what other requirements depends on it. This will help the
requirement engineers to track what other components may
be affected if a modification is done to a particular
requirement. This parameter will indicate if the requirements
of the proposed tool can be linked to other software artefacts
like design, code, and test cases.
The above-mentioned quality parameters are the basic
quality points which should be achieved for assuring quality.
The second research question "What types of testing method
are better for testing of web services?" is about to analyze
selected studies to show the best approaches. Testing is
done through two approaches which are Black box testing
and white box testing. Other testing methods are derived
from these main two methods. Black box testing is the
functional testing of a system which is mainly done with
interfaces with defining input and output of the system.
White box testing is unit testing which is based on detailed
testing of each component of the system including a code of
the software application. The second research question is
helpful for the service provider to choose testing methods
before providing services. To answer this question, our
focus will be on parameters and testing method used in each
study.
Last research question "How quality of web services can
be further improved?" is helpful for researchers to identify
new issue areas which are not yet addressed. This needs
more attention towards the limitation and gaps of the existing
testing techniques. By identifying limitation of selected
studies, our task is to highlight those issues which can
significantly improve the quality of web services if solved.
B. Inclusion Criteria
Criteria for inclusion of study are:
1) Papers which are published from 1st January 2006 to
1st June 2014
2) Papers which are based on testing of web services
3) Papers which address for specific testing problem
4) Papers which have practical support for testing
technique
C. Exclusion Criteria
Following criteria is used to exclude a study:
1) Papers which are published before 2006
2) Papers which addresses issues related to web service
but not helpful in quality assurance.
D. Search Process
For search process, all the research work recently done
on testing of web services have collected. Firstly, different
digital databases are searched i.e. "ACM Digital Library",
"IEEE Xplore" and "Science Direct Elsevier" .
The search was based on search terms and text strings.
Search terms are "Testing", "Quality Assurance" and
"web services". The strings include "White Box testing on
Web Services", "Testing on Web Services", "Black Box
Testing on Web services", "Testing issues of web services",
and "Quality assurance of web services".
The first list of research data is obtained from applying
search terms to selected databases that are automatic search.
Then the manual searching is done for text strings which
provided the most specific list of studies. Next step is to
analyze retrieved list for practical support of technique. At
this stage, all required data have achieved but for up to date
research these are further checked for their date of
publication as no study should be older than the year 2006.
The next step was to sort out retrieved techniques depending
on their related work and relevancy to designed research
criteria. By following above three steps, twenty most recent
and diverse studies are selected which were further studied
one by one to perform analysis based on our quality
parameters questions and research evaluation criteria.
1393
E. Data Extraction
To answer research questions on which research analysis
is based, we have performed data extraction to extract
relevant material from selected studies. For that purpose,
guidelines from K. Petersen et al [2]have taken such as to
explore the introduction and methodology when information
from the abstract is not enough to conduct an analysis of a
technique. Following is the required extracted material from
all studies:
• The source of studies i.e. From journal or
conference, the title, and the names of author(s).
• The structure of each study conducted.
• The technique, pros and cons of technique
• Supporting facts of each technique.
• Tool support for a technique
• Adaptive facts required for technique
• Information needed to answer all research questions
The results of data extractions are explained in Result
section to elaborate their significance regarding research
questions.
III. RESULTS & DISCUSSION
A. Quality of Web Services
As a result of search performance, twenty research
studies are included and data extraction is done to extract an
adequate amount of data from each study to formulate
results. For the first research question, we have formulated
parameter results into tabular form as given in Table 1. The
symbols "P" and "A" are used for present and absent
respectively to show that the specific technique addresses
what parameters.
For reviewing selected studies, three main research
questions have defined in section I1(A). For assurance of
quality of web services through the testing mechanism for
reliability, compatibility, code maintainability, availability,
complexity measures, effectiveness, vulnerability, efficiency
and performance measures have been analyzed in detail.
Zhang et al in [4] have presented a WSRTM (Web Service
Regression Testing Model). Including WSRTM by T. Zhang,
50% testing approaches are applicable for testing the
reliability of web services [1 , 5-14]. For proper functioning
of web services, it is important for web service to work
orderly, have the capability of efficient integration and
operation.Except for X. Y. Guo et al [1] and M. Yan et al[15],
all analyzed techniques account for compatibility. Efficiency
means the quality or property of being efficient and near to
half techniques caters for efficiency quality parameters; all
other techniques are not considering efficiency in their
techniques. It is an essential need and most important aspect
that to ensure efficiency while performing and providing
services to their user. Besides efficiency, the effectiveness of
provided services is also important. Only M. M. Eler[7] and
T. D. Cao [12] did not cater for the effectiveness. Almost
65% testing technique ensures the efficiency of web services.
Efficiency parameter is assured by 65% techniques[l , 4-6, 8,
10, 15-19].
 TABLE l. ANALYSIS RESULT OF P ARAMETERSUSED FOR FIRST RESEARCH QUESTION

Selected Studies For Analysis

.- -- = =-=- -- N=- !::!.=- --!::!. .-=-

Pa.-ametel·
ee
'"' '"'=-"'
;n- ee ;:::- ;n- ;:::- 00- N ;;;
=- Vi Vi Vi =- Vi Vi =- 00-
Vi Vi !::!. =-
'" '" '" '" '" '" '" '" '" '" '" '"
A P P A A A P P A P P A P P P A A A A P
Reliability.
p p p p p P A A P P P P P P P P P P P A
Compatibility.
p p p p p P A P P P A P P A A - A P - P
Efficiency.
A p P A A A A P P A - - P A - P P P P A
Interopernbility
A p P A A A P A A A A A A A A P A P A A
Code Maintainability.
p p p p p P A P P P P P P P A P P P P P
Effectiveness.
A A A P A A P A A A A A A P P P A P P A
Complexity Measures.
A A A P P P P P A A A P A P P A P P A P
Performance Measures.

100 (e.g. from requirements to Components). Except for M. M.

90
80
70
60
50
40
30
20
10
o
B.
Figure 1. Results of evaluated parameter.
The analysis shows that only techniques proposed by T.
Zhang et al[5], T. Masood et al [6] , S.H. Shafm et al [4],
M.M. Eler et al [7], Y. Jiang et al [11], T.D. Cao et al [12], c.
Ma et al [12], N. EI Ioini et al [14] and M. S. Jokhio et al
[22] take into account code maintainability to ensure quality
in Web Services. Interoperability means that the web
services are able to operate with other systems. Only T.
Zhang et al [5], T. Masood et al [6], X'y'Guo et al [8], S.
Hanna et al [9], N. El Ioini et al [10], A. C. Ma et al [20], G.
Morales et al [21], N. EI Ioini et al [1O]and M. S. Jokhio et al
[22] proposed techniques that take into account
interoperability which is essential for testing web services.
Interoperability is an important factor while ensuring the
quality of a system but about more than half of the
techniques are not following it. While testing web services, it
is important to assure traceability that all predefined
functional requirements are fulfilled. Assurance of
Traceability may horizontal through all the test
documentation for a given test level (e.g. system testillg,
from test conditions through test cases to test scripts) or
vertical through the layers of development documentation
Eler et al [7] all other techniques ensure the traceability
parameter of testing.
Here is the graphical view of results ill figure 1, which
shows the major contribution for quality assurance of web
services, is done through maintaining Compatibility issues,
the effectiveness of services and Traceability of operations.
Other than this Efficiency and Performance measure also
catered for quality purpose.
Testing Method Used for Quality Assurance
The Survey has provided few repeatedly identified
testing methods through which testing of the web a service is
done such as Load testing, Regression Testing, Performance
Testing, Model Based Testing, Goal & Mutation Method,
Rule Based Testillg, Passive Testing, Contract Mutation
Method, Worst-Input Mutation Approach, Built-In Structural
Testing, Black Box test case method and Automated testing.
In a study[16] authors have presented an abstract model to
generate test cases which deal with black box testing. [5]
and [6] introduces new approaches for regression testing. [4]
Presents a mutation testing approach which can be classified
as white box testing because it involves modification of
program code to generate mutants. [17]Introduces a new
black box testing approach. A study[15] is based on a load
testing technique, which tests the overall behavior or
performance of the system under peak conditions. Hence, it
can also be categorized as black box testing approach.
[7]Presents a structural testing approach, which is clearly, a
white box testing method as it deals with the program
structure of web services. [8] Introduces a performance
testing method which is based on black box testing. A study
[9] presents a fault-based robustness testing technique which
is again a black box testing approach. In [10] authors have

1394
introduced a mechanism for test report generation which is The reason of more techniques from black box method can
an XML- based method for reporting of test results. be summarized as [23]:
According to our analysis, the majority of techniques are 1) Easy to use: Since this type of testing does not
model or state based which are working for web services involve code, testing is relatively easy as the testers
testing. While conducting analysis, we concluded that major do not need to have knowledge of internal structure
methods are model and state-based (Black-box Approach). of the system. Hence generation of test cases is a
Black box testing considers the system as a black box. relatively easy task.
This work mainly consider the inputs provided to the system 2) Less Time: As the testers only have to interact with
and the output that should be generated from these inputs as system's GUI, no time is utilized in analyzing the
a result, without going into the details of internal program structure of the code. This simplicity saves the time
structure. The tools used to perform black box testing mostly of testers and makes the testing process a lot quicker.
interact with the system's Graphical User Interface (GUI).
TABLE II. CHALLENGING AREAS FOR QU ALITY A SSURANCE OF W EB SERV ICES
I 2
Runtime Functionality Ability of system to do work for intended purpose
System Qualities 4
Performance )
Ability of system to response in time and throughput behavior of the system
6 7
Security Ability of a system to resist unauthorized attempts
, Availability the quality of being at hand when needed
10 I
Usability ease of use and learning ability of the system
IL
Interoperability " the ability of diverse systems and organizations to work together
14 15 10
Non-Runtime Modifiability Ability of system to accommodate changes in the software
System Qualities 7
Portability " Ability of a computer program to be ported from one system to another
I>
Reusability LV
Ability of existing application to reused in the new application
li
Testa bility LL
Ability of a software to demonstrates its fau lts
LJ L4
Business Cost and Schedule L)
System cost with respect to time expected life of the project and utilization of COTS
Qualities system
,6
Marketability " system use with respect to market competition
" " Appropriateness Aligmnent of team, software structure, expertise allocation and availability of the
jV

for Organization human input

31 32
Architecture Conceptual
jj
The integrity of a structure that is a composition of a number of small architectural
Qualities Integrity components.
34 35
Correctness. To satisfy all functional requirements of the system
36
Domain 37
Sensitivity " Potential of a system component to picking something being measured
Specific Qualities 39
Compatibility 40
The capability of orderly, efficient integration and operation.

3) Simplicity: When the testers have to deal with large
and complex systems, black box testing makes the
testing process simpler because testers are only
concerned with the valid and invalid inputs and the
desired output.
For testing web services, it requires testing from different
perspectives with less or no details of user side. And as the
services can be small and large so it has to use a method
which can be applicable for every sized service. So black
box methods are more convenient to use due to simplicity
and time efficient approach. As the web services are
combination of interfaces, the reason for using black box
based method is as followed:
1) Black box nature of web services
2) client side code is absent i.e. Separate Client side
C. Challenges for Quality Assurance
According to the analysis of previous studies, we
concluded that both runtime and non-runtime quality
measures must be fulfilled. In addition to above parameters,
some other system architectural parameters are also
important. Here some Quality parameters are outlined below
in TABLE 2 which can also helpful in the better testing of
web services. Regarding system architecture it should have
small structures meeting all requirements.
The results of comparison helped in specifying more
efficient testing techniques. As testing of web services
involves five different perspectives Developer who build that
service, Provider who owns the service integrator, third Part
and the end user who is always anonymous it is a
challenging task to assure quality at every point. Note that it
is possible that the one technique will be the best in testing
one criterion and the worst in testing another one. But the
important thing which needs to improve is to close the gap
between developer and user because ultimately only end user
going to use the service builds by the developer. But this is
not an easy task too. The other challenging issue is that
increasing number of APIs, as there is different API for each
service. Besides this issue, up gradation of services cause
also quality problems as it is possible that not every user
upgrade his service. So there is need to create such testing
methods that can work for all version of service instead of

1395
 maintammg quality for the different version at a time.
Important point is to assure performance of web services
through testing. To assure better quality in web services in
addition to above, there should be more extensive work
needed to:
• Determine the number of test attempts in terms of
frequency that should be necessary for assuring
quality
• Determine the required levels of testing according to
each perspective (i.e. Developer, Provider, Integrator,
Third Party and End User)
• Integration of automated test for small modules of
system i.e. APIs
• More test methods at integrator and third party level
• Here below in the table 2, few quality attributes are
mentioned which are helpful in finding atesting
domain to improve quality performance for every
type of services.
IV. CONCLUSION
Testing web Services is a significant problem that should
be studied carefully; the testing has to be extensive and
comprehensive to all important levels (unit, component, and
system level).In this paper, several selected web services are
evaluated based on several known important quality
assurance parameters. Quality is tested through different
techniques and among these major contributions are towards
four parameters which are efficiency, effectiveness,
compatibility and traceability parameters. According to
detailed survey, 75% techniques are a model based hence the
better techniques are those which are based on some
automatic test generation, model or state based. Such
techniques provide better ways in the utilization of web
services data with respect to different functional and not-
functional constraints of the system. For quality assurance of
web services, testing at three main levels such as an end to
end service to service and interface to interface testing is
important due to their special infrastructure. As the most
concerned factor is the user who needs better GUI with new
change and also provide interoperability and compatibility
with user machine. The other important factor which needs
more attention may include security, availability,
performance, conceptual integrity and usability which should
cater for a better quality of web services. This work can be
further enhanced by modifying parameters, increasing
number of selected studies and also by covering all the
digital databases. Moreover, it can be extended by including
testing on security issues of Web Services.
REFERENCES
[I] Guo, x.-y. , et al. Design and implementation of perfonnance testing
model for web services. in Infonnatics in Control, Automation and
Robotics (CAR), 2010 2nd International Asia Conference on. 2010.
IEEE.
[2] Petersen, K., et a!. Systematic mapping studies in software
engineering. in 12th international conference on evaluation and
assessment in software engineering. 2008. sn.
1396
View publication stats
[3] Ladan, M.l. Web services testing approaches: A survey and a
classification. in International Conference on Networked Digital
Technologies. 2010. Springer.
[4] Shafm, S.H., L. Zhang, and X. Xu. Automated testing of Web
Services system based on OWL-So in lnfonnation and
Communication Technologies (WJCT), 2012 World Congress on.
2012. IEEE.
[5] Zhang, T , et al. An approach of end user regression testing for
semantic web services. in Management and Service Science (MASS),
20 II International Conference on. 2011. IEEE.
[6] Masood, T , A. Nadeem, and S. Ali. An automated approach to
regression testing of web services based on WSDL operation changes.
in Emerging Technologies (lCET), 2013 IEEE 9th International
Conference on. 2013. IEEE.
[7] Eler, M.M., et a!. Built-in structural testing of web services. in
Software Engineering (SBES), 2010 Brazilian Symposium on. 2010.
IEEE.
[8] Luo, L. and X. Bai, Web services-based t est report generation.
Tsinghua Science & Technology, 2005. 10(3): p. 282-287.
[9] Chen, 1., et a!., Worst-input mutation approach to web services
vulnerability testing based on SOAP messages. Tsinghua Science and
Technology, 2014. 19(5): p. 429-441.
[10] EI [oini, N. and A. Sillitti. Open web services testing. in 2011 IEEE
World Congress on Services. 2011. IEEE.
[11] Jiang, Y., et a!. Test-data generation for web services based on
contract mutation. in Secure Software Integration and Reliability
improvement, 2009. SSIRl 2009. Third IEEE International
Conference on. 2009. IEEE.
[12] Cao, T-D. , et a!. Testing of web services: tools and experiments. in
Services Computing Conference (APSCC), 2011 [EEE Asia-Pacific.
2011. IEEE.
[13] Van, M. , et al. WS-TaaS: a testing as a service platfOlm for web
service load testing. in Parallel and Distributed Systems ([CP ADS),
2012 IEEE 18th International Conference on. 2012. IEEE.
[14] EI Joini, N. , A. Sillitti, and G. Succi. Using Rules for Web Service
Client Side Testing. in 2013 IEEE Ninth World Congress on Services.
2013. IEEE.
[15] Van, M., et a!. Building a TaaS platfonn for web service load testing.
in 2012 IEEE International Conference on Cl uster Computing. 2012.
IEEE.
[16] Casado, R., 1. Tuya, and M. Younas. An Abstract Transaction Model
for Testing the Web Services Transactions. in JCWS. 2011.
[17] Askarunisa, A. , K.A.J. Punitha, and A. Abirami. Black box test case
prioritization techniques for semantic based composite web services
using OWL-So in Recent Trends in InfOlmation Technology
(lCRTIT), 2011 International Conference on. 2011. IEEE.
[18] Hanna, S. and M. Munro. Fault-based web services testing. in
lnfonnation Technology: New Generations, 2008. lTNG 2008. Fifth
International Conference on. 2008. IEEE.
[19] Lecue, F. and N. Mehandjiev, Seeking quality of web service
composition in a semantic dimension. IEEE Transactions on
Knowledge and Data Engineering, 2011. 23(6): p. 942-959.
[20] Ma, C., et al. Web services testing based on stream x-machine. in
2010 10th International Conference on Quality Software. 2010. IEEE.
[21] Morales, G., et a!. Timed extended invariants for the passive testing
of web services. in Web Services (lCWS), 2010 IEEE International
Conference on. 2010. IEEE.
[22] Jokhio, M.S., et a!. Web services testing via goal and mutation. in
Engineering of Complex Computer Systems (lCECCS), 2013 18th
International Conference on. 2013. IEEE.
[23] Test Plant. 2016; Available fi'Om: www.redstonesoftware.com.