Introduction

Mapping Planning and Conduction

Analysis and Discussion of Results

Conclusion

A systematic mapping study on test

generation from Input/Output Transition
Systems
Sofia Costa Paiva
Adenilso Simao
Instituto de Ci
encias Matem
aticas e de Computac
ao
S
ao Paulo University, Brazil

August 27, 2015

1 / 19

Introduction

Mapping Planning and Conduction

Analysis and Discussion of Results

Introduction
Model-Based Testing
Problem Statement

Mapping Planning and Conduction

Goal and Research Questions
Search Strategy and Extraction Process

Analysis and Discussion of Results

General Information about Studies
IOTS Model Characteristics
Test Selection Criteria
Technology
Test Execution
Map of selected studies
Threats to validity

Conclusion

Conclusion

2 / 19

Introduction

Mapping Planning and Conduction

Analysis and Discussion of Results

Conclusion

Context
Software testing:
Demonstrate the actual and expected behaviors of a system
are at least similar
Manual and laborious process

Model-Based Testing (MBT)

Strategy to minimize time and resource limitations
Automate the generation of high-quality test suites from
behavioural models.
A model of desired implementation behaviour: the starting
point of MBT.

3 / 19

Introduction

Mapping Planning and Conduction

Analysis and Discussion of Results

Conclusion

Techniques used in MBT

Finite State Machines FSM
Test generation methods
from FSM guarantee
fault coverage using a
fault model

Input/Output Transition
Systems - IOTS
Existing works merely
indicates that test suite
must be generated in a
nondeterministic way

4 / 19

Introduction

Mapping Planning and Conduction

Analysis and Discussion of Results

Conclusion

Goal
Several recent studies have addressed issues related to test
generation from IOTSs
IOTS is a suitable formalism to model complex systems, such
as communication protocols and distributed systems

Systematic mapping study

Important step to identify relevant studies and shortcomings
Rigorous process of assessment and interpretation of all
available studies
Reported systematic mapping studies regarding MBT not
including IOTS formalism.

5 / 19

Introduction

Mapping Planning and Conduction

Analysis and Discussion of Results

Conclusion

Goal and Research Questions

Goal
Identify methods for test case generation from IOTSs through
a systematic mapping.
Research questions:
RQ1. What are the main characteristics of IOTS models
applied in the studies?
RQ2. What are the test selection criteria used in the studies?
RQ3. What are the technologies used in the test generation
from IOTSs?
RQ4. What are the features of the test execution
implemented in the studies?

6 / 19

Introduction

Mapping Planning and Conduction

Analysis and Discussion of Results

Conclusion

Mapping Planning and Conduction

Search strategy : using control studies

Keywords: test generation and IOTS
Engines: IEEE Xplore, Springer Link, Scopus, Web of
Knowledge, ACM Digital Library, ScienceDirect, Compendex
and Oxford Journals

Figure : Mapping Process

7 / 19

Introduction

Mapping Planning and Conduction

Analysis and Discussion of Results

Conclusion

Taxonomy of MBT approaches

8 / 19

Introduction

Mapping Planning and Conduction

Analysis and Discussion of Results

Conclusion

Taxonomy of MBT approaches

9 / 19

Introduction

Mapping Planning and Conduction

Analysis and Discussion of Results

Conclusion

Analysis and Discussion of Results

General information about studies

Figure : Distribution of studies over

the years

Figure : Distribution of
studies into forums

10 / 19

Introduction

Mapping Planning and Conduction

Analysis and Discussion of Results

Conclusion

Analysis and Discussion of Results

General information about studies

Computational support
TGV (6 studies)
TorX (1 study)
UPPAAL (1 study)

Tipe of evidence:
68% provide only examples
13% report a case study
8% report experimental results
10% only discuss the application of the approach

11 / 19

Introduction

Mapping Planning and Conduction

Analysis and Discussion of Results

Conclusion

Analysis and Discussion of Results

RQ1. IOTS Model Characteristics
Untimed (92%) and
Timed (8%)
Deterministic (88%) and
non-deterministic (12%)
Discrete (98%), hybrid
(1%) and continuous
(1%)

12 / 19

Introduction

Mapping Planning and Conduction

Analysis and Discussion of Results

Conclusion

Analysis and Discussion of Results

RQ2. Test Selection Criteria

Structural model coverage (14 studies)
Requirements coverage: test purpose (26 studies)
Requirements coverage: properties (5 studies)
Test case specification (7 studies)
Test case specification: ioco theory (31 studies)
Fault-based (12 studies)

13 / 19

Introduction

Mapping Planning and Conduction

Analysis and Discussion of Results

Conclusion

Analysis and Discussion of Results

RQ3. Technology
Random/non-deterministic (26 studies)
Search-based algorithms (26 studies)
Model-checking (8 studies)
Simbolic execution (10 studies)
Constraint solving (6 studies)
Inference rules (5 studies)
Data abstraction (1 study)

14 / 19

Introduction

Mapping Planning and Conduction

Analysis and Discussion of Results

Conclusion

Analysis and Discussion of Results

RQ4. Test Execution
Online (34 studies)
Offline (25 studies)
Communication:
Synchronous (26 studies)
Asynchronous (10 studies)
Distributed (12 studies)

Online and synchronous test generation have been widely used

adopted by the classical method of Tretmans.

Online testing use randomize choice techniques

provided only a probabilistic guarantee of coverage for
(unrealistic) a long time execution.

15 / 19

Introduction

Mapping Planning and Conduction

Analysis and Discussion of Results

Conclusion

Analysis and Discussion of Results

Map of selected studies regarding Test Generation criteria and
Technology

16 / 19

Introduction

Mapping Planning and Conduction

Analysis and Discussion of Results

Conclusion

Threats to validity

Selection bias was treated by defining our inclusion criteria for

gathering the largest possible amount of papers that would fit
the test generation from the IOTSs domain
A specialist validated the results so that inaccuracy in data
extraction and misclassification could be alleviated.

17 / 19

Introduction

Mapping Planning and Conduction

Analysis and Discussion of Results

Conclusion

Conclusion
Lack of a standard of IOTS model characteristics may restrict
the adoption of each method.
Widely used coverage criteria, such as state and transition
coverage, have been applied to IOTSs in only few recent
studies.
Most of test generation technologies are non-deterministic
There exist no standard fault models, leading to most test
generation algorithms being nondeterministic.

Completeness is guaranteed in theory, by repeating the

process an unbounded number of times.
Recent studies on IOTS-testing have adopted concepts of
FSM-testing
However, fault models (which guarantee the completeness of
test suites) has played a relatively small role in IOTS testing.

Future work:
Replication of study and include more evidence
Quality evaluation
18 / 19

Introduction

Mapping Planning and Conduction

Analysis and Discussion of Results

Conclusion

A systematic mapping study on test

generation from Input/Output Transition
Systems
Sofia Costa Paiva
sofialc@icmc.usp.br
Instituto de Ci
encias Matem
aticas e de Computac
ao
S
ao Paulo University, Brazil

August 27, 2015

19 / 19