Software Verification and Validation

Lecture No. 11
 Module 65:
Model based testing
Model based Testing

 What is a model?
A model is a description of a
system’s behavior.
 Models are simpler than the
systems they describe.
 Models help us understand
and predict system’s behavior.
For Testing:
State-machines as models
Specifications as models
e.g., UML diagrams
Formal Specifications as
models e.g., Object Z
…
 Model based Testing

Specifications:

?  Users needs (Requirements)

System Under Test
 Objectives (specific)
 Informal specification
 Formal specification

The answer will help test team to establish a clear relationship

between system under test, specification and objective to satisfy.
 Model based Testing

 Conformance testing:
 To confirm if an implementation conforms to its standard
 External tester applies a sequence of inputs to IUT and verifies
its behavior
 Issue1: preparation of conformance tests in coverage of IUTs all
aspects
 Issue2: time required to run test should not be unacceptably
long
 Two main limitations
 Controllability: the IUT cannot be directly put into a desired
state, usually requiring several additional state transitions

 Observability: prevents the external tester from directly

observing the state of the IUT, which is critical for a test to
detect errors
 Model based Testing

 Formal conformance testing techniques based on Finite state

machine (FSM)
 Generate a set of input sequences that will force the FSM
implementation to undergo all specified transitions

 Black box approach: only the outputs generated by the IUT

(upon receipt of inputs) are observable to the external tester

 Requirement of a Fault Model

 Model based Testing

 Conformance Testing

conformance relation
abstract abstract
model of S model of I

assumptions/ assumptions/
test hypothesis test hypothesis
conformance relation
precise
implementation I
specification S
Model based Testing

 Fault model:
 A fault model is a hypothetical
model of what types of faults
may occur in an
implementation
 Most fault models are
"structural", i.e. the model
is a refinement of the
specification formalism (or
of an implementation
model)
 E.g. mutations of the
specification or of a
correct implementation
Model based Testing

 Fault model:
 It may be used to construct
the fault domain used for
defining what "complete test
coverage" means single fault
hypothesis (or multiple
faults)
 A fault model is useful for the
following problems:
 Test suite development for
given coverage objective
 Formalization of "test
purpose“
 coverage evaluation and
optimization (existing suite)
 Diagnostics
Model based Testing

 Models represented as:

 Finite state machines
 Statecharts
 Grammars
 Markov chains
 Stochastic Automata
Networks
 Algebraic specications
such as typed attributed
graph transformation
systems TAGTS
 Formal logic e.g.
Temporal logic
 ...
Model based Testing

 We study systems specified

as state machine
 As one example of test case
extraction form model
 Module 66:
State machine for Testing
State machine as model - Testing
 Procedural software
– unit = single program, function,
or procedure
 Object oriented software
– unit = class
– unit testing = intra-class
testing
– integration testing = inter-class
testing (cluster of classes)
 dealing with single methods
separately is usually
– too expensive (complex
scaffolding)
 Therefore, methods are usually
tested in the context of the class
they belong to
State machine as model - Testing

 Natural representation with

finite state Machines
– States correspond to
certain values of the
Attributes
– Transitions correspond to
methods
 FSM can be used as basis for
testing
– e.g. “drive” the class
through all transitions, and
verify the response and the
resulting state
State machine as model - Testing

 Test cases are sequences of

method calls that traverse the
state machine
 State machine model can be
derived from:
– Specification
– Code
 also using reverse
engineering techniques
 or both …
State machine as model - Testing

 It is very important to
highlight that it is possible
to have a state machine
based model of:
 a single class
 a complete system
 Let us consider some
examples:
 State machine as model - Testing

 Item basket: Single Class “basket” example:

 State machine as model - Testing

 Seminar registration system: complete system example

State machine as model - Testing

 State-machine from code of a

system
 Assume we had code:
public class stack {
public final intmax = 10;
public Boolean isFull { … }
public add{ … }
public delete { … }
public destroy { … }
public { … }
public create{ … }
}
State machine as model - Testing

 Extracting states and

transitions:
 States:
– Initial: before creation
– Empty: number of
elements = 0
– Holding: number of
elements >0, but less than
the max Capacity
– Full: number elements =
max
– Final: after destruction
State machine as model - Testing

 Extracting states and

transitions:

 Transitions:
– create, destroy
– actions that triggers the
transition
 ex. Add, delete
 State machine as model - Testing

 Arranging transitions
 Initial -> Empty: action = “create”
– e.g. “s = new Stack()” in Java
 Empty -> Holding: action = “add”
 Empty -> Full: action = “add”
– if MAXcapacity=1
 Empty -> Final: action = “destroy”
– e.g. destructor call in C++, garbage collection in
Java
 Holding -> Empty: action = “delete”
– if s.size() = 1
State machine as model - Testing
State machine as model - Testing

 Fault model for FSM:

 Output fault the machine
provides an output different
from the one specified by the
output function
 Transfer fault the machine
enters a different state than
that specified by the transfer
function
State machine as model - Testing

 Fault model for FSM:

 Transfer faults with additional
states: number of states of the
system is increased by the
presence of faults, additional
states is used to model certain
types of errors
 Additional or missing
transitions: one basic
assumption is that the FSM is
deterministic and completely
defined (fully specified). So
the faults occur when it turns
out to be non-deterministic
and/or incompletely (partially)
specified
State machine as model - Testing

 Fault types:
• Missing or incorrect
 Transitions – new state is
legal but incorrect
 Events – valid message
ignored
• Extra, missing or corrupt state
• Sneak path: messages are
accepted when they should not
• Illegal message failure
• Trap door: system cannot get
out of that state
State machine as model - Testing

 Coverage Criteria:
We can choose one or more
of the following test
selection criteria:
 All states – testing passes
through all states
 All events – testing forces
all events to occur at least
once
 All actions – testing forces
all actions to be produced at
least once
State machine as model - Testing

 Test strategies:
 Possible strategies
 All round-trip paths
 All transition sequences
beginning and ending in
the same state
 All simple paths from initial
to final state
 This strategy will help you to
find
 All invalid or missing states
 Some extra states
 All event an action faults
 Module 67:
State machine as model - Testing
State machine as model - Testing

 We have a system model

represented as state-
machine
 We extract test cases
considering:
 Sequences in the system
 Test requirement
State machine as model - Testing

• We build round-trip path tree

– Which is built form a state
transition diagram
– We includes all round-trip
paths
• Transition sequences
beginning and ending in
the same state
• Simple paths for initial
to final state. If a loop is
present, we use only
one iteration
State machine as model - Testing

Our test strategy is based on

round-trip path trees will
reveal:
• All state control faults
• All sneak paths – messages
are accepted when they should
not
• Many corrupt states -
unpredictable behavior
• Sequencing Constraints
– Guard Conditions
– Requirement for FSM test
cases development
State machine as model - Testing
 State machine as model - Testing

 Considering our shopping cart example:

State machine as model - Testing

 We have seen:
 How can we extract state
machine from code and
specifications
 State machine
representation of a
system
 Model based testing
using state machine as a
model
 Lets have an example
State machine as model - Testing

 State transition testing is

useful where some aspect of
the system can be described in
what is called a ‘finite state
machine’.
 system can be in a (finite)
number of different states,
and the transitions from one
state to another are
determined by the rules of the
‘machine’.
 This is the model on which the
system and the tests are
based
State machine as model - Testing

 advantages of the state

transition technique
 model can be as detailed or
as abstract as you need it
to be.
 In case a part of the
system is more important
(that is, requires more
testing) a greater depth of
detail can be modeled.
 In case the system is less
important (requires less
testing), the model can use
a single state to signify
what would otherwise be a
series of different states.
 Module 68:
State machine based testing - example
State machine based testing – example

 Scenario:
 XYZ Consulting was
requested to develop a
shopping cart for Marks
and Spenser (a leading
shopping mall in the UK)
and the following
requirements have been
identified.
 The system would only
allow registered users to do
the shopping and
registration process is
already working.
 
State machine based testing – example

 Scenario:
 Your consultancy is
requested to implement
login to the system,
selection of items on sale
and updating them into the
cart.
 Customers has to make ten
purchases in one go and
then system allows them to
verify their payment details
and the delivery address.
 Customers can replace
items during the purchase
process and update
quantities.
State machine based testing – example
/fail
Identify User /fail

/pass
/start
Select Item Confirm Order
/reset /purchase
Idle

/continue Confirm
Add to Cart
Delivery Addr
#(Item) = 10 /proceed /proceed
/cancel Receive
Checkout
Payment
status = OK && Qty > ordered
State machine based testing – example

Test Suite - Structure

S No S-ID T-ID Inputs Exp Output Pre-Cond Post-Cond

We adopt
S No Intent Inputs Exp Output Pre-Cond Post-Cond
 State machine based testing – example
/fail
Identify User /fail

/pass
/start
Select Item Confirm Order
/reset /purchase
Idle

/continue Confirm
Add to Cart
Delivery Addr
#(Item) = 10 /proceed /proceed
/cancel Receive
Checkout
Payment
status = OK && Qty > ordered
 State machine based testing – example

Test Suite

S No Intent Inputs Exp Output Pre-Cond Post-Cond

1 Invalid Enter U name Invalid user Idle State Idle State
user Enter Passwd
Press Enter

Sequence: Idle  Identify User  Idle

State machine based testing – example
/fail
Identify User /fail

/pass
/start
Select Item Confirm Order
/reset /purchase
Idle

/continue Confirm
Add to Cart
Delivery Addr
#(Item) = 10 /proceed /proceed
/cancel Receive
Checkout
Payment
status = OK && Qty > ordered
 State machine based testing – example

Test Suite
S No Intent Inputs Exp Output Pre-Cond Post-Cond
1 Invalid Enter U name Invalid user Idle State Idle State
user Enter Passwd
Press Enter
2 Valid Enter U name Valid User Idle State Select Item
user Enter Passwd
purchasi Press Enter Cart Empty
ng

Sequence: Idle  Identify User  Idle

Sequence: Idle  Identify User  select Item

State machine based testing – example

/fail
Identify User /fail
/pass
/start
Select Item Confirm Order
/reset /purchase
Idle

/continue Confirm
Add to Cart
Delivery Addr
#(Item) = 10 /proceed /proceed
/cancel Receive
Checkout
Payment
status = OK && Qty > ordered
 State machine based testing – example
S No Intent Inputs Exp Output Pre-Cond Post-Cond
1 Invalid user Enter U name Invalid user Idle State Idle State
Enter Passwd
Press Enter
2 Valid user Enter U name Valid User Idle State Select Item state
Test Suite

purchasing Enter Passwd

Press Enter Cart Empty
3 Valid user Enter U name/Pwd Valid User Idle State Select Item state
purchasing Press Enter
Select Item Cart Not Empty
Add to Cart
Continue
4 Possible Sneak path, found due to lack of system check

Sequence: Idle  Identify User  Idle

Sequence: Idle  Identify User  select Item
Sequence: Idle  Identify User  *Select Item  *Add to Cart