Software Quality Engineering: Wrap-Up Conclusion

Dr.
Tamim Ahmed Khan 4/14/2021
Software Quality Engineering
Week 8
Agenda
 Wrap-up
 Conclusion
– Structural Testing
– Unit Testing
 From Unit to Integration Testing
– OO Protocol Testing
– Finite State Machines for Testing
SEN 309: Software Quality Enginering 1

Dr. Tamim Ahmed Khan 4/14/2021
Unit Testing (Wrap up)

 While developing unit test strategy, we
consider:
– Test Requirement
– Documentation requirement
– Tools / Test utilities?
 Determine extent of unit testing (i.e., in
advance).
– do not just “test until time expires”
– prioritize, so that important tests definitely
performed
 Decide how and where to get the test input
 Estimate the resources required
– use historical data if available
 Arrange to track time, defect count, type &
source
Unit Test Cases Guidelines

 Considering our white-box testing knowledge, we write test
cases to test operations ensuring:
– all instructions execute (statement coverage)
– all paths, including both sides of all branches execute(decision coverage)
– use of all called objects
– handling of all data structures
– handling of all files
– We studied CFG-based testing and DFG-based testing techniques
 Considering our black-box testing knowledge, we write test
cases to test operations considering:
– normal parameter values
– limit parameter values
– outside parameter values
– We studied equivalence classes based and decision table
based testing techniques.

OO Protocol
 Procedural software
– unit = single program, function, or procedure
 Object oriented software
– unit = class, class methods
– unit testing = intra-class testing
– integration testing = inter-class testing (cluster of
classes)
 dealing with single methods separately is usually
– too expensive (complex scaffolding)
 Therefore, methods are usually tested in the
context of the class they belong to
Class in terms of State-Machine

 Natural representation with finite state Machines
– States correspond to certain values of the Attributes
– Transitions correspond to methods
 FSM can be used as basis for testing
– e.g. “drive” the class through all transitions, and verify
the response and the resulting state
 Test cases are sequences of method calls that
traverse the state machine
 State machine model can be derived from:
– Specification
– Code
 also using reverse engineering techniques
 or both …

Assume we had code…

public class stack
{
public final intmax = 10;
public Boolean isFull { … }
public add{ … }
public delete { … }
public destroy { … }
public stack{ … }
public create{ … }
}
Extracting States and transitions

 States:
– Initial: before creation
– Empty: number of elements = 0
– Holding: number of elements >0, but less than
the max Capacity
– Full: number elements = max
– Final: after destruction
 Transitions:
– create, destroy
– actions that triggers the transition
 ex. Add, delete

Arranging transitions
 Initial -> Empty: action = “create”
– e.g. “s = new Stack()” in Java
 Empty -> Holding: action = “add”
 Empty -> Full: action = “add”
– if MAXcapacity=1
 Empty -> Final: action = “destroy”
– e.g. destructor call in C++, garbage collection in
Java
 Holding -> Empty: action = “delete”
– if s.size() = 1
State machine – stack

State based testing

The tests are derived from a state model of the
system. We can derive the state model in several
way, e.g. from
• Expected system behavior
• State part of a UML design or requirements
specification.
• Other state diagrams
Most system will, however, have a large number of

states
Coverage Criteria
We can choose one or more of the following test
selection criteria:
 All states – testing passes through all states
 All events – testing forces all events to occur
at least once
 All actions – testing forces all actions to be
produced at least once

Test Strategies
 Possible strategies
– All round-trip paths
– All transition sequences beginning and ending in
the same state
– All simple paths from initial to final state
 This strategy will help you to find
– All invalid or missing states
– Some extra states
– All event an action faults
State machine based testing – example

 Scenario:
– XYZ Consulting was requested to develop a
shopping cart for Marks and Spenser (a leading
shopping mall in the UK) and the following
requirements have been identified.
– The system would only allow registered users to
do the shopping and registration process is
already working.
– So we are working on a module of the whole
system making our testing a unit testing process
for the moment


 Scenario:
– Your consultancy is requested to implement login
to the system, selection of items on sale and
updating them into the cart.
– Customers have to make ten purchases in one go
and then system allows them to verify their
payment details and the delivery address.
– Customers can replace items during the purchase
process and update quantities.

/fail
Identify User /fail
/pass
/start
Select Item Confirm Order
/reset /purchase
/continue
Idle
Confirm Delivery
Add to Cart
Addr
#(Item) = 10 /proceed /proceed
/cancel
Checkout Receive Payment
status = OK && Qty > ordered


Test Suite - Structure
S No S-ID T-ID Inputs Exp Output Pre-Cond Post-Cond
We adopt
S No Intent Inputs Exp Output Pre-Cond Post-Cond

/fail
Identify User /fail
/pass
/start
/reset /purchase
Idle
/continue
Confirm Delivery
Add to Cart
Addr
/cancel


Test Suite
S No Intent Inputs Exp Output Pre-Cond Post-Cond
1 Invali Enter U Invalid user Idle State Idle State
d user name
Enter
Passwd
Press
Enter
Test Sequence: Idle  Identify User  Idle
Test Case: Start State: Idle
End State: Idle
Inputs: UID = Tamim, PWD=‘123’
Expected Output = “Invalid User”

/fail
Identify User /fail
/pass
/start
/reset /purchase
/continue
Idle
Confirm Delivery
Add to Cart
Addr
/cancel


Test Suite
S# Intent Inputs Exp Output Pre-Cond Post-Cond
1 Invalid Enter U name Invalid user Idle State Idle State
user Enter Passwd Uname or pwd
Press Enter not present
2 Valid Enter U name Valid User Idle State, Select Item
user Enter Passwd
purchasi Press Enter Cart Empty U_Name and
ng PWD present
Sequence: Idle  Identify User  Idle

Sequence: Idle  Identify User  select Item
/fail
Identify User /fail
/pass
/start
/reset /purchase
Idle
/continue
Confirm Delivery
Add to Cart
Addr
/cancel


/fail
Identify User /fail
/pass
/start
/reset /purchase
Idle
/continue
Confirm
Add to Cart
Delivery Addr
/cancel
Receive
Checkout
Payment

S No Scenario name Inputs Exp Output Start End state
state
1 Invalid user Enter U name Invalid user Idle State Idle State
Enter Passwd
Test Suite
Press Enter
2 Valid user Enter U name Valid User Idle State Select Item
purchasing Enter Passwd state
Press Enter Cart Empty
3 Select items Select Item Valid User Valid user Select item
Add to Cart identified
Continue Cart Not
Empty Select
item
4 Possible Sneak path, found due to lack of system check
Sequence: Idle  Identify User  Idle
Sequence: Idle  Identify User  select Item
Sequence: Select Item  Select item

MODEL BASED TESTING
Model based Testing

 What is a model?
• A model is a description of a system’s behavior.
• Models are simpler than the systems they describe.
• Models help us understand and predict system’s
behavior.
• For Testing:
• State-machines as models
• Specifications as models e.g., UML diagrams
• Formal Specifications as models e.g., Object Z
•…

Model based Testing

Implementation Model:
 Users needs
System Under Test ? (Requirements)
 Objectives (specific)
 Informal specification
 Formal specification
We intend to understand what should be the relation
between SUT and its specification that will allow us
to draw test cases from model and test SUT
The answer will help test team to establish a clear
relationship between system under test,
specification and objective to satisfy.
Model based testing

 This means, Tester shall consider model
– It could be system specifications
 Use case, sequence, state-charts, class, … [UML artefacts]
– It may be design artefacts
– It may be a mathematical representation such as
 Z, VDM, Object-Z
– It may be a graphical representation such as:
 Typed Attributed Graph Transformation systems
 State-machine
– Please note that CFG, DFG etc. are not under
consideration here,

Model based Testing

 Conformance testing:
 To confirm if an implementation conforms to its standard
– External tester applies a sequence of inputs to
Implementation Under Test (IUT) and verifies its behavior
– Issue1: preparation of conformance tests in coverage of
IUTs all aspects
– Issue2: time required to run test should not be
unacceptably long
 Two main limitations
– Controllability: the IUT cannot be directly put into a
desired state, usually requiring several additional state
transitions
– Observability: prevents the external tester from directly

observing the state of the IUT, which is critical for a test to
detect errors
Model based Testing

 Formal conformance testing techniques
based on Finite state machine (FSM)
– Generate a set of input sequences that will
force the FSM implementation to undergo all
specified transitions
– Black box approach: only the outputs

generated by the IUT (upon receipt of inputs)
are observable to the external tester
 Requirement of a Fault Model

Model based Testing
 Conformance Testing
conformance relation
abstract abstract
model of S model of I
assumptions/ assumptions/
test hypothesis test hypothesis
conformance relation
precise
implementation I
specification S
Model based Testing

 Fault model:
 A fault model is a hypothetical model of what types
of faults may occur in an implementation
– Most fault models are "structural", i.e. the model
is a refinement of the specification formalism (or
of an implementation model)
 E.g. mutations of the specification or of a
correct implementation

Model based Testing

 Fault model:
–It may be used to construct the fault domain used
for defining what "complete test coverage" means
single fault hypothesis (or multiple faults)
 A fault model is useful for the following problems:
–Test suite development for given coverage objective
–Formalization of "test purpose“
–coverage evaluation and optimization (existing suite)
–Diagnostics
Model based Testing

 Models represented as:
– Finite state machines
– Statecharts
– Grammars
– Markov chains
– Stochastic Automata Networks
– Algebraic specications such as typed attributed
graph transformation systems TAGTS
– Formal logic e.g. Temporal logic
– ...

Model based Testing

 In order to complete our earlier discussion
regarding state-machine extraction from code:
– We study systems specified as state machine
– As one example of test case extraction form
model
State machine as model - Testing

 Procedural software
– unit = single program, function, or procedure
 Object oriented software
– unit = class
– unit testing = intra-class testing
– integration testing = inter-class testing (cluster of
classes)
 dealing with single methods separately is usually
– too expensive (complex scaffolding)
 Therefore, methods are usually tested in the
context of the class they belong to


 Natural representation with finite state Machines
– States correspond to certain values of the
Attributes
– Transitions correspond to methods
 FSM can be used as basis for testing
– e.g. “drive” the class through all transitions, and
verify the response and the resulting state

 It is very important to highlight that it is
possible to have a state machine based model
of:
– a single class
– a complete system
 Let us consider some examples:


 Item basket: Single Class “basket” example:
 I want to test addItm, removeItem,
checkout,
 addItem [empty->filled] ; addItem ;
removeItem
 Seminar registration system: complete

system example

Coming back to our stack example…
Question?
 What if Account
– we have concept level status: enum
balanace: int
documents to consider isActive(): boolean
– Could be isBlocked(): boolean
isClosed(): boolean
 UML documents gotBalance(): int
 Specifications block()
unblock()
 etc. close()
deposit(amount: int)
withdraw(amount: int)

Answer!
 We develop state-machines considering these
artifacts…
 But how…
Account
status: enum
balanace: int
isActive(): boolean
isBlocked(): boolean
isClosed(): boolean
gotBalance(): int
block()
unblock()
close()
deposit(amount: int)
withdraw(amount: int)
Fault Models
 A fault model is a hypothetical model of what types of faults
may occur in an implementation
– Most fault models are "structural", i.e. the model is a refinement of
the specification formalism (or of an implementation model)
 E.g. mutations of the specification or of a correct implementation
– It may be used to construct the fault domain used for defining what
"complete test coverage" means
 E.g. single fault hypothesis (or multiple faults)
 A fault model is useful for the following problems:
– Test suite development for given coverage objective
– Formalization of "test purpose"
– For existing test suite: coverage evaluation and optimization
– Diagnostics

Fault Model for FSM

 Output fault the machine provides an output different from
the one specified by the output function
 Transfer fault the machine enters a different state than that
specified by the transfer function
 Transfer faults with additional states: number of states of the
system is increased by the presence of faults, additional states
is used to model certain types of errors
 Additional or missing transitions: one basic assumption is that
the FSM is deterministic and completely defined (fully
specified). So the faults occur when it turns out to be non-
deterministic and/or incompletely (partially) specified
Round-trip paths and Sequencing Constraints

• A round-trip path tree
– Is built form a state transition diagram
– Includes all round-trip paths
• Transition sequences beginning and ending in the same state
• Simple paths for initial to final state. If a loop is present, we use
only one iteration
• Is used to
– Check conformance to explicit behavioral models
– Find sneak paths
A test strategy based on round-trip path trees will reveal:
• All state control faults
• All sneak paths – messages are accepted when they
should not
• Many corrupt states - unpredictable behavior
• Sequencing Constraints
– Guard Conditions
– Requirement for FSM test cases development

Transitions with Constraints

Each transition in a state diagram has the form
trigger-signature [guard] / activity. All parts are
optional
 trigger-signature: usually a single event that
triggers a potential change of state.
 guard: a Boolean condition that must be true
for the transition to take place.
 activity: an action that is performed during
the transition.
Sneak path test cases

A sneak path – message accepted when it
should not be accepted – can occur if
 There is an unspecified transition
 The transition occur even if the guard
predicate is false

Model-based testing considering FSM

 We found out what sequences to be tested
– We finalized that we shall test all round-trip paths
considering all transitions
 We specified what coverage metric be insured
– We finalized:
 All sneak paths
 All trap doors
 All paths
 Be covered
 Now we are ready to develop test scenarios:
– We consider all positive and all negative cases
Summary
 Unit Testing – Conclusion
 OO Protocol Testing
 State-Machine based testing
 Model-based testing
 Next Lectures
– Specification-based Testing
– Grey-box testing: An Introduction
– Testing Classes Clusters
 Method-Message Path Testing
– Testing Object-Oriented Systems

Software Quality Engineering: Wrap-Up Conclusion

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Software Quality Engineering: Wrap-Up Conclusion

Uploaded by

Copyright:

Available Formats

Dr.

Tamim Ahmed Khan 4/14/2021

Software Quality Engineering

SEN 309: Software Quality Enginering 1

Unit Testing (Wrap up)

Unit Test Cases Guidelines

SEN 309: Software Quality Enginering 2

Class in terms of State-Machine

SEN 309: Software Quality Enginering 3

Assume we had code…

Extracting States and transitions

SEN 309: Software Quality Enginering 4

State machine – stack

SEN 309: Software Quality Enginering 5

State based testing

Most system will, however, have a large number of

SEN 309: Software Quality Enginering 6

State machine based testing – example

SEN 309: Software Quality Enginering 7

State machine based testing – example

State machine based testing – example

status = OK && Qty > ordered

SEN 309: Software Quality Enginering 8

State machine based testing – example

S No S-ID T-ID Inputs Exp Output Pre-Cond Post-Cond

State machine based testing – example

status = OK && Qty > ordered

SEN 309: Software Quality Enginering 9

State machine based testing – example

State machine based testing – example

status = OK && Qty > ordered

SEN 309: Software Quality Enginering 10

State machine based testing – example

Sequence: Idle  Identify User  Idle

status = OK && Qty > ordered

SEN 309: Software Quality Enginering 11

State machine based testing – example

State machine based testing – example

SEN 309: Software Quality Enginering 12

MODEL BASED TESTING

Model based Testing

SEN 309: Software Quality Enginering 13

Model based Testing

Model based testing

SEN 309: Software Quality Enginering 14

Model based Testing

– Observability: prevents the external tester from directly

Model based Testing

– Black box approach: only the outputs

 Requirement of a Fault Model

SEN 309: Software Quality Enginering 15

Model based Testing

Model based Testing

SEN 309: Software Quality Enginering 16

Model based Testing

Model based Testing

SEN 309: Software Quality Enginering 17

Model based Testing

State machine as model - Testing

SEN 309: Software Quality Enginering 18

State machine as model - Testing

State machine as model - Testing