Politecnico

di Milano

Testing

V&V-1
1
Testing

Program testing can be used to show the presence of

bugs, but never to show their absence. (Dijkstra
1972)

Software
inputs under test outputs

Is output ok
no yes
wrt input?
Fail Success

2 V&V-1
Definitions

A test case t includes

A set of inputs for the system
A hypothesis on the state of the system at the time of
the test case execution
The expected output
Test set T
A set of test cases
If P is our system
A test case t is successful if P(t) is correct
A test set T is successful if P correct for all t in T

3 V&V-1
How to select test sets?

Randomly → random testing

It is "blind", it does not "look for bugs”
But it can be effective as it allows testers to generate
and execute hundreds of thousand of test cases
Using statistics it can be a powerful tool
Systematically → systematic testing
Use characteristics/structure of the software artifacts
(e.g., code) – white-box testing
Use information on the behavior of the system (e.g.,
specifications) – black-box testing

4 V&V-1
The testing process

5$)&2%0)$2
)$+$%&.1- %(.&$(.0
3#$%./.%0&.1-9
5$)&2%0)$2
1 6$-$(0&.1-
)1*(%$2%1'$

,'$-&./.%0&.1-
5$)&2 1/2$"#$%&$' ($)*+&) 2
%0)$)

!"#$%&$'
($)*+&)
5$)&2
($)*+&)
31/&40($2 5$)&2($#1(&
718#0(.)1-
3 *-'$(2&$)&

4
5 V&V-1
Testing vs Analysis

testing: scope-complete:
a few cases of arbitrary size all cases within a small bound
 Politecnico
di Milano

Testing activities

V&V-1
7
V&V activities and software artifacts
(the V model)

Actual needs
and constraints Delivered
User acceptance (alpha, beta test)
Package

Review

System test Integrated

RASD
System
Analysis/
Review

High level Integration test

Design Subsystems
document
Verification Analysis/
Review
Validation Unit/
Units/
Legend Component Unit test Components
Specs
8 V&V-1
Unit Testing

Conducted by the developers themselves

Aimed at testing sections of code
Why unit testing?
Find problems early
Facilitate changes
Guide the design
Visual feedback

9 V&V-1
Unit testing and scaffolding

The problem: components may not work in isolation

Thus, we need to simulate components that are
missing.
Example: B is used by A and uses C. We want to unit
test B
Driver for B
A
(simulates A) Oracle:
checks that,
A calls all functions of B
after test
execution,
B B the system is
in a correct
B calls function X of C state and the
output is
correct
C Stub of C

10 V&V-1
Integration Testing

Aimed at exercising
interfaces
modules’ interactions

11 V&V-1
Integration Faults

Inconsistent interpretation of parameters or values

Example: Mixed units (meters/yards) in Mars Climate Orbiter
Violations of value domains, capacity, or size limits
Example: Buffer overflow
Side effects on parameters or resources
Example: Conflict on (unspecified) temporary file
Omitted or misunderstood functionality
Example: Inconsistent interpretation of web hits
Nonfunctional properties
Example: Unanticipated performance issues
Dynamic mismatches
Example: Incompatible polymorphic method calls

12 V&V-1
Example from the Apache web server

Apache web server, version 2.0.48

Response to normal page request
Impossible to find iton secure
with unit (https)
port testing.
Inspection
Which problem do we haveand some dynamic
here?
techniques can find it.
static void ssl_io_filter_disable(ap_filter_t *f)
{ bio_filter_in_ctx_t *inctx = f->ctx;

inctx->ssl = NULL;
inctx->filter_ctx->pssl = NULL;
}
No obvious error, but Apache
leaked memory slowly (in
normal use) or quickly (if
13
exploited for a DOS attack)
V&V-1
Example from the Apache web server

Apache web server, version 2.0.48

Response to normal page request on secure (https)
port

static void ssl_io_filter_disable(ap_filter_t *f)

{ bio_filter_in_ctx_t *inctx = f->ctx;
SSL_free(inctx -> ssl);If the interface of the module
inctx->ssl = NULL; where f is defined offers
inctx->filter_ctx->psslSSL_free
= NULL;and clearly specifies
}
how it should be used, the
problem could be found during
integration testing
14 V&V-1
 Integration Plan + Test Plan

... Integration test plan

... drives and is driven
by the project “build
Build Plan Test Plan plan”
A key feature of the
... system architecture
System Architecture and project plan

15 V&V-1
How to achieve integration & testing

Big bang (better to avoid it)

Incrementally (facilitates bug tracking)

16 V&V-1
Big Bang integration & testing

An extreme and desperate approach: Test only after

integrating all modules
Does not require stubs/drivers/oracles
The only excuse, and a bad one
Minimum observability, diagnosability, efficacy,
feedback
High cost of repair
Recall: Cost of repairing a fault rises as a function of time
between error and repair

17 V&V-1
Incremental integration & testing

It occurs while components are released

E.g., as soon as a first version of components A and
B is released, we integrate them and test the
integration
Then, if tests pass, we integrate also the first version
of C that has been released in the meanwhile
And we test such new integration

18 V&V-1
Some strategies for integration & testing

Based on the hierarchical structure of the system

Top-down
Bottom-up
Threads: a portion of several modules that offers a
user-visible function
Critical modules

19 V&V-1
Top down .

Top

stub A stub B stub C

Working from the top level (in terms of “use” or

“include” relation) toward the bottom.
No drivers required if program tested from top-level
interface (e.g., GUI, CLI, web app, etc.)

20 V&V-1
Top down ..

Top

A stub B stub C

Write stubs of called or

used modules at each step
stub X stub Y
in construction

21 V&V-1
Top down ...

Top

A B C

As modules replace
stubs, more functionality
stub X stub Y
is testable

22 V&V-1
Top down ... complete

Top

A B C

... until the program is

complete, and all
X Y functionality can be tested

23 V&V-1
Bottom Up .

Driver
Starting at the leaves of the
“uses” hierarchy, we never
need stubs

24 V&V-1
Bottom Up ..

Driver Driver ... but we must

construct drivers for
each module (as in unit
testing) ...
X Y

25 V&V-1
Bottom Up ...

Driver

A
... an intermediate
module replaces a driver,
and needs its own driver
...
X Y

26 V&V-1
Bottom Up ....

Driver Driver

A B

X Y

27 V&V-1
Bottom Up .....

Driver Driver Driver

A B C

... so we may have

several working
X Y
subsystems ...

28 V&V-1
Bottom Up (complete)

Top

A B C

... that are eventually

X Y
integrated into a
single system.

29 V&V-1
Thread ...

Top

A C

A “thread” is a portion of several

modules that together provide a user-
X
visible program feature.

30 V&V-1
Thread ...

Top

A B C

Integrating one thread,

then another, etc., we
X Y
maximize visibility for the
user

31 V&V-1
Thread ...

Top

A B C

we can minimize stubs

and drivers, but the
X Y
integration plan may be
complex

32 V&V-1
Critical Modules

Strategy: Start with riskiest modules

Risk assessment is necessary first step
May include technical risks (is X feasible?), process
risks (is schedule for X realistic?), other risks
May resemble thread process in tactics for flexible
build order
E.g., constructing parts of one module to test
functionality in another
Key point is risk-oriented process
Integration & testing as a risk-reduction activity,
designed to deliver any bad news as early as possible

33 V&V-1
Choosing a Strategy

Structural strategies (bottom up and top down) are

simpler
But thread and critical modules testing provide better
process visibility, especially in complex systems
Possible to combine
Top-down and bottom-up are reasonable for relatively
small components and subsystems
Combinations of thread and critical modules integration
testing are often preferred for larger subsystems

34 V&V-1
System Testing

Conducted on a complete integrated system

Independent teams (black box)
Functional and non-functional requirements
Testing environment should be as close as possible
to production environment
System testing can be functional or... see next slides

35 V&V-1
Types of System Testing:
Performance Testing
Purpose
Identify bottlenecks affecting response time, utilization,
throughput
Benchmarking, i.e., establish a performance baseline
and possibly compare it with different versions of the
same product or a different competitive product
Prerequisites
Expected workload
Acceptable performance
Identifying
Inefficient algorithms
Query optimization possibilities
Hardware/Network issues

36 V&V-1
Types of System Testing:
Load Testing
Purpose
Expose bugs such as
memory leaks,
mismanagement of
memory, buffer
overflows
Identify upper limits of
components

How
Increase the load until
threshold
Load the system with the
maximum load it can
operate for a long period

37 V&V-1
Types of System Testing:
Stress Testing
Purpose
Make sure that the system
recovers gracefully after failure

How
Trying to break the system
under test by overwhelming its
resources or by taking
resources away from it

Examples
Double the baseline number
for concurrent users/HTTP
connections
Randomly shut down and
restart ports on the network
switches/routers that connects
servers

38 V&V-1
 Politecnico
di Milano

Short Overview of Some Testing Techniques

V&V-1
39
The testing procedure

5$)&2%0)$2
)$+$%&.1- %(.&$(.0
3#$%./.%0&.1-9
5$)&2%0)$2
)1*(%$2%1'$
.-#*&26$-$(0&.1-

,'$-&./.%0&.1-
5$)&2%0)$)
1/2$"#$%&$' ($)*+&)
.-#*&) 0-'2
%1(($)#1-'.-6
.-.&.0+ )&0&$
5$)&2%0)$)
!"#$%&$'
($)*+&)

31/&40($2 5$)&2($#1(&
718#0(.)1-
*-'$(2&$)&
5$)&2
($)*+&)
5$)&2$"$%*&.1- $-:.(1-8$-& ;)%0//1+'.-6<
40 V&V-1
The testing procedure

5$)&2%0)$2
)$+$%&.1- %(.&$(.0
3#$%./.%0&.1-9
5$)&2%0)$2
)1*(%$2%1'$
.-#*&26$-$(0&.1-

,'$-&./.%0&.1-
5$)&2%0)$)
1/2$"#$%&$' ($)*+&)
.-#*&) 0-'2
%1(($)#1-'.-6
.-.&.0+ )&0&$
5$)&2%0)$)
!"#$%&$'
($)*+&)

31/&40($2 5$)&2($#1(&
718#0(.)1-
*-'$(2&$)&
5$)&2
($)*+&)
5$)&2$"$%*&.1- $-:.(1-8$-& ;)%0//1+'.-6<
41 V&V-1
A trivial example of software under test
package programLogic;
public class Money {
private int fAmount;
private String fCurrency;
public Money(int amount, String currency) {
fAmount = amount;
fCurrency = currency; }
public int amount() { … }

public String currency() { … }

public Money add(Money m) {… }

public boolean equals(Object anObject) { … }

 A scaffolding for Money – MoneyTest (1)

package testSuite;
import static org.junit.Assert.*;
import org.junit.Before;
import org.junit.Test;
import programLogic.Money;
public class MoneyTest {
@Test
public void testAmount() {
Money money1EUR = new Money(100, "EUR");
Money money2EUR = new Money(1000, "EUR");
Money money1$ new Money(111, "$");
assertEquals(100, money1EUR.amount());
assertEquals(1000, money2EUR.amount());
assertEquals(111, money1$.amount());
}
A scaffolding for Money – MoneyTest (2)
@Test
public void testCurrency() {
Money money1EUR = new Money(100, "EUR");
Money money2EUR = new Money(1000, "EUR");
Money money1$ new Money(111, "$");
assertEquals("EUR", money1EUR.currency());
assertEquals("EUR", money2EUR.currency());
assertEquals("$", money1$.currency());
}
@Test
public void testAdd() {
Money money1EUR = new Money(100, "EUR");
Money money2EUR = new Money(1000, "EUR");
Money money1$ new Money(111, "$");
assertTrue("100 EUR + 1000 EUR = 1100 EUR",
money1EUR.add(money2EUR).equals(new Money(1100, "EUR")));
assertNull("100 EUR + 111 $ not possible", money1EUR.add(money1$));
}
…
}
How do we get ready to test execution
phase?
5$)&2%0)$2
)$+$%&.1- %(.&$(.0
3#$%./.%0&.1-9
5$)&2%0)$2
)1*(%$2%1'$
.-#*&26$-$(0&.1-

,'$-&./.%0&.1-
5$)&2%0)$)
1/2$"#$%&$' ($)*+&)
.-#*&) 0-'2
%1(($)#1-'.-6
.-.&.0+ )&0&$
5$)&2%0)$)
!"#$%&$'
($)*+&)

31/&40($2 5$)&2($#1(&
718#0(.)1-
*-'$(2&$)&
5$)&2
($)*+&)
5$)&2$"$%*&.1- $-:.(1-8$-& ;)%0//1+'.-6<
45 V&V-1
Reminder - how to select test sets?

Randomly → random testing

It is "blind", it does not "look for bugs”
But it can be effective as it allows testers to generate
and execute hundreds of thousand of test cases
Using statistics it can be a powerful tool
Systematically → systematic testing
Use characteristics/structure of the software artifacts
(e.g., code) – white-box testing
Use information on the behavior of the system (e.g.,
specifications) – black-box testing

46 V&V-1
Systematic testing:
black-box vs white-box
White-box testing is suitable for unit testing
Covering a small portion of software is possible

Black-box testing is suitable for integration, system

and acceptance testing
Specs usually smaller than code
Specs help identifying missing functionalities in the
system

47 V&V-1
 White-box testing: an example
1 //read i,j,n
2 while (i<n && j<n) { i<n && j<n
F
3 if (i < j)
4 i = i + j; T
5 else if (j < i)
T F
6 j = j + i; i<j
7 else break;
8 } i = i+j T F
j<i
9 //It continues...
j = j+i

(a) Define a test set that satisfies the

edge coverage criterion
(b) Define a test set that satisfies the condition coverage criterion
(c) Define a test suite that covers the following path:
1, 2, 3, 4, 8, 2, 3, 5, 6, 8, 2, 9

48 Testing
 White-box testing: edge coverage
1 //read i,j,n
2 while (i<n && j<n) { i<n && j<n
F
3 if (i < j)
4 i = i + j; T
5 else if (j < i)
T F
6 j = j + i; i<j
7 else break;
8 } i = i+j T F
j<i
9 //It continues...
j = j+i

Possible test set:

⟨i = 3, j = 4, n = 10⟩; ⟨i = 3, j = 3, n = 10⟩

49 Testing
 White-box testing: condition coverage
1 //read i,j,n
2 while (i<n && j<n) { i<n && j<n
F
3 if (i < j)
4 i = i + j; T
5 else if (j < i)
T F
6 j = j + i; i<j
7 else break;
8 } i = i+j T F
j<i
9 //It continues...
j = j+i

T
i<n && j<n i< n: T; j<n: F
F i< n: F; j<n: T

Possible test set:

⟨i = 3, j = 4, n = 10⟩; ⟨i = 4, j = 3, n = 10⟩; ⟨i = 3, j = 3, n = 10⟩

50 Testing
 White-box testing: path coverage
1 //read i,j,n
2 while (i<n && j<n) { i<n && j<n
F
3 if (i < j)
4 i = i + j; T
5 else if (j < i)
T F
6 j = j + i; i<j
7 else break;
8 } i = i+j T F
j<i
9 //It continues...
j = j+i

Define a test suite that covers the following path:

1, 2, 3, 4, 8, 2, 3, 5, 6, 8, 2, 9
Possible test set:
⟨i = 3, j = 4, n = 10⟩

51 Testing
A kind of black-box testing:
Model-Based Testing (MBT)
We can use models to devise test cases
The actual behavior of the software under test is
checked against the behavior specified by the model

52 V&V-1
Example of MBT from a state diagram:
online shopping user account

53 V&V-1
How do we identify test cases?

We can “cover” the state diagram

State coverage:
Every state in the model should be visited by at least
one test case
Transition coverage
Every transition between states should be traversed by
at least one test case.
This is the most commonly used criterion
A transition can be thought of as a (precondition,
postcondition) pair

54 V&V-1
Transition coverage

create – suspend – cancel – end

create – activate – suspend –
cancel – end
…. We continue until we
consider all transitions

55 V&V-1
MBT and automation

5$)&2%0)$2
)$+$%&.1- %(.&$(.0
3#$%./.%0&.1-9
5$)&2%0)$2
)1*(%$2%1'$
.-#*&26$-$(0&.1-

,'$-&./.%0&.1-
5$)&2%0)$)
1/2$"#$%&$' ($)*+&)
.-#*&) 0-'2
%1(($)#1-'.-6
.-.&.0+ )&0&$
5$)&2%0)$)
!"#$%&$'
($)*+&)

31/&40($2 5$)&2($#1(&
718#0(.)1-
*-'$(2&$)&
5$)&2
($)*+&)
5$)&2$"$%*&.1- $-:.(1-8$-& ;)%0//1+'.-6<
V&V-1
Random/Statistical Testing

=*&180&$' &$)&26$-$(0&.1- 3$$'

$-:.(1-8$-&
3#$%./.%0&.1-9
5$)&2%0)$2
)1*(%$2%1'$
.-#*&26$-$(0&.1-

,'$-&./.%0&.1-
5$)&2%0)$)
1/2$"#$%&$' ($)*+&)
.-#*&) 0-'2
%1(($)#1-'.-6
.-.&.0+ )&0&$
5$)&2%0)$)
!"#$%&$'
($)*+&)

31/&40($2 5$)&2($#1(&
718#0(.)1-
*-'$(2&$)&
5$)&2
($)*+&)
5$)&2$"$%*&.1- $-:.(1-8$-& ;)%0//1+'.-6<
V&V-1
Random testing: strengths and weaknesses

+ -
Completely automatic Input validity
Can be very extensive May find the same bug
Unbiased over and over
Allows testers to identify May identify several
completely unforeseen minor bugs not relevant
issues to most of the use cases
for the software
May result in frustration

58 V&V-1
 Capture & reply testing

5$)&2
%0)$)
A0-*0+ =*&180&$'
$"$%*&.1- $"$%*&.1- ;0/&$( /.()&280-*0+ $"$%*&.1-<

5$)&2
($)*+&)
31/&40($2 5$)&2($#1(&
>(0%+$
*-'$(2&$)&

5$)&2$"$%*&.1- $-:.(1-8$-& ;)%0//1+'.-6<

,& .) 0?+$ &12($%1('20-'2($@$"$%*&$ Often used for GUI
testing
Auto-regression testing
First test expensive
Low level recorded tests
59 V&V-1
 From Utting, Pretschner, Legeard,
Utting, Pretschner and Legeard
The elements of MBT A Taxonomy of Model-Based Testing
Journal Software Testing, Verification & Reliability
Volume 22 Issue 5, August 2012 Pages 297-312
Environment
Subject
SUT Software Under Test

Shared test&dev model

Redundancy
Separate test model

Model Deterministic / Non−Det.

Characteristics Timed / Untimed
Discrete / Hybrid / Continuous

Pre−Post
Transition−Based
Paradigm History−Based The model is the main input
Functional
Operational to the test generation
Structural Model Coverage
Data Coverage
Test Selection Requirements Coverage
Test Case Specifications
Criteria Random&Stochastic
Test Fault−Based
Generation Manual
Random generation
Graph search algorithms
Technology Model−checking
Symbolic execution
Theorem proving

Test
Manual vs automated and
Execution On/Offline Online / Offline
integrated with test generation
60 V&V-1

Figure 2. Overview of the Taxonomy

 Utting, Pretschner and Legeard
A summary
y do not, Source
Subject code
Environment
Adapted from Utting, Pretschner, Legeard,
on cannot
SUT
A Taxonomy of Model-Based Testing
Journal
Shared Software
test&dev modelTesting, Verification & Reliability
Informal reqs
Redundancy Volume 22 Issue 5, August 2012 Pages 297-312
Separate test model
e that ex- Input to test
Model
generation
ut of the
Deterministic / Non−Det.
Models
Characteristics Timed / Untimed
adaptor is Discrete / Hybrid / Continuous

e software Pre−Post
Transition−Based
n the test Pseudo random
Paradigm
numbers
History−Based
Functional
Operational

Structural Model Coverage

Data Coverage
volves the Test Selection Requirements Coverage
Test Case Specifications
Criteria
he model, Test
Random&Stochastic
Fault−Based
nsforming Generation Manual
Random generation
ifications, Technology Graph search algorithms
Model−checking
g up the Symbolic execution
Theorem proving
kes a sig- Test
d execut- Execution On/Offline Online / Offline

the SUT 61 V&V-1

heck their
Conclusion

Verification and validation take place in most phases of

development process
Many techniques available
A lot of possibilities for automation
We need to avoid the attitude: yes, I’m great in my work so I do
not need verification/validation
DEVELOPER
We need to plan for the V&V process carefully
QA person
We need to pay attention to the cohesion of
the team

SOFTWARE
62 V&V-1
 Politecnico
di Milano

References

V&V-1
63
References

C. Ghezzi, M. Jazayeri, D. Mandrioli Fundamentals of Software

Engineering, 2nd Edition, Prentice Hall, 2003 (Italian translation
Pearson Ed. 2004)
M. Pezzè, M. Young Software Testing and Analysis: Process,
Principles, and Techniques. 2008, John Wiley & Sons.
Michael Fagan, Advances in Software Inspections, IEEE
Transactions on Software Engineering, July 1986
https://doi.org/10.1109/TSE.1986.6312976
NASA, Software Formal Inspections Guidebook, Office of Safety
and Mission Assurance, NASA-GB-A302 approved August 1993,
https://ntrs.nasa.gov/search.jsp?R=19980228472
NASA, Software Formal Inspections Standard, NASA-STD-8739.9,
June 2013 (supersedes NASA-GB-A302),
https://standards.nasa.gov/standard/nasa/nasa-std-87399
Check list for inspections of Java code by Christopher Fox
(available on the course web site)
A tool for supporting code review (among other things)
http://phabricator.org

64 V&V-1
References

Code reviews for teams too busy to review code

(video)
https://www.youtube.com/watch?v=1m3eRFeCInY
Bacchelli, Bird, Expectations, Outcomes, and
Challenges Of Modern Code Review
http://research.microsoft.com/pubs/180283/ICSE%20
2013-codereview.pdf

65 V&V-1