A Framework for Testing and Analysis

(c) 2007 Mauro Pezzè & Michal Young Ch 2, slide 1

 Learning objectives
• Introduce dimensions and tradeoff between
test and analysis activities
• Distinguish validation from verification
activities
• Understand limitations and possibilities of test
and analysis

(c) 2007 Mauro Pezzè & Michal Young Ch 2, slide 2

 Verification and validation
• Validation:
does the software system meets the user's real
needs?
are we building the right software?

• Verification:
does the software system meets the
requirements specifications?
are we building the software right?

(c) 2007 Mauro Pezzè & Michal Young Ch 2, slide 3

 Validation and Verification

SW
Actual Specs
Requirements System

Validation Verification
Includes usability Includes testing,
testing, user inspections, static
feedback analysis

(c) 2007 Mauro Pezzè & Michal Young Ch 2, slide 4

Verification or validation depends on
the specification
12345678

Example: elevator response

Unverifiable (but validatable) spec: ... if a user

presses a request button at floor i, an available
elevator must arrive at floor i soon...
Verifiable spec: ... if a user presses a request
button at floor i, an available elevator must
arrive at floor i within 30 seconds...

(c) 2007 Mauro Pezzè & Michal Young Ch 2, slide 5

 Validation and Verification Activities
ramework for Test and Analysis

Actual Needs and

User Acceptance (alpha, beta test) Delivered
Constraints
Package
Review

System System Test System

Specifications Integration
Analysis /
Review

Subsystem Integration Test

Design/Specs Subsystem
Analysis /
Review
validation
Unit/Component Unit/
Specs Module Test Components

User review of external behavior as it is verification

determined or becomes visible

Validation
Legend

(c) 2007 Mauro Pezzè & Michal Young Ch 2, slide 6

 ever
You can’t always get what you want

Property
Decision Pass/Fail
Program Procedure

Correctness properties are undecidable

the halting problem can be embedded in almost
every property of interest

(c) 2007 Mauro Pezzè & Michal Young Ch 2, slide 7

 Degrees of Freedom 19

Getting what you need ...

Perfect verification of

• optimistic inaccuracy: we may

arbitrary properties by
logical proof or
exhaustive testing
(infinite effort) accept some programs that do
Theorem proving:
not possess the property (i.e.,
Unbounded effort to
verify general properties
it may not detect all
violations).
Model Checking:
Decidable but possibly – testing
intractable checking of
simple temporal properties
• pessimistic inaccuracy: it is
not guaranteed to accept a
Data flow
Typical
testing
program even if the program
analysis technique does possess the property
Precise analysis of
simple syntactic being analyzed
properties
– automated program analysis
techniques
Simplified Optimistic • simplified properties: reduce
the degree of freedom for
properties inaccuracy

Pessimistic
inaccuracy
simplifying the property to
check
Figure 2.2: Verification trade-off dimensions

(c) 2007 Mauro Pezzè & Michal Young Ch 2, slide 8

 Example of simplified property:
Unmatched Semaphore Operations
original problem simplified property
Java prescribes a
if ( .... ) { more restrictive, but
... statically checkable
lock(S); construct.
}
Static
...
checking for
if ( ... ) {
match is synchronized(S) {
...
necessarily ...
unlock(S);
inaccurate ... ...
}
}

(c) 2007 Mauro Pezzè & Michal Young Ch 2, slide 9

 Some Terminology
• Safe: A safe analysis has no optimistic
inaccuracy, i.e., it accepts only correct
programs.
• Sound: An analysis of a program P with respect
to a formula F is sound if the analysis returns
true only when the program does satisfy the
formula.
• Complete: An analysis of a program P with
respect to a formula F is complete if the
analysis always returns true when the program
actually does satisfy the formula.

(c) 2007 Mauro Pezzè & Michal Young Ch 2, slide 10

 Summary
• Most interesting properties are undecidable,
thus in general we cannot count on tools that
work without human intevention
• Assessing program qualities comprises two
complementary sets of activities: validation
(daes the software do what it is supposed to
do?) and verification (does the system behave
as specificed?)
• There is no single technique for all purposes:
test designers need to select a suitable
combination of techniques

(c) 2007 Mauro Pezzè & Michal Young Ch 2, slide 11