Testing Methods

 Black-box (functional)
 White-box (structural)
– Background
– Statement coverage
– Decision coverage
– (Short circuit) condition coverage
– Dataflow coverage
 Coverage terminology
 Coverage tools

1
 Black-Box Testing

 Also called “functional testing”

 Testing based on requirements
 Major methods:
– Equivalence partitioning on inputs
– Equivalence partitioning on outputs
– Boundary value analysis
– Extreme value analysis
– Syntax testing

2
 Equivalence Partitioning

 On inputs:
– Identify input variables
e.g. side lengths, number of sides
– Identify valid and invalid value sets for each variable
e.g. fl j l > 0g valid; fl j l  0g invalid; fl j l not a numberg invalid
– Build complete test cases from those sets
e.g. (3, 4, 5), (-1, 3, 3), (3, a, 3), (3), (3, 4, 5, 6, 7)
 On outputs:
– Identify output events, output variables, (sets of) variable values
e.g. triangle types, angle types
– Build test cases that generate each one
e.g. (3, 3, 3), (2, 2, 3), (3, 4, 5), (2, 10, 2)

3
 Boundary Value Analysis

 Find some n such that as n either increases or decreases,

we go from one EC to another (e.g. valid to invalid)
 Identify values of n that are
– Close to boundary on each side
– On boundary (if possible)
EC 1 EC 2  Generate test cases for each
n
 n can be integer or real; input value, output value, number
of repetitions, etc.
 For number representing month: 0, 1, 12, 13
 For TRIANGLE example:
– On outputs: (3, 3, 2.99), (3, 3, 3), (3, 3, 3.01), (3, 4,
4.99), (3, 4, 5), (3, 4, 5.01), . . .
– Number of occurrences: (3, 3), (3, 3, 3), (3, 3, 3, 3), . . .

4
 Extreme Value Analysis

 Find an n again
 Choose extremely large/small values for n
 Generate test cases for each
 n can be integer or real
 n can be input value, output value, number of repetitions/occurrences,
etc.
 For TRIANGLE:
– On inputs/outputs: (0.000004, 0.000002, 0.000003), (43728947,
43928410, 54386880)
– On number of repetitions: () (empty test case), (3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3)

5
 Syntax Testing

-f
put -g filename
-m filename

get filename

 Basic steps:
– Identify target language and make explicit if not already
– Define syntax formally with state machine
– Write test cases to cover:
– All transitions
– All incomplete inputs
– Each change of a valid token to an invalid one
 For above grammar: (put -f foo), (put -g foo), (put -m foo bar), (get foo), (put -f), (get),
(put -f -g), . . .

6
 White-Box Testing

 Also called “structural testing”

 Basic idea: if 2 test cases execute code differently, cannot
be test-equivalent
while (p!=null) {
if (p->n==y) {
return p;  What criteria do we use to decide “executing the code dif-
}
p = p->next;
ferently”?
} ...
– “Coverage criteria”
 Hierarchies of coverage criteria
 Levels of coverage

7
 Statement Coverage

 A test suite achieves (100%) statement coverage on code if:

For every statement in the code, there is at least one test case which causes the program to
execute that statement
 Example: consider the insurance code

premium = 500;
if ((age<25) && (sex==male) && (!married)) {
premium += 1500;
} else {
if (married || (sex==female)) {
premium -= 200;
}
if ((age>45) && (age<65)) {
premium -= 100;
}
}

(from Software Testing in the Real World, Edward Kit)

8
 Statement Coverage

 In the insurance code:

– 3 “if” statements, 4 assignment statements
 Two test cases can execute all statements:
– One with ((age<25) && (sex==male) && (!married)) true
– One with ((age<25) && (sex==male) && (!married)) false,
but (married || (sex==female))
and ((age>45) && (age<65)) both true
 Statement coverage also called block coverage
– (to be precise: block coverage is just very closely related)

9
 Decision Coverage

 Decision: entire conditional expression after an if, while, etc.

 A test suite achieves (100%) decision coverage on code if:
– For every decision in the code, there is at least one test case that evaluates the decision
to true
– For every decision in the code, there is at least one test case that evaluates the decision
to false
 Also called “branch coverage”
 Three test cases achieve 100% decision coverage on insurance code:
– One with ((age<25) && (sex==male) && (!married)) true
– One with (age<25) false, but (married || (sex==female))
and ((age>45) && (age<65)) both true
– One with (age<25) false, but (married || (sex==female))
and ((age>45) && (age<65)) both false

10
 (Short-Circuit) Condition Coverage

 Condition: atomic expression with no logical operators, within decision

– Logical operator: &&, ||, !, etc.
– Every decision contains one or more conditions
– e.g. ((age<25) && (sex==male) && (!married)) : three conditions
 A test suite achieves (100%) condition coverage on code if:
– For every condition in the code, there is at least one test case that evaluates the condi-
tion to true
– For every condition in the code, there is at least one test case that evaluates the condi-
tion to false
 For many languages we need to assume short-circuit evaluation: e.g.
when evaluating B && C, if B false, don’t evaluate C
 Use phrase “short-circuit (SC) condition coverage” for this
 Need at least 5 test cases for SC condition coverage on insurance code

11
 Comparing Coverage Criteria

 Statement, decision, SC condition = “coverage criteria”

 If a test suite achieves decision coverage, it achieves statement coverage
– (There are unimportant, rare exceptions)
– Decision coverage “stronger than” statement coverage
 If a test suite achieves SC condition coverage, it achieves decision coverage
– SC condition coverage “stronger than” decision coverage
 US Military Standard DO-178B defines criteria for testing software at different safety lev-
els
– Highest level: needs SC condition coverage to satisfy

12
 Dataflow Coverage

 Definition statement (“def”): statement which assigns a value to a variable; e.g.

– “x = 3;” is a definition of x
– “add(&tree, i);” is a definition of tree
 Use statement: statement which uses the value of a variable
– Predicate use (p-use): statement which uses value in a decision; e.g.
“if (x == y) {...}” is a predicate use of x and y
– Calculation use (c-use): all other uses; e.g.
– “x = y+3;” is a calculation use of y
– “add(&tree, i);” is a calculation use of i
 Def-use pair: def of a variable x + a later use (in same method) of x, that can be reached
from the def by a path q with no other defs of x in-between
– Parameter decls considered “virtual defs” here

13
 Def-Use Coverage

 A test suite achieves (100%) def-use coverage on code if:

– For every def-use pair, there is at least one test case that executes the path q between
the def and the use
 C-use coverage: def-use coverage for c-uses
 P-use coverage: def-use coverage for p-uses

 Statement, decision, condition coverage: controlflow criteria

 Def-use, C-use, P-use coverage: dataflow criteria
 Some studies have shown that test suites achieving dataflow criteria are more effective than
those achieving controlflow criteria

14
 Coverage Terminology

 A test suite achieves (90%) statement coverage on code if 90% of the statements are exe-
cuted
– Similarly 90%, 80%, etc., decision coverage, def-use coverage, etc.
 Some statements cannot be executed (“dead code”)
– Consider these coverage elements infeasible for statement coverage
– Similarly, infeasible decisions, conditions, etc.
 Generally only count feasible coverage elements
– Thus say “100% feasible statement coverage”, etc.
 Minimal test suite for a criterion: one that achieves the criterion in the smallest number of
test cases

15
Coverage Tools

 Step 1: run program that either

– Compiles your software under test
(SUT) in a special way
– Adds statements to your SUT and
compiles it
– Instruments your SUT bytecode
 Step 2: run your compiled program
(your SUT)
 Step 3: run coverage reporting pro-
gram
– Can tell you which statements,
etc. in your SUT were executed
 Most tools measure only statement
coverage
 For C: gcc / gcov
 For Java: Cobertura

16