Software Testing

No issue is meaningful unless it can be put to the test of decisive verification. C.S. Lewis, 1934

2001-10-10

PUM-I, Mariam Kamkar, IDA

Real life examples

First U.S. space mission to Venus failed. (reason: missing comma in a Fortran do loop) December 1995: AA, Boeing 575, mountain crash in Colombia, 159 killed. Incorrect one-letter computer command (Cali, Bogota 132 miles in opposite direction, have same coordinate code) June 1996: Ariane-5 space rocket, self-destruction, $500 million. (reason: reuse of software from Ariane-4 without recommended testing).
2001-10-10 PUM-I, Mariam Kamkar, IDA 2

Real life examples

Australia: Man jailed because of computer glitch. He was jailed for traffic fine although he had actually paid it for 5 years ago. Dallas Prisoner released due to program design flaw: He was temporary transferred from one prison to another (witness). Computer gave him temporary assignment.

2001-10-10

PUM-I, Mariam Kamkar, IDA

Why Testing and Analysis?

Software is never correct no matter which developing technique is used Any software must be verified. Software testing and analysis are
important to control the quality of the product (and of the process) very (often too) expensive difficult and stimulating

2001-10-10

PUM-I, Mariam Kamkar, IDA

Statement Coverage
Begin if ( y >= 0) then y = 0; abs = y; end; test case-1:
input: y = 0 expected result: 0 actual result: ?
2001-10-10 PUM-I, Mariam Kamkar, IDA 5

begin y >= 0
yes

y=0 abs = y

Branch Coverage
Begin if ( y >= 0) then y = 0; abs = y; end;
test case-1: input: y = 0 expected result: 0 actual result: ?
2001-10-10

begin y >= 0
no yes

y=0

abs = y test case-2: input: y = -5 expected result: 5 actual result: ?

6

PUM-I, Mariam Kamkar, IDA

Begin if ( x < 10 && y > 20) { z = foo (x,y); else z =fie (x,y); } end;
test case-1: input: x = -4, y = 30 expected result: ... actual result: ?
2001-10-10 no

x<10 && y>20

yes

z=fie (x,y)

z=foo (x,y)

test case-2: input: x = 12, y = 12 expected result: ... actual result: ?

7

PUM-I, Mariam Kamkar, IDA

Condition / Branch Coverage

Begin no yes if ( x < 10 && y > 20) { x < 10 z = foo (x,y); else z =fie (x,y); no y > 20 } yes end; z=fie (x,y) z=foo (x,y)
x<10 y>20 ----------------------------------test-case-1: t t test-case:2 t f test-case-3: f t test-case-4 f f
2001-10-10 PUM-I, Mariam Kamkar, IDA 8

Path Coverage
no

x <> 0

yes

x = 0, z = 12 x = 2, z = 6 z = sin(x)

z = z-x
yes

z > 10

no

z=0

z=z/x

x = 0, z = 7 x = 0, z = 13 x = 1, z = 5 x = 2, z = 15

2001-10-10

PUM-I, Mariam Kamkar, IDA

Path with loops

a d b c e

a c,b,d e d

2001-10-10

PUM-I, Mariam Kamkar, IDA

10

Input

P r o g r a m
Output

Failure?

Oracle
2001-10-10 PUM-I, Mariam Kamkar, IDA 11

Glass box testing! White box testing! Structural testing!

2001-10-10

PUM-I, Mariam Kamkar, IDA

12

Glass box testing

logical decision loops internal data structure paths ...

2001-10-10

PUM-I, Mariam Kamkar, IDA

13

Error, Fault, Failure

Can lead to

Can lead to Human error Fault

Failure

2001-10-10

PUM-I, Mariam Kamkar, IDA

14

Black box testing! Functional testing!

2001-10-10

PUM-I, Mariam Kamkar, IDA

15

Black box testing

incorrect or missing functions interface errors performance error
input

output
2001-10-10 PUM-I, Mariam Kamkar, IDA 16

Black box testing

Equivalence partitioning Boundary value analysis Decision Table Cause-Effect Exhaustive testing

2001-10-10

PUM-I, Mariam Kamkar, IDA

17

Equivalence partitioning

Invalid inputs

Valid inputs

outputs
2001-10-10 PUM-I, Mariam Kamkar, IDA 18

Specification: the program accepts four to eight inputs which are 5 digit integers greater than 10000.
input values
Less than 10000 Between 10000 and 99999 More than 99999

Number of input values

Less than 4
2001-10-10

Between 4 and 8

More than 8
19

PUM-I, Mariam Kamkar, IDA

Guidelines
If an input condition specifies a range: one valid and two invalid equivalence classes. A specific value: one valid and two invalid equivalence classes. A member of a set: one valid and one invalid equivalence classes. A boolean: one valid and one invalid class.
2001-10-10 PUM-I, Mariam Kamkar, IDA 20

10

Boundary value analysis

10000 9999 10001 99998

99999 100000

Less than 10000

Between 10000 and 99999

More than 99999

2001-10-10

PUM-I, Mariam Kamkar, IDA

21

Cause-Effect
#AUSES
C1: command is credeit C2: command is debit C3: account number is valid C4: transaction amount is valid

%FFECTS
E1: print invalid command E2: print invalid account
number

E3: print debit amount not valid E4: debit account print E5: credit account print

2001-10-10

PUM-I, Mariam Kamkar, IDA

22

11

C2

C3
not

and

E3

C4

2001-10-10

PUM-I, Mariam Kamkar, IDA

23

Data Flow Testing

DEF(S) = {x | statement S contains a definition of variable x} USE(S) = {x | statement S contains a use of variable x} S1 : S2 : i = 1; while (i <= n) DEFS (S1) = {i} USE(S2) = {i, n} (d1-i) (u1-i, u2-n)

definition-use chain (du chain) = [x, S, S] du-1: [i, S1, S2]

2001-10-10

(d1-i), (u1-i)
24

PUM-I, Mariam Kamkar, IDA

12

Data Flow testing

s = 0; i = 1; while (i <= n) { s + = i; i ++ } print (s); print (i); print (n);
2001-10-10

(d1-s) (d2-i) (u1-i, u2-n) (u3-s, u4-i, d3-s) (u5-i, d4-i)

du-s

du-s (u6-s) (u7-i) (u8-n)

PUM-I, Mariam Kamkar, IDA 25

Data Flow Testing

s = 0; i = 1; while (i <= n) { s + = i; i ++ } print (s); print (i); print (n);
2001-10-10

(d1-s) (d2-i) (u1-i, u2-n) (u3-s, u4-i, d3-s) (u5-i, d4-i) (u6-s) (u7-i) (u8-n)
PUM-I, Mariam Kamkar, IDA

dd: def-def dk: def-kill du: def-use ...

26

13

Program Slicing
s = 0; i = 1; while (i <= n) { s + = i; i ++ } print (s); print (i); print (n); i = 1; while (i <= n) { i ++ } print (i);

2001-10-10

PUM-I, Mariam Kamkar, IDA

27

Static Software Testing

Inspection (design, code)
overview preparation inspection rework follow-up

2001-10-10

PUM-I, Mariam Kamkar, IDA

28

14

 Walkthroughs (design, code, chapter of users guide,)

presenter coordinator secretary maintenance oracle standards bearer user representative
PUM-I, Mariam Kamkar, IDA 29

2001-10-10

Levels of Software Testing

Component Integration System Acceptance

2001-10-10

PUM-I, Mariam Kamkar, IDA

30

15

Components
driver Boundary conditions independent paths interface ...

Component to be tested

stub

stub

Test cases

2001-10-10

PUM-I, Mariam Kamkar, IDA

31

Classes of Integration Testing

Bottom-up Top-down Big bang Sandwich Regression

2001-10-10

PUM-I, Mariam Kamkar, IDA

32

16

2001-10-10

PUM-I, Mariam Kamkar, IDA

33

Bottom-up

ABCDEFG

BEF

DG

2001-10-10

PUM-I, Mariam Kamkar, IDA

34

17

Top-down

ABCD

ABCDEFG

2001-10-10

PUM-I, Mariam Kamkar, IDA

35

System Testing
Recovery testing (fault tolerant) Security testing Stress testing (volume, resources,) Performance testing (real-time, embedded system)

2001-10-10

PUM-I, Mariam Kamkar, IDA

36

18

Acceptance Testing
Alpha test: at the developers site, controlled environment Beta test: at one or more customer site.

2001-10-10

PUM-I, Mariam Kamkar, IDA

37

Termination Problem How decide when to stop testing

The main problem for managers! Termination takes place when
resources (time & budget) are over found the seeded faults some coverage is reached

2001-10-10

PUM-I, Mariam Kamkar, IDA

38

19

Oracle

Scaffolding

What can be automated?

Test case generation Termination

2001-10-10

PUM-I, Mariam Kamkar, IDA

39

Testing object-oriented systems

Easier
modularity encapsulation Quicker development Small methods reuse Interface identified early inheritance polymorphism Dynamic binding Complex interfaces More integration

Harder

2001-10-10

PUM-I, Mariam Kamkar, IDA

40

20

Goals of software testing: Historical Evolution

Years Prevent software faults Measure SQA test objectives 1980 Find faults 1981: Deutsch. Software project V and V
1979: Myers, Art of software testing cost complexity # applic.

1970

Establish confidence
1972 June, First formal conference software testing, university of North Carolina, Bill Hetzel.

1960
1957, Charles Baker distinguished debugging from testing

1950

Not distinguished from debugging

2001-10-10

PUM-I, Mariam Kamkar, IDA

41

And
Testing can show the presence, but never the absence of errors in software. E. Dijkstra, 1969

2001-10-10

PUM-I, Mariam Kamkar, IDA

42

21