Validation And

Verification

1
Testing
 Testing is a crucial part of software quality
assurance
 In industry it is not unusual for testing to
occupy between 30% to 49% of the project
budget
 For safety critical software testing may occupy
a cost of 3 to 5 times as much as the rest of
the software engineering activities

2
Testing Objectives
 Testing is a process of executing a program
with the intent of finding errors
 A good test case is one that has a high
probability of finding an as-yet undiscovered
error
 A successful test is one that uncovers an as-
yet undiscovered error
 Testing cannot show the absence of defects, it
can only show that software errors are present

3
What are we trying to achieve?
 Verification
- Are we building the product right?
- i.e. does the product conform to its specification?
 Validation

- Are we building the right product?

- i.e. does the product meet the users’ expectations
 Activities of this type are often referred to as V&V

4
 Types of testing
 There are two basic types of testing
- White box testing
 Where the structure of the code is known
 Typically undertaken by the developer of the code

 Test code on module by module basis

- moving toward the complete system

- Black box testing
 Tests all of the code according to the functional
specification
 Typically undertaken by a separate authority

5
White box testing
 Consider the internals of the modules that
make up the program
 Test the logical paths through the software
with test cases that apply to conditional
statements and loops
 However we cannot for many modules
carry out exhaustive testing

6
Example of Exhaustive Testing

 Consider a C module ( say 100 lines)

- Suppose it consists of two nested for
loops that can execute between 1 and 20
each dependent upon input conditions
- If inside the inner loop there are 4 if-else
constructs
How many tests?
How Long would it take?

7
Basis Set Testing
 Basis Set Testing
- Can be used to determine the number of
distinct execution paths through the code
- This can be used to determine the maximum
number of tests needed to ensure that every
statement is executed at least once
 There may be fewer tests
 You can draw a Flow Graph to understand
this process

8
Flow Graphs if
If (condition) { 1
/*statements for if*/ 2
} 1
else {
/* statements for else */ 3

} 4 2 3

Node represents set of

non-branching code
statements 44
4

9
Flow Graphs While/for
While (condition) { 1
statement for while 2
1
} 3

10
Flow Graphs do-while

Do {
/* statements for while*/ 1
} while condition 2
1

11
Flow Graphs switch
Switch (variable) { 1
case 1: /* statement for 1 */; break; 2
case 2 : /* statement for 2*/; break; 3
case 3: /* statement for 3*/; break ; 4
}5
2

1 3 5

12
 Example
order (&a, &b, &c); 1
if (a < (b+c) && c>0) { 2, 3
is_a_triangle = True; 4
*type = class_ord(a, b, c); 5
Notice the if statement has
two conditions joined by
&& and how this is
handled

1 2 3 4 5

13
Note on AND and OR
Condition 1
If we have
- if (condition 1 && condition 2) Condition 2
• We have two nodes one for each condition
• If we have
- if (condition 1|| condition 2) Statement
• We have one node
end
Condition 1&2

statement

end
14
 Flow Graphs - Why do they help
 Flow graphs tell us the maximum number of test we
need to apply to ensure that we have traversed every
independent path in the code
 To do this we calculate the cyclometric complexity, this is
given by
- V(G) = E - N + 2
- where V(G) is the cyclometric complexity of graph G
- E is the number of edges
- N is the number of nodes

15
Cyclometric Complexity
 Number of Edges = 6
 Number of Nodes = 5
 Cyclometric Complexity = 6 - 5 + 2 = 3
 This gives the number of independent
paths through the code
- Shown graphically overleaf

1 2 3 4 5
16
 Distinct Paths
Path 1
a<c+b AND
1 2 3 4 5 c> 0

Path 2

1 a < c+ b AND
2 3 4 5 c <= 0

Path 3
a > c+b
1 2 3 4 5
17
 What this means
 To test this function we need to carry out three tests
- One where the dat supplied has the largest number<less
than the sum of the other two and no data <=0
 e.g a = 7, b = 9, c= 3
- One where the largest side is >= sum of the
 e.g. a = 15, b = 9, c = 5
- One where the largest side is > = sum of the other two and
where the smallest side is <= 0
 e.g. a = -1, b= -2, c=-2 (although all conditions actually fail first case-
this is redundant

18
An Alternative Method of
Computing Cyclometric Complexity
 V(G) = P + 1
- Where P is the number of predicates
- A predicate is a distinct conditional expression
 AND (&&) makes 2, OR (||) makes 1

 For example

- if ( a < ( b+ c) && c>0) {

- if a > ( b+ c) And c >0
gives 2
 Therefore V(G) is 2 + 1 = 3 for the example

- Replace && by || the answer is 2 ( 1+1)

 Flow graph helps because we can see the paths

19
Testing Documentation
 There are two aspects to test documentation
- The Test Plan
 This can start to be developed at the analysis stage and
can further refined during design and implementation
- The Test Log
 This is a log that records the tests applied and the
results of each test
 For a module this may look the table on the following

page

20
Test Log ( for classify)
Expected Actual
Conditions Comment
Result Result
a=3, b=3,
True True Pass
c=4
Problem
a=10, b=2,
False True bad return
c=7
from order
a=-7, b=3,
False False Pass
c=6
21
Testing 2

Top-Down and Bottom Up

testing
Black Box Testing
Organising Testing 22
 Putting it all together
 So far, we have only considered testing individual
modules. How do we test the complete program?
 Using white box testing we can use either a top-down or
bottom up strategy for testing
 Top down

- start at the highest level module (say main) and work

downward
 Bottom up

- start at the lowest level module and work up

23
 Bottom Up Testing
 Select the lowest level modules
- or more generally a set ( or cluster) of modules
- Test these in a test harness ( or driver program)
 A test harness is a program that
 supplies the input for the module under test
 logs the output

- In the simplest case this can be a program that takes input from the
keyboard and prints out results to the screen
- In general tables and files are used for input, the output may be
checked by a program
 possibly from a file

24
Bottom Up Testing The Triangle
Recognition Program main
 Start with lowest level modules type side_a
side_b
is_a_triangle side_c
a classifylarge
b Small
order c type class_ord
x medium
- So test y
• greater greater
•class_ord
•order (which calls the previously tested greater)
•classify (which calls the previously tested class_ord and order)
•main (which calls the previously tested classify)

25
Using Test Harness with greater
 Greater requires two tests
- where parameter 1 is greater than parameter 2 and vice versa
int pa1 = 2, pa2 = 3, pb1 = 3, pb2 = 2;
greater (&pa1 &pa2)
if (pa = = 3 && pa2 = = 2)
printf (“success test 1\n”);
else
printf (“eror in test 1 %d, %d\n”, pa1, pa2);
greater (&pb1, &pb2);
if (pb1 = = 3 && pb = = 2)
printf (“success test 2\n”);
else
printf (“error in test 2 %d, %d\n”, pb1, pb2);

26
Using Test Harness with
class_ord
 The test harness for class_ord is given
 Note that it is simpler to allow the user to
type in the data and write out the result to
the screen
 Note that there are 5 tests applied

27
 Bottom Up Testing Problems
 The test harness need to be written
- They should be as simple as possible
 even test programs can have errors!!!
 The whole system doesn’t appear until the end
- Thus problems due to the program’s structure may only be apparent
later
- If such problems do occur and they involve the rewriting of previously
tested lower level modules they must be re-tested
 This is called regression testing
 This is why keeping a test log is important

- so that the tests can be reapplied

28
Top Down Testing
 This is the converse of Bottom Up Testing
- Start at the top and work down
 So in our triangle example

-Test main
- Test main with classify
- Test main & classify with class_ord
- Test main & classify with order and class_ord
- Test complete program (greater added)

29
Top Down Testing
 Can be carried out Breadth-First
- M1, M2, M3, M4, M5, M6, M7, M8
 Can be carried out Depth First

- M1, M2, M5, M6, M3, M7, M4, M8

M1

M2 M3 M4

M5 M6 M7 M8
30
Top Down Testing
 Difficult to produce stubs for complex lower
level functions
 Difficult to observe test ouput

- Many higher parts of programs have no I/O

 Always have a working version of the system
for demos, etc
 Can be integrated with program development
 Structural problems should be found

31
 Black Box Testing
 Black box testing focuses on the functionality of the
software
- Rather than is structure, as in white box testing
 The set of tests are derived by considering the
Requirement Specification for the software
 In general

- White box testing is performed early in the testing

stage
- Black box testing toward the end

32
 Equivalence Partitioning
 One way of carrying out black box testing is to follow an
approach called Equivalence Partitioning
 Basically, we consider input provided to the program and
break it up into a number of distinct classes ( or sets)
- For example, positive numbers, strings, negative numbers,
etc.
 For a given partition, the program should handle all members
of the class in the same way
 We can also consider output provided by the program and
break this up into a set of classes

33
Equivalence Partitioning (2)
 Once the partitions have been identified,
test cases are taken from each of the
partitions
 For each partition it is good idea to take a
number of values
- The midpoint of the partition
- The boundaries of the partition

34
 Equivalence Partitioning Example 1
 For example, suppose that input to a program can be an integer
constrained in value between - 100 and 100
- We should select a lower boundary value (-100)
- An upper boundary value (100)
- A typical value (2)
 We may also select out of range values
- too small (-101)
- too big (101)
 It is also a good idea to test with zero, as this is sometimes treated
differently
- Therefore test with 0

35
Equivalence Partitioning
Example 2
 A program accepts
- Between 4 and 8 input values
- Each input value is a 5 digit integer that is >1000
 We have two partitions

- The number of inputs

- The value of the inputs
 Following our guidelines we should test

36
 Equivalence Partitioning
Example 2
3 4 7 10 11
No. Of Input
values
Less than 4 4 to 7 More than 10

9999 10000 50000 99999

100000
Input
values
Less than 10000 In Range More than 99999

Test out all combinations of number and values (25 tests)

37
 Equivalence Partitioning - Output
 The output of a program can also be partitioned and
the same rules applied
 For example

- Suppose a program produces betwen 3 and 6 outputs

whose value is in the range 1000-2500
- select input to produce
 3 Values at 2500
 3 Values at 1000

 6 Values at 2500

 6 Values at 1000

38
Testing as a Process
 Testing needs to be organised so that
- The tests are cost efficient
- Provide good test coverage
- Systems are built on previously tested
components
 The testing process is shown on the next
slide

39
 The Testing Process
Test a module’s control structure, uses
white box techniques carried out by
developer

Unit Testing Validation testing, does it meet the

user’s expectations, black box test

Integration
Testing
Verification of
system and overall Validation
program structure Test
Uses black box and
limited white Final operational System Test
box Tests 40
Testing with Customers (1)
 The developers of software will not always foresee
exactly how the customer will use their software
- Therefore testing with the customer is important
 When the software is a custom product for one specific
product a series of Acceptance Tests are undertaken
- where the customer determine whether his/her
requirements are satisfied
 ranges from a ‘test-drive’ to systematic testing procedure

41
 Testing with Customers (2)
 Where software is to be used with many
customers, the developer must test with a
representative sample
 Most developers use alpha and beta testing
 Alpha Test
-Customer is taken developer’s site and asked to use the
software with developer watching and recording problems any
errors
 Beta Test
- Number of customers use the software on their own sites
- Developer is not present
42
- Customer records problems