Software testing

ECE 452 Introduction to Software Engineering

Lecture 24: Software Testing

Manish Parashar parashar@ece.rutgers.edu Department of Electrical & Computer Engineering Rutgers University

Testing Testing is is the the process process of of executing executing a a program program with with the the intent intent of of finding finding errors. errors.
Glen Glen Myers Myers

ECE 452 - Introduction to Software Engineering

Limits of software testing

Good testing will find bugs z Good testing is based on requirements
z z

Exhaustive testing
There are 10 14 possible paths! If we execute one test per millisecond, it would take 3.170 years to test this program!!
loop < 20x

BUT: Testing can only prove the presence of bugs - never their absence

ECE 452 - Introduction to Software Engineering

ECE 452 - Introduction to Software Engineering

Testability = how easy can a program be tested

z z

What Testing Shows

errors requirements conformance performance

operability - it operates cleanly observability

x

the results are easy to see x distinct output is generated for each input x incorrect output is easily identified z

controllability
x x

processing can be controlled tests can be automated & reproduced

z z z z

decomposability - software modules can be tested independently simplicity - no complex architecture and logic stability - few changes are requested during testing understandability - program is easy to understand

an indication of quality

ECE 452 - Introduction to Software Engineering

ECE 452 - Introduction to Software Engineering

Who tests the software

Test cases
Describe how to test a system/module/function z Description must identify
z
y system

developer
Understands the system but, will test gently and, is driven by delivery

independent tester
Must learn about the system, but, will attempt to break it and, is driven by quality

state before executing the test to be tested y parameter values for the test y expected outcome of the test
y function

ECE 452 - Introduction to Software Engineering

ECE 452 - Introduction to Software Engineering

Test case examples

z

Test case design

z

system state before executing the test

y

system state before executing the test

y

ResourcePool is not empty

z z

ResourcePool is not empty removeAgent(anAgent) anAgent is in NOT the ResourcePool

z z

function to be tested
y

function to be tested
y

Bugs lurk in corners and congregate at boundaries

Boris Beizer

removeAgent(anAgent) anAgent is in the ResourcePool

z

parameter values for the test

y

parameter values for the test

y

expected outcome of the test

y

expected outcome of the test

y

OBJECTIVE CRITERIA CONSTRAINT

to uncover errors in a complete manner with a minimum of effort and time

ResourcePool = ResourcePool\anAgent

AgentNotFoundException is thrown

ECE 452 - Introduction to Software Engineering

ECE 452 - Introduction to Software Engineering

Adequate set of test cases

z

Black box testing

requirements output input events

Test the interface of the components

y all

public methods

Is one test case per method enough? z How many do we need?

z

ECE 452 - Introduction to Software Engineering

ECE 452 - Introduction to Software Engineering

White box testing

our goal is to ensure that all statements and conditions have been executed at least once...

Why bother with white box testing?

z

Black box testing:

y Requirements

fulfilled

y Interfaces available and working

Question: Why white box testing?

ECE 452 - Introduction to Software Engineering

ECE 452 - Introduction to Software Engineering

Why cover?
logic errors and incorrect assumptions are inversely proportional to a paths execution probability z we often believe that a path is not likely to be executed; in fact, reality is often counter intuitive z typographical errors are random; its likely that untested paths will contain some
z

Exhaustive testing
If-then-else

There are 520 =1014 (approx.) possible paths!

loop < 20x

ECE 452 - Introduction to Software Engineering

ECE 452 - Introduction to Software Engineering

Selective testing
a selected path

Path coverage testing

1) 2) Derive a logical complexity measure Use it to define a basis set of execution paths

First, we compute the cyclomatic complexity: number of simple decisions P +1 (compound predicates have to be split) number of enclosed areas +1

In this case, V (G) = 4 V(G) provides an upper bound of tests that must be executed to guarantee coverage of all program statements

loop < = 20x

ECE 452 - Introduction to Software Engineering

ECE 452 - Introduction to Software Engineering

Cyclomatic complexity
A number of industry studies have indicated that the higher V(G), the higher the probability of errors.

Path coverage set

Cyclomatic complexity defines the number of independent paths in the basis set z Path coverage set = set of paths that will execute all statements and all conditions in a program at least once z Goal: Define test cases for basis set z Path coverage set is not unique
z

modules

V(G) modules in this range are more error prone

ECE 452 - Introduction to Software Engineering

ECE 452 - Introduction to Software Engineering

Path coverage testing

1

Documenting test cases

Name z Number z Description of system state before running the test case z Values for the inputs z Expected outputs z Short description (if needed)
z
ECE 452 - Introduction to Software Engineering

Next, we derive the independent paths: Since V(G) = 4, there are four paths

Path 1:
4 5 3

1,2,3,6,7,8 1,2,3,5,7,8 1,2,4,7,8 1,2,4,7,2,47,8

Path 2:
6

Path 3: Path 4:
7

Finally, we derive test cases to exercise these paths.

ECE 452 - Introduction to Software Engineering

Path coverage testing - remarks

you dont need a flow chart, but the picture will help when you trace program paths z count each simple logical test, compound tests count as 2 or more z path coverage testing should be applied to critical modules
z
ECE 452 - Introduction to Software Engineering

Loops
z z

Cornerstone of every program Loops can lead to non-terminating programs

ECE 452 - Introduction to Software Engineering

Loop testing
simple loop

Loop testing: simple loops

Minimum conditions - simple loops
1. skip the loop entirely 2. only one pass through the loop 3. two passes through the loop 4. m passes through the loop m < n 5. (n-1), n, and (n+1) passes through the loop where n is the maximum number of allowable passes

nested loops

concatenated loops

unstructured loops

ECE 452 - Introduction to Software Engineering

ECE 452 - Introduction to Software Engineering

Nested loops
Just extending simple loop testing: number of tests explodes z Reduce the number of tests:
z
y start

Black box testing

requirements output input events

at the innermost loop; set all other loops to minimum values y conduct simple loop test; add out of range or excluded values y work outwards while keeping inner nested loops to typical values y continue until all loops have been tested

ECE 452 - Introduction to Software Engineering

ECE 452 - Introduction to Software Engineering

Categories of errors in black box testing

incorrect or missing functions interface errors z errors in data structures or external database access z performance errors z initialization and termination errors
z z

Example
abs(integer x) Return x if x >= 0 Return -x if x<0 z The concrete value of x does not matter!

ECE 452 - Introduction to Software Engineering

ECE 452 - Introduction to Software Engineering

Equivalence partitioning
Partitioning is based on input conditions

Equivalence classes (1)

Valid data
user supplied commands responses to system prompts file names computational data physical parameters bounding values initiation values output data formatting commands responses to error messages graphical data (e.g., mouse picks)

mouse picks on menu output format requests responses to prompts

user queries numerical data

command key input

ECE 452 - Introduction to Software Engineering ECE 452 - Introduction to Software Engineering

Equivalence classes (2)

Invalid data
data outside bounds of the program physically impossible data proper value supplied in wrong place

Defining equivalence classes

Input condition is a range: one valid and two invalid classes are defined z Input condition requires specific value: one valid and two invalid classes are defined z Input condition is boolean: one valid and one invalid class are defined
z

ECE 452 - Introduction to Software Engineering

ECE 452 - Introduction to Software Engineering

Boundary value analysis

mouse picks on menu output format requests responses to prompts

Boundary value testing

z z

Range a..b
y test

cases: a, b, just below a, just above b

output domain

user queries numerical data

Number of values:
y test

cases: max, min, just below min, just above max

z z

Output bounds should be checked Boundaries of data structures shall be checked (e.g. arrays)
ECE 452 - Introduction to Software Engineering

command key input

ECE 452 - Introduction to Software Engineering

Comparison testing
z z z

Automating software testing

Manual software testing is time consuming z Software testing has to be repeated after every change (regression testing) z Write test drivers that can run automatically and produce a test report
z

Safety critical systems Develop several implementations for a specification Run tests with all and compare results if not identical: investigate in detail Problems:
y expensive y errors y all

in specification implementations return the same but incorrect result

ECE 452 - Introduction to Software Engineering

ECE 452 - Introduction to Software Engineering