Professional Documents
Culture Documents
When Should You Stop Testing
When Should You Stop Testing
Topics
Why Do We Test Software? When Should We Stop Testing? Software Coverage Is Code Coverage Enough? Whats Missing? What Else is Missing? The Two Key Problems of Program Testing Properties of an Ideal Test Set Categories of Test Data Adequacy Criteria Functional Coverage Business Usage Coverage Using Metrics as a Guide There are No Silver Bullets
Software Coverage
One concept that could guide us in our decision to stop testing is to consider some form of code coverage There are a few variants of code coverage including - Statement coverage (all executable statements) - Branch coverage (all branches are executed at least once) - Path coverage (all paths are executed at least once) - Loop coverage (all loops are executed) Code coverage that provides a complete cover is a good necessary criterion for enough testing. A complete cover is one where: 1) every instruction in a module or program has been executed at least once 2) Every decision (branch or case statement) has been taken at least once in each possible way
Whats Missing?
Test scenarios not covered: - Various combinations of path execution (basic path execution is not the same as all paths or even the most used business paths) - Combinations of logical decisions in addition to just taking a branch in all possible ways (example: 10 yes-no branches can be covered in two paths, taking all Yes branches in a path and taking all No branches in a path) (all combinations for 10 Yes-No branches would need 2**10 paths)
Functional Coverage
To use all the information about a programs intended use you must test it from a functional perspective - Guided by a formal spec, use cases, user stories or whatever is available to say how the program should behave A minimal necessary condition is to test all functions completely - Complete meaning all expected paths through the function, plus alternative paths - Need to translate paths into input values to drive the paths Testing all functions is not sufficient - Does not cover interface testing (how program works with other system, programs and data sources) - Does not address dependencies between parts of the programs and between various functions