Lecture 4

Software Testing Fundamentals

Prof. Mostafa Abdel Aziem Mostafa
 Contents
3.1 General Testing Principles
(The Seven Key Principles)
3.2 Fundamental Test Process
3.3 Psychology of Testing
3.1 General Testing Principles
(The Seven Key Principles)

1.Testing shows presence of Defects

2.Exhaustive Testing is Impossible!
3.Early Testing
4.Defect Clustering
5.The Pesticide Paradox
6.Testing is Context Dependent
7.Absence of Errors Fallacy
 General Testing Principles
The Seven Key Principles
1.Testing shows the presence of Defects
( it is not a ghost. It is a fact )
• We test to find Faults (as known as Defects)
• As we find more defects, the probability of
undiscovered defects remaining in a system
reduces ( decreasing nature).
• However Testing cannot prove that there
are no defects present
 Why Testing is necessary
Testing Pearls of Wisdom
• “The probability of the existence of more errors in a section of a
program is proportional to the number of errors already found in
that program”
• “Do not plan a test effort under the tacit assumption that no errors
will be found”
• “A good test is one that has a high probability of detecting an as
yet undiscovered error”
• “A successful test is one that detects an as-yet undiscovered error”

Myers 2004
 General Testing Principles
The Seven Key Principles

2.Exhaustive Testing is Impossible!

• We have learned that we cannot test

everything (i.e. all combinations of inputs
and pre-conditions).

• That is we must Prioritise our testing effort

using a Risk Based Approach.
 Why Testing is Necessary
Why don’t we test everything ?
System has 20 screens
Average 4 menus / screen
Average 3 options / menu
Average of 10 fields / screen
2 types of input per field
Around 100 possible values

Approximate total for exhaustive testing

20 x 4 x 3 x 10 x 2 x 100 = 480,000 tests
Test length = 1 sec then test duration = 17.7 days
Test length = 10 sec then test duration = 34 weeks
Test length = 1 min then test duration = 4 years
Test length = 10 mins then test duration = 40 years!

It is not a matter of time. But,time is money ( salary is taken by hour. So

second is valuable for software houses)
 Urgency of Equilibrium

Number of Cost of
• If you test too little, the Missed Bugs Testing
probability of software
failure increases Q
u
a Testing
Under Equilibrium
• If you try to test too n
t Testing Over
much, the development i Testing
t
cost becomes y
unaffordable

• So, we need to conduct Amount of Testing

some sort of equilibrium
 General Testing Principles
The Seven Key Principles

3.Early testing
• Testing activities should start as early
as possible in the development life
cycle

• These activities should be focused on

defined objectives – outlined in the
Test Strategy

• Remember from our Definition of

Testing, that Testing doesn’t start
once the code has been written!
 Cost of Fault Correction

100
90
80
70
Relative
60
Multiples 50
Cost
40
30
20
10
0
Reqs Des Code Unit Accept Use
 General Testing Principles
The Seven Key Principles

4. Defect Clustering
• Defects are not evenly distributed in a system
• They are ‘clustered’
• In other words, most defects found during testing are usually
confined to a small number of modules ( 80% of uncovered errors
focused in 20% modules of the whole application) “Pareto law”

• Similarly, most operational failures of a system are usually confined

to a small number of modules
• An important consideration in test prioritisation!
 General Testing Principles
The Seven Key Principles

5. The Pesticide Paradox ( ‫ارقة‬/‫لحشرية مف‬//‫تا‬/‫لمبيدا‬//‫)ا‬

• Testing identifies bugs, and programmers respond to fix

them
• As bugs are eliminated by the programmers, the software
improves
• As software improves the effectiveness of previous tests
erodes (‫آكل‬///‫) ت‬
• Therefore we must learn, create and use new tests based on
new techniques to catch new bugs ( i.e. It is not a matter of
repetition. It is a matter of learning and improving)
• N.B It's called the "pesticide paradox" after the agricultural phenomenon, where bugs such as the boll weevil build up tolerance to
pesticides, leaving you with the choice of ever-more powerful pesticides followed by ever-more powerful bugs or an altogether
different approach.’ – Beizer 1995
 General Testing Principles
The Seven Key Principles

6. Testing is Context (background)Dependent

• Testing is done differently in different contexts

• For example, safety-critical software is tested differently from an e-commerce
site
• Whilst, Testing can be 50% of development costs, in NASA's Apollo program (it
was 80% testing)
• 3 to 10 failures per thousand lines of code (KLOC) typical for commercial
software
• 1 to 3 failures per KLOC typical for industrial software
• 0.01 failures per KLOC for NASA Shuttle code!
• Also different industries impose different testing standards
 General Testing Principles
The Seven Key Principles
7.Absence of Errors Fallacy ( ‫لطة‬//‫ا‬/‫)مغ‬

• If we build a system and, in doing so, find and fix

defects ....
It doesn’t make it a good system
• Even after defects have been resolved, it may still be
unusable and/or does not fulfil the users’ needs and
expectations

• Visit the following site:

http://www.guru99.com/software-testing-seven-principles.h
tml
3.2 Fundamental Test Process
The five stages of the fundamental test process

• Test Planning and Control

• Test Analysis and Design

• Test Implementation and Execution

• Evaluating Exit Criteria and Reporting

• Test Closure Activities

 Fundamental Test Process
Fundamental Test Process

• The process always starts with planning and

ends with test closure activities
• Each phase may have to be executed a
number of times in order to fulfil exit or
completion criteria
• Although logically sequential, the activities in
the process may overlap or take place
concurrently
 Fundamental Test Process
1.Test Planning and Control
Test Planning
• Specifies how the test strategy and project
Test Plan
A document describing the scope, approach, resources and schedule of intended test activities

apply to the software under test

• Principally:
– verify the mission,
– define the Test objectives
– Specify the Test Activities required to meet the
mission and objectives
 Fundamental Test Process
Test Planning and Control
Test Planning (continued)

• Major Tasks are :-

– Identify the objectives of testing
– Determine Scope
– Determine the Test Approach
– Determine the required test resources
– Implement the test policy and/or the test strategy
– Schedule test analysis and design tasks
– Schedule test implementation, execution and evaluation
– Determine the Exit Criteria

• More on Test Planning in Test Management section

 Fundamental Test Process
Test Planning and Control
Test Control
• The ongoing activity of comparing actual progress against the plan
• Reporting status, including deviations from the plan
• Taking actions necessary to meet the mission and objectives of the project
• Test Planning takes into account the feedback from monitoring and control
activities.

• Major Tasks are :-

– measure and analyse results
– Monitor and document progress, test coverage and exit criteria
– initiate corrective actions
– make decisions

• More on Test Control in Test Management section

 Fundamental Test Process
2. Test Analysis and Design
• General testing objectives are transformed into tangible Test Conditions (An item or event of a
component or system that could be verified by one or more test cases, e.g. a function, transaction, feature, quality attribute, or structural element )

and Test Cases (A set of input values, execution preconditions, expected results and execution
Post-conditions, developed for a particular objective or test condition, such as to exercise a particular program path or to verify compliance with a
specific requirement).

• Tests should be designed using the test design techniques selected in the test planning
activity

• Major tasks are:

– Review the Test Basis - in doing so evaluate testability of Test Basis and Test Object(s)
– From Analysis of Test Basis and Test Items, identify and prioritize Test Conditions and associated
Test Data
– Test Conditions and associated Test Data are documented in a Test Design Specification
– Design and prioritize the Test Cases
– Identify Test Data required to support Test Cases
– Design the test environment set-up
– Identify any required infrastructure and tools
 Fundamental Test Process
Test Analysis and Design

• Each Test Case is specified in terms of :-

– its objective
– the initial state of the system
– the input
– the expected result.
 Fundamental Test Process
Testing Pearl of Wisdom

• “The act of designing tests is one of the most effective

error prevention mechanisms known
... The thought process that must take place to create
useful tests can discover and eliminate problems at every
stage of development.“

Beizer 1983
 Fundamental Test Process
3. Test Implementation and Execution

• Test Cases are combined in a particular order to form Test Procedures

Specifications (a.k.a Test Scripts) A document specifying a sequence of actions for the execution of a test. Also
known as test script or manual test script
• Test Data is created
• The test environment is created
• Major tasks are:

Create Test Scripts and Data

– Develop, implement and prioritise Test Cases
– Create the Test Scripts
– Create test data
– Preparing test harnesses ( dedicated resources)
– Write automated test scripts
– Create test suites from the test cases for efficient test execution
Check the Environment
– Verify that the test environment has been set up correctly
 Fundamental Test Process
Test Implementation and Execution

• Major tasks (continued):

Execute the Tests

– Execute the Test Cases (manually or through tools)
according to the planned sequence.
– Log ( record ) the outcome of test execution
– Test execution records should uniquely identify the
versions of the software under test, test tools and
Testware
– It should be possible to establish that all testing has
been carried out by reference to the test records.
 Fundamental Test Process
Test Implementation and Execution

• Major tasks (continued):

Check the Results
– Compare actual results with expected results
– Report discrepancies as Incidents
– Analyse Incidents to establish Root cause
– Repeat, as necessary, test activities as result of
action taken for each discrepancy
– The test coverage levels achieved for those
measures specified as test completion criteria
should be recorded.
 Fundamental Test Process
Testing Pearl of Wisdom

• “carefully inspect the results of each test”

Myers - 2004

Ref: Myers, The Art of Software Testing, J Wiley and Sons, 1979
 Fundamental Test Process
4. Evaluating Exit Criteria and Reporting
• Exit criteria ensures that the testing of the application is
completed and ready.
• Test execution is assessed against the objectives defined in
Test Planning
• This should be done for each Test Level (i.e. test stage)
A group of test activities that are organized and managed together.

• Major tasks are:

– Check test logs against the exit criteria specified in Test Planning
– If the exit criteria has not been met
• Assess if more tests are needed
• Assess which test activities may need to be repeated
– Assess if the exit criteria specified should be changed
– Produce a test summary report for stakeholders review
 Fundamental Test Process
How to measure exit criteria ?
1. All the planned requirements must be met

2. All the high Priority bugs should be closed

3. All the test cases should be executed

4. If the scheduled time out is arrived

5. Test manager must sign off the release

Note: All these parameters can be met by percentages( not 100 %)

 Fundamental Test Process
5 Phases of the Fundamental Test Process

Fix test design and repeat

Fix component or test cases/scripts
and repeat

Test Planning Test Analysis Test Implementation Evaluating Exit Criteria Test Closure
and Control and Design and Execution and Reporting Activities

Fix test design and repeat

Fix component test plan and repeat

 3.3 Psychology of Testing
The Testing approach
• Historically testing was viewed as showing the system meets its
requirements

• This has evolved to a stage where testing is performed with the

primary aim of finding faults rather than proving correctness. It is
perceived as a destructive ( negative ) process

• Seeking to find failures (the right approach) can be viewed as

criticism of the product and/or its author

• But looking for failures as constructive! ( useful)

– Time can be saved
– Risks reduced
– Costs reduced
– Skills improved
 Psychology of Testing
The Testing approach

• It is important that the Objectives of testing are clearly

understood as humans will moderate their behaviour
accordingly (however sub-consciously‫وعي‬///‫)ب‬:-
– “Iftesting is showing the system meets its requirements
then I will just produce tests that show this.”
– “If testing is aimed at finding faults then I will be
measured on this so I will put effort into designing tests
that are more likely to find faults.”
• The Testing approach is different from a
Developer’s
 Psychology of Testing
Testing Pearl of Wisdom

• “Testing is an extremely creative and intellectually (‫كري‬///‫ )ف‬challenging

task”

• “Tests must be written for invalid and unexpected, as well as valid and
expected, input conditions”

Myers - 1979
 Psychology of Testing
Traits of Good Testers
• A Tester needs:

– good communication skills

– good observation skills
– people handling skills
– Curiosity ( testing activities interest )
– patience
– reliability
– Thoroughness ( carefulness)
– an inquisitive nature ‫ي‬( ‫ضول‬///‫)ف‬
– attention to detail
– creativity in terms of identifying likely faults
– Experience

• However as with most other disciplines an effective test team

will need a mix of skills so it is difficult to generalise
 Psychology of Testing
Developer vs Tester Relationship

• The relationship between a Developer and a Tester is not

normally an easy one because:-
– testers point out problems with software
– developers like to think their software is perfect
– testers are perceived as delaying the project by
finding faults in the system
– when the development slips, testers normally have to
work long hours to test the product, which in turn can
cause anger
• It is important that they work together
• It is also important that they have mutual respect for
each other.
• Collaboration is the right approach – we work to a
common goal!
• Communicate findings objectively, not subjectively
 Psychology of Testing
Independent testing ( (‫الختبار‬//‫حيادية ا‬
• The right approach could enable Developers to test the code

• However, passing this responsibility to trained and professional

testing resources has many benefits (such as higher defect find rates)

• Authors tend to bring across assumptions they have made when

developing the software. They are less likely to write tests to show
faults in their own software (human nature)

• With testing performed by independent testers, testing effort is

focused and not compromised by development effort and bias

• It is generally believed that objective independent testing is more

effective
 Psychology of Testing
Independent testing

• There are several levels of Independence

(from Low to High):
– Tests designed by the person(s) who wrote the software under test
– Tests designed by another person(s) (e.g. from the development
team).
– Tests designed by a person(s) from a different organizational group
(e.g. an independent test team).
– Tests designed by a person(s) from a different organization or
company (e.g. outsourcing to an in-house or external test specialist
organisation)