SOFTWARE TESTING

(ITS 64704)
Testing Throughout the Software Life Cycle
(Part 2)
Topics

Testing in Software Development Models

Testing Levels

Testing of New Product Version

Overview of Test Types

Testing Levels

Drivers and stubs

Component Test
Test first

Top down

Test Levels Bottom up

Integration Test
Ad-hoc
System Test
Big bang
Acceptance Test
Testing Levels

Drivers and stubs

Component Test
Test first

Top down

Test Levels Bottom up

Integration Test
Ad-hoc
System Test
Big bang
Acceptance Test
 System Testing

• After integration testing, system testing (3rd level of testing) may be required
before the system is delivered
• System testing verifies if specific requirements are met.
• Reasons for system testing:

Discrepancy of viewpoints

• Low level testing (unit and integration) are tested against technical specification, mostly from
developer’s viewpoint
• System testing examines the system from customer’s perspective.

Discover error at higher level

• Some functions and system properties result from interaction of system components
• Only observable and testable at the level of system testing
System Testing: Test Object

With completed integration testing, the final assembled

software system is available

System testing is concerned with the behaviour of the

whole system
System Testing: Test Harness

 The test environment should correspond as much as possible to the final

production environment
 No stubs and drivers used
 Actual hardware and software products should be installed in the test
environment (hardware equipment, system software, driver software, network,
external system, etc)
System Testing: Test Harness

 To save cost, time and effort, the system testing is often performed in the
customer’s production environment instead of a separate test environment.
 This needs to be avoided due to:

In system testing, failures will occur.

• This brings the risk that the customer’s production environment will be affected.
• Expensive failures and loss of data in the customer’s productive system can be the result.

The testers have little or no control over the parameters and configurations of the production
environment.
• Through the simultaneous operation of other customer systems, the test criteria might be
subtly modified.

The system testing that has already been conducted is hard or even impossible to reproduce.
 System testing: Test Objectives

Two classes of requirements are considered separately:

Functional requirements
• Specify the behavior which the system or system parts have to perform.
• They describe “what” the system(part) has to perform. Their implementation
is required in order for the system to be suitable for use.
• Characteristics of the quality attribute “functional stability” according to [ISO
25010]
Non-functional requirements
• Describe “how” well certain attributes of the system(part) behavior should
be performed.
• Quality attributes according to [ISO 25010] such as : reliability, performance
efficiency, usability, etc. Indirectly, maintainability and portability may also
influence the customer‘s satisfaction.
System testing: Test Strategy

Functional
Requirement
System testing: Test Strategy

Testing based on requirements

• The approved requirements specification document is

used as test basis.
• The system specification will also be verified by a
review.
• For each requirement (functional and non-functional)
test cases are derived and recorded in the system
specification.
System testing: Test Strategy

Business process-based testing

• Used if a software system has the purpose of automating or

supporting the business process of a customer.
• A business process analysis (mostly created as part of the
requirements analysis) shows which business processes are
relevant, how often they are used and in which context they
appear (persons, companies, external systems etc.)
• Subsequently, test scenarios (sequences of test cases) are set up
on the basis of this analysis that represent typical business
situations.
 System testing: Test Strategy

Use Case testing

• System test cases can be derived which describe how the user handles the
system and which actions are typically carried out.
• Different user groups each possess their own user profiles. Typical
interactions or use cases with a realistic frequency can be identified for these
groups.
• From these typical interactions, test scenarios can be derived.
• On the basis of the frequency that various actions are performed in using the
software, the tester determines how important the corresponding test
scenario is and with which priority it therefore has to be included in the test
plan.
System testing: Test Strategy

Non-
Functional
Requirement
System testing: Test Strategy

Load Testing

• Measurements of the system behavior in relation to increasing system load (e.g.

Number of parallel users, number of transactions).

Performance Testing

• Measurements of processing speed and response time for specific use cases, normally
in relation to increasing load.

Volume testing

• Observation of system behavior in relation to the amount of data transmitted (e.g.

processing large volumes of data).

Stress testing

• Observation of system behavior when overloaded

System testing: Test Strategy

Testing (data) security

• Test of access authorisations to the system or data.

Testing reliability

• During nonstop operation (e.g. amount of failures per hour with given user
profile).

Testing robustness

• Against incorrect operation, programming errors, hardware failure etc.

• Tests also include system recovery handling.

Testing compliance/conversion of data

• Verifying the compatibility of existing systems. Import/export of database etc.

System testing: Test Strategy

Testing different configurations

• of the system, e.g., different versions of operating systems, language, hardware

platform etc.

Testing usability

• Verifying the suitability of operation, understandability, and of system output etc.,

in relation to the needs of the user.

Testing documentation

• Testing conformity with the system behavior (e.g. user manual)

Testing portability/maintainability

• Understandability and current content of development documents, modular system

structure etc.
System testing: Test Strategy
(Quality Characteristics)
 System Testing: Problems

Indistinct customer If there are no requirements, any system behavior is initially acceptable or cannot
requirements be assessed.

The user or customer will obviously have a certain idea of what they expect of
“their” software system. So requirements do exist.

However, these are not to be found in written form but only exist “in the minds”
of some persons that are involved in the project.

The testers then have the challenging task to retrospectively gather all this
information about the desired expected behavior.
 System Testing: Problems

Neglected If the testers identify the original requirements, they will realize that in the minds
decisions of the various persons there are very different opinions and expectations
regarding one single matter.

No wonder, because during the project it has been neglected to record the
requirements in a written form, to agree upon and approve them.

Therefore the system test not only has to gather the requirements, but also needs
to seek clarification and obtain decisions which have been omitted for many
months. This needs to be done when it is actually too late.

This gathering of information is very time and cost intensive. Test and acceptance
of the system are delayed.
 System Testing: Problems

Projects fail If requirements are not documented, the developers are missing clear objectives.
The probability that the implemented system fulfills the implicit customer
requirements is therefore extremely low.

Nobody can sincerely hope that even a partially suitable system can be produced
under these project conditions.

In these kinds of projects, the system test can often only “officially” confirm the
failure of the project.

*
Testing Levels

Drivers and stubs

Component Test
Test first

Top down

Bottom up
Integration Test
Test Levels Ad-hoc
System Test
Big bang

Regulatory
acceptance test

Contractual
acceptance test
Acceptance Test
User acceptance

Operational test
Acceptance Test: Definition

• Previous test levels are performed by the producer or the designing project
group, before the software is passed to the customers/users.

• Before the software is deployed, (especially if an individual software specified

for customer has been designed), the final test level performed is the
acceptance testing.

• The main focus here is the view and opinion of the client/user. Finding defects
is not the main focus in acceptance testing. The goal is to establish confidence
in the product and assess its fitness for the intended use.

• Acceptance testing is a special form of system testing which is often

performed at the customer‘s site.
Acceptance Test: Types

1) Contract acceptance testing

• performed explicitly against a contract's acceptance criteria.

• The test criteria are therefore the acceptance criteria recorded in the designed
contract that should be clearly defined and unambiguous.
• acceptance environment should correspond as much as possibe to the future
production environment

2) Regulation acceptance testing

• Performed against any regulations and standards that the system must adhere to.
• e.g Quality models, ISO standards, etc
Acceptance Test: Types

3) Operational acceptance testing

• mostly to performed by the system administrators of the customer. Test performed

include:
• Testing of backup/restore procedures
• Disaster recovery
• User management
• Maintenance tasks
• Data load and migration tasks
• Periodic checks of security vulnerabilities.

4) User acceptance testing

• Testing by means of representative customers

• Must not replace the system testing
• Alpha testing is performed at the developer‘s site
• Beta testing (also called field-testing) is performed at the customer‘s site
Topics

Testing in Software Development Models

Testing Levels

Testing of New Product Version

Overview of Test Types

 Test of new product version
(Maintenance testing)

• Until now, it has been assumed that a software development project is

completed after successful acceptance testing and delivery of the new
product. The reality is different.

• The initial delivery a software product is merely at the beginning of its life
cycle. Once deployed, the software is often in service for years or even
decades. The software product, its environment, its data and its configuration
will be corrected, modified and extended many times during this period.

• Each time a new version of the original product is developed, it needs to be

tested.
 Software Maintenance

• In contrast to »classic« industrial products, software

maintenance is not about keeping up the usability by regular
maintenance or reparing damages e.g., from wear. Software does
not wear down.

• We are talking about software maintenance when a product is

adapted to changing requirements or when failures are removed
that had already been in the system.
Software Maintenance
 Modification

• Planned enhancement changes

• Apart from maintenance work caused by defects or failures there is also
modification and extension work that the project management has
planned from the beginning.
• This is part of the normal product enhancements.
• This includes, for example,
 Adjustments due to a planned modification of neighboring system.
 Implementation of a planned functionality that could not be
delivered at time of initial deployment due to lack of time.
 Extensions required because of market changes.

• A software project is therefore in no way completed with the delivery of the

first product version.
 Modification: Types of

Changes of
Defect fixing Quality improvement
environment
• Defect from • Updating operating • Improving factors
operating system for quality, e.g.,
• Emergency changes • Updating database maintainability,
(“Hot Fixes“) management system performance,
• Upgrades of usability without
commercial change of functional
software amount
• Patches of external
components
 Maintenance Testing: Migration & Retirement

Maintenance testing for retirement of

Maintenance testing for migration
the system
• Includes operational tests of the new • Includes testing of data migration or
(software/hardware) environment as archiving if long data retention periods
well as of the changed software are required.
• Includes conversion testing
• of the software that is used for the
conversion of the data (e.g. data
from another application will be
migrated into the system being
maintained)
• the (new) data that result from the
conversion
 Regression Testing

• Through maintenance and also by enhancements, some parts of the existing

software are changed or new software parts are added.
• In both cases, the modified software has to be tested again. This is called
regression testing.

DEFINITION:
Regression testing is the repeated testing of an already tested program, after
modification.

OBJECTIVE
• to discover any defects introduced or uncovered as a result of the changes.
 Extent of regression testing

1. Repeating all tests that detected the fixed defect (re-testing of defects)

2.Testing all program statements that have been fixed or modified (testing
modified functionality)

3.Testing all program parts that have been added (testing new functionality)

4.The whole system (complete regression testing)

• Both, the pure re-testing of defects (1) as well as testing only the modified
program parts (2 and 3) are not enough.
• Simple local modifications can have unexpected side effects on other (also
distant) parts of the software system.
 Complete regression testing

• In addition to testing fixed defects and added functions, all existing test cases
actually need to be repeated.

• Only then would the test have the same value as the same test that has been
performed in the original version of the program.

• Such complete regression testing would also have to be performed when the
system environment has been changed, since this can potentially have an
effect on each part of the system.
Topics

Testing in Software Development Models

Testing Levels

Testing of New Product Version

Overview of Test Types

 Basic test types

Non-functional
Functional testing
testing

Structure-based Change-oriented
testing testing

All test types can be implemented on these test levels:

Component System Integration Acceptance

testing testing testing testing
Basic test types

Functional testing

• Functionality describes, "what" the system should perform.

• Functionality is delivered from a system, a sub system, or from
one component.
• Test basis are requirements specification, use-cases, or
functional specification.
• Functional testing proves the external visible behaviour of the
software (Black-box test design approach)
• Application of specification-oriented test design approach, to
derive exit criteria (also known as test completion criteria) and
test cases from the functionality of the software or the system.
Basic test types

Non Functional testing

• Non-functional tests prove on the basis of software and system

attributes, "how good" the system works.
• Non-functional tests are e.g.:
• Performance testing - Suitability testing
• Load testing - Maintainability testing
• Stress testing - Reliability testing
• Portability test
• Can be implemented on all test levels.
• Non-functional testing mostly proves the external visible behaviour of
the software
• The basis are quality models (e.g. ISO 9126, ISO 25010).
Basic test types

Structure based testing

• Structure-based testing (White-box test design approach) is based on

the internal structure of a component or the architecture of the
software or the system.
• Basics are e.g.
• the control flow within components,
• the calling-hierarchy of procedures or structured menus or
• the structure of abstract models of the software (finite state machine)
• The objective is to cover all elements of the considered structure via
testing.
• Suitable and sufficient test cases need to be designed.
• Structure oriented testing are applied in component and integration
testing, or as an addition in higher test levels (e.g. to cover menu
structures).
 Basic test types
Change oriented testing
• Testing in relation to modifications (Re-testing and regression testing)
Re-testing
• If a failure was detected and corrected, a re-
test must be conducted to ensure that the
failure was successfully eliminated
• Re-testing must be done several times and
must be capable of repetition.
Regression testing
• When tested programs (or program parts)
were modified, it needs to be shown that no
new failures were built in or failures which
were undetected have become visible
• Regression tests must be executed if the
software environment was changed.
• Comprises functional, non-functional and also
structure oriented testing (on all test levels).
• Re-testing must be done several times and
must be capable of repetition.
Comparison of test levels

*
THANK YOU