The New Software Testing Standard

Stuart Reid

Testing Solutions Group

London, UK

Abstract In May 2007 ISO formed a working group to develop a new standard
on software testing – a new area for ISO. This initiative is closely-supported by
IEEE and BSI, both of which are providing existing standards as source docu-
ments to the project. The proposed standard, ISO/IEC 29119, comprises four
parts. The first covers ‘concepts and terminology’, the second ‘test process’, the
third ‘test documentation’, and the fourth ‘test techniques’. An extension to the
standard is already under development in the form of a test process assessment
standard. This paper describes progress on the development of ISO/IEC 29119,
which is now due for publication in 2012, and the challenges with creating a ge-
neric testing standard that is applicable to all organizations and all types of project.

1 Introduction

Software testing was an established and integral part of the software development
process well before life cycle models were defined, with references to a separate
software testing activity being made as early as 1954. Today estimates for the pro-
portion of life cycle costs spent on testing vary from about 20% up to 80% for
safety-critical systems (these estimates typically include testing costs across the
full life cycle, and so comprise both static and dynamic testing). Despite its long
history and high costs, testing has traditionally been poorly covered in standards;
this corresponds with similar poor coverage in academic courses and research.
This paper covers a new international software testing standard, the develop-
ment of which started in 2007 and which is now expected to be published in 2012.
Some standards on different aspects of software testing have been published by
bodies such as IEEE and BSI; these have either covered testing in individual life
cycle phases (e.g. BS 7925-2 Software Component Testing (BSI 1998b)) or spe-
cific aspects, such as test documentation (e.g. IEEE 829 Test Documentation
(IEEE 2008)). The new international standard (ISO/IEC 29119) is intended to be
far more inclusive, covering testing processes at various levels, as well as test
documentation, techniques for designing test cases, a testing vocabulary and the
concepts on which the standard is based.

C. Dale and T. Anderson (eds.), Achieving Systems Safety: Proceedings of the Twentieth 237
Safety-Critical Systems Symposium, Bristol, UK, 7-9th February 2012,
DOI 10.1007/978-1-4471-2494-8_17, © Springer-Verlag London Limited 2012
238 Stuart Reid

1.1 A software testing model

The underlying model used as the basis of the new standard is shown in Figure 1,
and comprises four basic entities with the test processes forming the central core
of the model. The test documentation is produced as a result of executing the test
processes, thus the test documentation describes the outputs of the test processes.
The requirement to use techniques to perform the testing (e.g. designing test cases)
is defined as part of the processes, while the actual techniques are defined sepa-
rately. The terminology used by the other parts of this model is defined in the vo-
cabulary.

Fig.1. Underlying model of software testing

The processes covered by this model work at three different layers as shown in
Figure 2. The organizational test process defines those activities required to man-
age the testing across a number of projects and would typically include the devel-
opment and implementation of an organizational test policy and the organizational
test strategy. Test management processes cover those activities that define and im-
plement test plans; these could be at the overall project level or be concerned with
the test management of individual test phases or test types (e.g. usability testing).
The fundamental test processes define the actual testing activities (e.g. test design
and test execution), but could include both static and dynamic testing.

1.2 Why testing standards?

Historically the demand for new standards has often been driven by serious fail-
ures and even major disasters (e.g. boiler standards and steamboat accidents, water
standards and cholera epidemics). Although there are examples of inadequate
software testing and associated serious failures (e.g. Ariane 5), the development of
software testing standards appears to be more a balance between commercialism
and an altruistic drive for increased professionalism in the industry.
Software testing standards provide a number of potential benefits to different
stakeholders. For instance, use of such a standard could act as a form of guarantee
to the consumer of a testing service that the testing is of a certain quality. From