SOFTWARE PROCESS-Improvement and Practice, Vol.

2, 281-289 (1996)

CMM and I S 0 9001

Martyn A. Ould
Quality 6 Technical Director, Praxis plc, 20 Manvers Street, Bath
BAl l P X , UK

Practice Section

In their own ways, both I S 0 9001 and the Capability Maturity Model (CMM) have provided
guidance and urgency to the improvement of software processes. However, there is a point
where each calls for sfatistical analysis of the performance of the processes in use, treating
software development as a manufacturing activity. This paper argues that this is an unhelpful
demand that should be reconsidered by the writers of standards. 0 1996 by John Wiley &
Sons Ltd and Gauthier-Villars
Softw. Process Improve. Pract., 2, 281-289 (1996)

No. of Figures: 0 No. of Tables: 0 No. of References: 8

KEY WORDS: IS0 9001;CMM; SPC; software process improvement

INTRODUCTION IS0 9001 and the CMM brings combined benefits.

A great deal - perhaps too much - effort is
expended by organizations trying to answer the
question ’which should we use: IS0 9001 or the
SEI CMM?’. One might as well ask ’which should
we use: code inspections or good programming
practices?’. We have had a Quality Management
System registered under IS0 9001 (or its precursor
British Standard 5750 Part 1) since 1986, and we
have a TickIT+ certificate. We are looking at the
relevance of the CMM (Paulk et al. 1993)framework
to further improving our software engineering
practices. Drawing particularly on our own experi-
ence, I argue in this paper that intelligent use of
~ ~ ~~~ ~~ ~~
Correspondence to: M.A. O d d , Quality & Technical Director,
Praxis plc, 20 Manvers Street, Bath, BAl IPX, UK.
*Origin-ally a UK certification scheme speafic to software
development that amplified I S 0 9001 and included IS0 9OOO-
3 ( I S 0 9000-3: 1991 - Guide to the application of I S 0 9001 to the
dtwelopment, and maintolance of software), TickIT is
increasingly being adopted internationally. TickIT-registered
auditors are required to be well versed in software development.
ccc 1077-48661%1020281-1 7.50
0 1996 by John Wiley & Sons, Ltd. and Gauthier-Villars
I also argue that the tendency of both schemes
to push towards a statistical view of software
development wdl be a fruitless one which should
be replaced with approaches appropriate to the
intellectual nature of software development as a
collaborative design activity, and that responsive-
ness to change should be promoted at least as
strongly as the ability to hone a stable process.
USING I S 0 9001 AND CMM TOGETHER
The spirit and the letter of I S 0 9001
There are no guarantees in this world. Buy the car
with the best safety record on the market and you
can still drive it like an idiot. A bad driver can
pass a driving test.
ISO9001 is no different. An Iso 9001 certificated
Quality Management System (QMS) does not
guarantee the quality of the software developed
under it. Bad software engineers can operate a
good QMS. In our consultancy work we are used
 Practice Section
to finding organizations for whom their QMS is
an irritating appendage.
All this is obvious, but it tells us that if we
aspire to an IS0 9001 certificate for our QMS and
we want to get some real benefit from the achievement
it represents then we must live the spirit of the
standard as much as the wording.
In the rest of this paper, you can assume that
when I talk about an IS0 9001 QMS, I mean one
that is used in the spirit in which I S 0 9001 is
intended. In particular, that means not treating it
as something that is simply tacked onto daily life,
or as a bureaucratic ordeal. The sort of QMS I am
thinking about is a QMS that is effective in
promoting the business's effectiveness.
The QMS as repository
Some years ago I visited a large organization that
was well into a major Total Quality Management
(TQM) programme. I had been invited to spend a
day addressing about 30 of their IT delivery
managers on the topic of I S 0 9001 QMSs. I took
on the assignment with some trepidation since, at
the time, I felt rather apologetic about our rather
traditional QMS when in the company of people
in the midst of exciting TQM programmes, even
though I had real evidence that 'quality' had (and
still has) a real meaning in the company and with
our staff. I S 0 9001 has this reputation of being a
recipe for bureaucracy and the stifling of inno-
vation - allegedly quite the reverse of TQM
with its emphasis on innovation. Nevertheless, I
prepared my one-day seminar.
At the end of the day, I was interested by the
comments from the people who came. The gist of
their feedback was that a QMS was just the thing
they needed in their TQM programme. So far they
had only heard a great deal of exhortations to
measure, to control, to improve; they had received
training in team building, process analysis, cause-
effect graphing and all the paraphernalia of TQM,
but they did not know what to do next (or even
first). Some people had indeed started measuring
things and putting graphs up at their desks, but
when it came down to it, they had no foundations
on which to build.
They recognized that the real point of a QMS
was that it could act as the focus they lacked; a
repository for things they agreed as a group were
essential to achieving quality in their delivery of
S o b . Process Improve. Pract., Vol. 2, 281-289 (19%)
282
M. A. Ould
IT systems to the organization, something to bring
cohesion and hence lasting benefit.
This 'repository' aspect of a QMS is the first that
I want to draw attention to. Our QMS is the major
repository of our process, at least in so far as we
want to capture that process in terms of process
descriptions, as opposed to, say, training courses
or in-house or purchased tools. It is also 'our'
repository; it was originally developed by our
engineers and it continues to be maintained by
our engineers, with an internal quality assurance
team having editorial responsibility for it.
Building a QMS guided by the CMM
Interestingly, when we look back at the order in
which we developed our QMS we find it ran
roughly as follows:
0 have a standard format for documents and a
standard approach to version control;
have a standard way of peer reviewing anything
and of closing defects, and start applying it
to everything;
institute a project-oriented approach under
which all work is managed;
have a standard approach to the planning and
reporting of projects;
0 start to define common technical processes and
put in place standards for the products of
those processes;
0 concentrate on managing requirements;
...
Does this look familiar? It bears a very close
resemblance to the order in which one ought to
be doing things to lift oneself from level 1 in the
CMM to (round about) level 3. It was about getting
projects under basic control before worrying about
technical issues. I once had discussions with a
client who wanted to introduce quality-oriented
practices into his IS department, was aware of the
requirements of I s 0 9001, and had been reading
a lot about measurement. Basically his brain was
spinning from all the ideas that had been presented
to him. I led him through CMM levels 2 and 3
and showed him how we had, without having the
CMM to hand (back in 1983-1986), gone through
that same foundation laying and then up through
the building floor by floor. In particular, I could
demonstrate to him why his uncertainty about
measurement in an organization with an ad hoc
0 1996 by John Wiley L Sons, Ltd. and Gauthier-ViIlars
& Practice Section
process was making his brain spin; it was premature
to think about measurement without significant
foundations in place.
What we see is that the CMM gives us a good
plan of action for achieving IS0 9001 certification
starting from an ad hoc process, and IS0 9001
provides us with the notion of a Quality Manage-
ment System, which reinforces the notion of the
process asset repository and gives us a framework
for building our process capability. The two notions
neatly support each other (I rather wish we had
had the progression defined by the CMM when
we started our QMS development). Establishing
good software engineering practice by building
through levels 2 and 3 will generate a QMS
(whether or not we choose to call it that) and
should, I assert, lead to an I s 0 9001 certificate for
that QMS (pace Mark Paulk’s (1995) detailed
comparison). There are some pluses and minuses
in each scheme but my point here is that the CMM
and IS0 9001 are not alternatives from which one
must be chosen.
The effective QMS
IS0 9001‘s requirement that there should be an
auditable trail showing that the plan-do-check-act
cycle has been carried out is often turned inside
out and portrayed as a demand for unnecessary
bureaucracy. Those who run an I s 0 9001 QMS
properly - in other words in a way that is fit for
their purpose - know that the only records you
keep are those you need anyway for good quality
management. It is hard to convince people who
have seen or experienced an ineffective QMS that
bureaucracy is not a must-have of IS0 9001. In
our own QMS, because we have a wide variety of
work for different clients on different architectures
for different sorts of systems, we devolve engineer-
ing decisions about process to individual develop-
ment projects and this extends to the choice of
quality controls and records for the project. (This
is not a carte blanche to say ’we are not bothering’
since the quality management plan for the project
must be produced and must be approved by
management as appropriate for the job in hand.)
The rule for project managers is then simple: ‘if
you are doing something that you think does not
add value to the engineering of quality into this
system then you have only yourself to blame’. I s 0
8 1996 by JohnWiley & Sons, Ltd. and Gauthier-Villan
CMM and I s 0 9001
9001 does not require us to do things that do not
make sense for our business.
BEYOND LEVEL 3: GREAT
EXPECTATIONS
So, we can see a helpful degree of convergence of
the two schemes at the point which I might
characterize by I S 0 9001 certification. There are
differences of emphasis in the two schemes, but
these are of no great import to us here. Neither
says ’what is not required is forbidden’.
Beyond this point of rough convergence things
get harder.
Measurement in I S 0 9001
IS0 9001 leaves some openings in two areas. First,
it calls for ‘preventive action‘ which will include
ways of detecting, analysing and eliminating
’potential causes of non-conformities’. I s 0 9000-3
does not address this topic, based as it was on the
earlier version of Is0 9001 which did not deal
with it specifically. So we are left rather in the air
currently. Secondly, I S 0 9001 requires the supplier
to ‘identify the need for statistical techniques
required for establishing, controlling and venfying
process capability and product characteristics’ and
then to have ’procedures to implement and control
the application’ of those techniques. ‘Process capa-
bility‘ is used here as a technical term defined
(Ishikawa 1990) as ’the performance of a process
over a certain period of time while in the statistically
controlled state’; one way of putting it might be
‘the degree of variation in the output of the process
that is due to the process itself‘.
IS0 9000-3 elaborates this by requiring the choice
and use of quantitative measures of quality as a
trigger for ‘remedial action’ should performance
not meet targets defined for those measures, and
as a way of defining ‘specific improvement goals’.
This is an open-ended definition because, as it
admits, ’there are currently no universally accepted
measures of software quality‘. This (properly) lets
IS0 off the hook when it comes to trying to define
any ’universal’ measures, but it leaves the more
practical question to be a’mwered by the individual
organization: ’what will our measures be and how
will we use them to assess statistical control or
lack of it?‘.
Softw. Process Improve. Pract., VoI. 2,281-289 (1996)
283
 Practice Section
Measurement in CMM levels 4 and 5
For its part, the CMM also takes us into unknown
waters in levels 4 and 5. I say 'unknown' because
it seems clear that they are, at present, largely
speculative. In his first book Humphrey's (1989)
coverage of levels 4 and 5 was comparatively weak,
reflecting the fact that the role of measurement for
process improvement was still poorly understood.
However it, and the subsequent authoritative text
from the SEI (1995), proposed the translation of
the techniques of statistical process control (SPC)
(e.g. from Deming (1986) and Ishikawa (1990,
sections 2.8 and 4.7.7), and the use of defect
analysis for process improvement. I believe the ice
becomes very thin at these two levels.
Humphrey says that 'Levels 4 and 5 are relatively
unknown territory for the software industry. There
are only a few examples of Level 4 and 5 software
projects and organizations. There are too few to
draw general conclusions about the characteristics
of Level 4 and 5 organizations. The characteristics
of these levels have been defined by analogy with
other industries and the few examples in the
software industry exhibiting this level of process
capabilitf. Little appears to have changed since
1989 to provide any more examples to validate the
definitions of levels 4 and 5. I know one organiza-
tion in the UK that could reasonably lay claim to
having a level 5 operation as far as measurement
and process improvement are concerned. It is a
maintenance group dedicated to handling change
requests to a single system some decades old. Not
quite the average situation of an IT development
department moving new and legacy systems into
client/server with RAD, or of a group developing
new systems in avionics, or air traffic control, or
medical instrumentation.
There is the danger that, whilst we might find
a few organizations that meet the current level 4
and 5 requirements, they only validate the model
for organizations like them. We will only get a useful
validation if we perceive that those organizations
are representative of the types of situations prevalent
in our industry and that we can, by induction,
demonstrate that levels 4 and 5 offer an effective
route for the generality of organizations. I see no
evidence that this has happened or will happen.
Softw. Procress Improve. Pract., Vol. 2,281-289 (19%)
284
M. A. O d d
The CMM and SPC
The CMM explicitly imports the work of Shewhart,
Deming and Juran on statistical process control
and calls for its use on the software development
process. It requires that projects actively bring
process variation within acceptable quantitative
boundaries, using notions such as mean and vari-
ance. It talks about 'idenhfying special causes of
variation within a measurably stable process and
correcting ... the circumstances that drove the
transient variation to occur' (my italics). It follows
traditional SPC principles: 'without controlling the
process within statistically narrmu boundaries (small
variations in process measures), there is too much
noise in the data to determine objectively whether
a specific process improvement has an effect' (my
italics), and it deduces that level 4's aim of bringing
the process under statistical control must be
achieved before level 5's aim of process improve-
ment can be attempted. Carnegie Mellon's book
The Capability Maturity Model: Guidelines for Improv-
ing the Sofhoare Process (1995) is expressly clear
about this use of SPC (pp. 19-20).
We need to look closely at the appropriateness
of SPC to software development, since both IS0
9001 and levels 4 and 5 of the CMM rely on it. In
particular, we need to ask 'can we give meaning
to the notions of SPC and process capability when
applied to the software development process?'.
SPC AND SOFTWARE DEVELOPMENT
First we need to understand a little more about
the style of SPC called for in the Ch4M and IS09001.
A glance at Ishikawa's book (1990, section 2.1)
shows us that of the Seven Tools of Quality Control
(which include Pareto charts and histograms) the
main one is the 'control chart' which involves
'statistically calculated control limit lines'. These
control limits bound the statistical variation
inherent in the process; the wider they are apart
the less the capability of the process (Humphrey
1989, chapter 15). Note that they are statistically
calculated, on the assumption that the feature we
are measuring about our process conforms to a
normal distribution. It is the control chart style of
SPC that makes possible what the CMM calls for
in levels 4 and 5. (My investigations suggest,
however, that most people in software development
interpret SIT as Simply People Counting.)
0 19% by JohnWiley & Sons, Ltd. and Gauthier-Villars
 Practice Section
Control charts are important to SPC because
they allow the separation of what are called special
causes and common causes. ‘Special causes’ are those
that are due to a specific situation: things that are
the responsibility of the individual worker, or due
to faulty inputs to the process, or to machinery
going out of alignment. ’Common causes’ are those
that are a property of the process itself things
whose prevention is in the hands of management
as process owners. The control chart allows us
to discriminate between these; given a set of
measurements, statistical formulae give us upper
and lower control limits which bound the inherent
variability of the process. A value falling outside
those limits is thus due to a special cause. A
process which is ’under statistical control’ is one
that is free from variations other than those inherent
in the process itself - the measurements consistently
fall between the upper and lower control limits.
Only such a process, the argument goes, can
sensibly be the subject of process improvement,
for, if the process is not under statistical control
(i.e. free from special causes), we might f b d
ourselves changing the process in response to
effects generated by a special cause, with the result
that we actually make the process worse. Hence
the assumption underlying the two-stage approach
of levels 4 and 5: first get the process under control
by removing special causes (level 41, and then
improve the process by removing common, i.e.
systemic, causes.
Software development is not a manufacturing
process
However, in everyday software engineering situ-
ations we cannot expect to be able to apply
manufacturing-style SPC; the software process -
or any small part of it - is not a manufacturing
process which we can adjust the knobs on. It is a
design activity. As such it is largely an intellectual
and a sociological activity, prone to many influences
including changes in the type of work being done,
subtle differences in the complexity of solutions,
changes in the technologies which the development
group might be forced to use, changes in staff or
recruitment policy, changes in general morale due
to external factors, a new project manager on the
client’s side, and, perhaps above all, the learning
that goes on inside the heads of the individuals -
let alone any planned process improvements being
8 1996 by John Wiley & Sons, Ltd. and Gauthier-Viars
CMM and I S 0 9001
put in place. On a manufacturing line, if I give
the green knob half a turn I know - because there
is a physical or chemical causal model available
to me - that the widgets will be 0.01 nun thinner.
In software engineering we simply do not have
that strong causal model. If we have causal models
at all they will be very complex, and, as a
result, our ability to use measurement in the
manufacturing sense will be correspondinglyweak-
ened. Why otherwise does COCOMO have 15 cost
drivers and Mark I1 Function Point Analysis
even more?
Moreover, we have no a priori reason to suppose
that our software engineering processes behave in
a normally distributed way. If we are to use
statistics we will need other forms that do not
make this assumption.
Is this a counsel of despair? No, but it is a
counsel of caution, especially to the definers of
international standards. Am I against measure-
ment? No, but I believe we have to be clear about
what is a valid role for measurement in a sociological
design activity carried out in an environment of
learning. In particular we need statistical techniques
that only make assumptions that are true of the
software engineering environment and that use
more complex causal models than SPC.
How can we use measurement?
I know that in everyday rough-and-ready terms I
can use measurement in order to spot a potential
problem: the module that seems to be taking more
time to code than one would expect for its size;
the module that has undergone more than its
quota of changes given its size; the area of the
specification that has generated the largest number
of change requests from the users (what we might
call Spotting Potential Chaos). However, this is
not the same as using SPC with its assumption of
an underlying statistical distribution model. By all
means let us make simple measurements of defect
densities and let us spot potential troublemakers,
but this is not an issue for means, or subgroup
ranges, or standard deviations, or upper and lower
control limits, or control charts, or statistical
analysis.
I also know that I can keep my measurements
over an extended period of time and perhaps, only
perhaps, spot a trend. How do I respond to that
trend? Suppose the trend represents an improve-
Softw. Process Improve. Pract., Vol. 2, 281-289 (1996)
285
 Practice Section
ment. Can I congratulate myself on the process
improvements I have recently put in place, or is
it more likely that any improvement due to them
is swamped by the fact that this is the second such
system that this team has built?’ Suppose the trend
represents a degradation. What can I tell from it
except that, for some reason, things have become
worse?
So, all that my measurements are telling me is
that things are better or worse, but I get no more
help. (I can probably tell that things are going
wrong, albeit a bit later, through my (CMM level
2) measurements of planned and actual effort use
and schedule.) To get that help I must look to
defect analysis: how did that defect get inserted
and why did it escape detection until now? This
tells us clearly what could be done to improve our
production process and our verification process.
Again we will look for trends so that we can tackle
the big and/or expensive ones first, but this is still
not a statistical question, simply one of recognizing
patterns and thinking through what we do. This
for me is a more constructive aspect of levels 4
and 5 - the institutionalized use of defect analysis
to seek common causes and remove them - and
it is an aspect that is consonant with IS0 9001.
Deming would not, I suspect, have liked my
rough-and-ready approach, but I believe that we
are in danger of being seduced by the lure of SPC.
We are dealing with a sociological system, not a
physical one; a soft system, not a hard one. If we
demanded equivalent use of SPC by physicists we
would be demanding from them the measurement
and statistical analysis of their processes for
developing hypotheses, designing experiments, car-
rying out experiments, and updating hypotheses.
They do not do any of that; should we then accuse
them of being unscientific?
So, for the everyday software development
group’s environment, I do not believe we can
achieve a process which would in Deming‘s terms
be recognizably under statistical control: the local
influencing factors are too many and too strong,
and too unavoidable. SPC in its traditional manufac-
turing sense is inappropriate for the software
engineering design process. However, all is not
lost. I believe we can find a formulation of levels
4 and 5 that still retains the notion of defect
’ This would not be a process improvement since we have not
it - it is only in the heads of our engineers.
instituti~~lized
Softw. Pmcess Improve. F’ract., Vol. 2, 281-289 (1%)
286
M. A. Ould
prevention and process improvement but that
characterizes them in ways that are meaningful
for the software development process and the
generality of software engineering groups.
DEFECT PREVENTION AND PROCESS
IMPROVEMENT WITHOUT FORMAL SPC
For defect prevention, our use of measurement
must be restricted to its power as an indicator in
a very crude sense, that of pointing to potential
trouble spots (Spotting Potential Chaos). We might
use simple and robust data exploration (for
instance, ‘box plots’ described by Hoaglin et al.
(1983)) to help us decide what is an ’outlier’, i.e.
what we wish to regard as anomalous and worthy
of investigation, and we must then use our
engineering judgement to decide if there was a
genuine reason for the outlier or if it was an outlier
only in terms of our crude discriminator. No
algorithms or statistical formulae will replace that
engineering judgement. In other words, we do not
need to use a statistical algorithm to tell us which
are the potential anomalies. We will look at anything
that looks anomalous and make a decision. I
believe this is both practical and methodologically
sound (enough).
For process improvement we must still rely on
defect analysis, but our use of defect analysis
should be restricted to the search for patterns, not
for statistical significance. We must use engineering
judgement to spot those patterns. No pattern
matching algorithm or spurious classification
scheme wdl replace that engineering judgement.
We have now disconnected anomaly detection
based on measurement from process improvement
based on defect analysis, and thereby have effec-
tively removed the ordering of levels 4 and 5. We
are now using measurement in a realistic way, but
it is not the way that either I s 0 9001 or the CMM
want us to use it.2
A level 5 organization in my definition would
therefore be one that was routinely exploiting the
measurement of quality in the refinement of its processes.
This would mean dropping the requirement for
SPC and distribution-based statistical control and
* I S 0 9001 might of course be e x 4 as it was not defined
expressly for Quality Management Systems for software
development.
0 1996 by John Wiley & Sons, Ltd. and G a u t h i e r - V i
 Practice Section
replacing it with the use of the 'simpler' approaches
of histograms, box plots, Pareto charts, etc., as
indicators. As a result, for example:
an individual project would measure defect
densities for some or all of its products;
the project would use coarse criteria based on
those densities to identdy potential trouble
spots;
the organization would consolidate defect den-
sity data at the organization level for compari-
son by projects and for identification by manage-
ment of processes needing attention;
the project would carry out causal analysis of
defects in order to make local process improve-
ments;
the organization would consolidate project-
level improvements at the organisational level
as improvements to the 'corporate process'.
WHO IS CMM LEVEL 5 AIMED AT?
The timid organization?
There is a very strong message in the CMM about
the importance of organizational focus and the
institutionalization of proven practice. At the higher
levels the degree of caution called for seems to be
increased and increased; the Technology Change
Management KPA requires 'pilot efforts' to be
performed where appropriate 'to assess new and
unproven technologies before they are incorporated
into normal practice'. What is a new technology?
Presumably one that is new to the organization.
What is an unproven technology? One that is
unproven by the organization or unproven by
anyone anywhere? What level of proof is required
for a technology to be proven? Praxis has been an
innovative user of mathematically formal methods
for system specification because, as engineers, we
believed that that was a proper way in which to
increase the quality of specifications and resulting
systems. The problem of finding a convincing pilot
for new technology is a familiar one, and I would
hazard that the majority of innovations come
because someone knew in their engineering bones
that 'this is the right way to do things'. In the
eyes of the CMM, this sort of 'gut-feel' decision-
making puts at risk the established process and
lacks organizational focus. Must a company refuse
0 19% by JohnWiley & Sons, Ltd. and Gauthier-Villars
Ch4M and I s 0 9001
to use the next version of Oracle until they have
proven their capability of using it with a pilot?
Pilots are a natural risk management strategy, but
can we always be so risk-averse in the face of an
ever-increasing rate of product change beneath our
feet? Standing back, perhaps what is needed is the
capability of adopting new technologies, new products,
and new infrastructures, quickly and effectively, rather
than of analysing them slowly.
The boring organization?
One might guess that only an organization that is
well insulated from its environment and that
internally has a high degree of stability of work
area, staff, infrastructure, and technology has any
chance of attaining level 5. The organization is
probably a high volume organization, repeating
its process hundreds or thousands of times.
The definition of level 5, as it stands, seems to
be a blind alley in that it defines a target that is
irrelevant to all but a tiny minority of organizations
that find themselves in one, very unusual situation.
I feel we should be looking for a definition that is
wider, not because it will let more people 'in', but
because a wider definition could more usefully
characterize the sorts of organization that buyers
of software will be looking for; those able to take
on new work, to adapt to changing circumstances,
to deal with the large forces at work in a social
activity like software development, and to reliably
adapt to new technologies whilst improving its
processes as far as stability permits. A wider
definition would also represent a realistic targef
for more organizations.
This all suggests a level 5 that promotes sound
continuous process improvement at the organiza-
tional level whilst recognizing at the same time
the sociological and intellectual content of software
development and the reality of change that software
developers actually have to manage.
The call for strict SPC could breed only inward-
looking, ossdymg, risk-averse, dull, and increas-
ingly outdated software development organiza-
tions. The search for stability of process called for
in the CMM is only there to make the separation
of special from common causes possible with SPC -
but is not this a case of the tail wagging the dog?
(Remember that I am talking here about level 5
competence for an organization undertaking a number
of projects. The situation for the individual project
Softw. Procffs Improve. Pract., Vol. 2,281-289 (19%)
287
@ Practice Section
is of course quite different; a project can be a
relatively closed world with a single problem to
solve, and with fixed infrastructure and techno-
logies. Its chances of achieving stability are greater
given the greater likelihood of a stable environment,
and, more importantly, it is to its advantage to
achieve that stability and to exploit it. The major
example given in The Capability Maturity Model:
Guidelines for Improving the Software Process is for
the Flight Software Project at IBM Houston which
is just such a project.)
T h e learning organization?
This surely is what both I S 0 9001 and the upper
levels of the CMM are trying to get at. We want
to say ‘this organization has got its process act
sorted out and learns’. In this way, level 5 could
be used to speclfy an organization that can be
trusted to deal with the change that is endemic in
our world, not one that has spent its time trying
to isolate itself to the point where SPC can be
applied (even if it were applicable). The mistake
is to spec* a learning mechanism - SPC - which
requires a priori that software engineering processes
operated by software engineers exhibit two-dimen-
sional statistical variation conforming to a normal
distribution. Organizational learning is a large,
complex, multi-dimensional thing not reducible to
the application of SPC.
SUMMARY
IS0 9001 and the CMM up to level 3 are mutually
supportive and complementary. An organization
should not see them as mutually exclusive. It can
aspire to a level 3 rating whilst using an IS0 9001
QMS. It can aspire to an IS0 9001 certificate for
its QMS whilst using the progression in the CMM
to get there. Anyone doing either will want to
take from each whatever suits their business.
Defect prevention and process improvement,
called for in general terms in IS09001 and detailed
in CMh4 levels 4 and 5, are traditionally discussed
in terms of SPC, but control chart based SPC is
neither appropriate to, nor is it necessary for,
defect prevention and process improvement in the
software development process. Cruder approaches
can be used that recognize the intellectual and
sociological context of software development as a
S o h . procesS Improve. Pract., Vol. 2,281-289 (1996)
288
M. A. Ould
design activity. The resulting ‘interpretation’ of
levels 4 and 5 would still characterize organizations
committed to continuous improvement but do it
in a way that makes it applicable to the generality
of development organizations, rather than to a
narrow set whose processes can be more
mechanistic.
The CMM’s higher levels recognize an organiza-
tion’s ability to hone its process over time, but
perhaps the most important process a software
development group must acquire is that of taking
on and adapting to change very quickly. If change
is the norm and stability the exception in today’s
software development environment, then perhaps
the industry - and the CMM - should value the
ability to deal with change above the ability to
achieve stability.
If our aim is to capture the characteristics that
make for a learning organization, SPC is not the
foundation we should build our international
standards on. We need to drop the distinction
between CMM levels 4 and 5, which arises princi-
pally from the distinction between special and
common causes, and to concentrate on the real
learning process at the individual and the organiza-
tional level.
ACKNOWLEDGEMENTS
My thanks go to colleagues who have contributed
to the debate in Praxis and helped shape my views,
in particular Dave Deans, Mike Hewson, Tim
Huckvale, George May, Chris Miller, and Chris
Warren, and to others in the wider software
engineering community.
REFERENCES
Deming, W. E. 1986. Out of the Crisis. Cambridge
University Press.
Hoaglin, D. C., Mosteller, F. and Tukey, J. W. (eds) 1983.
Understanding Robust and Exploratory Data Analysis. Wiley.
Humphreys, W. S. 1989.Managing the Software Process.
Addison-Wesley.
Ishikawa, K. 1990. Introduction to Quality Control. 3A
Corporation (translated by J. Loftus).
ISO.1994. ISO 9001:1994. Quality systems - Model for
0 1996 by John Wiley & Sons,Ltd. and Gauthier-Villars
@ Practice Section CMh4 and I S 0 9001

quality assurance in design/development, production, Paulk, M. C. 1995. How IS0 9001. compares with the
installation and servicing. CMM. IEEE Software, 12, No. 1, 7 M 3 .

Paulk, M. C., Curtis, B., Chrissis, M. B. and Weber, C.

V. 1993. Capability Maturity Model for Software. Version Software Engineering Institute. 1995. The Capability
1 .l. Software Engineering Institute, CMU/SEI-93-TR-24, Maturity Model: Guidelines for Improving the Software
Camegie Mellon University, Pittsburgh, PA. Process. Camegie Mellon University, Addison-Wesley.

EDITOR’S COMMENT

The Capability Maturity Model and the ISO9OOX standards build the basis of software process improvement.
There is some debate how both are related and to what extend they contribute to improved software
development. Ould points out that both are useful for driving software process improvement into the right
direction, but that they cannot guarantee high quality software. He clearly demands that the spirit of the
standard has to be lived (in contrast to just the following the words of its definition). He claims, that the
intelligent use of IS09001 and CMM brings combined benefits. Additionally, he sketches his experiences
with a statistical view on software process improvement. This experience is that software processes
cannot be controlled by statistical means, but that software process should be understood as collaborative
design activity. The experiences of Ould are based on building up a quality management system (which
served as repository of knowledge about the software process) and from several consulting projects. That
is why the practical experiences presented in this paper should be worthwhile for all practitioners who
plan to introduce quality management.
-VG

@ 19% by John Wiley & Sons, Ltd. and Gauthier-Villars S o h . Process Improve. Pract., Vol. 2,281-289 (19%)

289