Professional Documents
Culture Documents
1 s2.0 S0950584905001023 Main PDF
1 s2.0 S0950584905001023 Main PDF
www.elsevier.com/locate/infsof
Abstract
The Standish Group reported in their 1994 CHAOS report that the average cost overrun of software projects was as high as 189%. This
figure for cost overrun is referred to frequently by scientific researchers, software process improvement consultants, and government
advisors. In this paper, we review the validity of the Standish Group’s 1994 cost overrun results. Our review is based on a comparison of the
189% cost overrun figure with the cost overrun figures reported in other cost estimation surveys, and an examination of the Standish Group’s
survey design and analysis methods. We find that the figure reported by the Standish Group is much higher than those reported in similar
estimation surveys and that there may be severe problems with the survey design and methods of analysis, e.g. the population sampling
method may be strongly biased towards ‘failure projects’. We conclude that the figure of 189% for cost overruns is probably much too high to
represent typical software projects in the 1990s and that a continued use of that figure as a reference point for estimation accuracy may lead to
poor decision making and hinder progress in estimation practices.
q 2005 Elsevier Ltd All rights reserved.
* Corresponding author. Tel.: C47 924 333 55; fax: C47 67 82 82 01. 2. Users of the cost overrun figures
E-mail address: magnej@simula.no (M. Jørgensen).
0950-5849/$ - see front matter q 2005 Elsevier Ltd All rights reserved. Among the users of the cost overrun figures in the 1994
doi:10.1016/j.infsof.2005.07.002 CHAOS report are scientific researchers, software process
298 M. Jørgensen, K. Moløkken-Østvold / Information and Software Technology 48 (2006) 297–301
improvement groups, and government advisors. The main 52.7% is identical to the percentage of so-called
use seems to be to support arguments for more research, challenged projects. Yet even the definition of challenged
better estimation processes and improved project manage- projects is not easy to interpret. It is defined in [1] as ‘The
ment methods. These are all laudable goals, well motivated project is completed and operational but over-budget, over
by the ‘software development crisis’ implied by a 189% the time estimate, and offers fewer features and functions
average cost overrun. Unfortunately, there are several than originally specified.’ The problem here is the use of
examples of situations where the 189% result may have ‘and’ instead of ‘or’, combined with the following definition
hindered progress. The following three real-world examples of successful projects [1]: ‘The project is completed on-time
illustrate this. and on-budget, with all features and functions as initially
specified.’ Consider a project that is on-time, and on-budget,
Example 1. A project had a 146% cost overrun, i.e. the
but not with all specified functionality. Is this project to be
actual cost was about 2.5 times the estimated cost. A report
categorized as challenged or successful? If it is categorized
on the project’s performance stated that the cost overrun
as challenged, this is not consistent with the provided
was not that bad, because it was better than the industry
definition of challenged projects; while if it is categorized as
average of 189%. We have found several examples of this
successful, it fails to meet the criteria given for a successful
type of use.
project.
Example 2. A consultancy company claimed to be in the To determine whether we were the only ones confused by
elite class of software development companies, based on a these descriptions, we conducted a simple survey of the
comparison of its own high cost overrun figures with the actual use of the 1994 CHAOS report cost overrun figure.
189% cost overrun. The company’s cost mean cost overrun We examined 50 randomly sampled web documents
was high, but lower than 189%. applying the search term: ((Standish Group) AND (189%
OR 89%)) and the search engine www.yahoo.com. We
Example 3. A recent UK study of software projects reports
found the following:
an average cost overrun of 18% (The state of IT project
management in the UK 2002–2003, www.computerweek- † 50% of the documents described the result as ‘189% cost
lyms.com/pmsurveyresults/surveyresults.pdf). Here, an overrun’, 40% as ‘189% of original estimate’, and 10%
adjusted version of the 189% cost overrun figure from the as ‘89% cost overrun’. ‘189% of original estimate’ and
1994 CHAOS report is used to support an argument for an ‘89% cost overrun’ seem to reflect the same under-
enormous improvement in estimation performance since standing of the result, i.e. we found two different
1994. Readers of that study may get the impression that the interpretations of cost overrun that were used with
improvement of cost estimation processes does not require a almost the same frequency.
great deal of systematic work and focus, whereas, in fact, as † 70% of the documents related the result to ‘53% of the
the lack of progress reported in [3] suggests, it is a very projects’ (without explicitly pointing out that this 53%
demanding and difficult enterprise. referred to challenged projects only), 16% to ‘all
projects’, 8% to ‘challenged and cancelled projects’,
and 6% explicitly pointed out that the average cost
overrun is based on ‘challenged projects’ only.
3. The meaning of ‘189% average cost overrun’
Generally, in recent reports and press releases the
Standish Group seems to apply the interpretation ‘189%
Before we compare the CHAOS report results with those
average cost overrun of challenged projects’; see, for
of other studies it is important to clarify what the expression
example, the press release March 2003 (www.stan-
‘189% average cost overrun’ actually means. This turned
dishgroup.com/press/article.php?idZ2). This means that
out to be more difficult than expected. In fact, we were
many, perhaps the majority, of users interpret the results
unable to find in the CHAOS reports an explicit definition of
differently from the Standish Group.
the cost overrun measure that is applied. Only informal
descriptions were presented; and even they were inconsist-
ent with each other. The following quotations provide
typical examples of how the CHAOS reports describe the 4. A comparison with other cost estimation
189% average cost overrun: accuracy studies
† ‘. 52.7% of projects will overrun their initial cost We conducted a systematic search for estimation surveys
estimates by 189%’, page 41 in [2]. [3]. This systematic search was based on (i) a manual search
† ‘The average cost overruns for combined challenged and in about 100 software development, project management
cancelled projects is 189%.’, page 42 in [2]. and quality assurance journals (see www.simula.no\BEST-
† ‘. 52.7% of projects will cost 189% of their original web or contact the authors for a list of the journals included)
estimates’, page 14 in [1]. and (ii) a variety of searches for studies applying terms
M. Jørgensen, K. Moløkken-Østvold / Information and Software Technology 48 (2006) 297–301 299
Magne Jørgensen is a Professor at the University of Oslo and Simula Kjetil Moløkken-Østvold is a Postdoctoral Researcher at Simula Research
Research Laboratory. His research interests include software cost Laboratory. His research interests include software cost estimation, process
estimation, uncertainty assessments in software projects, expert judgment improvement and public management of software projects. He received his
processes, and learning from experience. He received a D. Scient. degree in Masters degree from the University of Oslo in 2002, and his PhD from the
Informatics from the University of Oslo, Norway in 1994. He has 10 years University of Oslo in 2004. kjetilmo@simula.no.
industry experience as a consultant and software project manager. His
publications on software cost estimation and related topics are listed at:
www.simula.no. magne.jorgensen@simula.no.