Overruns or Underestimates?: Thucydides (C. 460 BC - C. 395 BC)

Overruns or Underestimates?
"... their judgment was based more on blind

wishing than upon any sound prediction; for it is
the habit of mankind to entrust to careless hope
what they long for, and to use sovereign reason
to thrust aside what they do not desire."
Thucydides (c. 460 BC – c. 395 BC)
A Political Perspective on
Software Cost Estimation
Eduardo Miranda
Carnegie Mellon University
October 15th, 2012
© Eduardo Miranda, 2012
Estimation politics
 The purposeful shrinking or padding of an estimate to

achieve an objective or favor a course of action not
warranted by the unaltered estimate
– Shrinking consists in reducing the estimate below what the estimator
honestly believes will be required
– Padding refers to the practice of increasing the estimate beyond what

the estimator honestly believes will be required to develop a system
 An adjustment to compensate for what you think others

are or will be doing

Loss leader strategy ≠ Target costing ≠ Political
behavior
Loss leader strategy Target costing

Price ≠ Budget ≠ Cost ≠ Estimate
 Estimate
 – A range of values within which an
organization believes it is possible for it to
achieve the objectives of the project with a
defined probability
 Price
– What the seller or developer will get for doing
the work. Depending on the strategic intent
of the seller it could be higher than the cost
estimate, with the difference being the
expected profit or mark-up or they could be
lower, a loss, with the hope of entering a new
 market, acquire a new technology or recoup
it in follow-up work
 Budget
– A definite amount in terms of effort,
schedule, and resources, hopefully chosen
from the range of possible values defined by
the estimate, to which the budgetee or
executor of the project commits to
 Margin
– The desired profit
 Cost
– The amount of money or resources that has
been used to produce or acquire something,
and hence is not available for use anymore
 Profit (or loss)
– The difference between the price and the
cost

Agenda
 The cone of uncertainty revisited
 What is the evidence?
 Successful projects, price and cost, the consequences of

overruns
 What can be done?
 Conclusion

The cone of uncertainty
Relative
Size
Barry Boehm,
Range Software
Engineering
Economics,
Prentice Hall,
1981
Todd Little,
Schedule
Estimation and
Uncertainty
Surrounding
the Cone of
Uncertainty,
IEEE Software,
May/June 2006
Plans and Product Detailed Development

Feasibility
Requirements Design Design and Test
Phases and Milestones

Some plausible explanations
 Cognitive bias
– Overconfidence
– Anchoring
– …….
 Individual & administrative behaviors
– Student syndrome
– Parkinson’s law
 Political behavior
– Shrinking
– Padding
 All of the above

Cognitive bias
Overconfidence Anchoring
Variability and Calibration of Expert Judgment in Software Anchoring and Adjustment in Software Estimation, J. Aranda, 2005
Estimation during the Bid Phase of a Project – An Exploratory
Survey, Pedro Faria, University of Coimbra, 2011

Individual & administrative behaviors
Parkinson’s Law Student syndrome

0.1 1.8
0.09 1.6
0.08
1.4
0.07
1.2
0.06
1
0.05
0.8 Normal
0.04
0.6
0.03 Parkinson's
Law
0.4
0.02
0.01 0.2
0 0
10 15 20 25 30
Task duration

Cognitive bias vs. political behavior as cause of
chronic underestimations
(Politics)
(Cognitive bias)
B. Flyvbjerg, From Nobel Prize To Project Management: Getting Risks Right, 2006

So, what is the evidence?

"The Montreal Olympics can no more have a
deficit, than a man can have a baby"
 Montreal won the bid for the 1976 Olympic
Games in May 1970, the estimated budget
was $120 M
 On Jan. 29, 1973 J. Drapeau, Montreal’s
mayor announced that the games would
cost $310 million
 The 1972 Munich Olympics had cost
around $600 million

Montreal’s Olympic Stadium and Toronto’s CN
Tower
•Completed in 1976
•Budget 63 million
•40,500 cubic meters of concrete
•Headcount peak 1,500
•Cost per cubic meter of concrete
•Original budget 130 million. $1,555
•Final construction 770 millions

•Total cost including repairs and
interest: 1.5 billion
•Planned completion 1976
•Completed 1987
•400 000 cubic meters of concrete
•Headcount at peak of construction
10,000
•Cost per cubic meter of concrete
$1,925

The C-130 Avionic Modernization Program (AMP)
according to GAO
GAO-08-467SP Assessments of
Major Weapon Programs

C-130 AMP
According to the C-130

AMP PO the amount of
wiring and the number of
harnesses and brackets
needed for the installation
had been underestimated
by 400 percent
GAO-08-467SP
Assessments of Major
Weapon Programs

The EMS Commercialization Project

EMS Project
 Migrate 2,000,000 SLOC of application code from
mainframe shared memory to client-server architecture
– Fortran
– Assembly
– Hard real-time
– Custom interfaces to SCADA system
– Complicated GUI
– Safety critical application
– Two organizations that have never worked together and with very
different cultures (public service, business) located 600 km away
– Productizing the system
– Deployment in a foreign country
– Generating documentation
 Constraints
– 8 months from bid to promised date + 2 months ready on-site
– 1.5 M budget (hardware excluded)

Sensitivity chart for EMS
Unfeasible
Need to rewrite
CPF MPT
Feasible

ERP Implementations
Medium Organizations $m
Hardware
–Application, Web, and database servers including 0.8+
storage
Software ERP application Suite License

–HR, Financials, Distribution $3.2+
–1,000 seats
Implementation 9 months to complete pilot

site including process engineering, apps
configuration, and testing $9.3+
–30 external consultants @ $1,200 a day
–30 internal staffers @ $100,000
Deployment
–3 external consultants at 9 sites for 3 months
–9 internal staffers at each site for 6 months
–5 days of user training at an average burdened user $7.5+
salary of $50,000
–3 full-time training staff at an average burdened
salary of $100,000
Total 20.8+

The Validation Of A Political Model Of Information
Systems Development Cost Estimating, A. Lederer and
J. Prasad, 1991

Software Management And Cost Estimating Error, A.
Lederer and J. Prasad, 1999
Uses Practices Methods Result
Justify using cost- Multiple

benefit analysis
(+) phased
Staff projects (+)
estimates
Monitor project (+) User (+)
management
Schedule Structured
signs-off estimate
projects
Estimate (+) (+)
Select projects prepared Accuracy of
proposal stage (-)
(+) estimates is part Project
of IS performance overruns
Evaluate review
estimators (+)
Evaluate Estimates and
developers actuals are
Charge users (+) audited
Accuracy of Informal
estimates is part Software
of users management and
cost estimating
performance error, A. Lederer
& J. Prasad,
review 1999

Other studies
 Any other cost estimation inhibitors? A. Magazinović & J. Pernstål,
Chalmers University, 2008
– The results of this study were used for validation of the results of the Lederer and Prasad study
that was conducted in the form of a questionnaire, answered by 112 software professionals with
a response rate of 28%
– Pressures from manager, users or others to increase or reduce the estimate was fully validated
by the issues found in both cases.
– Removal of padding was partly validated by an issue found in the VCC case.
 Better sure than safe? Over-confidence in judgment based software
development effort prediction intervals, M. Jørgensen et al, University of
Oslo, 2002
– Hidden agendas: Software professionals have goals other than just a high correspondence
between confidence level and hit rate. In particular, the desire to be evaluated as a skilled
software developer may be an important agenda that leads to overly narrow effort Pis.
– Project managers favor narrow intervals and high confidence: Study D indicates that most
software developers preferred effort PIs that were much too narrow or based on a much too high
confidence level, even when they knew that they were much too narrow
 When planners lie with numbers, M. Wachs, 1989
– Planning, however, is not just analytical. We work in the fishbowl of politics and public-policy
making. We serve as staff to politicians, consultants to government bodies, and representatives
of private landowners and real estate developers. These roles are usually associated with
clearly articulated interests. Our agencies, employers, and clients favor particular policies or
programs for reasons that may be derived more directly from ideology, political commitments, or
economic self-interest than from the results of analytical studies.
– I have experienced this conflict between planning as science and planning as advocacy in my
own consulting, and have accumulated dozens of case studies from alumni who return to the
university to talk about their anxieties and conflicts as professionals.
Successful projects & the consequences
of underestimations

Budget overruns are not the same as project
failures
 Iridium
– 66 LEO satellites designed,
built, launched, and operated
successfully
– Completed on schedule
– Completed below budget -$5B
– Bankruptcy
– Entire system was sold for $25M
in 2000
 Titanic
– Six months late
– $100M budget, final cost $200M
– Financial and creative
blockbuster
– Revenue of $1.8B

Even if overruns do not equate to project
failures, underestimations are not free:
 They affect which projects get done
 How much we end up paying for them; and
 Might throw the organization into a vicious firefighting

cycle

Which projects get done?
“We have found it isn’t necessarily the best

ones, but those projects for which proponents
best succeed in conjuring a fantasy world of
underestimated costs, overestimated revenues,
undervalued environmental impacts and
overvalued regional development effects”
Machiavellian Megaprojects, Bent Flyvbjerg, 2005

Olympic gold
Atlanta Sydney Athens Beijing London Rio
Year 1996 2000 2004 2012 2012 2016
AUS$ Euros
Original US$ 800 US$ 15 US$ 2.4 US$14
1.7 1.5
budget millions billions billions billions
billions billions
Euros US$
Actual US$1.8 AUS$ 6 US $43*
12 15+
cost billions billions billions
billions billions
* Estimated. The actual cost has not been revealed by the Chinese government

The Freiman Curve
Budget
29
Underestimated projects cost more
Fatigue
Overtime (Eoa)
Learning Effort
contributed by bringing in
additional personnel (Ea)
FTE
Fatigue
Effort contributed thru overtime (Eob)
Communication Overhead (Eco)
Budgeted Effort (Eb)

Ramp-up Coaching
preparation
Td Ta Tl
Tb

Recovery cost
Project Parameters
Tb = 12 months
Ta = 1 month
Tl = 1 month
FTEb = 20 people
Teams = 4
Lag = 2 months
Ci = 0.025
Cr = 0.1
Cc = 0.1
Cd = 0.1
Co = 0.0
Co = 0.0
b  t   b  t   4a  t  c  u, t 
2
FTEa  t , u  
2a  t 

Underestimations lead to system inefficiencies
Number of + -
+
worked Fatigue Productivity
hours -
PM +
intervention
+ + Number +
- QA
of Rework
+ activities
defects
+ -
Rate of - Project
progress workload +
+ Other
Resources + projects’
allocated to delays
the project +/-
- -
+
-
Add - Organization Multitasking
+ resources al resource
- + availability
+
Underestimation Management
intervention + Scope + “Band + Organization
Aid”
cuts workload
projects
What can be done?

Before You Start
 Attachment to issues have career consequences. How do
you expect your advocacy to affect your goals?
 The introduction of an estimation or screening process

will impose limits in otherwise discretionary decisions,
and although people vary to the extent that they seek
power, they rarely relinquish it voluntarily.
 There is not unequivocal substantive solution to the

problem. Other parties have views and solutions of their
own and they are not likely to support you until they
recognize their own views in it.
 Not matter how good the process, it cannot succeed on

its own. It depends on others who can amend, delay,
obstruct and even block its deployment and operation.
Estimation Perspectives
Political: There are multiple and distinct interests in

organizations, each pursuing its own, occasionally
parochial, objectives
Cognitive: Our predisposition to judge future events

in a more positive light than is warranted by actual
experience
Epistemological: What do we know? How?

When?
Models: Measurement errors,
assumption errors, and scope
errors

Your options
 Reduce political influences
 Compensate for political influences

Reducing political influences (The inside view)
 Visibility
– What is the work to be done?
– What risks were considered and which ones were excluded?
– Worst, most likely and best case estimates for effort, cost and schedule
– Safe estimates (x) for effort, cost and schedule
– Staffing curves
– Correlations between work elements
 Sanity checks
– A final check that you should do whenever you solve a problem. It’s an
informal test to see whether your answer makes sense in the context of the
problem. It’s not detailed enough to verify that an answer is correct, but it is
good at detecting logic or assumption errors which produce answers that
cannot possibly be right
 Reliability
– Given the same outputs two persons should arrive at more or less the same
result for the same reason
 Traceability
– It should be possible to understand how the outputs are related to its inputs.
Not black magic.
 Advocates / Non-advocates estimates

Benchmarking & Compensating (The outside view)
 Rolls Royce’s Lead Time Model
 Class reference forecasting

Lead Time Model – Rolls Royce Controls
A. Powell, Right on Time: Measuring, Modelling and Managing Time-Constrained Software Development , PhD. Thesis, University of York, 2001

Class Reference Forecasting

Uplift curves
Project sponsor must

present justification for
the uplift not to be
applied
The British Department for Transport

Procedures for Dealing with
Optimism Bias in Transport
Planning
Guidance Document
June 2004

Reference Class Forecasting for DoD Projects – Cost Correction
Cost Overruns & Underruns For 45 DoD Projects According to
GAO 2008
 C130 – AMP
14 100.00%
12
90.00% proposed budget =
80.00% 3.9B
10 70.00%
60.00%  Acceptable chance
Frequency
6
50.00% of cost overrun =
40.00%
30.00%
10%
4
2
20.00%  Required uplift =
0
10.00%
0.00%
190%
-25 0 25 50 75 100 125 150 175 200 225 250 275 300 More
 Uplifted budget =
Percentage of Original Budget
Frequency Cumulative % 11.3B
Required Cost Uplift  Latest estimate with
350 half of original units
300 to be delivered =
Original Budget Adjustment
250 5.3B
200  Latest estimate with
150 original number of
100 units to be delivered
50 = 12.5B
0
0.00% 10.00% 20.00% 30.00% 40.00% 50.00% 60.00% 70.00% 80.00% 90.00% 100.00%
Acceptable Chance of Cost Overrun

Distribution Trendline
Reference Class Forecasting for DoD Projects – Schedule
Correction
Schedule Overruns & Underruns for 45 DoD Projects According
to GAO 2008  C130 – AMP
14 100.00%
proposed schedule
12
90.00% = 48 months
80.00%
10 70.00%  Acceptable chance
of cost overrun =
Frequency
8 60.00%
50.00%
6
40.00% 10%
4 30.00%
2
20.00%  Required uplift =
0
10.00%
0.00%
90%
Uplifted schedule =
-10 0 10 20 30 40 50 60 70 80 90 100 More
Percentage of Original Schedule 
Frequency Cumulative % 91 months
Required Schedule Uplift
 Latest estimate = 84
160
140
months
O riginal S chedule A djustm ent
120
100
80
60
40
20
0
0.00% 10.00% 20.00% 30.00% 40.00% 50.00% 60.00% 70.00% 80.00% 90.00% 100.00%
Acceptable Chance of Schedule Overrun Distribution Trendline

Applying Reference Class Forecasting to
software development and information systems
acquisitions
Collect Past
completed project data Perfromance project data base
Data
A1
Select reference projects

Reference
query Projects
A2
Generate schedule reference distribution

Class
Reference
Distributions cost reference distribution
A3
cost uplift
Calculate schedule uplift
Uplift
acceptable cost overrun chance
Values
acceptable delay chance
A4
adjusted cost
adjusted schedule
Adjust
Estimates
estimated cost
estimated schedule
A5

Summary
 Prices and targets are not estimates. They should be

recognized as such and the cost of following a given
strategy acknowledged
 Underestimations cause inefficiencies
 Political and cognitive biases distort the inputs used by

models and determine the fate of its outputs
 When an estimate can only be explained by delusion or
deception, it is safer to assume the latter

The estimation
problem isn’t
unsolvable, we
just need to
understand what
is involved and
perhaps use it as
a competitive
advantage

Overruns or Underestimates?: Thucydides (C. 460 BC - C. 395 BC)

Uploaded by

Copyright:

Available Formats

You might also like

Overruns or Underestimates?: Thucydides (C. 460 BC - C. 395 BC)

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Overruns or Underestimates?: Thucydides (C. 460 BC - C. 395 BC)

Uploaded by

Copyright:

Available Formats

Overruns or Underestimates?

"... their judgment was based more on blind

 The purposeful shrinking or padding of an estimate to

– Padding refers to the practice of increasing the estimate beyond what

 An adjustment to compensate for what you think others

© Eduardo Miranda, 2012

Loss leader strategy Target costing

© Eduardo Miranda, 2012

© Eduardo Miranda, 2012

 The cone of uncertainty revisited

 What is the evidence?

 Successful projects, price and cost, the consequences of

© Eduardo Miranda, 2012

Plans and Product Detailed Development

Phases and Milestones

© Eduardo Miranda, 2012

© Eduardo Miranda, 2012

Parkinson’s Law Student syndrome

© Eduardo Miranda, 2012

© Eduardo Miranda, 2012

© Eduardo Miranda, 2012

© Eduardo Miranda, 2012

•Final construction 770 millions

© Eduardo Miranda, 2012

© Eduardo Miranda, 2012

According to the C-130

© Eduardo Miranda, 2012

© Eduardo Miranda, 2012

© Eduardo Miranda, 2012

© Eduardo Miranda, 2012

Software ERP application Suite License

Implementation 9 months to complete pilot

© Eduardo Miranda, 2012

© Eduardo Miranda, 2012

Justify using cost- Multiple

© Eduardo Miranda, 2012

© Eduardo Miranda, 2012

© Eduardo Miranda, 2012

 How much we end up paying for them; and

 Might throw the organization into a vicious firefighting

© Eduardo Miranda, 2012

“We have found it isn’t necessarily the best

Machiavellian Megaprojects, Bent Flyvbjerg, 2005

© Eduardo Miranda, 2012

Atlanta Sydney Athens Beijing London Rio

Year 1996 2000 2004 2012 2012 2016

© Eduardo Miranda, 2012

Communication Overhead (Eco)

Budgeted Effort (Eb)

© Eduardo Miranda, 2012

© Eduardo Miranda, 2012

© Eduardo Miranda, 2012

 The introduction of an estimation or screening process

 There is not unequivocal substantive solution to the

 Not matter how good the process, it cannot succeed on

Political: There are multiple and distinct interests in

Cognitive: Our predisposition to judge future events

Epistemological: What do we know? How?