Professional Documents
Culture Documents
Risk Management Using Risk+ (V5)
Risk Management Using Risk+ (V5)
Risk Management Using Risk+ (V5)
A workshop on the principles and practices of Risk+ and increasing the Probability of Program Success
A Warning
Were going to cover a lot of material in 3 hours
MOTIVATION? Your motivation? Your motivation is your pay packet on Friday. Now get on with it. Noel Coward, English actor, dramatist, & songwriter (1899 1973)
We have to know the underlying statistical behavior of the processes driving the project This means cost, schedule, and technical performance measures with probabilistic models We need to know how these three statistical drivers are coupled What drives what? What are the multipliers between each random variable?
In building a risk tolerant IMS, were interested in the probability of a successful outcome
But the underlying statistics of the tasks influence this probability The statistics of the tasks, their arrangement in a network of tasks and correlation define how this probability based estimated developed.
There are real problems with those pesky Unknowns that get in the way of progress
11
Imprint of a bird on our west facing family room second story window on a bright afternoon The Bird survived
These classifications can be used to avoid asking the 3 point question for each task. Anchoring and Adjustment of all estimating processes produces a bias. Knowing this is necessary for credible Classification Uncertainty Overrun estimates.
Routine, been done before Routine, but possible difficulties Development, with little technical difficulty Development, but some technical difficulty Significant effort, technical challenge No experience in this area Low 0% to 2% Medium to Low 2% to 5% Medium Medium High High Very High 5% to 10% 10% to 15% 15% to 25% 25% to 50%
1 2 3 4 5 6
Were looking for knowledge of what is going to happen in he future, with a known level of confidence
13
14
15
George Louis Leclerc, Comte de Buffon, asked what was the probability that the needle would fall across one of the lines, marked here in green.A l sin That outcome will occur only if
Monte Carlo Simulation is named after the city, in Monaco, of casinos on the French Rivera. Monte Carlo
Examines all paths not just the critical path. Provides an accurate (true) estimate of completion: Overall duration distribution Confidence interval (accuracy range) Sensitivity analysis of interacting tasks Varied activity distribution types not restricted to a single distribution Schedule logic can include branching both probabilistic and conditional When resource loaded schedules are used provides integrated cost and schedule probabilistic model.
19
20
Here is some advice on how to depict this margin and where to place this margin. No matter how we show manage these two elements in the IMS, if we dont have margin we are late and over budget before we start.
21
http://www.ndia.org/Divisions/Divisions/Procurement/Documents/PMSCommittee/CommitteeDoc uments/WhitePapers/NDIAScheduleMarginWhitePaperFinal-2010(2).pdf
As the program proceeds we want to have Increased accuracy Reduced schedule risk Increasing visual confirmation that success can be reached
Programmatic Margin is added between Development, Production and Integration & Test phases Risk Margin is added to the IMS where risk alternatives are identified
Duration of Plan B < Plan A + Margin
Margin that is not used in the IMS for risk mitigation will be moved to the next sequence of risk alternatives
Plan B
This enables us to buy back schedule margin for activities further downstream This enables us to control the ripple effect of schedule shifts on Margin activities
Plan A
5 Days Margin
2 days will be added to this margin task to bring schedule back on track
Sensitivity Analysis
24
The schedule sensitivity of a task measures the closeness with which change in the task duration matches change in the project duration over the simulation. This closeness is the correlation between changes in individual activities and their impacts on other activities. A task with high schedule sensitivity is more likely to be a major driver of the project duration than a lower ranked task.
: Models of the Schedule
A measure of the frequency that an activity in the project schedule is critical (Total Float = 0) in a simulation If a task is critical in 500 of the 1,000 iterations of the simulation, it has a Criticality Index of 0.5 The higher the criticality index, the more certain it is that the task will always be critical in the project
: Models of the Schedule
task with high sensitivity may not be on or near the critical path. Thus a reduction in that tasks duration may have little effect on the project duration.
highlights critical or nearcritical activities with high. tasks are most likely to drive project
Schedule Sensitivity
These
: Models of the Schedule
Guiding the Risk Factor Process means weighting each level of risk
27
For tasks marked Low a reasonable approach is to score the maximum 10% greater than the minimum. The Most Likely is then scored as a geometric progression for the remaining categories with a common ratio of 1.5 Tasks marked Very High are bound at 200% of minimum.
Min
Most Max Likely 1.04 1.06 1.09 1.14 1.20 1.10 1.15 1.24 1.36 1.55
No viable project manager would like a task grow to three times the planned duration without intervention
High+
Very High Very High+
1.0
1.0 1.0
1.30
1.46 1.68
1.85
2.30 3.00
The geometric progress is somewhat arbitrary but it should : Examples of Monte Carlo
A geometric progression (1.534) of risk can be used. The phrases associated with increasing risk have been shown at the Naval Research Laboratory to correlate with an engineers sense of increasing risk.
The narrative for each risk factor needs to be developed. Each description is dependent on
Discipline Program stage Complexity Historical data Current risk state of the program
This is currently missing from our efforts to quantify schedule and cost risk.
: Examples of Monte Carlo
Accuracy
30
Given a specified final cost or project duration, what is the probability of achieving this cost or duration? Frequentist approach
Over many different projects, four out of five will cost less or be completed in less time than the specified cost or duration. We would be willing to bet at 4 to 1 odds that the project will be under the 80% point in cost or duration.
Bayesian approach
Accuracy is needed to plan reserves. Accuracy is needed when comparing competing proposals.
: What is the Purpose of Project Risk Analysis?
Structured Thinking
31
All estimates will be in error to some degree of variance. Trying to quantify these errors will result in bounds too wide to be useful for decision making. Risk analysis should be used to
Think about different aspects of the project Try to put numbers against probabilities and impacts Discuss with colleagues the different ideas and perceptions
Work must be represented in single units either task or work packages. The overall schedule margin must be related to the variation of individual units of work. The importance of the units of work must be shared among all participants (ordinal ranking of work and its risk). The schedule must be reasonable in some units of measure shared by all the participants.
Protecting Earned Value Schedules with Schedule Margin, Newbold, Budd, and Budd, http://www.prochain.com/pm/articles/ProtectingEVSchedules.pdf
33
34
35
A Small Diversion
Most Likely Isnt Likely to be the Most Likely When we say most likely what do we think this actually means? If you pick the wrong meaning, your Monte Carlo model will be seriously flawed.
These definitions lead to values that are almost always different from each other. Rolling up the best estimate of completion is
They are random variables drawn from the probability distribution function (pdf).
Actual project duration is an uncertain quality that can be modeled as a sum of random variables
PERT assumes probability distribution of the project times is the same as the tasks on the critical path. Because other paths can become critical paths, PERT consistently underestimates the project completion time.
: Managing Uncertainty in the IMS
1+1=3
Probability of occurrence as a function of the number of samples. The number of times a task duration appears in a Monte Carlo simulation.
: Managing Uncertainty in the IMS
Lies, Damn Lies, and Statistics Benjamin Disraeli But we know better, we know that any estimate without a variance is not trustworthy. We know that the variances have to be calibrated from past performance to be credible
41
42
This is a must own book for everyone in our business. It defines fundamental Laws of program and business management, which are many times ignored like the
Risk+ Installed Lets define the needed fields These are used by Risk+ to hold information and run the application. If there are conflicts, you can make changes in Risk+ to work around your fields.
A Simple IMS
44
The duration that is the most youd expect this task to complete in
Start with the Gantt View and Entry Table Set up both to match the Risk+ field usage
Initialize the Most Likely. This sets the Most Likely duration to the same value that is in the Duration field of your IMS.
The
planned duration now becomes the ML duration. If this planned duration is bogus then your model will be as well. Choose wisely.
The next well set the upper and lower limits of that ML value
This marks that ROW in the schedule as a work activity we want to see the Monte Carlo output for
The RISK ANALYSIS command starts the process going. Lets make 200 iteration and look at the DURARTION ANALYSIS for the activities we are watching.
Risk+ is picking a random number from under the normal distribution within the range of the
Least remaining and most remaining This is not some ordinary random number it is chosen through an algorithm called the Latin Hypercube more on that later.
Risk+ then plugs that number into the real DURATION field and does that for all the DURATIONS in the schedule Then the F9 key is pressed and the date is recorded for the finish of UID 41. This is done 200 times and a histogram of all the dates that appeared for those 200 time is
And We Get
54
Date: 11/29/2011 4:32:17 PM Samples: 500 Unique ID: 19 Name: End Work Package 3
0.20 1.0
Completion Std Deviation: 2.06 days 95% Confidence Interval: 0.18 days Each bar represents 1 day
Completion Probability Table Prob 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45 0.50 Date Wed 3/7/12 Thu 3/8/12 Thu 3/8/12 Fri 3/9/12 Fri 3/9/12 Fri 3/9/12 Mon 3/12/12 Mon 3/12/12 Mon 3/12/12 Mon 3/12/12 Prob 0.55 0.60 0.65 0.70 0.75 0.80 0.85 0.90 0.95 1.00 Date Tue 3/13/12 Tue 3/13/12 Tue 3/13/12 Wed 3/14/12 Wed 3/14/12 Wed 3/14/12 Thu 3/15/12 Thu 3/15/12 Fri 3/16/12 Tue 3/20/12
Cumulative Probability
Frequency
Fri 3/2/12
Mon 3/12/12
Tue 3/20/12
Completion Date
Risk +shows use the probability of finish on or before a date It does NOT show the probability of success. But even the on or before term is loaded with special meaning. It means for the 500 iterations of Risk+ using the upper and lower bounds of the duration, drawn from the probability density function (pdf) with the Normal (Gaussian) shape, 60% of the finish dates were recorded to be on or before 3/12/12.
Date: 11/30/2011 6:05:35 PM Samples: 200 Unique ID: 17 Name: (SA) Systems Requirements Completed
0.22 1.0
Completion Std Deviation: 4.49 days 95% Confidence Interval: 0.62 days Each bar represents 2 days
Completion Probability Table Prob 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45 0.50 Date Fri 5/4/12 Wed 5/9/12 Thu 5/10/12 Fri 5/11/12 Mon 5/14/12 Mon 5/14/12 Tue 5/15/12 Tue 5/15/12 Wed 5/16/12 Wed 5/16/12 Prob 0.55 0.60 0.65 0.70 0.75 0.80 0.85 0.90 0.95 1.00 Date Thu 5/17/12 Thu 5/17/12 Fri 5/18/12 Mon 5/21/12 Mon 5/21/12 Tue 5/22/12 Wed 5/23/12 Thu 5/24/12 Mon 5/28/12 Mon 6/4/12
Cumulative Probability
0.20 0.17
0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 Wed 5/16/12 Mon 6/4/12
Frequency
Wed 5/2/12
Completion Date
Date: 11/30/2011 10:30:05 PM Samples: 200 Unique ID: 17 Name: (SA) Systems Requirements Completed
0.16 1.0
Completion Std Deviation: 9.14 days 95% Confidence Interval: 1.26 days Each bar represents 3 days
Completion Probability Table Prob 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45 0.50 Date Thu 5/3/12 Tue 5/8/12 Wed 5/9/12 Mon 5/14/12 Tue 5/15/12 Thu 5/17/12 Fri 5/18/12 Mon 5/21/12 Wed 5/23/12 Wed 5/23/12 Prob 0.55 0.60 0.65 0.70 0.75 0.80 0.85 0.90 0.95 1.00 Date Fri 5/25/12 Mon 5/28/12 Wed 5/30/12 Wed 5/30/12 Fri 6/1/12 Mon 6/4/12 Wed 6/6/12 Fri 6/8/12 Thu 6/14/12 Wed 6/27/12
Cumulative Probability
0.14 0.12
0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 Thu 5/24/12 Wed 6/27/12
Frequency
Tue 4/24/12
Completion Date
58
59
60
61
Basic Principles with Probabilistic Cost Estimating are coupled with scheduling
Each of these CERs has uncertainty (standard error) CER input variables have uncertainty (technical uncertainty)
Must combine CER uncertainty with technical uncertainty for many CERs in an estimate
Usually cannot be done arithmetically; must use simulation to roll up costs derived from Monte Carlo samples
Add and multiply probability distributions rather than numbers Statistically combining many uncertain, or randomly varying, numbers Take random sample from each CER and input parameter, add and multiply as necessary, then record total system cost as a single sample Repeat the procedure thousands of times to develop a frequency histogram of the total system cost samples : Basic Principles of Probabilistic Cost
62
Cost = a + bXc
Cost Modeling Uncertainty
Cost Estimate
In the riskadjusted cost estimate, we now combine discrete risk events and the uncertainty of the input distributions with the uncertainty of the CERs Since the input distributions tend to be right skewed, the expected cost tends to be larger than the baseline estimate In addition, the riskadjusted cost distribution tends to be wider than the baseline estimate The difference between the expected cost of the riskadjusted estimate and the expected cost of the baseline estimate is, by definition, the amount of RISK dollars included in the riskadjusted estimate
: Basic Principles of Probabilistic Cost
64
Baseline versus Risk Adjusted Cost Estimates Usually Show a Cost Increase
Baseline vs. Risk-Adjusted Estimates Baseline:
Likelihood
50
100
150 FY$M
200
250
300
350
Cumulative Probability
$80
$100
$120
$140
$160
$180
$200
FY00$M
66
The Real Question Always Returns to But How Much Does It Cost? Really? This is impossible to answer precisely
Decisionmakers and cost analysts should always think of a cost estimate as a probability distribution, NOT as a deterministic number The best we can provide is the probability distribution If we think we can be any more precise, were fooling ourselves It is up to the decisionmaker to decide where he/she wants to set the budget The probability distribution provides a quantitative basis for making this determination
Low budget = high probability of overrun High budget = low probability of overrun
67
68
69
Without Integrating $, Time, and TPM youre driving in the rearview mirror
50% of all possible values are under this area of the curve. This is the definition of the median
72
TPM Trends & Responses directly impact risk and credibility of the IMS
Design Model
ROM in Proposal
28kg
Prototype Measurement
CA
SFR
SRR
PDR
CDR
TRR
Dr. Falk Chart modified
Not A Mitigation Plan Mitigation is too late, the risk has turned into an issue. The money has been spent, and the time has passed.
73
Ordinal
A variable is ordinally measurable if ranking is possible for values of the variable. For example, a gold medal reflects superior performance to a silver or bronze medal in the Olympics, or you may prefer French toast to waffles, and waffles to oat bran muffins. All variables that are cardinally measurable are also ordinally measurable, although the reverse may not be true.
Cardinal
A variable is cardinally measurable if a given interval between measures has a consistent meaning, i.e., if the measure corresponds to points along a straight line. For example, height, output, and income are cardinally measurable.
Classify and calibrate risk ranking in units meaningful to the decision makers
Risk
rank 1, 2, 3, 4, is NOT sufficient The Risk Rank must have a measurable value connected to the actual behavior of the system being assessed
Calibration coefficients between ordinal probability and consequences should also be used. Ordinal analysis assumes ordering of the risks. Cardinal analysis provides objective measures
Never multiply Likelihood by outcome. They are not numbers, they a probability distributions. Only convolution is possible
Level Likelihood E D C B A
Value
E
Near Certainty E 90% Highly Likely Likely 74% D 90% 40% C 60%
D
C B A A B C D E
Budget increase or unit Minor reduction in technical Able to meet key dates production cost increases. performance or supportability. < (1% of Budget) Moderate reduction in Minor schedule slip. Budget increase or unit technical performance or Able to meet key production cost increase supportability with limited milestones with no < (5% of Budget) impact on program objectives. schedule float. Significant degradation in Budget increase or unit technical performance or Program critical path production cost increase major shortfall in affected < (10% of Budget) supportability. Cannot meet key Exceeds budget increase or Severe degradation in technical program milestones. unit production cost performance. Slip > X months threshold
76
77
At least 5% of the interface design has been Conrow, altered Effective Risk Management: Some Keys to Success , Edmund AIAA Press, 2003
Risk Rank A B
Interpretation of Risk Ranking Normal business, technical & manufacturing processes are applied Normal business & technical processes are applied; new or innovative manufacturing processes Flight software development & certification processes Build & qualification of flight components, subsystems & systems Flight software qualification ISS thermal vacuum acceptance testing
C
D E F
5% C 35%
10% D 25% 10% E 35% 5% F 175%
responsibilities Work authorizations that are not always followed Issues with Budget and data reconciliation Lack of an integrated management system Baseline fluctuations and frequent replanning Current period and retroactive changes Improper use of management reserve EV techniques that do not reflect actual performance Lack of predictive variance analysis
Estimates (LRE) Progress not monitored in a regular and consistent manner Lack of vertical and horizontal traceability cost and schedule data for corrective action Lack of internal surveillance and controls Managerial actions not demonstrated using Earned Value 79
Set up the Risk+ fields, flags, views, and tables for the program standard IMS. Build an IMS that passes the DCMA 14 Point Assessment with all GREEN. Build the Ordinal Risk Ranking table for the various risk categories on the program. Assign risk ranking to each activities in the IMS, with the variances defined in the Ordinal Table. Run Risk+ to see the confidence in the deliverables. Develop the needed schedule margin to protect the delivery to at least the 80% confidence level.
Put margin in front of critical deliverables. Build a margin burn down chart and allocate schedule margin just like you do MR for the PMB. This real world advice is counter to the current DCMA guidance.
82
83
80
60
40
20
-20
-40
The Missing Link: Schedule Margin Management, Rick Price, PS10, PMICPM EVM World 2008
Baseline Plan
Oct 2011
Nov 2001
Current Plan with risks is the deterministic schedule Plan Margin 20% Risk Margin Current Plan with risks is the stochastic schedule The probability distribution can vary as a function of time
CDR ATLO FRR
Dec 2011
Jan 2012
Mean
Feb 2012
Mar 2012
80%
Apr 2012
SRR
PDR
87
88
References
References
89
Protecting Earned Value with Schedule Margin, http://www.prochain.com/pm/articles/ProtectingEVSchedules.pdf Depicting Schedule Margin in the Integrated Master Schedule, http://www.ndia.org/Divisions/Divisions/Procurement/Documents/PMSCom mittee/CommitteeDocuments/WhitePapers/NDIAScheduleMarginWhitePap erFinal-2010(2).pdf Effective Risk Management: Some Keys to Success, Second Edition, Edmund Conrow, AIAA Press. How to Lie with Statistics, Darrell Huff, Norton, 1954 (Available in paper back at any good book store) DID DIMGMT81650 A management method for accommodating schedule contingencies. It is a designated buffer and shall be identified separately and considered part of the baseline.
References
90
Interfacing Risk and Earned Value Management, Association for Project Management, 150 West Wycombe Road, High Wycombe, Buckinghamshire, HP12 3AE, United Kingdom. Practice Standard for Earned Value Management, Second Edition, Project Management Institute, 2011. Effective Opportunity Management for Projects, David Hillson, Taylor and Francis, 2004. Measuring Time: Improving Project Performance Using Earned Value, Mario Vanhoucke, Springer, 2009. Performance Based Earned Value, Paul Solomon and Ralph Young, Wiley, 2007.
Effective Risk Management: Some Keys to Success, Edmund Conrow, AIAA Press, 2003.
Niwot Ridge LLC 4347 Pebble Beach Drive Niwot, Colorado 80503
(: 303.241.9633 -: glen.alleman@niwotridge.com
91