Reserves on Schedule to Go (STG)

Based on Historical Data

Bob Bitten, Bob Kellogg, Marc Hayhurst, Shirin Eftekharzadeh
The Aerospace Corporation

Voleak Roeum
NASA Headquarters

2014 NASA Cost Symposium

12-14 August 2014
Outline

• Introduction/Background

• Methodology

• Results from PDR, CDR, SIR, and PSR

• Recommendations & Summary

2
Introduction
• NASA Center guidance requires that a mission hold Funded Schedule
Reserve (FSR) corresponding to the remaining Schedule to Go (STG) at
an increasing rate over the development lifecycle

• The rates of required FSR may not be adequate given that many projects
have experienced significant schedule growth

• This study looks at the project-held FSR and actual schedule growth for a
variety of NASA missions at the Preliminary Design Review (PDR), Critical
Design Review (CDR), Systems Integration Review (SIR), and Pre-Ship
Review (PSR)

• The data can then be used as guidance for adjusting project FSR
requirements and for establishing additional reserves at the center or HQ
level for future missions

3
NASA Project Lifecycle
Phase D:
• Background
Phase A: Phase B:
Phase C: System
Project Final Design Assembly, Phase E:
Conceptual Preliminary
Phase & Integration & Operations
Design Design
Fabrication Test, Launch &
Checkout

KDP-B KDP-C KDP-D KDP-E

Milestones Launch

Reviews
PDR CDR SIR PSR LRD

• Definitions:
– KDP = Key Decision Point = Transition from one Phase to the next
– PDR = Preliminary Design Review
– CDR = Critical Design Review
– SIR = Systems Integration Review
– PSR = Pre-Ship Review
– LRD = Launch Readiness Date

4
Reserve Tracking Assumptions
• JPL and GSFC Guidelines specify funded schedule reserves in months for
Phases C & D

PDR (KDP-C) SIR (KDP-D) Ship

Milestone to to to
SIR (KDP-D) Ship Launch
JPL & GSFC Reserve Guidelines
1 month/year 2 month/year 1 week/month
Schedule Margin Rate

• Most projects track Funded Schedule Reserve (FSR) in weeks or days

– 1 Month = 4 weeks
– 1 Week = 5 working days
Phase Months/Year Weeks/Year Days/Year
KDP-C to KDP-D 1 4 20
KDP-D to Ship 2 8 40
Ship to Launch 3 12 60

• Weeks used in this analysis for consistency across all projects

5
Example Project Lifecycle
• Funded Schedule Reserve (FSR) = No planned activities, but funded schedule
• FSR Rate = FSR/(Planned Activity + FSR) = FSR / (P + FSR)
Total
Schedule

Funded
Planned Schedule
Activity Reserve
Launch
KDP-B KDP-C KDP-D Ship KDP-E
P FSR P FSR P FSR

Critical Path
PDR CDR SIR PSR LRD

Phase ATP to PDR to SIR SIR to PSR PSR to Launch Total

PDR

Example Duration 0.75 2.0 1.0 0.25 4.0

(years)
Guideline FSR Rate 0 4 weeks/year 8 weeks/year 12 weeks/year

Guideline FSR 0 8 8 3 19
(weeks)

6
Outline

• Introduction/Background

• Methodology

• Results from PDR, CDR, SIR, and PSR

• Recommendations & Summary

7
Large Diversity of Missions Included in Analysis
• Data collected from 37 science missions providing a representative mix
Launch Mission Launch Mission
Theme Lead Org. Theme Lead Org.
Mission Date Type Mission Date Type
AIM Apr-07 Helio CAT 3 Other RHESSI Feb-02 Helio CAT 3 Other
Cloudsat Apr-06 Earth CAT 2 JPL SDO Feb-10 Helio CAT 1 GSFC
DAWN ct Sep-07 Planetary CAT 2 JPL Spitzer Aug-03 Astro CAT 1 JPL
Deep Impa Jan-05 Planetary CAT 2 JPL Stardust Feb-99 Planetary CAT 2 JPL
EO-1 Nov-00 Earth CAT 2 GSFC STEREO Oct-06 Helio CAT 2 GSFC
GALEX Apr-03 Astro CAT 3 JPL Swift Apr-04 Astro CAT 2 GSFC
GENESIS Aug-01 Planetary CAT 2 JPL Terra Dec-99 Earth CAT 1 GSFC
GLAST Jun-08 Astro CAT 2 GSFC WISE Jun-09 Astro CAT 2 JPL
GRAIL Sep-11 Planetary CAT 2 JPL GLORY Mar-11 Earth CAT 2 GSFC
IBEX Oct-08 Helio CAT 3 Other GPM Feb-14 Earth CAT 2 GSFC
ICESAT Jan-03 Earth CAT 2 GSFC IRIS Jun-13 Helio CAT 3 LM
Kepler Mar-09 Astro CAT 2 JPL Juno Aug-11 Planetary CAT 1 JPL
LRO Jun-09 Planetary CAT 2 GSFC LADEE Sep-13 Planetary CAT 2 ARC
MER R Jul-03 Planetary CAT 1 JPL LDCM Feb-13 Earth CAT 2 GSFC
MESSENGE Aug-04 Planetary CAT 2 APL MAVEN Nov-13 Planetary CAT 3 GSFC
MRO Aug-05 Planetary CAT 1 JPL MSL Nov-11 Planetary CAT 1 JPL
New Horiz Jan-06 Planetary CAT 1 APL RBSP Aug-12 Helio CAT 2 APL
OCO Feb-09 Earth CAT 2 JPL NuSTAR Jun-12 Astro CAT 3 JPL
PHOENIX Aug-07 Planetary CAT 2 JPL

• Includes mix of Science Themes

– 15 Planetary, 7 Astrophysics, 8 Earth Science and 7 Heliophysics
• Includes mix of different sized missions based on NASA 7120.5E Mission Category
– 8 Category 1 (CAT 1) missions with Life Cycle Cost (LCC) > $1B FY12$
– 22 Category 2 (CAT 2) missions with LCC >$250M but < $1B FY12$
– 7 Category 3 (CAT 3) missions with LCC < $250M FY12$
Study Definitions
– Project Actual Margin (FSR) = Actual historical margin held by project at milestone
– Actual Required Margin = Project Actual Margin + Schedule Growth = FSR + G
– Actual Schedule Margin Rate =
• Actual Required Margin/(Planned Activity + Project Actual Margin) = (FSR + G) / (P + FSR)
G
Planned Duration P FSR

Actual Duration

– External Events: factors resulting increased schedule growth which were outside of the control of
NASA and/or the project and unreasonable to anticipate and plan for
• NASA External: Contractor closes and relocates facility; labor strike; Hurricane delays launch
• Project External (Including above): center/headquarters directed launch date move; funding cut;
stand down

– Actual Required Margin less External Events =

• Actual Required Margin – External Event Duration = (FSR + G) - E
– Actual Required Margin Rate less External =
• Actual Required Margin less External Events/(Planned Activity + Project Actual Margin)
• ((FSR + G) – E) / (P + FSR) G
Planned Duration P FSR
Actual Duration E

9
Outline

• Introduction/Background

• Methodology

• Results from PDR, CDR, SIR, and PSR

• Recommendations & Summary

10
 Results of Historical Mission Examination
FSR Rates - All Missions
50.0
Weeks per year calculations
-Numbers: average weeks per year used between
45.0 each milestone
Weeks per year between Milestone

42.7
-Orange line: unadjusted calculation from the raw
40.0 schedule data

-Purple line: adjusted calculation removing NASA

35.0 external events 33.9

30.0 - Blue line: adjusted calculation removing Project

external events 28.4

25.0 -Black line: current FSR guidelines

24.3

20.0 20.5
19.2
14.9
15.0 14.7 13.0
13.5
10.6 12.0
10.0 10.5
8.0
5.0
4.0 4.0
0.0
PDR to SIR CDR to SIR SIR to PSR PSR to Launch
Schedule Phase
Average All Causes Average NASA External Removed
11
Average Project External Removed Current FSR Guidelines
 Results of Historical Mission Examination
FSR Rates - All Missions
50.0
On Schedule Missions
45.0 -Green line: average of the 8 missions
Weeks per year between Milestone

launched on schedule 42.7

40.0
-The FSR they budgeted was approximately
the actual required FSR
35.0
33.9

30.0 28.4

25.0 24.3

20.0 20.5
19.2
14.9
15.0 14.7 13.0
13.5 12.5
10.6 12.7 12.0
10.0 10.5 8.8

7.1 8.0
5.0
4.0 4.0
0.0
PDR to SIR CDR to SIR SIR to PSR PSR to Launch
Schedule Phase
Average All Causes Average NASA External Removed Average Project External Removed
12
Current FSR Guidelines Avg. Launched On Schedule
Results of Historical Mission Examination
FSR Rates - All Missions
50.0

45.0
Weeks per year between Milestone

PDR and CDR to SIR 4

40.0 -Minimal usage of FSR prior to CDR

35.0 -Increase in required FSR at CDR is partly

an artifact of non-usage from PDR to CDR
30.0 -PDR to SIR: 2 years; 8 wks FSR = 4 wks/yr
-CDR to SIR: 1 year; 8 wks FSR = 8 wks/yr
25.0
-Most missions had modest schedule growth
20.5 between PDR and SIR
20.0 192
14.9 -Missions that had growth post CDR were
15.0 14.7 1w
3.0
e
13.5
mostly attributable to instruments
12.5
10.6 12.7 12.0
10.0 10.5 8.8

7.1 8.0
5.0
4.0 4.0
0.0
PDR to SIR CDR to SIR SIR to PSR PSR to Launch
Schedule Phase
Average All Causes Average NASA External Removed Average Project External Removed
Current FSR Guidelines Avg. Launched On Schedule
13
Results of Historical Mission Examination
FSR Rates - All Missions
50.0
SIR to PSR
-Even with external events removed the I&T
45.0 phase is the most prominent with largest
Weeks per year between Milestone

42.7
variance from guidelines
40.0
- Expected as the most intense development
35.0 effort takes place in this phase 33.9

30.0 -Values near the guidelines were sufficient 28.4

for missions launched on schedule
25.0 24.3
-Nearly all missions launched on schedule
dul
20.0 also had constrained launch windows 20.5 19.2
14.9
15.0 14.7 13.0
13.5 12.5
10.6 12.7 12.0
10.0 10.5 8.8

7.1 8.0
5.0
4.0 4.0
0.0
PDR to SIR CDR to SIR SIR to PSR PSR to Launch
Schedule Phase
Average All Causes Average NASA External Removed Average Project External Removed
Current FSR Guidelines Avg. Launched On Schedule
14
Results of Historical Mission Examination
FSR Rates - All Missions
50.0
PSR to Launch
45.0 -External events such as launch vehicle
Weeks per year between Milestone

technical delays & range conflicts drive the 42.7

40.0 needs for FSR from ship to launch

35.0 - When these events are removed the

33.9
guidelines appear adequate for a nominal
30.0 launch processing phase
28.4

25.0 24.3

20.0 20.5
19.2
14.9
15.0 14.7 13.0
13.5 12.5
10.6 12.7 12.0
10.0 10.5 8.8

7.1 8.0
5.0
4.0 4.0
0.0
PDR to SIR CDR to SIR SIR to PSR PSR to Launch
Schedule Phase
Average All Causes Average NASA External Removed Average Project External Removed
Current FSR Guidelines Avg. Launched On Schedule
15
Further Analysis
• Examination of the average can be useful to asses overall trends
– However, there is much variation present in the schedule data
– Some missions were affected by “extraordinary” events which could not be planned for
– Data points to the far right can unfairly influence the average

• Another approach is to examine a cumulative distribution of the data to identify

the most probable outcomes
– Distributions were examined for potential FSR guidance consistent with NASA policy
• The 50% confidence level on the curve with Project External events removed is
suggested for Project held reserves
• The 70% confidence level on the curve with NASA External events removed is
suggested for HQ/Center held reserves

• The following slides show the cumulative distributions at the guideline specified
milestones (PDR, SIR, PSR) with the 50% and 70% levels highlighted

16
Required Schedule Reserve at PDR
Guideline: 4 weeks/year

Required FSR @ PDR

100%
4
70% w/o
90%
NASA
80%
External
7.1
50% w/o
70%
Project
10.6
External
Cumulative Probability

60% 13.5
7.2
50%

40%
Current FSR Guideline

30% All Causes

NASA External Removed

20%
Project External Removed
7 15
10% Launched On Schedule

4
0%
0 5 10 15 20 25 30 35 40 45 50
Weeks Per 17
Year
Required Schedule Reserve at SIR
Guideline: 8 weeks/year

Required FSR @ SIR

100% 8
70% w/o
90% NASA
External 24.3
80%
28.4
50% w/o
70%
Project 8.8
External
Cumulative Probability

60%
33.9
50%

40%
Current FSR Guideline

30% All Causes

NASA External Removed

20%
Project External Removed
13 18 Launched On Schedule
10%

0% 8
0 5 10 15 20 25 30 35 40 45 50
Weeks Per 18
Year
Required Schedule Reserve at PSR
Guideline: 12 weeks/year
Required FSR @ PSR
100% 12

70% w/o
90%
NASA
80%
External
42.7
19.2
50% w/o
70%
Project
External 11.8
Cumulative Probability

60%
13.4
50%

40%
Current FSR Guideline

30% All Causes

NASA External Removed

20%
Project External Removed

10%
12 17 Launched On Schedule

0% 12
0 5 10 15 20 25 30 35 40 45 50
19
Weeks Per Year
Outline

• Introduction/Background

• Methodology

• Results from PDR, CDR, SIR, and PSR

• Recommendations & Summary

20
Recommendations
• Examination of the cumulative distributions of historical data yielded the following
recommended values of FSR given the example project schedule
Phase ATP to PDR PDR to SIR SIR to PSR PSR to Total Total
Launch (Years) (months)
Example Duration (years) 0.75 2.0 1.0 0.25 4.0 48

Current Guidelines
FSR Rate (wks/year) 0 4 8 12 Total Total
(Weeks) (months)
Total FSR (weeks) 0 8 8 3 19 4

Recommended Project FSR at 50% confidence level

FSR Rate (wks/year) 0 7 13 12 Total Total
(Weeks) (months)
Total FSR (weeks) 0 14 13 3 30 7

Recommended Center/HQ FSR at 70% confidence level

FSR Rate (wks/year) 0 15 18 17 Total Total
(Weeks) (months)
Total FSR (weeks) 0 30 18 4.25 52.25 12

FSR above Proj. (wks) 0 16 5 1.25 22.25 5

21
Summary
• This study aimed to assess if the adequacy of the current guidelines for
carrying funded schedule reserve based on schedule-to-go

• Historical data from 37 missions were assessed to determine required FSR

over the project lifecycles
– Average weeks per year assessed for overall trends
– Cumulative distributions examined to establish recommendations for specific
confidence levels

• Recommended FSR rates are listed below

Phase PDR to SIR SIR to PSR PSR to Launch

Current Guidelines 4 8 12

Recommended Project FSR

7 13 12
at 50% confidence (weeks per year)
Recommended Additional Center/Headquarters FSR
8 5 5
at 70% confidence (weeks per year)
Recommended Total Project + Center/Headquarters FSR
15 18 17
at 70% confidence (weeks per year)

22