Professional Documents
Culture Documents
Measurable News Fall 2008 FINAL102208
Measurable News Fall 2008 FINAL102208
Abstract
The development of the Earned Schedule (ES) method by Lipke in
Did you hear? 2003 has been shown to be an important extension to the Earned
Value Management (EVM) method, increasing the utility of EVM data
The Measurable News for project schedule analysis, control and oversight. As ES provides
is now accepting a reliable time-based indicator of Schedule Performance, the objec-
sponsorship for tive of this paper is to investigate whether the SPI(t) exhibited similar
future issues. stability characteristics to those extensively reported for the Cost
With sponsorship, Performance Index (CPI) in EVM. This paper analyzes EVM data
you receive from three different countries for projects in three industry segments.
Thirty-seven projects were examined for SPI(t) stability and 26 for
• 4-color, 1/3-page ad on CPI stability. It has been found that while the behavior of SPI(t) is
front cover
broadly consistent with CPI, the widely reported CPI stability rule
• 4-color, full-page ad on cannot be generalized even within the US Defense Department (US
back cover DoD) project portfolio. Further research is required to develop im-
• Priority publication of proved understanding of project performance characteristics and the
two articles from profes- behavior of CPI and the SPI(t).
T
sionals in your organ-
ization he cancellation of the US Navy’s A-12 Avenger II stealth aircraft
program in January 1991 [1] [2] resulted in research during the
Too good to 1990s, which investigated the reliability of Earned Value Manage-
ment (EVM) cost prediction and the behavior of the Cost Perfor-
be true? mance Index (CPI)1 using US Defense Department (US DoD) project 2
data. These research findings have come to be regarded as generally ap-
Find out more!
Contact Managing Director *Reprint from Crosstalk, Journal of Defense Software Engineering, April 2008.
Gaile Argiro **The contents of this paper are the author’s personal views and conclusions which do not
reflect an endorsed position of the PMI College of Performance Management.
703.370.7885
1
Unless otherwise stated, all references to CPI and the SPI(t) refer to the cumulative values.
ExecAdmin@pmi-cpm.org
2
”Project” has been used consistently throughout this paper. In US government, particularly
the US DoD context, “program” may be the more appropriate term.
Continued on page 17.
Deltek wInsight:
• Saves you time by providing a common database for analysis and reporting on all projects across your firm, at the click
of a button
• Integrates with a wide range of project management tools including Deltek Cobra®, Deltek Open Plan®, Microsoft Project®,
Primavera®, MPM® and many more
©2008 Deltek, Inc. All rights reserved. All referenced trademarks are the property of their respective owners
Fall 2008, Issue 4
The Measurable News 3
Managing Director
Gaile Argiro
Integrating Systems Engineering with Earned
Story Editor
Value Management
Peter Schwarz By Paul J. Solomon, PMP.................................................................... 7
Designer
Amanda Mitchell
Advertising
Advertising inquiries, submissions, and pay-
ments (check or money order made payable
to the College of Performance Manage-
ment) should be sent to CPM headquarters.
Advertising rates are $1000 (taken) for inside
front or back cover (full-page ad only), $800
for other full-page ads, $500 for half-page ads,
Advertising Rates
and $300 for quarter-page ads. Issue sponsor-
ships are available at $2500 per issue. Business Inside front/back cover..................$ 1000
card ads are available for $100 per issues (or Full-page ad only — taken next issue
free with full-page ad). Rates are good from
January 1, 2008 – December 31, 2008. CPM Full page........................................... $ 800
reserves the right to refuse publication of any Size: 7.5 x 10 inches
ad for any reason.
Half page......................................... $ 500
Subscriptions Size: 5 x 7 inches or 10 x 3.5 inches
All CPM publications are produced as a ben-
efit for CPM members. All change of address Quarter page................................... $ 300
or membership inquiries should be directed Size: 5 x 3.5 inches
to: PMI Headquarters, Customer Care Depart-
ment, Four Campus Boulevard, Newtown Sponsor an issue............................. $2500
Square, PA 19073-3299 • Phone 610.356.4600 • includes ads on front and back cover
Fax 610.356.4647 • customercare@pmi.org.
All articles and letters represent the view of Business Card Ad.............................. $100
the authors and not necessarily those of CPM. Size: 2.3 x 1.3 inches
Advertising content does not signify endorse-
ment by CPM. Please notify CPM for single copy Contact Managing Director Gaile Argiro at
or reproduction requests. Appropriate charges
will apply.
PMI-CPM Headquarters for more information:
phone 703.370.7885 • ExecAdmin@pmi-cpm.org
© 2008 by the College of Performance Management. All rights reserved.
4
The Measurable News Fall 2008, Issue 4
Past President
Neil Albert
703.506.4600
PastPresident@pmi-cpm.org
Register now for …
IPM 2008
20th Annual International
Integrated Program Management Conference
November 17–19, 2008
Hilton Alexandria Mark Center, Alexandria, VA
Call 703.845.1010 and ask for “IPM Conference rate”
Co-sponsored
Co-sponsored by…
by…
How you finish
depends on
how you start!
P RO J E C T M A N AG E M E N T S E RV I C E S A N D E D U C AT I O N
• Operational Support
• In-house Training
w w w . p m a s s o c . c o m
1 . 5 6 1 . 6 9 4 . 1 6 4 7 CHARTING PROJECT SUCCESS
Fall 2008, Issue 4
The Measurable News 7
E
summarized in the following tables.
VM is capable of integrating a project’s Table 1, Defense Acquisitions, includes recom-
cost, schedule, and technical performance. mendations to hold contractors accountable for
However, an organization may comply with achieving desired outcomes. Table 2, Effective
GEIA EIA-748-B, the EVMS standard, yet Implementation of EVM, includes findings that
fail to integrate the project’s technical objectives agencies are not measuring actual vs. expected per-
into the Performance Measurement Baseline (PMB). formance in meeting performance goals. Table 3,
EVM data will be reliable and accurate only if the Best Practices, includes demonstrating that the de-
right base measures of technical performance are se- sign meets performance requirements.
lected and if progress is objectively assessed. The Office of Management and Budget (OMB)
Guidance and best practices for planning and Circular A-11 specifies that the EVMS must measure
measuring technical performance are provided by progress towards milestones, in terms of capability
SE and project management standards. of the investment to meet specified requirements and
EVM will provide more reliable information for quality.
analysis and decision-making if the EVMS guide-
lines are augmented by guidance regarding main- DOD Acquisition Transformation
taining the technical baseline, measuring technical The DOD issued acquisition policy and guides that
performance, and managing risk. specify measurement of technical performance
along with cost and schedule performance. The
GAO Procurement Findings and DOD guides that will enable implementation of the
Recommendations policy are based primarily on SE standards. In fact,
The Government Accountability Office (GAO) studied DOD states that revitalizing SE is a component of its
failures in acquisition of weapons systems and Infor- DOD’s Defense Acquisition Transformation Report
mation Technology (IT) systems. One report concluded to Congress.
that, if EVM is not properly implemented, the data may
the product scope. In comparison, the SE stan- states that the PMB may include technical and
dards and models address the product require- quality parameters.
ments and require assessment of progress against 2. EVMS encourages but does not require precise,
requirements, technical performance, design quantifiable measures. It states that objective
maturity, and the quality of the product being earned value methods are preferred but it also
developed. The Project Management Institute states that management assessment (subjective)
(PMI) standard, Project Management Body of may be used to determine the percentage of work
Knowledge (PMBOK® Guide) differentiates two completed. In contrast, the other standards speci-
components of scope, product scope and project fy objective measurement.
(work) scope. The PMBOK Guide also includes 3. EVMS is described as a risk management tool.
the quality baseline within the product scope and However, it contains no guidelines on risk man-
agement.
Earned value, when possible, should be based on the of the enabling work products. Successful peer re-
sum of two measures: views or testing are often used to determine the com-
• Progress towards completing the set of enabling pleteness of enabling work products against a process
work products, such as drawings or code. quality standard. Later, measurements are taken of the
• Progress towards meeting the product require- capability of the design to meet the requirements.
ments, including technical performance or Quality. Of primary importance is the performance to-
wards achieving a combination of:
Product Requirements and Plan • Schedule objectives for enabling work products
The product scope (also called the technical or qual- that meet process quality objectives and
ity baseline) is bounded by product requirements. The • Event-driven success criteria when the event is
technical baseline should be integrated into the PMB meeting a measurable product requirement.
along with work scope, schedule, and cost objectives. These should also become the criteria for taking
The CMMI defines requirements traceability as 100 percent of earned value in work packages.
the evidence of an association between a require-
ment and its source requirement, its implementa- Risk Management
tion, and its verification. Requirements traceability The most straightforward guidance for integrat-
is maintained from a requirement to its derived ing risk management with EVM is in the PMBOK®
requirements as well as to its allocation to func- Guide. First, new risk management activities should
tions, objects, people, processes, and work products. be planned, budgeted, and scheduled if they consume
Traceability should also be maintained from the resources. It is important to add the risk response
work products to the plan, including the schedule, activity to the master schedule and to establish a logi-
work packages, and planning packages. cal link between the risk response activity and its
dependent activity on the schedule. By adding the risk
Measure Progress
response activity to the PMB, management will use
The validity of schedule and cost performance in- earned value to monitor its progress.
formation depends on the accurate measurement of Second, the EAC should be based on both project
progress towards meeting the product requirements. performance and quantified risk assessments. This
However, progress towards meeting the product guideline may be applied at all WBS levels. At the
requirements may not be measurable early in devel- project level, EAC should be based on the project
opment. For example, a hardware or software com- manager’s assessment of overall project perfor-
ponent may require the completion and assembly of mance, including technical performance, and quanti-
many enabling work products, such as drawings or fication of the impacts of top risks.
coded software modules, before the integrated set of
work products may be tested or analyzed to determine Practical Examples
if it will meet the product requirements. So, until that This section provides practical examples of tying
assembly is completed, early measurements are taken earned value to technical performance:
Table 8: TPM per INCOSE SE Handbook. • Ex 1: Base
EV on
TPM per INCOSE SE Handbook
completing
• TPMs express the objective performance requirements.
drawings and
• Without TPM
meeting TPM
– Project manager could fall into the trap of relying on cost and schedule status alone
– Can lead to a product developed on schedule and within cost that does not requirements
meet all key requirements. • Ex 2: Entry
• Periodic recording of status of each TPM and exit crite-
– Provides continuing verification of degree of anticipated and actual
achievement of technical parameters.
ria for techni-
cal reviews
12
The Measurable News Fall 2008, Issue 4
Ex 1: Base EV on completing drawings Table 10, Net BCWP Based on Component Re-
and meeting TPM requirements quirements, shows the time-phased Budgeted Cost
This example uses technical performance measure- for Work Scheduled (BCWS), earned value (BCWP)
ment (TPM) to assess the capability of the design to based on completed drawings and negative BCWP
meet product requirements. TPM uses actual or pre- when the design fails to meet requirements.
dicted values from engineering measurements, tests, Schedule variance analysis should state that al-
experiments, models, simulations, or prototypes and though the drawings are ahead of schedule (+40), the
includes milestones for meeting the requirements. design has not met the planned requirements (-100),
The output of a work package is the design of a resulting in a net SV of -60. There will be an unfa-
component of a subsystem, a set of wire harnesses. vorable impact to both the cost and schedule objec-
There are two TPM requirements that are allocated tives as the drawings are reworked until the design
to the wire harnesses: meets the requirements.
• Maximum weight: 200 lb. In Ex 1, TPMs were used to determine if the
• Maximum diameter: 1 inch weight and diameter requirements were met at the
There are two measures of progress, completing component or work package level. However, perfor-
the drawings and meeting the requirements. The mance requirements are usually established at higher
schedule is shown in Table 9: WBS levels. If the design of a component is at the
• The budget is allocated as follows: work package level, completion of the design may
• Budget at Completion (BAC): 2000 hours depend on achieving planned TPMs values or other
• Budget per drawing: 40 hours quality objectives at the higher level. In this case,
• EV penalty if requirements not met: –300 hours earned value for a component level work package
During the early periods, EV is based on tradi- should be dependent on meeting both the component
tional measures, completion of the enabling work and higher level objectives.
products (drawings). Later, when there is a milestone Ex 2: Entry and exit criteria for technical
to measure and achieve the technical performance reviews
requirement, there is an EV penalty (negative earned An important control for ensuring integration of a
value) if the requirement is not met. EV is restored project’s technical and schedule objectives is to es-
when the requirement is finally met. The total possi- tablish milestones for the end of a stage of develop-
ble negative earned value is –300 hours, as follows: ment with the right exit or success criteria. These
• Component weight requirement (req.) not met: exit criteria should also be used for the work pack-
–100 hours ages that support that milestone. This example pro-
• Diameter req. not met: –200 hours vides guidance for defining the right exit criteria as
The schedule status at April month end follows: well as entry criteria.
• Cumulative (cum.) drawings completed: 41 Per IEEE 1220, 6.8.1.5.d, technical reviews are
• Diameter req. met conducted to:
• Component weight req. not met • Assure that all master schedule success criteria
have been met
• Assess development maturity
Table 9: Schedule for Drawings and Requirements. to date
Schedule Plan Jan. Feb. Mar. Apr. May Total • Assess the product’s ability to
Drawings 8 10 12 10 10 50 satisfy requirements
• Assure traceability of require-
Requirements met:
ments and validity of decisions
Weight 1 1 Exit criteria for preliminary de-
Diameter 1 1 sign reviews (PDR) and
critical or final design reviews
(CDR) are also in IEEE 1220.
Fall 2008, Issue 4
The Measurable News 13
Table 10: Net BCWP Based on Component Requirements. EACs to variances from the plans
Design (drawings) Jan. Feb. Mar. Apr. May Total for achieving all success criteria
for technical reviews and TPM
Planned drawings 8 10 12 10 10 50
variances.
BCWS – cur. 320 400 480 400 400 2000
BCWS – cum. 320 720 1200 1600 2000 2000 Confirm Integration of
Actual drawings
Technical Performance
completed 9 10 10 12 with EVM in IBR
BCWP (drawings) – cur. 360 400 400 480 The DOD guide (Integ SE) em-
phasizes reviewing the following
BCWP (drawings) – cum. 360 760 1160 1640
during an IBR:
Negative BCWP (reqs) – • Plans for event-based technical
cum. –100
reviews including entry and exit
Net BCWP (drawings criteria and independent subject
and reqs) 1540 matter expert participation
Schedule Variance (SV) 40 40 –40 –60 • Technical tasks and products
resulting from the IMS tasks
Some of the exit criteria for a system PDR are: • Correlation of the technical metrics and mea-
• Prior completion of subsystem reviews sures, IMP/IMS, EVMS
• Total system approach to detailed design satisfies For example, the IMS should include the criteria
the system baseline for completing technical reviews and milestones for
• Unacceptable risks are mitigated measuring technical performance as well as the TPM
• Issues for all subsystems, products, and life cycle planned values to be achieved at that milestone. The
processes are resolved IMS and work packages should also include comple-
• Trade-study data are adequate to substantiate that tion of the technical baselines.
subsystem and system requirements are achievable
Manage Requirements
Better Acquisition Management
The following guidance for managing requirements
Finally, a customer can manage the acquisition of
is derived from the CMMI-Acquisition Module
a new system more effectively if it ensures that the
(CMMI-AM)
supplier’s management process and reports integrate
1. Identify inconsistencies between requirements,
technical performance, with schedule and cost perfor-
project plans and work products.
mance and focuses on the requirements and quality.
1.3 Manage changes to the requirements.
Include Performance-Based Each change to a controlled requirement should
Progress in Contract be assessed for impact to the project perfor-
The customer should initiate performance-based mance, cost, and schedule baselines. These
acquisition management by including requirements baselines should be changed, as required, to ac-
for achieving technical performance objectives in its commodate the requirements change.
contracts, beginning with the solicitation. 1.4 Maintain bidirectional traceability among the
The customer can obtain insight into earned value requirements and the project plans and work
that is based on technical performance by requiring products.
the supplier to define milestones that meet desired 1.5 Identify inconsistencies between the project
performance outcomes such as: plans and work products and the requirements.
• Success criteria for major technical reviews.
• TPM planned values and measurement milestones. Award Fee
Also, EVM variance analysis reports should relate A second component of Defense Acquisition Trans-
schedule variances, schedule recovery plans and formation is Award Fee and Incentives Policy. In April
14
The Measurable News Fall 2008, Issue 4
PMI-CPM EVM Education Program has the right subject-matter experts and training capabilities to help your or-
ganization meet program training objectives and improve cost and schedule performance. PMI-CPM’s core EVM
education training capabilities encompass the full spectrum of program management:
• Practice Symposia: Lessons learned and best practices on the implementation of Earned Value
Management, updated on recent developments and integrating EVM with other management tech-
niques.
• Workshops: Workshops provide a forum to introduce and develop new and emerging practices
and techniques to improve program management using EVM, objectives are to identify potential so-
lutions. Workshops are not intended to provide training or instruction on EVM.
20th Annual International Integrated Program Management Conference • November 17–19, 2008
Hilton Alexandria Mark Center, Alexandria, VA
Joe Houser • VP Professional Education Training Seminar • KMSystems Group • jrhouser@kmsystemsgroup.com
Bill Chitty • Education Communications Coordinator • BAE Systems • bill.chitty@baesystems.com
PROJECTMANAGEMENT
FORTHE
N E X T G E N E R AT I O N
WHAT HAS YOUR PROJECT MANAGEMENT SYSTEM DONE FOR YOU TODAY?
Too many software companies claim to be all things to all people by piecing together separate modules that provide
the desired functionality with integration as an afterthought. SAFRAN provides integrated performance on the most
widely accepted and industry tested technology.
S AFRAN P ROJECT AND S AFRAN FOR M ICROSOFT P ROJECT
Project management software from Safran provides organizations with superior visibility and insight into how time and
resources are spent and into the overall health of their projects. Our solutions provide all of the power and functionality you
need out of the box based on your industry and business needs, integrated from the ground up; not after the fact.
Implementing Safran project management software decreased tangible and intangible project costs, increased
efficiency almost 100% in day-to-day project planning and control, and reduced bid preparation times 75% -
providing our customers with the competitive edge.
From the individual project office to the enterprise, Safran provides the most up-to-date and powerful EPM solution today.
SAFRAN’S“ONESOLUTION”APPROACHFORPROJECTͲCENTRICORGANIZATIONS:
SAFRANPROJECTlSAFRANFORMICROSOFTPROJECTlSAFRANPLANNERlSAFRANSCORECARD
ENTERPRISEPROJECTMANAGEMENTSOFTWAREANDSERVICESFOR:
ENERGYANDPOWERlA&E/CONSTRUCTIONlAEROSPACEANDDEFENSE
N ORTH A MERICA
GOVERNMENTlHIGHTECHlANSI748COMPLIANCE
simple | flexible | powerful
IPM 2008
Delivering A New “Steady-State”: Portfolios, Programs and Projects
20th Annual International Integrated Program Management Conference
November 17–19, 2008
Hilton Alexandria Mark Center, Alexandria, VA
Registration: Early $700 • Regular $750
Hotel Rate: $143 (phone 703.845.1010)
http://www.pmi-cpm.org/pages/events/IPM08/conf_program.html
V e cost
n anddperformance
o r
With MCR Program Management,
aligning / S e r v i c es • Project Management Process Design
For information
If your program is over budget but
under performing, MCR can help. to advertise, contact Managing Director
• IndependentGaile Argiro at
Data Analysis
703.370.7885 or ExecAdmin@pmi-cpm.org.
With the right tools, proven processes,
and subject-matter experts you need, • Integrated Baseline Reviews
MCR has the capabilities to meet your
objectives and improve cost
effectiveness and performance. • Information Systems Integration
Our Core Capabilities encompass the
full spectrum of program management, • Software Evaluations & Training
from start up to sustainment, including
Your ad
could go
w w w . p m a s s o c . c o m
www.mcri.com
30 Years of Program/
1 . 5 6 1 . 6 9 4 . 1 6 4 7 CHARTING PROJECT SUCCESS
www.pmassoc.com
Acquisition Management
here!
MCR, LLC
2010 Corporate Ridge • Suite 350 • McLean, VA 22102
see our ad onExperience.
703/506-4600 • Fax 703/506-8601
www.mcri.com p. 31 see our ad on p. 6
Your ad Your ad
could go could go
www.systalex.com
here! here!
V e n d o r / S e r v i c es
Fall 2008, Issue 4
The Measurable News 19
Continued from page 17. reported for the Cost Performance Index. The objec-
possible because the SPI is known to fail as a statistical tive of this research paper is to reexamine Cost Per-
predictor because it always returns to unity at project formance Index stability and to compare the stability
completion irrespective of duration based delay. The behavior of the SPI(t) with CPI.
SPI is also recognized as failing, nominally within the
Method for Evaluating Stability
final third of the project and also fails after the project’s
Planned Duration has been exceeded. EVM project data was loaded into a Microsoft Excel
Lipke proposed the ES method in 2003 as a solution “Stability Point Calculator” developed by Lipke. The
to these limitations and flaws of the EVM schedule in- calculator determines the observation number in a
dicators [3]. A series of studies provided initial valida- sequence of CPI and SPI(t) values at which all sub-
tion of the ES method, some by using real EVM project sequent observations are within a defined stability
data, Henderson [12] [13] [14] and Vandevoorde and limit. The stability limit used is .10. The calculator
Vanhoucke [15] and also by using simulated network enables the associated percentage complete at which
schedules, Vanhoucke and Vandevoorde [16]. The time stability occurs to be determined.
based ES derived SPI(t) has been shown to be reliable This calculator has been placed into the public
for both early and late finish projects. The SPI(t) only domain to encourage more broadly based CPI and
reverts to unity at project completion if on time com- SPI(t) stability research and is freely available from
pletion has been achieved. the ES website at http://www.earnedschedule.com/
A research study intended to validate the ES con- Calculator.shtml.
struct using DAES data was commissioned in 2004 and To determine the significance of the observations
undertaken by a US Air Force Institute of Technology of stability for both CPI and SPI(t), statistical hy-
Masters student. Unfortunately, this study was discon- pothesis testing is conducted. The test applied is the
tinued after an independent review determined: Sign Test at 0.05 level of significance3 [19]. The Sign
Test was used in this research because it does not
Results: The historical data collection procedures
depend upon the data having a normal distribution.
for the DOD and USAF do not allow for sufficient
In past research, the hypothesis test method chosen
testing of ES theory at this time. A statistical evalu-
implied that the data was normally distributed; how-
ation concluded that SPI(t) is different than SPI($);
ever, the normality of the data was not established.
however, the two variables are highly correlated.
Research by Lipke also suggests that:
The result of the analysis identified that SPI(t) per-
Results indicate the logarithm data representations of
forms similarly to SPI($) with the data contained in
the indexes are likely normally distributed, whereas the
the DAES database. In order for the ES Theory to
distributions for CPI, SPI, and CV are not. [20]
be fully investigated, additional data must be col-
The question to answer regarding stability is “Can
lected. This research shows that the necessary data
it be stated generally and reliably that the final value
may also not be available despite the best collection
of the performance index is within 0.10 of its value
efforts. The original schedule and planned duration
when the project is 20 percent complete?” The an-
information is critical to successful evaluation of the
swer to the question will be “yes” if the alternate
ES methodology. [17]
hypothesis is satisfied:
However, early interest by the Project Management
H1(CPI): |CPI(final) – CPI(20%)| < 0.10
Institute College of Performance Management resulted
H2 (SPI(t)): |SPI(t)(final) – SPI(t)(20%)| < 0.10
in the principles of ES being included as an “Emerg-
Two separate hypothesis tests are conducted, one
ing Practice Insert” in the Practice Standard for Earned
for CPI and one for the SPI(t). The result from the
Value Management published in 2004. [18]
hypothesis testing is recorded as Ha when the value
Following the initial validation of ES, interest de-
of the test statistic is in the critical region (0.05) and
veloped in ascertaining whether the SPI(t) exhibited
Ho (null hypothesis) when it is not.
similar stability characteristics to those extensively
3
Applying the Sign Test at the 0.05 level of significance means that the test is being applied at a 95% level of confidence.
20
The Measurable News Fall 2008, Issue 4
project set and for the composite of all. It can be Additional Analysis
seen that the majority of projects reach stability only Following the lack of CPI and SPI(t) stability find-
after the 20% completion point. ings additional analysis was conducted. Within each
Figure 1 summarizes within each 10 percent com- 10% complete percentile bands projects were catego-
plete percentile band where CPI and the SPI(t) sta- rized as follows:
bility occurred. This figure shows: • Cost at completion:
• The wide variability in the achievement of stabil- o Under or On Budget (UOB)
ity for both CPI and the SPI(t). o Over Budget (OvB).
Project performance heuristics or “rules of • Schedule at completion:
thumb” intended to be generally applicable (e.g., o Early or On Time finish (EOT)
the CPI stability rule) require an empirically es- o Late Finish (LF).
tablished consistency of behavior across a broad The purpose of this analysis is to determine if there
range of projects. These findings are a significant is a correlation between achieving earlier CPI and the
impediment to proposing and confirming broadly SPI(t) stability and improved project outcomes.
applicable CPI and SPI(t) stability heuristics. Figure 2 summarizes the analysis for CPI and
• That stability is usually achieved very late in the Figure 3 does the same for the SPI(t). With the data
project lifecycle, often later than 80% complete samples utilized, achievement of earlier stability is
for projects in these samples. not correlated with improved final cost and/or sched-
Zwikael et al. analyzed the Israeli hi‑tech project ule outcomes.
sample using visual inspection of charts and suggest- For UOB and EOT projects where cost and sched-
ed that CPI stability was, on average, achieved at the ule stability was achieved late (after say 60% com-
60% completion point [21]. That analysis broadly pletion) achieving earlier stability would have been
confirms this paper’s finding of CPI stability being disadvantageous to the final outcome(s) achieved
achieved much later in the project lifecyle than pre- because project performance progressively improved
viously reported. over the life of those projects.
!
Figure 1: Total Projects CPI and SPI(t) Stability Within Each 10 Percentile Band.
22
The Measurable News Fall 2008, Issue 4
!
Figure 2: Project Completion Categories by CPI Stability Band.
!
Figure 3: Project Completion Categories by SPI(t) Stability Band.
Fall 2008, Issue 4
The Measurable News 23
Figure 4 summarizes projects (with the required domain on the PMI Sydney Chapter website. [22]
comparative data), which achieved SPI(t) or CPI The purpose of the Popp study was to develop prob-
stability first. Achieving SPI(t) stability first implies ability distributions of cost Estimates at Complete
schedule management had a higher management pri- (EACs) based on the CPI at complete, current CPI
ority, achieving CPI stability first implies cost man- and percentage complete of projects based on his-
agement had the higher priority. tory. As stated in the report:
In the Australian IT projects sample, SPI(t) sta- Given a program has a CPI of X and a percent com-
bility was achieved first for the preponderance of plete of Y, what is the most likely finishing CPI. [22]
projects. For the other data samples the achievement In contrast to Christensen and associates research,
of cost or schedule stability first occurred in rough- which used data from the DAES database, the data
ly equal proportion. In only one project in these used by Popp was sourced from the Contracts Analy-
samples, an Australian IT project, was the cost and sis System (CAS) database maintained by the Office
schedule stability achieved simultaneously. of the Secretary of Defense Cost Analysis Improve-
ment Group (CAIG).4
Corroboration with Other Research The research undertaken by Popp did not focus
Because of the comprehensive contradiction to the on CPI stability. However, charts which can also be
previously published CPI stability research findings, used for assessing CPI stability were completed as
a further literature review was undertaken. This part of that study. These charts correlate the cumula-
review obtained a most unexpected source of inde- tive CPI for the percentage complete in each 10%
pendent corroboration for this paper’s CPI stability complete percentile band to the CPI Final for all
findings. In the mid 1990s Mr. Michael Popp initi- projects in that sample.
ated an internal US DoD research project within the Figure 5 is the first chart of interest from the Popp
US Naval Air Systems Command (NAVAIR). report, as it shows the correlation between the cumu-
The output was an internal but unclassified NA- lative CPI at 10–20% complete and the CPI Final for
VAIR report (the Popp report) which has, with Mr. all projects in the sample.
Popp’s permission, now been placed into the public
was not achieved. This paper has found that while prediction techniques and the supporting rationales
the behavior of the SPI(t) is broadly consistent with is “pending publication” as at 12th September 2007
CPI, the widely reported CPI stability rule cannot be [24]. The statistical prediction techniques developed
generalized to all projects using the EVM method or have been summarized in a presentation by Hender-
even within the US DoD project portfolio. However, son [25] which is available on the website.
the consistent behavior to CPI demonstrated by the A major advance to EVM practice and future
SPI(t) provides further support for the validity of the research opportunities would be development of a
SPI(t) metric and the ES method. broadly based EVM research database where com-
Additional analysis was unable to establish a pleted EVM project data could be submitted anony-
correlation between achieving earlier CPI and the mously for:
SPI(t) stability and improved outcomes at comple- • Research purposes
tion. In cases where projects achieved either under • Benchmarking completed project performance
budget and/or early finish outcomes with cost and/ • Assisting in the sizing of projects.
or schedule stability achieved late (ie. after say 60% Such knowledge bases are not unique in other disci-
completion), earlier cost and/or schedule stability plines, with an instructive Australian example being the
would have been disadvantageous to the actual final International Software Benchmarking Standards Group
outcome(s) achieved. This is because CPI and/or the (ISBSG), website at www.isbsg.org.
SPI(t) were progressively improving over the life of Improved data collection techniques to ensure that
those projects. baseline schedule information is captured and stored
The findings and corroboration of this paper require in the DAES database are also recommended.
significant review and revision to what has been re-
garded as a long settled EVM heuristic with regard Concluding Remarks and Future
to CPI stability and consequent practice including the Research
use of a stable CPI as evidence that an EVM system is While this paper has overturned long-standing find-
functioning properly and of a “reliable” EAC. [5] ings and belief on CPI stability, it is important that
Improvements to current EVM techniques for the strengths and limitations of the EVM method are
predicting future cost performance should be con- properly understood, particularly in an era of:
sidered as current techniques have relied on general- • Adoption of EVM by US Government agencies
izing research findings from limited data sources, through Office of Management Budget Circular
principally the DAES database. A-11 Part 7 mandate
Alternatives methods of cost and schedule predic- • Advocacy of the use of EVM cost predictors to
tion using well-established statistical principles and assess compliance to the Sarbanes Oxley Act [9]
methods developed by Lipke show promise as: • Increased interest and the adoption of EVM by
• These techniques allow generation of a range of organizations globally.
cost and schedule predictions from user defined Where projects have not exhibited “CPI stability”
Confidence Limit(s) EVM practitioners can now know that this is neither
• All information and data required for these pre- unique, nor is it necessarily an adverse reflection on
dictions comes from within the project itself. the management or execution of those projects.
This may reduce the current dependence on Various follow‑on research opportunities arise from
heuristics developed from external project data this paper, which may develop improved understanding
sources, which might not be applicable to the of project performance characteristics and generalisable
project of interest. heuristics. Suggestions include examining the perfor-
To promote trials of these statistical prediction mance characteristics of projects where:
techniques, a Microsoft Excel “Statistical Prediction The CPI stability rule does seem applicable ( e.g.,
Calculator” is also freely available from the ES web- the subset highlighted in the Popp report data) to
site at www.earnedschedule.com/Calculator.shtml. determine whether there are project characteristics
An academic paper fully describing the statistical which result in early CPI stability
26
The Measurable News Fall 2008, Issue 4
Early CPI stability was not achieved due to pro- Christensen, David S, PhD., “Using the Earned
gressively improving CPI performance over the proj- Value Cost Management Report To Evaluate The
Contractor’s Estimate at Completion”, Acquisition
ect lifecycle. Review Quarterly, Summer 1999: 283:295.
Academically oriented research aimed at estab- http://www.dau.mil/pubs/arq/99arq/chrisevm.pdf
lishing a theoretical rationale for project perfor- (26 Nov 2007)
mance instability would be another useful addition to Lipke, Walt. “Schedule is Different,” The Measurable
the project management body of knowledge. News, 2003, March: 10–15.
While Coleman et. al. [23] provide the sobering http://www.earnedschedule.com/Docs/Schedule%20
assessment that consistent with Christensen’s find- is%20Different.pdf (12 Sep 2007)
ings “average to good programs do not improve”, Stratton, Ray. “Not Your Father’s Earned Value,”
Projects@Work, (www.projectsatwork.com), 2005,
an understanding of project characteristics, which Feb 24
result in progressively improving CPI would, if these http://www.earnedschedule.com/Docs/Not%20Your%20
characteristics could be emulated in other programs, Father%27s%20Earned%20Value.PDF
be an extremely useful advance to practice. Such (14 Nov 2007)
research could offer significant opportunities for tan- Christensen, David S., Payne, Kirk. “Cost Performance
gibly improving project performance. Stability — Fact or Fiction?,” Journal of
Parametrics, (April 1992): 10:27–40.
Research opportunities are equally applicable
http://www.suu.edu/faculty/christensend/evms/
to project schedule performance. This paper also CPIstabilityJP.pdf (12 Sep 2007)
demonstrates that by using ES, research of schedule Christensen, David S. PhD., Templin, Carl. PhD.
performance using EVM data is possible and already “EAC Evaluation Methods: Do They Still Work?,”
leading to improved understanding of the dynamics Acquisition Review Quarterly, Spring 2002: 105:116.
of project schedule and project cost performance. http://www.suu.edu/faculty/christensend/evms/
eacevalmethods4.pdf (12 Sep 2007)
Acknowledgements Christensen, D. S., Heise, S. R. “Cost Performance Index
Stability,” National Contract Management Journal,
This research has been made possible due to the gen- Vol 25, (1993): 7–15.
erous assistance of the following individuals: http://www.suu.edu/faculty/christensend/evms/
• The Project Controls Manager from the United CPIstabilityNCMJ.pdf (12 Sep 2007)
Kingdom based construction company (who Fleming, Quentin., Koppelmann, Joel. The Earned Value
desire anonymity) for making available the UK Body of Knowledge. Proceedings of the 30th Annual
construction projects EVM data Project Management Institute 1999 Seminars and
Symposium. Philadelphia, Pennsylvania, USA
• Mr. Michael Popp of NAVAIR for making avail-
Fleming, Quentin, Koppelmann, Joel. “Sarbanes‑Oxley:
able the “Popp report” and providing permission Does Compliance Require Earned Value Management
for the report to be placed in the public domain on Projects,” Contracts Management, April 2004
on the PMI Sydney Chapter website. 26:28
The support, suggestions, general assistance, and Fleming, Quentin., Koppelmann, Joel. “If EVM is Good
review comments by the ES advocates and research- … Why Isn’t it used on all Projects?,” Contracts
Management, April 2004:26–28
ers, which significantly improved this paper is also
http://www.suu.edu/faculty/christensend/evms/WhyEVM.pdf
appreciated. Responsibility for any errors, omissions (12 Sep 2007)
or erroneous conclusions remains the sole responsi- Fleming, Quentin, & Koppelman, Joel. Earned Value
bility of the authors. Project Management. 3rd ed. Upper Darby, PA:
Project Management Institute, 2005
References Henderson, Kym. “Earned Schedule: A Breakthrough
Beach, Chester Paul Jr. “A-12 Administrative Inquiry. Extension to Earned Value Theory? A Retrospective
Report to the Secretary of Navy” Department of the Analysis of Real Project Data,” The Measurable
Navy, Washington DC, 1990. News, 2003, Summer: 13–23.
http://www.suu.edu/faculty/christensend/evms/beacha-1.pdf http://www.earnedschedule.com/Docs/Earned%20
(12 Sep 2007) Schedule%20-%20A%20Breakthrough%20
Extension%20to%20EVM.pdf (12 Sep 2007)
Fall 2008, Issue 4
The Measurable News 27
Abstract
“Earned Schedule,” touted as a major new extension of the EVM methodology, is neither new
nor profound. Through personal experience and the simple methods developed by Army EVM analysts in
the 1970s, I describe the origins of earned schedule and address errors in recently published literature that
purports to have overturned longstanding findings related to EVM data stability.
T
he Earned Value Management world has been developed an algorithm to approximate schedule
abuzz with papers and presentations touting status based on monthly reported EVM data. While
the concept of “earned schedule” (see www. this was some 30 years ago, I would not be surprised
earnedschedule.com for a description of the were I to learn that someone else had tripped to the
technique). As a past president of PMI-CPM in 2004, obvious connection even earlier, as EVM at that
I became personally involved when I helped forge point already was ten-year-old DoD policy.
a compromise in the PMI “Practice Standard for
Earned Value Management” between earned sched- Earned Schedule Isn’t Profound
ule proponents and skeptics. The standard intro- Emil’s algorithm was simple. We calculated the aver-
duced “Time-Based Schedule Measures” (now called age amount of work planned to have been performed
earned schedule) as “an emerging EVM practice” each week during a given period by dividing the
rather than giving it the endorsement that was sought planned value for the period by the number of weeks
by its more enthusiastic supporters. The compromise in the reporting period (typically four per month —
was meant to allow time for research to demonstrate the number of months used would be based on such
its value. factors as how stable the planned value was over the
Any such compromise involves give and take. I period). We then divided the cumulative schedule
overcame personal misgivings resulting from more variance by the average planned value per week to
than three decades of EVM analysis experience derive the approximate number of weeks the vari-
because I thought earned schedule showed promise ance represented from the plan. From there it was a
based on limited statistical results that conceivably simple step to relate the variance in weeks to the per-
might be demonstrated through more extensive re- cent complete (cumulative earned value divided by
search. That hoped-for result has not been achieved. budget at completion) and estimate the completion
Nevertheless, the advocacy not only continues but date. Our report to the Deputy Commanding General
now includes challenges to established EVM princi- might say, for example, “This contract is about one-
ples. This article addresses three things about earned third complete and about four weeks behind sched-
schedule that I find troubling: First, it’s not new. ule. If there are no improvements in performance, it
Second, it’s not profound. Third, its proponents are will complete at least twelve weeks later than
running roughshod over well-grounded research. planned.”
We couched our analyses carefully because we
Earned Schedule Isn’t New understood the limitations of EVM-based schedule
I learned about earned schedule soon after joining data. Even the name is a misnomer — a schedule
the Army Materiel Command headquarters staff variance is actually an accomplishment variance
as an EVM analyst in the late 1970s. My office that measures any difference in the volume of work
prepared independent assessments of monthly con- performed relative to the plan. If resources in the
tractor cost performance reports for the Command performance measurement baseline are planned
Group. Because we didn’t have access to schedule unrealistically, variance calculations may be mis-
information, the analysis branch chief, Emil Szten, leading. The variance doesn’t necessarily reflect the
Fall 2008, Issue 4
The Measurable News 29
critical path. And it will be zero at project comple- Henderson and Ofer Zwikael titled “Does Project
tion, no matter how late the work actually is com- Performance Stability Exist? A Re-examination of
pleted. These limitations are more than offset by its CPI [Cost Performance Index] and Evaluation of
proven value as an early warning indicator when SPI(t) [an earned schedule metric — Schedule Per-
work is not being accomplished — but as my former formance Index (time)] Stability” (see http://www.
Pentagon boss Gary Christle says, “if you want to stsc.hill.af.mil/crosstalk/2008/04/0804HendersonZ
understand the schedule — look at the schedule, not wikael.html) had as an objective “to reexamine CPI
at EVM data.” stability and to compare the stability behavior of the
That said, the schedule variance is derived from per- SPI(t) with CPI.” The authors’ conclusion that “the
formance against a time-phased resource consumption widely reported CPI stability rule cannot be general-
baseline linked to the schedule. It should be no surprise ized to all projects using the EVM method or even
that the information will correlate tightly with the proj- within the DoD project portfolio” is based on de-
ect schedule… it would be a problem if it did not. But monstrably faulty research.
correlation neither elevates earned schedule to a sci- The definitive research on CPI stability was per-
ence nor improves its management utility beyond the formed by graduate students at the Air Force Insti-
intuitive, simple calculations we did at Army to give tute of Technology under the guidance of Dr. David
the senior leadership a heads-up. Christensen. As the senior EVM analyst in the Office
of the Secretary of Defense in the 1990s I made the
Earned Schedule Proponents Defense Acquisition Executive Summary (DAES)
Misunderstand EVM Research database available to many researchers including
An April 2008 article published in CrossTalk, the Dr. Christensen and his students and sponsored their
Journal of Defense Software Engineering, by Kym research. The DAES database included quarterly
Humphreys & Associates, Inc., is a management consulting firm specializing in Earned Value Project
Management. Our experience encompasses the construction and utility industries, energy programs,
the aerospace industry, and all acquisition branches of the U.S. Dept. of Defense, and foreign govern-
ments. We have consulted and supported over 500 major organizations and trained more than 450,000
individuals. The principles of project management that we espouse are applicable to any project oriented
environment. Some of our specific areas of expertise include: Proposal Preparation, Project Scheduling,
Performance Measurement, Risk Assessment, Software Evaluation, System Audits, Customer Review
Preparations, and Training.
summary EVM data on all major defense contracts it exists on DoD contracts. Whether the oft-quoted
that were subject to oversight by the Office of the “rule” derived from their work is true in other man-
Secretary of Defense. The resulting theses and ar- agement environments is unknown, but the article
ticles constitute an unparalleled body of knowledge does not make a case for refuting it “within the DoD
grounded in actual, verifiable performance data from project portfolio.”
hundreds of defense contracts for weapon systems, On the other hand, given the fifteen years since
vehicles, ships, airplanes, information technology Christensen’s work was published, one might hy-
and more. pothesize that defense project managers and their
Henderson and Zwikael cite “an internal DoD contractors have learned how to keep their contracts
[Naval Air Systems Command] research project” off overseers’ radar through adroit baseline manage-
from the mid-1990s and state “In contrast to Chris- ment. Any reexamination of CPI stability must take
tensen and associates research, which used data that into consideration because increasing the budget
from the DAES database, the [NAVAIR] data was base is the easiest way to remain (or get back to)
sourced from the Contracts Analysis System [CAS] “green.” Not that I’m advocating such a reexamina-
database...” They are mistaken; DAES and CAS tion — the literature stands on its merits and has
are in fact the same database under different names, made an enormous contribution to improving US
now part of the “Defense Acquisition Management government program management.
Information Retrieval (DAMIR)” system. Ironically, Perhaps the larger issue is how our community
my office maintained the database and would have should respond to inadequate research. Lacking a
provided access to the NAVAIR staff. journal of our own, College of Performance Man-
In 1993, building on earlier work, Christensen agement members who wish to be published must
and Captain Scott Heise published their benchmark go elsewhere, where peer reviews for our special-
paper “Cost Performance Index Stability” in the Na- ized discipline may not meet our desired standards.
tional Contract Management Journal. They analyzed The CrossTalk article undoubtedly will be cited as
data from 155 defense contracts that met EVM re- gospel, given one co-author’s identification with the
quirements and distinguished between contracts hav- College. And that’s a shame.
ing stable and unstable performance measurement As for earned schedule, I believe it’s time to take it
baselines. Henderson and Zwikael fail to note that out of the “emerging practice” box in the practice stan-
the NAVAIR study did not use comparable criteria to dard. It should be recognized for what it’s been for at
select contracts from the same source data. Nor did least thirty years — a quick and easy sanity check for
it need to — it was performed for different reasons. comparison with project schedule analysis. Nothing
There is no way to determine the extent of overlap less — but nothing more until proved otherwise.
with the Christensen-Heise research or to compare
results. About the Author
Henderson and Zwikael also include information Wayne Abba retired in 1999 as
on 37 foreign projects: twelve “hi-tech” (unspeci- the Office of the Secretary of
fied) projects from Israel, twenty construction proj- Defense’s Senior Analyst for con-
ects from the United Kingdom and five IT projects tract performance measurement.
from Australia. Of the 37, only 24 were deemed He is an independent consultant
usable for CPI stability research. They offer no specializing in EVM in the pub-
evidence that these disparate projects implemented lic sector. He is past president
EVM consistently, as on DoD contracts, and their (2001–2002) of PMI-CPM and a
analysis lacks rigor. For example, Israeli data were contributing author of the forthcoming GAO Cost
analyzed “using visual inspection of charts.” Estimating and Assessment Guide: Best Practices
Henderson and Zwikael claim to have “overturned for Developing and Managing Capital Program
longstanding findings and beliefs on CPI stability.” Costs and of chapters 7 (“Project Cost Manage-
That claim does not withstand scrutiny. Christensen ment”) and 12 (“Procurement Management”) of the
et al have shown that CPI stability as they defined PMBOK® Guide — 4th edition update.
With MCR Program Support
aligning cost and performance
is easier than you think.
If your program is over budget but
under performing, MCR can help.
With the right tools, proven processes,
and subject-matter experts you need,
MCR has the capabilities to meet your
objectives and improve cost
effectiveness and performance.
– Risk Assessment
– Linking the Cost Estimate to the EV DAU
Performance Baseline
– EVMS Implementation/Certification
Equivalency
– Baseline Development/IBR Support for PM
– Business Case Analysis/Review Certification
– Capital Planning Consultation
– Staff Training & Development
Available
(32 Project/Acquisition
Management courses available) The MCR Training Institute (a
Defense Acquisition University
Keep your program aligned Equivalency provider) is offering
classes in Basic Earned Value
Management (BCF 102) and PM
Tools in a classroom setting taught
by experienced, certified instructors
who practice what they teach.
Contact mti@mcri.com
for more information.
MCR, LLC
2010 Corporate Ridge • Suite 350 • McLean, VA 22102 30 Years of Program/ Acquisition
703/506-4600 • Fax 703/506-8601
www.mcri.com Management Experience.
32
The Measurable News Fall 2008, Issue 4