Download as pdf or txt
Download as pdf or txt
You are on page 1of 32

Fall 2008, Issue 4

The Measurable News


The Magazine of the Project Management Institute’s College of Performance Management

Does Project Performance


Stability Exist?
A Re-examination of CPI
and Evaluation of SPI(t) Stability*
By Kym Henderson** and Dr. Ofer Zwikael

Abstract
The development of the Earned Schedule (ES) method by Lipke in
Did you hear? 2003 has been shown to be an important extension to the Earned
Value Management (EVM) method, increasing the utility of EVM data
The Measurable News for project schedule analysis, control and oversight. As ES provides
is now accepting a reliable time-based indicator of Schedule Performance, the objec-
sponsorship for tive of this paper is to investigate whether the SPI(t) exhibited similar
future issues. stability characteristics to those extensively reported for the Cost
With sponsorship, Performance Index (CPI) in EVM. This paper analyzes EVM data
you receive from three different countries for projects in three industry segments.
Thirty-seven projects were examined for SPI(t) stability and 26 for
• 4-color, 1/3-page ad on CPI stability. It has been found that while the behavior of SPI(t) is
front cover
broadly consistent with CPI, the widely reported CPI stability rule
• 4-color, full-page ad on cannot be generalized even within the US Defense Department (US
back cover DoD) project portfolio. Further research is required to develop im-
• Priority publication of proved understanding of project performance characteristics and the
two articles from profes- behavior of CPI and the SPI(t).

T
sionals in your organ-
ization he cancellation of the US Navy’s A-12 Avenger II stealth aircraft
program in January 1991 [1] [2] resulted in research during the
Too good to 1990s, which investigated the reliability of Earned Value Manage-
ment (EVM) cost prediction and the behavior of the Cost Perfor-
be true? mance Index (CPI)1 using US Defense Department (US DoD) project 2
data. These research findings have come to be regarded as generally ap-
Find out more!
Contact Managing Director *Reprint from Crosstalk, Journal of Defense Software Engineering, April 2008.
Gaile Argiro **The contents of this paper are the author’s personal views and conclusions which do not
reflect an endorsed position of the PMI College of Performance Management.
703.370.7885
1
Unless otherwise stated, all references to CPI and the SPI(t) refer to the cumulative values.
ExecAdmin@pmi-cpm.org
2
”Project” has been used consistently throughout this paper. In US government, particularly
the US DoD context, “program” may be the more appropriate term.
Continued on page 17.

See our VENDOR / SERVICES listing on page 18.


E R!
TT
BE s
EN
V ow h ng ORS
a
E
n udi AT
t 4 IS
h . cl IC
s ig ht 6 es in IND
n r
wI nsig eatu ING
W wI w f RN
NO ne WA
Y
A RL
E

Spending too much time reporting program status?


Produce standard reports with Deltek wInsightTM in minutes, not hours.
Meeting your mandated reporting requirements can be extremely time consuming. With Deltek wInsight, you can
streamline the reporting process, integrate seamlessly with Microsoft® Office and Microsoft® SharePoint and easily monitor
all of your projects.

Deltek wInsight:

• Gives you detailed information about project performance

• Saves you time by providing a common database for analysis and reporting on all projects across your firm, at the click
of a button

• Integrates with a wide range of project management tools including Deltek Cobra®, Deltek Open Plan®, Microsoft Project®,
Primavera®, MPM® and many more

Visit our website at www.deltek.com/wInsight

©2008 Deltek, Inc. All rights reserved. All referenced trademarks are the property of their respective owners
Fall 2008, Issue 4
The Measurable News 3

The Measurable News is an official publication


of the College of Performance Management
Contents
(CPM), 101 S. Whiting Street, Suite 320, Alexan-
dria, VA 22304, telephone 703.370.7885. The
Measurable News is published four times per
Does Project Performance Stability Exist?
year. A Re-examination of CPI and Evaluation of SPI(t) Stability
Editorial Staff By Kym Henderson and Dr. Ofer Zwikael.......................................... 1
Publisher
College of Performance Management

Managing Director
Gaile Argiro
Integrating Systems Engineering with Earned
Story Editor
Value Management
Peter Schwarz By Paul J. Solomon, PMP.................................................................... 7
Designer
Amanda Mitchell

Vice President of Communications


Robert Horrigan IPM-2008...............................................................16
Editorial Copy Vendor Services..................................................18
Editorial contributions, photos, and miscella-
neous inquiries should be addressed and sent to
the Editor at CPM headquarters. Please follow
the author guidelines posted on the PMI-CPM
The Trouble with Earned Schedule
Web site. Letters submitted to the editor will By Wayne Abba............................................................................... 28
be considered for publication unless the writer
requests otherwise. Letters are subject to edit-
ing for style, accuracy, space, and propriety.
All letters must be signed, and initials will be
used on request only if you include your name.
CPM reserves the right to refuse publication of
any letter for any reason. We welcome articles
relevant to project management. The Measur-
able News does not pay for submissions. Articles
published in The Measurable News remain the
property of the authors.

Advertising
Advertising inquiries, submissions, and pay-
ments (check or money order made payable
to the College of Performance Manage-
ment) should be sent to CPM headquarters.
Advertising rates are $1000 (taken) for inside
front or back cover (full-page ad only), $800
for other full-page ads, $500 for half-page ads,
Advertising Rates
and $300 for quarter-page ads. Issue sponsor-
ships are available at $2500 per issue. Business Inside front/back cover..................$ 1000
card ads are available for $100 per issues (or Full-page ad only — taken next issue
free with full-page ad). Rates are good from
January 1, 2008 – December 31, 2008. CPM Full page........................................... $ 800
reserves the right to refuse publication of any Size: 7.5 x 10 inches
ad for any reason.
Half page......................................... $ 500
Subscriptions Size: 5 x 7 inches or 10 x 3.5 inches
All CPM publications are produced as a ben-
efit for CPM members. All change of address Quarter page................................... $ 300
or membership inquiries should be directed Size: 5 x 3.5 inches
to: PMI Headquarters, Customer Care Depart-
ment, Four Campus Boulevard, Newtown Sponsor an issue............................. $2500
Square, PA 19073-3299 • Phone 610.356.4600 • includes ads on front and back cover
Fax 610.356.4647 • customercare@pmi.org.
All articles and letters represent the view of Business Card Ad.............................. $100
the authors and not necessarily those of CPM. Size: 2.3 x 1.3 inches
Advertising content does not signify endorse-
ment by CPM. Please notify CPM for single copy Contact Managing Director Gaile Argiro at
or reproduction requests. Appropriate charges
will apply.
PMI-CPM Headquarters for more information:
phone 703.370.7885 • ExecAdmin@pmi-cpm.org
© 2008 by the College of Performance Management. All rights reserved.
4
The Measurable News Fall 2008, Issue 4

PMI-CPM GOVERNING BOARD


2008–2010
President
John Singley, PhD, PMP
703.861.6700
President@pmi-cpm.org

Executive Vice President


Susan Wood
850.892.3747
ExecutiveVP@pmi-cpm.org

Vice President of Finance


Cathy Ahye Enjoy the season,
703.627.2022
VPFinance@pmi-cpm.org no matter where in
Vice President of Administration
the world you are…
Marilyn McCauley
937.878.7923
VPAdministration@pmi-cpm.org

Vice President of Conference & Events


Barry Schuler
703.456.0707
VPConf-Events@pmi-cpm.org

Vice President of Education & Certification


Joe Houser
301.529.6697
VPED@pmi-cpm.org

Vice President of Research & Standards


Kym Henderson
61.414.428.537
VPResearch-Standards@pmi-cpm.org

Vice President of Communications


Robert Horrigan
703.923.6094
VPCommunications@pmi-cpm.org

Past President
Neil Albert
703.506.4600
PastPresident@pmi-cpm.org
Register now for …
IPM 2008
20th Annual International
Integrated Program Management Conference
November 17–19, 2008
Hilton Alexandria Mark Center, Alexandria, VA
Call 703.845.1010 and ask for “IPM Conference rate”

The premier conference on Earned Value Management

Delivering A New “Steady-State”:


Portfolios, Programs and Projects
• Special Guest Speakers:

Michael Joyce, David M. Bowen, Jon C. Jones,


Lockheed Martin FAA Raytheon Co.

• Professional Education Training Seminars


• Practice Symposia
• Topical Workshops
• Networking Opportunities
• World's Experts on EVM
• Tools Track
Conference Fees: $750 * General Conference Information:
http://www.pmi-cpm.org/pages/events/IPM08/conf_program.html
or call 703.370.7885 * Exhibiting Information: erin@sceaonline.org
703.938.5090 * General Information: Gaile Argiro at 703.370.7885

Co-sponsored
Co-sponsored by…
by…
How you finish
depends on
how you start!
P RO J E C T M A N AG E M E N T S E RV I C E S A N D E D U C AT I O N

• Project Management Process Design

• Earned Value Management Implementation

• Independent Data Analysis

• Integrated Baseline Reviews

• Information Systems Integration

• Software Evaluations & Training

• Operational Support

• In-house Training

• Public Seminars on Project Scheduling


and Earned Value Management Systems

w w w . p m a s s o c . c o m
1 . 5 6 1 . 6 9 4 . 1 6 4 7 CHARTING PROJECT SUCCESS
Fall 2008, Issue 4
The Measurable News 7

Integrating Systems Engineering with


Earned Value Management
By Paul J. Solomon, PMP
Integrating Systems be inaccurate and misleading. Recent reports disclose
Engineering (SE) recurring weaknesses in procurement management and
recommendations for achieving desired outcomes, as
with Earned Value Management (EVM)

E
summarized in the following tables.
VM is capable of integrating a project’s Table 1, Defense Acquisitions, includes recom-
cost, schedule, and technical performance. mendations to hold contractors accountable for
However, an organization may comply with achieving desired outcomes. Table 2, Effective
GEIA EIA-748-B, the EVMS standard, yet Implementation of EVM, includes findings that
fail to integrate the project’s technical objectives agencies are not measuring actual vs. expected per-
into the Performance Measurement Baseline (PMB). formance in meeting performance goals. Table 3,
EVM data will be reliable and accurate only if the Best Practices, includes demonstrating that the de-
right base measures of technical performance are se- sign meets performance requirements.
lected and if progress is objectively assessed. The Office of Management and Budget (OMB)
Guidance and best practices for planning and Circular A-11 specifies that the EVMS must measure
measuring technical performance are provided by progress towards milestones, in terms of capability
SE and project management standards. of the investment to meet specified requirements and
EVM will provide more reliable information for quality.
analysis and decision-making if the EVMS guide-
lines are augmented by guidance regarding main- DOD Acquisition Transformation
taining the technical baseline, measuring technical The DOD issued acquisition policy and guides that
performance, and managing risk. specify measurement of technical performance
along with cost and schedule performance. The
GAO Procurement Findings and DOD guides that will enable implementation of the
Recommendations policy are based primarily on SE standards. In fact,
The Government Accountability Office (GAO) studied DOD states that revitalizing SE is a component of its
failures in acquisition of weapons systems and Infor- DOD’s Defense Acquisition Transformation Report
mation Technology (IT) systems. One report concluded to Congress.
that, if EVM is not properly implemented, the data may

Table 1: Defense Acquisitions.


GAO Report No. Title Findings and Recommendations
Contractors not held accountable for
achieving desired outcomes:
ª Cost goals
ª Schedule goals
ª Desired capabilities
Defense Acquisitions: • Programs do not capture early on
DOD Has Paid Billions in Award and the requisite knowledge needed
Incentive Fees Regardless of Acquisition to effectively manage program
06-66 Outcomes risks
Defense Acquisitions: Assessments of DOD needs to change its requirements
Selected Major Programs and budgeting processes to get desired
06-391 outcomes from the acquisition process
8
The Measurable News Fall 2008, Issue 4

DOD Guides Tables 4 and 5 show pertinent elements of the


guides. DoD guidance for the IBR is provided later
DOD guides that support the integration of techni-
in this article.
cal, schedule, and cost performance follow:
• Defense Acquisition Guidebook (DAG) EVMS Limitations
• SE Plan Preparation Guide (SEP)
If a contractor augments its EVM processes with SE
• Work Breakdown Structure Handbook, MIL-
processes, it can integrate technical, schedule, and
HDBK-881A (WBS)
cost performance and overcome the following limita-
• Integrated Master Plan and Integrated Master
tions of the EVMS.
Schedule Preparation and Use Guide (IMP/IMS).
• Guide for Integrating SE into DOD Acquisition 1. EVMS states that EV measures only the quantity
Contracts (Integ SE) of work accomplished, not the quality. It ad-
dresses the work scope of a project but ignores
Table 2: Effective implementation of EVM.
GAO Report No. Title Findings and Recommendations
06-250 IT: Agencies Need to Improve the • EVM can have impact on acquisition
Accuracy and Reliability of Investment success if properly implemented
Information • If not implemented effectively
decisions based on inaccurate
and potentially misleading EVM
information
• Agencies not measuring actual vs.
expected performance in meeting IT
performance goals.
08-345 OPM Improvements Needed to • EVM reporting was unreliable
Ensure Successful Retirement Systems because it reported program on
Modernization plan although
­– Test result had not met quality
goals
­– Lack of system maturation and
quality

Table 3: Best Practices Federal Policy on Performance Requirements and Quality.


GAO Report No. Title Findings and Recommendations
Best Practices: Best Practice Controls:
Better Support of Weapons System • Complete 90% of design drawings
06-110 Program Managers Needed to Improve • Complete subsystem and system
Outcomes design reviews
• Demonstrate with prototype that
Defense Acquisitions: design meets requirements
Major Weapon Systems Continue to • Agreement that drawings are
Experience Cost and Schedule Problems complete and producible
06-368 under DOD’s Revised Policy

Information Technology: Best Practices and Controls:


DOD’s Acquisition Policies and Guidance • Ensure that requirements are
Need to Incorporate Additional Best traceable, verifiable, and controlled.
04-722 Practices and Controls • Continually measure an acquisition’s
performance, cost, and schedule
DOD Systems Modernization: against approved baselines.
Planned Investment Needs to Be
06-215 Reassessed
Fall 2008, Issue 4
The Measurable News 9

the product scope. In comparison, the SE stan- states that the PMB may include technical and
dards and models address the product require- quality parameters.
ments and require assessment of progress against 2. EVMS encourages but does not require precise,
requirements, technical performance, design quantifiable measures. It states that objective
maturity, and the quality of the product being earned value methods are preferred but it also
developed. The Project Management Institute states that management assessment (subjective)
(PMI) standard, Project Management Body of may be used to determine the percentage of work
Knowledge (PMBOK® Guide) differentiates two completed. In contrast, the other standards speci-
components of scope, product scope and project fy objective measurement.
(work) scope. The PMBOK Guide also includes 3. EVMS is described as a risk management tool.
the quality baseline within the product scope and However, it contains no guidelines on risk man-
agement.

Table 4: Technical Reviews.


DoD Policy or Guide Policy DAG SEP WBS IMP / IMS Integ SE
Technical Reviews:
Event-driven timing of technical reviews X X X X X X
Success criteria of technical reviews X X X X X X
Include entry and exit criteria for
technical reviews in IMP and IMS X X
Assess technical maturity in technical
reviews X X X X

Table 5: Integrated Plans and TPMs.

DoD Guide DAG SEP WBS IMP / IMS Integ SE


Integrated Plans and Technical
Performance Measures (TPM):
Integrate SEP with IMP, IMS, TPMs, EVM
X X X X

Integrate WBS with requirements specification, statement of


work, IMP, IMS, EVMS X X X

Link risk management (including risk mitigation plans),


X
technical reviews, TPMs, EVM, WBS, IMS
Flow integrated program plans down to teammates,
subcontractors, vendors, lowest level suppliers and integrate
X
across SOW, SEP, IMP/IMS, and other plans and processes to
support critical path analysis, EVM, and risk management
Include in proposal a matrix that correlates Government SEP
X
with Offeror’s integrated SEP, SOW, IMP/IMS, WBS
Use TPMs to compare actual vs. planned technical
X X X X
development and design maturity
Use TPMs to report degree to which system requirements are
X X X
met in terms of Performance, Cost, Schedule
10
The Measurable News Fall 2008, Issue 4

4. EVMS provides little guidance for specifying the IEEE 1220


types of measures that should be the base mea-
Some of the guidelines, best practices and definitions
sures of EV such as TPMs and success criteria
from IEEE 1220 are in Table 6.
for technical reviews.
EIA 632
Standards and Models
Pertinent guidelines, best practices, and definitions
The limitations of EVMS may be overcome by in- from EIA 632 are in Table 7.
corporating best practices from other standards into
an organization’s EVM processes. The standards and INCOSE SE Handbook
models include: An important requirements management activity in
• Standard for Application and Management of the the INCOSE SE Handbook is the use of TPMs, as
SE Process (IEEE 1220) summarized in Table 8.
• Processes for Engineering a System (EIA 632)
• Capability Maturity Model Integration (CMMI®) Integrating SE with EVM
PMBOK Guide® The integration of SE with EVM can be achieved
• INCOSE Systems Engineering Handbook (INCOSE) by measuring progress towards meeting the planned
quality of the product being developed and built.

Table 6: IEEE 1220.


IEEE Std 1220 Standard for Application and
Topic Management of the Systems Engineering Process
Design 6.8.1.5.a Evaluate progress and completion of technical tasks
Maturity • Provide progress measurements of design maturity:
TPM 6.8.1.5.b. TPMs are key to progressively assess technical progress.
• Track relative to time with dates established for when:
­– Progress will be checked.
­– Full conformance will be met.
• Key technical parameters are measured relative to lower-level elements of the System
Breakdown Structure by estimate, analysis, or test, and values are rolled up to the system
level.
TPM 6.8.1.5.c Cost and schedule performance measurements are integrated with TPMs to:
• Provide current schedule and performance impacts.
• Provide an integrated corrective action to variances identified.
Quality, 6.8.6 Track product and process metrics
Product Metrics are collected, tracked and reported at pre-established control points...to enable:
Metrics • Overall system quality and productivity evaluation.
• Comparison of planned goals and targets.
• Benchmarking of the Systems Engineering Plan.

Table 7: EIA 632.


Topics ANSI/EIA-632 Processes for Engineering a System
TPM 4.2.1, Requirement 5, Technical Effort Definition
• Identify and track TPMs to determine the success of the system.
Note: Project the evolution of the parameter as a function of time toward the desired value at the
completion of development….based on verification, validation, planning, or historical data.

Product 4.2.2, Requirement 10, Progress Against Requirements


Metrics Identify product metrics, and their expected values, that will affect the quality of the product
and provide information toward satisfying acquirer and other stakeholder requirements, as
well as derived requirements.
Fall 2008, Issue 4
The Measurable News 11

Earned value, when possible, should be based on the of the enabling work products. Successful peer re-
sum of two measures: views or testing are often used to determine the com-
• Progress towards completing the set of enabling pleteness of enabling work products against a process
work products, such as drawings or code. quality standard. Later, measurements are taken of the
• Progress towards meeting the product require- capability of the design to meet the requirements.
ments, including technical performance or Quality. Of primary importance is the performance to-
wards achieving a combination of:
Product Requirements and Plan • Schedule objectives for enabling work products
The product scope (also called the technical or qual- that meet process quality objectives and
ity baseline) is bounded by product requirements. The • Event-driven success criteria when the event is
technical baseline should be integrated into the PMB meeting a measurable product requirement.
along with work scope, schedule, and cost objectives. These should also become the criteria for taking
The CMMI defines requirements traceability as 100 percent of earned value in work packages.
the evidence of an association between a require-
ment and its source requirement, its implementa- Risk Management
tion, and its verification. Requirements traceability The most straightforward guidance for integrat-
is maintained from a requirement to its derived ing risk management with EVM is in the PMBOK®
requirements as well as to its allocation to func- Guide. First, new risk management activities should
tions, objects, people, processes, and work products. be planned, budgeted, and scheduled if they consume
Traceability should also be maintained from the resources. It is important to add the risk response
work products to the plan, including the schedule, activity to the master schedule and to establish a logi-
work packages, and planning packages. cal link between the risk response activity and its
dependent activity on the schedule. By adding the risk
Measure Progress
response activity to the PMB, management will use
The validity of schedule and cost performance in- earned value to monitor its progress.
formation depends on the accurate measurement of Second, the EAC should be based on both project
progress towards meeting the product requirements. performance and quantified risk assessments. This
However, progress towards meeting the product guideline may be applied at all WBS levels. At the
requirements may not be measurable early in devel- project level, EAC should be based on the project
opment. For example, a hardware or software com- manager’s assessment of overall project perfor-
ponent may require the completion and assembly of mance, including technical performance, and quanti-
many enabling work products, such as drawings or fication of the impacts of top risks.
coded software modules, before the integrated set of
work products may be tested or analyzed to determine Practical Examples
if it will meet the product requirements. So, until that This section provides practical examples of tying
assembly is completed, early measurements are taken earned value to technical performance:
Table 8: TPM per INCOSE SE Handbook. • Ex 1: Base
EV on
TPM per INCOSE SE Handbook
completing
• TPMs express the objective performance requirements.
drawings and
• Without TPM
meeting TPM
­– Project manager could fall into the trap of relying on cost and schedule status alone
­– Can lead to a product developed on schedule and within cost that does not requirements
meet all key requirements. • Ex 2: Entry
• Periodic recording of status of each TPM and exit crite-
­– Provides continuing verification of degree of anticipated and actual
achievement of technical parameters.
ria for techni-
cal reviews
12
The Measurable News Fall 2008, Issue 4

Ex 1: Base EV on completing drawings Table 10, Net BCWP Based on Component Re-
and meeting TPM requirements quirements, shows the time-phased Budgeted Cost
This example uses technical performance measure- for Work Scheduled (BCWS), earned value (BCWP)
ment (TPM) to assess the capability of the design to based on completed drawings and negative BCWP
meet product requirements. TPM uses actual or pre- when the design fails to meet requirements.
dicted values from engineering measurements, tests, Schedule variance analysis should state that al-
experiments, models, simulations, or prototypes and though the drawings are ahead of schedule (+40), the
includes milestones for meeting the requirements. design has not met the planned requirements (-100),
The output of a work package is the design of a resulting in a net SV of -60. There will be an unfa-
component of a subsystem, a set of wire harnesses. vorable impact to both the cost and schedule objec-
There are two TPM requirements that are allocated tives as the drawings are reworked until the design
to the wire harnesses: meets the requirements.
• Maximum weight: 200 lb. In Ex 1, TPMs were used to determine if the
• Maximum diameter: 1 inch weight and diameter requirements were met at the
There are two measures of progress, completing component or work package level. However, perfor-
the drawings and meeting the requirements. The mance requirements are usually established at higher
schedule is shown in Table 9: WBS levels. If the design of a component is at the
• The budget is allocated as follows: work package level, completion of the design may
• Budget at Completion (BAC): 2000 hours depend on achieving planned TPMs values or other
• Budget per drawing: 40 hours quality objectives at the higher level. In this case,
• EV penalty if requirements not met: –300 hours earned value for a component level work package
During the early periods, EV is based on tradi- should be dependent on meeting both the component
tional measures, completion of the enabling work and higher level objectives.
products (drawings). Later, when there is a milestone Ex 2: Entry and exit criteria for technical
to measure and achieve the technical performance reviews
requirement, there is an EV penalty (negative earned An important control for ensuring integration of a
value) if the requirement is not met. EV is restored project’s technical and schedule objectives is to es-
when the requirement is finally met. The total possi- tablish milestones for the end of a stage of develop-
ble negative earned value is –300 hours, as follows: ment with the right exit or success criteria. These
• Component weight requirement (req.) not met: exit criteria should also be used for the work pack-
–100 hours ages that support that milestone. This example pro-
• Diameter req. not met: –200 hours vides guidance for defining the right exit criteria as
The schedule status at April month end follows: well as entry criteria.
• Cumulative (cum.) drawings completed: 41 Per IEEE 1220, 6.8.1.5.d, technical reviews are
• Diameter req. met conducted to:
• Component weight req. not met • Assure that all master schedule success criteria
have been met
• Assess development maturity
Table 9: Schedule for Drawings and Requirements. to date
Schedule Plan Jan. Feb. Mar. Apr. May Total • Assess the product’s ability to
Drawings 8 10 12 10 10 50 satisfy requirements
• Assure traceability of require-
Requirements met:
ments and validity of decisions
Weight 1 1 Exit criteria for preliminary de-
Diameter 1 1 sign reviews (PDR) and
critical or final design reviews
(CDR) are also in IEEE 1220.
Fall 2008, Issue 4
The Measurable News 13

Table 10: Net BCWP Based on Component Requirements. EACs to variances from the plans
Design (drawings) Jan. Feb. Mar. Apr. May Total for achieving all success criteria
for technical reviews and TPM
Planned drawings 8 10 12 10 10 50
variances.
BCWS – cur. 320 400 480 400 400 2000
BCWS – cum. 320 720 1200 1600 2000 2000 Confirm Integration of
Actual drawings
Technical Performance
completed 9 10 10 12 with EVM in IBR
BCWP (drawings) – cur. 360 400 400 480 The DOD guide (Integ SE) em-
phasizes reviewing the following
BCWP (drawings) – cum. 360 760 1160 1640
during an IBR:
Negative BCWP (reqs) – • Plans for event-based technical
cum. –100
reviews including entry and exit
Net BCWP (drawings criteria and independent subject
and reqs) 1540 matter expert participation
Schedule Variance (SV) 40 40 –40 –60 • Technical tasks and products
resulting from the IMS tasks
Some of the exit criteria for a system PDR are: • Correlation of the technical metrics and mea-
• Prior completion of subsystem reviews sures, IMP/IMS, EVMS
• Total system approach to detailed design satisfies For example, the IMS should include the criteria
the system baseline for completing technical reviews and milestones for
• Unacceptable risks are mitigated measuring technical performance as well as the TPM
• Issues for all subsystems, products, and life cycle planned values to be achieved at that milestone. The
processes are resolved IMS and work packages should also include comple-
• Trade-study data are adequate to substantiate that tion of the technical baselines.
subsystem and system requirements are achievable
Manage Requirements
Better Acquisition Management
The following guidance for managing requirements
Finally, a customer can manage the acquisition of
is derived from the CMMI-Acquisition Module
a new system more effectively if it ensures that the
(CMMI-AM)
supplier’s management process and reports integrate
1. Identify inconsistencies between requirements,
technical performance, with schedule and cost perfor-
project plans and work products.
mance and focuses on the requirements and quality.
1.3 Manage changes to the requirements.
Include Performance-Based Each change to a controlled requirement should
Progress in Contract be assessed for impact to the project perfor-
The customer should initiate performance-based mance, cost, and schedule baselines. These
acquisition management by including requirements baselines should be changed, as required, to ac-
for achieving technical performance objectives in its commodate the requirements change.
contracts, beginning with the solicitation. 1.4 Maintain bidirectional traceability among the
The customer can obtain insight into earned value requirements and the project plans and work
that is based on technical performance by requiring products.
the supplier to define milestones that meet desired 1.5 Identify inconsistencies between the project
performance outcomes such as: plans and work products and the requirements.
• Success criteria for major technical reviews.
• TPM planned values and measurement milestones. Award Fee
Also, EVM variance analysis reports should relate A second component of Defense Acquisition Trans-
schedule variances, schedule recovery plans and formation is Award Fee and Incentives Policy. In April
14
The Measurable News Fall 2008, Issue 4

2007, DOD issued a policy that award fee criteria be Conclusion


derived from contract objectives; technical/program- Integrating SE with traditional EVM enables the in-
matic, cost, and schedule. The policy stated that objective tegration of a project’s cost, schedule, and technical
criteria to measure contract performance toward these objectives.
objectives will be utilized. The previous policy memo, • It enhances EVM’s value as a project manage-
issued in March 2006, stated that in situations where ment tool.
there may be no identifiable milestone for a year or more, • It focuses on the customer’s requirements and
consideration should be given to apportioning some of needs.
the award fee pool for a predetermined interim period of • It is based on standards and models for systems
time, based on assessing progress toward milestones. engineering and project management.
The OMB issued a policy memo in December 2007
that applies to all federal agencies. The memo stated: References
• Link incentive and award fees to cost, sched- References and pertinent extracts of the cited guidance
ule, and performance results are provided at www.PB-EV.com.
• Use metrics that describe what is required
and at what point a contractor is considered About the Author
successful Paul Solomon is a consultant and co-author of Per-
If a company’s application of EVM effectively inte- formance-Based Earned Value.® He holds a BA and
grates the project scope of work with cost, schedule, and an MBA from Dartmouth College. He has led EVM
performance elements, as required by the FAR, then on Northrop Grumman programs including the B-2
earned value metrics indicate progress toward mile- Bomber, Global Hawk, and F-35 Joint Strike Fighter.
stones and toward achieving final cost objectives. He is a co-author of the EVMS standard and recipient
Consequently, valid earned value metrics may be used to of the DoD David Packard Excellence in Acquisition
determine award fee. The validity of the earned value Award. He was a Visiting Scientist at the Software En-
metrics should be ensured if a company is publicly owned gineering Institute. Contact: 818.212.8462; paul.solomon@
and has established and maintained effective internal pb-ev.com; 3307 Meadow Oak Dr., Westlake Village, CA 91361;
controls as required by the Sarbanes-OxleyAct of 2002. www.PB-EV.com (provides EVM resources and best practices).

If your program is OVER budget but


PMI-CPM’s Management EVM Educational Program can help you get back on track.
under performing,

PMI-CPM EVM Education Program has the right subject-matter experts and training capabilities to help your or-
ganization meet program training objectives and improve cost and schedule performance. PMI-CPM’s core EVM
education training capabilities encompass the full spectrum of program management:

• Practice Symposia: Lessons learned and best practices on the implementation of Earned Value
Management, updated on recent developments and integrating EVM with other management tech-
niques.
• Workshops: Workshops provide a forum to introduce and develop new and emerging practices
and techniques to improve program management using EVM, objectives are to identify potential so-
lutions. Workshops are not intended to provide training or instruction on EVM.

20th Annual International Integrated Program Management Conference • November 17–19, 2008
Hilton Alexandria Mark Center, Alexandria, VA
Joe Houser • VP Professional Education Training Seminar • KMSystems Group • jrhouser@kmsystemsgroup.com
Bill Chitty • Education Communications Coordinator • BAE Systems • bill.chitty@baesystems.com
PROJECTMANAGEMENT
FORTHE
N E X T  G E N E R AT I O N 


WHAT HAS YOUR PROJECT MANAGEMENT SYSTEM DONE FOR YOU TODAY?
Too many software companies claim to be all things to all people by piecing together separate modules that provide
the desired functionality with integration as an afterthought. SAFRAN provides integrated performance on the most
widely accepted and industry tested technology.
S AFRAN P ROJECT AND S AFRAN FOR M ICROSOFT P ROJECT
Project management software from Safran provides organizations with superior visibility and insight into how time and
resources are spent and into the overall health of their projects. Our solutions provide all of the power and functionality you
need out of the box based on your industry and business needs, integrated from the ground up; not after the fact.

Implementing Safran project management software decreased tangible and intangible project costs, increased
efficiency almost 100% in day-to-day project planning and control, and reduced bid preparation times 75% -
providing our customers with the competitive edge.
From the individual project office to the enterprise, Safran provides the most up-to-date and powerful EPM solution today.


SAFRAN’S“ONESOLUTION”APPROACHFORPROJECTͲCENTRICORGANIZATIONS:
SAFRANPROJECTlSAFRANFORMICROSOFTPROJECTlSAFRANPLANNERlSAFRANSCORECARD


ENTERPRISEPROJECTMANAGEMENTSOFTWAREANDSERVICESFOR:
ENERGYANDPOWERlA&E/CONSTRUCTIONlAEROSPACEANDDEFENSE
N ORTH A MERICA
GOVERNMENTlHIGHTECHlANSI748COMPLIANCE
simple | flexible | powerful

2132A CENTRAL AVENUE SE #253 l ALBUQUERQUE, NM 87106


PHONE: 505.265.2229 l FAX: 505.265.0229 l WWW.SAFRANNA.COM

© Safran North America, LLC, All Rights reserved.


WWW.SAFRANNA.COM
All other trademarks mentioned herein are the property of their respective owners
Plan Ahead…
Make your budget requests now for the
best conferences on Earned Value Management!

IPM 2008
Delivering A New “Steady-State”: Portfolios, Programs and Projects
20th Annual International Integrated Program Management Conference
November 17–19, 2008
Hilton Alexandria Mark Center, Alexandria, VA
Registration: Early $700 • Regular $750
Hotel Rate: $143 (phone 703.845.1010)
http://www.pmi-cpm.org/pages/events/IPM08/conf_program.html

EVM World 2009


PMI-CPM 25th Annual International Conference
May 27–29, 2009 • Naples Grande Resort
Naples, FL • 888.422.6177
www.naplesgranderesort.com • www.naples-florida.com
CPM Members: Early $600 • Regular $650
Non-CPM Members: Early $700 • Regular $750
Hotel: Government Rate — prevailing per diem
Corporate Rate — $169

PMI-College of Performance Management • 101 South Whiting Street, Suite 320


Alexandria, VA 22304 • 703.370.7885 • 703.370.1757• www.pmi-cpm.org
Fall 2008, Issue 4
The Measurable News 17

Continued from page 1. Background


plicable across all project types using EVM across The CPI has long been a key indicator used to ana-
multiple industry sectors. A finding regarded as par- lyze the cost performance of projects using Earned
ticularly significant was that CPI stabilizes by 20% Value Management (EVM).
of project completion. The first empiric confirmation of the widely re-
Lipke proposed the ES method in 2003 to provide ported and referenced CPI stability rule was by
time based measures of schedule performance utiliz- Christensen and Payne using data from 26 US Air
ing EVM data. Initial validation has shown that the Force completed contracts in 1992. The data used
time based ES derived Schedule Performance Index came from the cost library of the US Air Force Sys-
(time) (SPI(t)) to be reliable for both early and late tems Command Aeronautical Systems Division [5].
finish projects. For a technical description of the Christensen and Templin conveniently summa-
ES method the reader is referred to Lipke’s seminal rized the series of research findings subsequent to
paper, “Schedule is Different” [3]. For an excellent that paper in 2002:
easy to read non‑technical but comprehensive dis- … the range of the cumulative CPI from the 20
cussion of the ES method, refer to “Not Your Fathers percent completion point to contract completion
Earned Value” [4] by Stratton. was less than 0.20 for every contract. This result
Following the initial validation of ES, interest is usually interpreted to mean that the cumulative
developed in ascertaining whether SPI(t) exhibited CPI does not change by more than plus or minus
similar stability characteristics to those extensively 0.10 from its value at the 20 percent completion
reported for CPI. The objective of this research pa- point, and is used to evaluate the reasonableness
per is to reexamine CPI stability and to compare the of projected cost efficiencies on future work. [6]
stability behavior of the SPI(t) with CPI. Christensen and Payne [5] made the following ob-
This paper has found that while the behavior of servations on the perceived importance of CPI stability:
the SPI(t) is broadly consistent with CPI, the widely • A stable CPI is evidence that the contractor’s
reported CPI stability rule cannot be generalized to management control systems, particularly the
all projects utilizing the EVM method or even within planning, budgeting, and accounting systems, are
the US DoD project portfolio. However, the consis- functioning properly.
tent behavior to CPI demonstrated by SPI(t) provides • A stable CPI may thus indicate that the contrac-
further support for the validity of the SPI(t) metric tor’s estimated final costs of the authorized work,
and the ES method. termed “Estimated at Completion,” are reliable.
Additional analysis was unable to establish a cor- • In addition, knowing that the CPI is stable may
relation between achieving earlier CPI and SPI(t) help the analyst evaluate the capability of a con-
stability and improved outcomes at completion. In tractor to recover from a cost overrun by com-
certain cases where projects achieved either under paring the CPI with other key indicators, such as
budget and/or early finish outcomes with cost and/ the To-Complete Performance Index.
or schedule stability achieved late, earlier cost and/or Over time, the widely reported CPI stability find-
schedule stability would have been disadvantageous ings have been generalized as being applicable to all
to the actual final outcome(s) achieved. This is be- projects utilizing the EVM method [8] [9] [10] [11].
cause CPI and/or SPI(t) progressively improved over An extensive literature review has not found further
the life of those projects. empiric validation of the CPI stability rule beyond
This paper also demonstrates that by utilizing ES, the project data obtained in the initial paper and data
research of schedule performance using EVM data is from the US DoD Defense Acquisition Executive
now possible, and leading to improved understand- Summary (DAES) database.
ing of the dynamics of project schedule and project Concurrent research into the stability characteristics
cost performance. of the EVM Schedule Performance Index (SPI) was not
Continued on page 19.
P RO J E C T M A N AG E M E N T S E RV I C E S A N D E D U C AT I O N

V e cost
n anddperformance
o r
With MCR Program Management,
aligning / S e r v i c es • Project Management Process Design

is easier than you think. • Earned Value Management Implementation

For information
If your program is over budget but
under performing, MCR can help. to advertise, contact Managing Director
• IndependentGaile Argiro at
Data Analysis

703.370.7885 or ExecAdmin@pmi-cpm.org.
With the right tools, proven processes,
and subject-matter experts you need, • Integrated Baseline Reviews
MCR has the capabilities to meet your
objectives and improve cost
effectiveness and performance. • Information Systems Integration
Our Core Capabilities encompass the
full spectrum of program management, • Software Evaluations & Training
from start up to sustainment, including

– Program Risk Assessment • Operational Support


– Linking the Cost Estimate to the EV
Performance Baseline
www.DekkerLtd.com
– EVMS Implementation/Certification www.deltek.com • In-house Training
www.humphreys-assoc.com
– Baseline Development/IBR Support
– Business Case Analysis/Review
see our ad
– Capital Planning Consultation on p. 32 • Public Seminars see
on Project
our adScheduling
on p. 29
– Staff Training & Development see our ad on p. 2
(32 Project/Acquisition
and Earned Value Management Systems
Management courses available)

Keep your program aligned


by taking the next step with MCR.

Your ad

could go
w w w . p m a s s o c . c o m

www.mcri.com
30 Years of Program/
1 . 5 6 1 . 6 9 4 . 1 6 4 7 CHARTING PROJECT SUCCESS
www.pmassoc.com
Acquisition Management
here!
MCR, LLC
2010 Corporate Ridge • Suite 350 • McLean, VA 22102
see our ad onExperience.
703/506-4600 • Fax 703/506-8601
www.mcri.com p. 31 see our ad on p. 6

www.ProjectRX.com www.primavera.com/products/ infona@safran.com


costmanager/index.asp
see our ad on p. 4 see our ad on p. 15

www.aisc.com www.arescorporation.com www.lexemstrategy.com

Your ad Your ad

could go could go
www.systalex.com
here! here!

V e n d o r / S e r v i c es
Fall 2008, Issue 4
The Measurable News 19

Continued from page 17. reported for the Cost Performance Index. The objec-
possible because the SPI is known to fail as a statistical tive of this research paper is to reexamine Cost Per-
predictor because it always returns to unity at project formance Index stability and to compare the stability
completion irrespective of duration based delay. The behavior of the SPI(t) with CPI.
SPI is also recognized as failing, nominally within the
Method for Evaluating Stability
final third of the project and also fails after the project’s
Planned Duration has been exceeded. EVM project data was loaded into a Microsoft Excel
Lipke proposed the ES method in 2003 as a solution “Stability Point Calculator” developed by Lipke. The
to these limitations and flaws of the EVM schedule in- calculator determines the observation number in a
dicators [3]. A series of studies provided initial valida- sequence of CPI and SPI(t) values at which all sub-
tion of the ES method, some by using real EVM project sequent observations are within a defined stability
data, Henderson [12] [13] [14] and Vandevoorde and limit. The stability limit used is .10. The calculator
Vanhoucke [15] and also by using simulated network enables the associated percentage complete at which
schedules, Vanhoucke and Vandevoorde [16]. The time stability occurs to be determined.
based ES derived SPI(t) has been shown to be reliable This calculator has been placed into the public
for both early and late finish projects. The SPI(t) only domain to encourage more broadly based CPI and
reverts to unity at project completion if on time com- SPI(t) stability research and is freely available from
pletion has been achieved. the ES website at http://www.earnedschedule.com/
A research study intended to validate the ES con- Calculator.shtml.
struct using DAES data was commissioned in 2004 and To determine the significance of the observations
undertaken by a US Air Force Institute of Technology of stability for both CPI and SPI(t), statistical hy-
Masters student. Unfortunately, this study was discon- pothesis testing is conducted. The test applied is the
tinued after an independent review determined: Sign Test at 0.05 level of significance3 [19]. The Sign
Test was used in this research because it does not
Results: The historical data collection procedures
depend upon the data having a normal distribution.
for the DOD and USAF do not allow for sufficient
In past research, the hypothesis test method chosen
testing of ES theory at this time. A statistical evalu-
implied that the data was normally distributed; how-
ation concluded that SPI(t) is different than SPI($);
ever, the normality of the data was not established.
however, the two variables are highly correlated.
Research by Lipke also suggests that:
The result of the analysis identified that SPI(t) per-
Results indicate the logarithm data representations of
forms similarly to SPI($) with the data contained in
the indexes are likely normally distributed, whereas the
the DAES database. In order for the ES Theory to
distributions for CPI, SPI, and CV are not. [20]
be fully investigated, additional data must be col-
The question to answer regarding stability is “Can
lected. This research shows that the necessary data
it be stated generally and reliably that the final value
may also not be available despite the best collection
of the performance index is within 0.10 of its value
efforts. The original schedule and planned duration
when the project is 20 percent complete?” The an-
information is critical to successful evaluation of the
swer to the question will be “yes” if the alternate
ES methodology. [17]
hypothesis is satisfied:
However, early interest by the Project Management
H1(CPI): |CPI(final) – CPI(20%)| < 0.10
Institute College of Performance Management resulted
H2 (SPI(t)): |SPI(t)(final) – SPI(t)(20%)| < 0.10
in the principles of ES being included as an “Emerg-
Two separate hypothesis tests are conducted, one
ing Practice Insert” in the Practice Standard for Earned
for CPI and one for the SPI(t). The result from the
Value Management published in 2004. [18]
hypothesis testing is recorded as Ha when the value
Following the initial validation of ES, interest de-
of the test statistic is in the critical region (0.05) and
veloped in ascertaining whether the SPI(t) exhibited
Ho (null hypothesis) when it is not.
similar stability characteristics to those extensively
3
Applying the Sign Test at the 0.05 level of significance means that the test is being applied at a 95% level of confidence.
20
The Measurable News Fall 2008, Issue 4

The Data Stability Evaluation Results


A composite EVM data set was assembled compris- The results of the Sign Tests for testing the hypoth-
ing commercial sector data samples obtained from: esis “can it be stated generally and reliably that the
• 24 United Kingdom (UK) construction projects final value of the performance index is within 0.10 of
• 12 Israeli High Technology (Hi-Tech) projects its value when the project is 20 percent complete?”
• 9 Australian Information Technology (IT) projects. as previously described are tabulated in Table 1 be-
The EVM data consists of direct labor costs only with: low. Recall, the test result of Ha indicates stability of
• UK construction projects recorded in “person the performance indicators CPI and the SPI(t). As is
days” weekly with EVM values expressed as a shown, the test results did not have any test statistic
percentage of the Budget at Complete to further in the critical region (0.05). As a result, none of the
maintain data anonymity, null hypotheses can be rejected, for any of the three
• Israeli Hi-Tech projects recorded in United States samples as well as the composite of all samples. This
dollars monthly, and means that stability was not achieved for either CPI
• Australian IT projects recorded in Australian dol- or the SPI(t) by the time the project was 20 percent
lars weekly. complete.
An extensive review of the data was undertaken. This research does not support the previously ref-
Projects were excluded from the sample for a variety erenced generalizations that the CPI stability rule has
of reasons including: universal applicability for all projects utilizing the
• Lack of data integrity, EVM method. Because the SPI(t) index demonstrates
• Lack of Earned Value data at 20% of project a similar lack of stability to that found for CPI, the
completion, validity of the SPI(t) metric is supported due to the
• Partially incomplete Planned Value data, and consistent behavior demonstrated with CPI.
• Lack of required Actual Cost data. Table 2 summarizes the raw data in relation to the
Ten UK Construction projects are included in the numbers of projects that achieved stability before or
CPI stability research sample. Five of these proj- after 20% completion for the SPI(t) and CPI by each
ect were included although the
final Actual Cost data available Table 1: Hypothesis Test Results.
was between 96.7% to 99.0% CPI Stability SPI(t) Stability
complete. Including those five Test Statistic Test Result Test Statistic Test Result
projects is consistent with the ap-
UK
proach adopted by Christensen Construction 0.623 Ho 0.748 Ho
and Payne’s research [5] and as-
Australian IT 1.000 Ho 0.500 Ho
sumes that the difference between
CPI Final and the latest available Israeli Hi Tech 0.806 Ho 0.613 Ho

CPI has no material impact on the Composite 0.916 Ho 0.629 Ho


findings.
The outcome was a usable data
Table 2: Summary of Stability Achievement Related to 20% Completion.
sample of
• Twelve Israeli Hi-Tech proj- Stability UK Australian Israeli
Achieved Construction IT HI Tech Composite
ects for the SPI(t) and CPI
stability research SPI(t)
cum ≤ 20% 3 0 1 4
• Twenty UK construction proj-
ects for the SPI(t) stability and > 20% 17 5 11 33
ten for CPI stability research CPI
• Five Australian IT projects for cum ≤ 20% 2 0 1 3
the SPI(t) stability and four > 20% 8 4 11 23
for CPI stability research.
Fall 2008, Issue 4
The Measurable News 21

project set and for the composite of all. It can be Additional Analysis
seen that the majority of projects reach stability only Following the lack of CPI and SPI(t) stability find-
after the 20% completion point. ings additional analysis was conducted. Within each
Figure 1 summarizes within each 10 percent com- 10% complete percentile bands projects were catego-
plete percentile band where CPI and the SPI(t) sta- rized as follows:
bility occurred. This figure shows: • Cost at completion:
• The wide variability in the achievement of stabil- o Under or On Budget (UOB)
ity for both CPI and the SPI(t). o Over Budget (OvB).
Project performance heuristics or “rules of • Schedule at completion:
thumb” intended to be generally applicable (e.g., o Early or On Time finish (EOT)
the CPI stability rule) require an empirically es- o Late Finish (LF).
tablished consistency of behavior across a broad The purpose of this analysis is to determine if there
range of projects. These findings are a significant is a correlation between achieving earlier CPI and the
impediment to proposing and confirming broadly SPI(t) stability and improved project outcomes.
applicable CPI and SPI(t) stability heuristics. Figure 2 summarizes the analysis for CPI and
• That stability is usually achieved very late in the Figure 3 does the same for the SPI(t). With the data
project lifecycle, often later than 80% complete samples utilized, achievement of earlier stability is
for projects in these samples. not correlated with improved final cost and/or sched-
Zwikael et al. analyzed the Israeli hi‑tech project ule outcomes.
sample using visual inspection of charts and suggest- For UOB and EOT projects where cost and sched-
ed that CPI stability was, on average, achieved at the ule stability was achieved late (after say 60% com-
60% completion point [21]. That analysis broadly pletion) achieving earlier stability would have been
confirms this paper’s finding of CPI stability being disadvantageous to the final outcome(s) achieved
achieved much later in the project lifecyle than pre- because project performance progressively improved
viously reported. over the life of those projects.

!
Figure 1: Total Projects CPI and SPI(t) Stability Within Each 10 Percentile Band.
22
The Measurable News Fall 2008, Issue 4

!
Figure 2: Project Completion Categories by CPI Stability Band.

!
Figure 3: Project Completion Categories by SPI(t) Stability Band.
Fall 2008, Issue 4
The Measurable News 23

Figure 4 summarizes projects (with the required domain on the PMI Sydney Chapter website. [22]
comparative data), which achieved SPI(t) or CPI The purpose of the Popp study was to develop prob-
stability first. Achieving SPI(t) stability first implies ability distributions of cost Estimates at Complete
schedule management had a higher management pri- (EACs) based on the CPI at complete, current CPI
ority, achieving CPI stability first implies cost man- and percentage complete of projects based on his-
agement had the higher priority. tory. As stated in the report:
In the Australian IT projects sample, SPI(t) sta- Given a program has a CPI of X and a percent com-
bility was achieved first for the preponderance of plete of Y, what is the most likely finishing CPI. [22]
projects. For the other data samples the achievement In contrast to Christensen and associates research,
of cost or schedule stability first occurred in rough- which used data from the DAES database, the data
ly equal proportion. In only one project in these used by Popp was sourced from the Contracts Analy-
samples, an Australian IT project, was the cost and sis System (CAS) database maintained by the Office
schedule stability achieved simultaneously. of the Secretary of Defense Cost Analysis Improve-
ment Group (CAIG).4
Corroboration with Other Research The research undertaken by Popp did not focus
Because of the comprehensive contradiction to the on CPI stability. However, charts which can also be
previously published CPI stability research findings, used for assessing CPI stability were completed as
a further literature review was undertaken. This part of that study. These charts correlate the cumula-
review obtained a most unexpected source of inde- tive CPI for the percentage complete in each 10%
pendent corroboration for this paper’s CPI stability complete percentile band to the CPI Final for all
findings. In the mid 1990s Mr. Michael Popp initi- projects in that sample.
ated an internal US DoD research project within the Figure 5 is the first chart of interest from the Popp
US Naval Air Systems Command (NAVAIR). report, as it shows the correlation between the cumu-
The output was an internal but unclassified NA- lative CPI at 10–20% complete and the CPI Final for
VAIR report (the Popp report) which has, with Mr. all projects in the sample.
Popp’s permission, now been placed into the public

Figure 4: Summary of Projects Achieving SPI(t) or CPI Stability First. !


Post publication note: It is now known, thanks to a post publication correction to the paper that the “DAES and CAS are the same
4
!
database under different names”.
24
The Measurable News Fall 2008, Issue 4

The area of the chart enclosed


within the dashed lines bounds the
area in which the correlation plots
must occur for the Christensen de-
rived CPI stability rule to apply.
Those plots which occur outside
the enclosed area are also in con-
flict with the Christensen derived
CPI stability rule. The limited data
samples used in this analysis are suf-
ficient to show that the CPI stabil-
ity rule cannot be generalized even
within the US DoD project portfolio.
While research by Coleman et.
al. [23] using the Popp report data
sample was not principally directed
at examining the validity of the Figure 5: Correlation between Cumulative CPI at 10-20% Complete
and Final CPI (Popp)
CPI stability rule this research !
found that for:
• Development programs “at 20%
(completion) , programs with a cu-
mulative CPI below 0.89 improve”
which was “close to Christensen,
(findings) but with some excep-
tions”
• Production programs “at 20%
(completion), programs with
a cumulative CPI below 0.84
improve, again “close to Chris-
tensen, (findings) but with some
exceptions”.
Using the “enclosure” technique,
Figure 6 shows that the preponder- Figure 6: Correlation between Cumulative CPI at 70-80% Complete
ance of plots, occur within the area and Final CPI (Popp)
!
where the CPI stability rule applies
at 20% completion. The conclusion is that for the US does not significantly improve during the period
DoD project data used by Popp, CPI stability was also between 15% and 85% of contract performance; in
achieved very late in the project lifecycle, often as late fact, it tends to decline. [1]
as 70-80% completion. This finding is consistent with Some projects in the Popp sample show a trend of
the late CPI stability findings for the commercial sector CPI performance improvement, from CPI 20% and
project samples as shown in Figure 1. in a smaller number of cases, as late as CPI 80% to
While the underlying data was not available and CPI Final.
further research is required, these findings also con-
flict with the US DoD research findings quoted in
Summary and Conclusions
The initial objective of this paper, ascertaining
the Beach report into the A-12 cancellation that:
whether the SPI(t) demonstrates similar stability
DOD experience in more than 400 programs since
characteristics to those extensively reported for CPI
1977 indicates without exception that the cum CPI
Fall 2008, Issue 4
The Measurable News 25

was not achieved. This paper has found that while prediction techniques and the supporting rationales
the behavior of the SPI(t) is broadly consistent with is “pending publication” as at 12th September 2007
CPI, the widely reported CPI stability rule cannot be [24]. The statistical prediction techniques developed
generalized to all projects using the EVM method or have been summarized in a presentation by Hender-
even within the US DoD project portfolio. However, son [25] which is available on the website.
the consistent behavior to CPI demonstrated by the A major advance to EVM practice and future
SPI(t) provides further support for the validity of the research opportunities would be development of a
SPI(t) metric and the ES method. broadly based EVM research database where com-
Additional analysis was unable to establish a pleted EVM project data could be submitted anony-
correlation between achieving earlier CPI and the mously for:
SPI(t) stability and improved outcomes at comple- • Research purposes
tion. In cases where projects achieved either under • Benchmarking completed project performance
budget and/or early finish outcomes with cost and/ • Assisting in the sizing of projects.
or schedule stability achieved late (ie. after say 60% Such knowledge bases are not unique in other disci-
completion), earlier cost and/or schedule stability plines, with an instructive Australian example being the
would have been disadvantageous to the actual final International Software Benchmarking Standards Group
outcome(s) achieved. This is because CPI and/or the (ISBSG), website at www.isbsg.org.
SPI(t) were progressively improving over the life of Improved data collection techniques to ensure that
those projects. baseline schedule information is captured and stored
The findings and corroboration of this paper require in the DAES database are also recommended.
significant review and revision to what has been re-
garded as a long settled EVM heuristic with regard Concluding Remarks and Future
to CPI stability and consequent practice including the Research
use of a stable CPI as evidence that an EVM system is While this paper has overturned long-standing find-
functioning properly and of a “reliable” EAC. [5] ings and belief on CPI stability, it is important that
Improvements to current EVM techniques for the strengths and limitations of the EVM method are
predicting future cost performance should be con- properly understood, particularly in an era of:
sidered as current techniques have relied on general- • Adoption of EVM by US Government agencies
izing research findings from limited data sources, through Office of Management Budget Circular
principally the DAES database. A-11 Part 7 mandate
Alternatives methods of cost and schedule predic- • Advocacy of the use of EVM cost predictors to
tion using well-established statistical principles and assess compliance to the Sarbanes Oxley Act [9]
methods developed by Lipke show promise as: • Increased interest and the adoption of EVM by
• These techniques allow generation of a range of organizations globally.
cost and schedule predictions from user defined Where projects have not exhibited “CPI stability”
Confidence Limit(s) EVM practitioners can now know that this is neither
• All information and data required for these pre- unique, nor is it necessarily an adverse reflection on
dictions comes from within the project itself. the management or execution of those projects.
This may reduce the current dependence on Various follow‑on research opportunities arise from
heuristics developed from external project data this paper, which may develop improved understanding
sources, which might not be applicable to the of project performance characteristics and generalisable
project of interest. heuristics. Suggestions include examining the perfor-
To promote trials of these statistical prediction mance characteristics of projects where:
techniques, a Microsoft Excel “Statistical Prediction The CPI stability rule does seem applicable ( e.g.,
Calculator” is also freely available from the ES web- the subset highlighted in the Popp report data) to
site at www.earnedschedule.com/Calculator.shtml. determine whether there are project characteristics
An academic paper fully describing the statistical which result in early CPI stability
26
The Measurable News Fall 2008, Issue 4

Early CPI stability was not achieved due to pro- Christensen, David S, PhD., “Using the Earned
gressively improving CPI performance over the proj- Value Cost Management Report To Evaluate The
Contractor’s Estimate at Completion”, Acquisition
ect lifecycle. Review Quarterly, Summer 1999: 283:295.
Academically oriented research aimed at estab- http://www.dau.mil/pubs/arq/99arq/chrisevm.pdf
lishing a theoretical rationale for project perfor- (26 Nov 2007)
mance instability would be another useful addition to Lipke, Walt. “Schedule is Different,” The Measurable
the project management body of knowledge. News, 2003, March: 10–15.
While Coleman et. al. [23] provide the sobering http://www.earnedschedule.com/Docs/Schedule%20
assessment that consistent with Christensen’s find- is%20Different.pdf (12 Sep 2007)
ings “average to good programs do not improve”, Stratton, Ray. “Not Your Father’s Earned Value,”
Projects@Work, (www.projectsatwork.com), 2005,
an understanding of project characteristics, which Feb 24
result in progressively improving CPI would, if these http://www.earnedschedule.com/Docs/Not%20Your%20
characteristics could be emulated in other programs, Father%27s%20Earned%20Value.PDF
be an extremely useful advance to practice. Such (14 Nov 2007)
research could offer significant opportunities for tan- Christensen, David S., Payne, Kirk. “Cost Performance
gibly improving project performance. Stability — Fact or Fiction?,” Journal of
Parametrics, (April 1992): 10:27–40.
Research opportunities are equally applicable
http://www.suu.edu/faculty/christensend/evms/
to project schedule performance. This paper also CPIstabilityJP.pdf (12 Sep 2007)
demonstrates that by using ES, research of schedule Christensen, David S. PhD., Templin, Carl. PhD.
performance using EVM data is possible and already “EAC Evaluation Methods: Do They Still Work?,”
leading to improved understanding of the dynamics Acquisition Review Quarterly, Spring 2002: 105:116.
of project schedule and project cost performance. http://www.suu.edu/faculty/christensend/evms/
eacevalmethods4.pdf (12 Sep 2007)
Acknowledgements Christensen, D. S., Heise, S. R. “Cost Performance Index
Stability,” National Contract Management Journal,
This research has been made possible due to the gen- Vol 25, (1993): 7–15.
erous assistance of the following individuals: http://www.suu.edu/faculty/christensend/evms/
• The Project Controls Manager from the United CPIstabilityNCMJ.pdf (12 Sep 2007)
Kingdom based construction company (who Fleming, Quentin., Koppelmann, Joel. The Earned Value
desire anonymity) for making available the UK Body of Knowledge. Proceedings of the 30th Annual
construction projects EVM data Project Management Institute 1999 Seminars and
Symposium. Philadelphia, Pennsylvania, USA
• Mr. Michael Popp of NAVAIR for making avail-
Fleming, Quentin, Koppelmann, Joel. “Sarbanes‑Oxley:
able the “Popp report” and providing permission Does Compliance Require Earned Value Management
for the report to be placed in the public domain on Projects,” Contracts Management, April 2004
on the PMI Sydney Chapter website. 26:28
The support, suggestions, general assistance, and Fleming, Quentin., Koppelmann, Joel. “If EVM is Good
review comments by the ES advocates and research- … Why Isn’t it used on all Projects?,” Contracts
Management, April 2004:26–28
ers, which significantly improved this paper is also
http://www.suu.edu/faculty/christensend/evms/WhyEVM.pdf
appreciated. Responsibility for any errors, omissions (12 Sep 2007)
or erroneous conclusions remains the sole responsi- Fleming, Quentin, & Koppelman, Joel. Earned Value
bility of the authors. Project Management. 3rd ed. Upper Darby, PA:
Project Management Institute, 2005
References Henderson, Kym. “Earned Schedule: A Breakthrough
Beach, Chester Paul Jr. “A-12 Administrative Inquiry. Extension to Earned Value Theory? A Retrospective
Report to the Secretary of Navy” Department of the Analysis of Real Project Data,” The Measurable
Navy, Washington DC, 1990. News, 2003, Summer: 13–23.
http://www.suu.edu/faculty/christensend/evms/beacha-1.pdf http://www.earnedschedule.com/Docs/Earned%20
(12 Sep 2007) Schedule%20-%20A%20Breakthrough%20
Extension%20to%20EVM.pdf (12 Sep 2007)
Fall 2008, Issue 4
The Measurable News 27

Henderson, Kym. “Further Developments in Earned http://sydney.pmichapters-australia.org.


Schedule,” The Measurable News, 2004, Spring, 15-22. au/programs/customer/v_filedown.
http://www.earnedschedule.com/Docs/Further%20 asp?P=31&FID=738016087&FRF=n& (12 Sep
Developments%20in%20Earned%20Schedule.pdf 2007)
(12 Sep 2007) Coleman et al., “Predicting Final CPI”, Presentation
Henderson, Kym. “Earned Schedule in Action,” The to the 4th Joint Annual ISPA/SCEA International
Measurable News, 2005, Spring: 23–30. Conference, Orlando, Florida USA, June 2003
http://www.earnedschedule.com/Docs/Earned%20 Lipke, Walt, et al., “Prediction of Project Outcome - The
Schedule%20in%20Action.pdf (12 Sep 2007) Application of Statistical Methods to Earned Value
Management and Earned Schedule Performance
Vanhoucke, Mario, Ph. D, Vandevoorde, Stephan.
Indexes,” Publication pending.
“A Comparison of Different Project Duration
Forecasting Methods Using Earned Value Metrics,” Henderson, Kym. “Recent Advances in Project Prediction
International Journal of Project Management, Techniques”, Presentation to the IQPC IT Project
Volume 24, Issue 4 (May 2006): 289–302. Management Conference, Sydney Australia, 1 May
2007.
Note: This paper is available on-line at a cost of $USD 30
from Science Direct at: http://www.sciencedirect.com/ http://www.earnedschedule.com/Docs/Recent%20
Advances%20in%20Project%20Prediction%20
Vanhoucke Mario, Ph.D, Vandevoorde, Stephan. “A
Techniques.pdf (12 Sep 2007)
Simulation and Evaluation of Earned Value Metrics
to Forecast Project Duration,” Journal of Operational
Research Society, Volume 58: 1361–1374, Issue About the Authors
10 (October 2007); advance online publication, 13
Kym Henderson is a practicing information tech-
September, 2006
nology project manager with significant experience
Note: This paper is available on-line at a cost of $USD
10 at: www.palgrave-journals.com in project recoveries utilizing simplified EVM tech-
Attachment to e-mail from Ed Witte, SAF/AQX to Walt niques. He has a Bachelor of Business and a Master
Lipke dated 15th April 2005, “An analysis of the of Science (Computing) from the University of Tech-
Schedule Performance Index (SPI) in units of time: nology, Sydney. Kym has presented at many confer-
Overcoming the SPI($) limitations to accurately
ences internationally and published papers in various
portray schedule” performance.
publications and as proceedings of PMI Global Con-
Note: Summary of review findings leading to cancellation
of AFIT research project into the Earned Schedule gresses. He published the first independent validation
method. of the Earned Schedule method in 2003. Kym is the
Project Management Institute. Practice Standard for Immediate Past Education Director (2003–2007) of the
Earned Value Management. Project Management PMI Sydney Chapter and is the first non-US national
Institute 2004.
elected to the board of the PMI College of Performance
National Institute of Standards and Technology Dataplot,
Management commencing office as Vice President of
2005. Sign Test.
Research and Standards on 1 January 2008.
http://www.itl.nist.gov/div898/software/dataplot/refman1/
auxillar/signtest.htm (12 Sep 2007) Dr. Zwikael is a senior lecturer at the Victoria
Lipke, Walt. “A Study of the Normality of Earned Value Management School, Victoria University of Welling-
Management Indicators,” The Measurable News, ton, New Zealand. He also leads projects and pro-
2002, December: 1–16. gram groups in dozens of organisations, in Asia and
http://sydney.pmichapters-australia.org. Europe. Ofer is an accredited PMP, has acted for two
au/programs/customer/v_filedown.
years as a vice president in the Executive Board of
asp?P=31&FID=89297847&FRF=n& (12 Sep 2007)
the PMI’s Israeli chapter, and is currently a Director
Zwikael, Ofer, et al. “Evaluation of Models for
Forecasting the Final Cost of a Project,” Project at the New Zealand PMI Executive Board.
Management Journal, Volume 31, Number 1 (March
2000): 53–57.
Popp, Michael. “Probability Distributions of CPI at
Complete vs. CPI Today”, Internal NAVAIR Report,
Unpublished, 1996.
28
The Measurable News Fall 2008, Issue 4

The Trouble with Earned Schedule


By Wayne Abba

Abstract
“Earned Schedule,” touted as a major new extension of the EVM methodology, is neither new
nor profound. Through personal experience and the simple methods developed by Army EVM analysts in
the 1970s, I describe the origins of earned schedule and address errors in recently published literature that
purports to have overturned longstanding findings related to EVM data stability.

T
he Earned Value Management world has been developed an algorithm to approximate schedule
abuzz with papers and presentations touting status based on monthly reported EVM data. While
the concept of “earned schedule” (see www. this was some 30 years ago, I would not be surprised
earnedschedule.com for a description of the were I to learn that someone else had tripped to the
technique). As a past president of PMI-CPM in 2004, obvious connection even earlier, as EVM at that
I became personally involved when I helped forge point already was ten-year-old DoD policy.
a compromise in the PMI “Practice Standard for
Earned Value Management” between earned sched- Earned Schedule Isn’t Profound
ule proponents and skeptics. The standard intro- Emil’s algorithm was simple. We calculated the aver-
duced “Time-Based Schedule Measures” (now called age amount of work planned to have been performed
earned schedule) as “an emerging EVM practice” each week during a given period by dividing the
rather than giving it the endorsement that was sought planned value for the period by the number of weeks
by its more enthusiastic supporters. The compromise in the reporting period (typically four per month —
was meant to allow time for research to demonstrate the number of months used would be based on such
its value. factors as how stable the planned value was over the
Any such compromise involves give and take. I period). We then divided the cumulative schedule
overcame personal misgivings resulting from more variance by the average planned value per week to
than three decades of EVM analysis experience derive the approximate number of weeks the vari-
because I thought earned schedule showed promise ance represented from the plan. From there it was a
based on limited statistical results that conceivably simple step to relate the variance in weeks to the per-
might be demonstrated through more extensive re- cent complete (cumulative earned value divided by
search. That hoped-for result has not been achieved. budget at completion) and estimate the completion
Nevertheless, the advocacy not only continues but date. Our report to the Deputy Commanding General
now includes challenges to established EVM princi- might say, for example, “This contract is about one-
ples. This article addresses three things about earned third complete and about four weeks behind sched-
schedule that I find troubling: First, it’s not new. ule. If there are no improvements in performance, it
Second, it’s not profound. Third, its proponents are will complete at least twelve weeks later than
running roughshod over well-grounded research. planned.”
We couched our analyses carefully because we
Earned Schedule Isn’t New understood the limitations of EVM-based schedule
I learned about earned schedule soon after joining data. Even the name is a misnomer — a schedule
the Army Materiel Command headquarters staff variance is actually an accomplishment variance
as an EVM analyst in the late 1970s. My office that measures any difference in the volume of work
prepared independent assessments of monthly con- performed relative to the plan. If resources in the
tractor cost performance reports for the Command performance measurement baseline are planned
Group. Because we didn’t have access to schedule unrealistically, variance calculations may be mis-
information, the analysis branch chief, Emil Szten, leading. The variance doesn’t necessarily reflect the
Fall 2008, Issue 4
The Measurable News 29

critical path. And it will be zero at project comple- Henderson and Ofer Zwikael titled “Does Project
tion, no matter how late the work actually is com- Performance Stability Exist? A Re-examination of
pleted. These limitations are more than offset by its CPI [Cost Performance Index] and Evaluation of
proven value as an early warning indicator when SPI(t) [an earned schedule metric — Schedule Per-
work is not being accomplished — but as my former formance Index (time)] Stability” (see http://www.
Pentagon boss Gary Christle says, “if you want to stsc.hill.af.mil/crosstalk/2008/04/0804HendersonZ
understand the schedule — look at the schedule, not wikael.html) had as an objective “to reexamine CPI
at EVM data.” stability and to compare the stability behavior of the
That said, the schedule variance is derived from per- SPI(t) with CPI.” The authors’ conclusion that “the
formance against a time-phased resource consumption widely reported CPI stability rule cannot be general-
baseline linked to the schedule. It should be no surprise ized to all projects using the EVM method or even
that the information will correlate tightly with the proj- within the DoD project portfolio” is based on de-
ect schedule… it would be a problem if it did not. But monstrably faulty research.
correlation neither elevates earned schedule to a sci- The definitive research on CPI stability was per-
ence nor improves its management utility beyond the formed by graduate students at the Air Force Insti-
intuitive, simple calculations we did at Army to give tute of Technology under the guidance of Dr. David
the senior leadership a heads-up. Christensen. As the senior EVM analyst in the Office
of the Secretary of Defense in the 1990s I made the
Earned Schedule Proponents Defense Acquisition Executive Summary (DAES)
Misunderstand EVM Research database available to many researchers including
An April 2008 article published in CrossTalk, the Dr. Christensen and his students and sponsored their
Journal of Defense Software Engineering, by Kym research. The DAES database included quarterly

Humphreys & Associates, Inc.


A pragmatic, common sense approach to planning and controlling projects

Humphreys & Associates, Inc., is a management consulting firm specializing in Earned Value Project
Management. Our experience encompasses the construction and utility industries, energy programs,
the aerospace industry, and all acquisition branches of the U.S. Dept. of Defense, and foreign govern-
ments. We have consulted and supported over 500 major organizations and trained more than 450,000
individuals. The principles of project management that we espouse are applicable to any project oriented
environment. Some of our specific areas of expertise include: Proposal Preparation, Project Scheduling,
Performance Measurement, Risk Assessment, Software Evaluation, System Audits, Customer Review
Preparations, and Training.

2008–2009 Public Seminars


Earned Value Management Project US Review Team Advanced Earned Value
Systems Scheduling Workshop Management Techniques
December 3 – 5, 2008 January 27 – 29, 2009 TBA March 3 – 5, 2009
Dallas, TX Phoenix, AZ Phoenix, AZ
January 27 – 29, 2009
Phoenix, AZ

3111 North Tustin Avenue, Suite 250, Orange, CA 92865


(714) 685.1730 (Phone) (714) 685.1734 (Fax)
Email: humphreys@humphreys-assoc.com Website: http://www.humphreys-assoc.com
30
The Measurable News Fall 2008, Issue 4

summary EVM data on all major defense contracts it exists on DoD contracts. Whether the oft-quoted
that were subject to oversight by the Office of the “rule” derived from their work is true in other man-
Secretary of Defense. The resulting theses and ar- agement environments is unknown, but the article
ticles constitute an unparalleled body of knowledge does not make a case for refuting it “within the DoD
grounded in actual, verifiable performance data from project portfolio.”
hundreds of defense contracts for weapon systems, On the other hand, given the fifteen years since
vehicles, ships, airplanes, information technology Christensen’s work was published, one might hy-
and more. pothesize that defense project managers and their
Henderson and Zwikael cite “an internal DoD contractors have learned how to keep their contracts
[Naval Air Systems Command] research project” off overseers’ radar through adroit baseline manage-
from the mid-1990s and state “In contrast to Chris- ment. Any reexamination of CPI stability must take
tensen and associates research, which used data that into consideration because increasing the budget
from the DAES database, the [NAVAIR] data was base is the easiest way to remain (or get back to)
sourced from the Contracts Analysis System [CAS] “green.” Not that I’m advocating such a reexamina-
database...” They are mistaken; DAES and CAS tion ­— the literature stands on its merits and has
are in fact the same database under different names, made an enormous contribution to improving US
now part of the “Defense Acquisition Management government program management.
Information Retrieval (DAMIR)” system. Ironically, Perhaps the larger issue is how our community
my office maintained the database and would have should respond to inadequate research. Lacking a
provided access to the NAVAIR staff. journal of our own, College of Performance Man-
In 1993, building on earlier work, Christensen agement members who wish to be published must
and Captain Scott Heise published their benchmark go elsewhere, where peer reviews for our special-
paper “Cost Performance Index Stability” in the Na- ized discipline may not meet our desired standards.
tional Contract Management Journal. They analyzed The CrossTalk article undoubtedly will be cited as
data from 155 defense contracts that met EVM re- gospel, given one co-author’s identification with the
quirements and distinguished between contracts hav- College. And that’s a shame.
ing stable and unstable performance measurement As for earned schedule, I believe it’s time to take it
baselines. Henderson and Zwikael fail to note that out of the “emerging practice” box in the practice stan-
the NAVAIR study did not use comparable criteria to dard. It should be recognized for what it’s been for at
select contracts from the same source data. Nor did least thirty years — a quick and easy sanity check for
it need to — it was performed for different reasons. comparison with project schedule analysis. Nothing
There is no way to determine the extent of overlap less — but nothing more until proved otherwise.
with the Christensen-Heise research or to compare
results. About the Author
Henderson and Zwikael also include information Wayne Abba retired in 1999 as
on 37 foreign projects: twelve “hi-tech” (unspeci- the Office of the Secretary of
fied) projects from Israel, twenty construction proj- Defense’s Senior Analyst for con-
ects from the United Kingdom and five IT projects tract performance measurement.
from Australia. Of the 37, only 24 were deemed He is an independent consultant
usable for CPI stability research. They offer no specializing in EVM in the pub-
evidence that these disparate projects implemented lic sector. He is past president
EVM consistently, as on DoD contracts, and their (2001–2002) of PMI-CPM and a
analysis lacks rigor. For example, Israeli data were contributing author of the forthcoming GAO Cost


analyzed “using visual inspection of charts.” Estimating and Assessment Guide: Best Practices
Henderson and Zwikael claim to have “overturned for Developing and Managing Capital Program
longstanding findings and beliefs on CPI stability.” Costs and of chapters 7 (“Project Cost Manage-
That claim does not withstand scrutiny. Christensen ment”) and 12 (“Procurement Management”) of the
et al have shown that CPI stability as they defined PMBOK® Guide — 4th edition update.
With MCR Program Support
aligning cost and performance
is easier than you think.
If your program is over budget but
under performing, MCR can help.
With the right tools, proven processes,
and subject-matter experts you need,
MCR has the capabilities to meet your
objectives and improve cost
effectiveness and performance.

Our Core Capabilities encompass the


full spectrum of Integrated program
management, from start up to
sustainment, including:

– Risk Assessment
– Linking the Cost Estimate to the EV DAU
Performance Baseline
– EVMS Implementation/Certification
Equivalency
– Baseline Development/IBR Support for PM
– Business Case Analysis/Review Certification
– Capital Planning Consultation
– Staff Training & Development
Available
(32 Project/Acquisition
Management courses available) The MCR Training Institute (a
Defense Acquisition University
Keep your program aligned Equivalency provider) is offering
classes in Basic Earned Value
Management (BCF 102) and PM
Tools in a classroom setting taught
by experienced, certified instructors
who practice what they teach.
Contact mti@mcri.com
for more information.
MCR, LLC
2010 Corporate Ridge • Suite 350 • McLean, VA 22102 30 Years of Program/ Acquisition
703/506-4600 • Fax 703/506-8601
www.mcri.com Management Experience.
32
The Measurable News Fall 2008, Issue 4

You might also like