Software Metrics

Prepared for:
NAVAIR Software Competency
Prepared by:
Pete Christensen, John Kennedy, and Tom Ullrich
pchris@mitre.org, jkennedy@mitre.org,
tullrich@mitre.org
28 February 2001
Summary Slide
O DOD Impetus for Software Measurement
O What are Metrics?
O What can we do with Metrics?
O What Metrics are Appropriate?
O The PSM Approach
O Selecting the Right Measures
O How Good is Your Developer?
O Guidelines for Collecting Measures
O PSM and PEO(TSC)
O Recommendations

2
DOD Impetus for Software Measurement
O Most complex systems in procurement by DoD are software
intensive*
O “Software is highest risk component in DoD programs”*
O 1995 Standish Survey polled over 800 software developers
-52.7% of all projects were completed but incurred cost
and schedule overruns **
-Average cost overrun was 189% **
-Average schedule overrun was 222% **
-31.1% of all projects were cancelled **
-Challenged projects (the 52.7%) were delivered with only
61% of originally specified functionality **
O If you ain’t measuring, you ain’t managing!
* 95 Guidelines for Successful Acquisition and
Management of Software Intensive Systems
** 95 Standish Survey

3
 The Cost of Locating and Correcting Errors*

Relative Cost Relative

Errors Made During Errors Found During
Cost Multiplier Cost

0.1 X Requirements 2700 Beta Test 270 X

0.15 X Definition 600 System Test 90 X

0.3 X Decomposition 100 Integration Test 30 X

1.0 X Build 1 Unit Test 1.0 X

4
* Large System Study World Wide Military Command and Control System
What are Metrics?
O A metric is a measure of some aspect of a program, design,
or algorithm.
- It can be systematically calculated
- It can be used to make inferences about that program,
design, or algorithm.
O By systematically calculating values for programs of known
complexity
- We can infer the complexity of other programs from their
calculated values.
O For example:
- We know that programs, designs, and algorithms with y
value for metric x had problem z
O High defect rate, poor maintainability, etc.
- If my program has those metrics
O It will probably have similar problems

5
Measure Vs Metric
O Measures vs Metrics
- Measures are the numbers used to create the metrics
- Metrics are the numbers turned into information
- That information is used by managers to create great
software on time and under budget
- Measure: Rank: A linearly independent path through a
module
O Metric: Cyclomatic Complexity = 25

The total number of linearly independent paths

through a module
- Measure: Dollars Budgeted (BCWP)
- Measure: Dollars expended (ACWP)
O Metric: Cost Performance Index

6
What can we do with Metrics?
O Manage risk
- The probability of an issue becoming a problem!
O Managers can no longer manage by the seat of their pants
They need:
- Insight into the performance of the developer
- An objective, quantitative basis for evaluating product
quality and analyzing issues/problems
- A foundation for quantitative control of the program
management and engineering processes.
O Program performance vs program plans
O Cost and schedule control
O Quality & Configuration Control
O Defect tracking
O Staffing
O Process improvement

7
Candidate Modules for Simplification/Reuse
cad_cmn_symbol_database.symbol_state_ cad_cmn_symbol_database.symbol_state_
data_task().read_declutter_symbol_data(…) data_task().write_declutter_symbol_data(…)

8
Complex, well structured and essentially identical! Decomposition may appropriate in later builds.
Simplifies downstream maintenance and testing.
Cost Performance Metrics

S/W Cost

9000

8000

7000

6000 Where we are supposed to be

5000
$K

4000 What we have spent to

3000
get to this point

2000
Where we are BCWS
1000
BCWP
ACWP
0

102

106

110
104

108
9912

204
9906

9908

9910

112

202

206
4

8
2

10

12

Date

9
What Metrics are Appropriate?
O Program Managers must invest time and effort to collect
analyze and respond to metrics
O Metrics should be collected that relate to significant project
risks
- How volatile are my requirements? (size)
- How is my productivity? (cost & schedule)
- How mature is this release of software? (defects)
O Recommend focusing on a small number of useful metrics
- There is no panacea
- No one metric or class of metric is always appropriate
O OK...
- What are the right measures?

10
Some Measures Are Mandated
O NAVAIR AVION Instruction 5235.1 Dtd 18 Jun 1992
- AVIONICS Software Metrics
O Nine typical measures addressed
1 :Requirements Volatility
2 :Software Size
3 :Software Personnel and Effort
4 :Cyclomatic Complexity
5 :CSU Development Progress
6 :SPR Status
7 :Build Release Content
8 :Computer Resource Utilization
9 :Milestone Performance

11
The Need for A Disciplined Measurement
Methodology
O OUSD(A&T) investigated software development processes
- Measures were being collected that:
O Provided no insight into program goals or potential
areas of concern
O Increased development costs
O Were inconsistent or used inappropriately
- The process needed to be better managed
O See the Practical Software Measurement Web Site
O www.psmsc.com/
O PSM Discussion Forum on software and systems
measurement issues.
O Download the latest version of the PSM Guide
O Products & Services, News, Information Exchange

12
The PSM Approach
O The PSM approach defines three basic measurement
activities
- Tailor Measures, Apply Measures and Implement
Process
O Tailor the Measures:
- Identify/select effective and economical measures for the
project
O Apply the Measures:
- Collect, analyze, and act upon the data
O Implement the Process:
- Address the cultural and organizational issues
O The measurement process must be integrated into the
Contractor’s Software Development Process

13
The Practical Software Measurement Process
Project
Team

Revisit
Objectives, Issues and Software Process
Characteristics
Tailor
Measures
Data
Apply and
Measures Feedback
Objectives, Issues and Software Process
Characteristics
Implement
Collect, Analyze and Act! Process

14
PSM Identified Objectives and Issues
O Software project objectives fall into seven Common Issues
Areas
- Schedule and Progress
- Resources and Cost
- Product Size and Stability
- Product Quality
- Process Performance
- Technology Effectiveness
- Customer Satisfaction
O These seven Issue Areas are broken into
- 21 Measurement Categories and
O 50 discrete measures

O The Practical Software Measurement Manual Refers!

15
PSM Standard Issue, Category, Measures
Mapping

16
Selecting the Right Measures
O Project objectives and issues drive what will be measured
O Objectives are goals and requirements usually expressed in
terms of functionality, cost, schedule and quality
- Measures help managers determine if objectives have
been achieved
O Issues are areas of concern that present obstacles to
achieving objectives such as problems, risks, or lack of
information
- Measures provide insight into issues throughout the
project
O Risk analysis at project inception facilitates identification
and prioritization of issues
- Management can prevent risks from becoming actual
problems by applying the correct measures

17
Selecting the Right Measures
O Invest the time and resources to select measures that
- Address and permit objective evaluation of project
objectives and issues
- Are effective and economical for the project
- Help to quantify product quality, evaluate the
development process, and analyze problems
O Candidates for measurement include new software products
and artifacts (COTS or reuse) delivered
O Measures should be relevant and useful to both the Project
Manager and the technical manager/developer

Don’t measure something unless it provides

insight into objectives and issues

18
Correlation Between Product Quality Measures
20

16
15
14
Cyclomatic Complexity

SLOC

Static Path
# Files

Function Calls

Nesting Depth

Parameter Count

Measures
Correlation Between Measures in Complex Functions
19
Correlation Between Product Quality Measures:
Cyclomatic Complexity and Static Path Count
Cyclomatic Complexity = 10
Stat Path

1.00E+09

1.00E+08

1.00E+07

Low Complexity High Complexity

High Static Path Count High Static Path Count
1.00E+06
Static Path Count

Functions not normally 15.5% of Functions

(15.5%)
(0%)
expected in this Quadrant fall in this Quadrant
1.00E+05

Static Path
1.00E+04
= 1000
1.00E+03

73% of Functions fall 11.2% of Functions

1

10

..
in this Quadrant 1.00E+02 fall in this Quadrant
(73%) (11.5%)
1.00E+01
Low Complexity High Complexity
Low Path Count Low Path Count
1.00E+00

Cyclomatic Complexity

20
Be Specific and Document What You Want!
O Develop and disseminate explicit data definitions
- What’s in a name?
O SEI recommends developing a data definition checklist and
a schedule checklist for each type of data
O The PSM Guide has standard definition tables for issues,
measurement categories, and measures
O Document and coordinate requirements for collection and
reporting in formal plans
- Put it on Contract!
- Document measures in
O Software Measurement Plan, Risk Management Plan
and Financial Performance Plans
- Then use what you get!

21
Development Process and Environment Drive
What Measures Are Collected
O Management objectives and issues drive what is collected
O The Contractor’s software development process drives how
measurements will be taken
O Integrating PM’s requirements with development reality
involves answering three major questions
- What is the software development environment?
O Ada vs. Java, Windows vs. UNIX, Sun vs. Power PC
- Where can I measure in the development process?
O Waterfall, Spiral, OOA&D
- What measures does my environment and process
support?
O Data available from the current process may or may
not map to the ideal measurement requirements
O If requirements are critical, changes may be made to
software measurement approach

22
How Good is Your Developer?
O Measures should accurately represent the developer’s
process, including new management or technical practices
O CMM Level III is mandated for ACAT Level 1 Programs
- NAVAIR has expanded the requirement
- Your contractor should be using repeatable processes!
O If there is a well defined process
- measurements will provide useful information
conversely...
O If processes are ad-hoc
- it will be difficult to determine what is being measured
- Refer to slide #3!

23
Guidelines for Collecting Measures
O Collect and analyze measures at a level of detail sufficient to
identify and isolate software problems
O Granularity:
- Measures should be collected, processed and analyzed
often enough to be useful to managers and
- In enough detail to allow insight into the software
component and software activity level
O Independent analysis
- Developer and PM may have different priorities!
O Profit vs. Schedule and Performance!
- Objective communication can only be achieved when
both parties understand the data under discussion
O If the metrics lack independent analysis or granularity they
have questionable value!

24
Using Measures: Caution!
O Individual Software Developers may resent being measured
- They should be involved in the definition of metrics
O Do not use measures to find fault with individuals
- Fault is most likely due to
O Processes, Procedures, Tools or estimates
O If you measure people without their knowledge
- You may get accurate data until they find out about it
O Then you loose accuracy and trust
O Training may be required
O Consider all measures in conjunction with current cost, risks
and development environment
O Integrate software measurement into the management and
development processes throughout the software lifecycle

25
PSM and PEO (TSC)
O PEO (TSC) has implemented PSM
O Goals include:
- Reduce software upgrade & verification time & cost
- Increase the use of Common software components
- Meet performance & functional requirements
- Portability
- Maintenance affordability
O Program Managers are responsible for
- Developing and establishing a metrics program
- Collecting and analyzing the metrics and taking actions
as necessary
- Reporting standard metrics to the Technical Director

26
PEO (TSC) Software Measurement Strategy
O Provide Progress against plans
O Determine the extent to which system requirements are
being met
O Identify and monitor Critical product attributes such as
quality and performance
O Gain timely and accurate information about the efficiency
and effectiveness of the software processes
O Understand the status of current and impending risks
O Developed a standard set of core metrics based upon PSM
to help PMs select appropriate measures
O PEO TSC INSTRUCTION 4810.21 11 Jan 2000
- SOFTWARE DEVELOPMENT, VERIFICATION AND
VALIDATION (V&V), DESIGN REUSE, AND METRICS
POLICY

27
PEO (TSC) Software Measurement Guidelines By
Development Phase
Metrics Core Metrics Report During Activity
Category RA AD DD C&T I QT SI SQT
Schedule and Earned value schedule ) ) ) ) ) ) ) )
progress Milestone status ) ) ) ) ) ) ) )
Progress metrics
* * * * * * * *
Resources and Earned value cost ) ) ) ) ) ) ) )
cost Staff loading * * * * * * * *
Effort history
* * * * * * * *
Growth and Requirements stability ) ) ) ) ) ) ) )
stability Requirements ambiguity * * * * * * * *
Requirements completeness
Code size and content
) ) ) ) ) ) ) )
Code stability ) ) ) ) )
Build stability ) ) ) )
Requirements slippage ) ) ) ) )
) ) ) ) )
Product quality TR history and status ) ) ) )
CR history and status ) ) ) ) * * * *
Failure density
Requirements test coverage
) ) ) )
Structural test coverage ) ) ) )
Test effectiveness ) ) ) )
TR discovery rate ) ) ) )
) ) ) )
Development Earned value performance ) ) ) ) ) ) ) )
performance Age of Open TRs * * * *
Age of Closed TRs
Productivity
* * * *
Defect containment ) ) ) )
Defect removal ) ) ) )
effectiveness ) ) ) )
Technical Product performance ) ) ) ) ) )
adequacy
) Report to PEO * Report to PM (available to PEO) Gray = updates with changes only

28
Recommendations
O Review your contract and planning documents
- What metrics are required?
- Are they the right metrics?
O Go to the Practical Software Measurement web site and
start reading (www.psmsc.com/)
- What measures would be more helpful?
- What measures could you stop taking?
O Sign up for Practical Software Measurement Training
- Learn how to select, analyze, and utilize measures
O Implement Practical Software Measurement
- Deliver a high quality product on time and at or below
the estimated cost. Boring is good!

29
And Finally...

“When you cannot measure, when you cannot

express it in numbers, your knowledge is of a
meager and unsatisfactory kind: it may be the
beginning of knowledge, but you have scarcely, in
your thoughts, advanced to the stage of a
science.”

Lord Kelvin, 1800s

30
Questions?

31