Statistical Process Control (SPC) For Software Tutorial: Topics

Carnegie Mellon University
Software Engineering Institute
Statistical Process
Control (SPC) for
Software Tutorial
Anita D. Carleton and Mark C. Paulk
August 25, 1997 (SEI Symposium)

Pittsburgh, PA 15213-3890
Sponsored by the U.S. Department of Defense

© 1997 by Carnegie Mellon University 1

Topics
Introduction
Role of Measurement in Process Management
Activities
Process Measurement Perspectives and Issues
Analyzing Process Performance
Acting on the Results
Summary
Page 1
1
Module Objectives
Set expectations for the tutorial
Understand the types of skills and knowledge

that can be obtained from the tutorial
Outline the agenda

Outline
Who Are We?
Who Should Be Here?
What Will We Do?
What Will You Walk Away With?
Agenda
Page 2
2
About the SEI

Federally Funded Research
and Development Center
(FFRDC)
Awarded to Carnegie Mellon

University (CMU) based on
competitive procurement
Managed by Office of the

Secretary of Defense (OSD)
Acquisition Program
Integration (API)
Administered by Electronic
Systems Center (ESC)

Who Should Be Here?

Software Engineering Process Group
Quality Improvement Groups
Project Managers
Process Action Teams
Process Managers/Owners
Working Groups
Professionals
Professionalswhose
whosekey
keyfocus
focusis
ison
onusing
usingmeasurement
measurement
to
tomanage
manageand
andimprove
improvesoftware
softwareprocesses.
processes.
6
Page 3
3
Do You Have Questions Like...

Why should I use SPC? How do I address issues
of normalization and aggregation?
What is the business value of SPC?
When should I apply SPC
How can I apply SPC ? techniques? What should be
put under statistical control?
techniques to my
software Issues?
What SPC techniques

should senior
How much data to I need management be using?
to use SPC techniques?
What control charts should How do I know what

I use and why? techniques to apply to
what measures for which
How can I avoid using SPC for purposes?
7
performance evaluation?

What Will We Do?

Introduce and help you gain familiarity with some
useful processes, practices, and methods for
• using measurement data to control and
improve process performance
• understanding issues and examples of what
organizations are facing as they address
statistical process control for software
• understanding what it means to control and
predict the software process
Page 4
4
What Will You Walk Away With?

A conceptual understanding of
• the concepts and issues that lie behind the
statistical methods
• the steps associated with effectively
implementing and using the methods to
manage and improve software processes
• ways to get started in applying quantitative
principles to your work

Practical Software Measurement: Measuring for

Process Management and Improvement
The Role of Measurement in Process
Management
Practical
Software The Perspectives of Process
Measurement: Measurement
Measuring for
Planning Measurement for Process
Process Management
Management and
Improvement Applying Measurement to Process
Management
More About Analysis and Use of Process

Measures
Principles of Successful Process

Measurement
10
Page 5
5
Topics
Introduction
Role of Measurement in Process

Management Activities
Summary
11

Module Objectives
Understand the role of measurement in the
context of software process management
Understand that measures must be targeted to

support the business goals of your organization
Understand that the concept of process

management is founded on the principles of
statistical process control
12
Page 6
6
Outline
Process measures are driven by goals and issues
What is software process management?
Objectives of software process management
13

Without the right information,

you’re just another person with
an opinion.
Tracey O’Rourke, CEO of Allen-Bradley
14
Page 7
7
Every Organization Asks

Questions Like...
Are our customers satisfied
Are we achieving with our products and services?
the results we desire?
Are we achieving the Are we getting better?

growth required for ?
survival?
Are we meeting our
business objectives?
How can we
improve our
competitive Are we earning a fair
position? return on our
investment?
How can we improve response Can we reduce the cost of

to our customers’needs or increase the producing the product or
functionality of our products? service?
15

Measurement Clarifies and

Focuses Understanding of These
Issues
… but it all starts with business goals.
16
Page 8
8
Business Goals
Goals, objectives, strategies, and plans in all
organizations are based on two fundamental
needs.
1. providing competitive products or services
in terms of functionality, time-to-market,
quality, and cost
2. meeting commitments to customers with
respect to products and services
Success in meeting commitments means that

commitments must be achievable. This implies
the need to predict outcomes.
17

Why Measure Processes?

Processes are critical to executing strategies
and plans aimed at business objectives.
Processes must be controlled, predicted, and
improved to achieve organizational goals.
Measurement is needed to control, predict, and
improve processes. These are fundamental
management responsibilities.
18
Page 9
9
A Process Is the Organization of…
People Material Energy Equipment Procedures
Requirements
Ideas Products
Time Work activities &
Services
19

The Process Management Paradigm

Measure process performance
No Remove
Is process assignable
stable? causes
Yes
Is process No Change
capable? process
Yes
Continual
improvement
Measurement
Action 20
Page 10
10
Process Management
Is about successfully managing the work
activities you use for developing,
maintaining, and supporting software
products.
Is founded on the principles of statistical

process control.
21

Four Key Responsibilities of Process

Management
Define the
process Improve
Process
Measure the
process Define Control Measure
Process Process Process
Control the
process
Execute
Process
Improve the
process
22
Page 11
11
Define the Process

Design the processes that can meet or support
business and technical objectives.
Identify and define the issues, models, and
measures that relate to the performance of the
processes.
Provide the infrastructures (the set of methods,
people, and practices) that guide and support
software activities.
Ensure that the software organization has the
ability to execute and sustain the processes (skills,
training, tools, facilities, funds).
23

Measure the Process

Collect data that measure the performance of
each process.
Analyze the performance of each process
Retain and use the data to

• assess process stability and capability
• interpret the results of observations and
analyses
• predict future costs and performance
• provide baselines and benchmarks
• plot trends
• identify opportunities for improvement
24
Page 12
12
Control the Process

Determine whether or not the process is under
control (i.e., stable with respect to the mean and
inherent variance of measured performance).
Identify performance variations that are caused

by process anomalies (assignable causes).
Eliminate the sources of assignable causes, so

as to stabilize the process.
25

Improve the Process

Understand the characteristics of existing
processes and the factors that affect process
capability.
Plan, justify, and implement actions that will

modify the processes so as to better meet
business needs.
Assess the impacts and benefits gained and

compare these to the costs of changes made to
the processes.
26
Page 13
13
Relating Clarify Business Goals &Strategy Yes
Measurement to Identify and Prioritize Issues

New Goals,
Strategy?
Process Yes
Management—
No
Select and Define Measures New Issues?
The Bigger No
Yes
Picture Measure process performance New

Measures?
Is No Remove
process assignable
stable? causes
Yes
Is No
process Change
capable? process
Yes
Continual
improvement 27

Topics
Introduction

Activities
Process Measurement Perspectives

and Issues
Summary
28
Page 14
14
Module Objectives
Outline five perspectives that are central to
process management
29

Outline
Process performance
Stability
Compliance
Capability
Improvement and Investment
30
Page 15
15
Process Performance
Concern: What is the process producing now
with respect to measurable attributes of quality,
quantity, cost, and time?
Process performance can be determined in two

ways
• by measuring attributes of products that the
process produces
• by measuring attributes of the process itself
Business value: foundation for predicting and

improving quality, cost, and cycle time
31

Stability
Concern: Is the process that we are managing
behaving predictably?
Business value: foundation for estimating

(predicting) and making commitments
32
Page 16
16
Compliance
Concerns: Are the processes sufficiently
supported?
Are they faithfully executed?
Is the organization fit to execute the process?
Business value: foundation for controlling and

directing
33

Capability
Concerns: Is the process capable of delivering
products that meet requirements?
Does the performance of the process meet the

business needs of the organization?
Business value: foundation for making

commitments
34
Page 17
17
Improvement
Concerns: What can we do to improve the
performance of the process?
What would enable us to reduce the variability?
What would let us move the mean to a more

profitable level?
How do we know that the changes we have

introduced are working?
Business value: foundation for creating

competitive advantage
35

Process Management
Perspectives

Process
Performance
Process
Improve
Process
Control Define
Process
Control
Process
Measure
Process
Variability
Execute
Process
Stability
Capability
36
Page 18
18
Process Performance & Control

Process performance
• the manner in which the process fulfills its
purpose
Process performance measures

• quantify the degree to which the process is
fulfilling its purpose
Control — making the process behave

consistently
• allows prediction of results
• gives basis for measurable process
improvement
37

Process Management
Perspectives
Process
Performance
Improve
Process
Process
Control Define Control Measure
Variability
 Execute
Stability
Process
Capability
38
Page 19
19
Process Performance Variation and

Stability
Variation = process noise + process anomalies
Stable process = process anomalies
39

An Example Process in Statistical

Control
x
x x
x
x x
x x
x xx
Frequency x
of x
Measured x x
Values x
time
x x
x
x
Variation in
Measured Values
40
Page 20
20
An Example Out-of-Control
Process
x
x
x
x
x x
x x
xx
xx x x
Frequency x x
of
Measurement
Value time
xx
x x
Variation in
Measurement
41

Process Management
Perspectives
Process
Performance Improve
Process
Process
Control Define
Process
Control
Process
Measure
Process
Variability
Execute
Process
Stability
Capability
42
Page 21
21
Process Capability
Process capability = “Voice of the Process”
 stable process
Capable process =  +
 product conformance
Capability ratios
• compare the capability of a predictable process
to the specifications
• are attempts to compare the “voice of the
process” to the “voice of the customer”
Process capability DOES NOT EQUATE TO capable process.

The Concept of Process Capability

Elapsed Time Between Problem
Evaluation and Resolution
18
16
Customer
14
Requirement
12
Target
10
Upper Limit
Frequency 8 of 12 days
6
0
1 2 3 4 5 6 7 8 9
Days
UCL=10.7
Process Mean = 4.7 days
Days
Voice of the Process
44
Page 22
22
Upper Spec
Reduce Variation
Changing
Process Upper Spec
Capability Change the Specs
Mean
Upper Spec
Shift the Aim
45


No Remove
stable? causes
Yes
capable? process
Yes
Continual
improvement
Measurement
Action
46
Page 23
23
SPC for Software Premises

The software process is performed by people, not
machines.
The software process is (or can be) repeatable, but
not repetitive.
The act of measuring and analyzing will change
behavior – potentially in dysfunctional ways.
47

SPC Questions
Can the software process be stable? Is it?
Can the software process be capable? On what

dimensions – cost, schedule, quality?
Will control charts and other statistical process

control techniques provide timely insight for
process implementers?
48
Page 24
24
Topics
Introduction

Activities

Summary
49

Module Objectives
Describe some important analytic tools and show
how the tools can be used to help assess the
stability, capability, and performance of a
software process or activity
Understand the quantitative issues that underlie

effective use of control charts for guiding
process management and improvement actions
50
Page 25
25
Outline
Separating signals from noise
Evaluating process stability
Control charts
Evaluating process capability
51

Interpretation Requires Analysis
Data Analysis Interpretation
Input Transformation Output
52
Page 26
26
Two Ways to Obtain Statistical

Inferences
Enumerative Studies - action will be taken on the
materials within the frame studied.
• The aim of an enumerative study is descriptive:
to determine “how many” as opposed to “why
so many.”
Analytic Studies - action will be taken on the

process (or cause system) that produced the
data, not on the materials within the frame from
which the data came.
• The aim of an analytic study is to predict or
improve the behavior of the process in the
future. 53

Enumerative Studies -1
Software engineering examples of enumerative
studies include
• inspections of code modules to detect and
count existing defects
• functional or system testing of a software
product to ascertain the extent to which a
product has certain qualities
• measurement of software size to determine
project status or the amount of software under
configuration control
• measurement of staff hours expended, so that
the results can be used to bill customers or
track expenditures against budgets
54
Page 27
27
Enumerative Studies -2
An enumerative study answers questions like
• How many defects were found by inspecting the product
code?
• How many problem reports have been received from
customers?
• How many user manuals do we have on hand?
• How many employees are there in our organization?
• What percent have been trained in object-oriented
design methods?
• How large were the last five products we delivered?
• What was the average size of our code inspection teams
last year?
• How many staff hours were spent on software rework
last month?
55

Analytic Studies
Examples of analytical studies include
• evaluating software tools, technologies, or
methods— for the purpose of selecting among
them for future use
• tracking defect discovery rates to predict
product release dates
• evaluating defect discovery profiles to identify
focal areas for process improvement
• predicting schedules, costs, or operational
reliability
• using control charts to stabilize and improve
software processes or to assess process
capability
56
Page 28
28
Statistical Process Control

A phenomenon will be said to be
controlled when, through the use of
past experience, we can predict, at
least within limits, how the
phenomenon may be expected to
vary in the future.
Walter A. Shewhart, 1931
57

Stability Concepts
Stable process = Process In Statistical
Control
= Sources of Variability Due
to Common Causes
Quality Quality Quality
Planning Control Improvement
Sporadic
spike
Original zone of quality control
Cost of New zone of quality control

poor Chronic
quality waste
Lessons Learned
J.M. Juran Wilton, CT
Used with the express permission of the Juran Institute, Aug 1990. 58
Page 29
29
Why is it Important to Address

Stability? -1
Without stability and the associated knowledge of
what a process can do, we have no way to
distinguish signals in measured values from the
random noise that accompanies them.
Measurements then easily lead to inappropriate
actions.
Without a history of stable performance,

uncontrolled excursions can occur at any time.
We then have no rational basis for extrapolating
observed performance to future situations, and
all plans that we make are at risk.
59

Why is it Important to Address

Stability? -2
Without knowing what the stable level of
performance is, we have no basis for recognizing
assignable causes that signal improvement
opportunities.
Without stability, we have no repeatable process

to use as a baseline for process improvement. In
fact, some would say that we have not one
process, but many— all different. Effects of
improvement actions may then be
indistinguishable from other assignable causes.
60
Page 30
30
An Example Process in Statistical

Control
x
x x
x
x x
x x
x xx
Frequency x
of x
Measured x x
Values x
time
x x
x
x
Variation in
Measured Values
61

X-Bar and Range Charts for a

Process that Is In Control
UCL
O O
Average O O CLx
(X-bar) O
LCL
t1 t2 t3 t4 t5
UCLR
Range O
O
(R) CLR
O O
O
t1 t2 t3 t4 t5
62
Page 31
31
An Example Out-of-Control
Process
x
x
x
x
x x
x x
xx
xx x x
Frequency x x
of
Measurement
Value time
xx
x x
Variation in
Measurement
63

X-Bar and Range Charts for a

Process that Is Out of Control
O
UCL
O
O
Average O
O CLx
(X-bar)
LCL
t1 t2 t3 t4 t5
O
UCLR
Range O
(R) CLR
O O
O
t1 t2 t3 t4 t5
64
Page 32
32
Why Control Charts? -1

Backlog of Unresolved Problem Reports
Control charts UCL=32.9
let you know what 30
26
your processes can Individual

Values
22
18
CL=20.4
14
do, so that you can 10
LCL=7.89
set achievable goals. 0 5 10 15 20 25 30
They represent the “voice of the process.”

Control charts provide the evidence of stability that
justifies predicting process performance.
65

Why Control Charts? -2
Control charts separate O

UCL
signal from noise, so O
Average O O
O CLx
that you can recognize (X-bar)
a process change when LCL
t1 t2 t3 t4 t5
it occurs.
Control charts identify unusual events. They point
you to fixable problems and to potential process
improvements.
66
Page 33
33
Notice what the control charts

do— they seek to identify if the
process is behaving one way or
another. This, in effect, is the same
as asking if the process exists as a
well-defined entity, where the past
can be used to predict the future, or
if the process is so ill-defined and
unpredictable that the past gives
little clue to the future.
Donald J. Wheeler, 1995
67

Control Charts
Two broad classes of control charts
• variable data, which is continuous
• attribute data, which is discrete
Choice of what control chart to use should be

based on knowing the right assumptions!
Use the correct formulas for the kind of control

chart selected!
68
Page 34
34
Why Is It Important to
Understand the Distinction
Between Variables Data and
Attributes Data?
Because control limits for attributes data are often
computed in ways quite different from control
limits for variables data.
Unless you have a clear understanding of the

distinctions between the two kinds of data, you
can easily fall victim to inappropriate control
charting methods.
69

The Distinction Between Variables

Data and Attributes Data - 1
Variables data (sometimes called measurement
data) are usually measurements of continuous
phenomena.
Examples: measurements of length, weight,

height, volume, voltage, horsepower, torque,
efficiency, speed, and viscosity.
Software examples: elapsed time, effort

expended, years of experience, memory
utilization, cpu utilization, and cost of rework.
70
Page 35
35
The Distinction Between Variables

Data and Attributes Data -2
Attributes data occur when information is recorded
only about whether an item conforms or fails to
conform to a specified criterion or set of criteria.
Attributes data almost always originate as counts.
Examples: the number of defects found, the

number of defective items found, the number of
source statements of a given type, the number of
lines of comments in a module of n lines, the
number of people with certain skills or experience
on a project or team, and the percent of projects
using formal code inspections.
71

Detecting Instabilities and Out-

of-Control Situations -1
To test for instabilities in processes, we examine
control charts for instances and patterns that
signal nonrandom behavior.
Values falling outside the control limits and

unusual patterns within the running record
suggest that assignable causes exist.
72
Page 36
36
Detecting Instabilities and Out-of-

Control Situations -2
Test 1: A single point falls outside the 3-sigma
control limits.
Test 2: At least two of three successive values fall on
the same side of, and more than two sigma
units away from, the center line.
Test 3: At least four out of five successive values fall
on the same side of, and more than one sigma
unit away from, the center line.
Test 4: At least eight successive values fall on the
same side of the center line.
Source: Western Electric Handbook
73

Steps for 1
Select the process

2
Identify the product or process
characteristics that describe
Using Control
process performance
Charts to
Select the appropriate
control charts
Evaluate
4
over a period of time
Process 5
Use appropriate calculations
Stability
based on measurement data
to determine the center lines
and control limits for the
performance characteristics
6 10
Plot the measurement data on Identify and remove
the control charts assignable causes
8 7 9
Are all
Yes measured values No
Process is
within limits and
stable, continue Process is not stable
distributed randomly
measuring
around the
centerlines?
74
Page 37
37
Useful Control Charts

Hypothesis: begin with control charts for defect
data.
Most likely to be of value for software processes

• u-chart
• Z-chart
• XmR chart for attributes data
75

u-chart Assumptions
Assumes a Poisson process
Based on count of discrete events
Discrete events occur within well-defined, finite

regions
Events are independent
Events are comparatively rare
76
Page 38
38
u-chart Formulas
CLu = u = Σ ui / Σ ai
UCLu = u + 3 √ (u / ai)
LCLu = u - 3 √ (u / ai) Defects/ KSLOC for Inspected Modules
60
50
40 UCL
30
20 CL=19.5
10
0 LCL
0 5 10 15 20 25
Module Number
77

Z-chart
Scales u-chart in sigma units
Avoids variable control limits
σi = √ (u / ai) Defect Density of Inspected Modules
Z i = (ui - u) / σi 3
0
1
-1
3 5 7 9 11 13 15 17 19 21 23 25
Module
Number
78
Page 39
39
XmR Chart
Assumes standard deviation (sigma) is constant
across all observations
Does not assume Poisson or binomial data
Useful cross-check for u-chart or Z-chart!
79

XmR Chart
UCL=32.9
30
26
22
Individual CL=20.4
Values 18
14
10
LCL=7.89
0 5 10 15 20 25 30
16 UCL=15.4
14
12
Moving 10
Range 8
6
4
CL=4.7
2
0
0 5 10 15 20 25 30
Weeks of System
. Test.
Individual Values (X) and Moving-Range Charts

80
Page 40
40
Other Statistical Techniques

Control charts are not the only statistical
techniques that provide insight.
• confidence intervals
• prediction intervals
• test of hypotheses
• design of experiments
Most are outside the scope of this tutorial.
Even simple “run charts” can provide valuable

insights.
81

Economic Justification
Control charts are important for
• identifying special causes of variation and
removing them
• balancing between false alarms and missed
signals
• determining process stability and capability
Empirically, control charts are effective tools.
82
Page 41
41
Topics
Introduction

Activities

Summary
83

Module Objectives
Put measurement in the context of software
project management and software process
improvement.
Examine some obstacles and reasons for failure.
Outline the benefits derived from measuring

software products, processes, and resources.
84
Page 42
42
Outline
Finding and correcting assignable causes
Where to look when seeking improvements
Tools for finding root causes and solutions
85

Additional Roles of Measurement

When process measurements are analyzed, the
results point to one of three investigative
directions
• stability
• capability
• improvement
86
Page 43
43
Actions That
Clarify Business Goals &Strategy Yes
Follow
New Goals,
Identify and Prioritize Issues Strategy?
Evaluations of No
Yes
Process
Yes
Performance Measure process performance

No
New
Measures?
Is No Remove
process assignable
stable? causes
Yes
Is No
process Change
capable? process
Yes
Continual
improvement 87

More to Measuring for Process

Management and Improvement
• What is causing the problems that we see?
• Which factors should we change when we
wish to improve?
• How large a change should we make?
• What are the costs and benefits now? How will
they be after the changes? What will cost to
make the changes? How long will it take?
• What are the other characteristics of the
product, process, system, environment,
resources, and customer that we are
addressing?
88
Page 44
44
Need for Additional Information

Three reasons for gathering additional data
• to support or invalidate hypotheses about
possible causes of instability
• to identify, verify, or quantify cause-effect
relationships, so that the right amounts of the
changes can be made
• to estimate (or confirm) the business value of
the actions that are proposed (or taken) to
improve the process
89

Establishing Stability
90
Page 45
45
Finding and Correcting

Assignable Causes
Isolate the problem(s)
Assemble a group of people who work within the process

to brainstorm possible reasons for the unusual problem or
failure
Measure key parameters of some of the inputs to the

process and plot run charts (or contro charts) to see if
anything unusual is happening
Look at parallel input streams of supposedly similar items

as sources of erratic behavior when the inputs have
different characteristics
Measure and/or plot characteristics of intermediate

products 91

Noncompliance as an Assignable
Cause
When investigating compliance as a potential
source of instability, examine
• adherence to the process
• fitness and use of people, tools, technology,
and procedures
• fitness and use of support systems
• organizational factors, such as management
support, organizational changes, personnel
turnover, relocations, downsizing, and so forth
92
Page 46
46
Improve the Process
93

Business Questions
Is my process performing according to my
expectations?
Is a change in the numbers significant?

• likely to “overcontrol” without control charts
Can I localize a problem?

• process mostly stable with an isolated glitch
(special cause)
How confident can I be in my understanding of

the process?
• gut feel versus management by fact
94
Page 47
47
Where to Look When Seeking

Improvements
There are two places to look when seeking ways
to improve process performance
• process activities and subprocesses
• things used within the process that originate
outside the process
95

Inputs That Affect Process

Performance
E n titie s O rig in atin g O u tsid e a P ro ce ss
W h o se A ttrib u te s Can A ffe ct P ro ce ss
P e rfo rman ce
products and by-products guidelines and directions
from other processes • policies
resources • procedures
• people • goals
• facilities • constraints
• tools • rules
• raw materials • laws
• energy • regulations
• money • training
• time • instructions
96
Page 48
48
Tools for Finding Root

Causes and Solutions
97

Tools for Finding Root Causes

and Solutions
Scatter Diagrams
Run Charts
Cause-and-Effect Diagrams
Histograms
Bar Charts
Pareto Charts
98
Page 49
49
Application Areas for Analytic Tools
Problem Problem
Identification Analysis
histogram
Pareto chart scatter diagram

check sheet
cause & effect control chart
diagram
flow chart regression
brainstorming analysis
run chart
pie chart process
stratification capability
nominal group bar graphs analysis
technique
interrelationship force field
digraph analysis
affinity diagram design of
experiments management
presentation
checklist
99

Scatter Diagrams
Display empirically observed relationships
between two process characteristics.
A pattern in the plotted points may suggest that

the two factors are associated, perhaps with a
cause-effect relationship.
When the conditions warrant, scatter diagrams

are natural precursors to regression analyses
that reveal more precise information about
interrelationships in the data.
100
Page 50
50
Scatter Diagrams - Example
Development Effort (person months) vs. Size (KSLOC)
500
400
300
200
100
0
0 10 20 30 40 50 60 70
Size (thousands of executable source lines)
101

Run Charts
Specialized, time-sequenced form of scatter
diagram that can be used to examine data quickly
and informally for trends or other patterns that
occur over time.
Like control charts, but without the control limits

and center line.
102
Page 51
51
Run Charts - Example
16
12
Observed 8
Value 4
0
-4
Time
0 2 4 6 8 10 12 14 16
Observation Number 103

Cause-and-Effect Diagrams
Allow you to probe for, map, and prioritize a set
of factors that are thought to affect a particular
process, problem, or outcome.
They are especially helpful in eliciting and

organizing information from people who work
within a process and know what might be
causing it to perform the way it does.
104
Page 52
52
Cause-and-Effect Diagrams -
Example
change control board meets
only once a week
cannot isolate software
artifact(s) containing
problem reports the problem changes decisions
not released in a delays in approving
not logged in properly
timely manner changes
It takes too
long to
Collection Closure process our
Evaluation Resolution software
change
requests
cannot determine
information missing what needs to be done takes time to delays in shipping
from problem reports to fix the problem make changes changes and releases
delays enroute
must reconfigure
cannot replicate baselines
problem
105

Histograms
Displays of empirically observed distributions.
They show the frequencies of events that have

occurred over a given set of observations and
period of time.
Can be used to characterize the observed values

of almost any product or process attribute.
Examples: module size, defect repair time, time between
failures, defects found per test or inspection, and daily
backlogs.
Can also be helpful for revealing differences that

have taken place across processes, projects, or
106
times.
Page 53
53
Histograms - Example
20
18
16
Number of Days
14
12
10
8
6
4
2
0
34 36 38 40 42 44 46 48 50 52 54 56
Product-Service Staff Hours
107

Bar Charts
Similar in many ways to histograms.
They need not be based on measures of
continuous variables or frequency counts.
108
Page 54
54
Bar Charts - Example

Defect Analysis
45
40
35
Percent Defects
Injected
30
Found
25 Escaped
20
15
10
5
0
req’ts design code unit componentsystem customer
analysis test test test use
Software Activity 109

Pareto Charts
Special form of histogram or bar chart.
Help to focus investigations and solution finding

by ranking problems, causes, or actions in terms
of their amounts, frequencies of occurrence, or
economic consequences.
110
Page 55
55
Pareto Charts - Example

Profile of Defects Found in Product XYZ In Its First Two
Years of Operation (August 1996 through July 1998)
40
35
Percent of Total Defects
30
25
20
15
10
0
Syntax Error Ambiguous Interface Error Missing Function Memory
Requirements
Defect Type 111

Topics
Introduction

Activities
Summary
112
Page 56
56
Module Objectives
Summarize key concepts and principles
113

Four Key Responsibilities of

Process Management
Define the
process Improve
Process
Measure the
process Define Control Measure
Control the
process
Execute
Process
Improve the
process
114
Page 57
57
Process Management Perspectives
Process Performance Improve

Process
Process Control
Variability Define Control Measure
Stability
Capability Execute
Process
115


No Remove
stable? causes
Yes
capable? process
Yes
Continual
improvement
Measurement
Action
116
Page 58
58
Relating Clarify Business Goals &Strategy Yes
Measurement to Identify and Prioritize Issues

New Goals,
Strategy?
Process Yes
Management—
No
The Bigger No
Yes
Picture Measure process performance New

Measures?
Is No Remove
process assignable
stable? causes
Yes
Is No
process Change
capable? process
Yes
Continual
improvement 117

Economic Justification
Control charts are important for
• identifying special causes of variation and
removing them
• balancing between false alarms and missed
signals
• determining process stability and capability
Empirically, control charts are effective tools.
118
Page 59
59
Business Questions
Is my process performing according to my
expectations?
Is a change in the numbers significant?

• likely to “overcontrol” without control charts
Can I localize a problem?

• process mostly stable with an isolated glitch
(assignable cause)
How confident can I be in my understanding of

the process?
• management by facts versus gut feel
119

SPC for Software Premises

The software process is performed by people, not
machines.
The software process is (or can be) repeatable,

but not repetitive.
The act of measuring and analyzing will change

behavior – potentially in dysfunctional ways.
120
Page 60
60
SPC Questions
Can the software process be stable? Is it?
Can the software process be capable? On what

dimensions – cost, schedule, quality?
Will control charts and other statistical process

control techniques provide timely insight for
process implementers?
121

Summary -1
Use of SPC techniques is not widespread.
There is only limited evidence that SPC is being

applied by today’s highest maturity organizations
(Levels 4 and 5).
We believe that SPC techniques are applicable to

any organization with an existing measurement
process (Levels 2 and above).
UCL=32.9
30
26
22
Individual CL=20.4
18
Values
14
10
LCL=7.89
0 5 10 15 20 25 30
122
Page 61
61
Summary -2
The guidebook is intended to assist those
interested in moving towards process
performance measures and process
management.
The guidebook is intended to introduce

processes, practices, and methods for
• using measurement data to control and
improve process performance
• understanding issues and examples of what
organizations are facing as they address
statistical process control for software
• understanding what it means to control and
predict the software process 123


The Role of Measurement in Process

Management
Practical Software
Measurement: The Perspectives of Process
Measuring for Measurement
Process
Management and Planning Measurement for Process
Improvement Management
Applying Measurement to Process

Management
More About Analysis and Use of Process

Measures
Principles of Successful Process

Measurement
124
Page 62
62
If You Want to Use SPC, You

Should Read These Three Books
Wheeler, Donald J. Understanding Variation: The
Key to Managing Chaos. Knoxville, TN, SPC
Press, 1993.
Wheeler, Donald J. & Chambers, David S.

Understanding Statistical Process Control.
Knoxville, TN, SPC Press, 1992.
Wheeler, Donald J. Advanced Topics in

Statistical Process Control. Knoxville, TN, SPC
Press, 1995.
125

And These Two Papers

Deming, W. Edwards. "On Probability As a Basis
For Action." The American Statistician, Vol. 29,
No. 4 (November 1975): 146–152.
Hahn, Gerald J. & Meeker, William Q.

"Assumptions for Statistical Inference." The
American Statistician, Vol. 47, No. 1 (February
1993): 1–11.
126
Page 63
63
Rule one for solving any

problem… How do you
“know” you have a problem?
Rule two for solving any
problem… How will you
“know” when it is gone?
Jim Barr, 1996
(in an internet posting
to Comp.software-eng)
127

For Further Information

General SEI Web page
http://www.sei.cmu.edu/
Software Engineering Measurement & Analysis

http://www.sei.cmu.edu/technology/
measurement/welcome.html

http://www.sei.cmu.edu/products/publications/
97.reports/97hb003/97hb003abstract.html
Up-to-date copies of these slides (two-up)

http://www.sei.cmu.edu/technology/cmm/
Page 64
64

Statistical Process Control (SPC) For Software Tutorial: Topics

Uploaded by

Copyright:

Available Formats

You might also like

Statistical Process Control (SPC) For Software Tutorial: Topics

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Statistical Process Control (SPC) For Software Tutorial: Topics

Uploaded by

Copyright:

Available Formats

Carnegie Mellon University

Software Engineering Institute

August 25, 1997 (SEI Symposium)

Software Engineering Institute

Sponsored by the U.S. Department of Defense

Carnegie Mellon University

Process Measurement Perspectives and Issues

Analyzing Process Performance

Acting on the Results

Understand the types of skills and knowledge

Outline the agenda

Carnegie Mellon University

Who Should Be Here?

What Will We Do?

What Will You Walk Away With?

About the SEI

Awarded to Carnegie Mellon

Managed by Office of the

Carnegie Mellon University

Who Should Be Here?

Quality Improvement Groups

Process Action Teams

Do You Have Questions Like...

What SPC techniques

What control charts should How do I know what

Carnegie Mellon University

What Will We Do?

What Will You Walk Away With?

Carnegie Mellon University

Practical Software Measurement: Measuring for

More About Analysis and Use of Process

Principles of Successful Process

Role of Measurement in Process

Analyzing Process Performance

Acting on the Results

Carnegie Mellon University

Understand that measures must be targeted to

Understand that the concept of process

What is software process management?

Objectives of software process management

Carnegie Mellon University

Without the right information,

Tracey O’Rourke, CEO of Allen-Bradley

Every Organization Asks

Are we achieving the Are we getting better?

How can we improve response Can we reduce the cost of

Carnegie Mellon University

Measurement Clarifies and

Success in meeting commitments means that

Carnegie Mellon University

Why Measure Processes?

A Process Is the Organization of…

People Material Energy Equipment Procedures

Carnegie Mellon University

The Process Management Paradigm

Is founded on the principles of statistical

Carnegie Mellon University

Four Key Responsibilities of Process