Professional Documents
Culture Documents
Costs Benefits of Quality
Costs Benefits of Quality
6/21/2008 1
6/21/2008 2
1
6/21/2008
Contents
1. Cost of Quality (COQ) Equation
2. Cost of rework
3. Raytheon Case Study
4. BDM International – TRW Case Study
5. General Dyamics Case study
6. Motorola Case Study
7. Boeing Case Study
8. AIS Case Study
9. Space Shuttle Data
10. COQ Implementation
11. Extended COQ
12. Problems during implementation
Juran, "father" of quality
6/21/2008 3
To be completed
6/21/2008 4
2
6/21/2008
Answers
• There are about ‘5 to 15 errors’ per 1000
lines of code
• It takes about ‘75 minutes’ to find each error
• It takes about ‘2 to 9 hours’ to fix each error
6/21/2008 5
Source: Krasner, H., Using the Cost of Quality Approach for Software, Crosstalk, Nov 1998.
6/21/2008 6
3
6/21/2008
6/21/2008 Source: Krasner, Using The Cost of Quality Approach for Software Development, 1999. 7
Source: DeFeo, Joseph, The Tip of the Iceberg, QUALITY PROGRESS, MAY 2001.
6/21/2008 8
4
6/21/2008
6/21/2008 Source: DeFeo, Joseph, The Tip of the Iceberg, QUALITY PROGRESS, MAY 2001. 9
Project Cost
Project Cost
6/21/2008 10
5
6/21/2008
Cost of Rework
1. Effort to recreate the problem
– What the user did, what was the user’s configuration
– May require setting up hardware, database, etc
2. Effort to trace failure to the cause
• May require utilization of tools ( e.g. debuggers)
3. Effort to implement a fix
• May involve design change and changes to many components
4. Effort to develop and run tests
– May require setting up hardware, database
5. Effort to update and run regression test suite
– May require setting up hardware, database, etc
6. Effort to document the change, modify process or procedure
and manage configuration changes (CCB)
7. Effort to package the fix and ship it to customers
8. Effort to inform or train users of the changes
Adapted from: El Emam, Khaled, ‘The ROI from Software Quality’, K Sharp Technology Inc., 2004.
6/21/2008 11
6
6/21/2008
Use of COQ
1. Provide a measure to compare the success/failure of various
projects, and/or organizations
2. Provide cost/benefit justification and tracking for improvement
initiatives
3. Provide cost data to demonstrate the relationship of employee
efforts to the bottom line (e.g. effort vs. results)
4. Provide a basis for budgeting the quality management and
assurance functions
5. Identify quality improvement candidates through causal analysis
6. Compare proposed process improvements and identify the most
cost effective ones
7. Tune the quality costs on a particular project
• By altering the process prior to, or even in situ.
8. Determine the potential cost/risk impact of specific quality
trade-off decisions on specific projects
Source: Krasner, H., Using the Cost of Quality Approach for Software, Crosstalk, Nov 1998.
6/21/2008 13
Quality Total
cost of
Costs software
quality
Total Costs
Total Costs of Control
of Failure
Adapted from: Galin, D., Software Quality Assurance, Pearson, Addison-Wesley, 2004.
6/21/2008 14
7
6/21/2008
Examples of rework
6/21/2008 15
Requirements Development
Technical Solution
Product Integration
Verification
Validation
Organizational Process Focus
3 Process
Organizational Process Definition
Standardization Organizational Training
Defined
Integrated Project Management for IPPD
Risk Management
Integrated Teaming
Integrated Supplier Management
Decision Analysis and Resolution
Organizational Environment for Integration
Requirements Management
Project Planning
Basic Project Monitoring and Control
2 Managed Project Supplier Agreement Management
Management Measurement and Analysis
Process and Product Quality Assurance
Configuration Management
Risk
Rework
1 Initial
8
6/21/2008
30
Appraisal & Prevention Costs
25
20
18%
15
10
11%
5
0 1 6%
2 5%
1988 3
1990 4
Maturity Level 1992
1994 1996
Source: Haley, T. J., ‘Software Process Improvement at Raytheon’,
6/21/2008 IEEE SW, November 1996. 18
9
6/21/2008
45
Cost Predictability
40
40 % Overrun
35
30
25
20
15
10
0% 0 3 % Range
-5
Jan
1
Jul Jan July 2
Jan July Jan
July
3
1988
1990
Jan July Jan 4
July Jan
July
1992 Jan
1994
6/21/2008 19
Productivity Increase
200
180
160
140
120
100
80
170 %
60 Increase
40
20
0
1 2 3
1988 1990
4
1992
1994
1996
6/21/2008 20
10
6/21/2008
18
Defect Density
16
14
12
10
0
1
Jan
Jul Jan 2
July Jan
July 3
1988
1990
Jan
July
Jan
July
4
Jan
1992 1995 Jan
July
6/21/2008 21
Cost Reduction
1.2
0.8
0.6
0.4
0.2
0 1
2 3
1988
1990
4
1992
1994
6/21/2008 22
11
6/21/2008
60
TCoSQ
50
Rework
40
Cost of
Conformance
30
20
Appraisal
10 Rework
Prevention
0
87 88 89 90 91 92 93 94 95 96
Year
6/21/2008 23
12
6/21/2008
13
6/21/2008
6/21/2008 Reifer, D., Profiles of Level 5 CMMI Organizations, Crosstalk, January 2007, p24-28 27
14
6/21/2008
Source: Sandra A. Slaughter, Donald E. Harter, Mayuram S. Krishnan, ‘Evaluating the Cost of
Software Quality’, Communications of the ACM, August 1998.
6/21/2008 29
6/21/2008 30
15
6/21/2008
6/21/2008 31
COQ at BDM
Source: Sandra A. Slaughter, Donald E. Harter, Mayuram S. Krishnan, ‘Evaluating the Cost of
Software Quality’, Communications of the ACM, August 1998.
6/21/2008 32
16
6/21/2008
32 $
Total
9$
Source: Sandra A. Slaughter, Donald E. Harter, Mayuram S. Krishnan, ‘Evaluating the Cost of
6/21/2008 Software Quality’, Communications of the ACM, August 1998. 33
6/21/2008 34
17
6/21/2008
General Dynamics
Decision Systems
• Supplier of Communications and Information
Technology for military and government
customers
• 1,500 engineers
– 360 software engineers
• Process Improvement Budget
– 2.5% of base staffing of 360 engineers
General Dynamics
Customer Reported Unique Defects (CRUD)
6/21/2008 36
18
6/21/2008
General Dynamics
Phase Containment
• Detection of defects within the same phase in which it was created
6/21/2008 • Goal of 85%. Causal analysis is used to improve 37
General Dynamics
Productivity Increase
6/21/2008 38
19
6/21/2008
General Dynamics
Reduction of Rework
6/21/2008 39
General Dynamics
20
6/21/2008
General Dynamics
Better-Faster-Cheaper
21
6/21/2008
Defect Management
Level 1
Number of Defects Detected
Level 2
Level 3
Defect Management
100% Pourcentage du nombre 99% 99%
de défauts détectés
98% avant une livraison
97%
96%
95%
94%
94%
90%
88% 88%
86%
84%
80%
Version 1 Version 2 Version 3 Version 4 Version 5 Version 6
Time
(Based on 120 projects in Boeing Information Systems)
6/21/2008 44
22
6/21/2008
12% in rework
8%
3% 4%
1%
6/21/2008 45
23
6/21/2008
Space Shuttle
6/21/2008 47
1 2 3 4 5 6 7
1. Proposal / contract 1.0 1.3 2.4 3.3 6.8 26 96
Defect 2. System requirements - 1.0 1.8 2.4 5.1 19 72
occurred 3. Preliminary design - - 1.0 1.3 2.8 11 39
at stage: 4. Detailed design - - - 1.0 2.1 8.0 30
5. Unit test - - - - 1.0 3.8 14
6. System integration - - - - - 1.0 3.7
7. Operation/usage - - - - - - 1.0
6/21/2008 48
24
6/21/2008
Cost of non-quality
6/21/2008 49
Coûts de la non-qualité
Le coût des anomalies internes pendant le
développement
6/21/2008 50
25
6/21/2008
Coûts de la non-qualité
Le coût des anomalies externes
1. Pertes de clients
2. Pénalités
3. Réparations des défauts pendant l’exploitation
4. Remises pour défauts
5. Assistance technique
6. Modifications ou adaptations prohibitives
7. ‘Retrofits’ sur le produit
8. Entretien des matériels de maintenance corrective et
des compétences
6/21/2008 51
Les enjeux
• Commerciaux
– Relations avec le client
– Réputation
• Économiques
– Rencontre du coût prévu
– Rencontre de l’échéancier prévu
• Techniques
– Rencontre des objectifs
• De productivité,
• De maintenance
• De performance
• De sécurité, etc.
6/21/2008 52
26
6/21/2008
6/21/2008 53
Low Maturity
Defects CMMI 1
found
Mature
CMMI 3
27
6/21/2008
Extended COQ
Prevention costs
Adapted from: Galin, D., Software Quality Assurance, Pearson, Addison-Wesley, 2004.
6/21/2008 56
28
6/21/2008
Extended COQ
Managerial Preparation and Control Costs
• Costs of carrying out contract reviews
• Costs of preparing project plans, including quality plans
• Costs of periodic updating of project and quality plans
• Costs of performing regular progress control
• Costs of performing regular progress control of external
participants’ contributions to projects
Adapted from: Galin, D., Software Quality Assurance, Pearson, Addison-Wesley, 2004.
6/21/2008 57
Extended COQ
Managerial Failure Costs
• Unplanned costs for professional and other resources,
resulting from underestimation of the resources in the
proposal stage.
• Damages paid to customers as compensation for late
project completion, a result of the unrealistic schedule in
the Company’s proposal.
• Damages paid to customers as compensation for late
completion of the project, a result of management’s failure
to recruit team members.
• Domino effect: Damages to other projects planned to be
performed by the same teams involved in the delayed
projects. The domino effect may induce considerable
hidden external failure costs.
Adapted from: Galin, D., Software Quality Assurance, Pearson, Addison-Wesley, 2004.
6/21/2008 58
29
6/21/2008
Source: Krasner, H., Using the Cost of Quality Approach for Software, Crosstalk, Nov 1998.
6/21/2008 59
Houston, D., Cost of Software Quality: Selling Software Process Improvement to Managers, in Fundamental
6/21/2008 Concepts for the Software Quality Engineer, Edited by Daughtrey, T., ASQ Quality press, 2002 60
30
6/21/2008
A02 Requirements • • •
Development/Validation
A03 Product Concept Development • • •
Implementing COQ
BSCE Cost of Software Quality
6 5.499
5
Cost ($1000)
3.782 Prevention
4 Testing
3.316
3 Rework
2.462
Total
2
1.16 1.023
1 0.685 0.635
0
1995 1996
6/21/2008 62
31
6/21/2008
6/21/2008 Adapted from: Galin, D., Software Quality Assurance, Pearson, Addison-Wesley, 2004. 63
Conclusion
1. COQ is a useful technique
• it uses the common metric of $$$,
• it helps unify business and technical decisions about software
quality
2. COQ will make the economics of software quality visible in your
organization
• allowing quality to participate in decisions equally with cost
and schedule concerns
3. COQ has been used successfully to measure the impact (ROI) of
organizational software improvement programs (PI)
4. COQ is now being used to guide, as well as measure, effective
software quality improvement programs
5. Introducing COQ into an organization will cause useful
discussions about what is quality and the value of software quality
6/21/2008 Source: Krasner, H., Using the Cost of Quality Approach for Software, Crosstalk, Nov 1998. 64
32
6/21/2008
Summary
1. Cost of Quality (COQ) Equation
2. Cost of rework
3. Raytheon Case Study
4. BDM International – TRW Case Study
5. General Dyamics Case Study
6. Motorola Case Study
7. Boeing Case Study
8. AIS Case Study
9. Space Shuttle Data
10. COQ implementation
6/21/2008 65
6/21/2008 66
33