IBM Rational Software Conference

2009

Overview

§ Asking quality questions

§ Defect removal efficiency (DRE, CDRE)
§ test survivor game
§ Weighted DRE (DREw, CDREw)
4 Demo

§ Answering quality questions

1
IBM Rational Software Conference
2009

The Quality Questions

§ How good was our testing?

§ How good is our software?

2
IBM Rational Software Conference
2009

Cumulative Defect Removal Efficiency (Simple Method)

“Cumulative defect removal efficiency =
defects found before release
defects found before and after release
By this formula, if 100 defects are found in a program during its
entire life -- in both development and in production -- and 90 of the
defects are found before release, then the cumulative defect removal
efficiency is considered to be 90 percent.”
-- T.C. Jones, IBM Systems Journal, 1978
IBM Rational Software Conference
2009

Work-In-Process Defect Removal Efficiency

defects found prior test

defects found prior and current test

How good was my testing?

WIP DRE is retrospective.
IBM Rational Software Conference
2009

A DRE by Any Other Name

§ Defect Removal Effectiveness

§ Defect Fix Percentage
§ Defect Detection Effectiveness
§ Defect Detection Percentage
§ Defect Detection Rate

5
IBM Rational Software Conference
2009

What are Actual DREs?

6
IBM Rational Software Conference
2009

What are Actual CDREs?

< 80% 80-85% 85-90% 90-95% 95-99% >99%

7
IBM Rational Software Conference
2009

Simplifying Assumptions

§ All detection methods are equivalent

§ All fixes are good and singular
§ All defect causes are equivalent *
§ All defects are equivalent

8
IBM Rational Software Conference
2009

Let’s Play: Survivor Test Environments!

§ Volunteers, please? (you’ll win small prizes!)

4Quick and simple (needn’t even stand up)
§ Development environment
§ Acceptance environment
§ Release!

9
IBM Rational Software Conference
2009

Test Results
IBM Rational Software Conference
2009

Defect Detection
IBM Rational Software Conference
2009

Defect Removal Efficiency (DRE)

IBM Rational Software Conference
2009

Defect Removal Efficiency (DRE)

IBM Rational Software Conference
2009

Cumulative Defect Removal Efficiency (CDRE)

IBM Rational Software Conference
2009

Are these test results equivalent ?????

IBM Rational Software Conference
2009

Severity Weighting

“Obviously, it is important to measure defect

severity levels as well as recording numbers
of defects.” -- T. Capers Jones, 2008

16
IBM Rational Software Conference
2009

Weighted Defect Removal Efficiency (DREw)

Keep It Simple!
(or use quantified potential business impact)
IBM Rational Software Conference
2009

Weighted Defect Removal Efficiency (DREw)

IBM Rational Software Conference
2009

Weighted Defect Removal Efficiency (DREw)

IBM Rational Software Conference
2009

Cumulative DREw (CDREw)

IBM Rational Software Conference
2009

Why Measure Work-In-Process Testing?

§ Consistent WIP DRE lends predictive value

for product reliability from an unchanged
process
§ Consistent (WIP) DREw lends predictive value
for product releasability from a current
process

21
IBM Rational Software Conference
2009

Quality Questions

How good was our testing?

How good is our software?

22