STQA NOTES SYMCA

UNIT-8
SOFTWARE MEASUREMENT AND METRICS
Metrics for Software Maintenance

When development of a software product is complete and it is released to the market, it enters
the maintenance phase of its life cycle. During this phase the defect arrivals by time interval
and customer problem calls (which may or may not be defects) by time interval are the de
facto metrics. However, the number of defect or problem arrivals is largely determined by the
development process before the maintenance phase. Not much can be done to alter the quality
of the product during this phase. Therefore, these two de facto metrics, although important,
do not reflect the quality of software maintenance. What can be done during the maintenance
phase is to fix the defects as soon as possible and with excellent fix quality. Such actions,
although still not able to improve the defect rate of the product, can improve customer
satisfaction to a large extent. The following metrics are therefore very important: Fix backlog
and backlog management index Fix response time and fix responsiveness Percent delinquent
fixes Fix quality

Fix Backlog and Backlog Management Index

Fix backlog is a workload statement for software maintenance. It is related to both the rate of
defect arrivals and the rate at which fixes for reported problems become available. It is a
simple count of reported problems that remain at the end of each month or each week. Using
it in the format of a trend chart, this metric can provide meaningful information for managing
the maintenance process. Another metric to manage the backlog of open, unresolved,
problems is the backlog management index (BMI).

As a ratio of number of closed, or solved, problems to number of problem arrivals during the
month, if BMI is larger than 100, it means the backlog is reduced. If BMI is less than 100,
then the backlog increased. With enough data points, the techniques of control charting can
be used to calculate the backlog management capability of the maintenance process. More
investigation and analysis should be triggered when the value of BMI exceeds the control
limits. Of course, the goal is always to strive for a BMI larger than 100. A BMI trend chart or
control chart should be examined together with trend charts of defect arrivals, defects fixed
(closed), and the number of problems in the backlog.

Figure 4.5 is a trend chart by month of the numbers of opened and closed problems of a
software product, and a pseudo-control chart for the BMI. The latest release of the product
was available to customers in the month for the first data points on the two charts. This
explains the rise and fall of the problem arrivals and closures. The mean BMI was 102.9%,
indicating that the capability of the fix process was functioning normally. All BMI values
were within the upper (UCL) and lower (LCL) control limits葉he backlog management
process was in control. (Note: We call the BMI chart a pseudocontrol chart because the BMI
data are autocorrelated and therefore the assumption of independence for control charts is
violated. Despite not being "real" control charts in statistical terms, however, we found

Page 75 of 88
STQA NOTES SYMCA

pseudo-control charts such as the BMI chart quite useful in software quality management. Fix
Response Time and Fix Responsiveness

For many software development organizations, guidelines are established on the time limit
within which the fixes should be available for the reported defects. Usually the criteria are set
in accordance with the severity of the problems. For the critical situations in which the
customers' businesses are at risk due to defects in the software product, software developers
or the software change teams work around the clock to fix the problems. For less severe
defects for which circumventions are available, the required fix response time is more
relaxed. The fix response time metric is usually calculated as follows for all problems as well
as by severity level:

Mean time of all problems from open to closed. If there are data points with extreme values,
medians should be used instead of mean. Such cases could occur for less severe problems for
which customers may be satisfied with the circumvention and didn't demand a fix. Therefore,
the problem may remain open for a long time in the tracking report.

In general, short fix response time leads to customer satisfaction. However, there is a subtle
difference between fix responsiveness and short fix response time. From the customer's
perspective, the use of averages may mask individual differences. The important elements of
fix responsiveness are customer expectations, the agreed-to fix time, and the ability to meet
one's commitment to the customer. For example, John takes his car to the dealer for servicing
in the early morning and needs it back by noon. If the dealer promises noon but does not get
the car ready until 2 o'clock, John will not be a satisfied customer. On the other hand, Julia
does not need her mini van back until she gets off from work, around 6 P.M. As long as the
dealer finishes servicing her van by then, Julia is a satisfied customer. If the dealer leaves a
timely phone message on her answering machine at work saying that her van is ready to pick
up, Julia will be even more satisfied. This type of fix responsiveness process is indeed being
practiced by automobile dealers who focus on customer satisfaction.

In this writer's knowledge, the systems software development of HewlettPackard (HP) in

California and IBM Rochester's systems software development have fix responsiveness
processes similar to the process just illustrated by the automobile examples. In fact, IBM
Rochester's practice originated from a benchmarking exchange with HP some years ago. The
metric for IBM Rochester's fix responsiveness is operationalized as percentage of delivered
fixes meeting committed dates to customers.

Percent Delinquent Fixes

The mean (or median) response time metric is a central tendency measure. A more sensitive
metric is the percentage of delinquent fixes. For each fix, if the turnaround time greatly
exceeds the required response time, then it is classified as delinquent:

This metric, however, is not a metric for real-time delinquent management because it is for
closed problems only. Problems that are still open must be factored into the calculation for a
real-time metric. Assuming the time unit is 1 week, we propose that the percent delinquent of

Page 76 of 88
STQA NOTES SYMCA

problems in the active backlog be used. Active backlog refers to all opened problems for the
week, which is the sum of the existing backlog at the beginning of the week and new problem
arrivals during the week. In other words, it contains the total number of problems to be
processed for the week葉he total workload. The number of delinquent problems is checked at
the end of the week. Figure 4.6 shows the real-time delivery index diagrammatically.

It is important to note that the metric of percent delinquent fixes is a cohort metric. Its
denominator refers to a cohort of problems (problems closed in a given period of time, or
problems to be processed in a given week). The cohort concept is important because if it is
operationalized as a cross-sectional measure, then invalid metrics will result. For example,
we have seen practices in which at the end of each week the number of problems in backlog
(problems still to be fixed) and the number of delinquent open problems were counted, and
the percent delinquent problems was calculated. This cross-sectional counting approach
neglects problems that were processed and closed before the end of the week, and will create
a high delinquent index when significant improvement (reduction in problems backlog) is
made.

Fix Quality

Fix quality or the number of defective fixes is another important quality metric for the
maintenance phase. From the customer's perspective, it is bad enough to encounter functional
defects when running a business on the software. It is even worse if the fixes turn out to be
defective. A fix is defective if it did not fix the reported problem, or if it fixed the original
problem but injected a new defect. For mission-critical software, defective fixes are
detrimental to customer satisfaction. The metric of percent defective fixes is simply the
percentage of all fixes in a time interval (e.g., 1 month) that are defective.

A defective fix can be recorded in two ways: Record it in the month it was discovered or
record it in the month the fix was delivered. The first is a customer measure, the second is a
process measure. The difference between the two dates is the latent period of the defective
fix. It is meaningful to keep track of the latency data and other information such as the
number of customers who were affected by the defective fix. Usually the longer the latency,
the more customers are affected because there is more time for customers to apply that
defective fix to their software system.

There is an argument against using percentage for defective fixes. If the number of defects,
and therefore the fixes, is large, then the small value of the percentage metric will show an
optimistic picture, although the number of defective fixes could be quite large. This metric,
therefore, should be a straight count of the number of defective fixes. The quality goal for the
maintenance process, of course, is zero defective fixes without delinquency.

Software Metric
A software metric is a standard of measure of a degree to which a software system or
process

Page 77 of 88
STQA NOTES SYMCA

possesses some property. Even if a metric is not a measurement (metrics are functions, while
measurements are the numbers obtained by the application of metrics), often the two terms
are used as synonyms. Since quantitative measurements are essential in all sciences, there is a
continuous effort by computer science practitioners and theoreticians to bring similar
approaches to software development. The goal is obtaining objective, reproducible and
quantifiable measurements, which may have numerous valuable applications in schedule and
budget planning, cost estimation, quality assurance testing, software debugging, software
performance optimization, and optimal personnel task assignments.

Software Testing Metrics

A Metric is a quantitative measure of the degree to which a system, system component, or

process possesses a given attribute.

Metrics can be defined as “STANDARDS OF MEASUREMENT”.

Software Metrics are used to measure the quality of the project. Simply, Metric is a unit used
for describing an attribute. Metric is a scale for measurement.

Suppose, in general, “Kilogram” is a metric for measuring the attribute “Weight”. Similarly,
in software, “How many issues are found in thousand lines of code?”, here No. of issues is
one measurement & No. of lines of code is another measurement. Metric is defined from
these two measurements.

Test metrics example:

 How many defects exist within the module?

 How many test cases are executed per person?
 What is the Test coverage %?

What is Software Test Measurement?

Measurement is the quantitative indication of extent, amount, dimension, capacity, or size of
some attribute of a product or process.
Test measurement example: Total number of defects.
Please refer below diagram for a clear understanding of the difference between Measurement
& Metrics.

Page 78 of 88
STQA NOTES SYMCA

Test Metrics are used to,

1. Take the decision for next phase of activities such as, estimate the cost & schedule of
future projects.
2. Understand the kind of improvement required to success the project
3. Take a decision on process or technology to be modified etc.
Importance of Software Testing Metrics:

As explained above, Test Metrics are the most important to measure the quality of the
software.

Now, how can we measure the quality of the software by using Metrics?

Suppose, if a project does not have any metrics, then how the quality of the work done by a
Test analyst will be measured?

For Example A Test Analyst has to,

1. Design the test cases for 5 requirements

2. Execute the designed test cases
3. Log the defects & need to fail the related test cases
4. After the defect is resolved, need to re-test the defect & re-execute the corresponding
failed test case.

Metrics Life Cycle:

Page 79 of 88
STQA NOTES SYMCA

Types of Manual Test Metrics:

Testing Metrics are mainly divided into 2 categories.

1. Base Metrics
2. Calculated Metrics

Base Metrics:
Base Metrics are the Metrics which are derived from the data gathered by the Test Analyst
during the test case development and execution.

This data will be tracked throughout the Test Lifecycle. i.e. collecting the data like Total no.
of test cases developed for a project (or) no. of test cases need to be executed (or) no. of test
cases passed/failed/blocked etc.

Calculated Metrics:
Calculated Metrics are derived from the data gathered in Base Metrics. These Metrics are
generally tracked by the test lead/manager for Test Reporting purpose.

LIMITATIONS

Page 80 of 88
STQA NOTES SYMCA

As software development is a complex process, with high variance on both methodologies

and objectives, it is difficult to define or measure software qualities and quantities and to
determine a valid and concurrent measurement metric, especially when making such a
prediction prior to the detail design. Another source of difficulty and debate is in determining
which metrics matter, and what they mean.[3][4] The practical utility of software
measurements has therefore been limited to the following domains:

following domains:

 Scheduling
 Software sizing
 Programming complexity
 Software development effort estimation
 Software quality

Software metrics is a standard of measure that contains many activities which involve some
degree of measurement. It can be classified into three categories: product metrics, process
metrics, and project metrics.

 Product metrics describe the characteristics of the product such as size, complexity,
design features, performance, and quality level.

 Process metrics can be used to improve software development and maintenance.

Examples include the effectiveness of defect removal during development, the pattern
of testing defect arrival, and the response time of the fix process.

 Project metrics describe the project characteristics and execution. Examples include
the number of software developers, the staffing pattern over the life cycle of the
software, cost, schedule, and productivity.

Some metrics belong to multiple categories. For example, the in-process quality metrics of a
project are both process metrics and project metrics.

Scope of Software Metrics

Software metrics contains many activities which include the following −

 Cost and effort estimation

 Productivity measures and model
 Data collection
 Quantity models and measures
 Reliability models
 Performance and evaluation models

Page 81 of 88
STQA NOTES SYMCA

 Structural and complexity metrics

 Capability – maturity assessment
 Management by metrics
 Evaluation of methods and tools
Software measurement is a diverse collection of these activities that range from models
predicting software project costs at a specific stage to measures of program structure.

Cost and Effort Estimation

Effort is expressed as a function of one or more variables such as the size of the program, the
capability of the developers and the level of reuse. Cost and effort estimation models have
been proposed to predict the project cost during early phases in the software life cycle. The
different models proposed are −

 Boehm’s COCOMO model

 Putnam’s slim model
 Albrecht’s function point model
Productivity Model and Measures
Productivity can be considered as a function of the value and the cost. Each can be
decomposed into different measurable size, functionality, time, money, etc. Different possible
components of a productivity model can be expressed in the following diagram.

Data Collection
The quality of any measurement program is clearly dependent on careful data collection. Data
collected can be distilled into simple charts and graphs so that the managers can understand
the progress and problem of the development. Data collection is also essential for scientific
investigation of relationships and trends.

Page 82 of 88
STQA NOTES SYMCA

Quality Models and Measures

Quality models have been developed for the measurement of quality of the product without
which productivity is meaningless. These quality models can be combined with productivity
model for measuring the correct productivity. These models are usually constructed in a tree-
like fashion. The upper branches hold important high level quality factors such as reliability
and usability.

The notion of divide and conquer approach has been implemented as a standard approach to
measuring software quality.

Reliability Models
Most quality models include reliability as a component factor, however, the need to predict
and measure reliability has led to a separate specialization in reliability modeling and
prediction. The basic problem in reliability theory is to predict when a system will eventually
fail.

Performance Evaluation and Models

It includes externally observable system performance characteristics such as response times
and completion rates, and the internal working of the system such as the efficiency of
algorithms. It is another aspect of quality.

Structural and Complexity Metrics

Here we measure the structural attributes of representations of the software, which are
available in advance of execution. Then we try to establish empirically predictive theories to
support quality assurance, quality control, and quality prediction.

Capability Maturity Assessment

This model can assess many different attributes of development including the use of tools,
standard practices and more. It is based on the key practices that every good contractor should
be using.

Management by Metrics
For managing the software project, measurement has a vital role. For checking whether the
project is on track, users and developers can rely on the measurement-based chart and graph.
The standard set of measurements and reporting methods are especially important when the
software is embedded in a product where the customers are not usually well-versed in software
terminology.

Evaluation of Methods and Tools

This depends on the experimental design, proper identification of factors likely to affect the
outcome and appropriate measurement of factor attributes.

Page 83 of 88
STQA NOTES SYMCA

DEFECT DETECTION EFFICIENCY Fundamentals

DEFINITION
Defect Detection Efficiency (DDE) is the number of defects detected during a phase/stage
that are injected during that same phase divided by the total number of defects injected during
that phase.
ELABORATION

 defects:
o Are confirmed and agreed upon (not just reported).
o Dropped defects are not counted.
 phase:
o Can be any phase in the software development life cycle where defects can be
injected AND detected. For example, Requirement, Design, and Coding.
 injected:
o The phase a defect is ‘injected’ in is identified by analyzing the defects [For
instance, a defect can be detected in System Testing phase but the cause of the
defect can be due to wrong design. Hence, the injected phase for that defect is
Design phase.]

FORMULA

 DDE = (Number of Defects Injected AND Detected in a Phase / Total Number of

Defects Injected in that Phase) x 100 %

UNIT

 Percentage (%)

TARGET VALUE

 The ultimate target value for Defect Detection Efficiency is 100% which means that
all defects injected during a phase are detected during that same phase and none are
transmitted to subsequent phases. [Note: the cost of fixing a defect at a later phase is
higher.]

USES

 For measuring the quality of the processes (process efficiency) within software
development life cycle; by evaluating the degree to which defects introduced during
that phase/stage are eliminated before they are transmitted into subsequent
phases/stages.
 For identifying the phases in the software development life cycle that are the weakest
in terms of quality control and for focusing on them.

Page 84 of 88
STQA NOTES SYMCA

Defect Density Fundamentals

DEFINITION
Defect Density is the number of confirmed defects detected in software/component during a
defined period of development/operation divided by the size of the software/component.
ELABORATION
The ‘defects’ are:

 confirmed and agreed upon (not just reported).

 Dropped defects are not counted.

The period might be for one of the following:

 for a duration (say, the first month, the quarter, or the year).
 for each phase of the software life cycle.
 for the whole of the software life cycle.

The size is measured in one of the following:

 Function Points (FP)

 Source Lines of Code

DEFECT DENSITY FORMULA

USES

 For comparing the relative number of defects in various software components so that
high-risk components can be identified and resources focused towards them.
 For comparing software/products so that quality of each software/product can be
quantified and resources focused towards those with low quality

Cost of Quality (COQ) is a measure that quantifies the cost of control/conformance and the
cost of failure of control/non-conformance. In other words, it sums up the costs related to
prevention and detection of defects and the costs due to occurrences of defects.

 Definition by ISTQB: cost of quality: The total costs incurred on quality activities and
issues and often split into prevention costs, appraisal costs, internal failure costs and
external failure costs.
 Definition by QAI: Money spent beyond expected production costs (labor, materials,
equipment) to ensure that the product the customer receives is a quality (defect free)
product. The Cost of Quality includes prevention, appraisal, and correction or repair
costs.

Page 85 of 88
STQA NOTES SYMCA

EXPLANATION

 Cost of Control (Also known as Cost of Conformance)

o Prevention Cost
 The cost arises from efforts to prevent defects.
 Example: Quality Assurance costs
o Appraisal Cost
 The cost arises from efforts to detect defects.
 Example: Quality Control costs
 Cost of Failure of Control (Also known as Cost of Non-Conformance)
o Internal Failure Cost
 The cost arises from defects identified internally and efforts to correct
them.
 Example: Cost of Rework (Fixing of internal defects and re-testing)
o External Failure Cost
 The cost arises from defects identified by the client or end-users and
efforts to correct them.
 Example: Cost of Rework (Fixing of external defects and re-testing)
and any other costs due to external defects (Product
service/liability/recall, etc)

FORMULA / CALCULATION
Cost of Quality (COQ) = Cost of Control + Cost of Failure of Control
where
Cost of Control = Prevention Cost + Appraisal Cost
and
Cost of Failure of Control = Internal Failure Cost + External Failure Cost

Measurement Principle

Before I introduce a series of product metrics that

(1) Assist in the evaluation of the analysis and design models,
(2) Provide an indication of the complexity of procedural designs and source code, and
(3) Facilitate the design of more effective testing, it is important for you to understand basic
measurement principles. Roche suggests a measurement process that can be characterized by
five activities:

• Formulation. The derivation of software measures and metrics appropriate for the
representation of the software that is being considered.

• Collection. The mechanism used to accumulate data required to derive the formulated
metrics.

• Analysis. The computation of metrics and the application of mathematical tools.

Page 86 of 88
STQA NOTES SYMCA

• Interpretation. The evaluation of metrics resulting in insight into the quality of the
representation.

• Feedback. Recommendations derived from the interpretation of product metrics transmitted

to the software team.

Software metrics will be useful only if they are characterized effectively and validated so that
their worth is proven. The following principles [Let03b] are representative of many that can
be proposed for metrics characterization and validation:

• A metric should have desirable mathematical properties. That is, the metric’s value should
be in a meaningful range (e.g., 0 to 1, where 0 truly means absence, 1 indicates the maximum
value, and 0.5 represents the “halfway point”). Also, a metric that purports to be on a rational
scale should not be composed of components that are only measured on an ordinal scale.

• When a metric represents a software characteristic that increases when positive traits occur
or decreases when undesirable traits are encountered, the value of the metric should increase
or decrease in the same manner.

• Each metric should be validated empirically in a wide variety of contexts before being
published or used to make decisions. A metric should measure the factor of interest,
independently of other factors. It should “scale up” to large systems and work in a variety of
programming languages and system domains.

Software Metrics and Measures

Software Measures can be understood as a process of quantifying and symbolizing various
attributes and aspects of software.

Software Metrics provide measures for various aspects of software process and software
product.

Software measures are fundamental requirement of software engineering. They not only help
to control the software development process but also aid to keep quality of ultimate product
excellent.

According to Tom DeMarco, a (Software Engineer), “You cannot control what you cannot
measure.” By his saying, it is very clear how important software measures are.

Let us see some software metrics:

 Size Metrics - LOC (Lines of Code), mostly calculated in thousands of delivered

source code lines, denoted as KLOC.

Function Point Count is measure of the functionality provided by the software.

Function Point count defines the size of functional aspect of software.

Page 87 of 88
STQA NOTES SYMCA

 Complexity Metrics - McCabe’s Cyclomatic complexity quantifies the upper bound

of the number of independent paths in a program, which is perceived as complexity of
the program or its modules. It is represented in terms of graph theory concepts by
using control flow graph.
 Quality Metrics - Defects, their types and causes, consequence, intensity of severity
and their implications define the quality of product.

The number of defects found in development process and number of defects reported
by the client after the product is installed or delivered at client-end, define quality of
product.

 Process Metrics - In various phases of SDLC, the methods and tools used, the
company standards and the performance of development are software process metrics.
 Resource Metrics - Effort, time and various resources used, represents metrics for
resource measurement.

Page 88 of 88