Software Quality Metrics – Research, Analysis

and Recommendation
Tsvetelina Mladenova
Department of Computer Systems and Technologies
University of Ruse
Ruse, Bulgaria
tsmladenova@uni-ruse.bg

Abstract - Software Quality Testing has always been a crucial According to Campbell, “If you aren’t measuring, you
part of the software development process and lately, there has aren’t managing” [13]. His statement is another
been a rise in the usage of testing applications. While a well- confirmation that the process of testing and the process of
planned and performed test, regardless of its nature – measuring the results, of which the quality indicators
automated or manual, is a key factor when deciding on the
come, is critical and necessary.
results of the test, it is often not enough to give a more deep
and thorough view of the whole process. That can be achieved Section II gives a concise overview of some quality
with properly selected software metrics that can be used for models and software metrics.
proper risk assessment and evaluation of the development. The reviewed models consist of factors and criteria
which are further classified into three categories of
This paper considers the most commonly used metrics when software quality metrics.
measuring a performed test and examines metrics that can be Section III reviews the most used quality metrics when
applied in the development process. evaluating a software project concerning the development
quality, software size and complexity, time of defects
Keywords - Software Quality Testing, testing metrics,
quality metrics, quality measurements, quality measurements fixing and the overall quality of the software.
The conclusion of the paper is given in section IV and
an idea for development of a tool for quality checks is
I. INTRODUCTION proposed in section V.
Software quality assurance (SQA) is a critical technique
II. REVIEW OF SOFTWARE QUALITY MODELS AND
in the software development and maintenance, which helps
METRICS
to achieve quality of the developed product or service [1].
The goal of every software development cycle is to meet A. Software Quality Models
the functional requirements, which are set at the beginning
of the project. Along with that, the developed software This section reviews four quality assurance models –
should meet some quality standards. Software quality (SQ) Boehm quality model [14], FURPS model [15], Dromey
evaluation is a process that should be done in every key model [16] and the ISO / IEC 25010 standard [17].
stage of the development so that the time of the
development itself is minimized, as well as the effort and Boehm’s quality model showed in TABLE I. examines
the cost [2]. seven factors that include portability, reliability,
Software Quality can be viewed as an aggregation of efficiency, usability, testability, understanding and
flexibility of software.
some quality factors. The more factors the project covers
the more it can be classified as successful. Of course, the TABLE I. BOEHM QUALITY MODEL
factors taken in consideration sometimes may vary from
project to project but in the second section, we will review
Factors Criteria
four quality models, which consist of several factors each.
Portability Self-contentedness, device independence
Every model is backed up by a set of criteria that should
be met in order for the quality factor to be accepted as Reliability Self-contentedness, accuracy, completeness,
affirmative. These models are in the basis of the metrics robustness, integrity, consistency
Efficiency Accountability, device efficiency, accessibility
that measure the quality and the overall achievement of the
software. Usability Completeness
Dijkstra said, “Program testing can be used to show the Human Accountability, communicativeness, self-
presence of bugs, but never to show their absence.” [3]. engineering descriptiveness, structuredness
The process of software testing itself is a complex, time- (testability)
consuming and mandatory for every software development Understanding Consistency, structured, conciseness
life cycle (SDLC). Many authors are considering the Modifiability Structured, augment ability
different approaches when testing ( [4] [5] [6] [7] [8] [9] (Flexibility)
[10] [11] [12]) – were they automatically testing
procedures, manual testing, testing based on the principal
of the black box, etc.

978-1-7281-9308-3/20/$31.00 ©2020 IEEE

Authorized licensed use limited to: Miami University Libraries. Downloaded on June 14,2021 at 22:31:49 UTC from IEEE Xplore. Restrictions apply.
 TABLE II. DROMEY QUALITY MODEL
efficiency, reliability, operability, security, compatibility,
Factors Criteria
maintainability, and portability.
Correctness Functionality, reliability

Internal Maintainability, efficiency, reliability The reviewed quality models have several factors, as
Contextual Maintainability, reusability, portability,
well as criteria, such as efficiency, reliability, functionality
reliability and performance. From the comparison of TABLE V. it is
Descriptive Maintainability, efficiency, reliability, clear that some of the models does not cover factors like
usability maintainability and security as the ISO standard does. The
FURPS model, on the other hand, covers the factor of
supportability and the Dromey model is the only model,
from the reviewed in this paper that considers the factors
TABLE III. FURPS QUALITY MODEL of reusability and interoperability.
Factors Criteria
TABLE V. COMPARISON BETWEEN THE QUALITY MODELS
Functionality Capabilities, security
Dromey FURPS ISO
Usability Consistency, user documentation, training Factors Boehm
/IEC
materials Usability yes yes
Reliability Frequency and security of failure, recoverability,
predictability, accuracy, mean time between Efficiency yes yes yes
failure
Performance Speed efficiency, availability, accuracy, Flexibility yes yes
throughput, response time, recovery time,
resource usage Testability yes yes
Supportability Testability, extensibility, adaptability,
maintainability, compatibility, configurability, Maintainability yes
serviceability, install ability, localizability
Reliability yes yes yes yes

Portability yes yes yes

TABLE IV. ISO/IEC 25010 Reusability yes
Factors Criteria
Interoperability yes
Functional Functional appropriateness, accuracy
suitability Understandability yes yes
Performance Time behavior, resource utilization
efficiency Functionality yes yes
Reliability Maturity, fault tolerance, recoverability,
availability Performance yes yes
Operability Appropriateness reconcilability, ease of use,
user error protection, user interface aesthetics, Supportability yes
technical learn ability, technical accessibility
Security Confidentiality integrity, non-repudiation, Security yes
accountability, authenticity
Compatibility Co-existence, interoperability

Maintainability Modularity, reusability, analyzability,

modifiability, testability
Portability Adaptability, install-ability, replace-ability
TABLE II. denotes the quality model of Dromey. He
views the quality of software as a combination of
correctness, internal factors, contextual factors, and
descriptive factors. It gives the impression that the criteria
of maintainability is present in three out of four factors and
the reliability of a software is a criterion in all four factors.
FURPS quality model was first compiled and presented
by Grady [15] in 1992 and further developed by IBM
Rational Software into FURPS +. TABLE III. shows the
original quality model, presented by Grady.
ISO 25010 is one of the most used standard quality
model and contains eight quality factors, as shown in
TABLE IV. – functional suitability, performance
B. Software Quality Metrics
The definition of a software quality metric, as defined
by IEEE, is “a function whose inputs are software data
and whose output is a single numerical value that can be
interpreted as the degree to which software possesses a
given attribute that affects its quality” [18].
Furthermore, we can expand that definition by adding
that a software quality metric can be applied not only to
the software program itself but to software services and
processes as well [11]. The goal of the metrics and the
process of measurement is to supply information needed
for improvement and prediction. Using a well-selected
metrics can reduce the number of faults and defects
during the evaluation of the software as well as provide a
quantitative basis for making informative and rational
decisions about the software quality and the software
development [4]

Authorized licensed use limited to: Miami University Libraries. Downloaded on June 14,2021 at 22:31:49 UTC from IEEE Xplore. Restrictions apply.
 Many authors review the importance of having and
using a set of metrics. Some of the most notable
arguments are:
• Metrics are used to improve the quality and
productivity of the products and thus increasing
the Customer Satisfaction [11]
• Different Metrics can be used for monitoring
when a process or a service is going-out-of-
control [11]
• Metrics can improve the development of the
product.
• Metrics can give a clearer idea about the state of
the development team and its practices.
• Metrics can be used to enhance the software
maintenance [19]
III. ANALYSIS
A. Classification of Software Metrics
Figure 1 shows the three main categories of metrics -
project metrics, process metrics and product metrics.
Every group has a set of different test types that can
measure the quality of every step of the development, as
the software metric can be applied to each software
development phase and not only in the final stages [5].
Figure 1 Software Quality Metrics classification
1) Project Metrics
Some the project metrics are used in the early stages of
the project to measure the potential risks and to make a
stable plan for development. These types of metrics are
also used for monitoring the project state and costs, based
on experience and data gathered from previous projects.
2) Process Metrics
Authorized licensed use limited to: Miami University Libraries. Downloaded on June 14,2021 at 22:31:49 UTC from IEEE Xplore. Restrictions apply.
These types of metrics and measurements are observing
the process of development. The can retrieve information
about the defects found while programming the software,
in the project that is being developed now or in past
projects, and give results that can predict future problems.
These metrics are also used when a team efficiency
evaluation is needed.
3) Product Metrics
The product metrics are the most extensive metrics from
the three categories. They consider the developed product
itself and therefore are tested at the end of the development
phase. These metrics can include the size of the software,
measured by the Halstead’s metrics , the complexity of the
program, measured by Cyclomatic Complexity Metrics,
the performance of the program – network load, database
load, etc.; accessibility and security metrics, metrics
regarding the process of the test itself- test effort, test
efficiency, test effectiveness, etc.; and others.
B. Metrics examples and brief overview of well-known
and used metrics
1) Size and Complexity of Software
The most notable size and complexity metrics are no
doubt the Halsted and Cyclomatic Complex Metrics.
a) Halsted Metrics
The Halsted Metrics, proposed in 1977, are a lexical
analysis of source code. It measures the vocabulary of the
software by a set of four formulas [ [3] [2] [4] [5]]:
• Measure of vocabulary: n = n1 + n2 (1)
• Program length : N = N1 + N2 (2)
• Program volume: V = N log2 η (3)
• Program level: L = V* / V (4)
Where n1 = the number of unique operators
n2 = the number of unique operand
N1 = the total number of operators
N2 = the total number of operands
b) Complex Metrics
A way of measuring the complexity of a software is to
use the lines of code (LOC) or complexity metric based on
mathematical graph theory and proposed by McCable in
1997 [13].
While the metrics for lines of code, vocabulary of the
software, etc., are giving valuable information about the
essence of the software, they are not enough to give a
clearer presentation about the quality.
2) Performance Metrics
When referring to the performance metrics, we
usually mean indicators like speed, network load, database
workload. In the terms of web development, sometimes the
metrics that measure the performance can also measure the
number of visitors of the website and their behavior [20].
Depending on the user’s behavior and responses metrics
for accessibility and usability can be used to determine if
the software meets the requirements and if the users can IV. CONCLUSION
work and orient themselves in it. Software metrics and measurement helps organizations
to have a better understanding about the development
3) Maintenance Metrics process, the risks and quality of their products and
The maintenance metrics consist of functional services. The examples of quality metrics in this paper
metrics, periodicity metrics, equipment metrics [21]. Most compared to the examples in the researched resources is
of this metrics are measured after the development of the just a small fraction of the possibilities that are available.
software and it is not uncommon that they measure the The purpose of this overview is to cover the main
results from other metrics, like the performance metrics. classification of quality metrics. The conducted research
Periodical monitoring of key-parts of the software can led to the conclusion that quality evaluating is an extensive
reduce the cost of defects fixing. process that should be done in every phase in the software
development. Along with the research of the available and
4) Security Metrics used metrics, a research about the current tools of
Security metrics helps organizations to verify that their measurement was made. At the moment of this writing,
security measures are complaint with a policy, identify there are numerous tools that provide the service of
security strengths and weaknesses and identify security software testing but there are far-less tools that can be used
trends [22]. for the conducting of a Quality Check Plan (QCP).
Some of the most notable metrics that should be
measured are: V. FUTURE WORK
• Mean-time-to-detect and Mean-time-to-respond As said in the conclusion, there are no tools for
• Number of users with the highest level of access comprehensive and detailed plan for quality checks and
• Number of attacks on monthly basis measures. The development of such tool can be useful for
organizations that want to track their development, costs,
5) Metrics regarding the process of software testing risks and defects.
The process of software testing is required for every A well-designed system can help organizations obtain
software or a service. Every company has their own all the information it needs to do the listed above activities,
methodologies and tools for evaluating tests. The metrics as well as improve its products, services, processes and
for test effectiveness and test efficiency does not measure customer contentment. For the development of such tool a
the software. Their concern is the tests themselves. There further, more detailed research is needed, to compile as
is a big possibility that there are no two software that can much as possible quality metrics, from which the
be measured with exactly the same tests. Therefore, the organizations can choose from when drawing up their
companies has to design, create and perform different QCPs.
tests. The indicators that can show if the test itself is well
designed and suitable can be measured with some of the ACKNOWLEDGMENT
metrics review in TABLE VI. [23]. This work/paper is supported by project 20-FEEA-01
“Methods and tools for multimedia content analysis, and
TABLE VI. TEST EFFECTIVNESS METRICS automated document and big data processing”, funded by
Metric Name Formula the Research Fund of the “Angel Kanchev” University of
Passed Test Cases Ruse.
∗ 100
Percentage

Failed Test Cases VI. REFERENCES

Percentage ∗ 100

Fixed Defects
Percentage ∗ 100 [1] M. Lee, "Software Quality Factors and Software

Quality Metrics to Enhance Software Quality
Accepted Defects ℎ Assurance," British Journal of Applied Science &
Percentage ∗ 100
Technology, 2014.
Rejected Defects ℎ [2] D. Azar, H. Harmanani and R. Korkmaz, "A hybrid
Percentage ∗ 100
heuristic approach to optimize rule-based software
Average time for quality estimation models," Information and
Defect Repair Software Technology, vol. 51, pp. 1365-1376, 2009.
Number of tests run [3] E. W. Dijkstra, Notes on Structured Programming,
per period Eindhoven: Eindhoven: Technische Hogeschool
Test Design Eindhoven, 1970.
Efficiency [4] S. Balaji and D. M. Sundararajan Murugaiyan,
"Waterfall Vs V-Model Vs Agile: A Comparative
Study on SDLC," International Journal of
Information Technology and Business Management,
vol. 2, no. 1, 2012.

Authorized licensed use limited to: Miami University Libraries. Downloaded on June 14,2021 at 22:31:49 UTC from IEEE Xplore. Restrictions apply.
[5] E. Belachew, F. Gobena and S. Nigatu, "Analysis of
Software Quality Using Software Metrics,"
International Journal of Computational Science &
Application, vol. 8, 2018.
[6] S. U. Farooq and A. M. K. Quadri, "Software
measurements and metrics: Role in effective
software testing," International Journal of
Engineering Science and Technology, vol. 3, no. 1,
pp. 671-680, 2011.
[7] A. Kaur, B. Suri and A. Sharma, "Software Testing
product metrics - a survey," Proceedings of National
Conference on Challenges & Opportunities in
Information Technology, 2007.
[8] V. S. M. T. N. Khajuria, "Software Testing: A Error
Finding Technique," vol. 7, no. 2277-2723, p. 318,
2018.
[9] M.-C. Lee and T. Chang, "Software Measurement
and Software Metrics in Software Quality,"
International Journal of Software Engineerins and
Its Applications, vol. 7, no. 4, 2013.
[10] H. Maheshwari, I. Rana and P. Goswami, "A
REVIEW OF TOOLS AND TECHNIQUES USED
IN SOFTWARE TESTING," JETIR, vol. 6, no. 4,
pp. 262-266, 2019.
[11] B. N. Premal and K. V. Kale, "A brief overview of
software testing metrics," International Journal of
Computer Science and Engineering, vol. 1, no. 3/1,
pp. 204-211, 2011.
[12] A. A. Sawant, P. H. Bari and P. M. Chawan,
"Software Testing Techniques and Strategies," in
International Journal of Engineering Research and
Aplications {IJERA), 2012.
[13] L. K. Campbell, Software Metrics: Adding
Engineering Rigor to a Currently Ephemeral
Process, 1995.
[14] B. Boehm, J. Brow, M. Lipow, G. McLeod and M.
Merritt, Characteristics of software quality,
Amsterdam: North Holland Publishing, 1978.
Authorized licensed use limited to: Miami University Libraries. Downloaded on June 14,2021 at 22:31:49 UTC from IEEE Xplore. Restrictions apply.
[15] R. Grady, Practical spftware metrics for project
management and process improvement, Prentice
Hall, 1992.
[16] R. Dromey, "Concerning the Chimera (software
quality)," IEEE Software, vol. 1, pp. 33-43, 1996.
[17] ISO / IEC 25010: Software Engineering - system and
software quality requirement and evaluation
(SQuaRE) - system and software quality model,
2011.
[18] I. s. 1061, IEEE Standard for a Software Quality
Metrics Methodology, 1992.
[19] N. Joshi and P. Singh, "Role of Software Metrics on
Software Quality," IJREAT International Journal of
Research in Engineering & Advanced Technology,
vol. 1, no. 5, 2013.
[20] T. J. McCable, "A complexy measure," IEEE
Transaction on Software Engineering, pp. 308-320,
1976.
[21] J. Rashid, T. Mahmood and M. W. Nisar, "A Study
on Software Metrics and its Impact on Software
Quality," Technical Journal, University of
Engineering and Technology (UET) Taxila,
Pakistan , vol. 24, no. 1, 2019.
[22] I. Alsmadi, M. Al-Kabi and A. Wahbeh, Website
Performance measurement: Process and Product
Metrics.
[23] S. Seela and R. Yackel, "64 Essential Testing
Metrics for Measuring Quality Assurance Success,"
QASymphony Inc., [Online]. Available:
https://www.qasymphony.com/blog/64-test-
metrics/. [Accessed January 2020].