FEATURED ARTICLE

Managing Cost of Quality

for Quality 4.0 Using
Software, Systems, and
Data Quality Models
By Daniel Zrymiak
This article shows how a cost of quality approach, already customized for
software and information technology, can be applied toward the deployment and
implementation of Quality 4.0 in an organization. Existing models and quality
characteristics used for software, systems, and data quality can be leveraged to
identify cost categories and to support the creation of a cost of quality system for About the author
Quality 4.0.
Daniel Zrymiak is an
engagement manager at
Cost of Quality Applied to Software Ultranauts, based in Surrey,
British Columbia. He has
While the origins of quality costs and their categories originated in manufacturing
more than two decades of
and industrial engineering, the categories of external failure costs, internal failure
costs, appraisal costs, and prevention costs have been successfully applied to international experience
software and information technology. However, unlike the tangible manufactured in quality and project
product characteristics of the post-war era of the 20th century, software and infor- management, primarily in
mation technology are less tangible. Consequently, the success of such products or information technology,
deliverables is determined during the design and validation processes. operational excellence, and
consulting. Zrymiak is an ASQ
Software and information technology also do not exist as products unto themselves Fellow and has been awarded
but rather were created to perform or automate a particular service. In addition
ASQ’s Feigenbaum and Crosby
to technical deficiencies associated with the software or system, the business
requirements of that service must also be considered. Consequently, quality failure Medals and a Testimonial
can extend beyond functional adequacy to include reliability, responsiveness, and Award. He is a Quality Press
usability. Consider a shopping transaction website that has a three-to-five-minute author and reviewer, and
delay during processing; even though less time is spent than during the alternative committee chair for the ASQ
of travelling to a brick-and-mortar store and waiting in line for the same transac- Edwards Medal. Zrymiak
tion, a delay of this duration would represent an unmet expectation and a service remains active with the Quality
failure of the software.
Management Division as a
member leader and vice-chair
of governance and excellence.
He can be reached at:
13 QUALITY MANAGEMENT DIVISION | SPRING 2021 Dzrymiak.asq@gmail.com.
The Quality Management Forum
The cost of quality model attempts to justify investments in quality
activities that emphasize planning, prevention, and appraisal.
This is intended to create the resulting deliverable or solution
in a manner that is more stable, predictable, and cost-effective
than by fixing errors or defects as they emerge in use and are
reported back by the customer base. In the context of software,
the cost of quality categories can be summarized with the
following typical activities.
External Failures
These are reported incidents of software or system malfunctions
that prevent the proper use or application of the information
technology solutions. The impact and extent of the failures (and
subsequent external failure costs) depend on the criticality and
impact of the software. For example, while an online gaming
application isolated to a small percentage of users is of minimal
concern, software that regulates medical applications or trans-
portation arrangements could have catastrophic outcomes. The
protection against improper data acquisition or identity theft also
requires additional security measures. If damages are resolved,
there are still marketplace issues, contractual penalties, and
regulatory violations that must be addressed as well.
Internal Failures
These are found internally, prior to the realization of risks as
business-critical issues. However, the technical remedies are often
cumbersome, requiring the proper diagnosis and reconfiguration
or redesign of software or its associated systems and operating
environments. The internal failures area is complicated by the
evolving versions of concurrent or interdependent components,
applications, or infrastructure, which must be constantly
monitored. Even if the solution was previously determined to be
acceptable, externally directed product or system revisions can
also cause disruption and instability. For this reason, all failures
must be diagnosed from a variety of perspectives to determine
the potential technical and business impact.
Appraisal
When this category was first specified for manufactured
products, the types of activities included were primarily product
inspections, statistical controls, and process audits. For software
and information technology, this phase cannot simply be
14 QUALITY MANAGEMENT DIVISION | SPRING 2021
appended to finished products or work-in-progress phases.
Software and information technology solutions are made using
various frameworks and methods, including waterfall, agile,
Scrum, prototyping, and evolutionary. These approaches all
recognize that a finished solution is the result of releasing a
verified and validated deliverable, compatible with its operating
environment, suitable for its intended users, and representative of
the original design commitments.
Appraisal activities must occur throughout all stages or phases
of the design and development projects. The stages may be
done sequentially (characteristic of waterfall methods) or in a
more integrated fashion with repeated iterations leading up to
an acceptable release. Inspections are supplemented by reviews
of requirements and designs, audits of development processes,
verification that all explicit specifications are integrated, and
validation of the fitness for use. Appraisal activities continue after
the product or solution is released into its user environment, and
they are integrated into the ongoing maintenance activities for
the life of the product.
In addition to the direct activities, the supporting environment
and infrastructure must be included within this cost category. For
example, if a software or system was intended to manage the
concurrent transactions of 25,000 students registering for their
classes, it would be impractical to employ 25,000 testers to
simulate that quantity of users. A more practical approach would
be to simulate this activity through load, stress, and performance
testing resources, which automatically generate the necessary
test data and user traffic. This has resulted in the formation of a
distinct niche in software development dedicated to automated
software testing.
Prevention
Prevention costs are both deliberate and the result of unfortunate
experiences learned from product and system failures. While
there are finite ways for a tangible product to fail, software can
malfunction from infinite causes and conditions. This expands
the range of prevention activities needed to secure software
and information technology from failures related to functionality,
reliability, usability, portability, efficiency, maintainability, and
security. Within software design, development, and deployment,
such initiatives include research of market and customer pref-
erences, requirements review, project planning (that is, scope,
The Quality Management Forum
schedule, resources, risks, vendors), systems for configuration
management and version control, and training to build these
functional capabilities within the work environment.
As the need for quality is prevalent in software and information
technology, the costs associated with such activities must also be
monitored and optimized. The cost of quality system has been
leveraged for this purpose and has been used to demonstrate
the positive returns from investments in methods, practices, and
resources that have been shown to reduce the costs of nonconfor-
mance (representing both internal and external failures).
Quality 4.0: Applying Information and
Technology for Industry 4.0 Demands
Quality 4.0 is a concept that links the practice and application
of quality techniques to the emerging capabilities associated with
Industry 4.0. Some examples of Industry 4.0 include:
• Additive manufacturing: Enhancing existing systems with 3-D
models and computer controls
• Mobile computing: Expanding the devices used by front-line
employees to include smartphones, pads, tablets, and other
devices supporting instant interactions and communications
• Digital manufacturing: Integrating information technology
systems and databases into a holistic end-to-end resource (that
is, enterprise resource planning)
• Cloud computing: Migration of information technology
infrastructure to shared services
• Blockchain: Public transaction ledger for tracking and securing
transaction records
These examples represent potential augmentations to an
organization’s applications, devices, information management,
infrastructure, and archiving systems. As all these potential
solutions would require some form of design, development,
configuration, and deployment, the systems currently used for
software and data-quality models could be applied.
As with any innovation, risks are present that could result in
issues and external failure losses that would be detrimental to
the organization. The countermeasures for such failures would
15 QUALITY MANAGEMENT DIVISION | SPRING 2021
be categorized within prevention and appraisal activities, which
are similar to those currently used for software and information
technology. Attributes of Industry 4.0 are often applied to existing
operations, representing an upgrade to existing software and
information technology. This process can be simplified in a
five-step, high-level process:
Step 1—Analysis
The project or program is determined based on the business
need, the available budget and resources, and the opportunities
for innovation and superior delivery. This analysis needs to
incorporate the context of potential usage situations and the
perspectives of all relevant stakeholders, including customers,
vendors, regulatory agencies, partners, and the internal func-
tional areas within the organization. Preventive activities should
work to define the framework of the analysis, while appraisal
steps should strive to confirm the adequacy of such analysis
relative to the expectations of the proposed solution.
Step 2—Qualification
The components and vendors are qualified based on the
expected outcomes, the available budget and resources, and the
complexity of the system and solution. The preventive steps are
represented by the level of detail within documentation (that is,
request for proposals), and the appraisal steps should include
the assessment of potential vendors against the established
acceptance criteria. Technical solutions often require different
levels of vendor support for technical configurations and design
modifications.
Step 3—Development, Verification, and Validation
The Industry 4.0 solution is managed through its sub-stages
or phases of readiness. An increasingly popular framework is
Scrum, which applies agile methods to rapidly produce incre-
mental iterations until the solution is completed. This approach
infuses preventive activities, which include planning and
appraisal steps such as inspections and reviews, intermittently
throughout development. One variation, test-driven development,
actually integrates the appraisal tests as requirements against
which development can occur. The increased adoption of test
automation—and its recurring use for introductory “smoke”
testing or repeated regression testing—represents the adoption
The Quality Management Forum
of preventive and appraisal techniques as alternatives to external
failures discovered only after deployment to the user community.
Step 4—Change, Release, Deployment, and Migration
Any solution proposed for an organization must first be reviewed
for adequacy and suitability, must be released at a time that
minimizes interruptions of existing services and deliverables,
should be deployed in a controlled manner, and should migrate
all associated services and processes. This step also has to
incorporate the complexities associated with training and
adoption. These IT practices (that is, change, release, deploy-
ment, migration) have distinct preventive and appraisal steps to
ensure that proper reviews, verifications, and validations occur in
a manner that is commensurate with the complexity and risks of
the newly introduced solutions.
Step 5—Stabilization and Maintenance
The shifts to new technologies, techniques, and methods will
not be immediately seamless but will need adjustments to
address immediate problems and to resolve initial difficulties.
In this case, the preventive steps include the development of a
robust knowledge base and responsive approaches to events
or incidents. From these rapid responses, the systems can be
quickly validated and diagnosed, enabling the deployment of
workaround solutions while the root causes are more extensively
investigated.
Quality Characteristics, Business Risks,
and Costs of Failure
Quality characteristics can be identified from a review of two ISO
standards: (1) ISO 25010 Systems and Software Quality Models
and (2) ISO 25012 Data Quality. Unlike tangible products,
which have finite characteristics of size, dimension, functional
use, and other evaluation criteria, Quality 4.0 characteristics
integrate the principles of information and technology.
System and Software Quality Models
The characteristics of the system and software quality models
include elements of the quality-in-use model and elements
associated with system/software product quality.
16 QUALITY MANAGEMENT DIVISION | SPRING 2021
The quality-in-use model refers to the business needs of the
stakeholders and can be summarized by the following items:
• Effectiveness: Accuracy and completeness of usage relative to
the purpose
• Efficiency: Cost and resources needed to fulfill the purpose
• Satisfaction: Extent that user needs are satisfied by the solution
from the intended use
• Freedom from risk: Mitigation of potential risks, threats, or
adverse consequences
• Context coverage: The specified conditions and boundaries of
how the system can be used to fulfill the purpose
The business risks associated with these items are that use of
these systems can become delayed or disrupted due to their
complexity and the resources needed to operate and power
the systems. The failure to manage expectations and deliver the
desired outcomes over a timely period can increase costs without
delivering on lofty promises of innovation.
The potential costs of failure can extend beyond the technical
costs to include the introduction of new threats and adverse
consequences. For example, if user needs are left unmet because
the system was overloaded and could not complete transactions,
that could result in permanent loss of customer relationships.
Alternatively, if the systems were intended to regulate complex
arrangements such as transportation traffic, a malfunction could
lead to improper routing and costly diversions and accidents.
System/software product quality is associated with the following
characteristics:
• Functional suitability: Whether a system is complete, correct,
and appropriate
• Performance efficiency: The capacity, resource uses, and time
consumed
• Compatibility: The interoperability of the system with co-existing
entities
• Usability: The ease with which users can access, learn,
understand, and operate the system
The Quality Management Forum
• Reliability: The extent to which systems remain available,
tolerating and recovering from errors
• Security: The protection of confidentiality, authenticity, and
integrity of information
• Maintainability: The ability to modify modules, analyze, reuse,
and test components
• Portability: The ability to install, adapt, and replace systems or
to use them across environments
The business risks associated with these items are that systems
might not fit the purpose or might introduce new problems and
complexities that exceed the benefits of the innovation. If a
system has limited application, it will be an expensive innovation
for a confined segment of the organization. If the user experience
proves difficult, it will affect adoption or prompt bypassing of the
overly complex or unreliable system to create a “hidden factory”
condition by which work is done without being properly tracked
or captured. Problems with reliability and security lead to loss of
business confidence, which can permanently damage existing
and future business relationships.
The potential costs of failure can extend beyond the costs of the
technical solutions. Introducing the failure costs of the affected
stakeholders and their services can magnify the costs tenfold or
even higher. For example, if the systems cause the constituent
business processes to fail, the external failure costs can cascade
from the stakeholders and their client bases, amplifying the losses
and penalties involved.
Data Quality
The data quality characteristics are segmented by those elements
that are inherent to the data and those that are dependent on the
system.
Inherent data quality characteristics include:
• Accuracy: Representation of data values to true values or
intended attributes
• Completeness: Possession of all expected attributes and
instances within the data
17 QUALITY MANAGEMENT DIVISION | SPRING 2021
• Consistency: Data attributes are not contradictory but coherent
with other data
• Credibility: Representation of data values as truthful, authentic,
and believable by users
• Currentness: The timeliness and correct age for a specific
context of use
The business risks associated with these items are tied to the
relevance and importance of the information being handled. If
that information proves deficient or unusable, the value of the
technology managing that information will be diminished. Since
information is used for decision support, any flawed information
will cause incorrect decisions to be made based on fallacious
or erroneous indicators or assumptions. If the impacts of those
decisions are critical to life, security, or property, the risks are
infinite and have irreversible consequences.
The potential costs of failure can extend beyond those of
restoring the technical applications and infrastructure to support
databases and the resident information. If the data are used for
particular purposes or specified situations, the context of those
data can be assignable to overall failures in those situations. For
example, if information about water levels and precipitation was
inaccurate, and decisions derived from such information resulted
in uncontrolled flooding of urban areas, the cost of data failure
could be measured by the overall impact to society, traceable
back to the data maladies.
System-dependent data quality characteristics include:
• Accessibility: Availability for a specific use or user community
• Compliance: Adherence to standards, conventions, regulations,
or rules
• Confidentiality: Restriction of data to authorized use, along
with information security
• Efficiency: Ability for data to be processed with the available
resources for specific uses
• Precision: Exactness of data for a particular use
• Traceability: Audit trails of data access, usage, and changes
for future reference
The Quality Management Forum
• Understandability: Ability of the data to be read and under-
stood
• Availability: Data can be retrieved by authorized users within
the context of a purpose
• Portability: Data are installed, replaced, or moved between
systems remaining intact
• Recoverability: In the event of a failure, data operations can be
maintained and preserved
The business risks associated with these items are that systems
will manage the data in a manner that diminishes their integrity
or usability. The flawed systems may cause data to be lost,
corrupted, or disorganized such that their value to their constitu-
ent stakeholders and their processes is invalidated and nullified.
The potential costs of failure can include the ongoing cost
burdens of managing and storing data that consume time and
resources without providing commensurate benefit to their
intended users or to the interoperating applications and systems.
The discovery and diagnosis of these problems create additional
work, particularly unplanned recovery from unanticipated
disasters.
Business continuity practices have been established specifically
to address the potential failure conditions. The alternative to
planned business continuity is the wasteful and time-consuming
rework involved to recreate and replicate any information that
was lost or corrupted. Where this cannot be done automatically
or internally, customers or users may then be required to submit
redundant information, causing distress and frustration with
respect to customer engagement.
Managing Quality Costs and Optimizing
Returns on Investment
Having reviewed the breadth of areas that potentially require
quality safeguards, the projected costs of external failures would
include the expenses to design, develop, deploy, and maintain
Quality 4.0 deliverables, plus potential costs from losses to
stakeholders and their affected processes and customer solutions.
18 QUALITY MANAGEMENT DIVISION | SPRING 2021
These are all traceable back to the software, systems, and data
impacted by the Quality 4.0 innovations and upgrades applied.
When the potential costs and impacts are quantified, the proba-
bilities of such events could accommodate a realistic risk-based
estimate. For example, a 0.1% chance of a catastrophic failure
costing $1 million would translate into an estimated risk costing
$1,000. This figure could be traded against the prevention and
appraisal steps needed to mitigate the risk, so the costs are
planned and predictable, and the risks are stabilized.
Optimization would come from consolidating the potential
countermeasures taken for planning, project management,
change controls, reviews, verifications that specifications were
met, validations of fitness for intended uses in the required envi-
ronments, and other types of inspections and audits. By consoli-
dating these practices into workable and repeatable systems, the
existing risks are mitigated, and the robustness increases beyond
problem resolution to innovations and breakthroughs.
Quality 4.0 endeavors already have a prescribed business case
to justify their adoption. However, the presence of external and
internal failures forces the projects to adopt additional rework
and delays. The absence of failures increases the “purity” of
returns, and the predictability of future innovations for stakehold-
ers and their affected activities.
Conclusion: Key Outcomes
The cost of quality protocol can be applied to Quality 4.0. The
deployment can be elaborated using the same models and
practices as are currently applied to software, systems, and data.
Quality 4.0, comprising primarily software and information
technology, consists of intangible deliverables that are provided
to stakeholders, that are applied to processes and solutions, and
that require sufficient quality cost controls to mitigate losses from
potential internal and external failures.