Introduction to QSA

WEEK -1
 Four Stages of Software Development

• Software Requirements Specification

• Software Design
• Implementation (Coding & Module Testing)
• Integration & Testing

Each stage will require some sort of Software Quality

Assurance (SQA).

Zafar Iqbal Khan 3 Software Quality Assurance

 What is Software?
 What is Software?
– More than computer programs
– Computer programs, procedures, and possibly
associated documentation and data pertaining to the
operation of a computer system.

 Two major types of Software:

• Generic –Stand alone, sold on open market
• Customized –For specific customer

Zafar Iqbal Khan 4 Software Quality Assurance

 What is Software Quality?
• Software Quality (as per ISO/ IEC 9126):
The totality of functionality and features of a
software product that contribute to its ability to
satisfy stated or implied needs.

• Software Quality (as IEEE Std 610):

The degree to which a component, system or
process meets specified requirements and/or
user/customer needs and expectations.
Zafar Iqbal Khan 5 Software Quality Assurance
 Software Quality Attributes
• Safety • Modularity
• Security • Complexity
• Reliability • Portability
• Resilience • Usability
• Robustness • Reusability
• Understandability • Efficiency
• Testability • Learnability
• Adaptability
Zafar Iqbal Khan 6 Software Quality Assurance
 Software Quality
• Conformance to explicitly stated functional and performance
requirements, explicitly documented development standards,
and implicit characteristics that are expected of all
professionally developed software
– Software requirements are the foundation from which quality is
measured.
• Lack of conformance to requirements is lack of quality.
– Specified standards define a set of development criteria that guide
the manner in which software is engineered.
• If the criteria are not met, lack of quality will almost surely result.
– There is a set of implicit requirements that often goes
unmentioned.
• If software conforms to its explicit requirements but fails to meet its
implicit requirements, software quality is suspect.

Zafar Iqbal Khan 7 Software Quality Assurance

 Software Quality

In respective stages of software development

• The degree to which a system, component, or process meets
specified requirements.

• The degree to which a system, component or process meets

customer or user needs or expectations.

Zafar Iqbal Khan 8 Software Quality Assurance

 Software Quality Assurance

Zafar Iqbal Khan 9 Software Quality Assurance

 Software Quality Assurance
• To ensure quality in a software product, an organization must have a
three-prong approach to quality management:
– Organization-wide policies, procedures and standards must be established.
– Project-specific policies, procedures and standards must be tailored from
the organization-wide templates.
– Quality must be controlled; that is, the organization must ensure that the
appropriate procedures are followed for each project
• Standards exist to help an organization draft an appropriate software
quality assurance plan.
– ISO 9000-3
– ANSI/IEEE standards

• External entities can be contracted to verify that an organization is

standard-compliant.
Zafar Iqbal Khan 10 Software Quality Assurance
 Software Quality Assurance

 Defined as a planned and systematic approach to the evaluation of the

quality of and adherence to software product standards, processes, and
procedures.
 An umbrella activity that is applied throughout the software process.
 Consists of a means of monitoring the software engineering processes
and methods used to ensure quality.
 An effective approach to produce high quality software.

Zafar Iqbal Khan 11 Software Quality Assurance

 Software Testing

 Software Testing is the process of executing a system or

component under specified conditions with the intent of
finding defects/bugs and to verify that it satisfies specified
requirements.
 Main goal ==> To detect bugs
 Have different levels
 Static testing vs. Dynamic testing
 Manual testing vs. Automated testing

Zafar Iqbal Khan 12 Software Quality Assurance

 SQA VS Software Testing
• Software Quality • Software Testing
Assurance
• Process-oriented activity • Product-oriented activity
• Oriented to bug prevention • Oriented to bug detection

Zafar Iqbal Khan 13 Software Quality Assurance

 A Software Quality Plan
ISO
ISO9000
9000
model
model

Organization
Organization
quality
qualityplan
plan

Project
ProjectAA Project
ProjectBB Project
ProjectCC
quality
qualityplan
plan quality
qualityplan
plan quality
qualityplan
plan

Zafar Iqbal Khan 14 Software Quality Assurance

 SQA Activities
• Applying technical methods
– To help the analyst achieve a high quality specification and a high quality design

• Conducting formal technical reviews

– A stylized meeting conducted by technical staff with the sole purpose of uncovering quality
problems

• Testing Software
– A series of test case design methods that help ensure effective error detection

• Enforcing standards

• Controlling change
– Applied during software development and maintenance

• Measurement
– Track software quality and asses the ability of methodological and procedural changes to
improve software quality

• Record keeping and reporting

 Advantages of SQA
• Software will have fewer latent defects,
resulting in reduced effort and time spent
during testing and maintenance
• Higher reliability will result in greater
customer satisfaction
• Maintenance costs can be reduced
• Overall life cycle cost of software is reduced

Zafar Iqbal Khan 16 Software Quality Assurance

 Disadvantages of SQA
• It is difficult to institute in small
organizations, where available resources to
perform necessary activities are not available
• It represents cultural change - and change is
never easy
• It requires the expenditure of dollars that
would not otherwise be explicitly budgeted to
software engineering or QA

Zafar Iqbal Khan 17 Software Quality Assurance

 Quality Reviews
• The fundamental method of validating the quality of a product
or a process.
• Applied during and/or at the end of each life cycle phase
– Point out needed improvements in the product of a single person or
team
– Confirm those parts of a product in which improvement is either
not desired or not needed
– Achieve technical work of more uniform, or at least more
predictable, quality than what can be achieved without reviews, in
order to make technical work more manageable
• Quality reviews can have different intents:
– review for defect removal
– review for progress assessment
– review for consistency and conformance
 Requirements
Requirements
Quality Reviews
Analysis
Analysis Specification
Review
1x
Design Design
Design Review

3-6x
Code Code
Code Review

10x Test
Testing Review
Testing

15-70x Customer
Maintenance Feedback
Maintenance

40-1000x
[Adapted from Pressman 4th Ed]

Zafar Iqbal Khan 19 Software Quality Assurance

 Cost Impact of Software Defects
Errors from
Previous
Steps

Errors Passed Through Percent Efficiency

Amplified Errors 1:X for error

Newly Generated Errors detection

Errors
Passed to
Next Step

Zafar Iqbal Khan 20 Software Quality Assurance

 Defect Amplification and Removal

Preliminary Design
0
0 0%
10
Detailed Design
10 6
6
4 37
4x1.5 0% Code/Unit Testing
25 10
10
27 27x3 20%
94
37
25

116
To integration
testing...

Zafar Iqbal Khan 21 Software Quality Assurance

 Defect Amplification (cont’d)
94
Integration Testing
94
94
0 0 50%
47
Validation
0 47 Testing
47
0 24
94 0 50% System Testing
0 24
24
0 0 50%
12
47
0

24
Latent Errors

Zafar Iqbal Khan 22 Software Quality Assurance

Review Checklist for Systems Engineering

• Are major functions defined in a bounded and unambiguous

fashion?
• Are interfaces between system elements defined?
• Are performance bounds established for the system as a whole and
for each element?
• Are design constraints established for each element?
• Has the best alternative been selected?
• Is the solution technologically feasible?
• Has a mechanism for system validation and verification been
established?
• Is there consistency among all system elements?
Review Checklist for Software
Project Planning
• Is the software scope unambiguously defined and bounded?
• Is terminology clear?
• Are resources adequate for the scope?
• Are resources readily available?
• Are tasks properly defined and sequenced?
• Is the basis for cost estimation reasonable? Has it been
developed using two different sources?
• Have historical productivity and quality data been used?
• Have differences in estimates been reconciled?
• Are pre-established budgets and deadlines realistic?
• Is the schedule consistent?
 Review Checklist for Software
Requirements Analysis
• Is the information domain analysis complete, consistent, and
accurate?
• Is problem partitioning complete?
• Are external and internal interfaces properly defined?
• Are all requirements traceable to the system level?
• Is prototyping conducted for the customer?
• Is performance achievable with constraints imposed by other
system elements?
• Are requirements consistent with schedule, resources, and
budget?
• Are validation criteria complete?
Zafar Iqbal Khan 25 Software Quality Assurance
 Review Checklist for Software Design
(Preliminary Design Review)
• Are software requirements reflected in the software architecture?
• Is effective modularity achieved? Are modules functionally
independent?
• Is program architecture factored?
• Are interfaces defined for modules and external system elements?
• Is data structure consistent with software requirements?
• Has maintainability been considered?

Zafar Iqbal Khan 26 Software Quality Assurance

Review Checklist for Software Design
(Design Walkthrough)
• Does the algorithm accomplish the desired function?
• Is the algorithm logically correct?
• Is the interface consistent with architectural design?
• Is logical complexity reasonable?
• Have error handling and “ant bugging” been specified?
• Is local data structure properly defined?
• Are structured programming constructs used throughout?
• Is design detail amenable to the implementation language?
• Which are used: operating system or language dependent features?
• Is compound or inverse logic used?
• Has maintainability been considered?
 Review Checklist for Coding
• Is the design properly translated into code? (The results of the
procedural design should be available at this review)
• Are there misspellings or typos?
• Has proper use of language conventions been made?
• Is there compliance with coding standards for language style,
comments, module prologue?
• Are incorrect or ambiguous comments present?
• Are typing and data declaration proper?
• Are physical constraints correct?
• Have all items on the design walkthrough checklist been reapplied
(as required)?

Zafar Iqbal Khan 28 Software Quality

 Review Checklist for
Software Testing (Test Plan)
• Have major test phases been properly identified and sequenced?
• Has traceability to validation criteria/requirements been
established as part of software requirements analysis?
• Are major functions demonstrated early?
• Is the test plan consistent with the overall project plan?
• Has a test schedule been explicitly defined?
• Are test resources and tools identified and available?
• Has a test recordkeeping mechanism been established?
• Have test drivers and stubs been identified, and has work to
develop them been scheduled?
• Has stress testing for software been specified?
Zafar Iqbal Khan 29 Software Quality Assurance
 Review Checklist for Software
Testing (Test Procedure)
• Have both white and black box tests been specified?
• Have all independent logic paths been tested?
• Have test cases been identified and listed with
expected results?
• Is error handling to be tested?
• Are boundary values to be tested?
• Are timing and performance to be tested?
• Has acceptable variation from expected results been
specified?
Zafar Iqbal Khan 30 Software Quality Assurance
 Review Checklist for Maintenance

• Have side effects associated with change been considered?

• Has the request for change been documented, evaluated, and
approved?
• Has the change, once made, been documented and reported to
interested parties?
• Have appropriate FTRs been conducted?
• Has a final acceptance review been conducted to assure that all
software has been properly updated, tested, and replaced?

Zafar Iqbal Khan 31 Software Quality Assurance

Formal Technical Review (FTR)
• Software quality assurance activity that is performed by
software engineering practitioners
– Uncover errors in function, logic, or implementation for any
representation of the software
– Verify that the software under review meets its requirements
– Assure that the software has been represented according to
predefined standards
– Achieve software that is developed in a uniform manner
– Make projects more manageable
• FTR is actually a class of reviews
– Walkthroughs
– Inspections
– Round-robin reviews
– Other small group technical assessments of the software
The Review Meeting
• Constraints
– Between 3 and 5 people (typically) are involved
– Advance preparation should occur, but should involve no more that
2 hours of work for each person
– Duration should be less than two hours
• Components
– Product - A component of software to be reviewed
– Producer - The individual who developed the product
– Review leader - Appointed by the project leader; evaluates the
product for readiness, generates copies of product materials, and
distributes them to 2 or 3 reviewers
– Reviewers - Spend between 1 and 2 hours reviewing the product,
making notes, and otherwise becoming familiar with the work
– Recorder - The individual who records (in writing) all important
issues raised during the review
 Review Reporting and Recordkeeping

• Review Summary Report

– What was reviewed?
– Who reviewed it?
– What were the findings and conclusions?
• Review Issues List
– Identify the problem areas within the product
– Serve as an action item checklist that guides the
producer as corrections are made

Zafar Iqbal Khan 34 Software Quality Assurance

 Guidelines for FTR
• Review the product, not the producer
• Set an agenda and maintain it
• Limit debate and rebuttal
• Enunciate the problem areas, but don’t attempt to solve every problem that
is noted
• Take written notes
• Limit the number of participants and insist upon advance preparation
• Develop a checklist for each product that is likely to be reviewed
• Allocate resources and time schedules for FTRs
• Conduct meaningful training for all reviewers
• Review your earlier reviews (if any)

Zafar Iqbal Khan 35 Software Quality Assurance

 Reviewer’s Preparation
• Be sure that you understand the context of the
material
• Skim all product material to understand the location
and the format of information
• Read the product material and annotate a hardcopy
• Pose your written comments as questions
• Avoid issues of style
• Inform the review leader if you cannot prepare

Zafar Iqbal Khan 36 Software Quality Assurance

 Results of the Review Meeting
• All attendees of the FTR must make a decision
– Accept the product without further modification
– Reject the product due to severe errors (and perform
another review after corrections have been made)
– Accept the product provisionally (minor corrections are
needed, but no further reviews are required)
• A sign-off is completed, indicating participation and
concurrence with the review team’s findings

Zafar Iqbal Khan 37 Software Quality Assurance

 Software Reliability
• Probability of failure-free operation for a specified
time in a specified environment.
• This could mean very different things for different
systems and different users.
• Informally, reliability is a measure of the users’
perception of how well the software provides the
services they need.
– Not an objective measure
– Must be based on an operational profile
– Must consider that there are widely varying consequences
for different errors

Zafar Iqbal Khan 38 Software Quality

 IO Mapping
Subset of inputs
Input
InputSet
Set causing erroneous
outputs

Software
Software

Output
OutputSet
Set

Erroneous
outputs

Zafar Iqbal Khan 39 Software Quality Assurance

 Software Faults and Failures
• A failure corresponds to erroneous/unexpected runtime behavior
observed by a user.
• A fault is a static software characteristic that can cause a failure to
occur.
• The presence of a fault doesn’t necessarily imply the occurrence of a
failure.
Input Set

User A Erroneous
Inputs Inputs

User B User C
Inputs Inputs

[Adapted from Sommerville 5th Ed]

Zafar Iqbal Khan 40 Software Quality Assurance

 Reliability Improvements
• Software reliability improves when faults which are present in the
most frequently used portions of the software are removed.
• A removal of X% of faults doesn’t necessarily mean an X%
improvement in reliability.
• In a study by Mills et al. in 1987 removing 60% of faults resulted in a
3% improvement in reliability.
• Removing faults with the most serious consequences is the primary
objective.

Zafar Iqbal Khan 41 Software Quality Assurance

 Textbooks for Reading

Software Quality Assurance: From Theory to Implementation,

2012

This book covers several issues related to software quality assurance.

Some important chapters are: software quality challenge, what is
software quality, software quality factors, and software testing

Metrics and Models in Software Quality Engineering, 2010

This book is a reference in software metrics. It covers a comprehensive

breadth of measurement theory and software quality metrics

Zafar Iqbal Khan 42 Software Quality Assurance