Software Testing

(11CS 491)

Unit I – Introduction
Importance of Software Quality, Process and Code
Quality and SDLC Process, V-Model

Ashoka M
ISE, PESIT
Lecture Agenda
1. Course Introduction
2. Consequence of poor quality
3. Methods of enhancing quality & SQA
4. SDLC process & testing activity
5. Importance of testing and not testing
6. Validation & Verification

2
Software Testing – About the course

Subject relevance
◦ So far, covered the s/w construction extensively
◦ Detailed extension of s/w engg.
Difficultiesin real-world of software
Quality as an engineering discipline
◦ Higher level
◦ Engineering  Design - Production
Course Objectives
Software Testing – Course objective
At the end of the course you should be
◦ Able to apply proper testing technique
◦ Familiar with various types of testing
◦ Appreciative of difficulties and complexities in
software quality
◦ Exposed topics of software quality engineering,
Quality Assurance, Testing automation
◦ Able to plan proper quality approach in software
projects
Testing in SDLC
 Activity cost distribution
Wat erfall model
0 25 50 75 100

Specification Design Development Integration and testing

It erative development

0 25 50 75 1 00

Specification Iterative development System testing

Component-based software eng ineering

0 25 50 75 1 00

Specification Development Integration and testing

Development and evolution costs for long-lifetime syst ems

0 10 200 30 400

System development System evolution

 Product development costs

0 25 50 75 100

Specification Development System testing

Product development costs are typically 3-4

times that of comparable custom software.
Software Testing – Part of Quality
Engineering
“Traditional Testing” approaches – not
sufficient / efficient
Customer expectations, engineering and
business economics called for
◦ Total Quality Engineering
Management of “Quality” complexity in
modern day business / project context
What is Quality?
Popular Views about Quality:
◦ Quality related to luxury, class and taste
◦ Quality is related to quality of life.
◦ I know about quality when I see the product.
◦ It should work!

Professional Views about Quality:

◦ Conformance to requirements.
◦ Fitness for use.
What is software quality
Conformance to requirements
Lack of bugs - defects
Low defect rate (# of defects/size unit)
High reliability (number of failures per N
hours of operation)
Probability of failure-free operation in a
specified time
◦ Measured as Mean Time To Failure (MTTF)
What is Software Quality ?
According to the IEEE Software quality is:

The degree to which a system, component,

or process meets specified requirements.

The degree to which a system, component,

or process meets customer or user needs or
expectations.
What is Software Quality ?
According to Pressman Software quality is:

Conformance to explicitly stated functional

and performance requirements, explicitly
documented development standards, and
implicit characteristics that are expected of
all professionally developed software
Why quality is important
Software is a major component of computer
systems (about 80% of the cost) used for
◦ communication (e.g. phone system, email system)

◦ health monitoring, transportation (e.g. automobile,

aeronautics), economic exchanges (e.g.
ecommerce), entertainment, etc.

Software defects are extremely costly in

terms of money / reputation / loss of life
Consequences of poor quality
Many historic disasters attributed to software
1991 patriot missile failure inaccurate calculation of
time due to computer arithmetic errors
1988 shooting down of Airbus 320 by the USS
Vincennes - cryptic and misleading output displayed
by tracking software
On June 3, 1980, the North American Aerospace
Defense Command (NORAD) wrongly reported that
the U.S. was under missile attack.

9 hour breakdown of AT&T's longdistance telephone

network caused by an untested code patch
Consequences of poor quality
Ariane 5 crash June 4, 1996
◦ maiden flight of the European Ariane 5 launcher
crashed about 40 seconds after take-off
◦ loss was about half a billion dollars
◦ explosion was the result of a software error; Uncaught
exception due to floating point error: conversion from a
64bit integer to a 16bit signed integer applied to a
larger than expected number
Module was reused without proper testing from
Ariane 4
◦ Error was not supposed to happen with Ariane 4
◦ No exception handler
Consequences of poor quality
Mars Climate Orbiter September 23, 1999
◦ Mars Climate Orbiter, disappeared as it began to
orbit Mars.
◦ Cost about $US 125million
◦ Failure due to error in a transfer of information
between a team in Colorado and a team in
California
◦ One team used English units (e.g., inches, feet
and pounds) while the other used metric units for
a key spacecraft operation.
Consequences of poor quality
Mars Polar Lander December, 1999. Mars
Polar Lander, disappeared during landing on
Mars
Failure more likely due to unexpected
setting of a single data bit.
defect not caught by testing
independent teams tested separate aspects
Weak software = poor quality?
Software viruses and worms
– Blaster worm ($US 525 millions)
– Sobig.F ($US 500 millions – 1billions)
Exploit well known software vulnerabilities
Software developers do not devote enough
effort to applying lessons learned about the
causes of vulnerabilities.
Same types of vulnerabilities continue to be
seen in newer versions of products that were
in earlier versions.
Economics of failure
Monetary impact of poor software quality (Standish group 1995)
175,000 software projects/year. Average cost per project
◦ . Large companies $US 2,322,000
◦ . Medium companies $US 1,331,000
◦ . Small companies $US 434,000
31.1% of projects cancelled before completed - cost $81 billion
52.7% of projects exceed their budget costing 189% of original
estimates cost $59 billion
16.2% of software projects completed ontime and onbudget
◦ (9% for larger companies)
Large companies delivered systems have approximately only
42% of originally proposed features and functions
78.4% of smaller companies projects get deployed with 74.2%
of their original features and functions.
In-Process Impact
Serious impact on software creation company
◦ Business operations / Plans
◦ Wasted money and time
◦ Employee morale
◦ Credibility of teams
◦ Loss of customer / market
◦ Legal implications

These are difficult to notice and happen over a

period of time!
Some of the generic reasons for faiulre
The uniqueness of the software product
High complexity
Invisibility of the product
Limited opportunities to detect defects (“bugs”) only
opportunity is Product development
The environments in which software is developed
◦ Contracted
◦ Subjection to customer supplier relationship
Requirement for teamwork
◦ Need for cooperation and coordination with other development teams
◦ Need for interfaces with other software systems
◦ Need to continue carrying out a project while the team changes
◦ Need to continue maintaining the software system for years
 Quality Factors –
McCall's software quality factor model
Software Quality factors

Product Operation Factors

Product Revision factors

Correctness Product Transition Factors

Reliability
Maintainability
Efficiency
Testability Portability
Integrity
Flexibility Re-usability
Usability Inter-operability
Quality factors
Correctness
◦ accuracy, completeness of required output, upto-
dateness, availability of the information
Reliability
◦ Low failure rate
Efficiency
◦ Resource consumption to perform software function
Integrity
◦ software system security, access rights
Usability
◦ ability to learn, perform required task
Software Quality factors
Maintainability
◦ effort to identify and fix software failures
(modularity, documentation, etc)
Flexibility
◦ degree of adaptability (to new customers, tasks,
etc)
Testability
◦ support for testing (e.g. log files, automatic
diagnostics, etc)
Software Quality factors
Portability
◦ adaptation to other environments (hardware,
software)
Reusability
◦ use of software components for other projects
Interoperability
◦ ability to interface with other
components/systems
Methods of enhancing quality
Quality of product V/s Quality of Process
Holistic approach required
Integrated with Total Quality management
Quality Standards, Certifications
◦ ISO 9001, CMMI, 6-Sigma….
One of the commonly accepted
management framework / practice is QA –
Quality Assurance.
What is quality assurance? – SQA?
According to the IEEE Software quality
assurance is:
◦ A planned and systematic pattern of all actions
necessary to provide adequate confidence that an
item or product conforms to established technical
requirements.

◦ A set of activities designed to evaluate the process by

which the products are developed or manufactured.

Contrast with: quality control.

Software QA
According to D. Galin, Software quality
assurance is:
A systematic, planned set of actions necessary
to provide adequate confidence that the
software development process or the
maintenance process of a software system
product conforms to established functional
technical requirements as well as with the
managerial requirements of keeping the
schedule and operating within the budgetary
confines.
Objectives of SQA
Assuring an acceptable level of confidence
that the software will conform to functional
technical requirements.
Assuring an acceptable level of confidence
that the software will conform to managerial
scheduling and budgetary requirements.
Initiation and management of activities for
the improvement and greater efficiency of
software development and SQA activities.
Objectives of SQA
Only Quality Control (testing) is not
enough:
◦ What would you do if your software does not
pass the QC test?
◦ QC is a reactive approach, not proactive one.

Quality Assurance includes Proactive as

well as Reactive approaches.
Objectives of SQA
The purpose of quality assurance practices are to
minimize the number of defects.
◦ Practically, zero defect product is not possible to
achieve.
How much efforts are needed to minimize the
number of defects?
◦ are you developing a customized project or product?
◦ how critical your application is?
General principles of QA
Know what you are doing
Know what you should be doing
Know how to measure the difference
General principles of QA
Know what you are doing
◦ Understand what is being built, how it is being
built and what it currently does
Suppose a software development process
with management structure (milestones,
scheduling)
◦ reporting policies
◦ tracking
General principles of QA
Know what you should be doing
◦ having explicit requirements and specifications
◦ suppose a software development process with
◦ requirements analysis,
◦ acceptance tests,
◦ frequent user feedback
General principles of QA
Know how to measure the difference
◦ having explicit measures comparing what is being
done from what should be done
Four complementary methods:
◦ Formal methods: verify mathematically specified
properties
◦ Testing: explicit input to exercise software and check
for expected output
◦ Inspections: human examination of requirements,
design, code, ... based on checklists
◦ Metrics: measures a known set of properties related to
quality
Software Qualities and Process
 Qualities cannot be added after development
◦ Quality results from a set of inter-dependent activities
◦ Analysis and testing are crucial but far from sufficient.

 Testing is not a phase, but a lifestyle

◦ Testing and analysis activities occur from early in requirements
engineering through delivery and subsequent evolution.
◦ Quality depends on every part of the software process

 An essential feature of software processes is that

software test and analysis is thoroughly integrated and
not an afterthought
The Quality Process (cont)
Quality process: set of activities and
responsibilities
◦ focused primarily on ensuring adequate dependability
◦ concerned with project schedule or with product
usability

The quality process provides a framework for

◦ selecting and arranging activities
◦ considering interactions and trade-offs with other
important goals.
Interactions and tradeoffs
High dependability vs. time to market

Mass market products:

◦ better to achieve a reasonably high degree of
dependability on a tight schedule than to achieve
ultra-high dependability on a much longer schedule
Critical medical devices:
◦ better to achieve ultra-high dependability on a much
longer schedule than a reasonably high degree of
dependability on a tight schedule
 Properties of the Quality Process
 Completeness: Appropriate activities are planned to
detect each important class of faults
◦ Memory leaks are important class of faults for C++ program

 Timeliness: Faults are detected at a point of high

leverage (as early as possible) E.g. Java program
 Cost-effectiveness: Activities are chosen depending on
cost and effectiveness
◦ cost must be considered over the whole
development cycle and product life
◦ the dominant factor is usually the cost of
repeating an activity through many change
cycles.
 Planning and Monitoring
 The quality process
◦ Balances several activities across the whole development process
◦ Selects and arranges them to be as cost-effective as possible
◦ Improves early visibility
 Quality goals can be achieved only through careful
planning
 Planning is integral to the quality process
 A well designed quality process balances several
activities across the whole development process
 Selecting and arranging them to be a cost effective as
possible and to early visibility
 Process Visibility (key factor of software process)

 A process is visible to the extent that one can answer the

question
◦ How does our progress compare to our plan?
◦ Example: Are we on schedule? How far ahead or behind?
 The quality process has not achieved adequate visibility if
one cannot gain strong confidence in the quality of the
software system before it reaches final testing
◦ quality activities are usually placed as early as possible
 design test cases at the earliest opportunity (not ``just in time'')
 uses analysis techniques on software artifacts produced before actual code.
◦ motivates the use of “proxy” measures
 Ex: the number of faults in design or code is not a true measure of
reliability, but we may count faults discovered in design inspections as an
early indicator of potential quality problems
SQA
SQA: Comprehensive lifecycle approach concerned
with every aspect of the software product development
process
Includes
◦ comprehensive set of quality objectives
◦ measurable quality attributes (quality metrics) to assess
progress toward the objectives
◦ quantitative certification targets for all component of the
software development processes.
Takes into account:
◦ customer product requirements,
◦ customer quality requirements, and
◦ corporate quality requirements.
SQA Includes
Verification
performed at the end of a
phase to ensure that
requirements established
during previous phase have
been met
Validation
performed at the end of the
development process to
ensure compliance with
product requirements
SQA Includes
Defect Prevention
◦ prevents defects from occurring in the first place
◦ Activities: training, planning, and simulation
Defects detection
◦ finds defects in a software artifact
◦ Activities: inspections, testing or measuring
Defects removal
◦ isolation, correction, verification of fixes
◦ Activities: fault isolation, fault analysis,
regression testing
SQA Process activities
Requirements validation.
Design verification.
Static code checking (inspection/reviews).
Dynamic testing.
Process engineering and standards.
Metrics and continuous improvement
SQA Responsibilities for a Project
(cont...)
Typical SDLC phases and relevant Artifacts SQA Responsibilities
Requirements
RequirementsCollection
Collection Requirement
RequirementSpecs
Specs Reviews
Reviews

Analysis
Analysis Functional
FunctionalSpecs
Specs Reviews
Reviews

Architecture
Architecture&&Design
Design Design
DesignSpecs
Specs Reviews
Reviews

Development
Development Code
Code&&Executables
Executables
Implement
Implement
Test
TestCases
Cases
Testing
Testing

Deployment
Deployment Deployment
DeploymentDocs
Docs Review
Review
SQA Artifacts for a Project
Dev. Artifacts SQA Artifacts SQA Artifacts
(more)
Requirement
RequirementSpecs
Specs RS
RSReviews
Reviews

Functional
FunctionalSpecs
Specs FS
FSReviews
Reviews
Test
TestPlan
Plan

Design
DesignSpecs
Specs DS
DSReviews
Reviews
Test
TestCases
Cases

Code
Code&&Executables
Executables Bug
BugReports
Reports
More
MoreTest
TestCases
Cases

Deployment
DeploymentDocs
Docs DD
DDReviews
Reviews
SQA Responsibilities for a Project
 Review of documents developed by development team.
 Track the compliance with standards.
 Development of QA Plan (test plan + test cases).
 Implementation of test cases (Black Box or Glass Box
Testing).
 Management of bug repository.
 Participating in code and design reviews.
An Effective Testing Team
SQA Team should be composed of:
◦ Members with different background
◦ Members from different domains
◦ Technical gurus and user representatives
◦ Members with more analytical abilities.
Characteristics of Good SQA Engineer
 Experience & Education as a programmer or analyst.
 A thick skin.
 Good sense of humor.
 Tolerance for chaos.
 Firmness.
 Evidence oriented.
 Logical.
 Honest.
 Self sufficient.
Testing in different process models
Dev Process Model Testing approach
Testing in waterfall model Usually done toward the end of
development cycle
V-model Explicitly specifies testing activities in
each phase of development cycle

Spiral Applied to software increments, proposed

for eventually software development

Agile Used in agile development methods, such

as eXtreme programming (XP)

Test-driven development Requirements specified as tests

(TDD)
 Testing in waterfall model

Requirement
specification

Design
Coding and
unit testing

System testing
Acceptanc
e testing

Training
and delivery
Maintenance
 Testing in V-model

Maintenance
Requiremen
t analysis
Acceptanc
Validate requirements e testing
Develop system acceptance test
Design
System testing
Validate design
Develop integration test
Coding Integration
testing

Develop unit tests

unit testing
 Spiral testing

Final test activities, ready

(cost if test for acceptance test
activities increases Cost
with iterations)

Intermediate test activity

Time

Initial test
activity (prototype ,
evolves over time)
Importance of Testing
Minimize defects ‘leaking’ to the next stage
Customer satisfaction
Process and engineering efficiency
◦ Un-planned expenditure
◦ Delayed defect recognition and correction is
(very) costly
Feedback to Improve development practices
Un-earth hidden contradictions in
functionality / requirements
It is an important concern in project planning
Software Errors, software faults and
software failures
 Bug/defect/fault consequence of a human error
 results in non-conformance to requirements
 manife
 sts as failure in running software
9 causes of software errors
1. Faulty requirements definition
2. Client/developer communication failures
3. Deliberate deviations from software requirements
4. Logical design errors
5. Coding errors
6. Noncompliance with documentation and coding
instructions
7. Shortcomings of the testing process
8. User interface and procedure errors
9. Documentation errors
Development process relation to
defects
Majority of defects are introduced in earlier
phases

Relative cost of fixing defects

Compounding effect of defects on s/w
costs
S/W cost

1000x

100x

10x

Reqmnt Design Coding Post release

Verification and Validation
Verification Validation
Build the product right Build the right product

Reactive Proactive

Design v/s SRS Product V/s Need

Code v/s Design Product V/s Attributes

Mostly done by Team / Team and Customer &

Technical Domain Experts

Required Required
 Generic model to facilitate V & V in
any SDLC Phase
ETVX Model for Design Phase
(Entry, Task, Verification, Exit )

Exit Criteria
Entry Criteria Traceability matrix, Output
Customer approved Task ready for next phase
SRS  Evolve Architecture
 Create High Level Design
 Create Detailed Design
 Create Program Specs
Output:
Input: SRS Design docs,
Program Specs
Statements about Testing & QA
What is expected may not happen but what is
inspected will!
What is not tested will NOT work
You will not have time for testing; But you are
forced to spend time on reworking
If something can fail, it will
◦ Corollary – It fails when it can cause maximum
incovenience!
Topic Summary
Quality is an integral part of software
engineering
Poor quality has huge socio-economic
implication
A structured, scientific approach to enhance
software quality is imperative in business
Quality and it’s practice has influenced
overall development methodologies &
practices
THANKS!