Software Testing: ER/CORP/CRS/SE04/003 E&R Infosys 1

Software Testing
ER/CORP/CRS/SE04/003 E&R Infosys 1

Objectives
Quality – a culture
Software Testing Life Cycle
Software Testing Techniques
Types of testing
Introduction to Test automation and Test tools
Test Planning and Strategizing

Agenda
What is quality?
What is testing?
Testing life cycle
Testing Techniques
Types of Testing
Test Automation
Test Tools

What is Quality?
“The degree to which a system, component or

process meets requirements”
or
“The degree to which a system, component or
process meets customer or user needs or
expectations”
-I.E.E.E.

Quality is…
What the customer says it is…

 Features
 Cost
 Time
Not a goal, but a means for continuous
improvement

Dimensions of Software Quality
Functionality
 Completeness
 Correctness
 Compatibility
Performance
 Time
 Resources
Maintainability
 Correctability
 Expandability
 Testability

Dimensions of Software Quality (contd.)
Portability
 Hardware independence
 Interoperability
Reliability
 Error tolerance
 Availability
Usability
 Understandability
 Learnability
 Operability
 Communicativeness

Software Quality Issues
Challenges
Challenges Problem
Problem Symptoms
Symptoms
Scope User
User or
or Business
Business needs
needs not
not
ScopeCreep
Creep//Churning
Churning
requirements Met
Met
requirements
Fuzzy
Fuzzy Requirements
Requirements Poor
Poor End-user
End-user Experience
Experience
Lack
Lack of
of Skilled
Skilled Resources
Resources Difficult-to-Maintain
Difficult-to-Maintain
Systems
Systems
Lack
Lack of
of aa most
most suitable
suitable process
process
Build
Build and
and Release
Release Issues
Issues
Flaws
Flaws in
in Design
Design
Modules
Modules Don’t
Don’t Integrate
Integrate
Incomplete
Incomplete Test
Test Planning
Planning
Poor
Poor Performance
Performance under
under
Late
Late Testing
Testing Load
Load
Unreliable
Unreliable Data
Data for
for Testing
Testing Flaws
Flaws Discovered
Discovered Very
Very Late
Late
Improper
Improper Defect
Defect Tracking
Tracking Defect
Defect Leakage
Leakage

Testing – What Is it?
A process of demonstrating that errors are not

present?
OR
A way of establishing confidence that a program
does what it is supposed to do?
OR
A means of achieving an error-free program by
finding all errors?

Testing is…
A process of executing a program with the intent

of finding errors
A “DESTRUCTIVE”, yet creative process

Why Testing?
Verifies that all requirements are implemented

correctly (both for positive and negative
conditions)
Identifies defects before software deployment
Helps improve quality and reliability
Makes software predictable in behavior
Reduces incompatibility and interoperability
issues
Helps marketability and retention of customers

Cost of Ineffective Testing
Time
 Late Releases
 Projects need to be reworked or abandoned
Money
 Budget over-runs
 Defects are 100 to 1000 times more costly to find and
repair after deployment
Quality
 Developers unsure of product quality
 Products released with undiscovered or unresolved
defects

Testing Lifecycle
Development Lifecycle Iteration n
Project Requirements Analysis & Implementation
Planning Capture Design
Build 0 Build 1 Build 2
Test Lifecycle
Execute
Plan Design Setup
Test
Test Test Test Bed
Evaluate
Test
Defect Tracking/Enhancement Request

Testing Lifecycle
Requirements
Capture
Analysis The Scenarios Design and the Test Case

Development could normally start in
parallel with the Development Cycle.
Test planning & Test Execution Synchs up with the
Scenario Design Development Cycle during the
functional Testing Phases
New Version Test Case
Development
Defect Fixing
Test Execution
Cycle
Defects Test Cycle

Closure
Test Result
Analysis
Testing Lifecycle
Test requirements identification

 Validate for testability
Test Planning and Scenario design
 Develop Test Objectives
 Identify Test Items
 Resources and Schedules
Test Case Development
 Test Case Specification
 Pre-requisites, post-requisites and Acceptance criteria

Testing Lifecycle (contd.)
Prepare Test Bed

 Test Data Preparation
 Test Environment Setup
Test Execution
 Run Tests and Validate Results
 Bug Reporting
 Bug fixes and retesting
Test Result Analysis
 Defect Analysis
 Determining Test Set Coverage and Effectiveness

Start Testing Early
There are two basic forms of testing

 Verification Testing
 Validation Testing
Verification: “Are we building it right?”
Validation: “Are we building the right thing?”

Testing Stages – the V-Model
Requirement Analysis User Acceptance Testing
Functional Specifications System Testing
High-Level Design Integration Testing

Reviews
Verification Validation
Detailed Design Unit Testing
Code

Testing Techniques
Static Testing
Dynamic Testing
White Box Testing
Black Box Testing

Testing Techniques (contd.)
Static Testing/Reviews
 Scope
Requirements Feasibility
Technical Architecture
Design
Program specifications
Test Plans
Test Cases and Data
User Documents

Static Testing
 Advantages
Capture defects early, so saves rework cost
Checklist-based approach
Focuses on coverage
Group perspective
Highest probability of finding defects
 Disadvantages
Time-consuming
Cannot test data dependencies
High skill levels required

Dynamic Testing
 White-box/Glass-box Testing
Unit Testing
Integration/Interface Testing
 Black-box Testing
System Testing
Acceptance testing

White-Box/Glass-box Testing
Structural Testing process

Program Logic-driven Testing
Design-based Testing
Examines the internal structure of program

White-Box Testing – Advantages
High Code coverage – Exhaustive (thorough)

path testing
Program logic is tested
Internal boundaries are tested
Testing is algorithm specific

Black-box Testing
Specifications-based Testing
Business Transaction-driven Testing
No concern to internal behavior and program
structure

Black-box Testing – Advantages
User’s perspective
Focus on features, not implementation
Big-picture approach

Levels of Testing
Unit Testing
Integration Testing
}
Function Testing
Performance Testing System
Acceptance Testing Testing
Installation Testing

Unit Testing
E&R Infosys 28
Testing Stages – the V-model

Reviews
Code

What is Unit Testing?
Testing of individual units or groups of related

code is known as Unit Testing
Unit tested with a Unit Test Plan

Unit Test Plan should test for:
Standards specified (Ex. GUI related)

Field validations, boundary values
Basic functionality of the component
(add/delete/modify/query activities)
Code coverage and exception handling
Negative or destructive testing
Memory leaks
Data computation/integrity

Unit Test Plan should test for:
Usability
Standards specified (Ex. GUI related)
Field validations, boundary values
Basic functionality of the component
Functionality
(add/delete/modify/query activities)
Code coverage and exception handling
Negative or destructive testing
Memory leaks Performance
Data computation/integrity

Unit Testing – White Box Technique
Activities
 Identification of Testable units/modules
 Focus on code and logic
 Design logic-based Test cases and prepare Test data
 Run tests and verify output through path traversing

Unit Testing – White Box Technique
Logic Coverage method includes:

 Statement coverage
 Decision (branch) coverage
 Condition Coverage
 Path Coverage

Unit Testing - Black Box Technique
Commonly used:
 Equivalence partitioning
 Boundary value analysis
 Error Guessing
Less commonly used:
 Cause effect graphing
 State-transition testing

Equivalence Partitioning
Systematic process of identifying a set of input

conditions to be tested
The two distinct steps involved:

 Identify equivalence classes
 Identify test cases

Equivalence Classes – Valid Data

 User supplied command
 Responses to system prompts
 File names
 Computational data
 Physical parameters, bounding values, initialization
values
 Output data formatting commands
 Responses to error messages
 Graphical data (e.g. mouse picks)

Equivalence Classes – invalid Data

 Data outside bounds of the program
 Physically impossible data
 Proper value supplied in the wrong place

Boundary Value Analysis
Check edge values of all

 Output domain
 User queries
 Numerical data
 Mouse picks on menu
 Output format requests
 Responses to prompts
 Command key input

Error Guessing
Ad Hoc method to identify tests likely to expose

errors based on experience and intuition
Some areas to guess are:
 Empty or Null lists / strings
 Zero instances / Occurrences
 Blanks or null characters in strings
 Negative numbers

Advantage/Disadvantage
 Significantly reduces number of input conditions to be
tested
 Does not test combinations of input conditions
Boundary Value Analysis

 Boundary conditions sometimes subtle and difficult to
identify
 Does not test combinations of input conditions

Exercise
Writing a unit test plan…

Recap
What is quality
Dimensions of Quality
Testing Techniques
Unit Test

Integration Testing
E&R Infosys 44

Reviews
Code

Integration Testing
Combining and testing multiple components

together
Integration of modules, programs and functions
Tests Internal Program interfaces
Tests External interfaces for modules

Integration Testing
Banking System External
System
External
interface
CRM HR Financial
Inter
module
Online banking Loans Account Opening
interface
Java Based Based on C, C++ VB based
Intra
module
interface
Unit Module

Approach for testing interfaces
Intra-Module Inter-Module External
Test Techniques White Box Black Box Black Box
Test Development Isolated Simulate live

Environment environment development environment
Setup environment
Test Approach Check for Identify pre-process Identify
parameter / and approach to external /
procedure called. Interface. internal interface
Database Identify the post- and validate
updation process to validate against external
the interface system

Techniques based on order of Integration
Big-bang Integration (non-incremental)

Random Incremental Integration
Top-Down Integration
Bottom-up Integration

Top-down vs. Bottom-up
Top-down Integration
 Advantage
Key Interface defects trapped earlier
 Disadvantage
Core functionality tested late in the cycle
Bottom-up Integration
 Advantage
Core functionality tested early
 Disadvantage
Key Interface defects trapped late in the cycle

System Testing
E&R Infosys 51

Reviews
Code

System Testing – What is it?
A Black-box Testing technique

Specifications-based Testing
Typically independent team testing
Simulated environment testing
Live/Simulated user data
Tests the whole system
Functional and non-functional requirements tested
Business transaction-driven testing
Compatibility errors uncovered
Performance limitations uncovered

System Testing – Different Types
Functional testing (Sanity / Regression )

Performance and Scalability testing
Load/Stress testing
Volume Testing
Usability testing
Configuration testing
Installability testing
Disaster and Recovery testing
Reliability / Availability testing
Statistical Testing based on Usage Models

Functional Testing – Sanity Test
Very basic minimal number of tests to verify the

product for the feature / protocol compliance
Could be a subset of regression test used to
validate only high level functionality
Typically an initial testing effort to determine if a
new software version is performing well enough
to accept it for a major testing effort
Sanity testing will be performed in scenarios
when cursory testing is sufficient to prove that
the application is functioning properly

Why Sanity Testing?
Regression testing is more time taking and more

effort is required whereas Sanity testing is
quicker focusing on important functionality of
software
Saves time as only selected high level
functionality areas are only tested. Bugs found
this way would take almost triple the time when
found through regression testing
Efficient in scenarios wherein releases comes in
pretty often with very few changes
Functional Testing – Regression
Re-execution of one or more tests in subsequent

build of the application/product to ensure
 Revisiting and testing all prior bug-fixes in response
to a new fix/enhancement
 Re-testing all programs that might be affected by the
fix/enhancement
 Hidden Bugs are uncovered
The baseline for Regression Testing grows with
every build

Regression Testing
Should Cover
 Business Process depending on criticality
 User friendliness
 Cross functional dependencies
Activities
 System appreciation
 Preparation of Test case repository
 Automate test cases
 Execute Regression test suite

Without Regression Testing – Hidden Bugs
Software Builds
A B C
Unknown Unknown
Test PASS PASS? PASS?
Requirement 1
Fix Fix
Test FAIL FAIL PASS
Requirement 2
Test PASS BUG? BUG?

Requirement 3
New Hidden
Problem Bug

With Regression Testing – No Hidden Bugs
Software Builds
A B C
Test PASS PASS PASS

Requirement 1
Fix Fix
Test FAIL FAIL PASS
Requirement 2
Fix
Test PASS BUG PASS
Requirement 3
New
Problem

Acceptance Testing
Demonstrates that the system meets mutually

agreed Acceptance criteria
 Critical Requirements
 Minimum Performance level
 Maximum Defect Detection Rate
Typically, a sub-set of System testing

Alpha / Beta Testing
Forms of Acceptance testing

Testing in the production environment
Alpha testing is performed by end users within a
company but outside development group
Beta testing is performed by a sub-set of actual
customers outside the company

Performance Testing
Number of concurrent users accessing at any point in

given time
System’s performance under high volume of data
Stress testing for systems, which are being scaled up to
larger environments or implemented for the first time
Operational intensive transactions. (Most frequently used
transactions)
Volume intensive transactions (for both volume and
stress testing)

Installation Testing
Basic installation
Installation of various configurations
Installation on various platforms
Regression testing of basic functionality

System Testing
A Banking System
CRM HR Financial
Online banking Loans Account Opening
Java Based Based on C, C++ VB based

When do you stop testing?
When time runs out

Specific test case design methods
Target test coverage attained
Certain number of errors found*
Error detection rate drops below a threshold*
Predicted density of errors left drops below a threshold
Sequence test cases in order of priority
* Infosys Process Capability Baseline

Test Artifacts
Test Plan
Test Environment/Bed
Test Case
Test Data
Test Tools
Test Scripts
Test Log
Bug Reports

Traits of a good Tester
Destructive creativity
Detective skills
Appreciating the users’ perspective
Adapt to and Understand Requirements change
A skeptical, but not hostile attitude
An eagerness to embrace new technologies

Test Engineers
Role - Execute tests, record test scripts,

maintain statistics and metrics, check test data
setup, test environment setup, execute re-tests
Knowledge - Understanding of the importance of
testing, awareness of tools, how to progress
against a plan
Skills - Observation, accuracy, methodical, co-
ordination, problem solver
Experience - Following instructions, problem
reporting and solving, and relevant testing tools

Test Automation
Good candidates:
 Complex and time-consuming tests
 Tests requiring a great deal of precision
 Tests involving many simple, repetitive tests
 Tests involving many data combinations
Lower priority candidates:
 One-time only tests
 Testing peripheral devices
 Subjective assessment tests (look and feel based)

Test Automation
Effective
Automated Test
(After many runs)
Manual Test
Economic Evolvable
First Run of
Automated Test
Exemplary
Simple Vs Sophisticated Automation
Maintenance Cost
Implementation Cost
C
O
S
T
Simple Sophisticated
Automation Automation
Testing Tools
E&R Infosys 73
Types of Testing Tools
The software testing tools broadly of the

following type:
 Functional / regression
 Load /Stress: Performance testing under high load /
stress
 Web Testing: testing the Web Sites
 Data Generation: Generation of test data

Benefits
Reduces manpower and time

More coverage within the same testing time.
Testing products which are very difficult to test
manually. Example, Load Tests
Can look inside the software for memory leaks,
redundant code etc.

Test Tools (Client Server)
Category Tool Vendor

Source Code Testing BoundsChecker Numega
Pure Coverage Rational
Purify Rational
Jprobe Sitraka Software
ATTOLCoverage ATTOL Software
Functional Testing QA Run Compuware

Win Runner Mercury
Team Test Rational
QA Center Compuware
Performance Testing OptimizeIT VM Gear
Load Runner Mercury
QA Load Compuware

Test Tools (Web Applications)

Functional Testing Rational Robot Rational
Silk Test Segue Software
Performance Testing Silk Performer Segue Software
Web Load Radview
Link & HTML Testing NetMechanic Monte Sano
Linkbot Tetranet
Site Inspector
Doctor HTML Imageware

Test Tools (Network & Security)
Security & Communication SNMP Test Suite Interworking Labs Inc.
Sniffer Pro Network Associates Inc.
Lophtcrack
Saint, Satan, Sara SwTech
TCP Wrappers
TCP Dump

Going Forward…
E&R Infosys 79
Testing Lifecycle
Development Lifecycle Iteration n
Project Requirements Analysis & Implementation
Planning Capture Design
Build 0 Build 1 Build 2
Test Lifecycle
Execute
Plan Design Setup
Test
Test Test Test Bed
Evaluate
Test
Defect Tracking/Enhancement Request

Test Planning/Strategizing
Includes –
 Testing Objectives and Goals
 Test Strategy/Approach based on customer priorities
 Test Environment (Hardware, Software, Network,
Communication etc.)
 Features to test with priority/criticality
 Test Deliverables
 Test Procedure – Activities and tools
 Test Entry and Exit criteria
 Test Organization and Scheduling
 Testing Resources and Infrastructure
 Test Measurements/Metrics

Test Planning / Strategizing
Benefits
 Sets clear and common objectives
 Helps prioritize tests
 Facilitates Technical tasks
 Helps improve coverage
 Provides structure to activities
 Improves communication
 Streamlines tasks, roles and responsibilities
 Improves test efficiency
 Improves test measurability

Optimizing Testing
Categorize entire gamut of testing into Sanity / Regression

/Performance and Stress
Efficient Review Process
Use Tools to do the testing (both Automation and Simulators )
Look out for Automation if it helps ( leverage 24 hours a day )
Use Traceability Matrix
Tracking and Reporting of bugs and defects – Defining a process
Revisit the Test coverage and categorization on an ongoing basis
Risk strategy
Retest strategy

Recap
Integration Testing
System Testing
Test Automation
Testing Tools

E&R Infosys 85

Software Testing: ER/CORP/CRS/SE04/003 E&R Infosys 1

Uploaded by

Copyright:

Available Formats

You might also like

Software Testing: ER/CORP/CRS/SE04/003 E&R Infosys 1

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Software Testing: ER/CORP/CRS/SE04/003 E&R Infosys 1

Uploaded by

Copyright:

Available Formats

Software Testing

ER/CORP/CRS/SE04/003 E&R Infosys 1

ER/CORP/CRS/SE04/003 E&R Infosys 2

ER/CORP/CRS/SE04/003 E&R Infosys 3

“The degree to which a system, component or

ER/CORP/CRS/SE04/003 E&R Infosys 4

What the customer says it is…

ER/CORP/CRS/SE04/003 E&R Infosys 5

ER/CORP/CRS/SE04/003 E&R Infosys 6

ER/CORP/CRS/SE04/003 E&R Infosys 7

ER/CORP/CRS/SE04/003 E&R Infosys 8

A process of demonstrating that errors are not

ER/CORP/CRS/SE04/003 E&R Infosys 9

A process of executing a program with the intent

A “DESTRUCTIVE”, yet creative process

ER/CORP/CRS/SE04/003 E&R Infosys 10

Verifies that all requirements are implemented

ER/CORP/CRS/SE04/003 E&R Infosys 11

ER/CORP/CRS/SE04/003 E&R Infosys 12

Defect Tracking/Enhancement Request

ER/CORP/CRS/SE04/003 E&R Infosys 13

Analysis The Scenarios Design and the Test Case

Defects Test Cycle

Test requirements identification

ER/CORP/CRS/SE04/003 E&R Infosys 15

Prepare Test Bed

ER/CORP/CRS/SE04/003 E&R Infosys 16

There are two basic forms of testing

Verification: “Are we building it right?”

Validation: “Are we building the right thing?”

ER/CORP/CRS/SE04/003 E&R Infosys 17

Requirement Analysis User Acceptance Testing

Functional Specifications System Testing

High-Level Design Integration Testing

Detailed Design Unit Testing

ER/CORP/CRS/SE04/003 E&R Infosys 18

ER/CORP/CRS/SE04/003 E&R Infosys 19

ER/CORP/CRS/SE04/003 E&R Infosys 20

ER/CORP/CRS/SE04/003 E&R Infosys 21

ER/CORP/CRS/SE04/003 E&R Infosys 22

Structural Testing process

ER/CORP/CRS/SE04/003 E&R Infosys 23

High Code coverage – Exhaustive (thorough)

ER/CORP/CRS/SE04/003 E&R Infosys 24

ER/CORP/CRS/SE04/003 E&R Infosys 25

ER/CORP/CRS/SE04/003 E&R Infosys 26

ER/CORP/CRS/SE04/003 E&R Infosys 27

Requirement Analysis User Acceptance Testing

Functional Specifications System Testing

High-Level Design Integration Testing

Detailed Design Unit Testing

ER/CORP/CRS/SE04/003 E&R Infosys 29

Testing of individual units or groups of related

ER/CORP/CRS/SE04/003 E&R Infosys 30

Standards specified (Ex. GUI related)

ER/CORP/CRS/SE04/003 E&R Infosys 31

ER/CORP/CRS/SE04/003 E&R Infosys 32