First Vietnamese QA Firm

LQA – Toward the Perfection 1

 Testing Concepts

Presented by: Tham Nguyen

Hanoi, Feb 2017

LQA – Toward the Perfection 2

AGENDA
 Duration: 2 Hours
 Purpose: Introduce overview and fundamental concepts of Software
testing
 Audience: Testers
 Content:
1. What is quality? ISO 9126
2. V&V definition
3. What is testing? Testing objectives
4. Testing concepts/terms
5. Why testing?
6. General testing principles

LQA – Toward the Perfection 3

What is Quality?
• In the context of software engineering, software quality refers to two related but
distinct notions ( khác biệt ) that exist wherever quality is defined in a business
context:
 Software functional quality reflects how well it complies with ( tuân thủ ) or conforms to
(phù hợp ) a given design, based on functional requirement or specifications. ( thông số
kỹ thuật)

 Software structural quality refers to how it meets  non-functional requirement  that

support the delivery of the functional requirements

LQA – Toward the Perfection 4

Cost of Quality
• The "cost of quality" isn't the price of creating a quality product or service. It's the
cost of NOT creating a quality product or service.
• Quality Costs—general description
 Prevention Costs
 Appraisal Costs ( chi phí thẩm định )
 Failure Costs
 Internal Failure Costs
 External Failure Costs

LQA – Toward the Perfection 5

Cost of Quality
• The "cost of quality" isn't the price of creating a quality product or service.
• It's the cost of NOT creating a quality product or service.

• Quality Costs—general description

• Prevention Costs ( phòng ngừa )
• The costs of all activities specifically designed
• to prevent poor quality in products or services.

• Examples are the costs of:

• New product review
• Quality planning
• Supplier capability surveys (khảo sát năng lực của nhà cung cấp)
• Process capability evaluations (đánh giá khả năng xử lý)
• Quality improvement team meetings
• Quality improvement projects
• Quality education and training
• Appraisal Costs
• The costs associated with ( liên quan) measuring, evaluating or auditing ( kiếm toán ) products or services
• to assure conformance to quality standards and performance requirements.( yêu cầu về hiệu suất)

• These include the costs of:

• Incoming and source inspection/test of purchased material
• In-process and final inspection/test
• Product, process or service audits
• Calibration( hiệu chuẩn) of measuring and test equipment ( thử nghiệm )
• Associated supplies and materials ( vật tư và vật liệu liên quan)
• Failure Costs
• The costs resulting from products or services not conforming to requirements or customer/user needs.
• Failure costs are divided into internal and external failure categories.
• Internal Failure Costs
• Failure costs occurring prior to delivery or shipment of the product,
• or the furnishing ( cung cấp ) of a service, to the customer.

• Examples are the costs of:

• Scrap ( sắt vụn )

• Rework
• Re-inspection
• Re-testing
• Material review
• Downgrading ( hạ cấp )
• External Failure Costs
• Failure costs occurring after delivery or shipment of the product —
• and during or after furnishing of a service — to the customer.

• Examples are the costs of:

• Processing customer complaints
• Customer returns
• Warranty claims
• Product recalls

LQA – Toward the Perfection 6

Quality – Views
Different views:
• User’s view: quality is fitness for user’s needs
 Fit for use
• Product view: quality is meet attributes of a product
• Manufacturer view: quality is meet the requirements specification
• Value-based view: quality is the customers’ willingness to pay for a software

LQA – Toward the Perfection 7

Quality is People problem?
• It is wrong to assume that people alone are responsible for poor quality.

People Process

Quality

Tools
Figure 4: People, Process and Tools All Influence Product Quality

LQA – Toward the Perfection 8

The Quality Triangle
Product’s View

Product

User Requirements Requirements Specification

Manufacturer’s View
User’s View
Figure 5: The Quality Triangle

LQA – Toward the Perfection 9

The User Requirements – Requirement Specification gap

The gap is likely to include:

• Misunderstood requirements
• Ignored requirement
• Missing requirements
• Outdated requirements
• Unneeded requirements

LQA – Toward the Perfection 10

The Requirement Specification - Software Gap

The gap is likely to include:

• Wrong interpretation ( giải thích ) of requirement due to vagueness ( mơ hồ ) and
ambiguity ( không rõ rang ) in the specification ( thông số kĩ thuật )
• Requirements identified after development commenced and included in the software
but not described in the specification
• Changes to specified requirements identified after development commenced
• Features added by the developers to exploit technical opportunities
• Requirement ignored by the developers because they were too difficult to implement

LQA – Toward the Perfection 11

The Software – User Requirement Gap

• The gap occurs because the software doesn’t satisfy the user requirements
• The size of the gap is directly dependent on the side of other 2 sides of triangle
• The gap is most expensive to close since extensive rework will be necessary to
modify the software

LQA – Toward the Perfection 12

Closing Gaps between 3 views

• Spend more money on quality:

– Early requirement validation
– Configuration management
– Reviews, Inspection
– Testing
• Improve the processes that create the product

LQA – Toward the Perfection 13

ISO 9126 – Product quality model
ISO 9126 has been replaced by ISO/IEC 25010:2011

ISO 9126
Quality Model

Functionality Reliability Usability Efficiency Maintability Portability

Suitability Understandability Analysability Adaptability

Maturity
Accuracy Learnability Time behaviour Changeability Installability
Fault tolerance
Interoperability Operability Resource utilisation Stability Co-existence
Recoverability
Security Attractiveness Compliance Testability Replaceability
Compliance
Compliance Compliance Compliance Compliance

Figure 12: The ISO 9126 Quality Model

LQA – Toward the Perfection 14

ISO 25010 Product quality model

LQA – Toward the Perfection 15

Software Characteristics

• Functionality: the degree to which a product or system provides functions that meet
stated and implied needs when used under specified conditions
 Functional completeness. ( hoàn chỉnh)
 Functional correctness. ( đúng đắn)
 Functional appropriateness. (phù hợp)

LQA – Toward the Perfection 16

Software Characteristics
• Performance efficiency: the performance relative to the amount of resources
used under stated conditions.
 Time behavior. .
 Resource utilization. ( tận dụng nguồn tài nguyên )
 Capacity. (sức chứa)

LQA – Toward the Perfection 17

Software Characteristics
• Compatibility: ( tương thích) to which a product, system or component can
exchange information with other products, systems or components, and/or
perform its required functions, while sharing the same hardware or software
environment
 Co-existence. (đồng tồn tại )
 Interoperability. Khả năng tương tác

LQA – Toward the Perfection 18

Software Characteristics

• Usability: (khả năng sử dụng)to which a product or system can be used by specified
users to achieve specified goals with effectiveness, efficiency and satisfaction in a
specified context of use
 Appropriateness recognizability. ( tính phù hợp có thể nhận biết được)
 Learnability. Khả năng học hỏi
 Operability. Khả năng hoạt động
 User error protection. 
 User interface aesthetics. Tính thẩm mỹ
 Accessibility. Khả năng tiếp cận

LQA – Toward the Perfection 19

Software Characteristics

• Reliability: ( tính thực tế )to which a system, product or component performs specified
functions under specified conditions for a specified period of time
 Maturity. Trưởng thành
 Availability. Khả dụng
 Fault tolerance. Khả năng chịu lỗi
 Recoverability (khả năng phục hồi)

LQA – Toward the Perfection 20

Software Characteristics
• Security: bảo mật to which a product or system protects information and data so that
persons or other products or systems have the degree of data access appropriate to
their types and levels of authorization
 Confidentiality. (bảo mật)
 Integrity. Chính thực
 Non-repudiation. Không bác bỏ
 Accountability. Trách nhiệm giải trình
 Authenticity. Tính xác thực

LQA – Toward the Perfection 21

Software Characteristics
• Maintainability: the degree of effectiveness and efficiency with which a product or
system can be modified to improve it, correct it or adapt(thích ứng) it to changes in
environment, and in requirements
 Modularity. Tính mô dun
 Reusability. Khả năng Tái sử dụng
 Analyzability. Kn phân tích
 Modifiability. Kn sửa đổi
 Testability. Kn kiểm tra

LQA – Toward the Perfection 22

Software Characteristics

• Portability:tính khả chuyền Degree of effectiveness and efficiency with which a

system, product or component can be transferred from one hardware, software or
other operational or usage environment to another.
 Adaptability kn thích ứng
 Installabilitykn cài đặt
 Replaceabilitykn thay thế

LQA – Toward the Perfection 23

Verification
Software

Verification
Are we building the
system correctly?

User Requirements Requirements Specification

Figure 2: Verification and the Quality Triangle

– Verification confirms that work products properly reflect the requirements

specified for them.
– Did we build the system right?
– Example: SRS, design and code review by development team. Unit test,
integration and system test

LQA – Toward the Perfection 24

Validation
Software

Validation
Are we building the
correct system?

User Requirements Requirements Specification

Figure 1: Validation and the Quality Triangle

 Validation check whether a function needed and expected by

customers is present in a software product
 Did we build the right system, appropriate, fit-for-use?
 Example: SRS, Requirement prototype review by customer,
acceptance test, beta test, operational support
LQA – Toward the Perfection 25
Fundamentals of Testing

LQA – Toward the Perfection 26

What is Testing?

• Testing: the process consisting of all life cycle activities, both static and dynamic,
concerned with planning, preparation and evaluation of software product and related
work products to determine that they satisfy specified requirements, to demonstrate
that they are fit for purpose and to detect defects (ISTQB glossary)
• Testing: is the process of exercising or evaluating a system or system component by
manual or automated means to verify that it satisfies specified requirements (IEEE)

LQA – Toward the Perfection 27

Testing Objectives

• Primary:
– Execute a program with the intent of finding defects to
• Determine whether system meets specifications
• Determine whether system meets user’s needs
– Gaining confidence in and providing information about the level of quality
– Preventing defects
• Secondary:
– Continuously improve the testing process

LQA – Toward the Perfection 28

Testing Terms
 Process: sequence of steps performed for a given purpose. (IEEE)
 Software Process: a set of activities, methods, practices, and transformations that
people use to develop and maintain software and associated products.(SEI-CMM)
 Error: A human action that produces an incorrect result. Known as mistake
 Fault: An incorrect step, process, or data definition. Known as defect or bug
 Failure: a behavioral deviation from user requirement or product specification
 Errors  Faults  Failures
 Leakage: defects found in a product after product released

LQA – Toward the Perfection 29

Relation of failure, fault and error

A person makes
an error ...

… that creates
fault(s) in the
software ...

… that can cause

failure(s)
in operation

LQA – Toward the Perfection 30

Why is testing necessary?
• Because software is likely to have faults
• To learn about the reliability of the software
• Because failures can be very expensive
• To avoid being sued by customers
• To stay in business

LQA – Toward the Perfection 31

Software Failures
• Software failures can lead to:
– Loss of money
– Loss of time
– Loss of business reputation
– Injury/Death

LQA – Toward the Perfection 32

Examples
• Loss of Money/Time

– In May 1992, Pepsi fan a promotion in the Philippines.

customers could win a million pesos (approx. $40,000) if they bought a bottle
of Pepsi and found number 349 stamped on the underside of the bottle cap.

 Unfortunately, due to a software error, 800,000 bottle caps were produced

with number 349 instead of one, which was an equivalent of $32 billion in prize
money.

LQA – Toward the Perfection 33

Examples

• Loss of Business Reputation

– If the company website has some spelling mistakes in the text, potential customers may be put off
the company as it looks unprofessional.

• Injury/Death
– A failure of the train’s on-board systems or of the signaling system that controls the train’s
movements could lead to catastrophic results

LQA – Toward the Perfection 34

Why can't we test our own work ?

– We make assumptions
– We are emotionally attached to the product (it's our baby and there's nothing wrong
with it).
– We are so familiar with the product we cannot easily see the obvious faults
– We have a vested interest in passing the product as ok and not finding faults

LQA – Toward the Perfection 35

Independent Testing
 Tests by the person who wrote the item under test;
 Tests by another person within the same team, such as another programmer
 Tests by a person from a different organizational group, such as an independent
test team;
 Tests designed by a person from a different-organization or company, such as
outsourced testing or certification by an external body.

LQA – Toward the Perfection 36

Testing principles

LQA – Toward the Perfection 37

Testing Principles

• Testing shows presence of defects

• Exhaustive testing is impossible
• Early testing
• Defect clustering
• Pesticide paradox
• Testing is context dependent
• Absence-of-errors fallacy

LQA – Toward the Perfection 38

P1: Testing shows presence of defects
• Testing can show that defects are present, presence of defects but cannot prove
that there are no defects. Testing reduces the probability of undiscovered defects
remaining in the software but, even if no defects are found, it is not a proof of
correctness.
– Ex: We see many white swans, we cannot say 'All swans are white'. Otherwise,
we see one black swan, we can say 'Not all swans are white'.

LQA – Toward the Perfection 39

P2: Exhaustive testing is impossible

• Testing everything (all combinations of inputs and preconditions) is not feasible

except for trivial cases. Instead of exhaustive testing, we use risks and priorities to
focus testing efforts
• Ex: A software program designed to count the number of vowels (a, e, i, o, u) in a
string of five letters. To test this program with all possible inputs, require 265
(11,881376)cases
 How many test cases enough?  Pairwise testing.

LQA – Toward the Perfection 40

P3: Early Testing
 Testing activities should start as early as possible in the software or system
development life cycle and should be focused on defined objectives
 Early testing such as early test design and review activities , finds defects early
on when they are cheap to find and fix.

LQA – Toward the Perfection 41

P4: Defect Clustering

• A small number of modules contain most of the defects discovered during pre-
release testing or show the most operational failures

LQA – Toward the Perfection 42

P5: Pesticide Paradox

• If the same tests are repeated over and over again, eventually the same set of test
cases will no longer find any new bugs. To overcome this 'pesticide paradox', the test
cases need to be regularly reviewed and revised, and new and different tests need to
be written to exercise different parts of the software or system to potentially find
more defects.

LQA – Toward the Perfection 43

P6: Testing is context dependent
 Testing is done differently in different contexts. For example, safety-critical
software is tested differently from an e-commerce site.

LQA – Toward the Perfection 44

P7: Absence-of-errors fallacy
Finding and fixing defects does not help if the system built is unusable and does
not fulfill the users' needs and expectations.

LQA – Toward the Perfection 45

QnA/Discussion

LQA – Toward the Perfection 46

LQA – Toward the Perfection 47
Reference

• Foundations of Software Testing_ISTQB Certification, Chapter 1

• Foundation Syllabus_2011
• ISTQB Glossary of Testing Terms v3.01
– http://www.istqb.org/

LQA – Toward the Perfection 48

 Contact Us!

Website: http://lqa.com.vn
Email: hello@lqa.com.vn
Phone: +84 90 414 0881

LQA – Toward the Perfection 49

 <Document name>

PRESENTED BY: AUTHOR

LQA – Toward the Perfection 50

AGENDA

LQA – Toward the Perfection 51

QnA/Discussion

LQA – Toward the Perfection 52

LQA – Toward the Perfection 53
REFERENCE
 http://www.istqb.org/
 ..

LQA – Toward the Perfection 54

Contact Us!

Website: http://lqa.com.vn
Email: hello@lqa.com.vn
Phone: +84 90 414 0881
LQA – Toward the Perfection 55