Slides of Test and SQA

Software Testing and SQA
Chapter 9
Quản lý chất lượng PhầnMềm
Kiểm thử và Đảm bảo Chất lượng Phần mềm

Mục tiêu
Giới thiệu quá trình quản lý chất lượng và

các hoạt động QLCL chính
Giải thích vai trò của chuẩn trong QLCL
Giải thích khái niệm về độ đo phần mềm, đọ
đo dự báo và độ đo điều khiển
Giải thích độ đo có thể sử dụng ntn để đánh
giá chất lượng phần mềm và hạn chế của
cách đo phần mềm
Kiểm thử và Đảm bảo Chất lượng Phần mềm 9.2

Topics covered
Process and product quality

Quality assurance and standards
Quality planning
Quality control

Software quality management
Concerned with ensuring that the required

level of quality is achieved in a software
product.
Involves defining appropriate quality
standards and procedures and ensuring that
these are followed.
Should aim to develop a ‘quality culture’
where quality is seen as everyone’s
responsibility.

What is quality?
Quality, simplistically, means that a product should
meet its specification.
This is problematical for software systems
There is a tension between customer quality requirements
(efficiency, reliability, etc.) and developer quality
requirements (maintainability, reusability, etc.);
Some quality requirements are difficult to specify in an
unambiguous way;
Software specifications are usually incomplete and often
inconsistent.

The quality compromise
We cannot wait for specifications to improve

before paying attention to quality
management.
We must put quality management procedures
into place to improve quality in spite of
imperfect specification.

Scope of quality management
Quality management is particularly important

for large, complex systems. The quality
documentation is a record of progress and
supports continuity of development as the
development team changes.
For smaller systems, quality management
needs less documentation and should focus
on establishing a quality culture.

Quality management activities
Quality assurance
Establish organisational procedures and standards for
quality.
Quality planning
Select applicable procedures and standards for a
particular project and modify these as required.
Quality control
Ensure that procedures and standards are followed by
the software development team.
Quality management should be separate from
project management to ensure independence.

Quality management and software development

Process and product quality
The quality of a developed product is

influenced by the quality of the production
process.
This is important in software development as
some product quality attributes are hard to
assess.
However, there is a very complex and poorly
understood relationship between software
processes and product quality.

Chất lượng căn cứ vào tiến trình (1)
There is a straightforward link between process and product in

manufactured goods.
More complex for software because:
The application of individual skills and experience is
particularly important in software development;
External factors such as the novelty of an application or
the need for an accelerated development schedule may
impair product quality.
Care must be taken not to impose inappropriate process
standards - these could reduce rather than improve the product
quality.

Chất lượng căn cứ vào tiến trình (2)

Practical process quality
Define process standards such as how

reviews should be conducted, configuration
management, etc.
Monitor the development process to ensure
that standards are being followed.
Report on the process to project
management and software procurer.
Don’t use inappropriate practices simply
because standards have been established.

Đảm bảo chất lượng và chuẩn chất lượng
Standards are the key to effective quality

management.
They may be international, national,
organizational or project standards.
Product standards define characteristics that
all components should exhibit e.g. a
common programming style.
Process standards define how the software
process should be enacted.

Tầm quan trọng về chuẩn
Encapsulation of best practice- avoids

repetition of past mistakes.
They are a framework for quality assurance
processes - they involve checking compliance
to standards.
They provide continuity - new staff can
understand the organisation by
understanding the standards that are used.

Chuẩn sản phẩm và chuẩn tiến trình
Product standards Process standards

Design review form Design review conduct
Requirements document structure Submission of documents to CM
Method header format Version release process
Java programming style Project plan approval process
Project plan format Change control process
Change request form Test recording process

Các vấn đề về chuẩn
They may not be seen as relevant and up-to-

date by software engineers.
They often involve too much bureaucratic
form filling.
If they are unsupported by software tools,
tedious manual work is often involved to
maintain the documentation associated with
the standards.

Standards development
Involve practitioners in development. Engineers

should understand the rationale underlying a
standard.
Review standards and their usage regularly.
Standards can quickly become outdated and this
reduces their credibility amongst practitioners.
Detailed standards should have associated tool
support. Excessive clerical work is the most
significant complaint against standards.

ISO 9000
An international set of standards for quality

management.
Applicable to a range of organisations from
manufacturing to service industries.
ISO 9001 applicable to organisations which
design, develop and maintain products.
ISO 9001 is a generic model of the quality
process that must be instantiated for each
organisation using the standard.

ISO 9001
Management responsibility Quality system

Control of non-conforming products Design control
Handling, storage, packaging and Purchasing
delivery
Purchaser-supplied products Product identification and traceability
Process control Inspection and testing
Inspection and test equipment Inspection and test status
Contract review Corrective action
Document control Quality records
Internal quality audits Training
Servicing Statistical techniques

ISO 9000 certification
Quality standards and procedures should be
documented in an organisational quality
manual.
An external body may certify that an
organisation’s quality manual conforms to ISO
9000 standards.
Some customers require suppliers to be ISO
9000 certified although the need for flexibility
here is increasingly recognised.

ISO 9000 and quality management

Documentation standards
Particularly important - documents are the tangible

manifestation of the software.
Documentation process standards
Concerned with how documents should be developed,
validated and maintained.
Document standards
Concerned with document contents, structure, and
appearance.
Document interchange standards
Concerned with the compatibility of electronic
documents.

Documentation process

Document standards
Document identification standards

How documents are uniquely identified.
Document structure standards
Standard structure for project documents.
Document presentation standards
Define fonts and styles, use of logos, etc.
Document update standards
Define how changes from previous versions are
reflected in a document.

Document interchange standards
Interchange standards allow electronic documents

to be exchanged, mailed, etc.
Documents are produced using different systems
and on different computers. Even when standard
tools are used, standards are needed to define
conventions for their use e.g. use of style sheets
and macros.
Need for archiving. The lifetime of word processing
systems may be much less than the lifetime of the
software being documented. An archiving standard
may be defined to ensure that the document can be
accessed in future.

Quality planning
A quality plan sets out the desired product

qualities and how these are assessed and
defines the most significant quality attributes.
The quality plan should define the quality
assessment process.
It should set out which organisational
standards should be applied and, where
necessary, define new standards to be used.

Quality plans
Quality plan structure

Product introduction;
Product plans;
Process descriptions;
Quality goals;
Risks and risk management.
Quality plans should be short, succinct
documents
If they are too long, no-one will read them.

Software quality attributes
Safety Understandability Portability

Security Testability Usability
Reliability Adaptability Reusability
Resilience Modularity Efficiency
Robustness Complexity Learnability

Quality control
This involves checking the software

development process to ensure that
procedures and standards are being followed.
There are two approaches to quality control
Quality reviews;
Automated software assessment and software
measurement.

Quality reviews
This is the principal method of validating the quality

of a process or of a product.
A group examines part or all of a process or system
and its documentation to find potential problems.
There are different types of review with different
objectives
Inspections for defect removal (product);
Reviews for progress assessment (product and process);
Quality reviews (product and standards).

Types of review
Review type Principal purpose

Design or program To detect detailed errors in the requirements, design or code. A checklist of
inspections possible errors should drive the review.
Progress reviews To provide information for management about the overall progress of the
project. This is b oth a process and a product review and is concerned with
costs, plans and schedules.
Quality reviews To carry out a t echnical analysis of product components or documentation to
find mismatches between the specification and the component design, code or
documentation and to ensure that defined quality standards have been followed.

Quality reviews
A group of people carefully examine part or all

of a software system and its associated
documentation.
Code, designs, specifications, test plans,
standards, etc. can all be reviewed.
Software or documents may be 'signed off' at a
review which signifies that progress to the next
development stage has been approved by
management.

Review functions
Quality function - they are part of the general

quality management process.
Project management function - they provide
information for project managers.
Training and communication function -
product knowledge is passed between
development team members.

Quality reviews
The objective is the discovery of system

defects and inconsistencies.
Any documents produced in the process may
be reviewed.
Review teams should be relatively small and
reviews should be fairly short.
Records should always be maintained of
quality reviews.

Review results
Comments made during the review should be

classified
No action. No change to the software or documentation
is
required;
Refer for repair. Designer or programmer should correct
an identified fault;
Reconsider overall design. The problem identified in the
review impacts other parts of the design. Some overall
judgement must be made about the most cost-effective
way of solving the problem;
Requirements and specification errors may
have to be referred to the client.

Software measurement and metrics
Software measurement is concerned with deriving a

numeric value for an attribute of a software product
or process.
This allows for objective comparisons between
techniques and processes.
Although some companies have introduced
measurement programmes, most organisations still
don’t make systematic use of software
measurement.
There are few established standards in this area.

Software metric
Any type of measurement which relates to a

software system, process or related documentation
Lines of code in a program, the Fog index, number of
person-days required to develop a component.
Allow the software and the software process to
be quantified.
May be used to predict product attributes or to
control the software process.
Product metrics can be used for general predictions
or to identify anomalous components.

Predictor and control metrics

Metrics assumptions
A software property can be measured.

The relationship exists between what we can
measure and what we want to know. We can only
measure internal attributes but are often more
interested in external software attributes.
This relationship has been formalised and
validated.
It may be difficult to relate what can be measured
to desirable external quality attributes.

Internal and external attributes

The measurement process
A software measurement process may be

part of a quality control process.
Data collected during this process should be
maintained as an organisational resource.
Once a measurement database has been
established, comparisons across projects
become possible.

Product measurement process

Data collection
A metrics programme should be based on a

set of product and process data.
Data should be collected immediately (not in
retrospect) and, if possible, automatically.
Three types of automatic data collection
Static product analysis;
Dynamic product analysis;
Process data collation.

Data accuracy
Don’t collect unnecessary data

The questions to be answered should be decided
in advance and the required data identified.
Tell people why the data is being collected.
It should not be part of personnel evaluation.
Don’t rely on memory
Collect data when it is generated not after a
project has finished.

Product metrics
A quality metric should be a predictor of

product quality.
Classes of product metric
Dynamic metrics which are collected by measurements
made of a program in execution;
Static metrics which are collected by measurements
made of the system representations;
Dynamic metrics help assess efficiency and reliability;
static metrics help assess complexity, understandability
and maintainability.

Dynamic and static metrics
Dynamic metrics are closely related to software

quality attributes
It is relatively easy to measure the response time of a
system (performance attribute) or the number of failures
(reliability attribute).
Static metrics have an indirect relationship with
quality attributes
You need to try and derive a relationship between these
metrics and properties such as complexity,
understandability and maintainability.

Software product metrics
Software metric Description
Fan in/Fan-out Fan-in is a measure of the number of functions or methods that call some other function
or method (say X). Fan-out is the number of functions that are called by function X. A
high value for fan-in means that X i s tightly coupled to the rest of the design and
changes to X will have extensive knock-on effects. A high value for fan-out suggests
that the overall complexity of X m ay be high because of the complexity of the control
logic needed to coordinate the called components.
Length of code This is a measure of the size of a p rogram. Generally, the larger the size of the code of a
component, the more complex and error-prone that component is likely to be. Length of
code has been shown to be one of the most reliable metrics for predicting error-
proneness in components.
Cyclomatic complexity This is a m easure of the control complexity of a p rogram. This control complexity may
be related to program understandability. I discuss how to compute cyclomatic
complexity in Chapter 22.
Length of identifiers This is a measure of the average length of distinct identifiers in a p rogram. The longer
the identifiers, the more likely they are to be m eaningful and hence the more
understandable the program.
Depth of conditional This is a measure of the depth of nesting of if-statements in a program. Deeply nested if
nesting statements are hard to understand and are potentially error-prone.
Fog index This is a measure of the average length of words and sentences in documents. The higher
the value for the Fog index, the more difficult the document is to understand.

Object-oriented metrics
Object-oriented Description
metric
Depth of inheritance This represents the number of discrete levels in the inheritance tree where sub-
tree classes inherit attributes and operations (methods) from super-classes. The
deeper the inheritance tree, the more complex the design. Many different object
classes may have to be understood to understand the object classes at the leaves
of the tree.
Method fan-in/fan- This is directly related to fan-in and fan-out as described above and means
out essentially the same thing. However, it may be appropriate to make a
distinction between calls from other methods within the object and calls from
external methods.
Weighted methods This is the number of methods that are included in a class weighted by the
per class complexity of each method. Therefore, a simple method may have a complexity
of 1 and a large and complex method a much higher value. The larger the value
for this metric, the more complex the object class. Complex objects are more
likely to be more difficult to understand. They may not be logically cohesive so
cannot be reused effectively as super-classes in an inheritance tree.
Number of This is the number of operations in a super-class that are over-ridden in a sub-
overriding class. A high value for this metric indicates that the super-class used may not be
operations an appropriate parent for the sub-class.

Measurement analysis
It is not always obvious what data means

Analysing collected data is very difficult.
Professional statisticians should be consulted
if available.
Data analysis must take local circumstances
into account.

Measurement surprises
Reducing the number of faults in a program

leads to an increased number of help desk
calls
The program is now thought of as more reliable
and so has a wider more diverse market. The
percentage of users who call the help desk may
have decreased but the total may increase;
A more reliable system is used in a different way
from a system where users work around the
faults. This leads to more help desk calls.

Key points
Software quality management is concerned with

ensuring that software meets its required standards.
Quality assurance procedures should be
documented in an organisational quality manual.
Software standards are an encapsulation of best
practice.
Reviews are the most widely used approach for
assessing software quality.

Key points
Software measurement gathers information

about both the software process and the
software product.
Product quality metrics should be used to
identify potentially problematical components.
There are no standardised and universally
applicable software metrics.

Danh sách các test tools (nc opensource)
• jUnit (test unit cho Java)
• csUnit (test unit cho .NET)
• Nunit (.Net framework)
• jTestcase
• Jfunc (test function)
• Ant/Nant
Cách thực hiện:

• Nghiên cứu các công cụ kiểm thử
• Viết một phần mềm đơn giản bằng java hoặc .NET
• Sử dụng các công cụ kiểm thử để test cho phần mềm trên.
Các loại kiểm thử
• Unit test
• Regrestion test
• Function test
• Integrated test
• Test UML test
• DB test
Kiểm thử và Đảm bảo
Chất lượng Phần mềm
GVC Thạc Bình Cường, cuongtb@it-hut.edu.vn

Cuongtb-fit@mail.hut.edu.vn
Mob: 0913226660
Dept. of SE, Faculty of IT, HUT
Acknowledgments: Many thanks to
Prof. Ian Sommerville and Prof. Roger S. Pressman
for providing the materials for this course
Vài nét về môn học KT&BĐCL
PM
Tổng quan về môn học

Mục tiêu
Đối tượng
Tổ chức môn học
Yêu cầu kỹ năng và hiểu biết
Sách giáo trình
Tự học
Giới thiệu về đối tượng

Course goals
A. Knowledge of testing process, classification
scheme, terminology
B. Ability to classify test terminology
C. Knowledge of/skill with several test generation
methods
D. Understanding of possibilities, limitations, required
effort, mutual relations of test generation methods
E. Ability to determine given a situation which test
generation on method is suitable
F. Verification and validation
G. Software cost estimation
H. SQA Management
Course overview
1 Introduction: Concepts
2 V&V
3 Testing
4 Data-based black box testing
5 Structure-based white box testing
6 Structure-based black box testing
7 Integration testing
8 Software reliability engineering
9 Testing in industry
10 SW cost estimation
11 SW quality management
12 Configuration management
13 Additional: Real-time & Embedded systems design & Test
Required knowledge, skills
Language (speak/understand/write) [Vietnamese/English/Japanese/…]
IT Fundamentals (Computer science):
Skill in programming (making+correcting errors)
SE Fundamentals, software development process
Behavioural specification languages: process algebra, state machines,
petri nets
Mathematics:
Logic, sets, reasoning
Statistics

Textbooks and References
1. Ian Sommerville: “Software Engineering”, 7th Ed., 2004.
2. Roger S. Pressman: “Software Engineering: A Practitioner's Approach”, 6th
Ed., McGraw-Hill, 2004.
3. John Musa: “Software Reliability Engineering”, McGraw-Hill, 1998.
4. Barry W. Boehm et al.: “Software Cost Estimation with COCOMO II”,
Prentice Hall PTR, 2000.
5. Websites on SW Testing and SQA
6. Materials, Handouts
7. David E. Simon: “An Embedded Software Primer”, Addison-Wesley, 1999.
8. Frank Vahid and Tony Givargis: “Embedded System Design. A Unified
Hardware/Software Introduction”, John Wiley & Sons, Inc.2002.
9. Phillip A. Laplante: “Real-time Systems Design and Analysis. An
Engineer’s Handbook”, 2nd Ed., IEEE Press, 1997.
10. A.W. Leigh: “Real-time Software for Small Systems”, Sigma Press, 1987.

Outline
Course overview
Chapter 1: Introduction to the subject
What is... error/bug/fault/failure/testing?
Overview of the testing process
Classifying aspects of testing

Chapter 1
Introduction: Concepts
What is...
error/bug/fault/failure/testing?
Overview of the testing process
Classifying aspects of testing

What is...
error: something wrong in software itself
fault: manifestation of an error
(observable in software behaviour) bug
failure: something wrong in software
behaviour
(deviates from requirements)
requirements: software: output (verbose):
for input i, i=input(STDIN); input: 6 fault
error
give output 2*(i^3) i=double(i); doubling input..
i=power(i,3); computing power..
(so 6 yields 432) output(STDOUT,i); output: 1728
fault + failure
What is...
Testing:
by experiment,
find errors in software (Myers, 1979)
establish quality of software (Hetzel, 1988)
a successful test:
finds at least one error
gives quality judgment with maximum
confidence with minimum effort

What’s been said?
Dijkstra:
Testing can show the presence of bugs, but not the absence
Beizer:
1st law: Every method you use to prevent or find bugs leaves a residue
of subtler bugs, for which other methods are needed
2nd law: Software complexity grows to the limits of our ability to manage
it
Beizer:
Testers are not better at test design than programmers are at code
design
...
¾ Let’s start with a global look at the testing process

Concept map of the testing process

Dimensions of software testing
1. What is tested?
a. Software characteristics (design/embedded?/language/...)
b. Requirements (what property/quality do we want from the software)
c. Purpose (what development phase, what kind of errors, what next step)
2. How to test?
a. Method (adequacy criterion: how to generate how many tests)
b. Assumptions (limitations, simplifications, heuristics)
c. Organization (documentation, implementation, execution:platform,
interactive, batch)
d. Who tests? (programmer, user, third party)
3. How are the results evaluated?

Dimensions + concept map
2d
2c
1a
1b
2b 2a 1c

1b. Requirements
functional:
the behaviour of the system is correct
precise
non-functional:
performance, reliability, compatibility,
robustness (stress/volume/recovery),
usability, ...
possibly quantifiable, always vague

1c. Test purpose
Software development phases (V-model):
requirements acceptance
test
system
specification test
detailed integration
design test
implementation unit
code test
1c. Test purpose
What errors to find?
typical types
functional mistakes
unimplemented required features
‘software does not do all it should do’

implemented non-required features
‘software does things it should not do’

2a. Test method dimensions
data-based
structure-based
error seeding black box white box

typical errors
efficiency
...

2b. Assumptions, limitations
Single/multiple fault:
clustering/dependency of errors
Testability:
can software be tested?
time, resources, accessibility,
Perfect repair
...

2c. Test organization
Documentation
for reuse!
Implementation
platform
batch?
inputs, coordination, ...
Execution
duration, timing, interruptions
manual/interactive or automated
in parallel on several systems
Æ classify aspects of testing
smoke testing
requirements: only major functions
purpose: often system testing
method: error guessing
assumption: (regression testing) a bit of testing suffices
capture/replay tool
test implementation & execution: the tool records test input
behaviour and saves it for automatic reuse
traceability matrix
requirements, test set: the matrix shows relationship between
these, gives some coverage confidence

Chapter 2
Verification and Validation
Thẩm tra và Phê Chuẩn
GVC Thạc Bình Cường
Dept. of SE, Faculty of IT, HUT

Objectives
To introduce software verification and
validation and to discuss the distinction
between them.
To describe the program inspection process
and its role in V & V.
To explain static analysis as a verification
technique.
To describe the Cleanroom software
development process.

Topics covered
Verification and validation planning
Software inspections
Automated static analysis
Cleanroom software development

Verification vs validation
Verification:
"Are we building the product right?”.
The software should conform to its
specification.
Validation:
"Are we building the right product?”.
The software should do what the user really
requires.

The V & V process
Is a whole life-cycle process - V & V
must be applied at each stage in the
software process.
Has two principal objectives
The discovery of defects in a system;
The assessment of whether or not the
system is useful and useable in an
operational situation.

V& V goals
Verification and validation should establish
confidence that the software is fit for
purpose.
This does NOT mean completely free of
defects.
Rather, it must be good enough for its
intended use and the type of use will
determine the degree of confidence that is
needed.

V & V confidence
Depends on system’s purpose, user
expectations and marketing environment
Software function
The level of confidence depends on how critical the
software is to an organisation.
User expectations
Users may have low expectations of certain kinds of
software.
Marketing environment
Getting a product to market early may be more
important than finding defects in the program.

Static and dynamic verification
Software inspections. Concerned with
analysis of the static system representation to
discover problems (static verification)
May be supplement by tool-based document and
code analysis
Software testing. Concerned with exercising
and observing product behaviour (dynamic
verification)
The system is executed with test data and its
operational behaviour is observed

Static and dynamic V&V

Program testing
Can reveal the presence of errors NOT their
absence.
The only validation technique for non-
functional requirements as the software has
to be executed to see how it behaves.
Should be used in conjunction with static
verification to provide full V&V coverage.

Types of testing
Defect testing
Tests designed to discover system defects.
A successful defect test is one which reveals the
presence of defects in a system.
Validation testing
Intended to show that the software meets its
requirements.
A successful test is one that shows that a
requirements has been properly implemented.

Testing and debugging
Defect testing and debugging are distinct
processes.
Verification and validation is concerned with
establishing the existence of defects in a
program.
Debugging is concerned with locating and
repairing these errors.
Debugging involves formulating a hypothesis
about program behaviour then testing these
hypotheses to find the system error.
The debugging process

V & V planning
Careful planning is required to get the most
out of testing and inspection processes.
Planning should start early in the
The plan should identify the balance between
static verification and testing.
Test planning is about defining standards for
the testing process rather than describing
product tests.

The V-model of development

The structure of a software test plan
The testing process.

Requirements traceability.
Tested items.
Testing schedule.
Test recording procedures.
Hardware and software requirements.
Constraints.
Lập kế hoạch kiểm thử
Quá trình kiểm thử
Mô tả các giai đoạn chính của quá trình kiểm thử .
Lần theo dấu vết

Người sử dụng hầu như chỉ quan tâm đến hệ thống có đáp ứng các yêu
cầu và kiểm thử cần lập kế hoạch sao cho các yêu cầu được kiểm thử
mọt cách độc lập .
Tested items
The products of the software process that are to be tested should be
specified.
Testing schedule
An overall testing schedule and resource allocation for this schedule. This,
obviously, is linked to the more general project development schedule.
Lập kế hoạch kiểm thử(cont.)
Thủ tục ghi chép lại kiểm thử.
It is not enough simply to run tests. The results of the tests must be
systematically recorded. It must be possible to audit the testing
process to check that it been carried out correctly.
Các yêu cầu về phần cứng và phần mềm

This section should set out software tools required and estimated
hardware utilisation.
Ràng buộc
Constraints affecting the testing process such as staff shortages
should be anticipated in this section.

Software inspections
These involve people examining the source
representation with the aim of discovering
anomalies and defects.
Inspections not require execution of a system
so may be used before implementation.
They may be applied to any representation of
the system (requirements,
design,configuration data, test data, etc.).
They have been shown to be an effective
technique for discovering program errors.
Inspection success
Many different defects may be
discovered in a single inspection. In
testing, one defect ,may mask another
so several executions are required.
The reuse domain and programming
knowledge so reviewers are likely to
have seen the types of error that
commonly arise.
Inspections and testing
Inspections and testing are complementary
and not opposing verification techniques.
Both should be used during the V & V
process.
Inspections can check conformance with a
specification but not conformance with the
customer’s real requirements.
Inspections cannot check non-functional
characteristics such as performance, usability,
etc.
Program inspections
Formalised approach to document reviews
Intended explicitly for defect detection (not
correction).
Defects may be logical errors, anomalies in
the code that might indicate an erroneous
condition (e.g. an uninitialised variable) or
non-compliance with standards.

Inspection pre-conditions
A precise specification must be available.
Team members must be familiar with the
organisation standards.
Syntactically correct code or other system
representations must be available.
An error checklist should be prepared.
Management must accept that inspection will
increase costs early in the software process.
Management should not use inspections for
staff appraisal ie finding out who makes
mistakes.
The inspection process

Inspection procedure
System overview presented to inspection
team.
Code and associated documents are
distributed to inspection team in advance.
Inspection takes place and discovered errors
are noted.
Modifications are made to repair discovered
errors.
Re-inspection may or may not be required.

Inspection roles
Author or owner
The programmer or designer responsible for producing
the program or document. Responsible for fixing defects
discovered during the inspection process.
Inspector
Finds errors, omissions and inconsistencies in programs
and documents. May also identify broader issues that are
outside the scope of the inspection team.
Reader
Presents the code or document at an inspection meeting.
Inspection roles (cont.)
Scribe (Secretary)
Records the results of the inspection meeting.
Chairman or moderator
Manages the process and facilitates the inspection.
Reports process results to the Chief moderator.
Chief moderator
Responsible for inspection process improvements,
checklist updating, standards development etc.

Inspection checklists
Checklist of common errors should be used to
drive the inspection.
Error checklists are programming language
dependent and reflect the characteristic
errors that are likely to arise in the language.
In general, the 'weaker' the type checking,
the larger the checklist.
Examples: Initialisation, Constant naming,
loop termination, array bounds, etc.

Inspection checks 1a
•Data faults
• Are all program variables initialised before
their values are used?
• Have all constants been named?
• Should the upper bound of arrays be
equal to the size of the array or Size -1?
• If character strings are used, is a
delimiter explicitly assigned?
• Is there any possibility of buffer overflow?

Inspection checks 1b
•Control faults
• For each conditional statement, is the
condition correct?
• Is each loop certain to terminate?
• Are compound statements correctly
bracketed?
• In case statements, are all possible cases
accounted for?
• If a break is required after each case in
case statements, has it been included?
Inspection checks 1c
•Input/output faults
• Are all input variables used?
• Are all output variables assigned a
value before they are output?
• Can unexpected inputs cause
corruption?

Inspection checks 2a
•Interface faults
• Do all function and method calls have the
correct number of parameters?
• Do formal and actual parameter types
match?
• Are the parameters in the right order?
• If components access shared memory, do
they have the same model of the shared
memory structure?

Inspection checks 2b
•Storage management faults
• If a linked structure is modified, have all
links been correctly reassigned?
• If dynamic storage is used, has space been
allocated correctly?
• Is space explicitly de-allocated after it is no
longer required?
•Exception management faults
• Have all possible error conditions been
taken into account?
Inspection rate
500 statements/hour during overview.
125 source statement/hour during individual
preparation.
90-125 statements/hour can be inspected.
Inspection is therefore an expensive process.
Inspecting 500 lines costs about 40
man/hours effort - about £2800 at UK rates.

Automated static analysis
Static analysers are software tools for source
text processing.
They parse the program text and try to
discover potentially erroneous conditions and
bring these to the attention of the V & V
team.
They are very effective as an aid to
inspections - they are a supplement to but
not a replacement for inspections.

Static analysis checks 1
Fault class and Static analysis check
Data faults
• Variables used before initialisation
• Variables declared but never used
• Variables assigned twice but never used
between assignments
• Possible array bound violations
• Undeclared variables
Control faults
• Unreachable code
• Unconditional branches into loops

Static analysis checks 2
Input/output faults
• Variables output twice with no intervening
assignment
• Interface faults
• Parameter type mismatches
• Parameter number mismatches
• Non-usage of the results of functions
• Uncalled functions and procedures
Storage management faults
• Unassigned pointers
• Pointer arithmetic
Stages of static analysis
Control flow analysis. Checks for loops with
multiple exit or entry points, finds
unreachable code, etc.
Data use analysis. Detects uninitialised
variables, variables written twice without an
intervening assignment, variables which are
declared but never used, etc.
Interface analysis. Checks the consistency of
routine and procedure declarations and their
use

Stages of static analysis
Information flow analysis. Identifies the
dependencies of output variables. Does not
detect anomalies itself but highlights
information for code inspection or review
Path analysis. Identifies paths through the
program and sets out the statements
executed in that path. Again, potentially
useful in the review process
Both these stages generate vast amounts of
information. They must be used with care.
LINT static analysis
138% more lint_ex.c
#include <stdio.h>
printarray (Anarray)
int Anarray;
{ printf(“%d”,Anarray); }
main ()
{
int Anarray[5]; int i; char c;
printarray (Anarray, i, c);
printarray (Anarray) ;
}
139% cc lint_ex.c
140% lint lint_ex.c
lint_ex.c(10): warning: c may be used before set

lint_ex.c(10): warning: i may be used before set
printarray: variable # of args. lint_ex.c(4) :: lint_ex.c(10)
printarray, arg. 1 used inconsistently lint_ex.c(4) :: lint_ex.c(10)
printarray, arg. 1 used inconsistently lint_ex.c(4) :: lint_ex.c(11)
printf returns value which is always ignored
Use of static analysis
Particularly valuable when a language
such as C is used which has weak
typing and hence many errors are
undetected by the compiler,
Less cost-effective for languages like
Java that have strong type checking
and can therefore detect many errors
during compilation.
Verification and formal methods
Formal methods can be used when a

mathematical specification of the system is
produced.
They are the ultimate static verification
technique.
They involve detailed mathematical analysis
of the specification and may develop formal
arguments that a program conforms to its
mathematical specification.

Arguments for formal methods
Producing a mathematical specification
requires a detailed analysis of the
requirements and this is likely to
uncover errors.
They can detect implementation errors
before testing when the program is
analyzed alongside the specification.

Arguments against formal methods
Require specialized notations that cannot be

understood by domain experts.
Very expensive to develop a specification and
even more expensive to show that a program
meets that specification.
It may be possible to reach the same level of
confidence in a program more cheaply using
other V & V techniques.

Cleanroom software development
The name is derived from the 'Cleanroom'
process in semiconductor fabrication. The
philosophy is defect avoidance rather than
defect removal.
This software development process is based
on:
Incremental development;
Formal specification;
Static verification using correctness arguments;
Statistical testing to determine program reliability.

The Cleanroom process

Cleanroom process characteristics
Formal specification using a state transition
model.
Incremental development where the
customer prioritises increments.
Structured programming - limited control and
abstraction constructs are used in the
program.
Static verification using rigorous inspections.
Statistical testing of the system.

Formal specification and inspections
The state based model is a system

specification and the inspection process
checks the program against this model.
The programming approach is defined so that
the correspondence between the model and
the system is clear.
Mathematical arguments (not proofs) are
used to increase confidence in the inspection
process.

Cleanroom process teams
Specification team. Responsible for developing
and maintaining the system specification.
Development team. Responsible for developing and
verifying the software. The software is NOT
executed or even compiled during this process.
Certification team. Responsible for developing
a set of statistical tests to exercise the software
after development. Reliability growth models
used to determine when reliability is acceptable.

Cleanroom process evaluation
The results of using the Cleanroom process have
been very impressive with few discovered faults in
delivered systems.
Independent assessment shows that the
process is no more expensive than other
approaches.
There were fewer errors than in a 'traditional'
However, the process is not widely used. It is not
clear how this approach can be transferred
to an environment with less skilled or less
motivated software engineers.
Key points
Verification and validation are not the same
thing. Verification shows conformance with
specification; validation shows that the
program meets the customer’s needs.
Test plans should be drawn up to guide the
testing process.
Static verification techniques involve
examination and analysis of the program for
error detection.

Key points
Program inspections are very effective in discovering
errors.
Program code in inspections is systematically checked
by a small team to locate software faults.
Static analysis tools can discover program anomalies
which may be an indication of faults in the code.
The Cleanroom development process depends on
incremental development, static verification and
statistical testing.

Chương 3
Kiểm thử phần mềm

Mục tiêu
Thảo luận sự Phân biệt giữa kiểm thử

chấp nhận và kiểm thử phát hiện lỗi
Mô tả các nguyên lý của kiểm thử hệ
thống và kiểm thử thành phần
Mô tả chiến lược tạo ra các trường hợp
kiểm thử hệ thống
Hiểu các đặc trưng cần thiết của công
cụ sử dụng cho tự động hoá kiểm thử

Các chủ đề
Kiểm thử Hệ thống

Kiểm thử thành phần
Thiết kế trường hợp kiểm thử
Tự động hoá kiểm thử

The testing process
Component testing
Testing of individual program components;
Usually the responsibility of the component
developer (except sometimes for critical systems);
Tests are derived from the developer’s experience.
System testing
Testing of groups of components integrated to
create a system or sub-system;
The responsibility of an independent testing team;
Tests are based on a system specification.

Testing phases

Defect testing
The goal of defect testing is to discover

defects in programs
A successful defect test is a test which
causes a program to behave in an
anomalous way
Tests show the presence not the
absence of defects

Testing process goals
Validation testing
To demonstrate to the developer and the system customer
that the software meets its requirements;
A successful test shows that the system operates as
intended.
Defect testing
To discover faults or defects in the software where its
behaviour is incorrect or not in conformance with its
specification;
A successful test is a test that makes the system perform
incorrectly and so exposes a defect in the system.

The software testing process

Testing policies
Only exhaustive testing can show a program
is free from defects. However, exhaustive
testing is impossible,
Testing policies define the approach to be
used in selecting system tests:
All functions accessed through menus should be
tested;
Combinations of functions accessed through the
same menu should be tested;
Where user input is required, all functions must be
tested with correct and incorrect input.
System testing
Involves integrating components to create a
system or sub-system.
May involve testing an increment to be
delivered to the customer.
Two phases:
Integration testing - the test team have access to
the system source code. The system is tested as
components are integrated.
Release testing - the test team test the complete
system to be delivered as a black-box.

Integration testing
Involves building a system from its
components and testing it for problems that
arise from component interactions.
Top-down integration
Develop the skeleton of the system and populate
it with components.
Bottom-up integration
Integrate infrastructure components then add
functional components.
To simplify error localisation, systems should
be incrementally integrated.
Incremental integration testing

Testing approaches
Architectural validation
Top-down integration testing is better at discovering errors
in the system architecture.
System demonstration
Top-down integration testing allows a limited demonstration
at an early stage in the development.
Test implementation
Often easier with bottom-up integration testing.
Test observation
Problems with both approaches. Extra code may be required
to observe tests.

Release testing
The process of testing a release of a system
that will be distributed to customers.
Primary goal is to increase the supplier’s
confidence that the system meets its
requirements.
Release testing is usually black-box or
functional testing
Based on the system specification only;
Testers do not have knowledge of the system
implementation.

Black-box testing
Inputs causing
anomalous
Du lieu kiem thu Ie behaviour
SHe thong
Outputs which reveal

the presence of
Output test results Oe defects

Testing guidelines
Testing guidelines are hints for the testing
team to help them choose tests that will
reveal defects in the system
Choose inputs that force the system to generate
all error messages;
Design inputs that cause buffers to overflow;
Repeat the same input or input series several
times;
Force invalid outputs to be generated;
Force computation results to be too large or too
small.
Testing scenario
A student is studying Vietnam History and has been asked to write a
paper on ‘Vietnamese people's anti-American war during 1954-1975".
To do this, she needs to find sources from a range of libraries, including
the US ones. She logs on to the LIBSYS system and uses the search
facility to discover if she can access original documents from that time.
She discovers sources in various US university libraries and downloads
copies of some of these. However, for one document, she needs to
have confirmation from her university that she is a genuine student and
that use is for non-commercial purposes. The student then uses the
facility in LIBSYS that can request such permission and registers her
request. If granted, the document will be downloaded to the registered
library’s server (e.g. HUT's e-Library) and printed for her. She receives a
message from LIBSYS telling her that she will receive an e-mail
message from the Department of Information-Documentation Services
when the printed document is available for collection.
System tests
1. Test the login mechanism using correct and incorrect logins to check
that valid users are accepted and invalid users are rejected.
2. Test the search facility using different queries against known sources to
check that the search mechanism is actually finding documents.
3. Test the system presentation facility to check that information about
documents is displayed properly.
4. Test the mechanism to request permission for downloading.
5. Test the e-mail response indicating that the downloaded document is
available.

Use cases
Use cases can be a basis for deriving

the tests for a system. They help
identify operations to be tested and
help design the required test cases.
From an associated sequence diagram,
the inputs and outputs to be created for
the tests can be identified.

Collect weather data sequence chart

Performance testing
Part of release testing may involve testing the

emergent properties of a system, such as
performance and reliability.
Performance tests usually involve planning a
series of tests where the load is steadily
increased until the system performance
becomes unacceptable.

Stress testing
Exercises the system beyond its maximum
design load. Stressing the system often
causes defects to come to light.
Stressing the system test failure behaviour..
Systems should not fail catastrophically.
Stress testing checks for unacceptable loss of
service or data.
Stress testing is particularly relevant to
distributed systems that can exhibit severe
degradation as a network becomes
overloaded.
Component testing
Component or unit testing is the process of

testing individual components in isolation.
It is a defect testing process.
Components may be:
Individual functions or methods within an object;
Object classes with several attributes and
methods;
Composite components with defined interfaces
used to access their functionality.

Object class testing
Complete test coverage of a class involves

Testing all operations associated with an object;
Setting and interrogating all object attributes;
Exercising the object in all possible states.
Inheritance makes it more difficult to design
object class tests as the information to be
tested is not localised.

Weather station object interface

Weather station testing
Need to define test cases for reportWeather,

calibrate, test, startup and shutdown.
Using a state model, identify sequences of
state transitions to be tested and the event
sequences to cause these transitions
For example:
Waiting -> Calibrating -> Testing -> Transmitting
-> Waiting

Interface testing
Objectives are to detect faults due to

interface errors or invalid assumptions
about interfaces.
Particularly important for object-
oriented development as objects are
defined by their interfaces.

Interface testing

Interface types
Parameter interfaces
Data passed from one procedure to another.
Shared memory interfaces
Block of memory is shared between procedures or
functions.
Procedural interfaces
Sub-system encapsulates a set of procedures to
be called by other sub-systems.
Message passing interfaces
Sub-systems request services from other sub-
systems.
Interface errors
Interface misuse
A calling component calls another component and
makes an error in its use of its interface e.g.
parameters in the wrong order.
Interface misunderstanding
A calling component embeds assumptions about
the behaviour of the called component which are
incorrect.
Timing errors
The called and the calling component operate at
different speeds and out-of-date information is
accessed.
Interface testing guidelines
Design tests so that parameters to a called procedure
are at the extreme ends of their ranges.
Always test pointer parameters with null pointers.
Design tests which cause the component to fail.
Use stress testing in message passing systems.
In shared memory systems, vary the order in which
components are activated.

Test case design
Involves designing the test cases (inputs and
outputs) used to test the system.
The goal of test case design is to create a set
of tests that are effective in validation and
defect testing.
Design approaches:
Requirements-based testing;
Partition testing;
Structural testing.

Requirements based testing
A general principle of requirements

engineering is that requirements should
be testable.
Requirements-based testing is a
validation testing technique where you
consider each requirement and derive a
set of tests for that requirement.

LIBSYS requirements
• The user shall be able to search either all of
the initial set of databases or select a subset
from it.
• The system shall provide appropriate viewers
for the user to read documents in the
document store.
• Every order shall be allocated a unique
identifier (ORDER_ID) that the user shall be
able to copy to the account’s permanent
storage area.
LIBSYS tests
• Initiate user search for searches for items that are known to
be present and known not to be present, where the set of
databases includes 1 database.
• Initiate user searches for items that are known to be present
and known not to be present, where the set of databases
includes 2 databases
• Initiate user searches for items that are known to be present
and known not to be present where the set of databases
includes more than 2 databases.
• Select one database from the set of databases and initiate
user searches for items that are known to be present and
known not to be present.
• Select more than one database from the set of databases
and initiate searches for items that are known to be present
and known not to be present.

Partition testing
Input data and output results often fall into

different classes where all members of a class
are related.
Each of these classes is an equivalence
partition or domain where the program
behaves in an equivalent way for each class
member.
Test cases should be chosen from each
partition.

Equivalence partitioning

Equivalence partitions

Search routine specification
procedure Search (Key : ELEM ; T: SEQ of ELEM;
Found : in out BOOLEAN; L: in out ELEM_INDEX) ;
Pre-condition
-- the sequence has at least one element
T’FIRST <= T’LAST
Post-condition
-- the element is found and is referenced by L
( Found and T (L) = Key)
or
-- the element is not in the array
( not Found and
not (exists i, T’FIRST >= i <= T’LAST, T (i) = Key ))

Search routine - input partitions
Inputs which conform to the pre-conditions.

Inputs where a pre-condition does not hold.
Inputs where the key element is a member of
the array.
Inputs where the key element is not a
member of the array.

Testing guidelines (sequences)
Test software with sequences which

have only a single value.
Use sequences of different sizes in
different tests.
Derive tests so that the first, middle and
last elements of the sequence are
accessed.
Test with sequences of zero length.

Search routine - input partitions
Sequence Element
Single value In sequence
Single value Not in sequence
More than 1 value First element in sequence
More than 1 value Last element in sequence
More than 1 value Middle element in sequence
More than 1 value Not in sequence
Input sequence (T) Key (Key) Output (Found, L)

17 17 true, 1
17 0 false, ??
17, 29, 21, 23 17 true, 1
41, 18, 9, 31, 30, 16, 45 45 true, 7
17, 18, 21, 23, 29, 41, 38 23 true, 4
21, 23, 29, 33, 38 25 false, ??

Structural testing
Sometime called white-box testing.

Derivation of test cases according to
program structure. Knowledge of the
program is used to identify additional
test cases.
Objective is to exercise all program
statements (not all path combinations).

Structural testing

Binary search - equiv. partitions
Pre-conditions satisfied, key element in array.
Pre-conditions satisfied, key element not in
array.
Pre-conditions unsatisfied, key element in
array.
Pre-conditions unsatisfied, key element not in
array.
Input array has a single value.
Input array has an even number of values.
Input array has an odd number of values.

Binary search equiv. partitions

Binary search - test cases
Input array (T) Key (Key) Output (Found, L)

17 17 true, 1
17 0 false, ??
17, 21, 23, 29 17 true, 1
9, 16, 18, 30, 31, 41, 45 45 true, 7
17, 18, 21, 23, 29, 38, 41 23 true, 4
17, 18, 21, 23, 29, 33, 38 21 true, 3
12, 18, 21, 23, 32 23 true, 4
21, 23, 29, 33, 38 25 false, ??

Path testing
The objective of path testing is to ensure that
the set of test cases is such that each path
through the program is executed at least
once.
The starting point for path testing is a
program flow graph that shows nodes
representing program decisions and arcs
representing the flow of control.
Statements with conditions are therefore
nodes in the flow graph.

Binary search flow graph

Independent paths
1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 14
1, 2, 3, 4, 5, 14
1, 2, 3, 4, 5, 6, 7, 11, 12, 5, …
1, 2, 3, 4, 6, 7, 2, 11, 13, 5, …
Test cases should be derived so that all of
these paths are executed
A dynamic program analyser may be used to
check that paths have been executed

Test automation
Testing is an expensive process phase. Testing
workbenches provide a range of tools to reduce the
time required and total testing costs.
Systems such as Junit support the automatic
execution of tests.
Most testing workbenches are open systems because
testing needs are organisation-specific.
They are sometimes difficult to integrate with closed
design and analysis workbenches.

A testing workbench

Testing workbench adaptation
Scripts may be developed for user

interface simulators and patterns for
test data generators.
Test outputs may have to be prepared
manually for comparison.
Special-purpose file comparators may
be developed.

Key points
Testing can show the presence of faults in a system;
it cannot prove there are no remaining faults.
Component developers are responsible for
component testing; system testing is the
responsibility of a separate team.
Integration testing is testing increments of the
system; release testing involves testing a system to
be released to a customer.
Use experience and guidelines to design test cases in
defect testing.

Key points
Interface testing is designed to discover
defects in the interfaces of composite
components.
Equivalence partitioning is a way of
discovering test cases - all cases in a partition
should behave in the same way.
Structural analysis relies on analysing a
program and deriving tests from this analysis.
Test automation reduces testing costs by
supporting the test process with a range of
software tools.
Chương 4
Phương pháp kiểm thử
Kiểm thử và Đảm bảo Chất lượng Phần mềm 1

Nội dung chính
Data-based black box testing
classification
methods
boundary-value testing
equivalence partitioning
decision tables
error guessing, random testing
Example
Structure-based white box testing
Data/structure-based white box testing
Structure-based black box testing

1. What is tested?
2. How to test?
c. Organization (documentation, implementation, execution:platform, interactive, batch)

2d
2c
1a
1b
2b 2a 1c

Test purpose
Software development phase:
test
system
specification test
design test
implementation unit
code test

Test method dimensions
data-based
structure-based

typical errors
efficiency
...

Classifying data-based black box testing
all
requirements: some logical/mathematical input/output relation
purpose: unit testing, functional mistakes
method: black box, data-based
assumption: both single-/multiple-fault assumption
boundary value testing
method: typical errors (boundaries of domains)
equivalence partitioning/decision tables
method: efficiency

Running example: mortgage
input: gender [m,f], age [18-55], monthly salary [0-10000]
output: mortgage total for one person
requirements: category male female
mortgage young (18-35 years) 75 (18-30 years) 70
= salary*factor middle (36-45 years) 55 (31-40 years) 50
program: old (46-55 years) 30 (41-50 years) 35
Mortgage (male:Boolean, age:Integer, salary:Integer): Integer

begin
if (male) then
return ((18<=age<35)?(75*salary):(31<=age<40)?(55*salary):(30*salary))
else /* female */
return ((18<=age<30)?(75*salary):(31<=age<40)?(50*salary):(35*salary))
fi;
end
Testing the mortgage program
each test case = input combination +
expected output value
complete testing requires all input
combinations: infeasible
select some combinations
assumptions:
single/multiple fault
input type: quantity, enumerable, bounded
typical errors
efficiency: redundant input combinations
Boundary value testing
detect errors to do with input domain bounds

for integer input x with domain [a,b], test
input values around a and b
# tests = for n inputs, 4n+1 input
combinations y
d
assumes:
independent quantity inputs
single-fault assumption c
a b x
Robustness boundary value
also test values outside the domain
# tests =
for n input variables, 6n+1 input combinations
a test: <x_nom, y_max+1, expected output>
y
maximal value Å d
nominal value Å
minimal value Å c
a b x
Worst-case boundary value
multiple-fault assumption
# tests =
for n input variables, 5n input combinations
y
d
a b x
Worst-case robustness bv
multiple-fault assumption, tests also outside

the domain
# tests =
for n input variables, 7n input combinations
y
d
a b x
Subsume relations
AÆB Method A subsumes method B if A tests
at least what B tests (possibly more)
Which subsume relations for boundary
value variants? y
(Assume one fixed nominal d
value for each input)
boundary value
worst-case robustness c
worst-case robustness
a b x
Boundary value testing Mortgage
Observations
Input gender is not a quantity
Boundaries are not fixed:

Inputs gender and age are not independent!
Variant ‘boundary value’ may not detect error
(if nominal value for gender is ‘male’)
Many more errors in the program than we
can expect any boundary value variant to
detect

Boundary value testing summary
Coverage: not good

#tests: moderate to very many
Usage: straightforward, easy to implement
When to use:
independent inputs
enumerable quantities, e.g. age
(obviously) when suspecting boundary errors
See literature:
Zhu
Jorgensen
detect errors to do with computational mistakes
for integer input x with domain [a,b] partitioned in s_x
subdomains, test input values from each subdomain
assumes:
independent variables y
redundancy in subdomain g
weak normal variant:
• assumes:
f
– independent variables
– single-fault assumption
• #tests = max s_x for any input x e
a b c d x
strong normal variant:
assumes:
#tests =
Π s_x for each input x
y
g
a b c d x
weak robust variant:
tests outside domain
assumes:
independent variables
single-fault assumption
#tests = y
max s_x for any input x g
+ 2*(#inputs)
a b c d x
strong robust variant:
tests outside domain
assumes:
#tests = y
Π (s_x+2) for each g
input x
f
a b c d x
Subsume relations
Which subsume relations for equivalence

partition variants?
(Assume same value selected for each input, for
each subdomain) y
g
weak normal
f
strong normal weak robust
strong robust e
a b c d xx
Equivalence partitioning summary
Coverage: moderate
#tests: small to moderate
Usage: some study of requirements needed
When to use:
independent inputs
enumerable quantities
when suspecting computational errors
when redundancy can be assumed
may easily be combined with boundary value

Decision table testing
also known as cause-effect graphing
detect errors to do with computational mistakes
input are partitioned through conditions
assumes: dependent variables
#tests = depends on granularity of conditions/actions
conditions/actions rule 1 rule 2, 3 rule 4 rule 5-8 - : don’t care

c1: x>=0? y n y - _ : fixed value
c2: x<=y? y - n -
c3: z even? _ n n y
a: correct answer type 1 x x
a2: correct answer type 2 x
a3: faulty inputs x

Decision table testing Mortgage
conditions/actions rule 1 rule 2
c1: male?
c2: young?
c3: middle?
c4: old?
c5: < young?
c6: > old?
c7: 0 <= salary <= 10000?1
a1: wrong input
a2: 75*age
a3: 70*age
a4: 55*age
a5: 50*age
a6: 30*age
a7: 35*age
Decision table testing summary
Coverage: good
#tests: moderate to large
Usage: much study of requirement needed
When to use:
dependent inputs
when suspecting computational errors
may be combined with boundary testing (tricky)
See literature:
Zhu
Jorgensen
Error guessing & Random Testing
Error guessing
No method involved, just experiment and see if
something goes wrong
Some people have a knack for exposing failures
(e.g. young children!)
Random testing
Select input combinations randomly
Both can be very effective, but should be used in

addition to methodic testing

Guidelines data-based black box testing
Inputs dependent on each other?

Æ decision table
Suspect computational errors?
Æ decision table or equivalence partition (see 1st bullet)
Suspect boundary errors?
Æ boundary value
Combinations can and
should be constructed
Single-fault assumption? sensibly!
Æ strong/worst-case variant
Suspect errors outside the domain?
Æ robust variant
Want to know what a non-average user does?
Æ error guessing, random testing
Outline
classification
example
adequacy criteria
generate tests?
guidelines

2d
2c
1a
1b
2b 2a 1c

Test purpose
test
system
specification test
design test
implementation unit
code test

data-based
structure-based

typical errors
efficiency
coverage

Classifying structure-based white box testing
all criteria
requirements: some logical/mathematical
input/output relation
software characteristics: available, imperative
programming language
method: white box, structure-based, coverage,
efficiency
assumption: both single and multiple-fault
assumption

Running example: triangle
input: a,b,c: [1-..]
1. Triangle (a, b, c: Integer): String
output: equilateral/isosceles/ 2. IsATriangle: Boolean
scalene/not a triangle 3. begin
4. if (a<=b+c or b<=a+c or c<=b+c)
requirements: mathematically 5. then IsATriangle := true
correct 6.
rs!
else IsATriangle := false fi;
testing: input combination, 7. if (IsATriangle)
rro
expected output 8.
?) e
then if (a=b and b=c)
structure: each test is a path

9.
10.
(6
then return “Equilateral”
5
else if (a≠b and a≠c and b≠c)
then return “Scalene”
coverage: when ... 11.
12. else return “Equilateral” fi; fi;
paths/branches/... executed 13. else return “Not a triangle” fi;
14. end
isosceles equilateral scalene

Testing the triangle program
4. 1. Triangle (a, b, c: Integer): String
2. IsATriangle: Boolean
program 3. begin
5. 6.
graph 4. if (a<=b+c or b<=a+c or c<=b+c)
5. then IsATriangle := true
7. 6. else IsATriangle := false fi;
7. if (IsATriangle)
8. 13. 8. then if (a=b and b=c)
9. then return “Equilateral”
10. else if (a≠b and a≠c and b≠c)
9. 10. 11. then return “Scalene”
12. else return “Equilateral” fi; fi;
13. else return “Not a triangle” fi;
14. end
11. 12.
14.
4. (a<=b+c or b<=a+c or c<=b+c)? each test
represents a
yes no
5. IsATriangle := true 6. IsATriangle := false
path: a=b=c=1
7. IsATriangle?
yes no
8. (a=b and b=c)? 13. “Not a triangle”

yes no
9. “Equilateral” 10. (a≠b and a≠c and b≠c)?
yes no
11. “Scalene” 12. “Equilateral”
14. end
yes no
represents a
5. IsATriangle := true 6. IsATriangle := false
path: a=b=c=1:
7. IsATriangle? “Equilateral”
yes no coverage:
8. (a=b and b=c)? 13. “Not a triangle” nodes
yes no edges
9. “Equilateral” 10. (a≠b and a≠c and b≠c)? ...
yes no all paths feasible?
11. “Scalene” 12. “Equilateral” no! (this is in general
undecidable)
14. end
Program graphs & testing
infeasible paths:
combinations that cannot occur
(dependent nodes)
dead code
# paths:
possibly infinite (loops!)
coverage criteria:
...

Program graphs & testing
coverage criteria:
statement: each node is executed
branch/decision: each edge is executed
(multiple) condition:
let p_1, p_2, ... be the smallest parts in condition c
(atomic predicates)
condition coverage: each condition is executed so often
single fault Æ that each atomic predicate p_i has evaluated to both
truth values
multiple fault Æ multiple condition coverage: each condition is executed
so often that the atomic predicates have evaluated to all
possible truth value combinations
all paths
Subsume relations
A Æ B: Criterion A subsumes criterion B if

A tests at least what B tests
statement coverage
branch coverage condition coverage
branch+condition coverage
all paths multiple condition coverage
Note: multiple condition coverage tests more, but may show fewer failures
than branch+condition coverage! (Frankl & Weyuker, Provable
Improvements on Branch Testing, IEEE Trans. Softw. Eng.:19(10), 1993)
yes no represents a path:
5. IsATriangle := true 6. IsATriangle := false a=b=c=1: “Equilateral”
coverage:
7. IsATriangle?
no edges
yes
8. (a=b and b=c)? 13. “Not a triangle”
yes no
Exercise (5 min):
9. “Equilateral” 10. (a≠b and a≠c and b≠c)? Construct input combinations,
yes no such that each edge is executed.
11. “Scalene” 12. “Equilateral” Compare program output values
to required output values.
Which path is not feasible?
14. end
Observations
How to construct tests & get everything covered?
Guidelines:
Start with test generation based on black box method
Measure white box coverage for black box test set
Use tools!
Which coverage criterion?
Æ How many errors may remain?
Æ A bit stronger than branch coverage is very effective, e.g.
branch + condition coverage
Æ Industrial practice:
varies from statement coverage (100%?) to multiple condition
coverage (?%)

What we haven’t discussed
Loops:
once and zero times
maximal + minimal number of times
...
Alternative program graphs:
LCSAJ blocks per node
Stepwise testing: graph condensation
Discrepancies between graph and requirements:
feasible paths not required
required paths not present/feasible

Outline (Complementary)


Data/structure-based criteria
variable x of the program:

define occurrence: x gets a value (assignment)
use occurrence: the value of x is used
computational (in the value for assignment)
predicate (in a condition)
criteria:
for each definition occurrence of x at node n, for
each/some use occurrence that can be reached
from n, each/some path is executed
all definitions criterion


criteria:
all uses criterion


criteria:
all definition-use paths criterion

Structure-based white box testing summary
Coverage: poor to very good

but even for weak criteria 100% not always possible!
#tests: few to very many
Usage: hard to generate tests, easier to measure coverage
When to use:
in combination with black box generation methods
elaborate methods only for vital parts of software
if tools are available

Outline


1. What is tested?
2. How to test?

2d
2c
1a
1b
2b 2a 1c

Test purpose

test
system
specification test
design test
implementation unit
code test

data-based
structure-based

typical errors
efficiency
coverage

Classifying structure-based black box testing
all criteria
requirements: behaviour, defined in FSM
software characteristics: can be modelled as
an FSM
method: black box, structure-based,
coverage, efficiency
assumption: ...

Running example: musical toy
1. PlayMusic
input: 2. inputs pushRhythm, pushSong,
pushQuit;
event: a button is 3.
4.
outputs startRhythm, startSong, Quit;
state Rhythm, Song: Boolean;
pushed 5. begin Quit
6. while (true) do
output: 7. if (pushQuit) then
8. Rhythm
Rhythm, Song := false,Song
false;
event: sound changes 9. if (Rhythm and Song) then output
Quit fi
requirements: 10. elsif (pushRhythm) then
11. if (Rhythm) then output Rhythm fi;
FSM (behaviour) 12. Rhythm := true
13. elsif (pushSong) then
testing: ... 14. if (Song) then output Song fi;
15. Song := true fi
16. od;
17. end
FSM of musical toy
requirements
1. PlayMusic
PushQuit?/- 2. inputs pushRhythm, pushSong,
Off pushQuit;
PushRhythm? PushSong? 3. outputs startRhythm, startSong, Quit;
/StartRhythm! /StartSong! 4. state Rhythm, Song: Boolean;
5. begin
6. while (true) do
PushQuit? 7. if (pushQuit) then
/Quit! 8. Rhythm, Song := false, false;
9. if (Rhythm and Song) then output
Rhythm Song Quit fi
PushRhythm?/- PushSong?/- 10. elsif (pushRhythm) then
11. if (Rhythm) then output Rhythm fi;
PushSong? PushRhythm? 12. Rhythm := true
/StartSong! /StartRhythm! 13. elsif (pushSong) then
14. if (Song) then output Song fi;
15. Song := true fi
Both 16. od;
PushRhythm?/- PushSong?/- 17. end

What is an FSM?
PushQuit?/- FSM (Finite State Machine), or
PushRhythm?
Off Mealy machine: <S,I,O,δ,λ>
PushSong?
/StartRhythm! /StartSong! states S (usually has a start
state)
alphabets: inputs I, outputs O
PushQuit?
transfer function: δ:S×I→S
/Quit!
output function λ:S×I→O
Rhythm special no-output label: ‘-’
Song
PushRhythm?/- assumptions!
PushSong?/-
suitable for:
PushSong? PushRhythm? precise, abstract models
/StartSong! /StartRhythm! interfaces, protocols, embedded
software, ...
Both formal test generation
PushRhythm?/- PushSong?/-

Assumptions for FSM modelling
PushQuit?/- FSM must be
PushRhythm?
Off deterministic + input-enabled:
PushSong?
/StartRhythm! /StartSong! δ and λ are proper functions
finite:
PushQuit? I,O,S all finite
/Quit! strongly connected:
each state can be reached from
Rhythm Song any state
PushRhythm?/- PushSong?/- reduced:
non-equal states have different
PushSong? PushRhythm? observable behaviour
/StartSong! /StartRhythm!
Both
PushRhythm?/- PushSong?/-

Two FSM models:
implementation?
Specification Implementation
FSM model of
FSM model of software to be
tested
requirements
satisfies restrictions except
satisfies restrictions ‘reduced’
previous slide not really needed, but:
is used for test justification for FSM-based
testing
generation
#states must be estimated

Testing Impl versus Spec:
c?/- c?/-
s0 t0
b?/e!
? a?/e!
b?/e!
a?/e! b?/e!
a?/- b?/e!
t1 c?/e!
a?/- s1 c?/- t2
c?/- c?/- a?/-
test = sequence of inputs: (1) a? b? a? (2) b? c? a?

expected outcome = sequence of outputs
expected: (1) e! e! e! (2) e! - -
actually: (1) e! e! e! (2) e! e! e!
¾ Impl implements Spec if paths(Impl) = paths(Spec)

Testing Impl versus Spec:
c?/- c?/-
s0 t0
b?/e!
? a?/e!
b?/e!
a?/e! b?/e!
a?/- b?/e!
t1 c?/e!
a?/- s1 c?/- t2
c?/- c?/- a?/-
¾ Impl implements Spec if paths(Impl) = paths(Spec)

¾ possible errors:
1. wrong output (computational mistake)
2. extra/missing states (extra/missing features)
3. transition to wrong state (computational mistake)
¾ which paths to test??
Test generation from Spec c?/-
s0
b?/e!
a?/e! b?/e!
test each transition: <s,i?>
1. go to state s from arbitrary state a?/- s1
c?/-
(synchronizing sequence + transferring sequence)
2. apply input i?
(test transition)
3. check output λ(s,i)
(test transition)
4. check if in state δ(s,i)
(UIO-sequence/distinguishing sequence/W-set/...)
test fails iff something wrong in step 3. or 4.
s0
b?/e!
a?/e! b?/e!
c?/-
2. apply input i?
(test transition) a? b?
(test transition)
s0
b?/e!
a?/e! b?/e!
c?/-
2. apply input i?
(test transition) for s0: (empty)
3. check output λ(s,i) for s1: a?
(test transition)
s0
b?/e!
a?/e! b?/e!
c?/-
2. apply input i?
(test transition) e.g. for <s1,b?>: b?
(test transition)
s0
b?/e!
a?/e! b?/e!
c?/-
2. apply input i?
(test transition) e.g. for <s1,b?>: e!
(test transition)
s0
b?/e!
a?/e! b?/e!
c?/-
2. apply input i?
(test transition) a?
3. check output λ(s,i) output for state s1: -
(test transition)
s0
b?/e!
a?/e! b?/e!
The total set of tests:
a?/- s1
c?/-
transition synchronising transferring expected state

sequence sequence output verification
<s0,a>
<s0,b>
...
Homework: to be completed!

What we haven’t (yet) discussed
Why is it correct?
Non-deterministic systems
Data parameters
Literature: Lee and Yannakakis, “Principles and
Methods of Testing Finite State machines—A survey”,
Proc. of the IEEE:84(8), 1996.
Separate input or output transitions
Other implementation criteria (subsume rel)
Multi-process/distributed testing
Summary, guidelines

s0
b?/e!
a?/e! b?/e!
c?/-
2. apply input i?
(test transition)
(test transition)
Test generation assumptions
FSM = <S,I,O,δ,λ>
Spec FSM:
is deterministic, complete, reduced
has a synchronizing sequence
has an UIO sequence for each state
Impl FSM :
is deterministic, complete
has no more states than Spec
Generalised δ,λ functions:
δ(s, i_1 i_2 ... i_n) = t (target state of a sequence of inputs)
λ(s, i_1 i_2 ... i_n) = o_1 o_2 ... o_n (sequence of outputs)

Synchronising sequence s0
c?/-
b?/e!
a?/e! b?/e!
sequence of inputs
brings FSM to one designated state a?/- s1
c?/-
determine through successor graph:
node: set of states S’={s,t,...} (initially: S)
edge: from S’ to S’’ with i/o where b?/e! c?/-
S’’ = { δ(s,i) | s∈S’ and λ(s,i)=o }
{s0,s1}
synchronising sequence σ
all paths for σ from S end in same state {s}
σ does not always exist!! a?/e! a?/-

often there is a reliable reset
if exists, easy to determine in successor graph {s1}

Successor graph for musical toy
{Off,Rhythm,Song,Both}
PR?/R!
PS?/-
PR?/- PQ?/Q! PS?/S!
PQ?/-
{Rhythm,Both} {Off} {Song,Both}
PQ?/Q! PQ?/Q!
PR?/-
PS?/-
PS?/- PR?/-
PR?/R!
PS?/S! {Both}
synchronising sequence: PushQuit?

Another FSM + successor
graph
FSM: a?/0!
successor graph: {s1,s2,s3}
s1
b?/1! b?/0!
a?/0! a?/1! b?/0! b?/1!
{s1,s3} {s2} {s1} {s2,s3}

s2 s3
b?/1! b?/1! b?/0!
a?/0!
a?/1! a?/0!
b?/0! a?/1!
synchronising sequence? no! a?/0! b?/1!
(see Lee & Yannakakis for proof construction) {s3}
Relax: homing vs. synchronising
synchronising sequence: c?/-

s0
ignore outputs
always brings FSM to same end state
a?/e! b?/e!
b?/e!
does not always exist

s1
homing sequence: a?/- c?/-
check outputs
outputs determine which end state
always exists
extra test effort:
check outputs, determine end state
transfer to initial state (or directly to state-under-test)
UIO sequence s0
c?/-
b?/e!
a?/e! b?/e!
Unique Input/Output sequence determines
which first state a?/- s1
c?/-
inputs (+ expected outputs per state)
determine through successor graph with
initial states:
node: set of states S’={s,t,...} (initial: S) b?/e! c?/-
path from S’ to S’’ with i1/o1 i2/02 ...
{s0,s1}
S’’={ s∈S’ | λ(s,i1 i2...) = o1 o2... }
UIO sequence for s: path to {s} a?/e! a?/-
does not always exist!!
{s0} {s1}
For musical toy {current states}
{possible initial states}
{O,R,S,B}
{O,R,S,B}
PR?/R!
PS?/-
PQ?/- PS?/S!
PR?/- PQ?/Q!
{R,B} {R,B} {O}
{O,S} {O} {S,B} {S,B}
{R,B} {O} {O,R} {S,B}
{R,S,B}
PS?/S! PS?/- PS?/S! PS?/-

{B} {B} {B} {B}
{O} {S} {R} {B}
UIO sequence for any state: PushRhythm? PushSong?

Yet another FSM
FSM: a?/0! {s1,s2,s3}

successor graph
{s1,s2,s3}
with initial states:
s1
a?/0! b?/0!
a?/0! a?/1! b?/0! b?/1!
{s1} {s3} {s3} {s2}
s2 s3 {s1,s2} {s3} {s1} {s2,s3}
b?/1!
b?/1! a?/1! a?/0! b?/0! b?/1!
a?/0!
UIO sequences? {s3}
{s1}
{s1,s2}
s1: b?/0! s3: a?/1! {s2,s3}
s2: no UIO sequence!
Relax: UIO vs. W-set s0
c?/-
b?/e!
a?/e! b?/e!
UIO:
1 sequence of inputs/outputs per state s1
a?/-
does not always exist c?/-
W-set:
set W(s) of input (+ output) sequences for state s
for each distinct pair s,s’: W(s) contains a sequence that
distinguishes s from s’
always exists
extra test effort:
large number of sequences
to establish state verification for s, repeatedly go back to s to
test all sequences in W(s)

Other relaxations (1)
Spec/Impl FSM: in(y)? [NOT a]

s t
has data parameters (EFSM): /e! [x:=y]
only test control structure (mediocre coverage)
compute complete FSM (huge effort, very many tests)
combine with data-based black box methods
is non-deterministic: a?/e! t
randomize testing algorithm
s
a?/e! u
is incomplete:
test only specified parts s a?/0! t
b?/1! u

Other relaxations (2)
Impl FSM :
has n more states than Spec:
for complete coverage: extra test effort factor |I|n
No FSM model possible:
Labelled Transition System (LTS) model
input/output conformance (ioco) method
see KUN/UTwente courses on testing:
http://www.cs.kun.nl/~tretmans/testtechnieken/
a? t b?
u
s b?
f!
Structure-based black box testing summary
Coverage: complete (formal) to rather good

(relaxations)
#tests: moderate to very many
Usage: hard to model FSM and implement tests
When to use:
when requirements = (non-trivial) behaviour
when complete coverage is essential
unit/system/acceptance testing
tools for test generation (commercially) available

Chapter 5
Kiểm thử tích hơp

Nội dung chính
Kiểm thử tích hơp

Phân loại kiểm thử
Cách tiếp cận tích hợp
Ngữ nghĩa của hệ sinh kiểm thử và tính
phủ (coverage)
Thí dụ

Mục đích
Giai đoạn phát triển phần mềm:

Các yêu cầu Kiểm thử chấp nhận
Đặc tả Kiểm thử hệ thống
Thiết kế chi tiết Kiểm thử tích hợp
Lập trình Kiểm thử đơn vị

Phân loại kiểm thử tích hợp
Các yêu cầu: “Giao diện giữa các đơn vị CT

đươc thực hiện đúng đắn “
Đặc trưng phần mềm : Truy cập phần mềm tại
mức mô đun, không phải tới chính các đơn vị
chương trình
Mục đích: Kiểm thử tích hơp, giao diện/các lỗi
phối hợp
Giả định: Các đơn vị đã được kiểm thử trước đó

Sự tích hơp phần mềm được kiểm
thử
Sự hơp thành các đơn vị chương trình Main

Các đơn vị chương trình đã được kiểm
thử trước
Giao diện phải được kiểm thử D
Không có liên quan đến hành vi mức hệ
thống ! A B C
Lời gọi thủ tục : Cấu tạo đồ thị gọi
Để đảm bảo kiểm thử thực tế, mọi đơn
vị CT cần bao gồm biểu diễn về
E F G
Các con của nó có mặt trong đồ thị lời gọi
Ít ra có một cha trong đồ thị gọi

Tiếp cận tích hợp: big-bang
Đặt các đơn vị cùng với nhau khi Main
chúng cùng một phần (1 session

Dựa vào các kết quả kiểm thử của D
đơn vị CT
A B C
(relatively) Có ít các kiểm thử
Rất khó cô lập các lỗi
E F G

Tiếp cận tích hợp: top-down
start with top unit: Main Main
replace each unit called by Main with a
stub:
dummy software s
reacts as original unit would s s s
easy to program
1st session: test interfaces for Main
E F G

Tiếp cận tích hợp: top-down
second session: unit A Main
...
many stubs required s
many test sessions required A s s
1 session per unit
easy to isolate errors
s F G

Tiếp cận tích hợp: bottom-up
start with lowest-level unit: E Main
replace unit calling E with a driver:
dummy software
makes calls as original unit would D
harder to program d B C
many drivers required
usually fewer than #stubs for top-down
many test sessions required
E F G
1 session per unit
easy to isolate errors

Tiếp cận tích hợp: sandwich
mixture of top-down and bottom- Main
up
some stubs, some drivers required
D
test interfaces between portions
A B C
already tested
moderate #test sessions required
errors harder to isolate E F G
top-down bottom-up

Tiếp cận tích hợp: pair-wise
per session, test a pair of units: d

C/D
some drivers, some stubs required D
fewer than bottom-up or top-down
A B C
many test sessions required
1 session per pair
easy to isolate errors E s s

What is an integration test?
input values + driver/stub/unit set-up Main
in the composed program graph: unit-
paths with call/return markers

m1 <Main calls C> c1 <C calls F> f1 D
<F returns C> c2 ...
A B C
Main C F
1 1 2
m1 c1
2 5 3
2 E F G
5 3 f1
3 4
4 c2
6 m2 4 6

Sinh và phủ các kiểm thử tích hợp
Coverage in the call-graph: Main

minimally/usually: all edges (each call-return
interface is tested at least once)
D
maximally: all feasible paths
A B C
Test generation requires
(white box) the composed program graph
(grey box) composition of units’ call/return models, E F G
e.g. FSMs
Main 1 C F
1 2
2 5 3
2
5 3
3 4
4
6 4 6

Thực hiện ví dụ : bank machine
card
Bank Balance Machine
1 2 3
WELCOME
4 5 6
insert your card 7 8 9
0 Cancel

Đặc tả mức hệ thống FSM
InvalidCardIn?/CardOut! idle
ValidCardIn?/Screen2!
InvalidPIN?/Screen4!
process PIN
ValidPIN?/Screen5!
Cancel?/Screen7,CardOut!
process balance
Savings?/Screen6,CardOut! Checking?/Screen6,CardOut!
end session
CardRemoved?/Screen1!

Thông báo màn hình của máy
Ngân hàng
Screen 1 Screen 2 Screen 3 Screen 4
Enter your PIN: Invalid PIN!
Bank Balance Machine (Personal
Please enter your Identification failed!
Indentification
WELCOME Number) PIN again:
Your card is retained.
____ ____ Contact your bank.
Insert your card. (press Cancel to clear)
(press Cancel to clear)
Screen 5 Screen 6 Screen 7
Select your account: Your balance is

being printed on
Operation cancelled!
a receipt.
checkings ½ Please take your card.
savings ½ Please take your
Thank you.
receipt and card.
(press Cancel to abort) Thank you.

Phân rã chức năng ...
Bank Machine
Central Bank Card Balance Screen Receipts

Communication Management Management Management
Card Card Type PIN Validation

Input/Output Validation
... Không giúp đỡ gì vì đồ thị-gọi nhìn khác

hoàn toàn !
Đồ thị - gọi máy ngân hàng
Main
CardIn
ManageBalance
ValidatePIN
ValidateCard
CardOut
GetPIN
PrintReceipt
ContactBank
PrintScreen

Chapter 6
Phần mềm khả hiện
(More Reliable Software)
Một cách nhìn tổng quan về Nhanh
hơn và Rẻ hơn

Most Important Software Problem
1. Customers demand (in order):
A. More reliable software
B. Faster
C. Cheaper
2. Your success in meeting demands affects
market share, profitability, your career
3. Demands conflict, causing risk and
overwhelming pressure
Introduction

Software Reliability Engineering –
Developed to Address the Problem
1. It differs from other approaches by being primarily
quantitative.
2. You add and integrate software reliability
engineering (SRE) with other good processes and
practices; you do not replace them.
3. With SRE you control the development process, it
doesn’t control you:
A. Development process is not externally imposed.
B. You use quantitative information to choose the most
cost-effective software reliability strategies for your
situation.
Introduction

Outline
1. Nature of software reliability engineering

(SRE)
2. SRE process (with illustration)
Introduction
Reliability and Availability Definitions
1. Reliability: the probability that a system or a
capability of a system will continue to function
without failure for a specified period in a specified
environment. The period may be specified in natural
or time units.
A. Natural unit: unit other than time related to amount of
processing performed by software-based product, such as
runs, pages of output, transactions, telephone calls, jobs,
semiconductor wafers, queries, or API calls
B. Failure intensity (FI): failures per natural or time unit, an
alternative way of expressing reliability
2. Availability: the average (over time) probability that a
system or a capability of a system is currently
functional in a specified environment
Nature of SRE
How Does SRE Work?
1. Increase effective
Increase in resources
Effective A. Quantitatively characterize
Resources expected use
B. Focus resources on most
used and most critical
Original functions
Resources C. Maximize test effectiveness
by making test highly
representative of field
Nature of SRE – How Does SRE Work?

How Does SRE Work?
2. Apply resources to
maximize customer
Rel / Avail value
A. Set quantitative
objectives for major
quality
Time Price characteristics
(reliability and/or
availability, delivery
time, price)

How Does SRE Work?
Added Customer B. Choose software

Value - reliability strategies to
Matching Needs meet objectives
C. Track reliability in
Added Customer
Value - Focus system test against
objective as one of
Original
Original Customer the release criteria
Customer
ValueValue

SRE - A Proven, Standard, Widespread
Best Practice
1. Proven practice
A. Example: AT&T International Definity PBX
[6, pp 167-8]
a. Reduced customer-reported problems by factor of
10
b. Reduced system test interval by factor of 2
c. Reduced total development time by 30%
d. No serious service outages in 2 years of
deployment
Nature of SRE - Proven, Standard, Widespread Best Practice

SRE - A Proven, Standard,
Widespread Best Practice
B. AT&T Best Current Practice since 5/91
(based on widespread practice, documented
strong benefit/cost ratio, probing review) [6,
pp 219-254]
2. Standard practice
A. McGraw-Hill handbook [6]
B. AIAA standard since 1993
C. IEEE standards under development

SRE - A Proven, Standard,
Widespread Best Practice
3. Widespread practice
A. Users include Alcatel, AT&T, Bellcore, CNES
(France), ENEA (Italy), Ericsson Telecom,
Hewlett Packard, Hitachi, IBM, NASA’s Jet
Propulsion Laboratory, Lockheed-Martin,
Lucent Technologies, Microsoft, Mitre, Nortel,
Saab Military Aircraft, Tandem Computers, the
US Air Force, and the US Marine Corps.
B. Over 50 published articles by users of
software reliability engineering, continuing to
grow ([1]; [3], pp. 371 - 374)

SRE Is Widely Applicable
1. Technically speaking, you can apply SRE to any
software-based product, beginning at start of any
release cycle.
2. Economically speaking, the complete SRE process
may be impractical for small components (involving
perhaps less than 2 staff months of effort), unless
used in a large number of products. It may still be
worthwhile to implement it in abbreviated form.
3. Independent of development technology and
platform.
4. SRE requires no changes in architecture, design, or
code - but it may suggest changes that would be
beneficial.
Nature of SRE - Widely Applicable

Activities of SRE Process and Relation
to Software Development Process
1. List Associated
Systems
2. Implement Operational
Profiles
3. Engineer “Just Right”

Reliability
4. Prepare for Test 5. Execute Test
6. Guide Test
Requirements Design and

Test
and Architecture Implementation
SRE Process
Illustration - FONE FOLLOWER (FF) -
Product Description
1. Subscriber calls FF, enters planned phone
numbers (forwardees) to which calls are
to be forwarded vs time.
2. FF forwards incoming calls (voice or fax)
from network to subscriber as per
program. Incomplete voice calls go to
pager (if subscriber has one) and then
voice mail.
SRE Process
1. List Associated
Systems
Profiles

Reliability
6. Guide Test

Test
SRE Process - List Associated Systems
List Associated Systems of Product
1. Base product
2. Major variations of base product (for
substantially different environments,
platforms, or configurations)
3. Frequently used supersystems of base
product or variations
SRE Process - List Associated Systems

1. List Associated
Systems
Profiles

Reliability
6. Guide Test

Test
SRE Process - Implement OPs
Operation
Operation: major system logical task

performed for initiator, which returns control
to system when complete.
Illustrations - FF:
Process fax call, Phone number entry, Audit
section of phone number database
SRE Process - Implement OPs - Concepts

Operational Profile
Operational profile (OP): complete set of operations
with probabilities of occurrence
Illustration - FF:
Occur.
Operation Prob.
Process voice call, no pager, ans. 0.21
Process voice call, pager, ans. 0.19
Process fax call 0.17
Process voice call, pager, ans. on page 0.13
•
•
•
1
SRE Process - Implement OPs - Concepts

Develop Operational Profiles –
Step by Step
Develop operational profile for base product
and each variation (supersystems have same
operational profiles as those of their related
base product or variations).
SRE Process - Implement OPs – Develop OPs

Determine Operational Profile
1. Identify initiators of operations [1]

2. Create operations list [1]
3. Review operations list [1]
4. Determine occurrence rates [1]
5. Determine occurrence probabilities [1]
Steps 1, 2, 3 are mostly the same across base
product and variations. New release often requires
only slight change from previous release, all steps.
SRE Process - Implement OPs – Develop OPs

Apply Operational Profiles
Apply operational profile and criticality

information to increase efficiency of:
1. Development of all developed software
2. Test of all associated systems
SRE Process - Implement OPs - Apply OPs

Apply Operational Profiles to
Increase Development Efficiency
For developed software in base product and

each variation:
1. Review functionality to be implemented
(Reduced Operation Software or ROS)
2. Suggest operations where looking for
opportunities for reuse will be most cost-effective
3. Plan a more competitive release strategy
(operational development)

Operational Development -
Illustration
Proportion of operations developed
Release 1
Release 2
Release 3
Proportion of use represented

Apply Operational Profiles to Increase
Development Efficiency
4. Allocate development resources:
A. Among operations - for system engineering,
architectural design, requirements reviews, design
reviews
B. Among modules - for code, code reviews, unit test ([3],
pp. 118 - 119)

Apply Operational Profiles to Increase
Test Efficiency
1. Allocate new test cases of release among new
operations of base product and variations (Prepare
for Test activity)
2. Allocate test time (Prepare for Test and Execute
Test activities)

1. List Associated
Systems
Profiles

Reliability
6. Guide Test

Test
SRE Process - Engineer “Just Right” Reliability
Failure and Fault
Failure
Failure Fault
Fault
Departure
Departureofofsystem
system Defect
Defectin
insystem
system
behavior
behaviorin
inexecution
execution implementation
implementationthat
that
from
fromuser
userneeds
needs causes
causesthe
thefailure
failurewhen
when
executed
executed
User-oriented
User-oriented Developer-oriented
Developer-oriented

Engineer “Just Right” Reliability -
Step by Step
1. Define failure consistently over life of product

release and clarify with examples [1]
2. Choose common reference measure for all failure
intensities [1]
3. Set system FIO for each associated system [1]
4. For any software you develop:
A. Find developed software FIO [1]
B. Choose software reliability strategies to meet developed
software FIO and schedule objectives with lowest
development cost [1]

1. List Associated
Systems
Profiles

Reliability
6. Guide Test

and Architecture Test
Implementation
SRE Process - Prepare for Test
Prepare for Test
1. Specify new test cases for new operations
A. Distribute new test cases to new operations based on
operational profile [1]
Illustration - FF:
Allocate 17% of test cases to Proc. fax call operation
B. Detail new test cases for each new operation by
selecting from possible choices of input variable values
with equal probability [1]
Illustration - FF:
Forwardee = Local calling area
2. Specify test procedure, based on the test
operational profile [1]
SRE Process - Prepare for Test
1. List Associated
Systems
Profiles

Reliability
6. Guide Test

Test
SRE Process - Execute Test
Execute Test
1. Determine and allocate test time among

associated systems and types of test
(feature, load, regression) [1]
*2. Invoke test in accordance with operational
profile [1]
3. Identify system failures and when they
occurred - use data in Guide Test [1]

Invoke Test
Base Product
LoadF
F Supersystem
Test
R R R R RRR RR R
Load
F Test Supersystem
R R R R
Variations
F = Feature test Certify reliability
R = Regression test Track reliability growth
1. List Associated
Systems
Profiles

Reliability
6. Guide Test

Test
SRE Process - Guide Test
Guide Test
Process system failure data gathered in test

to:
*1. Track reliability growth of developed
software of base product and variations
*2. Certify reliability of:
A. Base product and variations that customers
will acceptance test
B. Supersystems
3. Guide product release
SRE Process - Guide Test
Track Reliability Growth
1. Execute CASRE software reliability estimation

program to obtain FI / FIO ratio
2. Plot FI / FIO ratio against time
*3. Interpret plot
SRE Process - Guide Test - Track Reliability Growth

Interpret Plot : Illustration - FF
18
16
Conventional test
14
12
10
FI/FIO
8
6
4 Operational-profile-driven test
2
0
0 0.1 0.2 0.3 0.4 0.5
Mcalls
18
16 Scheduled
test
14
completion Solutions:
12
1. Defer features
10 2. Rebalance major
FI/FIO
8 quality characteristics
6 3. Increase work hours
4
2
0
0 0.1 0.2 0.3 0.4 0.5
Mcalls
18 Possible causes:
16 1. Poor change control
14 2. Poor control of test
12 execution, resulting
10 Investigate in test selection
FI/FIO significant probabilities varying
8
6
upward in time
4
trends
2
0
0 0.1 0.2 0.3 0.4 0.5
Mcalls
18
16
14
12
10
FI/FIO
8
6
4 Terminate
2
test
0
0 0.1 0.2 0.3 0.4 0.5
Mcalls
Certify Reliability – Dem. Chart:
Illus. - FF – BP Supersystem
16
Failure 14
number
12 Reject
Mcalls
10 Fail. at Normalized
No. Failure Units
8
Continue
6 1 0.00375 0.75
2 0.00625 1.25
4 Accept 3 0.025 5
2
0 Failure intensity objective:
0 2 4 6 8 10
Normalized units 200 failures / Mcalls
SRE Process - Guide Test - Certify Reliability

SRE and You
1. SRE gives you a powerful way to
engineer software-based products so
you can be confident in the availability
and reliability of the software-based
product you deliver as you deliver it in
minimum time with maximum efficiency.
2. With SRE you control the process; it
doesn’t control you.
3. SRE is a vital skill for being competitive.
Conclusion - SRE and You

To Explore Further
1. More Reliable Software Faster and Cheaper, two
day course, described on internet at
http://members.aol.com/JohnDMusa/FLweb.htm
2. Software Reliability Engineering website:
http://members.aol.com/JohnDMusa/
Overview, briefing for managers, bibliography of
articles by software reliability engineering users,
course information, useful references, Question of
the Month.
3. Musa, J. D., Software Reliability Engineering: More
Reliable Software, Faster Development and
Testing, ISBN 0-07-913271-5, McGraw-Hill, 1998.
Detailed, extensive treatment of practice.
Conclusion - Explore
To Explore Further
4. Musa, Iannino, Okumoto; Software
Reliability: Measurement, Prediction,
Application, ISBN 0-07-044093-X, McGraw-
Hill, 1987. Practice plus extensive theoretical
background.
5. Musa, J.D., More Reliable Software Faster
and Cheaper. Overview of software
reliability engineering, suitable for managers
and anyone wanting a fast and broad but not
detailed understanding of the topic. May be
downloaded from:
http://members.aol.com/JohnDMusa/ARTweb.htm
To Explore Further
6. Lyu, M. (Editor), Handbook of Software
Reliability Engineering , ISBN 0-07-039400-8,
McGraw-Hill, 1996. Particularly useful for
CD/ROM of CASRE program.
7. IEEE Computer Society Technical Committee on
Software Reliability Engineering. Professional
organization of the field: publishes newsletter,
sponsors ISSRE annual international
conference): membership application at
http://www.tcse.org
Chapter 7
Software Testing in Industry
(By Maurice Siteur)

2d
2c
1a
1b
2b 2a 1c

Program
TMap
Testing in embedded software
CMM/TMM (process improvement)

Test tools

Testing
Testing is a process of planning, preparation

and measuring, with the aim to view the
characteristics of a product, that makes it
possible to make judgements about the
quality of that product.

Test structure
T Techniques
Life cycle
L
I O
Infrastructure Organization
TMap
Master test plan
Unit Functional Production

Unit test integration System test acceptance Acceptance
test test test

What makes testing difficult
Specifications are not always right

There are bugs in the software (you should
test for that)
Testing everything is impossible!!
Reproducing tests

What makes embedded systems different
Hardware and software (not available when

you would like it to be)
Mechanics, electronics, optics, ….
Stubs and drivers needed
Performance, timing, synchronisation,

resource usage
Regulatory requirements

Embedded software testing - example
Unit testing programs/components
Unit integration testing
System test
Software acceptance test
Hardware component test
Hardware integration test
Software/Hardware integration test
Manufacturers acceptance test
Client acceptance test

TMap essentials
Test planning
‘Risk based’ testing
Test design
Test reporting

Test planning
Looks like normal plan of approach
Approach/strategy
based on risk's
Use of quality attributes
Here test coverage is determined

Coverage levels
Theory of finite state machines

Test design
Logical and physical
Determination of test cases
Test case
Description
Starting point
Input
Expected result
Regression test / re use of test cases

Test reporting
Record findings of test execution

Report on progress
End report
Conclusions
Findings

Guidelines
System development 75%, testing 25%
Preparation 20%
Specification 35%
Execution 40%
Completion 5%
Planning and control 10% on top
Test automation can turn these figures up side

down.
Process improvement
Software process improvement

Capability Maturity Model
Test process improvement

Test Maturity Model
TPI (Homework)

CMM

Level 5: Optimization, Defect Prevention and
TMM Quality Control

- Test proces optimization
- Quality control
- Application of process data for defect prevention
Level 4: Management and Measurement

- Software quality evaluation
- Establish a test measurement program
- Establish an organization-wide review program
Level 3: Integration
- Control and monitor the test process
- Integrate testing into the software lifecycle
- Establish a technical training program
- Establish a software test organization
Level 2: Phase Definition

- Institutionalize basic testing techniques and methods
- Initiate a test planning process
- Develop testing and debugging goals
Level 1: Initial

Tool Support of testing activities
System development
Requirements
Acceptance criteria Test planning
Specifications
Static testing
Generating testsets
for system- and acc.testing
Design
Static testing
Generating testsets
on the basis of design
Coding
Static testing
Generating testsets
for progr.test
Dynamic testing
Test execution
• Capture and playback Aids White box testing
• Test drivers • Test design • Debugging
• Load en performance testing • Test data manipulation • Test coverage
• Monitoring
Test repository
Test management & control

• Test planning
• Configuration management
• Defect tracking / incident management
• Progress control

Tool selection and implementation
Identify your test activities and see where tools can be

supportive
Creation of long list
Creation of a shortlist
Arrange demo’s
Evaluation of selected tools
Review and select tool
Pilot project
Roll out
Are you ready for CAST?

1. What is tested?
2. How to test?

2d
2c
1a
1b
2b 2a 1c

Homework
Which CMM and TMM level has the following

organization?
- 300 employees, 20 IT
- Working on getting documentation done
- Project plans are made
- They do a survey on testing tool
- Test planning is on its way
- Test design is made during acceptance test
What would you suggest to improve testing using TMM?

Homework
Try to make a case for yourself what automated

testing would cost and manual testing.
(Assume the organisation made the initial
investment)

Chương 8
Ước lượng chi phí

phần mềm

Mục tiêu
Giới thiệu các khái niệm và nền tảng về chi

phí và giá cả phần mềm
Mô tả 3 độ đo để đánh giá năng suất phần
mềm
Giải thích tại sao nên sử dụng các kỹ thuật
khác nhau để đánh giá ước lượng phần mềm
Mô tả các nguyên lý của giải thuật cho mô
hình ước lượng chi phí COCOMO 2

Các chủ đề chính
Năng suất phần mềm Software productivity

Kỹ thuật ước lượng Estimation techniques
Mô hình hoá chi phí giải thuật Algorithmic
cost modelling
Khoảng thời gian và thực hiện Project
duration and staffing

Các vấn đề ước lượng cơ bản
Mất bao nhiêu chi phí cần thiết để hoàn

thành một hoạt động?
Mất bao nhiêu thời gian để hoàn thành một
hoạt động?
Tổng chi phí cho một hoạt động là gì?
ước lượng dự án và lịch biểu là các hoạt động
quản lý có liên hệ với nhau.

Các thành phần chi phí phần mềm
Các chi phí về phần cứng và phần mềm.

Chi phí đi lại và đào tạo.
Các chi phí nỗ lực (Nhân tố nổi trội trong hầu hết
các dự án)
Tiền lương các kỹ sư làm việc trong dự án;
Chi phí bảo hiểm và xã hội.
Các chi phí phải trả trước
Chi phí xây nhà, lò sưởi, chiếu sáng.
Chi phí mạng và truyền thông.
Chi phí các phương tiện công trình công cộng dùng
chung (thư viện, nhà ăn, v.v...).

Chi phí và giá cả
Ước lượng được đánh giá để phát hiện giá cả

cho các nhà phát triển làm ra các hệ thống
phần mềm.
Không có mối quan hệ đơn giản nào giữa chi
phí phát triển và giá cả mà khách hàng phải
trả..
Hội đồng tổ chức, nền kinh tế, chính trị và
các điều kiện kinh doanh có ảnh hưởng đáng
kể đến giá phải trả.

Các nhân tố giá cả phần mềm
Cơ hội thị trường A development organisation may quote a low price because it
wishes to move into a new segment of the software market.
Accepting a low profit on one project may give the opportunity
of more profit later. The experience gained may allow new
products to be developed.
Sự không chắc If an organisation is unsure of its cost estimate, it may increase
chắn của luợng its price by some contingency over and above its normal profit.
lượng chi phí
Các điều khoản A customer may be willing to allow the developer to retain
hơp đồng ownership of the source code and reuse it in other projects. The
price charged may then be less than if the software source code
is handed over to the customer.
Thay đổi yêu cầu If the requirements are likely to change, an organisation may
lower its price to win a contract. After the contract is awarded,
high prices can be charged for changes to the requirements.
Sức mạnh về tài Developers in financial difficulty may lower their price to gain
chính a contract. It is better to make a smaller than normal profit or
break even than to go out of business.

Năng suất phần mềm
Độ đo tỷ lệ mà các kỹ sư phát triển phần

mềm đưa ra ra phần mềm và các tài liệu liên
quan.
Không định hướng tới chất luợng mà thông
qua đảm bảo chất lượng là một nhân tố đánh
giá năng suất.
Một cách cần thiết, chúng ta muốn đo chức
năng của sản phẩm đưa ra trên đơn vị thời
gian.

Độ đo năng suất
Độ đo liên quan đến kích cỡ : Dựa vào đầu ra

của quá trình làm phần mềm. Có thể là só
dòng của mã nguồn, các lệnh mã đích v.v....
Độ đo liên quan đến chức năng: Dựa vào ước
lượng các chức năng của sản phẩm phân
phối. Điểm chức năng là cách tốt nhất để đo
cho loại này.

Measurement problems
Estimating the size of the measure (e.g. how

many function points).
Estimating the total number of programmer
months that have elapsed.
Estimating contractor productivity (e.g.
documentation team) and incorporating this
estimate in overall estimate.

Lines of code
What's a line of code?

The measure was first proposed when programs were
typed on cards with one line per card;
How does this correspond to statements as in Java which
can span several lines or where there can be several
statements on one line.
What programs should be counted as part of the
system?
This model assumes that there is a linear
relationship between system size and volume of
documentation.

Productivity comparisons
The lower level the language, the more

productive the programmer
The same functionality takes more code to implement in
a lower-level language than in a high-level language.
The more verbose the programmer, the higher
the productivity
Measures of productivity based on lines of code suggest
that programmers who write verbose code are more
productive than programmers who write compact code.

System development times
Analysis Design Coding Testing Documentation

Assembly code 3 weeks 5 weeks 8 weeks 10 2 weeks
High-level language 3 weeks 5 weeks 4 weeks weeks 2 weeks
6 weeks
Size Effort Productivity

Assembly code 5000 lines 28 weeks 714 lines/month
High-level language 1500 lines 20 weeks 300 lines/month

Function points
Based on a combination of program characteristics

external inputs and outputs;
user interactions;
external interfaces;
files used by the system.
A weight is associated with each of these and the
function point count is computed by multiplying
each raw count by the weight and summing all
values.
UFC = ∑(number of elements of given type) × (weight)

Function points
The function point count is modified by complexity

of the project
FPs can be used to estimate LOC depending on the
average number of LOC per FP for a given language
LOC = AVC * number of function points;
AVC is a language-dependent factor varying from 200-
300 for assemble language to 2-40 for a 4GL;
FPs are very subjective. They depend on the
estimator
Automatic function-point counting is impossible.

Object points
Object points (alternatively named application

points) are an alternative function-related measure
to function points when 4Gls or similar languages
are used for development.
Object points are NOT the same as object classes.
The number of object points in a program is a
weighted estimate of
The number of separate screens that are displayed;
The number of reports that are produced by the system;
The number of program modules that must be developed
to supplement the database code;

Object point estimation
Object points are easier to estimate from a

specification than function points as they are
simply concerned with screens, reports and
programming language modules.
They can therefore be estimated at a fairly
early point in the development process.
At this stage, it is very difficult to estimate
the number of lines of code in a system.

Productivity estimates
Real-time embedded systems, 40-160

LOC/P-month.
Systems programs , 150-400 LOC/P-month.
Commercial applications, 200-900
LOC/P-month.
In object points, productivity has been
measured between 4 and 50 object
points/month depending on tool support and
developer capability.

Factors affecting productivity
Application Knowledge of the application domain is essential for effective
domain software development. Engineers who already understand a
experience domain are likely to be the most productive.
Process quality The development process used can have a s ignificant effect on
productivity. This is covered in Chapter 28.
Project size The larger a project, the more time required for team
communications. Less time is available for development so
individual productivity is reduced.
Technology Good support technology such as C ASE tools, configuration
support management systems, etc. can improve productivity.
Working As I discussed in Chapter 25, a q uiet working environment with
environment private work areas contributes to improved productivity.

Quality and productivity
All metrics based on volume/unit time are

flawed because they do not take quality into
account.
Productivity may generally be increased at the
cost of quality.
It is not clear how productivity/quality metrics
are related.
If requirements are constantly changing then an
approach based on counting lines of code is not
meaningful as the program itself is not static;

Estimation techniques
There is no simple way to make an accurate

estimate of the effort required to develop a software
system
Initial estimates are based on inadequate information in a
user requirements definition;
The software may run on unfamiliar computers or use
new technology;
The people in the project may be unknown.
Project cost estimates may be self-fulfilling
The estimate defines the budget and the product is
adjusted to meet the budget.

Changing technologies
Changing technologies may mean that previous

estimating experience does not carry over to new
systems
Distributed object systems rather than mainframe
systems;
Use of web services;
Use of ERP or database-centred systems;
Use of off-the-shelf software;
Development for and with reuse;
Development using scripting languages;
The use of CASE tools and program generators.

Algorithmic cost modelling.

Expert judgement.
Estimation by analogy.
Parkinson's Law.
Pricing to win.

Algorithmic A model based on historical cost information that relates some software
cost modelling metric (usually its size) to the project cost is used. An estimate is made
of that metric and the model predicts the effort required.
Expert Several experts on the proposed software development techniques and
judgement the application domain are consulted. They each estimate the project
cost. These estimates are compared and discussed. The estimation
process iterates until an agreed estimate is reached.
Estimation by This technique is applicable when other projects in the same application
analogy domain have been completed. The cost of a new project is estimated by
analogy with these completed projects. Myers (Myers 1989) gives a
very clear description of this approach.
ParkinsonÕs ParkinsonÕsLaw states that work expands to fill the time available. The
Law cost is determined by available resources rather than by objective
assessment. If the software has to be delivered in 12 months and 5
people are available, the effort required is estimated to be 60 person-
months.
Pricing to win The software cost is estimated to be whatever the customer has
available to spend on the project. The estimated effort depends on the
customerÕs budget and not on the software functionality.

Pricing to win
The project costs whatever the customer has

to spend on it.
Advantages:
You get the contract.
Disadvantages:
The probability that the customer gets the system
he or she wants is small. Costs do not accurately
reflect the work required.

Top-down and bottom-up estimation
Any of these approaches may be used top-

down or bottom-up.
Top-down
Start at the system level and assess the overall
system functionality and how this is delivered
through sub-systems.
Bottom-up
Start at the component level and estimate the
effort required for each component. Add these
efforts to reach a final estimate.

Top-down estimation
Usable without knowledge of the system

architecture and the components that might
be part of the system.
Takes into account costs such as integration,
configuration management and
documentation.
Can underestimate the cost of solving difficult
low-level technical problems.

Bottom-up estimation
Usable when the architecture of the system is

known and components identified.
This can be an accurate method if the system
has been designed in detail.
It may underestimate the costs of system
level activities such as integration and
documentation.

Estimation methods
Each method has strengths and weaknesses.

Estimation should be based on several methods.
If these do not return approximately the same
result, then you have insufficient information
available to make an estimate.
Some action should be taken to find out more in
order to make more accurate estimates.
Pricing to win is sometimes the only applicable
method.

Pricing to win
This approach may seem unethical and un-

businesslike.
However, when detailed information is lacking it
may be the only appropriate strategy.
The project cost is agreed on the basis of an outline
proposal and the development is constrained by that
cost.
A detailed specification may be negotiated or an
evolutionary approach used for system
development.

Algorithmic cost modelling
Cost is estimated as a mathematical function of

product, project and process attributes whose
values are estimated by project managers:
Effort = A × SizeB × M
A is an organisation-dependent constant, B reflects the
disproportionate effort for large projects and M is a
multiplier reflecting product, process and people
attributes.
The most commonly used product attribute for cost
estimation is code size.
Most models are similar but they use different
values for A, B and M.
Estimation accuracy
The size of a software system can only be

known accurately when it is finished.
Several factors influence the final size
Use of COTS and components;
Programming language;
Distribution of system.
As the development process progresses then
the size estimate becomes more accurate.

Estimate uncertainty

The COCOMO model
An empirical model based on project experience.

Well-documented, ‘independent’ model which is not
tied to a specific software vendor.
Long history from initial version published in 1981
(COCOMO-81) through various instantiations to
COCOMO 2.
COCOMO 2 takes into account different approaches
to software development, reuse, etc.

COCOMO 81
Project Formula Description

complexity
Simple PM = 2.4 (KDSI)1.05 × M Well-understood applications
developed by small teams.
Moderate PM = 3.0 (KDSI)1.12 × M More complex projects where
team members may have limited
experience of related systems.
Embedded PM = 3.6 (KDSI)1.20 × M Complex projects where the
software is part of a strongly
coupled complex of hardware,
software, regulations and
operational procedures.

COCOMO 2
COCOMO 81 was developed with the assumption
that a waterfall process would be used and that
all software would be developed from scratch.
Since its formulation, there have been many
changes in software engineering practice and
COCOMO 2 is designed to accommodate different
approaches to software development.

COCOMO 2 models
COCOMO 2 incorporates a range of sub-models that

produce increasingly detailed software estimates.
The sub-models in COCOMO 2 are:
Application composition model. Used when software is
composed from existing parts.
Early design model. Used when requirements are
available but design has not yet started.
Reuse model. Used to compute the effort of integrating
reusable components.
Post-architecture model. Used once the system
architecture has been designed and more information
about the system is available.

Use of COCOMO 2 models

Application composition model
Supports prototyping projects and projects where

there is extensive reuse.
Based on standard estimates of developer
productivity in application (object) points/month.
Takes CASE tool use into account.
Formula is
PM = ( NAP × (1 - %reuse/100 ) ) / PROD
PM is the effort in person-months, NAP is the number of
application points and PROD is the productivity.

Object point productivity
DeveloperÕs experience Very low Low Nominal High Very high

and capability
ICASE maturity and Very low Low Nominal High Very high
capability
PROD (NOP/month) 4 7 13 25 50

Early design model
Estimates can be made after the requirements have

been agreed.
Based on a standard formula for algorithmic models
PM = A × SizeB × M where
M = PERS × RCPX × RUSE × PDIF × PREX × FCIL × SCED;
A = 2.94 in initial calibration, Size in KLOC, B varies from
1.1 to 1.24 depending on novelty of the project,
development flexibility, risk management approaches and
the process maturity.

Multipliers
Multipliers reflect the capability of the developers,
the non-functional requirements, the familiarity with
the development platform, etc.
RCPX - product reliability and complexity;
RUSE - the reuse required;
PDIF - platform difficulty;
PREX - personnel experience;
PERS - personnel capability;
SCED - required schedule;
FCIL - the team support facilities.

The reuse model
Takes into account black-box code that is

reused without change and code that has to
be adapted to integrate it with new code.
There are two versions:
Black-box reuse where code is not modified. An
effort estimate (PM) is computed.
White-box reuse where code is modified. A size
estimate equivalent to the number of lines of
new source code is computed. This then adjusts
the size estimate for new code.

Reuse model estimates 1
For generated code:

PM = (ASLOC * AT/100)/ATPROD
ASLOC is the number of lines of generated code
AT is the percentage of code automatically
generated.
ATPROD is the productivity of engineers in
integrating this code.

Reuse model estimates 2
When code has to be understood and

integrated:
ESLOC = ASLOC * (1-AT/100) * AAM.
ASLOC and AT as before.
AAM is the adaptation adjustment multiplier
computed from the costs of changing the reused
code, the costs of understanding how to integrate
the code and the costs of reuse decision making.

Post-architecture level
Uses the same formula as the early design
model but with 17 rather than 7 associated
multipliers.
The code size is estimated as:
Number of lines of new code to be developed;
Estimate of equivalent number of lines of new code
computed using the reuse model;
An estimate of the number of lines of code that have
to be modified according to requirements changes.

The exponent term
This depends on 5 scale factors (see next slide).

Their sum/100 is added to 1.01
A company takes on a project in a new domain. The
client has not defined the process to be used and
has not allowed time for risk analysis. The company
has a CMM level 2 rating.
Precedenteness - new project (4)
Development flexibility - no client involvement - Very high
(1)
Architecture/risk resolution - No risk analysis - V. Low .(5)
Team cohesion - new team - nominal (3)
Process maturity - some control - nominal (3)
Scale factor is therefore 1.17.
Exponent scale factors
Precedentedness Reflects the previous experience of the organisation with this type of
project. Very low means no previous experience, Extra high means
that the organisation is completely familiar with this application
domain.
Development Reflects the degree of flexibility in the development process. Very
flexibility low means a prescribed process is used; Extra high means that the
client only sets general goals.
Architecture/risk Reflects the extent of risk analysis carried out. Very low means little
resolution analysis, Extra high means a complete a thorough risk analysis.
Team cohesion Reflects how well the development team know each other and work
together. Very low means very difficult interactions, Extra high
means an integrated and effective team with no communication
problems.
Process maturity Reflects the process maturity of the organisation. The computation
of this value depends on the CMM Maturity Questionnaire but an
estimate can be achieved by subtracting the CMM process maturity
level from 5.

Multipliers
Product attributes
Concerned with required characteristics of the software
product being developed.
Computer attributes
Constraints imposed on the software by the hardware
platform.
Personnel attributes
Multipliers that take the experience and capabilities of the
people working on the project into account.
Project attributes
Concerned with the particular characteristics of the
software development project.
Effects of cost drivers
Exponent value 1.17
System size (including factors for reuse 128, 000 DSI
and requirements volatility)
Initial COCOMO estimate without 730 person-months
cost drivers
Reliability Very high, multiplier = 1.39
Complexity Very high, multiplier = 1.3
Memory constraint High, multiplier = 1.21
Tool use Low, multiplier = 1.12
Schedule Accelerated, multiplier = 1.29
Adjusted COCOMO estimate 2306 person-months
Reliability Very low, multiplier = 0.75
Complexity Very low, multiplier = 0.75
Memory constraint None, multiplier = 1
Tool use Very high, multiplier = 0.72
Schedule Normal, multiplier = 1
Adjusted COCOMO estimate 295 person-months

Project planning
Algorithmic cost models provide a basis for

project planning as they allow alternative
strategies to be compared.
Embedded spacecraft system
Must be reliable;
Must minimise weight (number of chips);
Multipliers on reliability and computer constraints > 1.
Cost components
Target hardware;
Development platform;
Development effort.

Management options

Management option costs
Option RELY STOR TIME TOOLS LTEX Total effort Software cost Hardware Total cost
cost
A 1.39 1.06 1.11 0.86 1 63 949393 100000 1049393
B 1.39 1 1 1.12 1.22 88 1313550 120000 1402025
C 1.39 1 1.11 0.86 1 60 895653 105000 1000653
D 1.39 1.06 1.11 0.86 0.84 51 769008 100000 897490
E 1.39 1 1 0.72 1.22 56 844425 220000 1044159
F 1.39 1 1 1.12 0.84 57 851180 120000 1002706

Option choice
Option D (use more experienced staff)

appears to be the best alternative
However, it has a high associated risk as
experienced staff may be difficult to find.
Option C (upgrade memory) has a lower cost
saving but very low risk.
Overall, the model reveals the importance of
staff experience in software development.

Project duration and staffing
As well as effort estimation, managers must

estimate the calendar time required to complete a
project and when staff will be required.
Calendar time can be estimated using a COCOMO 2
formula
TDEV = 3 × (PM)(0.33+0.2*(B-1.01))
PM is the effort computation and B is the exponent
computed as discussed above (B is 1 for the early
prototyping model). This computation predicts the
nominal schedule for the project.
The time required is independent of the number of
people working on the project.

Staffing requirements
Staff required can’t be computed by diving the

development time by the required schedule.
The number of people working on a project varies
depending on the phase of the project.
The more people who work on the project, the more
total effort is usually required.
A very rapid build-up of people often correlates with
schedule slippage.

Key points
There is not a simple relationship between

the price charged for a system and its
development costs.
Factors affecting productivity include
individual aptitude, domain experience, the
development project, the project size, tool
support and the working environment.
Software may be priced to gain a contract
and the functionality adjusted to the price.

Key points
Different techniques of cost estimation should be used
when estimating costs.
The COCOMO model takes project, product, personnel
and hardware attributes into account when predicting
effort required.
Algorithmic cost models support quantitative option
analysis as they allow the costs of different options to
be compared.
The time to complete a project is not proportional to
the number of people working on the project.

Chapter 10
Quản lý cấu hình

Mục tiêu
Diễn tả tầm quan trọng của quản lý cấu hình

QLCH (CM)
Mô tả các hoạt động chính của QLCH: Lập kế
hoạch, quản lý thay đổi, quản lý phiên bản và
XD kiến trúc
Thảo luận về các công cụ CASE để hỗ trợ tiến
trình QLCH

Các chủ đề chính của QLCH
Lập kế hoạch QLCH

Quản lý thay đổi
Quản lý phiên bản và phát hành
Kiến trúc hệ thống
Cac công cụ CASE cho QLCH

Quản lý cấu hình
Các phiên bản mới của HT phần mềm được tạo ra

khi HT thay đổi :
Khi thay đổi máy tính/Hệ điều hành OS;
Yêu cầu các chức năng khác đi;
Các yêu cầu thay đổi do khách hàng đưa ra cho phù hợp
(may đo).
QLCH liên quan đến quản lý hệ thống phần mềm
tiến hoá:
Sự thay đổi hệ thống là hoạt động của một nhóm đội
ngũ;
QLCH nhằm điều khiển chi phí và nỗ lực thay đổi một hệ
thống.

Quản lý cấu hình (Tiếp)
Liên quan đến sự phát triển và ứng dụng của

các thủ tục và các chuẩn nhằm quản lý sản
phẩm phần mềm tiến hoá.
QLCH có thể xem như một phần của quá
trình quản lý chất lượng (tổng quát hơn).
Khi đưa ra QLCH, Các hệ thống phần mềm
đôi khi còn được gọi đường cơ sở như chúng
đang ở thời điểm xuất phát cho việc phát
triển sau này.

Những họ hệ thống

Các chuẩn QLCH
CM should always be based on a set of standards

which are applied within an organisation.
Standards should define how items are identified,
how changes are controlled and how new versions
are managed.
Standards may be based on external CM standards
(e.g. IEEE standard for CM).
Some existing standards are based on a waterfall
process model - new CM standards are needed for
evolutionary development.

Concurrent development and testing
A time (say 2pm) for delivery of system

components is agreed.
A new version of a system is built from these
components by compiling and linking them.
This new version is delivered for testing
using pre-defined tests.
Faults that are discovered during testing are
documented and returned to the system
developers.

Frequent system building
It is easier to find problems that stem from

component interactions early in the process.
This encourages thorough unit testing -
developers are under pressure not to ‘break
the build’.
A stringent change management process is
required to keep track of problems that have
been discovered and repaired.

Configuration management planning
All products of the software process may

have to be managed:
Specifications;
Designs;
Programs;
Test data;
User manuals.
Thousands of separate documents may be
generated for a large, complex software
system.

The CM plan
Defines the types of documents to be

managed and a document naming scheme.
Defines who takes responsibility for the CM
procedures and creation of baselines.
Defines policies for change control and
version management.
Defines the CM records which must be
maintained.

The CM plan
Describes the tools which should be used to

assist the CM process and any limitations on
their use.
Defines the process of tool use.
Defines the CM database used to record
configuration information.
May include information such as the CM of
external software, process auditing, etc.

Configuration item identification
Large projects typically produce thousands of

documents which must be uniquely identified.
Some of these documents must be maintained for
the lifetime of the software.
Document naming scheme should be defined
so that related documents have related names.
A hierarchical scheme with multi-level names is
probably the most flexible approach.
PCL-TOOLS/EDIT/FORMS/DISPLAY/AST-INTERFACE/CODE

Configuration hierarchy

The configuration database
All CM information should be maintained in a

configuration database.
This should allow queries about configurations to be
answered:
Who has a particular system version?
What platform is required for a particular version?
What versions are affected by a change to component X?
How many reported faults in version T?
The CM database should preferably be linked to the
software being managed.

CM database implementation
May be part of an integrated environment to

support software development.
The CM database and the managed documents
are all maintained on the same system
CASE tools may be integrated with this so
that there is a close relationship between the
CASE tools and the CM tools.
More commonly, the CM database is
maintained separately as this is cheaper and
more flexible.

Change management
Software systems are subject to continual

change requests:
From users;
From developers;
From market forces.
Change management is concerned with
keeping track of these changes and ensuring
that they are implemented in the most cost-
effective way.

The change management process
Request change by completing a change request form
Analyze change request
if change is validthen
Assess how change might be implemented
Assess change cost
Submit request to change control board
if change is acceptedthen
repeat
make changes to software
submit changed software for quality approval
until software quality is adequate
create new system version
else
reject change request
else
reject change request

Change request form
The definition of a change request form is part of

the
CM planning process.
This form records the change proposed, requestor
of change, the reason why change was suggested
and the urgency of change (from requestor of the
change).
It also records change evaluation, impact analysis,
change cost and recommendations (System
maintenance staff).

Change request form

Change tracking tools
A major problem in change management is

tracking change status.
Change tracking tools keep track the status of
each change request and automatically
ensure that change requests are sent to the
right people at the right time.
Integrated with E-mail systems allowing
electronic change request distribution.

Change control board
Changes should be reviewed by an external group

who decide whether or not they are cost-effective
from a strategic and organizational viewpoint rather
than a technical viewpoint.
Should be independent of project responsible
for system. The group is sometimes called a change
control board.
The CCB may include representatives from client
and contractor staff.

Derivation history
This is a record of changes applied to a

document or code component.
It should record, in outline, the change made,
the rationale for the change, who made the
change and when it was implemented.
It may be included as a comment in code. If
a standard prologue style is used for the
derivation history, tools can process this
automatically.

Component header information
// BANKSEC project (IST 6087)
//
// BANKSEC-TOOLS/AUTH/RBAC/USER_ROLE
//
// Object: currentRole
// Author: N. Perwaiz
// Creation date: 10th November 2002
//
// © Lancaster University 2002
//
// Modification history
// Version Modifier Date Change Reason
// 1.0 J. Jones 1/12/2002 Add header Submitted to CM
// 1.1 N. Perwaiz 9/4/2003 New field Change req. R07/02

Version and release management
Invent an identification scheme for system

versions.
Plan when a new system version is to be
produced.
Ensure that version management procedures
and tools are properly applied.
Plan and distribute new system releases.

Versions/variants/releases
Version An instance of a system which is

functionally distinct in some way from other
system instances.
Variant An instance of a system which is
functionally identical but non-functionally
distinct from other instances of a system.
Release An instance of a system which is
distributed to users outside of the
development team.

Version identification
Procedures for version identification should

define an unambiguous way of identifying
component versions.
There are three basic techniques for
component identification
Version numbering;
Attribute-based identification;
Change-oriented identification.

Version numbering
Simple naming scheme uses a linear

derivation
V1, V1.1, V1.2, V2.1, V2.2 etc.
The actual derivation structure is a tree or a
network rather than a sequence.
Names are not meaningful.
A hierarchical naming scheme leads to fewer
errors in version identification.

Version derivation structure

Attribute-based identification
Attributes can be associated with a version with

the combination of attributes identifying that
version
Examples of attributes are Date, Creator,
Programming Language, Customer, Status etc.
This is more flexible than an explicit naming scheme
for version retrieval; However, it can cause
problems with uniqueness - the set of attributes
have to be chosen so that all versions can be
uniquely identified.
In practice, a version also needs an associated
name for easy reference.

Attribute-based queries
An important advantage of attribute-based

identification is that it can support queries so
that you can find ‘the most recent version in
Java’ etc.
The query selects a version depending on
attribute values
AC3D (language =Java, platform = XP, date = Jan
2003).

Change-oriented identification
Integrates versions and the changes made to create

these versions.
Used for systems rather than components.
Each proposed change has a change set that
describes changes made to implement that change.
Change sets are applied in sequence so that, in
principle, a version of the system that incorporates
an arbitrary set of changes may be created.

Release management
Releases must incorporate changes forced on

the system by errors discovered by users and
by hardware changes.
They must also incorporate new system
functionality.
Release planning is concerned with when to
issue a system version as a release.

System releases
Not just a set of executable programs.

May also include:
Configuration files defining how the release is configured
for a particular installation;
Data files needed for system operation;
An installation program or shell script to install the
system on target hardware;
Electronic and paper documentation;
Packaging and associated publicity.
Systems are now normally released on optical disks
(CD or DVD) or as downloadable installation files
from the web.

Release problems
Customer may not want a new release of the

system
They may be happy with their current system
as the new version may provide unwanted
functionality.
Release management should not assume
that all previous releases have been
accepted. All files required for a release
should be re-created when a new release is
installed.

Release decision making
Preparing and distributing a system release is

an expensive process.
Factors such as the technical quality of the
system, competition, marketing requirements
and customer change requests should all
influence the decision of when to issue a new
system release.

System release strategy
Factor Description
Technical quality of If serious system faults are reported which affect the way in which
the system many customers use the system, it may be necessary to issue a fault
repair release. Howeve r, minor system faults may be repaired by issuing
patches (often distributed over the Internet) that can be applied to the
current release of the system.
Platform changes You may have to create a new release of a software application when a
new version of the operating system platform is released.
LehmanÕs fifth law This suggests that the increment of functionality that is included in each
(see Chapter 21) release is approximately constant. Therefore, if there has been a system
release with significant new functionality, then it may have to be
followed by a repair release.
Competition A new system release may be necessary because a co mpeting product is
available.
Marketing The marketing department of an organisation may have made a
requirements commitment for releases to be available at a particular date.
Customer change For customised systems, customers may have made and paid for a
proposals specific set of system change proposals and they expect a system release
as soon as these have been implemented.

Release creation
Release creation involves collecting all files

and documentation required to create a
system release.
Configuration descriptions have to be written
for different hardware and installation scripts
have to be written.
The specific release must be documented to
record exactly what files were used to create
it. This allows it to be re-created if necessary.

System building
The process of compiling and linking software

components into an executable system.
Different systems are built from different
combinations of components.
This process is now always supported by
automated tools that are driven by ‘build
scripts’.

System building problems
Do the build instructions include all required
components?
When there are many hundreds of components making up
a system, it is easy to miss one out. This should normally
be detected by the linker.
Is the appropriate component version
specified?
A more significant problem. A system built with the wrong
version may work initially but fail after delivery.
Are all data files available?
The build should not rely on 'standard' data files. Standards
vary from place to place.

System building problems
Are data file references within components
correct?
Embedding absolute names in code almost always causes
problems as naming conventions differ from place to place.
Is the system being built for the right platform
Sometimes you must build for a specific OS version or
hardware configuration.
Is the right version of the compiler and other
software tools specified?
Different compiler versions may actually generate different
code and the compiled component will exhibit different
behaviour.

System building

CASE tools for configuration management
CM processes are standardised and involve

applying pre-defined procedures.
Large amounts of data must be managed.
CASE tool support for CM is therefore
essential.
Mature CASE tools to support configuration
management are available ranging from
stand-alone tools to integrated CM
workbenches.

CM workbenches
Open workbenches
Tools for each stage in the CM process are
integrated through organisational procedures and
scripts. Gives flexibility in tool selection.
Integrated workbenches
Provide whole-process, integrated support for
configuration management. More tightly
integrated tools so easier to use. However, the
cost is less flexibility in the tools used.

Change management tools
Change management is a procedural process so it

can be modelled and integrated with a version
management system.
Change management tools
Form editor to support processing the change request
forms;
Workflow system to define who does what and to
automate information transfer;
Change database that manages change proposals and is
linked to a VM system;
Change reporting system that generates management
reports on the status of change requests.

Version management tools
Version and release identification

Systems assign identifiers automatically when a new version is
submitted to the system.
Storage management.
System stores the differences between versions rather than all
the version code.
Change history recording
Record reasons for version creation.
Independent development
Only one version at a time may be checked out for change.
Parallel working on different versions.
Project support
Can manage groups of files associated with a project rather
than just single files.

Delta-based versioning

System building
Building a large system is computationally

expensive and may take several hours.
Hundreds of files may be involved.
System building tools may provide
A dependency specification language and
interpreter;
Tool selection and instantiation support;
Distributed compilation;
Derived object management.

Component dependencies

Key points
Configuration management is the management of

system change to software products.
A formal document naming scheme should be
established and documents should be managed in a
database.
The configuration data base should record
information about changes and change requests.
A consistent scheme of version identification should
be established using version numbers, attributes or
change sets.

Key points
System releases include executable code, data,

configuration files and documentation.
System building involves assembling components
into a system…
CASE tools are available to support all CM activities
CASE tools may be stand-alone tools or may be
integrated systems which integrate support for
version management, system building and change
management.

Slides of Test and SQA

Uploaded by

Copyright:

Available Formats

You might also like

Slides of Test and SQA

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Slides of Test and SQA

Uploaded by

Copyright:

Available Formats

Software Testing and SQA

Kiểm thử và Đảm bảo Chất lượng Phần mềm

 Giới thiệu quá trình quản lý chất lượng và

Kiểm thử và Đảm bảo Chất lượng Phần mềm 9.2

 Process and product quality

Kiểm thử và Đảm bảo Chất lượng Phần mềm 9.3

 Concerned with ensuring that the required

Kiểm thử và Đảm bảo Chất lượng Phần mềm 9.4

Kiểm thử và Đảm bảo Chất lượng Phần mềm 9.5

 We cannot wait for specifications to improve

Kiểm thử và Đảm bảo Chất lượng Phần mềm 9.6

 Quality management is particularly important

Kiểm thử và Đảm bảo Chất lượng Phần mềm 9.7

Kiểm thử và Đảm bảo Chất lượng Phần mềm 9.8

Kiểm thử và Đảm bảo Chất lượng Phần mềm 9.9

 The quality of a developed product is

Kiểm thử và Đảm bảo Chất lượng Phần mềm 9.10

 There is a straightforward link between process and product in

Kiểm thử và Đảm bảo Chất lượng Phần mềm 9.11

Kiểm thử và Đảm bảo Chất lượng Phần mềm 9.12

 Define process standards such as how

Kiểm thử và Đảm bảo Chất lượng Phần mềm 9.13

 Standards are the key to effective quality

Kiểm thử và Đảm bảo Chất lượng Phần mềm 9.14

 Encapsulation of best practice- avoids

Kiểm thử và Đảm bảo Chất lượng Phần mềm 9.15

Product standards Process standards

Kiểm thử và Đảm bảo Chất lượng Phần mềm 9.16

 They may not be seen as relevant and up-to-

Kiểm thử và Đảm bảo Chất lượng Phần mềm 9.17

 Involve practitioners in development. Engineers

Kiểm thử và Đảm bảo Chất lượng Phần mềm 9.18

 An international set of standards for quality

Kiểm thử và Đảm bảo Chất lượng Phần mềm 9.19

Management responsibility Quality system

Kiểm thử và Đảm bảo Chất lượng Phần mềm 9.20

Kiểm thử và Đảm bảo Chất lượng Phần mềm 9.21

Kiểm thử và Đảm bảo Chất lượng Phần mềm 9.22

 Particularly important - documents are the tangible

Kiểm thử và Đảm bảo Chất lượng Phần mềm 9.23

Kiểm thử và Đảm bảo Chất lượng Phần mềm 9.24

 Document identification standards

Kiểm thử và Đảm bảo Chất lượng Phần mềm 9.25

 Interchange standards allow electronic documents

Kiểm thử và Đảm bảo Chất lượng Phần mềm 9.26

 A quality plan sets out the desired product

Kiểm thử và Đảm bảo Chất lượng Phần mềm 9.27

 Quality plan structure

Kiểm thử và Đảm bảo Chất lượng Phần mềm 9.28

Safety Understandability Portability

Kiểm thử và Đảm bảo Chất lượng Phần mềm 9.29

 This involves checking the software

Kiểm thử và Đảm bảo Chất lượng Phần mềm 9.30

 This is the principal method of validating the quality

Kiểm thử và Đảm bảo Chất lượng Phần mềm 9.31

Review type Principal purpose

Kiểm thử và Đảm bảo Chất lượng Phần mềm 9.32

 A group of people carefully examine part or all

Giới thiệu quá trình quản lý chất lượng và

Process and product quality

Concerned with ensuring that the required

We cannot wait for specifications to improve

Quality management is particularly important

The quality of a developed product is

There is a straightforward link between process and product in

Define process standards such as how

Standards are the key to effective quality

Encapsulation of best practice- avoids

They may not be seen as relevant and up-to-

Involve practitioners in development. Engineers

An international set of standards for quality

Particularly important - documents are the tangible

Document identification standards

Interchange standards allow electronic documents

A quality plan sets out the desired product

Quality plan structure

This involves checking the software

This is the principal method of validating the quality

A group of people carefully examine part or all

Quality function - they are part of the general

The objective is the discovery of system

Comments made during the review should be

Software measurement is concerned with deriving a

Any type of measurement which relates to a

A software property can be measured.

A software measurement process may be

A metrics programme should be based on a

Don’t collect unnecessary data

A quality metric should be a predictor of

Dynamic metrics are closely related to software

It is not always obvious what data means

Reducing the number of faults in a program

Software quality management is concerned with

Software measurement gathers information

Tổng quan về môn học

establish quality of software (Hetzel, 1988)

unimplemented required features