6 Reviews

6.1 Introduction
6.2 The Principles of Reviews
6.3 Types of Reviews
6.4 Introducing Reviews
6.5 Success Factors for Reviews

1
6 Reviews
6.1 Introduction

 A successful review process requires planning,

participation and follow-up.
 Test managers have responsibilities for the planning
and follow-up activities.
 Testers / Test Analysts must be active participants
in the review process, providing their unique views.
They should have formal review training to better
understand their respective roles in any technical
review process.

2
6 Reviews

6.1 Introduction
6.2 The Principles of Reviews
6.3 Types of Reviews
6.4 Introducing Reviews
6.5 Success Factors for Reviews

3
Reviews Learning Objectives

The Learning Objective for the video session is to:

 (K2) Compare review types with each other and show
their relative strengths, weaknesses and fields of use.
The concept of checklists will also be introduced but a
practical session on their use is deferred to the Live
classroom event
 (K3) Use a review checklist to verify code and
architecture from a testers perspective
 (K3) Use a review checklist to verify requirements and
use cases from a testers perspective

4
Quiz 6.2a
6.2 The Principles of Reviews

Which of the following best describes the benefits of

Reviews?
a) To find defects
b) Not only to find defects but find defects earlier in
the life cycle
c) Not only to find defects but find defects earlier in
the life cycle and remove the causes from the
process

5
Quiz 6.2a (suggested answer)
6.2 The Principles of Reviews

Which of the following best describes the benefits of

Reviews?
a) To find defects
b) Not only to find defects but find defects earlier in the life cycle
c) Not only to find defects but find defects earlier in the life cycle
and remove the causes from the process

The answer is (c).

a) Not just find defects
b) Not just find defects early but …!

6
Review
6.2 The Principles of Reviews

 Review: a type of static testing

• an evaluation of software elements or project status
to ascertain discrepancies from planned results and
to recommend improvement
(IEEE 1028 Standard for Software Reviews)
 Objective:
not just to find defects, but also, to find defects
earlier in the life cycle and to remove the causes
from the process

7
Why reviews?
6.2 The Principles of Reviews

Objective:
• to find defects, but also, to find defects earlier and to
remove the causes from the process
 Early defects are often the most important
• defects have the characteristic to multiply themselves
top-down
• cost of rework rises exponentially
 60% of the defects have already been made before
coding/implementation has started

8
Cost Escalation Model
6.2 The Principles of Reviews

Cost of fixing errors escalates as we move project towards field use

9
More Benefits
6.2 The Principles of Reviews

 Other defects (not failures!!) are found compared to

dynamic testing
• e.g. deviations from std.‟s, missing requirements,
insufficient maintainability
 Development productivity improvements
 Reduced lead-times
 Reduced dynamic testing costs & time
(less test cycles and re-testing)
 Improved communication

10
Code Reviews & Dynamic Testing
6.2 The Principles of Reviews

Review of Dynamic testing

source code (faults) (failures)

11
Code Reviews & Dynamic Testing
6.2 The Principles of Reviews

 Code reviews are more cost effective than dynamic

testing
• dynamic testing done well finds fewer defects than
static testing done well
 But code reviews cannot find all the faults, e.g.:
• Complier introduced errors
• Processor specific errors
• Environment specific problems
 What about Static Analysis?

12
IEEE Std 1028-1997
6.2 The Principles of Reviews

 The ISTQB syllabus is broadly based on this

standard
 The syllabus identifies key principles and provides a
comparative description of commonly used formal
review types including:
• Management review
• Audit
• Walkthrough
• Technical review
• Inspection

13
Basic Review Process
6.2 The Principles of Reviews

Individual
preparation

Kick-off Planning

Rework/follow-up
Review
meeting

Metrics/sign-off
14
Source Documents
6.2 The Principles of Reviews

 Reviews are more effective Requirements & Specifications

when source documents are
available
 Without source documents a
review is only likely to find errors
of omission and internal
inconsistency in the product
Standards Rules /checklists
document

15
Checklists
6.2 The Principles of Reviews

 A list of questions
 A negative answer identifies
an error
 Interpret the rules
 Identify frequently made
mistakes
 Must be maintained

16
Checklist questions
6.2 The Principles of Reviews

 Does this document conform to your standards and

house-style?
 Is this document clear and unambiguous to the
intended readership?
 Are all statements consistent with other statements
in the same or related documents?

17
Checklist questions
6.2 The Principles of Reviews

 Is this document complete? (or has something vital

been missed out)
 Have irrelevant topics been left out?
 Are the diagrams and pictures clear and a useful aid
in explaining the text?
 Have you checked the references outside the
document?

18
Possible Review Outcomes
6.2 The Principles of Reviews

 Is it fit for purpose?

 Possible outcomes:
• Product can be used as is;
• Issues need to be dealt with,
but a further review is not
required;
• Issues need to be dealt with
and a further review is required

19
Review Roles
6.2 The Principles of Reviews

 Leader (for the complete

review process)
 Chair

 Author

 Reviewers

 Scribe

 Presenter

20
Quiz 6.2b
6.2 The Principles of Reviews

 What are the benefits of reviews compared to

dynamic testing and to other static testing
techniques?
 Five minutes to note down your ideas!

21
Quiz 6.2b (suggested answer)
6.2 The Principles of Reviews

 Benefits of reviews compared to dynamic testing

and to other static testing techniques include:
• Help to improve communications via the review
meeting participation
• Help to increase our understanding of key aspects of
the application via “walkthroughs”
• Provide an opportunity to check the effectiveness of
project management via management review
• Provide the evidence to check process via audit
• Can be very cost effective compared with dynamic
testing and static analysis – earlier in SDLC
• Can validate requirements conformance whereas
static analysis checks intrinsic code quality
22
Key Learning Points
6.2 The Principles of Reviews

You should now be able to

 Explain the benefits of reviews compared to
dynamic testing and other static testing techniques

23
6 Reviews

6.1 Introduction
6.2 The Principles of Reviews
6.3 Types of Reviews
6.4 Introducing Reviews
6.5 Success Factors for Reviews

24
Review Types and Objectives
6.3 Types of Review

 Review types (from Foundation):

• Informal
• Walkthrough
• Technical review
• Inspection
 Other review types include Management Review
and Audit
 Choice of review depends upon:
• Objectives of the review
• Culture and maturity of the organization performing
the review
25
Learning Objectives
6.3 Types of Review

 Compare review types with each other and show

their relative strengths, weaknesses and fields of
use. (K2)

26
Types of Formal Review
6.3 Types of Review

 Formal Review types:

• Management review
• Audit
Previously covered in Foundation
• Walkthrough
• Technical review
• Inspection

27
Types of Formal Reviews
6.3 Types of Review

 Management review (IEEE 1028-1997):

„A systematic evaluation of a software acquisition,
supply, development, operation, or maintenance
process performed by or on behalf of management
that monitors progress, determines the status of plans
and schedules, confirms requirements and their system
allocation, or evaluates the effectiveness of
management approaches used to achieve fitness for
purpose.‟
Strength: forum for monitoring and control
Weakness: may be biased by management over-control
Field of use: progress reporting, problem solving
28
Key Characteristics Management Review
6.3 Types of Review

To monitor progress, assess status, and make

decisions
• Carried out by or for managers having direct responsibility for the
project or system
• Carried out by or for a stakeholder or decision maker, e.g. a
higher level manager or director
• Checks consistency with and deviations from plans, or adequacy
of management procedures
• Includes assessment of project risks
• Outcome includes action items and issues to be fixed
• Participants expected to prepare, decisions are documented

Note that test managers should participate in and may

instigate management reviews of testing progress.

29
Types of formal reviews : Audits
6.3 Types of Review

 Audit: An independent examination of a software product,

software process, or set of software processes to assess
compliance with specifications, standards, contractual
agreements, or other criteria (IEEE 1028:1997)
 Strengths: Audits can be extremely formal, and are usually
performed to demonstrate conformance to some set of expectations,
most likely an applicable standard or a contractual obligation, e.g.
• ISO 9000:2000 QA Audit
• CMMI /TMMi
 Weaknesses: Audits are the least effective at revealing defects in
the product
 Field of use: Evaluate/find defects in/conformance to the process!

30
Key Characteristics Audit
6.3 Types of Review

 Main purpose: provide independent evaluation of

compliance to processes, regulations, standards etc.
• A lead auditor is responsible for the audit and acts as
the moderator
• Auditors collect evidence of compliance through
interviews, witnessing and examining documents
• Outcome includes observations, recommendations,
corrective actions and a pass/fail assessment

31
Review type : Walkthrough
6.3 Types of Review

 Walkthrough (IEEE 1028-1997)

„A static analysis technique in which a designer or
programmer leads members of the development team and
other interested parties through a software product, and the
participants ask questions and make comments about
possible errors, violation of development standards, and other
problems.‟

Strength: easy to do, little prep, also educational

Weakness: less evidence (review records)
Field of use: Design “trade-offs” and proof of concepts

32
Review type : Technical Review
6.3 Types of Review

 Technical review (IEEE 1028-1997)

„A systematic evaluation of a software product by a team of
qualified personnel that examines the suitability of the
software product for its intended use and identifies
discrepancies from specifications and standards. Technical
reviews may also provide recommendations of alternatives
and examination of various alternatives.‟

Strength: Technical rigour, evidence record, discuss alternatives

Weakness: analysis paralysis! Time consuming!
Field of use: Technical solution, integrity!

33
Review type : Inspection
6.3 Types of Review

 Inspection (IEEE 1028-1997)

„A visual examination of a software product to detect and
identify software anomalies, including errors and deviations
from standards and specifications. Inspections are peer
examinations led by impartial facilitators who are trained
in inspection techniques. Determination of remedial or
investigative action for an anomaly is a mandatory element of
a software inspection, although the solution should not be
determined at the inspection meeting.‟

Strength: proven track record, review records!

Weakness: needs formal training, checklists & time
Field of use: critical systems, process/product improvement

34
Reviews Quiz 6.3
6.3 Types of Review

Suggest a suitable review type for each situation:

1. Agreeing control activities to manage reduced time scales

2. Identifying defects and process improvements in a requirements document
3. Checking whether a team has conformed to a stated process flow
4. Assessing the effectiveness of metrics and reporting on a project
5. Getting feedback when time is short but you need quick ideas
6. Agreeing solutions to problems in a system architecture
7. Introducing a new business process to a group of business users
8. Checking whether a design process is a suitable choice for a team to use

35
Reviews Quiz 6.3 – Suggested Answer
6.3 Types of Review

Suggest a suitable review type for each situation:

1. Agreeing control activities to manage reduced time scales Management
review
2. Identifying defects and process improvements in a requirements document
Inspection
3. Checking whether a team has conformed to a stated process flow Audit
4. Assessing the effectiveness of metrics and reporting on a project
Management review
5. Getting feedback when time is short but you need quick ideas Informal
review or a walkthrough
6. Agreeing solutions to problems in a system architecture Technical review
7. Introducing a new business process to a group of business users
Walkthrough
8. Checking whether a design process is a suitable choice for a team to use
Audit – for example for compliance to a specific CMMi level
36
Reviews Quiz 6.2c
6.2 The Principals of Reviews - revision

Can you remember: in what order would the

phases of a formal review usually occur?

a) Planning, preparation, kick-off, meeting & follow-up

b) Kick-off, planning, preparation, meeting, rework &
follow-up
c) Preparation, planning, kick-off, meeting, rework &
follow-up
d) Planning, kick-off, preparation, meeting, rework &
follow-up

37
Reviews Quiz 6.2c – Suggested Answer
6.2 The Principals of Reviews - revision

Can you remember: in what order would the

phases of a formal review usually occur?

a) Planning, preparation, kick-off, meeting & follow-up

b) Kick-off, planning, preparation, meeting, rework &
follow-up
c) Preparation, planning, kick-off, meeting, rework &
follow-up
d) Planning, kick-off, preparation, meeting, rework
& follow-up

38
Key Learning Points
6.3 Types of Review

You should now be able to:

 Compare review types with each other and show
their relative strengths, weaknesses and fields of
use. (K2)

39
Assignment 6.4 – Introducing Reviews

1. What is your current experience with reviews?

2. What review types have you participated in?
3. What worked well?
4. Do you use any check lists?
• if so, document examples
5. What is your typical “reading rate”?
6. What suggestion for improvements do you have?
7. If you have not used reviews where could you apply
them?
Document your response ready for the virtual classroom
session.
40
6 Reviews

6.1 Introduction
6.2 The Principles of Reviews
6.3 Types of Reviews
6.4 Introducing Reviews
6.5 Success Factors for Reviews

41
Success Factors
6.5 Success Factors for Review

 Technical Factors
 Organizational Factors
 People Factors

42
Success Factors - Technical Factors (1/2)
6.5 Success Factors for Review

 Ensure the defined process for the review type is

followed correctly, particularly for more formal types of
review such as inspection
 Record the costs of reviews (including time spent) and
benefits achieved
 Review early drafts or partial documents to identify
patterns of defects before they are built into the whole
document
 Ensure that the documents or partial documents are
review-ready before starting a review process (i.e. apply
entry criteria)
 Use organization-specific checklists of common defects
43
Success Factors - Technical Factors (2/2)
6.5 Success Factors for Review

 Use more than one type of review, depending on

objectives: document cleanup, technical improvement,
transfer of information, or progress management
 Review or inspect documents on which important
decisions will be made, for example, inspect a proposal,
contract or high level requirement before a management
review authorizing major expenditure
 Sample a limited subset of a document for assessment
 Encourage finding the most important defects: focus on
content not format
 Continuously improve the review process!

44
Success Factors - Organizational Factors (1/2)
6.5 Success Factors for Review

 Ensure managers allow adequate time to be spent on

review activities, even under deadline pressure
 Remember, time and budget spent are not in proportion
to the defects found
 Allow adequate time for rework of defects identified by
reviews
 Never use the metrics from reviews for individual
performance appraisal
 Ensure that the right people are involved in the different
types of review
 Provide training in reviews, particularly the more formal
review types
45
Success Factors - Organizational Factors (2/2)
6.5 Success Factors for Review

 Support a test analyst forum to share experience,

checklists and ideas for improvement
 Ensure everyone participates in reviews and everyone
has their own documents reviewed
 Apply the strongest review techniques to the most
important documents
 Ensure a well-balanced review team of people with
different skills and backgrounds
 Support process improvement actions to address
systemic problems
 Recognize improvements gained through the review
process
46
Success Factors - People Factors
6.5 Success Factors for Review

 Educate stakeholders to expect that defects will be found

and to allow time for rework and re-review
 Ensure the review is a positive experience for the author
 Welcome the identification of defects in a blame-free
atmosphere
 Ensure comments are constructive, helpful and
objective, not subjective
 Do not review if the author does not agree or is not
willing
 Encourage everyone to think deeply about the most
important aspects of the documents being reviewed

47
Quiz 6.5a
6.5 Success Factors for Reviews

A group of test analysts who has been on a testing

course return enthusiastic about implementing reviews
on their project.

What factors do they need to explain to their manager

to support reviews being introduced successfully?

48
Quiz 6.5a (suggested answer)
6.5 Success Factors for Reviews

A group of test analysts who has been on a testing course return

enthusiastic about implementing reviews on their project.
What factors do they need to explain to their manager to support
reviews being introduced successfully?
 Technical Factors: for example, make sure everyone agrees and
understand the review process, make sure the documents are
reviewed at the right time, use the right types of review
 Organizational Factors: for example, management support in time
and budget, use of metrics to measure efficiency and effectiveness
of reviews (Return on Investment) but NOT measurement of people
 People Factors: for example, helping the motivation of the author
and reviews by use of objective reporting of defects, help
stakeholders to understand why the reviews are helping

49
Quiz 6.5b - Checklists
6.5 Success Factors for Reviews

A requirements document has the paragraph:

For all new investment products, whenever a new
account is opened the customer record must be
updated with the new account number and investment
product code. Please note that for joint accounts both
client records must be updated with the investment
code.

What observations do you have and what type of

checklist points might help you identify defects?

50
Quiz 6.5b Checklists (Suggested Answer)
6.5 Success Factors for Reviews

A requirements document has the paragraph:

(ref?)For all new investment products, whenever a new account is
opened the customer record must be updated with the new account
number and investment product code. Please note that for joint
accounts both client records must be updated with the investment code.
 The words in red are potential defects. For example, is the investment
product code the same data item as the investment code? Is the
customer record the same as the client record? Checklists would help
us identify items for consistency, completeness, accuracy, ambiguity,
adherence to standards and traceability. Examples are
• Consistency – is each data item uniquely and consistently named?
• Traceability – do the requirements have unique references so they can be
tracked when they change? Is there sufficient traceability to allow us to
track when requirements have been tested?
• Accuracy and ambiguity – the words ALL and NEVER are warning signs –
do we mean that?
51
Assignment 6.5

As a Test Analyst you should be using checklists to :

• Verify code and architecture from a testers
perspective
• Verify requirements and use cases from a testers
perspective
 Thinking about the quiz you have just done,
investigate the topic of checklists for the above
areas and produce what you believe to be
appropriate checklists
 Have your response ready for the Live Virtual
Classroom session

52
Conclusion
6 Reviews

 Reviews are an effective and efficient means to

improve the quality of the software product
• requirements, specification, code, test specifications
 There is more than inspection
• Objective determines the review type
 Lots of „proof” and knowledge available

JUST DO IT!

53
End of Chapter 6 Summary
6 Reviews

6.1 Introduction
6.2 The Principles of Reviews
6.3 Types of Reviews (1 LO)
6.4 Introducing Reviews Reader
Chapter 6
6.5 Success Factors for Reviews

 The other 2 LOs (K3) for use of checklists are

covered further in your assignments and in the
Virtual Classroom
 A table of review types is in your Reader.

54