@paul_gerrard

Agile Test Strategy

Paul Gerrard
paul@gerrardconsulting.com

gerrardconsulting.com
Overview
• What is Agile Test Strategy?
• Project Profiling
• (Test Strategy as) Agile Interventions
• Test Automation
• What’s Left?
• Summary
• Q&A

Intelligent Definition and Assurance Slide 2

What is Agile Test
Strategy?

Agile Strategy – an oxymoron?

Agile Test Strategy
• What do we mean by this?
1. AGILE Test Strategy – how to create a test
strategy in an Agile way?
2. AGILE Test Strategy – a test strategy for an Agile
project?

• We’ll look at how we created an Agile approach

to strategy, but we’ll spend more time on
strategy for an Agile project.
Intelligent Definition and Assurance Slide 4
Google “Agile Test Strategy”
• There are plenty of recipes out there
• Most offer a selection of techniques but don’t
provide much guidance on how to blend them
• We need to know how to make choices, not just
know what choices exist

• Strategy is a thought process, not a document

– Although you might document the ideas for reuse, as
a reminder or ‘for the record’.
Intelligent Definition and Assurance Slide 5
Agile governance
• Governance is the act of governing. It relates to
decisions that define expectations, grant power,
or verify performance
Wikipedia
• Define expectations – DEFINITION of need
• Grant power – DELEGATION to a project team
• Verify performance – ASSURANCE of solution.

Intelligent Definition and Assurance Slide 6

Strategy helps you decide what to do
A. The strategy presents some decisions that can be made
ahead of time
B. Defines the process or method or information that will
allow decisions to be made (in project)
C. Sets out the principles (or process) to follow for uncertain
situations or unplanned events

• In a structured/waterfall environment, most questions

answered off-the-shelf – “A-style, ready for anything”
• In an Agile environment – might have some ready-made
policies but we manage scope and adapt (mostly C?)
Intelligent Definition and Assurance Slide 7
Contexts of Test Strategy
Communication
Axioms Early
Testing De-
Risks Duplication

Test Opportunities
Goals
Strategy

Culture Automation
Contract User
Human involvement
resource Constraints
Skills
Artefacts
Environment Process
(lack of?) Timescales
Traditional v Agile test strategy
• Traditional – structured, goal/risk-driven
– Identify stakeholders; what are their goals?
– Product risk analysis
– Allocate risks/goals to test stages
– Formulate test stage definitions (entry/exit criteria,
environments, tools etc. etc.
• Agile – interventionist, consensus-driven
– Project profiling to set the testing theme
– Identify testing interventions (perhaps better, contributions) in
the Agile process
– Test policy overlays the process; catches exceptions.

Intelligent Definition and Assurance Slide 9

 Project Profiling

Select a profile for your project first,

then choose the aspects of test
strategy that suite your project
Template-driven? Bah!
• So this is just a template copy and edit process?
• Won’t you always end up with the same document?

• Profiling doesn’t need to be prescriptive

– No need to write a document if you don’t need to
– But if company policy or common sense dictates certain
approaches, save yourself some time
– Create a set of deeper, more detailed questions to be answered
(Pocketbook)
• Profilers are really just checklists: heuristic guidelines
designed to help you make choices and trade-offs.

Intelligent Definition and Assurance Slide 11

 Incident Mgt.

Using a Project Profiler to Derive Project Plan Tools

a Test Strategy and Project Plan Story Guideline

Test
(A government client example) Environment

Project Profiler
Cerise Waterfall
Test Plan Test Strategy
Project Manager Items
Orange
Risk Profiler
SCRUM/Agile
Consultation

Product Test Strategy

Green
Risks

Assurance

Unknowns

The Project Profiler (with Test Assurance) helps

Blue Project Managers to:
• Select a project style that fits (Waterfall or Agile)
• Identify the product risks that need testing
• Identify test activities to include in project plans
Business, Project Team • Carefully define the scope of the project
and Boards
Intelligent Definition and Assurance 12
Project profiling process
Task
1 Have the Information you need to hand
2 Project Profiler (section 3):
Select the statements that best match your project context. The Blue column indicates
that you need more information – consult your stakeholders, team or relevant Board(s).
3 Generic Risk Profiler (section 4):
Consider the generic project risks – which are significant for your project? Can testing
help?
4 Product Risk Profiler (Section 5):
Consider the functional and non-functional risks that most concern your stakeholders –
should they be tested?
5 Actions and Test Activities (Section 6):
Consider the actions that prompt your ‘next steps’ and the test activities that should be
incorporated into your project plan.
6 Create your Test Strategy from the Test Strategy Framework Template
7 Incorporate the activities from stage 5 and identified in 6 into your Project
Plan.

Intelligent Definition and Assurance 13

Project Profiler (part of section 3)
Project Aspect Cerise
Responsibility for Users will take
Acceptance responsibility for UAT; they
have UAT experience
Requirements (Sources New system replaces a
of Knowledge) well-understood existing
system; users have clear
vision of system goals and
prefer to document their
requirements up-front
Requirements Stability New system is a functional
replacement of an existing
system or a well-defined
process (requirements can
be fixed early on)
Visibility, Formality High visibility/risk to
general public; formal
progress reporting
required at board level;
fixed scope and
deliverables; formal
approvals and sign-offs
External Dependencies More than one or new
external suppliers
responsible for
development (and supplier
testing)
Etc. Etc.
Intelligent Definition and Assurance
Orange Green Blue
Users will be responsible Users will take part in UAT Users are unwilling/unable
for UAT but have no test or witness tests at critical to take part in UAT;
experience periods, and will review reluctant to make the
the outcome acceptance decision or not
known
Users want to collaborate Users put the onus of Inexperienced users who
to jointly define requirements elicitation on are unable or unwilling to
requirements and meet the project; requirements collaborate with
them incrementally or and the solution will evolve requirements gathering
iteratively
New system replaces an New system supports a New system supports a
existing system with new business need; new business need;
enhancements or an business process exists business process is not
established (but not but will change/evolve; yet known; users have no
necessarily documented users have experience of experience or
process) requirements requirements
High visibility/risk to Relatively low business- Potentially, high visibility,
business; formal progress risk; informal progress high risk project, uncertain
reporting required; some reporting is acceptable; impact on the business
defined deliverables, some partial solution may
deliverables will suffice,
emerge/evolve; some incremental/iterative
approvals and sign-offs delivery
Single, known supplier In-house development, no Dependencies on external
responsible for external dependencies suppliers, their
development (and supplier responsibilities or
testing) competence not yet known
Etc. Etc. Etc.
14
Project types - examples
Cerise Structured, waterfall style of project (and includes
COTS projects)
Orange Iterative/prototyping style of project using SCRUM in
a formal way and having dedicated resources for the
Business Analyst and Tester roles.
Green A project using SCRUM in a less formal way but not
having dedicated resources for the Business Analyst
and/or the Tester roles.
Blue Blue column statements describe where there is
insufficient information available to identify the style
of project and the recommendation must be that
some further investigation is required.

Intelligent Definition and Assurance 15

 (Test Strategy as)
Agile Interventions

I’m using Scrum/Sprint terminology,

but you don’t have to of course
Interventions (government client example)
No. Activity When?

• On the following
1 Story Challenge As stories are added to the
Product Backlog
2 Story Definition As stories are added to a

slides, we
Sprint Backlog
3 Daily Stand-Up Once per day during the

These activities are repeated for each Sprint iteration

Sprint
highlight 8 4

5
Story Refinement

Developer Testing
Occurs throughout the Sprint
as new information emerges
Occurs throughout the Sprint
interventions as the developer codes the
stories
6 Integration (and During and at the end of
• Some are test incremental
System) Testing
each sprint, including the
final sprint

phases, but some

aren’t

7 System Testing At the end of each sprint,

including the final sprint
8
User Acceptance At the end of each sprint,
Testing including the final sprint
Intelligent Definition
9 and Assurance
Non-functional Slideon
Expected to take place 17an
 1. Story Challenge
Suggest ‘what-ifs’ to
Project Level Test Activities
(This diagram shows three sprints, but there could be more or fewer)
challenge new stories
and define story
headlines Sprint Backlog Sprint Backlog Sprint Backlog

2. Story Definition
Introduce scenarios
to enhance the
Acceptance Criteria

Sprint 1 Sprint 2 Sprint 3

Developed Stories Developed Stories Developed Stories

New Code

Integration into 6. Integration Test 6. Integration Test 6. Integration Test

Existing Code base
Automated testing

Increasing
Scope of 7. System Test
Sys. Test
and UAT 8. User Test

Increasing Scope of Integration, Complete Tests after

System and Users Testing Final Sprint
 1. Story Challenge
Suggest ‘what-ifs’ to
Project Level Test Activities
(This diagram shows three sprints, but there could be more or fewer)
challenge new stories
and define story
headlines Sprint Backlog Sprint Backlog Sprint Backlog

2. Story Definition
Introduce scenarios
to enhance the
Acceptance Criteria

Sprint 1 Sprint 2 Sprint 3

Developed Stories Developed Stories Developed Stories

New Code

Integration into 6. Integration Test 6. Integration Test 6. Integration Test

Existing Code base
Automated testing

Increasing
Scope of 7. System Test
Sys. Test
and UAT 8. User Test

Increasing Scope of Integration, Complete Tests after

System and Users Testing Final Sprint
 1. Story Challenge
Suggest ‘what-ifs’ to
Project Level Test Activities
(This diagram shows three sprints, but there could be more or fewer)
challenge new stories
and define story
headlines Sprint Backlog Sprint Backlog Sprint Backlog

2. Story Definition
Introduce scenarios
to enhance the
Acceptance Criteria

Sprint 1 Sprint 2 Sprint 3

Developed Stories Developed Stories Developed Stories

New Code

Integration into 6. Integration Test 6. Integration Test 6. Integration Test

Existing Code base
Automated testing

Increasing
Scope of 7. System Test
Sys. Test
and UAT 8. User Test

Increasing Scope of Integration, Complete Tests after

System and Users Testing Final Sprint
 1. Story Challenge
Suggest ‘what-ifs’ to
Project Level Test Activities
(This diagram shows three sprints, but there could be more or fewer)
challenge new stories
and define story
headlines Sprint Backlog Sprint Backlog Sprint Backlog

2. Story Definition
Introduce scenarios
to enhance the
Acceptance Criteria

Sprint 1 Sprint 2 Sprint 3

Developed Stories Developed Stories Developed Stories

New Code

Integration into 6. Integration Test 6. Integration Test 6. Integration Test

Existing Code base
Automated testing

Increasing
Scope of
7. System Test
Int. Sys.
and UAT 8. User Test

Increasing Scope of Integration, Complete Tests after

System and Users Testing Final Sprint
 1. Story Challenge
Suggest ‘what-ifs’ to
Project Level Test Activities
(This diagram shows three sprints, but there could be more or fewer)
challenge new stories
and define story
headlines Sprint Backlog Sprint Backlog Sprint Backlog

2. Story Definition
Introduce scenarios
to enhance the
Acceptance Criteria

Sprint 1 Sprint 2 Sprint 3

Developed Stories Developed Stories Developed Stories

New Code

Integration into 6. Integration Test 6. Integration Test 6. Integration Test

Existing Code base
Automated testing

Increasing
Scope of
7. System Test
Int. Sys.
and UAT 8. User Test

Increasing Scope of Integration, Complete Tests after

System and Users Testing Final Sprint
 Test Activities in the Sprint
3. Daily Stand-Up
Report anomalies found, 5) Developer Testing
Daily Scrum Private ad-hoc tests and
stories tested, amended,
Stand-Up build/run automated unit
created
Meeting 24 tests
Hours
4. Story Refinement 6) Integration/System Testing
Refine scenarios to Incorporate automated unit
enhance story definition, tests into the CI regime.
create system tests as On weekly basis and at end of
stories, as required Sprint, deploy to System test
environment and tester runs
2-4 Weeks system tests.
Backlog tasks
expanded
by team
Sprint Backlog

Potentially Shippable
Product backlog Product increment
As prioritised by Product Owner

Intelligent Definition and Assurance 23

 Test Activities in the Sprint
3. Daily Stand-Up
Report anomalies found, 5) Developer Testing
Daily Scrum Private ad-hoc tests and
stories tested, amended,
Stand-Up build/run automated unit
created
Meeting 24 tests
Hours
4. Story Refinement 6) Integration/System Testing
Refine scenarios to Incorporate automated unit
enhance story definition, tests into the CI regime.
create system tests as On weekly basis and at end of
stories, as required Sprint, deploy to System test
environment and tester runs
2-4 Weeks system tests.
Backlog tasks
expanded
by team
Sprint Backlog

Potentially Shippable
Product backlog Product increment
As prioritised by Product Owner

Intelligent Definition and Assurance 24

Test Activities in the Sprint
3. Daily Stand-Up
Report anomalies found, 5) Developer Testing
Daily Scrum Private ad-hoc tests and
stories tested, amended,
Stand-Up build/run automated unit
created
Meeting 24 tests
Hours
4. Story Refinement 6) Integration/System Testing
Refine scenarios to Incorporate automated unit
enhance story definition, tests into the CI regime.
create system tests as On weekly basis and at end of
stories, as required Sprint, deploy to System test
environment and tester runs
2-4 Weeks system tests.
Backlog tasks
expanded
by team
Sprint Backlog

Potentially Shippable
Product backlog Product increment
As prioritised by Product Owner

Intelligent Definition and Assurance 25

Test Activities in the Sprint
3. Daily Stand-Up
Report anomalies found, 5) Developer Testing
Daily Scrum Private ad-hoc tests and
stories tested, amended,
Stand-Up build/run automated unit
created
Meeting 24 tests
Hours
4. Story Refinement 6) Integration/System Testing
Refine scenarios to Incorporate automated unit
enhance story definition, tests into the CI regime.
create system tests as On weekly basis and at end of
stories, as required Sprint, deploy to System test
environment and tester runs
2-4 Weeks system tests.
Backlog tasks
expanded
by team
Sprint Backlog

Potentially Shippable
Product backlog Product increment
As prioritised by Product Owner

Intelligent Definition and Assurance 26

4. Story Refinement (example definition)
Objectives  To define acceptance criteria for all stories that are included in a Sprint as they are worked
on by development
 To define scenarios that describe the tests and expected behaviours of the System
 Improve understanding of the requirement and communicate anomalies to developers
 To identify System Tests that exercise functionality of multiple stories that can be system
tested in this sprint
 To assure the completeness for stories in the current Sprint
What’s being tested?  Stories to be included in the current Sprint
Deliverables  Refined story definitions with defined acceptance criteria and scenarios, where appropriate
 System tests
Responsibilities (Orange)  Tester – challenges stories by suggesting potential scenarios, new stories, story merges and
splits; performs ad-hoc testing with/on behalf of developers; assures completeness of
stories.
 Developers – considers stories, evaluates impact on development
 Product Owner or Analyst – collates feedback and decisions on stories
 Product Owner – approves changes to stories, accepts completeness of stories
 Scrum Master – monitors progress; evaluates impact on resources and schedules
Responsibilities (Green)  Not performed in Green projects
Baseline  Story Guideline (reference 3)
Entry Criteria  On commencement of the Sprint
Exit Criteria  When all stories within a Sprint are considered complete
 Queries, anomalies, discrepancies and inconsistencies have been eliminated or explained
 System Tests appropriate to the Sprint have been defined
 Definition of acceptance is agreed with Product Owner

Intelligent Definition and Assurance 27

Test Automation

Could you create an Agile Test

Strategy without automation?
Brian Marick’s Testing quadrants

Intelligent Definition and Assurance 29

Test Automation Pyramid – Lisa
Crispin’s version (Google others)
• Pyramid reflects the
relative numbers of tests
• Focus on unit/component
– Acceptance of “Services”
• GUI are end-to-end
• Manual checking the
exception?

Intelligent Definition and Assurance 30

Where do you automate?
3. Non-Technical
testers write scripts Stories/
Scenarios
Tools Experts write
interface Test Code 4. Programmers
write low level HTTP
GUI Test Framework GET/POST calls
through a driver that
simulates a browser
GUI Test Tool Unit Test Framework
2. Technical Testers
code scripts directly
Browser HTTP Driver 1. Programmers
write unit tests or
execute embedded
HTTP/S HTTP/S unit tests using a
unit test framework
to test components
Inter/Intranet

HTTP/S
Test Code
Web Server

App. Server Unit Test Framework

API
DB Server
Distributed testing
• Use business stories and scenarios/acceptance
criteria to validate requirements
• Reuse those stories to feed ‘Acceptance-
Driven Development’ BDD/TDD process
• Automated tests are an Anti-Regression tactic
• Automated tests don’t replicate manual tests;
think of them as a ‘change trip-wire’ that
triggers an alarm, if tripped.

Intelligent Definition and Assurance 32

Deriving scenarios to test
• To understand feature scope? Story Challenge

• To get stakeholder to accept? Story Refinement

• To validate the requirement? Story Definition

• To estimate the work to build this feature?

Iteration Planning

• To system test this feature? System Testing

• To unit test this feature? Developer Testing

Scenarios are created to meet several goals

Intelligent Definition and Assurance 33
What’s Left?
Other aspects of test policy
• Definitions (of done etc.)
• Incident management
• Test automation
• Story format e.g. Gherkin
• Environment request and management
• Regression testing (at what levels)
• Test deliverables and documentation.

Intelligent Definition and Assurance Slide 35

The Three Amigos
• Business Analyst
– Liaises and manages stakeholders and their needs
– Transforms business requirements into specification
(at multiple levels)
• Developer
– Scopes, designs, builds, tests and delivers features
• Tester
– Challenges the thinking on the project
– Performs ‘Assurance in the small’.

Intelligent Definition and Assurance Slide 36

The tester’s contribution to testing
• _________ feature/story acceptance criteria
• _________ the developers to unit test (auto)
• _________ feature/story acceptance
• _________ acceptance test
• _________ service/UI level automation
• Scope: from low-level detail to system integration
• Liaison with integration testers and feedback

• Fill in the blanks yourself; negotiate with your team.

Intelligent Definition and Assurance Slide 37

Summary
Close
• Agile test strategy has its place and many aspects of test
can be pre-defined
• Importantly, we use a principled approach to deal with
the unexpected
• Project profiling can help
• Testing as interventions, rather than test phases
• The testing role is split at least three ways – the tester
doesn’t own testing – think TESTMASTER
• Test automation in the context of Specification by
Example, requirements validation, BDD, TDD.

Intelligent Definition and Assurance Slide 39

 @paul_gerrard
Agile Test Strategy
Paul Gerrard
paul@gerrardconsulting.com

gerrardconsulting.com