Just-In-Time Testing

Robert Sabourin
President
AmiBug.Com, Inc.
Montreal, Canada
rsabourin@amibug.com
June 21, 2021 © Robert Sabourin, 2003 Slide 1
AmiBug.Com, Inc.
 Overview
Introduction
Some Philosophy
Bug Flow
Testing Ideas
Test Triage
Developers
Testing Just In Time
Platforms
Exploratory Testing
Checklists
Scenarios
Status
Bibliography

June 21, 2021 © Robert Sabourin, 2003 Slide 2

AmiBug.Com, Inc.
 Just In Time Testing

• Robert Sabourin ,
Software Evangelist
• President
• AmiBug.Com Inc.
• Montreal, Quebec,
Canada
• rsabourin@amibug.com

June 21, 2021 © Robert Sabourin, 2003 Slide 3

AmiBug.Com, Inc.
 Just-In-Time Testing

Some Philosophy

June 21, 2021 © Robert Sabourin, 2003 Slide 4

AmiBug.Com, Inc.
 Fundamental Question
• How do you know when you are finished?

June 21, 2021 © Robert Sabourin, 2003 Slide 5

AmiBug.Com, Inc.
 Crosby on Quality
• “Quality is defined as conformance to
requirements”
• “Quality is not a measure of
GOODNESS”
– Phil B. Crosby, Quality is Free

June 21, 2021 © Robert Sabourin, 2003 Slide 6

AmiBug.Com, Inc.
 Edsger W. Dijkstra
• “Program testing can be used to show
the presence of bugs, but never to show
their absence”

June 21, 2021 © Robert Sabourin, 2003 Slide 7

AmiBug.Com, Inc.
 Boris Beizer
• “Why software has bugs – the
fundamental problem – Programming is
a bitch.”
• The Frozen Keyboard - 1986

June 21, 2021 © Robert Sabourin, 2003 Slide 8

AmiBug.Com, Inc.
 Ken Blanchard
• “Feedback is the breakfast of
champions!”

June 21, 2021 © Robert Sabourin, 2003 Slide 9

AmiBug.Com, Inc.
 Watts S. Humphrey
• “… the job of the software
engineer is to deliver high-
quality software products at
agreed cost and schedule …”
• “… even the most
experienced software engineer
injects about one defect for ten
lines of code …”

June 21, 2021 © Robert Sabourin, 2003 Slide 10

AmiBug.Com, Inc.
 C. Northcote Parkinson

Parkinson’s Law:
“…work expands so as to fill the
time available for its
completion…”

June 21, 2021 © Robert Sabourin, 2003 Slide 11

AmiBug.Com, Inc.
 Yoda

"No! Try not, Do. Or do not.

There is no try."

June 21, 2021 © Robert Sabourin, 2003 Slide 12

AmiBug.Com, Inc.
 Steve Covey

“…begin with the end in mind …

“…first things first …"

June 21, 2021 © Robert Sabourin, 2003 Slide 13

AmiBug.Com, Inc.
 Purpose of Testing

• Common definition:
– To find bugs before our customers do!
• Broader definition:
– The role of testing is to provide objective input to
facilitate business decisions!
– Keeps stakeholders aware of all issues or
concerns that relate to shipping a product!

June 21, 2021 © Robert Sabourin, 2003 Slide 14

AmiBug.Com, Inc.
 Bug Defined

• To make our job more fun, whenever

we have a concern with software, we
call it a “bug”.

June 21, 2021 © Robert Sabourin, 2003 Slide 15

AmiBug.Com, Inc.
 Just-In-Time Testing

• It’s all about people! (and the occasional

bug too)

June 21, 2021 © Robert Sabourin, 2003 Slide 16

AmiBug.Com, Inc.
 Just In Time Testing

Get Ready, Get Set,

Cause here it comes

June 21, 2021 © Robert Sabourin, 2003 Slide 17

AmiBug.Com, Inc.
 Just-In-Time Testing
Turbulence

June 21, 2021 © Robert Sabourin, 2003 Slide 18

AmiBug.Com, Inc.
 Just-In-Time Testing
Unprepared

June 21, 2021 © Robert Sabourin, 2003 Slide 19

AmiBug.Com, Inc.
 Just-In-Time Testing
So what exactly did they throw over the wall?

June 21, 2021 © Robert Sabourin, 2003 Slide 20

AmiBug.Com, Inc.
 Getting Things Done
• Metaphors
– RUTHLESS TRIAGE
– RING BELL
– WELL OILED MACHINES
• Themes
– Parallelism wherever and always
– Think and test 24/7, early and always
– Chunking
June 21, 2021 © Robert Sabourin, 2003 Slide 21
AmiBug.Com, Inc.
 First Things First
• Begin with the end in mind
– Gain Consensus
• Goals
– How do we know we are finished?
• Purpose
– Why are we doing this project?
– How will be react to change?
• Meaning
– What is a bug?
– What is a test?

June 21, 2021 © Robert Sabourin, 2003 Slide 22

AmiBug.Com, Inc.
 Just In Time Testing

Bug Flow

June 21, 2021 © Robert Sabourin, 2003 Slide 23

AmiBug.Com, Inc.
 Bug Flow
• We will be testing … imagine that we
actually find a bug!
– What are we going to do about it?
– How will we decide?
– When should we decide how to decide?
– When should we change how we decide?
– When should we review our past
decisions?
June 21, 2021 © Robert Sabourin, 2003 Slide 24
AmiBug.Com, Inc.
 Bug Flow
REFUSE

CHECK TRIAGE DESIGNATE

Entered Reviewed Prioritized Assigned

CORRECT

MANDATE

Unassigned Fixed Closed

CONFIRM

FAILURE

June 21, 2021 © Robert Sabourin, 2003 Slide 25

AmiBug.Com, Inc.
 Just In Time Testing

Developers

June 21, 2021 © Robert Sabourin, 2003 Slide 26

AmiBug.Com, Inc.
 Testability
• Influence development team
before software gets into your
hands
– Can application be controlled by
non-gui api for all business
functions?
– Can application logging be
controlled and logs viewed by
testing?
– Can everything which can be set
or changed be queried
programmatically?
– Is there a static equivalent for
dynamic guis?

June 21, 2021 © Robert Sabourin, 2003 Slide 27

AmiBug.Com, Inc.
 JIT Configuration
Management
• Make sure that
– Developers and testers
use different test servers
– Hand off to testing is
based on a baseline
– Smoke test before and
after delivery
– Keep previous build
– Know the last best build

June 21, 2021 © Robert Sabourin, 2003 Slide 28

AmiBug.Com, Inc.
 Testing
Just In Time

June 21, 2021 © Robert Sabourin, 2003 Slide 29

AmiBug.Com, Inc.
 Philosophy

We have precious little time to run tests!

We must always be prepared!

June 21, 2021 © Robert Sabourin, 2003 Slide 30

AmiBug.Com, Inc.
 Time

June 21, 2021 © Robert Sabourin, 2003 Slide 31

AmiBug.Com, Inc.

Getting
Things Done
REQ
FLOW
- Who manages them?
- What are they?
- Where can I find them?
- When are they updated?
- Why are they changing?
- How are they evolving?
- Do we observe turbulence?
June 21, 2021
- Who manages them?
- How are they prioritized?
- Where can I find them?
- Are the communicated?
- Do they get reprioritized?
- Are business drivers known?
- Are technical risks known?
BUG
Development FLOW
- Are builds delivered?
- Where do developers work?
- Configuration management?
- Source control? Baseline?
- Transition? Periodic?
Release - Smoke tests?
Cycle - Owners:Dev IT DBA SQA?
© Robert Sabourin, 2003 Slide 32
AmiBug.Com, Inc.
 Getting Things Done
Concern Being Prepared!

- Information Flow Corporate information Key business drivers Sales Market Finance

- Technology Flow Architecture Technology churn Tools Techniques Training

- Requirement Flow Defined Understood Interrupt Poll Prioritize Turbulence Status Truffle

- Bug Flow Defined Understood Business Technical Efficient Expedient Reassess

- Test Objectives Quality Factors Technical Risk Failure Modes Importance

- Test Strategy Plan Analytic Exploratory Checklists Parallel Chunking Scenarios Data

- Test Organization Scheduling Staffing Outsourcing Contractors Students

- Testing Lab Multi-tier Server Client Platforms Swap Pattern Synchronized

- Test Status Bug charts Test Plan Elaboration Status Pass Fail Execution Status

June 21, 2021 © Robert Sabourin, 2003 Slide 33

AmiBug.Com, Inc.
 Getting Things Done
Where tests are run!
During development task assignments
Developers Desktop Unit Testing

During development and integration task assignments

Development Servers Unit and Integration testing

During Integration and system test assignments

Test Lab Servers Integration and System Testing

Acceptance testing
Staging Servers As close as possible to live technologies

Operational system
Live Servers Site monitoring

June 21, 2021 © Robert Sabourin, 2003 Slide 34

AmiBug.Com, Inc.
 Getting Things Done
Where tests are run!

Developers Desktop - Synchronize releasing

builds from the development
Development Servers
team to the testing team!
Test Lab Servers
- Ensure a new builds passes
Staging Servers the Smoke Test before it
replaces the build currently
Live Servers being tested!

Never update servers when tests are running!

June 21, 2021 © Robert Sabourin, 2003 Slide 35
AmiBug.Com, Inc.
 Getting Things Done
Test Scheduling
Revised risks?
Adapt to change New test objectives? New chunks?
DAILY

Assign testing chunks to testing team members.

Triage Testing Chunks Analysis and exploration to more senior team members

Relative business importance of testing objective?

Prioritize Bugs Any test objective more important than any other?

Total budget effort

Track Progress Spread across testing objectives

Should the new build be tested at all?

Smoke Test On failure continue with previous build in test.

Each area of functionality has a simple test.

FAST Test Is functional area stable enough to test
BUILD

Does application still work as expected?

Regression Test Did we accidentally break something?

Have bugs really been fixed?

Confirmation Test Double check in test lab for each bug!

How well does the application behave in harsh conditions?

Stress Testing Treated as an experiment.
June 21, 2021 © Robert Sabourin, 2003 Slide 36
AmiBug.Com, Inc.
 Smoke Testing
• Smoke test is run on a
new build of software to
make sure all functions
operate well enough to
continue testing
• “Turn on a new
appliance at the store”

June 21, 2021 © Robert Sabourin, 2003 Slide 37

AmiBug.Com, Inc.
 FAST Testing
• Functional Acceptance
Simple Tests
– Wide in breadth, low
in depth
– Exercise every
function of the
application at least
once

June 21, 2021 © Robert Sabourin, 2003 Slide 38

AmiBug.Com, Inc.
 Regression Testing

• Previously executed tests are re-executed

against a new version of the application
– have code changes broken something that
used to work
– have we introduced new defects
– often automated

June 21, 2021 © Robert Sabourin, 2003 Slide 39

AmiBug.Com, Inc.
 Confirmation Testing

• Typically:
– Tester confirms that the
fixed bug is really fixed in
the appropriate software
build

June 21, 2021 © Robert Sabourin, 2003 Slide 40

AmiBug.Com, Inc.
 Getting Things Done
Testing Activities
Define test procedure and test cases.
Elaboration Methods, techniques, test cases. All must be repeatable.

Primarily to identify areas of weakness or instability!

Exploration Use exploratory testing techniques.
PLATFORM

Follow defined test procedures or execute automated test scripts.

Execution While testing identify bugs! (use checklists!)

Narrow down how to repeat bug. Be practical.

Bug Isolation May need triage to determine if more isolation is needed.

Track defects Open open over time. (TREND CURVE)

Track Progress Track test chunk status over time. (SPREED SHEET)

Execute in collaboration with development.

Stress Test Experiment As required report bugs and always report stress test results!

June 21, 2021 © Robert Sabourin, 2003 Slide 41

AmiBug.Com, Inc.
 Just In Time Testing

Exploratory Testing

June 21, 2021 © Robert Sabourin, 2003 Slide 42

AmiBug.Com, Inc.
 Getting Things Done

• Wisdom
– Exploratory Testing in parallel with
elaboration of chunks
– Elaborate higher priority chunks first
– Executing chunks as they are elaborated
– DO NOT WAIT FOR A COMPLETE SET

June 21, 2021 © Robert Sabourin, 2003 Slide 43

AmiBug.Com, Inc.
 Exploratory Testing

• Approach formalized by
James Bach
(www.satisfice.com)
• Used in General
Functionality and Stability
Test Procedure for
Windows 2000 Application
Certification

June 21, 2021 © Robert Sabourin, 2003 Slide 44

AmiBug.Com, Inc.
 Mandate to explore

William Clark Meriwether Lewis

The object of your mission is to explore the Missouri river,

& such principal streams of it, as, by its course and communication
with the waters of the Pacific ocean...may offer the most direct &
practicable water communication across this continent for the
purposes of commerce.

- Thomas Jefferson's letter to Meriwether Lewis, June 1803

June 21, 2021 © Robert Sabourin, 2003 Slide 45
AmiBug.Com, Inc.
 Make intelligent
decisions
Take notes
about your
decisions
Map out
where you
have been
Others can
use the
June 21, 2021 © Robert Sabourin, 2003 result Slide 46
AmiBug.Com, Inc.
 Chart as you explore
Further
exploration
yields a
good idea of
the state of
the world!

One bit at a
time

June 21, 2021 © Robert Sabourin, 2003 Slide 47

AmiBug.Com, Inc.
 Exploration Notes

- Tabular
- Chronological
- Schematic
- Point form
- Concise

June 21, 2021 © Robert Sabourin, 2003 Slide 48

AmiBug.Com, Inc.
 Exploratory Testing
• Test cases
– Not known in advance
– Defined & executed “on the fly” while you learn about the
product
• Testers need to “hone up” their skills in
making maps!
– Consistent note taking style
– Practice
• We need to have a standard template
– Uniform way to capture details

June 21, 2021 © Robert Sabourin, 2003 Slide 49

AmiBug.Com, Inc.
 Exploratory Testing
• During test we must capture
– Function, options or sub-functions being explored
– Test cases attempted
– Comments, notes, images or attachments
– Hints, reminders and observations which may be useful to
future testers
– Date, Platform, Build or Configuration under test
– Name of person running test
– Oracles, “strategy to assess correctness”
– Other relevant details

June 21, 2021 © Robert Sabourin, 2003 Slide 50

AmiBug.Com, Inc.
 An Exploratory Test
Process
Confirm Test Objective
Ensure context known

Ensure HW and SW OK
All tools available
Kick Off
Chunk of 90 to 120 min
Test, Plan, Discover
Prepare
Wrap up
Run Collect all notes data

Review results with

Complete Test Lead

Review
Reassess goals
Piece together map Follow Up
June 21, 2021 © Robert Sabourin, 2003 Slide 51
AmiBug.Com, Inc.
 Just In Time Testing

Scenarios

June 21, 2021 © Robert Sabourin, 2003 Slide 52

AmiBug.Com, Inc.
 Scenario Based
Testing
• Scenarios
– Your testing department should develop a series of typical, real, usage
scenarios for each user type
• Can be done in parallel with development effort
• Could be based on Use Case Analysis
• Could be based on Story Boards
– Testing chunks can be dedicated to scenario elaboration or execution
• Elaboration (scripting) by testing professionals
• Execution by contractors, students, support staff etc
• Lots of parallelism!
– Individual scenarios can cover many functions!

June 21, 2021 © Robert Sabourin, 2003 Slide 53

AmiBug.Com, Inc.
June 21, 2021 © Robert Sabourin, 2003 Slide 54
AmiBug.Com, Inc.
June 21, 2021 © Robert Sabourin, 2003 Slide 55
AmiBug.Com, Inc.
June 21, 2021 © Robert Sabourin, 2003 Slide 56
AmiBug.Com, Inc.
 Scenario Based
Testing
• User Experience Test Case
– Parameterize experience
– Walk through scenario from start to end
– Use pre selected input for each case
– Always run every test as if it were a user experience

June 21, 2021 © Robert Sabourin, 2003 Slide 57

AmiBug.Com, Inc.
 Just In Time Testing

Status

June 21, 2021 © Robert Sabourin, 2003 Slide 58

AmiBug.Com, Inc.
 JIT Status
• Reporting Status
– Test Objectives
– Test Results
– Scoreboard
– Bug Summary
– What we know
– What is left to do

June 21, 2021 © Robert Sabourin, 2003 Slide 59

AmiBug.Com, Inc.
 JIT Status
Build Chunk Execution Status
Test Objective Chunk Elaborated Go A B C D E F G H I J Latest Runs
Overall Stability OS001 NO FAIL FAIL PASS FAIL FAIL 3
Overall Stability OS002 YES FAIL FAIL FAIL 1
Overall Stability OS003 NO FAIL FAIL 0
Overall Stability OS004 NO FAIL FAIL 0
Overall Stability OS005 YES FAIL PASS PASS 1
Typical End User Usage Scenarios TE001 YES FAIL PASS PASS 1
Typical End User Usage Scenarios TE002 YES FAIL FAIL PASS PASS 2
Typical End User Usage Scenarios TE003 NO FAIL FAIL 0
Typical End User Usage Scenarios TE004 NO FAIL FAIL 0
Typical End User Usage Scenarios TE005 YES FAIL FAIL FAIL 1
Typical End User Usage Scenarios TE006 NO FAIL FAIL 0
Typical Administrator Usage Scenarios TA001 YES FAIL FAIL FAIL 1
Typical Administrator Usage Scenarios TA002 NO FAIL FAIL 0
Typical Administrator Usage Scenarios TA003 YES FAIL FAIL FAIL 1
Typical Administrator Usage Scenarios TA004 YES FAIL PASS PASS 1
Typical Administrator Usage Scenarios TA005 NO FAIL FAIL 0
Typical Administrator Usage Scenarios TA006 NO FAIL FAIL 0
Typical Administrator Usage Scenarios TA007 NO FAIL FAIL 0
Typical Administrator Usage Scenarios TA008 NO FAIL FAIL 0
Basic Operation of all Functions BO001 YES FAIL PASS PASS PASS 2
Data Integrity DI001 NO FAIL FAIL 0
Data Integrity DI002 YES FAIL FAIL PASS PASS 2
Data Integrity DI003 NO FAIL FAIL 0
Data Integrity DI004 YES FAIL PASS FAIL PASS FAIL FAIL 4
Data Integrity DI005 YES FAIL PASS PASS 1
Data Integrity DI006 NO FAIL FAIL 0

June 21, 2021 © Robert Sabourin, 2003 Slide 60

AmiBug.Com, Inc.
 JIT Status

Statistics Total Percent

Number System Test Chunks 26 100%
Number of System Test Chunks Elaborated 12 46%
Number of System Test Chunks To Elaborate 14 54%
Number of System Test Chunks PASS 7 27%
Number of System Test Chunks FAIL 19 73%

June 21, 2021 © Robert Sabourin, 2003 Slide 61

AmiBug.Com, Inc.
 JIT Status
Test Lead Must Track Efforts Every Day!

Number of Tests to elaborate?

Number of Tests Passed?
Number of Tests Failed?
Number of High Priority Bugs to be fixed?
Number of Bug Fixes to confirm?

Count every day and plot trend graphs!

Publish results.
June 21, 2021 © Robert Sabourin, 2003 Slide 62
AmiBug.Com, Inc.
 100
150
200
250

50

0
01/11/98

01/18/98

01/25/98
02/01/98

02/08/98

02/15/98

02/22/98

03/01/98
03/08/98

June 21, 2021

03/15/98

03/22/98

03/29/98

04/05/98

Open Bugs
Trend Chart
04/12/98
04/19/98

04/26/98

05/03/98
05/10/98

05/17/98

05/24/98

05/31/98
06/07/98

06/14/98

06/21/98

06/28/98

07/05/98

Open P1 & P2 Bugs

07/12/98

07/19/98

07/26/98

08/02/98

08/09/98

08/16/98

08/23/98

© Robert Sabourin, 2003

08/30/98

09/06/98

09/13/98

Bugs Pending validation (Corrigé)

09/20/98

09/27/98

10/04/98

10/11/98

10/18/98
10/25/98

11/01/98
JIT Status

11/08/98

11/15/98
11/22/98

11/29/98

12/06/98

12/13/98

01/14/99

01/18/99
Slide 63

01/25/99

02/01/99

02/08/99
AmiBug.Com, Inc.
 JIT Status

Trend Chart
Open Bugs By Type

June 21, 2021 © Robert Sabourin, 2003 Slide 64

AmiBug.Com, Inc.
 Finished?

How do you know you are finished?

June 21, 2021 © Robert Sabourin, 2003 Slide 65

AmiBug.Com, Inc.
 You know you are
finished when
…
… the only bugs left are the ones are
acceptable (based on your objective
test team input) ...

June 21, 2021 © Robert Sabourin, 2003 Slide 66

AmiBug.Com, Inc.
 You know you are
finished when
…
… the only bugs left are the ones are
acceptable (based on your objective
test team input) ...

At least for now!

June 21, 2021 © Robert Sabourin, 2003 Slide 67
AmiBug.Com, Inc.
 Thank You

• Questions?

June 21, 2021 © Robert Sabourin, 2003 Slide 68

AmiBug.Com, Inc.