Antecedent Analytical Variance Factors in

Qualitative Bowtie Risk Analysis

Phillip McKenzie

Research oral presentation submitted in fulfilment Part 1 of the

Degree of Master of Risk Management by Research

1
 Research problem
 Literature review
 What is bowtie analysis
 What is analytical variance
 Analytical variance sources and types
 Interim literature review findings
 Research objectives
 Research methods and tools

Agenda
2
Companies routinely apply risk assessment tools
and methodologies in their risk management
systems. One methodology that is growing in use
is qualitative bowtie analysis.

It has been observed that qualitative bowtie

analysis often produces inconsistent analytical
results (‘analytical variance’).

This is concerning as it calls into question the

reliability and validity of the methodology.

Problem
3
Qualitative bowtie analysis
ISO 2000
4
Quantitative bowtie analysis
ISO 2000
5
The research premise is that given a common
‘analytical subject’; and a common ‘analytical
process’; you would expect consistent ‘analytical
results’ across multiple analyses.

However, in practice the analytical results appear to

be highly at variance from each other.

Analytical Analytical Analytical

Subject Process Results

Simplified model of the analytical process

6
A literature review was performed on ‘qualitative
bowtie analysis’ and ‘analytical variance’.

This explored the factors occurring throughout the

risk analysis process which may be sources of the
observed analytical variance.

The findings of the literature review has produced a

typology of analytical variance which demonstrates
the sources of analytical variance and the related
types of variance factors.

Literature review
7
No. Analytical Element Common Analytical Element Synonyms

1 Hazard threat energy

2 Top Event hazardous event

3 Causes mechanisms, threats

4 Outcomes consequences

5 Controls barriers, safeguards, defences, mitigations

6 Defeating factors escalation factors, preconditions, active failures

7 Defeating factor controls escalation factor controls

Typical qualitative bowtie analysis in practice

8
Qualitative bowtie analysis employs an
epidemiological modelling approach. This
approach is associated with high information
complexity arising from barrier ‘control’ analysis.

Model type Search principle Analysis goals Example

Specific causes and well- Eliminate or contain Linear chain of events
Sequential
defined links causes domino, Trees / networks
Latent conditions,
Carriers, barriers, and Make defences and
Epidemiological Carrier-barriers,
latent conditions barriers stronger
Pathological systems
Control theory models,
Tight couplings and Monitor and control
Systemic Chaos models,
complex interactions performance variability
Stochastic resonance

Evolution of accident models

Hollnagel & Goteman 2004
9
Variance: “the fact or quality of being different,
divergent, or inconsistent” Oxford University Press 2014

Variance is the actual state of difference between

two or more things. The term ‘analytical variance’
therefore refers to the inconsistent results of
multiple comparative analyses.

In the case of this research, analytical variance is

the inconsistency observed in the analytical results
of qualitative bowtie analysis.

Analytical variance
10
 hazard

 Omissions of relevant analytical elements

top
 Inclusions of irrelevant analytical elements event

 Differences in characterisations of the same cause

analytical elements
 Differences in classifications of the same outcome

analytical elements
defeating
factor
 Differences in relationships between the
analytical elements
control

Observed analytical variance manifestation in

qualitative bowtie analysis 11
Analytical variance is discussed in the literature as
resulting from either ‘uncertainty’ or ‘variability’; with
uncertainty being the most prevalent term used.

A literature survey of uncertainty and variability

typologies in the domain of risk analysis was
undertaken.

This revealed a very wide spectrum of typologies

which use divergent terminology and describe
many different ‘types’ of uncertainty and variability.

Variance typologies
ANS and IEEE 1983; Ferson & Ginzburg 1996; Regan, Colyvan & Burgman 2002; Carey &
Burgman 2008; Markowski, Mannan & Bigoszewska 2009; Ferdous et al. 2012; Shahriar, Sadiq &
Tesfamariam 2012; Ferdous et al. 2013 12
 Subjective judgement

· Variability · Linguistic uncertainty

Naturally occurring, unpredictable Ambiguity, Vagueness, Underspecificity,
Carey & Burgman change Context dependence
2008 · Incertitude
Lack of model parameter knowledge,
Lack of model relationship knowledge
Variability Uncertainty Variability Uncertainty
(Aleatory Uncertainty) (Epistemic Uncertainty) (Aleatory Uncertainty) (Epistemic Uncertainty)
Variability Uncertainty · Objective uncertainty · Completeness uncertainty · Subjective uncertainty
(Aleatory Uncertainty) (Epistemic Uncertainty) Variability, Random behaviour Have all significant phenomena and Vagueness in interpretation
relationships been considered
Markowski, Mannan & · Subjective uncertainty
Lack of knowledge · Modelling uncertainty
Bigoszewska 2009 Analytical Subject Analytical
Analytical Subject Analytical Methodology Human Analysts Inadequacies andMethodology
deficiencies in Human Analysts
· Parameter uncertainty
Analytical Subject Analytical Methodology Human Analysts formulation of accident scenario
Imprecise data, Vague data, Missing
structure
data, Inadequate data
· Data (parameter) uncertainty · Completeness uncertainty · Model uncertainty · Data (parameter) uncertainty · Completeness uncertainty · Model uncertainty
Amount of data, Diversity of data List of initiating events, system failure Skill and accuracy of analyst, Amount of data, Diversity of data List of initiating events, system failure Skill and accuracy of analyst,
· Data (parameter)
sources, Accuracy uncertainty
of data sources · Completeness uncertainty
contributors, accident sequence, · Model uncertainty
Misapplication of method rules · Aleatory uncertainty
sources, Accuracy (variation)
of data sources contributors, accident sequence, Misapplication of method rules
Amount of data, Diversity of data List of initiating
definition events,
of system system
damage failure
states, list Skill and accuracy of analyst, Stochastic, Objective, Irreducible, definition of system damage states, list
ANS and IEEE 1983 sources, Accuracy of data sources contributors,
of system interactions,sequence,
accident accounting of Misapplication of method rules ANS and IEEE 1983 Random of system interactions, accounting of
definition of system damage states, list Ferdous et al. 2012
ANS and IEEE 1983 human factors · Epistemic uncertainty (knowledge) human factors
of system interactions, accounting of
· Model Imprecise, Incomplete, Ambiguous, · Model uncertainty
human uncertainty
factors
Limitations of binary logic models Ignorance, Inconsistent, Vague Limitations of binary logic models
· Model uncertainty
Limitations of binary logic models
· Variability (objective uncertainty) Shahriar, Sadiq & ·· Data uncertainty
Variability (epistemic)
(objective uncertainty) · Model uncertainty
Heterogeneity, stochasticity Impreciseness, stochasticity
Heterogeneity, Vagueness, Lack of Interdependency of event relationships
· Variability (objective uncertainty) Tesfamariam 2012
Ferson & Ginzburg Ferson & Ginzburg knowledge, Incompleteness
· Heterogeneity,
Ignorance (epistemic uncertainty)
stochasticity · Ignorance (epistemic uncertainty)
1996 1996
Ferson & Ginzburg Systematic measurement error, Systematic measurement error,
· Ignorance
incomplete (epistemic
information uncertainty) · incomplete information
1996 Aleatory uncertainty · Model uncertainty
Systematic measurement error, Natural variation, Random behaviour of Model adequacy, Mathematical and
incomplete information a system numerical approximations in the model,
· Epistemic uncertainty · Epistemic uncertainty · Linguistic uncertainty · Epistemic uncertainty · Epistemic
Assumptions uncertainty
or validation of the model · Linguistic uncertainty
Measurement error, Systematic error, Model uncertainty Vagueness, Context dependence, · Measurement
Epistemic uncertainty
error, Systematic error, Model uncertainty Vagueness, Context dependence,
· Epistemic uncertainty · Epistemic uncertainty · Linguistic uncertainty Regan, Colyvan & Lack of variation,
knowledge, Incompleteness · Quality uncertainty
Regan, Colyvan & Natural variation, Inherent randomness Ambiguity, Underspecificity, Natural Inherent randomness Ambiguity, Underspecificity,
Measurement error, Systematic error, Model uncertainty Vagueness,
IndeterminacyContext dependence,
of theoretical terms Error in hazard identification, Indeterminacy of theoretical terms
Burgman 2002 Burgman
Ferdous 2002
et al. 2013 · Data uncertainty
Regan, Colyvan & Natural variation, Inherent randomness Ambiguity, Underspecificity, Incomplete, Inconsistent or imprecise
Incorrectness in identification of
· Indeterminacy
Epistemic uncertainty
of theoretical terms consequences and their interactions · Epistemic uncertainty
Burgman 2002 data, Missing or unavailable data, Multi-
Subjective judgement Subjective judgement
· Epistemic uncertainty source data, Vagueness or inadequacy
Subjective judgement in input data
· Variability · Linguistic uncertainty ·· Variability
Quality uncertainty · Linguistic uncertainty
Naturally occurring, unpredictable Ambiguity, Vagueness, Underspecificity, Naturally occurring,
Knowledge unpredictable
deficiency about a system Ambiguity, Vagueness, Underspecificity,
· Variability · Linguistic uncertainty change Context dependence
Carey & Burgman change Context dependence Carey & Burgman
Naturally occurring, unpredictable Ambiguity, Vagueness, Underspecificity,
2008 · Incertitude
change Context dependence
2008 · Incertitude
Carey & Burgman
Lack of model parameter knowledge, Lack of model parameter knowledge,
2008 · Incertitude
Lack of model relationship knowledge Lack of model relationship knowledge
Lack of model parameter knowledge,
Lack of model relationship knowledge
· Objective uncertainty · Completeness uncertainty · Subjective uncertainty · Objective uncertainty · Completeness uncertainty · Subjective uncertainty
Variability, Random behaviour Have all significant phenomena and Vagueness in interpretation Variability, Random behaviour Have all significant phenomena and Vagueness in interpretation
· Objective uncertainty · Completeness uncertainty
relationships been considered · Subjective uncertainty relationships been considered
Markowski, Mannan & · Variability,
SubjectiveRandom
uncertainty
behaviour Have all significant phenomena and Vagueness in interpretation Markowski, Mannan & · Subjective uncertainty
Lack of knowledge · relationships
Modelling uncertainty
been considered Lack of knowledge · Modelling uncertainty
Bigoszewska 2009 · Subjective uncertainty Bigoszewska 2009
Markowski, Mannan & · Lack
Parameter uncertainty
Inadequacies and deficiencies in · Parameter uncertainty
Inadequacies and deficiencies in
of knowledge · Modelling
formulationuncertainty
of accident scenario formulation of accident scenario
Bigoszewska 2009 Imprecise data, Vague data, Missing Imprecise data, Vague data, Missing
Inadequacies
structure and deficiencies in structure
· Parameter uncertainty
data, Inadequate data data, Inadequate data
formulation of accident scenario
Imprecise data, Vague data, Missing
structure
data, Inadequate data
· Aleatory uncertainty (variation) · Aleatory uncertainty (variation)
Stochastic, Objective, Irreducible, Stochastic, Objective, Irreducible,
· Aleatory
Random uncertainty (variation) Random
Ferdous et al. 2012 Stochastic, Objective, Irreducible, Ferdous et al. 2012
· Random
Epistemic uncertainty (knowledge) · Epistemic uncertainty (knowledge)
Ferdous et al. 2012 Imprecise, Incomplete, Ambiguous, Imprecise, Incomplete, Ambiguous,
· Epistemic
Ignorance, uncertainty
Inconsistent, (knowledge)
Vague Ignorance, Inconsistent, Vague
Imprecise, Incomplete, Ambiguous,
Ignorance, Inconsistent, Vague
· Data uncertainty (epistemic) · Model uncertainty Shahriar, Sadiq & · Data uncertainty (epistemic) · Model uncertainty
Shahriar, Sadiq &
Impreciseness, Vagueness, Lack of Interdependency of event relationships Impreciseness, Vagueness, Lack of Interdependency of event relationships
Tesfamariam 2012 · Data uncertainty (epistemic) · Model uncertainty Tesfamariam 2012
Shahriar, Sadiq & knowledge, Incompleteness knowledge, Incompleteness
Impreciseness, Vagueness, Lack of Interdependency of event relationships
Tesfamariam 2012
knowledge, Incompleteness
· Aleatory uncertainty · Model uncertainty · Aleatory uncertainty · Model uncertainty
Natural variation, Random behaviour of Model adequacy, Mathematical and Natural variation, Random behaviour of Model adequacy, Mathematical and

Summary of uncertainty and variability typologies

Ferdous et al. 2013
· Aleatory
a system uncertainty
Natural variation, Random behaviour of
· Epistemic
a system uncertainty
Lack of knowledge, Incompleteness
· Epistemic uncertainty
· Lack
Data of
uncertainty
knowledge, Incompleteness
· Model
Model
uncertainty
numerical approximations in the model,
adequacy,
Assumptions Mathematical
or validation of theand
model
numerical approximations in the model,
· Assumptions
Quality uncertainty
or validation of the model
Error in hazard identification,
· Quality uncertainty Ferdous et al. 2013
a system
· Epistemic uncertainty
Lack of knowledge, Incompleteness
· Data uncertainty
numerical approximations in the model,
Assumptions or validation of the model
· Quality uncertainty
Error in hazard identification,
Incorrectness in identification of Incorrectness in identification of
Incomplete, Inconsistent or imprecise Incomplete, Inconsistent or imprecise

in risk analysis
Error in hazard and
consequences identification,
their interactions consequences and their interactions
Ferdous et al. 2013 · Data
data, uncertainty
Missing or unavailable data, Multi- data, Missing or unavailable data, Multi-
Incorrectness in identification of
Incomplete,
source data,Inconsistent
Vagueness or or imprecise
inadequacy source data, Vagueness or inadequacy
consequences and their interactions
data, Missing
in input data or unavailable data, Multi- in input data
source data, Vagueness or inadequacy
· in
Quality uncertainty
input data
Knowledge deficiency about a system
· Quality uncertainty
Knowledge deficiency about a system
13
· Quality uncertainty
Knowledge deficiency about a system
 Analytical subject
(knowledge, complexity, randomness)

 Analytical methodology
(elements, terminology, format, rules, tools)

 Human analysts
(language, skill, experience, cognition)

Variance sources
14
 Knowledge amount (inadequate, source diversity)
 Knowledge accuracy (errors, imprecise, inconsistent)
 Knowledge completeness (missing, ignorance, incomplete)
 Knowledge clarity (vague, ambiguous)
 Subject randomness (stochasticity, natural variation, unpredictability)
 Subject complexity (heterogeneity, irreducibility)

Variance source: analytical subject

15
Socio-Technical Systems (subject complexity)
Bostrom & Heinen 1977
 Bases (1) Authority (1) Cost (3) Acceptability (4) Equity (4) Consequences (4) Alternatives (4)

Control evaluation (2) Compatibility (4)

Context Type (1)

Selection Decision Efficiency (5)

Stakeholders (1)

3 Robustness (5)
Risk Aversion (1)

Selection Context 1 Control 4 Operating Status Dependencies (5)

Risk Types (2)

Define the context (2) 2 (2) Monitor and review

Maintainability (6)

Risk Level (2)

Operating Effect Ownership (6)

Risk Targets (4) Control analysis (2) Survivability (6)

Functionality (5) Availability (5) Reliability (5) Adequacy (5) Specificity (5) Objective Class (5) Means Class (5)

Model of control analysis (subject complexity)

UKOOA 1999; ISO 2009a; ISO 2009b; Standards Australia 2004; Sklet 2006; NOPSEMA 2014
17
 Elements (hazards, top-events, causes, controls, outcomes, defeating factors)
 Terminology (element definitions, element names, element characteristics)
 Format (structure, graphical presentation)
 Rules (logic, element identification criteria, element classification criteria)
 Tools (software, formulae)
 Propagation

Variance source: analytical methodology

18
 Language - ambiguity
 Language - vagueness
 Language - underspecificity
 Language - context dependence
 Human performance - skill
 Human performance - experience
 Human performance - cognition

Variance source: human analysts

19
 20
improvisation
No rules

Mistake
Correct

Mispliance
Bad rules

violation
Correct
Good rules

compliance

Misvention
Correct

Necessary
Violations

Optimising
performance

performance
Erroneous
Correct

Routine
Intended Actions

Knowledge Based Bad Rule Applied

Mistakes

Rule Based Good Rule Not Applied

Good Rule Misapplied
Error

Retrieval Failures
Lapses

Memory Failures Storage Failures

Input Failures
Unintended Actions

Reason 1990, 1997, 2008

Human error
Interference
Attention Failures
Slips Strong Habit Intrusion
Wrong Detections
Recognition Failures Non-detections
Misidentification

· Knowledge - amount
· Knowledge - accuracy
· Knowledge - completeness
· Knowledge - clarity
· Variability - randomness
· Variability - complexity
Analytical Subject
Variability
(Aleatory Uncertainty)
Typology of analytical variance sources and factors
· Language - ambiguity
· Limits - elements · Language - vagueness
· Limits - terminology · Language - underspecificity
· Limits - format · Language - context dependence
· Limits - rules · Performance - skill
· Limits - tools · Performance - experience
· Propagation · Performance - cognition
Analytical Methodology Human Analysts
Uncertainty
(Epistemic Uncertainty)
Analytical Variance
21
In addition to a new analytical variance typology; a
systems based conceptual model of the process
leading to analytical variance has been developed.

This conceptual model is used to demonstrate

where in the analytical process the variance
sources and the variance factor types occur; and
how they interact with each other to produce the
observed analytical variance.

Interim results
22
 Analytical Process
Control
· Limits - elements
· Limits - terminology
Analytical · Limits - format
· Limits - rules
Methodology · Limits - tools
· Variance propagation

Variance Methodology
Propagation Limits

Input Output
Knowledge
Analytical Uncertainty Human Analytical
Human
Subject Analyst Error Result
Knowledge
Variability
Processing
· Knowledge - amount
· Knowledge - accuracy
· Knowledge - completeness Analytical
· Language - ambiguity
· Knowledge - clarity
· Language - vagueness Variance
· Variability - randomness
· Language - underspecificity
· Variability - complexity
· Language - context dependence
· Performance - skill
· Performance - experience
· Performance - cognition

Systems based model of the process leading to

analytical variance 23
 Objective 1: To determine if the anecdotally
observed analytical variance in qualitative bowtie
analysis is consistent with the experiences of experts
in the field.

 Objective 2: To identify and describe the antecedent

factors inherent in the qualitative bowtie analysis
process which cause the observed analytical
variance.

 Objective 3: To determine which antecedent factors

are significant causes of the analytical variance.

Research objectives
24
 Participant sampling (snowballing)
 In-depth participant interviews
 Data coding of interviews
 Electronic survey (Likert & ranking)
 Descriptive statistical analysis results
 Identification of factors for future research

Research methods and tools

25
As this research involves interviews and electronic
surveys gather data, it is subject to SUHERC
approval; which is still pending.

Based on the Swinburne published risk criteria for

human research, the research is considered to be
low risk.

The necessary research information statement and

consent instruments have been prepared for the
research.

Human research ethics

26
  Experience Contextualisation
General demographical data from the expert
participant and how they are involved in the
field of qualitative bowtie analysis.

 Apprehending the phenomenon

(modes of appearing): This is based on the
experiential data relevant to the participant
and the phenomena as they appear to the
participant.

 Clarifying the phenomenon

This section of the interview calls for the use
of “imaginative variation” in exploring the
phenomenon beyond what is initially offered
by the participant.

Research interview protocol

Bevan 2014
27
Data coding scheme
Bouma & Ling 2005
28
 Bouma, G & Ling, R 2005, The research process, 5th Ed., Oxford
University Press, South Melbourne, Australia.
 Bevan, MT 2014, “A method of phenomenological interviewing”,
Qualitative health research, vol. 24, no. 1, pp. 136–44.
 ISO 2000, “17776 Petroleum and natural gas industries — Offshore
production installations — Guidelines on tools and techniques for hazard
identification and risk assessment,” International Standards Organisation, 
Geneva, Switzerland.
 ISO 2009a, “31000 Risk management — Principles and guidelines,”
International Standards Organisation, Geneva, Switzerland.
 ISO 2009b, “31010 Risk management — Risk assessment techniques,” 
International Standards Organisation, Geneva, Switzerland.
 Hollnagel, E & Goteman, O 2004, “The functional resonance accident
model,” Linköping, viewed 5 April, 2014,
<http://www.skybrary.aero/bookshelf/books/403.pdf>.
 ANS and IEEE 1983, PRA Procedures Guide: a guide to the performance
of probabilistic risk assessments for nuclear power plants, NUREG/CR-
2300., U.S. Nuclear Regulatory Commission., Washington, D.C.
 Ferson, S & Ginzburg, LR 1996, “Different methods are needed to
propagate ignorance and variability,” Reliability Engineering & System
Safety, vol. 54, no. 2-3, pp. 133–144.
 Regan, HM, Colyvan, M & Burgman, MA 2002, “A taxonomy and
treatment of uncertainty for ecology and conservation biology,” Ecological
Applications, vol. 12, pp. 618–628.
 Carey, JM & Burgman, M a 2008, “Linguistic uncertainty in qualitative risk
analysis and how to minimize it.,” Annals of the New York Academy of
Sciences, vol. 1128, pp. 13–17.
 Markowski, AS, Mannan, MS & Bigoszewska, A 2009, “Fuzzy logic for
process safety analysis,” Journal of Loss Prevention in the Process
Industries, vol. 22, no. 6, pp. 695–702.
 Ferdous, R, Khan, F, Sadiq, R, Amyotte, P & Veitch, B 2012, “Handling
and updating uncertain information in bow-tie analysis,” Journal of Loss
References
Prevention in the Process Industries, vol. 25, no. 1, Elsevier Ltd, pp. 8–
19.
 Shahriar, A, Sadiq, R & Tesfamariam, S 2012, “Risk analysis for oil & gas
pipelines: A sustainability assessment approach using fuzzy based bow-
tie analysis,” Journal of Loss Prevention in the Process Industries, vol.
25, no. 3, Elsevier Ltd, pp. 505–523.
Ferdous, R, Khan, F, Sadiq, R, Amyotte, P & Veitch, B 2013, “Analyzing
system safety and risks under uncertainty using a bow-tie diagram: An
innovative approach,” Process Safety and Environmental Protection, vol.
91, no. 1-2, Institution of Chemical Engineers, pp. 1–18.
Bostrom, R & Heinen, J 1977, “MIS problems and failures: A socio-
technical perspective,” MIS quarterly, no. September, pp. 17–33.
 UKOOA 1999, Industry guidelines on a framework for risk related
decision support, 1st ed, United Kingdom Offshore Operators
Association.
Standards Australia 2004, Handbook HB 436:2004 Risk Management
Guidelines Companion to AS/NZS 4360:2004, Standards Australia
International Ltd, Sydney.
 Sklet, S 2006, “Safety barriers: Definition, classification, and
performance,” Journal of Loss Prevention in the Process Industries, vol.
19, no. 5, pp. 494–506.
 NOPSEMA 2014, “Control Measures and Performance Standards,”
National Offshore Petroleum Safety and Environmental Management
Authority, viewed 5 April, 2014,
<http://www.nopsema.gov.au/assets/Guidance-notes/N-04300-GN0271-
Control-Measures-and-Performance-Standards.pdf>.
 Reason, J 1990, Human error, Cambridge University Press, Cambridge.
 Reason, J 1997, Managing the Risks of Organizational Accidents,
Reason, Ashgate Publishing Company, Aldershot.
 Reason, J 2008, The human contribution: unsafe acts, accidents and
heroic recoveries, Ashgate Publishing, Ltd.
29