Risk Analysis, Vol. 39, No. 5, 2019 DOI: 10.1111/risa.

13236

Rethinking Risk Assessment for Public Utility

Safety Regulation

Carl Danner1 and Paul Schulman2,∗

To aid in their safety oversight of large-scale, potentially dangerous energy and water infras-
tructure and transportation systems, public utility regulatory agencies increasingly seek to use
formal risk assessment models. Yet some of the approaches to risk assessment used by utilities
and their regulators may be less useful for this purpose than is supposed. These approaches
often do not reflect the current state of the art in risk assessment strategy and methodology.
This essay explores why utilities and regulatory agencies might embrace risk assessment tech-
niques that do not sufficiently assess organizational and managerial factors as drivers of risk,
nor that adequately represent important uncertainties surrounding risk calculations. Further,
it describes why, in the special legal, political, and administrative world of the typical public
utility regulator, strategies to identify and mitigate formally specified risks might actually di-
verge from the regulatory promotion of “safety.” Some improvements are suggested that can
be made in risk assessment approaches to support more fully the safety oversight objectives
of public regulatory agencies, with examples from “high-reliability organizations” (HROs)
that have successfully merged the management of safety with the management of risk. Fi-
nally, given the limitations of their current risk assessments and the lessons from HROs, four
specific assurances are suggested that regulatory agencies should seek for themselves and the
public as objectives in their safety oversight of public utilities.

KEY WORDS: High-reliability organizations; risk assessment; safety management; utility regulation

1. INTRODUCTION: THE SCOPE AND SCALE
OF PUBLIC UTILITY REGULATION
The regulation of safety by governmental agen-
cies occurs in a wide variety of contexts. One of the
most challenging may be the oversight of the safety of
utilities—gas, electricity, water supply, telecommuni-
cations, and transportation systems—by public utility
regulatory agencies in the United States. These tend
to differ from other health and safety agencies that
might be required to consider economic impacts in
their decisionmaking, but are not responsible for
1 Berkeley Research Group, Emeryville, CA, USA.
2 Center for Catastrophic Risk Management, University of Califor-
nia, Berkeley, CA, USA.
∗ Address correspondence to Paul Schulman, 3 Harold Drive, Mor-
aga, CA, USA; PaulSchulman@mills.edu.
broader economic, operational, or policy objectives
regarding the businesses their rules affect.3
The scale and scope of public utility oversight
includes many policy priorities of evident importance
aside from safety. These frequently include service
price regulation, upfront review and approval of
major projects or investments, control of entry into
the industry, and providing service to high-cost or fi-
nancially unattractive locations.4 Policies to promote
some new technologies (such as renewable energy or
energy-efficiency investments) and greenhouse gas
emissions abatement have also become priorities in
many jurisdictions.
3 For further review of these utility regulatory characteristics,
see Kahn, (1988), Clifton, Lanthier, and Schroter (2012), and
Gormley (1984).
4 For more detail, see Phillips (1993).

1044 0272-4332/19/0100-1044$22.00/1 
C 2018 Society for Risk Analysis
Public Utility Safety Regulation
The breadth and scale of these responsibilities
encourages utility regulators to develop a strategic
focus in safety oversight. One logical response is to
attempt risk assessments within the agency’s own
oversight procedures to help identify which available
resources and managerial effort within the utility
(the operator of the potentially hazardous facilities)
might best reduce the likelihood and severity of
adverse events. Such a formal effort to prioritize
potential problems and mitigation measures is based
on a broad foundation of rational analysis and
economic thinking, and has much to recommend it.
At the same time, the application of risk assess-
ment to safety management issues facing utilities
and regulators has been incomplete and often does
not reflect the latest advances and best practices
in the field. Risk assessment can also pose its own
risks to the successful practice and regulation of
safety management if it is not performed in a so-
phisticated manner, or not understood to require
other managerial initiatives as complementary
activities. Additionally, what risk assessment or
mitigation activities make sense for an operator
to pursue internally may not be the same as those
for a regulator to perform in its oversight role. We
highlight these issues and challenges in the analysis
that follows, and offer some explanations as to why
utilities and their regulators are susceptible to them.
Finally, we offer some suggestions for ways in which
utility regulators can enhance their oversight policies
to better advance safety objectives in the face of
challenging political environments.
We inform our analysis both by lessons learned
through the forensic study of disasters, as well as by
operational lessons that have been learned through
the experience of high-reliability organizations
(HROs). A key observation is that it may be at
least as important to consider who is attempting
to operate or oversee the safety of a complicated
system—in terms of the people, their roles and
practical realities of the organizations involved—as
it is to consider what equipment, technologies, or
investments they may use to that end.
2. CHALLENGES AND LIMITATIONS OF THE
REGULATORY AGENCY CONTEXT
The modern utility regulatory agency typically:
(1) pursues many broad and potentially conflicting
policy objectives;
1045
(2) using administrative decisionmaking proce-
dures that are often costly, slow, and inflexible;
(3) to reach decisions in a context with significant
political interests and pressures;
(4) while operating under administrative rules and
strictures that limit its operational flexibility,
and the resources it can acquire.5
All U.S. states and some territories or districts
have a utility regulatory commission of some form.6
The purposes, methods, legal procedures, and even
underlying origins of these institutions have long
been a fertile field of academic, professional, and
political interest. Any single summary would under-
state the full range of their activities, but an im-
portant common element is their oversight of the
rates (prices), range of offerings, and service quality
for certain utilities that tend to be legally franchised
monopolies—such as electricity, natural gas, or water
service.7 Some typical legal standards applicable to
this oversight include generally stated requirements
that rates be just and reasonable (and not unrea-
sonably discriminatory between customers), and that
service be provided in a safe and reliable manner.
These agencies also typically can investigate matters
of concern, punish violations of law or policy, and
make rules of general applicability for an industry.
The regulation of rate setting and service quality
for large utilities has also led logically to the over-
sight of many factors that can affect those prices
and services. Some examples include corporate
structure and mergers and acquisitions, the capital
structure and financing vehicles used by the firm,
accounting rules and practices, sources of fuel or
purchased commodities such as electricity or water,
or the terms on which utilities will interconnect or
perform wholesale business with other entities or
competitors. Environmental impacts receive agency
oversight in many instances, including through poli-
cies intended for environmental benefit rather than
reduced service costs or improved service quality. As
a general rule, conflicts are often perceived between
regulatory policy objectives, as well as between the
5 An exploration of these conditions and a suggested model for
“prudent regulators” in the face of them can be found in Beecher
(2008).
6 For a comparison of regulatory agencies and their legal and po-
litical contexts across nations, see Bignami and Zaring (2018).
7 For descriptions of these diverse priorities and their measure-
ment, see Coelli and Lawrence (2006).
1046
interests of various entities or groups affected by the
results of agency deliberations.8
U.S. regulatory agencies of all kinds have spe-
cific and often elaborate administrative procedures
used in formal decisionmaking, such as to review and
approve the rates and terms of service for particular
companies in the utilities context. The form and
substance of these procedures tend to be highly spec-
ified, as is required by federal and state law standards
for due process and transparency in agency decision-
making.9 The resulting use of rulemaking or adju-
dicatory procedures can require long lead times for
decisions, and impose their own significant litigation-
like resource demands on the agency, the regulated
entities, and interested parties who become involved.
The quality of information is also affected by the
complications that adversarial processes or political
debates tend to create. The difficulties for regulatory
agencies of obtaining relevant and current informa-
tion (or knowing whether they have received it) have
been the subject of much attention in research and
legal practice.10 The participation of interested par-
ties in adversarial processes and policy debates also
creates a strong incentive for the agency to appear
independent of the utilities, for example, by giving
them directives rather than collaborating on policy
or managerial concerns. In particular, quasi-judicial
adjudicatory procedures reinforce such incentives by
situating the regulatory agency as if it were a neutral
judge in a courtroom dealing with litigants who
happen to include the utility that is being overseen.
Regulatory agencies also are governmental
bureaucracies subject to the usual panoply of public-
sector administrative requirements, such as how they
acquire resources through a budgetary process; hire,
compensate, and dismiss employees; acquire goods
and services; and perform many other organizational
functions with lesser degrees of flexibility than
might exist for nongovernmental entities. These
requirements can tax the level of staff expertise
that agencies can maintain, particularly across many
missions at once, and their employees may have little
direct operational experience in the industries they
help oversee.
8 See Hausman and Neufeld (2012) and Clifton, Lanthier, and
Shroter (2012).
9 For example, the federal Administrative Procedure Act (APA),
5 U.S.C.A. §§ 501 et seq.
10 For an examination of the problem of “information asymmetry”
between a regulator and regulated entity, see Baldwin, Cave, and
Lodge (2013).
Danner and Schulman
The political nature of the leadership of reg-
ulatory agencies (which are usually headed by
appointed or elected commissioners) reflects the
public interest concerns involved, and can help
provide public accountability and encourage re-
sponsiveness to popular pressures and sentiments.
However, politically determined leadership may be
less likely to include subject-matter experts in fields
like safety management. Many utility regulatory
decisions are inherently political in the sense that
there is no analytical method that yields a single best
answer, and policies invariably cause differential
economic impacts on affected parties (Leone, 1986).
For all these reasons, agencies typically operate in a
milieu that is rich in political dynamics, both outside
and within the agency.11 These dynamics reinforce
incentives toward agency independence (from the
utilities), and also encourage agencies to assert that
their regulatory actions are definitive, well-founded,
and reflect the application of deep expertise they
may or may not actually possess.
The essential role of a regulatory agency is to
change the behavior of entities it oversees—from
what they otherwise would have done, to some
conduct considered to better promote a definition
of the public interest. To do so, the agency has
various tools such as issuing formal policy directives;
conducting investigations, inspections, or audits;
imposing penalties or requiring remediation of
problems; offering financial incentives; adopting
standards of operation; communicating informally
with regulated entities; issuing public statements;
developing or providing authoritative information;
and pursuing remedies through other institutions
(such as legislatures or courts).
However, while broad and potentially intrusive,
these tools do not typically include the ability to
take over a regulated entity’s operations, or the
practical capacity to become involved in an entity’s
day-to-day operations on more than an exceptional
or episodic basis. In addition, an agency’s exercise
of its capabilities and authority must occur within
the institutional context described above, that is,
while pursuing multiple and sometimes conflicting
objectives, through slow and arduous administrative
processes, with incomplete and often dated informa-
tion, while pressured by politics, and with the limits
to its resources (including expertise) and its flexibility
that governmental operating procedures impose.
11 For a classic study of regulatory politics, see Wilson (1980).
Public Utility Safety Regulation
It is also worth noting that there is a broad pro-
fessional consensus that safe operations in a complex
organization will not result solely from the top-
down imposition of operating rules or standards on
employees—either from an entity’s own leadership,
or from an outside agency.12 In other words, rules,
prescriptions, and orders alone cannot get the job
done. Strong efforts along those lines can even com-
promise safety by denying experienced operators the
latitude they need to solve complex problems (e.g.,
involving equipment, technology, unique operating
circumstances, and other “local” situations not
amenable to detailed work rules), and the ability to
use their own expertise to find effective real-time risk
mitigation approaches that cannot be specified in
advance. Similarly, managerial practices to promote
effective solutions to safety problems need to be per-
formed by those in the middle-level ranks of the reg-
ulated entity if consistent performance is the goal.13
This leads us to two important questions. First,
what does it take for a complicated organization
(like a utility) to operate safely, and how effectively
do the capabilities of a regulatory agency allow it to
oversee this operation, given the practical realities of
its institutional constraints, which might include the
overlapping authority of other public agencies? Sec-
ond, what contribution does formal risk assessment
currently make to the effectiveness of this safety
oversight and safe operations in the utilities?14 We
address these questions below.
3. SOME LIMITATIONS OF FORMAL RISK
ASSESSMENT FOR SAFETY REGULATION
We first turn to the risk assessment process, and
some of its limitations for safety oversight of complex
utilities. It is again important to note that safety is
only one of many organizational values addressed by
risk assessment in these utilities.15 Typically, separate
12 For arguments concerning safety and rules, see Hale, Borys, and
Else (2012) and Bieder and Bourrier (2013).
13 For example, the U.S. Occupational Safety and Health Adminis-
tration (OSHA) now describes important roles for middle-level
managers and supervisors in its guidelines for safety manage-
ment systems (OSHA, 2016).
14 The International Organization for Standardization (ISO) has
developed guidelines (ISO31000:2009) for risk assessment meth-
ods that are widely followed by technical organizations, includ-
ing public utilities, in their risk assessment methodology. For a
skeletal description of these guidelines, see ISO 31000:09 pre-
view (https://www.iso.org/obp/ui/#iso:std:iso:31000:ed-1:v1:en).
15 For an insightful look at “risk principles” for utility regulators,
see Beecher and Kihm (2016).
1047
assessments are conducted in risk categories such as
financial, environmental, reputational, compliance,
and legal categories, as well as a general enterprise
risk category. These may be independently per-
formed across different divisions or lines of business
for a complex utility, leading safety assessments
to have to compete with other kinds of identified
risks and their assessments for prioritization in a
risk register to guide investment planning for miti-
gation.16 Finally, the characteristics and incentives
of regulatory agencies and their environments may
necessarily put them in a disadvantageous position
to assess a utility’s risks in the same detailed manner
that its management might be able to accomplish.
3.1. Risk Versus Safety
There are significant differences between the
regulation of risk and the promotion of safety.
Generally speaking, risk refers to the inevitable
tendency of any human endeavor to produce results
that vary from those intended or expected. Risk
assessment and management then focus on how
large, likely, and consequential such variances might
be, and what might be done to reduce likelihoods
and/or their consequences. In the safety context, risk
classically has been analytically defined as a hazard
(an event or failure and the deaths, costs, and other
negative consequences it would cause) discounted
by the probability of its actual occurrence. In a more
recent glossary of terms (SRA, 2015), however, the
Society for Risk Analysis has recognized that risk
can also be a qualitative concept, offering definitions
such as: “Risk is the possibility of an unfortunate
occurrence” or “Risk is uncertainty about the sever-
ity of the consequences of an activity with respect
to something that humans value” (SRA, 2015, p. 5).
While some uncertainty can indeed be described by
a formal probability distribution (the SRA Glossary
terms this a “frequentist probability” perspective)
probability can also have different interpretations
(such as subjective probability).
Some frequentist analysts assert that a formal
probability distribution does reflect uncertainty.
However, the SRA Glossary asserts that uncertainty
for a person or a group of persons can also mean
“not knowing the true value of a quantity or the
future consequences of an activity or imperfect
or incomplete information/knowledge about a
hypothesis, a quantity, or the occurrence of an
16 See International Organization for Standardization (2009).
1048
event” (SRA, 2015, p. 6). In the SRA formulation,
in other words, risk is about more than a formal
calculation of consequences and probability. The
National Academy of Sciences (NRC, 2009) and
the U.S. Nuclear Regulatory Commission (USNRC,
2009) both suggest that risk assessments should
include the state of knowledge and uncertainty
about both likelihoods and consequences.
However, for many public utilities and their
regulators, this “epistemic” uncertainty is not fac-
tored into formal risk assessments. Rather, they use
classical definitions of risk to offer analytic preci-
sion in the allocation of resources across different
mitigation opportunities to maximize risk reduction
per dollar spent, or to optimize by spending to the
point where going further would no longer buy
added risk reduction that is judged to be worth the
cost. This precision, while often illusory because it
ignores epistemic uncertainties, is important to util-
ities to justify rate increases for safety investments
and for regulators in guiding their decisions in the
adjudication of utility rate cases.
However it is estimated, risk is generally about
a specified harm and its likelihood of occurrence.
But safety is increasingly recognized, as it was by
an international group of aviation regulators, to be
about “more than the absence of risk; it requires
specific systemic enablers of safety to be maintained
at all times to cope with the known risks, [and] to
be well prepared to cope with those risks that are
not yet known.”17 In this sense, risk analysis and risk
mitigation do not actually define safety, and even
the best and most modern efforts at risk assessment
and risk management cannot deliver safety on their
own. Psychologically and politically, risk and safety
are also different concepts, and this distinction is
important to regulatory agencies and the publics they
serve.
Risk is about loss while safety is about assur-
ance. These are two different states of mind. As
one illustration, psychologists Daniel Kahneman
and Amos Tversky have established that humans
can behave quite differently when confronted with
equivalent choices, depending upon whether they
are framed negatively in terms of potential losses or
positively in terms of possible gains.18
Public risk tolerances can also differ prior to and
after a major adverse event. Prospectively, the public
17 See the Safety Management International Collaboration Group
(2013, p. 2).
18 See Kahneman (2011).
Danner and Schulman
is likely to ignore or discount a risk relative to the
cost of its mitigation. After an accident or failure, the
public is likely to substitute a retrospective regret and
condemnation of those associated with the risk, in-
cluding regulators. This leads to a postaccident public
risk tolerance much lower than would have been im-
plied by the prior reluctance to pay the cost needed
to mitigate the risk’s likelihood or consequences.
This inconsistency in risk tolerance is heightened
because while a number of accidents or failures can
occur without invalidating a risk assessment, a single
failure can indicate to the public a lack of safety—
signaling a systemic problem that could cause more
calamities. Regulatory agencies must operate in a po-
litical world where the public seeks the assurance of
safety, and simply reducing a probability from a fre-
quentist perspective is not enough.19 If a bad event
occurs, the public and its elected representatives may
have little patience for an assertion that an agency’s
prior actions had at least made the accident less
likely, or may have postponed the catastrophe for a
little while. This reflects a public concept of safety
that rejects an assumption—implicit in a frequentist
probability estimate—that a single event or failure
may represent nothing more than a low-probability
bit of bad luck come to pass. Instead, it may be seen
as an indication that something more serious is wrong
with the management and regulation of an entire
“unsafe” and therefore “dangerous” endeavor.20
While utilities can face such a loss of public confi-
dence, these pressures can be felt even more strongly
by regulatory agencies that are inherently political to
begin with. In the United States, prominent operator
failure intensifies the incentives for regulators to po-
sition themselves as independent of the utility, so as
not to share in blame or approbation from the public.
Even if the regulator and operator had been able
to achieve some collaboration with regard to risk
assessment, it is unlikely to survive the first adverse
event significant enough to attract media or political
attention. This likely pressure for the regulator to
distance itself from the operator makes it all the
more difficult for these entities to (effectively) share
a managerial function like ongoing risk assessment,
or attempt to rely on shared efforts as the foundation
of safety oversight by the agency. Thus, to incor-
porate uncertainty in likelihood and consequence
19 For an analysis of this diverging safety standard relative to risk in
the area of nuclear waste, see Barke and Jenkins-Smith (1993).
20 For an in-depth analysis of this “amplification of risk” in the pub-
lic perceptions of safety, see Kasperson (1998).
Public Utility Safety Regulation
estimations in formal risk assessment techniques may
well require more of an ongoing working partnership
and less of an adversarial relationship than political
incentives may permit to exist between an American
public utility regulator and a utility operator.
3.2. Time and Risk
The statistical independence of single events
assumed in a probability distribution is often taken
to mean by the public that a particular event can
be expected to occur as frequently as its probability
distribution indicates it might. One Louisiana Con-
gressman’s famous post-Hurricane Katrina comment
was that “we’ve had our 100-year storm, now we can
stop worrying about one for another 99 years.”
However, two practical issues arise immediately.
Even if a frequency-based probability is accurately
assessed, the time period must be very long if the fre-
quency of actual occurrences of an event is to come
close to matching that implied by the probability
assigned to it. So while it might be safe to anticipate
something like 20 “50-year” events in a millennium,
simply by random chance it is considerably less likely
that any single 50- or 100-year period will produce
exactly the expected single storm, as opposed to two,
none, or a greater number. Second, in the absence
of a very long series of reliable data points, it can be
difficult to pin down the likelihood of a rare event
with any precision at all. For example, the knowledge
that two events occurred in a prior century might not
rule out the chance that these might recur as often
as once in every decade, as rarely as once in every
century, or somewhere in between or even beyond
the ends of that range.
In fact, calculation of a 50- or 100-year prob-
ability may be based on technical or mathematical
analyses that have little or no correspondence to
empirically observed frequencies or even validated
models for such events as severe floods, in part be-
cause of our lack of 1,000-year databases on which to
rely for a better statistical sense of the actual underly-
ing probability distribution or because flood models
are incomplete or outdated (Leskens et al., 2014).
While physical or statistical simulations of thousands
of Monte Carlo style randomized “runs” are often
employed to establish and confirm a probability in
the resulting frequency distribution of particular
specified events, these simulations may have limited
correspondences to the conditions of an activity or
operation as they will actually occur over the time pe-
riods (years, decades, or longer) being modeled. This
1049
problem is exacerbated by the reality that many of
the failures and accidents regulators are concerned
about could be catastrophic in each occurrence.
From a political or social standpoint, the public are
retrospectively risk averse to even one such disaster
and do not care about long-term averages.21
The main issue here is that regulators must make
use of risk assessment approaches that are appro-
priate to the specific decision options and choices
available to them in the administrative, political,
and social environment within which they have to
operate. This has been recognized by more recent
perspectives on the risk assessment process. The
National Research Council of the National Academy
of Sciences in a report on “Science and Decisions:
Advancing Risk Assessment” (2009) has asserted
that:
risk assessment should be viewed as a method for
evaluating the relative merits of various options for
managing risk rather than as an end in itself. Risk as-
sessment should continue to capture and accurately de-
scribe what various research findings do and do not tell
us about threats to human health and to the environ-
ment, but only after the risk-management questions that
risk assessment should address have been clearly posed,
through careful evaluation of the options available to
manage the . . . problems at hand. (NRC, 2009, p. 5)
3.3. Units of Frequency
Another practical challenge for regulatory risk
assessments involves the units of frequency measure-
ment often used for probability calculations. Many
utilities and their regulators seek to express risks
in time units of annual calendar years, for example,
the risk of a major levee failure per year. These are
the units within which budgets are frequently deter-
mined and the interval in which accounting annual
reports are issued. Many performance metrics are
also measured on an annual basis. However, it is
not clear that a calendar year is an appropriate unit
of measurement for the probability of risks to be
managed. Airlines, for example, assess safety risks
in units measured by flights or air miles flown, rather
than years.
Years may constitute neither a series of equiv-
alent nor independent units of risk or exposure.
Is a year an appropriate “run” for a failure risk
in a stretch of pipeline? What if a particular pipe
21 NancyLeveson, a safety engineer, argues that there is a “de-
creasing tolerance for single accidents” in modern society
(Leveson, 2011).
1050 Danner and Schulman

section is only pressurized in emergencies, or a water 3.4. Variance in the Architecture of Failure
pipeline is used only for overflow periods? Some
Many failures are not discrete or binary events,
years might have heavy use and others little or no
that is, it failed, or it didn’t. The binary simplification
use at all. What is the probability of failure in a dam’s
often used of “failure/nonfailure” can leave out
spillway system during a drought? Might not units of
intermediate conditions such as temporary disrup-
risk such as failure per given volume of gas or water
tion or degraded operation of equipment or service.
flow make more sense than a calendar period? For
There can be system conditions that lie between
backup generators at a nuclear power plant, it’s not
successful operation and failure in which physical
years but “starts” that are the principal measure to
assets are destroyed. Such instances can have un-
which failure probabilities are applied. For utility
certain outcomes, as a disruption can be limited
regulators, however, allowing for widely varying
and a system quickly restored to service, or lapse
units of measurement can make data demands more
over to a costly loss or catastrophe. Much depends
challenging while also complicating the effort to
upon the resilience of management in the utility.
compare different types of risks within the same
Unfortunately, resilience is a property that it difficult
or across different utilities on the same analytical
for regulators and risk analysts to assess beforehand.
basis.
This is so partly because every significant disruption
In addition, as mentioned, for many regula-
or accident has elements of uniqueness. This adds
tors, their rate-setting authority and responsibility
uncertainty to the consequence estimations of risk
might seem to compel calculation of comparative
assessments, even if sophisticated techniques might
“risk/spend” efficiencies in risk mitigation by their
yield more insights if provided with additional
utilities to support proposed investment decisions in
information. (Uncertainty will be treated in more
safety expenditures that can be recovered through
detail at a later point in this analysis.)
rate increases. Due to the annual budget cycles of
While attempts can be made to expand a
utilities and the rate case cycles of regulators, poten-
risk assessment to include intermediate modes of
tial efficiencies in risk reduction may be computed
degradation, their probabilities, and their follow-on
to the level of annualized risk reductions, down to
possibilities in a fault tree, the resulting complica-
hundreds of a percent improvement in fractions of
tions for model building can grow rapidly; those
an injury or fatality per year. One risk/spend analysis
models might be of limited usefulness if alternate
by a utility regarding helicopter risk mitigation
real-world scenarios are numerous or difficult to
of serious injuries and fatalities (SIFs) calculated
anticipate and specify. Disasters also have a way of
that “replacing a single engine with a twin engine
revealing latent and unanticipated mutual causes of
[helicopter] at $3M yields a risk/spend efficiency of
multiple failures that were simply beyond the prior
0.00815 reduction in SIFs/year per $1 million.”
imagination of the analysts.22
What meaning does a risk mitigation to this frac-
tion of a SIF per year have in the context of a man-
agement or regulatory frame of reference? How is 3.5. Unstable Probabilities
this to be communicated to the public? What is its
Frequent assumptions in probability estimations
relation to the management of safety?
that a specific event or failure is statistically indepen-
The assumption that it is possible to resolve and
dent of prior failures (or failures in subsequent years)
sensibly compare, let alone regulate, different risk
and its probability is stable over a measured interval
outcomes per dollar expenditures down to this an-
pose another challenge for utility risk assessments
nual level for “optimal” investment and rate-setting
in organizational and system settings where these
decisions would seem to be an invitation to false pre-
probabilities are not necessarily stable.23 As noted
cision and analytic error. Yet, public utility regulators
above, years themselves are not necessarily equiva-
who typically review and approve utility expenditure
lent units for probability. Further, failure in one year
levels in a detailed manner may be hard-pressed
to avoid some ostensibly precise risk abatement
22 For classic descriptions of organizational and managerial vari-
cost-effectiveness conclusions in authorizing partic-
ular investments or expenditures for safety-related ables at the root of accidents, see Reason (1997) and Vaughan
(2016).
purposes, if for no other reason than to preserve an 23 The assumptions of stable probability and independence of
appearance of comprehensive expertise and careful events are cornerstones of Poisson distributions frequently used
oversight important to their political standing. in risk analyses.
Public Utility Safety Regulation
or even the absence of failure over many years can
affect the probability of failure in subsequent years.
Many risk assessments differentiate possible
events into time-dependent (such as failure due to
pipe corrosion) and time-independent risks. Certain
risks, such as weather or incorrect operations failures
are often described as time invariant—that is, the
probability of failure is assumed to be constant
from year to year over the entire interval addressed
in the analysis. But ironically, a number of years
without failure can make operators and managers
complacent, lowering care and mindfulness in the
management of operations. This can actually in-
crease the probability of failure over successive
years. Indeed, “safety” is sometimes assumed to
exist on the basis of successive operations or years
that pass without an accident, when in effect these
periods can really represent only the “failure to fail”
under poorly managed conditions.
An analysis of the space shuttle Challenger
explosion by sociologist Diane Vaughan argued that
successful flights of the shuttle created a false sense
of security at NASA that ongoing O-ring failures
were unimportant, despite the fact that O-ring failure
was considered a first criticality threat to the safety
of flight. Vaughan termed this a “normalization of
deviance” within the culture of NASA that led to
the acceptance of this and other operations outside
of design conditions (Vaughan, 2016),
We offer this as only one example of how orga-
nizational and managerial variables can make a big
difference in both the likelihood and consequences
of events. Yet few risk assessments—and particu-
larly those used by utility regulators—treat these
variables carefully or fully in their analysis. This is in
part because they are difficult to measure, and also in
part because they can lead to normative assessments
of management performance that regulators may
find difficult to make and politically controversial
to act on. For example, contentions that a utility is
well or poorly managed may be relatively easy to
argue, but more challenging for political appointees
and civil servants to evaluate given the limits to the
information available to them in the litigious context
(including appeal rights by utilities) in which such
oversight is typically conducted.
4. LESSONS FROM HROs
We now turn to some different (and very largely
successful) risk analysis and management practices
in “high-reliability” organizations (HROs). The risk
1051
assessment practices noted above suggest that risk
has to be understood in more dimensions than are
currently addressed in the risk assessments often
used by utilities and their regulators. Importantly,
organizational and managerial factors if addressed in
a risk-informed management system can supplement
the picture of risk provided in those assessments—
and can aid both a regulatory agency in its safety
oversight, and the regulated organization’s own
safety management.
A risk-informed management system can help
bridge the gap between the practical realities of
actual operating technical systems, and the assump-
tions in those risk assessments that do not square
with practical realities and constraints. A brief look
at findings from research into HROs can illustrate
this.24 These organizations, while certainly not typi-
cal in the extreme hazards they are trying to manage
and the self-contained control systems they exercise
over relatively well-understood technical systems,
have successfully faced a variety of major challenges,
in difficult environments, with respect to managing
risk and safety. They offer instructive examples of
the ways that risk assessment assumptions can be
adjusted to safety management requirements to
promote more effective risk management and safety.
The concept of an HRO is empirically based, and
has followed from the observation that certain orga-
nizations operating complex systems achieve higher
rates of reliability and lower rates of catastrophic
events than might be expected from organization the-
ory and many case studies. Rather than hypothesiz-
ing what might be required to operate in this man-
ner, the research has tended to focus on identifying
successful operators and finding common elements in
what they are doing. When combined with unfortu-
nate lessons of organizational failures, the result has
been to create some compelling managerial models
for safe and productive operations.
To begin with, many risk assessments focus
primarily on risks of physical failure of technical
systems. Yet these systems are understood to be
sociotechnical systems—with operators and main-
tenance personnel and a variety of organizational
characteristics acting as important elements in their
functioning.25 For example, an airliner is not simply
the physical hardware but also the skills of pilots who
24 For a review of the HRO literature, see LaPorte (1996), LaPorte
and Consolini (1991), Roberts (1993), and Schulman (1993).
25 A pioneering
description of the sociotechnical systems approach
is Emery (1959).
1052
fly it, the maintenance personnel who keep it in air-
worthy condition, and the air traffic controllers who
direct and separate its flight path from other aircraft.
Similarly, a pipeline is not only the physical contain-
ment structure for gas and liquids, but the operators
who keep flows within acceptable pressures and per-
sonnel who thoroughly assure its integrity and tend
to leaks. In recognition of this, some risk analysts
have attempted to consider the likelihood of human
error as analogous to the failure probabilities of
machines. However, despite human factors research
into design strategies that can force or diminish
human error, the quest for “natural laws” to predict
and prevent such failures has been largely elusive.
Recent safety research now argues that human
error must be understood as embedded in the actual
context, particularly the organizational context, sur-
rounding behavior.26 Humans act in sociotechnical
systems, and the likelihood of human error depends
upon the character and challenge of the task and
surrounding organizational elements within which
it is conducted. In fact, many major failures and
accidents with technical systems—including Three-
Mile Island, Chernobyl, the Challenger explosion,
and Deepwater Horizon—turned out not to be the
isolated failure of an individual, but instead system
errors with design, management, and organizational
factors as their root causes.27
Along those lines, management is a useful frame
within which to consider a variety of human and
organizational variables, including those needed for
understanding and assessing risk in technical systems.
Management strategy is a preoccupation with HROs.
Yet, managerial factors are typically neglected in
non-HRO risk assessments. There may be consider-
ation of such concerns as “inadequate training” or
“incorrect operations,” but these are seldom parsed
out to include the layered set of management
practices and errors that underlie them. This is
understandable due to the difficulty of making the
attempt, as management factors and errors do not
connect as directly as physical failures to scenarios of
adverse outcomes and probabilities that can be spec-
ified. In addition, management and organizational
factors seem less quantifiable than physical condi-
tions and attributes. But their difficulty for making
26 See,for example, Hollnagel (2014), Reason (1997), and Dekker
(2011).
27 For organizational analyses of catastrophic accidents, see Per-
row (1999), Turner and Pidgeon (1997), Vaughan (2016), and
Bazerman and Watkins (2008).
Danner and Schulman
formal risk assessment estimates does not diminish
the importance of management and other organi-
zational variables for achieving safe and effective
operations.
In some ways HROs offer an alternative ap-
proach to risk assessment and safety management.
Research on HROs that have to manage very haz-
ardous technical systems—such as nuclear power
plants or airlines in commercial aviation and the
air traffic control centers that direct this aviation—
has revealed a set of quite distinctive organizational
and managerial practices. They have organized their
risk management around a set of worst-case events
that are surrounded by societal dread, for example,
the loss of nuclear containment, or a crash or colli-
sion of commercial aircraft under air traffic control.
These events are, for these organizations, ones that
must never happen. Thus, HROs organize to pre-
clude these events not simply on the basis of formal
probabilities, but also by managing to possibilities,
when probabilities are uncertain.28 Their view, and
that of the public, is that reducing the probabilities is
not enough when even one occurrence is too many
(Pool, 1997).
HROs operate under detailed and extensive risk
analyses of all of their operations, often as a re-
quirement of the regulatory oversight they face. For
American nuclear power plants, for example, it is
against federal regulations to operate these plants
“outside of analysis”—including unanalyzed condi-
tions whose dynamics cannot be reliably predicted,
even if no apparent problem or danger exists. Under
careful analysis, HROs identify “precursor” failures
or conditions that are in effect drivers of risk (that
could lead to the worst-case events), and they orga-
nize to stay away from these conditions.
For nuclear power plants, some of these pre-
cursors are physical conditions: temperatures and
pressures beyond the allowable bandwidth of nor-
mal operations. However, other precursor conditions
are those that affect safety management itself. Some
are conditions that deny information to operators,
such as telecommunications or sensor failures un-
der which the state of system elements would be un-
known, or display dropouts during which it would be
hard to monitor or control technical systems in real
time. Noise or other distractions in a control room
can also be treated as precursors. Even though these
28 Risk as possibility, not probability, of adverse events is also one
of the qualitative definitions of risk offered in the SRA Glossary
(SRA, 2015).
Public Utility Safety Regulation
precursor conditions are not emergencies as such,
management has made the explicit determination not
to tolerate them because of the lack of experience or
understanding (and thus diminished possibilities for
control) that they represent.
These precursor zones represent a sometimes
explicit, but at other times less formal, real-time
risk assessment of a kind (founded on possibility
but not necessarily probability) on the part of not
only HRO management but also individual nuclear
power control operators, air traffic controllers, and
pilots. Rather than trying to estimate poorly under-
stood risks in such circumstances, they avoid them
altogether. These individuals can actually refuse
to continue operations in such precursor zones.29
The management practice and culture in these or-
ganizations supports observance of these boundary
conditions, and provides a signaling language by
which operators can invoke them. Operators can
announce that they find themselves in “unstudied
conditions” or even state that they are “uncomfort-
able” operating in precursor conditions, and these
pronouncements will be taken seriously by their
supervisors and by higher management. Indeed,
being in conditions under which a risk is difficult to
understand and assess is itself considered a risk in its
own right (Roe & Schulman, 2008).
Another important feature of HROs is their
prospective focus and “positive skepticism” about
failure. The managers of these organizations believe
in (and the culture of the organizations reinforces)
a need to ward off complacency. Reliability and
safety are not defined retrospectively for them by
the many successful and uneventful operations that
lie behind. Rather, prospectively they are perceived
to be only as reliable as the first failure ahead, to
which a precursor condition could lead and against
which preemptive responses are made. Unlike many
organizations that base their confidence on lagging
indicators, the managers of these organizations “run
scared” about their reliability—they are always
worried about the unforeseen and the unknown. As
a maintenance department head at a nuclear power
plant instructed his personnel after the implementa-
29 Nuclear plant control operators can on their own authority
“scram” or shut down a reactor if conditions stray into precursor
zones; pilots can refuse to fly under precursor conditions such
as bad weather or doubts about the airworthiness of their air-
craft; and air traffic controllers can refuse to accept responsibil-
ity for maintaining airplane separations and close off a sector of
airspace if they feel conditions of air traffic load or congestion
are in a precursor state.
1053
tion of a new procedure: “Don’t think for a minute
that this technology can’t still surprise you.”
Such useful practices extend beyond formal risk
models. Many of the features observed in HROs are
focused on protecting against error, including “repre-
sentational” error. Representational errors are those
of mis-specification, mis-estimation, and misunder-
standing on the part of members of an organization
about the systems they are managing.30 These errors
can lead to overconfidence or hubris, complacency,
and/or misdirected attention.
A related issue is that some risk assessments
neglect epistemic uncertainties31 and thus can be
founded on representational error in their under-
standing of failure likelihood and their specification
of failure consequences. This in turn can lead to a
misleading ranking of the most important risks in a
risk register or heat map from which management
will take guidance. In this way, risk assessments can
themselves alter failure probabilities by conveying
representational errors to managers and leading
thereby to follow-on errors of action or inaction.
Acting or regulating based on incomplete or erro-
neous understanding can thus add to risk. Notice
that this recognition of the importance of represen-
tational error at the same time creates a management
and a regulatory responsibility to protect against
and detect likely sources of these errors—including
those that could be imported from inappropriate risk
assessment methods.
5. HOW RISK ASSESSMENT CAN ENHANCE
THE MANAGEMENT AND REGULATION
OF SAFETY
Against the backdrop of the safety and risk
strategies of HROs and considering the context of
regulatory agencies (including their organizational
incentives and capabilities), and the limitations of
classical risk assessment processes in relation to this
context, we offer some recommendations for the im-
provement of risk assessment that might contribute
to more effective safety management in and regula-
tion of modern public utilities.
First, effective safety management is about not
only “managing the unexpected” (Weick & Sutcliffe,
30 Psychologist Karl Weick (1995) terms these “errors of rendi-
tion.”
31 Epistemic uncertainties and the need to recognize them in risk
assessments are described in the document Guidance on the
Treatment of Uncertainties by the U.S. Nuclear Regulatory Com-
mission (2009).
1054
2015) but also the enlargement of expectancies, that
is, the appreciation of those involved that the range
of things that can go wrong is quite broad (and prob-
ably broader than they currently know). A substan-
tial and ubiquitous dose of this intellectual humility is
a hallmark of HROs, and risk assessment should en-
courage that expansion of the perceived range of pos-
sibilities rather than circumscribing expectancies in
quest of potentially false precision in risk estimates.
In this light we can see several possible improve-
ments for risk assessment to enhance both safety
management and regulation.
5.1. Modeling Safety Management
Beyond just equipment, technologies, and ex-
ternal events, management itself should be included
as an important risk factor. Reflecting that recog-
nition, a number of models of functions, strategies,
and practices for effective safety management in
organizations are currently prescribed by regulatory
agencies and industry organizations such as the
U.S. Nuclear Regulatory Commission, the Canadian
Nuclear Safety Commission, the Federal Aviation
Administration, the Institute of Nuclear Power
Operators, the Center for Chemical Process Safety,
and the Pipeline Safety Institute.32
These models are not simply prescriptions on
“how to manage” for safety. They also lay a foun-
dation for potential metrics for assessing the con-
tribution that existing practices, strategies, and cul-
ture in an organization make to either amplify or
mitigate risk. Many of the elements in these models
could be used as indicators of managerial capacity to
perform effective risk management, across the vary-
ing challenges we have highlighted. When integrated
with the physical failure analyses typically included
in a risk assessment, an analysis of organizational and
managerial vulnerability to these failures can provide
the larger system perspective that is important for en-
hancing safety through both management and regu-
lation.
Further, the risk mitigation that can be achieved
by improving elements of safety management is of-
ten a far cheaper option than large-scale capital in-
vestments in new equipment and technology, which
32 See,
for example, Center for Chemical Process Safety (2007),
Underwriters Laboratory (2013), Safety Management Interna-
tional Collaboration Group (2013), U.S. Federal Aviation Ad-
ministration (2017), and the Institute of Nuclear Power Opera-
tors (2013).
Danner and Schulman
are the ones utilities and their regulators typically
focus on in relation to cost recovery through rate
cases. Replacing older with newer gas pipelines, for
instance, might reduce risk only when they are in-
stalled correctly, inspected and maintained properly,
and operated within allowable pressure and temper-
ature limits. Without managerial and organizational
supports for these competencies, the risk reduction
that is physically possible through the new invest-
ment may not be fully realized, if at all.
One management model comes from the Insti-
tute for Nuclear Power Operators (INPO, 2013) and
offers a number of useful possibilities for metrics,
including both leading and lagging indicators for
assessing safety management and organizational
drivers of risk. These managerial factors include:
communication, decisionmaking, work planning, and
emergency response capacity. Significantly, some
of these safety management models overlap with
approaches that have been associated with effective
management in general, and high productivity levels
in competitive business firms.33 Such management
and “safety culture”34 elements could be oper-
ationalized and assessed through data collected
monthly, quarterly, yearly, or over two- or three-
year periods, through specific accounting practices
in utilities, observations by regulatory inspectors and
auditors, or through employee surveys conducted by
independent organizations.
The narrowness of current risk management per-
spectives is a reason why risk assessments used by
utilities and regulatory agencies often overlook (or
underweigh) the role of interconnected or interas-
set risks that cut across physical systems. Regula-
tory oversight agencies themselves, with specialized
regulatory units, often focus on localized concerns
for each given service or separate infrastructure, but
leave unexamined the catastrophic implications of
failures across several of these at once. As a result
of this limitation in risk analysis, neither utilities nor
their regulators fully understand the full range of
these interconnected risks.35
Adding these organizational and managerial
factors to risk assessments would have important
regulatory implications. The development and anal-
ysis of risk-related safety management metrics would
33 For support for this argument, see Bloom and van Reenan
(2006) and Oliver et al. (2017).
34 See, for example, Hopkins (2008).
35 For a recent review of the complexity of these interconnections,
see Roe and Schulman (2016).
Public Utility Safety Regulation
give a regulatory agency a consistent basis to assess
management practices of a utility, compare them
to other regulated organizations and industry best
practices, and identify deficiencies. The managerial
and organizational metrics themselves can enhance
the inspection and audit processes of the regulator.
Given the major role for many regulatory agen-
cies in controlling rates charged to customers, formal
risk assessments are increasingly used by utilities
to justify rate increases to fund safety, capacity,
or reliability improvements. As noted above, the
addition of organizational and managerial variables
would allow a regulatory agency to assess more
carefully the risk mitigation that might actually occur
through higher spending or investments in physical
equipment. It would supplement the question often
asked by regulators: “How much safety are we
willing to pay for in rate increases?” with another
prior, and perhaps more useful, question: “How
much safety are we likely to get from a proposed risk
mitigation investment, given the safety management
effort, capacity, and competence of the utility?”
A safety management score could even be
developed and used as a factor to adjust the risk
mitigation calculations offered by the utility with
regard to investments in physical assets and systems.
In other words, it is not just the equipment that
matters, but also how effectively and reliably it is
likely to be used. This could provide an incentive for
better and more updated risk assessment processes
as an element in better safety management for both
utilities and regulators.
5.2. The Representation of Uncertainty
A second potential improvement in risk assess-
ment methodology would be a better representation
of uncertainty as a key strategy. In its report on “Ad-
vancing Risk Assessment” the National Academy of
Sciences asserted that:
Just as a risk assessment itself should be more closely
tied to the questions to be answered, so should the
technical analyses supporting it. For example, descrip-
tions of the uncertainty and variability inherent in all
risk assessments may be complex or relatively simple;
the level of detail in the descriptions should align with
what is needed to inform risk-management decisions.
(NRC, 2009, p. 5)
As part of this alignment between uncertainty
descriptions and decision requirements, it is impor-
tant to recognize that there are differing types of un-
1055
certainty and specifying and describing the type con-
veys important information.
The U.S. Nuclear Regulatory Commission has
distinguished several types of epistemic uncertainty:
parametric uncertainty (in measurements assigned
to likelihoods and consequences), modeling un-
certainty (in causal understanding underlying risk
estimates), and incompleteness uncertainty (impor-
tant risk factors that may be left out in a risk analysis
such as the organizational and managerial variables
we have mentioned above) (U.S. NRC, 2009, pp.
13–19). While many modern risk assessments do
try to describe and characterize uncertainty, there
are still analyses presented to utility regulators that
misleadingly present probability as single-number
estimates and consequences as well in single-dollar
or fatality estimations and thus can instill a false
sense of confidence in precision and completeness to
managers and regulators.
In describing “black swan” events, Nassim Taleb
(2010) asserts that these events cannot really be
anticipated and contends that “appropriate prepa-
ration for these events is frequently hindered by the
pretense of knowledge of all the risks.”36 However,
there are risk analysis and management strategies
that can address different types of “black swans”
(Aven, 2015) and HROs illustrate that a proper
acknowledgment and understanding of these and
other uncertainties does not foreclose effective
safety or risk management.
Uncertainty concerning likelihoods leads HROs
to manage from possibility, and not simply proba-
bility. Uncertainty concerning consequences leads
HROs to manage against worst-case scenarios. Un-
certainty about both likelihoods and consequences
leads them to cease operations in precursor zones,
and stress emergency response preparations and the
promotion of resilience. HROs have been successful
not by eliminating uncertainty (though they are con-
stantly trying to reduce it and extend their knowledge
base through the range and depth of their analyses),
but by recognizing it and preparing for surprises.
It is important in good risk assessment practice
to preserve information about the existence of un-
certainty, not disguise it by single-value estimations
that suggest precision but may undermine accuracy.
A range of probability and consequence estimates
should be presented when there is disagreement
about “true” values. This does not always happen
36 As
quoted in Wikipedia, “Knightian Uncertainty” (https://en.
wikipedia.org/wiki/Knightianuncertainty).
1056
in the risk assessments used by utilities and their
regulators. One reason it doesn’t is to preserve the il-
lusion of precision in the calculation and application
of risk/spend optimizing models to guide investment
and rate-decisions. Another is the effort by both
utilities and regulators to reassure the public they
are protecting them through safety management of a
“scientific” nature.
One potential benefit of specifying uncertainty
can be useful information about the strategic in-
vestment value of research and development to
reduce epistemic uncertainties—especially in cases
where knowledge could likely be gained and where
important managerial decisions appear to turn on a
better or more precise understanding of underlying
conditions or variables. Spending a million dollars to
clarify the potential gains and risks from investing a
hundred million could make sense where the result-
ing information could tip the balance for or against
the decision. A “value of information” analysis can
even be part of a thorough risk analysis (Samson,
Wirth, & Rickard, 1989).
Sensitivity analysis should also be part of every
risk assessment, testing its conclusions across a range
of possible variation in key assumptions underlying
specific probability estimates and consequence costs,
as well as exogenous factors such as interest rates,
service demand, equipment replacement costs, and
other circumstances (e.g., trends such as climate
change), to see how sensitive model outputs such as
risk and risk rankings are to these variances.
5.3. Adding Validation to Risk Assessments
Finally, it would be helpful to have a validation
component in every risk assessment. A risk assess-
ment is both an analysis and a set of hypotheses.
The hypotheses are the predictions of the likelihood
of specific failures based on a model of failure
mechanics and estimates of exogenous events, as
well as an estimate of the consequences of these
failures, which should in turn be based on causal
models and social assumptions. A risk assessment
for a levee failure, for example, should be based on
weather models that help predict storm likelihoods
and hydrologic models that predict levee resistance
at different water levels. It should also be based
on flood models and loss-of-life estimates given
different flood levels in different areas. Over time,
it is possible to compare the assumptions of such
models to actual occurrences or new or corrected
information, and to test, correct, or modify them
Danner and Schulman
depending on how well they conform to ongoing
experience.37
This is particularly important when models fail
in repetitive ways, such as when “100-year storms”
keep recurring, or if catastrophes or major con-
sequences seem to come out of the blue so far as
risk models are concerned, as in the 2008 meltdown
of the U.S. financial sector and its international
economic consequences. Regarding this financial
crisis, the risk manager of Goldman Sachs later
observed that “we were seeing things that were
25 standard deviation moves several days in a
row” [from the modeled probability distribution of
occurrences]. However, as one economic analyst ob-
served, instead of astronomically unlikely bad luck,
“an alternative—and correct—explanation was that
the models Goldman-Sachs were using to evaluate
the riskiness of its strategies were totally wrong.”38
The long-run validation and improvement of
risk assessment models can be a challenge for
regulatory agencies headed by political appointees
(who tend to transition from office fairly frequently,
and may change in philosophy as political currents
shift), that rely on formal administrative processes
for decisionmaking (where each proceeding or case
may be considered as a fresh endeavor), and that
often have multiple and sometimes shifting policy
priorities assigned to them. Some organizational
creativity may be needed for such agencies to
institutionalize a consistent focus on risk model
development and improvement in the face of these
dynamics.39
6. CONCLUDING OBSERVATIONS
This analysis has described how enhanced risk
assessments could be used by utilities and their
regulators as instruments for upgrading the overall
safety management of critical public utilities. They
can be a means for learning and improvement,
and should not simply be employed as forensic
instruments to justify rate increases or for pro
forma regulatory compliance. Managerial metrics
and risk assessments incorporating them should
further be used by utilities and their regulators as
37 For
a good treatment of this function as a needed part of risk
assessment, see Goble and Bier (2013).
38 For an account of analytic errors underlying the 2008 financial
crisis, see Wolf (2015).
39 For an exploration of risk validation against the backdrop of
a major chemical accident in Seveso, Italy, see Lindhout and
Reniers (2017).
Public Utility Safety Regulation
drivers toward managerial improvement and excel-
lence rather than simply compliance and regulatory
adequacy.
Indeed, drifts and lapses from (mere) adequacy
are more likely to occur than drift away from a
commitment to excellence—because as we have de-
scribed in HROs, good safety management and good
risk assessment entail a continuing self-examination
and sensitivity to slips in standards and performance.
The best tend to maintain their position, while the
mediocre may decay.
Beyond the more informed use of improved risk
assessment, we can also offer a concluding suggestion
for the role of utility regulatory agencies in the over-
sight of safety management by complex, regulated
entities. We propose four broad and feasible objec-
tives for utility regulatory agencies to address if they
are to gain public support and function effectively
in the realm of safety. At the outset of this analysis,
we associated safety with assurance, not simply the
calculation and mitigation of loss. Broadly stated,
our suggestions form a baseline set of assurances
agencies might aspire to provide to themselves and
the public, given where they sit, the nature of their
task, and the special requirements for utility safety,
oversight, and management. These assurances are
that:
(1) The utilities are using modern and appropri-
ate technology that should operate safely if uti-
lized properly;
(2) The utilities’ management and staff have inter-
nalized a commitment to the safe operation of
their systems;
(3) The utilities’ management and staff are contin-
ually attentive to system operations and risks,
including effective monitoring of both techni-
cal and organizational conditions, and are per-
forming continuous improvement practices for
their own internal processes and procedures;
and
(4) The utilities’ management and staff are highly
competent and well-trained.
We offer these objectives for several reasons.
First, they are straightforward, and fit with the
concerns and preferences of the public toward the
prevention of catastrophic events. Regulators can ex-
plain these to the public, and be understood.
Second, these objectives are consistent with the
experience and literature we have discussed about
1057
how effective safety management systems function.
These approaches appear to work.
Finally, these objectives are feasible for utility
regulatory agencies to pursue, notwithstanding the
various constraints and limitations under which they
must operate. These do not depend on fine grada-
tions of probabilities or other forms of “precision”
in risk assessments that are hardly feasible anywhere
(much less in this context), and they will tend to
mitigate the various kinds of related risk assessment
problems we discussed above, such as representa-
tional error or undue managerial complacency. They
are also likely to satisfy the political incentives and
demands that agencies face, and to be feasible given
the resources, competencies, and administrative
procedures they are able to employ.
By contrast, we are far less sanguine about
the potential for cost-effectiveness or risk-benefit
calculations to yield optimal results (such as for
picking and choosing utility investments based on
safety considerations), particularly in the adver-
sarial, often information-challenged environments
in which regulators must operate. Careful and
creative risk assessments can be a highly useful
discipline to help inform a variety of management
and decision making processes. However, it is also
a safety management responsibility to understand
the difference between risk-informed and risk-based
decisionmaking.
Modern technology, commitment to safe op-
erations, consistent attention, and highly capable
personnel—these are four key qualities of a firm’s
operations on which we believe safety oversight
must focus, given a realistic view of a public utility
regulatory agency’s own capabilities. These qualities
fit well with the lessons that have been learned from
HROs, and the sad but necessary study of prior dis-
asters. It is not just equipment and technology that
can fail catastrophically, but also the organizations
that operate them—even, quite possibly, as an unin-
tended byproduct of poor risk assessment exercises
themselves.
ACKNOWLEDGMENTS
The authors would like to thank Herman B.
Leonard, Harvard University, David J. Teece,
University of California, Berkeley, Alex Tabarrok,
George Mason University, and Arthur O’Donnell,
California Public Utilities Commission, for their ad-
vice and suggestions on earlier drafts of this article.
We also benefited considerably from constructive
1058
comments provided by two anonymous reviewers
and the area editor of Risk Analysis.
REFERENCES
Aven, T. (2015). Implications of black swans to the foundations
and practice of risk assessment and management. Reliability
Engineering and System Safety, 134, 83–91.
Baldwin, R., Cave, M., & Lodge, M. (2013). Understanding regu-
lation: Theory, strategy and practice. Oxford: Oxford University
Press.
Barke, R. P., & Jenkins-Smith, H. (1993). Politics and scientific
expertise: Scientists, risk perception and nuclear waste policy.
Risk Analysis, 13(4), 425–439.
Bazerman, M. H., & Watkins, M. D. (2008). Predictable surprises.
Cambridge, MA: Harvard Business Review Press.
Beecher, J. (2008). The prudent regulator: Politics, independence,
ethics and the public interest. Energy Law Journal, 29, 577–
614.
Beecher, J., & Kihm, S. (2016). Risk principles for public utility
regulators. East Lansing: Michigan State University Press.
Bieder, C., & Bourrier, M. (Eds.). (2013). Trapping safety into
rules. London: CRC Press.
Bignami, F., & Zaring, D. (2018). Comparative law and regula-
tion: Understanding the global regulatory process. Northamp-
ton, MA: Edward Elgar.
Bloom, N., & van Reenan, J. (2006). Measuring and explain-
ing management practices across firms and countries. Cen-
ter for Economic Performance Discussion Paper No 716
(March).
Center for Chemical Process Safety. (2007). Human factors meth-
ods for improving performance in the process industries. Hobo-
ken, NJ: Wiley.
Clifton, J., Lanthier, P., & Schroter, H. (Eds.). (2012). The eco-
nomic and social regulation of public utilities. New York:
Routledge.
Coelli, T., & Lawrence, D. (2006). Performance measurement
and regulation of network utilities. Northampton, MA: Edward
Elgar.
Dekker, S. (2011). Drifting into failure. London: CRC Press.
Emery, F. E. (1959). Characteristics of socio-technical systems.
London: Tavistock Documents.
Goble, R., & Bier, V. (2013). Risk assessment can be a game-
changing information technology. Risk Analysis, 33(11), 1942–
1951.
Gormley, W. T. (1984). The politics of public utility regulation.
Pittsburgh: University of Pittsburgh Press.
Hale, A., Borys, D., & Else, D. (2012). Management of safety rules
and procedures: A review of the literature. Leicestershire, UK:
Institute of Occupational Safety and Health.
Hausman, W., & Neufeld, J. (2012). How politics, economics
and institutions shaped electricity regulation in the United
States. In Lanthier, Clifton, & Shroter (Eds.), The economic
and social regulation of public utilities (pp. 65–88). New York:
Routledge.
Hollnagel, E. (2014). Safety-I and Safety-II: The past and future of
safety management. Boca Raton, FL: CRC Press.
Hopkins, A. (2008). Safety, culture and risk. London: McPherson
Group.
Institute of Nuclear Power Operators. (2013). Traits of a healthy
nuclear safety culture, Retrieved from https://www.nrc.gov/
docs/ML1303/ML13031A707.pdf.
International Organization for Standardization (2009). ISO31000
preview. Retrieved from https://www.iso.org/obp/ui/#iso:sTd:
iSo:31000:eD-1:v1:en.
Kahn, A. (1988). The economics of regulation: Principles and insti-
tutions. Cambridge, MA: MIT Press.
Danner and Schulman
Kahneman, D. (2011). Thinking fast and slow. Princeton: Prince-
ton University Press.
Kasperson, R., Renn, O., Slovic, P., Brown, H. S., Emel, J., Goble,
R., & Ratick, S. (1998). The social amplification of risk. Risk
Analysis, 8(2), 177–189.
LaPorte, T. (1996). High reliability organizations: Unlikely, de-
manding and at risk. Journal of Contingencies and Crisis Man-
agement, 4(2), 60–71.
LaPorte, T., & Consolini, P. (1991). Working in practice but not
in theory. Public Administration Research and Theory, 1(1),
19–47.
Leone, R. (1986). Who profits: Winners, losers and government reg-
ulation. New York: Basic Books.
Leskens, J. G., Brugnach, M., Hoekstra, A. Y., & Schu-
urmans, W. (2014). Why are decisions in flood disas-
ter management so poorly supported by information from
flood models? Environmental Modelling and Software, 53,
53–61.
Leveson, N. (2011). Engineering a safer world. Cambridge, MA:
M.I.T. Press.
Lindhout, P., & Reniers, G. (2017). Risk validation by the reg-
ulator in Seveso companies: Assessing the unknown. Jour-
nal of Loss Prevention in the Process Industries, 49, 78–
93.
National Research Council. (2009). Science and decisions: Ad-
vancing risk assessment. Washington, DC: National Academies
Press.
Oliver, N., Senturk, M., Calvard, T., Potocnik, K. & Tomasella,
M. (2017). Collective mindfulness, resilience and team pefor-
mance. Academy of Management Annual Meeting Proceedings
1, 12905.
OSHA. (2016). Recommended practices for safety and health
programs, Retrieved from https://www.osha.gov/shpguidelines/
docs/OSHA_SHP_Recommended_Practices.pdf.
Phillips, C. (1993). Regulation of public utilities: Theories and prac-
tice (3rd ed). Arlington, VA: Public Utilities Reports.
Pool, R. (1997). When failure is not an option. M.I.T. Technology
Review, July 1.
Reason, J. (1997). Managing the risks of organizational accidents.
London: Ashgate.
Roberts, K. (1993). New challenges to understanding organiza-
tions. New York: Macmillan.
Roe, E., & Schulman, P. (2008). High reliability management. Stan-
ford, CA: Stanford University Press.
Roe, E., & Schulman, P. (2016). Reliability and risk: The challenge
of managing interconncted critical infrastructures. Stanford, CA:
Stanford University Press.
Safety Management International Collaboration Group. (2013).
Measuring safety performance, July, p. 2. Retrieved from
https://www.skybrary.aero/bookshelf/books/2395.pdf.
Samson, D., Wirth, A., & Rikard, J. (1989). The value of in-
formation from multiple sources of uncertainty in decision
analysis. European Journal of Operational Research, 39(3),
254–260.
Schulman, P. (1993). The negotiated order of organizational relia-
bility. Administration and Society, 25(3), 353–372.
Snook, S. (2002). Friendly fire. Princeton, NJ: Princeton University
Press.
Society for Risk Analysis (SRA). (2015). Glossary of risk terms.
Retrieved from http://www.sra.org/sites/default/files/pdf/SRA_
glossary_20150622.pdf.
Taleb, N. (2010). The black swan: The impact of the highly improb-
able (2nd ed). New York: Random House.
Turner, B., & Pidgeon, N. (1997). Man made disasters. Oxford:
Butterworth-Heinemann.
Underwriters Laboratory. (2013). Using leading and lagging safety
indicators to manage workplace health and safety risk.
U.S. Federal Aviation Administration. (2017). SMS: Safety man-
agement system manual. Washington, DC: FAA.
Public Utility Safety Regulation 1059

U.S. Nuclear Regulatory Commission. (2009). Guidance on the
treatment of uncertainties associated with PRAs in risk-informed
decision-making. Washington, DC: NRC.
Vaughan, D. (2016). The Challenger launch decision. Chicago:
University of Chicago Press.
Weick, K. (1995). Sensemaking in organizations. London: SAGE
Publication, Inc.
Weick, K., & Sutclilffe, K. (2015). Managing the unexpected.
Hoboken NJ: John Wiley.
Wilson, J. (1980). The politics of regulation. New York: Basic
Books.
Wolf, M. (2015). The shifts and the shocks. New York: Penguin
Books.