Risk Analysis, Vol. 24, No.
4, 2004
Redundancy as a Design Principle and
an Operating Principle
John S. Carroll∗
1. INTRODUCTION
In an era of increased concern over terrorist at-
tacks on nuclear installations and other targets, policy
makers must address questions such as: “Will more
(nuclear) security forces produce more (nuclear) se-
curity?” As Scott Sagan (this issue) points out, station-
ing more guards at every vulnerable point seems like a
simple and effective solution based on the engineering
principle of redundancy, but carries additional risks
that could make us less secure overall. Sagan articu-
lates three problems with redundancy as a strategy:
common-mode failures such as insider threats from
the guards themselves, diffusion of responsibility that
produces relaxed vigilance, and overcompensation or
expanding the mission in the belief that there is more
than adequate protective capability.
Sagan’s argument is innovative and persuasive.
In essence, policy makers and engineer designers are
familiar with technical logics that analyze problems
into subpieces, address each subpiece, and then cal-
culate overall effectiveness under assumptions such
as independence. In technical systems, independence
means that a flaw in one place is uncoupled from a
flaw elsewhere, so the probability of two flaws can
be calculated easily from the separate probabilities.
However, in human systems, independence rapidly
breaks down because people take action based on
their beliefs about what other people are doing: in
short, protective measures that are technically inde-
pendent may be socially coupled (and also technically
coupled in unanticipated ways, Perrow (1984)). Sagan
gives the example of the friendly fire downing of two
Black Hawk helicopters over the no-fly zone in Iraq
in which two F-15 pilots were supposed to “indepen-
∗ MIT Sloan School of Management.
955
dently” confirm the identity of the helicopters. How-
ever, the second pilot, unable to get a clear view, only
stated that he saw two helicopters; each pilot assumed
the other had made a correct identification.
2. DESIGN LOGICS AND OPERATING
LOGICS
Underlying this difficulty in managing human sys-
tems is a larger issue of the differences between de-
sign logics and operating logics (Carroll, 1998; Perin,
1998). Design logics, the province of engineers and
other technical specialists, deal with how things are
intended to work. As Sagan points out, the existing
regulations that call for five security guards at nuclear
power plants are designed to meet a maximum threat
from three outside attackers and one plant insider,
giving the guards a numerical advantage. Similarly,
the concrete containments of nuclear power plants are
designed to withstand a direct hit from a Boeing 707
jet, the largest commercial plane when the regulations
were written (but much smaller than modern jumbo
jets). Probabilistic risk analysis, the most elaborate
version of protective calculus, is based on a variety
of assumptions, including that plants will correspond
to their designs and be operated in accordance with
regulatory requirements. In short, design logics are di-
rected at anticipating threats and designing defenses
(redundant or “in depth” where necessary) to prevent
and mitigate known problems.
However, the world of actual operations always
diverges from the assumptions on which plans and
designs are based. Analyses of serious accidents at
nuclear power plants show that plants were outside
their intended operating conditions (Reason, 1990;
Rasmussen, 1990). The as-built plant was not identical
0272-4332/04/0100-0955$22.00/1 "
C 2004 Society for Risk Analysis
956
to the design, equipment was allowed to deteriorate,
repairs were not made in a timely fashion, opera-
tors and maintenance personnel worked in ways that
designers had not considered, and unanticipated tech-
nical challenges emerged (e.g., stress corrosion crack-
ing, outage risk). In contrast to design logics, operat-
ing logics demand continual alertness, maintenance
of doubt (Schulman, 1993), heedful interrelating
(Weick et al., 1999), and resilience in the face of sur-
prise (Wildavsky, 1988). This means that plants are
both less safe and more safe than we calculate: less
safe because hidden traps or “latent defects” (Reason,
1990) are widespread, and more safe because people
can improvise to handle risks and manage problems
in creative and unplanned ways (e.g., using fire hoses
to provide cooling water when all cooling pumps were
disabled).
Thus, there is no exact answer to the question
of how many guards are needed at a nuclear facil-
ity. Given a predictable threat (three outside attack-
ers and one insider) and a set of assumptions (well-
trained guards who react appropriately, attackers who
follow a standard attack plan, etc.), then a level of
safety can be calculated. However, as someone who
sat in on verbal reports following a simulated attack
against a nuclear power plant, I know that these as-
sumptions are routinely violated. Plants prepare care-
fully for these scheduled tests and, when weaknesses
are revealed, they make changes intended to reme-
diate the weaknesses, but they are not necessarily
re-tested at that point in time. Thus, any number of
weaknesses may remain, at least for some number of
years. Nor do we really know whether a “surprise” test
would achieve the same level of performance. Since
security forces face an intelligent opponent who is well
aware of the countermeasures being adopted, we can
be reasonably assured that terrorists will attack at a
time and place of their choice and in a manner di-
rected at whatever weaknesses they anticipate. If 19
terrorists can attack 4 airplanes, then 19 terrorists can
attack 1 nuclear power plant. Five guards are only “re-
dundant” if we assume that they (and more) are not
all needed at once.
The truth of the matter is that redundancy serves
to make us feel safe. At a time of public concern, policy
makers have “to let insiders and outsiders see that top
officials are doing something” (Sagan, this issue). Re-
dundancy is in part a persuasive argument addressed
to policy makers who allocate funds, plant employees
who must be motivated to do their jobs as well as pos-
sible, a public that needs reassurance in order to con-
tinue working and spending, and the terrorists them-
Carroll
selves, who may choose to attack a less formidable
target.
3. MANAGING FOR NUCLEAR SECURITY
However, if we want real safety beyond the per-
ception of safety, then we have to combine design log-
ics with operating logics. Sagan offers awareness as a
first step for opening a dialogue around the value of re-
dundancy, or how much is enough. But we can go fur-
ther. There are ways to address problems such as diffu-
sion of responsibility, complacency, and scope creep
that are linked to redundancy. Research on “social
loafing” (Karau & Williams, 1993) and accountability
(Tetlock, 1992) shows that responsibility does not dif-
fuse if people know they are individually accountable,
particularly to a diverse audience that may ask unpre-
dictable questions. Research on safety culture shows
that it is possible to maintain a questioning attitude
and avoid complacency (Carroll et al., 2002; Mearns,
et al., 2003; Reason, 1997). Above all, research and
experience show that safety comes from people, not
from technology alone (Weick et al., 1999). It is peo-
ple who notice vulnerabilities, design models, pay at-
tention to surprises, tell each other about discover-
ies and problems, create technical and organizational
innovations, and pass information across plants and
industries.
Far more resources have been invested in techni-
cal know-how than in understanding how people op-
erate within technical systems. Nuclear power plants
are required to spend millions of dollars on technical
safety analyses and safety equipment. On the people
side, we rely on procedures, training, and supervision
to make people do what is expected of them. Unfortu-
nately, we also need them to do more than is expected.
We want people to bring their experience, creativity,
imagination, and teamwork to the workplace. We can-
not “design in” safety through technical means alone;
discussions over safety have to engage the knowledge
and experience of a wide range of experts and prac-
titioners, including operations personnel and social
scientists. Policy makers should consider that invest-
ments in a greater variety of skills and viewpoints, a
reasonable amount of slack time for reflection and
observation, critical self-analysis that admits doubt,
willingness to improvise and learn from experience,
and managerial capabilities are just as important for
security against terrorists as they are for operational
excellence in general.
Redundancy as a Design or Operating Principle
REFERENCES
Carroll, J. S. (1998). Organizational learning activities in high-
hazard industries: The logics underlying self-analysis. Journal
of Management Studies, 35, 699–717.
Carroll, J. S., Rudolph, J. W., & Hatakenaka, S. (2002). Learning
from experience in high-hazard organizations. Research in Or-
ganizational Behavior, 24, 87–137.
Karau, S. J., & Williams, K. D. (1993). Social loafing: A meta-
analytic review and theoretical integration. Journal of Person-
ality and Social Psychology, 65, 681–706.
Mearns, K., Whitaker, S., & Flin, R. (2003). Safety cli-
mate, safety management practice and safety performance
in the offshore oil industry. Safety Science, 41, 641–
680.
Perin, C. (1998). Operating as experimenting: Synthesizing en-
gineering and scientific values in nuclear power produc-
tion. Science, Technology & Human Values, 23(1), 98–
128.
Perrow, C. (1984). Normal Accidents. New York: Basic Books.
957
Rasmussen, J. (1990). The role of error in organizing behavior.
Ergonomics, 33, 1185–1190.
Reason, J. (1990). Human Error. New York: Cambridge University
Press.
Reason, J. (1997). Managing the Risks of Organizational Accidents.
Brookfield, VT: Ashgate.
Sagan, S. D. (this issue). The problem of redundancy problem: Will
more nuclear security forces produce more nuclear security?
Risk Analysis.
Schulman, P. R. (1993). The negotiated order of organizational re-
liability. Administration and Society, 25, 353–372.
Tetlock, P. E. (1992). The impact of accountability on judgment and
choice: Toward a social contingency model. In M. P. Zanna
(Ed.), Advances in Experimental Social Psychology, Vol. 25
(pp. 331–376). New York: Academic Press.
Weick, K. E., Sutcliffe, K. M., & Obstfeld, D. (1999). Organizing for
high reliability: Processes of collective mindfulness. Research
in Organizational Behavior, 21, 81–123.
Wildavsky, A. (1988). Searching for Safety. New Brunswick, NJ:
Transaction Press.