See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/223642670

Nuclear power plant shift supervisor's decision making during

microincidents

Article  in  International Journal of Industrial Ergonomics · July 2005

DOI: 10.1016/j.ergon.2005.01.010

CITATIONS READS
63 1,146

3 authors:

Paulo Victor R. de Carvalho Isaac J. A. L. Santos

Nuclear Engineering Institute Comissão Nacional Energia Nuclear, Brazil
162 PUBLICATIONS   1,728 CITATIONS    26 PUBLICATIONS   573 CITATIONS   

SEE PROFILE SEE PROFILE

Mario Cesar Vidal

Federal University of Rio de Janeiro
72 PUBLICATIONS   630 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Descriptive Study of the Healthcare Assistance Provided by the Mobile Emergency Motorboat Service in Riverine, Coast and Maritime Regions View
project

Expanding the Reach of Health Surveillance in Detecting Epidemics and Disasters through the Monitoring of Social Media View project

All content following this page was uploaded by Paulo Victor R. de Carvalho on 08 December 2017.

The user has requested enhancement of the downloaded file.

 ARTICLE IN PRESS

International Journal of Industrial Ergonomics 35 (2005) 619–644

www.elsevier.com/locate/ergon

Nuclear power plant shift supervisor’s decision making

during microincidents
Paulo V.R. Carvalhoa,, Isaac L. dos Santosa, Mario C.R. Vidalb
a
Comissão Nacional de Energia Nuclear, Instituto de Engenharia Nuclear, Cidade Universitária,
Ilha do Fundão, Rio de Janeiro, RJ 21945-970, Brazil
b
Grupo de Ergonomia e Novas Tecnologias, GENTE, COPPE/UFRJ, Cidade Universitária, Ilha do Fundão, Rio de Janeiro, RJ, Brazil
Received 15 January 2004; received in revised form 9 November 2004; accepted 18 January 2005
Available online 29 March 2005

Abstract

The aim of this paper is to examine the cognitive processes through which operators make decisions when dealing
with microincidents during their actual work, and to determine whether they use a naturalistic or normative decision
making strategy. That is, do they try to recognize a microincident as familiar and base decisions on pattern recognition,
tacit knowledge, or condition–action rules (naturalistic), or do they need to concurrently compare and contrast options,
before selecting the best possible according standard operating procedures (normative)? The method employed for data
collection was a cognitive task analysis (CTA) based on operators’ activities. The main finding of this research was that
decision making is primarily based on naturalistic strategies. These findings contrast the normative behavior prescribed
by the organization’s work design and their standards of competency for training and evaluation operators work.
Relevance to industry: This study presents a situated method to describe how sharp end operators make decisions
during microincidents that occurs in normal operation, emphasizing how the sociotechnical environment affects their
cognitive strategies, which is one of the basic steps for an organization that wants to enhance the safety culture.
r 2005 Elsevier B.V. All rights reserved.

Keywords: Naturalistic decision making; Nuclear power plant operation; Cognitive strategies; Activity analysis

1. Introduction effects as seen in the Chernobyl disaster in 1996. In

Nuclear power plants (NPPs) are hazard envir-
onments where emergencies can have devastating
Corresponding author. Tel.: +55 21 22098196;
fax: +55 21 22098186.
E-mail address: paulov@ien.gov.br (P.V.R. Carvalho).
overall command of control room crew and of
handling any incident encountered, is the Shift
Supervisor (SS). The outcome of a crisis is
consequently dependent on the SS’s judgement,
decision making and situation awareness.
The aim of this study was to examine the
cognitive processes through which experienced

0169-8141/$ - see front matter r 2005 Elsevier B.V. All rights reserved.
doi:10.1016/j.ergon.2005.01.010
 ARTICLE IN PRESS
620 P.V.R. Carvalho et al. / International Journal of Industrial Ergonomics 35 (2005) 619–644
Shift Supervisors make decisions during micro-
incidents in normal plant operation, and to
determine whether they use a naturalistic or
normative decision making strategy. That is, do
they recognize the situation as familiar and base
decisions on condition–action rules (naturalistic),
or do they need to concurrently compare and
contrast options before selecting the best possible
(normative). Emphasis was on the individual Shift
Supervisors’ understanding of the situation and
the meaning he attached to the information or
events taking place. The method employed to
achieve this objective, was a cognitive task analysis
(CTA) based on field studies.
Investigations based on field studies during
NPPs normal operation are rare. Mumaw et al.
(1996) had already noticed this situation, indicat-
ing that the focus of the research has been on
emergency operations. They claimed that because
most abnormalities are preceded by normal
operating regimes, an understanding of people
performance during normal operation is critical.
After a set of field studies carried out in Canadian
nuclear power plants, Mumaw et al. (1996), and
Vicente et al. (1997) found that what makes
monitoring difficult is not the need to identify
subtle abnormal indications in a quiescent back-
ground, but rather the need to identify relevant
information against a noisy background. Their
findings emphasized the active problem-solving
nature of monitoring in NPPs, and highlight the
use of strategies for knowledge-driven and the
proactive adaptation of the man–machine inter-
face (control room) to support monitoring. Since
most of the operators’ adaptive strategies depends
on the SS’s authorization, to investigate the SS’s
decision making process can be seen as a comple-
ment of Mumaw, Vicente and colleagues’ studies.
Another issue to justify this investigation is that
the risk organizations performance criteria for
decision making (OPITO, 1997; INPO, 1997)
present elements that suggests a normative deci-
sion making model based on information gathered
from all available sources, appropriate resource
utilization, valid interpretation of information,
and valid action selection based on this informa-
tion. Additionally, a review of potential conse-
quences and probabilities against possible
response actions, the development of a plan of
action, and quick interventions are required. The
normative approach described by the industry
contrasts with research evidence that has indicated
that in emergencies, decision making is often based
on condition–action rules (Rasmussen, 1983) or
recognition (Klein, 1989). However, in normal
operation conditions, what happened with the
decision making processes?
2. Naturalistic decision making
Naturalistic decision making (NDM) is a
comparatively new term, referring to how people
make decisions in complex real world settings. The
study of NDM asks how experienced people,
working as individuals or groups in dynamic,
uncertain, and often fast paced environments,
identify and assess their situation, make decisions
and take actions whose consequences are mean-
ingful to them and to the larger organization in
which they operate (Zsambok and Klein, 1997;
Klein et al., 1993). The research approach is
holistic and ecological with emphasis on actual
decisions taken within real life contexts. NDM
researchers emphasize the study of experienced
people in order to gain insight into the way
decision makers utilize contextual information and
their domain knowledge, and to discover how
contextual factors affect decision making pro-
cesses. Decision making processes are believed to
be situation specific thus, it is imperative to
consider features of the context to understand
the decision. This objective distinguishes the NDM
ecological approach from the normative analysis
of judgment and decision making based on
laboratory experiments.
The main features of the naturalistic decision
tasks, which set NDM research apart from
normative studies, are outlined and discussed
below:
1. Ill structured problems: problems encountered in
real life dynamic environments are seldom
simple and easily understood. The decision
maker will normally have to gather more
information to construct hypotheses about
 ARTICLE IN PRESS
P.V.R. Carvalho et al. / International Journal of Industrial Ergonomics 35 (2005) 619–644
what is taking place and to be able to generate
options that might be appropriate responses.
Complex interactions and causal associations
may connect features of the situation to each
other.
2. Uncertain dynamic environments: goals often
change during a dynamic situation as the result
of shifting priorities. The dynamic conditions
may change what is important as the situation
evolves. Tradeoffs between different factors,
e.g. production and safety, have to take place,
often involving a great deal of uncertainty on
the decision maker’s behalf.
3. Shifting, ill defined or competing goals: natur-
alistic decision making typically takes place in
situations where information is incomplete and
imperfect. The decision maker may have
information about one part of the problem
but not about others. It is seldom that the
decision is dominated by a single, easily under-
stood goal. The decision maker is expected to be
driven by multiple purposes, some of which are
not clear and others, which will be contra-
dictory in nature.
4. Action/feedback/feedforward loops: dynamic
situations are characterized by a series of events
taking place over time. The decision maker
rarely has all the information desired available,
since incomplete, unreliable, missing, ambigu-
ous, and erroneous data is the order of the day
in most dynamic situations. On this basis, the
decision maker must take action in the early
stages, using anticipatory strategies or feedfor-
ward loops. This action may or may not be
appropriate but the feedback derived from the
resulting outcome may help the decision maker
to correct his/her plans. The feedback and
feedforward loops, however, can also compli-
cate matters. It may distort the interrelationship
between cause and effect in particular when a
time delay is involved. Action/feedback/feed-
forward loops may therefore be both of benefit
and detriment to the decision making process.
5. Time stress: decisions in NDM settings are
usually made under time pressure that has
several implications. Decision makers may
experience high levels of personal stress, with
the potential for loss of concentration and
621
fatigue thinking will change to less complex
reasoning methods. Decision making strategies
depending on exhaustive deliberation will be-
come unfeasible. This is in contradiction to
normative decision theories that are extremely
time consuming and involve complex calcula-
tions. The decision maker in an NDM environ-
ment simply does not have the time and
cognitive processing ability to concurrently
generate and evaluate options and simulta-
neously gather information as suggested by
normative theories (Amalberti, 1996).
6. Multiple players: many of the fields of interest to
NDM researchers, such as military or fire
service command and control involves more
than one decision maker. In fact, a large
proportion of the tasks involve some degree of
teamwork. The potential for breakdowns in
communication, dysfunctional relationships
and groupthink, always exist.
7. Organizational goals and norms: decisions are
not made independently from the organiza-
tional values. Organizations will have their
priorities and communicate them through
regulations, evaluations, reviews, or informal
emphases. The organizational culture is created
through these, setting an indicator of acceptable
practices, prohibitions and perspectives. The
decision maker is aware of what is expected,
and may act accordingly.
8. Poorly defined procedures: standard operating
procedures (SOPs) and emergence operating
procedures (EOPs) have been introduced in
many environments in order to describe to the
operators how they are supposed to do their
work. If good procedures exist, there is little
need for creative decision making, although a
problem may exist concerning which procedures
to select and implement. However, even when
the SOPs have been specified, operators may be
required to construct new procedures ad hoc to
deal with an incident not previously envisaged
and for which there are no SOPs.
Those characteristics came from the four areas
of research which dominate NDM, that are fire
ground (Klein et al., 1986), military (Hutchins,
1997), aviation (Orasanu and Fischer, 1997) and
 ARTICLE IN PRESS
622 P.V.R. Carvalho et al. / International Journal of Industrial Ergonomics 35 (2005) 619–644
medical (Gaba et al., 1995) decision making. With
this research, we wanted to investigate if a highly
structured organization, like a nuclear power
plant, share some of the characteristics that
defining NDM research.
In order to classify the SS’s decisions we use the
recognition primed decision model (RPD) (Klein,
1989). In the RPD model, rather than making a
concurrent evaluation of the relative advantages
and disadvantages of several courses of action (the
normative approach), expert decision makers in
actual situations select a course of action, which is
generated through recognition: the situation as
similar to a previous experience and evaluated for
its adequacy in that particular set of circum-
stances. RPD relies upon the serial generation and
evaluation of courses of action. A finding, which is
manifested in the observation that a potential
solution retrieved based on situational recognition
is not compared with an alternative in an attempt
to see which is most appropriate, but rather is
evaluated for its sufficiency, i.e., satisfying rather
than optimizing, in dealing with the problem
faced. This process is dependent on the expertise
of the decision maker. In their studies, Klein and
his colleagues found that managers from a number
Fig. 1. Basic methodological steps.
of different domains reported using a recogni-
tional style in making a large proportion of their
decisions (e.g. Kaempf et al., 1992; Klein et al.,
1993).
3. Methodological framework
Considering the ecological and holistic ap-
proach of the NDM research, the cognitive task
analysis (CTA) based on field studies is the natural
methodological framework for this research. CTA
is a ubiquitous description for a number of
methods used to elicit knowledge from workers
in specific domains. Seamster et al. (1997) defined
CTA as methods to identify and describe cognitive
structures such as knowledge base organization
and representational skills, and processes such as
attention, problem solving and decision making.
Among these methods, we adopted the ergonomic
work analysis (EWA), an approach used by the
French ergonomics school (e.g. Keyser de and
Nyssen, 1993), based on activity theory (Enges-
tron, 2000), in which the subjects are observed in
their actual work setting. Fig. 1 outlines the basic
steps of the methodology used.
 ARTICLE IN PRESS
P.V.R. Carvalho et al. / International Journal of Industrial Ergonomics 35 (2005) 619–644
This qualitative ethnographic framework implies
that the researcher collect empirical data while
interacting with people under study. The observation
in situ implies the daily taking of field notes
(supported by electronic media—audio and video
recording) that record naturally occurring talks and
interactions between observed actors. Particularly,
we want to stress that this method of field
observation is especially suitable for study of a
organization’s cultural issues, enabling access to the
backstage activities where workers hold the tacit
competencies that make possible all the cooperative
strategies essential to the accomplishment of daily
activity. This strategy of gathering data allows
grasping the vivid social scenes with accompanying
conflicts, misunderstandings, processes of negotiation
among actors, creations of consensual arrangements
to disrespect prescriptive rules y that often come
together with jargons, gestures, jokes, and so forth.
Finally, ethnographic research assumes that
there is no independence between the collection
of empirical data and all the interactions that
occur between the field observer and the insiders.
This is to say that all these interactions, occurring
between the researcher and people under study,
have to be considered as empirical data that will be
classified as part of the theoretical analysis.
4. The field studies
The research was based on a set of field studies
carried out in one nuclear power plant. Our first field
study was an exploratory study with two aims: (1) to
break the ice between researchers and operators; (2)
to get a preliminary understand of the strategies that
operators use to overcome the work environment
constraints. Pilot study results set the basis to the
second study, carried out during NPP startup and
shutdown. In this studies our aim were to classify
Shift Supervisor decision making according to the
RPD model and to check if this model is a useful
descriptor for this work environment.
4.1. The work environment
Since the NDM research approach attaches a
great significance to the environment as the root of
623
variation in decision making behavior, studying
the constraints in the environment, whether task
or work domain relevant to the operator, we
present a brief description of the NPP control
room, which constitutes the main operators work
environment.
The operation of the plant under study started
in the middle of 2000; however, most of I/C
equipment had been acquired 15 years previously
(due to delays in plant completion). Thus, the
control room and instrumentation/control system
use the process control technology developed in
the seventies at Germany (from where this plant
was purchased). Fig. 2 shows a schematic layout of
this control room that consists of stand-up control
panels, an operator desk with two workplaces, one
for the Reactor Operator and one for the
Secondary Circuit Operator, printers, a commu-
nication desk and bookshelves for operating
procedures. There are also work desks for the
Foreman and the SS, who also has a small room
that faces the control room. The control panels use
the mosaic technology, in which the traditional
hard-wired meters, strip-chart and control devices
(start/stop pushbuttons) are mounted according to
the functional diagrams of the various plant
systems. The alarms are presented in alarm
windows distributed across the panels and the
operator desks, according to the system to which
they belong. There are five CRTs placed in a panel
in front the operators’ desks: two CRTs present
plant variables (up to 8) in a bargraph and digital
form, other two CRTs present textual alarm
messages chronologically ordered in a list, and
the last one, located at the middle of the panel,
presents the automation system status. The two
operators also have a computer in their workplaces
for office work and for the safety parameter
display system, which was developed afterwards
(it did not belong to the original vendor contract).
From the pulpit, located in the middle of the room,
the Foreman reads procedures to the operators.
4.2. The pilot study: understanding the work
environment
In this exploratory pilot study, our aim was
to understand the work environment and the
 ARTICLE IN PRESS
624 P.V.R. Carvalho et al. / International Journal of Industrial Ergonomics 35 (2005) 619–644
Fig. 2. The NPP control room.
operator tasks, identifying some constraints on the
operators’ activities. We intended to identify
constraints according to the Schein’s (1999) three
levels of organizational culture: (1) artifacts:
‘‘visible organizational structures and processe-
syall the phenomena that one sees, hears and
feelsy’’ (2) espoused values: ‘‘strategies, goals,
philosophiesy’’ What the organization says about
itself. (3) underlying assumptions: ‘‘unconscious,
taken-for-granted beliefs, perceptions, thoughts
and feelings y the ultimate source of values and
action.’’
Method: two researchers, with some background
knowledge in nuclear operation, conducted ob-
servations in the control room, interviews with
operators, and other plant workers. The pilot
study takes about 2 weeks (10 days) and five
operator crews participated. During the study, the
researchers spent 4 h/day (40 h total) inside the
control room observing the operators’ work and
conducting debriefing interviews during work
intervals. The researchers also conducted inter-
views with field operators, maintenance, engineer-
ing personnel, and plant management. To collect
empirical data in this pilot study the only material
used were paper and pencil. The researchers did
not communicate to each other during the
observation period, and wrote independent sum-
maries of their findings. At the end of the
observation period, their findings were confronted
with and the confusing points were discussed with
operators.
Participants: The control room operator crew is
composed by four licensed operators—Shift
Supervisor, Foreman, Reactor Operator, Second-
ary System Operator—and 1 not licensed opera-
tor—the Auxiliary Panel Operator. The Shift
Supervisors and Foremen are Senior Reactor
Operators with ages ranging from 30 to 55 with
more than 10 years experience in NPP operation
(they were operators in the other NPP before they
came to this plant). Some Reactor and Secondary
Circuit Operators—ROs (age from 30 to 40), were
also experienced operators (5–10 years), but others
ROs were recently employed workers (1.5 year)
with ages from 20 to 25, and no previous
experience in plant operation. During 1.5 years
work time in the utility they had been trained.
 ARTICLE IN PRESS
P.V.R. Carvalho et al. / International Journal of Industrial Ergonomics 35 (2005) 619–644
4.3. Pilot study results
4.3.1. Artifact level
At the artifact level, we observe some hardware
constraints mainly related to the high automation
level and system complexity, both coupled with a
relatively poor information support. This is a
direct result from the paradigms of instrumenta-
tion and control (I&C) system design at the early
seventies (e.g. Bainbridge, 1983), when this plant
had been designed. Some of these constraints are
outlined below, and discussed in turn:
1. Poor human–system interface.
2. Poor information about the automation system
status.
3. Conventional alarm system design.
4. Communication systems.
5. Procedures.
1. Poor human– system interface: Fig. 3 shows
the problems with difficult display visualization
from an operator sitting position, and with the
inadequate position of graphic registers. The
minimum eyes’ height needed to see the last line
in the CRT of the main console from a seated
position is 1300 mm (inadequate for most of the
Brazilian operators), and the visual distance
between the operator and the CRT is 2230 mm,
that hinders the characters visualization, as
verbalized below:
Fig. 3. Visualization difficulties at the control room.
625
Shift Supervisor: ‘‘One, seven, five, seveny Is
it?’’
Operator: ‘‘One point eight.’’
Supervisor: ‘‘I cannot see it well! The guy who
design this y Or the guy sees very well, or y I
cannot accept that screen! Moreover, look, I can
see closely very well! You see, a yellow back-
ground, and some pale black letters. Terrible.
And I am not going to talk about the light
reflection effects y’’
2. Poor information about the automation system
status: the main reason for the highly automated
system is to avoid human intervention in emer-
gencies. According to the 30 min rule, after a
reactor unexpected shutdown (a reactor trip) there
is no need for manual actions for 30 min. Using
electronic logic gates, the technology available
when the I&C system had been purchased, the
automation system is composed by many building
blocks that act in order to operate and protect the
systems, subsystems, functions and equipment.
Considering the complexity of installation, the
number of automation loops is enormous and to
access automation system status, the operators
have only one display, in which they can see a logic
diagram that indicates the status of each logical
input/output of each automation building block.
When something goes wrong, the operator task is
to integrate all this information, from equipment
to system level, in order to have meaningful
picture about what the automation is really doing.
The difficult of this task was accessed in detail in
the other field studies presented in this paper.
3. Conventional alarm system design: the differ-
ent operational modes of the plant, startup,
shutdown, normal operation were not fully con-
sidered in the hardwired alarm system design.
Thus, during the change of operational modes, the
operators have to cope with many spurious and
nonsense alarms and they do not have the
possibility to bypass specific alarms or group of
alarms, without commit a procedure violation (see
Section 5).
4. Communication systems: voice communica-
tion links between the control room and the
plant plays a fundamental role for the
NPP operation and control. Field operators,
 ARTICLE IN PRESS
626 P.V.R. Carvalho et al. / International Journal of Industrial Ergonomics 35 (2005) 619–644
maintenance, instrumentation, engineering people
are in constant contact with the control room
operators by the various plant communications
systems. Communication activities also plays a
significant role in the Shift Supervisor job, since
that the Shift Supervisor is the main responsible
for the control room communications. Beside the
telephone system, the control room has two other
voice communication systems to the various areas
of the plant: a broadcast system, with loud
speakers spread around the plant and the inter-
communication system that connects the control
to some specific areas of the plant. Radio and
pagers are not aloud due possible electronic
interference with I&C. When the control room
needs to talk with some field operator, a call is
made at the general broadcast system and the field
operator has to go to the near intercommunication
system cabinet available to talk with the control
room. During the pilot study, we observe many
difficulties in this communication procedure be-
cause there are several blind points in the plant (no
loud speaker, or to noisy to be hear) and the field
operator have to stop their work to attend the
control room call.
Reactor/
Turbine trip
5 min after RTI yes
and Φ > 10-7A
no
∆Φ/∆t>0 yes
δt Φ > 10-8A
no
∆Φ/∆t>0 yes
δt Φ > 10-8A
no
Fig. 4. Procedure format—diagram.
5. Procedures: the plant procedures are written
on paper in English (which is not the mother
language of the operators). According to the
procedure format, (see Figs. 4 and 5) the operators
actions are presented in two ways: flow diagrams
to map the course of the system’s actions either
automatic or manual, and checklists, in which the
operators must fill the blanks with the detailed
manual actions they performed. This format is
justified to cope with the high degree of plant
automation: when the operators fill the checklists,
they become more alert about plant state. In the
Fig. 4 flow diagram, the rectangles S1 and S2
present actions that must be done when the
condition (in the decision box) is true. S1 and S2
are main actions labels that point out to another
page of the same procedure, where the detailed
steps of actions (manual or automatic) are
described as a checklist (Fig. 5). To copy with this
structure, the operators have to browse the
procedures continually, going from general to the
detailed parts and vice versa. This task (to browse
procedures) is performed by Foreman, who read
the procedures aloud to the Reactor and Second-
ary Circuit operators. The operators acknowledge
Insert all control rods S1
Increase boron
Concentration S2
Increase boron
Concentration S2
Function OK
 ARTICLE IN PRESS

P.V.R. Carvalho et al. / International Journal of Industrial Ergonomics 35 (2005) 619–644 627

Fig. 5. Procedure format—check list.

Fig. 6. Nominal operational mode.

the information received by repetition, and inform
the action carried out, configuring the nominal
operational mode (see Fig. 6). During the pilot
study, we observe that this nominal operational
mode is only possible when everything occurs
according to the procedural steps, that operators
have some difficulties with the procedure language,
and with the browsing process. Finally yet
importantly, the following question appeared
many times: ‘‘Do we (operators) really need to fill
all these check lists?’’
4.3.2. Espoused values
Espoused values give the official framework for
people behavior in the organization. The company
under which these field studies were carried out
 ARTICLE IN PRESS
628 P.V.R. Carvalho et al. / International Journal of Industrial Ergonomics 35 (2005) 619–644
operates the two Brazilian NPPs, delivering about
1800 MWe, which represents about 4% of the
electric energy produced in the country. The
current organizational arrangement was created
in 1997, from the merging of two companies with
different organizational cultures: an design &
engineering company, which was in charge of the
new nuclear units projects, and the nuclear
directorate of the state electric power utility. The
merger process itself was complex, suffers opposi-
tion from the unions and technical associations,
brought tensions to the workers, and some
experienced personnel were retired. When these
studies were carried out (summer 2001) the
merging process was already finished and the
organization, worried about the result of the
merging process, was developing a safety culture
assessment and enhancement program, with the
support and advice of the International Atomic
Energy Agency (IAEA) (INSAG-15, 2002). The
company has an official safety policy that gives the
maximum priority to safety issues, and the word
safety appears on the company’s vision and
mission statements. The safety culture IAEA
definition: ‘‘The assembly of characteristics and
attitudes in organizations and individuals which
establishes that, as an overriding priority, nuclear
plant safety issues receive the attention warranted
by their significance’’ (INSAG-4, 1991), was often
discussed during the safety culture self assessment
program, which involved most of the plant
personnel. Thus, it can be said that safety is one
of the most evident espoused values of the
organization.
4.3.3. Underlying assumptions
The underlying or basic assumptions are con-
structed by the workers during their daily activ-
ities, and have direct influence on the cognitive
strategies behind the decision making process.
The aim of the merger process described above
is related to the deregulation of the electric energy
market in Brazil: the privatization process of the
hydroelectric dams would be easier without the
nuclear generators in the same company. Under
this new deregulated market, the utility to survive
needs to supply the broadest possible service and
thus generate optimum profit on the invested
capital. Consumers want energy at lower possible
cost and greatest possible reliability, and the
ethical goal of society is to convert the fuel in
energy with safety and less possible damage to
environment. The above paragraph outlines some
conflicting goals in the environment in which
electric power generators, nuclear power plants
included, must act in the recent deregulated
market. From a reliable and not always equally
cost-conscious supplier (the former state com-
pany), with a defined market share, the power
producer will transform himself into a market-
oriented supplier. Thus, the liberalization of the
market demands a very different approach in
terms of organization and management (included
the risk management system), in order to built a
competitive organization with highest possible
profit, without jeopardizing safety. However, a
management change to fulfill all these require-
ments did not appear so quickly, and the new
production requirements emerged as underlying
assumptions, especially at the middle level man-
agers, SSs included. During the pilot study we
observe managers that offers informal gratification
(special lunches, extra holidays) if some work was
carried out below the time schedule.
Another issue that created underlying assump-
tions was the electric energy shortage due the low
level of Brazilian’s dams in 2000/2001. In this
situation, any NPP unexpected shutdown (or even
long time programmed shutdowns) could be a
serious problem to the Brazilian integrated electric
energy system. Thus, this situation causes more
pressure to the operator crews to keep the electric
energy production, avoiding unexpected shut-
downs, and to the instrumentation and mainte-
nance teams to do tests and maintenance
interventions as quickly as possible.
4.3.4. The Shift Supervisor’s work
Fig. 7 outline schematically the shared opera-
tional context where operators make decisions
during their work. The operators exchange in-
formation with each other and with the various
agents of the sociotechnical context in order to
construct their situation awareness and be able
to make correct decisions during their work.
Operators send and receive requests for tests,
 ARTICLE IN PRESS
P.V.R. Carvalho et al. / International Journal of Industrial Ergonomics 35 (2005) 619–644
Chiefs,
Planners
Plan
Espoused Request
values
Official
Rules
- Legislation
- Policy
DECISION MAKING
- Instructions
- Procedures
Action
Interfaces,
Equipment aids
Fig. 7. Schematic relation between a shared operational context and operator activities.
maintenance and general improvements to and
from the instrumentation, maintenance and en-
gineer workers.
The SS is ultimately responsible for the plant
operation. The SS tasks include the off site
communications activities and the authorization
for all tests/maintenance interventions carried out
during the shift. Thus, we saw that SSs are in the
center of the main operational decisions. In case of
doubt, operators asks to the Supervisor who gives
the final decision in many operational situations,
such as to authorize or not the use of a different
auxiliary procedure (sometimes not written, based
on rules created ad hoc), to waiting for other
operators and coordinating with, to solve opera-
tional problems, and so forth .
Because of this communication tasks, SSs have
direct contact with the higher hierarchical levels
(site operation manager) and with other people in
the plant (chief of instrumentation, maintenance,
engineering teams, etc.). Thus, SSs have informa-
tion and beliefs (underlying assumptions) different
from the other operator crewmembers. In this pilot
study, we noted the accumulation of roles by the
Supervisors: to follow procedures for checking the
actions performed by the Reactor Operator and
Secondary Circuit Operator; to think about the
process in conjunction with the Foreman, the
629
Instrumentation,
maintenance,
engineering
System
response
& state CONTEXT Basic
assumptions
Individual-
OPERATOR
& Group-
related:
- Attitudes
- Rules
- Routines
System Request
send
response
& state instructions
Field or area
operators
Operations Manager and the Shift Engineer; to
authorize the use of different auxiliary procedures,
especially in case of failure of some equipment; to
communicate with people around the plant to
authorize/suspend interventions.
From this pilot study, we have seen that NPPs
work environment share some of the character-
istics with the NDM research environment de-
scribed earlier, which motivated our next field
studies and set up their features.
4.4. Study 2: Shift Supervisor decision making
In this second study we combine the strongest of
the CTA methods—concurrent verbal protocols
analysis, video observations, field notes, multiple
analysts (4), hot reports (interviews during the
action) and an extensive data analysis phase
(coding exercises, cross checking among analyst
conclusions). Our aim was to examine the cogni-
tive processes through which Supervisors make
decisions when dealing with microincidents—MI
(Bressolle et al., 1996) during their actual work,
and to determine whether they use a naturalistic or
a normative decision making strategy.
We define microincident any event that pro-
vokes a rupture with the normal operation (the
nominal operation mode, Fig. 6), something that
 ARTICLE IN PRESS
630 P.V.R. Carvalho et al. / International Journal of Industrial Ergonomics 35 (2005) 619–644
brings the emergency of a new reality apprehen-
sion, sending the operators to work in a new type
of practical rationality. Microincidents are com-
plex entities with four basic properties: singularity,
unpredictability, importance (the discriminator
value from one event can be classified as MI),
and immanence to the situation.
Due the activities’ shared characteristics (a
finding from the pilot study), we used four analysts
at this time, two with nuclear operation back-
ground and the other two with an extensive
background in field analysis. From the pilot study
we saw that communication activities played an
important role in the SS work, and during the
programmed shutdown and startup of the plant,
the communication activities are intensified in
order to coordinate the operator crew work with
field operators and the test/maintenance teams.
Thus, the SSs’ workload increases during these
periods (e.g. Bourrier, 1999), and the probability
of the microincidents appearance will be greater.
In such way, we chose the planned shutdown and
startup of the NPP as the observation period for
this second filed study.
Method: In the study we use more than the
paper and pencil (to get field notes) we had used
before. Electronic media, such as audio and video
recorders has also been used. According the NDM
approach, the operators received only one instruc-
tion: behave as normal as possible in spite of the
presence of the analysts in the control room (this
became easier because operators already know the
study procedure and the analysts from the pilot
study). The procedure consisted of three phases:
(1) data collection, (2) data preparation, (3) data
analysis/coding exercise.
(1) Data collection: we use three video cameras
inside the control room and microrecorders in the
pocket of the four control room licensed opera-
tors. Each one of the four analyst followed one
operator, taking notes and changing the tape of
the microrecorder. Following completion and even
during work interval periods, debriefing interviews
were carried out, where some of the decisions
made by the SS were discussed. In order to probe
certain decisions more closely, we asked questions
such as: the cues which used to make a situation
assessment, the goals at particular parts of the
micro incident, whether any other courses of
action were considered when making a particular
decision, or whether the situation faced reminded
them of any previous experience.
(2) Data preparation: the verbal protocols
recorded were transcribed in chronological order,
identifying the operator and the other people who
talked with him. Considering that each operator
had one microrecorder, and the collaborative
nature of the control room crew work the
transcription from one operator can be used to
confirm (validate) the transcriptions from the
others. At the end of this transcription process,
we were able to rebuild all crew dialogs during the
observation period. The other set of data needed
for the study were the microincidents. The MIs
were identified after the concatenation of the field
notes coming from the four analysts (each analyst
followed one operator, so different events can be
perceived), together with the transcribed proto-
cols. First, we developed tables with the chron-
ological description of all observed events (e.g.
Table 1). The characterization of some of these
events as MIs was done based on the MIs
importance for the research objectives: the char-
acterization of SS decision making cognitive
strategies. For example, during the beginning of
the reactor shutdown (the period covered by Table
1) the only event classified as MI, was the last one,
the boiler problem. After MI selection, we
searched for all verbal protocol transcriptions
related to each MI, from the start of our
observations (with some MIs, discussions related
to the situation were noticed before the event) in
order to fill coding scheme tables.
(3) Data analysis/codification exercise: the main
purpose of the coding exercise was to identify
decision points in the verbal protocol and to
classify them according to whether they showed
evidence of serial or multiple generation and
assessment of options for action. The coding
scheme was based around the defining criteria of
Klein’s RPD model. Table 2 illustrates the coding
scheme. The analysis was conducted following the
principles of protocol analysis (Ericsson and
Simon, 1993) and content analysis (Krippendorff,
1980). The two analysts who have background in
NPP operation filled coding tables for each MI
 ARTICLE IN PRESS

P.V.R. Carvalho et al. / International Journal of Industrial Ergonomics 35 (2005) 619–644 631

Table 1
Event table during the beginning of the reactor shutdown

Time Op. condition Event

21:01 Load reduction Shift Supervisor announces in the general communication the beginning of load reduction
21:17 LAC 30 valve—does not open manually
22:16 Pump AP001 LAC 30—vibrating axis
22:25 Reactor Shift Supervisor announces in the general communication: suspend load reduction
410 MW according National Grid Regulator request by phone
22:31 Operators notice the oscillation in the electric power to the instrumentation
22:41 Foreman talk about spurious radiation alarms in the chimney
During the shift changeover the operators had already discussed this subject
22:44 Shift Supervisor announces in the general communication: restart load reduction
23:06 Turbine trip The turbine turned off; the plant disconnected to the electric power grid. Reactor
(325 MW) shutdown can be started through control rod insertion
23:34 Reactor trip Manual SCRAM. RO presses the button that liberates the control rods to fall inside the
reactor core (power 0.54%). Heat removal process begins
23:40 Heat removal RO notices that the fan of Boiler 2 is stuck, and boiler cannot start. Operator crew had
doubts about status of boiler 1(it has been under tests during previous shift)

Table 2
The initial coding scheme

Situation assessment (SA) Cues - SA/C Checklists - SA/CS Available checklists

Information sought Actively seeking information, not readily

provided without encouragement
SA/CI
Goals - SA/G Desired outcomes, sub and overall goals
Expectancies Related directly to the incident. Generating
hypothesis about how the incident might
develop. Based on deductive reasoning
SA/E
Mental imagery Visualization
SA/E(MI)
Action - SA/A Course of action generated based on
recognition of the problem, including SOPs
Evaluate action (EA) Run through action plan mentally. Referring
to a particular response plan.
Mental simulation Mental simulation of action plan
EA/MS
Modification - EA/ Reassessment and modification of action plan
M
Story building (SB) Use cues that won0 t directly fit present model
of incident. Story build from: knowledge of a
similar situation and expectations about the
situation
Option generation (OG) Generation of multiple options to deal with the
incident
Decision point (DP) Shift Supervisor makes a conscious decision to
act

independently. A reliability analysis of the coded was made, according to the verbal protocols
protocols based on simple match reliability coeffi- reliability analysis used in several NDM studies
cients (0 to 1) for each coded category of Table 2 (Calderwood et al., 1987; Klein et al., 1989). When
 ARTICLE IN PRESS
632 P.V.R. Carvalho et al. / International Journal of Industrial Ergonomics 35 (2005) 619–644
the two coders select the same verbal protocol in
one category, a match was achieved. The reliability
coefficient for each category was obtained trough
the division of the number of matches in the
category by the total number of verbal protocols
assigned to the category. Reliability analysis aims
the verification of the consistency of the coded
categories and especially in which categories
essential agreements—reliability coefficient greater
than .80 (Klein et al., 1989)—had been achieved.
Based on the reliability analysis findings, the more
fine-grained analysis was abandoned in favor of
the more meaningful distinctions among cate-
gories, as we will discuss in the results section.
Participants: almost the same operator crews  MI 2—Authorization/problems/suspension of
who participated in the pilot study participated in
this second study. We observed one crew during
the reactor shutdown for 12 h and other two crews
during the startup tests, reactor criticality up to
full power, more than 18 h.
4.5. Study two results
According to the proposed method, we will
divide the results presentation in (1) MIs descrip-
tion, (2) coding exercises results.
4.5.1. MIs description
Table 3 summarizes the six MIs identified
during the startup and shutdown of the plant.
Next, these MIs will be described with the help of
some of the protocols gathered.
MIs during the plant shutdown:
 MI 1—Difficulties to startup the boiler 1: the
boiler is needed during the shutdown of a
nuclear reactor in order to provide steam for
Table 3
The microincidents
Reactor shutdown—crew 1 Pre startup tests—crew 2
MI-1. Boiler 1 startup MI-4. Incompatibility between
procedure requirements
MI-2. Instrumentation tests
MI-3. Reactor heat removal circuit
blockade after pump shutdown
some utilities after the reactor trip, when the
main heat source is lost. This MI began when
the field operator noted that there was a scaffold
near the boiler 1. Because of the scaffold, the
operators were not sure if the maintenance
services in boiler 1 were really ended, despite the
information they got from the work permit
document. Thus, the operator decided to start
the redundant boiler 2, but this boiler did not
start. Finally, after send the panel operator to
the boiler area to get more information, the SS
decided to start the boiler 1. Fig. 8 presents
some of the verbal protocols transcriptions
related to this MI.
tests: several events were related to the Super-
visor’s decisions to authorize, or suspend
instrumentation tests if they jeopardized the
reactor refrigeration process. According to the
task schedule, instrumentation tests should be
initiated at 24, just after the reactor trip at 2330.
However, the test procedure indicated that the
tests should be done only when the reactor
reached the subcritical cold state. The subcri-
tical cold state could only be reached after at
least 6 h from the trip. In such way, there was an
inconsistency between 2 written information:
the test procedure and the shutdown task
planning. In the first moment, the Supervisor
decided to authorize the tests, against the test
procedure in order to accomplish with the
planned tasks. 2 h latter, after spurious alarms
and blocks in the automatic system, the Super-
visor under the pressure of the operator crew,
decided to suspend the tests. At 0440, after the
start of the Reactor Heat Removal System, the
Supervisor tried again to carry out the tests.
15 min later the same problems appeared
Increasing reactor power—crew 3
MI-5. Limitation system parameter
oscillation
MI-6. Leakage in the MKF tank
 ARTICLE IN PRESS

P.V.R. Carvalho et al. / International Journal of Industrial Ergonomics 35 (2005) 619–644 633

Fig. 8. The boiler microincident.

(blockades on the automatic system). At 0553, spray system. Some of this discussion is
he decided to suspend the tests again, until the presented below.
end of the reactor refrigeration process, in the
morning. Fig. 9 presents some of the verbal
RO arriving in the shift: ‘‘We saw that thing
protocols transcriptions related to this MI.
before, and we reached the conclusion: (y)
 MI 3—Reactor heat removal circuit blocks after
when you turn off the pump (y) the pressure of
the main refrigeration pump shutdown: according
the primary circuit becomes higher.’’
to the reactor shutdown procedure, two of the
Foreman arriving: ‘‘I read in the Science
four reactor refrigeration pumps have to be
Description (part of plant technical specifica-
turned off to finish the reactor refrigeration
tion) that the core DP is 3 bar. Now, we have 2
process (in this phase they become the main
bar, it is possible, it is colder.’’
reactor heat source). Just after the operator
RO arriving: ‘‘On the other time, we lowered the
turned off the pumps, one of the three heat
primary pressure here.’’
removal circuits in operation (there are four
Foreman arriving: ‘‘I told him that I will do, but
circuits, but one was already out of operation)
he (Shift Supervisor arriving) is not sure about.’’
was unexpectedly blocked by the automatic
RO arriving: ‘‘It can be lowered by hand! Here I
system, configuring the MI. The operators
know a trick that you can still have control, with
realized that the block was due to the overlap
lower pressure.’’
in shutdown curve pressure limits. For some
Foreman leaving: ‘‘You gonna cheat the big
unknown reason for the operators, the pump
brother!?’’ (Laughing)
shutdown caused a slight increase in the primary
circuit pressure (33–36 bar), enough to trigger
Now the Foreman made an observation about
the automatic system (the set point is 34 bar).
the stage of the refrigeration process.
This occurs at the end of the shift and operators
from two operation crews discussed the problem Foreman leaving: ‘‘We do our job well. You see, it
and the strategy to restart the refrigeration is in 69 degrees, up to 11(hour) to arrive to 50
circuit. They discussed 2 options: to open the (degrees), I think is reasonabley’’
breaker to bypass physically the interlock, or to RO leaving: ‘‘But it is like this for a long time y
lower the primary circuit pressure using the and it is not getting colder.’’
 ARTICLE IN PRESS
634 P.V.R. Carvalho et al. / International Journal of Industrial Ergonomics 35 (2005) 619–644
Fig. 9. The tests microincidents.
Foreman arriving: ‘‘Ask him to open there, the guy
is in a hurry!’’
RO leaving: ‘‘Order to open the breaker.’’
To open the breaker implies in automation
system bypass and the immediate start of the
refrigeration circuit.
Foreman arriving: ‘‘Wait! (y) the pressure is
high!’’
Foreman leaving: ‘‘I also think it is high. Anyway,
we have to lower the pressure a little bit. I will talk
with (Supervisor name).’’
Foreman leaving: ‘‘The (RO name) is talking that
this problem had already happened before with him.
In addition, they lowered the pressure manually ... in
the core. We change to manual, spray and later we
go back y just to bypass the block y and to return
with JN (heat removal circuit).’’
Supervisor arriving: ‘‘Ok. Let’s go.’’
After the pressurizer spray actuation, the
primary circuit pressure dropped, and the heat
removal circuit resumed its operation.
This MI exemplified some difficulties brought by
the automation system design (a prescribed formal
construct) and the shutdown procedure for the
operators. The unexpected block forced the
operators into a complicated diagnosis process to
restore the refrigeration capability of the heat
removal system. Based on their previous experi-
ence they mutually construct their situation
awareness (what should happen next) in order to
make decisions. They discussed ways to bypass the
interlock system, routine violations according to
Reason’s definition, or a performance improve-
ment, to restore the refrigeration circuit. The main
proposals were to start the circuit directly from the
circuit breaker, or a more subtle approach—the
one chosen—to use the spray system to lower the
primary pressure in about 2 bars, going inside the
limits of shutdown curve set point.
In this MI we also observed that the knowledge
acquired by some teams in unexpected situations
are not shared throughout the operator crews,
unless among operator crews who participate in
the repetition of the situation. This MI inspired
some questions such: is the shutdown curve set
point too tight? is there any problem with the
accuracy of the pressure transmitters? Is there
really need to have this interlock, in what
situations should it be important (since it is clearly
a slang during the reactor shutdown)? The
 ARTICLE IN PRESS
P.V.R. Carvalho et al. / International Journal of Industrial Ergonomics 35 (2005) 619–644
problem here is that such questions are dealing
with automation system design and philosophy of
the reactor control system whose answers lie
somewhere in the sociotechnical system (NPP
vendor, engineering department, regulator), but
certainly they are beyond the scope of the
operators. As such, the operators seem to have a
no explicit assignment to deal with such situations
as best as they can.
MIs during the plant startup:
 MI 4—Incompatibility between procedures: the
operators follow the procedure for the execution
of the primary circuit leakage test, before the
reactor startup. According to the test procedure,
the operator must electrically disconnect the
pump during the test. However, with the pump
turned off and the output valve closed and
locked, why electrically disconnect the pump?
The problem here is if the operators electrically
disconnect the pump, they also have to proceed
with the pump test that takes about 8 h. This test
has already been done the day before, and it is
not programmed to be done again in the task
planning. This situation raise the question about
the meanings behind procedures requirements
one more time and their temporality, as
verbalized by the Foreman in his dialog with
the RO:
Foreman: ‘‘Which is the relationship between the
Operation Manual, blocking this thing and the
test (pause)? We are doing this test in the same
region of the Operation Manual. You are here, in
this point, then he (the procedure writer) knew
that the valve already blocks it. Theoretically, the
person who wrote the procedure y he knew the
plant condition. Then there is a redundancy in
what is written here. I am wondering if there is
something more behind, in order to have this
double block: the pump and the valve. Which was
his (procedure writer) concern?’’
RO: ‘‘The valve, he is not mentioned it here y
no.’’
Foreman: ‘‘I know. The valve, he is requesting
there ... there, in the Operation Manual. He
suggests that this valve has to be closed during
635
the test. Because like him (the procedure) said,
the valve has to be open after the test. During the
test, it has to be closed. Then, why, if it is already
closed there, did the guy insist on the electrically
disconnect requirement!?’’
When the Foreman said: ‘‘y the guy who wrote the
procedure y he knew the plant condition’’, he
means that the person who elaborated the test
procedure should have specified the tests that need
to be done for reactor startup in an orderly way,
according to the plant state and previous tests
(there would be no sense in making pump tests
twice). We can see the difficulty to follow the
procedures in complex systems. In this case
operators face temporal incompatibility among
three documents: operation manual, test proce-
dure and task planning. Each one of them was
elaborated by a different group of specialists, in
different organizations, in different moments, and
in completely different contexts; however, even so,
to be followed without the need of human
intervention (interpretation), they must be compa-
tible with the rationale of the operation.
At the end of about 20 min discussion, and
consultation of engineering/instrumentation
diagrams, the Shift Supervisor decided not dis-
connect the pump electrically during the test,
against the test procedure requirement and ac-
cording to the mutual situation awareness they
achieved.
 MI 5—Limitation system parameter oscillation:
to solve the problem of an oscillation in a
limitation system parameter in low power
(12.5%), the RO, after phone consultation with
instrumentation technicians and Supervisor
authorization, increments the reactor power in
5%, in order to see if the oscillation stops. The
transcription below begins when the RO, after
stopping the power increase procedure, explains
the parameter oscillation to the Supervisor.
RO: ‘‘It is oscillating, man! y The problem
began y Do you see y From 12.5 it jumped at
once to 28! When it was in 12.5 it had to be
changed to 17.5 and it did not change y it
became stuck! Then, it was just passing through.
 ARTICLE IN PRESS
636 P.V.R. Carvalho et al. / International Journal of Industrial Ergonomics 35 (2005) 619–644
From that point, it went to 28, and now y Now it
is oscillating, look!’’
Supervisor: ‘‘It came back to 20.’’
RO at the phone: ‘‘Hello, control room. (Pause)
It is because the power is 12.5%, ok. (Pause) The
water flow is less than 10%. (Pause) Oh! Yes? I
do not know, I will look with the (SS name). Ok,
bye.’’
RO: ‘‘(Supervisor name)! He suggested increas-
ing 5% the power, to see if the feedwater flow
leaves the low flux region.’’
Supervisor: ‘‘Ok. You can increase.’’
This transcription illustrates the importance of
the operators’ mental model and situation aware-
ness to the installation performance and safety.
When the RO noticed the parameter oscillation, he
immediately suspended the power increase, pre-
sented the problem to the Supervisor and asked for
help from the instrumentation technician. The
oscillating parameter is formed by many signals.
One of them is the feedwater flow, which was low.
The instrumentation technician recognize the
pattern in which small variations in a low flow
can give spurious value signals, inferring this
should be the cause for the oscillation. The low
flow could only be increased, with an increase of
reactor power—but power increase was stopped
due to the oscillations—creating two conflicting
conditions. The final decision, to increase the
power in 5% has been successfully applied.
 MI 6—Leakage in an auxiliary water tank
(named MKF tank): the tank begins to drain,
threatening the power increase process (trip risk,
reactor with 33% of power). The MKF tank is
one of many examples of auxiliary equipment
that do not have alarms in the control room
panels. The only indication in the control room
is its level in one strip chart indicator located on
the auxiliary panel. Operators discuss strategies:
to fill the tank to avoid reactor trip (automatic
reactor shutdown), or to discover the cause of
the leak.
SCO answer the phone: ‘‘Control room speaking
y How critical is it? y Who is talking? y OK,
bye.’’
SCO calls for Supervisor: ‘‘Problem (SS name)!
The MKF’s water is going down! Nobody know
from where! Are there trip risk?’’
Supervisor: ‘‘Can we fill the tank while we do not
discover where the leak is?’’
The Supervisor immediately suggests an alter-
native to avoid the trip: to fill the tank. At this
moment, operators, instrumentation and mainte-
nance technicians went to the auxiliary panel to
look at the MKF tank level.
Foreman: ‘‘On the other time, it was a rubber hose,
do you remind?’’
SCO: ‘‘Yes. It was the conductivity meter’s hose, but
there was a conductivity meter there, at that time.’’
Phone rings—SCO at the phone: ‘‘Give me a break,
go to MKF because the level is going down faster! I
know that (name) is already there, but help him,
otherwise we will a have trip soon! OK, bye.’’
While the SCO tries to understand the leak by
sending more people to the tank, the Supervisor
look for ways to fill the tank.
Supervisor: ‘‘There is an input here, look! I am not
sure if it is to fill the tank. There must be some place
to fill water in that shy!’’
SCO: ‘‘To where this water is going to?’’
The SCO rejects the Supervisor’s dialog and still
tries to get more information to understand the
situation.
Supervisor: ‘‘No, the problem is y Until we
discover to where the water is going yWe will have
a trip!’’
They did not have a trip. The field operators
found and corrected the source of the leak—a drain
valve that remained open after maintenance work.
4.5.2. Coding exercise
The coding scheme initially used (Table 2)
included categories and subcategories such as
Situation Assessment (goals, cues, actions, expec-
tancies, information sought, mental imagery),
Evaluate Action (mental simulation, modifica-
tion), Story Building, Option Generation and
Decision Point. Table 4 depicts a sample analysis
of the boiler MI dialogs from one coder.
 ARTICLE IN PRESS

P.V.R. Carvalho et al. / International Journal of Industrial Ergonomics 35 (2005) 619–644 637

However, this initial coding scheme was found
to be too complex for the protocols gathered (they
were also too complex). This was illustrated in the
low reliability coefficients calculated, in some
subcategories as such as a .54 for story building.
A simplified coding scheme consisting of the main
categories i.e. situation assessment, evaluate ac-
tion, story building, option generation, and
decision points, were utilized. Unfortunately, even
that proved too difficult to apply and reliability
coefficients only improved slightly. The lowest
coefficient again was story building at .59 and
decision points remained the highest at .93.
Because of the problems associated with the
coding scheme, emphasis was placed on the
decision points (the highest reliability coefficients)
and how the decision was taken, as this was central
to Klein’s theory. Klein’s et al. (1993) observed
that the RPD decision making is more than a
simply recognition cognitive process: ‘‘we assert
that the decision is primed by the way the situation
is recognized and not completely determined by
that recognition’’ (p. 141). In addition, Kaempf et
al. (1992) stressed that central to the process of
RPD is an evaluation of the generated option and
an understanding of the ways in which the chosen
course of action will benefit the decision maker.
Thus, to better understand the decision points
found, we develop tables (see Table 5) for each
decision point, were the main features of each
decision (decision, input, instigated by, involved,
goal, reason, options and consequences, time) were
described. In Table 6 we present an example for
one decision point of the Boiler MI. At the end of
this process, we classify 15 SSs’ decision points
across the 6 microincidents analyzed. Twelve of
these decision points were classified as RPD style,
whether three of them multiple options were
considered to make the decision (see Table 7).
5. Discussion
5.1. Shift Supervisors decision making style
Clear examples (20%) of multiple generation
and concurrent evaluation of options were found
in the protocols (and were backed up in the debrief

Table 4
The coding of the boiler microincident

During shift changeover, we got the first protocols and decision regarding this MI
PO: ‘‘The boiler, 1t has been finished, is under test.’’
Situation assessment/sough of information: the boiler 1 was prepared in the previous shift
RO coming: ‘‘OK.’’
Foreman: ‘‘So, theoretically it is available.’’
Situation assessment /Goals: the operator needs the boiler available to shutdown the plant
RO coming: ‘‘But, is it hot?’’
Situation assessment /Mental simulation: some time is needed to heat the boiler
PO: ‘‘Ok. I’m going to heat.’’
Evaluate Action: Panel Operator evaluate steps to heat the boiler face operational needs
RO coming: ‘‘Because we will need it today!’’
PO: ‘‘I’m going to start the boiler 2.’’
Decision point—without option generation
2 h later, the operators realized that the boiler 2 could not be used. At this moment, we got the transcription below
SS: ‘‘Look at the state of this f... boiler! What is really happen there?! ’’
Situation assessment/sough of information: the former satisfactory action is no longer valid (boiler 2 did not start); situation must be
reevaluated to establish a new course of action
30 min later Panel Operator returns form boiler area
PO: ‘‘There is a scaffold there that he put sustaining the tubes (interruption)’’
SS: ‘‘Why this thing impede to start the boiler?!’’
PO: ‘‘No, nothing, nothing.’’
SS: ‘‘So let’s start the boiler!’’
Decision point without option generation: If the boiler 1 did not start, a new course of action should be generated and evaluated
 ARTICLE IN PRESS

638 P.V.R. Carvalho et al. / International Journal of Industrial Ergonomics 35 (2005) 619–644

sessions) which suggested that the design of the they called condition-action rules. ‘‘These deci-
study did not suppress evidence of such a decision sions depend on if...then rules. Cognitive demands
making strategy. These decisions occurred during require recognition of the condition and retrieval
the Reactor Heat Removal Blockade MI, and
during the Incompatibility Procedures MI. In Table 6
these two MIs time pressure were not important: The boiler MI decision
in the Reactor Heat Removal Blockade, the Category Boiler MI
refrigeration process was almost finished, and the
two operators crews together, during the shift Definition
changeover, spent the time they needed to discuss
Decision Start boiler 1 (Shift Supervisor)
the problem; in the procedure incompatibility MI, Input Boiler 2 did not start
the reactor was in the shutdown mode, during the Boiler 1 available (?)
pre startup tests. Instigated by Panel Operator in the field
Before concluding that SSs make the majority of Involved Panel Operator, Reactor Operator,
Foreman, Field Operator
their decisions (80%) using a recognitional strat-
Goal To have an operational boiler
egy, however, it is necessary to look more closely Reason To produce steam for plant users after
at the way that this figure has been generated in reactor shutdown
this study. Options Stop reactor refrigeration procedures until
The main criterion for a decision being classified a boiler is available
Consequences Delay in the task schedule
as RPD Style was that it should have been made
Time 30 min
without any other courses of action being con-
sidered. However, it may be the case that the set of
decisions, identified using this classification, con- Table 7
tains some decisions, which, on the face of it, Summary of the SSs’ decision styles
appear to be RPD, but are in fact reliant on a
Decision Recognition primed Multiple
cognitively slightly less sophisticated mechanism.
points during decision (RPD) style options
Orasanu and Fischer (1997) sowed the seeds of MIs (only one option)
doubt when addressed a similar issue in a paper
based upon their work on decision making in the Shutdown 10 9 1
Startup 5 3 2
airline cockpit. They had identified a class of
Total 15 80% 20%
decisions that they suggested were based on what

Table 5
The characterization of the decision points

Category Definition

Decision The decision taken: leading to a course of action (CoA), to do nothing, or to wait. For example, to
shut down the process or to wait to see how the incident escalated
Input Information leading to an altered assessment needing a solution. Identification of when the problem
solving related topic was introduced, and what new factors caused the change
Instigated by Who identified the need to tackle the problem?
Involved Team members involved from problem identification to decision making
Goal The objective of the decision. Stated verbally or inferred by the researcher
Reason Based on the goal. For example, the goal may be to shutdown the process, the reason was to
minimize escalation potential. Could be stated but frequently had to be inferred
Options and consequences Options available as alternative means of resolving the problem identified. This could be not to do
anything or to wait. The consequences referred to what would happen if these were selected in
contrast to the chosen CoA. Options and consequences could be stated but mostly had to be inferred
Time The time taken from when the problem was identified until the decision was made
 ARTICLE IN PRESS
P.V.R. Carvalho et al. / International Journal of Industrial Ergonomics 35 (2005) 619–644
of the associated response. These decisions are
most similar to Klein’s RPDs, but are prescriptive
in the domain. That is, they do not depend
primarily on the pilot’s past experience with
similar cases, but on responses prescribed by the
industry, company or FAA.’’ (Orasanu and
Fischer, 1997, p. 346)
Initially it was not clear whether the application
of condition–action rules (included in operating
procedures or implicit in the memory) could be
taken to represent a recognition-primed decision.
It might be suggested that since the basis of RPD is
the process of situation assessment, the application
of an appropriate rule following such a process of
situational recognition might be taken to be an
instance of recognition primed decision making.
Therefore, it could be suggested that application of
an rule certainly can be said to come from a
recognition primed decision provided that it is not
a ‘‘blind’’ choice based purely on recognition of
the type of situation where there are no other
option available.
To know which of the decisions identified in this
study were based on ‘‘blind’’ condition–action
rules it must be distinguished which of the chosen
course of action (CoA) were based on operational
procedures. According to the research method,
based on MI emergence, there is no operational
procedure available to handle each MI. Questions
related to blind choices, were also probed in the
debriefing sessions with a view to assessing the
rationale behind the application of the chosen
CoA (to understand the ways in which the chosen
course of action will benefit the decision maker).
In this study we found that SSs decision making
during microincidents is primarily (80%) based on
pattern recognition and implicit condition–action
rules according to other findings in NDM research
(Lipshitz et al., 2001). These rules appear derived
from experience and training rather than from
Standard Operating Procedures. As a result SSs
select a course of action which is generated
through recognition of the immediate situation
as similar to a previous experience and evaluated
for its adequacy in that particular set of circum-
stances and beliefs, relying upon a serial genera-
tion and evaluation of courses of action according
to the RPD model. A finding that is manifested in
639
the observation that a potential solution retrieved
based on situational recognition is not compared
with an alternative in an attempt to see which is
most appropriate, but rather is evaluated for its
sufficiency, i.e., satisfying rather than optimizing,
in dealing with the problem. The number of
options available to the Supervisor is limited by
organizational constraints: hardware (man–ma-
chine interface, poor information about automa-
tion system status), software (detailed and
sometimes incongruent procedures) and liveware
(pressure to accomplish the tasks demands, under-
lying assumptions, see next section). Fig. 10
outlines a simple model of SSs decision making
process during microincidents. We must note that
the time availability (seeing as the perception
people has about the time needed to do the job)
and constraints/resources available are the key
issues that aloud a more or less extensive reflection
about the situation.
5.2. The underlying assumptions and SSs decisions
Espoused values are what the organization says
it wants to be and do, usually generated by high-
level management based on industry insights. They
often take the form of slogans, posters and mission
statements (safety culture definition, for instance),
plans, procedures etc. designed to promote certain
types of behavior, attitudes or expectations. They
can assist change by becoming a memorable
prompt to thought or action. In particular, the
standards and criteria set out by agencies pair
reviews, such as the IAEA (OSART Guidelines)
and WANO (performance objectives and criteria
(PO&Cs)) are intended to hold the status of
espoused values and the basis for the organization
work design. However, even whether sometimes
espoused values are readily adopted and quickly
become artifacts, the field study suggested that
espoused values are not always matched by
situated action. At human performance level, the
key to understanding the behavior lies one cultural
level down, with the underlying assumptions.
Based on their underlying assumptions and socio-
technical constraints people modify the exposed
values creating new ways to do their jobs that go
well beyond their prescribed work tasks.
 ARTICLE IN PRESS

640 P.V.R. Carvalho et al. / International Journal of Industrial Ergonomics 35 (2005) 619–644

Fig. 10. A model for SSs’ decision making process.

According to the organization work design, the
SSs are the ultimate responsible for the control
room operation and safety. They also should serve
as the main control room communication channel
to the other parts of the plant, and even outside
the plant. Because of such task assignment,
Supervisor accumulates many roles, technical and
administrative. He must follow a procedure, for
checking the actions performed by the ROs and
SCOs and think about the process beyond the
instructions to give the final authorization for the
implementation of the ad hoc reconfigurations, as
we saw in almost all MIs. Together with the
Foreman, the Supervisor helps the operators in
organizing their access to resources (binders,
operations sheets and logs, for example), pass
instructions to panel operators, maintenance,
instrumentation/test and engineering people (see
 ARTICLE IN PRESS
P.V.R. Carvalho et al. / International Journal of Industrial Ergonomics 35 (2005) 619–644
for instance the boiler and tests MIs). He also co-
ordinates contingent reunions to collectively solve
problems caused by the MIs, exchanging informa-
tion and diagnoses/prognoses on the dynamics of
the system, criticizes other’s actions and receive
critics: ‘‘This thing y test liberation, it is alarm all
the time, man.’’ Knowing the operators were right,
he used his sense of humor to lower the tension:
‘‘We are in trip risk!’’, when the reactor had
already been tripped. He assists the operators not
just when asked, but most of the time sponta-
neously. It is in the case of spontaneous assistance
that the operators best exhibit the (ad hoc)
innovations concerning their activity. See for
instance the Reactor Heat Removal Blockade
MI, when the RO tells: ‘‘y I know a trick that
you can get the high control, with lower pressure.’’
In the shutdown period the Supervisor work-
load increases enormously, since beside the tasks
described above he also must sign and authorize
all work permits related to the operation. As we
saw, just after the reactor trip many people came
to the control with their work permits asking for
Supervisor authorization, as in the case of the
instrumentation tests: ‘‘Is it already shutdown? Are
we already liberated?’’
Plant managers, especially in the actual harsh
environment in which NPP has to operate, impose
other constraints on the Supervisor activity.
Primarily conceived as a government strategic
issue, NPPs received their budget from state, no
matter how much energy they produced. Nowa-
days they must sell the energy to the market in
competitive prices in order to survive. The Super-
visor acts as bridge between managers and
operators, workers with naturally diverse cultures
and assumptions. In the debrief interview after the
Tests MI, the Supervisor shed some light to the
situation: ‘‘Those tests, I should not have author-
ized, ok. I should not have authorized! However,
people tried to do the tests, although they have to be
done when the plant was in subcritical cold state. In
the test procedure, this is one of the requirements.
Nevertheless, the Plant tried to do the tests, in
advance. Because we have a very shortstop period!
In such way, I took the challenge and I authorized
the tests. But in the middle of the wave I felt it was
impossible and I ordered to stop.’’
641
It can be said that all these cultural issues are
directly related to the ways SSs perceive the time
availability to do their tasks (see Fig. 10),
introducing time pressure in the relatively low
pace environment of the NPP control room. For
instance, the SS confronted with an inconsistency
between two procedures (the test procedure and
the task schedule), made decisions in order to meet
the task schedule, to do tests just before reactor
SCRAM (fall of control rods), ignoring the test
procedure that said to do the tests only when the
reactor reached the subcritical cold state. There-
fore, spurious alarms and blockade in some
systems due automatic interlocks disturb the
reactor refrigeration process, and the Supervisor,
under pressure of the operator crew decided, 2 h
later, to suspend the tests. The same pattern of
liberation/problems/suspension of the tests was
repeated 2 h later (see Fig. 9).
Another example of the same decision pattern
occurred with another crew during startup, during
the leak of the auxiliary (MKF) tank. The SS (not
the same who made the decisions mentioned
above) immediately tried to find ways to fill the
tank, in order to avoid reactor trip. The strategy to
send field operators to look what happened in the
field had been done by the SCO.
The parameter oscillation MI emerged when the
reactor had 12% of total power. The RO noticed
the set point oscillation and stopped the power
increase procedure. He talked by phone to an
instrumentation technician about the situation.
The technician told that the low water flow
characteristic in this power level could be the
cause of the oscillation. He recommended increase
the power in 5%, to see if the oscillation stops. The
RO immediately talked to the SS about the
technician suggestion.
Although to increase 5% in the power of a 1300
MWe nuclear reactor to do a quick hypothesis test
should not be an expected procedure, the SS had
accepted the suggestion in order to stop the
oscillation in the parameter, and continue the
reactor startup process. No standard procedures
had used, the situation is not expected to occurred
in the reactor startup, and tacit knowledge based
on analogies (low flux could lead to oscillation
patterns) was the strategy used to solve this
 ARTICLE IN PRESS
642 P.V.R. Carvalho et al. / International Journal of Industrial Ergonomics 35 (2005) 619–644
problem, biased by the (implicit) need of connect
the reactor on the electrical energy network as
soon as possible. One more time, only one course
of action had been generated and tested according
satisfaction criteria: if the oscillation did not stop,
another action would be considered.
The field observations revealed the same pattern
of decision styles during microincidents among
three SSs when confronted with sociotechnical
constraints as such as incompatible procedures,
pressures to do tasks (perception of time avail-
able), difficulties regarding communication, and
with the information about automation system.
The cognitive strategies used were mainly based on
recognition and condition–action rules stored in
memory are biased by underlying assumptions
constructed by operators, or groups of operators
(SSs had different assumptions when compared
with the other control room operators). The
sociotechnical constraints act on meta-recognition
processes (see Fig. 10—the arrow that goes direct
to action) limiting the option generation, selection
and analysis (normally only the one option is
selected and tested), i.e., the reflection about the
situation and the critical thinking strategies
(Cohen and Thompson, 2001).
6. Conclusions
There were several methodological problems
associated with the use of verbal protocols in this
field studies. Firstly, the complexity of NPP
control room operators’ activities and the large
number of personnel involved made it difficult to
distinguish between what would in fact be Shift
Supervisors decisions and decisions made by other
team members, only authorized by SSs. Secondly,
the use of an actual work scenario also compli-
cated the matter as a number of questions arose
where the answer would lead to many outcomes
depending on the investigation purposes. There
were simply too many variables to control.
Thirdly, verbal protocols lend themselves to serial
generation of thought. There may be some
unconscious parallel processing taking place,
which the method failed to acknowledge.
Fourthly, transcription of the verbal protocols
also appeared a time consuming process that did
justify itself only regarding the quality of the data
collected. Finally, the original coding scheme
developed, was far to complex to deal with the
protocols involved. This was reflected in the low
reliability ratings. Although decision points were
relatively simple to identify, situation assessment,
evaluation of action, story building, and option
generation, was difficult to separate because of the
nature of the protocols obtained.
Beside all those limitations, we claim that there
is no other way to grasp so vivid information
about human work as such as activity analysis,
since the human beings must work in the
organization social tissue they construct and where
underlying assumptions emerge. We argue that the
main objective of this kind of applied ergonomic
research seldom is to hypothesize and test for
statistical significance, in order to add to general-
ized principles of human behavior. Rather, such a
kind of applied research tends to emphasize the
naturalistic (local) environment and what is
imperative within that environment to the indivi-
dual performing a particular task.
This does not mean the existence of a direct
causal relationship between organizational envir-
onment (as causes) and sharp end operators errors
(effects). What we are trying to say, based on the
results of this studies, is that patterns of actions
and decisions of sharp end operators can be
identified and theirs effects (positive or negative)
can be now discussed: in feed forward reflection
loops to avoid unintended effects that could
manifest themselves much later and in indirect
(complex) ways, as latent conditions. In trying to
understand how an accident occurs, we must
acknowledge that human actions cannot be seen
in a binary way, as correct or incorrect. Although
the correctness of the decisions made by SSs go
beyond the scope of this study, we showed
examples of many decisions that could be con-
sidered incorrect (like increase power to stop an
oscillation) if an incident occurs. However, they
can also be considered as an operational improve-
ment, since the course of actions selected solved in
effective ways the problems that the operators
faced at that moment. To capture this kind of
actions (that had not been captured in any other
 ARTICLE IN PRESS
P.V.R. Carvalho et al. / International Journal of Industrial Ergonomics 35 (2005) 619–644
way since microincidents, as such, are not
reported) and reflected about them (are there
correct or not?) we need a situated or naturalistic
approach and to produce a discussion inside the
organization.
Regarding risk management system, and more
generally the organization safety strategies our
study has shown a way to introduce a complex
systemic framework to understand some possible
ways for accident generation, finding coincidence
patterns of behavior in operator actions and
decisions. We have seen, for example, that all
SSs decide to accomplish their scheduled tasks
rather than follow test procedures; that operators
uses analogies, cooperation, hypothesis test, tacit
knowledge to solve their problems in unfamiliar
situations (microincidents), rather then standard
operational procedures. How such coincidence
patters of actions and decisions affects (and are
affected) the overall safety organizations strategies
can now be discussed based on empirical evidence
and a body of knowledge far greater than the
organization had before.
To the organization work environment, our
findings contrast with the normative behavior
prescribed by the organization work design and
their standards of competency for training and
evaluation of the operators work. What is evident
from the results of this study is that the traditional
nuclear industry’s standards of competence used
for operators performance assessment and train-
ing, differs from what is required for effective task
performance, since they are based on normative
decision theory and traditional (cognitivist) human
factors findings. This approach is static and
ignores important characteristics of work activities
(e.g. dynamism, context dependency and so on).
Indeed, as shown in our study, work is often
accomplished through a dynamic redistribution of
tasks or roles, involving interactions between
individuals, in a cooperative, opportunistic and
situated (naturalistic) way.
In order to be successful an organization has to
solve certain problems, a process that can be
supported, enhanced, endangered or stymied by
the underlying assumptions of the organizational
culture. Denied their existence, without a systemic
reflection, as often occurs in high-risk organiza-
643
tion, will not contribute to enhance safety and
brings tension to the organization’s social tissue.
References
Amalberti, R., 1996. La Conduite de Systèmes à Risques. Le
travail Humaine. Presses Universitaires de France, Paris
puf.
Bainbridge, L., 1983. Ironies of automation. Automatica 19,
775–779. Reprinted in Rasmussen, J., Duncan, K., Leplat,
J. (Eds.), New Technology and Human Error, Wiley,
Chichester, pp. 276–283.
Bourrier, M., 1999. Le nucléaire á l’épreuve de l’organizsation.
Le travail humaine 108 puf.
Bressolle, M.C., Decortis, B.P., Salembier, P., 1996. Traitement
cognitif et organisationnel des micro-incidents dans le
domaine du contròle aérien: analyse des boucles de
régulation formelles et informelles. Octarès, Toulouse,
France.
Calderwood, R., Crandall, B.W., Klein, G.A., 1987. Expert and
Novice Fire Ground Command Decisions. Klein Associates
Inc., Dayton, OH.
Cohen, M.S., Thompson, B.B., 2001. Training teams to take
initiative: critical thinking in novel situations. In: Salas, E.
(Ed.), Advances in Cognitive Engineering and Human
Performance Research. JAI Press, Greenwich.
Engestron, I., 2000. Activity theory as a framework for
analysing and redesigning work. Ergonomics 43 (7),
960–974.
Ericsson, K.A., Simon, H.A., 1993. Protocol Analysis:
Verbal Reports as Data. Cambridge University Press,
Cambridge.
Gaba, D.M., Howard, S.K., Small, S.D., 1995. Situation
awareness in anesthesiology. Human Factors 37, 20–31.
Hutchins, S.G., 1997. Decision making errors demonstrated by
experienced naval officers in a littoral environment. In:
Zsambok, C.E., Klein, G.A. (Eds.), Naturalistic Decision
Making. Lawrence Erlbaum Associates, Mahwah, NJ.
INPO, 1997. Excellence in Human Performance. Institute of
Nuclear Power Operators, Atlanta, GA.
INSAG – 4, 1991. Safety Culture. Safety Series No.75.
International Nuclear Safety Advisory Group, IAEA,
Vienna.
INSAG – 15, 2002. Key Practical Issues in Strengthening Safety
Culture. Safety Series No.75. International Nuclear Safety
Advisory Group, IAEA, Vienna.
Kaempf, G.L., Wolf, S.P., Thorsden, M.L., Klein, G.A., 1992.
Decision making in the AEGIS combat information center.
Technical Report No 1, Naval Command, Control and
Ocean Surveillance Center.
Keyser (de), V., Nyssen, A., 1993. Les erreurs humaines in
Anesthésie, Le travail humaine 56, puf.
Klein, G.A., 1989. Recognition-primed decisions. In: Rouse, W.
(Ed.), Advances in Man–Machine Systems Research. JAI
Press Inc., Greenwich.
 ARTICLE IN PRESS
644 P.V.R. Carvalho et al. / International Journal of Industrial Ergonomics 35 (2005) 619–644
Klein, G.A., Calderwood, R., Clinton-Cirocco, A., 1986. Rapid
decision making on the fire ground. Proceedings of the
Human Factors Society (30th Annual Meeting).
Klein, G.A., Calderwood, R., MacGregor, D., 1989. Critical
decision method for eliciting knowledge. IEEE Transactions
on Systems, Man, and Cybernetics 19, 462–472.
Klein, G.A., Orasanu, J., Calderwood, R., Zsambok, C.E.,
1993. Decision making in action: models and methods.
Ablex Publishing Corp., Norwood, NJ.
Krippendorff, K., 1980. Content Analysis. An Introduction to
its Methodology. SAGE Publications Ltd., London, UK.
Lipshitz, R., Klein, G., Orasanu, J., Salas, E., 2001. Taking
stock of naturalistic decision making. Journal of Behavioral
Decision Making 14, 331–352.
Mumaw, R.J., Vicente, J.K., Roth, E., 1996. A Model of
Operator Cognition and Performance During Normal
Operations. AECB pub., Ottawa, Canada.
OPITO, 1997. Offshore Petroleum Industry Training Organisa-
tion’s Approved Standards for Offshore Installation Man-
agers (dep OIM). Montrose, OPITO.
View publication stats
Orasanu, J., Fischer, U., 1997. Finding decisions in natural
environments: towards a theory of situated decision making.
In: Zsambok, C.E., Klein, G.A. (Eds.), Naturalistic
Decision Making. Lawrence Erlbaum Associates, Hillsdale,
NJ.
Rasmussen, J., 1983. Skills, rules and knowledge: signals, signs
and symbols, and other distinctions in human performance
models. IEEE Transactions on Systems, Man and Cyber-
netics 13, 257–266.
Seamster, T.L., Redding, R.E., Kaempf, G.L., 1997. Applied
Cognitive Task Analysis in Aviation. Ashgate Publishing
Ltd., Avebury, UK.
Schein, E., 1999. Corporate Culture Survival Guide. Jossey-
Bass Inc., San Francisco.
Vicente, J.K., Mumaw, R.J., Roth, E., 1997. Cognitive
Functioning of Control Room Operators. AECB pub.,
Ottawa, Canada.
Zsambok, C.E., Klein, G.A., 1997. Naturalistic
Decision Making. Lawrence Erlbaum Associates, Mahwah,
NJ.