Journal of Experimental Psychology: Applied Copyright 1995 by the American Psychological Association, Inc.

1995, Vol. 1, No. 4,305-320 1076-898X/95/$3.00

Expertise in Aeronautical Weather-Related Decision Making:

A Cross-Sectional Analysis of General Aviation Pilots
Mark Wiggins David O'Hare
University of Newcastle University of Otago

Weather-related crashes continue to account for a significant proportion of

general aviation (GA) accidents. In the present study, the decision-making
performance and information acquisition strategies of inexperienced, interme-
diate, and experienced pilots were examined from a cognitive skill perspective.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

Data were obtained from information search patterns and verbal protocols
This document is copyrighted by the American Psychological Association or one of its allied publishers.

during a series of 6 computer-based simulated flight scenarios. The results

revealed both quantitative and qualitative differences between the strategies
of experienced and inexperienced pilots. The implications of these results are
discussed in terms of weather-related decision-making training and the validity
of a cognitive skill model of aeronautical decision making.

Weather-related crashes are one of the common-
est causes of general aviation fatalities, accounting
for 20.8% of fatal general aviation (GA) crashes in
the United States between 1982 and 1988. (Air-
craft Owners and Pilots Association Air Safety
Foundation, 1991). Weather-related decision-
making (WRDM) skills can be defined as those
skills necessary to recognize and avoid meteorologi-
cal phenomena that present a hazard to the flight.
Most GA pilots are qualified to fly only in Visual
Meteorological Conditions (VMC), which are de-
fined by rules relating to cloud base and visibility.
Should such a pilot fail to conform to these
requirements and enter Instrument Meteorologi-
cal Conditions (IMC), the result will almost inevi-
tably involve loss of control of the aircraft resulting
in a fatal crash (Bryan, Stonecipher, & Aron,
1955). Because of the variable nature of operations
in the aviation environment, weather-related deci-
Mark Wiggins, Department of Aviation and Technol-
ogy, University of Newcastle, Newcastle, Australia; David
O'Hare, Department of Psychology, University of Otago,
Dunedin, New Zealand.
The computer used in this experiment was provided
by a science research grant from the New Zealand
Lottery Grants Board.
Correspondence concerning this article should be
addressed to David O'Hare, Department of Psychology,
University of Otago, P.O. Box 56, Dunedin, New Zea-
land. Electronic mail may be sent via Internet to
ohare@rivendell.otago.ac.nz.
sion making is often considered a skill that cannot
be prescribed during training. Rather it is expected
to develop gradually through practical experience.
However, in developing this type of experience,
relatively inexperienced pilots may be exposed to
hazardous situations with which they are ill-
equipped to cope. The aviation literature is replete
with tales of survival against the odds, which have
often been portrayed in hindsight as pivotal experi-
ences that marked the authors' transition from raw
novice to seasoned professional (e.g., Gann, 1961).
Although there is a growing recognition of the
need for WRDM to become an integral aspect of
pilot training, the characteristics necessary for an
optimal training program remain unclear. Unlike
the training environments necessary for develop-
ing skills such as forced-landing procedures, it is
more difficult to simulate deteriorating weather
conditions and thereby provide an opportunity to
practice WRDM skills. In addition, there are no
simple rules that inexperienced pilots can be
taught, particularly in regards to judging the sepa-
ration distances and precise significance of various
meteorological phenomena. Therefore, what is
required is a detailed identification of the skills
necessary for effective and efficient WRDM, which
can be used as the basis for the development of
training systems. The aim of the present study was
to develop a task that could be used to capture the
structure of expert WRDM in the laboratory.
Performance on this task was to be used to

305
 306 WIGGINS AND O'HARE
examine the basic differences between expert and
novice weather-related decision making to deter-
mine if expertise in an open dynamic environment,
such as aviation, can be characterized in similar
terms to those used to account for expertise in
more highly structured domains, such as chess and
physics (Larkin, McDermott, Simon, & Simon,
1980), The long-term goal of this research is to
provide a theoretical foundation for the training
and development of expert WRDM. As noted
above, the general aviation accident record sug-
gests that there is considerable need for improve-
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
ment.
This document is copyrighted by the American Psychological Association or one of its allied publishers.
Characteristics of the Weather-Related
Decision-Making Task
According to Woods (1988), the cognitive de-
mands associated with a particular task can be
represented along four dimensions: the extent to
which the process is constrained by the environ-
ment; the complexity of the interrelationships
between variables; the degree of risk associated
with the decision; and the uncertainty of the
information available. Particular problem-solving
strategies become more or less effective, depend-
ing on the relative values for these dimensions. For
example, Hammond, Hamm, Grassia, and Pearson
(1987) suggested that analytical strategies are best
suited to tasks with a high degree of risk and
uncertainty in which the problem-solving process is
unconstrained by the environment and the interre-
lationships between variables are complex.
However, weather-related decision making can
be conceptualized as a process that is severely
constrained by the environment. Moreover, the
interrelationship between variables is particularly
complex, and there is a high degree of risk and
often extreme uncertainty associated with the
available information. Indeed, Wiggins and O'Hare
(1993) contended that the nature of WRDM is
such that analytical strategies are likely to be less
effective than intuitive strategies in eliciting appro-
priate and timely decisions.
Support for this assertion can be derived from a
series of analyses of aeronautical decision making
with the Micro-Computer Decision Simulator
(MIDIS; Barnett, Stokes, Wickens, Davis, Rosen-
blum, & Hyman, 1987; Stokes, Kemper, & Marsh,
1992). Initial studies were designed to investigate
the relationship between information-processing
capability and decision-making performance dur-
ing simulated cross-country flights. A comparison
between the performance of inexperienced and
experienced pilots indicated that although several
measures of information processing were predic-
tive of inexperienced pilots' performance, no rela-
tionships were found for the experienced group.
Because information-processing capacity is an in-
trinsic element of analytical decision making, Stokes
et al. (1992) concluded that experienced pilots'
aeronautical decision making is mediated less by
analytical strategies and more by intuitive strate-
gies such as the direct retrieval of relevant sche-
mata from long-term memory. A similar conclu-
sion is proposed by Klein (1989) and Klein and
Klinger (1991) concerning the diagnosis of the
sources of fires by fireground commanders.
According to Ericsson and Smith (1991), "[R]e-
search on expertise may be one of the most rapidly
expanding areas within cognitive psychology and
cognitive science" (p. 1). The tasks that have been
extensively investigated (e.g., chess playing; phys-
ics problem solving) lack many of the key features
(e.g., ill-structured problems, time stress, and mul-
tiple goals) that characterize real-world decision
making (Orasanu & Connolly, 1993). The most
notable attempt to capture and model the essence
of expertise in dynamic environments, involving
ill-structured problems, time-pressure, and signifi-
cant risks has been made by Klein (1989).
According to Klein (1989), problem-solving and
decision-making expertise in dynamic tasks is char-
acterized by the serial evaluation of options avail-
able in long-term memory. The option selected is
dependent on an unconscious and rapid compari-
son between the characteristics associated with the
present situation and those embodied within pre-
existing memory structures. Empirical evidence for
the serial evaluation of options among experts has
been derived primarily from research involving
expert firefighters during their diagnosis of the
probable sources of fires (Klein, 1989). To trace
the decision-making process, Klein asked partici-
pants to "think aloud" as they diagnosed the
situation and selected the strategy that they consid-
ered most effective under the circumstances. Ex-
pert firefighters were found to evaluate options
sequentially, discarding those that did not meet
the needs of the current situation and retaining
and modifying as necessary, those strategies that
matched the existing situation. Moreover, these
 EXPERTISE IN AERONAUTICAL DECISION MAKING
strategies were successful in ameleriorating the
situations because they were based on a large
repertoire of prior experiences coupled with a
well-developed mental model of the system within
which they operated.
In contrast, novice problem-solving perfor-
mance was characterized by the evaluation of
options concurrently and the subsequent integra-
tion of this information with generalized problem-
solving strategies such as means-ends analysis and
backward chaining (Klein, 1989). The result is a
cumbersome and time-consuming strategy that is
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
often unsuited to the demands associated with
This document is copyrighted by the American Psychological Association or one of its allied publishers.
time-limited and uncertain environments. Ironi-
cally, this kind of analytical procedure has often
been recommended as a means of improving the
performance of novices (e.g., Benner, 1975).
Whereas the decision-making strategies of ex-
perts show the features of automatic processing as
identified by Shiffrin and Schneider (1977), the
decision making of novices more closely resembles
that of controlled processing because the informa-
tion pertaining to each situation must be mediated
within working memory. As working memory is
both capacity limited and subject to transience,
information acquired by novice performers is often
lost and/or overwritten such that task-related
information must be reacquired on a regular basis.
Hershey, Walsh, Read, and Chulef (1990) referred
to this reacquisition of task-related information as
information recursion. Kirschenbaum (1992) hypoth-
esized that the tendency to "re-look at" informa-
tion would differentiate expert and novice subma-
rine officers.
The most detailed model of cognitive skill acqui-
sition has been developed by Anderson (1982,
1987,1993). In the ACT* model (and its successor,
ACT-R), all cognitive behavior is controlled by
production rules. In the early stages of skill acqui-
sition, knowledge is encoded declaratively. Prac-
tice at a task results in the compilation of task-
specific productions. Therefore, the experienced
performer is not limited by the characteristics of
working memory, as productions are run autono-
mously. The features of automatic processing dis-
cussed above can also be derived from the ACT*
model (Anderson, 1992). The ACT* model strongly
suggests that as production systems become more
highly attuned with further task-related experi-
ence, experts are likely to search a problem space
more selectively and more rapidly. It is fundamen-
307
tal to the ACT*/ACT-R models that the system
learns by doing and that task-related experience
should be a better predictor of expert strategy than
any global measure of experience. Ericsson and
Charness (1994) highlighted the importance of
"deliberate practice" sustained over periods of
years in expert performance.
Process-tracing techniques are one method
through which the sequential nature of informa-
tion processing can be identified and the nature of
decision making and problem solving examined in
detail. In previous research, this has often involved
an analysis of eye movements or verbal protocol
data (Guindon, 1990; Lusk, 1993; Terranova,
Snyder, Seamster, & Treitler, 1989). The former
facilitates the identification of information-search
patterns, whereas the latter is designed to identify
the thought processes that are engaged during the
task. However, the use of verbal protocol data
exclusively has been criticized widely as being both
obtrusive and subjective (Nisbett & Wilson, 1977).
Bainbridge (1979) provides a thorough discussion
of the value and limitations of verbal protocol data
in the analysis of operator behavior in complex
systems. Sanderson, Verhage, and Fuld (1989)
showed how a combination of objective measures
and verbal protocol analysis can yield valuable
insights into complex process control behavior.
The present study was designed to use a combi-
nation information search and verbal protocol
methodology (see Sanderson et al., 1989) to inves-
tigate the differences in performance on a simu-
lated WRDM task between pilots of different
levels of experience at such tasks. The following
hypotheses were derived from previous research
on expertise in other domains, and the theoretical
implications of the recognition-primed (Klein,
1989), ACT*/ACT-R (Anderson, 1982,1993), and
automatic-controlled (Shiffrin & Schneider, 1977)
models of decision-making and problem-solving
skills.
Hypothesis 1: If expertise lies in the develop-
ment of task-specific, procedural knowledge, then:
(a) experts will be less able to verbalize this
procedural knowledge than novices who must
interpret declarative knowledge directly; (b) differ-
ences in decision quality will be less apparent when
expertise is distinguished on the basis of total flight
hours than on the basis of actual cross-country
flying experience; (c) general problem-solving abil-
ity will be unrelated to differences in performance
 308 WIGGINS AND O'HARE
within the expert group, but will be moderately
related to performance within the novice group.
Hypothesis 2: Experts' information search pat-
terns will be more structured and directed. Conse-
quently, experts will (a) acquire a smaller quantity
of information than novices; (b) spend less time
examining information; (c) be more concerned
with the implications to be drawn from informa-
tion gathered; (d) be less likely to articulate their
goals; (e) be less likely to return to information
already accessed (information recursion); (f) not
access information on the basis of the "surface"
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
representation of the problem provided by the
This document is copyrighted by the American Psychological Association or one of its allied publishers.
experimenter-defined information categories; (g)
show more confidence in their decisions.
Overview
Participants were asked to assume that they
were pilots in command of a VFR cross-country
flight and had reached some point en route where
a decision was required as to whether they should
continue toward the destination, divert to an
alternate aerodrome, or return to the point of
departure. Participants were advised that each
scenario would comprise an introductory phase, an
information search phase, a decision phase, and a
confidence phase. A practice scenario was pro-
vided to familiarize them with the structure and
format of the software. It was stressed that they
should, in each case, examine only that informa-
tion that they considered necessary in formulating
their decisions because the time period in which to
conduct the information search would be limited.
Participants were tested individually with six
aeronautical decision scenarios, of which three
were situated in the North Island region and three
in the South Island region of New Zealand. These
were presented alternately and counterbalanced
within each group such that half received a North
Island scenario initially and the other half received
a South Island scenario. Scenarios relating to both
regions were subsequently presented in a quasi-
random sequence within this counterbalanced
structure. This procedure was intended to control
for any biasing of information search patterns that
may have occurred because of the participants'
relative familiarity with one region and one area
over the other.
Prior to each experimental scenario, partici-
pants were provided with a brief introduction that
outlined the details of the proposed flight includ-
ing the departure point, the intended destination,
and the magnetic bearing and distance between
the two airports. In addition, the geographical
location of each decision point was described in
terms of its distance from the point of departure
along a magnetic bearing running directly from the
departure point to the destination.
Method
Participants
Forty participants were recruited, of whom 25
were residents in the Auckland region (North
Island), with the remainder residents in either the
Otago or Canterbury regions (South Island) of
New Zealand. They ranged in age from 18 to 67
years and had completed at least 2 hr as the
pilot-in-command of cross-country flights. These
were defined as instrument flight rules (IFR) or
visual flight rules (VFR) operations other than
those in training areas, for which the distance to
the nearest airport was at or beyond 20 nmi.1
Overall, participants reported a median 120 hr
(range 4-8,000) as the pilot-in-command of cross-
country flights. In comparison, the median for the
total number of hours in command was 360 (range
12-20,800). The number of cross-country hours
accumulated over the previous 12 months ranged
from 1 to 650 (mdn — 40). The total number of
hours accumulated during the previous 12 months
ranged from 2.6 to 800, with a median value of 130
hr experience.
Participants were grouped into three categories
defined as: (1) pilots with 2 to 100 hr of cross-
country flying experience; (2) pilots with 101 to
1,000 hr experience; and (3) pilots who had accu-
mulated in excess of 1,000 hr of cross-country
experience. Eighteen were classified into the
2-100-hr category, with 14 in the 101-1,000-hr
category, and 8 in the 1,000-hr category. Pilots in
the first category had accumulated a mean 44.35 hr
(SD = 29.26 hr) as the pilot-in-command of
cross-country flights, and those in the second and
third categories had accumulated a mean 229
(SD = 124.5) and 3,640 (SD = 2,601) hr, respec-
tively.
In aviation, distances are expressed in nautical miles.
 EXPERTISE IN AERONAUTICAL DECISION MAKING
Apparatus
The experiment was conducted with a NEC
ProSpeed 286 portable computer equipped with a
VGA card linked to a remote PCDIRECT visual
graphics adaptor (VGA) monitor. A computer
programmed in Turbo Basic by technicians at the
University of Otago, collected personal details and
information pertaining to the participant's flying
experience. The program also incorporated a test
of general problem-solving ability, and one prac-
tice and six experimental aeronautical-decision
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
scenarios presented sequentially.
This document is copyrighted by the American Psychological Association or one of its allied publishers.
Problem-Solving Ability
This was assessed with a syllogistic reasoning
task similar to the test of logical reasoning incorpo-
rated in the SPARTANS (Simple Portable Avia-
tion Relevant Task Battery and Answer Scoring
System) test battery (Baddeley, 1968; Stokes &
Raby, 1989). Participants were presented with
eight syllogisms and required to respond true or
false to the validity of the conclusion presented.
Both response latency and accuracy were re-
corded.
Aeronautical Scenarios
Each decision scenario included aeronautical
information normally available to pilots during
VFR cross-country flights within New Zealand.
This information was presented on the computer
screen in a menu-based format with a single main
menu and five submenus (see Figure 1). These
were designated as (1) the current state of the
aircraft; (2) the performance characteristics of the
aircraft; (3) the meteorological conditions; (4)
aerodrome specifications; and (5) the topographi-
cal map. Submenu 1 incorporated a number of
additional menus including cockpit views and the
distances and bearings to selected alternate air-
ports. Similarly, meteorological reports and termi-
nal area forecasts were included as additional
menus within Submenu 3.
Submenu 1 included information regarding the
amount of fuel and daylight remaining in which to
complete the flight. The values were designed such
that the endurance in each scenario was less than
90 min at the decision point. In addition, Submenu
1 included four verbal descriptions of weather
309
conditions corresponding to the forward, star-
board, port, and rear views as observed from the
cockpit. This information referred specifically to
the type of cloud visible, the approximate altitude
of the cloud base-ceiling above ground level, and
any noticeable change in these conditions, such as
an apparent increase in density or lowering of the
cloud base. Relative to the decision point, the
distances and magnetic bearings to five alternate
airports were available, with the distance repre-
sented in nautical miles and the bearings corrected
for magnetic variation. Information relating to the
outside air temperature, ground-speed, and alti-
tude of the aircraft at the decision point was also
provided in Submenu 1.
A series of aircraft performance characteristics
were available in Submenu 2, including a landing
performance table, cruise performance table, and
crosswind landing specifications. For each sce-
nario, information relating to one of three types of
single-engined aircraft was provided alternately in
order to minimize the possibility that the perfor-
mance characteristics accessed in one scenario
would be recalled and transferred to subsequent
scenarios.
Submenu 3 contained the meteorological infor-
mation that would normally be available to pilots
conducting cross-country flights within New Zea-
land. The weather patterns and characteristics
were devised in conjunction with air traffic control
authorities situated within the region relevant to
each scenario. As air traffic controllers are respon-
sible for issuing hourly metereorological reports,
they were expected to possess a thorough under-
standing of the local weather conditions and the
difficulties experienced by pilots. They were thus
considered qualified to ensure that the meteorologi-
cal conditions included within scenarios were not
uncommon for the various regions (Ryan, 1984).
Submenu 4 contained aerodrome information
relating to the destination airport and each of the
five alternate airports. These were computer-
generated reproductions of the information issued
by the New Zealand Civil Aviation Authority,
which included an aerial photograph of the air-
port, the type of runway surface, its serviceability,
and its dimensions.
A series of 10 computer-generated images of the
World Aeronautical Chart (WAC) appropriate to
each flight were available in Submenu 5 and
displayed in a "16 grays," Graphics Interchange
 310 WIGGINS AND O'HARE

15:20:13 I I ^^T I I 1 I
i I I i i i i

1 -Current State of the Aircraft

2 -Aircraft Performance Specifications
3 -Meteorological Information
4 -Aerodrome Diagrams
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

5 -Topographic Map
This document is copyrighted by the American Psychological Association or one of its allied publishers.

:
- • • • •"•••: ' •' ' "• : • I : • " '. . II ' ' • • • • - . ' ' '•'• i
1 ESC for Main Menu II II Please Enter Option I I SPACE for PreviousMenu
f' " •• "

15:20:26

1 —Fuel Remaining 6 -Ground Speed

2 -Daylight Remaining 7 -Outside Air Temperature
3 —Distance to Airports 8 -Cockpit Views
4 -Bearings to Airports
5 -Present Altitude

ESC for Main Menu Please Enter Option SPACE for Previous Menu

Figure 1. (top) The main menu and (bottom) the submenu for current state of the
aircraft. Time elapsed is indicated by the horizontal bar at the top of the screen. ESC =
escape.
 EXPERTISE IN AERONAUTICAL DECISION MAKING
Format (GIF). In each case, the first image por-
trayed a 120 run x 45 nm quadrant. It depicted the
decision point and the proposed track between the
departure and destination aerodromes. This was
displayed at 50% of the original size and provided
a broader perspective for those participants unfa-
miliar with the region. The remaining images
portrayed 32 x 20 nm quadrants of and about the
decision point. They were to 192% of the original
size and oriented toward true north.
Access to each of the 50 screens of information
(see Table 1) could be accomplished by depressing
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
This document is copyrighted by the American Psychological Association or one of its allied publishers.
the appropriate numeric key on the computer
keyboard. Pressing the space bar returned the user
to the screen accessed previously; the escape key
returned the user to the main menu. For those
screens where information was displayed as text,
prompts were provided at the base of each informa-
tion screen. Graphics screens did not include
prompts. Therefore, a card detailing the functions
of the computer keys was positioned permanently
to the side of the computer. A time line, displayed
in the top right-hand corner of both the informa-
tion and menu screens, indicated the proportion of
time remaining in which to complete the informa-
tion search.
For each of the six decision scenarios, the
computer recorded both the information screens
accessed by participants, and the duration in
seconds between the selection and subsequent exit
from a particular screen. Following completion of
each scenario, participants were presented with a
forced choice that involved either continuing to-
ward the proposed destination or returning to the
point of departure.
Table 1
Information Categories and the Number of
Information Screens Contained In Each Scenario
Information Information screen
category Number Percentage
Current state 5 10
Distance to airports 5 10
Bearings to airports 5 10
Cockpit views 4 8
Performance 3 6
Meteorological information 12 24
Aerodrome diagrams 6 12
Topographic map
10 20
311
Verbal Protocols
Following the practice decision-making sce-
nario, participants were given a description of
"Duncker's candle problem" as described by
Anderson (1980). The problem consists of finding
a way to attach a candle to a door. The following
objects are available on a table: the candle, some
matches, and a box of tacks. Participants were
instructed to think aloud as they worked on the
problem: to say everything they were thinking from
the start of the problem until they came up with an
answer (Ericsson & Simon, 1984). If they stopped
verbalizing for more than 3 s, the experimenter
prompted them to continue. On completion, par-
ticipants were told that they would be required to
think aloud during each of the weather-related
decision-making scenarios.
Results
Information Acquisition
To differentiate between the types of informa-
tion accessed, we divided the 50 computer screens
of aeronautical information available in each sce-
nario into eight distinct categories. These were
based on the submenus in the computer program
and the subject of the information screens con-
tained therein. Table 1 provides a list of these
categories, in addition to the number of informa-
tion screens, and the percentage of the total
number of screens that they represented. Encod-
ing the information screens according to these
categories allowed comparisons to be made be-
tween both the type of information accessed by
participants and the extent to which the sequence
with which their accessed information followed
these experimenter-defined categories.
For each aeronautical decision scenario, informa-
tion screens were coded in sequence with the
Hemi-Semi-Automated Protocol Analyzer
(SHAPA) computer software package for protocol
analysis (Sanderson, James, & Seidler, 1989). This
program assists in establishing patterns and se-
quences in both verbal and nonverbal protocol
data with the predicate-argument notation for
encoding. According to Sanderson et al. (1989),
predicates are general terms that represent the
particular activity of interest, whereas arguments
are specific to a situation and are used to qualify
predicates. SHAPA provides for a maximum of 20
predicates, each containing at least one argument.
 312 WIGGINS AND O'HARE
In addition to the frequency with which predi-
cates and arguments occur, the SHAPA computer
program also produces transition matrices, detail-
ing the sequence in which predicates occur. First,
second, and third order transition matrices are
provided, corresponding to the frequency of predi-
cate* given predicate A (first order), the frequency
of x given A then B (second order), and the
frequency of A: given A then B then C (third order;
Sanderson et a!., 1989). These facilitate the identi-
fication and quantification of interrelationships
that may exist between successive predicates gener-
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
ated during the performance of a task.
This document is copyrighted by the American Psychological Association or one of its allied publishers.
A 3 x 2 x 3 mixed factorial analysis of variance
(ANOVA) with repeated measures on the second
and third variables was conducted for each of the
measured variables (number of information screens
accessed; time spent examining the information
screens; number of information recursions). The
aim was to establish whether differences existed
between cross-country flying experience (three
levels), the relative familiarity of the scenario
characteristics (two levels), and responses across
trials (three levels).
Analysis of the results arising for the number of
information screens accessed during scenarios re-
vealed a significant main effect for cross-country
flying experience, F(2, 37) = 3.51, p < .05. Post
hoc tests with the Tukey (B) statistic indicated that
the difference lay between the inexperienced and
experienced groups, with novices accessing the
greatest number of information screens, and ex-
perts the least.
The ANOVA of the time spent examining the
information screens revealed a significant main
effect for cross-country flying experience, F(2,37) =
4.55, p < .05. Post hoc tests indicated that the
inexperienced group took significantly longer than
the experienced group to examine the information
screens.
The ANOVA for the number of information
recursions revealed a significant main effect for
cross-country flying experience, F(2, 37) = 7.88,
p < .01, with no main effects or interactions
involving familiarity or trials. Post hoc tests dis-
closed significant differences between the experi-
enced group and both the intermediate and inexpe-
rienced groups. The differences indicate that overall
the number of information recursions made during
scenarios decreased as a function of cross-country
flying experience, with novices making the most
recursions and experts the fewest.
Search Patterns
As outlined previously, the sequence in which
participants accessed the information screens was
recorded for each scenario. These screens were
categorized according to the information category
from which they originated, and the sequences
were examined with transition matrices. Transi-
tion matrices are used primarily for the identifica-
tion and tabulation of recurring patterns of behav-
ior during protocol analysis (Sanderson et al.,
1989). In the present study, the behavior of particu-
lar interest related to the frequency with which
successive information screens were accessed from
the same information category (predicate). This
provided an indication of the extent to which
information was being accessed according to the
experimenter-defined categories shown in Table 1
or in some other sequence. Tabulations of pairs of
predicates were referred to as first-order transition
matrices, and sequences involving triples or qua-
druples were labeled second- and third-order tran-
sition matrices respectively. Transition matrices
enabled the tabulation of frequencies and thus the
calculation of the proportion of the total number
of sequences that originated from the same infor-
mation category.
A 3 x 2 x 3 repeated measures multivariate
analysis of variance (MANOVA) was conducted to
examine the relationship between cross-country
flying experience (three levels), relative familiarity
with the characteristics associated with scenarios
(two levels), trials (three levels), and the three-
element vector of measured variables comprising
the proportions derived from the first-order, sec-
ond-order, and third-order transition matrices.
The results disclosed a statistically significant main
effect for cross-countryflyingexperience, F(2,37) =
3.46, p < .01, although there were no main effects
or interactions relating to familiarity or trials. In
order to identify the variable or variables within
which these differences lie, we conducted univari-
ate tests between the cross-country groups for each
of the three measured variables. Statistically signifi-
cant differences were evident between groups for
the proportions derived from each of the first-
order, F(2, 37) = 6.23,p < .01, second-order, F(2,
 EXPERTISE IN AERONAUTICAL DECISION MAKING
37) = 8.72, p < .01, and third-order, F(2, 37) =
11.82, p < .01, transition matrices.
Because all three measured variables showed
significant main effects for cross-country flying
experience, it was considered appropriate to deter-
mine whether any similarities existed between the
responses. Post hoc tests (Tukey B) for each of the
first-, second-, and third-order transition matrices
revealed a significant difference between the expe-
rienced group and the intermediate and inexperi-
enced groups. These results indicate that of the
total number of information screens accessed by
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
each pilot, the proportion of successive screens
This document is copyrighted by the American Psychological Association or one of its allied publishers.
from the same information category was lower for
those in the experienced group than for those
pilots in either the intermediate or inexperienced
groups. As Figure 2 reveals, the proportions de-
creased as a function of cross-country category for
all three transition matrices.
Decision Making
Following the information acquisition phase of
each scenario, participants were asked to select
from pairs of alternative strategies, those which
they would most prefer to undertake given the
particular circumstances. One of these pairs in-
volved a forced choice between either returning to
the departure point or continuing en route to the
destination. Overall, participants chose to return
to their point of departure in 43% of cases,
whereas 57% of selections involved continuing the
flight en route to the destination. Chi-square tests
did not reveal any statistically significant differ-
0.2
c 0.15
o
L.
a
CL
a 0.1
03
0)
0.05
Novices Intermediate
Cross-country group
Figure 2. Mean proportion of the sequences of information screens accessed from the
same category for first-order, second-order, and third-order transition matrices (TM).
313
ences between scenarios or between the sequence
with which the scenarios were presented. How-
ever, statistical analysis revealed differences be-
tween levels of cross-country flying experience.
Pilots in the intermediate and experienced groups
were more likely to choose to continue toward
their destinations than pilots categorized as inexpe-
rienced who were more likely to elect to return to
their take-off points, x2(2,N = 40) = 9.02, p < .02.
Response Latency
In addition to recording the particular informa-
tion screen selected, the computer program also
recorded the response latency for each selection.
Across participants, the mean response latency for
the forced choice between returning to the point of
take-off or continuing en route to the destination
was 18.9 s (SD = 17.97). The raw data were
transformed logarithmically prior to statistical
analysis to reduce any disparity between the covari-
ance matrices of groups. These data were sub-
jected to a 3 x 2 x 3 mixed factorial ANOVA to
determine whether differences existed between
cross-country experience (three levels), familiarity
(two levels), and trials (three levels), for the
response latency recorded for the forced choice
between continuing toward the destination or
returning to the take-off point.
The results disclosed a significant main effect for
cross-country flying experience, F(2, 37) = 3.29,
p < .01. An inspection of the data for the three
cross-country groups indicated that the mean re-
sponse latency for the intermediate group ex-
• First-Order TM
nSecond-Order TM
OThird-Order TM
Experts
 314 WIGGINS AND O'HARE
ceeded that for the inexperienced group, which in
turn exceeded that recorded for the experienced
group.
Confidence
The final aspect of the decision selection phase
involved participants choosing from a 7-point Likert
scale, the number that best represented their level
of confidence in the efficacy of their selections.
Overall, participants expressed a confidence level
of 2.8 (SD = 1.4) where a score of 1 indicated
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
absolute confidence in the decision. A 3 x 2 x 3
This document is copyrighted by the American Psychological Association or one of its allied publishers.
mixed factorial ANOVA was used to determine
whether differences in confidence existed between
cross-country flying experience (three levels), famil-
iarity (two levels), and trials (three levels). The
results of this analysis revealed a statistically signifi-
cant main effect for cross-country flying experi-
ence, F(2,37) = 5A5,p < .01, although there were
no main effects or interactions involving familiarity
or trials.
Post hoc tests with Tukey's B statistic indicated
that the inexperienced group was significantly less
confident than either the intermediate or the
experienced group. Inspection of the respective
category means showed that the level of decision-
making confidence was greatest for the experi-
enced group. This is consistent with Shanteau's
(1988) characterization of expert performance.
Problem-Solving Ability and
Information Acquisition
Overall, accuracy of responses in the syllogistic
reasoning task was high (mean correct = 7.05;
SD = 1.26) with a mean response latency of 25.12 s
(SD = 65). There was a nonsignificant correlation
between accuracy and latency, r(40) = - .19, p >
.05. Both accuracy and latency were entered into
multiple regression equations with the mean num-
ber of information recursions (log transformed) as
the criterion variable. As noted in the introduc-
tion, information recursions measure the amount
of reacquisition of task related information that
takes place. Separate equations were calculated
for each experience level (see Table 2).
Inspection of the means and standard deviations
for each of the three groups of pilots indicated that
valid comparisons could be made regarding their
performance during the syllogistic reasoning task.
Table 2
Multiple Regression Models for Each Cross-Country
Group Between the Accuracy and Response Latency
Associated With the Nonsense Syllogisms Task and
the Number of Information
Recursions During Scenarios
Problem solving (p)
Multiple
Group R Accuracy SD Latency SD
Inexperienced .584* -.486* 1.35 .475* 7.59
Intermediate .215 .234 0.63 .149 7.86
.447 .402 1.19 -.157 9.75
Experienced
Note. One pilot from the experienced group was excluded
from this analysis as an outlier.
*p < .05.
Indeed, the results indicated that this type of
problem-solving ability was unrelated to the perfor-
mance of intermediate or expert pilots in the
decision scenarios. However, it was predictive of
performance by the inexperienced pilots and thus
provides support for Hypothesis Ic.
Analysis Determined on the Basis of Total
Flight Hours
All the previous analyses were repeated with
pilots classified into three categories according to
total flying experience rather than cross-country
experience. The same criteria (hours of experi-
ence) were used as that applied for the cross-
country categories. No main effects or interactions
were found for the number of information screens
accessed or for time spent examining information
screens. A significant main effect of experience
group on number of information recursions was
found, F(2, 37) = 3.49, p < .05. The same analysis
on the basis of cross-country categorization was
significant (p < .01). Overall, these results sup-
port Hypothesis Ib that differences in perfor-
mance will be more evident as a function of
task-specific experience than of flying experience
in general.
Verbal Protocol Analysis
Because of the very large quantity of verbal
protocol data, 3 participants were randomly se-
lected from each group for detailed analysis. The
mean total flight hours of the three groups were
experts = 7,483; intermediate = 200; and novices =
 EXPERTISE IN AERONAUTICAL DECISION MAKING
55. The audiotapes were transcribed and entered
into the SHAPA system (Sanderson et al., 1989).
The verbalizations were divided into segments
containing one predicate. The predicates were
derived from Rasmussen's (1982) analysis of hu-
man error that had previously been applied to the
coding of aircraft accident reports (O'Hare, Wig-
gins, Batt, & Morrison, 1994). The six predicates
were detection (identifying information); diagno-
sis (discerning the meaning of the information);
goal (selecting a goal); strategy (identifying a
means to achieve the goal); procedure (specifying
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
tasks needed to carry out the strategy); and action
This document is copyrighted by the American Psychological Association or one of its allied publishers.
(implementation). Two independent raters coded
samples of two transcripts. The cross-coder reliabil-
ity was assessed by calculating values of Cohen's
kappa. These were 0.73 and 0.79 for the two
samples. These indicate excellent levels of agree-
ment (Fleiss, 1981).
A simple frequency analysis of the six predicates
showed that although the first three (detection,
diagnosis, and goal) were used frequently, in-
stances of strategy, procedure, and action were
rarely recorded. Therefore, for the present analy-
sis, these last three categories were collapsed into
one group referred to as tactics. The novice group
produced more verbalizations (n = 1,155) than
either the intermediate (« = 700) or expert groups
(n = 732), although these differences were not
statistically significant.
The distribution of the predicates across the
three groups (expert, intermediate, and novice) is
shown in Figure 3. The numbers represent the
50
* • Experts
tn
£ 40
a;
to 30
20
C
o
•H
10
o
Q.
O
Detection Diagnosis
Category
Figure 3. Proportions of total predicate encodings in each category obtained from the
concurrent verbal protocol data.
315
percentage of each group's total verbalizations
that were encoded into each category.
It can be seen that the expert pilots made fewer
explicit goal or detection references than other
pilots, but rather more diagnostic statements. Nov-
ice pilots were mostly concerned with detecting
appropriate information and with explicitly articu-
lating their goals. The very small number of partici-
pants in each group means that conventional
statistical tests lack sufficient power to detect true
differences. Despite this, a test of the absolute
number of goal statements produced by the three
groups approached statistical significance, F(2,6) =
4.13, p < .07, as did the absolute number of
detection statements, F(2, 6) = 3.38,p < .10. The
difference in absolute number of diagnosis state-
ments between the three groups was nonsignifi-
cant, F(2, 6) = 0.33, p > .7. However, the ratio of
diagnostic to detection statements was less than
one for all participants except for 2 of the 3 expert
pilots. Again, this difference approached signifi-
cance,^, 6) = 3.23,p < .11.
As noted previously, SHAPA provides a number
of techniques for exploring the sequential depen-
dencies in data in greater depth. The first-order
transition matrices show that the most common
transition for experts was from one diagnostic
statement to another (148 examples—more than
60% greater than the next most common transi-
tion), whereas this pattern was less frequent for
novices (64 examples). The most common pattern
for novices was from a goal statement to a detec-
tion statement (224 examples), which was much
^Intermediates
DNovices j
Goal Tactics
 316 WIGGINS AND O'HARE
less frequent for experts (86 examples). This differ-
ence approached statistical significance, F(2, 6) =
3.78,p < .08. These differences are consistent with
the working backward and working forward strate-
gies adopted by novices and experts respectively in
solving physics problems (Larkin et al., 1980).
The second-order transition matrices showed
that experts were most likely to transition to a third
diagnosis statement, whereas novices were most
likely to transition to a detection statement. Inspec-
tion of the frequency of predicate cycles supplied
by SHAPA confirms these observations. Further
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
detail of these analyses of the verbal protocols
This document is copyrighted by the American Psychological Association or one of its allied publishers.
obtained in the present study can be found in
O'Hare, Wiggins, and Jones (1995).
Discussion
Strategies of Information Acquisition
During the acquisition of aeronautical informa-
tion, experienced pilots accessed significantly fewer
information screens, made fewer information recur-
sions, and spent relatively less time examining the
information screens than inexperienced pilots.
These results are consistent with Hypotheses 2a, e,
and b, respectively, and suggest that experienced
pilots required relatively less task-related informa-
tion with which to formulate a decision that they
considered appropriate under the circumstances.
This is consistent with research in other domains,
such as financial analysis (Bouman, 1980) and
auditing (Bedard, 1989), and suggests that experi-
enced decision makers use information search
patterns that are more structured and efficient
than those of their inexperienced counterparts.
Further support for this assertion can be derived
from the results arising from the analysis of transi-
tion matrices. Of the total number of sequences of
information screens accessed by pilots, a signifi-
cantly greater proportion was accessed from the
same submenu by inexperienced pilots than by
experienced pilots. Therefore, those pilots who
were experienced in formulating weather-related
decisions were less likely to access information
according to the experimenter-defined categories,
supporting Hypothesis 2f. This is consistent with
research in other domains that suggests that ex-
perts operate on a different representation of the
problem than novices, who are more likely to
operate with the surface representation of a prob-
lem. Collectively, the results provide considerable
empirical support for the theoretical distinctions
proposed between the information acquisition strat-
egies of experienced and inexperienced pilots.
These results occurred independently of either
the particular characteristics of the scenarios (trial)
or the extent to which pilots were familiar with the
particular area. Thus it can be concluded that the
differences evident between experienced and inex-
perienced pilots, both qualitatively and quantita-
tively, were relatively consistent and not the prod-
uct of idiosyncracies associated with any particular
scenario.
Because we were unable to differentiate signifi-
cantly between the performance of intermediate
and inexperienced pilots in the experimental sce-
narios, we believe this suggests that either the
transition between stages was continuous instead
of stepwise or that the classification procedure
used did not distinguish effectively between pilots
at the two stages. This is a matter for further
research.
Verbal Protocol Analysis
Combined with differences in information acqui-
sition strategies, the distinction between automatic
and controlled processing also predicts differences
between experienced and inexperienced pilots in
terms of the extent to which information is verbal-
ized (Shanteau, 1988). Thus, in Hypothesis la, we
predicted that experts would be less able to verbal-
ize procedural knowledge than the novices who
interpret declarative information directly. Al-
though the novice pilots produced more verbaliza-
tions than both the intermediate and expert groups,
these differences were not statistically significant.
However, the hypotheses that experts would be
less likely to articulate their goals (Hypothesis 2d)
and would be more concerned with implications or
diagnosis (Hypothesis 2c) were confirmed.
Although they are based on a small sample,
these findings are strikingly consistent with those
of Larkin et al. (1980) who noted that when
attempting to solve elementary physics problems
novices "seem to require goals and sub-goals to
direct their search" (p. 1338). Larkin et al. hypoth-
esized that the management of these goals takes
up a substantial proportion of working memory
capacity, thus accounting in part for the relatively
 EXPERTISE IN AERONAUTICAL DECISION MAKING
slow and effortful problem-solving performance of
novices.
Aeronautical Decisions
Because of the nature of the terrain, coupled
with the daylight and fuel limitations, the decision
to return to the point of departure would, in most
of scenarios used, be ill-advised. A chi-square
analysis of the decisions indicated that pilots in the
intermediate and experienced groups were signifi-
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
cantly more likely to continue toward their destina-
This document is copyrighted by the American Psychological Association or one of its allied publishers.
tions than to divert. In contrast, pilots in the
inexperienced group chose with greater frequency
to return to the point of take-off. At the ab initio
stage of training, pilots are generally taught to
conduct a 180° turn if the conditions en route
prevent a landing at the destination proposed. This
recommendation is based on the assumption that
the meterological conditions are likely to be favor-
able in the direction from which the aircraft has
traveled recently. Thus in an experimental situa-
tion, it is possible that inexperienced pilots would
rapidly adopt the normative option of returning to
the point of departure (a 180° turn), rather than
actively formulating a decision on the basis of the
information available.
The analysis of the decision response latency
data disclosed a significant main effect between
cross-country groups. Further inspection of the
data revealed that the response latency for the
decision phase was greater among inexperienced
pilots than among expert pilots. This suggests that
inexperienced pilots did not quickly select the
more obvious option (to discontinue rather than
press on) but may have had greater difficulty
formulating an appropriate decision strategy dur-
ing the limited time available. Experienced pilots
required the least time to develop a strategy that
they considered effective.
The final aspect of the decision phase involved
the participants rating on a 7-point Likert scale
their confidence in the appropriateness of their
selections. The results revealed a significant main
effect for cross-country flying experience, with the
inexperienced pilots showing the least confidence
in their decisions and the experts the most. This is
broadly consistent with the differences predicted
in Hypothesis 2g, although the differences between
expert and intermediate pilots was not significant.
317
Problem-Solving Skills and Pilot Performance
As an individual develops experience in a particu-
lar domain, Klein (1989) and Klein and Klinger
(1991) suggested that task-specific experiences are
acquired. These experiences through further prac-
tice can be applied successfully to a variety of
problems and circumstances. This makes redun-
dant any reliance on general procedures for prob-
lem resolution. Consequently, the performance of
individuals at later stages of skill acquisition would
be expected to occur independent of any profi-
ciency associated with the application of problem-
solving procedures in general. Thus it was
predicted in Hypothesis Ic that general problem-
solving ability would be unrelated to the perfor-
mance of experts but would be moderately related
to the performance of novices.
Multiple regression analyses were conducted for
each of the three cross-country groups between the
number of information recursions made during
scenarios and both the accuracy and response
latency associated with the responses to the non-
sense syllogisms. A significant relationship was
found between the number of information recur-
sions made by novice pilots and both the accuracy
and response latency of their responses to the
syllogisms task. No relationships were found for
intermediate or expert pilots.
Measures of Experience
According to the principles embodied within the
cognitive expertise framework, the compilation of
task-related experiences requires that the indi-
vidual participate actively in the performance of a
particular task. This is an essential feature of
production system models of cognitive skill acquisi-
tion (Anderson, 1983; 1989; 1993). Thus in the
aeronautical domain, it might be expected that
flying experience, other than that associated with
weather-related decision making, would not lead
to the development of productions with which to
exercise skilled weather-related decisions. There-
fore, we predicted in Hypothesis Ib that total
flying experience would be less effective than
cross-country flying experience as a predictor of
differences between the weather-related decision-
making performance of inexperienced pilots and
either intermediate or experienced ones.
 318 WIGGINS AND O'HARE
The results for the number of information
screens accessed, the frequency of information
recursions, and the time spent examining the
information screens did not reveal any significant
effects either between- or within-subjects for those
groups classified according to total flying experi-
ence. In contrast, the analyses made on the basis of
pilot cross-country flying experience revealed sig-
nificant main effects for the three measured vari-
ables. Consequently, the performance between
groups was delineated more effectively by the
cross-country classification than by the total flying
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
This document is copyrighted by the American Psychological Association or one of its allied publishers.
experience classification. Combined, these results
provide empirical support for Hypothesis Ib: Dif-
ferences between experts and novices will be less
apparent when a global measure of experience,
such as total flying hours, rather than a proximal
measure, such as cross-country flying experience, is
used.
The distinction between the cross-country and
total experience classifications also provides sup-
port for the assumption that the acquisition of a
cognitive skill is dependent on the participation of
an individual repeatedly in the performance of
that skill (Ericsson & Charness, 1994). Thus, the
development of weather-related decision-making
skills is unlikely to occur if the pilot has not directly
experienced situations in which the application of
such skills are necessary. Moreover, it is not
possible to instill such skills solely through text-
books or even through demonstrations. The evi-
dence suggests that these resources need to be
combined with practical tasks in order to facilitate
the transition from the initial, declarative stage to
the advanced, procedural stages of skill acquisition
(Anderson, 1987).
In an interesting study of U.S. Air Force Class A
fighter mishaps, McKinney (1993) found that the
decisions made by highly experienced flight leads
in response to an unexpected mechanical malfunc-
tion were generally worse than those made by
less experienced flight leads. Experience was di-
chotomized into > 500 hr on type and less than
500 hr on type. Although flight leads are selected
as pilots of exceptional all-round ability, they do
not routinely engage in training in this particular
task. As in the present study, overall experience
was not synonymous with expertise in a particular
task.
Discussion
The present cross-sectional analysis of the
weather-related decision-making performance of
pilots with differing levels of cross-country flying
experience has provided evidence that expert-
novice differences can be captured in a laboratory
task. In particular, a qualitative distinction was
observed between both the information search and
problem-solving strategies used by inexperienced
and experienced pilots. This was interpreted as
qualified support for the notion that through
task-specific experience, individuals develop proce-
dures that can be generalized and applied subse-
quently to a variety of situations.
Quantitative differences were also observed be-
tween the information search strategies of inexpe-
rienced and experienced pilots, with the former
accessing a greater number of information screens;
making a greater number of information recur-
sions; and spending more time examining the
information screens than the latter. Inexperienced
pilots also exhibited a greater response latency in
selecting from a forced choice: whether to con-
tinue to the destination or return to the point of
take-off.
The main aim of cross-sectional analysis is to
determine whether differences exist between cer-
tain groups defined a priori. The main disadva-
tange associated with this discriminatory approach
is that transitory phases are often overlooked.
From a training perspective, this information is
extremely important if effective training programs
are to be developed. Trainers and educators must
not only be able to identify the quantity and quality
of the information required to effect the transition,
but they must also be able to identify the character-
istics associated with the transition. Thus future
research should undertake either a longitudinal
approach or a cluster analysis across participants
in order to determine at which point or points
along the continuum the significant transition from
novice to expert occur.
References
Anderson, J. R. (1980). Cognitive psychology and its
implications. San Francisco: Freeman.
Anderson, J. R. (1982). Acquisition of cognitive skill.
Psychological Review, 89, 369-406.
Anderson, J. R. (1983). The architecture of cognition.
Cambridge, MA: Harvard University Press.
 EXPERTISE IN AERONAUTICAL DECISION MAKING
Anderson, J. R. (1987). Skill acquisition: Compilation of
weak-method problem solutions. Psychological Re-
view, 94, 192-210.
Anderson, J. R. (1992). Automaticity and the ACT*
theory. American Journal of 'Psychology, 105, 165-180.
Anderson, J. R. (1993). Rules of the mind. Hillsdale, NJ:
Erlbaum.
Aircraft Owners and Pilots Association Air Safety Foun-
dation (1991). General aviation accident analysis book
for the years 1982-1988. Frederick, MD: AOPA Air
Safety Foundation.
Baddeley, A. D. (1968). A three-minute reasoning test
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
based on grammatical transformation. Psychonomic
This document is copyrighted by the American Psychological Association or one of its allied publishers.
Science, 10, 341-342.
Bainbridge, L. (1979). Verbal reports as evidence of the
process operator's knowledge. International Journal of
Man-Machine Studies, 11, 411-436.
Barnett, B., Stokes, A., Wickens, C. D., Davis, T.,
Rosenblum, R., & Hyman, F. (1987). A componential
analysis of pilot decision-making (pp. 842-846). In
Proceedings of the Human Factors Society 31st Annual
Meeting. Santa Monica, CA: Human Factors Society.
Beck, G. A. (1987). The problem of visual flight rules flight
into instrument meteorological conditions: A protocol
analysis of general aviation pilots' subjective weather
forecasting procedure. Unpublished master's thesis.
Columbus: Ohio State University.
Bedard, J. (1989). Expertise in auditing: Myth or reality?
Accounting, Organisations and Society, 14, 113-131.
Benner, L. (1975). D. E. C. I. D. E. in the hazardous
materials emergencies. Fire Journal, 69, 13-18.
Bouman, M. J. (1980). Application of information pro-
cessing and decision-making research. In G. R. Ung-
son & D. N. Braunstein (Eds.), Decision making: An
interdisciplinary inquiry. Boston, MA: Kent
Bryan, L. A., Stonecipher, J. W., & Aron, K. (1955).
AOPA's 180° rating. Washington, DC: AOPA Founda-
tion & Aircraft Owners and Pilots Association.
Ericsson, K. A., & Charness, N. (1994). Expert perfor-
mance: Its structure and acquisition. American Psy-
chologist, 49, 725-747.
Ericsson, K. A., & Simon, H. A (1984). Protocol analysis:
Verbal reports as data. Cambridge, MA: MIT Press.
Ericsson, K. A., & Smith, J. (1991). Toward a general
theory of expertise. Cambridge, England: Cambridge
University Press.
Fleiss, J. L. (1981). Statistical methods for rates and
proportions. New York: Wiley.
Gann, E. K. (1961). Fate is the hunter. London: Hodder
& Stoughton.
Guindon, R. (1990). Knowledge exploited by experts
319
during software system design. International Journal of
Man-Machine Studies, 33, 279-304.
Hammond, K. R., Hamm, R. M., Grassia, J., & Pearson,
T. (1987). Direct comparison of the efficacy of intui-
tive and analytical cognition in expert judgment. IEEE
Transactions on Systems, Man, and Cybernetics, 17,
753-770.
Hershey, D. A., Walsh, D. A., Read, S. J., & Chulef,
A. S. (1990). The effects of expertise on financial
problem solving: Evidence for goal directed, problem
solving scripts. Organizational Behavior and Human
Decision Processes, 46, 77-101.
Kirschenbaum, S. S. (1992). Influence of experience on
information-gathering strategies. Journal of Applied
Psychology, 77, 343-352.
Klein, G. A. (1989). Recognition-primed decisions
(RPD). In W. B. Rouse (Ed.), Advances in man-
machine systems (Vol. 5, pp. 47-92). Greenwich, CT:
JAI.
Klein, G., & Klinger, D. (1991). Naturalistic decision-
making. CSERIAC Gateway, 2, 1-4.
Larkin, J., McDermott, J., Simon, D. P., & Simon, H. A.
(1980). Expert and novice performance in solving
physics problems. Science, 208, 1335-1342.
Lusk, C. M. (1993). Assessing components of judgments
in an operational setting. In O. Svenson & A. J. Maule
(Eds.), Time pressure in human judgment and decision-
making (pp. 309-321). New York: Plenum Press.
McKinney, E. H. (1993). Flight leads and crisis decision
making. Aviation, Space, and Environmental Medicine,
64, 359-362.
Nisbett, R. E., & Wilson, T. D. (1977). Telling more
than we can know: Verbal reports on mental pro-
cesses. Psychological Review, 84, 231-259.
O'Hare, D., Wiggins, M., & Jones, G. (1995). A verbal
protocol analysis of expert aeronautical decision making.
Unpublished manuscript.
O'Hare, D., Wiggins, M., Batt, R., & Morrison, D.
(1994). Cognitive failure analysis for aircraft accident
investigation. Ergonomics, 37, 1855-1869.
Orasanu, J., & Connolly, T. (1993). The reinvention of
decision making. In G. A. Klein, J. Orasanu, R.
Calderwood, & C. E. Zsambok (Eds.), Decision mak-
ing in action: Models and methods. Norwood, NJ:
Ablex.
Rasmussen, J. (1982). Human errors: A taxonomy for
describing human malfunction in industrial installa-
tions. Journal of Occupational Accidents, 4, 311-333.
Ryan, A. P. (1984). Flying weather: Manual of New
Zealand aviation climatology. Wellington, New Zea-
land: New Zealand Meteorological Service.
 320 WIGGINS AND O'HARE
Sanderson, P. M., James, J. M., & Seidler, K. S. (1989).
SHAPA: An interactive software environment for
protocol analysis. Ergonomics, 32, 1271-1302.
Shanteau, J. (1988). Psychological characteristics and
strategies of expert decision-makers. Acta Psychologia,
68, 203-215.
Shiffrin, R. M., & Schneider, W. (1977). Controlled and
automatic human information processing: II. Percep-
tual learning, automatic attending, and general theory.
Psychological Review, 84, 127-190.
Stokes, A. F., Kemper, K. L., & Marsh, R. (1992).
Time-stressed flight decision making: A study of expert
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
and novice aviators (Technical Report ARL-93-1/
This document is copyrighted by the American Psychological Association or one of its allied publishers.
INEL-93-1). Urbana-Champaign, IL: Aviation Re-
search Laboratory, University of Illinois.
Stokes, A. F., & Raby, M. (1989). Stress and cognitive
performance in trainee pilots. In Proceedings of the
Human Factors Society 33rd Annual Meeting (pp.
883-887). Santa Monica, CA: Human Factors Society.
aaaar-sgf
Terranova, M., Snyder, C. E., Seamster, T. L., &
Treitler, I. E. (1989). Cognitive task analysis: Tech-
niques applied to airborne weapons training. In Pro-
ceedings of the Human Factors Society 33rd Annual
Meeting (pp. 1358-1362). Santa Monica, CA: Human
Factors Society.
Wiggins, M. W., & O'Hare, D. (1993). A skills-based
approach to training aeronautical decision-making. In
R. A. Telfer (Ed.), Aviation instruction and training
(pp. 430-476). Aldershot, England: Ashgate.
Woods, D. D. (1988). Coping with complexity: The
psychology of human behaviour in complex systems.
In L. P. Goodstein, H. B. Andersen, & S. E. Olsen
(Eds.), Tasks, errors and mental models (pp. 128-148).
London: Taylor & Francis.
Received June 27,1994
Revision received June 22,1995 m
Accepted June 28,1995
gsaaeaar _ «saaa
ZSSSgSS&m