Exploring the Underpinning Principles of the Multi-Crew Pilot Licence

by Melanie A Todd

BAvn, MSciTech(Avn)

A thesis submitted in fulfilment of the requirement for the degree of

Doctor of Philosophy

in the School of Psychology

Division of Education, Arts and Social Sciences

University of South Australia

Submitted March 2014

 Table of Contents
List of figures ............................................................................................................. vi
List of tables ............................................................................................................. vii
Glossary .................................................................................................................. viii
Summary ................................................................................................................... x
Acknowledgements ................................................................................................. xiii

Chapter One - Literature review ............................................................................. 1

1.1 Overview .......................................................................................................... 2
1.2 The discipline of Human Factors...................................................................... 5
General overview ................................................................................................ 5
Aviation Human Factors ..................................................................................... 6
1.3 Expertise ........................................................................................................ 10
Pilot skill............................................................................................................ 15
Technical skills ................................................................................................. 15
Psychomotor skills ............................................................................................ 16
Non-technical skills ........................................................................................... 17
Aeronautical decision-making ........................................................................... 18
Naturalistic Decision making............................................................................. 18
Recognition-Primed Decision Making ............................................................... 19
Fast and Frugal Heuristics ................................................................................ 20
NDM and Fast and Frugal ................................................................................ 21
Workload .......................................................................................................... 22
Situational awareness....................................................................................... 23
Memory............................................................................................................. 26
Summary .......................................................................................................... 27
1.4 Learning and skill acquisition ......................................................................... 29
Principles of adult learning................................................................................ 29
Situated learning ............................................................................................... 30
Skill acquisition ................................................................................................. 32
Simulation training ............................................................................................ 35
Transfer of training from simulator to aircraft .................................................... 36
Fidelity in simulation ......................................................................................... 39
Summary .......................................................................................................... 44
1.5 Competency-based Training .......................................................................... 47
Definition........................................................................................................... 47
ii
 Background ...................................................................................................... 47
Philosophy ........................................................................................................ 47
The Australian experience and perspective ...................................................... 49
Competency-based training in Aviation ............................................................ 53
Benefits of competency-based training ............................................................. 55
Criticisms of competency-based training .......................................................... 58
Contextual learning and assessment ................................................................ 61
Summary .......................................................................................................... 64
1.6 Regulation of pilot training and licensing ........................................................ 66
International regulation ..................................................................................... 66
Australian regulation ......................................................................................... 67
1.7 Pilot Training philosophy and methodologies ................................................. 68
History and international influence .................................................................... 68
Australian model ............................................................................................... 69
Innovations in pilot training ............................................................................... 69
Criticisms of pilot training models ..................................................................... 70
Summary .......................................................................................................... 71
1.8 Rationale for studies ...................................................................................... 73

Chapter Two - Experience, competence or syllabus? Influences on Flight

Hours at Licensing of Commercial Pilots ........................................................... 76
2.1 Flight Training ............................................................................................ 77
Competency-Based Training ............................................................................ 77
Competency-Based Training in Australia.......................................................... 78
Competency-Based Training in Australian Aviation .......................................... 80
2.2 Exploratory Study ....................................................................................... 82
2.3 Results ....................................................................................................... 83
2.4 Discussion.................................................................................................. 87
Regulatory minimum hours ............................................................................... 87
Syllabus requirements ...................................................................................... 88
2.5 Conclusion ................................................................................................. 90

Chapter Three - Flight Hours and Flight Crew Performance in Commercial

Aviation .................................................................................................................. 91
3.1 Introduction ................................................................................................ 92
3.2 Method ....................................................................................................... 97
Subjects ............................................................................................................ 97

iii
 Design and Procedure ...................................................................................... 98
Measures .......................................................................................................... 99
Statistical Analysis .......................................................................................... 101
3.3 Results ..................................................................................................... 102
3.4 Discussion................................................................................................ 105

Chapter Four - Pilot Experience and Performance in an Airline Environment

.............................................................................................................................. 108
4.1 Introduction .............................................................................................. 109
4.2 Literature review ...................................................................................... 112
General Aviation pilots .................................................................................... 112
Airline pilots .................................................................................................... 114
4.3 Methodology ............................................................................................ 117
Data sources .................................................................................................. 117
Design and procedure .................................................................................... 117
Statistical analysis .......................................................................................... 123
4.4 Results ..................................................................................................... 124
Airline A .......................................................................................................... 124
Airline B .......................................................................................................... 128
Airline C .......................................................................................................... 135
4.5 Discussion................................................................................................ 139
Simulator proficiency check results ................................................................ 140
Line check results ........................................................................................... 143
Limitations ...................................................................................................... 145
4.6 Conclusion ............................................................................................... 147

Chapter Five - Discussion and future directions .............................................. 149

5.1 Issues facing pilot training ........................................................................ 150
Background .................................................................................................... 150
5.2 Summary of findings ................................................................................ 154
Study 1 – Competency-based training ............................................................ 154
Study 2 – Pilot performance in an airline environment ................................... 155
Study 3 – Pilot performance in normal and non-normal operations ................ 156
5.3 Competency-based training ..................................................................... 157
Training and assessment................................................................................ 157
5.4 Pilot performance ..................................................................................... 161
5.5 Future training .......................................................................................... 164

iv
 5.6 Pilot performance in non-normal scenarios .............................................. 168
5.7 Future research ........................................................................................ 169
5.8 Conclusion ............................................................................................... 172

References ........................................................................................................... 173

Appendix 1 Permission from publishers .......................................................... 204

Appendix 2 Author Contribution ........................................................................ 206

v
 List of figures

Figure 2.1: Box Plots of Experience at Licence Issue as a function of Flight Training
Organisation ............................................................................................................ 85
Figure 4.1: Performance in proficiency simulator check as a function of entry
pathway ................................................................................................................. 125
Figure 4.2: Performance in simulator proficiency check as a function of flight hours
.............................................................................................................................. 126
Figure 4.3: Performance in line check as a function of entry pathway .................. 127
Figure 4.4: Performance in line check as a function of flight hours ....................... 128
Figure 4.5: Performance in simulator proficiency check manoeuvres as a function of
entry pathway ........................................................................................................ 129
Figure 4.6: Performance in simulator proficiency check as a function of entry
pathway ................................................................................................................. 130
Figure 4.7: Performance in simulator proficiency check manoeuvres as a function of
flight hours............................................................................................................. 131
Figure 4.8: Performance in simulator proficiency check as a function of flight hours
.............................................................................................................................. 132
Figure 4.9: Line check at 5 year mark as a function of entry pathway ................... 133
Figure 4.10: Line check at 5 year mark as a function of flight hours ..................... 134
Figure 4.11: Performance in simulator proficiency check as a function of entry
pathway ................................................................................................................. 135
Figure 4.12: Performance in simulator proficiency check as a function of flight hours
.............................................................................................................................. 136
Figure 4.13: Line check as a function of entry pathway ........................................ 137
Figure 4.14: Line check as a function of flight hours ............................................. 138

vi
 List of tables

Table 2.1: Descriptive Summary of Hours at Licence Issue .................................... 83

Table 3.1: Stabilised approach performance (at 500’), autopilot disconnect height,
and TEM performance by experience group ......................................................... 103
Table 3.2: Non-technical skills criteria and performance (mean, SD) by experience
group ..................................................................................................................... 104
Table 4.1: Airline A - Technical (T), Non-technical (N) and required regulatory
manoeuvres (M) .................................................................................................... 122
Table 4.2: Airline B - Technical (T), Non-technical (N) and required regulatory
manoeuvres (M) .................................................................................................... 122
Table 4.3: Airline C - Technical (T), Non-technical (N) and required regulatory
manoeuvres (M) .................................................................................................... 123
Table 4.4: Differences in performance between cadets and direct entry second
officers .................................................................................................................. 124
Table 4.5: Differences in performance between low- and high-hour second officers
.............................................................................................................................. 126
Table 4.6: Differences in performance between low- and high-hour first officers .. 130
Table 4.7: Overview of significant differences in performance across all three
airlines for simulator (S) and line (L) checks as a function of entry pathway (-
indicates not tested) .............................................................................................. 139
Table 4.8: Overview of significant differences in performance across all three
airlines for simulator (S) and line (L) checks as a function of flight hours (- indicates
not tested) ............................................................................................................. 140

vii
 Glossary

A330 Airbus A330 aircraft

AF447 Air France Flight 447

AFE Above Field Elevation

AIPA Australian and International Pilots Association

ANOVA Analysis of Variance

ANTA Australian National Training Authority

ATC Air Traffic Control

ATP Air Transport Pilot

ATSB Australian Transport Safety Bureau

ATPL Air Transport Pilot Licence

AVI08 Aviation Training Package

AQF Australian Qualifications Framework

AQTF Australian Quality Training Framework

BASI Bureau of Air Safety Investigation

BEA Bureau d’Enquêtes et d’Analyses pour la sêcuritê de I’aviation civile

BITRE Bureau of Infrastructure, Transport and Regional Economics

CAR Civil Aviation Regulation

CASA Civil Aviation Safety Authority

CBT Competency-based Training

CPL Commercial Pilot Licence

CRM Crew Resource Management

DHC-8 Bombardier Dash 8 aircraft

FAA Federal Aviation Administration

FCLTP Flight Crew Licensing and Training Panel

FOQA Flight Operations Quality Assurance

FTO Flight Training Organisation

GA General Aviation

viii
ICAO International Civil Aviation Organization

ILS Instrument Landing System

ILS MAP Instrument Landing System Missed Approach

IMC Instrument Meteorological Conditions

LOFT Line Orientated Flight Training

LOS Line Operational Simulations

LOSA Line Operational Safety Audit

LSD Fisher’s Least Significant Difference

MECIR Multi-Engine Command Instrument Rating

MPL Multi-Crew Pilot Licence

NCVER National Centre for Vocational Education Research

NDM Naturalistic Decision Making

NTF National Training Framework

NTS Non-Technical Skills

NTSB National Transportation Safety Board

PF Pilot Flying

PPL Private Pilot Licence

RPD Recognition-Primed Decision Making

RPT Regular Public Transport

SA Situational Awareness

SARPS Standards and Recommended Practices

S.D. Standard Deviation

SOP Standard Operating Procedure

SPL Student Pilot Licence

TAA Technically Advanced Aircraft

TAFE Technical and Further Education

TEM Threat and Error Management

VET Vocational Education and Training

VFR Visual Flight Rules

ix
 Summary

At just over 100 years old, aviation is relatively new compared to other modes of
transport, and in that time there have been innovations in nearly all aspects of the
industry. Improved aircraft manufacturing materials and techniques, and the
introduction of aviation human factors, have greatly improved the reliability and
safety of air travel. However, save the introduction of simulation and human factors
to the training syllabus, pilot training is still predominately based upon a model from
the Second World War.

In 2006, the International Civil Aviation Organization (ICAO) introduced a new type
of pilot licence. The Multi-Crew Pilot Licence (MPL) was designed to enable pilots to
enter an airline directly from the training environment and commence flying
immediately as part of a two-crew operation.

This new licence created much debate within the aviation industry, as it presented a
very different model from the traditional approach, where pilots would accumulate
significant flight hours in general aviation, before gaining entry into a large
passenger jet airline. In particular, some in the aviation industry argued that MPL
pilots would be less safe than their high-hour non-cadet peers. In addition, the
introduction of the MPL relied on competency-based training as a risk mitigator.
Competency based training is “…geared to the attainment and demonstration of
skills to meet industry-specified standards rather than to an individual’s achievement
relative to that of others in a group” (National Centre for Vocational Education
Research, 1999).

The three studies in this thesis were conducted with the overall aim of evaluating the
underpinning principles of the MPL, and test empirically some of the potential
criticisms of this new form of pilot training.

The first study examined Australian Commercial Pilot Licence data for the hours
pilots held at licence issue grouped by flight training organization. The results
showed that the effect of the flight training organization and airline client
requirements appeared to influence the amount of hours commercial pilots obtained
prior to licensing to a greater degree than would be expected in a competency-
based system. Another strong effect on the data appeared to be the hours-based
regulatory requirements that pilots had to meet to in order to hold a licence.

x
The second study used line operations safety audit (LOSA) observations of airline
pilots and examined their performance as a function of flight hour experience
against technical, non-technical and stabilized approach metrics. The pilots were
sorted by role into low- and high-hour groups for comparison. The results showed
no significant differences against the technical measures, however there were minor
differences in non-technical performance as a function of crew composition.

The third study examined airline pilot performance in both simulator and line (day-to-
day flying) checks, as a function of both flight hour experience and entry pathway to
the airline. The study covered both normal and non-normal scenarios. Pilots were
sorted into low- and high-hour groups and also grouped by entry pathway. Pilots
who entered as cadets with no previous flying experience were in the ‘cadet’ group,
and pilots who had flown for other airlines or in general aviation or the military were
in the ‘direct entry’ group. The results showed that all pilots were rated as proficient
in the checks. However, there were some differences in the level of performance
based on experience and entry pathway. Overall there was no systematic difference
across airlines that would indicate a problem with the cadet pathway or low-hour
pilots. The data may be useful to inform airlines on the areas to tailor their training in
order to reduce the differences that do exist.

The studies presented in this thesis demonstrate that;

 competency-based aviation training in Australia does not appear to be

occurring;
 low-hour pilots do not differ significantly from high-hour pilots in their
performance within an airline environment.
 the cadet pathway currently in place produces proficient pilots;

This highlights the need for regulators to understand the system they have before
introducing any change and also demonstrates the benefit of using data to inform
changes to pilot training. Further research into entry pathways to airlines and the
performance of MPL pilots is recommended.

This thesis is submitted as a Thesis Containing Published Research in accordance

with UniSA Academic Regulations for Higher Degrees by Research Clause 16.1.1b.

xi
Declaration

I declare that this thesis presents work carried out by myself and does not
incorporate without acknowledgement any material previously submitted for a
degree or diploma in any university; to the best of my knowledge it does not contain
any materials previously published or written by another person except where due
reference is made in the text; and all substantive contributions by others to the work
presented, including jointly authored publications, is clearly acknowledged.

Signed … ………………….

Date…02/08/2014…………………………….

xii
 Acknowledgements

I would like to sincerely thank my supervisors: Dr Matthew Thomas, for his endless
enthusiasm for this work, his immense patience, guidance, optimism and
encouragement; and Dr Richard Batt for his guidance, support, motivation and good
humour.

I would like to thank Dr Graham Edkins and Patrick Murray for convincing and
enabling me to start this project and the Civil Aviation Safety Authority for assistance
with the study in Chapter 2.

Thank you to Dr Stuart Godley, Malcolm Christie, Julian Walsh and Martin Dolan at
the ATSB for the excellent feedback and support for the study in Chapter 4.

Thank you to the airlines involved in the studies in Chapter 3 and 4 for enabling me
to conduct my research and for their willingness to contribute to the scientific
knowledge on pilot performance.

A very big thank you to Dr Kirsty McCulloch, for her ongoing advice and therapeutic
administration of wine.

A special thanks to my parents for teaching me that I can do anything if I work hard
enough at it and encouraging me to fulfil my dreams, and to my sister for her
ongoing support and encouragement.

Finally, an enormous thank you to my husband James, for his amazing support and
belief in me and his endless understanding when I was filled with doubt, stress and
worry, I am very lucky to have him.

xiii
Chapter One - Literature review

1
1.1 Overview

Despite the complexity of modern aircraft, adverse weather conditions and

increasing traffic levels, aviation as an industry is considered to be an ultra-safe
system. The safety of an ultra-safe system is measured as the chance of a
disastrous accident being around “5 x 10-7” (Amalberti, 2001). However, despite the
advancements in aeronautical engineering, including aircraft materials, engines,
avionics etc., the training system supporting and producing airline pilots has not
made comparable progress. In part this could be argued to be a product of the
success of the system and the high amount of regulation that coincides with ultra-
safe systems. This can result in a tendency for regulators and operators to maintain
the status quo, even when this means that additional restrictions are added to the
system and nothing is considered for removal (Amalberti, 2001).

However, this desire to fix problems by adding more to a system is also noticeable
when an aviation accident does occur, especially when it involves a large passenger
carrying jet aircraft. The accident is often accompanied by societal outrage that such
a disaster could occur and corresponding calls for justice, explanations and
punishment. An example of precisely this reaction, and the corresponding regulatory
‘solution’, is the accident involving a Bombardier DHC-8 at Buffalo, New York in
2009. The investigation highlighted issues with pilot selection and training, fatigue
and crew communication, amongst other things (NTSB, 2010).

In response to the accident at Buffalo, the United States Congress enacted

legislation to increase the amount of flying experience that first officers would need
in order to enter an airline (United States Congress, 2010).
Whether this decision was supported by empirical evidence is largely questionable
given a lack of available literature comparing pilot performance above and below the
thresholds stated in the legislation. However it is an example of a reaction to a
problem that occurs within an ultra-safe system which leads to increased restrictions
within that system.

The structure of pilot licences has remained fairly constant since 1948 and they are
not dramatically different to the earliest standards set out in the 1919 convention for
the Regulation of Aerial Navigation (Dow & Defalque, 2013). However, it has been

2
recognised that the traditional pilot training structure relies upon “prescriptive
requirements with much attention being given to experience” and does not take into
account advancements in aircraft systems, technology and systematic approaches
to training (Dow & Defalque, 2013).

This traditional training system relied upon pilots accumulating total flight hours,
instrument flight hours, cross-country flight hours and time in command (or solo
time, that is without an instructor) in order to meet the minimum requirements to
undertake a flight test and, upon successful completion, become licensed as a
commercial pilot. In many countries such as Australia and the United States, this
model then promoted the further accumulation of flight hours experience within
general aviation before commercial pilots could meet the hour requirement to hold
an Air Transport Pilot Licence and therefore be able to fly as a captain in an airline.

However, some attempts have been made to recognise that the aviation industry of
today is very different from the aviation industry of the 1940’s, when many of the
initial pilot training regulations were developed. The International Civil Aviation
Organization (ICAO) has enacted standards and recommended practices that state
all flight training should be conducted under a competency-based training system
(ICAO, 2006b) and has introduced a new type of pilot licence, the Multi-Crew Pilot
Licence (MPL), somewhat in response to the recognition that the industry has
evolved (ICAO, 2006a). The MPL was also formulated to produce airline pilots in a
faster training regime than normal, who would be first officers, able to occupy a
control seat of the cockpit (that is, conduct take-offs and landings) as soon as they
were finished training. Traditionally, pilots entering airlines at the completion of
airline training have been second officers (who do not conduct take-offs and
landings in the aircraft), who then progressed to the role of first officer after a
number of years or flight hours accrued.

The MPL created strong debate within the aviation industry, and numerous debates
on the topic of low-hour pilots (a cohort of which MPL is part). The one point which
remains constant is the view that hours equals experience and therefore skill, which
acts to ensures safe performance (MacFarlane, 2003; Sutton, 2008; ALPA, 2009).
Indeed, flight hours experience is emphasised in every part of aviation, from training
to airline selection to promotion and has been used in many studies of general

3
aviation (non-airline) pilots to evaluate performance as well as in accident and
research reports (Groff & Price, 2006; Bazargan & Guzhva, 2011; O’Hare &
Chalmers, 1999; Wiggins & Bollwerk, 2006; Wiggins & O’Hare, 2003). While it is
clear that flight hours and experience (and the resulting assumed skill level) are
seen as synonymous, what is lacking is any evidence that this apparent correlation
has been tested for airline pilots. Yet the training system has, for numerous years,
relied on them accumulating flight hours, often in general aviation, to a specified
minimum which allows them entry to an airline. To have never tested this particular
minimum appears quite odd and yet the “magic” number of 1,500 hours (the
minimum needed for an Air Transport Pilot Licence and the figure chosen by the
U.S. Congress as the minimum for first officers) has been used as a selection figure
in many airlines for many years (United States Congress, 2010).

As part of the introduction of the MPL, ICAO stated that competency-based training
would be a cornerstone of MPL training and would improve standardization
worldwide, which would act as a safety benefit (ICAO 2005). ICAO had already
introduced competency-based training as a requirement for flight training for the
commercial pilot licence, so its use in MPL should not have been considered
revolutionary or indeed innovative. However it became clear that competency-based
training in aviation had not been well examined in the literature to establish whether
or not it was occurring, and if it was occurring, whether it was producing the desired,
successful outcome.

This chapter reviews the previous research relating to the studies presented in
subsequent chapters. Section 1.2 provides an overview of the science of human
factors, which is the discipline to which the research studies of this thesis belong.
Section 1.3 presents existing research on expertise and decision making. Section
1.4 provides an overview of skill acquisition, including the use of simulation for skill
acquisition. Section 1.5 outlines the key concepts of competency-based training,
including the benefits and criticisms. Section 1.6 outlines the regulation of pilot
training and licensing. Section 1.7 presents an overview of pilot training. Finally,
Section 1.8 provides a summary and rationale for the studies reported in
subsequent chapters.

4
1.2 The discipline of Human Factors

General overview
The discipline of pilot training was one of the earliest endeavours of the science of
aviation human factors. The science of human factors covers a range of areas
including ergonomics, cognitive science, psychology, sociology and physiology.
Various definitions have been given to cover human factors, with the most often
cited being “Human factors (or ergonomics) may be defined as the technology
concerned to optimize the relationships between people and their activities by the
systematic application of the human sciences, integrated within the framework of
system engineering” (Edwards, 1988, p9). A simpler, but equally valid definition is
used by The Australian Transport Safety Bureau (ATSB) and is “the multi-
disciplinary science that applies knowledge about the capabilities and limitations of
human performance to all aspects of the design, operation, and maintenance of
products and systems” (ATSB, 2011a, p47). This definition allows human factors to
be placed outside of the strict ergonomic approach it started with, and ensures that
systemic influences on human performance and behaviour are considered as part of
the application of the science. At the most fundamental level, human factors aims to
optimise the interaction between humans and technology, including the individual’s
interaction with equipment, the physical environment, procedures and other people.

The origin of the science of human factors dates back to the late 1940s in the United
Kingdom (UK), where a group of scientists from various disciplines formed the
Ergonomics Research Society, which later became the Ergonomics Society
(Edwards, 1988). The society was formed with the aim of looking at the human
aspects of the work environment. In the United States of America (USA) similar
developments were occurring, which led to the formation of the Human Factors
Society. Both societies were born out of initial support for the war effort and in the
US, the focus stayed on the defence and aerospace industries (Edwards, 1988).

The specific discipline of aviation human factors was developed in response to

many of the human performance issues encountered by aviators during the Second
World War. In 1939, the head of the Cambridge University applied psychology
laboratory was tasked with exploring this area in depth within military aviation. The

5
outcomes of this project included “contributions in the areas of aircrew selection and
training, the effects of sleep loss and fatigue, and various aspects of visual
perception and display design” (Edwards, 1988, p7). The study also found that “The
hallmark of an expert is his ability to “get his act together”; this involves anticipation,
planning and timing” (Edwards, 1988, p8). Interestingly it was noted that “less-skilled
performers may well demonstrate equal prowess in the execution of such individual
subroutines as tracking particularly instruments, but they lack the ability to integrate
these partial skills into a total coherent performance” (Edwards, 1998, p8). These
findings still have relevance to pilot training today, and in particular foreshadow a
concern relating to the use of competency-based training in that the demonstration
of skill in a particular task does not necessarily guarantee overall competency in a
role (Capper, 2000; Descy & Tessaring, 2001).

Aviation Human Factors

Aviation human factors is the application of the human factors discipline within the
aviation industry. The aviation industry, as it applies to large passenger carrying
aircraft, is considered to be a highly reliable and ultra-safe system (Amalberti, 2001).
This status has been achieved as a result of the integration of human factors in
aircraft design (predominately and particularly ergonomics); pilot training – including
the introduction of non-technical training and in particular crew resource
management (CRM) training; air traffic control; aircraft maintenance and
engineering. Interventions by the International Civil Aviation Organisation (ICAO)
and various national regulators worldwide recognise and incorporate human factors
as a cornerstone of the industry and its best practices.

Some of the first examples of applied aviation human factors are in the area of pilot
selection, training and ergonomics and comes from the work conducted by the
aeromedical laboratory of the American military in the 1940s. This work was
conducted to understand and ultimately combat ‘human error’ accidents and
incidents and to improve pilot performance in general. One study examined “pilot-
error” experiences in reading and interpreting aircraft instruments with an outcome
from this being a recognition that without an improvement in instrument design to
accommodate interpretation by pilots, the “training time to reach proficiency in
instrument flight will continue to be unduly long…” (Fitts & Jones, 1947, p28).
Further work was done on the effect of force and amplitude cues on pilots learning

6
tracking tasks, and explorations of transfer of training effects on pilots (Briggs, Fitts
& Bahrick, 1957; Briggs & Waters, 1958). In addition, the aeromedical laboratory
also focused on psychological research of equipment design to further aid pilot
performance, in conjunction with appropriate selection of pilots and extensive
training (Fitts, 1947).

One of the mechanisms for exploring human factors in aviation has been through
accident investigation. Various international accident investigation agencies employ
human performance or human factors investigators to ensure their investigations
consider the role of the human and system in an accident. The seminal work of
Professor James Reason through the development and application of his Model for
Accident Causation helped to highlight the importance of investigating human
factors in high reliability systems, including aviation (Reason, 1991).

The ATSB, then known as the Bureau of Air Safety Investigation (BASI) was the first
investigation agency to apply Reason’s model to an investigation (BASI, 1994). The
resulting report highlighted a number of organisational influences within the airline
and regulator which contributed to the accident. The consideration of organisational
influences is now commonplace and often cited within investigation reports
internationally and has allowed investigators to highlight safety issues with
regulations, training, procedures and aircraft design. This consideration has become
embedded in the ATSB’s taxonomy and methodology in relation to accident analysis
and models of causation (Walker & Bills, 2008).

In addition, the various exploratory studies of, and research into, human
performance within aviation have allowed the industry to recognise the importance
of identifying underlying systemic problems, instead of focusing on the individual as
the ‘cause’ of an accident or incident (Reason, 1991; Dekker, 2011; Dismukes,
Berman & Loukopoulos, 2007). This work has allowed the aviation industry to
recognise the importance of building a robust and resilient system with
acknowledgement of the fact that human error is inevitable and the aim of ensuring
the system can cope with these errors and either prevent an incident or accident, or
mitigate the consequences.

7
Furthermore, research into human performance has allowed the identification of
factors which contribute to human error, generally by degrading performance, such
as fatigue, stress, distractions, interruptions, information processing limitations and
various areas which can influence performance in teams, such as communication
and team management (Martin, 2013; Wickens & McCarley, 2008; Dismukes, 2006;
Dismukes, Berman & Loukopoulos, 2007).

Equally important to the area of human factors, has been the recognition of the
importance of human performance in recovering systems from an unsafe state.
While human performance may contribute to unsafe conditions, it is also the
innovation, adaptation, compensation, intelligence, resilience and motivation of
humans that allows the unforeseen and undesired outcome to be anticipated,
avoided or recovered. There have been examples of pilots recovering from
situations which were either never anticipated; believed to be too remote to
conceivably occur; or from which recovery seemed impossible (Reason, 2008).
These recoveries have been labelled as ‘heroic’ by Professor James Reason and in
some cases this is due to the individual pilot that responded on the day as opposed
to the system support available. However, it has been acknowledged that in some
cases, the ‘heroic’ recovery of a situation could be anticipated to have occurred with
different individuals in the main recovery roles, primarily as a function of appropriate
selection, adequate training (including to the point of developing automaticity of
actions in personnel) and organisational support which in turn developed skill and
expertise in individuals, along with the resilience to cope in unforeseen and unsafe
situations (Reason, 2008).

Accident investigation, while primarily focused on uncovering the individual actions,

local conditions and organisational contributions to an accident, can also highlight
the positive side of human performance. For example, the following two
investigations which examined accidents involving high-capacity aircraft in airline
operations had findings relating to the positive decisions and judgement in the
reaction of the pilots. Specifically that: “The captain’s selection of Take-off/Go-
Around (TO/GA) thrust during the rotation manoeuvre very likely limited the adverse
consequences of the runway overrun” (ATSB, 2011a) and “The flight crew’s
responses to the warnings and cautions, the pitch-down events, and the

8
consequences of the pitch-down events, demonstrated sound judgement and a
professional approach” (ATSB, 2011b).

The contribution of the expertise and skill of personnel involved in ultra-safe

systems cannot be overlooked when examining the success of that system. It is for
this reason that the skill acquisition and expertise development of pilots is important
in assuring the on-going safety of aviation as an industry. The individual’s ability to
adapt, compensate and recover from potentially fatal events should be
acknowledged within the overall safety of the system, particularly when undue
consideration is given to humans as hazards, as opposed to their equal, but less
visible role as heroes (Reason, 2008).

Part of the individual’s ability to adapt will rely on the underpinning knowledge
obtained from both initial and on-going training. In aviation it is therefore important
that the training system provides the necessary foundation from which novice pilots
can progress to a level of expertise through knowledge and skill acquisition. The
training system is therefore an important part of ensuring aviation remains an ultra-
safe system by supporting the development of expertise through initial and recurrent
training, particularly that which will enhance the resilience of individuals.

9
1.3 Expertise

Expertise is generally regarded as a refinement of skill and judgement that is

obtained the longer an individual spends in a certain industry or role, the
acknowledged definition being “...that experts show high, outstanding, and
exceptional performance that is domain-specific, stable over time, and related to
experience and practice” (Herbig & Glockner, 2009).

There are many components that contribute to an individual’s expertise within a

specific domain, including aviation. It is recognised that pilot expertise is a
combination of the physical/psychomotor skills necessary to control the aircraft, as
well as the technical and non-technical knowledge and skills required to manage the
flight and successfully cope with non-normal situations. Within the area of non-
technical skills, the subjects of decision-making, workload, situational awareness
and memory are highlighted in the research as being points of difference between
novice and expert pilot performance in various scenarios (Wiggins & O’Hare, 1995;
Wiggins et al., 2002; Wiegmann, Goh & O’Hare, 2002; Dussault, Jouanin, Philippe &
Guezennec, 2005; Schriver, Morrow, Wickens & Talleur, 2008; Endsley, 1995).
Together, these attributes define what expert pilot performance involves and will be
used as the foundation for exploring expertise in this thesis.

However, it should be acknowledged that often expertise is viewed solely as a

function of time within a system or domain, with novices being new to the area and
experts having years of experience within the system. In most industries expertise is
linked to seniority, and the underlying assumption is that their experience gives the
individual expertise (McDaniel, Schmidt & Hunter, 1988; Potter & Fiedler, 1981). In
exploring Naturalistic Decision Making (NDM) and Recognition-Primed Decision
making (RPD) in fire-fighting, Klein used Fire Commanders as the ‘experts’, a link
which recognises the contribution of seniority and years in the role (Klein, 1993).

In contrast, the performance of experts versus non-experts, in respect of knowledge,

was studied in relation to predicting the outcome of the first round of the (soccer)
World Cup in 2002. The results showed the experts performed no better than the
non-experts in choosing who would go through to the second round. In addition, the
experts overestimated their ability to produce accurate results, whereas the non-
experts judged their ability to forecast reasonably well (Andersson, Ekman &
Edman, 2003). One difference between experts and novices that is common in the
10
literature is that while both groups apply the same general skills to a problem, the
novices lack the exposure of experts in accumulating process and strategies to
apply to the problem. They also appear to lack experience in knowledge retrieval
from previous events or from the literature (Jeffries, Turner, Polson & Atwood, 1981;
Klein, 1993; Wiggins, Stevens, Howard, Henley & O’Hare, 2002; Wiggins &
Bollwerk, 2006; Wiggins & O’Hare, 1995). When set a software design problem,
experts outperformed novices not because they were familiar with that software or
design rules, but because their “years of experience enable the procedures of the
schema to become automatic, freeing the designer to focus more on the details of
the specific problem” (Jeffries et al., 1981, p281). As these processes develop
further, the ability of the individual to deal with complex problems increases, thereby
adding to that individual’s level of expertise.

It is common in many industries to view the accumulation of hours or years of

exposure to the industry or events in question as building experience. This view is
particularly prevalent in regard to road users, in part because the variability in
performance as a function of age is highly visible to most of the population. In line
with intuitive views of driving ability, the correlation between age and experience has
been shown in the negative for both young and senior drivers. Experience is
generally measured as years of licensure and as age is often included in road
accident studies, this correlation has been easy to study. The accident rate of low
experience drivers in the UK was shown to decline with the number of years the
driver held a full licence, however it was noted that age was a major contributor to
this, meaning it was low experience young drivers that presented the greatest risk
(Clarke, Ward, Bartle & Truman, 2006). Specifically, the ability of inexperienced
drivers to respond to a risky scenario was lowered if the scenario was demanding in
terms of driving skill (Fisher, Laurie, Glaser, Connerney, Pollatsek, Duffy & Brock,
2002). At the other end of the experience spectrum, drivers who could be classified
as highly experienced based on years of licensure but whom were aged above 65
consistently exhibited a decreased performance in driving and navigating
concurrently, as well as performing less well at trip planning and scanning the
instruments and environment and made more safety-related errors (Dingus, Hulse,
Mollenhauer, Fleishman, McGehee & Manakkal, 1997). It is informative to note that
the intuitive view on experience in relation to driver’s performance had validity,
however it appears that experience will only act as a protector of performance to a
certain point beyond which performance is negatively affected.

11
In aviation, many studies have examined accident risk and pilot performance as a
function of expertise, with the measure of expertise being flight hours experience
(Bazargan & Guzhva, 2011; Goh & Wiegmann, 2002; Jarvis & Harris, 2007a, 2007b;
Jarvis & Harris, 2008; Li, Baker, Grabowski, Qiang, McCarthy &Rebok, 2003; Li &
Baker, 2007 Salvatore, Stearns, Huntley & Mengert, 1986; Wiggins & O’Hare,
1995), with pilot decision making often reviewed as the performance metric
(Schriver, Morrow, Wickens & Talleur, 2008; Wiegmann, Goh & O’Hare, 2002;
Wiggins et al., 2002; Wiggins & Bollwerk, 2006; Wiggins & O’Hare, 2003). Many of
these studies have found a positive link between experience and performance,
particularly in the area of accident involvement and causation, with some linking
increased experience with lower accident involvement and lower error rates and
improved performance (Bazargan & Guzhva, 2011; Jarvis & Harris, 2007b; Jarvis &
Harris, 2008; Li et al., 2003; McFadden, 1996; Myers, 1974; Pagan & De Voogt,
2008; Pagan, De Voogt & van Doorn, 2006; Salvatore et al., 1986; Causse, Dehais
& Pastor, 2010). However, while lower accident involvement has been linked to
higher experience, one study noted that while highly experienced pilots were less
likely to be involved in an accident, they were more likely to be involved in a fatal
accident (Bazargan & Guzhva, 2011).

There is an often-cited belief in the aviation industry, particularly amongst pilots, that
low-hour or less experienced pilots are more ‘unsafe’ and more likely to be involved
in an accident than their more experienced peers. This concept of a special
vulnerability of low hour pilots in general aviation, (those with around 100-200 and
100-300 hours total) was studied, with no evidence found of an increased risk of
accident involvement as a function of low total flight hours (O’Hare & Chalmers,
1999). While this appears to go against the findings of accident involvement being
higher for low-hour pilots, it is worth noting that low hour in the previous studies
would have included this cohort, but also those above 300 hours. It is possible that
the low hour pilots are not less safe when around the 100-300 hour range, however
as they progress beyond this their performance may become more variable in
certain scenarios.

The inherent assumption in measuring expertise as a function of time in a system is

the overemphasis of exposure in this equation. In some industries, the synonymous
use of experience and exposure may be quite valid – for instance in fire-fighting,
where it is reasonable that a fire-fighter who has been in the role for a number of

12
years will have been exposed to a number of different scenarios to develop their
experience upon. In an industry such as aviation, which relies heavily on flight hours
as a determinant of both skill and experience, it is by no means certain that the total
flight hours accumulated by a pilot have allowed them to develop a level of expertise
in the role (Hunter, 2001). In relation to decision making in particular, it has been
shown that flight hours accumulated within the preceding 3 months was more
significantly correlated with positive performance, rather than total flight hours,
which is often assumed to be the more important variable (Groff & Price, 2006;
Wiegmann, Goh & O’Hare, 2002; Wiggins & Bollwerk, 2006).

Given the importance of decision making in aviation, and particularly as a subset of

expertise, various studies have examined how expert pilots’ decision making differs
to that of novice or intermediate experienced pilots. The patterns of cue utilisation
and the time taken to make decisions differed between the groups. In general, more
experienced pilots tended to make more appropriate decisions more quickly,
especially when information was ambiguous or low in diagnosticity and especially if
it was task relevant experience (Schriver et al., 2008; Wiggins et al., 2002; Wiggins
& Bollwerk, 2006; Wiggins & O’Hare, 1995; Wiggins & O’Hare, 2003).

Interestingly, a study (McKinney & Davis, 2003) examining pilot decision making in
the context of deliberate practice and its effect on decision making in a crisis,
highlighted a positive effect of previous aircraft specific hours and a positive
influence of total hours on decision making when dealing with previously practiced
malfunctions. The pilots studied were all military pilots and the practiced malfunction
was one they had been previously exposed to in flight or in simulators. However, the
authors also introduced a malfunction that the pilots could not possibly have trained
for previously. In this scenario, and in contrast to the above findings, there was no
relationship between deliberate practice and decision making performance for hours
in the aircraft type or total hours. This went against the hypothesis, which was that
performance would improve due to deliberate practice in general as the flight itself
should have been less taxing in other ways, e.g. physical flying, communication and
so on. When this difference was explored further, it was noted that more
assessment errors were common with low-hour pilots and action phase errors more
common for more experienced pilots (McKinney & Davis, 2003). In this sense,
expertise did not help prevent errors but it did change the type of error the pilot was
more likely to make. This finding is interesting in the context of the concern about

13
low-hour pilots as it would seem that it is not error rate but error type that changes
as a function of experience.

It would appear that the idea of deliberate practice does work to an extent, but the
belief that a pilot’s total hours can be relied on as a measure of skill, including
decision making, in an emergency situation is not necessary correct, especially if
that emergency has never been considered or practiced by the pilot previously.
Within the area of expertise research and human factors, the idea of using flight
hours as a measure of skill has been discussed and cautioned against as “...relying
on someone’s title or years of experience can be inadequate” and that the
performance of an expert “...depends on acquiring domain-specific knowledge
rather than on a general ability advantage” (Charness & Tuffiash, 2008). Yet this is
precisely what the majority of airlines do on a regular basis when they require a
minimum amount of flight hours on behalf of a pilot applicant. Often these hour
requirements will specify a breakdown of hours in certain areas, however there is
still an assumption that the total hours acquired will have enabled the pilot to
develop the appropriate skills to operate safely.

The majority of studies of expertise and pilot performance in aviation have centred
upon the general aviation sector of industry. With regard to the increased risk of an
accident within general aviation, it was noted that this risk was “partly reflecting
inadequate training and flight experience” (Li & Baker, 2007). One study which
examined professional pilots found that less experienced pilots (defined as having
less than 5,000 hours) were twice as likely to be involved in an accident as their
more experienced peers. However, it should be noted that these pilots were not
flying for a major airline, they were flying for the commuter and air-taxi sector of
aviation, a sector which carries a 10-45 times higher crash rate per pilot than the
major airlines (Li et al., 2003). As such, the findings from this study, while
informative, do not relate to the ultra-safe sector of high-capacity regular public
transport (RPT) airline flying.

A study which examined accident causation as a function of pilot age, gender and
experience, defined air transport pilots with 300-2000 hours of total time as “...newly
licensed airline transport pilots”; 2000-5000 hours of total time as “...moderately
experienced airline transport pilots”; and 5000 hours or more total time as “the most
experienced pilots” (Bazargan & Guzhva, 2011). The results of this study indicated
that “the ratio of pilot error caused to total accidents steadily declines with pilot

14
experience” with significant differences between experience levels, showing low
experience pilots were more likely to make errors that resulted in an accident
although more experienced pilots were more likely to be involved in a fatal accident
(Bazargan & Guzhva, 2011). It should be noted that their sample also included pilots
with less than 100 total hours and 100-300 total hours, which by definition does not
cover the air transport sector and would be representative of private and commercial
pilots only and it was these groups which had the highest error rate.

Within the area of expertise, little research has been done to examine high capacity
RPT airline pilot performance as a function of total flight hours. Accordingly, the
current beliefs and views of many in the aviation industry relating to the safety of low
hour pilots, has not been appropriately tested or examined in depth.

Pilot skill
The overall skill of a pilot in successfully managing a flight is made up of various
elements including technical, non-technical and psychomotor skills. It is the
combination of the required hand-eye coordination, muscle movements, reactions
and other psychomotor skills along with the technical expertise and application of
technical knowledge with the non-technical aspects such as decision making,
leadership and communication which contribute to producing effective pilot
performance. One airline assesses six main areas during pilot selection: “technical
skill and aptitude; judgement and problem solving; communications; social
relationships, personality and compatibility with [airline]; leadership/subordinate
style; and motivation and ambition” (Harris, 2011, p121). It is therefore important to
ensure that any assessment of pilot skill encompasses more than just their technical
or psychomotor skills but also includes non-technical skills.

Technical skills
The technical skills of a pilot are centred upon their ability to understand and
implement technical knowledge relating to the aircraft and environment and use this
to successfully conduct a flight. Pilots are regularly assessed during flight tests and
simulator recurrent training for their ability to conduct various technical tasks such
as flying an approach and landing, adherence to standard operating procedures,
handling (that is, inputs to the automation and manual handling skills), flight

15
preparation and so on. The technical aspects of pilot skill are also tested through
various mandatory regulatory required manoeuvres, such as successfully recovering
from a stall, or engine failure during cruise or conducting an emergency descent.

A pilot’s technical skill involves being able to interact with the aircraft through their
technical knowledge to obtain a desired outcome, such as inputting the correct
control commands to comply with a heading or altitude restriction issued by Air
Traffic Control. These skills form the basis of pilot competency as without the
technical knowledge and skill, a pilot cannot interact effectively or safely with the
aircraft. This is recognised within pilot training, with the Civil Aviation Safety
Authority’s (CASA) Day Visual Flight Rules (VFR) syllabus for pilot training ensuring
the role of technical skills such as ‘operate radio’ and ‘manage fuel’ are embedded
early in the training regime and assessed as contributing to a pilot’s overall
competence within this role (CASA, 2008).

Psychomotor skills
In addition to the cognitive skills needed to fly a plane, pilots must also possess the
necessary psychomotor skills to manipulate the controls to achieve a desired
outcome (e.g. maintain an altitude, airspeed or conduct a balanced turn, which
involves the use of rudder pedals as well as manipulation of the control yoke/stick).
Psychomotor skills, due to their nature, tend to become automatic fairly quickly,
forming the basis for further skill acquisition in a task that may demand a higher
cognitive oversight (Ackerman, 1992). As such, once pilots acquire the necessary
psychomotor skills required to manipulate the aircraft safely, these skills become
predominately automatic, with very little conscious oversight required from the pilot.
While learning these skills will place a cognitive demand or load on pilots, once they
are learned, and in some cases overlearned in order to ensure the reaction is as
fast as possible in an emergency event (such as an engine failure after takeoff), the
associated mental load in producing a stable performance is reduced.

The validity of selection tests to predict the success of pilot performance in an airline
or military environment have always incorporated psychomotor testing as well as
personality assessment, intelligence tests and evaluation of spatial abilities and
cognition. To this end, many pilot selection processes rely on a battery of selection
tests which include testing candidates against various psychomotor metrics in order
to test for speed control, rate of change control and prediction, hand coordination,

16
direction control, drift correction, tracking and so on (Fleishman & Hempel Jr, 1956;
Harris, 2011). The predictive validity of selection testing has been further extended
with consideration of the required role within a dynamic environment, including the
effect of psychomotor skill acquisition in these environments (Ackerman, 1992).
Such research has highlighted the relationship between various abilities (including
psychomotor skills) as predictors of performance at various stages of skill
acquisition. In addition, automated versions of pre-world war two testing metrics for
psychomotor skills have been shown to be a superior predictor of performance than
the use of flight grades in flight training and number of flight hours in the military
environment (Griffin & Koonce, 1996).

Non-technical skills
Non-technical skills are “the cognitive, social and personal resource skills that
complement technical skills and contribute to safe and efficient task performance”
(Flin, O’Connor & Crichton, 2008, p1). The main focus of non-technical skills have
been focused upon situational awareness; decision making; communication;
teamwork; leadership; managing stress and fatigue (Flin, O’Connor & Crichton,
2008). These skills are emphasised as part of a larger skill set the individual needs
to perform a task safely.

Within aviation, the recognition for the need to focus on non-technical skills in
addition to the technical and psychomotor skills pilots possessed came from
accidents such as the accident at Tenerife, Canary Islands, involving two Boeing
747s in 1977. This accident was not the result of any kind of technical fault with the
aircraft involved, or technical skills of the pilots. Instead, it was found that a
breakdown in communication, team coordination, decision making, flight crew
fatigue and problems with leadership were all contributing factors to the accident
(Flin, O’Connor & Crichton, 2008). There was a realisation within the aviation
industry that they had been equipping their pilots with the technical skills to fly,
without giving them the same support for the non-technical skills and Tenerife, along
with other accidents, highlighted the need to further develop this support. Training
courses in cockpit resource management, which evolved to crew resource
management, were as a direct result of this realisation and today are commonplace
within airlines. Recently, in 2009, CASA mandated non-technical skills training

17
within airlines as a regulatory requirement, which they can be audited against. This
gives further recognition to the importance of this area.

As such, for an airline pilot, the non-technical skills listed above are as important as
the technical and psychomotor skills required to manage a flight. As the human is
generally the last defence in the system, it is recognised that non-technical skills can
assist individuals to cope with and respond effectively and safely to a non-normal or
off-nominal event (Flin, O’Connor & Crichton, 2008; Reason, 2008; Harris, 2011).

Aeronautical decision-making
Decision making is a well-studied area of psychology and human factors and is
often examined as a function of individual expertise (Shanteau, 1992; Klein, 1997;
Fisher et al., 2002; Herbig & Glockner, 2009). The application of decision making to
the aviation industry is known as aeronautical decision making and has focused on
pilots in particular. They have been studied extensively by researchers in an attempt
to understand and document successful decision making in dynamic, uncertain
environments, of which aviation is an example (Wiggins & O’Hare, 1995; Wiggins et
al., 2002; Wiegmann, Goh & O’Hare, 2002; Schriver et al., 2008; Kennedy, Taylor,
Reade & Yesavage, 2010).

Various models have been proposed to help document and understand successful
decision making, with some focusing on the more practical issues of real world
decision making (such as recognition-primed decision making and fast and frugal
decision making) and developing a model from this. Others have focused on the
analytical theory underpinning how individuals perform in laboratory settings, with
the results of that theory then applied to a more practical setting (Klein, Calderwood
& MacGregor, 1989; Klein, 1993; Gigerenzer, 2008; Andersson et al., 2003; Bryant,
2002).

Naturalistic Decision making

Naturalistic Decision Making (NDM) has been defined as “the study of how people
use their experience to make decisions in field settings” (Klein, 1997). NDM
recognises that people are required to make decisions in the ‘real world’, which
generally involves time pressure, high-stakes tasks with incomplete information and
uncertainty in a complex situation (Klein, 1997). The NDM framework is context

18
dependent and situation driven, making it ideal for training and evaluating novices in
decision making within certain scenarios. Unlike normative, analytical approaches to
decision making (which are generic), NDM allows the context of the domain under
training to be considered, thereby equipping novices with the strategies they need to
work in their chosen domain (Klein, 1997).

NDM has been acknowledged as an exception to the general problem of explaining

expertise in decision making and how to develop such expertise. However issues
such as “...what experts do in new situations” and “...how far experience can be
generalized” have not been adequately addressed, even within NDM (Herbig &
Glockner, 2009).

Recognition-Primed Decision Making

The Recognition-Primed Decision model was the result of work by various
researchers into naturalistic decision making in real world settings (Klein et al.,
1989, Klein, 1993). The research began with fire commanders regarding emergency
events, which called for rapid responses in a complex environment.

The RPD model has three levels; the first a simple match of recognising a situation
and being able to react immediately with an obvious solution. The second level
requires a course of action to be developed with some conscious evaluation of the
situation and the third is a complex strategy which involves recognising a problem
with a solution that requires modification in order to work, or rejection in favour of
the next suitable intervention (Klein, 1993). It is noted, however, that the use of the
term recognition-primed is deliberate, that is, the decision is primed by how the
situation is recognised, but does not rely solely on that recognition (Klein, 1993).

One of the main findings of the studies into RPD was that “experts rarely report
considering more than one option” (Klein, et al., 1989). The research highlighted the
differences between traditional decision making theories of using decision trees to
“make choices” and “consider alternatives” and what the fire commanders were
doing, which according to them was “acting and reacting on the basis of prior
experience…generating, monitoring, and modifying plans to meet the needs of the
situations” (Klein, 1993, p139). The benefit of this approach to real world decision
making was highlighted by the fact that the focus of the fire commanders was on

19
responding quickly with a workable response, not relying on optimising choices,
which could result in them losing control of the situation all together (Klein, 1993).

While RPD centres upon making decisions in a complex real world situation, often
under time pressure and with incomplete information, it is also recognised that
traditional methods of decision making, which focus on analytical approaches to
considering every problem, are equally valid in certain situations (Bryant, Webb &
McCann, 2003; Klein, 1993). Specifically, “when planning, decision makers are
more likely to have the time and access to resources” which will help them to
consider all options to arrive at the best solutions, not just to satisfy an immediate
goal (Bryant et al., 2003). The benefit of using intuitive methods such as NDM was
recognised as important in tactical situations “given the advantages of speed and
information processing efficiency conveyed by these strategies” (Bryant et al.,
2003).

Fast and Frugal Heuristics

Heuristics, or mental shortcuts, are efficient and effective decision making strategies
because they enable people to make fast judgements that predominately arrive at
accurate conclusions in spite of various constraints such as information processing
capacity, limited knowledge and ambiguous environments (Gigerenzer, 2008;
Andersson et al., 2003).

These ‘fast and frugal’ heuristics need little time, knowledge and cognition and
different strategies are often used depending on individual factors (including
experience) and cues from the environment (e.g. ambiguous information)
(Gigerenzer, 2008). When individuals state they are using a rule of thumb to make
decisions, this is known as the recognition heuristic. The recognition heuristic is
what is used by a non-expert in choosing between two products or things, and
works only when they have heard of one and not the other. For instance, German
students participating in a survey correctly identified Detroit as being larger than
Milwaukee; however a comparable number of American students only got it right
40% of the time. This is because German students recognised the name Detroit but
not Milwaukee and American students knew both, which meant they could not rely
on the recognition heuristic. While this heuristic does have limitations, it is useful
when there is a degree of uncertainty and only one recognisable option for the
individual (Gigerenzer, 2008).
20
Another heuristic, labelled ‘Take the Best’ is a strategy for rapid decision making
which centres upon the individual gathering enough information to make the
decision and no more. This heuristic has been used in the form of a fast and frugal
decision tree in a hospital in Michigan, USA to decide whether or not patients
presenting with chest pain should be admitted to coronary care or a regular nursing
bed. The decision tree has three levels and it is used by entering at the top. If the
first level provides the answer, the doctor stops there. If it does not, they continue to
the next level where the same thing happens. However there are only three levels
and sufficient information at each level to stop after the first step if the criterion is
met (Gigerenzer, 2008, p174).

NDM and Fast and Frugal

A comparison of NDM and the concept of ‘fast and frugal’ decision making (the use
of heuristics) was done with the intent of developing a framework for testing clear
and precise hypotheses to be tested and developed (Bryant, 2002). While NDM is
essentially a framework that can be used to study decision making in the real world,
it does not explain exactly how these decision making processes occur and NDM is
also indicative of expert performance in decision making (Bryant, 2002; Klein, 1997).
The fast and frugal approach however, aims “to develop models of cognition that are
simultaneously plausible on psychological and ecological grounds, as well as being
computationally specific” (Bryant, 2002). In this way, the fast and frugal heuristic
approach combines theoretical rigour as well as being indicative of real world
decision making.

Both models operate within various constraints such as ambiguity and time
pressure, with NDM focused on the actual and observable behaviours of experts in
the field. Fast and frugal emphasises formal modelling but has also been shown to
work in the real world. It does not rely on expertise for decision making and in one
particular heuristic – the recognition heuristic, expertise is not advantageous as the
benefit of applying this rule is that the novice is choosing the one scenario they
recognise out of the two on offer.

Overall the heuristic is predominately advantageous, however, on rare occasions it

can work against an individual. It is generally through inadvertently applying the
heuristic as a tool that can result in a less than optimal outcome. Despite this,
individuals generally get feedback about the efficacy of their decision and can adjust
21
it depending on the circumstance. Therefore, the benefit of RPD, NDM and fast and
frugal are centred on efficiency gains in dynamic situations with the chance that this
will not work in all scenarios covered by the adaptation of individuals to recover the
situation.

One limitation of NDM is that “no one has developed a concept of rationality from
the NDM perspective. Researchers have assumed that the ways experts solve
problems defines the “right” way” (Bryant, 2002). The main difference between the
two is that “NDM models are derived from expert behaviour but fast and frugal
heuristics from an analysis of the task or problem in its environmental context”
(Bryant, 2002). The benefit appears to be that while NDM works particularly well
with experts and indeed relies on expertise to a large extent, fast and frugal
heuristics can also rely on these ‘shortcuts’ of expertise but they by no means
depend on, or demand them. This would suggest that while expertise could be
beneficial in fast and frugal decision making, it is by no means a pre-requisite and in
some cases would be of little use.

However the underlying similarities, as well as the differences, between the two
approaches mean that they can complement each other in real world settings, with a
framework proposed to encapsulate the benefits of both to aid decision making in a
military environment (Bryant, 2002).

Workload
Workload is defined as the information processing ‘cost’ of performing a certain task
(Harris, 2011, p40). High workload has been shown to increase error rates in
aviation (Harris, 2011, p39). Workload will vary between individuals due, in part, to
the subjective nature of what is a high workload, or high task demands and also as
a function of individual differences. Other aspects of non-technical skills can be
affected by an individual’s workload, such as communication, situational awareness
and decision making (Flin, O’Connor & Crichton, 2008).

Research comparing the performance of experts and novices in a simulated flight

found that novices experienced a higher workload than experts when measured
through EEG and ECG changes. Specifically, data processing and motor
preparation appeared to be greater in novices, suggesting higher activity for novices
in the sensorimotor and visuomotor areas. It was also noted that the level of mental

22
activation was lower for experts, again indicative of a lower mental workload
(Dussault, Jouanin, Philippe & Guezennec, 2005). An examination of scanning
techniques for pilots also noted an effect of expertise, with experts demonstrating a
superior scanning pattern to novice pilots, which included more frequent fixations on
instruments but shorter dwell times on these instruments, highlighting an efficiency
of expert performance in scanning (Kasarskis, Stehwien, Hickox, Aretz & Wickens,
2001). This led to better landings by experts, which may be in part due to the more
effective scanning techniques they used. This finding is consistent with the cue
utilisation research in decision making regarding expert versus novice performance
(Wiggins & O’Hare, 2003).

The benefit of expertise on workload was also found in a medical setting, and
explained as an automaticity of motor learning on behalf of the expert surgeons
performing a laparoscopic task. Due to the automaticity of the actions associated
with the task, it is not surprising that mental workload would decrease as oversight
is not as high as it would be for a novice in the knowledge-based learning
environment (Zheng, Cassera, Martinec, Spaun & Swanstrom, 2010; Adams &
Ericsson, 2000; Rasmussen, 1983). The development of automaticity in experts
allows allocation of mental resources to areas such as situational awareness and
secondary tasks which will have a positive effect on workload management (Adams
& Ericsson, 2000; Zheng, Cassera, Martinec, Spaun & Swanstrom, 2010).

Situational awareness
An often cited definition of situational awareness is “…the perception of the
elements in the environment within a volume of time and space, the comprehension
of their meaning, and the projection of their status in the near future” (Endsley,
1995). Another definition is “the cognitive processes for building and maintaining
awareness of a workplace situation or event” (Flin, O’Connor & Crichton, 2008).
Irrespective of the definition used, situational awareness is ultimately concerned
with an individual’s understanding and interaction with their environment.

Situational awareness is typically presented as three hierarchical phases which are:

Level 1 – perception of the environment, Level 2 – Comprehension of the situation
and Level 3 - Projection of future status (Endsley, 1995). This hierarchy is
concerned with developing situational awareness as a construct and is useful in
identifying the point at which an individual becomes unaware of the actual situation.

23
For instance, it may be a misinterpretation of the environment, which would be seen
as a failure within level 1, a misunderstanding of the current situation, which occurs
within level 2, or an inability to understand how the current situation will either
develop into the future, or how it should develop in the future, which is a limitation
within level 3 (Endsley, 1995).

The benefit of having appropriate and accurate situational awareness for individuals
working within safety critical systems is obvious. Within aviation in particular, there
are many examples of pilots having insufficient situational awareness, irrespective
of the level, to the detriment of the safety of the flight. The same is true of surgeons
and anaesthetists, firefighters and other safety critical personnel (Flin, O’Connor &
Crichton, 2008). While it is acknowledged that situational awareness is necessary
for individuals to ensure the safe operating of a system, it is also worth noting that
labelling an accident or incident as being due to ‘a loss of situational awareness’ is
not particularly useful when seeking to improve a system. Instead, the contributing
factors which adversely affected an individual’s situational awareness should be
examined and explained.

Situational awareness can be affected by a number of other constructs such as

attention, working memory, workload, fatigue and stress (Endsley, 1995; Flin,
O’Connor & Crichton, 2008). Each of these areas can have a detrimental effect on
an individual’s ability to demonstrate effective situational awareness. As situational
awareness is reliant upon accurate perception, comprehension and projection of a
scenario, it is also susceptible to limitations within working memory, which often
dictates the extent to which an individual has the capacity to attend sufficiently to
their environment. Other factors such as distraction and interruptions will adversely
affect working memory, which in turn can impact on an individual’s ability to
accurately perceive or comprehend the situation. It is for this reason that distractions
and interruptions have been highlighted as a threat to safety, particularly in aviation,
with investigations highlighting how distraction and interruption at the pre-flight stage
resulted in errors being made in critical systems which then adversely affected the
aircraft’s takeoff performance (ATSB, 2011a; Loukopoulos, Dismukes & Barshi,
2003; Flin, O’Connor & Crichton, 2008; Trafton & Monk, 2008).

The workload associated with the novice interpreting cues or expert facing novel
problems will also adversely affect situational awareness. However, it has been
hypothesised that situational awareness and workload can diverge for various

24
reasons and it’s only “if the demands associated with this task and others exceed
the operator’s limited capacity,… will a decrement in SA be expected” (Endsley,
1995). This is an important distinction with regard to ensuring that workload is not
seen as a constant detriment to situational awareness, but rather the fact that
situational awareness can degrade if an individual’s workload is excessive. The
main difference in situational awareness as a function of expertise has been
identified as how the picture is put together by the individual. That is, there may not
be a difference in various aspects such as cues used, knowledge, imagery and so
on, but it is the way in which these are combined and decisions are made from this
combination that facilitates improved performance in experts (Randel, Pugh & Reed,
1996).

Both fatigue and stress will adversely affect situational awareness through their
impact upon information processing, attention and judgement. In particular, the
largest effect of fatigue and stress on situational awareness will be the narrowing or
reduction in attention that corresponds with human performance under these
conditions. It is well documented that humans under stress will narrow their
attention, which will make accurate perception of the environment difficult (Endsley,
1995). Similarly, fatigue will also act as a detriment upon both perception and
working memory, which will affect an individual’s situational awareness (Flin,
O’Connor & Crichton, 2008).

Effective situational awareness can also be a function of individual expertise, with

experts regularly outperforming novices, particularly in comprehending the
relevance of cues in the environment and what that means for the task being
undertaken (levels 2 & 3). As expertise is, in part, a function of exposure to various
scenarios and their related cues, novices will have fewer mental models available to
them and will have a corresponding rise in the effort required to comprehend the
environmental cues and accurately assess the situation (Endsley, 1995; Flin,
O’Connor & Crichton, 2008). The same problem however, is faced by experts when
they encounter novel situations, which require them to interpret cues accurately,
which in turn can result in a high demand on their working memory (Flin, O’Connor
& Crichton, 2008).

Research into training situational awareness has found some success, particularly
through the use of simulation to allow for various distracting and high workload tasks
to be presented and followed up by stopping the simulation to allow the student to

25
review the situation with the aim of developing and enhancing their situational
awareness (Flin, O’Connor & Crichton, 2008). There have also been numerous
attempts to measure and assess situational awareness in practical settings, with
varied success (Jones & Endsley, 2004; Flin, O’Connor & Crichton, 2008; Harris,
2011). The difficulty in assessing situational awareness is that it is a cognitive skill
and as such, is not visible to an assessor in the same way that a physical action
within the cockpit would be. However, under various assessment systems, such as
one used with anaesthetists, situational awareness is inferred from the actions of
the individuals which are observable, such as “conducts frequent scan of the
environment; increases frequency of monitoring in response to patient condition;
and keeps ahead of the situation by giving fluids/drugs” (Flin, O’Connor & Crichton,
2008, p35).

Memory
The role of memory in expert performance is dependent on a number of factors.
One advantage of experts in relation to both situational awareness and decision
making is that their expertise will reduce the load on working memory in operational
settings (Flin, O’Connor & Crichton, 2008). This reduction in load is associated with
the automaticity of performance in known tasks that is gained by individuals as they
progress to skill-based performance (Rasmussen, 1983).

The performance of experts in an experiment conducted in electronic warfare

showed that experts displayed a greater ability to remember and subsequently recall
the location of potential threats in the environment during the simulation (Randel,
Pugh & Reed, 1996). The difference in memory performance was explained as a
function of the importance of the position of hostile ships to the tactical situation and
what response would be required, which allowed the expert to construct a memory
structure to allow accurate recall of the threat (Randel, Pugh & Reed, 1996). This
then creates memory structures which can be used in the future to deal with similar
situations, thereby emphasising the benefit of expert performance in the use of both
working and long term memory (Chase & Ericsson, 1981).

The importance of memory and memory structure in accurate recall in expert

performance underlines the necessity of training in equipping novices with the
necessary information to achieve expert performance (Klein, 1997). This is an
important consideration for pilot training innovations as it emphasises the need for

26
training to be contextually relevant and allow pilots to create and utilise memory
structures to aid future performance.

Summary
Research into expertise has shown a consistent difference in performance between
novices and experts in both laboratory and real world settings. Of note, many
aviation research studies have demonstrated a difference in the decision making
performance and capabilities of experts compared to novices. The research shows
that experts often use heuristics and spend less time examining the options,
choosing instead to stop at, and therefore select, the first appropriate strategy to suit
the situation. Novices, in comparison, tend to evaluate all of the options to ensure
their selection is the most valid for the situation. This is both time consuming and
potentially dangerous in a field such as aviation, where due to the demands of bad
weather or an aircraft malfunction, the time available may be limited. It may be
possible to use tools such as NDM and RPD to explore how experts make decisions
with the view of educating novices in what cues are important. This mechanism to
enhance experience may be of use to airlines with regard to cadet pilot training.
However, further consideration of this is outside the scope of this chapter.

Additionally, the performance of experts in non-technical skills such as situational

awareness has been shown to exceed that of novices, typically due to the use of
memory and the resultant saving in workload that this affords the expert. That is,
experts have built up schema for various automatized tasks, which allows them the
cognitive capacity to attend to other stimuli and tasks which may be relevant. It is in
this way that the areas of workload, situational awareness, memory, automaticity
and skill link together to enhance expert performance.

It has been widely recognised that pilot performance relies not only on the relevant
technical and psychomotor skills, but also on the non-technical skills to support
these. Various accidents in aviation have demonstrated both the need for, and
importance of, non-technical skills in safe operations, including recovery from non-
normal situations (ATSB, 2011a, 2011b).

With regard to accident risk, the research appears to support, at least in part, the
assertion that experience offers a protective factor in relation to accident risk.
However some studies did show that experts still made errors (albeit of a different

27
type to novices) and the likelihood of experts being involved in a fatal accident was
increased in some sectors of aviation (McKinney & Davis, 2003; Bazargan &
Guzhva, 2011). In addition, a study examining the accident risk of low-hour pilots did
not find a special hypothesised vulnerability (O’Hare & Chalmers, 1999). Of
importance however is the lack of research into the difference between low- and
high-hour pilots (or novices and experts) in an airline environment, specifically
regarding those pilots who did not transition through general aviation which is where
most of the research on expertise in aviation was conducted. The following section
will examine the areas of skill acquisition and the use of simulation in further
developing skill and expertise.

28
1.4 Learning and skill acquisition

Principles of adult learning

Research into adult learning has highlighted the importance of using techniques and
methods for supporting the unique characteristics that adults have as learners,
including the importance of understanding and addressing their needs (Leagans,
1964; Knowles, 1984). With the advent of the concept of andragogy “the art and
science of helping adults learn” as opposed to the previous focus on pedagogy “the
art and science of teaching children” (Knowles, 1984, p6), adult learning evolved in
its own right. There is an important distinction between teaching children and
helping adults learn, which had been lacking in adult education prior to the concept
of andragogy. The first suggests a very directive and focused intervention, the
second an encouraging and supported approach to learning. After application of the
term andragogy spread, it was noted that in various situations andragogical
methods assisted children in learning and in some specific cases, such as basic
skills training, adult learning was best suited to a pedagogical style (Knowles, 1984).

The clearest distinction of pedagogy and andragogy lies in the underlying

assumptions and these also serve to illustrate how adult learning can encompass
both models. Under a pedagogy model, the assumptions of the individual are that
they are ‘directed to learn’; they often have little experience to use for learning; they
are ‘told when to learn’; progression through a course is sequential; and motivation
is primarily external. It is easy to see the applicability of this model to school children
but it is equally applicable to workers being sent on training courses by their
employer. In contrast, under the andragogy model, the assumptions of the individual
include that they are a ‘self-directed learner’; they have a great wealth of experience
to use as a resource in learning; they are ready to learn; they identify a need to
learn, i.e. how to conduct a task or solve a problem; and they are often internally
motivated to learn.

One of the identified issues in pilot training is the transition from pedagogy to
andragogy, especially when young pilots enter a university aviation degree straight
from school, where the model is one of pedagogy (Clark III, 2001). This can lead to
difficulties in both learning and teaching as the student pilots must adapt to
becoming self-motivated and forget the idea that they are being told what to think
and do, something that older pilots have less trouble with (Clark III, 2001). In

29
addition, the cognitive learning bias of the student may also make progression
through a university aviation course difficult, and while this is not something the
student can control, instructors should be aware of the different learning styles of
students and adapt as necessary (Quilty, 1996) This highlights the importance of
ensuring that any level of instruction aimed at student pilots is designed to take into
account various approaches, learning styles and recognises the difficulty some
students will have in transitioning from a ‘taught’ mind-set to one of ‘learning’ (Quilty,
1996).

An important note is that the experience of adults can sometimes work against the
learning process by providing “habitual ways of thinking and acting, preconceptions
about reality, prejudices and defensiveness about their past ways of thinking and
doing” (Knowles, 1984, p.10), a finding that has been echoed in pilot training
experiences (Clark III, 2001). The concept of previous experience providing a
negative effect on performance is an often cited reason as to why airlines prefer to
employ and train cadet pilots rather than employing direct entry pilots. While cadets
are often ab initio pilots and therefore have no existing skill or experience in
aviation, they are also unlikely to have any ‘bad habits’ or responses that are not
suited to an airline environment. That is, pilots who have operated single pilot
operations in the general aviation sector will be more experienced in the individual
single pilot skill of flying, but may take longer or have more trouble transitioning to a
team role in a highly regimented and controlled environment. In addition, their
previous experience can provide negative transfer by way of increasing the chances
of rule-based errors, also known as “strong but wrong” errors, where the response is
incorrect but the programming to conduct the response has been embedded
through repetition and is hard to stop or correct (Reason, 1991 p75).

Situated learning
Situated learning focuses on recognising the role of culture and context on learning.
In particular, it highlights the need to provide students with the contextually relevant
cues and information they will use as practitioners rather than the neat, controlled
classroom examples often used to convey information and learning in the traditional
sense. The theory is that this allows students to develop the knowledge they need to
perform the task they are learning about, rather than having the theory they need

30
but a lack of recognition of how to put it into practice in the real world (Seely Brown,
Collins & Duguid, 1989; Lave, 1991; Stein, 1998).

Situated learning has been likened to undertaking a ‘cognitive apprenticeship’

(Seely Brown et.al., 1989) in which the situated nature of knowledge is used to
ensure competent performance, rather than the more traditional approach of
providing information out of context and hoping learning and knowledge will result.
The analogy of knowledge as a tool is used, with acknowledgement that the
acquisition of a tool does not ensure the individual can use it and it is within this
context that the limitations of merely providing information to students, devoid of the
necessary context in which their actions will be carried out, becomes obvious. The
role of cognition, coupled with the cultural and social surroundings, presents a
powerful suggestion that learning within a ‘community of practice’ without the formal
learning structures relied upon in schools, can produce a master level of skill as a
student journeys from a ‘newcomer’ to an ‘old-timer’ (Lave, 1991).

It should be noted however, that some of the more extravagant claims of situated
learning have been challenged. Ideas such as actions being linked to the situation
they occur in, knowledge not transferring between tasks, training by abstraction
being less useful and the concept that instruction must occur in a complex, social
environment for it to be valid (Anderson, Reder & Simon, 1996). Rather, it has been
argued that elements of these ideas, seen as central to situated learning, may
indeed be beneficial to learning and knowledge acquisition, the end goal of which is
often task mastery, or at the very least, a level of skill in the required tasks. That is
not to say that situated learning should be the sole approach, to the detriment of
educational and psychological advances in the area of learning, but rather that the
more robust and useful elements should be incorporated into learning scenarios for
students. For example, it is important that an understanding of the learner’s end role
is incorporated into learning design, so that the knowledge which needs to transfer
across tasks can do so. To this end, it is important to ensure that classroom
instruction stays relevant to the end task but it should also be acknowledged that
students will need to build up their skills (e.g. through part-task training) before
combining them in a real world setting (Anderson et al., 1996).

31
As such, ensuring that pilots are taught the underlying (but useful and necessary)
theory in the classroom and are allowed to practice these skills in part-task trainers
or simulators, will aid their comprehension of the theory. But to say that the only way
they can learn the role is to be immersed in the whole concept from day one, to
learn directly from someone who is experienced, seems inefficient and could result
in individuals not possessing the requisite skills needed to perform safely.

Skill acquisition
Individual skill acquisition is an important area for consideration in any training
system, but it is particularly important when establishing a competency-based
training system and much work has been done in studying the area of skill, ability
and cognition (Fleishman & Bartlett, 1969; Fleishman & Mumford, 1989; Ackerman
& Cianciolo, 2000; Ackerman & Cianciolo, 2002; Ackerman, 1988). In particular, skill
acquisition studies have identified that “particular abilities related to skill acquisition
change as practice on the task continues” and “some abilities accounted for an
increasing portion of the variance as learning continued, whereas other abilities
accounted for a decreasing portion of the variance” (Fleishman & Mumford, 1989).
The learning benefits of practicing a task until automaticity were also highlighted,
particularly the obvious benefit of effort and attention decreasing as skill increases
(Neves & Anderson, 1981; Shiffrin & Dumais, 1981). These findings highlight the
need to understand individual skill acquisition rates when designing competency-
based training. This becomes especially relevant in a simulated environment and
highlights the need for a syllabus that will account for individual skill acquisition rates
during repetitive learning.

Experiments in the area of cognitive skill acquisition have often concluded that skill
acquisition is largely domain-dependent, with one experiment highlighting skill
development as “an iterative process that converts advice into plans and, ultimately,
converts these plans into behaviors” (Hayes-Roth et al., 1981, p249). In addition, it
was postulated that an individual’s ability to apply the “diagnosis and learning rules”
will depend on their familiarity and level of expertise in the particular domain (Hayes-
Roth, Klahr & Mostow, 1981, p249). That is, while there are general skills individuals
possess to address domain specific issues it is their knowledge of the domain that
will guide appropriate response and “affect the difficulty of learning in these
domains” (Hayes-Roth et al., 1981, p252). Expertise is considered to occur through

32
“the use of knowledge and not by analysis of knowledge” (Neves & Anderson, 1981,
p83). Domain knowledge is considered necessary for expert performance as without
this knowledge base, “task performance by a novice is poor or non-existent” (Chase
& Ericsson, 1981, p141).

Looking at this process from another direction is the application of schemata by a

student within the learning environment. That is, when students apply a schemata
that does not exactly fit to a problem it is possible to infer what the original source
schemata was. This is the basis for recognition-primed decision making, the rapid
application of the known knowledge, which does not entirely fit, to the new situation.
This is fundamental to what we claim are the differences between novices and
experts (the ability to apply solutions in ambiguous environments) and yet there are
some principles that have been suggested to provide students with a conceptual
model about a domain in which they are being taught. These principles include: “It
should be based on a domain with which the student is very knowledgeable…”, “The
target domain and the source domain should differ by a minimum number of
specifiable dimensions”, and “Operations that are natural within the target domain
should also be natural within the source domain” (Rumelhart & Norman, 1981,
p358).

These concepts are built upon Thorndike’s ‘Theory of identical elements’, which
found support for the concept of transfer of training being enhanced by the
commonality of the environment (Thorndike, 1923). Essentially the learning benefit
for students will be greatest when the commonality between the tasks, environment
and domain are highest, and when the application of knowledge from one area of
the domain is effortless and predominately correct for another area of the same
domain. This is a compelling argument for the need to ensure pilot training is
tailored to the end product we want – the ability for student pilots to progress to
being able to apply knowledge about one area of an aircraft or the aviation industry
to another area and achieve a correct and useful result. In this way, training MPL
students in an airline-like environment from the start of their training (particularly the
attention given to multi-crew operations), if done correctly, would appear to provide
immense benefits.

33
While some generic abilities and knowledge will underpin skills and knowledge
within various domains, interestingly when the subject of deliberate practice was
applied to skilled memory, it was found that performance in the area studied was not
transferable to other similar areas. A study examining skilled memory in the context
of increasing the number of digits held in, and retrieved from, memory found it was
possible with up to 250 hours of practice accumulated over a two year period, to
increase memory capacity from +/- 7 digits to about 80 digits (Chase & Ericsson,
1981). It should be noted that the authors do not believe the subject’s short-term
memory abilities increased, rather it was the development of an encoding and
retrieval system that meant an effective use of intermediate knowledge states in
long-term memory. However this skill was not transferable to remembering random
consonants, meaning the skill was highly domain dependent and indicative of expert
performance in the area of remembering digits only. It is worth noting that the
authors acknowledge that “with only a couple of hundred hours of practice, SF [the
subject] would be classified as a beginner at most skills” (Chase & Ericsson, 1981, p
143). However, in this field the subject’s performance was comparable to experts in
memory retrieval.

The performance benefit of expertise in memory recall and recognition of situations

has been highlighted in many studies, including that of expert chess players and
fire-fighters, and the rapid access (generally cue driven) to memory, could account
for why expert performance appears superior to the performance of a novice,
especially once their actions are automatic, which generally occurs through practice
(Chase & Ericsson, 1981; Neves & Anderson, 1981; Jeffries et al., 1981; Langley &
Simon, 1981; Klein, 1993; Chabris & Simons, 2010; Kahneman, 2011).

Another study exploring the role of competence in the self-efficacy skills relationship
examined the domain of information technology (IT), specifically six domains within
general computer use being examined for the effect of self-efficacy on competence.
The results showed little correlation between competence and self-efficacy, but did
identify that as competence increased, so did self-efficacy, finally tapering off at the
mastery level (Downey & Zeltmann, 2009). The effect was essentially a bell curve
and prompted the authors to emphasise the importance of tailoring training, as
“simply teaching skills is not enough” (Downey & Zeltmann, 2009 p.107). The results
highlight the importance in teaching novices differently to experts, insofar as novice

34
training should be focused on building “basic skills and foundational concepts”
(p.107), whereas the higher competence groups will be affected more by
“momentary moods…” (p.107).

A practical examination of skill acquisition and transfer of training in the aviation

domain was achieved by manipulating the amount of pre-task training given to
subjects in an Air Traffic Control (ATC) training task. The results showed that
performance increased when subjects were given a longer training time; however
this advantage was attenuated with practice (Goska & Ackerman, 1996). This
difference in transfer based on practice time and ability-performance is an important
element of both competency-based training design and simulator transfer of training
considerations for MPL students.

Simulation training
Simulator training is widely used to train students prior to flight in an actual aircraft.
The need for simulation in aviation training has been highlighted over the years with
numerous accidents resulting from practising or examining non-normal or
emergency manoeuvres and situations in actual aircraft. A recent Australian
example involved an Embraer EMB-120 turboprop aircraft which was conducting an
instrument rating renewal test flight for the captain and involved an engine failure at
takeoff scenario. The recorded data from the aircraft showed that the thrust was
reduced on the left engine to flight idle instead of zero thrust, which increased the
drag and affected aircraft controllability, resulting in the aircraft impacting the ground
just past the runway, fatally injuring both pilots. In addition, this manoeuvre was
initiated at a speed that was too low to allow adequate margin for error (ATSB, 2012
p 57). Tragically, this flight was anticipated to be one of the last check flights the
operator conducted in the actual aircraft as they were planning on moving these
flights to a simulator (ATSB, 2012 p 53). Conducting these same manoeuvres in a
simulator is not only significantly safer, but can provide cost benefits as well. A
similar issue faced by the military is the constraints of training their pilots for warfare
in peacetime conditions, with similar limitations due to the danger of various
manoeuvres as well as an inability to control the environment, aircraft serviceability
and other operational demands (Bell & Waag, 1998).

35
However, despite the recognised and agreed need for simulators and simulation,
the use of simulation in aviation training has been a long discussed subject, with
debates surrounding training design, equipment fidelity and pilot acceptance
(Gerathewohl, 1969; Caro, 1973; Salas, Bowers & Rhodenizer, 1998; Alessi, 2000;
Bürki-Cohen et al., 1998). Numerous studies have examined the benefits to be
gained through transfer of training from simulation and a key point that is
consistently made in the literature is that the fidelity of the simulation training is the
most important aspect of transfer of training success. This was nicely summed up
as “it has to be emphasised that flight simulation does not provide training” (Harris,
2011, p126). While it has been noted that “Airline flight training is sometimes
considered synonymous with simulation” (Wiener, 1989), it is important to remember
that the simulator is only a tool that allows airlines to achieve a “desired result at
maximum safety, reasonable cost, and regulatory conformity”. To achieve the best
result possible in training airline pilots “Training must include a consideration of
curriculum, educational psychology, instructors, training techniques, materials,
equipment, philosophy, and policy” (Wiener, 1989).

Transfer of training from simulator to aircraft

There have been numerous studies conducted looking specifically at the transfer of
training between simulators and (often student) pilot performance in the aircraft
(Lintern, 1980; Caro, 1988; Lintern, Roscoe, Koonce & Segal, 1990; Lintern, Taylor,
Koonce & Talleur, 1993; Taylor, Lintern & Koonce, 1993; Taylor, Lintern, Koonce,
Kunde, Tschopp & Talleur, 1993; Westra & Lintern, 1995), and/or Flight Training
Devices and performance in the aircraft (Wiggins & Crognale, 2003; Wightman &
Lintern, 1985) including in the area of non-technical skills such as crew coordination
(Jentsch & Bowers, 1998; Cannon-Bowers & Salas, 1998). Studies have also
examined performance in a simulator to assess transfer of training and/or
performance from other training devices (Damos & Lintern, 1981; Lintern & Koonce,
1992; Lintern & Naikar, 1999; Lindo, Deaton, Cain & Lang, 2013).

In addition to the transfer of training skill between simulators and aircraft, the
transfer of training from computer-based training devices and simulators or aircraft
has also been extensively reviewed (Moroney, Hampton, Biers & Kirton, 1994;
Phillips, Taylor, Lintern, Hulin, Emanuel Jr & Talleur, 1995;Taylor, Lintern, Hulin,
Talleur, Emanuel & Phillips, 1997a; Taylor, Lintern, Hulin, Talleur, Emanuel &

36
Phillips, 1997b; Taylor, Talleur, Emanuel Jr, Rantanen, Bradshaw & Phillips, 2001;
McDermott, 2006). There are findings that question the need to use aircraft in
instrument proficiency checks given there was no difference in performance when
candidates did the recurrent training on a PC device or simulator or aircraft (Taylor,
Talleur, Bradshaw, Emanuel Jr, Rantanen, Hulin & Lendrum, 2003; Taylor, Talleur,
Rantanen & Emanuel Jr, 2005).The studies predominately focused on low hour
pilots and therefore these studies are still relevant today, as MPL pilots will be
transitioning to simulators with low actual aircraft time and will conduct the majority
of their training in an approved simulator. They will then move from the simulator
environment to line training and line operations in large aircraft.

There is evidence from studies that show students who have trained in the simulator
perform better in the aircraft than a control group (with no simulator training but the
same amount of flight hours). Lintern (1980), showed that students who learnt a task
in the simulator performed to a higher standard in landing practice in the aircraft
than students who were in the control group (no simulator training). A similar
positive effect of simulation was noted in the US military involving air to air combat
training (Bell & Waag, 1998). Gopher, Weil & Bareket (1994), looked at the transfer
of training from a computer game to an aircraft and found students in the two groups
who were trained on the computer game (but using different instruction methods)
outperformed the control group on flight performance scores.

However it should be noted that a study by Schmidt & Wulf (1997) reviewed the use
of continuous concurrent feedback and found it was detrimental to performance in
most circumstances. They listed aviation studies, such as Lintern (1980), as the
“troubling exceptions to this generalization” as Lintern (1980) indicated augmented
concurrent feedback was effective (Schmidt & Wulf, 1997). It is possible that the
complexity of the landing task in the aviation studies meant the feedback was
necessary, not just useful, an important consideration when rating the effectiveness
of transfer of training studies and their applicability to the real world. It is equally
important to consider the construct of the training and whether or not it should be
part-task or full-task training depending on the required skill and progression of
learners, as well as the abilities of the individual students (Goettl & Shute, 1996).

37
There has been exploration of ab-initio student performance in technically advanced
(general aviation) aircraft (TAA) in response to suggestions that transitioning
students to a modern cockpit environment would improve their performance. This
was seen as advantageous for pilots who were about to enter an airline and who
had received their training in traditional aircraft without exposure to “multi-crew and
jet-transport flying” (Dahlstrom et al., 2006). The results of their study of the
introduction of TAA aircraft to a flying school found that the performance of students
on the TAA was as good as traditional aircraft. The authors note the main expected
benefit from these aircraft would be “training that more effectively produces
competent first officers for large jet transport aircraft” (Dahlstrom, et al., 2006).
However they also caution that the assumption that the smaller GA TAA can
simulate the environment of a larger aircraft and is therefore more relevant for the
student than a traditional aircraft is somewhat questionable and needs further
exploration.

Interestingly, simulation is being used as a tool to both develop expertise as well as

to study it (O’Hare & Batt, 1999; Salas et al., 2009). The potential benefit of
developing expertise in a simulator within the context of high capacity public
transport aviation is that it presents an opportunity to develop expert performance
without the exposure to complications of the real world. This enables expertise to be
gained faster than if pilots were left to develop it through line flying, which would not
necessarily guarantee the individual had been exposed to the ‘right’ kind of learning
opportunities to develop the required knowledge, skills and performance that
underpin expertise as the line flying environment cannot be controlled to provide a
learning outcome. To this end, the simulator can be used to both collect data on
expert performance and to teach it (O’Hare & Batt, 1999; Salas et al. 2009).

Further study into ab-initio pilot training in large jet transport aircraft simulators
would be advantageous. For example, there have been no studies concerned with
the effect of low hour pilots receiving the majority of their training time in simulators,
with little actual aircraft time (beyond a certain amount), which is the basis for the
MPL.

38
Fidelity in simulation
It has been shown that if the foundation of the training has been considered with the
objectives of the learner at the core, then the level of fidelity within the actual
simulator is less important (Gerathewohl, 1969; Caro, 1973; Salas, Bowers &
Rhodenizer, 1998; Feinstein & Cannon, 2002; Beaubien & Baker, 2004; Dorsey,
Campbell & Russell, 2009). Altering the fidelity of simulation to suit the stage of
learning has shown to be beneficial i.e. low fidelity for novices up to high fidelity for
experts (Alessi, 2000). There is a belief in the aviation industry that a simulator
needs to mimic the aircraft as closely as possible in order to provide effective
training and useable skills. This belief, however, has been shown to be incorrect
(Caro, 1973; Roscoe, 1991; Salas et al., 1998; Alessi, 2000; Beaubien & Baker,
2004; Harris, 2011). The use of a low fidelity simulator within a well-defined and
designed training system can provide benefits in the area of transfer of training, with
some studies showing a significant decrease in the amount of flight hours necessary
in the actual aircraft to obtain a certain skill after initial training in the simulator
(Caro, 1973; Lintern, Roscoe, Koonce & Segal, 1990; Lintern, Taylor, Koonce &
Talleur, 1993; Taylor, Lintern & Koonce, 1993).

Achieving learning outcomes in simulator training has more to do with instructional

design than the fidelity of the machine. Yet aviation has often focused on making the
simulator mimic the aircraft, with the inherent assumption that this is all that is
necessary to ensure skill transfer and therefore learning. This is not the case and
has led to an over-reliance on what have been labelled ‘complex’ learning
environments (e.g. advanced high fidelity simulators) at the detriment of giving the
individual control over their learning environment and room to practice (Schneider,
1985; Salas et al., 1998; Dorsey et al., 2009; Alessi, 2000; Feinstein & Cannon,
2002; Beaubien & Baker, 2004; Harris, 2011). It has also led to a reliance on hours
in the simulator and hours in the aircraft, resulting in a strengthening of the hours
required within an industry that is supposedly recognising the importance of
individual competence and the variation in hours that individual competence will
present (Roscoe, 1991). Interestingly, Caro (1988) highlighted the benefit of training
in training devices, including a generic mock up, by showing the results of an Army
intake study which compared three groups of similarly experienced pilots who were
assigned to either a low fidelity mock up, a simulator or the actual aircraft. The study
showed that on their first attempt to conduct procedures in the actual aircraft (after

39
five training trials) performance was about equal across the groups. This highlights
the limits of relying on realism as a necessary requirement to obtain transfer of
training (Caro, 1988).

It is also important when using simulation to determine the desired outcome of the
simulator session. If the use is for training, the techniques and styles will be different
to those required if it is being used for assessment. The fidelity requirement of the
simulator itself will also differ, with assessment simulation sessions requiring higher
fidelity than training sessions due to the differences in performance expected from
the individual (Caro, 1988; Alessi, 2000; Harris, 2011).
An innovative use of technology was proposed and trialled in an American aviation
college, which involved a live video feed from the simulator back to a classroom of
students and allowed for two way communication. The instructor in the simulator
asked questions of the students in the classroom, requiring them to provide input
into things like the normal engine start procedure and even to deal with non-normal
situations (Fanjoy & Young, 2001). This intervention was encouraged and accepted
by pilots and instructors as useful and has the potential to allow for more efficient
and effective use of simulation in the future (Fanjoy & Young, 2001).
In addition ensuring that emphasis is placed on learning transfer, which may be
supported by feedback and facilitation of the session, will help maximise the benefits
of simulation (Garris, Ahlers & Driskell, 2002). It is for this reason that regarding the
simulator itself as the only tool needed in airline training will not provide an effective
learning environment for the individual, nor a desirable outcome for the organisation.

It has been recognised that perfect replication of both the real world and the aircraft
handling features in a simulator is not possible, and instead the focus should be on
obtaining a high level of realism, especially with regard to the quality of the visual
images (Padmos & Milders, 1992). Indeed the advancement of technology in recent
years has allowed the visual image support for simulators to improve significantly.
Coupled with this improvement in visual cues is a longstanding and on-going debate
about the usefulness of motion in simulation (Gerathewohl, 1969; Martin & Waag,
1978; Nataupsky, Waag, Weyer, McFadden & McDowell, 1979; Bürki-Cohen, Soja
& Longridge, 1998; Harris, 2011). In particular, the question regarding the use of
motion has related to whether or not simulators need full motion in order to be
considered suitable for training and checking in the airline environment (Bürki-

40
Cohen et al., 1998). The results are mixed, with some advantages found for
improved visual cues alone and some for motion alone, but the combination of
improved visual cues and high fidelity motion has not been studied in enough depth
to change the long standing regulatory requirement for the need to have motion in
airline training and checking operations (Bürki-Cohen et al., 1998).

In addition to the debate on motion versus fixed based simulators, the level of
fidelity of simulators for certain tasks has also been the focus of much discussion
and research. It has been argued that lower levels of fidelity (appropriately
designed) can provide individual competence acquisition without the economic
penalty of having to use full motion high fidelity simulators (Caro, 1988; Dahlstrom,
Dekker & Nahlinder, 2009). This is particularly true in the case of training non-
technical skills such as teamwork, where other avenues such a low fidelity
simulation, role play and case studies may achieve the aim more effectively than
high fidelity simulators (Beaubien & Baker, 2004).

Another consideration is the fidelity of the simulator being used, and trainers should
ensure that the fidelity of the simulator should include fidelity of operations, not just
how closely the simulator matches the aircraft capabilities. One example in aviation
of the fidelity of operations in simulators is the use of Line Orientated Flight Training
(LOFT) which is centred upon the simulator session being conducted as a normal
line flight, with various failures and non-normal scenarios added to the normal
operations (Wiener, 1989) However it should be noted that even the benefits of
LOFT can become negated if the crew are constantly overloaded with failures to the
detriment of learning non-technical skills (Salas et al., 1998).

While techniques such as LOFT and Line Operational Simulations (LOS) allow the
use of realistic event sets to evaluate crew’s performance, effective use of such
techniques can be limited in that the amount of work to develop a LOS scenario is
quite high and often requires regulatory approval, including any modifications and
changes to the scenarios. As pilots become exposed to multiple LOS sessions, they
will know what to expect, thereby reducing the learning and evaluation potential of
the scenario (Bowers et al., 1997). This led to the exploration of what was termed
‘rapidly reconfigurable event-sets’ which allow airlines to create training scenarios
for pilots that ensure they are not exposed to the same failures and events multiple

41
times, with the LOS style still followed (Bowers, Jentsch, Baker, Prince & Salas,
1997). The use of event-sets, especially for training areas such as teamwork, has
been highlighted as a way to represent the complexity of the system while remaining
realistic enough to ensure applicability to the real world. This approach has been
recommended for healthcare, given the similar complexity of the healthcare system
to aviation, and the strong need for teamwork to be integrated in training from the
very beginning (Hamman, 2004).

An important component of simulator training is evaluation. It is through evaluation

that the effectiveness of the training can be ascertained and any improvements
implemented (Feinstein & Cannon, 2002). However evaluating a system is quite
complex and is subject to many influences and limitations. There is confusion in the
literature with regard to terminology associated with evaluation which is hampering
the ability to understand validation literature (Feinstein & Cannon, 2002). One such
limitation is deciding when to evaluate. Evaluating the system at the wrong time –
e.g. before the system has had time to produce a meaningful outcome, can give a
false impression of the efficacy of the training intervention. This may result in a
program being cancelled as it was deemed ineffective when in fact it was a function
of the timing of the evaluation rather than the training itself (Hays & Singer, 1989).
The timing of evaluation is not only an issue when evaluating a single training
program it can also affect the comparison of two or more training programs. For
instance, a program which is rated higher than another earlier in the delivery may
not produce the same benefit of outcome of the program which received a lower
initial rating but which has proven to produce a better product when rated later in the
delivery (Hays & Singer, 1989). Choosing when to evaluate a program is an
important consideration for the implementation of concepts such as the MPL. If the
rating of the MPL pilots is conducted too early, compared to traditionally trained
pilots, it may show a false positive or false negative, neither of which is a helpful
outcome.

Evaluation of training is also limited by reliability of the raters (such as check and
training captains in a simulator check) and the validity of the metrics and measures
used in the evaluation (Hays & Singer, 1989; Brannick, Prince & Salas, 2002). Inter-
rater reliability in aviation is a known problem and can influence the effectiveness of
the training in a number of ways. One way is that the raters may be influenced in

42
their rating of pilots based on the rank of the pilot (and their perceived experience).
For example, a check captain may rate a captain 4 out of 5 on a particular metric,
but would give a first officer a 3 for exactly the same performance, solely due to their
rank. Differences in ratings are particularly prevalent in rating non-technical skills
such as CRM (Brannick et al., 2002). As such, drawing conclusions about the
success of a system based on check ratings of pilots alone would not be a robust
way of evaluating the training system. The validity of metrics refers to whether or not
these metrics are measuring what was intended to be measured. Validity refers to
both internal and external validity, both of which can influence the success of the
training. Internal validity, the inference of a causal link between two variables, can
be affected if the controls are not robust or the statistical interpretations are
incorrect. External validity, the extent to which the causal relationships can be
generalised, can be affected if the participants in the study are not representative of
the final target audience, or if another aspect of the training means the results
cannot be generalised to the greater participant population (Hays & Singer, 1989;
Feinstein & Cannon, 2002).

There are some key concepts which have been highlighted in the literature as ‘best
practice’ for measuring performance in simulator training. These include: multiple
levels of measurement; addressing the process of performance as well as the
observable outcome; describing, evaluating and diagnosing performance; and
providing a basis for remediation (Salas, Rosen, Held & Weissmuller, 2009). Using
multiple levels of measurement will help to ensure performance is captured across
all dimensions, and various sources are used, which will enable raters to have
confidence the performance is repeatable and therefore valid. Understanding the
process of performance may involve using expert models of a task to compare and
evaluate performance, which will aid understanding of how an individual conducted
the task and if this was correct, as opposed to an individual fortuitously producing
the required outcome. The use of standards for comparison of performance can
help describe, evaluate and diagnose performance, particularly if they allow a
desired profile to be constructed. This may standardise the evaluation to some
extent and allow for effective feedback to the individual. The basis for measuring
performance is to provide remediation if needed. That is, it should support the
instructional and learning process and allow for the provision of feedback and
results to guide further learning and improve performance (Salas et al., 2009).

43
Following these concepts will give the best chance that the performance being
evaluated in the simulator is indicative of individual skill and is transferable to the
real world. Both are key issues in aviation that are required to ensure that simulation
training is valid, robust and applicable and will become increasingly important in an
aviation training world that is moving to increase the amount of training in simulators
while reducing the amount of time spent training in actual aircraft.

Summary
Studies into adult learning have shown that different techniques and methods are
needed when teaching adults as opposed to children. While each group may
benefit, to some extent, from the approach used for the other group, that is adults
may benefit from a pedagogical style and children from an andragogy style in some
cases, it is more problematic for adults to be taught as children are. The transition
from a pedagogical style to andragogy is also important for young adults moving
from a school education system to university, particularly in aviation. It is therefore
important that pilot training schemes ensure they can support this transition,
particularly the focus on becoming self-motivated and developing analytical skills as
opposed to the more rote learning system in place for children.

Pilot training is ultimately about laying a strong foundation of knowledge from which
the relevant skills in aircraft manipulation, flight management and non-technical
skills, such as communication, can develop. For innovations such as the MPL, it is
extremely important that this skill acquisition occurs within a contextually relevant
environment which aids the pilot in developing the relevant knowledge and skills
required to successfully handle both normal and non-normal events. While the
situated learning nature of this kind of approach may assist in skill acquisition, it is
important that the system recognises the difference between novice and expert
performance. Identifying this difference will aid the transition to expert performance.
That is, it develops the pilot over time rather than immersing them in the complex
system at the start in the belief this alone will develop expertise.

A major part of developing competence, and to a degree, expertise, will rely upon
the context of the training. As discussed above, various experiments particularly in
the area of cognitive skill acquisition have shown that it is largely

44
domain-dependent, and a large part of expert performance relies upon the domain
specific knowledge developing over time. Any training course aiming to produce
competent pilots should focus upon ensuring a level of commonality between the
tasks being taught and the environment they are learning within, with the tasks
required and the check and training environment of the aviation industry. If this is
achieved, the student’s learning benefit will be enhanced and the transition to line
operations in an airline should be improved. However it must be recognised that
knowledge in one domain is not necessarily transferable to another domain and as
such, a pilot’s demonstrated competence in one area of flight training is not
necessarily a guarantee of their overall performance in the system.

Numerous studies have shown a positive effect of simulator training on student pilot
performance, often with the pilots who have been trained in the simulator showing a
reduced number of hours required to achieve competency on the same skill in the
actual aircraft (Damos & Lintern, 1981; Lintern, 1980; Caro, 1988; Lintern, 1991;
Lintern & Koonce, 1992; Wiggins & Crognale, 2003; Wightman & Lintern, 1985;
Lindo, Deaton, Cain & Lang, 2013). In addition, studies have shown that the level of
fidelity of the simulator does not need to be high, especially when training novices
and advancements in pilot performance have been attained from part-task trainers
and computers (Caro, 1973; Taylor, Lintern, Hulin, Talleur, Emanuel Jr & Phillips,
1997b; Salas, Bowers & Rhodenizer, 1998; Feinstein & Cannon, 2002). What is
more important is the fidelity of the training and ensuring that simulators are not
used as a substitute for adequate training systems (Wiener, 1989; Harris, 2011).

As the studies detailed above show, simulator training can provide a contextually
powerful training environment that is not always possible to achieve in the ‘real
world’. For instance, while flying within the general aviation sector will increase a
pilot’s flight hours, it may set up unwanted behaviours or indeed not provide the
opportunity to gain experience in all areas relevant to an airline pilot. A simulator on
the other hand, can create an environment where pilots can learn and develop skills
when real life makes this skill acquisition hard. For example, a simulator can be
used to safely train a pilot to handle an engine out event during take-off at night, a
manoeuvre which, due to its high risk nature, should never be trained for in a real
aircraft.

45
Used correctly, simulation in pilot training can enhance the skill acquisition and
performance of pilots, particularly in regard to non-normal and emergency events.
However, this benefit will only be possible if the training system supports the
learning style and knowledge acquisition of the pilot and ensures that the emphasis
is not placed on simulator fidelity to the detriment of training fidelity.

46
1.5 Competency-based Training

Definition
Competency based training has been defined in Australian guidance as “…training
geared to the attainment and demonstration of skills to meet industry-specified
standards rather than to an individual’s achievement relative to that of others in a
group” with competency broadly defined as “underpinning knowledge as well as
values and attitudes”. (NCVER, 1999).

It should be noted however, that one of the most often cited problems relating to
CBT appears to be a lack of consistency in the definition, application and
standardisation of terms such as competency, competence and performance
(Hyland, 1994; Thomson, Mathers & Quirk, 1996; Milligan, 1998; Smith, 1999; Misko
& Robinson, 2000; Voorhees, 2001; Descy & Tessaring, 2001; U.S. Department of
Education, 2002; McMullan, Endacott, Gray, Jasper, Miller, Scholes & Webb, 2003;
Draganidis & Mentzas, 2006; Cowan, Norman & Coopamah, 2007; Guthrie, 2009).

Background
The concept of competency-based training was developed primarily in the United
States of America (USA) in the 1960’s/ 1970’s (Biemans, Neiuwenhuis, Poell,
Mulder & Wesselink, 2004) and was initially focused on organisational and teacher
training. In the United Kingdom (UK), the implementation of competency-based
training occurred during the 1980’s and 1990’s, when the government of the time
overhauled vocational education and training and adopted a model which reduced
the amount of theory taught to craft and technician apprentices as part of their
training (Boreham, 2002).

Philosophy
A simplified version of the ‘for and against’ arguments for competency-based
learning is outlined by Gonczi (1999). Those in favour of this approach argue that
traditional training is more likely to have “unnecessarily broad and vague aims”
(Gonczi, 1999, p180) which focus on what broad base of knowledge teachers will
attempt to pass on to students and that the whole approach lacks any kind of clarity
of purpose or assessment. The inference from this being that competency-based

47
learning does not suffer from these limitations. However the argument against
competency-based learning emphasises the need for knowledge in general and
highlights the problems inherent in trying to “make simple what is complex” (Gonczi,
1999, p181) by predetermining what is needed and specifying what is to be
achieved by the learning process. This can result in a series of identified skills that
are required to be achieved independent of understanding of the context within
which the individual is required to implement these skills.

Behaviourist and holistic approaches to CBT

In a review of the term competence, Norris (1991) outlined the two main approaches
to competence (and by inference competency-based training), as behaviourist and
generic. Behavioural objectives are “outcome and product orientated”, and under
this view of competence, the term is “usually treated as something a person is or
should be able to do” (Norris, 1991). In contrast, the generic competency approach
“favours empirical investigation to establish competencies which discriminate
between average and expert performance” (Norris, 1991). The main point here is
that the behaviourist approach deems someone as competent or not, whereas the
generic approach seeks to understand the difference between someone who is just
or not quite competent, and someone who is an expert. The second approach is
somewhat more holistic than the first, as it addresses the broader issue of
performance, including basic skills, personal attributes, values and ethics, lifelong
learning as well as workplace competencies rather than whether or not an individual
can complete a task (Guthrie, 2009). This goes someway to addressing the
criticisms of Hyland in the philosophical debate quoted below.

A number of papers have debated the philosophy underpinning competency-based

training, with one side of the argument dismissing competency-based training as
“behaviouristic”, labelling the holistic approach taken by Australia as a strategy to
“merely serve to mitigate the worst features of competence strategies” (Hyland,
1996; Hyland, 1997 and noted by Velde, 1999). The alternative view offers a more
positive assessment of competency-based training, particularly in relation to the
benefit of practical competence within an educational setting and the transparency
of assessment and outcomes, while acknowledging that some criticism is indeed
warranted (Bridges, 1996).

48
With regard to the intention for teacher education to become competency-based, in
line with the vocational competency-based training model in the UK, it was strongly
argued that teachers need to be experts and this requirement was seen as being at
odds with the aim and approach of CBT (Hyland 1992, 1993). The alternate view of
competency-based training as providing “practical attributes” is similar to this
concern and it is argued that part of the perceived limitation of the competency-
based training approach lies in the assessment of individuals (Bridges, 1996).

The introduction of competency-based training in the UK represented a shift away

from the traditional dual system for apprentices that required both classroom and
on-the-job training with their employer (Boreham, 2002). This move to competency-
based training removed the theory component as far as practical, leaving students
to learn the minimum theory necessary to be deemed competent during
assessment.

The German approach to CBT does not modularise vocational training, instead it
focuses on the profession as a whole and aims to have flexibility in the system and
in their apprentices, which goes beyond simply acquiring qualifications (Misko,
1999).The German dual system has delivered productivity and adaptability gains,
and is similar in part to the UK pre-CBT training system (Boreham, 2002). Of
particular note, the benefits of the German system included the skills focus with
strong emphasis on technical knowledge and understanding (Boreham, 2002;
Phillimore, 1997). Another study also examined the German dual system and
compared it favourably against the Australian apprenticeship system that was in
place at the time (Phillimore, 1997). However, it was also noted that the German
system encouraged further training and was closely regulated to protect workers.
The German system also promoted skill acquisition through rewarding workers on
the skills they had acquired rather than focusing on providing training to attain the
required skills only. It was in this context that the need for reform to the Australian
vocational education industry was highlighted and the German system championed
(Phillimore, 1997)

The Australian experience and perspective

Vocational education and training (VET) in Australia started in the mid-to-late
nineteenth century, offering industry training and skills in areas such as mining and
trade-related industries. In the 1974 Kangan Report on “Needs in Technical and
49
Further Education” was the first step in the introduction of what is now known in
Australia as the Technical and Further Education (TAFE) system (Department of
Education, Employment and Workplace Relations, 2010a).

While the TAFE system has relied on behavioural objectives, top-down curriculum
development and benchmarking since inception, the training reform movement in
the 1980’s tightened “the formula for curriculum development and implementation”
(Billett et al., 1999). This was the start of national standards, accreditation
frameworks and assessment practices (Billett et al., 1999). The introduction of
competency-based training was a response to “the perceived inadequacy of
Australia’s skill formation system to meet the needs of industry in an increasingly
competitive international environment” (Smith, 1999, p106).

The next major VET reform was in 1992, with the establishment of the Australian
National Training Authority (ANTA). ANTA was established by the Australian
Government and was introduced to coincide with a new VET system which relied on
industry input. In 1994 the Fitzgerald Report on VET led to further development of
the system including states and territories taking responsibility for accreditation and
standards endorsement and a more industry-led structure of training. By the late
1990’s, ANTA had introduced New Apprenticeships, established the National
Training Framework (NTF) and expanded VET to schools and embarked on
developing training packages (Department of Education, Employment and
Workplace Relations, 2010a). The VET model introduced to Australia was strongly
based on the system in place in the UK, one of functional competency, which is
“workplace focused and performance-orientated” (Guthrie, 2009). This is as
opposed to behaviourist competency favoured in the United States which stresses
the importance of observing successful and effective performers and determining
what differentiates them from their less successful counterparts (Guthrie, 2009). In
building upon this concept, Australia also inherited the tendency to downplay the
importance of underpinning knowledge and a holistic approach to gaining
competency in a role, compared to the systems in place in Germany and Austria
(Guthrie, 2009). It should be noted that this view is at odds with the work of Gonczi
(1994; 2000) and Hager (2000), who have reviewed professional industries in
Australia and who have implemented competency-based training which focused on
the underpinning knowledge and skills required for the role.

50
The Australian Qualifications Framework (AQF) was introduced Australia-wide in
1995. The role of the AQF is to “provide a comprehensive, nationally consistent yet
flexible framework for all qualifications in post-compulsory education and training”
(AQF, 2007). The National Skills Framework is made up of the Australian Quality
Training Framework (AQTF) and underneath that, Training Packages, which are
sets of nationally endorsed standards and qualifications for assessment. Training
Packages are developed by industry through skills councils (for instance, the
Transport and Logistics Industry Skills Council which covers aviation) and outline
the skills and knowledge an individual needs to perform a certain role. In keeping
with the philosophy of competency-based training, they do not contain prescriptive
rules for how individuals should achieve these skills (AQF, 2007).The AQF was also
“designed to provide a clear and rational structure in which an increasingly de-
regulated training market can operate while maintaining credibility within the overall
education and training system” (Misko & Robinson, 2000, p67). As part of the AQF,
evidence guides were developed, to support the assessment of competency in four
areas: task skills, task management skills, contingency-management skills and
job/role environment skills. These guidelines went some way to recognising the
importance of strategies to deal with non-normal or non-routine problems (Misko &
Robinson, 2000, p69).

By 1997, most vocational education participants were adults, training or re-training

for job-related purposes, with one quarter of these participants over the age of 40
(Misko & Robinson, 2000, p62). This gives some weight to the concept of CBT
encouraging and enabling the concept of lifelong learning.

The Australian Quality Training Framework (AQTF) was introduced in 2002 and was
endorsed by training ministers from all states and territories of Australia. The revised
AQTF comprises of standards which aim to ensure a “nationally consistent, high-
quality training and assessment service” as part of the Australian VET system (AQF,
2010). The key features of AQTF 2007 include that it is “outcome focused; nationally
consistent; streamlined; and transparent” (AQF, 2010). Research identified that
AQTF was “perceived as constraining because it focused on compliance and audit”
although it was noted that the next version of AQTF was going to address this issue.
(Harris et al., 2007, p4). Since this time various developments have taken place in
AQTF as it evolves to best suit the needs of industry.

51
In answering the criticism that competency-based training does not produce an
“educated” workforce, in that it focuses on technical and interpersonal skills to the
detriment of individuals contributing to society in socially productive ways, it was
noted that “CBT was never designed to do so” (Misko & Robinson, 2000, p78).
Instead the focus on introducing competency-based training in Australia was to
make the nation more internationally competitive. The authors are therefore, quite
reasonably, critical of criticism directed at a part of CBT the concept was never
meant to address (Misko & Robinson, 2000).

In a review of competency-based assessment in Australia, it was noted that the

competency based approach formed a key part of educational policy in Australia,
New Zealand (NZ), the UK, the US and Canada (Gonczi, 1994). The difference
inherent in the Australian system is the involvement of professions in CBT. This
difference is important as vocational training is often designed to teach an individual
how to complete a task or job; whereas tertiary or professional education historically
focused on providing a foundation of knowledge to underpin the task performance
required of the individual. Typically countries which have embraced competency-
based training have focused on “middle level occupations” and have viewed
competency-based training as inappropriate for training professionals such as
lawyers or doctors (Gonczi, 1994). However, Gonczi has detailed the numerous
ways in which competency-based training and, in particular, competency-based
assessment, can actually provide more valid results that traditional training and
assessment methods. This is achieved by adopting a holistic approach which not
only considers the tasks individuals are required to perform, but also the context. In
this sense, the “notion of competence is relational. It brings together disparate
things – abilities of individuals (deriving from combinations of attributes) and the
tasks that need to be performed in particular situations” (Gonczi, 1994).It does not
rely on each required task becoming a required competency (e.g. install component
in engine) nor does it generalise attributes and assume they are transferable
between domains. As such, it moves away from the idea competence can be
assessed by direct observation of behaviour. However, it should be noted that the
professions which have introduced this approach have done so as part of their
qualifications for entry to the profession, rather than changing how a university will
structure a law degree for instance. In this sense, the change has occurred outside
of the traditional education path of university, which focuses on providing broad

52
knowledge and then assessing individuals against the curriculum rather than
competency standards (Gonczi, 1994).

The view that CBT is inappropriate for professions is consistent with that found in
the survey of organisations implementing CBT, that a “professional model of
skills...was incompatible with CBT” (James, 2001). Another limitation reported by
those surveyed was “sometimes CBT assumes that people are going to be
competent in less time than it used to take”, which is not only indicative of a problem
with the design of the training, but also fails to account for individual variations, a
concept that should be the cornerstone of competence acquisition. As mentioned
above, it is likely that criticism of CBT in relation to professions is likely aimed at the
behaviourist approach or the generalist approach, rather than the more beneficial
holistic approach.

Competency-based training in Aviation

One of the greatest areas of contention regarding the introduction of the Multi-Crew
Pilot Licence (MPL), is competency-based training. This is due, in part, to ICAO
assigning a minimum of 240 total flight hours for the issue of an MPL, thereby
ensuring that pilots would receive at least 240 hours total flight time when licenced.
It has been highlighted that in using an hours-based requirement, the philosophy of
competency-based training has not been fulfilled (Hunt, 2007). Indeed much of the
criticism regarding competency-based training hinges upon the fact that the MPL
specifies an hour requirement and many training schools have opted to increase the
training hours in their MPL syllabus above the ICAO requirements as they see this
as the only ‘safe’ way to introduce MPL (Oaster, 2007; Pryde, 2007; Hu, 2007).
However, competency-based training is not a new concept in aviation and has been
the intended underpinning philosophy in flight training in Australia for a number of
years. In 2003, the plan for introducing competency-based training for all pilot
licences in Australia was detailed (MacFarlane, 2003). The main reason for this was
to align aviation with the requirement for all Australian industries to introduce
competency-based training, as discussed above.

Standards and design tools for competency-based training vary in detail and
applicability between disciplines. For example, vocational education in Australia is
overseen by the Department of Education, Employment and Workplace Relations,
but their standards and design guidance may not be applicable to a flight operations

53
training regime. However research into competency-based training design has
produced numerous useful tools and guidance (de Croock, Paas, Schlanbusch &
van Merrienboer, 2002; Curry, Caplan & Knuppel, 1994; Dick, Carey & Carey, 2004;
Baldwin & Ford, 1988; Oprins, Burggraaff & van Weerdenburg, 2006).The
fundamentals of training design and needs assessment are ensuring “the training
design includes factors such as training content, principles of learning, and
sequencing of training activities” (Baldwin et al., 1988; Curry et al., 1994).

One example of training and assessment in aviation is research that led to the
design and evaluation of a computerised tool to establish a competency-based
training program aimed at developing complex skills, trialled in Air Traffic Control.
The research was born from the acknowledgement that the areas such as Air Traffic
Control require “a competency-based approach that focuses on the integration and
coordination of constituent skills and transfer of learning” (de Croock et al., 2002).
The result was a training design tool which supports the needs of a competency-
based approach in this, and similar, dynamic domains.

This approach is very similar to the training requirements of MPL students –

obtaining the skills necessary to operate in a dynamic and complex environment
and transferring these skills from a simulated environment to the real world. Given
both the dynamic nature of aviation and the stated incorporation of competency-
based training for MPL, the focus for the introduction and implementation of MPL
should be on the support and education that is needed to underpin the training
regime delivered to students. That is, recognition of the need for MPL students to
achieve competency in a holistic system, as per Gonczi’s (1994) comments above.

Competency-based training in Australian aviation is achieved through a national

VET pathway and is aimed at ensuring individuals achieve a nationally recognised
qualification. Currently, the AVI08 “Aviation Training Package” covers all aviation
personnel, from ground operations to flight operations e.g. air crewmen, cabin crew,
pilots and air traffic controllers (Department of Education, Employment and
Workplace Relations Transport and Logistics Industry Skills Council, 2010a, 2010b).
The review of these standards is conducted by an industry group including members
from CASA, airlines, flight training organisations, ground service organisations,
unions, Airservices Australia, the Australian Defence Force (ADF) and logistics
organisations. The training package for aviation is issued in two parts - a guide for
users (part 1) and units of competency (part 2). The guide outlines what

54
competency is and how competency-based training and assessment should work.
The units of competency break down each unit into tasks with corresponding
elements for required knowledge and skills; range statements; evidence guides and
method of assessment.

One of the limitations of AVI08 is linked to the comments of Gonczi (1994), above.
As AVI08 is designed to cover the entire aviation industry, the required
competencies obviously vary between personnel and tasks. As such, the
competency statements for a tug driver are extremely detailed and very skill
orientated (e.g. ‘baggage tug is coupled to baggage carts’). In contrast, the
competencies for a pilot include areas such as ‘manage flight’. In this way, most of
AVI08 is a combination of manual tasks contextualised in a highly sophisticated
system (aviation). The benefit to pilots will be in application of a holistic approach
discussed above, rather than being assessed against areas such as ‘manage flight’
and “non-normal or emergency situations are recognised”. (Department of
Education, Employment and Workplace Relations Transport and Logistics Industry
Skills Council, 2010a, 2010b; Gonczi, 1994). The holistic approach will require more
than an assessor directly observing performance as it will require assessing pilots
against their overall competency, which includes skill, task completion and personal
attributes and not just their ability to recognise a non-normal situation in isolation to
the rest of the tasks that go with managing it. As such, it will have a higher
assessment burden than other areas with AVI08 seem to require. However, the
holistic approach to competency-based training would seem the most fitting for
airline pilots, given the professional aspect of their role and the limitations in
reducing that role to a number of tasks to be performed, without due
acknowledgement of the broader knowledge and attributes needed to conduct a
successful flight.

Benefits of competency-based training

While CBT has been extensively debated, analysed, criticised and judged in the
literature by academia and practitioners, there have been some documented
benefits, at least according to the end users.

Some of the benefits noted in a survey of managers in Australia included the view
that “what you need is what you get”, that “we know workers are competent when
they are employed”, largely as a function of a national, consistent system (James,
55
2001). The broad benefits listed by managers as “improved morale and improved
confidence in skills”, covered a wide range of attributes, including adaptability to
change, attitudes and morale. The benefit to workers was that they could “manage
upwards” and clearly articulate the competencies they needed to achieve in order to
get a pay rise or progress within the organisation (James, 2001).

The introduction of CBT has also been shown to increase employee motivation and
productivity, as well as resulting in a higher skilled workforce and increased
competitiveness in both corporate and construction industries in Australia (Misko &
Robinson, 2000 p77; Gilling & Graham, 2000 p146; Kellie, 1998; Kellie, 1999 p128).

In contrast to documented criticisms of CBT, benefits have been found in the

implementation of CBT in nursing in Australia, specifically that it did provide a
holistic approach to training that was more than just meeting individual
competencies (such as taking a pulse) (James, 2001). Paradoxically, this holistic
approach is achieved at the expense of standardisation, one of the hallmarks of
CBT in Australia. As such, the benefit to be achieved from CBT, such as using
experienced practitioners and assessors and adapting the training, actually goes
against one of the design intents for CBT – that of a “national standardised system”
of training. The benefits of using a holistic approach for CBT include that the
assessment focuses on values, knowledge and attitudes, which serves to improve
on traditional training approaches deemed to be fundamentally about examining
memory recall (Hager, 1995; Hager & Gillis, 1995; NCVER, 1999 p2; Bowen-
Clewley, 2000). In the holistic approach, the student must demonstrate the task as
well as showing an understanding of the knowledge and values of the workplace, an
example of which is “showing empathy to patients” rather than trying to assess
“empathy” as a standalone value (Hager, 1995, p55).

Further studies within medicine have found performance improvements in residents

who were taught surgical approaches using the competency-based approach and
for registered nurses in operating rooms (Martin, Vashisht, Frezza, Ferone, Lopez,
Pahuja & Spence, 1998; Voorhees, 1996). The performance of the residents was
assessed immediately after learning and three weeks later with a resulting
performance benefit that was significant. The time taken to conduct the procedure
was also reduced and the residents were found to have fulfilled the aim of “achieve
a mastery level” through CBT (Martin et al., 1998).

56
CBT was also shown to benefit Italian MBA students when integrated into the
existing MBA program, based on the requirements of the law firms who traditionally
employed the graduates (Camuffo & Gerli, 2004). The success of this program may
be influenced by the fact that the MBA was a university course which was imparting
a substantial amount of knowledge to coincide with the CBT component which
focused on simulations, case studies and role plays (Camuffo & Gerli, 2004).

This performance improvement in a field as complex as law was mirrored in

Australia with the success of the New South Wales Law Society Accreditation
Scheme, which also relied on performance based assessment and client
satisfaction ratings, as well as simulation and workplace assessment (Hager, 2000;
Gonczi, 1994; Gonczi, 1999). Competency standards have also been introduced to
pharmacy, nursing and engineering in Australia, all of which are considered to be
professions rather than vocations, which supports the benefits of holistic
approaches to CBT in professions (Guthrie, 2009). As discussed previously, this
introduction of competency standards in professions has occurred through
professional associations, although changes to universities in the area of ‘graduate
attributes’ was also noted (Guthrie, 2009).

Additionally, in order to get the greatest benefit from CBT, it has been suggested
that holistic approaches to CBT should focus on developing curricula to enable
individuals to learn not only how to deal with problems they will encounter in their
workplace within a realistic setting but also to learn how an expert would perform in
the same situation (Gonczi, 1999; Diwakar, 2002). To achieve this, the distinction
between “…expertise as an outcome and the acquisition of expertise as a process”
must be understood (Gonczi, 1999 p187).

To this end, the integrated CBT approach concentrates on how an individual uses
experience to form expertise through concepts such as staged learning, stepping
from novice to expert, and problem-based learning (Gonczi, 1994). Problem-based
learning is closely aligned with the integrated learning and assessment model and
would provide the basis for individuals to develop and build on competence in the
workplace, turning that into expertise (Gonczi, 1999). In order to facilitate the move
toward expertise, it is suggested that teachers “need to understand the conditions
under which experience can lead to expertise and the teaching methods which will
facilitate the development of any generic skills” (Gonczi, 1999, p187).

57
Countering this however, is research showing that high levels of competence or
expertise is actually quite context specific, which calls into question the generic
competencies assumption that often underpins CBT (Shanteau, 1992, p259; Misko,
1999). Another review of competence in professional domains also alludes to this,
within the context of pharmacists, and notes that this industry “should take a
complex view of professional competence, focus on the integrated behaviours that
differentiate performance and use evaluation methods” as strategies to avoid over
simplifying competence (Torr, 2008).

Criticisms of competency-based training

The use of ambiguous language in the development of competency-based training
guidance has led to an oft cited criticism of non-standard definitions and the
mislabelling of various terms (Hyland, 1994; Thomson et al., 1996; Milligan, 1998;
Smith, 1999; Misko & Robinson, 2000; Descy & Tessaring, 2001; U.S. Department
of Education, 2002; McMullan et al., 2003; Draganidis & Mentzas, 2006; Cowan et
al., 2007; Guthrie, 2009). One review on competency management systems found
12 definitions of the term competency (Draganidis & Mentzas, 2006). Another
identified confusion regarding the definition of competence with definitions ranging
from broad statements to specific definitions, with statements like “competence is
essentially concerned with performance” and “it is about what people can do rather
than what they know” (Hyland, 1994).

It is reasonable to assume that before any system can be introduced, especially one
that is labelled as an educational reform, that these sorts of issues would have been
resolved. In essence the concept of CBT is one that should be easily transportable
between industries, countries and cultures. However it is hard to see how this is
achievable when the definitions and labels used to describe the CBT system are not
consistent or standardised.

A specific example of the issue this presents is given by the U.S. Department of
Education working group, whose mandate was to analyse the practices in defining
and assessing learner competencies in the United States. They noted that “The
bundles of different skills and knowledge that are given the same label in different
contexts is one reason there is often difficulty in achieving a common understanding
of what a given competency (like leadership) is, and then what it means to assess it”
(U.S. Department of Education, 2002, p9).
58
Another concern amongst critics of the approach of competency-based training is
that it focusses most strongly on performance, with less regard given to
underpinning knowledge and skills, with the inferred criticism that this results in a
test orientated learning environment, where students learn what they need to know
to pass the exams, not what they should gain as part of an education (Hyland, 1994;
Leung, 2002; Mulder, Weigel & Colllins, 2007). In addition, this disconnect between
underpinning knowledge and performance may mean that students miss out on the
background information needed to help them develop expertise, the lack of which
may result in rote learning and little integration of competencies with the
underpinning knowledge necessary to do the job (Mulder et al., 2007). This overall
criticism is one that has provoked heated debates within the education community,
with the divide being between those that view CBT as suitable for all professions,
and those that view it as unsuitable for training professionals such as lawyers and
doctors (Hyland, 1994; Gonczi, 1994; Milligan, 1998; James, 2001; Leung, 2002).
Within this criticism sits the concern that the link between competence and
performance is not direct; that is, two individuals may be deemed competent when
tested in an artificial environment, but this does not ensure consistent performance
in the real world and as such their performance on the job may differ dramatically
(Mulder et al., 2007). This concern has ramifications for the introduction of CBT in
domains which require standardised and consistent performance, such as within
aviation.

A focus of this criticism has been on the provision of education versus training. The
long standing view is that universities provide students with knowledge and skills
“education” whereas vocational training systems provide “training” and not
education. As such, it is argued that CBT may be suited to vocational training but is
incompatible with a university model (Hyland, 1994; Hyland, 1996; Milligan, 1998;
Leung, 2002; Priest, 2009).This difference between the intent of education versus
vocational training was recently emphasised by a review of education in the United
States, which identified the extremes to which this system had moved (Grubb &
Lazerson, 2005). On one side, the “professional” universities who educate in
professional streams such as medicine, engineering and law produce graduates
who are often criticised for their lack of ability to work in the “real world”, their lack of
practice in their discipline (such as lawyers who can’t draft contracts) and lack of
hands-on experience despite their strong academic grounding. Conversely,
institutions which focused on vocational training, such as for teachers, were

59
accused of not passing on critical thinking skills, focusing too much on theory and
not equipping graduates with either enough academic skills or practical skills (e.g.
the ability to discipline children and classroom management). The solution
suggested by the review mirrors many recommendations on improving competency-
based training (such as assessing interpersonal skills and other “non-essential”
skills which are nevertheless a critical part of the individual’s profession), as well as
building upon directing research in academia to better reflect practice (Grubb &
Lazerson, 2005). It appears that a balance of the two approaches is required.
However, despite the fact that many view CBT as the way to achieve this balance,
there is very little evidence that it actually achieves this.

Criticism regarding the implementation of CBT, especially in Australia, includes the

complaint “that the standards were too prescriptive and prevented enterprises from
interpreting the standards according to their own contexts” (Misko & Robinson, 2000
p66). This criticism appears to be related to the confusion and misuse of terms of
competence mentioned above, which has resulted in the perception that CBT is a
structured course which will deliver a set goal and no flexibility is allowed (James,
2002; Boreham, 2002; Biemans et al., 2004). The main concern regarding this view
is that CBT will produce people who can do specified tasks but cannot move beyond
these tasks (that is, they lack flexibility and the ability to transfer learned skills), who
cannot innovate or lead change in the workplace, such as adapting procedures or
training methods as they are seen as “fixed” and “approved” under a CBT system
(Billett et al., 1999; James, 2002; Boreham, 2002; Watson et al., 2002; Biemans et
al., 2004; Cartledge & Watson, 2008). Further complaints from industry included that
CBT “appears to discount tacit knowledge and experiential learning” (James, 2001)
and a European review of how entrepreneurs learn found that ‘the focus on
competencies, rather than specific knowledge, attitudes or skills for
professionalisation demands more complex, continuous learning environments”
(Lans et al., 2004). It is of interest to note the underlying assumption appears to be
that teaching CBT is a matter of following a series of markers and tasks, yet little
consideration has been given to how that will help an individual achieve competence
in the real world.

The frustration of educators within the CBT system appears predominately linked to
their lack of involvement in introducing the concept. This is especially true within the
Australian environment, where the prominence of industry working groups,

60
government and unions in introducing CBT meant there was little involvement from
educators and also a lack of acknowledgement that this would present an issue
(Cartledge & Watson, 2008; Gonczi, 2000; Smith, 2010). The main concern from
educators and researchers in introducing CBT was that the concept would not result
in the development of skilled workers, or any “deep-level learning and problem
solving capabilities needed for the new workplace” (Gonczi, 2000 p20).

Contextual learning and assessment

A major catalyst for the change from traditional approaches to training to a
competency-based system is often stated as the need to take into account people’s
abilities, skills and what competencies they are capable of displaying (Descy &
Tessaring, 2001). This is in direct contrast to an assessment which occurs within a
set time and space and relies upon a demonstration of competencies as they occur
before an assessor. There are conditions that need to be met in order to ensure that
certification mirrors the occupational (operational) requirements of the certificate and
ensures the skills are transferable and the assessment valid and balanced (Descy &
Tessaring, 2001; Bloch, Clayton & Favero, 1995; Cropley, 1995).

In order to address the changing environment in Europe, the need for competencies
to meet the new demands of job roles was identified (Descy & Tessaring, 2001). In
particular, the need for updated knowledge; higher standards of general education;
and identification of non-technical skills (explained here as communication, problem-
solving, inter-personal relationship building, autonomy) was emphasised. The
conclusion drawn from the examination of traditional training in Europe was that the
current systems of training and assessment cannot meet the requirements of these
necessary changes to the required competencies. The need for the certification or
qualifications to reflect the performance standards of the job being trained for is
considered very important, particularly for applicability to the industry (end user).
However, it was also noted that “A balance therefore has to be sought between the
precision of the reference framework through which an individual’s particular skills
can be readily identified and the degree of generality which will enable these skills to
be recognised in the broadest possible spectrum” (Descy & Tessaring, 2001, p80).
This is due, in part to the contextual element of learning, specifically that “Learning
is fundamentally contextual. Skills are acquired within a context and a community of
practice which therefore need to be taken into account for assessment” (Descy &

61
Tessaring, 2001, p81). This is particularly important for aviation and training
innovations such as MPL, which favour contextually relevant training environments
and assessment.

A related area of the contextual nature of learning is the ability to transfer

competency between similar domains. It has been largely noted that competence is
closely related to the domain and activities involved (Guthrie, 2009; Descy &
Tessaring, 2001; Capper, 2000), and specifically that “An individual wholly
competent to solve problems within his occupation will be able to transfer this
competence to a family of related occupations, but will not be capable of solving all
problems of whatever nature. This approach recognises the fundamentally
contextual character of competence” (Descy & Tessaring, 2001, p122).

The problems associated with contextual learning in complex systems such as

aviation have been identified, specifically the disconnect between “known, repetitive
and predictable settings” for assessment and “novel situations” that may actually be
encountered in real life (Capper, 2000, p147). What this essentially means is that it’s
likely the performance being assessed will be at the skills-based cognitive level of
human performance. That is, they are routine, highly practised tasks, whereas
rule-based and knowledge-based performance start to move into the area of more
experienced and expert performance. Rule-based performance involves applying a
known rule in a changed but familiar situation, with knowledge-based performance
focused on problem solving in a novel environment (Rasmussen, 1983). The
detriment of assessing only skill-based knowledge to determine overall competence
can be seen in the figure of “as many as 80% of aviation accidents occur when the
third level of performance is required, but has not been trained for” (Capper, 2000,
p149).

Knowledge-based performance is the deeper level of understanding and as such, a

system which does not evaluate this foundation of knowledge will not be able to
judge whether or not an individual possesses the relevant competence to face an
unexpected emergency event. In part, the very nature of the events means they are
rarely trained for in reality, particularly in conditions of change and uncertainty
(Capper, 2000).

The lack of focus of competency-based training on knowledge-based understanding

aligns with earlier mentioned criticism that an individual may not receive the
underlying understanding and knowledge they need for the job, rather they are
62
shown the tasks necessary to complete the job and pass the assessment. As such,
any assessment regime within competency-based training should take a holistic
approach to ensure an accurate reflection of an individual’s performance can be
attained through multiple metrics and stable performance over time (Scheele et.al.,
2008).

The influence an organisation, and specifically its culture, has on an individual’s

performance and assessment of skill is also contextual. That is, organisational
culture will influence the performance of the individual during training and
assessment (Capper, 2000). As such, we need to focus on more than just the
individual and, in aviation, start to examine team performance in novel situations.
Part of this focus should involve recognising the benefit of both novices and experts
in these situations, and an important consideration is that expertise may actually
reside with individuals who appear “inadequately experienced” yet the expert may
approach “novel situations with fatally channelled thinking” (Capper, 2000, p 167). It
is also difficult to assess novice and expert performance as “…many of the
heuristics used by novices are replaced by shortcuts in the hands of experts…”
(Epstein & Hundert, 2002), this can also be seen by the ability of more experienced
staff to conduct ‘workarounds’, that is, to step outside of the formal system to
achieve the desired goal (Brown & Duguid, 1991).

In summary, an individual may be capable of skill-based performance; however both

rule and knowledge-based performance will sometimes require team work in order
to achieve an optimal outcome. While the concept of assessment requires careful
consideration, especially when claiming validity, transferability or standardisation, it
would be “absurd to conclude we must therefore abandon the effort” as we need to
ensure a level of competence in professionals (Capper, 2000, p166). Rather, we do
need to understand the issues and factor them into the design of assessment, as
well as being careful to state only the level and limits of capability and competence it
is possible to actually judge. As such, it is worth emphasising that “whereas
performance is directly measurable, competence is an inferred quality” (Epstein &
Hundert, 2002).

63
Summary
Competency-based training is a cornerstone of Australian vocational education and
has been a requirement in Australian aviation for a number of years. Despite this,
there is still strong opposition to the concept of competency-based training, both
within the wider educational community and aviation in particular (Hyland, 1993;
Hyland, 1996; Oaster, 2007; Hunt, 2007).

The fundamental concerns surrounding the use of competency-based training are

centred upon philosophical differences between holistic and behaviourist
approaches to CBT and the relevance of CBT to professions. These criticisms are
focused upon the relevance of a vocational approach to training professionals, who
should have a sound knowledge base for their occupation, which appears at odds
with the ‘learn on the job’ approach of CBT (Bridges, 1996; Hyland, 1993; Hyland,
1996; Leung, 2002).

However, the benefits of CBT have been demonstrated, with research showing an
improvement in morale and motivation of employees, as well as increases in
productivity and competitiveness (James, 2001; Kellie, 1998). In addition, the
introduction of CBT into professions such as law and medicine have demonstrated
that it is able to be implemented for professions, however this appears to be at the
expense of standardisation, a hallmark of CBT (James, 2001; Hager, 1995; Martin
et al., 1998).

Despite the limitations of competency-based training, and the concerns of the

aviation industry regarding the use of CBT for innovations such as MPL, it is
probable that aviation is well suited to a holistic approach to CBT. This holistic
approach incorporates multiple assessment methods, the technical and non-
technical aspects of the job and is flexible enough to support the trainee’s learning
requirements (James, 2001). However this occurs at the expense of standardisation
and a desire to promote a national standardised system of training, which the
Australian approach to CBT has focused upon (NCVER, 1999; Hager, 1995). One of
the greatest benefits of CBT to aviation is the application of a holistic approach to
CBT curricula to enable individuals to learn the traits of expert performance;
however this is contingent upon the distinction between expertise as an outcome
and the process of acquiring that skill to be recognised and understood (Gonczi,
1999).

64
Within aviation, the introduction of CBT has been assumed to have occurred by
virtue of the regulatory requirement to do so, however whether or not this is actually
the case does not appear to have been examined. In addition, the paradox of having
a requirement for pilots to accumulate flight hours before they can be licensed whilst
apparently being trained under a competency-based system appears not to have
been widely identified or addressed (Hunt, 2007). Indeed, the extent to which it has
been identified is in response to the perceived ‘safety’ of increasing the hours
required for MPL students under a competency-based system (Oaster, 2007; Pryde,
2007; Hu, 2007).

65
1.6 Regulation of pilot training and licensing

International regulation
The International Civil Aviation Organization (ICAO) is a specialized agency of the
United Nations and was created from a conference held in Chicago in November
1944. This conference created a Convention on International Civil Aviation (also
known as the Chicago Convention), which was signed on 7 December 1944. ICAO
was set up as “a means to secure international cooperation and highest possible
degree of uniformity in regulations and standards, procedures and organisation
regarding civil aviation matters” (ICAO, 2010).

ICAO achieves this by issuing Standards and Recommended Practices (SARPS) in

Annexes to the Chicago Convention. Each contracting state, that is a state that has
ratified the Chicago Convention, is required to incorporate the SARPS into their
legislation as far as possible and lodge a statement of any difference with ICAO
where their legislation may differ from the SARPS.

In Australia the Civil Aviation Safety Authority (CASA) has developed Civil Aviation
Regulations to fulfil the requirement to standardise with ICAO (ICAO, 2006a; CASA,
2008). ICAO Annex 1 details the requirements for training and licensing pilots, and
the different levels of licence and the privileges that each licence offers. Once again,
contracting states to ICAO must incorporate these licence structures into their
legislation to aid global harmonisation. Presently, the pilot licence types include:
Student Pilot Licence (SPL); Private Pilot Licence (PPL); Commercial Pilot Licence
(CPL); Air Transport Pilot Licence (ATP or ATPL) with the most recent addition of
the Multi-crew Pilot Licence (MPL).

In recognition of the global licensing regulation of ICAO, Australia will convert an

overseas licence to an Australian licence once the applicant has passed a flight
rules and air law examination (covering Australian air law), a flight test, and obtained
an Australian medical certificate. If the licence is a CPL or ATPL issued in New
Zealand, the Australian licence can be issued under the Trans-Tasman Mutual
Recognition Act (1997).

66
Australian regulation
As the Australian aviation regulator, CASA is broadly responsible for developing
standards and regulations, implementing the regulations, auditing aviation operators
against regulations and conducting safety education activities.

As part of their role in standards and regulatory development, as well as fulfilling the
ICAO mandate as discussed above, CASA is responsible for pilot licensing. This
includes setting regulatory standards for the issue of pilot licences, providing an
approved training syllabus and guidance on flight training standards.

These regulatory standards include minimum flight hours for the issue of a pilot
licence, the hours of which vary with the licence being issued. Currently the
minimum total hours required for the issue of a Commercial Pilot Licence (CPL)
through conducting an approved flight training course is 150 hours, with 70 hours in
command (including 20 hours cross-country command) and 10 hours instrument
flying and 5 hours of instrument flight time in an aeroplane (CASA, 2008). If a pilot
does not attend an approved flight training course, they must have accumulated at
least 200 hours total, with a minimum of 190 hours of flight time as a pilot; 100 hours
in command; 20 hours of cross-country command; 10 hours of instrument time and
5 hours of instrument flight time in an aeroplane (CASA, 2008).

The current minimum hours required for the issue of a Multi-crew Pilot Licence
(MPL) are a minimum of 240 hours that includes: at least 40 hours of flight time as
pilot of an aeroplane; at least 10 hours in command; at least 5 hours cross-country
command, with any required hours not completed as flight time must be completed
as simulated flight time in an approved flight simulation training device for the
purpose. There are also a number of additional conditions MPL pilots must meet,
including having completed 12 take-offs, and at least 12 landings, in an aeroplane of
the type used for the MPL applicant’s flight test. Although, this number can be
lowered to between 6 and 12 if CASA is satisfied “the lower number does not
adversely affect the acquisition of the required skill by the applicant” (CASA, 2013).

67
1.7 Pilot Training philosophy and methodologies

History and international influence

The necessity of obtaining and training large numbers of pilots to serve the
demands of the Second World War saw the formation of dedicated research into
what skills were needed in the cockpit of military aircraft. At the same time, the
concept of ergonomics was gaining momentum as it was realised that the design of
the cockpit played an important part in the operation of the aircraft (Hawkins, 1987;
O’Hare & Roscoe, 1990).

Due to the high tempo demands of the war, pilot training was conducted with the
aim of rapidly giving pilots the skills they needed to fly the plane and carry out their
associated duties (O’Hare & Roscoe, 1990). After the war, a syllabus of training was
developed, the foundations of which came from the lessons learnt during this time,
including in the area of physiological limitations (such as visual illusions),
ergonomics (human/machine interface) and the environment (e.g. the effect of
altitude on human performance). Despite advancements in technology in aviation,
even as late as 1990 it was noted that training had not developed at the same rate,
with many of the instructional techniques from the second world war still in use
(O’Hare & Roscoe, 1990).

US recruitment of civilian pilots into airlines was generally from the military, with “up
to 75 per cent of new hire airline pilots were recruited after commencing their flying
career in military aviation” (Harris, 2011, p121) whereas in Europe, airline
recruitment is predominately of ab-initio trainees, or cadets, a system that has been
in place in Europe since the 1970’s (Hawkins, 1987; Harris, 2011).

Pilots who do not enter an airline from the military or another airline come from
general aviation. General aviation is the sector of the aviation industry which is not
airline or military and covers a wide spectrum of activity and aircraft types (Ritchie,
1988). In Australia, the majority of general aviation activity is made up of Charter
(flying passengers on-demand as opposed to on a fixed schedule) and Training
(Bureau of Infrastructure, Transport and Regional Economics, 2012). In 2010 in
Australia the general aviation sector of the industry accounted for 1,848,000 hours
flown, with regular public transport (i.e. airlines) accounting for 1,338,000 hours
flown. While the activity in general aviation was higher overall, there was a far

68
greater spread of activity within general aviation, including charter, training, aerial
work, private, business, agricultural and test & ferry flights (Bureau of Infrastructure,
Transport and Regional Economics, 2012). General aviation provides a training
pathway into the airlines by allowing pilots to accumulate the necessary flight hours
in order to enter an airline, generally through charter and aerial work

Australian model
The Australian approach to pilot training has established two distinct pathways for
civilian commercial pilots. There is the more traditional path of obtaining a private
pilot licence (PPL) before accumulating enough flight hours to meet the criteria to
conduct a flight test for commercial pilot licence. This route requires the individual to
attain 200 hours total, with 100 of these in command. In contrast, the accelerated
commercial pilot training path allows individuals to become licensed with 150 hours
total time and 70 hours in command. This pathway has typically been used by flight
training organisations when training pilots to enter an airline as a cadet. As such, the
requirements on this training pathway are more rigorous and require the student to
conduct the flight training without a significant break, unlike the traditional pathway.

Australia is unique in a global aviation sense in that it has a strong general aviation
sector of the industry, which provides an avenue for pilots to enter into the airlines
(Bureau of Infrastructure, Transport and Regional Economics, 2012). Typically in
Australia the entry pathways to an airline will be through the military, from general
aviation, from another airline or as a cadet. Cadets typically have no prior flight
experience before commencing training, and transition from training directly to the
airline. The position in which cadets then enter will vary according to the airline.
Typically they have entered as a second officer prior to moving to the role of first
officer, however some airlines have placed cadets directly into the first officer role.

Innovations in pilot training

In 2004 ICAO tasked its Flight Crew Licensing and Training Panel (FCLTP) with
reviewing current licensing and training standards (Annex 1 and Annex 6) with a
view to introducing a new pilot licence (Sutton, 2005), which would tailor flight
training to deliver an ‘airline pilot’. The traditional system produced a commercial
pilot who then had to be introduced to airline standard operating procedures and

69
culture. This licence is the Multi-Crew Pilot Licence. One of the drivers for the
establishment of the licence was the ‘pilot shortage’ problem facing airlines and the
realisation that a more effective flight training approach was needed in order to
reflect the skill set of airline pilots.

The panel was also asked to determine “whether competency-based flightcrew

standards could complement and/or replace existing standards based on
knowledge, skill and experience that have been in use since Annex 1 came into
being”. It was also to assess the extended use of flight simulators “in acquiring or
maintaining the competencies required for the various levels of licenses and ratings,
taking into account the type of simulator being used, and determine the credit to be
given” (Sutton, 2005).

After reviewing the current standards, the FCLTP recommended the introduction of
the MPL. ICAO has set the requirements for an MPL as 240 hours total time, with 10
hours in command during training, and the majority of that time is to be completed in
a certified simulator (ICAO, 2006a, ICAO 2006b). While this is a departure from CPL
hours (150 hours total time, 70 hours in command) the biggest change is in the
increased use of simulation and the decrease in both actual aircraft time and
command time for MPL pilots. In recognition of the need of stronger non-technical
skills in modern pilots, the MPL syllabus also incorporates CRM, threat and error
management and crew communication from the commencement of training.

According to Sutton (2005), one of the benefits of MPL is that it “would introduce
competency-based training in place of outdated or irrelevant experience
requirements”. As discussed in the earlier section on competency-based training, it
would appear that elements of competency-based training are suited to training
professionals; however this must be considered when devising the syllabus and
assessment methodologies.

Criticisms of pilot training models

It has been acknowledged that the first stages of commercial pilot training (including
the syllabus requirements) has not significantly changed since the 1930’s and for a
period of time that did not present a great problem (O’Hare & Roscoe, 1990; Harris,
2011). However, with the advent of modern jet aircraft, particularly highly automated

70
‘glass cockpit’ aircraft, this model is becoming more and more outdated. In
particular, the timing of the transition of new pilots from old style training aircraft, to
these new aircraft has been questioned, with the case argued that the relatively late
transition is making it harder for pilots to learn the skills they need as they are also
concurrently learning about multi-crew and transport operations (for a discussion of
the benefits and limitations of this approach, see Harris, 2011). This same issue was
noted in relation to ab-initio pilots, who conduct their training on light piston twin-
engine aircraft and who, as a result, “are hardly equipped to cope with the
complexity, speed, size and inertia of modern jet transports” (Smallwood, 2000,
p299). The need for contextually relevant training for ab-initio pilots, which will
enhance their performance once they are in an airline, is inherent in this statement.

What is clear is the need for a training needs analysis to be developed that clearly
defines the skills needed of an airline pilot in modern aircraft and construction of a
training and evaluation system that will ensure these skills are achieved and match
what is needed on the line (Harris, 2011). A basic review of the traditional approach
to training questions the need for teaching ab-initio cadets about piston engines and
visual low level navigation given the lack of piston engines in modern airliners and
the low likelihood they will ever rely on visual low level navigation to reach a
destination (Harris, 2011).

A more fundamental issue regarding pilot training was raised in relation to the
distinction between “training in skills and procedures and the judgmental education
of a professional airman. Traditional general aviation training is concentrated on the
former with little attention to the latter” (O’Hare & Roscoe, 1990, p62). This is in line
with the concerns raised in the competency-based training section which also
highlighted the need to educate pilots rather than train them in the various tasks
associated with being a pilot with the belief this will produce overall competence.

Summary
Historically pilot training pathways have been through the military, general aviation
or cadet schemes. The advancements made during the second world war in both
aircraft technology and ergonomics enhanced the reliability of aircraft and pilot
performance. However these advancements also changed the training needs of

71
pilots, which haven’t kept pace with the advancing technology of aircraft (Harris,
2011).

In recognition of this need, the International Civil Aviation Organization researched

and then implemented a training innovation aimed at producing an airline ready first
officer capable of occupying the right hand control seat from entry to the airline. This
licence, the MPL, is focused on providing contextually relevant training for cadets
who will take on the first officer role at the completion of training.

However, it should be acknowledged that competency-based training, one of the

major changes of the MPL, may not be without its problems and limitations, and
may not be able to replace the current flight hours focus of industry.

72
1.8 Rationale for studies

The aim of the studies in this thesis is to examine the current pilot training system
from different perspectives to address the question of ‘what makes a good airline
pilot?’ To do this, the current commercial pilot licence training system in Australia is
examined, with particular attention focused upon the competency-based training
requirements that are currently within the Civil Aviation Safety Authority’s (CASA)
pilot training syllabus (CASA, 2008). In addition, the performance of airline pilots is
examined to determine the extent to which pilot experience (in this case flight hours)
contributed to successful performance in normal and non-normal operations. To do
this, we examined data from normal line flights and gathered data relating to
simulator checks that these pilots were required to do to maintain their licence. We
also examined the effect of the entry pathway of pilots into an airline to establish the
extent to which this altered their performance. Direct-entry pilots (that is, pilots who
had come from other airlines, the military or general aviation) were compared to
cadet (often ab-initio) pilots who had undergone training for a specific airline. This
allowed exploration of the pilot training system as a whole, as well as enabling a
focused review of the desired end product of recent pilot training innovations – the
performance of airline pilots in normal and non-normal scenarios.

This thesis presents a number of studies that were conducted to bridge the gaps in
the literature by exploring the existing commercial pilot training system and in
particular, the difference in the performance of pilots above and below the 1,500
hour threshold. With an increase in demand of pilots globally and a constant desire
by airlines to save money, coupled with recent accidents which highlight skill
shortages of pilots in relation to advanced systems, pilot training now more than
ever needs to produce pilots who can operate in these ultra-safe yet complex
systems (National Transport Safety Board, 2010; Bureau d’Enquêtes et d’Analyses
pour la sêcuritê de I’aviation civile, 2012; Boeing, 2013, p31). The MPL is a licence
which may provide a means for the kind of innovative and tailored training that is
needed. However given that airlines are only now training MPL pilots, their
performance in an airline will not be able to be studied for a few years. As MPL
pilots were not able to be studied, we instead focused on low-hour CPL pilots, with
the view that much of the opposition to MPL was directed toward the low-hours
aspect of the licence. As such, the three studies aimed to explore the underpinning

73
principles of MPL – competency-based training and reduced flight hours experience
in the context of airline operations.

The aim of the first study (Chapter 2) was to examine commercial pilot licence
records to determine the overall spread of experience at licence issue and examine
whether or not there was an effect of the flight training organisation which conducted
the training. A series of one-sample t tests were used to test for significant variation
from the CASA-prescribed regulatory minimum hour requirements. Additionally, a
subset of the data was examined to test for the effects of a specific flight training
organisation’s syllabus on the flight hour accumulation of the students.

The aim of the second study (Chapter 3) was to examine whether there were any
systematic differences in crew performance (against technical and non-technical
metrics) in normal flight operations evident as a function of flight hour experience.
The hypothesis examined was that there would be a difference in performance
based on flight hours and, specifically, that both the technical and non-technical
metrics would demonstrate better performance for the high-hour pilots.

The third study (Chapter 4) examined pilot performance in an airline environment by

examining the performance of low- and high-hour second and first officers against
technical and non-technical metrics as well as regulatory required manoeuvres in
both a simulator check flight and a line check flight. In addition, the entry pathway of
pilots into the airline was also examined against the same metrics, by comparing
direct-entry pilots with cadet (generally ab-initio) pilots. The aims of the study were
to test for a difference in performance in the simulator proficiency check for normal
and non-normal operations, based on the entry pathway to the airline and the hours
experience of the pilots and to compare the results with the previous study (Chapter
3) in examining if there are differences in pilot performance in the line checks based
on entry pathway or hours.

It is anticipated that the findings of this thesis will provide useful information for
airlines (including their training departments), regulators and pilots regarding the
extent to which competency-based training systems are in place in Australian
aviation as well as the effect of both flight hours accumulated and entry pathway on
the performance of airline pilots. By understanding the differences in performance

74
as a function of pilot hours and/or entry pathway, airlines and regulators can use the
available evidence to make informed decisions about appropriate training
innovations and interventions.

75
Chapter Two - Experience, competence or syllabus? Influences on
Flight Hours at Licensing of Commercial Pilots1

1
Todd, M.A. & Thomas, M.J.W. (2013). Experience, Competence, or Syllabus? Influences on Flight
Hours at Licensing of Commercial Pilots. International Journal of Aviation Psychology, 23(2), 169-
180.

76
2.1 Flight Training

The provision of flight training worldwide is guided by the International Civil Aviation
Organisation (ICAO), with the categories of pilot licences (private, commercial, air
transport) being detailed in Annex 1 (ICAO, 2006a). Each contracting state to ICAO
recognises these licence types in their legislation and nominates the required hours
to be obtained by student pilots to achieve each licence type.

While the basic structure of the pilot training and licensing system has not changed
considerably since the Second World War, there have been some recent advances
in training philosophy. The introduction of the Advisory Circular for the Advanced
Qualification Program in the United States in 1991 (FAA, 2006) allowed approved
airlines to tailor their training and checking to the needs of the individual through the
use of task analysis and data. This represents a move from conducting training and
check to meet standardised regulatory criteria, to a system that benefits the
individual pilots and the organisation by relying on data to ensure an equivalent or
increased level of safety compared to meeting the more traditional targets set by
regulators (for more on evidence based training, see IATA, 2011).

The introduction of the Multi-Crew Pilot Licence (MPL) to Annex 1 by ICAO (ICAO,
2006a) is another shift in training philosophy that focuses on providing relevant
training to ab-initio pilots for direct airline entry. At the core of the MPL is a focus on
competency-based training, which should represent a shift from meeting regulatory
standardised flight hours to meeting the needs of the individual students.

Competency-Based Training
Since its inception, the concept of competency-based training (CBT) has been much
debated around the world. Various studies have been conducted on the introduction
of CBT into various domains, particularly nursing and teaching. CBT appears to
have gained the greatest following in the US, England, Australia, New Zealand,
Canada and parts of Europe (Gonczi, 1994; Biemans, Nieuwenhuis, Poell, Mulder &
Wesselink, 2004). It has been, or currently remains, government policy in most of
these nations that vocational education and training (VET) incorporate CBT as far
as possible.

77
The perceived wisdom of this policy has been contested by numerous authors
(namely Hyland, 1992, 1993, 1997 and Bridges 1996), who have openly argued the
relative soundness of the philosophy underpinning CBT. Despite any disputes
regarding the concept of CBT, one point which everyone appears to agree is
confusing and counterproductive is the wide variety of definitions given for the
elements of CBT, including the use of the terms competence, competency and CBT
interchangeably (Hyland, 1994; Milligan, 1998; McMullan et al., 2003, Draganidis &
Mentzas, 2006; Cowan, Norman & Coopamah, 2007). In both International and
Australian studies, various authors have noted the disparity of definitions for
competence; competency-based training; competency and competent (Hyland,
1992, 1994; Milligan, 1998; McMullan et al., 2003; Draganidis & Mentzas, 2006;
Cowan et al., 2007). The lack of standardization at the definition stage presents a
problem for learners, teachers, assessors and employers alike. One implicit concern
in the literature is that if there is no standard definition for everyone to work to, how
can the program of training deliver a defined outcome?

Researchers in this field (Gonczi, 1994; Hyland, 1993; Biemans, et al., 2004;
Chapman, 1999; Watson, Stimpson, Topping & Porock, 2002; Cowan et al., 2007;
James, 2001; James, 2002; McMullan et al., 2003) have also noted the problems
with assessment in CBT. One of the main challenges appears to be consistency in
assessment ratings across assessors. However a more complex problem is how
assessors can identify and assess the more ‘intangible’ qualities of an individual,
which are often necessary for a job but are rarely captured in traditional
competency-based assessment of individual units of competence (James, 2001).
These ‘intangible’ qualities are often quoted as ‘the ability of an individual to fit in to
the workplace’ or for a nurse or doctor it may be the ‘compassionate, human side’ of
patient care (Chapman, 1999).

Competency-Based Training in Australia

The concept of CBT was introduced to Australia in the late 1980’s (ANTA, 1999).
Originally competency-based training was limited to vocational training, but
gradually moved into professions as well (such as law – see Gonczi, 1994, and
aviation in 1999, see below). This move has been commented on extensively in the
literature, with authors such as Gonczi (1994) highlighting the benefits of this
approach for professional training. Predominately however, competency-based

78
training remains a vocational training mechanism in most of the world. Many
arguments have been made for the applicability of competency-based training to
vocational training with significantly less support for the concept within professional
training (Hyland, 1994, Milligan, 1998, Leung, 2002). With the exception of Gonczi
(1994) and James (2001, 2002) – in part, there has been very little literature on the
concept of training for professions within Australia using CBT.

Studies in Australia on the introduction of competency-based training have revealed

mixed results, often depending on the domain. Reviews have occurred in nursing,
forestry and fishing, manufacturing, services, construction and agriculture and
teaching (James, 2001; James, 2002; Milligan, 1998; Cowan et al., 2007; Chapman,
1999; Watson et al., 2002). The main problems with the introduction of CBT in
Australia seem to centre on a few key concepts as discussed above, indicating
parallels with the worldwide experience in implementing CBT (e.g. problems with
definitions, assessment, etc. see Gonczi, 1994; James, 2001; James, 2002).

However a few authors have commented on the introduction of more effective

assessment protocols (Gonczi, 1994, McMullan et al., 2003). In particular Gonczi’s
(1994) work on assessment methodology for law society candidates. Here, the
realisation was made that the individuals would be working in a complex, dynamic
and changing environment. The assessment module was therefore comprised of
more traditional approaches to CBT – a written test and peer review/refereeing,
supplemented by a take home case (a mock file) that the individuals had to work
through and a videotaped simulation. The mock file was a complex family law matter
related to a divorce case and required the individuals to demonstrate a range of
knowledge and identify approach strategies to determine and obtain the most
favourable outcome for their client. This assessment approach allowed assessors to
judge candidates on a number of markers as they observed individuals within
realistic scenarios they were likely to encounter in their working life (Gonczi, 1994).
This observation technique helped in areas such as ‘develop relationship with client’
as the assessor was able to observe the individual do so, rather than rely on the
traditional approach of candidates nominating their competency on the basis that
they had a number of interactions with clients.

79
Competency-Based Training in Australian Aviation
The International Civil Aviation Organization (ICAO) recently recommended the use
of CBT for pilot training worldwide (see Annex 1 and Annex 6, ICAO, 2006a,b). For
some nations this will be a new concept; however the Australian Civil Aviation
Safety Authority (CASA) introduced CBT requirements as part of the Day Visual
Flight Rules (VFR) syllabus for private and commercial pilot training in 1999. This
requirement brought private and commercial pilot training in Australia in line with
government policy on vocational education and training.

In March 2003, the competency standards relating to private and commercial pilots
were consolidated into a standard, produced by the Department of Education,
Science and Training, titled ‘TDA03 – Aviation Training Package’. In August 2008,
the competency standards were again modified to ‘AVI08 – Aviation Training
Package’, with a revised Day VFR syllabus that included competency standards in
an abbreviated form (CASA, 2008).

Given that the training syllabus requirements for commercial pilot training has been
in place since 1998, it can be reasonably expected that Australia has been training
commercial pilots under CBT for a number of years now. Indeed, a recent revision
(March 2008) to the Day VFR syllabus has included further elements of competency
that instructors must train for and assessors must rate pilots against. These further
competencies are in the area of non-technical skills and cover the ‘intangible’ skills
pilots require, such as ‘Manage Flight’ (effective lookout, situational awareness,
assess situations and make decisions, set priorities and manage tasks, maintain
effective communications and interpersonal relationships); ‘Threat and Error
Management’ (to recognise and manage threats; recognise and manage errors,
recognise and manage undesired aircraft state), amongst others (CASA, 2008).

It could therefore also be reasonably expected that the evidence for the success of
CBT in Australian aviation must be well documented and easily available to anyone
interested in the field of study. This is not the case. While the regulator has
continued to add requirements to the pilot training syllabus, and while chief flying
instructors and flying schools will say they are operating to a CBT regime, there
seems to be a lack of definitive, objective evidence to back this up. There are a
number of reasons why this may be so, including (but not limited to):

 a lack of understanding on the part of the instructors and flying schools about
what competency-based training actually is/involves;

80
  a lack of understanding by the regulator;
 external pressures and directions on flying schools; and
 a lack of guidance on how to implement a CBT regime (for schools) and how
to assess one (for CASA staff).

An investigation by the Australian Transport Safety Bureau (ATSB) into a midair

collision at Latrobe Valley in 2007 highlighted the lack of CBT at the flying school
involved, including a lack of guidance for instructors relating to sending students
solo, and a lack of a complete and fully implemented CBT system at the flying
school (ATSB, 2010).

The fundamental problem with CBT in aviation (and not just Australian aviation) is a
logical problem relating to time. CBT relies heavily on the concept that individuals
will obtain competency at a given task, or set of tasks, within varying timeframes.
That is, student A may take 2 weeks to become competent in task C, whereas
student B takes 3 weeks. The idea that students will obtain competency in their own
timeframe is a cornerstone of the CBT approach. This concept then may well be at
odds with any approach which specifies a limit of any kind, which aviation training
does (Hunt, 2007). In Australia, this limit on commercial pilots ranges from needing
a minimum of 150 hours total time, to 200 hours total time, depending on the type of
commercial training the student is undertaking. Within the 150 hours total time, a
student must also obtain 70 hours in command (i.e. solo flight without an instructor
on board). While these limits have been modified over time and often are legacies
from military training and civil training after the Second World War, the concept of
having a minimum amount of hours that a student must reach before they are
licensed presents as somewhat against the pure concept of competency-based
training, which is ideally about individuals attaining skill and not just hours.

81
2.2 Exploratory Study

In an exploratory study of pilot hours at commercial pilot licence issue, we examined

688 records for a one year period to determine the overall spread of experience and
examine whether or not there was an effect of flight training organisation (which
conducted the pilot training). All flight training organisations in the study were
conducting training to the CASA Day VFR syllabus requirements and were therefore
operating to a competency-based training system. Descriptive analyses were
undertaken on key flight hour variables available in the licensing data-set, including
1) total hours; 2) instrument flying hours; 3) command hours; and 4) cross-country
command hours, at licence issue. Further, a series of one-sample t tests were
utilised to test for significant variation from the CASA prescribed regulatory minimum
hour requirements under Civil Aviation Regulation CAR5.111. In order to examine
possible effects of a specific Flight Training Organisation’s (FTO) syllabus on the
flight hour experience levels of pilots at licence issue, a sub-set of data was created
which grouped the four flight training organisations conducting airline cadet training.
Given the non-normality of the sample, Kruskal-Wallis tests with Mann-Whitney post
hoc tests were conducted.

82
2.3 Results

Analysis of mean flight hours at licence issue showed significant differences from the
CAR 5.111 minimum requirements, as summarised in Table 2.1. In all instances,
mean hours at licence issue were significantly higher than prescribed regulatory
minima.
Table 2.1: Descriptive Summary of Hours at Licence Issue
Regulatory Mean SD Min. Max.
Minimum

Total Hours 150 189.0 30.9 150 467.8

Command 70 74.4 8.6 70 184.5

Cross-Country Command 20 53.6 10.3 20 138.9

Instrument 10 18.5 10.9 10 95.8

(N=688)

For total flying hours at CPL issue, trainee pilots had accrued on average 39.0 hours
more than the regulatory minimum, t(687) = 33.0, p < .001.

This pattern was repeated for the individual components of the flight-training
program. For command hours at CPL issue, trainee pilots had accrued on average
4.4 hours more than the regulatory minimum, t(687) = 13.5, p < .001.

For cross-country command hours, trainees had accrued on average 34 hours more
than the regulatory minimum, t(687) = 85.6, p < .001.

Instrument hours also followed the trend of exceeding the regulatory requirement,
with an average of 8.5 hours greater than the minimum, t(687) = 20.4, p < .001.

Analysis of the data sub-set of the four training organisations demonstrated variation
in hours at licence issue within, but more evidently between, individual flight training
organisations.

83
84
Figure 2.1: Box Plots of Experience at Licence Issue as a function of Flight Training
Organisation

Analysis of total flying hours indicated a significant effect of Flight Training

Organisation, H(3) = 179.19, p < 0.001. Post hoc analysis using Mann-Whitney tests
indicated all Flight Training Organisation’s were significantly different in hours (FTO
1 v 2 U = 154, p<0.001; FTO 1 v 3 U = 243.5, p<0.001; FTO 1 v 4 U = 63, p<0.001;

85
FTO 2 v 3U = 772, p<0.001; FTO 2 v 4 U = 377.5, p<0.001), except between FTO 3
v FTO 4, which were not significant (U = 483.5, p = .659)

In relation to the command hours component, once again a significant effect of

Flight Training Organisation was found H(3) = 82.73, p < 0.001. In this case post
hoc analysis demonstrated that all FTO’s were significantly different in mean
command hours for candidates (FTO 1 v 2 U = 1829, p<0.001; FTO 1 v 3 U =
1697.5, p<0.001; FTO 1 v 4 U = 226, p<0.001; FTO 2 v 3U = 697, p<0.001; FTO 2 v
4 U = 298, p<0.001; FTO 3 v 4 U = 110, p <0.001).

Cross-country command hours also demonstrated a significant effect of Flight

Training Organisation H(3) = 50.75, p < 0.001. Subsequent post hoc analysis
demonstrated that all FTO’s were significantly different in hours (FTO 1 v 2 U =
2057, p<0.001; FTO 1 v 3 U = 1472, p<0.001; FTO 1 v 4 U = 610, p<0.001; FTO 2 v
3U = 918.5, p<0.001; FTO 3 v 4 U = 324, p<0.012), except between FTO 2 v FTO 4,
which were not significant (U = 640, p = .142)

The results for Instrument hours also showed a significant effect of Flight Training
Organisation on flying hours at licence issue H(3) = 151.23, p < 0.001. In this case,
post hoc analysis indicated that all Flight Training Organisations were significantly
different (FTO 1 v 2 U = 1738, p<0.001; FTO 1 v 3 U = 107.5, p<0.001; FTO 1 v 4 U
= 33.5, p<0.001; FTO 2 v 3U = 176, p<0.001; FTO 2 v 4 U = 134.5, p<0.001; FTO 3
v 4 U = 203, p <0.001).

86
2.4 Discussion

If flight training organisations did manage to implement CBT and were free from
both the regulatory and airline imposed requirements for flight hours of cadets, we
could expect to have seen a much different result from our data. It is plausible that
given the same syllabus but no hours targets, that the spread of hours obtained by
individuals would follow a normal distribution. Therefore we would expect to have
seen outliers at each end, some obtaining full competency in under 100 hours,
some taking up to 250 hours and beyond. If this system was achievable, we could
be confident that the individual had in fact met the necessary requirements to hold a
CPL and could demonstrate consistency of performance, which is a necessity to
ensure validity of performance under CBT. While we believe pilots currently licensed
at 150 hours are competent, we also note that the ones that were licensed close to
the target of 150 hours may have actually been competent before this time.

Based on the results from the exploratory study, it is not possible to judge whether
or not a CBT approach (in its strictest sense) is occurring in Australian aviation
training. The study highlighted an artificial variation in the data, with instances of
tightly clustered hours being indicative of the FTO’s meeting the minimum regulatory
requirements. We also noted a clear variation between FTO’s which appears
indicative of the effect of syllabus on pilot hours. Therefore, from the results
presented here, the variables of 1) regulatory minimum hours, and 2) syllabus
requirements, appear to be of greater influence on flight training hours at CPL issue.

Regulatory minimum hours

The current flight training regime has developed to a point where the minimum flight
hours at licence issue has become a ‘safety net’ that neither the industry nor the
regulator want to alter in case it dramatically alters the safety of the system (for
arguments on overregulation in ultra-safe systems for precisely this reason see
Amalberti, 2001).

This reluctance to examine and alter the system could be one of the underpinning
reasons for some of the greatest criticisms of new training and licensing innovations
in international aviation (see AIPA, 2010). The belief of sections of the aviation
industry that any deviation from what is known (i.e. flight hour minimums) will result
in a loss of system safety (see Sutton, 2005; IFALPA, 2008) coupled with an
87
emotive topic (a reduction in pilot training hours) has resulted in strong opposition to
potential advancements in aviation training (see AIPA, 2010). The reduction in pilot
training hours was recently the subject of an Australian Parliamentary Senate
Inquiry. The Australian and International Pilots Association (AIPA) commented in
their submission that “AIPA is adamant that practical and appropriate piloting
experience is a significant risk mitigator that cannot be replaced by theory or
simulator training, only supplemented” and “AIPA is concerned that the widespread
shift in emphasis on airline pilot recruiting from those pilots who had several
thousand hours of flight experience in General Aviation to cadet pilots whose only
experience is in a training school environment has and will continue to increase the
risk of an aviation accident” (AIPA, 2010, p3).

However part of these criticisms cannot be based on the reality of CBT, as it has
never truly existed in aviation in the manner it was designed for. If a student has
become proficient or competent in less than 150 hours total time, there is no way for
them to have been legally licensed. As such, their training must continue until 150
hours is reached. Not only is this counter-intuitive in the world of CBT, it is wasting
the student’s time and money. It also means that it is impossible to see the spread
of individual competency within the licensing data, as the range will be truncated by
the numbers at the lower end representative of the minimum hours required by the
regulator and not the performance of the student/s.

Syllabus requirements
Another variable which will affect the number of hours a student has by the time
their licence is issued is the influence of the flight training school syllabus and it is
reasonable that this will be the biggest influence on the fluctuations in the data.
Many schools follow the Day VFR syllabus as a basis and have amended it to
develop their own syllabus over time. One of the reasons for the largest variations in
syllabus between training schools will be dictated by the client. In Australia this will
depend on what the school is training the students for. Many Australian schools
conduct training for overseas airlines and as such, their syllabus will be affected by
the requirements of that airline or their countries’ regulatory authority. A survey of
two flight training organisations conducting flight training during our study period
shows that both organisations were training airline cadets for international airlines;
however one also trained university students gaining a degree in aviation. The

88
feedback from one of the flight training organisations was that the fluctuation in
instrument flight hours in their data was likely in response to client demands and the
necessity for the cadets to hold a multi-engine command instrument rating (MECIR)
following their CPL issue. This FTO found that adding instrument training during the
CPL stage helped their students at the MECIR training time, despite MECIR
occurring after CPL issue. The same organisation noted that their student’s
command time did not vary from the CASA syllabus requirements as they were
sending students out solely to ‘make up the numbers’ before the flight test. This sort
of syllabus variation by an FTO will vary the figures of these students at licence
issue but it is hardly seems reflective of individual competency.

The training sector of the Australian aviation industry has taken measures to
introduce competency-based training, particularly those conducting ab-initio cadet
pilot training. However there are difficulties for the FTO’s trying to achieve this goal.
They appear to be caught between the requirement to meet the minimum hours
imposed by the regulator for the issue of a commercial licence and the demands of
the airline they are training the cadets for, in particular the airline’s syllabus
requirements. The results of our study demonstrate consistent evidence of
clustering of flight hours around these minimum hours requirements from the
regulator and also due to the syllabus of the FTO which may well be influenced by
contractual obligations between the FTO and the airline.

The effect of this situation appears to have set up a conflict between the desire of
FTO’s to incorporate true CBT and the requirement to continue with obtaining a
known benchmark that requires a specific flight hour number to be obtained by a
cadet before they can become licensed.

89
2.5 Conclusion

Competency-based training is a complex area that appears to be difficult to

implement in practice, due to a number of issues in definition, implementation and
assessment. This article has given a brief overview of the current situation relating
to competency-based training in Australian aviation. Despite a clear regulatory
requirement for CBT to be used in flight training in Australian aviation since 1999,
there appears to be little evidence that the system in place is a true competency-
based system. This may well be due to the difficulty for FTO’s in implementing a
CBT system with minimum regulatory requirements for obtaining a flight crew
licence and any airline syllabus requirements in addition to these regulatory
minimums. The very concept of having to meet a minimum amount of hours is at
odds with the underlying philosophical concept of CBT.

It is therefore important that aviation regulators base decisions regarding flight

training and flight training innovations on existing data and not an ideal view of what
should be happening in the aviation training system. This study should serve as an
indicator of what is happening in Australian flight training organisations and act as a
starting point for further study in this area. As such regulators need to examine and
review the system they have, not the system they think they should have under the
current regulatory framework. Only then will we get a clear, accurate picture of the
existing aviation training system, which can then be used as a starting point to build
upon. Using the data from further study we can develop the most applicable,
relevant and useful pilot training system to meet the future needs of the aviation
industry.

90
 Chapter Three - Flight Hours and Flight Crew Performance in
Commercial Aviation2

2
Todd, M.A. & Thomas, M.J.W. (2012). Flight Hours and Flight Crew Performance in Commercial
Aviation. Aviation, Space, and Environmental Medicine, 83(8), 776-782.

91
3.1 Introduction

The concept of experience is currently the subject of much debate within the
aviation industry. With a global pilot shortage, a number of recent accidents and the
introduction of new training pathways for airline pilots, the question of whether or not
a ‘low hour’ pilot has sufficient skills to perform on the flight deck of modern aircraft
is particularly relevant. New legislation introduced in the US has raised the number
of hours needed by First Officers flying for an airline to 1500 (United States
Congress, 2010). In Australia, a senate inquiry on pilot training was held, which
considered introducing the same amendment in Australia (Parliament of Australia,
2010).

The importance of flight training is widely recognised by airlines, regulators, safety

investigators and the aviation industry as a primary tool in maintaining the ultra-safe
system of commercial aviation. However the focus on hours attained at the time of
licence issue appears to give little regard to individual skill development and
learning requirements, which may have been somewhat overlooked in the syllabus.
There is an implicit suggestion in the legislation that if an individual has attained the
required hours to be issued with a pilot licence, they also possess the requisite skill.
In many debates and discussions that have been held regarding low hour pilots, the
one thing that remains constant is the view that hours equals experience (Sutton,
2008; MacFarlane, 2003; ALPA, 2009).

While we do not examine the broader pilot training regime in this study, it is worth
noting that experience (hours) is emphasised in both training, and later the
selection, of pilots for airline entry. As such, the present study examines the
question of pilot experience, specifically the variables of total flying hours. It seems
that within the aviation industry experience is used as a measure in and of itself,
which is assumed to be a perfect correlate to performance. This is certainly the view
presented during submissions to the Australian Senate inquiry. It should be noted
that this approach focuses on hours only and not skill or competence. In fact, these
general arguments appear to use hours as a surrogate for competence. This
synonymous use of experience and hours within industry will be used in this article
as the basis to explore differences in pilots’ technical and non-technical skills.

Previous studies have explored the relationship between experience and

performance, including the role of non-technical skills such as situational awareness
92
and decision making in aviation (Wiegmann, Goh & O’Hare, 2002; Wiggins &
Bollwerk, 2006; Wiggins & O’Hare, 2003). The importance of experience as a
determinant of performance has been widely recognised and studied, particularly
the impact of performance on accident rates amongst pilots. The results of these
studies have been divided, with some finding a significant effect of experience (Goh
& Wiegmann, 2002; Wiegmann, Goh & O’Hare, 2002; Wiggins & O’Hare, 2003) and
some finding none (Groff & Price, 2006; O’Hare & Chalmers, 1999).

One area that has been studied extensively in the literature is that of accidents
involving flight into adverse weather by pilots not qualified to fly in instrument
conditions (that is, flight with sole reference to instruments such as when in cloud,
instead of flight using visual cues outside the aircraft). Specifically, the aspects of
decision making and experience of the pilot in these kinds of accidents have been
reviewed with the result that less-experienced pilots were more likely to be involved
in these types of accidents (Goh & Wiegmann, 2002; Wiegmann, Goh & O’Hare,
2002). In comparing expert and novice performance in perception of weather
images, Wiggins and O’Hare found a significant effect for expertise and judgement
(Wiggins & O’Hare, 2003). They classified experts as pilots with greater than 1000
hours cross-country flying experience (mean 5520.33 and mean total flying
experience of 6873.37 hrs) and novices as under 1000 hours (mean 281.71 and
mean total flying experience of 801.17 hours).

Another study, which examined General Aviation accidents in degraded visibility,

found that while pilots in the younger age group were involved in a smaller
proportion of accidents, both number of years as a pilot and total number of flight
hours flown were not significant (Groff & Price, 2006). The effect of low flight hours
experience on accident risk in New Zealand pilots in General Aviation was explored,
specifically the concept of a special vulnerability for pilots with around 100 to 200
hours total time compared to the rest of the pilot population. The same comparison
was conducted for pilots with around 100 to 300 hour total time compared to the rest
of the pilot population. The authors found no evidence of an increased risk of
accident involvement for the low total flight hour groups (O’Hare & Chalmers, 1999).

In their review of accident causation and the effect of age, experience and gender,
Bazargan and Guzhva defined air transport pilots with 300-2000 hours of total time
as “...newly licensed airline transport pilots”; 2000-5000 hours of total time as
“...moderately experienced airline transport pilots”; and 5000 hours or more total

93
time as “the most experienced pilots”(Bazargan & Guzhva, 2011). The results of this
study indicated that “the ratio of pilot error caused to total accidents steadily
declines with pilot experience” with significant differences between experience
levels, showing low experience pilots were more likely to make errors that resulted
in an accident although more experienced pilots were more likely to be involved in a
fatal accident (Bazargan & Guzhva, 2011). It should be noted that their sample also
included pilots with less than 100 total hours and 100-300 total hours, which by
definition does not cover the air transport sector and would be representative of
private and commercial pilots only and it was these groups which had the highest
error rate.

An examination of professional pilots and their crash risk as a function of age and
flight experience found that pilots with less than 5000 hours were “nearly twice as
likely as their more experienced counterparts to be involved in a crash” (Li, Baker,
Grabowski, Qiang, McCarthy & Rebok, 2003). The authors did note that crash risk
appeared to stabilise once 10,000 hours total flight time had been achieved. These
pilots were all operating in the commuter or air taxi sectors of aviation (i.e. not major
airlines), a sector which carries a 10-45 times higher fatal crash rate per pilot flight
hour than major airlines (Li et al., 2003).

Another area that has been studied extensively for the effect of experience on
performance is driver training and licensing. Experience is just one of the variables
examined in the area of driver performance, with age also being considered for
effect. In the case of driver performance, age and experience are often closely
aligned.

Studies in the area of driver performance have examined the effect of experience on
decision making, risk taking and interaction with automation (Dingus, Hulse,
Mollenhauer, Fleishman, McGehee & Manakkal, 1997; Fisher, Laurie, Glaser,
Connerney, Pollatsek, Duffy & Brock, 2002). A driver study from the UK examined
the effect of age, experience and time of day on young driver accidents and found a
correlation between low experience (experience in this case measured as years of
licensure) and accident rate, although the authors state that age was a major
contributor to this (Clarke, Ward, Bartle & Truman, 2006).

The majority of the studies on experience and performance in the aviation industry
have focused on general aviation (GA) or non-major airlines. Whilst these studies
have shown strong links between experience and performance, the relative
94
influence of training and post-training experience in largely expert, or “non-novice”,
populations has not been explored to the same extent, neither has performance in
an airline environment been reviewed in this manner. Moreover, while new training
regimes in aviation have focussed on competency, rather than the measure of flight
hour experience, there is still a focus in the aviation industry on flight hours, which
they appear to use as a surrogate for competency (ICAO, 2006a).

Given the regulatory emphasis placed on Threat and Error Management (TEM) by
the International Civil Aviation Organisation (ICAO, 2006a; ICAO 2006b) and
Australia’s Civil Aviation Safety Authority (CASA, 2008), and the introduction of new
training methods which rely, in part, on the introduction and use of TEM for low hour
pilots, there is a need to determine whether or not flight experience (flight hours)
does affect TEM performance.

The question of what are the minimum hours required to ensure adequate
performance of pilots has come to prominence in both the United States and
Australia, particularly in the wake of the accident involving a DHC-8 aircraft operated
by Colgan Air, which crashed while on approach to Buffalo, New York, U.S. in
February 2009 (NTSB, 2010). The high profile Air France Flight 447 accident
involving an A330 in July 2009 has further fuelled both public emotion and industry
debate about pilot training and experience. One of the direct outcomes from the
Colgan Air accident was the introduction of new legislation in the US, which
increased the number of hours needed by First Officers in order to fly for an airline
to 1500 (United States Congress, 2010). A similar change to legislation was
discussed in Australia during a senate inquiry on pilot training (Parliament of
Australia, 2010).

In summary, previous research in the area of pilot performance as a function of

experience, as highlighted above, fails to provide conclusive evidence to inform the
current debate with respect to performance, safety and risk. Previous studies have
tended to examine non-normal events, and in particular, aircraft accidents. Many of
these studies have focused only on the non-technical and cognitive areas of pilot
performance as opposed to the physical, manipulative skill required of pilots. The
airline environment requires a mix of both cognitive and manipulative skill for
technical and non-technical performance, with a cognitive element to technical
performance, such as interaction with systems and automation (Parasuraman &
Mouloua, 1996) to accurately fly the aircraft to the required standard.

95
In this article we will highlight the results of the exploration of the current industry
beliefs with a view to establishing if there are significant differences, in both
technical and non-technical performance of Captains and First Officers as a function
of the hours (or experience) they have. This study aims to examine whether there
are any systematic differences in crew performance evident as a function of flight
hour experience. The hypothesis to be examined is that there will be a difference in
performance based on flight hours, and specifically, that both the technical and non-
technical metrics will demonstrate better performance for the high-hour pilots.

96
3.2 Method

Subjects
Participants were drawn from the pilot group of one high capacity Regular Public
Transport (RPT) operator during commercial airline flying of short-haul jet
operations, with a total of 302 normal flight sectors observed. All flight sectors were
crewed by two pilots, a Captain and a First Officer, with 302 crews observed. Due to
strict anonymity involved with data-collection and the naturalistic nature of the field
observations, it was not possible to determine if an individual pilot was observed
more than once, as is the case of all LOSA observations. Flight sectors were
chosen to provide a stratified sample representative of the airline’s normal route
structure. Flights ranged from 30 min to 5 h, and observations were made during
both day and night operations (Thomas & Ferguson, 2010). All pilots were given the
opportunity to opt out of the study, with this occurring in approximately 1 in 50
flights, and all data were collected under non-jeopardy conditions and de-identified.
All pilots held, as a minimum, a commercial pilot licence and no participants held a
multi-crew pilot licence. The training pathway of the pilots was not collected.

In order to compare the performance of pilots based on their experience, four

groups were created to reflect both low and high hour First Officers and Captains.
First Officers were grouped into those with 1500 hours and below, and those above
1500 hours. 1500 hours was chosen as the divide based upon the current legislative
changes in the United States, and the fact that 1500 hours total time is necessary
for pilots to hold an Air Transport Pilot Licence (ATPL). The Captains were grouped
below and above 5000 hours total time. This value was selected after consultation
with training Captains and is consistent with the average amount of hours for
Captains in the sample, as well as with the earlier study listing 5000 hours as “the
most experienced pilots” (Bazargan & Guzhva, 2011). These thresholds were
chosen as they are used by industry as a standard for promotion for Captains and
often for entry of First Officers and yet there is little or no scientific evidence behind
their use, as noted in the literature.
The data for this study were collected as part of a Line Operations Safety Audit
(LOSA) (Helmreich, Klinect & Wilhelm, 2001; Klinect, Murray, Merritt & Helmreich,
2003). LOSA is an observational methodology developed by the University of Texas
to collect safety data during normal flight operations in order to evaluate the safety
of the system, with the stated purpose “...to capture operational reality” (Klinect,

97
Murray, Merritt & Helmreich, 2003). To do this, trained observers sit in the jump seat
of a flight deck and collect data against various metrics, such as the operational
environment, threats, crew performance and so on. As such, the methodology is
predicated, in part, upon anonymous and confidential data collection and an
emphasis that LOSA is not a check of individual performance. That is, the LOSA
observers do not give feedback/de-briefings on crew performance, they are there
only to observe and collect data on normal operations in order to provide a snapshot
of the performance of the system (Klinect, Murray, Merritt & Helmreich, 2003).

The study protocol was approved by the University of South Australia Human
Research Ethics Committee.

Design and Procedure

The study was a between-groups factorial design in order to test the hypothesis
relating to the hours experience of the crew and their performance.

Subjects were observed during normal flight operations by a team of 15 trained

expert observers on the flight deck trained in the LOSA methodology with a mix of
piloting experience. Over a 6-wk period of data collection, observers were assigned
flights to observe according to the stratified sample described above. Observers
made detailed notes pertaining to crew performance from the pre-flight preparation
through to the end of the flight, including notation of stabilised approach criteria at
the 1000’ and 500’ above field elevation (AFE). At the completion of the flight,
observers completed structured forms that prompted analysis of each external
threat and crew error observed. These forms also prompted the recording of crew
demographic data and the assessment of the crew’s non-technical skills across the
flight. Forms were then returned directly to the University of South Australia and
entered prior to a process of data cleaning to ensure accurate classification of
threats and errors. All observers attended a 4 day training course that introduced in
detail each of the aspects of data collection and included practice observations as
well as assessment of interrater reliability. Interrater reliability and observer
calibration was established utilising standardised videos which the observers group
rated during training. The interrater reliability was established during training using
the mean within-group agreement (rwg) statistic (Yule, Flin, Maran, Rowley,
Youngson & Paterson-Brown, 2008). The rwg values ranged from 0.73 to 1.0 across
the markers in performance, the scale being 1 - perfect agreement, 0 - perfect

98
disagreement and anything greater than 0.7 being acceptable. The operator
involved in the LOSA followed the aircraft manufacturer’s standard operating
procedures and checklists, which all observers were familiar with.

From the 302 observations, a total of 287 records were complete and suitable for
analysis. The observers and pilots were blind to the specific hypothesis regarding
flight time and performance. Observers were also blind to the hour experience of the
pilots at the commencement of the flight.

Measures
To examine individual performance against technical measures, the experience
groups of low hour captains (<5000 h), high hour captains (>5,000 h), low hour first
officers (<1,500 h) and high hour first officers (>1,500 h) were used. The
independent variable was the experience group and the dependent variable was
performance against the stabilised approach criteria.
For non-technical performance, the flight crew as a whole were rated. With respect
to crew composition for non-technical performance, four groups were created, using
the same thresholds of experience mentioned above, which were high hour captain,
low hour captain, high hour first officer and low hour first officer. The independent
variable was again experience group, with the dependent variable being
performance against the non-technical skills criteria. This enabled specific analyses
of sectors where both crewmembers were in ‘low’ experience groups.

Individual Technical Performance: The primary markers for technical performance

were a set of stabilized approach criteria. The approach-and-landing phase of flight
is one of the only times pilots have the opportunity to manually fly the aircraft, as
most of the flight is conducted using automation (e.g. autopilot and auto-throttles).
According to the Flight Safety Foundation’s Approach-and-landing Accident
Reduction tool kit, unstabilized approaches are “frequent factors in approach-and-
landing accidents” (Flight Safety Foundation, 2000). The FSF identified three
essential parameters that must be stabilized for a safe approach. These are: aircraft
track; flight path angle; and airspeed. In addition, they also recommend the aircraft
is in the correct landing configuration, sink rate (or vertical speed) is no greater than
1,000 foot per minute and all briefings and checklists are completed. The six criteria
outlined by the FSF at both 1000’ AFE and 500’ AFE were used to benchmark crew

99
performance. These were: airspeed; vertical speed; appropriate thrust set; approved
landing configuration; on proper flight-path; and briefings and checklists completed.

The pilot flying (PF) was assessed on whether they met each stabilised approach
criteria at the 1000’ or 500’ AFE target.

In addition to this, information was collected on the altitude at which the autopilot
was disconnected on approach by the pilot flying.

Crew Non-Technical Performance: Observers rated crew on NTS using a

performance rating scale against four NTS criteria: Communication, Situational
Awareness, Task Management and Decision Making. Each of these criteria has
subsections, resulting in 16 NTS markers. These are built on the original 12 markers
developed by the University of Texas, as detailed in the Federal Aviation
Administration (FAA) Advisory Circular on Line Operations Safety Audits (FAA,
2006). See Table 3.2 for the NTS criteria subsections.

The flight crew were given a rating from 1-4 (1 is poor, 2 marginal, 3 standard and 4
outstanding) against each NTS performance criteria for five phases of flight – Pre-
departure; takeoff; cruise; descent, approach, land; and taxi.

These ratings measured the crew performance of NTS in each phase of flight, that
is, they were not rated as individuals, but rather as a crew, with overall ratings
assigned to each criterion in addition to each subsection.

Threat and Error Management: With respect to Crew’s TEM performance,

observers used the TEM Model, developed at the University of Texas as a
standardized and highly structured method to collect operational performance data
(Helmreich, Klinect & Wilhelm, 2001; Klinect, Murray, Merritt & Helmreich, 2003).
This method allowed for classification of the crew’s management of operational
threats, defined as aspects of normal flight operations that have the potential to
impact negatively on safety (e.g. adverse weather). It also allowed for analysis of
crew errors (Helmreich, Klinect & Wilhelm, 2001). The crew’s performance was
represented by several global measures which included: 1) proportion of threats
effectively managed; 2) proportion of threats that were inconsequential; 3) the
number of errors made; 4) proportion of errors effectively managed; and 5)
proportion of errors that were inconsequential.

100
Statistical Analysis

For the technical performance dependent variables of stabilised approach criteria

and autopilot disconnect height, analyses were performed for the role of pilot flying.
Chi square analyses were utilised for categorical variables and one way ANOVA
utilised for continuous variables. For the non-technical dependent variables,
analyses were performed at the crew level. Statistical tests were interpreted using
the alpha level of 0.05 for statistical significance. Post-hoc testing using the LSD test
was used for the non-technical data to determine the significance of variance
between groups.

101
3.3 Results

A total of 287 sectors of normal flight operations were analysed. During these
sectors a total of 845 (Mean = 3.21, SD = 1.71) threats and 811 (Mean = 3.14, SD =
1.96) errors were observed and analyzed. These findings are in line with typical
observations of normal commercial flight operations (Helmreich, 2000). The most
frequently occurring threats included those relating to adverse weather, air traffic
control requirements, and ground handling events. Common error types included
aircraft handling, management of aircraft systems and instruments, and
management of aircraft automation.

For Captains, 15% were classified as low hour (<5000 hours, n=24) and 85% were
classified as high hour Captains (>5000 hours, n=137). For First Officers, 17% were
classified as low hour (<1500 hours, n=21) and 83% were classified as high hour
(>1500 hours, n=103).

For the stabilised approach criteria, at both 500’ and 1000’, no statistically significant
difference was found between, or within, Captain and First Officer experience.
Details of the results for stabilised approach performance at 500’ can be seen in
Table 3.1. The only measure which showed any difference (and is not part of the
stabilised approach criteria) was the height at autopilot disconnect, with low hour
First Officers disconnecting at a significantly lower average altitude of 655’,
compared with 1168’ for high hour First Officers, F(1,111)=6.396, p=.013.

102
Table 3.1: Stabilised approach performance (at 500’), autopilot disconnect height, and TEM
performance by experience group
Stabilised approach Capt (L) Capt (H) χ2 sig. FO (L) FO (H) χ2 sig.

Airspeed 100% 97.7% .565 .452 100% 94.8% 1.143 .285

Vertical speed 100% 96.2% .954 .329 100% 95.8% .906 .341

Appropriate thrust 100% 96.2% .939 .332 100% 95.8% .906 .341

Landing configuration 100% 98.5% .359 .549 100% 96.9% .674 .412

On flight path 95.8% 92% .911 .823 100% 86.4% 3.218 .200

Briefings & checks 100% 99.2% .177 .674 100% 99% .214 .644

Autopilot disconnect Capt (L) Capt (H) F sig. FO (L) FO (H) F sig.

Height at autopilot disconnect 949.6 1198.0 2.292 .132 655.0 1168.2 6.396 .013

Threat & Error Management Capt (L) Capt (H) χ2 sig. FO (L) FO (H) χ2 sig.

Threat response (detect & manage) 81.1% 77.8% 1.224 .747 77.9% 78.3% .504 .918

Threat outcome (inconsequential) 87.7% 85.5% .373 .541 87.2% 85.5% .312 .576

Error response (detect & manage) 54.2% 46.4% 3.617 .306 58.5% 61.8% .387 .943

Error outcome (inconsequential) 95.8% 93.1% 1.058 .589 92.7% 94.2% .326 .849

Percentage values for stabilised approach criteria refer to the proportion of flight sectors where the criteria were met. Percentage values
for TEM metrics refer to the proportion of flight sectors where the criteria in parentheses were met.

Against the non-technical criteria, the group with a low hour Captain and a low hour
First Officer were significantly different to the other three groups against the marker
cooperation (p=0.002). For the group with a high hour Captain and low hour First
Officer, the significant difference was against the monitoring and cross-check
marker (p=0.020) against the group with high hour Captains and high hour First
Officers only, as highlighted in Table 3.2.

103
Table 3.2: Non-technical skills criteria and performance (mean, SD) by experience group
Variables Low (C) & Low (C) & High (C) & High (C) & F sig.
Low (FO) Low (FO) High (FO)
High (FO)

Communication

Communication Environment 2.80 (0.45) 2.93 (0.13) 2.84 (0.26) 2.95 (0.29) .849 .470

Leadership-Followship 2.95 (0.10) 2.85 (0.31) 3.02 (0.35) 2.93 (0.34) .484 .694

Inquiry 2.95 (0.10) 2.82 (0.32) 2.88 (0.15) 2.86 (0.33) .180 .910

Assertiveness 2.56 (0.88) 2.94 (0.13) 2.78 (0.27) 2.85 (0.30) 1.53 .214

Cooperation 2.50 (1.00)* 2.98 (0.06) 2.82 (0.29) 2.98 (0.18) 5.48 .002

Statement of Plans & Change 2.93 (0.12) 2.98 (0.06) 3.00 (0.37) 2.96 (0.19) .129 .943

Situation Awareness

Vigilance 3.00 (0.00) 2.79 (0.32) 2.88 (0.18) 2.92 (0.32) .619 .604

Monitoring and Cross-Check 2.85 (0.13) 2.81 (0.31) 2.69 (0.31)* 2.90 (0.18) 3.446 .020

Task Management

Briefing and Planning 2.56 (0.88) 2.98 (0.07) 2.95 (0.13) 2.94 (0.32) 2.128 .102

Workload Management 3.00 (0.00) 2.86 (0.31) 2.93 (0.10) 2.88 (0.34) .278 .841

Workload Assignment 3.00 (0.00) 3.00 (0.00) 2.91 (0.15) 2.93 (0.33) .209 .890

Automation Management 2.95 (0.10) 2.85 (0.31) 2.80 (0.25) 2.86 (0.29) .289 .833

Fatigue & Stress Management 3.00 (0.00) 3.00 (0.00) 2.98 (0.07) 2.90 (0.33) .413 .744

Decision Making

Contingency Planning 3.00 (0.00) 2.91 (0.18) 2.96 (0.10) 2.97 (0.29) .115 .951

Problem Identification 3.00 (0.00) 2.73 (0.37) 2.91 (0.19) 2.98 (0.24) 2.218 .093

Evaluation of Plans 3.00 (0.00) 3.00 (0.00) 3.00 (0.00) 2.95 (0.14) .618 .606

* indicates significance in post-hoc testing

Against the TEM metrics, no statistically significant difference was found between
the low and high hour pilot groups, as highlighted in Table 3.1.

104
3.4 Discussion

There was no statistically significant difference between the performance of

Captains and First Officers against the stabilised approach criteria used during the
observations. This is not entirely unexpected as most airlines have developed
stabilised approach criteria and procedures, which all flight crew are regularly
assessed against, both as the pilot flying and as the pilot not flying. There was also
no difference in TEM performance and very little difference in the non-technical
skills performance between the experience groups of Captains and First Officers.

Although there was a significant variation in autopilot disconnection height observed

for low experienced First Officers against high hour First Officers, there was no
difference observed in the safety of the flight against the stabilised approach criteria
for the low hour group.

There are a number of reasons likely as to why the increased use of automation is
particularly noticeable in this group of pilots. For instance the increased reliance on
automation flying the aircraft to a lower altitude during approach may be indicative of
the lower experience of the group, that is, ‘expert’ performance has not yet
developed (Bazargan & Guzhva, 2011; Rasmussen, 1983; Rasmussen, 1987). As
such, these First Officers may be relying on automation to manage the flight for a
longer period of time, thereby reducing the time spent manually hand-flying, than
their more experienced colleagues. This would allow the First Officers more time to
conduct the other necessary tasks during approach, and would also free up their
cognitive capacity to do so, which would be a beneficial flight management strategy.

The increased altitude for disconnect for the more experienced First Officers may be
an indication of the automaticity of their actions during approach (Rasmussen,
1983), and that their level of cognitive investment and oversight is much lower as a
function of greater experience and expertise (Dismukes, Berman & Loukopoulos,
2007). It is equally possible that the high hour First Officers are choosing to
disconnect at a higher altitude in order to build up hand-flying time as a way of
gaining experience in preparation for upgrading to Captain. The authors feel both
scenarios are equally valid and neither present any detriment to the safety of the
flight.

105
Of interest is the lack of significant variation in TEM performance within the Captain
and First Officer groups. The performance of low hour pilots in each group (less
than 1500 hrs for First Officers and 5000 hrs for Captains) did not differ from the
performance of high hour pilots in these groups.

However against the non-technical criteria, there was an effect of experience for
First Officers on two measures. In crews with a low-hour captain and a low-hour
First Officer, the cooperation of the crew was scored as lower than the other groups.
This could be indicative of the low-hour First Officer’s being new to the flight deck
and not yet familiar enough to work in close cooperation with the Captain. It may
also be due to the low-hour Captains having recently upgraded and may be
indicative of them adjusting to the new role. Against the criteria of monitoring and
cross-check (which sits under situation awareness), the crews with a high-hour
Captain and a low-hour First Officer rated lower, possibly suggesting that low-hour
First Officers are not yet familiar enough with operations to monitor and cross-check
as well as their high-hour colleagues. It may also be a function of high-hour
Captains not completing the monitoring and cross-check functions as thoroughly as
their colleagues, which may in part be due to managing the low-hour First Officer.
However it is important to note that the variations in these two metrics were the only
two variations in a large array of performance assessment. Furthermore, the
variation is of a relatively small magnitude, with these scores being approximately
10% lower. Despite the significance, which may well be from chance, there is very
little difference between the hour groups, and overall no real difference in
performance in the dataset. We would caution the reader of the importance of
ensuring that minor differences are not be overstated in the context of such a very
large number of non-significant findings.

There was no evidence in our study that low hour pilots were systematically different
or less able in their performance when compared to their more experience
colleagues, despite the arguments often quoted in any discussion of pilot training
and selection.

This study was a naturalistic study, which by its nature means the authors did not
have a level of experimental control that you would find in a laboratory study and
therefore the results will not be sensitive to subtle variations. However, during
normal line operations, we are not seeing any major difference in performance
based on experience, which begs the question do we need an experimental

106
simulator study to see the same thing? One benefit which we believe exists in
running a simulator study on experience versus performance is in exploring the non-
normal operations, some of which are only possible to safely explore in a simulator.
One of the limitations of the current study is a limitation of LOSA – it provides a
snapshot of normal operations. As such, tools such as Flight Operations Quality
Assurance (FOQA) data monitoring programs may give a more accurate overview of
stabilised approach performance during normal operations in an airline, in particular
the ability to use FOQA data to monitor the safety and efficiency of operations
(Chidester, 2007). While the LOSA snapshot of normal operations is useful, it may
be of further benefit to explore the same hypothesis but with tighter experimental
control and under non-normal flight conditions. This may go some way to answering
the question of whether or not low hour First Officers are performing less well than
their more experienced cohort and if they are performing well enough to be
operating in an airline environment with minimal total flight hours. A further limitation
of field based LOSA observations is that some observers may have been known to
the crew they were rating.

Another limitation of the current study is that the pilot flying may have been
dynamically controlled by the crew, based on the perceived complexity of the sector
and experience of the crewmembers. In that sense, there may have been some
compensatory behaviour by the crew that may have been the result of an informal
management strategy of inexperienced crewmembers, a type of “performance
protection” put in place by Captains to ensure that the harder sectors were flown by
the more experienced pilot. While there is no evidence of this, we also cannot
discount it.

In conclusion, the findings of this research provide concrete evidence to inform

legislators, regulators, safety groups, pilots and industry in the on-going debate
surrounding pilot hours and inferred performance. Given our current flight training
and checking regimes, this research suggests that the performance of a pilot
certainly does not differ dramatically at 1500 hours, when compared to 1499 hours
or less. There is a continued need for scientific rigour, rather than political
commentary, to inform the debate on commercial pilot training and licensing, in
particular the individual differences that make up the competence of a pilot instead
of adherence to an arbitrary threshold that might somehow ensure performance and
therefore safety.

107
 Chapter Four - Pilot Experience and Performance in an Airline
Environment3

3
Todd, M.A. & Thomas, M.J.W. (2013). Pilot Experience and Performance in an Airline Environment.
Aviation Research investigation AR-2012-023. Australian Transport Safety Bureau, Canberra, ACT.

108
4.1 Introduction

Significant debate has occurred within the aviation industry regarding the issues of
pilot training and experience, particularly with regard to the introduction of the Multi-
crew Pilot Licence (MPL) by the International Civil Aviation Organisation (ICAO,
2006). The main concern being presented by sectors of the industry that are not in
favour of the MPL concept is the possibility that low-hour pilots are not as competent
as their high-hour peers.

As part of this debate, two main issues are currently attracting attention and
discussion. The first is a focus on the hours a pilot has accumulated, generally with
an inherent assumption that these hours ensure a level of individual skill. The
second is a focus on competency, both as an assumed result of flight hour
accumulation and because competency-based training is quoted as the cornerstone
of aviation training, in particular with regard to MPL. These issues and assumptions
have received considerable attention following two high profile international aviation
accidents in 2009, one involving a Bombardier DHC-8 at Buffalo, New York, and the
other an Airbus A330, operating as Air France flight 447 (AF447), en-route from Rio
de Janeiro, Brazil to Paris, France (NTSB, 2010; BEA, 2012).

The AF447 accident highlighted the issue of competency and, in particular, the
relevance of current simulator and non-normal operations training for pilots, as well
as raising issues around automation use and systems knowledge, both of which are
subsets of pilot competency. Of interest is the fact that the flight crew of AF447 had
a high number of flight hours, which appears at odds with the assumption that high
flight hours will ensure adequate performance in a non-normal event.

In response to the DHC-8 accident, which was viewed in part as being the result of
an hours/skill issue, the United States Congress enacted legislation to increase the
amount of flying experience that first officers would need, in order to enter an airline,
to 1,500 hours.

The Australian Senate held an inquiry into pilot training (Parliament of Australia
Senate, 2010) at the end of 2010 to explore the need to enact similar legislation in
Australia. In response to the Australian inquiry, submissions from flight training

109
organisations and airlines highlighted the importance of cadets as a recruitment
option for the airlines, a system that has been in place around the world for a
number of years. Cadets are pilots who, generally, have limited or no prior flight
training prior to starting their cadetship, and are trained specifically for an airline at a
flight training organisation. After training, these pilots enter the airline as a second or
first officer, depending on the airline and its requirements.

Those supportive of cadet schemes emphasise the ability of the airlines to ‘mould’
the pilots to the airline environment, thereby shaping their behaviours and instilling
the required performance from the start of their training. Part of the justification from
ICAO in relation to the introduction of MPL also highlighted these benefits, along
with the use of competency-based training as a mitigator of the risks associated with
changes in the training program, and the introduction of multi-crew cooperation
techniques such as crew resource management (CRM) and threat and error
management (TEM) as strong benefits of the changed program.

Those who are not supportive of cadet schemes are of the view that in order to be a
successful airline pilot, an individual needs to have accumulated flight hours either
with the military, in general aviation or other (often smaller) airlines, prior to entering
a major airline as a second or first officer.

Despite the on-going debate and regulatory guidance on pilot training, which
outlines the minimum hours required and the areas that must be covered, there is
very little research on the differences, if any, between pilots in airlines who have
entered from these various pathways, including as a cadet. While there are some
studies which have examined the effect of flight hours on accident risk; judgement;
ability to interpret weather and one on the effect of flight hours on performance in
normal flight operations, there is very little looking at performance in non-normal
scenarios, specifically as a function of both flight hours and entry pathway. In this
report, ‘entry pathway’ refers to the difference between cadets and direct entry
pilots. Direct entry pilots enter an airline having already been licensed and have
generally accumulated a number of flight hours. Direct entry pilots typically build up
these hours in the military, general aviation (non-airline flying) or other airlines.

110
This report differentiates between normal events, which are routine, expected
events that do not require additional training or specific targeted intervention by a
pilot; and non-normal events, which involve a non-routine event or problem with the
aircraft, its systems, other crew, or the environment, which is not routine and is
outside of what is expected in a ‘normal’ flight.
The aims of this study are to:
(1) test for a difference in performance in the simulator proficiency check for normal
and non-normal operations, based on the entry pathway to the airline
(2) test for a difference in performance in the simulator proficiency check for normal
and non-normal operations, based on the hours experience of the pilots
(3) compare the results with the authors’ previous study (Todd & Thomas, 2012), in
examining if there are differences in pilot performance in the line checks based on
entry pathway or hours.
This study will also address some of the limitations of the previous study, which did
not consider simulator proficiency data and entry pathway.
It can be expected that if the concerns of sectors of the industry relating to low-
hour/cadet pilots are valid, that high-hour/direct entry pilots will perform better. If
ICAO’s stance on MPL and the airlines’ defence of cadets is valid, there will be no
difference in performance or cadets will perform better than their direct entry peers.

111
4.2 Literature review

There have been a number of studies into the effect of experience on performance
of pilots, including the non-technical areas such as decision-making (Wiegmann et
al, 2002; Wiggins & Bollwerk, 2006; Wiggins & O’Hare, 2003; Todd & Thomas,
2012), with many studying the link between experience and accident rates (Goh &
Wiegmann, 2002; Groff & Price, 2006; Bazargan & Guzhva, 2011; Li et al, 2003;
O’Hare & Chalmers, 1999). It is important to note that most of these studies use
pilot populations that are quite different to airline pilots. In particular, most of the low
experience groups in the following studies did not receive the same standard of
training as airline cadets who, after initial accelerated competency based training,
are transitioned to multi-crew operations in large, complex aircraft.

One area of interest to researchers in the experience/performance debate is

accidents involving flight into instrument meteorological conditions (IMC) by visual
flight rules (VFR) rated pilots. That is, pilots who are not qualified to fly in instrument
conditions, which require them to fly with sole reference to instruments such as
when in cloud, instead of conducting the flight with reference to visual cues outside
the aircraft. The focus of this research has typically centred on the aspects of
decision making and pilot experience, with some studies finding an effect of low
experience on accident risk (Goh & Wiegmann, 2002; Wiegmann et al, 2002).

General Aviation pilots

The propensity for pilots of different experience levels to be involved in accidents
has been studied with a number of different pilot cohorts. In general aviation, a
review of 72 accidents which occurred in degraded visibility found no effect of total
flight hours, nor years of experience, although they did find that pilots in the younger
age group were involved in a smaller proportion of these kinds of accidents (Groff &
Price, 2006). In a similar type of study, the effect of low experience on accident risk
for New Zealand pilots in general aviation was reviewed with a particular focus on
the concept that pilots with around 100-200 hours total time had a ‘special
vulnerability’ compared to their colleagues with more than 200 hours total time
(O’Hare & Chalmers, 1999). The same comparison was conducted for pilots with
about 100-300 hours total time compared to the rest of the pilot population. Both

112
studies found no evidence of increased accident involvement of low-hour pilots
(O’Hare & Chalmers, 1999).

Another study of general aviation pilots examined the effect of age, experience and
gender on accident causation and found that lower experience pilots were more
likely to make errors that resulted in an accident than their more experienced peers
(Bazargan & Guzhva, 2011). However, they found that while lower experience pilots
were more likely to be in an accident, the more experienced pilots were more likely
to be involved in a fatal accident. The sample used by the authors included pilots
with less than 100 hours and less than 100-300 hours, meaning while it may cover
some commercial pilots, the sample is not representative of air transport operations.
The authors also defined air transport pilots with 300-2000 hours of total time as
‘...newly licensed airline transport pilots’; 2000-5000 hours of total time as
‘...moderately experienced airline transport pilots’; and 5000 hours or more total time
as ‘the most experienced pilots’ (Bazargan & Guzhva, 2011).

Also in general aviation, a beneficial effect of experience on performance was

found, particularly in the performance of older pilots (Causse et al, 2010). It was
found that older, more experienced pilots outperformed older low-hour pilots who
had around 90 hours total flight time. A similar study also found a ‘protective’ effect
of experience against age related deteriorations in performance (Kennedy et al,
2010). However, contrary to their hypothesis, they did not find improved decision-
making with increased expertise, although the results of their study may have been
influenced by the expertise groups being very close (moderate versus high
expertise, all with instrument ratings with VFR pilots excluded from the study
entirely) (Kennedy et al, 2010). One study examining both retrospective and
prospective indicators of accident risk also used flight hours as a metric, grouping
them into hours flown in the last 6 months, 12 months and a pilot’s entire career.
The author found that single-engine time in the last 12 months; piston-powered time
for all three hours measures; and landings in the last 6-12 months were all
significantly related to accident involvement (Hunter, 2001). However, an important
caveat issued for this study was that the outcomes may well have been an effect of
exposure and in particular ‘a pilot cannot gain the experience that is usually
regarded as a protective factor’ without this exposure (Hunter, 2001).

113
A significant effect for expertise was found when differences in perception of
weather images as a function of expertise was examined (Wiggins & O’Hare, 2002).
To differentiate between experts and novices, they classified experts as pilots who
had accumulated greater than 1,000 hours cross-country flying (mean cross country
experience 5,520 hours, mean total experience 6,873 hours) and novices as pilots
who had accumulated less than 1,000 hours (mean cross country experience 281
hours, mean total experience 801 hours).

In a later study examining heuristic-based approaches to in-flight decision making, it

was noted that ‘greater levels of experience tend to be associated with the selection
of more optimal alternatives, irrespective of the information acquisition strategy
employed, but that this experience is defined by the hours accumulated as pilot in
command rather than by the number of flight hours in total’ (Wiggins & Bollwerk,
2006). The authors also found a significant effect of the hours accumulated in the 90
days preceding the testing resulted in an increased likelihood the optimal alternative
would be selected, thereby concluding that ‘...recent experience, rather than total
experience, the license held, or performance during the familiarization scenarios, is
most predictive of the selection of the optimal alternative’ (Wiggins & Bollwerk,
2006). This is an interesting finding in relation to the on-going perception within the
aviation industry that total flight hours alone can accurately predict performance and
mirrors an earlier study which also noted that task specific experience was more of
a predictor of performance than total experience (Wiggins & O’Hare, 1995).

Airline pilots
Interestingly, in a study of air transport pilot involvement in general aviation
accidents, one study found that in general, air transport pilots (ATP) were safer than
private pilots (Salvatore et al, 1986). The two groups had comparable exposure, that
is, time in general aviation aircraft, but the ATP group were involved in fewer fatal
accidents and fewer ‘pilot-induced’ accidents, indicating that the majority of ATP
accidents were from factors not related to pilot skill (e.g. aircraft maintenance). One
exception to this appears to be that ATP accidents more often involved aerobatic
flight and for the ATP fatal accidents, nearly 50 per cent occurred during aerobatics.
The authors note that ATPs tended to have less experience in the aircraft type for
the fatal aerobatic accidents, pointing to a consistent finding in terms of experience
(Salvatore et al, 1986). The authors also caution that with an increase in pilot

114
experience ‘...comes the pilot’s willingness to encounter more challenging flight
regimes and, perhaps consequently, to accept increased risk’ (Salvatore et al,
1986). This note is important to remember during any debate on flight experience
and the performance of more experienced pilots. This concept is highlighted from a
different perspective in another study examining the influence of flight experience on
mid-air collision risk perception, which found that mid-air collisions typically involved
highly experienced pilots and that high-hour pilots were less likely to scan properly
for traffic and had become somewhat complacent given the low perception of risk
their experience had provided, and the amount of successful flights they had
completed (Shuch, 1992).

There have been studies in the air transport or airline sector of aviation that examine
accident rates and experience, including one which examined professional pilots’
risk of an accident as a function of age and flight experience. This study found that
pilots with less than 5,000 hours were almost twice as likely to be involved in an
accident as their more experience colleagues (Li et al, 2003). The authors found a
stabilising effect once a total flight time of 10,000 hours had been accumulated.
However, it should be noted that these pilots were all operating in the commuter or
air taxi sectors of aviation (that is, not a major airline), which the authors note carries
a 10 – 45 times higher fatal crash rate per pilot flight hour than major airlines.
As part of a gender study examining accident risk for airline pilots, it was found that
‘airline pilots had less pilot-error accidents as experience (total flying hours)
increases’. An effect of major airlines was also found, with pilots who flew for major
airlines being less likely to have a pilot-error accident than those who flew for a non-
major airline (McFadden, 1996). Of interest in relation to airline training, she also
commented that ‘once pilots accumulate additional flying hours and training within
the airline business, their pilot-error accident rates decrease markedly’ (McFadden,
1996). This statement appears to give some support to the idea that cadet training
and development in a wholly airline environment will aid performance, with respect
to a lower involvement in pilot-error accidents. This was also reflected in the findings
of Salvatore et al (1986) above.

A study of pilot performance as a function of flight hours in normal line operations

found no statistically significant differences between low- and high-hour first officers
or captains against a series of technical measures, although there were minor

115
differences in non-technical measures as a function of crew composition (Todd &
Thomas, 2012). There was a difference in the use of automation by low-hour first
officers, who kept the autopilot engaged until a significantly lower altitude. For this
study, low-hour first officers were classified as those with less than 1,500 hours, and
high-hour first officers as those with more than 1,500 hours. For captains, low-hour
was classified as less than 5,000 hours, and high-hour above 5,000 hours (Todd &
Thomas, 2012).

Despite the topic of pilot hours and performance attracting significant debate, there
is little in the literature exploring this issue in relation to airline operations.

116
4.3 Methodology

Data sources
Data were collected from three airlines, one of which (Airline A) supplied data for
one aircraft type, and two of which (Airline B and Airline C) supplied data for two
aircraft types. All aircraft types are classified as large passenger carrying jet aircraft
capable of conducting high-capacity regular public transport (RPT) operations.

Performance measures were from simulator checks and line checks from each
airline. De-identified demographic information, which included entry pathway and
hours at entry, was also obtained.

For Airline A, the performance of second officers was measured. For Airlines B and
C, the performance of first officers was measured. All pilots held at least a
commercial pilot licence and none were trained under the multi-crew pilot licence
system.

Design and procedure

Data sources

Airline A
Data were collected from proficiency check forms for all second officers who had
entered the airline in the previous 13 years. The data were collected by the
operator’s check and training captains during the pilot’s proficiency check in the
simulator, which covered non-normal operations.

A total of 473 records considered complete and suitable for analysis were received
from the airline, covering simulator sessions (cadets n=78, direct entry n=395; low-
hour n=69, high-hour n=404) and line checks (cadet n=22, direct entry n=175; low-
hour n=23, high-hour n=174) conducted by the airline’s check and training captains.
The airline’s flight training department oversees the calibration of check and training
captains to ensure a consistent application of their rating system in both the
simulator and line check environment.

117
The check and training captains and pilots were blind to the specific hypothesis
regarding entry pathway, flight hours and performance.

Airline B
Data were collected from proficiency check forms for a selection of first officers on
two aircraft types operated by the airline. The data were collected by the operator’s
check and training captains during the pilot’s proficiency check in the simulator,
which covered non-normal operations, and during the pilot’s line check, which
covers normal operations.

Inter-rater reliability and calibration was established by the airline and on-going
calibration of the check and training captains by the flight training department occurs
yearly. In addition, the check and training captains are provided with word pictures
of performance at each of the five levels of rating (1=unsatisfactory, 5=excellent) for
each measure being tested.
When the data were grouped into entry pathway to test the effect of pathway on
performance, a total of 116 records (cadet n=35, direct entry n=81) were completed
and considered suitable for analysis for the simulator sessions (non-normal events)
and 151 records (cadet n=45, direct entry n=106) were considered suitable for the
line check data.

When the data were grouped into hours below and above our thresholds to test the
effect of hours on performance, a total of 169 records (low-hour n=19, high-hour n=
150) were complete and considered suitable for analysis for the simulator sessions
(non-normal events). For the line check data, 168 records (low-hour n=10, high-hour
n=158) were complete and suitable for analysis.

The check and training captains and pilots were blind to the specific hypothesis
regarding flight hours and performance.

Airline C
Data were collected from proficiency check forms for a selection of first officers on
one aircraft type operated by the airline. The data were collected by the operator’s
check and training captains during the pilot’s proficiency check in the simulator,

118
which covered non-normal operations, and during the pilot’s line check, which
covers normal operations.

Inter-rater reliability and calibration was established by the airline and on-going
calibration of the check and training captains by the flight training department occurs
yearly. In addition, the check and training captains are provided with word pictures
of performance at each of the five levels of rating (1=unsatisfactory, 5=excellent) for
each measure being tested.

When the data were grouped into entry pathway to test the effect of pathway on
performance, a total of 167 records (cadet n=33, direct entry =134) were completed
and considered suitable for analysis for the simulator sessions (non-normal events)
and 251 records (cadet n=56, direct entry n=195) were considered suitable for the
line check data.

When the data were grouped into hours below and above our thresholds to test the
effect of hours on performance, a total of 167 records (low-hour n=26, high-hour
n=141) were complete and considered suitable for analysis for the simulator
sessions (non-normal events). For the line check data, 251 records (low-hour n=51;
high-hour n=200) were complete and suitable for analysis.

The check and training captains and pilots were blind to the specific hypothesis
regarding flight hours and performance.

Independent variables

Airline entry pathway

For each airline, to test the effect of training pathway on performance, two groups
were created, one for cadet pilots and one for direct entry.

For Airline A, the measure used to test the effect of pathway on performance was a
proficiency simulator session, which contained a number of metrics listed below
under dependent variables, conducted at the time the pilot entered the airline, and
the check to line flight test. Cadet second officers entered the airline with an
average of 246 hours (S.D. 94 h), which they built up during training, while direct

119
entry second officers had, on average, 3,826 hours (S.D. 1,344 h). Data about the
type of previous flying experience for direct entry pilots into Airline A (airline, general
aviation, military) was not provided, however, the airline reported that the majority of
the direct entry second officers were from a general aviation (or similar)
environment. At testing, direct entry SOs had, on average, 4,729 hours (S.D. 1,427
h) and cadets had, on average, 1,042 hours, (S.D. 431 h).

For Airline B, the measure used to test the effect of pathway was a proficiency
simulator session conducted 5 years after the first officers had joined the airline, and
the most recent line check for these pilots. This time period was chosen to
overcome any learning effect as a possible confound. Cadet first officers entered the
airline with, on average 278 hours (S.D. 42 h) predominately built up during training,
while direct entry first officers had, on average, 6734 hours (S.D. 2,853 h). Of the 81
direct entry first officers, 76 had previous airline experience, 3 were from general
aviation and 2 were from the military. At testing, cadet first officers had, on average,
3,984 hours (S.D. 836 h), and direct entry first officers had, on average, 11,158
hours (S.D. 3,172 h).

For Airline C, the measure used to test the effect of pathway on performance was a
proficiency simulator session, which contained a number of metrics listed below
under dependent variables, conducted during the first five years after entering the
airline, and the check to line flight test. Cadet pilots typically entered the airline with,
on average 561 hours (S.D. 457 h) predominately acquired during training, while
direct entry pilots entered with, on average, 5,291 hours (S.D. 3,110 h). Of the 134
direct entry pilots, 32 had previous airline experience, 99 were from general aviation
and 3 were from the military. At testing, cadet pilots had, on average, 1,185 hours
(S.D. 511 h) and direct entry pilots had, on average, 6,923 hours (S.D. 3,471 h).

Total flight hours

The effect of flight hours on performance were examined against the same metrics
used in the previous study (Todd & Thomas, 2012). Second officers from Airline A
were grouped into those with below 1,500 hours and those above; and first officers
from Airlines B and C into below 1,500 hours and those above. The divide of 1,500
hours was chosen based upon the current legislative changes in the United States,

120
and the fact that 1,500 hours total time is necessary for pilots to hold an Air
Transport Pilot Licence (ATPL).

Due to the grouping, some high-hour pilots at the time of testing had entered the
airline as cadets. For Airline A, 93 per cent of low-hour second officers had entered
the airline as cadets and 3 per cent of high-hour second officers had entered as
cadets. For Airline B, 100 per cent of low-hour first officers entered as cadets, and
29 per cent of high-hour first officers entered as cadets. For Airline C proficiency
simulator check, 79 per cent low-hour first officers entered as cadets and 5 per cent
of high-hour first officers entered as cadets. For the line check data, 91 per cent of
low-hour first officers entered as cadets and 2 per cent of high-hour first officers
entered as cadets.

Dependent variables

Technical Performance
For each airline, pilots were rated against technical measures during the simulator
proficiency check designed to test technical knowledge and handling, with various
normal and non-normal technical scenarios (Table 4.1 to Table 4.3). They were
assessed using a score range of 1 (unsatisfactory) to 5 (excellent). In addition, they
were also required to demonstrate proficiency in various regulatory required
manoeuvres.
In addition to be above, for Airline B, proficiency was measured by a rating of pass
(P), pass with repeat (1P) or fail (F) for each manoeuvre.

Non-Technical Performance
For each airline, pilots were rated against various non-technical markers and again,
pilots were rated on the 5 point ordinal scale (Table 4.1 to Table 4.3).
Checks included both normal events, which are routine, expected events that do not
require additional training or specific targeted intervention by a pilot; and non-normal
events, which involve a non-routine event or problem with the aircraft, its systems,
other crew, or the environment, which is not routine and is outside of what is
expected in a ‘normal’ flight (such as an engine failure).

121
Table 4.1: Airline A - Technical (T), Non-technical (N) and required regulatory manoeuvres (M)
Simulator proficiency check metrics Line check metrics
Approach and Landing (T) Checklist usage (T)
Pilot Flying duties (T) Automation use (T)
Pilot Monitoring duties (T) Operational knowledge (T)
Technical knowledge (T) Technical knowledge (T)
Adherence to standard operating procedures (T) Adherence to standard operating procedures (T)
Situation awareness (N) Error recovery (T)
Communication (N) Communication (N)
Assertion (N) Conflict resolution (N)
Missed Approach (M) Environmental awareness (N)
ILS approach (M) Weather assessment (N)
Holding (M) Leadership (N)
Departure (M) Handover (N)
Normal Landing (M) Traffic awareness (N)
Emergency descent (M) Teamwork (N)
Stall (M)
Engine failure during cruise (M)
Normal takeoff (M)

Table 4.2: Airline B - Technical (T), Non-technical (N) and required regulatory manoeuvres (M)
Simulator proficiency check metrics Line check metrics
Non precision approach (M) Approach (M)
ILS approach and missed approach (MAP) (M) Landing (M)
Holding (M) Technical (T)
En route NDB_VOR (M) Teamwork (N)
Engine inoperative landing (M) Descent (T)
Engine inoperative ILS and MAP (M) Configuration (T)
V1 cut (engine failure during takeoff) (M) Takeoff (M)
Preparation (T) Flight management (T)
Automation (T) Automation (T)
Handling (T) Handling (T)
Procedure (T) Adherence to SOPs (T)
Systems knowledge (T) Operational (T)
Situation awareness (N) Conflict resolution (N)
Crew Resource Management (CRM) (N) Communication (N)
Command (N) Airmanship (N)
Briefings (N)
Aircraft (T)
Traffic awareness (T)
Weather assessment (T)
Terrain awareness (T)
Threat awareness (T)

122
Table 4.3: Airline C - Technical (T), Non-technical (N) and required regulatory manoeuvres (M)
Simulator proficiency check metrics Line check metrics
Takeoff (M) Takeoff (M)
Rejected takeoff (M) Initial climb (M)
Instrument approach (M) Instrument approach (M)
Engine failure (M) Cruise (T)
Engine fire in cruise (M) Visual approach (M)
Climb (Obstacle Clearance Procedure – OCP) (M) Abnormal emergency (M)
Abnormal emergency (M) Descent planning (T)
Approach and landing (T) Landing (T)
Systems knowledge (T) Pre-flight planning (T)
Procedures (T) Systems knowledge (T)
Support duties (N) Procedures (T)
Communication (N) Ground handling (T)
Situation awareness (N) Communication (N)
Leadership and management (N) Leadership and management (N)
Decision making (N) Situation awareness (N)
Decision making (N)

For technical and non-technical performance, the 1-5 ordinal scales were grouped
into the following categories: below standard (1), meeting the standard (2-3), and
exceeding the standard (4-5). These groups were formed on the basis of the rating
scales and word pictures used by the airlines involved in the study.

Statistical analysis
Chi square analyses were utilised for categorical variables and statistical tests were
interpreted using the type 1 error rate of 0.01 for statistical significance. To control
for small cell size, Fisher’s exact post hoc test was performed on the data. A
limitation of using chi square for multiple comparisons is the increased propensity
for type I error to occur, which was why the significance level of 0.01 was used
(Wiggins & Stevens, 1999). However, the results that fall between 0.01 and
0.05 have also been included in the results to highlight overall trends in the data
(Mendenhall et al., 1981).

123
4.4 Results

Overall there were no significant differences between the groups for any dependent
variable in relation to meeting the standard required in any of the three airlines. The
results below show the differences in each group exceeding the standard. That is,
they all passed, however, performance varied between the groups within the ‘pass’
standard. While the line check data from Airline C did contain some fail items for
some pilots, these pilots successfully completed the subsequent check flight.
Significant results are labelled with an asterisk (*) in the figures.

Airline A

Simulator proficiency check

Entry pathway study
Of the seventeen metrics, the performance of cadets differed to the direct entry
second officers with respect to seven of the 17 metrics. The difference was
measured as a percentage of pilots exceeding the standard, with a higher number of
direct entry second officers exceeding the standard compared to cadets (Figure
4.1). The metrics the groups differed on were engine failure in the cruise, normal
landing, ILS approach, situation awareness, assertion, pilot flying and approach and
landing (Table 4.4).

Table 4.4: Differences in performance between cadets and direct entry second officers
Metric Cadet (%) Direct entry (%) χ2 (1, n=473) p (2-tailed)
Engine failure in the cruise 74.4 87.8 9.63 0.004
Normal landing 73.1 88.4 12.50 0.001
ILS approach 69.2 90.6 26.56 <0.001
Situation awareness 75.6 88.4 8.88 0.006
Assertion 78.2 94.7 23.94 <0.001
Pilot flying 79.5 91.6 10.37 0.003
Approach and landing 73.1 91.4 21.26 <0.001

For the metric pilot monitoring, the difference was not significant at 0.01, but was
within the range of 0.01-0.05 (cadets = 82.1%, direct entry = 90.9%, χ2(1, n = 473)
= 5.38 p=0.027, 2-tailed).

124
Figure 4.1: Performance in proficiency simulator check as a function of entry pathway

Flight hour study

The results for the comparison of hours versus performance mirror those for entry
pathway for this cohort of pilots. Of the seventeen metrics, the performance of the
low-hour pilots differed to the high-hour group with respect to seven metrics. The
difference was measured as a percentage of pilots exceeding the standard, with a
higher number of high-hour second officers exceeding the standard compared to
low-hour second officers (Figure 4.2). The metrics the groups differed on were
engine failure in the cruise, normal landing ILS approach, situation awareness,
assertion, pilot monitoring, pilot flying and approach and landing (Table 4.5).

125
Table 4.5: Differences in performance between low- and high-hour second officers
Metric Low-hour (%) High-hour (%) χ2 (1, n=473) p (2-tailed)
Engine failure in the cruise 73.9 87.6 9.00 0.005
Normal landing 71 88.4 14.59 0.001
ILS approach 68.1 90.3 25.93 <0.001
Situation awareness 73.9 88.4 10.39 0.004
Assertion 78.3 94.3 20.54 <0.001
Pilot flying 78.3 91.6 11.27 0.002
Approach and landing 73.9 90.8 16.44 <0.001

As above, the metric pilot monitoring was significant when tested at 0.05
significance (low-hour = 81.2%, high-hour = 90.8%, χ2(1, n = 473) = 5.84 p=0.031,
2-tailed).

Figure 4.2: Performance in simulator proficiency check as a function of flight hours

126
Line check
Entry pathway study
For the line check data, the only significant difference between cadets and direct
entry second officers was with respect to the one non-technical metric of leadership
(cadets = 72.7%, direct entry = 95.4% , χ2 (1, n = 197) = 15.26 p=0.002, 2-tailed).
The other thirteen metrics showed no significant difference (Figure 4.3).

Figure 4.3: Performance in line check as a function of entry pathway

127
Flight hour study
For the line check data, the only significant difference between low- and high-hour
second officers, were with respect to the non-technical measure of leadership (low-
hour = 78.3%, high-hour = 94.8%, χ2 (1, n = 197) = 8.45 p=0.014, 2-tailed) (Figure
4.4).

Figure 4.4: Performance in line check as a function of flight hours

Airline B

The data collected and analysed for Airline B was for proficiency and line checks at
the 5 year mark, in order to account for any learning effect that may have been
present at airline entry.

Simulator proficiency check

Entry pathway study
The only significant difference between the cadets and the direct entry first officers
was that the cadets scored lower on the measure of crew resource management
(CRM) at the significance level of 0.05 but not 0.01 (cadet = 62.9%, direct entry =
80.3%, χ2 (3, n = 116) = 10.39 p=0.018, 2- tailed). There was no significant

128
difference between the two groups on any of the other technical or non-technical
measures, or for the manoeuvres.
While the other measures showed no significant difference, there was a trend
towards a lower performance of cadets overall (Figure 4.5, Figure 4.6).

Figure 4.5: Performance in simulator proficiency check manoeuvres as a function of entry

pathway
P=pass F=fail 1P=pass with repeat

129
Figure 4.6: Performance in simulator proficiency check as a function of entry pathway

Flight hour study

For the four technical and three non-technical metrics, the pilots in the low-hour
group had a significant difference with respect to all metrics (Figure 4.7, Figure 4.8
and Table 4.6). High-hour pilots scored significantly higher in these areas against
their low-hour colleagues, with a greater number of these pilots exceeding the
standard. The low-hour pilots, on the other hand, had a higher repeat rate for the
required regulatory manoeuvres: engine inoperative landing (low-hour = 21.1%,
high-hour = 3.3%, χ2 (2, n = 169) = 10.59 p=0.016, 2-tailed) and non-precision
approach (low-hour = 21.1%, high-hour = 4.7%, χ2 (2, n = 169) = 7.53 p=0.032, 2-
tailed).
Table 4.6: Differences in performance between low- and high-hour first officers
Metric Low-hour (%) High-hour (%) χ2 df (n=169) p (2-tailed)
Command 10.5 66 27.15 3 <0.001
CRM 36.8 73.4 12.28 3 0.003
Situation awareness 26.3 68 22.63 3 <0.001
System knowledge 63.2 85.3 6.98 2 0.042
Procedure 10.5 54.6 28.55 4 <0.001
Handling 15.8 69.3 28.64 3 <0.001
Automation 63.2 81.3 6.77 2 0.021

130
Figure 4.7: Performance in simulator proficiency check manoeuvres as a function of flight
hours

131
Figure 4.8: Performance in simulator proficiency check as a function of flight hours

Line check
Entry pathway study
The only significant difference was with respect to the metric of communication,
measured by a higher percentage of cadets exceeding the standard at the 0.05 level
(cadets = 95.5%, direct entry = 84.9%, χ2 (1, n = 151) = 7.48 p=0.036, 2-tailed)
(Figure 4.9).

132
Figure 4.9: Line check at 5 year mark as a function of entry pathway

Flight hour study

For the line check data, the only significant differences between low- and high-hour
first officers, were with respect to four technical metrics, measured by a higher
percentage of high-hour pilots exceeding the standard (terrain awareness, low-hour
= 70% high-hour = 96.2%, χ2 (2, n = 168) =13.45 p=0.010, 2-tailed; and traffic
awareness, low-hour = 70% high-hour = 95.6%, χ2 (3, n = 168) =23.16 p=0.002, 2-
tailed). It should be noted that for terrain awareness, neither group scored lower
133
than a 3 (Figure 4.10). Two metrics were significant at 0.05 level - SOP adherence,
low-hour = 70%, high-hour = 84.2, χ2 (3, n = 168) = 14.43 p=0.037, 2-tailed and
approach, (low-hour = 70%, high-hour =88%, χ2 (3, n = 168) = 17.15 p=0.044, 2-
tailed).

Figure 4.10: Line check at 5 year mark as a function of flight hours

134
Airline C

Simulator proficiency check

Entry pathway study
The only significant differences between the cadets and the direct entry groups was
that the cadets scored lower on the measure of leadership and management at the
significance level of 0.01 (cadet = 44.8%, direct entry = 73.9%, χ2 (3, n = 159)
=10.06 p=0.007, 2-tailed). One metric was significant at the 0.05 level only Situation
awareness (cadet= 50%, direct entry= 73.9%, χ2 (3, n = 166) = 9.73 p=0.03, 2-
tailed). There was no significant difference between the two groups on any of the
other technical or non-technical measures, or for the manoeuvres (Figure 4.11).

Figure 4.11: Performance in simulator proficiency check as a function of entry pathway

135
Flight hour study
The only significant differences between the low- and high-hour groups was that the
low-hour pilots scored lower on the measures of leadership and management at the
significance level of 0.01 (low-hour= 39.1%, high-hour= 75.2%, χ2 (3, n = 159) =
11.50 p=0.004, 2-tailed) and situation awareness at the significance level of 0.01
(low-hour= 36%, high-hour= 75.2%, χ2 (3, n = 166) = 18.07 p=0.001, 2-tailed). One
metric was significant at the 0.05 level only – Abnormal emergency (low-hour=64%,
high-hour= 84.6%, χ2 (2, n = 155) = 6.60 p=0.03, 2-tailed). There was no significant
difference between the two groups on any of the other technical or non-technical
measures, or for the manoeuvres (Figure 4.12).

Figure 4.12: Performance in simulator proficiency check as a function of flight hours

136
Line check
Entry pathway study
The only significant differences between the cadets and the direct entry first officers
was that the cadets scored lower on the measures of landing (cadet=33.9%, direct
entry = 58%, χ2 (4, n = 251) =10.74 p=0.01, 2-tailed) and leadership and
management at the significance level of 0.01 (cadet = 26.4%, direct entry = 52.6%,
χ2 (3, n = 169) = 11.17 p=0.003, 2-tailed). There was no significant difference
between the two groups on any of the other technical or non-technical measures, or
for the manoeuvres (Figure 4.13).

Figure 4.13: Line check as a function of entry pathway

137
Flight hour study
The only significant differences between the low- and high-hour groups was that the
low-hour first officers scored lower on the measures of landing (low-hour = 31.4%,
high-hour = 58%, χ2 (4, n = 251) = 12.5 p=0.005, 2-tailed) and leadership and
management at the significance level of 0.01 (low-hour = 25%, high-hour = 52%, χ2
(3, n = 169) = 11.55 p=0.003, 2-tailed). One metric was significant at the 0.05 level
only – Visual approach (low-hour = 20.4%, high-hour= 38.3%, χ2 (4, n = 224) =
10.13 p=0.03, 2-tailed). There was no significant difference between the two groups
on any of the other technical or non-technical measures, or for the manoeuvres
(Figure 4.14).

Figure 4.14: Line check as a function of flight hours

138
4.5 Discussion

The subject of pilot performance is one that often draws heated debate and
entrenched positions amongst the aviation industry. Often these positions have
been formed and debates have occurred without scientific data to support or oppose
the various stances. Despite this, the concept that the performance of low-hour and
cadets pilots would be fundamentally different to their high-hour direct entry
colleagues has not previously been explored to any great extent.

While the results of this study indicated some significant differences between the
pilot groups for each airline, it is worth noting that there are a significant number of
similarities in their performance (Table 4.7 and Table 4.8). In the line check
environment, all three airlines showed very few differences between the
performance of cadets and direct entry pilots, and between low- and high-hour
pilots. This finding is largely consistent with the authors’ previous study examining
the effect of flight hours on performance in the line operational environment (Todd &
Thomas, 2012).

Table 4.7: Overview of significant differences in performance across all three airlines for
simulator (S) and line (L) checks as a function of entry pathway (- indicates not
tested)
Metric Airline A Airline B Airline C

Engine failure in the cruise (M) S -

Normal landing (M) S L

ILS approach (M) S

Pilot flying (T) S -

Approach and landing (T) S

Leadership/ command (N) L SL

Situation awareness (N) S

Assertion (N) S - -

139
Table 4.8: Overview of significant differences in performance across all three airlines for
simulator (S) and line (L) checks as a function of flight hours (- indicates not
tested)
Metric Airline A Airline B Airline C

Engine failure in the cruise (M) S -

Normal landing (M) S L
ILS approach (M) S

Non precision approach (M) - S

Engine inoperative landing (M) - S -

Pilot flying (T) S -

Approach and landing (T) S

System knowledge (T) S

Procedure (T) S

Handling (T) - S -
Automation (T) S -

Terrain awareness (T) L -

Traffic awareness (T) L -

Leadership/ command (N) L S SL

CRM/ Communication (N) S

Situation awareness (N) S S S

Assertion (N) S - -

Simulator proficiency check results

In the simulator proficiency checks, there was a difference between the cadet and
direct entry second officers and low- and high-hour second officers for Airline A. The
differences in performance were fairly consistent between the entry pathway and
flight hour groups. Since this data was collected at airline entry, data from Airline B
was examined at the 5 year mark to account for any learning effect that may have
been present at airline entry. Very few significant differences between the cadets
and direct entry first officers in this simulator proficiency check were found.
However when the data were analysed for flight hour experience, there were a few
more differences between the low- and high-hour first officers. For Airline C, whose
simulator proficiency checks were conducted within the first 5 years of joining the
airline, a consistent difference on one metric only was found, between cadets and
direct entry pilots and low- and high-hour first officers, with one other significant
difference in the flight hour data.

For Airline A, the differences in performance against the six technical and two non-
technical metrics are worthy of further exploration. The technical metrics include the
regulatory required manoeuvres, such as an ILS approach, and the main difference

140
between the groups appears to occur during the approach and landing phase of
flight. Given the role of the second officer is to act as a relief pilot in cruise flight it is
not particularly surprising to see a difference in performance between cadet and
direct entry second officers against this metric. While both groups are now limited to
a non-control seat outside of the cruise phase of flight, it is conceivable that prior to
their direct entry as a second officer, these pilots had flown either as first officers in
other airlines, in the military, or in general aviation. They are therefore more likely to
have had a higher exposure to the approach and landing phase of flight, and
experience in a control seat during this phase.

For the non-normal metric of engine failure, it is also possible that previous
exposure to an airline environment, and therefore simulator sessions, has aided
these second officers in their understanding of what the simulator session will
involve, and the necessary actions on their part. However, irrespective of this, it is
important for second officers, who will occupy a control seat during cruise, to be
proficient at handling an engine failure in this phase of flight. While each pilot met
the standard for this metric, the number of cadets and low-hour pilots were over-
represented in this marking group, pointing to a potential need for future training to
target this difference. The other technical metrics relate to flying role, which again, is
something that a direct entry second officer with previous airline experience will
have some familiarity with, compared to a cadet.

The non-technical metrics covered areas such as situation awareness and

assertion. It is possible that the prior exposure to, and familiarity with, airline
operations has aided direct entry second officers in performance against this metric.
It is also possible that the direct entry pilots have developed greater familiarity with
the need for, and technique of, good situation awareness through the flying they
conducted prior to entry into the airline. The metric of assertion may also be
explained in the same way, especially if these pilots had operated in a team
environment previously.

The potential confounds of exposure and familiarity are also likely of pilots with more
than 1,500 hours, which may well explain the similarity in the results between the
entry pathway and flight hour groups for Airline A and Airline C.

141
For the results from the simulator proficiency check for Airline B, the differences in
performance against the metrics are quite interesting even in the bigger picture of
the trend towards lower performance against all metrics for cadets. The significant
difference against the CRM metric at the 0.05 level is not too surprising given it
typically covers communication and workload management. Direct entry first officers
are more likely to have had higher exposure to, and familiarity with, these concepts
prior to entry to this airline and it is not surprising that their communication would be
stronger, given the necessity of operating in a team environment for longer than the
5 years in this airline alone. This also applies to the results from Airline C, which
showed that direct entry pilots were scoring higher in both situation awareness and
leadership and management.

The overall trend in performance of low-hour first officers may well be due to a
similar effect. As many of the direct entry first officers had previous experience flying
in an airline, it is likely they have had to fly the required manoeuvres a number of
times before entry to this airline, as well as having to demonstrate technical and
non-technical proficiency in the simulated environment. It is worth noting that the
low-hour pilots were all cadets (n=19), and 96 of the high-hour pilots (n=150) had
previous airline experience, although the high-hour group also included 43 cadets. It
is therefore possible that the previous airline experience of the high-hour first
officers has allowed these pilots to develop a more in-depth understanding of airline
operations and the requirements of proficiency checks in an airline environment, the
inexperience of which may be influencing the performance of the low-hour pilots.

The difference in performance between the low- and high-hour pilots against all
technical and nontechnical metrics is surprising, given there was a lack of significant
variation when the effect of entry pathway was considered on the same dataset. It
seems from these results that the effect of low hours on performance is stronger
than any effect of entry pathway, despite all of the low-hour pilots being cadets.

The difference in performance against the two manoeuvres is quite interesting,

given that the performance of the low-hour pilots was not significantly different
against the non-normal manoeuvres that are generally perceived as being more
challenging – such as the V1 cut (engine failure during takeoff) and the one engine
inoperative Instrument Landing System Missed Approach (ILS MAP). This result is

142
also not consistent with the results from the airline entry pathway study, despite all
low-hour pilots being cadets.

For Airline C the only differences in the simulator proficiency checks for technical
metrics was at the 0.05 level of significance and were on the metrics of cruise
engine fire (for entry pathway only) and abnormal emergency (for both groups).
Given the number of comparisons and that this result was at the 0.05 level, it is
possible the difference was due to chance only. However, it is also worth exploring
these differences with a view to appropriate training intervention.

Line check results

An interesting outcome from the line check data for entry pathway for Airline B was
that the only metric showing a difference was communication at the 0.05 level. For
our purposes, it is worth considering that there is a difference in communication in
the line environment as a function of entry pathway and to treat the result as valid.
That is, if a pilot entered the airline as a cadet, they appear not to communicate as
effectively in the line check environment as their direct entry peers. As mentioned
above, this may be a function of the previous exposure to an airline environment
providing the direct entry pilots with a greater knowledge of what communication
strategies are required in the line environment, and the skills to implement these
effectively.

The significant differences in the line check data for Airline B as a function of flight
hours was also interesting. Given these metrics - SOP adherence and approach at
0.05; and terrain awareness and traffic awareness at 0.01 are high-level categories
it is possible that the performance of the pilots differed on a subset of these metrics
and not the overarching metric itself, per se. It may be that a low-hour pilot can fly
an approach adequately, however, they may be receiving a lower rating because
they did not fly a certain segment to a high standard, or did not follow a procedure
exactly and as such this difference is being captured under the heading of
‘approach’.

However for metrics such as terrain and traffic awareness, the finding is not exactly
surprising. It is reasonable that a pilot will develop a greater ability to plan and
operate adequately within their environment once they build up exposure to, and

143
experience in, that environment. Issues such as terrain awareness in particular will
require strong local knowledge and this may take time to adequately acquire. It
should be noted that neither group scored lower than a 3 for terrain awareness,
indicating they were both sitting at the high end of ‘meeting the standard’. The issue
of exposure aiding performance may also be true of SOP adherence, which is
another area where it is reasonable to anticipate that performance will improve with
both exposure and practice. Low-hour pilots will be limited in the exposure they
have to operating with, and following, SOPs in the line environment.

For the line check data from Airline A, the difference in performance between the
cadets and direct entry second officers for the metric of leadership could be a
function of the role of a second officer not requiring leadership on the line.
Therefore, cadets who have only ever flown as second officers will not be as familiar
with the principles underpinning this metric as pilots who have had previous
exposure to the concept and who have implemented it in previous roles, and/or
airline operations prior to entry into this airline.

That the same difference exists for low-hour pilots from Airline A, with the inclusion
of communication, is not a particularly surprising finding. It is reasonable that low-
hour pilots will not have had great exposure to the line operating environment, unlike
many of their high-hour colleagues and it is probable that this lack of previous
exposure is impacting their ability to communicate effectively. As is the case with
leadership, it is possible the influence of role is a factor here also, specifically that
second officers are typically not required to communicate to the same extent as first
officers and captains.

The line check data for Airline C was very similar, with cadets and low-hour pilots
scoring lower on leadership and management and landing metrics. Once again, the
prior exposure of direct entry pilots to the landing phase of flight and to the concept
of leadership may be influencing the results. The only difference in technical metric
for line check data was at the 0.05 significance level and indicated low-hour pilots
were scored lower on the visual approach metric. However, given both groups had a
low percentage exceeding the standard (20% low-hour v 38% high-hour), it is
possible there is a difficulty within this metric that is influencing the performance of

144
both groups. These results are close to those of Airlines A and B, indicating a
consistent effect of prior experience being beneficial.

Overall however, the results from Airline C showed very little difference in
performance in both simulator proficiency checks and line checks for each of the
entryway and flight hour groups, indicating that cadets and low-hour pilots were
performing to an acceptable level.

Limitations
It is important that the various figures throughout the results section are interpreted
correctly. They show the percentage of each pilot group for each airline that
exceeded the standard. As outlined previously, the ratings were divided into: below
standard (unsatisfactory), meeting the standard and exceeding the standard. As
such, the figures show the percentage of pilots who exceeded the standard,
however, it should be noted that all pilots passed the simulator proficiency or line
check (and therefore at least met the standard) and were suitable to continue flying.
It should also be noted that the idea of a simulator proficiency check is to ensure
that a pilot is proficient in a number of manoeuvres, as well as technical and non-
technical skills and knowledge. It is reasonable to assume there will be an individual
variation in performance within this environment that may well occur due to issues
unrelated to individual skill, such as stress or the physical fitness/wellbeing of the
pilot, both of which are beyond the scope of this study to consider.

The other potential influencing factor is that the metrics pilots are rated against are
the overarching areas of proficiency and as such, any sub-element of that metric
may influence performance. For example, if a procedure has recently changed or is
not often used, it may be the execution of the procedure rather than the approach
which is causing the pilot to repeat the manoeuvre, or receive a lower rating. It is
therefore possible that the variation in performance that is being seen is not a
function of pilot proficiency or ability, but rather the subtleties of an underlying
procedure and/or other training issues.

Another factor that must be acknowledged is the possibility that one or more of the
differences might be to chance alone, due to the limitation that multiple comparisons
were required. However, this was addressed by using the lower significance level of

145
0.01 with exploration of results that were significant at 0.05 to ensure overall trends
in performance were identified.

The other possible influence on the data, which should be considered, is that of
marking bias. Marking bias by check captains is reported to be an inherent problem
in the recurrent measure of flight crew proficiency checks and generally involves the
pilots under check receiving a mark consistent with their performance for their rank
and/or experience. That is, a cadet or low-hour pilot performing well may only
receive a 3, whereas the same performance from a more experienced first officer or
captain may attract a 4. Interestingly the data for Airline B showed a potential bias of
a low-hour ‘stop’ at 3, with high-hour pilots receiving marks clustered around 3-4.
Therefore the reader should be mindful of the possibility that what the data is
showing is a difference in marking and not in performance.

146
4.6 Conclusion

This study has provided an overview of pilot performance as a function of both

airline entry pathway, and flight hours within three high-capacity regular public
transport airlines.

There is a level of unease within the wider aviation industry regarding the concept of
low-hour pilots in general, and cadets and MPL pilots in particular. Despite the
prevalent existence of cadet training schemes for a number of airlines around the
world for a number of years, there remains a level of opposition within Australia to
such a concept. With the development of training innovations such as the MPL, this
opposition has certainly not abated.

The results indicate that while there are differences in performance between the
various groups, the performance of the cadets and low-hour pilots against their
direct entry and more experienced brethren is remarkably similar. All pilots within
the cadet entry and low-hours groups passed the simulator and line checks required
of them, meaning they are proficient to be operating as second and first officers
within the respective airlines, as rated by the airlines’ check and training staff. The
differences in performance were between ‘meeting the standard’ and exceeding it,
and are subject to the caveats listed in the discussion. However given that the
cadets met the standard and are therefore proficient, the evidence is demonstrating
that the cadet pathway for low-hour pilots is a valid option for airlines. Contrary to
the expressed concerns of some sectors of the industry, there was no evidence
indicating that cadets or low-hour pilots within these airlines were any less
competent or proficient than their direct entry and high-hour peers.

The inclusion of simulator proficiency data and airline entry pathway in this study
was to address some of the limitations from a previous study, which occurred within
the line operating environment only, and without the data to study entry pathway. By
studying simulator proficiency checks, this study was able to examine the
performance of cadet and low-hour pilots in non-normal scenarios, which is not
possible during line operations. Aircraft accidents such as AF447 and the DHC-8
accident at Buffalo, New York, highlight the importance of appropriate training for
non-normal events and the need for competent, proficient performance of all flight

147
crew to successfully handle non-normal events that occur in the operational
environment.

The differences in the performance of pilots against all metrics were variable within
airlines and inconsistent across all three airlines. This suggests that the differences
between the groups were not of a systemic nature that would highlight an area of
concern for industry. While the metric normal landing showed a difference across
two of the three airlines, none of the rest of the required regulatory manoeuvres or
technical metrics were significantly different in more than one airline. For non-
technical metrics, both leadership and situation awareness were significantly
different in all three airlines. Although this is understandable given the low
experience of cadet and low-hour pilots, focused exposure to those metrics during
initial airline training may reduce this difference as was seen with cadets at the 5-
year mark in Airline B.

While differences in performance were discovered as a function of flight hours in

some non-normal scenarios, the fact remains that these differences did not result in
the low-hour pilots being marked as unsatisfactory. Nevertheless, the data from
these comparisons may well further inform future training within high-capacity RPT
airline operations to ensure that low-hour pilots are receiving suitable training
interventions for the areas they are not performing as well on.

While the subject of low-hour and cadet pilots have received significant international
attention recently (e.g. the Australian senate inquiry and American legislative
changes), it is important to ensure that any on-going debate is informed by relevant
studies and science. With the advancement of aircraft technology and a growing
shortage of pilots; regulators, airlines and industry bodies need to ensure that the
underpinning training aids pilots by providing targeted training to prepare them for
any non-normal events during flight.

148
Chapter Five - Discussion and future directions

149
5.1 Issues facing pilot training

Background
Aviation, at just over 100 years old, is a relatively new means of transportation for
society. Despite this, it is aviation that has made some of the greatest
advancements in technology and engineering in the past decades. Compared to just
50 years ago, air travel is safer, more reliable, cheaper and accessible to a wider
range of people than at its inception.

In the wake of world war two the airline industry advanced dramatically and the
International Civil Aviation Organization (ICAO), a United Nations body, was formed
to oversee and administer flight standards, rules of the air and governing
conventions for aviation on a global scale. Nations who were signatories to the
convention which formed ICAO were required to establish their own aviation
regulatory body to administer these global standards in local legislation. In Australia,
this function is currently provided by the Civil Aviation Safety Authority (CASA).

Historically, many pilots entered an airline directly from the military. Following both
world wars, this entry pathway was the most prevalent given the amount of pilot
training the military had conducted over this time (Harris, 2011). However the other
pathway to airlines was through the burgeoning general aviation sector. In countries
such as Australia, Canada and the United States of America, the reliance on
general aviation to transport goods and people to remote locations was, and
remains, high (Bureau of Infrastructure, Transport and Regional Economics, 2012).
General aviation has existed from the very beginning of the industry and covers
many operation types such as charter flights, emergency and medical flights,
agricultural flying and flight training (Bureau of Infrastructure, Transport and
Regional Economics, 2012). If a pilot does not enter employment in an airline
following military training and service, they will typically accumulate their flight hours
in general aviation. The third option of a pilot training pathway is to join an airline as
a cadet, with either no or little previous flying experience, whereupon the cadet is
trained for the airline in a flight training organisation. The cadet then typically enters
as a second officer, meaning they do not occupy a control seat for take-off or
landing. They only actively fly the aircraft when it is cruising at altitude and the
primary flight crew are resting. However some airlines have employed cadets as first

150
officers, meaning they will conduct take-offs and landings in passenger carrying jet
aircraft at the successful completion of their training.

Despite various advancements within the aviation system, the training system for
pilots has remained relatively unchanged for over 50 years. To put that in
perspective, aircraft have become much more sophisticated, with computer-
operated flight controls and integrated control systems, even to the point of using
cutting edge materials, such as carbon fibre composite, in the construction of
aircraft. Engine reliability has improved so dramatically that long flights over water
without a suitable place to land are seen as commonplace, and occur as part of a
normal flight. It is this complex, evolving, sophisticated system that requires pilots;
not the system of yesteryear with failing components, low reliability engines and a
need for multiple flight crewmembers in the cockpit just to be able to manually
manage the aircraft’s systems. Yet pilot training has not evolved nearly as fast. It
has, however, incorporated improvements in simulation into initial and recurrent
training and improved the training given to pilots in human factors, both of which are
excellent initiatives that have produced demonstrable safety benefits.

The recent developments implemented by ICAO to amend pilot training standards

and licence types to allow licences such as the Multi-Crew Pilot Licence (MPL), has
created much debate and discord within the aviation industry (Sutton, 2008). The
implementation of the MPL, which will allow pilots with no prior piloting experience to
enter an airline at the end of their training as a first officer (thereby conducting take-
offs and landings) is a major innovation within pilot training. Typically low-hour
commercial pilots (who had been issued a commercial pilot licence or CPL) entered
an airline as a cadet second officer. The current pilot licensing system in Australia
offers two options for pilots gaining a CPL. Pilots can first attain a private pilot
licence and then acquire 200 hours of total flight time with 100 hours solo (pilot in
command) before conducting a flight test for the issue of a CPL (CASA, 2013). The
other pathway is to enter a flight training organisation under the Civil Aviation
Regulations 5.111 requirements, which state that the flight hours can be reduced to
150 hours total time and 70 hours solo as long as the student undergoes continuous
training without any substantial break and the training is completed at the one
school (CASA, 2013).

151
The introduction of the MPL was designed to build upon the cadet tradition, but
instead of entering as a second officer, the MPL pilot will be a first officer. However,
there are a number of crucial differences to this licence and training. One is that
MPL pilots can only fly as part of a two pilot crew, that is they are not allowed to fly
any aircraft of any size on their own. To do so they would need to hold a CPL.
Additionally, the MPL training syllabus has placed greater emphasis on two crew
operations, including areas such as crew resource management and threat and
error management. It also reduced the amount of time the MPL student pilot spends
in small, piston-engine aircraft by increasing the amount of time they spend in large,
jet aircraft simulators. The syllabus also reduced the amount of solo time the MPL
student had in these small aircraft. The MPL requirements are for students to
accumulate at least 240 hours in total, with at least 40 hours as pilot of an
aeroplane, at least 10 hours of solo time and at least 5 hours of cross-country flight
time as pilot in command of an aeroplane (CASA, 2013). These command hour
requirements are significantly lower than the CPL requirements mentioned above,
however there is also the requirement that any aeronautical experience that is not
completed as flight time must be completed in an approved flight simulator. The
combination of these amendments led to strong opposition from aviation unions
such as the Australian and International Pilot’s Association, who likened the MPL to
“handing the keys of the Ferrari to a P-Plater” (AIPA, 2010).

One of the reasons for the strong debate surrounding the introduction of the MPL,
and pilot training innovations in general, is the importance of the pilot training
system as a foundation of aviation safety. This foundation was recently recognised
by the Federal Aviation Administration (FAA), the United States of America’s
aviation regulator body, when releasing new pilot training syllabus requirements.
The FAA acknowledged that the new syllabus should “contribute to a reduction in
aviation accidents” (FAA, 2013). In addition, many accident investigations have
highlighted the need for improvements in pilot training following various high-profile
accidents, emphasising the need for pilot training to ensure a strong knowledge-
base and skill development for pilots to assist them in facing emergency and non-
normal situations (BEA, 2012; NTSB, 2010).

Despite the on-going debates and opposition from within the aviation industry to
concepts such as the MPL, no dedicated research has been undertaken to look at

152
the underpinning elements of the MPL or to examine whether any of this opposition
is based on scientific data. While there has been research on competency-based
training, skill acquisition, use of simulation and pilot performance as a function of
flight hours, none of this research considered large passenger carrying jet aircraft
operations involving major airlines and in some cases the findings were
contradictory. As such, this thesis examined the evidence surrounding competency-
based training as well as pilot performance in an airline environment. The aims of
the studies within this thesis were: to examine whether or not there was any
evidence of competency-based training influencing pilot hours at licence issue; to
examine pilot performance as a function of flight hours in normal line operations
(day-to-day flights) against technical, non-technical and threat and error
management metrics; and to examine pilot performance as a function of both flight
hours and airline entry pathway (cadet or direct-entry) during line checks and
simulator checks covering normal and non-normal operations against technical,
non-technical and regulatory required manoeuvres metrics.

The first study examined the current commercial pilot licence (CPL) training system
in Australia to determine whether there was any evidence of competency-based
training in practice. This was achieved by reviewing 688 records of pilots applying
for a CPL and analysing the hours they had accrued as a function of flight training
organization, regulatory requirements and finally, looking for any evidence of
individual variation within these hours.

The second study expanded on the idea of individual pilot competence by reviewing
the performance of low- and high-hour pilots during normal operations in a major
airline. Data were collected during in-flight observations and the performance of the
pilots on various technical, non-technical and threat and error metrics were analysed
as a function of which group they belonged to, which was based on their hours
accumulated (grouped for low- and high-hour).

In the third study, the performance of low- and high-hour pilots, and cadet and
direct-entry pilots was examined to determine if there was any difference in their
performance based on the hours they had accumulated as well as their entry
pathway into an airline. Three major airlines participated in this study, with data
being collected during normal line operations flight checks, as well as during non-

153
normal flight scenarios in a simulator, conducted as a flight review/check. Pilot
performance within both the line and simulator environment was rated using metrics
for technical, non-technical and regulatory required manoeuvres. These ratings
were again analysed as a function of the group each pilot belonged to, this time for
both hours accumulated (low/high) and the entry pathway to the airline (with military,
general aviation and other airline all being classified as direct entry, and cadets).

5.2 Summary of findings

Study 1 – Competency-based training

An examination of the flight hours at licence issue for commercial pilot licence (CPL)
students showed a stronger effect for both the regulatory minimum hours and the
flight training organisation (FTO) syllabus than the individual variation expected of
students trained within a competency-based system. The research presented in this
thesis reinforces the issue already identified within the aviation domain of the
incompatibility of the regulatory required minimum hours to be licensed sitting within
a competency-based training regime (Hunt, 2007). The results of chapter 2
identified that this struggle may in fact be preventing Australian FTOs from operating
a competency-based training system in the true sense. When the results are viewed
as a whole it appears that individual variation is occurring, albeit without a normal
distribution due to the minimum hour requirement truncating the lower end of the
data.

However further analysis of the data by function of FTO showed that there appeared
to be a stronger effect of both the syllabus used by the training school, often due to
a client airline’s specifications, and the regulatory required minimums, particularly
those for command (solo) hours. Despite the requirement for Australian FTOs to
have implemented a competency-based system, the data showed that the client
requirements on the FTO resulted in syllabus design which produced CPL students
who met the requirements of both the client as well as the Civil Aviation Safety
Authority (CASA) regulations. These CPL students could not be licensed in under
the minimum flight hours set by CASA, even if they were displaying a level of
competence commensurate with holding a CPL. This appears to have set up a
conflict between the desire of the FTOs to implement competency-based training
and the reality of the student pilots still having to accumulate the minimum flight
hours before CASA will issue their licence.
154
The lack of demonstrable evidence for a true competency-based training system in
aviation is of concern given it acts as a risk mitigator for the introduction of the Multi-
Crew Pilot Licence (MPL). While there may be valid reasons as to why a true
competency-based system cannot be seen or cannot be followed in aviation, the
concern is that ICAO is relying on a risk mitigator that doesn’t appear to exist in the
strictest sense. The results of chapter 2 showed that despite Australia having had a
requirement for competency-based training in aviation since 1999, there was not
much evidence of its use, in terms of what is meant by competency-based training.
It appears that the incompatibility of having an hour based requirement within a
competency-based training system (which by definition allows individuals to acquire
skill at their own rate) has meant that most FTOs have such a system in name only,
with the reality being a reliance on syllabus and hours. The existence of minimum
hours indicates a desire by the aviation industry to keep a ‘safety net’ in place,
which industry does not appear to want to alter in case this alters the safety of the
system, an argument that has been shown to constrict the on-going viability of
industries over the long term (Amalberti, 2001).

While competency-based training does appear at some levels to be suited to

aviation and indeed is being relied upon by ICAO (ICAO, 2011), it is important that
when introducing new training systems and licence types that regulators (both
internationally and locally) understand the system they currently have before they
attempt to amend it. In this case it is probable that competency-based training is not
occurring in its strictest sense. However, how much of a problem, if at all, that is to
the client airlines of these FTOs was not possible to determine. Indeed, the fact that
competency-based training in Australian aviation is closer to a hybrid model may not
be detrimental to aviation safety so long as the on-going checking within an airline
can adequately monitor a pilot’s performance.

Study 2 – Pilot performance in an airline environment

The study in chapter 3 set out to examine whether or not there was any evidence of
a performance difference based solely on the amount of hours a pilot had
accumulated, within the context of normal day-to-day flying operations in a major
airline. The results from this study showed that there was no significant difference
between the performance of low- and high-hour captains and first officers against

155
technical and threat and error management metrics. There were some differences in
non-technical performance against two metrics, however the magnitude of the
observed differences were only around 10% and need to be considered within the
context of the majority of the results demonstrating no significant difference. Low-
hour first officers were those with less than 1,500 hours, high-hour first officers had
greater than 1,500 hours. Low-hour captains were those with less than 5,000 hours
and high-hour captains those with greater than 5,000 hours.

Of particular interest was exploration of the performance of first officers with less
than 1500 hours total flight experience, and those with greater than 1500 hours,
particularly in light of the industry commentary on cadets, MPL and low-hour pilots
being potentially ‘unsafe’. In contrast to this expectation of industry, the results
showed that performance did not appear to dramatically differ once this particular
divide was crossed. This division of 1500 hours was chosen as it is the amount of
hours a pilot requires to be issued with an Air Transport Pilot Licence (which lets
them command an airliner) and is also the new requirement introduced by the
United States Congress, and enacted by the FAA, as the minimum hours needed for
American pilots to enter an airline as a first officer. The introduction of this threshold
in the US was in response to the DHC-8 accident at Buffalo. The investigation into
this accident highlighted a number of issues within the training system of that
particular airline, as well as many other issues affecting pilot performance such as
inadequate fatigue management (NTSB, 2010). However, this sort of regulatory
intervention seems at odds to the reality of airline operations seen in our study,
which highlights the dangers of reacting to public pressure following an accident
without examining whether or not the problem exists within the wider aviation
industry and not just within the one airline involved in an accident.

Study 3 – Pilot performance in normal and non-normal operations

In recognition of the need to examine more than just normal line operations within
an airline, the final study was conducted with 3 airlines to examine normal and non-
normal operations as well as whether or not the airline entry pathway of a pilot
influenced their performance. The results from this study, in chapter 4 showed that
both the low-hour and cadet pilot groups studied were deemed as proficient (and in
some cases exceeded the required standard), as their high-hour and direct-entry
peers were. There were differences in the performance of the low-hour and cadet

156
pilot groups compared to their peers. However, these differences showed up as a
higher amount of high-hour and direct-entry pilots exceeding the required standard.
These differences were against various metrics with leadership/command and
situation awareness the only metrics that appeared as a significant difference
between low- and high-hour pilots across all three airlines. It is reasonable to expect
that both situation awareness and leadership will be less developed in low-hour
pilots and first officers given the evidence in the literature relating to novice
performance in these areas (Endsley,1995; Flin, O’Connor & Crichton, 2008)
However, this difference was not seen for all three airlines when the data were
examined in relation to entry pathway. Overall, the results showed that there was no
systematic or consistent difference in performance between the groups across all
three airlines, meaning it is unlikely that there is an industry wide problem that is
specific to low-hour or cadet pilots. However the study did highlight the areas in
which cadets and low-hour pilots were not performing as well, which could serve as
evidence based indicators of future training interventions within those airlines.

Indeed there are many reasons why the low-hour and cadet pilots received lower
ratings during these simulator and line flying checks that may be independent of
their actual competence. However, for the purposes of informing the aviation
industry of potential training interventions that may assist low-hour and cadet pilots,
these results were considered and explored as being significant differences that
were directly reflective of individual competence. In this way, the results allow the
training system within these airlines to be examined, based upon the airline specific
differences, to allow any alterations that may assist these pilots in improving their
performance above that required to meet the standard.

5.3 Competency-based training

Training and assessment

There are a number of issues within competency-based training which do not aid in
its application, especially in complex systems such as aviation. One of the major
on-going problems highlighted in the literature is a lack of consistency in the
definition of competence and how that is applied during an assessment (Hyland,
1994; Cowan et al., 2007; Guthrie, 2009). Many airlines use a scale of performance
that ranges from 1-5, with 1 being a fail due to the pilot not being proficient or
competent. The rest of the scale for most airlines represents the pilot being

157
proficient or competent but assigns differing levels to their performance, where 2
may be ‘baseline competent’, 3 is acceptable, 4 acceptable for captains, with 5
being excellent (Todd & Thomas, 2013). This is an interesting twist on competence
as surely you are competent or you are not. The idea that you can ‘rate’
competence, that someone is a little bit more competent than their peer is
complicating a system that needs no further complexity added to it. Grading a pilot
based upon their performance may be useful to address areas in which they are not
performing as well as their peers. But the idea that you can assign differing levels of
competence to a pilot will do nothing to reduce the confusion around the use of the
term competent within aviation.

To this end, the issue is less about ‘competence’ per se and is more related to skill
and proficiency. As passengers what we would expect of pilots is that they are all
competent and ideally, also highly proficient. The marking system in place in most
airlines is a good indication of the level of skill of an individual within the larger
cohort.

It is worth emphasising that each pilot who passes a check is by necessity

‘competent’. However what is reflected in the rating scale is an individual’s skill and
proficiency rather than ‘level of competence’ (as discussed in chapter 4). By moving
away from the labelling associated with competence, individuals can now move
beyond being just ‘competent’ and further develop this skill into expertise.

Within competency-based training, one of the limitations identified is the underlying

assumption that by breaking down a role into a multitude of tasks and then
assessing someone against each of those tasks will ensure that they are competent
for the entirety of the role. This is not necessarily the case, as highlighted by Gonczi
(1994). However, it should be noted that if an individual is rated higher than a peer
during an assessment, it is reasonable that this increased level of skill is indicative
of a higher level of expertise. One benefit of experts is their capacity to handle off-
nominal or non-normal events by virtue of having ‘something in reserve’. It is this
additional level of knowledge, skill and ability that aids experts in challenging and
critical situations and this element is not currently captured in the behaviouristic
approach to competency-based training which the AVI08 training package for the
aviation industry reflects. The adoption of a holistic assessment approach may

158
assist in identifying and assessing expertise in professional domains such as in the
area of pilot performance.

It has been shown that assessment of competence is an area mired with complexity
and limitations. One of the topics which has dominated the literature is
understanding what it is that is being assessed (Saunders, 1998; Thomson,
Saunders & Foyster, 2001; Thomson, Mathers & Quirk, 1996; Voorhees, 2001). In
aviation, assessments occur regularly within an airline environment, including the
pilot’s initial type rating in a simulator, followed by a check to line assessment
(assessing whether or not the pilot can conduct day-to-day line operations
proficiently), recurrent simulator proficiency checks and annual line checks. Overall,
airline pilots undergo a large amount of assessment over a year and indeed the
course of their career. However, the obvious question is what is it that is being
assessed in these multiple checks? At first glance it would appear that the pilots are
demonstrating proficiency against a number of metrics that include technical, non-
technical and regulatory required manoeuvres which are designed to give an airline
a level of assurance that the pilot’s performance is at an acceptable standard. The
results of the third study (chapter 4) show that this is largely the case.

The subject of assessing competence in complex systems such as aviation has

raised concerns that there is a conflict between competent performance
demonstrated for assessment that occurs within a known and predictable setting,
and the performance required from pilots in complex, novel situations such as
during an emergency (Capper, 2000). Given the limitations of assessing overall
competence through assessment against individual tasks, coupled with the
predictable assessment environment, is this level of assurance actually valid? It
would seem equally possibly that the training and checking system is producing
pilots who can deal with day-to-day normal flight operations and who can respond to
non-normal events during a simulator proficiency check, but this cannot be a
guarantee that the same pilot will be able to successfully deal with an in-flight
emergency that is unexpected and/or untrained for. For example, a pilot may
struggle to get through initial or recurrent training due to a number of factors,
however if they do pass the training but subsequently it is revealed (for instance
from an accident investigation) that they could only cope in normal or routine
operations, then this raises questions as to how the checking system might be able

159
to discover this deficiency. If the training environment presents something new and
unexpected to a pilot, they may not be able to demonstrate the required skills or
reaction, but this is often, and easily, explained away as the nerves or low skill level
expected of a trainee (depending on the stage of training).

While this is a valid concern of competency-based training, it is also a concern that

is relevant to an hours-based traditional model of training. That is, if a pilot
accumulates sufficient hours to qualify for the licence type they are training for, and
are deemed to be able to pass a flight test, then they will be tested for licence issue
during a test flight conducted with the aim of assessing the pilot’s skills and
competence. However, it can be argued that the pilot will know what areas they are
to be tested on and their training will have prepared them to demonstrate a level of
ability in these areas. As such, unless they encounter a genuine emergency event
during the check flight, even the traditional flight test approach will not be able to
assess whether or not a pilot will react correctly during a true emergency event.
What is necessary in pilot training is to ensure the underpinning foundation of
knowledge exists to allow pilots to develop the relevant knowledge-based, rule-
based and skill-based competencies to ensure they have a good chance of
performing well in novel, unexpected events (Rasmussen, 1983; Capper, 2000).

In this sense, it seems intuitive that you would design a training system in aviation to
ensure that a pilot is competent, not merely that they meet a target number of hours.
It would seem far more important for a pilot to continue to experience a training
environment that will support their skill development until they reach the point in
which it can be stated that they are or are not competent to perform their role.

Guidance from IATA in relation to competency in aviation highlights the fact that
competency is achieved through task analysis and applied through instructional
design, specifying that it is not achieved through a “box ticking exercise” (IATA,
2011). That is, the design of a competency-based system in aviation needs to
recognise that competence is not a distillation of tasks but rather provides
competency relating to higher level components of the job, such as weather related
decision-making. Skills such as contingency planning relating to systems failures
should be seen as competencies and implementing this approach in aviation will
add a level of sophistication to the concept of ‘pilot competencies’ which appears to

160
be currently missing. Implementation of such a system would appear to come a lot
closer to the reality of competency-based training however if this continues to occur
commensurate with a licensing system that requires a minimum number of hours
before a pilot can be licensed, then it is hard to see how the aviation industry will
break away from the belief that the accumulation of hours equals competence.

5.4 Pilot performance

This belief that flight hours experience equates to competence is a common

perception within the aviation industry which states that the skill of a pilot can be
determined from the amount of hours that pilot has accumulated. In this way, flight
hours have become synonymous with experience. Indeed much of the research in
aviation regarding pilot performance, particularly decision making, has focused upon
the amount of hours the pilot has accumulated as a divider or determinate of
performance. However, these studies focused upon general aviation and air taxi
services, which differ significantly from major airlines (Bazargan & Guzhva, 2011; Li
et al., 2003). As such the findings in this thesis, while in agreement with some of the
research, are in contrast to some of the studies which have found a difference in
performance based on flight hours. Specifically contrasting the findings relating to
the performance of low-hour pilots (Wiegmann et al., 2002; Bazargan & Guzhva,
2011; Li et al., 2003) and in agreement with research which has found other
determinates of performance stronger than total flight hours alone (Wiggins &
Bollwerk, 2006; Wiggins & O’Hare, 2003;O’Hare & Chalmers, 1999; Groff & Price,
2006).

Certainly the contribution of technical and non-technical skills to a pilot’s overall

ability in a flying role are both very important; however it remains undetermined as
to whether the general aviation experience can ‘corrupt’ non-technical skills, making
it harder for pilots to integrate into multi-crew environments, where decisions are
shared, communication is vital and a shared understanding of flight management is
crucial. Given the success of general aviation pilots in airlines for many years, and
the proficiency of direct entry pilots in chapters 3 and 4, it is obviously not
impossible for pilots from this pathway to learn and adhere to these concepts.
However, any reform of pilot training would be remiss in not examining this factor
given that we need pilots in airlines who can draw on both their technical and non-

161
technical knowledge to successfully handle in-flight non-normal events (Harris,
2011; Flin, O’Connor & Crichton, 2008).

General aviation is an historically important pathway to the airlines, particularly in

Australia, and obviously offers some benefits to some individuals, perhaps in
allowing pilots to further enhance their existing skills acquired during training.
However it is possible that recruitment to the airlines is from the higher end of
general aviation pilot performance group in terms of consistency of performance
given the lack of on-going checks and balances in the majority of general aviation,
unlike the frequent testing and checking within an airline.

It should be noted that in the same way that a pilot having accumulated 5,000 hours
in general aviation is no guarantee of their performance in an airline, the same may
be true of a pilot who has built up 5,000 hours in an airline, however the airline
check and training system will provide a level of skill assurance in terms of how
closely monitored and checked the pilots are during their career. Given the level of
scrutiny in the airlines it would be surprising to see a pilot who had built up 5,000
hours in an airline suddenly perform poorly solely because their hours were
accumulated within an airline. It is far more likely that the airline’s check and training
system would have identified that the pilot was having problems earlier than the
5,000 hour point and either successfully re-trained them, or facilitated their exit from
the airline system This is supported in part by the findings in chapter 4, which
showed a positive effect on performance based, in part, on the time cadets had
spent in an airline environment.

Another question which arises given the proliferation of cadet schemes and the
introduction of the MPL is does it make a difference if a pilot has never flown single-
pilot commercial operations prior to entering an airline? Specifically, do we need
pilots to be operating in the world of single-pilot operations when their flying will
actually occur as part of a crew? If the only world the MPL pilots have ever known is
the world they operate in for the rest of the career, is that a benefit? In line with
some aspects of the situated learning literature, it is intuitive that this continuation
would create an entry, training and experience pathway to support pilots in what is
becoming an increasingly complex role, insofar as interaction with automation and
the growing complexity of modern aircraft is concerned, however this concept needs

162
to be tested. The existing cadet schemes in various airlines around the world would
support the view that single-pilot operations are in fact not a necessary part of
experience for airline pilots.

However, once again, the results of chapters 3 & 4 did not show a dramatic
difference in performance as a function of total flight hours alone, which is
consistent with previous research, some of which has highlighted the more powerful
effect of recent flying rather than total flying experience on performance, and that
task specific experience was a more stable predictor of performance than total
experience (Wiggins & Bollwerk, 2006; Wiggins & O’Hare, 1995). This again
highlights the need for airlines, training schools and regulators to focus upon the
concept of individual pilot competence as something that is separate and
distinguishable from the total hours accumulated by a pilot.

The desire of parts of the aviation industry to persist with an hours based system
can be seen in the comments by the Australian International Pilot Association’s
(AIPA) submission to the Australian senate committee on pilot training, which stated
a concern that any move away from airlines recruiting pilots from general aviation
(who have accumulated several thousand hours) to a system which takes cadets
from a training school environment “…has and will continue to increase the risk of
an aviation accident” (Australian International Pilot Association, 2010, p3). This
statement is surprising given that at least two high profile airline accidents (The
DHC-8 at Buffalo and Air France 447) involved pilots who had accumulated their
hours in general aviation prior to entry to the airlines (National Transportation Safety
Board, 2010; Bureau d’Enquêtes et d’Analyses pour la sécurité de l’aviation civile
2012). Accidents such as that involving Air France flight 447 should act as a warning
against relying upon total flight hours to assure performance during non-normal
operations as the flight crew had a high number of total hours. In relation to cadet
performance, one of Australia’s most well-known in-flight incidents, an A380
uncontained engine failure near Singapore in 2010, had a first officer at the controls
that had entered the airline as a cadet some 15 years previously (Australian
Broadcasting Corporation, 2011). This would suggest that pilot performance is made
up of more than just hours and relies on training, checking and individual skill and
competence because if cadets are inherently unsafe, then the first officer’s
performance should have been demonstrably affected by this. In relation to this

163
event it would seem to mean that the first officer was either inherently skilled
regardless of training and his time as a cadet did not adversely affect this, or,
perhaps his time as a cadet helped his response. Certainly the findings from the
studies in chapters 3 & 4 highlight that low-hour pilots are no less proficient than
their high hour peers. While there were some differences in performance, they did
not support the assertion of AIPA that cadets and/or low-hour pilots will lead to
accidents.

In support of the need for a focus on the quality of training, not just the hours
accumulated by pilots, the president of the Flight Safety Foundation stated
“Mandating an arbitrary number of hours experience required to be in a cockpit
makes the dangerous assumption that specific knowledge will be obtained simply
due to hours in the air. This leaves too much to chance” and “There are countless
examples of pilots with many thousands of hours, who lacked the critical knowledge
to avert a tragedy” (Voss, 2012).

5.5 Future training

The results of the studies of this thesis, combined with previous research and in light
of recent accidents that have reflected performance problems in high stress,
complex and confusing situations concur with this observation and should serve as
a warning against assuming that hours equates to performance. The aviation
industry needs to focus on providing contextually relevant training to help pilots
perform appropriately to sudden, unexpected and complex in-flight events. Simply
relying upon the number of hours a pilot had accumulated to ensure this
performance is not necessarily adequate.
Some of the approaches within the aviation industry to develop the contextually
relevant training include Federal Aviation Administration (FAA) pilot training syllabus
amendments and research which considers the effect of startle and stress on pilot
performance (FAA, 2013; Martin, Murray & Bates, 2012; Casner, Geven & Williams,
2013). This research has highlighted the need for pilot training which focuses on
teaching pilots how to handle unexpected critical events, in order to both raise the
expectancy of such an event occurring and also to provide pilots with the skills they
will need to deal with these situations. The FAA training syllabus revision
emphasizes pilot’s manual handling skills and adds safety-critical tasks to the

164
training requirements, such as recovery from a stall or in-flight upset event, similar
to that which occurred in the Colgan Air accident at Buffalo (FAA, 2013). In addition,
training models which focus on the importance of developing and assessing core
skills for pilots have been implemented within airlines with encouraging results
(Mavin & Dall’Alba, 2010; Munro & Mavin, 2012).

Innovations in pilot training, such as the MPL, may result in higher numbers of low-
hour cadet pilots entering airlines, which may provide an opportunity for airlines to
train these cadets in expert approaches to non-normal situations. There is a
significant amount of research into areas such as decision making that has provided
methods to elicit information from experts with the view of understanding the
differences between experts and novices and in the future, teaching these expert
strategies to novices (Klein, Calderwood & MacGregor, 1989; Wiggins & O’Hare,
1995; Wiggins et al., 2002). Use of these methods could prove to be useful in
determining the strategies used by experts and these, along with training
innovations that incorporate modern technology, may provide a way to pass on
expertise to student pilots, particularly in a classroom setting (Capper, 2000; Fanjoy
& Young, 2001). The need to tailor airline training to the strengths of pilot groups
based on experience has been recommended previously, along with the caution
against treating all flight hours experience as equal and the need to ensure the
selection process addresses the tasks we need pilots to perform, not their likelihood
of completing the training (Bramble & Koonce, 1998; Damos, 1996).

The differences in how experts and novices make decisions appear to centre on the
cues they seek and use as well as knowing when to stop looking. Experts rely on
fast decisions and ‘best fit’ solutions, that is, solutions which fit or solve the majority
of the problem they are facing. Novices, in contrast, continue searching each
plausible scenario until they have developed the best possible solution and do not
see a part-fit scenario as being adequate (Wiggins & Bollwerk, 2006; Klein, 1997;
Schriver, Morrow, Wickens & Talleur, 2008). The use of cues in particular may
explain, in part, why the results from chapter 4 showed a difference between low-
and high-hour pilots with respect to the non-technical skills of leadership/command
and situation awareness.

165
To ensure that a training system will promote skill acquisition, there should be a
commonality between the tasks being taught, the environment in which the learning
is occurring and the domain in which an individual performs (Fleishman & Bartlett,
1969; Rumelhart & Norman, 1981; Ackerman, 1988). It has also been demonstrated
that performance on a task will improve with practice, most probably due to the fact
that the task has been practiced until automaticity is reached, which has the benefit
of reducing both the effort and attention required on the part of the individual in
performing the task (Fleishman & Mumford, 1989). This is of particular relevance to
pilot training and has been the cornerstone of training pilots to handle emergency
situations with little or no notice in a time-critical phase of flight. For instance, the
required responses to an engine failure at take-off are taught progressively through
the pilot training syllabus to ensure that students become familiar with the necessary
actions prior to conducting a solo flight. The necessity for this is obvious as it would
be inherently dangerous to send a pilot solo without equipping them with the ability
to respond to an emergency situation that may occur during take-off. As a result, the
response to such an event moves from the cognitively intensive knowledge-based
reaction to an automatic, skill-based response over the course of the student’s
training, providing a strong, automatic response that can be utilised in such an
emergency. While development of this particular skill is a necessary part of training,
the potential exists for student pilots to practice some tasks to the level of
automaticity that will be detrimental to them in future operations. This ‘strong but
wrong’ response may result in a pilot reverting to a behaviour or reaction that is
appropriate in the context in which it was learnt, but completely inappropriate for the
situation they now face (Reason, 1991, p75).

Even if an individual does acquire the relevant and necessary skills to perform a
task, their performance can also be adversely affected when fatigued, confused or
stressed. An example of degraded performance under confusing or stressful
situations is evident in both the Colgan Air accident at Buffalo, New York, in which
the captain reacted inappropriately to the stalling situation by using an incorrect
control input due to confusion about what kind of stall it was, hence exacerbating the
event, and the AF447 flight in which the crew became confused about whether the
aircraft was stalling or in an overspeed situation and also reacted inappropriately
(National Transportation Safety Board, 2010; Bureau d’Enquêtes et d’Analyses pour
la sécurité de l’aviation civile, 2012).

166
Another major innovation in the MPL training regime is the incorporation of
increased simulator time, with an equivalent reduction in the time spent in an actual
aircraft during training. The MPL substitutes the time in a small single-engine
aircraft, and to some extent small twin-engine aircraft, with an increased amount of
time in smaller high-performance aircraft simulators and large, jet aircraft simulators.
This substitution has created a lot of angst in industry yet the literature reviewed in
this thesis shows a positive effect of simulation on student pilot learning and
performance (Damos & Lintern, 1981; Lintern, 1980; Caro, 1988; Lintern, 1991;
Lintern & Koonce, 1992; Wiggins & Crognale, 2003; Wightman & Lintern, 1985;
Lindo, Deaton, Cain & Lang, 2013). It is also aligned with the concept of situated
learning and the ability to develop novices into experts by manipulating the
environment and the learning strategies they are taught (Lave, 1991; Anderson et
al., 1996; Clark III, 2001).

By exposing MPL pilots to increased amount of simulation, it is reasonable to

assume that they will be gaining a higher exposure to the airline operational
environment and the performance characteristics of large jet aircraft. This is not a
substitute for actual hours in these jet aircraft and the students will still get that in
their base and line training, which all new airline pilots have to do, regardless of
entry pathway or licence type. For MPL pilots, the base check is a requirement they
must pass in order to become licensed, as such, the system is ensuring that MPL
pilots can fly the actual aircraft before they proceed to line training. There is no such
guarantee with CPL pilots, although there is the on-going assumption that they will
be able to fly high capacity (airline) jet aircraft because of the amount of flight hours
they have accumulated. That is, those pilots will possess a CPL regardless of
whether or not they can fly a jet aircraft and indeed many CPL pilots won’t fly for an
airline for a range of reasons.
However MPL pilots must demonstrate proficiency on the aircraft before they will be
licensed. If anything, the MPL regime appears more demanding of the students than
the traditional CPL route. If the MPL pilots fail to perform during their base training
they don’t just miss out on entry to an airline, they do not attain the licence they
were training for either.

167
5.6 Pilot performance in non-normal scenarios

With regard to pilot training and the concept of experience, it is beneficial to break
this concept into three “tiers”. The first are the “stick and rudder skills” the physical
act of flying; the second are the flight management skills (decision making, non-
technical skills etc.); and the third is the resilience a pilot has from their skill and/or
experience in which to successfully deal with uncertain or abnormal situations.

The physical act of flying is the process which becomes automatic over time,
requiring very little conscious attention or oversight by the pilot once it has been
successfully consolidated (Rasmussen, 1983). As discussed above, this can lead to
responses which are deemed ‘strong but wrong’ and further compound an
emergency situation (Reason, 1991, p75). The extent to which these “hands on”
flying skills are affected by the type of training pathway a student participates in is
an area that requires further review. While it is very likely that pilots will not be able
to pass a relevant flight test without having automatized some of their performance,
it is possible that this base level of skill may not be as embedded as the aviation
system demands or has assumed from the training pathway the pilot followed.
Aviation as an industry needs to ensure that this foundation skill is not eroded by
changes to training regimes.

Of greater importance to the safety of operations is the ability of pilots to make

accurate and safe decisions, often in non-optimal or non-normal situations. The
degree to which a pilot has been exposed to decision making in known events will
play some part in how well they respond, which is a possible influence on the results
in chapter 4, which showed a benefit for direct entry first officers against some of the
more practised regulatory manoeuvres. In addition, the individual resilience and
coping mechanisms of the pilot may also assist performance, but previous practice
will not necessarily aid pilots when they face unexpected or un-trained for events
(McKinney & Davis, 2003). Given the increased complexity of the aviation system as
a whole, and aircraft in particular, it is important that pilots are able to respond to
surprising, confusing and uncertain events in a robust, efficient and effective
manner.

168
This thesis has examined some of the underpinning non-technical and threat and
error management skills used by low- and high-hour pilots and found very little
difference between the cohorts. However it is important to ensure that this level of
familiarity and proficiency in the “softer” side of flight, the domain of human factors,
has as firm a foundation as the physical manipulative skills pilots rely on without
conscious oversight. In fact, it is easy to argue that these attributes are more
important, especially in light of recent accidents in Australia that have highlighted
effective decision making and crew resource management in ensuring the safe
outcome of the flight following an emergency situation (ATSB, 2011a; ATSB, 2013).

In the past, pilot hours have been used as an indicator for these three distinct skills;
however the obvious question arising from this premise is, are flight hours alone
really the most effective way to judge these skills? Given this thesis has highlighted
the danger in relying on hours as a sole measure of skill, then perhaps the aviation
industry needs to consider what the most effective way of creating a pilot who
encompasses all of these very necessary skills is. Paradoxically, the main reason
for industry angst regarding the reduction in hours for future pilot training may very
well be the fact that the various parties are talking at cross purposes. One party may
be using experience as an indicator of manipulative skill, another as an indicator of
decision making skills. Despite this possible miscommunication, one question which
remains in the debate about flight hours assuring competence is does flying around
the outback of Australia in a small aircraft in general aviation really equip pilots to
understand and deal with an automation anomaly in an Airbus A330? And if it does
help, how does it do this? This is an area which needs to be addressed in future
research.

5.7 Future research

The complexity of the airline environment appears to have very little correlation to
the world of general aviation in many respects. While single-pilot multi-engine flight
is certainly one of the most demanding roles a pilot can do, is it accurate to say that
a pilot who performs well in this environment will do well in an airline environment?
Anecdotally the answer appears to be yes, however is this actually a function of the
individual’s personality and skill – i.e. the reason they perform well in the single-pilot
multi-engine general aviation role is the same reason they perform well in an airline
169
and not because the skill was built up in general aviation, but because it existed
from the start? There is a chance we are in fact confusing the end result with the
process and assuming we need the process when in fact the process did not alter
the pilot, the pilot possessed the relevant potential and skills from the start. The only
thing the process of going through the general aviation experience did was expose
them to an environment that is vastly different to an airline. The question arises
then, if this is the case, is it possible that the process may hinder the abilities of
pilots who are ‘average’ in that they find it harder to assimilate into an airline
environment after operating in the relative ‘freedom’ of general aviation where they
are in control of all decisions, actions and outcomes (from their perspective)? If this
is correct, it at least shows that the training and checking of the airline is working, it
is ensuring a level of competence amongst the pilots. It is for this reason that the
general aviation pathway to an airline needs further study in order to determine
what, if anything, is aiding general aviation pilots in the airline environment and what
is detrimental to their performance in an airline.

It must be noted that single-pilot operations is a separate issue to whether or not

student pilots need to go solo during their training. One of the biggest proposed
changes the MPL system will introduce is minimal solo time for pilots during their
training. There may well be a confidence element and solo decision making
influence that is necessary for pilots to be exposed to, but if this occurs solely during
training, will it adversely impact the ability of a pilot once they are in an airline?
Outside of aviation this requirement to go solo in order to ‘build confidence’ and
improve decision making does not appear to exist. In comparable teamwork
domains such as healthcare and maritime, it is not considered necessary for
surgeons or ship captains to operate solo in order to have improved confidence and
performance. So why does this belief prevail in aviation? This is an area that needs
further research to help ensure that future pilot training advances are designed to
produce the best possible pilot candidate from that training by removing
unnecessary training elements. This can then allow for an evidence-based approach
to training which addresses what pilots need for improved performance and allows
the removal of legacy items that are no longer needed.

Irrespective of the noted benefits of simulation and transfer of training, in relation to

the MPL, the question still remains about whether substituting flying hours and/or

170
time in a “real” aircraft (albeit a small piston-engine aircraft) for a simulated jet
aircraft will ensure skill acquisition and ultimately, safe performance. This is an
important area of future research and should be examined as soon as practical with
a cohort of MPL students and their CPL equivalents.

Another question relevant to on-going innovation within pilot training is: does a pilot
get through a training system, either the more traditional model or a competency-
based training system, because of their inherent abilities (both technical and non-
technical), regardless of the training regime? While general aviation was specifically
discussed above, a broad look at the military and general aviation entry pathways
and training systems may assist in being able to identifying the benefits (if they
exist) of both of these training systems on pilot performance in an airline. As such,
any research which could highlight a benefit from either system would be
advantageous to ensure any future pilot training innovation incorporated those
benefits. It may not be possible to integrate many or all elements of the military
system into the civilian training world due, in part to the differences in the nature of
the aircraft involved and the types of flying. However the levels of complexity
between military aircraft and large jet aircraft are comparable and it would be useful
to determine the extent to which this level of complexity is being addressed within
military pilot training.

171
5.8 Conclusion

By examining the current pilot training systems, as well as rigorously evaluating the
performance of pilots trained under new training regimes, our understanding of the
benefits of current and future training innovations will improve. It is also important
that any new training system ensures all elements of successful pilot performance
(technical, non-technical and flight management) are addressed. By exploring the
principles highlighted in this thesis, along with future research, the aviation industry
can ensure all future pilots are as well-equipped as possible to face the challenges
of flying complex aircraft in demanding conditions, safely.

172
References

173
Ackerman, P.L. (1988) Determinants of individual differences during skill acquisition:
Cognitive abilities and information processing. Journal of Experimental
Psychology: General, 117(3), 288-318.

Ackerman, P.L. (1992). Predicting individual differences in complex skill acquisition:

Dynamics of ability determinants. Journal of Applied Psychology, 17(5), 598-
614.

Ackerman, P.L. & Cianciolo, A.T. (2000). Cognitive, perceptual-speed, and

psychomotor determinants of individual differences during skill acquisition.
Journal of Experimental Psychology: Applied, 6(4), 259-290.

Ackerman, P.L. & Cianciolo, A.T. (2002). Ability and task constraint determinants of
complex task performance. Journal of Experimental Psychology: Applied,
8(3), 194-208.

Adams, R.J. & Ericsson, A.E. (2000). Introduction to cognitive processes of expert
pilots. Journal of Human Performance in Extreme Environments, 5(1), 44-62.

Alessi, S. (2000). Simulation design for training and assessment. In O’Neil, H. &
Andrews, D.H. (Eds.) Aircrew training and assessment, (pp. 197-222).
Lawrence Erlbaum Associates: Mahwah, NJ, USA.

Anderson, J.R., Reder, L.M. & Simon, H.A. (1996). Situated learning and education.
Educational Researcher, 25(4), 5-11.

Andersson, P., Ekman, M. & Edman, J. (2003). Forecasting the fast and frugal way:
A study of performance and information-processing strategies of experts and
non-experts when predicting the World Cup 2002 in soccer. Unpublished
manuscript, Center for Economic Psychology, Stockholm School of
Economics, Sweden.
174
Australian Broadcasting Corporation (2011). Four corners program transcript from
the report “QF 32” first broadcast Monday 28 March 2011. Retrieved 13
October 2013 from
http://www.abc.net.au/4corners/content/2011/s3175853.htm

Australian and International Pilots Association (2010). Australian and international

pilots association submission and response to the terms of reference; pilot
training and airline safety including consideration of the Transport Safety
Investigation Amendment (Incident Reports) Bill 2010. AIPA, Mascot, NSW,
Australia.

Australian National Training Authority (1999). Research at a glance: competency-

based training in Australia. Leabrook, Australia: National Centre for
Vocational Education Research.

Australian Transport Safety Bureau (2010). Midair collision Latrobe valley

aerodrome, Victoria, 1 December 2007. Canberra, ACT, Australia.

Australian Transport Safety Bureau (2011a). Tailstrike and runway overrun,

Melbourne airport, Victoria, 20 March 2009, A6-ERG, Airbus A340-541.
Canberra, ACT, Australia.

Australian Transport Safety Bureau (2011b). In-flight upset, 154 km west of

Learmonth, WA, 7 October 2008, VH-QPA, Airbus A330-303. Canberra,
ACT, Australia.

Australian Transport Safety Bureau (2012). Loss of control – Embraer S.A. EMB-
120ER Brasilia, VH-ANB. Canberra, ACT, Australia.

Australian Transport Safety Bureau (2013). In-flight uncontained engine failure,

Airbus A380-842, VH-OQA overhead Batam Island, Indonesia, 4 November
2010. Canberra, ACT, Australia.

175
Australian Qualifications Framework (2010). The Australian qualifications
framework. Retrieved 2 October 2010 from
http://www.aqf.edu.au/AbouttheAQF/TheAQF/tabid/108/Default.aspx

Australian Qualifications Framework Advisory Board (2007). Australian qualifications

framework: implementation handbook. Carlton, Vic, Australia.

Baldwin, T.T. & Ford, J.K. (1988). Transfer of training: A review and directions for
future research. Personnel Psychology, 41(1), 63-105.

Baltes, P.B., Reese, H.W., & Lipsitt, L.P. (1980). Life-span developmental
psychology. Annual Review Psychology, 31, 65-110.

Barrett, P. (2012). The international pilot training consortium: A coordinated effort to

improve pilot, instructor and evaluator training. ICAO Training Report, 2(2), 6-
9.

Bazargan, M. & Guzhva, V.S. (2010). Impact of gender, age and experience of
pilots on general aviation accidents. Accident Analysis and Prevention, 43,
962-970.

Bureau d’Enquêtes et d’Analyses pour la sécurité de l’aviation civile (2012). Final

report on accident on 1st June 2009 to the Airbus A330-203 registered F-
GZCP operated by Air France flight AF447 Rio de Janeiro – Paris. Bureau
d’Enquêtes et d’Analyses pour la sécurité de l’aviation civile. Paris, France.

Beaubien, J.M. & Baker, D.P. (2004). The use of simulation for training teamwork
skills in healthcare: How low can you go? Quality and Safety in Health Care,
13, 51-6.

Bell, H.H. & Waag, W.L. (1998). Evaluating the effectiveness of flight simulators for
training combat skills: A review. International Journal of Aviation Psychology,
8, 223-42.

176
Biemans, H., Nieuwenhuis, L., Poell, R., Mulder, M. & Wesselink, R. (2004).
Competence-based VET in the Netherlands: background and pitfalls. Journal
of Vocational Education and Training, 56(4), 523-538.

Billett, S., McKavanagh, C., Beven, F., Angus, L., Seddon, T., Gough, J., Hayes, S.,
& Robertson, I. (1999). The CBT decade, teaching for flexibility and
adaptability, an overview. NCVER, Leabrook, SA, Australia.

Bloch, B., Clayton, B., & Favero, J. (1995). Who assesses? In Hall, W.C. (Ed.) Key
aspects of competency-based assessment (1995). NCVER, Leabrook, SA,
Australia.

Boeing Commercial Airplanes (2013). Current market outlook 2013-2032. Boeing

Commercial Airplanes, Seattle, WA, USA.

Boreham, N. (2002). Work process knowledge, curriculum control and the work-
based route to vocational qualifications. British Journal of Educational
Studies, 50(2), 225-237.

Bowen-Clewley, L. (2000). Assessing against competency standards in the

workplace – a New Zealand perspective. In Arguelles, A. & Gonczi, A. (Eds.)
Competency based education and training: a world perspective (2000).
Noriega Editores, Balderas, Mexico.

Bowers, C., Jentsch, F., Baker, D., Prince, C. & Salas, E. (1997). Rapidly
reconfigurable event-set based line operational evaluation scenarios.
Proceedings for the 41st Annual Meeting of the Human Factors and
Ergonomics Society, 912-15.

Bramble Jr, W.J. & Koonce, J.M. (1998). The path to airline employment: Flight
experience and performance in a full-mission flight simulation. Proceedings
for the 42nd Annual Meeting of the Human Factors and Ergonomics Society,
797-800.

177
Brannick, M.T., Prince, C. & Salas, E. (2002). The reliability of instructor evaluations
of crew performance: Good news and not so good news. International
Journal of Aviation Psychology, 12, 241-61.

Bridges, D. (1996). Competence-based education and training: Progress or villainy?

Journal of the Philosophy of Education: Great Britain, 30(3), 361-376.

Briggs, G.E., Fitts, P.M. & Bahrick, H.P. (1957). Effects of force and amplitude cues
on learning and performance in a complex tracking task. Journal of
Experimental Psychology, 54(4), 262-68.

Briggs, G.E. & Waters, L.K. (1958). Training and transfer as a function of
component interaction. Journal of Experimental Psychology, 56(6), 492-500.

Brown, J.S. & Duguid, P. (1991). Organizational learning and communities-of-

practice: Towards a unified view of working, learning and innovation.
Organization Science, 2(1).

Bryant, D. (2002). Making naturalistic decision making “fast and frugal”. Defence
Research Development Canada. Toronto, Canada.

Bryant, D.J., Webb, R.D.G. & McCann, C. (2003). Synthesizing two approaches to
decision making in command and control. Canadian Military Journal, Spring
2003.

Bureau of Air Safety Investigation (1994). Investigation report 9301743, Piper PA-
31-350 chieftain VH-NDU, Young, NSW, 11 June 1993. Department of
Transport, Canberra ACT.

Bureau of Infrastructure, Transport and Regional Economics (2012). Statistical

report aviation: General aviation activity 2010. Department of Infrastructure
and Transport, Canberra, ACT.

Bürki-Cohen, J., Soja, N.N. & Longridge, T. (1998). Simulator platform motion – The
need revisited. The International Journal of Aviation Psychology, 8, 293-317.
178
Camuffo, A. & Gerli, F. (2004). An integrated competency-based approach to
management education: an Italian MBA case study. International Journal of
Training and Development, 8(4): 240-257.

Cannon-Bowers, J.A. & Salas, E. (1998). Individual and team decision making
under stress: Theoretical underpinnings. In Cannon-Bowers, J.A. & Salas, E.
(Eds.) Making Decisions under Stress: Implications for Individual and Team
Training (1998). American Psychological Association, Washington, DC.

Capper, P. (2000). Understanding competence in complex work contexts. In

Arguelles, A. & Gonczi, A. (Eds.) Competency based education and training:
a world perspective (pp. 147-172). Noriega Editores, Balderas, Mexico.

Caro, P.W. (1973). Aircraft simulators and pilot training. Human Factors, 15: 502-9.

Caro, P.W. (1988). Flight training and simulation. In Wiener, E.L. & Nagel, D.C.
(Eds.) Human Factors in Aviation (1988). Academic Press, San Diego, CA,
USA.

Cartledge, D., & Watson, M. (2008).Creating place: Design education as vocational

education and training. NCVER, Adelaide, SA, Australia.

CASA (2008). Day (VFR) Syllabus – aeroplanes. Civil Aviation Safety Authority,
Canberra, Australia.

CASA (2013). Civil Aviation Legislation Amendment Regulation 2013 (No.1). Civil
Aviation Safety Authority, Canberra, Australia.

Casner, S.M., Geven, R.W. & Williams, K.T. (2013). The effectiveness of airline pilot
training for abnormal events. Human Factors, 55(3): 477-485.

Causse, M., Dehais, F. & Pastor, J. (2010). Flight experience and executive
functions predict flight simulator performance in general aviation pilots. In: 4th
International Conference on Research in Air Transportation – ICRAT 2010,
04 June 2010, Budapest, Hungary.
179
Chabris, C. & Simons, D. (2010). The invisible gorilla and other ways our intuition
deceives us. Harper Collins: London, England.

Chapman, H. (1999). Some important limitations of competency-based education

with respect to nurse education: an Australian perspective. Nurse Education
Today, 19, 129-135.

Charness, N. & Tuffiash, M. (2008). The role of expertise research and human
factors in capturing, explaining, and producing superior performance. Human
Factors, 50, 427-432.

Chase, W.G. & Ericsson, K.A. (1981). Skilled memory. In Anderson, J.R. (Ed.)
Cognitive Skills and Their Acquisition (1981), pp 141-230. Lawrence Erlbaum
Associates: NJ, USA.

Chidester, T.R. (2007). Intramural monitoring. In Statler, I.C. (Ed.), The Aviation
System Monitoring and Modeling Project: A Documentation of its History and
Accomplishments: 1999-2005. NASA Technical Publication, TP-2007-
214556. National Aeronautics and Space Administration, Washington, D.C.

Clark III, J.F. (2001). Teaching the pilots of the new millennium: Adult cooperative
education in aviation education. Collegiate Aviation Review, 19(1), 29-40.

Clarke, D.D., Ward, P., Bartle, C. & Truman, W. (2006). Young driver accidents in
the UK: The influence of age, experience, and time of day. Accident Analysis
and Prevention, 38, 871-878.

Cowan, D.T., Norman, I. & Coopamah, V.P. (2007). Competence in nursing

practice: A controversial concept – A focused review of literature. Accident
and Emergency Nursing, 15, 20-26.

Cropley, M. (1995).Validity. In Hall, W.C. (Ed) Key aspects of competency-based

assessment (1995). NCVER, Leabrook, SA, Australia.

180
Curry, D.H., Caplan, P. & Knuppel, J. (1994).Transfer of training and adult learning
(TOTAL). Journal of Continuing Social Work Education, 6(1), 8-14.

Dahlstrom, N., Dekker, S. & Nahlinder, S. (2006). Introduction of technically

advanced aircraft in ab-initio flight training. International Journal of Applied
Aviation Studies, 6(1), 131-144.

Dahlstrom, N., Dekker, S., van Winsen, R. & Nyce, J. (2009). Fidelity and validity of
simulator training. Theoretical Issues in Ergonomics Science, 10, 305-14.

Damos, D.L. & Lintern, G. (1981). A comparison of single- and dual-task measures
to predict simulator performance of beginning student pilots. Ergonomics,
24(9), 673-684.

Damos, D.L. (1996). Pilot selection batteries: Shortcomings and perspectives.

International Journal of Aviation Psychology, 6(2), 199-209.

de Croock, M.B.M., Paas, F., Schlanbusch, H. & van Merrienboer, J.J.G. (2002).
ADAPT: Tools for training design and evaluation. Educational Technology
Research and Development, 50(4), 47-58.

Department of Education, Employment and Workplace Relations (2010a).The

history of VET. Department of Education, Employment and Workplace
Relations. Retrieved on 2 October 2010 from
http://www.training.com.au/Pages/menuitem3e365e26c4085888a392e51017
a62dbc.aspx

Department of Education, Employment and Workplace Relations Transport and

Logistics Industry Skills Council (2010a). AVI08 aviation training package
Volume 1: A guide for users. Department of Education, Employment and
Workplace Relations.

Department of Education, Employment and Workplace Relations Transport and

Logistics Industry Skills Council (2010b). AVI08 aviation training package
Volume 2: Units of competency. Department of Education, Employment and
Workplace Relations.
181
Dekker, S. (2011). Drift into failure: From hunting broken components to
understanding complex systems. Ashgate, Surrey, U.K.

Descy, P. & Tessaring, M. (2001). Training and learning for competence: second
report on vocational training research in Europe: synthesis report (revised
edition). Luxembourg: Office for Official Publications of the European
Communities.

Dick, W., Carey, L. & Carey, J. (2004). Systemic design of Instruction. Upper Saddle
River NJ: Pearson Education.

Dingus, T.A., Hulse, M.C., Mollenhauer, M.A., Fleishman, R.N., McGehee, D.V. &
Manakkal, N. (1997). Effects of age, system experience, and navigational
technique on driving with an advanced traveler information system. Human
Factors, 39(2), 177-199.

Dismukes, K. (2006). Concurrent task management and prospective memory: pilot

error as a model for the vulnerability of experts. In proceedings of the Human
Factors and Ergonomics Society 50th Annual Meeting, 2006, 909-913.

Dismukes, R.K., Berman, B.J. & Loukopoulos, L.D. (2007). The limits of expertise:
rethinking pilot error and the causes of airline accidents. Ashgate, Aldershot,
U.K.

Diwakar, V. (2002). Commentary: The baby is thrown out with the bathwater. British
Journal of Medicine, 325(7366), 695-696.

Dorsey, D., Campbell, G. & Russell, S. (2009). Adopting the instructional science
paradigm to encompass training in virtual environments. Theoretical Issues in
Ergonomics Science, 10: 197-215.

Dow, J. & Defalque, H. (2013). The MPL – A systems approach to pilot training –
Gains acceptance by states and industry. ICAO Training Report, 3(2), 31-3.

182
Downey, J.P., & Zeltmann, S. (2009). The role of competence level in the self-
efficacy – skills relationship: an empirical examination of the skill acquisition
process and its implications for information technology training. International
Journal of Training and Development, 13(2), 96-110.

Draganidis, F & Mentzas, G (2006). Competency based management: a review of

systems and approaches. Information Management & Computer Security, 14
(1), 51-64.

Dussault, C., Jouanin, J-C., Philippe, M. & Guezennec, C-Y. (2005). EEG and ECG
changes during simulator operation reflect mental workload and vigilance.
Aviation, Space, and Environmental Medicine, 76(4), 344-51.

Edwards, E. (1988). Introductory overview. In Wiener E.L. & Nagel, D.C. (Ed.)
Human Factors in Aviation (1988), pp 3-25. Academic Press, San Diego, CA,
USA.

Endsley, M.R. (1995). Toward a theory of situation awareness in dynamic systems.

Human Factors, 37(1), 32-64.

Epstein, R.M. & Hundert, E.M. (2002). Designing and assessing professional
competence. The Journal of the American Medical Association, 287(2), 226-
235.

FAA (2006). Advisory Circular 120-54A: Advanced qualification program (AQP).

Federal Aviation Administration, Washington, U.S.

FAA (2013). 14 CFR Part 121 Qualification, service, and use of crewmembers and
aircraft dispatchers. Department of Transportation, Washington DC, USA.

Fanjoy, R.O. & Young, J.P. (2001). The transfer of flight training procedures from an
advanced airline flight simulator to the classroom. Collegiate Aviation Review,
19(1), 41-8.

183
Feinstein, A.H. & Cannon, H.M. (2002). Constructs of simulation evaluation.
Simulation and Gaming, 33, 425-40.

Fisher, D.L., Laurie, N.E., Glaser, R., Connerney, K., Pollatsek, A., Duffy, S.A. &
Brock, J. (2002). Use of a fixed-base driving simulator to evaluate the effects
of experience and PC-based risk awareness training on drivers’ decision.
Human Factors, 44(2), 287-302.

Fitts, P.M. & Jones, R.E. (1947). Psychological aspects of instrument display: I:
analysis of 270 “pilot-error” experiences in reading and interpreting aircraft
instruments. Memorandum Report T6EAA-694-12A U.S. Air Force, Dayton,
Ohio.

Fitts, P.M. (1947). Psychological research on equipment design in the AAF.

American Psychologist, 2(3), 93-98.

Fleishman, E.A. & Hempel Jr, W.E. (1956). Factorial analysis of complex
psychomotor performance and related skills. The Journal of Applied
Psychology, 40(2), 96-104.

Fleishman, E.A. & Mumford, M.D. (1989). Abilities as causes of individual

differences in skill acquisition. Human Performance, 2(3), 201-223.

Fleishman, E.A. & Bartlett, C.J. (1969). Human abilities. Annual Review of
Psychology, 20, 349-380.

Flight Safety Foundation (2000). FSF ALAR briefing note 7.1 – Stabilized approach.
Flight Safety Digest, August-November 2000, 133-8.

Flin, R., O’Connor, P. & Crichton, M. (2008). Safety at the sharp end. Ashgate,
Surrey, England.

Garris, R., Ahlers, R. & Driskell, J.E. (2002). Games, motivation and learning: A
research and practice model. Simulation and Gaming, 33, 441-67.

184
Gerathewohl, S.J. (1969). Fidelity of simulation and transfer of training: A review of
the problem, AM69-24. Federal Aviation Administration, Washington, DC.

Gilling, J. & Graham, D. (2000). Making the difference: Competency based

education and training in Australian manufacturing industry. In Arguelles, A. &
Gonczi, A. (Eds.) Competency based education and training: a world
perspective (pp. 131-146). Noriega Editores, Balderas, Mexico.

Gigerenzer, G. (2008). Gut Feelings: The intelligence of the unconscious. New York
U.S.A.: Penguin Books.

Goettl, B.P. & Shute, V.J. (1996). Analysis of part-task training using the backward-
transfer technique. Journal of Experimental Psychology: Applied, 2(3), 227-
249.

Goh, J. & Wiegmann, D. (2002). Human factors analysis of accidents involving

visual flight rules flight into adverse weather. Aviation, Space, and
Environmental Medicine, 73(8), 817-22.

Gonczi, A. (1994). Competency based assessment in the professions in Australia.

Assessment in Education: Principles, Policy & Practice, 1(1).

Gonczi, A. (1999). Competency-based learning: A dubious past – an assured

future? In Boud, D and Garrick, J (Eds.) Understanding learning at work,
pp180-197. Routledge, London, England.

Gonczi, A. (2000). Review of international trends and developments in competency

based education and training. In Arguelles, A. & Gonczi, A. (Eds.)
Competency based education and training: a world perspective (pp. 15-40).
Noriega Editores, Balderas, Mexico.

Gopher, D., Weil, M. & Bareket, T. (1994). Transfer of skill from a computer game
trainer to flight. Human Factors, 36, 387-405

185
Goska, R.E. & Ackerman, P.L. (1996). An aptitude-treatment Interaction approach to
transfer within training. Journal of Educational Psychology, 88(2), 249-259.

Griffin, G.R. & Koonce, J.M. (1996). Review of psychomotor skills in pilot selection
research of the U.S. military services. The International Journal of Aviation
Psychology, 6(2), 125-147.

Groff, L.S. & Price, J.M. (2006). General aviation accidents in degraded visibility: A
case control study of 72 accidents. Aviation, Space, and Environmental
Medicine, 77(10), 1062-1067.

Grubb, W.N. & Lazerson, M. (2005). Vocationalism in higher education: The triumph
of the education gospel. The Journal of Higher Education, 76(1), 1-25.

Guthrie, H. (2009). Competence and competency-based training: what the literature

says. NCVER, Adelaide, SA, Australia.

Hager, P. (1995). Assessment of knowledge, attitudes and values. In Hall, W.C.

(Ed) Key aspects of competency-based assessment (pp. 47-58). NCVER,
Leabrook, SA, Australia.

Hager, P. (2000). Judgement and the law society of NSW specialist accreditation
scheme. In Arguelles, A. & Gonczi, A. (Ed.) Competency based education
and training: a world perspective (pp. 173-186). Noriega Editores, Balderas,
Mexico.

Hager, P. & Gillis, S. (1995). Assessment at higher levels. In Hall, W.C. (Ed) Key
aspects of competency-based assessment (pp. 59-72). NCVER, Leabrook,
SA, Australia.

Hamman, W.R. (2004). The complexity of team training: What we have learned from
aviation and its application to medicine. Quality and Safety in Health Care, 13
72-9.

Harris, D. (2011). Human performance on the flight deck. Ashgate, Surrey, England.
186
Harris, R., Clayton, B. & Chappell, C. (2007). Supporting vocational education and
training providers in building capability for the future. NCVER, Adelaide, SA,
Australia.

Hawkins, F.H. (1987). Human factors in flight. Avebury Technical, Aldershot,

England.

Hayes-Roth, F., Klahr, P. & Mostow, D.J. (1981). Advice taking and knowledge
refinement: An iterative view of skill acquisition. In Anderson, J.R. (Ed.)
Cognitive Skills and Their Acquisition (pp. 231-253). Lawrence Erlbaum
Associates, New Jersey, USA.

Hays, R.T. & Singer, M.J. (1989). Training effectiveness evaluation. In Hays, R.T. &
Singer M.J. (Eds.) Simulation Fidelity in Training System Design, pp. 112-59,
New York, NY: Springer-Verlag.

Helmreich, R.L. (2000). On error management: lessons from aviation. BMJ, 320,
781-5.

Helmreich, R.L., Klinect, J.R. & Wilhelm, J.A. (2001). System safety and threat and
error management: The line operations safety audit (LOSA). In Proceedings
of the Eleventh International Symposium on Aviation Psychology, 1-6.

Herbig, B. & Glockner, A. (2009). Experts and decision making: First steps towards
a unifying theory of decision making in novices, intermediates and experts.
Bonn 2009/2, Max Planck Institute for Research on Collective Goods, Bonn,
Germany.

Hu, Y. (2007). CAAC MPL Program – A training for the future. Presentation at the
ICAO Regional Symposium on the Implementation of Multi-Crew Pilot
Licence, Hong Kong, China, 23-24 August 2007.

Hunt, G. (2007). Competency-based systems in MPL training – Are we really

changing ab initio pilot training? Presentation at the ICAO Regional
Symposium on the Implementation of Multi-Crew Pilot Licence, Hong Kong,
China, 23-24 August 2007.
187
Hunter, D.R. (2001). Retrospective and prospective validity of aircraft accident risk
indicators. Human Factors, 43(4), 509-518.

Hyland, T. (1992). Expertise and competence in further and adult education.

Professional Development in Education, 18(1), 23-28.

Hyland, T. (1993). Professional development and competence-based education.

Educational Studies, 19(1), 123-129.

Hyland, T. (1994). Experiential learning, competence and critical practice in higher

education. Studies in Higher Education, 19(3), 327-340.

Hyland, T. (1997). Reconsidering competence. Journal of Philosophy of Education:

Great Britain, 31(3), 491-503.

IATA (2011). Guidance material and best practices for MPL implementation.
International Air Transport Association, Montreal, Canada.

ICAO (2010). Foundation of the International Civil Aviation Organization (ICAO).

International Civil Aviation Organization. Retrieved 19 May 2010 from
http://www.icao.int/cgi/goto_m.pl?icao/en/hist/history02.htm

ICAO (2006a). Annex 1 – Personnel licensing (10th edition). International Civil

Aviation Organization, Montreal, Canada.

ICAO (2006b). Procedures for air navigation services - Training (Doc 9868).
International Civil Aviation Organization, Montreal, Canada.

ICAO (2011). Update on ICAO’s competency-based approach to training and

assessment. RASG-APAC/1-IP/8. International Civil Aviation Organization,
Montreal, Canada.

188
IFALPA (2008). Position statement: Multi-crew pilot licence (MPL). International
Federation of Air Line Pilot Associations; 2007. Retrieved 30 May 2008 from
http://www.ifalpa.org/positionstatements/07POS01%20-
%20Multi%20Crew%20Licence%20(MPL).pdf

James, P. (2001). The double edge of competency training: contradictory

discourses and lived experience. Journal of Vocational Education and
Training, 53(2), 301-324.

James, P. (2002). Discourses and practices of competency-based training:

implications for worker and practitioner identities. International Journal of
Lifelong Education, 21(4), 369-391.

Jarvis, S. & Harris, D. (2007a). Looking for an accident: Glider pilots’ visual
management and potentially dangerous final turns. Aviation, Space, and
Environmental Medicine, 78(6), 597-600.

Jarvis, S. & Harris, D. (2007b). Accident rates for novice glider pilots vs. pilots with
experience. Aviation, Space and Environmental Medicine, 78(12), 1155-8.

Jarvis, S. & Harris, D. (2008). Investigation into accident initiation events by flight
phase, for highly inexperienced glider pilots. International Journal of Applied
Aviation Studies, 8(2) 211-224.

Jeffries, R., Turner, A.A., Polson, P.G. & Atwood, M.E. (1981). The processes
involved in designing software. In Anderson, J.R. (Ed.) Cognitive Skills and
Their Acquisition (pp. 255-283). Lawrence Erlbaum Associates, New Jersey,
USA.

Jentsch, F. & Bowers, C.A. (1998). Evidence for the validity of PC-based
simulations in studying aircrew coordination. International Journal of Aviation
Psychology, 8, 243-60.

Jones, D.G. & Endsley, M.R. (2004). Use of real-time probes for measuring situation
awareness. The International Journal of Aviation Psychology, 14(4), 343-367.

189
Kahneman, D. (2011). Thinking, fast and slow. Penguin, London, England.

Kasarskis, P., Stehwien, J., Hickox, J., Aretz, A. & Wickens, C. (2001). Comparison
of expert and novice scan behaviours during VFR flight. In Proceedings of the
11th International Symposium on Aviation Psychology. The Ohio State
University, Columbus, OH.

Kellie, D. (1998). Catching a wave: an Australian case study on building

competence in search of competitive advantage. International Journal of
Training and Development, 2(4), 259-275.

Kellie, D. (1999). The Australian way: competency-based training in the corporate

sector. International Journal of Training and Development, 3(2), 118-131.

Kennedy, Q., Taylor, J.L, Reade, G. & Yesavage, J.A. (2010). Age and expertise
effects in aviation decision making and flight control in a flight simulator.
Aviation, Space, and Environmental Medicine, 81(5), 489-497.

Klein, G. A., Calderwood, R. & MacGregor, D. (1989). Critical decision method for
eliciting knowledge. IEEE Transactions on Systems, Man and Cybernetics
19(3): 462-472.

Klein, G.A. (1993). A recognition-primed decision (RPD) model of rapid decision

making. In Klein, G.A., Orasanu, J., Calderwood, R. & Zsambok, C. (Eds.),
Decision making in action: Models and methods, pp.138-147. Norwood, NJ:
Ablex.

Klein, G. (1997). Developing expertise in decision making. Thinking and Reasoning,

3(4), 337-352.

Klinect, J., Murray, P., Merritt, A. & Helmreich, R. (2003). Line operation safety
audits (LOSA): Definition and operating characteristics. In Proceedings of the
12th International Symposium on Aviation Psychology, 663-668.

190
Knowles, M.S. (1984). Andragogy in action. Jossey-Bass, San Francisco, CA,
United States of America.

Langley, P. & Simon, H.A. (1981). The central role of learning in cognition. In
Anderson, J.R. (Ed.) Cognitive Skills and Their Acquisition (pp. 361-380).
Lawrence Erlbaum Associates, New Jersey, USA.

Lans, T., Wesselink, R., Biemans, H.J.A. & Mulder, M. (2004).Work-related lifelong
learning for entrepreneurs in the agri-food sector. International Journal of
Training and Development, 8(1), 73-89.

Lave, J. (1991). Situated learning in communities of practice. Perspectives on

socially shared cognition, 63, 82.

Leagans, J.P. (1964). A concept of needs. Journal of Cooperative Extension,

Spring, 89-96.

Leung, W.C. (2002). Competency-based medical training: review. British Medical

Journal, 325(7366), 693-696.

Li, G., Baker, S.P., Grabowski, J.G., Qiang, Y., McCarthy, M.L. & Rebok, G.W.
(2003). Age, flight experience, and risk of crash involvement in a cohort of
professional pilots. American Journal of Epidemiology, 157(10), 874-880.

Li, G. & Baker, S.P. (2007). Crash risk in general aviation. Journal of the American
Medical Association, 297(14), 1596-98.

Lindo, R.S., Deaton, J.E., Cain, J.H. & Lang, C. (2013). Methods of instrument
training and effects on pilots’ performance with different types of flight
instrument displays. Aviation Psychology and Applied Human Factors, 2(2),
62-71.

Lintern, G. (1980). Transfer of landing skill after training with supplementary visual
cues. Human Factors, 22(1), 81-88.

191
Lintern, G. (1991). An informational perspective on skill transfer in human-machine
systems. Human Factors, 33(3), 251-266.

Lintern, G. & Koonce, J.M. (1992). Visual augmentation and scene detail effects in
flight training. International Journal of Aviation Psychology, 2(4), 281-301.

Lintern, G. & Naikar, N. (1999). Specification of training simulators by cognitive work

analysis. In: Jensen RS, ed. Proceedings of the tenth international
symposium on aviation psychology. Columbus, OH: Ohio State University,
414-420.

Lintern, G., Roscoe, S.N., Koonce, J.M. & Segal, L.D. (1990). Transfer of landing
skills in beginning flight training. Human Factors, 32(3), 319-327.

Lintern, G., Taylor, H.L., Koonce, J.M. & Talleur, D.A. (1993). An incremental
transfer study of scene detail and field of view effects on beginning flight
training. In Proceedings of the 7th International Symposium on Aviation
Psychology, 737-742.

McDaniel, M.A., Schmidt, F.L. & Hunter, J.E. (1988). Job experience correlates of
job performance. Journal of Applied Psychology, 73(2), 327-330.

McDermott, J. (2006). Computer-based flight simulation: A cost effective way for

general aviation pilots to improve their instrument proficiency. International
Journal of Applied Aviation Studies, 6(1), 155-163.

McFadden, K.L. (1996). Comparing pilot-error accident rates of male and female
airline pilots. Omega, International Journal of Management Science, 24(4),
443-450.

McKinney Jr., E.H. & Davis, K.J. (2003). Effects of deliberate practice on crisis
decision performance. Human Factors, 45(3), 436-444.

McMullan, M., Endacott, R., Gray, M.A., Jasper, M., Miller, C.M.L., Scholes, J. &
Webb, C. (2003). Portfolios and assessment of competence: a review of the
literature. Journal of Advanced Nursing, 41(3), 283-294.
192
MacFarlane, R. (2003). Competency-based licensing system planned for all pilot
licences in Australia. ICAO Journal, No.4. International Civil Aviation
Organization, Montreal, Canada.

Martin, E.L. & Waag, W.L. (1978). Contributions of platform motion to simulator
training effectiveness: Study 1 – Basic contact. Air Force Human Resources
Laboratory, Air Force Systems Command, Brooks Air Force Base, TX.

Martin, M., Vashisht, B., Frezza, E., Ferone, T., Lopez, B., Pahuja, M. & Spence,
R.K. (1998). Competency-based instruction in critical invasive skills improves
both resident performance and patient safety. Surgery, 124(2), 313-317.

Martin, W., Murray, P.S. & Bates, P.R. (2012). The effects of startle on pilots during
critical events: A case study analysis. In Proceedings of the 30th EAAP
Conference – working towards zero accidents, 387-394.

Mavin, T. & Dall’Alba, G. (2010). A model for integrating technical skills and NTS in
assessing pilots’ performance. In Proceedings of the 9th International
Symposium of the Australian Aviation Psychology Association. AAvPA,
Canberra, Australia.

Mendenhall, W., Scheaffer, R.L. & Wackerly, D.D (1981). Mathematical statistics
with applications (second edition). PWS Publishers, Boston, United States.

Milligan, F. (1998). Defining and assessing competence: the distraction of outcomes

and the importance of educational process. Nurse Education Today, 18, 273-
280.

Misko, J. (1999). Competency-based training: review of research. NCVER,

Leabrook, SA, Australia.

Misko, J. & Robinson, C. (2000). Competency-based training in Australia. In

Arguelles, A. & Gonczi, A. (Ed.) Competency based education and training: a
world perspective (pp. 61-84). Noriega Editores, Balderas, Mexico.

193
Molesworth, B., Wiggins, M.W. & O’Hare, D. (2006). Improving pilots’ risk
assessment skills in low-flying operations: The role of feedback and
experience. Accident Analysis and Prevention, 38, 954-960.

Moroney, W.F., Hampton, S., Biers, D.W. & Kirton, T. (1994). The use of personal
computer-based training devices in teaching instrument flying: a comparative
study. In Proceedings of the Human Factors and Ergonomics Society 38th
Annual Meeting, 95-99.

Mulder, M., Weigel, T. & Collins, K. (2007). The concept of competence in the
development of vocational education and training in selected EU member
states: a critical analysis. Journal of Vocational Education and Training,
59(1), 67-88.

Munro, I. & Mavin, T.J. (2012). Crawl–walk-run. In Proceedings of the 10th

International Symposium of the Australian Aviation Psychology Association,
Canberra, ACT.

Myers Jr, H.B. (1974). Pilot accident potential as related to total flight experience
and recency and frequency of flying. Naval Postgraduate School, Monterey,
CA, United States of America.

Nataupsky, M., Waag, W.L., Weyer, D.C., McFadden, R.W. & McDowell, E. (1979).
Platform motion contributions to simulator training effectiveness: Study III –
Interaction of motion with field-of-view. Air Force Human Resources
Laboratory, Air Force Systems Command, Brooks Air Force Base, TX.

National Centre for Vocational Education Research (1999). Research at a glance:

Competency-based training in Australia. NCVER, Leabrook, SA, Australia.

National Transportation Safety Board (2010). Loss of control on approach Colgan

Air, Inc. operating as Continental Connection Flight 3407, Bombardier DHC-
8-400, N200WQ, clarence center, New York, February 12, 2009. National
Transportation Safety Board, Washington, United States of America.

194
Neves, D.M. & Anderson, J.R. (1981). Knowledge compilation: Mechanisms for the
automatization of cognitive skills. In Anderson, J.R. (Ed.) Cognitive Skills and
Their Acquisition (pp. 57-84). Lawrence Erlbaum Associates, New Jersey,
USA.

Norris, N. (1991). The trouble with competence. Cambridge Journal of Education,

21(3), 331-342.

O’Hare, D. & Roscoe, S. (1990). Flightdeck performance: The human factor. Iowa
State University Press, Iowa, U.S.A.

O’Hare, D. & Batt, R. (1999). A pilot for all seasons: beyond simulation. In D.O’Hare
(Ed.), Human performance in general aviation (pp.173-190).
Aldershot:Ashgate.

O’Hare, D. & Chalmers, D. (1999). The incidence of incidents: A nationwide study of

flight experience and exposure to accidents and incidents. International
Journal of Aviation Psychology, 9(1), 1-18.

Oaster, B. (2007). Proof of concept – MPL training program. Presentation at the

ICAO Regional Symposium on the Implementation of Multi-Crew Pilot
Licence, Hong Kong, China, 23-24 August 2007.

Oprins, E., Burggraaff, E. & van Weerdenburg, H. (2006). Design of a competence-

based assessment for air traffic control training. International Journal of
Aviation Psychology, 16(3), 297-320.

Padmos, P. & Milders, M.V. (1992). Quality criteria for simulator images: A literature
review. Human Factors, 34, 727-48.

Pagan, B.J. & De Voogt, A. (2008). Gyroplane accidents 1985-2005:

Epidemiological analysis and pilot factors in 223 events. Aviation, Space, and
Environmental Medicine, 79(10), 983-5.

195
Pagan, B.J., De Voogt, A.J., Van Doorn, R.R.A. (2006). Ultralight aviation accident
factors and latent failures: A 66-case study. Aviation, Space, and
Environmental Medicine, 77(9), 950-2.

Parasuraman, R. & Mouloua, M. (1996). Automation and human performance.

Lawrence Erlbaum Associates, Mahwah, NJ, U.S.

Parliament of Australia Senate. (2010). Pilot training and airline safety. Retrieved 20
October 2010 from
http://www.aph.gov.au/senate/committee/rat_ctte/pilots_2010/info.htm

Phillimore, J. (1997). Trade unions and the national training reform agenda in
Australia, 1983–1996. International Journal of Training and Development,
1(1), 34-48.

Priest, A. (2009). Getting the knowledge-skills mix right in high-level vocational

education and training qualifications. NCVER, Adelaide, SA, Australia.

Phillips, S.I., Taylor, H.L., Lintern, G., Hulin, C.L., Emanuel, T.W. Jr, & Talleur, D.A.
(1995). Developing performance measures for evaluating personal computer-
based aviation training devices within an FAR part 141 pilot training school.
In Proceedings of the 8th International Symposium on Aviation Psychology.

Potter, E.H. & Fiedler, F.E. (1981). The utilization of staff member intelligence and
experience under high and low stress. The Academy of Management
Journal, 24(2), 361-376.

Pryde, T. (2007). MPL beta test – Progress and lessons learned. Presentation at the
ICAO Regional Symposium on the Implementation of Multi-Crew Pilot
Licence, Hong Kong, China, 23-24 August 2007.

Quilty, S.M. (1996). Cognitive learning bias of college students in an aviation

program. Journal of Air Transportation World Wide, 1(1), 51-62.

196
Randel, J.M., Pugh, H.L. & Reed, S.K. (1996). Differences in expert and novice
situation awareness in naturalistic decision making. International Journal of
Human-Computer Studies, 45, 579-597.

Rasmussen, J. (1983). Skills, rules, and knowledge; signals, signs, and symbols,
and other distinctions in human performance models. IEEE Transactions on
Systems, Man and Cybernetics, SMC-13(3), 257-266.

Rasmussen, J (1987). Cognitive control and human error mechanisms. In:

Rasmussen, J., Duncan, K. & Leplat, J. (Eds.) New technology and human
error (pp53 – 61). John Wiley & Sons, Chichester, UK.

Reason, J. (1991). Human error. Cambridge University Press, Cambridge UK.

Reason, J. (2008). The human contribution: Unsafe acts, accidents and heroic
recoveries. Ashgate, Surrey, England.

Ritchie, M.L. (1988). General aviation. In Wiener E.L. & Nagel, D.C. (Eds.) Human
Factors in Aviation (pp. 561-589). Academic Press, San Diego, CA, USA.

Roscoe, S.N. (1991). Simulator qualification: Just as phony as it can be.

International Journal of Aviation Psychology, 1(4), 335-339.

Rumelhart, D.E. & Norman, D.A. (1981). Analogical processes in learning. In

Anderson, J.R. (Ed.) Cognitive Skills and Their Acquisition (pp. 335-359.
Lawrence Erlbaum Associates, New Jersey, USA.

Salas, E., Bowers, C.A. & Rhodenizer, L. (1998). It is not how much you have but
how you use it: Toward a rational use of simulation to support aviation
training. International Journal of Aviation Psychology, 8: 197-208.

Salas, E., Rosen, M.A., Held, J.D. & Weissmuller, J.J. (2009). Performance
measurement in simulation-based training: A review and best practices.
Simulation and Gaming, 40, 328-76.

197
Salvatore, S., Stearns, M.D., Huntley Jr, M.S. & Mengert, P. (1986). Air transport
pilot involvement in general aviation accidents. Ergonomics, 29(11), 1455-
1467.

Saunders, J. (1998). Indicators of competency: Profiling employees and the

workplace. NCVER, Leabrook, SA, Australia.

Scheele, F., Teunissen, P., Van Luijk, S., Heineman, E., Fluit, L., Mulder, H.,
Meininger, A., Wijnen-Meijer, M., Glas, G., Sluiter, H. & Hummel, T. (2008).
Introducing competency-based postgraduate medical education in the
Netherlands. Medical Teacher, (30), 248-253.

Schmidt, R.A. & Wulf, G. (1997). Continuous concurrent feedback degrades skill
learning: Implications for training and simulation. Human Factors, 39, 509-25.

Schneider, W. (1985). Training high-performance skills: Fallacies and guidelines.

Human Factors, 27, 285-300.

Schriver, A.T., Morrow, D.G., Wickens, C.D. & Talleur, D.A. (2008). Expertise
differences in attentional strategies related to pilot decision making. Human
Factors, 50(6), 867-878.

Seely Brown, J., Collins, A. & Duguid, P. (1989). Situated cognition and the culture
of learning. Educational Researcher, 18, 32-42.

Shanteau, J. (1992). Competence in experts: The role of task characteristics.

Organizational Behavior and Human Decision Processes, 53, 252-266.

Shiffrin, R.M. & Dumais, S.T. (1981). The development of automatism. In Anderson,
J.R. (Ed.) Cognitive Skills and Their Acquisition (pp. 111-139). Lawrence
Erlbaum Associates, New Jersey, USA.

Shuch, H.P. (1992). The influence of flight experience on midair collision risk
perception. Accident Analysis and Prevention, 24(6), 655-660.

198
Smallwood, T. (2000). The airline training pilot. Ashgate, Surrey, England.

Smith, E. (1999). Ten years of competency-based training: the experience of

accredited training providers in Australia. International Journal of Training
and Development, 3(2), 106-117.

Smith, E. (2010). A review of twenty years of competency-based training in the

Australian vocational education and training system. International Journal of
Training and Development, 14(1), 54-64.

Stein, D. (1998). Situated learning in adult education. ERIC Digest 3. Clearinghouse

on Adult, Career and Vocational Education. Columbus, OH.

Sutton, O. (2005). Shortcut or fast track? Air Transport World. Retrieved 24 April
2008 from
http://www.atwonline.com/channels/safetySecurity/article.html?articleID=1428

Taylor, H.L., Lintern, G. & Koonce, J.M. (1993). Quasi-transfer as a predictor of

transfer from simulator to airplane. The Journal of General Psychology,
120(3), 257-276.

Taylor, H.L., Lintern, G., Koonce, J.M., Kunde, D.R., Tschopp, J.M. & Talleur, D.A.
(1993). Scene content, field of view and amount of training in first officer
training. In Proceedings of the 7th International Symposium on Aviation
Psychology.

Taylor, H.L., Lintern, G., Hulin, C.L., Talleur, D., Emanuel, T. & Phillips, S.
(1997a).Transfer of training effectiveness of personal computer-based
aviation training devices. Federal Aviation Administration, Washington DC,
DOT/FAA/AM-97/11.

Taylor, H.L., Lintern, G., Hulin, C.C., Talleur, D.A., Emanuel, T.W. Jr. & Phillips, S.
(1997b). Effectiveness of personal computers for instrument training. In
Proceedings of the 9th International Symposium on Aviation Psychology,
1276-1280.

199
Taylor, H.L., Talleur, D.A., Emanuel, T.W. Jr, Rantanen, E.M., Bradshaw, G. &
Phillips, S. (2001). Incremental training effectiveness of personal computers
used for instrument training. In Proceedings of the 11th International
Symposium on Aviation Psychology.

Taylor, H.L., Talleur, D.A., Bradshaw, G.L., Emanuel, T.W. Jr., Rantanen, E., Hulin,
C.L. & Lendrum, L. (2003). Effectiveness of personal computers to meet
recency of experience requirements. Federal Aviation Administration,
Washington DC, DOT/FAA/AM-03/3.

Taylor, H.L., Talleur, D.A., Rantanen, E.M. & Emanuel, T.W. Jr. (2005). The
effectiveness of a personal computer aviation training device (PCATD), A
flight training device (FTD), and an airplane in conducting instrument
proficiency checks. In Proceedings of the 13th International Symposium on
Aviation Psychology.

Thomas, M.J.W. & Ferguson, S.A. (2010). Prior sleep, prior wake, and crew
performance during normal flight operations. Aviation, Space, and
Environmental Medicine, 81, 665-70.

Thomson, P., Mathers, R. & Quirk, R. (1996). The grade debate: should we grade
competency-based assessment? NCVER, Leabrook, SA, Australia.

Thomson, P., Saunders, J. & Foyster, J. (2001). Improving the validity of

competency-based assessment. NCVER, Leabrook, SA, Australia.

Thorndike, E.L. (1923). Education psychology: Briefer course. Routledge, Abingdon,

Oxon, UK.

Todd, M.A. & Thomas, M.J.W. (2012). Flight hours and flight crew performance in
commercial aviation. Aviation, Space, and Environmental Medicine, 83(8),
776-782.

200
Todd, M.A. & Thomas, M.J.W. (2013). Pilot experience and performance in an
airline environment. Aviation research investigation AR-2012-023. Australian
Transport Safety Bureau, Canberra, ACT.

Torr, A. (2008). A complex view of professional competence. Retrieved 13 June

2011 from http://www.voced.edu.au/content/ngv38681 NCVER, Adelaide,
SA, Australia.

Trafton, J.G. & Monk, C.A. (2008). Task interruptions. In Reviews of human factors
and ergonomics, 3. Human Factors and Ergonomics Society.

U.S. Department of Education, National Center for Education Statistics (2002).

Defining and assessing learning: exploring competency-based initiatives,
NCES 2002-159, prepared by Jones, E.A., & Voorhees, R.A., with Paulson,
K. For the Council of the National Postsecondary Education Cooperative
Working Group on Competency-Based Initiatives. Washington, DC.

United States Congress (2010). Airline safety and Federal Aviation Administration
extension act of 2010. United States Government Printing Office.
Washington, DC.

Velde, C. (1999). An alternative conception of competence: implications for

vocational education. Journal of Vocational Education and Training, 51(3),
437-447.

Voorhees, M. (1996). Using competency-based education in the perioperative

setting. Nursing Management, 27(8), 35, 38.

Voorhees, R.A. (2001). Competency-based learning models: A necessary future.

New Directions for Institutional Research, 110, Summer 2001, 5-13.

Voss, W.R. (2012). Testimony of William R. Voss, President and CEO, flight safety
foundation to U.S. Senate Committee on Commerce, Science and
Transportation’s Subcommittee on Aviation Operations, Safety, and Security,
Hearing on Implementation of HR5900. Retrieved 3 November 2013 from
http://flightsafety.org/media-center/speeches-and-presentations/voss-
testimony-on-HR-5900 FSF, Alexandria, Virginia, U.S.
201
Walker, M.B. & Bills, K.M. (2008). Analysis, causality and proof in safety
investigations. Australian Transport Safety Bureau, Canberra, ACT, Australia.

Watson, R., Stimpson, A., Topping, A. & Porock, D. (2002). Clinical competence
assessment in nursing: a systematic review of the literature. Journal of
Advanced Nursing, 39(5), 421-431.

Westra, D.P. & Lintern, G. (1995). Transfer of training and cost-effectiveness.

Vertical Flight Training. Federal Aviation Administration, Washington DC, 61-
86.

Wickens, C.D. & McCarley, J.S. (2008). Applied attentional theory. CRC Press,
Boca Raton, FL, USA.

Wiegmann, D.A., Goh, J. & O’Hare, D. (2002). The role of situation assessment and
flight experience in pilots’ decisions to continue visual flight rules flight into
adverse weather. Human Factors, 44(2), 189-197.

Wiener, E.L. (1989). Human factors of advanced technology (“glass cockpit”)

transport aircraft. National Aeronautics and Space Administration, Ames
Research Center, California, United States of America.

Wiggins, M.E. & Crognale, M.W. (2003). Use of training devices in general aviation
training programs. Federal Aviation Administration, Washington DC.

Wiggins, M.W. & Bollwerk, S. (2006). Heuristic-based information acquisition and

decision making among pilots. Human Factors, 48(4), 734-746.

Wiggins, M.W. & O’Hare, D. (1995). Expertise in aeronautical weather-related

decision making: A cross-sectional analysis of general aviation pilots. Journal
of Experimental Psychology: Applied, 1(4), 305-320.

Wiggins, M.W. & O’Hare, D. (2003). Expert and novice pilot perceptions of static in-
flight images of weather. The International Journal of Aviation Psychology,
13(2), 173-187.
202
Wiggins, M.W. & Stevens, C. (1999). Aviation social science: Research methods in
practice. Ashgate, Aldershot, England.

Wiggins, M.W., Stevens, C., Howard, A., Henley, I. & O’Hare, D. (2002). Expert,
intermediate and novice performance during simulated pre-flight decision-
making. Australian Journal of Psychology, 54(3), 162-167.

Wightman, D.C. & Lintern, G. (1985). Part-task training for tracking and manual
control. Human Factors, 27(3), 267-283.

Yule, S., Flin, R., Maran, N., Rowley, D., Youngson, G. & Paterson-Brown, S.
(2008). Surgeons’ non-technical skills in the operating room: Reliability
testing of the NOTSS behavior rating system. World Journal of Surgery, 32,
548-56.

Zheng, B., Cassera, M.A., Martinec, D.V., Spaun, G.O. & Swanstrom, L.L. (2010).
Measuring mental workload during the performance of advanced
laparoscopic tasks. Surgical Endoscopy, 24(1), 45-50.

203
 Appendix 1 Permission from publishers

As this thesis contains publications in which copyright was assigned to the

publisher, the following statements provide permission for their use in this thesis.

Chapter 2

‘Experience, Competence or Syllabus? Influences on Flight Hours at Licensing of

Commercial Pilots’ International Journal of Aviation Psychology 23(2), 169-180.

Excerpt from: http://journalauthors.tandf.co.uk/permissions/reusingOwnWork.asp

3. Copyright

3.2 Retained rights

In assigning Taylor & Francis or the journal proprietor copyright, or granting an

exclusive license to publish, you retain:

the right to include an article in a thesis or dissertation that is not to be published

commercially, provided that acknowledgment to prior publication in the journal is
made explicit;…

Chapter 3

‘Flight hours and Flight Crew Performance in Commercial Aviation’ Aviation, Space,
and Environmental Medicine 83(8), 776-782.

From: Pam C. Day [mailto:pday@asma.org]

Sent: Wednesday, 26 February 2014 7:41 AM
To: Todd Melanie
Subject: RE: Copyright for manuscript 3271 [SEC=UNOFFICIAL]

You are allowed to published your article in your thesis.

Pam Day

Managing Editor

204
Chapter 4

‘Pilot experience and performance in an airline environment’ Australian Transport

Safety Bureau, Canberra ACT.

Creative Commons licence

With the exception of the Coat of Arms, ATSB logo, and photos and graphics in
which a third party holds copyright, this publication is licensed under a Creative
Commons Attribution 3.0 Australia licence.

Creative Commons Attribution 3.0 Australia Licence is a standard form license

agreement that allows you to copy, distribute, transmit and adapt this publication
provided that you attribute the work.

The ATSB’s preference is that you attribute this publication (and any material
sourced from it) using the following wording: Source: Australian Transport Safety
Bureau.

Copyright in material obtained from other agencies, private individuals or

organisations, belongs to those agencies, individuals or organisations. Where you
want to use their material you will need to contact them directly.

205
 Appendix 2 Author Contribution

The contribution of the second author, Dr Matthew Thomas, to the three publications
in this thesis, was in line with that expected from a Higher Degree Research
supervisor. Dr Thomas provided methodological guidance and assistance with the
interpretation of results, final structure of the papers and editing comments.

Melanie A Todd… ………………………..

Dr Matthew JW Thomas… ...

206