This is the author pre-publication version.

The final paper that should be used for referencing is:

Y. Lim, A. Gardi, S. Ramasamy, R. Sabatini, “A Virtual Pilot Assistant System for Single Pilot Operations of Commercial
Transport Aircraft”, Australian International Aerospace Congress (AIAC17), pp. 139-145, Melbourne, Australia, 2017.

Student Paper

A Virtual Pilot Assistant System for Single Pilot

Operations of Commercial Transport Aircraft
Yixiang Lim, Alessandro Gardi, Subramanian Ramasamy and Roberto Sabatini

School of Engineering – Aerospace Engineering and Aviation

RMIT University, Melbourne, VIC - 3000, Australia

Abstract
The paper presents the conceptual design of a Virtual Pilot Assistant (VPA) for single-pilot
operations of commercial transport aircraft. In particular, a thorough requirement analysis was
performed by taking into account existing literature on single pilot operations from general
aviation and military standards. The analysis also considers relevant human factors
engineering and system design aspects of conventional two-pilot aircraft and Remotely-Piloted
Aircraft Systems (RPAS). The VPA performs a real-time assessment of the single-pilot’s
cognitive states and provides useful and timely alerts based on predictions on the performance
levels of the pilot. In case of pilot incapacitation, the VPA enables the single-pilot aircraft to
be operated as an RPAS, allowing ground operators to take over as necessary and make an
emergency landing at a safe and suitable location.
Keywords: Cognitive Systems, Human Factors Engineering, Intelligent Decision Support
Systems, Single-Pilot Operations.

Introduction
Single-pilot operations (SiPO) have traditionally been employed in the domain of General
Aviation (GA) and military jet aircraft. However, advances in Communications, Navigation,
Surveillance (CNS), as well as Air Traffic Management (ATM) and Avionics (CNS+A)
technologies are now creating an opportunity for commercial transport aircraft to transition to
SiPO. For instance the avionics systems of recent larger single-pilot GA jet aircraft, such as
the Embraer Phenom, Cessna Citation and ONE Aviation Eclipse, already feature most of the
enabling CNS+A advances. These recent avionics integrate higher levels of decision support
and flight automation to allow SiPO within the cockpit. While some of the concepts from GA
single-pilot cockpits are transferrable to commercial transport aircraft, the evolution from
conventional two-pilot to single-pilot operations requires more extensive operational and
safety assessment to ensure the aircraft is not compromised in case of pilot incapacitation. In
the literature, in-depth research on SiPO has been conducted by NASA since the mid-2000s
[1-3], and also by TU Delft [4] and MITRE [5] in more recent years. The main considerations
for SiPO, as highlighted in the literature, revolve around five aspects: operational, safety,
technical, human factors as well as certification. This paper presents a brief overview of these
various aspects before presenting the system design. In particular, a Virtual Pilot Assistant
(VPA) system is proposed to fulfil the new set of operational requirements imposed by SiPO,
to ensure sufficient levels of safety and human-system interactions, thus providing a pathway
for the certification of SiPO for commercial transport aircraft.

Operational Considerations
In view of the reduction in flight crew size, a new concept of operation is required to re-
distribute the tasks of ‘aviate, navigate, communicate and manage’ among operational

139
personnel. A number of alternative single-pilot configurations have been considered:
1. Single-pilot on-board, fully responsible for taking over the duties of the second pilot.
2. Single-pilot on-board, supported by on-board automation.
3. Single-pilot on-board, supported by on-board back-up personnel.
4. Single-pilot on-board, supported by a ground-based team member.
5. Single-pilot on-board, supported by a network of ground-based team members.
The ideal SiPO concept of operations will likely incorporate elements of all these
configurations – a combination of increased automation (thorough an intelligent assistant
system), ground support personnel (dispatcher or remote pilot) and in-flight personnel (flight
attendant or air marshal) support would be provided to offset the increased workload
experienced by the single pilot as a result of him/her inheriting some of the secondary pilot’s
tasks. Increased automation will likely assume the bulk of lower-level ‘aviate and navigate’
tasks, such that the single-pilot will be able to focus on managing on-board systems and
communicating with ground support. Ground support personnel could either involve a
dispatcher – through an evolution of the role of current Airline Operational Center (AOC)
operators – allowing the AOC to manage most of the flight planning activities; or a remote
pilot, to provide additional support in monitoring and/or piloting the single-pilot aircraft
during off-nominal, workload-intensive phases or single-pilot incapacitation events. These
roles can be distributed in the form of a ground-based network such that different types of
ground personnel can provide different forms of support throughout the various flight phases.
On-board support is provided by having specially trained, non-piloting crew members to either
temporarily assume control of the aircraft, activate emergency procedures, or transfer authority
to ground personnel. Some of the safety, technical, human factors and certification challenges
introduced by these new operational modes will be discussed in following sections.

Safety Considerations
The adoption of SiPO’s concept of operations requires key safety issues to be considered and
addressed before SiPO can be implemented. Three key issues that require consideration are:
emergency procedures in the event of single-pilot incapacitation; the negative effect of
increased on-board automation; as well as the security, performance and vulnerability of the
communications links.
With respect to single-pilot incapacitation, conventional flight decks rely on the second pilot
to provide an additional layer of redundancy that SiPO does not have. The challenge is for
SiPO to demonstrate that single-pilot incapacitation does not compromise flight safety. In
place of a second pilot, the SiPO concept of operations must rely on different channels to
build the required redundancy. Physical redundancy is afforded through on-board support
personnel – flight attendants and/or flight marshals to first identify an incapacitation event,
then to provide on-site assistance and support to the incapacitated pilot – but this form of
redundancy requires locked cockpit doors protocols to be revised. Ground personnel provide
another layer of redundancy, allowing the aircraft to be flown as a remotely piloted vehicle
while emergency landing procedures are coordinated on the ground with ATM operators.
Automated systems provide the next layer of redundancy, with a possibility being the use of
psychophysiological monitoring systems for early detection and preliminary diagnosis of pilot
incapacitation; as well as automated functions to override controls and/or transfer them to
ground operators.
Because SiPO’s concept of operations requires higher levels of automation than conventional
aircraft operations, greater control authority have to be given to these systems, which imply
greater risk to safety in the event of a system failure. Additionally, there are concerns about
17th Australian Aerospace Congress, 26-28 February 2017, Melbourne
140
the short term (loss of situational awareness) and long term (complacency, skill degradation)
effects induced by an increased use of automation. The safety implications of increased on-
board automation should be carefully addressed by developing adequate human-machine
interfaces and interactions.
The increased collaboration between on-board and ground personnel places a greater emphasis
on the reliability, performance and security of the air-ground communications network.
Depending on the level of support provided, high performance and secure links will be
required (e.g., remote pilot support during pilot incapacitation). Network security will
therefore also be a key issue, with interference activities such as the intentional/unintentional
spoofing and jamming of key communication signals posing a significant threat to flight
safety.

Technical Considerations
To address the operational and safety requirements, current CNS+A technologies need to be
leveraged to realise the SiPO concept of operations. The key CNS considerations include:
implementing a communications network for realising the new forms of air-ground
collaboration; automated flight management as well as trajectory planning and optimisation
capabilities [6]; enhanced single-pilot situational awareness through on-board Line of Sight
(LOS) and Beyond LOS (BLOS) surveillance systems [7]; as well as cognitive human
machine interfaces (CHMI) for assessing and mitigating single-pilot workload, or in the case
of pilot incapacitation, detection of incapacitation, automatic transfer of authority and
execution of emergency landing procedures.

Human Factors Considerations

In addition to the safety considerations regarding increased automation as highlighted earlier,
other key human factors issues include:
1. Workload/fatigue: single-pilot workload/fatigue must not exceed the limits of the pilot and
compromise his/her ability to maintain flight safety. The pilot shall therefore be supported
by intelligent on-board support systems, which monitor pilot workload, fatigue and
situational awareness and perform necessary task reallocations as required.
2. Crew resource management (CRM): the transition to a single-pilot cockpit would mean
that different CRM concepts would need to be adopted, which includes pilot boredom due
to lack of human interaction, transfer of authority and accountability between on-board
pilot, on-board automation and ground support personnel, as well as automation
monitoring techniques.
3. Selection and training: single-pilots as well as ground and on-board support personnel
require new selection criteria and training standards. The current pilot/co-pilot mentorship
training scheme would not be possible in SiPO.
Certification Considerations
There are no certifications and standards relating to SiPO for commercial aviation at this
point. However, there is ample literature covering conventional two-pilot operations, GA
SiPO as well as remotely piloted aircraft systems (RPAS), which can be assumed as the basis
for the development of commercial SiPO certification standards. In terms of operational
certification standards, SiPO for transport aircraft would require an evolution of current
Federal Aviation Regulations (FAR) Parts 25 and 121, in particular FAR 25.1523 and
Appendix D, specifying the requirements for minimum flight crew. With regards to training
and flight crew selection, references can be made to EASA Acceptable Means of Compliance
(AMC) and Guidance Material (GM) to Annex III (AMC2.ORO.FC.115), as well as AMC1
17th Australian Aerospace Congress, 26-28 February 2017, Melbourne
141
FCL.720.A(b)(2)(i). The operational requirements for RPAS are involved if the single-pilot
aircraft becomes remotely operated in case of single-pilot incapacitation – reference can be
made to the International Civil Aviation Organization (ICAO) Doc. 10019 “Manual on
RPAS”, Radio Technical Commission for Aeronautics (RTCA) DO-304 chapters 1, 2 as well
as Appendix D, as well as Australia’s Civil Aviation Safety Authority (CASA) AC 101-1.
The safety aspects are covered in FAR 25.1309 as well as the Advisory Circular (AC) 25.1309
and relate to the safety analysis of avionics equipment, systems and installations. The Society
of Automotive Engineers (SAE) ARP-4754A and ARP 4761 relate to the standards for
avionics systems development and design as well as the safety assessment, while Joint
Authorities for Rulemaking on Unmanned Systems (JARUS) AMC RPAS 1309 and European
Organisation for Civil Aviation Equipment (EUROCAE) ER-010 outline the safety
assessment for RPAS.
Concerning technical certifications, the reader is referred to the ICAO Doc. 9869 “Manual for
Required Communications Performance (RCP)”, Doc. 9613 and Doc. 9224 (respectively on
Performance-Based Navigation and Surveillance), as well as RTCA DO-238 on
communications data link, SAE ARP 4109/9A on Flight Management Systems (FMS),
JARUS D.04 on RPAS Command and Control (C2) datalink RCP and NATO Standardization
Agreement (STANAG) 4586 Appendixes B1 and B2 on datalink and C2 interfaces.
For human factors standards and guidelines, references should be made to FAA STD-004
(human factors program), AC 25.1302-1 (design of systems), RTCA DO-344, Volume 2
Appendix D (RPAS control stations) and the US Department of Defence DOD 11-S-3613
Chapter 10 (manned/unmanned teaming).

System Design
The proposed VPA system is an on-board autonomous pilot support system that interfaces
with various subsystems (including the flight control, FMS, sense-and-avoid and
communications subsystems) in the aircraft. The top-level requirements for the VPA design
are as follows:
1. Decrease pilot workload by assuming control of specific flight tasks, which include:
computing an sharing optimized flight plans through a Next-Generation FMS (NG-FMS);
temporarily assuming control of the aircraft in the event of pilot incapacitation, with the
capability to perform CAT II (< 200ft decision height) and CAT III (< 100ft decision
height) landing; systems monitoring through Integrated Vehicle Health Management
(IVHM) and Avionics Based Integrity Augmentation (ABIA) systems to provide
cautions/warnings to the pilot if required [8].
2. Decrease flight deck complexity through psychophysiological monitoring sensors that
assess the pilot’s cognitive state and trigger changes in adaptive interfaces to prompt
appropriate automation modes to the pilot.
3. Increase aircraft surveillance capabilities through advanced avionics systems, including: a
surveillance system for autonomous Separation Assurance and Collision Avoidance
(SA&CA); a weather surveillance system augmented by ground forecasts; as well as
autonomous strategic/tactical rerouting and deconfliction.
4. Facilitate collaborative work- and information-sharing through a combination of direct
LOS and beyond LOS air-to-ground communication channels, supplemented by ground-to-
ground channels between the ground support and ATM.
The VPA system architecture is illustrated in Figure 1 and comprises of four major systems:
the communications, surveillance, flight management/control and HMI systems. The
communications system enables data-sharing between the single-pilot, ground support and
17th Australian Aerospace Congress, 26-28 February 2017, Melbourne
142
ATM via a network comprising various LOS and BLOS data links. In case of emergency, a
reliable, secure high-speed C2 link enables the ground support personnel to assume direct
control of the aircraft’s flight management and control systems. The surveillance system
consists of an Airborne Surveillance and Separation Assurance Processing (ASSAP)
subsystem, which interfaces with the NG-FMS, to provide automated SA&CA capabilities.
The NG-FMS is interlinked to the Flight Control Unit (FCU), autopilot and Flight Control
System (FCS) to provide guidance, navigation and control, as well as trajectory optimisation,
planning, negotiation and validation functions. An IVHM subsystem automates the
management and monitoring of aircraft systems, providing appropriate updates, warnings or
alerts to the pilot (via a cognitive HMI) and the ground crew.

Adaptive/Cognitive HMI

Pilot

Cognitive HMI
Manual Control

Flight Control Flight Management Communications

(Navigation and Guidance) IVHM
FCU
Comms
Autopilot NG-FMS Ground RLOS/BRLOS
receivers/transmitters

FCS
ASSAP Traffic / weather
updates

GO AOCO ATCO
Cooperative/ HMI HMI HMI
Non-cooperative
sensors Surveillance
Resilient, high-
speed C2 link

Figure 1: VPA system architecture.

The CHMI is a crucial component of the VPA system, providing the necessary reductions in
workload as well as incapacitation-detecting capabilities that will support the case for SiPO
certification. The CHMI assists the pilot with several intelligent functions such as information
management, adaptive alerting, situation assessment as well as dynamic task allocation. The
design of the CHMI in the VPA system is based on the guidelines specified in FAR AC
25.1302-1 with respect to the human factors engineering and system redundancy for safe and
effective operations.
The CHMI utilises physiological sensors, which are integrated into the flight deck, to monitor
the pilot in real-time. Cognition models are used to estimate the pilot’s cognitive states, such
as fatigue, stress, attention and mental workload based on the physiological data collected.
Important physiological indicators include brain activity (e.g., blood oxygenation levels),
cardiac activity (e.g., heart rate, heart rate variability) and eye activity (e.g., blink rate, eye
movements and pupil diameter). Brain activity provides information on cognitive workload,
and can be tracked either electrically via electroencephalography (EEG) or optically via
functional near-infrared (fNIR) spectroscopy. Cardiac activity can be measured with wearable
devices such as wristbands or smart shirts [9], and is utilised to assess stress and workload of
the pilot. Eye activity can be tracked remotely with multi-camera systems and is also a good
17th Australian Aerospace Congress, 26-28 February 2017, Melbourne
143
indicator of workload – blink rates and duration are inversely correlated and decrease with
increasing workload [10]. Additionally, pilot attention can be inferred from gaze patterns,
which are correlated with information sampling [11]. An inference engine is used to manage
task distribution between the automation systems (e.g., NG-FMS) and human operators (air-
and ground-based) based on the external environment and their cognitive states. If a high pilot
workload is inferred, the CHMI scales up decision support by prompting a transition to higher
levels of system automation, reduces level of detail of displayed information, and/or
transferring non-critical tasks to the ground crew. On the other hand, if the system infers that
the pilot is losing situational awareness at a high level of automation, the CHMI either
prompts a reduction in level of automation or triggers appropriate alerts to keep the pilot in the
loop. Adaptive alerting is designed to provide cues that complement the cognitive state of the
pilot, based on the system’s assessment of the situation. Alerts are prioritised and are provided
through a combination of visual, auditory and haptic feedbacks; for instance, multi-sensory
feedback increases the pilot’s perceptual bandwidth with more channels to process
information. Figure 2 illustrates the architecture of the CHMI, which captures several
elements from [9].

Cognitive HMI
Avionics databus Intelligent Module
Primary flight data
Surveillance data Cognition-complexity model Fatigue
C/N/S integrity
Uncertainty analysis module Stress
Flight plan

Inference engine

Pilot
Task Manager
Brain activity
Eye activity Sensors Information Priority
Cardiac selector selector Workload
Voice

Interface Control Module Alerting Module

Ground station
Weather
Comms GUI page manager Mode selector
Traffic
System Visual
Terrain Format and function manager Auditory
CRM/SRM Haptic

Task Automator Display/loudspeaker

Figure 2: Architecture of the cognitive HMI.

Conclusions
Given the advances in Communication, Navigation, Surveillance, Air Traffic Management
and Avionics (CNS+A) technologies, the evolution from commercial two-pilot to Single Pilot
Operations (SiPO) is possible in the next few decades. To allow this evolution, various
operational, safety, technical, human factors and certification aspects need to be considered in
the design of a Virtual Pilot Assistant (VPA) system. In particular, this paper presents a new
17th Australian Aerospace Congress, 26-28 February 2017, Melbourne
144
concept of operations requiring greater air-to-ground collaboration as well as on-board
automation. Additionally, the key safety and human factors issues concerning this concept of
operations are assessed, technical considerations in addressing these issues are provided and
the status of current certification standards was evaluated. The VPA is a viable system
implementation providing enhanced surveillance, communication and flight management
capabilities, as well as adaptive task allocation and human-machine interfaces. By monitoring
the pilot’s workload through on-board psychophysiological monitoring devices, the VPA
adapts its level of support in its various autonomous functions through a context-aware
inference engine. Additionally, adaptive alerts and task allocation are provided to the pilot
based on the VPA’s assessment of the situation. The psychophysiological monitoring devices
are also capable of detecting an incapacitation event, allowing the VPA to assume temporary
autonomous control while simultaneously alerting and transferring control authority to the
ground support personnel. The VPA interfaces with the trajectory optimisation, negotiation
and validation as well as Separation Assurance and Collision Avoidance (SA&CA) functions
of the Next Generation Flight Management System (NG-FMS) to provide autonomous flight
planning and real-time re-optimisation capabilities in such an event. In consideration of
current standards, advances in the certification and training processes are required to address
all safety, technical and regulatory implications before systems such as the VPA can be
operationally deployed.

References
1. S. Deutsch and R. W. Pew, "Single pilot commercial aircraft operation," BBN Report No. 8436,
2005.
2. D. Comerford, S. L. Brandt, J. Lachter, S.-C. Wu, R. Mogford, V. Battiste, et al., "NASA’s
Single-Pilot Operations Technical Interchange Meeting: Proceedings and Findings," NASA,
Ames Research Center, Moffett Field, California, 2013.
3. C. A. Wolter and B. F. Gore, "A Validated Task Analysis of the Single Pilot Operations
Concept," 2015.
4. A. Faber, "Single Pilot Commercial Operations: A Study of the Technical Hurdles," Master of
Science, Department of Control and Simulation, TU Delft, Delft University of Technology, Delft,
Netherlands, 2013.
5. J. Graham, C. Hopkins, A. Loeber, and S. Trivedi, "Design of a single pilot cockpit for airline
operations," in Systems and Information Engineering Design Symposium (SIEDS), 2014, 2014,
pp. 210-215.
6. A. Gardi, R. Sabatini, and S. Ramasamy, "Multi-objective optimisation of aircraft flight
trajectories in the ATM and avionics context," Progress in Aerospace Sciences, vol. 83, pp. 1-36,
5// 2016.
7. S. Ramasamy, R. Sabatini, A. Gardi, and J. Liu, "LIDAR obstacle warning and avoidance system
for unmanned aerial vehicle sense-and-avoid," Aerospace Science and Technology, 2016.
8. R. Sabatini, T. Moore, and C. Hill, "A New Avionics-Based GNSS Integrity Augmentation
System: Part 2–Integrity Flags," Journal of Navigation, vol. 66, pp. 501-522, 2013.
9. J. Liu, A. Gardi, S. Ramasamy, Y. Lim, and R. Sabatini, "Cognitive pilot-aircraft interface for
single-pilot operations," Knowledge-Based Systems, vol. 112, pp. 37-53, 2016.
10. L. L. Di Stasi, A. Antolí, M. Gea, and J. J. Cañas, "A neuroergonomic approach to evaluating
mental workload in hypermedia interactions," International Journal of Industrial Ergonomics,
vol. 41, pp. 298-304, 2011.
11. C. D. Wickens, "Visual Attention Control, Scanning, and Information Sampling," in Applied
Attention Theory, ed: CRC Press, 2007, pp. 41-61.

17th Australian Aerospace Congress, 26-28 February 2017, Melbourne

145