Recommended Practices for Human Factors Evaluation Development

Process for Advanced Avionics

Lisette Lyne and R. John Hansman

MIT International Center for Air Transportation
Department of Aeronautics & Astronautics
Massachusetts Institute of Technology
Cambridge, MA 02139 USA

July 2001
 Table of Contents

1. Introduction..............................................................................................................2
2. Objectives.................................................................................................................3
3. Recommendations...................................................................................................4
3.1 Integrated Human Centered Systems Approach............................................4
3.1.1 Model the System and Operator (or Operators) as a Closed Loop
Feedback Process.......................................................................................4
3.1.2 Determine the Information that the Operator Requires to Perform the
Task.............................................................................................................5
3.1.3 Use the Information Requirements to Determine the
Display/Automation Requirements..........................................................5
3.1.4 Develop Prototype Systems.......................................................................5
3.1.5 Perform Simulation Evaluations..............................................................5
3.1.6 Integrated Simulation Testing..................................................................5
3.1.7 System Evaluation.....................................................................................5
3.1.8 Field Development Phase..........................................................................5
3.2 Operator Directed Process................................................................................6
3.2.1 Functional Analysis...................................................................................7
3.2.2 Automation Model.....................................................................................7
3.2.3 Incremental Instantiation.........................................................................7
3.2.4 Training Material......................................................................................8
3.2.5 Certification................................................................................................8
3.2.6 Configuration Management......................................................................9
4. Recommended Practices for Fulfilling FAR Part 23 Regulations............10
4.1 Human Factors Requirements........................................................................10
4.2 Proposed Human Factors Evaluation Plan...................................................13
5. Other Issues/Comments.......................................................................................17
5.1 Limitation of Using Low Fidelity Simulators to Capture Accurate
Performance Data............................................................................................17
5.2 Evaluation Measures.......................................................................................17
6. References...............................................................................................................18
Appendix A - Guidance for Reviewing Certification Plans to Address
Human Factors for Certification of Transport Airplane Flight decks (Policy
Statement Number ANM-99-2).................................................................................19
Appendix B – Human Factors in Certification Presentation..............................20
Appendix C - Application Guide for Obtaining a supplemental Type
Certificate (AC 21-40).................................................................................................21

1
1. Introduction
Advanced avionic systems are currently being developed for use in general aviation aircraft. The
avionics include both primary flight displays and multi-functional displays. In order to support
the human factors development of such displays, a research project was undertaken to review
current FAA guidelines relating to human factors requirements necessary for certification, and
other relevant FAA documentation. FAR Part 23 is commonly used for certifying avionics for
general aviation aircraft. Specifically, Part 23 is used for normal, utility, acrobatic, and commuter
category aircraft. Part 23 was reviewed and the human factors requirements listed in this
document. The human factors requirements are presented in the regulations in very general
terms, and focus primarily on presentation of warning information, location of instruments,
visibility of instruments, pilot workload, and warning, caution, and advisory light color schemes.
Little guidance is given in the regulations on how to measure, test and satisfy the human factors
related regulations.

A variety of FAA documents were reviewed in order to find further guidance for developing a
human factors evaluation plan. The list of documents reviewed can be found in the reference
section. The “Human Factors and Operations Checklist for Standalone GPS Receivers” [8] gives
a very detailed series of bench tests and flight tests to be performed, including detailed test
procedures and evaluation considerations. Although this document provides valuable test
procedures, methods of evaluation and measures, the measures and procedures are targeted at one
piece of equipment and have not been developed in a way that can be generalized. The document
also considers the human factors testing as a final task to be performed after the unit has been
designed and built, rather than an on-going process during the design and development stages.

The most comprehensive document identified was “Guidance for Reviewing Certification Plans
to Address Human Factors for Certification of Transport Airplane Flight Decks” by Sharon Hecht
of the Northwest Mountain Region [9]. The purpose of the document is to provide guidance to
FAA certification teams that will enable them to conduct an effective review of an applicant’s
Human Factors Certification Plan, submitted at the beginning of a certification application. The
document outlines nine steps, which provide the suggested format for a certification plan.
Sections 3 and 4 of the document provide an outline of the certification requirements and
methods of compliance using the example of FAR 25. Various methods of compliance are
suggested such as engineering evaluations or analyses, mock-up evaluations, part-task
evaluations, simulator evaluations and in-flight evaluations. Appendix A of the document also
provides discussions on how the human factors requirements specified in FAR Part 25 can be
satisfied. Although the document provides a more detailed description on how to satisfy the
certification requirements for Part 25, the descriptions lack specific guidaence and the document
does not assist the developer in feeding human factors considerations into the design and
development phases.

This document is intended to be a working document and will be updated as further relevant
documentation is found. The current state of the document reflects the compilation of material
found as a result of an initial literature review. There may however be other relevant documents
that have not been included due to time constraints. This information will be incorporated in
future updates.

2
2. Objectives
In the absence of a comprehensive FAA human factors guide, this document has been complied
with two aims. The first is to provide the user with an approach to incorporate human factors into
the design and development phases, by using an integrated human centered systems approach and
an Operator Directed Process. The second is to provide a suggested evaluation plan in
accordance with Part 23 regulations, which can be partially or fully implemented during the
various phases as suggested by the integrated human centered systems approach. The overall aim
of performing human factors evaluations and incorporating the results into the design and
development phase is to ensure the equipment is developed in accordance with the following
principles: to meet the intended function, prevent designs that are susceptible to misuse, identify
usability issues, minimize input error and enhance safety.

3
3. Recommendations

This section provides a detailed description of two methods of incorporating human factors into
the design and development phases of a product. The first is the Integrated Human Centered
Systems approach and the second is the Operator Directed Process, the latter of which focuses
primarily on complexity.

3.1 Integrated Human Centered Systems Approach

The Integrated Human Centered Systems Approach applies known techniques of human centered
design but maintains a Systems Engineering methodology in the development process. The
human is considered as a functional component of the closed loop information system. System
level trades are considered to evaluate the allocation of capability and responsibility between the
human and other components of the information systems such as the sensors, displays, or
automation systems.

A key element of the integrated approach is practical consideration for the actual operating
environment. Many proposed information system elements which look good on paper, in theory
and in static models fail in dynamic operation. A simple example would be a decision aid, which
did not consider real world behavior such as the variability in pilot response time to controller
instructions or the possibility of a blocked communication.

The key steps in the Integrated Human Centered Systems Approach are described below.

3.1.1 Model the System and Operator (or Operators) as a Closed Loop
Feedback Process

The first step in the process is to create a model of the system with the operators represented as
single elements, or as more complicated subsystems if necessary. These elements will have
inputs in the form of sensory data, and outputs in the form of control actions on various other
system elements. Fig. 1 shows an example of a closed loop model of a portion of the current
aerospace system. In this model, Air Traffic Control, the Pilot and the Airline Operations Center
(AOC) are all considered functional elements.

Surveillance:
Flight Decision Enrout e: 12.0 s
Strips ATC Terminal: 4.2 s
Aids
Dis pl ays ADS: 1s

Fli ght Plan

Vec tors
A mendments

Voic e
Pi lot
AOC:
Airl ine ACARS
Operati ons (Datalink )
Center
CDU MCP Controls

Trajec tory St at e Display s

Manual Cont rol
Commands Com mands

Flight
A ut opil ot
I ni tial Management Aircraft
Autothrust
Clearances Comput er

St ate

Navigation

Fig. 1. Simple Closed Loop Model of the ATM System.

4
 3.1.2 Determine the Information that the Operator Requires to Perform the
Task

Information requirements are defined by the inputs necessary for the operator to perform and
manage the necessary tasks. Typically a functional analysis and time line analysis of the
operation are conducted to determine a base set of information requirements. For evolutionary
systems it is more applicable to identify key issues and obtain operational insight by conducting
focused interviews and surveys of operators currently using similar systems.

3.1.3 Use the Information Requirements to Determine the

Display/Automation Requirements

Once the information requirements have been identified, the functional requirements for the
display can be derived. These requirements often highlight issues that will need to be dealt with
before continuing. It is useful at this stage to identify information which may be unobservable or
difficult to display, and perhaps re-examine its impact on task requirements.

3.1.4 Develop Prototype Systems

Based on the results of the information requirements analysis and an assessment of technological
capability, prototype information systems are developed to explore various system options and to
address issues. The systems are typically developed on rapid prototyping part-task simulators
based on graphical workstations which allow easy exploration of different system options at the
cognitive level. In many cases, fundamental issues are identified and resolved in the prototyping
process when the degree of fidelity is matched to the functional requirements of the task.

3.1.5 Perform Simulation Evaluations

Iterative simulation evaluations of prototype information system options are conducted using
controller subject populations. Both performance metrics and subjective metrics are used for
evaluation purposes.

3.1.6 Integrated Simulation Testing

For some development systems, it is necessary to run more complex simulation studies to
investigate the interaction dynamics between multiple agents (controllers & pilots). This can be
conducted in distributed simulation facilities with combinations of real and simulated systems.

3.1.7 System Evaluation

Based on the result of the simulation evaluations, system level assessments are conducted with
regard to the potential benefits and impact of the information system. This would include
development requirements, system effectiveness, safety implications and cost-benefit analysis.

3.1.8 Field Development Phase

For those systems which have favorable cost-benefit profiles, preliminary systems are developed
for field studies with live controllers and ultimately live aircraft. The results of these field studies
are used to develop system specifications which are used to procure operational systems. In some
cases it has been found beneficial to transition the technology into the field in incremental stages.
The technology is first introduced to operating central facilities in a non-interfering “shadow

5
mode”. Once operational issues have been resolved and controller acceptance has been obtained
in this manner, the technology is more easily incorporated into operational facilities. This
method has been used in initial TCAS implementation.

3.2 Operator Directed Process

The Operator Directed Process is another development process for developing systems using
automation with which humans interact. This process allows the consideration of the human
operator early in the development process. This process is similar to the Integrated Human
Centered Systems Approach, but is expanded to incorporate training material, certification and
configuration management.

Functional Automation Model is derived from

Analysis Functional Analysis, operator and
expert user input.
Automation
Model
Training material and procedures are
Derived from Automation Model.
Training Training Representation is created.
Material
Specification changes
must be consistent with Software specification is
Automation Model. Software derived from Training Material.
Specification

Software

System is certified against

Automated Model. Certification

Configuration Management verifies and

maintains consistency with Automated Model. Configuration
Management

Figure 2: Operator Directed Process (Waterfall Model)

The Operator Directed Process (ODP) is shown schematically in Figure 2. The major difference
in this process is that the training material is the source of the system specification rather than
vice versa. Developing training material early forces consideration of fundamental issues in
human-machine interaction early in the process. This contrasts with existing development cycles
that use training material to document system design. The intent is to develop a less error-prone
and more understandable system by requiring consistency between the training material,
procedural usage, and the software, and by limiting the complexity of the system through the
articulation of a model for the operator. This enables the explicit consideration of the human
operator early in the development process.

An iterative version of the ODP is shown in Figure 3.

6
Figure 3: Operator Directed Process (Iterative Model)

3.2.1 Functional Analysis

The first stage of this process is to determine the functionality that the automation system
requires. This analysis needs to be based on the existing environment in which the automation
must function and the anticipated operational and procedural usage of the automation. Several
other researchers have published work to guide this process (Boy 1998, Vicente 1999).

3.2.2 Automation Model

The key element of Operator Directed Process is the creation of an Automation Model suitable
for the pilot. It is derived based on the functional analysis and input from current design
engineers, operators, and expert users. This is a representation of the automation which can be
articulated and used operationally by the pilot and is a necessary construct for effective
monitoring. The purpose of creating this model early in the process is to use it to limit the
complexity of the automation, either by limiting the behaviors and functionality of the system, or
by consistently abstracting the system at a higher level. This model is intended to be a high level
description of the system which captures the philosophical and design goals which lead to
specific design criteria at more detailed levels. The primary goal of the automation model is that
it must be capable of describing and explaining all the behaviors of the system that matter, and all
of the derived operational procedures.

3.2.3 Incremental Instantiation

The waterfall model of system design consists of a linear set of steps, which are followed to
create a product. The waterfall model flows information and design considerations “downstream”
to be dealt with by the next stage. The major stages of this process are needs analysis and

7
specification, design, implementation, testing, and maintenance and upgrades. Similarly, for
software, these typically consist of the creation of functional analysis, followed by a software
specification, system instantiation and finally the development of documentation. Boehm (1983)
has shown that this development approach is inappropriate, and can be proven to be incorrect.
This is due to the unknowns in the development process, which requires the design of systems in
the absence of complete understanding of the problem to be solved, or its solution. By contrast,
the Operator Directed Process utilizes Boehm’s (1983) “spiral” model. This consists of a series of
repeating stages of iteration, where updates are made to an operational prototype of the final
system. In Figure 3, the sections are delineated by gray boxes to indicate that these encompass the
necessarily iterative stages of design and require human-in-the-loop testing. It is recognized that
in order to effectively design, document and evaluate early revisions of a system, it may be
necessary to create and evaluate prototypes in a manner consistent with the spiral model. The
reverse arrows shown in Figure 3 show the manner in which “downstream” events can impact
earlier stages and result in another iterative cycle. Determining when to iterate is dependent on
the size of the system. Simple systems may be able to be validated by inspection, whereas more
sophisticated systems may require full simulations in order to determine their effectiveness.

3.2.4 Training Material

One of the concerns is that any complex engineering model may not be an appropriate
representation for pilots. To overcome this issue, the ODP process derives training material based
on the automation model. This derivation assures that the proposed automation system can be
presented in a form amenable to training. The training material description of the system can then
be presented to pilots for feedback. In designing this process, few limitations have been placed on
the form or content of the training material. Rather than attempting to prescriptively specify the
form, structure, or nature of the training material, the goal is to explicitly require the
consideration of the specifics of knowledge transfer to the pilot. Domain-specific training experts
are likely to have an understanding of the appropriate material and how it should be presented.
For some applications the presentation of a structural model of the system may be sufficient
training. For others, a detailed explanation of how the system is to be used procedurally in
various operational scenarios may be more appropriate (Sherry 1999, Leveson 1998).

3.2.5 Certification

The current aircraft certification processes were originally designed for the mechanical and
electrical aspects of aircraft airframes. This approach has been successful, as shown by the
reduction in airframe-related incidents. Unfortunately, it does not appear that the approach is as
effective in the fields of software design or human factors, likely due to their implicit complexity.
The human factors aspects of certification have been recognized as being inadequate: Current
standards for type certification and operations have not kept pace with changes in technology and
increased knowledge about human performance. For example, flight crew workload is the major
human performance consideration in existing Part 25 regulations; other factors should be
evaluated as well, including the potential for designs to induce human error and reduce flight
crew situation awareness. (FAA, 1996) Currently, certification authorities do not have the means
or criteria available to require aircraft designers to create systems which address human factors
issues. With the exception (noted above) of workload issues, certification authorities do not have
the means to conduct an evaluation of human factors issues early in design. This has resulted in
the evaluation of aircraft flight decks being conducted during flight tests when a design is nearly
finalized at the end of the development cycle. At this stage, changes are both expensive and
difficult to make. After design is completed, flight-testing is also able to consider human factors
issue. However, if problems are found at this stage, it is again too expensive to change the

8
automation, and procedures are often designed to compensate. By imposing a process-oriented
solution, it may be possible to minimize the use of procedures in fixing design vulnerabilities.

3.2.6 Configuration Management

The concerns outlined above are focused in type certification of aircraft and components. This
refers to the certification of initial equipment from the primary manufacturer. A secondary
concern is that changes made to the system need to be approved as “Supplementary Type
Certificate” (STC). Any individual or company can apply to modify an existing type-certified
airplane through the STC process, but may not be aware of the design decisions made by the
original manufacturer. The “philosophy” of the flight deck, the operating assumptions, and other
consistencies designed into the system are not currently documented as part of the certification
process, and so cannot be considered during the STC process. As such, it is possible for approval
of a flight deck modification which is not consistent with the original manufacturer’s design. This
lack of “rationale capture” is a concern in current aircraft and certification processes. The basis
for design decisions is not documented during development, nor is it required by certification.
This lack of documentation makes it difficult for inconsistencies to be discovered and evaluated
by regulatory agencies, and for the underlying basis for design to be used when upgrades and
changes are made to these systems. If the Automation Model can be captured during initial design
and made explicit to parties who modify aircraft, it may be possible to maintain more consistent
systems through the life cycle of the systems (Littman, 1987).

9
4. Recommended Practices for Fulfilling FAR Part 23 Regulations
This section outlines a number of human factors requirements as outlined in FAR Part 23 and
proposes a five step evaluation plan consisting of engineering evaluations and analyses, mock-up
evaluations, part-task evaluations, simulator evaluations and in-flight evaluations.

4.1 Human Factors Requirements

Table 1 outlines a number of human factors requirements as outlined in FAR Part 23.

FAR Section (Part 23) Requirement Relevant discussion from FAR Part
25, extracted from “Guidance for
Reviewing Certification Plans to
Address Human Factors for
Certification of Transport Airplane
Flight Decks” by Sharon Hecht [9].

General Human
Factors Requirements

23.1309 (b) (3) (3) Warning information must

be provided to alert the crew to
unsafe system operating
conditions and to enable them
to take appropriate corrective
action. Systems,
controls, and associated
monitoring and warning means
must be designed to minimize
crew errors that could create
additional hazards.
23.1321(a) (a) Each flight, navigation, and The applicant may wish to perform
power plant instrument for use analyses of the visual angles to each
by any required pilot during of the identified instruments. Final
takeoff, initial climb, final assessments of the acceptability of
approach, and landing must be the visibility of the instruments may
located so that require a simulator with a high
any pilot seated at the controls degree of geometric fidelity and/or
can monitor the airplane's flight the airplane.
path and these instruments with
minimum head and eye
movement. The power plant
instruments for
these flight conditions are those
needed to set power within
power plant limitations.
23.1321(e) (e) If a visual indicator is Demonstrations and tests intended
provided to indicate to show that these indications of
malfunction of an instrument, it instrument malfunctions, along with

10
 must be effective under all
probable cockpit lighting
conditions.
23.1523 The minimum flight crew must
be established so that it is
sufficient for safe operation
considering--
[(a) The workload on individual
crewmembers and, in addition
for commuter category
airplanes, each crewmember
workload determination must
consider the
following:
(1) Flight path control,
(2) Collision avoidance,
(3) Navigation,
(4) Communications,
(5) Operation and monitoring of
all essential airplane systems,
(6) Command decisions, and
(7) The accessibility and ease of
operation of necessary controls
by the appropriate crewmember
during all normal and
emergency operations when at
the
crewmember flight
station.]
(b) The accessibility and case of
operation of necessary controls
by the appropriate
crewmember; and
(c) The kinds of operation
authorized under Sec. 23.1525.
23.1543 For each instrument--
(a) When markings are on the
cover glass of the instrument,
there must be means to
maintain the correct alignment
of the glass cover with the face
of the dial; and
(b) Each arc and line must be
11
other indications and alerts, are
visible under the expected lighting
conditions will typically employ the
use of production quality hardware
and careful control of lighting
conditions (e.g., dark, bright
forward field, shafted sunlight).
Simulators and aircraft are often
used, although supporting data from
laboratory testing may also be
useful.
The applicant may choose to use
workload analyses (such as time-
line analysis) to evaluate certain
workload issues. Other evaluations
of workload typically involve
trained pilots in either a high
fidelity simulation or in actual
airplanes. There are a number of
possible workload assessment
techniques that can be successfully
employed. An efficient means for
selecting test conditions is to focus
on those operational and/or failure
scenarios that are likely to result in
the highest workload conditions.
Dispatch under the Minimum
Equipment List (MEL) also should
be considered, in combination with
other failures that are likely to result
in significantly increased workload.
Since no objective standard for
workload is available, applicants
may wish to compare the workload
in the new/modified airplane with
that in a well-understood,
previously certified airplane.
The applicant may choose to use
computer modeling to provide
preliminary analysis showing that
there are no visual obstructions
between the pilot and the instrument
markings. Where head movement
is necessary, such analyses also can
be used to measure its magnitude.
 wide enough and located to be
clearly visible to the pilot.
[(c) All related instruments
must be calibrated in
compatible units.]
System-Specific HF
Requirements
23.1381(b) The instrument lights must be
installed so that their direct
rays, and rays reflected from
the windshield or other surface,
are shielded from the pilot's
eyes.
Specific Crew
Interface
Requirements
23.1322 If warning, caution, or advisory
lights are installed in the
cockpit, they must, unless
otherwise approved by the
Administrator, be--
(a) Red, for warning lights
(lights indicating a hazard
which may require immediate
corrective action);
(b) Amber, for caution lights
(lights indicating the possible
need for future corrective
action);
12
Other analysis techniques can be
used to establish appropriate font
sizes, based on research-based
requirements. Mock-ups also can
be helpful in some cases. The data
collected in these analysis and
assessment can be used to support
final verification in the flight deck,
using subjects with vision that is
representative of the pilot
population, in representative
lighting conditions.
The applicant may be able to
develop analytical techniques that
identify potential sources of glare
and reflections, as a means for
reducing the risk of problems
identified after the major structural
features have been committed.
Mock-ups also may be a useful
means for early assessments.
However, analysis results typically
must be verified in an environment
with a high degree of geometric and
optical fidelity. Both internal (e.g.,
area and instrument lighting) and
external (e.g., shafting sunlight)
sources of reflection should be
considered.
Compliance with this requirement is
typically shown by a description of
each of the warning, caution, and
advisory lights. Evaluations may
also be useful to verify the
chromaticity (e.g., red looks red,
amber looks amber) and
discriminability (i.e., colors can be
distinguished reliably from each
other) of the colors being used,
under the expected lighting levels.
These evaluations can be affected
by the specific display technology
 (c) Green, for safe operation being used, so final evaluation with
lights; and flight quality hardware is
(d) Any other color, including sometimes needed. A description
white, for lights not described of a well-defined color-coding
in paragraphs (a) through (c) of philosophy that is consistently
this section, provided the color applied across flight deck systems
differs sufficiently from the can be used to show how the design
colors avoids “possible confusion.”
prescribed in paragraphs (a)
through (c) of this section to
avoid possible confusion.
[(e) Effective under all probable
cockpit lighting conditions.]
Table 1: Human Factors Requirements

4.2 Proposed Human Factors Evaluation Plan

Engineering Evaluations or Analyses

(These assessments can involve a number of techniques, including such things as procedure
evaluations (complexity, number of steps, nomenclature, etc); reach analysis via computer
modeling; time-line analysis for assessing task demands and workload; or other methods,
depending on the issue being considered.)

All requirements have been taken from Part 23 except basic functionality and identification
testing.

Regulation Requirement Test

Basic functionality and Information must be Poster boards, tree diagrams
identification testing easily obtained from the outlining functionality, natural
display and accurately mappings etc.
input.

Equipment Location Flight, navigation and Ergonomic computer modeling

Instrument Reflections
Table 2: Engineering Evaluations and Analysis
power plant instruments
must be located so the
pilot can monitor flight
path and instruments with
minimum head and eye
movements
Pilots eyes must be Computer modeling of direct and
shielded from direct and reflected rays.
reflected rays of the
instruments

Mock-up Evaluations

13
These types of evaluations use physical mock-ups of the flight deck and/or components. They
are typically used for assessment of reach and clearance, thus, they demand a high degree of
geometric accuracy.

Regulations Requirement Test

Equipment Location
Flight, navigation and Physical mock-up to assess reach
power plant instruments and clearance, measure visual
must be located so the angles to each instrument and
pilot can monitor flight measure head and eye movements
path and instruments with required to read/input information
minimum head and eye into the instruments.
movements
Table 3: Mock-up Evaluations

Part-Task Evaluations
These types of evaluations use devices that emulate (using flight hardware, simulated systems, or
combinations) the crew interfaces for a single system or a related group of systems. Typically
these evaluations are limited by the extent to which acceptability may be affected by other flight
deck tasks.

Regulation Requirement Test

Basic functionality and Information must be easily Correctness of information
identification testing obtained from the display and extraction, performance and
accurately input. user input (e.g. no. of errors)

Warning Information Warning information must be Recognition of alerts, no. of

able to alert crew to unsafe correct actions and no. of
system operating conditions. errors.
Warning systems must lead to
pilots taking corrective actions
and minimize crew errors
Workload Excessive workload associated Workload analyses for each of
with flight path control, the separate systems
collision avoidance,
navigation, communications,
operation and monitoring all
essential airplane systems and
command decisions must be
avoided.
Alignment of cover glass Pilots must be able to maintain Ability to maintain alignment
correct alignment of the cover using the instrument, test for
glass (when markings are on no. of correct responses.
the cover glass of an
instrument) with the face of
the dial.
Visibility of instrument Pilots must be able to see Ability to see markings with
markings clearly all markings on an varying lighting conditions,

14

Visibility and color coding of
warning, cautionary and
advisory lights
instrument head positions etc, test for no.
of correct responses.
Ensure adherence of warning, Ability to see, identify and
cautionary and advisory lights discriminate lights under all
with the following color- probably lighting conditions.
coding scheme, red (warning), Test no. of correct responses.
amber (caution) and green
(safe). Lights must be visible
under all probably lighting
conditions, verify chromaticity
and discriminability.
Table 4: Part-Task Evaluations

Simulator Evaluations
These types of evaluations use devices that present an integrated emulation (using flight
hardware, simulated systems, or combinations) of the flight deck and the operational
environment. They also can be “flown” with response characteristics that replicate, to some
extent, the responses of the airplane. Typically, these evaluations are limited by the extent to
which the simulation is a realistic, high fidelity representation of the airplane, the flight deck, the
external environment, and crew operations. The types of pilots (test, instructor, airline) used in
the evaluations and the training they receive may significantly affect the results and their utility.

Basic functionality and Information must be easily Correctness of information

identification testing obtained from the display and extraction, performance and
accurately input. user input (e.g. no. of errors)
of all systems, tested during a
real flight scenario.

Warning Information Warning information must be Recognition of alerts, no. of

Visual Malfunction
Workload
Alignment of cover glass
able to alert crew to unsafe
system operating conditions.
Warning systems must lead to
pilots taking corrective actions
and minimize crew errors
Pilots must be able to see
visual malfunction indicators
under all probably cockpit
lighting conditions.
Excessive workload associated
with flight path control,
collision avoidance,
navigation, communications,
operation and monitoring all
essential airplane systems and
command decisions must be
avoided.
Pilots must be able to maintain
correct alignment of the cover
glass (when markings are on
correct actions and no. of
errors, tested using all systems
during a real flight scenario
No. of correct identification of
visual malfunction indicator
with varying lighting
conditions.
Workload analyses during a
real flight scenario.
No. of correct instrument
readings during a real flight
scenario.

15

Visibility of instrument
markings
Visibility and color coding of
warning, cautionary and
advisory lights
the cover glass of an
instrument) with the face of
the dial.
Pilots must be able to see Ability to see markings with
clearly all markings on an varying lighting conditions,
instrument head positions during a real
flight scenario
Ensure adherence of warning, Ability to see, identify and
cautionary and advisory lights discriminate lights under all
with the following color- probably lighting conditions,
coding scheme, red (warning), during a real flight scenario.
amber (caution) and green
(safe). Lights must be visible
under all probably lighting
conditions, verify chromaticity
and discriminability.
Table 5: Simulator Evaluations

In-Flight Evaluations
These types of evaluations use the actual airplane. Typically, these evaluations are limited by the
extent to which the flight conditions of particular interest (e.g., weather, failures, unusual
attitudes) can be located/generated and then safely evaluated in flight. The types of pilots (test,
instructor, airline) used in the evaluations and the training they receive may significantly affect
the results and their utility.

In-flight evaluations follow the same format as simulator evaluations and additionally include
tests to ensure shielding of pilots eyes from direct and reflected rays and suitability of equipment
location.

16
5. Other Issues/Comments
During the past five months a series of human factors evaluations were performed using both
EFIS and HITS displays. As a results of these studies, the following issues/problems were
identified.

5.1 Limitation of Using Low Fidelity Simulators to Capture Accurate

Performance Data

Low fidelity simulators can be constructed quickly and at low cost by using a computer monitor,
joystick and off the shelf software packages, such as Microsoft Flight Simulator. However, the
simulator results may not be a good representation of the display/instrument under investigation.
The validity of results depends on the measures being taken during the experiment. Performance
data from a low fidelity simulator may provide a poor representation of the display due to
inaccurate flight dynamics or poor/slow response flight controls. A high fidelity simulator or in-
flight testing is highly recommended for performance measures. However, other measures such
as situational awareness can be accurately obtained using a low fidelity simulator. The advantage
of using a simulator rather than in-flight studies is the ability to control the environment and limit
the number of variables.

5.2 Evaluation Measures

Human factors evaluations require a number of different measures in order to obtain a true
understanding of the displays strengths and weaknesses. Performance data provides evidence for
the pilot’s ability to use the display (providing a high fidelity simulator is used). Subjective data
is another valuable method of data collection as this gives the opinion of the user, which may lead
to a difference conclusion than the performance data. Both types of data should be collected
synchronized and analyzed together.

17
6. References

[1] R. Hansman et al., Integrated Human Centered Systems Approach to the Development of
Advanced Cockpit and Air Traffic Management Systems, 16 th Digital Avionics Systems
Conference, October 1997.

[2] K.M. Joseph and D.W.Jahns, Enhancing GPS Receiver Certification by Examining
Relevant Pilot-Performance Databases, Office of Aviation Medicine, February 2000.

[3] S.A. Shappell and D. A. Weigmann, The Human Factors Analysis and Classification
System – HFACS, DOT/FAA/AM-00/7, Office of Aviation Medicine, Washington DC,
February 2000.

[4] S. Vakil, Analysis of Complexity Evolution Management and Human Performance Issues
in Commercial Aircraft Automation Systems, Thesis submitted to the Massachusetts
Institute of Technology, May 2000.

[5] D.A Weignmann, A Human Error Analysis of Commercial Aviation Accidents Using the
Human Factors Analysis and classification system, Office of Aviation Medicine,
February 2001.

[6] K.W. Williams, Comparing Text and Graphics in Navigation Display Design, Office of
Aviation Medicine, February 2000.

[7] Application Guide for Obtaining a Supplemental type Certificate, Advisory Circular AC
21-40, FAA, US Department of Transportation, May 1998.

[8] FAA Aircraft Certification Human Factors and Operations Checklist for Standalone GPS
Receivers (TSO C-129 Class A), Research and Special Programs Administration, Volpe
National Transportation Systems Center, April 1995.

[9] Guidance for Reviewing Certification Plans to Address Human Factors for Certification
of Transport Airplane Flight Decks, Department of Transportation, FAA, Policy
Statement Number ANM-99-2.

[10] Description of the FAA Avionics Certification Process, FAA, Aircraft Certification
Service, Aircraft Engineering Division, Avionics Systems Branch, April 23 1997.

18
Appendix A - Guidance for Reviewing Certification Plans to Address
Human Factors for Certification of Transport Airplane Flight decks
(Policy Statement Number ANM-99-2)

19
Appendix B – Human Factors in Certification Presentation

20
Appendix C - Application Guide for Obtaining a Supplemental Type
Certificate (AC 21-40)

21