Pilot Trainee

Validation Report
Board of Pilot Commissioners
For the Bays of San Francisco, San Pablo, and
Suisun

Project Conducted by:

Selection Services Program
California Department of Human Resources
June 2019

Template Revision Date: 12-8-16

 Validation Report

For the classification of

Pilot Trainee

Board of Pilot Commissioners

For the Bays of San Francisco, San Pablo, and Suisun

June 2019

Steve Shriver
Consultant

California Department of Human Resources

Selection Services Program
1515 S Street, North Bldg., Suite 500
Sacramento, CA 95811
 Validation Report
Pilot Trainee
TABLE of CONTENTS

TABLE of CONTENTS ................................................................................. i

LISTING of APPENDICES .......................................................................... ii
INTRODUCTION ........................................................................................ 1
BACKGROUND AND PURPOSE ............................................................... 1
METHOD .................................................................................................... 1
DEVELOPMENT OF EXAMINATION PLAN ............................................... 2
DEVELOPMENT OF EXAMINATION PROCESS SCORING MODEL ........ 3
DEVELOPMENT OF SELECTION PROCEDURES .................................... 4
PASS POINT SETTING .............................................................................. 5
PILOT TESTING ......................................................................................... 6
RESULTS ................................................................................................... 7

i
 Validation Report
Pilot Trainee
LISTING of APPENDICES

APPENDIX A: Examination Panel Subject Matter Expert Participants ..... 10

APPENDIX B: Written Examination Content Specifications ..................... 13
APPENDIX C: Simulation Examination Content Specifications ................ 18
APPENDIX D: Examination Item/KSAPC Linkage ................................... 24
APPENDIX E: Pass Point Data ................................................................ 31

ii
 Validation Report
Pilot Trainee
INTRODUCTION

This report documents the development and validation of the selection process used to
select Pilot Trainees into the Pilot Trainee Training Program (Training Program), as used
by the Board of Pilot Commissioners (BOPC) for the Bays of San Francisco, San
Pablo, and Suisun.

The process was specifically designed to comply with both the letter and spirit of equal
employment opportunity laws and court precedents as well as the methodology
acknowledged in the Federal Uniform Guidelines on Employee Selection Procedures and
the Standards for Educational & Psychological Testing (American Educational Research
Association, American Psychological Association, and the National Council on
Measurement in Education, 1999). Subsequent sections of this report detail the specific
procedures followed in order to develop a content-valid selection instrument.

BACKGROUND AND PURPOSE

BOPC entered into a contractual agreement with the Selection Services Program
(SSP) to develop a valid selection instrument for the Pilot Trainee position as used by
BOPC. Examination development activities were based on and supported by the job
analysis completed in November 2016. The intent of this report is to document the
evidence of content validity for all selection procedures developed and used to select
new Pilot Trainees into the Training Program.

Upon entering the Pilot Trainee program, the trainees begin hands-on training
specifically tailored for the Pilot Trainee’s background, skills, and needs. The initial pilot
training program is one to three years. During this time, the Pilot Trainees perform their
pilotage duties under careful supervision of veteran pilots on board working vessels in
the various ports throughout the bays of San Francisco, San Pablo, and Suisun. Pilot
Trainee performance on each trip is evaluated in writing as to the candidate’s basic ship
handling skills, knowledge of local waters, and ability to handle the requirements of the
job. These evaluations are collected and reviewed by the Pilot Evaluation Committee
before the pilot license is issued. Failure to meet any performance benchmark may
result in a trainee being placed on probation.

METHOD

Prior to examination development, to ensure that the examination reflected the

actual tasks performed by San Francisco Bar Pilot Trainees, SSP analysts reviewed
the 2016 job analysis with Bar Pilot SMEs to ensure the essential tasks and
corresponding knowledge, skills, abilities, and personal characteristics (KSAPCs)
identified in the job analysis were still critical to the job and expected at entry into
the Pilot Trainee position. Examination development activities were based on a
review of these job analysis results and the identification of those KSAPCs most
appropriate for assessment. Documentation demonstrating the relationship between
the job tasks and KSACPs to the exam content is available in the appendix section
of this report.
1
 Validation Report
Pilot Trainee

(See APPENDIX D for the Examination Item/KSAPC Linkage)

The Board’s Pilot Evaluation Committee identified licensed, working pilots to provide
subject matter expertise in interviews and focus groups to support exam
development activities. These licensed pilots were consulted to identify the major
subject matter areas and to develop the examinations. The participating pilots had
varying degrees of experience ranging from newly licensed to very experienced. All
individuals involved with examination development activities signed an examination
security agreement which specified that they would keep all examination materials
secure, discuss the examination only during Board-sponsored meetings, and avoid
involvement in any examination-oriented review program for prospective pilot
candidates. Signers included Pilot Evaluation Committee (PEC) members, PSC
staff, staff at the California Maritime Academy, and pilots who served as subject
matter experts (SMEs) in all phases of written and simulation examination
development scoring, and/or established the passing score for the written and
simulator examinations.

Standardized procedures and protocols to administer both the written examination

and simulator examination were developed to ensure that every candidate had the
same examination experience.

Involved Project Processes and Activities:

• Conducted a job analysis study of the Bar Pilot Trainee to identify critical
and expected upon entry knowledge, skills and abilities for testing purposes
• Developed detailed content specifications for the written examination
• Developed detailed content specifications for the simulator examination
• Developed multiple-choice questions based on the content specifications
and job relatedness for new trainees
• Provided critical review of items by evaluating accuracy of the content and
making editorial revisions
• Selected items for the published examination based on content
specifications and SME feedback
• Evaluated scenarios and rating scales for the simulator examination for
compliance with test and measurement principles
• Evaluated the standardized protocols that were developed to administer the
simulator examination for compliance with test and measurement principles
• Facilitated the process for establishing criterion-referenced passing scores
(modified Angoff) for the written and simulator examinations
• Determined the passing scores for the written and simulator examinations
• Developed the ranked list of candidates who passed both the written and
simulator examinations

DEVELOPMENT OF EXAMINATION PLAN

KSAPC statement were reviewed to determine which assessment methodology would

2
 Validation Report
Pilot Trainee

be the most effective method of measurement. It is not uncommon for a KSAPC to be

measurable using a variety of methods. For example, a person’s ability to review
information and take appropriate action may be measured using a written examination, a
structured interview, or a job simulation activity, among other methods.

When considering the design of an appropriate selection procedure, it is strongly

recommended that agencies consider using methods that might achieve their assessment
goals while minimizing potential bias. Since most KSAPCs can be measured in a variety
of ways, selecting the assessment technique that is the most effective for the broadest
array of potential incumbents complies with the true spirit of anti-discrimination law.

SSP considers the variety of assessment procedures available when making the final
recommendation to the agency in terms of the best approach for creating a valid and fair
selection procedure. Issues considered included many pertinent variables such as
administration resources, cost, efficiency, development resources, predictive validity of
the various examination modalities, anticipated candidate pool size, vacancies to be filled,
anticipated candidate characteristics, and job requirements to ensure that assessment
modalities do not conflict with actual job requirements.

Upon consideration of all of the available information collected through this job analysis
and examination development process, it was confirmed that the assessment procedures
would consist of an Experience Evaluation, Written Examination and a Simulation
Examination.

DEVELOPMENT OF EXAMINATION PROCESS SCORING MODEL

SSP project staff, in collaboration with the Bar Pilot Subject Matter Experts (SMEs),
included as part of the examination development process a scoring model for the overall
examination process for the Pilot Trainee position. The selection process consisted of
three equally weighted components: experience, written examination and simulator
examination. Through this process, it was determined that the scoring model would
integrate all selection procedures in the following manner:

Experience: The process begins with an candidate application where they are
tasked with demonstrating that they have sufficient experience to meet the
requirements of 7 CCR, Division 2, Article 4, Subsection 213(e). The
application is evaluated and those candidates who can establish proof of
sufficient experience and other requirements are able to continue on in the
examination process by sitting for the written examination. To sit for the written
examination candidates must possess sufficient tug, deep draft, and/or piloting
experience.

Written Examination: Candidates complete the Written Examination. A pass

point for this examination was established with Bar Pilot SMEs post
examination administration. Candidates are required to pass the Written
Examination in order to continue on in the examination process.

Simulation Examination: Candidates who passed the Written Examination

3
 Validation Report
Pilot Trainee

would continue on to complete the Simulation Examination. In the bridge

simulator examination, a variety of real world complications and hazards are
presented, and the candidates are evaluated on how well they handle
unforeseen events. A separate pass point was established with Bar Pilot
SMEs post examination administration.

All three components must be passed in order for the candidate to make the final
list. Final scores for each component were weighted to ensure equal weight for
each component, and then the three components were totaled. Total scores
were used to rank order candidates, thus allowing the commission to select
individuals who scored the highest first into the Trainee Program, until the Board
has exhausted the list of candidates.

DEVELOPMENT OF SELECTION PROCEDURES

Written Examination:
Five (5) examination development meetings were conducted to draft, review, finalize,
and approve all content for the Written Examination as well as the BOPC’s item bank.
Exam items were renewed and referenced to current sources. The examination
development meetings were convened on February 4-6, 2019 and February 25-26,
2019 were led by SSP project staff with the participation of several current and/or
former Bar Pilots who served as SMEs. During these meetings, SMEs reviewed the
Job Analysis data and confirmed the content areas to be assessed by the exam. SMEs
and SSP then reviewed each established examination item to ensure the content
corresponded with important and expected at entry KSAPCs and the difficulty level for
all examination components were appropriate. The resulting final version of the
examination was approved by the SMEs. Quality reviews of materials were performed
prior to administration.

The written examination consists of 150 multiple-choice items, with each item being
weighted equally at one point. Questions on the written examination were distributed
with respect to the weights of four subject matter areas: (1) pre-transit planning, (2)
master/pilot transition, (3) route piloting, and (4) docking, undocking, & anchorage.
Weights of subject matter areas were derived from job analysis results.

(See APPENDIX B for Written Examination Content Specifications)

Simulation Examination:
Thirteen (13) examination development meetings were conducted to review, edit,
finalize and approve all content for the Simulator Examination. The examination
development meetings convened on the following dates in 2019 January 22, February
15-16, March 12, March 27, April 4, April 15, May 10, and May 17. The development of
the Simulation Examination involved several SMEs who worked to coordinate,
program, test, and validate the examination procedure and content. These SMEs
included: a computer programmer (who is also an experienced mariner), a Coordinator
who was a retired pilot, multiple active pilots, and CalHR staff. The Simulation
Examination evaluates the candidates on the following seven subject matter areas: (1)
4
 Validation Report
Pilot Trainee

situational awareness, (2) appropriate response, (3) ability to respond correctly under
stress, (4) communication and bridge presence, (5) fundamental shiphandling, (6)
bridge resource management, and (7) rules of the road. The number of measurement
opportunities were distributed according to the weights of subject matter areas from the
Job Analysis

The simulator’s examination development process involved multiple repetitions of

developing items, live testing with pilots, and revising content as necessary. Multiple
revisions were necessary to ensure that nearly all of the possible actions that could be
taken by candidates were identified and built into the system. The scoring system was
designed around the metrics of +2 for highly effective, +1 for acceptable, 0 for
ineffective, -1 for an allision (touching), and -2 for a collision. Evaluation forms were
designed such that the candidates’ evaluations were standardized. Numerous
evaluation form revisions were necessary to achieve consensus regarding the
wording, rating scales, and order of presentation of the measurement opportunities.
Simulator raters were trained on the importance of ensuring the that the examination
experience was consistent for each candidate. To support this, raters were briefed on
the potential negative consequences associated with inconsistency (e.g., appeals,
lawsuits, impact of unqualified candidates, negative press, candidate perceptions) and
its impact on BOPC To ensure there were an adequate number of items developed to
appropriately assess the skill of the candidates, 60 measurement opportunities were
developed.

(See APPENDIX C for Simulation Examination Content Specifications)

(See APPENDIX A for Examination Panel SME Information)

PASS POINT SETTING

The Uniform Guidelines require that the pass/fail cutoffs should be “…set so as to be
reasonable and consistent with the normal expectations of acceptable proficiency in the
workforce” (Section 5H). For licensing examinations, it is necessary to establish a
passing score (cut score) based on the concept of minimal acceptable competence. The
methodology most frequently applied and which was used for the written and simulator
examination is the Angoff method.

1. Select a representative sample SMEs who are truly experts in the content area
and are diverse in terms of ethnicity, gender, geography, seniority (with a
minimum of one year experience), and “functional areas” of the target position.
Supervisors and trainers can also be included.

2. Facilitate a discussion with the SME panel to clarify and define the concept of a
“minimally qualified applicant.” The definition should be limited to an applicant
who possesses the necessary, baseline levels of the KSAPC measured by the
examination item to successfully perform the first day (before training) on the job.

3. Ask the SMEs to provide their ratings regarding probability of a minimally

qualified applicant answering the examination item correctly.
5
 Validation Report
Pilot Trainee

4. Average the ratings across all SMEs for every item. Add these average ratings
to establish the preliminary pass point.

The written and simulator assessments both require cut scores to be established. There
are no predetermined standards to compare performance when determining if the
standard set for passing is correct, or the number of points awarded for simulator
activities is correct. Thus, SMEs were needed to establish appropriate pass points for
both exam components. Participating SMEs provided their judgments based on training
received from CalHR staff regarding Minimally Qualified Candidates, guidance from Bar
Pilots with current and previous exam development and administration experience,
examinations’ documentation, and their own expertise. CalHR staff, and Bar Pilots
involved with exam development and administration activities, instructed SMEs to use
their evaluative skills and to maintain consistent scoring principles for each candidate.
While consensus amongst SME raters appears to be a necessary component for expert
judgments, SME agreement does not guarantee correctness. Differences in ratings
between the SMEs are likely a result of having different perspectives due to different
settings or work experiences (e.g., deep sea versus inland piloting). SMEs were asked
to be highly discriminating when setting the passing scores or performing evaluations,
and were reminded of the expectation to set pass points appropriate for the Bar Pilot
Trainee level, rather than of an experienced, licensed Bar Pilot – the level of participating
SMEs.

For the written examination component, there are 150 points possible. The Angoff
method (as described above) was used to set the final pass point. Seven pilots, serving
as SMEs, evaluated the exam items based on the performance expected of minimally
competent trainees. After the SMEs received training in the Angoff process, they
responded to the following question for each item “What percentage of minimally
competent candidates (pilot trainees) would answer the item correctly?” The data was
aggregated across all items and all raters and divided by seven, the number of SMEs.
The results from the workshop established the passing score (cut score) at 106. The
passing score was optimally established—the closest score above was 108 and the
closest below was 100 which left clear gaps on both the pass and fail sides of the
passing score.

For this simulator examination component, there are 120 points possible. The Angoff
session for the simulator component included fourteen SMEs. Similar to the written
examination, they received necessary training, and proceeded to provide their
percentage of minimally acceptable candidates who would answer the item correctly.
Data was again aggregated across all 60 items and divided by the number of
participants. The final pass point was set to 52 points.

(See APPENDIX E for Pass Point Data)

PILOT TESTING

6
 Validation Report
Pilot Trainee

Written Examination: In order to maintain the security of the written examination, it was
decided that the examination would not be piloted amongst SMEs. In lieu of a pilot test,
the examination was evaluated by SMEs as part of a final review, and later reviewed
again by CalHR staff to ensure examination quality. Such efforts, along with information
learned from frequent discussions with SMEs helped ensure: the clarity of instructions
and exam items, and that previously established time limits remained appropriate for
administration.

Simulator Examination: Piloting of the simulator component occurred throughout the

exam development process, as participating examination raters and other invited SMEs
were asked to complete the assessment activities for the exam development focus group
to view and evaluate. The benefits of this pilot testing included: ensuring that the
programming software was operating appropriately, observing the variables and
responses involved as candidates navigate throughout the examination and its items,
ensuring the user-friendliness of tools (e.g., rating forms) developed for use by raters
and exam proctors, ensuring the user-friendliness of candidate materials and
instructions, establishing appropriate administration time limits, providing practice for
raters to evaluate candidate and to determine how to rate newly identified observances,
and other efforts to improve the overall quality of this component. Quality improvement
modifications were consistently implemented as a result of these efforts.

RESULTS

Experience Points: BOPC staff evaluated 35 candidate applications and determined

that 33 met or exceeded the minimum requirements to qualify for sitting for the written
examination. In addition, some individuals were awarded points for their experience.
Candidate scores ranged from 0 to 35 points.

Written Examination: Candidates were provided 4 ½ hours to complete the 150

multiple-choice item examination. The administration occurred in the California Maritime
Academy’s cafeteria, with candidates being seated at large round tables, two candidates
to a table. The arrangement helped ensure that candidates were not able to observe
other candidate’s responses. Additionally, two versions of the written examination were
administered to further deter candidate cheating. Of the 33 candidates who qualified to
sit for the written exam, 32 appeared on the date of the exam with one no-show.

Of the 32 candidates who sat for the examination 28 successfully passed, enabling them
to continue forward to take the Simulator examination component. The highest score on
the written exam was 139 and the lowest score was 73. The mean score was 119.66
and the median score was 123.5.

Simulation Examination: The SMEs involved with examination development activities

received extensive training and guidance for evaluating and scoring the candidates. For
training and practice purposes, the SME evaluators proceeded as if the person piloting
the simulator was an actual pilot. Each evaluator first conducted their evaluation
independently before convening to discuss results with other raters. These experiences
helped confirm rater consistency and that no aspects included in the design of the
7
 Validation Report
Pilot Trainee

simulation had a negative impact on candidates’ performance.

The processes applied for training the evaluators involved during actual administration
were quite similar to what occurred during administration. The evaluators observed and
rated pilots during numerous dry runs until the evaluators were confident of the
consistency of their evaluations. Evaluators were made aware of the consequences and
impacts of their ratings by being reminded of two perspectives. The first necessity being
that pilot trainees are expected to protect the public’s health, safety and welfare. The
second is the interest to ensure that the candidates’ are assessed fairly and consistently
in the examination program and selection process, where each candidate has the same
opportunity to succeed throughout the selections process. The evaluators included
representation from BOPC’s Pilot Evaluation Committee, state licensed pilots from
another jurisdiction, and industry representatives with command experience on deep
draft vessels.

The California Maritime Academy hosted the simulation examination on a full bridge
simulator over three days, June 5th, 6th, 7th, and 8th, 2019. On the 5th the candidates who
successfully passed the written examination were fully briefed on the process, received
materials designed to assist in preparing for the examination which they were allowed to
take from the site, received a hands- on orientation to the bridge, and observed the
vessel’s track through the simulation exercise.

All candidates were given a scheduled time to report to their simulator exercise. Each
candidate was given 25 minutes prior to their simulator exercise to study materials
relating to the exam in the map room at California Maritime Academy. This information
included vessel transit information for the day in the bay where they were going to be
transiting. There were six evaluators assigned to observe the candidates and participate
in discussions of the candidate’s performance based on the scoring criteria. Three
evaluators would be assigned and responsible for evaluating each candidate. They were
stationed in the simulator with the candidate so they could observe all candidate actions.
A fourth evaluator was stationed in a room above the simulator where measuring
equipment was used to determine measurements for certain items. After the simulation
was completed, the three evaluators in the simulator conducted a “hot wash” session
where they discussed the candidate’s performance on each item based on the scoring
criteria. The fourth evaluator provided the measurements for items that required it, but
was not involved in providing a final score for the candidate. The three evaluators would
provide final scores for the candidate on each item based on their observations and the
discussion with the other evaluators after the session. To obtain a final score for each
candidate, the scores of the three evaluators were averaged for each item. To obtain a
final score, the average scores for each of the 60 items were added together.

Similar to how the pass point was set for the written examination, an Angoff workshop
was conducted with 14 SMEs who had been involved in the simulation. All evaluators
were intimately familiar with the exam content and the evaluation process. The
evaluators have trained pilots enabling them to have an in-depth appreciation of the skill
set needed by pilot trainees. They were asked to evaluate the level of performance that a
minimally competent candidate would exhibit on each of the 60 measurement
8
 Validation Report
Pilot Trainee

opportunities. They were asked to judge the level of performance based on the
examination rating system, +2, +1, 0, -1, and -2. The passing score was a scaled 52. Of
the twenty-seven candidates who took the simulator exam, nineteen obtained a passing
score.

Scaling: The three components of the assessment process received equal weighting.
With the total possible points varying for each exam component, it was necessary to
scale the experience points and the simulation exercise points to be comparable to the
maximum score achievable on the written examination. Thus, the maximum experience
points and the maximum simulator scores were scaled to equal 150. The maximum
experience points was 90, which was scaled to equal 150 and the maximum score
achievable on the simulation exercise was 120, which was also scaled to equal 150.

PROCESS FOR ESTABLISHING LISTING OF SUCCESSFUL CANDIDATES

The points awarded from each exam component were combined to establish the exam’s
total possible points. All three components must be passed by the candidate to pass the
exam overall. Candidates, who possessed sufficient qualifying experience and
successfully passed both the written and simulator components were placed on a list,
ranked according to the total points earned. This allows the commission to select
individuals who scored the highest first into the Trainee Program, until the Board has
exhausted the list of candidates. The list was presented in a separate report to the
BOPC.

9
 Validation Report
Pilot Trainee

APPENDIX A: Examination Panel Subject Matter Expert Participants

Written Exam Development Meetings

Date: February 4, 5, and 6, 2019

Name Classification Contact

1 Tom Burger Bar Pilot EMAIL REDACTED
2 Randy Pinetti Bar Pilot EMAIL REDACTED
3 Steve Teague Bar Pilot EMAIL REDACTED
4 Larry Teague Bar Pilot EMAIL REDACTED

Written Exam Development Meetings

Date: February 25 and 26, 2019

Name Classification Contact

1 Tom Burger Bar Pilot EMAIL REDACTED
2 Pete McIsaac Bar Pilot EMAIL REDACTED
3 Steve Teague Bar Pilot EMAIL REDACTED
4 William Lemke Bar Pilot EMAIL REDACTED

Written Examination Angoff Session SMEs

Date: June 3, 2019

Name Classification Contact

1 Steve Teague Bar Pilot EMAIL REDACTED

2 John Carlier Bar Pilot EMAIL REDACTED

3 Jesse Pullin Bar Pilot EMAIL REDACTED

4 Kevin Freese Bar Pilot EMAIL REDACTED

5 Drew Aune Bar Pilot EMAIL REDACTED

6 Erik Fawcett Bar Pilot EMAIL REDACTED

7 Matt Lingo Bar Pilot EMAIL REDACTED

10
 Validation Report
Pilot Trainee

Simulator Examination Development Meetings (Main Crew)

Dates: January 22, February 15, 16, March 12, 27, April 4, 15, & May 10 &17, 2019

Name Classification Contact

1 Steve Teague Bar Pilot EMAIL REDACTED
2 Eric Robinson Bar Pilot EMAIL REDACTED
3 Mark Haggerty Bar Pilot EMAIL REDACTED
4 Dan Boriolo Bar Pilot EMAIL REDACTED

5 John Carlier Bar Pilot EMAIL REDACTED

6 Allen Garfinkle Ship Master EMAIL REDACTED
Test Proctor with
7 J.D. Gates California Maritime EMAIL REDACTED
Academy (CMA)
8 Victor Schisler Test Programmer, CMA EMAIL REDACTED

Simulator Examination Development Meetings (Individual Simulation Runs)

Name Classification Contact

1 Jesse Pullin Bar Pilot EMAIL REDACTED
2 Kevin Freese Bar Pilot EMAIL REDACTED
3 Drew Aune Bar Pilot EMAIL REDACTED
4 Dylan Epperson Bar Pilot EMAIL REDACTED
5 Tom Miller Bar Pilot EMAIL REDACTED
6 Erik Fawcett Bar Pilot EMAIL REDACTED
7 Robert Carr Bar Pilot EMAIL REDACTED
8 Paul Ruff Bar Pilot EMAIL REDACTED
9 Cevan LeSieur Bar Pilot EMAIL REDACTED
10 Dustin Slack Bar Pilot EMAIL REDACTED
11 Don Cloes Bar Pilot EMAIL REDACTED
12 Sam D’Alusio Bar Pilot EMAIL REDACTED
11
 Validation Report
Pilot Trainee

13 Brett Nelson Bar Pilot EMAIL REDACTED

14 Matt Stevens Bar Pilot EMAIL REDACTED

Simulator Examination Evaluators

Dates: June 6, 7, and 8, 2019

Name Classification Contact

1 Eric Robinson Bar Pilot EMAIL REDACTED

2 John Carlier Bar Pilot EMAIL REDACTED

3 Paul Amos Pilot (Portland) EMAIL REDACTED
4 Anne Macintyre Pilot (Portland) EMAIL REDACTED

5 Chris Carson Ship Master EMAIL REDACTED

6 Tony Mocuin Ship Master EMAIL REDACTED

Simulator Examination Angoff Session

Date: June 8, 2019

Name Classification Contact

1 Steve Teague Bar Pilot EMAIL REDACTED
2 John Carlier Bar Pilot EMAIL REDACTED
3 Eric Robinson Bar Pilot EMAIL REDACTED
4 Dan Boriolo Bar Pilot EMAIL REDACTED
5 Mark Haggerty Bar Pilot EMAIL REDACTED
6 Paul Amos Pilot (Portland) EMAIL REDACTED
7 Anne McIntyre Pilot (Portland) EMAIL REDACTED
8 Chris Carson Ship Master EMAIL REDACTED
9 Tony Mocuin Ship Master EMAIL REDACTED
10 Robert Carr Pilot EMAIL REDACTED
11 Paul Ruff Pilot EMAIL REDACTED
12 Victor Schisler Test Programmer EMAIL REDACTED
13 J.D. Gates Test Proctor EMAIL REDACTED
14 Allen Garfinkle Ship Master EMAIL REDACTED

12
 Validation Report
Pilot Trainee
APPENDIX B Written Examination Content Specifications

Subject Matter Areas in Written Examination

SMEs worked as a group to evaluate and integrate task and KSAPC statements into
four subject matter areas. Weights (percentages) to each area were based on the task
ratings in the job analysis. Below is a table showing the four subject matter areas, area
definitions, and weights.

Subject Matter Area Definition Weight

Prepare for safe transit of vessel in a timely
Pre-Transit Planning 12%
manner; determine feasibility of vessel movement
Exchange of information between master and pilot
Master-Pilot
pertaining to transit; acclimate to vessel by 6%
Transition
analyzing equipment and vessel position
Take the conn; implement plans for safe transit of
Route Piloting 44%
vessel from start to destination
Docking, Undocking, Initiate or terminate transit; approach or depart
38%
& Anchorage dock

Written Examination Specifications

SSP utilized linkage data of job tasks and knowledge, skills, and abilities and personal
characteristics (KSAPC) established by SMEs to organize KSAPC statements into
Subject Matter Areas. The following tables represent the content specifications in the
written examination. The specifications are organized in terms of Subject Matter Area
and the KSAPCs that candidates must possess to perform job tasks. Each KSAPC,
although linked to multiple job tasks, was listed only once to avoid redundancy.

13
 Validation Report
Pilot Trainee
CONTENT AREA SUBAREA ASSOCIATED KSAPC
I. Pre-Transit Planning A. Environment 12 Knowledge of the effects of salinity on vessels’ draft to allow for safe underkeel
(12%) – Prepare for safe clearance.
transit of vessel in a 13 Knowledge of available resources (e.g., Army Corps charts, Local Notice to
timely manner; Mariners) to obtain latest channel depths and characteristics to ensure safe passage
determine feasibility of of vessels.
vessel movement 14 Knowledge of available resources to determine predicted tides and currents to
ensure timely transit.
15 Knowledge of the effect of environmental factors on tide and current predictions to
efficiently carry out transit operations.
17 Knowledge of the effect of various environmental conditions on vessels’ ability to
ensure timely transit.
B. Geographic Conditions 19 Knowledge of bridge clearances and configurations to avoid allision.
20 Ability to identify and interpret information from charts to ensure safe transit.
21 Knowledge of various types of port configurations and channels (e.g., turning basins,
overhead structures) to ensure safe passage.
C. Vessel Capabilities 24 Knowledge of the effect of vessel speed on squat, heel, and sinkage to determine
safe and efficient transit.
25 Knowledge of the effect of vessel’s draft and trim on its handling to safely navigate
through various water conditions.
26 Knowledge of tugboat characteristics and bollard pull to determine tug effectiveness.
27 Knowledge of the effect of environmental conditions on handling techniques of
different classes of vessels to ensure safe transit.
28 Knowledge of the effect of interactions between vessels in close quarters to prevent
damage.
29 Knowledge of various types of vessel maneuvering characteristics for effective
shiphandling.
D. Transit Planning 32 Ability to identify potential conflicts (e.g., vessel traffic, debris, current change, tide
levels) on transit routes to ensure safe passage.
34 Ability to coordinate meeting location, availability, et cetera with assigned tugboats
for effective use.
35 Knowledge of underkeel clearance, and its effect on the vessel in various locations
along route, to ensure safe passage.

14
 Validation Report
Pilot Trainee

CONTENT AREA SUBAREA ASSOCIATED KSAPC

II. Master-Pilot Transition A. Leadership/Supervision Knowledge of crew responsibilities on various vessels to carry out timely transit
9
(6%) – Exchange of and Management operations.
information between Ability to coordinate and cooperate with Bridge Team to ensure effective transit
8
master and pilot operations.
pertaining to transit; Ability to coordinate passing arrangements with other vessels with regards to your own
acclimate to vessel by 10
vessel limitations.
analyzing equipment B. (Bridge and Vessel) Knowledge of factors (e.g., environment, traffic, geographic conditions) that affect
and vessel position. 36
Equipment reliability of estimated times of arrival when meeting other vessels.
Knowledge of various types of equipment (e.g., rudder, bitt [strength], thruster)
37
necessary to ensure safe transit operations.
Knowledge of use and limitations of various shipboard navigational equipment and
38
radar systems for vessel maneuvering and collision avoidance.
C. Communication Ability to communicate in standard nautical terminology with shore-based support
39 systems, bridge-to-bridge, and working channels to comply with protocols and
procedures.
40 Ability to communicate with Vessel Traffic Service to ensure safe navigation.
Ability to read written documents of varying complexity including departmental policy,
43
manuals, and guides to comprehend and communicate information to others.
D. Vessel Characteristics Knowledge of different tugboat characteristics to determine their suitability for specific
45
tasks.
Ability to interpret and interpolate draft mark readings to adjust shiphandling techniques
47
accordingly.
Ability to determine information about vessel maneuvering limitations, special
49
requirements, and unusual characteristics to ensure effective shiphandling.

15
 Validation Report
Pilot Trainee

CONTENT AREA SUBAREA ASSOCIATED KSAPC

III. Route Piloting A. Technical Knowledge of applicable guidelines or standards (e.g., Regulated Navigation Areas,
(44%) – Take the conn, 1 speed limits, Rules of the Road, escort regulations) as they relate to maritime
implement plans for safe operations safety.
transit of vessel from Knowledge of shiphandling principles and methods to carry out safe and efficient
2
start to destination transit operations.
Knowledge of vessel movement operational guidelines to ensure compliance and
3
safe piloting.
Knowledge of the basic principles of physics (e.g., hydrodynamics) to assess the
4
effect of external conditions on vessels and the vessels’ effect on surroundings.
Ability to operate specialized global positioning and tracking technology (e.g.,
5
Electronic Chart Display, Automatic Radar Plotting Aid) to ensure safe transit.
Ability to read and interpret charts/soundings in order to determine conditions and
6
navigate waterways safely.
Ability to perform mathematical calculations (e.g., basic algebra) to effectively
7
navigate vessels safely in various geographic and environmental conditions.
B. Shiphandling Ability to adjust maneuvering techniques based on vessel’s type and size to ensure
50
efficient shiphandling.
Knowledge of hydrodynamic forces in narrow channels and shallow water to
52
appropriately maneuver vessel in challenging conditions.
Knowledge of vessel’s wake effect to reduce negative effects on surrounding
53
environments.
Knowledge of the implications caused by vessels’ speed relative to engine speed for
56
effective shiphandling.
Ability to properly maneuver the vessel under changing channel configurations to
54
ensure safe passage.
Ability to safely handle the vessel depending on changing vessel factors (e.g.,
55
underkeel clearance, heel) for effective maneuvering.
57 Knowledge of factors that affect pivot point to adjust shiphandling.
C. Traffic Management Knowledge of factors (e.g., weather, traffic, geographic conditions) constraining
59
other vessels’ transit to plan transit operations accordingly.
Knowledge of water depths and configurations in bays, channels, rivers, harbors,
61 and anchorages, and their effects on navigation to manage transit planning
accordingly.
Ability to properly apply Rules of the Road and/or Vessel Traffic Service regulations
62
to ensure safe transit.
D. Situational Awareness Ability to strategize and prioritize information from multiple sources to carry out
75
transit operations in a timely manner.
76 Ability to perform multiple tasks simultaneously when responding to unfolding events

16
 Validation Report
Pilot Trainee

to ensure a safe transit.

Ability to adapt to sudden changing conditions (e.g., environmental, mechanical) in
77
order to ensure vessel safety.
Ability to observe actions and/or response of others to ensure compliance with
78
orders.
Ability to follow up with others (e.g., personnel, vessels) to ensure predetermined
79
arrangements are carried out.
Ability to choose and take the appropriate course of action to mitigate an unforeseen
80
event or hazard.
Ability to apply past experiences to present or future circumstances to ensure
81
mistakes are not repeated.
Ability to assess crew’s effectiveness in responding to pilot’s orders to ensure safe
82
vessel transit.

CONTENT AREA SUBAREA ASSOCIATED KSAPC

IV. Docking, Undocking, & A. Anchorage 63 Knowledge of factors that affect anchoring of a vessel to ensure its safety.
Anchorage 65 Ability to assess potential hazards at local anchorages to ensure safety of vessel.
(38%) – Initiate or Ability to carry out anchoring procedures while vessel is moving, mooring, or in
terminate transit; 66
emergency situations to ensure vessel safety.
approach or depart Knowledge of various navigation techniques to precisely anchor a vessel under
dock. 67
various conditions.
B. Docking and Knowledge of acceptable protocol and etiquette for radio communication to ensure
68
Undocking proper docking and undocking of vessel.
Knowledge of the effect of existing traffic conditions on departure/arrival times to
69
ensure safe transit.
Knowledge of capabilities and limitations of different types of tugboats to safely dock
70
and undock the vessel.
Knowledge of various methods to determine distances, speed, and position of own
73
vessel and of vessels in close proximity.
Knowledge of environmental conditions (e.g., current, wind) for approach to berth in
74
order to safely dock and undock vessel.

17
 Validation Report
Pilot Trainee
APPENDIX C: Simulation Examination Content Specifications

Subject Matter Areas in Simulation Examination

SMEs worked as a group to determine which tasks were to be measured in simulation

examination. Definitions for the seven subject matter areas in the simulation
examination are as follows:

1) Situation awareness: Assesses the candidate’s appreciation of the situation the

vessel is in at the all times during the exercise including relative motion, traffic, aids
to navigation, effect of wind, current and other forces on the vessel, and the
candidate’s ability to accurately filter and prioritize available information.

2) Appropriate response: Assesses the candidate’s responses in routine transit,

including timeliness and appropriateness of rudder commands, engine orders and
other orders to the bridge team.

3) Ability to respond correctly under pressure: Separately assess the candidate’s

ability to response under emergency or non-routine situations of increased stress.

4) Communication and bridge presence: Assesses the candidate’s ability to use

concise, clear, and pertinent communications using proper terminology on radios,
with the bridge, and evaluation teams. In addition, assesses the candidate’s
professional composure, demeanor, and ability to communicate warranted self-
confidence, which inspires confidence in the bridge team that the vessel’s navigation
is in good hands.

5) Fundamental ship handling: Assesses the candidate’s knowledge of proper and

timely use of engine and rudder commands and understanding the ship’s response
to those commands.

6) Bridge resource management: Assesses the candidate’s proper use of all

personnel and equipment resources available during the exercise.

7) Rules of the road: assesses the candidate’s proper application of the navigational
rules of the road to the situations presented. The Inland Rules will apply throughout
this exercise.

18
 Validation Report
Pilot Trainee

Simulation Examination Specifications

Tasks selected for the simulation examination were derived from the job analysis which
SMEs identified as critical tasks for successful job performance. The Simulator
Examination Specification matrices on the following pages represent how SMEs
categorized each job task per Subject Matter Area to allow for measurement
opportunities in the Simulator Examination.

Simulator Measurement Opportunities

The table below shows the weight of each Subject Matter Areas and corresponding
measurement opportunities in the Simulator Examination. Weighting was computed by
taking the total number of tasks within a Subject Matter Area and divide by the sum of
all tasks within the seven Subject Matter Areas.

Measurement
Subject Matter Area Tasks Weight
Opportunities
Situation awareness 19 18% 11
Appropriate response 21 20% 12
Ability to respond correctly
22 21% 12
under pressure
Communication and bridge
10 9% 6
presence
Fundamental ship handling: 22 21% 12
Bridge resource management 7 7% 4
Rules of the road 5 5% 3
TOTAL 106 100% 60

19
 Validation Report
Pilot Trainee

I. Pre-Transit Planning

Appropriate Responses

Respond Under Stress

Situation Awareness

Communication and

Fundamental Ship

Rules of the Road

Bridge Resource
Bridge Presence
Tasks

Management
Handling
1 Determine factors (e.g., weather, traffic, underkeel x x x
clearances, tugboat availability) affecting the desired
route or alternative route to plan safe transit routes
accordingly.
4 Calculate underkeel clearances by reviewing channel x x
sounding charts and local area data for vessel safety.
5 Calculate overhead clearances based on height of x x x x
tide at locations along route to ensure safe and
uneventful transit.
10 Identify constraints on vessel maneuverability based x x
on vessel factors (e.g., type, draft, air draft, vessel
particulars) to plan efficient transit routes.
14 Develop a personal, standardized conference practice x x
with Master to effectively execute Master-Pilot
exchange.

II. Master-Pilot Transition

Appropriate Responses

Respond Under Stress

Situation Awareness

Communication and

Fundamental Ship

Rules of the Road

Bridge Resource
Bridge Presence

Tasks

Management
Handling

15 Exchange information (e.g., pilot card, tug escort form) x x x x

with Master to determine vessels’ handling
characteristics and readiness for emergency
situations.
16 Communicate with Master regarding existing x x x x
underkeel clearance, traffic, and environmental
conditions to ensure departure/arrival times are
feasible.
19 Establish radio communication with Vessel Traffic x x x x
Service to report vessel location and destination, and
to receive information about other vessel movements.

20
 Validation Report
Pilot Trainee

III. Route Piloting

Appropriate Responses

Respond Under Stress

Situation Awareness

Communication and

Fundamental Ship

Rules of the Road

Bridge Resource
Bridge Presence
Tasks

Management
Handling
2 Evaluate environmental conditions to determine, and x x x x
adjust accordingly, appropriate vessel operations
(e.g., transit, maneuvering) in compliance with Vessel
Traffic Service safety regulations.

3 Review timely navigational information to evaluate x x x

risks posed by unusual factors and/or special
circumstances that may affect vessel movement.
6 Adjust to bridge environment and layout in order to x x x x
carry out safe transit operations.
7 Monitor environmental and traffic conditions to x x x x
appropriately update estimated arrival times.
12 Plan transit routes by evaluating traffic conditions to x x x x
ensure vessel safety.
13 Determine current at locations (e.g., along planned x x x
route, at berth) to develop plans for maneuvering
vessel.
17 Maintain communication with Bridge Resource Team x x x
to ensure free flow of information.
18 Communicate with Marine Exchange when checking x
in with escorted vessels to comply with California
State regulations.
20 Communicate with other vessels in a timely manner to x x x x
make passing arrangements and ensure the safety of
vessels.
21 Communicate time of arrival with Lift Bridge Operator x x x x
to arrange safe and timely transits.
25 Monitor Vessel Traffic Service and radio x x x
communications to determine traffic conditions and
ensure vessel safety.
32 Determine the relative motion of other craft on the x x x x
water to assess the risk of collision.
33 Utilize navigational equipment (e.g., radar, Electronic x x x x
Chart Display) to determine information regarding
vessels’ transit (e.g., position, course, speed).
34 Assess potential risk caused by vessel transit to x x x x x
minimize harm to property, persons, and the
environment.

21
 Validation Report
Pilot Trainee

35 Determine vessels’ clearance from objects in close x x

proximity through all available means to avoid
collisions/allisions.
40 Prioritize actions during transit to ensure essential x x x
operations are carried out in a timely manner.
41 Perform multiple tasks simultaneously when x x x
responding to unfolding events to ensure a safe
transit.
43 Observe actions and/or response of others to ensure x x x x
compliance with orders.
44 Follow up with others (e.g., personnel, vessels) to x x x
ensure predetermined arrangements are carried out.
46 Maintain awareness of current situation when x x x x
interrupted by events to ensure vessel safety and
necessary actions are completed.

IV. Docking, Undocking, & Anchorage

Appropriate Responses

Respond Under Stress

Situation Awareness

Communication and

Fundamental Ship

Rules of the Road

Bridge Resource
Bridge Presence
Tasks

Management
Handling
8 Assess immediate navigation and traffic conditions to x x x x
identify potential hazards to vessel.
9 Assess dock and berth characteristics to avoid x x
potential obstructions.
11 Modify transit plan due to vessel responsiveness, x x x x
traffic/berth conflicts, or environmental conditions for
efficient transit operations.
22 Coordinate with Master and Bridge Resource Team to x x x x
ensure timely availability of crew for anchoring,
mooring or other evolutions.
23 Establish working radio communication frequencies to x x x
facilitate communication with other vessels and/or
tugboats.
24 Communicate with Master regarding line handling and x x x x
docking plan to ensure appropriate actions are being
taken.
26 Determine the vessel’s responsiveness to changes in x x
course/speed and environmental conditions to ensure
safe and effective maneuvering.

22
 Validation Report
Pilot Trainee

27 Observe vessel’s response to helm and engine orders x x

to ensure safe and timely maneuvering.
28 Take corrective actions (e.g., rerouting, speed x x x x
adjustment) to ensure vessel arrives at predetermined
points in an appropriate manner.
29 Position vessel for approach to berth to ensure vessel x x x x
docks safely.
30 Maneuver vessel while ensuring safety of tugboats, x x x x
vessel, and terminal facilities during transit and
undocking/docking operations.
31 Control vessel at an appropriate speed when in transit x x x x
and/or approach to berth or anchorage to ensure
vessel safety.
36 Position vessel alongside berth to ensure safe cargo x x x x
operations.
37 Monitor line handling that may affect ship personnel, x x x
equipment, or facility to ensure effective mooring and
unmooring.
38 Anchor vessel in appropriate locations to ensure safety x x x x x
and compliance with vessel traffic regulations.
39 Ensure final mooring arrangement is sufficient for x x x
prevailing conditions to ensure vessel is safely
moored.
42 Adapt to sudden changing conditions (e.g., x x x x
environmental, mechanical) in order to ensure vessel
safety.
45 Choose and take the appropriate course of action to x x x x
mitigate an unforeseen event or hazard.

23
 Validation Report
Pilot Trainee

APPENDIX D: Examination Item/KSAPC Linkage

Written Examination Item Linkage

Exam KSAPC #
Item #

1 1, 6, 7, 8, 9, 13, 14, 35, 43, 47, 79, 82

2 4, 6, 12, 17, 24, 25, 52, 55
3 2, 5, 6, 19, 32, 38, 65, 67, 77, 80, 81
4 2, 4, 29, 49, 56, 57, 63, 66, 75, 76
5 2, 4, 26, 28, 37, 45, 68, 70, 77, 78, 79
6 2, 3, 4, 26, 70, 73, 74
7 2, 27, 29, 49,50, 56
8 2, 4, 7, 24, 25, 27, 29, 35, 49, 50, 52, 54, 55, 56, 57, 67
2, 4, 5, 15, 17, 19, 25, 27, 29, 36, 38, 47, 49, 50, 52, 54, 55, 59, 73, 74, 77, 80,
9
81, 82,
10 2, 5, 6, 19, 32, 38, 65, 67, 77, 80, 81
11 2, 4, 10, 24, 25, 27, 29, 35, 52, 54, 55, 56, 61
12 2, 4, 8, 29, 37, 49, 50, 54, 56, 57, 75, 80, 81, 82
13 6, 13, 20, 21, 32, 61
14 4, 24, 25, 29, 55, 61, 77
15 2, 4, 29, 37, 49, 50, 55, 56, 57, 63, 80, 81
16 6, 13, 20, 21, 32, 35, 63, 65, 67
17 6, 13, 14, 20
18 2, 4, 12, 13, 15, 17, 24, 25, 35
19 2, 4, 29, 49, 73, 74, 77
20 2, 4, 25, 27, 29, 49, 50, 57, 74
2, 4, 6, 7, 8, 9, 13, 15, 20, 21, 27, 37, 49, 52, 56, 57, 61, 63, 66, 75, 76, 77, 78,
21
79, 82
22 2, 4, 10, 24, 25, 27, 29, 35, 52, 54, 55, 56, 61
23 4, 12, 24, 25, 52, 56, 77
24 4, 52, 54, 55, 76, 77, 80
25 2, 4, 10, 24, 25, 27 ,29, 35, 52, 54, 55, 56, 61
26 2, 4, 26, 28, 37, 45, 68, 70, 77, 78, 79
27 2, 4, 8, 15, 25, 27, 29, 37, 47, 49, 50, 52, 54, 55, 57, 74, 76, 78, 82
28 1, 6, 8, 9, 13, 14, 15, 32, 37, 38, 43, 59, 61, 73, 78, 79, 82
29 1, 6, 7, 20, 47
30 2, 4, 7, 24, 25, 27, 36, 49, 50, 52, 55, 56, 57, 61
31 2, 4, 5,15, 27, 38, 52, 75, 76, 81,49,50,55,59,61
2, 4, 6, 7, 15, 20, 21, 24, 25, 27, 28, 29, 32, 35, 36, 49, 50, 52, 53, 54, 55, 56,
32
59, 61, 76, 78, 80, 81, 82
33 2, 4, 29, 49, 56, 57, 63, 66, 75, 76
34 2, 4, 6, 8, 9, 13, 14, 21, 37, 49, 52, 57, 61, 63, 66, 76, 77, 78, 79, 80, 82
35 2, 4, 7, 15, 27, 32, 36, 59, 74

24
 Validation Report
Pilot Trainee

Exam KSAPC #
Item #

36 38,75,49
37 4, 12, 24, 25, 52, 56, 77
2, 4, 5, 7, 21, 26, 28, 29, 32, 34, 38, 47, 52, 53, 56, 57, 63, 67, 74, 75, 76, 77,
38
78, 79, 81, 82
39 2, 4, 26, 29, 37, 45, 49, 50, 70
40 2, 5, 6, 7, 63, 67
41 4, 25, 45, 55, 57
42 2, 4, 29, 37, 49, 56
43 6, 15, 20, 35, 61, 65
44 2, 4, 7, 17, 26, 37, 38, 47, 56, 57, 74, 76, 77, 79, 80, 81
45 15, 20, 27, 50, 54, 55, 61, 75, 77, 81
46 2, 5, 6, 7, 63, 67
47 2, 4, 25, 28, 32, 35, 52, 54, 53, 56, 80
48 2, 4, 6, 7, 15, 20, 21, 24, 25, 28, 29, 32, 35, 52, 53, 56, 76, 78, 80, 81, 82
49 2, 4, 7, 24, 25, 27, 36, 49, 50, 52, 55, 56, 57, 61
50 2, 4, 26, 37, 45, 49, 50, 57, 70
51 2, 4, 24, 25, 27, 28, 29, 31
52 2, 4, 29, 52, 54, 55, 56, 57, 66, 80, 81
2, 4, 6, 9, 15, 20, 27, 29, 32, 35, 49, 50, 52, 54, 55, 56, 57, 75, 76, 77, 78, 79,
53
80, 81, 82
54 6, 7, 13, 19, 20, 31, 32
55 2, 4, 25, 27, 29, 49, 50, 57, 74
56 2, 4, 8, 9, 25, 29, 31, 49, 66, 78, 79, 80
57 2, 4, 26, 28, 37, 45, 68, 70, 77, 78, 79
58 2, 4, 6, 8, 9, 13, 14, 21, 37, 49, 52, 57, 61, 63, 66, 76, 77, 78, 79, 80, 82
59 2, 4, 37
60 2, 37
61 2, 4, 17, 27, 56, 57
62 4, 37
63 2, 27, 29, 49, 50, 56
64 2, 4, 14, 15, 26, 27, 29, 37, 45, 49, 50, 57, 70, 74, 78
2, 4, 6, 7, 8, 9, 13, 15, 20, 21, 27, 37, 49, 52, 56, 57, 61, 63, 66, 75, 76, 77, 78,
65
79, 82
66 2, 4, 24, 26, 29, 37, 45, 49, 50, 57, 70, 75, 78
67 2, 4, 10, 24, 25, 27, 29, 35, 52, 54, 55, 56, 61
68 2, 4, 7, 24, 25, 29, 35, 52, 56, 57, 67, 27, 49, 50, 54, 55, 61
69 4, 37
70 2, 4, 26, 28, 37, 45, 68, 70, 77, 78, 79
71 6, 7, 47, 63, 66, 67
72 2, 4, 7, 24, 25, 27, 29, 35, 49, 50, 52, 54, 55, 56, 57, 61, 67
73 2, 4, 29, 37, 50, 52, 55
74 2, 3, 4, 24, 27, 55

25
 Validation Report
Pilot Trainee

Exam KSAPC #
Item #

75 2, 4, 37
76 2, 4, 24, 26, 28, 45, 56
2, 4, 6, 13, 15, 27, 29, 35, 49, 50, 54, 55, 56, 57, 61, 63, 65, 66, 67, 75, 76, 77,
77
78, 79, 81 82
78 7, 49, 73, 74
2, 4, 6, 7, 8, 9, 13, 15, 20, 21, 27, 37, 49, 52, 56, 57, 61, 63, 66, 75, 76, 77, 78,
79
79, 82
80 2, 4, 7, 15, 27, 32, 36, 59, 74
81 2, 4, 7, 37, 56, 57, 74, 79
82 2, 4, 29, 49, 56, 57, 63, 66, 75, 76
83 2, 4, 24, 26, 29, 37, 45, 49, 50, 57, 70, 75, 78
84 2, 4, 29, 49, 56, 57, 63, 66, 75, 76
2, 4, 6, 9, 20, 15, 27, 29, 32, 35, 49, 50, 52, 54, 55, 56, 57, 61, 75, 76, 77, 78,
85
79, 80, 81, 82
86 2, 4, 21, 50, 52, 56, 57, 61, 73, 74, 80, 81
87 2, 4, 25, 29, 47, 56
88 2, 4, 24, 26, 28, 45, 56
89 2, 4, 10, 24, 25, 27, 29, 35, 52, 54, 55, 56, 61
90 7, 47
91 2, 4, 29, 37, 57, 49, 50
92 2, 4, 8, 9, 15, 21, 27, 29, 35, 49, 50, 55, 56, 57, 61, 66, 74, 78, 82
2, 4, 6, 7, 8, 9, 13, 15, 20, 21, 27, 37, 49, 52, 56, 57, 61, 63, 66, 75, 76, 77, 78,
93
79, 82
94 2, 4, 25, 27, 29, 49, 50, 57, 74
95 2, 4, 8, 9, 15, 27, 37, 39, 52, 61, 68, 74, 75, 78, 79
96 2, 4, 10, 24, 25, 27, 29, 35, 52, 54, 55, 56, 61
97 2, 4, 7, 24, 52, 50
98 2, 4, 7, 24, 25, 28, 29, 36, 49, 50, 52, 55, 56, 80, 81
99 2, 27, 29, 49,50, 56
100 7, 27 ,59, 74, 77
101 2, 4, 29, 27, 49, 50, 55
102 2, 4, 24, 25, 27, 28, 29, 31
103 2, 4, 24, 25, 27, 28, 29, 31
104 2, 4, 25, 27, 29, 49, 50, 57, 74
105 4, 15, 21, 27, 49, 54, 74, 77
2, 4, 10, 24, 25, 27, 28, 29, 34, 35, 36, 37, 49, 50, 52, 54, 55, 56, 61, 76, 77,
106
78, 79, 80, 81, 82
107 2, 4, 6, 17, 25, 27
2, 4, 5, 15, 17, 19, 25, 27, 29, 36, 38, 47, 49, 50, 52, 54, 55, 59, 73, 74, 77, 80,
108
81, 82
109 5, 38, 73
110 2, 4, 6, 7, 12, 14, 17, 20, 21, 24, 25, 28, 29, 32, 35, 52, 53, 56, 80, 81

26
 Validation Report
Pilot Trainee

Exam KSAPC #
Item #

111 4, 15, 21, 27, 49, 54, 74, 77

112 5, 38, 73
2, 4, 5, 15, 17, 19, 25, 27, 29, 36, 38, 47, 49, 50, 52, 54, 55, 59, 73, 74, 77, 80,
113
81, 82
2, 4, 10, 24, 25, 27, 28, 29, 34, 35, 37, 36, 49, 50, 52, 54, 55, 56, 61, 76, 77,
114
78, 79, 80, 81, 82
115 2, 37, 38
2, 4, 10, 24, 25, 27, 28, 29, 34, 35, 36, 37, 49, 50, 52, 54, 55, 56, 61, 76, 77,
116
78, 79, 80, 81, 82
117 2, 4, 7, 37, 56, 57, 74, 79
118 2, 4, 25, 27, 29, 49, 50, 57, 74
119 2, 4, 8, 9, 13, 14, 20, 21, 25, 35, 37, 57, 61, 63, 66, 75, 78
120 2, 4, 24, 26, 29, 37, 45, 49, 50, 57, 70, 75, 78
121 7,49,73,74
122 2, 4, 24, 26, 29, 37, 45, 49, 50, 57, 70, 75, 78
123 1, 2, 3, 6, 9, 75
124 2, 4, 24, 25, 29, 35, 47, 49, 50, 52, 55, 56, 57, 67, 81
125 2, 4, 5, 15, 27, 38, 49, 50, 52, 55, 59, 61, 75, 76, 81
126 2, 4, 10, 24, 25, 27, 29, 35, 52, 54, 55, 56, 61
127 2, 4, 14, 24, 25, 29, 35, 37, 52, 56, 57, 76, 78, 79, 81, 82
128 2, 4, 7, 24, 25, 27, 29, 35, 49, 50, 52, 54, 55, 56, 57, 61, 67
129 2, 4, 7, 14, 15, 17
130 2, 4, 8, 9, 15, 21, 27, 29, 35, 49, 50, 55, 56, 57, 61, 66, 74, 78, 82
131 2, 4, 12, 13, 15, 17, 24, 25, 35
132 2, 4, 29, 37, 57
133 4, 37
134 2, 4, 8, 15, 25, 27, 29, 37, 47, 49, 50, 52, 54, 55, 57, 74, 76, 78, 82
135 2, 4, 7, 24, 25, 27, 29, 35, 49, 50, 52, 54, 55, 56, 57, 61, 67
136 2, 4, 7, 10, 14, 15, 17, 20, 21, 24, 27, 28, 32, 52, 54, 56, 61, 74, 77, 81
137 4, 7, 15, 27, 36, 50, 57, 59, 63, 67, 74, 77
2, 4, 6, 7, 8, 9, 13, 15, 20, 21, 27, 37, 49, 52, 56, 57, 61, 63, 66, 75, 76, 77, 78,
138
79, 82
139 2, 4, 25, 27, 29, 49, 50, 57, 74
2, 4, 6, 7, 8, 9, 13, 15, 20, 21, 27, 37, 49, 52, 56, 57, 61, 63, 66, 75, 76, 77, 78,
140
79, 82
141 2, 4, 29, 37, 49, 52, 56
142 15, 17, 27, 38, 59, 77
143 1, 2, 3, 6, 9, 75
144 2, 4, 7, 24, 25, 27, 29, 35, 49, 50, 52, 54, 55, 56, 57, 61, 67
145 2, 8, 37, 43, 49, 50, 75
2, 4, 5, 7, 9, 15, 21, 26, 27, 28, 29, 32, 34, 36, 38, 47, 49, 50, 52, 53, 56, 57,
146
59, 61, 63, 65, 66, 67, 74, 75, 76, 77, 78, 79, 81, 82

27
 Validation Report
Pilot Trainee

Exam KSAPC #
Item #

2, 3, 4, 7, 6, 12, 13, 14, 15, 17, 19, 20, 24, 25, 27, 32, 35, 36, 47, 55, 61, 74,
147
77, 80, 81
148 2, 4, 27, 35, 52, 55, 56, 61
1, 2, 3, 5, 6, 7, 8, 9, 10, 12, 13, 14, 17, 19, 20, 21, 24, 25, 26, 27, 29, 32, 34,
149 35, 36, 37, 38, 39, 40, 43, 45, 47, 49, 50, 52, 53, 54, 55, 56, 57, 59, 61, 62, 63,
65, 67, 68, 69, 70, 73, 74, 75,76, 77, 78,79, 80, 81, 82, 83, 84,85, 86, 87
150 6, 7, 47, 63, 66, 67

28
 Validation Report
Pilot Trainee

Simulator Examination Item Linkage

Exam Task #
Item #

1 2, 10, 11, 12, 13, 15, 22, 40, 41, 42, 43, 44, 45, 46
2 12, 15, 33, 34, 46
3 2, 3, 6, 7, 12, 13, 26, 27, 28, 30, 31, 32, 33, 34, 35, 41, 46
4 1, 2, 7, 8, 11, 12, 20, 23
5 2, 3, 6, 7, 12, 13, 26, 27, 28, 30, 31, 32, 33, 34, 35, 41, 46
6 2, 3, 6, 7, 12, 13, 26, 27, 28, 30, 31, 32, 33, 34, 35, 41, 46
7 2, 6, 12, 17, 20, 32, 34, 40, 41, 43, 46
8 1, 2, 3, 4, 5, 6, 8, 10, 11, 12, 17, 33, 34, 35, 40, 41, 42
9 2, 6, 12, 17, 20, 32, 34, 40, 41, 43, 46
10 26, 27, 28, 29
11 1, 2, 7, 8, 11, 12, 20, 23
12 1, 2, 7, 8, 11, 12, 20, 23
13 26, 27, 28, 29
14 2, 3, 6, 7, 12, 13, 26, 27, 28, 30, 31, 32, 33, 34, 35, 41, 46
15 1, 2, 7, 8, 11, 12, 20, 23
16 2, 3, 6, 7, 12, 13, 26, 27, 28, 30, 31, 32, 33, 34, 35, 41, 46
17 2, 3, 6, 7, 12, 13, 26, 27, 28, 30, 31, 32, 33, 34, 35, 41, 46
18 2, 6, 12, 17, 20, 32, 34, 40, 41, 43, 46
19 2, 3, 6, 7, 12, 13, 26, 27, 28, 30, 31, 32, 33, 34, 35, 41, 46
20 2, 6, 12, 17, 20, 32, 34, 40, 41, 43, 46
21 2, 3, 6, 7, 12, 13, 26, 27, 28, 30, 31, 32, 33, 34, 35, 41, 46
22 2, 6, 12, 17, 20, 32, 34, 40, 41, 43, 46
23 1, 2, 7, 8, 11, 12, 20, 23
24 1, 2, 7, 8, 11, 12, 20, 23
25 1, 2, 3, 5, 8, 10, 11, 12, 17
26 26, 27, 28, 29
27 2, 3, 6, 7, 12, 13, 26, 27, 28, 30, 31, 32, 33, 34, 35, 41, 46
28 1, 2, 6, 8, 10, 17, 33, 41, 42, 45
29 2, 6, 12, 17, 20, 32, 34, 40, 41, 43, 46
30 1, 2, 7, 8, 11, 12, 20, 23
31 2, 3, 6, 7, 12, 13, 26, 27, 28, 30, 31, 32, 33, 34, 35, 41, 46
32 2, 6, 12, 17, 20, 32, 34, 40, 41, 43, 46
33 1, 2, 7, 8, 11, 12, 20, 23
34 1, 2, 7, 8, 11, 12, 20, 23
35 1, 2, 7, 8, 11, 12, 20, 23
36 2, 3, 6, 7, 12, 13, 26, 27, 28, 30, 31, 32, 33, 34, 35, 41, 46
37 2, 6, 12, 17, 20, 32, 34, 40, 41, 43, 46
38 2, 3, 6, 7, 12, 13, 26, 27, 28, 30, 31, 32, 33, 34, 35, 41, 46
39 1, 2, 7, 8, 11, 12, 20, 23

29
 Validation Report
Pilot Trainee

Exam Task #
Item #

40 2, 3, 6, 7, 12, 13, 26, 27, 28, 30, 31, 32, 33, 34, 35, 41, 46
41 1, 2, 3, 17, 33, 40, 41, 42, 45
42 1, 2, 3, 17, 33, 40, 41, 42, 45
43 1, 2, 11, 16, 17, 18, 19
44 1, 4, 6, 12, 33, 34, 35
45 1, 2, 7, 8, 11, 12, 20, 23
46 2, 3, 6, 7, 12, 13, 26, 27, 28, 30, 31, 32, 33, 34, 35, 41, 46
47 2, 6, 12, 17, 20, 32, 34, 40, 41, 43, 46
48 1, 2, 7, 8, 11, 12, 20, 23
49 2, 3, 6, 7, 12, 13, 26, 27, 28, 30, 31, 32, 33, 34, 35, 41, 46
50 1, 2, 7, 8, 11, 12, 20, 23
51 2, 3, 6, 7, 12, 13, 26, 27, 28, 30, 31, 32, 33, 34, 35, 41, 46
52 2, 3, 6, 7, 12, 13, 26, 27, 28, 30, 31, 32, 33, 34, 35, 41, 46
53 1, 2, 7, 8, 11, 12, 20, 23
54 2, 3, 6, 7, 12, 13, 26, 27, 28, 30, 31, 32, 33, 34, 35, 41, 46
55 2, 6, 12, 17, 20, 32, 34, 40, 41, 43, 46
56 2, 3, 6, 7, 12, 13, 26, 27, 28, 30, 31, 32, 33, 34, 35, 41, 46
57 2, 3, 6, 7, 12, 13, 26, 27, 28, 30, 31, 32, 33, 34, 35, 41, 46
58 2, 3, 6, 7, 12, 13, 26, 27, 28, 30, 31, 32, 33, 34, 35, 41, 46
59 2, 6, 12, 17, 20, 32, 34, 40, 41, 43, 46
60 2, 6, 12, 17, 20, 32, 34, 40, 41, 43, 46

30
 Validation Report
Pilot Trainee

APPENDIX E: Pass Point Data

Written Examination

EXAMINATION MAC SCORE TOTAL EXAMINATION MAC SCORE TOTAL

ITEM # AVERAGE POSSIBLE ITEM # AVERAGE POSSIBLE
1 0.69 1 43 0.80 1
2 0.60 1 44 0.75 1
3 0.75 1 45 0.90 1
4 0.76 1 46 0.71 1
5 0.69 1 47 0.68 1
6 0.64 1 48 0.70 1
7 0.71 1 49 0.61 1
8 0.75 1 50 0.66 1
9 0.76 1 51 0.75 1
10 0.80 1 52 0.69 1
11 0.60 1 53 0.80 1
12 0.80 1 54 0.75 1
13 0.68 1 55 0.86 1
14 0.66 1 56 0.68 1
15 0.69 1 57 0.61 1
16 0.62 1 58 0.60 1
17 0.75 1 59 0.67 1
18 0.66 1 60 0.61 1
19 0.74 1 61 0.74 1
20 0.77 1 62 0.84 1
21 0.73 1 63 0.70 1
22 0.67 1 64 0.68 1
23 0.71 1 65 0.74 1
24 0.50 1 66 0.73 1
25 0.66 1 67 0.69 1
26 0.58 1 68 0.81 1
27 0.78 1 69 0.57 1
28 0.77 1 70 0.69 1
29 0.93 1 71 0.88 1
30 0.66 1 72 0.71 1
31 0.59 1 73 0.76 1
32 0.72 1 74 0.72 1
33 0.68 1 75 0.71 1
34 0.71 1 76 0.66 1
35 0.74 1 77 0.73 1
36 0.78 1 78 0.86 1
37 0.70 1 79 0.74 1
38 0.65 1 80 0.64 1
39 0.84 1 81 0.54 1
40 0.69 1 82 0.69 1
41 0.68 1 83 0.64 1
42 0.73 1 84 0.49 1

31
 Validation Report
Pilot Trainee

Written Examination
EXAMINATION MAC SCORE TOTAL EXAMINATION MAC SCORE TOTAL
ITEM # AVERAGE POSSIBLE ITEM # AVERAGE POSSIBLE
85 0.82 1 123 0.83 1
86 0.64 1 124 0.69 1
87 0.61 1 125 0.69 1
88 0.61 1 126 0.64 1
89 0.69 1 127 0.66 1
90 0.69 1 128 0.63 1
91 0.69 1 129 0.81 1
92 0.72 1 130 0.69 1
93 0.69 1 131 0.66 1
94 0.69 1 132 0.74 1
95 0.66 1 133 0.74 1
96 0.66 1 134 0.82 1
97 0.70 1 135 0.81 1
98 0.69 1 136 0.64 1
99 0.71 1 137 0.83 1
100 0.79 1 138 0.74 1
101 0.74 1 139 0.74 1
102 0.69 1 140 0.68 1
103 0.68 1 141 0.63 1
104 0.72 1 142 0.65 1
105 0.56 1 143 0.90 1
106 0.61 1 144 0.74 1
107 0.70 1 145 0.79 1
108 0.69 1 146 0.67 1
109 0.74 1 147 0.83 1
110 0.70 1 148 0.83 1
111 0.70 1 149 0.71 1
112 0.76 1 150 0.71 1
113 0.66 1
114 0.65 1
115 0.88 1
116 0.61 1
117 0.66 1
118 0.71 1
119 0.62 1
120 0.62 1 MAC Total 106.02
121 0.78 1 Total Poss. 150
122 0.65 1 Pass Score 106

32
 Validation Report
Pilot Trainee

Simulator Examination
EXAMINATION MAC SCORE TOTAL EXAMINATION MAC SCORE TOTAL
ITEM # AVERAGE POSSIBLE ITEM # AVERAGE POSSIBLE
1 1.23 2 31 0.50 2
2 0.88 2 32 0.90 2
3 1.38 2 33 0.46 2
4 1.46 2 34 0.76 2
5 0.99 2 35 1.10 2
6 1.11 2 36 0.83 2
7 1.42 2 37 1.20 2
8 1.00 2 38 1.05 2
9 1.06 2 39 1.28 2
10 0.86 2 40 0.76 2
11 1.60 2 41 1.61 2
12 1.31 2 42 1.44 2
13 0.89 2 43 0.12 2
14 1.29 2 44 1.35 2
15 1.30 2 45 0.88 2
16 0.68 2 46 1.77 2
17 0.04 2 47 1.05 2
18 0.78 2 48 0.36 2
19 1.26 2 49 0.78 2
20 1.78 2 50 0.95 2
21 -0.22 2 51 0.16 2
22 0.76 2 52 0.00 2
23 0.73 2 53 0.48 2
24 0.36 2 54 0.00 2
25 1.28 2 55 0.00 2
26 0.19 2 56 0.00 2
27 -0.66 2 57 1.00 2
28 1.69 2 58 1.09 2
29 1.11 2 59 1.00 2
30 1.04 2 60 1.12 2

MAC Total 52.57

Total Poss. 120
Pass Score 52

33