Program Evaluation: FAA Implementation of IACRA

(Integrated Airman Certification and Rating Application)

Joseph C. McGeorge

University of Illinois
 FAA Implementation of IACRA 2

A. Evaluation Rationale

Since the inception of IACRA (Integrated Airman Certification and Rating Application)

there have been many views expressed in the aviation community as to the effectiveness and

efficiency of the system. The primary application method for managing pilot applications and

certifications is critical to our society because of the scale and dynamic environment of the

aviation industry. Because this decision affected so many people, attitudes and satisfaction range

across the spectrum with respect to the program, an evaluation is necessary to either prove the

success of the implementation or point out flaws that can be improved to increase the efficiency

and acceptance of the system.

The FAA has commissioned this evaluation and is providing full support at all levels and

locales with respect to data collection, observations, and compliance with results. The FAA is

interested in the general industry acceptance, timely action, and reduced errors within the airman

application process. Feedback is required from the professional as well as general aviation

community to determine the success of the program. There should be no surprise that IACRA

was met with resistance as pilot certification has been a paper process since the early 20th

century. An evaluation is critical to determine if IACRA application processing is faster, less

costly (paid for by taxpayer dollars), reduces application errors, and meets the need of the

aviation community. Pilots, instructors, FAA administrators, examiners, those who retain

examining authority, politicians, and anyone else involved in the field of aviation will be

interested in the results of the evaluation.

B. Purpose
 FAA Implementation of IACRA 3

The purpose of the IACRA evaluation is to determine program acceptance, efficiency,

and improvement of the pilot certification process. The program changed the way an entire

segment of the economy handled paperwork, applications, and certificate renewal for airmen.

The FAA is a publicly funded body and with a program of this magnitude, an evaluation is

necessary. Efficiency issues in government bodies affect the entire society and this change was a

large shift in the way the FAA operates. Results will be used to find inadequacies, make

improvements, and determine general attitudes and acceptance from those who have a role/stake

in the industry.

Since the FAA is a public body providing certification services to all levels of pilots; this

evaluation will seek the satisfaction and acceptance of those pilots. Results will be organized

into experience levels based on certification to determine the most relevant problems with

respect to those results. By organizing the results in this manner, the evaluation will be able to

most effectively isolate and correct the most common problems. It is at the request of the

Administrator (FAA) that pilots of all levels be given consideration in this evaluation and that all

those concerned with airman certification have a voice in the conduct of the FAA with respect to

new certificate application and renewals.

C. Audience

Primary

1) Pilots – Primary applicants.

2) FAA officials/administrators – Primary enforcement body who ensures the legitimacy of

certificates issued.

3) FAA inspectors/examiners – Primary certificate issuers based on results of flight

examinations.
 FAA Implementation of IACRA 4

4) Flight school administrators – Primary recommending body.

5) Flight Instructors – Primary applicants who use additional features of the site by means of the

student recommendation function.

6) IACRA designers – Need access to common system problems to determine where changes

need to be made.

Secondary

1) Government bodies interested in digitization of records and services – May consider

application systems similar to IACRA.

2) Politicians – Responsible for the effective use of taxpayer dollars.

3) System maintenance personnel – Are those who ensure the proper operation of the online

system. They will need to be aware of pertinent problems and misunderstandings encountered

by users.

Tertiary

1) Taxpayers – Those who pay for the operation of the FAA.

D. Key Questions

1) To what extent has the implementation of IACRA reduced application errors in the airman

certification process?

2) What is the average delay between application submission and receipt of the certificate or

rating by the applicant? How does this compare to the paper application system?

3) What problems are most common to applicants that use the system?

4) Are computer-compatibility issues causing problems for applicants? If so, what are they?

5) To what extent has satisfaction and acceptance of the system improved or declined in the

aviation community concerning IACRA? FAA internal acceptance?

 FAA Implementation of IACRA 5

6) Are the desired outcomes coming to fruition with the implementation of IACRA? If not, what

barriers are causing the lack of success?

7) How does the efficiency and output of IACRA compare to that of its predecessor (the paper

system)?

8) In what ways can the online application be improved to ensure less confusion and mistakes

among applicants?

9) The FAA still accepts paper applications for pilot certification. Could this method be

eliminated and IACRA be the sole means of the airman application process? Are there enough

facilities, equipment and support in the country to facilitate this change?

10) Is there a monetary benefit/savings from the implementation of IACRA?

E. Evaluation Design

(see Appendix A for data collection instruments)

Since IACRA has already been implemented and general instructions and operations

have been given to the flying public, this evaluation will be primarily summative. The

application process is in full use and is currently the primary application source for airman

applications. Major problems have already been identified through informal evaluations, but the

FAA is concerned with the outcomes of the system implementation.

To determine the outcomes, this evaluation will utilize the Kirkpatrick model in order to

determine the success of the program. Since IACRA is truly a means to an end (submitting

applications) a level 1 combined with elements of a level 4 evaluation are appropriate. In

generalBecause the FAA uses taxpayer dollars to operate, many stakeholders throughout the

country will be interested in the results of the evaluation. Therefore, much of the data will need

to be collected and presented in a quantitative manner in order to satisfy those concerned.

 FAA Implementation of IACRA 6

Several methods of data collection will be employed to perform the evaluation. Since

applications for FAA certificates and ratings are conducted online, a survey will be used

following the submission of an application. By utilizing surveys in this manner, the evaluation

will have a high degree of demographic information (e.g. ages, locations, aeronautical

experience, ratings currently held) that will help to identify those types of people who are having

the most trouble with the system. The survey will be voluntary in nature. When pilots finish

their applications, they will most likely have frustrations and suggestions, which will create an

opportune moment to offer a survey. It is likely that response rates will be higher due to the

freshness of issues present in the participants’ minds. No sample will be required to collect these

surveys as the opportunity will be offered to anyone who completes an application.

Along with individual surveys, the evaluation will conduct focus group interviews around

the country to collect information from pilots in a collaborative environment. Gathering

participants will not be difficult as flight schools and flying clubs provide a close knit group of

people who are directly affected by the implementation of IACRA. Because the evaluation

requires the opinions of pilots across the country and throughout the aviation industry, a

stratified random sampling method is appropriate. There will be a total of eight focus group

sessions aimed at different training bodies within the aviation community. Focus groups will be

conducted at four flight schools, two of which are 14CFR Part 61 flight schools and two of

which are 14CFR Part 141 flight schools. One focus group will be held with administrators and

trainers at Flight Safety and another type-certificate training facility in Indianapolis which is

representative of airline pilot training. The final two focus group interviews will be conducted

with instructors who work in general aviation. The preceding interviews should cover the vast

majority of the aviation training community with interviewees from different methods of training
 FAA Implementation of IACRA 7

representing their groups. Surveys and focus group interviews will collect data from pilots but

information also needs to be collected from those within the FAA.

To address the concerns of those within the FAA individual interviews will be conducted

with those who are assigned to the program. Representatives at the help desk, programmers, and

administrators will need to be interviewed to see if the implementation of IACRA has resulted in

fewer problems and/or issues with pilot certification. The evaluation will use purposive

sampling in selecting the appropriate candidates for the interviews. With the data collected from

these methods, the evaluation will be able to answer or at least shed light on the key questions

asked in section D of the evaluation.

F. Evaluation Management Plan

Key
Questions Data Sources Personnel
Method Addressed Timeline Instrument Involved
Individual 1,5,6,7,8,9,1 May - June Administrative Individual Evaluator
Interviews 0 officer of interview
certification, Guide
help desk
employees,
processing
employees,
system
technicians,
examiners
Focus Group 1,2,3,4,5,8,9 March - Pilots, Focus group Evaluator,
Interviews April instructors, interview evaluator’s
pilot school guide assistant
administrative
staff
Survey 1,2,3,4,5,8,9 March - Airman Survey Evaluator,
August certificate technical
applicants staff
(pilots,
instructors)

IACRA Program Evaluation Budget

 FAA Implementation of IACRA 8

Survey Budget
Survey construction $ 2,000.00
Survey management $ 1,000.00
Data organization and reporting $ 12,000.00
Total: $ 15,000.00

Focus Group Interviews Budget

Transportation $ 4,000.00
Compensation $ 4,000.00
Data organization and reporting $ 8,000.00
Total: $ 16,000.00

Individual Interview Budget

In-person interviews (compensation) $ 1,500.00
Transportation $ 1,000.00
Phone interviews (compensation) $ 1,500.00
Data organization and reporting $ 4,000.00
Total: $ 8,000.00

Final Report and Recommendations

Written report and recommendations $ 10,000.00
Total: $ 10,000.00

Combined Total Budget: $ 49,000.00

G. Data Analysis

Data analysis is quite possibly the most important part of an evaluation. It is the part

where evidence is determined. In the case of this evaluation, data will be analyzed in different

ways. The online survey is constructed in a way that information can be organized based on the

level of pilot responding to the questions. This step is important because student pilots may find

much more difficulty in filling out the application than commercial pilots. It is important to
 FAA Implementation of IACRA 9

make a distinction between different experience levels. The first step is to utilize the quantitative

data. The survey is composed of a general scale from strongly agree to strongly disagree.

Responses can be gathered and averaged based on weighted responses.

Example:

Strongly Agree = 3, Agree = 2, Disagree = 1, Strongly Disagree = 0

If the sum of 10 responses = 20, then take the sum of the weighted responses (20) and divide by

the total number of responses (10) and this will provide the mean response. The mean is 2 which

means airmen generally agree with the statement presented on the survey. By taking mean

responses from the quantitative data, the evaluator can gather consensus from the questions

presented on the survey.

The survey also has a section for qualitative responses. In the beginning of the survey

total flight hours and number of years as a pilot are asked of the applicant. Along with the

accomplished certificate information of the respondent, the flight hours and years as a pilot will

also be included to give better context to the answers provided in those sections. By analyzing

the survey data in these ways, the evaluation will be able to more adequately define problems

that are being experienced by the entire aviation community as well as problems experienced by

applicants of specific experience levels. Other demographic information is unnecessary to this

evaluation.

The focus group interviews will be organized in much the same way as the qualitative

sections of the survey. The major difference is the specific groups of people who are being

interviewed. For instance, discussion with representatives from a flight school will generate

more insightful data than just having information from a survey which has responses from

certified flight instructors. The qualitative information gathered from these interviews will be
 FAA Implementation of IACRA 10

organized in a manner to show overall concerns across the training community as well as facility

specific concerns. In other words, flight instructors that work in general aviation, administrators

at Flight Safety, or a flight school with examination authority will all have different concerns

about changes to IACRA.

Data collected from the service side of the FAA is just as important as that collected from

the airmen. FAA employees can give valuable insight as to the common questions and problems

with the system. People working at the help desk will have an intricate knowledge of the minor

problems caused by IACRA. Basic categorization of responses will be used for these interviews

to give further insight to the problems that may be causing errors within the application system.

A description of the procedures that will be used to analyze the evaluation data.

H. Evaluation Constraints

Though the full support of the FAA has been given to this evaluation, there are certain

barriers that exist which may impede the implementation of recommendations to IACRA. The

primary evaluation tools for data collection from the aviation community are being conducted on

a voluntary basis. If not enough participants choose to complete the application survey or

participate in the focus groups, the evaluation will be in serious jeopardy of not having an

appropriate set of data from which to draw conclusions. The survey has a unique barrier in that

applications are usually completed shortly before a flight exam. The barrier is that applicants

will feel as though they are too busy preparing to pass the exam than to take the time to complete
 FAA Implementation of IACRA 11

a short survey. A barrier that exists with conducting focus group interviews is the worry that

information will be collected from a less than diverse sample group. The industry is spread

throughout the country and a cost-effective method for conducting focus group interviews has to

be feasible. As stated in previous sections, certain operations and pilot types will be targeted to

represent their similar groups. Trips will have to be made across the country to conduct these

interviews and if not enough pilots are present, the focus-group data will be less reliable. If the

evaluation overcomes the previous barriers, other problems still exist.

Even after the evaluation is complete, there are still barriers that will disallow the

recommended outcomes of the evaluation. The FAA is a bureaucratic body which requires

approval from different levels to allow system changes, especially to the way in which

applications are handled. Even if legitimate recommendations are made, one FAA manager

could block or slow the evaluation implementation for fear that the system will break down due

to such changes. The application system needs to be available anytime flight exams are being

conducted which means on a daily basis. The website can only be shutdown overnight so that

changes or updates can be made and if the system crashes or is down for a significant period of

time the aviation community will become dissatisfied with IACRA.

I. Reporting the Results

When the evaluation is complete, a final report will be written which includes a one page

executive summary giving the primary findings of the evaluation. The report will then include

activities and processes that took place to come to the conclusions and recommendations of the

evaluation. The data collected from the evaluation tools will be compiled and organized in so as

to present the data in a clear way. After the data analysis has been given, a section for

recommendations based on each of the key questions will be provided with in depth justification
 FAA Implementation of IACRA 12

for each recommendation. The final portion of the report will include other data collected from

the evaluation that can be considered important but were not essential in answering the key

questions of the evaluation.

The primary beneficiaries of the evaluation data are the FAA’s administrative officers of

airman certification as they will be the group who are able to authorize changes to the system.

The final report will be supplied in advance of a meeting where any final questions and answers

can take place between the evaluator and the client. When the client is fully satisfied the

executives can decide on which recommendations they will take to improve or change the

IACRA system. To ensure that evaluation results are used by the client, the evaluation report

and executive meeting will be conducted in a way where complete emphasis is placed on

recommendations based on answers to the key questions. In other words, the key questions were

developed early in the evaluation and the client agreed that they were questions which needed to

be answered. If the evaluation answers the key questions, then the client will have difficulty in

denying the changes which may solve any problems associated with the key questions.

Once the results have been filtered by the executives and final decisions are made on the

system changes, instructions will be passed onto those who maintain the system as well as those

who maintain the help desk. The general report will not be made available to the aviation

community; however, changes to the system based on the report will be delivered through a link

on the IACRA website describing what changes have been made as well as the justification for

those changes. The report will be made available by request to political bodies that are

concerned with the evaluation, but will not be published in a general manner unless otherwise

instructed to do so by the FAA. Reporting results in this manner is the most efficient way of
 FAA Implementation of IACRA 13

delivering pertinent information to those concerned and it is likely that IACRA will become a

better system because of the evaluation.

Appendix A
1. Select the airman certificates(s) that you currently hold:

Student Pilot
Recreational Pilot
Private Pilot
Commercial Pilot
Airline Transport Pilot
Certified Flight Instructor
Certified Ground Instructor

2. Total number of flight hours:

3. Total number of years as a pilot:

4. After logging into IACRA:

Strongly Strongly
Agree Agree Disagree Disagree N/A
a. Selecting the appropriate role was easy.
b. Starting or retrieving an application was easy.
c. I was able to easily navigate through the menus.
d. I was able to find all of the information I needed.
e. Submitting my application was easy.
f. The website worked well on my computer.
 FAA Implementation of IACRA 15
g. Overall, it was easy to use from start to finish.

5. Compared to a paper application, I felt that IACRA:

Strongly Strongly
Agree Agree Disagree Disagree N/A
a. Is faster.
b. Is easier to complete.
c. Caused me to make fewer errors.
d. Is an overall better way to complete an airman application.

6. The IACRA help desk:

Strongly Strongly
Agree Agree Disagree Disagree N/A
a. Was easy to contact over the phone.
b. Was helpful in answering my questions.
c. Was friendly.
d. Made the application process much easier.
e. Provided good overall support.

7. When using IACRA:

Strongly Strongly
Agree Agree Disagree Disagree N/A
a. The application instructions were clear.
b. The application questions were clear.
c. Entering information was easy.
d. The system prevented me from making application errors.
e. I felt that the website protected my personal information.
f. I was satisfied with using a digital signature.
g. I liked that I could save my application information.
h. Overall, I was satisfied with the system.
 FAA Implementation of IACRA 16

8. What did you like most about IACRA?

9. What did you like least about IACRA?

10. Do you have any suggestions for improving IACRA?