1 Running Head: PROTOTYPE FORMATIVE EVALUATION

Formative Evaluation of ILR Familiarization Prototype

Luay Askar

California State University, Monterey Bay

IST622 Assessment and Evaluation

Dr. Bude Su

July 24, 2018

 PROTOTYPE FORMATIVE EVALUATION 2

Table of Contents

Introduction....................................................................................................................................4
Methodology...................................................................................................................................5
Prototype.....................................................................................................................................5
Learners......................................................................................................................................6
Cognitive and Affective Domain...............................................................................................6
Tryout conditions and process..................................................................................................6
Evaluand.................................................................................................................................7
Process.....................................................................................................................................7
Conditions...............................................................................................................................8
Results.............................................................................................................................................8
Entry conditions.........................................................................................................................9
Intended..................................................................................................................................9
Observed.................................................................................................................................9
Instruction..................................................................................................................................9
Intended..................................................................................................................................9
Observed.................................................................................................................................9
Outcomes..................................................................................................................................10
Intended................................................................................................................................10
Observed...............................................................................................................................10
Data Analysis........................................................................................................................10
Recommendations....................................................................................................................12
Summary......................................................................................................................................13
References.....................................................................................................................................15
Appendices....................................................................................................................................16
Appendix A Consent Form.....................................................................................................16
Appendix B Demographics Questionnaire.............................................................................17
Appendix C Pre-Instructional Self-Assessment Questionnaire..............................................18
Appendix D Pre-Test Questionnaire.......................................................................................20
Appendix E Observation Checklist........................................................................................22
Appendix F Post-Test Questionnaire......................................................................................23
Appendix G Level 1 Evaluation..............................................................................................25
Appendix H Excel Worksheet..................................................................................................27
PROTOTYPE FORMATIVE EVALUATION 3

Appendix I Link to Prototype Itcdland...................................................................................28

 PROTOTYPE FORMATIVE EVALUATION 4

Introduction

Formative Assessment is one of the main tools that provide feedback to the instructor

during the process of learning (Hudson and Bristow, 2006). Likewise, instructional designers use

formative evaluation to assess and modify their instructional design model during the process of

development. This assessment is an iterative process that requires preparation, assessment,

evaluation, and feedback.

The designer developed a fifteen minutes interactive instructional prototype for training

teachers at The Defense Language Institute (DLI), in Middle East School II, on the basics of

Interagency Language Roundtable (ILR) descriptors (ILR, 2011). These descriptors are used in

assessing language proficiency and assigning an ILR level to authentic reading passages. The

prototype is a self-paced instructional module and is one of the four modules for the final

capstone under construction that aims to cover the Diagnostic Assessment (DA) familiarization

and its main element, ILR, that is required for conducting DA. The designer carried out a

formative evaluation for a set of instructions to estimate the effectiveness of the content,

delivery, navigation and the learning outcome of the prototype. This pilot test was introduced to a

small group sample that is representative of the intended learners.

 PROTOTYPE FORMATIVE EVALUATION 5

Methodology

Prototype

The DLI is aiming towards teachers adhering to a student-centered teaching

methodology. In order to support this goal, teachers are required to prepare teaching materials

and activities to enhance learning in addition to preparing assessment material to diagnose areas

that need improvement for each learner. This diagnosis should be followed by an interventional

instructional plan to overcome any difficulty. During the “DA specialist certification course”

training, most of the participants expressed lack of the basic knowledge about DA and

specifically about the ILR descriptors. The Needs analysis confirmed this lack of knowledge

which has a negative influence on the outcome of the teaching-learning process and on achieving

the organizational goal.

The proposed training will consist of four modules:

 Module One (ILR familiarization and passage rating) Reading.

 Module Two (ILR familiarization and passage rating) Listening.
 Module Three (DA familiarization) DA Protocol.
 Module Four (DA familiarization) DA Reports.

Module One prototype is used for the formative evaluation. This module is demonstrated

by using Adobe Captivate 2017 program, which will facilitate the learning process in interactive

instructional steps. The module will start with an introduction to ILR descriptors’ definition

followed by addressing the different ILR levels. Then, the learners will interact with non-graded

knowledge check activities that will allow the learner to go back and forth reviewing the

descriptors. The last activity is a graded quiz where the learners are required to read some Arabic

passages, rate, and assign an appropriate ILR level and then submit the answer. Upon submitting
 PROTOTYPE FORMATIVE EVALUATION 6

the answer, immediate feedback is provided. Learners can review the quiz as needed. The

published module will be sent as a zipped file to the participants through their work email.

Learners

The target audience are Arabic language teachers at Middle East School II. The audience

are of different gender and diverse years of teaching experience at DLI. Their educational

background varies between College Graduates (four years) to those with a Doctoral Degree in

different educational and professional fields (Appendix B). A pre-instructional self-assessment

survey (Appendix C) was conducted for nine teachers who volunteered to participate in this

survey. Two of the participants were excluded because they showed a high level of experience

and knowledge in the pre-instructional survey. The survey result of the remaining seven teachers

(four females and three males) revealed that 57.5% of them are somewhat familiar with the ILR

scale.

Upon completing this instructional prototype, learners will be able to identify and assign

appropriate ILR level for different Arabic reading passages with 85% accuracy because of

diversity and complexity of the Arabic language structure.

Cognitive and Affective Domain

A pre-test questionnaire was conducted to evaluate the learners’ knowledge level in using

the ILR descriptors (Cognitive - Knowledge domain). In addition, after completing the training, a

post-test questionnaire was conducted to assess the effectiveness of the module in changing the

audience’ knowledge and attitude in using the ILR descriptors (Cognitive - Affective domain).
 PROTOTYPE FORMATIVE EVALUATION 7

Tryout conditions and process

Formative evaluation for the prototype was conducted while developing the final training

module in order to provide the designer with the feedback that can be used to modify and

validate the module.

Evaluand. During the tryout, the designer will receive feedback from the participants

regarding:

 Content: How relevant are the contents to the needs and abilities of the participant?
 Instructions: Whether instructions are clear and adequate for the audience to navigate

the module?
 Delivery method: What is the audience preference as a delivery mode for the

instructions?
 Learning outcome: Did the module change the participants’ knowledge and attitude in

using the ILR descriptors to select authentic material for teaching and assessment

uses?
Process. The tryout process consists of six steps:
1. Survey. The participants were teachers from Middle East School II, who accepted

voluntarily to participate in the tryout after receiving a short summary of the

training prototype. The teachers were given the following forms:

 Consent form which they read and signed (Appendix A).
 Demographics questionnaire (Appendix B).
 Pre-instructional self-assessment questionnaire (Appendix C).
 Pre-test questionnaire (Appendix D) to assess the teachers’ pre-training

knowledge.
 PROTOTYPE FORMATIVE EVALUATION 8

2. Training. The forms were collected from the teachers, then the prototype was

emailed as a zipped file to them with instructions on how to unzip and run the

module (Appendix I includes the link to itcdland).

3. Observation. During the training, the designer observed in-person three of the

participants to assess the ease of navigation and look for any glitches while

proceeding through the training, in addition to having a direct feedback or

concerns from the trainees without interfering with the training process (Appendix

E).
4. Post-test questionnaire I. Teachers were given the post-training questionnaire

(Appendix G) to cover level one of “Kirkpatrick’s four evaluation levels”

(Kirkpatrick, 2006). This questionnaire is given to the teachers to provide

evaluative feedback regarding the training module and trainees’ satisfaction. Also,

feedback about the delivery method and packaging of the prototype.

5. Post-test questionnaire II. Teachers were given the post-training questionnaire

(Appendix F) to cover level two of “Kirkpatrick’s four evaluation levels” that

aimed to measure the extent of learning and change of teachers’ attitudes

following this prototype training.

6. Analysis. The last step was analyzing the data collected and giving

recommendations.

Conditions. The tryout was conducted under the same environmental condition that is

expected for the final training module. The teachers received the training at their workplace

using their work computers. The training module was delivered to the teachers through their

work email with instructions on how to unzip the file and download it on their desktop in order

to activate the captivate module.

 PROTOTYPE FORMATIVE EVALUATION 9

Results

The data collected from the formative evaluation of this prototype was analyzed and

interpreted to modify and improve the prototype instructions and thus the final product. The

areas covered in this analysis included content, delivery, and assessment.

Entry conditions

Intended. The sample of volunteered teachers was nine teachers from the Middle East

School II. They were six females and three males. The participants were briefed about the

training and after signing the consent, teachers were given the demography and self-assessment

questionnaires that were collected from them at a later time. There was no pre-requisite in order

to participate in this training.

Observed. Upon conducting the self-assessment questionnaire, two female teachers were

excluded from the training because the results of their self-assessment questionnaire revealed

higher scores. The rest of the teachers were provided with pre-test questionnaire.

Instruction

Intended. The intention was to provide teachers with a summary of the training followed

by a consent form to sign, and a self-assessment questionnaire. This would be followed by a pre-

test questionnaire before starting the training. The teachers would receive an email with the

training in a zipped file. Instructions would be provided in the email for downloading the

prototype and activating it. During the training, the designer will observe participants to identify

any technical issue and observe the reaction of the trainee with the contents. After completing the

training, teachers would be given level one and level two Kirkpatrick’s evaluation levels as post-

test questionnaires. In addition, teachers would be asked to provide any recommendation about

the module.
 PROTOTYPE FORMATIVE EVALUATION 10

Observed. The training was carried out as it was intended. All teachers were able to

access the prototype module from their work computer. Also, they were able to navigate

throughout the steps with no problem or issue except for one teacher who had a problem with the

“next” button that required a double click to function.

Outcomes

Intended. The training was intended to bridge the gap between the learners’ current and

the required knowledge and attitude in using ILR descriptors in selecting teaching material.

Therefore, this training intends to familiarize teachers with the ILR descriptors and enables them

to identify and select level appropriate authentic material for use in classroom and assessment

activities. Getting more knowledge about the ILR descriptors is the cornerstone for

implementing diagnostic teaching and diagnostic assessment that are important in enhancing the

language learning process.

Observed. As for this prototype module, the teachers were able to run the module by

downloading the captivate zipped file and activating it on their desktop. Teachers were able to

follow all the instructions without facilitation, also there were no challenging technical problems

reported except for one teacher who had a problem with the “next” button that required a double

click to function. Afterward, the content, delivery method and learning outcomes were evaluated

(Appendix G). Positive feedback and recommendations were submitted by the participants.

Data Analysis. Data collected before, during, and after conducting this training was used

to assess whether there was a gain in knowledge about the ILR descriptors and their different

levels, or a change in the attitude of the teachers toward the use of ILR descriptors in selecting

authentic material for use in classroom or assessment. To evaluate the results of the pre- and

post-test, the designer assigned points for the responses as shown in the table below:
 PROTOTYPE FORMATIVE EVALUATION 11

Response Point

Poor 1

Fair 2

Good 3

Very Good 4

Excellent 5

The table below shows the Pre- and Post-Test teachers’ scores collected

No. Pre-Test Post Test

1 12 36
2 23 31
3 38 43
4 40 46
5 20 23
6 35 51
7 36 40
 PROTOTYPE FORMATIVE EVALUATION 12

The following Bar chart shows Pre- and Post-Test teachers’ scores collected

Pre-Post Training Comparison

60
50
40
30
20
10
0
1 2 3 4 5 6 7
Pre Test Post Test

In order to find out whether using the module changed teachers’ knowledge and attitude

in using ILR descriptors, the designer analyzed the data using “T-test paired two samples for

means” as shown in the table below.

t-Test: Paired Two Sample for Means

Variable 1 Variable 2
29.1428571 38.5714285
Mean 4 7
115.476190 89.6190476
Variance 5 2
Observations 7 7
Pearson Correlation 0.7133773
Hypothesized Mean
Difference 0
df 6
t Stat -3.22174897
 PROTOTYPE FORMATIVE EVALUATION 13

0.00904865
P(T<=t) one-tail 7
1.94318028
t Critical one-tail 1
0.01809731
P(T<=t) two-tail 3
2.44691185
t Critical two-tail 1

The research hypothesis (H1) states that there is a difference between the pre-test (µ₁) and

post-test (µ₂) results in this training, that is H1: µ₁< µ₂. Therefore, the null hypothesis is (H0: µ₁≥

µ₂). The mean score for the pre-test is 29.143 and for the post-test is 38.571. Comparing the t

Stat with the t Critical value of one tail, the result revealed t Stat=3.221> t 0.05,6=1.943, this

indicates that there is a statistically significant result in the data collected and that we can reject

the null hypothesis. Also, the P0.05 value (0.009) is less than 0.05 and this confirms the rejection

of the null hypothesis.

The designer also conducted an Effect Size (d) test to confirm that it is practically

significant and the result revealed d= 0.87, this result is > 0.8 and thus indicates that there is a

large practical significance on applying this training (Appendix H).

Recommendations

Conducting formative evaluation for a prototype is essential for a project under

development. Through this evaluation, the designer collects information that will assist in the

development of the final product. Recommendations can be collected from different resources;

directly from the participants (verbally or written), during the observation process (the designer

can observe any technical issue or change in participants’ behavior while performing a certain

action), or form analyzing the collected data. The following are the collected recommendations:
 PROTOTYPE FORMATIVE EVALUATION 14

1. Based on the teachers’ feedback, they recommended adding more samples and exercises

to the prototype. This is a valid point and will be taken into consideration in the final

project while taking the time constraint into consideration.

2. Teachers also recommended adding samples for the ILR plus levels. This is also a valid

point but designer already pointed out at the beginning of the prototype that the module

will discuss the ILR base levels only due to time constraint.
3. During the observation, there were some technical issues with some glitches in the

function of certain buttons that required double-click in order to function. This issue was

noticed previously during the construction of the prototype and it is related to the device

being used to conduct the training. The designer will try to make sure it will function

properly on different devices in the final product.

Summary

Formative evaluation is an essential tool that should be conducted during the different

stages of the instructional design process. This type of evaluation will provide the designer with

valuable information that can be used to clarify, modify and reconstruct the design while in

progress.

The designer held a formative evaluation for the prototype “ILR Familiarization” using

seven teachers as a sample from the target audience. Teachers participated voluntarily in the

tryout and were given several types of questionnaires after signing a consent form. These

questionnaires were designed to gather information regarding demography, self-assessment, pre-

test and post-test questionnaires, thus covering level one and level two from “Kirkpatrick’s four

levels of evaluation” that are used to assess the effectiveness of a training.

 PROTOTYPE FORMATIVE EVALUATION 15

The research hypothesis was that ILR familiarization training module would have a

significate effect in improving teachers’ knowledge and use of the ILR descriptors in selecting

and assigning levels to passages. The data collected was analyzed using a t-test for paired two

samples for means. The findings revealed there is a statistically significant difference between

the pre and post-test knowledge which indicates that there was an improvement in the

performance of the participants after taking this training.

The feedback and recommendations collected from the audience and the observations

conducted by the designer during the training process will be used to modify and improve the

prototype and set a solid base for the final training module.

References

Hudson, J. & Bristow, D. (2006). Formative assessment can be fun as well as educational.

Advanced Physiological Education. 10, 33-37. https://doi.org/10.1152/advan.00040.2005

Interagency Language Roundtable (2011) [Website]. Descriptions of Proficiency Levels.

Retrieved from http://govtilr.org/Skills/ILRscale1.htm

Kirkpatrick, L. & Kirkpatrick, J. (2006). Evaluating Training Programs: The Four Levels,

3rd edition. Oakland, CA: Berrett-Koehler Publishers, Inc.

 PROTOTYPE FORMATIVE EVALUATION 16

Appendices

Appendix A

Informed Consent Form for learners

Thank you for your approval in participating in the prototype “ILR Familiarization Prototype”.

The goal for this training is to introduce ILR descriptors that will help in preparing authentic

material through selecting ILR-Level appropriate scripts. Your participation in the formative

evaluation of this prototype will be of great importance to improve the training module. Any

feedback or response is of high confidentiality, whether you continue your participation or

withdraw at any time. The module will be specifically directed to your colleagues from the

faculty member in the school, so your participation and opinion are appreciated.

By signing and placing the date in the space provided below, you indicate that you have read the

above information and have expressed your consent to participate in the formative evaluation.

You also understand that you can withdraw from the participating in the prototype at any time.

: Date ________________ Signature : ______________________________

 PROTOTYPE FORMATIVE EVALUATION 17

Appendix B
Demographics Questionnaire

Please indicate your responses to the following questions by marking questions

1. Gender:
□ male
□ female

2. Age:
□ 22 – 28 □ 29 -35 □ 36 – 42

□ 43 – 49 □ 50 – 56 □ 57 - older

3. Educational level:

□ High school/GED
□ Some college
□ 2-year college degree (Associated)
□ 4-year college degree (BA, BS)
□ Master’s degree
□ Doctoral degree

4. Years of experience:

a. Years of teaching experience at DLI: ____ years ___ months.

b. Years of teaching experience in total: ____ years ___ months.

 PROTOTYPE FORMATIVE EVALUATION 18

Appendix C

Pre-Instructional Self-Assessment Questionnaire

1. Do you use ILR descriptors in selecting material for classroom use?
a) Yes
b) No
2. Do you use ILR descriptors in selecting material for assessment use?
a) Yes
b) No
3. If your answer is (b) in the previous question, what is the reason?
a) time-consuming
b) No knowledge

Please choose the answer that best matches your response.

Not at all Somewhat Moderately Extremely

The ILR descriptors
Familiar Familiar Familiar Familiar

1. How familiar are you in 1 2 3 4

defining ILR scale?

2. How familiar are you in 1 2 3 4

identifying ILR descriptors for
level 1?

3.How familiar are you in 1 2 3 4

identifying ILR descriptors for
level 2?

4. How familiar are you in 1 2 3 4

identifying ILR descriptors for
level 3?
 PROTOTYPE FORMATIVE EVALUATION 19

5. How familiar are you in 1 2 3 4

selecting ILR level 1 material?

6. How familiar are you in 1 2 3 4

selecting ILR level 2 material?

7. How familiar are you in 1 2 3 4

selecting ILR level 3 material?

8. How familiar are you in 1 2 3 4

justifying the selection of ILR
level 1 material?

9. How familiar are you in 1 2 3 4

justifying the selection of ILR
level 2 material?

10. How familiar are you in 1 2 3 4

justifying the selection of ILR
level 3 material?
 PROTOTYPE FORMATIVE EVALUATION 20

Appendix D
Pre-test Questionnaire

Please rate your knowledge level in doing the following:

Fair Good Very Good Excellen

Poor
t

1. I can define the ILR

1 2 3 4 5
scale.

2. I can identify ILR

1 2 3 4 5
descriptors for level 1.

3. I can identify ILR

1 2 3 4 5
descriptors for level 2.

4. I can identify ILR

1 2 3 4 5
descriptors for level 3.

5. I can select level 1

1 2 3 4 5
authentic material using
the ILR descriptors.
6. I can select level 2
1 2 3 4 5
authentic material using
the ILR descriptors.
7. I can select level 3
1 2 3 4 5
authentic material using
the ILR descriptors.
8. I can justify selecting
1 2 3 4 5
level 1 descriptors for a
passage.
 PROTOTYPE FORMATIVE EVALUATION 21

9. I can justify selecting

1 2 3 4 5
level 2 descriptors for a
passage.
10. I can justify selecting
1 2 3 4 5
level 3 descriptors for a
passage.
 PROTOTYPE FORMATIVE EVALUATION 22

Appendix E

Observation Checklist

Checklist Yes No Comments

Participant completed the

entire Module.

Participant is following the

instruction.

Participant is able to navigate

with difficulty.

Participant overcame a

difficulty without an assistant.

Participant overcame a

difficulty with an assistant.

 PROTOTYPE FORMATIVE EVALUATION 23

Appendix F

Post-test Questionnaire

Please rate your knowledge level in doing the following:

Fair Good Very Good Excellen

Poor
t

1. I can define the ILR

1 2 3 4 5
scale.

2. I can identify ILR

1 2 3 4 5
descriptors for level 1.

3. I can identify ILR

1 2 3 4 5
descriptors for level 2.

4. I can identify ILR

1 2 3 4 5
descriptors for level 3.

5. I can select level 1

1 2 3 4 5
authentic material using
the ILR descriptors.
6. I can select level 2
1 2 3 4 5
authentic material using
the ILR descriptors.
7. I can select level 3
1 2 3 4 5
authentic material using
the ILR descriptors.
8. I can justify selecting
1 2 3 4 5
level 1 descriptors for a
passage.
 PROTOTYPE FORMATIVE EVALUATION 24

9. I can justify selecting

1 2 3 4 5
level 2 descriptors for a
passage.
10. I can justify selecting
1 2 3 4 5
level 3 descriptors for a
passage.

Appendix G
Level One Evaluation
 PROTOTYPE FORMATIVE EVALUATION 25

Instructions:
Please rate your degree of agreement concerning the statements below. Feel free to skip any
statement you don’t feel comfortable answering. Your responses will be only used for the
purpose of formative evaluation of the ILR Familiarization. The confidentiality of the survey and
your anonymity will be ensured.
The first 8 questions select one option, the last two questions write a short statement.
1=Strongly Disagree 2=Disagree 3=Neutral 4=Agree 5=Strongly Agree
1. It is easy to navigate through the training module.
1 2 3 4 5
1
2. The amount of time it took me to complete the training is acceptable
1 2 3 4 5
1
3. The activities are functioning easily
1 2 3 4 5
1
4. The content of the module is well presented
1 2 3 4 5
1

5. The instructions are clear

1 2 3 4 5
1

6. The contents are logically organized

1 2 3 4 5
1

7. The training is related to my job

1 2 3 4 5
1

8. Overall the training is an effective learning experience

1 2 3 4 5
1

9. What things did you like the most about this training?
 PROTOTYPE FORMATIVE EVALUATION 26

10. What things did you like the least about this training?

Appendix H

Excel Worksheet
 PROTOTYPE FORMATIVE EVALUATION 27

Appendix I

Link to the prototype

 PROTOTYPE FORMATIVE EVALUATION 28

http://itcdland.csumb.edu/~laskar/mist/ist526/Final%20Project_Askar_Luay/