Item Development

Presented to: Majmaah University

Neil Wilkinson

Creating better test items

The FACTS of good test items

Face valid

Accurate

Consistent with current practice

Test blueprint representative

Statistically appropriate

Creating good test items

Right
Expertise

Sound
Processes

Efficient
Tools

xx

Right
Right
Expertis
Expertise
e

Sound
Processes

Subject matter expertise

ensures test is valid

Content development expertise

ensures test is superiorly crafted

Psychometric expertise

ensures test is reliable and fair

Efficient
Tools

xxx

Right
Expertise

Sound
Sound
Processes
Processes

Efficient
Tools

Write items

Gap analysis
of item bank

Review of field
test results

Review &
approve items

Field test
items

xxx

Sound item writing processes

Assignments

Gap analysis

Training

Good item writing

practices
Style guidelines
Cultural
sensitivity
Intellectual
property rights
Confidentiality

Review
procedures

Editorial review
Content review
Acceptance
criteria

Item
performance
monitoring
Feedback
mechanisms

Item performance analysis and feedback

Why analyze items?

Statistical behavior of bad items is fundamentally different

from that of good items

Provides quality control indicating items which should be

reviewed by content experts

Feedback on how items are performing can be used in

future item writing efforts providing continuous quality
improvement in the item writing process

Distractor analysis

High-scoring candidates should select the correct option

Look

at discrimination for each distractor

Low-scoring candidates should select from among the

distractors at random

Look at p-values for each distractor

Option:

A
KEY

N
Mean

241
219.63

p-value
PtBis

9
215.11

3
196.67

11
210.73

70
210.03

0.72

0.03

0.01

0.03

0.21

+0.22

-0.02

-0.10

-0.06

-0.19

Total

334

Diagnosing potential problems

Option:
N

C
KEY

113

23

194

233.00

177.00

215.08

215.48

218.54

p-value

0.01

0.01

0.34

0.07

0.58

PtBis

+0.06

-0.16

-0.07

-0.02

+0.09

Mean

Total
334

Feedback into the item development process

Sharing performance statistics to authors of individual items

may be problematic due to time between authoring and use,
but...

Item performance statistics can still inform the item

development process.

Share and discuss item performance and reasons for poorly

performing items.

Show examples of good, average, and poorly performing

items.

Right
Expertise

Sound
Processes

Efficient
Efficient
Tools
Tools

xxx

Item Bank
A repository of items with their associated metadata & historical
record

Right
Expertise

Sound
Processes

Efficient
Efficient
Tools
Tools

Item Bank Data

The item itself

Item author

Item type

Content area reference from test blueprint

Cognitive level

Revision history

Item status

Performance statistics

xxx

Right
Expertise

Efficient
Efficient
Tools
Tools

Sound
Processes

xxx

Item Bank Features

Item authoring

Test construction

Remote item authoring

Test assembly

Support of multiple item

types

Export features

Item banking

Ancillary features

Search capabilities

Security and access

Import/export capabilities

Workflow management

Batch editing capabilities

Project tracking

Right
Expertise

Sound
Processes

Efficient
Tools

Effective
Item
Development

Guidelines for an effective item development process

An effective item development process begins well

before the first test item is written with
development of test blueprints.

Right
Expertise

Sound
Processes

Efficient
Tools

Effective
Item
Development

Guidelines for an effective item development process

Carefully consider the associated information that is

required and stored for each item.

Right
Expertise

Sound
Processes

Efficient
Tools

Effective
Item
Development

Guidelines for an effective item development process

Recognize that item development requires more than

subject matter expertise.

Right
Expertise

Sound
Processes

Efficient
Tools

Effective
Item
Development

Guidelines for an effective item development process

Item authors require training in item writing.

Right
Expertise

Sound
Processes

Efficient
Tools

Effective
Item
Development

Guidelines for an effective item development process

Item development requires more than authoring,

sound review processes are also essential.

Right
Expertise

Sound
Processes

Efficient
Tools

Effective
Item
Development

Guidelines for an effective item development process

Style and grammar guidelines and review processes

are necessary to ensure items are testing what
they are meant to test.

Right
Expertise

Sound
Processes

Efficient
Tools

Effective
Item
Development

Guidelines for an effective item development process

Use item performance analysis to guide further review

of the items and to provide feedback to item
authors.

Writing Good Multiple-Choice Test Items

What Makes a Content-Appropriate Item

Reflects specific content

Able to be classified on the test blueprint

Does not contain trivial content

Independent from other items

Not a trick question

Vocabulary suitable for candidate population

Not opinion-based

Source: Haladyna, T.M. (1996). Writing Test Items to Evaluate Higher Order Thinking. Boston, USA: Allyn & Bacon.

Anatomy of a Multiple-Choice Item

Which city has been awarded

the 2016 Summer Olympics?

Stem

Distractors

A. Chicago
B. Madrid
C. Rio de Janeiro
D. Tokyo

Answer Options
Key

Not opinion based

Avoid the use of opinion based material in items

Poor Stem:
Which are the best football team in the world?
a. Barcelona
b. Manchester United.
c. Bayern Munich
d. Athletico Madrid

Better Stem:

Which football team won the European championship in

1999?

Guidelines for Writing the Stem

Place most of the phrasing in the stem.

Poor Stem:
Type II diabetes is
a.
b.
c.
d.

also called juvenile-onset.

characterised by insulin dependency.
primarily seen in adults over 40.
often managed by drug therapy.

Better Stem:

What is the most frequent age of onset for Type II

diabetes?

Guidelines for Writing the Stem

Avoid excessive, unnecessary wording.

Poor Stem:

Certified individuals are required to undertake

continuing education. What is the minimum number of hours
required in a two-year period?
a.
b.
c.
d.

10
15
20
25

Better Stem:

How many hours of continuing education must a certified

individual take in a two-year period?

Guidelines for Writing the Stem

Clearly describe the problem or what is being

asked of the candidate.

Poor Stem:
Continuing education is
a. required every two years.
b. important to life-long learning.
c. an optional activity.
d. undertaken at certified centers.

Better Stem:

How many hours of continuing education is a certified

individual required to undertake in a 5-year cycle?

Guidelines for the Answer Options

Maintain similarity in the answer options.

Keep the answer options relatively equal in

length.

Keep all answer options grammatically correct

with the stem and parallel.

Avoid cueing the right answer.

Avoid words such as always and never.

Avoid repetitive wording in the answer options.

Vary location of key.

Place answer options in logical order.

Number of Answer Options

Four is most common.

Recent research indicates three may be sufficient

if the two distractors are strong and plausible.

Rarely have three functional distractors.

Using more options does little to improve

item and test score statistics and typically
results in implausible distractors.*

*Rodriquez, M.C. (2005). Three options are optimal for multiple-choice items: A meta-analysis of 80 years of research.
Educational Measurement: Issues and Practice, 24(2), 3-13 (p. 11).

Ten steps to an Effective Item Writing Process

Ten Steps to an Effective Item Writing Process

Use item bank inventories to direct item writing

activities.

Ten Steps to an Effective Item Writing Process

Provide a training program for item writers.

Ten Steps to an Effective Item Writing Process

Develop a style guide.

Ten Steps to an Effective Item Writing Process

Require all items to be classified according to the

test blueprint.

Ten Steps to an Effective Item Writing Process

Ensure all items are validated with current

references and the validations are recorded.

Ten Steps to an Effective Item Writing Process

Editorially review all items for grammar and

style.

Ten Steps to an Effective Item Writing Process

Include an item review process by Subject

Matter Experts (SMEs).

Ten Steps to an Effective Item Writing Process

Share acceptance criteria with item writers and

item reviewers.

Ten Steps to an Effective Item Writing Process

Use statistical analysis of items to guide further

review.

Ten Steps to an Effective Item Writing Process

Provide feedback to item writers.

Thank You!