STAGES OF TEST DEVELOPMENT

& COMMON TEST TECHNIQUES

 STAGES OF TEST DEVELOPMENT OUTLINE

Stating the Writing and

Specifications Items on native
problem moderating
speakers
items

Calibration of Analysis of On non-native

Validation sales results speakers

Handbooks for
test takers
Training staff
SHOULD BE A TEST DESIGNED BY A SINGLE
PERSON OR A TEAM? TEAM!

• IT IS DIFFICULT TO DESIGN A TEST BECAUSE OF…

…OBJECTIVITY
…CRITICISMS
…NATIVE COMMAND OF THE LANGUAGE
STEPS…
  
STATING THE PROBLEMS
Proficiency
Final Diagnostic progress
placement
Kinds?
Purposes?

Constraints?

Questions Abilities?
Expertise

Facilities Backwash

Time How
How detailed?
accurate?
STATING THE PROBLEM

Once the problem is clear, it is also

important to:
Gather information on already existing
test designed for similar situations!
 WRITING SPECIFICATIONS FOR THE TEST

• A) Content
• B) Structure, timing, medium / channel and
techniques.
• C) Criterial levels of performance
• D) Scoring procedure
A. CONTENT
Include clear specifications regarding
• Skills (sub-skills)
• Types of texts
• Addressees
• Length of texts
• Topics
• (Readability)
• Structural range
• Vocabulary range
• Dialect, accent and style
• Speed of processing
B) STRUCTURE, TIMING, MEDIUM/CHANNE L AND
TECHNIQUES
• Test structure
• Number of items
• Number of passages
• Timing
• Medium
• Kinds of test technique(s )
C) CRITERIAL LEVELS OF PERFORMANCE
• Accuracy
• Approppriacy
• Range
• Flexibility
• Size
D) SCORING PROCEDURES
• What rating scales will be used?

• How many people will rate each piece of work?

• What happens if two or more raters disagree about a

piece of work?
WRITING AND MODERATING ITEMS
• A) Sampling
• B) Writing items
• C) Moderating items
A) SAMPLING
How the texts are going to be chosen?
B) WRITING ITEMS
• Try to look at the test through the eyes of the test takers!
• An item without a key is incomplete!
• “The best way to identify items that have to be improved
or abandoned is through the process of moderation”.
C) MODERATING ITEMS
• Intervention of two colleagues
 INFORMAL TRIALLING OF ITEMS
ON NATIVE SPEAKERS

• 20 or more …
• They should be similar to the group being tested in
terms of:
Age
Education
General background
TRIALLING OF THE TEST ON A GROUP OF NON-
NATIVE SPEAKERS SIMILAR TO THOSE FOR
WHOM THE TEST IS INTENDED

• Problems in administration and scoring can be noted.

ANALYSIS OF RESULTS OF THE TRIAL;
MAKING OF ANY NECESSARY CHANGES

• Statistical and qualitative analysis.

How difficult are the items?
Discover misinterpretations (to be modified or dropped)
 CALIBRATION OF SCALES

• It is important to collect samples of performance and

assign each of them to a point on the relevant scale.
 VALIDATION

Low -stakes test

TOEFL

PO P
QUIZES
High -stakes test
 WRITING HANDBOOKS FOR TEST TAKERS,
TEST USERS AND STAFF

Rationale Development Description Sample items

Advices on
Test scores Test administration
preparing for taking Training materials
interpretations
the test
 TRAINING STAFF

Interviewers

Raters

Procters NEED TO BE
TRAINED

Scorers
Computer
operaters
COMMON TEST TECHNIQUES
 OUTLINE
Definition

Multiple
Gap filling COMMON TEST choice
TECHNIQUES

Short
True or
answers
False
 WHAT ARE THE TEST TECHNIQUES?
Means of eliciting behavior
Reliable and
valid
behavior
Behavior which
can be reliably
scored

Less time
and effort Beneficial
backwash

TECHNIQUES
 MULTIPLE CHOICE ITEMS
- Stem
-Distractor
 MULTIPLE CHOICE ITEMS

* Advantages
- Rapid and economical scoring
- Test taker is not required to produce language
 MULTIPLE CHOICE ITEMS

* Disadvantages
-Performance may give incorrect picture of candidates’
ability.
- Guessing may have unknown effect on scores.
- Lack of distractors.
- Backwash may be harmful.
- Cheating may be facilitated.
- Difficult to write items.
MULTIPLE CHOICE ITEMS

ADVANTAGES DISADVANTAGES

Performance

Guessing
Reliable
Lack of
scoring
distractors
Testing of
receptive skills Washback Difficult to
write items
CHEATING Great demand
on time and
expertise
YES / NO AND TRUE / FALSE ITEMS

Multiple
choice (2
options)

Too
informal

Reliability and
Reasons? Validity
YES / NO AND TRUE / FALSE ITEMS

Weakness:
Test taker has a 50% chance of choosing
the correct response
 SHORT-ANSWER ITEMS

READING TESTS LIESTENING TESTS

 SHORT-ANSWER ITEMS

Advantages:
- Less guessing
- No distractors
- Cheating is more difficult.
- Easier to write items
 SHORT-ANSWER ITEMS

Disadvantages:
- Responses may take longer.
- Test taker has to produce language.
- Scoring may be invalid or reliable.
- Scoring may take longer.
 GAP FILLING ITEMS

- Work well in tests of grammar and vocabulary

- Does not work well:
+ where the grammatical element is
discontinuous
+ where minor or subtle differences of
meaning are concerned ( grammar or vocabulary)
 GAP FILLING ITEMS
It is a valuable technique as long as the context is
provided! will, might,
• A: What will he do? could,may, etc
• B: I think he ___ resign.
But
A: I wonder who that is.
B: It ___ be the doctor.
A. How can you be so certain?
THANK YOU !