Testing problem for writing

1. We have to set tasks that are properly

representative of the population of tasks that we
should expect the students to be able to perform.
2. The tasks should elicit valid samples of writing
(which truly represent the student’s ability)
3. It is essential that the samples of writing can and
will be scored reliably.
 Representative tasks

i) Specify all possible content (operations, types

of text, addressees, length of texts, topics, dialect and
style)
ii) Include a representative sample of the
specified content: try to select a representative set
of tasks. The more tasks (within reason) that we set,
the more representative of a candidate’s ability (the
more valid). If a test includes a wide ranging and
representative sample of specifications, the test is
more likely to have a beneficial backwash effect.
 Elicit a valid sample of writing ability

Set as many separate tasks as is feasible:

Candidates performance on a task or different tasks can’t
be always consistent, so we have to offer candidates as
many “fresh starts” as possible, and each task may
represent a fresh start.
Test only writing ability, nothing else. Test writing
not creativity, imagination, general knowledge, etc.
Restrict candidates: Writing tasks should be well
defined, candidates should know just what it is required.
(specify length, give notes, clues or pictures).
 Ensure valid and reliable scoring

Set tasks which can be reliably scored.

Set as many tasks as possible
Restrict candidates
Give no choice of tasks
Ensure long enough samples
Create appropriate scales for scoring
Calibrate the scale to be used
Select and train scorers
Follow acceptable scoring procedures
 Holistic vs Analytic Scoring

Holistic (Impressionistic) involves the assignment of a

single score to a piece of writing on the basis of an overall
impression of it. An advantage is that it can be very rapid.
Analytic Scoring requires a separate score for each of a
number of aspects of a task. Advantages:
1.It disposes of the problem of uneven development of
subskills
2. Scorers are compelled to consider aspects of
performance which they might otherwise ignore
3. The very fact that the scorer has to give a number of
scores will tend to make scoring more reliable.