ASSESSING SPEAKING:

A FRAMEWORK FOR TESTING

SPOKEN INTERACTION
 A Framework for testing Spoken
Interaction
1. Operations (Activities/skills) that are involved in spoken interaction
1. Informational Routine skills (e.g. telling a story)
2. Improvisation skills (e.g. requesting clarification)

2. Conditions – the conditions under which the tasks are performed

1. Processing under normal time constraints
2. Degree of reciprocity/participation in developing interaction
3. Further conditions on the testing situation

3. Criteria for assessing quality of output

1. Criteria for assessing the handling of routine skills
2. Criteria for assessing the handling of improvisational skills
3. Criteria for assessing the handling of microlinguistic skills
4. Analytic vs Global Impression marking scheme
Formats for testing speaking
Indirect Direct
Types _______________________Types

Response Interaction
Elicitation between SS/
Examiner

Purpose of framework:
To offer a tentative checklist of points in each of its three
parts that might usefully be considered in the construction of
language tests which lay claim to be assessing the construct of
spoken language interaction.
Basic types of speaking
1. Imitative
2. Intensive
3. Responsive
4. Interactive
5. Extensive
E.g. Indirect Type of Speaking
test
Direct Type of speaking test
Extensive speaking
Involves lengthy stretches of discourse
Minimal verbal interaction
E.g.
Picture- cued story- telling
Retelling a story, news event
Direct Type
Micro-Skills of Speaking
 Criteria for assessment for
Micro and macro skills of speaking
Micro- skills: phonemes, words, collocations, phrasal units
They include production English stress patterns, reduced forms,
production of fluent speech, use of strategic devices (pauses,
fillers).
Macro- skills: fluency, discourse, function, style, cohesion,
nonverbal communication.
They include the appropriate accomplishment of communicative
functions, use of appropriate styles, registers, conversation rules,
etc.
Testing Speaking
A strong case for testing spoken language
performance directly, in realistic situations.
If we wish to make statements about the capacity
for spoken interaction – should not use MCQ tests
(i.e., indirect tests of speaking where spoken
language is conspicuously absent).
To test speaking ability we should require candidates
to demonstrate their ability to use language in ways
which are characteristic of interactive speech.
 Testing Speaking
The more direct we can make a test and the more we
can incorporate contextual and interactional features
of real-life activity into our tests, the more confidently
we can extrapolate and make statements about what
candidates should be able to do in that real-life
context.
The fewer features of the real-life activity we are able
to include and the less direct the test, the more
difficult it will be to translate performance in the test
into statements about what candidates will be able to
do with the language.
HOW IS ORAL COMMUNICATION DEFINED?
 Defining the domain of knowledge, skills, or attitudes to be
measured is at the core of any assessment.
 Most people define oral communication narrowly, focusing
on speaking and listening skills separately.
 Traditionally, when people describe speaking skills, they
do so in a context of public speaking.
 Recently, however, definitions of speaking have been
expanded (Brown 1981). One trend has been to focus on
communication activities that reflect a variety of settings:
one-to-many, small group, one-to-one, and mass media.
 HOW IS ORAL COMMUNICATION DEFINED?
Con’t
Another approach has been to focus on using communication to achieve
specific purposes: to inform, to persuade, and to solve problems.
A third trend has been to focus on basic competencies needed for
everyday life -- for example, giving directions, asking for information, or
providing basic information in an emergency situation.
The latter approach has been taken in the Speech Communication
Association's guidelines for elementary and secondary students.
Many of these broader views stress that oral communication is an
interactive process in which an individual alternately takes the roles of
speaker and listener, and which includes both verbal and nonverbal
components.
Construction of tests
Theory – Nature of Speaking as a Skill (Bygate, 1987)
Features of spoken interaction
Weir’s (1993) 3 parts
framework:
1. Operations – Activities/Skills involved in
Spoken interaction
2. Conditions under which the tasks are
performed – e.g. time constraints, the number of
people involved and their familiarity with each other
3. Criteria for assessing quality of output – the
expected level of performance in terms of various
relevant criteria, e.g. accuracy, fluency or
intelligibility
 1. Operations: What types of tasks in spoken
interaction to include?
a) To speak a FL, obvious that we need to understand some
grammar and vocabulary (operate at the microlinguistic
level) and have an idea of how sentences are put
together.
b) However, we spend most of out time actually using
sentences to perform a variety of language fuctions and
in so doing spend little time reflecting on the accuracy of
our own or others’ speech.
c) To test whether learners can speak, it is necessary to get
them to take part in direct spoken language activities.
 1. Operations: What types of tasks in spoken
interaction to include? Con’t
d.We are no longer interested in testing whether candidates
merely know how to assemble sentences in the abstract.
e.We want candidates to perform relevant language tasks and
adapt their speech to the circumstances, making decisions
under time pressure, implementing them fluently, and
making necessary adjustments as unexpected problems
arise.
f.Hence, there is a strong case for testing spoken language
performance directly, in realistic situations, rather than
testing hypothetical knowledge of what might be said.
Operations: What types of tasks in spoken interaction to
include? …Con’t

g. Speaking tests should require candidates to

demonstrate their ability to use language in
ways which are characteristic of interactive
speech.
h.The more direct we can make a test and the
more we can incorporate contextual and
interactional features of real-life activity into
our tests,
i. the more confidently we can extrapolate and
make statements about what candidates should
be able to do in real-life context.
Bygate (1987) – description of how speakers
organise in routines what they have to communicate.
They have a repertoire of both informational and
interactional routines which reflect their familiarity with
certain kinds of communication.
Routines are normally recurring patterns of organization of
communication, and can be found firstly in the organization
of information, and secondly in the organization of
interaction.
Also, we need to be aware of improvisational skills which
are brought into play when an interaction falters.
In making decisions on what activities to test it would be
useful to refer to a listing of what has been taught or is
necessary in the target situation. (see Table 1)
Table 1 – Summary checklist of spoken interaction
skills (p.34 Weir,1993)

A. Routine Skills
1. Informational – frequently recurring patterns of
information structure: conventional ways of organizing
speech:
◦ Expository: narration, description, instruction, comparison, story
telling, giving directions, explanations, presentations.
◦ Evaluative: drawing of conclusions, justifications, preferences.
2. Interactional – typical ordered sequences of turns as
in:
Telephone conversations, service encounters, meetings,
discussions, interviews, conversations, decision making.
 Table 1 – Summary checklist of spoken interaction
skills (p.34 Weir,1993)
con’t
B. Improvisation Skills
1. Negotiation of meaning
Use of discourse processing strategies to evaluate communicative
effectiveness and make any necessary adjustments in the course of an event
Speaker may: check understanding, ask opinion, respond to clarification
request, check common ground.
Listener may: indicate understanding through gesture or summarizing,
indicate uncertainty, use elicitation devices to get topic clarified, express
agreement/disagreement.
2. Management of interaction
Agenda management: choice of topic, introduce topic, develop topic, bring it
to a close, change topic.
Turn taking: who speaks, when and for how long.
 2. Conditions
We have focused on the operations we might expect to see in manytests
of spoken interaction, let us now look at the conditions under which
these tasks are performed.
See Table 2 – Checklist of performance conditions (Speaking)(p.39, Weir,
1993)
 Table 2 – Checklist of performance conditions (Speaking)(p.39,
Weir, 1993)

1. Processing under normal time constraints

◦ Tolerance of silence in short turns. Planning time in long turns(Short
speaking turns are more common: usually more spontaneous loosely-
strung- together phrases, rather than neat sentences. Long turns, e.g.
oral presentations or lectures, require more planning decisions, tend
to be more prepared.)
2. Degree of reciprocity/participation in
developing interaction
◦ Equal speaking rights. Shared responsibility for continuance of the
interaction.
◦ speaking rights and responsibility in the maintenance of an
interaction
◦ varies depending on whether it is a lecture, interview,
conversation, etc.
 Table 2 – Checklist of performance conditions
(Speaking)
Con’t
3.Further conditions in the testing situation
◦Purpose (Reason for doing it)
◦help to define the structure and focus of the interaction, as well as the
outcome towards which the participants will be required to work.

Interlocutors
◦Number of participants in the interaction : dialogue, group discussion.
◦Complexity of the task can be related to:
◦Status: the social/professional status in real life of those involved: student,
teacher, examiner, etc. Register: formal/informal
◦Familiarity: participants known or unknown to each other.
◦Gender: male or female examiners/interlocutors for male or female students.
 Table 2 – Checklist of performance conditions
(Speaking)…Con’t
◦Setting
-Attempt should be made within the constraints of the test situation to
simulate reality as closely as possible.
◦Role
Appropriate to age, experience and culture: friend/friend,
undergraduate/supervisor, student/teacher.
◦Topic
Specificity, familiarity, interest
◦Channel
Channel of communication can have an obvious impact on the
performance, e.g. simulating a telephone conversation face to face in the
same room vs simulating a conversation with an interlocutor in another
room.
 Table 2 – Checklist of performance conditions
(Speaking) Con’t
Input dimensions
◦concern with the speech of the interlocutor which the candidate has to process – e.g. rate of
utterance, accent of the examiner, clarity of articulation of the examiner, length of the discourse.
◦Hence dimensions of tasks given should be realistic, in terms of:

Size: processing appropriately sized input

Complexity: language used, subject talked about.
Range: topics covered, lexical fields
Assessing quality of output
Problems in assessing spoken language
- Practical constraints – administrative costs,
difficulties, logistics of testing large numbers of
candidates.
- Issues of validity – what is relevant for
teaching purposes may not necessarily be efficient,
practicable, or cost effective for inclusion in tests
- Assessing speech reliably – Speech is transient,
fleeting and cannot be checked later – needs double
marking
3. Criteria for assessing quality of
output
The third element that needs to be considered in
test task design is how we are to measure the
quality of the output which results from the
spoken language tasks we adopt.
In order to measure the quality of spoken
performance, we first need to establish criteria of
assessment. The following is a set of criteria for
assessment of the output of communicative.
 Table 3 – Checklist for Assessing quality of
output (p.43, Weir, 1993)
3.1 Criteria for Assessing the handling of
routine skills
◦ Effectiveness - To what extent can the candidate
demonstrate an ability to meet effectively the demands of
required informational or interactional routines?
◦ Fluency –smoothness of execution of the task
◦ Appropriateness – the sociocultural ability to take into
account setting, topic, role relationships, formality
required
◦ Coherence – organization of discourse in long turns
 Table 3 – Checklist for Assessing quality of
output con’t
3.2 Criteria for Assessing the handling of improvisational
skills
Effectiveness in
- ability to negotiate meaning in cases of comprehension or production difficulties
manifested on the part of the of the candidate or his/her interlocutor
-ability to manage interaction (agenda in turn taking) actively and flexibly. –
particularly important where speakers are expected to be active participant.
Fluency – smoothness of execution.
Ability to negotiate meaning would for example, include the ability to use
communication strategies with ease when in difficulties.
Appropriateness :
this could include for example, the degree of politeness and suitability of timing in
turn taking or suitability of the language used in request for clarification or
disagreement.
 Table 3 – Checklist for Assessing quality of
output con’t

3.3 Criteria for Assessing the handling of

microlinguistic skills
Accuracy focusing on both intelligibility and grammar
Range: adequacy and variety of vocabulary employed;
adequacy and variety of structures employed.
CON’T
Besides the criteria that might be used to evaluate the
various aspects of spoken interaction described above for
assessing the handling of routines and the handling of any
improvisation and for operating at the microlinguistic level,
testers also need to decide whether they will treat these
criteria separately in an analytic scheme or try to collapse
them into some form of global impression banding.
Table 4 - Analytic marking Scheme (p.43, Weir, 1993)
Table 5 – Global impression marking scheme (p.44, Weir,
1993)
CON’T

The decision on whether to use an analytical or a

global impression band will largely rest on the
degree to which one can describe in behavioural
terms the different levels of proficiency that
student performances will result in.
Analytical scales are recommended – due to our
limited understanding of the continuum of
proficiency in speaking.
 FORMATS FOR TESTING SPOKEN
INTERACTION
Having established what it is that needs to be tested , the test
writer must decide which is the best format for operationalising
the test specifications.
The range of formats that we are going to see a while later
embraces the
direct types such as interaction between students (example 2.1)
and the face-to-face interview (example 3.1)
indirect types such as response elicitation (example 1.1)
 FORMATS FOR TESTING SPOKEN
INTERACTION…con’t
Directness here is a function of how closely a task relates to real-
life performance (in terms of operations and conditions), and
how far performance on the task can be assessed in terms
which allow of direct comparison with that target performance.
The more indirect the task the more difficult it will be to translate
test results into statements about what candidates can or cannot
do in terms of the real-life activity under review.
IELTS
Individual Long turn Sample
Speaking Marking Scheme
IELTS
Conclusion