Chapter 4: Deciding What to

Test
Presented by Cristina O’Farrill Perea
Points that will be discussed
• Test purpose
• Test criteria
• Construct definition

• Concern about the level of models.

If it is described as either construct-based
models, or as behavioral-performance.
What do you look for when designing a test?
Harris and Bell (1994: 42-43) suggest that teachers should ask the
following questions when considering how to design their assessment
for a learning programme:
• What do we hope to achieve?
• How can the important issues be identified?
• Who should be involved?
• What may be the effects of evaluating?
• How can information be collected and analyzed?
• Who can best collect and analyze the information?
• What are the most applicable sources of information?
• What are the constraints (e.g. time, manpower, political)?
Before testing
There needs to be a PURPOSE for testing.

• What do we test?
• What is the information we need, and
why do we need it?

4.1 The Test desing cycle

Example of Test Purpose (Architect)
Fulcher and Davidson (2009) use
architect as a metaphor for test
development.
When architects begin to design a
building, they must have a clear idea of
its purpose.

Without it we could have design

chaos.

Just as it is difficult to evaluate the

success of a building without a
purpose, it is impossible to evaluate a
test.
Test Criterion

• see video
https://www.youtube.com/watch?v=kNpPOfwwkQE
 Construct definition
Constructs are the abilities of the learner
that we believe underlie their test
performance, but which we cannot directly
observe.
e.g. Intelligence, attitude

Other resource
Cronbach and Meehl (1955) define construct as:
Groups of functionally related behaviors, attitudes, processes, and experiences. Instead of seeing
intelligence, love, or fear we see indicators or manifestations of what we have agreed to call
intelligence, love, or fear.
Example of Construct Definition
• Test criterion and Construction:
Read second paragraph (p. 97)
Stamina test (p. 99) The test must be indirect- removed from reality. In other words it
is impossible to have a direct test because it would be too costly, unethical, and extremely
dangerous.
Practice
Excersice 4.2 Identifying constructs
What other constructs do you think you may need to include in a test
for lifeguards?
Where do constructs come from?
• Constructs are usually described in models.
They are general and abstract by nature,
whereas test frameworks is a selection of
constructs that are relevant to a particular
purpose.
• Together these 3 levels constitute the levels
of test architecture, from the most abstract (models) to the
most specific (test specifications)

**(test specifications are discussed in

chapter 5)

Figure 4.2 ( from Fulcher and Davison, 2009: 217)

What does this model help us recognize?
• This conceptualization recognizes that there are a very large number
of language constructs and infinite language use situations, but only
some constructs or abilities will be realized in each situation. (p. 102)
• As Lado (1961:26) put it,
“The situations in which language is the medium of communication are potentially almost infinite.
No one, not even the most learned, can speak and understand his native language in any and all the
situations in which it can be used”.

• In other words, Lado realized that language was not only a tool for
humans to get things done, but the means by which personal and
cultural meaning was encoded.
Lado’s model of language use that contained
constructs (1961:6)
• In order for communication to be
successful it needs to have:
1. meaning- sounds, words,
grammatical units
2. Cultural meaning- organization of
behavior
3. Personal meaning-life experiences

=Language is an act of being. Figure 4.3 Langauge, culture and the individual (Lado 1961:6)
 How it can help us
Serves as an heuristic that can help us to
select the constructs that are relevant to
a test or assessment we need to design
for a particular context.
How it can cannot help us
Cannot tell us which forms we may wish
to test, or what types of intercultural
communication we should focus on. This
is the role of the test designer when
creating a test framework.

For example, If I wished to construct a test for shopkeepers in a tourist district, one element of intercultural
communincation that I may wish to test is the ability to establish “rapport”.
“Rapport is a customer’s perception of having an enjoyable interaction with a service provider employee, characterized
by a personal connection between the two interactants. “ (p.117; Gremler and Gwinner, 2000:92)
 Other models
Type of Model Problem Are the models described as
construct-based
models, or as behavioral-
performance?
Canale & Swain’s Model (Fig. 1. Attempt to reduce the grain and give a great deal of detail about Conclusion:
4.4 p.107) what kinds of competence or abilities are relevant to particular All models act as starting points for
kinds of interactions. our own ideas and research.
Bachman’s Model 2. Too complex We attempt to select from them the
(fig. 4.5 p. 109) 3. No appearance of interactional competence-Not really a constructs or other information that
Construct {e.g. last paragraph p. 108-110} we think may be useful to our own
situation. It is describing our own
Common European 1. People treat it as if it were really a framework- they take the
testing purpose and context, and
Framework of Reference scales and attempt to use them directly to score actual tests.
providing a rationale for the relevance
(CEFR) 2. It is indeed an abstract behavioral model
of the constructs that we have
3. Davidson and Fulcher notices that a key construct in the
selected to use, that constitutes the
literature, not mentioned in the CEFR, is the ‘ability to establish
test framework.
rapport’.
However, we have also seen that, in
4. Atheoretical
the process of constructing the test
framework-of moving between the
models and the framework- we may
very well generate our own constructs
that are particularly relevant to the
context.
Constructs are usually described in _______________.

• A. models
• B. test framework
• C. test specifications
• Answer:

A. Models
 ______________________ are the abilities of the learner
that we believe underlie their test performance, but which we cannot
directly observe.

a. Test criterion
b. Constructs
c. Test purpose
• Answer:

b. constructs
The first step in the Test Design cycle is:

• A. construct definition
• B. Test purpose
• C. test criterion
• Answer:

B. test purpose
Constructs cannot be concretely identified by observation.

a. True
b. False
• Answer:

A. True
According to Cronbach and Meehl:

Is a fish a hypothetical construct ? Explain

• According to Cronbach and Meehl a fish is not a hypothetical
construct because, despite variation in species and varieties of fish,
there is an agreed upon definition for a fish with specific
characteristics that distinguish a fish from a bird. Furthermore, a fish
can be directly observed.