Language Testing

http://ltj.sagepub.com/

Communicating the theory, practice and principles of language testing to

test stakeholders: Some reflections
Lynda Taylor
Language Testing 2013 30: 403
DOI: 10.1177/0265532213480338

The online version of this article can be found at:

http://ltj.sagepub.com/content/30/3/403

Published by:

http://www.sagepublications.com

Additional services and information for Language Testing can be found at:

Email Alerts: http://ltj.sagepub.com/cgi/alerts

Subscriptions: http://ltj.sagepub.com/subscriptions

Reprints: http://www.sagepub.com/journalsReprints.nav

Permissions: http://www.sagepub.com/journalsPermissions.nav

Citations: http://ltj.sagepub.com/content/30/3/403.refs.html

>> Version of Record - Jul 18, 2013

What is This?

Downloaded from ltj.sagepub.com at St Petersburg State University on December 15, 2013

 /$1*8$*(
Article 7(67,1*

Language Testing
30(3) 403­–412
Communicating the theory, © The Author(s) 2013
Reprints and permissions:
practice and principles of sagepub.co.uk/journalsPermissions.nav
DOI: 10.1177/0265532213480338
language testing to test ltj.sagepub.com

stakeholders: Some reflections

Lynda Taylor
University of Bedfordshire, UK

Abstract
The 33rd Language Testing Research Colloquium (LTRC), held in June 2011 in Ann Arbor,
Michigan, included a conference symposium on the topic of assessment literacy. This event brought
together a group of four presenters from different parts of the world, each of whom reported
on their recent research in this area. Presentations were followed by a discussant slot that
highlighted some thematic threads from across the papers and raised various questions for the
professional language testing community to consider together. One point upon which there was
general consensus during the discussion was the need for more research to be undertaken and
published in this complex and challenging area. It is particularly encouraging, therefore, to see a
coherent set of studies on assessment literacy brought together in this special issue of Language
Testing and it will undoubtedly make an important contribution to the steadily growing body of
literature on this topic, particularly as it concerns the testing of languages. This brief commentary
revisits some of the themes originally raised during the LTRC 2011 symposium, considers how
these have been explored or developed through the papers in this special issue and reflects on
some future directions for our thinking and activity in this important area.

Keywords
Assessment literacy, language assessment literacy, test stakeholders

LTRC 2011 in Michigan celebrated 50 years since the publication in 1961 of the
seminal work in our field by Professor Robert Lado, one-time Director of the English
Language Institute at the University of Michigan. His Language Testing volume
helped to inform and shape subsequent research and practice in our field perhaps
more than most publications. Although Lado himself made no explicit reference
in his book to assessment literacy (AL) or language assessment literacy (LAL), it

Corresponding author:
Lynda Taylor, Senior Lecturer in Language Assessment, Centre for Research in English Language Learning
and Assessment (CRELLA), University of Bedfordshire, 47 Montague Road, Cambridge, CB4 1BU, UK.
Email: lynda.taylor@beds.ac.uk

Downloaded from ltj.sagepub.com at St Petersburg State University on December 15, 2013

404 Language Testing 30(3)

could be argued that the publication was in its essence an exercise in developing
assessment literacy, particularly as far as the testing of languages was concerned.
Though he did not use the term ‘assessment literacy,’ Lado was quite clear about
the target readership for his book and its intended function. He was writing for a
variety of different audiences who would be involved in language teaching and/
or learning, who would be making use of language tests and who would, therefore,
need to understand something about language testing. The volume’s subtitle – The
Construction and Use of Foreign Language Tests – confirms that Lado was keen
to target not just ‘test makers’ (though these undoubtedly constituted an important
audience for him) but a much wider range of key stakeholder constituencies
including: existing teachers of foreign languages; prospective language teachers
and, presumably, those in training; linguists and language specialists; teachers; and
graduate students. Lado articulated his aims in writing his book as follows: “so that
increasingly we may speak on the basis of knowledge rather than from opinions
and hypotheses alone valuable as they are at proper stages in the development of
man’s [sic] thought” (1961, pp. vii–viii). His goal seems to have been to disseminate
language testing knowledge, skills and competence at a time when “great advances
in transportation and communication” were stimulating a need for “more effective
teaching of languages” along with the need for “more effective testing of their use”
(1961, pp. 1–2). Reading Lado’s words from half a century ago, it is tempting to
think that little has changed in some respects; the debate over what constitutes the
effective teaching and testing of languages continues to this day in a world that is
still characterized by globalization and technological advances.

An era of literacy or ‘literacies’

There is, however, one aspect of our world that has changed significantly. We seem to
have entered an era in which it is now necessary to develop ‘multiple literacies’ as indi-
viduals and as communities, and we find ourselves under pressure to acquire an ever
expanding body of knowledge, skills and competence relating to a growing number of
domains in everyday life; this includes the language and discourse patterns belonging to
these domains. Brumfit (2010, p. 13) described this contemporary trend as follows:
“recent changes in global economics, politics and language use have placed the issue of
interpreting multiple linguistic identities in the centre of our concerns.”
The traditional understanding of literacy as ‘the ability to read and write’ expanded
many decades ago to embrace not only competency in reading and writing but also the
critical and effective use of these skills in people’s lives, as well as the use of language
(both written and oral) for specific purposes and in specific domains. Thus the notion of
literacy began to take on a sociocultural as well as a functional dimension. Since then,
the notion of ‘academic literacy’ has become fairly well-established in the context of
studying in higher education. Hyland and Hamp-Lyons (2002, p. 4) applied the term
‘academic literacy’ to “the complex set of skills (not necessarily only those relating to the
mastery of reading and writing) which are increasingly argued to be vital underpinnings
or cultural knowledge required for success in academic communities.” (See also the col-
lection of papers on this topic edited by Blue, 2010.)

Downloaded from ltj.sagepub.com at St Petersburg State University on December 15, 2013

Taylor 405

Other notions of literacy to have emerged, mainly since the late 20th century, are simi-
larly metaphorical in nature, often associated with education or with society more
broadly. Teachers and lecturers in schools and universities, for example, sometimes com-
plain that their students fail to acquire the ‘biblical literacy’ (a form of ‘cultural literacy’)
that is fundamental for a full understanding and appreciation of so much Western litera-
ture, art and music from earlier centuries. Where once this knowledge and understanding
could be generally assumed among students within most Western educational contexts,
specially designed resources may be needed nowadays to familiarize students with the
great religious themes that once inspired writers, artists, composers and others in Western
culture and civilization (see Dyas & Hughes, 2005).
At a more applied level, notions of ‘technological literacy’ (e.g., to operate standard
household appliances) and ‘computer literacy’ are now acknowledged throughout soci-
ety; and in a technological age, we are sensitive to the reality that those who are not
computer-literate risk becoming increasingly marginalized or disadvantaged as a result.
Some social commentators express concern at increasingly low levels of ‘civic literacy’
in contemporary society; that is, citizens are seen as lacking the knowledge and skills
needed to participate effectively in the community, government and politics, leading to a
potential threat to democracy and good governance.
Other metaphorical expressions to have entered common usage in recent years include
‘risk literacy,’ ‘health literacy,’ ‘media literacy’ and ‘emotional literacy.’ In each case, the
use of the term ‘literacy’ stands for know-how and awareness linked to the preceding item
in the multi-word phrase. The focus is on the ability to understand the content and dis-
course associated with a given domain or activity and on being able to engage with and
express oneself appropriately in relation to this.
It is hardly surprising, then, that ‘assessment literacy’ (AL) has been added to the
growing list of ‘literacies’ to be acquired in contemporary life, together with ‘language
assessment literacy’ (LAL) as a potentially subordinate or overlapping category. Both
are relatively new fields of as far as theoretical and empirical research is concerned,
but both are generally acknowledged today as an important focus for attention, debate,
policymaking and action in education and wider society. Some writers highlight an
imperative for work in this area in response to the huge growth of testing and assess-
ment worldwide and the increasing number of people involved in it; the role of lan-
guage assessment in particular has expanded in education and wider society as a result
of globalization and migration (for a fuller discussion see Taylor, 2009). Outcomes
from empirical research investigating the nature and development of language assess-
ment literacy are urgently needed not just to inform and underpin existing policy and
practice but also to inspire and shape new and innovative initiatives for disseminating
core knowledge and expertise in language assessment to a growing range of test
stakeholders.

Research topics and questions

At the LTRC 2011 symposium, a set of wh- questions was used as a convenient spring-
board for identifying relevant research topics and questions concerning AL/LAL:

Downloaded from ltj.sagepub.com at St Petersburg State University on December 15, 2013

406 Language Testing 30(3)

• Who are the key stakeholder groups that need to develop AL/LAL?
• What sort of content input is needed for developing AL/LAL?
• Where are the specific domains and contexts in which this needs to be done?
• When is the best time for this to be done?
• What methods or approaches are likely to be most effective?

The papers presented in the LTRC 2011 symposium demonstrated how a broad range
of research methodologies (quantitative, qualitative and mixed methods) could be suc-
cessfully employed to investigate such questions. Subsequent discussion during the sym-
posium highlighted four particular areas in which further research into AL and LAL
might generate important insights for our field. The four areas related to the following:

1) How we understand and define the AL/LAL construct

How should we distinguish assessment literacy from language assessment literacy? Is it
possible to think and speak in terms of ‘a core’ relating to the former generic concept, and
for the latter some sort of ‘bolt-on’ component relating to the teaching, learning and
assessment of language? Or should we be thinking in terms of a more integrated con-
struct? Is language assessment literacy primarily knowledge-based? Is it a skill-based
competence? Or is it a set of principles? Or is it something that integrates all three, as
Davies (2008) seemed to suggest?

2) Our language and discourse when engaging with a non-specialist

audience
Would it help to define our terms and phraseology more narrowly and in ways that are
accessible to a broader and non-specialist audience? Even if the concepts of assessment
literacy and language assessment literacy are still evolving – in an emergentist sense –
we nevertheless need to develop a discourse and a way of communicating that reflects
and meets the needs of differing stakeholder groups and varying professional contexts.
In his 1961 volume, Robert Lado was keenly aware of the need for communicative effec-
tiveness if his book was to speak to its target audience and achieve its intended objec-
tives. He wrote as follows in the preface to his volume: “The style has been kept as
non-technical as possible because of the variety of backgrounds of the intended readers.
Linguistic terminology is not always familiar to the average teacher and test maker; and
testing terminology is not often familiar to language teachers and linguists” (1961,
p. vii). Once again, it is tempting to feel that little has changed in this regard over half a
century; in fact, it could be argued that the terminology and discourse used in applied
linguistics and language testing today risk being even more highly technical and thus
inaccessible to the non-specialist than they were 50 years ago.

3) How we identify, evaluate and respond to varying user needs

Growing awareness and understanding of test impact since the early 1990s, particularly
its multidimensional nature within education and wider society, has made it clear that

Downloaded from ltj.sagepub.com at St Petersburg State University on December 15, 2013

Taylor 407

language testers and test developers need to be sensitive to many different types of test
stakeholder and the varying ways in which they find themselves engaging with and
understanding assessment issues. Identifying the range of relevant stakeholders and eval-
uating their specific needs in relation to what test scores mean in their context and, con-
sequently, how scores can or cannot be used, is becoming a priority in a world where
assessment occupies such a central role. Ideally the promotion and development of
assessment literacy will be achieved ‘by design’ rather than being a corrective after-
thought. This means that engagement with test stakeholder groups needs to take place at
the outset of any project to develop a new or revised test or testing system, not simply
initiated at a point where the test is ready to be implemented. Similarly, some degree of
familiarization with the principles and practice of assessment will ideally be embedded
from an early stage in the training of language teachers, not simply offered as a ‘bolt-on’
option. AL/LAL development activities may need to be integrated within professional
development programmes or briefing sessions for other stakeholder groups, such as civil
servants or politicians. In each case, the primary content relevant to the needs of that
stakeholder group (e.g., will it be measurement theory? practical know-how? ethical
principles?) has to be extracted and translated into a language that particular group can
access and understand.

4) Our understanding of how AL/LAL grows and matures over time

Relatively little is understood still about how AL or LAL develops within individuals or
across stakeholder groups who are not trained assessment professionals. Is it possible,
for example, to think of someone moving from being a ‘novice’ in AL/LAL to achieving
a level of proficiency and, if so, what might that progression curve look like? Are there
recognizable and meaningful steps of achievement along the way that can be discerned
and even accredited in some formal or informal way? Might there be a place for certifi-
cating language assessment literacy in some contexts? And could there be a role in this
for a professional language testing organization at national, regional or international
level?

The contribution of this special issue to the ongoing AL/

LAL debate
The five papers published in this special issue echo various aspects of the four areas
identified above, and they build upon the earlier body of research reported at the LTRC
2011 assessment literacy symposium. In some cases the studies reported here confirm or
replicate previous findings. In other cases they explore new territory and contexts, help-
ing to broaden our understanding and advance our thinking in this area.
The papers by Scarino, by Malone, by Jeong and by O’Loughlin all touch upon the
central question of how we understand and define the AL/LAL construct. They highlight
the importance of balancing theoretical assessment knowledge with practical know-how,
while still allowing some place in the mix for attention to ethical principles. The papers
also confirm the impact on assessment literacy of the professional beliefs, perceptions
and practices which are invariably located within any local community of practice, for

Downloaded from ltj.sagepub.com at St Petersburg State University on December 15, 2013

408 Language Testing 30(3)

example language teachers, course instructors, university administrators, government

policy makers. The authors challenge us to consider more deeply how to take account of
these factors and whether they need to be integrated within the AL/LAL construct.
The papers by Jeong, by O’Loughlin and by Pill and Harding touch upon issues
associated with the language and discourse of language testing, sometimes in terms of
its overly technical nature for a non-specialist audience, or in relation to its tendency to
generate misconceptions regarding measurement precision and test score infallibility.
They challenge us to seek new and effective ways of communicating the theory and
practice of language testing in transparent and accessible terms. This may require inno-
vative modes of presentation tailored for specific audiences and contexts, for example,
a specially designed online module for university administrators or immigration person-
nel on how to use test scores safely as part of complex decision-making processes, or an
animated cartoon for teachers, parents and children showing how assessment supports
successful teaching and learning. Creative and imaginative approaches using web-based
tools are likely to be far more effective at reaching some key stakeholder groups than
the print-based approach (e.g., paper leaflets or test handbooks) which language testers
and test developers have traditionally adopted with stakeholders. Furthermore, even
though much of the relevant information that stakeholders need can be found some-
where on test producers’ websites, its presentation and packaging too often reflects the
documentation-based approach of the past, where the burden is on the stakeholders to
find, read and understand the information that is relevant to them. Brumfit (2010, p. 14)
speculates that literacy has become another term for ‘interpretation’ “in a world of great
and greater communication, we are presented with more and more data to interpret.”
This view of the world helps to explain the growing need for assessment literacy, but it
also implies that those who need to develop such literacy are likely to have less time and
energy to spend seeking out what is relevant and useful to their requirements; the onus
of responsibility for making key information more accessible must surely lie with those
who already know where it is located.
All five papers alert us to the growing diversity of individuals and collective bodies,
typically referred to as stakeholders, all of whom require some degree of AL/LAL though
not always at the highest level of expertise. There have been various attempts to list cat-
egories of test stakeholder (e.g., Rea-Dickins, 1997; Taylor, 2000; Saville, 2009) and it
is self-evident that different stakeholder groups experience differing degrees of interest
in a particular test (or in testing more generally) according to their level of assessment
involvement. What becomes much clearer from the papers in this special issue, however,
is the extent to which these different stakeholder groups need differential levels of assess-
ment literacy according to their specific roles and responsibilities. We might think of
some stakeholders as being closer to the heart of assessment – the ‘assessment core’;
such groups are likely to require comprehensive training in measurement theory, techni-
cal know-how and ethical principles. Other stakeholder groups might place themselves
some distance away from the core or closer to the periphery; they may be interested in or
concerned with a particular dimension of assessment use, for example a classroom-based
learning context, a university admissions system or a public forum such as a parliamen-
tary enquiry. Instead of simply listing a comprehensive range of potential test stakehold-
ers, perhaps we need a more sophisticated model such as that proposed in Figure 1 where

Downloaded from ltj.sagepub.com at St Petersburg State University on December 15, 2013

Taylor 409

concentric circles expand outwards from an ‘expert core’ of assessment knowledge, skill
and principles, with each successive ring (or segment of a ring) representing the level of
content/input that is required to meet the needs of a particular set of stakeholders.
Taking a more discriminating perspective in this way may assist us in identifying the
specific range and depth of testing expertise that is relevant when selecting the content
or input that will best match each group’s needs. It should also make the process of
developing AL/LAL more manageable and achievable. Not everyone needs to know or
be able to do everything to the same level. What is important is that they should be
competent in the knowledge, skills and understanding necessary for their context of
activity. The resulting profile for each stakeholder group is likely to look somewhat dif-
ferent. For example, a profile for test writers may cover a wide range of content dimen-
sions fairly evenly and in some depth. A profile for classroom language teachers,
however, may end up focusing strongly on the practical know-how needed for creating
tests but have a much lighter focus on measurement theory or ethical principles; the lat-
ter may need to be touched upon only briefly at a surface level. While a profile for
university administrators will address those aspects of the assessment literacy construct
that relate to understanding the nature of test instruments and the meaning of their
scores for decision-making purposes, other aspects such as how to construct and vali-
date tests need not receive much attention.
Figure 2(a–d) attempts to illustrate what differential assessment literacy might look
like for these three groups and for the community of professional language testing
experts. It should be noted that the labelled dimensions on the eight axes (i.e. knowledge
of theory, technical skills, etc.) are hypothesized from the discussion of possible AL/
LAL components across various papers in this special issue, while the values (i.e. 0–4)
are hypothesized according to the different stages of literacy suggested by Pill and
Harding. The diagrams are for illustrative purposes only, to show how it might be

Figure 1.  Levels of AL/LAL differentiated according to stakeholder constituency.

Downloaded from ltj.sagepub.com at St Petersburg State University on December 15, 2013

410 Language Testing 30(3)

(a) Knowledge of (b) Knowledge of

theory theory
4 4
Scores and 3 Scores and
Technical skills 3 Technical skills
decision making decision making
2 2
1 1
Personal Principles and Personal Principles and
beliefs/atudes 0 concepts 0
beliefs/atudes concepts

Language Language
Local pracces Local pracces
pedagogy pedagogy

Sociocultural Sociocultural
values values
(c) Knowledge of (d) Knowledgeof
theory theory
4 4
Scores and Scores and
3 Technical skills 3 Technical skills
decision making decision making
2 2
1 1
Personal Principles and Personal Principles and
beliefs/atudes 0 concepts 0
beliefs/atudes concepts

Language Language
Local pracces Local pracces
pedagogy pedagogy
Sociocultural Sociocultural
values values

Figure 2.  Differential AL/LAL profiles for four constituencies.

(a) Profile for test writers.
(b) Profile for classroom teachers.
(c) Profile for university administrators.
(d) Profile for professional language testers.

possible to conceptualize and represent differential AL/LAL; the actual characterization

for each group is naturally open to debate.
In relation to understanding how AL/LAL develops and matures over time, the Pill
and Harding paper is particularly insightful in making a useful interdisciplinary link with
understandings of literacy drawn from other educational fields, for example scientific or
mathematical literacy. As the authors suggest, this approach helps move us away from a
notion of AL/LAL as a polarized dichotomy (i.e. one is either literate or illiterate in this
area) towards a continuum which identifies and describes progressive stages of literacy
along the way, for example nominal, functional, procedural/conceptual and multidimen-
sional. This characterization merits further thought and exploration to see if it might
provide us with a ‘literacy ladder’ for assessment; that is, a way of conceptualizing AL/
LAL progression which could assist in designing modular courses in language testing
and even in certificating levels of achievement as part of a broader professional develop-
ment programme for certain stakeholder groups if appropriate to do so.
While Fulcher’s (2012) expanded working definition of language assessment literacy
helpfully draws together a broad range of elements that could be key components of the
construct, it does not address the issue of the degree or depth that might be necessary for

Downloaded from ltj.sagepub.com at St Petersburg State University on December 15, 2013

Taylor 411

any of the elements depending upon the nature and extent of the stakeholder’s involve-
ment in assessment. Figure 2(a–d) attempts to operationalize the range of key compo-
nents making up the AL/LAL construct and to grade them on some sort of a continuum
of depth/intensity.
Finally, it is worth noting that several of papers in this special issue make it clear that
language assessment literacy is not necessarily a value-free concept. It is important to
consider, therefore, what or whose values shape any programme to develop language
assessment literacy. Two of the papers commented on a certain tension experienced
between the perspectives of ‘professional language testers’ and of ‘non-language testers.’
This observation resonates with the tension commented on by Brumfit (2010) in his dis-
cussion of language education in British society. He described a tension between ‘exper-
tise and popular understanding’:

there has been an increase of linguistically expert academics at the same time as experts of all
kinds have been increasingly marginalised by governments (see Brumfit 2001, Chapter 6).
Several of the tensions mentioned above reflect differences between specialist and popular
views of language and its role. (Brumfit, 2010, p. 21)

Brumfit’s comments referred primarily to the tension that arose in the late 1990s between
specialist and popular views concerning primary and secondary language education
issues in the UK. A similar tension can be discerned in the world of language testing;
there is always the danger that testing experts risk adopting a somewhat superior attitude
as far as the value and significance of their professional expertise is concerned but which
leads to them being marginalized in a way that is counterproductive for all concerned.
The development of AL/LAL across a range of stakeholder groups is likely to be most
successful when it reflects a dynamic and iterative process informed by a collaborative
ongoing dialogue taking place at the interface between language testing experts and non-
specialist test stakeholders.
In her Guest Editorial to this valuable special issue, Inbar-Lourie highlights four
strands to emerge from the authors’ contributions: first, the breadth of scope in terms of
how the AL/LAL construct can be interpreted and defined; second, the nature and extent
of the language component within an AL/LAL construct; third, the site of authority for
determining the precise nature of AL/LAL; and fourth, what constitutes the essential
core components of AL/LAL. This final paper has offered some additional reflections on
these strands and has attempted to suggest some practical methods for conceptualizing
range and depth of content and for differentiating profiles of stakeholder need and levels
of proficiency. Hopefully, some of these ideas can be refined more fully to reveal new
insights and approaches in this important area for our field.

Funding
This research received no specific grant from any funding agency in the public, commercial or
not-for-profit sectors.

References
Blue (2010). (Ed.) Developing academic literacy. Bern, Switzerland: Peter Lang.

Downloaded from ltj.sagepub.com at St Petersburg State University on December 15, 2013

412 Language Testing 30(3)

Brumfit, C (2010). Literacy or literacies? Academic identities in a language-sharing world. In

G. Blue (Ed.), Developing academic literacy. Bern, Switzerland: Peter Lang.
Davies, A. (2008). Textbook trends in teaching language testing. Language Testing, 25(3), 327–347.
Dyas, D., & Hughes, E. (2005). The Bible in Western culture: The student’s guide. Abingdon,
Oxon: Routledge.
Fulcher, G. (2012). Assessment literacy for the language classroom, Language Assessment Quar-
terly, 9(2), 113–132.
Hyland, K., & Hamp-Lyons, L. (2002). EAP: Issues and directions. Journal of English for Aca-
demic Purposes, 1(1), 1–12.
Lado (1961). Language testing: The construction and use of foreign language tests. London: Longman.
Rea-Dickins, P. (1997). So, why do we need relationships with stakeholders in language testing?
A view from the UK. Language Testing, 14(3), 304–314.
Saville, N. (2009). Developing a model for investigating the impact of language assessment within
educational contexts by a public examination provider. PhD dissertation, University of Bed-
fordshire.
Taylor, L. (2000). Stakeholders in language testing. Cambridge ESOL Research Notes 2, 2-4.
Taylor, L. (2009). Developing assessment literacy. Annual Review of Applied Linguistics, 29,
21–36. Cambridge: Cambridge University Press.

Downloaded from ltj.sagepub.com at St Petersburg State University on December 15, 2013