FACULTY OF LANGUAGES, LITERATURES, AND CIVILISATIONS

DEPARTMENT OF ENGLISH
LICENCE 3
Specialty : Language Didactics (Didactique de la Langue Anglaise) DLA
Academic year : 2018-2019
Semester6
Course Title : Materials evaluation
Prerequisite : Theory of materials
Duration : 25 hours Lecture
Instructor : Pr AGBA Yoboue Kouadio Michel
Course Aim and Outcomes
Aim
The aim of this course is to allow the students to learn about the guidelines for
second/foreign language materials evaluation.
Specific Learning Outcomes
The learning outcomes the course is intended to produce are as follows :
- The students will be able to set some guidelines for second/foreign language
materials evaluation ;
- The students will be able to carry out the task of materials evaluation basing on some
set criteria.

Course Schedule

DATE / TOPIC Tasks/activities Materials/Sou

SESSION rces
Session 1 Materials evaluation : Nature and Through explanations in Booklet on
need plenary the students are materials
enlightened on what evaluation
materials evaluation is and
why it is importance

Session 2 Materials evaluation methods: Through explanations in Booklet on

- macro vs micro evaluation plenary the students are materials
- predictive vs retrospective enlightened on the principles evaluation
evaluation and procedures of some
- pre-use, in-use and post- evaluation methods
use evaluations
- extranal vs internal
evaluation
- impressionistic vs
systematic evaluation

1
Session 3 Frameworks and checklists for
materials evaluation
Through explanations in Booklet on
plenary the students are materials
enlightened on some major evaluation
frameworks and checklists
for materials evaluation

Session 4 developing criteria for materials Through explanations in Booklet on

evaluation plenary the students are materials
enlightened on the process evaluation
of setting some criteria for
materials evaluation
Session 5 Research session The students do personal The Internet,
research activities on course Books and
contentAssignment : Articles on
Evaluating a second/foreign materials
language teaching material. evaluation
(To be handed over no later
than

two weeks after the end of

course)

Grading Procedures: Grades for the different credit will be based on:

 Assignment (40 %)
 Final exam(60 %)
 Passing grade: 10/20

Readings
- Candlin, C. & Breen, M. (1979) Evaluating, adapting and innovating language teaching
materials, In C. Yorio, K. Perkins and J. Schacter (Eds.), On TESOL ’79: The learner in focus.
(pp. 86-108). Washington, DC: TESOL.
- Cunningsworth, A. (1984). Evaluating and selecting EFL teaching material. London, UK:
Heinemann.
- Cunningsworth, A. (1995). Choosing Your Coursebook, Oxford: Heinemann
- Ellis, R. (1997). The empirical evaluation of language teaching materials English Language
Teaching Journa1, 51( 1) 36-42
- Hutchinson, T. (1987). What’s underneath? An interactive view of materials development. In
L. E. Sheldon (Ed.), ELT Textbooks and Materials: Problems in evaluation and development
(pp. 37-44). ELT Documents 126. London, UK: The British Council.
- Johnson, R. (1986). “Selecting a coursebook: a realistic approach”, in S. Holden(ed.), (1986),
Techniques of Teaching From Theory to Practice, Papers from the 1985 Bologna conference.
Oxford. Modern English Publications/British Council 54-7

2
- Jolly, D., & Bolitho, R. (1998). A framework for materials writing. In B. Tomlinson (Ed.),
Materials Development for Language Teaching (pp. 90-115). Cambridge, UK: CUP.
- Hutchinson, T. (1987), `What's underneath?: an interactive view of materials evaluation', in L
Sheldon (ed. ) (1987), ELT Textbooks and Materials: Problems in Evaluation and
Development, ELT Documents 126, Oxford: Modern English Publications/the British Council,
37-44
- McDonough, J., Shaw, C. and Masuhara, H., 2013. Materials And Methods In ELT: A Teacher's
Guide. 3rd ed. West Sussex: Wiley-Blackwell.
- McDonough, J., & Shaw, C. (2003). Materials and methods in ELT: A teacher's guide. Malden,
MA: Blackwell Pub.
- McGrath, I. (2002). Materials Evaluation and Design for Language Teaching. Edinburgh
Textbooks in Applied Linguistics. Edinburgh University Press
- Tomlinson B. (2003. Materials evaluation. In B. Tomlinson (ed) Developing materials for
language teaching. London; continuum, 15-36)
- Tomlinson, B., 2003. Developing Materials For Language Teaching. London: Bloomsbury.
- Tomlinson, B., 2011. Materials Development In Language Teaching. 2nd ed. Cambridge:
Cambridge University Press.

3
 MATERIALS EVALUATION: Nature and need

 What are materials?

" Materials" is a common term used to describe the tangible teaching and learning aids
whether print, non-print, digital or any combination thereof, used by teachers and learners
in instructional programs (Pender County Schools, 2012). They "can be in the form (…) of a
textbook, a workbook, a cassette, a CD-ROM, a video, a photocopied hand-out, a newspaper,
a paragraph written on a whiteboard: anything which presents or informs about the
language being learned" (Tomlinson, 2011, xiii–xiv) and which is used in the classroom to
facilitate the process of teaching and learning (Morales, 2012 as cited by Sesma and Fontes,
2016, p. 1). Basically then, materials serve as: (1) a resource for language presentation
materials (spoken and written), (2) a source of activities for learner practice and
communicative interaction, (3) a reference source for learners on grammar, vocabulary,
pronunciation, (4) a source of stimulation and ideas for classroom activities, (5) a syllabus
(where they reflect learning objectives that have already been determined), (6) and a
support for teachers (Cunningsworth, 1995, p. 7). As such, they can be informative
(informing the learner about the target language), instructional (guiding the learner in
practicing the language), experiential (providing the learner with experience of the language
in use), eliciting (encouraging the learner to use the language) and exploratory (helping the
learner to make discoveries about the language) (Tomlinson, 2012, p.143). It therefore goes
without saying that materials "the center of instruction and one of the most important
influences on what goes on in the classroom"(Kitao 1998, p. 1)
 The ‘what’ and ‘why’ of materials evaluation
Tomlinson’s (2003) describes materials evaluation as “ a procedure that involves
measuring the value (or potential value) of a set of learning materials’. It aims at examining
whether materials reflect the needs of the learners, the aims, methods and values of a
specific teaching program (Cunnings worth, 1995: 7) . It is about ‘checking all the sources of
information, updating outdated information and providing more contextually relevant
information’ ( course developer’s toolkit, p.12), it includes analyzing the tasks and contents
to find out whether they are appropriate for a particular group of learners, and to decide to
adopt, update or substitute a textbook after having identified its specific weaknesses and
strength (cunningsworth, 1995; Riazi, 2003). Thus materials evaluation is carried out in order
to ‘’ to see 'inside' the materials and to take more control over their design and use”. (
Littlewood, 1998 in Khodabakhshi 2014) , to measure the value (or potential value) of a set
of learning materials” (Tomlinson (2003, p.15). It is finally “an activity of gathering
information to be used in making educational decisions” (Genesee and Upshur, 1999, p.140)

4
 MATERIALS EVALUATION METHODS

There are miscellaneous proposed approaches and methods. to materials evaluation. Let us
summarize some of the most widely-used.
 predictive and retrospective evaluations
A predictive evaluation is designed to make a decision regarding what materials to use.
Teachers who are required to carry out a predictive evaluation determine which materials
are best suited to their purposes. Once the materials have been used, further evaluation
may be conducted to find out whether the materials have worked out for them, and this
type of evaluation is called retrospective evaluation (1997). such an evaluation is suitable
for teacher to determine whether it is valuable to use materials again, which activities 'work'
and which do not, and how to improve the materials to make them more efficient for future
use. So, textbook analysis can be carried out in two directions: the analysis of selected
textbooks in order to determine whether they are suitable or not for classroom usage, i.e.
predictive analysis, and the analysis of textbooks that are already in use, i.e. retrospective
analysis (Skopinskaja, 2003, p. 39).
Using a slightly different terminology, Cunningsworth (1995, p. 15) describes how textbooks
can be evaluated for potential and for suitability. In the first case, the textbook is assessed
according to a predefined set of criteria, but without any particular learners or courses in
mind. This approach is more frequent when a new textbook is published and may result in
book reviews being written. However, since classroom situations introduce many unknown
elements and require constant adapting on the side of the teacher, this type of analysis has
its flaws as the textbook cannot be matched to a particular class or group of students. On
the other hand, the evaluation of textbooks for suitability is always done with a certain
audience in mind. In this case, as Cunningsworth (1995, p. 15) stipulates, there are very
specific learning objectives, learners’ backgrounds, resources available, etc. In practice, it can
be said that both types of analysis are done equally frequently and sometimes the same
textbook is analyzed first for potential (before it is used in class, as the teacher is selecting a
textbook for the upcoming academic year) and then for suitability (while it is used in class, so
the teacher can confirm that the choice he/she made was correct). In case the textbook is
assessed positively in both of these situations, then it will be used in future; in case the
evaluation for suitability proves that the textbook does not comply with all the needs of the
learners, then the teacher has a clear signal that another textbook should be chosen.
 Macro and micro evaluations
Ellis (1997) suggests the micro-evaluation versus macro-evaluation. While in the former type
the teacher chooses a specific teaching task of interest and makes it undergo a detailed
empirical evaluation, the latter refers to an inclusive appraisal to understand if the whole
textbook is efficient. A micro-evaluation is characterized by a narrow focus on some specific

5
aspect of the curriculum or the administration of the program. Thus, in the case of materials,
we might ask whether a particular task is effective or efficient. This may be informed by a
desire to obtain information that will speak generally about the effectiveness and efficiency
of the materials in achieving learning goals (Tomlinson, 2003:216). Thus there is a
relationship between macro- and micro-evaluations, as macro-evaluations address every
aspect of a course/programme (i.e. both administrative and curricular matters). Micro-
evaluations of tasks can help to inform macro-evaluations of curricular matters by providing
information about the materials used, the teachers implementing the course and the
performance (and learning) of the learners. However, micro-evaluations are of value in
themselves as they provide a basis for deciding whether specific tasks work, and serve as a
source of teacher self-reflection and development.
 The Impressionistic and Systematic methods
The impressionistic method means attaining a general synopsis of the contents presented in
the coursebook (Widodo, 2015). This method deals with analyzing a textbook based on a
general impression obtained by means of checking the textbook contents in view of
organization, layout, the presented topics as well as the visuals, and so forth (Montasser,
2013). In . In practice, it allows the evaluator to quickly review content and form an opinion
on its strengths and weaknesses, its design, and its structure. Of particular interest when
reviewing content through the impressionistic method are language elements, the types of
exercises used in learning, and the author’s view of the learning process. So, This evaluation
method is predictive in nature and it is more effectively applied in the pre-use stage for
developing an early overall impression on the potential impact of content. (Ellis, 1997;
McGrath (2002). In the case of global textbooks ( ie. textbooks intended for the international
market), such an overview typically involves glancing at the publisher’s ‘blurb’( i.e. the brief
description of the book on the back cover, and the, skimming through the book looking at
organization, topic, lay out and visuals (Cunningsworth’s (1995).
The impressionstic method’s weakness lies in the fact that it may be perceived as superficial
even when findings are the result of systematic information gathering.
Another recognized method is systematic evaluation which uses checklists. Such
checklists make it easier to establish comparison among diverse materials (Montasser,
2013). Widodo (2015) highlights that by using a checklist we compare, identify, or verify. As
a rather easy method, an evaluation checklist introduces us a list of criteria considered for
any efficacious learning and/or teaching materials. Evaluators can be the teachers,
researchers, or even the students who can rate the quality of the given material according to
the mentioned criteria (Soori, Kafipur, & Soury, 2011). Thus, the systematic or checklist
method contrasts system ( and therefore ostensible objectivity) with impression ( and
implicitly subjectivity). In most literal sense, a checklist consists of a list of items which is
‘referred to for comparison, identification or verification’ (Collins English Dictionary, 1992),
the items being ‘checked off’ ( or ticked) once their presence has been confirmed.
The checklist has at least four advantages:

6
 1. It is systematic, ensuring that all elements that are deemed to be important are
considered.
2. It is cost effective, permitting a good deal of information to be recorded in a
relatively short span of time
3. The information is recorded in a convenient format, allowing for easy comparison
beaten competing sets of material
4. It is explicit, and, provided the categories are well understood by all involved in the
evaluation work for decision making.
The systematicity of a textbook is well brought out by Skierso, (19912: 440, citing Tucker
1978):
“A textbook evaluation checklist should consist of a comprehensive criteria based on the
basis linguistic, psychological, and pedagogical principles underlying modern methods of
language leering. These criteria ‘ should be discreet enough to focus attention on one
characteristic at a time or on a single group of related characteristics’ (Tucker 1978, p.
219)
However, pre-existing checklists can become out-of-date and the criteria used may not be
transparent or based on assumptions shared by everyone (McGrath, 2002). Sheldon (1988:
242) has also written how considerable modification of any set of culturally restricted
criteria is necessary to make them applicable to most local contexts. Consequently the
problem is that no set of criteria is applicable to all situations and, as Byrd (2001) says, it is
important that there is a fit between the materials and the curriculum, students and
teachers. Matthews (1985), Cunningsworth (1995) and Tomlinson (2012) have also stressed
the importance of relating evaluation criteria to what is known about the context of learning
and Makundan and Ahour (2010) in their review of 48 evaluation checklists were critical of
most checklists for being too context bound to be generalizable.
 External and internal evaluations
Tomlinson (2003) identifies two types of materials evaluation that may be carried out in two
complementary stages. First, Teachers/evaluators may want to review all the materials
that have come out during a given period of time to find materials suitable for
adoption/selection at some future date. However, there is no point in doing a full evaluation
for selection purposes if a preliminary evaluation can show that those materials will be of
little use for a particular group. Thus an external evaluation will offer a brief overview of the
materials from the outside (cover, introduction, table of contents), and may then followed
by a closer and more detailed internal evaluation.
- External evaluation
For Tomlinson (2003: 54), the aim of external evaluation is basically that of examining the
organization of the materials as stated explicitly by the author/publisher by looking at:
 the ‘blurb’, or the claims made on the cover of the teacher’s/students’ book
 the introduction and table of contents

7
that should enable the evaluator to assess what Tomlinson he calls analysis in that ‘it asks
questions about what the materials contain, what they aim to achieve and what they ask
learners to do’ (Littlejohn, 2011 makes a similar distinction). He also considers it useful to
scan the table of contents page in that it often represents a ‘bridge’ between the external
claim made for the materials and what will actually be presented ‘inside’ the materials
themselves. At this stage there is a need to consider why the materials have been produced.
Presumably because the author/publisher feels that there is a gap in the existing market that
these materials are intended to fill: it will be necessary to investigate this further to see
whether the objectives have been clearly spelt out.
As an illustration, here is an example of one such ‘blurb’ taken from a well-known EFL
textbook published in 2012:
 … an integrated skills series which is designed to offer flexibility with different
teaching and learning styles. Fun for learners to use and easy for teachers to adapt…
• Fully integrated grammar, skills and lexical syllabuses provide a balanced learning
experience
 Engaging topics motivate students and offer greater personalization • A wide range
of approaches exploit different learning styles
 Clearly structured grammar presentations are reinforced with extensive practice
 Contextualized vocabulary focuses on authentic real-world language
 A variety of listening and speaking activities develop learning fluency
 Learner training throughout the Student’s Book and Workbook maximizes skills
development.
Other factors to take into account at this external stage are as follows:
 Are the materials to be used as the main ‘core’ course or to be supplementary to it?
 Is a teacher’s book in print and locally available?
 What visual material does the book contain (photographs, charts, diagrams) and is it
there for cosmetic value only or is it integrated into the text?
 Is the layout and presentation clear or cluttered? Is the material too culturally biased
or specific?
 Do the materials represent minority groups and/or women in a negative way? Do
they present a ‘balanced’ picture of a particular country/society?
 What is the cost of the inclusion of digital materials (e.g. CD, DVD, interactive games,
quizzes and downloadable materials from the web)? How essential are they to
ensure language acquisition and development?
 The inclusion of tests in the teaching materials (diagnostic, progress, achievement);
would they be useful for your particular learners?
After completing this external evaluation, and having funds and a potential group of learners
in mind, we can arrive at a decision as to the materials’ appropriacy for adoption/selection
purposes.

8
 - Internal evaluation
Internal evaluation refers to an in-depth investigation into the materials. The essential issue
at this stage is to analyze the extent to which the aforementioned factors in the external
evaluation stage match up with the internal consistency and organization of the materials as
stated by the author/publisher. In order to perform an effective internal inspection of the
materials, we need to examine at least two units (preferably more) of a book or set of
materials to investigate the following factors:
 The presentation of the skills in the materials.
 The grading and sequencing of the materials.
 Where reading/‘discourse’ skills are involved, is there much in the way of appropriate
text beyond the sentence?
 Where listening skills are involved, are recordings ‘authentic’ or artificial?
 Do speaking materials incorporate what we know about the nature of real interaction
or are artificial dialogues offered instead?
 The relationship of tests and exercises to (1) learner needs and (2) what is taught by
the course material.
 Do you feel that the material is suitable for different learning styles?
 Are the materials engaging to motivate both students and teachers alike, or would
you foresee a student/teacher mismatch?
In the internal evaluation stage we need to examine the following criteria: the treatment
and presentation of the skills, the sequencing and grading of the materials, the type of
reading, listening, speaking and writing materials contained in the materials, appropriacy
of tests and exercises, self-study provision and teacher–learner ‘balance’ in use of the
materials.
 Pre-use , in-use and post-use evaluation
Cunningsworth (1995) and Tomlinson (1998) categorize the evaluation into pre-use, while-
use or in-use, and after-use types which is dependent on the aims of evaluation.
- Pre-use evaluation:
For Tomlinson (1998) involves making predictions about the potential value of materials for
their users. He claims that pre-use evaluation is impressionistic and consists of a teacher
flicking through a book to gain a quick impression of its potential value. Cunningsworth
(1995) accentuates that the objective of a pre-use evaluation is to examine the course
book’s future or potentiality. Pre-use evaluation is called predictive by Ellis (1997).
- In-use evaluation
Tomlinson (1998) mentions that while-use evaluation measuring the value of materials while
using them or while observing them being used. It can be more objective and reliable than
pre-use evaluation as it makes use of measurement rather than prediction. However, it is
limited to measuring what is observable (e.g. ‘Are the instructions clear to the learners?’)
and cannot claim to measure what is happening in the learners’ brains. It can measure short-
9
term memory through observing learner performance on exercises but it cannot measure
durable and effective learning because of the delayed effect of instruction. Exactly what can
be measured in a whilst-use evaluation is controversial but I would include the following:
- Clarity of instructions
- Clarity of layout
- Comprehensibility of texts
- Credibility of tasks
- Achievability of tasks
- Achievement of performance objectives
- Potential for localization
- Practicality of the materials
- Teachability of the materials
- Flexibility of the materials
- Appeal of the materials
- Motivating power of the materials
- Impact of the materials
- Effectiveness in facilitating short-term learning
Most of the above can be estimated during an open-ended, impressionistic observation of
materials in use but greater reliability can be achieved by focusing on one criterion at a time
and by using pre-prepared instruments of measurement. . Information collected can serve
the following purposes:
 To provide feedback on how well the book works in practice and how effectively it
achieves it aims.
 To document effective ways of using the textbook and assist other teachers in using
it.
 To keep a record of adaptations that were made to the book.

 Post-use evaluation
Finally, after-use evaluation is proposed to determine what happened on account of using
the instructional materials (Tomlinson, 1998), and Cunningsworth (1995) names this type as
post-use evaluation affirming that it allows retrospective evaluation. In this regard, Post-use
evaluation seems to be beneficial for recognizing the merits and demerits of a certain
textbook following its continual use. Post-use evaluation is also valuable in determining if
the evaluated textbook is good enough for future use. It can answer such important
questions as:
- What do the learners know which they did not know before starting to use the
materials?
- What do the learners still not know despite using the materials?
- What can the learners do which they could not do before starting to use the
materials?
- What can the learners still not do despite using the materials?

10
 - To what extent have the materials prepared the learners for their examinations?
To what extent have the materials prepared the learners for their postcourse
- use of the target language?
- What effect have the materials had on the confidence of the learners?
- What effect have the materials had on the motivation of the learners?
- To what extent have the materials helped the learners to become independent
learners?
- Did the teachers find the materials easy to use?
- Did the materials help the teachers to cover the syllabus?
- Did the administrators find the materials helped them to standardize the teaching in
their institution?
In other words, it can measure the actual outcomes of the use of the materials and thus
provide the data on which reliable decisions about the use, adaptation or replacement of the
materials can be made. Ways of measuring the post-use effects of materials include:
- tests of what has been ‘taught’ by the materials;
- tests of what the students can do;
- examinations;
- interviews;
- questionnaires;
- criterion-referenced evaluations by the users;
- post-course diaries;
- post-course ‘shadowing’ of the learners;
- post-course reports on the learners by employers, subject tutors, etc.
The main problem, of course, is that it takes time and expertise to measure post-use effects
reliably (especially as, to be really revealing, there should be measurement of pre-use
attitudes and abilities in order to provide data for post-use comparison).

11
 FRAMEWORKS AND CHECKLISTS FOR MATERIALS EVALUATION

In order to make materials evaluation more operable and easier to conduct, many
frameworks or checklists have been developed.
 McDonough and Shaw’s (2003) checklist
• What visual material does the textbook contain? Is it there for cosmetic value or is it
actually integrated in the text?
• Is the layout and presentation clear or cluttered?
• Is the material too culturally biased or specific?
• Do the materials represent minority groups and/or women in a negative way? Do
they present a “balanced” picture of a particular country/society?
• Is the textbook accompanied by audio/video material? Is it essential to possess this
extra material in order to use the textbook successfully?
• If tests are included are they useful for your particular learners?
 Breen and Candlin’s (1987) framework: (in Sheldon 1987, ELT Document 126)
PHASE ONE: Initial questions
1. What do the materials aim to do and what do they contain?
2. What do the materials make your learners do while they are learning?
3. How do the materials expect you to teach the learners in the classroom?
4. Are the materials the only resource in classroom language learning?
PHASE TWO: Your learners and the materials
1. Are the materials appropriate to your learners’ needs and interests?
2. Are the materials appropriate to your learners’ own approaches to language
learning?
3. Are the materials appropriate to the classroom teaching/learning process?
4. Seven design features of materials for classroom work
5. Discovering learners’ criteria for good materials
 Dougill’s framework (Dougill 1987, in Sheldon 1987, ELT Document 126)
 FRAMEWORK
 THE UNITS
 SUBJECT-MATTER
 FORM
 COURSE COMPONENTS

12
  Littlejohn’s framework for analyzing materials (in Tomlinson 1998:214-6)
Publication:
1. Place of learners’ materials in the set
2. publication form: worksheets vs. bound book; durable vs. consumable
3. Subdivision of materials into sections;
4. Subdivision of sections into sub-sections
5. Continuity
6. Route (whether teaching order is predetermined; i.e. can a section be skipped)
7. Access (how access into the materials are supported, for example, whether there are
contents lists, wordlists, and indexes)
Design
1. Aims
2. Principles of selection
3. Principles of sequencing
4. Subject matter and focus of subject matter
5. Types of learning/teaching activities
6. Participation: who does what with whom
7. Learner roles
8. Teacher roles
9. Role of materials as a whole
 Cunningsworth’s (1984) checklist (Cunningsworth 1984:74-9)
Section I. General impression
Section II. Appropriacy
Section III. Motivation and the Learner
Section IV. Cultural Differences
Section V. Padagogic Analysis and the Language Content
 Grammar
 Vocabulary
 Phonology:
Section VII. Supplementary Materials
Other
o Cost Effectiveness:
o Author’s Claim
 Grant’s (1987) evaluation criteria (Grant 1987)
Stage 1
• Communicative? Is the textbook communicative? Will the students be able to use the
language to communicate as a result of using the textbook?
• Aims? Does it fit in with our aims and objectives?
• Teachable? Does the course seem reasonably easy to use, well-organised and easy to
find your way around?

13
 • Available add-ons? Are there available additional materials such as teacher’s book,
tapes, workbooks?
• Level? Does the level seem about right?
• Your impression? What is you overall impression of the book?
• Student interest? Are your students likely to find the textbook interesting?
• Tried and tested? Has the course been tried and tested in real classrooms? Where?
By whom? What were the results? How do you know?
Stage 2
Questionnaire Part 1: Does the book suit your students?
Questionnaire Part 2: Does the book suit the teacher?
Questionnaire Part 3: Does the textbook suit the syllabus and examination?

As Tomlinson ( 1999, p.1, cited in Tomlinson 2013, p36 ) puts it " ‘there can be no one model
framework for the evaluation of materials; the framework used must be determined by the
reasons, objectives and circumstances of the evaluation’ . Most of the lists in the
publications above are to some extent subjective as they are lists for pre-use evaluation and
this involves selection and prediction.
A useful exercise he (Tomlinson 2013: 36-37) therefore says, for anybody writing or
evaluating language teaching materials would be to evaluate the checklists and criteria lists
from a sample of the publications above against the following criteria:
- Is the list based on a coherent set of principles of language learning?
- Are all the criteria actually evaluation criteria or are they criteria for analysis?
- Are the criteria sufficient to help the evaluator to reach useful conclusions?
- Are the criteria organized systematically (e.g. into categories and subcategories which
facilitate discrete as well as global verdicts and decisions)?
- Are the criteria sufficiently neutral to allow evaluators with different ideologies to make
use of them?
- Is the list sufficiently flexible to allow it to be made use of by different evaluators in
different circumstances?

14
 DEVELOPING CRITERIA FOR MATERIALS EVALUATION

1. Brainstorm a list of universal criteria

Universal criteria are those which would apply to any language learning materials anywhere
for any learners.
They derive from principles of language learning and the results of classroom observation
and provide the fundamental basis for any materials evaluation. Brainstorming a random list
of such criteria is a very useful way of beginning an evaluation by phrasing the criteria as
specific questions
Here are the universal criteria used in Tomlinson and Masuhara (2013) to evaluate six
current global coursebooks.
To what extent is the course likely to:
 provide extensive exposure to English in use?
 engage the learners affectively?
 engage the learners cognitively?
 provide an achievable challenge?
 help learners to personalize their learning?
 help the learners to make discoveries about how English is typically used?
 provide opportunities to use the target language for communication?
 help the learners to develop cultural awareness?
 help the learners to make use of the English environment outside the classroom?
 cater for the needs of all the learners?
 provide the flexibility needed for effective localization?
 help the learners to continue to learn English after the course?
 help learners to use English as a lingua franca?
 help learners to become effective communicators in English?
 achieve its stated objectives?
2. Subdivide some of the criteria
If the evaluation is going to be used as a basis for revision or adaptation of the materials, it
is useful to subdivide some of the criteria into more specific questions.
For example:
Are the instructions:
 succinct?
 sufficient?
 self-standing?
 standardized?
 separated?
 sequenced?
 staged?
Such a subdivision can help to pinpoint specific aspects of the materials which could gain
from revision or adaptation.

15
 3. Monitor and revise the list of universal criteria:
Monitor the list and rewrite it according to the following criteria:
• Is each question an evaluation question?
If a question is an analysis question (e.g. ‘Does each unit include a test?’) then you can only
give the answer a 1 or a 5 on the 5-point scale which is recommended later in this suggested
procedure. However, if it is an evaluation question (e.g. ‘To what extent are the tests likely
to provide useful learning experiences?’) then it can be graded at any point on the scale.
• Does each question only ask one question?
Many criteria in published lists ask two or more questions and therefore cannot be used in
any numerical grading of the materials. For example, Grant (1987) includes the following
question which could be answered ‘Yes; No’ or ‘No; Yes’: ‘following question which could be
answered ‘Yes; No’ or ‘No; Yes’: ‘1 Is it attractive? Given the average age of your students,
would they enjoy using it?’ (p. 122). This question could be usefully rewritten as:
1. Is the book likely to be attractive to your students?
2. Is it suitable for the age of your students?
3. Are your students likely to enjoy using it?

• Is each question answerable?

This might seem an obvious question but in many published lists of criteria some questions
are so large and so vague that they cannot usefully be answered. Or sometimes they cannot
be answered without reference to other criteria, or they require expert knowledge of the
evaluator. For example:
‘Is it culturally acceptable?’ (Grant, 1987, p. 122)
‘Does it achieve an acceptable balance between knowledge about the language and practice
in using the language?’ (Ibid.)
‘Does the writer use current everyday language, and sentence structures that follow normal
word order?’ (Daoud and Celce-Murcia, 1979, p. 304)
• Is each question free of dogma?
The questions should reflect the evaluators’ principles of language learning but should not
impose a rigid methodology as a requirement of the materials. If they do, the materials
could be dismissed without a proper appreciation of their potential value. For example, the
following examples make assumptions about the pedagogical procedures of coursebooks
which not all coursebooks actually follow:
 ‘Are the various stages in a teaching unit (what you would probably call presentation,
practice and production) adequately developed?’ (Mariani, 1983, p. 29)
 Do the sentences gradually increase in complexity to suit the growing reading ability
of the students? (Daoud and Celce-Murcia, 1979, p. 304)

16
 • Is each question reliable in the sense that other evaluators would interpret it in the
same way?
Some terms and concepts which are commonly used in applied linguistics are amenable to
differing interpretations and are best avoided or glossed when attempting to measure the
effects of materials. For example, each of the following questions could be interpreted in a
number of ways:
 Are the materials sufficiently authentic?
 Is there an acceptable balance of skills?
 Do the activities work?
 Is each unit coherent?
There are a number of ways in which each question could be rewritten to make it more
reliable and useful. For example:
 Do the materials help the learners to use the language in situations they are likely to
find themselves in after the course?
 Is the proportion of the materials devoted to the development of reading skills
suitable for your learners?
 Are the communicative tasks useful in providing learning opportunities for the
learners?
 Are the activities in each unit linked to each other in ways which help the learners?

4. Categorize the list

It is very useful to rearrange the random list of universal criteria into categories which
facilitate focus and enable generalizations to be made. An extra advantage of doing this is
that you often think of other criteria related to the category as you are doing the
categorization exercise.
Possible categories for universal criteria would be:
 Learning Principles
 Cultural Perspective
 Topic Content
 Teaching Points
 Texts
 Activities
 Methodology
 Instructions
 Design and Layout
5. Develop media-specific criteria
These are criteria which ask questions of particular relevance to the medium used by the
materials being evaluated (e.g. criteria for books, for audio cassettes, for videos, etc.).
Examples of such criteria would be:
 Is it clear which sections the visuals refer to?
 Is the sequence of activities clearly signaled?
 Are the different voices easily distinguished?

17
  Do the gestures of the actors help to make the language meaningful in realistic ways?
Obviously these criteria can also be usefully categorized (e.g. under Illustrations, Layout,
Audibility, Movement)
6. Develop content-specific criteria
These are criteria which relate to the topics and/or teaching points of the materials being
evaluated. ‘Thus there would be a set of topic related criteria which would be relevant to the
evaluation of a business English textbook but not to a general English coursebook; and there
would be a set of criteria relevant to a reading skills book which would not be relevant to the
evaluation of a grammar practice book and vice versa’ (Tomlinson, 1999, p. 11).
Examples of content-specific criteria would be:
 Do the examples of business texts (e.g. letters, invoices, etc.) replicate features of
real-life business practice?
 Do the reading texts represent a wide and typical sample of genres?
7. Develop age-specific criteria
These are criteria which relate to the age of the target learners. Thus there would be criteria
which are only suitable for 5-year-olds, for 10-year-olds, for teenagers, for young adults and
for mature adults. These criteria would relate to cognitive and affective development, to
previous experience, to interests and to wants and needs.
Examples of age-specific criteria would be:
 Are there short, varied activities which are likely to match the attention span of the
learners?
 Is the content likely to provide an achievable challenge in relation to the maturity
level of the learners?
8. Develop local criteria
These are criteria which relate to the actual or potential environment of use. They are
questions which are not concerned with establishing the value of the materials per se but
rather with measuring the value of the materials for particular learners in particular
circumstances. It is this set of criteria which is unique to the specific evaluation being
undertaken and which is ultimately responsible for most of the decisions made in relation to
the adoption, revision or adaptation of the materials.
Typical features of the environment which would determine this set of materials are:
 the type(s) of institution(s);
 the resources of the institution(s);
 class size;
 the background, needs and wants of the learners;
 the background, needs and wants of the teachers;
 the language policies in operation;
 the syllabus;
 the objectives of the courses;

18
  the intensity and extent of the teaching time available;
 the target examinations;
 the amount of exposure to the target language outside the classroom.
Examples of local criteria would be:
 To what extent are the stories likely to interest 15-year-old boys in Turkey?
 To what extent are the reading activities likely to prepare the students for the
reading questions in the Primary School Leaving Examination in Singapore?
 To what extent are the topics likely to be acceptable to parents of students in Iran?

9. Develop other criteria

Other criteria which it might be appropriate to develop could include teacher-specific,
administrator-specific, gender-specific, culture-specific or L1-specific criteria and, especially
in the case of a review for a journal, criteria assessing the match between the materials and
the claims made by the publishers for them.
10. Trial the criteria
It is important to trial the criteria (even prior to a small, fairly informal evaluation) to ensure
that the criteria are sufficient, answerable, reliable and useful. Revisions can then be made
before the actual evaluation begins.
11. Conducting the evaluation
The most effective way of conducting an evaluation is to:
 make sure that there is more than one evaluator;
 discuss the criteria to make sure there is equivalence of interpretation;
 answer the criteria independently and in isolation from the other evaluator(s);
 focus in a large evaluation on a typical unit for each level (and then check its
typicality by reference to other units);
 give a score for each criterion (with some sets of criteria weighted more heavily than
others);
 write comments at the end of each category;
 at the end of the evaluation aggregate each evaluator’s scores for each criterion,
category of criteria and set of criteria and then average the scores;
 record the comments shared by the evaluators;
 write a joint report.
What is recommended above is a very rigorous, systematic but time-consuming approach to
materials evaluation which I think is necessary for major evaluations from which important
decisions are going to be made. However for more informal evaluations (or when very little
time is available) I would recommend the following procedure:
1. Brainstorm beliefs
2. Decide on shared beliefs
3. Convert the shared beliefs into universal criteria
4. Write a profile of the target learning context for the materials
5. Develop local criteria from the profile
6. Evaluate and revise the universal and the local criteria
7. Conduct the evaluation

19
 ASSIGNMENT

 Evaluate the 5e book from the “Lets’ keep in Touch” series. Use an
evaluation tool based on the criteria in one of the models in the course
book (Frameworks and checklists for materials evaluation, p.12), and
ideas from the last section of the course book entitled ‘Developing
criteria for materials evaluation’ (pp.14-19).
 Start with an introduction (a brief presentation of the textbook and its
content), then describe your evacuation tool, before evaluating the
textbook against it. Conclude with a summary of the main characteristics
of the book and say the type of syllabus you believe underlies this
textbook.

 To be handed over in two weeks after reception of the course book.

20