Study Guide in Language Learning Materials Development FM-AA-CIA-15 Rev.

0 10-July-2020

Module 2 – Materials Evaluation

Module No. 2

Materials Evaluation

MODULE OVERVIEW

Enumerated below are topics that you need to study in this module.
1) Definition and principles in materials evaluation
2) Qualities each unit of material should reflect
3) Types of materials evaluation

MODULE LEARNING OBJECTIVES

At the end of the lesson, the students will be able to:

4) evaluate language learning materials used in a basic education classroom using the principles in materials
evaluation;
5) examine the K to 12 English curriculum and cite materials used to facilitate mastery of a specific
competency; and
6) list down specific competencies from the K to12 English curriculum and identify possible language learning
materials to be used for these competencies.

LEARNING CONTENTS

Definition of MATERIALS EVALUATION

Materials evaluation is a procedure that involves measuring the value (or potential value) of a set of
learning materials. It involves making judgements about the effect of the materials on the people using
them and it tries to measure some or all of the following:
 the appeal of the materials to the learners
 the credibility of the materials to learners, teachers and administrators;
 the validity of the materials (i.e. Is what they teach worth teaching?);
 the reliability of the materials (i.e. Would they have the same effect with different groups of target
learners?);the ability of the materials to interest the learners and the teachers;
 the ability of the materials to motivate the learners;
 the value of the materials in terms of short-term learning (important, for example, for performance on
tests and examinations);the value of the materials in terms of long-term learning (of both language and
of communication skills);the learners’ perceptions of the value of the materials;
 the teachers’ perceptions of the value of the materials;
 the assistance given to the teachers in terms of preparation, delivery and assessment;
 the flexibility of the materials (e.g. the extent to which it is easy for a teacher to adapt the materials to
suit a particular context);
 the contribution made by the materials to teacher development;
 the match with administrative requirements (e.g. standardization across classes, coverage of a
syllabus, preparation for an examination).

PANGASINAN STATE UNIVERSITY 1

Study Guide in Language Learning Materials Development FM-AA-CIA-15 Rev. 0 10-July-2020

Module 2 – Materials Evaluation

Principles of MATERIALS EVALUATION

Many evaluations are impressionistic, or at best are aided by an ad hoc and very subjective list of
criteria. In my view it is very important that evaluations (even the most informal ones) are driven by a set of
principles and that these principles are articulated by the evaluator(s) prior to the evaluation. In this way
greater validity and reliability can be achieved and fewer mistakes are likely to be made. In developing a
set of principles it is useful to consider the following.

The evaluator’s theory of learning and teaching

All teachers develop theories of learning and teaching which they apply in their classrooms (even
though they are often unaware of doing so). Many researchers (e.g.Schon,1983) argue that it is useful for
teachers to try to achieve an articulation of their theories by reflecting on their practice. For example Edge
and Wharton (1998,p. 297) argue that reflective practice can not only lead to ‘perceived improvements in
practice but, more importantly, to deeper understandings of the area investigated’. Ina similar way I am
going to argue that the starting point of any evaluation should be reflection on the evaluator’s practice
leading to articulation of the evaluator’s theories of learning and teaching. In this way evaluators can make
overt their predispositions and can then both make use of them in constructing criteria for evaluation and
be careful not to let them weight the evaluation too much towards their own bias. At the same time
evaluators can learn a lot about themselves and about the learning and teaching process.
Here are some of my theories, which I have articulated as a result of reflection on my own and other
teachers’ practice:
 Language learners succeed best if learning is a positive, relaxed and enjoyable experience.
 Language teachers tend to teach most successfully if they enjoy their role and if they can gain some
enjoyment themselves from the materials they are using.
 Learning materials lose credibility for learners if they suspect that the teacher does not value them.
 Each learner is different from all the others in a class in terms of his or her personality, motivation,
attitude, aptitude, prior experience, interests, needs, wants and preferred learning style.
 Each learner varies from day to day in terms of motivation, attitude, mood, perceived needs and
wants, enthusiasm and energy.
 There are superficial cultural differences between learners from different countries (and these
differences need to be respected and catered for)but there are also strong universal determinants of
successful language teaching and learning.
 Successful language learning in a classroom (especially in large classes) depends on the generation
and maintenance of high levels of energy.
 The teacher is responsible for the initial generation of energy in a lesson; good materials can then
maintain and even increase that energy.
 Learners only learn what they really need or want to learn.
 Learners often say that what they want is focused language practice but they often seem to gain more
enjoyment and learning from activities which stimulate them to use the target language to say
something they really want to say.
 Learners think, say and learn more if they are given an experience or text to respond to than if they
are just asked for their views, opinions and interests.
 The most important thing that learning materials have to do is to help the learner to connect the
learning experience in the classroom to their own life outside the course.
 The more novel (or better still bizarre) the learning experience is the more impact it is likely to make
and it is more likely to contribute to long- term acquisition.
 The most important result that learning materials can achieve is to engage the emotions of learners.
Laughter, joy, excitement, sorrow and anger can promote learning. Neutrality, numbness and nullity
cannot.

PANGASINAN STATE UNIVERSITY 2

Study Guide in Language Learning Materials Development FM-AA-CIA-15 Rev. 0 10-July-2020

Module 2 – Materials Evaluation

Learning theory
Research into learning is controversial as there are so many variables involved and local
circumstances often make generalization precarious. However, it is important that the materials evaluator
considers the findings of learning research and decides which of its findings are convincing and applicable.
The conclusions which convince me are that:
 Deep processing of intake is required if effective and durable learning is to take place (Craik and
Lockhart,1972). Such processing is semantic in that the focus of the learner is on the meaning of the
intake and in particular on its relevance to the learner.
 Affective engagement is also essential for effective and durable learning. Having positive attitudes
towards the learning experience and developing self-esteem while learning are important determiners
of successful learning. And so is emotional involvement. Emotions must be ‘considered an essential
part of learning’ (Williams and Burden,1997, p. 28) as they’ are the very center of human mental
life . . . [they] link what is important for us to the world of people, things and happenings’ (Oatley and
Jenkins,1996, p. 122).
 Making mental connections is a crucial aspect of the learning process. In order for learning to be
successful, connections need to be made between the new and the familiar, between what is being
learned and the learner’s life and between the learning experience and its potential value in the future.
 Experiential learning is essential (though not necessarily sufficient) and, in particular, apprehension
should come to the learner before comprehension (Kolb,1984; Kelly,1997; Tomlinson and
Masuhara,2000;Kolb and Kolb,2009).
 Learners will only learn if they need and want to learn and if they are willing to invest time and energy
in the process. In other words, both instrumental and integrative motivation are vital contributors to
learning success (Dornyei and Ushioda,2009).
 Multidimensional processing of intake is essential for successful learning and involves the learner
creating a mental representation of the intake through such mental processes as sensory imaging
(especially visualization), affective association and the use of the inner voice(Masuhara,1998,2005;
Tomlinson, 2000a,2000b,2001a, 2001c,2003,2011b; de Guerro,2005; Wiley,2006; Tomlinson and
Avila,2007). As Berman (1999, p. 2) says, ‘we learn best when we see things as part of a recognized
pattern, when our imaginations are aroused, when we make natural associations between one idea
and another, and when the information appeals to our senses.’ One of the best ways of achieving
multidimensional representation in learning seems to be a whole person approach which helps the
learner to respond to the learning experience with emotions, attitudes, opinions and ideas (Jacobs and
Schumann,1992;Schumann,1997,1999; Arnold,1999).
 Materials which address the learner in an informal, personal voice are more likely to facilitate learning
than those which use a distant, formal voice(Beck et al.,1995; Tomlinson,2001b). Features which
seem to contribute toa successful personal voice include such aspects of orality as:
o Informal discourse features (e.g. contracted forms, ellipsis, informal lexis)
o The active rather than the passive voice
o Concreteness (e.g. examples, anecdotes)
o Inclusiveness (e.g. not signaling intellectual, linguistic or cultural superiority over the learners)
o Sharing experiences and opinions
o Sometimes including casual redundancies rather than always being concise. (Tomlinson,2001b

As a materials evaluator I would convert the assertions above into criteria for the assessment of learning
material. For example, I would construct such criteria as:
o To what extent are the materials related to the wants of the learners?
o To what extent are the materials likely to help the learners to achieve connections with their own
lives?

PANGASINAN STATE UNIVERSITY 3

Study Guide in Language Learning Materials Development FM-AA-CIA-15 Rev. 0 10-July-2020

Module 2 – Materials Evaluation

o To what extent are the materials likely to stimulate emotional engagement?

o To what extent are the materials likely to promote visual imaging?

Second language acquisition research (SLA)

SLA research is so far inconclusive and has stimulated many disagreements and debates (e.g. about
the value of the explicit teaching of discrete language points). However, there is now a sufficient consensus
of opinion on certain facilitating features of language learning for them to be useful in helping to articulate
the principles to be used as a basis of materials evaluation. In Tomlinson (2011a, pp. 6–23) I discussed the
principles of second language acquisition which I think SLA researchers would agree are relevant to the
development of materials for the teaching of languages. Some of these principles are summarized below:
 Materials should achieve impact (through novelty, variety, surprise, bizarreness, attractive presentation
and appealing content).
 Materials should help learners to feel at ease (e.g. through the use of white space to prevent clutter and
the use of texts and illustrations which they can relate to their own culture, through a supportive
approach which is not always testing them and through the use of a personal voice).
 Materials should help the learners to develop confidence (e.g. through ‘pushing’ learners slightly beyond
their existing proficiency by involving them in tasks which are challenging but achievable).
 What is being taught should be perceived by learners as relevant and useful (Stevick,1976;
Krashen,1982; Wenden and Rubin,1987).
 Materials should require and facilitate learner self-investment (e.g. through giving learners responsibility
for making decisions and through encouraging them to make discoveries about the language for
themselves (Rutherford and Sharwood-Smith,1988; Tomlinson,1994,2007; Bolitho et al.,2003).
 Learners must be ready to acquire the points being taught both in terms of linguistic, developmental
readiness and of psychological readiness too(Meisel et al.,1981; Pienemann,1985,2005).
 Materials should expose the learners to language in authentic use (ideally to a rich and varied input
which includes unplanned, semi-planned and planned discourse and which stimulates mental
response). See Mishna(2005), Rilling and Dantas-Whitney (2009) and Tomlinson (2012).
 The learners’ attention should be drawn to linguistic features of the input (so that they are alerted to
subsequent instances of the same feature in future input (Seliger,1979; White,1990; Schmidt,1992;
Ortega,2009).
 Materials should provide the learners with opportunities to use the target language to achieve
communicative purposes (in order to automatize existing procedural knowledge, to check the
effectiveness of their existing hypotheses (Swain,1985,2005) and to develop strategic competence
(Canale and Swain,1980)).
 Materials should take into account that the positive effects of instruction are usually delayed (and
therefore should not expect effective production immediately to follow initial presentation but should
rather ensure recycling and frequent and ample exposure to the instructed features incommunicative
use).
 Materials should take into account that learners differ in learning styles (Oxford and Anderson,1995;
Oxford,2002; Anderson,2005) (and should therefore ensure that they cater for learners who are
predominantly visual, auditory, kinaesthetic, studial, experiential, analytic, global, dependent or
independent).Materials should take into account that learners differ in affective attitudes(Wenden and
Rubin,1987) (and therefore materials should offer variety and choice).
 Materials should maximize learning potential by encouraging intellectual, aesthetic and emotional
involvement which stimulates both right and left brain activities (through a variety of non-trivial activities
requiring a range of different types of processing).
 Materials should provide opportunities for outcome feedback (i.e. feedback on the effectiveness of the
learner in achieving communication objectives rather than just feedback on the accuracy of the output)

In addition to the requirements listed in Tomlinson (2011a) I would like to add that materials should:

PANGASINAN STATE UNIVERSITY 4

Study Guide in Language Learning Materials Development FM-AA-CIA-15 Rev. 0 10-July-2020

Module 2 – Materials Evaluation

o help the learner to develop cultural awareness and sensitivity (Tomlinson, 2000b; Byram and
Masuhara,2013);reflect the reality of language use;
o help learners to learn in ways similar to the circumstances in which they will have to use the
language;
o help to create readiness to learn (e.g. by helping learners to draw their attention to the gap
between their use of a feature of communication and the use of that feature by proficient users of
the language, or by involving the learners in a task in which they need to learn something new in
order to be successful);
o achieve affective engagement (Tomlinson, 2010).

Richards (2001, p. 264) suggests a rather different and briefer list of the ‘qualities each unit in the
materials should reflect’:
 Gives learners something they can take away from the lesson.
 Teaches something learners feel they can use.
 Gives learners a sense of achievement.
 Practices learning items in an interesting and novel way.
 Provides a pleasurable learning experience.
 Provides opportunities for individual practice.
 Provides opportunities for personalization.
 Provides opportunities for self-assessment of learning.

The important thing is for materials evaluators to decide for themselves which findings of SLA
research they will use to develop principles for their evaluation. Ultimately what matters is that an
evaluation is principled, that the evaluator’s principles are made overt and that they are referred to when
determining and carrying out the procedures of the evaluation. Otherwise the evaluation is likely to be ad
hoc and mistakes will be made. A textbook selected mainly because of its attractive appearance could turn
out to be very boring for the learners to use; a review which overemphasizes an irritating aspect of the
materials (e.g. a particular character in a video course) can give a distorted impression of the value of the
materials; a course selected for national use by a ministry of education because it is the cheapest or
because it is written by famous writers and published by a prestigious publisher could turn out to be a very
expensive disaster.

Types of MATERIALS EVALUATION

There are many different types of materials evaluation. It is possible to apply the basic principles of
materials evaluation to all types of evaluation but it is not possible to make generalizations about
procedures which apply to all types. Evaluations differ, for example, in purpose, in personnel, in formality
and in timing. You might do an evaluation in order to help a publisher to make decisions about publication,
to help yourself in developing materials for publication, to select a textbook, to write a review for a journal or
as part of a research project. As an evaluator you might be a learner, a teacher, an editor, a researcher, a
Director of Studies or an Inspector of English. You might be doing a mental evaluation in a bookshop, filling
in a short questionnaire in class or doing a rigorous, empirical analysis of data elicited from a large sample
of users of the materials. You might be doing your evaluation before the materials are used while they are
being used or after they have been used. In order to conduct an effective evaluation you need to apply your
principles of evaluation to the contextual circumstances of your evaluation in order to determine the most
reliable and effective procedures.

PRE-USE EVALUATION
Pre-use evaluation involves making predictions about the potential value of materials for their
users. It can be context-free, as in a review of materials for a journal, context-influenced as in a review of

PANGASINAN STATE UNIVERSITY 5

Study Guide in Language Learning Materials Development FM-AA-CIA-15 Rev. 0 10-July-2020

Module 2 – Materials Evaluation

draft materials for a publisher with target users in mind or context-dependent, as when a teacher selects a
coursebook for use with her particular class. Often pre-use evaluation is impressionistic and consists of a
teacher flicking through a book to gain a quick impression of its potential value (publishers are well aware
of this procedure and sometimes place attractive illustrations in the top right-hand corner of the right-hand
page in order to influence the flicker in a positive way).Even a review for a publisher or journal, and an
evaluation for a ministry of education is often ‘fundamentally a subjective, rule of thumb activity’
(Sheldon,1988, p. 245) and often mistakes are made. Making an evaluation criterion-referenced can
reduce (but not remove) subjectivity and can certainly help to make an evaluation more principled, rigorous,
systematic and reliable. This is especially true if more than two evaluators conduct the evaluation
independently and then average their conclusions. For example, in the review of eight adult EFL courses
conducted by Tomlinson et al. (2001), the four evaluators devised one-hundred-and-thirty-three criteria
together and then used them independently and in isolation to evaluate the eight courses before pooling
their data and averaging their scores. Even then, though, the reviewers admitted that, ‘the same review,
conducted by a different team of reviewers, would almost certainly have produced a different set of results’
(p. 82).
Making use of a checklist of criteria has become popular in materials evaluations and certain
checklists from the literature have been frequently made use of in evaluations(e.g. Cunnings worth
(1984,1995), Skierso (1991), Brown (1997), Gearing, (1999)). The problem though is that no set of criteria
is applicable to all situations and, as Byrd(2001) says, it is important that there is a fit between the materials
and the curriculum, students and teachers. Matthews (1985), Cunnings worth (1995) and Tomlinson
(2012)have also stressed the importance of relating evaluation criteria to what is known about the context
of learning and Makundan and Ahour (2010) in their review of 48evaluation checklists were critical of most
checklists for being too context bound to be generalizable. Makundan and Ahour (2010) proposed that a
framework for generating flexible criteria would be more useful than detailed and inflexible checklists (a
proposition also made by Ellis (2011) and stressed and demonstrated by Tomlinson (2003b)). Other
researchers who have proposed and exemplified frameworks for generating evaluation criteria include:

o McGrath (2002), who suggests a procedure involving materials analysis followed by first glance
evaluation, user feedback and evaluation using context-specific checklists.
o Riazi (2003), who suggests, surveying the teaching/learning situation, conducting a neutral
analysis and the carrying out of a belief-driven evaluation.
o Rubdy (2003), who suggests a dynamic model of evaluation in which the categories of
psychological validity, pedagogical validity and process and content validity interact.
o Mukundan (2006), who describes the use of a composite framework combining checklists,
reflective journals and computer software to evaluatively textbooks in Malaysia.
o McDonough et al. (2013), who focus on developing criteria evaluating the suitability of materials in
relation to usability, generalizability, adaptability and flexibility.

Tomlinson and Masuhara (2004, p. 7) proposed the following criteria for evaluating criteria:
a) Is each question an evaluation question?
b) Does each question only ask one question?
c) Is each question answerable?
d) Is each question free of dogma?
e) Is each question reliable in the sense that other evaluators would interpret it in the same way?

Tomlinson (2012) reports these criteria and gives examples from the many checklists in the literature of
evaluation criteria which their use exposes as inadequate in terms of specificity, clarity, answerability,
validity and generalizability.

WHILST-USE EVALUATION
This involves measuring the value of materials while using them or while observing them being
used. It can be more objective and reliable than pre-use evaluation as it makes use of measurement rather

PANGASINAN STATE UNIVERSITY 6

Study Guide in Language Learning Materials Development FM-AA-CIA-15 Rev. 0 10-July-2020

Module 2 – Materials Evaluation

than prediction. However, it is limited to measuring what is observable (e.g. ‘Are the instructions clear to the
learners?’) and cannot claim to measure what is happening in the learners’ brains. It can measure short-
term memory through observing learner performance on exercises but it cannot measure durable and
effective learning because of the delayed effect of instruction. It is therefore very useful but dangerous too,
as teachers and observers can be misled by whether the activities seem to work or not. Exactly what can
be measured in a whilst-use evaluation is controversial but I would include the following:

o Clarity of instructions
o Clarity of layout
o Comprehensibility of texts
o Credibility of tasks
o Achievability of tasks
o Achievement of performance objectives
o Potential for localization
o Practicality of the materials
o Teachability of the materials
o Flexibility of the materials
o Appeal of the materials
o Motivating power of the materials
o Impact of the materials
o Effectiveness in facilitating short-term learning

Most of the above can be estimated during an open-ended, impressionistic observation of materials
in use but greater reliability can be achieved by focusing on one criterion at a time and by using pre-
prepared instruments of measurement. For example, oral participation in an activity can be measured by
recording the incidence and duration of each student’s oral contribution, potential for localization can be
estimated by noting the times the teacher or a student refers to the location of learning while using the
materials and even motivation can be estimated by noting such features as student eye focus, proximity to
the materials, time on task and facial animation. Whilst-use evaluation receives very little attention in the
literature, but Jolly and Bolitho (2011) describe interesting case studies of how student comment and
feedback during lessons provided useful evaluation of materials, which led to improvements being made in
the materials during and after the lessons. Also, Tomlinson and Masuhara (2010) report materials
development projects in which whilst-use evaluation was made use of.

POST-USE EVALUATION
Post-use evaluation is probably the most valuable (but least administered) type of evaluation as it
can measure the actual effects of the materials on the users. It can measure the short-term effect as
regards motivation, impact, achievability, instant learning, etc., and it can measure the long-term effect as
regards durable learning and application. It can answer such important questions as:
o What do the learners know which they did not know before starting to use the materials?
o What do the learners still not know despite using the materials?
o What can the learners do which they could not do before starting to use the materials?
o What can the learners still not do despite using the materials?
o To what extent have the materials prepared the learners for their examinations?
o To what extent have the materials prepared the learners for their post-course use of the target
language?
o What effect have the materials had on the confidence of the learners?
o What effect have the materials had on the motivation of the learners?
o To what extent have the materials helped the learners to become independent learners?
o Did the teachers find the materials easy to use?

PANGASINAN STATE UNIVERSITY 7

Study Guide in Language Learning Materials Development FM-AA-CIA-15 Rev. 0 10-July-2020

Module 2 – Materials Evaluation

oDid the materials help the teachers to cover the syllabus?

oDid the administrators find the materials helped them to standardize the teaching in their
institution?
In other words, it can measure the actual outcomes of the use of the materials and thus provide the
data on which reliable decisions about the use, adaptation or replacement of the materials can be made.
Ways of measuring the post-use effects of materials include:
o tests of what has been ‘taught’ by the materials;
o tests of what the students can do;
o examinations;
o interviews;
o questionnaires;
o criterion-referenced evaluations by the users;
o post-course diaries;
o post-course ‘shadowing’ of the learners;
o post-course reports on the learners by employers, subject tutors, etc.

The main problem, of course, is that it takes time and expertise to measure post-use effects reliably
(especially as, to be really revealing, there should be measurement of pre-use attitudes and abilities in
order to provide data for post-use comparison).But publishers and ministries do have the time and can
engage the expertise, and teachers can be helped to design, administer and analyse post-use instruments
of measurement. Then we will have much more useful information, not only about the effects of particular
courses of materials but about the relative effectiveness of different types of materials. Even then, though,
we will need to be cautious, as it will be very difficult to separate such variables as teacher effectiveness,
parental support, language exposure outside the classroom, intrinsic motivation, etc.
For a description of the process of post-use evaluation of piloted materials see Donovan (1998), for
descriptions of how publishers use focus groups for post-use evaluation of materials see Amrani (2011)
and for suggestions of how teachers could do post-use micro-evaluations of materials see Ellis (1998,
2011). For reports of projects which conducted post-use evaluation of materials in many different countries
see Tomlinson and Masuhara (2010).

LEARNING ACTIVITY 1

Name: ___________________________________________ Student No. _______________________

(Surname, Given Name, MI)

Note: Upon submitting your activity, follow this name format: SG2_Learning Activity 1

GENERAL DIRECTIONS: Watch the link with actual demonstration teaching in face-to-face class.
Afterwards, answer the following questions provided. Make sure to write/encode your chosen link below
your name, by that, the instructor is guided as to what demonstration teaching is watched by the student in
answering the provided questions. Each item is given 6-point.

https://www.youtube.com/watch?v=EXdMsKJjp_c

1) Do the materials provide useful opportunities for the learners to think for themselves? Explain.
2) Are the target learners likely to be able to follow the instructions? Explain.
3) Are the materials likely to cater for different preferred learning styles? Explain.
4) Are the materials likely to achieve affective engagement? Explain.
5) Does the teacher effectively facilitate the class with the materials provided? Explain.

PANGASINAN STATE UNIVERSITY 8

Study Guide in Language Learning Materials Development FM-AA-CIA-15 Rev. 0 10-July-2020

Module 2 – Materials Evaluation

REFERENCES

Tomlinson B. (2013). Developing Materials for Language Teaching. Retrieved at

https://www.scribd.com/document/430048129/E-Book-Developing-Materials-for-Language-Teaching-Brian-
Tomlison-2013-Pdf-pdf

PANGASINAN STATE UNIVERSITY 9