Evaluation and Adaptation of

Materials
Group 1
Casia, Disomimba, Cuizon, & Antiola
 OUTLINE
What is materials evaluation? (CASIA)
Principles in materials evaluation (CASIA)
• The evaluator’s theory of learning and teaching (CASIA)
• Learning theory (DISOMIMBA)
Second Language Acquisition Research (SLA) (DISOMIMBA)
Types of Materials Evaluation (CUIZON)
• Pre-use evaluation
• Whilst-use evaluation
• Post-use evaluation
Standard approaches to Materials Evaluation (CUIZON)
Developing Criteria for Materials Evaluation (ANTIOLA)
Conclusion (ANTIOLA)
What is Materials
Evaluation?
Materials Evaluation...
• is a procedure that involves measuring the value (or potential
value) of a set of learning materials
• involves making judgements about the effect of the materials
on the people using them, and,
• it tries to measure some or all of the following;
• the appeal of the materials to the learners;
• the credibility of the materials to learners, teachers and
administrators;
• the validity of the materials
• the reliability of the materials
• the ability of the materials to interest the learners and the
teachers;
• the ability of the materials to motivate the learners;
• the value of the materials in terms of short-term learning the
value of the materials in terms of long-term learning;
• the learners’ perceptions of the value of the materials;
• the teachers’ perceptions of the value of the materials;
• the assistance given to the teachers in terms of preparation,
delivery and assessment;
• the flexibility of the materials
• the contribution made by the materials to teacher
development;
• the match with administrative requirements
 EVALUATION
• focuses on the users of the
materials and makes
judgements about their
effects.
• No matter how structured,
criterion referenced and
rigorous an evaluation is, it
will be essentially
subjective.
VS. ANALYSIS
• focuses on the materials and
it aims to provide an
objective analysis of them.
• It ‘asks questions about what
the materials contain, what
they aim to achieve and what
they ask learners to do’.
How can we use both?
Littlejohn (2011)
• Analysis of the target situation of use.
• Materials analysis.
• Match and evaluation (determining the appropriacy
of the materials to the target situation of use).
• Action.
 Principles in Materials
Evaluation
I. The evaluator’s theory of learning and teaching
II. Learning theory
I. The evaluator’s theory of learning and
teaching
• All teachers develop theories of learning and teaching which
they apply in their classrooms (even though they are often
unaware of doing so).
• The following are the theories of Brian Tomlinson
• Language learners succeed best if learning is a positive, relaxed and
enjoyable experience.
• Language teachers tend to teach most successfully if they enjoy their
role and if they can gain some enjoyment themselves from the materials
they are using.
• Learning materials lose credibility for learners if they suspect that the
teacher does not value them.
• Each learner is different from all the others in a class in terms of his or
her personality, motivation, attitude, aptitude, prior experience, interests,
etc.
• Each learner varies from day to day in terms of motivation, attitude, mood,
perceived needs and wants, enthusiasm and energy.
• There are superficial cultural differences between learners from different
countries (and these differences need to be respected and catered for) but
there are also strong universal determinants of successful language
teaching and learning.
• Successful language learning in a classroom (especially in large classes)
depends on the generation and maintenance of high levels of energy.
• The teacher is responsible for the initial generation of energy in a lesson;
good materials can then maintain and even increase that energy.
• Learners only learn what they really need or want to learn.
• Learners often say that what they want is focused language practice but
they often seem to gain more enjoyment and learning from activities
which stimulate them to use the target language to say something they
really want to say.
• Learners think, say and learn more if they are given an experience or text
to respond to than if they are just asked for their views, opinions and
interests.
• The most important thing that learning materials have to do is to help the
learner to connect the learning experience in the classroom to their own
life outside the course.
• The more novel (or better still bizarre) the learning experience is the
more impact it is likely to make and it is more likely to contribute to
long- term acquisition.
• The most important result that learning materials can achieve is to
engage the emotions of learners. Laughter, joy, excitement, sorrow and
anger can promote learning. Neutrality, numbness and nullity cannot.
II. Learning theory

• There are many types of learning theories.

• Research into learning is controversial as there are so many
variables involved and local circumstances often make
generalization precarious.
 Deep Processing Intake

• Such processing is semantic in that the focus

of the learner is on the meaning of the intake
and in particular on its relevance to the
learner.
 Affective Engagement

• Emotions must be ‘considered an essential part of

learning’ as they ‘are the very centre of human
mental life . . . [they] link what is important for us
to the world of people, things and happenings’
 Making Mental Connection

• In order for learning to be successful, connections

need to be made between:
• the new and the familiar,
• what is being learned and the learner’s life, and
• the learning experience and its potential value in the
future.
 Experiential Learning
(Kolb, 1984; Kelly, 1997; Tomlinson and Masuhara, 2000;
Kolb and Kolb, 2009).
• Is essential (though not necessarily sufficient) and, in
particular, apprehension should come to the learner before
comprehension
 Multidimensional Processing of Intake

• is essential for successful learning and involves the

learner creating a mental representation of the intake
through such mental processes as sensory imaging
 Features that contribute to successful personal voice
include:

 Informal discourse features

 The active rather than the passive voice
 Concreteness
Inclusiveness
 Sharing experiences and opinions
Sometimes including casual redundancies rather than always
being concise. (Tomlinson, 2001b)
 Second Language
Acquisition Research (SLA)
 Principles of second language
acquisition
Materials should:
• achieve impact
• help learners to feel at ease
• help the learners to develop confidence
• require and facilitate learner self-investment
• expose the learners to language in authentic use.
• provide the learners with opportunities to use the target language
to achieve communicative purposes.
• take into account that the positive effects of instruction are
usually delayed.
• take into account that learners differ in learning styles.
• take into account that learners differ in affective attitudes.
• maximize learning potential.
• provide opportunities for outcome feedback
In addition to the requirements listed in Tomlinson (2011a)
materials should:

• help the learner to develop cultural awareness and sensitivity

(Tomlinson, 2000b; Byram and Masuhara, 2013);
• reflect the reality of language use;
• help learners to learn in ways similar to the circumstances in
which they will have to use the language;
• help to create readiness to learn.
• achieve affective engagement.
Richards (2001) ‘qualities each unit in the materials should
reflect’:

• Gives learners something they can take away from the lesson.
• Gives learners a sense of achievement.
• Practises learning items in an interesting and novel way.
• Provides a pleasurable learning experience.
• Provides opportunities for individual practice.
• Provides opportunities for personalization.
• Provides opportunities for self-assessment of learning
Types of Materials
Evaluation
Pre-use evaluation
• involves making predictions about the potential
value of materials for their users.
Whilst-use evaluation
• involves measuring the value of materials while using them
or while observing them being used.
• It can be more objective and reliable than pre-use evaluation
as I makes use of measurement rather than prediction.
Post-use evaluation
• is probably the most valuable (but least administered)
type of evaluation as it can measure the actual effects
of the materials on the users.
• What do the learners know which they did not know before starting
to use the materials?
• What do the learners still not know despite using the materials?
• What can the learners do which they could not do before starting to
use the materials?
• What can the learners still not do despite using the materials?
• To what extent have the materials prepared the learners for their
examinations?
• To what extent have the materials prepared the learners for their
post-course use of the target language?
• What effect have the materials had on the confidence of the
learners?
• What effect have the materials had on the motivation of the
learners?
• To what extent have the materials helped the learners to become
independent learners?
• Did the teachers find the materials easy to use?
• Did the materials help the teachers to cover the syllabus?
• Did the administrators find the materials helped them to
standardize the teaching in their institution?
Standard approaches to
Materials Evaluation
 Standard Approaches to Materials
Evaluation
• the writer explained about the lists of criteria to conducting
an evaluation.
• As we know that most of the literature on materials
development has so far focused on the materials evaluation.
Developing Criteria for
Materials Evaluation
 Developing criteria for materials evaluation

• It is extremely useful to develop a set of formal criteria for

use on a particular evaluation and then to use that set as a
basis for developing subsequent context-specific sets
1. Brainstorm a list of universal criteria
• Universal criteria are those which would apply to any
language learning materials anywhere for any learners.
 Examples of universal criteria would be:

• Do the materials provide useful opportunities for the learners

to think for themselves?
• Are the target learners likely to be able to follow the
instructions?
• Are the materials likely to cater for different preferred
learning styles?
2. Subdivide some of the criteria

• If the evaluation is going to be used as a basis for revision or

adaptation of the materials, or if it is going to be a formal
evaluation and is going to inform important decisions, it is
useful to subdivide some of the criteria into more specific
questions.
Are the instructions:

• succinct?
• sufficient?
• self-standing?
• standardized?
• separated?
• sequenced?
• staged?
3. Monitor and revise the list of universal criteria

• Is each question an evaluation question?

For example: (Does each unit include a test?) Analysis
For example: (To what extent are tests likely to provide useful
learning experiences?) Evaluation
Does each question only ask one question?
• For xample: Is it attractive? Given the average age of your students,
would they enjoy using it?’
Is each question answerable?
• For example: ‘Is it culturally acceptable?’
Is each question free of dogma?
Is each question reliable in the sense that other evaluators
would interpret it in the same way?
4. Categorize the list
• It is very useful to rearrange the random list of universal
criteria into categories which facilitate focus and enable
generalizations to be made.
 Possible categories for universal criteria would be:

• Learning Principles
• Cultural Perspective
• Topic Content
• Teaching Points
• Texts
• Activities
• Methodology
• Instructions
• Design and Layout
5. Develop media-specific criteria

• These are criteria which ask questions of particular relevance

to the medium used by the materials being evaluated.
Examples of such criteria would be:

• Is it clear which sections the visuals refer to?

• Is the sequence of activities clearly signalled?

• Are the different voices easily distinguished?

• Do the gestures of the actors help to make the language

meaningful in realistic ways?
6. Develop content-specific criteria
• These are criteria which relate to the topics and/or teaching points of
the materials being evaluated.

• Thus there would be a set of topic related criteria which would be

relevant to the evaluation
7. Develop age-specific criteria

• These are criteria which relate to the age of the target

learners. Whether it is suitable for 5-year-olds, for 10-year-
olds, for teenagers, for young adults or for mature adults.
 8. Develop local criteria
• These are criteria which relate to the actual or potential environment
of use.

• They are actually related to measuring the value of the materials for
particular learners in particular circumstances.
9. Develop other criteria
• Other criteria which it might be appropriate to develop could
include teacher-specific, administrator-specific, gender-
specific, culture-specific or L1-specific criteria and,
especially in the case of a review for a journal, criteria
assessing the match between the materials and the claims
made by the publishers for them.
10. Trial the criteria
• It is always important to trial the criteria to ensure that the
criteria are sufficient, answerable, reliable and useful.
11. Conducting the evaluation
• make sure that there is more than one evaluator;
• discuss the criteria to make sure there is equivalence of
interpretation;
• answer the criteria independently and in isolation from the
other evaluator(s);
• focus in a large evaluation on a typical unit for each level
(and then check its typicality by reference to other units);
• give a score for each criterion (with some sets of criteria
weighted more heavily than others);
• write comments at the end of each category;
• at the end of the evaluation aggregate each evaluator’s scores
for each criterion, category of criteria and set of criteria and
then average the scores;
• record the comments shared by the evaluators;
• write a joint report.
Informal evaluations
• Brainstorm beliefs
• Decide on shared beliefs
• Convert the shared beliefs into universal criteria
• Write a profile of the target learning context for the materials
• Develop local criteria from the profile
• Evaluate and revise the universal and the local criteria
• Conduct the evaluation
Conclusion
 Conclusion
• Materials evaluation is initially a time-consuming and difficult undertaking.

• Approaching it in the principled, systematic and rigorous ways suggested

above can not only help to make and record vital discoveries about the
materials being evaluated but can also help the evaluators to learn a lot
about materials, about learning and teaching and about themselves.