Approaches To Evaluation: Chapter Overview

11 APPROACHES TO EVALUATION
CHAPTER OVERVIEW
This chapter examines the following aspects of approaches to evaluation:
• The focus of evaluation • Process-focused evaluation

• Audience for evaluation • Descriptive evaluation
• Quantitative and qualitative approaches • 9LÅLJ[P]LL]HS\H[PVU
• Product-focused evaluation • 0TWSLTLU[PUNYLÅLJ[P]LL]HS\H[PVU
• Formative and summative evaluation
Case study 19 Evaluating an in-service program for English language teachers .LVќYL`*YL^LZ
Case study 20 Evaluating the content of an EAP program 1VUH[OHU5L^[VU
Case study 21 Evaluating an English course for tertiary-level learners +H]PK*YHIIL
Introduction
We have discussed two ways of thinking about curriculum in this book. One is the dominant
philosophy in educational planning that considers curriculum as a set of processes to develop
LѝJPLU[HUKLќLJ[P]L^H`ZVMHJOPL]PUNKLÄULKSLHYUPUNV\[JVTLZ;OLYPNVY^P[OÔPJOJ\YYPJ\S\T
development is carried out – drawing on procedures that include needs analysis, planning learning
V\[JVTLZKLZPNUPUNHJV\YZLHUKZ`SSHI\ZMYHTL^VYRHUK\ZPUNLќLJ[P]LTL[OVKZVM[LHJOPUNHUK
learning – will determine the success of the outcomes. Curriculum processes are seen as ways of
bringing order, control, and direction into language teaching and language course design. From this
perspective, evaluation is viewed as essential to maintaining the quality and success of a language
WYVNYHT(Z2PLS`JVTTLU[Z !¸BWYVNYHTL]HS\H[PVUDPZHIV\[LќLJ[P]LULZZHUK[YHKP[PVUHSS`
the answer has been sought in terms of test results, a language learning theory, or a particular
syllabus.”
The alternative understanding of curriculum that we discussed in Chapters 1 and 9 focuses more on
[OL]HS\LZ[OH[[OLJ\YYPJ\S\TYLÅLJ[ZHUKOV^[OLJ\YYPJ\S\TPZYLHSPaLK[OYV\NO[OLWYVJLK\YLZHUK
HJ[P]P[PLZVM[LHJOPUNHUKSLHYUPUN-YVT[OPZWLYZWLJ[P]LL]HS\H[PVUOHZ]LY`KPќLYLU[NVHSZ0[ZNVHS
is exploration and understanding.
>L ^PSS JVUZPKLY IV[O HWWYVHJOLZ [V L]HS\H[PVU PU [OPZ JOHW[LY HUK JOHYHJ[LYPaL LќLJ[P]LULZZ
oriented evaluation as WYVK\J[MVJ\ZLKL]HS\H[PVU, and the alternative approach as WYVJLZZMVJ\ZLK
L]HS\H[PVU. However, before comparing these two approaches, it is necessary to clarify some basic
issues that arise in planning an evaluation.
What kinds of evaluations are conducted in your institution? What are their main purposes?
277
278 • Curriculum Development in Language Teaching
11.1 The focus of evaluation

,]HS\H[PVUTH`MVJ\ZVUTHU`KPќLYLU[HZWLJ[ZVMHSHUN\HNLWYVNYHTPUJS\KPUN[OLMVSSV^PUN!
• ;OLV\[JVTLZ! to discover what gains in performance the students achieved.
• *\YYPJ\S\TKLZPNU![VHZZLZZ[OLX\HSP[`VMWYVNYHTWSHUUPUNHUKVYNHUPaH[PVU
• ;OLVYL[PJHS\UKLYWPUUPUNZ![VÄUKV\[PM[OL`YLÅLJ[J\YYLU[\UKLYZ[HUKPUNZVMZLJVUKSHUN\HNL
acquisition.
• ;OLZ`SSHI\ZHUKWYVNYHTJVU[LU[!to determine how relevant and engaging it is, how easy or
KPѝJ\S[HUKOV^Z\JJLZZM\S[LZ[ZHUKHZZLZZTLU[WYVJLK\YLZ^LYL
• ;LHJOPUNTL[OVKZ!to provide insights about the extent to which the methods were suitable and
were implemented appropriately.
• 4H[LYPHSZVMPUZ[Y\J[PVU![VÄUKV\[PM[OL[LHJOPUNTH[LYPHSZHUKYLZV\YJLZHYLYLSL]HU[H[[OL
right level, and well designed.
• <ZLVMYLZV\YJLZ! how textbooks and technology are used.
• ;OL[LHJOLYZ!to learn how well they conducted their teaching and what their perceptions are of
the program.
• *SHZZYVVTSHUN\HNL! the nature of the language that is used by teachers and students.
• ;LHJOLY[YHPUPUN!to assess whether the training teachers have received is adequate for the program.
• ;OLPUZ[P[\[PVU! to assess to what extent the culture and practices of the institution facilitate
Z\JJLZZM\SJ\YYPJ\S\TPTWSLTLU[H[PVU"[VHZZLZZ[OLL_[LU[[VÔPJO[OLPUZ[P[\[PVUYLÅLJ[Z
standards of best practice. (See Appendix 1 for a statement of best practice at the
institutional level.)
• ;OLZ[\KLU[Z![VÄUKV\[ÔH[[OL`SLHYULKMYVT[OLWYVNYHT[OLPYWLYJLW[PVUZVMP[HUKOV^
they participated in it.
• 4VUP[VYPUNVMW\WPSWYVNYLZZ!to identify any problems that students encountered during the
program.
Choose one item from the list above? How could you collect information needed to explore
this issue?
/V^L]LYHZ2PLS`HUK9LH+PJRPUZLTWOHZPaL [OLNVHSZVML]HS\H[PVUHYL[VSLHYU from

the information gathered during an evaluation and to put it to good use: evaluation should provide
[OL IHZPZ MVY ¸KLJPZPVUTHRPUN HJ[PVU HUK HJ[\HS \[PSPaH[PVU¹ *YP[PJHS X\LZ[PVUZ [OH[ ULLK [V IL
answered include what can be learned from an evaluation? What are the implications for practice?
What decisions and recommendations can be made? Who will be responsible for recommended
actions, and what will be the impact of any changes made?
In evaluating a curriculum, a number of questions can be asked of the curriculum as a whole. For
example:
• +VLZ[OLYHUNLVMJV\YZLZVќLYLKJVYYLZWVUK[V[OLULLKZVMSLHYULYZ&
• Is the curriculum coherent, and do the courses represent a rational approach to achieving the
institution’s mission?
• Have courses been developed based on sound educational principles, with due attention given
[VYLJVNUPaLKJ\YYPJ\S\TKL]LSVWTLU[WYVJLZZLZ&
11 Approaches to evaluation • 279
• Have course descriptions been developed, including aims, goals, syllabuses, learning outcomes,
HUKJV\YZLVYNHUPaH[PVU&
• Are teaching materials and tests of high quality, have they been carefully selected or developed,
and are they regularly reviewed and revised?
• Are mechanisms in place to monitor the quality of teaching and learning?
• Is the curriculum subject to ongoing review and renewal? Is there ongoing interest in identifying
strengths and weaknesses and bringing about improvements in all aspects of the curriculum?
Monitoring the quality of teachers in a program is also often central to curriculum evaluation and
involves a number of issues. These include KL[LYTPUPUN [OL W\YWVZL VM HWWYHPZHS (e.g., to reward
teachers for good performance, to help identify needs for further training, to reinforce the need for
JVU[PU\V\ZZ[HќKL]LSVWTLU[[VOLSWPTWYV]L[LHJOPUN[VWYV]PKLHIHZPZMVYJVU[YHJ[YLUL^HSHUK
promotion, or to demonstrate an interest in teachers’ performance and development). Other important
aspects of monitoring involve KLJPKPUNVU[OLMVJ\ZVMHWWYHPZHS (e.g., lesson plans, teaching skills,
use of resources such as textbooks and technology, teacher discourse, teacher-made classroom
materials, course outlines and handouts, class assignments, as well as participation in professional
development activities) and OV^[OLHWWYHPZHS^PSSILJVUK\J[LK(e.g., appraisal by a supervisor or
colleague; self-appraisal using lesson reports, a teaching journal, audio/video recordings of lessons
or through the use of student appraisal).
11.2 Audience for evaluation

;OLYL HYL THU` KPќLYLU[ SL]LSZ VM PU]VS]LTLU[ PU SHUN\HNL[LHJOPUN WYVNYHTZ HUK [OPZ JYLH[LZ
KPќLYLU[RPUKZVMH\KPLUJLZMVYL]HS\H[PVU-VYL_HTWSLPUKL]LSVWPUNHUL^[L_[IVVRZLYPLZMVYW\ISPJ
ZJOVVSZM\UKLKI`[OLTPUPZ[Y`VMLK\JH[PVUVѝJLYZPU[OLTPUPZ[Y`ÔVTHÙV[ILZWLJPHSPZ[ZPU
language teaching) might be primarily interested in how the money provided for the project is spent
and whether all components of the project (student books, teacher guides, and workbooks) are
H]HPSHISLPUZJOVVSZI`HZWLJPÄJKH[L;LHJOLYZ[LHJOPUN[OLTH[LYPHSZTPNO[ILWYPTHYPS`JVUJLYULK
[OH[ [OL IVVRZ WYV]PKL Z\ѝJPLU[ TH[LYPHS MVY HSS [OL JSHZZLZ VU [OL ZJOVVS [PTL[HISL (U V\[ZPKL
consultant might be interested in the design of the materials and the kind of classroom interaction
and language practice they provide for. Vocational training centers might be interested in whether
the course prepares school leavers for vocational training programs conducted in English. Therefore,
L]HS\H[PVUOHZ[VZH[PZM`HSSPU[LYLZ[LKWHY[PLZ;OLX\LZ[PVUZ[OH[KPќLYLU[Z[HRLOVSKLYZHZR^PSSHSZV
KPќLYHUKTH`PUJS\KL[OLMVSSV^PUN!
:[\KLU[Z!
• What did I learn?
• How well did I do compare to others?
• How well will I rate this course?
• How will this help me in the future?
• Do I need another course?
;LHJOLYZ!
• How well did I teach?
• What did my students learn?
• >LYLT`Z[\KLU[ZZH[PZÄLK^P[O[OLJV\YZL&
• How useful were the materials and course work?

• /V^LќLJ[P]L^HZ[OLJV\YZLVYNHUPaH[PVU&
*\YYPJ\S\TKL]LSVWLYZ!
• Is the design of the course and materials appropriate?
• What aspects of the course need replacing or revising?
• Do teachers and students respond favorably to the course?
• Do teachers need additional support with the course?
(KTPUPZ[YH[VYZ!
• Was the time frame of the course appropriate?
• Were the management and monitoring of the course successful in identifying and rectifying
problems?
• Were clients’ expectations met?
• Were testing and assessment procedures adequate?
• Were resources made use of?
:WVUZVYZ!
• >HZ[OLJVZ[VM[OLJV\YZLQ\Z[PÄLK&
• Did the course deliver what was promised?
• Was the course well managed?
• Is the reporting of the course adequate?
Can you suggest other questions an evaluation may seek to answer?
As we see above, two types of participants are typically involved in evaluation, and these may be
considered PUZPKLYZ and V\[ZPKLYZ. Insiders refers to teachers, students, and anyone else closely
involved in the development and implementation of the program. For example, formative evaluation,
discussed below, is often carried out by teachers who can monitor a course as it develops to check
[OLL_[LU[[VÔPJOP[PZ^VYRPUNÔH[KPѝJ\S[PLZHYLLUJV\U[LYLKOV^LќLJ[P]L[OLTH[LYPHSZHYL
HUK ÔH[ TVKPÄJH[PVUZ ^V\SK LUZ\YL [OL ZTVV[O Y\UUPUN VM [OL WYVNYHT :[\KLU[Z HYL VM[LU RL`
participants in the summative evaluation (discussed below) of the program, providing evidence of their
NHPUZPUSHUN\HNLWYVÄJPLUJ`HUKJVTWSL[PUNL]HS\H[PVUZVU[OL^H`[OLWYVNYHT^HZ[H\NO[HUK[OL
relevance of what they have learned to their needs. The involvement of key insiders in the process of
designing and carrying out the evaluation is often an important factor in the success of the evaluation
because, as a consequence, they will have a greater degree of commitment to acting on its results.
(UL_HTWSLVMHU¸PUZPKLY¹WLYZWLJ[P]LVUHJV\YZLPZNP]LUPU/HYK`ÔV\ZLKHULUKVM
course questionnaire to obtain students’ feedback on a course for adults with the following questions:
1. What did you like about this course?

2. What did you not like about it?
3. Are there any activities or skills you think we should have spent more time on?
4. Are there any activities or skills you think we should have spent less time on?
5. In which area(s) do you feel that you made the most progress?
6. Was the total number of hour per day __ too few? __ just right? __ too many?
7. Was the amount of homework __ not enough? __ just right __? __ too much?
8. Other comments/ suggestions?
Do you think open-ended questions such as those above are preferable to providing choices
for the students to check for each question?
Outsiders refers to those other participants who are not involved in the program itself, and who may
be asked to give an objective view of aspects of the program. They may be consultants, supervisors,
or administrators whose job it is to supplement the teachers’ perceptions of what happened in a
course with independent observation and opinion.
11.3 Quantitative and qualitative approaches

;^VKPќLYLU[RPUKZVMPUMVYTH[PVUHYLPU]VS]LKPUL]HS\H[PVU¶X\HU[P[H[P]L and X\HSP[H[P]L. Quantitative
measurement refers to the measurement of something that can be expressed numerically. Many
tests are designed to collect information that can be readily counted and presented in terms of
frequencies, rankings, or percentages. Other sources of quantitative information are checklists,
surveys, and self-ratings. Quantitative data seek to collect information from a large number of people
VUZWLJPÄJ[VWPJZHUKJHUNLULYHSSÌLHUHSàLKZ[H[PZ[PJHSS`ZV[OH[JLY[HPUWH[[LYUZHUK[LUKLUJPLZ
LTLYNL;OLPUMVYTH[PVUJVSSLJ[LKJHUILHUHSàLKMHPYS`ZPTWSÌLJH\ZLZ\IQLJ[P]LKLJPZPVUZHYLUV[
\Z\HSS`PU]VS]LK;YHKP[PVUHSS`X\HU[P[H[P]LKH[HHYLYLNHYKLKHZ¸YPNVYV\Z¹VYJVUMVYTPUN[VZJPLU[PÄJ
WYPUJPWSLZ VM KH[H JVSSLJ[PVU [OV\NO [OL SPTP[H[PVUZ VM X\HU[P[H[P]L PUMVYTH[PVU HYL HSZV YLJVNUPaLK"
hence the need to complement such information with qualitative information.
Qualitative measurement, on the other hand, refers to measurement of something that cannot be
expressed numerically and that depends more on subjective judgment or observation. Information
obtained from classroom observation, interviews, journals, logs, and case studies is generally qualitative.
Qualitative approaches are more holistic and naturalistic than quantitative approaches and seek to collect
information in natural settings for language use and on authentic tasks rather than in test situations.
They are normally more exploratory and seek to collect a large amount of information from a fairly small
U\TILYVMJHZLZ;OLPUMVYTH[PVUVI[HPULKPZTVYLKPѝJ\S[[VHUHSàLILJH\ZLP[PZVM[LUVWLULUKLK
and must be coded or interpreted. Qualitative data are sometimes regarded as “soft” or less rigorous
than quantitative data, but such information is essential in many stages of program evaluation.
In language program evaluation, both quantitative and qualitative approaches to collecting information
HYLVM[LU\ZLKILJH\ZL[OL`ZLY]LKPќLYLU[W\YWVZLZHUKJHUJVTWSLTLU[LHJOV[OLY-VYL_HTWSL
in assessing students’ achievement at the end of a course on spoken English, the following
procedures might be used:
• 7LYMVYTHUJLVUHUVYHSWYVÄJPLUJ`[LZ[X\HU[P[H[P]L
• Observation of students’ performance on classroom tasks with evaluation using a holistic rating
scale (qualitative).
• Students’ self-assessment of improvement in their speaking skills (qualitative).
Do you think open-ended questions such as those above are preferable to providing choices
for the students to check for each question?
11.4 Product-focused evaluation

;^VTHQVYW\YWVZLZMVYSHUN\HNLWYVNYHTL]HS\H[PVUHYL\Z\HSS`PKLU[PÄLKPUYLSH[PVU[VJ\YYPJ\S\T
as product: program HJJV\U[HIPSP[` and program KL]LSVWTLU[. Accountability refers to the extent
to which those involved in a program are answerable for the quality of their work. Accountability-
VYPLU[LKL]HS\H[PVU\Z\HSS`L_HTPULZ[OLLќLJ[ZVMHWYVNYHTVYWYVQLJ[H[ZPNUPÄJHU[LUKWVPU[ZVMHU
LK\JH[PVUHSJ`JSLHUKPZ\Z\HSS`JVUK\J[LKMVY[OLILULÄ[VMHUL_[LYUHSH\KPLUJLVYKLJPZPVUTHRLY
Development-oriented evaluation, by contrast, is designed to improve the quality of a program as it
PZILPUNPTWSLTLU[LK0[TH`PU]VS]LZ[HќÔVHYLWHY[PJPWH[PUNPU[OLWYVNYHTHZ^LSSHZV[OLYZÔV
HYLUV[HUKTHÒH]LH[LHJOLYKL]LSVWTLU[MVJ\Z>LPYHUK9VILY[Z
Weir and Roberts (1994, 42) describe a view of evaluation typical of a product-focused orientation
JOHYHJ[LYPaLKI`!
• a need for both insider and outsider commitment and involvement to ensure adequate evaluation;
• a central interest in improvement, as well as the demonstration of the “product value” of a
program or project or their components;
• an associated commitment to a deeper professional understanding of the processes of
educational change, as well as the results of that change;
• systematic documentation for evaluation purposes both during implementation and at the
beginning and end of a program or project’s life;
• a willingness to embrace both qualitative and quantitative methodology appropriate to the
purpose of the evaluation and the context under review.
From this perspective, evaluation is an aspect of quality assurance and involves making measurements
HUKJVTWHYPZVUZ^P[OÄUKPUNV\[OV^^LSS[OLJ\YYPJ\S\TZLY]LZP[ZZ[HRLOVSKLYZHUKHJOPL]LZP[Z
V\[JVTLZ2L`^VYKZPU[OPZWYVJLZZHYLLќLJ[P]LULZZHUKLѝJPLUJ`HZQ\KNLKI`L_[LYUHSNVHSZHUK
standards. Relevant questions have to do with end–means relations. Following Tyler (1949), from this
perspective evaluation seeks to answer the following questions:
• How successful was the curriculum in achieving its learning outcomes?
• Was it based on a correct understanding of language and language learning?
• What did students learn?
• How well did the syllabus, the materials, teaching resources, and tests work?
• What was the quality of teaching and how did it contribute to the course outcomes?
• /V^ZH[PZÄLK^LYLKPќLYLU[Z[HRLOVSKLYZLN[LHJOLYZHKTPUPZ[YH[VYZZ[\KLU[ZWHYLU[Z
employers)?
• Does the curriculum compare favorably with others of its kind?
• How could it be improved?
(UZ^LYPUN [OLZL X\LZ[PVUZ PU]VS]LZ JVSSLJ[PUN PUMVYTH[PVU HIV\[ KPќLYLU[ HZWLJ[Z VM H SHUN\HNL
WYVNYHTPUVYKLY[V\UKLYZ[HUKOV^[OLWYVNYHT^VYRZHUKOV^Z\JJLZZM\SS`LUHISPUNKPќLYLU[RPUKZ
of decisions to be made about the program, such as whether the program responds to learners’ needs,
whether further teacher training is required for teachers working in the program, or whether students are
SLHYUPUNZ\ѝJPLU[S`MYVTP[(VRPUV[LK[OH[[OPZHWWYVHJO[VL]HS\H[PVUPZIHZLKVUZL]LYHSHZZ\TW[PVUZ!
• The most valuable form of knowledge is based on empirical evidence.
• Empirical data is valued more highly than other forms of data, and the “harder” they are, the better.
• .LULYHSPaHIPSP[`ZOV\SKILWVZZPISLMYVT[OLKH[H
• :JPLU[PÄJRUV^SLKNLPZVIQLJ[P]LUL\[YHSHUKMYLLMYVT]HS\LZ
• ;OLHPTPZ[VYLTV]L[OLPUÅ\LUJLVMO\THUZ\IQLJ[P]P[`
With the global spread of English and the substantial investment required to achieve goals for national
education in most countries, curriculum evaluation has become of increasing interest to governments,
educators, and curriculum planners. Funding for national curriculum projects in many parts of the
world is often linked to a requirement to provide evaluation reports that demonstrate accountability,
that help guide improvement of ongoing projects, and that document what happens in curriculum
projects. Increasingly, schools, program administrators, and teachers have had to be accountable
for the funds they received or for the programs they have been responsible for, and this has created
the need for an understanding of the nature of curriculum evaluation. The scope of evaluation has
moved from a concern with test results to the need to collect information and make judgments about
all aspects of the curriculum, from planning to implementation (Hewings and Dudley-Evans 1996).
11.5 Formative and summative evaluation

Two major types of product-centered evaluation can be distinguished, and these are known as
MVYTH[P]L and Z\TTH[P]L L]HS\H[PVU. Formative evaluation is carried out as part of the process of
WYVNYHTKL]LSVWTLU[PUVYKLY[VÄUKV\[ÔH[PZ^VYRPUN^LSSÔH[PZUV[HUKÔH[JHUILKVUL[V
ensure successful achievement of outcomes. It focuses on ongoing development and improvement
of the program. Typical issues that relate to formative evaluation include the following:
• +PZ[YPI\[PVUVM[PTLVUKPќLYLU[J\YYPJ\S\TVIQLJ[P]LZ
• Placement of students at an appropriate level.
• Suitability of the textbook and teaching resources.
• Appropriateness of use of materials and technology.
• Appropriateness of teaching methods and procedures.
• Quality of language used by students.
• +PѝJ\S[PLZL_WLYPLUJLKI`Z[\KLU[Z
• Students’ level of engagement with the course.
• Balance between exercises and tasks.
• Pacing of the course.
• Suitability of ongoing assessment procedures.
Information collected during formative evaluation is used to address problems that have been
PKLU[PÄLKHUK[VPTWYV]L[OLKLSP]LY`VM[OLWYVNYHTHZ^LZLLPU[OLMVSSV^PUNL_HTWSLZ
,_HTWSL! During the implementation of a new primary course in an EFL context, it is found that rather than
using the task-oriented communicative methodology that provides the framework for the course, a number
of teachers are resorting to a teacher-dominated drill and practice mode of teaching that is not in harmony
with the course philosophy. In order to address this problem, a series of Saturday morning workshops are
held to identify the kinds of problems teachers are having with the materials. Videos are used to model more
appropriate teaching strategies, and teachers agree to attempt to implement in their classrooms some of the
techniques they have seen demonstrated and to report back on their experiences at subsequent workshops.
,_HTWSL! A few weeks after a course on integrated skills has started, it is found that there are
KPќLYLU[WLYJLW[PVUZVMÔH[[OLWYPVYP[PLZPU[OLJV\YZLHYL;LHJOLYZHYLZWLUKPUN]LY`KPќLYLU[
HTV\U[ZVM[PTLVUKPќLYLU[JVTWVULU[ZVM[OLJV\YZLHUKHYLLTWOHZPaPUNKPќLYLU[[OPUNZ(
series of meetings are held to review teachers’ understanding of the course objectives and to further
JSHYPM`[OL^LPNO[PUN[OH[ZOV\SKILNP]LU[VKPќLYLU[JV\YZLJVTWVULU[Z7LLYVIZLY]H[PVUPZ[OLU
suggested as a way for teachers to compare teaching styles and priorities and to enable them to
achieve a consensus concerning teaching practices.
,_HTWSL! A ten-week course on spoken English has been started for a group of low-level learners.
Pronunciation is not a major element of the course, because it is assumed that most pronunciation
problems will sort themselves out after a few weeks. However, four weeks after the course has
commenced, teachers report that a number of students have persistent and major pronunciation
problems that the course is not addressing. It is decided to refocus one section of the course to
include a pronunciation component. Individual diagnostic sessions are held with students who have
the most serious pronunciation problems, and laboratory work as well as classroom time is allotted to
systematic pronunciation work for the remainder of the course.
The other aspect of a product-focused or ends–means approach to evaluation is Z\TTH[P]L

L]HS\H[PVU:\TTH[P]LL]HS\H[PVUPZJVUJLYULK^P[OKL[LYTPUPUN[OLLќLJ[P]LULZZVMHWYVNYHTP[Z
LѝJPLUJ`HUKP[ZPTWHJ[HUK[HRLZWSHJLHM[LYHWYVNYHTOHZILLUPTWSLTLU[LK)LSV^HYLZVTL
of the issues that could be addressed:
• ;OLLќLJ[P]LULZZVMHJV\YZLHZYLÅLJ[LKPUP[ZYLZ\S[ZVYV\[JVTLZ
• Adequacy of program objectives.
• Adequacy of course design and syllabus.
• ,ќLJ[P]LULZZVMTH[LYPHSZYLZV\YJLZHUK[LZ[Z
• :\P[HIPSP[`HUKLќLJ[P]LULZZVM[LHJOPUNTL[OVKZ
• Perceptions of students and teachers on the course.
• Reports of problems experienced during the course.
0U VYKLY [V KLJPKL PM H JV\YZL PZ LќLJ[P]L JYP[LYPH MVY LќLJ[P]LULZZ ULLK [V IL PKLU[PÄLK ;OLYL HYL
THU` KPќLYLU[ TLHZ\YLZ VM H JV\YZL»Z LќLJ[P]LULZZ HUK LHJO TLHZ\YL JHU IL \ZLK MVY KPќLYLU[
purposes. For example:
4HZ[LY`VMVIQLJ[P]LZ6UL^H`VMTLHZ\YPUN[OLLќLJ[P]LULZZVMHJV\YZLPZ[VHZR¸/V^MHYOH]L
the objectives been achieved?” Each objective or learning outcome in the course is examined, and
criteria for successful achievement of each objective are chosen. In a course on speaking skills,
for example, an objective might be: 0UNYV\WKPZJ\ZZPVUZZ[\KLU[Z^PSSSPZ[LU[VHUKYLZWVUK[V[OL
VWPUPVUZVMV[OLYZPU[OLPYNYV\WThe extent to which the students have mastered this objective at the
end of the course can be assessed by the teacher’s observing students during group discussions and
recording on a scale the extent to which they listen and respond to opinions. If students’ performance
VU [OPZ VIQLJ[P]L PZ WVVY YLHZVUZ ^V\SK OH]L [V IL PKLU[PÄLK 7LYOHWZ MVY L_HTWSL PUZ\ѝJPLU[
opportunities were provided in the course for students to practice this task, or perhaps the materials
YLSH[PUN[V[OPZVIQLJ[P]L^LYL[VVKPѝJ\S[VYUV[Z\ѝJPLU[S`PU[LYLZ[PUN
/V^L]LY THZ[LY` VM VIQLJ[P]LZ KVLZ UV[ WYV]PKL H M\SS WPJ[\YL VM [OL LќLJ[P]LULZZ VM H JV\YZL
6IQLJ[P]LZJHUILHJOPL]LKKLZWP[LKLMLJ[ZPUHJV\YZL:[\KLU[ZTHÒH]LYLHSPaLK[OH[[OL[LHJOPUN
VYTH[LYPHSZ^LYLWVVYVYPUZ\ѝJPLU[HUKZVZWLU[HSV[VML_[YH[PTLPUWYP]H[LZ[\K`VYVU[OL0U[LYUL[
11 Approaches to evaluation •
to compensate for it. Or perhaps mastery of an objective was achieved, but the same objective could
have been covered in half the amount of time devoted to it. Or the program might have achieved its
learning outcomes, but students have a very negative perception of it because it was not stimulating
or the pacing was inappropriate.
7LYMVYTHUJL VU [LZ[Z Apart from the relatively informal way of assessing mastery of objectives,
formal tests are probably the commonest means used to measure achievement. Such tests might
IL\UP[[LZ[ZNP]LUH[[OLLUKVMLHJO\UP[VM[LHJOPUNTH[LYPHSZJSHZZ[LZ[ZVYX\PaaLZKL]PZLKI`
teachers and administered at various stages throughout the course, or formal exit tests designed
to measure the extent to which objectives have been achieved. Achievement tests can have an
PTWVY[HU[^HZOIHJRLќLJ[VU[LHJOPUNHUKSLHYUPUN;OL`JHUPUÅ\LUJLKLJPZPVUTHRPUNVUJOHUNLZ
needed to a program, such as which objectives require more attention or revision. Brindley (1989b,
43) reports, however, that in programs he studied in Australia, teachers preferred to rely on informal
methods of ongoing assessment rather than formal exit tests, but that this sometimes meant that
teachers’ preferences for informal measures of assessment clashed with the requirements of the
institution:
This [approach] does not seem to be sufficiently explicit to meet the expectations and require-
ments of either administrators or learners for more formal information on learners’ achieve-
ment of a course or a unit … The informal methods of ongoing assessment provided by
teachers do not provide the kind of explicit information on achievement required by learners
and administrators.
+PќLYLU[ [`WLZ VM [LZ[Z HYL JVTTVUS` \ZLK [V TLHZ\YL JOHUNLZ PU SLHYUPUN H[ [OL LUK VY H[
intermediate stages) of a course. Examples include the following:
• Institutionally or teacher-prepared tests, such as exit tests, designed to measure what students
have learned in the course.
• 0U[LYUH[PVUHS[LZ[ZZ\JOHZ;6,-30,3;:VYH*HTIYPKNLWYVÄJPLUJ`[LZ[PM[OLZLHYLYLSH[LK[V
the course aims and content.
• Textbook tests such as those provided in teachers’ manuals or as part of a commercial course.
• Student records, such as information collected throughout the course on course work or
JVU[PU\V\ZHZZLZZTLU[;OPZPUMVYTH[PVUTHÌL\ZLK[VHYYP]LH[HÄUHSZJVYLVYNYHKLMVYH
Z[\KLU[^P[OV\[\ZPUNHÄUHS[LZ[
Tests can provide a direct measure of achievement, particularly if they are based on student
performance, that is, they are criterion-referenced. However, it is not always easy to be sure whether
changes in learning as measured by tests are a direct result of teaching or are linked to other factors.
4LHZ\YLZVMHJJLW[HIPSP[À course might lead to satisfactory achievement of its objectives and good
levels of performance on exit tests yet still be rated negatively by teachers or students. Alternatively,
if everyone liked a course and spoke enthusiastically of it, could this be more important than the fact
that half the students failed to reach the objectives? Acceptability can be determined by assessments
of teachers and students. Reasons for a course being considered acceptable or unacceptable might
YLSH[L[VZ\JOMHJ[VYZHZ[PTL[HISPUNJSHZZZPaLJOVPJLVMTH[LYPHSZVY[LHJOLYZ»[LHJOPUNZ[`SLZ
9L[LU[PVU YH[L VY YLLUYVSSTLU[ YH[L ( TLHZ\YL VM H JV\YZL»Z LќLJ[P]LULZZ [OH[ TH` IL PTWVY[HU[
from an institution’s point of view is the extent to which students continue in the course throughout
its duration and the percentage of students who reenroll for another course at the end. If there is a
ZPNUPÄJHU[KYVWV\[YH[LPZ[OPZ[Y\LVMV[OLYJV\YZLZPU[OLPUZ[P[\[PVUHUK[OLJVTT\UP[`VYPZP[H
factor of a given course only?
,ѝJPLUJ`VM[OLJV\YZLAnother measure of the success of a course is how straightforward the course

^HZ[VKL]LSVWHUKPTWSLTLU[;OPZTHÌLHYLÅLJ[PVUVM[OLU\TILYVMWYVISLTZ[OH[VJJ\YYLKK\YPUN
[OLJV\YZL[OL[PTLZWLU[VUWSHUUPUNHUKJV\YZLKL]LSVWTLU[[OLULLKMVYZWLJPHSPaLKTH[LYPHSZHUK
teacher training, and the amount of time needed for consultations and meetings.
/V^PZ[OLLќLJ[P]LULZZVMJV\YZLZPU`V\YPUZ[P[\[PVUKL[LYTPULK&
11.6 The importance of documentation

The more documentation that is available about a course, the easier it is to arrive at decisions about
it. Relevant documentation includes the following:
*V\YZLZ[H[PZ[PJZ!information on why students chose the course, student numbers, attendance,

JSHZZZPaLKYVWV\[Z\ZLVMMHJPSP[PLZZ\JOHZSPIYHY`VYZLSMHJJLZZJLU[LY;OPZRPUKVMPUMVYTH[PVU
provides an overview of the nature of the course and its mode of operation and may reveal
certain patterns or problems.
9LSL]HU[JV\YZLKVJ\TLU[Z!compilation of all relevant documentation about the course, such as

descriptions, publicity materials, statements of aims, objectives and syllabus, course materials,
teaching guides, newsletters, newspaper articles, reports of planning meetings.
*V\YZL^VYR!examples of tests, class assignments, students’ work.
>YP[[LU JVTTLU[Z! anything that has been written about the course by external assessors,
teachers, learners, managers.
0UZ[P[\[PVUHSKVJ\TLU[Z!anything that is available about the school or institution, hiring policy, job
descriptions, needs analyses that have been conducted, reports of previous courses.
*V\YZLYL]PL^Z!a written account of a course, prepared by the teacher or teachers who taught
[OLJV\YZL;OPZZOV\SKILIV[OKLZJYPW[P]LHUKYLÅLJ[P]L0[ZOV\SKILHUHJJV\U[VMOV^[OL
course progressed, what problems occurred, the perceived strengths and weaknesses of the
course, and suggestions for the future. A well-written review is a useful resource for others who
will teach the course and also provides a record of the course, but more often than not the need
for and the value of such reviews are overlooked. As Weir and Roberts (1994, 12) comment:
“One shudders to think how many times the wheel has been reinvented in ELT programs and
projects around the world. Where is the collective memory of decades of projects? Where does
one go to learn from the mistakes and successes of similar projects in the past?”
11.7 Evaluating the evaluation

;OLW\YWVZLVML]HS\H[PVUPZ[VWYVTV[LYL]PL^YLÅLJ[PVUHUKYL]PZPVUVM[OLJ\YYPJ\S\TIHZLKVU
JHYLM\S JVTWPSH[PVU VM PUMVYTH[PVU MYVT H ]HYPL[` VM KPќLYLU[ ZV\YJLZ 0U VYKLY [V THRL KLJPZPVUZ
IHZLKVU[OLL]HS\H[PVUP[PZÄYZ[ULJLZZHY`[VYL]PL^[OLWYVJLZZVML]HS\H[PVU[VLUZ\YL[OH[[OL
evaluation was adequately designed. Questions related to the following aspects of design will help in
determining whether the evaluation process is satisfactory:
• :JVWL!+VLZ[OLYHUNLVMPUMVYTH[PVUJVSSLJ[LKPUJS\KLHSS[OLZPNUPÄJHU[HZWLJ[ZVM[OLWYVNYHT
being evaluated?
• (\KPLUJL!Does the information collected adequately serve the needs of all the intended
audiences?
• 9LSPHIPSP[`!/HZ[OLPUMVYTH[PVUILLUJVSSLJ[LKPUZ\JOH^H`[OH[[OLZHTLÄUKPUNZ^V\SKIL
obtained by others?
• 6IQLJ[P]P[`!Have attempts been made to make sure that there is no bias in the collecting and
processing of information?
• 9LWYLZLU[H[P]LULZZ!Does the information collected accurately describe the program?
• ;PTLSPULZZ!Is the information provided timely enough to be of use to the audiences for the
evaluation?
• ,[OPJHSJVUZPKLYH[PVUZ!Does the evaluation follow accepted ethical standards, e.g., such that
JVUÄKLU[PHSP[`VMPUMVYTH[PVUPZN\HYHU[LLKHUKPUMVYTH[PVUVI[HPULKPUHWYVMLZZPVUHSHUK
acceptable manner?
Once it has been determined that the evaluation meets acceptable standards of adequacy, it is
necessary to decide how to make use of the information obtained. The processes involved normally
PUJS\KL H YL]PL^ VM HSS PUMVYTH[PVU [OH[ ^HZ JVSSLJ[LK KPZZLTPUH[PUN ÄUKPUNZ [V YLSL]HU[ WHY[PLZ
KLJPKPUN VU ÔH[ JOHUNLZ TH` ULLK [V IL THKL PKLU[PM`PUN JVZ[Z HUK ILULÄ[Z VM WYVWVZLK
changes, developing a plan for the implementation of changes, identifying those responsible for
[HRPUNMVSSV^\WHJ[PVUHUKLZ[HISPZOPUNWYVJLK\YLZMVYYL]PL^VM[OLLќLJ[P]LULZZVMJOHUNLZ
The kinds of changes that might be needed are numerous. For example, the revision or replacement of some
of the course objectives might be required. Sometimes the decision is made to prepare supplementary
materials to complement the textbook or to select a new textbook to replace the book currently being
used. In some cases replacement of some of the face-to-face components of the course with online
delivery might be needed, or perhaps the syllabus itself needs to be rewritten so that the sequence of
skills taught within a course is reordered. Other changes could include the revision or replacement of tests,
[OLVYNHUPaH[PVUVMPUZLY]PJL[YHPUPUNMVY[LHJOLYZVYZLTPUHYZMVYZ[Hќ[VZOHYL[LHJOPUNL_WLYPLUJLZ[OL
development of a peer review process for teachers or of a materials writing project.
/HZ`V\YPUZ[P[\[PVU\UKLYNVULZPNUPÄJHU[JOHUNLZPU[OLUH[\YLVMP[ZWYVNYHTZPUJV\YZLZ&
If so, what kind of information led to these changes?
11.8 Procedures used in conducting evaluations

Many of the procedures used in conducting evaluation are similar to those described elsewhere in this
IVVR[OV\NO[OLPYW\YWVZLZTHÌLKPќLYLU[*OHW[LYVUULLKZHUHS`ZPZMVYL_HTWSLKPZJ\ZZLZ
a number of the procedures mentioned here from the perspective of needs analysis. Here we will
consider their role in evaluation and possible advantages or limitations of each procedure.
Tests
+PќLYLU[ [`WLZ VM [LZ[Z JHU IL \ZLK [V TLHZ\YLJOHUNLZPUSLHYUPUNH[[OLLUKVYH[PU[LYTLKPH[L
stages) of a course. These tests may be:
• institutionally prepared tests such as exit tests designed to measure what students have learned
in the course;
• PU[LYUH[PVUHS[LZ[ZZ\JOHZ;6,-30,3;:VYH*HTIYPKNLWYVÄJPLUJ`[LZ[PM[OLZLHYLYLSH[LK[V
the course aims and content;
• textbook tests such as those provided in teachers’ manuals or as part of a commercial course;
• student records, such as information collected throughout the course based on coursework or
JVU[PU\V\ZHZZLZZTLU[;OPZPUMVYTH[PVUTHÌL\ZLK[VHYYP]LH[HÄUHSZJVYLVYNYHKLMVYH
Z[\KLU[^P[OV\[\ZPUNHÄUHS[LZ[
(K]HU[HNLZ! Tests can provide a direct measure of achievement, particularly if they are based on
student performance, that is, they are criterion-referenced.
+PZHK]HU[HNLZ! It is not always easy to be sure that changes in learning as measured by tests
are a direct result of teaching or are linked to other factors. And if there is poor performance on
achievement tests, this does not identify the cause of the problem. Is it the teacher, the materials,
the students, or the course? Further investigation is normally needed. Student evaluation should not
ILJVUM\ZLK^P[OJV\YZLL]HS\H[PVU0UHKKP[PVUZV\UK[LZ[Z¶[LZ[Z[OH[YLÅLJ[WYPUJPWSLZVMYLSPHIPSP[`
HUK]HSPKP[`¶HYLKPѝJ\S[[VJVUZ[Y\J[
Comparison of two approaches to a course

;^V KPќLYLU[ ]LYZPVUZ VM H JV\YZL TPNO[ IL [H\NO[ HUK [OL YLZ\S[Z HZ TLHZ\YLK I` Z[\KLU[
HJOPL]LTLU[ JVTWHYLK ;OL JVTWHYH[P]L HWWYVHJO ZLLRZ [V JVTWHYL [OL LќLJ[Z VM [^V VY TVYL
KPќLYLU[[LHJOPUNJVUKP[PVUZ0[TLHZ\YLZ[OLLѝJPLUJ`VM[OLJ\YYPJ\S\TI`JVTWHYPUN[OLYLSH[P]L
LќLJ[P]LULZZVM[^VKPќLYLU[^H`ZVM[LHJOPUNP[
(K]HU[HNLZ!The comparative approach seeks to control all relevant factors and to investigate factors
rigorously.
+PZHK]HU[HNLZ! ;OPZ HWWYVHJO \Z\HSS` PTWVZLZ HY[PÄJPHS JVUZ[YHPU[Z VU [OL [LHJOLY 6UL [LHJOLY MVY
example, might be required always to give explicit error correction and another to give only indirect error
JVYYLJ[PVU/V^L]LYILJH\ZLVMO\THUMHJ[VYZP[PZ]LY`KPѝJ\S[[VTHPU[HPU[OLZLRPUKZVMKPќLYLUJLZ
It is usually impossible to control all relevant variables and the results are therefore usually inconclusive.
Interviews
Interviews with teachers and students can be used to get their views on any aspect of the course.
Normally, structured interviews provide more useful information than unstructured interviews.
(K]HU[HNLZ!0UKLW[OPUMVYTH[PVUJHUILVI[HPULKVUZWLJPÄJX\LZ[PVUZ
+PZHK]HU[HNLZ!Interviews are very time-consuming and only a sample of teachers or students can
normally be interviewed in depth; hence the representativeness of their views may be questionable.
Questionnaires
These can be used to elicit teachers’ and students’ comments on a wide range of issues.
(K]HU[HNLZ! Questionnaires are easy to administer and information can be obtained from large
numbers of respondents.
+PZHK]HU[HNLZ!Questionnaires need to be carefully designed if they are to elicit unbiased answers,
HUK PUMVYTH[PVU TH` IL KPѝJ\S[ [V PU[LYWYL[ -VY L_HTWSL PM Z[\KLU[Z PUKPJH[L [OH[ [OL` MV\UK H
WHY[PJ\SHY\UP[PUHJV\YZLKPѝJ\S[MVSSV^\WPU]LZ[PNH[PVUTHÌLULLKLK[VKL[LYTPULL_HJ[S`Ô`
[OL`WLYJLP]LKP[[VILKPѝJ\S[>HZP[[OL\UP[P[ZLSMVY^HZP[IHKS`[H\NO[&
Teachers’ written evaluations

Teachers can complete a course evaluation using a structured feedback form that elicits comments
on all aspects of the course.
(K]HU[HNLZ!Teachers are in a good position to report on a course, and a well-designed evaluation

MVYTWYV]PKLZPUMVYTH[PVUX\PJRS`PUH^H`[OH[PZLHZ`[VZ\TTHYPaL
+PZHK]HU[HNLZ! The information obtained may be impressionistic and biased, because it presents
only the teacher’s point of view.
Diaries and journals

Teachers can keep an ongoing record of their impressions and experiences of a course. Diaries
provide a narrative record of things the teacher does, problems encountered, critical incidents, time
allocation, and other issues.
(K]HU[HNLZ! Diaries and journals provide relatively detailed and open-ended information and can
capture information that may be missed by other means.
+PZHK]HU[HNLZ!0[PZKPѝJ\S[[VKLJPKLOV^[V\ZL[OLPUMVYTH[PVUVI[HPULK+PHYPLZHYLPTWYLZZPVUPZ[PJ
and unsystematic. They also require cooperation and a time commitment on the part of the teacher.
Teachers’ records
Use can be made of available written records of courses, such as reports of lessons taught, material
covered, attendance, students’ grades, and time allocation.
(K]HU[HNLZ!Records can provide a detailed account of some aspects of the course.
+PZHK]HU[HNLZ! Not all of the information collected may be relevant. Some information may be
impressionistic and represent only the teacher’s point of view.
Student logs
Students might be asked to keep an account of what happened during a course, how much time they spent
VUKPќLYLU[HZZPNUTLU[ZOV^T\JO[PTL[OL`HSSVJH[LK[VOVTL^VYRHUKV[OLYV\[VMJSHZZHJ[P]P[PLZ
(K]HU[HNLZ!Student logs provide the students’ perspective on the course and gives insights that the
teacher may not be aware of.
+PZHK]HU[HNLZ!They require the cooperation of students and time commitment. Students may not
ZLL[OLILULÄ[VMZ\JOHUHJ[P]P[`
Case studies
A teacher may conduct a case study of a course or some aspect of a course. For example, the
teacher might document how he or she made use of lesson plans throughout a course, or trace the
progress of a particular learner.
(K]HU[HNLZ! Case studies provide detailed information about aspects of a course, and over time the
HJJ\T\SH[LKPUMVYTH[PVUMYVTJHZLZ[\KPLZJHUWYV]PKLHYPJOWPJ[\YLVMKPќLYLU[KPTLUZPVUZVMHJV\YZL
+PZHK]HU[HNLZ!The information collected may not be typical or representative, and case studies are
time-consuming to prepare.
Student evaluations
Students can provide written or oral feedback on a course both during the course and after it has
been taught, commenting on features such as the teacher’s approach, the materials used, and their
relevance to the students’ needs.
(K]HU[HNLZ!Student evaluations are easy to obtain, provide feedback on a wide range of topics, and
enable large numbers of learners to be involved.
+PZHK]HU[HNLZ!0UMVYTH[PVUVI[HPULKTHÌLZ\IQLJ[P]LHUKPTWYLZZPVUPZ[PJHUKPZZVTL[PTLZKPѝJ\S[
[VPU[LYWYL[VYNLULYHSPaL
Audio or video recordings

3LZZVUZJHUILYLJVYKLK[VWYV]PKLL_HTWSLZVMKPќLYLU[[LHJOPUNZ[`SLZHUKSLZZVUMVYTH[Z
(K]HU[HNLZ!Recordings can provide a rich account of teaching in real time and record information
[OH[PZKPѝJ\S[[VKVJ\TLU[PUV[OLY^H`Z
+PZHK]HU[HNLZ! The presence of the recording instrument or person making the recording can
IL KPZY\W[P]L HUK JHU IPHZ [OL KH[H .VVK YLJVYKPUNZ JHU IL KPѝJ\S[ [V VI[HPU HS[OV\NO TVIPSL
technology has made it easier.
Observation
Regular observation of classes may be made by other teachers or a supervisor. Observation is usually
TVYL\ZLM\SPMP[PZZ[Y\J[\YLKPUZVTL^H`Z\JOHZI`NP]PUNHZWLJPÄJ[HZR[V[OLVIZLY]LYHUKI`
providing procedures for the observer to use (e.g., checklists or rating scales).
(K]HU[HNLZ! Observers can focus on any observable aspect of the lesson and can provide an
objective eye, identifying things that may not be apparent to the teacher. If teachers observe each
V[OLY»ZJSHZZLZP[HSZVWYV]PKLZHIHZPZMVYMVSSV^\WKPZJ\ZZPVUHUKYLÅLJ[PVU
+PZHK]HU[HNLZ! The observer’s presence may be intrusive. As noted earlier, observation is a
ZWLJPHSPaLKZRPSSHUKYLX\PYLZWYLWHYH[PVUHUKL_WSPJP[N\PKHUJLPMP[PZ[VIL\ZLM\S
Choose the three most useful or practical procedures above that could be applicable in your
teaching context.
As the examples above illustrate, the evaluation procedures chosen will depend on the kind of issue
[OH[ULLKZJSHYPÄJH[PVUVYYL]PL^>LPYHUK9VILY[Z Z\TTHYPaL[OLMVJ\ZHUKWYVJLK\YLZ
available in program evaluation as represented in Table 11.1.
Table 11.1 Issues and procedures in evaluation

FOCUS PROCEDURES
Teacher beliefs Pre/Post: questionnaires
interviews
observations
review of lesson plans

Teacher abilities Observations/videotapes
Self-assessment quizzes
Pre/Post: questionnaires
interviews
observations
review of lesson plans

FOCUS PROCEDURES
Teacher practices Record of activities
Lesson plan reviews
Observations/videotapes
Interviews
Questionnaires
Student behaviours Student interviews
Student questionnaires
Teacher logs
Observation
Teacher interviews
Student learnings Chapter/unit tests
Standardized test
Teacher logs
Student assignments
Comparison of present term grades to previous grades (or grades of

another group of students)
Student interviews
Teacher questionnaires
Student questionnaires
Teacher interviews
11.9 Process-focused evaluation: descriptive and reflective evaluation

Evaluation may also involve collecting information about how a curriculum is implemented – not in
order to improve it or to measure its outcomes but to gain a better understanding of the processes
of teaching and learning that occur in the program. This approach to evaluation has two aspects:
VULPZ[VJVSSLJ[PUMVYTH[PVUVUKPќLYLU[HZWLJ[ZVM[OLJ\YYPJ\S\TMVYHU`VULÔVTHÙLLKZ\JO
information. This will be called KLZJYPW[P]LL]HS\H[PVU. Below are some examples of issues that might
be explored from this perspective:
• ;PTPUN!OV^T\JO[PTLPZZWLU[VUKPќLYLU[HJ[P]P[PLZ
• ;PTLVU[HZR! the percentage of time that students are actively engaged on learning tasks.
• *SHZZYVVTTHUHNLTLU[!OV^[OL[LHJOLYVYNHUPaLKHUKTHUHNLKJSHZZYVVTHJ[P]P[PLZ
• .YV\WPUNHYYHUNLTLU[Z! the teacher’s use of whole-class and small-group teaching.
• 7HY[PJPWH[PVUWH[[LYUZ! how students participated during group activities.
• -LLKIHJR! the kinds of feedback strategies the teacher made use of.
• (J[PVUaVUL![OL[LHJOLY»Z[`WPJHSHJ[PVUaVULK\YPUNSLZZVUZ
• <UWSHUULKHJ[P]P[PLZ! the teacher’s departures from plans.
• <ZLVM[OL[L_[IVVRHUK[LJOUVSVN`! time spent with teaching resources and how they are used.
What kind of information of this kind would you be interested to learn about for your teaching?
As the name suggests, then, descriptive evaluation requires collecting as much information as
WVZZPISL[OH[^V\SKWYV]PKLHJVTWYLOLUZP]LHUKVIQLJ[P]LHJJV\U[VMKPќLYLU[HZWLJ[ZVMHJV\YZL
Many of the procedures described above can be used to collect information of this kind, such as
observation, questionnaires, interviews, teachers’ reports, and student logs. The following examples
illustrate this aspect of evaluation:
,_HTWSL! A teacher is teaching a course on reading skills and has developed a course which
focuses on a wide variety of reading skills, such as skimming, scanning, reading for details, surveying
a text, critical reading, and vocabulary development. All of the skills receive regular focus throughout
[OLJV\YZL;OL[LHJOLYPZPU[LYLZ[LKPUÄUKPUNV\[ÔH[[OLZ[\KLU[ZWLYJLP]L[VIL[OLTHPUWVPU[
VM[OLJV\YZL:[\KLU[ZJVTWSL[LHZOVY[X\LZ[PVUUHPYLH[KPќLYLU[[PTLZK\YPUN[OLJV\YZLPUVYKLY
[VKLZJYPIL[OLPYWLYJLW[PVUZVMÔH[[OLJV\YZLPZZLLRPUN[VHJOPL]L([[PTLZ[OLYLPZHKPќLYLU[
WLYJLW[PVUVU[OLWHY[VMZ[\KLU[ZHZ[V[OLW\YWVZLVMKPќLYLU[HJ[P]P[PLZVYL]LUVMÔVSLSLZZVUZ
(M[LYYLÅLJ[PUNVU[OPZWOLUVTLUVU[OL[LHJOLYJVTLZ[V\UKLYZ[HUK[OH[SLHYULYZ»WLYJLW[PVUZVM
HJV\YZLTH`YLÅLJ[ÔH[[OL`HYLTVZ[PU[LYLZ[LKPUVYÔH[[OL`MLLS[OLÙLLKTVZ[OLSW^P[OH[H
particular point in time.
,_HTWSL! A teacher is interested in learning more about teacher–student interaction in her

own classroom. She invites a colleague to visit her class and to carry out a series of classroom
VIZLY]H[PVUZ;OLVIZLY]LYPZNP]LU[OL[HZRVMUV[PUNOV^VM[LU[OL[LHJOLYPU[LYHJ[Z^P[OKPќLYLU[
students in the class and the kind of interaction that occurs. This involves noting the kinds of
questions the teacher asks and the extent to which she acknowledges and follows up on students’
questions. From the data collected by the observer, the teacher is able to assess the extent to which
she or the students control classroom interaction and gets a better understanding of how she uses
X\LZ[PVUZ[V¸ZJHќVSK¹SLZZVUJVU[LU[
,_HTWSL!([LHJOLY^HU[Z[VÄUKV\[TVYLHIV\[OV^Z[\KLU[ZJHYY`V\[NYV\W^VYRHUKÔL[OLY
OLPZHKLX\H[LS`WYLWHYPUNZ[\KLU[ZMVYNYV\W^VYR[HZRZ/LHYYHUNLZ[VYLJVYKKPќLYLU[NYV\WZ
VMZ[\KLU[ZJHYY`PUNV\[HNYV\W^VYR[HZRHUKYL]PL^Z[OLYLJVYKPUNZ[VÄUKV\[[OLL_[LU[[V
which students participate in group discussions and the kind of language they use. On reviewing the
recordings, the teacher is pleased to note that the strategy of assigning each member of a group a
KPќLYLU[YVSLK\YPUNNYV\W[HZRZ¶Z\JOHZJVVYKPUH[VYSHUN\HNLTVUP[VYVYZ\TTHYPaLY¶PZWYV]PUN
LќLJ[P]LPULUZ\YPUN[OH[NYV\WTLTILYZWHY[PJPWH[LHJ[P]LS`PU[HZRZ
Such examples suggest the variety of ways in which teachers seek to assess and monitor aspects
of their teaching, whether by asking students to describe how they understand the goals of the
course and the use of classroom activities, asking colleagues to collect information about patterns
of classroom interaction in order to understand better the kind of communication that occurs during
teaching, or by monitoring group-work activities to understand better the kinds of participation they
facilitate. Much classroom action research or teacher inquiry can be regarded as evaluation of this
kind and may be either teacher initiated or carried out by others.
The other aspect of process evaluation we refer to as YLÅLJ[P]LL]HS\H[PVUThis view of evaluation is
inquiry-based and is concerned with knowledge building, with understanding, and with explanation.
Perhaps the term L]HS\H[PVU does not really suit this orientation to understanding the curriculum as a
“lived experience,” since the focus is not on judgment but on exploration and understanding. The goal
is less diagnoses and improvement and more on a holistic exploration of teaching in context. Context
here is not viewed as a set of limitations but rather as central to the process by which curriculum is
enacted. This is a sociocultural approach to evaluation in which classrooms are seen to have a rich life
that unfolds over time, as events and processes interact and shape the way participants think, feel,
and act. It draws on social and situated perspectives on learning, on the classroom as a community
of practice, as a site where learners exercise their agency in identity formation, and as an ecology,
OH]PUN H SPML [V IL THUHNLK >YPNO[ ;OL JSHZZYVVT PZ ]PL^LK HZ OH]PUN ZVJPHS WHY[PJPWH[PVU
structures that can enhance or inhibit learning opportunity. This includes both the discourse and the
HJ[P]P[PLZVMJSHZZYVVTSPMLÔPJOHќLJ[OV^TLHUPUNPZTHKLHUKRUV^SLKNLJVUZ[Y\J[LK3H]LHUK
Wenger 1991; Wenger 1998; Lantolf 2000; Hawkins 2004). Learning is tied to artifacts, identities,
and the cultural space in which it is situated. These artifacts include books, whiteboards, tablets, and
SHW[VWZ[OH[NP]LÅLZO[VPKLU[P[PLZPUWYHJ[PJL/VSSHUKHUK3H]L
What kinds of classroom processes would a visitor typically observe in your classes?
(YLÅLJ[P]L]PL^VML]HS\H[PVU]PL^Z[OLJSHZZYVVTHZHJVTWSL_LJVSVNPJHSZP[LPUÔPJO[LHJOLYZHUK
learners have to navigate among unfolding events and processes in order to participate in teaching and
SLHYUPUN-YVTHZP[\H[LKZVJPHSWLYZWLJ[P]LVUSLHYUPUN[OLJSHZZYVVTPZJVUJLW[\HSPaLKHZHULTLYNPUN
“community of practice” (Lave and Wenger 1991). This shifts the focus to people jointly engaged in a
mutual enterprise, with a shared repertoire of actions, discourses, and tools (Wenger 1998).
Questions that evaluation seeks to answer from this perspective include the following:
• How do teachers and learners understand the curriculum?
• What is the nature of language teaching and learning?
• What does it mean to the participants?
• What roles do they participate in?
• What is the nature of the experiences they participate in?
• What learning opportunities arise during lessons?
• What do these activities mean to them?
• What underlying values and beliefs underlie the curriculum?
• /V^KVLZ[OL[LHJOLYYLHSPaLOPZVYOLYWYPUJPWSLZHUK]HS\LZPU[LHJOPUN&
• How are teacher and learner identities negotiated through their interactions?
How do you think questions such as those above could be explored? What research
procedures could be used?
8\LZ[PVUZ VM [OPZ RPUK YLÅLJ[ [OL LJVSVNPJHS \UKLYZ[HUKPUN VM J\YYPJ\S\T YLMLYYLK [V PU *OHW[LYZ
HUK (J[P]P[PLZ [OH[ PU]VS]L H YLÅLJ[P]L HUK LJVSVNPJHS \UKLYZ[HUKPUN VM L]HS\H[PVU HYL
VIZLY]H[PVU PUX\PY` WHY[PJPWH[PVU HUK JYP[PJHS YLÅLJ[P]P[` ;\KVY JVU[YHZ[Z [OL WYVK\J[
focused technological approach to curriculum with an ecological approach and, like Aoki,
suggests the tensions that can arise between these two perspectives. Tudor notes that a feature
VM H [LJOUVSVNPJHS HWWYVHJO PZ [OH[ P[ PZ SPURLK [V H ZWLJPÄJ WYVK\J[ HUK WYLZLU[Z H WVZP[P]L HUK
JVUÄKLU[ HWWYVHJO [V HJOPL]PUN V\[JVTLZ ;OL LJVSVNPJHS HWWYVHJO VU [OL V[OLY OHUK PZ SLZZ
JVUÄKLU[ HUK KLWLUKLU[ VU MHJ[VYZ [OH[ HYL UV[ JLY[HPU VY Ä_LK /L Z\NNLZ[Z [OH[ [OVZL ÔV
are not closely connected with classroom realities such as planning committees and educational
authorities are likely to favor a technological approach. However, classroom teachers operate
PU H KPќLYLU[ JVU[L_[ ¶ HU LJVSVNPJHS VUL [OH[ KYH^Z VU THU` KPќLYLU[ ZV\YJLZ! [OL ILSPLMZ HUK
expectations of students, parents, textbook writers and administrators, and above all the individual
teacher, for whom teaching is far more than following the teacher’s manual or making use of the
classroom technology.
11.10 Implementing reflective evaluation

9LÅLJ[P]LL]HS\H[PVUTHRLZ\ZLVMHU\TILYVMZ[YH[LNPLZMVYL_WSVYPUN[OLUH[\YLHUKTLHUPUNVM
[OL[LHJOLY»ZJSHZZYVVTWLKHNVN`(SL_HUKLYL_WSHPUZ[OLKPќLYLUJLIL[^LLU[LHJOPUNHUK
pedagogy:
Pedagogy I define as the discourse which attends the act of teaching. Teaching and pedagogy
are not the same. Teaching is a practical and observable act. Pedagogy encompasses that act
together with the purposes, values, ideas, assumptions, theories and beliefs which inform,
shape and seek to justify it.
Developing an understanding of pedagogy, then, involves examining not just the “what” and “how” of
teaching – content knowledge and classroom methodology respectively – but the questions of “what
kind of teacher am I and what kind of teacher do I want to become?” A teacher’s pedagogy is based
on answers to questions such as these from Bartlett (1990, 206–207):
• What caused me to want to become a second language teacher?
• Do these reasons still exist for me now?
• What does it mean to be a teacher?
• Is the teacher I am the person I am?
• Where did the ideas I embody in second language teaching come from historically?
• How did I come to appropriate them?
• Why do I continue to endorse them now in my teaching?
• Whose interests do these ideas serve?
• Who has power in my classroom and how is it expressed?
• How do power relationships in my classroom influence my interactions with students?
• How might I teach differently?
• What is the nature of knowledge that guides my teaching of content?
• Who creates this knowledge? How did this knowledge emerge during the evolution of
teaching?
• Whose interests does this knowledge about language teaching serve?
• How do/can I personally work to uncover the contradictions in my teaching?
• How does what I do affect the opportunities in life of students?
• What connections do I make with organizations outside the school or centre to demon-
strate my active role in society?
• Do I wish to uncover the “hidden curriculum” – the inconsistencies – in my teaching?
(J[P]P[PLZ [OH[ PU]VS]L YLÅLJ[P]L L]HS\H[PVU HUK [OH[ LUNHNL [LHJOLYZ PU L_WSVYPUN [OLPY WLKHNVN`
include journal writing, narrative inquiry, analysis of critical incidents, peer observation, and case
HUHS`ZPZ9PJOHYKZHUK-HYYLSS9LÅLJ[P]L[LHJOPUNL_WSVYH[VY`WYHJ[PJLHUKSLZZVUZ[\K`HYL
further examples of approaches that teachers sometimes use in this process of inquiry, and these are
KPZJ\ZZLKIYPLÅÌLSV^
Reflective teaching
9LÅLJ[P]L [LHJOPUN YLMLYZ [V [LHJOPUN [OH[ PZ HJJVTWHUPLK I` JYP[PJHS WYVJLZZPUN HUK YL]PL^ VM
WYHJ[PJL+L^L`WYVWVZLK[OPZHWWYVHJOÔLUOL^YV[LVM¸YLÅLJ[P]LPUX\PY`¹ 9VNLYZ
JSHYPÄLZ[OLUV[PVUVMYLÅLJ[P]LPUX\PY`HUKYLÅLJ[P]L[LHJOPUN!
a reflective teacher does not merely seek solutions, nor does he or she do things in the same
way every day without an awareness of both the source and the impact of his or her actions.
Rather, from his or her practice and the students’ learning, the teacher seeks meaning and
creates from this a theory to live by, a story that provides structure for the growth of the
students and of the teacher. When the teacher seeks solutions, he or she also pursues con-
nections and relationships between solutions so that a theory might grow. This theory guides
practice (which includes but is not limited to problem solving) until it encounters a situation
where the theory no longer serves, at which point, through more reflection, it is either revised,
refined, or discarded, and a new theory is born.
9LÅLJ[P]L [LHJOPUN PZ IHZLK VU [OL PKLH [OH[ L_WLYPLUJL HSVUL KVLZ UV[ ULJLZZHYPS` [YPNNLY JYP[PJHS
\UKLYZ[HUKPUN I\[ [OH[ L_WLYPLUJL SPURLK [V YLÅLJ[PVU JHU SLHK [V H KLLWLY \UKLYZ[HUKPUN VM [OL
TLHUPUN VM [LHJOPUN 9LÅLJ[PVU KYH^Z VU HJJV\U[Z VM [LHJOPUN LWPZVKLZ HUK PUJPKLU[Z PUJS\KPUN
activities such as journal writing, narratives, critical incident analysis, peer observation, and audio
or video recording of lessons. The teacher then reviews the data, either individually or with another
teacher, and poses questions about how and why things happened during the lesson, what value
systems they represent, and what alternatives might be available.
>O`KV`V\[OPURL_WLYPLUJLPZPUZ\ѝJPLU[HZHIHZPZMVY[LHJOLYSLHYUPUN&
Exploratory practice
This refers to a form of practitioner inquiry elaborated originally by Allwright (2003) that involves focus
on an issue or question (such as the teacher’s role in the classroom), exploring the issue from multiple
KPYLJ[PVUZ HUK \ZPUN PUMVYTH[PVU [OH[ YLZ\S[Z HZ H IHZPZ MVY JYP[PJHS YLÅLJ[PVU HUK H^HYLULZZ YHPZPUN
:LL*OHW[LY /HURZ¶PKLU[PÄLZZL]LUWYPUJPWSLZ[OH[JOHYHJ[LYPaLHUL_WSVYH[VY`
WYHJ[PJLHWWYVHJO0ZZ\LZHUKHYLYLMLYYLK[VHZ¸ÔH[¹PZZ\LZ"HUKHZ¸ÔV¹PZZ\LZHUK
6 and 7 as “how” issues:
1. Focus on quality of life as the fundamental issues.

2. Work to understand it before thinking about solving problems.
3. Involve everybody as practitioners developing their own understandings.
4. Work to bring people together in a common enterprise.
Work cooperatively for mutual development.
6. Make it a continuous enterprise.
7. 4PUPTPaL[OLI\YKLUI`PU[LNYH[PUN[OL^VYRMVY\UKLYZ[HUKPUNPU[VUVYTHSWLKHNVNPJWYHJ[PJL
Central to the notion of exploratory practice is that it involves adding a dimension to one’s existing
teaching practice rather than intervening in some way to change practice (as is the case with action
research). The processes involved could include peer observation, discussions, narratives, blogs,
reviewing video or written accounts of lessons, or analysis of critical incidents in order to arrive at
new understandings of the meaning of everyday classroom life. An example of this approach is given
in Appendix 1 in Chapter 9.
An example of an issue a teacher could focus on in exploratory practice is, “Why don’t
students talk more in my class?” Can you suggest other issues that could be the focus of
exploratory practice?
Lesson study
This is a form of collaborative inquiry in which a group of teachers co-plan a lesson that focuses on
a particular piece of content of a unit of study, as described in the example below from the Lesson
Study Project site (Cerbin and Kopp n.d.).
Lesson study is a form of classroom inquiry in which several teachers collaboratively plan,
teach, observe, revise and share the results of a single class lesson. Teachers work through the
steps listed below.
1. Form a Team: Teams are usually composed of 3–6 instructors.
2. Develop Learning Goals: Team members articulate what they would like students to know
and be able to do as a result of the lesson.
3. Design the Lesson: The team designs a lesson to achieve the learning goals.
4. Plan the Study: The team decides how to observe and collect evidence of student learning.
5. Teach and Observe: One team member teaches the lesson while others observe and collect
evidence of student learning.
6. Analyze and Revise: The team discusses the results and assesses student progress toward
learning goals.
7. Document and Disseminate: The team documents the lesson study and shares their work
with colleagues.
In a lesson study, teachers carefully explore how student learning, thinking and behav-
ior change as a result of the lesson. The practice of lesson study can lead to instructional
improvement as teachers become more knowledgeable about how their students learn and
think and how instruction affects student thinking.
As outlined above, throughout the planning process they draw on outside resources, including
textbooks, research, and teaching theories, and engage in extended conversations while focusing
VUZ[\KLU[SLHYUPUNHUK[OLKL]LSVWTLU[VMZWLJPÄJV\[JVTLZ6UJL[OLWSHUOHZILLUKL]LSVWLK
one member of the team volunteers to teach it, while the others observe. (Sometimes outsiders
HYL HSZV PU]P[LK [V VIZLY]L (M[LY [OL SLZZVU [OL NYV\W KPZJ\ZZLZ [OLPY ÄUKPUNZ PU H JVSSVX\P\T VY
panel discussion.
Conclusions
Two perspectives on evaluation have been examined in this chapter, one that is product and outcome
MVJ\ZLK HUK VUL [OH[ PZ WYVJLZZ MVJ\ZLK ;OL MVYTLY YLÅLJ[Z H [YHKP[PVUHS HWWYVHJO [V L]HS\H[PVU
PU ÔPJO WYVNYHT LќLJ[P]LULZZ PZ [OL WYPTHY` JVUJLYU -VYTH[P]L L]HS\H[PVU ZLLRZ [V HKKYLZZ
HU` WYVISLTZ [OH[ TPNO[ HYPZL K\YPUN H WYVNYHT [OH[ JV\SK PUÅ\LUJL [OL Z\JJLZZ VM P[Z V\[JVTLZ
Summative evaluation seeks to assess the extent to which the program’s goals and learning outcomes
OH]L ILLU HJOPL]LK 7YVJLK\YLZ MVY JVUK\J[PUN LќLJ[P]LULZZMVJ\ZLK L]HS\H[PVU PU]VS]L JVSSLJ[PUN
both quantitative and qualitative information and could address many aspects of a program, including
the curriculum design, the teaching, the teachers, the teaching resources, the administration of
the program, and the tests and assessment processes that are used. Process-focused evaluation
complements but does not replace product-focused evaluation. It is both descriptive and explanatory
and seeks to achieve a deeper understanding of how the curriculum works and how it is understood
by teachers and learners.
Discussion questions
1. /V^ ^V\SK `V\ JOHYHJ[LYPaL [OL KPќLYLUJL IL[^LLU WYVK\J[MVJ\ZLK HUK WYVJLZZMVJ\ZLK
evaluation?
2. If you were involved in developing an evaluation of courses in your institution, what would the
goals of such an evaluation be?
3. Give examples of quantitative and qualitative information that you could make use of for the
activity in question 2 above.
4. /V^ ^V\SK `V\ JOHYHJ[LYPaL ¸LќLJ[P]LULZZ¹& >OH[ JYP[LYPH ^V\SK `V\ \ZL [V HZZLZZ [OL
LќLJ[P]LULZZVM[OLJV\YZLZ`V\[LHJO&
>OH[HYLZVTLVM[OLNVHSZVMKLZJYPW[P]LHUKYLÅLJ[P]LL]HS\H[PVU&
6. Review the best-practice framework in Appendix 1 in this chapter. How well would it work in your
teaching context? Would you need to make changes to it if you found it useful?
7. 9LHK*HZLZ[\K` I`.LVќYL`*YL^LZH[[OLLUKVM[OPZJOHW[LY
• >OH[JYP[LYPH^LYL\ZLK[VKL[LYTPUL[OLLќLJ[P]LULZZVM[OLWYVNYHT&
• What were some of the distinctive features of the program?
• To what extent did the program incorporate both a product and a process perspective?
8. Read Case study 20 by Jonathan Newton.
• >OH[JYP[LYPH^LYL\ZLK[VKL[LYTPUL[OLLќLJ[P]LULZZVM[OLPUUV]H[PVU&
• What factors mitigated against the teachers’ uptake of the innovation?
• Review the quote from Kiely at the beginning of this chapter: “program evaluation is about
LќLJ[P]LULZZHUK[YHKP[PVUHSS`[OLHUZ^LYOHZILLUZV\NO[PU[LYTZVM[LZ[YLZ\S[ZH
SHUN\HNLSLHYUPUN[OLVY`VYHWHY[PJ\SHYZ`SSHI\Z¹/V^PZ[OPZYLÅLJ[LKPU*HZLZ[\K`&
9. Read Case study 21 by David Crabbe.

• What other questions could the teachers have investigated?
• Can you suggest other ways in which the teachers could have explored their own teaching?
• The teachers decided that the next step would be to evaluate more explicitly the impact
VMSLHYUPUNJVU]LYZH[PVUZVUHUV\[VMJSHZZSLHYUPUNHJ[P]P[`HUKVUHZWLJ[ZVMJVUÄKLUJL
HUKWYVÄJPLUJ`/V^JV\SK[OL`KV[OPZ&
APPENDIX 1 Best practice in English language teaching*

1. Institution
PHYSICAL FACILITIES
A quality language centre is characterised by:
• clean and safe premises;
• JSHZZYVVTZHUKVѝJLZÔPJOHYLUV[V]LYJYV^KLK"
• adequate ventilation, heating, cooling and lighting; and
• HKLX\H[LWYLJH\[PVUZPUJHZLVMÄYL
The physical facilities contribute to an atmosphere conducive to learning.
MANAGEMENT AND ADMINISTRATION

A quality language centre:
• is under the direction of an appropriately-trained and experienced management and
administration team which is knowledgeable about the design, implementation and evaluation
of ELT programs;
• ZLLRZ[VH[[YHJ[HUKYL[HPUHZ[HќVM[YHPULKKLKPJH[LKWYVMLZZPVUHS,3;WYHJ[P[PVULYZ"
• YLJVNUPZLZ[OLPTWVY[HUJLVMHWWYVWYPH[LZHSHY`HUKILULÄ[ZHZ^LSSHZ[OLPTWVY[HUJLVMZ[Hќ
development. The centre has clearly stated policies concerning these issues;
• reviews its employment conditions and procedures periodically in light of generally accepted ELT
standards and local market conditions; and
• [HRLZHJJV\U[VMPUW\[MYVT[LHJOLYZZ\WWVY[Z[HќHUKZ[\KLU[ZPUTHRPUNKLJPZPVUZYLNHYKPUN
personnel practices, management of resources and program evaluation.
The management strives to engender a positive teaching and learning environment.
6WDσ
TEACHING STAFF
• LTWSV`Z [LHJOLYZ ÔV OH]L PU[LYUH[PVUHSS` YLJVNUPZLK X\HSPÄJH[PVUZ PUSHUN\HNL[LHJOPUN"
• YLJVNUPZLZ[OH[[OLU\TILYVMJVU[HJ[OV\YZWYLWHYH[PVUOV\YZHUKVѝJLWYLZLUJLVM[LHJOLYZ
KPYLJ[S`PUÅ\LUJL[LHJOLYLќLJ[P]LULZZ"
• ensures equal opportunity regarding all aspects of employment, including the possibility of job
security of employment; and
• WYV]PKLZZ\WWVY[PU[OLMVYTVMHKTPUPZ[YH[P]LZ`Z[LTZVѝJLZWHJL[LSLWOVULZK\WSPJH[PUN
facilities (with clear guidelines about copyright laws), and space for professional development
seminars and workshops.
SUPPORT STAFF
(X\HSP[`SHUN\HNLJLU[YLYLJVNUPZLZ[OL]P[HSYVSL[OH[UVU[LHJOPUNZ[HќWSH`PUZ\WWVY[PUN[YHPUPUN
activities and contributing to the quality of the service provided to clients.
STAFF DEVELOPMENT
• HJ[P]LS`Z\WWVY[ZHUKLUNHNLZPUJVU[PU\PUNZ[HќKL]LSVWTLU[MVYHSSLTWSV`LLZ;OLYLPZ
continuous, ongoing in-service training, using a range of learning modes conducted by members
VMZ[HќHZ^LSSHZI`PU]P[LK[YHPULYZMYVTV\[ZPKL"
• RLLWZ[VHTPUPT\T[OLU\TILYVM\UKLY[YHPULK[LHJOPUNZ[HќHUKZ\WWVY[Z[Hќ;OLYLPZH
WSHUPUWSHJL[VLUZ\YL[OH[Z[HќKL]LSVWTLU[VWWVY[\UP[PLZHYLTHKLH]HPSHISLHZHWWYVWYPH[L"
• supports membership in professional organisations, attendance at workshops and conferences,
and participation in professional activities outside the workplace; and
• LUNHNLZPUHUKVYLUJV\YHNLZYLZLHYJOVU]HYPV\ZHZWLJ[ZVM,3;I`Z[Hќ
3. Program management
CURRICULUM
• designs and implements curricula that are informed by an analysis of learner needs and the
HZZLZZTLU[VM[OLSLHYULYZ»SL]LSZVMWYVÄJPLUJ`"
• documents curricula, and such documentation includes details of program goals and objectives,
expected learner outcomes, teaching materials, methodology, assessment criteria, and
evaluation procedures;
• assesses student progress on a regular basis. The instruments for assessment are selected
VYKL]LSVWLKHJJVYKPUN[VWYPUJPWSLZNLULYHSS`YLJVNUPZLKPU[OLÄLSKVM,3;HUKHYLJ\S[\YHSS`
appropriate. They relate directly to the stated goals and objectives of the training program.
Students are regularly informed of their progress;
• HJRUV^SLKNLZ[OH[MHJ[VYZZ\JOHZJSHZZZPaLJV\YZLSLUN[OHUKJV\YZLPU[LUZP[`HYLVM[LU
beyond the control of the training provider. However, curricula are developed with these
considerations in mind;
• engages in regular evaluation of its curriculum and courses in response to changing student
needs, new trends in ELT and the changing global context. Teachers and students are involved
in this evaluation which leads to program re-design, with the development of new approaches,
new components, and/or new courses; and
• seeks periodic external evaluation through consultation with experienced, recognised
WYVMLZZPVUHSZPUHWWYVWYPH[LÄLSKZZ\JOHZHWWSPLKSPUN\PZ[PJZHUK,3;THUHNLTLU[;OLZL
PUKP]PK\HSZ^VYR^P[OZ[Hќ[VZOHYLL_WLY[PZLHUK[VWYV]PKLVIQLJ[P]LHWWYHPZHSZVM[OLWYVNYHT»Z
LќLJ[P]LULZZ
4. Resources
• provides instructional materials to facilitate successful language learning. These resources are
up-to-date and accessible to all teachers and include print materials, video tape recorders and
cassettes, audio tape recorders and cassettes, as well as a range of realia;
• recognises the contribution that computerised language instruction and self-access resources
THRL[VLќLJ[P]LSHUN\HNLSLHYUPUNHUKÔLYL]LYWVZZPISLHPTZ[VWYV]PKLZ\JOYLZV\YJLZ"
• maintains a resource collection of relevant books, journals and other materials which is easily
accessible to teachers and students; and
• documents procedures for the selection, evaluation, purchase and upkeep of equipment and
materials and ensures that all concerned are actively involved in decision-making related to
these matters.
Document prepared by EL centers in Indonesia (IALF), Thailand (ELCA), Laos (VUC), and Cambodia
(ACE), for establishing quality standards for language training centers in Southeast Asia. Reprinted
with permission.
CASE STUDY 19 Evaluating an in-service program for English language teachers

.LVɈYL`*YL^LZ
What was the context for the evaluation?
The context for the evaluation is the trialing of an in-service program for English language (EL)
teachers in a developing country. The teachers are all under-trained and some untrained, most with
HSV^SL]LSVM,3WYVÄJPLUJ`;OL`[LHJO,UNSPZOH[Q\UPVYZLJVUKHY`SL]LSPUYLZV\YJLWVVYZJOVVSZ
in remote areas. The in-service program aimed to provide them with the opportunity to improve their
SL]LSVM,UNSPZOWYVÄJPLUJ`HZ^LSSHZ^P[OPUZ[Y\J[PVUHUKWYHJ[PJLPUSLHYULYJLU[LYLKTL[OVKVSVN`
[HRPUNPU[VHJJV\U[SHYNLJSHZZZPaLZ¶PUZVTLJHZLZV]LYZ[\KLU[ZWLYJSHZZ
The in-service program was funded by a bilateral aid program, and designed and delivered by an
experienced provider of EL Teacher Training. A feature of the design was to train local trainers to
produce training materials and deliver the training program, alongside international ELT specialists.
What was the goal of the evaluation?

The overall goal of the evaluation was to determine whether the model was successful in improving
the skills of teachers. Previous training activities had required teachers to be relieved of teaching
duties in their district schools and to attend training held in the capital city – this was a more intensive,
JLU[YHSPaLKHWWYVHJO[V[OL[YHPUPUNVM[LHJOLYZ;OLTVKLSMVY[OLUL^HWWYVHJO^HZMVY[LHJOLYZ
to remain teaching in their schools and undertake training at a district location for two-day sessions,
twice a month. Their school principals would be asked to give them one day free from teaching – Friday
– and this would be combined with a Saturday. Teachers would be paid travel and accommodation
costs to attend the training. In addition to the face-to-face component, teachers would be given
home-study activities to complete prior to the following training session. During school holidays,
more intensive one-week sessions were held. This model meant that schools retained their valued
English teachers while they were undertaking further training. If the model proved to be successful the
Ministry of Education would look at replicating it in other districts with a local provider.
Who was the audience for the evaluation?

The evaluation had several audiences – the international aid agency funding the initiative, the national
Ministry of Education as the partner stakeholder, the local schools where the teachers taught, and
the INSET provider who was responsible for designing and delivering the course for the local context.
How did you conduct the evaluation?

The training provider worked closely with the Ministry of Education, given that the aim of the activity
was for the Ministry to take over this initiative once the aid funding had ceased. The evaluation looked
at several components:
• The local trainers who were being mentored to deliver the program.
• The participants – under-trained school teachers, many of whom were not familiar with certain
aspects of the program, such as home-study modules.
• The local schools – how this in-service model impacted on the teaching of English in schools.
Was it successful? Was it sustainable?
What kind of information did you collect?

Information consisted of baseline data at the beginning of the program, mostly about the skills of
the teachers and their beliefs about teaching English. As the program progressed, it measured their
,3WYVÄJPLUJ`¶ÔPJO^HZHMVJ\ZVM[OLÄYZ[[^VJVTWVULU[Z¶HUK[OLU[OLPYJVTWL[LUJLHUK
JVUÄKLUJLPUTL[OVKVSVN`PU[OL[OPYKJVTWVULU[;OLTLJOHUPZTZ\ZLK^LYLSHUN\HNLWYVÄJPLUJ`
tests, questionnaires, focus groups, and longitudinal case studies of pre-selected individual teachers.
The trainers were also evaluated. The principal mechanism for this was observation of them delivering
training. Prior to the commencement of the INSET program, the teacher trainers, who were all TESOL
X\HSPÄLK\UKLY[VVRH;YHPU[OL;YHPULYZ program which provided opportunities for them to practice
training methodologies and to evaluate their progress as teacher trainers. The availability of local
Z\P[HIS`X\HSPÄLKHUKL_WLYPLUJLK[YHPULYZ[VJVUK\J[PUZLY]PJL[YHPUPUNILJHTLHUPZZ\LHZZVTL
of the trainers struggled with the role of teacher trainer; for this program to be replicated on a larger
scale across the country, there would be a need for a cadre of suitable trainers.
:KDWZHUH\RXUðQGLQJVDQGUHFRPPHQGDWLRQV"
;OLÄUKPUNZ^LYL[OH[!
• The model of local delivery over an extended period of time worked well. It was well-received
by local school administrations and principals, as it meant they did not have to release teachers
MYVT[LHJOPUN[V\UKLY[HRLPUZLY]PJL[YHPUPUN;OPZPZZPNUPÄJHU[PUHJV\U[Y`ÔLYL[LHJOLYZHYL
in very short supply.
• ;OLWHY[PJPWHU[ZNLULYHSS`YLZWVUKLKWVZP[P]LS`[V[OLTVYLSVJHSPaLKHWWYVHJO7HY[PJPWH[PVU
YH[LZ^LYLNVVKHUKKPKUV[KLJSPULZPNUPÄJHU[S`HZ[OLWYVNYHTJVU[PU\LK3VJHSKLSP]LY`TLHU[
that the teachers did not need to spend extended periods of time in the capital city, away from
their families. This was particularly important for ensuring the ongoing participation of female
teachers – a key objective of the initiative. Also the training delivered locally appeared to be more
relevant, as participants could refer to the situation in their schools – many of which lacked the
resources of schools in the nation’s capital. There was a belief that the program and the trainers
had taken into account the local context.
• Some of the trainers struggled with the role of teacher trainer, despite holding TESOL
X\HSPÄJH[PVUZHUK\UKLY[HRPUNHJVTWYLOLUZP]L;YHPU[OL;YHPULY program. One trainer had to
ILYLWSHJLKK\YPUN[OLÄYZ[JVTWVULU[(ZV\YJLVMZ\P[HISLSVJHS[LHJOLY[YHPULYZ^PSSILH
challenge for the delivery of this program in the future.
• ;OLWYVNYLZZPU,3WYVÄJPLUJ`^HZMVYTHU`WHY[PJPWHU[ZZSV^(TVYLPU[LUZP]LHWWYVHJO
WHY[PJ\SHYS`H[[OLILNPUUPUNTHÒH]LZLLUTVYLYHWPKPTWYV]LTLU[PUWYVÄJPLUJ`;\P[PVU
for two days twice a month was a bit piecemeal for some, and they struggled to improve. A
WVZZPISLYL]PZPVU[V[OLKLZPNU^V\SKIL[OLPUJS\ZPVUVMHUPU[LUZP]LWLYPVKVM,3WYVÄJPLUJ`
training – perhaps during the school vacation – to “kick-start” the program and set up productive
study techniques and familiarity with self-study resources.
• The introduction of home-study materials had some initial challenges, but once participants got
\ZLK[VOV^[OLZLÄ[[LKPU^P[O[OLWYVNYHT[OL`YLZWVUKLK^LSSHUKJVTWSL[PVUYH[LZMVY
work done from home was high.
• ;OL05:,;WYVNYHTPZILPUNZ\ITP[[LKMVYHJJYLKP[H[PVU\UKLY[OL5H[PVUHS8\HSPÄJH[PVUZ
Framework. This was seen by participants and stakeholders as an important aspect of the
[YHPUPUNHZP[^PSSWYV]PKLZ\JJLZZM\SWHY[PJPWHU[Z^P[OHYLJVNUPaLKX\HSPÄJH[PVUVUÔPJO[V
develop their teaching careers.
.LVќYL`*YL^LZOHZILLUPU]VS]LK^P[O,3;HUKPU[LYUH[PVUHSLK\JH[PVUMVYV]LY`LHYZ/LOHZOLSK
,3;THUHNLTLU[WVZP[PVUZH[[OL5H[PVUHS<UP]LYZP[`VM:PUNHWVYL[OL)YP[PZO*V\UJPS0UKVULZPH[OL
0UKVULZPH(\Z[YHSPH3HUN\HNL-V\UKH[PVU0(3-HUK0+70,3;:(\Z[YHSPH/LPZJ\YYLU[S`HUPUKLWLUKLU[
JVUZ\S[HU[HUKOHZYLJLU[S`JVUK\J[LKJVUZ\S[HUJPLZPU0UKVULZPH=PL[UHT;PTVY3LZ[LY3HV7+9
*HTIVKPHHUK[OL<(,/LPZH[Y\Z[LLVM[OL3HUN\HNLHUK+L]LSVWTLU[JVUMLYLUJLZLYPLZ
CASE STUDY 20 Evaluating the content of an EAP program

1VUH[OHU5L^[VU
What was the context for the evaluation?
;OPZWYVNYHTL]HS\H[PVU^HZJHYYPLKV\[H[HSHYNL:V\[OLHZ[(ZPHU\UP]LYZP[`ÔPJOH[[YHJ[ZZPNUPÄJHU[
numbers of international students for whom English is a second language. English is the medium of
instruction at this university, and so all prospective students must obtain a passing grade on an
PUOV\ZL,UNSPZO7YVÄJPLUJ`;LZ[,7;PUVYKLY[VILHJJLW[LKVUTHPUZ[YLHTJV\YZLZ;V[OPZLUK
[OL,UNSPZO+LWHY[TLU[Y\UZHUPU[LUZP]L-V\UKH[PVU)YPKNPUN,UNSPZO7YVÄJPLUJ`7YVNYHT[VWYLWHYL
hundreds of students each year for the EPT and for their future academic study at the university.
Pass rates on the EPT were proving to be modest at best, and international students often struggled
with the English language demands of their mainstream study even when they had passed the test.
The program management team in the English Department responded by implementing a plan to
improve pass rates on the EPT and upgrade the English for Academic Purposes (EAP) content of
[OLPY WYVNYHT ([ P[Z JVYL [OPZ WSHU PU]VS]LK JOHUNPUN [OL WYVÄJPLUJ` WYVNYHT MYVT H [YHKP[PVUHS
grammar-based curriculum to a task-based approach focused on engaging students in academic
\ZLZVM,UNSPZO(JJVYKPUNS`ZLUPVYZ[HќYLKLZPNULK[OLJ\YYPJ\S\THUKWYVK\JLKPUOV\ZLUL^ZL[Z
of ostensibly task-based teaching materials. However, these changes were not entirely successful.
:VTL VM [OL [LHJOPUN Z[Hќ YLZPZ[LK [OL UL^ J\YYPJ\S\T VY ^LYL \UJSLHY HIV\[ ÔH[ ^HZ L_WLJ[LK
VM [OLT ;OL Z[Hќ ÔV OHK KLZPNULK [OL J\YYPJ\S\T HSZV SHJRLK JVUÄKLUJL PU [OL X\HSP[` VM [OLPY
work, having carried it all out in-house. Students were also reported to be resistant to this new
communicative orientation in their classes. At this point, the management team initiated an evaluation
of the program.
What was the goal of the evaluation?

The goal was to evaluate the extent to which these recent curriculum innovations aligned with
international standards for best practice in task-based teaching and to provide feedback and
guidance on improving the program. Three objectives were set:
1. To identify strengths and weaknesses in the new task-based curriculum.
2. To establish an agreed set of principles and action steps for enhancing the curriculum.
3. To put in place a plan for ongoing evaluation of the curriculum drawing on data from key
stakeholders including management, teachers, and students.
Who was the audience for the evaluation?

The evaluation had three main audiences – senior management in the English Department who were
YLZWVUZPISL MVY ZL[[PUN [OL J\YYPJ\S\T ZLUPVY [LHJOPUN Z[Hќ ÔV ^LYL YLZWVUZPISL MVY ^YP[PUN [HZR
based classroom materials, and teachers who implemented the curriculum.
What information was collected and how?

First, an analysis of curriculum documents was carried out focusing on course outlines, detailed
lesson plans, and sample units of classroom materials. The analysis examined four aspects of task-
based teaching:
1. The nature of the tasks themselves.
2. The nature of the activity cycles (teaching units and lessons) within which tasks are set.
3. The sequencing of tasks within and across courses.
4. Task-based assessment.
This evaluation was guided by a taxonomy of task features and a set of principles for task-based
teaching proposed by Ellis (2003, pp. 9–10 and 276–278). From this analysis I drew up a summary of
the main areas of alignment and misalignment between the program and current understandings of
good practice in task design and task-based language teaching (TBLT). It was clear from the analysis
that tasks in the new curriculum were not consistently designed on task-based principles and that
little attention had been given to the design of task-based activity cycles or to sequencing tasks so
as to provide progression across the program. In particular, the materials showed little awareness
of how to incorporate systematic coverage of grammar and vocabulary in a task-based curriculum.
:LJVUK HSS [LHJOPUN HUK THUHNLTLU[ Z[Hќ JVTWSL[LK HU VUSPUL Z\Y]L` KLZPNULK [V VI[HPU
information on their beliefs about language learning and teaching and perceptions of the recent shift
towards task-based teaching. The survey included Likert-scale items and short-answer questions
on topics such as preferred teaching approaches, issues and challenges faced in delivering the
new curriculum, and a series of questions on attitudes to and experience of task-based and
JVTT\UPJH[P]L SHUN\HNL [LHJOPUN 0[ ^HZ L]PKLU[ MYVT [OL Z\Y]L` YLZ\S[Z [OH[ THU` Z[Hќ OHK SP[[SL
understanding of the principles of task-based teaching and were not convinced that a move away
MYVTWYPVYP[PaPUNNYHTTHY[LHJOPUNHUKSLHYUPUN^HZHNVVK[OPUN;OL`WLYJLP]LK[OH[[OLYL^HZH
mismatch between the new curriculum and the high-stakes EPT, and that coverage of grammar and
vocabulary was not systematic enough.
How did you use the information?

First, I drew on information from the evaluation to design and run a series of professional learning
ZLTPUHYZHUK^VYRZOVWZVU[HZRIHZLK[LHJOPUNMVY[LHJOPUNZ[HќVU[OLWYVNYHT0UJS\KLKPULHJO
^VYRZOVW^HZHMVY\TPUÔPJO[OLZ\Y]L`YLZ\S[Z^LYLYLWVY[LKIHJR[VZ[HќHUKLHJORL`PZZ\L
discussed. As an ancillary activity, a group of teachers observed me teaching a “model” task-based
lesson to students in the program, followed by a debrief session with the teachers.
:LJVUK0KL]LSVWLKHUKYHUHZLYPLZVMWYVMLZZPVUHSKL]LSVWTLU[ZLZZPVUZMVYZLUPVYZ[HќHUK[OVZL
responsible for designing materials. Work carried out during these sessions included giving materials
KLZPNU[LHTZZHTWSLZL[ZVMTH[LYPHSZ[VHUHSàLHUKYLKLZPNUHM[LYÔPJOLHJO[LHTWYLZLU[LK[OLPY
YLKLZPNULK\UP[ZPUWSLUHY`ZLZZPVUZ0U[OLÄUHSYV\UK[HISLZLZZPVU^P[O[OLTHUHNLTLU[[LHTHU
action plan was developed detailing the further steps to be taken to address issues highlighted in
the evaluation.
:KDWZHUH\RXUðQGLQJVDQGUHFRPPHQGDWLRQV"
;OLL]HS\H[PVUPKLU[PÄLKZ\IZ[HU[PHS^LHRULZZLZPU[OLWYVJLZZVMPTWSLTLU[PUN[OPZUL^[HZRIHZLK
J\YYPJ\S\T:WLJPÄJHSS`[OLPZZ\LZ^LYL!
1. PUZ\ѝJPLU[ L_WLY[PZL HUK HSSVJH[PVUJVVYKPUH[PVU VM YLZV\YJLZ SLHKPUN [V WVVY PUZ[HU[PH[PVU VM
TBLT in teaching materials and unit outlines;
2. lack of teacher professional learning through the induction period for the new curriculum, leading
to teacher confusion and resistance;
3. failure to “sell” this new approach to students or to provide learning training, leading to student
KPZHќLJ[PVU
An action plan was developed to address four priority areas: the structure of the curriculum, materials
KLZPNU[LHJOLYWYVMLZZPVUHSSLHYUPUNHUKSLHYULY[YHPUPUN-VYLHJOHYLHRL`^LHRULZZLZ^LYLÄYZ[
PKLU[PÄLKHUKKPZJ\ZZLK;OLUH[PTLIV\UKTLHZ\YLHISLVIQLJ[P]L^HZZL[-PUHSS`HZL[VMHJ[PVU
steps and responsibilities were agreed on. To underpin this action plan, we established a process of
VUNVPUNL]HS\H[PVUVM[OLWYVNYHTÔPJOLTWOHZPaLKNH[OLYPUNL]PKLUJLVM[OLPTWHJ[VMJ\YYPJ\S\T
innovation on student satisfaction and on learning outcomes over future iterations of the program.
Reference
Ellis, R. 2003. ;HZR)HZLK3HUN\HNL3LHYUPUNHUK;LHJOPUNOxford: Oxford University Press.
+Y 1VUH[OHU 5L^[VU PZ H ZLUPVY SLJ[\YLY H[=PJ[VYPH<UP]LYZP[`VM>LSSPUN[VU5LÂLHSHUK/LOHZ

^VYRLK PU SHUN\HNL [LHJOPUN HUK [LHJOLY LK\JH[PVU MVY `LHYZ /PZ YLZLHYJO PU[LYLZ[Z PUJS\KL
ZLJVUKSHUN\HNL]VJHI\SHY`HJX\PZP[PVUSLHYUPUN[OYV\NO[HZRZHUKPU[LYJ\S[\YHSSHUN\HNL[LHJOPUN
HUKSLHYUPUN/LOHZW\ISPZOLK^PKLS`PUJS\KPUN[^VJVH\[OVYLKIVVRZ Teaching ESL/EFL Listening
and Speaking and Workplace Talk in Action: An ESOL Resource .
CASE STUDY 21,]HS\H[PUNHU,UNSPZOJV\YZLMVY[LY[PHY`SL]LSSLHYULYZ

+H]PK*YHIIL
What is the context and general purpose of the evaluation?
The teaching team at a tertiary college in a large Asian city, where English is a foreign language, is
HNYV\WVMÄ]L[LHJOLYZ,HJOVM[OLTOHZH[SLHZ[MV\Y`LHYZ»JSHZZYVVTL_WLYPLUJL;OLPY,UNSPZO
classes are for students who are majoring in a range of professional subjects ranging from commerce
to engineering. The programme is largely task-based, designed around a common-core approach in
which all students are exposed to the use of English in professional rather than technical contexts as
HWYLWHYH[PVUMVYM\[\YL^VYRWSHJLJVTT\UPJH[PVU(SS[OL[LHJOLYZHYLJVUÄKLU[[OH[[OLPYJSHZZLZ
are going well, in the sense that the students participate in class activity and appear to be engaged.
Test scores show the usual range of performance.
The teachers recently decided that they will meet regularly to explore their teaching in more depth.
;OPZ^HZWYVTW[LKWHY[SÌ`[OLMHJ[[OH[[^VVM[OLTHYLLUYVSSLKPUHUVUSPULTHZ[LY»ZX\HSPÄJH[PVU
which expects a critical approach to learning and teaching, searching for evidence of the process and
outcome of learning beyond the anecdotal.
How did the teachers make sure that their evaluation was designed in a way that was
going to be useful?
0U[OLÄYZ[[LHTKPZJ\ZZPVU[OL`ZWLU[ZVTL[PTLKPZJ\ZZPUNL_HJ[S`ÔH[ they wanted to evaluate
in their course. They came up with an initial list of three questions:
• Are the materials that are being used (a mix of published materials and teacher-made materials)
having a positive or negative impact on learning?
• What learning is taking place from the classroom communicative tasks that form the main focus
of the course?
• What language learning do the learners do outside the classroom?
At a subsequent meeting, these questions were critically evaluated. Were they the right questions?
Can they be answered, and if so how useful would the answers be for guiding classroom practice?
;OL [LHT LUKLK \W YLQLJ[PUN [OL ÄYZ[ X\LZ[PVU ILJH\ZL P[ Z\NNLZ[Z [OH[ [OL TH[LYPHSZ ^LYL HU
instrument that have a direct impact on learning, whereas the team was more interested in learning
WYVJLZZLZHUKV\[JVTLZÔPJOTPNO[ILWHY[PHSS`PUÅ\LUJLKI`[OLTH[LYPHSZI\[Z\JOHUPUÅ\LUJL
^V\SKILKPѝJ\S[[VLZ[HISPZO
The second question was considered to be more important, but again, the team had similar
reservations. A task is not so much an instrument as a bundled set of learning opportunities, and
each implementation of a task has a life of its own. On that view, anything they found out about a
task might not be so valid for the next use of that task.
The third question remained important to the team. They knew that they had to know about all the
learning going on in the background if they were to understand what the impact of the classroom
activity was.
When it came to discussion of the purpose of the evaluation, there was an even bigger issue with
these questions. The answers would only capture the value of the status quo from a teacher’s
perspective. What would the impact be if the learners themselves were engaged in the evaluation?
Would that process of engaging the learners themselves in evaluating their learning prompt changes
in learning behaviour and learning outcomes? Their approach had shifted from a focus on the
correlation between instrumental means and learning ends to a focus on the stakeholders as agents
and the impact of focusing on that agency.
This was an interesting new direction. Just through discussion amongst themselves, the team had
shifted their own perspective on what they wanted to do. They had moved from evaluating the status
quo to evaluating a change in the status quo. On the basis of this shift in thinking, the team came up
with just one evaluation question:
• What is the impact on a learning activity when learners are asked to keep track of and evaluate
their own learning over the period of the course?
The team liked the dynamic nature of this evaluation. By asking this question, they would get
evaluative data from students on the learning experiences of the course but at the same time be
evaluating a shift in the responsibilities of learners and teachers.
What information was collected and how?

Having settled the purpose and focus of the evaluation, the team worked on how they would get the
learners to keep track of and evaluate their learning. Firstly, they discussed with the learners what
they were intending to do, suggesting that talking about their learning might well enhance the learning
process. They then collected information from the following activities:
• They asked the learners to share what their learning goals were from day to day on a web-based
discussion group and to comment on their individual progress against those goals. This gave
the teachers good information on what learning the learners thought was necessary and what
success they felt they had had.
• They set up a learning conversation in each lesson. The conversation ran like a thin thread
through the lessons, reminding the learners of the learning goals and opportunities and checking
on what they felt they could and could not do at times during the lesson and at the end. The
teachers were interested in the quality of the thinking about learning that appeared in the
conversations. They noted down the issues that were raised and what strategies emerged.
• 0U[OLÄUHSJV\YZLL]HS\H[PVUX\LZ[PVUUHPYL[^VVM[OLX\LZ[PVUZHZRLK[OLSLHYULYZ[VYH[L[V
ÔH[L_[LU[[OLÒHKILULÄ[LKMYVT[OLSLHYUPUNJVU]LYZH[PVUHUK[OLKPZJ\ZZPVUNYV\W
At the end of this process, the teachers had information about goals and progress, challenges and
strategies, and learner perceptions of the value of thinking and talking about learning.
What was learned from the evaluation?

There were several lessons that the teachers took from the evaluation:
• ;OLÄYZ[NVHSZ[OLZ[\KLU[ZZL[^LYL\UYLHSPZ[PJLN¸0^HU[[VILHISL[V\UKLYZ[HUK,UNSPZO
movies”) and inevitably not achieved over the course. Later goals, with the help of the classroom
JVU]LYZH[PVUZILJHTLTVYLZWLJPÄJHUKZOVY[[LYT¸0^HU[[VPTWYV]LT`Å\LUJ`PUTHRPUN
simple statements”; “I want to be able to improve my pronunciation”; “My vocabulary needs
building up”). The students recorded progress against them.
• The contributions to classroom conversations about learning were more detailed when they took
place immediately after a shared learning activity rather than before an activity.
• The learners themselves had a good range of suggestions for strategies on how to achieve their
learning goals.
• ;OL\ZLVM[OLÄYZ[SHUN\HNLPU[OLVUSPULJVU]LYZH[PVUZLUHISLKTVYLJVTWSL_Z[H[LTLU[Z
HIV\[SLHYUPUN[VILTHKLHUK[O\ZKPKUV[WYL]LU[[OVZL^P[OSV^LYWYVÄJPLUJ`MYVT
contributing. (Two of the teachers were native speakers of the students’ L1.)
• Not all learners participated fully in all the evaluation activities. For whatever reason, not all
SLHYULYZ^PSSLUNHNL^P[OL]LY`[OPUN[OH[H[LHJOLYVYNHUPaLZ
How useful was the learning gained from the evaluation?

The teachers found the expression of personal learning goals was productive in prompting learning
activity. While the course had objectives, these did not always relate to an individual student’s
immediate learning needs. The teachers made sure that personal goal setting was a feature of their
course.
While it was not possible to demonstrate that learner engagement in monitoring their learning actually
led to better learning outcomes, the students evaluated the activity reasonably highly and reported
it as useful. Because of this, the teachers continued with it, and the next step would be to evaluate
more explicitly the impact of learning conversations on out-of-class learning activity and on aspects
VMJVUÄKLUJLHUKWYVÄJPLUJ`
+H]PK*YHIILPZJ\YYLU[S`[OL+LHUVM,K\JH[PVUH[=PJ[VYPH<UP]LYZP[`VM>LSSPUN[VU/LOHZL_[LUZP]L
WYHJ[PJHS HUK HJHKLTPJ L_WLYPLUJL PU [OL ÄLSK VM ,3; HUK W\ISPZOLZ VU J\YYPJ\S\T THUHNLTLU[
JSHZZYVVTWYHJ[PJLHUKSLHYULYH\[VUVT`/LOHZSP]LKHUK[H\NO[PU,\YVWL(MYPJH:V\[OLHZ[(ZPH
HUK5LÂLHSHUK

Approaches To Evaluation: Chapter Overview

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Approaches To Evaluation: Chapter Overview

Uploaded by

Copyright:

Available Formats

11 APPROACHES TO EVALUATION

• The focus of evaluation • Process-focused evaluation

11.1 The focus of evaluation

/V^L]LYHZ2PLS`HUK9LH+PJRPUZLTWOHZPaL [OLNVHSZVML]HS\H[PVUHYL[VSLHYU from

11.2 Audience for evaluation

• How useful were the materials and course work?

Can you suggest other questions an evaluation may seek to answer?

1. What did you like about this course?

8. Other comments/ suggestions?

11.3 Quantitative and qualitative approaches

11.4 Product-focused evaluation

11.5 Formative and summative evaluation

The other aspect of a product-focused or ends–means approach to evaluation is Z\TTH[P]L

,ѝJPLUJ`VM[OLJV\YZLAnother measure of the success of a course is how straightforward the course

11.6 The importance of documentation

*V\YZLZ[H[PZ[PJZ!information on why students chose the course, student numbers, attendance,

9LSL]HU[JV\YZLKVJ\TLU[Z!compilation of all relevant documentation about the course, such as

*V\YZL^VYR!examples of tests, class assignments, students’ work.

11.7 Evaluating the evaluation

11.8 Procedures used in conducting evaluations

Comparison of two approaches to a course

Teachers’ written evaluations

(K]HU[HNLZ!Teachers are in a good position to report on a course, and a well-designed evaluation

Diaries and journals

Audio or video recordings

Table 11.1 Issues and procedures in evaluation

review of lesson plans

review of lesson plans

Lesson plan reviews

Comparison of present term grades to previous grades (or grades of

11.9 Process-focused evaluation: descriptive and reflective evaluation

,_HTWSL! A teacher is interested in learning more about teacher–student interaction in her

11.10 Implementing reflective evaluation

• What caused me to want to become a second language teacher?

• Do these reasons still exist for me now?

• What does it mean to be a teacher?

• Is the teacher I am the person I am?

• How did I come to appropriate them?

• Why do I continue to endorse them now in my teaching?

• Whose interests do these ideas serve?

• Who has power in my classroom and how is it expressed?

• How do power relationships in my classroom influence my interactions with students?

• How might I teach differently?

• What is the nature of knowledge that guides my teaching of content?

• Whose interests does this knowledge about language teaching serve?

• How do/can I personally work to uncover the contradictions in my teaching?

• How does what I do affect the opportunities in life of students?

• Do I wish to uncover the “hidden curriculum” – the inconsistencies – in my teaching?

1. Focus on quality of life as the fundamental issues.

1. Form a Team: Teams are usually composed of 3–6 instructors.

9. Read Case study 21 by David Crabbe.

APPENDIX 1 Best practice in English language teaching*

The physical facilities contribute to an atmosphere conducive to learning.

MANAGEMENT AND ADMINISTRATION

The management strives to engender a positive teaching and learning environment.

CASE STUDY 19 Evaluating an in-service program for English language teachers

What was the goal of the evaluation?

Who was the audience for the evaluation?

How did you conduct the evaluation?

What kind of information did you collect?

CASE STUDY 20 Evaluating the content of an EAP program

What was the goal of the evaluation?

Who was the audience for the evaluation?

/V^L]LYHZ2PLS`HUK9LH+PJRPUZLTWOHZPaL [OLNVHSZVML]HS\H[PVUHYL[VSLHYU from

,ѝJPLUJ`VM[OLJV\YZLAnother measure of the success of a course is how straightforward the course

,_HTWSL! A teacher is interested in learning more about teacher–student interaction in her

+Y 1VUH[OHU 5L^[VU PZ H ZLUPVY SLJ[\YLY H[=PJ[VYPH<UP]LYZP[`VM>LSSPUN[VU5L^ALHSHUK/LOHZ

CASE STUDY 21,]HS\H[PUNHU,UNSPZOJV\YZLMVY[LY[PHY`SL]LSSLHYULYZ