LANGUAGE ASSESSMENT QUARTERLY, 3(4), 333368 Copyright 2006, Lawrence Erlbaum Associates, Inc.

Watching for Washback: Observing the Influence of the International English Language Testing System Academic Writing Test in the Classroom
Anthony Green
University of Cambridge ESOL Examinations

Previous studies of washback (the influence of a test on teaching and learning) have provided insights into the complexity of educational systems and test use, especially in relation to the role of the teacher, but have given insufficient attention to the relationship between observed practices and test design features. In this article a washback model is proposed that incorporates both test design and participant characteristics. The model is used to predict behaviour on preparation courses directed toward the Academic Writing component of the International English Language Testing System (IELTS) test. 197 learners and 20 teachers were observed over 51 classroom hours. These encompassed 22 IELTS preparation classes and, for comparison, 13 classes of English for academic purposes (EAP). Evidence was found for substantial areas of common practice between IELTS and other forms of EAP but also for some narrowing of focus in IELTS preparation classes that could be traced to test design features.

Before offering a place to an international student, most universities in English-speaking countries will require evidence of the students language ability. As increasing numbers of students choose to travel to access global educational opportunities, there has been rapid growth in the use of language tests for this purpose. In the United Kingdom the most widely recognised test of English for academic purposes (EAP) is the International English Language Testing System (IELTS). Between 1995 and 2005 the number of candidates rose from under 50,000 to over half a million per year (International English Language Testing
Correspondence should be addressed to Anthony Green, Validation Unit, University of Cambridge ESOL, 1 Hills Road, Cambridge, CB1 2EU, UK. E-mail: green.a@cambridgeesol.org

334

GREEN

System, 2005). The rapid expansion of the test has brought with it increased demand for test preparation books and courses. IELTS is used as a means of determining whether candidates should be accepted into English-medium courses and whether they will require further language support. However, concern has been expressed that preparation for tests like IELTS, because of the limitations on what can realistically and equitably be tested in a few hours, may not develop the full range of skills required for successful university study, particularly in the area of academic writing (Deakin, 1997; Read & Hirsh, 2005). J. Turner (2004), for example, argues that what the IELTS test or the TOEFL test delivers underspecifies the complexity of language issues in the academic context (p. 98). Her concern is that education in academic literacy is being supplanted by training in test taking. But what influence does the IELTS writing test really have on teaching, and how different are writing classes in IELTS preparation courses from other forms of EAP? To investigate these questions, this article compares the practices observed in writing classes of two types: IELTS preparation classes directed at success on the test and, as a suitable point of comparison, presessional EAP writing classes provided by universities to prepare learners for academic study.

LITERATURE REVIEW Washback studies, investigating the effects of tests on the teaching and learning directed toward them, have often involved direct observation of the behaviour of teachers and learners in the classroom. The inclusion of an observational element in such studies has been recommended as a means of contextualising, corroborating, or correcting data from surveys and interviews (Alderson & Wall, 1993; C. Turner, 2001; Wall, 1996; Watanabe, 2004). Table 1 summarises the methods and findings of recent case study investigations of washback in language education that have included an observational element. These studies covered a wide range of educational contexts, with observation either focussing on a small number of participants observed intensively over a sustained period (Alderson & Hamp-Lyons, 1996; Read & Hayes, 2003) or on a lighter sampling of classes to allow for observation of larger numbers of teachers and a broader perspective (Hawkey, 2006; Wall, 2005). With the exception of Burrows (1998, 2004) and Hawkey (2006), all included comparisons between different types of class. Wall (1996, 2005) and Cheng (2005) focused on changes over time as a new test is introduced. This approach also informed work relating to the recent update of the Test of English as a Foreign Language (TOEFL; Hamp-Lyons, 2005; Wall & Hork, 2004). Alderson and Hamp-Lyons (1996) and Watanabe (1996, 2004) compared the practices of teach-

TABLE 1 Studies of Washback in Language Education That Include Observation and Interview Data Observations Purpose-designed instrument, 2 teachers 16 classes: 8 general English, 8 TOEFL preparation over 1 week 3 focus group interviews with 312 students; 9 teachers in group and individual sessions N and Frequency Interviews Key Findings

Study

Exam

Institutions

Alderson and Hamp-Lyons (1996)

Test of English as a Foreign Language (TOEFL)

Specialised language institute in United States

Brown (1998)

International English Language Testing System (IELTS)

University-based language institute

Records of instructional focus and materials, 2 teachers

All classes designated for writing over 10 weeks; 70 hours in IELTS and 30 hours in non-IELTS EAP course

Not reported

TOEFL washback may be generated more by teachers, materials, writers, and administrators than by test. Amount and type of washback vary according to test status, relation of test to nontest practice, degree to which teachers and material writers think about appropriate methods and their willingness to innovate. IELTS preparation instruction closely focussed on the test with regular timed practice of IELTS tasks. IELTS preparation was more successful than EAP course in improving IELTS scores from entry to exit, but small samples (9 IELTS, 5 EAP) limit interpretation.

335

(continued)

TABLE 1 (Continued) Institutions COLT, 4 teachers Two 4-hour lessons per teacher Observations N and Frequency Interviews Key Findings

336
Adaptation of COLT, 3 teachers 30 teacher interviews including those observed (conducted prior to the observation phase) Follow-up 31 70-min classes interviews with c. 6 months observed before exam over teachers 2 years Cambridge IIS instrument, 10 teachers 10 IELTS preparation classes Teachers vary in responses to innovations in assessment in line with individual differences and experience. With Cheng (2003) and Wall (2005), links washback to theories of change. Following innovation, teachers engaged in exam-like activities e.g., role-plays, and used (conservative) exam prep materials in class, but beliefs and attitudes to learning did not change. Differences in how teachers cope with change. Extensive support required in implementation of innovations. Teacher preference for 120 students, 21 task-based, often teachers, and 15 interrelated macroskills receiving activities, involving institution microskills relevant to administrators IELTS. Focus on IELTS in focus groups but willingness to use a range of materials and methods.

Study

Exam

Burrows (1998; 2004)

Certificates in Spoken and Written English (CSWE)

Adult English Migrant English Programme (AMEP) in Australia

Cheng (2005)

Hong Kong Certificate Examinations in English (HKCEE)

3 HK secondary schools (main study)

Hawkey (2006)

IELTS

10 language schools in UK, Japan, Cambodia

Read and Hayes (2003); Hayes and Read (2004)

IELTS

2 language schools in New Zealand

COLT and Cambridge IIS instrument, 2 teachers

Wall (2005)

Sri Lankan O level

c. 50 secondary schools in 11 areas of Sri Lanka

Purpose-designed observation checklist

22 hours of 32-hour 23 teacher interviews; course and 28 weekly hours of 320interviews with hour (8-month) the teachers course observed 5 lessons observed 64 teachers in focus groups; over 2 years follow-up (3964 different interviews with classes observed each class per round) at observation varying periods ahead of exam

Watanabe (1996; 2004)

Various Japanese university entrance examinations

3 high schools in Japan

Adaptation of COLT, 5 teachers

964 min of exam preparation and 833 min of regular classes over 6 months

Teacher follow-up interviews with each observation

Differential teacher practices at least partially linked to institutional context. Greater pressure to teach to the test in private language schools. Teachers used textbook content but failed to follow suggested methods, e.g., reading for gist. Proportion of classes was dedicated to exam preparation. Neglect of speaking skills traceable to exam content. Teachers vary in their approaches to exam preparationinformed by attitudes towards the exam. School culture an important factor. Material designed for exam preparation may sometimes be used for other purposes.

337

338

GREEN

ers in test preparation and non-test-preparation classes, while Read and Hayes (2003; Hayes & Read, 2004) compared two approaches to test preparation. Three of the observational washback studies listed in Table 1 investigated IELTS preparation classrooms. Brown (1998) compared practices in two courses provided by the same institution: IELTS preparation and non-IELTS EAP. Read and Hayes (2003) compared two IELTS preparation courses at different institutions: one an intensive preparation course, the other combining IELTS preparation with other forms of EAP. The ongoing IELTS Impact Study (IIS; Hawkey, 2006) also includes observational data of IELTS preparation classes. These studies all found IELTS to affect behaviour. Brown (1998) found that students in the 10-week IELTS preparation course used IELTS preparation textbooks, completed one Task 1 and one Task 2 essay each week (and no other writing), performed three timed practice examinations, were informed about IELTS scoring criteria, received feedback on the accuracy of their written work, and were instructed in strategies for writing under timed conditions. In contrast, students in the EAP course worked on a 1,000-word project, did no timed writing, were instructed in strategies for writing in academic contexts, and were encouraged to develop research skills. Read and Hayes (2003) combined a broad survey with targeted classroom observation of two teachers to provide mutual corroboration of findings. The briefer and more intensive of the two IELTS classes they observed was more narrowly concerned with the test and included more test practice under timed conditions. In common with other washback studies, Hawkey (2006) found variation between the 10 teachers he observed in how they conducted their classes, notably in the number of opportunities they provided for learners to communicate together in English. Teachers were willing to employ a variety of teaching methods and to use material both within and beyond the textbook. However, both the institutions providing the courses and the students, who were motivated to succeed on the test, appeared to constrain teachers to focus their instruction on IELTS. In class, teachers showed a preference for task-based activities, targeting microskills they believed to be relevant to the test. Data from observational studies have informed insights into the complexity of educational systems and test use, especially in relation to the role of the teacher. However, a shortcoming identified by Bachman (2005) is the lack of a coherent evidential link between test design characteristics and the practices observed or reported in the classroom. In studies that involve predicting the effects of a test on instruction or learning, appeal is more often made to the views of educational authorities (Cheng, 2005; Ferman, 2004; Qi, 2004), teachers (Banerjee, 1996), or widely held public perceptions (Watanabe, 1996) than directly to the design of the test instrument (as evidenced by available test materials or test specifications). How might the design of the IELTS Academic Writing component be expected to influence instruction?

WATCHING FOR WASHBACK

339

Watanabe (2004) suggested two sources of evidence that may be used to relate observed practices to the influence of a test. One is evidence that test design features are reflected in teaching or learning. The other is the absence of such features in teaching or learning not directed toward the test (or directed toward an alternative test). Of the three IELTS-related studies, only Brown (1998), in common with Alderson and Hamp-Lyonss (1996) study of TOEFL preparation, incorporated a comparison with courses that were not directed toward the test. The inclusion of just two classes in each of the Australian studies limits their generalisability, while the lack of a nontest comparison in Hawkey (2006) makes it difficult to disentangle test influence from teacher variables. This study involves observation of a larger number of teachers and learners than Brown (1998) and Read and Hayes (2003) but includes the comparison with nontest EAP classes missing from Hawkey (2006). In considering the mechanisms of washback, a growing body of theory relates test design, test use, and classroom behaviours, although as Wall (2005) argued, too little is sufficiently informed by empirical evidence. Most of this work takes the form of recommendations to test developers. Chapman and Snyder (2000) provided a framework for relating tests to educational practices, and Brown (2000) cited Hughes (1989), Heyneman and Ransom (1990), Kellaghan and Greaney (1992), Bailey (1996), and Wall (1996) in identifying features of a test that may be manipulated in efforts to improve instruction. These embrace both contexts for test use and technical qualities of the test instrument. Drawing together these two elements in washback theory, Green (2003) proposed the predictive model of test washback set out in Figure 1. The model starts from test design characteristics and related validity issues of construct representation identified with washback by Messick (1996) and encapsulated in Resnick and Resnicks (1992) formulation of overlap, or the extent of congruence between test design and skills developed by a curriculum or required in a target language use domain. Test design issues are most closely identified with the direction of washbackwhether effects are likely to be judged beneficial or damaging to teaching and learning. The model relates design issues to contexts of test use, including the extent to which participants (including material writers, teachers, learners, and course providers) are aware of and are equipped to address the demands of the test and are willing to embrace beliefs about learning embodied therein. These features are most closely related to washback variability (differences between participants in how they are affected by a test) and washback intensity. Washback will be most intensehave the most powerful effects on teaching and learning behaviours where participants see the test as challenging and the results as important (perhaps because they are associated with high stakes decisions, such as university entrance; Bailey, 1999; Cheng, 2005; Hughes, 1993; Watanabe, 2001). A survey of IELTS preparation courses conducted in tandem with this study (Green, 2003) indicated that the test was regarded both as important and challeng-

340

GREEN

FIGURE 1 (2003).

A model of washback direction, variability, and intensity adapted from Green

ing by a majority of learners in the IELTS preparation courses involved (70% rated success on the test as very important). It also indicated that almost all the learners were taking the test because they intended to enter higher education in the United Kingdom. Most demonstrated at least a basic knowledge of the test format and viewed the direct test of writing as a positive feature. In short, the conditions for intense washback to a majority of participants would seem to be in place. Beyond a general prediction that the direct testing of writing in IELTS would encourage the teaching of writing in test preparation programs, the washback model suggests that features of task design will impact on the nature of that instruction. The IELTS Academic Writing component (AWC) is one of four skill-based subtests in the IELTS battery intended to assess the language ability of candidates who intend to study or work where English is used as the language of communication (International English Language Testing System, 2005). On the AWC, candidates are required to compose two short essays in 1 hour. The first

WATCHING FOR WASHBACK

341

(Task 1) is a description of a diagram or table, and the second (Task 2) is a discursive essay written in response to a point of view, argument, or problem. Drawing both on the available literature (Banerjee, 2000; Chalhoub-Deville & Turner, 2000; Coffin, 2004; Douglas, 2000; Hale et al., 1996; Mickan & Slater, 2003; Moore & Morton, 1999; Thorp & Kennedy, 2003) and on the views of course providers, a framework developed by Weigle (2002) was used to relate the design of the IELTS AWC to theories of academic literacy. Information about the test was derived from sample test materials published by the IELTS partners and from unpublished specifications for item writers obtained from Cambridge English for Speakers of Other Languages (Cambridge Assessment, Cambridge, UK) under a research agreement. The review highlighted the similarities between the two IELTS tasks and paradigmatic reports or essays assigned by participant teachers in universities but also highlighted differences between the design of the IELTS AWC and the broader concerns of EAP. Briefly, the following areas emerged as the key differences between the two. IELTS tasks involve

abstract and impersonal topics, but these are not targeted at learners chosen
academic subjects

a limited range of text types (descriptions of iconic data and five-paragraph

persuasive essays)

composition of texts based on personal opinions about how the world should
be. They do not involve

literature reviews, summaries, and other genres of relevance to the academic

context arguments based in reading and research (as university assignments often do). In short, IELTS may imply an approach to instruction that passes over features of canonical EAP, such as the integration of source material in learners writing, learning of subject- specific vocabulary and text types, and strategies for coping with the length of university-level written assignments. Preparing for IELTS may involve learning how to shape texts to meet the expectations of examiners (as expressed through the scoring criteria) rather than those of university staff. The scoring criteria may imply a focus in the classroom on grammar, vocabulary, and organisation of text, with limited attention given to quality of content. The timed conditions may lead to practice in composing short texts under timed conditions, perhaps in response to past (or imitation) examination papers.

342

GREEN

It should be noted that the IELTS AWC has undergone continual modification and revision since its inception in 1989. The test has changed in certain respects since this research was conducted. Where these changes are of relevance to the study, this is indicated in the text. Regular updates on IELTS developments can be found on the IELTS Web site (http://www.ielts.org). METHODS Methodology The literature review has pointed to specific features that might indicate the influence of IELTS AWC test design on instruction. To investigate whether these predictions about the likely impact of IELTS would be borne out in this context and to explore how instruction varied between IELTS preparation and EAP classes in academic writing, a series of classroom observations were carried out at selected U.K. centres. The observations provided evidence of how preparation for the IELTS AWC was conducted in practice and how this compared with other forms of EAP writing provision. The courses included in the study involved IELTS preparation in combination with varying proportions of EAP and general English classes. However, comparisons were made at the narrower level of the writing class: between classes directed toward the IELTS AWC and EAP classes directed at preparing learners for writing in English at U.K. universities. Instrumentation One observation instrument that has been widely used in washback studies (Burrows, 1998; Cheng, 2005; Read & Hayes, 2003; Watanabe, 1996) is the Communicative Orientation to Language Teaching (COLT) observation schedule (Spada & Frhlich, 1995). The scheme is designed to be used flexibly and in real time, describing classroom events at the level of activities and their constituent episodes. These are described qualitatively and recorded quantitatively under a series of headings to build a picture of the balance of patterns of classroom organisation, content, and student modality. Spada and Frhlich (1995) explain the terms thus: Separate activities include such things as a drill, a translation task, a discussion or a game. Three episodes of one activity would be: teacher introduces dialogue, teacher reads dialogue aloud, individual students read parts of dialogue aloud (p. 14). As Read and Hayes (2003) discovered, the COLT schedule could not, without adaptation, identify features of direct test preparation, such as learning test-taking strategies. Read and Hayes (2003) chose to supplement the COLT with the draft IIS observation schedule, an instrument developed by the IELTS partners in collaboration with researchers from the University of Lancaster at the inception of the IIS (Saville, 2000). This schedule includes lists of text types and activities antici-

WATCHING FOR WASHBACK

343

pated to occur in preparation classes and is specifically designed to record instances of IELTS preparation. To address the needs of this study, but avoiding duplication, elements of the IIS instrument were incorporated into the COLT schedule, and the resulting instrument was further refined through piloting. The observation schedule appears in the Appendix. Among the modifications to the COLT, a Test References section was included with three categories: IELTS, Other, and Test Strategies. The IELTS section was used to record mentions of the IELTS test, a separate note being made of the context. The Other section was used to record mention of tests or assessments other than IELTS (such as course exit tests). The Test Strategies section was used as a record of test-taking strategy instruction, and the specific strategies were also noted. Copies of all materials used in class were collected for later analysis, and details were recorded separately on a second page of the observation form (see the Appendix). Under the heading content, the distinction made in the COLT between personal or broad topics was extended to further differentiate broad from academic. An academic topic was indicated where teachers and students treated the topic as academic subject matter: The primary focus would be learning about the topic, rather than exploiting the topic to learn about language. Additional notes were made of aspects of instruction, such as homework assignments, that were of interest but not captured by the schedule. The adapted schedule was piloted with five classes, including 240 min of presessional EAP and 150 min of IELTS preparation (60 in an IELTS intensive course and 90 in a course combining IELTS preparation with EAP). Observations were recorded in real time at intervals of 1 min, and the time in hours and minutes was entered at each episode boundary. To provide an estimate of internal consistency, two classes observed at the beginning of the data collection exercise were video recorded and reanalyzed in real time 3 months later. The 82.5% level of agreement between the two sets of observations suggests that the instrument was being used consistently during the study. Interrater reliability was investigated in cooperation with an IIS consultant, who was trained in the use of the schedule and independently observed the two videocassettes. These ratings showed complete agreement on the number of activities observed, with minor discrepancies in timing. There was agreement on 72% of the observed categories, with most of the differences being on the minor focus of an activity. For example, the first observer recorded that one activity was led by the teacher, but the second also recorded that this involved learners working in small groups. Participants and Settings To provide a cross section of courses, a range of institutions were approached by telephone to invite participation. These institutions were selected following an

344

GREEN

earlier survey of U.K. course providers; they had indicated willingness to participate in further research and were conveniently located. Three of these (two universities and one private language school) declined the invitation. The remaining institutionssix universities, three colleges of further education, and four private language schoolsrepresented a variety of courses for students intending to enter higher education in the United Kingdom, both in IELTS preparation and, for purposes of comparison, presessional courses in EAP offered by universities to prepare international students for the language demands of academic study. The intention was to observe a minimum of one writing-focused class for each month of a course, although this did not always prove possible. As summarised in Tables 2 and 3, some 36 scheduled classes were observed covering over 51 hours, involving a total of 197 learners and given by 20 different teachers. The classes included eight different EAP groups. Each of these was observed either once or twice over periods ranging from 8 to 12 weeks to give a total of 13 classes. There were 12 IELTS preparation groups. Each of these was observed on between one and three occasions over periods of between 4 and 12 weeks to give a total of 22 preparation classes. Following each observation, teachers were briefly interviewed about the class. The interviews focused on five issues: the aims of the class, the extent to which the aims had been met, the place of the focal class in a teaching sequence, the extent to which the class could be described as typical of writing classes on the course, and the influence of the IELTS AWC. Because of other commitments, teachers did not always have the time to take part in these interviews. Nonetheless, 22 of the classes were accompanied by interview data (8 EAP and 14 IELTS). Analysis Comparisons were made between writing-focused IELTS preparation classes and EAP classes across courses (Table 4). The length of time spent on each activity provided an index of how much time and what proportion of class time (calculated as a percentage) was given to each form of participant organisation, content focus, content control, and student modality. Mentions of tests or of test-taking strategies were treated as simple frequency data. As the data were not normally distributed, differences between classes were evaluated for significance (p < .05) through nonparametric MannWhitney U tests. The activities observed were also reviewed qualitatively in the light of the washback predictions to explore (a) how far the predictions were borne out in IELTS preparation and (b) how far they differentiated between writing-focused IELTS preparation and EAP classes. Follow-up interviews provided an opportunity to probe aspects of teachers behaviour.

TABLE 2 Summary of IELTS Preparation Classes Observed

Centre IELTS IELTS 1 (F) 2 (M) 6 12 8 8

Course Type

Class Title

Teacher

Week

No. of Weeks Activities

Hours/ Week

Course Hours/ Week Obs Min

No. of Ss

IELTS and EAP IELTS and Gen English

IELTS and Gen English IELTS and Gen English IELTS writing IELTS writing 6 (M) 6 10 5 (M) 6 10

IELTS IELTS

3 (M) 4 (F)

4 8

6 8

IELTS

IELTS

IELTS and EAP IELTS IELTS IELTS IELTS IELTS 10 (F) 11 (M) 12 (F) 9 (M) 8 8 8 10 7.50 8 (M) 8

IELTS

7 (F)

4.5 8 8 7.5 9 10 8.08

IELTS and EAP

IELTS and EAP

College A College B College B College B College C College D College D College E College E College E College E College E University A University A University B University B University B University B University C College F College G College G Average

IELTS and EAP IELTS and EAP IELTS and EAP

3 2 6 10 3 2 7 1 5 1 3 5 3 5 2 6 2 6 2 3 4 6

23 23 23 23 21 23 23 25 25 25 25 25 22.5 22.5 20 20 20 20 25 21 25 25 22.95

6 7 7 4 4 6 7 6 3 4 6 4 9 5 5 5 2 3 6 5 3 3 5.00

82 137 92 94 95 122 118 148 114 67 119 114 93 96 55 46 50 27 90 79 52 38 87.64

7 7 9 4 8 10 7 12 8 12 11 12 7 7 9 7 8 7 8 14 5 6 8.41

345

346
TABLE 3 Summary of EAP Classes Observed Class Title 12 (F) 13 (M) 15 (F) 16 (M) 17 (F) 18 (M) 19 (F) 20 (M) 8 8 8 9.25 8 9 7.5 7.5 7.5 8.38 8 10 12 7.5 12 8 10 10 Teacher Week No. of Weeks Activities Hours/ Week Course Hours/ Week Obs Min No. of Ss 4 6 3 11 4 8 2 6 2 6 7 7 2 25 25 23 23 20 20 28 28 28 28 25 25 25 24.85 6 5 9 5 5 5 4 3 7 7 4 3 4 5.15 61 58 127 79 123 126 89 52 97 95 93 98 73 90.08 11 11 7 5 15 11 13 14 14 14 12 10 12 11.46

Centre

Course Type

IELTS and EAP

Study skills

IELTS and EAP

Reading and writing

EAP

Reading and writing

EAP

Academic writing

EAP

Academic writing

College G College F College A College A University D University D University E University E University E University E University F University F University F Average

EAP EAP EAP

Reading and writing Reading and writing Reading and writing

TABLE 4 A Comparison of Timing, Episodes, and Activities in IELTSand EAP-Focused Classes (Minutes) Episodes per Activity 3.67 3.71 3.86 4.50 3.50 2.33 2.86 3.33 2.67 4.00 2.33 4.00 2.56 2.20 2.80 3.60 4.00 3.00 2.67 3.00 3.33 2.00 3.18 2.17 4.00 3.56 3.20 3.00 4.40 3.75 3.33 3.57 2.86 3.50 4.00 2.75 3.39 0.39

Course Type IELTS Preparation

Centre College A College B College B College B College C College D College D College E College E College E College E College E University A University A University B University B University B University B University C College F College G College G

Observed time 82 137 92 94 95 122 118 148 114 67 119 114 93 96 55 46 50 27 90 79 52 38 87.64 61 58 123 126 89 52 97 95 93 98 73 127 79 90.08 0.68

Activities 6 7 7 4 4 6 7 6 3 4 6 4 9 5 5 5 2 3 6 5 3 3 5.00 6 5 9 5 5 5 4 3 7 7 4 3 4 5.15 0.89

Episodes 22 26 27 18 14 14 20 20 8 16 14 16 23 11 14 18 8 9 16 15 10 6 15.68 13 20 32 16 15 22 15 10 25 20 14 12 11 17.31 0.61

Min per Activity 13.67 19.57 13.14 23.50 23.75 20.33 16.86 24.67 38.00 16.75 19.83 28.50 10.33 19.20 11.00 9.20 25.00 9.00 15.00 15.80 17.33 12.67 18.32 10.17 11.60 14.11 15.80 24.60 25.20 22.25 17.33 13.86 13.57 23.25 32.67 18.25 18.67 0.84

Average IELTS Preparation EAP

College G College G College A College A University D University D University E University E University E University E University F University F University F

Average EAP Significance (MannWhitney U test)

347

348

GREEN

RESULTS: FREQUENCY DATA The observed IELTS and academic writing classes were of similar length. The ratio of episodes to activities was also very similar across class types; there were approximately 10 episodes to every 3 activities, with each activity taking up just over 18 min on average. Table 5 compares participant organisation by course type. The results of the MannWhitney test displayed in the final row of Table 5 indicate that there were no significant (p < .05) differences between class types in participant organisation. The predominant form across classes was TeacherStudents/Class. Calculating this as a percentage of total class time for each course type gives 56% of presessional EAP class time and 54% of IELTS preparation class time (see Table 5). This did not generally involve lectures, but whole-class interactions centred on the teacher. The differences found by Read and Hayes (2003) between teachers in how they organised their classes were not repeated across the broader sample here, nor did students spend as much of their time in class on individual test practice. However, the proportion of class time spent on individual work on the same exercise did increase as IELTS classes progressed: an average of 20% of class time in the first half of these courses compared with 33% in the second half. Reflecting the focused atmosphere of the classes observed, discipline was not recorded as the main focus of any activity, although occasionally, a teacher reminded students to speak English rather than their first language. Procedural issues took up 9% of time in IELTS and 12% of time in presessional EAP courses. Observed variation in the amount of time spent on procedures also appeared to be a matter of teacher style, rather than of test focus. The results displayed in Table 6 reveal some evidence of differences between class types in content. Forty-six percent of IELTS class time involved a major focus on language form (grammar and vocabulary) as compared with 22% of presessional EAP class time. However, this did not prove to be significant in the analysis (p = .07). Across classes observed, topics were mostly broad: relating to issues of general rather than personal interest. Fifty-eight percent of time in IELTS preparation classes and 48% of time in presessional EAP classes was spent working with topics in this category (p = .41). Little time (4% of IELTS and 1% of presessional EAP classes) was spent on immediate personal topics (and these typically occurred only during brief introductory episodes). These proportions are similar to those observed by Read and Hayes (2003). Academic topicsthose topics that became the focus of the class in their own rightoccurred only in two presessional EAP classes, making up 2% of the total presessional EAP class time observed. Again, the differences between courses were not significant (p = .19). Across the classes observed, topics were generally limited to one or two activities. Switches of topic would occur with each change of task. However, there was evidence that topics were pursued in more depth in at least some EAP classes. Four

TABLE 5 A Comparison of Participant Organization in IELTS- and EAP-Focused Classes (Minutes)

Course Type 82 137 92 94 95 122 118 148 114 67 119 114 93 96 55 46 50 27 90 79 52 38 40 48 51 64 53 26 61 57 37 66 73 59 66 67 22 15 38 11 58 49 38 29 0 0 5 0 0 0 0 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 39 71 26 36 4 22 9 17 0 0 0 0 10 0 22 15 9 0 25 5 0 9 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 13 6 38 74 33 43 75 0 46 55 17 29 11 16 3 15 5 25 14 0

Centre

Observed Time TC/S SC/S

Individual Same

Individual Different

Group Same

Group Different 0 0 0 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

IELTS preparation

College A College B College B College B College C College D College D College E College E College E College E College E University A University A University B University B University B University B University C College F College G College G

349

(continued)

350
TABLE 5 (continued) Observed Time TC/S SC/S Individual Same Individual Different Group Same Group Different 87.64 61 58 123 126 89 52 97 95 93 98 73 127 79 90.08 0.68 46.73 31 41 64 34 28 64 62 50 78 53 52 36 20 47.15 0.90 0.41 0 0 14 0 0 0 0 2 0 0 24 0 23 4.85 0.09 14.50 3 8 49 12 13 17 16 0 16 35 6 17 0 14.77 0.68 0.00 0 0 0 0 0 0 0 0 0 0 0 5 22 2.08 0.06 23.55 26 18 0 20 43 45 7 0 6 6 20 40 0 17.77 0.69 0.77 0 0 0 0 0 0 0 0 0 0 0 0 3 0.23 0.73

Course Type

Centre

Average IELTS preparation EAP

College G College G College A College A University D University D University E University E University E University E University F University F University F

Average EAP Significance (MannWhitney U test)

TABLE 6 A Comparison of Content in IELTS- and EAP-Focused Classes (Minutes)

Course Type 3 20 18 0 7 13 15 7 8 3 5 5 4 9 3 4 9 1 14 15 5 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 81 0 82 37 101 40 0 15 57 100 29 37 87 28 22 44 0 40 14 13 35 0 10 0 10 21 47 0 13 0 57 26 0 0 43 22 0 0 0 51 52 2 0 0 13 74 0 37 0 34 0 0 9 8 3 10 29 8 0 0 24 19 0 24 0 0 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 4 0 0 29 0 0 0 0 0 0 0 40 0 0 0 9 0 0 72 105 78 44 50 113 77 48 60 8 53 41 45 38 48 0 44 25 71 41 19 10

Centre

Observed Time Procedural Discipline Form Function Discourse Sociolinguistic Immediate Broad

Academic 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

IELTS preparation College A College B College B College B College C College D College D College E College E College E College E College E University A University A University B University B University B University B University C College F College G College G

82 137 92 94 95 122 118 148 114 67 119 114 93 96 55 46 50 27 90 79 52 38

(continued)

351

352
TABLE 6 (Continued) Discipline 0.00 0 0 0 0 0 0 0 0 0 0 0 0 0 0.00 1.00 0 25 0 24 33 47 22 0 48 0 0 0 38 18.23 0.07 0 3 0 16 0 45 74 0 0 0 0 0 0 10.62 0.22 54 37 90 45 6 0 0 47 3 92 89 5 0 36.00 0.08 0 3 0 0 29 0 0 0 0 0 0 0 0 2.46 0.28 0 0 0 0 2 0 0 0 5 0 0 0 0 0.54 0.72 39.18 16.09 13.27 0.18 3.73 Form Function Discourse Sociolinguistic Immediate Broad 49.55 0 29 58 73 99 35 0 0 9 0 89 71 60 40.23 0.41 Academic 0.00 0 0 0 0 0 0 0 23 0 0 0 0 0 1.77 0.19 7.77 8 7 18 3 2 16 7 5 28 7 3 21 10 10.38 0.36

Course Type

Centre

Observed Time Procedural

Average IELTS preparation EAP

87.64

College G College G College A College A University D University D University E University E University E University E University F University F University F

Average EAP Significance (ManWhitney U test)

61 58 123 126 89 52 97 95 93 98 73 127 79 90.08 0.68

WATCHING FOR WASHBACK

353

presessional EAP classes (29% of those observed) remained focused on a single topic for the duration of a class, while IELTS classes tended to switch topic more frequently; only one IELTS class (5%) remained with a single topic throughout. Eleven of the IELTS classes (50% of those observed) included more than five topics, while this was true of just three (21%) of the presessional EAP classes. IELTS was mentioned by participants a total of 129 times during IELTS classes, compared with 10 times during presessional EAP classes (Table 7). Of these 10 mentions, 9 were on combination IELTS/EAP courses (courses in which students were studying EAP but were also following an IELTS preparation course strand). In these cases, teachers mentioned how the class content could be applied to the test, or students asked for information about the test. Specific test strategies or test-taking tips were provided by teachers on a total of 67 occasions, or just over three times per class on average. Just 2 of the 22 IELTS classes observed (both given by the same teacher) included no explicit mention of the IELTS test, although even here, it remained the implicit focus for class activities and did appear in class materials. References to IELTS and mention of test-taking strategies were the only quantitative features recorded on the observation schedule to show significant differences between class types (p = .00012 for references to IELTS and p = .00002 for mention of test-taking strategies). Tests other than IELTS were mentioned just five times during presessional EAP classes on courses unrelated to IELTS. One class accounted for four of these mentions. This included 50 min of explicit preparation for a course exit test to be held 3 days after the observation. The teacher introduced the test format, describing the timing and format of the tasks, and gave 25 min to a practice writing exercise. Results for content control are also displayed in Table 7. In both IELTS and presessional EAP classes the teacher or text most often held control of class content (82% of the time in IELTS and 73% of the time in presessional EAP classes). There was no significant difference between course types (p = .21). Students did not hold sole control of content during any of the classes observed, but control was shared between teacher, text, and students a little more often in presessional EAP (27% of class time) than in IELTS classes (18% of class time). Again, the differences were not significant (p = .07), but the degree of teacher control was somewhat higher and student control lower than that observed by Read and Hayes (2003). Modality was similar across course types (Table 8). Listening (mostly during teacher-centred activities) took up just over half of class time in both types of class (p = .93), while writing activities took up around 10% (p = .81). IELTS teachers were, like those observed by Read and Hayes (2003) and by Hawkey (2006), eclectic in their use of materials, using a variety of books and self-produced materials. Four of seven books used in IELTS classes included a reference to IELTS in their titles. Of the remaining three, two were intended as preparation material for other tests. The four IELTS titles were all course books directed

TABLE 7 A Comparison of Test Focus (Frequency) and Content Control (Minutes) in IELTS- and EAP-Focused Classes Course Type IELTS Preparation Centre College A College B College B College B College C College D College D College E College E College E College E College E University A University A University B University B University B University B University C College F College G College G IELTS 7 7 19 2 2 6 4 3 3 14 5 14 9 5 3 3 0 0 7 5 8 3 5.86 0 3 3 3 0 0 0 0 1 0 0 0 0 0.77 0.00 Other 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.00 0 0 0 0 0 1 0 0 0 0 0 4 0 0.38 0.06 Strategies 6 4 3 3 1 3 3 1 1 6 4 6 7 3 2 3 0 0 7 1 9 0 3.32 0 0 0 0 0 0 0 0 0 0 0 1 0 0.08 0.00 T/Text 45 72 66 89 63 95 105 54 114 66 119 114 75 61 46 19 50 26 81 79 52 38 69.50 35 47 27 59 43 77 85 31 97 95 15 61 65 56.69 0.21 T/S/Text 37 0 3 0 31 26 0 58 0 0 0 0 18 35 6 30 0 0 9 0 0 0 11.50 26 11 7 20 62 31 0 21 0 0 75 37 23 24.08 0.07

Average IELTS Preparation EAP

College G College G College A College A University D University D University E University E University E University E University F University F University F

Average EAP Significance (MannWhitney U test)

354

TABLE 8 A Comparison of Student Modality in IELTS- and EAP-Focused Classes (Minutes)

Course Type 40 46 33 47 39 22 48 91 39 67 72 70 53 66 24 11 37 10 61 38 39 29 12 16 45 19 18 16 9 17 0 0 0 0 10 0 9 15 9 0 10 7 0 0 0 0 14 0 29 30 4 0 60 0 0 0 10 29 16 8 4 17 0 29 14 9 0 23 0 28 0 26 0 40 0 0 0 41 3 0 0 4 0 0 0 0 0 0 12 0 0 0 0 6 9 0 0 0 0 0 0 0 0 0 0 0 19 5 0 0 18 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 24 0 0 9 22 48 0 15 0 46 0 14 1 6 8 0 0 0 0 0 0

Centre

Observed time Listening Speaking Reading Writing S&R S&W L&R R&W

Other 0 0 5 8 0 15 0 0 0 0 0 0 0 0 0 0 0 0 39 0 0 0

IELTS Preparation

College A College B College B College B College C College D College D College E College E College E College E College E University A University A University B University B University B University B University C College F College G College G

82 137 92 94 95 122 118 148 114 67 119 114 93 96 55 46 50 27 90 79 52 38

(continued)

355

356
TABLE 8 (Continued) Listening Speaking Reading Writing S&R S&W L&R R&W Other 44.64 31 46 50 15 48 47 62 52 36 53 56 36 22 42.62 0.93 3 8 12 0 0 8 0 0 6 0 7 17 29 6.92 0.35 26 2 1 1 13 17 4 0 0 7 0 41 8 9.23 0.46 0 0 12 20 62 25 11 0 0 0 20 0 0 11.54 0.81 3 0 0 27 0 17 5 0 6 12 3 0 7 6.15 0.61 0 0 0 0 0 10 0 0 3 0 0 0 0 1.00 0.22 0 0 13 16 0 0 0 0 0 5 7 0 0 3.15 0.16 9.64 12.41 7.50 2.32 0.82 0.14 8.77 0 0 39 0 0 0 7 0 0 22 0 4 7 6.08 0.96 3.05 0 3 0 0 0 0 0 0 0 0 0 0 0 0.23 0.34

Course Type

Centre

Observed time

Average IELTS Preparation EAP

87.64

College G College G College A College A University D University D University E University E University E University E University F University F University F

Average EAP Significance (MannWhitney U test)

61 58 123 126 89 52 97 95 93 98 73 127 79 90.08 0.68

WATCHING FOR WASHBACK

357

toward the test, rather than collections of practice test material. In presessional EAP courses observed, none of the titles included the acronym IELTS, and none was intended primarily as a test preparation text.

ACTIVITIES AND EPISODES: QUALITATIVE OBSERVATIONS RELATING TO TEST DESIGN ISSUES The frequency data were indicative of broad similarities between courses in teaching method. There was little variation associated with the type of course in participant organisation, content control, or student modality. In contrast, the qualitative observations captured in the activities and episodes sections of the schedule highlighted differences in the content that were not well captured in the frequency data. These qualitative observations are presented here in relation to the task description categories developed by Weigle (2002). Subject Matter and Stimulus During work in class, topics were similar in both IELTS and EAP classes. EAP classwork did not generally relate to specific academic disciplines to any greater extent than did IELTS preparation, although, as noted previously, the IELTS classes did involve more frequent changes in topic. Where learners were directed to explore topics further, IELTS teachers suggested reading magazines such as The Economist to learn about topics that might occur in IELTS; EAP teachers set projects relating to students academic subjects. To this extent, the EAP classes did, as predicted, take greater account of subject specialisms. Genre, Rhetorical Task, and Pattern of Exposition IELTS-directed classes, across teachers and institutions, involved activities with a clear and direct relationship to the IELTS AWC. Frequently observed activities included question analysis, brainstorming ideas, forming an outline plan with topic sentences and sentence and paragraph building, all apparently directed toward test-like writing practice exercises. In comparison with IELTS preparation, there was a greater variety of activities on EAP courses, and these appeared to take account of more features of academic writing. Limitations on the selection of topics for Task 2 and the choice of data for presentation in Task 1 may have attracted the test preparation teaching strategies observed, including

providing lists of topics encouraging learners to read about potential IELTS topics in the media

358

GREEN

practice in planning and writing essays (but without incorporating source

material in ways consistent with academic writing) encouraging memorisation of formulaic phrases teaching relevant vocabulary and structures (such asfor AWC Task 1 phrases for describing trends and reporting on information in graphs and diagrams or learning past rather than present forms of new verbs). However, IELTS preparation and EAP courses also had much in common. EAP classes incorporated concern with the discursive essay genre; the rhetoric of description and evaluation; and cause and effect, comparison, and problem and solution patterns of exposition. Cognitive Demands As observed by Brown (1998), Hayes and Read (2004), and Hawkey (2006), IELTS classes were explicitly directed toward test success. In preparation classes, there were frequent mentions of IELTS and of strategies for dealing with the tasks. Most of the materials used were either taken from IELTS textbooks or chosen by the teacher to reflect the test. Frequent essay writing practice involved test practice under timed conditions and completing tasks closely modelled on IELTS (often assigned as homework). Learners were encouraged to become thoroughly familiar with the test tasks. Instruction in EAP courses included stages that did not feature in IELTS preparation. As these courses progressed, learners were often encouraged to go beyond the classroom to locate material and to integrate this into their writing. Learners were never observed to leave an IELTS preparation classroom to carry out other activities, but EAP classes observed included one group visit to a library, and in another, individuals left the classroom to carry out research work elsewhere. Attention was given in EAP classes to issues of plagiarism, the appropriate use of sources, and the compilation of bibliographies. Specification of Audience, Role, Tone, and Style In IELTS classes, primary attention was given to the expectations of examiners, rather than to university lecturers or educated nonspecialist readers: the audiences specified at the time in the task instructions (since January 2006, audiences are no longer specified in the IELTS task rubrics). Audience and the role of the writer were not a major focus of instruction in the observed EAP classes, although one did include a discussion of university teachers expectations of written work. Both IELTS preparation and EAP classes gave time to formal language, but explicit teaching of features of academic style (hedging) was observed only in EAP classes.

WATCHING FOR WASHBACK

359

Performance Conditions: Length and Time Allowed As predicted, reflecting the test format, timed writing activities were more frequent in IELTS preparation classes, but these were also observed in two EAP classes. Some IELTS classes did, contrary to expectations, provide opportunities for redrafting and error correction, though there would be limited opportunities for these under test conditions. When questioned about this in a follow-up interview, one teacher reported that he saw the relevance of these exercises to IELTS in the awareness of essay structure they developed. This may be an instance of teacher beliefs about language learning and skill building outweighing the direct influence of the test format in guiding behaviour. The length of the test tasks directly influenced the requirements set by teachers for written work: Essays assigned in IELTS preparation classes were to be 150 or 250 words in length. Two IELTS teachers observed provided formulaic openings to Task 1, and most included sentence expansion activities as methods of increasing the number of words in a response. Teachers sometimes related such activities directly to the IELTS length requirements. Prompts and Transcription Mode Writing task prompts assigned in IELTS classes employed similar wording to standard test tasks. In addition to producing full-length essays, IELTS preparation students were given practice in writing essay plans based on a range of IELTS prompts. Analyzing prompts for key words was another frequently observed activity. Rating Scales and Criteria The IELTS rating scales were updated in 2005. At the time of this study, these included task fulfilment; coherence and cohesion (Task 1); arguments, ideas, and evidence; communicative quality (Task 2); and vocabulary and sentence structure (both tasks). The new scoring criteria are task achievement (Task 1); task response (Task 2); coherence and cohesion; lexical resource; and grammatical range and accuracy. Bridges and Shaw (2004) provided an overview of their revision. Although the differences were not statistically significant, the frequency data suggested the possibility of a greater focus on form in IELTS classes. Qualitative data suggested that more attention was given to grammar exercises in IELTS classes, compared with a stronger focus on discourse in EAP classes. This might reflect the importance IELTS afforded to vocabulary and sentence structure, the only criterion used to score both Task 1 and Task 2. However, attention to form could also reflect assumptions about learning and the needs of students; teachers might assume that learners could quickly improve their language skills through instruction focused on this area. It is also possible that IELTS learners made more grammatical errors and that teachers were reacting to this with remedial activities.

TABLE 9 Summary of Findings Related to Weigles (2002) Writing Task Characteristics Dimension Subject matter Observations Lists of typical IELTS topics were prepared for students on IELTS preparation courses. IELTS topics were broad and were not explored beyond task demands. Topics were sometimes sustained for longer in EAP courses, and these included some attention to students academic disciplines. IELTS-like task stimuli heavily used in IELTS courses. Focus on topics of general interest. Students were encouraged to read newspapers or serious magazines outside class to learn about relevant topics. This was not the case in EAP courses. IELTS classes limited to writing only Task 1and Task 2type essays. EAP courses involved wider range. Rhetorical tasks limited to those required for the test: predominantly description for Task 1 and evaluation or hortation for Task 2. No integration of academic sources (unlike EAP, where this was a focus). IELTS classes limited to Task 1 and Task 2. EAP involved greater variety, for example, classes on writing definitions IELTS classes involved reproducing information: from graph or from personal knowledge. EAP classes included more writing from source material. Some mention in IELTS classes of university lecturers (the specified audience), but more attention given to examiner expectations, including teaching of ways to impress the examiner. IELTS classes did not focus on the role of the university student. This was an occasional focus in EAP classes. IELTS preparation taught formal tone, but not including features of an academic style such as hedging, which were included in EAP classes. Essays for the IELTS classes were short (100300 words). Writing assignments were generally longer in EAP classes. IELTS students were encouraged to count words. Teaching focus in some IELTS preparation classes on how to use more wordssentence expansion. Timed essay practice activities were more frequent in IELTS classes. However, redrafting of work and error correction were common practice in both EAP and IELTS classes. IELTS preparation involved question analysis based on the generic IELTS task prompts. Students were sometimes given a choice of topic for IELTS practice essays, but this was always limited to IELTS-like tasks. Little word processing observed in IELTS writing (and none in class). Use of IT was observed in EAP classes. Feedback on essays was mostly in the form of band scores in IELTS preparation classes. IELTS preparation involved (a) Teaching of organisational templates for coherence, argument structure (b) Teaching grammar points relevant to testerror analysis/useful structures (c) Encouraging use of more formal and varied vocabulary

Stimulus

Genre Rhetorical task

Pattern of exposition Cognitive demands

Specification Audience

Role Tone, style Length

Time allowed

Prompt wording Choice of prompts Transcription mode Scoring criteria

360

WATCHING FOR WASHBACK

361

Postobservation interviews with teachers did not cast light on this as teachers did not generally seem to be aware of an emphasis on grammar in IELTS classes.

RESULTS: POSTOBSERVATION TEACHER INTERVIEWS Although the analysis of the frequency data revealed few differences between classes in organisation, modality, or content, interviews with teachers indicated that they approached the two class types very differently. Thirteen of the 14 IELTS classes for which interview data were available included mention of the IELTS writing test as an aim of the class. The one IELTS preparation class said not to be influenced by the test came in the first week of a 6-week course. It emerged from the postobservation interview that the central activity (finding supporting examples for popular proverbs from students countries) was intended as a means of preparing students for the demands of Task 2 without directly introducing test-like questions. Thus, although the teacher maintained that the test did not directly influence the class, the demands of Task 2 were the ultimate goal. Class aims in IELTS preparation courses included both practice in performing directly test-derived tasks (Task 1 or Task 2 writing practice) and a wide variety of other activities intended to build test-relevant skills. These aims were categorised as follows:

gaining an overview of test demands building grammar and vocabulary related to test demands analyzing Task 2 questions learning about thesis statements, topic sentences, and paragraph structure supporting propositions with evidence selecting data in response to Task 1 focussing on specific areas of difficulty through self- or peer correction understanding the IELTS assessment criteria.

Some of the aims for presessional EAP classes were similar to those reported for IELTS:

learning how to describe processes learning how to construct paragraphs learning about a problem and solution essay structure reviewing the tense system debating an issue.

362

GREEN

Others seemed to have no parallel among the IELTS classes:

learning how to write definitions learning about hedging in academic writing distinguishing ones own ideas from others learning to integrate source material learning how to construct a bibliography.

Both IELTS and presessional EAP classes were said to follow a similar cycle, with input from the teacher, practice writing tasks, and diagnostic feedback. From the interview data it appeared to be the content of this cycle, not the process, that differentiated presessional EAP from IELTS preparation classes. This reported emphasis was borne out in the content of the activities and episodes observed. As Brown (1998) found in the courses he observed, the IELTS preparation cycle was closely tied to test content and practice of test tasks. Teachers said that the frequency of test practice intensified as the courses progressed, and this was again consistent with the observational data reported previously. In the presessional EAP classes, teachers built toward longer writing tasks, with learners being given greater independence (for research activities and library work) as the courses neared completion. Where courses involved a final test (as with University F), there might be some attention given to this, with students having opportunities for test practice. However, in contrast to the IELTS preparation classes, which focused on the test throughout, the teacher of the test familiarisation class observed at University F reported that this would be the one session (in week seven of the eight-week course) to concentrate on the test. In one combination course the teacher reported that as the test date approached, IELTS preparation leaked across the curriculum, exerting an influence on the content of classes beyond the identified IELTS component, with students requesting practice in test tasks during non- IELTS lessons. This was consistent with the observed mention of IELTS in non-IELTS classes at College A and University E, where learners requested more information about the AWC in nontest classes. Teachers of IELTS courses claimed that writing class content was entirely dictated by the AWC. Conversely, those of EAP courses either dismissed the idea that IELTS had any influence on their classes or suggested that it served as a useful baseline for their teaching; they could assume that learners arriving in their courses with an IELTS score would have some knowledge of how to write a basic five-paragraph essay. Student Work It was plain that the work collected from IELTS and EAP classes differed in the variety of tasks completed by learners. Reflecting the focus reported by teachers and seen in the classes observed, all written work collected from IELTS classes con-

WATCHING FOR WASHBACK

363

sisted of responses to practice IELTS tasks (with varying degrees of guidance from supporting materials). EAP tasks ranged from timed writing exercises on broad topics of general interest based on personal knowledge or experience (similar to IELTS Task 2) to extended projects on topics relating to students academic disciplines and including tables of contents, references, and bibliographies. The task responses also differed in their presentation. All but two of the practice IELTS tasks were handwritten, while the work collected from presessional EAP courses, with the exception of work done in class under time constraints, was all word processed. As anticipated, there was a much greater range in the length of EAP task responses than of IELTS responses. IELTS tasks collected ranged in length from 98 to 445 words (compared with the IELTS requirements of 150 and 250 words), while the presessional EAP essays ranged from 128 to 3,495 words. It will be interesting to see whether the recent introduction of a computer-based IELTS, which offers test takers the option of word processing their responses, will lead to greater use of computers in IELTS preparation courses. Of the six IELTS teachers, five marked student work using IELTS band scores, with one providing a breakdown of the score by the criteria used on the test. The five teachers giving scores often added a comment to the awarded score, such as good 6 or 5.0+. Two of the three EAP teachers provided scores (one as marks out of 20, the other as percentages), while the third made written comments but did not give a score. One of the teachers giving scores used an analytical style of reporting that might have been influenced by IELTS. This used the criteria content and task achievement, organisation and coherence, range and accuracy of language, and (clearly beyond the scope of IELTS) improvement between drafts.

CONCLUSIONS Although the primary concern of this article has been differences between IELTS and EAP classes, it should be emphasised that, as the washback model would predict, there were found to be considerable areas of overlap. Organisation, content control, and student modality were all very similar across classes. Differences in language content were nonsignificant. Excluding the references to the test, many of the activities in the IELTS classes observed might not have been out of place in the EAP classes. Both class types involved brainstorming and planning, with frequent practice in extensive writing. Although there was some variation, both encouraged a formal, objective style of writing; offered instruction in discourse-level organisation; were concerned with the requirements implicit in task instructions; and involved work (often in the form of remediation) on grammar and vocabulary. There was also evidence that teacher variables (such as common beliefs about the value of editing and redrafting) may encourage practices that cannot be predicted from test design. Given the lack of differences between class types in the frequency data, it appears that many of the differences observed between classes

364

GREEN

might be linked rather to teacher or institutional variables, such as levels of professional training and beliefs about effective learning, than to the influence of the test. In the evidence from qualitative observations and from examples of student work, self-correction of essays emerged as another area of variation between teachers, rather than between IELTS and EAP classes. Further research is needed to account for the relative influence of tests, training, resources, and prior beliefs on teaching methods. It seems clear, in this study as elsewhere, that tests exert a less direct influence on this aspect of teaching than on content. Although there was evidence of common practice across course types, and although courses were taught by teachers with varying levels of experience and attitudes toward the test, IELTS preparation classes differed in consistent and salient ways from EAP classes. As predicted by the washback model, differences between classes could be traced to test design features, and test preparation focused narrowly on these features could not be said to offer the same range of skills and knowledge as EAP programs. The focus on the test in IELTS preparation classes directed learners away from their academic subjects and toward the topics and text types featured in the test. Writing in IELTS preparation classes was time constrained and brief, as it is in the test, while EAP learners also worked toward more extensive and less speeded assignments. This study may also point to some of the limitations of watching for washback. Superficially at least, as reflected in the frequency data, the IELTS classes looked very similar to the EAP classes. However, watching classes may not tell us enough about how they are experienced: about which aspects of a class are attended to by the learners, what they learn from them, and whether the attention given to the test is rewarded with improvements in test scores. The teacher interviews showed that even if the content often appeared similar, they approached the two class types in very different ways. This may have been equally true of the learners. There are also limitations in the explanatory power of the observations. Interviews provided some opportunity to probe why teachers believed certain activities might be useful in preparing for IELTS, but why do learners choose to study in IELTS courses? And why do courses and course materials take the shapes seen here? Although its effectiveness has not been demonstrated, among the teachers observed, there was a consistent approach to preparing for the IELTS AWC that centred on building relevant writing skills. The shared nature of the approach and the relationship of this to the test design are indicative of washback. As the skills required by the IELTS AWC are, for the most part, relevant to writing in higher education, there was much in common between IELTS and other forms of EAP instruction. However, the restrictions on the IELTS tasks outlined in the review of the literature were also reflected in the narrower focus of IELTS-directed classes. This suggests that learners will need to pass beyond IELTS preparation if they are to be fully equipped with the language skills they will need for academic study.

WATCHING FOR WASHBACK

365

REFERENCES
Alderson, J. C., & Hamp-Lyons, L. (1996). TOEFL preparation courses: A study of washback. Language Testing, 13, 280297. Alderson, J. C., & Wall, D. (1993). Does washback exist? Applied Linguistics, 14, 115129. Bachman, L. F. (2005). Building and supporting a case for test use. Language Assessment Quarterly, 2, 134. Bailey, K. M. (1996). Working for washback: A review of the washback concept in language testing. Language Testing, 13, 257279. Bailey, K. M. (1999). Washback in language testing. Princeton, NJ: ETS. Banerjee, J. (1996). The design of the classroom observation instruments (Internal Report). Cambridge, UK: University of Cambridge Local Examinations Syndicate. Banerjee, J. (2000). Using English language screening tests in your institution. Paper presented at the BALEAP Professional Issues Meeting, Trent University, Nottingham, UK. Bridges, G., & Shaw, S. (2004). IELTS writing: Revising assessment criteria and scales. Research Notes, 18, 812. Brown, J. D. H. (1998). An investigation into approaches to IELTS preparation, with particular focus on the Academic Writing component of the test. In S. Wood (Ed.), IELTS research reports (Vol. 1, pp. 2037). Sydney: ELICOS/IELTS. Brown, J. D. (2000). University entrance examinations: Strategies for creating positive washback on English language teaching in Japan. Shiken JALT Testing and Evaluation SIG Newsletter, 3(2), 48. Burrows, C. (1998). Searching for washback: An investigation of the impact on teachers of the implementation into the adult migrant English program of the assessment of the certificates in spoken and written English. Unpublished doctoral dissertation, Macquarie University, Sydney. Burrows, C. (2004). Washback in classroom-based assessment: A study of the washback effect in the Australian Adult Migrant English Program. In L. Cheng, Y. Watanabe, & A. Curtis (Eds.), Washback in language testing: Research contexts and methods (pp. 113128). Mahwah, NJ: Lawrence Erlbaum Associates, Inc. Chalhoub-Deville, M., & Turner, C. E. (2000). What to look for in ESL admission tests: Cambridge certificate exams, IELTS, and TOEFL. System, 28, 523539. Chapman, D. W., & Snyder, C. W. (2000). Can high stakes testing improve instruction: Re-examining conventional wisdom. International Journal of Educational Development, 20, 457474. Cheng, L. (2005). Studies in language testing: Vol. 21. Changing language teaching through language testing: A washback study. Cambridge, UK: Cambridge University Press. Coffin, C. (2004). Arguing about how the world is or how the world should be: The role of argument in IELTS tests. Journal of English for Academic Purposes, 3, 229246. Deakin, G. (1997). IELTS in context: Issues in EAP for overseas students. EA Journal, 15, 728. Douglas, D. (2000). Assessing languages for specific purposes. Cambridge, UK: Cambridge University Press. Ferman, I. (2004). The washback of an EFL national oral matriculation test to teaching and learning. In L. Cheng, Y. Watanabe, & A. Curtis (Eds.), Washback in language testing: Research contexts and methods (pp. 191210). Mahwah, NJ: Lawrence Erlbaum Associates, Inc. Green, A. (2003). Test impact and English for academic purposes: A comparative study in backwash between IELTS preparation and university presessional courses. Unpublished doctoral dissertation, University of Surrey, Roehampton, UK. Hale, G., Taylor, C., Bridgeman, B., Carson, J., Kroll, B., & Kantor, R. (1996). A study of writing tasks assigned in academic degree programs. Princeton, NJ: ETS. Hamp-Lyons, L. (2005). The effect of the newly developed TOEFL format on EFL/ESL teaching and learning in four countries. ILTA Online Newsletter, 1. Retrieved November 4, 2005, from http:// www.iltaonline.com

366

GREEN

Hawkey, R. (2006). Studies in language testing: Vol. 24. Impact theory and practice: Studies of the IELTS test and Progetto Lingue 2000. Cambridge, UK: Cambridge University Press. Hayes, B., & Read, J. (2004). IELTS test preparation in New Zealand: Preparing students for the IELTS Academic Module. In L. Cheng, Y. Watanabe, & A. Curtis (Eds.), Washback in language testing: Research contexts and methods (pp. 97112). Mahwah, NJ: Lawrence Erlbaum Associates, Inc. Heyneman, S. P., & Ransom, A. W. (1990). Using examinations and testing to improve educational quality. Educational Policy, 4, 177192. Hughes, A. (1989). Testing for language teachers. Cambridge, UK: Cambridge University Press. Hughes, A. (1993). Backwash and TOEFL 2000. Unpublished manuscript, University of Reading, Reading, UK. International English Language Testing System. (2005). The IELTS handbook. Cambridge, UK: University of Cambridge Local Examinations Syndicate. Kellaghan, T., & Greaney, V. (1992). Using examinations to improve education: A study of fourteen African countries. Washington, DC: The World Bank. Messick, S. (1996). Validity and washback in language testing. Language Testing, 13, 241256. Mickan, P., & Slater, S. (2003). Text analysis and the assessment of academic writing. In R. Tulloh (Ed.), IELTS research reports (Vol. 5, pp. 5988). Canberra: IELTS Australia. Moore, T., & Morton, J. (1999). Authenticity in the IELTS Academic Module Writing Test: A comparative study of Task 2 items and university assignments. In R. Tulloh (Ed.), IELTS research reports (Vol. 2, pp. 64106). Canberra: IELTS Australia. Qi, L. (2004). Has a high-stakes test produced the intended changes? In L. Cheng, Y. Watanabe, & A. Curtis (Eds.), Washback in language testing: Research contexts and methods (pp. 171190). Mahwah, NJ: Lawrence Erlbaum Associates, Inc. Read, J., & Hayes, B. (2003). The impact of the IELTS test on preparation for academic study in New Zealand. In R. Tulloh (Ed.), IELTS research reports (Vol. 5, pp. 153206). Canberra: IELTS Australia. Read, J., & Hirsh, D. (2005). English language levels in tertiary institutions (Export Levy Programme Project E4). Wellington: Education New Zealand. Resnick, L. B., & Resnick, D. P. (1992). Assessing the thinking curriculum: New tools for educational reform. In B. G. Gifford & M. C. OConner (Eds.), Changing assessments: Alternative views of aptitude, achievement and instruction (pp. 3775). Boston: Kluwer Academic. Saville, N. (2000). Investigating the impact of international language examinations. Research Notes, 2, 47. Spada, N., & Frhlich, M. (1995). COLT observation scheme: Coding conventions and applications. Sydney: National Centre for English Language Teaching and Research, Macquarie University. Thorp, D., & Kennedy, C. (2003). What makes a good IELTS answer in academic writing? (IELTS Research Report). London: British Council. Turner, C. (2001). The need for impact studies of L2 performance testing and rating: Identifying areas of potential consequences at all levels of the testing cycle. In C. Elder, A. Brown, A. K. Hill, N. Iwashita, T. Lumley, T. McNamara, & K. OLoughlin (Eds.), Experimenting with uncertainty: Essays in honour of Alan Davies (pp. 138149). Cambridge, UK: Cambridge University Press. Turner, J. (2004). Language as academic purpose. Journal of English for Academic Purposes, 3, 95109. Wall, D. (1996). Introducing new tests into traditional systems: Insights from general education and from innovation theory. Language Testing, 13, 334354. Wall, D. (2005). Studies in language testing: Vol. 22. The impact of high-stakes testing on classroom teaching: A case study using insights from testing and innovation theory. Cambridge, UK: Cambridge University Press. Wall, D., & Hork, T. (2004). TOEFL impact study: The influence of next generation TOEFL on preparation classes in Central and Eastern Europe. Language Testing Update, 36, 4041.

WATCHING FOR WASHBACK

367

Watanabe, Y. (1996). Does grammar translation come from the entrance examination? Preliminary findings from classroom-based research. Language Testing, 13, 318333. Watanabe, Y. (2001). Does the university entrance examination motivate learners? A case study of learner interviews. In A. Murakami (Ed.), Trans-equator exchanges: A collection of academic papers in honour of Professor David Ingram (pp. 100110). Akita, Japan: Akita University. Watanabe, Y. (2004). Methodology in washback studies. In L. Cheng, Y. Watanabe, & A. Curtis (Eds.), Washback in language testing: Research contexts and methods (pp. 1936). Mahwah, NJ: Lawrence Erlbaum Associates, Inc. Weigle, S. C. (2002). Assessing writing. Cambridge, UK: Cambridge University Press.

368