Critical Review Assignment EDUC 8510: Education Research Methods

John Jordan, November 16, 2011 Prof. Jean Turner, FA 2011

Critical Review: Chang, A. C., and Read, J. (2006). The effects of listening support on the listening performance of EFL learners. TESOL Quarterly, 40(2), 375-397. From 2009 to 2010 I taught Academic Listening at the Asian University for Women in Chittagong, Bangladesh. The courses aim was to bolster the students listening skills to prepare them for university work when they entered the undergraduate program. It was my first listening skills course, and I struggled building the students confidence and ability to listen to materials and demonstrate their comprehension. Any improvement in my ability to teach listening I gleaned from doing the course was largely through a process of trial and error. I chose this article as it explicitly addressed effective ways to improve listening performance and studied how different listening support affected students comprehension of listening texts in a similar context as the one I had taught in. In fact, one of the studys authors, Anna Ching-Shyang Chang, teaches listening at a university in Taiwan. Article Summary This study investigated how different listening support strategies affected the listening comprehension of university students. The authors tried four different listening support strategies: previewing the test questions, repetition of the listening input, providing background knowledge of the topic of the listening material, and vocabulary instruction. In addition, the authors studied how the learners level of listening proficiency interacted with the four types of listening support. The authors used 160 Taiwanese students, already divided into 4 classes. A pretest (the listening section of the TOEIC exam) determined that the four groups had no significant difference in listening proficiency. Based on the pretest, the 160 students were put in two language-level groups, with the high proficiency group scoring above the median pretest score, and the low proficiency group below it. Before listening, each group was allowed to preview the test questions. Group 1 had no other treatment. The other three groups each received

Critical Review Assignment

one of the other listening support strategies. The groups listened to two listening texts and took listening tests designed by the teacher consisting of 30 multiple-choice questions (15 for each text) that covered general ideas and details of the two texts. After the exam, some students were selected for interviews to add qualitative data about their impressions of the treatments and the tests. The researchers did a 4x2 ANOVA (four treatments, two listening-proficiency levels) on the data and found a statistically significant effect for both type of listening support and listening proficiency level. T-tests were done on each group to measure the effects of each listening support. In particular, topic previewing proved to be the best form of support for both groups. Vocabulary instruction did not appear to help; in fact, the higher proficiency group did significantly worse after receiving vocabulary instruction. The listening proficiency of the students also mattered. For example, higher-level students benefited more from repeated input and previewing questions than their lower-proficiency counterparts, but topic preview helped regardless of level. Critique Abstract The articles abstract is generally concise and comprehensive. At 173 words, it includes nearly all the necessary elements that Turner (2011) calls for. The first two sentences include the topic, purpose, and rationale for the study. The fourth sentence tells about the participants; the fifth, sixth, and seventh sentences summarize the results. The last sentence mentions implications and leads for further investigation. The methods, procedures, and data analysis of this study, however, are not explicitly mentioned in the abstract. Introduction

Critical Review Assignment

The article has a comprehensive introduction that expands the topic and rationale for this study. The researchers explain the difficulty foreign language learners can have with listening, how test anxiety factors to listening comprehension evaluation, and how prelistening activities can aid learners. The authors explain that while it is recognized that prelistening activities are important, little research has been done to show how effective different types are. The authors summarize the relevant literature on two of the most-studied listening support strategies, previewing questions and repeated input. This discussion leads in to the studys purpose, explained in prose and then given explicitly as two research questions, one about the different effects of the four strategies, and one about the effect of learner listening proficiency to the four types. Methods The article divides the Methods section into four sections: Participants, Instruments and Materials, Procedure, and Data Analysis. In general, the authors are very explicit and systematic in their research. When they cannot be, they say so. For example, the participants were not randomly assigned into groups, although analysis was done and shown in the study to show that the groups were comparable. The authors say, [I]t should be noted that, according to the requirements of strict experimental design, the fact that the students were not randomly assigned to groups leaves open the possibility that extraneous variables may have influenced results (p. 381). The materials for the study (the listening texts, the listening tests, the background information given the topic preparation group, and the vocabulary lists) are all explained quantitatively and qualitatively to the degree that the materials are reproducible. The procedures, too, are well-explained and easily reproducible. The authors present a table that clearly explains how the treatments were administered (Appendix A). There is, however, a potential confounding

Critical Review Assignment

element of their design: All groups were allowed to preview test questions. Groups with other treatment still previewed questions. Although the authors present their four treatments as separate, in fact they are testing three other treatments against a control group that only previews questions. The procedures used make it difficult to tease out the effects of previewing questions from the other three forms of treatment. Thus, the research questions are not quite accurate, and could be better stated as something like, Given that students preview test questions, what is the effect of other listening support strategies to listening comprehension of EFL students? The authors rationalize their choice by pointing out that the higher proficiency group actually scored significantly lower after getting vocabulary instruction compared to the group that only previewed test questions, and report, [C]ontrary to expectation, PQ was not simply a control condition but a form of listening support in its own right (p. 389). Previewing questions may indeed be a form of listening support in its own right, but it was one that was given to all students, and therefore the authors cannot justify their research as treating previewing as a separate treatment, because there were no groups without this treatment with which to compare. Analysis The authors state that SPSS was used to analyze their data. They performed a 4x2 ANOVA. They do not, however, state that their data met the assumptions for using ANOVA, and it is not clear if the test scores would yield normally distributed data in a population. Results Much of the data analysis was discussed under the Results section. Again, the authors make no mention of meeting the assumptions of the statistics used, nor do they set alpha. After finding a statistically significant interaction among listening support and listening proficiency, the authors apparently do t-tests to check the observed effect. They do not explicitly state this,

Critical Review Assignment

but it could be inferred from information presented as such: The most noticeable difference was found in the PQ subgroup (t = 3.53; p = .001), followed by the RI group ( t = 2.84; p = .007) (p. 387).1 Again, if they did use t-tests, there is no indication that their data was normally distributed. All this means that the analysis is potentially invalid. In addition, the authors apparently use eta-squared (again, this is not explicitly stated, but can be inferred by the use of 2 in the text and within tables presented in the article) to measure the observed difference between different groups, while omega-squared might have been a more appropriate statistic.2 Finally, no formal research hypotheses are given. These flaws damage what are actually quite compelling and clearly presented data. The chart on page 387 (Appendix B), for example, graphically displays their results in an easy to comprehend manner, particularly the effects listening proficiency had on previewing questions and repeated input, and the overall effect topic previewing had. Discussion/Conclusions The Discussion and Conclusions sections are separate, but work together well in this paper. The authors combine their data with qualitative information they gathered from post-test interviews and insight from their literature review and experience as teachers to make sense of their information. On page 393 is a clear list of conclusions drawn from this study. The authors list limitations of their research and put the rather limited scope of the study within the larger frameworks of both listening and teaching listening in general. Overall, it clearly summarizes the findings and how they relate to FL listening teaching. One flaw, however, is that the authors

I acknowledge Prof. Turner for helping me understand parts of the data analysis section, especially those parts not covered in class. 2 Again, thanks to Prof. Turner for pointing me to this information, namely, The omega squared for a particular study will yield an estimate smaller than eta squared or R-squared, which tend to overestimate the strength of association (p. 159), from: Vogt, W. P. (1993). Dictionary of Statistics and Methodology. Newbury Park, CA: Sage Publications.

Critical Review Assignment

bring in new literature not covered in their review in this section. Overall, though, this section does what the APA Publication Manual (2010) asks of it. Conclusion Overall, I found this study to be comprehensive, systematic, and useful. It is well organized in a format that closely follows APA (2010) format and Turners (2011) elements of good research. The literature reviewed set up well-thought-up research questions. The methods are explained in a way as to make the study quite easy to duplicate. Rationale behind the choices made by the researchers was given and explained. Results were clearly shown and related to existing literature and application to language teaching. However, since the authors have not demonstrated how their data met the assumptions of the analytical statistics they used, a reader cannot determine if an appropriate and reliable analysis has been done. This fact has the potential to make the entire output moot. On the other hand, the research provides valuable insight to foreign language educators teaching listening skills. The authors have shown that some strategiesnamely, providing background information before listening to a text and repeating inputcan be very beneficial in increasing listening comprehension; they have also shown how even a minimal difference in listening proficiency can affect what a teacher should prepare students for a listening task. The authors have also demonstrated that vocabulary instruction has little to no benefit as listening support, and may even be detrimental. This is valuable information for a listening teacher, and a main reason research is done. However, the lack of transparency about the data analysis done provides a threat the the studys meaningfulness.

Critical Review Assignment

References American Psychological Association. (2010). Publication manual of the American Psychological Association (6th ed.). Washington, DC: American Psychological Association. Turner, J. (2011). Using non-parametric and parametric statistics in language education research. Manuscript in preparation.

Critical Review Assignment Appendix A: Table of four treatments

From: Chang, A. C., and Read, J. (2006). The effects of listening support on the listening performance of EFL learners. TESOL Quarterly, 40(2). p. 384.

Critical Review Assignment Appendix B: Scores by treatment and listening proficiency

From: Chang, A. C., and Read, J. (2006). The effects of listening support on the listening performance of EFL learners. TESOL Quarterly, 40(2). p. 387.