Student Evaluation Questionnaires: Dispelling Misconceptions and Looking at the Literature.

#1 Aline Germain-Rutherford Carleton University, April 2003

1. A Historical perspective
#2 The evaluation of teaching performance is not a new phenomena. As Doyle already reminded us: In Antioch in AD350, any father who was dissatisfied with the instruction given to his son could examine the boy, file a formal complaint to a panel and laymen, and ultimately transfer his son to another teacher if the teacher could be shown to have neglected his duty (Doyle, 1983, quoted by Marsh, 1987, p. 257) #3 But to go back to the 20th Century, students evaluation questionnaires were introduced at Harvard in 1926 (the first questionnaire was developed by students at Harvard), then at the University of Washington, Purdue University and the University of Texas, also in the mid- 1920s. Thousands of studies have been published on students evaluation of teaching effectiveness, with a sharp increase in the 1970s and the 1980s; Marsh (1987) reports 1055 publications on the topic between 1976 and 1984. Remmers, with the first systematic research program started in 1927, is considered as the father of research into students evaluation of teaching effectiveness.(Marsh, 1987 : 257). However many studies conducted and published in the 1970s proved methodologically unsound, even though many of their conclusions served as basis for tenure and promotion policies and subsequent research. This changed in the 1980s when research paradigms and methodological standards evolved. #4 Huguette Bernard (2002) distinguishes 3 periods in the research of student evaluation since the 1970s: 1970-80, the focus was more on the idea of a formative evaluation and studies were looking at indicators of teaching efficiency. 1980-1990, the focus was more on the administrative use of evaluation and studies were examining the validity of student ratings, a well as were looking for multiple ways to conduct a comprehensive evaluation 1990 2000, the focus was/is more on the notion of Teaching Dossier and studies look at the impact and the effectiveness of students evaluation. In the last decade of the 20th Century, [Arreola (2002) notes that] the demands for accountability in Higher Education reached an all-time high. Our Society is questioning the value it is receiving from its investment in Higher Education institutions. In a world dominated by the Internet and ubiquitous media, Universities are not isolated Ivory Towers any more but see-through glass buildings which cant claim anymore to be the primary repository knowledge and information.

#5 Further more, Market forces, responding to societys perception of the failure of traditional higher education, brought forth the rapid emergence and success of private online educational institutions, as well as corporate universities (Arreola, 2002: xv). For Arreola, all these factors explain a growing interest today in the assessment and evaluation of faculty performance. Bernard & Bourque (1997) and Donald & Saroyan (1991) note that most Canadian universities use a student questionnaire to evaluate the effectiveness of teaching, either generic for the whole university or specific for each faculty or department.

2. Students and Facultys perceptions on students evaluation

However, within this context, and examining the research done on the topic, perceptions both by faculty and students are very negative concerning the use of students evaluation. #6 a. In a study on students perception of the effectiveness of students evaluation, Bernard (2002) reports that most students surveyed (out of a 400 sample) thought that: - Evaluation is important to improve teaching and formation of students But, it has - Little impact on formation - Has little influence on faculty practices - Faculty dont read their comments - They are not considered for promotion - Universities are not interested by teaching - Teaching is not very important to Faculty - Evaluation is important but not useful - They call it: Opration bidon b. Before looking at the Facultys perception, lets go back to the purpose of Students Evaluation of Teaching Effectiveness. #7 The most widely declared and accepted purposes of students evaluation of teaching effectiveness are (Centra, 1993; Doyle, 1983; Murray, 1984; Bernard & Bourque, 1997, Marsh, 1987): A measure of teaching effectiveness to be used in administrative decision making (promotion, tenure, renewal of contract) A diagnostic feedback to faculty about the effectiveness of their teaching that will be useful for the improvement of teaching (Formative evaluation) Information for students to use in the selection of courses and instructors. Marsh (1987) noted 2 other purposes : #8

A measure of the quality of the course, to be used in course improvement and curriculum development (however this measure is not very reliable as the research shows that the questionnaires do not give much information on the course itself). An outcome or a process description for research on teaching.

c. What is the perception of faculty on students evaluation? #9 Faculty generally believe that students evaluation are biased by a number of factors which they believe to be unrelated to teaching effectiveness. In a survey conducted by Marsh & Overall (1979), faculty listed specific characteristic which would cause bias to student ratings: 72% listed the course difficulty, 68% grading leniency, 63% instructor popularity 62% student interest in subject before course 60% course workload, 60% class size, 55 % reason for taking the course. Bernard (2002), surveying 1200 faculty found that the facultys perception is that: - All Faculty must be evaluated and All Course must be evaluated But that - Student should not be the only source of information and that Faculty must be allowed to take part. - Evaluation helps to improve teaching - Students written comments are an interesting source of information but also a lieu de dfoulement And that the - Questionnaires are not a good tool for evaluation The most common questions and issues raised by faculty, as reported by Arreola, 2002, Marsh, 1987, Abrami, 1998, Bernard 2002) are: #10 1. Arent student ratings just a popularity contest, based on instructor style rather than on the content of their delivery? 2. Arent student ratings forms just plainly unreliable and invalid? Students evaluation questionnaires provide inconclusive evidence about teaching effectiveness. 3. Arent students too immature, inexperienced and capricious to make any consistent judgments about the instructor and instruction?

4.Dont students have to be away from the course and possibly the college/university for several years before they are able to make correct judgements about the instructor and instruction? 5. Isnt it true that I can buy good student ratings just by giving easy grades? #11 6. Isnt true that students who are required to take a course tend to rate the course more harshly than those taking it as an elective? 7. Do majors in a course rate it differently than non-majors? 8. Isnt it generally easier to get good ratings in upper year courses? 9. Isnt there a gender bias in student ratings? Dont female faculty tend to get lower ratings than male faculty? 10. Isnt it more difficult for Math and Sciences faculty to get good ratings? And a somewhat related question: what is the generality of student ratings across different courses taught by the same instructor? Is there an effect of the specific course being taught on the rating of the instructor? #12 11. Isnt it true that the size of the class affects student ratings? 12. Does the time of the day the course is taught affect student ratings? 13. Does the rank of the instructor (instructor, assistant professor, associate professor or professor) affect student ratings? 14. Does the workload/difficulty level of a course cause a potential bias to student ratings? 15. Isnt it true that the only faculty who are really qualified to teach or evaluate their peers teaching are those who are actively involved in conducting research in their field? 16. What good are student ratings in an effort to improve instruction? 17. Is the Students Evaluation Questionnaire the appropriate tool to measure the instructors impact on student learning?

3. What does the research tell us?

#14 A. Validity and reliability concerns 1. Arent student ratings just a popularity contest, based on instructor style rather than on the content of their delivery?

Teaching is multifaceted, and there is no single criterion of effective teaching. The design of instruments to evaluate teaching effectiveness therefore should reflect this multidimensionality. When such tool are designed and used, research shows very clearly that students do discriminate among dimensions of teaching. Factor analysis of questionnaires presenting several dimensions (Aleamoni and & Spencer , 1973), Aleamoni (1976), Marsh (1987) shows that students ratings are not unduly affected by overlapping of dimensions: i.e. the instructors characteristics (warm, friendly, approachable) had no effect on the lack of organization of the same instructor, or the poor stimulating strategies used in the course. Feldman (1986) studied the relation between student ratings and 14 categories of personality of instructors (defined by personality self-description and students and colleagues personality descriptions of the instructor ). The only positive correlations were positive self-regard, self-esteem (0.30) and enthusiasm and energy(0.30). These correlations were higher when personality was inferred by student ratings and colleague ratings. He concludes that there is a relation between student ratings and some traits of personality of the instructor, but this relation is not viewed as a bias, since the effect of personality is seen also on other related indicators of teaching effectiveness. Instead, Feldman views these positive correlations as supporting the validity of student ratings. The issue therefore is now on the selection of the dimensions to be measured. Feldmans list (1976) of the19 characteristics of the superior university teachers (Photocopie 1), derived from a review of the literature on the students point of view, or Chickering and Gamsons (1987) 7 principles for good practice in undergraduate education could serve as a good basis to define and select dimensions of teaching to be measured in such a questionnaire. The weight of each dimension has also to be determined according to specific criteria, depending on the purpose of the evaluation questionnaire and who will use it. Marsh (1987) describes the process put in place by the University of Sydney when they developed the StudentsEvaluations of Educational Quality (SEEQ) instrument: (Photocopis 2 and 3) #15 A large item pool was obtained from a literature review, instruments in current usage, and interviews with faculty and students about characteristics which they see as constituting effective teaching. Then, students and faculty were asked to rate the importance of items, faculty were asked to judge the potential usefulness of the items as a basis for feedback, an open-ended student comments on pilot instruments were examined to determine if important aspects had been excluded. These criteria, along with psychometric properties, were used to select items and revise subsequent versions. This systematic development constitutes evidence for the content validity of SEEQ and makes it unlikely that it contains any irrelevant factors. (Marsh, 1987: 266) However, Arreola (2002) notes that most universities use home-made questionnaires, developed without rigorous psychometric and statistical procedures. Bernard & Bourque (1997) report that 70% of the 16 universities in Qubec use a questionnaire which has

never been validated. Donald & Saroyan (1991) describe a similar situation across Canada. #16 2. Arent student ratings forms just plain unreliable and invalid? To test the validity of students rating, Marsh reviewed and conducted several studies examining the empirical relations between students evaluations of teaching effectiveness and other indicators that have been posited to reflect effective teaching (Marsh, 1987: 286). Indeed, teaching effectiveness is multifaceted and student learning is not the one single criterion to be measured, eventhough it seems the most obvious. Marsh suggests other criteria should be examined such as the rating of former students, changes in student behaviours, instructor self-evaluations, the evaluation of peers and/or administrators, the frequency of occurrence of specific behaviours observed (Murrays Teaching Behaviours Inventory) by trained observers, and the effects of experimental manipulations. Of these criteria, studies show that student ratings are significantly and consistently related to the ratings of former students and students achievement in multisection courses. A study on the experimental manipulation of clarity of speech of the instructor (Land and Comb, 1981) also supports the validity of students ratings, with results which demonstrate highly significant correlations between specific clarity behaviours and student ratings. #17 Two elements determine the reliability of a questionnaire: the correlations among responses to different items designed to measure the same component of effective teaching, and the agreement among ratings by different students in the same class. Already Remmers, supported by 20 years of research, wrote in 1949: (a)There is warrant for ascribing validity to student ratings not merely as measures of students attitudes toward instructors for which validity and reliability are synonymous but also as measured by what students actually learn of the content of the course (Remmers, et al.,1949, p. 26); (b) undergraduate judgment as a criterion of effective teaching can no longer be waved aside as invalid and irrelevant (Remmers, 1950, p. 4, quoted by Marsh, 1987, p. 258). Remmers conclusions are still supported by studies today. #18 However, eventhough student ratings are good predictors of instructors effectiveness, they are not perfect predictors. It is therefore necessary to be realistic in our expectations of what student ratings can achieve in terms of feedback and information, to interpret carefully the results, but also to supplement this evaluation with additional information: Bernard (2002) suggests: - Study the environment (group of students - room schedule resources - etc.) - Study the students characteristics - Peer, chair and alumni ratings - Examination of course outlines - Teaching portfolio and self-assessment #19 3. Arent students too immature, inexperienced and capricious to make any consistent judgments about the instructor and instruction? And

4. Dont students have to be away from the course and possibly the college, for several years before they are able to make correct judgements about the instructor and instruction? The argument here is that former students who evaluate courses with the added perspective of time and experience on the job will have a different judgement than the students who are just completing the course. However not only all reviewed studies indicate a stability of students ratings from one year to the other, but a cross-sectional study (Marsh & Overall, 1980) involving 100 end-of-class ratings and ratings with the same students several years after graduation show a correlation of 0.83. These studies demonstrate that students ratings are quite stable, and that added perspective does not alter the ratings given at the end of a course. #20 5. Isnt it true that I can buy good student ratings just by giving easy grades , that instructors who inflate grades are rewarded by positive evaluations? Marsh defines a bias as student ratings being influenced by variables that are unrelated to teaching effectiveness, to the extent that this influence generalizes across all rating factors rather than being specific to the particular factors most logically related to the influence (for instance, class size would be considered a bias if it has an effect on dimensions other than Group Interaction and Individual Rapport). In that sense, grading leniency could be considered a bias. However, and even though the literature does show that class-average grades are moderately correlated with classaverage ratings of teaching effectiveness, Marsh (1980 and 1983) examined the relationship among expected grades, prior subject interest and student ratings. Across all rating dimensions, prior subject interest precedes expected grades in terms of effect. Prior subject interest however cant be considered a bias to student ratings since it seems to affect only dimensions most logically related to it, particularly Learning/Value, not the other variables. Higher student interest in the subject creates a more favourable learning environment and facilitates effective teaching. Howard & Maxwell (1980, 1982) are reported to show similar results (Marsh 1987). They conclude that the influence of student motivation upon student performance, grades, and satisfaction [ratings] appear to be a more potent contributor to the covariation between grades and satisfaction than does the direct contaminating effect of grades upon student satisfaction.(p. 818, quoted by Marsh, 1987: 318) In his review of the literature Arreola (2002) reports also a positive, however extremely weak, relationship between grading and student ratings. #21 6. Isnt true that students who are required to take a course tend to rate the course more harshly than those taking it as an elective? And 7. Do majors in a course rate it differently than non-majors? In a study examining 6 different reasons to take a course: a) major requirement, b) major elective, c) general interest, d) general education requirement, e) minor/related field and f) other, Marsh (1980) reports that taking a course for general interest and as a major

elective had a positive effect on all factors, and that taking a course as a major requirement or as a general education requirement had a negative effect on all factors. In fact, general interest had the strongest effect on all variables ( although usually less than 0.20). B. The effect of Specific Background Characteristics : #22 8. Isnt it generally easier to get good ratings in upper year courses? A review of the literature shows that 1st year students tend to rate a course more harshly than 2nd year students, 2nd year students tend to rate a course more harshly than 3rd year students, etc and that graduate level courses tend to receive slightly higher ratings. 9. Isnt there a gender bias in student ratings? Dont female faculty tend to get lower ratings than male faculty? The literature tells us that gender has no effect on student ratings (women and men rate the same way) and that female and male instructors are rated the same way. #23 10. Isnt it more difficult for Math and Sciences faculty to get good ratings? And a somewhat related question: What is the generality of student ratings across different courses taught by the same instructor? Is there an effect of the specific course being taught on the rating of the instructor? Studies in the 1970s and 1980s reviewed by Marsh (Feldman, 1976, Centra and Creech, 1976, over 100 institutions, Neumann & Neumann, 1985, ect), comparing ratings across disciplines (Humanities, Social sciences, Maths, engineering, physical sciences, business administration, etc), or across orientations (hard/soft, applied/pure, living organisms/non life disciplines) show an effect of the discipline on student ratings, although small. Humanities and arts are usually rated higher than social sciences, and social sciences are rated higher than physical sciences and engineering. On the basis of this research, authors argue that student ratings should only be compared within similar disciplines. They suggest also that discipline differences observed in student ratings may reflect the different roles of teaching in these disciplines. As for the importance or not of a course effect on student ratings, Marsh (1981) studied ratings of 1364 courses in 341 sets, where instructors either taught the same course twice, or the same instructor taught 2 different courses, or the same course was taught by 2 different instructors, or 2 different courses taught by different instructors. Marshs results showed that the effect of the teacher on student ratings of teaching effectiveness was much larger than was the effect of the course being taught, and that ratings could be generalized across different offerings of the same course by the same instructor . He concludes that students evaluation primarily reflect the effectiveness of the instructor rather than the influence of the course, and some instructors may be uniquely suited to teaching some specific courses.(Mars, 1987:278) #24 11. Isnt it true that the size of the class affects student ratings?

This belief is one of the oldest and most popular in education. A review of the literature (Marsh, 1987, Feldman 1984, Centra, 1979, and Arreola, 2002) shows that class size is moderately correlated with the most logical related variables (for instance Group Interaction and Individual Report for the SEEQ instrument), but not with other dimensions, nor with the overall ratings of the course instructor. Interestingly, Centra (1979) found in his study of class-size effect that classes in the 35100 ranges received the lowest ratings, whereas larger and smaller classes received higher ratings. When compared to the rating of small class, Arreola (2002) argues also that low ratings for large classes are not due to the class-size effect but to the fact that required and first year courses tend to be offered in large class setting. Since research tells us that required course are usually rated harsher, and that 1rst year students tend to rate also more harshly, this lower rating is expected. 12. Does the time of the day the course is taught affect student ratings? There is too little research on this aspect. The research done doesnt see a time-of-the-day effect in student ratings. #25 13. Does the rank of the instructor (Instructor, assistant professor, associate professor or professor) affect student ratings? Marsh (1987) notes no relation between rank and global ratings, but a slightly positive correlation with the dimension Breadth of coverage and slightly negative correlation with the dimension Group Interaction. 14. Is the Workload/Difficulty effect a potential bias to student ratings? Paradoxically, Marshs own studies (1980, 1982, 1983) as well as his review of the literature on this topic show a positive correlation with student ratings. Marsh studies found that in pairs of courses taught by the same instructor, courses perceived to have higher level of workload/difficulty achieved also higher ratings. #26 15. What good are student ratings in effort to improve instruction? Student ratings prove to be a valid and reliable source for obtaining overall evaluation of teaching effectiveness, even though it is only one among others. Abrami (1998), in a large meta-analysis of teaching dimensions came to the conclusion that global items can be used for administrative summative decisions about teaching. However, studentscomments are more useful for diagnostic feedback for the instructor than class-average ratings. Marsh (1987) reports studies where faculty judged ratings items as superior for personnel decision but judged the written comments as superior for self-improvement. Interestingly Bernard (1994) and Bernard & Bourque (1997) find that only instructors who already have an interest in improving their teaching will use the students comments in a constructive way. According to Bernard & Bourque (1997) and Centra (1993), in order to use student ratings and comments as a valuable mean to improve ones teaching effectiveness, certain conditions have to be in place:

#27 The instructor has to obtain new information on his/her teaching He/she has to value this feedback He/she has to know how to make changes He/She has to be motivated

Examining potential changes in teaching effectiveness due to feedback from student ratings Marsh (1987) and Arreola (2002)s reviews of the literature show that mid-term feedback is modestly effective. As Marsh explains, many instructional characteristics cannot be easily altered in the second half of the course (p. 339). However all the studies show that instructors significantly improved their ratings when feedback was supplemented with consultation. #28 In fact, the key finding that emerges here is that student ratings can be used to improve instruction if used as part of a personal consultation between the faculty member and a faculty development resource person(Arreola, 2002 : 84). #29 16. Is the Student Evaluation Questionnaire the appropriate tool to measure the instructors impact on student learning? In Marshs view, The assumption that effective teaching and student learning are synonymous is unwarranted. A more reasonable assumption that is consistent with the construct validation approach is that student learning is only one indicator of effective teaching, even if a very important indicator (Marsh, 1987 : 285). Large class, multisection courses appear to be an ideal situation to study the relation between students learning and student ratings. Indeed, different groups of students are studying the same material presented by different instructors. Cohen (1981) studied 68 multisection courses. Across all the courses, student achievement was consistently correlated with students ratings of teaching effectiveness. Cohens study shows that sections for which instructors are evaluated more highly by students tend to do better on standardized examinations. This is also confirmed by Abramis analysis (1998). However the authors caution that , considering the difficulty in setting up a methodologically sound, multisection validity study, the relation might not be as direct as it seems. #30 17. Isnt it true that the only faculty who are really qualified to teach or evaluate their peers teaching are those who are actively involved in conducting research in their field? Not only do all studies done on the topic fail to demonstrate the validity of peer ratings based on classroom visitation as an indicator of effective teaching (classroom observation are too punctual, disruptive for the instructor), but Murray (1980), when he compared student ratings and peer ratings, found that peer ratings was 1) less sensitive, reliable and valid, 2) more threatening and disruptive of faculty morale and 3) more affected by non-instructional factors such as research productivity (p. 45) than student ratings.

This put into question the use of peer evaluations for personnel decisions. As for the relationship between research productivity and teaching effectiveness, Marsh (1987) reviewed 13 empirical studies on this aspect and most studies found no significant relationship between effectiveness of teaching and research productivity. The only dimensions which appeared slightly positively correlated with research productivity was student ratings of : Breadth and coverage and Instructors knowledge (0.27). Teaching effectiveness and research productivity, even though not or slightly correlated, are a function of both ability and time allocation, as seen in the figure 4.1 (Marsh, 1987, p. 301) Photocopie 4

Conclusion
#31 A review of the literature indicates that students evaluation of teaching effectiveness questionnaires, if well designed, properly administrated and its results carefully interpreted , are valid and reliable tools, and can provide valuable formative information to the instructor for teaching enhancement purposes and summative information to the administration for personnel decision. Furthermore, Marsh (1987), Bernard & Bourque (1997) and Arreola (2002) show the strong tie between students evaluation of teaching effectiveness and faculty development. As explained by the authors, hand-in-hand with any evaluation system should also exist resources and opportunities to enhance or improve teaching effectiveness. A teaching evaluation system, without reference to faculty development programs is usually viewed by faculty as being primarily punitive in intent. On the other hand, faculty development programs, implemented without clear reference to information generated by faculty evaluation systems tend to attract only those already motivated and not those in greater needs. #32 Bernard & Bourque (1997) therefore propose an integrated strategy where faculty development, teaching evaluation and teaching improvement system, and valorisation of teaching are all interconnected and interdependent. (Figure 1., Bernard, 1997, p. 55) (Photocopie 5)

References
Abrami, P.C. (1998). Improving judgements about teaching effectiveness using teacher rating forms. Centre for the study of learning and performance. Working Paper. Concordia University. Aleamoni, L. M. (1976). On the Invalidity of Student Ratings for Administrative Personnel Decisions. Journal of Higher Education, 47 (5), 607-610.

Aleamoni, L. M. & Spencer, R. E. (1973). The Illinois course evaluation questionnaire: A description of its development and a report of its results. Educational and Psychological Measurement, 33, 669-684. Arreola, R. A. ( 2002). Developing a Comprehensive Faculty Evaluation System. Anker Publishing Company, Inc. Bolton, MA Bernard. H. (1994). La pertinence de lvaluation faite par les tudiants et les stratgies damlioration de lenseignement suprieur. Communication prsente lors du Sminaire international itinrant de lAIPU en Belgique. Bernard, H. (2002). Using Evaluation to Improve Teaching. Address delivered at the University of Ottawa, November 2002. Bernard. H., & Bourque, S. (1997). Portrait des politiques et des pratiques dvaluation, damlioration et de valorisation de lenseignement des universits qubcoises. Res Academica, Vol. 15 (1 et 2), 33-60 Centra, J. A. (1979). Determining faculty effectiveness. San Francisco: Jossey-Bass Centra, J. A. (1993). Reflective Faculty Evaluation. Enhancing Teaching and Determining Faculty Effectiveness. San Francisco: Jossey-Bass Centra, J. A. & Creech, F. R. (1976). The relationship between student, teacher, and course characteristics and student ratings of teacher effectiveness (Project Report 76-1). Princeton, NJ: Educational Testing Service. Chickering, W.A. & Gamson, Z.F.(1987). Seven Principles for Good Practice in Undergraduate Education. The Wingspread Journal. AAHE Bulletin, March. Cohen, P. A. (1981). Student Ratings of Instruction and Student Achievement: A MetaAnalysis of Multisection Validity Studies. Review of Educational Research, 51, 281-309 Donald, J., & Saroyan, A. (1991). Assessing the Quality of Teaching in Canadian University Education: The Quality and Evaluation of Teaching. Revista Iglu, 1, 157-173 Doyle, K. O. (1983). Evaluating teaching. Lexington, MA: Lexington Books. Feldman, K. A. (1976). Grades and College Students Evaluations of their Courses and Teachers. Research in Higher Education, 4, 60-111 Feldman, K. A. (1984). Class Size and Students Evaluations of College Teacher and courses: A Closer Look. Research in Higher Education, 21, 45-116

Feldman, K. A. (1986). The Perceived Instructional Effectiveness of College Teachers as Related to their Personality and Attitudinal Characteristics: A Review and Synthesis. Research in Higher Education, 24, 139-213 Howard, G. S & Maxwell, S. E. (1980). The Correlation between Student Satisfaction and Grades: A Case of Mistaken Causation? Journal of Educational Psychology, 72, 810820 Howard, G. S & Maxwell, S. E. (1982). Do Grades Contaminate Student Evaluations of Instruction. Research in Higher Education, 16, 175-188 Land M. L. & Combs, A. (1981). Teacher Clarity, Student Instructional Ratings, and Student Performance. Paper presented at the annual meeting of the American Educational Research Association, Los Angeles. Marsh, H. W. (1980). The Influence of Student, Course and Instructor Characteristics on Evaluations of University Teaching. American Educational Research Journal, 17-219237 Marsh, H. W. (1981). The Use of Path Analysis to Estimate Teacher and Course Effect in Student Ratings Of Instructional Effectiveness. Applied Psychological Measurement, 6, 47-60 Marsh, H. W. (1983). Marsh, H. W. (1987). StudentsEvaluations of University Teaching: research Findings, Methodological Issues, and Directions for Future Research. International Journal of Educational Research, 11 (3), 253-388 Marsh & Overall (1979) (Marsh & Overall, 1980) Murray (1980), Murray, 1984 Neumann & Neumann, 1985 Remmers, et al.,1949, Remmers, 1950