Norana Cantrell ETAP 680 Research Proposal

Online Class Grades versus Traditional Class Grades:

A Research Proposal Examining the
Effect Online Classes has on Grading

INTRODUCTION AND STATEMENT OF PROBLEM

It is common for research reports to use instructor assigned grades as a measure of student

success when studying online courses and traditional face-to-face courses. The purpose of this

study is to determine if educators have different expectations of online students and if they grade

student work different in an online course as opposed to a traditional face-to-face course.

Instructor-assigned grades from online courses will be compared with instructor-assigned grades

for their classroom-based counterparts. Additionally, other instructors will grade the student work

for both classes to see if there are any significant differences without knowing which students are

from which classes. This work attempts to investigate whether instructors are more or less lenient

when it comes to grading online course work as a result of instructor’s perceptions of online

students, online courses, or for further reasons.

LITERATURE REVIEW

It is frequently questioned whether online learning courses can provide a quality of learning

that is consistent with traditional classes. These two types of courses are regularly compared and

the measure for the quality of learning that tends to be used are the grades that the students received

on assignments during the course and/or final grades at the end of the course.

In 2004, Bernard et al compiled a meta-analysis of all the research completed that

compared distance education with classroom instruction. They examined a total of 232 studies

and found that, overall, there was no difference in the level of success between the two types of

1
Norana Cantrell ETAP 680 Research Proposal

courses. Among the various measures of success were achievement, retention, and attitude.

However, levels of student achievement was generally measured by instructor-assigned grades.

Also, the question of consistent course quality from instructor to instructor was a factor that was

not examined. Does an “A” in an online course taught by one instructor mean the same as an “A”

in the same online course taught by a different instructor?

This concern of inconsistent quality between online learning courses has been approached

previously. One of the methods presented to rectify this has been the implementation of standard

grading and assessment practices (Trotter, 2008; Wyss, Freedman, Siebert, 2014). One article uses

voluntary teaching standards for online courses as a response to concerns of inconsistent quality

(Trotter, 2008). Trotter discusses using the National Teaching Standards for Online Learning as a

way to provide quality assurance and consistency with online courses.

Wyss, Freedman, & Siebert (2014) uses a discussion rubric to evaluate student

contributions to required course discussions. While this may allow for more consistent grading

practices, what is examined in this study is the effect that the rubric has on the quality of student

submissions as determined by instructor-assigned grades. It was found that students tend to score

higher when give a discussion rubric to self-evaluate their work prior to submission. However, is

it that students submit higher quality work or that the instructor is more lenient with the grades

assigned?

Attempts to implement standardized grading practices have been made to examine their

effects on consistent quality of work and assessment. Harris (2012) examines the varying levels

on teacher expectations and academic standards. This study discovers how instructors harbor

inconsistent grading practices despite standards-based reform having been implemented in their

schools. One of the teachers interviewed explained the reason for varying standards by saying that

2
Norana Cantrell ETAP 680 Research Proposal

“our kids have a lot of needs. There is a low foundation knowledge as well as low reading ability.

It is difficult to reach all the standards” (Harris, 2012, pg 136). Even though standard practices for

grading and evaluating have been implemented in their school, instructors still assess their students

differently.

Multiple studies have been done comparing the effectiveness of online and distance

learning with classroom learning. Generally, they use instructor-assigned grades as the measure

of student success and learning. This is seen in the vast majority of the 232 studies in the Bernard

et al (2004) meta-analysis article comparing distance education with classroom instruction. Even

more recently, Lopez Soblecher, Gonzalez Gaya, and Hernandez Ramirez (2014) compared online

education with classroom learning for a vocational training program. Even in this study, where it

would be a fairly simple process to evaluate student learning with a skill based assessment, teacher

assigned grades are used as the measure of student success. Since it has been shown that instructors

have inconsistent grading practices (Harris, 2012), is it realistic to use instructor-assigned grades

as the measure of student success and learning?

One aspect that is of interest is perceptions of the instructors towards students who enroll

in these online learning classes. Wickersham & McElhany (2010) found some particularly

interesting results. One of the findings was that “students enroll in online courses perceiving them

as being easier than traditional classroom courses” (pg 10). Could this kind of perception affect

the instructor-assigned grades administered in a distance course?

Another factor that has been considered is the level of instructor bias against online

learning, and the underlying causes and reasons for it. Several studies examined these perceptions

(Barker, 2007; Broady-Ortmann, 2002; Lee & Busch, 2005; Major, 2010; Wickersham &

McElhany, 2010). Generally, it was found that online learning courses were seen as being more

3
Norana Cantrell ETAP 680 Research Proposal

time-consuming than classroom education, as well as also finding existing concerns about the

quality of learning, as discussed previously.

Broady-Ortmann (2002) found that “teachers were more concerned with problems and

issues regarding practical day-to-day operations than with the overall objectives of the professional

organizations” (pg 113). It was discussed that while quality was less of a concern for teachers, the

amount of time spent on non-teaching related activities was an issue which may have affected

instructor’s willingness to participate in online courses. One very extensive article (Major, 2010)

found that increased time demands, increased structure, and additional instructor responsibilities

were considerable factors in an instructors’ perceptions on an online learning class.

Lee & Busch (2005) and Tabata & Johnsrun (2008) both found similar results - that two

important concerns in distance education courses is that instructors viewed online learning courses

as being more time consuming and of inferior quality as their face-to-face counterparts. Among

the time-consuming issues were technology use and lack of technology skills, as well as attitudes

towards technology in general. These were seen as factors that make distance education more time

consuming. In a similar vein, Wickersham & McElhany (2010) found instructors believed that

online courses were more time consuming for the students with matters not directly relevant to

education. “[Students] learning is effected as more time is spent on ‘fixing’ the technological

problems as opposed to learning with technology” (pg 7).

If teachers already have inconsistent grading practices, could it be possible or likely that

negative views on distance education and its increased amount of time consumed for the online

class, that instructors have a grading bias either for or against online students when compared to

their classroom counterparts?

4
Norana Cantrell ETAP 680 Research Proposal

METHODOLOGY

Design

This research proposal is for a nonexperimental, comparative design. The proposed study

will be comparing the grades assigned by instructors who are aware of which students are in an

online class and which students are enrolled in a traditional face-to-face class with those are not

aware. This will provide an opportunity to examine the relationship between the knowledge that

a student is enrolled in a certain type of class, with any grading effect that might exist.

The sample group of instructors will teach two sections of the same course – one that is

fully online only and one that is classroom-based. During these courses, they will administer six

(6) assignments standardized based on length. Four (4) assignments will be two (2) pages in

length, one midterm assignment to be five (5) pages in length, and one final assignment to be

five (5) pages in length.

Each instructor will have two graders assigned to them. The instructor is responsible for

all aspects of the course including assigning grades to the six (6) standard assignments. The

graders will be responsible only for providing additional grades for the six (6) standard

assignments. The graders will be drawn from the same sample pool as the instructors and

assigned randomly, accounting for subject matter specialty.

This will be a double-blind experiment so that the instructors will not know who their

graders are, and the graders will not know who their instructors are. In addition, the graders will

5
Norana Cantrell ETAP 680 Research Proposal

not know the names of the students whose work they are grading. They will be defined only by

randomly generated eight (8) digit identification numbers.

The courses examined will be administered during the Spring semester. However, only

the instructor will examine the assignments and administer grades during that time. Due to the

amount of time involved in grading all of the assignments as graders, this portion will be

completed during the Summer semester when the majority of the instructors and graders

involved in the research will not have teaching obligations to fulfill. In addition, all graders

involved will received a stipend for their time involved in grading the assignments.

For instance, Prof. Smith will teach two (2) sections of Introduction to Philosophy 101,

one that is fully online and one that is face-to-face. After he has graded the standard

assignments, they will be passed on to Professors Johnson and Professors Brown through an

intermediary. Both Professors Johnson and Professors Brown teach Philosophy as well.

Professor Johnson is teaching two (2) sections of Philosophy 202: Plato – one section that is fully

online and one section that is face-to-face. There are two other professors that will grade

Professor Johnson’s standard assignments, and so on and so forth. In this way, all courses will

be assigned one instructor and two graders.

Sample

The sample group, from which all instructors and graders will be drawn, will be chosen

using cluster sampling by selecting naturally occurring groups. The clusters will be based on a

combination of several criteria. The first criteria is that they will all be community college

instructors at one of two large community colleges in New York City. Second criteria for the

6
Norana Cantrell ETAP 680 Research Proposal

sample will be that all of the instructors in this study must be currently teaching two sections of

the same course – one fully online course and one classroom-based course. The estimated

number of instructors in this sample is 65 based on the course listings and teaching assignments

of previous semesters. Instructors are permitted by their respective community colleges to teach

online courses once they have completed a six-hour online teaching training seminar.

Instrumentation

Since grades will be the sole form of measurement, all instructors and graders will receive

a grading rubric to follow when evaluating the student work. The rubric implemented will be a

modified version of the discussion rubric using in the Wyss, Freedman, and Siebert (2014) study.

A copy of the rubric can be found on the final page of this document. Each instructor and grader

will received one copy of the rubric at the beginning of the course, as well as one copy to

accompany each of every student’s individual assignment. The intention is that they will use a

copy rubric as a worksheet to calculate each grade for each paper for each student. Such that,

when grading the midterm paper for a class of twenty-five (25) students, both the instructor and

grader will receive 25 copies of the rubric, one for each paper that they have to grade. In addition,

each student will receive a copy of the rubric when they receive each assignment. Grades

examined will not include any penalty for late assignments submitted.

The data that will be collected is in the form of grades for assignments and final papers.

Each of the six (6) assignments will be graded on a range between 0 and 100 points. The

assignments will then be averaged with the Midterm and Final assignments each being worth

30% for a total of 60%, and then the four (4) small assignments being worth a total of 40%. These

7
Norana Cantrell ETAP 680 Research Proposal

will be collected from questionnaires given to each of the Instructors and the graders with unique

identifiers for each student and spaces for grades for each of their assignments.

In this study, grades assigned to online and face-to-face students by instructors and

graders will be compared to look for significant differences. Grades for individual assignments,

across rubric categories, as well as overall combined total grades will be examined and compared.

Researchers will examine the differences in grades between classes based on the format of the

course.

Several statistical measures will be implemented to determine the existence and size of

the grading effect. Specifically, measures including averages and standard distributions on all

four test groups will be examined. A correlation coefficients 2x2 matrix between the grades

assigned by the instructors and grades assigned by the graders will be examined. This correlation

coefficient is expected to be very close to 1.00 if there were no difference between the grades

for the two sections. So this measurement will show the bulk of the effect that is being examined.

Analysis of covariance will be explored when comparing the face-to-face grades with the online

grades. These statistical analyses will provide an excellent picture of the size of effect that exists

in the difference of grading between online courses and traditional face-to-face courses.

LIMITATIONS

The design of this study has a few limitations that are present, as well as a few aspects

that might present as limitations but, in the end, will not be likely to affect the results.

It may be possible for graders to figure out which instructor they are assigned to as a

primary, or vice versa. This issue may come up particularly in small departments of one of the

8
Norana Cantrell ETAP 680 Research Proposal

community colleges where, using the example referenced in the “Design” section of this research

proposal, there is only one instructor who teachers Philosophy 202: Plato. However, the sample

consists of Philosophy professors from two separate community colleges. Thus reducing this

likelihood.

It is possible that some of the graders will be more lenient or stricter with their grading

practices since these grades will not be given out to the students. At first glance, this is presented

as a concern. But once this is examined more closely, it becomes obvious that this will have a

limited, if any, effect on the results as long as the graders are consistent with their own grading

standards.

What is of concern in this study is the difference between grades among online students

versus traditional students. However, similarly, as long as each grader is consistent across their

own standards for each class, this will not present itself as a limitation.

A further limitation that could be a concern is the grading rubric that is being

implemented. It is a very broad and generic. Due to the design of this study, it becomes a

necessity as the courses, instructors, and graders will be using this to evaluate a range of six (6)

different assignments covering a large range of subject matter and topics. This rubric will be used

to evaluate short and medium length papers in lower level and more advanced classes in

Anthropology, Economics, Educations, History, Philosophy, Psychology, and Sociology, among

others. However, this necessity means that the rubric, while providing some guide for instructors

and graders, is still largely open to interpretation. Despite this, as with the above mentioned

limitation, individual grader consistency will reduce this limitation.

9
Norana Cantrell ETAP 680 Research Proposal

In addition, due to the narrow focus of the sample size, generalizability to the greater

population may be an issue. All of the faculty included in this study are community college

professors in the New York City area. The narrow focus of these attributes could limit how

applicable the results are to the larger population of all e-learning professors.

Overall, there are several limitations that could affect the weight and importance of the

results found. However, the researchers are confident that this study will provide interesting and

important results regardless of these few limitations.

Overall, this research proposal could have a significant impact on distance education and

its grading practices. Depending on the size and direction of the effect found, replication studies

would allow to increase the generalizability to the larger population as well further research that

would be needed to determine possible causes of such an effect. The data provided by this

research proposal, as well as any following related research, could have a significant impact in

instructor grading practices and policies for online learning classes, as well as the trainings

provided by instructors who intend to teach such courses in an online format.

10
 Norana Cantrell ETAP 680 Research Proposal

Assignment Grading Rubric

Student Name/ID#: ______________________

Category Marginal (0-9 pts.) Average (10-19 pts.) Excellent (20-25 pts.) Points earned
Research Research is minimal or none Research is substantive Research is substantive and
at all used in a professional manner
Scholarship No attempt at clarifying Rephrasing and summaries of Indicative that the student can
course content is made course content as interpreted clarify course content for
by others themselves
Citations No references or citations References and citations are References and citations are
regularly used but with some always used correctly and
errors appropriately with no errors
Personal- No reference to individual Course content and individual Connections are made
Practical experience is made, or no experience is referred to, but between course content and
Knowledge reference to course content no connections are made individual experience
is made
Supportive and No criticism or feedback is Only surface criticisms and/or In depth support and critical
Critical given of the assigned support of assigned readings analysis of assigned readings is
Feedback readings is provided provided

Total:

11
Norana Cantrell ETAP 680 Final Paper

References

Barker, K. (2007). E-learning Quality Standards for Consumer Protection and Consumer

Confidence: A Canadian Case Study in E-learning Quality Assurance. Journal of Educational

Technology & Society, 10(2), 109-119.

Bernard, R.M, et al. (2004). How does distance education compare with classroom instruction?

A Meta-Analysis of the empirical literature. Review of Education Research, 74 (3), pp 379-439.

Broady-Ortmann, C. (2002). Teachers' Perceptions of a Professional Development Distance

Learning Course: A Qualitative Case Study. Journal of Research on Technology in Education

(International Society for Technology in Education), 35(1), 107.

Harris, D. M. (2012). Varying Teacher Expectations and Standards: Curriculum Differentiation

in the Age of Standards-Based Reform. Education & Urban Society, 44(2), 128-150. doi:

10.1177/0013124511431568

Lee, J., & Busch, P. (2005). Factors Related to Instructors' Willingness to Participate in Distance

Education. Journal of Educational Research, 99(2), 109-115.

López Soblechero, M., González Gaya, C., & Hernández Ramírez, J. (2014). A Comparative

Study of Classroom and Online Distance Modes of Official Vocational Education and Training.

Plos ONE, 9(5), 1-9. doi:10.1371/journal.pone.0096052

Major, C. (2010). Do Virtual Professors Dream of Electric Students? University Faculty

Experiences with Online Distance Education. Teachers College Record, 112(8), 2154-2208.

12
Norana Cantrell ETAP 680 Final Paper

Tabata, L., & Johnsrud, L. (2008). The Impact of Faculty Attitudes Toward Technology,

Distance Education, and Innovation. Research in Higher Education, 49(7), 625-646. doi:

10.1007/s11162-008-9094-7

Trotter, A. (2008). Voluntary Online-Teaching Standards Come Amid Concerns Over Quality.

Education Week, 27(26), 1-15.

Wickersham, L. E., & McElhany, J. A. (2010). Bridging the Divide: Reconciling Administrator

and Faculty Concerns Regarding Online Education. Quarterly Review of Distance Education,

11(1), 1-12.

Wyss, V., Freedman, D., & Siebert, C. (2014). The Development of a Discussion Rubric for

Online Courses: Standardizing Expectations of Graduate Students in Online Scholarly

Discussions. Techtrends: Linking Research & Practice to Improve Learning, 58(2), 99-107. doi:

10.1007/s11528-014-0741-x

13