1

Using the International Society for Technology in Education (ISTE) Classroom Observation

Tool (ICOT) to observe an online doctoral class:

Brennan-Jones, C.

Nova Southeastern University Fischler School of Education and Human Services

cyntbren@nova.edu

8015 105th Court

Vero Beach, FL 32967

Miller, M.

Nova Southeastern University Fischler School of Education and Human Services

mhobbs@nova.edu

1656 Northlake Springs Court

Decatur, GA. 30033

Williams, N. Y.

Nova Southeastern University Fischler School of Education and Human Services

nicowil@nova.edu

2122 Little Torch Street

Riviera Beach, FL 33407

Keywords or short phrases:

classroom observation tool, distance education, ICOT, ISTE, technology integration

Running head: USING THE ISTE ICOT TO OBSERVE AN ONLINE CLASS 2

Abstract

Three doctoral students used the International Society for Technology in Education

(ISTE) Classroom Observation Tool (ICOT) to observe a synchronous (real-time) online

doctoral class meeting. The authors found that this classroom observation tool, while viable for

face-to-face classroom settings, would be useful for online classroom observations only if

modifications were made. With the growing popularity and numbers of distance education and

online classes, the authors find there is also a subsequent need for appropriate classroom

observation tools.

Brief Biographical Paragraph

The authors are all currently enrolled in doctoral programs at Nova Southeastern

University Fischler School of Education and Human Services. Cynthia Brennan and Nicole

Williams’ concentrations are in Instructional Leadership, and Melanie Miller’s concentration is

in Educational Leadership. All three authors have a minor in Curriculum and Instruction. As part

of a group project in their online EDD 7914 Technology—Integrated Teaching and Learning

course, the authors utilized an online classroom observation tool. Subsequently, they wrote this

article.
Running head: USING THE ISTE ICOT TO OBSERVE AN ONLINE CLASS 3

Introduction

As part of a group project in an online doctoral course, authors Brennan, Miller, and

Williams (2010) utilized the International Society for Technology in Education (ISTE)

Classroom Observation Tool (ICOT) to observe a synchronous online class meeting. ISTE, “the

premier membership association for educators and education leaders engaged in improving

learning and teaching by advancing the effective use of technology in PK-12 and higher

education” (ISTE, 2010) offers educators the free online ICOT. The ICOT “provides a set of

questions to guide classroom observations of a number of key components of technology

integration” (ISTE ICOT, 2010). The “ICOT is based on observation tools developed since 1998

during ISTE's work with numerous educational technology initiatives, including the National

Educational Technology Standards (NETS), Technology Innovation Challenge Grants, Preparing

Tomorrow's Teachers to Use Technology programs, and Enhancing Education Through

Technology Grants” (Felix, 2008). While the three authors all agree the ICOT is a suitable tool

for an educator or administrator to use when observing components of technology integration in

a “face-to-face” classroom, the authors argue that the ICOT as a tool needs significant

modifications if it is to be considered a useful tool for online class observations.

Description of Activity

Using the ICOT, the three authors separately observed a recorded instructor-facilitated

online class Elluminate session wherein two groups of students presented their PowerPoint

presentations on emerging technologies in the field of education synchronously to the class and

to the instructor. Problems with how to answer the ICOT questions came up consistently. For

example, because only the instructor used a webcam, it was impossible to ascertain whether

students in the class were using desktop or laptop computers (ICOT, Question 8). The “Three-
Running head: USING THE ISTE ICOT TO OBSERVE AN ONLINE CLASS 4

Minute Chart” (ICOT, Question 10) and the “Estimated time technology was used” (ICOT,

Question 11) were also, for the most part, not relevant for a synchronous meeting of an online

class, where all students and teachers are using technology for teaching and learning at all times.

The NETS Teacher Standards listed on the ICOT that the authors checked off as being

addressed were as follows: 1A.1. operating system procedures; 1A.2. routine hardware and

software problems; 1A.3. content-specific tools; 1A.4. productivity tools; 1A.5. multimedia

tools; 1A.6. interactive communication tools; 1A.7. curriculum-based presentations/publications;

1A.9. appropriate technology selected; 2A.1. developmentally appropriate learning activities;

2A.2 technology-enhanced instructional activities; 3A.1. learning experiences address content

standards; 3A.2. learning experiences address student technology standards; 3D.1. class

management facilitates engagement with technology; 3.D.2 technology integrated as a teacher

tool; 4A. 1. Student learning of subject matter assessed with technology; 4A.2. teacher assesses

student technology skills; 6D.1. safe and healthy use of technology promoted (ICOT, 2010).

However, it should be noted that not all the authors observed each of the NETS Standards listed;

there were numbers of discrepancies between the authors’ ICOTs.

The discrepancies noted both in the NETS Standards check-off list and elsewhere in the

ICOT show that both training and discussion are needed before utilizing the tool. In fact, ICOT

advises in its Training Resources section that, “Observers discuss the data with partners or the

whole group to reach consensus on how to apply the ICOT. How will you define technology as

"in use?" How will you decide if a technology use is "Essential?" What will it take to determine

if a particular standard has been addressed?” (ICOT, 2010).

In addition to training before using the tool, modifications to the questions are needed

if the ICOT is to be used for the observation of online classes. For example, the portions of the
Running head: USING THE ISTE ICOT TO OBSERVE AN ONLINE CLASS 5

ICOT which are useful for measuring student engagement when reviewing a lesson taught in a

face-to-face classroom are not as useful when observing an online class. The authors agree that it

is more difficult when observing an online class to ascertain how many of the students are

actively engaged in a lesson, but it is possible to build in ways to measure engagement. For

instance, Tally and Goldenberg (2005), in a study of the success of using digitized primary

sources in an online task, used “the student task [which] became a pilot assessment of students’

historical thinking, as revealed through the software” (p. 6). These authors hypothesized “that

well-designed software could support teachers and students in working closely with documentary

sources, and thereby improve history teaching and learning” (p. 2). Tuttle (2008) suggested using

21st-century technologies such as online quizzes and web-based surveys to assist with effective

student observations. Either of the two tools might be helpful in measuring online student

engagement with a lesson. Cook and Germann (2010) suggest that qualitative methods may be

useful as well as “numerical counts for postings and readings” (p. 6) for assessment of student

engagement with online learning material. Looking at data generated by the WebCT courseware

utilized in their study of online classes “offered virtual snapshots of student activities and

engagement” (p. 15), but, the authors concluded, “clearly further studies with more qualitative

data may broaden the picture in lieu of relying on snapshots” (p. 16).

As a final example of a difficulty encountered by the authors while using the ICOT for

this activity, answering the ICOT question “How essential was technology to the teaching and

learning activities” was especially problematic when observing an online class. For instance,

what if online students are utilizing the most basic of technology to sit passively while listening

to a lecture or viewing a PowerPoint? As the ICOT is currently written, an activity that involves
Running head: USING THE ISTE ICOT TO OBSERVE AN ONLINE CLASS 6

an online student listening to a lecture would receive the highest rating for technology

integration.

Discussion

Other available classroom observation tools, such as Barnett and Brooks-Young’s

Technology Use Observation Tool (2003), pose similar limitations when considering its use for

online class observations. Like the ICOT, Barnett and Brooks-Young’s tool offers opportunities

to pinpoint where technology integration may be lacking in a face-to-face classroom, but the

authors of this article argue that it too offers little usefulness to an observer of an online

classroom. Eib (2001), who also created an observation formula to assess the amount of

technology use in the classroom, stated that the best way to ensure that technology was being

utilized was to “use a list of instructional improvement priorities that support your school

improvement plan and reflect research-based best practices to create a rubric for observations.”

While the authors agree that using best practices and instructional priorities is a solid way to

build a rubric to assess the use and effectiveness of technology in the classroom, Eib’s

observation formula would also need modification to be useful for effective observations of an

online course.

Conclusions

Technology is a basic essential in every online activity. However, does the ICOT

measure effective technology in an online classroom environment? The authors respectfully

suggest that the answer is “no.” In order to effectively measure key components of technology

integration, for online classroom observations, the questions found in the ICOT must be

modified to reflect that basic technology, i.e. being online, is a given, and find ways to measure

higher levels of technology integration and online student engagement. With the growing
Running head: USING THE ISTE ICOT TO OBSERVE AN ONLINE CLASS 7

popularity of distance education and online classes, the authors suggest that there is also a

subsequent need for appropriate online classroom observation tools.

Running head: USING THE ISTE ICOT TO OBSERVE AN ONLINE CLASS 8

References

Barnett, H. (2003). Technology professional development: Successful strategies for teacher change.
Syracuse, NY: ERIC Clearinghouse on Information & Technology. (ERIC Document
Reproduction Service No. ED477616)

Barnett, H. & Brooks-Young, S. (2002). Technology Use Lesson: Observation Tool. Retrieved June 25,
2010, from http://members.tripod.com/sjbrooks_young/techobstool.pdf

Cook, S. and Germann, C. (2010). The tell-tale data: Virtual whispering and final student grades. Online
submission. Syracuse, NY: ERIC Clearinghouse on Information & Technology. (ERIC Document
Reproduction Service No. ED508149)

Eib, B. J. (2001). Beyond the bells and whistles: Evaluating technology use in the classroom. Principal
Leadership, 9, 16-23.

Felix, K. (2008). ISTE Classroom observation tool (ICOT). MultiMedia & Internet at Schools, 5, 41.

International Society for Technology in Education (ISTE). (2010). Retrieved June 22, 2010, from
http://www.iste.org/

International Society for Technology in Education (ISTE). (n.d.). The ISTE Classroom
Observation Tool (ICOT). Retrieved June 17, 2010, from http://www.iste.org/icot/

Tally, B. & Goldenberg, L. B. (2005). Fostering historical thinking with digitized primary
sources. Journal of Research on Technology in Education, 38(1), 1-21.

Tuttle, H. G. (2008). Digital age assessment: Part 1. Technology & Learning, 28(8). Online
version. Retrieved June 26, 2010 from http://www.techlearning.com/article/8592