Joe Goldberg

IDDE 611
Dr. Lei

Comprehensive Evaluation Report

Site Title: Teach with Movies

Site URL: http://www.teachwithmovies.org/

Site Creators: James Frieden and Deborah W. Elliott

Evaluation Contact Consultant: Joe Goldberg

Syracuse University IDDE CAS student and
Teaching and Curriculum doctoral candidate
Jagold05@syr.edu

Goals and Learning Objectives of the Project:

This project aims to objectively analyze and evaluate the efficacy of the designated

website in order to identify its strengths and weaknesses so as to ultimately offer

recommendations on potential improvements.

 Website General Description

Purpose of the Site:

TeachWithMovies.com bills itself as “the premiere site on the Internet showing teachers

how to create lesson plans using movies and film.” Spanning most of the core and some of the

special content areas, and offering film-related lessons suitable for ages ranging from 3 years

old to college-aged, the husband and wife team that founded the site in 1998 realized “that

carefully selected feature films could supplement curriculum and foster social-emotional

learning” and build the site as “a way of sharing this concept.”

Within five years of inception, the site boasted tens of thousands of monthly visitors and

was selected as a contributor to the Gateway to Educational Materials (GEM), a project

sponsored by the U.S. Department of Education “to provide a central clearinghouse for the best

lesson plans and teacher resources on the Internet.”

Target Audience(s) of the Site

By its own account, “Educators comprise 80% of the site's audience,” and the “next

largest contingent is made up of home schooling parents.”

Content of the Site

This site specializes in offering film-related lesson plans, with step-by-step instructional

guides, background information, and assessments are organized by length of film, classroom

content area, age, and several other niche categories, as reflected by the content toolbar links:

 Snippets & Shorts

 English
 Social Studies
 Sciences
 Other
 10 Best
 SEL
 Alpha
  Age
 FAQ
 What’s New
 Articles
 Reward Films
 Set Up The Sub
 Movie Worksheets
 Documentaries/Nonfiction
 TV Show Lesson Plans

Special and/or Unique Features of the Site

This website employs a variety of eye-catching DVD covers of popular Hollywood films

set in relief of brightly colored framing and attractive logo. A free newsletter subscription is

offered as well as links to the site’s Facebook and Twitter pages.

Methods

A Brief Description of the Evaluation

I employed two chiefly quantitative evaluative instruments to judge this website,

including the Content Validity Scale© (Arnone & Small, 1999) and the Website Motivational

Analysis Checklist (WebMAC) Professional©, Version 2.0 (Arnone & Small, 1999, revised in

2004). The former uses a five point Likert scale across ten items examining the credibility,

accuracy, reliability, and presentation of the site (see details below).

The latter, “designed for professional educators” in the classroom, requires the evaluator

to rate 32 items by agreeing or disagreeing with each either “strongly” or “somewhat.” For any

items that are not applicable, the evaluator will next determine if the site would have benefited

from it, didn’t require it, or was better off for not including it. The scores for each item are then

tabulated by parsing them into four thematic categories: stimulating, meaningful, organized, and

easy-to-use (S.M.O.E.). The values for each category are charted to determine if the site is

“highly motivating,” “needs some improvement,” or “needs much improvement.” Finally, the
values are combined (stimulating and meaningful, as well as organized and easy-to-use,

respectively) to determine a “value dimension” and a “summary motivation” and then graphed to

identify site’s “Expectation for Success.” This instrument also features two yes or no questions

about whether the evaluator would visit the site again or recommend it to another, as well as two

open-ended questions for qualitative responses for things that may not fit within the parameters

of the 32 pre-determined evaluative factors.

Evaluators

In addition to myself, two other people evaluated the website using the two evaluative

instruments detailed above, and the median scores of all three were used to plot how the website

ranked (shown below).

Jane Cook (pseudonym), a high school level library media specialist, was the first to

complete both evaluations of the website. She was specifically selected for her vast experience in

evaluating resources of all kinds, and specifically websites. Indeed, she regularly instructs on the

topic.

Sidney Lane (pseudonym), another student in IDE 611, was the second person to evaluate

the website. She was specifically selected for her familiarity with the course, this assignment,

and her understanding of the goals of this class.

Data Analysis

After all three evaluators rated the website using both evaluative instruments, the data

from each was collected, tabulated, and charted separately for comparative purposes. Following

that, the mean scores of all three were calculated and charted from each instrument so that the

overall results could be determined for gleaning recommendations.

 Results

Content Validity Scale Results

Median scores of the three evaluations are based on the following Likert scale:

Strongly Disagree 1 2 3 4 5 Strongly Agree

# Items Mean
Score
1 The source of information for this Web site is credible. 5
2 There is a way to contact the author of this Web site, if necessary. 3.7
3 The factual information or content of the Web site seems accurate. 5
4 If the Web site presents concepts or principles in its domain (e.g. science, 5
art), they are appropriately presented without confusing or missing
information.
5 There are no typographical or spelling errors that could potentially cause 5
the information at this Web site to be misunderstood.
6 The content is appropriate for the intended audience. 5
7 The links from this site appear to be credible. 5
8 This Web site appears to be free of bias. 4.7
9 The information at this Web site is current enough for the type of 5
information it includes.
10 The links from this Web site appear current and unbiased 5

Website Motivational Analysis Checklist Results

 Quality Scale Mean Score
Stimulating 0-24 19.67
Meaningful 0-24 22.33
Organized 0-24 24
Easy-to-Use 0-24 21

S + M = 42 (V) O + E = 45 (XS)

Interpretation and Synthesis of Overall Findings

The data show that this website is highly effective overall and carries a high expectation

for success with its target audiences. With the first evaluative instrument, the site boasted near

perfect scores, excepting its ease of contacting the site’s authors and minimal bias. On the second

instrument, the site scored similarly well, as I will detail next by the categories of that evaluative

tool.

Stimulating

This website earned high scores for its motivational qualities, and the evaluators gave

high marks for its aesthetic qualities, including its screen layout, eye-catching visuals on the
homepage, with information presented very clearly and without typos. One area for improvement

in this regard, however, is the site’s lack of interactive opportunities.

Meaningful

The information and links on the website was consistently well-regarded by the

evaluators. The website’s material was uniformly rated as relevant, accurate, and largely

unbiased.

Organized

The website earned its highest marks with its organization. The site affords multiple ways

to access film titles and accompanying lesson plans, whether by genre, content area, age group,

alphabetically by title, and by several other categorical options. Even just hovering the cursor

above some of the tabs displayed dropdown menus with subcategories within the tab. Though the

home page offers myriad information, its clear sections, tabs, and layout obviate overwhelming

the visitor.

Ease-of-Use

The website also rated rather highly for its utility and intuitiveness. Largely because of its

organization, navigating the site is very easy, and search options make this all the more so. Most

of the links work as designed, and they all load at fast speeds. The evaluators indicated, however,

that improvements could be achieved by adding a help function of some kind to assist visitors for

any of their needs.

Overall Ratings

All told, the site ranked in the “highly motivating” and “awesome website” ranges with a

high expectation for success. The evaluators unanimously indicated that they would visit the site
again and recommend it to a friend. The qualitative responses, which I will share next, confirm

the same basic findings interred in the quantitative data.

Qualitative Responses

Strengths

 “Educational information can be used across multiple levels (e.g., elementary to

college) and the site provides clear use of movies and/or clips in an educational

context. It also provides valid background information that instructors may use to

prepare both students and themselves to appropriately use video media, including

time frames and evaluation.”

 “Well organized with quick and easy navigation features.”

 “Aesthetically pleasing, easy to navigate, useful info not easily available elsewhere.”

Areas for improvement

 “The site has many distracting advertisements on its homepage, and the font choice is

slightly out of date. There is also a Biology link that no longer exists (Predicting

Combinations for Alleles).”

 “Improve access to the website authors/contributors contact info.”

 “Interactivity opportunities.”

Addendum

Though the two evaluative instrument employed to measure the efficacy of this website

resulted in very high scores, the site and its visitors would profit from assessing it through yet

another lens. Upon further review, there are some significant shortcomings for visitors who may

have disabilities in accessing the site and its content. Applying version 2.0 of the Web Content

Accessibility Guidelines (WCAG) reveals that the website appeals to the senses only minimally,
as there are no sounds, with no apparent assistive technologies, such as screen readers or

enlargers. Operability through use of a computer mouse is fully functional, though the same may

not be said for using a keyboard, and it remains unclear if the site is compatible with assistive

devices. The content is mostly clear and well-organized, though distracting advertisements

distract and detract. Finally, the website is ironically insipid and limited in its accessibility for

assistive technology and diverse users.

Recommendations

The recommendation stemming from this evaluation is that the website authors build on

its many existing strengths by primarily increasing its interactivity with its visitors. This may

come in the following forms: an online chat option for customer help and other queries, and,

relatedly, increase contact information and means for visitors to communicate with the site’s

authors and other contributors. Additional interactive opportunities should also be considered,

including a message or comment board. While the site scored highly for its overall appearance,

the authors might nevertheless consider updating the font to a more modern look, and attempt to

present their advertisements in a less obtrusive way. Additionally, while nearly all links are

working properly and are relevant, the stray one or two broken or outdated ones should be

repaired. Overall, this is a very strong website, and a few upgrades could take it to new heights.

Additional Recommendations

Further recommendations include building on the website’s strongly defined content

structure, replete with headers and tabs, by adding multi-modal content which would benefit all

users, and particularly those who are otherwise excluded. In this way, adding video and audio

content, including closed captioning, would substantively improve this site. So, too, would

alternative text options, and content that is not device dependent.

 Reflection

This project was a particularly useful as it concerns digital instruction in particular and

operating as a layperson in the 21st Century generally. Given that the wonted way of obtaining

information has evolved from printed to primarily digital texts, an extra level of scrutiny must be

applied to the reliability of the source and attendant content. Absent that, audiences are

increasingly vulnerable to misinformation, or even disinformation, given the lack of safeguards

(imperfect though they may be) that are part and parcel of printed newspapers, textbooks, and

other similarly established sources with oversight of reported information. In an instructional

position, this is not only critical to teach students but to apply in one’s own pedagogic practices

(and personal life) as well.

I had worried that eliciting partners and working as a group might present some

challenges, but I was very relieved to discover how smoothly this ended up going. I appreciated

that the guidelines allowed for partners to come from our class or from elsewhere, which not

only made the process easier for me to accomplish, but it gave a wider range of perspectives.

Having three evaluators was especially productive, as evidenced by the responses that served to

confirm and disconfirm judgments made about the site between the three.

The one salient thought that remained with me in evaluating the website with the two

instruments is that they might benefit from being updated. While I think they are rather

foresighted in many of the evaluative items, the most recent update goes back to 2004. Of

course, in the fast-evolving world of digital media, that is quite dated. Additional items could

likely be added to set the standard for what might qualify as a paragon for a website today,

including live chat features, social media integration, video, and other modern elements. I
wonder, too, if there is room for improvement in detailing specific things to look for that make

for a finding of bias, especially given that bias may often be subtle and surreptitious.

I think I will substantively profit from the experience of this project by applying the

lessons mentioned here across four fronts: my personal life, my classroom, my program at SU,

and potential future consulting endeavors that I partake in as a professional. While I like to think

I had some a priori knowledge relating to this subject, it did underscore for me many nuances

that I hadn’t fully considered in thinking of effective, reliable websites and their roles in

pedagogy and beyond.

Resources

Content Validity Scale© (Arnone & Small, 1999)

Web Content Accessibility Guidelines (WCAG) 2.0 (Caldwell, et al, 2008)

Website Motivational Analysis Checklist (WebMAC) Professional©, Version 2.0 (Arnone &
Small, 1999, revised in 2004).

http://www.teachwithmovies.org/