Six Facets of

Instructional Product Evaluation

Review

Ne
ce

ed
an

s
As
en

se
int

ss
Ma

me
nt
ve
Im

ati
p

rm
ac

Fo
t

Effectiveness
Evaluation Functions
Development Activitie
Review Product Conceptualization

Needs Assessment Design

Formative Evaluation Development

Effectiveness Evaluation Implementation

Impact Evaluation Institutionalization

Maintenance Evaluation Project Re-conceptualization

 EMCC Design Document
• Urban Science Course Environment
Dimensions of effective technology
enhanced learning environments:
● Task-Oriented
● Challenging
● Collaborative
● Constructionist
● Conversational
● Responsive
● Reflective
● Formative
 Task-Oriented
Academic Authentic

The tasks faculty set

for students define the
essence of the
learning environment.
If appropriate, tasks
should be authentic
rather than academic.
 Task-Oriented Example

Students in online
instructional
design courses
are tasked with
designing
interactive
modules for real
clients.
 Challenging
Simple Complex

The notion that

interactive learning is
easy should be
dispelled. Learning is
difficult and students
should not spoon fed
simplified versions of
their fields of study.
 Challenging Example

In a Masters of
Public Health
program, students
confront problems
as complex and
difficult as the ones
they’ll face in the
real world.
 Collaborative
Unsupported Integral

Web-based tools for

group work and
collaboration can
prepare students
for team work in
21st Century
work environments.
 Collaborative Example
Art, dance, and
music students are
collaborating to
produce online
shows with digital
versions of their
works and
performances for
critique by
international experts.
 Constructionist
Replication Origination

Faculty should
engage students in
creating original
knowledge
representations
that can be shared,
critiqued, and
revised.
 Constructionist Example
Students in fields
ranging from aero-
engineering to zoo
management are
producing digital
portfolios as integral
components of
their academic
programs.
 Conversational
One-way Multi-faceted

Students must have

ample time and
secure spaces for
in-depth
discussions,
debates, arguments,
and other forms of
conversation.
 Conversational Example
New knowledge and insight are being
constructed in conversation spaces
such as the e-learning forums found in
BlackBoard, WebCT, Desire2Learn,
and other
online
learning
authoring
platforms.
 Responsive
Superficial Genuine

In learning
communities, both
faculty and students
have a mutual
responsibility to
respond quickly,
accurately, and
with respect.
 Responsive Example
This is an area
where R&D are
needed. Some
universities are
seeking to establish
supportive online
networks that will
continue throughout
a career, indeed
throughout a life.
 Reflective
Shallow Deep

Both faculty and

learners must
engage in deep
reflection and
metacognition. These
are not instinctive
activities, but they
can be learned.
 Reflective Example

Teacher preparation
students are keeping
electronic journals to
reflect upon the
children they teach,
and their roles as
advocates for
children.
 Formative
Fixed Assessment Developmental
Learning
environments can be
designed to allow
students to develop
prototype solutions
over time rather than
to find one right
answer that someone
else has defined.
 Formative Example
Faculty should
engage their
students in ongoing
efforts to evaluate
and refine their
work related to
authentic tasks to
encourage lifelong
learning.
Traditional Course Online Course
Task-Oriented
Challenging
Collaborative
Constructionist
Conversational
Responsive
Reflective
Formative
Heuristic Review
 What is usability?
• The concern with
designing software
applications which
people find easy to
use and personally
empowering.
• Usable computer
programs are logical,
intuitive, and clear to
the people who use
them.
 Web Site Usability
• “ The most common user action
on a Web site is to flee.”
Edward Tufte
• “at least 90% of all commercial
Web sites are overly difficult to
use…..the average outcome of
Web usability studies is that test
users fail when they try to perform
a test task on the Web. Thus, when
you try something new on the
Web, the expected outcome is
failure.”

Jakob Neilsen
 Typical Web Usability Problems
• bloated page design
• internally focused
design
• obscure site
structures
• lack of navigation
support
• writing style
optimized for print Jakob Neilsen
http://www.useit.com/
 Key Usability Principles
• Structure - organize
meaningfully
• Simplicity - make
common tasks easy
• Visibility - all data
needed for a task
• Feedback - keep users
informed
• Tolerance - allow cancel,
back
• Reuse - reduce the
users' need to remember
 Nielsen’s Web Usability Rules
• Visibility of system status
• Match between system and
real world
• User control and freedom
• Consistency and standards
• Error prevention
• Recognition rather than recall
• Flexibility and efficiency of
use
• Help users recognize,
diagnose, and recover from
errors
• Help and documentation
• Aesthetic and minimalist
design
Two Major Ways to Evaluate Usability

• Heuristic Review
– quick and relatively inexpensive
– based on expert analyses
– no user involvement
• Usability Testing
– finds more problems
– user involvement increases validity
– when designers see problems live, it
has a huge impact
 Heuristic Review
 Several experts individually compare a
product to a set of usability heuristics
– Typical heuristic:
• Visibility of
system status
• The system should
always keep users
informed about
what is going on,
through appropriate
feedback within
reasonable time.
 Heuristic Review
 Violations of the heuristics are
evaluated for their severity and extent
Severity Scale:
1 Cosmetic: fix if possible.
2 Minor: fixing this should be given low priority.
3 Medium: fixing this should be given medium priority.
4 Major: fixing this should be mandatory before the system is
launched. If the problem cannot be fixed before launch, ensure
that the documentation clearly shows the user a workaround.
5 Catastrophic: fixing this is mandatory; no workaround possible.

Extensiveness Scale:
1 Single case
2 Several places
3 Widespread
 Heuristic Review
 At a group meeting, violation reports are
categorized and assigned
 Heuristics violated are
identified

 Average severity and

extensiveness ratings are
compiled

 Opportunities for
improvement are clarified

 Feasible solutions are

recommended
 Heuristic Review
• Example of Opportunity For Improvement

Opportunity 1 (4 reports. Avg. Severity=2.25, Avg. Extent=2.34,

Heuristics Used: 1, 3)

Consider providing more user feedback about where they are and
what they should do next. Examples cited:
• No Page progress indicator
• No indication of how to start

Suggestions:
• Provide a page-progress indicator, such as “page 3 of 12”
• Put a “Click a section below to start:” on the first screen, as a TOC
header
 Heuristic Review
• Advantages
– Quick: Do not need to find or schedule users
– Easy to review problem areas many times
– Inexpensive: No fancy equipment needed

• Disadvantages
– Validity: No users involved
– Finds fewer problems (50% less in some cases)
– Getting good experts can be challenging
– Building consensus with experts is sometimes
difficult
 Another Weakness

• Some people
believe that
heuristic evaluation
is too subjective.
• Human judges are
prone to poor
judgment at times.
 Usability Standards
http://www.astd.org/ASTD/marketplace/ecc/ecc_home
ASTD offers certification of e-learning courses,
including 8 usability standards:
-Navigation
-Orientation
-Feedback cues
-Link cues
-Links labeled
-Help
-Legibility
-Text quality
 Heuristics for E-Learning Evaluation

1. Visibility of
system status:
The e-learning
program keeps the
learner informed
about what is
happening, through
appropriate
feedback within
reasonable time. •red for a problem
•yellow for a warning
•green for OK
 Heuristics for E-Learning Evaluation
2. Match between system
and the real world: The
e-learning program’s
interface employs words,
phrases and concepts
familiar to the learner or
appropriate to the content,
as opposed to system-
oriented terms. Wherever
possible, the e-learning
program utilizes real-world
conventions that make
information appear in a
natural and logical order.
 Heuristics for E-Learning Evaluation
3. Error Recovery and
Exiting: The e-
learning program
allows the learner to
recover from input
mistakes and
provides a clearly
marked “exit” to leave
the program without
having to go through
an extended dialogue.
 Heuristics for E-Learning Evaluation
4. Consistency and
standards: When
appropriate to the
content and target
audience, the e-
learning program
adheres to general
software conventions
and is consistent in
its use of different
words, situations, or
actions.
 Heuristics for E-Learning Evaluation
5. Error prevention: The e-learning program is
carefully designed to prevent common problems
from occurring in the first place.
 Heuristics for E-Learning Evaluation
6. Navigation support:
The e-learning program
makes objects, actions,
and options visible so
that the user does not
have to remember
information when
navigating from one part
of the program to
another. Instructions for
use of the program are
always visible or easily
retrievable.
 Heuristics for E-Learning Evaluation
7. Aesthetics:
Screen displays do
not contain
information that is
irrelevant, and
“bells and whistles”
are not gratuitously
added to the e-
learning program.
 Heuristics for E-Learning Evaluation
8. Help and
documentation: The e-
learning program
provides help and
documentation that is
readily accessible to the
user when necessary.
The help provides
specific concrete steps
for the user to follow. All
documentation is written
clearly and succinctly.
 Heuristics for E-Learning Evaluation
9. Interactivity: The e-learning program provides
content-related interactions and tasks that support
meaningful learning.
 Heuristics for E-Learning Evaluation
10.Message
Design: The
e-learning
program
presents
information in
accord with
sound
principles of
information-
processing
theory.
 Heuristics for E-Learning Evaluation
11.Learning Design:
The interactions in
the e-learning
program have been
designed in accord
with sound principles
of learning theory.
 Heuristics for E-Learning Evaluation
12.Media Integration: The inclusion of media in the
e-learning program serves clear pedagogical
and/or motivational purposes.
 Heuristics for E-Learning Evaluation
13.Instructional
Assessment:
The e-learning
program provides
assessment
opportunities that
are aligned with
the program
objectives and
content.
 Heuristics for E-Learning Evaluation
14.Resources:
The
e-learning
program
provides
access to all
the resources
necessary to
support
effective
learning.
 Review
• The purpose of review is to
ensure that the development
team is well-informed about
previous work done in the area
during the early stages of
product conceptualization.
• Designers must avoid recreating
the wheel.
 Review
I can do
• The two primary methods used better
are reviewing the related than this!
literature and reviewing
competing products.
• Regularly reviewing competing
products is a great professional
development practice.
 Needs Assessment
• The purpose of needs
assessment is to identify the
critical needs that an instructional
product is supposed to meet.
• Needs assessment provides
essential information to guide the
design phase of the development
process.
 Needs Assessment
• The primary methods are:
– task analysis,
– job analysis, and
– learner analysis.
• One of the most important results is
a list of specific goals and
objectives that learners will
accomplish with the new product.
 Formative Evaluation
• The purpose is to collect
information that can be used for
making decisions about
improving interactive learning
products.
• Formative evaluation starts with
the earliest stages of planning
and continues through
implementation.
 Formative Evaluation
• Provided the results are used,
formative evaluation usually
provides the biggest payoff for
evaluation activities.
• Clients may be reluctant to
accept the results of formative
evaluation, especially as a
program nears completion.
 Effectiveness Evaluation
• The purpose is to estimate
short-term effectiveness in
meeting objectives.
• It is a necessary, but
insufficient, approach to
determining the outcomes
of interactive learning.
 Effectiveness Evaluation
• Evaluating implementation is
as important as evaluating A connection with
outcomes. the server could not
be established?
• If you don’t understand how
instructional products were
actually implemented, you
can’t interpret results.
 Impact Evaluation
• The purpose is to estimate the
long-term impact on
performance, both intended
and unintended.
• It is extremely difficult
to evaluate the impact
of interactive learning
products.
 Impact Evaluation
• Evaluating impact is
increasingly critical because of
emphasis on the bottom line.
• More and more clients expect
impact evaluation to include
“return-on-investment” (ROI)
approaches.
 Maintenance Evaluation
• The purpose of maintenance
evaluation is to ensure the
viability of an interactive
product over time.
• Maintenance is one of the
weakest links of web-based
learning environments.
 Maintenance Evaluation
• Document analysis, interviews,
observations, and automated data
collection are among the methods
used in maintenance evaluation.
• Very few education and training
agencies engage in serious
maintenance evaluation.
 Planning is the key to successful
instructional product evaluation.
• Evaluation requires good
planning, careful
implementation, and
systematic follow-up.
• A major challenge is
getting clients to identify
the decisions they face.
• Clear decisions drive the
rest of the planning.
 Heuristics for E-Learning Evaluation
15.Feedback:
The
e-learning
program What the heck?
provides
feedback that
is contextual
and relevant
to the
problem or
task in which
the learner is
engaged.