CHAPTER 10

Introducing Evaluation
Chapter Goals
Explain the concepts and terms used to discuss
evaluation
Discuss Hutch World Case Study
Examine different techniques used at different stages
of development
Show and discuss how developers deal with real world
problems and constraints
 Introduction
Identify needs/

establish

requirements

(Re)Design

Evaluate

Build an

interactive

version

Final product
 Introduction
Designing useful and attractive products requires skill
and creativity.
As products evolve from initial ideas through conceptual
design and prototypes, iterative cycles of design and
evaluation help to ensure that they meet users' needs.

But how do evaluators decide what and when to

evaluate?

The Hutchworld and 1984 OMS case studies described

how one team did this`
 What is Evaluation ?
 a systematic and objective assessment of an ongoing or
completed project, programme or policy, its design,
implementation and results.
 a structured process of assessing the success of a project in
meeting its goals and to reflect on the lessons learned.

 The aim is to:

 determine the relevance and fulfilment of objectives,
efficiency, effectiveness, impact and sustainability
 ensure that product meets the users’ needs
Two main types of Evaluation
1. Formative Evaluation is done at different stages of
development to check that the product meets users
needs.

2. Summative Evaluation assesses the quality of a

finished product.
 What to evaluate

 Early ideas for conceptual model

 Early prototypes of the new system
 Later, more complete prototypes
 Usability:
measuring the performance of typical users on typical tasks.
Satisfaction, can be evaluated through questionnaires and
interviews
 user experience:
evaluating more subjective user-experience goals, like
emotionally satisfying, motivating fun etc.
 Why you need to Evaluate
• To check that users can use the product and like it.
• Designers need to check that they understand users’
requirements.
• To determine the relevance and fulfilment of objectives,
efficiency, effectiveness, impact and sustainability
• Toganazzini’s 5 reasons to evaluate:
1. Problems are fixed before the product is shipped, not after
2. The team can concentrate on real problems, not imaginary
ones
3. Engineers code is sharply reduced
4. Time to market is sharply reduced
5. Upon 1st release, your sales department has a rock solid
design it can sell

•
When to evaluate
Throughout design
From the first descriptions, sketches etc. of users needs
through to the final product
Design proceeds through iterative cycles of ‘design-test-
redesign’
New product
mockups, sketches, and other low fidelity prototyping
techniques are used to represent design ideas
Upgrading existing products
Compare user performance and attitudes and contrast
new products with the previous versions
Key points
 Evaluation & design are closely integrated in user-centered
design.

 Some of the same techniques are used in evaluation &

requirements but they are used differently
(e.g., interviews & questionnaires)

 Triangulation involves using a combination of techniques to

gain different perspectives

 Dealing with constraints is an important skill for evaluators

to develop.
 Chapter 11:
An Evaluation Framework
Chapter Goals
Explain the concepts and terms used to discuss
evaluation
Discuss Hutch World Case Study
Examine different techniques used at different stages
of development
Show and discuss how developers deal with real world
problems and constraints
 Chapter Goals
Key concepts & terms to discuss evaluation
Description of evaluation paradigms & techniques
Introduce and explain the DECIDE framework.
Discuss the conceptual, practical, and ethical issues
involved in evaluation.
 Evaluation Paradigms
Any kind of evaluation is guided explicitly or
implicitly by a set of beliefs, which are often
under-pined by theory. These beliefs and the
methods associated with them are known as an
‘evaluation paradigm’

The set of beliefs and practices (i.e., the methods

or techniques) which guide any type of evaluation
Or associated with any kind of evaluation.
 Four evaluation paradigms
1. “Quick and dirt” evaluation
2. Usability testing
3. Field Studies
4. Predictive Evaluation
1. “Quick and Dirty”
Quick & Dirty evaluation describes the common
practice in which designers informally get
feedback from users or consultants to confirm that
their ideas are in-line with users’ needs and are
liked.
Quick & Dirty evaluations are done any time.
The emphasis is on fast input to the design process
rather than carefully documented findings.
2. Usability testing (See Detail Notes)
Usability testing involves recording typical users’
performance on typical tasks in controlled settings.
As the users perform these tasks they are watched
& recorded on video & their key presses are
logged.
This data is used to calculate performance times,
identify errors & help explain why the users did
what they did.
User satisfaction questionnaires & interviews are
used to elicit users’ opinions.
Recall HutchWorld and OMS Case studies
3. Field Studies
Field studies are done in natural settings
The aim is to understand what users do naturally
and how technology impacts them.
In product design field studies can be used to:
- identify opportunities for new technology
- determine design requirements
- decide how best to introduce new technology
- evaluate technology in use.
3. Field Studies
Two Approaches
Outsider – observing and recording what
happens as an outsider looking in
Insider – participant in study that explores the
details of what happens in a particular setting
4. Predictive Evaluation
Experts apply their knowledge of typical users,
often guided by heuristics, to predict usability
problems.
Note: heuristics- design principles used in practice
Another approach involves theoretically based
models.
A key feature of predictive evaluation is that users
need not be present
Relatively quick & inexpensive
Evaluation Techniques
Observing users
Asking users their opinions
Asking experts their opinions
Testing users’ performance
Modeling users’ task performance to predict the
efficacy of a user interface
Observing Users
Techniques
notes
audio
video
interaction log
Asking users their opinions
Questions like:
what do you think about the product?
does it do what you want?
do you like it?
does the aesthetic design appeal to you?
did you encounter problems?
would you use it again?
Asking experts their opinions
Use heuristics to step through tasks
Typically use role-playing to identify problems
It is inexpensive and quick to ask experts rather than
perform laboratory and field evaluations
User Testing
Recall HutchWorld example
Usually conducted in a controlled environment
Users perform well-defined tasks
Data can be collected and statistically analyzed
Modeling users’ task performance
Model human-computer interaction to predict the
efficiency and problems in the design
This is successful for systems with limited
functionality

Table 11.2 - page 347

 DECIDE:
A framework to guide evaluation

Determine the goals the evaluation addresses.

Explore the specific questions to be answered.
Choose the evaluation paradigm and techniques
to answer the questions.
Identify the practical issues.
Decide how to deal with the ethical issues.
Evaluate, interpret and present the data.
 Determining the Goals
What are the goals of the evaluation?
Who wants it and why?
Goals influence the paradigm for the study.
Some examples of goals:
 Check that evaluators have understood user needs
 Check to ensure that the final interface is consistent.
 Investigate how technology affects working practices.
 Improve the usability of an existing product .
 Explore the Questions
All evaluations need goals & questions to guide them
so time is not wasted on ill-defined studies.

For example, the goal of finding out why many

customers prefer to purchase paper airline tickets
rather than e-tickets can be broken down into sub-
questions:
- What are customers’ attitudes to these new tickets?
- Are they concerned about security?
- Is the interface for obtaining them poor?
 Choose Evaluation Paradigm and
Techniques

Evaluation Paradigms determine which type of

techniques will be used.

Trade-Offs

Combinations of Techniques
-HutchWorld
 Identifying Practical Issues
For example, how to:

select users
stay on budget
staying on schedule
evaluators
select equipment
 Decide on Ethical Issues
Consideration for peoples rights.
Develop an informed consent form
Participants have a right to:
- know the goals of the study
- what will happen to the findings
- privacy of personal information
- not to be quoted without their agreement
- leave when they wish
“do unto others only what you would not mind
being done to you”
 Evaluate, Interpret, and Present Data

Reliability
Validity
Biases
Scope
Ecological Validity
 Pilot Studies
Pilot Study is a small trial run of the main study.
Pilot studies are always useful for testing plans for
an evaluation, before launching the main study
Often evaluators run several pilot studies.
User studies

User studies involve looking at how people behave in

their natural environments, or in the laboratory, both
with old technologies and with new ones.
Case studies
1984 OMS
Background
Voice mail system for Olympic Games Contestants and
their families could send and receive messages
Developed by IBM
Reason for intense evaluation
 IBM’s reputation at stake
 Olympics a high profile event
Evaluating the 1984 OMS
• Early tests of printed scenarios & user guides
 Early simulations of telephone keypad
 An Olympian joined team to provide feedback
 Interviews & demos with Olympians outside US
 Overseas interface tests with friends and family.
 Free coffee and donut tests
 Usability tests with 100 participants.
 A ‘try to destroy it’ test
 Pre-Olympic field-test at an international event
 Reliability of the system with heavy traffic
 Iterative testing of user guides
 Overseas tests
 Pre-Olympic field tests
HutchWorld Case Study
 HutchWorld: clinical study of computer-mediated social
support for cancer patients and their caregivers

To address the needs of cancer patients and their caregivers,

Microsoft Research and the Fred Hutchinson Cancer
Research Center developed HutchWorld, an online
community environment, to provide computer-mediated
social and informational support. In a controlled clinical
study, we deployed HutchWorld to bone marrow transplant
patients and their caregivers and assessed the impact of
Internet access and HutchWorld on their quality of life. We
found that Internet access and the use of HutchWorld helped
to buffer study participants against reductions in life
satisfaction and social support following the transplant
procedure. In particular, participants used the Internet to seek
out support from family and friends
HutchWorld Case Study
Virtual community
Collaboration
Microsoft’s Virtual Worlds Research Group
Fred Hutchinson Cancer Research Center
Uses
Chatting, storytelling, discussions and emotional support
Why ??
Cancer patient isolation issues
Development of HutchWorld

Many informal meetings with patients, carers & medical

staff early in design

Early prototype was informally tested on site

Designers learned a lot e.g.
- language of designers & users was different
- asynchronous communication was also needed

Redesigned to produce the portal version

HutchWorld Design Question ?
Needs
Useful
Engaging
Easy to use
Emotional satisfaction
Early Ideas
What resource are available to patients?
What are the specific needs of the user’s?
What should it look like?
How will users interact within the Virtual community
Design Team Efforts !!!
Interviews with patients, caregivers, family, friends,
clinicians, social support groups, former friends, and
experts
Reading of latest research literature and HutchWorld
web pages
Visiting Fred Hutch research facilities, and the Hutch
school for pediatric patients and juvenile patient family
members
Problems
Inadequate non-verbal feedback
Potential for mis-understanding
No :
 Facial expressions
 Body language
 Tone of voice
Features of HutchWorld
Availability
Anytime, day or night
Regardless of geographic location
Design to resemble the outpatient facility
This real-world metaphor helped users infer the
functionality
Synchronous chat environment was selected for
realism
3D photographic avatars
Testing HutchWorld
Test 1:
six computers
scaled-back prototype
Microsoft specialists trained Hutch volunteers
events were hosted in the prototype
Test 1 observations:
general usage of the prototype
usage of the space during unscheduled times
Testing HutchWorld (cont’d)
Test 1 results:
small user community
critical mass concept – not enough participants to fill the
chat room for successful conversation
lack of interest
patient availability
patients preferred asynchronous communication (via email,
journals, etc.)
prototype did not include original computer uses
patients played games and searched the internet
Redesigning HutchWorld
a more “unified” product was desired that included a
variety of communication, information, and
entertainment tasks

new support:
more asynchronous communication
information-retrieval tools
email, a bulletin board, text-chat
games and other entertainment tasks
a web page creation tool
a way to check if anyone is around to chat with
Usability testing

User tasks investigated:

- how users’ identify was represented
- communication
- information searching
- entertainment

User satisfaction questionnaire

Triangulation to get different perspectives

Usability Tests
 Seven participants
 four had used chat rooms
 all had browsed the web
 given five minutes to get familiar with software
 A running commentary was given by each during exploration (what
each was looking at, thinking, or confused by)
 After five minutes, a series of structured tasks were given focusing on
how the participants:
 dealt with their virtual identity
 communicated with others
 retrieved desired information
 found entertainment
Questionnaire
After the test participants were asked to fill out a
questionnaire about their experience with HutchWorld
What did you like about HutchWorld?
What did you not like about HutchWorld?
What did you find confusing or difficult to use in
HutchWorld?
How would you suggest improving HutchWorld?
Usability Findings
The back button did not always work.
Users ignored navigation buttons
more prominent buttons were needed
Users expected that objects in 3D would do something when
clicked on
provide links to web pages when objects are clicked
Users did not realize other real people were interacting with
them in the world
wording was changed in the overview description
Users did not notice the chat window and instead chatted with
people on the participation list
instructions on where to chat were clarified
Future of HutchWorld
Evaluation of the effects of the software at the Fred
Hutchinson Center
Investigation will include:
How the computers and software impact the social
wellbeing of the patients and their caregivers?
What type of computer-based communication best supports
this patient community?
What are the general usage patterns of the system?
How might any medical facility use computers and
software like HutchWorld to provide social support for its
patients and caregivers?
Findings from the usability test
The back button didn’t always work
Users didn’t pay attention to navigation buttons
Users expected all objects in the 3-D view to be clickable.
Users did not realize that there could be others in the 3-D world
with whom to chat,
Users tried to chat to the participant list.
 Key points
 Evaluation & design are closely integrated in user-centered design.

 Some of the same techniques are used in evaluation & requirements

but they are used differently
(e.g., interviews & questionnaires)

 Triangulation involves using a combination of techniques to gain

different perspectives

 Dealing with constraints is an important skill for evaluators to

develop.