Jan M.
Noyes and Stella Mills
Heuristic Evaluation of Websites: Lessons Learned
Jan M. Noyes1 and Stella Mills2
1
University of Bristol, UK
2
University of Gloucestershire, UK
Abstract
Heuristic evaluation is a technique that
has been developed over the last 15 years as
a result of the growth and interest in the
usability of systems and products. It is a so-
called ‘discount usability engineering’
method since it comprises a so-called ‘quick
and dirty’ means of evaluation that can be
carried out by informed non-experts. This
paper overviews the use of heuristic
evaluations in usability testing, and presents
two heuristic evaluations of websites carried
out by the authors. From the lessons learned,
a series of recommendations are made in
order to help computer and other
professionals, who need to conduct heuristic
evaluations.
Keywords: Heuristic evaluation, discount
usability engineering, human-computer
interaction, usability, Internet, website
design.
Designing websites needs to be
accompanied by evaluation in order to
ensure systems and products are well
designed for human use. However, the
engineering emphasis traditionally has
concentrated on bringing a product to market
as quickly as possible in order to begin to
realise the costs of development. Economic,
logistic and practical reasons often have
precluded carrying out a full or even part
evaluation. In response to this, the so-called
‘discount usability engineering’ methods
32
have been developed (Nielsen & Mack,
1994). One example of these methods is
heuristic evaluation. “Heuristic evaluation is
an easy to use, easy to learn, discount
usability evaluation technique used to
identify major usability problems of a
product in a timely manner with reasonable
cost” (Zhang, Johnson, Patel, Paige, &
Kubose, 2003, p. 25). Further, it has been
defined as a specialist report method, and is
thought to be one of the most effective in
identifying usability problems (Fu, Salvendy,
& Turley, 2002) and, more recently, the
interactivity of web-based learning systems
(Evans & Sabry, 2003). Heuristic evaluation
provides the focus of the current paper,
which aims to introduce the method as a
useful tool for evaluating websites. The
various steps of undertaking an heuristic
analysis are given, together with a discussion
of these in the light of two website
evaluations; issues raised are also considered
at the end of each step.
What is Heuristic Evaluation?
An heuristic evaluation is essentially a
systematic inspection of a user interface by
having a small set of evaluators examine an
interface and judge its compliance with
recognised usability principles (Nielsen &
Molich, 1990). The overall aim of an
heuristic evaluation is to identify problems
with the user interface. Besides finding
problems, an heuristic evaluation helps the
designer to prioritize further refinement.
Although this type of evaluation does not
 Heuristic Evaluation of Websites: Lessons Learned

provide explicit technical solutions, often the followed by a discussion of the lessons
act of identification of problems leads to learnt.
suggestions concerning their rectification.
The focus of an heuristic evaluation is Step 1. Defining the Heuristics
usually on the whole of the design rather The heuristics are features relating to the
than on specific aspects (Lewis & Wharton, usability of the interface that will have been
1997). It is carried out as part of the overall determined prior to the evaluation. Two
iterative design process, and typically examples of heuristics might be
involves a small group of people acting as compatibility, i.e. the operation of the
evaluators and examining the interface in interface should be in keeping with the
order to check the extent to which it meets expectations of the user from using similar
recognised design principles, i.e. the interfaces, and reversibility, i.e. users have
heuristics. These heuristics are often in the the opportunity to ‘undo’ their actions.
form of guidelines. Given that Smith and Mosier (1986)
Heuristic evaluation was developed as a generated 944 guidelines for user-system
‘discount usability engineering’ method, i.e. interaction, a large number of heuristics are
“an easy and fast method of finding potential possible for some interfaces. One of the
problems” (Nielsen & Mack, 1994, p. 25). initial problems, therefore, when carrying
This is an important point since many studies out an heuristic evaluation is to determine
have attempted to compare heuristic which heuristics to use.
evaluation and empirical testing methods. A systematic attempt at defining
The results have been mixed with some appropriate heuristics was carried out by
researchers, e.g. Jeffries, Miller, Wharton, Nielsen (1994). He put forward a set of nine
and Uyeda (1991) and Virzi, Sorce, and heuristics which were derived from a factor
Herbert (1993) finding that the heuristic analysis of 11 studies covering 249 usability
method will identify the largest number of problems. These were: visibility of system
usability problems, while others, e.g. status, match between system and the real
Desurvire, Lawrence, and Atwood (1991) world, user control and freedom, consistency
and Karat, Campbell, and Fiegel (1992) and standards, error prevention, recognition
indicating that empirical testing identified rather than recall, flexibility and efficiency
more problems. The difficulty here might lie of use, aesthetic and minimalist design, and
in distinguishing the severity and importance helping users recognize, diagnose and
of the usability problems identified recover from errors. A tenth heuristic, help
compared with their actual number. and documentation, was added later to this
Measurements have tended to focus on list by Nielsen, while Muller, McClard, Bell,
quantity, whereas the type of usability Dooley, Meiskey, Meskill, Sparks, and
problem may be more important. Kamper Tellam (1995) extended this original list to
(2002) concluded that heuristic evaluation is include three more heuristics. These were:
more appropriate when used earlier in the respect the user and his/her skills, provide a
design process, whereas empirical testing is pleasurable experience with the system, and
of more use later on. This may help explain support quality work. Although it is
some of these earlier findings of mixed unrealistic to expect all these heuristics to
success. relate to every interface, they do provide a
We turn now to the steps in the process starting point when carrying out an heuristic
of conducting an heuristic evaluation; each evaluation.
step is then illustrated with our experiences, User characteristics and the context of
use are important when defining heuristics.

International Journal of The Computer, the Internet and Management Vol. 14.No.1 (January-April, 2006) pp 32- 49

33
Jan M. Noyes and Stella Mills

Clearly, if the system is to be used by Evaluating websites: Our experiences

novices then the heuristics need to reflect
their needs so that, for example, a minimalist
style of interface may not be suitable.
Context of use needs also to be ascertained.
If the system is to be used in a safety critical
situation, for example, then the interface
must reflect good practice in ensuring
cognitive efficiency. In other words, the
more specific the heuristics can be, the more
closely the evaluation will mimic the actual
working situation and, consequently, the
system is more likely to satisfy the users’
needs in achieving their goals.
Given the numerous generic evaluation
principles available, ranging from those
directly related to system design (e.g.
Nielsen & Mack, 1994) to those based on
psychological principles (e.g. Gardiner &
Christie, 1987), it is necessary to choose the
heuristics carefully. Following a typical user-
centred design approach, a good starting
point is to consider the users’ characteristics
and the context of use of the system. The
principles given in Table 1 have been
derived previously from the literature and
used effectively by one of the authors in an
evaluation context.

Insert Table 1 about here.

Table 1

Some Generic Principles for Website Evaluation

1. The presentation is of great importance when decision making is necessary from the
interpretation of the information displayed (Mills, 1995).
2. Information must be within the user’s capability to comprehend instantly and
simultaneously (Mills, 1995).
3. Information supplied should be easy to receive and absorb and clear in all respects
(Mills, 1995).
4. Operational errors should be rectifiable without consequence (Mills, 1995).
5. Users should be able to master the system and use it effectively (Mills, 1995).
6. Information should be consistently displayed (Nielsen & Mack, 1994; Parlangeli,
Marchigiani, & Bagnara, 1999).
7. Clearly marked exits should be available at all times (Parlangeli et al., 1999).
8. The design style of the interface should reflect the users’ characteristics (Mills &
Araújo, 1999).

Charlton, Gittings, Leng, Little, and
Neilson (1999) refuted the claim that the
web provides a simple graphical interface
that is usable by anyone by arguing that
usability is more than utilising a set of skills
possessed by the user. They admit, however,
that many web users already possess skills
that enable them to navigate satisfactorily
through menu structures, icons and browsers.
Even so, in evaluating websites, it is
necessary to supplement generic principles,
such as those given above in Table 1, with
more specific guidelines. It must be pointed
out, however, that guidelines are by no
means unanimously agreed by evaluators
since some of them have yet to be tested
fully. Even so, it is possible to be specific in
some cases and to give some guidance as to
what constitutes a more usable interface.
Table 2 shows a list of ergonomic principles
derived from the literature.

34
 Heuristic Evaluation of Websites: Lessons Learned

Table 2

Ergonomic Principles for Website Evaluation

1. Text should always be in a style that is understandable without being patronising (Noyes
& Mills, 1998).
2. Text should not use a variety of fonts unless a special effect is required (Noyes & Mills,
1998).
3. Upper case letters should only be used for the first letter of words, normally only at the
beginning of a sentence, the exception being some short words used in warnings (Noyes
& Mills, 1998).
4. The minimum number of colours should be used to gain the desired effect (Noyes &
Mills, 1998).
5. The use of colours should reflect the time the user spends viewing a particular page
(Shneiderman, 1998).
6. The various facets of each page should be clear instantaneously to the user; these include
navigation aids, exit points and links to other pages (Shneiderman, 1998).
7. Navigation support should clearly indicate the user’s progress through the system
(Shneiderman, 1998).
8. Metaphors should be life-like and instantaneously recognisable by the user (Noyes &
Mills, 1998).
9. The software must accommodate the technical limitations of the hardware (Mills &
Araújo, 1998).
10. Sound should always match the user’s expectation (Noyes & Mills, 1998).
11. Sound should always be distinct from any background noise (Noyes & Mills, 1998).
12. Any sound should only provide the minimum information for comprehension (Noyes &
Mills, 1998)
13. Music can be used to communicate the flow of time and emotion (Robertson, 1996).
14. Animation should be used purposefully (Shneiderman, 1998).
15. Graphics should be used purposefully (Noyes & Mills, 1998).
16. No more than four different graphics should be used (Shneiderman, 1998).
17. Video should be used in short sequences to illustrate static procedures (Shneiderman,
1998).

Lessons learned themselves for easy general application.

In applying the heuristics, it is often
necessary to derive the set from first
principles, depending on the users and the
purpose of the software. However, in this
case, we have been able to use generic
questions which cover the main design areas
of websites and the user groups will be
ubiquitous due to the nature of the Internet.
Thus, given that businesses want to target the
general populace and so must be designed
for the ‘average’ web user (whoever that
person may be!), the heuristics lend
They have, in fact, been used profitably to
evaluate a number of different websites and
have yielded design modifications that have
improved the accessibility and general usage
of the websites. But it should be stressed that
beyond the rather specialised type of
software that a website is (after all, most
websites are used primarily for information
seeking and purchasing and do not have
much data input from the user, for example),
it may be necessary at least to fine tune the
heuristics for the software under evaluation.

International Journal of The Computer, the Internet and Management Vol. 14.No.1 (January-April, 2006) pp 32- 49

35
Jan M. Noyes and Stella Mills
From our experience, when deriving
heuristics, it is better to have a small number
of, say up to 10, categories and then, if
necessary, include sub-heuristics under each
category. While this may lengthen the list of
heuristics, it focuses the evaluator on a
particular evaluation section and allows a
more in-depth evaluation to be achieved
more quickly. This is especially
advantageous if the system is to be used by a
variety of users as it allows the evaluator to
examine each part of the system for relevant
potential problems for each user category.
In using the heuristics, it is sometimes
necessary to ignore sections that do not
apply. For example, a website may not have
any sound associated with it. However, it is
not recommended that such sections be
removed as the presence of these sections
reminds the designer that future
modifications to the website may include
these other media. Also, if these heuristics
are used for comparisons, particularly of
competitors’ websites, then it is much easier
to compare the design features and their
characteristics. Interestingly, it has been our
experience that few companies have thought
about the website’s design giving
competitive advantage as most companies
seem still to be focussed on their own sites.
Step 2. Identifying who should carry out the
Heuristic Evaluation
A further issue relates to the people
carrying out the heuristic evaluation. There
is a need for someone to define the heuristics
as well as having someone act as evaluator
and actually carry out the practical
evaluation. The 10 heuristics defined by
Nielsen (1994) require the evaluators to
know the meaning of the terms both from a
human-computer interaction (HCI)
viewpoint and domain area perspective. This
often means that the evaluators need to be
‘double experts’ – experts in HCI and the
domain area (Nielsen, 1992). It is generally
accepted that the person defining the
36
heuristics is a usability expert, but the
situation with regard to the evaluators of the
interface is not so clear. In some situations, it
is quite likely that the person who
determines the heuristics (the ‘defining
evaluator’) could assume the role of
evaluator (the ‘user evaluator’), but it may be
more appropriate to have ‘real users’
evaluate the interface.
Nielsen (1992) considered this issue in a
number of experimental studies. He defined
three groups of user evaluators: (i) ‘novices’
who knew about computers, but with no
specialist usability expertise; (ii) ‘single
experts’ who were usability experts, but with
no knowledge of the system interface, and
(iii) ‘double experts’ who had knowledge
about both the system and usability
attributes. Nielsen found there was a
systematic difference in the performance of
the evaluators with the double experts
locating 60% of the usability problems,
single experts finding 41%, and the novices
an average of 22%. An heuristic evaluation
is likely to be more successful when carried
out by individuals with knowledge of
usability engineering and system knowledge.
However, we would argue beyond this and
for evaluators to have domain knowledge
(i.e. be ‘triple’ experts). This would allow
comparison within the field of application to
be made intrinsically, thus leading to amore
knowledgeable evaluation. This is
particularly appropriate for specialist
domains such as sports or project
management software.
The number of evaluators also needs to
be determined. It is possible to conduct an
heuristic evaluation with only one individual,
who acts as both the defining and user
evaluator. However, the obvious difficulty is
that a single evaluator may not be
particularly good at evaluating the interface,
because of a limited knowledge base. This
was shown by Nielsen (1992) in an analysis
of six projects. He found that one evaluator
only located 35% of the usability problems

associated with the interface while five
evaluators found 75% of the usability
problems. Increasing the number of
evaluators to 15 resulted in 90% of the
problems being identified. Work by Virzi
(1992) also supported this finding in that
four to five evaluators were found to
highlight 80% of the usability problems.
However, adding in more evaluators will
inevitably lead to more problems being
uncovered.
In conclusion, the optimum number of
evaluators is probably between three and
five. Some support for this comes from a
recent study evaluating ambient displays
where three to five evaluators were able to
identify between 40 to 60% of the usability
problems (Mankoff, Dey, Hsieh, Kientz,
Lederer, & Ames, 2003). However, the final
decision needs to be made according to the
specific characteristics of the interface under
evaluation. If, for example, there are health
and safety considerations, and the usability
of the interface is critical to the safe
functioning of the system, more evaluators
should be used in order to ensure that the
majority of the problems can be located
before the system becomes operational. If,
on the other hand, the interface is for a
website, where not uncovering all the
usability problems will result in no more
than a mild irritation for some users, then the
number of evaluators could be considerably
reduced.
Lessons learned
Having stated that the optimum number
of evaluators lies between three and five, we
have successfully carried out heuristic
evaluations with two evaluators. However,
both these individuals were human factors
specialists and had system knowledge, i.e.
so-called ‘double experts’.
User characteristics and context of use
are vital considerations. If the system is to be
used by a variety of users, the evaluator will
need to examine each section of the system
International Journal of The Computer, the Internet and Management Vol. 14.No.1 (January-April, 2006) pp 32- 49
37
Heuristic Evaluation of Websites: Lessons Learned
for relevant potential problems for each user
category. This assumes, of course, that the
evaluators have sufficient skill and
knowledge to adapt their evaluating to
different cognitive scenarios for role-playing
the different users’ needs and goals. It
supports using double experts. This may
increase the overall costs of the evaluation,
but the quality of the evaluation should
reflect this investment.
Step 3. Conducting the Heuristic Evaluation
A number of different techniques can be
used for the practical evaluation of the
interface. However, the basic premises are
the same. For example, the actual heuristic
evaluation is carried out by each individual
evaluating the interface usually alone and in
their own time. It is implicit in the technique
that the evaluator uses the interface, even if
only a mock-up is available. Ideally, most
interfaces being subject to an heuristic
evaluation would be close to product release,
i.e. an operational interface is present, but
there is an opportunity to make changes
(Fleming, 1998). However, given that the
evaluators in an heuristic evaluation need not
use the interface to perform a task as in the
real world, it is possible to carry out an
evaluation using only ‘pen and paper’.
Although there is a loss of ecological
validity in doing this, it might be appropriate
for systems that are not yet developed, and
this also allows for heuristic evaluations to
be carried out early in the life cycle. In the
simplest form of an heuristic evaluation, the
user evaluator could merely study the
interface without carrying out any pre-
described tasks. In these cases, often a
‘walkthrough’ technique is used (see Lewis,
Polson, Wharton & Rieman, 1990), where
the evaluator ‘walks through’ typical tasks
which would be carried out using the
interface.
A variation on this is a cognitive
walkthrough where the focus of the tasks is
on the cognitive aspects of the user-system
Jan M. Noyes and Stella Mills
interaction (Lewis & Wharton, 1997). The
benefit of a walkthrough is that it allows
using the interface to be simulated, and
specific tasks to be considered. Although
Lewis and Wharton (p. 718) argued that
“heuristic evaluation provides an overview
of a complete design rather than a detailed
examination of particular tasks”
incorporating a cognitive walkthrough into
the evaluation will allow a focus on specific
tasks.
There are various modifications of the
walkthrough technique, e.g. the
programming walkthrough (Bell, Rieman, &
Lewis, 1991) and the cognitive jogthrough
(Rowley & Rhoades, 1992). In the
jogthrough technique, the evaluation session
is recorded on video tape allowing the
evaluation (but not the test) to be conducted
in a much shorter time. Lewis and Wharton
Table 3a
Website Evaluation of an Agricultural Charity
Criterion
Fitness for Purpose
Integrity of information
Is the information reliable?
Is the information consistent with similar
information found elsewhere?
Identify users’ characteristics
Have the potential users experience of computers?
Have the potential users experience of website
exploration?
For this evaluation, are the users considered to be
browsers or searching with fully specified goals?
Is the website intended for specific age groups
such as children?
Identify users’ goals
Are the users’ goals consistent with their
characteristics?
System functionality
Is the information available within the website
specification consistent with the users’ goals?
38
reported that about three times more
problems could be found in the same amount
of evaluation time by adopting the
jogthrough technique. Finally, an extension
of the basic walkthough is the pluralistic
walkthrough (Bias, 1994). In this type of
walkthrough, the practical evaluation is
carried out by a number of different groups
of individuals who have been involved in the
system development, e.g. usability experts,
users and the system developers.
Our experiences
The heuristics were applied as a
checklist to two UK websites, one for an
agricultural charity and the other a small
wine producing company. Tables 3a and 3b
summarize the results of applying the
questions relating to the four areas for
evaluation.
Comments
Yes, but rather dated
Yes, as much as I could judge
This is assumed
Yes
Both were role-played
It appeared that adults using the
site for serious information were
envisaged as the users
Yes
 Heuristic Evaluation of Websites: Lessons Learned

Is the content of the website (or at least the main Yes, but rather thin and dated
links) clearly visible on entering?

Overall Design
Decision making Yes, but first impressions are
Is decision making necessary from the rather clouded by the colour
interpretation of the information displayed? combination and the large
amount of information and
choice
Is the design sufficiently clear to allow a fully
informed decision?
Information comprehension
Is the information comprehensible instantly and
simultaneously? Yes, the choices weren’t
immediately obvious

Clarity of information
Is the information supplied easily absorbed? Yes, after thought

Is the information clear in all respects? No – initially the site appears

cluttered with buttons too small
for quick comprehension
Rectification of operational errors
Are operational errors rectifiable without Yes, but at times the style is
consequence? rather emotional and does not,
therefore, give the impression of
scholarly independence (e.g.
articles by Brian Dobby). Also,
some articles are long for
reading especially with such a
small font but printing is easily
achieved
Are error messages comprehensible
instantaneously? No – section buttons are too
small and some links are not
obvious

Are fatal errors prevented? I found no errors as such

Effective mastery of system N/A

Can users master the system and use it effectively? N/A

Consistency
Is information consistently displayed throughout
the website? Yes, if they are experienced

Clearly marked exits

Are clearly marked exits available at all times? More or less but there were
some different routes to the
same article such as rural

International Journal of The Computer, the Internet and Management Vol. 14.No.1 (January-April, 2006) pp 32- 49

39
Jan M. Noyes and Stella Mills

theology arriving at redefining

sustainability
Style of interface
Does the design style of the interface reflect the The homepage is linked at the
users’ characteristics? top but not overly obvious
Some thought was needed. The
colour scheme of green and
yellow is rather eye straining
although the idea is good.
Perhaps green and white is a
better combination

If applicable, does the design style cater for any Definitely adults and probably
specified age groups? educated viewers searching for
information

Interface/page design
General aspects
Use of text
Is the text understandable instantaneously? Yes
Is the text free from any patronising of the user?
Are the number of different text fonts limited to Yes but at times rather
two? emotional
Are upper case letters only used for the first letter
of words? Yes
Use of colour
Is the minimum number of colours used to gain the Yes
desired effect?

Are soft colours used on pages that require the
user to read sections of text?
No – this needs rectification as
they were too bright
Black and white for text is good
as is white/green but why red
headings?

Clarity
Are the various facets, such as navigation aids, No – there are too many
exit points and links to other pages, of each page headings, boxes etc on the
clear instantaneously to the user? homepage. The search area is
too small
Navigation
Is the user’s progress through the system clearly
indicated? Not at all, only through the
browser facilities

Is the number of links to other pages adequate but
not excessive?
Probably yes but there are
various paths to the same pages.
New links text is very small and
there are links to nowhere such
as briefing papers

40
 Heuristic Evaluation of Websites: Lessons Learned

Is the website broad and shallow in terms of

navigation levels? Yes

Metaphors
Are metaphors life-like and instantaneously The search engine ‘go’ button
recognisable by the user? has to be pressed for bulletin
board queries. Pull-down box is
below and then need to click
‘search’
Use of multimedia facets
Does the software adequately accommodate the Yes except possibly for frames
technical limitations of the hardware? if used

Sound N/A
Are the sounds used those that the user would
expect in a given situation?
Are all sounds distinct from any background
noise?
Do all sounds provide the minimum information
for comprehension?
If music is used, does it communicate the flow of
time?
Is music used to create emotion?

Animation N/A
Is animation used to increase the user’s
enjoyment?
Is animation used to aid icon understanding?

Graphics
Are all graphics used purposefully? Pictures on homepage could be
made active as icons as on the
site map. This leads to an
inconsistency at present
Are no more than four different graphics used on
any one website? There are more than four photos
on the homepage
Video
Is video used in short sequences to illustrate static
procedures? N/A

International Journal of The Computer, the Internet and Management Vol. 14.No.1 (January-April, 2006) pp 32- 49

41
Jan M. Noyes and Stella Mills

Table 3b

An Evaluation of a Wine Producers’ Website

Fitness for Purpose

Integrity of information
Is the information reliable?
Yes – this is assumed.
Is the information consistent with similar information found elsewhere?
N/A
Identify users’ characteristics
Have the potential users experience of computers?
No evidence of users being consulted
Have the potential users experience of website exploration?
Possibly
For this evaluation, are the users considered to be browsers or searching with fully
specified goals?
Both possibly.
Is the website intended for specific age groups such as children?
Adults.
Identify users’ goals
Are the users’ goals consistent with their characteristics?
Goals not clearly indicated
System functionality
Is the information available within the website specification consistent with the users’
goals?
Yes probably.
Is the content of the website (or at least the main links) clearly visible on entering?
Yes but there is little leading of the user to the goals

Overall Design
Decision making
Is decision making necessary from the interpretation of the information displayed?
Yes.
Is the design sufficiently clear to allow a fully informed decision?
No - the information is far too cluttered. Scrolling is used.
Information comprehension
Is the information comprehensible instantly and simultaneously?
Yes – but it is difficult to find
Clarity of information
Is the information supplied easily absorbed?
Possibly - but the design is too cluttered. The user is given far too much choice without
clarity. The navigation does not lead the user or help.
Is the information clear in all respects?
Yes - except for too much text.
Rectification of operational errors
Are operational errors rectifiable without consequence?

42
 Heuristic Evaluation of Websites: Lessons Learned

Yes
Are error messages comprehensible instantaneously?
None found
Are fatal errors prevented?
None found
Note all error messages are browser dependent; the system appears not to have its own
error messages.
Effective mastery of system
Can users master the system and use it effectively?
Yes but not very effectively. There is far too much information almost ‘thrown’ at the
user. Pictures and colours are used indiscriminately.
Consistency
Is information consistently displayed throughout the website?
Probably
Clearly marked exits
Are clearly marked exits available at all times?
No – some links are not clear.
Style of interface
Does the design style of the interface reflect the users’ characteristics?
Only superficially through the use of pictures.
If applicable, does the design style cater for any specified age groups?
Yes – adult

Interface/page design
General aspects
Use of text
Is the text understandable instantaneously?
Yes
Is the text free from any patronising of the user?
Yes
Are the number of different text fonts limited to two?
Yes
Are upper case letters only used for the first letter of words?
Yes
Use of colour
Is the minimum number of colours used to gain the desired effect?
No – colours have no obvious metaphor and are used indiscriminately
Are soft colours used on pages that require the user to read sections of text?
Yes – text mirrors WORD conventions
Clarity
Are the various facets, such as navigation aids, exit points and links to other pages, of
each page clear instantaneously to the user?
Not really – there are too much text and graphics
Navigation
Is the user’s progress through the system clearly indicated?
No – the left bar helps and at times the upper bar too but no structure becomes
evident.

International Journal of The Computer, the Internet and Management Vol. 14.No.1 (January-April, 2006) pp 32- 49

43
Jan M. Noyes and Stella Mills

Is the number of links to other pages adequate but not excessive?

Rather too many
Is the website broad and shallow in terms of navigation levels?
No – it appears square.
Metaphors
Are metaphors life-like and instantaneously recognisable by the user?
I wasn’t aware of any metaphors really

Use of multimedia facets

Does the software adequately accommodate the technical limitations the hardware?
Yes - although the machine used in the evaluation was of a good specification.
Sound
Are the sounds used those that the user would expect in a given situation?
N/A
Are all sounds distinct from any background noise?
N/A
Do all sounds provide the minimum information for comprehension?
N/A
If music is used, does it communicate the flow of time?
N/A
Is music used to create emotion?
N/A
Animation
Is animation used to increase the user’s enjoyment?
N/A
Is animation used to aid icon understanding?
No - but icon meaning is generally clear.
Graphics
Are all graphics used purposefully?
No - some pictures seem to serve little purpose and add to the clutter
Are no more than four different graphics used on any one website?
Yes - graphics are restricted to pictures.
Video
Is video used in short sequences to illustrate static procedures?
N/A

Lessons learned problems in navigation, user confusion, use

As with any heuristic evaluation, there
was an element of role-playing; in both
cases, the evaluator was a human factors
expert with previous experience of heuristic
evaluation and so was able to identify
possible problems from both naïve and
experienced users. In addition, the
evaluator’s knowledge of interface design
was essential in order to identify potential
of colour and other facets important
particularly in website design. Domain
knowledge and practical knowledge of
targeted users also facilitated the evaluation.
Step 4: Recording and Presenting the Data
The exact way in which the heuristics
are checked against the design needs to be
determined. A checklist approach could be

44

used or more detailed answers elicited. Some
evaluations might use a combination of
approaches. Recording of this information is
usually carried out by the evaluator, although
some heuristic evaluations have an
independent observer who records comments
being made by the evaluator (Nielsen, 2003).
The advantage of this is that the evaluator
does not lose track of the procedure whilst
having to stop to make notes; the
disadvantage is inaccurate reporting by the
observer who misinterprets the evaluator’s
responses.
Lessons learned
For the evaluation, the heuristics were
listed as questions rather than stated design
principles; this is rather akin to research aims
being re-phrased into research questions and
has been found easier to administer. The
questions are shown in Table 3. Questions
also allow for a clearer categorization of
answer, as this can be recorded simply as
‘yes’, ‘no’, or ‘maybe’, following
suggestions used in Usability Context
Analysis quality documentation (Bowden,
Thomas, & Bevan, 1995), where a minimal,
positive answer, essentially ‘yes’, will
indicate the compliance of the website with
the associated principle. Conversely, a
negative answer indicates non-compliance.
Thus, a basic usability measure is possible
by simply counting the positive responses to
the applicable questions. In essence, it is a
simplified Nielsen (2003) scale. If a tabular
format of recording the results of the
heuristic evaluation is preserved, then a third
column can be used for comments and
suggesting solutions. Alternatively, these can
be discussed in a separate document whilst
referring to the table; this is helpful if the
problems are numerous and need fairly
drastic remedies. In business terms, this
tabular approach may be sufficient but we
have found that some companies prefer to
have a prose document highlighting the main
findings; this gives good scope for
International Journal of The Computer, the Internet and Management Vol. 14.No.1 (January-April, 2006) pp 32- 49
45
Heuristic Evaluation of Websites: Lessons Learned
suggesting solutions and wider connected
issues.
It will be noted that Tables 3a and 3b
both use the same heuristics but the results
are presented slightly differently; the first is
in true tabular form while the second is more
as a list of questions and answers. In the
latter, bold type has been used to highlight
the answers. We are content working with
either format and we use whichever one the
client prefers. However, this does emphasize
that there is not a generally accepted way of
presenting the data from an heuristic analysis
and we would favour any presentation style,
which gives clarity, while highlighting issues
of concern. Severity ratings (see below) were
not used formally but in the accompanying
report for both businesses, emphasis was
given to those factors, which may cause a
user to give up or move to another site.
Step 5: Assessing the Severity of the
Usability of the Websites
A further step in the heuristic evaluation
is for the evaluators to rate all of the
problems identified. As can be seen from the
example given, this can be done verbally in
the ‘comments’ column or quantitatively
using the following 5-point scale: ‘0 = I do
not agree that this is a usability problem’; ‘1
= Cosmetic problem only: need not be fixed
unless extra time is available’; ‘2 = Minor
usability problem: fixing this should be
given low priority’; ‘3 = Major usability
problem: important to fix, so should be given
high priority’; ‘4 = Usability catastrophe:
imperative to fix this before real users use
the interface’ (Nielsen, 2003). Compilation
of this data will indicate the order in which
usability problems need to be addressed and
can also be used to provide a crude usability
rating for the system.
Our experiences
In evaluating websites for a number of
clients in industry, it has been our experience
to be asked to evaluate sites which are
Jan M. Noyes and Stella Mills
already live. Often managers are aware that
all is not well with the website but they are
unable to technically define the problems. In
such circumstances, we have found that it is
much better to use a verbal system of rating,
as has been given in the examples (Tables 3a
and 3b) rather than try to quantify the
problems in a rather subjective hierarchy.
Managers generally want the website to be
usable in all aspects and so it has been our
experience to list all problems that detract
from the objectives of the website as all
these need to be fixed for the user to move
freely within the website. However, should
the website be evaluated heuristically in its
early stages, then some quantitative ranking
of the problems along the lines of Nielsen’s
(2003) hierarchy above could be useful.
Lessons learned
While the quantitative ranking of
problems suggested by Nielsen (2003) may
be useful in systems development, it has
been our experience that for live websites of
businesses, it can be ‘overkill’, adding little
to a more verbal approach. Managers want to
know what must be done to fix problems
which they can perceive but may not be in a
position to suggest viable solutions. Thus, in
ranking usability problems, it is often the
solutions which are more important; for
example, if a problem has no solution within
a budget, then practically, it has to remain,
regardless of the consequences for the users.
Issues and recommendations
Issue 1. Defining the heuristics
The primary consideration is to have
someone define the heuristics who knows
about interface design and the principles of
good design from a user perspective; in
addition, domain knowledge is helpful as
this contextualizes the system and should
lead to more user-centred and specialised
heuristics. It can be argued that knowledge
of the system under evaluation is also
46
advantageous, but if the end-users of the
system will not be familiar with a previous
version then system knowledge may hinder
the defining evaluator’s derivation of the
heuristics.
Recommendation 1: The evaluator has
computing and human factors knowledge of
interface design and domain knowledge (i.e.
is a ‘triple’ expert).
It is important that the heuristics reflect
the user categories as well as the context of
use in terms of any special target users. In
website design, for example, most businesses
have a target audience for their products and
it is likely that these people will also visit the
website. Consequently, it is important that
the design of the website reflects the needs
of these users as otherwise they may be
alienated from using the site. In addition, the
company may want to attract a different
target group and the needs of these users also
must be accommodated in the design. This
can lead to conflict and there is no easy
design solution. Heuristic evaluation can,
however, be instrumental in identifying the
design features that conflict; e.g. animation
may be useful for attracting young people
but may not be desirable for the older
generation (Greenwood, 2000).
Recommendation 2: Heuristics need to
reflect user characteristics and context of
use.
Issue 2. Identifying the evaluators
Generally it has been found easy to
apply the heuristics, but this has been done
by experienced human factors evaluators and
it is hard to see how an inexperienced user
without any appropriate knowledge or
training could find the same type of design
problems. Indeed, the fact that after design,
such problems have been found, suggests the
need for the human factor expert’s
evaluation. We would argue that a single
expert evaluator is more beneficial than a
larger number of inexperienced evaluators

since it is impossible for novice users to role-
play experienced users. However, it is not
difficult for experienced users to role-play
novice users. In addition, the use of one
expert evaluator is much easier and cheaper
strategically and operationally.
Recommendation 3: Single evaluators,
expert in computing, human factors and
domain knowledge are likely to be more
beneficial than multiple novice users.
As with any evaluation, the personal
bias of the evaluator plays a role in
identifying the areas of concern. In this
sense, questions perhaps ease the situation
by narrowing the scope of answers, whereas
guidelines or traditional heuristics may allow
a wider answer. But the problem of bias is
one that is inherent in any evaluation and at
least it is minimalized through using a
generically derived table.
Recommendation 4: Bias should be reduced
by using generically derived heuristics.
Issue 3. Carrying out the evaluation
The evaluator works through a series of
typical tasks using the interface in order to
compare the attributes of the system with a
list of recognised usability principles.
Recording of the results is usually carried
out by the evaluator. At the end of the
practical evaluation, the results are compiled
to gauge the extent to which the interface
meets these pre-determined usability criteria.
If a number of evaluators have been
employed, it is important that they do not
confer since this may introduce bias.
Recommendation 5: The format for
recording the results of the evaluation will
be determined by the nature of the heuristic
evaluation and the skills of the evaluator(s).
Recommendation 6: There are benefits to be
attained from having an independent
observer record the results.
International Journal of The Computer, the Internet and Management Vol. 14.No.1 (January-April, 2006) pp 32- 49
47
Heuristic Evaluation of Websites: Lessons Learned
Issue 4. Severity Ratings
The calculation of severity ratings helps
assess the extent of the problems associated
with the usability of the interface. It also
allows comparisons to be made between
interfaces.
Recommendation 7: Severity ratings are
useful in that they provide a ranking of the
problems found; however, for managerial
use, this may be more useful in a verbal form
than a quantitative ranking.
Issue 5. Further analyses
Nielsen (1993) suggested that it is useful
to hold a debriefing session at the end of the
evaluation in order to allow the good aspects
of the interface to be highlighted and to
begin to give some consideration to how
usability issues can be addressed. The end-
result of the practical evaluation is usually a
list of usability problems that need to be
rectified before product release. This is
useful in the development stages of the
website but, for live websites, a written
report can give an overview and context of
the problems, together with possible
solutions. Such a report can be passed
around management for action as required.
There should be further studies to ascertain
that the problems have been fixed.
Recommendation 8: If the resource is
available, further confirmatory and
verification exercises will be beneficial.
Conclusion
In this paper, we have discussed our
experiences of using heuristic evaluations for
two websites which are live and whose
managers were concerned about usability
problems in the sites. These experiences,
together with other heuristic evaluations not
detailed formally here, suggest that heuristic
evaluation can be an inexpensive and useful
method of finding many of the problems that
users encounter. Clearly, this is part way to a
solution to each problem so that the
Jan M. Noyes and Stella Mills
evaluator who can also suggest suitable and
practical solutions can add value to the
design of the website. This is beneficial since
more usable websites may mean more profit
and success for the business.
References
Bell, B., Rieman, J., & Lewis, C. (1991).
“Usability testing of a graphical
programming system: Things we missed
in a programming walkthrough”.
Proceedings of CHI ‘91 (New Orleans,
LA) (pp. 7-12). New York: ACM.
Bias, R. G. (1994). “The pluralistic usability
walkthrough: Coordinated empathies”. In
J. Nielsen & R. L. Mack (Eds.), Usability
inspection methods (pp. 63-76). New
York: Wiley.
Bowden, R., Thomas, C., & Bevan, N. (Eds.)
(1995). Usability context analysis: A
practical guide. Teddington, UK:
National Physical Laboratory.
Charlton, C., Gittings, C., Leng, P., Little, J.,
& Neilson, I. (1999). “Bringing the
internet to the community”. Interacting
with Computers, 12(1), 51-61.
Desurvire, H., Lawrence, D., & Atwood, M.
(1991). “Empiricism versus judgement:
Comparing user interface evaluation
methods on a new telephone-based
interface”. SIGCHI Bulletin, 23, 58-59.
Evans, C. & Sabry, K. (2003). “Evaluation
of the interactivity of web-based learning
systems: Principles and process”.
Innovations in Education and Teaching
International, 40(1), 89-99.
Fleming, J. (1998). User Testing: How to
Find Out What Users Want [Online].
Available:
http://www.ahref.com/guides/design/199
806/0615jef.html
Fu, L., Salvendy, G., & Turley, L. (2002).
“Effectiveness of user testing and
heuristic evaluation as a function of
performance classification”. Behaviour
48
and Information Technology, 21(2), 137-
143.
Gardiner, M. M. & Christie, B. (1987).
Applying cognitive psychology to user-
interface design. Chichester: Wiley.
Greenwood, P. J. (2000). Animation in
electronic commerce web sites: Potential
usefulness to Internet marketers.
Unpublished Master’s Thesis, University
of Gloucestershire, UK.
Jeffries, R., Miller, J. R., Wharton, C., &
Uyeda, K. M. (1991). “User interface
evaluation in the real world: A
comparison of four techniques”.
Proceedings of CHI ’91 Conference on
human factors in computing systems
(New Orleans, LA) (pp. 119-124). New
York: ACM.
Kamper, R. J. (2002). “Extending the
usability of heuristics for design and
evaluation: Lead, follow, and get out of
the way”. International Journal of
Human-Computer Interaction, 14(3&4),
447-462.
Karat, C. M., Campbell, R., & Fiegel, T.
(1992). Comparison of empirical testing
and walkthrough methods in user
interface evaluation. Proceedings of CHI
’92 Conference on human factors in
computing systems (pp. 397-404). New
York: ACM.
Lewis, C. & Wharton, C. (1997). “Cognitive
walkthroughs”. In M. Helander, T. K.
Landauer & P. Prabhu (Eds.), Handbook
of human-computer interaction (2nd Ed.)
(pp. 717-732). North-Holland: Elsevier.
Lewis, C., Polson, P., Wharton, C., &
Rieman, J. (1990). “Testing a
walkthrough methodology for theory-
based design of walk-up-and-use
interfaces”. Proceedings of CHI ’90
Conference on human factors in
computing systems (Seattle, WA) (pp.
235-242). New York: ACM.
Mankoff, J., Dey, A.K., Hsieh, G., Kientz, J.,
Lederer, S., & Ames, M. (2003).
“Heuristic evaluation of ambient

displays”. Proceedings of CHI ’03
Conference on human factors in
computing systems (Fort Lauderdale, FL)
(pp. 169-176). New York: ACM.
Mills, S. (1995). “Usability problems of
acoustical fishing aids”. Displays, 16(3),
115-121.
Mills, S. & Araújo, M. M. T. (1998).
“Learning from virtual reality – a lesson
for a designer”. HCI Letters, 1, 28-31.
Mills, S. & Araújo, M. M. T. (1999).
“Learning through virtual reality: A
preliminary investigation”. Interacting
with Computers, 11, 453-462.
Muller, M. J., McClard, A., Bell, B., Dooley,
S., Meiskey, L., Meskill, J. A., Sparks,
R., & Tellam, D. (1995). “Validating an
extension to participatory heuristic
evaluation: Quality of work and quality
of work life”. Proceedings of CHI ’95
Conference on human factors in
computing systems (pp. 115-116). New
York: ACM.
Nielsen, J. (1992). “Finding usability
problems through heuristic evaluation”.
Proceedings of CHI ’92 Conference on
human factors in computing systems
(Monterey, CA) (pp. 373-380). New
York: ACM.
Nielsen, J. (1993). Usability engineering.
London: AP Professional.
Nielsen, J. (1994). “Enhancing the
explanatory power of usability
heuristics”. Proceedings of CHI ’94
Conference on human factors in
computing systems (Boston, MA) (pp.
152-158). New York: ACM.
Nielsen, J. (2003). [Online]. Available:
http://ww.useit.com/papers/heuristic
Nielsen, J. & Mack, R. L. (1994). Usability
inspection methods. New York: Wiley.
Nielsen, J. & Molich, R. (1990). “Heuristic
evaluation of user interfaces”.
Proceedings of CHI ’90 Conference on
human factors in computing systems
International Journal of The Computer, the Internet and Management Vol. 14.No.1 (January-April, 2006) pp 32- 49
49
Heuristic Evaluation of Websites: Lessons Learned
(Seattle, WA) (pp. 249-256). New York:
ACM.
Noyes, J. M. & Mills, S. (1998). Display
design for human-computer interaction.
Cheltenham: Cheltenham & Gloucester
College of Higher Education.
Parlangeli, O., Marchigiani, E., & Bagnara,
S. (1999). “Multimedia systems in
distance education: Effects of usability
on learning”. Interacting With
Computers, 12(1), 37-49.
Robertson, P. (1996). Music and the mind.
London: Channel 4 Television.
Rowley, D. E. & Rhoades, D. G. (1992).
“The cognitive jogthrough: A fast-paced
user interface evaluation procedure”.
Proceedings of CHI ’92 Conference on
human factors in computing systems
(Monterey, CA) (pp. 389-395). New
York: ACM.
Shneiderman, B. (1998). Designing the user
interface strategies for effective human-
computer interaction (3rd Ed.). Reading,
MA: Addison Wesley Longman.
Smith, S. L. & Mosier, J. N. (1986).
Guidelines for designing user interface
software. Report ESD-TR-86-278, USAF
Electronic Systems Center, Hanscom
AFB, MA.
Virzi, R. A. (1992). “Refining the test phase
of usability evaluation: How many
subjects is enough?” Human Factors, 34,
457-468.
Virzi, R. A., Sorce, J. F., & Herbert, L. B.
(1993). “A comparison of three usability
evaluation methods: heuristic, think-
aloud, and performance testing”.
Proceedings of the Human Factors and
Ergonomics Society 37th Annual Meeting
(pp. 309-313). Santa Monica, CA: The
Human Factors and Ergonomics Society.
Zhang, J., Johnson, T. R., Patel, V. L., Paige,
D. L., & Kubose, T. (2003). “Using
usability heuristics to evaluate safety of
medical devices”. Journal of Biomedical
Informatics, 36, 23-30.