Libri, 2005, vol. 55, pp.

96–121 Copyright Saur 2005

Printed in Germany All rights reserved
Libri
ISSN 0024-2667

Usability Assessment of Academic

Digital Libraries: Effectiveness,
Efficiency, Satisfaction, and Learnability
Judy Jeng
School of Communication, Information, and Library Studies, Rutgers,
The State University of New Jersey, New Brunswick, NJ, U.S.A.

This study is to develop and evaluate methods and in-
struments for assessing the usability of digital libraries. It
discusses the dimensions of usability, what methods have
been applied in evaluating usability of digital libraries, their
applicability, and criteria. It is found in the study that there
exists an interlocking relationship among effectiveness, ef-
ficiency, and satisfaction. It provides operational criteria for
effectiveness, efficiency, satisfaction, and learnability. It dis-
covers users’ criteria on ”ease of use,” ”organization of
information,” ”terminology and labeling,” ”visual attractive-
ness,” and ”mistake recovery.” Common causes of ”user lost-
ness” were found. ”Click cost” was examined.

Introduction Dimensions of usability

The development of digital library has reached a
mature stage. However, evaluation has not kept
pace. This study is to develop and evaluate meth-
ods and instruments for assessing the usability
of digital libraries. Several authors have pointed
out that little work is being done to understand
usability of digital libraries in any context (Bland-
ford, Buchanan, and Jones 2004, 69; Blandford, Stel-
maszewska, and Bryan-Kinns 2001, 181; Borgman
et al. 2000, 229; Theng, Mohd-Nasir, and Thim-
bleby 2000a, 238; Thomas 1998). Blandford and
Buchanan (2002b) call for a need for further work
on methods for analyzing usability, including
an understanding of how to balance rigor, ap-
propriateness of techniques, and practical limi-
tations. Popp (2001) also found that the literature
on usability testing on library Web sites was
small.
Usability is a multidimensional construct that can
be examined from various perspectives. It is also
an elusive concept and is determined by the tasks,
the users, the product, and the environment. We
have seen in the literature that the term usability
has been used broadly and means different things
to different people.
One perspective is to look at usability from
interface effectiveness point of view. This view
makes sense as usability has theoretical base on
human-computer interaction. Back in 1994, Rubin
(1994) alerted that system designers were slow to
respond to the guidelines established from the re-
search in human-computer interaction. Today,
there are many usability studies focusing on in-
terface design. Kim (2002, 26) has found that ”the
difference between interface effectiveness and us-
ability is not clear.” Interface is one of the most im-

Judy Jeng, School of Communication, Information, and Library Studies, Rutgers, The State University of New Jersey,
4 Huntington Street, New Brunswick, NJ 08901-1071, U.S.A. E-mail: judyjeng@scils.rutgers.edu
This paper has received honourable mention in the 2005 Libri Best Student Award Competition.

96

portant aspects of usability as it is the medium that
users communicate and interact with the system.
Usability can also be related to usefulness,
usableness, and ease of use. Gluck (1997) dif-
ferentiates usefulness and usableness. Usableness
refers to functions such as ”Can I turn it on?”
”Can I invoke that function?” Usefulness refers to
functions such as ”Did it really help me?” ”Was it
worth the effort?” Blandford and Buchanan (2003)
also discuss the difference between usefulness and
usableness. Usefulness is generally taken to mean
”supporting the required functionality.” Usable-
ness, at its simplest, means ”can be used.” Pearrow
(2000, 4) argues strongly that ”a Web site that is
not usable is useless.”
Landauer (1995) defines usability as ”ease of
operation” and usefulness as ”serving an intended
purpose” although also comments that the two
are hard to separate in the context of evaluation.
Davis and his colleagues (Davis 1989; Davis, Ba-
gozzi, and Warshaw 1989) also make a distinction
between usefulness and ease of use. In their Tech-
nology Acceptance Model (TAM) they defined
”perceived usefulness” as the extent to which an
information system will enhance a user’s per-
formance and that ”perceived ease of use” as the
extent to which a person believes a system will be
free of effort. Furtado et al. (2003) also consider
usability from ease of use point of view and add
that usability should also include ease of learning.
Grudin (1992), however, considers that usefulness
is the issue of whether the system can be used
to achieve some desired goal and can be broken
down into utility and usability, where utility is
the question of whether the functionality of the
system in principle can do what is needed, and
usability is the question of how well users can use
that functionality.
Shackel (1991, 24) reports that the definition of
usability was probably first attempted by Miller
(1971) in terms of measures for ”ease of use,” and
these were first fully discussed and a detailed for-
mal definition was attempted by Shackel (1981):
the capability in human functional terms to be used easily
and effectively by the specified range of users, given
specified training and user support, to fulfill the specified
range of tasks, within the specified range of environmental
scenarios.
Perhaps the most widely cited definition is the
ones of ISO and Nielsen. The International Stand-
Usability Assessment of Academic Digital Libraries
ards Organization (1994, 10) defines usability as
”the extent to which a product can be used by
specified users to achieve specified goals with
effectiveness, efficiency, and satisfaction in a spe-
cified context of use.” Nielsen (1993) points out that
usability has five attributes: learnability, efficiency,
memorability, error recovery, and satisfaction.
Brinck, Gergle, and Wood (2002) share a similar
perspective that usability is: functionally correct,
efficient to use, easy to learn, easy to remember,
error tolerant, and subjectively pleasing. In ad-
dition, Booth (1989) outlines that usability has
four factors: usefulness, effectiveness (ease of use),
learnability, and attitude (likeability). Shackel
(1986) suggests that usability has the criteria of
effectiveness, learnability, flexibility, and user at-
titude. Hix and Hartson (1993) classify usability
into initial performance, long-term performance,
learnability, retainability, advanced feature usage,
first impression, and long-term user satisfaction.
Hix and Hartson are unique in that they take
one step further to differentiate performance and
satisfaction into initial and long-term measures.
Blandford and Buchanan (2003) gave usableness
the definition of 1) how efficiently and effective-
ly users can achieve their goals with a system,
2) how easily users can learn to use the system
(learnability), 3) how well the system helps the
user avoid making errors, or recover from errors,
4) how much users enjoy working with the system
or whether they find it frustrating, and 5) how
well the system fits within the context in which
it is used. Blandford and Buchanan’s definition
about usableness is similar to those definitions
cited in this paragraph about usability.
It is also worth noting that satisfaction is found
to be the most cited attribute of usability while
usefulness is the one often overlooked (Thomas
1998).
In addition to those views mentioned above,
Gould (1988) defines usability into more com-
ponents, including system performance (reli-
ability, responsiveness), system functions, user
interface, reading materials, language transla-
tion, outreach program, ability for customers to
modify and extend, installation, field mainte-
nance and service-ability, advertising, and sup-
port group users. Rushinek and Rushinek (1986)
found that system responsiveness (response time)
is the most important variable affecting users’
happiness.
97
Judy Jeng
Figure 1. The Four Principle Components in a Human-Ma-
chine System
Karoulis and Pombortsis (2003) suspect that
usability (effectiveness, efficiency, and satisfaction)
and learnability of educational environment are
positively correlated and wonder how far one
affects the other, although they did not carry out
a study to actually examine this possible cor-
relation.
Usability can also be grouped into two large
categories: inherent usability (Kurosu and Kashi-
mura 1995) and apparent usability (Kurosu and
Kashimura 1995; Tractinsky 1997). Inherent us-
ability is mainly related to the functional or dy-
namic part of interface usability. It includes those
attributes focus on how to make the product easy
to understand, easy to learn, efficient to use, less
erroneous, and pleasurable. On the other hand,
apparent usability is more related to the visual
impression of the interface. At times, inherent us-
ability and apparent usability may be contradictory
(Fu 1999). For example, in Web page design,
graphics enhance apparent usability but slows
down the system. Pearrow (2000) also discusses
that form versus function are at the both ends of a
continuum. A truly great Web site should be both
aesthetically pleasing and truly usable.
In terms of usability framework, Thomas
(1998) categories usability attributes into three
categories: outcome, process, and task. We may
apply Thomas’s grouping and categorize usability
criteria as follows: the outcome group includes
effectiveness, efficiency, and satisfaction. The pro-
cess group includes ease of use, interface, learn-
ability, memorability, and error recovery. The task
group includes functionality and compatibility.
Shackel (1991) views usability as the dynamic
98
interplay of four components: user, task, tool, and
environment. This relationship may be illustrated
as shown in Figure 1 which is a modified version
from Shackel (1991, 23), Bennett (1972; 1979) and
Eason (1981).
Usability has user focus. Dumas and Redish
(1993, 4) define usability as ”people who use the
product can do so quickly and easily to accomplish
their task.” Guillemette (1995, 215) refers usability
to ”the degree to which an information system
can be effectively used by target users in the per-
formance of tasks.” Clairmont, Dickstein, and
Mills (1999) also state, ”Usability is the degree
to which a user can successfully learn and use a
product to achieve a goal.”
Usability is different from functionality. Dumas
and Redish (1993) use the videocassette recorder
(VCR) as an example to illustrate the difference
between the two: VCRs may have high func-
tionality (the feature works as it was designed to
work) but they have low usability (people cannot
use them quickly and easily to accomplish their
task).
Usability is also not equivalent to accessibility.
In a nutshell, accessibility involves making Web
site content available to and usable by people with
disabilities.
Table 1 compares various perspectives on the
attributes of usability.
Usability has several aspects, including interface
design, functional design, data and metadata, and
computer systems and networks (Arms 2000). Us-
ability is a property of the total system. All the com-
ponents must work together smoothly to create an
effective and easy-to-use digital library.
Usability can be tackled from various directions.
Blandford and Buchanan (2002a) suggest that us-
ability is technical, cognitive, social, and design-
oriented and it is important to bring these different
perspectives together, to share views, experiences
and insights. Indeed, digital library development
involves interplay between people, organization,
and technology. The usability issue should look at
the system as a whole.
In order to design systems for a variety of users,
Lamb (1995) suggests usability issues should be
extended beyond interface usability to include con-
tent usability, organizational usability, and inter-
organizational usability.
In addition to those views, usability can also
be examined from the perspectives of graphic de-
 Usability Assessment of Academic Digital Libraries

Table 1. Attributes of Usability

Authors Attributes
Booth (1989) usefulness, effectiveness, learnability, attitude
Brinck et al. (2002) functionally correct, efficient to use, easy to learn, easy to remember, error tolerant, and subjectively
pleasing
Clairmont et al. (1999) successfully learn and use a product to achieve a goal
Dumas & Redish (1993) perform tasks quickly and easily
Furtado et al. (2003) ease of use and learning
Gluck (1997) useableness, usefulness
Guillemette (1995) effectively used by target users to perform tasks
Hix & Hartson (1993) initial performance, long-term performance, learnability, retainability, advanced feature usage, first
impression, and long-term user satisfaction
ISO (1994) effectiveness, efficiency, satisfaction
Kengeri et al. (1999) effectiveness, likeability, learnability, usefulness
Kim (2002) interface effectiveness
Nielsen (1993) learnability, efficiency, memorability, errors, satisfaction
Oulanov & Pajarillo (2002) affect, efficiency, control, helpfulness, adaptability
Shackel (1981) ease of use, effectiveness
Shackel (1986, 1991) effectiveness, learnability, flexibility, user attitude

Usability is the broad discipline of applying sound sci-

sign, navigation, and content (Spool et al. 1999).
Turner (2002) categorizes usability into naviga-
tion, page design, content, accessibility, media use,
interactivity, and consistency. Clausen (1999) ad-
vocates that a high quality Web page should have
the criteria of accuracy, authority, currency, rele-
vant links, browsability, navigation, consistency,
use of frames, use of graphics, connectivity, inter-
activity, user-friendliness, and originality.
There are several approaches in evaluating us-
ability. One view is offered by Eberts (1985) who
categorizes those approaches into four: empirical,
anthropomorphic, cognitive, and predictive mod-
eling approaches. The empirical approach empha-
sizes obtaining a representative sample of target
users and analyzes user performance and attitudes
under specified conditions. The anthropomorphic
approach evaluates user-system interaction in com-
parison to effective interpersonal communication.
The cognitive approach applies the theory and
methodology of cognitive psychology to interface
design. The predictive approach uses an analytic
tool which examines or manipulates an abstract
representation of the interface in order to forecast
usability. Blandford et al. (2004), on the other hand,
roughly group usability evaluation into two kinds:
empirical and analytical. Empirical techniques in-
volve testing systems with users, whereas ana-
lytical techniques involve usability personnel
assessing systems using established theories and
methods.
In conclusion, usability can be defined as (Pear-
row 2000, 12):
entific observation, measurement, and design principles
to the creation and maintenance of Web sites in order to
bring about the greatest ease of use, ease of learnability,
amount of usefulness, and least amount of discomfort for
the humans who have to use the system.
Usability evaluation at academic sites
There are a number of methods to evaluate us-
ability. The techniques include formal usability
testing, usability inspection, card sort, category
membership expectation, focus groups, question-
naires, think aloud, analysis of site usage logs,
cognitive walkthrough, heuristic evaluation, claims
analysis, concept-based analysis of surface and
structural misfits (CASSM), paper prototyping,
and field study (Askin 1998; Blandford et al. 2004;
Campbell 2001; Kantner and Rosenbaum 1997;
Keith et al. 2003; Nielsen and Mack 1994; Pearrow
2000; Popp 2001; Rosson and Carroll 2002; Snyder
2003). The areas of digital library usability assess-
ment have covered breadth of coverage, naviga-
tion, functionality, utility, interface, metadata
appropriateness, and awareness of library re-
sources.
Table 2 is a review of usability tests in selected
academic digital libraries. More detailed de-
scriptions and discussions will be available in the
author’s doctoral dissertation.
Although the literature on usability tests in
libraries is still small, interest and involvement in
testing library Web sites grew exponentially, with
an explosion of work done in 1999. The survey of

99
Judy Jeng

Table 2. Usability Assessment at Academic Sites

100

Popp (2001) indicates that there were only two
tests in 1996, but there were 15 in 1997, 36 in 1998,
and jumped to 82 in 1999. The literature review of
this study found that there is continued interest
since 1999. Many usability studies reported in
library literature are published after 2000. It is
clear that libraries are rapidly moving forward in
evaluating the usability of their Web sites.
Usability evaluation model
This paper proposes an evaluation model for as-
sessing usability of academic digital libraries. The
proposed evaluation model applies the definition
of ISO 9241-11 (1994) that examines effectiveness,
efficiency, and satisfaction. In addition, the mod-
el includes learnability (see Figure 2). The ISO
definition (1994, 10) defines usability as ”the
extent to which a product can be used by specified
users to achieve specified goals with effectiveness,
efficiency, and satisfaction in a specified context
of use.” The ISO definition, however, does not ex-
plicitly specify operational criteria on what to
evaluate. Karoulis and Pombortsis (2003) sus-
pect that usability (effectiveness, efficiency, and
satisfaction) and learnability of educational en-
vironment are positively correlated and wonder
how far one affects the other. Karoulis and
Pombortsis never actually carried out a study to
examine this possible correlation, nor did they
provide operational criteria.
In this study, effectiveness is to evaluate if the
system as a whole can provide information and
functionality effectively and will be measured by
how many answers are correct. Efficiency is like-
wise to evaluate if the system as a whole can be
used to retrieve information efficiently and will
be measured by 1) how much time it takes to
complete tasks and 2) how many steps required.
Satisfaction will look into the areas of ease of
use, organization of information, clear labeling,
visual appearance, contents, and error corrections
and will be measured by Likert scales and ques-
tionnaires. Belkin and Vickery (1985, 195) have
stated, ”Although the idea behind satisfaction as
a criterion is simple, it is obvious that actually
measuring it is not, since it must be a multifaceted
variable.” This study assesses satisfaction from
the perspectives of user’s reaction to ease of use,
organization of information, clear labeling, visual
appearance, contents, and error corrections. Ease
Usability Assessment of Academic Digital Libraries
of use is to evaluate user’s perception on the ease
of use of the system. Organization of information
is to evaluate if the system’s structure, layout,
and organization meets the user’s satisfaction.
Labeling is to evaluate from user’s perception if the
system provides clear labeling and if terminology
used are easy to understand. Visual appearance
evaluates the site’s design to see if it is visually
attractive. Contents evaluate the authority and
accuracy of information provided. Error is to test
if users may recover from mistakes easily and if
they make mistakes easily due to system’s design.
Learnability is to measure learning effort and is
examined by asking subjects to cross search the
other test site (Rutgers participants search Queens
site and vice versa). It is then measured by 1) how
soon the subjects know how to begin searching,
2) how many questions are answered correctly,
and 3) how much time to answer questions.
Figure 2 is a diagram illustrating this evaluation
model. It is suspected that there exists an inter-
locking relationship among effectiveness, efficien-
cy, and satisfaction. It is also interesting to ex-
amine how learnability interacts with these three
attributes.
This evaluation model considers both quanti-
fying elements of performance (time, accuracy
rate, steps to complete tasks) as well as subjective
criteria (satisfaction). The evaluation approach is
empirical.
Testing of the model and instruments
Two academic library Web sites were selected for
this study: the Rutgers University Libraries Web
site (http://www.libraries.rutgers.edu) and the
Queens College Web site (http://qcpages.qc.edu/
Library/). These two sites provide rich resources
for their local holdings, connections to electronic
resources and digital collections, and are one kind
of digital library.
Francisco-Revilla et al. (2001) report digital li-
braries are increasingly being defined as ones that
collect pointers to WWW-based resources rather
than hold the resources themselves. Greenstein
(2000) shares this view and says that the digital
library is known less for the extent and nature
of the collections it owns than for the networked
information space it defines through its online ser-
vices. Paepcke et al. (1996) also state that a digital
library provides a single point of access to a wide
101
Judy Jeng
Figure 2. A Model of Usability Evaluation for Digital Library
range of autonomously distributed sources. These
views can be seen as calling a library’s Web site
as a portal-typed digital library. Covi and Kling
(1996, 672) specifically include a library’s Web site
as one kind of digital library:
Digital libraries include personal, distributed, and
centralized collections such as online public access catalogs
and bibliographic databases, distributed document
databases, scholarly and professional discussion lists and
electronic journals, other online databases, forums, and
bulletin boards.
102
The experiments of this study were divided into
two stages: stage one was conducted in February/
March of 2004 and stage two was conducted in
September/October of 2004. The Rutgers Uni-
versity Libraries revamped its Web site in Sep-
tember 2004 which in part prompts the interest
to repeat the usability testing. The other reason
of re-running the usability testing was to confirm
the interlocking relationship among effectiveness,
efficiency, and satisfaction that was discovered in
stage one.
 Usability Assessment of Academic Digital Libraries

Table 3. Effectiveness of Rutgers Subjects Searched Rutgers Site and Queens Subjects Searched Queens Site

Rutgers Web Site Queens Web Site

Stage 1 Stage 2 Stage 1 Stage 2
NR1=5 NR2=15 NQ1=6 NQ2=15
Question 1 Find a book 80% 100% 83% 100%
Question 2 Find a journal 100% 80% 83% 93%
Question 3 Use database 80% 60% 83% 93%
Question 4 Use database 100% 60% 83% 80%
Question 5 Use database 100% 67% 83% 80%
Question 6 Find encyclopedia 40% 53% 33% 40%
Question 7 Find e-book 80% 33% 50% 67%
Question 8 Find information 100% 93% 100% 87%
Question 9 Find information 100% 100% 100% 100%
Average 87% 72% 78% 82%

There were a total of forty-one subjects for the
study. The first stage had a total of eleven subjects
(five from Rutgers and six from Queens). The
second stage had a total of thirty subjects (fifteen
from Rutgers and fifteen from Queens). Those
were graduate and undergraduate students. Con-
venience sampling method was used for re-
cruitment. The convenience sampling method is
common in usability test.
A list of nine questions was designed to test the
effectiveness, efficiency, and user satisfaction of
the system. These nine questions were designed
to be representative of typical uses of a library’s
Web site (see Appendix A). Question 1 is to locate
a book. Question 2 is to locate a journal. Questions
3 to 5 are to use databases. Question 6 is to use an
e-encyclopedia. Question 7 is to locate an e-book.
Questions 8 and 9 are to seek instructions on
library services.
There was a pre-test questionnaire that collects
demographic data, including gender, age, status
(undergraduate, master’s, or doctoral student),
major, years at the institution, original nationality
if coming from foreign country, and frequency of
using the site. After completing a task, the subject
was asked to rank satisfaction and to write down
comments. There was also a post-test questionnaire
that specifically examines satisfaction in the areas
of ease of use, organization of information, clear
labeling, visual appearance, contents, and error
corrections (see Appendix B).
Each subject was asked to search two sites with
the sequence alternated. When searching the other
institution’s site, questions 3–5 were eliminated be-
cause of proxy server authorization and to limit
the session to a reasonable timeframe. The pur-
pose of searching the other institution’s site was to
measure learnability.
Results
Effectiveness
Effectiveness in this study is measured by how
many answers are correct. Table 3 indicates the
effectiveness scores of Rutgers subjects searched
Rutgers site and Queens subjects searched Queens
site respectively. Questions 6 (find an encyclopedia
article) and 7 (find an e-book) gave subjects more
difficulties on both Rutgers and Queens sites. This
is consistent on stages one and two.
Efficiency
Efficiency in this study is measured by 1) how
much time it takes to complete a task correctly
and 2) how many keystrokes/clicks (or steps/
movements). If the answer is wrong, the time
spent and the keystrokes/clicks are not counted
in the calculation. If a subject decides to give up
a question, the time spent and the steps on the
particular question are also not counted. Table 4
indicates time used by Rutgers subjects to search
Rutgers site and Queens subjects to search Queens
site on stages one and two respectively.
As indicated in Table 4, question 9 (find in-
formation to set up remote access) is the easiest on
both Rutgers and Queens sites. This is symbolic
that it is important that library users are able to
remotely access a library’s electronic resources
easily. Although the instruction is readily available
and easy to locate, it is equally important that

103
Judy Jeng

Table 4. Time Used by Rutgers Subjects to Search Rutgers Site and Queens Subjects to Search Queens Site

Rutgers Web Site Queens Web Site

Stage 1 Stage 2 Stage 1 Stage 2
NR1=5 NR2=15 NQ1=6 NQ2=15
Question 1 3 min. 17 sec. 2 min. 12 sec. 1 min. 29 sec. 1 min. 28 sec.
Question 2 1 min. 9 sec. 1 min. 39 sec. 1 min. 5 sec. 1 min. 26 sec.
Question 3 6 min. 11 sec. 3 min. 5 sec. 2 min. 28 sec. 2 min. 38 sec.
Question 4 3 min. 18 sec. 1 min. 33 sec. 1 min. 12 sec. 2 min.
Question 5 6 min. 33 sec. 2 min. 28 sec. 4 min. 39 sec. 2 min. 41 sec.
Question 6 4 min. 12 sec. 2 min. 50 sec. 2 min. 20 sec. 4 min. 33 sec.
Question 7 6 min. 17 sec. 5 min. 25 sec. 1 min. 36 sec. 2 min. 11 sec.
Question 8 2 min. 4 sec. 46 sec. 1 min. 8 sec. 1 min. 9 sec.
Question 9 1 min. 23 sec. 29 sec. 1 min. 10 sec. 46 sec.
Average 3 min. 49 sec. 2 min. 15 sec. 1 min. 54 sec. 2 min. 1 sec.

Table 5. Number of Steps by Rutgers Subjects to Search Rutgers Site and Queens Subjects to Search Queens Site

Rutgers Web Site Queens Web Site

Stage 1 Stage 2 Stage 1 Stage 2
NR1=5 NR2=15 NQ1=6 NQ2=15
Question 1 Find a book 9 9 7 7
Question 2 Find a journal 5 7 6 6
Question 3 Use database 20 10 8 10
Question 4 Use database 9 6 7 9
Question 5 Use database 19 8 17 11
Question 6 Find encyclopedia 15 12 11 18
Question 7 Find e-book 27 20 7 9
Question 8 Find information 6 3 2 3
Question 9 Find information 4 2 2 2
Average 13 9 7 8

the instruction is easy to follow. This study does
not evaluate if the instruction is clear and easy to
follow.
In addition to using time as an indicator to assess
efficiency, number of movements/steps is also
used in this study. This is counted by the number
of keystrokes or clicks. A string of characters when
performing a title or author search is counted as
one step. Each press on Enter key or each click on
mouse is also counted as one step. Table 5 indicates
the numbers of steps by Rutgers and Queens
subjects searching their own sites.
As indicated in Table 5, question 7 (find e-book)
took subjects more steps to answer in Rutgers site.
Three participants suggested that Rutgers should
make e-book collections more noticeable on its
Web page. It is hard to dig out e-book collections
in the cluster of Rutgers site.
The use of electronic databases to find articles
is not an easy task for participants. Four partici-
pants commented that it is necessary to know
those databases in order to select from the long
list which one to use. Two participants said that
they had difficulty in combining terms for ques-
tion 3 (find an article about nursing homes and
mental illness). Two participants suggested meta-
searching which allows users to enter search
strings to search all available databases so that
they do not need to learn various interfaces and
commands.
Relationships among effectiveness, efficiency, and
satisfaction
One of the research goals of this study is to ex-
amine the relationships among effectiveness, ef-
ficiency, and satisfaction. This section reports the
results of stages one and two respectively.
Effectiveness and satisfaction
Effectiveness in this study is measured by whether
the subjects answer the questions correctly. Two
one-way analysis of variance were conducted to
evaluate the relationships between correctness of
answers and satisfaction for stage one and two

104

Table 6. Differences of Satisfaction Ranking between Correct
or Incorrect Answers (1=easy to use/high satisfaction,
5=difficult to use/low satisfaction)
Satisfaction Satisfaction
(Stage 1) (Stage 2)
N=11 N=30
Correctness of M SD M SD
Answer
Wrong Answer 4.00 1.17 3.93 1.33
Correct Answer 2.20 1.16 2.09 1.20
respectively. The dependent variable was satisfac-
tion (1=easy to use/high satisfaction to 5=difficult
to use/low satisfaction) and the independent vari-
able was correctness of answers (0=wrong answer
and 1=correct answer).
The ANOVA for stage one was significant, F (1,
163) = 57.57, p < .001. The strength of relationship
between satisfaction and correctness of answers, as
assessed by η2, was strong, with the correctness of
answer factor accounting for 26% of the variance
of the dependent variable. Post-hoc tests were not
performed because there were fewer than three
factors.
The ANOVA for stage two was also significant, F
(1, 445) = 185.48, p < .001. Because the p-value was
less than .05, the null hypothesis that there were
no differences between the groups was rejected.
The η2 of .29 indicates a strong relationship be-
tween satisfaction and correctness of answers.
Post-hoc tests were not performed because there
were fewer than three factors.
Based on the findings of the two ANOVA
analyses that there were significant differences in
the means of satisfaction between right and wrong
answers, an interpretation may be made that
subjects felt less satisfied with the system when
they failed to perform the task correctly.
Table 6 indicates the means of satisfaction rank-
ing with regard to correctness of answers in both
stages. The rankings are similar in both stages.
Efficiency and satisfaction
Correlation coefficients were computed among
time spent on answering questions, numbers of
steps, and satisfaction. Using the Bonferroni ap-
proach to control for Type I error across the
three correlations, a p-value of less than .017 was
required for significance.
Usability Assessment of Academic Digital Libraries
The correlation between time spent on an-
swering questions and the numbers of steps for
stage one was significant, r (134) = .78, p < .001;
for stage two was also significant, r (445) = .77, p
< .001. This means the longer the time that a
subject spent on answering a question, the more
the steps involved. The effect sizes were large for
both stages.
The correlation between the numbers of steps
and satisfaction for stage one was significant, r
(134) = .51, p < .001; for stage two was also sig-
nificant, r (445) = .57, p < .001. This means the
more steps taken to answer a question, the lower
the satisfaction. The effect sizes for both stages
were large.
The correlation between the time spent and
satisfaction for stage one was significant, r (134) =
.46, p < .001; for stage two was also significant,
r (445) = .61, p < .001. This means the more time
spent on answering a question, the lower the satis-
faction. The effect size for stage one was medium,
for stage two was large.
Based on those statistical results, a conclusion is
drawn that there is correlation between efficiency
and satisfaction. Users feel less satisfied when the
task takes more steps to get to the answer.
Effectiveness and efficiency
Effectiveness and steps
Two ANOVAs were conducted, for stages one and
two respectively, to evaluate whether the group
means of steps on correct answers and incorrect
answers differ significantly from each other. The
independent variable, effectiveness, included two
levels: correct and incorrect answers. The de-
pendent variable was the number of steps.
The ANOVA for stage one was significant,
F (1, 163) = 29.19, p < .001. The strength of re-
lationship between effectiveness and steps, as
assessed by η2, was strong, with the effectiveness
factor accounting for 15% of the variance of the
dependent variable.
The ANOVA for stage two was also significant,
F (1, 445) = 82.84, p < .001. The strength of re-
lationship between effectiveness and steps, as
assessed by η2, was strong, with the effectiveness
factor accounting for 16% of the variance of the
dependent variable.
105
Judy Jeng

Table 7. Effectiveness and Steps Table 8. Effectiveness and Tim

Stage 1 Stage 2 Stage 1 Stage 2

N=11 N=30 N=11 N=30
Correctness of M SD M SD Correctness of
M SD M SD
Answer Answer
Wrong Answer 16.97 1.47 13.39 9.29 4 min. 3 min. 2 min.
Wrong Answer 25 sec.
Correct Answer 8.18 0.71 6.74 5.55 12 sec. 53 sec. 25 sec.
2 min. 1 min. 1 min.
Correct Answer 12 sec.
23 sec. 48 sec. 36 sec.
Figure 3. Significant Differences in Steps Related to Correct-
ness of Answers
Figure 4. Significant Differences in Time Related to Correct-
ness of Answers

As indicated in Table 7 and Figure 3, incorrect

answers involve more steps while correct answers
involve less steps. When the subject knows how to
get the answer, it takes them less steps while when
they don’t know how to find the information, they
struggle.
Effectiveness and time
Two ANOVAs were conducted to evaluate
whether the group means of time on correctness of
answers and incorrect answers differ significantly
from each other. The independent variable, the
effectiveness factor, included two levels: correct
answer and incorrect answer. The dependent
variable was time spent on completing tasks.
The ANOVA for stage one was significant, F (1,
163) = 15.70, p < .001. The strength of relationship
between effectiveness and time, as assessed by η2,
was medium, with effectiveness factor accounting
for 9% of the variance of the dependent variable.
The ANOVA for stage two was also significant,
F (1, 445) = 107.44, p < .001. The strength of re-
lationship between effectiveness and time, as as-
sessed by η2, was strong, with effectiveness fac-
tor accounting for 20% of the variance of the
dependent variable.
Table 8 and Figure 4 indicate the means of time
for correct and incorrect answers for stages one
and two.
Based on the analyses from one-way ANOVAs
and correlation coefficients, it is found that there
exist interlocking relationships among effective-
ness, efficiency, and satisfaction. Figure 2 indicates
these relationships.
Learnability
Learnability is in some sense the most fundamental
usability attribute (Nielsen 1993). The system
should be easy to learn so that the user can rapidly
start getting some work done with the system.
As Dzida et al. (1978) reported, learnability is
especially important for novice users.
Learnability in this study is to examine learning
effort of using a new Web site and to measure how
easy a site for new visitors to orient themselves
and get a good overview of what the site offers.
Learnability is inferred from the amount of time
required to achieve user performance standards

106

Table 9. Learnability of Both Sites in Stage One
Rutgers Web Site
Correctness
Rutgers Subjects NR1=5 83%
Queens Subjects NQ1=6 75%
Table 10. Learnability of Both Sites in Stage Two
Rutgers Web Site
Correctness
Rutgers Subjects NR2=15 77%
Queens Subjects NQ2=15 67%
and if they can perform the tasks correctly. The
design of this study asked Rutgers subjects to
search Queens site and Queens subjects to search
Rutgers site. It is found that all subjects can begin
searching without difficulty and begin to perform
searches almost immediately. The experimenter
counted the time subjects took before their first
click and found that they almost started im-
mediately.
Learnability is also examined by how many
questions subjects can answer correctly and how
much time it takes on the new Web site. This is to
determine if they can perform tasks at a proficient
level. As indicated in Table 9, it seems that both
Rutgers and Queens subjects were able to complete
tasks correctly and faster in Queens site. However,
this finding is not confirmed in stage two. Table 10
indicates that subjects feel more comfortable using
their home institution’s site than the new site.
Satisfaction
Satisfaction is a multi-dimensional construct. This
study applies Likert scale to measure its directions
and intensity (1= easy to use/high satisfaction,
5=difficult to use/low satisfaction). Factors of sat-
isfaction are carefully identified as shown in the
model (see Figure 2). In addition to Likert scales
used after each task and in post-test questionnaire,
participants’ comments were solicited. There were
also open-ended questions. In addition to overall
satisfaction and reaction, satisfaction was further
examined in the areas of ease of use, organization
of information, terminology, labeling, visual ap-
pearance, content, error correction, best features,
and worst features.
Usability Assessment of Academic Digital Libraries
Queens Web Site
Speed Correctness Speed
3 min. 4 sec. 90% 2 min. 20 sec.
2 min. 75% 1 min. 42 sec.
Queens Web Site
Speed Correctness Speed
2 min. 14 sec. 71% 2 min. 14 sec.
2 min. 18 sec. 81% 1 min. 56 sec.
The use of Likert scale is known as an economi-
cal way of measuring user satisfaction (Pearson
1977).
Overall satisfaction
The scores for calculating overall satisfaction are
gathered from the Likert scales after completing
each task. As indicated in Table 11, the overall
satisfaction ratings by all subjects are very close
between the two sites in both stages one and two.
The satisfaction ratings for question 1 (locate a
book) on Rutgers site are worse than the cor-
responding ratings on Queens site in both stages.
Thirteen subjects (32 percent) commented that a
simple search by title on Rutgers catalog resulted
in too many hits when it was supposed to be a
concise search. Rutgers catalog provides two op-
tions for searching: ”contains” and ”begins with.”
A title search of Gone with the Wind uses ”contains”
results in 42 hits while the same search string using
”begins with” results in 12 hits. The default is set
to ”contains” by the system. Most users did not
realize the difference and did not change it. One
subject said, ”I typed Gone with the Wind without
author’s name in the search for title and it wasn’t
the first search result, which I thought it should
have been.” Another subject said, ”I don’t like it
when a lot of unrelated things show up.”
Ease of use
Table 12 indicates the ratings of ease of use by
Rutgers and Queens subjects in both stages one
and two. Queens site was rated easier to use than
Rutgers site by both Rutgers and Queens subjects
107
Judy Jeng

Table 11. Satisfaction Rating by Question (1=easy to use/high satisfaction, 5=difficult to use/low satisfaction)

Rutgers site Queens site

Stage 1 Stage 2 Stage 1 Stage 2
N=11 N=30 N=11 N=30
Question 1 Find a book 3.2 2.7 1.9 1.7
Question 2 Find a journal 1.9 2.2 2.3 2.5
Question 3 Use database 3.4 2.7 1.8 2.5
Question 4 Use database 2.2 1.9 1.7 2.4
Question 5 Use database 2.8 2.5 3.2 2.6
Question 6 Find encyclopedia 2.7 3.4 3.7 4.3
Question 7 Find e-book 4.1 3.7 2.9 3.2
Question 8 Find information 2.2 1.9 2.1 2.2
Question 9 Find information 2.0 1.3 1.3 1.6
Overall 2.6 2.5 2.5 2.6

in stage one. However, in stage two, subjects
rated their own institution’s site easier to use. It
is worth noting that Rutgers site receives better
rating in stage two, after its revamp of Web site
appearance.
It is found that subjects evaluate ”ease of use”
from the perspectives of ”easy to get around,” ”can
follow directions easily,” ”easy navigation,” ”clear
description,” ”intuitive,” and ”user-friendly.”
Organization of information
Table 13 indicates the ratings of organization of
information. Queens site received better ratings
from both Rutgers and Queens subjects in stage
one. However, Rutgers site, after its redesign of its
Web site, receives better ratings in stage two by
both Rutgers and Queens subjects.
It is found that subjects evaluate ”organization”
from the perspectives of ”simple,” ”straightfor-
ward,” ”logical,” ”easy to look up things,” and
”placing common tasks upfront.”
Terminology
Table 14 indicates how subjects rate the ter-
minology used in both sites and if categories are
clearly labeled. Rutgers site receives better ratings
than Queens site in both stages one and two and
by both Rutgers and Queens subjects.
It is found that subjects evaluate ”terminology”
from the perspectives of ”simple,” ”straightfor-
ward,” ”understandable,” ”generic,” ”label sec-
tions clearly,” ”no jargon,” ”clear descriptions/
explanations,” and ”from user’s perspective.” It
is also found that thirteen subjects (31 percent)
expressed various degrees of problems with ter-
minology used in both Rutgers and Queens site.
The concerns are centered on:
• the library site assumes that users have common sense
of library terms.
• users do not know the difference between index and
database
• the terms are library-oriented, such as articles, encyclo-
pedia, serial, periodicals, and index.
• need better description/explanation to library jargons.
Attractiveness
Table 15 indicates how subjects rate the degrees of
visual attractiveness of both sites. It is interesting
to find that subjects give their home institution’s
site better rating, in both stages one and two.
It is found that subjects evaluate ”attractiveness”
from the perspectives of ”appropriate graphics,”
”readability,” ”appropriate color,” ”not too com-
plicated,” and ”appropriate size of font.”
Mistake recovery
Table 16 indicates how subjects rate both sites’
features on the ease of recovering from mistakes.
The Rutgers site receives better rating from both
Rutgers and Queens subjects in stage one. How-
ever, in stage two, subjects rated their own site
easier to recover from mistakes.

108
 Usability Assessment of Academic Digital Libraries

Table 12. Ease of Use (1=easy to use, 5=difficult to use) Table 16. Mistake Recovery (1=easy, 5=difficult)

Ease of use Rutgers site Queens site Mistake Recovery Rutgers site Queens site
Stage 1 Stage 2 Stage 1 Stage 2 Stage 1 Stage 2 Stage 1 Stage 2
Rutgers subjects 2.6 2.5 2.0 3.1 Rutgers subjects 2.2 1.9 2.6 3.1
Queens subjects 3.0 2.7 2.0 2.4 Queens subjects 2.0 2.4 2.7 1.9

Table 13. Organization of Information (1=clear, 5=unclear) Table 17. Overall Reaction (1=satisfied, 5=unsatisfied)

Organization of Overall Reaction Rutgers site Queens site

Rutgers site Queens site
Information Stage 1 Stage 2 Stage 1 Stage 2
Stage 1 Stage 2 Stage 1 Stage 2
Rutgers subjects 2.4 2.1 2.2 3.2
Rutgers subjects 3.0 1.9 2.2 3.1 Queens subjects 3.2 2.2 1.8 2.4
Queens subjects 2.6 2.1 1.6 2.6

Table 18. User Lostness (N2=30)

Table 14. Terminology (1=clear, 5=unclear)
Rutgers Site Queens Site
Terminology Rutgers site Queens site
Stage 1 Stage 2 Stage 1 Stage 2 Rutgers Subjects 6 11
Queens Subjects 8 6
Rutgers subjects 3.0 2.2 3.4 2.9
Total 14 17
Queens subjects 2.4 1.7 2.4 2.1

Table 19. Navigation (N2=30)

Table 15. Attractiveness (1=attractive, 5=unattractive)
Rutgers site Queens site
Attractiveness Rutgers site Queens site
Stage 1 Stage 2 Stage 1 Stage 2 Rutgers subjects 11 7
Rutgers subjects 2.6 2.1 3.2 2.6 Queens subjects 13 12
Queens subjects 3.0 2.8 2.0 2.3 Total 24 19

Table 20. Click Cost (N2=30)

Rutgers site Queens site

It is found that subjects evaluate ”mistake re- Rutgers subjects 10 11
covery” from the perspectives of ”easy naviga- Queens subjects 12 11
tion.” It is helpful to have navigation bar and Back Total 22 22
button. It is also important to be able to start over.
A ”Help” section is also suggested.

Specific comments on improvements for both

Overall reaction
The scores for calculating overall reaction are
from the Likert scales in post-test questionnaires.
This gives participants another opportunity to
rate the test sites and provide overall reaction af-
ter examining specific areas such as ease of use,
organization of information, terminology, attrac-
tiveness, and mistake recovery. Table 17 indicates
the overall reaction to both Rutgers and Queens
sites. It turns out that subjects are more pleased
with Queens site in stage one, but are more
pleased with Rutgers site in stage two, after
Rutgers revamped its site.
Rutgers and Queens sites, their best features, worst
features, and desired features are also received.
Lostness
User lostness issue was examined in the stage two
of this study. It is found that fourteen subjects (46
percent) felt lost in Rutgers site while there were
seventeen subjects (57 percent) felt lost in Queens
site (see Table 18). An examination of Rutgers and
Queens subjects separately indicates that subjects
are lost more frequently on a new site than their
institution’s site.

109
Judy Jeng
It is found that subjects felt loss for the reasons
of site design, navigation, tasks, lacking of con-
fidence, and mistake recovery.
Navigation
Navigation disorientation is found by Brinck,
Gergle, and Wood (2002) among the biggest
frustrations for Web users. The navigation issue
was examined in stage two of this study. Twenty-
four subjects (71 percent) said Rutgers site is easy
to navigate while nineteen subjects (56 percent)
said Queens site is easy to navigate. Table 19
indicates further breakdown by Rutgers and
Queens subjects.
Subjects’ comments regarding navigation indi-
cate that links should be stable, self-explanatory,
consistent, and easy to recover from mistakes. Sev-
eral participants commented that Queens site’s
drop-down menu is over sensitive and the links are
”ghosting” (i.e. disappear when mouse moves).
Click Cost
”Click cost” is defined by McGillis and Toms
(2001) that ”users are very reluctant to click unless
they are fairly certain they will discover what
they are looking for.” The click cost issue was first
examined in stage two of this study. Forty-four
subjects (73 percent) expressed that they expect
the click(s) will eventually lead them to the correct
answer. Table 20 is a breakdown by Rutgers and
Queens subjects.
As one subject indicates, ”a click was supposed
to imply what you could expect.” Another subject
said, ”I expect, but it doesn’t always.” One said,
”I just hoped for the best.” Users of library Web
sites come to the site to look up information. They
want to be able to get to the answer easily and
rapidly. Each click should get them closer to the
information.
Demographic factors on performance
This study examines the relationships between
demographic factors, such as gender, age, sta-
tus, major, ethnic background, years at the in-
stitution, and frequency of using library Web site,
on effectiveness. A series of ANOVAs and the
Pearson Product-Moment Correlation Coefficient
were conducted. It is found that there does not
110
exist statistically significant relationships between
those demographic factors and effectiveness.
Discussion
Number of subjects
How many subjects are needed in the study of
this kind? This study recruited forty-one subjects,
including twenty from Rutgers University and
twenty-one from Queens College. Each subject
was asked to perform a total of fifteen tasks (nine
on their institution’s site and six on the other site).
Are these enough both to evaluate usability and
to examine the relationships among effectiveness,
efficiency, and satisfaction?
Nielsen and Molich (1990) found that not quite
half of all major usability problems were detected
with three participants. Nielsen (1993) then found
that using six participants allowed the evaluator
to identify the majority of important problems.
Virzi (1992) found that 80 percent of the usability
problems were detected with five participants and
90 percent were detected with ten participants.
The same findings were confirmed by Rubin
(1994). Dumas and Redish (1993) recommend
six to twelve participants for a typical usability
testing. They said that additional participants
are less and less likely to reveal new information.
Nielsen (2000) strongly advocates using five users
in a formal usability testing. He states, ”Elaborate
usability tests are a waste of resources. The best
results come from testing no more than five users
and running as many small tests as you can af-
ford.” This study includes forty-one subjects on
the grounds that it is not only a usability test but
also a research project to study the relationships
among attributes. It needs more subjects to en-
sure sufficient statistical power. Each subject per-
formed fifteen tasks. The unit of analysis is search.
Therefore there are a total of 615 sets of scores of
effectiveness, efficiency, and satisfaction. These
should provide enough statistical power to analyze
the relationships among effectiveness, efficiency,
and satisfaction.
Another reason of using a large number of
subjects is to help to eliminate the investigator’s
judgment call whether a problem discovered from
the usability test is user-specific or a true usability
problem. The small sample size makes it difficult
to determine.

Interlocking relationships among effectiveness,
efficiency, and satisfaction
The study was divided into two phases: the phase
one included eleven subjects and was conducted
in February/March of 2004; the phase two had
thirty subjects and was conducted in September/
October 2004, after Rutgers revamped its Web
page. Statistical analyses were performed sepa-
rately. The first phase has a total of 165 sets of
scores. The second phase has a total of 450 sets of
scores. These should provide sufficient statistical
power to analyze the relationships among ef-
fectiveness, efficiency, and satisfaction. It is found
in both phases that these relationships exist.
Because the relationships are confirmed in both
stages respectively, the investigator of this study
thereby does not perform a combined analysis
using the whole 615 sets of scores.
The study of Frøkjær, Hertzum, and Hornbæk
(2000) found effectiveness and efficiency are only
weakly correlated. The study of Walker et al.
(1998) found user satisfaction is not determined
by efficiency. This study found the strength of re-
lationship between effectiveness and steps was
strong (η2 = .15 in stage one; η2 = .16 in stage
two) and the strength of relationship between
effectiveness and time was medium (η2 = .09) in
stage one and was strong (η2 = .20) in stage two.
This study also found the strength of relationship
between effectiveness and satisfaction were strong
in both stages (η2 = .26 in stage one; η2 = .29 in
stage two). However, this study also realizes that
effectiveness, efficiency, and satisfaction are three
separate criteria. Each has its specific emphasis
and should be measured separately.
Conclusion
The goals of this study were to develop a model of
evaluating usability for academic digital libraries;
use the model for development of measures, in-
struments, and methods for evaluating academ-
ic digital libraries; test the measures, instruments,
and methods by applying them in two academic
digital libraries; generalize the model, instru-
ments, and methods for use in academic librar-
ies; and to study the relationships among
effectiveness, efficiency, and satisfaction. Those
research objectives are achieved. A model and
Usability Assessment of Academic Digital Libraries
a suite of instruments were designed and suc-
cessfully tested. The relationships among effective-
ness, efficiency, and satisfaction were found. In
addition, the user lostness and click costs issues
were studied; and users’ criteria on usability at-
tributes such as ”ease of use,” ”organization of
information,” ”terminology,” ”attractiveness,” and
”mistake recovery” were found. Users’ comments
on improvements were especially helpful on
understanding what digital library users care the
most.
Digital library is an information system over
a network that is organized, well-managed, and
supports the creation, use, and searching of digital
objects. Digital library should be looked as a tool
that supports user’s information task. Users are
looking for an information system that is easy and
intuitive to use. This study applies user-centered
formal usability testing technique to measure
usability from the perspectives of effectiveness,
efficiency, satisfaction, and learnability which
considers both performance elements as well as
satisfaction.
The evaluation model proposed in this study
considers both the quantifying elements such as
time, accuracy rate, steps to complete tasks, as well
as subjective criteria, satisfaction, which is further
examined in the areas of ease of use, organization
of information, labeling, visual appearance, con-
tent, and error correction. Although the model is
tested in two academic libraries in this study, it
may be generalized to other digital libraries or
information systems.
The contributions of this study are: 1) an evalua-
tion model for digital libraries; 2) benchmarks
of performances; 3) the finding of interlocking
relationships among effectiveness, efficiency, and
satisfaction; 4) operational criteria and strategy
to measure effectiveness, efficiency, satisfaction,
and learnability; 5) user’s criteria regarding ease
of use, organization of information, labeling,
visual appearance, content, and error correction;
6) causes of user lostness; 7) what contributes to
easy navigation; 8) the finding of click cost; 9) the
finding that demographic factors such as gender,
age, status, major, and ethnic background do not
have statistical significance on performance; 10) a
review of how usability have been and should be
defined in the context of the digital library; and
11) what usability evaluation methods have been
applied in academic digital libraries.
111
Judy Jeng
References
Adams, A., and A. Blandford. 2002. Acceptability of
medical digital libraries. Health Informatics Journal 8
(2): 58–66.
Allen, Maryellen. 2002. A case study of the usability
testing of the University of South Florida’s virtual li-
brary interface design. Online Information Review 26
(1): 40–53.
Arms, William Y. 2000. Digital libraries. Cambridge,
Mass.: MIT Press.
Askin, A. Y. 1998. Effectiveness of usability evaluation
methods at a function of users’ learning stages. Mas-
ter’s thesis, Purdue University.
Augustine, Susan, and Courtney Greene. 2002. Dis-
covering how students search a library Web site: A
usability case study. College & Research Libraries 63
(4): 354–65.
Belkin, Nicholas J., and Alina Vichery. 1985. Interaction in
Information Systems: A review of research from document
retrieval to knowledge-based systems. Wetherby, West
Yoorkshire, Great Britain: British Library.
Benjes, Candice, and Janis F. Brown. 2001. Test, revise,
retest: Usability testing and library Web sites. Internet
Reference Services Quarterly 5 (4): 37–54.
Bennett, J. L. 1972. The user interface in interactive sys-
tems. Annual Review of Information Science and Tech-
nology 7, 159–96.
Bennett, J. L. 1979. The commercial impact of usability
in interactive systems. In Man-Computer Communica-
tion, Infotech State-of-the-Art, Vol. 2, ed. B. Shackel,
1–17. Maidenhead: Infotech International.
Bishop, Ann Peterson. 2001. Logins and Bailouts: Meas-
uring access, use, and success in digital libraries. The
Journal of Electronic Publishing 4 (2). http://www.press.
umich.edu/jep/04-02/bishop.html (accessed Jan-
uary 23, 2002).
Blandford, Ann, and George Buchanan. 2002a. Usability
for digital libraries. Proceedings of the Second ACM/
IEEE-CS Joint Conference on Digital Libraries, 424.
New York: ACM Press.
Blandford, Ann, and George Buchanan. 2002b. Work-
shop report: Usability of digital libraries @ JCDL’02.
http://www.uclic.ucl.ac.uk/annb/DLUsability/
SIGIR.pdf (accessed June 11, 2003).
Blandford, Ann, and George Buchanan. 2003. Usability
of digital libraries: A source of creative tensions with
technical developments. TCDL Bulletin. http://
w w w. i e e e - t c d l . o rg / B u l l e t i n / s u m m e r 2 0 0 3 /
blandford/blandford.html (accessed October 30,
2004).
Blandford, Ann, George Buchanan, and Matt Jones.
2004. Usability of digital libraries. International Jour-
nal on Digital Libraries 4 (2): 69–70.
Blandford, Ann, Suzette Keith, Iain Connell, and Helen
Edwards. 2004. Analytical usability evaluation for
112
digital libraries: A case study. Proceedings of the Fourth
ACM/IEEE Joint Conference on Digital Libraries, 27–36.
New York: ACM Press.
Blandford, Ann, Hanna Stelmaszewska, and Nick
Bryan-Kinns. 2001. Use of multiple digital libraries:
A case study. Proceedings of the First ACM/IEEE-CS
Joint Conference on Digital Libraries, 179–188. New
York: ACM Press.
Booth, Paul. 1989. An introduction to human-computer in-
teraction. London: Lawrence Erlbaum Associates.
Borgman, Christine L., Anne J. Gilliland-Swetland,
Gregory H. Leazer, Richard Mayer, David Gwynn,
Rich Gazan, and Patricia Mautone. 2000. Evaluating
digital libraries for teaching and learning in under-
graduate education: A case study of the Alexandria
Digital Earth ProtoType (ADEPT). Library Trends 49
(2): 228–50.
Brinck, Tom, Darren Gergle, and Scott D. Wood. 2002.
Designing Web sites that work: Usability for the Web.
San Francisco: Morgan Kaufmann.
Campbell, Nicole. 2001. Usability assessment of library-
related Web sites: Methods and case studies. Chicago:
LITA, American Library Association.
Chisman, Janet, Karen Diller, and Sharon Walbridge.
1999. Usability testing: A case study. College & Re-
search Libraries 60 (6): 552–69.
Clairmont, Michelle, Ruth Dickstein, and Vicki Mills.
1999. Testing of usability in the design of a new information
gateway. http://www.library.arizona.edu/library/
teams/access9798 (accessed February 14, 2003).
Clark, Jason A. 2004. A usability study of the Belgian-
American Research Collection: Measuring the func-
tionality of a digital library. OCLC Systems & Services:
International Digital Library Perspectives 20 (3): 115–
27.
Clausen, Helge. (1999). Evaluation of library Web sites:
The Danish case. The Electronic Library 17 (2): 83–87.
Covi, Lisa, and Kling, Rob. 1996. Organizational di-
mensions of effective digital library use: Closed
rational and open natural systems models. Journal
of the American Society for Information Science 47 (9):
672–89.
Davis, F. (1989). Perceived usefulness, perceived ease of
use, and user acceptance of information technology.
MIS Quarterly 13, 319–339.
Davis, F. D., R. P. Bagozzi, and P. R. Warshaw. (1989).
User acceptance of computer technology: A compari-
son of two theoretical models. Management Science
35, 982–1003.
Dickstein, Ruth, and Vicki Mills. 2000. Usability testing
at the University of Arizona library: How to let the
users in on the design. Information Technology and
Libraries 19 (3): 144–51.
Dorward, Jim, Derek Reinke, and Mimi Recker. 2002.
An evaluation model for a digital library services
tool. Proceedings of the Second ACM/IEEE-CS Joint

Conference on Digital Libraries, 322–323. New York:
ACM Press.
Dumas, Joseph. S., and Janice C. Redish. 1993. A prac-
tical guide to usability testing. Norwood, NJ: Ablex
Publishing Co.
Dzida, W., S. Herda, and W. D. Itzfeldt. 1978. User-
perceived quality of interactive systems. IEEE Trans-
actions on Software Engineering 4 (4): 270–76.
Eason, K. D. 1981. A task-tool analysis of the manager-
computer interaction. In Man-Computer Interac-
tion, ed. B. Shackel. Amsterdam: Sijthoff and Noord-
hoff.
Eberts, R. E. 1985. Human-computer interaction. In Hu-
man Factors Psychology, ed. P. A. Hancock, 249–304.
Amsterdam: Elsevier.
France, Robert K., Lucy Terry Nowell, Edward A. Fox,
Rani A. Saad, and Jianxin Zhao. 1999. Use and us-
ability in a digital library search system. http://
www.dlib.vt.edu/Papers/Use_usability.PDF (ac-
cessed March 15, 2003).
Francisco-Revilla, Luis, Frank Shipman, Richard Fruta,
Unmil Karadkar, and Avital Arora. 2001. Managing
change on the Web. Proceedings of the First ACM/
IEEE-CS Joint Conference on Digital Libraries, 67–76.
New York: ACM Press.
Frøkjær, Erik, Morten Hertzum, and Kasper Hornbæk.
2000. Measuring usability: Are effectiveness, ef-
ficiency, and satisfaction really correlated? Proceed-
ings of the CHI2000 Conference on Human Factors in
Computing Systems, 345–52. New York: ACM Press.
Fu, Limin Paul. 1999. Usability evaluation of Web page
design. PhD diss., Purdue University.
Furtado, Elizabeth, João José Vasco Furtado, Fernando
Lincoln Mattos, and Jean Vanderdonckt. 2003. Im-
proving usability of an online learning system by
means of multimedia, collaboration and adaptation
resources. In Usability evaluation of online learning
programs, ed. Claude Ghaoui, 69–86. Hershey, PA:
Information Science Publishing.
Gluck, Myke. 1997. A descriptive study of the us-
ability of geospatial metadata. Annual Review of
OCLC Research. http://www.oclc.org/research/
publications/arr/1997/gluck/gluck_frameset.htm
(accessed February 11, 2002).
Gould, J. D. 1988. How to design usable systems. In
Handbook of Human Computer Interaction, ed. Martin
Helander, 757–89. New York: Elsevier.
Greenstein, Daniel. 2000. Digital libraries and their
challenges. Library Trends 49 (2): 290–303.
Grudin, J. 1992. Utility and usability: Research issues
and development contexts. Interacting with Computers
4 (2): 209–17.
Guillemette, Ronald A. 1995. The evaluation of usability
in interactive information systems. In Human factors
in information systems: Emerging theoretical bases, ed.
Jane M. Carey, 207–221. Norwood, NJ: Ablex.
Usability Assessment of Academic Digital Libraries
Hammill, Sarah J. (2003). Usability testing at Florida
International University Libraries: What we learned.
Electronic Journal of Academic and Special Librarianship
4 (1).
Hartson, H. Rex, Priya Shivakumar, and Manuel A.
Pérez-Quinones. 2004. Usability inspection of digital
libraries: A case study. International Journal on Digital
Libraries 4 (2): 108–23.
Hennig, Nicole. 1999. Web site usability test. http://
macfadden.mit.edu:9500/webgroup/usability/
results/index.html (accessed February 26, 2003).
Hessel, Heather, and Tom Burton-West. 2003. Tools for
improving WebVoyage usability: Search log analysis and
usability testing. http://library.pub.getty.edu:8100/
vugm03.html (accessed February 2, 2005).
Hix, Deborah, and H. Rex Hartson. 1993. Developing
user interfaces: Ensuring usability through product &
process. New York: John Wiley.
International Standards Organization. 1994. Ergonomic
requirements for office work with visual display terminals.
Part 11: Guidance on usability (ISO DIS 9241-11).
London: International Standards Organization.
Kantner, Laurie, and Stephanie Rosenbaum. 1997. Us-
ability studies of www sites: Heuristic evaluation
vs. laboratory testing. Proceedings of the 15th Annual
International Conference on Computer Documentation,
153–160. New York: ACM Press.
Karoulis, Aaroulis, and Andreas Pombortsis. 2003.
Heuristic evaluation of Web-based ODL programs.
In Usability evaluation of online learning programs, ed.
Claude Ghaoui, 88–109. Hershey, PA: Information
Science Publishing.
Keith, Suzette, Ann Blandford, Bob Fields, and Yin Leng
Theng. 2003. An investigation into the application of
claims analysis to evaluate usability of a digital library
interface. Paper presented at the usability workshop
of JCDL 2002. http://www.uclic.ucl.ac.uk/annb/
DLUsability/Keith15.pdf (accessed June 13, 2003).
Kengeri, Rekha, Cheryl D. Seals, Hope D. Harley, Hi-
mabindu P. Reddy, and Edward A. Fox. 1999. Us-
ability study of digital libraries: ACM, IEEE-CS,
NCSTRL, NDLTD. International Journal on Digital
Libraries 2: 157–69.
Kim, Kyunghye. 2002. A model of digital library in-
formation seeking process (DLISP model) as a frame
for classifying usability problems. PhD diss., Rutgers
University.
Krueger, Janice, Ron L. Ray, and Lorrie Knight. 2004.
Applying Web usability techniques to assess student
awareness of library Web resources. Journal of Aca-
demic Librarianship 30 (4): 285–93.
Kurosu, Masaaki, and Kaori Kashimura. 1995. Apparent
usability vs. inherent usability: Experimental analy-
sis on the determinants of the apparent usability.
Conference on Human Factors and Computing Systems,
292–293. New York: ACM Press.
113
Judy Jeng
Lamb, R. 1995. Using online resources: Reaching for the
*.*s. In Digital Libraries’95, ed. F. M. Shipman, R. Fu-
ruta, and D. M. Levy, 137–46. Austin, TX: Depart-
ment of Computer Science, Texas A&M University.
Lan, Su-Hua. 2001. A study of usability evaluation of
information architecture of the University Library
Web site: A case study of National Taiwan University
Library Web site. Bulletin of the Library Association of
China 67: 139–54.
Landauer, Thomas K. 1995. The trouble with computers:
Usefulness, usability and productivity. Cambridge,
MA: MIT Press.
McGillis, Louise, and Elaine G. Toms. 2001. Usability of
the academic library Web site: Implications for de-
sign. College & Research Libraries 62 (4): 355–67.
Miller, R. B. (1971, April 12). Human ease of use cri-
teria and their tradeoffs. IBM Report TR 00.2185.
Poughkeepsie, NY: IBM Corporation.
Neumann, Laura J., and Ann Peterson Bishop. 1998.
From usability to use: Measuring success of testbeds
in the real world. http://forseti.grainger.uiuc.edu/
dlisoc/socsci_site/dpc-paper-98.html (accessed Au-
gust 24, 2002).
Nielsen, Jacob. 1993. Usability engineering. Cambridge,
MA: Academic Press.
Nielsen, Jacob. 2000. Why you only need to test with 5 users.
http://www.useit.com/alertbox/20000319.html
(accessed May 1, 2003).
Nielsen, Jakob, and Robert L. Mack, eds. 1994. Usability
inspection methods. New York: Wiley.
Nielsen, Jacob, and Rolf Molich. 1990. Heuristic evalua-
tion of user interfaces. Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems:
Empowering People, 249–56. New York: ACM Press.
Oulanov, Alexai, and Edmund F. Y. Pajarillo. 2002.
CUNY+ Web: Usability study of the Web-based GUI
version of the bibliographic database of the City Uni-
versity of New York (CUNY). The Electronic Library
20 (6): 481–87.
Paepcke, Andreas, S. B. Cousins, Héctor García-Molina,
S. W. Hassan, S. P. Ketchpel, M. Roscheisen, and Terry
Winograd. 1996. Using distributed objects for digital
library interoperability. Computer 29 (5): 61–68.
Pearrow, Mark. 2000. Web site usability handbook.
Rockland, MA: Charles River Media.
Pearson, Sammy Wray. 1977. Measurement of computer
user satisfaction. PhD diss., Arizona State University.
Popp, Mary Pagliero. 2001. Testing library Web sites:
ARL libraries weigh in. Paper presented at the As-
sociation of College and Research Libraries, 10th Na-
tional Conference, Denver, Colorado, March 15–18,
2001. http://www.ala.org/ala/acrl/acrlevents/
popp.pdf (accessed January 11, 2005).
Rosson, M. B., and J. M. Carroll. 2002. Usability engineer-
ing: Scenariio-based development of human-computer
interaction. San Francisco: Morgan Kaufmann.
114
Rubin, Jeffrey. 1994. Handbook of usability testing: How to
plan, design, and conduct effective tests. New York: John
Wiley & Sons.
Rushinek, Avi, and Sara F. Rushinek. 1986. What makes
users happy? Communications of the ACM 29 (7):
594–98.
Shackel, Brian. 1981. The concept of usability. Proc-
eedings of IBM Software and Information Usability Sym-
posium, Poughkeepsie, NY, September 15–18, 1–30; and
in J. L. Bennett, D. Case, J. Sandelin, and M. Smith
eds. 1984. Visual Display Terminals: Usability Issues
and Health Concerns, 45–88. Englewood Cliffs, NJ:
Prentice-Hall.
Shackel, Brian. 1986. Ergonomics in design for usability.
In People & computers: Designing for usability. Pro-
ceedings of the second conference of the BCS HCI
specialist group, ed. M. D. Harrison and A. F. Monk.
Cambridge: Cambridge University Press.
Shackel, Brian. 1991. Usability – Context, framework,
definition, design and evaluation. In Human Factors
for Informatics Usability, ed. Brian Shackel and Si-
mon J. Richardson, 21–37. New York: Cambridge
University Press.
Snyder, Carolyn. 2003. Paper prototyping: The fast and
easy way to design and refine user interfaces. Boston:
Morgan Kaufmann.
Spool, Jared. M., T. Scanlon, C. Snyder, W. Schroeder,
and T. DeAngelo. 1999. Web site usability: A designer’s
guide. San Francisco, CA: Morgan Kaufmann.
Sumner, Tamara, Michael Khoo, Mimi Recker, and Mary
Marlino. 2003. Understanding educator perceptions
of ”quality” in digital libraries. Proceedings of the 3rd
ACM/IEEE-CS Joint Conference on Digital Libraries,
269–79. New York: ACM Press.
Theng, Yin Leng, Norliza Mohd-Nasir, and Harold
Thimbleby. 2000a. Purpose and usability of digital
libraries. Proceedings of the Fifth ACM Conference on
Digital Libraries, 238–39. New York: ACM Press.
Theng, Yin Leng, Norliza Mohd-Nasir, and Harold
Thimbleby. 2000b. A usability tool for web evaluation
applied to digital library design. Poster presented
at the WWW9 Conference, Amsterdam. http://
citeseer.nj.nec.com/theng00usability.html (accessed
February 24, 2003).
Thomas, Rita Leigh. 1998. Elements of performance
and satisfaction as indicators of the usability of
digital spatial interfaces for information-seeking:
Implications for ISLA. PhD diss., University of
Southern California.
Tractinsky, Noam. 1997. Aesthetics and apparent us-
ability: Empirically assessing cultural and methodo-
logical issues. Proceedings of the SIGCHI Conference on
Human Factors in Computing Systems, 115–122. New
York: ACM Press.
Turner, Steven. 2002. The HEP test for grading Web site
usability. Computers in Libraries 22 (10): 37–39.

Virzi, R. 1992. Refining the test phase of usability
evaluation: How many subjects is enough? Human
Factors 34, 457–468.
Walbridge, Sharon. 2000. Usability testing and libraries:
The WSU experience. Alki: The Washington Library
Association Journal 16 (3): 23–24.
Editorial history:
paper received 30 May 2005;
accepted 4 August 2005.
Usability Assessment of Academic Digital Libraries
Walker, Marilyn A., Jeanne Fromer, Giuseppe Di Fab-
briziio, Craig Mestel, and Don Hindle. 1998. What
can I say? Evaluating a spoken language interface
to email. Proceedings of CHI ’98, 582–589. New York:
ACM Press.
115
Judy Jeng

Appendix A. Usability Testing Questions

The goal of this test is to evaluate the usability of the library’s Web site. I will ask you a series of questions
and would like you to think out loud while you look for the answer. Some questions are easy and some
are more difficult. Do not worry if you can’t find the answer every time. Please remember that we are
testing the effectiveness of the site design and this is not a test of you. The whole test should take less than
an hour. I thank you.

116
Usability Assessment of Academic Digital Libraries

117
Judy Jeng

118
Usability Assessment of Academic Digital Libraries

119
Judy Jeng

120
Usability Assessment of Academic Digital Libraries

121