A Comparison of Two Current E-commerce Sites

IBM Ease of Use

User Involvement Team
Mail Stop 9446
11400 Burnet Road
Austin, TX 78758
Roger Tilson: rtilson@us.ibm.com
Jianming Dong: jdong@us.ibm.com
Shirley Martin: smartirL!@us.ibm.com
Eric Kieke: ekieke@us.ibm.com
The increasing importance of e-commerce is apparent in
the recent study conducted by researchers at the GW
I Abstract (Graphics, Visualization, and Usability) Center at the
The IBM Ease of Use Group asked sixteen participants to Georgia Institute of Technology. In their summary of the
compare two e-commerce web sites that sold designer fmdings from the eighth survey, the researchers report that
clothing and two that sold computer products. The primary “electronic commerce is taking off both in terms of the
goal of the study was to pinpoint factors or issues affecting number of users shopping as well as the total amount
the usability of e-commerce sites that need further research. people are spending via Internet based transactions.” Over
The sites that participants used were selected because they half of the 10,000 respondents report having purchased
had received good reviews in the press. One surprising items online. The most cited reason for using the web for
finding from this study, however, was that fourteen out of personal shopping was convenience (65%) followed by
sixteen participants preferred Clothing Site B to Clothing availability of vendor information (60%), no pressure from
Site A. This paper discusses the factors that influenced sales people (55%), and saving time (53%).
participants’ preferences. Specifically, Site B provided Although the issue of security remains the primary reason
participants with easy-to-use product lists, more why more people do not purchase items online, the GVA
obviousness for order links, more feedback on items in the survey also indicates that faith in the security of e-
shopping cart, and more navigation control after adding an commerce is increasing. As more people gain confidence in
item to the shopping list. current encryption technologies, we can expect more and
more users to frequently purchase items online.
7.7 Keywords This shift in the purpose of the mtemet provides businesses
e-commerce, web site design and organizations with many opportunities. It also presents
challenges. Web site designers can borrow upon what they
have learned about e-information web sites when they
Opportunities and challenges of begin facilitating e-commerce, but they must also acquire
some new knowledge to address the challenges that e-
e-commerce commerce presents.
Currently, the intemet can be used for two purposes:
e-information and e-commerce. E-information consists of In fact, many current e-commerce sites are not fully taking
advertising and providing information. E-information sites advantage of the opportunity e-commerce presents. In
sometimes provide email addresses, phone numbers, or Internet World (formerly WebWeek), columnist Whit
street addresses so customers can receive more information Andrews discusses his own sad experiences with bad
or order products, but their primary purpose is not to sell usability when trying to purchase things online. He says,
products online. In contrast, the primary goal of “Problems with consumer adoption of intemet commerce
e-commerce is to selI goods and services online. have almost no connection to issues of technology. . . What
happens when regular folks ny to buy things on the Internet
The first web sites to go online were almost solely e- is insupportably bad business.”
information sites. Increasingly, the intemet is being used
for e-commerce. Many companies have already established Whit Andrews quotes Mark Hurst, president of a consulting
e-commerce web sites, and in the next few years, many firm specializing in Internet ease of use: “It’s ease of use,
more e-commerce sites will come online. Companies will it’s ease of use. Why doesn’t the industry get that?”
develop e-commerce web sites for different reasons: to
Permission to make digital or hard copies of all or part of this work for
reach new customers, to stay abreast with their competitors, personal or classroom use is granted without fee provided that copies
and to meet their customers’ expectations and needs. are not made or distributed for profti or commercial advantage, and that
copies bear this notice and the full citation on the first page. To copy
otherwise, to republish, to post on servers. or to redistribute to lists,
requires prior specific permission and/or a fee.
01998 ACM 1-5811%004-Xl9B/0009 $5.00

87
2 Current and future
e-commerce research
Because practitioners and researchers know little about the
habits of virtual shoppers or how to create effective e-
commerce sites, the IBM Ease of Use Group has begun a
series of studies that will provide e-commerce site
developers with information they need to facilitate easy
shopping. This paper reports the findings from an initial
study that was used to pinpoint factors affecting the
usability of some current e-commerce sites. Future studies
will focus on specific factors in more detail. The data from
this body of research will form the foundation of
e-commerce guidelines to be made available to site
designers within and outside IBM via the IBM Ease of Use
web site (www.ibm.comleasy).
Participants in this initial study used two sites that sold
clothing and two sites that sold computer products. This
paper will discuss the participants’ comparison and use of
the two clothing sites. With the exception of one or two
additional features, Clothing Site A and Clothing Site B
enabled participants to complete the same basic task-to
purchase clothing merchandise. Each of the sites had
received good reviews in the press. Both are “state of the
art” e-commerce sites, but both sites presented usability
problems. One of the surprising findings was that fourteen
out of sixteen participants preferred Site B over Site A.
Although we could not determine from the data the precise
cause and effect for the participants’ preferences, or the
relative significance of each of the issues participants
discussed, we did glean from their comments and from our
observations some of the reasons for the participants’
preference for Site B. These reasons included issues that
are familiar to designers of e-information web sites, as well
as some new issues that are particular to e-commerce.
3 Methods
Eighteen people were asked to evaluate four e-commerce
sites. Two of the eighteen people participated in pilot
testing. The other sixteen participated in data collection.
Participants used Netscape 4.03 to browse, shop, and
compare the sites. Participants viewed the sites on a 17”
monitor with an 800x600 screen resolution.
We asked participants to compare two sites of the same
type so that they could tell us what they liked and disliked
baaed on the comparison. The comparative data was used to
determine factors that influence the use of e-commerce
sites.
Half of the participants had purchased an item online, and
half had not. We were not sure whether and how online
shopping experience would influence participants’ use and
88
evaluation of the sites, so we wanted a balance of users
who had and had not purchased items online.
One limitation of this study was that it did not continue
through the transaction process. Because of difficulties
enabling participants to purchase items online using a credit
card, the participants’ use and evaluation of the sites
stopped when they had added four items to the shopping
cart and had clicked check out. We hope to examine the
fmal check out processes, order cancellations, returns, etc.
in later studies.
4 Structure and Navigation
One issue pertinent to both e-commerce and
e-information sites is the amount of time users must spend
navigating born the home page to information about
products or services, and then to other parts of the site.
With both e-commerce and e-information, designers need
to simplify and expedite this navigation process.
Although facilitating easy navigation is important for both
e-commerce and e-information, it may sometimes be more
challenging in the case of e-commerce sites.
E-commerce sites sometimes present information about
hundreds or thousands of products, which complicates
designing the navigation and snucture of the site. If more
items must be presented, hierarchies may become broader
and/or deeper, product categories may become more
numerous, and product lists may become more
complicated.
Such was the case with the sites in this study. An e-
information web site that aims to attract people to real
stores to buy its clothing would not need to provide online
pictures and information for thousands of items of
merchandise. The site instead could present a few new
fashions for the current season. If the goal is to sell online,
however, the site must somehow facilitate navigation to
each of the items that it wants to sell online, which, in the
case of the two sites that participants used, was hundreds of
items.
Successful sites such as Amazon.com illustrate that
facilitating easy navigation is possible, and that the returns
can be tremendous. And for sites with fewer products and
services, navigation may not be more complicated in the
case of e-commerce. Regardless of the number of products
a site sells, the structural and navigation issues that our
participants indicated were important in the design of the
two clothing sites would likely be important for some other
e-commerce sites as well. The following is a discussion of
some of those issues.
4.7 Clicks required to view an item
When participants evaluated the two clothing sites, three
(19O/,) commented that the process of purchasing items
seemed to take longer at Site A than at Site B. None of the
participants commented specifically about the time it took
them to navigate from the home page to a particular piece
of merchandise, but analysis showed that the sites differed
significantly in the number of clicks required to view
product information.
Site A typically required 7 clicks to view an item of
merchandise: 2 clicks to enter the online product catalog, 1
for a category such as casual clothing, 3 to select an item
from a dropdown menu, and 1 to dismiss a security
message.
In contrast, Site B required a maximum of 4 clicks: 1 click
to enter the online shopping catalog, 1 to select a main Figure 1: The product organization at Site A. Five
category such as Menswear, 1 to select a subcategory such participants wanted separate categories for men’s and
as knit shirts, and 1 to select a specific item. women’s clothing.
Site B also provided easy and direct access to some of its
most popular items. In the fust task, participants were
asked to add a parka to their shopping cart. The list of
parkas was on level 4 of the hierarchy at Site B, but 4.3 Scanning and selecting items from a
participants could view level 3 and level 4 at the same time
(See Figure 1). When they arrived at level 3, the category list of products
“Outenvear” already displayed a list of parkas. With this Participants in this study generally liked the design of Site
design, viewing parkas required only 3 clicks at Site B, B’s product list more than they liked the design of Site A’s.
compared to 7 at Site A. Five participants (31%) indicated they like the product list
at Site B (see Figure 1). Only one participant indicated he
liked the dropdown menus as they were used at Site A, and
4.2 Helpful product organization four participants (25%) indicated they disliked them (see
At level two in the hierarchy of Site B, participants chose Figure 2).
which category they wanted to view, and the first two The product list at Site A enabled participants to scan and
categories listed were “Menswear” and “Womenswear.” select items with optimal ease. Site B employed dropdown
Five participants (3 1%) said they preferred Site B partly menus, which was not the best use of the screen real-estate.
because they liked this distinction. Knowing which
category to choose was easy for participants when using Site B used simple hypertext links to display product lists,
Site B. and users could initially see 10 subcategories and over 10
items in each subcategory.
Site A divided boys’ clothing IYom girls’ clothing, but did
not divide “Menswear” and “Womenswear.” Instead, Site A used dropdown menus, and only one item was
participants were first asked to choose between casual initially visible in each subcategory of products. The page
clothing, fleece clothing, outerwear, and other categories. with the dropdown menus also required scrolling. With an
Some participants did not know where they would need to 800x600 resolution, only one dropdown menu was visible
go to accomplish the fast task, which was to find a parka above the first fold. ‘Thus, initially, participants could see
and add it to their shopping cart. only one subcategory and one product in the subcategory
above the fold.

89

Figure 2: Simple Menu List at Site B. Participants could
select a subcategory from the column on the left and
then an item from the column on the right.
Jackets and Pullovers
Shirts
m IAll-Temn Cychng Jersey
. Shorts and Pants
All.Tsrran Cycling Shorts &Knlckers
. Skirts and Dresses
BaysIds Twll Bullon-Fmnl %,I 9 they needed to take to order items. Obviously, if the goal of
Sweaters
Commando Swealer
Figure 3: Dropdown menus at Site A.
Selecting items was also easier at Site B. The dropdown
menus at Site A typically required three clicks to view an
item. First, the participants needed to click the down arrow,
then the item, and then the View button.
Site B required one or two clicks to view an item of
merchandise, depending upon whether or not the
participant wanted to select a different subcategory (i.e.
something besides “Outerwear”) before selecting an item.
Dropdown menus work best in situations in which users are
interacting with content or information on the screen and
still need access to other tools or other types of
information. In the case of Clothing Site A, participants
only needed to select one item from the screen and move
on. Dropdown menus in this case did not make good use of
the available screen space.
90
4.4 Returning to different levels affer
adding an item to the shopping cart
At many e-commerce sites, users will need to be able to
purchase multiple items conveniently. At Clothing Site A,
this process was cumbersome. Each time users added
something to the shopping cart, they were given two
options for continuing shopping. One was to use the search
find other products, and the other was to return to the top of
the shopping hierarchy. Four participants (25%) wanted
additional choices that would enable them to return to the
product category they had most recently used.
Clothing Site B provided these exna choices, although the
labels for the links were not very clear. Links at Site B
were called “Main Directory” and “Product Directory.”
“Main Directory” returned users to the top of the hierarchy.
“Product Directory” returned users to the last product
category they had used.
5 Obviousness and Feedback
Just as with str~~ctural and navigation issues, some of the
obviousness and feedback issues that were important to
participants in this study are important in the design of e-
information sites as well. Others, such as feedback on items
saved in the “shopping cart” and security messages are
particular to e-commerce.
The worst and most embarrassing usability problem for
both sites was that participants did not lmow which link
a site is to sell merchandize, users need to be able to
recognize the “Order” links. For Site A, this problem was
caused by poorly designated “Order” links. For Site B, the
problem was caused primarily by a poorly presented
additional feature.
5.7 Feedback on saved items
One new issue that arises with e-commerce is the
importance of providing feedback when an item is and is
not saved in the shopping cart. Because users of most sites
need to be able to purchase multiple items, they need clear
indication of when an item is safely saved. E-information
sites do not need to provide this type of feedback
Four participants (25%) stated specifically that they wanted
to be able to view the shopping list after each item was
added. Three participants (19%) stated they would like for
the shopping cart to be visible on the screen at all times.
Site B provided a view of the shopping cart after each item
was added, but this view caused some problems. Four
participants (25%) indicated some confusion that there was
no “Continue Shopping” button. Participants worried that
the items in the list would not be saved if they were to click
any link other than “Continue Shopping.”
Unlike Site B, Site A did not provide a view of the user’s
saved items in the shopping cart after an item was added.
Instead, Site A provided a link to the shopping cart. Five
participants (31%) either clicked on the link to view it or
did not see the link and wanted to be able to view the
shopping cart.
A persistentview of the shopping cart might help solve one
of the biggest potential usability problems of online
shopping. Nine participants (56%) in this study indicated
confusion or uncertainty concerning when items were
saved in the shopping cart. Five participants (31%)
continued shopping thinking they had added items to their
cart when they had not. Similarly, some participants tried
checking out becausethey were not certain they had saved
the item in the list. A persistent view of the shopping cart
would likely help users see when items have been added
and when they have not been added.
5.2 Obviousness of “Order” /inks
For e-information and e-commerce,usersneedto be able to
recognize which graphics are links and which are purely
decorative. Ten participants (63%) did not at fust recognize
the “Order” links at Site B. The problem was that the link
did not look like a link or a button.
Figure 4: Unclear “Order” link at Site A.
Had the link been underlined and blue, participants may
have recognized it more easily. Had it been button-like,
they may also have recognizedit more easily.
Also, for Site A the “Order” links did not appearabove the
first fold. The links followed varying lengths of text and
varying numbers of pictures. One participant did not lmow
that she neededto scroll below product pich,uesto &d the
“Order” links.
In contrast,the information for eachitem of merchandiseat
Site B was concise enough that it could fit above the first
fold. The order buttons consistently appearedin the same
place beneatha product picture above the first fold.
5.3 Presentation of additional features
One of the benefits of online shopping is that it can
potentially offer additional features that make shopping
easier than it is offline. One such additional feature is a
regislry, or place where users can store items that they want
others to buy for them for special occasions, such as
weddings, birthdays, etc. Designers, nevertheless,need to
91
ensure that these additional features do not confUse the
shoppingprocess.
The biggest usability problem with Site B was that
participants became confused by the presentation of the
registry. They thought that it was the link they neededto
take to savean item to the shopping cart.
Figure 5: Add to registry and order link at Site B.
Nine out of sixteen participants mistook “add to Wish List”
for the link they would use to order the item. The “order
ITEM NOW’ soundedtoo much like checking out, and the
participants were asked to save additional items to the
shopping cart before checking out.
Site A also causedsomeproblems with the presentationof
an additional feature. After adding an item to the shopping
cart, participants were given an opportunity to return to the
top of the hierarchy to continue shopping or to use a
catalog order number to order an additional item. The
problem was that three participants mistook the catalog
order number feature for an entry box for a site search.
5.4 Security messages
The issue of security is what keeps many people from
buying products online. Although people will gradually
over time becomemore comfortable with the security of e-
commerce sites, designers must in the meantime hy to
make shoppersfeel secure.
This issueis both a structure and navigation issue as well as
a feedbackissue. Currently, unless the user requestsnot to
be notified, the Netscapebrowser displays messageswhen
users enter and exit a secure area. Users are concerned
about security issues and want to know their transactions
are secure,but they do not want to be hassledby too many
recurring messages.
Recurring security messageswere a problem with Site A.
Sevenparticipants (44%) indicated annoyanceor confusion
at the recurring messages.Site A displayed three messages
in the processof adding one item to the shopping cart. At
least two of these messageswere unnecessary.A message
that the transaction was insecurewas followed immediately
with a messagethat the transaction was secure.Then, after
specifying size and color, participants needed to dismiss
another message that the transaction was secure. In
contrast, Site B displayed no security messagesin the
processof adding an item to the shopping list and returning
to the top of the hierarchy. The security messagecame
when usersclicked check out.
 6 Important Usability Factors and
Future Research
The main purpose of this study was to pinpoint some of the
important factors that affect the usability of e-commerce
sites. Participants in this study wanted and needed help
with the following:
l Knowing when an item was saved or not saved in the
shopping cart
l Returning to different parts of the site after adding an
item to the shopping cart
. Easy scanning and selecting items in a list
l Effective categorical organization of products
. Simple navigation l?om the home page to information
and order links for specific products
l Clear presentation of additional features, such as a
registry, that do not complicate the shopping process
l Obvious shopping links or buttons
. Minimal and effective security notifications or
messages
Although users did not explicitly or implicitly comment on
the following, they may have wanted or preferred this
assistance as well:
l Consistent placement of order links on a page
l Placement of order links above the first fold
. Consistent layout of product information
For most of the above issues, we need to conduct further
studies in which we con&o1 for variables. For instance, a
persistent view of the shopping cart might be beneficial, but
there is the issue of bandwidth. Any firm and specific
recommendation would be premature without lmowing the
implications on load time.
92
Future research needs to control for specific variables. For
instance, participants could use a site version that uses a
persistent view of the shopping cart and a different group
would use a version without a persistent view. The first
goal, as far as feedback on the shopping cart is concerned,
would be to see if a persistent view of the shopping cart
helps users recognize when an item is added to the cart and
when one is not added. If it is indeed of value, then we
need to experiment the implications on load time using
different speed modems.
Similarly, we also need to present test participants with
different prototypes for navigating after saving an item to
the shopping cart. We could ask participants which they
prefer.
Until we have more specific information and connol for
specific variables, designers need to be aware of the issues
discussed here and by other researchers. E-commerce site
designers can test and report their fmdings to other
practitioners. The IBM Ease of Use Group will conduct
follow-on studies of the issues that the initial research
indicated was important.
7 References
Anclrews, Whit. “At Far Too Many Sites, ‘Buyer Be Lost’
Applies.” Internet World. Feb. 2, 1998.
http://www.intemetworld.com/
“GVU’s 8th WWW User Survey.” GVU’s Www User Surveys
http://www.gvu.gatech.edu/user~surveys/suivey-1997-10/