Professional Documents
Culture Documents
Developing An Evaluation Instrument For E-Commerce Web Sites From The First-Time Buyer's Viewpoint PDF
Developing An Evaluation Instrument For E-Commerce Web Sites From The First-Time Buyer's Viewpoint PDF
Developing An Evaluation Instrument For E-Commerce Web Sites From The First-Time Buyer's Viewpoint PDF
Abstract: This paper presents the process of developing an evaluation instrument specifically for the evaluation
of e-Commerce Web sites from the first-time buyer’s viewpoint. The development process is based on theoretical
discussions of the Web evaluation and Web user satisfaction literature. A draft evaluation instrument was
developed. To enhance its reliability and validity, several iterative trials on e-Commerce Web sites were
conducted. Some modifications were made to the instrument. The final version is capable of evaluating e-
Commerce Web sites effectively. The instrument provides implications to both Web evaluation practitioners and
academics.
Dran, Zhang, and Small (1999) adopted Web site from a first-time buyer’s viewpoint.
Kano’s Model of Quality as a theoretical There is a need to provide theoretical
framework to evaluate the quality of Web sites. justifications when selecting adequate
This model separated product and service evaluation criteria for e-Commerce Web sites.
quality into three levels according to customer
expectations: expected, normal, and exciting. 3. Web user satisfaction
These researchers believe that quality in a
product or service is not what the provider or For any business, the key to success is repeat
seller put into it, but what the client or customer business from the same customers (Barnes,
receives from it. Thus, a Web site should try to 1999). It is the same in the Web environment.
satisfy its customers’ needs in order to ensure A Web site can be considered successful if
repeat visits from them, and gain their loyalty. users are satisfied and revisit it. Satisfied users
may spend longer at a Web site, may revisit
In regard to Web design, Shneiderman (1997) the Web site later, and may recommend the
provided an Objects/Actions Interface (OAI) Web site to others (Zhang et al., 1999). It is
model for Web-site design. This encourages crucial to determine what makes a user
designers of Web sites to focus on analyzing satisfied with the Web site, as well as what are
the relationship between task and Web potential causes of dissatisfaction. To this end,
interface. Wan and Chung (1998) looked at Web evaluators must first know who the users
problems in Web design from the perspective are, what the key goals of those users are, and
of network analysis. They suggested that care then they have to know what steps the users
must be taken when designing the homepage, are going to take to use that site (Bacheldor,
which is the entrance to the Web site. A 2000).
homepage should keep the center or median
in a Web site. Gehrke and Turban (1999) 3.1 Who are the users?
suggested five major categories that should be The users of a Web site are various groups,
considered when designing a Web site for a such as suppliers, buyers, shareholders, or
business: page loading, business content, stakeholders. It is also very important to
navigation efficiency, and security and distinguish between first-time, intermittent, and
marketing/consumer focus. They argued that frequent buyers of a Web site (Shneiderman,
page loading is the most important factor in 1997). For example, first-time buyers usually
Web-site design. Thelwall (2003) suggested need an overview to understand the range of
shifting the focus on evaluating Web design services, and to know what is not available,
from individual pages to aggregated collections and buttons to select actions. In contrast,
based upon Web directories, domains, and frequent buyers demand shortcuts or macros
entire site. to speed-repeated tasks, compact in-depth
information and extensive services to satisfy
Undertaking a usability study usually needs their varied needs. The user group focused on
high consumer or user involvement, and in this paper is first-time buyers.
sometimes the study needs to be conducted in
an experimental environment. Nielsen (1993,
1995) provided guidelines and criteria to 3.2 What is first-time buyers’ goal?
evaluate the usability of Web sites design and The goal of first-time buyers is to conduct e-
suggested that every design project, including Commerce transaction activities. Several
Web site development, should be subjected to models and frameworks have been proposed
usability testing and other validation methods. to categorize the activities conducted by
Toh and Pendse (1997) also suggested that buyers in the e-Commerce transaction process
Web pages should be designed for usability (Gebauer and Scharl, 1999, Schubert and
and understanding. However, Web sites with Selz, 1997, Schubert and Selz, 1999, Lincke,
good usability cannot guarantee users’ 1998, Liu et al., 1997). For example, Gebauer
preference (Tullis, 1998). and Scharl (1999) described the e-Commerce
transactions process including information,
Although some researchers have tried to negotiation, settlement, and after-sales
provide ways of evaluating e-Commerce Web phases. Schubert and Selz (1997, 1999) and
sites specifically (e.g. Boyd, 2002, Merwe and Schubert and Dettling (2002) divided online
Bekker, 2003), the selection of evaluation transaction process into information,
criteria still requires more theoretical agreement, settlement, and community
justification. Overall, frameworks and criteria phases. Merwe and Bekker (2003) regard the
have been proposed to evaluate e-Commerce transaction process as consisting of need
Web sites. However, few have evaluated the recognition, gathering information, evaluating
information, and making the purchase. Overall, for completing transactions on the e-
these models share a certain degree of Commerce Web sites (see Table 1).
similarity. This paper has adopted the four- Table 1: Activities in each transaction phases
phase model suggested by Gebauer and
Transaction Activities
Scharl (1999). Schubert and Dettling (2002) Phase
also adopted this model as a basis to extend Information 1.find news; 2.find information for
their original model. Details of each phase are specific subjects; 3.find a new
described as follows. product’s information; 4.find a
new product’s price; 5.find a
The information phase comprises both known product’s information;
searching for a particular electronic catalog or 6.find a known product’s price;
information, and locating required information 7.overview company; 8.check
and commodities within the Web site. Buyers financial status.
seek and collect information on potential Negotiation 1.negotiate contract; 2.negotiate
price; 3.negotiate volume;
products or services in this phase. The Web
4.negotiate delivery date.
functions supporting the activities in this phase Settlement 1.conduct payment; 2.monitor the
are, for example, the company overview, goods or services; 3.exchange
product catalogs, news releases, and the financial documentation.
financial statements. After-Sale 1.find maintenance information;
2.ask questions; 3.feedback
The negotiation phase serves to establish a expression; 4.request training
contract, fixing details such as product program
specifications, and payment. Buyers seek
transaction information and decision support 3.3 Why are they satisfied?
by assessing the value of special offerings, by
identifying new bargaining options, and by Kim et al. (2003) suggested that the factors
increasing the negotiations. The Web functions that affect user satisfaction on the Web are
supporting these activities are, for example, attractiveness and informativeness.
email addresses, phone numbers, fax Attractiveness is defined as the quality of
numbers, and online communication physical settings of the Web site that attracts
applications that support the buyer to be able customers and/or involvement (Kim et al.,
to deal with the suppliers online. 2003). It depends on three criteria:
customization, interactivity, and vividness.
In the settlement phase, transaction activities Informativeness is defined as logical settings of
and procedures, which are part of the contract, the Web, which provide visitors with useful and
are comparatively well defined. Web sites to understandable information (Kim et al., 2003).
support transaction settlement include extranet It comprises three evaluation criteria:
systems, and various tools to process orders understandability, reliability, and relevance.
internally and between transaction partners,
facilitate order tracking, and support payment Zhang et al. (1999) adapted Herzberg’s Two-
processes. The Web functions are, for Factor Theory to explain the difference
example, the payment function, the document between satisfaction and dissatisfaction. Job
exchange, and the order status. dissatisfaction occurs when a group of
“hygiene” factors are absent (Zhang et al.,
In the after-sale phase, proper access to the 1999). Hygiene factors describe extrinsic
transaction file is crucial. Without this, factors that impact on employees’ relationship
communication problems and delays can to the context or environment where they do
occur. The electronic support of after-sale their jobs. These hygiene factors remove job
activities is diverse. It ranges from simple dissatisfaction; however, they do not cause
electronic mail services to automated people to become highly satisfied and
helpdesks and sophisticated electronic motivated in their work. In contrast, job
maintenance manuals. The Web functions satisfaction is determined by a group of
supporting the activities in this phase include, intrinsic factors named “motivators” (Zhang et
for example, the email service, electronic al., 1999). Motivators describe employees’
maintenance manuals, FAQs, and training relation to what they are doing. One example
programs. of this used by Zhang et al. (1999) is that fast
loading time will not result in user
Based on a hierarchical decomposition dissatisfaction, but may not be enough to
(Shneiderman, 1997) of user’s activities in guarantee user satisfaction.
these phases, nineteen activities are specified
In addition, Zhang et al. (1999) identified three buyers to complete a transaction activity.
components contributing to Web user Secondly, several evaluation criteria are
satisfaction or dissatisfaction with a Web presented according to the three failure points.
interface: information seeking strategy, user They provide more detailed measurements on
characteristics, and Web environment. The the degree of satisfaction that buyers perceive
strategy or approach a person uses to seek when they reach each failure point. The
information may be analytic (planned, goal- following sections will discuss more details on
driven, deterministic, and formal) or browsing each step of the two-step process.
(opportunistic, data driven, heuristic, informal
and continuous) (Zhang et al., 1999). The Web 4.1 Identify three failure points
interface that supports these two strategies is
different. For the analytic searching strategy, it Three failure points are identified in the
is heavily dependent on the functionality of proposed satisfaction model, which are
search engine algorithms, while the browsing numbered 1, 2, and 3 in Figure 2.
strategies require a Web user interface that
supports “easy and flexible control, high-quality To find the
Failure Point 1
display, and rapid response time”. The factors function on the
of user characteristic and Web environment Web
can be considered as either a hygiene or Failure Point 3
motivating factor, depending on the individual
differences (Zhang et al., 1999).
To conduct To produce
4. Development of evaluation transaction customer
instrument activities satisfactions
This paper proposes a model to demonstrate
and show how an e-Commerce Web site can
satisfy its buyers (see Figure 1).
To find the Failure Point 2
function on the Figure 2: The three failure points in the
Web proposed satisfaction model
Failure point 1 occurs when: buyers cannot
access the Web site, or buyers cannot find the
function or information they want. Failure point
2 occurs when: the function does not work, or
To conduct To produce buyers do not know how to use the function or
transaction customer the information is useless. Failure point 3
activities
satisfactions occurs when buyers do not feel satisfied
although they may not be dissatisfied.
Dissatisfied Satisfied
Section 4.2 Choice of evaluation criteria
Section I Section II III
Although many evaluation criteria are
proposed in the literature, the focus here is to
Failure point 1, 2 Failure point 3
choose those criteria which serve to determine
whether the first-time buyer can pass through
Figure 3: The three failure points and the three failure points in Figure 2 to become
customer satisfaction satisfied with the Web site. Four suitable
Failure points 1 and 2 can be used to measure criteria have been chosen: ease-of-
whether the buyer is dissatisfied with the site. identification, ease-of-use, usefulness and
The failure point 3 is used to measure whether interactivity. Ease-of-identification is used to
the buyer is satisfied with the site. Section I measure whether the buyer has passed
represents those buyers, who are dissatisfied through failure point 1. Ease-of-use is used to
with the Web function, because they can not measure whether the buyer has passed
complete the required activity by using it. through failure point 2. Usefulness and
Section II covers those buyers who can interactivity are used to measure whether the
conduct their activities or the Web function buyer has passed through failure point 3.
fulfills their needs. However, they do not want Table 2 shows the ability of each criterion to
to use other functions. Section III represents measure the three failure points.
those buyers who are satisfied by the function.
They will try to find other functions to complete
the rest of their business activities.
Table 2: The ability of each criterion to measure the three failure points
Ease-of-identification Ease-of-use Usefulness Interactivity
communication between the Web provider and Previous work applied original five-point Likert
its buyers. Functions belonging to this category scales (1 to 5) to evaluate the effectiveness of
have the highest level of interactivity. different criteria in Web environment, such as
user satisfaction (e.g. Sing, 2004), and ease-
4.3 Choice of scoring systems of-use (e.g. Misic and Johnson, 1999, Dai and
Grundy, 2003, Lii, Lim, and Tseng, 2004).
According to the previous discussion on Instead of using the original scoring system (1
satisfaction and dissatisfaction (see Figure 3), to 5), this paper has adapted a system ranging
the effectiveness of the four criteria (ease-of- from 0 to 10. The purpose here is to make a
identification, ease-of-use, and usefulness) is larger variance between the results obtained
based on the degree of satisfaction the buyer from the Web site evaluation. The evaluator
perceives. To measure a degree, it is therefore will receive the benefit of it being easier to
more appropriate to use a multiple-scale monitor the gap between superior and poor
scoring system rather than two-scale (e.g. Yes designed Web functions and Web sites.
or No). Modified five-point Likert scales have Together with the criterion interactivity, which
therefore been chosen for this purpose. is a multiple-choice measure, the scores of
each scale for each criterion are shown in
Table 3.
Table 3: Scoring systems for the four criteria
Criterion Levels and Scores
Ease-of- very easy, right easy normal very difficult (2.5) can not find (0)
identification away (10.0) (7.5) (5.0)
Ease-of-use very easy, no help easy, no help normal, can use, very difficult, do not know how to use
needed (10.0) needed but need help need much help or does not work
(7.5) (5.0) (2.5) (0)
Usefulness of very useful (10.0) useful normal not useful (2.5) no usefulness or can
information (7.5) (5.0) not find (0)
Interactivity interactive content dynamic static content
(10.0) content (2.0)
(2.0)
Based on the scoring systems in Table 3 and 5. Conclusion
the nineteen transaction activities in Table 1, a
draft evaluation instrument was developed. It This paper has focused on developing an
comprises a series of questions asking how evaluation instrument for e-Commerce Web
effective the buyer perceived the evaluation sites from a first-time buyer’s viewpoint. It has
criteria after conducting the nineteen proposed a useful evaluation instrument. As
transaction activities. Several iterative field the importance of e-Commerce increases, the
tests were conducted to enhance the instrument will be especially important for
instrument’s reliability and validity. A group of those businesses that are currently embracing
management, computer science, and e-Commerce to evaluate their Web sites. Not
education students in New Zealand were only can it differentiate the ability of the site to
involved. To enhance reliability, they were support first-time buyers to conduct transaction
asked to evaluate the same e-Commerce Web activities, but also measures how well each
site within twenty minutes using the draft Web function supports the transaction
instrument. If there was a significant difference activities.
among their results, some modifications would
be made to the instrument. Then, they were The instrument has several strengths. Firstly, it
asked to evaluate on another Web site until the can evaluate different types of e-Commerce
difference of the results was not significant. To Web sites. Guideline-based models are
enhance the validity, one management student generally grounded on practical experience.
was asked to use the instrument to evaluate These guidelines usually assess “good” or
forty e-Commerce Web sites and differentiate “bad” Web resources, particularly in the
them. In the end, many suggestions were usability test. The limitation of this kind of
received and contributed to modifying the model comes from the difficulty in applying it to
instrument. Some of the nineteen transaction various kinds of sites. Compared to this kind of
activities identified previously were combined, model, the proposed instrument is capable of
and fourteen transaction activities were assessing miscellaneous sites. Secondly, the
selected. Three instructions were added to the evaluation instrument does not need to access
instrument. The final version of the evaluation specific information in the company (such as
instrument is shown in Appendix 1. company’s marking strategy) to select
evaluation criteria. As Bauer and Scharl (2000)
have noted, designing evaluation criteria
usually requires access to company evaluate sites from diverse industries. It can be
information, which frequently is not available. employed more often to evaluate e-Commerce
The evaluation instrument has overcome this sites in the future.
difficulty. Thirdly, it is easy to use. Usually,
evaluators need specific background about the References
terms used in the frameworks when using the
evaluation frameworks. However, it has been Agarwal, R. and Venkatesh, V. (2002)
through several iterative tests. Evaluators can “Assessing a firm's Web presence: A
use this instrument easily by following the heuristic evaluation procedure for the
steps and descriptions within it and without measurement of usability” Information
knowing specific terms. Fourthly, the Systems Research, Vol 13 No 2 pp
evaluation time is less when using the 168-186.
instrument to assess sites in comparison with Bacheldor, B. (2000) “Web-site design:
using other evaluation models (e.g. Merwe and Simplicity pays for business-to-
Bekker, 2003). Finally, it is a cheap evaluation business sites” Informationweek, News
instrument in comparison with some evaluation Release 14 February 2000, Available
software or services. from Information.com:
http://www.informationweek.com/773/d
However, the proposed instrument has some esig2.htm [Accessed 28 August 2000].
limitations. Firstly, it assumes that evaluators Barnes, H. (1999) “Getting past the hype:
search information based on a browsing Internet opportunities for b-to-b
strategy, not an analytical strategy. Buyers with markets” Marketing News, Vol 33 No 3
browsing strategy undertake an information pp 11-12.
seeking approach that depends heavily on the Bauer, C. and Scharl, A. (2000) “Quantitative
information environment and the buyer’s evaluation of Web site content and
recognition of relevant information. They do structure” Internet Research:
not depend on the functions of search engines, Electronic Networking Applications and
unlike the analytical strategy which depends Policy, Vol 10 No 1 pp 31-43.
on careful planning, recall of query terms, Boyd, A. (2002) “The goals, questions,
iterative query reformulation, and an indicators, measures (GQIM) approach
examination of the result (Zhang et al., 1999). to the measurement of customer
Thus, future research should focus on satisfaction with e-commerce Web
developing another instrument based on an sites” Aslib Proceedings, Vol 54 No 3
analytical strategy. Secondly, some Web pp 177-187.
functions may not be accessed because they Cox, J. and Dale, B. G. (2002) “Key quality
are password protected or are required to factors in Web site design and use: An
conduct an actual transaction with the examination” International Journal of
company, for example, the order online, chat Quality & Reliability Management, Vol
with a seller, or the payment function. Thus, 19 No 7 pp 862-888.
their usefulness and ease-of-use can not be Cunliffe, D. (2000) “Developing usable Web
evaluated fully. Even the proposed evaluation sites: A review and model” Internet
instrument has provided specific criteria to Research: Electronic Networking
measure. However, the full range of Application and Policy, Vol 10 No 4 pp
measurement is not created until they are 295-307.
accessed. Finally, platforms of e-Commerce Dai, X. and Grundy, J. (2003) “Customer
are still in the stage of evolution. Dominant perceptions of a thin-client micro-
players, such as Cisco, Dell, IBM, and Ariba, payment system: Issues and
are continually developing newer generation of experiences” Journal of End User
platforms. The fourteen Web functions and Computing, Vol 15 No 4 pp 62-77.
transaction activities chosen in the evaluation Day, A. (1997) “A model for monitoring Web
form might need to be extended in the future. site effectiveness” Internet Research:
More effective Web functions have to be Electronic Networking Applications and
added in and calculated when one uses the Policy, Vol 7 No 2 pp 1-9.
form to measure e-Commerce Web sites. Dran, G. M., Zhang, P. and Small, R. (1999)
“Quality Web sites: an application of
In conclusion, the evaluation instrument is the Kano model to Web-site design”
capable of evaluating e-Commerce Web sites. Paper presented at the Fifth Americas
It is based on a theoretical discussion, and can Conference on Information Systems,
assist an evaluator to oversee the site easily. Milwaukee USA.
This instrument can also be applied to Gebauer, J. and Scharl, A. (1999) “Between
flexibility and automation: An
Criteria Form 1
Criterion 1: How easy is it to use the function to Criterion 2.2: How useful is the information found?
find one piece of information?
A – Very easy. A – The content of the information is three times the screen.
B – Easy. B – The content of the information is two times the screen.
C – Not easy. C – The content of the information is one screen.
D – Difficult. D – The content of the information is less than one screen.
E – The function could not work. E – Useless.
Criterion 2.1: How informative is the Web Criterion 3: Describe the function and the information found
function? after conducting the activity.