Usability Testing: Evaluating Travel Websites

March 8, 2006

Yu-Ju Lin (Sharon)

Siribang-orn Puttitwong (Aey)

Ya-Wen Yang (Michelle)

 Contents

Introduction 3

Procedure 3

User persona 4

User scenario 5

Evaluation criteria 5

Result of evaluation 10

User Comments 13

Discussion of Website Evaluation 15

Website with bad design 21

Proposed design 22

Who did what 23

2
Introduction
It’s time to relax. If you keep working for few years, a good vacation not only
relieves your body and mind, but also eliminates your tenseness for the busy life. But
how do we select a good trip for our coming vacation? Most people will choose to take a
look at the travel websites to search for further information. I am no exception. What is a
good travel website? From the perspective of the information, interaction and interface
design, how do they help you to find out the most informative website that meets your
needs? Therefore, through our evaluation report, hopefully, you’ll have a wonderful trip
in the near future.

Procedure
After reading the instruction of the assignment, we begin with creating the user
persona and user scenario that serve as our guiding situations for the entire evaluating
processes. With guiding situations in mind, we explore the provided websites and read
the given examples in order to get overall ideas for the rest of evaluating processes. After
that we have a meeting to continue working on the evaluation.
First, we select four websites that will be evaluated. That is, www.orbitz.com,
www.travelocity.com, www.travel.com and www.expedia.com. When we agree with the
choices, we start to create evaluation criteria by adapting from the provided examples and
the guiding situations. Next, we assume a user role to evaluate the selected websites by
using the agreed evaluation criteria. After evaluating the selected websites, we assume a
designer role to summarize the evaluation results and discuss about the good and bad
designs of the evaluated websites. The worst website, www.travel.com, is selected for
redesigning purpose. Finally, we discuss about the good and bad designs as well as the
problems of this website in detail in order to brainstorm team’s ideas for redesigning it.

3
User persona
To begin the process, we began with a user persona to describe travel website
evaluation.
Name: Debbie
Age: Early thirties
Occupation: Pflugerville Middle School
teacher
Home: Austin, TX

Bio:
Debbie has been teaching in middle school for three years. She loves her job but
she really needs a vacation to get away from schoolwork. Single, and Being working for
3 years, she has little saving and plans to relax herself in beautiful seashore with sunshine
in this spring break. There is a little late for her to go through all the information, booking
the flight, hotel, car and everything. The spring break is coming soon. Compared to other
colleges, they are all well prepared for their vacation that makes her even more anxious.
Motivation: Anxious to surf online to find a nice, reasonable, and safe trip to Caribbean.
Needs:
She plans to spend about 2,500 (3,000 up most) dollars for her vacation. That will
be great for her to get off the work on Friday and directly rush to airport and start her
spring break. She would like to stay in Caribbean until the last day of the vacation as long
as the budget is allowed.
She doesn’t like either to transfer flights or spend too much time on flights. She
would like to choose an airway company with shorter flight hours. And of course, the
amenities in the hotel are also under her consideration. She would like to have a sea-view
room. She is expecting in the little island there could be a certain festival for the tourists
to take part in. Most important of all, safety is always her prior concern.

4
User scenario:
Next we created a user scenario that describes how the user would achieve their
goal on travel websites. Debbie is always exhausted after work but now she has to make
quick decision for her plan. She knows Caribbean will be a good place for her to spend
spring break. But she has no idea which island she should choose, how much the flight
and the hotel would cost her. Her goal is clear—to arrange a decent trip within a short
time. She has no patience to go from one website to another to book flight, hotel, and car
separately. She also hopes the website will provide the lowest bargain at the same level
flight, or the same rating hotels.
So the first step, she will look for the site with a search engine, simple and clear.
She ignores the flashy ads, and she will not read if the whole website is full of texts. If
the website provides the full packages (flight, hotel, and car), she will stay longer and try
to compare the differences of different flight schedules and hotels, especially for the
hotel’s rating, amenities.
Before she decides to click on booking button, she will also hopes the website
can provide the information about Caribbean—Are there any activities during the spring
break? Are there any travelers’ recommendation or comments about certain hotels, or
how to arrange the trip, where will be the most beautiful attractions that you will never
want to miss. She hopes the website can provide “one-stop” service.
As for the safety concern, she would like to know if the websites also provide the
travel insurance, or provide more information about the weather, and what the travel to
Caribbean necessities are. If the website cannot satisfy her, the user scenario will repeat
again at other travel website.

Evaluation criteria
The criteria are composed of two major parts. The first part aims to acknowledge
users’ overall feelings of each selected website and the second part aims to gain the
insight information about the information, interface, and interaction designs of each
website. The following is the evaluation criteria that we use.
For each word below, please indicate how well it describes the website you have been
looking at throughout the study.

5
 Disagree Agree
Attractive * * * * * * * * * *
Believable * * * * * * * * * *
Boring * * * * * * * * * *
Concise * * * * * * * * * *
Engaging * * * * * * * * * *
Easy to use * * * * * * * * * *
Time efficiency * * * * * * * * * *
Helpful * * * * * * * * * *
Interesting * * * * * * * * * *
Likeable * * * * * * * * * *
Website revisit * * * * * * * * * *
Frustrating * * * * * * * * * *

Please select the appropriate dot for each following questions:

Information Design
1. Do you feel that you found the needed information?
Not at all satisfied Very satisfied
* * * * * * * * * *

2. Is the acquired information accurate?

Inaccurate Accurate
* * * * * * * * * *

3. Does the website fulfill its intended purpose?

Not Agree Agree

* * * * * * * * * *

4. Does the title of the page match the content information?

6
Not Agree Agree

* * * * * * * * * *

5. Does the website present most updated information?

Not Agree Agree

* * * * * * * * * *

6. Does the website offer external links to relevant sites?

Not Agree Agree

* * * * * * * * * *

7. Does the website provide customer service?

Not Agree Agree

* * * * * * * * * *

8. How easy is it to find specific information in this Web site?

Very difficult Very easy

* * * * * * * * * *

9. Is the information used in the site easy to understand?

Very difficult Very easy
* * * * * * * * * *

7
Interface Design

10. How easy is the site’s navigation?

Very difficult Very easy
* * * * * * * * * *

11. How easy is it to read the texts in this Web site?

Very difficult Very easy

* * * * * * * * * *

12. How easy is it to concentrate on searching for information (because of distractions)?

Very difficult Very easy

* * * * * * * * * *

13. Does the flow/process of information flow smoothly?

Not Agree Agree

* * * * * * * * * *

14. Does the navigation provide reversibility?

Not Agree Agree

* * * * * * * * * *

8
15. Are the fonts, sizes, colors of texts appropriate and consistency?

Not Agree Agree

* * * * * * * * * *

16. Are the tables, charts, or graphics on the website is readable?

Not agree Agree

* * * * * * * * * *

Interaction Design

17. In terms of searching process, how easy is it to search for the desired information?

Very difficult Very easy

* * * * * * * * * *

18. Compared to what you expected, how quickly did the tasks go?

Very slowly Very quickly

* * * * * * * * * *

19. Are you willing to register to be a member before you book the ticket/ the product?

Not agree Agree

* * * * * * * * * *

9
Results of evaluation
There are four figures showing the evaluation results. The first one shows the
users’ overall feelings towards the evaluated websites. For the rest of the figures, they
demonstrate information, interface, and interaction designs of the evaluated websites
respectively.

Overall Evaluation

Frus tra ting

We bs ite re vis it

Like a ble

Inte re s ting

He lpful
Evaluation Criteria

www.exp ed ia.co m

Time e ffic ie nc y www.travel.co m

www.o rb it z.co m
Ea s y to us e www.travelo cit y.co m

Enga ging

Conc is e

Boring

Be lie va ble

Attra c tive

0 1 2 3 4 5 6 7 8 9 10

Average Point

Figure 1: Overall evaluation

The above figure shows user’s overall feeling towards the evaluated websites.
Users felt that www.travel.com is the most frustrating and boring sites compared to the
others. Www.expedia.com is the site that gains the highest score in six positive criteria.
Finally, both www.orbitz.com and www.travelocity gain the highest score in three
positive categories.

10
 Information Design

Is the information used in the site easy to understand?

How easy is it to find specific information in this Web site?

Does the w ebsite provide customer service?

Does the w ebsite offer external links to relevant sites? www.exp ed ia.co m

www.travel.co m
Criteria

Does the w ebsite present most updated information? www.o rb itz.co m

www.travelo city.co m

Does the title of the page match the content information?

Does the w ebsite fulfill its intended purpose?

Is the acquired information accurate?

Do you feel that you found the needed information?

0 1 2 3 4 5 6 7 8 9 10

Average Point

Figure 2: Information Design

Using four websites, users agreed that both www.expedia.com and

www.orbitz.com are the easiest websites to understand. However, users felt that
www.travelocity.com is the easiest site that allows them to find specific information and
www.travel.com is the only site that offers them external useful links. Moreover, users
agreed that www.orbitz.com presents the most updated information. Www.travel.com
was rated in the lowest rank compared to the others, especially in the aspects of matching
between title of the page and the content, fulfillment of the site’s intended purpose,
accuracy of acquired information, and the success in finding needed information. Finally,
all websites offer customer service but only www.expedia.com and www.travelocity.com
do the best job in making the service easy to spot on their homepages.

11
 Interface Design

Are the tables, charts, or graphics on the w ebsite is

readable?

Are the fonts, sizes, colors of texts appropriate and

consistency?

Does the navigation provide reversibility?

www.exp ed ia.co m
Criteria

www.travel.co m
Does the flow /process of information flow smoothly? www.o rb itz.co m
www.travelo city.co m
How easy is it to concentrate on searching for
information (because of distractions)?

How easy is it to read the texts in this Web site?

How easy is the site’s navigation?

0 1 2 3 4 5 6 7 8 9 10

Average Point

Figure 3: Interface Design

Noticeably, www.travel.com was placed in the lowest rank in all criteria while
www.orbitz.com was rated in the highest rank in many criteria. Www.expedia.com and
www.travelocity.com were rated nearly equal level in all criteria.

12
 Interaction Design

Are you w illing to

register to be a member
before you book the
ticket/ the product?

www.e xpe dia .c o m

Compared to w hat you
Criteria

www.tra ve l.c o m
expected, how quickly www.o rbitz.c o m
did the tasks go? www.tra ve lo c ity.c o m

In terms of searching
process, how easy is it
to search for the
desired information?

0 1 2 3 4 5 6 7 8 9 10

Average Point

Figure 4: Interface Design

Users felt that they are most willing to register to www.orbitz.com and that it’s
acceptable to register to www.expedia.com and www.travelocity.com prior to booking
the ticket or buying service as well as product. Users also felt that they don’t want to
register to www.travel.com if the site asks them to.
Comparing to their expectation, users agreed that they completed the task quickly
when they used www.orbitz.com and that they spent the longest time when they used
www.travel.com.
In users’ opinion, www.orbitz.com provides the best search function since they
felt that it’s easy to use and to find the desired information and they like www.travel.com
less than the others.

User comments
www.travelocity.com
 Provide flexible travel date (unique feature): useful information for users to make
a decision
 Show clear information about departure and arrival time as well as travel time +
offer better deal at the same screen
 There is a guiding image showing the current process on which users are [search,
select, review, reserve, and confirm].
 Non-registered users can book flight, hotel, and car.

13
  Drop down menus
 Provide users (travelers) review [only for hotel]
www.orbitz.com
 Very simple
 Provide travel package that makes it very convenient for users
 Use tab for main sections
 Clean and simple look [less text, image = less distraction]
 Less affiliating company to provide package service
www.travel.com
 Provide the way to book flight, hotel, and car on the main page
 The search default is “hotel”
 Offer links to pertinent information such as driving directions, insurance, and
weather
 Show interesting deals from other sites as well as show price comparison
 Users are not required to register to the site before they can book anything.
 Busy and cluttered
 Provide incorrect information
 Inappropriate use of font sizes
 The tab for main sections is on the right.
 Advertisement is on the left. It should be put on the right.
www.expedia.com
 Provide the way to book flight, hotel, and car on the main page
 Look nice and clean
 Offer convenient booking for flight, hotel, and car (combined search)
 Doesn’t offer many links to pertinent information
 Users have to register to the site before they can actually book anything
 Use tab for main sections

14
Discussion of Website Evaluation

Followed by clockwise explanation, here is www.orbitz.com. In the Homepage

we see above, the tagline is pretty clear with a pithy phrase “ And Go” for the users to
understand that the website you’re browsing is the travel website. Differentiated with the
general travel websites, in the orbitz.com, the users can write the feedback to the web-
master for the design of the whole website, as shown below. Though it’s not the required
questionnaire, if you have any question to the layout or format in the website, you’re
welcomed to write the feedback to them via e-mail.

15
 In the Membership area, also with the assistance of the tab, the users not only can
sign in the website to conduct your own trip on-line, but also contacts with the customer
service for any problem, any time by e-mail or on phone. Specially, the Orbitz provides a
unique function for the customer, the TLC. It’s a travel alert to inform or remind the
customers for the trip after you’re booking.
Below the membership area, the promotion and best deal part for the current
popular trip plays an important role to offer the users another good price for the trip they
are looking for. Without the overload graph or pop-up commercial, the simple picture and
the organized texts easily catch the user’s attention to click for further understanding the
current promotion or best deals.
Next, instead of the general layout, the navigation part, Orbitz offers the multiple
and special search engine for the users to look for the information they need in a more
convenient and immediate method. In the multiple search, users can combine the flight,
hotel and car search together to get the information they need. Or, they can use the
different way to sort with their own special needs, such as flexible dates or multiple-city
search.
Also, as mentioned before, the Orbitz provides a special service to the customers
to get a safer and more comfortable trip, TLC. The users can know more details about it
in the column below the search part.

16
17
 The interface design of Travelocity is really powerful, especially the function of
setting up arrival/departure time. Function (+- 1~3days) makes the searching result even
more helpful for the users to make decisions. Users see the different prices in the table.
Sometimes, users may shorten or extend traveling days because of the budget. The table
will make users search what they want with efficiency.

18
 The homepage design of www.expedia.com is really good. It uses tabs at the top
of the page to show users its main sections and provides local navigation at the bottom of
the page. This thoughtful navigation plan will ensure that users will know what they can
find from the site and won’t get lost easily. Besides, the logo “Expedia.com” links to the
homepage so users can start over whenever they want. Additionally, the greatest feature
in its homepage is the search feature. It is placed in the proper position which is very easy
to spot. Moreover, users have many choices to select for searching such as flight, hotel,
or combined search—flight+hotel, or flight+hotel+car. Finally, the screen looks very
clean, simple, and relaxing.

19
 On the top-left of the screen, the site offers users a chance to modify their search.
To modify their search, they have three options which are “Start search over”, “Change
your search”, and “Narrow your search”. Besides, each search result also allows users to
change the provided options which are flight and car. Users can sort the search results by
Expedia picks, Price, Hotel Name, City, and Hotel Class. Finally, each search result
provides thorough details about flight, hotel, car, and total price.

20
Website with Bad Design

Why Is A Bad Website Design?

For information websites, the most important criterion is to provide accurate
information. Unfortunately, TRAVEL.com is the first one that we find to provide wrong
information—it only takes 2 hours to go from Austin to St. Croix, Caribbean. Wrong
information can be so lethal that it could totally destroy web viewers’ confidence.
Apparently, TRAVEL.com is kind of proud of “one-stop travel site” appeal.
However, the interface design is neither user-friendly nor time-efficient. The website

21
doesn’t provide the possibility to book vacation package. Besides, the website is text-
heavy. It is designed for customers to “read” but not to “scan”.
As the part for providing the relevant links, some of the concerns are thoughtful,
some are risky. For example, under the search engine, a table is provided to compare the
prices of website competitors. Unless TRAVEL.com can guarantee for the lowest price
nationwide, even worldwide; they are under the risk to send their money out.
There are several inappropriate and inconsistent font, color, and size in the website
design. What worse is, there are some links and buttons can only be understood by the
designer, such as “World Home” and “World Web Links”. Clicking on certain link of the
website will take the viewers into the same website of “World Web Links”. The
redundancy shows the website lacks of organization and carefulness.

Proposed design

22
 In our redesign, we move the tagline from right button to the left, below the
tagline, and also add one more tab, customer service and a hyperlink for the users to write
the feedback for the design of the web pages. In the left hand side, more function is
increased in the searching engine, such as the multiple search and the special search.
Without too many annoying pop-up promotion, one beautiful photo with easy hyperlinks
to the best destination or best deals make the whole web pages more clean and simple.
Though the travel.com tries to offer a table to guarantee their lowest price for the
customers as reference, the easy link with a simple button will attract the user to click in,
instead of the table with texts.

Who did what

All the work in our evaluation report is collaboratively completed by three of us.
But we still have some paper work respectively to finish, as shown below. After
collecting the paper work from the other teammates, Aey and Michelle, I combine all of
the parts together and e-mail to others.
1. Details of task
 1). User Persona (Michelle)
 2). User Scenario (Michelle)
2. Procedure (Steps or things you do) (Aey)
3. Evaluation Criteria (Aey, Michelle, Sharon)
4. Evaluation Results or Discussion
 1). Results of the evaluation (Aey)
 2). Discussion of Website Evaluation (Michelle, Sharon, Aey)
 3). User’s comments (Aey, Michelle, Sharon)
 4). Web sites Screen Shots (Michelle, Sharon, Aey)
5. Website with Bad Design- including brief introduction (Michelle)
6. Redesign of Website with Bad Design (Sharon)

23