User Testing Report

Created by Sana A. Indap for Coursera Course

Evaluating Designs with Users
8/14/20
Executive Summary

This report documents a Formative user testing study by Sana Indap on

Expedia.com focused on how expert users plan their trips (flights and hotel
accommodation). Formative tests are used in an iterative process to find problems and
make improvements in a product. The researcher performed usability testing sessions
with 2 individuals to identify what usability issues exist in the interface when used by
frequent flyers to plan trips.

Expedia Group is an American online travel shopping company for consumer and
small business travel. Its websites, which are primarily travel fare aggregators and travel
metasearch engines, include CarRentals.com, Expedia.com, HomeAway, Hotels.com,
Hotwire.com, Orbitz, Travelocity, trivago and Vrbo. Expedia was created as a division of
Microsoft in 1996 and became its own company in 1999. The website lets users plan
trips, book flights, cars, cruises, and much more. Some aspects of the website are easy to
use while others are a little more complex, but overall the website isn’t too confusing or
cumbersome.

The criteria for participants are preferably working adult professionals who have
booked flights/stays before. Methods used to recruit people were by text, messenger, and
instagram.

Instruments
The instruments designed and used to carry out this study are as follows:

-User test script

-Tasks
○ Task 1 - Your manager asks you to help her plan a few trips for the company. She has
heard of a website called “expedia.com ” that can help and encourages you to use it. Plan
a round trip from London to Mexico City for under £ 550.00 from 16 th September 2020
to 19th September 2020. Email the itinerary to UserTestingBhs@gmail.com

○ Task 2 - Four people from the London office want to attend a conference in New York
from 9th November 2020 to 11th November 2020. What is the cheapest total price of the
trip? *Note - any arrival/departure time is okay.
○ Task 3 - Your manager wants to join the London team in New York (your office is in
Manchester), but then she wants to go to London for a week then return to Manchester.
She plans to fly business class for the entire trip. What is the cheapest price for her trip?

○ Task 4 - The Manchester office manager has a meeting in New York on 16th October
2020. She wants to leave on 15th October after 3pm, and wants to arrive in the morning
before 11.30am. How many flight options do you have?

○ Task 5 - Help your manager book a place to stay from October 16-18. Find the top
reviewed hotel that has wifi for under $350/night in New York City.

○ Task 6 - You want to surprise your family with a visit over Christmas but money is
tight. Set up a fare alert for a trip from Manchester to Mexico City from 22nd December
2020 to 26th December 2020.

-Pre-test interview
○ Have you used expedia.com before?
○ Tell me about the last trip you planned:
a. What do you usually use to plan your trip?
b. What is your primary purpose for travelling?
c. What is your primary concern?
d. What is your budget?
○ What information is the most important when you are planning your trip?
○ How often do you travel?

-Debrief interview
Debriefing guidelines: The focus of the debrief is to walk through any problems that were
encountered during the tasks (question #1). If any time remains afterwards, ask
higher-level questions 2-5.

1. Review parts of the test where the user struggled:

What difficulties did you have on ____? I noticed you struggled with____, can you tell
me what happened? You paused here, tell me more about that.

2. Preferences:
What did you think of the site?
What did you like/dislike?
Which parts of this page are most/least important to you?

3. Changes:
If you had 3 wishes to make this better for you, what would they be? Why?

4. Understanding:
How would you describe this to a friend?

5. Use Cases: Under what circumstances would you use this? Why?

-SUS questionnaire
-Analysis Methods:
○ List of Issues: Analysis of video and logging sheet reviews
○ Analysis of critical incidents, additional issues and user suggestions
○ Success rate and average task time analysis

Findings and Recommendations:

Pretest Interview Insights

P1 is a senior in highschool and has never used Expedia or any travel sites before,
but her family usually used Google Flights. P2 was an adult who has had experience with
travel sites before.
To book flights, P1 would usually just search up a flight from where she is to her
destination on google. Not a lot of planning was done before booking a flight, she figured
out what to do as she was there. P2 would not do a direct search on google, rather find
flights on a travelling website. From there, she would often book flights from the airline
themselves as they offered more deals.

Sites regularly used:

-Google Flights
-Expedia
-Kayak
-Tripadvisor
Budget:
P1: $5000
P2: $5000

Summary Results
The following is the amount of time it took each participant to fulfill each task:
S: Success P: Partial Success F: Failure

User Task 1 Task 2 Task 3 Task 4 Task 5 Task 6

P1 S P F F S F

P2 S S S F S F

Task Level of Success Clock Time Average Time on

Task

Task 1
P1: S 6:45-6:50 5 min

P2: S 12:13-12:17 4 min

Task 2
P1: P 6:50-6:54 4 min

P2: S 12:17-12:20 3 min

Task 3
P1: F 6:54-7:00 6 min

P2: S 12:20-12:24 4 min

Task 4
 P1: F 7:00-7:04 4 min

P2: F 12:24-12:29 5 min

Task 5
P1: S 7:04-7:08 4 min

P2: S 12:29-12:31 2 min

Task 6
P1: F 7:08-7:15 7 min

P2: F 12:31-12:34 3 min

P1 average time: 5 min

P2 average time: 3.5 min

Success 6

Partial Success 1

Failure 5

Total 12

Success Rate 0.5 50%

Failure Rate 0.416 41.6%

Key Findings and Recommendations:

After carrying out the user testing sessions, the researcher analysed the data and notes to
identify and document any potential issues.
The severity was determined by using Nielsen’s severity ratings for usability problems :
0 - I don't agree that this is a usability problem at all
1 - Cosmetic problem only: need not be fixed unless extra time is available on project
2 - Minor usability problem: fixing this should be given low priority
3 - Major usability problem: important to fix, so should be given high priority
4 - Usability catastrophe: imperative to fix this before product can be released

Finding 1
Although the last task required users to set up a fare alert, no apparent fare alert option
was given when users booked their flights. Therefore, neither could complete the task.
This is not a good sign for less wealthy users as they may need this option but it is not
given to them.
Severity: 4

Recommendation: let users set up a fare alert by including it as an option.

Finding 2
Expedia does not allow the user to select a specific time for their flight, this means that it
is hard to find a flight with the user’s exact needs. The website only gives ranges for
times.
Severity: 3

Recommendation: let users enter specific times suitable to their needs instead of
premade options.

Finding 3:
Selecting hotels proved to be time-wasting, as the page would keep reloading to find a
hotel as soon as an option in the sidebar was selected.
Severity: 2

Recommendation: Since Expedia doesn’t reload when selecting options for flights, it
shouldn’t for hotels either so make sure the page doesn’t reload when entering in options.
Finding 4:
The expedia website starts on “stays” instead of “flights”. Because of this, for the first
task both users thought they had to enter their destination there.
Severity: 2
Recommendation: Put the flights tab in front of the stays tab since more users come to
expedia to book flights.

Finding 5:
The calendar was confusing when entering dates. P2 thought she had to enter the return
date at the top of the calendar instead of clicking on the date in the calendar.
Severity: 1

Recommendation: explicitly mention that users need to click on the date instead of
entering it in at the top.

Finding 6:
Since the amount of people travelling was a very small box, sometimes it was hard to see
it and change the number and P1 entered the wrong number in twice because of this. (P2
had no problems with it)
Severity: 1

Recommendation: Make the selection tool a little larger so users can see it easier.

Debrief Insights

Likes:
- Flights booked in order for multi city flights- “It makes the flights booked in
order for me so I’ll book my first one and then I’ll get ready to do my second one
and then the third one, I don’t have to like, um re-enter anything in.”- P1
- P1 liked how the sidebar on the website was structured, specifically when
finding free wifi for task 5, although P2 had difficulties with finding it. “It can
like give me like a lot of options to choose from on the side to like narrow it down
like when I was searching for free wifi, it was pretty easy to find.”- P1
- The site wasn’t too confusing and got the job done. “It gets the work done.”- P2

Dislikes:
- P1 did not like how the website appeared on the flight booking page or the
confirmation page. “I disliked um honestly the flight booking page was kinda
ugly...it could’ve been, a little bit more like friendly to the eyes I think, cause it
was like very overwhelming a little bit yeah.” -P1
- “After I booked the flight, like there was just a lot on the page at the end
like the confirmation page, like I just.. Ok yeah. That’s it (laughter)” -P1
- No option for seating preferences within each class (economy, business, etc.),
or choosing different classes for each person traveling. “It didn’t let me do
 seating preferences, like the task with the four travelers, what if they wanted to sit
in completely different places?”- P1
- Both participants had problems with the fare alerts. P1 tried to ask for help
in the help tab to see where the fare alerts were, but when she asked the bot it
did not respond. “Another wish is that the bot worked better because the bot
didn’t work!” - P1 “I did not find alerts, looks like it’s a missing feature.” - P2
- The website reloaded every time a filter from the sidebar was selected for the
hotel option. “The hotel was pretty bad, that whole thing because it was searching
for every filter I was putting in but that was not the case with the flights.” -P2
- Specific times for flights weren’t an option, the site only gave premade
options. “It doesn’t have granularity… I have to do the mental math.”- P2

Limitations:
P1 was not an adult so she had never had experience with any travel sites before. This
made the user test a little bit harder for her, but it was still manageable. P2 had used
Expedia before, but she was still not a vivid user so some aspects of the site were still
harder for her. Luckily, the tests could be conducted in person as P2 lives with the
moderator and P1 is someone the moderator has visited every week and is in the same
social bubble.

Survey Results

Questionnaire

Answer the following questions based on the scale of 1 -5 where 1 is strongly disagree
and 5 is strongly agree

1. I think that I would like to use this system frequently

1 2 3 4 5
2. I found the system unnecessarily complex
1 2 3 4 5
3. I thought the system was easy to use
1 2 3 4 5
4. I think that I would need support of a technical person to be able to use this system
1 2 3 4 5
5. I found the various functions in this system were well integrated
1 2 3 4 5
6. I thought there was too much inconsistency in this system
1 2 3 4 5
7. I would imagine that most people would learn to use this system very quickly
1 2 3 4 5
8. I found the system very cumbersome/inefficient to use
1 2 3 4 5
9. I felt confident using the system
1 2 3 4 5
10. I needed to learn a lot of things before I could get going with this system
1 2 3 4 5

Questionnaire Responses
 User Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Individual user
scores

P1 4 1 4 2 4 1 4 2 2 3 72.5

P2 3 3 4 1 3 4 3 3 4 2 60

SUS 66.25
score

Conclusion
There were some problems with the Expedia site scoring it slightly below average, (an
above average score is >68) but overall the site is okay for people who know how to book
flights and it gets the job done. It might take a little learning to operate the website, but
once users learn how to navigate it, the website is pretty simple and easy to use. To get a
more accurate result, more users would need to be tested on the site.

References
Nielsen, J. (1994) Heuristic Evaluation. In J. Nielsen. & R. L. Mack (Eds.)
Usability Inspection Methods. New York, NY: John Wiley & Sons.

Appendices
Appendix A: User Test Script
Pretest Checklist
● Clear cookies
● Login with email: (See Note** Below) password: (See Note** Below)
● Remove login saved memory
● Double check success criteria
● Fill in the blanks in this Test Script, User Testing Plan, and Consent Form with the
name of the website you
chose for analysis.
● Prepare shearable Tasks document
● Prepare shearable post-test questionnaire
● Print logging sheet
● Prepare Test Script and Consent Form forms
● Start screen recording
Note**: Email: UserTestingBhs@gmail.com / Password: Test1ng@
Post-test Checklist
● Stop recording, save audio and video to backup drive
● File informed consent form
● File logging sheet
Introduction to demo
Moderator (with a big smile!): Thanks for coming in today! We’re constantly trying to
improve our product, and getting your frank feedback is a really important part of that.
The goal for today’s session is to test the website -expedia.com. I’m here to learn from
you so I’ll ask a lot of questions, but I’m not testing you. There are no right or wrong
answers. I’ll start this session by asking some background questions. Then I’ll show you
some things we’re working on, and ask you to do some tasks. As you work on the tasks,
please think aloud. This means that you should try to give a running commentary on what
you're doing as you work through the tasks. Tell me what you're trying to do and how you
think you can do it. If you get confused or don't understand something, please tell me. If
you see things you like, tell me that too. I want to emphasize that, you won’t hurt my
feelings by telling me what you think. In fact, frank, candid feedback is the most helpful.
If you do get stuck, I’m going to try not to answer your questions or tell you what to do.
I’m just trying to see what you would do if you were using it on your own. But don’t
worry--I’ll help you if you get completely stuck.
Do you have any questions before we begin?
Consent Form
Present Consent form, summarize it, and obtain signature
Pre-test Questionnaire
1. Have you used ____________________ before?
2. Tell me about the last trip you planned.
● What do you usually use to plan your trip?
● What is your primary purpose for travelling?
● What is your primary concern?
● What is your budget?
3. What information is the most important when you are planning your trip?
4. How often do you travel?
Task Instructions
Prepare a shareable link and present them, one at a time. Read each task aloud and give
the shareable link to the
participant.
SUS questionnaire
Link to form: https://forms.gle/8HTQynstRJMBkyCJ8
Debrief
1. Review parts of the test where the user struggled: What difficulties did you have on
expedia.com? I noticed you struggled with____, can you tell me what happened? You
paused here, tell me more about that.
2. Preferences: What did you think of the site? What did you like/dislike?Which parts of
this page are most/least
important to you?
3. Changes: If you had 3 wishes to make this better for you, what would they be? Why?
4. Understanding: How would you describe this to a friend?
5. Use Cases: Under what circumstances would you use this? Why?
Conclusion
This has been incredibly helpful. Today, you mentioned … [Moderator: Try to briefly
summarize some key parts of the
discussion or issues.] Your input is really valuable for me and the team as we think about
the next steps for these ideas.
We really appreciate your taking the time to come in, and answering all of my questions.
Thanks SO much!

Appendix B: Raw Questionnaire Responses:

P1:
1. Have you used expedia.com before?
“No. mm mm.”
2. Tell me about the last trip you planned.
“Um ok, ok last trip we planned was probably- to I’ll say Mexico, in like um in like
winter break.”
● What do you usually use to plan your trip?
“We probably just use, honestly just Google like we Google like what are the top… yeah
we’re not really good at like planning trips like we’re actually really bad at it.
● What is your primary purpose for travelling?
“Um to, ok since we haven’t been to many new places it’s usually to like explore new
places but also we just- don’t plan very well.”
● What is your primary concern?
“Uh that we won’t be experiencing everything that the place has.”
● What is your budget?
(asks her mom) “Mommy what’s our travel budget?” (her mom responds) “Uh which
includes the tickets?... For a vacation.. 5000 dollars?”
3. What information is the most important when you are planning your trip?
“Umm..okay let’s see i think like- Mommy do you know?” (her mom responds) “Uh the
weather there, where we’re going.”
4. How often do you travel?
“Probably like twice a year, we’ll say.”

P2:
1. Have you used expedia.com before?
“Yes.”
2. Tell me about the last trip you planned.
“Uh I think it was Cornell.” (Ithaca, New York)
● What do you usually use to plan your trip?
“Expedia.”
● What is your primary purpose for travelling?
“Vacations.”
● What is your primary concern?
“Cost!”
● What is your budget?
“Usually, for family vacations my budget is $5000.”
3. What information is the most important when you are planning your trip?
“Uh.. when. The dates that I am planning. If that’s working for everybody.”
4. How often do you travel?
“Twice a year.”

Appendix C: Findings
Description System Location Severity
Although the last task required Confirmation page, bottom. (Does not 4
users to set up a fare alert, no exist on the website, though, but this is
apparent fare alert option was where it would be located.)
given when users booked their
flights. Therefore, neither could
complete the task. This is not a
good sign for less wealthy users
as they may need this option but
it is not given to them.

Expedia does not allow the user Sidebar on flight booking page 3
to select a specific time for their
flight, this means that it is hard to
find a flight with the user’s exact
needs. The website only gives
ranges for times.

There was not a number provided Booking page for flights 3

for the amount of flights
available as one of the tasks
wanted users to do, so both users
failed the task as they had to
count the flights themselves and
even then counted a lot less
flights than the success rate (192,
and P1 only counted 3)

Selecting hotels proved to be Sidebar on hotel booking page 2

time-wasting, as the page would
keep reloading to find a hotel as
soon as an option in the sidebar
was selected.

The expedia website starts on Beginning page on the website 2

“stays” instead of “flights”.
Because of this, for the first task
both users thought they had to
enter their destination there.

The calendar was confusing Beginning page on website 1

when entering dates. P2 thought
she had to enter the return date at
the top of the calendar instead of
clicking on the date in the
calendar.

The calendar sometimes wouldn’t Beginning page on website 1

fully fit the screen and the page
had to be reloaded in order to
access the full calendar.

Since the amount of people Beginning page on website 1

travelling was a very small box,
sometimes it was hard to see it
and change the number and P1
entered the wrong number in
twice because of this. (P2 had no
problems with it)

Can’t choose where each person Booking page for flights 1

will sit specifically. (economy
class, business class, etc. and
where in each class would they
sit ex. Seat 40 and seat 34 in
economy, seat 2 in economy and
seat 10 in business, etc.)

Not a very nice looking booking Booking and confirmation page 0

and confirmation page.
Appendix D: Consent Form
I agree to participate in the study of a flight booking website being conducted as part of
the Coursera course: Evaluating Designs with Users. I consent to the recording of this
test. This recording will be used for research and product improvements only. I
understand that participation in this usability study is voluntary and I agree to
immediately raise any concerns or areas of discomfort during the session with the study
administrator. Please sign below to indicate that you have read and you understand the
information on this form and that any questions you might have about the session have
been answered.

Date:_________

Please print/sign your name: __________

Thank you! We appreciate your participation.