HUMAN COMPUTER INTERACTION AND

COMMUNICATION
USABILITY TEST PLAN - EXAMPLE

EMPLOYEE WEBSITE TEST PLAN

As we prepare to go live with a new employee website for the university, it is important that we evaluate the
effectiveness of the preliminary design before dedicating resources to its development. Conducting a usability study
on the preliminary design will enable us to gather objective data on the design and determine how well the design
supports our end users’ ability to complete tasks.
We will use these data to recommend design changes as necessary, as well as to establish a baseline against which
future usability evaluations can be measured.
Our overall objectives for the study are to:
 Evaluate the overall effectiveness of the new employee website for users performing basic tasks
 Identify any obstacles to completing basic tasks
 Test multiple navigation structures (alphabetical vs. metrics-based)
 Test the effectiveness of employee announcement channels

RESEARCH QUESTIONS
In an effort to achieve the objectives listed above, in this study will try to answer these questions:

 How quickly can users perform common tasks?

 Can users quickly and easily navigate to the links used to perform tasks (test things like printing a W-2, updating
address, updating email address, viewing pay information, entering vacation and/or sick time, etc.)
 Do users notice/read/retain information from the “news” channels?
RECRUITING PARTICIPANTS
We will select 15 participants to test two different designs. Participants will have the following characteristics and
will be recruited from the approximately 500 employees in the university’s administrative offices. To match the
university’s overall employee demographics, we will ensure that half of the test participants are 50 years of age or
older and that 60% of the test participants are female.

Desired number
Characteristic of participants

Participant type
Pilot 1
Regular 14

Total number of participants 15

Website design
Design 1 7

Design 2 7

Overall characteristics
All participants:
 Are familiar with the university and its processes
 Are comfortable using online tools and websites to manage personal and
business transactions

Age
< 50 years 7
> 50 years 7

Gender
Male 6
Female 9

Note: Employees who are part of the development team or organization will be excluded from the test.
METHODOLOGY
This usability study will be an exploratory test. We will collect data about how long it takes participants to
complete each task, error and success rates, and qualitative data about participants’ experiences using the
system.

We will also be able to collect qualitative data from the participants’ verbal protocol – the running commentary
that participants make as they think aloud. This will give us indicators about what participants were confused by
and why, and their general impressions of the site.

PROCESS

During the test, each evaluator will work through seven task paths. To ensure that results are not affected by
transfer of learning, we will randomize the task order for the participants.
We will conduct a total of 15 individual 30-minute usability study sessions, including one pilot session.
Participants will be divided into two groups, with each group testing different design elements. Design 1 will
contain a “quick links” section and menu items that are arranged based on use. Design 2 will not contain a “quick
links” section and will use menu items that are arranged alphabetically.
We will use 10 minutes of each session to explain the session to the participant, review basic background
information with the participant, and then conduct a post-test debriefing interview.
During the middle 20 minutes of the session, participants will be asked to use the employee website to perform
basic tasks, following the instructions provided in each scenario.

The general structure of each session will be as follows:

Pre-test arrangements
Have the participant review and sign recording permissions.

Introduction to the session (5 minutes)

Discuss:
 Participant’s experience with usability studies and focus groups
 Importance of their involvement in the study
 Moderator’s role
 Room configuration, recording systems, observers, etc.
 The protocol for the rest of the session
 Thinking aloud
 How participants currently use the website and how they use other online systems in their job

Tasks (20 minutes)

 Participants will follow the instructions for each scenario to complete basic tasks.
Post-test debriefing (10 minutes)
Users will be asked a series of open-ended questions about their experience and will also be asked to
complete a brief questionnaire.

TASKS

Participates will begin the test at the employee website home page and will be asked to use the site to complete
the following tasks:
1. You are interested in attending the next PSC Town Hall meeting. When and where is the meeting being
held?

2. In this scenario, you want to update your mailing address. How would you use the website to do that?

3. You are planning to take a two-week vacation to Hawaii at the end of April. Where would you go to
enter your vacation?

4. You recently travelled to UCCS for an all-day meeting. Where would you go to submit an expense report
for the trip?

5. You know that open enrollment is coming up in the next month or two. Where would you go to find out
more?

6. You want to find your net pay for last month. Where would you go to find that information?

7. You need to access the HRMS system to complete a task, where would you go to do that?

We will briefly interview each participant at the beginning of the session to get some impression of the
participant’s web-use habits and how the employee website can better fit into his or her life.
Finally, we will end each session by having the participant fill out a short questionnaire and participate in a brief
interview where we ask broad question about their experience with the site.
TEST ENVIRONMENT AND LOGISTICS
We’ll use a controlled setting to conduct the sessions. The study will take place at the 1800 Grant Street office.
Participants will use a Windows PC and Internet Explorer with a high-speed connection to the Internet. The PC
that the participant uses will have Morae recorder installed on it and a web camera attached. The Morae
software will record the participant’s face and voice, along with what’s happening on the screen. The software
will compile all of this data against a timeline, enabling us to pinpoint time on task for each activity.
The test moderator will administer the usability tests in-person while the analyst and observer(s) will monitor
the test sessions remotely, using Morae, from an adjoining office.

PROJECT TIMELINE AND TESTING SCHEDULE

What When
Initial working meeting 1/21
 Review project objectives, scope, deliverables
 Identify participant selection criteria
 Agree on final schedule

Deliver final screener and test plan 2/4

Recruit participants 2/8-2/10

Deliver final session script 2/4

Dry run of the session script 2/11

14 usability test sessions of 30 minutes each 2/15 – 2/16

Debrief on observations and findings 2/18

Deliver final written report and recommendations 2/25

Session schedule

The tables below show a daily schedule of sessions for the study:

Time Design 1
2/15/2011

11:30 – 12:00 Meet up,

set up

12:00 – 12:30 Session 1

12:45 – 1:15 Session 2

1:30 – 2:00 Session 3

2:15 – 2:45 Session 4

3:00 – 3:30 Session 5

3:30 – 4:00 Team Debrief

Time Design 2 Design 1

2/16/2011 2/16/2011

8:00 – 9:00 Meet up,

set up

9:00 – 9:30 Session 1

9:45 – 10:15 Session 2

10:30 – 11:00 Session 3

11:15 – 11:45 Session 4

11:45 – 12:30 Lunch

12:30 – 1:00 Session 5

1:15 – 1:45 Session 6

2:00 – 2:30 Session 7

2:45 – 3:15 Session 6

 Time Design 2 Design 1
2/16/2011 2/16/2011

3:30 – 4:00 Session 7

4:00 – 4:30 Team Debrief

MEASURES

To answer the questions below, we will collect both performance and preference data during the test sessions:

 How quickly can participants locate the specified link?

 Do participants go straight to the area where the link is located, or do they search in other areas first?
 Do participants read the text in the communication channels?
 Can they answer questions based on information from the communication channels?
 How easily do participants grasp the organizational structure?
 How often to participants need prompting to find something?

Performance:

 Number of tasks completed with and without assistance—I will track two levels of prompting when
participants need assistance.

None Participant completed a task without prompting.

Try again Participant completed a task when asked, “Can you think of any other place to look?”

Preference:

 Perceived amount of time and number of steps

 Overall ease of use
 Usefulness of terms and labeling
 Feelings about overall design
DELIVERABLES

The following deliverables will be created as part of the usability testing process:

 Participant characteristics and recruiting strategy; detailed research questions; and final schedule
and deliverables.

 Draft and final copies of participant selection criteria in a screening script.

 A test plan (this document) describing the general approach and schedule for the study.

 Draft and final copies of the session checklist (a script that is also used for note taking during the
sessions and observers can use to follow along).

 Dry run of the session checklist.

 14 usability study sessions of 30 minutes each (along with one pilot session).

 Daily debriefings with observers.

 Morae recordings of each session.

 A final written report of complete findings.

The final report will:
 Briefly summarizes the background of the study, including the goals, methodology, logistics, and
participant characteristics
 Present findings for the original questions to investigate
 Give quantitative results and discuss specifics as appropriate to the question and the data
 Discuss the implications of the results
 Provide recommendations

 Suggest follow-on research