Common Industry Format for Usability Test Report

Comments and questions about this format: iusr@nist.gov

Assignment 2 Part 2
Usability Engineering
Website (www.yourwebsite.com)

Test Date
Report Date
Student name and
number
Assignment 2 Part 2 Website
(www.yourwebsite.com)

//Edit the Table of Contents

Executive Summary .................................................................................................................... 1

Introduction ................................................................................................................................. 1
Full Product Description/Overview of site ............................................................................ 1
User Profile............................................................................................................................ 1
Scope ..................................................................................................................................... 1
Test Objectives ...................................................................................................................... 1

Method ......................................................................................................................................... 1
Participants ............................................................................................................................ 2
Context of Site/Application Use in Test ................................................................................ 2
Tasks …………………………………………………………………………………2
Test Facility ................................................................................................................... 2
Participants Computing Environment ........................................................................... 2
Display Devices ..................................................................................................... 2
Experimental Design ............................................................................................................. 3
Procedure ....................................................................................................................... 3
Participant Task Instructions ......................................................................................... 3
Usability Metrics ................................................................................................................... 4

Results .......................................................................................................................................... 4
Data Analysis......................................................................................................................... 4
Data Scoring .................................................................................................................. 4
Data Reduction .............................................................................................................. 5

Presentation of Results .......................................................................................................... 5

Effectiveness Results ..................................................................................................... 5
Recommendations ................................................................................................................. 6

References .................................................................................................................................... 6

Appendix A – Data Collection Forms ........................................................................................ 6

Screening Questionnaire................................................................................................ 7
Orientation Script .......................................................................................................... 7
Pre Test Questionnaire .................................................................................................. 7
Video Consent Release Form ........................................................................................ 7
Task Scenarios............................................................................................................... 7
Post Test Questionnaire ................................................................................................. 7
Data Logger ................................................................................................................... 7
Facilitators Checklist ..................................................................................................... 7

Appendix B – Screen Shots ......................................................................................................... 7

Error! No text of specified style in document. Error! No text of specified style in document.
page ii
Assignment 2 Part 2 Website
(www.yourwebsite.com)

// NOTE – Comments in red will give you a rough idea of the content in that section. You should
remove them in your report, headings excepted.

Executive Summary
This section provides a high level overview of the test. The intent of this section is to provide information
for decision-makers. These people may not read the technical body of this document. This section shall
begin on a new page and end with a page break to facilitate its use as a stand-alone summary. A high level
overview of the test that includes:
a) Name and description of the website.
b) Summary of method including number and type(s) of participants and tasks.
c) List of findings.
d) Reason for and nature of the test.

Introduction

Full Product Description/Overview of site

Website name and date tested.

User Profile
The user population for which the site is intended.

Scope
The parts of the site that were evaluated. Reason for focusing on a product subset, if the whole product was
not tested.

Test Objectives
The objectives for the test and any areas of specific interest. NOTE: Possible objectives include testing user
performance of work tasks and subjective satisfaction in using the product

Method

Usability Engineering Semester 1, 2018

Page 1
Assignment 2 Part 2 Website
(www.yourwebsite.com)

Participants
a) The total number of participants tested.
b) Segmentation of user groups tested, if more than one.
c) Key characteristics and capabilities of user group.
d) How participants were selected; whether they had the essential characteristics.
e) Differences between the participant sample and the user population. EXAMPLE: Actual users might
attend a training course whereas test subjects were untrained.
f) Table of participants by characteristics, including demographics, professional experience,
computing experience and special needs. The characteristics shall be complete enough so that an
essentially similar group of participants can be recruited. Characteristics should be chosen to be
relevant to the product’s usability; they should allow a customer to determine how similar the
participants were to the customers’ user population.

Context of Site/Application Use in Test

Tasks
a) The task scenarios for testing.
b) Why these tasks were selected. EXAMPLES: The most frequent tasks, the most troublesome tasks.
c) Any task data given to the participants.
d) Completion or performance criteria established for each task.

Test Facility
a) The setting and type of space in which the evaluation was conducted. EXAMPLES: Usability lab, cubicle office,
meeting room, home office, home family room, manufacturing floor
b) Any relevant features or circumstances that could affect the results. EXAMPLES: Video and audio
recording equipment, one-way mirrors, or automatic data collection equipment
c) Computer configuration, including model, OS version, required libraries or settings.
d) Browser name and version; relevant plug-in names and versions.
e) If screen-based, screen size, resolution.
f) Any hardware or software used to record data.

Participants Computing Environment

The computing configuration that was used for the test -:
Computer model and specification:
Operation System:
Internet connection:
Mouse:
Browser:
Plug-ins:
Other software:

Display Devices
The monitor configuration used for the test is as follows:
Screen size:
Resolution:
Frequency:
Colour:

Usability Engineering Semester 1, 2018

Page 2
Assignment 2 Part 2 Website
(www.yourwebsite.com)

Experimental Design

Include - Number of participants, number of tasks they performed, environment for each of them

Discuss –

the control variables in this test: browser, screen size, resolution, same internet connection for each
participant, etc.

The measures recorded during the test per user:

 Goal achievement, Number assists per task: ,Number of errors,Time-On-Task:

 Satisfaction, ease of task achievement, usefulness of search functions, ease of finding way
around the site, amount of information on the web pages,adequacy of the amount of
instructions on the website and overall satisfaction with the website using the Likert scale.

Procedure
- Discuss the entire procedure from welcoming to thanking the users after the test
-Time limits on tasks.
- Verification that the participants knew and understood their rights as human subjects.
- Number and roles of people who interacted with the participants during the test session.
- Whether participants were paid or otherwise compensated.

-Measures that were recorded during the test per user are: discuss how you defined the measures

 Example - Number assists per task: count = zero at the beginning of a task, which is
incremented each time the facilitator helps the user.

-When and how the facilitator can assist and give hints must be discussed. You may use a table
Task numbers Time limit Assists allowed? Type of assistance
(minutes)

Participant Task Instructions

 General instructions given to the participants (here or in an Appendix).

 Instructions on how participants interact with any other person present, include how to ask for
assistance and interact with other participants, if applicable.
 Task instruction summary.

Usability Engineering Semester 1, 2018

Page 3
Assignment 2 Part 2 Website
(www.yourwebsite.com)

Usability Metrics

Effectiveness can be measured using -

Goal achievement:

Completion rate:

Assisted completion rate:

Error rating:

And any other

Efficiency:

Time on task:

Success in benchmark time rate:

Satisfaction:

Satisfaction measured using the Likert scale

Sample questions -

1. How do you rate your overall satisfaction of the site?

2. How useful do you think the website is?
3. How easy do you think the site was to use to achieve the tasks you were given?
…………………………….
……………………………

Results

Data Analysis
Data Scoring
Use the various metrics to discuss

Example - Goal achievement:

Task Completion/Performance criteria

number

Usability Engineering Semester 1, 2018

Page 4
Assignment 2 Part 2 Website
(www.yourwebsite.com)

1 100 % if the task is …

50% complete if …
0% complete if…

Satisfaction:

Metric Measure Scale Measure

Overall Satisfaction Not 3 2 1 0 1 2 3 Very Satisfied

Satisfied

Data Reduction
Group tasks into types to discuss

Presentation of Results
Discuss Success rates
a) The number of participants who completely and correctly achieve each task goal.
b) Errors are instances where test participants did not complete the task successfully, or had to attempt
portions of the task more than once.
c) The unassisted completion rate (i.e. the rate achieved without intervention ) as well as the assisted rate
(i.e. the rate achieved with intervention) where these two metrics differ.

Satisfaction ratings

a) One or more measures of user satisfaction. Questionnaires to measure satisfaction and associated attitudes
are commonly built using Likert and semantic differential scales. A variety of instruments are available for
measuring user satisfaction (such as the SUS).

Example Effectiveness Results

Task type Task 1 Task 2 Task 3

Participant 1 100 100 100
Participant 2 100 100 100
Participant 3 100 20 100
Participant 4 100 100 100
Participant 5 100 100 100
Goal Mean 100 84 100
Achievement Median 100 100 100
(%) Range 100-100 20-100 100-100
SD 0.00 35.78 0.00
Participant 1 0 0 1
Participant 2 0 0 1
Participant 3 2 3 0
Participant 4 1 0 0

Usability Engineering Semester 1, 2018

Page 5
Assignment 2 Part 2 Website
(www.yourwebsite.com)

Task type Task 1 Task 2 Task 3

Participant 5 1 1 0
Error Rating Mean 0.8 0.8 0.4
Median 1 0 0
Range 0-2 0-3 0-1
Standard 0.84 1.30 0.55
Deviation
Assisted 0/5 0/5 0/5
Completion Rate
Unassisted 5/5 4/5 5/5
Completion Rate
Completion Rate 5/5 4/5 5/5

Recommendations
Findings

List your findings – positive / negative for every task and give an explanation

Observations
//List your observation

Recommendations:
//List the recommendation and include the rationale behind it

Overall Conclusion

// Conclude with an overall idea about the website, issues and recommendations

References
List your references here

Appendix A – Data Collection Forms

Usability Engineering Semester 1, 2018

Page 6
Assignment 2 Part 2 Website
(www.yourwebsite.com)

Screening Questionnaire

Orientation Script

Pre Test Questionnaire

Video Consent Release Form

Task Scenarios

Post Test Questionnaire

Data Logger

Facilitators Checklist

Appendix B – Screen Shots

Embed screenshots of website and highlight the usability issues/ observation

Usability Engineering Semester 1, 2018

Page 7