INFO90004 Evaluating the User Experience The University of Melbourne

Assignment 2 (40%) Usability test plan, report and video

presentation
Assignment 2 is based on a group project that runs through the semester. In the
assignment, you will plan and conduct a user-based usability evaluation of a website, i.e.,
usability study. The project is divided into a series of related tasks, focusing on an
evaluation of a website. You will be able to complete some of these tasks during class
time; others will need to be finished between classes. Although this is a group
assignment, there are some individual tasks that must be submitted separately.
In Part A, you will prepare a plan for a usability study (test plan). This is worth 5% of
the marks for the unit. Through preparing a detailed usability test plan, you will be able
to receive feedback on the test approach that you will conduct in Part B of this
assignment.
As part of this assignment, you also need to demonstrate that you have engaged with
the ethical concerns around conducting a usability study.
In Part B, you will conduct the planned study with 4-6 moderated participants and 15
or more unmoderated participants (using an Optimal Workshop product) and prepare a
group report worth 20% as well as an individual reflection (5%) on how your group
worked together, what you learned from the project, what you observed as a test
participant, and what you would do differently if you were to start the project again.
The individual reflection is submitted privately.
You will also act as a participant for another group of students who are studying a
different website. There is a penalty of 20% of the total possible marks for part B for
any student who does not register or show up for this scheduled session.
In Part C, you will prepare a five-minute video (10%) showing what you found and your
recommendations. This video must include highlights from your testing, which should
be used to support your findings and recommendations. These might be charts and
images from Optimal Workshop, or videos of participants from your live study. We
encourage you to post a link to your video on the subject Discussion forum, for other
students to view and comment on. You are also required to (individually) peer review
another group’s video.

Part A: Part B: Part C:

Evaluation Plan Evaluation Report Video
Group Evaluation Plan Evaluation Report Video presentation
(5 marks) (20 marks) (10 marks)
Individual Act as a participant Peer review of
for another group another group’s
presentation
Reflection (0 marks; Hurdle for
(5 marks) video mark)

Copyright © 2024, The University of Melbourne 1

INFO90004 Evaluating the User Experience The University of Melbourne

Your task
Imagine that you have been contacted by one of the organisations whose website is
listed below. They want to know how they could improve their website and have asked
you for advice. You must conduct a moderated study face to face or using Zoom, Loop11
or a similar tool and a fully remote study using at least one of the Optimal Workshop
tools to complete your project. For example:
• If you want to test the structure of the website, you could use Treejack.
• If you want to test the layout of the pages, you could use Chalkmark.
• If you want to suggest a different structure for the website, you could use
Optimal Sort.
For your Optimal Workshop study or studies, aim for at least 15 participants per study.
Some of these may be the same people who complete your moderated study. You could
consider inviting your tutor or lecturers to complete the study, or post it on the
subject’s Discussion forum if you are looking for participants.
For your moderated study, you should aim to have 4-6 participants.

What can you evaluate?

You should select a website for evaluation from the list below. These sites enable you to
do or plan something. Tools that lead to good evaluations typically have several
screens or views of information and different ways to interact with it.

• Customise clothes and get them made to order – http://www.eshakti.com

• Find a job – http://www.seek.com.au
• Adopt a pet - https://www.adoptapet.com.au/
• Find out about weather and climate - http://www.bom.gov.au/
• Learn about the birds you see – https://www.birdsinbackyards.net/
• Plan practical care for a sick friend or relative –
http://www.gathermycrew.org.au/
• Learn about road rules, get a driving licence, or register a vehicle -
https://www.vicroads.vic.gov.au/
• Design and order stationery for a business - https://www.vistaprint.com.au/
• Design a kitchen for your home – https://planner.kaboodle.com.au/
• Build a customised personal computer – https://dreampc.com.au/

All of the websites listed above let you create an account for free (you can create
accounts with disposable email accounts), or interact with them without creating an
account. You do not need to order products or pay for these sites to do this
assignment.

When you choose a site for testing, think about:

• Does the topic of the site interest you?
• Are you able to complete tasks on the website?
• Does the site offer scope for interesting tasks and scenarios?
• What issues can you see with the tool? What would you like to explore in
usability testing?

Copyright © 2024, The University of Melbourne 2

INFO90004 Evaluating the User Experience The University of Melbourne

PART A1: Usability Test Plan (5%)

Cover Sheet: The cover sheet must be signed by each group member and attached to
the front of your assignment when you submit it on Turnitin.

Submission: Submit the complete report (PDF or Word) to Turnitin, using the link in
LMS. Label your file as follows:
[Tutor Name]_Group [number]_[optional group name]_Test Plan,
e.g.: Melissa_Group 14_Usability Stars_Test Plan.pdf.

There is not a formal template for your usability test plan, but it must include the
following components:

Each section should start on a new page, and follow the structure below (with page
length):
Max.
Section Notes/Assessment criteria
length
Include group name/number and names & student
Title Page
numbers of all group members
Scope ½ page Like an Executive Summary, this introduces your test.
What are you testing? What is your overall method for
the test? Who will you test with, and where will you
test?
Are there specific ethical considerations that you need
to acknowledge or consider?
We are looking here for an understanding of what you
are testing and why.
Objectives ½ page What do you hope to find out from testing?
Try to write SMAR(T) objectives that describe the goals
of your evaluation. A useful way to think about
objectives is as relating to those six usability goals from
Preece, Rogers and Sharp.
Project team 1 page Introduce the team. What relevant special skills or
interests do you bring to this project? What will each
person be responsible for?
A good project team description will include a role title
(User Experience Specialist?) for team members and
may even include a photo of each person.
Test format ½ to 1 What tools are you going to use for your evaluation?
page What will you test with each tool? How do these tests
relate to your overall objectives?
Include an outline of your online as well as in-person
testing: Which online tool/s do you plan to use? What
will users be asked to do in each part of your study?
How will you record the users' interactions or outcomes
so that you can use this material in your video
presentation?

Copyright © 2024, The University of Melbourne 3

INFO90004 Evaluating the User Experience The University of Melbourne

Users and Recruitment 1 page Who are the target users for the site? Are there any
other potential users?
How many participants will you test in each test? Who
is included? Who is excluded? Is there a screener
questionnaire? How will consent be obtained/given?
Test Scenarios (Task task 5-6 tasks & scenarios relevant to the problem for both
scenarios) for Optimal table (1- remote and moderated testing. You might use the same
Workshop and 2 pages) tasks for both parts of your study, or you might use
moderated testing different tasks.
Do the tasks vary according to their representativeness
and criticality?
Is appropriate detail provided in the scenarios?
Is it clear which tasks are suited to which type of
testing?
Key dates and 1 page Prepare an outline of the key dates and milestones for
milestones for the your project. See the “Project Timeline” doc for a
project template.
References Are references used to justify and explain relevant
aspects of the plan?
You should integrate references that are about the site
itself, or its context, as well as references about the
methods that you have chosen.
Full test protocol As described in class, and following an example
provided, please supply a full test protocol for your
facilitated 1:1 user test as your first Appendix. (Note
that you do not need to include the individual tasks at
this stage, but you should include a “Task 1”
placeholder block with any pre- or post-task questions)
Appendices Are the Appendices complete and clearly referenced in
• Group contract the main test plan?
• Minutes of all Project minutes should include your updated project
meetings timeline
• Any other
relevant HURDLE REQUIREMENT: Each member of the group
materials must complete the Human Ethics Student Report
including Ethics form. Attach the completed form, together with the
Report, Consent Consent Form and Plain Language Statement (PLS), as
form, and PLS an appendix.
Markers will also provide feedback on your overall
test plan:
• Is the plan as a whole coherent?
• Is the plan well presented?
• Does the plan clearly describe what will happen
during the evaluation?

Copyright © 2024, The University of Melbourne 4

INFO90004 Evaluating the User Experience The University of Melbourne

PART B1: Usability Test Report (20%)

Submission: Submit the complete report (PDF or PowerPoint) to Turnitin, using the
link in LMS. Label your file as follows:
[Tutor Name]_Group [number]_[group name]_Report,
e.g.: Wei_Group 13_Usability Experts_Report.pdf.

Your usability test report must be written as a set of slides which follows the structure
below. Although the report is written on slides, it is for reading rather than for use as a
presentation– think of this as a document in landscape format, that will be read on a
screen or on paper, rather than a presentation that will be projected onto a screen.
Refer to the samples that we will show you in Week 8.

Section Slides Notes and Assessment criteria

Title Page 1 Include group name/number and names & student
numbers of all group members
Please provide a link to your video presentation
(assignment 2 Part C). (This slide should include
the link to the video).
Executive Summary 2 Is the Executive Summary a clear and useful
summary of the entire document?
Introduction 1 This includes the scope of the testing, when it was
conducted, etc. Include your objectives here.
Method 1 Are the method, procedure and materials clearly
The testing method described?
– tools, instruments, Are the data collected relevant to the objectives?
metrics. Did you Are test scenarios provided (e.g. in an appendix)?
conduct a pilot test?
Do you have ethics
approval?
Test scenarios 1 Include a copy of the task table
Participants 1 Who tested the product? Do they match your ideal
users? Why/why not?
Findings 1-2 slides per 4-6 key findings, What did you learn about the
Number, Title, finding system from doing the test? Try to use quotes from
Summary, participants or descriptive statistics to illustrate a
Evidence, Severity particular point. Use numbers where relevant to
rating, Brief support your findings (e.g., 4 out of 5 participants
recommendation had trouble with …).
(optional) Include qualitative and quantitative findings.
Are the findings clear and comprehensive? Is the
data analysed and presented appropriately? Are
the results summarised effectively with
appropriate reference to appendices?
Recommendations 1-2 slides per 4-6 key recommendations, Recommendations
Number, Title, recommendation must be actionable and prioritised.
Summary, Link to

Copyright © 2024, The University of Melbourne 5

INFO90004 Evaluating the User Experience The University of Melbourne

Section Slides Notes and Assessment criteria

relevant findings, Are the conclusions & recommendations clear and
Priority justified by the evidence presented earlier??
Limitations 1 slide Are the limitations of the method and results
indicated
Next Steps 1 slide Is more research required?
What are the next steps for the organisation –
what advice do you have for them that goes
beyond your recommendations?
References 1-2 slides Use APA format for your referencing. Please
support your work with at least 8-10 references.
Are references used to justify and explain relevant
aspects of the report?
Remember to use references that support your
method and recommendations as well as
references that describe the context of the site you
are studying.
Appendices Submit as a Are the Appendices complete and clearly
• minutes of separate Word referenced in the main report?
all meetings doc or PDF file. Are participants’ names anonymised in the report?
• final full test You do not need to provide transcripts of each
protocol session!
• selected raw
data (e.g.
answers to
interview
questions)
• any other
relevant
materials
Quality of report Markers will also provide feedback on your
overall test report:
• Is the report as a whole coherent?
• Is the report well presented?
• Is there a logical progression between the
sections of the report?

Copyright © 2024, The University of Melbourne 6

INFO90004 Evaluating the User Experience The University of Melbourne

PART B2: Individual Reflection on Group Work (5%)

This is an individual requirement.

This document should begin with a table or list showing

• Your name and student number;

• Your group name and number; and
• The name and number of the group that you were a participant for.

You are to write a reflection (300 – 500 words – about 1 A4 page) on your experiences
working with your group and as a participant for another group.

Your reflection could answer questions like

• How did this group work together effectively?

• What challenges did you face as a group or in working with your group?
• What did you contribute to the group?
• How could you have improved your contribution?
• What would you do differently if you were to start the project again?
• What did you learn from working with this group?
• What did you learn from being a test participant?
• How did/would your experience as a test participant impact on the decisions
you made in conducting your own study?

This is a personal reflection on your own experiences in this group project. You are not
expected to provide references to support your discussion.

This reflection is due a few days after your group evaluation report and video.

Copyright © 2024, The University of Melbourne 7

INFO90004 Evaluating the User Experience The University of Melbourne

PART C: Video Presentation of Results (10%)

As a group, you will prepare a video presentation (5 minutes) describing what you
found in your evaluation.

The video must include edited “highlights” from your usability evaluation that illustrate
the key findings. These may be images from Optimal Workshop or Zoom, or short
excerpts from videos of live studies. You should use them to support your findings and
recommendations. The video must be posted on YouTube or a similar site and a link
must be supplied in your evaluation report (Assignment 2 Part B1). If you are unable to
post a video to YouTube, please let us know and we will find a suitable alternative e.g.,
the LMS.

Your video must include:

Section Time Notes
Titles 10 sec Showing your group name and members
Introduction and context 10-15 A brief overview of the website that you
seconds studied and how you tested it.
Highlights of testing 3 min 30 Key findings – what interesting things did
you learn? Include annotated “highlights”
that illustrate, justify, and provide
evidence for your key findings,
showcasing the users’ voice and
connecting it to your analysis.
Recommendations 1 min Recommendations – an outline of your
major recommendations. These should
be connected to your findings.

Upload your video to YouTube or another online video platform and provide a link to it
in your report (with password, if you choose to password protect the video).

The video will be assessed based on the following criteria:

• Is the video engaging and interesting?
• Is the video well structured, clear and understandable?
• Does the video highlight the users’ voice, supporting and justifying your analysis
using material from different sources?
• Are the recommendations well explained and justified?
• Is the video produced to an acceptable standard?

You will also be required to (individually) review another group’s video presentation.
The peer review is a hurdle for your video mark (that is, you will score 0 for your video
if you do not complete the peer review).

Copyright © 2024, The University of Melbourne 8

INFO90004 Evaluating the User Experience The University of Melbourne

Assignment 2 FAQ
1. How many tests should we conduct in this project?
You will develop and conduct TWO tests – that is, one Optimal Workshop study and
one live (moderated) study. You may find that it’s OK to use one set of tasks and any
associated questionnaires, or you may prefer to use some different tasks.
You might also (optionally) run a pilot test. This can be done informally with a
member of your team or with a friend or family member. It is up to you whether you
document this in your Assignment 2 Part B1 report (We suggest putting the results
in an appendix rather than in the main section, however, mention in your main
report that you conducted a pilot test).

2. What sort of tests should be run? (e.g., cognitive walkthrough, heuristic

evaluation)
This is a usability study with real users, not an expert evaluation. Look at the lecture
and reading material on testing with users. You will give the user some tasks to do,
and observe them completing them. You should also think about pre-test and post-
test questions.
The cognitive walkthrough and heuristic evaluation method are expert evaluation
methods. They are not methods that are used when testing with users. If you want to
use them to inform your tests, that’s fine.

3. How many test scenarios do we need? What template should we use?

The assignment document requires five to six test scenarios.
You do not need to provide a formal task breakdown using the template from the
cognitive walkthrough; it is sufficient to list the test scenarios in a task table,
although it is useful to justify why you selected these tasks for evaluation.

4. We’re having trouble writing objectives.

Try to write objectives that describe the goals of your evaluation. A useful way to
think about objectives is as relating to those six usability goals from Preece, Rogers
and Sharp.
So:
• Can users complete the tasks with less than a 5% error rate? (safe to use)
• Do users remember how to use the system based on previous tasks? (easy to
remember how to use)
• Do users recognise what the icons are used for? (easy to learn)
Alternatively, you could look at what Carol Barnum says about the ‘five Es’ (Barnum
2011, p.108): A successful design should be effective, efficient, engaging, error
tolerant, and easy to learn.

5. What other appendices do we need besides the full test protocol?

We have asked to see minutes of your team meetings. These do not need to be
formal but should show what was agreed in the meeting and what the tasks were
(including who was to do them). Ideally, you should use your project timeline to
check in on progress of key tasks.
If there is other material that you think is relevant but is not required in the main
test, you should include it here.

Copyright © 2024, The University of Melbourne 9