The User Testing Field Guide

The User Testing
Field Guide
A PRACTICAL GUIDE FOR USABILITY TESTING
The User Testing Field Guide | A Practical Guide for Usability Testing 2
The User Testing

Field Guide
A PRACTICAL GUIDE FOR USABILITY TESTING
Where prototyping meets user testing.
Marvel is a design collaboration platform em-

powers individuals and teams to effortlessly create
user-centric digital products and experiences.
marvelapp.com
We’ve also created a space for our community to

share all things design and product, from opinions
to processes.
blog.marvelapp.com
Design: Filippo Chiumiento

Illustrations: Kai Tsang
Edits: Charlotte Hall
Content: Naomi Francis
© 2020 Marvel Prototyping Ltd.
To learn more about our user testing feature, visit:
marvelapp.com/user-testing
Tables of contents
1 4
Introduction User Testing Methods

Product and Testing at Marvel 07 Moderated User Testing 34
Why Bother With User Research? 10 Remote Unmoderated User Testing 38

Common Objections to User Research 12 Guerilla User Testing 41
Prototyping and User Testing 14
2 5
Create a Test Plan Analysing Data

The building blocks of a test plan 18 Methodologies 45
Insight document 46
3 6
Recruiting Participants Conclusion

Recruiting External Users 27 Takeaways 49
Recruiting Internally for High Level 30

Feedback
Chapter One
Introduction
Welcome to The User

Testing Field Guide
This guide will help you carry out fast and cost effective usability tes-
ting that’ll leave you with actionable feedback directly from the peo-
ple that matter. As a result, you’ll be able to iterate on the real issues
at hand and ship products which delight your users.
User testing is one of the most powerful methods, terminology and tools. But don’t
tools in the product design process. All worry! We’re here to help.
it means is evaluating the performance
of your product through testing with With this handbook, we’ll explain the ins
representative users, who will complete and outs of every stage of the user testing
a number of set tasks which will help you process. On top of that, we’ve created
measure success. separate interactive templates for you to
print and fill out whenever you run a ses-
Implementing the right testing tools and sion - including a test plan template and a
processes can help you validate pretty template for capturing results.
much anything. From features, copy,
visuals or sketches, ideas and even com- Throughout this handbook, you’ll find
petitive products. references to our prototyping and user
testing platform which can help amplify
If you’re new to user testing, it might your research process. To learn more,
seem daunting. Especially with various visit: marvelapp.com/user-testing
“If you think good

design is expensive,
you should look at the
cost of bad design.”
- Dana Chisnell
How we approach
building product and
testing at Marvel
Pleasurable
Usable
Reliable
Functional
At Marvel, we approach usability as one mething functional, reliable and usable?

of several critical parts of our process in Shouldn’t that be enough?
creating a great feature or product.
The heady heights of the final level of user
It’s safe to say that every company strives needs is making experiences pleasurable.
to create something that is easy to use.
However, without the fundamentals as This is where you’ll hear people say they
illustrated above in Aaron Walter’s ‘Hie- ‘love’ a product rather than just ‘like’ it. It’s
rarchy of User Needs’ it won’t matter how where you can postive, human emotions
usable your product is if it isn’t functional from interactions with a digital product.
(broken features) or reliable (constantly
going offline). This all comes from having a deep unders-
tanding of the end-user, constantly tes-
Only once the basics are covered can you ting, listening and iterating. User testing is
achieve success in usability. That said, the key to opening up these insights and
we’re firm believers that the best digital helping you create incredible products
products and services reach even higher and features that your users love.
levels. But what comes after making so-
HOW WE TEST
As a prototyping platform, it should come as no surprise to hear we send

out a lot of prototypes to pretty much everyone! From prospective users
to loyal customers, there’s no question too big or small that can’t be
helped by a prototype.
• When needed, break larger flows down into smaller chunks

To make sure we’re moving as fast as possible and getting focused
feedback, we mix testing with large prototypes with others that have
been broken down to just show part of a flow or feature.
• Segmentation, segmentation, segmentation

Like most products, we do not have one type of user. Breaking down
by role, subscription value, company size and feature usage is a
must.
WHAT DO WE TEST?
1 Existing feature improvement tweaks.

This is the most common type of testing we do at Marvel and provi-
des the quickest turnaround in feedback.
2 Short term roadmap feature.

We use detailed prototypes with features that are planned in the
short term, as there’s often more clarity around the requirements and
feature set.
3 Long term roadmap.

For things that we know we want to build but may be more than 6-12
months away, we use loosely defined prototypes of wireframes and
sketches with the bare bones of what we want to achieve.
4 R&D, innovation, random ideas.

Sometimes an idea is a bit leftfield or not fully formed, in these cases
we send simple 2-3 screen prototypes, super basic and a simple
narrative of what we’re trying to achieve.
WHO WE TEST WITH
1 Customer champions
These are some of our most engaged and vocal customers who love
to test new things and let us know what they think. On any given
week, we’ll have 5-10 prototypes circulating within a group of 100
customer champions.
Within the group, we have a mixture of small and large companies in

different sectors so we can get a better understanding of how featu-
res impact teams of every size.
2 Users who have submitted a feature request

We receive hundreds of feature requests each week which are ma-
naged through a platform called Conflux. It allows users to not just
submit new ideas, but upvote others added by the community.
This makes it easy to quickly find lists of users who are interested in
improvements on a particular part of the platform and send over a
prototype to user test.
3 Potential customers or new market opportunities

When we build features which target a new market or profession, we
ask our customers for intros to different departments and roles.
4 Users who have tried a particular feature

We frequently use a messaging platform called Intercom which
allows us to filter users based on their usage, then show in-product
questions or send emails.
5 Random users
We do random spot testing too and select a group of active users
within the platform and reach out to them via email or in-app messa-
ging using Intercom.
Why bother with

User Testing?
If you’ve ever found issues with a design, feature or product once it’s
in development, don’t worry, you’re not alone. In fact, software spe-
cialists spend “about 40 to 50% of their time on avoidable rework.”
Ouch. This is where the costs start rac- vering defects after releasing software
king up - fixing issues once they are in costs 30 times more than catching them
development is eye-wateringly expensive, at the design stage.
regardless of company size.
Creating code-free prototypes using
If you’re a startup, redesigning and re- mockups, then testing them with partici-
building things the wrong way could pant will save you time, money and result
cause you to run out of money before in better user experiences. .
you’ve reached product market fit. Bigger
companies have to move quickly to stay But implementing a new tool and pro-
competitive and maximise shareholder va- cess can be tough, even if it does save
lue. So as a roll on effect, time-to-market so much time and effort in the long run.
becomes an issue too. We’ve put together a few stats to back you
up when it comes to getting buy-in from
IBM found, in a recent report, that disco- the wider business.
Running regular user testing sessions can see positive

results in the following:
Revenue: There is a common myth that $1 billion”. Even if you’re not a billion-do-
user testing costs a lot of money to run, llar unicorn, a rising tide lifts all boats.
however as you can see from the abo-
ve stats, it’s in no way close to what you Happiness: Not just for you and your
could be paying if you don’t do it. Every $1 team, but for your customers! You’ll no
invested in UX results in a return between doubt see an increase in NPS, ratings and
$2 and $100. customer retention. Forrester says, “inten-
tional and strategic user experience has
Experience: Gathering feedback from the potential to raise conversion rates by
your actual audience, rather than acting as much as 400%”.
on assumptions, can quickly result in an
improvement in your product’s design and Time: Effective user testing generates
user experience. actionable results from your end-user, this
means you or your design team can get to
Forrester Research found, “The revenue work on the right problems - so you can
impact from a 10 percentage point impro- iterate and hand over to dev faster. After
vement in a company’s customer expe- all, 85% of issues related to UX can be de-
rience score can translate into more than tected by performing a usability test.
It’s important to remember that user do. With tools like Marvel you can create a
testing is an iterative process, you test prototype in 5 minutes and start testing a
and gather results, implement changes few clicks later. No special setup or skills
then repeat until your participants are needed.
completing tasks as you hoped. On top
of that, it’s a process that should be used Combining Marvel with the methodolo-
for every idea, design, feature, design or gies, tips, tricks and templates in this gui-
product launch. de gives you the super powers you need
to get started and make usability testing
The secret to success is having different work for you.
methods and tools at your disposal that
match the scale of the testing you need to
The Common
Objections of User
Testing
Only 55% of companies are currently conducting any user experien-
ce testing, despite being such a crucial stage in the process. User
testing often gets cast aside or swept under the rug.
We’ve included a list of the common objections you might hear, so

you can be sure to squash them before they start costing you time,
money and effort.
• No dedicated researchers. • Tight deadlines.

Researchers are an invaluable resour- Having more time to test is always a
ce but you can still conduct testing benefit but there are user testing me-
without not essential to carrying out thodologies which can be launched in
testing. It’s something that can be run less than 10 minutes. It doesn’t have to
by anyone with this guide. Especially get in the way of your project timeli-
with tools like Marvel User Testing ma- nes. We’ll cover these later on in this
king the process more automated. eBook.
• No usability lab. • No budget.

With user testing software like Mar- Needing a huge budget for user tes-
vel’s, it’s possible to run tests and ting is a thing of the past, now with
record video and audio of your partici- remote unmoderated testing and user
pants without an expensive lab setup. testing software, you can cut your
No fancy cameras or one-way mirror research costs in half.
required.
Create your first user test in Marvel for
free by clicking here.
• You already know your users.
Any business or team carries out user
research to build personas which then
guide the product, messaging and • It’s too hard to recruit.
so on. So whilst you might know your Admittedly, this was more difficult be-
user, but you will never understand fore the internet, but it certainly isn’t
how they use and experience your now.
product if you do not run regular usa-
bility testing. From scouring your own user base, to
utilising recruitment platforms or even
tools like Craiglist - your audience can
• It takes too long to do. be sourced.
In the past, setting up a plan, running
a session, then waiting for results We’ve put together a ton of free and
could be a long and painful process. paid ways to source your participants
These days you can draw some sket- on our blog.
ches on a piece of paper, turn it into
an interactive prototype and start
gathering audio and video results of
participants using it withn an hour.
The key is to match the level of tes-

ting with what you are testing. A
ground-breaking new feature aimed at
a new market will require more plan-
ning than testing a new layout of your
app or website.
By the end of this guide, you’ll be able

to create your own participant list,
ready and waiting to help you test!
The Role of
Prototyping in User
Testing
You often hear the phrase ‘test early and often’ - the idea is that
waiting for everything to be polished and fully functional before you
start testing can seriously drag out the time it takes to get meaning-
ful feedback and discover problems (not to mention the costs invol-
ved if changes are required)
So, how do you get to the wrong idea, as Fast forward to today and ‘code-free’ pro-
fast as possible? totypes have become the defacto stan-
dard when testing early stage ideas.
In the past you typically saw coded,
functional prototypes put in front of users, With platforms like Marvel, you can create
where most of the interactivity works, interactive prototypes that look and feel
but dummy data is used to get it out to just like the real thing, in just a few minu-
users quickly. Whilst this is close to the tes using static images or sketches.
real-thing, it can often take weeks to put
together and is often outside the ability of It’s a game changer for your process and
the creative teams as it requires develop- means that every person on your team is
ment. empowered with the tools to create proto-
types and start testing.
Types of prototypes you can use for User Testing:

Prototypes can come in a few different forms, selecting the right one comes down to
how fast you want to move and the skill set of your team.
LOW-FIDELITY (PAPER SKETCHES)
Sketching is the quickest and easiest way to get early stage ideas in
front of the end-user or stakeholders. Grab a pen and paper, then start
drawing the basic parts of each screen.
The limitations of what you can do with quick sketching often helps
focus on just the fundamentals of what makes the idea work. Complex
layouts get distilled into a handful of rectangles and text. Of course it
will never elicit the same reaction as finished designs and does require
some imagination from users, however what you lose in that area, you
make up for in rapid speed and iteration.
The great thing about this stage is that anyone involved in the project
can do it, we’ve even seen testing sessions where the end-user jumped
in and started sketching what they thought would make more sense.
Once you have your sketches, grab the Marvel iOS or Android mobile
or tablet apps to take photos of each sketch, then link them together to
form an interactive prototype in just a few minutes.
Pro Tip: When sketching your app or websites on paper, it helps to have
some print templates to keep things tidy. Download them here.
WIREFRAMES
When you are a step further along with your idea, you typically start
creating wireframes. Wireframes act like the building blocks of each
screen and will act as a skeleton for your everything that comes after
including visuals, interactions and content.
It’s often said that the minimalism of wireframes allow users test ses-
sions to be more focused on how it works, rather than distract with
visuals. However there’s no one size fits all approach.
Wireframes are usually created in tools like Sketch, Adobe Illustrator and
Marvel.
Pro Tip: Export your wireframes from your favourite tool as images, then
head to marvelapp.com and drag them into a project to start creating your
interactive prototype. You can also make them in Marvel!
HI-FIDELITY
If you have progressed to the full design of your product or feature, then
you are ready for hi-fidelity prototyping.
These types of prototypes use finished visuals and are almost indistin-
guishable from the real-thing which makes it brilliant for getting genuine
reactions in user test sessions.
It takes longer to design the hi-fidelity mockups to put into the prototy-
pe, but the trade off is often worth it.
In an ideal world, you would be able to move through from low to high
fidelity over time in your user tests. But don’t get hung up on everything
being perfect, some feedback will always be better than none!
Now you understand the ‘why’ of user testing and where prototyping comes in, let’s
jump into the ‘how’. The best place to start is from the beginning, creating your test
plan.
Chapter Two
Creating a Test Plan
Create a test plan.
A prototype isn’t the only thing you need

to kick off user testing, you also need a
plan - and the more detailed the better.
Now is the opportunity to break down

the specifics of your upcoming user test,
where you’ll define things like:
• The feature, or product, you’re testing

• The business case
• Your test’s objectives
• Your audience
• Your test tasks
Tackling this early on will help you create

effective user tests that have clear objec-
tives and focus, generate both quantita-
tive and qualitative data and have proven
business value.
To create your own unique plan, we’ve

created a template that can be found
alongside this download.
To carry out a successful user test, it’s hel-

Specify the pful to define the purpose of the feature
feature, or and clearly outline what’s being tested.

This will help you to determine success in
product, you’re the later analytical stages.
testing In this stage, you should outline the key

business and user experience goals of this
feature so you can ensure that your usabi-
lity test objectives and tasks are linked to
these.
It’s also an opportunity to outline anything

that is out of the ordinary, for example,
any special conditions or context that is
needed to understand the direction of this
test.
It’s essential to break down why you need

Outline to run this test, its results should prove
business that your work is driving towards the busi-

ness goals you outlined in the last step.
reasoning Noting this down, alongside the potential
business costs of not carrying out the-
se tests, may also come in handy if, and
when, you need to get buy in from the
wider business.
So far, you’ve outlined what you’re testing

Define your and why you’re testing it - with this baseli-
test objectives ne in place, it’s the perfect time to outline

your objectives. What specifically do you
hope to learn from users in this test?
For each feature, 2-3 objectives should be

specified, which will then have 2-3 tasks
paired to them.
It’s important to avoid broad objectives - it

might come back to bite you when you’re
creating your test tasks later on. Instead,
try to come up with specific concerns
you might have, aspects you predict to be
challenging, potential problems that have
been highlighted by developers or wider
teams.
For example, let’s say the feature you’re

testing is a new add to cart design, objec-
tives could be:
• Objective 1: Users notice the new add

to cart button
• Objective 2: User find their cart easily
once adding an item
• Objective 3: Identify any design usabi-
lity issues in this current flow
Before diving into recruiting participants

Specify for user testing, you’ll need to define what
audience participants you are using. You’ll more

than likely have all your personas mapped
out, so just choose which personas are
core users of this particular feature or
product.
A helpful exercise can be defining their

telling characteristics or demographics:
• Age
• Profession
• Location
Or context driven characteristics:
• Owns a car
• Has a child under 3 years old
• Works remotely
Tasks refer to any activity or action you’d

Create Tasks like your participant to achieve during the
test. They should reflect the test objecti-
ves you outlined earlier on.
Studies show that there is more benefit

and a higher quality of results during a
test which has more varied tasks than sim-
ply a higher number of participants.
A good way to determine whether you

have a varied set of tasks is by asking
open-ended and closed questions.
WHAT ARE OPEN-ENDED TASKS?
Open-ended tasks give your participants

the opportunity to find solutions on their
own through providing them with minimal
information. Perfect for early stage testing
where you’re finding areas of interest or
identifying usability issues.
When these tasks are the supported by

open questions, you’ll receive open-en-
ded, descriptive answers - you might need
to remind users to think outloud during
these tasks.
For example, an open ended task would

be: ‘Please spend 3 minutes exploring the
website as you normally would’ and an
open question would be, “What would you
expect to be able to do on this website?”.
WHAT ARE CLOSED TASKS?
Closed tasks generate more quantitative

results as you’re giving them clear gui-
dance, giving your research the focus
you need. Perfect for testing complex
products, specific features or generating
feedback that’ll optimise conversion.
Craft these tasks carefully along with

some closed questions, as you’ll want to
avoid leading questions that impact the
validity of your data.
For example, a closed task might be,

‘Search and select a mug to purchase
using our new search functionality’, and a
closed question would be, ‘How did you
find the new search functionality?’.
Chapter Three
Recruiting
Participants
Recruiting Participants for

User Testing
Sourcing a pool of participants that are

truly representative of your target audien-
ce is essential to great research. With the
right participants, you’ll generate more
accurate results that will be applied to the
design stages that follow testing.
Thanks to your detailed test plan, you’ll

know exactly who you’re testing with -
now it’s just how to get a hold of them!
There are plenty of methods for recrui-

ting participants for user testing, some of
which we’ll outline in the following pages.
If you need more check out our blog post.
You’ll find there are different techniques

and some may be easier, or more challen-
ging, depending on who you’re targeting.
In this chapter, we’ll talk about recruiting

the following users:
• Existing users
• Prospective users
• Internal participants
How To Recruit
Participants for User
Testing
It’s vital you get feedback from your people who represent your cu-
rrent and prospective user base when creating new ideas, features
and products. Both can provide a unique and valuable perspective
that’ll be essential to the success of your project.
The overall objectives of the project will markets, designing for a new persona, or
determine which type of participant you’ll you’ve created a brand new product to
need. bring to market and your users have no
idea who you are (yet), recruiting prospec-
For example, if you’re planning new tive users is the best approach.
features or enhancements that will affect
existing user journeys in your product - it’s Each has its pros and cons, but regardless
best to approach your current user base. of approach, segmenting particpants by
However, if you’re entering into new attributes or demographics is key.
RECRUITMENT TIPS:
Offer rewards: To get people’s attention, and hold it, it’s

best to include a reward for taking part, something along
the lines of an Amazon voucher, cash, free or early access
to features or company swag would work just fine!
Recruit between 5 and 15 users: Testing with 5 users might

give you a valid overview of the majority result, however the
more users you test with the more representative of your
wider user base.
TO RECRUIT YOUR CURRENT USERS:
Creating a reusable participant list from your existing users

is must. It will become invaluable for setting up a repeata-
ble user testing program for your team or business.
At Marvel, we have our ‘Customer Champions’ list. These

are super engaged users who graciously offer their time to
test all our new features. Here’s how we recruited them:
1 Via Sales, Success and Support teams

Utilise your customer-facing teams who interact with
your most vocal and engaged users everyday, they
can tag and forward users who might be of interest.
2 In-product messaging
In-app messaging using free tools such as Intercom or
Crisp can pop-up anywhere in your product to recruit
users. It’s one of our favourite methods!
3 Setting up a beta program
Try annoucing a beta program where your users can
join to get early access to new features and improve-
ments. This will give you a high-quality pool of users.
TO RECRUIT PROSPECTIVE USERS:
When you’re searching for users that sit outside of your current base, or you’re in the
early stages of your business and don’t have any users yet - don’t panic. There are
plenty of methods to use to reach your core audience:
1 RECRUITMENT PLATFORMS 3 TARGETED ADS
Online recruitment platforms are the quic- A common, cheaper, tactic to find prospec-
kest way to find candidates from a huge tive users is to create highly targeted ads on
pool of consumers and professionals. They Google, Facebook, Instagram and Twitter
help manage the entire recruiting process, aimed at recruiting users within that criteria.
including qualifying candidates based on
your needs all the way to compensation. The ads would direct users to a form (Type-
form is perfect) or a link to book a meeting
Some can even locate participants based on in your calendar (Calendly is a popular
very detailed criteria - how about a 25 year- scheduling platform). This helps you get the
old Chef from Turkey that loves Tennis? Or exact type of person you need, at relatively
students studying law in Brazil that use Ma- low cost (don’t forget an incentive in the
cBooks? Anything is possible. Both options advert!).
below support Marvel prototypes too!
You can also jump into LinkedIn Groups or
• User Interviews ($) - Great for US-based Facebook Groups that match the persona
participants. you’re going after. These groups can con-
• Testing Time ($$) - Great for European tain hundreds even thousands of your target
and Spanish-speaking partcipants. audience.
2 GUERRILLA 4 LISTING WEBSITES
This recruiment technique is as simple as One of the cheapest ways to recruit users
you can get! Grab your prototype and your is through listing websites, like Craigslist or
test plan and head to the street! Gumtree, where your ads can reach thou-
sands of people, globally.
This usually works best for consumer-facing
apps and services. You will only have a few Recruiting this way might mean that quality
minutes to run your session so make sure may vary, so be as clear as possible in your
you adapt your test to the situation. You can details
read more about Guerrilla Testing here.
Recruiting Internally
for High Level
Feedback
We’re big believers in creating an environment where the design

process is more open and inclusive, giving all people across the busi-
ness the ability to share their own perspectives on solutions.
Internal testing is a great way to imple- Not only will you gather insights, but ma-
ment this kind of inclusion. Putting pro- nage communication, collaboration and
totypes in front of your colleagues is a camaraderie with new launches.
fantastic way to gather feedback from
your team and direct stakeholders, and
pull insights from the wider business. When to test internally
New features, or products, can often • If the changes to your product will
impact other departments. For example, impact other departments
a redesign has moved functionality that’ll • If you are creating internal apps, fea-
lead to an increase in tickets for the su- tures or services that aren’t externally
pport team, or perhaps the introduction facing
of multiple currencies in your checkout • To create an inclusive design and pro-
might change how finance reports on blem solving culture
revenues. • Early and often in your process
METHODS TO RECRUIT INTERNAL USERS:
The great thing about internal testing is that it’s fast and
free - as your audience is within your reach either in person
or over email, Slack or phone. The size of the business will
determine the best method for scheduling internal testing.
1 Email Blast:
Ping an email to the relevant teams with details of the
test, the time you’ll need and how many volunteers you
need. Sending a Calendly invite link works great.
2 Meeting Shout Out:

If you have a company all hands, why not highlight
details of the test and what you need there - try reser-
ving a room, or create an online meeting for a couple
of hours and ask for drop-ins.
3 Scheduling a coffee
This could be done in-person or over Slack or Zoom.
Ask for a quick 1:1 so you can walk them through the
test and ask questions as you go.
Pro tip: A super quick way to gather rich feedback from internal
users is to create a User Test in Marvel which provides a single URL
which you can share in Slack channels, emails or your documents.
This allows colleagues to quickly use your prototype and leave audio
and video feedback. Click here to create your first test.
Note: Remember, internal testing is not a replacement for testing

with real-world users, it should be done in conjunction and as part of
a multi-stage process.
Chapter Four
User Testing
Methods
Three different methods of

user testing.
User testing, or usability testing, comes

in many forms and there isn’t really a one
size fits all solution – especially if you’re
working without a research team.
In the following chapters, we’ll define

and guide you through three of the most
popular methods:
• Moderated User Testing

• Remote Unmoderated User Testing
• Guerilla User Testing
As designers, you might experience time

constraints, budget limitations or dry
spells in other resources.
That’s why introducing varied user tes-

ting methodologies can present you with
quick wins, that help you bridge the gap
between your product and end-user and
might end up saving your user experien-
ce.
A Guide to Moderated
User Testing
Moderated testing is often prefered because it leads to higher-qua-
lity feedback over unmoderated testing. Face-to-face or remote, it
works well for detailed, controlled and in-depth sessions, especially
around complex prototypes and flows.
Moderated user testing sessions have impact the results. For moderated testing,
both the participant and researcher pre- you might set conditions of where the
sent throughout. test must be carried out, for example, in a
restaurant if it’s a bill splitting app.
Unlike other testing methods, you’ll be
live - querying, interacting and answe- As you’ll be coordinating schedules and,
ring any questions the participants might in some cases, finding a space too - you’ll
have. Which means you have a real oppor- find moderated user testing uses the most
tunity to build a strong rapport with users, resources.
so be sure to include a good mix of open
and closed tasks and questions. But don’t let that put you off! You’ll be
rewarded with responses that surpass
These sessions can be carried out remote- most unmoderated methods.
ly, in-person, or anywhere you can set up
your test and control the conditions. This
is important, having different environmen-
tal conditions for each participant could
PERFECT FOR
• Generating both qualitative and quantitative data: having a range

in questions and data points is always a plus - but having interactive
sessions can also lead to candid feedback that might not have come
up otherwise.
• If you’re studying a prototype that has a complicated flow, or

limited functionality/fidelity that would benefit from your additional
questioning throughout, you have the opportunity to learn much
more about the customer journey and pain points.
• Catching extra details: Being in person means you can observe

body language you might have missed otherwise, highlighting areas
that might cause confusion. Whilst moderating over the phone
might encourage the user to be more verbal about their thought
process.
HOW TO RUN YOUR OWN SESSION:
Create your test plan

Using our template that will help you specify your
1 test goals, identify your users and define your tasks
and questions. You’ll find it in the ZIP folder down-
load.
Recruit participants
Identify which method works best for you out of
those we outlined in the previous chapter.
2
Choose the location and share details

As participants will be carrying out tests with you pre-
3 sent, you’ll need to decide the best place to run the
session. This can be in the street, an office or online via
video conferencing platforms like Zoom or Google Han-
gouts.
Make sure they are prepped with enough context before-

hand and any requirements such as devices and prototy-
pe links.
Note: Use Marvel to prototype and user test, so all the hard work
of recording is done for you. It automatically records video, audio
and interactions whilst your participant is carrying out the test.
To run remote moderated user testing, pair Marvel with a comms
platform like Zoom or Google Hangouts to instruct people while
testing.
Write a script
Having a standard set of questions to ask during the
session helps bring some uniformity to the results you
gather, but don’t afraid to go with the flow if things don’t
4
go according to plan!
Define how you’ll measure the test

You can use the templates we’ve outlined for recording
5 observations, goal completion and other notes in this
folder.
Note: A quick and easy way to track usage and other metrics is to
test using Marvel. Misclicks, duration, device type and other data
points are automatically recorded.
Run the test
We’re live! Run through your test plan. Some things to bear
in mind when sitting in on tests with users:
6
a. Make the participant feel comfortable
b. If things go very quiet, encourage them to think aloud
c. Avoid leading questions and the urge to jump in and
help. Try starting questions with ‘How might you’.
Capture results
Our template will guide you through capturing whether tas-
7 ks were completed, how easily users navigate your product

and additional notes from your observations. Find it in the
ZIP folder download.
Your tech set up should capture most of the observational

body language, so try not to note down too much during the
session as this may negatively distract the user.
Note: Marvel automatically stores results when submitted.
AT THE END OF YOUR SESSION:
Once you’ve completed your tests, remember to thank your users and
provide their rewards. Give them your contact details if they have any fo-
llow up questions. Now, you’ve collated all this data, it’s time to go away
and analyse it and create actions to improve your prototype.
A Guide to
Remote Unmoderated
User Testing
The revolutionary user testing methodology that’s low cost and low
effort. With unmoderated user testing you can share and test proto-
types with your audience and get their feedback, wherever they are
in the world.
Unmoderated user testing just refers to muting. And a more natural environment
the fact that the user carries out the test means more realistic results.
without a moderator present. This could
mean you and your team are all hiding be- Remote usability testing is carried out
hind a one-way mirror, but we’d like to co- using software, like Marvel, where you can
ver remote. Why? Because with new user share a prototype with participants to run
testing software like Marvel, this method tests pretty much anywhere.
is so easy to do and reduces time spent as
a designer on user testing. Test participants are provided with the
pre-determined tasks to perform whilst
It’s now possible to share prototypes with the software records audio and visual data
users via a link and have them test pro- for you to watch and analyse when the
ducts or features in their natural envi- test is complete and the data sent right
ronment, like in the office, home or com- back to you.
PERFECT FOR
• Testing product usage in a specific context, for example, testing a

splitting the bill app in a restaurant. This is more specifically called
destination-based testing.
• Saving resources - you’ll find that the design or research team time
spent is cut in half or more through running tests remotely and un-
moderated. You won’t have to schedule appointments or take time
out to attend tests.
• Running fast user research. Multiple tests can be ran concurrently

which means you can gather feedback from several users at the
same time - which in some cases might even allow for you to gather
invaluable insights from your audience in just a day.

using our template that will help you specify your
load.
Create your Marvel prototype

Marvel turns sketches or designs into interactive
prototypes. No coding required. This means you can
quickly test engaging experiences that look and feel
2
like the real thing, just without the effort!
Create your test

Start a User Test project in Marvel and select the
3 prototype you’d like to test, choose whether you’d
like to add a goal to the user test - in which case
the test will end when a user reaches the goal.
Invite participants
Through simply sharing a link, or inviting via email,
you can invite participants to carry out the test
from their own device, in their own environment
(unless specified to be in a specific context). You
4
can include an intro and outro message if you like.
Wait
With all the hard work done, go work on another
5 project, have that meeting you’ve been meaning to
have or just make yourself a well deserved cuppa
whilst you wait for the results.
Measure success
Our template will guide you through capturing
whether tasks were completed, how easily users
navigate your product and additional notes from 6
your observations. Find it in the ZIP folder down-
load.
A Guide to Guerrilla
User Testing
One of the easiest usability testing techniques that’ll help validate, or
invalidate, your ideas by generating results fast. Cut costs and time
spent by eliminating recruitment with guerrilla ux testing.
If you’re looking for a method to get fast can run interviews within that location by
feedback on your prototypes, Guerrilla approaching people in the vicinity and
User Testing takes the cake. produce instant qualitative feedback. Whi-
lst it’s better to recruit participants who
It’s a technique that takes advantage of match your buyer persona, most of the
the audience that’s immediately available people you meet will be an expert in using
to you, by simply heading to the nearest some device.
local cafe, park or communal space with
your work in hand. Not needing a lab, paperwork or recruit-
ment makes guerrilla UX testing extremely
Designers and researchers all the same lean and agile.
PERFECT FOR
• Frequent tests throughout the design process due to it’s low cost
impact
• Testing user experience and identifying potential usability issues

early on in the process
• Running alongside broader usability testing methods which produ-

ce more metric driven results

using our template that will help you specify your
load.
Write a script
Whilst in Guerilla User Testing, you won’t be able
to control all the environmental conditions but you
can manage how you speak to the participant and
2
the way you carry out your test.
Choose your device

Mobile devices and tablets best suit this method of
3 testing. Make sure you have one you can use during
the testing session to hand to participants for use.
Note: The Marvel iOS app can launch test sessions on the
go! All you need is your iPhone or iPad.
Pick your spot
Go to the high-street, cafe, supermarket or wherever there

are new people you can approach to test your prototype.
Remember everyone in the area is a potential participant!
4
• Create a few qualification questions to make sure they
fit your criteria.
• Don’t be discouraged by people not being interested!
Interviews
It’s important to give the participant space to vocalise their
5 thoughts and feelings. Remember to be as quiet as possible,

embrace silences but if you can see that a user is proces-
sing a lot internally, give them gentle nudges to think aloud.
Capture Results
Our template will guide you through capturing whether

tasks were completed, how easily users navigate your pro-
duct and additional notes from your observations. Find it in
6
the ZIP folder download.
• Try to refrain from writing too much as this can be dis-

tracting for the user and you may get skewed results.
• Set your mind to observe and at the end of the test
summarise the top 3 usability problems experienced
and whether or not the user completed their task.
AT THE END OF YOUR SESSION:
Once you’ve completed guerrilla user testing you’ll have some actiona-
ble feedback which will highlight the core areas you need to work on to
improve your user experience.
Chapter Five:
Creating a Shareable
Insights Document
Creating a Shareable
Insights Document
So your user testing has come to an end

and you’ve collected your notes, video
and audio recordings. Now what?
The final step is creating a document of

the valuable insights you’ve discovered,
enabling you to share what you’ve lear-
ned with others - whether that’s with your
immediate team or throughout the wider
business.
Remember the more people who have

access to user research, the better for
business. Not only are there are plenty of
teams that will find this research useful,
there are others which might be directly
affected by any changes you make off the
back of it.
Here’s what you should know about crea-

ting an insights document.
What is an Insights Document?
If you plan on sharing your research with the document depends on the type of
your team, wider organisation or stakehol- testing and the audience. If you’re valida-
ders, an insights document is a great way ting something small with your team, then
to package up everything you’ve discove- simply condense it into a Slack message
red during your testing sessions (and get or email, and make it heavy on the bullet
buy-in when needed). points. If you need to present to stakehol-
ders, then make it longer.
The level of effort you need to put into
HOW TO STRUCTURE YOUR DOCUMENT
What this user research is about

1
A brief description on why you conducted these user tests and
what you set out to achieve
Who you tested with

2
A breakdown of the participant profiles.
What you tested

3
Provide links to prototypes and designs.
Highlights
4
This is a brief summary of the key highlights from your testing
and allows the reader to continue to the full summary if they want
to go further in depth. Keep this to 5 bullet points maximum and
make it concise.
Quotes
5
It’s easy to overlook quotes but often there’s things that are said
during test sessions that can cut through the noise and package
up the sentiment for the entire test session. Include the quote and
the profile of the participant for additional context.
Metrics
6 It can often be difficult to decide whether a user test was a suc-
cess through qualitative feedback alone. One of the most power-
ful ways to decide whether a design or prototype is the right solu-
tion is to track how users move through the flow. For example how
many users completely the goal or reached the correct screen?
What was the average time taken?
Pro Tip: When you create a user test in Marvel, the data is automa-
tically collected and displayed for every test so you can quickly see
which users had issues and surface designs that perform better.
Using this data, you can compare different participants or even di-
fferent tests and see how they perform. Keep a count during your
sessions and add this to your write ups.
Full summary
7 A longer overview of your findings during the test sessions. This
should constantly relate back to why you conducted the research.
Keep this concise and include links to evidence where possible.
Links
8 If you’ve collected photos, surveys, video and audio recordings
and other links along the way add them in here.
Chapter Six
Conclusion
Now, get out

in the field
We hope using this guide results in creating an invaluable channel
for user feedback that will ultimately transform not only your design
process but the also products you create. The premise is simple, vali-
date your ideas early on, remove assumptions through research and
build better products.
Take the time to figure out which methods We’ve mentioned a number of tools within
work best for you and your team, so you this book that can help you streamline the
can make the user testing process as whole process, whether you’re carrying
seamless as possible. As we outlined ear- out moderated, remote or guerrilla user
lier, there are a number of reasons compa- testing, including our own User Testing
nies skip user research, but now knowing tool.
what you know, be a part of the 45% that
doesn’t make those excuses - make de- To learn more about our offering, why
sign your competitive edge. not catch our next product webinar?
Register for free here.
Prototyping meets User Testing.

Find out more: marvelapp.com/user-testing

The User Testing Field Guide

Uploaded by

Copyright:

Available Formats

You might also like

The User Testing Field Guide

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

The User Testing Field Guide

Uploaded by

Copyright:

Available Formats

The User Testing

The User Testing

Where prototyping meets user testing.

Marvel is a design collaboration platform em-

We’ve also created a space for our community to

Design: Filippo Chiumiento

To learn more about our user testing feature, visit:

Introduction User Testing Methods

Why Bother With User Research? 10 Remote Unmoderated User Testing 38

Create a Test Plan Analysing Data

Recruiting Participants Conclusion

Recruiting Internally for High Level 30

Welcome to The User

“If you think good

At Marvel, we approach usability as one mething functional, reliable and usable?

As a prototyping platform, it should come as no surprise to hear we send

• When needed, break larger flows down into smaller chunks

• Segmentation, segmentation, segmentation

1 Existing feature improvement tweaks.

2 Short term roadmap feature.

3 Long term roadmap.

4 R&D, innovation, random ideas.

WHO WE TEST WITH

Within the group, we have a mixture of small and large companies in

2 Users who have submitted a feature request

3 Potential customers or new market opportunities

4 Users who have tried a particular feature

Why bother with

Running regular user testing sessions can see positive

We’ve included a list of the common objections you might hear, so

• No dedicated researchers. • Tight deadlines.

• No usability lab. • No budget.

The key is to match the level of tes-

By the end of this guide, you’ll be able

Types of prototypes you can use for User Testing:

LOW-FIDELITY (PAPER SKETCHES)

Creating a Test Plan

Create a test plan.

A prototype isn’t the only thing you need

Now is the opportunity to break down

• The feature, or product, you’re testing

Tackling this early on will help you create

To create your own unique plan, we’ve

To carry out a successful user test, it’s hel-

feature, or and clearly outline what’s being tested.

testing In this stage, you should outline the key

It’s also an opportunity to outline anything

It’s essential to break down why you need

business that your work is driving towards the busi-

So far, you’ve outlined what you’re testing

test objectives ne in place, it’s the perfect time to outline

For each feature, 2-3 objectives should be

It’s important to avoid broad objectives - it

For example, let’s say the feature you’re

• Objective 1: Users notice the new add

Before diving into recruiting participants

audience participants you are using. You’ll more

A helpful exercise can be defining their