Adobe

Start Free XD Trial

XD IDEAS / PROCESS / USER TESTING

Everything You
Need to Know
About Beta Testing
Nick Babich
Oct 11, 2019
 Adobe

Testing is an integral part of the product design cycle. Different phases of the
cycle require different types of testing. In this post, I’ll explore the type of
testing that can reduce the risk of product failure through customer
validation – beta testing.

Although the practice of beta testing is fairly common among product

teams, there is still a lot of misunderstanding on what beta testing really is
and what benefits it offers to product design teams. We will fill the gaps by
providing best practices for beta testing and things you need to consider
when running them.

What is beta testing?

Beta testing is a type of user acceptance testing where the product team
gives a nearly finished product to a group of target users to evaluate product
performance in the real world.

There is no standard for what a beta test should look like and how to set up
beta testing. The actual testing procedure should be relevant to your testing
goals. However, there are a few requirements that a product needs to
comply with in order to be ready for beta testing:

The product should be in the “Feature complete” state (it should have
all the features that are planned for the release version).
The product should be stable (test participants should not face
unpredictable crashes).
 Test participants should belong to the product’s target audience.
Adobe
Test participants should complete real-world scenarios while interacting
with an app, and they should do it in the real environment (not in lab
environments).

Alpha testing vs. beta testing:

what is the difference?
Before discussing different types and approaches to beta testing, it’s
essential to differentiate it from alpha testing.

Alpha testing is the testing phase that precedes the beta test phase. Unlike
the beta version, the alpha version is usually less stable and might have a
limited feature set. Alpha testing is done by an in-house team of developers,
designers, and QA specialists. Alpha testers usually mix black-box (testing
method in which the internal structure, design, implementation of the item,
product, and feature being tested is not known to the tester) and white-box
testing (testing method in which the internal structure, design, and
implementation of the item being tested is known to the tester) techniques
to discover bugs and crashes. Alpha testing can be controlled activity
(meaning, it’s possible to reach test participants and ask specific questions
about their experience).

“
In the modern software
development processes, beta
testing should be an integral
 Adobe part of the product
development cycle.

Beta testing is usually black-box testing, meaning test participants don’t

know anything about the backend and don’t have access to source code.
Since beta testing happens most of the time at the end user’s side, it cannot
be a controlled activity. Beta testing can provide extremely valuable insights
– genuine scenarios of interactions with a product.

For example, during beta testing, it’s possible to match the expected user
journey with the actual user journey, by gathering behavioral habits (i.e.
whether the user prefers to use mouse or keyboard during the interaction
with a software) and emotional responses on interactions (i.e. how your
design decisions make your users feel). It’s also possible to create an
empathy map based on the actual data from your users.

The role of A/B testing in product

design: phases of testing
In the modern software development processes, beta testing should be an
integral part of the product development cycle. Ultimately, the moment for
conducting a beta test in a development cycle depends on the nature of a
product you’re developing. Some teams reserve beta testing only for major
releases, while other teams conduct beta testing for both minor and major
releases.
 Adobe

Image by Olivier Le Moal

While the phases of testing vary based on the needs of the organization,
generally the procedure of testing looks like this:

Pre-alpha testing (in-house testing done by the product team during

the initial development stages)
Alpha testing (in-house testing done by the product team in a lab
setting with a more developed prototype; it’s possible to invite a small
group of trusted users to participate in the alpha test)
Beta testing (out-of-house testing where subjects use the product to
get initial reactions of finished product)
Release testing (final testing before shipping the product to the market)

Every phase should have clear exit criteria. For instance, for beta testing the
exit criteria can be zero open high-priority issues in a bug tracking system.

Types of beta testing

Below are a few common types of beta testing. Note that beta testing can
include several different beta tests of different types.
 Adobe
Closed beta testing vs. open beta testing

Closed beta version (also known as private beta) released to selected people
who test its features. Usually, the number of test participants is limited. For
example, if you plan to ship a product on the market in the next few months
and have a landing page that allows visitors to leave their emails in order to
get the latest updates about the product, you can choose beta testers from
people who signed up for product updates. Closed beta tests are better
suited for beta tests with a limited scope (only core features of the future
product). And since the number of unique issues found by each tester
decreases as the number of testers increase, it makes sense to limit the total
number of test participants.

More users do not equate to more problems being

uncovered.

On the other hand, open beta tests do not restrict access and allow anyone
to sign up for beta testing. Open beta tests usually follow a closed beta
testing phase. This type of beta testing is great when you want to collect
Adobe
quantitative data about your target users and their interaction patterns. This
type of testing also provides some insights on how well your infrastructure
scales – whether your backend can handle a real number of users. However,
it can be hard to analyze the results of open beta testing, especially if you
have a large number of test participants. Google for example, beta tested
Gmail and it took five years until the product was completely usable and fully
publicly accessible.

Technical beta testing

A technically focused beta test is beta testing with a group of tech-savvy

users (usually, it’s an internal group in the organization). The goal of this
testing is to uncover complex bugs and provide high-quality test execution
reports to the engineering team. The fact that tech-savvy test participants
have a tolerance to minor issues makes the testing more focused. Test
participants are more willing to complete the testing despite the problems
they face along the way.

Focused beta testing

Focused beta testing is used when a product team needs to gather feedback
on specific features. In order to collect the feedback, the team releases the
product to the market.

Marketing beta testing

Marketing beta testing has a goal to get media attention for your product.
Generally, this type of beta testing can help you evaluate your marketing
channels. But it’s also possible to use beta testing to understand customer’s
reactions to new product features. Some products like Instagram have both
active goals in their marketing beta testing.
 Adobe
What you need to run beta
testing
Like any other testing process, beta testing also begins with proper planning.
During this stage, the team prepares a testing strategy. Here are a few points
that an organization needs to consider:

Define the goals in advance

First and foremost, it’s important to know what goal (or goals) you aim to
achieve from your beta test. What exactly do you want to test? Is there a
particular user flow or specific feature? Based on your goal, you will be able
to select the testing scope and find the most relevant type of beta testing.

Recruit the right test participant

Identifying and recruiting the right participants is the major challenge. The
fact that test participants should have the necessary skills to use your
product can be a problem for some groups of products. An improper
number of participants is among the common reasons why beta testing fail.

But how do you calculate how many beta testers you need? Since all projects
are different, it’s hard to provide a one-size-fits-all formula for calculating the
number of test participants but you can rely on the triple constraint concept
to get this understanding. This concept says that cost is a function of time
and scope and that these three factors are related in a defined and
predictable way. If we apply this concept to beta testing, we will see that
when we increase the scope, we also have to increase time or cost to
preserve quality. In terms of the number of test participants, the right
amount of testers provide enough feedback to prove the statistical
significance of your findings. The article “How to RightSize Your Mobile App
Beta Program” shows how to do it in the context of a mobile app.
 Adobe

Use the triple constraint concept to determine how

many beta testers you need.

Define the length of the testing period

Both too short and too long test periods lead to non-representative test
results. It’s important to decide how many days you want to keep the beta
version available for beta testers before starting the testing. The timeline can
be determined by a set goal you have put in place, or by utilizing a set
portion of a budget, among other factors.

Write product documentation

The product should have How-to-Setup and How-to-Use instructions. The

instructions should be detailed, jargon-free, and reviewed for correctness.
Test participants should be able to access the instructions whenever they
Adobe
want.

Share the information about known issues

with test participants

If your product has some known issues that have a negative impact on user
experience, it’s better to share the list of issues with test participants before
starting beta testing. By doing that you make test participants more tolerant
of the issues.

Share results of beta testing with the

product team and stakeholders

Design team, development team, QA, product management – all should

receive results of beta testing.

Create a clear procedure for collecting

feedback

While some information about user behaviour should be collected

automatically (i.e. your product should have a built-in mechanism of sending
crash reports and other system information), it’s also important to create a
clear channel for communication between test participants and your product
team. Test participants should be able to share their thoughts on product
features and submit requests for changes in design.

Use tools for beta testing

Logging bugs, measuring productivity, and collecting feedback from users

are typical things you do during beta testing. It’s important to have a tool (or
a set of tools) that will simplify the process of beta testing. Here are a few
tools that can help you run effective beta testing:

https://buglife.com/
 https://instabug.com/
Adobe
https://www.bugsee.com/

Conclusion
Beta testing is a fantastic practice that allows you to test your product with
real users before it reaches the market. Though the purpose of beta testing
may vary depending on the product, the ultimate goal remains the same –
create products that will have an excellent user experience.

User Testing

Words by
Nick Babich

Nick Babich is UX architect and writer. Nick has spent the last 10 years
working in the software industry with a specialized focus on research and
development. He counts advertising, psychology, and cinema among his
myriad interests.

Related Content
→
 Adobe

US E R T ESTI NG U SER T ES TING

How to Design a Successful Design Inspiration for UX

User-Centered Beta for Your and UI Workflows
Product B Y D AN SILVE IRA

B Y LAURA H E RMAN Design Inspiration for UX and UI Workflows.

Over the years at Adobe, we’ve developed several

beta best practices. These are the 10 key themes.

L EAD ERSH IP INSIG HTS SOC IAL IM PAC T

Maintaining Team Announcing World

Engagement in the Time of Interaction Design Day 2020:
Remote Work Culture and Sustainability
B Y RACHEL RO SE N B Y LIN D SAY MU NRO
The pandemic has driven us towards remote and We’re thrilled to announce that World Interaction
Adobe
hybrid workplaces. So how can leaders keep their Design Day (IxDD) will return for its third year,
teams engaged? presented in…

Let's XD together.
Where teams create the world’s best experiences at scale,
powered by the leader in creative tools.

Start Free XD Trial

Get XD Ideas delivered to your inbox. Free.

Sign Up

Process

Principles

Perspectives
 Design Circle
Adobe
About

Inspiration

Guides

Career Tips

Copyright © 2019-2023 Adobe. All Rights Reserved. / Privacy / Terms of use / Cookie Preferences / Do not sell or

share my personal information