THE HEAP GUIDE TO RETENTION

The Heap Guide to Retention

A Smarter Approach to Product Analytics

The Heap Guide to Retention • heap.io 1

 Table of Contents

Introduction 3 Scenario 2: Hypotheses for Creating 15

Ongoing, Repeatable Value
Section 1: Why Retention Matters 4
16
Scenario 3: Hypotheses for Creating
The Business Case for Retention 5 Longer-Term Retention
Down the Funnel and Back 6
Section 3: Examples from Heap 18
Good vs. Bad Retention Graphs 7
Finding Our Retention Metric 18
Three Stages of Retention 8
Two Retention Metrics 20

Section 2: Improving Retention 9 A Hypothesis Story: Suggested Reports 22

Defining Retention 9 Suggested Reports: What We Learned 23

Measuring Baseline 10 Suggested Reports: Impact on Retention 24

Critical: Avoid Measuring Incorrectly 11 Suggested Reports: Group By 25

Choose the Stage to Target 12 Suggested Reports: Final Learnings 26

Scenario 1: Hypotheses for Encouraging 14

Closing 27
New Users to Stick Around

The Heap Guide to Retention • heap.io 2

 In what follows, we describe why retention should
remain a critical focus for Product Managers,
Introduction
INTRO

what hypothesis-driven PMs can do to measure

and increase retention. We’ll also provide multiple
examples from our own practice that suggests
Use the “leaky funnel” analogy or say your product is
how PMs might go about developing and testing
a “boat with a hole in it”: if your product can’t retain
hypotheses that address retention in their own
a solid base of users, your business can’t sustain
products. At the core is the idea that a PM’s job,
revenue, and your company can’t grow.
when done right, involves using analytics to focus in
Welcome to the Heap Guide to Retention. Our on problem areas and then creatively designing ways
goal in writing it is to give Product Managers more to address them.
opportunities to provide undeniable value for their
As you read, you’ll realize that many of our strategies
customers. In particular, we believe that Product
are about pursuing smaller wins. At Heap, we believe
Managers can increase retention rates most effectively
that incremental improvements, when implemented
when they take a data-driven, experimental, iterative
constantly and repeatedly, pay enormous dividends. As
approach to product development, one that involves
long as you’re being creative with your ideas and both
forming and testing hypotheses, figuring out what to
rigorous and methodical about testing them (including
measure, making small improvements, and learning
recognizing when hypotheses are wrong and having
from every single experiment, successful or not.
the humility and ambition to change directions as
needed), you’re doing it right.

Read on to improve your product or feature!

The Heap Guide to Retention • heap.io 3


SECTION ONE
01
Why Retention Matters
Retention is the True Measure of Product Value
Building a product-driven business takes energy. There’s
marketing to generate interest, and a sales apparatus to
bring deals home. On the product side, there’s the initial user
experience to develop, and the value your product should
demonstrate from the get go.
These activities are all important for getting customers in
the door. But once people start using your product, it’s your
retention strategy that keeps them there. Plenty of users
are willing to try a product once or twice. But if your product
or feature doesn’t provide ongoing value over time, or is too
difficult to use, or if the pains your product addresses can
be more easily solved by a competitor, those users won’t
stick around.
The Heap Guide to Retention • heap.io
In fact, from a product perspective, it’s not unreasonable
to say that retention is the best measure of product-
market fit. For the bulk of digital products—websites,
mobile apps, business platforms, and many more—
achieving product-market fit means producing a product
that users willingly return to.
It’s for these reasons that Product Managers should
be somewhat obsessed with retention: measuring it,
understanding it, developing their product or features
with an eye towards improving it. Other growth metrics
matter too, of course. Acquisition numbers tell you how
well you’re able to sell your product to customers who
haven’t used it yet. Activation numbers tell you how many
people you can get into your product. Retention alone
tells you if customers are finding ongoing value in
your product.
For hypothesis-driven PMs, there
will rarely be a time when increasing
retention is not a major goal.
44
The Business Case for Retention
Another way to think about retention is to measure business
impact. Most of us know that it’s far more expensive to acquire
a new customer than to retain a current one—anywhere from
5 to 25 times more expensive. From the business’ perspective,
a poor retention strategy puts unnecessary pressure on your
acquisition efforts—marketing and sales.
You can see how increasing or decreasing retention by even a
single percent has on your yearly revenue. Simply decreasing
retention rates to 4% gives you an increase of 7 customers a year.
These 7 customers translate to meaningful business value.
As a product manager, you partner with marketing and sales
to entice customers to your product. But you also have special
insight into what makes them stay.

The diagram at left gives a quick explanation. To the question, Impact of Monthly Churn Across a Year (# Users)
“would you be happy with a monthly retention rate of 95%?”
most product companies tend to answer yes. (When we ask this
100

question in presentations, the majority of the audience raises

its hand.) 90

Yet the numbers at the bottom of the chart display the 80

compounding effect of a 5% churn rate. Simply losing 5% of 70

your users a month means a yearly churn rate of 43%.

60

This 43% doesn’t only mean a loss of income (via diminished

50
subscription costs, ad sales, renewals, or whatever business Jan Dec

model your company supports). It also holds back growth,

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
since you need to grow by 43% per year to simply maintain
a constant number of users. 4% 100 96 92 88 85 82 78 75 72 69 66 64

5% 100 95 90 86 81 77 74 70 66 63 60 57

6% 100 94 88 83 78 73 69 65 61 57 54 51

The Heap Guide to Retention • heap.io 5

Down the Funnel and Back
One can also think about retention as the gift that keeps on
giving. Much of what happens further along the customer
journey depends on how well you’re retaining your customers.
Renewals, expansions, referrals, recommendations: all only
happen when your customers continue to find value in
your product.
Let’s think a little bigger. For each individual customer, retention
is key for moving them down the path. For your company as a
whole, however retention plays an outsized role on acquisition
and activation. Why? Because retained customers give you proof
points, case studies, and other documentable evidence that your
product works. Showing a market that you can provide ongoing
value to customers is the easiest way to get prospects excited
about what you can do for them.
The effects extend outwards. Demonstrate stickiness and you’ll
find partners more interested in working with you. Retain well-
known customers and word spreads. Having a stable of satisfied
customers to call on gives salespeople a critical tool for their
arsenal. And so on.
In fact, we could say that for most product-based companies,
retention is the primary objective for sustaining growth.

Key Metrics for Growth

Acquisition Activation Retention Revenue Referral

Customers Finding value Continue to come Renewals, up-sells, Customer advocates
signing up in your product back over time cross-sells on your behalf

The Heap Guide to Retention • heap.io 6

Good vs. Bad Retention Graphs Bad Retention Example

At right you can see the difference between an effective and a 120,000

non-effective retention strategy. The graph at top reflects a user 100,000

base with a strong initial period of acquisition and activation,

80,000
then a long-term flattening out, indicating high churn and
(assuming the company is putting effort into bringing in new 60,000

users) poor long-term retention. 40,000

If you’re at the early stages of this graph, your growth rates may 20,000

be encouraging. But they can also be misleading, since they -

don’t tell you how many of your new customers are going to stay.

1/14
3/14
5/14
7/14
9/14
11/14
1/15
3/15
5/15
7/15
9/15
11/15
1/16
3/16
5/16
7/16
9/16
11/16
1/17
3/17
5/17
7/17
9/17
11/17
If you’re regularly measuring retention and are observing a graph
that looks like this, it should be cause for alarm.
Good Retention Example
On the bottom you can see what a user graph looks like when
9,000
a company has a good retention strategy in place. In this case 8,000
the number of users continues to increase over time, ensuring 7,000

that your company’s ability to retain users matches its ability 6,000

to acquire them. 5,000

4,000

3,000

2,000

1,000

-
1/16 2/16 3/16 4/16 5/16 6/16 7/16 8/16 9/16 10/16 11/16 12/16

The Heap Guide to Retention • heap.io 7

Three Stages of Retention Retention Rate

The graph at right—whose shape is typical for many Stage 1 Stage 2 Stage 3
companies—shows the standard way we measure retention.
The initial number—62%—represents the number of acquired
users who actually engage with your product. Typically this
70%
number drops significantly after a short period of time, here
60%
a month, and levels off into a more standard user rate. That 50%

number is generally what you’ll use as your baseline as you run 40%

experiments designed to increase retention rates. 30%

20%
This number can also be helpful for forecasting revenue. If you
10%
know that 16% of acquired users are likely to keep using your
0%
product and you have a decent idea of how long they’ll stick

Same
month

After 2
months

After 4
months

After 6
months

After 8
months

After 10
months

After 12
months
around, you can produce a useful forecasting model.

In this graph, you also see how user numbers slowly decrease
over the course of the year, indicating slow but steady churn. NOTE: Retention doesn’t have to be measured on the level of
The final significant drop occurs at the end of the year, when the your entire product. It’s just as important to track retention of
large majority of users still in the product or using the feature a specific feature. Doing this can help you decide whether it
drop off entirely. makes more sense to prioritize further development, to sunset,
to refocus, to simply to do more investigation.
As a Product Manager, it’s your job to know the retention
rates at each of these stages—initial use, longer sustained
period of time, and a larger drop off at the end—and to develop
hypotheses around improving them.

The Heap Guide to Retention • heap.io 8

 SECTION TWO
02

Improving Retention that the “visits” metric doesn’t tell you what actions users
are taking in your product. For many products, this can be
a problem.
Defining Retention
For example, many gaming apps add “daily reward” games
In order to develop and test hypotheses about increasing
to the user experience, where users who visit the product
retention, it’s important to first measure your retention
on successive days win a prize. (Often the size of the
rates and assemble them in a graph like the one on the
prize increases the more days in a row a user visits the
previous page.
site.) But these kinds of visits rarely correlate with
long-term retention.
Doing this can be tricky. At bottom, your retention metrics
should measure how many users performed an event
once, and then performed that event again later. The main
difficulty is that it can take work to decide what event in
your product counts as “usage.”

In Google Analytics, for example, retention is measured

by raw visit numbers. If a user visits your site, then visits
again later, they are counted as retained. The problem is

The Heap Guide to Retention • heap.io 99

Simple visits metrics can similarly cloud your ability to assess
the overall satisfaction. If an enterprise company purchases a
Percentage of Monthly Return Users
CRM (for example), that product will get hundreds or thousands Length of time after customer registration
of daily visits. Measuring them alone can’t tell you if users are
getting the value you want them to be getting from it.
50%

For more focused product leaders, the question is less “how

many people come to my site” than “what is the behavior
that people take in my site that signifies value, and how
many people repeat that behavior over time?”. 30%

What this behavior is often depends on what your product aims

to do. For e-commerce sites, the behavior might be “purchase.”
For marketing sites, it might be “view the blog.” For subscription 10%
services, it might be “renew.” And so on.

The key is that when defining retention, you want to brainstorm 1 YEAR 2 YEAR 3 YEAR
with your team, and home in on the sorts of behaviors you want
your product to drive long-term. Evernote’s famous “smile graph.” The longer a customer
used Evernote, the more likely they were to return. That’s
stickiness.

The Heap Guide to Retention • heap.io 10

Measuring Baseline
To produce the retention graph that captures your company’s
user base—the graph that will graph that will give you the
baseline measurements against which you’ll want to run
experiments—you’ll want to measure the number of users who
have taken the action you’ve designated as your retention
measurement, and plot those measurements over different
time frames.
Since Heap captures all your data and enables retroactive
analysis, it’s easy to look back and see how many people took
a given action once, then when and how often they took that
action again. Here’s how.
—— Navigate to Analyze → Retention. Ensure the Retention
Analysis option is selected at the top, then select
the following:
—— Start Event: The start event is the event that you’d like to
use as the foundation of the retention report. This would
generally be something basic that indicates initial interest in
your product, like “signup,” “start trial,” or “purchase.
—— Return Event: Generally the repeated action that you want to
see over time. This should be the event that best defines a
user’s getting value from your product (as described on the
previous page).
The Heap Guide to Retention • heap.io
—— Group By (optional): Group by any user property to examine
retention across cohorts or segments of users (this can
include things like “job title,” “completed X event within
a certain date range,” “referred by a specific page,” and
many more).
—— Date Range: Select the date range you want to look at.
Unless you’re focusing on new user retention, you’ll likely
want to focus on a longer time frame.
—— Granularity: This determines the interval by which we view
how often a person completes the return event. It can be
set to Day, Week, or Month. The granularity you choose
depends on the expected usage of your product (is it the
sort of product that people should come to twice a week,
or once a month?).
Run the query and you’ll see a line graph; you can also switch to
table view to see a breakdown of activity for each cohort.
NOTE: this is only the most basic way to run a retention query.
As you develop your analytics practice to run more hypotheses,
you can start comparing customer segments and further break
down your user base. To learn more, see Heap Docs, or watch
the Heap University training course.
11
 Running a Retention Analysis in Heap
Main Search for reports Save as Report
Production

Dashboard
Retention Retention Analysis
Reports
Start Event
Analyze Session

Graph Return Event

Session
Funnel

Users Group By Date of Start Event

Retention Filters Add Filter

Influence Compare Users Add Comparison

Paths
Date Range Past 7 Days grouped by Day

Queries
First Time
Define
Run Query
Capture

Activate
Users retained from Click Login to Click - Save Report comparing 1 segment
Account

The Heap Guide to Retention • heap.io 12

12
Critical: Avoid Measuring Incorrectly
When assessing the effects of a new feature on overall retention,
a very common mistake is to simply measure your retention
rates before the feature launch and your retention rates after.
The problem with measuring this way is that it doesn’t tell you
how directly (or even if) your feature affected retention. (Briefly,
it measures correlation, when you really want causation.)
Churned
Users
Returned Users With
Users an Action
The Heap Guide to Retention • heap.io
A better process is to measure both the number of retained
users who used your feature and the number of churned
customers who didn’t use it. Comparing these two numbers
helps you hone your analyses to better measure causation. (It’s
also important to compare the number of retained customers
who did use your feature with the number of retained customers
who didn’t use it, for the same reason.)
If retained customers were using your new feature but churned
customers weren’t, that’s a good indication that your feature
did make a difference. (It’s not a conclusive result, but it’s far
more indicative than simply measuring retention pre- and post-
launch.) At very least, this result suggests that getting more
users to adopt your new feature may make them more likely to
return to your product.
13
Choose the Stage to Target Finally, if you feel like you’ve sufficiently experimented and
iterated with your on-boarding process and feel confident about
Once you’ve established your baseline retention metrics, you
your product’s ability to activate new users, it’s time to move on
can start focusing in on different stages of the customer journey
to the next part of the curve, and find ways to get users in front
and develop experiments aimed at improving them. In general,
of other parts of the product.
your experiments will target one or more of the following goals.
The next few pages will provide some hypotheses you can test
1. You can get more new users to stick around longer.
in each specific area.
2. You can raise the level of ongoing,
repeatable value you provide. Retention Rate

3. You can figure out how people can continue to get value Stage 1 Stage 2 Stage 3
Get more Raise this Get this line
from your product, even after using it for a long period 70%
new users to
stick around
line, get more
people to find
to flatten
out/get more
repeatable value people who
of time. longer
continue to
60% find value

Figuring out which of these stages to focus on can take some

50%
work. If you haven’t thought about retention in this way, a good
beginning strategy is to first put effort into short-term retention. 40%

This effort tends to have a cascading effect, since increasing

short-term retention tends to pay dividends on medium- and 30%

long-term retention.
20%

On the other hand, if your retention graph tails off strongly at

the end, you should absolutely tackle this issue first, for the 10%

simple reason that if you cannot retain customers, you cannot

sustain a growth business. In the following pages we offer some
0%
Same
month

After 2
months

After 4
months

After 6
months

After 8
months

After 10
months

After 12
months
strategies that can help.

The Heap Guide to Retention • heap.io 14


SCENARIO 1 anticipated? Are you pointing users to quick wins, so they can see
Hypotheses for Encouraging
New Users to Stick Around
If your goal is to encourage more new users to play with your
product, there are a number of standard experiments you can
run to improve things.
They tend to cluster around a few key areas, though as always
you’re encouraged to come up with other creative ideas.
Trial Periods
Depending on your product, you may want to offer (or may already
offer) a trial period. There are many experiments you can run here.
You can extend your trial period, or shorten it. You can add more
features to your trial period, or remove them. You can ask users to
provide a credit card number before the trial period, or move that
request to the end. And so on!
The Heap Guide to Retention • heap.io
On-Boarding Flow
Revisit your on-boarding flow. Is it so long that users drop off
before they get through it? Are there moments of friction you hadn’t
immediate value in your product?
Direction In the Product
Does your product intuitively direct people to the activities that will
gain them value? Could it be organized differently? Could you present
information in a different way (through guides, say)?
Cross-Functional Efforts
Cross-functional efforts can go far towards increasing the number
of initial users. If launching a new feature, you can write blog posts
about it. You can run an email campaign, targeting specific users
whom you think would be most interested.
You can hold a launch party. You can do sales enablement to get your
salespeople to mention the new feature more often. You can identify
a group of users from whom you’d most like feedback and interview
them. As with any business proposition, the possibilities truly are
limited only by your time and creativity.
15

SCENARIO 2
Hypotheses for Creating
Ongoing, Repeatable Value
As with stage one above, there are many directions you can
pursue as you test out ways to keep more users engaged in
your product.
Tiers of Usage
Spend time identifying what the beginner, intermediate, and
advanced features (or types of usage) in your product are. Then
figure out how to help users at different stages uncover them. You
can do this in your product (with in-app guides, say or by targeting
pop-ups to people who have signed up months ago) or through
marketing and outreach (you could have your Account Managers call
customers and make sure they know about your new feature, or you
could launch a social media or paid advertising campaign).
The Heap Guide to Retention • heap.io
Engagement With New Features
Examine your features and assess how discoverable they are. Are
your features easy to use for newer users? Are they enticing to your
existing user base? Are there features that will excite ongoing users,
giving them new experiences? Are there incentives for users to start
adopting different features?
Marketing
A general best practice is to make noise around new features, or
around existing features you think are particularly important. You
can partner with marketing to draw attention to them. An event? A
press release? A customer campaign?
Deprecate
One powerful strategy is to deprecate features that are not used
often but which make your product more complex. PMs can hate to
let go of features they’ve had a hand in creating, but doing this—or
testing the possibility out—can make for a simpler, easier-to-use
product, and more quickly get users to value.
16

SCENARIO 3
Hypotheses for Creating
Longer-Term Retention
As mentioned above, if you’re witnessing a major drop-off after a
period of time, it’s imperative that you deal with it immediately.
If you don’t fix this problem, you’re putting your entire business
at risk. (If the issues is retention of a feature, your whole
business may not be at risk, but it may be worth considering
whether the feature is needed or not.)
End-stage drop off is a thornier issue to tackle, since the
problems involved may have little to do with the product itself.
The issue could be a new competitor to the space. Or a new
direction in the way your customers work or do business. (It’s
not really the horse’s fault that it got replaced by the train and
the car.)
The Heap Guide to Retention • heap.io
Moreover, the fact that people are leaving after using the
product for a longer period of time dramatically extends the
length of the experimentation cycle, making it more difficult
to gather data on your hypotheses. When fixing this problem,
you’re rarely going to get the kind of quick wins you can earn
when dealing with the first two stages.
That said, there are a number of areas to investigate and
experiment with. If you’re only dealing with a feature, the
discussions may not be as intense. But they still merit the same
kind of attention.
Examine Product-Market Fit
As before, these discussions will touch some more strategic issues.
Take a deep dive into your product and business model to assess
whether you’ve really achieved product-market fit. Are you selling
the wrong product? Are you selling to the wrong people? Is your
pricing model appropriate?
The issue could come from any part of your business. Are you giving
customers enough support? Are you following up in the right ways?
Figuring out how to solve these issues will likely involve other key
stakeholders in the company. Should you expand your product?
Change your pricing model? Sell to a different market? And so on.
17
 Make Your Product Indispensable Monthly Subscription Amount
By quarterly cohort
Work hard to isolate the features that make your product
indispensable. Then double down on your key differentiator(s).
What can you do that your competitors cannot? How do you make
these features invaluable to customers, so that if you left the market
they would feel real pain? These discussions will likely involve
more than your product team, and may include any number of
C-level executives.

Talk to Customers
If you’re not doing this already, it’s critical that you talk to
customers who have churned. Find out why they’ve left and 2013 2014 2015 2016 2017 2018
take copious notes. Base your decisions on this information.
Dropbox’s retention graph, from the company’s S-1.
Nice and healthy!

The Heap Guide to Retention • heap.io 18

 SECTION THREE
03

Examples From Heap

Finding Our Retention Metric
As we noted above, the first part of developing a retention
strategy involves isolating the activity that identifies
a user’s finding value in your product. Settling on this
activity can take some time, and no small amount of
experimentation.

A great strategy for pinpointing this metric (and what we

did at Heap) is to convene a cross-functional team for
a meeting, then collectively come up with a list of 10-12
activities that people think may work.

Once you’ve agreed on these 10-12 candidates, you can

run retention analyses in Heap and see which activities
bring the biggest lift.

The Heap Guide to Retention • heap.io 19

19
When Isolating Your Retention Metrics:
Time span matters
Knowing how long it takes users to get to a
specific action changes your retention strategy,
often dramatically. If the key retention event is
something that’s taken in a user’s first day in the
product, you’ll be far more aggressive in trying to
get users to that event. If it typically takes a week,
a drip campaign may be more appropriate.
Measure something useful
As we saw with the Google Analytics example on
page (??), it’s important to measure something
useful. You could track the number of people who
click a log-in button, but simply logging in doesn’t
generally designate value. Similarly, usage may
simply be dependent on what software a company
has purchased. Most large companies have a CRM
in place, for example. As a result, many people at
that company will perform activities in the CRM,
often daily. That in itself tells you little about how
happy those users are with that CRM.
The Heap Guide to Retention • heap.io
Distinguish yourselves
from your competitors
At Heap, we first started tracking whether people
were running a query with our tool. But we then
found that lots of customers ran sample queries
just to see how the tool worked, giving us skewed
retention metrics. So we decided to get more
specific and see if people were using queries to
answer legitimate questions.
We settled on “saving a report” as the activity that
best indicated this. However, we soon realized
that when we looked at the reports people saved,
many were saving the exact same kinds of reports
they could get in Google Analytics. So we ended
up adding “uses the Group By feature” to our
metric, so that our retention metric is not “saved
a report that uses the Group By feature.” As GA
doesn’t provide a Group By feature, the fact that
users use it indicates that they’re gaining unique
value from Heap.
See below for more information.
20
Two Retention Metrics Retention of users who did View - Any App Page and then Execute Query, where
Date First Seen after 1/1/2017, by Whether User Has Done Sequence, then undefined

At Heap, we found two metrics that correlated strongly with

retention: “Saving Reports” and “Group By.” Retention Start Event

Start Event

Saving Reports View - Any App Page

Return Event
The first, “Saving Reports,” told us that when a Heap user saves Execute Query

a report in their first session, they are 2x more likely to be long- Group By Has done sequence Accept Invite → Save Report

term retained users of Heap.

Filters Date First Seen after 01/01/2017

This is important, we hypothesize, because saving a report Add Filter

is likely to indicate that a user is actually using Heap to run Compare Segments Add Segment

queries and produce an analysis that they found important Date Range Dec 31, 2016 - Apr 27, 2018 by Week

enough to save.
First Time

To find this, we tested all possible activation events and

correlated them with long-term retention. Because Heap enables
retroactive analysis, we could easily run these queries and
extend them into the past.

Importantly, these queries didn’t tell us whether it was the act

of saving a report that caused users to be retained, or whether
the fact that they saved reports in their first session indicated
something else about these customers that made them more
likely to keep using the product. The queries did, however, help
focus our research.

The Heap Guide to Retention • heap.io 21

Group By Did Suggested Reports improve query retention?

The second metric, “Group by” told us that when a Heap user
uses the “Group By” feature when running a query, their Retention Start Event

engagement rates tend to be 2x those of users who don’t use Start Event

Group By. View - Any App Page

Return Event

As noted above, we chose “Group By” because it was a feature View - Any App Page

that other tools, including Google Analytics, do not offer. So Group By Add Group By

repeatedly using “Group By,” we hypothesized, indicates that a Filters Add Filter

user is using Heap for something no other solution can provide.

Compare Users Has Run a Query
vs.

As with “Saving Reports,” this query didn’t tell us whether it was Has Run a Query

the act of using Group By that caused users to be retained (that Date Range Past Year by Month

by using Group By they realized how powerful Heap was, for First Time

example), or whether the fact that they used Group By indicated After 10 months

something else about these customers that made them more Has Run a Query
w/ GroupBy
51.86% (-10.15%)

likely to keep using the product (that they had already found Has Run a Query 21.85% (-22.84%)

value in Heap, for example, and were experimenting with Group

By to see what it could do).

The Heap Guide to Retention • heap.io 22

 Our hypothesis was that by putting plain-English
questions in our product and auto-filling queries to
match, we could give customers more direction about
the questions they could ask with Heap, and therefore
provide more value. The ultimate goal was to drive
increased usage.
SUGGESTED REPORTS
We called this new feature “Suggested Reports.”

A Hypothesis Story:
Suggested Reports
Earlier this year, we hypothesized that our customers would
benefit from having more guidance in the product. From
customer interviews, we learned that our UI didn’t give
customers clear enough information about what questions
they could answer and why those questions mattered. We also
learned that customers found our query builder difficult to use,
since it had so many options.

To solve these problems, we convened a team and held a series

of meetings. After running through various options, we settled
on a hypothesis we were all willing to go with.

The Heap Guide to Retention • heap.io 23

23
 The data showed that 13% of the users who saw the Reports
page started creating a Suggested Report, and only 5% of the
users who saw the Suggested Reports panel opened it.

SUGGESTED REPORTS
What We Learned
After rolling out Suggested Reports, we gave it some time and
started measuring. Or, rather, we looked at the data. (Since
Heap autocaptures all customer data, we never needed to “start
measuring”; Heap was already measuring anything, without
needing“track(‘event’)” tags.)
These numbers suggested a few options. Maybe the issue was
that Suggested Reports needed to be more discoverable. Maybe
we needed to do more work to show users why Suggested
Reports would be a good option for them. Or perhaps users were
aware of what Suggested Reports could do, but simply found it
not useful.
Clearly we needed to dive deeper.

What we found: Only 10% of users who could discover

Suggested Reports ended up using it. This number was far
lower than we had hoped, especially since our hypothesis was
that Suggested Reports would change the way many customers
used Heap.

To learn more, we broke that number down. It turns out that

there were two ways our product introduced Suggested Reports 4.70% 32.83%

to customers. The first was on the “Reports” page, which

Click - Open Suggested Click - Choose
View Analytics Page
presented the option directly. The second was on “Analyze” view, Reports Panel Suggested Report

which offered Suggested Reports via slide-in panel.

The Heap Guide to Retention • heap.io 24

24
SUGGESTED REPORTS

Impact on Retention Did Suggested Reports improve query retention? Retention Retention Analysis

Start Event

Executive Query

Return Event

One way to start answering these questions about Suggested Executive Query

Group By Users who have done Click Analyze (All)

Reports—was the issue that Suggested Reports wasn’t Filters Add Filter

discoverable enough, or was it simply not a useful feature?—was Compare Users

Date Range
Add Comparison

Past 14 Days grouped by Week

to measure the feature’s impact on retention. Our theory was First Time

that if Suggested Reports turned out to have a positive impact

Run Query

on retention, then customers who discovered Suggested Reports

After 2 weeks

were in fact finding the feature valuable. If so, our energy Has done Click Analyze (All) 56.54% 3.81%
Has not done Click Analyze (All) 41.47% 2.53%

might be better directed towards making Suggested Reports

more discoverable.
To learn more, we dug into the Heap data. What did we
find? Suggested Reports had an undeniably positive impact
on retention.
Put differently, if “run query at least once a week” was the
metric we used to determine whether a user was finding value
in Heap, Suggested Reports was undeniably correlated with
increased usage.

First, we found that that users who had successfully run a Heap While this information doesn’t prove causation (only correlation),
query with Suggested Reports had a 50% higher weekly query as a first step it was quite encouraging.
retention rate at 10 weeks vs users who hadn’t.

The Heap Guide to Retention • heap.io 25

25
 Did Suggested Reports increase Grouped Query Retention? Retention Retention Analysis

Start Event

View - Any App Page

Return Event

Run Query w/Group By

Group By Users who have done Click Analyze (All)

SUGGESTED REPORTS Filters Add Filter

Compare Users Add Comparison

Date Range Past 21 Days grouped by Week

First Time

Run Query

Group By After 3 weeks

Has done Click Analyze (All) 31.41% 2.91%

Has not done Click Analyze (All) 14.51% 2.18%

Digging deeper into the data gave us even more information.

A bit of background: we already knew that the activation metric

that signaled good retention for Heap was “running a query.”
That is, we’ve learned that users who run a query are most likely
to be those who find value in the product and continue to use it.
But, as we described above (page 20), to see if a user was
getting value from Heap that they couldn’t get anywhere else,
we started focusing in on customers who used the “Group By”
feature in our product, since “Group By” is a feature unique to
Heap. Our hypothesis was that if a customer is using “Group
By,” they’re getting undeniable value from our product. And, as it
turns out (page 18), users who use “Group By” are far more likely
to continue to use Heap.
Given this, we decided to look to see if use of Suggested
Reports was positively correlated with using Group By.
The results are at right. The data shows that people who used
Suggested Reports were WAY more likely to also use Group By.
Almost 50% more likely than people who didn’t use Suggested
Reports! This was great news for us.
(Note: as before, this doesn’t necessarily prove causation; this
information in itself doesn’t show that using Suggested Reports
caused users to also use Group By, or to find more value in
Heap. But it does show an association, which is useful, and
gives us directions for further investigation.)

The Heap Guide to Retention • heap.io 26

26
SUGGESTED REPORTS
Final Learnings
Whenever we launch a new feature, we like to examine the
hypotheses we had at the beginning and see how they turned
out. We also like to look at the data and see what next steps
it suggests.
Hypothesis:
Our hypothesis was the Suggested Reports would make it easier
for users to answer questions in Heap, and as a result would
increase retention.
What we found:
—— The good: It does seem that users who actually used
Suggested Reports found value in it. Compared to users
who did not use Suggested Reports, users who did use
Suggested Reports were more likely to run queries, use
Group By, and engage in the activation features that tend
to indicate retention in Heap.
The Heap Guide to Retention • heap.io
—— The bad: Of customers who use Heap and were exposed
to the Suggested Reports feature, only a very small
percentage—10%—actually used the feature.
Next Steps:
Given that Suggested Reports has a positive correlation with
retention, it seems worth continuing to build it out. Before doing
that, however, we clearly need to make Suggested Reports
more discoverable. There are many ways we can do that: we can
present it more prominently in the product; we can add guides
to the product that direct users to the feature; we can market the
feature more aggressively; we can direct our Account Managers
to help let customers know what Suggested Reports is and what
it can do for them.
On to develop more experiments!
27
27
 CLOSING
04

Closing We hope this has been helpful to you. If you have

questions or thoughts about any of the material in
this e-book, or simply want to know more about how
Of all the ways to measure a customer’s interaction with
Heap’s approach to product management can be
your product, retention is arguably the one that tells you
useful for you, we encourage you to get in touch at
the most about the level and kinds of value customers
support@heap.io.
get from your product. In this short book, we’ve tried to
offer some methods for accurately measuring retention,
We thank you very much, and look forward to working
figuring out which stages of retention most deserve
with you to innovate the future of product.
your attention, and then implementing hypotheses to
test various strategies for increasing the frequency and
duration of customer engagement.

At Heap, we believe that the key to improving retention

is to develop a hypothesis-driven, experimental,
and iterative approach to product development. Our At Heap, we believe that the key to
product framework is focused on helping PMs probe for improving retention is to develop
opportunities to develop their product, then to capitalize
a hypothesis-driven, experimental,
on those opportunities, knowing that an ongoing series
of small wins can add up to or pave the path towards
and iterative approach to
major innovation. product development.

The Heap Guide to Retention • heap.io 28

w w w. h e a p. i o