GoodUI DATASTORIES - ISSUE #7 - NOVEMBER 2014

www.goodui.org/datastories
GoodUI DATASTORIES - ISSUE #7

One Thing Leads To Another

When optimizing single pages in isolation, you can miss the full story. When Findsomeone.co.nz, a
dating site for New Zealanders, asked us increase Trial Signups on their home page, we wondered
whether we should quickly pitch the benefits to signup on the home page or first engage visitors with
search. If we found that searching lead to more sign-ups, then perhaps pushing search behaviour would
make sense. To answer this, we set the test scope to two pages: the home page and search results. Here is
what we found after two months of testing ...

Overview
One test was run across 2 months with the Control and 4 Variations

PAGE1: Homepage
A E

Unchanged
Same As Control

Primary Metric
SIGNUPS

PAGE2: Search Results

+21%
Signups

Interaction: Once someone reaches the bottom of the

page, the footer sticks or floats as users scroll.

Best Variation
Converted at 1.8% with 464 out of 25,327 visits

Control
Converted at 1.5% with 442 out of 29,241 visits

Changes That We Think Have Contributed To The Effect

w w49
.go dui.org/#49 Curiosity
idea

The headline urged to discover, while the search results beyond 20 listings were covered.

w w.24go dui.org/#24 Selling Benefits

idea

Benefits of what a profile offers to a user have been explained close to the signup button.

Visibility
By making the footer stick to the bottom of the window, visibility of the call to action increased.

Key Findings
FINDING Visitors Who Searched Were More Likely To Signup
We tested the effect that search has on sign-up rate. Replacing the entire page with a more
prominent and usable search box (Variation B) increased searches from the homepage by +12%
and decreased signups on the home page by -26%. However, it increased overall signups by
+10% (our primary metric). This strongly links searching to increased sign-ups.

FINDING The Benefit Signup Pitch Was Effective Before And After Search
We tested the effect of placing benefits and call to action at the bottom of the homepage
(Variation C) versus delegated to the bottom of the search results page (Variation E). We were
able to lift overall signups similarly by 20% and 21% using both methods.

How To Read Significance In The Results

Significant Winners (p-Value less than 0.05)

Possible Winners (p-Value between 0.2 and 0.05)

Best data for Insignificant (p-Value higher than 0.2) Best not to use these results
decision making to make any decisions
Possible Losers (p-Value between 0.2 and 0.05)

Significant Losers (p-Value less than 0.05)

A The Control
These were the original two pages. The homepage gave users the choice to search or signup. The search
results page pushed to signup through a series of actions such as “Messaging”. Some of the positive
aspects on these two pages included:

PAGE1: Homepage

Contrast
Social Proof

PAGE2: Search Results

Benefit
Buttons

Repeating
Primary
Action

Existing Elements Which We Think Worked Well

w w.go11dui.org/#1 Contrast
idea

The current buttons have good constrast achieved with color, size and depth.

w w.go5dui.org/#5 Repeating Primary Action

idea

On the search results page, the calls to action are repeated beside each profile.

w w.g18o dui.org/#18 Benefit Buttons

idea

The “Message Him/Her” button speak of a benefit first which leads people to signup.

w w.go4dui.org/#4 Social Proof

idea

The homepage shows how many users have already signed up.
B The B Variation & Its Effects
We believe that search is a kind of gradual engagement. Visitors who search stay on a site longer and
become more familiar with the site, which makes them better qualified to hear the pitch. Variation B
replaced the entire home page with a large background image of a happy couple and overlaid a larger,
more clickable search as the central action.

Hypothesis: If we entice more people to engage with the site by doing a search, it will increase sign-up rate.

PAGE1: Homepage

Curiosity
Central
Layout Full Screen
Photo

Exposing
Options
+10%
Signups
Bigger Click
Areas

+12%
Smart Searches
Defaults

The Control

PAGE2: Search Results Unchanged

Same As Control

Changes That We Think Have Contributed To The Effect

w w.g14o dui.org/#14 Exposing Options

idea

Options such as “Man” or “Woman” were exposed to decrease effort to find / select.

w w.38go dui.org/#38 Bigger Click Areas

idea

Made it easier to select a choice with a larger input field.

w w49
.go dui.org/#49 Curiosity
idea

Headline of “See Who’s Looking” is speaking to curiosity.

w w.28go dui.org/#28 Smart Defaults

idea

Defaulted region to Auckland, and age range to 30-45 due to highest popularity.

Central Layout
The layout was very central and focused to the primary call to action.

Full Screen Photo

The background image changed to a full screen one.

FINDING Visitors Who Searched Were More Likely To Signup

We tested the effect that search has on sign-up rate. Replacing the entire page with a more
prominent and usable search box (Variation B) increased searches from the homepage by +12%
and decreased signups on the home page by -26%. However, it increased overall signups by
+10%. This strongly links searching to increased sign-ups.
C The C Variation & Its Effects
We thought that if we kept search prominent but at the same time made a stronger pitch to sign up on the
home page, then we would cover both bases: some visitors will search right away, while others may need
some benefits to entice them to search. Moreover, return visitors will more readily find the call to action.

Hypothesis: If we place the benefits and a more prominent sign up action on the home page, we'll
increase the likelihood visitors will take that action.

PAGE1: Homepage

+20%
Signups

+0%
Searches
One Column
Layout

Prominance

Selling
Benefits
Positive
Message

The Control

PAGE2: Search Results Unchanged

Same As Control

Changes That We Think Have Contributed To The Effect

w w.go1dui.org/#1 One Column Layout

idea

The Signup action was centralized underneath search for additional visibility.

w w.24go dui.org/#24 Selling Benefits

idea

Benefits of what a profile offers to a user have been explained close to the signup button.
Furthermore, “Have Others Find You” and “Get Listed” also imply a clearer benefit.

Positive Message
Message tried to provide additional confidence by saying “Because You’re Awesome!”

Prominence
The whole signup section began to stand out after it was placed further away from the search
and began taking up more space.

FINDING Making The Signup More Prominent Worked

Variation C increased overall signups by +20% proving that making a clear, separate and promi-
nent benefit pitch has caused the effect.
D

Va
ria
tio
The D Variation & Its Effects

nW
as
St
op
pe
Ead
rl y
We thought that making a time-limited offer might entice visitors to sign up quickly. A 30 minute time-out
offer was created giving visitors the ability to send one free message to a profile .

Hypothesis: If we offer a bonus gift with limited time to claim it, then more visitors will sign up.

PAGE1: Homepage

+13%
Signups

+1%
Gift Searches

Benefit
Urgency

The Control

PAGE2: Search Results Unchanged

Same As Control

Changes That We Think Have Contributed To The Effect

w w.go2dui.org/#2 Gift
idea

Headline frames the offer as a special gift given to the user.

w w35
.go dui.org/#35 Urgency
idea

Urgency has been created with the help of an expiry counter set to 30min.

w w.24go dui.org/#24 Selling Benefits

idea

The benefit of being visible to others with a profile was made clear.

FINDING The Urgent Gift Only Hinted At More Signups

Offering the gift with the timeout upfront did seem to lift sign up conversions by +13%. However,
the significance wasn’t very strong (p-Value of 0.12) even though it was very consistent.

NEW HYPOTHESIS Making The Gift More Authentic Will Increase Signups
Currently, if someone cleared their cookies or opened the website in a new browser, the gift
would appear with a new counter. This might have felt inauthentic. If however the gift was made
more authentic with an expiry tied more firmly to an individual user, perhaps it might work better.

NEW HYPOTHESIS Making The Gift More Personalized Will Increase Signups
Similarly, the current gift felt quite generic. Perhaps another idea to test would be to make the
gifts more personalized. For example: a user searches through some profiles, which then could
be referenced in a unique gift (”send one message to profile x for free - it’s on us.”)
E

Be
The E Variation & Its Effects

st
Pe
rfo
rm
er
We thought that making our pitch on the search page instead of on the home page would target more
qualified visitors. They already engaged the site by searching, so it's the perfect time to show them why
they should get an account.

Hypothesis: If we make the pitch to signup following a search, we will increase signups.

PAGE1: Homepage Unchanged

Same As Control

PAGE2: Search Results

+21%
Signups

+0%
Searches

Curiosity

The Control

Visibility

Selling
Benefits

Interaction: Once someone reaches the bottom of the page, the footer
sticks or floats as users scroll.

Changes That We Think Have Contributed To The Effect

w w49
.go dui.org/#49 Curiosity
idea

The headline urged to discover more profiles, while the results beyond 20 listings were covered.

w w.24go dui.org/#24 Selling Benefits

idea

Benefits of what a profile offers to a user have been explained close to the signup button.

Visibility
By making the footer stick to the bottom of the window, visibility of the call to action increased.

FINDING The Benefit Signup Pitch Was Effective Before And After Search
We tested the effect of placing benefits and call to action at the bottom of the homepage
(Variation C) versus delegated to the bottom of the search results page (Variation E). We were
able to lift overall signups similarly by 20% and 21% using both methods.

NEW HYPOTHESIS Proving Socially That Others Are Looking Will Increase Signups
Perhaps this variation could be made stronger by adding social proof references of those “others
who are looking” with the intent of further raising signups. What if an additional variation showed
the number of people who went on a date due to the service? What if a new variation showed
how many people were online or active in the last few days?
 Test & Results
We ran one test with 4 variations, 3 of the home page and 1 of the search page. All visitors were entered
into the test on the home page. We ran this test over 5 weeks with 4 variations. We then dropped variation
D and kept the test running for another 2 weeks. Even though the dropped variation was showing improve-
ment, we knew we wouldn't have time to run all variations due to time constraints on the project.

Test Setup
Traffic Type Tool Used Page Scope
100% of All Visitors VWO.com Homepage + Search Results
Except IE Users

Test Duration Design & Build Time # of Variations

7 Weeks (from Aug 2014) 3 weeks Control + Four

Primary Metric: Signups Test Type

Tracked with page visits to the first step of the signup process. Split URLs

Test Start Sample Size

Particpants joined the test upon visiting The Homepage. Around 125,000

A B C D E PAGE2

Primary: Sign-Up Rate (On Home and Search Pages)

We measured this using page visits to the first step of the signup process

A 1.5% (440 of 29,241 visits) p-Value Improvement

B 1.7% (423 of 25,462 visits) 0.14 +10% (-4% to 24%)

C 1.8% (458 of 25,284 visits) <0.01 +20% (6% to 35%)

D 1.6% (317 of 18,604 visits) 0.12 +13% (-3% to 28%)

E 1.8% (461 of 25,327 visits) <0.01 +21% (7% to 35%)

Secondary: Search Rate (From Home Page)

We measured page visits to the search page with “from home” URL parameter

A 24% (6141 of 25,327 visits) p-Value Improvement

We used E data as B 27% (6910 of 25,462 visits <0.001 +12% (9% to 15%)
the control due to a
measuring error.
C 24% (6123 of 25,284 visits) 0.94 0% (-3% to 3%)

D 24% (4588 of 18,702 visits) 0.49 +1% (-2% to 4%)

E 24% (6141 of 25,327 visits) 1.00 0% (-3% to 3%)

Test Snapshot Over Time

End of Test
A

B
D was stopped to let
Variations

other variations get to

C
significance quicker.
Our best performer
D Stopped

Test Strategy: Stretching The Scope

Typical A/B tests contain variations with changes limited to a single
page. In this test however, we stretched the scope to two pages. We did
this in order to gain a deeper understanding of how timing and sequenc-
ing of certain calls to action affects their effectiveness.
 The Recommendation

PAGE1: Homepage
C B
For the home page, using what we learned from variations C and B, we recommend merging both varia-
tions into a new one. Borrowing from the effectiveness of C, we’d replace the secondary content below the
fold with clear benefits and a more prominent signup button. At the same time, knowing that searches
lead to more signups, we’d implement the improved search form as it was done in variation B.

PAGE2: Search Results

E
For the search results page, we’d surely implement the best performing variation E. This would allow the
second page to benefit from a +21% signup rate increase.

Looking Back & Process Learnings

Double Checking URL Parameters
Making assumptions about URL parameters is a sure way to mess up your metrics. On this site, we had
multiple ways to sign up that all lead to the same goal page. We used a URL parameter to tell us from
which page the sign up originated. As the test ran, we realized that the URL parameter didn't map perfectly
to the pages we expected and there were also more parameters we weren't aware of. As one result of this,
the search page specific signups have been incorrectly counted. Luckily, in this case it didn't affect our
primary metric that was measuring signups in general. In the future however, it would be good practice to
double check how parameters are passed.

Ensuring Exclusivity of Goal Pages

Failing to control the entry point to your goal page is another sure way to distort your metrics. For instance,
visitors had many ways of getting to the search page, but we wanted to exlusively track visits from our
improved search box on the home page. To allow this, we added a URL parameter "from=home" to the
search page whenever the search originated from home. We could then use that URL parameter to track
just the searches we were interested in. After you set your goal page, rule out any possibility that someone
can bypass the behavior you're testing and still get to that page.

Finding Broken Measurement Patterns

For reasons still unknown to us, we were not able to accurately track the search rate on the Control. Here
is why we strongly believed the 16% absolute search rate was a broken measurement. First of all, it is
something we observed in previous similar tests. Second, we saw that variations C, D, and E had a similar
rate of 24% while these three variations did not change anything related to improving search. Finally, we
also knew based on historical data and previous testing, that the approximate search conversion rate was
around 23-24%. Therefore, we chose to correct the control by using the search data of variation E as the
new control and still be were able to extract insights. As a rule of thumb, the clear pattern to look out for is
when there are multiple variations with a similar lift, while not having any changes to deserve it.

Error Pattern In Absolute Search Rate

Broken
Measurement
A 16%

B 27%
Variations

C 24%

D 24%

E 24%

The Control’s search The normal This searche rate for B makes sense due
rate was too low search rate to the “search push” in the variation.

Additional Variation Ideas

Going forward, if we were to run more tests, we’d focus on the search results page. Here are some varia-
tion ideas which could be possible candidates for exploration:

F Socially Proven Signup - Showing the activeness of other people’s profiles.

G Socially Proven Signup II - Showing how many “Others Like You” have joined today.

H Rapid Signup - Elaborating the button to a few key fields in order to actually create an account.

If you’d like to see a particular test, have an idea or comments about this issue, please send your emails to jakub@linowski.ca.

Thank you. We hope you enjoyed the issue and learned from our insights. :)

Also a big thanks to Visual Website Optimizer, www.VWO.com, for supporting us by providing Datastories with an awesome a/b testing tool.

LINOWSKI
INTERACTION DESIGN

GoodUI DATASTORIES - ISSUE #7 - NOVEMBER 2014