Good UI Data Story Issue 6

GoodUI DATASTORIES - ISSUE #6 - OCTOBER 2014
GoodUI DATASTORIES - ISSUE #6
Closing The Checkout Gap

When Findsomeone.co.nz, a dating site for New Zealanders, asked us to raise revenue from their Gold
Membership Upgrade page, we didn’t intend to change the existing 2-page process for selecting one of 3
plans and then paying. Previous studies have shown that building a multi-page process rather than a
1-page form neither hurts nor raises conversions. However, a multi-step process can be used for gradual
engagement, and gradual engagement is generally a good thing. However, over the course of the test we
realized gradual engagement in this case was complicating matters. Would turning the choice of 3 options
into one primary CTA make any difference? Which of the three plans should we recommend and set as
default? Would creating a concise single page checkout work better? Here is what we found ...
Overview
Three tests were run across 3 months with the Control and 7 Variations
A H
+8%
Revenue
Best Variation
Measuring Generated $6.54 per visitor
Revenue
Control
Generated $6.08 per visitor
Key Causes Of Effect

idea
24 Selling Benefits
The headline was rewritten and the three benefits underneath were shown more clearly.
idea
51 Price Illusions
Removed dollar signs.
Possible Key
Difference
Exposing Fields
idea
20
Exposed the checkout fields on the first page instead of creating a separate checkout page.
Hence two pages were merged into a single one.
idea
35 Urgency
Button label states the goal and adds a sense of urgency (“Become a gold member now”).
idea
18 Benefit Button
Button was named as “Become a Gold Member” to carry a benefit.
Easier Plan Switching

The ability to switch plans was added.
Shifted Alternative Choices

Moved all secondary payment options to the footer and visually simplified it.
Changed Plan Label

Changed plan label for 1 month to “A Month Or So” with annotation (“no term”).
Default Plan Setting

The default plan was set to the 1 Month option.
Updated Amount Descriptions

Framed the 3 and 6 month plans as “paying for themselves” and the 1 month plan promoting the
3 month plan (”Don’t want to rush? Save at least $30 with the 3-month plan”) .
Test Snapshot
Test 1 Test 2 Test 3
A
The control was
carried over across B
three tests
C
D
Variations
Our best performer

The Control Insignificant Variations (p-Value higher than 0.2)
Significant Winners (p-Value less than 0.05) Almost Significant Winners (p-Value less than 0.05)
Significant Losers (p-Value between 0.2 and 0.05) Almost Significant Losers (p-Value between 0.2 and 0.05)
A The Control
This was the original two page purchase page. The first page asked the user to chose a plan type, followed
by a second checkout page. Some of the positive aspects on these two pages included:
P1: Plan Selection
idea
49
Curiosity
idea
24
Selling
Benefits
idea
22
Gradual
Engagement
P2: Checkout
idea
50
Reassurances
Existing Elements Which We Think Worked Well:

idea
Read more on
these ideas at 24 Selling Benefits
GoodUI.org
The site shows benefits of the membership, as well as saving benefits of higher end plans.
idea
22 Gradual Engagement
Gradual engagement is used with the first page asking to make a more subtle choice.
idea
50 Reassurances
Site is reassuring security on the checkout page with the headline.
idea
49 Curiosity
The headline suggests a hint of curiosity.
B The B Variation & Its Effects
Variation B was a small change in the direction of keeping more focus. The fourth purchase option was
removed to see if it would drive more attention to the three subscription options.
Results are grey if they
aren’t conclusive (p-Value is
higher than 0.20). A -1%
Compared To A -1% change also makes this very

similar to the control.
Revenue
P1: Plan Selection
The Control
idea
16
Keeping
Focus
P2: Checkout Same As Control
Differences Which We Think Have Contributed To The Effect:

idea
16 Keeping Focus
The footer with an additional purchase option was removed.
C The C Variation & Its Effects
Variation C was going to test if a single button plan recommendation would work better than a UI pattern
showing three choices more explicitly.
Compared To A -15%
Revenue
P1: Plan Selection
idea
7
Recommending
idea
18
The Control
Benefit Button
idea
16
Keeping
Focus

idea
7 Recommending
The UI pattern reduces perceived choice by making it look like there is only one button.
idea
18 Benefit Button
idea
16 Keeping Focus
D The D Variation & Its Effects
Variation D variation was very similar to C and existed to check if recommending the first 1-Month plan
would make a difference.
Compared To A -32%
Revenue
P1: Plan Selection
idea
7
Recommending
Default Set
To 1Month
idea
18
Benefit Button The Control
idea
16
Keeping
Focus

idea
7 Recommending
idea
18 Benefit Button
idea
16 Keeping Focus

E The E Variation & Its Effects
Variation E variation was very similar to D and existed to check if recommending the 3-Month plan would
make a difference.
Compared To A -17%
Revenue
P1: Plan Selection
idea
7
Recommending
Default Set
To 3Months
idea
18
Benefit Button The Control
idea
16
Keeping
Focus

idea
7 Recommending
idea
18 Benefit Button
idea
16 Keeping Focus

F The F Variation & Its Effects
Variation F kept the Plan Selection page the same as E, and modified the Checkout Page thinking that
perhaps customers wanted an easier way to change plans on the second step.
Results are light red if
they are somewhat
Compared To A -11% conclusive (p-Value is

between 0.2 and 0.05).
Revenue
P1: Plan Selection Same As E
idea
7
Recommending
The Control
Default Set
To 3Months
idea
18
Benefit Button
idea
16
Keeping
Focus
P2: Checkout
idea
24
Selling
Benefits
Easier Plan
Switching The Control

idea
7 Recommending
idea
18 Benefit Button
idea
16 Keeping Focus
idea
Three benefits were repeated at the top of the page.

The ability to switch plans was added on the Checkout Page.

G The G Variation & Its Effects
Variation G was very similar to D and E except it tried to increase the visibility of the three plans on the
first plan’s selection page.
Compared To A -17%
Revenue
P1: Plan Selection
idea
7
Recommending The Control
More Visible
Plans
idea
18
Benefit Button
idea
16
Keeping
Focus

idea
7 Recommending
idea
18 Benefit Button
idea
16 Keeping Focus
More Visible Plans

The three plan options were more visible
H
Be
The H Variation & Its Effects
st
Pe
rfo
rm
er
The main idea behind variation H was to take both pages and merge them into a single one while at the
same time allowing them to make an informed decision about each of the plans.
Results are green if
they are conclusive
Compared To A +8% (p-Value is higher than
0.05).
Revenue
P1: Plan Selection

Changed
idea
Plan Label
24
Selling Default Plan
Benefits Setting
idea
51
Price Illusions
Updated
Amount
Easier Plan Descriptions
Switching
idea
20 idea
Exposing 35
Fields
Urgency
idea
18 The Control
Benefit Button
Shifted
Alternative
Choices
Three Months Selected State

idea
The headline was rewritten and the three benefits underneath were shown more clearly.
idea
51 Price Illusions
Removed dollar signs.
Possible Key
Difference
Exposing Fields
idea
20
Exposed the checkout fields on the first page instead of creating a separate checkout page.
Hence two pages were merged into a single one.
idea
35 Urgency
Button label states the goal and adds a sense of urgency (“Become a gold member now”).
idea
18 Benefit Button

The ability to switch plans was added.
Shifted Alternative Choices

Moved all secondary payment options to the footer and visually simplified it.
Changed Plan Label

Changed plan label for 1 month to “A Month Or So” with annotation (“no term”).

Updated Amount Descriptions

Framed the 3 and 6 month plans as “paying for themselves” and the 1 month plan promoting the
3 month plan (”Don’t want to rush? Save at least $30 with the 3-month plan”) .
Test 1 of 3 & Findings
Preparation
Prior to running this test, we ran 4 short pre-tests (1-2 days long) to get a very rough idea of daily traffic
(e.g., is it dozens or hundreds?), daily conversions (e.g., it is 0.5% or 10%), and see if goal tracking with a
particular CSS query was technically working.
Hypothesis
Going into this test, we knew the traffic would be low and planned just 1 variation. The client also
suggested removing the “message pack” footer, so we added that as an additional variation. We started
with the idea that we could reduce the 3 buttons of choices to just 1 primary CTA. The other choices would
be secondary “options, which we would turn into radio buttons. This would allow our preferred option to be
pre-selected. We learned from the pre-tests that 1 month was the main choice with 90%+ of people choos-
ing that option. However, our assumption was that plans of longer duration held more value for the busi-
ness and should be the focus of our efforts. For example, we assumed monthly subscribers would quit
after 1-2 months. However, we did not have an opportunity verify this assumption. By making the 3-month
plan the default option, we hoped to pull some people from the 1 month plan to the longer 3 month plan.
Test Setup
Visitor Types Tool Used Page Scope

100% of All Traffic VWO.com Two Pages
Test Duration Design & Build Time # of Variations

11 Days (from Jun 2014) 1 week Control + Two
Primary Metric: Revenue Test Type

Calculated manually by tracking purchase page visits A/B Test
Results of Primary Metric: Revenue

Tracked with page visits to a goal page.
A Generated $5.27 per visitor p-Value Improvement
B Generated $5.23 per visitor 0.9 -1% (-19% to -21%)
C Generated $4.48 per visitor 0.04 -15% (-31% to -5%)

Data is shown at a 95% confidence interval. Revenue margin of error is approx. $0.5
Results of Secondary Metric: Purchased 1 Month Plan (Visits)

A Converted at 15% (14% to 17%) with 271 out of 1765 visits p-Value Improvement
B Converted at 15% (13% to 17%) with 271 out of 1772 visits 1 -0.4% (-18% to 17%)
C Converted at 12% (10% to 14%) with 215 out of 1792 visits 0.007 -22% (-38% to -4.9%)
Data is shown at a 95% confidence interval and adjusted for multiple variations using the Bonferroni correction.

A Converted at 2.1% (1.4% to 3%) with 37 out of 1765 visits p-Value Improvement
B Converted at 2.1% (1.4% to 3%) with 37 out of 1772 isits 1 -0.4% (-50% to 49%)
C Converted at 3.3% (2.5% to 4.4%) with 59 out of 1792 visits 0.05 +57% (-2.3% to 109%)

A Converted at 0.7% (0.4% to 1.4%) with 13 out of 1765 visits p-Value Improvement
B Converted at 0.6% (0.3% to 1.2%) with 11 out of 1772 isits 0.86 -16% (-91% to 64%)
C Converted at 0.3% (0.1% to 0.8%) with 6 out of 1792 visits 0.17 -55% (-116% to -24%)
FINDING Footer Removal With 4th Purchase Choice Didn’t Have Any Effect
Removing the footer did not produce any notable effect on the revenue. Both variations A and B had a very
similar revenue per visitor. We did however still recommend removing the footer as it seemed to generate
more clicks to the three plans which was an indication of more people starting off in the sales funnel.
FINDING Pushing 3 Month Plan Dropped Overall Sales

Our change in C to make it seem as if there was just one plan (with 3 Month as the default) was a mixed
success. The default choice did indeed shoot up visitors the 3-month and even 6-month payment pages,
but it reduced interest in the 1-month plan, which hurt overall revenue by -15%. Hence we were pushing a
longer term plan which customers possibly did not inherently feel comfortable with.
FINDING People Were Plan Switching Across Two Pages

We also realized the visits to payment pages were inflated by double-counting, since the choices were not
mutually exclusive (only the final purchase was). We believed people were making a choice, then going
back to Page 1, and making another choice. We realized people were probably trying to compare plans or
finding something on Page 2 that didn’t match their expectations. For example, we noticed on Page 1 the
monthly price was prominent, but on Page 2 it was the larger, total amount due that was prominent.
Perhaps there was reverse anchoring at work, priming people for a smaller amount then hitting them with
a larger amount.
Hypothesis
We realized it was not going to be just a matter of pushing people onto the higher plans. We wanted to
better understand how people make their choice between the three plans and what shift in behavior would
produce the right balance between the plans to lift overall revenue. Since the 3-month and 6-month plans
cost less per month, we knew any successful outcome meant keeping or lifting the dominant 1-month
purchases.
For this experiment, we clarified the prices and savings on Page 1, to create a smoother transition into
Page 2. We did this on all variations. Next, we wanted to know if simply changing the default would make
a difference. So, we set up a “first is default” and a separate “middle is default” variation.
We also surmised that the radio button is too subtle, that people didn’t pay attention to it, continued to the
default, and realized it’s not for them. So, in variation G we tried a compromise visual treatment, similar to
the Control in that it presented 3 distinct options but still with a preselected default, more concise text,
and still as a secondary choice.
In variation F, we also wanted to address the idea that people were going back and changing their plan
choice. We set up a variation identical to our “middle default” option on the first page, but added a way to
switch plans directly on the second page, so there would be no more going back.
Test Setup


15 Days (from Jun 2014) 2 weeks Control + Four

Results of Primary Metric: Revenue

D Generated $4.39 per visitor 0.001 -32% (-48% to -13%)
E Generated $5.34 per visitor 0.02 -17% (-35% to -4%)
F Generated $5.73 per visitor 0.13 -11% (-30% to +11%)
G Generated $5.38 per visitor 0.03 -17% (-35% to +5%)


D Converted at 8.5% (7% to 10%) with 93 out of 1096 visits 0.0001 -57% (-71% to -42%)
E Converted at 14% (12% to 16%) with 156 out of 1121 visits 0.0003 -30% (-45% to -14%)
F Converted at 16% (14% to 18%) with 184 out of 1138 visits 0.028 -18% (-34% to -1.9%)
G Converted at 15% (13% to 18%) with 174 out of 1123 visits 0.009 -22% (-37% to -5.3%)

D Converted at 7.2% (5.8% to 8.9%) with 79 out of 1096 visits 0.0001 +241% (145% to 301%)
E Converted at 4.3% (3.2% to 5.6%) with 48 out of 1121 visits 0.004 +103% (29% to 160%)
F Converted at 3.4% (2.5% to 4.7%) with 39 out of 1138 visits 0.059 +62% (-4% to 118%)
G Converted at 2.7% (1.9% to 3.8%) with 30 out of 1123 visits 0.39 +26% (-33% to 82%)

D Converted at 0.3% (0.1% to 0.8%) with 3 out of 1096 visits 0.92 -0.64% (-125% to 124%)
E Converted at 0.5% (0.2% to 1.2%) with 6 out of 1121 visits 0.36 +94% (-83% to 196%)
F Converted at 0.2% (0% to 0.7%) with 2 out of 1138 visits 0.52 -36% (-140% to 92%)
G Converted at 0.4% (0.1% to 1.0%) with 4 out of 1123 visits 0.78 +29% (-112% to 145%)
FINDING Still No Revenue Lift

All variations without exception lifted 3-month plans and all of them reduced 1-month plans. As a result,
revenue was still negative across the board.
FINDING Default Plan Choices Did Make A Difference

One odd thing we noticed was that making the 1 month plan default resulted in the greatest drop in 1
month plans and the greatest lift in 3 month plans. On variation D with the 1 Month plan as the default
there was a -57% drop in 1 Month Plan Sales and a +241% increase in 3 Month plan sales.
FINDING Two Page Funnel Was Hindering

At this point, we realized it was more than a matter of changing defaults and bolding prices. We were not
giving the user what they needed to make their decision and were hitting a ceiling. It also became clear
that the 2-page funnel was hindering our efforts. At this point, we collected more metrics on the site,
specifically to see if people were actually going back and changing their minds. We found that about 30%
of people changed their initial plan choice, and this behavior was likely complicating matters for us.
So, we decided to finally merge those pages into one and then keep improving.
Hypothesis
For Variation H, we felt just combining the two pages into one would be an improvement by removing the
chance for conflicting or inconsitent information. We decided to create just one strong headline, capturing
the idea that most people would be primarily interested in sending the first message. This was inline with
the knowledge that most people are interested in purchasing the 1 Month subscription. We also decided to
show fewer and more concise benefits.
At the same time we wanted the customer to make an informed choice. For instance, we decided to soften
the name to “A month or so” and label it with “No term” – making it overall more appealing. We also
decided to use the higher Monthly price with larger font size as a subtle anchor to suggest other plans had
more value and add explicit messages to that effect. Overall we wanted to create a “here are the facts, you
make your own choice” impression.
Test Setup


42 Days (from Aug 2014) 1 week Control + One

Results of Primary Metric: Revenue (Clicks)

H Generated $4.54 per visitor 0.003 +8% (-2% to 18%)


H Converted at 17% (16% to 18%) with 1119 out of 6438 visits 0.005 +12% (4% to 20%)

H Converted at 5.5% (16% to 18%) with 355 out of 6438 visits 0.17 -9.3% (-23% to 4%)

H Converted at 1.5% (1.2% to 1.8%) with 97 out of 6438 visits 0.4 +13% (-18% to 44%)
FINDING Single Page Checkout Worked Best

The original control page relied on the concept of gradual engagement to get a visitor to make a choice on
the first page and to then uphold that choice by paying on the second page. In reality, the sales messaging
was divided, so that, for example, on the second page the % saving was missing and the total price
replaced the monthly price that was central on the first page. Furthermore, the 2-page funnel constrained
visitors’ ability to compare the options. Once they made their first choice, they could no longer see the
other options, which may have created doubt. And certainly, going back to the previous page would break
their momentum.
Although Luke Wroblewski’s well-known experiments have shown that single page forms perform similarly
to multi-page forms, that may not hold for every kind of multi-step process. In some cases, it may be
better to split a complex process into several steps. In this case, consolidating the choices, sales copy, and
payment on one page was likely more effective.
FINDING Helping The Customer Make a Decision

A lot of our initial ideas were focused on removing friction, convincing the visitor, or even pushing the
visitor into a choice. None of these strategies were attuned to the user’s decision-making process. So,
ultimately these tactics didn’t work or lead to unexpected results. In striking example, making the 1-month
plan the default in variation D resulted in a 57% drop in visits to its payment page and a 573% increase in
visits to the 3 month payment page.
We believe what worked was a balanced presentation of the choices and their advantages and disadvan-
tages, so the user could make the choice that is right for them. For example, in variation H the higher price
for the 1 month plan was paired with the label “No term” – disadvantage and advantage. Instead of pushing
one choice, we cross-promoted the choices. For example, the 1 month plan says “Don’t want to rush? Save
at least $30 with the 3-month plan”. This states a potential disadvantage (rushing) and suggests another
plan as a better option. We also renamed this plan from “Month” to “A Month Or So”, which may have
provided a time context that more people could related to.
Another important change we made was to interpret the prices for our visitors. Instead of just stating a
price and a saving %, we tried to explain what it would mean for them. For example, the 3-month and
6-month plans say “pays for itself in X months”, putting that price clearer context. Likewise, the 6-month
plan not only states a price but suggests who this plan might best suit: “take your time meeting people”.
Overall, we feel this variation better engaged the visitors’ thought process and left less doubt as a result.
FINDING Benefit Headline and Fewer Benefits

We believe this made our headline was more concrete than the original “Ready to meet someone
special?” First, we have interpreted the benefit for the user. After all, who wants to “send unlimited mes-
sages”? The real benefit is being able to make first contact. We put that right in our headline, connecting a
real goal to the action: “Make A Great Impression, Send the First message”. We’ve then taken the list of
benefits and distilled them to just 3 most immediate benefits, removing technical and abstract benefits
like “extra privacy options” and “organize singles events”.
The Recommendation
Since variation H improved the overall revenue by +8% conclusively with a p-Value of
H 0.03, we recommended to take the variation an implement it on the live site.
Looking Back & Key Learnings

Measuring Non-Exclusive Metrics
After the first test, we realized the non-mutually exclusive click and page visit metrics were showing a
double-counting as some users were changing their minds (moving between the two pages). We assumed
visitors would make their choice and then proceed to payment or drop out. We found the non-exclusive
metrics provided invaluable insight, but we also realized that double counting can complicate analysis. In
the future, we may add a separate metric, using custom Javascript and cookies, to track the user’s first and
even second choice among non-exclusive choices. That could shed even more light on changes in choices
that result from our variations.
Flattening Two Page Funnels

We also realized that in this case we should have from the start created a 1-page layout and built all future
variations on that basis. A multi-page layout creates non-exclusive metrics, which are harder to analyze. It
divides our efforts over two pages, causing duplicated or inconsistent messaging on the different pages.
There was overlapping pricing information shown on two separate pages in various ways. It would have
been easier to work with a single page while minimizing the chance for one page to confuse users by what
was shown on the next one.
Additional Variation Ideas

Going forward, if we were to run more tests on the single page H variation here are some ideas which
could be possible candidates for exploration:
I Fewer Form Fields - The credit card field is one which could be detected automatically.
J Curiosity - Reminding about the profiles which were looked at by the user.
K Social Proof - Hinting at who else signed up today from the area the user searched for.
If you’d like to see a particular test, have an idea or comments about this issue, please send your emails to jakub@linowski.ca.
Thank you again for supporting the project! We hope you enjoyed the issue and learned from our insights. :)
Also a big thanks to Visual Website Optimizer, www.VWO.com, for supporting us by providing Datastories with an awesome a/b testing tool.
LINOWSKI
INTERACTION DESIGN
GoodUI DATASTORIES - ISSUE #6 - OCTOBER 2014

Appendix: Screenshots
A - The Control Page 1
A - The Control Page 2
H Variation -The Winner

Good UI Data Story Issue 6

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Good UI Data Story Issue 6

Uploaded by

Copyright:

Available Formats

GoodUI DATASTORIES - ISSUE #6 - OCTOBER 2014

GoodUI DATASTORIES - ISSUE #6

Closing The Checkout Gap

Key Causes Of Eﬀect

Easier Plan Switching

Shifted Alternative Choices

Changed Plan Label

Default Plan Setting

Updated Amount Descriptions

Our best performer

P1: Plan Selection

Existing Elements Which We Think Worked Well:

Compared To A -1% change also makes this very

P1: Plan Selection

P2: Checkout Same As Control

Differences Which We Think Have Contributed To The Effect:

P1: Plan Selection

P2: Checkout Same As Control

Differences Which We Think Have Contributed To The Effect:

P1: Plan Selection

P2: Checkout Same As Control

Differences Which We Think Have Contributed To The Effect:

Default Plan Setting

P1: Plan Selection

P2: Checkout Same As Control

Differences Which We Think Have Contributed To The Effect:

Default Plan Setting

Compared To A -11% conclusive (p-Value is

P1: Plan Selection Same As E

Differences Which We Think Have Contributed To The Effect:

Easier Plan Switching

Default Plan Setting

P1: Plan Selection

P2: Checkout Same As Control

Differences Which We Think Have Contributed To The Effect:

More Visible Plans

P1: Plan Selection

Three Months Selected State

Differences Which We Think Have Contributed To The Effect:

Easier Plan Switching

Shifted Alternative Choices

Changed Plan Label

Default Plan Setting

Updated Amount Descriptions

Visitor Types Tool Used Page Scope

Test Duration Design & Build Time # of Variations

Primary Metric: Revenue Test Type

Results of Primary Metric: Revenue

A Generated $5.27 per visitor p-Value Improvement

B Generated $5.23 per visitor 0.9 -1% (-19% to -21%)

C Generated $4.48 per visitor 0.04 -15% (-31% to -5%)

Results of Secondary Metric: Purchased 1 Month Plan (Visits)

Results of Secondary Metric: Purchased 3 Month Plan (Visits)

Results of Secondary Metric: Purchased 6 Month Plan (Visits)

FINDING Pushing 3 Month Plan Dropped Overall Sales

FINDING People Were Plan Switching Across Two Pages

Visitor Types Tool Used Page Scope

Test Duration Design & Build Time # of Variations

Primary Metric: Revenue Test Type

Results of Primary Metric: Revenue

A Generated $6.49 per visitor p-Value Improvement

D Generated $4.39 per visitor 0.001 -32% (-48% to -13%)

E Generated $5.34 per visitor 0.02 -17% (-35% to -4%)

F Generated $5.73 per visitor 0.13 -11% (-30% to +11%)