GoodUI Data Stories Issue 2

GoodUI DATASTORIES - ISSUE #2 - JUNE 2014
GoodUI DATASTORIES - ISSUE #2
Third Time’s A Charm

A test is complete when there is no more skepticism left on the table. On this project we were tasked with
increasing Trial Signups for ManageWP.com - a management tool for users with multiple Wordpress
sites. Here it actually took three tests for us and the client to finally become convinced of the slight, but
positive improvement. It’s probably a good thing not to trust the results too soon especially when just
beginning to learn about conversion optimization. It’s very easy to make little mistakes that introduce
noise into the data and skew the results. Trust develops the more tests that you do and as you begin to
collect data in various ways. When at the end of the day though both you and the client become
convinced, well that is well worth it.
The Overview
Three tests were ran across 4 months with the Control and 4 Variations
A
E
Measuring Trial
Signups
+12%
Relative Improvement
Control Best Variation

Converted at 7.2% with 693 out of 9597 visits Converted at 8.1% with 760 out of 9395 visits
Test Process Snapshot

Test 1 Test 2 Test 3
A
E
A The Control
This wasn’t the first time this homepage was going to be put to the test. Managewp.com was already
heavily optimized with numerous tests done in the past. Clear headline. Check. Social proof. Check. Not
one but two visible calls to action with a free mention. Check. Clearly a challenge, just the way we like it.
B The B Variation - No Pain
Variation B has taken a minimal approach and made the one input field central to the page. Its core mes-
saging is conveying the removal of pain associated with managing mulitple Wordpress sites, as well as
suggesting a time benefit.
Benefit Buttons
idea
18
The main button label was changed to “Get Your Time Back”.
GoodUI
Selling Benefits
idea
Ideas 24
We spoke to the benefit of removing management pain and savings of time. A specific time refer-
ence was used to indicate just how much time per site can be saved.
Social Proof
idea
4
We replaced testimonials with ratings and number of downloads from the “Wordpress Community”
Additional
Ideas Informality & Humour
“Based on our mad scientist estimates” writing
Central Form
The form was placed centrally for additional attention.
No Screenshot
The application screenshot was removed in order to direct more attention on the messaging.
C The C Variation - Breeze To Start
Variation C has emphasized that the software trial is within an arm’s reach away as it only takes 30
seconds to get started for free. It also talks about three key product features. The variation also tries to
frame the email grab as a short two stepped approach that’s tied to a benefit.
Benefit Buttons
idea
18
The main button label was changed to “Get Your Time Back”.
GoodUI
Selling Benefits
idea
Ideas 24
The benefit of time savings was emphasized with the key headline. Three key benefits were
displayed that highlighted: No More Passwords, One Click Automation and Security & Support.
Social Proof
idea
4
Additional
Central Form
No Screenshot
Speed Of Signing Up
Speed of signing up was highlighted with a “Got 30 Seconds? Give It a Try Now:” message.
Free Mention
We mentioned the trial is free.
D The D Variation - Zero Risk
Variation D has emphasized low risk, ease of use, and security while also taking the minimalist approach.
These are all strong reassurances to the fear of exposing one’s website to an unknown software.
Benefit Buttons
idea
18
The main button label was changed to “Take a Test Drive”.
GoodUI
Selling Benefits
idea
Ideas 24
The ease of managing Wordpress sites was communicated.
Social Proof
idea
4
Additional
Central Form
No Screenshot
Additional Reassurances
A badge was used to decrease fears of the software. This was done with multiple mentions of
security and 24/7 support.
E The E Variation - Zero Risk II
Variation E builds out on D by additionally speaking about top three product benefits and the fact that the
software is free for up to five sites.
Selling Benefits
idea
GoodUI 24
The ease of managing Wordpress sites was communicated.
Ideas
Social Proof
idea
4
Additional
Central Form
No Screenshot
Additional Reassurances
A badge was used to decrease fears of the software. This was done with multiple mentions of
security and 24/7 support.
Free Mention
We mentioned the trial is free, and informed that it is so for up to 5 sites.
Test 1 of 3
Ok, we can do this, we thought. We set out to do our standard approach with a shotgun strategy. Three
variations were created which represented a key theme that differentiated them from one another. We then
hoped one variation was going to do better and we’d take it from there. The key differences were:
Variation B was going to speak to the removal of management pain as a benefit

Variation C was going to speak to the speed of signing up
Variation D was going to speak to the low risk trial, awesome security and reliable support
The test setup was simple as well. One metric was measuring people initiating the signup process when
clicking on the main call to action. Out of curiosity we also tracked people clicking on secondary pages in
the header (Learn More) and (Plans), in case the new concepts were too simple and they were causing
people to leave in the hunt for additional information.
Test Setup
Visitor Types Tool Used Page Scope

100% of All Traffic Visual Website Optimizer Single Homepage
Test Duration Design & Build Time # of Variations

28 Days (from Jan 2014) 2-3 weeks Control + Three
Primary Metric: Trial Signups Test Type

Tracked mouse clicks on main signup button Split URL Test
Results of Primary Metric: Trial Signups (Clicks)

Tracked with mouse clicks on main signup button
A Converted at 4% (3.2% – 4.9%) with 121 out of 3,025 visits p-Value Improvement
B Converted at 3.3% (2.5% – 4.3%) with 79 out of 2384 visits. 0.42 -17% (-46% – 14%)
C Converted at 3.8% (2.9% – 4.8%) with 91 out of 2406 visits 0.96 -5.4% (-36% – 26%)
D Converted at 5% (4% – 6.1%) with 124 out of 2489 visits 0.19 +25% (-8.9% – 58%)
Shown with a 95% confidence interval.
Results of Secondary Metric: Visits To Other Pages

Tracked with mouse clicks on “Learn More” and “Plans” link in header
A Converted at 13% (11% – 14%) with 381 out of 3025 visits p-Value Improvement
B Converted at 14% (13% – 16%) with 342 out of 2384 visits. 0.15 +14% (-3.8% – 32%)
C Converted at 12% (11% – 14%) with 295 out of 2406 visits 0.97 -2.7% (-20% – 15%)
D Converted at 15% (13% – 17%) with 369 out of 2489 visits 0.044 +18% (0.02% – 35%)
As the tested approached its 28th day and variation D was pressing on hard with B and C dragging behind,
the client hinted at a slight adjustment. As we began losing faith in B and C having a chance to win, we
both decided to stop the test, break off the losers and then restart it.
D Was Promising
Variation D definitely was promising at this point. It’s p-Value of 0.19 was suggesting that there was 81%
chance of some real effect. The data showed that A had 4% of users signing up, while D increased it to 5%
(with a +25% relative lift). We began to think that the reassurance of zero risk and reinforced security was
a contributing factor. Perhaps users wish to know that their data, settings and websites are safe when they
expose it to the trial of this service.
People Were Leaving To Other Pages

The other insight which we gained from the data in this test was that both B and D were indicating an
increase of people clicking on the top header and searching for additional information in either “Learn
More” or “Plans” (B +14%, and D +18%). Both concepts were so light weight that possibly users were not
ready to make a trial decision just yet. Variation C, having a bit more informative content about the product
benefits however, was closer to the control in this respect.
Negative Connotation Is a Bad Frame

The worst performer so far was variation B in respect to signups. We began interpreting this that the
“Management Pain” messaging might have negative connotations and perhaps isn’t the best direction.
Maybe more positive and constructive wording might work better.
Neither us, nor the client were yet fully confident in the above results with the p-Value not being low
enough, and so we began to gear up for the second test.
Test 2 of 3
The second test was setup in order to build further confidence in variation D. We didn’t change anything
from the first test in terms of design, but we did setup additional metrics which we wanted to capture. First
we decided to track Trial Signups in an complementary way from when users would reach a deeper goal
page. We also broke down the single “visits” metric into two more granular metrics corresponding to the
distinct pages they could visit (Learn More, and Plans). Finally, we also introduced a method to filter out
existing users and customer from our target audience - new users. To do this, we had the actual applica-
tion itself create a cookie for each user that logged in which we then excluded from the test. We also
waited a week or so for some of this tagging to take effect.
Test Setup

New Users Visual Website Optimizer Single Homepage

23 Days (from Feb 2014) 2 days Control + One


A Converted at 4.9% (4.3% – 5.5%) with 208 out of 4278 visits p-Value Improvement
D Converted at 5.1% (4.5% – 5.9%) with 187 out of 3643 visits 0.58 +5.6% (-14% – 25%)
Results of Secondary Metric: Trial Signups (Deep)

Tracked with goal page visits to (https://managewp.com/wp-admin/?registered=1)
D Converted at 3.3% (2.8% – 3.9%) with 120 out of 3643 visits 0.15 -16% (-36% – 5.7%)
Results of Secondary Metric: Visits To Learn More Page

Tracked with “Learn More” page visits
D Converted at 11% (10% – 12%) with 412 out of 3643 visits < 0.0001 +76% (56% – 95%)
Results of Secondary Metric: Visits To Plans Page

Tracked with “Plans” page visits
A Converted at 10% (9.1% – 11%) with 427 out of 4278 visits p-Value Improvement
D Converted at 12% (11% – 14%) with 453 out of 3643 visits 0.0005 +25% (11% – 39%)
Less than a month into the test, we stopped it short. Something was still wrong.
Were People Not Really Signing Up?

The Trial Signup (Deep) metric of people actually reaching a further goal page was beginning to show a
-16% drop when comparing D to the control. The client of course didn’t like this, nor did we. To us however
this was a sign of something technical in nature which went wrong. We assumed that for the most part,
the deeper we look at in the funnel (ex. clicks, vs actual goal page visits), there might be a drop in overall
numbers, but proportionally the ratios should more or less remain the same. So if we had +5.6% users
clicking on the main button of D, we should have seen more or less something similar in terms of deeper
signups. The -16% was too large of a difference, as our funnel wasn’t really that deep (a click vs. an actual
valid one field form submission which would go to the next page).
More People Were Leaving To “Learn More” Than To “Plans”

Ok, it did turn out that variation D was somewhat light in explaining the product as +76% people were
clicking on “Learn More”. We also understood that the Learn More page in itself wasn’t the greatest in
pushing people back onto the trial signup process, hence whoever ventured there was possibly a lost
cause in terms of Trial Signups. We knew we had to do some basic explanation on the homepage in order
to keep more people trying out the software.
One more time, we’d have to readjust the test.

Test 3 of 3
In the third test we were going to make sure we’re capturing true signups more accurately. We did this by
changing to a more flexible goal page pattern to capture deeper signups. We also began to capture sign-
ups in four distinct ways: clicks on button, goal page visits, with a hidden INPUT field that was passing
variation values, and form submissions. Hopefully the four ways would help us compare the data and
identify if some were more reliable than others.
We also duplicated variation D into E and made some design adjustments. Overall, we thought that we
could stop people from leaving to the other secondary pages. By informing users more about the product,
we were going to keep them on the page and generate more trial signups - we hypothesized. The variation
E changes included: 1. Addition of three explanatory paragraphs about product benefits 2. Answering of
pricing concerns by showing that the trial is free, and always free for up to five users 3. Changed the
headline by using the softer word “update” instead of “manage” and also adding the word “quickly”. With
this we started the third test.
Test Setup

New Users Visual Website Optimizer Single Homepage

42 Days (from Apr 2014) Less than 1 week Control + Two


D Converted at 6.9% (6.3% – 7.5%) with 658 out of 9576 visits 0.97 -0.98% (-0.43% – 24%)
E Converted at 7.8% (7.2% – 8.4%) with 729 out of 9395 visits 0.056 +12% (-0.43% – 24%)
Results of Secondary Metric: Trial Signups (Deep)

Tracked with goal page visits to (*?registered=1*)
D Converted at 7.2% (6.6% – 7.8%) with 688 out of 9576 visits 0.99 -0.05% (-12% – 11%)
E Converted at 8.1% (7.5% – 8.7%) with 760 out of 9395 visits 0.045 +12% (0.03% – 24%)
Results of Secondary Metric: Trial Signups (Database)

Tracked with hidden INPUT field that was passing data into the database.
D Converted at 6.4% (5.9% – 7%) with 616 out of 9576 visits 0.30 -7% (-19% – 4.7%)
Results of Secondary Metric: Trial Signups (Form Submit)

Tracked with a FORM submit custom conversion goal
D Converted at 6.7% (6.1% – 7.3%) with 638 out of 9576 visits 0.81 -2.8% (-15% – 9%)
Results of Secondary Metric: Visits To Learn More Page

Tracked with mouse clicks on “Learn More” link in header
A Converted at 6.4% (5.9% – 7%) with 619 out of 9597 visits p-Value Improvement
D Converted at 9.8% (9.2% – 11%) with 942 out of 9576 visits < 0.0001 +53% (39% – 66%)
E Converted at 8.5% (7.8% – 9.1%) with 794 out of 9395 visits < 0.0001 +31% (18% – 44%)
Results of Secondary Metric: Visits To Plans Page

Tracked with mouse clicks on “Plans” link in header
A Converted at 15% (14% – 16%) with 1444 out of 9597 visits p-Value Improvement
D Converted at 15% (14% – 15%) with 1402 out of 9576 visits 0.65 -2.7% (-10% – 5%)
E Converted at 13% (12% – 14%) with 1210 out of 9395 visits < 0.0001 -14% (-22% – -6.9%)
As the test approached our targeted 10k visits per variation, we began seeing some interesting results.
Three Out Of Four Metrics Were Corresponding

The Clicks, the Deep Goal Page Visits, and the Database data were all showing a +12% improvement of E
over A. More so, they were showing this with a p-Value of around or less than 0.05 which to us was
indicated that the effect was real. The one metric that was standing out of the crowd (+18% instead of
+12%) was the data captured with Form Submits. This we interpreted as something that could be just
ignored since we had some bad luck with capturing form submissions in the past. Technically, form
submissions sometimes can fire in unpredictable ways - until we’d figure out how to capture them more
accurately, we’d assume that this metric was off. We also had a reason to believe that the Database data
(actual signups being tagged with variation codes) having more weight in reliability and so we went along
with a +12% increase.
Less Secondary Page Visits, More Trials

The data did finally show that in variation E less people visited the “Learn More” page in comparison to the
D variation. By showing additional pricing information, variation E also decreased in visits to the “Plans”
page by -14% in comparison to the control. It turned out that answering some concerns up front can help
keep people on the page and directing them to the main call to action.
The Decision
The third test provided enough confidence to suggest variation E as the winner and recommend
its implementation. The relative increase of Trial Signups by +12% with a p-Value of less than
0.05, combined with multiple data sources was a good indicator of a real improvement.
Looking Back & Key Learnings

Metric Ratios Should Hold Proportionally The Narrower They Are
When you have a funnel through which users or customers progress, and you’re measuring conversion in
multiple ways along this linear process, then the closer the measurements are to each other, the ratios of
the data should be closer to. So, that if on page 1 of 5, you have two metrics capturing the same conver-
sion, they should be in line with each other - otherwise something is faulty. However, if you’re comparing
the conversion of page 1 to page 4 deeper in, the further the ratios of the conversion data can stem apart.
Cookie Segmentation Takes A While

We introduced a cookie segmentation method and the cookie was still too warm. It took a while before the
exclusion of customers and existing users actually took effect. This we noticed by looking at the base
conversion across all three tests. Notice that when comparing the number of Trial Signup Clicks, the
control converted at 4% in Test 1, then at 4.9% in Test 2, and finally 6.9% in Test 3. Surely there are month
to month fluctuations between multiple tests, but our exclusion cookie was being created right before we
started Test 2 and required a customer to login before it would take effect. Hence existing customers were
being slowly but gradually excluded and this raised the baseline. It’s probably better to use a cookie that
has been already established for a longer time rather than eating it straight out of the oven. :)
More Flexible Pattern Matching Is Good

Technically, the new and more flexible pattern matching on the Trial Signups (Deep) metric was showing
cleaner data that was more in line with other metrics. Hence matching for:
*?*registered=1*
might have been better than matching for:
https://managewp.com/wp-admin/?registered=1
Calibration With Multiple Data Sources

It’s a good thing not to trust the results, especially when just beginning to learn about conversion optimi-
zation. There can be noise and garbage coming in that is skewing your data because the tool isn’t used
appropriately. It takes time to build trust in the tool and figure out how to use it for the best results. Until
we feel confident, it’s good to retest and compare the data to multiple sources.
Additional Variation Ideas

Below are a few additional ideas we’d be eager to test if we would run the test again.
F Screenshot or no screenshot
G Gradual Engagement. Would it help to first ask for a URL (to manage) instead of email?
H Highlighting additional or other benefits
I Emphasizing “FREE for up to 5 Sites” even further
If you’d like to see a particular test, have an idea or comments about this issue, please send your emails to jakub@linowski.ca.
Thank you again for supporting the project!
GoodUI DATASTORIES - ISSUE #2 - JUNE 2014

Appendix: Screenshots
A - The Control
E - Zero Risk II

GoodUI Data Stories Issue 2

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

GoodUI Data Stories Issue 2

Uploaded by

Copyright:

Available Formats

GoodUI DATASTORIES - ISSUE #2 - JUNE 2014

GoodUI DATASTORIES - ISSUE #2

Third Time’s A Charm

Control Best Variation

Test Process Snapshot

Variation B was going to speak to the removal of management pain as a benefit

Visitor Types Tool Used Page Scope

Test Duration Design & Build Time # of Variations

Primary Metric: Trial Signups Test Type

Results of Primary Metric: Trial Signups (Clicks)

Results of Secondary Metric: Visits To Other Pages

People Were Leaving To Other Pages

Negative Connotation Is a Bad Frame

Visitor Types Tool Used Page Scope

Test Duration Design & Build Time # of Variations

Primary Metric: Trial Signups Test Type

Results of Primary Metric: Trial Signups (Clicks)

Results of Secondary Metric: Trial Signups (Deep)

Results of Secondary Metric: Visits To Learn More Page

Results of Secondary Metric: Visits To Plans Page

Were People Not Really Signing Up?

More People Were Leaving To “Learn More” Than To “Plans”

One more time, we’d have to readjust the test.

Visitor Types Tool Used Page Scope

Test Duration Design & Build Time # of Variations

Primary Metric: Trial Signups Test Type

Results of Primary Metric: Trial Signups (Clicks)

Results of Secondary Metric: Trial Signups (Deep)

Results of Secondary Metric: Trial Signups (Database)

Results of Secondary Metric: Trial Signups (Form Submit)

Results of Secondary Metric: Visits To Learn More Page

Results of Secondary Metric: Visits To Plans Page

Three Out Of Four Metrics Were Corresponding

Less Secondary Page Visits, More Trials

Looking Back & Key Learnings

Cookie Segmentation Takes A While

More Flexible Pattern Matching Is Good

might have been better than matching for:

Calibration With Multiple Data Sources

Additional Variation Ideas

H Highlighting additional or other benefits

I Emphasizing “FREE for up to 5 Sites” even further

Thank you again for supporting the project!

GoodUI DATASTORIES - ISSUE #2 - JUNE 2014

You might also like