Product Roadmap via A/B Testing

Market Needs + Product Roadmap Deconstruct Roadmap Product Implementation Growth Conversion
Customer Product Managers or into Testable Chunks via A/B Testing CRO Data Scientist or
Expectations + business Analysts Analysts
Business Goals Business Analyst or UX Developers or CRO
Designers Analysts

01 02 03 04 05

Program Management
(Dedicated CRO program Manager)
 A/B Testing Process

01 Define Goal (conversion)

Before your create a test, your need 01
to know what, exactly, you’re
hoping to accomplish. 04 Crown the New Champion
Once you’ve launched your test, you
need to let it run for a long enough
02 05 period of time.
02 Brainstorm/Generate Idea
Once you’ve determined a goal you CONTINUOUS
want to accomplish, you’ll need to IMPROVEMENT
generate ideas.
05 Test Until Statistically Significant
By this point, you’ve set a goal, select
a page and determine what you want
03 Implement Changes 03 04 to test.

You will be need to implement

changes with highest potential,
intuition and best practices.
 A/B Testing Process

VARIATION A

30%
50% visitors Conversion rate
see variation A

VARIATION B

16%
50% visitors Conversion rate
see variation B
 A/B Test Results

VARIATION CONVERSION RATE IMPROVEMENT CONFIDENCE CONV./VISITORS STATUS

ORIGINAL 0.15% (+0.10%) - - 5/5,134 Stopped

VARIATION 01 0.25% (+0.20%) +150.00% 97.01% 13/5,152 Winner

A/B Testing Line Graph

Total
Totalvisitors
visitors 100

90

80

Average
Averageconversion
conversionrate
rate 70

60

50

Original
Original
40

30

20

10
Variation
Variation01
01
0
Mar-02 Mar-04 Mar-06 Mar-08 Mar-10 Mar-12 Mar-14 Mar-16
 A/B Test Tracking

OWNER
LANDING PAGE (LIST NAME STATISTICALLY
STATUS TEST START END PREDICTION RESULT NOTES NEXT STEPS
NAME OF DRI FOR SIGNIFICANT ?
TEST HERE)

Variation B performed 8% better than

Version B will perform
How to Create the original version and was
LP Header swapped better because the original Chose Variation B as winner, setting
Visuals for John Smith Completed 5/17/17 5/20/17 Success ✓ statistically significant at 95%
with subheader sub header is more up a/b on CTA placement next
Marketing confidence. Declared version B the
actionable
winner

Version B (image with Variation B performed 4% better than Chose Variation B. setting up a/b test
Demo Landing Amanda Different image on B
Not Started 5/27/17 6/20/17 background) will perform Inconclusive X A - not statistically significant but testing two different types of images
Page Morgan version
better declared B as winner. next

Version A will perform Version A performed 3% better, not

Call Sales Different Headline. A is Chose Variation A as winner, running
John Smith Not Started 6/23/17 7/6/17 better because it matches Inconclusive X statistically significant. Test ran for 3
Landing Page the title, B is descriptive template-level A/B test next
promo copy weeks.

Test "A" converted 33%better than

2018
Amanda Copy bullets vs long version B will perform Test "B." We are100% certain that the Chose Variation A as winner, setting
Predictions In Progress 6/17/17 7/12/17 Success ✓
Morgan form paragraphs better changes in Test "A" will improve your up a/b on CTA placement next
Landing Page
conversion rate.
Get the full version of this presentation and
access our library of hundreds of corporate
resources

Join You Exec Plus today and take your career to new
heights. Receive premium corporate resources in your
inbox every week.

Start today