A/B testing business case

You have been working for a company called OnlySolution as a Product Analyst for quite some time.
It has a similar product and business model to JustAnswer – a very successful and awesome company.
OnlySolution is a data driven company so any changes they implement on their web-site they do via A/B
testing.
As your third assignment you have been asked to analyze the results of an A/B experiment.
Results with the recommendations should be presented in format of presentation.

Test #1. Rating Bar redesign

Since OnlySolution helps connect customer and Expert so customer can ask questions to an Expert, an
important thing for the company is how customer rates Expert’s Answer. But the problem is that customers
often do not rate an Expert. So the company decided to do an A/B test where the rating bar with five stars was
moved from the bottom of the page to the top of the page.
Primary metric: Share of customers who rated Experts (0% - 100%);
Other Metrics of Interest: LTV8 per customer (Sum of all payments minus sum of all refund divided by the
numbers of customers.
Expected Results: Calculate Share of customers who rated Experts and LTV8 per customer for Control and
Variation. Calculate change (Lift = (Variation/Control-1)) in primary metrics when compared Control to
Variation. Calculate 90% Confidence Interval for a lift to see if it crosses the zero to say if the changes in
primary metric (lift) is significant. Answer the question is the test winner or not? Give any other relevant
recommendations to the team.

You can find the dataset here:

https://justanswer-my.sharepoint.com/:x:/p/myron_tsisinskyi/Edr8fJFmtzZPt_QS9CjZaAABxFX7ntAGnqGRHNiI
66lQkw?e=nMmYL1

Test #2. Credit Card page redesign

OnlySolution funnel consists of the two main pages: Landing page (LP) where all people lands and can type
their question and Credit Card (CC) page where you can pay for the service fulling the form with credit card
details.
In order to increase the conversation rate, team did an A/B test where the variation had a new design of Credit
Card page.
Primary metric: Conversion Rate (0% - 100%);
Other Metrics of Interest LTV8 per converter, LTV8 per LP visit (sum of all LTV dollars values divided by the
number of LP visits);
Expected Results: Calculate conversion rate for Control and Variation, calculate the lift (Variation/Control-1)
and check if the if the changes in primary metric (lift) is significant by calculating 90% Confidence Interval for a
lift. Calculate LTV8 per converter and LTV8 per LP visit. Calculate cumulative conversion rate by days for Control
and Variation and visualize it with the line plot. Answer the question is the test winner or not? Give any other
relevant recommendations to the team.
You can find the dataset here:
https://justanswer-my.sharepoint.com/:x:/p/myron_tsisinskyi/EaGvIwhR2ztEpYVS3Jd-
vG8B150_tW2nLxSmrJvFiA8wTA?e=xyLlY7

Test #3. Landing page redesign

OnlySolution funnel consists of the two main pages: Landing page (LP) where all people lands and can type
their question and Credit Card (CC) page where you can pay for the service fulling the form with credit card
details.
In order to increase the conversation rate, team did an A/B test where the variation had a new design of an
Landing Page.
Primary metric: Conversion Rate (0% - 100%);
Other Metrics of Interest LTV8 per converter, LTV8 per LP visit (sum of all LTV dollars values divided by the
number of LP visits);
Expected Results: Calculate conversion rate for Control and Variation, calculate the lift (Variation/Control-1)
and check if the if the changes in primary metric (lift) is significant by calculating 90% Confidence Interval for a
lift. Calculate LTV8 per converter and LTV8 per LP visit.
Since the LP experiment was running simultaneously with the Credit Card experiment we’ve got four different
funnel scenarios for a customer:
1) Old Landing Page & Old Credit Page
2) New Landing Page & Old Credit Page
3) New Landing Page & New Credit Page
4) Old Landing Page & New Credit Page
Calculate conversion rate for all these segments.
Answer the question is the test winner or not? Give any other relevant recommendations to the team.
Dataset is the same as for Test #2