GG E-commerce Experiment

April 22, 2022

Group 6
● Farin Cyntiya Garini - KM_G2DA2083
● Andrew - KM_G2DA2011
● Muhammad Rizal Indrasta - KM_G2DA2317
● Rahma Agustina - KM_G2DA2146
● Nurul Aeni - KM_G2DA2299

GG e-commerce is trying to sell more online classes, they want to perform the new process on a
landing page on their website.
● On the old page, they only have “Start Free Trial,” where students will be asked for credit
card information, and then they will be enrolled in a free trial in 14 days. After 14 days,
they will be charged unless they cancel.
● On the new page, students will see “Access Course Materials,” where students can
access videos and learning materials without any credit card information. After 14 days,
they will get a notification to upgrade their account to the paid version if they want to
continue studying.
Which page leads to better conversion?
Group 6 - Homework B
Group 6 - Google Colab

Goal/Objective

Help GG e-commerce finds out which page leads to better conversion to sell more online
classes.
General Problem

Students only use the free trial version of online classes without upgrading their accounts to the
paid version after 14 days. Therefore, GG e-commerce needs to decide whether they should
implement the new page or keep the old page.

Root Cause Analysis

● Students couldn’t give their credit cards information.

● Students were afraid they couldn't afford the paid version of online classes once the free
trial period ended.
● Students wanted to try using the free trial version first.
● Students needed notifications to remind them that their free trial period had ended.

Problem Statement

● Students didn’t have credit cards.

● Students didn’t have enough budget to upgrade their accounts.
● Students needed to make sure that the study materials fit their needs.
● Students wanted to be aware of being charged once the free trial period ended.

Proposed Solution

Choose the new landing page that leads to better conversion on GG e-commerce’s website to
sell more online classes.
Key Metrics

The conversion rate of online classes on GG e-commerce’s landing page (converted).

Population

The users of GG e-commerce’s online classes in January, 2017.

Business Hypothesis

The new page will result in more online classes sold as the conversion rate increases.

Hypothesis

H0: The new page doesn’t lead to better conversion than the old page.

𝑝𝑛𝑒𝑤 − 𝑝𝑜𝑙𝑑 ≤ 0

H1: The new page leads to better conversion than the old page.

𝑝𝑛𝑒𝑤 − 𝑝𝑜𝑙𝑑 > 0

Experiment Groups and Period

The total users of GG e-commerce online classes service are hundreds of people, the sample for
the experiment is 10% of the total users in school and college-age.

Control: Users with the old landing page.

Experiment: Users with the new landing page.

Experiment Monitoring

Monitor the conversion rate of online classes on GG e-commerce using a dashboard.

Analysis

Descriptive analysis:

Using python in Google Colab, we gained info about ab_data:

The data consists of 5 columns and 294478 rows with 290584 users, based on the number of
unique user_id.
By separating between categorical and numerical variables, we got the descriptive statistics as
the picture above.
● There are a total of 294478 conversions occurred with a mean (proportion) of 0,12 and a
standard deviation of 0,32.
● There are two groups (control & treatment) and two types of landing page (old & new).
● The most group is treatment with a frequency of 147276, while the most landing page is
the old page with a frequency of 147239.
● The control group has a higher conversion proportion (12,03%) than the treatment group
(11,89%).
Inferential analysis:

The purpose of this analysis is to test whether there is a statistically significant difference
between the control and experimental groups’ metrics that are in the form of proportions
(conversion rate) and the sample size is larger than 30. Therefore, we can use Central Limit
Theorem (CLT) to state that the sampling distributions of Control and Experimental groups are
asymptotically Normal, we can use 2-sample Z-test with the following hypothesis:

H0: The new page doesn’t lead to better conversion than the old page.

𝑝𝑛𝑒𝑤 − 𝑝𝑜𝑙𝑑 ≤ 0

H1: The new page leads to better conversion than the old page.

𝑝𝑛𝑒𝑤 − 𝑝𝑜𝑙𝑑 > 0

α = 5% = 0, 05

Using python in Google Colab, we gained a p-value of 0,90 which is larger than 0,05. Therefore,
the null hypothesis can not be rejected and is accepted, so the conclusion is the new page
doesn’t lead to better conversion than the old page.
Action Plan

Based on the result of the A/B testing analysis, since we got the conclusion that the new page
doesn’t lead to better conversion than the old page, GG E-commerce should keep using the old
landing page that only have the “Start Free Trial” option, where students will be asked for credit
card information, then they will be enrolled in a free trial in 14 days. After 14 days, they will be
charged unless they cancel.

Additional Action Plan

● Provide a preview of the special features the students will get if they upgrade their
account to the paid version.
● Add gamification features to drive engagement, like using points, exp (experience
points), etc.
● Simplify the process of upgrading the online classes accounts on GG e-commerce’s
landing page.
● Make sure to support many kinds of payment methods other than just using credit
cards.
● Provide special offers on the paid version of online classes (bundling, discounts, sales,
special prices, etc.).
● Do another experiment after adding more improvements to the landing page.