Giridhar Ramachandran

BM Term V Session 19
Marketing Experiments

2
Marketing Experiments

When would they become important?

• When we do not have past data that can be used to build multivariate models

• When we want a quick and effective way to establish causality

Recent applications

• Rise in online business and online marketing, makes it easier to run real time
experiments

3
Big honey’s problem

Mahesh, the brand manager of Big Honey wanted funds to run an advertising
campaign

But the CFO Malini is sceptical

“Can you convince me that sales of BigHoney will be hurt if you do not advertise?”

“You have requested $500,000. Can we do with less? How do you know this will be
the right amount?”

How does Mahesh convince Malini?

Typical challenge for marketing managers when they need to invest in the
marketing mix to increase future sales

Attributing an increase in sales to a specific marketing action is a major challenge

4
Marketing Experiments

One way to isolate such influences is through experiments

Mahesh could, on a smaller scale, measure the impact of his proposed campaign
on brand’s sales.

If he is designing an experiment, how would be able to determine causality?

• Launching the marketing campaign increases unit sales

• Not launching the marketing campaign causes no change to the sales figures
• Launching the marketing campaign today affects unit sales in the subsequent
time periods
• There are no other external factors (like competitive action) affecting unit
sales

5
Designing Marketing Experiments

Experiments involve at least two situations

A control group – where the independent variable is not changed, and

A test or treatment group – where the independent variable is changed

A statistically significant difference between the test and control groups suggests
causality

Such experiments are called as A/B testing, where A is the control group and B is
the treatment group

6
Experiment Types

After-only experiment

Supposing Mahesh designs an experiment with 1000 customers

500 customers are exposed to advertisement highlighting new packaging for a

month (test group)

Another 500 customers are exposed to the old advertisement for a month (control
group)

Supposing
Test group total sales = 1200 units
Control group total sales = 1000 units

Sales lift = 200 units

It is called After-Only because sales is measured after the exposure

7
Experiment Types

After-only experiment

This After-only design satisfies two of the four conditions for causality

Sales increase
in the short term, and
in subsequent periods

It cannot indicate whether

the increase might have occurred without the new ad
the preferences existed prior to the experiment

That can be addressed if the two groups are exposed to the same external
environment – store promotions, competitive reactions,..

8
Experiment Types

Deciding how to distribute customers between the test group and the control
group is critical

Key issue is extent to which participants in the test and control groups are similar
in terms of the factors relevant for the experiment

Two ways to achieve them

Randomization

Attribute matching

9
Experiment Types

Before-After experiment

Requires to measure the output of interest before and after the participants have
been exposed to the inputs

In our example

If we randomly divide 1000 participants into two groups (test and control)

Test group (500) Control group (500)

Exposed to old ad for a month Exposed to old ad for a month

Total sales = 1100 units Total sales = 1000 units
Exposed to ad highlighting the Exposed to old ad for one
packaging for one month month
Total sales = 1200 units Total sales = 1000 units

Sales lift = (1200 – 1000) – (1100 – 1000) = 100 units

10
Experiment Types

Field experiments

Conducted in a natural setting

Replicating real life situations

Advantage – people are rarely aware that they are participating in an experiment

Disadvantage –
Very difficult to control extraneous variables
Transparent to competition, and competitive reaction can cloud the results

Supposing Mahesh wants to test pricing on new packaging

  Product Price Sales Profit

City 1 $1.59 $1,000,000 $600,000
City 2 $1.89 $600,000 $540,000
City 3 $2.15 $500,000 $380,000

11
Experiment Types

Web experiments

Full factorial design

  Price
Ad copy $1.59 $1.89 $2.15
"Lasts longer" $1,315 $1,112 $1,206
"Tastes better" $957 $1,030 $1,500
"Good for you" $930 $820 $770

12