1 Using pre-experiment data on primary metric

for variance reduction

This is a short illustration of how pre-experimental data on the primary met-
ric can be used to reduce the variance in the treatment-effect estimate of an
experiment using ordinary least square (OLS) regression adjustment.
As an illustration, the home hrs played 1d metric is used. The data set
consists of 909 users and two weeks of daily data on the metric under no exper-
iment. We can view this as an AA test where the first 7 days are pre-exposure
and the last 7 days are under exposure and study the properties of an estimator
not using any variance reduction (the raw difference-in-means) and a regression
based adjustment, using the pre-exposure data as a control variable.
Call the metric home hrs played 1d Y for short. For each user two values
are available in the analysis. The pre and post average home hrs played 1d, i.e.,
7
1X
Ȳpre,i = Yi,t (1)
7 t=1

and
14
1X
Ȳpost,i = Yi,t , (2)
7 t=8
where i = 1, ..., 909.
The two treatment effect estimators considered here are (1) the raw differ-
ence in mean in Ȳpost,i between the treated and the control in the post period,
and (2) the regression estimator, regression Ȳpost,i on the treatment assignment
indicator using Ȳpre,i as a control variable/covariate. Let Di be the treatment
assignment indicator taking the value one if unit i was assigned treatment and
zero otherwise.

dE raw = 1 1 X
X
AT Ypost,i − Ypost,i , (3)
n1 n0
i:Di =1 i:Di =0

where n1 and n0 are the number of users in the treatment and control group,
respecitvely. For the regression estimator we find the least square estimate of
β2 from
Ypost,i = β0 + β1 Ypre,i + β2 Di + i . (4)
That is, βˆ2 = ATdE regression . Since, assuming that treatment assignment is
randomized, Di is independent of the outcome (Y), the relative difference in
variance of AT
dE raw and AT dE regression is exactly the squared correlation be-
tween Ypost,i and Ypre,i .
In this data set the squared correlation between Ypost,i and Ypre,i is approx-
imately 0.46, which implies that the expected relative variance reduction from
V ar(AT
dE regression )−V ar(AT
dE raw )
the regression adjustment is 46%, i.e., ≈ −0.46.
V ar(AT
dE raw )

1
 0.6

0.3
Estimate

0.0

−0.3

−0.6

Raw Regression
Estimator

Figure 1: Distribution of estimates over random samples for the regression and
raw estimator

1.1 Simulations based on the empirical data

To illistrate this a small simulation is conducted. From the 909 users, a random
sample of 200 are drawn. The sample is evenly splitted into treatment and
control. Both estimators are applied and the estimate recorded. This is repeated
for 10000 random samples. Figure 1 displays the distribution of the estimates
from the random samples for both estimators. It is clear that the variance of
the regression estimator is smaller than that of the raw estimator. The variance
of the regression estimator is 0.0104, and for the raw 0.0194. This implies the
relative variance reduction (0.0104-0.0194)/0.0194=-0.466, close to the expected
theoretical value.
Since this is an AA test, the power is equal to the level, 0.05. However, to
illustrate how the variance reduction affects the power empirically, a simulation
with hypothetical effects is conducted. The same scheme as above is applied
but here, the users assigned to treatment gets a hypothetical treatment effect
added to there metric in the post period. The effect is varied between 0 and
0.8 home hrs played 1d. How large treatment effect that gives what power of
course depends on the sample size, this span of treatment effect were choosen
to obtain large variablility in power. The emperical power is caluclated as the
ratio of the 10000 samples with p-values smaller than or equal to 0.05 for both
estimators and for each hypothetical treatment effect. That is, the empirical
power is simply the prortion of samples for which the treatment effect would
have been concluded significanlty different from zero.
Figure 2 displays the power curves of the two estimators. It is clear, as

2
 1.00

0.75

Estimator
Power

Regression
0.50
Raw

0.25

0.00
0.0 0.2 0.4 0.6 0.8
Hypothetical treatment effect

Figure 2: Power for the regression and raw estimator for various hypothetical
treatment effects.

expected from the variance reduction results, that the power is higher for the
regression estimator for all hypothetical treatment effects until the power is 1
for both estimators.
To more clearly display the gain in power in terms of sample size, Figure
3 displays the power curves for a fixed treatment effect of 0.5, over different
sample sizes. It is clear that using regression, the desired power (usually 0.8)
is achieved for much small sample sizes using regression rather than the raw
estimator.

3
 1.0

0.9

0.8 Estimator
Power

Regression
Raw
0.7

0.6

100 150
Sample size

Figure 3: Power for the regression and raw estimator for various hypothetical
treatment effects.