ECONOMETRIC

METHODS
BITS Pilani
Pilani Campus
 BITS Pilani
Pilani Campus

N V M Rao
Lecture – 3 & 4
2
 SAMPLE COVARIANCE: EXAMPLE CALCULATION

Definition of sample covariance:

Cov( X,Y )  ( X 1  X )(Y1  Y )  ...  ( X n  X )(Yn  Y )

1
n
1 n
=  ( X i  X )(Yi  Y )
n i 1

This sequence introduces the concept of sample variance and illustrates it with an example.

1
 SAMPLE COVARIANCE: EXAMPLE CALCULATION

Definition of sample covariance:

Cov( X,Y )  ( X 1  X )(Y1  Y )  ...  ( X n  X )(Yn  Y )

1
n
1 n
=  ( X i  X )(Yi  Y )
n i 1

Given a sample of n observations on two variables X and Y, the sample covariance is the
average of the products of their deviations about their sample means.

2
 COVARIANCE RULES

1. If Y = V + W,
Cov(X, Y) = Cov(X, V) + Cov(X, W)

This sequence states three simple rules for manipulating covariance expressions. Proofs
are provided at the end. First, the addition rule.

1
 COVARIANCE RULES

1. If Y = V + W,
Cov(X, Y) = Cov(X, V) + Cov(X, W)

2. If Y = bZ, where b is a constant,

Cov(X, Y) = Cov(X, bZ) = bCov(X, Z)

Next, the multiplication rule, for cases where a variable is multiplied by a constant.

2
 COVARIANCE RULES

1. If Y = V + W,
Cov(X, Y) = Cov(X, V) + Cov(X, W)

2. If Y = bZ, where b is a constant,

Cov(X, Y) = Cov(X, bZ) = bCov(X, Z)
Example: Cov(X, 3Z) = 3Cov(X, Z)

An example.

3
 COVARIANCE RULES

1. If Y = V + W,
Cov(X, Y) = Cov(X, V) + Cov(X, W)

2. If Y = bZ, where b is a constant,

Cov(X, Y) = Cov(X, bZ) = bCov(X, Z)
Example: Cov(X, 3Z) = 3Cov(X, Z)

3. If Y = b, where b is a constant,
Cov(X, Y) = Cov(X, b) = 0

Finally, a primitive rule that is often useful.

4
 COVARIANCE RULES

1. If Y = V + W,
Cov(X, Y) = Cov(X, V) + Cov(X, W)

2. If Y = bZ, where b is a constant,

Cov(X, Y) = Cov(X, bZ) = bCov(X, Z)
Example: Cov(X, 3Z) = 3Cov(X, Z)

3. If Y = b, where b is a constant,
Cov(X, Y) = Cov(X, b) = 0
Example: Cov(X, 10) = 0

An example.

5
 COVARIANCE RULES

Example: suppose Y = b1 + b2Z

Cov(X, Y) = Cov(X, [b1 + b2Z])

Here is a typical use of the rules. Suppose that a variable Y is a linear function of another
variable Z and we wish to analyze Cov(X, Y).

6
 COVARIANCE RULES

Example: suppose Y = b1 + b2Z

Cov(X, Y) = Cov(X, [b1 + b2Z])

= Cov(X, b1) + Cov(X, b2Z)

The first covariance rule has been used.

7
 COVARIANCE RULES

Example: suppose Y = b1 + b2Z

Cov(X, Y) = Cov(X, [b1 + b2Z])

= Cov(X, b1) + Cov(X, b2Z)
= 0 + Cov(X, b2Z)

The third rule has been used.

8
 COVARIANCE RULES

Example: suppose Y = b1 + b2Z

Cov(X, Y) = Cov(X, [b1 + b2Z])

= Cov(X, b1) + Cov(X, b2Z)
= 0 + Cov(X, b2Z)
= b2Cov(X, Z)

And now the second. This is as far as we can go with this example.

9
 COVARIANCE RULES

1. If Y = V + W, Cov(X, Y) = Cov(X, V) + Cov(X, W)

1 n
Cov ( X , Y )   ( X i  X )(Yi  Y )
n i 1
1 n
  ( X i  X )([Vi  Wi ]  [V  W ])
n i 1
1 n
  ( X i  X )([Vi  V ]  [Wi  W ])
n i 1
1 n 1 n
  ( X i  X )(Vi  V )   ( X i  X )(W i  W )
n i 1 n i 1
 Cov ( X ,V )  Cov ( X ,W )

The proofs of the rules are not important, so feel free to skip the rest of this sequence.
In each case the proof starts with the definition of Cov(X, Y).

10
 COVARIANCE RULES

1. If Y = V + W, Cov(X, Y) = Cov(X, V) + Cov(X, W)

1 n
Cov ( X , Y )   ( X i  X )(Yi  Y )
n i 1
1 n
  ( X i  X )([Vi  Wi ]  [V  W ])
n i 1
1 n
  ( X i  X )([Vi  V ]  [Wi  W ])
n i 1
1 n 1 n
  ( X i  X )(Vi  V )   ( X i  X )(W i  W )
n i 1 n i 1
 Cov ( X ,V )  Cov ( X ,W )

We now substitute for Y. Yi separates into Vi and Wi.

11
 COVARIANCE RULES

1. If Y = V + W, Cov(X, Y) = Cov(X, V) + Cov(X, W)

1 n
Cov ( X , Y )   ( X i  X )(Yi  Y )
n i 1
1 n
  ( X i  X )([Vi  Wi ]  [V  W ])
n i 1
1 n
  ( X i  X )([Vi  V ]  [Wi  W ])
n i 1
1 n 1 n
  ( X i  X )(Vi  V )   ( X i  X )(W i  W )
n i 1 n i 1
 Cov ( X ,V )  Cov ( X ,W )

And the mean value of Y separates into the mean values of V and W.

12
 COVARIANCE RULES

1. If Y = V + W, Cov(X, Y) = Cov(X, V) + Cov(X, W)

1 n
Cov ( X , Y )   ( X i  X )(Yi  Y )
n i 1
1 n
  ( X i  X )([Vi  Wi ]  [V  W ])
n i 1
1 n
  ( X i  X )([Vi  V ]  [Wi  W ])
n i 1
1 n 1 n
  ( X i  X )(Vi  V )   ( X i  X )(W i  W )
n i 1 n i 1
 Cov ( X ,V )  Cov ( X ,W )

Now we change the order of the V and W components. We group the V terms together.

13
 COVARIANCE RULES

1. If Y = V + W, Cov(X, Y) = Cov(X, V) + Cov(X, W)

1 n
Cov ( X , Y )   ( X i  X )(Yi  Y )
n i 1
1 n
  ( X i  X )([Vi  Wi ]  [V  W ])
n i 1
1 n
  ( X i  X )([Vi  V ]  [Wi  W ])
n i 1
1 n 1 n
  ( X i  X )(Vi  V )   ( X i  X )(W i  W )
n i 1 n i 1
 Cov ( X ,V )  Cov ( X ,W )

And similarly the W terms.

14
 COVARIANCE RULES

1. If Y = V + W, Cov(X, Y) = Cov(X, V) + Cov(X, W)

1 n
Cov ( X , Y )   ( X i  X )(Yi  Y )
n i 1
1 n
  ( X i  X )([Vi  Wi ]  [V  W ])
n i 1
1 n
  ( X i  X )([Vi  V ]  [Wi  W ])
n i 1
1 n 1 n
  ( X i  X )(Vi  V )   ( X i  X )(W i  W )
n i 1 n i 1
 Cov ( X ,V )  Cov ( X ,W )

We can now break up the expression as shown.

15
 COVARIANCE RULES

1. If Y = V + W, Cov(X, Y) = Cov(X, V) + Cov(X, W)

1 n
Cov ( X , Y )   ( X i  X )(Yi  Y )
n i 1
1 n
  ( X i  X )([Vi  Wi ]  [V  W ])
n i 1
1 n
  ( X i  X )([Vi  V ]  [Wi  W ])
n i 1
1 n 1 n
  ( X i  X )(Vi  V )   ( X i  X )(W i  W )
n i 1 n i 1
 Cov ( X ,V )  Cov ( X ,W )

This gives us the result we want.

16
 COVARIANCE RULES

2. If Y = bZ, where b is a constant,

Cov(X, Y) = Cov(X, bZ) = bCov(X, Z)

1 n
Cov ( X , Y )   ( X i  X )(Yi  Y )
n i 1
1 n
  ( X i  X )(bZ i  bZ )
n i 1
1 n
 b  ( X i  X )( Z i  Z )
n i 1
 bCov ( X , Z )

Next, the multiplication rule, for cases where a variable is multiplied by a constant.

17
 COVARIANCE RULES

2. If Y = bZ, where b is a constant,

Cov(X, Y) = Cov(X, bZ) = bCov(X, Z)

1 n
Cov ( X , Y )   ( X i  X )(Yi  Y )
n i 1
1 n
  ( X i  X )(bZ i  bZ )
n i 1
1 n
 b  ( X i  X )( Z i  Z )
n i 1
 bCov ( X , Z )

The Y terms have been replaced by the corresponding aZ terms.

18
 COVARIANCE RULES

2. If Y = bZ, where b is a constant,

Cov(X, Y) = Cov(X, bZ) = bCov(X, Z)

1 n
Cov ( X , Y )   ( X i  X )(Yi  Y )
n i 1
1 n
  ( X i  X )(bZ i  bZ )
n i 1
1 n
 b  ( X i  X )( Z i  Z )
n i 1
 bCov ( X , Z )

a is a common factor.

19
 COVARIANCE RULES

2. If Y = bZ, where b is a constant,

Cov(X, Y) = Cov(X, bZ) = bCov(X, Z)

1 n
Cov ( X , Y )   ( X i  X )(Yi  Y )
n i 1
1 n
  ( X i  X )(bZ i  bZ )
n i 1
1 n
 b  ( X i  X )( Z i  Z )
n i 1
 bCov ( X , Z )

Hence we obtain the result.

20
 COVARIANCE RULES

3. If Y = b, where b is a constant,
Cov(X, Y) = Cov(X, b) = 0

1 n
Cov ( X , Y )   ( X i  X )(Yi  Y )
n i 1
1 n
  ( X i  X )(b  b )
n i 1
1 n
  ( X i  X )(b  b )
n i 1
0

Now for the third rule.

21
 COVARIANCE RULES

3. If Y = b, where b is a constant,
Cov(X, Y) = Cov(X, b) = 0

1 n
Cov ( X , Y )   ( X i  X )(Yi  Y )
n i 1
1 n
  ( X i  X )(b  b )
n i 1
1 n
  ( X i  X )(b  b )
n i 1
0

The Y terms have been replaced by the corresponding a terms.

22
 COVARIANCE RULES

3. If Y = b, where b is a constant,
Cov(X, Y) = Cov(X, b) = 0

1 n
Cov ( X , Y )   ( X i  X )(Yi  Y )
n i 1
1 n
  ( X i  X )(b  b )
n i 1
1 n
  ( X i  X )(b  b )
n i 1
0

The mean value of a is just a itself.

23
 COVARIANCE RULES

3. If Y = b, where b is a constant,
Cov(X, Y) = Cov(X, b) = 0

1 n
Cov ( X , Y )   ( X i  X )(Yi  Y )
n i 1
1 n
  ( X i  X )(b  b )
n i 1
1 n
  ( X i  X )(b  b )
n i 1
0

The summation is equal to zero because every term in it has a zero factor.

24
 ALTERNATIVE EXPRESSION FOR SAMPLE COVARIANCE

1 n 
Cov ( X , Y )    X iYi   XY
n  i 1 

An alternative expression for sample covariance is shown above. This sequence shows
how it can be derived from the original definition.

1
 ALTERNATIVE EXPRESSION FOR SAMPLE COVARIANCE

1 n
Cov( X , Y )   ( X i  X )(Yi  Y )
n i 1
1 n
  ( X iYi  X iY  XYi  XY )
n i 1

  X 1Y1  X 1Y  XY1  XY
1
n
 ...
 X nYn  X nY  XYn  XY 
1 n n n

   X iYi   X iY   XYi  nXY 
n  i 1 i 1 i 1 

We start with the original definition.

2
 ALTERNATIVE EXPRESSION FOR SAMPLE COVARIANCE

1 n
Cov( X , Y )   ( X i  X )(Yi  Y )
n i 1
1 n
  ( X iYi  X iY  XYi  XY )
n i 1

  X 1Y1  X 1Y  XY1  XY
1
n
 ...
 X nYn  X nY  XYn  XY 
1 n n n

   X iYi   X iY   XYi  nXY 
n  i 1 i 1 i 1 

The first step is to multiply out the two terms.

3
 ALTERNATIVE EXPRESSION FOR SAMPLE COVARIANCE

1 n
Cov( X , Y )   ( X i  X )(Yi  Y )
n i 1
1 n
  ( X iYi  X iY  XYi  XY )
n i 1

  X 1Y1  X 1Y  XY1  XY
1
n
 ...
 X nYn  X nY  XYn  XY 
1 n n n

   X iYi   X iY   XYi  nXY 
n  i 1 i 1 i 1 

We will now write out the expression in full, without using the S sign. First put i equal to 1
and write down the four corresponding terms.

4
 ALTERNATIVE EXPRESSION FOR SAMPLE COVARIANCE

1 n
Cov( X , Y )   ( X i  X )(Yi  Y )
n i 1
1 n
  ( X iYi  X iY  XYi  XY )
n i 1

  X 1Y1  X 1Y  XY1  XY
1
n
 ...
 X nYn  X nY  XYn  XY 
1 n n n

   X iYi   X iY   XYi  nXY 
n  i 1 i 1 i 1 

We do the same with all the other observations.

5
 ALTERNATIVE EXPRESSION FOR SAMPLE COVARIANCE

1 n
Cov( X , Y )   ( X i  X )(Yi  Y )
n i 1
1 n
  ( X iYi  X iY  XYi  XY )
n i 1

  X 1Y1  X 1Y  XY1  XY
1
n
 ...
 X nYn  X nY  XYn  XY 
1 n n n

   X iYi   X iY   XYi  nXY 
n  i 1 i 1 i 1 

We now sum vertically. The first summation is the summation of the first terms in the n
lines.

6
 ALTERNATIVE EXPRESSION FOR SAMPLE COVARIANCE

1 n
Cov( X , Y )   ( X i  X )(Yi  Y )
n i 1
1 n
  ( X iYi  X iY  XYi  XY )
n i 1

  X 1Y1  X 1Y  XY1  XY
1
n
 ...
 X nYn  X nY  XYn  XY 
1 n n n

   X iYi   X iY   XYi  nXY 
n  i 1 i 1 i 1 

Now the second terms.

7
 ALTERNATIVE EXPRESSION FOR SAMPLE COVARIANCE

1 n
Cov( X , Y )   ( X i  X )(Yi  Y )
n i 1
1 n
  ( X iYi  X iY  XYi  XY )
n i 1

  X 1Y1  X 1Y  XY1  XY
1
n
 ...
 X nYn  X nY  XYn  XY 
1 n n n

   X iYi   X iY   XYi  nXY 
n  i 1 i 1 i 1 

Now the third terms.

8
 ALTERNATIVE EXPRESSION FOR SAMPLE COVARIANCE

1 n
Cov( X , Y )   ( X i  X )(Yi  Y )
n i 1
1 n
  ( X iYi  X iY  XYi  XY )
n i 1

  X 1Y1  X 1Y  XY1  XY
1
n
 ...
 X nYn  X nY  XYn  XY 
1 n n n

   X iYi   X iY   XYi  nXY 
n  i 1 i 1 i 1 

The fourth term is the same for each observation.

9
 ALTERNATIVE EXPRESSION FOR SAMPLE COVARIANCE

1 n n n

Cov( X , Y )    X iYi   X iY   XYi  nXY 
n  i 1 i 1 i 1 
1 n  1 n  1 n 
   X iYi     X iY     XYi   XY
n  i 1  n  i 1  n  i 1 
1 n  1 n  1 n 
   X iYi   Y   X i   X   Yi   XY
n  i 1   n i 1   n i 1 
1 n 
   X iYi   Y X  XY  XY
n  i 1 
1 n 
   X iYi   XY
n  i 1 

This is the last line of the previous slide.

10
 ALTERNATIVE EXPRESSION FOR SAMPLE COVARIANCE

1 n n n

Cov( X , Y )    X iYi   X iY   XYi  nXY 
n  i 1 i 1 i 1 
1 n  1 n  1 n 
   X iYi     X iY     XYi   XY
n  i 1  n  i 1  n  i 1 
1 n  1 n  1 n 
   X iYi   Y   X i   X   Yi   XY
n  i 1   n i 1   n i 1 
1 n 
   X iYi   Y X  XY  XY
n  i 1 
1 n 
   X iYi   XY
n  i 1 

We break up the expression into its four components.

11
 ALTERNATIVE EXPRESSION FOR SAMPLE COVARIANCE

1 n n n

Cov( X , Y )    X iYi   X iY   XYi  nXY 
n  i 1 i 1 i 1 
1 n  1 n  1 n 
   X iYi     X iY     XYi   XY
n  i 1  n  i 1  n  i 1 
1 n  1 n  1 n 
   X iYi   Y   X i   X   Yi   XY
n  i 1   n i 1   n i 1 
1 n 
   X iYi   Y X  XY  XY
n  i 1 
1 n 
   X iYi   XY
n  i 1 

The mean value of Y is a common factor in the second summation.

12
 ALTERNATIVE EXPRESSION FOR SAMPLE COVARIANCE

1 n n n

Cov( X , Y )    X iYi   X iY   XYi  nXY 
n  i 1 i 1 i 1 
1 n  1 n  1 n 
   X iYi     X iY     XYi   XY
n  i 1  n  i 1  n  i 1 
1 n  1 n  1 n 
   X iYi   Y   X i   X   Yi   XY
n  i 1   n i 1   n i 1 
1 n 
   X iYi   Y X  XY  XY
n  i 1 
1 n 
   X iYi   XY
n  i 1 

Similarly, the mean value of X is a common factor in the third summation.

13
 ALTERNATIVE EXPRESSION FOR SAMPLE COVARIANCE

1 n n n

Cov( X , Y )    X iYi   X iY   XYi  nXY 
n  i 1 i 1 i 1 
1 n  1 n  1 n 
   X iYi     X iY     XYi   XY
n  i 1  n  i 1  n  i 1 
1 n  1 n  1 n 
   X iYi   Y   X i   X   Yi   XY
n  i 1   n i 1   n i 1 
1 n 
   X iYi   Y X  XY  XY
n  i 1 
1 n 
   X iYi   XY
n  i 1 

The term in square brackets is by definition the mean value of X.

14
 ALTERNATIVE EXPRESSION FOR SAMPLE COVARIANCE

1 n n n

Cov( X , Y )    X iYi   X iY   XYi  nXY 
n  i 1 i 1 i 1 
1 n  1 n  1 n 
   X iYi     X iY     XYi   XY
n  i 1  n  i 1  n  i 1 
1 n  1 n  1 n 
   X iYi   Y   X i   X   Yi   XY
n  i 1   n i 1   n i 1 
1 n 
   X iYi   Y X  XY  XY
n  i 1 
1 n 
   X iYi   XY
n  i 1 

Similarly for Y.

15
 ALTERNATIVE EXPRESSION FOR SAMPLE COVARIANCE

1 n n n

Cov( X , Y )    X iYi   X iY   XYi  nXY 
n  i 1 i 1 i 1 
1 n  1 n  1 n 
   X iYi     X iY     XYi   XY
n  i 1  n  i 1  n  i 1 
1 n  1 n  1 n 
   X iYi   Y   X i   X   Yi   XY
n  i 1   n i 1   n i 1 
1 n 
   X iYi   Y X  XY  XY
n  i 1 
1 n 
   X iYi   XY
n  i 1 

Two of the terms cancel and we arrive at the alternative expression. Note that the 1/n factor
applies only to the first term.

16
 SAMPLE VARIANCE AND VARIANCE RULES

Definition of sample variance

1 n
Var ( X )   ( X i  X ) 2
n i 1
1 n
  ( X i  X )( X i  X )
n i 1
 Cov ( X , X )

Given a set of observations on a random variable X, the variance of X is its average squared
deviation about its sample mean.

1
 SAMPLE VARIANCE AND VARIANCE RULES

Definition of sample variance

1 n
Var ( X )   ( X i  X ) 2
n i 1
1 n
  ( X i  X )( X i  X )
n i 1
 Cov ( X , X )

The sample variance can be treated as a special case of a sample covariance. To see this,
write out the quadratic.

2
 SAMPLE VARIANCE AND VARIANCE RULES

Definition of sample variance

1 n
Var ( X )   ( X i  X ) 2
n i 1
1 n
  ( X i  X )( X i  X )
n i 1
 Cov ( X , X )

We obtain the expression for the sample covariance of X with itself.

3
 SAMPLE VARIANCE AND VARIANCE RULES

Variance Rule 1:
If Y = V + W, Var(Y) = Var(V) + Var(W) + 2Cov(V, W)

Proof:
Var(Y) = Cov(Y, Y) = Cov(Y, [V + W])
= Cov(Y, V) + Cov(Y, W)
= Cov([V + W], V) + Cov([V + W], W)
= Cov(V, V) + Cov(W, V)
+ Cov(V, W) + Cov(W, W)
= Var(V) + Var(W) + 2Cov(V, W)

We can use this result to derive some variance rules from the covariance rules. The first
rule concerns the expansion of the variance of the sum of two random variables.

4
 SAMPLE VARIANCE AND VARIANCE RULES

Variance Rule 1:
If Y = V + W, Var(Y) = Var(V) + Var(W) + 2Cov(V, W)

Proof:
Var(Y) = Cov(Y, Y) = Cov(Y, [V + W])
= Cov(Y, V) + Cov(Y, W)
= Cov([V + W], V) + Cov([V + W], W)
= Cov(V, V) + Cov(W, V)
+ Cov(V, W) + Cov(W, W)
= Var(V) + Var(W) + 2Cov(V, W)

Write the variance of Y as its covariance with itself and substitute for one of the Y terms.

5
 SAMPLE VARIANCE AND VARIANCE RULES

Variance Rule 1:
If Y = V + W, Var(Y) = Var(V) + Var(W) + 2Cov(V, W)

Proof:
Var(Y) = Cov(Y, Y) = Cov(Y, [V + W])
= Cov(Y, V) + Cov(Y, W)
= Cov([V + W], V) + Cov([V + W], W)
= Cov(V, V) + Cov(W, V)
+ Cov(V, W) + Cov(W, W)
= Var(V) + Var(W) + 2Cov(V, W)

Expand using the first covariance rule.

6
 SAMPLE VARIANCE AND VARIANCE RULES

Variance Rule 1:
If Y = V + W, Var(Y) = Var(V) + Var(W) + 2Cov(V, W)

Proof:
Var(Y) = Cov(Y, Y) = Cov(Y, [V + W])
= Cov(Y, V) + Cov(Y, W)
= Cov([V + W], V) + Cov([V + W], W)
= Cov(V, V) + Cov(W, V)
+ Cov(V, W) + Cov(W, W)
= Var(V) + Var(W) + 2Cov(V, W)

Now substitute for the other Y in each term.

7
 SAMPLE VARIANCE AND VARIANCE RULES

Variance Rule 1:
If Y = V + W, Var(Y) = Var(V) + Var(W) + 2Cov(V, W)

Proof:
Var(Y) = Cov(Y, Y) = Cov(Y, [V + W])
= Cov(Y, V) + Cov(Y, W)
= Cov([V + W], V) + Cov([V + W], W)
= Cov(V, V) + Cov(W, V)
+ Cov(V, W) + Cov(W, W)
= Var(V) + Var(W) + 2Cov(V, W)

Use the first covariance rule again, twice.

8
 SAMPLE VARIANCE AND VARIANCE RULES

Variance Rule 1:
If Y = V + W, Var(Y) = Var(V) + Var(W) + 2Cov(V, W)

Proof:
Var(Y) = Cov(Y, Y) = Cov(Y, [V + W])
= Cov(Y, V) + Cov(Y, W)
= Cov([V + W], V) + Cov([V + W], W)
= Cov(V, V) + Cov(W, V)
+ Cov(V, W) + Cov(W, W)
= Var(V) + Var(W) + 2Cov(V, W)

Cov(V, V) is just Var(V).

9
 SAMPLE VARIANCE AND VARIANCE RULES

Variance Rule 1:
If Y = V + W, Var(Y) = Var(V) + Var(W) + 2Cov(V, W)

Proof:
Var(Y) = Cov(Y, Y) = Cov(Y, [V + W])
= Cov(Y, V) + Cov(Y, W)
= Cov([V + W], V) + Cov([V + W], W)
= Cov(V, V) + Cov(W, V)
+ Cov(V, W) + Cov(W, W)
= Var(V) + Var(W) + 2Cov(V, W)

And Cov(W, W) is Var(W).

10
 SAMPLE VARIANCE AND VARIANCE RULES

Variance Rule 1:
If Y = V + W, Var(Y) = Var(V) + Var(W) + 2Cov(V, W)

Proof:
Var(Y) = Cov(Y, Y) = Cov(Y, [V + W])
= Cov(Y, V) + Cov(Y, W)
= Cov([V + W], V) + Cov([V + W], W)
= Cov(V, V) + Cov(W, V)
+ Cov(V, W) + Cov(W, W)
= Var(V) + Var(W) + 2Cov(V, W)

Note that the order of the variables makes no difference when defining covariance and
hence Cov(W, V) is the same as Cov(V, W).

11
 SAMPLE VARIANCE AND VARIANCE RULES

Variance Rule 2:
If Y = bZ, where b is a constant, Var(Y) = b2Var(Z)

Proof:
Var(Y) = Cov(Y, Y) = Cov(Y, bZ)
= bCov(Y, Z)
= bCov(bZ, Z)
= b2Cov(Z, Z)
= b2Var(Z)

Next, multiplication of a random variable by a scalar (constant). This has the effect of
multiplying the variance by the square of the scalar.

12
 SAMPLE VARIANCE AND VARIANCE RULES

Variance Rule 2:
If Y = bZ, where b is a constant, Var(Y) = b2Var(Z)

Proof:
Var(Y) = Cov(Y, Y) = Cov(Y, bZ)
= bCov(Y, Z)
= bCov(bZ, Z)
= b2Cov(Z, Z)
= b2Var(Z)

Write the variance of Y as its covariance with itself and substitute for one of the Y terms.

13
 SAMPLE VARIANCE AND VARIANCE RULES

Variance Rule 2:
If Y = bZ, where b is a constant, Var(Y) = b2Var(Z)

Proof:
Var(Y) = Cov(Y, Y) = Cov(Y, bZ)
= bCov(Y, Z)
= bCov(bZ, Z)
= b2Cov(Z, Z)
= b2Var(Z)

Now use the second covariance rule.

14
 SAMPLE VARIANCE AND VARIANCE RULES

Variance Rule 2:
If Y = bZ, where b is a constant, Var(Y) = b2Var(Z)

Proof:
Var(Y) = Cov(Y, Y) = Cov(Y, bZ)
= bCov(Y, Z)
= bCov(bZ, Z)
= b2Cov(Z, Z)
= b2Var(Z)

Substitute for the other Y term.

15
 SAMPLE VARIANCE AND VARIANCE RULES

Variance Rule 2:
If Y = bZ, where b is a constant, Var(Y) = b2Var(Z)

Proof:
Var(Y) = Cov(Y, Y) = Cov(Y, bZ)
= bCov(Y, Z)
= bCov(bZ, Z)
= b2Cov(Z, Z)
= b2Var(Z)

Finally, use the second covariance rule again.

16
 SAMPLE VARIANCE AND VARIANCE RULES

Variance Rule 3:
If Y = b, where b is a constant, Var(Y) = 0

Proof:
Var(Y) = Cov(Y, Y)
= Cov(b, b)
=0

Obviously, the variance of a constant is zero.

17
 SAMPLE VARIANCE AND VARIANCE RULES

Variance Rule 3:
If Y = b, where b is a constant, Var(Y) = 0

Proof:
Var(Y) = Cov(Y, Y)
= Cov(b, b)
=0

To prove this formally, we use the third covariance rule.

18
 SAMPLE VARIANCE AND VARIANCE RULES

Variance Rule 4:
If Y = V + b, where b is a constant, Var(Y) = Var(V)

Proof:
Var(Y) = Var(V + b)
= Var(V) + Var(b) + 2Cov(V, b)
= Var(V)

Lastly, it is useful to define a fourth variance rule as shown.

19
 SAMPLE VARIANCE AND VARIANCE RULES

Variance Rule 4:
If Y = V + b, where b is a constant, Var(Y) = Var(V)

Proof:
Var(Y) = Var(V + b)
= Var(V) + Var(b) + 2Cov(V, b)
= Var(V)

To prove it, begin by using the first variance rule.

20
 SAMPLE VARIANCE AND VARIANCE RULES

Variance Rule 4:
If Y = V + b, where b is a constant, Var(Y) = Var(V)

Proof:
Var(Y) = Var(V + b)
= Var(V) + Var(b) + 2Cov(V, b)
= Var(V)

Using the third variance and covariance rules, the last two terms are both zero.

21
 SAMPLE VARIANCE AND VARIANCE RULES

Variance Rule 4:
If Y = V + b, where b is a constant, Var(Y) = Var(V)

Proof:
Var(Y) = Var(V + b)
= Var(V) + Var(b) + 2Cov(V, b)
= Var(V)

0 V
0 V+b

Why does adding a constant to a variable not affect its variance?

22
 SAMPLE VARIANCE AND VARIANCE RULES

Variance Rule 4:
If Y = V + b, where b is a constant, Var(Y) = Var(V)

Proof:
Var(Y) = Var(V + b)
= Var(V) + Var(b) + 2Cov(V, b)
= Var(V)

0 V
0 V+b

In the diagram, the black markers represent the observations in the sample and the red
marker represents the sample mean.

23
 SAMPLE VARIANCE AND VARIANCE RULES

Variance Rule 4:
If Y = V + b, where b is a constant, Var(Y) = Var(V)

Proof:
Var(Y) = Var(V + b)
= Var(V) + Var(b) + 2Cov(V, b)
= Var(V)

0 V
0 V+b

Adding a constant to V has the effect of shifting each of the observations by that amount to
the right. As a consequence, the sample mean is shifted by the same amount.

24
 SAMPLE VARIANCE AND VARIANCE RULES

Variance Rule 4:
If Y = V + b, where b is a constant, Var(Y) = Var(V)

Proof:
Var(Y) = Var(V + b)
= Var(V) + Var(b) + 2Cov(V, b)
= Var(V)

0 V
0 V+b

Hence the deviations from the sample mean are unaffected. It follows that the sample
variance is also unaffected, because it depends only on the (squared) deviations.

25
 SAMPLE VARIANCE AND VARIANCE RULES

n 1 2
E Var( X )  X
n

The sample variance is a biased estimator of the population variance. It can easily be
shown that it is downwards biased by a factor (n-1)/n.

26
 SAMPLE VARIANCE AND VARIANCE RULES

n 1 2
E Var( X )  X
n
n
s 
2
Var( X )
n 1
X

Knowing this, we can define an unbiased estimator of the population variance. Just
compensate for the bias by multiplying the sample variance by n/(n-1).

27
 SAMPLE VARIANCE AND VARIANCE RULES

n 1 2
E Var( X )  X
n
n
s 
2
Var( X )
n 1
X

E s   E 
2  n 
Var ( X )
n  1
X

E Var ( X )
n

n 1
n n 1 2
   X   X2
n 1 n

The bias is cancelled out, so s2, as defined, is an unbiased estimator.

28
 POPULATION COVARIANCE

population covariance of X and Y   XY

 E ( X   X )(Y  Y )

The population covariance of two random variables is defined to be the expected value of
the product of their deviations from their population means.

1
 POPULATION COVARIANCE

population covariance of X and Y   XY

 E ( X   X )(Y  Y )

n 1
E Cov ( X , Y )   XY
n

The sample covariance is a biased estimator of the population covariance. It is biased

downwards by a factor (n - 1)/n.

2
 POPULATION COVARIANCE

population covariance of X and Y   XY

 E ( X   X )(Y  Y )

n 1  n 
E Cov ( X , Y )   XY  E Cov ( X , Y )   XY
n n  1 

As with the variance, an unbiased estimator can be obtained by compensating for the bias.

3
 POPULATION COVARIANCE

population covariance of X and Y   XY

 E ( X   X )(Y  Y )

n 1  n 
E Cov ( X , Y )   XY  E Cov ( X , Y )   XY
n n  1 

If X and Y are independent, XY = 0

E ( X   X )(Y  Y )  E ( X   X )E (Y  Y )
 E ( X )  E (  X )E (Y )  E ( Y )
  X   X Y  Y   0  0  0

If two variables are independent, their population covariance is zero. To show this, start by
rewriting the population covariance as the product of the expected values of its factors.

4
 POPULATION COVARIANCE

population covariance of X and Y   XY

 E ( X   X )(Y  Y )

n 1  n 
E Cov ( X , Y )   XY  E Cov ( X , Y )   XY
n n  1 

If X and Y are independent, XY = 0

E ( X   X )(Y  Y )  E ( X   X )E (Y  Y )
 E ( X )  E (  X )E (Y )  E ( Y )
  X   X Y  Y   0  0  0

We are allowed to do this because (and only because) X and Y are independent (see the
sequence on independence in the Review chapter).

5
 POPULATION COVARIANCE

population covariance of X and Y   XY

 E ( X   X )(Y  Y )

n 1  n 
E Cov ( X , Y )   XY  E Cov ( X , Y )   XY
n n  1 

If X and Y are independent, XY = 0

E ( X   X )(Y  Y )  E ( X   X )E (Y  Y )
 E ( X )  E (  X )E (Y )  E ( Y )
  X   X Y  Y   0  0  0

The expected values of both factors are zero because E(X) = X and E(Y) = Y. E(X) = X
and E(Y) = Y because X and Y are constants.

6
 POPULATION VARIANCE OF THE SAMPLE MEAN

Preliminary result: if X and Y are independent,  X2 Y   X2   Y2

 X2 Y   X2   Y2  2 XY
  X2   Y2

This sequence derives the expression for the population variance of a sample mean. We
will make use of the first variance rule, shown above.

1
 POPULATION VARIANCE OF THE SAMPLE MEAN

Preliminary result: if X and Y are independent,  X2 Y   X2   Y2

 X2 Y   X2   Y2  2 XY
  X2   Y2

If X and Y are independent, their population covariance is zero (see the previous sequence)
and so the population variance of (X + Y) is the sum of the population variances of X and Y.

2
 POPULATION VARIANCE OF THE SAMPLE MEAN

Preliminary result: if X and Y are independent,  X2 Y   X2   Y2

 2 
 2
  1
X
 X 1  ... X n  
 n 
 2  X2 1 ... X n 
1
n
 2  X2 1 ...   X2 n 
1
n
 X2
 2  X ...   X   2 n X  
1 2 2 1 2

n n n

We will apply this result to the sample mean.

3
 POPULATION VARIANCE OF THE SAMPLE MEAN

Preliminary result: if X and Y are independent,  X2 Y   X2   Y2

 2 
 2
  1
X
 X 1  ... X n  
 n 
 2  X2 1 ... X n 
1
n
 2  X2 1 ...   X2 n 
1
n
 X2
 2  X ...   X   2 n X  
1 2 2 1 2

n n n

The second variance rule has been used.

4
 POPULATION VARIANCE OF THE SAMPLE MEAN

Preliminary result: if X and Y are independent,  X2 Y   X2   Y2

 2 
 2
  1
X
 X 1  ... X n  
 n 
 2  X2 1 ... X n 
1
n
 2  X2 1 ...   X2 n 
1
n
 X2
 2  X ...   X   2 n X  
1 2 2 1 2

n n n

And now the first. We are assuming that the observations are generated independently and
that therefore the population covariance terms are all zero.

5
 POPULATION VARIANCE OF THE SAMPLE MEAN

Preliminary result: if X and Y are independent,  X2 Y   X2   Y2

 2 
 2
  1
X
 X 1  ... X n  
 n 
 2  X2 1 ... X n 
1
n
 2  X2 1 ...   X2 n 
1
n
 X2
 2  X ...   X   2 n X  
1 2 2 1 2

n n n

Each observation comes from the distribution of X, so each has population variance X2.

6
 POPULATION VARIANCE OF THE SAMPLE MEAN

Preliminary result: if X and Y are independent,  X2 Y   X2   Y2

 2 
 2
  1
X
 X 1  ... X n  
 n 
 2  X2 1 ... X n 
1
n
 2  X2 1 ...   X2 n 
1
n
 X2
 2  X ...   X   2 n X  
1 2 2 1 2

n n n

(Of course, once the sample has been drawn, each observation is fixed. We are talking
about the potential distribution before the sample is drawn. Potentially each observation is
drawn from a distribution with population variance x2.)
7
 POPULATION VARIANCE OF THE SAMPLE MEAN

Preliminary result: if X and Y are independent,  X2 Y   X2   Y2

 2 
 2
  1
X
 X 1  ... X n  
 n 
 2  X2 1 ... X n 
1
n
 2  X2 1 ...   X2 n 
1
n
 X2
 2  X ...   X   2 n X  
1 2 2 1 2

n n n

There are n identical terms in the summation. Hence we obtain the expression for the
population variance of the sample mean. We will use it in the next sequence.

8
 CORRELATION COEFFICIENT: EXAMPLE CALCULATION

Population correlation coefficient

 XY
 XY 
 X2  Y2

The population correlation coefficient, XY, for two variables X and Y is defined above.

1
 CORRELATION COEFFICIENT: EXAMPLE CALCULATION

Population correlation coefficient

 XY
 XY 
 X2  Y2

It has maximum value 1 when there is an exact positive relationship and minimum value -1
when there is an exact negative relationship. It is zero if there is no association.

2
 CORRELATION COEFFICIENT: EXAMPLE CALCULATION

Population correlation coefficient

 XY
 XY 
 X2  Y2

Sample correlation coefficient

n
Cov ( X , Y )
n-1 Cov ( X , Y )
rXY  
n n Var ( X ) Var (Y )
Var ( X ) Var (Y )
n-1 n-1

Given a sample of observations, it can be estimated by replacing the population covariance

and variances by their unbiased estimators.

3
 CORRELATION COEFFICIENT: EXAMPLE CALCULATION

Population correlation coefficient

 XY
 XY 
 X2  Y2

Sample correlation coefficient

n
Cov ( X , Y )
n-1 Cov ( X , Y )
rXY  
n n Var ( X ) Var (Y )
Var ( X ) Var (Y )
n-1 n-1

The n/(n-1) terms cancel and the expression simplifies as shown. Note that, even if there is
no real association between X and Y, rXY will generally not be exactly equal to zero. It will
have some random distribution around zero.
4
 CORRELATION COEFFICIENT: EXAMPLE CALCULATION

Population correlation coefficient

 XY
 XY 
 X2  Y2

The population correlation coefficient, XY, for two variables X and Y is defined above.

1
 CORRELATION COEFFICIENT: EXAMPLE CALCULATION

Population correlation coefficient

 XY
 XY 
 X2  Y2

It has maximum value 1 when there is an exact positive relationship and minimum value -1
when there is an exact negative relationship. It is zero if there is no association.

2
 CORRELATION COEFFICIENT: EXAMPLE CALCULATION

Population correlation coefficient

 XY
 XY 
 X2  Y2

Sample correlation coefficient

n
Cov ( X , Y )
n-1 Cov ( X , Y )
rXY  
n n Var ( X ) Var (Y )
Var ( X ) Var (Y )
n-1 n-1

Given a sample of observations, it can be estimated by replacing the population covariance

and variances by their unbiased estimators.

3
 CORRELATION COEFFICIENT: EXAMPLE CALCULATION

Population correlation coefficient

 XY
 XY 
 X2  Y2

Sample correlation coefficient

n
Cov ( X , Y )
n-1 Cov ( X , Y )
rXY  
n n Var ( X ) Var (Y )
Var ( X ) Var (Y )
n-1 n-1

The n/(n-1) terms cancel and the expression simplifies as shown. Note that, even if there is
no real association between X and Y, rXY will generally not be exactly equal to zero. It will
have some random distribution around zero.
4
 CORRELATION COEFFICIENT: EXAMPLE CALCULATION
45

40

35
Y (hourly earnings)

30

25

20

15

10

0
0 2 4 6 8 10 12 14 16 18 20
S (highest grade completed)

The calculation of a sample correlation coefficient will be illustrated using the data on
schooling and earnings used to calculate sample covariance.

5
 CORRELATION COEFFICIENT: EXAMPLE CALCULATION
Obs S Y S -S Y -Y (S - S)(Y -Y ) (S - S)2 (Y -Y )2

1 15 17.24 1.75 3.016 5.277

2 16 15.00 2.75 0.776 2.133
3 8 14.91 -5.25 0.686 -3.599
4 6 4.50 -7.25 -9.725 70.503
5 15 18.00 1.75 3.776 6.607
6 12 6.29 -1.25 -7.935 9.918
7 12 19.23 -1.25 5.006 -6.257
8 18 18.69 4.75 4.466 21.211
9 12 7.21 -1.25 -7.015 8.768
10 20 42.06 6.75 27.836 187.890
... ... ... ... ... ...
... ... ... ... ... ...
19 12 7.50 -1.25 -6.725 8.406
20 14 8.00 0.75 -6.225 -4.668

Total 265 284.49 305.888

Ave 13.25 14.225 15.294
Here is the table constructed when calculating the sample covariance.

6
 CORRELATION COEFFICIENT: EXAMPLE CALCULATION
Obs S Y S -S Y -Y (S - S)(Y -Y ) (S - S)2 (Y -Y )2

1 15 17.24 1.75 3.016 5.277 3.063 9.093

2 16 15.00 2.75 0.776 2.133
3 8 14.91 -5.25 0.686 -3.599
4 6 4.50 -7.25 -9.725 70.503
5 15 18.00 1.75 3.776 6.607
6 12 6.29 -1.25 -7.935 9.918
7 12 19.23 -1.25 5.006 -6.257
8 18 18.69 4.75 4.466 21.211
9 12 7.21 -1.25 -7.015 8.768
10 20 42.06 6.75 27.836 187.890
... ... ... ... ... ...
... ... ... ... ... ...
19 12 7.50 -1.25 -6.725 8.406
20 14 8.00 0.75 -6.225 -4.668

Total 265 284.49 305.888

Ave 13.25 14.225 15.294
In addition, we need to calculate the sample variances of S and Y. To do this, we add two
new columns containing the squared deviations of S and Y about their sample means.

7
 CORRELATION COEFFICIENT: EXAMPLE CALCULATION
Obs S Y S -S Y -Y (S - S)(Y -Y ) (S - S)2 (Y -Y )2

1 15 17.24 1.75 3.016 5.277 3.063 9.093

2 16 15.00 2.75 0.776 2.133 7.563 0.601
3 8 14.91 -5.25 0.686 -3.599 27.563 0.470
4 6 4.50 -7.25 -9.725 70.503 52.563 94.566
5 15 18.00 1.75 3.776 6.607 3.063 14.254
6 12 6.29 -1.25 -7.935 9.918 1.563 62.956
7 12 19.23 -1.25 5.006 -6.257 1.563 25.055
8 18 18.69 4.75 4.466 21.211 22.563 19.941
9 12 7.21 -1.25 -7.015 8.768 1.563 49.203
10 20 42.06 6.75 27.836 187.890 45.563 187.890
... ... ... ... ... ... ... ...
... ... ... ... ... ... ... ...
19 12 7.50 -1.25 -6.725 8.406 1.563 45.219
20 14 8.00 0.75 -6.225 -4.668 0.563 38.744

Total 265 284.49 305.888

Ave 13.25 14.225 15.294
We calculate the squared deviations for all the observations in the sample.

8
 CORRELATION COEFFICIENT: EXAMPLE CALCULATION
Obs S Y S -S Y -Y (S - S)(Y -Y ) (S - S)2 (Y -Y )2

1 15 17.24 1.75 3.016 5.277 3.063 9.093

2 16 15.00 2.75 0.776 2.133 7.563 0.601
3 8 14.91 -5.25 0.686 -3.599 27.563 0.470
4 6 4.50 -7.25 -9.725 70.503 52.563 94.566
5 15 18.00 1.75 3.776 6.607 3.063 14.254
6 12 6.29 -1.25 -7.935 9.918 1.563 62.956
7 12 19.23 -1.25 5.006 -6.257 1.563 25.055
8 18 18.69 4.75 4.466 21.211 22.563 19.941
9 12 7.21 -1.25 -7.015 8.768 1.563 49.203
10 20 42.06 6.75 27.836 187.890 45.563 187.890
... ... ... ... ... ... ... ...
... ... ... ... ... ... ... ...
19 12 7.50 -1.25 -6.725 8.406 1.563 45.219
20 14 8.00 0.75 -6.225 -4.668 0.563 38.744

Total 265 284.49 305.888 217.760 1542.160

Ave 13.25 14.225 15.294 10.888 77.108
To obtain the sample variance of S, we sum its squared deviations and divide by the number
of observations, 20. We do the same for Y.

9
 CORRELATION COEFFICIENT: EXAMPLE CALCULATION
Obs S Y S -S Y -Y (S - S)(Y -Y ) (S - S)2 (Y -Y )2

1 15 17.24 1.75 3.016 5.277 3.063 9.093

2 16 15.00 2.75 0.776 2.133 7.563 0.601
3 8 14.91 -5.25 0.686 -3.599 27.563 0.470
4 6 4.50 -7.25 -9.725 70.503 52.563 94.566
5 15 18.00 1.75 3.776 6.607 3.063 14.254
6 12 6.29 -1.25 -7.935 9.918 1.563 62.956
7 12 19.23 -1.25 5.006 -6.257 1.563 25.055
8 18 18.69 4.75 4.466 21.211 22.563 19.941
9 12 7.21 -1.25 -7.015 8.768 1.563 49.203
10 20 42.06 6.75 27.836 187.890 45.563 187.890
... ... ... ... ... ... ... ...
... ... ... ... ... ... ... ...
19 12 7.50 -1.25 -6.725 8.406 1.563 45.219
20 14 8.00 0.75 -6.225 -4.668 0.563 38.744

Total 265 284.49 305.888 217.760 1542.160

Ave 13.25 14.225 15.294 10.888 77.108
Cov(S,Y) Var(S) Var(Y)
Thus we have the sample covariance and variances.
10
 CORRELATION COEFFICIENT: EXAMPLE CALCULATION
Obs S Y S -S Y -Y (S - S)(Y -Y ) (S - S)2 (Y -Y )2

Cov (S ,Y )  15.294

Var (S )  10.888

Var (Y )  77.108

15.294 10.888 77.108

We save these figures. Cov(S,Y) Var(S) Var(Y)

11
 CORRELATION COEFFICIENT: EXAMPLE CALCULATION
Obs S Y S -S Y -Y (S - S)(Y -Y ) (S - S)2 (Y -Y )2

Cov (S ,Y )  15.294

Var (S )  10.888

Var (Y )  77.108

Cov(S ,Y ) 15.294 15.294

rSY     0.55
Var(S )Var(Y ) 10.888  77.108 28.975

Inserting the appropriate figures, we find that the sample correlation coefficient is 0.55. It is
positive because there is a positive association between S and Y in the sample.

12