Random variables

Van Nam Tran1

1
University of Technology and Education

March 10, 2020

Definition (Covariances)
The Covariance of two random variables X and Y is defined as

Cov[X, Y ] = E[(X − E[X])(Y − E[Y ])].

If Cov[X, Y ] > 0 then there exists some positive linear relationship between X
and Y so that when X is above (below) average then Y tends to be above
(below) average.

Example
If X is ice cream consumption and Y is the cream consumption would be
associated with above average temperature.

If Cov[X, Y ] < 0, there exists some negative linear relationship between X and
Y so that when X is above (below) average then Y tends to be below (above)
average.
For example, if X is ice cream consumption and Y is rainfall, one would
expect that Cov[X, Y ] < 0, that is above (below) average ice cream
consumption would be associated with below (above) average rainfall.
Van Nam Tran Random variables January, 2018 2/ 15
Definition
The correlation coefficient ρ is

Cov[X, Y ]
ρ= p
V ar[X]V ar[Y ]

Remark
The sign of ρ is always the same as the sign of Cov[X, Y ].

Van Nam Tran Random variables January, 2018 3/ 15

Independence

X and Y is called independent if and only if for all functions f (x) and g(x) we
have
E[f (X)]g(Y )] = E[f (X)]E[G(Y )]

Remark (Meaning of independence)

Two random variables are independent if the realization of one does not affect
the probability distribution of the other.
In discrete random variables we have
P r[X = xi , Y = yj ] = P r[X = xi ]P r[Y = yj ]

Van Nam Tran Random variables January, 2018 4/ 15

Properties of Covariance

If X and Y are independent then Cov[X, Y ] = 0.

If Cov[X, Y ] = 0 it does not follow that X and Y are independent.

If c is a nonrandom constant then Cov[X, c] = 0.

Cov[X, X] = V ar[X]

Cov[X, Y ] = E[XY ] − E[X]E[Y ]

E[XY ] = E[X]E[Y ] + Cov[X, Y ]

E[XY ] = E[X]E[Y ] iff Cov[X, Y ] = 0.

Van Nam Tran Random variables January, 2018 5/ 15

Linear Combinations of random variables

Given two random variables X1 , X2 . Suppose

Y = aX1 + bX2 + c.

Then

E[Y ] = aE[X1 ] + bE[X2 ] + c

V ar[aX1 + bX2 ] = a2 V ar[X1 ] + 2abCov[X1 , X2 ] + b2 V ar[X2 ]

The matrix  
V ar[X1 ] Cov[X1 , X2 ]
 
Cov[X1 , X2 ] V ar[X2 ]
is positive semi-definite.

Van Nam Tran Random variables January, 2018 6/ 15

If V ar[X1 ] 6= 0 and V ar[X2 ] 6= 0 then − ≤ ρ ≤ 1.

If ρ = 1 then X2 = αX1 + β where α > 0

If ρ = −1 then X2 = αX1 + β where α < 0

Van Nam Tran Random variables January, 2018 7/ 15
The Capital Asset Pricing Model (CAPM)

Van Nam Tran Random variables January, 2018 7/ 15

Sum of uncorrelated random variables

Definition
Random variables X1 , X2 , . . . , Xn are called uncorrelated if Cov[Xi , Xj ] = 0
for all i 6= j. otherwise we say that X1 , X2 , . . . , Xn are correlated

Remark
If X1 , X2 , . . . , Xn are independent then they are uncorrelated.

Theorem
Given that X1 , X2 , . . . , Xn are uncorrelated it follows that if

Y = a1 X1 + a2 X2 + · · · + an Xn

then
V ar[Y ] = a21 V ar[X1 ] + a22 V ar[X2 ] + · · · + a2n V ar[Xn ].
Van Nam Tran Random variables January, 2018 8/ 15
The sample mean

Consider n uncorrelated random variables X1 , X2 , . . . , Xn with the following

properties:

A1. E[Xi ] = µ for i = 1, 2 . . . , n.

A2. V ar[Xi ] = σ 2 for i = 1, 2, . . . , n

A3. Cov[Xi , Xj ] = 0 for i 6= j

Definition
An estimator θ̂ of a parameter θ is said to be unbiased if

E[θ̂] = θ

Van Nam Tran Random variables January, 2018 9/ 15

Example
n
1 1X
Let X̄ = (X1 + x2 + · · · + Xn ) = Xi (sample mean) as an estimator of
n n i=1
the unknown parameter µ. Then E[X̄] = µ if A1 is satisfied.
If A1, A2, A3 are satisfied,
σ2
V ar[X̄] = .
n

Van Nam Tran Random variables January, 2018 10/ 15

The linear Regression Model for Beginner

The classical linear regression model with a constant and one regressor
satisfies the following assumptions:

A1. Yi = α + βXi + ei for i = 1, 2, . . . , n

A2. E[ei ] = 0

A3. V ar[ei ] = σ 2 for i = 1, 2, . . . , n.

A4. Cov[ei , ej ] = 0 for i 6= j

Xn
A5. Xi is nonrandom with (Xj − X̄)2 6= 0.
j=1

Van Nam Tran Random variables January, 2018 10/ 15

The least square estimator of β is then
n
X
(Xi − X̄)(Yi − Ȳ ) n
i=1
X
β̂ = n = ai Yi
X
2 i=1
(Xj − X̄)
j=1

Xi − X̄
where ai = n .
X
2
(Xj − X̄)
j=1

Theorem
β̂ is an unbiased estimator of β or E[β̂] = β.

The variance of β̂ is given by

σ2
V ar[β̂] = n
X
(Xj − X̄)2
j=1

Van Nam Tran Random variables January, 2018 11/ 15

Linear Combinations of Correlated random variables

Let X1, X2 , . . . , Xn be correlated random variables and

n
X
Y = a1 X1 + a2 X2 + · · · + an Xn = ai Xi .
i=1

Then
n
X
E[Y ] = ai E[Xi ]
i=1

n X
X n
V ar[Y ] = ai aj Cov[Xi , Xj ]
i=1 j=1
Xn n
X n
X
= a2i V ar[Xi ] + 2 ai aj Cov[Xi , Xj ]
i=1 i=1 j=i+1

Van Nam Tran Random variables January, 2018 12/ 15

  
X1
 X2 
If X =  .  then V ar[X] is defined as
 
 .. 
Xn

V ar[X] =E (X − E[X])(X − E[X])T = E[XX T ] − E[X]E[X]T

 
 
V ar[X1 ] Cov[X1 , X2 ] Cov[X1 , X3 ] · · · Cov[X1 , Xn ]
 Cov[X1 , X2 ] V ar[X2 ] Cov[X2 , X3 ] · · · Cov[X2 , Xn ]
=
 
.. .. .. .. .. 
 . . . . . 
Cov[X1 , Xn ] Cov[X2 , Xn ] Cov[X3 , Xn ] · · · V ar[Xn ]

Theorem
If Y = aT X then
V ar[aT X] = aT V ar[X]a ≥ 0.

If Y = AX then
V ar[AX] = AV ar[X]AT .

Van Nam Tran Random variables January, 2018 13/ 15

The linear regression model for Experts

The classical linear regression model is defined by the following assumptions

A1: Y = βX + e where Y is an n × 1 random vector, X is n × p, β is a p × 1
random vector and e is an n × 1 random vector.
A2. E[e] = 0
A3. V ar[e] = σ 2 I where I is an n × n identity matrix.
A4. X is nonrandom
  with  rank[X]
 = p.  
Y1 1 X1 e1
 Y2  1 X2     e2 
 α
Note that A1.  .  =  . + . 
    
 ..   ..  β  .. 


Yn 1 Xn en
It turn out that β̂ = (X T X)−1 X T Y . It then can be show that β̂ = β + Ae
where A = (X T X)−1 X T is nonrandom by A4.

Van Nam Tran Random variables January, 2018 14/ 15

Theorem
E[β̂] = β.

V ar[β̂] = σ 2 (X T X)−1

Van Nam Tran Random variables January, 2018 15/ 15