Appendix C

Standard Statistical Results

C.1 THE LAW OF LARGE NUMBERS
Let x 1 , x 2 , . . . be a sequence of independent, identically distributed random
variables (I I D), each with expected value µ and variance σ 2 . Define the
sequence of averages
n
x
i=1 i x + x2 + . . . + x n
yn = = 1 , n = 1, 2, . . . .
n n

Then the law of large numbers states that yn converges to µ as n → ∞, that is,
V ar[ yn ] → 0.
The mean of yn is
1 1
E[ yn ] = (E[x 1 ] + E[x 2 ] + + . . . + E[x n ]) = nµ = µ.
n n

For the variance of yn , we have

 n n n
x σ2

1 1  1
 
i=1 i
V ar[ yn ] = V ar = 2 V ar  x i  = 2 V ar [x i ] = 2 nσ 2 = ,
n n   n n n
i=1  i=1

where we have used the fact that the variance of the sum of independent random
variables is the sum of their variances, see Appendix C.2.
We have therefore shown that as n → ∞, V ar[ yn ] → 0.

C.2 THE CENTRAL LIMIT THEOREM

Let x 1 , x 2 , . . . be a sequence of I I D random variables, each with expected value
µ and variance σ 2 . If we define ui = x i − µ, then
 
E [ui ] = E [u] = 0, V ar[ui ] = E u2 = σ 2 .

Let
n

sn = ui + nµ.
i=1

So
n

sn = xi .
i=1
Computational Finance Using C and C#: Derivatives and Valuation. DOI: 10.1016/B978-0-12-
803579-5.00019-X 315
Copyright © 2016 Elsevier Ltd. All rights reserved.
316 APPENDIX | C Standard Statistical Results

We now introduce the normalized value z n , as follows:

n
s n − nµ 1 
zn = √ = √ ui .
σ n σ n i=1

The central limit theorem states that as n tends to infinity the probability
distribution of z n tends to a normal distribution with zero mean and unit
variance, mathematically z n → N(0, 1) as n → ∞.

Proof. From Appendix C.5 equation (C.5.3),

n
 t  
  
Mz n = E [exp (t z n )] = E exp  ,
 
√ ui 
 σ n i=1  
  

and using (C.5.5)

   n
t
Mz n = Mu √ .
σ n
Equation (C.5.1) then yields
   2  
t t 1 t
Mu √ ≈1+ √ E [u] + √ E u2 + +.
σ n σ n 2 σ n
√
As n → ∞ t/(σ n) → 0, an
2  
t2
  
t t 1 t
Mu √ →1+ √ E [u] + √ E u2 = 1 + .
σ n σ n 2 σ n 2n

Thus,
 n n
t2 t2
  
t
Mu √ = 1+ →1+ as n → ∞,
σ n 2n 2
where we have use the fact that t << 1 see Grimmett and Welsh (1986).
We have therefore shown that as n → ∞

t2 t2
Mz n (t) → 1 + →e2 .
2

However from Appendix D.1, the moment generating function Mz (t) for a
standard normal distribution (µ = 0, σ 2 = 1) is

t2
Mz (t) = e 2 , where z ∼ N(0, 1).

Thus, we have proved that z n → N(0, 1) as n → ∞.

 The Variance and Covariance of Random Variables Section| C.3 317

C.3 THE VARIANCE AND COVARIANCE OF RANDOM

VARIABLES
C.3.1 Variance
One Variable
Let X be a variate from a given distribution, and Z be the following linear
function of this variate:
Z = a + bX,
where a and b are constants. Then

E[Z] = E[a] + E[bX] = a + bE[X]

and
V ar[Z] = E[(Z − E[Z])2 ]
= E[(a + bX − a − bE[X])2 ]
= E[(bX − bE[X])2 ]
= E[b2 (X − E[X])2 ]
= b2 E[(X − E[X])2 ].

Therefore, the mean is a + bE[X], and the variance is b2 V ar[X].

Two Variables
Let Z = a + b1 X1 + b2 X2 , where a, b1 , and b2 are constants.
Then mean is E[Z] = E[a] + E[b1 X1 ] + E[b2 X2 ] = a + b1 E[X1 ] + b2 E[X2 ].
The variance V ar[Z] is computed as follows:
 
V ar[Z] = E {a + b1 X1 + b2 X2 − a − b1 E[X1 ] − b2 E[X2 ]}2
 
= E {b1 (X1 − E[X1 ]) + b2 (X2 − E[X2 ])}2
   
= b21 E (X1 − E[X1 ])2 + b22 E (X2 − E[X2 ])2
   
+ 2b1 b2 E (X1 − E[X1 ]) E (X2 − E[X2 ])

= b21 V ar[X1 ] + b22 V ar[X2 ] + 2b1 b2 Cov[X1 , X2 ],

where Cov[X1 , X2 ] is the covariance between X1 and X2 . If X1 and X2 are I I D

random variables, then Cov [X1 , X2 ] = 0, and we thus have
V ar[Z] = b21 V ar[X1 ] + b22 V ar[X2 ].

Three Variables
Let Z = a + b1 X1 + b2 X2 = b3 X3 , where a, b1 , b2 , and b3 are constants.
318 APPENDIX | C Standard Statistical Results

Then mean is E[Z] = E[a] + E[b1 X1 ] + E[b2 X2 ] + E[b3 X3 ] = a + b1 E[X1 ] +

b2 E[X2 ] + b3 E[X3 ].
The variance V ar[Z] is computed as follows:

 
V ar[Z] = E {a + b1 X1 + b2 X2 + b3 X3 − a − b1 E[X1 ] − b2 E[X2 ] − b3 E[X3 ]}2
 
= E {b1 (X1 − E[X1 ]) + b2 (X2 − E[X2 ]) + b3 (X3 − E[X3 ])}2
     
= b21 E (X1 − E[X1 ])2 + b22 E (X2 − E[X2 ])2 + b23 E (X3 − E[X3 ])2
   
+ 2b1 b2 E (X1 − E[X1 ]) E (X2 − E[X2 ])
   
+ 2b2 b3 E (X2 − E[X2 ]) E (X3 − E[X3 ])
   
+ 2b1 b3 E (X1 − E[X1 ]) E (X3 − E[X3 ])

= b21 V ar[X1 ] + b22 V ar[X2 ] + b23 V ar[X2 ] + 2b2 b3 Cov[X2 , X3 ]

+ 2b1 b2 Cov[X2 , X3 ] + 2b1 b3 Cov[X1 , X3 ].

If X1 , X2 , and X3 are I I D, all the covariance terms are zero and the variance is

V ar[Z] = b21 V ar[X1 ] + b22 V ar[X2 ] + b23 V ar[X3 ].

Variance of n Variables
We will now derive an expression for the sum of n I I D random variables.
Let Z = a + i=1 bi X i , where a and bi ,i = 1, . . . , n are constants.
n

Then we have E[Z] = E[a] + E =a+

n n
b X
i=1 i i
b E[X i ]
i=1 i
and

  n n 2
  
V ar[Z] = E   +
 
 a b i X i − a − b i E[X i ]  
 
  i=1 i=1  
  n  n 2 
  

= E  

 bi X i − bi E[X i ] 
 
  i=1 i=1  
  n 2
  
= E  
 
 bi (X i − E[X i ])  
  i=1  
n    n n
= b2i E (X i − E[X i ])2 +


bi b j E (X i − E[X i ]) X j − E[X j ]
i=1 i=1 j=1( j,i)
n
 n
 n

= b2i V ar[X i ] + bi b j Cov[X i , X j ].
i=1 i=1 j=1( j,i)
 The Variance and Covariance of Random Variables Section| C.3 319

As before if all the X variables are I I D, then the covariance terms are zero
and we have
n

V ar [Z] = b2i V ar[x i ].
i=1

If in addition all the bi terms are one and all the X variable have variance σ 2 ,
we obtain
n

V ar [Z] = V ar[x i ] = nσ 2 .
i=1

C.3.2 Covariance
The covariance between two variables X and Y is defined by
   
Cov[X,Y ] = E (X − E[X])(Y − E[Y ]) = E XY − Y E[X] − X E[Y ] + E[X]E[Y ]

= E[XY ] − E[Y ]E[X] − E[X]E[Y ] + E[X]E[Y ]

= E[XY ] − E[X]E[Y ].

By symmetry, it can be seen that Cov[X,Y ] = Cov[Y, X].

Two Variables
Let Z1 = a + bX and Z2 = c + dY , where a, b, c, and d are constants.
We have

Cov[Z1 , Z2 ] = Cov[a + bX, c + dY ]

= E[(a + bX)(c + dY )] − E[(a + bX)]E[(c + dY )]
 
= E[ac + bcX + adY + bdXY ] − (a + bE[X])(c + dE[Y ])

= ac + bcE[X] + adE[Y ] + bdE[XY ] − ac

+ bcE[X] − adE[Y ] − bdE[X]E[Y ]
 
= bd E[XY ] − E[X]E[Y ] ,

∴ Cov[Z1 , Z2 ] = bdCov[X,Y ].

Three Variables
Let Z1 = a + b1 X1 + b2 X2 and Z2 = c + dY , where a, b1 , b2 , c, and d are
constants.
We have

Cov[Z1 , Z2 ] = Cov[a + b1 X1 + b2 X2 , c + dY ]
= E[(a + b1 X1 + b2 X2 )(c + dY )] − E[(a + b1 X1 + b2 X2 )]E[(c + dY )]
= E[(a + b1 X1 )(c + dY ) + b2 X2 (c + dY )]
 
− E[(a + b1 X1 )] + E[b2 X2 ]E[c + dY ]
320 APPENDIX | C Standard Statistical Results

= E[(a + b1 X1 )(c + dY )] + E[b2 X2 (c + dY )]

− E[(a + b1 X1 )]E[c + dY ] − E[b2 X2 ]E[c + dY ]
 
= E[(a + b1 X1 )(c + dY )] − E[(a + b1 X1 )]E[c + dY ]
 
− E[(b2 X2 )(c + dY )] − E[b2 X2 ]E[c + dY ] ,

∴ Cov[Z1 , Z2 ] = b1 dCov[X1 ,Y ] + b2 dCov[X2 ,Y ].

Four Variables
Let Z1 = a + b1 X1 + b2 X2 + b3 X3 and Z2 = c + dY , where a, b1 , b2 , b3 , c, and
d are constants.
We have
Cov[Z1 , Z2 ] = Cov[a + b1 X1 + b2 X2 + b3 X3 , c + dY ]
= E[(a + b1 X1 + b2 X2 + b3 X3 )(c + dY )]
− E[(a + b1 X1 + b2 X2 + b3 X3 )]E[(c + dY )]
= E[(a + b1 X1 + b2 X2 )(c + dY ) + b3 X3 (c + dY )]
 
− E[(a + b1 X1 + b2 X2 )] + E[b3 X3 ]E[c + dY ]

= E[(a + b1 X1 + b2 X2 )(c + dY )] + E[b3 X3 (c + dY )]

− E[(a + b1 X1 + b2 X2 )]E[c + dY ] − E[b3 X3 ]E[c + dY ]
 
= E[(a + b1 X1 + b2 X2 )(c + dY )] − E[(a + b1 X1 + b2 X2 )]E[c + dY ]
 
− E[(b3 X3 )(c + dY )] − E[b3 X3 ]E[c + dY ]

= Cov[(a + b1 X1 + b2 X2 ), c + dY ] + Cov[b3 X3 , c + dY ],
∴ Cov[Z1 , Z2 ] = b1 dCov[X1 ,Y ] + b2 dCov[X2 ,Y ] + b3 dCov[X3 ,Y ].

Covariance of n Variables
In a similar manner to that outlined above,
 n  n
Cov a + bi X i , c + dY  = d bi Cov[X i ,Y ].
 
i=1 i=1

For the most general case, let Z1 = a + i=1 bi X i and Z2 = c + M

n 
j=1 b j Yj .
So
 n
 M
   n
 M
 
Cov[Z1 , Z2 ] = Cov a + bi X i , c + d j Yj = Cov a + bi X i , c + d j Yj .
i=1 j=1 i=1 j=1

So
 n
 M
  n  M
 
Cov a + bi X i , c + d j Yj = Cov bi X i , d j Yj
i=1 j=1 i=1 j=1
 Conditional Mean and Covariance of Normal Distributions Section| C.4 321

n
   M 
= bi Cov X i , d j Yj
i=1 j=1
n
 M
 
= bi Cov d j Yj , X i ,
i=1 j=1
n 
 M
 
∴ Cov[Z1 , Z2 ] = bi d j Cov[Yj , X i ] .
i=1 j=1

C.3.3 Covariance Matrix

Let X denote the n element vector containing the random variates X i ,i =
1, .. . , n. The mean
 and variance of the ith variate are then E[X i ] and
E (X i − E[X i ])2 respectively. The covariance Cov[X]i j between the ith and


jth variates is E[(X i − E[X i ]) X j − E[X j ] ]. The elements of n by n covariance
matrix Cov[X] are then
Cov[X]i j = E[(X i − E[X i ]) X j − E[X j ] ], i = 1, . . . , n, j = 1, . . . , n.


(C.3.1)
We will now show that Cov[X + A] = Cov[X] where A is an n element vector
containing the constants Ai ,i = 1, . . . , n. Since E[X i + Ai ] = E[X i ] + Ai , we
obtain
V ar [(X + A)i ] = V ar [X i + Ai ]
   
= E (X i + Ai − E[X i + Ai ])2 = E (X i − E[X i ])2
and
Cov [X + A]i j = E[(X i + Ai − E[X i + Ai ]) X j + A j − E[X j + A j ] ]

= E (X i − E[X i ]) X j − E[X j ]



= Cov[X]i j . (C.3.2)

C.4 CONDITIONAL MEAN AND COVARIANCE OF NORMAL

DISTRIBUTIONS
Let X = [X1 /X2 ] be distributed as N p (µ, Σ) with µ = [µ1 /µ2 ], and Σ =
[(Σ11 |Σ12 )/(Σ21 |Σ22 )], and |Σ22 | > 0.
We will prove that the conditional distribution of X1 , given that X2 = x 2 , is
normal and has
Mean = µ1 + Σ12 Σ22 −1
(x 2 − µ2 ), and covariance = Σ11 − Σ12 Σ22
−1
Σ21 .
−1
Let the inverse of Σ be Σ , where

Σ11 Σ12 +
Σ−1 = *. /. (C.4.1)
21 22
,Σ Σ -
322 APPENDIX | C Standard Statistical Results

So Σ−1 Σ = I p , where I p represents the p × p unit matrix, and

*. Σ11 Σ12 +/ *. Σ11 Σ12 +/ = *. Iq 0 +/ . (C.4.2)

21 22
, Σ Σ - , Σ21 Σ22 - , 0 I p−q -
Multiplying out these matrices yields the following equations:

Σ11 Σ11 + Σ21 Σ21 = Iq , (C.4.3)

Σ Σ11 + Σ Σ22 = 0,
21 22
(C.4.4)
Σ Σ12 + Σ Σ22 = 0,
11 12
(C.4.5)
Σ Σ12 + Σ Σ22 = I p−q .
21 22
(C.4.6)

Multiplying equation (C.4.5) on the left by (Σ11 )−1 and on the right by Σ22
−1 gives

(Σ11 )−1 Σ12 = −Σ12 Σ22

−1
. (C.4.7)

Multiplying equation (C.4.3) on the left by (Σ11 )−1 yields

Σ11 + (Σ11 )−1 Σ12 Σ21 = (Σ11 )−1 , (C.4.8)

and substituting for (Σ11 )−1 Σ12 from equation (C.4.7) into equation (C.4.8) gives

(Σ11 )−1 = Σ11 − Σ12 Σ22

−1
Σ21 . (C.4.9)

The joint probability density function of x is

 
1
f (x) = (2π)−p/2
|Σ| −1/2
exp − (x − µ) Σ (x − µ) ,
T −1
2

writing x, µ, and Σ−1 in their partitioned form and expanding gives


1
f (x) = (2π)−p/2 |Σ|−1/2 exp − (x 1 − µ1 )T Σ11 (x 1 − µ1 )
2

+2(x 1 − µ1 )T Σ12 (x 2 − µ2 ) + (x 2 − µ2 )T Σ22 (x 2 − µ2 ) . (C.4.10)

The conditional distribution of x 1 given the value of x 2 is thus obtained by

dividing this density by the marginal density of x 2 and treating x 2 as constant in
the resulting expression. The only portion of the resultant that is not constant is
the portion involving terms in x 1 . It can easily be shown that
 
1
f (x 1 |x 2 ) ∝ exp − (x 1 − µ1 )T Σ11 (x 1 − µ1 ) + 2(x 1 − µ1 )T Σ12 (x 2 − µ2 ) ,
2

where the constant of proportionality is obtained using f (x 1 |x 2 )dx 1 = 1.
 Moment Generating Functions Section| C.5 323

If we let G = (x 1 − µ1 )T Σ11 (x 1 − µ1 ) + 2(x 1 − µ1 )T Σ12 (x 2 − µ2 ), we then

obtain

G = (x 1 − µ1 )T Σ11 (x 1 − µ1 ) + (x 1 − µ1 )T Σ12 (x 2 − µ2 )
+ (x 2 − µ2 )T Σ21 (x 1 − µ1 ),
 T  
G = x 1 − µ1 + (Σ11 )−1 Σ12 (x 2 − µ2 ) Σ11 x 1 − µ1 + (Σ11 )−1 Σ12 (x 2 − µ2 )
− (x 2 − µ2 )T Σ21 (Σ12 )−1 (x 2 − µ2 ), (C.4.11)

where for instance we have used the fact that the scalar quantity
 
(x 1 − µ1 )T Σ12 (x 2 − µ2 ) = (x 2 − µ2 )T Σ21 (x 1 − µ1 ).

Since the last term in equation (C.4.11) only involves constants (as far as
f (x 1 |x 2 ) is concerned), it follows that

1 T
f (x 1 |x 2 ) ∝ exp − x 1 − µ1 + (Σ11 )−1 Σ12 (x 2 − µ2 )
2
 
× Σ11 x 1 − µ1 + (Σ11 )−1 Σ12 (x 2 − µ2 ) ,

which is the density of a multivariate normal distribution that has a mean of

µ1 − (Σ11 )−1 Σ12 (x 2 − µ2 ), which from equation (C.4.7) can be expressed as
µ1 + Σ12 Σ22
−1
(x 2 − µ2 ). The covariance matrix is (Σ11 )−1 , which from equation
−1
(C.4.9) can be written as Σ11 − Σ12 Σ22 Σ21 .

C.5 MOMENT GENERATING FUNCTIONS

If x is a random variable with probability distribution f x (x), then the moment
generating function Mx (t) is defined by
 ∞
Mx (t) = E e t x = e t x f x (x)dx.
 
−∞

We can expand the above expression as follows:

 
1
E e t x = E 1 + t x + (t x)2 + + ,
 
2
1  
Mx (t) = 1 + tE [x] + t 2 E x 2 + +. (C.5.1)
2
Now
d k (Mx (t)) dk   t x 
 k tx 
d e  
= E e = E = E x k et x .
dt k dt k dt k
For t = 0, we thus have
d k (Mx (t)) d k (Mx (0)) 
k 0
  
= = E x e = E xk . (C.5.2)
dt k t=0 dt k
324 APPENDIX | C Standard Statistical Results

Moment generating function of a linear function of a random variable x

If the random variable y is defined as y = ax+b, then the moment generating
function of y, M y (t) is obtained as follows:

M y (t) = Ma x+b (t) = E e t y = E e at x+bt = e bt E e t x .

     

Therefore,
M y (t) = e bt Mx (at). (C.5.3)
Moment generating function of a linear combination of random variables
Let z = x + y where x and y are independent random variables then
Mz (t) = E e t z = E e x+y = E e t x e t y .
     

Since x and y are independent,

E e t x e t y = E e t x E e t y = Mx (t)M y (t).
     

More generally, if s n = x i where x i ,i = 1, . . . , n are independent variables,

n
i=1
then
n

Ms n (t) = Mx i (t). (C.5.4)
i=1
If x i ,i = 1, . . . , n are I I D, then we have
 n 
  
n
Ms n (t) = E exp *.t x i +/ = E e t x = (Mx (t))n . (C.5.5)
 , i=1 - 