Moment Generating Functions

Continuous Distributions

The Uniform distribution from a to b

 1
 a xb
f  x  b  a
 0 otherwise
0.4
f  x
0.3

0.2


1 
0.1 
ba 

0
0 5 10 x 15
a b
The Normal distribution
(mean m, standard deviation s)
 x  m 2
1 
f  x  e 2s 2

2s
The Exponential distribution

 e  x x0
f  x  
 0 x0

0.2

0.1

0
-2 0 2 4 6 8 10
Weibull distribution with parameters a and b.

a
 xb
Thus F  x  1 e b

a
 xb
and f  x   F   x   a x b 1e b
x0
 The Weibull density, f(x)

0.7
(a = 0.9, b = 2)
0.6

0.5 (a = 0.7, b = 2)
0.4

0.3 (a = 0.5, b = 2)

0.2

0.1

0
0 1 2 3 4 5
 The Gamma distribution
Let the continuous random variable X have
density function:
  a a 1   x
 x e x0
f  x     a 
 x0
 0

Then X is said to have a Gamma distribution

with parameters a and .
Expectation of functions of
Random Variables
X is discrete

E  g  X     g  x  p  x    g  xi  p  xi 
x i

X is continuous


E  g  X     g  x  f  x  dx

Moments of Random Variables
The kth moment of X.

mk  E  X k 

  xk p  x  if X is discrete
 x
 
  x k f  x  dx if X is continuous
-
the kth central moment of X

m  E X  m 

0 k
k
 
   x  m k p  x  if X is discrete
 x
 
   x  m k f  x  dx if X is continuous
-

where m = m1 = E(X) = the first moment of X .

Rules for expectation
Rules:
1. E c   c where c is a constant

2. E  aX  b  aE  X   b where a, b are constants

3. var  X   m  E  X  m  

0 2
2
 
 
 E X   E  X   m2  m12
2 2

4. var  aX  b   a 2 var  X 
Moment generating functions
Moment Generating function of a R.V. X

  etx p  x  if X is discrete
 x
mX  t   E e    
tX

  etx f  x  dx if X is continuous
 
Examples
1. The Binomial distribution (parameters p, n)

mX  t   E etX    etx p  x 
x

n
n x
  e   p 1  p 
tx n x

x 0  x
n t x  n  x n x
 
n n
    e p 1  p      a b
n x

x 0  x  x 0  x 


 a  b  e p 1 p 
n n
t
2. The Poisson distribution (parameter )
x
p  x  e  x  0,1, 2,
x!
The moment generating function of X , mX(t) is:

mX  t   E etX    etx p  x    e
n
 x
tx
e 
x x 0 x!

 e 
x
 t  x
u
e 
  e
e e using e  
t
u

x 0 x! x 0 x !

e

 et 1 
3. The Exponential distribution (parameter )
 e  x x  0
f  x  
 0 x0
The moment generating function of X , mX(t) is:
 
mX  t   E etX    e tx
f  x  dx   e tx
 e  x
dx
 0


  t   x 
 t   x e
  e dx   
0  t   0
 
 t
   t
undefined t  
4. The Standard Normal distribution (m = 0, s = 1)
1  x22
f  x  e
2
The moment generating function of X , mX(t) is:

mX  t   E etX    f  x  dx
e tx



1  x22
 e
tx
e dx
 2

1  x2 22 tx
 
 2
e dx
We will now use the fact that
  xb 2
1  2
 2 a e 2a
dx  1 for all a  0, b

We have
completed
the square
 
1  x2 22 tx 1
mX  t  
t2 x2 2 tx t 2

 dx  e 2 

e e 2 dx
 2  2

t2 1   x2t 2 t2
e 2

 2
e dx  e 2







This is 1
4. The Gamma distribution (parameters a, )
  a a 1   x
 x e x0
f  x     a 
 x0
 0

The moment generating function of X , mX(t) is:


mX  t   E etX    f  x  dx
e tx


 a

  etx xa 1e  x dx
0
 a 

 a a 1   t  x
 x e dx
0
 a 
We use the fact

ba a 1 bx
0   a  x e dx  1 for all a  0, b  0

 a a 1   t  x
mX  t    x e dx
0
 a 

  t
a  a a
 a 1   t  x   
a 
 x e dx   
   t  0  a    t 






Equal to 1
 Properties of
Moment Generating Functions
 1. mX(0) = 1
   
mX  t   E etX , hence mX  0   E e0 X  E 1  1

Note: the moment generating functions of the following

distributions satisfy the property mX(0) = 1

Binomial Dist'n mX  t   e p  1  p 
n
t
i)

ii) Poisson Dist'n mX  t   e


 et 1 
  
iii) Exponential Dist'n mX  t    
 t2  t 
iv) Std Normal Dist'n mX  t   e 2
a
  
v) Gamma Dist'n mX  t    
   t 
 m2 m3 mk
2. mX  t   1  m1t  t 
2
t 
3
 t 
k

2! 3! k!
We use the expansion of the exponential function:
2 3
u u uk
eu  1  u     
2! 3! k!
 
mX  t   E etX
 t2 2 t3 3 tk k 
 E 1  tX  X  X   X  
 2!
2
3!
3
k!
k

t
  t
 1  tE  X   E X  E X   E X 
2!
2

3!
3
 
t
k!
k
 
t2 t3 tk
 1  t m1  m2  m3   mk 
2! 3! k!
 k
d
3. mX  0   k mX  t   m k
k 
dt t 0

Now m 2 2 m3 3 mk k
mX  t   1  m1t  t  t   t 
2! 3! k!
m2 m3 2 mk k 1
mX  t   m1  2t  3t   kt 
2! 3! k!
m3 2 mk k 1
 m1  m2t  t   t 
2!  k  1!
and mX  0   m1
m4 mk
mX  t   m2  m3t  t   t k 2 
2!  k  2 !
and mX  0   m2
continuing we find mX  0  mk
k 
 Property 3 is very useful in determining the moments of a
random variable X.
Examples
i) Binomial Dist'n mX  t    e p  1  p 
n
t

   
n 1
mX  t   n e p  1  p
t
pet
m  0   n  e p  1  p   pe   np  m  m
n 1
0 0
X 1

m  t   np  n  1  e p  1  p   e p  e   e p  1  p  et 
n2 n 1
t t t t
X  
 npe  e p  1  p   n  1  e p    e p  1  p  
t t n2 t t

 npe  e p  1  p   ne p  1  p 
n2
t t t

 np  np  1  p   np  np  q   n 2 p 2  npq  m2
ii) Poisson Dist'n mX  t   e
  et 1 

mX  t   e

 et 1  et   e  e 1t t

 

mX  t    e
   et  1   2e e 1 2t  e e 1t
 et 1 t t t

 

mX  t    e
 
2  e 1  2t
t
 e  2  e 
t  et 1 t et  1
 

 e

2  e 1  2t
t
 e  3  e 
t  et 1 t

 e
 3  e 1 3t
t
  3 e
 
2  e 1  2t
t

 e
 
 et 1 t
To find the moments we set t = 0.

m1  mX  0    e   
 e0 1  0

m2  mX  0    e     e  e 10   2  
2  e 1  0
0 0

m3  mX  0    3e0  3 2e0t  e0   3  3 2  

   
iii) Exponential Dist'n mX  t    
   t 
d   t 
1
d  
mX  t    
dt    t  dt
   1   t   1      t 
2 2

mX  t     2    t   1  2    t 
3 3

mX  t   2  3   t   1  2  3     t 

4 4

mX  t   2  3   4    t 
 4
 1   4!     t 
5 5

mX  t    k !     t 
k   k 1
Thus
1
m1  m  mX  0       
2


2
m2  mX  0   2    
3

2

k!
mk  mX  0    k !    
k   k 1

k
The moments for the exponential distribution can be calculated in an
alternative way. This is note by expanding mX(t) in powers of t and
equating the coefficients of tk to the coefficients in:

m2 m3 mk
mX  t   1  m1t  t 
2
t 
3
 tk 
2! 3! k!
 1 1
mX  t      1  u  u 2  u3 
  t 1 t 1 u

t t2 t3
 1  2  3 
  
Equating the coefficients of tk we get:

mk 1 k!
 or mk 
k!  k
k
The moments for the standard normal distribution

mX  t   e
t2
2

We use the expansion of eu.

 k 2 3
u u u uk
eu    1  u     
k 0 k ! 2! 3! k!
     
2 3 k
t2 t2 t2

mX  t   e  1   
t2 2 2 2
2 t2
2     
2! 3! k!
1 4 1 6 1 2k
 1 2 t  2 t  3 t 
1 2
 k t 
2 2! 2 3! 2 k!
We now equate the coefficients tk in:
m2 mk m2 k
mX  t   1  m1t  t 
2
 t  k
 t 
2k

2! k!  2k  !
If k is odd: mk = 0.

m2 k 1
For even 2k:  k
 2k  ! 2 k !
or

m2 k  k
2k !
2 k!

2! 4!
Thus m1  0, m2   1, m3  0, m4  2 3
2 2  2!
 Summary

Moments
Moment generating functions
Moments of Random Variables
mk  E  X k 
  xk p  x  if X is discrete
 x
 
  x k f  x  dx if X is continuous
-
The moment generating function
  etx p  x  if X is discrete
 x
mX  t   E etX    
  etx f  x  dx if X is continuous
 
Examples
1. The Binomial distribution (parameters p, n)
n x
p  x     p 1  p 
n x
x  0,1, 2, ,n
 x

mX  t   e p  1  p   e p  q 
n n
t t

2. The Poisson distribution (parameter )

x
p  x  e  x  0,1, 2,
x!
mX  t   e
 
 et 1
3. The Exponential distribution (parameter )
 e  x x  0
f  x  
 0 x0
 
 t
mX  t      t
undefined t  

4. The Standard Normal distribution (m = 0, s = 1)

1  x22
f  x  e
2
mX  t   e
t2
2
 5. The Gamma distribution (parameters a, )
  a a 1   x
 x e x0
f  x     a 
 x0
 0
a
  
mX  t    
   t 
6. The Chi-square distribution (degrees of freedom n)
(a  n/2,   1/2
   2 n 1  1 x
n
1
 2 x2 e 2 x  0
f  x      n2 

 0 x0
n
 12  2
mX  t    1   1  2t 
n2

 2 t 
Properties of Moment Generating Functions

1. mX(0) = 1
m2 m3 mk
2. mX  t   1  m1t  t 
2
t 
3
 tk 
2! 3! k!
k
d
3. mX  0   k mX  t   m k
k 
dt t 0

i.e. mX  0   m1
mX  0   m2
mX  0   m3 , etc
The log of Moment Generating Functions

Let lX (t) = ln mX(t) = the log of the moment generating

function

Then l X  0   ln mX  0   ln1  0
1 mX  t  mX  0 
l X  t   mX  t   l X  0    m1  m
mX  t  mX  t  mX  0 

mX  t  mX  t    mX  t  
2

l X  t  
 mX  t  
2

mX  0  mX  0    mX  0  
2

l X  0    m2   m1   s 2
2

 mX  0  
2
Thus lX (t) = ln mX(t) is very useful for calculating the
mean and variance of a random variable

1. l X  0   m
2. l X  0   s 2
Examples
1. The Binomial distribution (parameters p, n)

mX  t   e p  1  p   e p  q 
n n
t t

l X  t   ln mX  t   n ln  e p  q 
t

1
m  l X  0  n
1
l X  t   n t et p p  np
e pq pq

l X  t   n
 
et p et p  q  et p et p  
e p  q
2
t

p  p  q  p  p
s  l X  0   n
2
 npq
 p  q
2
2. The Poisson distribution (parameter )

mX  t   e
 
 et 1

l X  t   ln mX  t    et  1  
l X  t   et m  l X  0   

l X  t   e t s 2  l X  0   
3. The Exponential distribution (parameter )
 
 t
mX  t      t
undefined t  

l X  t   ln mX  t   ln   ln    t  if t  
1
l X  t      t 
1

 t
1
l X  t   1   t   1 
2

  t 
2

1 1
Thus m  l X  0   and s  l X  0  
2

 2
4. The Standard Normal distribution (m = 0, s = 1)
mX  t   e
t2
2

l X  t   ln mX  t   t2
2

l X  t   t , l X  t   1

Thus m  l X  0   0 and s 2  l X  0   1

5. The Gamma distribution (parameters a, )
a
  
mX  t    
   t 

l X  t   ln mX  t   a ln   ln    t  
 1  a
l X  t   a   
 t   t
a
l X  t   a  1   t   1 
2

  t 
2

a a
Hence m  l X  0   and s  l X  0   2
2

 
6. The Chi-square distribution (degrees of freedom n)
mX  t   1  2t 
n2

n
l X  t   ln mX  t    ln 1  2t 
2
n 1 n
l X  t     2  
2 1  2t 1  2t
2n
l X  t   n  11  2t   2  
2

1  2t 
2

Hence m  l X  0   n and s 2  l X  0   2n

 Summary of Discrete Distributions
Moment
generating
Name probability function p(x) Mean Variance function MX(t)
Discrete 1 N+1 N2-1 et etN-1
Uniform p(x) = N x=1,2,...,N 2 12 N et-1
Bernoulli p x=1 p pq q + pet
p(x) = q x=0

Binomial N Np Npq (q + pet)N
p(x) =  x  pxqN-x
Geometric p(x) =pqx-1 x=1,2,... 1 q pet
p p2 1-qet
Negative  x-1  k kq  pet  k
Binomial p(x) =  k-1  pkqx-k p p2  
1-qet
x=k,k+1,...
Poisson x -   e(e -1)
t
p(x) = x! e x=1,2,...
Hypergeometric  A  N-A  A A  AN-n not useful
   n N n N 1-NN-1
 x  n-x       
p(x) =
N
 
n
 Summary of Continuous Distributions
probability Moment generating
Name density function f(x) Mean Variance function MX(t)
Continuous  1 a+b (b-a)2 ebt-eat
Uniform a xb 2
f ( x)   b  a 12 [b-a]t
0 otherwise
Exponentia le  lx x  0 1 1   
l f ( x)    2   t  for t < 
0 x0
Gamma  la a a a
 x a 1e lx x  0   
f(x) = f(x)   G ( a )  2   t  for t < 
0 x0
2 (1/2)n/2 n/21 -(1/2)x n 2n  1 
n/2
x e x? 0 1-2t for t < 1/2
f(x) =  (n/2)  
n d.f. 0 x<0
Normal 1 2 2 m s2 etm+(1/2)t s
2 2
f(x) = e-(x-m) /2s
2 s
Weibull   1 -x/  (
2/
)
+1  2 not
f(x) = 
x e x? 0 

2/
 ( )-[( )]
+2

+1


 avail.
0 x<0