Moment Generating Function

Definition Let X = (X1 , . . . , Xn )T be a random vector and

t = (t1 , . . . , tn )T 2 Rn . The moment generating function (MGF) is
STAT/MTHE 353: 5 – Moment Generating
defined by
Functions and Multivariate Normal Distribution T
MX (t) = E et X

for all t for which the expectation exists (i.e., finite).

T. Linder Remarks:
Pn
MX (t) = E e i=1 ti Xi
Queen’s University
For 0 = (0, . . . , 0)T , we have MX (0) = 1.
Winter 2017 If X is a discrete random variable with finitely many values, then
T
MX (t) = E et X is always finite for all t 2 Rn .
We will always assume that the distribution of X is such that
MX (t) is finite for all t 2 ( t0 , t0 )n for some t0 > 0.

STAT/MTHE 353: 5 – MGF & Multivariate Normal Distribution 1 / 34 STAT/MTHE 353: 5 – MGF & Multivariate Normal Distribution 2 / 34

Connection with moments

The single most important property of the MGF is that is uniquely
determines the distribution of a random vector:
Let k1 , . . . , kn be nonnegative integers and k = k1 + · · · + kn . Then
Theorem 1 @k @k
MX (t) = E et1 X1 +···+tn Xn
Assume MX (t) and MY (t) are the MGFs of the random vectors X and @tk11 · · · @tknn @tk11 · · · @tknn
✓ ◆
Y and such that MX (t) = MY (t) for all t 2 ( t0 , t0 )n . Then @k
= E et1 X1 +···+tn Xn
@tk11 · · · @tknn
FX (z) = FY (z) for all z 2 Rn
= E X1k1 · · · Xnkn et1 X1 +···+tn Xn
where FX and FY are the joint cdfs of X and Y .
Setting t = 0 = (0, . . . , 0)T , we get

Remarks: @k
MX (t) = E X1k1 · · · Xnkn
FX (z) = FY (z) for all z 2 R clearly implies MX (t) = MY (t).
n @tk11 · · · @tknn t=0

Thus MX (t) = MY (t) () FX (z) = FY (z)

Most often we will use the theorem for random variables instead of For a (scalar) random variable X we obtain the kth moment of X:
random vectors. In this case, MX (t) = MY (t) for all t 2 ( t0 , t0 )
dk
implies FX (z) = FY (z) for all z 2 R. MX (t) = E Xk
dtk t=0

STAT/MTHE 353: 5 – MGF & Multivariate Normal Distribution 3 / 34 STAT/MTHE 353: 5 – MGF & Multivariate Normal Distribution 4 / 34
 Theorem 2
Assume X 1 , . . . , X m are independent random vectors in Rn and let
X = X 1 + · · · + X m . Then
m
Y
MX (t) = MX i (t)
i=1 Example: MGF for X ⇠ Gamma(r, ) and X1 + · · · + Xm where the Xi
are independent and Xi ⇠ Gamma(ri , ).
Proof:
Example: MGF for X ⇠ Poisson( ) and X1 + · · · + Xm where the Xi
MX (t) = E et
T
X
= E et
T
(X 1 +···+X m ) are independent and Xi ⇠ Gamma( i ). Also, use the MGF to find
T T E(X), E(X 2 ), and Var(X).
= E et X1
· · · et Xm

T T
= E et X1
· · · E et Xm

= MX 1 (t) · · · MX m (t) ⇤

Note: This theorem gives us a powerful tool for determining the

distribution of the sum of independent random variables.

STAT/MTHE 353: 5 – MGF & Multivariate Normal Distribution 5 / 34 STAT/MTHE 353: 5 – MGF & Multivariate Normal Distribution 6 / 34

Theorem 3 Applications to Normal Distribution

Assume X is a random vector in Rn , A is an m ⇥ n real matrix and
b 2 Rm . Then the MGF of Y = AX + b is given at t 2 Rm by Let X ⇠ N (0, 1). Then
Z 1
MY (t) = et
T
b
MX (AT t) 1 2
MX (t) = E(etX ) = etx p e x /2 dx
1 2⇡
Z 1 Z 1 ⇥ ⇤
1 1 2 1 1
(x t)2 t2
Proof: = p e 2 (x 2tx) dx = p e 2 dx
1 2⇡ 1 2⇡
T T Z 1
MY (t) = E et Y
= E et (AX+b) 2 1 1 2
= et /2 p e 2 (x t) dx
= et
T
b T
E et AX = et
T
b
E e(A
T
t)T X 1 2⇡
| {z }
T N (t,1) pdf
= et b
MX (AT t) ⇤
t2 /2
= e

We obtain that for all t 2 R

Note: In the scalar case Y = aX + b we obtain
2
MX (t) = et /2
tb
MY (t) = e MX (at)

STAT/MTHE 353: 5 – MGF & Multivariate Normal Distribution 7 / 34 STAT/MTHE 353: 5 – MGF & Multivariate Normal Distribution 8 / 34
 Moments of a standard normal random variable Sum of independent normal random variables
P1 k
Recall the power series expansion ez = k=0 zk! valid for all z 2 R. Recall that if X ⇠ N (0, 1) and Y = X + µ, where > 0, then
Apply this to z = tX Y ⇠ N (µ, 2 ). Thus the MGF of a N (µ, 2 ) random variable is
✓X 1 ◆ t)2 /2
tX (tX)k MY (t) = etµ MX ( t) = etµ e(
MX (t) = E(e ) = E
k! 2 2

⇥ ⇤
k=0 = etµ+t /2
1
X E (tX) k 1
X E Xk
= = tk
k! k!
k=0 k=0 2
Let X1 , . . . , Xm be independent r.v.’s with Xi ⇠ N (µi , i) and set
X = X1 + · · · + Xm . Then
and to z = t2 /2 m m
X1 i Y Y Pm Pm
t2 /2 t2 /2 MX (t) = MXi (t) = etµi +t
2 2
i /2 = et i=1 µi + i=1
2
i t2 /2
MX (t) = e =
i=0
i! i=1 i=1

Matching the coefficient of tk , for k = 1, 2, . . . we obtain This implies

✓X
m m
X ◆
8 2
> k! X⇠N µi , i
< , k even,
k/2 i=1 i=1
E(X ) = 2 (k/2)!
k
Pm
>
:0, i.e., X1 + · · · + Xm is normal with mean µX = µi and
k odd. Pm i=1
2
variance X = i=1 i2 .
STAT/MTHE 353: 5 – MGF & Multivariate Normal Distribution 9 / 34 STAT/MTHE 353: 5 – MGF & Multivariate Normal Distribution 10 / 34

Multivariate Normal Distributions For any nonnegative definite n ⇥ n matrix C the following hold:

Linear Algebra Review (1) C has n nonnegative eigenvalues 1 , . . . , n (counting multiplicities)

and corresponding n orthogonal unit-length eigenvectors b1 , . . . , bn :
Recall that an n ⇥ n real matrix C is called nonnegative definite if it
is symmetric and Cbi = i bi , i = 1, . . . , n
T n
x Cx 0 for all x 2 R
where bTi bi = 1, i = 1, . . . , n and bTi bj = 0 if i 6= j.
and positive definite if it is symmetric and
(2) (Spectral decomposition) C can be written as
T n
x Cx > 0 for all x 2 R such that x 6= 0
C = BDB T

Let A be an arbitrary n ⇥ n real matrix. Then C = AT A is
nonnegative definite. If A is nonsingular (invertible), then C is
positive definite.
Proof: AT A is symmetric since (AT A)T = (AT )(AT )T = AT A.
Thus it is nonnegative definite since
where D = diag( 1 , . . . , n ) is the diagonal matrix of the
eigenvalues of C, and B is the orthogonal matrix whose ith column
is bi , i.e., B = [b1 . . . bn ].
(3) C is positive definite () C is nonsingular () all the eigenvalues
i are positive

xT (AT A)x = xT AT Ax = (Ax)T (Ax) = kAxk2 0 ⇤

STAT/MTHE 353: 5 – MGF & Multivariate Normal Distribution 11 / 34 STAT/MTHE 353: 5 – MGF & Multivariate Normal Distribution 12 / 34
 (4) C has a unique nonnegative definite square root C 1/2 , i.e., there
exists a unique nonnegative definite A such that
Lemma 4
C = AA
If ⌃ is the covariance matrix of some random vector
X = (X1 , . . . , Xn )T , then it is nonnegative definite.

Proof: We only prove the existence of A. Let

1/2 1/2 Proof: We know that ⌃ = Cov(X) is symmetric. Let b 2 Rn be
D 1/2 = diag( 1 , . . . , n ) and note that D 1/2 D 1/2 = D. Let
arbitrary. Then
A = BD 1/2 B T . Then A is nonnegative definite and

A2 = AA = (BD 1/2 B T )(BD 1/2 B T ) bT ⌃b = bT Cov(X)b = Cov(bT X) = Var(bT X) 0

= BD 1/2 B T BD 1/2 B T = BD 1/2 D 1/2 B T so ⌃ is nonnegative definite

= C ⇤
Remark: It can be shown that an n ⇥ n matrix ⌃ is nonnegative definite
if and only if there exists a random vector X = (X1 , . . . , Xn )T such that
Remarks: Cov(X) = ⌃.
If C is positive definite, then so is A.
If we don’t require that A be nonnegative definite, then in
general there are infinitely many solutions A for AAT = C.
STAT/MTHE 353: 5 – MGF & Multivariate Normal Distribution 13 / 34 STAT/MTHE 353: 5 – MGF & Multivariate Normal Distribution 14 / 34

Defining the Multivariate Normal Distribution Definition Let µ 2 Rn and let ⌃ be an n ⇥ n nonnegative definite
matrix. A random vector X = (X1 , . . . , Xn ) is said to have a
Let Z1 , . . . , Zn be independent r.v.’s with Zi ⇠ N (0, 1). The multivariate normal distribution with parameters µ and ⌃ if its
multivariate MGF of Z = (Z1 , . . . , Zn )T is multivariate MGF is
Pn n
Y T
µ+ 12 tT ⌃t
MZ (t) = E et
T
Z
=E e i=1 ti Zi = E e ti Zi MX (t) = et
i=1
n
Y Pn
2 2 1 T
= eti /2 = e i=1 ti /2 = e2t t

i=1
Notation: X ⇠ N (µ, ⌃).

Remarks:
Now let µ 2 R and A an n ⇥ n real matrix. Then the MGF of
n If Z = (Z1 , . . . , Zn )T with Zi ⇠ N (0, 1), i = 1, . . . , n, then
X = AZ + µ is Z ⇠ N (0, I), where I is the n ⇥ n identity matrix.
T T 1 T
t)T (AT t)
We saw that if Z ⇠ N (0, I), then X = AZ + µ ⇠ N (µ, ⌃), where
MX (t) = et µ
MZ (AT t) = et µ
e 2 (A
⌃ = AAT . One can show the following:
T 1 T
AAT t T
µ+ 12 tT ⌃t
= et µ
e2t = et
X ⇠ N (µ, ⌃) if and only if X = AZ + µ for a random n-vector
T
where ⌃ = AA . Note that ⌃ is nonnegative definite. Z ⇠ N (0, I) and some n ⇥ n matrix A with ⌃ = AAT .

STAT/MTHE 353: 5 – MGF & Multivariate Normal Distribution 15 / 34 STAT/MTHE 353: 5 – MGF & Multivariate Normal Distribution 16 / 34
 Mean and covariance for multivariate normal distribution

Consider first Z ⇠ N (0, I), i.e., Z = (Z1 , . . . , Zn )T , where the Zi are If X ⇠ N (µ, ⌃), then X = AZ + µ for a random n-vector
independent N (0, 1) random variables. Then Z ⇠ N (0, I) and some n ⇥ n matrix A with ⌃ = AAT .

E(Z) = E(Z1 ), . . . , E(Zn )

T
= (0, · · · , 0)T We have
E(AZ + µ) = AE(Z) + µ = µ
and
8 Also,
<1 if i = j,
E (Zi E(Zi ))(Zj E(Zj )) = E(Zi Zj ) = Cov(AZ + µ) = Cov(AZ) = A Cov(Z)AT = AAT = ⌃
:0 if i 6= j
Thus
Thus E(X) = µ, Cov(X) = ⌃

E(Z) = 0, Cov(Z) = I

STAT/MTHE 353: 5 – MGF & Multivariate Normal Distribution 17 / 34 STAT/MTHE 353: 5 – MGF & Multivariate Normal Distribution 18 / 34

Theorem 6
Joint pdf for multivariate normal distribution
If X = (X1 , . . . , Xn )T ⇠ N (µ, ⌃), where ⌃ is nonsingular, then it has a
Lemma 5 joint pdf given by
If a random vector X = (X1 , . . . , Xn )T has covariance matrix ⌃ that is 1 1
2 (x µ)T ⌃ 1
(x µ)
not of full rank (i.e., singular), then X does not have a joint pdf. fX (x) = p e , x 2 Rn
(2⇡)n det ⌃

Proof sketch: If ⌃ is singular, then there exists b 2 Rn such that b 6= 0 Proof: We know that X = AZ + µ where
Pn
and ⌃b = 0. Consider the random variable bT X = i=1 bi Xi : Z = (Z1 , . . . , Zn )T ⇠ N (0, I) and A is an n ⇥ n matrix such that
AAT = ⌃. Since ⌃ is nonsingular, A must be nonsingular with inverse
Var(bT X) = Cov(bT X) = bT Cov(X)b = bT ⌃b = 0
A 1 . Thus the mapping
Therefore P (bT X = c) = 1 for some constant c. If X had a joint pdf h(z) = Az + µ
f (x), then for B = {x : bT x = c} we should have 1
Z Z is invertible with inverse g(x) = A (x µ) whose Jacobian is
T 1
1 = P (b X = c) = P (X 2 B) = · · · f (x1 , . . . , xn ) dx1 · · · dxn Jg (x) = det A
B
By the multivariate transformation theorem
But this is impossible since B is an (n 1)-dimensional hyperplane 1 1
fX (x) = fZ (g(x))|Jg (x)| = fZ A (x µ) | det A |
whose n-dimensional volume is zero, so the integral must be zero. ⇤
STAT/MTHE 353: 5 – MGF & Multivariate Normal Distribution 19 / 34 STAT/MTHE 353: 5 – MGF & Multivariate Normal Distribution 20 / 34
 Proof cont’d: Since Z = (Z1 , . . . , Zn )T , where the Zi are independent Special case: bivariate normal
N (0, 1) random variables, we have For n = 2 we have
" # " #
Yn ✓ ◆ 2
1 zi2 /2 1 1
Pn 2 1 1 T µ1 1 ⇢ 1 2
fZ (z) = p e =p e 2 i=1 zi = p e 2z z µ= and ⌃= 2
i=1
2⇡ (2⇡)n (2⇡)n µ1 ⇢ 1 2 2

2
so we get where µi = E(Xi ), i = Var(Xi ), i = 1, 2, and

fX (x) = fZ A 1 (x µ) | det A 1 | Cov(X1 , X2 )

⇢ = ⇢(X1 , X2 ) =
1 1 1 T 1 1 2
1
= p e 2 (A (x µ)) (A (x µ))
| det A |
(2⇡)n Thus the bivariate normal distribution is determined by five scalar
1 1 T 1 T 1
1 parameters µ1 , µ2 , 12 , 22 , and ⇢.
= p e 2 (x µ) (A ) A (x µ)
| det A |
(2⇡)n
⌃ is positive definite () ⌃ is invertible () det ⌃ > 0:
1 1 T 1
= p e 2 (x µ) ⌃ (x µ)
(2⇡)n det ⌃ det ⌃ = (1 ⇢2 ) 2 2
>0 () |⇢| < 1 and 2 2
>0
1 2 1 2

1
since | det A 1
|= p and (A 1 T
) A 1
=⌃ 1
(exercise!) ⇤ so a bivariate normal random variable (X1 , X2 ) has a pdf if and only if
det ⌃
the components X1 and X2 have positive variances and |⇢| < 1.
STAT/MTHE 353: 5 – MGF & Multivariate Normal Distribution 21 / 34 STAT/MTHE 353: 5 – MGF & Multivariate Normal Distribution 22 / 34

Thus the joint pdf of (X1 , X2 )T ⇠ N (µ, ⌃) is

We have
" # 1 " # 1 1 (x1 µ1 )2
+
(x2 µ2 )2 2⇢(x1 µ1 )(x2 µ2 )
2 2 ⇢2 ) 2 2
1 1 ⇢ 1 2 1 2 ⇢ 1 2 f (x1 , x2 ) = p e 2(1 1 2 1 2
⌃ = 2
= 2 2⇡ 1 2 1 ⇢2
⇢ 1 2 2 det ⌃ ⇢ 1 2 1

and
Remark: If ⇢ = 0, then
T 1
(x µ) ⌃ (x µ) (x1 µ1 )2 (x2 µ2 )2
" #" # 1 1
2 + 2
h i 1 2
⇢ x1 µ1 f (x1 , x2 ) = e2 1 2

= x1 µ1 , x2 µ1 2 1 2 2⇡ 1 2
(1 ⇢2 ) 2 2 2
1 2 ⇢ 1 2 1 x2 µ1 1 (x1 µ1 )2
1 (x2 µ2 )2
2 2
" # = p e · 2 1
p e 2 2

h i 2 1 2⇡ 2 2⇡
1 2 (x1 µ1 ) ⇢ 1 2 (x2 µ2 )
= 2 2 x1 µ1 , x2 µ1 2 = fX1 (x1 )fX2 (x2 )
(1 ⇢2 ) 1 2 1 (x2 µ2 ) ⇢ 1 2 (x1 µ1 )
1 2 Therefore X1 and X2 are independent. It is also easy to see that
= 2 2 2 (x1 µ1 ) 2 2⇢ 1 2 (x1 µ1 )(x2 µ2 ) + 2
1 (x2 µ2 ) 2
(1 ⇢2 ) 1 2 f (x1 , x2 ) = fX1 (x1 )fX2 (x2 ) for all x1 and x2 implies ⇢ = 0. Thus we
✓ ◆
1 (x1 µ1 ) 2 (x2 µ2 ) 2 2⇢(x1 µ1 )(x2 µ2 ) obtain
= 2 + 2
(1 ⇢2 ) 1 2 1 2
Two jointly normal random variables X1 and X2 are independent if and
only if they are uncorrelated.
STAT/MTHE 353: 5 – MGF & Multivariate Normal Distribution 23 / 34 STAT/MTHE 353: 5 – MGF & Multivariate Normal Distribution 24 / 34
 In general, the following important facts can be proved using the Marginals of multivariate normal distributions
multivariate MGF: Let X = (X1 , . . . , Xn )T ⇠ N (µ, ⌃). If A is an m ⇥ n matrix and
b 2 Rm , then
(i) If X = (X1 , . . . , Xn )T ⇠ N (µ, ⌃), then X1 , X2 , . . . Xn are
Y = AX + b
independent if and only if they are uncorrelated, i.e.,
Cov(Xi , Xj ) = 0 if i 6= j, i.e., ⌃ is a diagonal matrix. is a random m-vector. Its MGF at t 2 Rm is
(ii) Assume X = (X1 , . . . , Xn )T ⇠ N (µ, ⌃) and let T
MY (t) = et b
MX (AT t)
X 1 = (X1 , . . . , Xk )T , X 2 = (Xk+1 , . . . , Xn )T T
µ+ 12 ⌧ T ⌃⌧
Since MX (⌧ ) = e⌧ for all ⌧ 2 Rn , we obtain
Then X 1 and X 2 are independent if and only if T
b (AT t)T µ+ 12 (AT t)T ⌃(AT t)
MY (t) = et e
Cov(X 1 , X 2 ) = 0k⇥(n k) , the k ⇥ (n k) matrix of zeros, i.e., ⌃ T
(b+Aµ)+ 12 tT A⌃AT t
can be partitioned as = et
" #
⌃11 0k⇥(n k) This means that Y ⇠ N (b + Aµ, A⌃AT ), i.e., Y is multivariate normal
⌃= with mean b + Aµ and covariance A⌃AT .
0(n k)⇥k ⌃22
Example: Let a1 , . . . , an 2 R and determine the distribution of
where ⌃11 = Cov(X 1 ) and ⌃22 = Cov(X 2 ).
Y = a1 X1 + · · · + an Xn .
STAT/MTHE 353: 5 – MGF & Multivariate Normal Distribution 25 / 34 STAT/MTHE 353: 5 – MGF & Multivariate Normal Distribution 26 / 34

Note the following: 2

3
For some 1  m < n let {i1 , . . . , im } ⇢ {1, . . . , n} such that µ i1
6 . 7
i1 < i2 < · · · < im . Let ej = (0, . . . , 0, 1, 0, . . . , 0)t be the jth unit Aµ = 6 . 7
4 . 5
vector in Rn and define the m ⇥ n matrix A by
µ im
2 3
eTi1 and the (j, k)th entry of A⌃AT is
6 . 7
A=6 4 . 5
. 7
(A⌃AT )jk = A ⇥ (ik th column of ⌃)
eTim j
= (⌃)ij ik = Cov(Xij , Xik )
Then 2 32 3 2 3
eTi X1 X i1
6 .1 7 6 . 7 6 . 7
AX = 6 . 76 . 7 6 . 7
4 . 54 . 5 = 4 . 5 Thus if X = (X1 , . . . , Xn )T ⇠ N (µ, ⌃), then (Xi1 , . . . , Xim )T is mul-
eTim Xn X im tivariate normal whose mean and covariance are obtained by picking out
the corresponding elements of µ and ⌃.
Thus (Xi1 , . . . , Xim )T ⇠ N (Aµ, A⌃AT ). Special case: For m = 1 we obtain that Xi ⇠ N (µi , 2
where
i ),
µi = E(Xi ) and i2 = Var(Xi ), for all i = 1, . . . , n.

STAT/MTHE 353: 5 – MGF & Multivariate Normal Distribution 27 / 34 STAT/MTHE 353: 5 – MGF & Multivariate Normal Distribution 28 / 34
 Conditional distributions
Recall that X = AZ + µ for some Z = (Z1 , . . . , Zn )T where the Zi are
T
Let X = (X1 , . . . , Xn ) ⇠ N (µ, ⌃) and for 1  m < n define independent N (0, 1) random variables and A is such that AAT = ⌃.

X 1 = (X1 , . . . , Xm )T , X 2 = (Xm+1 , . . . , Xn )T Let Z 1 = (Z1 , . . . , Zm )T and Z 2 = (Zm+1 , . . . , Zn )T . We want to

determine such A in a partitioned form with dimensions corresponding to
We know that X 1 ⇠ N (µ1 , ⌃11 ) and X 2 ⇠ N (µ2 , ⌃22 ) where the partitioning of ⌃:
µi = E(X i ), ⌃ii = Cov(X i ), i = 1, 2. " #
B 0m⇥(n m)
Then µ and ⌃ can be partitioned as A=
C D
" # " #
µ1 ⌃11 ⌃12
µ= , ⌃= We can write ⌃ = AAT as
µ2 ⌃21 ⌃22 " # " #" #
⌃11 ⌃12 B 0m⇥(n m) BT CT
where ⌃ij = Cov(X i , X j ), i, j = 1, 2. Note that ⌃11 is m ⇥ m, ⌃22 is =
⌃21 ⌃22 C D 0(n m)⇥m DT
(n m) ⇥ (n m), ⌃12 is m ⇥ (n m), and ⌃21 is (n m) ⇥ m. Also, " #
T T
BB BC
⌃21 = ⌃T12 . =
CB T CC T + DD T
We assume that ⌃11 is nonsingular and we want to determine the
conditional distribution of X 2 given X 1 = x1 .

STAT/MTHE 353: 5 – MGF & Multivariate Normal Distribution 29 / 34 STAT/MTHE 353: 5 – MGF & Multivariate Normal Distribution 30 / 34

Since B is invertible, given X 1 = x1 , we have Z 1 = B 1 (x1 µ1 ). So

We want to solve for B, C and D. First consider BB T = ⌃11 . We
given X 1 = x1 , we have that the conditional distribution of X 2 and the
choose B to be the unique positive definite square root of ⌃11 :
conditional distribution of
1/2
B = ⌃11 CB 1
(x1 µ1 ) + DZ 2 + µ2

Recall that B is symmetric and it is invertible since ⌃11 is. Then are the same.
⌃21 = CB T = implies
But Z 2 is independent of X 1 , so given X 1 = x1 , the conditional
C = ⌃21 (B ) T 1
= ⌃21 B 1 distribution of CB 1 (x1 µ1 ) + DZ 2 + µ2 is the same as its
unconditional distribution.
Then ⌃22 = CC T + DD T gives
We conclude that the conditional distribution of X 2 given X 1 = x1 is
T T 1 1 T multivariate normal with mean
DD = ⌃22 CC = ⌃22 ⌃21 B B (⌃21 )
= ⌃22 ⌃21 (BB) 1
⌃12 = ⌃22 ⌃21 ⌃111 ⌃12 E(X 2 |X 1 = x1 ) = µ2 + CB 1
(x1 µ1 )
1 1
Now note that X = AZ + µ gives = µ2 + ⌃21 B B (x1 µ1 )
= µ2 + ⌃21 ⌃111 (x1 µ1 )
X 1 = BZ 1 + µ1 , X 2 = CZ 1 + DZ 2 + µ2
and covariance matrix ⌃22|1 = DD T = ⌃22 ⌃21 ⌃111 ⌃12
STAT/MTHE 353: 5 – MGF & Multivariate Normal Distribution 31 / 34 STAT/MTHE 353: 5 – MGF & Multivariate Normal Distribution 32 / 34
 Special case: bivariate normal Equivalently, the conditional distribution of X2 given X1 = x1 is
T
Suppose X = (X1 , X2 ) ⇠ N (µ, ⌃) with 2 2
" # " # N µ2 + ⇢ (x1 µ1 ), 2 (1 ⇢2 )
2 1
µ1 1 ⇢ 1 2
µ= and ⌃ = 2
µ1 ⇢ 1 2 2

We have If |⇢| < 1, then the conditional pdf exists and is given by
2
µ2 + ⌃21 ⌃111 (x1 µ1 ) = µ2 + ⇢ (x1 µ1 ) 2
1 x2 µ2 ⇢ 2 (x1 µ1 )

and 1 1
2 (1 ⇢2 )
fX2 |X1 (x2 |x1 ) = p e 2 2

⇢2 2 2
2⇡(1 ⇢2 )
⌃22 ⌃21 ⌃111 ⌃12 = 2
2
1 2
2 = 2
2 (1 ⇢2 ) 2
1
Thus the conditional distribution of X2 given X1 = x1 is normal with
(conditional) mean Remark: Note that E(X2 |X1 = x1 ) = µ2 + ⇢ 2
1
(x1 µ1 ) is a linear
(affine) function of x1 .
2
E(X2 |X1 = x1 ) = µ2 + ⇢ (x1 µ1 ) 2
1 Example: Recall the MMSE estimate problem for X ⇠ N (0, X ) from
2
the observation Y = X + Z, where Z ⇠ N (0, Z ) and X and Z are
and variance
2 independent. Use the above the find g ⇤ (y) = E[X|Y = y] and compute
Var(X2 |X1 = x1 ) = 2 (1 ⇢2 ) ⇥ ⇤
the minimum mean square error E (X g ⇤ (Y ))2 .
STAT/MTHE 353: 5 – MGF & Multivariate Normal Distribution 33 / 34 STAT/MTHE 353: 5 – MGF & Multivariate Normal Distribution 34 / 34