Chapter6 (Multivariate Normal Distribution)

QMT581: MULTIVARIATE METHODS
CHAPTER 6: Multivariate Normal

Distribution
Ida Rosmini Othman

rosmini@tmsk.uitm.edu.my
@ y
Department of Statistics Ida Rosmini Othman 1
Multivariate Normal Density and Properties
UNIVARIATE NORMAL DISTRIBUTION
X ~ N ( μ ,σ 2 )
1
e −[( x − μ ) / σ ]
2
f ( x) = /2
−∞ < x < ∞
2πσ 2
1
THE EXPONENTIAL PART
⎛ x−μ ⎞
2
⎟ = ( x − μ )(σ ) ( x − μ )
2 −1
⎜
⎝ σ ⎠
⇓
(x − μ )' Σ −1 (x − μ)
p-DIMENSIONAL NORMAL DENSITY
X ~ N p (μ, Σ)
1 − ( x −μ )' Σ −1
( x −μ ) / 2
f ( x) = e
( 2π ) p / 2 Σ
1/ 2
−∞ < xi < ∞, i = 1, 2, L, p
2
CONTOURS
Note that contours of constant density are
x values that satisfy
ellipsoids defined by the x-
(x − μ)' Σ −1 (x-μ ) = c 2
These ellipsoids are centered at μ and have axes

± c λi ei where λ1,…,λλp are the eigenvalues of Σ
and e1,…,ep are the corresponding eigenvectors.
EXAMPLE 4.1: BIVARIATE NORMAL
μ1 = E ( X 1 ), μ 2 = E ( X 2 )
σ 11 = Var( X 1 ), σ 22 = Var( X 2 )
( )
ρ12 = σ 12 / σ 11 σ 22 = Corr( X 1 , X 2 )
⎡σ σ 12 ⎤ −1 1 ⎡ σ 22 − σ 12 ⎤
Σ = ⎢ 11 ⎥ ,Σ = ⎢− σ
⎣σ 21 σ 22 ⎦ σ 11σ 22 − σ 122 ⎣ 12 σ 11 ⎥⎦
σ 11σ 22 − σ 122 = σ 11σ 22 (1 − ρ122 )
3
EXAMPLE 4.1: SQUARED DISTANCE
(x − μ )' Σ −1 (x − μ)
1
= [ x1 − μ1 , x2 − μ 2 ]
σ 11σ 22 (1 − ρ122 )
⎡ σ 22 − ρ12 σ 11 σ 22 ⎤ ⎡ x1 − μ1 ⎤
⎢ ⎥⎢ ⎥
⎢⎣− ρ12 σ 11 σ 22 σ 11 ⎥⎦ ⎣ x2 − μ 2 ⎦
1 ⎡⎛ x − μ ⎞ 2 ⎛ x − μ ⎞ 2 ⎛ x1 − μ1 ⎞⎛ x2 − μ 2 ⎞⎤
= ⎢⎜ 1 1 ⎟ +⎜ 2 2 ⎟ − 2 ρ12 ⎜ ⎟⎜ ⎟⎥
1 − ρ122 ⎢⎜⎝ σ 11 ⎟⎠ ⎜⎝ σ 22 ⎟⎠ ⎜ σ ⎟⎜ σ ⎟⎥
⎝ 11 ⎠⎝ 22 ⎠
⎣ ⎦
EXAMPLE 4.1: DENSITY FUNCTION
1
f ( x1 , x2 ) =
2π σ 11σ 22 (1 − ρ122 )
2 2
1 ⎛ x1 − μ1 ⎞ ⎛ x2 − μ 2 ⎞
exp{− [⎜ ⎟ +⎜ ⎟
2(1 − ρ12 ) ⎜⎝ σ 11 ⎟⎠ ⎜⎝ σ 22 ⎟⎠
2
⎛ x1 − μ1 ⎞⎛ x2 − μ 2 ⎞
− 2 ρ12 ⎜ ⎟⎜ ⎟]}
⎜ σ ⎟⎜ σ ⎟
⎝ 11 ⎠⎝ 22 ⎠
4
EXAMPLE 4.1: BIVARIATE

DISTRIBUTION
σ11 = σ22, ρ12 = 0
EXAMPLE 4.1: BIVARIATE

DISTRIBUTION
σ11 = σ22, ρ12 = 0.75
5
EXAMPLE 4.2: BIVARIATE CONTOUR
Bivariate normal, σ 11 = σ 22
eigenvalues and eigenvectors
1 1
λ1 = σ 11 + σ 12 , e1 ' = [ , ]
2 2
1 −1
λ2 = σ 11 − σ 12 , e 2 ' = [ , ]
2 2
EXAMPLE 4.2: POSITIVE CORRELATION
6
Properties Of The Multivariate Normal Density
RESULT 4.2 & 4.3
Let X ~ N(μ,
(μ Σ))
Linear combinations of the components of X are
normally distributed.
If X ~ Np(μ, Σ), then every linear combination
a’X + d is distributed as N1(a’μ + d, a’Σa). Converse
is also true.
Iff X ~ Np(μ,
( Σ)) andd A andd d are q x p matrix
i andd q x
1 vector, respectively, then AX + d is distributed as
Nq(Aμ + d, AΣA,).
RESULT 4.2 & 4.3: Example 4.4
7
RESULT 4.4
All subsets of the components

p of X have a MVN
distribution
If X ~ Np(μ, Σ) and we partition X, μ, and Σ as
⎡X ⎤ ⎡μ ⎤ ⎡Σ Σ12 ⎤
X = ⎢ 1 ⎥, μ = ⎢ 1 ⎥, Σ = ⎢ 11
⎣X2 ⎦ ⎣μ 2 ⎦ ⎣Σ 21 Σ 22 ⎥⎦
Where X1 is q x 1, then X1 ~ Nq(μ1, Σ11). (It is also

true that X2 ~ Np-q(μ2, Σ22))
RESULT 4.4: Example 4.5
8
RESULT 4.5
Zero covariance implies that the corresponding
components of X are distributed independently.
Suppose X1 and X2 are q1 x 1 and q2 x 1,
respectively, and
⎡ X1 ⎤ ⎛ ⎡ μ1 ⎤ ⎡ Σ11 Σ12 ⎤ ⎞
⎢ X ⎥ ~ N q1+ q 2 ⎜⎜ ⎢μ ⎥, ⎢Σ ⎟
Σ 22 ⎥⎦ ⎟⎠
⎣ 2⎦ ⎝ ⎣ 2 ⎦ ⎣ 21
Then X1 and X2 are independent iff Σ12 = 0.
A related l If X1 ~ Nq1(μ
l d result: ( 1, Σ11) andd X2 ~ Nq2(μ
( 2,
Σ22) and X1 and X2 are independent, then
⎡ X1 ⎤ ⎛ ⎡ μ1 ⎤ ⎡Σ11 0 ⎤ ⎞
⎢ X ⎥ ~ N q1+ q 2 ⎜⎜ ⎢μ ⎥, ⎢ 0 Σ ⎥ ⎟⎟
⎣ 2⎦ ⎝⎣ 2⎦ ⎣ 22 ⎦ ⎠
RESULT 4.5: Example 4.6
9
RESULT 4.6
The conditional distribution of any

y subset of
components of X given any other disjoint subset
of components is MVN
More precise: With X partitioned as in Property 3,
the conditional distribution of X1, given X2 = x2, is
MVN, with
9 Mean: μ1 + Σ12 Σ −221 (x 2 − μ 2 )
9 Covariance matrix: Σ11 − Σ12 Σ −221Σ 21
RESULT 4.7
Certain q
quadratic forms in a MVN vector have a
chi-square distribution.
More precise: If X ~ Np(μ, Σ) then
(a) ( X − μ)' Σ −1 ( X − μ) : χ p2
(b) The probability inside the solid ellipsoid
{x : ( X − μ))' Σ −1 ( X − μ) ≤ χ p2 (α )} is 1 − α ,
where χ p2 (α ) denotes the upper (100α )th
percentile of the χ p2 distribution
10
χ2 DISTRIBUTION
X 1 : N ( μ1 , σ 12 ), X 2 : N ( μ 2 , σ 22 ), L,
X i − μi
X ν : N ( μν , σ ν2 ); Z i = : N (0,1)
σi
2
⎛ x − μi ⎞
ν
χ = ∑ ⎜⎜ i
2
⎟⎟ , ν : degrees of freedom (d.f.)
i =1 ⎝ σ i ⎠
⎧
⎪
1
f n ( χ ) = ⎨ 2 Γ(n / 2)
2 n/2
χ ( )
2 n / 2 −1 − χ 2 / 2
e ,χ2 > 0
⎪⎩ 0, χ2 ≤ 0
(Gamma distribution with α = n / 2 − 1, β = 2)
χ2 DISTRIBUTION CURVE
1
EXERCISE
Exercise 4.1, 4.3, 4.4(a), 4.19(a)
MLE ofThe Parameters of Np(μ, Σ) And Their Properties
MLEs
Suppose we have a random sampleX1, … Xn from
Σ) How should we estimate μ and Σ?
(μ Σ).
Np(μ,
MLE: maximizing the joint density function of
the random sample with respect to the
parameters.
Joint density (likelihood function):
n
⎧Joint density of ⎫ 1 − ∑ ( x j −μ )'Σ −1 ( x j −μ ) / 2

⎨ ⎬=
j =1
e
X
⎩ 1 2
, X , L , X n ⎭ (2π ) Σ
np / 2 n/2
2
MLEs: FACT 1
The unique MLEs, μ̂ and Σ̂ , in this setting
are
n −1
μˆ = X Σˆ = S n = S
n
MLEs: FACT 2
In this setting X and S are sufficient
t ti ti
statistics.
3
RECALL UV RESULT
Distribution of sample mean
X : N ( μ , σ 2 / n)
Distribution of sample variance
(n − 1) s 2 = ∑ (X j − X ) : σ 2 χ n2−1
n
2
j =1
n
(n − 1) s = σ
2 2
∑Z
j =1
2
j , σZ j : N (0, σ 2 )
MLEs: FACT 3
Taking X1, … Xn to be a random sample
f
from Np(μ, Σ) we hhave
( Σ),
X ~ N p (μ, Σ / n)
(n-1)S ~ Wishart with n – 1 df and

covariance parameter Σ, i.e. Wn-1 (Σ)
X and S are independent
4
WISHART DISTRIBUTION
The Wishart distribution is defined as
ffollows:
ll
If Z1, … Zm are iid Np(0, Σ), then the
distribution of
∑
m
j =1
Z j Z 'j
is called the Wishart distribution with m df

and covariance (matrix) parameter Σ.
MLEs: FACT 4 (A CLT)

Let X1, … Xn be a random sample from ANY p-
variate distribution with mean μ and covariance
matrix Σ. Then for large n – p,
n ( X − μ)
is distributed approximately as Np(0, Σ) and
) S −1 ( X − μ)
n( X − μ)'
is distributed approximately as χ p
2
5
Assessing the Assumption of Univariate Normal
Q-Q PLOT
Known as normal probability plot.
Plot of the sample quantiles versus the
quantiles of a standard normal distribution.
“Straightness” of the points in all plots
indicate conformance to a MVN
assumption.
assumption
Q-Q PLOT
Details of construction:
Order the original observations to get x(1),
x(2), …, x(n), and their corresponding
probability values ( j − 1 ) / n
2
Calculate the standard normal quantiles q(1),
q(2), …, q(n).
Plot the pairs of observations (q(1), x(1)), (q(2),
x(2)), …, (q(n), x(n)) and examine the
“straightness” of the outcome.
1
EXAMPLE 4.9
Given the following data
1.71, 1.26, -1.00, -0.10, 2.30, 0.16, 1.54,
0.80, 0.41, 0.62
Construct the Q-Q plot and comment on its
appearance.
EXAMPLE 4.9
2
EXAMPLE 4.9
MEASUREMENT OF STRAIGHTNESS
Q-Q plot is only a graphical diagnostic for

UNV
UNV.
It would be good to have an objective
measure of “straightness” to accompany
the plot.
Such measure is the correlation between
the x(j)’s and q(j)’s:
3
Assessing the Assumption of Normal
MEASUREMENT OF STRAIGHTNESS
∑ (x
j =1
( j) − x )(q( j ) − q )
rQ =
n n
∑ (x
j =1
( j) − x) 2
∑ (q
j =1
( j) − q )2
Reject
j UNV at significan
g ce level α if rQ < rQ,
Q α ,n ,
where the critical values rQ < rQ,α ,n are found in

Table 4.2 of the textbook.
TABLE 4.2
4
EXAMPLE 4.11
Calculate the correlation coefficient rQ
ffrom th Q Q plot
the Q-Q l t off Example
E l 4.9
4 9 andd test
t t
for normality.
Assessing the Assumption of Bivariate Normal
QUADRATIC FORMS
Check if roughly 50% of sample observations

lie in the ellipse given by
{all x such that (x − x)' S −1
(x − x) ≤ χ 22 (0.5) }
5
EXAMPLE 4.12
Exercise 1.4
EXAMPLE 4.12
⎡155.60 ⎤ ⎡7476.45 303.62 ⎤

x=⎢ ⎥ , S=⎢ ⎥
⎣ 14.70 ⎦ ⎣ 303.62 26.19 ⎦
χ 22 (0.5) = 1.39
'
⎡ x − 155.60 ⎤ ⎡ .000253 −.002930 ⎤ ⎡ x1 − 155.60 ⎤
d =⎢ 1
2
⎥ ⎢ ⎥⎢ ⎥
⎣ x2 − 14.70 ⎦ ⎣ −.002930 .072148 ⎦ ⎣ x2 − 14.70 ⎦
[ x1, x2 ] ' = [[108.28,17.05]] ⇒ d 2 = 1.61 > 1.39
Four out of 9 observations are with d 2 < 1.39
Greater than 50% ⇒ reject bivariate normality
However, sample size (n = 10) is too small to reach the conclusion
1
Assessing the Assumption of Multivariate Normal
CHI-SQUARE PLOT
Use the same idea as in Quadratic Forms,
bbutt applies
li it to
t the
th entire
ti p x 1 vector
t off
observed variables and incorporates the Q-
Q plot idea.
CHI-SQUARE PLOT
d 2 = ( x − x ) S −1 ( x − x ) : squared distance
Order the squared distance d (21) ≤ d (22) ≤ L ≤ d (2n )
1 1
qc , p (( j − ) / n) : 100( j − ) / n quantile of the
2 2
chi - square distribution with p degrees of freedom
1
Graph all (qc , p (( j − ) / n), d (2j ) )
2
The plot should resemble a straight line through
the origin having slope 1
1 1
Note that qc , p (( j − ) / n) = χ p2 (1 − ( j − ) / n)
2 2
2
EXAMPLE 4.13
Construct a chi-square plot of the
li d distances
generalized di t i Example
in E l 4.12.
4 12
EXAMPLE 4.13: CHI-SQUARE PLOT FOR

EXAMPLE 4.12
1
EXAMPLE 4.13: CHI-SQUARE PLOT FOR

EXAMPLE 4.12
CHI-SQUARE PLOT FOR COMPUTER

GENERATED 4-VARIATE NORMAL DATA
1
CLASS EXERCISE
Exercise 4.23, 4.26

Chapter6 (Multivariate Normal Distribution)

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Chapter6 (Multivariate Normal Distribution)

Uploaded by

Copyright:

Available Formats

QMT581: MULTIVARIATE METHODS

CHAPTER 6: Multivariate Normal

Ida Rosmini Othman

Department of Statistics Ida Rosmini Othman 1

Multivariate Normal Density and Properties

UNIVARIATE NORMAL DISTRIBUTION

Department of Statistics Ida Rosmini Othman 2

THE EXPONENTIAL PART

Department of Statistics Ida Rosmini Othman 3

Multivariate Normal Density and Properties

p-DIMENSIONAL NORMAL DENSITY

Department of Statistics Ida Rosmini Othman 4

These ellipsoids are centered at μ and have axes

Department of Statistics Ida Rosmini Othman 5

Multivariate Normal Density and Properties

EXAMPLE 4.1: BIVARIATE NORMAL

Department of Statistics Ida Rosmini Othman 6

EXAMPLE 4.1: SQUARED DISTANCE

Department of Statistics Ida Rosmini Othman 7

Multivariate Normal Density and Properties

EXAMPLE 4.1: DENSITY FUNCTION

Department of Statistics Ida Rosmini Othman 8

EXAMPLE 4.1: BIVARIATE

σ11 = σ22, ρ12 = 0

Department of Statistics Ida Rosmini Othman 9

Multivariate Normal Density and Properties

EXAMPLE 4.1: BIVARIATE

σ11 = σ22, ρ12 = 0.75

Department of Statistics Ida Rosmini Othman 10

EXAMPLE 4.2: BIVARIATE CONTOUR

Department of Statistics Ida Rosmini Othman 11

Multivariate Normal Density and Properties

EXAMPLE 4.2: POSITIVE CORRELATION

Department of Statistics Ida Rosmini Othman 12

RESULT 4.2 & 4.3

Department of Statistics Ida Rosmini Othman 13

Properties Of The Multivariate Normal Density

RESULT 4.2 & 4.3: Example 4.4

Department of Statistics Ida Rosmini Othman 14

All subsets of the components

Where X1 is q x 1, then X1 ~ Nq(μ1, Σ11). (It is also

Department of Statistics Ida Rosmini Othman 15

Properties Of The Multivariate Normal Density

RESULT 4.4: Example 4.5

Department of Statistics Ida Rosmini Othman 16

Department of Statistics Ida Rosmini Othman 17

Properties Of The Multivariate Normal Density

RESULT 4.5: Example 4.6

Department of Statistics Ida Rosmini Othman 18

The conditional distribution of any

9 Covariance matrix: Σ11 − Σ12 Σ −221Σ 21

Department of Statistics Ida Rosmini Othman 19

Properties Of The Multivariate Normal Density

Department of Statistics Ida Rosmini Othman 20

Properties Of The Multivariate Normal Density

Department of Statistics Ida Rosmini Othman 22

Department of Statistics Ida Rosmini Othman 23

MLE ofThe Parameters of Np(μ, Σ) And Their Properties

⎧Joint density of ⎫ 1 − ∑ ( x j −μ )'Σ −1 ( x j −μ ) / 2

Department of Statistics Ida Rosmini Othman 24

Department of Statistics Ida Rosmini Othman 25

MLE ofThe Parameters of Np(μ, Σ) And Their Properties