Chapter4 Statistical Hypothesis Testing

Chapter 4: Statistical Hypothesis Testing
Christophe Hurlin
November 20, 2015
Christophe Hurlin () Advanced Econometrics - Master ESA November 20, 2015 1 / 225
Section 1
Introduction
Christophe Hurlin (University of Orléans) Advanced Econometrics - Master ESA November 20, 2015 2 / 225
1. Introduction
The outline of this chapter is the following:

Section 2. Statistical hypothesis testing
Section 3. Tests in the multiple linear regression model
Subsection 3.1. The Student test
Subsection 3.2. The Fisher test
Section 4. MLE and Inference
Subsection 4.1. The Likelihood Ratio (LR) test
Subsection 4.2.The Wald test
Subsection 4.3. The Lagrange Multiplier (LM) test
1. Introduction
References
Amemiya T. (1985), Advanced Econometrics. Harvard University Press.
Greene W. (2007), Econometric Analysis, sixth edition, Pearson - Prentice

Hil (recommended)
Ruud P., (2000) An introduction to Classical Econometric Theory, Oxford
University Press.
1. Introduction
Notations: In this chapter, I will (try to...) follow some conventions of
notation.
fY ( y ) probability density or mass function

FY ( y ) cumulative distribution function
Pr () probability
y vector
Y matrix
Be careful: in this chapter, I don’t distinguish between a random vector

(matrix) and a vector (matrix) of deterministic elements (except in section
2). For more appropriate notations, see:
Abadir and Magnus (2002), Notation in econometrics: a proposal for a
standard, Econometrics Journal.
Section 2
Statistical hypothesis testing
2. Statistical hypothesis testing
Objectives
The objective of this section is to de…ne the following concepts:
1 Null and alternative hypotheses

2 One-sided and two-sided tests
3 Rejection region, test statistic and critical value
4 Size, power and power function
5 Uniformly most powerful (UMP) test
6 Neyman Pearson lemma
7 Consistent test and unbiased test
8 p-value
Introduction
1 A statistical hypothesis test is a method of making decisions or a

rule of decision (as concerned a statement about a population
parameter) using the data of sample.
2 Statistical hypothesis tests de…ne a procedure that controls (…xes)
the probability of incorrectly deciding that a default position (null
hypothesis) is incorrect based on how likely it would be for a set of
observations to occur if the null hypothesis were true.
Introduction (cont’d)
In general we distinguish two types of tests:
1 The parametric tests assume that the data have come from a type
of probability distribution and makes inferences about the parameters
of the distribution
2 The non-parametric tests refer to tests that do not assume the data
or population have any characteristic structure or parameters.
In this course, we only consider the parametric tests.
A statistical test is based on three elements:
1 A null hypothesis and an alternative hypothesis

2 A rejection region based on a test statistic and a critical value
3 A type I error and a type II error

De…nition (Hypothesis)
A hypothesis is a statement about a population parameter. The formal
testing procedure involves a statement of the hypothesis, usually in terms
of a “null” or maintained hypothesis and an “alternative,” conventionally
denoted H0 and H1 , respectively.
Introduction
1 The null hypothesis refers to a general or default position: that there

is no relationship between two measured phenomena or that a
potential medical treatment has no e¤ect.
2 The costs associated to the violation of the null must be higher than
the cost of a violation of the alternative.
Example (Choice of the null hypothesis)

In a credit scoring problem, in general we have: H0 : the client is not
risky(acceptance of the loan) versus H1 : the client is risky (refusal of the
loan).
De…nition (Simple and composite hypotheses)

A simple hypothesis speci…es the population distribution completely. A
composite hypothesis does not specify the population distribution
completely.
Example (Simple and composite hypotheses)

If X t (θ ) , H0 : θ = θ 0 is a simple hypothesis. H1 : θ > θ 0 , H1 : θ < θ 0 ,
and H1 : θ 6= θ 0 are composite hypotheses.
De…nition (One-sided test)

A one-sided test has the general form:
H0 : θ = θ 0 or H0 : θ = θ 0
H1 : θ < θ 0 H1 : θ > θ 0
De…nition (Two-sided test)

A two-sided test has the general form:
H0 : θ = θ 0
H1 : θ 6 = θ 0

De…nition (Rejection region)

The rejection region is the set of values of the test statistic (or
equivalently the set of samples) for which the null hypothesis is rejected.
The rejection region is denoted W. For example, a standard rejection
region W is of the form:
W = fx : T (x ) > c g
or equivalently
W = fx1 , .., xN : T (x1 , .., xN ) > c g
where x denotes a sample fx1 , .., xN g , T (x ) the realisation of a test
statistic and c the critical value.
Remarks
1 A (hypothesis) test is thus a rule that speci…es:
1 For which sample values the decision is made to ”fail to reject H0” as
true;
2 For which sample values the decision is made to ”reject H0”.
3 Never say "Accept H1", "fail to reject H1" etc..
2 The complement of the rejection region is the non-rejection region.
Remark
The rejection region is de…ned as to be:
W = fx : T (x ) 7 c g
|{z}
| {z }
test statistic critical value
T (x ) is the realisation of the statistic (random variable):
T (X ) = T (X1 , .., XN )
The test statistic T (X ) has an exact or an asymptotic distribution D

under the null H0 .
d
T (X ) D or T (X ) ! D
H0 H0

Decision
Fail to reject H0 Reject H0
Truth H0 Correct decision Type I error
H1 Type II error Correct decision
De…nition (Size)
The probability of a type I error is the (nominal) size of the test. This is
conventionally denoted α and is also called the signi…cance level.
α = Pr ( Wj H0 )
Remark
For a simple null hypothesis:
α = Pr ( Wj H0 )
For a composite null hypothesis:
α = sup Pr ( Wj H0 )
θ 0 2H 0
A test is said to have level if its size is less than or equal to α.
De…nition (Power)
The power of a test is the probability that it will correctly lead to
rejection of a false null hypothesis:
power = Pr ( Wj H1 ) = 1 β
where β denotes the probability of type II error, i.e. β = Pr W H1 and

W denotes the non-rejection region.
Example (Test on the mean)

Consider a sequence X1 , .., XN of i.i.d. continuous random variables with
Xi N m, σ2 where σ2 is known. We want to test
H0 : m = m 0
H1 : m = m 1
with m1 < m0 . An econometrician propose the following rule of decision:
W = fx : x N < c g
where X N = N 1 ∑N i =1 Xi denotes the sample mean and c is a constant

(critical value). Question: calculate the size and the power of this test.
Solution
The rejection region is W= fx : x N < c g . Under the null H0 : m = m0 :
σ2
XN N m0 ,
H0 N
So, the size of the test is equal to:
α = Pr ( Wj H0 )
= Pr X N < c H0
XN m0 c m
= Pr p < p 0 H0
σ/ N σ/ N
c m
= Φ p0
σ/ N
Solution (cont’d)
The rejection region is W= fx : x N < c g . Under the alternative
H1 : m = m 1 :
σ2
XN N m1 ,
H1 N
So, the power of the test is equal to:
power = Pr ( Wj H1 )
XN m1 c m
= Pr p < p 1 H1
σ/ N σ/ N
c m
= Φ p1
σ/ N
Solution (cont’d)
In conclusion:
c m
α=Φ p0
σ/ N
m c
β=1 p1
power = 1 Φ
σ/ N
We have a system of two equations with three parameters: α, β (or power)
and the critical value c.
1 There is a trade-o¤ between the probabilities of the errors of type I
and II, i.e. α and β : if c decreases, α decreases but β increases.
2 A solution is to impose a size α and determine the critical value and
the power.
Solution (cont’d)
In order to illustrate the tradeo¤ between α and β given the critical value
c, take an example with σ2 = 1 and N = 100:
H0 : m = m0 = 1.2 H1 : m = m1 = 1
σ2 σ2
XN N m0 , XN N m1 ,
H0 N H1 N
We have
W = fx : x N < c g
c m
α = Pr ( Wj H0 ) = Φ p0 = Φ (10 (c 1.2))
σ/ N
c m
β = Pr W H1 = 1 Φ p1 =1 Φ (10 (c 1))
σ/ N
4.5
Density under H0
4 Density under H1
3.5
α=Pr(W[H0)=5% β=1-Pr(W|H1)=36.12%
2.5
1.5
0.5
0
0.7 0.8 0.9 1 1.1 1.2 1.3 1.4 1.5 1.6
Click me!
Fact (Critical value)

The (nominal) size α is …xed by the analyst and the critical value is
deduced from α.

Xi N m, σ2 , N = 100 and σ2 = 1 . We want to test
H0 : m = 1.2 H1 : m = 1
An econometrician propose the following rule of decision:
W = fx : x N < c g
where X N = N 1 ∑N i =1 Xi denotes the sample mean and c is a constant

(critical value). Questions: (1) what is the critical value of the test of
size α = 5%? (2) what is the power of the test?
Solution
We know that:
m c
p0
α = Pr ( Wj H0 ) = Φ
σ/ N
So, the critical value that corresponds to a signi…cance level of α is:
σ
c = m0 + p Φ 1
(α)
N
NA: if m0 = 1.2, m1 = 1, N = 100, σ2 = 1 and α = 5%, then the

rejection region is
W = fx : x N < 1.0355g
Solution (cont’d)
σ
W= x : x N < m0 + p Φ 1
(α)
N
The power of the test is:
c m
power = Pr ( Wj H1 ) = Φ p1
σ/ N
Given the critical value, we have:
m0 m
power = Φ p 1 +Φ 1
(α)
σ/ N
NA: if m0 = 1.2, m1 = 1, N = 100, σ2 = 1 and α = 5%:
1.2 1
power = Φ p +Φ 1
(0.05) = 0.6388
1/ 100

Xi N m, σ2 with σ2 = 1 and N = 100. We want to test
H0 : m = 1.2 H1 : m = 1
The rejection region for a signi…cance level α = 5% is:
W = fx : x N < 1.0355g
where X N = N 1 ∑N i =1 Xi denotes the sample mean. Question: if the

realisation of the sample mean is equal to 1.13, what is the conclusion of
the test?
Solution (cont’d)
For a nominal size α = 5%, the rejection region is:
W = fx : x N < 1.0355g
If we observe
x N = 1.13
This realisation does not belong to the rejection region:
xN 2
/W
For a level of 5%, we do not reject the null hypothesis H0 : m = 1.2.
De…nition (Power function)

In general, the alternative hypothesis is composite. In this case, the power
is a function of the value of the parameter under the alternative.
power = P (θ ) 8 θ 2 H1

H0 : m = m 0
H1 : m < m 0
Consider the following rule of decision:
σ
W= x : x N < m0 + p Φ 1
(α)
N
Questions: What is the power function of the test?
Solution
As in the previous case, we have:
m0 m
power = P (m ) = Φ p +Φ 1
(α) with m < m0
σ/ N
NA: if m0 = 1.2, N = 100, σ2 = 1 and α = 5%.
1.2 m
P (m ) = Φ 1.6449 with m < m0
1/10
Power function P (m )
1.2
0.8
0.6
0.4
0.2
0
0.7 0.8 0.9 1 1.1 1.2
Example (Power function)

Consider a test H0 : θ = θ 0 versus H1 : θ 6= θ 0 , the power function has this
general form:
De…nition (Most powerful test)

A test (denoted A) is uniformly most powerful (UMP) if it has greater
power than any other test of the same size for all admissible values of the
parameter.
αA = αB = α
βA βB
for any test B of size α.
UMP tests
How to derive the rejection region of the UMP test of size α ?
=> the Neyman–Pearson lemma
Lemma (Neyman Pearson)

Consider a hypothesis test between two point hypotheses H0 : θ = θ 0 and
H1 : θ = θ 1 . The uniformly most powerful (UMP) test has a rejection
region de…ned by:
LN (θ 0 ; x )
W = xj <K
LN (θ 1 ; x )
where LN (θ 0 ; x ) denotes the likelihood of the sample x and K is a
constant determined by the size α such that:
LN (θ 0 ; X )
Pr < K H0 =α
LN (θ 1 ; X )

H0 : m = m 0
H1 : m = m 1
with m1 > m0 . Question: What is the rejection region of the UMP test of
size α?
Solution
Since X1 , .., XN are N .i.d. m , σ2 , the likelihood of the sample
fx1 , .., xN g is de…ned as to be (cf. chapter 2):
!
1 1 N
2σ2 i∑
2
LN (θ; x ) = exp (xi m)
σN (2π )N /2 =1
Given the Neyman Pearson lemma the rejection region of the UMP test of
size α is given by:
LN (θ 0 ; x )
<K
LN (θ 1 ; x )
where K is a constant determined by the size α.
Solution (cont’d)
1
exp 1
2σ2 ∑N
i =1 (xi m0 ) 2
σN (2π )N /2
<K
1
exp 1
2σ2 ∑N
i =1 (xi m1 ) 2
σN (2π )N /2
This expression can rewritten as:
1
exp ∑N
i =1 (xi m1 ) 2 ∑N
i =1 (xi m0 ) 2 <K
2σ2
() ∑Ni=1 (xi m1 ) 2 ∑N
i =1 (xi m0 ) 2 < K 1
where K1 = 2σ2 ln (K ) is a constant.
Solution (cont’d)
∑N
i =1 (xi m1 ) 2 ∑N
i =1 (xi m 0 ) 2 < K1
() 2 (m0 m1 ) ∑ N 2
i =1 xi + N m1 m02 < K1
() (m0 m1 ) ∑ N
i =1 xi < K2
where K2 = K1 N m12 m02 /2 is a constant.
Solution (cont’d)
( m0 m1 ) ∑ N
i =1 xi < K2
Since m1 > m0 , we have

1 N
∑ xi > K3
N i =1
where K3 = K2 / (N (m0 m1 )) is a constant.
The rejection region of the UMP test for H0 : m = m0 against
H0 : m = m1 with m1 > m0 has the general form:
W = fx : x N > Ag
where A is a constant.
Solution (cont’d)
W = fx : x N > Ag
Determine the critical value A from the nominal size:
α = Pr ( Wj H0 )
= Pr ( x N > Aj H0 )
X N m0 A m0
= 1 Pr p < p H0
σ/ N σ/ N
A m0
= 1 Φ p
σ/ N
Solution (cont’d)
A m
α=1 Φ p 0
σ/ N
So, we have
σ
A = m0 + p Φ 1
(1 α)
N
The rejection region of the UMP test of size α for H0 : m = m0 against
H0 : m = m1 with m1 > m0 is:
σ
W= x : x N > m0 + p Φ 1
(1 α)
N
Fact (UMP one-sided test)

For a one-sided test
H0 : θ = θ 0 against H1 : θ > θ 0 (or H1 : θ < θ 1 )
the rejection region W of the UMP test is equivalent to the rejection

region obtained for the test
H0 : θ = θ 0 against H1 : θ = θ 1
with for θ 1 > θ 0 (or θ 1 < θ 0 ) if this region does not depend on the value
of θ 1 .

H0 : m = m 0
H1 : m > m 0
Question: What is the rejection region of the UMP test of size α?
Solution
Consider the test:
H0 : m = m 0
H1 : m = m 1
with m1 > m0 . The rejection region of the UMP test of size α is:
σ
W= x : x N > m0 + p Φ 1
(1 α)
N
W does not depend on m1 . It is also the rejection region of the UMP
one-sided test for
H0 : m = m 0
H1 : m > m 0
Fact (Two-sided test)

For a two-sided test
H0 : θ = θ 0 against H1 : θ 6= θ 0
the non rejection region W of the test of size α is the intersection of the
non rejection regions of the corresponding one-sided UMP tests of size
α/2
Test A: H0 : θ = θ 0 against H1 : θ > θ 0
Test B: H0 : θ = θ 0 against H1 : θ < θ 0
So, we have:
W = WA \WB

H0 : m = m0
H1 : m 6 = m0
Question: What is the rejection region of the test of size α?
Solution
Consider the one-sided tests:
Test A: H0 : m = m0 against H1 : m < m0
Test B: H0 : m = m0 against H1 : m > m0

The rejection regions of the UMP test of size α/2 are:
σ
WA = x : x N < m0 + p Φ 1
(α/2)
N
σ
WB = x : x N > m0 + p Φ 1
(1 α/2)
N
Solution (cont’d)
The non-rejection regions of the UMP test of size α/2 are:
σ
WA = x : xN m0 + p Φ 1
(α/2)
N
σ
WB = x : xN m0 + p Φ 1
(1 α/2)
N
The non rejection region of the two-sided test corresponds to the
intersection of these two regions:
W = WA \WB
Solution (cont’d)
So, non rejection region of the two-sided test of size α is:

σ σ
W= x : m0 + p Φ 1
(α/2) xN m0 + p Φ 1
(1 α/2)
N N
Since, Φ 1
(α/2) = Φ 1
(1 α/2) , this region can be rewritten as:
σ
W= x : jx N m0 j p Φ 1
(1 α/2)
N
Solution (cont’d)
σ
W= p Φ 1 (1 α/2)
x : jx N m0 j
N
Finally, the rejection region of the two-sided test of size α is:
σ
W= x : jx N m0 j > p Φ 1
(1 α/2)
N
Solution (cont’d)
σ
W= x : jx N m0 j > p Φ 1
(1 α/2)
N
NA: if m0 = 1.2, N = 100, σ2 = 1 and α = 5%:
1
W= x : jx N 1.2j > Φ 1
(0.975)
10
W = fx : jx N 1.2j > 0.1960g

If the realisation of jx N 1.2j is larger than 0.1960, we reject the null
H0 : m = 1.2 for a signi…cance level of 5%.
De…nition (Unbiased Test)

A test is unbiased if its power P (θ ) is greater than or equal to its size α
for all values of the parameter θ.
P (θ ) α 8 θ 2 H1
By construction, we have P (θ 0 ) = Pr ( Wj H j0 ) = α.
De…nition (Consistent Test)

A test is consistent if its power goes to one as the sample size grows to
in…nity.
lim P (θ ) = 1 8θ 2 H1
N !∞

H0 : m = m 0
H1 : m < m 0
The rejection region of the UMP test of size α is
σ
W= x : x N < m0 + p Φ 1
(α)
N
Question: show that this test is (1) unbiased and (2) consistent.
Solution
σ
W= x : x N < m0 + p Φ 1
(α)
N
The power function of the test is de…ned as to be:
P (m ) = Pr ( Wj H1 )
σ
= Pr X N < m0 + p Φ 1
( α ) m < m0
N
m0 m
= Φ p + Φ 1 (α)
σ/ N
Solution
m0 m
P (m ) = Φ p +Φ 1
(α) 8 m < m0
σ/ N
The test is consistent since:
lim P (m ) = 1
N !∞
The test is unbiased since
P (m ) α 8 m < m0
lim P (m ) = Φ Φ 1
(α) = α
m !m 0
Solution
The decision ”Reject H0” or ”fail to reject H0” is not so informative!
Indeed, there is some ”arbitrariness” to the choice of α (level).
Another strategy is to ask, for every α, whether the test rejects at

that level.
Another alternative is to use the so-called p-value— the smallest level

of signi…cance at which H0 would be rejected given the value of the
test-statistic.
De…nition (p-value)
Suppose that for every α 2 [0, 1], one has a size α test with rejection
region Wα . Then, the p-value is de…ned to be:
p-value = inf fα : T (y ) 2 Wα g
The p-value is the smallest level at which one can reject H0 .
The p-value is a measure of evidence against H0 :
p-value evidence
< 0.01 Very strong evidence against H0
0.01 0.05 Strong evidence against H0
0.05 0.10 Weak evidence against H0
> 0.10 Little or no evidence against H0
Remarks
1 A large p-value does not mean ”strong evidence in favor of H0”.

2 A large p-value can occur for two reasons:
1 H0 is true;
2 H0 is false but the test has low power.
3 The p-value is not the probability that the null hypothesis is true!
For a nominal size of 5%, we reject the null H0 : βSP 500 = 0.

For a nominal size of 5%, we fail to reject the null H0 : βC = 0.
Summary
Hypothesis testing is de…ned by the following general procedure
Step 1: State the relevant null and alternative hypotheses (misstating the
hypotheses muddies the rest of the procedure!);
Step 2: Consider the statistical assumptions being made about the sample
in doing the test (independence, distributions, etc.)— incorrect
assumptions mean that the test is invalid!
Step 3: Choose the appropriate test (exact or asymptotic tests) and thus
state the relevant test statistic (say, T).
Summary (cont’d)
Step 4: Derive the distribution of the test statistic under the null
hypothesis (sometimes it is well-known, sometimes it is more tedious!)— for
example, the Student t-distribution or the Fisher distribution.
Step 5: Determine the critical value (and thus the critical region).
Step 6: Compute (using the observations!) the observed value of the test
statistic T , say tobs .
Step 7: Decide to either fail to reject the null hypothesis or reject in favor
of the alternative assumption— the decision rule is to reject the null
hypothesis H0 if the observed value of the test statistic, tobs is in the
critical region, and to ”fail to reject” the null hypothesis otherwise
Key concepts
1 Null and alternative hypotheses

2 Simple and composite hypotheses
3 One-sided and two-sided tests
4 Rejection region, test statistic and critical value
5 Type I and type II errors
6 Size, power and power function
7 Uniformly most powerful (UMP) test
8 Neyman Pearson lemma
9 Consistent test and unbiased test
10 p-value
Section 3
Tests in the multiple linear regression model
3. Tests in the multiple linear regression model
Objectives
In the context of the multiple linear regression model (cf. chapter 3), the
objective of this section is to present :
1 the Student test

2 the t-statistic and the z-statistic
3 the Fisher test
4 the global F-test
5 To distinguish the case with normality assumption and the case
without any assumption on the distribution of the error term
(semi-parametric speci…cation)
Be careful: in this section, I don’t distinguish between a random vector

(matrix) and a vector (matrix) of deterministic elements. For more
appropriate notations, see:
Abadir and Magnus (2002), Notation in econometrics: a proposal for a
standard, Econometrics Journal.
Model
Consider the (population) multiple linear regression model:
y = Xβ + ε
where (cf. chapter 3):
y is a N 1 vector of observations yi for i = 1, .., N
X is a N K matrix of K explicative variables xik for k = 1, ., K and

i = 1, .., N
ε is a N 1 vector of error terms εi .
β = ( β1 ..βK )> is a K 1 vector of parameters
Assumptions
Fact (Assumptions)
We assume that the multiple linear regression model satisfy the
assumptions A1-A5 (cf. chapter 3)
We distinguish two cases:
1 Case 1: assumption A6 (Normality) holds and ε N 0, σ2 IN

2 Case 2: the distribution of ε is unknown (semi-parametric
speci…cation) and ε ??
Parametric tests
The βk are unknown features of the population, but:
1 One can formulate a hypothesis about their value;

2 One can construct a test statistic with a known …nite sample
distribution (case 1) or an asymptotic distribution (case 2);
3 One can take a ”decision” meaning ”reject H0” if the value of the
test statistic is too unlikely.
Three tests of interest:
H0 : β k = a k or H0 : β k = a k
H1 : β k < a k H1 : β k > a k
H0 : β k = a k
H1 : β k 6 = a k
H0 : Rβ = q
H1 : Rβ 6= q
where ak = 0 or ak 6= 0.
For that, we introduce two types of test
1 The Student test or t-test

2 The Fisher test of F-test
Subsection 3.1
The Student test
3.1. The Student test
Case 1: Normality assumption A6
Assumption 6 (normality): the disturbances are normally distributed.

2
εj X N 0N 1, σ IN
Reminder (cf. chapter 3)
Fact (Linear regression model)

b and σ
Under the assumption A6 (normality), the estimators β b2 have a
…nite sample distribution given by:
1
b
β N β,σ2 X> X
b2
σ
(N K) χ2 (N K)
σ2
Moreover, βb and σb2 are independent. This result holds whether or not the
b is
matrix X is considered as random. In this last case, the distribution of β
conditional to X.
Remarks
1 b is also normally distributed:

Any linear combination of β
1
b
Aβ N Aβ,σ2 A X> X A>
2 b has a joint normal distribution.

Any subset of β
b
βk N βk , σ2 mkk
1
where mkk is k th diagonal element of X> X .
Reminder
If X and Y are two independent random variables such that
X N (0, 1)
Y χ2 ( θ )
then the variable Z de…ned as to be
X
Z = p
Y /θ
has a Student’s t-distribution with θ degrees of freedom
Z t( θ )
Student test statistic

Consider a test with the null:
H0 : β k = a k
Under the null H0 :

b
β k ak
p N (0, 1)
σ mkk H0
b2
σ
(N K) χ2 (N K)
σ2 H0
and these two variables are independent...
Student test statistic (cont’d)
b
β k ak
p N (0, 1)
σ mkk H0
b2
σ
(N K) χ2 (N K)
σ2 H0
So, under the null H0 we have:

b
βk a k
p
σ m kk b
β k ak
q = p t(N K)
b 2 (N K )
σ b mkk
σ H0
σ 2 (N K )
De…nition (Student t-statistic)

Under the null H0 : βk = ak , the Student test-statistic or t-statistic is
de…ned to be:
b
βk ak
Tk = t(N K)
b b
se βk H0
where N is the number of observations, K is the number of explanatory

variables (including the constant term), t(N K ) is the Student
t-distribution with N K degrees of freedom and
p
b b
se βk b mkk
=σ
1
with mkk is k th diagonal element of X> X .
Remarks
1 Under the assumption A6 (normality) and under the null

H0 : βk = ak , the Student test-statistic has an exact (…nite sample)
distribution.
Tk t(N K)
H0
2 The term seb b

βk denotes the estimator of the standard error of the
OLS estimator b
βk and it corresponds to the square root of the k th
b β
diagonal element of V b (cf. chapter 3):
1
V b =σ
b β b 2 X> X
Consider the one-sided test:
H0 : β k = a k
H1 : β k < a k
W = f y : Tk ( y ) < A g
where A is a constant determined by the nominal size α.
α = Pr ( Wj H0 ) = Pr Tk ( y ) < A j Tk t(N K)
H0
α = Pr Tk ( y ) < A j Tk t(N K) = FN K (A)

H0
where FN K (.) denotes the cdf of the Student’s t-distribution with N K

degrees of freedom. Denote cα the α-quantile of this distribution:.
A = FN 1 K ( α ) = c α
The rejection region of the test of size α is de…ned as to be:
W = f y : Tk ( y ) < c α g
De…nition (One-sided Student test)

The critical region of the Student test is that H0 : βk = ak is rejected in
favor of H1 : βk < ak at the α (say, 5%) signi…cance level if:
W = f y : Tk ( y ) < c α g
where cα is the α (say, 5%) critical value of a Student t-distribution with

N K degrees of freedom and Tk (y ) is the realisation of the Student
test-statistic.
Example (One-sided test)
Consider the CAPM model (cf. chapter 1) and the following results
(Eviews). We want to test the beta of MSFT as
H0 : βMSFT = 1 against H1 : βMSFT < 1
Question: give a conclusion for a nominal size of 5%.
Solution
Step 1: compute the t-statistic
b
βMSFT 1 1.9898 1
TMSFT (y ) = = = 3.1501
b b
se βMSFT 0.3142
Step 2: Determine the rejection region for a nominal size α = 5%.
TMSFT t(20 2)
H0
W = f y : Tk ( y ) < 1.7341g
Conclusion: for a signi…cance level of 5%, we fail to reject the null
H0 : βMSFT = 1 against H1 : βMSFT < 1
Solution (cont’d)
0.5
0.45 Density of Ts under H0
0.4
0.35
0.3
0.25
0.2 α=5%
0.15
DF = 18
0.1
Critical value = -1.7341
0.05
0
-4 -3 -2 -1 0 1 2 3 4
Consider the one-sided test
H0 : β k = a k
H1 : β k > a k
W = f y : Tk ( y ) > A g
α = Pr ( Wj H0 ) = Pr Tk ( y ) > A j Tk t(N K)
H0
α=1 Pr Tk ( y ) < A j Tk t(N K)

H0
or equivalently
1 α = FN K (A)
where FN K (.) denotes the cdf of the Student’s t-distribution with N K
degrees of freedom. Denote c1 α the 1 α quantile of this distribution:
A = FN 1 K ( 1 α ) = c1 α
W = f y : Tk ( y ) > c1 αg
De…nition (One-sided Student test)

favor of H1 : βk > ak at the α (say, 5%) signi…cance level if:
W = f y : Tk ( y ) > c1 αg
where c1 α is the 1 α (say, 95%) critical value of a Student

t-distribution with N K degrees of freedom and Tk (y ) is the realisation
of the Student test-statistic.
H0 : βMSFT = 1 against H1 : βMSFT > 1
Solution
b
βMSFT 1 1.9898 1
TMSFT (y ) = = = 3.1501
b b
se βMSFT 0.3142
TMSFT t(20 2)
H0
W = fy : Tk (y ) > 1.7341g
Conclusion: for a signi…cance level of 5%, we reject the null
H0 : βMSFT = 1 against H1 : βMSFT > 1
Solution (cont’d)
0.5
0.45
Density of Ts under H0
0.4
0.35
0.3
0.25
0.2
0.15
DF = 18 α=5%
0.1
Critical value = 1.7341
0.05
0
-4 -3 -2 -1 0 1 2 3 4
Consider the two-sided test
H0 : β k = a k
H1 : β k 6 = a k
The non-rejection region is de…ned as the intersection of the two

non-rejection regions of the one-sided test of level α/2:
W = W A \ WB
H0 : βk = ak against H1 : βk < ak WA = fy : Tk (y ) > cα/2 g

H0 : βk = ak against H1 : βk > ak WB = f y : T k ( y ) < c1 α/2 g
W = fy : cα/2 < Tk (y ) < c1 α/2 g
Since the Student’s t-distribution is symmetric, cα/2 = c1 α/2
W = fy : c1 α/2 < T k ( y ) < c1 α/2 g
The rejection region is then de…ned as to be:
W = fy : jTk (y )j > c1 α/2 g
De…nition (Two-sided Student test)

favor of H1 : βk 6= ak at the α (say, 5%) signi…cance level if:
W = fy : jTk (y )j > c1 α/2 g
where c1 α/2 is the 1 α/2 (say, 97.5%) critical value of a Student

t-distribution with N K degrees of freedom and Tk (y ) is the realisation
of the Student test-statistic.
H0 : βMSFT = 1 against H1 : βMSFT 6= 1
Solution
b
βMSFT 1 1.9898 1
TMSFT (y ) = = = 3.1501
b b
se βMSFT 0.3142
TMSFT t(20 2)
H0
W = fy : jTk (y )j > 2.1009g

Conclusion: for a signi…cance level of 5%, we reject the null
0.5
Density of Ts under H0
0.45
0.4
0.35
0.3
0.25
0.2
0.15
α=2.5% DF = 18 α=2.5%
0.1
|Critical value| = 2.1009
0.05
0
-4 -3 -2 -1 0 1 2 3 4
Rejection regions
H0 H1 Rejection region
β k = ak β k > ak W = f y : Tk ( y ) > c1 αg
β k = ak β k < ak W = f y : Tk ( y ) < c α g
β k = ak β k 6 = ak W = fy : jTk (y )j > c1 α/2 g
where cβ denotes the β-quantile (critical value) of the Student

t-distribution with N K degrees of freedom.
De…nition (P-values)
The p-values of Student tests are equal to:
Two-sided test: p-value = 2 FN K ( jTk (y )j)
Right tailed test: p-value = 1 FN K (Tk (y ))

Left tailed test: p-value = FN K ( Tk (y ))
where Tk (y ) is the realisation of the Student test-statistic and FN K (.)
the cdf of the Student’s t-distribution with N K degrees of freedom.
Consider the previous CAPM model. We want to test:
H0 : c = 0 against H1 : c 6= 0

Question: …nd the corresponding p-values.
Solution
Since we consider two-sided tests with N = 20 and K = 2:
p-valuec = 2 F18 ( jTc (y )j) = 2 F18 ( 0.9868) = 0.3368

006
p-valuec = 2 F18 ( jTMSFT (y )j) = 2 F18 ( 6.3326) = 5.7e
Fact (Student test with large sample)

For a large sample size N
Tk t(N K) N (0, 1)
H0
Then, the rejection region for a Student two-sided test becomes
W = y : jTk (y )j > Φ 1
(1 α/2)
where Φ (.) denotes the cdf of the standard normal distribution. For
α = 5%, Φ 1 (0.975) = 1.96, so we have:
W = fy : jTk (y )j > 1.96g
Case 2: Semi-parametric model
Assumption 6 (normality): the distribution of the disturbances is

unknown, but satisfy (assumptions A1-A5):
E ( εj X) = 0N 1
V ( ε j X ) = σ 2 IN
Problem
1 The exact (…nite sample) distribution of b b2 are unknown.

βk and σ
2 As a consequence the …nite sample distribution of Tk (y ) is also
unknown.
3 But, we can use the asymptotic properties of the OLS estimators (cf.
chapter 3). In particular, we have:
p d
b
N β β ! N 0, σ2 Q 1
where
1 >
Q = p lim X X = EX xi xi>
N
De…nition (Z-statistic)
Under the null H0 : βk = ak , if the assumptions A1-A5 hold (cf. chapter
3), the z-statistic de…ned by
b
βk ak d
Zk = ! N (0, 1)
b asy b
se βk H0
p
b asy b
where se βk =σb mkk denotes the estimator of the asymptotic
standard error of the estimator b
βk and mkk is k th diagonal element of
1
X> X .
Rejection regions
The rejection regions have the same form as for the t-test (except for the
distribution)
H0 H1 Rejection region
β k = ak β k > ak W = y : Zk (y ) > Φ 1
(1 α)
β k = ak β k < ak W = y : Zk (y ) < Φ 1
(α)
β k = ak β k 6 = ak W = y : jZk (y )j > Φ 1
(1 α/2)
where Φ (.) denotes the cdf of the standard normal distribution.
The p-values of the Z-tests are equal to:
Two-sided test: p-value = 2 Φ( jZk (y )j)
right tailed test: p-value = 1 Φ (Zk (y ))

left tailed test: p-value = Φ ( Zk (y ))
where Zk (y ) is the realisation of the Z-statistic and Φ (.) the cdf of the
standard normal distribution.
Summary
Normality Assumption Non Assumption

Test-statistic t-statistic z-statistic
b
βk a k b
βk a k
De…nition Tk = p
b m kk
σ Zk = p
b m kk
σ
Exact distribution Tk t(N K) —
H0
d
Asymptotic distribution — ZK ! N (0, 1)
H0
Subsection 3.2
The Fisher test
3.2. The Fisher test
Consider the two-sided test associated to p linear constraints on the

parameters βk :
H0 : Rβ = q
H1 : Rβ 6= q
where R is a p K matrix and q is a p 1 vector.
Example (Linear constraints)

If K = 4 and if we want to test H0 : β1 + β2 = 0 and β2 3β3 = 4, then
we have p = 2 linear constraints with:
R β = q
(2 4 ) (4,1 ) (2 1 )
0 1
β1
1 1 0 0 B β C 0
B 2 C=
0 1 3 0 @ β A 4
3
β4

If K = 4 and if we want to test H0 : β2 = β3 = β4 = 0, then we have
p = 3 linear constraints with:
R β = q
(3 4 ) (4,1 ) (3 1 )
1 0
0 β1 0 11
0 1 0 0 B β C 0
@ 0 0 1 0 AB 2 C = @ 0 A
@ β A
0 0 0 1 3 0
β4
Case 1: Normality assumption A6
De…nition (Fisher test-statistic)

Under assumptions A1-A6 (cf. chapter 3), the Fisher test-statistic is
de…ned as to be:
> 1
1 b
1
b
F= Rβ q b 2 R X> X
σ R> Rβ q
p
where β b denotes the OLS estimator. Under the null H0 : Rβ = q, the

F -statistic has a Fisher exact (…nite sample) distribution
F F(p,N K)
H0
Reminder
If X and Y are two independent random variables such that
X χ2 ( θ 1 )
Y χ2 ( θ 2 )
then the variable Z de…ned by
X /θ 1
Z =
Y /θ 2
has a Fisher distribution with θ 1 and θ 2 degrees of freedom
Z F(θ 1 ,θ 2 )
Proof
Under assumption A6, we have the following (conditional to X)
distribution
1
b N β,σ2 X> X
β
b2
σ
(N K) χ2 (N K)
σ2
Proof (cont’d)
b
Consider the vector m = R β q. Under the null
H0 : Rβ = q
We have
b
E (m) = RE β q = Rβ q=0
>
V (m) = E b
Rβ q b
Rβ q
b R>
= RV β
1
= σ 2 R X> X R>
Proof (cont’d)
We can base the test of H0 on the Wald criterion:
1
W = m> (V (m)) m
(1 1 ) (1 p ) p p p 1
> 1 1
= b
Rβ q 2
σ R X X >
R > b
Rβ q
Under assumption A6 (normality)
W χ2 (p )
H0
b2
σ
(N K) χ2 (N K)
σ2
These two variables are independent.

Proof (cont’d)
W χ2 (p )
H0
b2
σ
(N K ) χ2 (N K )
σ2
So, the ratio of these two variables has a Fisher distribution
W
p
F= F(p,N K)
b 2 (N K ) H 0
σ
σ 2 (N K )
Proof (cont’d)
> 1 1
b
Rβ q σ 2 R X> X R> b
Rβ q /p
F=
b2
σ
σ2
(N K ) / (N K)
After simpli…cation, the F-statistic is de…ned by:
> 1
1 b 2 >
1
> b
F= Rβ q b R X X
σ R Rβ q
p
Under the null H0 : Rβ =q :
F F(p,N K)
H0

Under assumptions A1-A6 (cf. chapter 3), the Fisher test-statistic can
be de…ned as a function of the SSR of the constrained (H0 ) and
unconstrained model (H1 ):
SSR0 SSR1 N K
F=
SSR1 p
where SSR0 denotes the sum of squared residuals of the constrained model
estimated under H0 and SSR1 denotes the sum of squared residuals of the
unconstrained model estimated under H1 .

Under assumptions A1-A6 (cf. chapter 3), the Fisher test-statistic can
be de…ned as to be:
1 b >
F= βH 1 b
β X> X b
β b
β
H0 H1 H0
b2 p
σ
b denotes the OLS estimator obtained in the constrained model

where β H0
b denotes the OLS estimator obtained in the
(under H0 ) and β H1
unconstrained model (under H1 ).
De…nition (Constrained OLS estimator)

b of
Under suitable regularity conditions, the constrained OLS estimator β C
β, obtained under the constraint Rβ = q, is given by:
1 1 1
b =β
β b X> X R> R X> X R> b
Rβ q
C UC UC
b
where β UC is the unconstrained OLS estimator.
Example (Fisher test and CAPM model)

Consider the extended CAPM model (…le: Chapter4_data.xls):
rMSFT ,t = β1 + β2 rSP 500,t + β3 rFord ,t + β4 rGE ,t + εt
where rMSFT ,t is the excess return for Microsoft, rSP 500,t for the SP500,
rFord ,t for Ford and rGE ,t for general electric. We want to test the
following linear constraints:
H0 : β2 = 1 and β3 = β4
Question: write a Matlab code to compute the F-statistic according to

the three alternative de…nitions.
Solution
In this problem, the null H0 : β2 = 1 and β3 = β4 can be written as:
R β = q
(2 4 ) (4,1 ) (2 1 )
0 1
β1
0 1 0 0 B β C 1
B 2 C=
0 0 1 1 @ β A 0
3
β4
Consider the Fisher test
H0 : Rβ = q
H1 : Rβ 6= q
Since the Fisher test-statistic is always positive, the rejection region is

de…ned as to be:
W = fy : F (y ) > Ag
α = Pr ( Wj H0 ) = Pr F (y ) > Aj F F(p,N K)
H0
α = Pr ( Wj H0 ) = Pr F (y ) > Aj F F(p,N K)
H0
or equivalently
α=1 Pr F (y ) < Aj F F(p,N K)

H0
Denote d1 α the 1 α quantile of the Fisher distribution with p and

N K degrees of freedom.
A = d1 α
W = fy : F (y ) > d1 αg
De…nition (Rejection region of a Fisher test)

The critical region of the Fisher test is that H0 : Rβ = q is rejected in
favor of H1 : Rβ 6= q at the α (say, 5%) signi…cance level if:
W = fy : F (y ) > d1 αg
where d1 α is the 1 α critical value (say 95%) of the Fisher distribution

with p and N K degrees of freedom and Fk (y ) is the realisation of the
Fisher test-statistic.
Example (Fisher test and CAPM model)

where rMSFT ,t is the excess return for Microsoft, rSP 500,t for the SP500,
rFord ,t for Ford and rGE ,t for general electric. We want to test the
following linear constraints:
H0 : β2 = 1 and β3 = β4
Question: given the realisation of the Fisher test-statistic (cf. previous

example), conclude for a signi…cance level α = 5%.
Solution
Step 1: compute the F-statistic (cf. Matlab code)
F (y ) = 4.3406
Step 2: Determine the rejection region for a nominal size α = 5% for

N = 24, K = 4 and p = 2
F F(2,20 )
H0
W = fy : F (y ) > 3.4928g
Conclusion: for a signi…cance level of 5%, we reject the null H0 : Rβ = q
against H1 : Rβ 6= q
1.2
Density of F under H0
1
0.8 DF1 = 2
DF2 = 20
Critical value = 3.4928
0.6
0.4
α=5%
0.2
0
0 1 2 3 4 5 6
De…nition (Student test-statistic and Fisher test-statistic )

Consider the test
H0 : β k = a k versus H1 : βk 6= ak
the Fisher test-statistic corresponds to the squared of the corresponding

Student’s test-statistic
F = T2k
Proof
Consider the test H0 : βk = ak against H1 : βk 6= ak , then we have:
R= 0 0 .. 1 0 0
k th position
q = ak
As a consequence :
b
Rβ q=b
βk ak
1
b 2 R X> X
σ b b
R> = V βk
Proof (cont’d)
So, for a test H0 : βk = ak against H1 : βk 6= ak , the Fisher test-statistic
becomes
> 1 1
b
F = Rβ q b 2 R X> X
σ R> b
Rβ q
So, we have:
2
b
βk ak
F=
b b
V βk
and the F test-statistic is equal to the squared t-statistic:
F = T2k
The p-value of the F-test is equal to:
p-value = 1 Fp,N K (F (y ))
where F(y ) is the realisation of the F-statistic and Fp,N K (.) the cdf of
the Fisher distribution with p and N K degrees of freedom.
De…nition (Global F-test)

In a multiple linear regression model with a constant term
yi = β1 + ∑K
k =2 βk xik + εi
the global F-test corresponds to the test of signi…cance of all the

explicative variables:
H0 : β2 = .. = βK = 0
Under the assumption A6 (normality), the global F-test-statistic satis…es:
F F(K 1,N K )
H0
Remarks
1 The global F-test is a test designed to see if the model is useful

overall.
2 The null H0 : β2 = .. = βK = 0 can be written as:
R β = q
(K 1 K ) (K ,1 ) (K 1 1 )
0 1
0 1 0 0 .. 0 0 1 0 1
B C β 1 0
B 0 0 1 0 .. 0 C B .. C B .. C
B CB C B C
B .. .. 0 1 .. .. C @ .. A = @ .. A
@ .. .. .. .. .. .. A
βK 0
0 .. 0 0 .. 1
Corollary (Global F-test)

In a multiple linear regression model with a constant term
yi = β1 + ∑K
k =2 βk xik + εi
the global F-test-statistic can also be de…ned as:
R2 N K
F=
1 R2 K 1
where R 2 denotes the (unadjusted) coe¢ cient of determination.
Example (Global F-test and CAPM model)
Question: write a Matlab code to compute the global F-test, the critical
value for α = 5% and the p-value. Compare your results with Eviews.
Case 2: Semi-parametric model
Assumption 6 (normality): the distribution of the disturbances is

unknown, but satisfy (assumptions A1-A5):
E ( εj X) = 0N 1
V ( ε j X ) = σ 2 IN
Problem
1 The exact (…nite sample) distribution of b b2 are unknown. As a

βk and σ
consequence the …nite sample distribution of F(y ) is also unknown.
2 But, we can express the F-statistic as a linear function of the Wald
statistic.
3 The Wald statistic has a chi-squared asymptotic distribution (cf. next
section)
De…nition (F-test-statistic and Wald statistic)

The Fisher test-statistic can expressed as a linear function of the Wald
test-statistic as
1
F = Wald
p
> 1
1 b b
1
b
Wald = Rβ q R Vasy β R> Rβ q
p
Under assumptions A1-A5, the Wald test-statistic converges to a
chi-squared distribution
d
Wald ! χ2 (p )
H0
Key concepts of Section 3
1 Student test
2 Fisher test
3 t-statistic and z-statistic
4 Global F-test
5 Exact (…nite sample) distribution under the normality assumption
6 Asymptotic distribution
Section 4
MLE and Inference
4. MLE and inference
Introduction
Consider a parametric model, linear or nonlinear (GARCH, probit,

logit, etc.), with a vector of parameters θ = (θ 1 : .. : θ K )>
We assume that the problem is regular (cf. chapter 2) and we

consider a ML estimator b
θ
The …nite sample distribution of bθ is unknown, but bθ is

asymptotically normally distributed (cf. chapter 2).
We want to test a set of linear or nonlinear constraints on the true

parameters (population) θ 1 , .., θ K .
De…nition (Null hypothesis)

Consider a null hypothesis of p linear and/or nonlinear constraints
H0 : c (θ) = 0p 1
| {z }
p 1
where c (θ) is a vectorial function de…ned as:
c: RK ! Rp
θ 7! c (θ)
Notations
1 c (θ) is a p 1 vector of functions c1 (θ) , .., cp (θ):

0 1
c1 (θ)
B c2 (θ) C
c (θ) = B
@ .. A
C
cp (θ)
2 In the case of p linear constraints, we have:
H0 : c (θ) = Rθ q=0

Consider the two linear constraints θ 1 = θ 2 + θ 3 and θ 2 + θ 4 = 1. We
have p = 2 constraints such that:
θ1 θ2 θ3 0
H0 : c (θ) = =
(2,1 )
θ2 + θ4 1 0
The function c (θ) can be written as Rθ q. For instance if K = 4 and θ

= (θ 1 θ 2 θ 3 θ 4 )> , we have
0 1
θ1
1 1 1 0 B θ2 C 0
c (θ) = Rθ q= B C
0 1 0 1 @ θ3 A 1
θ4
Example (Nonlinear constraints)

Consider the linear and nonlinear constraints
θ1 θ2 = 0 θ 21 θ3 = 0
We have p = 2 constraints such that:
θ1 θ2 0
H0 : c (θ) = =
(2,1 ) θ 21 θ3 0
Assumptions
1 The functions c1 (θ) , .., cp (θ) are di¤erentiable.

2 There is no redundant constraint (identi…cation assumption).
Formally, we have
∂c (θ)
(row) rank = p 8θ 2 Θ
∂θ>
with 0 1
∂c1 (θ) ∂c1 (θ) ∂c1 (θ)
∂θ 1 ∂θ 2 .. ∂θ K
B C
∂c (θ) B ∂c2 (θ) ∂c2 (θ)
..
∂c2 (θ) C
B ∂θ 1 ∂θ 2 ∂θ K C
=B C
∂θ> B .. .. .. .. C
(p,K ) @ A
∂cp (θ) ∂cp (θ) ∂cp (θ)
∂θ 1 ∂θ 2 .. ∂θ K
Consider the two-sided test
H0 : c (θ) = 0 versus H1 : c (θ) 6= 0

We introduce three di¤erent asymptotic tests (the trilogy..)
1 The Likelihood Ratio (LR) test

2 The Wald test
3 The Lagrance Multiplier (LM) test
For each of the three tests, we will present:
1 the test-statistic
2 its asymptotic distribution under the null
3 the (asymptotic) rejection region
4 the (asymptotic) p-value
Subsection 4.1
The Likelihood Ratio (LR) test
4.1. The Likelihood Ratio (LR) test
De…nition (Likelihood Ratio (LR) test statistic)

The likelihood ratio (LR) test-statistic is de…ned by as to be:
LR = 2 `N b
θH 0 ; y j x `N b
θH 1 ; y j x
where `N (θ; y j x ) denotes the (conditional) log-likelihood of the sample y ,

b
θH 0 and b
θH 1 are respectively the maximum likelihood estimator of θ under
the alternative and the null hypothesis.
Comments
Consider the ratio of likelihoods under H1 (no constraint) and under H0
(with c (θ) = 0).
LN bθH 0 ; y j x
λ=
LN bθH 1 ; y j x
1 λ > 0 since both likelihood are positive.

2 λ < 1 since LN (H0 ) cannot be larger than LN (H1 ) . A restricted
optimum is never superior to an unrestricted one.
3 If λ is too small, then doubt is cast on the restrictions c (θ) = 0.
4 Consider the statistic LR= 2 ln (λ): if λ is "too small", then LR is
large (rejection of the null)...
De…nition (Asymptotic distribution and critical region)

Under some regularity conditions (cf. chapter 2) and under the null
H0 : c (θ) = 0, the LR test-statistic converges to a chi-squared
distribution with p degrees of freedom (the number of restrictions
imposed):
d
LR ! χ2 (p )
H0
The (asymptotic) critical region for a signi…cance level of α is:
W = y : LR (y ) > χ21 α (p )
where χ21 α (p ) is the 1 α critical value of the chi-squared distribution

with p degrees of freedom and LR(y ) is the realisation of the LR
test-statistic.
De…nition (p-value of the LRT test)

The p-value of the LR test is equal to:
p-value = 1 Gp (LR (y ))
where LR(y ) is the realisation of the LR test-statistic and Gp (.) is the cdf
of the chi-squared distribution with p degrees of freedom.
Example (LRT and Poisson distribution)

Suppose that X1 ,X2 , ,XN are i.i.d. discrete random variables, such that
Xi Pois (θ ) with a pmf (probability mass function) de…ned as:
exp ( θ ) θ xi
Pr (Xi = xi ) =
xi !
where θ is an unknown parameter to estimate. We have a sample
(realisation) of size N = 10 given by f5, 0, 1, 1, 0, 3, 2, 3, 4, 1g . Question:
use a LR test to test the null H0 : θ = 1.8 against H1 : θ 6= 1.8 and give a
conclusion for signi…cance level of 5%.
Solution
The log-likelihood function is de…ned as to be:
N N
`N (θ; x ) = θN + ln (θ ) ∑ xi ln ∏ xi !
i =1 i =1
In the chapter 2, we found that the ML estimator of θ is the sample mean:

N
1
b
θ=
N ∑ Xi
i =1
Given the sample f5, 0, 1, 1, 0, 3, 2, 3, 4, 1g , the estimate of θ (under H1 ,

with non constraint) is b
θ H 1 = 2, and the corresponding log-likelihood is
equal to:
`N bθ H1 ; x = ln (0.104)
Solution (cont’d)
Under the null H0 : θ = 1.8, we don’t need to estimate θ and the
log-likelihood is equal to:
N N
`N ( θ H 0 ; x ) = 1.8N + ln (1.8) ∑ xi ln ∏ xi ! = ln (0.0936)
i =1 i =1
The LR test-statistic is equal to:
0.0936
LR (y ) = 2 ln = 0.21072
0.104
Solution (cont’d)
LR (y ) = 0.21072
For N = 10, p = 1 (one restriction) and α = 0.05, the critical region is:
W = y : LR (y ) > χ20.95 (1) = 3.8415
and the p-value is
pvalue = 1 G1 (0.21072) = 0.6462
where G1 (.) is the cdf of the χ2 (1) distribution.

Conclusion: for a signi…cance level of 5%, we fail to reject the null
H0 : θ = 1.8.
Subsection 4.2
The Wald test
4.2. The Wald test
De…nition (Wald test-statistic)

The Wald test-statistic associated to the test of H0 : c (θ) = 0 is de…ned
as to be:
> > 1
∂c ∂c
Wald = c b
θH 1 b b asy b
θH 1 V θH 1 b
θH 1 c b
θH 1
∂θ> ∂θ>
where b
θH 1 is the maximum likelihood estimator of θ under the alternative
hypothesis (unconstrained model) and V b asy b
θH 1 is an estimator of its
asymptotic variance covariance matrix.
4.2. The Wald test
Remark
0 1 1
>B
B ∂c b ∂c b >CC
Wald = c b
θH 1 b asy b
B > θH 1 V θH 1 >
θ H 1 C c b
θH 1
@
| {z } | {z }|
∂θ {z }|
∂θ A | {z }
{z }
1 p p K K K K p p 1
4.2. The Wald test
Example (Wald test-statistic)

Consider a model with K = 3 parameters θ = (θ 1 : θ 2 : θ 3 )> with
θ1 θ2 = 0 θ 21 θ3 = 0
We have two constraints (p = 2) and:
θ1 θ2 0
H0 : c (θ) = =
(2,1 ) θ 21 θ3 0
θH 1 = (θ 1 : θ 2 : θ 3 )> the ML estimator of θ under the alternative

Denote b
b asy b
hypothesis and V θH 1 the estimator of its asymptotic variance
covariance matrix. Question: write the Wald test-statistic.
4.2. The Wald test
Solution
Here we have K = 3 and p = 2
!
b
θ1 b
θ2
c b
θH 1 = 2
b
θ1 b
θ3
∂c b 1 1 0
θH 1 =
∂θ > 2b
θ1 0 1
> > 1
∂c ∂c
Wald = c b
θH 1 b b asy b
θH 1 V θH 1 b
θH 1 c b
θH 1
∂θ> ∂θ>
4.2. The Wald test
Remark
In the case of linear constraints
H0 : Rθ q=0
we have
H0 : c (θ) = 0
with
c (θ) = Rθ q
∂c
(θ) = R
∂θ>
4.2. The Wald test
De…nition (Wald test-statistic and linear constraints)

Consider the test of linear constraints H0 : c (θ) = Rθ q = 0. The
Wald test-statistic is de…ned as to be:
> 1
Wald = Rb
θH 1 q b asy b
RV θH 1 R> Rb
θH 1 q
where b
θH 1 is the maximum likelihood estimator of θ under the alternative
hypothesis (unconstrained model) and V b asy b
θH 1 is an estimator of its
asymptotic variance covariance matrix.
4.2. The Wald test

H0 : c (θ) = 0, the Wald test-statistic converges to a chi-squared
imposed):
d
Wald ! χ2 (p )
H0
W = y : Wald (y ) > χ21 α (p )

with p degrees of freedom and Wald(y ) is the realisation of the Wald
test-statistic.
4.2. The Wald test
Proof
Under some regularity conditions, we have
p d
N b
θH 1 θ0 ! N 0, I 1
(θ0 )
We use the delta method for the function c (.) . The function c (.) is a
continuous and continuously di¤erentiable function not involving N, then
p d ∂c ∂c
N c b
θH 1 c (θ0 ) ! N 0, >
(θ0 ) I 1
(θ0 ) >
(θ0 )>
∂θ ∂θ
Under the null H0 : c (θ0 ) = 0, we have
1/2 p
∂c ∂c d
>
(θ0 ) I 1
(θ0 ) >
(θ0 )> Nc b
θH 1 ! N (0, Ip )
∂θ ∂θ
where Ip is the identity matrix of size p.
4.2. The Wald test
Proof (cont’d)
The Wald criteria is de…ned as to be:
Wald criteria
!>
> 1/2
∂c ∂c >
= N c b
θH 1 (θ0 ) I 1
(θ0 ) (θ0 )
∂θ> ∂θ>
1/2
∂c ∂c >
(θ0 ) I 1
(θ0 ) (θ0 ) c b
θH 1
∂θ> ∂θ>
> 1
∂c ∂c
= N c b
θH 1 (θ0 ) I 1
(θ0 ) (θ0 )> c b
θH 1
∂θ> ∂θ>
So, under the null H0 : c (θ0 ) = 0,, we have
d
Wald criteria ! χ2 (p )
H0
4.2. The Wald test
Proof (cont’d)
>
Wald Criteria = N c b
θH 1
1
∂c ∂c
(θ0 ) I 1
(θ0 ) (θ0 )> c b
θH 1
∂θ> ∂θ>
A feasible Wald test-statistic is given by
>
Wald = N c b
θH 1
> 1
∂c 1 ∂c
θH 1 bI
b b
θH 1 b
θH 1 c b
θH 1
∂θ> ∂θ>
4.2. The Wald test
Proof (cont’d)
Since
b asy b 1b 1 b
V θH 1 =N I θH 1
We have …nally
> > 1
∂c ∂c
Wald = c b
θH 1 b b asy b
θH 1 V θH 1 b
θH 1 c b
θH 1
∂θ> ∂θ>
and
d
Wald ! χ2 (p )
H0
4.2. The Wald test
De…nition (p-value of the Wald test)

The p-value of the Wald test is equal to:
p-value = 1 Gp (Wald (y ))
where Wald(y ) is the realisation of the Wald test-statistic and Gp (.) is

the cdf of the chi-squared distribution with p degrees of freedom.
4.2. The Wald test
De…nition (z-statistic)
Consider the test H0 : θ k = ak versus H1 : θ k 6= ak . The z-statistic
corresponds to the square root of the Wald test-statistic and satis…es
b
θk ak d
Zk = r ! N (0, 1)
H0
b asy b
V θk
where b
θ k is the ML estimator of θ k obtained under H1 (unconstrained
model). The critical region for a signi…cance level of α is:
n α o
W = y : jZk (y )j > Φ 1 1
2
where Φ (.) denotes the cdf of the standard normal distribution.
4.2. The Wald test
Computational issues
The Wald test-statistic depends on the estimator of the asymptotic
variance covariance matrix:
> > 1
∂c ∂c
Wald = c b
θH 1 b b asy b
θH 1 V θH 1 b
θH 1 c b
θH 1
∂θ> ∂θ>
b asy b 1b 1 b
V θH 1 =N I θH 1
where I b
θH 1 denotes the average Fisher information matrix.
4.2. The Wald test
Computational issues (cont’d)

Three estimators are available for the average Fisher information matrix:
N
1
Actual Average Fisher Matrix: bI A b
θ =
N ∑ bI i b
θ
i =1
> !
N
1 ∂ì (θ; yi j xi ) ∂ì (θ; yi j xi )
BHHH estimator: bI B b
θ =
N ∑ ∂θ b ∂θ b
i =1 θ θ
N
1 ∂2 ì (θ; yi j xi )
Hessian based estimator: bI c b
θ =
N ∑ ∂θ∂θ> b
i =1 θ
4.2. The Wald test
1 These estimators are asymptotically equivalent, but the corresponding

estimates may be very di¤erent in small samples.
2 Thus, we can obtain three di¤erent values for the Wald statistic
given the choice of the estimator for Vasy b
θH 1 (cf. exercises).
3 In general, the estimator A is rarely available and the estimator B

(BHHH) gives erratic results.
4 Most of the software use the estimator C (Hessian based estimator).
4.2. The Wald test
Subsection 4.3
The Lagrange Multiplier (LM) test
4.3. The Lagrange Multiplier (LM) test
Introduction
Consider the set of constraints c (θ) = 0. Let λ be a vector of Lagrange
multipliers and de…ne the Lagrangian function
`N (θ ; y j x ) = `N (θ; y j x ) + λc (θ)
The solution to the constrained maximization problem is the root of

>
∂ `N ( θ ; y j x ) ∂` (θ; y j x ) ∂c (θ)
= N + λ
∂θ ∂θ ∂θ>
∂ `N ( θ ; y j x )
= c (θ)
∂λ
>
∂ `N ( θ ; y j x ) ∂` (θ; y j x ) ∂c (θ)
= N + λ
∂θ ∂θ ∂θ>
1 If the restrictions are valid, then imposing them will not lead to a
signi…cant di¤erence in the maximized value of the likelihood function.
In the …rst-order conditions, the meaning is that the second term in
the derivative vector will be small. In particular, λ will be small.
2 We could test this directly, that is, test
H0 : λ = 0
which leads to the Lagrange multiplier test.
There is an equivalent simpler formulation, however. If the restrictions
c (θ) = 0 are valid, the derivatives of the log-likelihood of the
unconstrained model evaluated at the restricted parameter vector will
be approximately zero.
∂`N (θ; y j x )
=0
∂θ b
θH 0
The vector of …rst derivatives of the log-likelihood is the vector of

(e¢ cient) scores.
De…nition (LM or score test)

For these reasons, this test is called the score test as well as the
Lagrange multiplier test.
Guess
Let us assume that θ is scalar, i.e. K = 1, then the LM statistic is simply
de…ned as:
2
sN bθH 0 ; Y j x
LM =
V sN b θH 0 ; Y j x
Since bI N b
θH 0 = V sN b
θH 0 ; Y j x , we have:
2
sN b
θH 0 ; Y j x
LM =
bI N b
θH 0

The LM test-statistic or score test associated to the test of H0 :
c (θ) = 0 is de…ned as to be:
>
LM = sN b
θH 0 ; Y j x bI N 1 b
θH 0 sN b
θH 0 ; Y j x
where b
θH 0 is the maximum likelihood estimator of θ under the null
hypothesis (constrained model), sN (θ; Y j x ) is the score vector of the
unconstrained model and bI N b θH 0 is an estimator of the Fisher
information matrix of the sample evaluated at b
θH 0 .
Remark
Since:
1
b asy b
V θH 0 = bI N b
θH 0
there is another expression for the LM statistic.

The LM test-statistic or score test associated to the test of H0 :
c (θ) = 0 is de…ned as to be:
>
LM = sN b
θH 0 ; Y j x V θH 0 sN b
b asy b θH 0 ; Y j x
where b
θH 0 is the maximum likelihood estimator of θ under the null
hypothesis (constrained model), sN (θ; Y j x ) is the score vector of the
unconstrained model and V b asy b θH 0 is an estimator of the asymptotic
variance covariance matrix of b
θH 0 .
Remark
The LM test-statistic can also be de…ned by:
>
∂c b b asy b ∂c b
LM = λ> θH 0 V θH 0 θH 0 λ
∂θ> ∂θ>
where λ denotes the Lagrange Multiplier associated to the constraints
c (θ) = 0.
The LM test-statistic can be obtained from the following auxiliary
procedure:
Step 1: Estimate the constrained model and obtain b
θH 0 .
Step 2: Form the gradients for each observation of the unrestricted
model evaluated at b
θH 0
gi b
θH 0 ; yi j xi 8i = 1, ..N
Step 3: Run the regression of a vector of 1 on the variables

gi b
θH 0 ; yi j xi 8i = 1, ..N, then
LM = N R2
where R 2 denotes the (unadjusted) coe¢ cient of determination of this

auxiliary regression.
Computational issues
1 The LM test-statistic depends on the estimator of the asymptotic

variance covariance matrix:
>
LM = sN b
θH 0 ; Y j x V θH 0 sN b
b asy b θH 0 ; Y j x
b asy b 1b 1 b
V θH 0 =N I θH 0
where I b
θH 0 denotes the average Fisher information matrix.
2 Thus, we can obtain three di¤erent values for the LM statistic
given the choice of the estimator for Vasy b
θH 0 (cf. exercises).

H0 : c (θ) = 0, the LM test-statistic converges to a chi-squared
imposed):
d
LM ! χ2 (p )
H0
W = y : LM (y ) > χ21 α (p )

with p degrees of freedom and LM(y ) is the realisation of the LM
test-statistic.
De…nition (p-value of the LM test)

The p-value of the LM test is equal to:
p-value = 1 Gp (LM (y ))
where LM(y ) is the realisation of the LM test-statistic and Gp (.) is the

cdf of the chi-squared distribution with p degrees of freedom.
Subsection 4.4
A comparison of the three tests
4.4. A comparison of the three tests
Source: Pelgrin (2010), Lecture notes, Advanced Econometrics
Summary
Test Requires estimation under
LRT H0 and H1
Wald H1
LM H0
Computational problems
If the ML maximisation problem is complex (with local extrema) and

if it has no closed form solution (nonlinear models: GARCH, Markov
Switching models etc.), it may be particularly di¢ cult to get a ML
estimates bθ through a numerical optimisation of the log-likelihood.
If the constraints c (θ) = 0 are not valid in the data, the (numerical)
convergence of the optimisation algorithm may be very problematic
under the null H0 .
Asymptotic comparison
The three tests have the same asymptotic distribution under the null
H0 : c (θ) = 0:
d
LRT ! χ2 (p )
H0
d
Wald ! χ2 (p )
H0
d
LM ! χ2 (p )
H0
Theorem (Asymptotic comparison)

The three tests are asymptotically equivalent. Under some regularity
conditions and under the null H0 : c (θ) = 0, the di¤erences between the
three test statistics converge to 0 as N tends to in…nity:
p
LRT LM ! 0
H0
p
LRT Wald ! 0
H0
p
LM Wald ! 0
H0
Fact (Finite sample properties)

The …nite sample properties of the three tests may be di¤erent,
especially in small samples. For small sample size, they can lead to
opposite conclusion about the rejection of the null hypothesis.
Key concepts of Section 4
1 Likelihood Ratio (LR) test

2 Wald test
3 Lagrange Multiplier (LM) test
4 Computational issues
5 Comparison of the three tests (the trilogy) in …nite samples
End of Chapter 4
Christophe Hurlin (University of Orléans)

Chapter4 Statistical Hypothesis Testing

Uploaded by

Copyright:

Available Formats

You might also like

Chapter4 Statistical Hypothesis Testing

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Chapter4 Statistical Hypothesis Testing

Uploaded by

Copyright:

Available Formats

Chapter 4: Statistical Hypothesis Testing

November 20, 2015

The outline of this chapter is the following:

Amemiya T. (1985), Advanced Econometrics. Harvard University Press.

Greene W. (2007), Econometric Analysis, sixth edition, Pearson - Prentice

fY ( y ) probability density or mass function

Be careful: in this chapter, I don’t distinguish between a random vector

Statistical hypothesis testing

1 Null and alternative hypotheses

1 A statistical hypothesis test is a method of making decisions or a

In this course, we only consider the parametric tests.

1 A null hypothesis and an alternative hypothesis

1 A null hypothesis and an alternative hypothesis

1 The null hypothesis refers to a general or default position: that there

Example (Choice of the null hypothesis)

De…nition (Simple and composite hypotheses)

Example (Simple and composite hypotheses)

De…nition (One-sided test)

De…nition (Two-sided test)

1 A null hypothesis and an alternative hypothesis

De…nition (Rejection region)

1 A (hypothesis) test is thus a rule that speci…es:

T (x ) is the realisation of the statistic (random variable):

The test statistic T (X ) has an exact or an asymptotic distribution D

1 A null hypothesis and an alternative hypothesis

For a composite null hypothesis:

A test is said to have level if its size is less than or equal to α.

where β denotes the probability of type II error, i.e. β = Pr W H1 and

Example (Test on the mean)

with m1 < m0 . An econometrician propose the following rule of decision:

where X N = N 1 ∑N i =1 Xi denotes the sample mean and c is a constant

So, the size of the test is equal to:

Fact (Critical value)

Example (Test on the mean)

An econometrician propose the following rule of decision:

where X N = N 1 ∑N i =1 Xi denotes the sample mean and c is a constant

NA: if m0 = 1.2, m1 = 1, N = 100, σ2 = 1 and α = 5%, then the

Example (Test on the mean)

The rejection region for a signi…cance level α = 5% is:

where X N = N 1 ∑N i =1 Xi denotes the sample mean. Question: if the

For a level of 5%, we do not reject the null hypothesis H0 : m = 1.2.

De…nition (Power function)

Example (Test on the mean)

Consider the following rule of decision:

NA: if m0 = 1.2, N = 100, σ2 = 1 and α = 5%.

Example (Power function)

De…nition (Most powerful test)

=> the Neyman–Pearson lemma

Lemma (Neyman Pearson)

Example (Test on the mean)

This expression can rewritten as:

where K2 = K1 N m12 m02 /2 is a constant.

Since m1 > m0 , we have

Fact (UMP one-sided test)

H0 : θ = θ 0 against H1 : θ > θ 0 (or H1 : θ < θ 1 )

the rejection region W of the UMP test is equivalent to the rejection

Example (Test on the mean)

Question: What is the rejection region of the UMP test of size α?