Graduate Econometrics I

Chapter 4: Asymptotic theory of M-estimators

Germain Van Bever

Lectures 10-11

1
General framework

◮ Assume that the q−dimensional random vectors z1 , z2 , . . . are i.i.d. from a

general parametric model, that is, share the same distribution function Fθ,τ for
θ ∈ Θ ⊆ Rp and τ ∈ T .

◮ We want to estimate θ, test hypotheses on θ, etc.

◮ The parameter τ is not of interest, i.e., it is a nuisance parameter.

◮ We denote EP or Eθ,τ the expectation w.r.t. the c.d.f. Fθ,τ . Often, we drop the
subscripts when there is no ambiguity.

◮ Denote θ0 and τ0 the “true parameters”. Keep in mind that θ0 and τ0 can be
anywhere in Θ and T , respectively.

2
Questions

◮ Given an estimator θ̂n (more precisely a sequence of estimators), we ask if this

estimator is consistent, i.e., if
p
θ̂n −
→ θ0 as n → ∞,

holds for every θ0 ∈ Θ and every τ0 ∈ T ; ...

◮ ... and if θ̂n is asymptotically normal, i.e., if

√ d
n(θ̂n − θ0 ) −
→ Np (0, Σ(θ0 , τ0 )) as n→∞

holds for every θ0 ∈ Θ and every τ0 ∈ T and some “asymptotic” variance-

covariance matrix Σ(θ0 , τ0 ).

3
M-estimators

◮ Consider estimators that minimize a (data-dependent) criterion function.

◮ More precisely estimators θ̂n = θ̂n (z1 , . . . , zn ) that solve

min Qn (z1 , . . . , zn , θ).

θ∈Θ

◮ Note that θ 󰀁→ Qn (z1 , . . . , zn , θ) is a “random function” as it depends on the

random variables z1 , . . . , zn .

4
Example: GMM estimator

◮ Setting: Suppose there is a function g : Rq × Θ → Rm so that

Eθ (g(z1 , θ)) = 0

(regardless of τ ), where for an integrable function h we set

󰁝
Eθ (h(z1 )) = h(z)dFθ,τ (z).
Rq

◮ Given a symmetric nonnegative definite (possible data-dependent) weighting

matrix Wn the GMM estimator θ̂n (based on this weighting matrix) is defined
as a solution (if it exists) of
󰀥 n 󰀦′ 󰀥 n 󰀦
1󰁛 1󰁛
min g(zi , θ) Wn g(zi , θ) .
θ∈Θ n n i=1
i=1

5
A general consistency result

Theorem 1
Suppose Q : Θ → R is a nonrandom function so that
p
sup |Qn (z1 , . . . , zn ; θ) − Q(θ)| −
→ 0,
θ∈Θ

and so that for every ε > 0 we have

inf Q(θ) > Q(θ0 ).

θ:󰀂θ−θ0 󰀂>ε

Then, any solution θ̂n of

min Qn (z1 , . . . , zn , θ)
θ∈Θ
p
satisfies θ̂n −
→ θ0 .

6
A consequence

Corollary 2 (Corollary “cons”)

Suppose Θ is compact. Assume Q : Θ → R is a nonrandom function that is contin-
uous on Θ, and that θ0 is the unique minimizer of Q. If
p
sup |Qn (z1 , . . . , zn ; θ) − Q(θ)| −
→ 0,
θ∈Θ

then any solution θ̂n of

min Qn (z1 , . . . , zn , θ)
θ∈Θ
p
satisfies θ̂n −
→ θ0 .

7
Lemma “GMMC”

Lemma 3 (without proof)

p
In the GMM setting, if Wn −→ W (non-stochastic), θ 󰀁→ E(g(z1 , θ)) is continuous,
Θ is compact, and if
󰀐 󰀐
󰀐 n 󰀐
󰀐 −1 󰁛 󰀐 p
sup 󰀐n g(zi , θ) − E(g(z1 , θ))󰀐 −
→ 0,
θ∈Θ 󰀐 i=1
󰀐

then
p
sup |Qn (z1 , . . . , zn , θ) − Q(θ)| −
→0
θ∈Θ

for Q(θ) = E(g(z1 , θ))′ W E(g(z1 , θ)), the latter being continuous.

8
Theorem “ULLN”

Theorem 4 (without proof)

Suppose Θ is compact. Let q : Z × Θ → R for Z ⊆ Rq the range space of z1 . Let
q(z, θ) be continuous in θ for every z ∈ Z, and let q(z1 , θ) be a random variable for
every θ ∈ Θ. If 󰀕 󰀖
E sup |q(z1 , θ)| < ∞,
θ∈Θ

then 󰀏 󰀏
󰀏1 󰁛 n 󰀏
󰀏 󰀏 p
sup 󰀏 q(zi , θ) − E(q(z1 , θ))󰀏 −
→ 0,
θ∈Θ 󰀏 n 󰀏
i=1

and θ 󰀁→ E(q(z1 , θ)) is continuous on Θ.

Remark: An analogous statement holds for q taking values in Rl or Rl×m if one

replaces the absolute values in the two displays by corresponding vector or matrix
norms.

9
Consistency of the GMM estimator

Theorem 5
In the GMM setting suppose that:
p
1. Wn −
→ W ; W symmetric, positive definite, and non-random.
2. Eθ0 (g(z1 , θ)) = 0 if and only if θ0 = θ.
3. Θ is compact.
4. g(z, θ) is continuous in θ for every z ∈ Z, the latter being the range space of
z1 , and g(z1 , θ) is a random variable for every θ ∈ Θ.
5. Eθ0 (supθ∈Θ 󰀂g(z1 , θ)󰀂) < ∞.
Then, every sequence of minimizers θ̂n of the GMM objective function
󰀥 n 󰀦′ 󰀥 n 󰀦
1󰁛 1󰁛
g(zi , θ) Wn g(zi , θ)
n i=1 n i=1

p
satisfies θ̂n −
→ θ0 .

10
Consistency of the MLE
Theorem 6
Let zi be i.i.d. with density f (z, θ) for θ ∈ Θ ⊆ Rp and z ∈ Z ⊆ Rq , where
󰁕
f (z, θ) > 0 for every z ∈ Z and every θ ∈ Θ, and Z f (y, θ)dy = 1 for all θ ∈ Θ.
Assume that:
1. Θ is compact;
2. the map θ 󰀁→ f (z, θ) is continuous on Θ for every z ∈ Z, and f (z1 , θ) is a
random variable for every θ ∈ Θ;
3. E(supθ∈Θ | log(f (z1 , θ))|) < ∞;
4. E (log (f (z1 , θ0 )/f (z1 , θ))) > 0 for every θ ∈ Θ so that θ ∕= θ0 .
Then, the MLE θ̂n , i.e., any solution to the minimization problem
n
󰁛
min n−1 [− log(f (zi , θ))],
θ∈Θ
i=1

p
satisfies θ̂n −
→ θ0 , the latter denoting the true parameter.

11
Mean-value-theorem

Proposition 7 (without proof)

Let f : Bδ (x0 ) → R be differentiable with gradient ∇f , where Bδ (x0 ) denotes the
ball of radius δ around x0 ∈ Rl . Let x1 ∈ Bδ (x0 ). Then, there exists an α ∈ (0, 1)
so that
f (x1 ) = f (x0 ) + ∇f (αx1 + (1 − α)x0 )(x1 − x0 ).

12
Asymptotic normality of the GMM estimator

Theorem 8
In the GMM setting, assume conditions (1)-(5) in the GMM consistency theorem
above, and suppose that:
1. θ 󰀁→ g(z, θ) is continuously differentiable for every z ∈ Z.
2. VC(g(z1 , θ0 )) =: Ω(θ0 ) is positive definite.
󰀃 󰀄
3. E supθ∈Θ 󰀂 ∂θ ∂
g(z1 , θ)󰀂 < ∞.
󰀃∂ 󰀄
4. E ∂θ g(z1 , θ0 ) := G(θ0 ) is so that G′ (θ0 )W G(θ0 ) is nonsingular.
5. θ0 is in the interior of Θ.
Then,
√ d
→ N(0, Σ(θ0 )),
n(θ̂n − θ0 ) −
where Σ(θ0 ) is defined as
−1 −1
[G′ (θ0 )W G(θ0 )] G′ (θ0 )W Ω(θ0 )W G(θ0 ) [G′ (θ0 )W G(θ0 )] .

13
Estimation of Σ(θ0 )

Proposition 9
Under the conditions of the previous theorem, and if
󰀕 󰀖
′
E sup 󰀂g(z1 , θ)g (z1 , θ)󰀂 < ∞,
θ∈Θ

then the estimator

󰁫 󰁬−1 󰁫 󰁬−1
Σ̂n = Ĝ′ Wn Ĝ Ĝ′ Wn Ω̂Wn Ĝ Ĝ′ Wn Ĝ

with
󰁛n n
󰁛
∂
Ĝ = n−1 g(zi , θ̂n ) and Ω̂ = n−1 g(zi , θ̂n )g ′ (zi , θ̂n ),
i=1
∂θ i=1

p
satisfies Σ̂n −
→ Σ(θ0 ).

14
Further result

Lemma 10 (without proof)

In the context of the asymptotic normality result for GMM estimators, it holds that
󰀅 󰀆−1
Σ(θ0 ) ≥ G′ (θ0 )Ω−1 G(θ0 ) .

◮ Note that the right-hand-side is the asymptotic covariance matrix with weight-
ing matrix proportional to Ω−1 .

◮ Setting Wn equal to a consistent estimator of Ω−1 one obtains the efficient

GMM estimator.

◮ For estimating Ω−1 one typically needs an estimator for θ0 in the first place,
leading to two-step GMM, iterated GMM, or continuously-updated GMM esti-
mators.

15
J-test

◮ In case m > p one can use the so-called J-test to test the hypothesis whether
there exists a θ ∈ Θ so that

E(g(z1 , θ)) = 0,

against the alternative that

E(g(z1 , θ)) ∕= 0 for every θ ∈ Θ.

◮ The test statistic is

󰀥 n
󰀦′ 󰀥 n
󰀦
󰁛 󰁛
−1 −1 −1
n n g(zi , θ̂n ) (Ω̂) n g(zi , θ̂n ) ,
i=1 i=1

which converges (under some conditions) under the null hypothesis in distribu-
tion to a χ2 distributed random variable with m − p degrees of freedom.

16
A general asymptotic normality result for M-estimators

Theorem 11 (without proof)

Suppose θ̂n minimizes Qn (z1 , . . . , zn , .) over Θ and:
p
1. θ̂n −
→ θ0 .
2. θ0 is in the interior of Θ.
3. Qn (z1 , . . . , zn , θ) is twice continuously differentiable in θ.
√ ∂ d
4. n ∂θ Qn (z1 , . . . , zn , θ0 ) −
→ N (0, Σ(θ0 )).
5. there exists a nonrandom function H(.) which is continuous at θ0 and so that
󰀐 󰀐
󰀐 ∂ 󰀐 p
sup 󰀐
󰀐 ′
Q (z
n 1 , . . . , z n , θ) − H(θ) 󰀐−
󰀐 → 0.
θ∈θ ∂θ∂θ

6. H(θ0 ) is nonsingular.
√ d
Then n(θ̂n − θ0 ) −→ N (0, H(θ0 )−1 Σ(θ0 )H(θ0 )−1 ).

17