4.

Method of Moment Estimators

Ismaı̈la Ba

ismaila.ba@umanitoba.ca
STAT 3100 - Winter 2024

1 / 14
Methodology

Let X1 , . . . , Xn be a random sample from a distribution with pmf or

pdf f (x; θ), where θ = (θ1 , . . . , θp ) are parameters whose values are
unknown.
The population moments µ′1 , . . . , µ′p will generally be functions of
the parameters θ1 , . . . , θp :

µ′1 = µ′1 (θ1 , . . . , θp )

µ′2 = µ′2 (θ1 , . . . , θp )
..
.
µ′p = µ′p (θ1 , . . . , θp ).

We recall that µ′k = E(X k ) for k = 1, . . . , p.

2 / 14
Methodology (2)

The sample moments M1 , . . . , Mp are consistent and unbiased

estimators of µ′1 , . . . , µ′p . We plug these (sample moments) in on the
left hand side and substitute θ̂1 , . . . , θ̂p for θ1 , . . . , θp to obtain

M1 = µ′1 (θ̂1 , . . . , θ̂p )

M2 = µ′2 (θ̂1 , . . . , θ̂p )
..
.
Mp = µ′p (θ̂1 , . . . , θ̂p ).

for k = 1, . . . , p.
1 Pn k
We recall that Mk = n i=1 Xi

3 / 14
Methodology (3)
This results in p equations in p unknowns. Solving for the θ̂i ’s will
yield the moment estimators θ̂1 . . . , θ̂p
θ̂1 = θ̂1 (M1 , . . . , Mp )
θ̂2 = θ̂2 (M1 , . . . , Mp )
..
.
θ̂p = θ̂p (M1 , . . . , Mp ).

Takeaway Message
The basic idea of the method of moments is to equate certain sample
characteristics, such as the mean, to the corresponding population
expected values. Then solving these equations for unknown parameter
values yields the estimators.
4 / 14
Examples - Bernoulli distribution

Example 1
Suppose that X1 , . . . , Xn are iid Bernoulli trials with unknown success
probability p. Since there is only one parameter to be estimated, the
estimator is obtained by equating µ′1 to M1 . Since E(X ) = p (so
µ′1 = µ′1 (p) = p) and M1 = (X1 + . . . + Xn )/n, we next write X̄n = p̂.
Solving for p̂ yields p̂ = X̄n .
The moment estimator of p is X̄n . Since E(X̄n ) = p and
V(X̄n ) = p(1 − p)/n, we have that X̄n is unbiased and (mean squared
error) consistent for p.

5 / 14
Examples - Generalized negative binomial distribution

Example 2
Let X1 , . . . , Xn be a random sample from a generalized negative binomial
distribution with parameters r and p. Note that E(X ) = r (1 − p)/p and
V(X ) = r (1 − p)/p 2 . We have E(X 2 ) = r (1 − p)(r − rp + 1)/p 2 .
Here µ′1 = r (1 − p)/p and µ′2 = r (1 − p)(r − rp + 1)/p 2 . Thus, we set

r̂ (1 − p̂)
M1 =
p̂
r̂ (1 − p̂)(r̂ − r̂ p̂ + 1)
M2 = .
p̂ 2
Solving for p̂ and r̂ yields

X̄n (X̄n )2
p̂ = 1 Pn 2
r̂ = 1 Pn 2
2 − (X̄n )2 − X̄n
n i=1 Xi − (X̄n ) n i=1 Xi

6 / 14
Examples - Exponential distribution
Example 3

Let X1 , . . . , Xn be a random sample from an exponential distribution (“rate”

parameterization) with parameter λ. Since E(Xi ) = 1/λ, it follows that
µ′1 = 1/λ so that M1 = 1/λ̂ and λ̂ = 1/M1 = n/ ni=1 Xi is the moment
P
estimator of λ.
Note that Y := ni=1 Xi has a GammaR (n, λ) distribution with density
P

λn y n−1 e −λy
fY (y ) = 1(0,∞) (y ).
Γ(n)

Set V = 1/Y . We take v = g (y ) = 1/y so that y = 1/v and

dy /dv = −1/v 2 . The transformation is monotonically decreasing (larger Y
yields smaller V ). Using the transformation theorem, we have

λn e −λ/v
fV (v ) = fY (1/v ) · | − 1/v 2 | = 1(0,∞) (v ).
v n+1 Γ(n)
7 / 14
Examples - Exponential distribution
Example 3 continued
fV (v ) is the density of an Inverse Gamma distribution with scale
parameter λ and shape parameter n. ( Verify that) E(V ) = λ/(n − 1) for
n > 1 and V(V ) = λ2 /[(n − 1)2 (n − 2)] for n > 2. Thus, we have
!
1 n n
E(λ̂) = E =E = E(nV ) = λ for n > 1
M1 Y n−1
n2
V(λ̂) = V (nV ) = n2 V(V ) = λ2 for n > 2.
(n − 1)2 (n − 2)

The bias is B(λ̂) = λ/(n − 1) → 0 as n → ∞, and the mean squared error is

n2 λ2
MSE(λ̂) = λ2
+ →0 as n → ∞.
(n − 1)2 (n − 2) (n − 1)2

Conclusion : λ̂ is asymptotically unbiased and is MSE consistent for λ.

8 / 14
Examples - Gamma distribution

Example 4

Let X1 , . . . , Xn be a random sample from a gamma distribution with

parameters α and β. Then, µ′1 = E(Xi ) and µ′2 = E(Xi2 ) = α(α + 1)β2 . The
moment estimators of α and β are obtained by solving

M1 =α̂β̂
M2 =α̂(α̂ + 1)β̂2 .

Since α̂(α̂ + 1)β̂2 = α̂2 β̂2 + α̂β̂2 and the first equation implies M12 = α̂2 β̂2 ,
the second equation becomes

M2 = M12 + α̂β̂2 .

9 / 14
Examples - Gamma distribution
Example 4 continued
Now dividing each side of this second equation by the correspond side of
the first equation and substituting back gives the estimators

M12 M2 − M12
α̂ = β̂ = .
M2 − M12 M1

Note that
n
 n 2 
n
 n 2 
1 X 2 1 X  1 X 2 1 X   n − 1 2
M2 − M12 = X − 2  Xi  =  Xi −  Xi   = S .
n i=1 i n i=1 n i=1 n i=1 n n

The moment estimators α̂ and β̂ become

n X̄n2 n − 1 Sn2
α̂ = β̂ = .
n − 1 Sn2 n X̄n
10 / 14
Examples - Two parameter exponential distribution

Example 4 continued
Note that α̂β̂ = X̄n and α̂β̂2 = [(n − 1)/n]Sn2 . Based on Chapter 3, we can
show α̂ is consistent for E2 (X )/V(X ) and β̂ is consistent for V(X )/E(X ).
Furthermore, α̂β̂ is consistent for E(X ) and α̂β̂2 is consistent for V(X ).

Example 5

The two parameter exponential distribution, denoted Exp(β, η), has density

1 −(x−η)/β
f (x) = e 1(η,∞) (x).
β

If X ∼ Exp(β, η), then E(X ) = η + β and V(X ) = β2 so that

E(X 2 ) = β2 + (η + β)2 . Now, let X1 , . . . , Xn be a random sample from a
two parameter exponential distribution with parameters β and η.

11 / 14
Examples - Two parameter exponential distribution

Example 5 continued
Therefore, we have

µ′1 = η + β
µ′2 = β2 + (η + β)2 .

The method of moments equations become

M1 = η̂ + β̂
M2 = β̂2 + (η̂ + β̂)2 .
q
We have that M2 = β̂2 + M12 so that β̂ = M2 − M12 . Substituting this
q
into the first equation yields η̂ = M1 − M2 − M12 .

12 / 14
Examples - Two parameter exponential distribution

Example 5 continued
From the previous example, we have M2 − M12 = (n − 1)Sn2 /n so that we
P
can write η̂ = X̄n − (n − 1)/nSn . Now, since X̄n → β + η and
p
P P P
(n − 1)/nSn → β, we have η̂ → η. We also have β̂ → β. However, neither
p

of these are unbiased.

13 / 14
Exercises

Exercise 1
Derive the mean and variance of the inverse gamma distribution, assuming
n > 2.

Exercise 2
iid
Suppose that X1 , . . . , Xn ∼ Exp(β). Find the method of moments estimator
for β and determine if the estimator is biased (or asymptotically unbiased,
or unbiased). Determine the MSE of β̂ and and determine if β̂ is consistent.

Exercise 3
Suppose, in Example 5, that β is known. Does the method of moments
estimator of η change ?

14 / 14