THE GAUSS–MARKOV THEOREM

Submitted To:
Dr. Amir Raza
Submitted By:
Ayat Mahfooz
BS 19031

Department of Statistics
 Government College Women University, Sialkot Pakistan
The Gauss Markov theorem says that, under certain conditions, the ordinary least squares (OLS)
estimator of the coefficients of a linear regression model is the best linear unbiased estimator
(BLUE), that is, the estimator that has the smallest variance among those that are unbiased and
linear in the observed output variables.

The theorem was named after Carl Friedrich Gauss and Andrey Markov, although Gauss' work
significantly predates Markov's. But while Gauss derived the result under the assumption of
independence and normality, Markov reduced the assumptions to the form stated above. A further
generalization to non-spherical errors was given by Alexander Aitken.

Statement:
In statistics, the Gauss–Markov theorem states that the ordinary least squares (OLS) estimator has
the lowest sampling variance within the class of linear unbiased estimators, if the errors in the
linear regression model are uncorrelated, have equal variances and expectation value of zero. The
errors do not need to be normal, nor do they need to be independent and identically distributed
(only uncorrelated with mean zero and homoscedastic with finite variance). The requirement that
the estimator be unbiased cannot be dropped, since biased estimators exist with lower variance.
See, for example, the James–Stein estimator (which also drops linearity), ridge regression, or
simply any degenerate estimator.

Assumptions:
The Gauss–Markov theorem holds when we adhere to the four assumptions of OLS: a)

linearity

b) no multicollinearity
c) strict exogeneity
d) spherical errors
If we make these four assumptions, then β^ is BLUE, the best (minimum-variance) linear unbiased
estimator.

Applications:
The Gauss-Markov theorem states that, under the usual assumptions, the OLS estimator βOLS is
BLUE (Best Linear Unbiased Estimator). To prove this, take an arbitrary linear, unbiased estimator
ˉβ of β. Since it is linear, we can write ˉβ=Cy in the model y=βX+ε. The Gauss Markov
assumptions guarantee the validity of ordinary least squares for estimating regression coefficients.
Checking how well our data matches these assumptions is an important part of estimating
regression coefficients.
Proof:
Since β = By is an unbiased estimator, it follows that E (β ) = BE(y) = BXβ = β, which implies
that BX = I. Now let us write B = (X′ X)−1X′ + G. Then BX = I implies that GX = 0. It follows that

(1) D(β ) = BD(y)B′

= σ2 {(X′ X) -1X′ + G} {X(X′ X)-1 + G′}
= σ2 (X′ X)-1 + σ2GG′
= D (βˆ) +σ2GG′

Therefore, for any constant vector p of order k, there is the identity (2)

V (p′ β ) = p′ D (βˆ) p + σ2p′ GG′ p

≥ p′D (βˆ) q = V (p′ βˆ);

And thus the inequality V (p′ β ) ≥ V (p′ βˆ) is established.

The tactic of taking arbitrary linear combinations of the elements of βˆ is to avoid the difficulty
inherent in the fact that βˆ is a vector quantity for which there is no uniquely defined measure of
dispersion. An alternative approach, which is not much favored, is to use the generalized variance
that is provided by the determinant of the D (β). A version of the Gauss–Markov Theorem that
uses this measure can also be proved. It is worthwhile to consider an alternative statement and
proof of the theorem, which also considers the variance of an arbitrary linear combination of the
elements of βˆ.

Alternative Statement:
Let q′ y be a linear estimator of the scalar function p′ β of the regression parameters in the model
(y; Xβ, σ2I). Then q′ y is an unbiased estimator, such that E (q′ y) = q′ E(y) = q′ Xβ = p′ β for all β,
if and only if q′ X = p′. Moreover, q′ y has the minimum variance in the class of all unbiased linear
estimators if and only if q′ y = q′ X(X′ X) −1X′ y = p′ (X′ X) −1X′ y

Therefore, since p is arbitrary, it can be said that βˆ = (X′ X) −1X′ y is the minimum variance
unbiased linear estimator of β.

Proof:
It is obvious that q′ X = p′ is the necessary and sufficient condition for q′ y to be an unbiased
estimator of p′ β. To find the unbiased estimator of minimum variance, consider the following
criterion:
Minimize V (q′ y) = q′D(y) q = σ2q′q

Subject to E (q′ y) = p′ β or, equivalently X′ q = p.

To evaluate the criterion, we form the following Lagrange an expression:

L = q′ q − 2λ′ (X′q − p)

Differentiating with respect to q and setting the results to zero gives, after some rearrangement, the
condition q′ = λ′ X′

Post multiplying by X gives q′ X = p′ = λ′ X′ X, whence λ′

= q′ X (X′ X)-1 = p′ (X′ X) −1

On post multiplying by y and on substituting the expression for λ′, we get

q′ y = q′ X(X′ X) −1X′y

= p′ (X′ X)−1 X′ y = p′ β

This proves the second part of the theorem.