Module 1 | Lesson 1 (Part 2)

Weighted Least Squares

Weighted Least Squares
By the end of this video, you will be able to…


• Derive the weighted least squares criterion given varying

measurement noise variance


• Compare weighted least squares to ordinary least squares

Method of Weighted Least Squares
• Suppose we take measurements with multiple multimeters, some of which are
better than others

vs.
Method of Weighted Least Squares
Consider the general linear measurement model for m measurements and n
unknowns:
y1 x1 v1
⋮ =H ⋮ + ⋮
ym xn vm

y = Hx + v
In regular least squares, we implicitly assumed that each noise term was of equal
variance:
σ2 0
2 2
𝔼[vi ] =σ (i = 1,…, m) R = 𝔼[vvT ] = ⋱
2
0 σ
Method of Weighted Least Squares
If we assume each noise term is independent, but of different variance,
σ12 0
2 2
𝔼[vi ] = σi , (i = 1,…, m) T
R = 𝔼[vv ] = ⋱
2
0 σm
Then we can define a weighted least squares criterion as:

ℒWLS(x) = e R e T −1 e1 y1 x1
e12 e22 em2 where ⋮ =e= ⋮ −H ⋮
= + +…+ em ym xn
σ12 σ22 σm2

The higher the expected noise, the lower the weight we place on the
measurement.
Re-deriving regular least squares
What happens if all of our variances are the same?
2 2 2
e1 e2 em
ℒWLS(x) = + +…+
σ2 σ2 σ2
1 2 2
= 2 (e1 + … + em)
σ

Since our variance is constant, it will not affect our final estimate! This results in
the same estimate as regular least squares:
2
σ1 = 2
σ2 = 2
…σm = σ → x̂WLS = x̂LS = argmin x ℒLS(x) = argmin x ℒWLS(x)
2
Minimizing the Weighted Least Squares
Criterion
Expanding our new criterion,
ℒWLS(x) = eT R−1e
T −1
= (y − Hx) R (y − Hx)

We can minimize it as before, but accounting for the new weighting term:

∂ℒ T −1 T T −1
x̂ = argmin x ℒ(x) ̂
=0=−y R H+x H R H
∂x x=x̂

HT R−1Hx̂WLS = HT R−1y
The weighted normal equations!
Method of Weighted Least Squares
The weighted normal equations

x̂ = (HT R−1H)−1HT R−1y

Let’s adapt our example:

Resistance Measurements (Ohms)

# Multimeter A ( σ = 20 Ohms ) Multimeter B ( σ = 2 Ohms )
1 1068
2 988
3 1002
4 996
Method of Weighted Least Squares
σ12
Once we define the 1 1068 400
1 988 σ22 400
relevant quantities, we H= y= R= =
plug-and-chug to get: 1 1002 σ32 4
1 996 4
σ42
̂xWLS = (HT R−1H)−1HT R−1y
−1
−1 −1
400 1 400 1068
400 1 400 988
= [1111] [1111]
4 1 4 1002
4 1 4 996

1/400 + 1/400 + 1/4 + 1/4 ( 400 4 )

1 1068 988 1002 996
= + + +
400 4
= 999.3 Ohms
Ordinary versus Weighted Least Squares

Least Squares Weighted Least Squares

T T −1
Loss / Criterion ℒLS(x) = e e ℒWLS(x) = e R e

Solution ̂xLS = (HT H)−1HT y ̂xWLS = (HT R−1H)−1HT R−1y

m≥n m≥n
Limitations
σi2 > 0

Accurate noise modelling is crucial to utilize various sensors effectively!

Summary | Weighted Least Squares

• Measurements can come from sensors that have different

noisy characteristics
• Weighted least squares lets us weight each measurement
according to noise variance