Lecture 2

Linear Inverse Problems and

Introduction to Least Squares

Surprise Quiz!
Lemma (you can assume)

( a b) 2 a 2 b 2

x y
x
y

Given the Lemma, prove by induction the

following inequality:
(a1 a2 ... an ) 2 a12 a22
an2

...
x1 x2 ... xn
x1 x2
xn

Once you have proven the aforesaid

inequality, derive the Cauchy Schwartz
Inequality from it.

Solution

( a b)
a
b

is given
x y
x
y
2

Substitute, b = b+c and y = y+z

(a b c)2 a 2 (b c)2 a 2 b 2 c 2

,follows from the Lemma

x yz
x
yz
x
y
z

Thus,

(a1 a2 )2 a12 a22

is given

x1 x2
x1 x2

And,

(a1 a2 a3 ) 2 a12 a22 a32

has been proven

x1 x2 x3
x1 x2 x3

Thus if for n-1 if we have,

(a1 a2 ... an 1 )2 a12 a22

a2

... n 1 , then we can substitute

x1 x2 ... xn 1
x1 x2
xn 1

an 1 an 1 an and xn 1 xn 1 xn and prove it for n.

Thus the proof follows by induction

Solution to Problem 1: part 2

Put ak k k and xk k2 in the above inequality
(11 2 2 ... n n ) 2 (11 ) 2 ( 2 2 ) 2
( n n ) 2

...

12 22 ... n2
12
22
n2
(11 2 2 ... n n ) 2 (12 22 ... n2 )( 12 22 ... n2 )
2

2
2
i i i i , QED
i
i
i

How many solutions?

Case 1
Case 2

y1 ax1 b
y1 ax1 b
y2 ax2 b

Case 3

y1 ax1 b
y2 ax2 b
y3 ax3 b

How many solutions?

Case 1
Case 2

y1 ax1 b
y1 ax1 b

Infinite
One

y2 ax2 b

Case 3

y1 ax1 b
y2 ax2 b
y3 ax3 b

None

Under-determined System
y1 ax1 b

Matrix-Vector Notation:
a
y1 x1 1
b

The system is Fat

Y

Perfectly Determined System

y1 ax1 b

y1 x1 1 a
y x 1 b
2 2

y2 ax2 b

The system is Square

Over-Determined System
y1 x1 1
y x 1 a
2 2 b
y3 x3 1

y1 ax1 b
y2 ax2 b
y3 ax3 b

The system is Tall

Y

Curve Fitting in Noisy Observation

Least Squares: Line

The observation is noisy
yi axi b i , i
Fit a line that minimizes the sum of least
squared error, i yi (axi b), i i.e.
min yi ( axi b)

a ,b

Like any other minimization, set gradient = 0

Least Squares: Line contd.

yi (axi b)

yi (axi b)

Least Squares: Line contd.

yi (axi b)

2 yi (axi b) 0
i

yi (axi b)

2 xi yi (axi b) 0
i

Least Squares: Line contd.

yi a xi b1,
i

2
x
y

a
x
i i i b xi

We have the solution:

a

xi yi

xi

yi

yi

xi2

xi

2
xi

xi

xi yi

2
xi

xi

Homework
Find the least squares fit for a second degree
polynomial of the form,

y ax bx c
2

General Form
y1

Y ...

yn

x1p

...
xnp

p 1
1

...
p 1
n

...
a1
...
...

an
...

The least squares problem is expressed as:

min Y

2
2

Solution
Y

2
2

(Y )T (Y )

Solution
Y

2
2

(Y ) (Y )
T

Y T Y Y T T T Y T T
Taking the gradient
Y

2
2

Solution
Y

2
2

(Y ) (Y )
T

Y T Y Y T T T Y T T
Taking the gradient
Y

2
2

T Y T Y 2 T 0
The Normal Equations
T LS T Y LS ( T ) 1 T Y

Properties of Least Squares

LS is a linear function of Y

T
1
T
Pseudo Inverse: ( )
This is also the left inverse of a tall matrix ,

T
1
T
i.e. ( ) I