Chapter 5

Orthogonality

5-1
Yi-Hsin Pang
§5.1 Inner Product in ℝ
∀ = , , and = , ,
 Inner product (or scalar product) of and
, ≜ = + +
 Norm (Euclidean length) of

≜ , = + + −

 Distance between and

, ≜ − = − + − + −

5-2
Yi-Hsin Pang
 θ : Angle between and ( , ≠ )
,
cos = θ

 Projection ( ≠ )
 Scalar projection of onto
,
= cos =
θ
 Vector projection of onto
, ,
= = =
,
5-3
Yi-Hsin Pang
 §5.4, §6.4 Inner Product Spaces
 The conjugate transpose (or adjoint, Hermitian) of
a complex-valued matrix A, denoted by , is
defined by
≜ ∗ for any ∈ ℂ'×)
where ∗ = complex conjugate of A
 e.g.
+ 2 −+ 2 −+ 3++
• .= → .∗ = →. =
3−+ 6 + 7+ 3++ 6 − 7+ 2 6 − 7+
1−+
= → = 1 + + −2+ , c.f. = 1−+ 2+
2+
•

→ = 1 − + + 2+ = 6 > 0
= 1−+ + 2+ = −4 − 2+ 5-4
Yi-Hsin Pang
Definition of Inner Product Space
 V: a vector space over 5 (5 = ℝ 78 ℂ)
 Inner product ⋅,⋅ : a mapping from V × V to5
⋅,⋅
, ∈; , ∈5

For all , , < ∈ ;, α, > ∈ 5, ⋅,⋅ satisfies:

(i) , ≥ 0. , =0↔ = B
(ii) , = , ∗ (If 5 = ℝ, , = , )
(iii) +> ,< = ,< + > ,<
 A vector space V with an inner product ⋅,⋅ is
called an inner product space. Notation: (V, ⋅,⋅ )
5-5
Yi-Hsin Pang
 (ii) + (iii)  <, +> = ∗
<, + > ∗ <,
pf: <, +> = + > ,< ∗

= ,< + > ,< ∗

= ∗ , < ∗ + >∗ , < ∗

= ∗ <, + > ∗ <,

5-6
Yi-Hsin Pang
Examples of Inner Product Spaces
 ℝ) : ∀ = , … , ) and = ,…, )
 , ≜ = = + + ⋯+ ) )
 Given E = F , … , F) with F , … , F) , > 0.
(F , … , F) are called weights),
, ≜F +F + ⋯ + F) ) )
 ℂ) : ∀ = , … , ) and = ,…, )
∗ ∗
 , ≜ = + + ⋯ + ) )∗
→ , = = + +⋯+ ) ≥0
, =0↔ = 0, … , 0
, = = ∗ ∗ = ∗ = , ∗
5-7
Yi-Hsin Pang
 C[a, b]: f, g ∈ C[a, b], complex-valued functions
J J
 G, H ≜ IK G H∗ L = IK G H L

if f, g are real-valued

 Given a positive function w ∈ C[a, b] (w is

called a weight function)
J
G, H ≜ IK F G H∗ L

5-8
Yi-Hsin Pang
Some Definitions Derived from Inner Product
(V, ⋅,⋅ ): an inner product space (We simply say V is
an inner product space hereafter)
∀ , ∈;
 Norm (or length) of : ≜ ,
 θ : Angle between and ( , ≠ )
,
cos =

 Scalar and vector projections of onto ( ≠ )

, , ,
= cos = = = =
,
5-9
Yi-Hsin Pang
 Orthogonality
 If , = 0  angle θ = 90°between and . 
we say that and are orthogonal. Notation: ⊥
 , ≠ and ⊥  and are linearly
independent.
 : vector projection of onto
θ
= + − ⊥ −
 Pythagorean theorem
⊥ → + = +
pf: ∵ + = + , + = , + , + , + ,
⊥ → , = , =0
→ + = , + , = + 5-10
Yi-Hsin Pang
 Review of Some Differentiations and
Integrals
L QU
T = PT QU I T QU L = Q T QU + S (P ≠ 0)
L
L
sin P = P cos P I cos P L = sin P + S (P ≠ 0)
L Q

L
cos P = −P sin P I sin P L = − cos P + S (P ≠ 0)
L Q

I cos V cos W L = I cos V − W L + I cos V + W L

1 1
= sin V − W + sin V + W +S V±W ≠0
2 V−W 2 V+W

I sin V sin W L = I cos V − W L − I cos V + W L

1 1
= sin V − W − sin V + W +S V±W ≠0
2 V−W 2 V+W 5-11
Yi-Hsin Pang
I sin V cos W L = I sin V + W L + I sin V − W L
−1 −1
= cos V + W + cos V − W +S V±W ≠0
2 V+W 2 V−W

1 + cos 2 1 1
Y cos V L = Y L = + sin 2 +S
2 2 4

1 − cos 2 1 1
Y sin V L = Y L = − sin 2 +S
2 2 4

5-12
Yi-Hsin Pang
Example – Fourier Series (Preview)
 Z ≜ {1, cos , sin , cos 2 , sin 2 , … , cos W , sin W }
 For ∈ −^, ^ , ; ≜ span Z
 For any f, g ∈ V, (Note: f and g are 2π-periodic functions)
`
G, H ≜ Ia` G H L

 For any m, k = 1, 2, …, n, and m ≠ k

` Uc`
 1, cos P = Ia` 1 ⋅ cos P L = sin P b =0
Q Uca`
`
 cos V , cos P = Ia` cos V cos P L =0
`
 sin V , sin P = Ia` sin V sin P L =0
`
 sin V , cos P = Ia` sin V cos P L =0
Z is a basis for V 5-13
Yi-Hsin Pang
  For any k = 1, 2, …, n
 d
` Uc` = 2^
= 1,1 = Ia` 1 L = |Uca`
`
 cos P = Ia` cos P L =^
`
 sin P = Ia` sin P L =^
 For any f ∈ V,
f U , f U , hij QU f U , jkl QU
G = 1 + ∑)Qc cos P + sin P
hij QU jkl QU

` `
= I G L + ∑)Qc m Ia` G n cos Pn Ln ⋅ cos P
` a` `
`
+ Ia` G n sin Pn Ln ⋅ sin P ]
`

= pqrs + ∑)Qc rQ cos P + tQ sin P 5-14

Yi-Hsin Pang
Fourier Series – 數學中美麗的詩篇
 Summary: For any f ∈ V,
G = pqrs + ∑)Qc rQ cos P + tQ sin P
`
rQ = Ia` G cos P L , P = 0, 1, … , W
`
`
tQ = Ia` G sin P L , P = 1, … , W
`

 ak, bk: Fourier coefficients

 n → ∞ : Fourier series of a continuous 2π-periodic
functions
 Finite n : An approximation of the Fourier series of a
continuous 2π-periodic functions

5-15
Yi-Hsin Pang
 Properties of Norm
∀ , ∈ ; (; is an inner product space)
 Cauchy-Schwarz Inequality: , ≤
 Triangle Inequality: + ≤ +
Proof of Cauchy-Schwarz Inequality:
1∘ If = → hold
2∘ Assume ≠ . For any scalar • ∈ 5
0≤ −• = , − ,• − • , + • ,•
,• + • , = •∗ , + • ,
, ,
Set • = →0≤ −• = −

“=” holds ↔ =• 5-16

Yi-Hsin Pang
Frobenius Norm
 = r‚ƒ ∈ ℝ'×) and „ = t‚ƒ ∈ ℝ'×)
, „ ≜ ∑'
‚c ∑ ƒc r‚ƒ t‚ƒ
)

→ … = , = ∑'
‚c ∑ ƒc r‚ƒ
)
: Frobenius norm of

1 1 −1 1
 Example:
.= 1 2 ‡= 3 0
3 3 −3 4

, „ = −1 + 3 − 9 + 1 + 0 + 12 = 6
„ … = 1 + 9 + 9 + 1 + 0 + 16 = 6
5-17
Yi-Hsin Pang
 §5.2 Orthogonal Subspaces – Example
1 2 Š
1 2 3
.= 2 4 = Š Š ∈ℝ × .• = = ∈ℝ ×
2 4 6 Š
3 6

 ‰ = span Š , Š = span 1 2 3
 ‹ : ∈‹ ↔ = ↔ ,Š = ,Š =0
1 2 3 +2 +3 Š ,Š
= = = =
2 4 6 2 +4 +6 Š ,Š

2 3
+2 +3 =0→ = −1 + > 0
0 −1
‹ = span 2 −1 0 , 3 0 −1
5-18
Yi-Hsin Pang
Example (Cont.)
 ‰ = span Š , Š = 1 2 3
‹ = span 2 −1 0 , 3 0 −1 ≜ span{Ž̅ , Ž̅ }
Note that: Š , Ž̅ = Š , Ž̅ =
∀• ∈ ‰ , ∀‘ ∈ ‹ , •, ‘ =
We say that R(A) and N(AT) are orthogonal, denoted by
‰ ⊥‹
Note that: ‰ ∩‹ =

5-19
Yi-Hsin Pang
 Example (Cont.)
The orthogonal complement of a set S, denoted by
Z “ , is defined by


Z“ ≜ , = 0, ∀ ∈ Z
‰ “ =‹
‹ “ =‰

[Exercise] Z = 1, 0 → Z “ =? Z“ “
=? Z“ “
= Z? 5-20
Yi-Hsin Pang
Example (Cont.)
 A basis for ‰ = 1 2 3 ≜•
A basis for ‹ = 2 −1 0 , 3 0 −1 ≜•
• ∪• = 1 2 3 , 2 −1 0 , 3 0 −1 :
a basis for ℝ
∀ ∈ ℝ , there exist unique • ∈ ‰ and ‘ ∈ ‹ s.t.
=•+‘
We say that ℝ is the direct sum of ‰ and ‹
Notation: ℝ =‰ ⨁‹

5-21
Yi-Hsin Pang
 Example (Cont.)
1 2 Š
1 2 3
.= 2 4 = Š Š ∈ℝ × .• = = ∈ℝ ×
2 4 6 Š
3 6
 Similarly, we have
‰ = span 1, 2 ⊥‹ = span 2, −1
‰ “
=‹ ‹ “
=‰
ℝ =‰ ⨁‹
 In general, for any ∈ ℝ'×)
‰ ⨁‹ = ℝ'

‰ ⨁‹ = ℝ)

5-22
Yi-Hsin Pang
Example – Least-Norm Solution
 Find the least-norm solution of A = Ž̅
1 2 1
.= 2 4 , Ž̅ = 2
3 6 3

5-23
Yi-Hsin Pang
A Useful Theorem
 For any ∈ ℂ) , ∈ ℂ' , A ∈ ℂ'×)
A , = ,A

5-24
Yi-Hsin Pang
§5.3 Least Square Problems
Consider the problem A = Ž̅, where A ∈ ℝ'×) ,
∈ ℝ) , and Ž̅ ∈ ℝ'


Ž̅ ∈ ‰ ?
No Yes

No sol ‹ ={ ) }?
No Yes

Least square sol General sol Unique sol

= ™+ š = ™

™ ∈‰ , A ™ = Ž̅
š ∈‹ ,A š = '
5-25
Yi-Hsin Pang
Problem of Least Square Solution
System of equation: A = Ž̅, A ∈ ℝ'×) , ∈
ℝ) , and Ž̅ ∈ ℝ'


 Residual vector: ›̅ ≜ Ž̅ − A for each ∈ ℝ)

 Find a least square solution œ s. t. ›̅ is
minimized

5-26
Yi-Hsin Pang
 Least Square Solution
≜ the vector projection of Ž̅ onto R(A)  is
the vector closest to Ž̅ in R(A)


Ž̅
 A = is always consistent
 ›̅ ≜ Ž̅ − ⊥ R(A) → ›̅ ∈ ‹
→ = ›̅ = Ž̅ − = Ž̅ − A
R(A)
•

Least square solution œ of A = Ž̅ is the solution of

= Ž̅  called the normal equation

5-27
Yi-Hsin Pang
Least Square Solution (Cont.)
If A ∈ ℝ'×) and rank(A) = n, then the normal
= Ž̅ has a unique solution

equation
œ= ad Ž̅
which is the unique least square solution of A = Ž̅
pf: 1. Show that ATA is nonsingular
Let <= → <∈‹
But < ∈ ‰ ⊥‹ → <= ž
∵ rank(A) = n → < = • → ATA is nonsingular
2. ∵ ATA is nonsingular
∴œ= ad
Ž̅ is the unique sol. of = Ž̅
5-28
In fact, rank(A) = rank (Exercise) Yi-Hsin Pang
 Least Square Solution (Cont.)
 The vector projection of Ž̅ onto R(A):
= œ= ad Ž̅
The matrix P ≜ ad is called the projection matrix

5-29
Yi-Hsin Pang
The Meaning of Projection Matrix

5-30
Yi-Hsin Pang
 Example
 Find the least square solution œ
1 1 3
−2 3 = 1
2 −1 2

Ž̅
Sol:
Solve the normal equation: =
1 1 1 1
9 −7 5
= −2 3 −2 3 = Ž̅ =
−7 11 4
2 −1 2 −1

83 71
œ= ,
50 50
5-31
Yi-Hsin Pang
Best Fitting Polynomial
 Given data
x x1 x2 … xm
y y1 y2 … ym
 Find the best fitting polynomial
y = p(x) = c0 + c1 x + c2 x2 + … + cn xn n<m

•s + • +• + ⋯ + •) )
=
…

•s + • ' +• ' + ⋯ + •)
)
' = '

5-32
Yi-Hsin Pang
Best Fitting Polynomial
 The equivalent problem: find the least square
solution of
•s
1 ⋯ )
•
1 ⋯ )
• =
1 ⋮ ⋮ ⋮ ⋮
⋮
1 ' ' ⋯ )
' •) '

Vandermonde matrix

5-33
Yi-Hsin Pang
Example
 Find the quadratic polynomial p(x) = c0 + c1 x + c2
x2 that gives the best least squares fit to the data in
Table I.
x –1 0 1 2
y 0 40 –60 80

(Ans: p(x) = – 17 – 11 x + 25 x2 )

5-34
Yi-Hsin Pang
Example – Solution
1 −1 1 •s 0
1 0 0 = • 40
= Ž̅ =
1 1 1 • −60
1 2 4 80
Solve the normal equation: = Ž̅
4 2 6 60
= 2 6 8 Ž̅ = 100
6 8 18 260
−17
= Ž̅ → œ = −11 → p(x) = – 17 – 11 x + 25 x2
25
5-35
Yi-Hsin Pang
Coordinates Metrology
 Given m points: (x1, y1), (x2, y2), …, (xn, yn)
 Find the best fit circle
−• + −• =8
 The equivalent problem
2 • +2 • + 8 −• −• = +
→2 • +2 • +• = + • =8 −• −•
Find the least square solution of
2 2 1 +
•
2 2 1 • +
=
⋮ ⋮ ⋮ • ⋮
2 ) 2 ) 1 )+ ) 5-36
Yi-Hsin Pang
 Linear Regression
 Given data:

x x1 x2 … xn Mean = X, standard deviation = σX

y y1 y2 … yn Mean = Y, standard deviation = σX

X ≜ ∑)Qc Q, ¦§ ≜ ∑)Qc Q −X ,
) )

Y ≜ ∑)Qc Q, ¦¨ ≜ ∑)Qc Q −Y ,
) )

 Find a best fitting line y = p(x) = a x + b

1
r ⋮
= ⋮ ⋮ ,x = ,b =
t
) 1 ) 5-37
Yi-Hsin Pang
Linear Regression (Cont.)
 Normal equation
1 ∑)Qc
… r … ⋮
)
⋮ ⋮ = )
= Q Q
1 … 1 t 1 … 1 ∑)Qc
) 1 )
Q

r W − ∑)Qc Q ∑)Qc Q Q
= « q
t ) ∑ª¬p Uª a ∑«
ª¬p Uª
q
− ∑)Qc Q ∑)Qc Q ∑)Qc Q

-® ±iB U,²
r= ° = t = Y − rX
-¯ §¨ ±iB U,U
,

∑«
ª¬p Uª aµ ²ª a¶
°§¨ ≜
)-¯ -®
: correlation coefficient

Cov , ≜ ∑)Qc Q −X Q − Y : covariance of x an y

)

Cov , = ∑)Qc Q −X Q − X = ¦§
)
5-38
Yi-Hsin Pang
 §5.5 Orthonormal Sets
 E = • , • , … , •) : a basis for V with dimV = n
 For any ‘ in V, ‘ = • • + • • + ⋯ + •) •)
Find c1, …, cn (the coordinate vector of ‘)
(M1) ‘ ¸ = • , • , … , •) a ‘ → only valid for V = ℝ)
(M2) ‘ = • • + • • + ⋯ + •) •)
‘, •ƒ = • • , •ƒ + • • , •ƒ + ⋯ + •) •) , •ƒ , º = 1, … , W
• ,• • ,• ⋯ •) , • • ‘, •
• ,• • ,• ⋯ •) , • • ‘, •
⋮ =
⋮ ⋮ ⋮ ⋮
• , •) • , •) ⋯ •) , •) •) ‘, •)

[Q] (M1) = (M2) for V = ℝ ?

)
5-39
Yi-Hsin Pang
 If •‚ , •ƒ = 0 for i ≠ j,
• ,• 0 ⋯ 0 • ‘, •
0 • ,• ⋯ 0 • ‘, •
⋮ =
⋮ ⋮ ⋮ ⋮
0 0 ⋯ •) , •) •) ‘, •)
)
‘, •Q ‘, •Q
•Q = , P = 1, … , W → ‘ = » •Q
•Q •Q
Qc

 If •‚ , •ƒ = 0 for i ≠ j and •Q = 1 for k = 1, …, n

)

→ ‘ = » ‘, •Q •Q
Qc 5-40
Yi-Hsin Pang
Definition – Orthogonal/Orthonormal Sets
 V: an inner product space
 E = • , • , … , •) : a set of nonzero vectors in V
/ a basis for V
if •‚ , •ƒ = 0 for i ≠ j
E is called an orthogonal set / an orthogonal basis
if •Q = 1 for k = 1, …, n
E is called an orthonormal set / an orthonormal basis
• , • = 0 for i ≠ j 0, + ≠ º
¾ ‚ ƒ •‚ , •ƒ = ¼‚ƒ ≜ ½
•Q = 1 for k = 1, …, n 1, + = º
Kronecker delta 5-41
Yi-Hsin Pang
Properties of Orthonormal Bases
V: an inner product space
E = • , • , … , •) : an orthonormal basis for V
 For any ‘ in V, ‘ = ∑Qc ‘, •Q •Q
)

 If = ∑)Qc Q •Q and = ∑)Qc Q •Q , then

, = ∑)Qc Q Q∗
 (Parseval’s Theorem) If = ∑)Qc Q •Q , then
= ∑)Qc Q

5-42
Yi-Hsin Pang
Properties of Orthonormal Bases – Proof
0, + ≠ º
<key> •‚ , •ƒ = ¼‚ƒ ≜ ½
1, + = º
 Let ‘ = • • + • • + ⋯ + •) •)
→ ‘, •Q = ∑)ƒc •ƒ •ƒ , •Q = •Q , P = 1, … , W

 , = ∑)Qc •
Q Q , ∑ ƒc ƒ •ƒ =
)

∑)Qc Q ∑ )
ƒc
∗
ƒ • ,
Q ƒ • = ∑ )
Qc
∗
Q Q

 = , = ∑)Qc ∗
Q Q = ∑)Qc Q

5-43
Yi-Hsin Pang
Example
Z≜{ , cos , sin , … , cos W , sin W } is an
` ` ` ` `


orthonormal basis for ; ≜ span Z and ∈ −^, ^ ,

 For any f, g ∈ V,
`
G, H ≜ Ia` G H L

G = 2sin = 2^ ⋅ − ^⋅ cos 2 ,

` `

H = 2cos = 2^ ⋅ + ^⋅ cos 2 ,
` `
`
Ia` 4sin cos L = G, H = 2^ 2^ − ^ ^ = ^
`
Ia` 4sin¿ L = G = 2^ + ^ = 3^

5-44
Yi-Hsin Pang
Least Square Problem with Orthonormal Sets
 A = • , • , … , •) ∈ ℝ'×)
 If • , • , … , •) is an orthonormal set in ℝ'
Rank(A) = n
• • ,• ⋯ •) , •
= ⋮ • , … , •) = ⋮ ⋮ = I)
•) • , •) ⋯ •) , •)

The least square solution: œ = ad

Ž̅ = Ž̅
The vector projection of Ž̅ onto R(A): = œ = Ž̅

5-45
Yi-Hsin Pang
Notes on œ = Ž̅ and = Ž̅
• Ž̅, •
= Ž̅, •Q
œ= Ž̅ = ⋮ Ž̅ = ⋮ → Q

•) Ž̅, •) P = 1, … , W

Ž̅, •
= Ž̅ = • , … , •) ⋮ = ∑)Qc Ž̅, •Q •Q
Ž̅, •)

5-46
Yi-Hsin Pang
§6.4, 5.5 Unitary/Orthogonal Matrices
 Q = • , • , … , •) : an n × n matrix (Q ∈ 5)×) )
 If • , • , … , •) is an orthonormal set in 5)
 Q is called a unitary matrix for 5 = ℂ

 Q is called an orthogonal matrix for 5 = ℝ

 Example – Rotation matrix of ℝ

cos − sin
sin cos
is an orthogonal matrix

5-47
Yi-Hsin Pang
 Inverse of a Unitary/Orthogonal Matrix
 If Q = • , • , … , •) is a unitary matrix
• • ,• ⋯ •) , •
Q Q = ⋮ • , … , •) = ⋮ ⋮ = I)
•) • , •) ⋯ •) , •)

Rank(Q) = n  Q–1 exists, and Qa = Q Q Qa = Q

 Q Q = QQ = I)

 If Q = • , • , … , •) is an orthogonal matrix
Qa = Q Q Q = QQ = I)

5-48
Yi-Hsin Pang
Properties of a Unitary/Orthogonal Matrix
 Q = Á , Á , … , Á) : an n × n matrix (Q ∈ 5)×) )
 The following statements are equivalent

a) Á , Á , … , Ál is an orthonormal set in 5l
b) Q Q = QQ = I) . That is, Qa = Q
c) Q , Q = , for any , ∈ 5)
d) Q = for any ∈ 5)

5-49
Yi-Hsin Pang
Properties of a Unitary/Orthogonal Matrix –
Sketch Proof
 b)  c): Q , Q = ,Q Q = ,
 d)  a):
 Let = ƒ , º = 1, … , W → Q = Áƒ
→1= = Q = Áƒ
 Let = ÂQ + ƒ , P, º = 1, … , W → Q = ÁQ + Áƒ
→2= = ÁQ + Áƒ = ÁQ + Áƒ + 2Re ÁQ , Áƒ

∵ ÁQ = Áƒ = 1 ∴ Re ÁQ , Áƒ = 0
 Let = ÂQ + +ƒ , P, º = 1, … , W → Im ÁQ , Áƒ = 0
 ∴ ÁQ , Áƒ = ¼Qƒ , P, º = 1, … , W

5-50
Yi-Hsin Pang
§5.6 Gram-Schmidt Orthogonalization
Process
 V: an inner product space
 , ,…, ) : a linearly independent set in ;
Gram-Schmidt Process + normalization
• , • , … , •) : an orthonormal set in ;
with span • , • , … , •) = span , ,…, )

5-51
Yi-Hsin Pang
Example
1 −2 −1
2 0 1
A = Š ,Š ,Š =
2 −4 2


4 0 0
 Find an orthonormal basis Á , Á , Á for R(A)

5-52
Yi-Hsin Pang
Example (Cont.)
 Š = 1, 2, 2, 4 , 8 = Š =5
Á = Š ⁄8 = Æ 1, 2, 2, 4

 Š = −2, 0, −4, 0
8 = Š ,Á = −2,
= 8 Á = aÆ 1, 2, 2, 4
Š − = ¿Æ −2, 1, −4, 2
8 = Š − =4
Á = Š − ⁄8 = Æ −2, 1, −4, 2
5-53
Yi-Hsin Pang
Example (Cont.)
 Š = −1, 1, 2, 0
8 = Š ,Á = 1,
8 = Š ,Á = −1,
= 8 Á + 8 Á = Æ 3, 1, 6, 2
Š − = Æ −4, 2, 2, −1
8 = Š − =2
Á = Š − ⁄8 = Æ −4, 2, 2, −1

Á ,Á ,Á is an orthonormal basis for R(A)

5-54
Yi-Hsin Pang
Example (Cont.)
 In the above process,
Š =Á 8 8Qƒ = Šƒ , ÁQ
Š =Á 8 +Á 8
Š =Á 8 +Á 8 +Á 8
8 8 8
Š ,Š ,Š = Á ,Á ,Á 0 8 8
0 0 8

 QR factorization of A
5-55
Yi-Hsin Pang
Gram-Schmidt QR Factorization
 A: an m × n matrix of rank n
 A = QR, where

Q: an m × n matrix with orthonormal column vectors

R: an n × n upper triangular matrix with positive
diagonal entries (rkk > 0, k = 1, …, n)
 Note: R is invertible

5-56
Yi-Hsin Pang
 QR Factorization and Least Square Solution
Least square problem: A = Ž̅ (A ∈ ℝ'×) , Ž̅ ∈ ℝ' )
œ = Ž̅


 Normal equation:

If A = QR → R Q QR œ = R Q Ž̅
Q Q = I) → R Rœ = R Q Ž̅ → Rœ = Q Ž̅

 Interpretation:
A = Ž̅ → Š + Š +⋯+ ) Š) = Ž̅
→ Š , ÁQ + Š , ÁQ + ⋯ + ) Š) , ÁQ = Ž̅, ÁQ , k = 1, …, n

5-57
Yi-Hsin Pang
 Example
 Find the least square solution of A = Ž̅, where
1 −2 −1 −1
2 0 1 ̅Ž = 1
A=
2 −4 2 1
4 0 0 −2
1 −2 −4
5 −2 1
 Sol:
2 1 2
QR factorization of A: A = 0 4 −1
Æ 2 −4 2
0 0 2
4 2 −1

1 −2 −4 −1
5 −2 1 −1
2 1 2 1
Normal equation: 0 4 −1 = = −1
Æ 2 −4 2 1
0 0 2 2
4 2 −1 −2
back-ward substitution  œ = −2/5, 0, 1 5-58
Yi-Hsin Pang
Example
 V = span{1, x}
 ∀G, H ∈ V, G, H ≜ Is G H L
 Starting from {1, x}, find an orthonormal basis for V.
Š = 1, Š = 1,1 = 1 → • = Š ⁄ Š =1
Š = ,
= Š ,Š Š = ,1 1 =

Š − = − , Š − = − , − =
Šq a q
→• = = 12 −
Šq a q Ans: 1, 12 −
5-59
Yi-Hsin Pang
 §5.5, 5.7 Orthogonal Bases for Functions
 Least square approximation of functions
A function G ∈ S r, t
• , • , … , •) : orthogonal basis for V
V = span(• , • , … , •) )

G ≈ GÊ ≜ ∑)ƒc
f,•Ë
q •ƒ : least square approximation of f in V
•Ë
G − GÊ is minimized
• , • , … , •) : orthonormal basis for V
G ≈ GÊ ≜ ∑)ƒc G, •ƒ •ƒ : least square approximation of f in V
5-60
Yi-Hsin Pang
 Example
Find p(x) = a0 + a1x to minimize T U − p(x) on
0, 1


T U − p(x) = Y T U − p(x) L
s
 Sol:
1. V = span{1, x}. ∀G, H ∈ V, G, H ≜ Is G H L

2. Basis {1, x}  1, 12 − : orthonormal basis for V

Gram-Schmidt
3. T U ≈ T U , • • + T U , • • • = 1, • = 12 −

→ T U ≈ 4e − 10 + 6 3 − e = p(x)
5-61
Yi-Hsin Pang
Example (Cont.)
p(x) = 4e 10 6 3 e 0.8731 1.6903

T U , p(x)

TU
p(x)

x
5-62
Yi-Hsin Pang
Fourier Series
 For a 2π-periodic function f(x), find H p
r
q s
∑)Qc rQ cos P tQ sin P to minimize

G H on ^, ^

 Z ≜ {1, cos , sin , … , cos W , sin W } : an orthogonal

basis for ; ≜ span Z and ∈ ^, ^
f U , f U , hij QU f U , jkl QU
G ≈ 1 ∑)Qc cos P sin P
hij QU jkl QU

`
rQ I G cos P L , P 0, 1, … , W
` a`
`
tQ I G sin P L , P 1, … , W
` a` 5-63
Yi-Hsin Pang
Fourier Series in Complex Form
 For 2π-periodic complex-valued functions f (x)
and H( )
`
G, H ≜ Ia` G H∗ L

 For a 2π-periodic function f (x), find H

∑)Qca) •Q T ‚QU to minimize
`
G H ≜ Ia` G H L

5-64
Yi-Hsin Pang
Fourier Series in Complex Form (Cont.)

` ‚ Qa' U 2^, +G P V
T ‚QU , T ‚'U Ia` T L ½ 2^¼Q'
0, +G P ≠ V

→ Z ≜ {T ‚QU , P W, W 1, … , W} : an orthogonal
basis for ; ≜ span Z and ∈ ^, ^
) )
G , T ‚QU
G ≈ » T ‚QU » •Q T ‚QU
T ‚QU
Qca) Qca)

f U ,Ì ÍªÎ `
•Q I G T a‚QU L
Ì ÍªÎ Ï a`

5-65
Yi-Hsin Pang