Linear Algebra

Grinshpan

Gram-Schmidt orthogonalization

Given a list of non-orthogonal linearly independent vectors v1 , . . . , vk in Rn , it is

possible to transform it into a list of orthonormal vectors u1 , . . . , uk so that
span(u1 , . . . , uj ) = span(v1 , . . . , vj ), j = 1, . . . , k,
and the orientation of the configuration is preserved. The procedure takes k steps.
Gram–Schmidt procedure

Step 1: u1 = v1 /∥v1 ∥

Step 2: u2 = v2 ⊥ /∥v2 ⊥ ∥, where v2 ⊥ = v2 − projspan(v1 ) v2

Step 3: u3 = v3 ⊥ /∥v3 ⊥ ∥, where v3 ⊥ = v3 − projspan(v1 , v2 ) v3

...
Step k: uk = vk ⊥ /∥vk ⊥ ∥, where vk ⊥ = vk − projspan(v1 , ..., vk−1 ) vk

For each j = 2, . . . , k, vj ⊥ ≠ 0 by linear independence.

Each uj is a unit vector orthogonal to span(v1 , . . . , vj−1 ).
Each uj is a linear combination of vectors v1 , . . . , vj with positive coefficient of vj .
Hence span(u1 , . . . , uj ) = span(v1 , . . . , vj ), j = 1, . . . , k.
By construction, u1 , . . . , uj−1 form an orthonormal basis for span(v1 , . . . , vj−1 ). Thus

vj ⊥ = vj − (vj ⋅ u1 )u1 − . . . − (vj ⋅ uj−1 )uj−1 , j = 2, . . . , k.

We may write the resulting equations as follows:

v1 = ∥v1 ∥u1
v2 = (v2 ⋅ u1 )u1 + ∥v2 ⊥ ∥u2
⋮
vk = (vk ⋅ u1 )u1 + . . . + (vk ⋅ uk−1 )uk−1 + ∥vk ⊥ ∥uk .
These equations can be also put in the matrix form:
⎛∥v1 ∥ u1 ⋅ v2 u1 ⋅ v3 . . . u 1 ⋅ vk ⎞
⎜ 0 ∥v2 ⊥ ∥ u2 ⋅ v3 . . . u 2 ⋅ vk ⎟
(v1 v2 . . . vk ) = (u1 u2 . . . uk ) ⎜ ⎟.
⎜ ⋮ ⋮ ⋮ ... ⋮ ⎟
⎝ 0 0 0 . . . ∥vk ⊥ ∥ ⎠
Thus every n × k matrix A with linearly independent columns is a product of an n × k
matrix Q with orthonormal columns and a k × k invertible upper-triangular matrix R.
This is known as QR factorization. Note that Rjj = vj ⋅ uj .

⎛1⎞ ⎛1⎞ ⎛3⎞

⎜1⎟ ⎜2⎟ ⎜4⎟
Example. The vectors v1 = ⎜ ⎟, v2 = ⎜ ⎟, v3 = ⎜ ⎟ are linearly independent.
⎜1⎟ ⎜3⎟ ⎜2⎟
⎝1⎠ ⎝4⎠ ⎝1⎠

They span a three-dimensional subspace of R4 but are not orthogonal.

Do you see an orthonormal basis for span(v1 , v2 , v3 ) ? If not, apply Gram–Schmidt:

⎛1⎞
1 ⎜1⎟
u1 = v1 /∥v1 ∥ = ⎜ ⎟
2 ⎜1⎟
⎝1⎠

⎛1⎞ ⎛1⎞ ⎛−3⎞ ⎛−3⎞

⎜2⎟ 5 ⎜1 ⎟ 1 ⎜−1⎟ 1 ⎜−1⎟
v2 ⊥ = v2 − (v2 ⋅ u1 )u1 = ⎜ ⎟ − ⎜ ⎟ = ⎜ ⎟ , u2 = v2 ⊥ /∥v2 ⊥ ∥ = √ ⎜ ⎟
⎜3⎟ 2 ⎜1⎟ 2 ⎜ 1 ⎟ 2 5⎜1⎟
⎝4⎠ ⎝1⎠ ⎝3⎠ ⎝3⎠

⎛3⎞ ⎛1⎞ ⎛−3⎞ ⎛−7⎞ ⎛−7⎞

⎜4⎟ 5 ⎜1⎟ 2 ⎜−1⎟ 1 ⎜ 11 ⎟ 1 ⎜ 11 ⎟
v3 ⊥ = v3 − (v3 ⋅ u1 )u1 − (v3 ⋅ u2 )u2 = ⎜ ⎟ − ⎜ ⎟ + ⎜ ⎟ = ⎜ ⎟, u3 = v3 ⊥ /∥v3 ⊥ ∥ = √ ⎜ ⎟
⎜2⎟ 2 ⎜1⎟ 5 ⎜ 1 ⎟ 10 ⎜−1⎟ 6 5 ⎜−1⎟
⎝1⎠ ⎝1⎠ ⎝3⎠ ⎝−3⎠ ⎝−3⎠

Now u1 , u2 , u3 is an orthonormal basis for span(v1 , v2 , v3 ) .

The frame u1 , u2 , u3 has the same orientation as the frame v1 , v2 , v3 .
The associated QR factorization is
⎛ 2 − 2√5 − 6√5 ⎞
1 3 7

⎛1 1 3⎞ ⎜ 1 ⎟ ⎛2 5 5 ⎞
⎜ − √1 √ ⎟
11
√
⎜1 2 4⎟ ⎜ 2 2 5 6 5 ⎟⎜
5 − √45 ⎟
⎜ ⎟ = ⎜1 ⎟ ⎜0 ⎟.
⎜1 3 2⎟ ⎜ 1
√ − 6√5 ⎟
1 ⎟ ⎜ ⎟
⎝1 ⎠ ⎜ ⎝ ⎠
4 1 ⎜ 2 2 5 ⎟ 0 0 √3
⎝ 21 2√3 5 − 6√3 5 ⎠ 5

⎛ 1 0 0⎞
Note that Q⊺ Q = ⎜0 1 0⎟ and that QQ⊺ = u1 u⊺1 + u2 u⊺2 + u3 u⊺3 is the orthogonal
⎝ 0 0 1⎠
projection onto span(v1 , v2 , v3 ) = im Q.