Lesson 3

Escuela Técnica Superior de Ingeniería Numerical Methods in
Universidad Loyola Andalucía Engineering
Numerical Methods in
Engineering
Part I
Lesson 3: “Singular Value Problems”
Outline
 Singular Value Problem
 The Eigenvalue Problem
 Power iteration method
 Inverse power iteration
 Rayleigh quotient iteration
 QR algorithm
2
Singular Value Problem

 Solution of high-dimensional space problems based on lower
dimensional subspaces.
 This allows for the simplification of complex data
environments.
 Determination of eigenvalues and eigenvectors is of great
relevance.
 These provide interesting information on dynamic systems.
 Applications cover from structural analysis to data analytics.
3
The Eigenvalue Problem

 Let 𝐴 be an 𝑛 × 𝑛 matrix, the eigenvalue problem reads as:
𝐴𝑣 = 𝜆𝑣
where 𝜆 and 𝑣 are the eigenvalues and eigenvectors of 𝐴,
respectively.
 The eigenvalues 𝜆 of 𝐴 can be obtained from:

𝐴 − 𝜆𝐼 = 0 → 𝜆1 , 𝜆2 , … , 𝜆𝑛
 The 𝑖-th eigenvector is obtained by solving the following system:

𝐴 − 𝜆𝑖 𝐼 𝑣𝑖 = 0
4
The Eigenvalue Problem (example)

 E.g., let us consider the following matrix:
1 3
𝐴=
2 2
 The eigenvalues obtained from:
1−𝜆 3
𝐴 − 𝜆𝐼 = 0 → =0
2 2−𝜆
1−𝜆 2−𝜆 −6=0
 Thus, 𝜆1 = 4 and 𝜆2 = −1.

5

 The related eigenvectors must satisfy:
𝐴 − 4𝐼 𝑣1 = 0
𝐴 + 𝐼 𝑣2 = 0
 Solving the first system, we can obtain 𝑣1 :
−3 3 𝑣1,1 0 1
= → 𝑣 =
2 −2 𝑣1,2 0 1
1
 And the second system provides 𝑣2 :

2 3 𝑣2,1 0 −3
= → 𝑣 =
2 3 𝑣2,2 0 2
2
6

 In this problem we have a dominant eigenvalue, 𝜆1 = 4, as it is
larger than the other eigenvalues, i.e., 𝜆2 = −1.
 In this case we say that 𝜆1 and 𝑣1 are the dominant
eigenvalue and eigenvector.
 This means that this eigenvalue will dominate the calculation
after several steps, and the solution will move closer to a
dominant eigenvector direction.
 This is basically to the fact that any solution of the system can
be expressed as a linear combination of the eigenvectors of
the system (see next slide).
7

 Let us consider the previous matrix 𝐴 with eigenvalues 𝜆1 =
4 and 𝜆2 = −1; and eigenvectors 𝑣1 = 1 1 𝑇 and 𝑣2 =
−3 2 𝑇 .
 Let us now observe what happens when we multiply 𝐴 times a
given vector 𝑥0 = −5 5 𝑇 :
1 3 −5 10
𝑥1 = 𝐴𝑥0 = =
2 2 5 0
1 3 10 10
𝑥2 = 𝐴𝑥1 = 𝐴2 𝑥0 = =
2 2 0 20
1 3 10 70
𝑥3 = 𝐴𝑥2 = 𝐴3 𝑥0 = =
2 2 20 20
1 3 70 250 25Τ26
𝑥4 = 𝐴𝑥3 = 𝐴4 𝑥0 = = = 260
2 2 60 260 1
8
Eigenvalue problem
 Multiplying 𝐴 by the vector 𝑥0 several times has resulted in
moving the vector very close to the dominant eigenvector of
𝐴, 𝑣1 = 1 1 𝑇 .
 This can be easily seen if we express 𝑥0 as a linear
combination of the eigenvectors 𝑣1 and 𝑣2 , 𝑥0 = 𝑣1 + 2𝑣2 =
−5 5 𝑇 , then:
1 −3
𝑥1 = 𝐴𝑥0 = 𝜆1 𝑣1 + 𝜆2 2𝑣2 = 4 + −1 2
1 2
1 −3
𝑥2 = 𝐴2 𝑥0 = 𝜆12 𝑣1 + 𝜆22 2𝑣2 = 42 + (−1)2 2
1 2
3 3 3 3 1 3 −3
𝑥3 = 𝐴 𝑥0 = 𝜆1 𝑣1 + 𝜆2 2𝑣2 = 4 + (−1) 2
1 2
4 4 4 4 1 4 −3
𝑥4 = 𝐴 𝑥0 = 𝜆1 𝑣1 + 𝜆2 2𝑣2 = 4 + (−1) 2
1 2
9
Eigenvalue problem
 The eigenvalues determination requires finding the roots of
the equation 𝐴 − 𝜆𝐼 = 0.
 On the other hand, finding the eigenvectors imply the use of
Gaussian elimination.
 For large matrices (𝑛 > 3), finding the roots of the equation
becomes complex.
 Therefore, the use of approximate methods is essential.
10
Power iteration method

 As in the case of root-finding, there is no direct method for
computing eigenvalues.
 In large systems, the traditional exact method for determining
the eigenvalues of the system (based on the roots of 𝐴 − 𝜆𝐼 =
0) is unmanageable.
 To keep the numbers getting out of hand, it is necessary to
normalize the vector.
 Two operations: normalization and multiplication: power
iteration, as in the case of fixed-point iteration.
 This method only provides the largest eigenvalue and
eigenvector.
11

 Consider the eigenvalue equation
𝐴𝑣 = 𝜆𝑣
 If we multiply both sides by 𝑣 𝑇 , we get:

𝑣 𝑇 𝐴𝑣 = 𝑣 𝑇 𝜆𝑣 = 𝜆𝑣 𝑇 𝑣
 Thus, given an approximate eigenvector, the Rayleigh quotient,

which is the best approximate eigenvalue, is:
𝑣 𝑇 𝐴𝑣
𝜆= 𝑇
𝑣 𝑣
12

Algorithm
Give initial vector 𝑣0
for j = 1, 2, …
𝑣𝑗−1
𝑢𝑗−1 =
𝑣𝑗−1
𝑣𝑗 = 𝐴𝑢𝑗−1
𝑇
𝜆𝑗 = 𝑢𝑗−1 𝐴𝑢𝑗−1
end
𝑣𝑗
𝑢𝑗 =
𝑣𝑗
13
Inverse power iteration

 Power iteration is limited to locating the eigenvalue of the
largest magnitude.
 If power iteration is applied to the inverse of the matrix, the
smallest eigenvalue can be found.
 Consider the eigenvalue equation, 𝐴𝑣 = 𝜆𝑣, it can be
reformulated as:
1
𝑣 = 𝐴−1 𝑣
𝜆
which presents the same eigenvector.
 Thus, the largest magnitude eigenvalue of 𝐴−1 gives the
smallest eigenvalue of 𝐴.
14
Inverse power iteration

 Ok, now we have shown we are able to get the largest and the
smalles eigenvalues of a matrix but how do we find the rest?
 We will make use of the shifting concept.
 In other words, it can be proved that the eigenvectors of a
shifted matrix 𝐴 − 𝑠𝐼 are the same as those of the matrix 𝐴,
while the eigenvalues are 𝜆𝑖 − 𝑠 ∀𝑖 = 1,2, … , 𝑛:
𝐴 − 𝑠𝐼 𝑣 = 𝜆 − 𝑠 𝑣
 I.e., if we want to know an eigenvalue 𝜆 close to 10, we set 𝑠 =
10.
 From the inverse power iteration we will get the smallest value
of 𝜆 − 𝑠, which after shift back, yields the expected 𝜆.
15
Inverse power iteration method

Algorithm
Give initial vector 𝑣0 and shift 𝑠
for j = 1, 2, …
𝑣𝑗−1
𝑢𝑗−1 =
𝑣𝑗−1
Solve 𝐴 − 𝑠𝐼 𝑣𝑗 = 𝑢𝑗−1
𝑇
𝜆𝑗 = 𝑢𝑗−1 𝑣𝑗
end
𝑣𝑗
𝑢𝑗 =
𝑣𝑗
16
Rayleigh Quotient Iteration

 If we use the Rayleigh quotient as the updated shift in the inverse power
iteration method, it leads to the Rayleigh Quotient Iteration (RQI) method.
Algorithm
Give initial vector 𝑣0 and shift 𝑠
for j = 1, 2, …
𝑣𝑗−1
𝑢𝑗−1 =
𝑣𝑗−1
𝑇
𝜆𝑗−1 = 𝑢𝑗−1 𝐴𝑢𝑗−1
Solve 𝐴 − 𝜆𝑗−1 𝐼 𝑣𝑗 = 𝑢𝑗−1
end
𝑣𝑗
𝑢𝑗 =
𝑣𝑗
17
QR algorithm
 The power method and inverse power method can be used to
obtain the largest and smallest eigenvalues, and be also
extended to determine the remaining values.
 However, there exists a more efficent method based on the
QR decomposition, which basically reads that for any matrix 𝐴,
there exists an orthogonal matrix 𝑄 and an upper triangular
matrix 𝑅 such that
𝐴 = 𝑄𝑅
 Recall that an orthogonal matrix 𝑄 has the property 𝑄 −1 = 𝑄 𝑇
18
QR algorithm
 The QR method consists of the following iterative steps:
 Transform 𝐴 into a tridiagonal (Hessian) matrix 𝐻
 Decompose 𝐻 in 𝑄𝑅
 Multiply 𝑄 and 𝑅 in reverse order to form a new 𝐻, 𝐻 = 𝑅𝑄
 The diagonal of 𝑯 will converge to the eigenvalues.
 There is a built-in QR decomposition function in Matlab,
[Q R]=qr(A).
19
Lesson check
 Eigenvalue and eigenvector problem
 Power matrix method to obtain largest value
 Inverse power matrix method to obtain smallest value
 Inverse power along with shift properties to obtain the rest
 QR algorithm (most efficient)
20

Lesson 3

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Lesson 3

Uploaded by

Copyright:

Available Formats

Escuela Técnica Superior de Ingeniería Numerical Methods in

Universidad Loyola Andalucía Engineering

Singular Value Problem

The Eigenvalue Problem

 The eigenvalues 𝜆 of 𝐴 can be obtained from:

 The 𝑖-th eigenvector is obtained by solving the following system:

The Eigenvalue Problem (example)

 Thus, 𝜆1 = 4 and 𝜆2 = −1.

The Eigenvalue Problem

 And the second system provides 𝑣2 :

The Eigenvalue Problem

The Eigenvalue Problem

Power iteration method

Power iteration method

 If we multiply both sides by 𝑣 𝑇 , we get:

 Thus, given an approximate eigenvector, the Rayleigh quotient,

Power iteration method

Inverse power iteration

Inverse power iteration

Inverse power iteration method

Rayleigh Quotient Iteration

You might also like