Lecture 9

Last lecture:
Muller’s method
Repeated roots
Linear system of equations
This lecture:

Linear algebra:
Determinant
Matrix operations
Eigenvalue Problems
 Chapter 3

SYSTEM OF EQUATIONS
 Linear Systems
• Need to solve linear equations for x’s with constant coefficients

𝑎11 𝑥1 + 𝑎12 𝑥2 +. . . +𝑎1𝑛 𝑥𝑛 = 𝑏1

𝑎21 𝑥1 + 𝑎22 𝑥2 +. . . +𝑎2𝑛 𝑥𝑛 = 𝑏2
⋮
𝑎𝑛1 𝑥1 + 𝑎𝑛2 𝑥2 +. . . +𝑎𝑛𝑛 𝑥𝑛 = 𝑏𝑛
• aij and bi are constants
 Mathematical background
• It is convenient to write system of equations in matrix-vector form

𝐴][𝑥] = [𝑏] or [𝐴]{𝑥} = {𝑏

𝑎11 𝑎12 ⋯ 𝑎1𝑛 𝑥1 𝑏1

𝑎21 𝑎22 … 𝑎2𝑛 𝑥2 𝑏2
⋮ ⋮ ⋮ ⋮ = ⋮
⋮ ⋮ ⋮
𝑎𝑛1 𝑎𝑛2 … 𝑎𝑛𝑛 𝑥𝑛 𝑏𝑛
 Matrix Notations
Matrix (m  n) - a set of numbers arranged in rows (m) and columns (n)

Column 4 (second index)

 a11 a12 a13 a14  a1 n 

a a22 a23 a24  a2 n 
 21 
 a31 a32 a33 a34  a3 n 
A =  
Row 3

 a41 a42 a43 a44  a4 n  (first index)

       
 
 am 1 am 2 am 3 am 4  amn 
3.1.1 Definitions of numerous special matrices

• Identity matrix • Diagonal matrix

1 0 0 0 𝑎11 0 0 0
0 1 0 0 0 𝑎22 0 0
[𝑰] = [𝐴] =
0 0 1 0 0 0 𝑎33 0
0 0 0 1 0 0 0 𝑎44

[B][I] = [I][B] = [B]

3.1.1 Definitions of numerous special matrices

• lower triangular zeroes

𝑎11 0 0 0
𝑎21 𝑎22 0 0
[𝐴] =
𝑎31 𝑎32 𝑎33 0
𝑎41 𝑎42 𝑎43 𝑎44

• upper triangular
𝑎11 𝑎12 𝑎13 𝑎14
0 𝑎22 𝑎23 𝑎24
[𝐴] = 0 0 𝑎33 𝑎34
0 0 0 𝑎44
zeroes
3.1.1 Definitions of numerous special matrices

➢ Banded matrix – all elements are zero,

with the exception of a band centered on the main diagonal

𝑎11 𝑎12 0 0
𝑎21 𝑎22 𝑎23 0 Tridiagonal –
[𝐴] =
0 𝑎32 𝑎33 𝑎34
3 non-zero bands
0 0 𝑎43 𝑎44
band
 Transpose of A:

For 𝑎11 𝑎12 ⋯ 𝑎1𝑛

𝑎21 𝑎22 … 𝑎2𝑛 𝑎11 𝑎21 ... 𝑎𝑛1
𝑎12 𝑎22 ... 𝑎𝑛2
A= ⋮ ⋮ ⋮  AT= . . . = transpose of A= A′
⋮ ⋮ ⋮ ... ... ...
𝑎𝑛1 𝑎𝑛2 … 𝑎𝑛𝑛 𝑎1𝑛 𝑎2𝑛 ... 𝑎𝑛𝑛

𝑥= 4 −2 3 ; 𝑦 = 3 1 −2

4 3
′
𝑥 = −2 ; y′ = 1
3 −2

Trace of A: tr[A] = σ𝑛𝑖=1 𝑎𝑖𝑖

3.1.1 Definitions of numerous special matrices

• Symmetric matrices

𝑎 𝑏 𝑐 𝑑
𝑏 𝑒 𝑓 𝑔 aij = aji
[𝐴] =
𝑐 𝑓 ℎ 𝑖 T
𝑑 𝑔 𝑖 𝑗 [A] = [A]

5 2 −1 4 6
2 −2 8 −5 1
[𝐵] = −1 8 3 2 0
4 −5 2 1 3
6 1 0 3 −4
Matrix Determinant
Compute the determinant D
2 x 2 matrix D = a11 a12 = a a − a a
11 22 12 21
a11 a12
a21 a22
a21 a22
3 x 3 matrix

a11 a12 a13 a11 a12 a13 a11 a12

D = a21 a22 a23 = a21 a22 a23 a21 a22
a31 a32 a33 a31 a32 a33 a31 a32

a11 a12 a13 a11 a12

D= a21 a22 a23 a21 a22
a31 a32 a33 a31 a32

= a11a22 a33 + a12 a23a31 + a13a21a32

−a31a22 a13 − a32 a23a11 − a33a21a12
 Matrix Determinant
Compute the determinant D for n x n matrix D = a11a22 a33 + a12 a23a31 + a13a21a32
a11 a12 a13
−a31a22 a13 − a32 a23a11 − a33a21a12
D = a21 a22 a23 = a11 (a22 a33 − a32 a23 ) − a21 (a12 a33 − a13a32 ) + a31 (a12 a23 − a22 a13 )
a31 a32 a33
a22 a23 a a13 a a13
= a11 − a21 12 + a31 12
a32 a33 a32 a33 a22 a23
= a11M 11 − a21M 21 + a31M 31
=σ3𝑖=1 (−1)i+j aijMij for j=1
Element Minor
a11 a12 a13 a11 a12 a13 a11 a12 a13 a11 𝑎22 𝑎23
M11=|𝑎 |
32 𝑎33
a21 a22 a23 a21 a22 a23 a21 a22 a23 𝑎12 𝑎13
a21 M21=|𝑎 |
a31 a32 a33 a31 a32 a33 a31 a32 a33 32 𝑎33
a31 𝑎12 𝑎13
M31=|𝑎 |
=M11 =M21 =M31 22 𝑎23

Minor Mij = determinant of the matrix after ith row and jth column of A are crossed out
 Matrix Determinant

5 7 3 4
Example: 8 9 2 3
D=
1 5 3 8
3 6 9 0
9 2 3 8 2 3 8 9 3 8 9 2
=5* 5 3 8 -7* 1 3 8 +3* 1 5 8 -4* 1 5 3 = -36
6 9 0 3 9 0 3 6 0 3 6 9
5 7 3 4
Example: 8 9 2 3 Choose the row or column with many zeroes.
D=
1 5 3 8
3 6 9 0
5 7 3 5 7 3 5 7 3
=-4* 8 9 2 + 3∗ - 8∗ 8 9 2 + 0∗ 8 9 2
= -36
1 5 3 1 5 3 1 5 3
3 6 9 3 6 9 3 6 9
No need to calculate
Do not use this method for D if n>3 unless there are a lot of zeros in matrix.
3.1.2 Matrix Operations
Matrix identity
[A] = [B] if and only if aij = bij for ALL i and j
Matrix Addition and Subtraction
[C] = [A] + [B]  Cij = Aij + Bij
[C] = [A] − [B]  Cij = Aij − Bij

Commutative
[A] + [B] = [B] + [A]
[A] − [B] = −[B] + [A]
Associative
( [A] + [B] ) + [C] = [A] + ( [B] + [C] )
( [A] + [B] ) − [C] = [A] + ( [B] − [C] )
( [A] − [B] ) + [C] = [A] + (−[B] + [C] )
Matrix Multiplication
➢ How the rows and columns line up in matrix multiplication?

3 1 
Consider   5 9 
A  B = 8 6   
 7 2 
0 4 

5 9 
7 2 
 

3 1   3x5+1x7=22 29 
8 6   
   82 84 
0 4   28 8 
Matrix Multiplications

Recall how matrix multiplication works

g j
a b c    ag + bh + ci aj + bk + cl 
d e   h k  =  
 f  dg + eh + fi dj + ek + fl 
 i l 

g j  ga + jd gb + je gc + jf 
h k  a b c 
=  ha + kd hb + ke hc + kf 
  d e 
 i l  
f
 ia + ld ib + le ic + lf 
[A]*[B]  [B]*[A]
Matrix Multiplication

➢ Matrix multiplication can be performed only if the inner dimensions are equal

For C = AB 

for i =1 to m
for j=1 to l

cij = σ𝑛𝑘=1 aik bkj

end
end
 Matrix Inverse
Matrix inverse, [A]−1 , for a non-singular square matrix [A] is defined such that
[A]−1 [A] = [A] [A]−1 = [I]
• Example:
1 −1 3 1 −1 −1
A= 2 1 2 A-1 = − 6Τ5 7Τ5 4Τ5
−2 −2 1 − 2Τ5 4Τ5 3Τ5
1 −1 3 1 −1 −1 1 0 0
 AA-1 = 2 1 2 − 6Τ5 7Τ5 4Τ5 = 0 1 0
−2 −2 1 − 2Τ5 4Τ5 3Τ5 0 0 1
• Inversion of 2x2 matrices can be done easily as follows:
−1 1
−1 𝑎 𝑏 𝑑 −𝑏
𝐴 = =
𝑐 𝑑 𝑎𝑑 − 𝑏𝑐 −𝑐 𝑎

• For n>2, it is easier to compute A-1 numerically. To be discussed later.

 Singular vs non-singular Matrix
◊ The matrix A is non-singular if A-1 exists.
A non-singular

◊ If A is not singular, Ax = b  x = A-1 b a unique solution

◊ If A is singular, Ax=b  x may have infinitely many or no solution.

A is singular
A is singular

2 lines collapse into 1

2 lines never cross
infinitely many solution
no solution
 Matrix Augmentation
 a11 a12 a13 a14 
a a 22 a 23 a 24 
A =  21
a 31 a 32 a 33 a 34 
 
a41 a 42 a43 a44 

 a11 a12 a13 a14 1 0 0 0

 
 a 21 a 22 a 23 a 24 0 1 0 0
A=
a 31 a 32 a 33 a 34 0 0 1 0
 
 a 41 a 42 a 43 a 44 0 0 0 1

➢ Whatever you do to left-hand-side, do to the right-

hand side (useful when solving system of equations)
 Augmented Matrix
a11 x1 + a12 x2 + a13 x3 + a14 x4 = b1
a21 x1 + a22 x2 + a23 x3 + a24 x4 = b2
a31 x1 + a32 x2 + a33 x3 + a34 x4 = b3
a41 x1 + a42 x2 + a43 x3 + a44 x4 = b4

 a11 a12 a13 a14 b1 

a a22 a23 a24 
b2 
 A b =  a
 21

a32 a33 a34 b3 

31
 
 a41 a42 a43 a44 b4 
 Eigenvalue Problems

Goal: Given an nxn matrix A, find scalar  and nonzero vector x such that
Ax=x
’s are eigenvalues, and x is corresponding eigenvector

 may be complex even if A is real

Spectrum = (A) = set of eigenvalues of A

Spectral radius = (A) = max{||:   (A)}

 Characteristic Polynomial
Equation A x = x is equivalent to
(A- I) x = 0
which has nonzero solution x if, and only if, its matrix is singular

Eigenvalues of A are roots i of characteristic polynomial

det (A- I) = 0
of degree n

An nxn matrix A corresponds to an nth order polynomial thus it always has n

eigenvalues, but they may not be real nor distinct

Complex eigenvalues of a real matrix occur in complex conjugate pairs:   i

(, x) forms an eigenpair of A

 Example

Consider A=
4 −1 (A- I) x = 0
−2 3
o The equation for finding  is: det (A -  I) = 0
NOTE: one of the equations is redundant.
4 −  −1
 =0
−2 3 −  Just pick one to work with.
 ሺ4 − 𝜆)ሺ3 − 𝜆) − 2 = 0
 𝜆2 − 7𝜆 + 10 = 0 is characteristic polynomial
Choose 𝑥2 =1  𝑥1 = 0.5
 ሺ𝜆 − 2)ሺ𝜆 − 5) = 0
o The solution for the eigenvalues is thus: 𝜆1 = 2 & 𝜆2 = 5 𝑥1 (1)
0.5
 𝑥 =
o Eigenvector #1 (for 𝜆 = 𝜆1 = 2): 2 1
Eqn: (A -  1 I) x = 0

4 − 𝜆1 −1 𝑥1 4−2 −1 𝑥1 • Similarly, for 𝜆 = 𝜆2 = 5,

= 0  =0
−2 3 − 𝜆1 𝑥2 −2 3 − 2 𝑥2
𝑥1 (2)
−1
𝑥2 =
1
2 −1 𝑥1
 =0
−2 1 𝑥2

 2𝑥1 − 𝑥2 = 0
& −2𝑥1 + 𝑥2 = 0
 Lecture 10
Last lecture:
Determinant
Matrix operations
Eigenvalue problems
This lecture:

Eigenvalue decomposition
Cramer's rule
Matrix norm
Gauss elimination
 Eigenvalue Value Decomposition
❖ Given an nxn matrix A, suppose there are n ’s and nonzero vector x such that
A x (i) =  i x (i) for i=1,2,…n
i.e.,
𝜆1 0 0 … 0
0 𝜆2 0 … 0
A [x (1) x (2) x (3) … x (n) ] = [x (1) x (2) x (3) … x (n) ] 0 0 𝜆3 … 0
… … … ... …
0 0 0 . . . 𝜆𝑛
❖ Let P = [x (1) x (2) x (3) … x (n) ] = nxn matrix , D = diag()= diagonal matrix with dii =  i.
AP=PD
❖ Multiply P-1 from the left on both sides,
P-1 A P = P-1 P D = D
=> A = P D P-1
❖ mth power of A: Am = (P D P-1) m = P D P-1 * P D P-1 * P D P-1 …* P D P-1 = P D m P-1
If max(|  i|) <1, Am → 0 for large m (very important for proving convergence of iterative methods)
3.1.3 Cramer's rule for system of equations
a11 x1 + a12 x2 = b1 a22b1 − a12b2 a11b2 − a21b1
  x1 = , x2 =
a21 x1 + a22 x2 = b2 a22 a11 − a12 a21 a22 a11 − a12 a21

a11 a12
Denominator is D=
a21 a22
b1 a12
Numerators are:
D1 = a22b1 − a12b2 =
b2 a22
a11 b1
D2 = a11b2 − a21b1 =
a12 b2
D (new matrix) kth column of as with bs (i.e., aik  bi )
xk = k
D(a ij )
 Dk
Cramer’s Rule

To find xk for the following system

a11 x1 + a12 x2 + + a1n xn = b1
a21 x1 + a22 x2 + + a2 n xn = b2

an1 x1 + an 2 x2 + + ann xn = bn
Replace kth column of as with bs (i.e., aik  bi )
𝐷(𝑛𝑒𝑤 𝑚𝑎𝑡𝑟𝑖𝑥 ሻ
𝑥𝑘 =
D(aijሻ
 Example

3 1 − 1  x1   2  − − −(1)
Find solution to 1 4 1   x  = 12  − − −(2) using Cramer’s rule
   2  
2 1 2   x3  10  − − −(3)

Solution:
3 1 −1 2 1 −1
D 1 28
D = 1 4 1 = ... = 28 x1 = 1 = 12 4 1 = =1
D 28 28
2 1 2 10 1 2

3 2 −1 3 1 2
D 1 56 D3 1 84
x2 = 2 = 1 12 1 = =2 x3 = = 1 4 12 = =3
D 28 28 D 28 28
2 10 2 2 1 10

Caution: do not use this method for n>3; inefficient

3.2 Matrix Norms
◊ Scalar: If a= -10 and b =0.1 are two scalar numbers, we can compare their magnitudes:
|a|=10 > |b| = 0.1
◊ Vectors: If u = 3 ex + 4 ey & v = 5 ex + 12 ey are two vectors,
 |u| = 5 & |v| = 13 are their magnitudes
we conclude: |u| < |v|.
◊ Matrices: How do we compare the "magnitude" of two matrices A & B?
Answer: need to use NORM ||A ||.
◊ Norm ||A || will be used in estimating the accuracy of the solution x for system Ax = b.
3.2.1 Properties of a norm
A properly defined norm must possess the following properties:
i. || A || ≥ 0; || A || = 0 if and only if A=0
ii. || kA || = |k| || A ||
iii. || A + B || ≤ || A || + || B || -- triangle inequality.
iv. || AB || ≤ || A || * || B ||
3.2.2 Lp-norm of a vector x

◊ For a vector x, we define the Lp-norm of x as

𝑛
|| x ||p = (|x1| p + |x2| p + ... + |xn| p)1/p = (σ𝑖=1 |xi| p )1/p

◊ Commonly used norms:

p = 1: || x ||1 = sum of magnitude of each component.
𝑛
p = 2: || x ||2 = || x ||e = 𝑥12 + 𝑥22 +. . . +𝑥𝑛2 = (෌𝑖=1 𝑥𝑖2 )1/2
is also called Euclidean norm of vector x.
p→∞: || x || = maximum of {|x1|, |x2|, ... |xn|}
= maximum norm.

* Example: For u = 5 ex + 12 ey, || u ||1 = 17, || u ||2 = 13, || u || = 12

* In general, || u ||1 ≥ || u ||2 ≥ || u ||.

 Inner product

◊ Scalar product of vectors u and v:

<u, v> = u1v1 + u2v2 + ... + unvn or simply u  v = uj vj.

It is also called inner product of vectors u and v.

◊ Unit vector u: u  u = 1, i.e. a unit vector u has a length of 1.

◊ Cauchy-Schwartz inequality:
For all real or complex valued vectors u and v ,
|<u, v>|2 ≤ <u, u>  <v, v>
or |<u, v>| ≤ || u ||2  || v ||2
3.2.3 Lp-norm of a matrix A
row sum:
Define
5 −5 −7 17
𝑛 max= 17
A= −4 2 −4 10
= max. column sum norm = 𝐴 ∞
𝑝 = 1, 𝐴 1 = max ෍ 𝑎𝑖𝑗 −7 −4 5 16
1≤𝑗≤𝑛
𝑖=1
column sum: 16 11 16
𝑛

𝑝 = ∞, 𝐴 = max ෍ 𝑎𝑖𝑗 = max. row sum norm max= 16 = 𝐴 1

∞
1≤𝑖≤𝑛
𝑗=1

n n
𝐴 = 52 + −5 2 + −7 2 + ⋯ + −4 2 + 52 =15
Ae=  aij2 = Frobenius norm 𝑒

i =1 j =1 or Euclidean norm

90 −5 −54
Note: || A ||2 = spectral norm = max{|i|1/2}
A*A= −5 40 7
= smallest among all norms −54 7 90

where i’s are the eigenvalues of (A*A). eig(A*A)= 35.7666, 44.5113, 144.7221
& A* = conjugate transpose of A.
2 =144.7221
𝐴 1/2=12.0301
3.3 Gauss Elimination Method and Gauss-Jordan Method
◊ It is a direct method (in contrast to iterative method)
◊ It is good for dense matrix (i.e. aij have very few zeroes).
3.3.1 Gauss elimination method
3 1 − 1  x1   2  − − −1
◊ Example: Solve x in : 1 4 1   x  = 12  − − −2
   2   pivot
2 1 2   x3  10  − − −3
element
Soln.  3 1 −1 2  ....1
 
i. Augmented matrix: [A | b] = 1 4 1 12 ....2
 2 1 2 10 ....3
ii. Reduction (to reduce A to an upper triangular matrix)
Reduce the 1st column below a11:

R1=> R 1’ [3 1 -1 2 … (1’)

R 2- R 1* (1/3) [0 3.6667 1.3333 11.3333 … (2’) f21=1/3

Scratch: R 2: [1 4 1 12 ] =a21/a11
R 1 * (1/3): [1 1/3 -1/3 2/3 ]
1
Subtract  [0 11/3 4/3 11
3 ]
pivot row [3 1 -1 2] … (1’)

[0 3.6667 1.3333 11.3333 ] … (2’)

[2 1 2 10 ] … (3) f31= 2 /3 = a31 /a11

Need to reduce 2 to 0
R3- R1’ * (2/3) => R3’:

Scratch: R3: [2 1 2 10 ]
R1 * (2/3): [2 2/3 -2/3 4/3 ]

Subtract => [0 1/3 8/3 26/3 ] … (3’)

Result is:
3 1 −1 2  ....1' Pivot
 0 3.6667 1.3333 11.3333 ....2 ' element
  for 2nd
0 0.3333 2.6667 8.6667  ....3'
column

Need to get rid of this guy now.

 Reduce the 2nd column below a22:
3 1 −1 2  ....1' pivot
0 3.6667 1.3333 11.3333 ....2 ' Element a22
 
0 0.3333 2.6667 8.6667  ....3'
f32=0.3333/3.6667=1/11
Need to reduce 0.3333 to 0

R 1'  R 1" 3 1 −1 2  ....1"

R 2 '  R 2" 0 3.6667 1.3333 11.3333 ....2"
0.3333  
R 3'-R 2'*( )  R 3 " 0 0 2.5455 7.6365 ….3
3.6667 3 1 −1
• Now A has become an upper-triangular matrix U. U= 0 3.6667 1.3333
0 2.5455 7.6365
• Put fij to get a lower triangular matrix (for later use):
 1 0 0
L = 1/ 3 1 0 
 2 / 3 1/11 1 
 3 1 −1 2  ....1"
iii. Back substitution 0 3.6667 1.3333 11.3333 ....2"
 
0 0 2.5455 7.6365  ....3"
 Lecture 11
Last lecture:
Cramer's rule
Matrix norm
Gauss elimination
This lecture:

Gauss elimination
3.3 Gauss Elimination Method and Gauss-Jordan Method

3.3.1 Gauss elimination method

3 1 − 1  x1   2  − − −1
◊ Example: Solve x in : 1 4 1   x  = 12  − − −2
   2  
2 1 2   x3  10  − − −3

Soln.  3 1 −1 2  ....1
 
i. Augmented matrix: [A | b] = 1 4 1 12  ....2
 2 1 2 10  ....3
ii. Reduction (to reduce A to an upper triangular matrix)

3 1 −1 2  ....1"
0 3.6667 1.3333 11.3333 ....2"
 
 0 0 2.5455 7.6365 ….3


 1 0 0 3 1 −1
L = 1/ 3 1 0  U= 0 3.6667 1.3333
0 0 2.5455
 2 / 3 1/11 1 
 3 1 −1 2  ....1" 3x1+x2-x3=2
iii. Back substitution 0 3.6667 1.3333 11.3333 ....2" 3.6667x2+1.3333x3=7.6365
 
0 0 2.5455 7.6365  ....3" 2.5455x3= 7.6365

• Solve for x3 (from 3") : x3= 7.6365 / 2.5455 = 3.0

• R2”: a22*x2 +a23*x3 =b2 => x2=(b2 - a23*x3 )/a22
• Solve for x2 (from 2") : x2= (11.3333-1.3333*3.0)/3.6667=2.0
• R1”: a11*x1 +a12*x2 +a13*x3 =b1 => x2=(b1 - a12*x2 –a13*x3 )/a11
* Solve for x1 (from 1") : x1= (2 -x2+ x3)/3 = 1.0
Gauss Elimination Derivation
a11 x1 + a12 x2 + ... + a1n xn = b1 Pivot eqn for 1st column
Begin with a 21 x1 + a22 x2 + ... + a2 n xn = b2 a11=pivot element
a 31 x1 + a32 x2 + ... + a3n xn = b3

an1 x1 + an 2 x2 + ... + ann xn = bn

Row2- Row1* a21 / a11:  a21   a21   a21  a21
 21
a − a11  1
x +  22
a − a12  2
x + ... +  2n
a − a1n  n
x = b2 − b1
 a11   a11   a11  a11

 a31   a   a  a
Row3- Row1* a31 / a11:  a31 − a11  x1 +  a32 − 31 a12  x2 + ... +  a3n − 31 a1n  xn = b3 − 31 b1
 a11   a11   a11  a11

Repeat the forward elimination for this column to get

a11 x1 + a12 x2 + ... + a1n xn = b1
 a21 
 x2 + ... + a2 n xn = b2
a22 =
 2 b − b1 
 a11 
Now 1st column done!
 x2 + ... + a3n xn = b3
a32

an 2 x2 + ... + ann

 xn = bn 𝑓21 = 𝑎21 /𝑎11
𝑓31 = 𝑎31 /𝑎11
➢ Multipliers used in reducing the 1st column: ...
 Gauss Elimination a11 x1 + a12 x2 + ... + a1n xn = b1
a22 x2 + ... + a2 n xn = b2
Now work on reducing 2nd column.  x2 + ... + a3n xn = b3
a32

Pivot equation:  x2 + a23

a22  x3 + + a2 n xn = b2
an 2 x2 + ... + ann
 xn = bn
(a'22 = pivot element for the 2nd column)

Row3- Row2*a'32 /a'22:  a   a   a 

 − 32 a23
+  a33   x3 + +  a3 n − 32 a2 n  xn =  b3 − 32 b2 
 
a22   
a22   
a22 

…
Repeat the elimination of ai2 and get
a11 x1 + a12 x2 + a13 x3 + + a1n xn = b1
 x2 + a23
a22  x3 + + a2 n xn = b2
 a 
 x3 +
a33 + a3n xn = b3  = b3 − 32 b2 
 
a22 

an3 x3 +  xn = bn
+ ann
f 32 = a32
'

/ a22
➢ Multipliers used in reducing the 2nd column: f 42 = a42
'

/ a22
...
 Gauss Elimination

➢ Continue to work on the 3rd & 4th columns … to get

a11 x1 + a12 x2 + a13 x3 + + a1n xn = b1

 x2 + a23
a22  x3 + + a2 n xn = b2
 x3 +
a33 + a3n xn = b3

( n −1)
ann xn = bn( n −1)

f 43 = a43
"
/ a3"3
➢ Multipliers used in reducing the 3rd column:
f53 = a53
" "
/ a33
...
Back Substitution

Now we can perform back substitution to get {x}

( n −1)
Last row: ann xn = bn( n −1)
Simple division  ( n −1)
bn
xn = ( n −1)
ann
Substitute this into (n-1)th equation

an( n−−1,2)n −1 xn −1 + an( n−−1,2)n xn = bn( n−1−2)

Solve for xn-1
Repeat the process to solve for xn-2 , xn-3 , …. x2, x1
 3.3.2 Gauss -Jordan Method
* Consider the same example.
* Start from rows 1" to 3" after reduction step, NO BACK SUBSTITUTION!
* (row 1")/3.0, (row 2")/3.6667, (row 3")/2.5455 

3 1 −1 2  ....1" 1 0.3333 −0.3333 0.6667 . . . . 𝑖

0 3.6667 1.3333 11.3333 ....2" All diagonal elements are 1
   0 1 0.3636 3.0909 . . . . 𝑖𝑖
0 0 2.5455 7.6365….3 0 0 1 3.0 . . . . 𝑖𝑖𝑖

1 0 −0.4545 0.3636 . . . . 𝑖′
* (row i) - (row ii)*0.3333 
0 1 0.3636 3.0909 . . . . 𝑖𝑖′
0 0 1 3.0 . . . . 𝑖𝑖𝑖′
𝑅𝑖′ + 0.4545 × ∗ሺ𝑅 3∗) ⇒ 𝑅 1∗ 1 0 0 1 ....1 ∗
R 𝑖𝑖′ − 0.3636 × ሺ𝑅 3 ∗) ⇒ 𝑅 2 ∗ 0 1 0 2 ....2 ∗  x = (x1, x2, x3)T = (1.0000, 2.0000, 3.0000)T.
ሺ𝑟𝑜𝑤 𝑖𝑖𝑖′) ⇒ 𝑅 3 ∗ 0 0 1 3 ....3 ∗
• Gauss-Jordan method involves 50% more work.

• Gauss-Jordan method is used mainly to find inverse matrix

3.3.3 Pivoting

−0.002 4 4 𝑥1 7.998 −− −1 ◊ Back substitution:

Consider −2 2.906 −5.387 𝑥2 = −4.481 −− −2
3 −4.031 −3.112 𝑥3 −4.143 −− −3 x3 = 13/10.5 = 1.238

◊ Exact solution is (x1, x2, x3)T = (1, 1, 1)T. x2 = (-8002 + 4005 x3 )/(-3997) = 0.762

◊ Augmented matrix: −0.002 4 4 7.998 −− −1 x1 = (7.998 - 4.0 x3 -4.0 x2) / (-0.002) = 1.0
−2 2.906 −5.387 −4.481 −− −2
3 −4.031 −3.112 −4.143 −− −3
◊ inaccurate results due to round off error.
◊ Reduction using standard Gaussian elimination
& keep ONLY 4 significant digits: ◊ If aii=0, we have to interchange rows in
order to continue the computation.
R2 – (-2)/(-0.002)*R1, R3 – (3)/(-0.002)*R1,

−0.002 4 4 7.998 −− −1′

0 −3997 −4005 −8002 −− −2′
0 5996 −5997 11990 −− −3′
R3 ′ – 5996/(-3997)*R2 ′,
−0.002 4 4 7.998 −− −1
0 −3997 −4005 −8002 −− −2
0 0 10.5 −13 −− −3
 Applying partial PIVOTING (still use ONLY 4 significant digits)

−0.002 4 4 𝑥1 7.998 − −1 𝑥1
3 −4.031 −3.112 −4.143 … 1′
−2 2.906 −5.387 𝑥2 = −4.481 − −2 𝑥2 = −4.481 … 2′
−2 2.906 −5.387
3 −4.031 −3.112 𝑥3 −4.143 − −3 𝑥3
−0.002 4 4 7.998 … 3′
Rearrange A by exchanging row 3 with row 1
3 −4.031 −3.112 −4.143… 1
The element with the largest magnitude in the column is now f21= -0.6667 0 0.2187 −7.4616 −7.243… 2
in the pivot position (as diagaonl element) f31= -0.6667E-4 0 3.997 3.998 7.995 … 3

interchange row 3 with row 2

3 −4.031 −3.112 −4.143… 1′ 3 −4.031 −3.112 −4.143 … 𝑖
0 3.997 3.998 7.995 … 2′ 0 3.997 3.998 7.995 … 𝑖𝑖
0 0.2187 −7.4616 −7.243… 3′ f32= 0.05470 0 0 −7.680 −7.680… 𝑖𝑖𝑖

◊ Back substitution:
x3 = −7.680 /(−7.680) = 1.000
x2 = (7.995 -3.998)/3.997 =1.000
x1 = (−4.143 + 4.031 + 3.112)/3 =1.000
which are exact within 4 significant digits
 Lecture 12
Last lecture:
Gauss elimination
Gauss -Jordan Method
Pivoting
This lecture:

Gauss elimination code

Operation count
Tridiagonal matrix
 MATLAB M-File: GaussPivot From a textbook
function x = GaussPivot(A,b)
% x = GaussPivot(A,b): Gauss elimination with pivoting. by Chapra
% input:
% A = coefficient matrix
% b = right hand side vector
% output:
% x = solution vector
[m,n]=size(A);
if m~=n, error('Matrix A must be square'); end
nb=n+1;
Aug=[A b]; largest element in {w}
% forward elimination
for k = 1:n-1
kp1=k+1
% partial pivoting
[big,i]=max(abs(Aug(k:n,k))); [big,i] = max(w)
ipr=i+k-1;
Partial if ipr~=k
Pivoting %pivoting the rows
Aug([k,ipr],:)=Aug([ipr,k],:); index of the largest element in w
end
for i = kp1:n
factor=Aug(i,k)/Aug(k,k); (excluding everything above akk)
Aug(i,k)=0;
Aug(i,kp1:nb)=Aug(i,kp1:nb)-factor*Aug(k,kp1:nb);
end
end Aug = (k=2)
% back substitution 6. 2.000 2.000 4.000 1.000
x=zeros(n,1); 0 2.333 2.333 -2.333 -0.833
x(n)=Aug(n,nb)/Aug(n,n); 0 1.000 1.000 4.000 2.000
for i = n-1:-1:1
x(i)=(Aug(i,nb)-Aug(i,i+1:n)*x(i+1:n))/Aug(i,i);
0 -0.333 1.667 2.333 0.833
end w
3.3.4 Operation count in solving system of equations
• Computing determinant |A| using minors
Determinants for small matrices are:
1×1 𝑎11 = 𝑎11

𝑎11 𝑎12
2×2 = 𝑎11∗𝑎22 − 𝑎12∗𝑎21  2 M/D
𝑎21 𝑎22

𝑎11 𝑎12 𝑎13

𝑎 𝑎23 𝑎 𝑎23 𝑎 𝑎22
3 × 3 𝑎21 𝑎22 𝑎23 = 𝑎11∗ 22 − 𝑎12∗ 21 + 𝑎13∗ 21
𝑎32 𝑎33 𝑎31 𝑎33 𝑎31 𝑎32
𝑎31 𝑎32 𝑎33
 （3+3*2） M/D

4 × 4: 𝑎11∗𝑀11 −𝑎12∗𝑀12 + 𝑎13∗𝑀13 − 𝑎14∗𝑀14  （4+4*(3+3*2）M/D

n×n  O(n!） M/D

 Operation Count
Important issue for a large matrix
Gauss elimination routine uses O(n3/3) M/D
Cramer’s rule:
Using expansion by minors: (n+1)! M/D;
Using Gauss elimination for Determinants: O(n4/3) M/D
Back-substitution uses O(n2/2)

n (n+1)! n3/3
10 3628800 333
20 2.4329E+18 2667
40 8.1592E+47 21333

How much does it cost? $2.5E+37 • 2018 US GDP= $ 20.54 trillion = $ 2*1013
 a11 x1 + a12 x2 + a13 x3 + + a1n xn = b1
Operation Count  x2 + a23
a22  x3 + + a2 n xn = b2
for k = 1:n-1
for i = k+1:n
(n-1)
 x3 +
a33 + a3n xn = b3
factor = Aug(i,k)/Aug(k,k)
rows
Aug(i,k:nb) = Aug(i,k:nb)-
( n −1)
factor*Aug(k,k:nb) ann xn = bn( n −1)
end
end
n col.
Outer Loop Inner Loop Addition/Subtraction Multiplication/Division +1
k i flops flops f 21 = a21 / a11
f31 = a31 / a11
1 2, n (n − 1)(n) (n − 1)(n + 1) f 41 = a41 / a11
2 3, n (n − 2)(n − 1) (n − 2)(n) ...

k k + 1, n (n − k )(n − k + 1) (n − k )(n − k + 2)

n −1 n, n (1)(2) (1)(3)
n (n2 +n )/2 M/D
 i(i-1) = (n3 –n )/3
i=1
n3/3+n2/2-5n/6 Back substitution: (n2 -n )/2 A/S

Total operation counts for elimination stage = 2n3/3 + O(n2)

Total operation counts for back substitution stage = n2
3.3.6 Tri-diagonal algorithm (Thomas algorithm)

Tridiagonal Matrix = banded matrix with bandwidth = 3

Diagonal f
 f1 g1 Superdiagonal g   x1   r1 
e f2 g2  x   r 
 2  2   2 
 e3 f3 g3   x3   r3 
    
Subdiagonal e     
=
 ei fi gi

 xi   ri 
    
    
    
en −1 f n −1 g n −1   xn −1  rn −1 
 
 en f n   xn   rn 

➢ Special case of saving storage, 3  n instead of n  n

➢ Solve by Gauss elimination idea (but not the code)
 (1)  f1 g1   x1   r1 
Tridiagonal Matrix (2)

(3)
e
 2

f2
e3
g2
f3 g3
 x   r 
 2   2 
  x3   r3 
    
Gauss Elimination …     
=
(i)  ei fi gi   xi   ri 
    
f1 g1 r1     
    
en −1 f n −1 g n −1   xn −1  rn −1 
(2) - e2/f1* (1):  
f 2' g2 r2'  en f n   xn   rn 

0 𝑒3 𝑓3 𝑔3 𝑟3

e e2
f = f 2 − 2 g1
' r2' = r2 − r1
2
f1 f1

  x1   r1 
(1)
 f1 g1
(2)  f 2   r  
 g2  2   2 
x
(3) – e3/f2* (2) (3)
   x3    
… f3 g3
 3
  r
     
(i) – ei/fi-1* (i-1) (i)
  x    = f = f −
ek
g k −1
 k k
  i     f k −1
 k = 2,3, ,n
    
r = r − k r e
(n-1) – en-1/fn-2* (n-2)   x   

 k k f k −1 k −1
   n −1   
(n) – en/fn-1*(n-1)    xn   
 

Use factor = ek / fk−1 to eliminate subdiagonal element

Apply the same matrix operations to right hand side r
 Tridiagonal Matrix
Gauss Elimination
(1)
 f1 g1   x1   r1 
(2) e f2 g2  x   r 
 2  2   2 
(3)  e3 f3 g3   x3   r3 
…
    
(i)
    
=
 ei fi gi

 xi   ri 
    
     
    
en −1 f n −1 g n −1   xn −1  rn −1 
 
 en f n   xn   rn 

(1)  f1 g1   r1 
 
 1 x   
(2)
 f 2 g2 
 x2   2 
r 
(3)  f3   
 

… 
g3 
 3 x   r3
    
      ek
(i) =   f k = f k − f g k −1
 f i gi   xi   r i   k −1
      k = 2,3, ,n
  r = r − k r e
  
   k k f k −1 k −1
f n −1 g n −1  xn −1   rn −1 
 
x   

 f n   n   rn 
 Tridiagonal Matrix
➢ Back substitution
→ x1 = (r1 -g1x2)/f1
 f1 g1   x1   r1 
0 f 2' g2   x   r' 
  2   2 
 0 f3' g3   x3   r3' 
    
     
=
 0 fi ' gi

 xi   ri '  → xi = (ri -gixi+1)/fi
    
    
    
0 f n'−1 g n −1   xn −1  rn' −1  xn-1 = (rn-1 -gn-1xn)/fn-1
  →
 0 f n'   xn   rn'  → xn = rn/fn

rn
xn =
fn
rk − g k xk +1
xk = k = n − 1, n − 2, ,3, 2,1
fk
 Lecture 13
Last lecture:
Gauss elimination code
Operation count
Tridiagonal algorithm
This lecture:

L-U decomposition based on Gaussian elimination

Crout reduction
Evaluation of Determinant
3.4 L-U Decomposition and Matrix Inversion
3.4.1 L-U Decomposition based on Gaussian elimination
3 1 − 1  x1   2 
1 4 1     
   x2  = 12 
2 1 2   x3  10 
 
3 1 −1 2  ....1"
 
Reduction using Gauss elimination  0 3.6667 1.3333 11.3333 ....2"
0 0 2.5455 7.6365  ....3"

 1 0 0
Put “multiplying coefficients” to get L L = 1/ 3 1 0 
 2 / 3 0.0909 1 

 1 0 0 3 1 −1   3 1 −1
LU = 1/ 3 1 0  0 3.6667 1.3333  = 1 4 1  = A
 
 2 / 3 0.0909 1  0 0 2.5455  2 1 2 

Why?
3.4.1 L-U Decomposition based on Gaussian elimination

Reduction in Gauss elimination: After the first column is reduced,

′
consider now reducing 2nd column below 𝑎22 .
a11 x1 + a12 x2 + ... + a1n xn = b1  a12   a12 
1 0 0 0 ... 0  a   a 
 x2 + ... + a2 n xn = b2
a22 Let
0
 1 0 0 ... 0   22   22 
 0 −m3 1 0 0  a32  0
 x2 + ... + a3 n xn = b3
a32 M2 = 
...
, we want M 2  = 

 0 −m4 0 1 ... 0 a
 42   0 
... ... ... ... ... ...  ...   ... 
     
an 2 x2 + ... + ann
 xn = bn  0 −mn 0 0 0 1   an 2   0 

This can be accomplished by using:

′ ′
𝑚𝑖 = 𝑎𝑖2 /𝑎22 , i=2+1,…,n

• Mk is called “elementary elimination matrix” or “Gauss transformation”

• Mk is a lower triangular matrix with unit main diagonal  Mk is nonsingular.
 ′ ′
3.4.1 L-U Decomposition based on Gaussian elimination 𝑚𝑖 = 𝑎𝑖2 /𝑎22 , i=2+1,…,n

1 0 0 0 ... 0 0 0 1 0 0 0 ... 0 0
0 1 0 0 ... 0 0 1 0 1 0 0 ... 0 0
0 −𝑚3 1 0 ... 0 𝑚3 0 0 0 1 0 ... 0 0
𝑀2 = , 𝒎= , 𝒆2 = , 𝑀3 = , 𝒕= ,
0 −𝑚4 0 1 ... 0 𝑚4 0 0 0 −𝑡4 1 ... 0 𝑡4
... ... ... ... ... ... … … ... ... ... ... ... ... …
0 −𝑚𝑛 0 0 0 1 𝑚n 0 0 0 −𝑡n 0 0 1 𝑡n

• Mk = I - m𝑒𝑘𝑇 , where m = [0,…,0, mk+1,…,mn]T and ek is the kth column of the identity matrix.
• 𝑀𝑘−1 =I+ m 𝒆𝑇𝑘 , which means that Lk= 𝑀𝑘−1 is the same as Mk except that the signs of the multipliers are reversed. Why?
Mk 𝑀𝑘−1 = I- m𝑒𝑘𝑇 + m𝑒𝑘𝑇 - m 𝑒𝑘𝑇 m 𝑒𝑘𝑇 = I
• If Mj , j > k, is another elementary elimination matrix, with vector of multipliers t, (e.g. M3 = I - t𝑒3𝑇 ) then 0
Mk Mj = I - m𝑒𝑘𝑇 - t𝑒𝑗𝑇 + m 𝑒𝑘𝑇 t 𝑒𝑗𝑇 = I - m 𝑒𝑘𝑇 - t 𝑒𝑗𝑇 since 𝑒𝑘𝑇 t = 0 0
𝑚3
𝑒2𝑇 m = [0,1,0,…,0] 𝑚4 = 0
0
0
0 𝑚n
𝑒2𝑇 t = [0,1,0,…,0] • Thus, their product is essentially their “union."
−𝑡4 = 0
• Because they have the same form, a similar result
holds for the product of their inverses, LkLj .
−𝑡n
3.4.1 L-U Decomposition based on Gaussian elimination A=LU
1 0 0 0 ... 0 0 0 1 0 0 0 ... 0 0
0 1 0 0 ... 0 0 1 0 1 0 0 ... 0 0
0 −𝑚3 1 0 ... 0 𝑚3 0 0 0 1 0 ... 0 0
𝑀2 = , 𝒎= , 𝒆2 = , 𝑀3 = , 𝒕= ,
0 −𝑚4 0 1 ... 0 𝑚4 0 0 0 −𝑡4 1 ... 0 𝑡4
... ... ... ... ... ... … … ... ... ... ... ... ... …
0 −𝑚𝑛 0 0 0 1 𝑚n 0 0 0 −𝑡n 0 0 1 𝑡n

For A{x} = {b}, to reduce the 1st column  multiply M1 on both sides
to reduce the 2nd column  multiply M2 on both sides AGAIN
…
Mn-1 …M2M1A{x} = Mn-1 …M2M1 {b} = M{b}
After reduction is complete, LHS becomes U{x} with U=MA

 U{x} =M{b}

 M-1U{x} = {b}
 M-1 is the L matrix we are looking for .

L= ( Mn-1 …M2M1 )-1 = 𝑀1−1 𝑀2−1 … 𝑀𝑛−1

−1
 L1 L2…Ln-1
3.4.1 L-U Decomposition based on Gaussian elimination

L= L1 L2…Ln-1

= I + f1 𝑒1𝑇 + f2 𝑒2𝑇 +…+ fn-1 𝑒𝑛−1

𝑇

1 0 0 1 0 0 0 ... 0
𝑎21 /𝑎11 1 0 𝑎21 /𝑎11 1 0 0 ... 0
′ ′ ′ ′
𝑎 /𝑎 𝑎32 /𝑎22 1 𝑎 /𝑎 𝑎32 /𝑎22 1 0 ... 0
f1 = 31 11 , f2 = ′ ′ , f3 = ′′ ′′ …  L= 31 11 ′ ′ ′′ ′′
𝑎41 /𝑎11 𝑎42 /𝑎22 𝑎43 /𝑎33 𝑎41 /𝑎11 𝑎42 /𝑎22 𝑎43 /𝑎33 1 ... 0
… … … ... ... ... ... ... ...
′ ′ ′′ ′′ ′ ′ ′′ ′′
𝑎𝑛1 /𝑎11 𝑎𝑛2 /𝑎22 𝑎𝑛3 /𝑎33 𝑎𝑛1 /𝑎11 𝑎42 /𝑎22 𝑎𝑛3 /𝑎33 … … 1

They are the multipliers used in the

reduction of each column.
 WHY LU Decomposition?
Equations in the form of [A] {x} = [L] [U] {x} = {b}
3 1 − 1  x1   b1 
1 4 1     
Suppose we have 100 different {b}’s.    x2  =  b 2 
2 1 2   x3   b 
 3
We DO NOT want repeat Gauss elimination 100 times!
LU Advantage:
Once we have [L] and [U],
we can solve for many different {b}’s without repeating the decomposition process.
LU factorization / decomposition
[ A ] = [L] [U]
does not involve the RHS {b} !!!
Very efficient for large matrices !
How to solve for x efficiently using LU Decomposition?

[L] [U] {x} = {b}

First let [U] {x} = {y} = intermediate variables
 [L] {y} = {b}
Solve {y} given [L] and {b} using forward substitution.
 O(n2) operation count
Then solve {x} from

[U] {x} = {y}

using back substitution
 again, O(n2) operation count!
3.4.2 Crout reduction

 Note: for this method, U has 1’s in the diagonal.

 Idea: consider a 4x4 matrix A. Decompose it into LU as

𝑎11 𝑎12 𝑎13 𝑎14 𝑙11 0 0 0 1 𝑢12 𝑢13 𝑢14

𝑎21 𝑎22 𝑎23 𝑎24 𝑙 𝑙22 0 0 0 1 𝑢23 𝑢24
A= 𝑎
31 𝑎32 𝑎33 𝑎34 = 𝑙21 𝑙32 𝑙33 0 0 0 1 𝑢34
C
31
𝑎41 𝑎42 𝑎43 𝑎44 𝑙41 𝑙42 𝑙43 𝑙44 0 0 0 1
with uii =1 for i=1 to 4.

◊ Procedure: (you need to multiply L & U out to get cij, then compare each cij with aij in the original matrix A)
• First column in C gives the 1st column in L; it is easy to find:

l11 = a11, l21 = a21, l31 = a31, l41 = a41  li1 = ai1
• Now, the 1st row in C gives the 1st row in U as follows: l11 u12 = a12, l11 u13 = a13, l11 u14 = a14  u1j = a1j/ l11
• Now, the 2nd column in C gives the 2nd column in L:
3.4.2 Crout reduction

• Now, 2nd column in C gives the 2nd column in L:

𝑎11 𝑎12 𝑎13 𝑎14 𝑙11 0 0 0 1 𝑢12 𝑢13 𝑢14
l21 u12 + l22 = a22 𝑎21 𝑎22 𝑎23 𝑎24 𝑙21 𝑙22 0 0 0 1 𝑢23 𝑢24
𝑎31 𝑎32 𝑎33 𝑎34 =
l31 u12 + l32 = a32 𝑙31 𝑙32 𝑙33 0 0 0 1 𝑢34
𝑎41 𝑎42 𝑎43 𝑎44 𝑙41 𝑙42 𝑙43 𝑙44 0 0 0 1
l41 u12 + l42 = a42
or li1 u12 + li2 = a i2, i=2, 3, 4
so that li2 = ai2 - li1 u12, i=2, 3, 4 Generalize to: 𝑗−1

lij = aij - ෍ lik𝑢𝑘𝑗

𝑘=1
o The 2nd row in C gives the 2nd row in U as:
𝑗−1
l21u13 + l22 u23 = a23 1
uij = l ( aij - ෍ lik𝑢𝑘𝑗 )
ii
l21u14 + l22 u24 = a24 𝑘=1

so that u2j = (a2j - l21u1j)/l22, j=3, 4

◊ Pivoting in Crout reduction
• The 3rd column in C gives the 3rd column in L …
Need to avoid small lii or lii = 0 in computing uij.
 Need to record how rows are interchanged.
3.4.2 Crout reduction

◊ Example (of pivoting in Crout reduction)

* Consider 0 2 1 ...1 1 0 0
A= 1 0 0 ...2 & let p=I= 0 1 0
3 0 1 ...3 0 0 1
* Interchange row 1 with row 3 (to avoid dividing by l11 =0) for A & p
3 0 1 0 0 1
A’= 1 0 0  p= 0 1 0 * Interchange row 2 with row 3
0 2 1 1 0 0 3 0 1 3 0 0 1 0 1/3 0 0 1
3 0 1 𝑙11 0 0 1 𝑢12 𝑢13  0 2 1 = 0 𝑙22 0 0 1 𝑢23  p= 1 0 0
 1 0 0 = 𝑙 𝑙 0 0 1 𝑢 1 0 0 1 𝑙32 𝑙33 0 0 1 0 1 0
21 22 23
0 2 1 𝑙31 𝑙32 𝑙33 0 0 1 𝑙21 =0
𝑙11 =3 𝑙11 𝑢12 =0  𝑢12 =0 𝑙31 =1 𝑙21 𝑢12+𝑙22 =2  𝑙22 =2 − 𝑙21 𝑢12 =2
𝑙21 =1  𝑙11 𝑢13 =1  𝑢13 =1/3 𝑙21 𝑢13+𝑙22 𝑢23 =1  𝑢23 =(1-𝑙21 𝑢13)/𝑙22 =1/2
𝑙21 𝑢12+𝑙22 =0  𝑙22 =0 𝑙31 𝑢12+𝑙32 =0  𝑙32 =-𝑙31 𝑢12 =0
𝑙31 =0
𝑙21 𝑢13+𝑙22 𝑢23 =0  𝑢23 =-𝑙21 𝑢13 /𝑙22 Finally, 1 ∗ (1/3) +𝑙32 𝑢23 +𝑙33 =0  𝑙33 =-1/3
dividing by 𝑙22 =0
 Pivoting in Crout reduction
* We can use permutation matrix p on
3 0 1 3 0 0 1 0 1/3 0 0 1 Gauss elimination with partial pivoting.
 0 2 1 = 0 2 0 0 1 1/2 & p= 1 0 0 Ax = b
1 0 0 1 0 −1/3 0 0 1 0 1 0
pA x = LU x = p b
* Suppose b = [5, -1, -2]T is the RHS of Ax = b.
𝑥1 Matlab
0 2 1 5
1 0 0 𝑥2 = −1 >> A=[0 2 1;1 0 0;3 0 1]
3 0 1 𝑥3 −2 >> [L U P]=lu(A)
* Multiply final p on both sides of Ax = b, L= >> y=L\(P*b)
1.0000 0 0
y=
pA = LU 0 1.0000 0
0.3333 0 1.0000 -2.0000
0 0 1 5 −1 U= 5.0000
pb= 1 0 0 −1 = −2 = b’ 3.0000 0 1.0000 -0.3333
0 1 0 −2 5 0 2.0000 1.0000
0 0 -0.3333 >> x=U\y
* Now solving P= x=
0 0 1 -1.0000
3 0 0 1 0 1/3 𝑥1 −1 𝑥1 −1 1 0 0
2.0000
𝑥2 = 2 0 1 0
0 2 0 0 1 1/2 𝑥2 = −2  1.0000
1 0 −1/3 0 0 1 𝑥3 5 𝑥3 1
3.4.3 Evaluation of Determinant
* Determinant of a triangular matrix:
a11 a12 a13 a1n a22 a23 ... a2 n a33 ... a33
0 a22 a23 a2 n 0 a33 ... a3n
D= 0 a33 a3n = a11 * = a11 * a22 a3n = ... = a11*a22*a33…*ann
0
0
0 ann
ann
0 ann

* Determinant of matrix product:

det(AB) = det(A)*det(B)

* Determinant of A=LU 1 0 0 0 𝑢11 𝑢12 𝑢13 𝑢14

𝑙21 1 0 0 0 𝑢22 𝑎23 𝑢24
A= LU  det(LU)=det(L)*det(U) = 𝑙
31 𝑙32 1 0 0 0 𝑢33 𝑢34
𝑙41 𝑙42 𝑙43 1 0 0 0 𝑢44

= u11*u22*u33…*unn

* Operation count: O(2n3/3) M/D +A/S

3.4.4 Matrix Inversion

◊ Inverse matrix can be obtained using Gaussian-Jordan method

1 −1 2
* Consider: A= 3 0 1
1 0 2 1 −1 2 1 0 0
* To find A-1, add unit vectors to A: Aug = ൥3 0 1 อ 0 1 0൩
1 0 2 0 0 1
3 0 1 0 1 0
* Pivoting: R2 R 1  ൥1 −1 2 อ 1 0 0൩
1 0 2 0 0 1
3 0 1 0 1 0
* Gauss elimination:  ቎0 −1 1.667 อ 1 −0.333 0൩
0 0 1.667 0 −0.333 1

* Continue with Gauss-Jordan elimination to 1 0 0 0 0.4 −0.2 0 0.4 −0.2

reduce the upper triangular matrix to obtain ቎0 1 0 อ −1 0 1 ൩  A-1 = −1 0 1
identity matrix on the left of the augmented 0 0 1 0 −0.2 0.6 0 −0.2 0.6
matrix
◊ No need to worry about rows are interchanged, no matter how often.
3.5 Condition Number and Error Analysis

Use graphical method for Ax= b,

2 equations with a slight uncertainty in b’s:
 2 almost parallel lines of finite thickness

x2 x2

Diamond=region of
solution uncertainty

x1 x’1 x”1 x1
Small change in the intercepts of the lines
 Large change in x1 & x2
 Inaccurate solution  Ill-conditioned system
3.5 Condition Number and Error Analysis

3.5.1 Ill-conditioned system 10 7 8 7 32 

7  23
 5 6 5   
 Consider the following 4x4 system: A x = x=
33 = b
8 6 10 9
   
* Exact solution: x = (1, 1, 1, 1)T 7 5 9 10  31
31.9   0.1
 23.1  
• Let b be changed to:    0.1
b *=  0.1
32.9 
   
  
Solve A x* = b*  x* = (-7.2, 14.6, -2.5, 3.1)T  31.1
0.1

i.e. b differs by (0.1, -0.1,0.1, -0.1)T with || b - b* ||2 =0.2

 Large change in x:
x - x* = (8.2, 13.6, 3.5, 2.1) T

|| x - x* ||2 =16.397 Ratio = 82

Example of ill-conditioned system of Equations

10 7 8 7 32 
7  23
5 6 5   
 Consider the following 4x4 system: A x =  =b
8 6 10 9 x= 33
   
7 5 9 10  31

* Exact solution: x = (1, 1, 1, 1)T 31.99   0.01

 23.01  
  0.01
 
• Now let b** (= A x** ) = 32.99   0.01
   
 31.01   0.01

i.e. b** differs from b by (0.01, -0.01, 0.01, -0.01)T; || b - b** ||2 =0.02
Solve for x** from A x** = b**  x** = (0.18, 2.36, 0.65, 1.21)T

(x* = (-7.2, 14.6, -2.5, 3.1)T)

 large change in x;
|| x - x** ||2 =1.6397
Ratio = 82
BAD approximation x** to the true solution can almost
satisfy the relevant equations! WHY???
det(A) =1. Not a problem.
How to measure the sickness of a matrix?
3.5.2 Condition number

◊ Definition: Cond (A) = || A || || A-1 || 10 7 8 7 25 −41 10 −6

7 5 6 5 
 −41 68 −17 10
1.01 0.99 25.25 −24.75 For A = 8 6 10 9 
, A-1 =
◊ Consider A= , A-1 =  
10 −17 5 −3
0.99 1.01 −24.75 25.25 7 5 9 10 
−6 10 −3 2
 || A ||∞ = 2, || A-1 ||∞ = 50.
Use maximum column sum  ||A||1 = 33, ||A-1||1 =136
Hence Cond(A) = 100.
 cond(A) = 33*136 = 4216 (quite big!)
This is not a small number:
4 4
y y Using spectral norm, cond(A) =2984.1
1.01x + 0.99y = 2.02
3 0.99x + 1.01y = 1.98 3  similar magnitude.
2 2
◊ Note Cond (A) = || A || || A-1 ||  || AA-1 || = || I || =1
1 1

0
◊ How does cond(A) affect solution accuracy?
0
1.01x + 0.99y = 1.98
-1 -1 0.99x + 1.01y = 2.02

-2 -2
-2 -1 0 x 1 2 3 4 -2 -1 0 x 1 2 3 4
3.5.3 Error analysis and condition number

* Eqn. A x = b; (x denotes the exact solution) (4) & (5)  || b || / || A || ≤ || x || ≤ || A-1 || || b || (6)

* Numerical solution is 𝑥෤ which is an approximate solution (3) & (6) 

* Error in x: e = x - 𝑥෤ 1 |𝑟| |𝑒|

≤ | 𝑥 | ≤ ||A|| ||𝐴−1 ||
|𝑟|
(7)
||A|| ||𝐴−1|| | 𝑏 | |𝑏|
* Residual of eqn. r = b - A𝑥෤ = b - A (x - e) = A e
due to round-off error. Or 1 |𝑟| |𝑒| |𝑟|
≤ | 𝑥 | ≤ 𝑐𝑜𝑛𝑑(𝐴) (7’)
 e = A-1 r 𝑐𝑜𝑛𝑑(𝐴) | 𝑏 | |𝑏|

 || e || = || A-1 r || ≤ || A-1|| || r || (1) * Typically, || r || is of the order of machine epsilon

* Also note: r = A e  || r || = || A e || ≤ || A || || e || (e.g. ~10-7 for single precision)
 || e || ≥ || r || / ||A|| (2) and || b || ~ O(1) or larger for a non-homogeneous system.
(1) & (2)  || r || / ||A|| ≤ || e || ≤ || A-1|| || r || (3) * If || b || is close to zero, the system is nearly homogeneous
* Relative error of the solution: || e || / || x || = ? which means that the solution x should be close to 0.
* If Cond(A) = O(1), small round-off error
* Ax=b  || b || ≤ || A || || x || (4)
 small relative error in solution || e || / || x ||.
* x = A-1 b  || x || ≤ || A-1 || || b || (5)
* If Cond(A) is large, say O(105), the relative error in solution, || e || / || x ||, can be of O(10-2).
Further computation using x suffers significant loss of accuracy.
* If Cond(A) is of O(105) or O(106), one remedy is to use double precision to obtain x.
However, even double precision is not enough if cond(A) = O(1014).
* Cond(A) is a very important measure of the matrix system.
3.6 Iterative Method
3.6.1 When do we use the iterative method?
Interchange Row 2 with Row 3:
* A sparse matrix with a lot of zero elements.
6 −2 1 𝑥1 11
Reduction step in Gauss elimination method reduces 0 to 0.
−2 7 2 𝑥2 = 5
 wasteful
1 2 −5 𝑥3 −1
 Iterative method w/o touching those 0 elements is preferred.
* Diagonal dominance must be achieved before starting Diagonal dominance now achieved
𝒏

| aii |> ෍ |aij | Divide the ith row by aii,

𝒋=𝟏,𝒋≠𝒊
express xi as a function of other x's for i=1, 2, 3
11 2 1
Consider 6 −2 1 𝑥1 11 x1 = 6
+ 6 x2 - 6 x3
1 2 −5 𝑥2 = −1 5 2 2
−2 7 2 𝑥3 5 x2 = 7 + 7 x 1 - 7 x3
1 1 2
Row 2: 2<1+5=6 x3 = 5 + 5 x 1 + 5 x2
Diagonal dominance
Row 3: 2<1+7=8 not achieved
3.6.2 Jacobi method

 Starting iteration using the following procedure:

(m+1) (𝑚) (𝑚)
𝑥1 = 1.8333 + 0.3333𝑥2 - 0.1667𝑥3
(m+1) (𝑚) (𝑚)
𝑥2 = 0.7143 + 0.2857𝑥1 - 0.2857𝑥3
(m+1) (𝑚) (𝑚)
𝑥3 = 0.2 + 0.2𝑥1 + 0.4 𝑥2
Initial guess: x(0) = (0, 0, 0)T 
x(1) = (1.8333, 0.7143, 0.2)T
x\ m 0 1 2 3 4 ... 7
x1 0 1.8333 2.0380 2.0848 2.0042 ... 2.0000
x2 0 0.7143 1.1809 1.0530 1.0014 ... 1.0004
x3 0 0.2 0.8524 1.0800 1.0382 ... 0.9984

Note: all x values on the RHS are from the mth iteration
 𝑎12 𝑎13 𝑎14 𝑏1
 Convergence consideration 0 − − −
𝑎11 𝑎11 𝑎11 𝑎11
𝑎11 𝑥1 + 𝑎12 𝑥2 + 𝑎13 𝑥3 + 𝑎14 𝑥4 = 𝑏1
𝑎21 𝑎23 𝑎24 𝑏2
𝑎21 𝑥1 + 𝑎22 𝑥2 + 𝑎23 𝑥3 + 𝑎24 𝑥4 = 𝑏2 − 0 − −
𝑎31 𝑥1 + 𝑎32 𝑥2 + 𝑎33 𝑥3 + 𝑎34 𝑥4 = 𝑏3 𝑎22 𝑎22 𝑎22 𝑎
𝐶= 𝑎31 𝑎32 𝑎34 , d = 22
𝑎41 𝑥1 + 𝑎42 𝑥2 + 𝑎43 𝑥3 + 𝑎44 𝑥4 = 𝑏4 𝑏3
− − 0 −
𝑎33 𝑎33 𝑎33 𝑎33
Rewrite the matrix equation 𝑎41 𝑎42 𝑎43
− − − 0 𝑏4
𝑎12 𝑎13 𝑎14 𝑏1 𝑎44 𝑎44 𝑎44
𝑥1 = − 𝑥 − 𝑥 − 𝑥 + 𝑎44
𝑎11 2 𝑎11 3 𝑎11 4 𝑎11
𝑎21 𝑎23 𝑎24 𝑏2  For system of equations, convergence requirement on C (not A)
𝑥2 = − 𝑥 − 𝑥 − 𝑥 +
𝑎22 1 𝑎22 3 𝑎22 4 𝑎22
𝑎31 𝑎32 𝑎34 𝑏3 𝜆𝑖 ≤ 1, 𝜆 = 𝑒𝑖𝑔(𝐶 )
𝑥3 = − 𝑥1 − 𝑥2 − 𝑥4 +
𝑎33 𝑎33 𝑎33 𝑎33
𝑎41 𝑎42 𝑎43 𝑏4  Why?
𝑥4 = − 𝑥1 − 𝑥2 − 𝑥3 +
𝑎44 𝑎44 𝑎44 𝑎44

𝑥 (𝑚+1) = 𝐶𝑥 (𝑚) + 𝑑; 𝐶ii = 0
Convergence consideration

𝑥 (𝑚+1) = 𝐶𝑥 (𝑚) + 𝑑
𝜆𝑖 of 𝐶 is difficult to find.
(𝑚+1) (𝑚)
𝑥𝑒𝑥𝑎𝑐𝑡 = 𝐶𝑥𝑒𝑥𝑎𝑐𝑡 + 𝑑 Diagonal dominance is much easier to check
Subtract 
𝑒 (𝑚+1) = 𝐶𝑒 (𝑚)
= 𝐶 2 𝑒 (𝑚−1) = ⋯
= 𝐶 𝑚+1 𝑒 (0)
 Eigenvalue decomposition: C = P D P-1
𝜆1 0 0 … 0
0 𝜆2 0 … 0
D= 0 0 𝜆3 … 0
… … … ... …
0 0 0 ... 𝜆𝑛

 𝐶 𝑚+1 = P 𝐷𝑚+1 P-1

 If max(|𝜆𝑖 |) <1  𝐷𝑚+1 → 0  𝑒 (𝑚+1) → 0

 3.6.3 Gauss-Seidel point iterative method

 Improve over Jacobi method by using the most recently m x1 x2 x3

updated results on 0 0 0 0
1 1.833333 1.238095 1.061905
𝑥 (m+1) = 𝐶𝑥 (𝑚) + 𝑑; 𝐶ii = 0
 2 2.069046 0.934694 1.061905
(m+1) (𝑚) (𝑚) 3 1.967913 1.002041 0.987687
𝑥1 = 1.833333 + 0.333333𝑥2 - 0.166667𝑥3
4 2.002731 0.994350 0.994399
(m+1) (m+1) (𝑚) 5 1.999049 1.002381 0.998286
𝑥2 = 0.714286 + 0.285714𝑥1 - 0.285714𝑥3
6 2.001078 1.000218 1.000762
(m+1)
𝑥3
(m+1)
= 0.2 + 0.2𝑥1 + 0.4𝑥2
(m+1) 7 1.999945 1.000090 1.000303
8 1.999979 0.999898 1.000025
 Initial guess: x(0) = (0, 0, 0) T  9 1.999961 0.999987 0.999955

1
𝑥1 = 1.833333 + 0.3333 * 0 - 0.166667 * 0 = 1.833333
1
𝑥2 = 0.714286 + 0.285714 * 1.833333 - 0.285714 * 0  Necessary and sufficient condition -- the magnitude of the
= 1.238095 largest eigenvalue of C (not A) is less than 1.
1  Fortunately, many engineering problems of practical
𝑥3 = 0.2 + 0.2*1.833333 + 0.4*1.238095 = 1.061905
importance satisfy this requirement.
 3.6.4 Relaxation method

𝒊−𝟏 𝒏
 General algorithm description for Ax = b: 1 (𝑚+1) − (𝑚)
𝒏
(𝑚+1)
𝑥𝑖 =𝑥𝑖
(𝑚)
+ a [bi - ෍ aij𝑥𝑗 ෍ aij𝑥𝑗 (𝑚)
-a𝑖𝑖𝑥𝑖 ]
ii
෍ aijxj = bi 𝒋=𝟏 𝒋=𝒊+𝟏

𝒋=𝟏 (𝑚)
=𝑥𝑖 + [ xi( m +1) − xi( m ) ]
𝒊−𝟏 𝒏
aii xi = bi - ෍ aijxj − ෍ aijxj  Introduce relaxation factor w:
𝒋=𝟏 𝒋=𝒊+𝟏
𝒏
𝑥𝑖
(𝑚+1) (𝑚)
=𝑥𝑗 + w[ xi( m +1) − xi( m ) ]
𝒊−𝟏
Jacobi: 𝑥 (𝑚+1) = 1 [b -෍ a 𝑥 (𝑚) − ෍ aij𝑥𝑗(𝑚) ]
𝑖 a i ii
ij 𝑗 w >2: solution diverges
𝒋=𝟏 𝒋=𝒊+𝟏
𝒊−𝟏
𝒏 1<w <2: over-relaxation
(𝑚)
G-S: (𝑚+1) 1 (𝑚+1) − ෍ aij𝑥 ]
𝑥𝑖 = a [bi -෍ aij𝑥𝑗 𝑗 w<1: under-relaxation
ii 𝒋=𝒊+𝟏
𝒋=𝟏
𝒏
𝒊−𝟏
(𝑚)
( m +1) 1 (𝑚+1) − ෍ aij𝑥 ]
Let x = [bi - ෍ aij𝑥𝑗 𝑗
i aii 𝒋=𝒊+𝟏
𝒋=𝟏

(𝑚)
 Add & subtract 𝑥𝑖 to G-S scheme:
 −4 1 1 1 𝑥1 1
1 −4 1 1 𝑥2 1
Example: 𝑥3 = 1
1 1 −4 1
1 1 1 −4 𝑥4 1

Use relaxation method, set convergence at | 𝑥 (m+1) - 𝑥 (𝑚) | < 10-5.

w Number of iteration
1.0 24
1.1 18
1.2 13
1.3 11
1.4 14
1.5 18
1.6 24
1.7 35
1.8 55
1.9 >100
 3.7 System of Nonlinear Equations

Circle x2 + y2 = r2
◊ Consider a 2-equation system to start with:
f1(x1, x2) = 0
f1(x,y)=0
f2(x1, x2) = 0 f2(x,y)=0

An example is the intersection of a circle and an ellipse.

Ellipse (x/a)2 + (y/b)2 = 1

◊ For m variables, it can be cast in the vector form as f (x) = 0

with x = (x1, x2,..., xn)T,
& f = (f1, f2,..., fn)T,
roots: r = (r1, r2,..., rn)T.

An example of 3 surface intersection is shown on the right.

 3.7.1 Fixed-point iteration method

• Rearanging: f (x) = 0  x = h(x) and r = h(r),

• Iteration scheme: x(m+1) = h(x(m)) Let l = max || hn || in the domain of x, we have
• Initial guess: x(0) = (x1(0), x2(0)) || e(m+1)|| ≤ l || e(m) ||  || e(m+1) || ≤ lm+1 || e(0) ||.
(m+1)th iteration x1(m+1) = h1(x1(m), x2(m)), It is clear that we need l <1 for convergence.
x2(m+1) = h2(x1(m), x2(m))
• Convergence behavior:
• Advantage of the method: easy to implement.
x(m+1) -r = h(x(m)) -r = h(x(m)) - h(r)  h(m)  (x(m) - r) + ...
• Disadvantage of the method: too slow to converge.
𝜕ℎ1 𝜕ℎ1 𝜕ℎ1
...
𝜕𝑥1 𝜕𝑥2 𝜕𝑥𝑚
𝜕ℎ2 𝜕ℎ2 𝜕ℎ2
where h = ... nxn matrix
𝜕𝑥1 𝜕𝑥2 𝜕𝑥𝑚
... ... ... ...
𝜕ℎ𝑚 𝜕ℎ𝑚 𝜕ℎ𝑚
𝜕𝑥1 𝜕𝑥2
... 𝜕𝑥𝑚

That is e(m+1) = x(m+1) - r  h(m)  ( x(m+1) - r) = h(m)  e(m)

so that || e(m+1) || ≤ || h(m) || || e(m+1) ||
 Fixed-Point Iteration
Example: Solve
𝑓1 (𝑥1 , 𝑥2 , 𝑥3 ) = 𝑥12 + 50𝑥1 + 𝑥22 + 𝑥32 − 200 = 0

𝑓2 (𝑥1 , 𝑥2 , 𝑥3 ) = 𝑥12 + 20𝑥2 + 𝑥32 − 50 = 0

𝑓3 (𝑥1 , 𝑥2 , 𝑥3 ) = −𝑥12 − 𝑥22 + 40𝑥3 + 75 = 0

Rearrange:

𝑥1 = 𝑔1 (𝑥1 , 𝑥2 , 𝑥3 ) = (200 − 𝑥12 − 𝑥22 − 𝑥32 )/50

𝑥2 = 𝑔2 (𝑥1 , 𝑥2 , 𝑥3 ) = (50 − 𝑥12 − 𝑥32 )/20
𝑥3 = 𝑔3 (𝑥1 , 𝑥2 , 𝑥3 ) = (𝑥12 + 𝑥22 − 75)/40
 Fixed-Point Iteration
Example: Rearranged eqns:
𝑥1 = (200 − 𝑥12 − 𝑥22 − 𝑥32 )/50
𝑥2 = (50 − 𝑥12 − 𝑥32 )/20

𝑥3 = (𝑥12 + 𝑥22 − 75)/40

Initial guess: x=[2 2 2]=> 3

Sum |e(i)| =  | xi
( m +1)
m x1 x2 x3 − xi( m ) |
i =1
0 2 2 2
𝑥1 = (200 − 4 − 4 − 4)/50=3.76 1 3.76 2.1 -1.675 5.535
൞ 𝑥2 = 50 − 4 − 4 /20 = 2.5 2 3.5729355 1.6528388 -1.41131 0.897916
𝑥3 = (4 + 4 − 75)/40=−1.675 3 3.6502092 1.7621168 -1.4875564 0.262798
4 3.6271619 1.7231574 -1.4642729 0.08529
5 3.6346066 1.7349801 -1.4718606 0.026855
6 3.6322621 1.7311631 -1.469487 0.008535
7 3.6330071 1.732364 -1.4702437 0.002703
8 3.6327712 1.7319822 -1.4700044 0.000857
9 3.632846 1.732103 -1.4700803 0.000272
10 3.6328223 1.7320647 -1.4700562 8.61E-05
11 3.6328298 1.7320769 -1.4700639 2.73E-05
12 3.6328274 1.732073 -1.4700614 8.65E-06
13 3.6328282 1.7320742 -1.4700622 2.74E-06
14 3.6328279 1.7320738 -1.4700620 8.69E-07
15 3.6328280 1.7320740 -1.4700620 2.76E-07
3.7.2 Newton's method for nonlinear equations

 Fixed-point iteration is too slow.

 Taylor series expansion for f(x): f(x) = f(x(m)) + f(m)(x - x(m)) + ...

Set f(x) = 0   f(m)  (x(m+1) - x(m)) = - f(x(m)) (*)

𝜕𝑓1 𝜕𝑓1 𝜕𝑓1

...
𝜕𝑥1 𝜕𝑥2 𝜕𝑥𝑛
Denote 𝜕𝑓2 𝜕𝑓2 𝜕𝑓2
J(x(m)) =  f(m) = ... = Jacobian
𝜕𝑥1 𝜕𝑥2 𝜕𝑥𝑛
... ... ... ...
𝜕𝑓𝑛𝑚 𝜕𝑓𝑛 𝜕𝑓𝑛
...
𝜕𝑥1 𝜕𝑥2 𝜕𝑥𝑛

(m) (m)
J(x(m)) x(m+1) = J(x ) x - f(x )
(m)
Eqn. (*) 

 x(m+1) = x(m) – J-1(x(m)) f(x(m))

 Convergence: || e(m+1) || ≤ B || e(m) ||2; quadratic

Example: Intersection of Three Surfaces
Solve the nonlinear system of 3 eqns.:
 f ( x, y, z ) = x 2 + y 2 + z 2 − 14 = 0  f1 ( x1, x2 , x3 ) = x12 + x22 + x32 − 14 = 0
 
 g ( x, y , z ) = x + 2 y − 9 = 0 or  f 2 ( x1, x2 , x3 ) = x12 + 2 x22 − 9 = 0
2 2

 h ( x, y , z ) = x − 3 y 2 + z 2 = 0 
  3 1, 2 3
f ( x x , x ) = x1 − 3 x2
2
+ x3 =0
2

𝜕𝑓1 𝜕𝑓1 𝜕𝑓1

𝜕𝑥1 𝜕𝑥2 𝜕𝑥3 2𝑥1 2𝑥2 2𝑥3
Jacobian 𝜕𝑓2 𝜕𝑓2 𝜕𝑓2 = 2𝑥1 4𝑥2 0
[𝐽] =
𝜕𝑥1 𝜕𝑥2 𝜕𝑥3 1 −6𝑥2 2𝑥3
𝜕𝑓3 𝜕𝑓3 𝜕𝑓3
𝜕𝑥1 𝜕𝑥2 𝜕𝑥3
Newton-Raphson Method
Solve the nonlinear system
 2 x1 2 x2 2 x3   x1   f1 
2 x     
 1 4 x2 0  x2  = −  f 2 
 1 −6 x2 2 x3   x3  f 
 3

 xnew   xold   x1 

     
 ynew  =  yold  + x2 
 z   z   x 
 new   old   3 
 Intersection of 3 surfaces
function [J,f] = threesurfaces(x,varargin)  2 x1 2 x2 2 x 3   x 1   f1 
2 x    
f1=x(1)^2 + x(2)^2 + x(3)^2 - 14;
4 x2 0   x 2  = −  f 2 
f2=(x(1)^2 + 2*x(2)^2 - 9);  1 
f3=(x(1) - 3*x(2)^2 + x(3)^2);  1 − 6 x2 2 x 3   x 3  f 
 3
f=[f1;f2;f3];
 x new   x old   x 1 
J=[2*x(1) 2*x(2) 2*x(3);2*x(1) 4*x(2) 0;1 -6*x(2) 2*x(3)];      
end y = y +
 new   old   2   x
 z   z   x 
x =  new   old   3 
>> x=[1;1;1];
1.6667
>> [J f]=threesurfaces(x) 2.1667
J = 4.6667
2 2 2 >> [J f]=threesurfaces(x)
2 4 0
1 -6 2 J =
f = 3.3333 4.3333 9.3333
-11 3.3333 8.6667 0
-6 1.0000 -13.0000 9.3333
-1 f =
>> x=x-J\f 15.2500
3.1667
9.3611
 Intersection of three surfaces
>> x=x-J\f
J =
x =
3.1231 3.6231 5.7918
1.5641
3.1231 7.2461 0
1.8407
1.0000 -10.8692 5.7918
3.2207
>> [J f]=threesurfaces(x) f =
J = 0.1063
3.1282 3.6815 6.4414 0.0017
3.1282 7.3629 0 0.1029
1.0000 -11.0444 6.4414
f = >> x=x-J\f
2.2076 x =
0.2230
1.7721 1.5616
1.8113
>> x=x-J\f
2.8777
x =
1.5616
1.8115
2.8959
 Problem with Newton’s method for nonlinear system – cost is high!

# of function evaluation in each iteration: n

# of derivative evaluation in each iteration: n2
# of algebraic operation (M/D+A/S) for solving a linear system
in each iteration: 2n3/3
!! the most expensive part of the whole computation is in inverting J

 Practical remedy:
after every 5 iterations for xk
Suppose it takes a total of n=15 iterations using this method.

If we only update J(x) after every 5 iterations for xk

carry out the solution using LU decomposition,

store the LU matrix coefficient,

 only 3x 2n3/3 FLOPs instead of 15x 2n3/3 FLOPs

The convergence will be slower since J(x) is not updated immediately;

but overall cost is greatly reduced.
 Lecture 14
Last lecture:
L-U decomposition based on Gaussian elimination
Crout reduction
Evaluation of Determinant
This lecture:
Condition Number and Error Analysis
Iterative Methods
3.5 Condition Number and Error Analysis

Use graphical method for Ax= b,

2 equations with a slight uncertainty in b’s:
 2 almost parallel lines of finite thickness

x2 x2

Diamond=region of
solution uncertainty

x1 x’1 x”1 x1
Small change in the intercepts of the lines
 Large change in x1 & x2
 Inaccurate solution  Ill-conditioned system
3.5.1 Ill-conditioned system

 Example : 10 7 8 7 32 

7 5 6 5   23
* Consider the following 4x4 system: A x =  x=   =b
8 6 10 9  33
   
7 5 9 10  31
* Exact solution: xexact = (1, 1, 1, 1)T
31.9   0.1
 23.1  
• Let b be changed to:    0.1
b *=  0.1
32.9 
   
  
Solve A x* = b*  x* = (-7.2, 14.6, -2.5, 3.1)T  31.1
0.1

i.e. b differs by (0.1, -0.1,0.1, -0.1)T with || b - b* ||2 =0.2

 Large change in x:
xexact - x* = (8.2, 13.6, 3.5, 2.1) T

|| xexact - x* ||2 =16.397 Ratio = 82

Example of ill-conditioned system of Equations

10 7 8 7 32 
7  23
5 6 5   
 Consider the following 4x4 system: A x =  =b
8 6 10 9  x = 33
   
7 5 9 10  31

* Exact solution: xexact = (1, 1, 1, 1)T 31.99   0.01

 23.01  
  0.01
 
• Now let b** (= A x** ) = 32.99   0.01
   
 31.01   0.01

i.e. b** differs from b by (0.01, -0.01, 0.01, -0.01)T; || b - b** ||2 =0.02
Solve for x** from A x** = b**  x** = (0.18, 2.36, 0.65, 1.21)T

(x* = (-7.2, 14.6, -2.5, 3.1)T)

 large change in x;
|| xexact - x** ||2 =1.6397
Ratio = 82
BAD approximation x** to the true solution can almost
satisfy the relevant equations! WHY???
det(A) =1. Not a problem.
How to measure the sickness of a matrix?
 3.5.2 Condition number

◊ Definition: Cond (A) = || A || || A-1 ||

10 7 8 7
7 5 6 5 
25 −41 10 −6
 , −41 68 −17 10
A-1 =
For A =  8 6 10 9  10 −17 5 −3
 
7 5 9 10  −6 10 −3 2

Use maximum column sum  ||A||1 = 33, ||A-1||1 =136

 cond(A) = 33*136 = 4216 (quite large!)

Using spectral norm, cond(A) =2984.1

 similar magnitude.

◊ Note Cond (A) = || A || || A-1 ||  || AA-1 || = || I || =1

◊ How does cond(A) affect solution accuracy?

3.5.3 Error analysis and condition number

* Eqn. A x = b; (x denotes the exact solution) (4) & (5)  || b || / || A || ≤ || x || ≤ || A-1 || || b || (6)

* Numerical solution is 𝑥෤ which is an approximate solution (3) & (6) 

* Error in x: e = x - 𝑥෤ 1 |𝑟| |𝑒|

≤ | 𝑥 | ≤ ||A|| ||𝐴−1 ||
|𝑟|
(7)
||A|| ||𝐴−1|| | 𝑏 | |𝑏|
* Residual of eqn. r = b - A𝑥෤ = b - A (x - e) = A e
due to round-off error. Or 1 |𝑟| |𝑒| |𝑟|
≤ | 𝑥 | ≤ 𝑐𝑜𝑛𝑑(𝐴) (7’)
 e = A-1 r 𝑐𝑜𝑛𝑑(𝐴) | 𝑏 | |𝑏|

 || e || = || A-1 r || ≤ || A-1|| || r || (1) * Typically, || r || is of the order of machine epsilon

* Also note: r = A e  || r || = || A e || ≤ || A || || e || (e.g. ~10-7 for single precision)
 || e || ≥ || r || / ||A|| (2) and || b || ~ O(1) or larger for a non-homogeneous system.
(1) & (2)  || r || / ||A|| ≤ || e || ≤ || A-1|| || r || (3) * If || b || is close to zero, the system is nearly homogeneous

* Relative error of the solution: || e || / || x || = ? which means that the solution x should be close to 0.
* If Cond(A) = O(1), small round-off error
* Ax=b  || b || ≤ || A || || x || (4)
 small relative error in solution || e || / || x ||.
* x = A-1 b  || x || ≤ || A-1 || || b || (5)
 1 |𝑟| |𝑒| |𝑟|
≤ ≤ 𝑐𝑜𝑛𝑑(𝐴) (7’)
𝑐𝑜𝑛𝑑(𝐴) | 𝑏 | |𝑥| |𝑏|

* If Cond(A) is large, say O(105), the relative error in solution, || e || / || x ||, can be of O(10-2).
Further computation using x suffers significant loss of accuracy.
* If Cond(A) is of O(105) or O(106), one remedy is to use double precision to obtain x.
However, even double precision is not enough if cond(A) = O(1014).
* Cond(A) is a very important measure of the matrix system.
3.6 Iterative Method
3.6.1 When do we use the iterative method?
Interchange Row 2 with Row 3:
* A sparse matrix with a lot of zero elements.
6 −2 1 𝑥1 11
Reduction step in Gauss elimination method reduces 0 to 0. −2 7 2 𝑥2 = 5
 Wasteful 1 2 −5 𝑥3 −1

 Iterative method w/o touching those 0 elements is preferred. Diagonal dominance now achieved
* Diagonal dominance must be achieved before starting
𝒏 Divide the ith row by aii,
| aii |> ෍ |aij | express xi as a function of other x's for i=1, 2, 3
𝒋=𝟏,𝒋≠𝒊
11 2 1
Consider 6 −2 1 𝑥1 11 x1 = + x2 - x3
6 6 6
1 2 −5 𝑥2 = −1
5 2 2
−2 7 2 𝑥3 5 x2 = 7 + 7 x1 - 7 x3
1 1 2
Row 2: 2<1+5=6 x3 = 5 + 5 x1 + 5 x2
Diagonal dominance
Row 3: 2<2+7=8 not achieved
3.6.2 Jacobi method

 Starting iteration using the following procedure:

(m+1) (𝑚) (𝑚)
𝑥1 = 1.8333 + 0.3333𝑥2 - 0.1667𝑥3
(m+1) (𝑚) (𝑚)
𝑥2 = 0.7143 + 0.2857𝑥1 - 0.2857𝑥3
(m+1) (𝑚) (𝑚)
𝑥3 = 0.2 + 0.2𝑥1 + 0.4 𝑥2
Initial guess: x(0) = (0, 0, 0)T 
x(1) = (1.8333, 0.7143, 0.2)T
x\ m 0 1 2 3 4 ... 7
x1 0 1.8333 2.0380 2.0848 2.0042 ... 2.0000
x2 0 0.7143 1.1809 1.0530 1.0014 ... 1.0004
x3 0 0.2 0.8524 1.0800 1.0382 ... 0.9984

Note: all x values on the RHS are from the mth iteration
 𝑎12 𝑎13 𝑎14 𝑏1
 Convergence consideration 0 − − −
𝑎11 𝑎11 𝑎11 𝑎11
𝑎11 𝑥1 + 𝑎12 𝑥2 + 𝑎13 𝑥3 + 𝑎14 𝑥4 = 𝑏1
𝑎21 𝑎23 𝑎24 𝑏2
𝑎21 𝑥1 + 𝑎22 𝑥2 + 𝑎23 𝑥3 + 𝑎24 𝑥4 = 𝑏2 − 0 − −
𝑎31 𝑥1 + 𝑎32 𝑥2 + 𝑎33 𝑥3 + 𝑎34 𝑥4 = 𝑏3 𝑎22 𝑎22 𝑎22 𝑎
𝐶= 𝑎31 𝑎32 𝑎34 , d = 22
𝑎41 𝑥1 + 𝑎42 𝑥2 + 𝑎43 𝑥3 + 𝑎44 𝑥4 = 𝑏4 𝑏3
− − 0 −
𝑎33 𝑎33 𝑎33 𝑎33
Rewrite the matrix equation 𝑎41 𝑎42 𝑎43
− − − 0 𝑏4
𝑎12 𝑎13 𝑎14 𝑏1 𝑎44 𝑎44 𝑎44
𝑥1 = − 𝑥 − 𝑥 − 𝑥 + 𝑎44
𝑎11 2 𝑎11 3 𝑎11 4 𝑎11
𝑎21 𝑎23 𝑎24 𝑏2  For system of equations, convergence requirement on C (not A)
𝑥2 = − 𝑥 − 𝑥 − 𝑥 +
𝑎22 1 𝑎22 3 𝑎22 4 𝑎22
𝑎31 𝑎32 𝑎34 𝑏3 𝜆𝑖 ≤ 1, 𝜆 = 𝑒𝑖𝑔(𝐶 )
𝑥3 = − 𝑥1 − 𝑥2 − 𝑥4 +
𝑎33 𝑎33 𝑎33 𝑎33
 Why?
𝑎41 𝑎42 𝑎43 𝑏4
𝑥4 = − 𝑥1 − 𝑥2 − 𝑥3 +
𝑎44 𝑎44 𝑎44 𝑎44


𝑥 (𝑚+1) = 𝐶𝑥 (𝑚) + 𝑑; 𝐶ii = 0
Convergence consideration

𝑥 (𝑚+1) = 𝐶𝑥 (𝑚) + 𝑑
𝜆𝑖 of 𝐶 is difficult to find.
(𝑚+1) (𝑚)
𝑥𝑒𝑥𝑎𝑐𝑡 = 𝐶𝑥𝑒𝑥𝑎𝑐𝑡 + 𝑑 Diagonal dominance is much easier to check
Subtract 
𝑒 (𝑚+1) = 𝐶𝑒 (𝑚)
= 𝐶 2 𝑒 (𝑚−1) = ⋯
= 𝐶 𝑚+1 𝑒 (0)
 Eigenvalue decomposition: C = P D P-1
𝜆1 0 0 … 0
0 𝜆2 0 … 0
D= 0 0 𝜆3 … 0
… … … ... …
0 0 0 ... 𝜆𝑛

 𝐶 𝑚+1 = P 𝐷𝑚+1 P-1

 If max(|𝜆𝑖 |) <1  𝐷𝑚+1 → 0  𝑒 (𝑚+1) → 0

 Lecture 15
Last lecture:
Condition Number and Error Analysis
Jacboi Methods
Diagonal dominance
This lecture:
Gauss Seidel method
Relaxation method
Nonlinear system

Interpolation
 3.6.3 Gauss-Seidel point iterative method

 Improve over Jacobi method by using the most recently m x1 x2 x3

updated results on 0 0 0 0
𝑥 (m+1) = 𝐶𝑥 (𝑚) + 𝑑; 𝐶ii = 0 1 1.833333 1.238095 1.061905
 2 2.069046 0.934694 1.061905
(m+1) (𝑚) (𝑚) 3 1.967913 1.002041 0.987687
𝑥1 = 1.833333 + 0.333333𝑥2 - 0.166667𝑥3
4 2.002731 0.994350 0.994399
(m+1) (m+1) (𝑚) 5 1.999049 1.002381 0.998286
𝑥2 = 0.714286 + 0.285714𝑥1 - 0.285714𝑥3
6 2.001078 1.000218 1.000762
𝑥3
(m+1) (m+1)
= 0.2 + 0.2𝑥1 + 0.4𝑥2
(m+1) 7 1.999945 1.000090 1.000303
8 1.999979 0.999898 1.000025
 Initial guess: x(0) = (0, 0, 0) T  9 1.999961 0.999987 0.999955

1
𝑥1 = 1.833333 + 0.3333 * 0 - 0.166667 * 0 = 1.833333
1
𝑥2 = 0.714286 + 0.285714 * 1.833333 - 0.285714 * 0  Necessary and sufficient condition -- the magnitude of the
largest eigenvalue of C (not A) is less than 1.
= 1.238095
 Fortunately, many engineering problems of practical
1
𝑥3 = 0.2 + 0.2*1.833333 + 0.4*1.238095 = 1.061905
importance satisfy this requirement.
 3.6.4 Relaxation method

 General algorithm description for Ax = b:

𝒏

෍ aijxj = bi 𝒊−𝟏 𝒏
(𝑚+1) (𝑚) 1 (𝑚+1) − ෍ a 𝑥 (𝑚) (𝑚)
𝒋=𝟏 𝑥𝑖 =𝑥𝑖 + a [bi - ෍ aij𝑥𝑗 ij 𝑗 -a𝑖𝑖𝑥𝑖 ]
𝒏 ii 𝒋=𝒊+𝟏
𝒊−𝟏 𝒋=𝟏
aii xi = bi - ෍ aijxj − ෍ aijxj (𝑚)
=𝑥𝑖 + [ xi( m +1) − xi( m ) ]
𝒋=𝟏 𝒋=𝒊+𝟏

𝒊−𝟏 𝒏  Introduce relaxation factor w:

Jacobi: 𝑥 (𝑚+1) = 1 [b -෍ a 𝑥 (𝑚) − ෍ aij𝑥𝑗 𝑚 ]
𝑖 a i ij 𝑗
ii 𝒋=𝒊+𝟏 𝑥𝑖
(𝑚+1)
=𝑥𝑗
(𝑚)
+ w[ xi( m +1) − xi( m ) ]
𝒋=𝟏
𝒏
𝒊−𝟏
G-S: (𝑚+1) 1 (𝑚+1) − ෍ aij𝑥
(𝑚)
] w >2: solution diverges
𝑥𝑖 = [b ෍ aij𝑥𝑗
- 𝑗
aii i
𝒋=𝒊+𝟏 1<w <2: over-relaxation
𝒋=𝟏
𝒊−𝟏
( m +1) 1 w<1: under-relaxation
Let x = [bi - ෍ aij𝑥𝑗(𝑚+1)− σ𝒏𝒋=𝒊+𝟏 aij𝑥𝑗(𝑚) ]
i aii
𝒋=𝟏

(𝑚)
 Add & subtract 𝑥𝑖 to G-S scheme:
 −4 1 1 1 𝑥1 1
1 −4 1 1 𝑥2 1
Example: 𝑥3 = 1
1 1 −4 1
1 1 1 −4 𝑥4 1

Use relaxation method, set convergence at | 𝑥 (m+1) - 𝑥 (𝑚) | < 10-5.

w Number of iteration
1.0 24
1.1 18
1.2 13
1.3 11
1.4 14
1.5 18
1.6 24
1.7 35
1.8 55
1.9 >100
 3.7 System of Nonlinear Equations

Circle x2 + y2 = r2
◊ Consider a 2-equation system to start with:
f1(x1, x2) = 0
f1(x,y)=0
f2(x1, x2) = 0 f2(x,y)=0

An example is the intersection of a circle and an ellipse.

Ellipse (x/a)2 + (y/b)2 = 1

◊ For m variables, it can be cast in the vector form as f (x) = 0

with x = (x1, x2,..., xn)T,
& f = (f1, f2,..., fn)T,
roots: r = (r1, r2,..., rn)T.

An example of 3 surface intersection is shown on the right.

 3.7.1 Fixed-point iteration method

• Rearanging: f (x) = 0  x = h(x) and r = h(r),

• Iteration scheme: x(m+1) = h(x(m))
• Initial guess: x(0) = (x1(0), x2(0))
Let l = max || hn || in the domain of x, we have
(m+1)th iteration x1(m+1) = h1(x1(m), x2(m)),
(m+1) (m), (m)) || e(m+1)|| ≤ l || e(m) ||  || e(m+1) || ≤ lm+1 || e(0) ||.
x2 = h2(x1 x2
It is clear that we need l <1 for convergence.
• Convergence behavior:
x(m+1) - r = h(x(m)) - r = h(x(m)) - h(r)  h(m)  (x(m) - r) + ...
• Advantage of the method: easy to implement.
𝜕ℎ1 𝜕ℎ1 𝜕ℎ1
... • Disadvantage of the method: too slow to converge.
𝜕𝑥1 𝜕𝑥2 𝜕𝑥𝑛
𝜕ℎ2 𝜕ℎ2 𝜕ℎ2
where h = ... nxn matrix
𝜕𝑥1 𝜕𝑥2 𝜕𝑥𝑛
... ... ... ...
𝜕ℎ𝑛 𝜕ℎ𝑛 𝜕ℎ𝑛
𝜕𝑥1 𝜕𝑥2
... 𝜕𝑥𝑛

That is e(m+1) =- x(m+1) + r  h(m)  (- x(m+1) + r) = h(m)  e(m)

so that || e(m+1) || ≤ || h(m) || || e(m) ||
3.7.2 Newton's method for nonlinear equations

 Fixed-point iteration is too slow.

 Taylor series expansion for f(x): f(x) = f(x(m)) + f(m)(x - x(m)) + ...

Set f(x) = 0   f(m)  (x(m+1) - x(m)) = - f(x(m)) (*)

𝜕𝑓1 𝜕𝑓1 𝜕𝑓1

...
𝜕𝑥1 𝜕𝑥2 𝜕𝑥𝑛
Denote 𝜕𝑓2 𝜕𝑓2 𝜕𝑓2
J(x(m)) =  f(m) = ... = Jacobian
𝜕𝑥1 𝜕𝑥2 𝜕𝑥𝑛
... ... ... ...
𝜕𝑓𝑛 𝜕𝑓𝑛 𝜕𝑓𝑛
...
𝜕𝑥1 𝜕𝑥2 𝜕𝑥𝑛

Eqn. (*)  J(x(m)) x(m+1) = J(x(m)) x(m) - f(x(m))

 x(m+1) = x(m) – J-1(x(m)) f(x(m))

 Convergence: || e(m+1) || ≤ B || e(m) ||2; quadratic

Example: Intersection of Three Surfaces
Solve the nonlinear system of 3 eqns.:
 f ( x, y, z ) = x 2 + y 2 + z 2 − 14 = 0  f1 ( x1, x2 , x3 ) = x12 + x22 + x32 − 14 = 0
 
 g ( x, y , z ) = x + 2 y − 9 = 0
2 2
or  f 2 ( x1, x2 , x3 ) = x12 + 2 x22 − 9 = 0
 h ( x, y , z ) = x − 3 y 2 + z 2 = 0 
  3 1, 2 3
f ( x x , x ) = x1 − 3 x2
2
+ x3 =0
2

𝜕𝑓1 𝜕𝑓1 𝜕𝑓1

𝜕𝑥1 𝜕𝑥2 𝜕𝑥3
2𝑥1 2𝑥2 2𝑥3
𝜕𝑓2 𝜕𝑓2 𝜕𝑓2
Jacobian [𝐽] = = 2𝑥1 4𝑥2 0
𝜕𝑥1 𝜕𝑥2 𝜕𝑥3 1 −6𝑥2 2𝑥3
𝜕𝑓3 𝜕𝑓3 𝜕𝑓3
𝜕𝑥1 𝜕𝑥2 𝜕𝑥3

 2 x1 2 x2 2 x3   x1   f1   xnew   xold   x1 

2 x          
 1 4 x2 0  x2  = −  f 2    ynew  =  yold  + x2 
 1 −6 x2 2 x3   x3  f 
 3  z   z   x 
 new   old   3 
Matlab for solving Intersection of 3 surfaces
function [J,f] = threesurfaces(x,varargin)  2 x1 2 x2 2 x 3   x 1   f1 
2 x    
f1=x(1)^2 + x(2)^2 + x(3)^2 - 14; 4 x2 0   x 2  = −  f 2 
f2=(x(1)^2 + 2*x(2)^2 - 9);  1 
f3=(x(1) - 3*x(2)^2 + x(3)^2);  1 − 6 x 2 2 x 3   x 3  f 
 3
f=[f1;f2;f3];  x new   x old   x 1 
J=[2*x(1) 2*x(2) 2*x(3);2*x(1) 4*x(2) 0;1 -6*x(2) 2*x(3)];      
end  y new  =  yold  +  x 2 
z   z   x 
x =  new   old   3 
>> x=[1;1;1];
1.6667
>> [J f]=threesurfaces(x) 2.1667
J = 4.6667
2 2 2 >> [J f]=threesurfaces(x)
2 4 0
1 -6 2 J =
f = 3.3333 4.3333 9.3333
-11 3.3333 8.6667 0
-6 1.0000 -13.0000 9.3333
-1 f =
>> x=x-J\f 15.2500
3.1667
9.3611
 Intersection of three surfaces
>> x=x-J\f
J =
x =
3.1231 3.6231 5.7918
1.5641
3.1231 7.2461 0
1.8407
1.0000 -10.8692 5.7918
3.2207
>> [J f]=threesurfaces(x) f =
J = 0.1063
3.1282 3.6815 6.4414 0.0017
3.1282 7.3629 0 0.1029
1.0000 -11.0444 6.4414
f = >> x=x-J\f
2.2076 x =
0.2230
1.7721 1.5616
1.8113
>> x=x-J\f
2.8777
x =
1.5616
1.8115
2.8959
 Problem with Newton’s method for nonlinear system – cost is high!

# of function evaluation in each iteration: n

# of derivative evaluation in each iteration: n2
# of algebraic operation (M/D+A/S) for solving a linear system
in each iteration: 2n3/3
!! the most expensive part of the whole computation is in inverting J

 Practical remedy:
Suppose it takes a total of n=15 iterations using this method.

If we only update J(x) after every 5 iterations for xk

carry out the solution using LU decomposition,

store the LU matrix coefficient,

 only 3x 2n3/3 FLOPs instead of 15x 2n3/3 FLOPs

The convergence will be slower since J(x) is not updated immediately;

but overall cost is greatly reduced.
 Supplemental Reading for
Chapter III System of Equations

Example 1 Mechanics of matrix multiplication-- illustration

1 0 0 1 2 3  1 0 0 0 0 0  1 2 3
0 0 2   4 5 6  =  0 0 0  +  0 0 2    4 5 6 
i)         
0 0 0 7 8 9  0 0 0 0 0 0  7 8 9

 1 2 3  0 0 0    1 2 3 
  + 14 16 18  = 14 16 18

=  0 0 0     
 0 0 0   0 0 0    0 0 0 
     

1 0 2 1 2 3  1 0 0 0 0 2  1 2 3
0 0 0   4 5 6  =  0 0 0  +  0 0 0    4 5 6 
ii)         
0 0 0 7 8 9  0 0 0 0 0 0  7 8 9

 1 2 3 14 16 18  15 18 21

  +  0 0 0  =  0 0 0 
=  0 0 0     
 0 0 0   0 0 0    0 0 0 
     

0 0 1 1 2 3 7 8 9
1 0 0 4 5 6 = 1 2 3 From the left these
iii)      matrices permute rows
0 1 0 7 8 9 4 5 6

 1 2 3  0 0 1   3 1 2 
4 5 6 1 0 0 = 6 4 5 From the right these
iv)      matrices permute columns
7 8 9 0 1 0 9 7 8
Example 2 Matrix operation – trigonometric function of matrix
The trigonometric functions can be extended to matrices! For instance, the
cosine of a square matrix A is given by

( −1)n 2 n 1 1 1
cos A =  A = I − A2 + A4 − A6 + L .
n =0 (2n )! 2! 4! 6!
1 0
If  =   , find the value of the determinant of cos  .
1 2
Solution:
1 0 1 0  4  1 0
* Note: =  =>  2 =   ,  =  2 4 ,
3(1 + 2 ) 2 
2
1 2 3 2 

 1 0 8  1 0
6 =  2 4 6 ,  =  8 …
3(1 + 2 + 2 ) 2 
2 4 6
3(1 + 2 + 2 + 2 ) 2 

 1 0 
 2n =  2 4
3(1 + 2 + 2 ... + 2
2n − 2
) 2 2n 
n
1 2 1 4 1 6 1 8 1
Thus cos  = I −  +  −  +  + ... + ( −1)  2n
2 4! 6! 8! (2n)!

 1 1 n
1 
1 − + + (−1) + ... 0 

=
2! 4! (2n)! 
n 2n 
 1 2 1 4 2
c21 1 − 2 + 2 + (−1) + ...
 2! 4! (2n)! 

cos(1) 0 
= .
 c21 cos(2)

Thus, det( cos  ) = cos(1) cos(2)

* Comment: you can define exp(A) using series in a similar manner.
Example 3 Matrix operation – patterns in multiplications
0 0 0 0 0 1
1 0 0 0 0 0

0 1 0 0 0 0
Let A =   2005
0 0 1 0 0 0 . What is the third row of A ?
0 0 0 1 0 0
 
0 0 0 0 1 0

Solution:
• First we note that A is almost like an identity matrix. In order to get to A2005,
we try the first few powers to have a better understanding.
0 0 0 0 1 0 0 0 1 0 0 0
0 0 0 0 0 1 0 0 0 1 0 0
 
1 0 0 0 0 0 2 2 0 0 0 0 1 0
• A 2 = AA =  4
, A = A A = 
0 1 0 0 0 0 0 0 0 0 0 1
0 0 1 0 0 0 1 0 0 0 0 0
   
0 0 0 1 0 0 0 1 0 0 0 0

& A6 = A2 A4 = I
Thus, A2005= A2004+1 = (A6)334+1 = IA = A
Hence the third row is: (0 1 0 0 0 0).
Example 4 Matrix operation – rotation of axes.
 1 3
− − 
If A =  2 2  , what is A12 ?
 3 − 
1
 2 2 

Solution 1:
Note: A3 = I
(you can verify this by simply multiplying A three times)
Thus, A12 = (A3)4 = I4 = I.
Solution 2:
cos − sin  
A= with  = 120 = 2/3 in the problem.
cos 
We note:
 sin 
Thus, A u = rotation of u vector by 120 counter-clockwise.
(You should have seen this concept in Pre-calculus.)
Thus, A12 u = rotation of u vector by 12x120 =1440 = 4 x 360
counter-clockwise
=u
It means that A12 must be an identity matrix I.
Example 5 Norm of an inverse matrix
Prove or give a counterexample: if A is a non-singular matrix, then
-1 -1
|| A || = || A ||
-1 -1 -1
Solution: Note: || A A || = || I || =1 and || AA ||  || A || || A ||
-1 -1
 || A ||  1/ ||A || = || A ||
-1 -1
 Thus, in general, || A ||  || A || & the equality holds only for some
special cases.

Example 6 Proving Cauchy-Schwarz inequality

For all real or complex valued vectors u and v ,
2 2
Prove that |<u, v>| ≤ <u, u>  <v, v> or |<u, v>| ≤ || u ||2  || v ||2
Poof: (we consider real valued vectors for simplicity)
Consider the L2-norm of the vector u- v in which  is real.

By definition, ||u- v|| 22  0.

That is, 0 ≤ ||u- v|| 22 = <u- v, u- v >

2
= <u, u > - <u, v> - <v, u> +  <v, v>
Let a=<v, v>, b=2<u, v>, c =<u, u > and
2
y = f() = a +b + c  0 (parabola opening up)
For the above to hold for all real values of , the discriminator
2
b -4ac ≤ 0 must hold.
2
That is, <u, v> ≤ <u, u >  <v, v>
2
or |<u, v>| ≤ || u ||2  || v ||2
which is the Cauchy-Schwarz inequality.
Example 7 Gaussian elimination
Solve the following linear system of equations.
3 y + 4 z = 14.8
4 x + 2 y − z = −6.3
x − y + 5 z = 13.5
Solution: (Gaussian Elimination)
0 3 4 14.8 
* Define an augmented matrix A = 4 2 − 1 − 6.3 .
1 − 1 5 13.5 

* First, apply pivoting. Exchange R1 with R2: =>

4 2 − 1 − 6.3 − −r1
A = 0 3 4 14.8  − −r 2
1 − 1 5 13.5  − −r 3

* Reduction:
No need to work on row 2 since a21 = 0 already.
4 2 − 1 − 6.3  − −r1
r3 − r1 / 4  r3 : A = 0 3 4 14.8  − −r 2
0 − 1.5 5.25 15.075 − −r 3

4 2 − 1 − 6.3  − −r1
r3 + r 2 / 2 => A = 0 3 4 14.8  − −r 2
0 0 7.25 22.475 − −r 3

* Back substitution:
z = 22.475/7.25 = 3.1
y = (14.8-4*3.1)/3 = 0.8
x = (-6.3-2*0.8+3.1)/4 = -1.2
=> (x, y, z) = (-1.2, 0.8, 3.1)
Example 8 Solving tri-diagonal system
6.4 3.2 0 0 − 1.6
3.2 − 1.6 4.8 0 32 
For an augmented matrix A = 
0 4.8 − 9.6 7.2 − 78 
 
0 0 7.2 4.8 20.4 
Find the solution.

Solution:

6.4 3.2 0 0 − 1.6 − − r1

 0 − 3.2 4.8 0 32.8 
3.2  − −r 2
r2  r2 − * r1 => A=
6.4 0 4.8 − 9.6 7.2 − 78  − −r 3
 
0 0 7.2 4.8 20.4  − −r 4

6.4 3.2 0 0 − 1.6  − − r1

4.8  0 − 3.2 4.8 0 32.8  − −r 2
r3  r3 − * r 2 => A=
− 3.2 0 0 − 2.4 7.2 − 28.8 − −r 3
 
0 0 7.2 4.8 20.4  − −r 4

6.4 3.2 0 0 − 1.6  − − r1

7.2  0 − 3.2 4.8 32.8  − −r 2
r4  r4 − * r 3 => A=
0
− 2.4 0 0 − 2.4 7.2 − 28.8 − −r 3
 
0 0 0 26.4 − 66  − −r 4

Using back substitution

=> x1 = 1.5 , x2 = −3.5 , x3 = 4.5 , and x4 = −2.5 .

* Can you come up with the operation count for solving tri-diagonal system?
Example 9 Diagonalization (or eigenvalue decomposition) of a matrix
Consider the following matrix A:
 2 2 − 4
A =  2 − 1 − 2
 
4 2 − 6

Find matrices U, V, and S such that A can be expressed as

 s1 0 0
A = USV= U  0 s2 0 V
 
 0 0 s3 
Solution: (We will use this opportunity to illustrate the use of Matlab, too).
% Define the matrix:
» A=[2 2 -4; 2 -1 -2;4 2 -6];
% Compute the eigenvalues:
» p=eig(A)
ans =
-1.0000
-2.0000
-2.0000
*(NOTE: we have a repeated eigenvalue of -2)
% Compute three eigenvectors:
» v1=null(A-p(1)*eye(3),'r')
v1 =
1.0000
0.5000
1.0000
» v2=null(A-p(2)*eye(3),'r')
v2 =
-0.5000 1.0000
1.0000 0
0 1.0000
% Form a matrix W based on eigenvectors:
>> W=[v1 v2]
W =
 1.0000 -0.5000 1.0000
0.5000 1.0000 0
1.0000 0 1.0000
% Find the inverse of W:
» C=inv(W)
C =
4 2 -4
-2 0 2
-4 -2 5
-1
% Compute W AW:
» D=inv(W)*A*W
D =
-1 0 0
0 -2 0
0 0 -2
* We note that the diagonal elements in D are the eigenvalues:
1 0 0
D =  0 2 0
 
 0 0 3 
where 3 =2 in this particular case.
* Generalize:
1 0 ... 0 
 0  ... 0 
W AW=D= 
-1 2 
 ... ... ... ... 
 
 0 0 0 n 
Hence, W(W-1 A W) W-1 = W (D) W-1
That is, A = W (D) W-1
* For the present problem, we thus have
  1 − 0.5 1 − 1 0 0  4 2 − 4
A = W (D) W-1 = 0.5 1 1  0 − 2 0  − 2 0 2
   
 1 0 1  0 0 − 2 − 4 − 2 5 

Hence we can identify that S=D, U=W, V= W-1

* This process is called diagonalization or eigenvalue decomposition of a matrix.
* As an example, it is now much easier to compute Ak since
A2= W (D) W-1 W (D) W-1 = W (D2) W-1
A3= W (D) W-1 W (D2) W-1 = W (D3) W-1

k 0 0
...
 1 
… Ak= W Dk W-1 = W
0 k2 ... 0  W-1
 
 ... ... ... ... 
 0 0 0 kn 
which allows one to compute exp(A) in terms of Taylor series expansion.
* This eigenvalue decomposition is also quite useful in solving system of first
order linear ordinary differential equations.
Example 10 Study on ill-conditioned Hilbert matrix
Consider the Hilbert matrix (using 5x5 as an example).
 1 1/ 2 1/ 3 1/ 4 1 / 5
1 / 2 1/ 3 1/ 4 1/ 5 1 / 6

H 5 = 1 / 3 1/ 4 1/ 5 1/ 6 1 / 7
 
1 / 4 1/ 5 1/ 6 1/ 7 1 / 8
1 / 5 1/ 6 1/ 7 1/ 8 1 / 9

1 j + j −2
The element Bij = 1/(i+j-1) comes from the integral Bij = 0 x dx

Investigate its singular behavior using simple Matlab commands.

Solution:
* First, we can generate H5 using Matlab command hilb(5):
>>h5=hilb(5)
h5 =
1.0000 0.5000 0.3333 0.2500 0.2000
0.5000 0.3333 0.2500 0.2000 0.1667
0.3333 0.2500 0.2000 0.1667 0.1429
0.2500 0.2000 0.1667 0.1429 0.1250
0.2000 0.1667 0.1429 0.1250 0.1111
>> format long
% find the determinant
>> det(h5)
ans =
3.749295132517936e-012
(Comment: det(h5) is nearly 0 => h5 is nearly singular)
% find the eigenvalues:
>> p=eig(h5)
ans =
0.00000328792877
0.00030589804015
0.01140749162342
 0.20853421861101
1.56705069109823
% find the condition number:
>> cond(h5)
ans =
4.766072502411117e+005
(Comment: 1.5670506911/0.00000328792877 = 476607.251 = cond(h5);
Such a large value for cond(A) implies a great difficult for the system)
* For n=2 to 7, we can compute the condition numbers and determinants.
The results are shown in the graphs below.

1.E+08 Cond
1.E+07
1.E+06
1.E+05
1.E+04
1.E+03 Cond(H(n))
1.E+02
1.E+01
1.E+00
1 2 3 4 5 6 n
 determinant
1.E+00
1.E-02 det(H(n))
1.E-04
1.E-06
1.E-08
1.E-10
1.E-12
1.E-14
1.E-16
1.E-18
1 2 3 4 5 6 n
Example 11 Study on ill-conditioned Vandermonde matrix
Let x = (x0, x1, x2, x3, x4)T = (2, 3, 4, 5, 6)T
The 5x5 Vandermonde matrix is
1 x0 x02 x03 x04 
 1 3 9 27 81 
  12 4 8 16
1
V5= 1
x1 x12
x 22
x13
x 23
x14 

x 24  =
 1 4 16 64 256 
 1 5 25 125 625 
x2
1 x34 

1
x3
x4
x32
x 42
x33
x 43

x 44 
 1 6 36 216 1296 
Find its determinant, condition number, eigenvalues, and inverse matrix.
Solution: (using Matlab)
...(input the matrix V5 in Matlab here)…

% display the matrix

>>V5
V5 =
1 2 4 8 16
1 3 9 27 81
1 4 16 64 256
1 5 25 125 625
1 6 36 216 1296
>> det(V5)
ans =
288
>> cond(V5)
ans =
1.598135832298317e+005
>> eig(V5)
ans =
1.0e+003 *
1.40985304217870
0.02867963640067
0.00219297656599
 0.00026194547045
0.00001239938420
* Inverse of V5: (the five columns are given below)
15 -40 45 -24 5
-14.25 44.6666667 -54 30 -6.416666667
4.958333333 -17.333333 22.75 -13.333333 2.958333333
-0.75 2.8333333 -4 2.5 -0.583333333
0.041666667 -0.1666667 0.25 -0.1666667 0.041666667

* From the magnitude of the condition number, we can see that it will be
difficult to get accurate solution to the system of equations with V5 as the
|| e ||
coefficient matrix. The relative error in the solution, , can be as high as
|| x ||
-2
O(10 ) on a single precision machine if the constant vector b is such that
|| b || ~ O(1). This is one of the major limitations on using the method of
undetermined coefficient to interpolate the data.
* We can also see that the ratio, max(i)/min(i), of the eigenvalue is
1409.85304217870 / 0.01239938420 = 113703.473

If we use the maximum row sum, cond(v5)= 232213.33.

If we use maximum column sum, cond(v5)=286524.
* As expected, 113703.473 based on the eigenvalue ratio gives the smallest
condition number comparing with the maximum row sum or column sum.
* There are two reasons that max(i)/min(i)=113703.473 does not agree
with cond(v5)=159813.5. First, v5 is a very ill-conditioned matrix as
evidenced by its magnitude based on various norms. In Matlab, condition
number is computed numerically; so it is not surprising that the numerical
result from an ill-conditioned matrix can be in serious error. Second, in
Matlab, cond(A)= max(i)/min(i). However, the i’s are not obtained
from the solution of the characteristic polynomials. Instead, they come
from an procedure called “svd” (singular value decomposition which is
similar to eigenvalue decomposition in concept) using an iterative QR
method. For v5, using “svd(v5)” command gives
ans =
1.0e+003 *
1.48696146371225
0.03274633836924
0.00249352754071
0.00025493521822
0.00000930434969
from which we can see that 1486.96146/0.00930435=159813.6 is the
cond(v5) given by Matlab.
The problem is that 5 =0.00930434969 is in serious error.
The characteristic polynomial for v5 is
P() = 5 -14414 +439843 -1006342 +24468 -288.
Using Newton’s method, we can obtain all 5 real roots of the above p() as:
1409.853042178700
28.67963640066610
2.19297656598769
0.26194547044672
0.01239938420049
which agree with the eigenvalue solution given by Matlab.
Thus, we conclude that
i) min(i) = 0.012399384… NOT 0.00930435
ii) cond(v5)= 113703.473, NOT 159813.583
iii) svd command in Matlab can give large error. Caution!
Example 12 Jacobi method for system of equations (on Excel sheet)
Start from the initial guess of x(0) = (0, 0, 0)T, implement the following
Jacobi iterative procedure on the Excel spreadsheet.

x1n +1 = 1.8333 + 0.3333 x2n - 0.1667 x3n

x2n +1 = 0.7143 + 0.2857 x1n - 0.2857 x3n

x3n +1 = 0.2 + 0.2 x1n n

+ 0.4 x2
Solution:
n x1 x2 x3
0 0 0 0
=1.8333+0.3333*D2 =0.7143+0.2857*C2 =0.2+0.2*C2
1 -0.1667*E2 -0.2857*E2 +0.4*D2
=1.8333+0.3333*D3 =0.7143+0.2857*C3 =0.2+0.2*C3
2 -0.1667*E3 -0.2857*E3 +0.4*D3
=1.8333+0.3333*D4 =0.7143+0.2857*C4 =0.2+0.2*C4
3 -0.1667*E4 -0.2857*E4 +0.4*D4

Results:
n x1 x2 x3 sum(e(i))
0 0 0 0
1 1.8333 0.7143 0.2 2.74760
2 2.03803619 1.18093381 0.85238 1.32375
3 2.084813493 1.053041973 1.0799808 0.40227
4 2.004246097 1.001380711 1.0381795 0.17403
5 1.99399567 0.99030523 1.0014015 0.05810
6 1.996435103 0.997884153 0.9949212 0.01650
7 2.00004142 1.000432514 0.9984407 0.00967
8 2.000304095 1.000457331 1.0001813 0.00203
 9 2.000022207 1.000035086 1.0002438 0.00077
10 1.999871061 0.999936705 1.0000185 0.00047
11 1.999875824 0.999957884 0.9999489 9.6E-05
12 1.999894482 0.999979124 0.9999583 4.9E-05
13 1.99989999 0.999981762 0.9999705 2E-05
n n −1 n n −1 n n −1
sum(e(i))=| | x1 − x1 | + | x2 − x2 | + | x3 − x3 |
Computation stops here since the roundoff error in coefficients are around 5E-5.

1.E+01
sum |e(i)|
1.E+00
1.E-01
1.E-02
1.E-03
1.E-04
1.E-05
1.E-06
1.E-07
1.E-08
1.E-09
0 5 10 15 20 n
Example 13 Gauss-Seidel method for system of equations (on Excel sheet)
Start from the initial guess of x(0) = (0, 0, 0)T, implementing the following
Jacobi iterative procedure on the Excel spreadsheet.

x1n +1 = 1.8333 + 0.3333 x2n - 0.1667 x3n

x2n +1 = 0.7143 + 0.2857 x1n +1 - 0.2857 x3n

x3n +1 = 0.2 + 0.2 x1n +1 + 0.4 x2n +1

Solution:
n x1 x2 x3
0 0 0 0
=1.8333+0.3333*D2 =0.7143+0.2857*C3
1 -0.1667*E2 -0.2857*E2 =0.2+0.2*C3+0.4*D3
=1.8333+0.3333*D3 =0.7143+0.2857*C4
2 -0.1667*E3 -0.2857*E3 =0.2+0.2*C4+0.4*D4
=1.8333+0.3333*D4 =0.7143+0.2857*C5
3 -0.1667*E4 -0.2857*E4 =0.2+0.2*C5+0.4*D5

Results:
n x1 x2 x3 sum |e(i)|
0 0 0 0
1 1.8333 1.23807381 1.0618895 4.13326
2 2.068933017 1.002012326 1.0145915 0.51899
3 1.998138300 0.995299311 0.9977474 0.09435
4 1.998708771 1.000274668 0.9998516 0.00765
5 2.000016282 1.000047043 1.0000221 0.00171
6 1.999912000 0.999968552 0.9999698 0.00024
7 1.999894549 0.999978495 0.9999703 2.8E-05
 Comparison of convergence behavior between Jacobi method and
Gauss-Seidel method:

1.E+01 sum |e(i)| Jacobi

1.E+00 G-S
1.E-01
1.E-02
1.E-03
1.E-04
1.E-05
1.E-06
1.E-07
1.E-08
1.E-09
0 5 10 15 20 n

Comments: G-S iteration converges clearly better than Jacobi.

Example 14 Over-relaxation method (on Excel sheet)
Start from the initial guess of x(0) = (0, 0, 0,0)T, implementing the over-
relaxation method for solving the following a 4x4 system of equations on Excel
spreadsheet.

 -41 -41 11 11   xx21   11 

 1 1 -4 1   x3  =  1 
 1 1 1 -4   x4   1 
Solution:
* We first rewrite the system as
⎯ ⎯
xn+1 n n n
= ( x2 + x3 + x4 -1)/4; x1n +1 = xn1 +  (xn+1 - xn1 ) =>
1 1

⎯ n +1 ⎯
xn+1 = ( x1
n n
+ x3 + x4 -1)/4; x2n +1 = xn2 +  (xn+1 - xn2) =>
2 2

⎯ ⎯
xn+1 = ( x1
n +1 n +1
+ x2
n
+ x4 -1)/4; x3n +1 = xn +  (xn+1 - xn) =>
3 3 3 3

⎯ n +1 n +1 n +1 ⎯
xn+1
= ( x1 + x2 + x3 -1)/4; x4n +1 =
n
x4 +  (x n+1 n
- x4)
4 4

Excel Formula:
n x1 x2 x3 x4
0 0 0 0 0
=C2+ww*((-1+D2 =D2+ww*((-1+E2 =E2+ww*((-1+F2 =F2+ww*((-1+C3
1 +E2+F2)/4-C2) +F2+C3)/4-D2) +C3+D3)/4-E2) +D3+E3)/4-F2)
=C3+ww*((-1+D3 =D3+ww*((-1+E3 =E3+ww*((-1+F3 =F3+ww*((-1+C4
2 +E3+F3)/4-C3) +F3+C4)/4-D3) +C4+D4)/4-E3) +D4+E4)/4-F3)
=C4+ww*((-1+D4 =D4+ww*((-1+E4 =E4+ww*((-1+F4 =F4+ww*((-1+C5
3 +E4+F4)/4-C4) +F4+C5)/4-D4) +C5+D5)/4-E4) +D5+E5)/4-F4)
=C5+ww*((-1+D5 =D5+ww*((-1+E5 =E5+ww*((-1+F5 =F5+ww*((-1+C6
4 +E5+F5)/4-C5) +F5+C6)/4-D5) +C6+D6)/4-E5) +D6+E6)/4-F5)
Screen shot: (ww=omega is defined on cell A4; a value of 1.25 is specified)

Results of iteration (=ww=1.25):

n x1 x2 x3 x4 sum |e(i)|
0 0 0 0 0
1 -0.3125 -0.4101563 -0.5383301 -0.7065582 1.968E+00
2 -0.7515764 -0.8338562 -0.8941646 -0.9107346 1.423E+00
3 -0.9492169 -0.9646972 -0.9716616 -0.9865588 4.818E-01
4 -0.9886075 -0.9922094 -0.9968895 -0.9963936 1.020E-01
5 -0.9983145 -0.9993219 -0.9989120 -0.9998230 2.227E-02
6 -0.9998141 -0.9997161 -1.0000699 -0.9999193 3.148E-03
7 -0.9999544 -1.0000533 -0.9999597 -1.0000100 6.783E-04
8 -1.0000186 -0.9999830 -1.0000137 -1.0000023 1.962E-04
9 -0.9999950 -1.0000077 -0.9999981 -0.9999997 6.640E-05
10 -1.0000030 -0.9999983 -1.0000008 -1.0000007 2.095E-05
11 -0.9999992 -1.0000006 -1 -0.9999998 7.827E-06
n n −1 n n −1 n n −1 n n −1
sum(e(i))=| | x1 − x1 | + | x2 − x2 | + | x3 − x3 | + | x4 − x4 |
1.E+01 sum |e(i)|
1.E+00
1.E-01
1.E-02
1.E-03
1.E-04
1.E-05
1.E-06
 =1.0
1.E-07  =1.1
1.E-08  =1.2
1.E-09  =1.25
1.E-10  =1.3
1.E-11  =1.4
1.E-12
0 5 10 15 20 n

This figure shows that:

• the optimal relaxation factor is around =1.25.
• There is a big difference in convergence between =1.25 and =1.0
which is Gauss-Seidel iteration.