MA 214 Lecture 6

Introduction to Numerical Analysis
( Linear Systems: Modified Gaussian elimination method, LU-Factorization
Methods)
MA 214, Spring 2023-24.
MA 214 - NA Spring 2023-24 1 / 39

Modified Gaussian elimination method with partial pivoting
Step 1: Define s1 = max { |a11 |, |a21 |, |a31 | } .
MA 214 - NA Spring 2023-24 2 / 39

Note that s1 ̸= 0 : why?
MA 214 - NA Spring 2023-24 2 / 39

Let k be the least number such that s1 = |ak1 |.
MA 214 - NA Spring 2023-24 2 / 39

Interchange the first row and the kth row.
MA 214 - NA Spring 2023-24 2 / 39

Interchange the first row and the kth row. Rewritten system is
(1) (1) (1) (1)
a11 x1 + a12 x2 + a13 x3 = b1
(1) (1) (1) (1)
a21 x1 + a22 x2 + a23 x3 = b2
(1) (1) (1) (1)
a31 x1 + a32 x2 + a33 x3 = b3 .
MA 214 - NA Spring 2023-24 2 / 39

Interchange the first row and the kth row. Rewritten system is
(1) (1) (1) (1)
a11 x1 + a12 x2 + a13 x3 = b1
(1) (1) (1) (1)
a21 x1 + a22 x2 + a23 x3 = b2
(1) (1) (1) (1)
a31 x1 + a32 x2 + a33 x3 = b3 .
where (1) (1) (1)
a11 = ak1 , a12 = ak2 , a13 = ak3 ,
(1) (1) (1)
ak1 = a11 , ak2 = a12 , ak3 = a13 ;
(1) (1)
b1 = bk , bk = b1
MA 214 - NA Spring 2023-24 2 / 39
Now eliminate the x1 variable from the second and third equations of
the system
(1) (1) (1) (1)
a11 x1 + a12 x2 + a13 x3 = b1
(1) (1) (1) (1)
a21 x1 + a22 x2 + a23 x3 = b2
(1) (1) (1) (1)
a31 x1 + a32 x2 + a33 x3 = b3 .
MA 214 - NA Spring 2023-24 3 / 39

Now eliminate the x1 variable from the second and third equations of
the system
(1) (1) (1) (1)
a11 x1 + a12 x2 + a13 x3 = b1
(1) (1) (1) (1)
a21 x1 + a22 x2 + a23 x3 = b2
(1) (1) (1) (1)
a31 x1 + a32 x2 + a33 x3 = b3 .
Proceed as in naive Gaussian elimination method.
MA 214 - NA Spring 2023-24 3 / 39

Note the new system is given by
(1) (1) (1) (1)
a11 x1 + a12 x2 + a13 x3 = b1
(2) (2) (2)
0 + a22 x2 + a23 x3 = b2
(2) (2) (2)
0 + a32 x2 + a33 x3 = b3 ,
MA 214 - NA Spring 2023-24 4 / 39

(1) (1) (1) (1)
a11 x1 + a12 x2 + a13 x3 = b1
(2) (2) (2)
0 + a22 x2 + a23 x3 = b2
(2) (2) (2)
0 + a32 x2 + a33 x3 = b3 ,
(2) (2)
where the coefficients aij , and bk are given by
(2) (1) (1)
aij = aij − mi1 a1j , i, j = 2, 3
(2) (1) (1)
bi = bi − mi1 b1 , i = 2, 3.
MA 214 - NA Spring 2023-24 4 / 39

{ }
(2) (2)
Step 2: Define s2 = max |a22 |, |a32 | .
MA 214 - NA Spring 2023-24 5 / 39

{ }
(2) (2)
(2)
Let l be the least number such that sl = |al2 |.
MA 214 - NA Spring 2023-24 5 / 39

{ }
(2) (2)
(2)
Interchange the second row and the lth rows. The rewritten system is
(1) (1) (1) (1)
a11 x1 + a12 x2 + a13 x3 = b1
(3) (3) (3)
0 + a22 x2 + a23 x3 = b2
(3) (3) (3)
0 + a32 x2 + a33 x3 = b3 ,
MA 214 - NA Spring 2023-24 5 / 39

{ }
(2) (2)
(2)
Interchange the second row and the lth rows. The rewritten system is
(1) (1) (1) (1)
a11 x1 + a12 x2 + a13 x3 = b1
(3) (3) (3)
0 + a22 x2 + a23 x3 = b2
(3) (3) (3)
0 + a32 x2 + a33 x3 = b3 ,
(3) (3)
where the coefficients aij , and bi are given by
(3) (2) (3) (2) (3) (2) (3) (2)
a22 = al2 , a23 = al3 , al2 = a22 , al3 = a23 ,
(3) (2) (3) (2)
b2 = bl , bl = b2
MA 214 - NA Spring 2023-24 5 / 39
(1) (1) (1) (1)
a11 x1 + a12 x2 + a13 x3 = b1
(3) (3) (3)
0 + a22 x2 + a23 x3 = b2
(4) (4)
0 + 0 + a33 x3 = b3 ,
MA 214 - NA Spring 2023-24 6 / 39

(1) (1) (1) (1)
a11 x1 + a12 x2 + a13 x3 = b1
(3) (3) (3)
0 + a22 x2 + a23 x3 = b2
(4) (4)
0 + 0 + a33 x3 = b3 ,
(4) (4)
where the coefficient a33 , and b3 are given by
MA 214 - NA Spring 2023-24 6 / 39

(1) (1) (1) (1)
a11 x1 + a12 x2 + a13 x3 = b1
(3) (3) (3)
0 + a22 x2 + a23 x3 = b2
(4) (4)
0 + 0 + a33 x3 = b3 ,
(4) (4)
(4) (3) (3)
a33 = a33 − m32 a23 ,
(4) (3) (3)
b3 = b3 − m32 b2 .
MA 214 - NA Spring 2023-24 6 / 39

(1) (1) (1) (1)
a11 x1 + a12 x2 + a13 x3 = b1
(3) (3) (3)
0 + a22 x2 + a23 x3 = b2
(4) (4)
0 + 0 + a33 x3 = b3 ,
(4) (4)
(4) (3) (3)
a33 = a33 − m32 a23 ,
(4) (3) (3)
b3 = b3 − m32 b2 .
Note that the variable x2 has been eliminated from the last equation.
MA 214 - NA Spring 2023-24 6 / 39

(1) (1) (1) (1)
a11 x1 + a12 x2 + a13 x3 = b1
(3) (3) (3)
0 + a22 x2 + a23 x3 = b2
(4) (4)
0 + 0 + a33 x3 = b3 ,
(4) (4)
(4) (3) (3)
a33 = a33 − m32 a23 ,
(4) (3) (3)
b3 = b3 − m32 b2 .
Note that the variable x2 has been eliminated from the last equation.
This phase is called Forward elimination phase with partial
MA 214 - NA Spring 2023-24 6 / 39
The reduced system is

(1) (1) (1) (1)
a11 x1 + a12 x2 + a13 x3 = b1
(3) (3) (3)
0 + a22 x2 + a23 x3 = b2
(4) (4)
0 + 0 + a33 x3 = b3 ,
MA 214 - NA Spring 2023-24 7 / 39

The reduced system is

(1) (1) (1) (1)
a11 x1 + a12 x2 + a13 x3 = b1
(3) (3) (3)
0 + a22 x2 + a23 x3 = b2
(4) (4)
0 + 0 + a33 x3 = b3 ,
Now do the Backward substitution phase.
MA 214 - NA Spring 2023-24 7 / 39

Recall the system
6x1 + 2x2 + 2x3 = −2
2 1
2x1 + x2 + x3 = 1
3 3
x1 + 2x2 − x3 = 0.
was solved using Gaussian Elimination method with four digit rounding
arithmetic.
MA 214 - NA Spring 2023-24 8 / 39

Recall the system
6x1 + 2x2 + 2x3 = −2
2 1
2x1 + x2 + x3 = 1
3 3
x1 + 2x2 − x3 = 0.
was solved using Gaussian Elimination method with four digit rounding
arithmetic. Recall the reduced system after Step 1 (in naive Gauss
elimination) was
6.000x1 + 2.000x2 + 2.000x3 = −2.000
0.000x1 + 0.0001x2 − 0.3333x3 = 1.667
0.000x1 + 1.667x2 − 1.333x3 = 0.3334
MA 214 - NA Spring 2023-24 8 / 39
The final system is (with m32 = 0.00005999)
6.000x1 + 2.000x2 + 2.000x3 = −2.000
0.000x1 + 1.667x2 − 1.333x3 = 0.3334
0.000x1 + 0.0000x2 − 0.3332x3 = 1.667
MA 214 - NA Spring 2023-24 9 / 39

6.000x1 + 2.000x2 + 2.000x3 = −2.000
0.000x1 + 1.667x2 − 1.333x3 = 0.3334
0.000x1 + 0.0000x2 − 0.3332x3 = 1.667
with back substitution, we obtain the approximate solution as
x1 = 2.602, x2 = −3.801 and x3 = −5.003.
MA 214 - NA Spring 2023-24 9 / 39

6.000x1 + 2.000x2 + 2.000x3 = −2.000
0.000x1 + 1.667x2 − 1.333x3 = 0.3334
0.000x1 + 0.0000x2 − 0.3332x3 = 1.667
x1 = 2.602, x2 = −3.801 and x3 = −5.003.
Recall, the solution obtained without pivoting was
x1 = 1.335, x2 = 0 and x3 = −5.003,
MA 214 - NA Spring 2023-24 9 / 39

6.000x1 + 2.000x2 + 2.000x3 = −2.000
0.000x1 + 1.667x2 − 1.333x3 = 0.3334
0.000x1 + 0.0000x2 − 0.3332x3 = 1.667
x1 = 2.602, x2 = −3.801 and x3 = −5.003.
Recall, the solution obtained without pivoting was
x1 = 1.335, x2 = 0 and x3 = −5.003,
whereas the actual solution is
x1 = 2.6, x2 = −3.8 and x3 = −5.
MA 214 - NA Spring 2023-24 9 / 39
Some Easily Solvable Systems
Here are some easily solvable system of linear equations :
MA 214 - NA Spring 2023-24 10 / 39


1 Invertible Diagonal Matrices
MA 214 - NA Spring 2023-24 10 / 39


2 Invertible Lower Triangular Matrices
MA 214 - NA Spring 2023-24 10 / 39


2 Invertible Lower Triangular Matrices
3 Invertible Upper Triangular Matrices
MA 214 - NA Spring 2023-24 10 / 39

Some Easily Solvable Systems: Invertible Diagonal Matrices
Invertible Diagonal Matrices:
MA 214 - NA Spring 2023-24 11 / 39

These matrices look like
 
d1 0 0 ··· 0
 0 d2 0 ··· 0
. . .
 .. .. · · · · · · .. 
0 0 0 · · · dn
MA 214 - NA Spring 2023-24 11 / 39

 
d1 0 0 ··· 0
 0 d2 0 ··· 0
. . .
 .. .. · · · · · · .. 
0 0 0 · · · dn
and di ̸= 0 for each i = 1, 2, · · · , n.
MA 214 - NA Spring 2023-24 11 / 39

 
d1 0 0 ··· 0
 0 d2 0 ··· 0
. . .
 .. .. · · · · · · .. 
0 0 0 · · · dn
and di ̸= 0 for each i = 1, 2, · · · , n.
In this case, the solution x = (x1 , x2 , · · · , xn )T is given by
( )T
b1 b2 bn
x= , ,··· , .
d1 d2 dn
MA 214 - NA Spring 2023-24 11 / 39
Easily Solvable Systems: Invertible Lower Triangular Matrices
Invertible Lower Triangular Matrices
MA 214 - NA Spring 2023-24 12 / 39

Invertible Lower Triangular Matrices

 
l11 0 0 ··· 0
l21 l22 0 ··· 0 
. . .. 
 .. .. · · · ··· . 
ln1 ln2 ln3 · · · lnn
and lii ̸= 0 for each i = 1, 2, · · · , n.

MA 214 - NA Spring 2023-24 12 / 39
The linear system takes the form

    
l11 0 0 · · · 0 x1 b1
l21 l22 0 · · · 0  x2  b2 
. .    
 .. .. · · · · · · ...   ...  =  ... 
ln1 ln2 ln3 · · · lnn xn bn
MA 214 - NA Spring 2023-24 13 / 39

    
The linear system takes the form l11 0 0 ··· 0 x1 b1
l21 l22 0 ··· 0  x2  b2 
    
    
 .. .
.
.
.   ..  =  .. 
 . . ··· ··· .  .  .
ln1 ln2 ln3 ··· lnn xn bn
From the first equation, we obtain x1 as

b1
x1 = .
l11
MA 214 - NA Spring 2023-24 14 / 39

    
l21 l22 0 ··· 0  x2  b2 
    
    
 .. .
.
.
.   ..  =  .. 
 . . ··· ··· .  .  .

b1
x1 = .
l11
Put x1 in the second equation
b1
b2 − l21
l11
⇒ x2 = ,
l22
MA 214 - NA Spring 2023-24 14 / 39

    
l21 l22 0 ··· 0  x2  b2 
    
    
 .. .
.
.
.   ..  =  .. 
 . . ··· ··· .  .  .

b1
x1 = .
l11
b1
b2 − l21
l11
⇒ x2 = , and so on.
l22
MA 214 - NA Spring 2023-24 14 / 39

    
l21 l22 0 ··· 0  x2  b2 
    
    
 .. .
.
.
.   ..  =  .. 
 . . ··· ··· .  .  .

b1
x1 = .
l11
b1
b2 − l21
l11
⇒ x2 = , and so on.
l22
This procedure is called the forward substitution.

MA 214 - NA Spring 2023-24 14 / 39
Easily Solvable Systems: Invertible Upper Triangular Matrices
Invertible Upper Triangular Matrices
MA 214 - NA Spring 2023-24 15 / 39

Invertible Upper Triangular Matrices

 
u11 u12 u13 · · · u1n
 0 u22 u23 · · · u2n 
 . .. .. 
 .. . ··· ··· . 
0 0 0 · · · unn
and uii ̸= 0 for each i = 1, 2, · · · , n.

MA 214 - NA Spring 2023-24 15 / 39

 
u11 u12 u13 · · · u1n−1 u1n    
 0 u22 u23 · · · u2n−1 x1 b1
 . u2n 
 x2  b2 
 .. .. ..  .  =  . 
 . ··· ··· .   ..   .. 
 0 0 0 · · · un−1n−1 unn 
xn bn
0 0 0 ··· 0 unn
MA 214 - NA Spring 2023-24 16 / 39


 
u11 u12 u13 · · · u1n−1 u1n    
 0 u22 u23 · · · u2n−1 x1 b1
 . u2n 
 x2  b2 
 .. .. ..  .  =  . 
 . ··· ··· .   ..   .. 
 0 0 0 · · · un−1n−1 unn 
xn bn
0 0 0 ··· 0 unn
From the last equation, we get

bn
xn = .
unn
MA 214 - NA Spring 2023-24 16 / 39

 
The linear system takes the form u11 u12 u13 ··· u1n−1 u1n    
x1 b1
 0 u22 u23 ··· u2n−1 u2n 
  x2   b2 
 . . .    
 . . .  .  =  . 
 . . ··· ··· .  .  .
  .  .
 0 0 0 ··· un−1n−1 unn 
xn bn
0 0 0 ··· 0 unn

bn
xn = .
unn
Put xn in the penultimate equation to get
bn
bn−1 − un−1,n
unn
xn−1 = .
un−1,n−1
MA 214 - NA Spring 2023-24 17 / 39

 
The linear system takes the form u11 u12 u13 ··· u1n−1 u1n    
x1 b1
 0 u22 u23 ··· u2n−1 u2n 
  x2   b2 
 . . .    
 . . .  .  =  . 
 . . ··· ··· .  .  .
  .  .
 0 0 0 ··· un−1n−1 unn 
xn bn
0 0 0 ··· 0 unn

bn
xn = .
unn
Put xn in the penultimate equation to get
bn
bn−1 − un−1,n
unn
xn−1 = .
un−1,n−1
This procedure is called the backward substitution.
MA 214 - NA Spring 2023-24 17 / 39
LU Factorization
In general, if we can write
MA 214 - NA Spring 2023-24 18 / 39

LU Factorization

A = LU.
MA 214 - NA Spring 2023-24 18 / 39

LU Factorization

A = LU.
( )
Ax = b ⇒ L Ux = b.
MA 214 - NA Spring 2023-24 18 / 39

LU Factorization

A = LU.
( )
Ax = b ⇒ L Ux = b.
Use forward substitution for
Lz = b.
MA 214 - NA Spring 2023-24 18 / 39

LU Factorization

A = LU.
( )
Ax = b ⇒ L Ux = b.
Use forward substitution for
Lz = b.
Then use backward substitution to solve
Ux = z.
MA 214 - NA Spring 2023-24 18 / 39

LU Factorization (contd.)
Gaussian elimination method can also be used to get

 
a11 a12 a13
 (2) (2) 
U =  0 a22 a23  .
(3)
0 0 a33
and
MA 214 - NA Spring 2023-24 19 / 39


 
a11 a12 a13
 (2) (2) 
U =  0 a22 a23  .
(3)
0 0 a33
and  
1 0 0
L = m21 1 0 .
m31 m32 1
MA 214 - NA Spring 2023-24 19 / 39


 
a11 a12 a13
 (2) (2) 
U =  0 a22 a23  .
(3)
0 0 a33
and  
1 0 0
L = m21 1 0 .
m31 m32 1
Check! A = LU.
MA 214 - NA Spring 2023-24 19 / 39
Is LU factorization UNIQUE?
MA 214 - NA Spring 2023-24 20 / 39

Clearly if a matrix has an LU decomposition, the matrices L and U are

not unique.
MA 214 - NA Spring 2023-24 20 / 39


not unique.
For any invertible diagonal matrix D
A = LU
MA 214 - NA Spring 2023-24 20 / 39


not unique.
For any invertible diagonal matrix D
A = LU = (LD)(D−1 U) = L̃Ũ.
MA 214 - NA Spring 2023-24 20 / 39

There are three special LU decompositions that we will discuss now.
MA 214 - NA Spring 2023-24 21 / 39

1 Doolittle’s factorization
MA 214 - NA Spring 2023-24 21 / 39

2 Crout’s factorization
MA 214 - NA Spring 2023-24 21 / 39

2 Crout’s factorization
3 Cholesky’s factorization
MA 214 - NA Spring 2023-24 21 / 39

LU Factorization: Doolittle’s factorization
Definition (Doolittle’s factorization)

A matrix A is said to have a Doolittle’s factorization if there exists a
lower triangular matrix L with all diagonal elements as 1, and an upper
triangular matrix U such that
A = LU.
MA 214 - NA Spring 2023-24 22 / 39

Definition (Doolittle’s factorization)

A matrix A is said to have a Doolittle’s factorization if there exists a
lower triangular matrix L with all diagonal elements as 1, and an upper
triangular matrix U such that
A = LU.
Theorem
Let A be an invertible n × n matrix such that all of its n − 1 leading
principal minors are non-zero. Then A has an LU-decomposition where L
is a unit lower triangular matrix and U is an upper triangular matrix.
That is, A has a Doolittle’s factorization.
MA 214 - NA Spring 2023-24 22 / 39
Definition (Minors of a Matrix)

1 Let A be an n × n matrix. A sub-matrix of order k (< n) of the
matrix A is a k × k matrix obtained by removing the n − k rows and
n − k columns from A.
The determinant of such a sub-matrix of order k of A is called

minor of order k of the matrix A.
MA 214 - NA Spring 2023-24 23 / 39

Definition (Principal Minors of a Matrix)

1 The principal sub-matrix of order k of the matrix A is obtained
by removing n − k rows and the same n − k columns from A.
The determinant of the leading principal sub-matrix of order k of A
is called a principal minor of order k of the matrix A.
2 A principal sub-matrix and the corresponding principal minor are
called leading principal sub-matrix and leading principal minor
respectively, if remove the last n − k rows and columns .
MA 214 - NA Spring 2023-24 24 / 39

LU Factorization: Doolittle’s factorization (contd.)
Computing the Doolittle’s factorization for a 3 × 3 matrix A
MA 214 - NA Spring 2023-24 25 / 39

Write A = LU as
    
a11 a12 a13 1 0 0 u11 u12 u13
a21 a22 a23  = l21 1 0  0 u22 u23 
a31 a32 a33 l31 l32 1 0 0 u33
MA 214 - NA Spring 2023-24 25 / 39

Write A = LU as
    
a11 a12 a13 1 0 0 u11 u12 u13
a21 a22 a23  = l21 1 0  0 u22 u23 
a31 a32 a33 l31 l32 1 0 0 u33
The right hand matrix multiplication yields
a11 = u11 , a12 = u12 , a13 = u13 ,
a21 = l21 u11 , a31 = l31 u11 .
MA 214 - NA Spring 2023-24 25 / 39

Write A = LU as
    
a11 a12 a13 1 0 0 u11 u12 u13
a21 a22 a23  = l21 1 0  0 u22 u23 
a31 a32 a33 l31 l32 1 0 0 u33
The right hand matrix multiplication yields
a11 = u11 , a12 = u12 , a13 = u13 ,
a21 = l21 u11 , a31 = l31 u11 .
These gives first column of L and the first row of U.
MA 214 - NA Spring 2023-24 25 / 39
    
a11 a12 a13 1 0 0 u11 u12 u13
a21 a22 a23  = l21 1 0  0 u22 u23 
a31 a32 a33 l31 l32 1 0 0 u33
Multiply row 2 of L times columns 2 and 3 of U, to obtain

a22 = l21 u12 + u22 , a23 = l21 u13 + u23 .
MA 214 - NA Spring 2023-24 26 / 39

    
a11 a12 a13 1 0 0 u11 u12 u13
a21 a22 a23  = l21 1 0  0 u22 u23 
a31 a32 a33 l31 l32 1 0 0 u33
Multiply row 2 of L times columns 2 and 3 of U, to obtain

a22 = l21 u12 + u22 , a23 = l21 u13 + u23 .
These can be solved for u22 and u23 .
MA 214 - NA Spring 2023-24 26 / 39

    
a11 a12 a13 1 0 0 u11 u12 u13
a21 a22 a23  = l21 1 0  0 u22 u23 
a31 a32 a33 l31 l32 1 0 0 u33
Multiply row 3 of L with columns 2 and 3 of U to obtain

l31 u12 + l32 u22 = a32 , l31 u13 + l32 u23 + u33 = a33 .
MA 214 - NA Spring 2023-24 27 / 39

    
a11 a12 a13 1 0 0 u11 u12 u13
a21 a22 a23  = l21 1 0  0 u22 u23 
a31 a32 a33 l31 l32 1 0 0 u33

l31 u12 + l32 u22 = a32 , l31 u13 + l32 u23 + u33 = a33 .
These equations yield values for l32 and u33 , completing the construction
of L and U.
MA 214 - NA Spring 2023-24 27 / 39

    
a11 a12 a13 1 0 0 u11 u12 u13
a21 a22 a23  = l21 1 0  0 u22 u23 
a31 a32 a33 l31 l32 1 0 0 u33

l31 u12 + l32 u22 = a32 , l31 u13 + l32 u23 + u33 = a33 .
These equations yield values for l32 and u33 , completing the construction
of L and U.
In this process, we must have u11 ̸= 0, u22 ̸= 0 in order to solve for L.
MA 214 - NA Spring 2023-24 27 / 39
Example:
Consider the matrix A given by
 
1 1 −1
 1 2 −2 .
−2 1 1
MA 214 - NA Spring 2023-24 28 / 39

Example:     
1 1 −1 1 0 0 u11 u12 u13
 1 2 −2 = l21 1 0  0 u22 u23 
−2 1 1 l31 l32 1 0 0 u33
MA 214 - NA Spring 2023-24 29 / 39

Example:     
1 1 −1 1 0 0 u11 u12 u13
 1 2 −2 = l21 1 0  0 u22 u23 
−2 1 1 l31 l32 1 0 0 u33
u11 = 1, u12 = 1, u13 = −1,
MA 214 - NA Spring 2023-24 29 / 39

Example:     
1 1 −1 1 0 0 u11 u12 u13
 1 2 −2 = l21 1 0  0 u22 u23 
−2 1 1 l31 l32 1 0 0 u33
u11 = 1, u12 = 1, u13 = −1,
a21 a31
l21 = = 1, l31 = = −2
u11 u11
u22 = a22 − l21 u12 = 2 − 1 × 1 = 1
MA 214 - NA Spring 2023-24 29 / 39

Example:     
1 1 −1 1 0 0 u11 u12 u13
 1 2 −2 = l21 1 0  0 u22 u23 
−2 1 1 l31 l32 1 0 0 u33
u11 = 1, u12 = 1, u13 = −1,
a21 a31
l21 = = 1, l31 = = −2
u11 u11
u22 = a22 − l21 u12 = 2 − 1 × 1 = 1
u23 = a23 − l21 u13 = −2 − 1 × (−1) = −1
MA 214 - NA Spring 2023-24 29 / 39

Example:     
1 1 −1 1 0 0 u11 u12 u13
 1 2 −2 = l21 1 0  0 u22 u23 
−2 1 1 l31 l32 1 0 0 u33
u11 = 1, u12 = 1, u13 = −1,
a21 a31
l21 = = 1, l31 = = −2
u11 u11
u22 = a22 − l21 u12 = 2 − 1 × 1 = 1
u23 = a23 − l21 u13 = −2 − 1 × (−1) = −1
l32 = (a32 − l31 u12 )/u22 = (1 − (−2) × 1)/1 = 3
MA 214 - NA Spring 2023-24 29 / 39

Example:     
1 1 −1 1 0 0 u11 u12 u13
 1 2 −2 = l21 1 0  0 u22 u23 
−2 1 1 l31 l32 1 0 0 u33
u11 = 1, u12 = 1, u13 = −1,
a21 a31
l21 = = 1, l31 = = −2
u11 u11
u22 = a22 − l21 u12 = 2 − 1 × 1 = 1
u23 = a23 − l21 u13 = −2 − 1 × (−1) = −1
l32 = (a32 − l31 u12 )/u22 = (1 − (−2) × 1)/1 = 3
u33 = a33 − l31 u13 − l32 u23 = 1 − (−2) × (−1) − 3 × (−1) = 2
MA 214 - NA Spring 2023-24 29 / 39
Thus we obtain the Doolittle’s factorization of A as

  
1 0 0 1 1 −1
A= 1 1 0 0 1 −1
−2 3 1 0 0 2
MA 214 - NA Spring 2023-24 30 / 39


  
1 0 0 1 1 −1
A= 1 1 0 0 1 −1
−2 3 1 0 0 2
Further, taking b = (1, 1, 1)T
MA 214 - NA Spring 2023-24 30 / 39


  
1 0 0 1 1 −1
A= 1 1 0 0 1 −1
−2 3 1 0 0 2

    
1 0 0 z1 1
 1 1 0 z2  = 1
−2 3 1 z3 1
MA 214 - NA Spring 2023-24 30 / 39


  
1 0 0 1 1 −1
A= 1 1 0 0 1 −1
−2 3 1 0 0 2

    
1 0 0 z1 1
 1 1 0 z2  = 1
−2 3 1 z3 1
Forward substitution yields z1 = 1, z2 = 0, z3 = 3.

MA 214 - NA Spring 2023-24 30 / 39
  
1 0 0 1 1 −1
A= 1 1 0 0 1 −1
−2 3 1 0 0 2
Further, taking b = (1, 1, 1)T .
MA 214 - NA Spring 2023-24 31 / 39

  
1 0 0 1 1 −1
A= 1 1 0 0 1 −1
−2 3 1 0 0 2
Further, taking b = (1, 1, 1)T . Keeping the vector z = (1, 0, 3)T as the
right hand side, we now solve the upper triangular system
    
1 1 −1 x1 1
0 1 −1 x2  = 0
0 0 2 x3 3
MA 214 - NA Spring 2023-24 31 / 39

  
1 0 0 1 1 −1
A= 1 1 0 0 1 −1
−2 3 1 0 0 2
Further, taking b = (1, 1, 1)T . Keeping the vector z = (1, 0, 3)T as the
right hand side, we now solve the upper triangular system
    
1 1 −1 x1 1
0 1 −1 x2  = 0
0 0 2 x3 3
Backward substitution yields x1 = 1, x2 = 3/2, x3 = 3/2.
MA 214 - NA Spring 2023-24 31 / 39
LU Factorization: Crout’s factorization
If we ask the upper triangular matrix to have special property in the LU

decomposition, it is known as Crout’s factorization.
MA 214 - NA Spring 2023-24 32 / 39


Definition (Crout’s factorization)
A matrix A is said to have a Crout’s factorization if there exists a
lower triangular matrix L, and an upper triangular matrix U with all
diagonal elements as 1 such that
A = LU.
MA 214 - NA Spring 2023-24 32 / 39


Definition (Crout’s factorization)
A matrix A is said to have a Crout’s factorization if there exists a
lower triangular matrix L, and an upper triangular matrix U with all
diagonal elements as 1 such that
A = LU.
For invertible matrices, one can easily obtain Doolittle’s factorization
from Crout’s factorization and vice versa.
MA 214 - NA Spring 2023-24 32 / 39
—————————————————————————–
MA 214 - NA Spring 2023-24 33 / 39

LU Factorization: Cholesky’s factorization
Definition (Cholesky’s factorization)

A matrix A is said to have a Cholesky’s factorization if there exists a
lower triangular matrix L such that
A = LLT .
MA 214 - NA Spring 2023-24 33 / 39


A = LLT .
Before we prove theorem on Cholesky’s factorization, we recall the
concept of a positive definite matrix.
MA 214 - NA Spring 2023-24 33 / 39


A = LLT .
Before we prove theorem on Cholesky’s factorization, we recall the
concept of a positive definite matrix.
Definition (Positive Definite Matrix)
A symmetric matrix A is said to be positive definite if
xT Ax > 0 for every non-zero vector x.

MA 214 - NA Spring 2023-24 33 / 39
We state some useful results concerning positive definite matrices.
MA 214 - NA Spring 2023-24 34 / 39


Lemma
The following statements concerning a symmetric n × n matrix A are
equivalent.
1 The matrix A is positive definite.
MA 214 - NA Spring 2023-24 34 / 39


Lemma
equivalent.
2 All the principal minors of the matrix A are positive.
MA 214 - NA Spring 2023-24 34 / 39


Lemma
equivalent.
2 All the principal minors of the matrix A are positive.
3 All the eigenvalues of the matrix A are positive.
MA 214 - NA Spring 2023-24 34 / 39

LU Factorization: Cholesky’s factorization (contd.)
Theorem
Let A be an n × n matrix with real entries such that
A is symmetric and
positive definite.
Then A has a unique factorization
A = LLT
where L is a lower triangular matrix with a positive diagonal.
MA 214 - NA Spring 2023-24 35 / 39

Theorem
A is symmetric and
positive definite.
A = LLT
Proof: Is by Induction
MA 214 - NA Spring 2023-24 35 / 39

Theorem
A is symmetric and
positive definite.
A = LLT
n = 1 Trivial
MA 214 - NA Spring 2023-24 35 / 39

Theorem
A is symmetric and
positive definite.
A = LLT
n = 1 Trivial
Assume the theorem for k for some k ∈ N and prove the theorem for
n = k + 1.
MA 214 - NA Spring 2023-24 35 / 39
Let A be a (k + 1) × (k + 1) symmetric positive definite matrix.
MA 214 - NA Spring 2023-24 36 / 39

Let ( )
Ak a
A= ,
aT a(k+1)(k+1)
where
MA 214 - NA Spring 2023-24 36 / 39

Let ( )
Ak a
A= ,
aT a(k+1)(k+1)
where
Ak is the k × k principal sub-matrix of A
a = (a1(k+1) , a2(k+1) , · · · , ak(k+1) )T
Observe: Ak is symmetric and positive definite
MA 214 - NA Spring 2023-24 36 / 39

Let ( )
Ak a
A= ,
aT a(k+1)(k+1)
where
Ak is the k × k principal sub-matrix of A
a = (a1(k+1) , a2(k+1) , · · · , ak(k+1) )T
Observe: Ak is symmetric and positive definite
There exists a unique lower triangular matrix Lk such that
Ak = Lk LTk
MA 214 - NA Spring 2023-24 36 / 39
Let the required lower triangular matrix be of the form
( )
Lk 0
L= ,
lT l(k+1)(k+1)
where the real number l(k+1)(k+1) and the vector
l = (l1(k+1) , l2(k+1) , · · · , lk(k+1) )T are to be chosen such that
MA 214 - NA Spring 2023-24 37 / 39

( )
Lk 0
L= ,
lT l(k+1)(k+1)
( ) ( )( T )
Ak a Lk 0 Lk l
= .
aT a(k+1)(k+1) lT l(k+1)(k+1) 0T l(k+1)(k+1)
MA 214 - NA Spring 2023-24 37 / 39

( )
Lk 0
L= ,
lT l(k+1)(k+1)
( ) ( )( T )
Ak a Lk 0 Lk l
= .
aT a(k+1)(k+1) lT l(k+1)(k+1) 0T l(k+1)(k+1)
⇒ Lk l = a
MA 214 - NA Spring 2023-24 37 / 39

( )
Lk 0
L= ,
lT l(k+1)(k+1)
( ) ( )( T )
Ak a Lk 0 Lk l
= .
aT a(k+1)(k+1) lT l(k+1)(k+1) 0T l(k+1)(k+1)
⇒ Lk l = a
A forward substitution yields the vector l. (Lk is invertible !?)
MA 214 - NA Spring 2023-24 37 / 39
Finally, we have
lT l + l(k+1)(k+1)
2
= a(k+1)(k+1)
and this gives
2
l(k+1)(k+1) = a(k+1)(k+1) − lT l,
Noting that l2(k+1)(k+1) is positive and taking square root, we are done.
MA 214 - NA Spring 2023-24 38 / 39

MA 214 Lecture 6

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

MA 214 Lecture 6

Uploaded by

Copyright:

Available Formats

Introduction to Numerical Analysis

( Linear Systems: Modified Gaussian elimination method, LU-Factorization

MA 214, Spring 2023-24.

MA 214 - NA Spring 2023-24 1 / 39

MA 214 - NA Spring 2023-24 2 / 39

MA 214 - NA Spring 2023-24 2 / 39

MA 214 - NA Spring 2023-24 2 / 39

MA 214 - NA Spring 2023-24 2 / 39

MA 214 - NA Spring 2023-24 2 / 39

MA 214 - NA Spring 2023-24 3 / 39

Proceed as in naive Gaussian elimination method.

MA 214 - NA Spring 2023-24 3 / 39

MA 214 - NA Spring 2023-24 4 / 39

MA 214 - NA Spring 2023-24 4 / 39

MA 214 - NA Spring 2023-24 5 / 39

MA 214 - NA Spring 2023-24 5 / 39

MA 214 - NA Spring 2023-24 5 / 39

MA 214 - NA Spring 2023-24 6 / 39

MA 214 - NA Spring 2023-24 6 / 39

MA 214 - NA Spring 2023-24 6 / 39

MA 214 - NA Spring 2023-24 6 / 39

The reduced system is

MA 214 - NA Spring 2023-24 7 / 39

The reduced system is

Now do the Backward substitution phase.

MA 214 - NA Spring 2023-24 7 / 39

MA 214 - NA Spring 2023-24 8 / 39

MA 214 - NA Spring 2023-24 9 / 39

MA 214 - NA Spring 2023-24 9 / 39

MA 214 - NA Spring 2023-24 9 / 39

Here are some easily solvable system of linear equations :

MA 214 - NA Spring 2023-24 10 / 39

Here are some easily solvable system of linear equations :

MA 214 - NA Spring 2023-24 10 / 39

Here are some easily solvable system of linear equations :

MA 214 - NA Spring 2023-24 10 / 39

Here are some easily solvable system of linear equations :

MA 214 - NA Spring 2023-24 10 / 39

MA 214 - NA Spring 2023-24 11 / 39

MA 214 - NA Spring 2023-24 11 / 39

MA 214 - NA Spring 2023-24 11 / 39

Invertible Lower Triangular Matrices

MA 214 - NA Spring 2023-24 12 / 39

Invertible Lower Triangular Matrices

These matrices look like

and lii ̸= 0 for each i = 1, 2, · · · , n.

The linear system takes the form

MA 214 - NA Spring 2023-24 13 / 39

From the first equation, we obtain x1 as

MA 214 - NA Spring 2023-24 14 / 39

From the first equation, we obtain x1 as

MA 214 - NA Spring 2023-24 14 / 39

From the first equation, we obtain x1 as

MA 214 - NA Spring 2023-24 14 / 39

From the first equation, we obtain x1 as

This procedure is called the forward substitution.

Invertible Upper Triangular Matrices

MA 214 - NA Spring 2023-24 15 / 39

Invertible Upper Triangular Matrices

These matrices look like

and uii ̸= 0 for each i = 1, 2, · · · , n.

The linear system takes the form

MA 214 - NA Spring 2023-24 16 / 39