Part 4 - Linear Systems - Direct Methods

GNE 333 - Engineering Analysis 1
Part 4 – Linear Systems
Direct Methods
Dr. Yves Mansour

yves.mansour@lau.edu.lb
03/27/2024 GNE 333 - Engineering Analysis 1 – Part 4 – Linear Systems - Direct Methods 1
Objectives
 Knowing how to solve small sets of linear equations with Cramer’s rule.
 Understanding how to implement forward elimination and back substitution
as in Gauss elimination.
 Understanding partial and complete pivoting techniques.
 Understanding LU factorization and how it is used to obtain the solution of a
linear system of equations.
 Recognizing that Cholesky’s method provides an efficient way to
decompose a symmetric matrix.
Cramer’s rule
Cramer’s rule is a solution technique that is best suited to small numbers of
equations.
For example, a 3×3 set of linear equations,
[A]
This rule states that each unknown in the system may be expressed as:
| | | | | |
𝑏1 𝑎 12 𝑎 13 𝑎 11 𝑏1 𝑎 13 𝑎11 𝑎1 2 𝑏1
𝑏2 𝑎 22 𝑎 23 𝑎 21 𝑏2 𝑎 23 𝑎 21 𝑎2 2 𝑏2
𝑏3 𝑎 32 𝑎 33 𝑎 31 𝑏3 𝑎 33 𝑎 31 𝑎3 2 𝑏3
𝑥1 = 𝑥 2= 𝑥 3=
det ⁡([ 𝐴]) det ⁡([ 𝐴 ]) det ⁡([ 𝐴])
Proof: https://en.wikipedia.org/wiki/Cramer’s_rule
Example 9.2
Example 9.2
Example 9.2
X1=-25.3 X2=-30 X3= 19.8
Solution of an Upper Triangular System
Many methods transform a linear system into a triangular one:
[ ][ ] [ ]
𝑥1 𝑏1
𝑎 11 𝑎12 𝑎 13 … 𝑎1 𝑛
𝑥2 𝑏2
0 𝑎22 𝑎 23 … 𝑎2 𝑛
⟺ … ¿ … ¿ 0 ¿ 𝑎𝑛𝑛 ¿ 𝑥3 = 𝑏 3
0 0 𝑎3 3 … 𝑎3 𝑛
⋮ ⋮
⋮ ¿⋱ ⋱ ¿ ¿
𝑥𝑛 𝑏𝑛
Back substitution:
𝑏𝑛
𝑅𝑜𝑤 𝑛→ 𝑥𝑛 =
𝑎𝑛 𝑛
𝑏𝑛 −1 −𝑎 𝑛𝑛 𝑥𝑛
𝑅𝑜𝑤 ( 𝑛− 1 ) : 𝑎𝑛 − 1 ,𝑛 − 1 𝑥𝑛 −1 + 𝑎𝑛𝑛 𝑥 𝑛 =𝑏𝑛 − 1 → 𝑥 𝑛 − 1=
𝑎 𝑛 − 1, 𝑛 −1
Naive Gauss Elimination
Gauss elimination method consists in turning a linear system into an upper triangular one.
The solution of the system does not change if we replace by
𝑎 11 𝑥 1 +𝑎 12 𝑥 2+ 𝑎1 3 𝑥3 + … +𝑎 1 𝑛 𝑥 𝑛 =𝑏1 𝑅
( 1)
𝑎2 1 𝑥 1 +𝑎 2 2 𝑥 2+ 𝑎 2 3 𝑥 3 +… +𝑎 2 𝑛 𝑥𝑛 =𝑏 2 𝑅
( 2)
𝑎 3 1 𝑥 1+ 𝑎 3 2 𝑥 2+ 𝑎 3 3 𝑥 3 +… + 𝑎3 𝑛 𝑥𝑛 =𝑏 3
( 𝑅3 )
⋮ ⋮⋮ ⋮ ⋮
𝑎 𝑛 1 𝑥 1 +𝑎 𝑛 2 𝑥 2 +𝑎 𝑛 3 𝑥3 + … +𝑎 𝑛 𝑛 𝑥𝑛 =𝑏𝑛 ¿ ( 𝑅𝑛 )
𝑎 11 𝑥1 +𝑎 12 𝑥 2 +𝑎 1 3 𝑥 3 +… +𝑎 1 𝑛 𝑥 𝑛 =𝑏1
(1) (1) (1 ) ( 1)
( 𝑅1) For = ,
Step (1) 0 +𝑎 𝑥 2+ 𝑎 𝑥 3 + …+𝑎 𝑥 𝑛 =𝑏 For = ,
22
(1)
23
(1)
2𝑛
(1 )
2
( 1)
( 𝑅(12 ) )
0 +𝑎 𝑥2 + 𝑎 𝑥 3 +… +𝑎 𝑥 𝑛 =𝑏
32 33
⋮ ⋮ ⋮⋮ ⋮
3𝑛 3
( 𝑅(13 ) )
(1) (1)
0+ 𝑎𝑛 2 𝑥 2+ 𝑎 𝑛 3 𝑥 3 +… +𝑎 (𝑛1𝑛) 𝑥𝑛 =𝑏𝑛( 1) ¿ ( 𝑅𝑛 )
(1)
Naive Gauss Elimination (cont.)
𝑎 11 𝑥1 +𝑎 12 𝑥 2 +𝑎 1 3 𝑥 3 +… +𝑎 1 𝑛 𝑥 𝑛 =𝑏1
(1) (1) (1 ) ( 1)
( 𝑅1)
0 +𝑎 𝑥 2+ 𝑎 𝑥 3 + …+𝑎 𝑥 𝑛 =𝑏
22
(1)
23
(1)
2𝑛
(1 )
2
( 1)
( 𝑅(12 ) )
0 +𝑎 𝑥2 + 𝑎 𝑥 3 +… +𝑎 𝑥 𝑛 =𝑏
32 33
⋮ ⋮ ⋮⋮ ⋮
3𝑛 3
( 𝑅(13 ) )
(1) (1)
0+ 𝑎𝑛 2 𝑥 2+ 𝑎 𝑛 3 𝑥 3 +… +𝑎 (𝑛1𝑛) 𝑥𝑛 =𝑏𝑛( 1) ¿ ( 𝑅𝑛 )
(1)
𝑎 11 𝑥1 +𝑎 12 𝑥 2 +𝑎 1 3 𝑥 3 +… +𝑎 1 𝑛 𝑥 𝑛 =𝑏1 For = ,
(1 ) ( 1)
( 𝑅1)
Step (2) 0 +𝑎(1) 𝑥 + 𝑎(1)
𝑥 + …+𝑎 𝑥 =𝑏 For = ,
22 2 23
(2)
3 2𝑛 𝑛 2
(2 ) (2)
(𝑅 ) (1)
2
0+ 0+ 𝑎 𝑥 3 + …+𝑎 𝑥 𝑛 =𝑏
33
⋮ ⋮ ⋮⋮ ⋮
3𝑛 3
( 𝑅3 )
(2)
(2)
0+ 0+ 𝑎𝑛 3 𝑥 3 +…+ 𝑎 (𝑛2𝑛) 𝑥 𝑛 =𝑏(𝑛2) ¿ ( 𝑅(2)
𝑛 )
At the end of step we obtain the following upper triangular linear system:
[ ][ ] [ ]
𝑥1 𝑏1 For = , (Step)
𝑎11 𝑎12 𝑎13 … 𝑎1 𝑛 (1)
𝑥 𝑏 For = ,
0 𝑎(1) 𝑎(1) … 𝑎(12 𝑛) (𝑛 −1)
2 2
22 23
… ¿ … ¿ 0 ¿ 𝑎 𝑛𝑛 ¿ 𝑥 3
= 𝑏 (2)
For = ,
0 0 𝑎(2)
33 … 𝑎(2)
3𝑛
3
⋮ ⋮
⋮ ¿⋱ ⋱ ¿ ¿ 𝑥𝑛 (𝑛−1)
𝑏𝑛
Back substitution:
What if  “Naive”
Because the rows of the system matrix and right hand vector receive the same
treatment, Gauss elimination can be applied to the augmented matrix
[ ][]
𝑎1,𝑛+1
𝑎11 𝑎12 𝑎13 … 𝑎1𝑛 For = , (Step)
𝑎2,𝑛+1
0 𝑎22 𝑎23 … 𝑎2𝑛 For = ,
… ¿ … ¿ 0 ¿ 𝑎𝑛𝑛 ¿ 𝑎3,𝑛+1
0 0 𝑎33 … 𝑎3𝑛 𝑏1
⋮ For = ,
⋮ ¿⋱ ⋱ ¿ ¿ 𝑏2
𝑎𝑛 ,𝑛+1
𝑏3
Back substitution:
(𝑛− 1)
⋮
𝑎 𝑛,𝑛+1 𝑏𝑛
𝑥𝑛 = (𝑛− 1)
𝑎
𝑛𝑛
𝑛
𝑎 ( 𝑖 − 1)
𝑖 ,𝑛 +1 − ∑ 𝑎(𝑖
𝑖 𝑗
−1 )
𝑥𝑗
𝑗 =𝑖+ 1
𝑥𝑖 = (𝑖 − 1)
𝑓𝑜𝑟 𝑖=𝑛 − 1, 𝑛 −2 , … , 2 , 1
𝑎 𝑖𝑖
Example 9.3
See “GaussNaive.m”
Example 9.3
Gauss Elimination – Pivoting
If we use naive Gauss elimination to solve the linear system:
or
the normalization of the first row in Step 1 would involve division by = 0.

Solution: Switch line with any line where , is called the “pivot” at step 1.
Since problems may also arise when the pivot element is close, rather than exactly equal,
to zero, line is switched with line where is maximum ( in this case)
So, before each elimination step , we search rows to to determine row with the
maximum pivot , then we switch rows and .
This is called “partial pivoting”.
Gauss Elimination – Pivoting (cont.)
If columns as well as rows are searched for the largest element and then switched, the
procedure is called “complete pivoting”.
Complete pivoting is rarely used because:

1 – Partial pivoting is often sufficient,
2 – Switching columns changes the order of the ’s and, consequently, adds significant
and usually unjustified complexity to the computer program.
See “GaussPivot.m” for partial pivoting.

Notice the use of to determine the index of the largest element of vector x.
Gauss Elimination – Determinant Evaluation
The determinant of an upper triangular matrix is equal to the diagonal product.
The forward-elimination step of Gauss elimination results in an upper triangular system,

consequently:
(1) (2) (𝑛 −1)
𝐷=𝑎 11 𝑎 22 𝑎 33 … 𝑎𝑛𝑛
There is a slight modification to the equation when partial pivoting is employed. For such
cases, the determinant changes sign every time a row is switched, consequently:
( 𝑛 −1 ) 𝑝
𝐷=𝑎 11 𝑎(1)
(2)
22 𝑎 33 … 𝑎𝑛𝑛 ( −1 )
where represents the number of times that rows are switched.
LU Factorization – Overview
Gauss elimination changes both the system matrix and right-hand-side vector .
It becomes inefficient when solving a system with the same coefficients , but with
different right-hand-side vector , which is often encountered in engineering problems.
LU factorization methods separate the time-consuming modification of the matrix from
the manipulations of the right-hand side .
Consider for example a 33 system:
𝐴=𝐿 .𝑈
– Lower triangular matrix with 1s on the diagonal
– Upper triangular matrix
LU Factorization – Overview (cont.)
Solution of factorized system: [ 𝑑𝑖 ]
[ ][ ] [ ] ¿
1 0 0 𝑑1 𝑏1
Solution of a lower
𝑙21 1 0 𝑑2 = 𝑏2
triangular system:
𝑙31 𝑙32 1 𝑑3 𝑏3
= Solution of an upper
triangular system:
Example 10.2-Shortcut method
Example 10.2-Shortcut method
Example 10.2
LU Factorization
problem statement. Use LU factorization to solve the following system.
[ ][ ][ ]
3 − 0.1 − 0.2 1 0 0 𝑢11 𝑢12 𝑢13
0.1 7 − 0.3 = 𝑙 21 1 0 0 𝑢 22 𝑢23
0.3 − 0.2 10 𝑙 31 𝑙32 1 0 0 𝑢33
LU Factorization – General Formulation
[ 10∙∙∙0
𝑙21 1 0 ¿ ¿ ¿
For = 1, 2, …,
][
𝑙32¿1¿∙ ¿∙¿∙ ¿∙¿∙¿∙¿∙¿∙ ¿ 𝑙𝑛−1,𝑛−2¿1¿0¿𝑙𝑛1¿∙¿∙¿∙¿𝑙𝑛,𝑛−1¿1¿ ×
𝑢1 𝑢12 𝑢13 ∙ ∙ 𝑢1𝑛
0 𝑢2 𝑢23 ¿ ¿ ¿ ]
0¿𝑢3 ¿∙ ¿∙¿∙¿¿∙¿∙¿∙¿∙¿∙ ¿ ∙¿𝑢𝑛−1,𝑛−1¿𝑢𝑛−1,𝑛¿0¿∙¿∙¿∙¿0¿𝑢𝑛𝑛¿ =[𝑎𝑖𝑗]
𝑖−1
𝑢𝑖𝑗 =𝑎𝑖𝑗 − ∑ 𝑙𝑖𝑘 𝑢𝑘 𝑗 for = , …, See “myLU.m”.
𝑘=1
𝑖 −1 Matlab has its own LU factorization:
𝑎 𝑗 𝑖 − ∑ 𝑙 𝑗 𝑘 𝑢 𝑘𝑖
𝑘=1 >> [L,U] = lu(A)
𝑙 𝑗𝑖= for = , …,
𝑢𝑖 𝑖
LU Factorization – Concluding Remarks
LU factorization considered so far is called Doolittle decomposition ().
Another variant is Crout decomposition ()
For example, for a 33 system, Crout decomposition is:
𝐴=𝐿 .𝑈
Finally, notice that if for any row of , LU factorization fails.
Ex: if , then (Crout decomposition) or (Doolittle decomposition), or becomes singular!
In this case, partial pivoting is applied: if , we search the rows of to determine row with
the maximum pivot , then we switch rows and .
Cholesky Factorization – Overview
Many engineering applications require solving a symmetric linear system ().
Special solution techniques are available for such systems. They offer computational
advantages: half the storage is needed and half the computation time is required.
One of these techniques is Cholesky factorization (also called Cholesky decomposition).
For example, for a 33 system:
Solution:
{𝑑} lower triangular system
upper triangular system
Cholesky Factorization – General Formulation
[ 𝑢1 0 ∙ ∙ ∙ 0
𝑢12 𝑢2 0 ¿ ¿ ¿
For = 1, 2, …,
][
𝑢23¿𝑢3 ¿∙ ¿∙¿∙ ¿∙¿∙¿∙¿∙¿∙ ¿ ∙¿𝑢𝑛−1,𝑛−1¿0¿𝑢1𝑛¿∙¿∙¿∙¿𝑢𝑛−1,𝑛¿𝑢𝑛𝑛¿ ×
𝑢1 𝑢12 𝑢13 ∙ ∙ 𝑢1𝑛
0 𝑢2 𝑢23 ¿ ¿ ¿ ]
0¿𝑢3 ¿∙ ¿∙¿∙¿¿∙¿∙¿∙¿∙¿∙ ¿ ∙¿𝑢𝑛−1,𝑛−1¿𝑢𝑛−1,𝑛¿0¿∙¿∙¿∙¿0¿𝑢𝑛𝑛¿ =[𝑎𝑖𝑗]
√
𝑖 −1 MATLAB has a built-in function that
𝑢𝑖𝑖 = 𝑎𝑖𝑖 − ∑ 𝑢
2
generates the Cholesky factorization:
𝑘𝑖
𝑘=1 >> U = chol(A)
𝑖 −1
𝑎 𝑖𝑗 − ∑ 𝑢 𝑘𝑖 𝑢 𝑘 𝑗
𝑘=1
𝑢𝑖𝑗 = for = , …,
𝑢𝑖 𝑖
Example 10.6
Analytical solution
https://www.youtube.com/watch?v=CBkdQSu7NnY
See “Example_10_6.m”
Matlab Left Division ()
• When we use left division with the backslash operator, MATLAB invokes a highly
sophisticated algorithm to obtain a solution.
• First, MATLAB checks to see whether is in a format where a solution can be
obtained without full Gauss elimination:
If is sparse, banded, triangular (or easily transformed into triangular form), or
symmetric, the solution is obtained with the efficient techniques that are available
for such systems.
• If none of these simplified solutions are possible, a general triangular factorization
is computed by Gauss elimination with partial pivoting and the solution obtained
with substitution.
• This is why is much more efficient than .

Part 4 - Linear Systems - Direct Methods

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Part 4 - Linear Systems - Direct Methods

Uploaded by

Copyright:

Available Formats

GNE 333 - Engineering Analysis 1

Part 4 – Linear Systems

Dr. Yves Mansour

X1=-25.3 X2=-30 X3= 19.8

the normalization of the first row in Step 1 would involve division by = 0.

Complete pivoting is rarely used because:

See “GaussPivot.m” for partial pivoting.

The forward-elimination step of Gauss elimination results in an upper triangular system,

You might also like