Numerical Methods (CIE-301)

Unit-3
(Systems of Linear Equations)
Dr. Muhammad Majid Gulzar (CIE-KFUPM)
 Contents (Unit-3):
1) Naive Gauss Elimination (Sec 9.2)

2) Pitfalls of Elimination Methods (Sec 9.3)

3) Techniques for Improving Solutions (Sec 9.4)

4) Gauss-Jordan Method (Sec 9.7)

5) Tri-diagonal Systems (Sec 11.1)

Dr. Muhammad Majid Gulzar (CIE-KFUPM)

 Solving Small Numbers of Equations:
1) Graphical Method

Dr. Muhammad Majid Gulzar (CIE-KFUPM)

 Solving Small Numbers of Equations:
2) Determinants and Cramer’s Rule

Dr. Muhammad Majid Gulzar (CIE-KFUPM)

 Solving Small Numbers of Equations:
3) The Elimination of Unknowns

Dr. Muhammad Majid Gulzar (CIE-KFUPM)

Naive Gauss Elimination (Sec 9.2)
 Naive Gauss Elimination:
 The procedure consists of two steps:
 Forward Elimination: The system is reduced to upper triangular form. A sequence of
elementary operations is used.

 Back Substitution: Solve the system starting from the last variable.

 This basic approach can be extended to large sets of equations by developing a systematic scheme or
algorithm to eliminate unknowns and to back-substitute. Gauss elimination is the most basic of these
schemes.
𝑎𝑎11 𝑎𝑎12 𝑎𝑎13 𝑥𝑥1 𝑏𝑏1 𝑎𝑎11 𝑎𝑎12 𝑎𝑎13 𝑥𝑥1 𝑏𝑏1
𝑎𝑎21 𝑎𝑎22 𝑎𝑎23 𝑥𝑥2 = 𝑏𝑏2 ⇒ 0 𝑎𝑎22 ′ 𝑎𝑎23 ′ 𝑥𝑥2 = 𝑏𝑏2 ′
𝑎𝑎31 𝑎𝑎32 𝑎𝑎33 𝑥𝑥3 𝑏𝑏3 0 0 𝑎𝑎33 ′ 𝑥𝑥3 𝑏𝑏3 ′

Dr. Muhammad Majid Gulzar (CIE-KFUPM)

 Naive Gauss Elimination:

2𝑥𝑥1 + 4𝑥𝑥2 − 3𝑥𝑥3 = 3 2 4 −3 𝑥𝑥1 3

2.5𝑥𝑥1 − 𝑥𝑥2 + 3𝑥𝑥3 = 5� ⇔ 2.5 −1 3 𝑥𝑥2 = 5
𝑥𝑥1 − 6𝑥𝑥3 = 7 1 0 −6 𝑥𝑥3 7

Standard form Matrix form

Dr. Muhammad Majid Gulzar (CIE-KFUPM)

 Naive Gauss Elimination (Example):
Use Gauss elimination to solve the given equations.
Note: Carry 6 significant figures during the computation.

0.1
𝑅𝑅2 − 𝑅𝑅
3 1

0.3
𝑅𝑅3 − 𝑅𝑅
3 1

0.190000
𝑅𝑅3 + 𝑅𝑅
7.00333 1

Dr. Muhammad Majid Gulzar (CIE-KFUPM)

 Naive Gauss Elimination (Example):

Verification

Dr. Muhammad Majid Gulzar (CIE-KFUPM)

 Naive Gauss Elimination (Class Activity):
Use Gauss elimination to solve the given equations.

6 −2 2 4 𝑥𝑥1 16
12 −8 6 10 𝑥𝑥2 26
𝑥𝑥3 =
3 −13 9 3 −19
−6 4 1 −18 𝑥𝑥4 −34

Forward Elimination

6 −2 2 4 𝑥𝑥1 16 6 −2 2 4 𝑥𝑥1 16 6 −2 2 4 𝑥𝑥1 16

0 −4 2 2 𝑥𝑥2 −6 0 −4 2 2 𝑥𝑥2 −6 0 −4 2 2 𝑥𝑥2 −6
𝑥𝑥3 = 𝑥𝑥3 = 𝑥𝑥3 =
0 −12 8 1 −27 0 0 2 −5 −9 0 0 2 −5 −9
0 2 3 −14 𝑥𝑥4 −18 0 0 4 −13 𝑥𝑥4 −21 0 0 0 −3 𝑥𝑥4 −3

Eliminate 𝑥𝑥1 from equations 2,3,4 Eliminate 𝑥𝑥2 from equations 3,4 Eliminate 𝑥𝑥3 from equations 4

Dr. Muhammad Majid Gulzar (CIE-KFUPM)

 Naive Gauss Elimination (Class Activity):
Use Gauss elimination to solve the given equations.

6 −2 2 4 𝑥𝑥1 16
0 −4 2 2 𝑥𝑥2 −6
𝑥𝑥3 =
0 0 2 −5 −9
0 0 0 −3 𝑥𝑥4 −3

Backward Substitution

−3 −9 + 5
𝑥𝑥4 = = 𝟏𝟏 𝑥𝑥3 = = −𝟐𝟐
−3 2

−6 − 2(−2) − 2(1) 16 + 2(1) − 2(−2) − 4(1)

𝑥𝑥2 = = 𝟏𝟏 𝑥𝑥1 = = 𝟑𝟑
−4 6

Dr. Muhammad Majid Gulzar (CIE-KFUPM)

Pitfalls of Elimination Methods (Sec 9.3)
 Pitfalls of Elimination Methods:
1) Division by Zero

0 1 𝑥𝑥1 1
=
 The normalization of the first row would involve division by 0. 1 1 𝑥𝑥2 2

10−10 1 𝑥𝑥1 = 1
 Problems also can arise when a coefficient is very close to 0. 1 1 𝑥𝑥2 2

 Solution: Pivoting

Dr. Muhammad Majid Gulzar (CIE-KFUPM)

 Pitfalls of Elimination Methods:
2) Round-Off Errors
 Solution: Use more significant figures

3) Ill-Conditioned Systems
 A small changes in coefficients result in large changes in the solution.
 An alternative interpretation of ill-conditioning is that a wide range of answers can
approximately satisfy the equations.
 As round-off errors can induce small changes in the coefficients, these artificial changes can
lead to large solution errors for ill-conditioned systems.
 Solution: Use more significant figures, Pivoting & Scaling

Dr. Muhammad Majid Gulzar (CIE-KFUPM)

 Pitfalls of Elimination Methods:
 Exceptional Cases

No Solution Infinite Solution Ill-Conditioned Systems

Dr. Muhammad Majid Gulzar (CIE-KFUPM)

 Pitfalls of Elimination Methods:
 How many solutions does a system of equations 𝐴𝐴𝐴𝐴 = 𝐵𝐵 have?

Unique Solution No Solution Infinite Solution

𝐷𝐷𝐷𝐷𝐷𝐷 𝐴𝐴 ≠ 0 𝐷𝐷𝐷𝐷𝐷𝐷 𝐴𝐴 = 0 𝐷𝐷𝐷𝐷𝐷𝐷 𝐴𝐴 = 0

Reduced Matrix has one or multiple zero Reduced Matrix has one or multiple zero
Reduced Matrix has no zero rows
rows corresponding 𝐵𝐵 elements ≠ 0 rows corresponding 𝐵𝐵 elements = 0

1 2 1 1 2 2 1 2 2
𝑋𝑋 = 𝑋𝑋 = 𝑋𝑋 =
3 4 2 2 4 3 2 4 4

1 2 1 1 2 2 1 2 2
𝑋𝑋 = 𝑋𝑋 = 𝑋𝑋 =
0 −2 −1 0 0 −1 0 0 0

Dr. Muhammad Majid Gulzar (CIE-KFUPM)

Techniques for Improving Solution (Sec 9.4)
 Techniques for Improving Solutions:
1) Partial Pivoting
 Before each row is normalized, it is advantageous to determine the largest available
coefficient in the column below the pivot element.

 The rows can then be switched so that the largest element is the pivot element.

2) Scaling
 Scaling is used in minimizing round-off errors for cases where some of the equations in a
system have much larger coefficients than others.

Dr. Muhammad Majid Gulzar (CIE-KFUPM)

 Improving Solutions (Example):
Use Gauss elimination to solve the given equations.
a) Without Partial Pivoting b) With Partial Pivoting
Solution Without Partial Pivoting

1
𝑅𝑅
0.0003 1

𝑅𝑅2 − 𝑅𝑅1

Dr. Muhammad Majid Gulzar (CIE-KFUPM)

 Improving Solutions (Example):
However, due to subtractive cancellation, the result is very sensitive to the number of
significant figures carried in the computation.

Solution Without Partial Pivoting

Note: 𝑥𝑥1 is highly dependent on the number of significant figures because, we are subtracting two almost-equal numbers.

Dr. Muhammad Majid Gulzar (CIE-KFUPM)

 Improving Solutions (Example):
Use Gauss elimination to solve the given equations.
a) Without Partial Pivoting b) With Partial Pivoting
Solution With Partial Pivoting

𝑅𝑅1 ↔ 𝑅𝑅2

0.0003
𝑅𝑅2 − 𝑅𝑅1
2.9997𝑥𝑥2 = 1.9998 1

Dr. Muhammad Majid Gulzar (CIE-KFUPM)

 Improving Solutions (Example):
This case is much less sensitive to the number of significant figures in the computation.

Solution With Partial Pivoting

Note: Thus, a pivot strategy is much more satisfactory.

Dr. Muhammad Majid Gulzar (CIE-KFUPM)

 Techniques for Improving Solutions:
1) Partial Pivoting
 Before each row is normalized, it is advantageous to determine the largest available
coefficient in the column below the pivot element.

 The rows can then be switched so that the largest element is the pivot element.

2) Scaling
 Scaling is used in minimizing round-off errors for cases where some of the equations in a
system have much larger coefficients than others.

Dr. Muhammad Majid Gulzar (CIE-KFUPM)

 Improving Solutions (Example):
Use Gauss elimination to solve the given equations.
a) Without Pivoting & Without Scaling
b) With Pivoting & With Scaling (Such that the maximum coefficient in each row should be 1)
c) With Pivoting & Without Scaling

Dr. Muhammad Majid Gulzar (CIE-KFUPM)

 Improving Solutions (Example):
Use Gauss elimination to solve the given equations.
a) Without Pivoting & Without Scaling

1
𝑅𝑅2 − 𝑅𝑅1
2

Note: Although 𝑥𝑥2 is correct, 𝑥𝑥1 is 100% in error due to round-off.

Dr. Muhammad Majid Gulzar (CIE-KFUPM)

 Improving Solutions (Example):
Use Gauss elimination to solve the given equations.
b) With Pivoting & With Scaling (Such that the maximum coefficient in each row should be 1)
1
𝑅𝑅 Scaling
100000 1

Pivoting

0.00002
𝑅𝑅2 − 𝑅𝑅1
1

Note: Pivoting & Scaling leads to the correct answer.

Dr. Muhammad Majid Gulzar (CIE-KFUPM)

 Improving Solutions (Example):
Use Gauss elimination to solve the given equations.
c) With Pivoting & Without Scaling

Pivoting

2
𝑅𝑅2 − 𝑅𝑅1
1

Note: For this case with Pivoting & without Scaling also leads to the correct answer.

Dr. Muhammad Majid Gulzar (CIE-KFUPM)

 Improving Solutions (Class Activity):
Use Gauss elimination to solve the given equations.
a) Without Pivoting
b) With Pivoting

Dr. Muhammad Majid Gulzar (CIE-KFUPM)

Gauss-Jordan Method (Sec 9.7)
 Gauss-Jordan Method:
 The Gauss-Jordan method is a variation of Gauss elimination.

 The method reduced the general system of equations 𝐴𝐴𝐴𝐴 = 𝐵𝐵 to 𝐼𝐼𝐼𝐼 = 𝐵𝐵 where 𝐼𝐼 is an
identity matrix.

 The elimination step results in an identity matrix rather than a triangular matrix.

 Only Forward Elimination is done and no Back Substitution is needed.

 It has the same problems as Naive Gaussian elimination and can be modified to do
partial scaled pivoting.

Dr. Muhammad Majid Gulzar (CIE-KFUPM)

 Gauss-Jordan Method:

2𝑥𝑥1 + 4𝑥𝑥2 − 3𝑥𝑥3 = 3 2 4 −3 𝑥𝑥1 3

2.5𝑥𝑥1 − 𝑥𝑥2 + 3𝑥𝑥3 = 5� ⇔ 2.5 −1 3 𝑥𝑥2 = 5
𝑥𝑥1 − 6𝑥𝑥3 = 7 1 0 −6 𝑥𝑥3 7

Standard form Matrix form

Dr. Muhammad Majid Gulzar (CIE-KFUPM)

 Gauss-Jordan Method (Example):
Use Gauss-Jordan technique to solve the given equations.
Note: Carry 6 significant figures during the computation.

1
𝑅𝑅
3 1

𝑅𝑅2 − 0.1 × 𝑅𝑅1

𝑅𝑅3 − 0.3 × 𝑅𝑅1

Dr. Muhammad Majid Gulzar (CIE-KFUPM)

 Gauss-Jordan Method (Example):
1
𝑅𝑅
7.00333 2

𝑅𝑅1 + 0.0333333 × 𝑅𝑅2

𝑅𝑅3 + 0.190000 × 𝑅𝑅2

1
𝑅𝑅
10.01200 3

𝑅𝑅1 + 0.0680629 × 𝑅𝑅3

𝑅𝑅2 + 0.0418848 × 𝑅𝑅3

Dr. Muhammad Majid Gulzar (CIE-KFUPM)

Tri-diagonal Systems (Sec 11.1)
 Tri-diagonal Systems:
 The non-zero elements are in the main diagonal, super diagonal and sub diagonal.

 𝑎𝑎𝑖𝑖𝑖𝑖 = 0 𝑖𝑖𝑖𝑖 |𝑖𝑖 − 𝑗𝑗| > 1

 Occur in many applications

 Needs less storage (4𝑛𝑛 − 2 compared to 𝑛𝑛2 + 𝑛𝑛 for the general case)

 Efficiently solved by Gaussian Elimination.

 Forward Elimination 5 1 0 0 0 𝑥𝑥1 𝑏𝑏1
3 4 1 0 0 𝑥𝑥2 𝑏𝑏2
 Backward Substitution
0 2 6 2 0 𝑥𝑥3 = 𝑏𝑏3
0 0 1 4 1 𝑥𝑥4 𝑏𝑏4
0 0 0 1 6 𝑥𝑥5 𝑏𝑏5
Dr. Muhammad Majid Gulzar (CIE-KFUPM)
 Tri-diagonal Systems:
 Elements in the main diagonal, sub diagonal and 𝐵𝐵 need update, so update 𝑎𝑎, 𝑑𝑑 and 𝑏𝑏.

 Elements in the super diagonal are not affected.

𝑑𝑑1 𝑐𝑐1 𝑥𝑥1 𝑏𝑏1 𝑑𝑑1 𝑐𝑐1 𝑥𝑥1 𝑏𝑏1

𝑎𝑎1 𝑑𝑑2 𝑐𝑐2 𝑥𝑥2 𝑏𝑏2 𝑑𝑑2 𝑐𝑐2 𝑥𝑥2 𝑏𝑏2
𝑎𝑎2 𝑑𝑑3 ⋱ 𝑥𝑥3 = 𝑏𝑏3 ⇒ 𝑑𝑑3 ⋱ 𝑥𝑥3 = 𝑏𝑏3
⋱ ⋱ 𝑐𝑐𝑛𝑛−1 ⋮ ⋮ ⋱ 𝑐𝑐𝑛𝑛−1 ⋮ ⋮
𝑎𝑎𝑛𝑛−1 𝑑𝑑𝑛𝑛 𝑥𝑥𝑛𝑛 𝑏𝑏5 𝑑𝑑𝑛𝑛 𝑥𝑥𝑛𝑛 𝑏𝑏5

Dr. Muhammad Majid Gulzar (CIE-KFUPM)

 Tri-diagonal Systems:
𝑑𝑑1 𝑐𝑐1 𝑥𝑥1 𝑏𝑏1 𝑑𝑑1 𝑐𝑐1 𝑥𝑥1 𝑏𝑏1
𝑎𝑎1 𝑑𝑑2 𝑐𝑐2 𝑥𝑥2 𝑏𝑏2 𝑑𝑑2 𝑐𝑐2 𝑥𝑥2 𝑏𝑏2
𝑎𝑎2 𝑑𝑑3 ⋱ 𝑥𝑥3 = 𝑏𝑏3 ⇒ 𝑑𝑑3 ⋱ 𝑥𝑥3 = 𝑏𝑏3
⋱ ⋱ 𝑐𝑐𝑛𝑛−1 ⋮ ⋮ ⋱ 𝑐𝑐𝑛𝑛−1 ⋮ ⋮
𝑎𝑎𝑛𝑛−1 𝑑𝑑𝑛𝑛 𝑥𝑥𝑛𝑛 𝑏𝑏5 𝑑𝑑𝑛𝑛 𝑥𝑥𝑛𝑛 𝑏𝑏5

Forward Elimination Backward Substitution

𝑎𝑎𝑖𝑖−1 𝑏𝑏𝑛𝑛
𝑑𝑑𝑖𝑖 ← 𝑑𝑑𝑖𝑖 − 𝑐𝑐 𝑥𝑥𝑛𝑛 =
𝑑𝑑𝑖𝑖−1 𝑖𝑖−1 𝑑𝑑𝑛𝑛

1
𝑎𝑎𝑖𝑖−1 𝑥𝑥𝑖𝑖 = 𝑏𝑏 − 𝑐𝑐𝑖𝑖 𝑥𝑥𝑖𝑖+1 for 𝑖𝑖 = 𝑛𝑛 − 1, 𝑛𝑛 − 2, . . . , 1
𝑏𝑏𝑖𝑖 ← 𝑏𝑏𝑖𝑖 − 𝑏𝑏 2 ≤ 𝑖𝑖 ≤ 𝑛𝑛 𝑑𝑑𝑖𝑖 𝑖𝑖
𝑑𝑑𝑖𝑖−1 𝑖𝑖−1

Dr. Muhammad Majid Gulzar (CIE-KFUPM)

 Tri-diagonal Systems (Example):
Solve the following tridiagonal system.

5 2 𝑥𝑥1 12 5 1 2 12
1 5 2 𝑥𝑥2 9 5 1 2 9
𝑥𝑥3 = ⇒ 𝐷𝐷 = , 𝐴𝐴 = , 𝐶𝐶 = , 𝐵𝐵 =
1 5 2 8 5 1 2 8
1 5 𝑥𝑥4 6 5 6

Forward Elimination Backward Substitution

𝑎𝑎𝑖𝑖−1 𝑏𝑏𝑛𝑛
𝑑𝑑𝑖𝑖 ← 𝑑𝑑𝑖𝑖 − 𝑐𝑐 𝑥𝑥𝑛𝑛 =
𝑑𝑑𝑖𝑖−1 𝑖𝑖−1 𝑑𝑑𝑛𝑛

1
𝑎𝑎𝑖𝑖−1 𝑥𝑥𝑖𝑖 = 𝑏𝑏 − 𝑐𝑐𝑖𝑖 𝑥𝑥𝑖𝑖+1 for 𝑖𝑖 = 𝑛𝑛 − 1, 𝑛𝑛 − 2, . . . , 1
𝑏𝑏𝑖𝑖 ← 𝑏𝑏𝑖𝑖 − 𝑏𝑏 2 ≤ 𝑖𝑖 ≤ 𝑛𝑛 𝑑𝑑𝑖𝑖 𝑖𝑖
𝑑𝑑𝑖𝑖−1 𝑖𝑖−1

Dr. Muhammad Majid Gulzar (CIE-KFUPM)

 Tri-diagonal Systems (Example):
Solve the following tridiagonal system.

5 2 𝑥𝑥1 12 5 1 2 12
1 5 2 𝑥𝑥2 9 5 1 2 9
𝑥𝑥3 = ⇒ 𝐷𝐷 = , 𝐴𝐴 = , 𝐶𝐶 = , 𝐵𝐵 =
1 5 2 8 5 1 2 8
1 5 𝑥𝑥4 6 5 6

Forward Elimination
𝑎𝑎1 1×2 𝑎𝑎1 1 × 12
𝑑𝑑2 = 𝑑𝑑2 − 𝑐𝑐 = 5 − = 𝟒𝟒. 𝟔𝟔 𝑏𝑏2 = 𝑏𝑏2 − 𝑏𝑏 = 9 − = 𝟔𝟔. 𝟔𝟔
𝑑𝑑1 1 5 𝑑𝑑1 1 5

𝑎𝑎2 1×2 𝑎𝑎2 1 × 6.6

𝑑𝑑3 = 𝑑𝑑3 − 𝑐𝑐 = 5 − = 𝟒𝟒. 𝟓𝟓𝟓𝟓𝟓𝟓𝟓𝟓 𝑏𝑏3 = 𝑏𝑏3 − 𝑏𝑏 = 8 − = 𝟔𝟔. 𝟓𝟓𝟓𝟓𝟓𝟓𝟓𝟓
𝑑𝑑2 2 4.6 𝑑𝑑2 2 4.6

𝑎𝑎3 1×2 𝑎𝑎3 1 × 6.5652

𝑑𝑑4 = 𝑑𝑑4 − 𝑐𝑐3 = 5 − = 𝟒𝟒. 𝟓𝟓𝟓𝟓𝟓𝟓𝟓𝟓 𝑏𝑏4 = 𝑏𝑏4 − 𝑏𝑏3 = 6 − = 𝟒𝟒. 𝟓𝟓𝟓𝟓𝟓𝟓𝟓𝟓
𝑑𝑑3 4.5652 𝑑𝑑3 4.5652

Dr. Muhammad Majid Gulzar (CIE-KFUPM)

 Tri-diagonal Systems (Example):
After the Forward Elimination
𝐷𝐷𝑇𝑇 = 5 4.6 4.5652 4.5619 𝐵𝐵𝑇𝑇 = 12 6.6 6.5652 4.5619

Backward substitution
𝑏𝑏4 4.5619
𝑥𝑥4 = = = 𝟏𝟏
𝑑𝑑4 4.5619

𝑏𝑏3 − 𝑐𝑐3 𝑥𝑥4 6.5652 − 2 × 1

𝑥𝑥3 = = = 𝟏𝟏
𝑑𝑑3 4.5652

𝑏𝑏2 − 𝑐𝑐2 𝑥𝑥3 6.6 − 2 × 1

𝑥𝑥2 = = = 𝟏𝟏
𝑑𝑑2 4.6

𝑏𝑏1 − 𝑐𝑐1 𝑥𝑥2 12 − 2 × 1

𝑥𝑥1 = = = 𝟐𝟐
𝑑𝑑1 5

Dr. Muhammad Majid Gulzar (CIE-KFUPM)

 Tri-diagonal Systems (Class Activity):
10 −1 𝑥𝑥1 16
Solve the following tridiagonal system. −2 10 −1 𝑥𝑥2 8
𝑥𝑥3 =
−2 10 −1 4
−2 10 𝑥𝑥4 2

Forward Elimination Backward Substitution

𝑎𝑎𝑖𝑖−1 𝑏𝑏𝑛𝑛
𝑑𝑑𝑖𝑖 ← 𝑑𝑑𝑖𝑖 − 𝑐𝑐 𝑥𝑥𝑛𝑛 =
𝑑𝑑𝑖𝑖−1 𝑖𝑖−1 𝑑𝑑𝑛𝑛

1
𝑎𝑎𝑖𝑖−1 𝑥𝑥𝑖𝑖 = 𝑏𝑏 − 𝑐𝑐𝑖𝑖 𝑥𝑥𝑖𝑖+1 for 𝑖𝑖 = 𝑛𝑛 − 1, 𝑛𝑛 − 2, . . . , 1
𝑏𝑏𝑖𝑖 ← 𝑏𝑏𝑖𝑖 − 𝑏𝑏 2 ≤ 𝑖𝑖 ≤ 𝑛𝑛 𝑑𝑑𝑖𝑖 𝑖𝑖
𝑑𝑑𝑖𝑖−1 𝑖𝑖−1

1.721
1.211
Answer: 𝐷𝐷𝑇𝑇 = 10 9.8 9.796 9.796 𝐵𝐵𝑇𝑇 = 16 11.2 6.286 3.283 𝑥𝑥 =
0.676
0.335
Dr. Muhammad Majid Gulzar (CIE-KFUPM)