Simultaneous Linear Equations

Topic: Gaussian Elimination

6/14/2007 http://numericalmethods.eng.usf.e 1
du
 Gaussian Elimination
One of the most popular techniques for
solving simultaneous linear equations of the
form [A ][X ] = [C]

Consists of 2 steps
1. Forward Elimination of Unknowns.

2. Back Substitution
 Forward Elimination
The goal of Forward Elimination is to transform
the coefficient matrix into an Upper Triangular
Matrix

⎡ 25 5 1⎤ ⎡25 5 1 ⎤
⎢ 64 8 1⎥ → ⎢ 0 − 4.8 − 1.56⎥
⎢ ⎥ ⎢ ⎥
⎢⎣144 12 1⎥⎦ ⎢⎣ 0 0 0.7 ⎥⎦
 Forward Elimination
Linear Equations
A set of n equations and n unknowns

a11 x1 + a12 x2 + a13 x3 + ... + a1n xn = b1

a21 x1 + a22 x2 + a23 x3 + ... + a2 n xn = b2
. .
. .
. .
an1 x1 + an 2 x2 + an 3 x3 + ... + ann xn = bn
 Forward Elimination
Transform to an Upper Triangular Matrix
Step 1: Eliminate x1 in 2nd equation using equation 1 as
the pivot equation

⎡ Eqn1⎤
⎢ ⎥ × (a21 )
⎣ a11 ⎦

Which will yield

a21 a21 a21
a21 x1 + a12 x2 + ... + a1n xn = b1
a11 a11 a11
 Forward Elimination
Zeroing out the coefficient of x1 in the 2nd equation.
Subtract this equation from 2nd equation

⎛ a21 ⎞ ⎛ a21 ⎞ a21

⎜⎜ a22 − ⎟ ⎜
a12 ⎟ x2 + ... + ⎜ a2 n − ⎟
a1n ⎟ xn = b2 − b1
⎝ a11 ⎠ ⎝ a11 ⎠ a11

Or Where

'
x2 + ... + a2' n xn = b2' a21
a22 '
a22 = a22 − a12
a11
M
a21
a2' n = a2n − a1n
a11
 Forward Elimination
Repeat this procedure for the remaining
equations to reduce the set of equations as

a11 x1 + a12 x2 + a13 x3 + ... + a1n xn = b1

'
a22 x2 + a23
'
x3 + ... + a2' n xn = b2'
'
a32 x2 + a33
'
x3 + ... + a3' n xn = b3'
. . .
. . .
. . .

an' 2 x2 + an' 3 x3 + ... + ann

'
xn = bn'
 Forward Elimination

Step 2: Eliminate x2 in the 3rd equation.

Equivalent to eliminating x1 in the 2nd equation
using equation 2 as the pivot equation.

⎡ Eqn2 ⎤
Eqn3 − ⎢ ⎥ × (a32 )
⎣ a22 ⎦
 Forward Elimination
This procedure is repeated for the remaining
equations to reduce the set of equations as

a11 x1 + a12 x2 + a13 x3 + ... + a1n xn = b1

'
a22 x2 + a23
'
x3 + ... + a2' n xn = b2'
"
a33 x3 + ... + a3" n xn = b3"
. .
. .
. .

an" 3 x3 + ... + ann

"
xn = bn"
 Forward Elimination
Continue this procedure by using the third equation as the pivot
equation and so on.
At the end of (n-1) Forward Elimination steps, the system of
equations will look like:

a11 x1 + a12 x 2 + a13 x3 + ... + a1n x n = b1

'
a22 x2 + a23
'
x3 + ... + a2' n xn = b2'
a x + ... + a x = b
"
33 3
"
n n
"
3
. .
. .
. .

( n −1) (n −1 )
ann xn = bn
 Forward Elimination
At the end of the Forward Elimination steps

⎡a11 a12 a13 L a1n ⎤ ⎡ x1 ⎤ ⎡ b1 ⎤

⎢ ' ' ' ⎥ ⎢ ⎥ ⎢ ' ⎥
⎢ a 22 a23 L a2 n ⎥ ⎢ x2 ⎥ ⎢ b2 ⎥
⎢ "
a
33 L a3n x3 = b3 ⎥
"
⎥ ⎢ ⎥ ⎢ "

⎢ ⎥⎢ ⎥ ⎢ ⎥
⎢ M M ⎥⎢ M ⎥ ⎢ M ⎥
⎢⎣ ( n −1) ⎥ ⎢ ⎥ ⎢ (n -1 ) ⎥
ann ⎦ ⎣ xn ⎦ ⎣bn ⎦
 Back Substitution
The goal of Back Substitution is to solve each of
the equations using the upper triangular matrix.

⎡a11 a12 a13 ⎤ ⎡ x1 ⎤ ⎡ b1 ⎤

⎢0 a ⎥ ⎢ ⎥ ⎢
a23 ⎥ ⎢ x 2 ⎥ = ⎢b2 ⎥ ⎥
⎢ 22

⎢⎣ 0 0 a33 ⎥⎦ ⎢⎣ x 3 ⎥⎦ ⎢⎣b3 ⎥⎦
Example of a system of 3 equations
 Back Substitution
Start with the last equation because it has only
one unknown

( n −1)
b
xn = n
( n −1)
a nn

Solve the second from last equation (n-1)th

using xn solved for previously.
This solves for xn-1.
 Back Substitution
Representing Back Substitution for all equations
by formula

(i −1)
− ∑ aij(i −1) x j
n
bi
j =i +1
xi = For i=n-1, n-2,….,1
aii(i −1)
and

( n −1)
b
xn = n
( n −1)
a nn
 Example: Rocket Velocity
The upward velocity of a rocket
is given at three different times
Time, t Velocity, v

s m/s
5 106.8
8 177.2
12 279.2

The velocity data is approximated by a polynomial as:

v (t ) = a1t 2 + a 2 t + a 3 , 5 ≤ t ≤ 12.

Find: The Velocity at t=6,7.5,9, and 11 seconds.

 Example: Rocket Velocity
Assume
v (t ) = a 1 t 2 + a 2 t + a 3 , 5 ≤ t ≤ 12.
Results in a matrix template of the form:
⎡ t 12 t1 1⎤ ⎡ a1 ⎤ ⎡ v1 ⎤
⎢ 2 ⎥ ⎢a ⎥ = ⎢v ⎥
⎢t2 t2 1⎥ ⎢ 2⎥ ⎢ 2⎥
⎢ t 32 t3 1 ⎥⎦ ⎢⎣ a 3 ⎥⎦ ⎢⎣ v 3 ⎥⎦
⎣
Using date from the time / velocity table, the matrix becomes:
⎡ 25 5 1⎤ ⎡ a 1 ⎤ ⎡106 . 8 ⎤
⎢ 64 8 1⎥⎥ ⎢ a ⎥ = ⎢177 . 2 ⎥
⎢ ⎢ 2⎥ ⎢ ⎥
⎢⎣144 12 1⎥⎦ ⎢⎣ a 3 ⎥⎦ ⎢⎣ 279 . 2 ⎥⎦
 Example: Rocket Velocity
Forward Elimination: Step 1
⎡ Row1⎤
Row2 − ⎢ ⎥ × (64) =
⎣ 25 ⎦
Yields

⎡ 25 5 1 ⎤ ⎡ a 1 ⎤ ⎡ 106.81 ⎤
⎢ 0 − 4.8 − 1.56⎥ ⎢a ⎥ = ⎢− 96.21⎥
⎢ ⎥ ⎢ 2⎥ ⎢ ⎥
⎢⎣144 12 1 ⎥⎦ ⎢⎣a 3 ⎥⎦ ⎢⎣ 279.2 ⎥⎦
 Example: Rocket Velocity
Forward Elimination: Step 1
⎡ Row1⎤
Row3 − ⎢ ⎥ × (144) =
⎣ 25 ⎦
Yields

⎡25 5 1 ⎤ ⎡ a 1 ⎤ ⎡ 106.8 ⎤
⎢ 0 − 4.8 − 1.56 ⎥ ⎢a ⎥ = ⎢ − 96.21⎥
⎢ ⎥ ⎢ 2⎥ ⎢ ⎥
⎢⎣ 0 − 16.8 − 4.76⎥⎦ ⎢⎣a 3 ⎥⎦ ⎢⎣− 336.0⎥⎦
 Example: Rocket Velocity
Forward Elimination: Step 2
⎡ Row2 ⎤
Row3 − ⎢ ⎥ × (−16.8) =
⎣ − 4 .8 ⎦
Yields

⎡25 5 1 ⎤ ⎡ a 1 ⎤ ⎡ 106.8 ⎤
⎢ 0 − 4.8 − 1.56⎥ ⎢a ⎥ = ⎢− 96.21⎥
⎢ ⎥ ⎢ 2⎥ ⎢ ⎥
⎢⎣ 0 0 0.7 ⎥⎦ ⎢⎣a 3 ⎥⎦ ⎢⎣ 0.735 ⎥⎦

This is now ready for Back Substitution

 Example: Rocket Velocity
Back Substitution: Solve for a3 using the third equation

0.7a3 = 0.735
0.735
a3 =
0.7
a3 = 1.050
 Example: Rocket Velocity
Back Substitution: Solve for a2 using the second equation

− 4.8a2 − 1.56a3 = −96.21

− 96.21 + 1.56a3
a2 =
− 4.8
-96.21 + 1.56(1.050 )
a2 =
-4.8
a2 = 19.70
 Example: Rocket Velocity
Back Substitution: Solve for a1 using the first equation

25a1 + 5a 2 + a3 = 106.8
106.8 − 5a 2 − a3
a1 =
25
106.8 − 5(19.70 ) − 1.050
a1 =
25

a1 = 0.2900
 Example: Rocket Velocity
Solution: ⎡ a1 ⎤ ⎡0.2900⎤
The solution vector is ⎢a ⎥ = ⎢ 19.70 ⎥
⎢ 2⎥ ⎢ ⎥
⎢⎣ a3 ⎥⎦ ⎢⎣ 1.050 ⎥⎦

The polynomial that passes through the three data points is

then:

v(t ) = a1t 2 + a 2 t + a3
= 0.2900t 2 + 19.70t + 1.050, 5 ≤ t ≤ 12
 Example: Rocket Velocity
Solution:
Substitute each value of t to find the corresponding velocity

v(6 ) = 0.2900(6 ) + 19.70(6 ) + 1.050

2

= 129.69 m / s.
v(7.5) = 0.2900(7.5) + 19.70(7.5) + 1.050
2

= 165.1 m / s.
v(9 ) = 0.2900(9 ) + 19.70(9 ) + 1.050
2

= 201.8 m / s.
v(11) = 0.2900(11) + 19.70(11) + 1.050
2

= 252.8 m / s.
 Pitfalls

Two Potential Pitfalls

-Division by zero: May occur in the forward elimination
steps. Consider the set of equations:
10 x2 − 7 x3 = 7
6 x1 + 2.099 x2 + 3x3 = 3.901
5 x1 − x2 + 5 x3 = 6
- Round-off error: Prone to round-off errors.
 Pitfalls: Example
Consider the system of equations:
Use five significant figures with chopping

⎡ 10 − 7 0⎤ ⎡ x1 ⎤ ⎡ 7 ⎤
⎢− 3 2.099 6⎥ ⎢ ⎥ ⎢ ⎥
⎢ x
⎥ ⎢ 2⎥ ⎢ = 3.901⎥
⎢⎣ 5 − 1 5⎥⎦ ⎢⎣ x3 ⎥⎦ ⎢⎣ 6 ⎥⎦

At the end of Forward Elimination

⎡10 −7 0 ⎤ ⎡ x1 ⎤ ⎡ 7 ⎤
⎢ 0 − 0.001 6 ⎥ ⎢ x ⎥ = ⎢ 6.001 ⎥
⎢ ⎥ ⎢ 2⎥ ⎢ ⎥
⎢⎣ 0 0 15005⎥⎦ ⎢⎣ x3 ⎥⎦ ⎢⎣15004⎥⎦
 Pitfalls: Example
Back Substitution
15004
x3 = = 0.99993
−7 15005
⎡10 0 ⎤ ⎡ x1 ⎤ ⎡ 7 ⎤
⎢ 0 − 0.001 6 ⎥ ⎢ x ⎥ = ⎢ 6.001 ⎥
⎢ ⎥⎢ 2 ⎥ ⎢ ⎥ 6.001 − 6 x3
⎢⎣ 0 0 15005⎥⎦ ⎢⎣ x3 ⎥⎦ ⎢⎣15004⎥⎦ x2 = = −1.5
− 0.001

7 + 7 x 2 − 0 x3
x1 = = −0.3500
10
 Pitfalls: Example

Compare the calculated values with the exact solution

⎡ x1 ⎤ ⎡ 0 ⎤
[X ] exact ⎢ ⎥ ⎢
= ⎢ x 2 ⎥ = ⎢− 1⎥⎥
⎢⎣ x3 ⎥⎦ ⎢⎣ 1 ⎥⎦
⎡ x1 ⎤ ⎡ − 0.35 ⎤
[X ] calculated = ⎢⎢ x 2 ⎥⎥ = ⎢⎢ − 1.5 ⎥⎥
⎢⎣ x3 ⎥⎦ ⎢⎣0.99993⎥⎦
 Improvements

Increase the number of significant digits

Decreases round off error

Does not avoid division by zero

Gaussian Elimination with Partial Pivoting

Avoids division by zero

Reduces round off error

 Partial Pivoting

Gaussian Elimination with partial pivoting applies row switching to

normal Gaussian Elimination.
How?
At the beginning of the kth step of forward elimination, find the maximum of

akk , ak +1,k ,................, ank

If the maximum of the values is a pk In the pth row, k ≤ p ≤ n,
then switch rows p and k.
 Partial Pivoting

What does it Mean?

Gaussian Elimination with Partial Pivoting ensures that

each step of Forward Elimination is performed with the
pivoting element |akk| having the largest absolute value.
 Partial Pivoting: Example
Consider the system of equations

10 x1 − 7 x 2 = 7
− 3x1 + 2.099 x 2 + 3x 3 = 3.901
5x 1 − x 2 + 5x 3 = 6
In matrix form
⎡ 10 7 0⎤ ⎡ x1 ⎤ ⎡ 7 ⎤
⎢− 3 2.099 6⎥ ⎢ x ⎥ ⎢3.901⎥
⎢ ⎥ ⎢ 2⎥ = ⎢ ⎥
⎢⎣ 5 − 1 5⎥⎦ ⎢⎣ x3 ⎥⎦ ⎢⎣ 6 ⎥⎦

Solve using Gaussian Elimination with Partial Pivoting using five

significant digits with chopping
 Partial Pivoting: Example
Forward Elimination: Step 1
Examining the values of the first column
|10|, |-3|, and |5| or 10, 3, and 5
The largest absolute value is 10, which means, to follow the
rules of Partial Pivoting, we switch row1 with row1.

Performing Forward Elimination

⎡ 10 7 0⎤ ⎡ x1 ⎤ ⎡ 7 ⎤ ⎡10 −7 0⎤ ⎡ x1 ⎤ ⎡ 7 ⎤
⎢− 3 2.099 6⎥ ⎢ x ⎥ = ⎢3.901⎥
⎢
⎢⎣ 5
⎥⎢ 2 ⎥ ⎢ ⎥
− 1 5⎥⎦ ⎢⎣ x3 ⎥⎦ ⎢⎣ 6 ⎥⎦
⇒ ⎢ 0 − 0.001 6⎥ ⎢ x ⎥ = ⎢6.001⎥
⎢
⎢⎣ 0 2.5
⎥⎢ 2 ⎥ ⎢ ⎥
5⎥⎦ ⎢⎣ x3 ⎥⎦ ⎢⎣ 2.5 ⎥⎦
 Partial Pivoting: Example
Forward Elimination: Step 2
Examining the values of the first column
|-0.001| and |2.5| or 0.0001 and 2.5
The largest absolute value is 2.5, so row 2 is switched with
row 3

Performing the row swap

⎡10 −7 0⎤ ⎡ x1 ⎤ ⎡ 7 ⎤ ⎡10 −7 0⎤ ⎡ x1 ⎤ ⎡ 7 ⎤
⎢ 0 − 0.001 6⎥ ⎢ x ⎥ = ⎢6.001⎥
⎢
⎢⎣ 0 2.5
⎥⎢ 2 ⎥ ⎢ ⎥
5⎥⎦ ⎢⎣ x3 ⎥⎦ ⎢⎣ 2.5 ⎥⎦
⇒ ⎢0
⎢ 2 . 5 5 ⎥ ⎢ x ⎥ = ⎢ 2.5 ⎥
⎥⎢ 2 ⎥ ⎢ ⎥
⎢⎣ 0 − 0.001 6⎥⎦ ⎢⎣ x3 ⎥⎦ ⎢⎣6.001⎥⎦
 Partial Pivoting: Example
Forward Elimination: Step 2

Performing the Forward Elimination results in:

⎡10 − 7 0 ⎤ ⎡ x1 ⎤ ⎡ 7 ⎤
⎢ 0 2.5 5 ⎥ ⎢ x ⎥ = ⎢ 2.5 ⎥
⎢ ⎥⎢ 2 ⎥ ⎢ ⎥
⎢⎣ 0 0 6.002⎥⎦ ⎢⎣ x3 ⎥⎦ ⎢⎣6.002⎥⎦
 Partial Pivoting: Example
Back Substitution
Solving the equations through back substitution

⎡10 − 7
6.002
0 ⎤ ⎡ x1 ⎤ ⎡ 7 ⎤ x3 = =1
⎢ 0 2.5 5 ⎥ ⎢ x ⎥ = ⎢ 2.5 ⎥ 6.002
⎢ ⎥⎢ 2 ⎥ ⎢ ⎥
⎢⎣ 0 0 6.002⎥⎦ ⎢⎣ x3 ⎥⎦ ⎢⎣6.002⎥⎦ 2.5 − 5 x 2
x2 = =1
2.5

7 + 7 x 2 − 0 x3
x1 = =0
10
 Partial Pivoting: Example
Compare the calculated and exact solution
The fact that they are equal is coincidence, but it does
illustrate the advantage of Partial Pivoting

⎡ x1 ⎤ ⎡ 0 ⎤ ⎡ x1 ⎤ ⎡ 0 ⎤
[X ] calculated = ⎢⎢ x2 ⎥⎥ = ⎢⎢− 1⎥⎥ [X ] exact = ⎢⎢ x 2 ⎥⎥ = ⎢⎢− 1⎥⎥
⎢⎣ x3 ⎥⎦ ⎢⎣ 1 ⎥⎦ ⎢⎣ x3 ⎥⎦ ⎢⎣ 1 ⎥⎦
 Summary

-Forward Elimination
-Back Substitution
-Pitfalls
-Improvements
-Partial Pivoting