BITS Pilani

Hyderabad Campus

Lecture -9:
SYSTEM of LINEAR ALGEBRAIC EQUATIONS
Direct Methods: Gauss Elimination
1
Solving Systems of Equations

• A linear equation in n variables:

a1x1 + a2x2 + … + anxn = b

• For small (n ≤ 3), linear algebra provides several tools to solve

such systems of linear equations:

– Graphical method
– Cramer’s rule
– Method of elimination

• Nowadays, easy access to computers makes the solution of

very large sets of linear algebraic equations possible

CHE F242 Numerical Methods for Chemical Engineers, BITS, Pilani-Hyderabad Campus P-2
Graphical Method

Graphical solution of set of two

simultaneous algebraic equations.

Interaction of lines represents the

solution

What about graphical representation

of system of three simultaneous
algebraic equations ?

Solution is the point of intersect of

three planes.

CHE F242 Numerical Methods for Chemical Engineers, BITS, Pilani-Hyderabad Campus P-3
Graphical Method

Graphical Depiction of singular and ill-

conditioned systems
a) No solution
b) Infinite solutions
c) Ill-conditioned system (slopes are
very close )
a)

c) b)

CHE F242 Numerical Methods for Chemical Engineers, BITS, Pilani-Hyderabad Campus P-4
Determinants and Cramer’s Rule

Ax  b [A] : coefficient matrix

a11 a12 a13  a11 a12 a13   x1  b1 

  a a a   x   b 
A  a21 a22 a23   21 22 23   2   2 
a31 a32 a33  a31 a32 a33   x3  b3 

b1 a12 a13 a11 b1 a13 a11 a12 b1

b2 a22 a23 a21 b2 a23 a21 a22 b2
b3 a32 a33 a31 b3 a33 a31 a32 b3
x1  x2  x3 
D D D
D : Determinant of A matrix
CHE F242 Numerical Methods for Chemical Engineers, BITS, Pilani-Hyderabad Campus P-5
Computing the Determinant
a11 a12 a13
D  a21 a22 a23
a31 a32 a33

Ax  B D11 

a22 a23
a32 a33
 a22 a33  a32 a23

a21 a23
a11 a12 a13  D12   a21 a33  a31 a23
A  a21 a22 a23  a31 a33
a21 a22
a31 a32 a33  D13   a21 a32  a31 a22
a31 a32
a22 a23 a21 a23 a21 a22
Determinant of A  D  a11  a12  a13
a32 a33 a31 a33 a31 a32
CHE F242 Numerical Methods for Chemical Engineers, BITS, Pilani-Hyderabad Campus P-6
Gauss Elimination

 a11 a12 a13 b1 

• Solve Ax = b a b2 
 21 a22 a23
a31 a32 a33 b3 
• Consists of two phases: Forward
–Forward elimination  Elimination
–Back substitution a11 a12 a13 b1 
 0 a' '
a23 b2' 
 22
• Forward Elimination  0 0 ''
a33 b3'' 
reduces Ax = b to an upper
triangular system Tx = b’ 
b3'' b2'  a23
'
x3
• Back substitution can then x3  '' x2  ' Back
a33 a22
solve Tx = b’ for x Substitution
b1  a13 x3  a12 x2
x1 
a11
CHE F242 Numerical Methods for Chemical Engineers, BITS, Pilani-Hyderabad Campus P-7
Forward Elimination
x1 - x2 + x3 = 6 x1 - x2 + x3 = 6
-(3/1) 3x1 + 4x2 + 2x3 = 9 0 +7x2 - x3 = -9
-(2/1) 2x1 + x2 + x3 = 7 -(3/7) 0 + 3x2 - x3 = -5

x1 - x2 + x3 = 6
0 7x2 - x3 = -9
0 0 -(4/7)x3=-(8/7)
Solve using BACK SUBSTITUTION: x3 = 2 x2=-1 x1 =3

0
0 0
0 0 0
0 0 0 0
0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0 0
0 0 0 0 0 0 0 0

CHE F242 Numerical Methods for Chemical Engineers, BITS, Pilani-Hyderabad Campus P-8
Back Substitution

1x0 +1x1 –1x2 +4x3 = 8

– 2x1 –3x2 +1x3 = 5

2x2 – 3x3 = 0

x3 = 2 2x3 = 4

CHE F242 Numerical Methods for Chemical Engineers, BITS, Pilani-Hyderabad Campus P-9
Back Substitution

1x0 +1x1 –1x2 = 0

– 2x1 –3x2 = 3

x2 = 3 2x2 = 6

CHE F242 Numerical Methods for Chemical Engineers, BITS, Pilani-Hyderabad Campus P-10
Back Substitution

1x0 +1x1 = 3

x1 = –6 – 2x1 = 12

CHE F242 Numerical Methods for Chemical Engineers, BITS, Pilani-Hyderabad Campus P-11
Back Substitution

1x0 +1x1 = 3

x1 = –6 – 2x1 = 12

CHE F242 Numerical Methods for Chemical Engineers, BITS, Pilani-Hyderabad Campus P-12
x0 = 9 1x0 = 9

CHE F242 Numerical Methods for Chemical Engineers, BITS, Pilani-Hyderabad Campus P-13
Forward Elimination

  a ji 
a ji  a ji  aii    0
 aii 

CHE F242 Numerical Methods for Chemical Engineers, BITS, Pilani-Hyderabad Campus P-14
Forward Elimination
M
U
L
T
I
P 4x0 +6x1 +2x2 – 2x3 = 8
L
I
E
R
S
-(2/4) 2x0 +5x2 – 2x3 = 4

-(-4/4) –4x0 – 3x1 – 5x2 +4x3 = 1

-(8/4) 8x0 +18x1 – 2x2 +3x3 = 40

CHE F242 Numerical Methods for Chemical Engineers, BITS, Pilani-Hyderabad Campus P-15
Forward Elimination

M
U
L
T
I 4x0 +6x1 +2x2 – 2x3 = 8
P
L
I
E
R – 3x1 +4x2 – 1x3 = 0
S

-(3/-3) +3x1 – 3x2 +2x3 = 9

-(6/-3) +6x1 – 6x2 +7x3 = 24

CHE F242 Numerical Methods for Chemical Engineers, BITS, Pilani-Hyderabad Campus P-16
Forward Elimination

4x0 +6x1 +2x2 – 2x3 = 8

M
U
L
T
– 3x1 +4x2 – 1x3 = 0
I
P
L
I 1x2 +1x3 = 9
E
R

(2/1) 2x2 +5x3 = 24

CHE F242 Numerical Methods for Chemical Engineers, BITS, Pilani-Hyderabad Campus P-17
Forward Elimination

4x0 +6x1 +2x2 – 2x3 = 8

– 3x1 +4x2 – 1x3 = 0

1x2 +1x3 = 9

3x3 = 6

CHE F242 Numerical Methods for Chemical Engineers, BITS, Pilani-Hyderabad Campus P-18
Operation count in
Forward Elimination
b11

2n 0 b22

2n 0 0 b33

2n 0 0 0 b44

2n 0 0 0 0 b55

2n 0 0 0 0 0 b66

2n 0 0 0 0 0 0 b77

2n 0 0 0 0 0 0 0 b66
2n 0 0 0 0 0 0 0 0 b66

TOTAL
1st column: 2n(n-1)  2n2 2(n-1)2 2(n-2)2 …….
TOTAL # of Operations for FORWARD ELIM INATION :
n
2n  2(n  1)  ...  2 * (2)  2 * (1)  2 i 2
2 2 2 2

i 1

n(n  1)(2n  1)
2
6
 O(n 3 )
CHE F242 Numerical Methods for Chemical Engineers, BITS, Pilani-Hyderabad Campus P-19
Pitfalls of Elimination Methods

Division by zero
It is possible that during both elimination and back-substitution phases a
division by zero can occur.

For example:
2x2 + 3x3 = 8 0 2 3
4x1 + 6x2 + 7x3 = -3 A= 4 6 7
2x1 + x2 + 6x3 = 5 2 1 6

Solution: pivoting (to be discussed later)

CHE F242 Numerical Methods for Chemical Engineers, BITS, Pilani-Hyderabad Campus P-20
Pitfalls (cont.)

Round-off errors

• Because computers carry only a limited number of significant figures,

round-off errors will occur and they will propagate from one iteration to the
next.

• This problem is especially important when large numbers of equations (100

or more) are to be solved.

• Always use double-precision numbers/arithmetic. It is slow but needed for

correctness!

• It is also a good idea to substitute your results back into the original
equations and check whether a substantial error has occurred.

CHE F242 Numerical Methods for Chemical Engineers, BITS, Pilani-Hyderabad Campus P-21
Pitfalls (cont.)
ill-conditioned systems - small changes in coefficients result in large
changes in the solution. Alternatively, a wide range of answers can
approximately satisfy the equations.
(Well-conditioned systems – small changes in coefficients result in small
changes in the solution)

Problem: Since round off errors can induce small changes in the coefficients, these
changes can lead to large solution errors in ill-conditioned systems.
Example: b1 a12 10 2
x1 + 2x2 = 10
b2 a22 10.4 2 2(10)  2(10.4)
1.1x1 + 2x2 = 10.4 x1    4 x2  3
D 1(2)  2(1.1)  0.2

x1 + 2x2 = 10 b1 a12 10 2
1.05x1 + 2x2 = 10.4 b2 a22 10.4 2 2(10)  2(10.4)
x1    8 x2  1
D 1(2)  2(1.05)  0.1

CHE F242 Numerical Methods for Chemical Engineers, BITS, Pilani-Hyderabad Campus P-22
ill-conditioned systems (cont.) –
• Surprisingly, substitution of the erroneous values, x1=8 and x2=1, into the original
equation will not reveal their incorrect nature clearly:
x1 + 2x2 = 10 8+2(1) = 10 (the same!)
1.1x1 + 2x2 = 10.4 1.1(8)+2(1)=10.8 (close!)

IMPORTANT OBSERVATION:
An ill-conditioned system is one with a determinant close to zero

• If determinant D=0 then there are infinitely many solutions  singular system

• Scaling (multiplying the coefficients with the same value) does not change the
equations but changes the value of the determinant in a significant way.
However, it does not change the ill-conditioned state of the equations!
DANGER! It may hide the fact that the system is ill-conditioned!!

How can we find out whether a system is ill-conditioned or not?

Not easy! Luckily, most engineering systems yield well-conditioned results!

• One way to find out: change the coefficients slightly and recompute & compare

CHE F242 Numerical Methods for Chemical Engineers, BITS, Pilani-Hyderabad Campus P-23
Techniques for Improving Solutions
• Use of more significant figures – double precision arithmetic

• Pivoting
If a pivot element is zero, normalization step leads to division by zero. The
same problem may arise, when the pivot element is close to zero. Problem
can be avoided:
– Pivoting
Switching the rows below so that the largest element is the pivot element.

– Partial pivoting
• Searching for the largest element in all rows and columns then switching.
• This is rarely used because switching columns changes the order of x’s
and adds significant complexity and overhead  costly

• Scaling
– used to reduce the round-off errors and improve accuracy

CHE F242 Numerical Methods for Chemical Engineers, BITS, Pilani-Hyderabad Campus P-24
Gauss-Jordan Elimination

a11 0 0 0 0
0 0 0 0 b11

0 x22 0 0 0 0 0 0 0 b22

x33 0 0 0 0 0 0 b33
0 0

0 0 0 x44 0 0 0 0 0 b44

x55 0 0 0 0
0 0 0 0 b55

0 0 0 0 0 x66 0 0 0 b66

0 0 0 0 0 0 x77 0 0 b77

0 0 0 0 0 0 0 x88 0
b66

0 0 0 0 0 0 0 0 x99 b66

CHE F242 Numerical Methods for Chemical Engineers, BITS, Pilani-Hyderabad Campus P-25
Gauss-Jordan Elimination: Example
1 1 2  x1  8  1 1 2| 8 
 1  2 3  x   1  Augmented Matrix :  1  2 3 | 1 
  2   
 3 7 4  x3  10  3 7 4 |10

1 1 2 | 8  Scaling R2: 1 1 2 | 8 
R2  R2 - (-1)R1 0 1 5 | 9  R2  R2/(-1) 0 1 5| 9 
   
R3  R3 - ( 3)R1 0 4 2| 14  0 4 2| 14 

R1  R1 - (1)R2 1 0 7 | 17  1 0 7 | 17 
0 1  5 |  9  Scaling R3: 0 1  5 |  9 
   
R3  R3-(4)R2 0 0 18 | 22  R3  R3/(18) 0 0 1 |11 / 9

R1  R1 - (7)R3 1 0 0 | 8.444  RESULT:

0 1 0 |  2.888
R2  R2-(-5)R3
 
0 0 1 | 1.222  x1=8.45, x2=-2.89, x3=1.23

Time Complexity?CHE F242 O(n3for

Numerical Methods ) Chemical Engineers, BITS, Pilani-Hyderabad Campus P-26