Gauss Elimination

Somnath Roy
Department of Mechanical Engineering

1
Gauss elimination
Earliest documented methodology for solving system of linear equations.
Credit- Chinese mathematicians (109 AD), Sir Isaac Newton (1670), Gauss (1910)
formalized the notations. Later used in computer programs.

2
 a11 x1  a12 x2  ......  a1n xn  b1 (1)
a21 x1  a22 x2  ......  a2 n xn  b2 (2)
.
an1 x1  an 2 x2  ......  ann xn  bn (n)

Forward elimination: subtract (1)*a21/a11 from (2), (1)*a31/a11 from (3) and so on
a11 x1  a12 x2  ......  a1n xn  b1 (1)
 x2  ......  a2 n xn  b2
a22 (2)
.
an 2 x2  ......  ann
 xn  bn (n)
a1 j ai1
where aij  aij  This is valid if the pivot
a11
a11≠0
subtract (2’)*a31/a’22 from (3’), (2’)*a41/a’22 from (3’) and so on

a11 x1  a12 x2  ............  a1n xn  b1 (1)

 x2  .............  a2 n xn  b2
a22 (2)
 x3  ...  a3n xn  b3
a33 (3)
.
a2 j ai2
an3 x3  ...  ann
 xn  bn (n ) where aij  aij  ,  0
a22
a22

After n-1 steps

a11 x1  a12 x2  ............  a1n xn  b1 (1)
 x2  .............  a2 n xn  b2
a22 (2)
 x3  ...  a3n xn  b3
a33 (3)
.
a ( n 1) nn xn  b( n 1) n (n n 1 )
( n 1) ( n  2)
a ( n  2) ( n 1) n a ( n  2) n ( n 1)
where a nn a nn  ( n  2)
, a( n  2) ( n 1)( n 1)  0
a ( n 1)( n 1)
Back substitution: subtract eqn (n)*ai-1,n/an,n from eqn (i-1) [ if an,n≠0]
a*11 x1  a*12 x2  .........  a *1( n 1) xn 1  0  b *1 (1)
a*22 x2  ...........  a *2( n 1) xn 1  0  b *2 (2)
a 33 x3  ..  a
* *
x
3( n 1) n 1 0b *
3 (3)
.
.
a*( n 1)( n 1) xn 1  0  b* n 1 (n-1)
a nn xn  b
* *
n (n)

b*1
x1  (1)
a*11
After similar n-1 steps b*2
a*11 x1  b*1 (1) x2  (2)
a*22
a *
x b
22 2
*
2 (2) b*3
x3  (3)
a x b
*
33 3
*
3 (3) a*33 These steps are valid if a*ii≠0
.  .
. .
a*( n 1)( n 1) xn 1  b*n 1 (n-1) b*n 1
xn 1  (n-1)
a*( n 1)( n 1)
a*nn xn  b*n (n)
b*n
xn  (n)
a*nn
 Restriction on success of Gauss-Elimination
This method fails if any of the pivots is zero at any step.

Equations may rearranged to get non zero pivot. However, this may not work in
following cases

A pivot becomes zero during forward elimination if the equation (row) is linear
combination of previous equations.

A pivot becomes zero during back substitution if the equation (row) is linear
combination of subsequent equations also.

6
 Success of Gauss-Elimination
Gauss Elimination will be successful for equations (rows/columns) with no linear
dependence.

For full rank matrices or matrices with non-zero determinants.

Infinite solutions can not be calculated using Gauss-Elimination

7
Example-1 forward elimination
2w  3x  4 y  5 z  14 (12 )
2w  3x  4 y  5 z  14 (1)
w  2x  3y  z  3 (2) 3.5 x  y  1.5 z  4 (22 )
3w  x  y  3 z  0 (3) 8 y  9 z  17 (32 )
w x  2y  z  5 (4)  y / 7  9 / 7 z  10 / 7 (4 2 )

2w  3x  4 y  5 z  14 (11 ) 2 w  3 x  4 y  5 z  14 (13 )
3.5 x  y  1.5 z  4 (21 ) 3.5 x  y  1.5 z  4 (23 )
3.5 x  7 y  10.5 z  21 (31 ) 8 y  9 z  17 (33 )
0.5 x  1.5 z  2 (41 ) (9 / 7  9 / 56) z  10 / 7 +17/56 (43 )

8
Example-1 (contd.)- back substitution
2 w  3x  4 y  5 z  14 (13 )
3.5 x  y  1.5 z  4 (23 )
8 y  9 z  17 (33 )
9 / 8 z  9 / 8 (43 )
2w  3 x  4 y  9 (1 ) 4 2 w  2 (16 )

3.5 x  y  2.5 (24 ) 3.5 x  3.5 (26 )

8 y  8 (34 ) 8 y  8 (36 )

9 / 8 z  9 / 8 (4 4 ) 9 / 8 z  9 / 8 (46 )

2 w  3x  5 (15 )  w  1
3.5 x  3.5 5
(2 )  x  1
  
8 y  8 (35 )   
 y  1
9 / 8 z  9 / 8 (45 )  z  1

9
Cost of computation
As this is used in computer programs for large matrix, it is important to estimate the floating
point operations during this process.

Consider a matrix (A)nxn: Associated b-vector has n elements also.

Forward elimination:
First step does not operate on first row. Therefore, total number of calculations on matrix
components=n2-n.
Next step takes (n-1) 2-(n-1). The last step takes 22-2 steps

10
Cost of computation
Total no. of operations in forward substitution=
n2-n+ (n-1) 2-(n-1)+…. + 22-2=n(n+1)(2n+1)/6 – n(n+1)/2=(n3-n)/3

In back substitution, The number of operations are small=1+2+…n.

Total number of operations is of the order of n3

11
Example-2 forward substitution
x  2 y  4 z  3 (1 )
x  2 y  4 z  3 (1)
5 y  13 z  13 (2)
x  2 y  z  2 (2)
5z  5 (3)
3x  y  z  4 (3)

x  2 y  4 z  3 (1 ) Backward substitution

5 z  5 (2) will follow
5 y  13z  13 (3)

12
Example-3 break of elimination steps
x  2 y  4 z  3 (1 )
x  2 y  4 z  3 (1)
 y  5 z  5 (2)
x  y  z  2 (2)
02 (3)!!!!!!
2 x  3 y  3z  1 (3)

x  2 y  4 z  3 (1 )
 y  5 z  5 (2) The process breaks here. Row permutation will not
 y  5z  7 (3) work

13
Matrix operations during Gauss elimination
x  2 y  4 z  3 (1) 1 2 4   x   3
   
x  2y  z  2 (2)  1 2 1   y    2 
3x  y  z  4 (3) 3 1 1   z   4 

1 2 4   x   3
x  2 y  4 z  3 (1 )    
 0 0 5   y    5 
5 z  5 (2)
0 5 13   z  13 
5 y  13z  13 (3)

1 2 4   x  3
x  2 y  4 z  3 (1 )    
 0 5 13   y   13 
5 y  13z  13 (2)
0 0 5   z   5 
5z  5 (3)
Matrix operations during row combinations
Multiplying row 1 by l and subtracting from row 2 is represented by
multiplication with matrix
 1 0 0
 l 1 0 
 
 0 0 1 

 1 0 0   a1 a2 a3   a1 a2 a3 
 l 1 0   b b2  
b3   b1  la1 b2  la2 b3  la3 
  1
 0 0 1   c1 c2 c3   c1 c2 c3 
  1 0 0 1 0 0 
Inverse of  l 1 0 is  l 1 0
   
 0 0 1  0 0 1 

As again addition of row 1 multiplied by l with row 2 brings back the

original matrix.
Matrix operations during row permutations
Interchanging row 3 and row 2 is obtained by multiplication with permutation
matrix
1 0 0 
0 0 1 
 
 0 1 0 

1 0 0   a1 a2 a3   a1 a2 a3 
0 0 1  b b2 b3    c1 c2 c3 
  1
 0 1 0   c1 c2 c3   b1 b2 b3 

Inverse of permutation matrix is the matrix itself

Matrix operations during Gauss elimination (ex 2)
1 2 4   x  3
   
 1 2 1   y    2 
3 1 1   z   4 

 1 2 4   x   3  1 2 4   1 0 0  1 2 4 
   
  0 0 5   y    5   0 0 5    1 1 0  1 2 1 
    
   
 3 1 1   z   4   3 1 1   0 0 1  3 1 1 

1 2 4  1 0 0  1 2 4   1 0 0   1 0 0  1 2 4   1 0 0  1 2 4 
1 2 4   x   3 0 0 5    0 1 0  0 0 5    0 1 0   1 1 0  1 2 1    1 1 0  1 2 1 
   
  0 0 5   y    5             
 0 5 13   z  13  0 5 13   3 0 1  3 1 1   3 0 1   0 0 1  3 1 1   3 0 1  3 1 1 

1 2 4   x  3 1 2 4  1 0 0  1 2 4  1 0 0   1 0 0  1 2 4   1 0 0  1 2 4 
    0 5 13   0 0 1  0 0 5   0 0 1   1 1 0  1 2 1    3 0 1  1 2 1 
  0 5 13   y   13             
 0 0 5   z   5  0 0 5  0 1 0  0 5 13  0 1 0   3 0 1  3 1 1   1 1 0  3 1 1 