Lecture 8 & 9

Gauss Elimination and LU Decomposition

Course Website
https://sites.google.com/view/kporwal/teaching/mtl107
Basic concepts: linear system of equations

 
x1
▶ Find x =
x2
which satisfies

a11 x1 + a12 x2 = b1
a21 x1 + a22 x2 = b2

or,
Ax = b
 
a11 a12
where A =
a21 a22
umerical Methods for Computational Science and Engineering
Basic concepts: linear system of equations
Basic concepts: linear system of equations (cont.)
Basic concepts: linear system of equations (cont.)
Solving a 2 × 2 system.
I Unique solution if and only if lines are not parallel.

CSE, Lecture 5, Oct 3, 2013 Figure 1: Solving 2 × 2 system of equations

Range space and null space

We can write Ax = b as

a1 x1 + a2 x2 + · · · + an xn = b

where aj is the j−th column of A.

If A is nonsingular, then the columns aj are linearly independent.
Then there is a unique way of writing any vector b as a linear
combination of these columns.
Hence, if A is nonsingular, then there is a unique solution for the
system Ax = b.
Range space and null space (cont.)

▶ In general, for a square n × n system there is a unique solution

if one of the following equivalent statements hold:
▶ A is nonsingular;
▶ det(A) ̸= 0;
▶ A has linearly independent columns or rows;
▶ there exists an inverse A−1 satisfying AA−1 = I = A−1 A;
▶ range(A)=Rn ;
▶ null(A)=0.
range(A) = all vectors that can be written as linear combination of
columns of A.
null(A) = nullspace of A = all vectors z for which Az = 0.
Example (Range and nullspace)

 
1 1
A=
3 3
  
1
null(A) = α , α ∈ R,
−1
  
1
range(A) = β , β ∈ R.
3
 
2
Give all solutions of Ax = b with b= .
6
Example (Almost singularity)

Let’s
 look atlinear
 system
 of equations

1+ε 1 x1 2+ε
= where 0 < ε ≪ 1.
3 3 x2 6
 
1
The unique solution is x= .
1
System matrix above is almost singular, i.e., a small perturbation
makes it singular.
The close-by problem has many solutions, some are far apart from
x: ill-conditioned problem.
Vector norms

A vector norm is a function ∥·∥ : Rn → R+ such that

1. ∥x∥ ≥ 0, for all x ∈ Rn and ∥x∥ = 0 ⇐⇒ x = 0.
2. ∥αx∥ = |α| ∥x∥ for all α ∈ R.
3. ∥x + y∥ ≤ ∥x∥ + ∥y∥, for all x, y ∈ Rn .
This generalizes absolute value or magnitude of a scalar.
Frequently used norms are
▶ The spectral norm or Euclidean norm or 2-norm or l2 norm:
It measures the Euclidean length of a vector and is defined by
v
u n
uX 2
∥x∥2 = t |xi | .
i=1
Vector norms (cont.)

▶ The infinity norm or maximum norm:

It measures the largest element in modulus and is defined by

∥x∥∞ = max |xi |

1≤i≤n

▶ The 1-norm or l1 norm:

It measures the sum of all the elements in modulus and is
defined by
X n
∥x∥1 = |xi | .
i=1
 Numerical Methods for Computational Science and Engineering
Vector norms
Vector norms (cont.)
Vector norms (cont.)

Manhattan Norm

‘Unit circles’ for `1 , `2 , and `1 norms (in 2D).

Figure 2: Errors

’Unit Circles’ of l1 , l2 and l∞ norms

NumCSE, Lecture 5, Oct 3, 2013 11/33
Vector norms (cont.)

In the finite dimensional case considered here, all norms are

equivalent, meaning that for two norms ∥·∥a and ∥·∥b there are
positive constants C1 , C2 such that

C1 ∥x∥a ≤ ∥x∥b ≤ C2 ∥x∥a , ∀x.

We can choose freely in which norm we want to measure distance.

Often a good choice simplifies the argument when proving a result.
Nevertheless the result holds in any norm, the difference is just a
constant. In particular for the three norms above we have
√
∥x∥∞ ≤ ∥x∥2 ≤ n ∥x∥∞

∥x∥∞ ≤ ∥x∥1 ≤ n ∥x∥∞

√
∥x∥2 ≤ ∥x∥1 ≤ n ∥x∥2
Matrix norms

A matrix norm is a function ∥A∥ : Rm×n → R+ such that

1. ∥A∥ ≥ 0, for all A ∈ Rm×n and ∥A∥ = 0 ⇐⇒ A = 0.
2. ∥αA∥ = |α| ∥A∥ for all α ∈ R.
3. ∥A + B∥ ≤ ∥A∥ + ∥B∥, for all A, B ∈ Rm×n .
Note that matrix norms are often also required to satisfy the
submultiplicative property,

∥AB∥ ≤ ∥A∥ ∥B∥ .

Some matrix norms are induced by vector norms by the definition

∥A∥ = sup ∥Ax∥

∥x∥=1
Matrix norms (cont.)

For the vector norms we have introduced before, the corresponding

induced matrix norms are
▶ The spectral norm or 2-norm:

∥A∥2 = sup ∥Ax∥2 = σmax (A)

∥x∥2 =1

It can be shown that ∥A∥2 is equal to the largest eigenvalue of

A⊤ A. It follows that the 2-norm is invariant under orthogonal
transformations, i.e., we have

∥QA∥2 = ∥AQ∥2 = ∥A∥2

whenever Q ⊤ Q = I .
Matrix norms (cont.)

▶ The infinity norm or maximum row sum norm:

n
X
∥A∥∞ = sup ∥Ax∥ = max |aij |
∥x∥∞ =1 1≤i≤n
j=1

The last identity comes from the observation that the vector x
which maximizes the supremum is given by x = (±1, ±1, · · · , ±1)⊤
with the sign of the entries chosen according to the sign of the
entries in the row of A with the largest row sum.
Matrix norms (cont.)

▶ The 1-norm or maximum column sum norm:

n
X
∥A∥1 = sup ∥Ax∥ = max |aij |
∥x∥1 =1 1≤j≤n
i=1

The last identity holds because the supremum is attained for

the value x = (0, 0, · · · , 0, 1, 0, · · · , 0)⊤ with 1 at the column
position of A with the largest column sum.
▶ Frobenius norm:
The commonly used Frobenius norm does not arise from
vector norms:
v v
uXn u n
2
uX
σi2 (A)
u
∥A∥F = t |aij | = t
i,j=1 i=1
Orthogonal vectors and matrices
Orthogonal vectors: Two vectors u, v ∈ Rn are orthogonal if

u⊤ v = 0

Vector u is orthonormal if ∥u∥2 = 1.

A square matrix Q is orthogonal if its columns are pairwise
orthonormal, i.e.,

Q ⊤Q = I Hence, also Q −1 = Q ⊤ .

Important property: for any orthogonal matrix Q and vector x:

∥Qx∥2 = ∥x∥2

Hence,
∥Q∥2 = Q −1 2 = 1.

Linear systems: Problem statement

We consider linear systems of equation of the form

n
X
aik xk = bi , i = 1, 2, · · · , n
k=1

or,
Ax = b
The matrix elements aik and the right-hand side elements bi are
given. We are looking for the unknowns xk .
Direct vs. iterative methods

Ax = b
A is given, real, nonsingular, n × n and b is real, given vector.
Such problems are ubiquitous!
Two types of solution approaches:
1. Direct method: yield exact solution in absence of roundoff
error.
▶ Variations of Gaussian elimination.
2. Iterative method: iterate in a similar fashion to what we do
for nonlinear problems.
▶ Use only when direct methods are ineffective.
Existence and uniqueness of LU decomposition

The decomposition of the matrix A into a (unit) lower triangular

matrix L and an upper triangular matrix U, A = LU, is called LU
decomposition. The process that computes the LU decomposition
is called Gaussian elimination.
Theorem
The square matrix A ∈ Rn×n has a unique decomposition A = LU
if and only if the leading principal submatrices
Ak = A(1 : k, 1 : k) k = 1, 2, · · · , n − 1 are nonsingular.

Theorem
If A is nonsingular, then one can find a row permutation P such
that PA satisfies the conditions of the previous theorem, that is PA
= LU exists and is unique.
Gaussian elimination for Ax = b

x1 x2 x3 x4 1
a11 a12 a13 a14 b1
a21 a22 a23 a24 b2
a31 a32 a33 a34 b3
a41 a42 a43 a44 b4

1. Permute rows i = 1, ..., 4 (if necessary) such that a11 ̸= 0.

This element is called the pivot.
2. Subtract the multiple li1 = ai1 /a11 of row 1 from row i,
i = 2, · · · , 4.
′ = a − l a , k, i = 2, · · · , 4
3. Set aik ik i1 1k
4. Set bi′ = bi − li1 b1 , i = 2, · · · , 4
Gaussian elimination for Ax = b (cont.)

x1 x2 x3 x4 1
a11 a12 a13 a14 b1
0 ′
a22 ′
a23 ′
a24 b2′
0 ′
a32 ′
a33 ′
a34 b3′
0 ′
a42 ′
a43 ′
a44 b4′

′ ̸= 0.
1. Permute rows i = 2, ..., 4 (if necessary) such that a22
This is next pivot.
′ = a′ /a′ of row 2 from row i,
2. Subtract multiples li2 i2 22
i = 3, · · · , 4.
′′ = a′ − l ′ a′ , k, i = 3, · · · , 4
3. Set aik ik i2 2k
4. Set bi′′ = bi′ − li2
′ b ′ , i = 3, · · · , 4
2
Gaussian elimination for Ax = b (cont.)

x1 x2 x3 x4 1
a11 a12 a13 a14 b1
0 ′
a22 ′
a23 ′
a24 b2′
0 0 ′
a33 ′
a34 b3′
0 0 ′
a43 ′
a44 b4′

′′ ̸= 0.
1. Permute rows i = 3, ..., 4 (if necessary) such that a33
This is the next pivot.
′′ = a′′ /a′′ of row 3 from row 4.
2. Subtract multiples l43 43 33
′′′ = a′′ − l ′′ a′′ ,
3. Set a44 44 43 34
4. Set b4′′′ = b4′′ − li1
′′ b ′′ .
3
Gaussian elimination for Ax = b (cont.)

x1 x2 x3 x4 1 x1 x2 x3 x4 1
u11 u12 u13 u14 c1 a11 a12 a13 a14 b1
0 u22 u23 u24 c2 ⇐⇒ 0 ′
a22 ′
a23 ′
a24 b2′
0 0 u33 u34 c3 0 0 ′
a33 ′
a34 b3′
0 0 0 u44 c4 0 0 0 ′
a44 b4′

Actual storage scheme

x1 x2 x3 x4 1
a11 a12 a13 a14 b1
l21 ′
a22 ′
a23 ′
a24 b2′
l31 ′
l32 ′′
a33 ′′
a34 b3′′
l41 ′
l42 ′
l43 ′′′
a44 b4′′′

plus vector that stores info on permutations

Gaussian elimination for Ax = b (matrix notation)

We stick with the 4 × 4 example. Let

    
1 1 1
 l21 1   1   1
L1 =   L2 = 
′
 L3 = 
 l31 1   l32 1   1
l41 1 ′
l42 1 ′′
l43

Then, we executed the following

U = L−1 −1 −1
3 P3 L2 P2 L1 PA

which can be interpreted as

U = L−1 −1 −1 −1 −1 −1
3 (P3 L2 P3 )(P3 P2 L1 P2 P3 )(P3 P2 P1 )A

L3−1 (P3 L−1 −1 −1 −1 −1 −1

2 P3 )(P3 P2 L1 P2 P3 ) = L , (P3 P2 P1 ) = P
Algorithm: Solving Ax = y by LU decomposition

1. Compute LU factorization with column pivoting

LU = PA.

Then LUx = PAx = Pb.

2. Solve
Lc = Pb
by forward substitution.
3. Solve
Ux = c
by backward substitution.
Note: L is a unit lower triangular matrix, i.e., it has ones on the
diagonal.
Computing the determinant of A

Since
det(A) = det(P −1 ) det(L) det(U)
and
det(L) = 1,
and
det(P −1 ) = det(P) = (−1)V
where V is the number of row permutations in the Gaussian
elimination, we have
n
Y
det(A) = (−1)V uii
i=1

The uii are the diagonal elements of U.

Algorithm: LU factorization

1 for k=1:n-1
2 for i=k+1:n
3 l(i,k) = a(i,k)/a(k,k);
4 for j=k+1:n
5 a(i,j) = a(i,j) - l(i,k)*a(k,j);
6 end
7 end
8 end
Complexity: LU factorization

In step k of the LU factorization:

Divide ai,k for i = k + 1, ..., n by the pivot to get li,k
Update ai,j , for k + 1 ≤ i, j ≤ n by li,k ak,j .
The cost of the Gaussian elimination algorithm in terms of floating
point operations (flops) is thus

(n − 1) + (n − 2) + · · · + 1 + 2(n − 1)2 + (n − 2)2 + · · · + 12

1 1
= n(n − 1) + n(n − 1)(2n − 1)
2 3
2 3 1 2 1
= n − n − n
3 2 6
2
= n3 + O(n2 )
3
Algorithm: forward and Backward substitution

▶ The complexity of forward substitution is thus n2 + O(n) flops.

▶ The complexity of Backward substitution is thus n2 + O(n)
flops.
Exercise
Complexity: complete system solve

The three steps of Gaussian elimination costs thus

2 3 2
n + O(n2 ) + 2n2 + O(n) = n3 + O(n2 ) = O(n3 )
3 3

▶ Important: LU factorization costs O(n3 ) while the actual

solve costs only O(n2 ).
▶ Can solve economically for several right-hand sides! Costs are
O(n3 + n2 k) (not O(n3 k)) for k right-hand sides
Computing the inverse of a matrix

▶ We can compute A−1 by decomposing LU = PA once, and

then solving Ax = b for n right hand sides [e1 , e2 , ..., en ], the
columns of the identity matrix I .
▶ Cost(approx): 32 n3 + n(2n2 ) = 38 n3 .
▶ However, typically we will try to avoid computing the inverse
A−1 the need to compute it explicitly is rare.
▶ General rule: Don’t compute the explicit matrix inverse.
 Lecture 10
Pivoting of Matrices

Course Website
https://sites.google.com/view/kporwal/teaching/mtl107
Gaussian elimination with partial pivoting

To solve the linear system of equation A x = b we proceed as

follows:
1. Compute LU factorization with partial pivoting

LU = PA

Then LUx = PAx = Pb.

2. Solve Lc=Pb by forward substitution
3. Solve Ux=c by backward substitution
The cost is 32 n3 + O(n2 ) for the factorization and 2n2 + O(n) for
each forward and backward substitution, in total
2 3
n + O(n2 ) FLOPs
3
Need for pivoting

    
0 1 x1 4
Ax=b ⇐⇒ =
1 1 x2 7

x1 x2 1
0 1 4
1 1 7

Pivot a11 = 0. We exchange rows 1 and 2.

x1 x2 1
1 1 7
0 1 4
Need for pivoting (cont.)

Let’s consider the numerical example (5 decimal digits)

x1 x2 1
0.00035 1.2654 3.5267
1.2547 1.3182 6.8541

x1 x2 1
→ 0.00035 1.2654 3.5267 with
0 -4535.0 -12636

l21 = 1.2547/0.00035 ≈ 3584.9

u22 = 1.3182 − 3584.9 × 1.2654 ≈ 1.3182 − 4536.3 ≈ −4535.0

c2 = 6.8541 − 3584.9 × 3.5267 ≈ 6.8541 − 12643 ≈ −12636.
Need for pivoting (cont.)

Backsubstitution gives

x2 = −12636/(−4535.0) ≈ 2.7863

x1 = (3.5267 − 1.2654 × 2.7863)/0.00035 ≈ 2.5714

The exact solution (up to 5 decimal digits) is

x1 = 2.5354, x2 = 2.7863

We have cancellation in the computation of x1 ! The very small

pivot in the first elimination step induced large numbers u22 and
c2 . Information, e.g. a22 , got lost.
GE with partial pivoting (GEPP)

The most common strategy for pivoting is to search for the

maximal element in modulus: The index p of the pivot in the k-th
step of GE is determined by
   
 k−1   (k−1) 
apk  = max aik 
i≥k

If p > k then rows p and k are exchanged.

This strategy implies that |lik | ≤ 1
GE with partial pivoting (GEPP) (cont.)

Let’s consider the

x1 x2 1
1.2547 1.3182 6.8541
0.00035 1.2654 3.5267

x1 x2 1
→ 1.2547 1.3182 6.8541 with
0 1.2650 3.5248

l21 = 0.00027895

x2 = 2.7864
x1 = (6.8541−1.3182×2.7864)/1.2547 ≈ 3.1811/1.2547 ≈ 2.5353
There is a deviation from the exact solution in the last digit only.
GEPP stability

Question: Does the incorporation of a partial pivoting procedure

guarantee stability of the Gaussian elimination algorithm, in the
sense that roundoff errors do not get amplified by factors that
grow unboundedly as the matrix size n increases?
We have seen that |lik ≤ 1|. What about |uik |?
We would like to be sure that
 
 (k) 
gn (A) = max ai,j 
i,j,k

i.e.,the elements in U do not grow too fast in the course of GE.

Unfortunately, it is not possible in general to guarantee stability of
GEPP.
Wilkinson’s counter example


1,
 if i = j or j = n,
n
An = (aij )i,j=1 with −1 if i > j,

0 otherwise.


1 0 0 0 0 1
 
 −1
 1 0 0 0 1 

 −1 −1 1 0 0 1 
A6 =  
 −1
 −1 −1 1 0 1 

 −1 −1 −1 −1 1 1 
−1 −1 −1 −1 −1 1
Partial Pivoting do not trigger change of row and last column
grows.
Complete pivoting

In complete pivoting we look for (one of) the largest element in

modulus    
 (k−1)   (k−1) 
aqr  = max aij 
k≤i,j≤n

at each stage of GE and interchange both row q and column r with

row i and column j, respectively.
This strategy that does guarantee stability. However, searching in
each stage (n − k)2 elements makes it significantly more expensive
than partial pivoting.
Instances in which (scaled) partial pivoting really fails appear to be
extremely rare in practice → GEPP is the commonly used
approach.
When pivoting is not needed

Definition
A matrix A ∈ Rn×n is diagonally dominant, if
X
|aii | ≥ |aik | , i = 1, ..., n.
k̸=i

Theorem
If A is nonsingular and diagonally dominant then the LU
factorization can be computed without pivoting.
Proof

We show that after reduction of the first row, the reduced system
is again diagonally dominant. We have
(1) ai1 a1k
aik = aik − , i, k = 2, ..., n.
a11
For the diagonal elements we get the estimate
   
ai1 a1i 
 
 ai1 a1i 
 (1)  
aii  = aii − ≥ |aii | − 
 , i = 2, ..., n.
a11  a11 

We now want to show that

  n  
 (1)  X  (1) 
aii  ≥ aik  , i = 2, ..., n.
k=2,k̸=i
Proof (cont.)
Now for the sum of the moduli of the off-diagonal elements of row
i (i = 2, . . . , n) we get
n n  
X  
 (1)  X  a i1 a 1k 
aik  = aik − 
 a11 
k=2,k̸=i k=2,k̸=i
n   n
X  ai1  X
≤ |aik | +   |a1k |
a11 
k=2,k̸=i k=2,k̸=i
n ( n
 )
X  ai1  X
= |aik | − |ai1 | +   |a1k | − |a1i |
a11 
k=1,k̸=i k=2
 
 ai1 
≤ |aii | − |ai1 | +   {|a11 | − |a1i |}
a11 
 
 ai1 a1i   (1) 
= |aii | − 
  ≤ a  .
ii
a11