Lecture 13

Iterative Methods for Linear Systems

Course Website
https://sites.google.com/view/kporwal/teaching/mtl107
Iterative Methods for Linear Systems

I Learn simple and e↵ective iterative methods for problems

where direct methods are ine↵ective.
I Analyze methods to see when and where they can be applied
and how e↵ective they are.
I Understand modern algorithms, specifically preconditioned
conjugate gradients
References
I U. Ascher and C. Greif: Numerical methods. Chapter 7.
I Y. Saad: Iterative Methods for Sparse Linear Systems. SIAM,
2003. 2nd ed.
I Barrett et al.: Templates for the Solution of Linear Systems.
SIAM, 1994. Online at http://www.netlib.org/linalg/
html_templates/Templates.html
Why not use a direct method? €"
-
ou : e- A E-

d÷
-

" 10-0*00

104×104
* dit
Ax=b, A nonsingular n ⇥ n.
fun:c Job

I
Iterative method: Starting from initial guess x0 , generate iterates
x 1 , x 2 , ..., x k , ..., hopefully converging to solution x.
But why not simply use LU decomposition, or
x=A \b in Matlab ?
The matrix size n must be large, and the matrix A must be
somehow special to consider iterative methods. "
10
A is very Big
=) LU is
large
Ocn )
> → n

expensive
.
?⃝
Comparison of direct and iterative linear solvers
'

Ji AI
Direct solvers
box
1. Computation is numerically stable in many relevant cases.
2. Can solve economically for several right-hand sides.
3. Accuracy can be improved via ‘iterative refinement.’
4. ‘Essentially’ a black box. =
cannot .

5. But: fill-in limits usefulness (memory, flops).

Iterative solvers
1. Only a rough approximation to x is required. ¥-5
2. A good x0 approximating x is known (warm start)
3. Matrix often only implicitly needed via matvec product.
4. Good preconditioner often necessary for convergence.
5. Quality often dependent on ‘right’ choice of parameters, e.g.
start vector, basis size, restart (see that later).
Typical scenarios

Direct solvers
1. Inverse Iteration
2. Determinants
3. Many linear systems with the same matrix A
4. ‘Difficult’ applications (e.g. circuit simulation)
Iterative solvers
1. Inexact Newton-Methods
2. Many linear systems with ‘slightly changing’ matrices
3. Matrix-free applications (e.g. matvec product via FFT)
4. Very large problems
Stationary iterative methods: Motivation

Let A be so large that it is impossible to compute its LU

factorization (fill-in). Then, we try to solve Ax = b iteratively.
Let xk be an approximation to the solution x⇤ of Ax = b. Then

x⇤ = xk + e k

ek is the error.

rk is the residual.
Aek = Ax⇤
D
Axk = b Axk = rk
r
.

②
x⇤ = xk + A
r 1
k
I'm
Of course, the above assumption prevents us from solving this for
the error ek .
Stationary iterative methods: Motivation (cont.)

Let M be an invertible matrix with the following properties

1. M is a good approximation to A.
2. Mz = r can be solved (relatively) cheaply.
3. M can be formed (relatively) cheaply.
Then, instead of solving (1) we iterate according to

rk = b Axk
1
xk+1 = xk + M rk
M is called a preconditioner.
Stationary iterative methods: Motivation (cont.)

Procedure:

I
rk = b Axk Compute residual
Mzk = rk Solve with preconditioner
xk+1 = xk + zk Update approximate
Remarks:
I In step 1, we have to multiply a matrix with a vector. In
practice: we need to have a procedure that computes y Ax.
I In step 2, we have to solve a system of equations with the
preconditioner which is “by definition” relatively easy to do.
This step is usually the most expensive one.
Stationary iterative methods: Theory

Definition
Let A be nonsingular. Then A = M -N, with M nonsingular, is
called a matrix splitting.
We consider the iteration M
- '
Hsiu't
014+1 :

m
-
'
b-
1
xk+1 = xk + M rk () Mxk+1 = Nxk + b.
"
m ( ra )
-
'
=
m-i-b-A.si , )
Note:
I If M 1 = A 1 , then one-step convergence: xk+1 = A 1 b.
I In practice, M 1 should be a good (but cheaper to invert)
approximation to A 1 (‘preconditioner’).
Formulation as fixed-point iteration

1 1
xk+1 = xk + M rk = M (Mxk + rk )
•

1
=M ((M
-
A)xk + b)
1 1
=M Nxk + M b
=) xk+1 = G xk + c fixed point iteration
G =M 1N =I M 1A is called the iteration matrix.
Theorem
The error satisfies ek+1 = G ek . si
"
: Gsi # c

Exercise
 =-D
>

Theorem on convergence Aoi

M N
A =
-

Theorem
Let A = M-N be a matrix splitting with M invertible. Then,
1. The iteration
Mxk+1 = Nxk + b
converges for any x(0) if and only if
1 1
⇢(G ) < 1, G =M N=I M A.

Here ⇢(G ) = max{| | : is an eigenvalue of G} is the

spectral radius of G.
2. If, for any vector norm, kG k < 1, then iteration converges.

See: Numerical Analysis (9th Ed.), Richard L. Burden, J. Douglas

Faires, 2011
 5in ,
= Goin t Gb→
If converge
3- Ji s t
unique
-
-

I
'
=
GI
>
+
c. I
( GB )
"
=
'
I' =
CI
-

G)

Lem_ma_ If SCG )< 1 ,

then
'
CF -

G)
-

exists and
"

G) Gta 't
-

4- -

= It - -

=
Eai
j=o
Proof SCG )< 1 => all the
=

eigenvalues are
strictly less than

one
-

GÉ= Jd .

can not non -

zero
There
solution this
only solution
.

'

to Gsi
>
=
I
'

(I 9) -

I :O
"
can not have
non -

zero Sokka
-1
=s
(I -

G) exists
 BEIT .

If SCGJCI ⇐
It Ghai =o or It G "=o
k→•
ue¥tw
"→
that ..

Sn= It at -
gh

( I -

G) Sn= I -
gmhhti

Gh )
"
It (I -
G) Sn=1tfI -

n→D not

It Sn I It anti
G)
-

(
= ' =
I -

n→a
- so
n

=
I
-1
→
It 5h G)
=
( I -

n→o
 III. Assume SCGKL
4=-93-1 exists .

Inti : Cs Jiu
I
t
GÑW=c→
=
GCGI'm t
-
E)
'
E
=

Garsia ,
+ ai *

'

( Ghi Gh F
:
=

Guido F)
-

+ - -

becase SCGKL -10 E

•

(Ef )É
"
It I'm =

n→8
"

=
(I -
G) E

converse Assume 5in converges

satisfy
'

I which
I. =a +É
then SCGKI -

>
⇐ It Ghz =D
6*8
Consider É and define 5io= É -
É

Éu+ I'm )
'

É -

,
: GCI -

Jio )
'

=
GCI -

=Gz
'

It CI -

Jinn )=0
k→D >

It Gh I =D

4-so

= ,
g( G) 21

"
It 119 I → 0

m¥
Convergence rate

I Earlier we derived a convergence rate for fixed point iterations

xk+1 = g (xk ) for solving nonlinear systems.
I Here, the situation is similar:xk+1 = g (xk ) = G xk + c.
I How many iteration steps are required to reduce the error
norm by an order of magnitude?

0.1 ke0 k = ⇢(G )k ke0 k

1
k⇡
log10 ⇢(G )
1
rate = ⇡ log10 ⇢(G )
k
The rate is higher (faster convergence), the smaller ⇢.
Remarks

I (1) in the convergence theorem is based entirely on the

eigenvalues of the iteration matrix G .
I There is however a big di↵erence of the convergence behavior
of normal (diagonalizable by a unitary matrix) and nonnormal
matrices: Compare in Matlab the ✓ behaviour ◆
of the error
0.9 0
norm kek k with the two matrices ,
0 0.9
✓ ◆
0.9 10
.
0 0.9
I Pay attention on the norm! e.g.,
✓ ◆
0.1 0.4
G=
0.4 0.8

Here, kG k1 = 1.2, kG k2 = maxi | i (G )| = 0.9815.

Practicalities

Starting vector, iterates and convergence

Improve initial vector x(0) so that xk ! x⇤ = A 1 b in fewer steps.
Where to get x(0) from?
There are various possibilities (some better than others):
I Zero/random vector.
I Insights into underlying problem.
I Solution from a ‘similar’ previously-solved problem.
Stopping criterion
A few practical possibilities:
I krk k  ⇠ kbk
I krk k  ⇠ r(0)
Practical schemes: Jacobi and Gauss–Seidel iteration
In -5

Let A = L + D + U where D is diagonal, L is strictly lower

-0

T¥
triangular and U is strictly upper triangular. Let E = D + L.
0 1 0 1 0
7 3 1 7 0 0 7 0
A= @ 3 10 2 A =) D = @ 0 10 0 A ,E = @ 3 10
1 7 15 0 0 15 1 7
1
Jacobi iteration: xk+1 = xk + D 1 rk -
Gauss-Siedel iteration: xk+1 = xk + E 1 rk

I >
Fun ( Asia
'
5in
-

+ D-
=

D-
'
I
Is)
¥¥É☒t¥t
-

'
D- ca
-
-
Jacob
5in Ñ
'

u+ , = -1 D-

f ri = 5- A#u

= 5in + D-
'

( 5 -
CD.IL.tv )Ñu )

¥u
D÷b D- ( Ltu )Ñu
'
=
+ -

Ñhti ( Ltu )Ñu

' '
=
D- 5 -
D-
__
Gaus Seidel -

Fi
'
-

Eun = Éu 1- ( Dtl)

=
5in 1- ④ + c)
"

( I -
AÑÉU )
5in +
(☐t_É ) ( D¥L5YDtLt§
=
-

sin
5in
cDtHÑu
=
Jacobi example

÷
7x1 = 3 10x2 x3 ,
Ten
10x
-
2 = 4 + 3x1 2x3 ,
.
15x3 = 2 x1 7x2

Evaluate right hand side at current iterate k and left hand side as
unknown at step k + 1:

(k+1) 1 (k) (k)

x1 = (3 10x2 x3 )
7
(k+1) 1 (k) (k)
x2 = (4 + 3x1 2x3 )
10
(k+1) 1 (k) (k)
x3 = (2 x1 7x2 )
15
Gauss–Seidel example

00
7x1 = 3 10x2 x3 , (0.1)
10x2 3x1 = 4 2x3 , (0.2)
15x3 + x1 + 7x2 = 2 (0.3)

Evaluate right hand side at current iterate k and left hand side as
unknown at step k + 1:

(k+1) 1 (k) (k)

x1 = (3 10x2 x3 ) (0.4)
7

8-
(k+1) 1 (k+1) (k)
x2 = (4 + 3x1 2x3 ) (0.5)
10
(k+1) 1 (k+1) (k+2)
x3 = (2 x1 7x2 ) (0.6)
15
(0.7)

Always use most recent information.

Properties of Jacobi and Gauss-Seidel relaxation

I Jacobi more easily parallelized

I Jacobi matrix M is symmetric
I GS converges whenever Jacobi converges and often (but not
always) twice as fast
I Both methods are simple but slow. Used as building blocks for
faster, more complex methods.
Thin If It is strictly diagonally dominant
then for any choice of ¥0 )
both Jacobi and Gauss -
Seidel

converge to unique solution at

A5i=b .

Thin Stein Rosenberg

.
-

If aij < 0
i=fj and aii so

then only one of the following

statement hold :

"

"←"
"" "
"
Iii ) r CSCG ;) <
Slag )
(iii ) SIG ; )=fCGg ) = 0

Civ ) SCG ;) SC Gg ) I
-

: -