Tutorial 9

Iterative Methods
and Matrix Norms
 Iterative processes

In an iterative process, the k+1 step is defined via:

 0
u a
 k 1  k
u  Tu

Eigenvector decomposition dramatically simplifies the solution:

u  k   T k u  0   T k  c1 v1  ...  c n v n   c11k v1  ...  c n  n k v n (1)

The iterative process converges if:

1  i  n,  i  1  lim u  k   0
k 

2
 Spectral radius of a matrix

The iterative process u  k   T k u  0  converges if the spectral radius

of T is below 1, where spectral radius of a matrix is defined as:

(T)  max  k
1 k  n

The largest eigenvalue (the first after ordering) governs the dynamic
system, i.e.:
u  k   c1  k1 v1

Unfortunately, straightforward calculation of spectral radius is itself a

complicated task, and for large matrices is solved by ITERATIVE (!!)
process.
3
Vector Norms
v
A norm on the vector space is a mapping, R  R  , satisfying the
n 1

following axioms:

a. Positivity : v  0; v 0v 0
b. Linearity : cv  c v ;
c. Triangle inequality : vw  v  w ;

The popular example of the norm is the p-norm:

n

v
p
vp p
k 1 p  
k 1

and its three special cases, p=1: sum of abs. values, p=2: Euclidian
norm, and p=∞: the maximal absolute value element of the vector.
4
Norms Equivalence

For any two norms A and B, defined in a finite dimensional vector

space, there exist positive constants C≥c>0, such that:

v, c v B  v A  C v B (2)

Proof (outline):

Due to property (b) of norms, it is enough to show (2) on the unit

sphere, v A
 1. Due to property (c) the norm on this sphere is a
continuous function, thus it has the minimum m and maximum M, which
are both positive due to property (a). Taking C=1/m and c = 1/M
satisfies (2).

5
 Induced Matrix Norm

The induced matrix norm is defined via the vector norm A , by

considering the matrix as a linear operator in the vector space (isn’t that
the essence of what is matrix for ?!).

The induced norm of the matrix T is the maximum factor by which it can
multiply the norm ( = the ‘length’ ) of the input vector.
Tv
T  max A
 max T v A
A v v A
v A
1

The last equality is due to the fact that matrices are linear operators and
vector length can be reduced.

6
 The Norm Axioms are Satisfied

The properties of vector norms (p.4) are naturally extended to matrices.

Now let us prove that natural matrix norm satisfies these properties.

T A
 max T v A
v A
1

(a) Positivity: Since vector norm is non-negative, the maximum is taken

over non-negative numbers and therefore non-negative by itself: T A
0
Now, assume that matrix T has at least one non-zero element, tjk, and let
us show that in this case T A
0:

 . Then the vector w  T v

T
Consider the vector, v [ 01 ,...,0 ,1
k 1 k , 0 k 1...0 n ]
will have at least one non-zero element
7
 The Norm Axioms are Satisfied

n
w  T v  w j   t ji vi  t jk 1  t jk  w A 0
i 1

T A
 max Tu A
 Tv A
0
u A
1

(b) Linearity:
cT A
 max  cT  v A
 c max T v A
c T A
v A
1 v A
1

Follows from the fact that

 cT v   ct ij v j  c   t ij v j  c T v 
i 1..n; j1..m i 1..n ; j1..m

8
 Natural Matrix Norm

(c) Triangle inequality: Consider

AB A
 max  A  B  v A
 max A v  B v A
v A
1 v A
1

The right hand side is the norm of the sum of two vectors and therefore
must satisfy the triangle inequality for the vector norm:

max A v  B v
v 1 A
 max A v
v 1
 A
 Bv A

A A

searching for the maximums independently is a looser constraint – it can

yield the same vector v or two other vectors u and w in case they give
even greater maximum:
max A v
v 1
 A
 Bv A
  max Au
u 1 A
 max A w
w 1 A
 A  B
A A A

9
 Matrix norm for Convergence Estimation
The norms of matrices are a useful alternative to the spectral radius in
evaluation of matrix convergence due to the following

Statement: if some induced norm of A, A  1 , then A is convergent.

Proof: Apply A to any vector v, and get:

Av  A  v
k k
Continuing the iterative process we obtain: A v  A v

Since A  1 the process converges.

10
 Spectral radius is not a norm.
The matrix norm is a sufficient but not necessary condition for testing for
convergence: on the next slide we will see an example of the converging
matrix with Euclidian norm >1.

Question: Is the spectral radius of a matrix a valid norm? NO!

0 1 
Proof: Consider the matrix A   
 0 0 
Its both eigenvalues are 0, and therefore the spectral radius=0. However,
A≠0 and therefore its norm must be positive.

In fact, this matrix gives a convergence within one iteration (A2=0).

11
 Example – See the Gap
Statement: The Euclidian (p=2) norm of a matrix equals its maximum
singular value.

Proof: Done in class.

0.5 1 
Consider the matrix: A 
 0 0.25 

We have that A  1.5; A  1.25; and A  1.14

 1 2

and yet, the eigenvalues are 0.5 and 0.25, implying a

spectral radius of 0.5.

This matrix leads to convergence.

12
 Gerschgorin Circle Theorem

Let A be an n×n matrix. For each 1 ≤ i ≤ n, the Gerschgorin disk is

defined as  
 n

D i   z  a ii   a ij , z  C
 j1 
 j  i 
The Gerschgorin domain is defined as a union of all the Gerschgorin
disks: n
D A  D i
i 1

Gerschgorin Circle Theorem: All eigenvalues of Matrix A lie in its

Gerschgorin domain DA.

13
 Proof of Gerschgorin Circle Theorem
v
Let v be an eigenvector of A with an eigenvalue λ. Let u  be a
v 
corresponding “normalized” eigenvector.

Let ui be a maximal entry of u: ui = 1. Writing an i-th component of the

eigenvalue equation Au= λu, we obtain:
n n
 a iju j  u i   a iju j     a ii  u i
j1 j1, j i
Now, we obtain
n n n n
   a ii  u i    a ii   a iju j   a iju j   a ij  u j   a ij
 j1, j i j1, j i j1, j i j1, j i
u i 1
n
   a ii    a ij
j1, j i
14