Professional Documents
Culture Documents
CH07 4a
CH07 4a
Residual Vectors
SOR Method
Optimal
SOR Algorithm
Outline
Relaxation Techniques
2 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
Outline
Relaxation Techniques
2 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
Outline
Relaxation Techniques
2 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
Outline
Relaxation Techniques
2 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
Outline
Relaxation Techniques
3 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
Relaxation Techniques
4 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
Relaxation Techniques
4 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
Relaxation Techniques
4 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
Relaxation Techniques
4 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
r = b Ax
Relaxation Techniques
5 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
r = b Ax
Comments
A residual vector is associated with each calculation of an
approximate component to the solution vector.
Relaxation Techniques
5 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
r = b Ax
Comments
A residual vector is associated with each calculation of an
approximate component to the solution vector.
The true objective is to generate a sequence of approximations
that will cause the residual vectors to converge rapidly to zero.
Numerical Analysis (Chapter 7)
Relaxation Techniques
5 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
ri
(k )
(k )
(k )
Relaxation Techniques
6 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
ri
(k )
(k )
(k )
xi
(k )
(k )
(k )
(k 1)
= (x1 , x2 , . . . , xi1 , xi
Relaxation Techniques
(k 1) t
, . . . , xn
6 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
ri
(k )
(k )
(k )
xi
(k )
(k )
(k )
(k 1)
= (x1 , x2 , . . . , xi1 , xi
(k )
= bm
is
i1
X
(k )
amj xj
j=1
(k 1) t
, . . . , xn
Relaxation Techniques
n
X
(k 1)
amj xj
j=i
6 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
(k )
rmi
= bm
i1
X
(k )
amj xj
n
X
j=1
(k 1)
amj xj
j=i
= bm
i1
X
j=1
(k )
amj xj
n
X
(k 1)
amj xj
(k 1)
ami xi
j=i+1
for each m = 1, 2, . . . , n.
Relaxation Techniques
7 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
rmi = bm
i1
X
(k )
amj xj
j=1
n
X
(k 1)
amj xj
(k 1)
ami xi
j=i+1
rii
= bi
i1
X
j=1
(k )
aij xj
is
n
X
(k 1)
aij xj
(k 1)
aii xi
j=i+1
Relaxation Techniques
8 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
rmi = bm
i1
X
(k )
amj xj
j=1
n
X
(k 1)
amj xj
(k 1)
ami xi
j=i+1
= bi
rii
i1
X
(k )
aij xj
j=1
(k 1)
(k )
+ rii
n
X
(k 1)
aij xj
= bi
i1
X
(k )
aij xj
j=1
Numerical Analysis (Chapter 7)
(k 1)
aii xi
j=i+1
so
aii xi
is
Relaxation Techniques
n
X
(k 1)
aij xj
j=i+1
R L Burden & J D Faires
8 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
(k 1)
aii xi
(k )
rii
= bi
i1
X
(k )
aij xj
n
X
(k 1)
aij xj
j=i+1
j=1
i1
n
X
X
1
(k )
(k )
(k 1)
bi
xi =
aij xj
aij xj
aii
j=1
Relaxation Techniques
j=i+1
9 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
(k 1)
aii xi
(k )
rii
= bi
i1
X
(k )
aij xj
n
X
(k 1)
aij xj
j=i+1
j=1
i1
n
X
X
1
(k )
(k )
(k 1)
bi
xi =
aij xj
aij xj
aii
j=1
j=i+1
aii xi
Numerical Analysis (Chapter 7)
(k )
+ rii
(k )
= aii xi
Relaxation Techniques
9 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
(k 1)
aii xi
(k )
+ rii
(k )
= aii xi
(k )
xi
(k 1)
= xi
rii
aii
Relaxation Techniques
10 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
Relaxation Techniques
11 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
(k )
(k )
xi+1 = (x1 ,. . ., xi
(k 1)
(k 1) t
).
, xi+1 , . . ., xn
Relaxation Techniques
11 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
(k )
(k )
xi+1 = (x1 ,. . ., xi
(k 1)
(k 1) t
).
, xi+1 , . . ., xn
(k )
= bm
i1
X
(k )
amj xj
j=1
Relaxation Techniques
n
X
is
(k 1)
amj xj
j=i
11 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
(k )
rmi
= bm
i1
X
(k )
amj xj
n
X
(k 1)
amj xj
j=i
j=1
ri,i+1 = bi
i
X
j=1
(k )
aij xj
n
X
(k 1)
aij xj
j=i+1
Relaxation Techniques
12 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
(k )
rmi
= bm
i1
X
(k )
amj xj
n
X
(k 1)
amj xj
j=i
j=1
ri,i+1 = bi
i
X
(k )
aij xj
j=1
= bi
i1
X
j=1
n
X
(k 1)
aij xj
j=i+1
(k )
aij xj
n
X
(k 1)
aij xj
(k )
aii xi
j=i+1
Relaxation Techniques
12 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
= bi
i1
X
(k )
aij xj
j=1
n
X
(k 1)
aij xj
(k )
aii xi
j=i+1
i1
n
X
X
1
(k )
(k )
(k 1)
bi
xi =
aij xj
aij xj
aii
j=1
j=i+1
(k )
Relaxation Techniques
13 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
= bi
i1
X
(k )
aij xj
j=1
n
X
(k 1)
aij xj
(k )
aii xi
j=i+1
i1
n
X
X
1
(k )
(k )
(k 1)
bi
xi =
aij xj
aij xj
aii
j=1
j=i+1
(k )
Relaxation Techniques
13 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
Outline
Relaxation Techniques
14 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
Relaxation Techniques
15 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
(k )
xi
(k 1)
xi
r
+ ii
aii
Relaxation Techniques
15 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
(k )
xi
(k 1)
xi
r
+ ii
aii
to
(k )
(k )
xi
(k 1)
= xi
Relaxation Techniques
rii
aii
15 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
(k )
xi
(k 1)
xi
r
+ ii
aii
to
(k )
(k )
xi
(k 1)
= xi
rii
aii
Relaxation Techniques
15 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
(k )
xi
(k 1)
= xi
rii
aii
Relaxation Techniques
16 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
(k )
xi
(k 1)
= xi
rii
aii
Relaxation Techniques
16 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
(k )
xi
(k 1)
= xi
rii
aii
Relaxation Techniques
16 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
(k )
xi
(k 1)
= xi
rii
aii
Relaxation Techniques
16 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
(k )
xi
(k 1)
= xi
rii
aii
Relaxation Techniques
16 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
rii
= bi
i1
X
j=1
(k )
aij xj
n
X
in the form
(k 1)
aij xj
(k 1)
aii xi
j=i+1
Relaxation Techniques
17 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
rii
= bi
i1
X
(k )
aij xj
n
X
in the form
(k 1)
aij xj
(k 1)
aii xi
j=i+1
j=1
(k )
xi
(k 1)
xi
r
+ ii
aii
Relaxation Techniques
17 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
rii
= bi
i1
X
(k )
aij xj
n
X
in the form
(k 1)
aij xj
(k 1)
aii xi
j=i+1
j=1
(k )
xi
(k 1)
xi
r
+ ii
aii
xi
i1
n
X
X
(k 1)
(k )
(k 1)
bi
= (1 )xi
+
aij xj
aij xj
aii
j=1
Relaxation Techniques
j=i+1
17 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
i1
n
X
X
(k )
(k 1)
(k )
(k 1)
bi
aij xj
aij xj
xi = (1 )xi
+
aii
j=1
Relaxation Techniques
j=i+1
18 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
i1
n
X
X
(k )
(k 1)
(k )
(k 1)
bi
aij xj
aij xj
xi = (1 )xi
+
aii
j=1
j=i+1
as
(k )
aii xi
i1
X
(k )
aij xj
= (1
(k 1)
)aii xi
j=1
n
X
(k 1)
aij xj
+ bi
j=i+1
Relaxation Techniques
18 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
aii xi
i1
X
(k )
aij xj
(k 1)
= (1 )aii xi
n
X
(k 1)
aij xj
+ bi
j=i+1
j=1
Relaxation Techniques
19 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
aii xi
i1
X
(k )
aij xj
(k 1)
= (1 )aii xi
n
X
(k 1)
aij xj
+ bi
j=i+1
j=1
Relaxation Techniques
19 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
Relaxation Techniques
20 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
Relaxation Techniques
20 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
c = (D L)1 b
Relaxation Techniques
20 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
c = (D L)1 b
Relaxation Techniques
20 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
Example
The linear system Ax = b given by
4x1 + 3x2
= 24
3x1 + 4x2 x3 = 30
x2 + 4x3 = 24
has the solution (3, 4, 5)t .
Relaxation Techniques
21 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
Example
The linear system Ax = b given by
4x1 + 3x2
= 24
3x1 + 4x2 x3 = 30
x2 + 4x3 = 24
has the solution (3, 4, 5)t .
Compare the iterations from the Gauss-Seidel method and the
SOR method with = 1.25 using x(0) = (1, 1, 1)t for both
methods.
Relaxation Techniques
21 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
Relaxation Techniques
22 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
(k )
= 0.75x2
(k )
= 0.75x1 + 0.25x3
(k )
= 0.25x2 6
x1
x2
x3
+6
(k )
(k 1)
+ 7.5
(k )
Relaxation Techniques
22 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
(k )
= 0.75x2
(k )
= 0.75x1 + 0.25x3
(k )
= 0.25x2 6
x1
x2
x3
+6
(k 1)
(k )
+ 7.5
(k )
and the equations for the SOR method with = 1.25 are
(k )
x1
(k )
x2
(k )
x3
(k 1)
= 0.25x1
=
=
(k 1)
0.9375x2
+ 7.5
(k )
(k 1)
(k 1)
0.9375x1 0.25x2
+ 0.3125x3
(k )
(k 1)
0.3125x2 0.25x3
7.5
Relaxation Techniques
+ 9.375
22 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
(k )
x1
(k )
x2
(k )
x3
1
1
1
5.250000
3.812500
5.046875
3.1406250
3.8828125
5.0292969
3.0878906
3.9267578
5.0183105
Relaxation Techniques
7
3.0134110
3.9888241
5.0027940
23 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
(k )
x1
(k )
x2
(k )
x3
1
1
1
5.250000
3.812500
5.046875
3.1406250
3.8828125
5.0292969
3.0878906
3.9267578
5.0183105
7
3.0134110
3.9888241
5.0027940
(k )
x1
(k )
x2
(k )
x3
1
1
1
6.312500
3.5195313
6.6501465
2.6223145
3.9585266
4.6004238
3.1333027
4.0102646
5.0966863
Relaxation Techniques
7
3.0000498
4.0002586
5.0003486
23 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
Solution (3/3)
Relaxation Techniques
24 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
Solution (3/3)
For the iterates to be accurate to 7 decimal places,
the Gauss-Seidel method requires 34 iterations,
Relaxation Techniques
24 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
Solution (3/3)
For the iterates to be accurate to 7 decimal places,
the Gauss-Seidel method requires 34 iterations,
as opposed to 14 iterations for the SOR method with = 1.25.
Relaxation Techniques
24 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
Outline
Relaxation Techniques
25 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
Relaxation Techniques
26 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
Relaxation Techniques
26 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
Relaxation Techniques
27 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
Relaxation Techniques
27 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
Theorem (Ostrowski-Reich)
If A is a positive definite matrix and 0 < < 2, then the SOR method
converges for any choice of initial approximate vector x(0) .
Relaxation Techniques
27 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
Theorem (Ostrowski-Reich)
If A is a positive definite matrix and 0 < < 2, then the SOR method
converges for any choice of initial approximate vector x(0) .
The proof of this theorem can be found in Ortega, J. M., Numerical
Analysis; A Second Course, Academic Press, New York, 1972, 201 pp.
Numerical Analysis (Chapter 7)
Relaxation Techniques
27 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
=
1+
1 [(Tj )]2
Relaxation Techniques
28 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
=
1+
1 [(Tj )]2
Relaxation Techniques
28 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
=
1+
1 [(Tj )]2
Relaxation Techniques
28 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
Example
Find the optimal choice of for the SOR method for the matrix
4
3
0
4 1
A= 3
0 1
4
Relaxation Techniques
29 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
Solution (1/3)
This matrix is clearly tridiagonal, so we can apply the result in the
SOR theorem if we can also show that it is positive definite.
Relaxation Techniques
30 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
Solution (1/3)
This matrix is clearly tridiagonal, so we can apply the result in the
SOR theorem if we can also show that it is positive definite.
Because the matrix is symmetric, the theory tells us that it is
positive definite if and only if all its leading principle submatrices
has a positive determinant.
Relaxation Techniques
30 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
Solution (1/3)
This matrix is clearly tridiagonal, so we can apply the result in the
SOR theorem if we can also show that it is positive definite.
Because the matrix is symmetric, the theory tells us that it is
positive definite if and only if all its leading principle submatrices
has a positive determinant.
This is easily seen to be the case because
4 3
det(A) = 24, det
= 7 and
3 4
Relaxation Techniques
det ([4]) = 4
30 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
= D 1 (L + U)
Relaxation Techniques
31 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
= D 1 (L + U)
0 3 0
4 0 0
0 1
= 0 14 0 3
0
1 0
0 0 1
4
Relaxation Techniques
31 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
= D 1 (L + U)
0 3 0
4 0 0
0 1
= 0 14 0 3
0
1 0
0 0 14
0
0.75 0
0
0.25
= 0.75
0
0.25 0
Relaxation Techniques
31 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
so that
= D 1 (L + U)
0 3 0
4 0 0
0 1
= 0 14 0 3
0
1 0
0 0 14
0
0.75 0
0
0.25
= 0.75
0
0.25 0
0.75
0
0.25
Tj I = 0.75
0
0.25
Relaxation Techniques
31 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
Relaxation Techniques
32 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
Relaxation Techniques
= (2 0.625)
32 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
(Tj ) =
= (2 0.625)
0.625
Relaxation Techniques
32 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
(Tj ) =
and
=
= (2 0.625)
0.625
2
2
q
1.24.
=
1 + 1 0.625
1 + 1 [(Tj )]2
Relaxation Techniques
32 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
(Tj ) =
and
=
= (2 0.625)
0.625
2
2
q
1.24.
=
1 + 1 0.625
1 + 1 [(Tj )]2
This explains the rapid convergence obtained in the last example when
using = 1.25.
Numerical Analysis (Chapter 7)
Relaxation Techniques
32 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
Outline
Relaxation Techniques
33 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
Relaxation Techniques
34 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
Relaxation Techniques
34 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
Relaxation Techniques
35 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
i
P
1 h Pi1
j=1 aij xj nj=i+1 aij XOj + bi
aii
Relaxation Techniques
35 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
i
P
1 h Pi1
j=1 aij xj nj=i+1 aij XOj + bi
aii
Relaxation Techniques
35 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
i
P
1 h Pi1
j=1 aij xj nj=i+1 aij XOj + bi
aii
Relaxation Techniques
35 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
i
P
1 h Pi1
j=1 aij xj nj=i+1 aij XOj + bi
aii
Relaxation Techniques
35 / 36
Residual Vectors
SOR Method
Optimal
SOR Algorithm
i
P
1 h Pi1
j=1 aij xj nj=i+1 aij XOj + bi
aii
Relaxation Techniques
35 / 36
Questions?