ENUME 2024 - Lecture Notes

Roman Z.
Morawski
phone: (22) 234-7721, e-mail: roman.morawski@pw.edu.pl
Numerical Methods (ENUME)

1. INTRODUCTION
Lecture notes for Spring Semester 2023/2024
R. Z. Morawski: Numerical methods (ENUME) – 1. Introduction 1-1

Warsaw University of Technology, Faculty of Electronics and Information Technology
Educational objectives
 The programme of study, leading to the B.Sc. degree in Information and Communications
Technology, is aimed at providing graduates with knowledge and skills specific of information
technology, control and robotics, electronics and telecommunications.
 Classes in the ENUME course are intended to contribute to achieving the above goal by enabling
the students of Information and Communications Technology to acquire the knowledge and skills
necessary for:
 independently solving elementary numerical problems,
 commissioning the solution of more complex numerical tasks to experts of numerical
methods, and
 evaluating the solutions they provide.

Lectures
Contents:
Analysis of numerical problems and algorithms
Solving linear algebraic equations
Solving nonlinear algebraic equations
Approximation of functions
Solving ordinary differential equations

Lectures
Schedule:
Date Lectures Tests
February 22 2h
February 29 2h
March 7 2h
March 14 2h
March 21 2h
March 28 2h
April 4 1h 1 h (Test#1)
April 11 2h
April 18 2h
April 25 2h
May 9 2h
May 16 2h
May 23 2h
June 6 1h 1 h (Test#2)
June 13 2 h (Tests#1'&2')
Total 26 h 4h
Note: Test #1' is fully equivalent to Test #1, and Tests #2' is fully equivalent to Test #2; all the ENUME students are eligible to take all the tests.

Lectures
Teaching materials:
R. Z. Morawski, Lecture notes for ENUME students, Spring 2024
R. Z. Morawski, A. Miękina: Solved problems in
numerical methods, Oficyna Wydawnicza PW,
Warszawa 2021
Electronic textbooks on numerical methods available
in the Main Library of Warsaw University of
Technology (examples on the next slide)

Lectures
Addresses of selected e-books on numerical methods, available in the Main Library of WUT:
https://primo-48tuw.hosted.exlibrisgroup.com/permalink/f/1nmcehm/48TUW_ejournals3710000000360322
https://primo-48tuw.hosted.exlibrisgroup.com/permalink/f/1nmcehm/48TUW_WTU01000196082

Project classes
Tutors:
S. Kruszewski, e-mail: szymon.kruszewski @pw.edu.pl
P. Mazurek, e-mail: pawel.mazurek@pw.edu.pl
J. Wagner, e-mail: jakub.wagner@pw.edu.pl
Teaching materials:
P. Mazurek & J. Wagner, Introduction to MATLAB (3 parts)
J. Wagner, Guidelines for report preparation
J. Wagner, GitLab guidelines
Assignments:
A. Accuracy of computation ca. 8 h of homework
B. Approximation of functions ca. 10 h of homework
C. Ordinary differential equations ca. 12 h of homework

Grading
Partial grading
Project assignment A: accuracy of computation 15 pts
Project assignment B: approximation of functions 15 pts
Project assignment C: ordinary differential equations 20 pts
Test #1 20 pts
Test #2 30 pts
100 pts
Final grading
precondition: ≥ 25 pts from project assignments & ≥ 25 pts from tests
0 ≤∑pts < 50  2
50 ≤∑pts < 60  3
60 ≤∑pts < 70  3.5
70 ≤∑pts < 80  4
80 ≤∑pts < 90  4.5
90 ≤∑pts ≤100  5

1.1. Mathematical modelling in engineering practice
1.2. Vectors and matrices – recapitulation of linear algebra
Basic definitions
 x1 
x 
x   2    x1 x2  xN  – N -dimensional vector
T
 
 
 xN 
 a1, N   a1,1  aM ,1 
T
 a1,1 a1,2 a2,1
a a2,2  a2, N   a1,2 a2,2  aM ,2 
A  2,1   – M  N -dimensional matrix
           
   
 aM ,1 aM ,2  aM , N   a1, N a2, N  aM , N 

The most important norms of vectors:
N

2
x2 xn – the Euclidean norm
n 1
x 
 sup  xn n  1, ..., N  – the maximum norm
The most important norms of matrices:
A 2  sr  AT A  – the norm induced by x 2
where sr    – the operator of spectra radius determination

N 
A 
 sup  an , n  1, ..., N  – the norm induced by x 
 1 
A useful inequality:
Ax p
 A p x p
for p  2, 

Spectral characteristics of matrices
Definitions:
 the eigenvalue ( n ) and eigenvector ( v n ) of a square ( N  N ) non-singular matrix A :
A  v n  n  v n and v n 2
 1 for n  1, ..., N
 the spectral radius of A :
sr  A   sup  1 , ..., N 
Properties:
 For any matrix A , the eigenvectors ( v n ) are linearly independent, and:
A  V  V  Λ  A  V  Λ  V 1
where: V   v1 ... v N  and Λ  diag 1 , ..., N 
 For a symmetrical real-valued matrix A , the eigenvectors ( v n ) are orthonormal, and – consequently
– the matrix V is unitary, i.e.:
1
V   v1 ... v N   VT
1
which implies: A  V  Λ  V .
T

1.3. Numerical algorithm (NA) and its description
R
Numerical problem: D   W:
D  vectors of data : d1, d2 , ... 
W  vectors of results: w1 , w 2 , ... 
R – requirements
N
Example: y  PN  x; a    an x n
n0
d   xˆ a0 a1 ... a N 
T
vectors of data:
vectors of results: w   ŷ 
requirements: R :yˆ  PN  xˆ; a  .

Numerical algorithm = an ordered sequence of operations, transforming an element of D in an element
of W satisfying the requirements R .
Forms of NA description:
 (traditional) mathematical notation
 programming languages
 flow diagrams
 sequential notation

Example: The Horner's algorithm for computing the value of a polynomial:
yN  aN
yn -1  yn xˆ  an 1 for n  N , N  1, ..., 1
yˆ =y0
 the C language:
y  a  N  ; for ( n  N ; n  0; n  )  y  y  x  a  n  1 ;
 the flow diagram:  the sequential notation:
 xˆ   xˆ 
yˆ  a N a   a   xˆ 
 0   0   a0   xˆ 
 a1   a1   
 a   yˆ  W
nN 

  D  

   a1    
 0  
1 ŷ        yˆ 
1  aN 1   aN 1   
     yˆ 
a
 N  ˆ
y 
yˆ  yˆ  xˆ  a n 1

Roman Z. Morawski

2. ACCURACY AND COMPLEXITY OF COMPUTING
R. Z. Morawski: Numerical methods (ENUME) – 2. Accuracy and complexity of computing 2-1

2.1. Effects of the finite representation of numbers in computers
Example: Subtracting numbers:
369.711 ( 0.0005) – relative error 1.3  104 %

369.702 ( 0.0005) – relative error 1.3  104 %
0.009 ( 0.0010) – relative error 11 %
Example: Computing zeros of the polynomial:

20
y  x  a19 x  ...  a1 x  a0    x  n 
20 19
n 1
with the coefficient a19 disturbed at the level 6  106 %  complex zeros, e.g. 13.99  j 2.51.

2.2. Floating-point representation of numbers in computers
The general form of the floating-point representation:

x   m  10c
m  0.m1m2 ...mL 1mL mL 1..., where mi  0, ..., 9 , m1  0
The rounding principle:

x   m  10c
0.m1m2 ...mL if 0  mL 1  5
m   L
0.
 1 2m m ...m L  10 if 5  mL 1  9
The limit relative error:

x  x m  10c  m  10c sup m  m 5  10  L 1
  x       5  10 L
 eps
x m  10 c
inf m 10 1
The convenient representation of disturbed data:

x  x
  x      x  x 1    , where    eps,  eps 
x

2.3. The model of error propagation
The numerical algorithm for solving the numerical problem,  : D  W , has the form:
0   1    K 1  
 vK   w
1 2 K
dv  v  ...  v 
where:     K     K 1  ...    2    1
~ 
 d
1  ~v 1  2   ~v2  K   w
~
T  T  T 
η(1) η( 2) η( K )
 
where:   v ( k )   T     v ( k 1)   η  for k  1, ..., K
k k
After linearisation:
  v ( k )   T ( k )    v ( k 1)   η k  for k  1, ..., K
with T 
k
being a matrix of the coefficients of relative errors propagation

Propagation of error across a scalar function of a scalar variable
y    x
y    x   y  y    x  x 
where y , x – absolute errors corrupting y and x , respectively
x 1 2  x 
2
y    x      x  x  12    x  x     y  x   x   2 x    x     

2
x  x 
y  y  y  x   x    12 x 2   x   2   , where   x x
y x x 2
  y       x    2    x   2  
1
y y y
dy
y x dy d ln  y 
  y   T  , where T 
2
  1       
dx y dx x  x
d ln  x  x  x
x x  x

Propagation of error across a scalar function of a vector variable
N
 : y    x1 , x2 ,..., xN     y    Tn n
n 1
xn  y  ln  y 
where:  n    xn  and Tn  
y  xn xn  xn
 ln  xn  x  x
n n
Example:  n   xn for n  1, 2, ...

x1 x2 x1 x2
  x1  x2   1   2    x1  x2   1   2  
x1  x2 x1  x2 x1  x2 x1  x2 x1  x2
1   2  0
  x1a x2b   a 1  b 2 for a , b  eps 1

  x1x   x2 1  x2 ln  x1   2
2
1
 ln  x    
ln  x 

Basic rules of "epsilon" calculus
1   1 1   2   1   1   2
1     1  a for a  eps 1
a
ln 1     
e1  1    e
Example: Two methods for computing y  x12  x22 :

 x1   v1  x1 
2
A1 :     2
  y  v1  v2 
 x2   v2  x2 
 x  v  x  x 
A2 :  1    1 1 2    y  v1v2 
 x2   v2  x1  x2 

 x1   v1  x1 
2
A1 :     2
  y  v1  v2 
x
 2   v2  x2 

y1   x1 1  1   1  1    x2 1   2   1   2  1  3 
2 2

y1   x12 1  21 1  1   x22 1  2 2 1  2  1  3 
y1   x12 1  21  1   x22 1  2 2   2  1  3 
y1   x
2
1 
 x22    x12  21  1   x22  2 2  2   1  3 
  x12 x22    x12 x22  

y1  y 1    21  1    2 2  2    1  3   y 1    21  1    2 2  2    3 
  y y    y y  
x12 x22 x12 x22
  y1   T11  T2 2  K11  K 2 2  K 33 , where: T1  2 , T2  2 , K1  , K 2   , K3  1
y y y y
  y1   T1 eps  T2 eps  K1 eps  K 2 eps  K3 eps
x12  x22  x12  x22 
  T1  T2  eps   K1  K 2  K 3  eps  2 eps    1 eps
y  
y 

 x1   v1  x1  x2 
A2 :    
x v  x  x    y  v1v2 
 2  2 1 2

  y 2   T1 eps  T2 eps  K s eps  Ko eps  K m eps   T1  T2  eps   K s  K o  K m  eps

x12  x22
  y 2  2 eps  3
eps
y
   GRG 2
O
 GR
The above assessment differs from that derived for A1 :
x12  x22  x12  x22 

  y1   2 eps    1 eps
y y
    
O
 GR G
 GR 1
only by the component generated by rounding errors:

1 x
 1  3   GR
G
1   GR 2
G
3 x2
Notice: The dots indicating the exact values of variables (e.g. x , y , …) are usually omitted, if their absence does not
imply ambiguity.
2.4. Propagation of errors in the data
The model of the propagation of error across an elementary operation:

  v ( k )   T ( k )    v ( k 1)   η k 
under an assumption that η   0 for k  1, ..., K , enables one to evaluate the component of the error
k
in the final result of computation, caused by the errors in the data:

 O  y   T K   T K 1   T1  ε  T  ε , where T  T K   T K 1    T1
If the small relative changes of the data d cause large relative changes of the result w , then the
numerical problem is called ill-conditioned.
The conditioning of the numerical problem (and consequently  O  y  ) is characterised by the matrix T
which does not depend on the numerical algorithm used for solving this problem.

Example: Solving quadratic equation:
y 2  2 x1 y  x2  0 for x1  0, x2  0
y1  x1  x12  x2 , y2  x1  x12  x2
x1  y1 x1 y1 x1
T1,1        0,1
y1  x1 y1 y1  x1 x  x2 2
1
x2  y1 x2 1 1 x1   1
T1,2      1     0, 
y1  x2 y1 2  x1  y1  2  x1  x2   2 
2

x y x y2  x1
T2,1  1  2  1     1, 0 
y2  x1 y2 y2  x1 x1  x2
2
x2  y2 x2 1 1 x1  1 
T2,2     1     ,1
y2  x2 y2 2  x1  y2  2  x1  x2   2 
2

Ti , j  1 for i, j  1, 2  very good numerical conditioning.

The conditioning of the numerical problem may be characterised by the so-called condition number,
d
i.e. the worst-case amplification of the aggregated relative error in the data :
d
  
 w 
 w  w d
cond(d)  sup  d   cond  d  
 d

 
w d
 d 
 
where: d  d  d and w  w
 w


2.5. Propagation of rounding errors
The model of the propagation of error across an elementary operation:

  v ( k )   T ( k )    v ( k 1)   η k 
under an assumption that d   0 , enables one to evaluate the component of the error in the result of
 , caused by the rounding errors:
computation w
K
 G
 w    K  k  η k 
k 1
where K    T 
K
k
 T  K 1    T  k 1 for k  1,..., K  1; K  K   I
1 1 1
Example: Two algorithms for computing y    for x  0 :
x 1  x x  1  x 
 v1  1 / x 
A1 :  x   v  1 / 1  x    y  v1  v2 
 2  
 v1  x   1 
A2 :  x   v  1  x    y  
 2   v1v2 

 v1  1 / x 
A1 :  x       y  v1  v2 
v
 2  1 / 1  x  
1 1 
y1   1  1   1  2   1  o 
x 1  x  1  s  
1
1  2   s  1  o 
1
y1   1  1  
x 1 x 
 1 1  1 1 
y1    
  1    2   s   1   o 

 x 1  x   x 1  x 
 1 1 
y1  y 1  1   2 s  1  o 
  
 xy 1  x  y 
 G  y1   1  x 1  x2  x s  o   G  y1   1  x eps  x eps  x eps  eps   3 x  2  eps
v  x   1 
A2 :  x    1 
  y 
 v2  1  x   v1v2 
1  d
y 2   y 1  d  s  m    G  y 2   3eps
x 1  x  1   s  1  m 
x  13  sup  G  y1   sup  G  y 2  .
Numerical correctness of a numerical algorithm
An algorithm A  d  for d  D is numerically correct if:

 Interpretation #1: when applied to the exact data, it provides the results being the exact solutions of
the numerical problem for slightly disturbed data.
 Interpretation #2: the contribution of rounding errors to the total error in the result of computation
w remains in a "practically acceptable" relationship to the contribution of the errors in the data d .

Example: An algorithm for computing y  x12  x22 is numerically correct because:
y   x12 1  1   x22 1   2   1   s 
y   x12 1  1   s   x22 1  2   s  
y   x1 1  12 1  12  s     x2 1  12  2  12  s  
2 2
Example: The "classical" algorithm for solving quadratic equations:

 x1   x1   y1  x1  v 
x      for x1  0, x2  0
 2  v  x1  x2   y2  x1  v 
2
is numerically unstable because of the subtracting close numbers when x2  0 :
x12  x2  y2 x12  x2
K 2,     
y2   x  x2
2
1  x1  x  x22
1
x2 0

2.6. Using computers for the analysis of computational accuracy
The most important techniques for evaluation of the coefficients of error propagation:
 "epsilon" calculus
 symbolic differentiation
 numerical differentiation
 statistical simulation

2.7. Non-numerical applications of the methods for accuracy analysis
Implementation of numerical algorithms in floating-point processors:

 estimation of limit errors
 estimation of the minimum number of digits of mantissa, sufficient for providing required accuracy
 estimation of admissible level of measurement errors to guarantee the required accuracy of
computation using measurement data
 selection of algorithms providing most accurate results of computation
 correction of numerically unstable algorithms
Other engineering tasks:

 accuracy analysis of measuring devices and systems
 accuracy analysis of electronic circuits such as filters, bridges, converters
 sensitivity analysis of electronic circuits
 "small-signal" analysis of electronic circuits

Example: Estimate the sensitivity of the cut-off frequency 3dB of the filter:
R1
u1(t) R2 C2 u2(t)
to the scattering of the parameters of its elements:

R1  R1 1  1  , R2  R2 1  2  , C 2  C2 1   2  , where 1 ,  2 ,  2  1%
U2  s 1
K s  
U1  s   R1C2 
1  R1 R2   1  s 
 1  R1 R2 
R1 1  1 
1
1  R1 R2 
1  R1 R2  R2 1   2 
3dB    3dB  
R1C2 R1C 2 R1 1  1  C2 1   2 

 R1 
 3dB  3dB  1   1   2   1   2 
 R1  R2 
R1 R R
  3dB    1   2   1   2   2 1  1  2   2
R1  R2 R1  R2 R1  R2
R2 R1  2 R2  R  3R2
 3dB  
 1  2   2    1   1%  1  1%
R1  R2 R1  R2 R
 1  R2  R1  R2

2.8. Numerical complexity
f NA  N  – the dependence of the number of operations on the dimension of the problem N
CLASS OF EXEMPLARY COMPUTING TIME

CLASS OF ALGORITHMS
f NA  N  N  10 N  60
polynomial (effective) algorithms
ON  0.0001 s 0.0006 s
K
ON3 0.001 s 0.216 s
 K ,  k  N : f NA  N     k N k
k 0 ON5 0.1 s 780 s

exponential (ineffective) algorithms O  2N  0.001 s 336 600 years
K
 K ,  k  N 0 N  N 0 : f NA  N     k N k
k 0
O  3N  0.059 s 1.3  1015 years
Engineering measures of numerical complexity:

 computing time measured on a reference computer
 size of memory occupied in a reference computer

Roman Z. Morawski

3. SOLVING LINEAR ALGEBRAIC EQUATIONS
R. Z. Morawski: Numerical methods (ENUME) – 3. Solving linear algebraic equations 3-1

3.1. Formulation and numerical conditioning of the problem
The system of linear algebraic equations to be solved has the form:

Ax  b
where:
 a1,1 a1,2 ... a1, N   b1 
a a2,2 ... a2, N  b 
 2,1   2
A   a3,1 a3,2 .. a3, N  , det  A   0 , and b   b3  , with an ,m , bn R
   
       
 aN ,1 aN ,2 ... aN , N  bN 

The propagation of errors in the data b :
A  (x  x)  b  b  x  A1  b   x  A  1  b
1 1
b  A x  b  A  x   A 
x b
x b b
 A 1  A   cond( A) 
x b b
where: cond( A) is a condition number of the matrix A defined as follows:
cond( A)  cond p ( A)  A p  A 1 for p  2 or p  
p
The propagation of errors in the data b and A :

A
cond( A) 
x A b
  cond( A) 
x A b
1  cond( A) 
A

Example: The condition number of the Hilbert's matrix:
1
am , n  m, n  1,..., N
m  n 1
N 2 5 10 50
cond ( A) 19.28 4.77 105 1.60 1013 8.51 1019
General classification of the methods for solving A  x  b :

 finite (non-iterative) methods
 iterative methods

3.2. Finite methods for solving A  x  b
Solving a system with an upper-triangular matrix
a1,1 x1  a1,2 x2    a1, N 1 xN 1  a1, N xN  b1

a2,2 x2    a2, N 1 xN 1  a2, N xN  b2
  
aN 1, N 1 xN 1  aN 1, N xN  bN 1
aN , N xN  bN
The algorithm:
 N

bN  n  n ,  
b  a x
xN  and xn     n 1  for n  N  1,,1
aN , N an , n
The number of operations:

L  ,    12 N 2  12 N  O  12 N 2  and L *, /   12 N 2  12 N  O  12 N 2 

Method of Gauss elimination
The two-phase algorithm:

 the phase of elimination: a sequence of linear transformations leading to the conversion
of A  x  b into a system with an upper-triangular matrix having the same solution
 the phase of solving the system with an upper-triangular matrix
1 1
a1,1 x1  a1,2 x2    a1,1N xN  b11 
1 1

a2,1 x1  a2,2 x2    a2,1N xN 1
 b2 
 A   x  b 
1 1

         
aN1,1 x1  aN1,2 x2    aN1, N xN  bN1 
where: A   A and b   b
1 1

Elimination, step #1: zeroing of the elements number 2, ..., N in the first column of A 
1
1
by elimination of x1 from the equations number 2, ..., N , under an assumption that a1,1  0:
an ,1
1
df
ln ,1  1 for n  2,, N
a1,1
an 2,  an1,  ln ,1a1,1 for   1,, N 

 for n  2,, N
bn 2   bn1  ln ,1b11 

1 1
a1,1 x1  a1,2 x2    a1,1N xN  b11
 2
0  a2,2 x2    a2, 2N xN  b2 2 
A   x  b
2
 2
        
0  aN 2,2 x2    aN 2,N xN  bN 2 

Elimination, step #2: zeroing of the elements number 3, ..., N in the second column of A 
2
 2
by elimination of x2 from the equations number 3, ..., N , under an assumption that a2,2  0:
an ,2
2
df
ln ,2   2 for n  3, , N
a2,2
an3,  an 2,  ln ,2 a2, 2 for   2, , N 

 for n  3, , N
bn 3  bn 2   ln ,2b2 2  

1 1
a1,1 x1  a1,2 x2    a1,1N xN  b11
 2
0  a2,2 x2    a2, 2N xN  b2 2
A   x  b
3
 3
        
0  0    aN 3,N xN  bN 3
After N – steps: A    x  b  , where A   is an upper-triangular matrix.

N N N

LU factorisation
The Gauss method of elimination is equivalent to the following decomposition of A :

A  LU
1 0  0
l 1  0
 2,1 
where U  A  and L   l3,1 l3,2  0
N
 
   
lN ,1 lN ,2  1 
The system of equations:

LUx  b
may be solved in two steps – by solving two simpler systems of equations:
L  y  b and U  x  y
with the triangular matrices L and U .

Example: The in-place processing of the matrix A leading to its LU factorisation:
 3 1 6   x1   2 
 2 1 3   x    7 
   2  
1 1 1   x3   4 
1
a2,1 2
3 1 6 2 1
l2,1 
3 1 6 2   3 1 6 2
a 3  2
  1,1 2 1 17 
a 2 1 17 
 2 1 3 7    3 3 1 3   l3,2  3,2
 2
 2  3 3 1 3 
1 a
a
1 1 1 4  l  3,1  1  13 32 1 103  2,2
 13 2 1 8
3,1 1
a1,1 3
Hence:
1 0 0   3 1 6
L   23 1 0  and U  0 13 1
 
 3 2 1 
1
0 0 1

The number of operations:
 the LU factorisation: L  ,    O  1
3 N 3  and L *, /   O  13 N 3 
 the solution of two systems with triangular matrices: L  ,    O N 2 and L *, /   O N 2    
Some practical comments:
 The LU factorisation is especially convenient and efficient if the system of linear equations is solved
several times, for the same matrix A and various vectors b .
 To minimize numerical errors, the rows and columns of the matrix A are re-ordered at each step of
the LU factorisation as to maximize an ,n  0 . This operation is equivalent to the multiplication of the
n
system by a zero-one pivoting matrix P :

P  A  x  P b  L U  x  P b
Then, two triangular systems of linear equations ale solved to find the solution:
L  y  P  b and U  x  y

Cholesky-Banachiewicz factorisation
A symmetric matrix A is positive definite if  x  0 : xT  A  x  0 .
Any positive definite matrix A may be factorised in the following way:

A  L  LT
 l1,1 0  0 
l l  0 
where L    is a unique matrix with positive diagonal elements
2,1 22
     
 
l l
 N ,1 N ,2  l N ,N 

The elements of the matrix L may be found by comparison of the LHS and RHS of:
 a1,1 a1,2  a1, N   l1,1 0  0  l1,1 l2,1  lN ,1 
a a2,2  a2, N   l2,1 l22  0   0 l2,2  lN ,2 
 2,1   .
                
     
 aN ,1 aN ,2  aN , N  lN ,1 lN ,2  lN , N   0 0  lN , N 
i.e. by solving the following system of linear equations:
2
l1,1  a1,1 , ln,1  l1,1  an,1 for n  2,..., N
2
l2,1  l2,2
2
 a22 , ln,1  l2,1  ln,2  l2,2  an,2 for n  3,..., N
etc.
Its solution has the form:
n 1 
ln , n  a n , n   l 2
n ,i 
i 1 
n 1
 for n  1,, N
a ,n   l ,i  ln,i 
l ,n  i 1
for   n  1, , N 
ln , n 
The number of operations: L  ,    O  1

6 N 3  , L *, /   O  16 N 3  and L   N.
Residual correction (iterative improvement) of the solution
1
The solution x̂ , obtained by means of the LU or LLT factorisation, may be given the form:
xˆ 1  x  x
where x is the exact solution of A  x  b .
1
The equation resulting from the substitution of x  xˆ  x to A  x  b :

A  xˆ 1  x  b 
may be given the form:
A  x  A  xˆ    b
1
An estimate xˆ of x may be obtained by solving this equation, using the already obtained results of
the LU or LLT factorisation of A . It may be then used for correction of the solution x̂  :
1
xˆ  2  xˆ 1  xˆ

3.3. Computing determinants and inverse matrices
To calculate the determinant det  A  with a possibly small numerical error, one should use the LU or
LLT factorisation of the matrix A in the following way:
N
det  A   det  L  U   det  L   det  U   det  U    un,n
n 1
2
 N 
det  A   det  L  L    det  L      ln,n 
T 2
 n1 

To calculate the inverse matrix A  1 in an efficient and accurate way, one should use the equality
A  A 1  I which may be re-written in the form:
 y1,1 y1,2  y1, N  1 0  0
y y2, N  0
y2,2  1  0
A  
2,1
         
 
 y N ,1 yN ,2  y N , N  0 0  1

y1 y2  yN e1 e2  e N
or in the form of a set of N systems of linear equations with the same matrix A :
A  y1  e1
A  y 2  e2
…
A  y N  eN
The use of the LU (or LLT) factorisation is recommended for solving those equations:
L  U  y1  e1
L  U  y 2  e2
…
L  U  y N  eN

3.4. Iterative methods for solving A  x  b
A distinctive feature of an iterative algorithm: the repetition of identical or similar sequences of

operations (from Latin iterare = to repeat).
Example: The equation a  x  b ( a  0 ) may be solved by means of the following iterative algorithm:
xi 1  xi      a  xi   b  for i  0,1, ...
The speed of its convergence depends on  because:

i

i
  
x  x   x  x      a  x  x   b  , where x i   xi   x , x i 1  xi 1  x
i 1
xi 1  xi     a  x i   1    a   x i 
x i 1  xi  if 1    a  1
The convergence is guaranteed if:

1  1    a  1  2    a  0  2    sign(a )  a  0
2  γ   0 , 2 a  for a  0
     sign(a)  0  
a  γ   2 a , 0  for a  0

General properties of iterative methods
A linear iterative method for solving A  x  b is defined by the formula:

xˆ    M  xˆ    w
i 1
for i  0,1, ...,
i
 0
where x̂ is the initial approximation of the solution, while M and w satisfy the coherence condition:
x  M x  w.
The method is convergent if sr  M   1; the speed of convergence, i.e. the speed of approaching zero
by the norm:
xˆ  i   xˆ  i   x for i  0,1, ...,
increases with decreasing sr  M  .
Typical stop conditions:

xˆ    xˆ  
i 1 i
xˆ  i 1  xˆ i   x and i 
x
xˆ
where x and  b are indicators of expected accuracy.

Example: Gauss-Seidel method
The Gauss-Seidel method is based on the following decomposition of A :

A LU
where:
L is a lower-triangular matrix (with a non-zero main diagonal),
U is an upper-triangular matrix (with a zero main diagonal).
A heuristic development of the Gauss-Seidel method includes the following steps:

L  x  U  x  b
L  xˆ 
i 1
 U  xˆ    b for i  0,1, 2, 
i
xˆ    L1  U  xˆ    L1  b for i  0,1, 2, 

i 1 i
i.e.:
M  L1  U and w  L1  b

The computing algorithm for an iteration of the Gauss-Seidel method consists in solving the following
equation with a triangular matrix:
L  xˆ 
i 1
 U  xˆ    b for i  0,1, 2, 
i
or:
L  xˆ 
i 1
 v  for i  0,1, 2, 
i
where:
v     U  xˆ    b for i  0,1, 2, 
i i
The Gauss-Seidel method is convergent for:

 any positive definite matrix A ;
 any matrix A with a strongly dominant diagonal, i.e. if:
N N
an , n   an , for n  1,, N or an , n   a ,n for n  1,, N
 1  1
 n  n

Roman Z. Morawski

4. SOLVING NONLINEAR ALGEBRAIC EQUATIONS
R. Z. Morawski: "Numerical methods (ENUME) – 4. Solving nonlinear algebraic equations" 4-1

4.1. System of nonlinear algebraic equations
 x1   f1  x  
x   
f  x 
f  x   0 , where: x   2  and f  x    2 
    
   
x
 N f
 N  x  
Example:
 sin  x1  x2   1 
 
f x  
cos  x2  x 3   1
 tan  x3  x4   1
 
x  x
 1 2 3 4  x  x

4.2. General characteristics of iterative algorithms
An iterative algorithm (IA) for solving a numerical problem d  w has the form:

w i 1   w i  , w i 1 ,; d  for i =0,1,...
where:
i   is an operator defining IA
w i  is the i th approximation of the solution to the problem d  w
Examples:
The Heron algorithm for solving w2  d  0 ( d  0 ), i.e. for computing d:
 d 
w i 1  12  wi   i   for i  0,1,...
w 0   max  d ,1  ,
 w 
The Newton algorithm for solving f  w; d   0 :
w  i 1
wi


f w i  ; d for i  0,1,...
f   w  ; d 
i

An IA is globally convergent if there exists a non-empty set W0 of the starting points w   that:
0
w i   w  i   w
 
i
 0 for i  0,1,...
An IA is locally convergent if it is convergent for any (even very small) vicinity W0 of the convergence
 . The analysis of local convergence may be performed by means of the error function:
point w

w i 1  i wi  ; d  for i  0,1,...
taking into account that the error should diminish to zero for i   . The local convergence of a scalar
 by:
IA is characterised by an approximation of the function  i in the vicinity of w
 
w 
i 1
 C  w   (or w   C  w 
i i 1 i
in the scalar case)
where:
C  0 is the coefficient of local convergence and   [1,  ) is the exponent of local convergence
The attainable (relative) accuracy of a scalar IA is characterised by an interval:

w  i 
   K  eps,  K  eps  (or   w     K  eps,  K  eps  in the scalar case)
i 
 i 
w
where K is the indicator of attainable accuracy

4.3. Analysis of scalar one-point IAs
A scalar one-point IA is defined by the formula:

w i 1   wi  ; d  for i  0, 1,...
which will be further processed using a simplified notation:
wi 1    wi  for i  0, 1,... and w    i
i
The parameters of local convergence may be determined using the Taylor series expansion of   wi 
:
in the vicinity of w
wi 1    w      w   wi  w   12    w   wi  w   ...
2

w
2
   
wi 1  w    w   w     w   wi  w   2    w   wi  w   ...
1

         
i 1 0  i   i 
Hence:
 i 1     w   i  12    w   i2  ...

For "small"  i  i  0  , the following approximation of:
 i 1     w   i  12    w   i2  16    w   i3  ...
is justified:
   w   0  i+1     w  i
   w   0 and    w   0   i+1  12    w   i2
   w      w   0 and    w   0   i+1  16    w   i3
Thus, the parameters of local convergence C and  , i.e. the parameters of the function

 i 1  C   i are:
1
C      w  and   1, 2, ...
!
A scalar one-point IA is convergent if:

C  1 for   1 and C   for   2, 3, ...

3
Example: Analysis of the local convergence of IA for computing d , i.e. for solving the equation
w3  d :
wi 1  wi  152  wi3  d  for d  1, 8
 
Since   w   w  152 w3  d , the exponent of convergence is   1 because:
   w   1  152  3w2  0  
2

w d
 1  2
5 d 0
3
3
Consequently:
C  1  52 d 3  35 for d  1, 8
2
So, AI is convergent because C  1.

Example: Analysis of the local convergence of the Heron algorithm for computing d:
 d 
w0  max d ,1 and wi 1  12  wi   for i  0,1,...
 wi 
 d
Since   w  12  w   , the exponent of convergence is   2 because:
 w
 d  d 1
   w  12 1     0 and    w     0
 w2  w  d w3 w d d
Thus:
1
C
2 d

Evaluation of attainable accuracy
The evaluation of attainable accuracy is based on the following principle:

w i  wi 1  i  
i 
 w 1  i 
where i is the error resulting from propagation and accumulation of rounding errors introduced
during the iterations number 1, ..., i . If i 1  1, then:
 1 
i +1  C  w  i  i
where:
 1 
C  w  i is the error component "inherited" from the previous iteration
i is the error component introduced during the current iteration

3
Example: Evaluation of attainable accuracy of the IA for computing d:
wi 1  wi  152  wi3  d  dla d  1, 8
 
w i 1  w i  152  w i3 1   pi   d  1  mi 1  oi  1  oi 
For i   :

w 1  i 1   w 1  i   152  w 3 1  i  1   pi   w 3  1   mi  oi  1  oi 

3
 
 
w 1  i 1   w 1  i   152  w 3 1  3i   pi   w 3  1  mi  oi  1  oi 

w 1  i 1   w 1  i  152 w 2  3i   pi  1   mi  oi  1  oi  
w 1  i 1   w 1  1  2
5 w 2 i  152 w 2 pi  oi 
i 1  1  52 w 2 i  152 w 2 pi   oi
 i 3   23
i 1  C i  2
15 w  pi   oi  i 1  i 
2 3
5
23
15 eps     5
3 i 1
0    5   15 eps
 0 
23
eps
i 1  15
 23
6 eps  4eps  K 4
1 5
3

 For all AI with   1:
K  eps 

sup i  d  d  D 
1  sup  C  d  d  D 
 For all AI with   2, 3,... :

K  eps  sup  i d  D 
 1 
(in this case, the error component "inherited" from the previous iteration C  w  i is
negligible in comparison to the error component introduced during the current iteration i )

4.4. Methods for solving scalar equations f  x   0
Bisection method
For f  x  continuous in  a0 , b0  and satisfying the condition f ( a0 )  f (b0 )  0 , the bisection method is
defined by the formulae: x0  1
2  a0  b0  and:
ai 1  ai , bi 1  xi gdy f ( ai )  f ( xi )  0 
xi 1  1
 i 1 i 1 
a  b with   for i  0, 1, 2, 
ai 1  xi , bi 1  bi gdy f ( ai )  f ( xi )  0 
2
% Function defining equation

f=@(x) 2*sin(x)+sin(5*x);a=6;b=8;
xx=linspace(6,8,100);yy=f(xx);
plot(xx,yy,'LineWidth',1);hold on;grid on
xlabel('x');ylabel('f(x)');set(gca,'FontSize',12);
title(['a = ',num2str(a),', b = ',num2str(b)],...
'FontWeight','normal');
set(gca,'FontSize',10);
% Implementation of bisection method
for i=1:5
z(i)=(a+b)/2;
if sign(f(z(i)))==sign(f(a)),a=z(i);else b=z(i);end
line([z(i),z(i)],[0,f(z(i))],'LineStyle','--',...
'Color','red','LineWidth',1);
text(z(i)-0.02,f(z(i))+0.2,num2str(i-1))
end
plot(z,f(z),'o');

The local convergence of bisection method is characterised by the parameters:   1 and C  12
because:
 12  ai  xi  if f ( ai )  f ( xi )  0 
xi 1    xi    1  and   x   2
1
 2  xi  bi  if f ( ai )  f ( xi )  0 
The number of iterations necessary for reducing the error below  is:
 | b  a |
I  min i  log 2 
  

Regula falsi method
  1
xi  x0 
xi 1  xi  f ( xi ) for i  1, 2, ...   f ( x )
f ( xi )  f ( x0 ) C   x  x0   1
 f ( x0 )

The parameters  and C for the regula falsi method may be estimated as follows:
x  x0 x0 f ( x)  xf ( x0 )
  x  x  f ( x) 
f ( x)  f ( x0 ) f ( x)  f ( x0 )
 

 x   0
x f ( x )  xf ( x0  
)  f ( x )  f ( x0   0
)  x f ( x )  xf ( x0  
)  f ( x )  f ( x0 
)
 f ( x)  f ( x0 )
2

 x0 f ( x )  f ( x0 )    f ( x )  f ( x0 )    x0 f ( x )  xf ( x0 )   f ( x )
 f ( x)  f ( x0 )
2
Since f ( x )  0 :
 x0 f ( x )  f ( x0 )    f ( x0 )    xf ( x0 )  f ( x )  f ( x )
   x    x  x0   1
  f ( x0 )
2
f ( x0 )
f ( x )
Thus:   1 and C   x  x0   1
f ( x0 )

Newton's (tangent) method
  2
f ( xi ) 
xi 1  xi  for i  0, 1,...   f ( x )

f ( xi ) C   1

2
f ( x )

The parameters  and C for the Newton's method may be estimated as follows:
 f ( x)  f ( x) f ( x)
2
f ( x) f ( x) f ( x)
  x  x     x   1    0
f ( x )  f ( x)  f ( x) x  x
2 2
 f '( x) f ( x)  f ( x) f ( x)   f ( x)  f ( x) f ( x)  2 f ( x) f ( x)

2
   x  
 f ( x)
4
Since f ( x )  0 :
 f '( x ) f ( x )   f ( x )
2
f ( x )
   x   
 f ( x )
4
f ( x )
f ( x )
Thus:   2 and C     x   1 1
2 2
f ( x )

Secant method

   1 5  1  1.618
 2 
xi  xi 1 
2
5 1
xi 1  xi  f ( xi ) for i  1, 2,...    f   x  
1
f ( xi )  f ( xi 1 )  C   12 
  f    

x

The parameters  and C for the secant method may be estimated as follows:
f  xi  xi  xi 1 
xi 1  xi  for i  1, 2,...
f  xi   f  xi 1 
 f  x   f   x  i  12 f   x   i2  ...   i  i 1 
 i 1   i 
f  x   f   x  i  12 f   x   i2  ...  f  x   f   x   i 1  12 f   x   i21  ...
Since f  x   0 :
 f   x   i  12 f   x  i2    i   i 1 
 i 1  i 
f   x    i   i 1   12 f   x   i2   i21 

After dividing by  i   i1 :
 f   x   i  12 f   x   i2 
 i 1   i 
f   x   12 f   x    i  i 1 
f   x   i  12 f   x    i2   i 1 i   f   x   i  12 f   x  i2
  Cˆ  i  i 1
f   x   12 f   x    i  i 1 
1
f   x    1 
where Cˆ  1
. Since  i 1  C  i   i  C  i 1   i 1    i  :
2
f   x  C 
1
1 1
1   1
 i 1  Cˆ  i  i 1 ˆ ˆ 
 C i  i   C C i 
C 

Since  i 1  C  i , the following equations must be satisfied by C and  :
1
1 

1   and Ĉ C C

Their solution has the form:
1
 
 
5 1
  12 1  5 and C = Cˆ 2

Example: A comparison of the convergence speed of three methods when applied for solving
the equation x 2  x  2  0 .
NUMBER OF ITERATIONS
METHOD
1 2 3 4 5 6
BISECTION (   1)
1.3  10-1 6.3  10-2 3.1  10-2 1.6  10-2 7.8  10-3 3.9  10-3
x0  3 , x1  1.5
SECANT (   1.618)
7.1  10-2 1.5  10-2 7.5  10-4 7.5  10-6 3.8  10-9 1.9  10-14
x0  3 , x1  1.5
NEWTON (   2)
1.0  10-1 5.9  10-3 2.3  10-5 3.5  10-10 K eps K eps
x0  3

Muller's method
The method is based on approximation of f  x  around xi by means of a quadratic function:

fˆ  x  xi   ai  x  xi   bi  x  xi   ci
2
 version I – the approximation by means of the Taylor series referring to f  xi  , f   xi  and f   xi 

(a generalised tangent method using second derivative):
1
ai  f   xi  , bi  f   xi  , ci  f  xi 
2
 version II – the interpolation of f  xi  2  , f  xi 1  and f  xi  (a generalised secant method derived

from three-point interpolation):
1
 ai     xi  xi 1   xi  xi 1    f  xi   f  xi 1  
2
b       and ci  f  xi 
 i     xi  xi  2   xi  xi2    f  xi   f  xi 2 
2
The iterative formula, implemented in complex arithmetic, is identical for both versions:
2ci
xi 1  xi 
bi  sgn  bi  bi2  4ai ci
The convergence exponent is:   3 for version I, and   1.84 for version II.

4.5. Newton-Raphson method for solving system of equations f ( x)  0
The method is based on the assumption that:

f ( x)   f1 ( x) f 2 ( x)... f N ( x) 
T
where x   x1 x2 ... x N  . The method is defined by the following formula:

T
 f1 f1 
xi 1  xi  f   xi   f  xi  
1
 x 
 xN 
  1 
  for i  0, 1,... ; where f   x       
1   f f N 
xi 1  xi   f   xi   f  xi    N
 

 x1 xN 
The corresponding algorithm has the form:
 compute f  xi  and f   x i  ;
 solve f   xi   x i  f  xi  with respect to xi  xi 1  xi ;
 compute xi 1  x i  xi ;
 return to  if xi  x or xi xi   x .

4.6. Methods for finding roots (zeros) of polynomials
The roots xn of a polynomial:

N
f N  x   aN x  aN 1 x
N N 1
   a1 x  a0  aN   x  xn 
n 1
may be: real and/or complex; single and/or multiple.
Finding real roots and linear deflation
Any method for solving scalar nonlinear equations, described in § 3.3, may be used for finding real
zeros of a polynomial. A zero already found may be eliminated using the so-called linear deflation.
For any single real root x of f N  x  , the following relationship holds:

f N  x   f N 1  x    x  x  , where f N 1  x   bN 1 x N 1  bN  2 x N  2    b1 x  b0
The coefficients b0 , ..., bN 1 may be determined using an algorithm for dividing the polynomial f N  x 
by  x  x  , called the Horner's algorithm:
bN 1  aN , and bn  an 1  x  bn 1 for n  N  2, ..., 0

Finding complex roots and quadratic deflation
A pair of complex roots x and x  may be found using the Mueller's method. They may be eliminated
using the so-called quadratic deflation. The coefficients of the quadratic polynomial:
m  x    x  x   x  x    x 2  px  r
whose values are p  2Re  x  i r   x , are used for finding the polynomial :
2
f N  2  x   bN  2 x N  2    b1 x  b0
such that:
f N  x    x 2  px  r  f N  2  x 
The solution to this problem has the form:

bN  2  aN
bN 3  a N 1  pbN  2
bn  an 2  pbn1  rbn  2 for n  N  4, ..., 0

Accuracy of roots estimation
Example: The errors  n of the estimates xn  xn 1   n  of roots xn ( n  1, 2 , 3 ) of the polynomial:
f  x   x 3  30 x 2  299 x  990
may be evaluated in the following way:
f  xn   xn3  30 xn2  299 xn  990   n
xn3 1   n   30 xn2 1   n   299 xn 1   n   990   n
3 2
xn3 1  3 n   30 xn2 1  2 n   299 xn 1   n   990   n

 xn3  30 xn2  299 xn  990    3 xn3  60 xn2  299 x n   n   n
Since xn  30 xn  299 xn  990  0 , the errors may be determined as follows:
3 2
n n
n  
3xn3  60 xn2  299 xn  3x
2
n  60 xn  299  xn

Roman Z. Morawski

5. APPROXIMATION OF FUNCTIONS
R. Z. Morawski: Numerical methods (ENUME) – 5. Approximation of functions 5-1

INTERPOLATION SMOOTHING APPROXIMATION
-0.02 -0.02
fˆ  x  f x  fˆ  x  f x
-0.04 -0.04
-0.06 -0.06
-0.08 -0.08
-0.1 -0.1
y y
-0.12 -0.12
-0.14 -0.14
-0.16 -0.16
-0.18 -0.18
-0.2 -0.2
-1 -0.5 0 0.5 1 -1 -0.5 0 0.5 1
x x

5.1. Formulation of the interpolation problem
Having N  1 pairs:
xn , f  xn  for n  0, ..., N
determine a function fˆN  x  interpolating a function f  x , i.e. satisfying the following conditions:
fˆN  xn   f  xn  for n  0, ..., N
The solution of the interpolation problem consists in estimation of the parameters p of a known
function fˆ  x   fˆ  x; p  , where fˆ  x  is (most frequently):
N N N
 an algebraic polynomial fˆN  x; p    pn x n with p   p0 ... pN 

N
T
n 0
 a polynomial spline function of order 2, 3 or 5
 a trigonometric polynomial: fˆN  x; p    pn cos nx  with p   p0 ... pN 

N
T
n 0
a  a x  ...  a x N M 1
 a rational function: fˆN  x; p    
0 1 N  M 1 T
with p a ... a b ... b
1  b1 x  ...  bM x M
0 N M 1 1 M

5.2. Interpolation of discrete values of an unknown function with an algebraic polynomial
There exists a unique polynomial fˆN  x; p    pn x n of order N , such that:

N
n 0
fˆN  x   f  xn  for n  0, ..., N
If f  x  is N  1 times differentiable in the interval  x 0 , xN  , then the error of interpolation

 
fˆN  x; p   f  x  is subject to the following assessment:
ˆf  x; p   f  x   YN 1
  
N
 x  x  , where Y  sup f  N 1
 x x   x 0 , xN 
N
 N  1! n0 n N 1
Conclusion: a "large" error of interpolation may happen in the vicinity of "large" changes of f  x  or
its derivatives.
x  x

N
The Lagrange polynomial: fˆN  x    f  xn  Ln  x  with Ln  x 
N
n 0  0 xn  x
 n

5.3. Interpolation of discrete values of an unknown function with a polynomial spline function
The function s  x; x0 , ..., xN  , where x0  x1  ...  x N , is a polynomial spline function of order M , if:
 it is a polynomial of order not higher than M in each subinterval  xn , xn1  , i.e.:
s  x   an , M  x  xn   ...  an ,1  x  xn   an ,0 for x   xn , xn1  , n  0, 1, ..., N  1

M
 it is continuous with M 1 derivatives in  x0 , xN 
The function s  x   s  x; x0 , ..., xN  is an interpolating spline function if:

s  xn   yn for n  0, 1, ..., N
The coefficients of an interpolating spline function an , M , ..., an ,1 , an ,0 , whose number is N  M  1 , have

to satisfy the following MN  M  N  1 equations:
 the N  1 interpolation conditions
 the M  N  1 continuity conditions for the derivatives of order 0, ..., M  1 at x1 ,..., xN 1

The remaining degrees of freedom, whose number is:
N  M  1   MN  M  N  1  M 1
may be used for satisfying some additional boundary conditions:
 for a spline function of order 2 (alternatively):
a) s   x0   1
b) s   xN   1
c) s   x0   s   xN 
 for a spline function of order 3 (alternatively):
a) s   x0   2 , s   xN   2
b) s   x0   1 , s   xN   1
c) s   x0   s   xN , s   x0   s   xN 

Example: Design a spline function of order 2:
s  x   an  x  xn   bn  x  xn   cn for x   xn , xn1  , n  0, 1, 2
2
interpolating the following points:

n 0 1 2 3
xn 0 1 2 3
yn 1 2 4 3
and satisfying the following boundary condition: s   x0   0 .

The interpolation conditions have the form:
 s  xn   an  xn  xn   bn  xn  xn   cn  yn  cn  yn for n  0, 1, 2
2
 s  x3   a2  x3  x2   b2  x3  x2   c2  y3  a2  b2  c2  y3
2
The continuity conditions for the function have the form:

 s  x1   a0  x1  x0   b0  x1  x0   c0  a1  x1  x1   b1  x1  x1   c1  s  x1 
2 2
 a0  b0  c0  c1
 s  x2   a1  x2  x1   b1  x2  x1   c1  a2  x2  x2   b2  x2  x2   c2  s  x2 
2 2
 a1  b1  c1  c2
The continuity conditions for the first derivative of the function have the form:
 s   x1   2a0  x1  x0   b0  2a1  x1  x1   b1  s   x1   2 a0  b0  b1
 s   x2   2 a1  x2  x1   b1  2 a2  x2  x2   b2  s   x2   2a1  b1  b2
The boundary condition has the form:

 s   x0   2a0  x0  x0   b0  0  b0  0

The values of cn result from the interpolation conditions: c0  1, c1  2 and c2  4 .
The remaining conditions assume the form of algebraic equations with respect to an and bn :
a2  b2  4  3  a2  b2  1 2 a0  b0  b1
a0  b0  1  2  a0  b0  1 2a1  b1  b2
a1  b1  2  4  a1  b1  2 b0  0
The solution of those equations has the form: 4.5
a0  1 , a1  0 and a2  3 4
b0  0 , b1  2 and b2  2 3.5
Hence: 3
y=s(x)

 x 2  1 for x  0,1

2.5
s  x  2  x 1  2 for x  1, 2



2
 3  x  2 
2
 2  x  2  4 for x   2, 3
 1.5
1
0 0.5 1 1.5 2 2.5 3
x

5.4. Least-squares approximation of a function on the basis of its discrete values
A vector of parameters p   p1 ... pK  is sought for, the vector that makes the linear combination
T
of linearly independent functions  k  x  k  1, 2 , ..., K  :
fˆ ( x; p)   pkk ( x)
K
k 1
best approximate a sequence of discrete values of an unknown function f  x :
 f  xn  n  1, 2, ..., N 
with respect to the following (least-squares) criterion:

1 N ˆ 
J 2 p     fˆ  xn ;p  f  xn 
N
or J 2 p  f  xn ;p  f  xn 
2 2
  N n1 
 
n1
Examples of k  x  :
x k1 k  1, ..., K  , cos k 1 x k  1, ..., K  , sinc  x  xk  k  1, ..., K 

The necessary condition of the minimum has the form:
J 2 p 
 0 for k  1, ..., K
 pk

ΦT  Φ  p  ΦT  y
where:
 1  x1  2  x1   K  x1  
 
 1  x2  2  x2   K  x2  
Φ   and y   f  x1  f  x2  ... f  xN 
 
T
     
  x  2  x N   K  xN 
 1 N
The solution may be presented in an analytical form:

pˆ  ΦT  Φ  ΦT  y
1
or numerical form:
pˆ  arg p ΦT  Φ  p  ΦT  y 
The Cholesky-Banachiewicz method may be used because ΦT  Φ is a positive definite matrix.

5.5. Uniform approximation of a function on the basis of its discrete values
T
of linearly independent functions k  x  k  1, 2 , ..., K  :
fˆ ( x; p)   pkk ( x)
K
k 1
best approximate a sequence of discrete values of an unknown function f  x :
 f  xn  n  1, 2, ..., N 
with respect to the following criterion:
J  p  sup  fˆ  xn ; p  f  xn  n  1,...,N 

Numerical implementation:
 the algorithms of non-differentiable optimisation
 the Remez algorithm
 the least-squares approximation based on the first-kind Chebyshev polynomials

5.6. Least-squares approximation of a known function by means of algebraic polynomials
T
of linearly independent functions  k  x  k  1, 2 , ..., K  :
fˆ ( x; p)   pkk ( x)
K
k 1
best approximate a given function f  x  with respect to the following criterion:

b b
 fˆ  x;p  f  x  dx or J p  1  fˆ  x;p  f  x dx
b  a a
J 2 p   
2 2
  2  
a
where the approximated function f  x  is assumed to satisfy the condition:

b
 f 2  x  dx  
a

The necessary condition of the minimum has the form:
J 2 p
 0 for k  1, 2 , ..., K  Ap  b
 pk
where:
b 
ak , j   k  x   j  x dx  k |  j for j  1, 2, ..., K 

a  for k  1, 2, ..., K
b 
bk   f  x k  x  dx  f | k 
a

A particularly simple solution may be obtained if k  is a subset of an orthogonal basis, i.e.:
b
k |  j   k  x  j  x  dx  0 for k  j
a
Then ak , j  0 for k , j  1, 2, ..., K ; j  k , i.e. the matrix A  diag ak ,k  .
If, moreover, k 2
 k , k  1 for k  1, ..., K , then:
A  I  p  b  pk  f | k for k  0, 1, ..., K

A sequence of linearly independent functions 1  x , 2  x  , ... may be transformed into a sequence
of orthogonal functions  1  x, 2  x, ... by means of the following procedure called Gram-
Schmidt orthogonalisation.
The inner product of functions in the above procedure is defined as follows:

b
k |  j   w x  k  x  j  x  dx
w
a
b
where w x  is a weighing function such that w x   0 and  w( x) dx   .

a
The orthogonalisation of    x  x
k
k
k  0,1, ... for various intervals  a, b weighing functions
w x  generates various families of orthogonal polynomials.

Example: Legendre polynomials – orthogonal in  1, 1 with w  x   1:
P0  x   1, P1  x   x
2k  1 k 1
Pk  x   xPk 1  x   Pk  2  x  for k  2, 3, ...
k k
1
2
  Pk2  x  dx 
2
Pk for k  0 ,1, ...
2
1
2k  1
1
Example: Chebyshev polynomials of the I kind – orthogonal in  1, 1 with w  x   :
1 x 2
T0  x   1, T1  x   x
Tk  x   2 xTk 1  x   Tk 2  x  for k  2, 3, ...
1  for k  j  0

1 w  x  Tk  x  T j
 x  dx   / 2 for k  j  0
 for k  j
0

Example: Hermite polynomials – orthogonal in (, ) with w  x   exp  x 2 :
H 0  x   1, H1  x   2 x
H k  x  2 xH k 1  x   2k  2 H k 2  x  for k  2 , 3, ...

  exp x 2  H k2  x  dx  2k k!  for k  2 , 3, ...

2
Hk 2

Notice: If the functions 1  x  , 2  x , ... are orthogonal with respect to the inner product:
b
k |  j   w  x  k  x   j  x  dx
w
a
then the functions k  x  w x  k  x are orthogonal with respect to the inner product:
b
k |  j   k  x   j  x dx
w
a

5.7. Approximation of a known function by means of a rational function (Padé method)
The parameters p   a0 ... aL b1 ... bM  of a rational function:

T
a  a x  ...  a x L
fˆ  x; p   0 1 L
for L =0,1, ...; M  1, 2, ...; L  M
1  b1 x  ...  bM x M
are determined in such a way as to satisfy the equality up to L  M  1 terms:
1  b x  ...  b
1 M x M  c0  c1 x  c2 x 2  c4 x 3  ...  a0  a1 x  ...  aL x L
where c0  c1x  c2 x 2  c4 x3  ... is the MacLaurin expansion of f  x  .
Thus, the parameters a0 , ..., aL , b1 , ..., bM should satisfy the following equations:
l 1
c0  a0 , cl   ck bl  k  al for l  1, ..., L
k 0
l 1
cl   ck bl  k  0 for l  L  1, ..., M
k 0
l 1
cl  
k l M
ck blk  0 for l  M  1, ..., M  L

5.8. Comparison of selected methods of interpolation and approximation
Two methods of interpolation, viz.:

 the interpolation by means of a Lagrange polynomial
 the interpolation by means of a polynomial spline of order 3
and two methods of approximation, viz.:
 the least-squares approximation by means of an algebraic polynomial
 the least-squares approximation by means of a linear combination of Chebyshev polynomials
are applied for interpolation and approximation of the function:
 24  e z 10 sin  z 4  e z 10 cos  z 4
y  f  x  sin  2  , where z  5 x  5 , x  1, 1
 x  4  10  e z 10 sin  z 4  e z 10 cos  z 4
The data for this purpose are generated after the formula:
n 1
xn  1  2 , yn  f  xn  for n  1, ..., N
N 1
The accuracy of interpolation and approximation has been assessed using the criterion:
1
1 ˆ 
 f  x   f  x  dx (root-mean-square error =RMS error)
2
f  
2 1  

Error-free data
RMS error

Error-free data
105
y
Lagrange interpolation
Cubic spline interpolation
LS polynomial approximation (N=100)
100 LS approximation with Chebyshev polynomials (N=100)
10-5
RMS error
10-10
10-15
10-20
0 10 20 30 40 50 60
number of nodes/order of polynomial

Error-free data
105
y
10-5
RMS error
10-10
10-15
10-20
0 10 20 30 40 50 60

Error-free data
105
y
10-5
RMS error
10-10
10-15
10-20
0 10 20 30 40 50 60

Error-free data
105
y
10-5
RMS error
10-10
10-15
10-20
0 10 20 30 40 50 60

Error-free data
105
y
10-5
RMS error
10-10
10-15
10-20
0 10 20 30 40 50 60

Error-free data
105
y
10-5
RMS error
10-10
10-15
10-20
0 10 20 30 40 50 60

Error-free data
105
y
10-5
RMS error
10-10
10-15
10-20
0 10 20 30 40 50 60

Error-free data
105
y
10-5
RMS error
10-10
10-15
10-20
0 10 20 30 40 50 60

Error-free data
105
y
10-5
RMS error
10-10
10-15
10-20
0 10 20 30 40 50 60

Error-free data
105
y
10-5
RMS error
10-10
10-15
10-20
0 10 20 30 40 50 60

Error-free data
105
y
10-5
RMS error
10-10
10-15
10-20
0 10 20 30 40 50 60

Error-free data
105
y
10-5
RMS error
10-10
10-15
10-20
0 10 20 30 40 50 60

Error-free data
105
y
10-5
RMS error
10-10
10-15
10-20
0 10 20 30 40 50 60

Error-free data
105
y
10-5
RMS error
10-10
10-15
10-20
0 10 20 30 40 50 60

Error-free data
105
y
10-5
RMS error
10-10
10-15
10-20
0 10 20 30 40 50 60

Error-free data
105
y
10-5
RMS error
10-10
10-15
10-20
0 10 20 30 40 50 60

Error-free data
105
y
10-5
RMS error
10-10
10-15
10-20
0 10 20 30 40 50 60

Error-free data
105
y
10-5
RMS error
10-10
10-15
10-20
0 10 20 30 40 50 60

Error-corrupted data
  1012   1010
105 1010
Lagrange interpolation Lagrange interpolation
Cubic spline interpolation Cubic spline interpolation
LS polynomial approximation (N=100) LS polynomial approximation (N=100)
LS approximation with Chebyshev polynomials (N=100) LS approximation with Chebyshev polynomials (N=100)
100 105
RMS error
10-5 100
10-10 10-5
10-15 10-10
0 10 20 30 40 50 60 0 10 20 30 40 50 60
number of nodes/order of polynomial number of nodes/order of polynomial

Error-corrupted data
  108   106
RMS error
RMS error

Roman Z. Morawski

6. SOLVING ORDINARY DIFFERENTIAL EQUATIONS
R. Z. Morawski: Numerical methods (ENUME) – 6. Solving ordinary differential equations 6-1

6.1. Introductory example: dynamics of an electrical network
Using the elemental equations and Kirchhoff's laws, one may model the following network:
R1 R2
C1 C2
e(t)
u1 u2
with two ordinary differential equations (ODEs):

 R1  R2 1 
   1 
 u1(t )   R1R2C1 R2C1   u1 (t )   
      R1C1   e(t ) for t [0,T ]
u2 (t )  1 1  u2 (t )  
      0 
 R2C2 R2C2 
which for R1  1 kΩ , R2  1 , C1  C2  1 μF and e(t ) 1(t ) V takes on the form:
 u1(t )  1001 1  u1 (t )  103 
        for t  0
u2 (t ) 
  1 1 u2 (t )  0 
  
u (t ) A u(t ) b

% Symbolic solution of ODE
A=[-1001 1;1 -1];b=[1000;0];syms u(t) [2 1]
ODE=diff(u,t)==A*u(t)+b;IniCon=u(0)==[0;0];
uSol=dsolve(ODE,IniCon);u2(t)=simplify(uSol.u2);
fprintf('u2(t) = %s\n',vpa(u2(t)));
fplot(@(t) u2(t),[0,6]);grid on;
xlabel('t');ylabel('u_2(t)');
u2(t) = 9.9900e-04*exp(-1.0010e+03*t) - 1.0010*exp(-0.9990*t) + 1

6.2. Formulation of initial-value problem (IVP)
In scalar notation: Find the functions y1  t  , , yM  t  satisfying the following set of ODEs:
dym  t 
 f m  t , y1  t  , , y M  t   for t   0, T 
dt
given the initial conditions, i.e. the values of ym  0  for m  1,, M .
In vector notation: Find a vector of functions y  t    y1  t  yM  t  satisfying the following system
T
of ODEs:
dy  t 
 f  t , y  t   for t   0, T 
dt
given the initial condition, i.e. the vector y  0    y1  0  yM  0  
T

6.3. Design of numerical methods for solving IVPs
A numerical method for solving an IVP is defined by a recursive operator:

y n  S  y n1, y n2 , ...  for n  0, ..., N
generating a sequence y n  being an estimate of the sequence y  tn  , where:  
0  t0  t1    tn    t N  T
Such a method may be designed by replacing the derivative in:
dy  t 
 f  t , y  t   for t   0, T  with a formula of numerical differentiation
dt

Example: The Taylor series of the solution y  t  :
y  t   y  tn1   y  tn1  t  tn 1   ...
for t  tn takes on the form:
y  tn   y  tn 1   y  tn 1  tn  tn 1   ...  y  tn 1   f  tn 1 , y  tn 1    tn  tn 1   ...
Which is the basis for the so-called forward Euler method:
yn  yn1  hn1 f  tn1, yn1 
where yn is an estimate of y  tn  and hn  tn  tn 1 .
The algorithm for solving the system of ODEs from Section 6.1:
u  t   A  u  t   b  e  t  for t  [0, T ]
based on the above method takes on the form:
un  un1  h   A  un1  b  e  tn1   for n  1, 2, ... and u 0  u  0 

6.4. General properties of numerical methods for solving IVPs
Convergence of methods for solving IVPs
A numerical method for solving an IVP is convergent if for any set of ODEs, having a unique solution
 
y  t  , the approximate solution y n  , dependent on the step h , converges to y  tn  for h  0 , i.e.
the global error of the numerical solution is diminishing to a zero:
en  y n  y  tn  
h0
 0 for n  0,1,

Local error and order of methods for solving IVPs
An estimate of the error generated by the operator:

y n  S  y n1, y n2 , ...  for n  0, ..., N
during a single step of solving IVP is called local error:
rn  S  y  tn1  , y  tn2  , ...   y  tn 
Its dependence on the step of integration h :

rn  rn  h   rn  0   rn  0  h  12 rn 2  0  h 2  
is used for the determination of the order of the method characterising its local accuracy.
A method is said to be of order p if:

rn  0   0 , rn  0   0 , …, rn
p
 0   0 , but rn p 1  0   0
For the method of order p :
1
rn  h   rn   0  h p 1
p 1
 p  1!

Example: The local error and the order of the forward Euler method:
yn  yn1  hf  tn1 , yn1 
will be determined using a simplified notation:
y  tn   yn , y  tn   y n , y  tn   y n , …
f  tn , y  tn    f  tn , yn   yn
The Taylor series of the RHS of the formula yn  yn1  hf  tn1 , yn1  has the form:
yn  y n1  hyn 1   y n  y n h  12 y n h 2  ...  h  y n  y nh  ...

  
yn1 yn 1
Hence:
yn  y n    y n h  y n h    12 y nh 2  y nh 2   ...  y n  12 y nh 2
and the conclusion:
p  1, rn  h    12 y n h2

Stability of methods for solving IVPs
For any square matrix A , having no repeated eigenvalues 1 , 2 , ... C :

A  V  Λ  V 1 , where Λ  diag 1 , 2 , ...  and V is the matrix of eigenvectors
This decomposition may be used for transformation of a system of linear ODEs:

y  t   A  y  t   B  u  t 
into a set of independent scalar ODEs, viz.:
y  t   V  Λ  V 1  y  t   B  u  t 
V 1  y  t   Λ  V 1  y  t   V 1  B  u  t 
   
z t  zt 
z  t   Λ  z  t   V 1  B  u  t  , where: z  t   V 1  y  t  .
Thus, the results of stability analysis obtained for a scalar ODE, called "test equation":
y  t     y  t  with  C , for y  0   1 and t   0, T 
may be generalised on linear systems of ODEs.

Example: The forward (explicit) Euler method:
yn  yn1  h  f  tn1, yn1 
when applied to the test equation, y  t     y  t  , generates a difference equation:
yn  yn 1  h yn 1  1  h  yn 1
Thus, it is stable if:
1  h  1  1  h  1  1  Re  h    Im  h    1
2 2 2
i.e. for h in the unite circle whose centre is located at  1, 0  .

% REGION OF STABILITY FOR EXPLICIT EULER METHOD.
% NOTATION: h*lambda = x+j*y
% Range of x and y coordinates:
N=300;xx=linspace(-2.25,0.25,N);yy=linspace(-1.25,1.25,N);
% Generation of grid with all points (x,y):
[x,y] = meshgrid(xx,yy);
Im[h*lambda]
% Representation of inequality in logical matrix:
hl=x+1j*y;I=abs(1+hl)<1;
% Plot of points (x,y) that satisfy inequality:
scatter(x(I),y(I),0.3);grid on
xlabel('Re[h*lambda]');ylabel('Im[h*lambda]');
axis('equal');axis([-2.25 0.25 -1.25 1.25]);

Example: The backward (implicit) Euler method:
yn  yn1  h  f tn , yn   yn  arg y  y  yn1  h  f tn , y 
when applied to the test equation, y t     y t  , generates a difference equation:
1
yn  yn1  h yn  yn1
1  h
Thus, it is stable if:
1
1  1 h 1  1 Reh   Im h   1
2 2
1 h
i.e. for all h with Reh  0 .
% REGION OF STABILITY FOR IMPLICIT EULER METHOD.
% Range of x and y coordinates:
N=300;xx=linspace(-0.25,2.25,N);yy=linspace(-1.25,1.25,N);
% Generation of grid with all points (x,y):
[x,y] = meshgrid(xx,yy);
hl=x+1j*y;I=1./abs(1-hl)<1;
scatter(x(I),y(I),0.3);grid on
axis('equal');axis([-0.25 2.25 -1.25 1.25]);

6.5. Single-step methods
Generalised Runge-Kutta formula for single-step methods
K
Butcher's table of coefficients:
yn  yn 1  h   wk f k
k 1 c1 a1,1 a1,2  a1,K
where: c2 a2,1 a2,2  a2,K
 K
     
f k  f  tn 1  ck h, yn 1  h   ak , f  for k  1, 2,, K cK a K ,1 aK ,2  aK ,K
  1 
ck   0,1 and wk   0,1 for k  1, 2,, K w1 w2  wK
K
w
k 1
k 1

Runge-Kutta formula for explicit single-step methods
If ak ,  0 for   k , then the method is explicit, and the values of f k may be determined recursively:
 k 1

f k  f  tn 1  ck h, yn 1  h   ak , f  for k  1, 2,, K ;
  1 
Runge-Kutta formula for implicit single-step methods
Otherwise it is implicit, and therefore the values of f k must be determined by solving the system of
algebraic equations (nonlinear, if the function f is nonlinear:
  K

f t
  n 1 1

 c h , y n 1  h   a f
1,   

 f1  
 1
 f   f  t  c h, y  h  a f  
K
 2     n 1 2 n 1  2,   

  
 1
    
 fK   
  K

f t
  n 1 K c h , y n 1  h   a f
K ,   
   1 

Order and stability of explicit single-step methods
The maximal possible order: Regions of stability:

 K for K  1, 2, 3, 4 3

 K 4
p  K    K  1 for K  5, 6, 7

 K  2 for
K 3
K 8 2
K 2
1
K 1

Im h 
-1
-2
-3
-3 -2 -1 0
Re h  

Lower-order explicit single-step methods
1. The forward Euler method ( K  1, p  1): 0 0

yn  yn1  h  f  tn1, yn1  1
2. The modified (midpoint) Euler method ( K  2 , p  2 ): 0 0 0

yn  yn 1  hf  tn 1  12 h, yn 1  12 hf  tn 1 , yn 1  
1 1
2 2 0
0 1
or

yn  yn1  hf tn  1 , yn  1
2 2
 with yn  1  yn 1  12 hf  tn 1 , yn 1  and tn  1  tn 1  12 h
2 2

3. The Heun method ( K  2 , p  2 ): 0 0 0
yn  yn1  12 h  f  tn1 , yn1   f  tn1  h, yn1  hf  tn1 , yn1    1 1
1
0
1
2 2
or
 f  tn1 , yn1   f  tn , yˆ n  
yn  yn1  h  with yˆ n  yn1  hf  tn1 , yn1 
2
Example: The IVP from Section 6.1:

u  t   A  u  t   b  e  t  for t  [0, T ]
may be solved by means of the Heun method using the following algorithm:
uˆ n  u n 1  h  A  u n 1  b  e  tn 1   yˆn  yn1  hf  tn1, yn1 
 A  u n1  b  e  tn 1     A  uˆ n  b  e  tn    f  tn1 , yn1   f  tn , yˆ n 
u n  u n1  h   yn  yn1  h 
2 2
for n  1, 2, ... and u 0  u  0 

4. The Kutta method ( K  3 , p  3 ): 0 0 0 0
1 1
0 0
yn  yn1  16 h  f1  4 f 2  f3  2 2
1 1 2 0
with: f1  f  tn1, yn1  1
6
2
3
1
6
f 2  f  tn1  12 h, yn1  12 hf1 

f3  f  tn1  h, yn1  hf1  2hf 2 
5. The "classical" Runge-Kutta method ( K  4 , p  4 ): 0 0 0 0 0

1 1
yn  yn1  16 h  f1  2 f 2  2 f3  f 4  2 2 0 0 0
1 1
2 0 2 0 0
with: 1 0 0 1 0
f1  f  tn1, yn1  1
6
1
3
1
3
1
6
f 2  f  tn1  12 h, yn1  12 hf1 

f3  f  tn1  12 h, yn1  12 hf 2 
f 4  f  tn1  h, yn1  hf3 

Overview of implicit single-step methods
1. The backward Euler method ( K  1, p  1): 1 1

yn  yn1  h  f  tn , yn  1
2. The family of implicit Butcher (Gauss-Legendre) methods: p  2 K , stability for Re  h   0 .
Example: The implicit Butcher method of order 4 ( K  2 , p  4 ): 1

 3 1
4
1
 3
2 6 4 6
1
2  6
3 1
4  6
3 1
4
1 1
2 2
3. The family of implicit Radau methods: p  2 K  1, stability for Re  h   0 .
Example: The implicit Radau IIA method of order 3 ( K  2 , p  3 ): 1

3
5
12  121
yn  yn1  h  34 f1  14 f 2 
3 1
1 4 4
3 1
4 4
with:
f1  f  tn 1  13 h, yn 1  h  125 f1  121 f 2  
f 2  f  t n 1  h, yn 1  h  34 f1  14 f 2  

When applied to the IVP from Section 6.1:
u t   A  ut   b  et  for t [0,T ]
the Radau IIA method of order 3 yields:
un  un1  h 34 f1  14 f2  for n 1, 2, ... and u0  u0
with:
f1  A  u n1  h  125 f1  121 f 2   b  e tn1  13 h  1
h=2.5
h=1.4
f 2  A  u n1  h  43 f1  14 f 2   b  e tn 

h=0.01
0.8
or:
hA   f1   A  un1  b  e tn1  h
0.6
I  hA 5 1 1
 12
   
12  3
 34 hA 1    
I  4 hA f2   A  un1  b  e tn   0.4
 
0.2
0
0 1 2 3 4 5 6
time [ms]

4. The family of implicit Lobatto methods: p  2 K  2 , stability for Re  h   0 .
Example: The fourth-order implicit Lobatto IIIA method ( K  3 , p  4 ): 0 0 0 0

1
2
5
24
1
3  241
1 2 1
1 6 3 6
1 2 1
6 3 6

6.6. Linear multi-step methods
Definition of linear multi-step methods
 The K -step linear method is defined by the following formula:

K K
yn    k yn  k  h   k  f  tn  k , yn  k  with y0  y  0  , …, yK 1  y   K  1 h 
k 1 k 
where   0 or   1 , K  1, K    , max K , K   K .  
 The method is called explicit if   1 (  0  0 ) since then yn depends only on yn 1 , yn  2 , …, yn  K
and f  tn1, yn1  , f  tn2 , yn2  ,…, f  tn K , yn K  .
 The method is called implicit if   0 (  0  0 ) since then yn depends also on f  tn , yn  ; therefore,

it is necessary to solve at each step a (nonlinear) algebraic equation:
yn  0  f  tn , yn   F  yn1 , yn2 , ...
where:
K K
F  yn 1 , yn  2 , ...    k yn  k  h  k  f  tn  k , yn  k 
k 1 k 1

Design of linear multi-step methods and its local accuracy
For the sake of convenience, the formula defining a designed multi-step method will be re-written in
the form:
K K
yn    k yn  k  h  k  yn  k
k 1 k 
resulting from the substitution yn1  f  tn1, yn1  ,…, yn  K  f  tn K , yn K  .
The simplest method for determination of the parameters  k and k , given K ,  and K  , consists in:
 the development of the RHS of the above formula into the Taylor series at tn ;
 the solution of K  K     1 linear algebraic equations resulting from setting the coefficients at
K  K 
the term without h to 1 and zeroing the coefficients at the terms with h1 , h 2 , …, h .
The above procedure provides, as a by-product, an estimate of the local error, viz. the term with
K  K  1
h   .

Example: The parameters 1 ,  2 and 1 of the following two-step method for solving IVPs:
yn  1 yn 1   2 yn  2  h1 yn 1
satisfy the set of equations:
1   2  1,  1  2 2  1  0, 1
2 1  2 2  1  0
resulting from the development of the RHS into the Taylor series:
yn  1  y n  y n h  12 y nh 2  16 y n h3  ...
  2  y n  2 y n h  2 y n h 2  43 y n h3  ...
 h1  y n  y nh  12 y n h 2  ...
The local error may be estimated as follows:
rn  yn  y n
 1   2  1 y n   1  2 2  1  y n h   12 1  2 2  1  y nh 2    16 1  34  2  12 1  y  h 3
   n
0 0 0
The solution of the equations is: 1  0,  2  1, 1  2 . Therefore:

rn     16 1  43  2  12 1  y n h3   13 y n h3

Stability of multi-step methods
The application of a multi-step method for solving y  t     y  t  results in the difference equation:
K K K
yn   k ynk  h    k  ynk
k 1 k 0
Z
   z Y  z   h z Y  z   F  y , y
k 0
k
k 1
k
k 1
0 1 , ... 
Its solution in the Z domain has the form:

F  y 0 , y 1 , ... 
Y z 1
 K
, where  0  1, and some  k and k are zeros
  z
k 0
k
k
 h k z  k 
This solution is stable if the zeros of its denominator, i.e. of the equation:
K K
 z
k 0
k
k
 h  k z k  0
k 0
are located inside of the unit circle. The border of the region of stability, i.e. the set of h values for
which this condition is satisfied, may be determined by solving the above equation with respect to h
j
and substituting z  e :
K
 e  k
j K  k 
h   kK0 for   0, 2 
 k

k 0
e j  K  k 

Example: The so-called Shichman method (the Gear implicit method of order p  K  2 ) is defined by
the formula:
yn  34 yn 1  13 yn  2  32 hyn
The local accuracy of this method may be assessed as follows:
rn  yn  y n  4
3  y  y  h  y h
n n
1
2 n
2
 16 y nh3  ...
 13  y  2 y h  2 y h
n n n
2
 43 y nh3  ...
 23  y n h   y n
rn  43    16 y nh 3   13    34 y 
nh     9 
3 2 4
9  y nh 3  92 y h3

The equation for the border of the stability region of the method:
yn  34 yn 1  13 yn  2  32 hyn
has the form:
h  1
2  3  4e  j
 e  j 2  for   0, 2 
or:
Re  h   12 3  4cos    cos  2   and Im  h   12  4sin    sin  2   for   0, 2 
% REGION OF STABILITY FOR SHICHMAN METHOD.
% Generation of x-y grid:
N=200;xx=linspace(-1,4,N);yy=linspace(-2.5,2.5,N);
[x,y]=meshgrid(xx,yy);hlambda=x+1j*y;
syms z hl;eq=(3-2*hl)*z^2-4*z+1==0;zz=solve(eq,z);
z1=double(subs(zz(1),hl,hlambda));
Im[h*lambda]
z2=double(subs(zz(2),hl,hlambda));
L=(abs(z1)<=1) & (abs(z2)<=1);
scatter(x(L),y(L),0.1);grid on
axis('equal');axis([-1 4 -2.5 2.5]);

The application of the Shichman method for solving the IVP from Section 6.1:
u   t   A  u  t   b  e  t  for t [0,T ]
requires the solution of the following system of linear algebraic equations at each step:
u n  43 u n 1  13 u n  2  23 h   A  u n  b  e  t n   for n  1, 2, ... and u 0  u  0 
with respect to un , i.e. of the system:
 I  23 hA  un  43 un1  13 un2  23 hbe  tn  for n  1, 2, ... and u 0  u 0 

Explicit Adams methods (Adams-Bashforth methods)
K
yn  yn1  h k f  tnk , ynk  , rn  c p 1 yn p 1 h p 1
k 1
K p c p 1 1 2 3 4 5 6 7
1
1 1  1
2
5 3 1
2 2  
12 2 2
3 23 16 5
3 3  
8 12 12 12
251 55 59 37 9
4 4   
720 24 24 24 24
95 1901 2774 2616 1274 251
5 5   
288 720 720 720 720 720
19087 4277 7923 9982 7298 2877 475
6 6    
60480 1440 1440 1440 1440 1440 1440

Regions of stability of Adams-Bashforth methods
0.8
0.6
0.4
0.2
Im h  
K 2
K 1
K 3
K 4
0
-0.2
-0.4
-0.6
-0.8
-1
-2 -1.5 -1 -0.5 0 0.5
Re h  

Implicit Adams methods (Adams-Moulton methods)
K
yn  yn1  h k  f  tnk , ynk  , rn  c p 1 yn p 1 h p 1
k 0
K p c p 1 0 1 2 3 4 5 6 7
1
0 1 1
2
1 1 1
1 2
12 2 2
1 5 8 1
2 3 
24 12 12 12
19 9 19 5 1
3 4 
720 24 24 24 24
3 251 646 264 106 19
4 5  
360 720 720 720 720 720
863 475 1427 798 482 173 27
5 6  
60480 1440 1440 1440 1440 1440 1440
275 19087 65112 46461 37504 20211 6312 863
6 7   
24192 60480 60480 60480 60480 60480 60480 60480

Regions of stability of Adams-Moulton methods
4
1
Im h  
K 1
K 4
0 K 3
-1
-2
K 2
-3
-4
-6 -4 -2 0 2
Re h  

Explicit Gear's methods
K
yn   k ynk  h  1 f  tn1, yn1  , rn  c p 1 yn p 1 h p 1
k 1
K p c p 1 1 2 3 4 5 6 1
1
1 1  1 1
2
1
2 2  0 1 2
3
1 3 1
3 3   3  3
4 2 2
1 10 1
4 4   6 2 4
5 3 3
1 65 5 1
5 5   10 5  5
6 12 3 4
1 77 3 1
6 6   15 10 5  6
7 10 2 5

Regions of stability of explicit Gear methods
2
1.5
0.5
Im h  
K 2
K 4
K 3
K 1
-0.5
-1
-1.5
-2
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2 2.5
Re h  

Implicit Gear's methods
yn   k ynk  h  0* f tn , yn  , rn  c p 1 yn p 1h p 1

K
k 1
K p c p 1 1 2 3 4 5 6 0
1
1 1 1 1
2
2 4 1 2
2 2 
9 3 3 3
3 18 9 2 6
3 3 
22 11 11 11 11
12 48 36 16 3 12
4 4  
125 25 25 25 25 25
10 300 300 200 75 12 60
5 5  
137 137 137 137 137 137 137
20 360 450 400 225 72 10 60
6 6   
343 147 147 147 147 147 147 147

Regions of stability of implicit Gear methods
8
2
 
K 2
K 3
K 1
K 4
Im h 
-2
-4
-6
-8
-5 0 5 10 15
Re h 

6.7. Implementation of linear multi-step methods
Stiff systems of ODEs
A system of linear ODEs y   Ay is called stiff if:

sup  1 , 2 , ...
 1, where 1 , 2 , ... are eigenvalues of A .
inf  1 , 2 , ...
Example: The system of ODEs:

 667 333
y t   
   y t 
 666  334 
is stiff since 1  1 and 2  1000 .
0
Its solution for y  0     has the form:
3
y1  t   et  e1000t and y2  t   2et  e1000t

The integration of a stiff system of ODEs is problematic due to the following step constraints:
const.
 the stability requirement: sup  1 , 2 , ... h  const.  h  ;
sup  1 , 2 , ...
1 1
 the required interval of integration: T  Nh  h .
inf  1 , 2 , ... N inf  1 , 2 , ...
They imply a severe requirement for the number of steps:

1 const. 1 sup  1 , 2 , ...
h N 
N inf  1 , 2 , ... sup  1 , 2 , ... const. inf  1 , 2 , ...
If the forward Euler method is used for integration of the system from the previous example, then:
1 sup  1 , 2 , ... 1 1000
N     500
const. inf  1 , 2 , ... 2 1
to reproduce the response of the circuits within its one time constant.
If the number of steps needed to find a solution for a given interval  0,T  is increasing, then:
 the number of arithmetic operations needed is increasing,
 the accumulated error due to the rounding errors of operations is increasing.
The above dilemma may be resolved by adaptation of the step size, the adaptation based on
the evaluation of the local error.

Step-doubling method for step control
1  2 h
Let yn be an estimate of the solution obtained for h , and yn – for :
2
p 1
h
h p 1  O  h p  2  and yn
yn1  y  tn   
  2
 y  tn   2      O  h p  2 
the main part  2 
of the error the main part
of the error
The coefficient  may be determined from those equations:
yn   yn 
p 1 2 1
 2 h
1
y  y  2      h p 1
   p
n n
2
   2  1 h p 1
and used for evaluation of the local error:
y n   y n  yn   y n 
2 1 2 1
rn  y n  y  t n   2
1 1 p
and rn  yn  y  t n  
2 2
2 1
p
2p 1
The rules for step selection are as follows:
 1
 if rn  rmax , then double the step h ;
1  2
 if rn  rmax , but rn  rmax , then continue without changing h ;
 if rn   rmax , the divide the step h by 2.

2

Prediction-correction methodology
The main limitation of the step-size control, based on the local error assessment, is implied by
the stability constraints.
Those constraints are much less severe for implicit methods than for explicit methods, but implicit
methods require the solution of a nonlinear algebraic equation at each step.
The computational complexity of this operation may be significantly reduced by using an initial
approximation of the solution obtained by means of a corresponding explicit method (prediction
phase):
K K
yn    k y n  k  h   k f  t n  k , yn  k 
0
k 1 k 1
and next the equation:

K K
yn  h 0 f  tn , yn     k yn  k  h  k f  tn k , yn k 
k 1 k 1
0 
is solved with respect to yn using an iterative method starting at y n (correction phase).


ENUME 2024 - Lecture Notes

Uploaded by

Copyright:

Available Formats

You might also like

ENUME 2024 - Lecture Notes

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

ENUME 2024 - Lecture Notes

Uploaded by

Copyright:

Available Formats

Roman Z.

Numerical Methods (ENUME)

Lecture notes for Spring Semester 2023/2024

R. Z. Morawski: Numerical methods (ENUME) – 1. Introduction 1-1

R. Z. Morawski: Numerical methods (ENUME) – 1. Introduction 1-2

R. Z. Morawski: Numerical methods (ENUME) – 1. Introduction 1-3

R. Z. Morawski: Numerical methods (ENUME) – 1. Introduction 1-4

R. Z. Morawski: Numerical methods (ENUME) – 1. Introduction 1-5

R. Z. Morawski: Numerical methods (ENUME) – 1. Introduction 1-6

R. Z. Morawski: Numerical methods (ENUME) – 1. Introduction 1-7

R. Z. Morawski: Numerical methods (ENUME) – 1. Introduction 1-8

1.2. Vectors and matrices – recapitulation of linear algebra

R. Z. Morawski: Numerical methods (ENUME) – 1. Introduction 1-9

The most important norms of matrices:

A 2  sr  AT A  – the norm induced by x 2

where sr    – the operator of spectra radius determination

R. Z. Morawski: Numerical methods (ENUME) – 1. Introduction 1-10

R. Z. Morawski: Numerical methods (ENUME) – 1. Introduction 1-11

R. Z. Morawski: Numerical methods (ENUME) – 1. Introduction 1-12

R. Z. Morawski: Numerical methods (ENUME) – 1. Introduction 1-13

 the flow diagram:  the sequential notation:

R. Z. Morawski: Numerical methods (ENUME) – 1. Introduction 1-14

Numerical Methods (ENUME)

Lecture notes for Spring Semester 2023/2024

R. Z. Morawski: Numerical methods (ENUME) – 2. Accuracy and complexity of computing 2-1

Example: Subtracting numbers:

369.711 ( 0.0005) – relative error 1.3  104 %

Example: Computing zeros of the polynomial:

R. Z. Morawski: Numerical methods (ENUME) – 2. Accuracy and complexity of computing 2-2

The general form of the floating-point representation:

The rounding principle:

The limit relative error:

The convenient representation of disturbed data:

R. Z. Morawski: Numerical methods (ENUME) – 2. Accuracy and complexity of computing 2-3

R. Z. Morawski: Numerical methods (ENUME) – 2. Accuracy and complexity of computing 2-4

y    x      x  x  12    x  x     y  x   x   2 x    x     

R. Z. Morawski: Numerical methods (ENUME) – 2. Accuracy and complexity of computing 2-5

Example:  n   xn for n  1, 2, ...

  x1a x2b   a 1  b 2 for a , b  eps 1

R. Z. Morawski: Numerical methods (ENUME) – 2. Accuracy and complexity of computing 2-6

Example: Two methods for computing y  x12  x22 :

R. Z. Morawski: Numerical methods (ENUME) – 2. Accuracy and complexity of computing 2-7

y1   x12 1  21  1   x22 1  2 2   2  1  3 

  x12 x22    x12 x22  

R. Z. Morawski: Numerical methods (ENUME) – 2. Accuracy and complexity of computing 2-8

The above assessment differs from that derived for A1 :

x12  x22  x12  x22 

only by the component generated by rounding errors:

The model of the propagation of error across an elementary operation:

in the final result of computation, caused by the errors in the data:

R. Z. Morawski: Numerical methods (ENUME) – 2. Accuracy and complexity of computing 2-10

R. Z. Morawski: Numerical methods (ENUME) – 2. Accuracy and complexity of computing 2-11

R. Z. Morawski: Numerical methods (ENUME) – 2. Accuracy and complexity of computing 2-12

The model of the propagation of error across an elementary operation:

R. Z. Morawski: Numerical methods (ENUME) – 2. Accuracy and complexity of computing 2-13

An algorithm A  d  for d  D is numerically correct if:

R. Z. Morawski: Numerical methods (ENUME) – 2. Accuracy and complexity of computing 2-15

Example: The "classical" algorithm for solving quadratic equations:

is numerically unstable because of the subtracting close numbers when x2  0 :