Basic Concepts in Numerical Analysis
Outline
Basic Concepts in
Numerical Analysis
Larry Caretto
Mechanical Engineering 501AB
Seminar in Engineering Analysis
November 6, 2017
Review Last Class
• Phase plots, critical points, and stability
• Look at system of two linear
homogenous, autonomous equations
– dy/dt = Ay (no function of time)
• Critical points and stability depend on
matrix eigenvalues which depend on
determinant properties
• Described various critical points: node,
center, saddle point and spiral
3 4
Interpolation
• Start with N data pairs xi, yi
• Find a function (polynomial) that can be
used for interpolation
• Basic rule: the interpolation polynomial
must fit all points exactly
• Denote the polynomial as p(x)
• The basic rule is that p(xi) = yi
• Many different forms
5 6
ME 501A Seminar in Engineering
Analysis
November 6, 2017
• Review last class
• Midterm Exam November 15 covers
material on differential equations and
Laplace transforms (no phase plots)
• Overview of numerical solutions
– Initial value problems in first-order
equations
– Systems of first order equations and initial
value problems in higher order equations
– Boundary value problems
– Stiff systems and eigenvalues
2
Numerical Analysis Problems
• Numerical solution of algebraic
equations and eigenvalue problems
• Solution of one or more nonlinear
algebraic equations f(x) = 0
• Linear and nonlinear optimization
• Constructing interpolating polynomials
• Numerical quadrature
• Numerical differentiation
• Numerical differential equations
Newton Polynomials
• p(x) = a0 + a1(x – x0) + a2(x – x0)(x – x1)
+ a3(x – x0)(x – x1)(x – x2) + …
+ an-1(x – x0)(x – x1)(x – x2) … (x – xn-2)
– n – 1 data points numbered 0 to n – 2
• Terms with factors of x – xi are zero
when x = xi
– Have p(xi) = yi to find ai, i = 0 to n – 1
• a0 = y0, a1 = (y1 – y0) / (x1 – x0)
• y2 = a0 + a1(x2 – x0) + a2(x2 – x0)(x2 – x1)
– Solve for a2 using results for a0 and a1
Page 1
Basic Concepts in Numerical Analysis November 6, 2017

Newton Polynomials II Divided Difference Table

• y2 = a0 + a1(x2 – x0) + a2(x2 – x0)(x2 – x1) • Enter data on xi and yi in rows of table
y y skipping one row between entries
y2  y0  1 0 ( x2  x0 )
y2  a0  a1 ( x2  x0 ) x1  x0 • Start with yi data as zeroth divided
a2  
( x2  x0 )( x2  x1 ) ( x2  x0 )( x2  x1 ) difference
• Could continue in this fashion to • First divided difference, Fi = (yi+1 – yi) /
determine coefficients from data (xi+1 – xi)
• Use alternative scheme – not derived – Second (or later) divided difference is
difference of first (or later) differences
here – known as divided difference
table to compute ak from same data – ai coefficients are initial divided differences
7 8

Divided Difference Table Divided Difference Example

x0 y0 a0 0 0 a0
y1  y0 10  0
F0  a1 F0   1 a1
x1  x0 10  0
F1  F0 a2 3 1
x1 y1 S0  10 10 S0   .1 a2
x2  x0 20  0
y y S S 40  10 .15  .1
F1  2 1 T0  1 0 F1  3 T0  
1
x2  x1 x3  x0 20  10 30  0 600
F2  F1  a3 63  a3
x2 y2 S1  20 40 S1   .15
x3  x1 30  10
y3  y 2 S 2  S1 100  40
F2  T1  F2  6
x3  x2 x4  x1 30  20
F3  F2
x3 y3 S2  30 100
x4  x2
y  y3
F3  4 9 10
x4  x3

Divided Difference Example II Start at Any Point in Data Table

• Divided difference table gives a0 = 0, a1 x-1 y-1
= 1, a2 = .1, and a3 = 1/600
• Polynomial p(x) = a0 + a1(x – x0) + a2(x
x0 y0 a0
– x0)(x – x1) + a3(x – x0)(x – x1)(x – x2)
y1  y0
= 0 + 1(x – 0) + 0.1(x – 0)(x – 10) + F0  a1
x1  x0
(1/600)(x – 0)(x – 10)(x – 20) = x + F1  F0
x1 y1 S0  a2
0.1x(x – 10) + (1/600)x(x – 10)(x – 20) x2  x0
y2  y1 S1  S 0
• Check p(30) = 30 + .1(30)(20) + (1/600) F1  T0 
x2  x1 x3  x0
(30)(20)(10) = 30 + 60 + 10 = 100 (correct) F2  F1
x2 y2 S2   a3
y  y2 x3  x1
11
F2  3 12
x3  x2

ME 501A Seminar in Engineering

Analysis Page 2
Basic Concepts in Numerical Analysis November 6, 2017

Difference Example from x = 10 Divided Difference Calculation II

0 0 Final data point • Divided difference table gives a0 = 10,
not shown is x a1 = 3, a2 = .15, and a3 = 1/600
= 40, y = 200
• Polynomial p(x) = a0 + a1(x – x0) + a2(x
10 10 a0
– x0)(x – x1) + a3(x – x0)(x – x1)(x – x2)
40  10
F0   3 a1 = 10 + 3(x – 10) + 0.15(x – 10)(x – 20) +
20  10
63 (1/600)(x – 10)(x – 20)(x – 30) = x +
20 40 S0   .15a2
30  10 0.1x(x – 10) + (1/600)x(x – 10)(x – 20)
100  40 1 / 5  .15 1
F1  6 T0   • Check p(40) = 10+3(30)+.15(30)(20) +
30  20 40  10 600
10  6 1
30 100 S1    a3 (1/600)(40)(30)(20)=10+90+90+10=200
40  20 5 (correct)
200  100
F2   10 13 14
40  30

Divided Difference Code Constant Step Size

for ( i = 0; i < n; i++ ) • Divided differences work for equal or
D[0][i] = y[i]; unequal step size in x
• If x = h is a constant we have simpler
for ( k = 1; k < n; k++ ) results
for ( i = 0; i < n – k; i++ ) – Fk = yk/h = (yk+1 – yk)/h
D[k][i] = ( D[k-1][i+1] – – Sk = 2yk/2h2 = (yk+2 – 2yk-1 + yk)/2h2
D[k-1][i] ) / ( x[i+k] – x[i] );
– Tk = 3yk/6h3 = (yk+3 – 3yk+2 + 3yk+1 – yk)/6h3
– nyk is called the nth forward difference
• D[k][i] is ith value of kth divided difference
• Code for n data points (0 to n–1) – Can also define backwards and central
differences
15 16

Interpolation Approaches Cubic Spline Overview

• When we have N data points how do we
interpolate among them?
– Order N-1 polynomial not good choice
– Use piecewise polynomials of lower order
(linear or quadratic)
– Can match first and or higher derivatives
where piecewise polynomials join
– Cubic splines are piecewise cubic
polynomials that match first and second
derivatives [as well as (xk,yk) values]
17
• Have N cubic polynomials, ai + bix + cix2
+ dix3, with end point of 1 polynomial the
start of next, requires N + 1 data points
– Data points numbered 0 to N with
polynomials numbered 1 to N
• Need 4N equations to get N values for
polynomial coefficients: ai, bi, ci, and di
• Each polynomial fits data points at ends:
pk(xk-1) = yk-1 and pk(xk) = yk, k = 1, N
18

ME 501A Seminar in Engineering

Analysis Page 3
Basic Concepts in Numerical Analysis November 6, 2017

Cubic Spline Overview II Splines in MATLAB

• Have continuity of first and second
derivatives: pk-1’(xk) = pk’(xk) and pk-1’’(xk)
= pk’’(xk)
• Matching data points gives 2N equations
and derivative continuity gives 2N – 2
• Have 4N – 2 equations for 4N unknown
polynomial coefficients
• Different models of end point behavior
used to provide additional 2 equations
19
• Use spline function in MATLAB to get
one or more interpolated points
– xIn is array of y data for spline fit
– yIn is array of x data for spline fit
– Apply spline to x which can be a single data
point or an array using command below
>> spline(xIn, yIn, x)
– Generally uses not-a-knot end slopes
• Also has routine unmkpp to get details
of resulting spline coefficients
20

Cubic Spline Interpolation

Newton Interpolating Polynomial
5
0.7

0.6 4

0.5
3 Polynomial
Y Value
y value

Known f'
0.4 Data
Natural
2
No Knot
0.3 Data
1
0.2

0.1 0

0 -1
0 1 2 3 4 5 6 0 1 2 3 4 5 6

x values X Values
Results show effect of different High-order polynomials can
methods used to treat end points 21
give unrealistic fits to data 22

Polynomial Applications Derivative Expressions

• Data interpolation • Obtain from differentiating interpolation
• Approximation functions in numerical polynomials or from Taylor series
quadrature and solution of ODEs • Series expansion for f(x) about x = a
• Basis functions for finite element
df 1 d2 f 1 d3 f
methods f ( x )  f (a)  ( x  a)  ( x - a) 2  ( x - a)3  ....
dx xa 2! dx 2 xa
3! dx3 xa
• Can obtain equations for numerical
differentiation • Note: d0f/dx0 = f 
1 dn f
f ( x)   ( x - a) n
• Statistical curve fitting (not discussed and 0! = 1 n 0 n ! dx n xa
here) usually used in practice
• What is error from truncating series?
23 24

ME 501A Seminar in Engineering

Analysis Page 4
Basic Concepts in Numerical Analysis November 6, 2017

Truncation Error Derivative Expressions

• If we truncate series after m terms • Look at finite-difference grid with equal
m
1 dn f 
1 dn f spacing: h = x so xi = x0 + ih
f ( x)   n
( x - a) n   n
( x - a) n
n  0 n ! dx xa n  m 1 n ! dx x a • Taylor series about x = xi gives f(xi + kh)
Terms used Truncation error, m = f[x0 + (i+k)h] = fi+k in terms of f(xi) = fi
• Use theorem of mean to write truncation df 1 d2 f 1 d3 f
f ( xi  kh)  f ( xi )  kh  (kh) 2  (kh)3  .....
error as single term at unknown dx x  xi 2! dx 2 x  xi
3! dx 3 x  xi

location, , between x and a • Compact derivative notation


1 dn f 1 d m1 f
m  
n  m 1 n ! dx
n
( x - a) n 
(m  1) ! dx m1
( x - a) m 1 fi' 
df
f i '' 
d2 f
... f i n k 
dn f
xa x  dx x  xi dx 2 x  xi
dx n x  xi  k
25 26

Derivative Expressions II Derivative Expressions III

• Combine all definitions for compact • Apply general f i '' (kh) 2 f i ''' ( kh) 3
f i  k  f i  f i ' kh   .
series notation equation for k 2! 3!
df 1 d2 f 1 d3 f
= 1 and k = –1
f ( xi  kh)  f ( xi )  kh  (kh) 2  (kh)3  .....
dx 2! dx 2 3! dx 3 f i '' h 2 f i ''' h 3
x  xi x  xi x  xi
f i 1  f i  f i ' h    .....
f i '' (kh) 2 f i ''' ( kh) 3 2! 3! f '' h 2 f i ''' h 3
f i  k  f i  f i ' kh    ..... f i 1  f i  f i ' h  i   .....
2! 3! 2! 3!
f f ''
f h f h '''
f f
• Use this formula to get expansions for f i '  i 1 i  i  i  .....  i 1 i  Ah Forward
h 2! 3! h
various grid locations about x = xi and
f i  f i 1 f i '' h f i ''' h f  f i 1
use results to get derivative expressions fi'     .....  i  Ah
h 2! 3! h Backward
27 28

Derivative Expressions IV Order of the Error

• Subtract fi+1 and fi-1 expressions • Forward and backward derivative have
'' 2 ''' 3
fi h f h error term that is proportional to h
fi 1  fi  f i 'h   i  .....
2! 3! f ''h 2 f '''h3 • Central difference error is proportional
fi 1  fi  fi 'h  i  i  ..... to h2
2! 3!
2 f '''h3 2 f i '''h5 • Error proportional to hn called nth order
fi 1  f i 1  2 f i 'h  i   .....
3! 5! • Reducing step size by a factor of 
f f f '''h 2 fi '''''h 4 f f reduces nth order error by n
fi '  i 1 i 1  i   .....  i 1 i 1  Ah 2 n
2h 3! 5! 2h  h2 
• Result called central difference expression  2   1  
29  h1  30

ME 501A Seminar in Engineering

Analysis Page 5
Basic Concepts in Numerical Analysis November 6, 2017

Order of the Error Notation Higher Order Derivatives

• Write the error term for nth error term as • Add fi+1 and fi-1 expressions; solve for f i
''

O(hn) ''
fi h 2
f h ''' 3

– Big oh notation, O, denotes order f i 1  f i  f i ' h   i  .....

2! 3! f '' h 2 f i ''' h 3
– Recognizes that factor multiplying hn may f i 1  f i  f i ' h  i   .....
2! 3!
change slightly with h
f '' h 2 2 f i '''' h 4 2 f i '''''' h 6
f i 1  f i 1  2 f i  2 i    .....
First order forward First order backward 2! 4! 6!
f f f i  f i 1
f i  i 1 i  O (h) f i 1  f i 1  2 f i f i '''' h 2 f i '''''' h 4 f  f  2 fi
 
'
fi'   O (h)
h h f i ''     .....  i 1 i 21  O h2
h2 3! 5! h
Second order central f i 1  f i 1 • f’’ is second-order, central difference
fi'   O(h 2 )
2h expression for second derivative
31 32

Higher Order Directional Specific Taylor Series

• We can get higher truncation error • General f '' (kh) 2 f i ''' ( kh) 3
f i  k  f i  f i ' kh  i   .....
expressions at the expense of more equation 2! 3!
computations f i '' h 2 f ''' h 3
• k=2 f i2  f i  2 fi 'h  4 8 i  .....
• Get second order forward and backward 2! 3!
'' 2 ''' 3
derivative expressions from previous fi 2  fi  2 fi 'h  4 i
f h
8 i
f h
 .....
results and fi+2 and fi-2, respectively • k = -2 2! 3!
'' 2 ''' 3
f h f h
• Combine fi+2 and fi-2 equations with • k=3 f i 3  f i  3 f i ' h  9 i  27 i  .....
2! 3!
previous expressions for fi+1 and fi-1 to f '' 2
h f ''' 3
h
• k = -3 f i 3  f i  3 f i ' h  9 i  27 i  .....
eliminate first order error term 2! 3!
33 34

Second Order Forward Second Order Backwards

• Subtract 4fi+1 from fi+2 to eliminate h2 • Add 4fi-1 to –fi-2 to eliminate h2 term
term  '' h
2
''' h
3
  h2 h3 
f i  2  4 f i 1   f i  2 f i h  4 f i
'
 8 fi  ...  f i  2  4 f i 1    f i  2 f i ' h  4 f i ''  8 f i '''  ...
 2 6   2 6 
 h 2
h 3
 h3  h 2
h 3
 h3
 4 f i  f i ' h  f i ''  f i '''  ...  3 f i  2 f i 'h  4 f i '''  ...  4 f i  f i ' h  f i ''  f i '''  ..  3 f i  2 f i ' h  4 f i '''  ...
 2 6  6  2 6  6

h3 Second h3 Second
f i  2  4 f i 1  3 f i  2hf i '  4 f i '''  ... order  f i  2  4 f i 1  3 f i  2hf i '  4 f i '''  ...
6 6 order
error error
 f i  2  4 f i 1  3 f i h2 f i 2  4 f i 1  3 f i h2
fi '   f i '''  ... fi '   f i '''  ...
2h 3 2h 3
35 36

ME 501A Seminar in Engineering

Analysis Page 6
Basic Concepts in Numerical Analysis November 6, 2017

Other Derivative Expressions Other Derivative Formulas

• Can continue in this fashion  11 f i  18 f i 1  9 f i  2  2 f i 3 h3
– Write Taylor series for fi+1, fi-1, fi+2, fi-2, fi+3, fi '   f i ''''  ...
fi-3, etc. 2h 3
– Create linear combinations with factors that 11 f i  18 f i 1  9 f i  2  2 f i 3 '''' h
3
fi 
'
 fi  ...
eliminate desired terms 2h 4
– Eliminate fi term to obtain central difference
– Keep only terms in fk with k  i for forward f i  2  8 f i 1  8 f i 1  f i  2 h4
fi '   f i '''''  ...
difference expressions 2h 30
– Keep only terms in fk with k  i for forward
 f i  2  16 f i 1  30 f i  16 f i 1  f i  2 h4
difference expressions f i ''  2
 f i ''''''  ...
– Results in numerical analysis texts/online 12h 90
37 38

Order of Error Examples Roundoff Error

• Table 1 in online notes shows error in • Possible in derivative expressions from
first derivative for ex around x = 1 subtracting close differences
– Using first- and second-order forward and • Example f(x) = ex: f’(x)  (ex+h – ex-h)/(2h)
second-order central differences and error at x = 1 is (e1+h – e1-h)/(2h) - e
– Step h = 0.4, 0.2, and 0.1
– Error ratio for doubling step size 3.004166  2.722815
E  2.718282  4.5 x10 3
• 4.01 to 4.02 for central differences 2(0.1)
• 2.07 to 2.15 for first-order forward differences
Second order error
2.7185536702  2.7180100139
• 4.32 to 4.69 for second-order forward E  2.718281828459  4.5 x10 9
2(0.0001)

log 2 
 1   log( 2 )  log( 1 )  d log( ) 2.71828210028724  2.71828155660388
n E  2.718281828  5.9 x109
log 2  log(h2 )  log(h1 ) d log(h)
h 2(0.0000001)
 h1 
39 40

Figure 2-1. Effect of Step Size on Error Richardson Extrapolation

1.E+01
1.E+00 Roundoff
• Uses finite-difference method with two
1.E-01
Error Truncation step sizes to get improved accuracy
1.E-02
1.E-03 Dominates Error • Start with E = F(h) + TE = F(h) + O(hn)
1.E-04 ࢊ࢒࢔ሺࣕሻ
Error

1.E-05 = – E is exact result

ࢊ࢒࢔ሺࢎሻ
1.E-06
1.E-07 Const – F(h) is finite difference approximation with
1.E-08 step size h
1.E-09 Roundoff
1.E-10 Error Starts – Truncation error, TE, is O(hn)
1.E-11
1.E-17 1.E-15 1.E-13 1.E-11 1.E-09 1.E-07 1.E-05 1.E-03 1.E-01
– Actually have an infinite series for error
hn  d n f  
hn  d n f  
Step Size TE       n    Ak h k
41 n!  dx n  x  k n n!  dx  xa k n 42

ME 501A Seminar in Engineering

Analysis Page 7
Basic Concepts in Numerical Analysis November 6, 2017

Richardson Extrapolation II Richardson Extrapolation III

• Look at evaluating error with two step • Solve equation from previous slide for E
sizes, h and kh  
k n E  E  k n F h   F kh   O h n 1
F h   F kh 
 Oh   RE  Oh 
n
– Exact value will not change k n 1 n 1
E
– Create sum to display

first error term k 1
n

E  F h   TE  F h    Am h m  F h   An h n  A h m
m
• The formula for the Richardson extrapo-
mn m  n 1

lation, RE, has a higher order of the error
E  F kh   TE    F kh   An kh    A kh
n m
m
m  n 1 – Truncation error for RE shown below
kn
– Multiply first equation by and subtract the   Bm 
A h   A kh
m
second equation to eliminate the An term kn m
m m  kn  km
k E  E  k F h   F kh   k An h  An kh   O h
n n n n n
 
n 1
TE  k  n 1
k n 1
k  n 1
 B m h m
k n 1
Am
k  n 1
May be hn+243 44

Richardson Extrapolation IV Richardson Extrapolation V

• Richardson extrapolation for forward
• What does this mean?
dcos(x)/dx at x = 1 and h = 0.1 & h = 0.2
k n F h   F kh 
E
k n 1
   
 O h n 1  RE  O h n 1 – What are k and n? k = h2/h1 = 2; n = order = 1
f i 1  f i k n F h   F kh 
• E is the exact result, F(h) is a finite fi ' 
h
 O h1   RE 
k n 1

difference result with step size h cos1.1  cos1
 0.8670618 2  0.86706  0.88972
1
f i ' h  0.1 
– If we have two nth-order finite difference 0.1 21  1
cos1.2  cos1
results, with two step sizes h and kh, we f i ' h  0.2    0.8897228  0.84440093
can use this formula to get an improved 0.2
result with an error order of n + 1 (or higher • Extrapolation closer to correct value of
if the error term has every other power of h) dcos(x)/dx|x=1 = –sin(1) = –0.84147098
45 46

Richardson Extrapolation VI
• Richardson extrapolation for central
dcos(x)/dx at x = 1 and h = 0.1 & h = 0.2
– What are k and n? k = h2/h1 = 2; n = order = 2
f i 1  f i 1 k n F h   F kh 
fi ' 
2h
 
 O h2 RE 
k n 1


cos1.1  cos0.9
f i ' h  0.1   0.84006922  0.84007   0.83587
1

20.1
21  1
cos1.2  cos0.8
f i ' h  0.2    0.8358872  0.841468
20.2
• Extrapolation closer to correct value of
dcos(x)/dx|x=1 = –sin(1) = –0.84147098
47

ME 501A Seminar in Engineering

Analysis Page 8