University of Technology

Petroleum Technology Department

Numerical Analysis

Lecture 1: Introduction to Numerical Analysis Semester 1 (2020-2021)

Doaa Saleh Mahdi

Lecturer
Doaa Saleh Mahdi
• Teaching Experience

❖ Petroleum Production Technology

❖ Reservoir Engineering
❖ Reservoir simulation
❖ Integrated Field Development and Management

• Contact details:

❖Email: 150082@uotechnology.edu.iq

❖
 Lecture Content
• Introduction
• Use of numerical methods in a variety of petroleum engineering
problems
• The Purpose of Numerical methods in Petroleum Engineering
• Noncomputer Methods
 Introduction

• The primary objective of this chapter is to introduce you to

mathematical modeling and its role in engineering problem solving.

• We will also illustrate how numerical methods process.

 Use of Numerical methods in a variety
of petroleum engineering problems
• Numerical methods are techniques by which mathematical problems
are formulated so that they can be solved with arithmetic operations.
• Although there are many kinds of numerical methods, they have one
common characteristic: they invariably involve large numbers of
tedious arithmetic calculations.
• It is little wonder that with the development of fast, efficient Digital
computers, the role of numerical methods in engineering problem
solving has increased dramatically in recent years.
 Use of Numerical methods in a variety
of petroleum engineering problems
• Considerable practical experience in reservoir engineering and
computer simulation is required in order to take full advantage of the
technology of reservoir simulation.
• Because of the fact that assumptions are involved at several stages of
the process of model development and the flow equations can only
be modelled approximately.
• it is necessary to use good engineering judgement in conducting
reservoir studies and in the interpretation of results obtained.
 The Purpose of Numerical methods
in Petroleum Engineering
• In order to be able to develop a hydrocarbon reservoir optimally it is
necessary to be able to quantify the relative performance of the
reservoir under different depletion schemes.
• Although this might seem a logical enough statement it must be
appreciated that optimally here does not mean a ‘technical’ optimum
in the sense of maximising the number of barrels recovered, rather it
means an ‘economical’ optimum reflecting the return on an
investment.
• It should always be appreciated that hydrocarbon reserves, whether
on a world-wide basis or on an individual well basis, have a price tag
associated with them and the volume of reserves will fluctuate
depending on the perceived future value of the commodity.
 The Purpose of Numerical methods
in Petroleum Engineering
• With this in mind it is clearly not possible to determine either the
technical or economical optimum on a trial-and-error basis with the
reservoir itself; one must either predict the future performance
analytically or numerically.
• Analytical approaches to reservoir analysis and performance
prediction have already been presented in some detail in the
Reservoir Engineering Modules. In summary they are:
❖The Material Balance approach (zero-dimensional)
❖The Buckley Leverett / Welge description of immiscible flow (one-
dimensional)
❖The method of pseudo functions (reduces a two-dimensional problem to
one-dimension)
 The Purpose of Numerical methods
in Petroleum Engineering
• All of these methods are excellent diagnostic tools but cannot hope to
capture all the nuances and accurately project the future performance of
large, complicated reservoir systems.
• In particular there is no substantial geological information carried in these
analytical approaches. (In the case of Material Balance there is none
whatsoever!) As such these approaches can say very little, if anything,
about, the optimum placement of future wells and it is therefore highly
unlikely that any expensive engineering decisions will be made on the
strength of such rudimentary modelling alone.
• Clearly, an analytical approach is useful in performing screening studies,
and as a lead into a more in depth investigation, but it will be hard pressed
to come up with anything meaningful when trying to model complex
situations since much of the physics of the situation will not be captured.
 The Purpose of Numerical methods
in Petroleum Engineering
• On a slightly softer but nonetheless important note, the best reservoir
simulators of the current generation are, in fact, excellent data
administrators.
• They are able to display data to assist in error checking and can manipulate
large volumes of data efficiently.
• This can give some reassurance to the Engineer that data will at least be
handled consistently throughout his study.
• The audit trail of a reservoir simulation study makes information easier to
share with other disciplines and other organisations, and helps promote
technical credibility.
• The plethora of “spreadsheet” engineering, whilst providing much
valuable information and insight, is often extremely difficult to follow by
anyone other than the original author.
 The Purpose of Numerical methods
in Petroleum Engineering
• A final point in favour of reservoir simulation is that it can provide
answers to a wide range of “what if” questions efficiently.
• Comparing the outcomes of various alternative development
schemes with the current best view of the situation allows the
development to be optimised in a systematic fashion.
• It is the ‘relative’ information gained through simulation that is most
important in being able to maximise the return of a development.
• Although the ‘absolute’ answer to the nearest barrel will never be
known it is obviously extremely useful to be able to rank alternatives
to maximise the chances of realising the full potential value of the
hydrocarbon resource.
 Noncomputer Methods
• Beyond providing increased computational firepower, the widespread
availability of computers (especially personal computers) and their
partnership with numerical methods has a significant influence on the
actual engineering problem-solving process.
• In the pre-computer Era there were generally three different ways in
which engineers approached Problem solving:
 Noncomputer Methods
1) Solutions were derived for some problems using analytical, or
exact, methods:
• These Solutions were often useful and provided excellent insight into
the behavior of some Systems.
• However, analytical solutions can be derived for only a limited class
of Problems.
• These include those that can be approximated with linear models
and those that have simple geometry and low dimensionality.
• Consequently, analytical solutions are of limited practical value
because most real problems are nonlinear and involve Complex
shapes and processes.
 Noncomputer Methods
2) Graphical solutions were used to characterize the behavior of
systems:
• These graphical Solutions usually took the form of plots or no
mographs.
• Although graphical techniques can often be used to solve complex
problems, the results are not very precise.
• Furthermore, graphical solutions (without the aid of computers) are
extremely tedious and awkward to implement.
• Finally, graphical techniques are often limited to problems that can
be described using three or fewer dimensions.
 Noncomputer Methods
3) Calculators and slide rules
• These were used to implement numerical methods manually.
• Although in theory such approaches should be perfectly adequate for
solving complex Problems, in actuality several difficulties are
encountered.
• Manual calculations are Slow and tedious.
• Furthermore, consistent results are elusive because of simple
Blunders that arise when numerous manual tasks are performed.
 University of Technology
Petroleum Technology Department

Numerical Analysis
Lecture 2: Analytical and Numerical methods Semester 1 (2020-2021)

Doaa Saleh Mahdi

 Lecture Content
• Numerical Methods and Engineering Practice
• A simple mathematical model
• Conservation Laws and Engineering
• Examples
Numerical Methods and Engineering
Practice
• Since the late 1940s the widespread availability of digital computers
has led to a veritable explosion in the use and development of
numerical methods.
• At first, this growth was somewhat limited by the cost of access to
large mainframe computers, and, consequently, many engineers
continued to use simple analytical approaches in a significant portion
of their work.
• Needless to say, the recent evolution of inexpensive personal
computers has given us ready access to powerful computational
capabilities.
Numerical Methods and Engineering
Practice
There are several additional reasons why you should study numerical
methods:
1. Numerical methods are extremely powerful problem-solving tools. They
are capable of handling large systems of equations, nonlinearities, and
complicated geometries that are not uncommon in engineering practice
and that are often impossible to solve analytically. As such, they greatly
enhance your problem-solving skills.
2. During your careers, you may often have occasion to use commercially
available Prepackaged, or “canned,” computer programs that involve
numerical methods. The intelligent use of these programs is often
predicated on knowledge of the basic theory underlying the methods.
3. Many problems cannot be approached using canned programs. If you are
conversant with numerical methods and are adept at computer
programming, you can design your own programs to solve problems
without having to buy or commission expensive software.
Numerical Methods and Engineering
Practice
4. Numerical methods are an efficient vehicle for learning to use computers. It is
well known that an effective way to learn programming is to actually write
computer programs. Because numerical methods are for the most part
designed for implementation on computers, they are ideal for this purpose.
Further, they are especially well-suited to illustrate the power and the
limitations of computers. When you successfully implement numerical
methods on a computer and then apply them to solve otherwise intractable
problems, you will be provided with a dramatic demonstration of how
computers can serve your professional development. At the same time, you
will also learn to acknowledge and control the errors of approximation that are
part and parcel of large-scale numerical calculations.
5. Numerical methods provide a vehicle for you to reinforce your understanding
of Mathematics. Because one function of numerical methods is to reduce
higher Mathematics to basic arithmetic operations, they get at the “nuts and
bolts” of some otherwise obscure topics. Enhanced understanding and insight
can result from this Alternative perspective.
 Mathematical Models
• Knowledge and understanding are prerequisites for the effective
implementation of any tool. No matter how impressive your tool
chest, you will be hard-pressed to repair a car if you do not
understand how it works. This is particularly true when using
computers to solve engineering problems.
• Although they have great potential utility, computers are practically
useless without a fundamental understanding of how engineering
systems work.
• This understanding is initially gained by empirical means—that is, by
observation and experiment. However, while such empirically derived
information is essential, it is only half the story.
 Mathematical Models
• Over years and years of observation
and experiment, engineers and
scientists have noticed that certain
aspects of their empirical studies
occur repeatedly.
• Such general behavior can then be
expressed as fundamental laws that
essentially embody the cumulative
wisdom of past experience.
• Thus, most engineering problem
solving employs the two split
approaches of observation and
theoretical analysis (Fig. 1).
 Mathematical Models
• Thus, a mathematical model can be broadly defined as a formulation or
equation that expresses the essential features of a physical system or
process in mathematical terms.
• In a very general sense, it can be represented as a functional relationship of
the form
Dependent variable = f (independent variables, parameters, forcing
function) (1)
• Where the dependent variable is a characteristic that usually reflects the
behavior or state of the system; the independent variables are usually
dimensions, such as time and space, Along which the system’s behavior is
being determined; the parameters are reflective of the System’s properties
or composition; and the forcing functions are external influences acting
Upon the system.
 Mathematical Models
• The actual mathematical expression of Eq. (1) can range from a simple
algebraic relationship to large complicated sets of differential
equations.
• For example, on the basis of his observations, Newton formulated his
second law of motion, which states that the time rate of change of
momentum of a body is equal to the resultant force acting on it

• The mathematical expression, or model, of the second law is the well-

known equation
• F = ma 2
 Mathematical Models
• Where F = net force acting on the body (N, or kg m/s2), m = mass of
the object (kg), and a = its acceleration (m/s2).
• The second law can be recast in the format of Eq. (1) by merely
dividing both sides by m to give
• a = F/m (3)
• Where a = the dependent variable reflecting the system’s behavior, F =
the forcing function and m = a parameter representing a property of the
system
 Mathematical Models
• Note that for this simple case there is no independent variable
because we are not yet predicting how acceleration varies in time or
space. Equation (3) has several characteristics that are typical of
mathematical models of the physical world:
1. It describes a natural process or system in mathematical terms.
2. It represents an idealization and simplification of reality. That is, the
model ignores negligible details of the natural process and focuses
on its essential manifestations. Thus, the second law does not
include the effects of relativity that are of minimal importance when
applied to objects and forces that interact on or about the earth’s
surface at velocities and on scales visible to humans.
 Mathematical Models
3. Finally, it yields reproducible results and, consequently, can be used
for predictive purposes. For example, if the force on an object and
the mass of an object are known, Eq. (3) can be used to compute
acceleration. Because of its simple algebraic form, the solution of Eq.
(2) can be obtained easily. However, other mathematical models of
physical phenomena may be much more complex, and either cannot
be solved exactly or require more sophisticated mathematical
techniques than simple algebra for their solution.

• To illustrate a more complex model of this kind, Newton’s second law

can be used to determine the terminal velocity of a free-falling body
near the earth’s surface. Our falling body will be a parachutist.
 Mathematical Models
• A model for this case can be derived by expressing the acceleration as
the time rate of change of the velocity (dv/dt) and substituting it into
Eq. (3) to yield:
(dv/dt)=F/m (4)
• Where v is velocity (m/s) and t is time (s). Thus, the mass multiplied by
the rate of change of the velocity is equal to the net force acting on
the body. If the net force is positive, the object will accelerate. If it is
negative, the object will decelerate. If the net force is zero, the object’s
velocity will remain at a constant level.
• Next, we will express the net force in terms of measurable variables
and parameters.
 Mathematical Models
• For a body falling within the vicinity of the earth, the net force is
composed of two opposing forces: the downward pull of gravity FD
and the upward force of air resistance FU:
F=FD+FU (5)

• If the downward force is assigned a positive sign, the second law can
be used to formulate the force due to gravity, as
FD=mg (6)
• Where g = the gravitational constant, or the acceleration due to
gravity, which is approximately equal to 9.8 m/s2.
 Mathematical Models
• Air resistance can be formulated in a variety of ways. A simple
approach is to assume that it is linearly proportional to velocity and
acts in an upward direction, as in
• FU=-CV (7)
• where c = a proportionality constant called the drag coefficient (kg/s).
Thus, the greater the fall velocity, the greater the upward force due to
air resistance. The parameter c accounts for properties of the falling
object, such as shape or surface roughness, that affect air resistance.
For the present case, c might be a function of the type of jumpsuit or
the orientation used by the parachutist during free-fall.
• The net force is the difference between the downward and upward
force.
 Mathematical Models
• Therefore,
• Eqs. (4) through (7) can be combined to yield:
(8)

• or simplifying the right side:

• (9)
 Mathematical Models
• Equation (9) is a model that relates the acceleration of a falling object
to the forces acting on it. It is a differential equation because it is
written in terms of the differential rate of change (dv/dt) of the
variable that we are interested in predicting.
• However, in contrast to the solution of Newton’s second law in Eq. (3),
the exact solution of Eq. (9) for the velocity of the falling parachutist
cannot be obtained using simple algebraic manipulation.
• Rather, more advanced techniques such as those of calculus, must be
applied to obtain an exact or analytical solution.
 Mathematical Models
• For example, if the parachutist is initially at rest (v = 0 at t = 0), calculus
can be used to solve Eq. (9) for:
(10)

• Note that Eq. (10) is cast in the general form of Eq. (1), where v(t) = the
dependent variable, t = the independent variable, c and m =
parameters, and g = the forcing function.
 Analytical Solution to the Falling
Parachutist Problem
Example 1: A parachutist of mass 68.1 kg jumps out of a stationary hot
air balloon. Plot the velocity-time relationship and compute the treminal
velocity. The drag coefficient is equal to 12.5 kg/s.
Solution: Inserting the parameters into Eq. (10) yields:

Note: termenial velocity represents the constant velocity in the velocity-

time curve. For simplify use (0.1) as an accepted value between the new
calculated velocity and previous velocity.
• Now calculate v(t) for different suggested times until reach a
difference between V new –V old = 0.1. (Note: This will be applied for all
problems in this chapter).
 Analytical Solution to the Falling
Parachutist Problem
Example 1

For t = (0):
V(0) = 53.39 (1-e-0.18355 × 0) = 0
For t = (2):
V(2) = 53.39 (1-e-0.18355 × 2) = 16.405
For t = (4):
V(4) = 53.39 (1-e-0.18355 × 4) = 27.769
Follwing the same calculation for t = 6,8, 10, 12, 14, 16, 20, … ∞) as
follows:
 Analytical Solution to the Falling
Parachutist Problem
Example 1

t, s V, m/s t, s V, m/s
0 0 18 51.429
2 16.405 20 52.032
4 27.769 22 52.449
6 35.642 24 52.738
8 41.095 26 52.939
10 44.873 28 53.077
12 47.49 30 53.174
14 49.303 32 53.24
16 50.559 ∞ 53.39
 Analytical Solution to the Falling
Parachutist Problem
Example 1
 Analytical Solution to the Falling
Parachutist Problem
Example 1
• According to the model, the parachutist accelerates rapidly (above Figure).
A velocity of 44.87 m/s (100.4 mi/h) is attained after 10 s.
• The terminal velocity is the constant velocity that is reached after a
sufficiently long time (in this example):
Terminal velocity = 53.39 m/s.
• This velocity is constant because, eventually, the force of gravity will be in
balance with the air resistance. Thus, the net force is zero and acceleration
has ceased. Equation (10) is called an analytical, or exact, solution because it
exactly satisfies the original differential equation. Unfortunately, there are
many mathematical models that cannot be solved exactly. In many of these
cases, the only alternative is to develop a numerical solution that
approximates the exact solution.
 Numerical Solutions
• As mentioned previously, numerical methods are those in which the
mathematical Problem is reformulated so it can be solved by
arithmetic operations.
• This can be illustrated for Newton’s second law by realizing that the
time rate of change of velocity can be approximated by:

(11)

Where v and t = differences in velocity and time, respectively, computed

over finite intervals, v(ti) = velocity at an initial time ti , and v(ti+1) =
velocity at some later time ti+1
 Numerical Solutions
• This figure shows the use of a
finite difference to approximate
the first derivative of v with
respect to t.
 Numerical Solutions
• Note that dv/dt ∼= ∆v/∆t is approximate because ∆t is finite. Remember
from calculus that:

• Equation (11) represents the reverse process.

• Equation (11) is called a finite divided difference approximation of the
derivative at time ti . It can be substituted into Eq. (9) to give:

(12)
 Numerical Solutions
• Notice that the term in brackets is the right-hand side of the differential
equation itself [Eq. (9)]. That is, it provides a means to compute the rate of
change or slope of v.
• Thus, the differential equation has been transformed into an equation that
can be used to determine the velocity algebraically at ti+1 using the slope and
previous values of v and t.
• If you are given an initial value for velocity at some time ti , you can easily
compute velocity at a later time ti+1. This new value of velocity at ti+1 can in
turn be employed to extend the computation to velocity at ti+2 and so on.
Thus, at any time along the way,
New value = old value + slope × step size
• Note that this approach is formally called Euler’s method.
 Numerical Solution to the Falling
Parachutist Problem
Example 2: A parachutist of mass 68.1 kg jumps out of a stationary hot air
balloon. Use the numerical method to plot the velocity-time relationship and
compute the treminal velocity . The drag coefficient is equal to 12.5 kg/s.
Employ a step size of 2 s for the calculation.

Solution: the following equation (Eq. 12) is used to calculate v at different t:

• At the start of the computation (ti = 0), the velocity of the parachutist
is zero (v0 = 0).
• Using this information and the given parameter values, Eq. (12) can be
used to compute velocity at ti+1 = 2 s:
 Numerical Solution to the Falling
Parachutist Problem
Example 2:
• Thus, velocity for the first interval (t = 0 to 2 s):

• For the next interval (from t = 2 to 4 s), the computation is repeated,

with the result:

• The calculation is continued in a similar fashion to obtain additional

values:
 Numerical Solution to the Falling
Parachutist Problem
Example 2:
t, s V, m/s t, s V, m/s
0 0 24 53.17
2 19.6 26 53.25
4 32 28 53.3
6 39.86 30 53.33
8 44.82 32 53.36
10 47.97 34 53.37
12 49.96 36 53.38
14 51.22 38 53.38
16 52.02 40 53.38
18 52.52 42 53.39
20 52.84 44 53.39
22 53.04 ∞ 53.39
 Numerical Solution to the Falling
Parachutist Problem
Example 2:
The results of the numerical
solution are plotted and
compared with analytical
solution for the same
problem in the following
Figure.
 Numerical Solution to the Falling
Parachutist Problem
Example 2:
• It can be seen (from the above Figure) that the numerical method captures
the essential features of the exact solution. However, because we have
employed straight-line segments to approximate a continuously curving
function, there is some discrepancy between the two results.
• One way to minimize such discrepancies is to use a smaller step size. For
example, applying Eq. (12) at l-s intervals results in a smaller error, as the
straight-line segments track closer to the true solution.
• Using hand calculations, the effort associated with using smaller and
smaller step sizes would make such numerical solutions impractical.
• However, with the aid of the computer, large numbers of calculations can be
performed easily. Thus, you can accurately model the velocity of the falling
parachutist without having to solve the differential equation exactly.
 Conservation Laws
• Aside from Newton’s second law, there are other major organizing
principles in engineering. Among the most important of these are the
conservation laws.

• Although they form the basis for a variety of complicated and

powerful mathematical models, the great conservation laws of science
and engineering are conceptually easy to understand. They all boil
down to

Change = increases – decreases (13)

 Conservation Laws
• simple, Eq. (13) embodies one of the most fundamental ways in which
conservation laws are used in engineering—that is, to predict changes
with respect to time.

• We give Eq. (13) the special name time-variable (or transient)

computation.

• Aside from predicting changes, another way in which conservation

laws are applied is for cases where change is nonexistent. If change is
zero, Eq. (13) becomes
 Conservation Laws
• Change = 0 = increases − decreases

or

• Increases = decreases

Example 3: The below figure shows a flow balance for steady

incompressible fluid flow at the junction of pipes. Compute the flow out
of the fourth pipe. Note, the flow in the below figure is in m3/s.
Conservation Laws
 Conservation Laws
Solution:
If no change occurs, the increases and decreases must be in balance.
This case, which is also given a special name—the steady-state
computation—has many applications in engineering.
Thus, the flow into a junction must be balanced by flow going out, as in:
Flow in = flow out
For the above junction, the balance can be used to compute that the
flow out of the fourth pipe as follows:
 Conservation Laws
Flow in = flow out
Flow in- flow out = 0

Flow in pipe 1+ Flow in pipe 2- Flow out pipe 3- Flow out pipe 4= 0

100+ 80-120- Flow out pipe 4= 0

Flow out pipe 4= 100+80-120 = 60 m3/s

 Conservation Laws
Example 4

The volume flow rate through a pipe is given by Q = vA, where v is the
average velocity and A is the cross-sectional area.

Use volume-continuity to solve for the required area in pipe 3 in the

below junction.
Conservation Laws
 Conservation Laws
Solution:

For the above junction, the balance can be used to compute that the flow out of the fourth pipe as follows:

Q1 in = Q2 out + Q3 out

Q1 in - Q2 out - Q3 out = 0

40-20- Q3 out = 0

Q3 out = 40-20 = 20 m3/s

Now, the following equation is used to calculate the area in the pipe 3

Flow rate = Area × Velocity

Q=AV; thus, A=Q/V

 Conservation Laws
For pipe 3: A3 = Q3/V3

A3= 20/ 6 = 3.33 m2

For the falling parachutist, steady-state conditions would correspond to the case where the net force was zero, or
[Eq. (8) with dv/dt = 0]

mg = cv (15)

Thus, at steady state, the downward and upward forces are in balance, and Eq. (1.15) can be solved for the
terminal velocity

v = mg/c

Although Eqs. (13) and (14) might appear simple; they represent the two fundamental ways that conservation laws
are employed in engineering.
 University of Technology
Petroleum Technology Department

Numerical Analysis
Lecture 3: Numerical solution of differential equations
Semester 1 (2020-2021)

Doaa Saleh Mahdi

CISE301_Topic8L1 1
Learning Objectives

• Recall basic definitions of ODEs:

• Order
• Linearity
• Initial conditions
• Solution
• Classify ODEs based on:
• Order, linearity, and conditions.
• Classify the solution methods.

2
Learning Objectives

• Derive Euler formula using the Taylor series expansion.

• Solve the first order ODEs using Euler method.
• Assess the error level when using Euler method.
• Appreciate different types of errors in the numerical solution of
ODEs.
• Improve Euler method using higher-order Taylor Series.

3
Derivatives

Derivatives

Partial Derivatives
Ordinary Derivatives

dv u
dt y
u is a function of
v is a function of one
more than one
independent variable
independent variable

4
Differential Equations
Differential
Equations

Ordinary Differential Equations Partial Differential Equations

2
d v u u
2 2

+ 6tv = 1 − 2 =0
dt 2 y 2
x
involve one or more involve one or more
Ordinary derivatives of partial derivatives of
unknown functions unknown functions

5
Differential Equations

6
 Ordinary Differential Equations

Ordinary Differential Equations (ODEs) involve one or

more ordinary derivatives of unknown functions with
respect to one independent variable
Examples :
dv(t )
− v(t ) = et x(t): unknown function
dt
d 2 x(t ) dx(t )
2
−5 + 2 x(t ) = cos(t )
dt dt
t: independent variable

7
Example of ODE:
Model of Falling Parachutist
The velocity of a falling parachutist is given by:

dv c
= 9.8 − v
dt M
M : mass
c : drag coefficien t
v : velocity

8
Definitions

Ordinary
differential
equation
dv c
= 9.8 − v
dt M

9
Definitions (Cont.)

dv c
= 9.8 − v (Dependent
dt M variable)
unknown
function to be
determined

10
Definitions (Cont.)

dv c
= 9.8 − v
dt M
(independent variable)
the variable with respect to which
other variables are differentiated

11
 Order of a Differential Equation
The order of an ordinary differential equation is the order
of the highest order derivative.
Examples :
dx(t )
− x(t ) = et First order ODE
dt
d 2 x(t ) dx(t )
2
−5 + 2 x(t ) = cos(t ) Second order ODE
dt dt
3
 d x(t )
2
 dx(t )
 2
 − + 2 x 4 (t ) = 1 Second order ODE
 dt  dt

12
Order of a Differential Equation

13
Solution of a Differential Equation
A solution to a differential equation is a function that
satisfies the equation.

Example : Solution x(t ) = e −t

dx(t )
+ x(t ) = 0 Solution
Proof :
Proof :
x(t ) = e −t

dt dx(t )
dx(t )
= −e −t

= −e −t
dt
dx(t )
+ x(t ) = −e −t + e −t = 0
dt
dt
dx(t )
+ x(t ) = −e −t + e −t = 0
dt

14
 Solution of a Differential Equation

• Example: Proof that:

y = xe x
is a solution of d2y
2
dy
−2 + y = 0
dx dx
• solution

dy
y = xe x
= xe x + e x
dx
d2y d2y
= xe x
+ e x
+ e x
= xe x
+ 2e x

dx 2 dx 2

xe + 2e − 2 xe − 2e + xe = 0
x x x x x

15
 Solutions of Ordinary Differential
Equations
Homework

x(t ) = cos(2t )
is a solution to the ODE
2
d x(t )
2
+ 4 x (t ) = 0
dt
16
Linear ODE

17
Linear ODE

18
Linear ODE

19
Linear ODE

20
 Linear ODE

Examples :
dx(t )
− x(t ) = et
dt Linear ODE
d 2 x(t ) dx(t )
2
− 5 + 2t 2
x(t ) = cos(t ) Linear ODE
dt dt
3
 d x(t )  dx(t )
2
Non-linear ODE
 2
 − + x(t ) = 1
 dt  dt

21
 Nonlinear ODE
An ODE is linear if
The unknown function and its derivatives appear to power one
No product of the unknown function and/or its derivatives
Examples of nonlinear ODE :
dx(t )
− cos( x(t )) = 1
dt
d 2 x(t ) dx(t )
2
− 5 x(t ) = 2
dt dt

22
Auxiliary Conditions

Auxiliary Conditions

Initial Conditions Boundary Conditions

 All conditions are at one • The conditions are not at one

point of the independent variable
point of the independent
variable

23
Boundary-Value and
Initial value Problems
Initial-Value Problems Boundary-Value Problems

 The auxiliary conditions • The auxiliary conditions are not at one

are at one point of the point of the independent variable
independent • More difficult to solve than initial
variable value problems

x + 2 x + x = e −2 t x + 2 x + x = e −2t

x(0) = 1, x (0) = 2.5 x(0) = 1, x(2) = 1.5
same different

24
Classification of ODEs

ODEs can be classified in different ways:

• Order
• First order ODE
• Second order ODE
• Nth order ODE
• Linearity
• Linear ODE
• Nonlinear ODE
• Auxiliary conditions
• Initial value problems
• Boundary value problems

25
Analytical Solutions

• Analytical Solutions to ODEs are available for linear ODEs and special
classes of nonlinear differential equations.

26
Numerical Solutions

• Numerical methods are used to obtain a graph or a table of the

unknown function.
• Most of the Numerical methods used to solve ODEs are based
directly (or indirectly) on the truncated Taylor series expansion.

27
Classification of the Methods
Numerical Methods
for Solving ODE

Single-Step Methods Multiple-Step Methods

Estimates of the solution Estimates of the solution

at a particular step are at a particular step are
entirely based on based on information on
information on the more than one step
previous step

28
Taylor Series Method
The problem to be solved is a first order ODE:

dy( x)
= f ( x, y ), y( x0 ) = y0
dx
Estimates of the solution at different base points:

y( x0 + h), y( x0 + 2h), y( x0 + 3h), ....

are computed using the truncated Taylor series
expansions.

29
 Taylor Series Expansion
Truncated Taylor Series Expansion
h k  d k y
n 

y ( x0 + h )  
k !  dx k

k =0
 x = x 00 , y =
= y
y 00 

dy h 22 d 22 y h nn dd nn yy
 y ( x0 ) + h + 22
+ ... +
dx xy==xy00 , 2! dx xx==xx00,, dxnn xx==xx00,,
nn!! dx
00 yy==yy00 yy==yy00

The nth order Taylor series method uses the

nth order Truncated Taylor series expansion.

30
Euler Method

• First order Taylor series method is known as Euler Method.

• Only the constant term and linear term are used in the Euler method.

• The error due to the use of the truncated Taylor series is of order
O(h2).

31
First Order Taylor Series Method
(Euler Method)
dydy
yy( x( 0x0++h)h=
) =y (yx(0x)0+) h+ h + O+(O
2
h 2()h )
x = xx0 ,= x0 ,
dxdx
y = yy = y
0 0

Notation
Notation: :
xxn n==x0x0+ +nhnh
, , yny=n = y ( yxn( ),
xn ),
dydy = f ( x , y )
dx x = x ,
= f (i x ,i y )
i i
x =
dxy = y i
ix ,
y =i yi
Euler
EulerMethod
Method
yyi +1 ==yiy+ +h hf (fx(i ,xy,i )y )
i +1 i i i

32
Euler Method
Problem :
Given the first order ODE : y ( x) = f ( x, y )
with the initial condition : y0 = y ( x0 )
Determine : yi = y ( x0 + ih ) for i = 1,2,...

Euler M ethod :
y0 = y ( x0 )
yi +1 = yi + h f ( xi , yi ) for i = 1,2,...

33
Interpretation of Euler Method
y2

y1

y0

x0 x1 x2 x

34
Interpretation of Euler Method

Slope=f(x0,y0)
y1
y1=y0+hf(x0,y0)
hf(x0,y0)
y0

x0 x1 x2 x
h

35
Interpretation of Euler Method
y2 y2=y1+hf(x1,y1)
Slope=f(x1,y1)
hf(x1,y1)
Slope=f(x0,y0)
y1 y1=y0+hf(x0,y0)
hf(x0,y0)
y0

x0 x1 x2 x
h h

36
Example
Use Euler method to solve the ODE:

dy
= 1+ x ,
2
y(1) = −4
dx
to determine y(1.01), y(1.02) and y(1.03).

37
Example

f ( x, y) = 1 + x , 2
x0 = 1, y0 = −4 , h = 0.01
Euler M ethod
yi +1 = yi + h f ( xi , yi )

Step1 : y1 = y0 + h f ( x0 , y0 ) = −4 + 0.01(1 + (1) 2 ) = −3.98

( )
Step 2 : y2 = y1 + h f ( x1 , y1 ) = −3.98 + 0.01 1 + (1.01) = −3.9598
2

Step 3 : (
y3 = y2 + h f ( x2 , y2 ) = −3.9598 + 0.01 1 + (1.02) = −3.9394
2
)

38
Example

f ( x, y) = 1 + x ,
2
x0 = 1, y0 = −4 , h = 0.01
Summary of the result:

i xi yi
0 1.00 -4.00
1 1.01 -3.98
2 1.02 -3.9595
3 1.03 -3.9394

39
Example

f ( x, y) = 1 + x ,
2
x0 = 1, y0 = −4 , h = 0.01
Comparison with true value:

i xi yi True value of yi
0 1.00 -4.00 -4.00
1 1.01 -3.98 -3.97990
2 1.02 -3.9595 -3.95959
3 1.03 -3.9394 -3.93909

40
Example

f ( x, y) = 1 + x ,
2
x0 = 1, y0 = −4 , h = 0.01

A graph of the
solution of the
ODE for
1<x<2

41
Types of Errors
• Local truncation error:
Error due to the use of truncated Taylor series to compute x(t+h) in one
step.
• Global Truncation error:
Accumulated truncation over many steps.
• Round off error:
Error due to finite number of bits used in representation of numbers. This
error could be accumulated and magnified in succeeding steps.

42
Second Order Taylor Series Methods
dy ( x)
Given = f ( y, x), y ( x0 ) = y0
dx
Second order Taylor Series method
2 2
dy h d y
yi +1 = yi + h + 2
+ O(h )
3

dx 2! dx
2
d y
2
needs to be derived analytically.
dx
43
Third Order Taylor Series Methods
dy ( x)
Given = f ( y, x), y ( x0 ) = y0
dx
Third order Taylor Series method
2 2 3 3
dy h d y h d y
yi +1 = yi + h + 2
+ 3
+ O(h )
4

dx 2! dx 3! dx
2 3
d y d y
2
and 3 need to be derived analytically.
dx dx
44
High Order Taylor Series Methods
dy ( x)
Given = f ( y, x), y ( x0 ) = y0
dx
th
n order Taylor Series method
dy h 2 d 2 y hn d n y n +1
yi +1 = yi + h + 2
+ .... + n
+ O(h )
dx 2! dx n! dx
2 3 n
d y d y d y
2
, 3 ,....., n need to be derived analytically.
dx dx dx

45
Higher Order Taylor Series Methods

• High order Taylor series methods are more accurate than Euler
method.

• But, the 2nd, 3rd, and higher order derivatives need to be derived
analytically which may not be easy.

46
Example
Second order Taylor Series Method
Use Second order Taylor Series method to solve :
dx
+ 2 x 2 + t = 1, x(0) = 1, use h = 0.01
dt

to determine x(0.01), x(0.02) and x(0.03).

d 2 x(t )
What is : 2
?
dt

47
Example
Use Second order Taylor Series method to solve :
dx
+ 2 x + t = 1, x(0) = 1,
2
use h = 0.01
dt
dx
= 1− 2x2 − t
dt
d 2 x(t ) dx
2
= 0 − 4 x − 1 = − 4 x (1 − 2 x 2
− t ) −1
dt dt
2
h
xi +1 = xi + h(1 − 2 xi − ti ) + ( − 1 − 4 xi (1 − 2 xi − ti ))
2 2

48
Example

f (t , x) = 1 − 2 x 2 − t , t0 = 0, x0 = 1, h = 0.01
2
h
xi +1 = xi + h(1 − 2 xi − ti ) + ( − 1 − 4 xi (1 − 2 xi − ti ))
2 2

2
Step 1 :

x1 = 1 + 0.01(1 − 2(1) 2
− 0) +
(0.01)
2
(−1 − 4(1)(1 − 2 − 0)) = 0.9901
2
Step 2 :

x2 = 0.9901 + 0.01(1 − 2(0.9901) 2

− 0.01) +
(0.01)
2
(−1 − 4(0.9901)(1 − 2(0.9901) 2 − 0.01)) = 0.9807
2
Step 3 :
x3 = 0.9716

49
Example

f (t , x) = 1 − 2 x 2 − t , t0 = 0, x0 = 1, h = 0.01
Summary of the results:

i ti xi Homework:
Calculate the True
0 0.00 1 value of xi.

1 0.01 0.9901
2 0.02 0.9807
3 0.03 0.9716

50
 University of Technology
Petroleum Technology Department

Numerical Analysis
Lecture 4: Numerical solution of differential equations (Part two)
Semester 1 (2020-2021)

Doaa Saleh Mahdi

1
Learning Objectives
 To be able to solve first order differential
equations using the Midpoint Method.
 To be able to solve first order differential
equations using the Heun’s Predictor
Corrector Method.
 Convert a single (or a system of) high order
ODE to a system of first order ODEs.
 Use the methods discussed earlier in this
topic to solve systems of first order ODEs.

2
Topics Lecture 4
•Review Euler Method
• Heun’s Method
• Midpoint Method
•Solution of a system of first order ODEs.
•Conversion of a high order ODE to a system of
first order ODEs.
•Conversion of a system of high order ODEs to
a system of first order ODEs.
•Use different methods to solve systems of first
order ODEs.

3
Euler Method
Problem Euler M ethod
y ( x)  f ( x, y ) y0  y ( x0 )
y ( x0 )  y0 yi 1  yi  h f ( xi , yi )
for i  1,2,...

2
Local Truncation Error O(h )
Global Truncation Error O(h)
4
Types of Errors
• Local truncation error:
Error due to the use of truncated Taylor series to compute x(t+h) in one
step.
• Global Truncation error:
Accumulated truncation over many steps.

5
Introduction
Problem to be solved is a first order ODE :
y ( x)  f ( x, y ), y( x0 )  y0
 The methods proposed in this lesson
have the general form:
yi 1  yi  h 
 For the case of Euler:   f ( xi , yi )
 Different forms of  will be used for
the Midpoint and Heun’s Methods.
6
Midpoint Method
Problem M idpoint M ethod
y ( x)  f ( x, y ) y0  y ( x0 )
h
y ( x0 )  y0 y 1  yi  f ( xi , yi )
i 2
2

yi 1  yi  h f ( x 1 ,y 1 )
i i
2 2

Local Truncation Error O(h3 )

Global Truncation Error O(h2 )
7
Motivation
 The midpoint can be summarized as:
 Euler method is used to estimate the solution
at the midpoint.
 The value of the rate function f(x,y) at the mid
point is calculated.
 This value is used to estimate yi+1.
 Local Truncation error of order O(h3).
 Comparable to Second order Taylor series
method.

8
Midpoint Method

( xi , yi )

x0 x 1 xi 1
i
2
h
y 1  yi  f ( xi , yi ) , yi 1  yi  h f ( x 1 ,y 1 )
i 2 i i
2 2 2
9
Midpoint Method

slope  f ( xi , yi )

( xi , yi )

x0 x 1 xi 1
i
2
h
y 1  yi  f ( xi , yi ) , yi 1  yi  h f ( x 1 ,y 1 )
i 2 i i
2 2 2
10
Midpoint Method

(x ,y )
slope  f ( xi , yi )
1 1
i i
2 2

( xi , yi )

x0 x 1 xi 1
i
2
h
y 1  yi  f ( xi , yi ) , yi 1  yi  h f ( x 1 ,y 1 )
i 2 i i
2 2 2
11
Midpoint Method
slope  f ( x 1 ,y 1 )
i i
(x 1 ,y 1 ) 2 2

i i
2 2

( xi , yi )

x0 x 1 xi 1
i
2
h
y 1  yi  f ( xi , yi ) , yi 1  yi  h f ( x 1 ,y 1 )
i 2 i i
2 2 2
12
Midpoint Method
slope  f ( x 1 ,y 1 )
i i
(x 1 ,y 1 ) 2 2
i i
2 2

( xi , yi )

x0 x 1 xi 1
i
2
h
y 1  yi  f ( xi , yi ) , yi 1  yi  h f ( x 1 ,y 1 )
i 2 i i
2 2 2
13
Example 1

Use the M idpoint M ethod to solve the ODE

y ( x)  1  x 2  y
y (0)  1
Use h  0.1. Determine y(0.1) and y(0.2)

14
Example 1
Problem : f ( x, y )  1  x 2  y , y0  y (0)  1, h  0.1
Step 1 :
h
y 1  y0  f ( x0 , y0 )  1  0.05(1  0  1)  1.1
0 2
2

y1  y0  h f ( x 1, y 1)  1  0.1(1  0.0025  1.1)  1.2103

0 0
2 2
Step 2 :
h
y 1  y1  f ( x1 , y1 )  1.2103  .05(1  0.01  1.2103)  1.3213
1 2
2

y2  y1  h f ( x 1, y 1)  1.2103  0.1( 2.3438)  1.4446

1 1
2 2
15
Heun’s Predictor
Corrector

16
 Heun’s Predictor Corrector Method
Problem Heun ' s M ethod
y ( x )  f ( x, y ) y0  y ( x0 )
y ( x0 )  y0 Predictor : yi01  yi  h f ( xi , yi )

Corrector : yi11  yi 
h
2

f ( xi , yi )  f ( xi 1 , yi01 ) 

Local Truncation Error O(h3 )

Global Truncation Error O(h2 )

17
Heun’s Predictor Corrector
(Prediction)

( xi 1 , yi01 )

( xi , yi )

xi xi 1

Prediction 0
yi 1  yi  h f ( xi , yi )
18
Heun’s Predictor Corrector
(Prediction)

slope  f ( xi , yi ) ( xi 1 , yi01 )

( xi , yi )

xi xi 1

Prediction y 0
i 1  yi  h f ( xi , yi )
19
Heun’s Predictor Corrector
(Correction)

f ( xi , yi )  f ( xi 1 , yi01 )
slope 
2
( xi 1 , yi01 )

( xi , yi ) ( xi 1 , yi11 )

xi xi 1

y1
i 1  yi 
h
2

f ( xi , yi )  f ( xi 1 , yi01 ) 
20
Example 2
Use the Heun' s M ethod to solve the ODE
y ( x)  1  x  y
2

y (0)  1
Use h  0.1. One correction only
Determine y(0.1) and y(0.2)

21
Example 2
Problem : f ( x, y )  1  y  x 2 , y0  y ( x0 )  1, h  0.1

Step 1 :
Predictor : y10  y0  h f ( x0 , y0 )  1  0.1( 2)  1.2

Corrector : y11  y0 
h
2

f ( x0 , y0 )  f ( x1 , y10 ) 
 1  (0.1 / 2)( 2  (1  1.2  (0.12 )))
 1  (0.1 / 2)( 2  2.21)  1.2105
Step 2 :
Predictor : y 20  y1  h f ( x1 , y1 )  1.4326

Corrector : y12  y1 
h
2

f ( x1 , y1 )  f ( x2 , y 20 )  
 1.2105  (0.1 / 2)( 2.2205  (1  1.4326  (0.2 2 )))
 1.2105  (0.1 / 2)(2.2205  2.4726)  1.4452
22
Summary
 Euler, Midpoint and Heun’s methods are
similar in the following sense:
yi 1  yi  h  slope
 Different methods use different estimates of
the slope.
 Both Midpoint and Heun’s methods are
comparable in accuracy to the second
order Taylor series method.

23
Comparison
Method Local
truncation
Global
truncation
error error

Euler M ethod yi 1  yi  h f ( xi , yi ) O(h 2 ) O ( h)

Heun' s M ethod
Predictor : yi01  yi  h f ( xi , yi ) O(h 3 ) O(h 2 )
k 1
Corrector : y  yi 
i 1
h
2

f ( xi , yi )  f ( xi 1 , yik1 ) 
h
M idpoint y 1  yi  f ( xi , yi ) O(h 3 ) O(h 2 )
i 2
2

yi 1  yi  h f ( x 1 ,y 1 )
i i
2 2

24
Solving a System of First Order ODEs
 Methods discussed earlier such as Euler,
Midpoint and Heun’s method are used to
solve first order ordinary differential
equations.
 The same formulas will be used to solve
a system of first order ODEs.
 In this case, the differential equation is a
vector equation and the dependent variable is
a vector variable.

25
Euler Method for Solving a System of
First Order ODEs
Recall Euler method for solving a first order ODE:

dy ( x)
Given  f ( y, x), y (a)  ya
dx

Euler Method :
y (a  h)  y (a )  h f ( y (a), a)
y (a  2h)  y (a  h)  h f ( y (a  h), a  h)
y (a  3h)  y (a  2h)  h f ( y (a  2h), a  2h)

26
Example - Euler Method
Euler method to solve a system of n first order ODEs.
 f1 (Y , x)   y1 ( x0 ) 
 f (Y , x)  y ( x )
d Y ( x)
Given  F (Y , x)   2 , Y ( x )   2 0 
dx  ...  0
 ... 
   
 f n (Y , x)  yn ( x0 )
Euler Method :
Y ( x0  h)  Y ( x0 )  h F (Y ( x0 ), x0 )
Y ( x0  2h)  Y ( x0  h)  h F (Y ( x0  h), x0  h)
Y ( x0  3h)  Y ( x0  2h)  h F (Y ( x0  2h), x0  2h)

27
Solving a System of n First Order ODEs
 Exactly the same  y1 ( x) 
 y ( x) 
formula is used but Y ( x)   2  Y is n 1 vector
the scalar variables  ... 
 
and functions are  n 
y ( x )
replaced by vector  d y1 
variables and vector  dx   f1 (Y , x) 
values functions. d y   
d Y ( x)  2   f 2 (Y , x) 
 dx   F (Y , x)
 Y is a vector of dx    ... 
 ...   
length n.  n  n
d y f (Y , x ) 
 F(Y,x) is a vector  dx 
valued function.

28
 Example :
Euler method for solving a system of first order ODEs.
 y1 ( x)   y2   y1 (0)   1

 y ( x) 1  y   F (Y , x ), Y ( 0)   y ( 0)    1 
 2   1  2   
Two steps of Euler Method with h  0.1
STEP 1 :
Y ( x0  h)  Y ( x0 )  h F (Y ( x0 ), x0 )
Y (0  h)  Y (0)  h F (Y (0),0)
 y1 ( x0  h)  y1 (0.1)   y1 (0)   y2 (0)    1  0.1   0.9
 y ( x  h)  y (0.1)   y (0)  0.11  y (0)  1  0.1(1  1)   1.2 
 2 0 2   2   1     
STEP 2 :
Y ( x0  2h)  Y ( x0  h)  h F (Y ( x0  h), x0  h)
 y1 ( x0  2h)  y1 (0  2(0.1)   y1 ( x0  h)   y2 ( x0  h)    0.9  0.1(1.2)   0.78

 y ( x  2h)  y (0  2(0.2)  y ( x  h)  0 .11  y ( x  h)  1.2  0.1(1  (0.9)   1.39 
 2 0 2   2 0   1 0     
STEP 3 :
Y ( x0  3h)  Y ( x0  2h)  h F (Y ( x0  2h), x0  2h)
 y1 ( x0  3h)  y1 (0  3(0.1)   y1 ( x0  2h)   y2 ( x0  2h)    0.78  0.1(1.39)   0.64

 y ( x  3h)  y (0  3(0.2)  y ( x  2h)  0 .11  y ( x  2h)  1.39  0.1(1  (0.78)   1.57 
 2 0 2   2 0   1 0     
29
Methods for Solving a System of First Order ODEs

 We have extended Euler method to solve

systems of first order ODEs.

 Other methods used to solve first order ODE can

be easily extended to solve systems of first
order ODEs.

30
High Order ODEs
 How do solve a second order ODE?

x  3x  6 x  1
 How do solve high order ODEs?

31
The General Approach to Solve ODEs
High order ODE Convert System of first order ODEs Solve

 z1   z2 
 z   1  3z  6 z ,
x  3x  6 x  1 Convert
 2  2 1
Solve
x (0)  1; x(0)  4  4
Z (0)   
1 
Second order ODE Two first order ODEs

32
Conversion Procedure
High order ODE Convert System of first order ODEs Solve

1. Select the dependent variables

One way is to take the original dependent
variable and its derivatives up to one degree less
than the highest order derivative.
2. Write the Differential Equations in terms of
the new variables. The equations come from the
way the new variables are defined or from the
original equation.
3. Express the equations in a matrix form.
33
Remarks on the Conversion Procedure
High order ODE Convert System of first order ODE Solve

1. Any nth order ODE is converted to a system of n

first order ODEs.
2. There are an infinite number of ways to select
the new variables. As a result, for each high
order ODE there are an infinite number of set of
equivalent first order systems of ODEs.
3. Use a table to make the conversion easier.

34
Example of Converting a High Order
ODE to First Order ODEs
Convert x  3 x  6 x  1, x (0)  1; x(0)  4
to a systemof first order ODEs

1. Select a new set of variables

(Second order ODE  We need two variables )
z1  x
One degree less than the
z 2  x highest order derivative

35
Example of Converting a High Order
ODE to First Order ODEs
old new Initial Equation
name name cond.
x z1 4 z1  z 2
x z2 1 z2  1  3 z 2  6 z1

 z1   z2   4
 z   1  3z  6 z , Z (0)  1 
 2  2 1  
36
Example of Converting a High Order
ODE to First Order ODEs
Convert
x  2 x  7 x  8 x  0
x(0)  9, x (0)  1; x(0)  4

1. Select a new set of variables (3 of them)

z1  x
z 2  x One degree less than the
highest order derivative
z3  x
37
Example of Converting a High Order
ODE to First Order ODEs
old new Initial Equation
name name cond.
x z1 4 z1  z 2
x z2 1 z2  z3
x z3 9 z3  2 z3  7 z 2  8 z1
 z1   z2   4
 z    z , Z (0)  1 
 2  3   
 z3   2 z3  7 z 2  8 z1  9 
38
 Conversion Procedure for Systems of
High Order ODEs
System of high order ODEs Convert System of first order ODE Solve

1. Select the dependent variables

Take the original dependent variables and their
derivatives up to one degree less than the
highest order derivative for each variable.
2. Write the Differential Equations in terms of
the new variables. The equations come from the
way the new variables are defined or from the
original equation.
3. Express the equations in a matrix form.
39
Example of Converting a High Order
ODE to First Order ODEs
Convert
x  5x  2 x  8 y  0
y  2 xy  x  2
x(0)  4; x (0)  2; x(0)  9; y (0)  1; y (0)  3
1. Select a new set of variables ((3  2) variables )
z1  x
z 2  x One degree less
z3  x
than the highest
order derivative
z4  y
z5  y
40
Example of Converting a High Order
ODE to First Order ODEs
old new Initial Equation
name name cond.
x z1 4 z1  z 2
x z2 2 z2  z3
x z3 9 z3  5 z3  2 z2  8 z 4
y z4 1 z4  z5
y z5 3 z5  2  z2  2 z1 z4
41
Solution of a Second Order ODE
 Solve the equation using Euler method. Use h=0.1
x  2 x  8 x  2
x(0)  1; x (0)  2
Select a new set of variables : z1  x, z2  x
The second order equation is expressed as :

  1  
z z   1 
Z  F (Z )      2
 , Z (0)   
 z2  2  2 z 2  8 z1    2

42
Solution of a Second Order ODE
 z2  1
F (Z )    , Z (0)   , h  0.1
2  2 z 2  8 z1    2
Z (0  0.1)  Z (0)  hF ( Z (0))
1  2   0.8 
    0.1   

 2  2  2 ( 2)  8(1)    2.2 
Z (0  0.2)  Z (0.1)  hF ( Z (0.1))
 0.8    2.2   0.58 
   0.1   
  2.2   2  2( 2 . 2)  8( 0.8)    2.2 
43
Summary
 Formulas used in solving a first order ODE
are used to solve systems of first order
ODEs.
 Instead of scalar variables and functions, we
have vector variables and vector functions.

 High order ODEs are converted to a set of

first order ODEs.

44
 University of Technology
Petroleum Technology Department

Numerical Analysis
Lecture 5:Runge-Kutta Methods Semester 1 (2020-2021)

Doaa Saleh Mahdi

1
Learning Objectives

• Runge-Kutta Methods
• To understand the motivation for using Runge Kutta method and the
basic idea used in deriving them.
• Use Runge Kutta of order 2 to solve ODEs.
• Boundary Value Problem
• Examples

2
Simplified method for the computation of numerical solution
of system

1-Euler method is the simplest method for the computation of

numerical solution of system . The Taylor series expansions for
y(x) and z(x) are as follows :
2:
Example
Example
 Runge-Kutta Methods
Runge–Kutta method of different orders can
be used to solve the system , but here we are
presenting only classical Runge–Kutta method
(also known as Runge–Kutta method of order
4). Keeping in mind the method for a single
equation, classical Runge–Kutta method for
the solution of the system is as follows

8
Second Order Runge-Kutta Method
K1  h f ( xi , yi )
K 2  h f ( xi   h, yi   K1 )
yi 1  yi  w1K1  w2 K 2
Problem :
Find  ,  , w1, w2
such that yi 1 is as accurate as possible.

9
2nd Order Runge-Kutta Methods
K1  h f ( xi , yi )
K 2  h f ( xi   h, yi   K1 )
yi 1  yi  w1K1  w2 K 2

Choose  ,  , w1, w2 such that :

1 1
w1  w2  1, w2  , and w2  
2 2
10
Choosing , , w1 and w2
1
For example, choosing   1, then   1, w1  w2 
2
Second Order Runge - Kutta method becomes :
K1  h f ( xi , yi )
K 2  h f ( xi  h, yi  K1 )

2
h
2

yi 1  yi  K1  K 2   yi  f ( xi , yi )  f ( xi 1, yi 1 )
1 0

This is Heun ' s Method with a Single Corrector

11
Choosing , , w1 and w2
1 1
Choosing   then   , w1  0, w2  1
2 2
Second Order Runge - Kutta method becomes :
K1  h f ( xi , yi )
h K1
K 2  h f ( xi  , yi  )
2 2
h K1
yi 1  yi  K 2  yi  h f ( xi  , yi  )
2 2
This is the M idpoint M ethod
12
2nd Order Runge-Kutta Methods
Alternative Formulas
1 1
 w2  ,  w2  , w1  w2  1
2 2
1 1
Pick any nonzero  number :    , w2  , w1  1 
2 2

Second Order Runge Kutta Formulas (select   0)

K1  h f ( xi , yi )
K 2  h f ( xi   h, yi   K1 )
 1  1
yi 1  yi  1   K1  K2
 2  2
13
Second order Runge-Kutta Method
Example
Solve the following system to find x (1.02) using RK2
x (t )  1  x 2  t 3 , x (1)  4, h  0.01,   1

STEP 1 :
K1  h f (t0  1, x0  4)  0.01(1  x02  t03 )  0.18
K 2  h f (t0  h, x0  K1 )
 0.01(1  ( x0  0.18)2  (t0  .01)3 )  0.1662
x (1  0.01)  x (1)  K1  K 2  / 2
 4  (0.18  0.1662) / 2  3.8269
14
Second order Runge-Kutta Method
Example

STEP 2

K1  h f (t1  1.01, x1  3.8269)  0.01(1  x12  t13 )  0.1668

K 2  h f (t1  h, x1  K1 )
 0.01(1  ( x1  0.1668)2  (t1  .01)3 )  0.1546

x (1.01  0.01)  x (1.01)  K1  K 2 

1
2
1
 3.8269  (0.1668  0.1546)  3.6662
2

15
 x (t )  1  x (t )  t , x(1)  4,
2 3

Solution for t  [1,2]

Using RK2,   1

CISE301_Topic8L4&5 16
2nd Order Runge-Kutta RK2

Typical value of   1, Know as RK2

Equivalent to Heun' s method with a single corrector
k1  f ( xi , yi )
k2  f ( xi  h, yi  k1 h )

yi 1  yi  k1  k2 
h
2
Local error is O ( h 3 ) and global error is O (h 2 )

17
Higher-Order Runge-Kutta

Higher order Runge-Kutta methods are available.

Derived similar to second-order Runge-Kutta.

Higher order methods are more accurate but

require more calculations.

18
3rd Order Runge-Kutta RK3

Know as RK3
k1  f ( xi , yi )
h 1
k2  f ( xi  , yi  k1h )
2 2
k3  f ( xi  h, yi  k1h  2k2h )

yi 1  yi  k1  4k2  k3 
h
6
4 3
Local error is O (h ) and Global error is O ( h )
19
4th Order Runge-Kutta RK4

k1  f ( xi , yi )
h 1
k2  f ( xi  , yi  k1h )
2 2
h 1
k3  f ( xi  , yi  k2h )
2 2
k4  f ( xi  h, yi  k3h )

yi 1  yi  k1  2k2  2k3  k4 

h
6
5 4
Local error is O ( h ) and global error is O ( h )
20
Higher-Order Runge-Kutta
k1  f ( xi , yi )
1 1
k2  f ( xi  h, yi  k1h )
4 4
1 1 1
k3  f ( xi  h, yi  k1h  k2h )
4 8 8
1 1
k4  f ( xi  h, yi  k2h  k3h )
2 2
3 3 9
k5  f ( xi  h, yi  k1h  k4h )
4 16 16
3 2 12 12 8
k6  f ( xi  h, yi  k1h  k2h  k3h  k4h  k5h )
7 7 7 7 7
yi 1  yi  7k1  32k3  12k4  32k5  7k6 
h
90
21
Example
4th-Order Runge-Kutta Method RK4

dy
 1 y  x 2

dx
y (0)  0.5
h  0.2
Use RK 4 to compute y (0.2) and y (0.4)

22
Example: RK4
Problem :
dy
 1 y  x2 , y (0)  0.5
dx
Use RK 4 to find y (0.2), y (0.4)

23
4th Order Runge-Kutta RK4

k1  f ( xi , yi )
h 1
k2  f ( xi  , yi  k1h )
2 2
h 1
k3  f ( xi  , yi  k2h )
2 2
k4  f ( xi  h, yi  k3h )

yi 1  yi  k1  2k2  2k3  k4 

h
6
5 4
Local error is O ( h ) and global error is O ( h )
24
Example: RK4 See RK4 Formula
Problem : h  0.2
dy
 1 y  x2 , y (0)  0.5 f ( x, y )  1  y  x 2
dx
x0  0, y0  0.5
Use RK 4 to find y (0.2), y (0.4)
k1  f ( x0 , y0 )  (1  y0  x02 )  1.5

k2  f ( x0  h, y0  k1h )  1   y0  0.15   x0  0.12  1.64

1 1
2 2
Step 1

k3  f ( x0  h, y0  k2h )  1   y0  0.164    x0  0.12  1.654

1 1
2 2
k4  f ( x0  h, y0  k3h )  1   y0  0.16545   x0  0.2 2  1.7908

y1  y0 
h
k1  2k2  2k3  k4   0.8293
6
25
Example: RK4
Problem :
h  0.2
dy
 1 y  x2 , y (0)  0.5 f ( x, y )  1  y  x 2
dx
x1  0.2, y1  0.8293
Use RK 4 to find y (0.2), y (0.4)
k1  f ( x1, y1 )  1.7893
1 1
k2  f ( x1  h, y1  k1h )  1.9182
2 2
Step 2

1 1
k3  f ( x1  h, y1  k2h )  1.9311
2 2
k4  f ( x1  h, y1  k3h )  2.0555

y2  y1 
0.2
k1  2k2  2k3  k4   1.2141
6
26
Example: RK4
Problem :
dy
 1 y  x2 , y (0)  0.5
dx
Use RK 4 to find y (0.2), y (0.4)

Summary of the solution

xi yi
0.0 0.5
0.2 0.8293
0.4 1.2141
27
Summary
 Runge Kutta methods generate an
accurate solution without the need to
calculate high order derivatives.
 Second order RK have local truncation
error of order O(h3) and global truncation
error of order O(h2).
 Higher order RK have better local and
global truncation errors.
 N function evaluations are needed in the
Nth order RK method.
28
Example :
RK2 method for solving a system of first order ODEs

29
 Example :
RK2 method for solving a system of first order ODEs

30
 Example :
RK2 method for solving a system of first order ODEs

31
32
33
 Example :
RK2 method for solving a system of first order ODEs
 y1 ( x)   y2   y1 (0)   1
 y ( x)  1  y   F (Y , x), Y (0)   y (0)   1 
 2   1  2   
Two steps of second order Runge  Kutta M ethod with h  0.1
STEP 1 :
 y2 (0)   0.1
K1  h F (Y (0),0)  0.1   
1  y1 (0) 0.2
 y2 (0)  0.2  0.12
K 2  h F (Y (0)  K1,0  h)  0.1   
1  ( y1 ( 0)  0.1)   0. 19 
Y (0  h)  Y (0)  0.5( K1  K 2)
 y1 (0.1)   1 1   0.1 0.12   0.89
 y (0.1)   1   2  0.2  0.19    1.195 
 2         
34
 y1 ( x)   y2   y1 (0)   1
 y ( x)  1  y   F (Y , x), Y (0)   y (0)   1 
 2   1  2   

STEP 2 :
 y2 (0.1)  0.1195
K1  h F (Y (0.1),0.1)  0.1   
1  y1 ( 0. 1)   0. 1890 
 y2 (0.1)  0.189  0.1384
K 2  h F (Y (0.1)  K1,0.1  h)  0.1   
1  ( y1 ( 0.1)  0.1195)   0.1771
Y (0.1  h)  Y (0.1)  0.5( K1  K 2)
 y1 (0.2)   0.89 1  0.1195 0.1384   0.7611
 y (0.2)   1.195   2  0.1890   0.1771    1.3780 
 2         
35
 4th Order Runge-Kutta RK4

k1  f ( xi , yi )
h 1
k2  f ( xi  , yi  k1h )
2 2
h 1
k3  f ( xi  , yi  k2h )
2 2
k4  f ( xi  h, yi  k3h )

yi 1  yi  k1  2k2  2k3  k4 

h
6
5 4
Local error is O ( h ) and global error is O ( h )
36
 Boundary-Value and
Initial Value Problems
Initial-Value Problems Boundary-Value Problems

The auxiliary conditions are at  The auxiliary conditions are not at •

one point of the one point of the independent
independent variable variable
More difficult to solve than initial •
value problem

2 t
2 t
x  2 x  x  e x  2 x  x  e
x(0)  1, x (0)  2.5 x(0)  1, x(2)  1.5
same different

39
 Shooting Method

40
 University of Technology
Petroleum Technology Department

Numerical Analysis
Lecture 6: Solution of Systems of Linear Equation using Numerical
Methods Semester 1 (2020-2021)

Doaa Saleh Mahdi

۱
Learning Objectives

• Vectors
• Matrices
• Systems of Linear Equations
• Cramer’s Rule
• Naive Gaussian Elimination
• Problems with Naive Gaussian Elimination
• Gauss-Jordan Method
• Examples

۲
VECTORS
Vector : a one dimensional array of numbers
Examples :
 2
row vector [1 4 2] column vector  
1 
1 0  0  0 
0  1 0  0 
Identity vectors e1 =  , e2 =  , e3 =  , e4 =  
0  0  1 0 
       
0  0  0  1
۳
MATRICES
Matrix : a two dimensional array of numbers
Examples :
0 0 0 1 0
zero matrix   identity matrix  
0 0 0 0 1
1 0 0 0 1 2 0 0
0 4 0 0 3 4 1 0
diagonal  , Tridiagonal 
0 0 0 0 0 1 4 1
   
0 0 0 6 0 0 2 1

٤
MATRICES
Examples :
1 2 1 3
 2 1 − 1 0 4 1 0
symmetric  1 0 5 , upper triangular 
0 0 4 1
− 1 5 4   
0 0 0 1

٥
Determinant of a MATRICES

٦
Determinant of a MATRICES

۷
Determinant of a MATRICES
Defined for square matrices only
Examples :
 2 3 − 1
 
det  1 0 5  = 2
0 5 3 -1 3 -1
-1 -1
5 4 5 4 0 5
− 1 5 4 
= 2(−25) − 1(12 + 5) − 1(15 − 0) = −82

۸
Systems of Linear Equations
A system of linear equations can be presented
in different forms

2 x1 + 4 x2 − 3x3 = 3  2 4 − 3  x1  3

2.5 x1 − x2 + 3x3 = 5  ⇔ 2.5 − 1 3   x2  = 5
x1 − 6 x3 = 7   1 0 − 6  x3  7 
Standard form Matrix form

۹
Cramer’s Rule

۱۰
Cramer’s Rule

۱۱
Cramer’s Rule

۱۲
Cramer’s Rule

۱۳
Cramer’s Rule is Not Practical

Cramer' s Rule can be used to solve the system

Cramer' s Rule is not practical for large systems .

To solve N by N system requires (N + 1)(N - 1)N! multiplications.
To solve a 30 by 30 system, 2.38 × 1035 multiplications are needed.
It can be used if the determinants are computed in efficient way

۱٤
Naive Gaussian Elimination
 The method consists of two steps:
 Forward Elimination: the system is
reduced to upper triangular form. A sequence
of elementary operations is used.
 Backward Substitution: Solve the system
starting from the last variable.

 a11 a12 a13   x1   b1  a11 a12 a13   x1   b1 

a a22 a23   x  = b  ⇒ 0 a22 ' a23 '  x  = b '
 21  2  2   2  2 
 a31 a32 a33   x3  b3   0 0 a33 '  x3  b3 '

۱٥
Elementary Row Operations

 Adding a multiple of one row to another

 Multiply any row by a non-zero constant

۱٦
Forward Elimination
 ai1  
aij ← aij −  a1 j (1 ≤ j ≤ n ) 
 a11  
To eliminate x1 2 ≤ i ≤ n
a  
bi ← bi −  i1 b1
 a11  

 ai 2  
aij ← aij −  a 2 j (2 ≤ j ≤ n )
 a 22  
To eliminate x2 3 ≤ i ≤ n
a  
bi ← bi −  i 2 b2
 a 22  

۱۷
Forward Elimination

 aik  
aij ← aij −  akj (k ≤ j ≤ n)
 akk  
To eliminate xk k + 1 ≤ i ≤ n
 aik  
bi ← bi −  bk
 a kk  

Continue until xn −1 is eliminated.

۱۸
Backward Substitution
bn
xn =
a n ,n
bn −1 − a n −1,n xn
xn −1 =
a n −1,n −1
bn − 2 − a n − 2,n xn − a n − 2,n −1 xn −1
xn − 2 =
a n − 2, n − 2
n
bi − ∑ ai , j x j
j = i +1
xi =
a i ,i
۱۹
Example 1
Solve using Naive Gaussian Elimination :
Part 1 : Forward Elimination ___ Step1 : Eliminate x1 from equations 2, 3
x1 + 2 x2 + 3x3 = 8 eq1 unchanged ( pivot equation)
2
2 x1 + 3x2 + 2 x3 = 10 eq 2 ← eq 2 −  eq1
1
 3
3x1 + x2 + 2 x3 = 7 eq3 ← eq3 −  eq1
1
x1 + 2 x2 + 3 x3 = 8
− x2 − 4 x3 = −6
− 5 x2 − 7 x3 = − 17

۲۰
Example 1
Part 1 : Forward Elimination Step2 : Eliminate x2 from equation 3
x1 + 2 x2 + 3x3 = 8 eq1 unchanged
− x2 − 4 x3 = −6 eq 2 unchanged ( pivot equation)
 −5
− 5 x2 − 7 x3 = − 17 eq3 ← eq3 −  eq 2
 −1 

 x1 + 2 x2 + 3 x3 = 8

⇒  − x2 − 4 x3 = −6
 13 x3 =13


۲۱
Example 1
Backward Substitution

b3 13
x3 = = =1
a3,3 13
b2 − a2,3 x3 − 6 + 4 x3
x2 = = =2
a 2, 2 −1
b1 − a1, 2 x2 − a1,3 x3 8 − 2 x2 − 3 x3
x1 = = =1
a1,1 a1,1
 x1  1 
The solution is  x2  = 2
 x3  1 
۲۲
Example
Forward Elimination
 6 −2 2 4   x1   16 
 12 − 8 6 10   x   26 
  2 = 
 3 − 13 9 3   x3   − 19 
     
− 6 4 1 − 18  x4  − 34
Part 1 : Forward Elimination
Step1 : Eliminate x1 from equations 2, 3, 4
6 − 2 2 4   x1   16 
0 − 4 2 2  x   − 6 
  2 = 
0 − 12 8 1   x3  − 27
     
0 2 3 − 14  x4   − 18 
۲۳
Example
Forward Elimination
Step2 : Eliminate x2 from equations 3, 4
6 − 2 2 4   x1   16 
0 − 4 2 2  x   − 6 
   2 = 
0 0 2 − 5   x3   − 9 
     
0 0 4 − 13  4  − 21
x
Step3 : Eliminate x3 from equation 4
6 − 2 2 4   x1   16 
0 − 4 2 2   x  − 6
   2 = 
 0 0 2 − 5  x3  − 9
     
0 0 0 − 3  x4   − 3 
۲٤
 Example
Forward Elimination

Summary of the Forward Elimination :

 6 −2 2 4   x1   16  6 − 2 2 4   x1   16 
 12 − 8 6 10   x   26  0 − 4 2 2   x   − 6
  2 = ⇒   2 = 
 3 − 13 9 3   x3   − 19  0 0 2 − 5  x3  − 9
           
− 6 4 1 − 18  4 
x − 34   0 0 0 − 3  x4   − 3

۲٥
Example
Backward Substitution

6 − 2 2 4   x1   16 
0 − 4 2 2   x   − 6 
   2 = 
0 0 2 − 5  x3  − 9
    
0 0 0 − 3  x4   − 3
Solve for x4 , then solve for x3 ,... solve for x1
−3 −9+5
x4 = = 1, x3 = = −2
−3 2
− 6 − 2(−2) − 2(1) 16 + 2(1) − 2(−2) − 4(1)
x2 = = 1, x1 = =3
−4 6

۲٦
Naive Gaussian Elimination
o The method consists of two steps
o Forward Elimination: the system is reduced to
upper triangular form. A sequence of elementary
operations is used.
 a11 a12 a13   x1   b1  a11 a12 a13   x1   b1 
a a22 a23   x  = b  ⇒ 0 a22 ' a23 '  x  = b '
 21  2  2   2  2 
 a31 a32 a33   x3  b3   0 0 a33 '  x3  b3 '

o Backward Substitution: Solve the system starting

from the last variable. Solve for xn ,xn-1,…x1.

۲۷
Problems with Naive Gaussian Elimination
o The Naive Gaussian Elimination may fail for
very simple cases. (The pivoting element is zero).
0 1  x1  1 
1 1  x  = 2
   2  
o Very small pivoting element may result in
serious computation errors
10 −10 1  x1  1 
   = 
 1 1  x2  2
۲۸
Example 2
Solve the following system using Gaussian Elimination
with Scaled Partial Pivoting :

1 − 1 2 1  x1   1 
3 2 1 4 x   1 
   2 = 
5 8 6 3  x3   1 
4 2 5 3  x   − 1
  4  

۲۹
Example 2
Initialization step
Scale vector:
1 − 1 1  x1   1 
2 disregard sign
3 2     
4   x2   1 
1 find largest in
 = magnitude in
each row
5 8 3  x3   1 
6
     
4 2 3  x4  − 1
5
Scale vector S = [2 4 8 5]
Index Vector L = [1 2 3 4]
۳۰
Why Index Vector?
 Index vectors are used because it is much
easier to exchange a single index element
compared to exchanging the values of a
complete row.
 In practical problems with very large N,
exchanging the contents of rows may not
be practical.

۳۱
Example 2
Forward Elimination-- Step 1: eliminate x1

Selection of the pivot equation

1 − 1 2 1  x1   1 
3 2 4 x   1 
 1  2  =   ⇒ S = [2 4 8 5]
5 8 6 3  x3   1   L = [1 2 3 4 ]
     
4 2 5 3  x4  − 1
 al ,1   1 3 5 4 
Ratios =  i
i = 1,2,3,4 =  , , ,  ⇒ max corresponds to l4
 Sli   2 4 8 5 
equation 4 is the first pivot equation Exchange l4 and l1
L =[4 2 3 1 ]
۳۲
Example 2
Forward Elimination-- Step 1: eliminate x1
Update A and B
1 − 1 2 1  x1   1 
3 2 1 4 x   1 
  2 =  First pivot
5 8 6 3  x3   1 
      equation
4 2 5 3  x4  − 1
0 − 1.5 0.75 0.25   x1  1.25 
0 0.5 − 2.75 1.75   x  1.75 
⇒    2 = 
0 5.5 − 0.25 − 0.75  x3  2.25
     
4 2 5 3   4   −1 
x
۳۳
Example 2
Forward Elimination-- Step 2: eliminate x2
Selection of the second pivot equation
0 − 1.5 0.75 0.25   x1  1.25 
0 0.5 − 2.75 1.75   x  1.75 
   2 = 
0 5.5 − 0.25 − 0.75  x3  2.25
     
4 2 5 3   4   −1 
x
S = [2 4 8 5 ] L=[ 4 2 3 1]

 al , 2   0.5 5.5 1.5 

Ratios :  i i = 2,3,4 =  ⇒ L = [ 41 3 2 ]
 Sli   4 8 2 
۳٤
Example 2
Forward Elimination-- Step 3: eliminate x3
Third pivot
0 − 1.5 0.75 0.25   x1   1.25 
0   x  2.1667  equation
 0 − 2 . 5 1 . 8333   2 =  
0 0 0.25 1.6667  x3   6.8333
     
 4 2 5 3   4 
x − 1 
L = [ 4 1 2 3]
0 − 1.5 0.75 0.25   x1   1.25 
0 0 − 2 .5 1 .8333  x  2.1667 
   2 =  
0 0 0 2   x3   9 
     
 4 2 5 3   4 
x − 1 
۳٥
Example 2
Backward Substitution
0 − 1.5 0.75 0.25   x1   1.25  L = [ 4 1 2 3]
0 0 − 2.5 1.8333  x  2.1667
   2 = 
0 0 0 2   x3   9 
4 3  x   −1 
 2 5  4  
b3 9 b − a 2,4 x4 2.1667 − 1.8333x4
x4 = = = 4.5, x3 = 2 = = 2.4327
a3,4 2 a 2,3 − 2.5
b1 − a1,4 x4 − a1,3 x3 1.25 − 0.25 x4 − 0.75 x3
x2 = = = 1.1333
a1,2 − 1.5
b4 − a 4,4 x4 − a 4,3 x3 − a 4,2 x2 − 1 − 3 x 4 − 5 x3 − 2 x 2
x1 = = = −7.2333
al1 ,1 4

۳٦
Example 3
Solve the following sytstem using Gaussian Elimination
with Scaled Partial Pivoting

1 − 1 2 1  x1   1 
3 2 1 4 x   1 
  2 = 
5 − 8 6 3  x3   1 
     
4 2 5 3  x4  − 1

۳۷
Example 3
Initialization step

1 − 1 2 1  x1   1 
3 2 1     
4   x2   1 
 =
5 − 8 6 3  x3   1 
     
4 2 5 3  x4  − 1
Scale vector S = [2 4 8 5]
Index Vector L = [1 2 3 4]
۳۸
 Example 3
Forward Elimination-- Step 1: eliminate x1

Selection of the pivot equation

1 − 1 2 1  x1   1 
3 2 1 4 x   1 
  2  =   ⇒ S = [2 4 8 5]
5 − 8 6 3  x3   1   L = [1 2 3 4 ]
     
4 2 5 3  x4  − 1
 al ,1  1 3 5 4 
 i 
Ratios =  i = 1,2,3,4 =  , , ,  ⇒ max corresponds to l4
 Sli   2 4 8 5 
equation 4 is the first pivot equation Exchange l4 and l1
L =[4 2 3 1 ]
۳۹
Example 3
Forward Elimination-- Step 1: eliminate x1
Update A and B
1 − 1 2 1  x1   1 
3 3 1 4 x   1 
  2 = 
5 − 8 6 3  x3   1 
     
4 2 5 3  x4  − 1
0 − 1.5 0.75 0.25   x1  1.25 
0 0.5 − 2.75 1.75   x  1.75 
⇒    2 = 
0 − 10.5 − 0.25 − 0.75  x3  2.25
     
4 2 5 3   4   −1 
x
٤۰
Example 3
Forward Elimination-- Step 2: eliminate x2
Selection of the second pivot equation
0 − 1.5 0.75 0.25   x1  1.25 
0 0.5 − 2.75 1.75   x  1.75 
   2 = 
0 − 10.5 − 0.25 − 0.75  x3  2.25
     
4 2 5 3   x4   − 1 
S = [2 4 8 5 ] L=[ 4 2 3 1 ]

 al , 2 
 i   0.5 10.5 1.5 
Ratios :  i = 2,3,4 =   ⇒ L = [ 4 3 2 1]
 Sli   4 8 2 
٤۱
Example 3
Forward Elimination-- Step 2: eliminate x2
Updating A and B
0 − 1.5 0.75 0.25   x1  1.25 
0 0.5 − 2.75 1.75   x  1.75 
   2 = 
0 − 10.5 − 0.25 − 0.75  x3  2.25
     
4 2 5 3   x4   − 1 
L = [ 4 1 3 2]
0 0 0.7857 0.3571  x1  0.9286
0 0 - 2.7619 1.7143   x  1.8571
   2 = 
0 − 10.5 − 0.25 − 0.75   x3   2.25 
     
 4 2 5 3   4 
x − 1 
٤۲
 Example 3
Forward Elimination-- Step 3: eliminate x3
Selection of the third pivot equation
0 0 0.7857 0.3571  x1  0.9286
0 0 − 2.7619 1.7143   x  1.8571
   2 = 
0 − 10.5 − 0.25 − 0.75   x3   2.25 
     
4 2 5 3   x4   − 1 
S = [2 4 8 5 ] L=[ 4 3 2 1 ]

 al ,3   2.7619 0.7857 
Ratios :  i
i = 3,4 =   ⇒ L = [ 4 3 2 1]
 Sli   4 2 
٤۳
Example 3
Forward Elimination-- Step 3: eliminate x3
0 0 0.7857 0.3571  x1  0.9286
0 0 − 2.7619 1.7143   x  1.8571
   2 = 
0 − 10.5 − 0.25 − 0.75   x3   2.25 
     
 4 2 5 3   4 
x − 1 
L = [ 4 3 2 1]
0 0 0 0.8448  x1  1.4569
0 0 − 2.7619 1.7143   x  1.8571
   2 = 
0 − 10.5 − 0.25 − 0.75   x3   2.25 
     
4 2 5 3   x4   − 1 

٤٤
Example 3
Backward Substitution
0 0 0 0.8448  x1  1.4569 L = [ 4 3 2 1]
0 0 − 2.7619 1.7143  x2  1.8571
    = 
0 − 10.5 − 0.25 − 0.75   x3   2.25 
4
 2 5 3   x4   − 1 
bl4 1.4569 bl3 − al3 ,4 x4 1.8571 − 1.7143x4
x4 = = = 1.7245, x3 = = = 0.3980
al4 ,4 0.8448 a l3 ,3 − 2.7619
bl2 − al2 ,4 x4 − al2 ,3 x3
x2 = = −0.3469
al2 ,2
bl1 − al1 ,4 x4 − al1 ,3 x3 − al1 ,2 x2 − 1 − 3 x 4 − 5 x3 − 2 x 2
x1 = = = −1.8673
al1 ,1 4

٤٥
Remarks:
 We use index vector to avoid the need to move
the rows which may not be practical for large
problems.
 If we order the equation as in the last value of
the index vector, we have a triangular form.
 Scale vector is formed by taking maximum in
magnitude in each row.
 Scale vector does not change.

٤٦
Gauss-Jordan Method
 The method reduces the general system of
equations AX=B to IX=B where I is an identity
matrix.

 Only Forward elimination is done and no

backward substitution is needed.

 It has the same problems as Naive Gaussian

elimination and can be modified to do partial
scaled pivoting.

 It takes 50% more time than Naive Gaussian

method.
٤۷
Gauss-Jordan Method
Example

2 − 2 2   x1  0
4 2 − 1  x  = 7 
   2  
2 − 2 4   x3  2
Step 1 Eleminate x1 from equations 2and 3

eq1 ← eq1 / 2 
 1 − 1 1   x1  0
4  
eq 2 ← eq 2 −  eq1 ⇒ 0 6 − 5  x2  = 7 
1 
0 0 2   x3  2
  
2
eq3 ← eq3 −  eq1
1 
٤۸
Gauss-Jordan Method
Example

1 − 1 1   x1  0
0 6 − 5  x  = 7 
   2  
0 0 2   x3  2
Step 2 Eleminate x 2 from equations 1 and 3
eq 2 ← eq 2 / 6  1 0 0.1667   x1  1.1667 
 −1     x  = 1.1667 
eq1 ← eq1 −  eq 2 ⇒ 0 1 − 0.8333  2  
 1   0 0
  2   x3   2 

٤۹
Gauss-Jordan Method
Example

1 0 0.1667   x1  1.1667 
0 1 − 0.8333  x  = 1.1667 
   2  
0 0 2   x3   2 
Step 3 Eleminate x 3 from equations 1 and 2

eq3 ← eq3 / 2 
 1 0 0  x1  1 
 0.1667      x  = 2
eq1 ← eq1 −   eq 3 ⇒
  0 1 0   2  
 1   0 0 1   x  1 
 − 0.8333      3  
eq 2 ← eq 2 −  eq3
 1  

٥۰
Gauss-Jordan Method
Example

2 − 2 2   x1  0
4 2 − 1  x  = 7 
   2  
2 − 2 4   x3  2
is transformed to
1 0 0  x1  1   x1  1 
0 1 0  x  = 2 ⇒ solution is  x  = 2
   2    2  
0 0 1  x3  1   x3  1 
٥۱
 University of Technology
Petroleum Technology Department

Numerical Analysis

Lecture 7: Solution Boundary Value Problems

Semester 1 (2020-2021)

Doaa Saleh Mahdi

1
Outlines
 Gauss-Jordan Method
 Boundary Value Problem
 Shooting Method
 Finite Difference Methods
 Examples

2
Learning Objectives
 Grasp the difference between initial value
problems and boundary value problems.
 Appreciate the difficulties involved in solving the
boundary value problems.
 Use the shooting method to solve boundary
value problems.
 Use the finite difference method to solve BVP.

3
Gauss-Jordan Method
 The method reduces the general system of
equations AX=B to IX=B where I is an identity
matrix.

 Only Forward elimination is done and no

backward substitution is needed.

 It has the same problems as Naive Gaussian

elimination and can be modified to do partial
scaled pivoting.

 It takes 50% more time than Naive Gaussian

method.
4
Gauss-Jordan Method
Example

2  2 2   x1  0
4 2  1  x   7 
   2  
2  2 4   x3  2
Step 1 Eleminate x1 from equations 2and 3

eq1  eq1 / 2 
 1  1 1   x1  0
4  
eq 2  eq 2   eq1  0 6  5  x2   7
1 
0 0 2   x3  2
  
2
eq 3  eq 3   eq1
1 
5
Gauss-Jordan Method
Example

1  1 1   x1  0
0 6  5  x   7
   2  
0 0 2   x3  2
Step 2 Eleminate x 2 from equations 1 and 3
eq 2  eq 2 / 6  1 0 0.1667   x1  1.1667 
 1   
eq1  eq1   eq 2  0 1  0.8333  x   1.1667 
 2  
 1   0 0
  2   x3   2 

6
Gauss-Jordan Method
Example

1 0 0.1667   x1  1.1667 
0 1  0.8333  x   1.1667 
   2  
0 0 2   x3   2 
Step 3 Eleminate x 3 from equations 1 and 2

eq 3  eq 3 / 2 
 1 0 0  x1  1
 0.1667      x    2
eq1  eq1    eq 3 
  0 1 0   2  
 1   0 0 1   x  1 
  0.8333      3  
eq 2  eq 2   eq 3
 1  

7
Boundary-Value and
Initial Value Problems
Initial-Value Problems Boundary-Value Problems

 The auxiliary conditions  The auxiliary conditions are

are at one point of the not at one point of the
independent independent variable
variable  More difficult to solve than
initial value problem

2 t
x  2 x  x  e x  2 x  x  e 2t
x(0)  1, x (0)  2.5 x(0)  1, x(2)  1.5
same different

8
Shooting Method

9
The Shooting Method

Target

10
The Shooting Method

Target

11
Solution of Boundary-Value Problems
Shooting Method for Boundary-Value Problems

1. Guess a value for the auxiliary conditions at one

point of time.
2. Solve the initial value problem using Euler,
Runge-Kutta, …
3. Check if the boundary conditions are satisfied,
otherwise modify the guess and resolve the
problem.
 Use interpolation in updating the guess.
 It is an iterative procedure and can be
efficient in solving the BVP.
12
Solution of Boundary-Value Problems
Shooting Method
Boundary-Value Initial-value
convert
Problem Problem
1. Convert the ODE to a system of
first order ODEs.
2. Guess the initial conditions that
Find y ( x) to solve BVP are not available.
3. Solve the Initial-value problem.
y  2 y  y  x
2
4. Check if the known boundary
conditions are satisfied.
y (0)  0.2, y (1)  0.8 5. If needed modify the guess and
resolve the problem again.

13
Example 1
Original BVP
y  4 y  4 x  0
y (0)  0, y (1)  2

0 1 x

24
Example 1
Original BVP
y  4 y  4 x  0
y (0)  0, y (1)  2

2. 0

0 1 x

25
Example 1
Original BVP
y  4 y  4 x  0
y (0)  0, y (1)  2

2. 0

0 1 x

26
Example 1
Original BVP
y  4 y  4 x  0
y (0)  0, y (1)  2

2. 0

to be
determined

0 1 x

27
Example 1
Step1: Convert to a System of First Order ODEs
y  4 y  4 x  0
y (0)  0, y (1)  2
Convert to a systemof first order Equations
 y 1   y 2   y1 (0)  0
 y   4(y  x),  y (0)  ? 
 2  1   2   

The problem will be solved using RK2 with h  0.01

for different values of y 2 (0) until we have y(1)  2
28
Example 1
Guess # 1
y  4 y  4 x  0
y (0)  0, y (1)  2

Guess#1
y (0)  0
Thus _ from _ RK 2
0 1 x
y (1)  0.7688
-0.7688

29
Example 1
Guess # 2
y  4 y  4 x  0
y (0)  0, y (1)  2
0.99
Guess#2
y (0)  1
From _ RK 2 0 1 x

y (1)  0.99

30
Example 1
Interpolation for Guess # 3
y  4 y  4 x  0 y(1)

y (0)  0, y (1)  2
0.99
Guess y (0) y(1)

1 0 -0.7688
0 1 2 y’(0)
2 1 0.9900
-0.7688

31
Example 1
Interpolation for Guess # 3
y  4 y  4 x  0 y(1)

y (0)  0, y (1)  2 2

Guess 3
0.99
Guess y (0) y(1)
1.5743
1 0 -0.7688
0 1 2 y’(0)
2 1 0.9900
-0.7688

32
Example 1
Guess # 3
y  4 y  4 x  0 2.000

y (0)  0, y (1)  2

Guess#3
y (0)  1.5743

0 1 x

This is the solution to the

y(1)=2.000
boundary value problem.

33
Summary of the Shooting Method
1. Guess the unavailable values for the
auxiliary conditions at one point of the
independent variable.
2. Solve the initial value problem.
3. Check if the boundary conditions are
satisfied, otherwise modify the guess and
resolve the problem.
4. Repeat (3) until the boundary conditions
are satisfied.

34
Properties of the Shooting Method

1. Using interpolation to update the guess often

results in few iterations before reaching the
solution.

2. The method can be cumbersome for high order

BVP because of the need to guess the initial
condition for more than one variable.

35
Solution of Boundary-Value Problems
Finite Difference Method
Boundary-Value convert
Algebraic
Problems Equations
Find the unknowns y1, y2, y3

y4=0.8
Find y ( x) to solve BVP y3=?
y y1=?
y  2 y  y  x
2
y2=?
y0=0.2
y (0)  0.2, y (1)  0.8
0 0.25 0.5 0.75 1.0 x
x0 x1 x2 x3 x4

36
Solution of Boundary-Value Problems
Finite Difference Method
 Divide the interval into n sub-intervals.
 The solution of the BVP is converted to
the problem of determining the value of
function at the base points.
 Use finite approximations to replace the
derivatives.
 This approximation results in a set of
algebraic equations.
 Solve the equations to obtain the solution
of the BVP.
37
Finite Difference Method
Example
y  2 y  y  x 2
y (0)  0.2, y (1)  0.8 To be
determined
Divide the interval
[0,1 ] into n = 4 y y3=? y4=0.8
intervals y1=?
Base points are y2=?
x0=0
x1=0.25 y0=0.2

x2=.5
x3=0.75 0 0.25 0.5 0.75 1.0 x
x4=1.0 x0 x1 x2 x3 x4
38
Finite Difference Method
Example
y  2 y  y  x 2
y (0)  0.2, y (1)  0.8
Divide the interval Replace
[0,1 ] into n = 4 yi 1  2 yi  yi 1
intervals y  2
central difference formula
h
Base points are y y
y  i 1 i 1 central difference formula
x0=0 2h
x1=0.25 y  2 y  y  x 2
x2=.5 Becomes
x3=0.75 yi 1  2 yi  yi 1 yi 1  yi 1
2  yi  xi 2
h2 2h
x4=1.0

39
Second Order BVP
d2y dy
2
 2  y  x 2
with y (0)  0.2, y (1)  0.8
dx dx
Let h  0.25
Base Points
x0  0, x1  0.25, x2  0.5, x3  0.75, x4  1
dy y ( x  h)  y ( x) yi 1  yi
 
dx h h
d2y y ( x  h)  2 y ( x )  y ( x  h) yi 1  2 yi  yi 1
2
 2

dx h h2
40
Second Order BVP
d2y dy
 2  y  x 2
dx 2 dx
yi 1  2 yi  yi 1 yi 1  yi
2
 2  yi  xi i  1,2,3
2
h h
x0  0, x1  0.25, x2  0.5, x3  0.75, x4  1
y0  0.2, y1  ?, y2  ?, y3  ?, y4  0.8
16 yi 1  2 yi  yi 1   8 yi 1  yi   yi  xi 2

24 yi 1  39 yi  16 yi 1  xi 2
41
Second Order BVP
24 yi 1  39 yi  16 yi 1  xi
2

i 1 24 y2  39 y1  16 y0  x1
2

i2 24 y3  39 y2  16 y1  x2
2

i3 24 y4  39 y3  16 y2  x3
2

 39 24 0   y1   0.252  16(0.2) 
 16  39 24   y    0.5 2 
  2   
 0 16  39  y3  0.752  24(0.8)
Solution y1  0.4791, y2  0.6477, y3  0.7436
42
Second Order BVP
d2y dy
 2  y  x 2
dx 2 dx
yi 1  2 yi  yi 1 yi 1  yi
2
2  yi  xi i  1,2,...,100
2
h h
x0  0, x1  0.01, x2  0.02 ... x99  0.99, x100  1
y0  0.2, y1  ?, y2  ?, ... y99  ?, y100  0.8
10000 yi 1  2 yi  yi 1   200 yi 1  yi   yi  xi 2

10200 yi 1  20199 yi  10000 yi 1  xi 2

43
Summary of the Discretiztion Methods
 Select the base points.
 Divide the interval into n sub-intervals.
 Use finite approximations to replace the
derivatives.
 This approximation results in a set of
algebraic equations.
 Solve the equations to obtain the solution
of the BVP.

44
Remarks
Finite Difference Method :
 Different formulas can be used for
approximating the derivatives.
 Different formulas lead to different
solutions. All of them are approximate
solutions.

45
Example

46
Example

47
 University of Technology
Petroleum Technology Department

Numerical Analysis

Lecture 8: Least Squares Curve Fitting

Semester 1 (2020-2021)

Doaa Saleh Mahdi

۱
Outlines
 Introduction to Least Squares
 Linear Regression
 Nonlinear Least Squares
 Nonlinear Least Squares Problems
 Examples

۲
 Motivation
Given a set of experimental data:
x 1 2 3
y 5.1 5.9 6.3

• The relationship between

x and y may not be clear.
• Find a function f(x) that
best fit the data
1 2 3

۳
Motivation
 In engineering, two types of applications are
encountered:
 Trend analysis: Predicting values of dependent
variable, may include extrapolation beyond data points
or interpolation between data points.
 Hypothesis testing: Comparing existing mathematical
model with measured data.
1. What is the best mathematical function f that
represents the dataset?
2. What is the best criterion to assess the fitting of
the function f to the data?

٤
Curve Fitting
Given a set of tabulated data, find a curve
or a function that best represents the
data.
Given:
1. The tabulated data
2. The form of the function

3. The curve fitting criteria

Find the unknown coefficients

٥
Least Squares Regression
Linear Regression
 Fitting a straight line to a set of paired
observations:
(x1, y1), (x2, y2),…,(xn, yn).
y=a0+a1x+e
a1-slope.
a0-intercept.
e-error, or residual, between the model
and the observations.
٦
Selection of the Functions
Linear f ( x) = a + bx
Quadratic f ( x) = a + bx + cx 2
n
Polynomial f ( x) = ∑ ak x k
k =0
m
General f ( x ) = ∑ ak g k ( x )
k =0

g k ( x) are known.
۷
Decide on the Criterion
1. Least Squares Regression :
n
minimize ∑ i
( y − f ( x i )) 2

i =1

2. Exact Matching (Interpolation) :

y i = f ( xi )

۸
Least Squares Regression
Given: xi x1 x2 …. xn
yi y1 y2 …. yn

The form of the function is assumed to be

known but the coefficients are unknown.

ei = ( yi − f ( xi )) 2 = ( f ( xi ) − yi ) 2
2

The difference is assumed to be the result of

experimental error.

۹
Determine the Unknowns

We want to find a and b to minimize :

n
Φ ( a , b) = ∑ ( a + bxi − yi ) 2
i =1

How do we obtain a and b to minimize : Φ ( a, b) ?

۱۰
Determine the Unknowns
Necessary condition for the minimum :

∂ Φ ( a, b)
=0
∂a
∂ Φ ( a, b)
=0
∂b

۱۱
Determining the Unknowns

∂ Φ ( a , b) n
= ∑ 2(a + bxi − yi ) = 0
∂a i =1

∂ Φ ( a , b) n
= ∑ 2(a + bxi − yi )xi = 0
∂b i =1

۱۲
Normal Equations

 n   n 
n a +  ∑ xi  b =  ∑ yi 
 i =1   i =1 
 n   n 2  n 
 ∑ xi  a +  ∑ xi  b =  ∑ xi y i 
     
 i =1   i =1   i =1 

۱۳
Solving the Normal Equations
 n   n  n 
n  ∑ xi yi  −  ∑ xi   ∑ yi 
b=  i =1   i =1   i =1 
2
 n   n 
n  ∑ xi  −  ∑ xi 
2

 i =1   i =1 

1  n   n 
a =   ∑ yi  − b  ∑ xi  
n   i =1   i =1  
۱٤
Example 1: Linear Regression
Assume :
f(x) = a + bx x 1 2 3
y 5.1 5.9 6.3
Equations :
 n   n 
n a +  ∑ xi  b =  ∑ yi 
 i =1   i =1 
 n   n 2  n 
 ∑ xi  a +  ∑ xi  b =  ∑ xi y i 
     
 i =1   i =1   i =1 
۱٥
Example 1: Linear Regression
i 1 2 3 sum
xi 1 2 3 6
yi 5.1 5.9 6.3 17.3
xi2 1 4 9 14
xi yi 5.1 11.8 18.9 35.8
Equations :
3 a + 6 b = 17.3
6a + 14 b = 35.8
Solving : a = 4.5667 b = 0.60
۱٦
Multiple Linear Regression
Example: t 0 1 2 3

x 0.1 0.4 0.2 0.2

Given the following data:
y 3 2 1 2

Determine a function of two variables:

f(x,t) = a + b x + c t
That best fits the data with the least sum of the
square of errors.
۱۷
Solution of Multiple Linear Regression
Construct Φ , the sum t 0 1 2 3
of the square of the
x 0.1 0.4 0.2 0.2
error and derive the
necessary conditions by y 3 2 1 2
equating the partial
derivatives with respect
to the unknown
parameters to zero, then
solve the equations.

۱۸
Solution of Multiple Linear Regression
n
f ( x, t ) = a + bx + ct , Φ ( a , b, c ) = ∑ (a + bxi + cti − yi )2
i =1
Necessary conditions :
∂Φ ( a , b, c ) n
= 2∑ (a + bxi + cti − yi ) = 0
∂a i =1

∂Φ ( a , b, c ) n
= 2∑ (a + bxi + cti − yi ) xi = 0
∂b i =1

∂Φ ( a , b, c ) n
= 2∑ (a + bxi + cti − yi ) ti = 0
∂c i =1
۱۹
System of Equations
n n n
a n + b ∑ xi + c ∑ ti = ∑ y i
i =1 i =1 i =1
n n n n
a ∑ xi + b ∑ ( x i ) 2 + c ∑ ( x i t i ) = ∑ ( x i y i )
i =1 i =1 i =1 i =1
n n n n
a ∑ ti + b∑ ( xi ti ) + c ∑ (ti ) = ∑ (ti yi )
2

i =1 i =1 i =1 i =1

۲۰
Example 2: Multiple Linear Regression
i 1 2 3 4 Sum
ti 0 1 2 3 6
xi 0.1 0.4 0.2 0.2 0.9
yi 3 2 1 2 8
xi2 0.01 0.16 0.04 0.04 0.25
xi ti 0 0.4 0.4 0.6 1.4
xi yi 0.3 0.8 0.2 0.4 1.7
ti2 0 1 4 9 14
ti yi 0 2 2 6 10

۲۱
Example 2: System of Equations

4a + 0.9b + 6c = 8
0.9a + 0.25b + 1.4c = 1.7
6a + 1.4b + 14c = 10

Solving :
a = 2.9574, b = −1.7021, c = −0.38298
f ( x, t ) = a + bx + ct = 2.9574 − 1.7021 x − 0.38298 t

۲۲
Polynomial Regression
 The least squares method can be extended to fit
the data to a higher-order polynomial

f ( x ) = a + bx + cx ,
2 2
ei = ( f ( x ) − yi ) 2

( )
n
Minimize Φ ( a , b, c ) = ∑
2
a + bxi + cxi2 − yi
i =1
Necessary conditions :
∂Φ ( a , b, c ) ∂Φ ( a , b, c ) ∂Φ ( a , b, c )
= 0, = 0, =0
∂a ∂b ∂c
۲۳
Equations for Quadratic Regression
( )
n
Minimize Φ ( a, b, c ) = ∑
2
a + bxi + cxi2 − yi
i =1

∂Φ ( a, b, c )
( )
n
= 2∑ a + bxi + cxi2 − yi = 0
∂a i =1

∂Φ ( a, b, c )
( )
n
= 2∑ a + bxi + cxi2 − yi xi = 0
∂b i =1

∂Φ ( a, b, c )
( )
n
= 2∑ a + bxi + cxi2 − yi xi2 = 0
∂c i =1
۲٤
Normal Equations
n n n
a n + b ∑ xi + c ∑ xi2 = ∑ yi
i =1 i =1 i =1
n n n n
a ∑ xi + b ∑ 2
xi +c∑ 3
xi = ∑ xi y i
i =1 i =1 i =1 i =1
n n n n
a ∑ xi2 + b ∑ xi3 + c ∑ xi4 = ∑ xi2 yi
i =1 i =1 i =1 i =1

۲٥
Example 3: Polynomial Regression
Fit a second-order polynomial to the following data
xi 0 1 2 3 4 5 ∑=15
yi 2.1 7.7 13.6 27.2 40.9 61.1 ∑=152.6

xi2 0 1 4 9 16 25 ∑=55
xi3 0 1 8 27 64 125 225
xi4 0 1 16 81 256 625 ∑=979
xi y i 0 7.7 27.2 81.6 163.6 305.5 ∑=585.6

xi2 yi 0 7.7 54.4 244.8 654.4 1527.5 ∑=2488.8

۲٦
Example 3: Equations and Solution
6 a + 15 b + 55 c = 152.6
15 a + 55 b + 225 c = 585.6
55 a + 225 b + 979 c = 2488.8
Solving . . .
a = 2.4786, b = 2.3593, c = 1.8607
f ( x ) = 2.4786 + 2.3593 x + 1.8607 x 2

۲۷
How Do You Judge Functions?
Given two or more functions to fit the data,
How do you select the best?

Answer :
Determine the parameters for each function,
then compute Φ for each one. The function
resulting in smaller Φ (least sum of the squares
of the errors) is the best.

۲۸
Example showing that Quadratic is
preferable than Linear Regression

y y

x x

Linear Regression Quadratic Regression

۲۹
Fitting with Nonlinear Functions
xi 0.24 0.65 0.95 1.24 1.73 2.01 2.23 2.52

yi 0.23 -0.23 -1.1 -0.45 0.27 0.1 -0.29 0.24

It is required to find a function of the form :

f ( x ) = a ln( x ) + b cos( x ) + c e x
to fit the data.
n
Φ ( a , b, c ) = ∑ ( f ( xi ) − yi ) 2
i =1

۳۰
Fitting with Nonlinear Functions
n
Φ ( a , b, c ) = ∑ ( a ln( xi ) + b cos( xi ) + c e xi − yi ) 2
i =1
Necessary condition for the minimum :
∂ Φ ( a , b, c ) 
= 0
∂a

∂ Φ ( a , b, c ) 
= 0 ⇒ Normal Equations
∂b 
∂ Φ ( a, b, c ) 
= 0
∂c  ۳۱
Normal Equations
n n n n
a ∑ (ln xi ) + b∑ (ln xi )(cos xi ) + c ∑ (ln xi )( e ) = ∑ yi (ln xi )
2 xi

i =1 i =1 i =1 i =1

n n n n
a ∑ (ln xi )(cos xi ) + b∑ (cos xi ) + c ∑ (cos xi )( e ) = ∑ yi (cos xi )
2 xi

i =1 i =1 i =1 i =1

n n n n
a ∑ (ln xi )( e ) + b∑ (cos xi )( e ) + c ∑ ( e ) = ∑ yi ( e xi )
xi xi xi 2

i =1 i =1 i =1 i =1

Evaluate the sums and solve the normal equations.

۳۲
 Example 4: Evaluating Sums
xi 0.24 0.65 0.95 1.24 1.73 2.01 2.23 2.52 ∑=11.57
yi 0.23 -0.23 -1.1 -0.45 0.27 0.1 -0.29 0.24 ∑=-1.23

(ln xi)2 2.036 0.1856 0.0026 0.0463 0.3004 0.4874 0.6432 0.8543 ∑=4.556

ln(xi) cos(xi) -1.386 -0.3429 -0.0298 0.0699 -0.0869 -0.2969 -0.4912 -0.7514 ∑=-3.316

ln(xi) * exi -1.814 -0.8252 -0.1326 0.7433 3.0918 5.2104 7.4585 11.487 ∑=25.219

yi * ln(xi) -0.328 0.0991 0.0564 -0.0968 0.1480 0.0698 -0.2326 0.2218 ∑=-0.0625

cos(xi)2 0.943 0.6337 0.3384 0.1055 0.0251 0.1808 0.3751 0.6609 ∑=3.26307

cos(xi) * exi 1.235 1.5249 1.5041 1.1224 -0.8942 -3.1735 -5.696 -10.104 ∑=-14.481

yi*cos(xi) 0.223 -0.1831 -0.6399 -0.1462 -0.0428 -0.0425 0.1776 -0.1951 ∑=-0.8485

(exi)2 1.616 3.6693 6.6859 11.941 31.817 55.701 86.488 154.47 ∑=352.39

yi * exi 0.2924 -0.4406 -2.844 -1.555 1.523 0.7463 -2.697 2.9829 ∑=-1.9923

۳۳
Example 4: Equations & Solution
4.55643 a − 3.31547 b + 25.2192 c = −0.062486
− 3.31547 a + 3.26307 b − 14.4815 c = −0.848514
25.2192 a − 14.4815 b + 352.388 c = −1.992283

Solving the above equations :

a = − 0.88815, b = − 1.1074 , c = 0.012398
Therefore,
f ( x ) = −0.88815 ln( x ) − 1.1074 cos( x ) + 0.012398 e x

۳٤
Example 5
xi 1 2 3
Given:
yi 2.4 5 9

Find a function f(x) = ae bx that best fits the data.

( )
n
Φ ( a, b) = ∑ ae
2
bxi
− yi
i =1
Normal Equations are obtained using :
∂Φ
( )
n
= 2∑ ae bxi − yi e bxi = 0
∂a i =1 Difficult to Solve
∂Φ
( )
n
= 2∑ ae bxi − yi a xi e bxi = 0
∂b i =1 ۳٥
Linearization Method
Find a function f(x) = ae bx that best fits the data.
Define g ( x ) = ln( f ( x )) = ln( a ) + b x
Define zi = ln( yi ) = ln( a ) + bxi
Let α = ln( a ) and zi = ln( yi )

( )
n
Instead of minimizing : Φ ( a, b) = ∑ ae
2
bxi
− yi
i =1
n
Minimize : Φ (α , b) = ∑ (α + bxi − zi )2 ( Easier to solve)
i =1
۳٦
Example 5: Equations
n
Φ (α , b) = ∑ (α + b xi − zi )2
i =1
Normal Equations are obtained using :
∂Φ n
= 2∑ (α + b xi − zi ) = 0
∂α i =1

∂Φ n
= 2∑ (α + b xi − zi ) xi = 0
∂b i =1
n n n n n
α n + b ∑ xi = ∑ zi and α ∑ xi + b∑ xi2 = ∑ ( xi zi )
i =1 i =1 i =1 i =1 i =1
۳۷
Evaluating Sums and Solving
xi 1 2 3 ∑=6
yi 2.4 5 9
zi=ln(yi) 0.875469 1.609438 2.197225 ∑=4.68213
xi2 1 4 9 ∑=14
xi z i 0.875469 3.218876 6.591674 ∑=10.6860

Equations :
3 α + 6 b = 4.68213 α = ln(a ), a = eα
6 α + 14 b = 10.686 a = e 0.23897 = 1.26994
Solving Equations :
f ( x ) = ae bx = 1.26994 e 0.66087 x
α = 0.23897, b = 0.66087
۳۸
Power Equation

۳۹
Power Equation

٤۰
Power Equation

٤۱
Power Equation

٤۲
٤۳
Activities

٤٤
 University of Technology
Petroleum Technology Department

Numerical Analysis

Lecture 9 : Interpolation
Semester 1 (2020-2021)

Doaa Saleh Mahdi

۱
Outlines
Introduction
InterpolationProblem
Lagrange Interpolation
Linear and Quadratic Interpolation
Newton’s Divided Difference Method
Inverse Interpolation

۲
Introduction
Interpolation was used for x sin(x)
long time to provide an
0 0.0000
estimate of a tabulated
function at values that are 0.1 0.0998
not available in the table.
0.2 0.1987

What is sin (0.15)? 0.3 0.2955

0.4 0.3894

Using Linear Interpolation sin (0.15) ≈ 0.1493

True value (4 decimal digits) sin (0.15) = 0.1494
۳
The Interpolation Problem
Given a set of n+1 points,

(x0 , f ( x0 ) ), (x1, f ( x1 ) ), ...., (xn , f ( xn ) )

Find an nth order polynomial f n (x )
that passes through all points, such that:

f n ( xi ) = f ( xi ) for i = 0,1, 2,..., n

٤
 Example
Temperature Viscosity
An experiment is used to determine (degree)
the viscosity of water as a function
of temperature. The following table
0 1.792
is generated:
5 1.519

10 1.308
Problem: Estimate the viscosity
when the temperature is 8 degrees.
15 1.140

٥
Interpolation Problem
Find a polynomial that fits the data
points exactly.
V : Viscosity
n
V(T) = ∑ ak T k T : Temperature
k =0 ak : Polynomial
Vi = V(Ti ) coefficients
Linear Interpolation: V(T)= 1.73 − 0.0422 T
V(8)= 1.3924
٦
Existence and Uniqueness
Given a set of n+1 points:
(x0 , f ( x0 ) ), (x1, f ( x1 ) ), ...., (xn , f ( xn ) )
Assumption: x0 , x1 ,..., xn are distinct

Theorem:
There is a unique polynomial fn(x) of order ≤ n
such that:
f n ( xi ) = f ( xi ) for i = 0 ,1,...,n
۷
Examples of Polynomial Interpolation
Linear Interpolation Quadratic Interpolation

 Given any two points, Given any three points there

there is one polynomial of is one polynomial of order ≤
order ≤ 1 that passes 2 that passes through the
through the two points. three points.

۸
Linear Interpolation
Given any two points, ( x0 , f ( x0 ) ), ( x1 , f ( x1 ) )

The line that interpolates the two points is:

f ( x1 ) − f ( x0 )
f1 ( x) = f ( x0 ) + (x − x0 )
x1 − x0
Example :
Find a polynomial that interpolates (1,2) and (2,4).

4−2
f1 ( x) = 2 + (x − 1) = 2 x
2 −1

۹
Quadratic Interpolation
 Given any three points: (x0 , f ( x0 ) ), (x1 , f ( x1 ) ), and (x2 , f ( x2 ) )
 The polynomial that interpolates the three points is:
f 2 ( x ) = b0 + b1 ( x − x0 ) + b2 ( x − x0 )( x − x1 )
where :
b0 = f ( x0 )
f ( x1 ) − f ( x0 )
b1 = f [ x0 , x1 ] =
x1 − x0
f ( x2 ) − f ( x1 ) f ( x1 ) − f ( x0 )
−
x2 − x1 x1 − x0
b2 = f [ x0 , x1 , x2 ] =
x2 − x0

۱۰
General nth Order Interpolation
Given any n+1 points: (x0 , f ( x0 ) ), (x1 , f ( x1 ) ), ..., (xn , f ( xn ) )
The polynomial that interpolates all points is:

f n ( x) = b0 + b1 (x − x0 ) + b2 ( x − x0 )(x − x1 ) + ... + bn ( x − x0 )...(x − xn −1 )

b0 = f ( x0 )
b1 = f [ x0 , x1 ]
....
bn = f [ x0 , x1 , ... , xn ]

۱۱
 Divided Differences
f [ xk ] = f ( xk ) Zeroth order DD
f [ x1 ] − f [ x0 ]
f [ x0 , x1 ] = First order DD
x1 − x0
f [ x1 , x2 ] − f [ x0 , x1 ]
f [ x0 , x1 , x2 ] = Second order DD
x2 − x0
............
f [ x1 , x2 ,..., xk ] − f [ x0 , x1 ,..., xk −1 ]
f [ x0 , x1 ,..., xk ] =
xk − x0

۱۲
Divided Difference Table
x F[ ] F[ , ] F[ , , ] F[ , , ,]
x0 F[x0] F[x0,x1] F[x0,x1,x2] F[x0,x1,x2,x3]
x1 F[x1] F[x1,x2] F[x1,x2,x3]
x2 F[x2] F[x2,x3]
x3 F[x3]

 i −1 
(x − x j )
n
f n ( x) = ∑  F [ x0 , x1 ,..., xi ] ∏
i =0  j =0 

۱۳
Divided Difference Table
x F[ ] F[ , ] F[ , , ]
xi f(xi)
0 -5 2 -4
1 -3 6
0 -5
-1 -15 1 -3
-1 -15
Entries of the divided difference
table are obtained from the data
table using simple operations.

۱٤
Divided Difference Table
x F[ ] F[ , ] F[ , , ] xi f(xi)
0 -5 2 -4 0 -5
1 -3 6 1 -3
-1 -15 -1 -15

The first two column of the

table are the data columns.
Third column: First order differences.
Fourth column: Second order differences.
۱٥
Divided Difference Table
x F[ ] F[ , ] F[ , , ] xi yi
0 -5 2 -4 0 -5
1 -3 6
-1 -15 1 -3
-1 -15
− 3 − (−5)
=2
1− 0
f [ x1 ] − f [ x0 ]
f [ x0 , x1 ] =
x1 − x0

۱٦
Divided Difference Table
x F[ ] F[ , ] F[ , , ] xi yi
0 -5 2 -4 0 -5
1 -3 6
-1 -15 1 -3
-1 -15
− 15 − (−3)
=6
−1−1

f [ x2 ] − f [ x1 ]
f [ x1 , x2 ] =
x2 − x1

۱۷
Divided Difference Table
x F[ ] F[ , ] F[ , , ] xi yi
0 -5 2 -4 0 -5
1 -3 6
-1 -15 1 -3
-1 -15

6 − (2)
= −4
− 1 − ( 0)
f [ x1 , x2 ] − f [ x0 , x1 ]
f [ x0 , x1 , x2 ] =
x2 − x0
۱۸
Divided Difference Table
x F[ ] F[ , ] F[ , , ] xi yi
0 -5 2 -4 0 -5
1 -3 6
-1 -15 1 -3
-1 -15

f 2 ( x) = −5 + 2( x − 0) − 4( x − 0)( x − 1)

f2(x)= F[x0]+F[x0,x1] (x-x0)+F[x0,x1,x2] (x-x0)(x-x1)

۱۹
Two Examples
Obtain the interpolating polynomials for the two examples:

x y x y

1 0 2 3

2 3 1 0

3 8 3 8

What do you observe?

۲۰
Two Examples
x Y
x Y
1 0 3 1 2 3 3 1
2 3 5 1 0 4
3 8 3 8

P2 ( x) = 0 + 3( x − 1) + 1( x − 1)( x − 2) P2 ( x) = 3 + 3( x − 2) + 1( x − 2)( x − 1)
= x −1
2 = x2 −1

Ordering the points should not affect the interpolating polynomial.

۲۱
Properties of Divided Difference
Ordering the points should not affect the divided difference:

f [ x0 , x1 , x2 ] = f [ x1 , x2 , x0 ] = f [ x2 , x1 , x0 ]

۲۲
Example
 Find a polynomial to x f(x)
interpolate the data.
2 3

4 5

5 1

6 6

7 9

۲۳
Example
x f(x) f[ , ] f[ , , ] f[ , , , ] f[ , , , , ]
2 3 1 -1.6667 1.5417 -0.6750
4 5 -4 4.5 -1.8333
5 1 5 -1
6 6 3
7 9
f 4 = 3 + 1( x − 2) − 1.6667( x − 2)( x − 4) + 1.5417( x − 2)( x − 4)( x − 5)
− 0.6750( x − 2)( x − 4)( x − 5)( x − 6)

۲٤
 Summary
Interpolating Condition : f ( xi ) = f n ( xi ) for i = 0, 1, 2, ..., n
* The interpolating Polynomial is unique.
* Different methods can be used to obtain it
- Newton Divided Difference
- Lagrange Interpolation
- Other methods

Ordering the points should not affect the interpolating polynomial.

۲٥
Lagrange Interpolation

۲٦
The Interpolation Problem
Given a set of n+1 points:

(x0 , f ( x0 ) ), (x1, f ( x1 ) ), ...., (xn , f ( xn ) )

Find an nth order polynomial: f n (x )
that passes through all points, such that:

f n ( xi ) = f ( xi ) for i = 0,1, 2,..., n

۲۷
Lagrange Interpolation
Problem: xi x0 x1 …. xn
Given
yi y0 y1 …. yn

Find the polynomial of least order f n (x) such that:

f n ( xi ) = f ( xi ) for i = 0,1,..., n
n
Lagrange Interpolation Formula: f n ( x) = ∑ f ( xi )  i ( x)
i =0

(x − x )
n
 i ( x) = ∏ (x − x )
j =0, j ≠i
j

i j
۲۸
Lagrange Interpolation

 i ( x) are called the cardinals.

The cardinals are n th order polynomials :
0 i ≠ j
 i (x j ) = 
1 i = j

۲۹
Lagrange Interpolation Example
P2 ( x ) = f ( x0 ) 0 ( x ) + f ( x1 ) 1 ( x ) + f ( x2 ) 2 ( x) x 1/3 1/4 1

 0 ( x) =
( x − x1 ) (x − x2 )
=
( x − 1 / 4 ) (x − 1) y 2 -1 7
(x0 − x1 ) (x0 − x2 ) (1 / 3 − 1 / 4) (1 / 3 − 1)
 1 ( x) =
(x − x0 ) (x − x2 ) = (x − 1 / 3) (x − 1)
(x1 − x0 ) (x1 − x2 ) (1 / 4 − 1 / 3) (1 / 4 − 1)
 2 ( x) =
(x − x0 ) (x − x1 ) = (x − 1 / 3) (x − 1 / 4)
(x2 − x0 ) (x2 − x1 ) (1 − 1 / 3) (1 − 1 / 4)
P2 ( x) = 2{− 18( x − 1 / 4)( x − 1)}− 1{16( x − 1 / 3)( x − 1)}
+ 7{2( x − 1 / 3)( x − 1 / 4)}

۳۰
Example
Find a polynomial to interpolate: x y

0 1
Both Newton’s interpolation
method and Lagrange 1 3
interpolation method must
give the same answer. 2 2

3 5

4 4

۳۱
Newton’s Interpolation Method
0 1 2 -3/2 7/6 -5/8

1 3 -1 2 -4/3

2 2 3 -2

3 5 -1

4 4

۳۲
Interpolating Polynomial
3 7
f 4 ( x ) = 1 + 2( x ) − x ( x − 1) + x ( x − 1)( x − 2)
2 6
5
− x ( x − 1)( x − 2)( x − 3)
8

115 95 2 59 3 5 4
f4 ( x) = 1 + x− x + x − x
12 8 12 8

۳۳
Interpolating Polynomial Using
Lagrange Interpolation Method

∑
4

f 4 ( x) = f ( xi )  i =  0 + 31 + 2  2 + 5 3 + 4  4
i =0
( x − 1) ( x − 2) ( x − 3) ( x − 4) ( x − 1)( x − 2)( x − 3)( x − 4)
0 = =
(0 − 1) (0 − 2) (0 − 3) ( 0 − 4) 24
( x − 0) ( x − 2 ) ( x − 3) ( x − 4) x( x − 2)( x − 3)( x − 4)
1 = =
(1 − 0) (1 − 2) (1 − 3) (1 − 4) −6
( x − 0) ( x − 1) ( x − 3) ( x − 4) x( x − 1)( x − 3)( x − 4)
2 = =
( 2 − 0) ( 2 − 1) ( 2 − 3) ( 2 − 4) 4
( x − 0) ( x − 1) ( x − 2) ( x − 4) x( x − 1)( x − 2)( x − 4)
3 = =
(3 − 0) (3 − 1) ( 3 − 2) ( 3 − 4) −6
( x − 0) ( x − 1) ( x − 2) ( x − 3) x ( x − 1)( x − 2)( x − 3)
4 = =
( 4 − 0) ( 4 − 1) ( 4 − 2) ( 4 − 3) 24
۳٤
 Inverse Interpolation
Error in Polynomial Interpolation

۳٥
Inverse Interpolation
Problem : Given a table of values
Find x such that : f ( x ) = yk , where yk is given

xi x0 x1 …. xn
yi y0 y1 …. yn

One approach:
Use polynomial interpolation to obtain f n (x) to interpolate the
data then use Newton’s method to find a solution to x

f n ( x ) = yk
۳٦
Inverse Interpolation
Inverse interpolation: xi x0 x1 …. xn
1. Exchange the roles
yi y0 y1 …. yn
of x and y.

2. Perform polynomial
Interpolation on the yi y0 y1 …. yn
new table.
xi x0 x1 …. xn
3. Evaluate

x = f n ( yk )

۳۷
Inverse Interpolation

x
y

x y

۳۸
Inverse Interpolation
Question:

What is the limitation of inverse interpolation?

• The original function has an inverse.

• y1, y2, …, yn must be distinct.

۳۹
Inverse Interpolation
Example

x 1 2 3
Problem : y 3.2 2.0 1.6

Given the table. Find x such that f ( x ) = 2.5

3.2 1 -.8333 1.0417
2.0 2 -2.5
1.6 3

x = f 2 ( y ) = 1 − 0.8333( y − 3.2) + 1.0417( y − 3.2)( y − 2)

x = f 2 ( 2.5) = 1 − 0.8333( −0.7) + 1.0417( −0.7)(0.5) = 1.2187
٤۰
Errors in polynomial Interpolation
 Polynomial interpolation may lead to large
errors (especially for high order polynomials).

BE CAREFUL

 When an nth order interpolating polynomial is

used, the error is related to the (n+1)th order
derivative.

٤۱
10th Order Polynomial Interpolation
2

1.5 10 th order interpolating polynomial

0.5

true function

-0.5
-5 -4 -3 -2 -1 0 1 2 3 4 5

٤۲
Summary
 The interpolating polynomial is unique.
 Different methods can be used to obtain it.
 Newton’s divided difference
 Lagrange interpolation
 Others
 Polynomial interpolation can be sensitive to
data.
 BE CAREFUL when high order polynomials
are used.

٤۳
 University of Technology
Petroleum Technology Department

Numerical Analysis

Lecture 10 : Numerical Differentiation

Semester 1 (2020-2021)

Doaa Saleh Mahdi

۱
Outlines
 First order derivatives
 High order derivatives
 Richardson Extrapolation
 Examples

۲
Motivation
 How do you evaluate the Time Displacement

derivative of a tabulated (second) (meters)

function. 0 30.1

 How do we determine the 5 48.2

velocity and acceleration
from tabulated 10 50.0

measurements.
15 40.2

۳
Recall
df f ( x + h) − f ( x )
= lim
dx h→0 h

Taylor Theorem :
f ( 2 ) ( x ) h 2 f ( 3) ( x ) h 3
f ( x + h) = f ( x ) + f ' ( x ) h + + + O(h 4 )
2! 3!
E = O(h ) ⇒ ∃ real , finite C , such that : E ≤ C h
n n

E is of order h n ⇒ E is approaching zero at rate similar to h n

٤
Three Formula
df ( x) f ( x + h) − f ( x)
Forward Difference : =
dx h

df ( x) f ( x) − f ( x − h)
Backward Difference : =
dx h

df ( x) f ( x + h) − f ( x − h)
Central Difference : =
dx 2h

Which method is better? How do we judge them?

٥
The Three Formulas

٦
Forward/Backward Difference Formula
Forward Difference : f ( x + h) = f ( x ) + f ' ( x ) h + O ( h 2 )
⇒ f ' ( x ) h = f ( x + h) − f ( x ) + O ( h 2 )
f ( x + h) − f ( x )
⇒ f ' ( x) = + O ( h)
h
_______________________________________________________
Backward Difference : f ( x − h) = f ( x ) − f ' ( x ) h + O ( h 2 )
⇒ f ' ( x ) h = f ( x ) − f ( x − h) + O ( h 2 )
f ( x ) − f ( x − h)
⇒ f ' ( x) = + O ( h)
h

۷
Central Difference Formula
Central Difference :
f ( 2 ) ( x) h 2 f ( 3) ( x) h 3 f ( 4 ) ( x ) h 4
f ( x + h) = f ( x ) + f ' ( x ) h + + + + ...
2! 3! 4!
f ( 2 ) ( x ) h 2 f ( 3) ( x ) h 3 f ( 4 ) ( x ) h 4
f ( x − h) = f ( x ) − f ' ( x ) h + − + + ...
2! 3! 4!
f ( 3) ( x ) h 3
f ( x + h) − f ( x − h) = 2 f ' ( x ) h + 2 + ...
3!
f ( x + h) − f ( x − h)
⇒ f ' ( x) = + O(h 2 )
2h

۸
The Three Formula (Revisited)
df ( x) f ( x + h) − f ( x)
Forward Difference : = + O ( h)
dx h

df ( x) f ( x) − f ( x − h)
Backward Difference: = + O ( h)
dx h

df ( x) f ( x + h) − f ( x − h)
Central Difference : = + O(h 2 )
dx 2h

Forward and backward difference formulas are comparable in accuracy.

Central difference formula is expected to give a better answer.

۹
Higher Order Formulas
f ( 2) ( x)h 2 f (3) ( x)h 3 f ( 4) ( x)h 4
f ( x + h) = f ( x ) + f ' ( x ) h + + + + ...
2! 3! 4!
f ( 2) ( x)h 2 f (3) ( x)h 3 f ( 4) ( x)h 4
f ( x − h) = f ( x ) − f ' ( x ) h + − + + ...
2! 3! 4!
f ( 2) ( x ) h 2 f ( 4) ( x ) h 4
f ( x + h) + f ( x − h) = 2 f ( x ) + 2 +2 + ...
2! 4!
f ( x + h) − 2 f ( x ) + f ( x − h)
⇒ f ( 2) ( x ) = 2
+ O ( h 2
)
h

f ( 4) (ξ )h 2
Error = −
12
۱۰
Other Higher Order Formulas
f ( x + h) − 2 f ( x) + f ( x − h)
f ( 2) ( x ) =
h2
f ( x + 2h ) − 2 f ( x + h ) + 2 f ( x − h ) − f ( x − 2h )
f (3) ( x ) =
2h 3
f ( x + 2h ) − 4 f ( x + h ) + 6 f ( x ) − 4 f ( x − h ) + f ( x − 2h )
f ( 4) ( x ) =
h4

Central Formulas with Error = O ( h 2 )

Other formulas for f ( 2) ( x ), f (3) ( x )... are also possible.
You can use Taylor Theorem to prove them and obtain the error order.

۱۱
Example
 Use forward, backward and centered difference
approximations to estimate the first derivate of:
f(x) = –0.1x4 – 0.15x3 – 0.5x2 – 0.25x + 1.2
at x = 0.5 using step size h = 0.5 and h = 0.25
 Note that the derivate can be obtained directly:
f’(x) = –0.4x3 – 0.45x2 – 1.0x – 0.25
The true value of f’(0.5) = -0.9125
 In this example, the function and its derivate
are known. However, in general, only tabulated
data might be given.
۱۲
Solution with Step Size = 0.5
 f(0.5) = 0.925, f(0) = 1.2, f(1.0) = 0.2
 Forward Divided Difference:
f’(0.5) ≈ (0.2 – 0.925)/0.5 = -1.45
|εt| = |(-0.9125+1.45)/-0.9125| = 58.9%
 Backward Divided Difference:
f’(0.5) ≈ (0.925 – 1.2)/0.5 = -0.55
|εt| = |(-0.9125+0.55)/-0.9125| = 39.7%
 Centered Divided Difference:
f’(0.5) ≈ (0.2 – 1.2)/1.0 = -1.0
|εt| = |(-0.9125+1.0)/-0.9125| = 9.6%
۱۳
Solution with Step Size = 0.25
 f(0.5)=0.925, f(0.25)=1.1035, f(0.75)=0.6363
 Forward Divided Difference:
f’(0.5) ≈ (0.6363 – 0.925)/0.25 = -1.155
|εt| = |(-0.9125+1.155)/-0.9125| = 26.5%
 Backward Divided Difference:
f’(0.5) ≈ (0.925 – 1.1035)/0.25 = -0.714
|εt| = |(-0.9125+0.714)/-0.9125| = 21.7%
 Centered Divided Difference:
f’(0.5) ≈ (0.6363 – 1.1035)/0.5 = -0.934
|εt| = |(-0.9125+0.934)/-0.9125| = 2.4%
۱٤
Discussion
 For both the Forward and Backward difference,
the error is O(h)
 Halving the step size h approximately halves the
error of the Forward and Backward differences
 The Centered difference approximation is more
accurate than the Forward and Backward
differences because the error is O(h2)
 Halving the step size h approximately quarters
the error of the Centered difference.

۱٥
Richardson Extrapolation
f ( x + h) − f ( x − h)
Central Difference : f ' ( x) = + O(h 2 )
2h

Can we get a better formula?

Hold f ( x) and x fixed :

f ( x + h) − f ( x − h)
φ ( h) =
2h
φ (h) = f ' ( x) − a2 h 2 − a4 h 4 − a6 h 6 − ...

۱٦
Richardson Extrapolation
Hold f ( x ) and x fixed :
f ( x + h) − f ( x − h)
φ (h) =
2h
φ ( h ) = f ' ( x ) − a2h 2 − a4h 4 − a6h 6 − ...
2 4 6
h h h h
φ ( ) = f ' ( x ) − a2   − a4   − a6   − ...
2 2 2 2
Use two derivate
h 3 4 15
φ ( h ) − 4φ ( ) = −3 f ' ( x ) − a4h − a6h 6 − ... estimates to
2 4 16 compute a third,
4 1
⇒ f ' ( x ) = φ ( h / 2) − φ ( h ) + O ( h 4 ) more accurate
3 3 approximation

۱۷
Richardson Extrapolation Example
 Use the function:

f(x) = –0.1x4 – 0.15x3 – 0.5x2 – 0.25x + 1.2

Starting with h1 = 0.5 and h2 = 0.25, compute

an improved estimate of f’(0.5) using
Richardson Extrapolation

 Recall the true value of f’(0.5) = -0.9125

۱۸
Solution
 The first-derivative estimates can be computed
with centered differences as:
f (0.5 + h ) − f (0.5 − h )
φ (h) = at x = 0.5
2h
f (1) − f (0) 0.2 − 1.2
φ (0.5) = = = −1.0, | ε t | = 9.6%
1 1
f (0.75) − f (0.25)
φ (0.25) = = −0.934375, | ε t | = 2.4%
0.5
The improved estimate can be obtained by applying :
4 1 4 1
f ' (0.5) ≅ φ ( h / 2) − φ ( h ) = ( −0.934375) − ( −1) = −0.9125
3 3 3 3
Which produces the exact result for this example
۱۹
Richardson Extrapolation Table
D(0,0)=Φ(h)

D(1,0)=Φ(h/2) D(1,1)

D(2,0)=Φ(h/4) D(2,1) D(2,2)

D(3,0)=Φ(h/8) D(3,1) D(3,2) D(3,3)

۲۰
Richardson Extrapolation Table

 h 
First Column : D ( n,0) = φ  n 
2 

Others :
4m 1
D ( n, m ) = m D ( n, m − 1) − m D( n − 1, m − 1)
4 −1 4 −1

۲۱
Example
Evaluate numerically the derivative of :
f(x) = x cos(x) at x = 0.6

Use Richardson Extrapolation with h = 0.1

Obtain D(2,2) as the estimate of the derivative .

۲۲
Example
First Column
f(x + h) - f(x - h)
Φ (h) =
2h
f(0.7) - f(0.5)
Φ (0.1) = = 1.08483
0.2
f(0.65) - f(0.55)
Φ (0.05) = = 1.08988
0.1
f(0.625) - f(0.575)
Φ (0.025) = = 1.09115
0.05
۲۳
Example
Richardson Table
D (0,0) = 1.08483 , D(1,0) = 1.08988, D(2,0) = 1.09115

4m 1
D ( n, m ) = m D( n, m − 1) − m D( n − 1, m − 1)
4 −1 4 −1
4 1
D(1,1) = D(1,0) − D(0 ,0) = 1.09156
3 3
4 1
D( 2 ,1) = D( 2 ,0) − D (1,0) = 1.09157
3 3
16 1
D ( 2 ,2) = D ( 2 ,1) − D (1,1) = 1.09157
15 15
۲٤
Example
Richardson Table

1.08483 This is the best

1.08988 1.09156 estimate of the
derivative of the
1.09115 1.09157 1.09157 function.

All entries of the Richardson table are estimates of the

derivative of the function.
The first column are estimates using the central
difference formula with different step size h.

۲٥