Systems of Non-Linear Equations

Examples of Nonlinear Equations

Characteristic equation in process control

1 4s 16s 2 3s 3 7s 4 0

polynomial

Van der Waals eq.

2

n V
P
a
b

RT

V n

complex expression

Friction factor f for the turbulent flow in a pipe

e
1
9.35
1.14 2.0 log
f
D Re f

Implicit function

Root Finding: Graphic Method

f ( x) 0

Unique solution
f(x)

Many solutions

No solution

f(x)

f(x)

x 0

Bisection Method
a
xl

xm

xu

root

If

f ( xl ) f ( xu ) 0

Then, there is at least one real root between xu and xl

Choose a mid-point xm and check

f ( xl ) f ( xm ) 0 ?
f ( xu ) f ( xm ) 0 ?

or

Example
xm xmold
Error
100
xm

f ( x) e x 0

Solve
Iteration xl

xu

f(xl)

f(xu)

xm

f(xm)

Error

0.632

0.5

0.1065

---

0.5

0.1065

0.632

0.75

0.2776 33%

0.5

0.75

0.1065

0.2776 0.625

0.0896 20.0%

0.5

0.625

0.1065

0.0896 0.5625

0.00728

0.5625 0.625

(the root is: 0.56714329)

11.1%

0.00728 0.0896 0.59375 0.0415 5.3%

Maximum Possible Error

We note that the initial interval is

xu xl 0

After n iterations, the new interval becomes

xu xl 0
2

xu xl 0

maximum possible error

2n

Rate of Convergence
k+1 iteration

k iteration

a xk 1 C a xk

root
For convergence to occur:

C 1

C : rate of convergence. For bisection method, C = 0.5

p = 1 : converge linearly
p = 2 : converge quadratically

More Than One Root

f (x)

roots

root
xu

xl

f (x)

f ( xl ) f ( xu ) 0

roots
xl

xu

f ( xl ) f ( xu ) 0
x

multiple roots

f (x)

xl

xu

f ( xl ) f ( xu ) 0
x

Singular Point
1

1
f x ln 2 1
x
x

f (x)

xl
xu

f ( xl ) f ( xu ) 0

Bisection method cannot recognize the difference

between a root and a singularity point

Successive Substitution Method

If we can rearrange f(x) = 0 as x = g(x)
The iterative scheme can be written as xn1 g xn
iteration function
Select an arbitrary value x0 and
calculate g(x0)
Let x1 = g(x0)
Calculate g(x1)

When xn+1 = g(xn)

iteration xn1 g xn

f(x) = 0 and xn = a

How Does It Work?

y=x

y
xn+1 = g(xn)

xnx2

x1

y = g(x)

x0
initial guess

Example
e.g.

f x x 3 0
2

(a) x x 3 x

xn1 xn2 3 xn

(b) x 3 / x

xn1 3 / xn

(c) x 0.5x 3 / x

xn1 0.5 x 3 / xn

Results
f x x 2 3 0 root 3 1.732051
xn1 3 / xn

xn1 0.5 x 3 / xn

Iteration # n

xn1 xn2 3 xn

2.0

2.0

2.0

3.0

1.5

1.75

9.0

2.0

1.732143

87.0

1.5

1.732051

Diverging

Oscillating

Converging

Not every g(x) can give us a solution. How to select the best g(x) ?

Conditions of Convergence
0 g x 1

1 g x

y=x

y=g(x)
y=x

y=g(x)

x0

1 g x 0

g x 1

y=x

y=x

x0

y=g(x)
x0

x0

y=g(x)
x

Conditions of Convergence
2

xn a
g xn g a xn a g a
g a ...

g xn xn 1

iteration function

g a a

xn a
xn1 a xn a g a
g a ...

en2
en1 g a en g a ...
2

for en1 en

g a 1

is required

en1
a xn1

g a

lim
lim
n en
n a xn

f x x 2 3 0 root a 3

g xn xn2 3 xn
g xn 2 xn 1 0
g a 1

diverging

g xn 3 / xn
g xn 3 / xn2

oscillating

g a 1

g x 0.5 x 3 / xn

g xn 0.5 1 3 / xn2
g a 0

converging

What is the Best Way to Find g(x)?

g x x f x
g x 1 f x
Condition for convergence requires

1 g x 1
0 f x 2
1. must have the same sign as f (x)
2. Optimal value of is obtained when

f x

Example
f x x 2 3 0

g ( x) x f x x x 2 3
Iterative scheme:
Find :
let

xn1 xn xn2 3

0 f x 2

f x 1

If x0 = 2 1 4

1
2xn 1
2 xn

f x x 2 3 0

1 2
xn 1 xn xn 3
4
n

xn

a xn

a xn+1/a xn

2.0

2.68 10-1

0.0668

1.75

1.79 10-2

0.130

1.734375

2.32 10-3

0.134

1.732361

3.10 10-4

0.134

1.7320923

4.15 10-5

0.134

1.7320564

5.56 10-6

0.134

1.7320509

1.00 10-7

0.134

a xn1
en1

lim
g a
lim
n en
n a xn

1
g a 1 21.732 0.134
4

linear convergence

Newton-Raphson Method
Taylor series expansion of f(x) around x0
2

x x0
f x f x0 x x0 f x0

2!

f x0 ...

If x1 is the root, then f(x1) = 0

2

x1 x0
f x1 f x0 x1 x0 f x0

2!

f x0 ... 0

f x0 x1 x0 f x0 0
x1 x0

f x0
f x0

Similar to the successive substitution

1
method when f x

Newton-Raphson Method
f(x)

f(x)

a
x2 x1
x2 x1

f x1
f x1

x0
x1 x0

x
f x0
f x0

Example
f x 3x sin x e x
f x 3 cos x e x

Start with x0 = 0
f x0
1
0
0.3333333
x1 x0

3.0
f x0

0.068418
f x1
0.3333333
0.36017
x2 x1

2.54934
f x1

6.279 10 4
f x2
0.36017
0.3604217
x3 x2
2.50226
f x2

Exact solution is 0.360421703

Convergence of the NewtonRaphson Method

Taylor series expansion of f(x) around xn
2

x xn
f x f xn x xn f xn

2!

If a is a root

a xn 2
f a f xn a xn f xn
2!

f 0

2
f xn a xn f

a xn
2!
f xn
f xn

xn 1

a xn1

a xn 2

2!

f
f xn

a xn1 M a xn
en1 M en

Sufficient Condition for

Convergence
a xn1 M a xn

M a xn1 M a xn

M a xn1 M a x0

2n

a xn1

1
2n
M a x0

Sufficient condition for the Newton-Raphson method to converge

to a is
M a x0 1

Pitfalls of the Newton-Raphsons

Method
f x2 0

f(x)

x1

x0

x2
x

The success of the Newton-Raphson method depends on the initial guess

Oscillation of the Solution

f(x)

x1

x0
x

There is no root between x0 and x1, but the Newton-Raphson

method oscillates between these two points

Newton-Raphson Method for

Multiple Roots
f (x)

double root

f ( x) x 3x 1

triple root

f ( x) x 3x 1

f (x)

f (x)

quadruple root

f ( x) x 3x 1

Convergence of Newton-Raphson
Method for Multiple Roots
f ( x) x a q x
m

Newton-Raphson method:
xn1 g xn xn

f xn
f xn

x a q x
x a qx
f x
g x x
x

x a qx mqx
f x
x a m qx mx a m1 qx
m

g x 1

d
q x
qx
x a
x a qx mqx
dx x a qx mqx

g a 1

1
m

Convergence of Newton-Raphson
Method for Multiple Roots
xn1 a g xn g a
2

xn a
g xn g a xn a g a
g

2!

xn1 a xn a g a

xn a

2!
m 1
xn1 a xn a g a
en
m

0
g

g a 1

For m = 1

1
m

xn a
xn1 a xn a g a
g

2!

xn 1

xn a
g a Me 2
a

2!

Over-Relaxation Method
xn 1 xn

f xn
f xn

How to choose a to make g(a) = 0 ?

g a 1

g a 0 when m

Example
f x x3 5.56 x 2 9.1389 x 4.6899 0

has a double root a = 1.23

f xn
xn 1 xn 2
f xn
n

xn

e (%)

1.0

18.70

1.219629012

0.84

1.229971477

2.33 101

1.22999787

1.73 104

1.230000091

7.4 106

Modified Newton-Raphson Method

f ( x) x 3x 1

Define

K ( x) f x

The new function has the same roots

as f(x)

K x
xi 1 xi
K x

This method is introduced in the textbook, but

not very common

f x
u ( x)
f x

ux
xi 1 xi
ux

f xn f xn
x n 1 xn
f xn 2 f xn f xn

This method is preferred. Why?

Example
Use standard and modified Newton-Raphson methods
2

f
(
x
)

3
x

1
to evaluate the multiple root of
xn3 5 xn2 7 xn 3
f x
xn
x n1 xn
3xn2 10 xn 7
f x

Take initial guess x0 = 0

f (x)

xn

e (%)

100

0.4285714

57

0.6857143

31

0.8328654

17

0.9133290

8.7

0.9557833

4.4

0.9776551

2.2

Linearly convergent
toward 1

Modified Method
x n 1 xn
xn

f xn f xn
f xn 2 f xn f xn

3x

3
n

2
n

5 xn2 7 xn 33xn2 10 xn 7

10 xn 7 xn3 5 xn2 7 xn 36 xn 10
2

Take initial guess x0 = 0

n

xn

e (%)

100

1.105263

11

1.003082

0.31

1.000002

0.00024

converge quadratically

Secant Method
f x

f(x)

xn-1

xn 1 xn

f xn xn1 xn
f xn 1 f xn

xn x

f xn 1 f xn
xn 1 xn

Example of Using Secant Method

Solve

f ( x) e x x 0

(recall the root is 0.56714329)

First iteration
x1 0
x0 1

f x1 1.00000

f x0 0.63212

0.632120 1
x1 1
0.61270 t 8.0%
1 0.63212

Second iteration
x0 1

f x0 0.63212

x1 0.61270

f x1 0.07081

0.070811 0.61270
x2 0.61270
0.56384 t 0.58%
0.63212 0.07081

Systems of Nonlinear Equations

Consider a two-component isomerization reaction occurring in a CSTR

kAB

kBA

if the reaction
is 1st order:
if the reaction
is 2nd order:

Vk BACA QCA Vk ABCB QCA 0

Vk BACA Vk ABCB QCB QCB0
Vk BACA Vk ABCB QCA QCA 0
2

Vk BACA Vk ABCB QCB QCB0

2

A system of nonlinear equations

Solving Nonlinear Equations

Consider the following two non-linear functions:

f x, y 0

g x, y 0

Taylor expansion
f xn , yn
f xn , yn

y yn
...
f x , y 0 f xn , y n x x n

x
y
g xn , yn
g xn , yn
g x , y 0 g xn , y n x x n
y yn
...
x
y

In a Matrix Form
f xn , yn

g xn , yn

f xn , yn
x xn
f xn , yn
y

g xn , yn y yn

g
x
,
y
n n

J: Jacobian

JDZ h
xn 1 xn Dxn
y y Dy
n 1 n n

DZ

Example
f x, y x xy 10 0
2

g x, y y 3xy 2 57 0

f xn , yn

g xn , yn

f xn , yn
x xn
f xn , yn
y

g x , y
g xn , yn y yn
n n

solution (2,3)
x
x 2 xy 10
Dxn
2 x y

Dy
3 y 2 1 6 xy
2

y
3
xy
57

xn , y n n

xn , y n

Initial guess (1.5, 3.5)

1.5 Dxn
2.5
6.5
36.75 32.5 Dy 1.625

Dxn 0.536
Dy 0.656

x1 x0 Dx0 1.5 0.536 2.036

y y Dy 3.5 0.656 2.844

1 0 0

Eigenvalues and Eigenvectors

Eigenvalues
det A I 0
a11

a12

a13

a1n

a21

a22

a23

a2 n

a31

a32

an1

an 2

ann

a33

f n n1n1 n2n2 ... 11 0 0

characteristic equation

(has n roots)

Example
Find the eigenvalues of

det A I 0

2
3
3 4

characteristic equation:

f 3 4 6 0
2 6 0

2, 3

Principal Minors
A principal minor of size j j can be obtained by striking out an equal
number of n j row and column from A
For example, to obtain a principal minor of 2 2, we can
strike out 1st row and 1st column, or 2nd row and 2nd column

a11 a12

A a21 a22
a31 a32

a13

a23
a33

principle
minor

a22

a23

a32

a33

Eigenvalues from Principal Minors

A characteristic equation can be expressed as:
n

f S j A

n j

j 0

where Sj is the sum of all principal minors of size j j

a11 a12

A a21 a22
a31 a32

a13

a23
a33

S0 = 1 (by definition)
S1 = a11 + a22 + a33 = trace(A)
S2

a11

a12

a21 a22

a11

a13

a31 a33

a22

a23

a32

a33

S3 = det(A)
For a 2 2 matrix

f 2 trA det A 0

Example
6 4 7
A 0
2 3
1 1
1

S0 S1 S2 S3 0
3

for n = 3

S0 1
S2

a11

S1 a11 a22 a33 6 2 1 9

a12

a21 a22

a11

a13

a31 a33

a22 a23
a32

S3 det A 32
f 3 92 30 32 0

a33

30

Properties of Eigenvalues
Upper bound of the eigenvalues:

Max i A
Proof

i v i Av i A v i
i v i A v i
i A

Eigenvectors
For each eigenvalue i, we can find a corresponding
eigenvector vi which fulfills the following equation

A i I vi 0
or

Av i i v i

To have a non-trivial solution for vi, det (A iI) = 0.

Therefore, rank (A iI) < rank (A) and eigenvectors have at
least one arbitrary coefficient

v i

Example
Find the eigenvalues and eigenvectors of the following matrix

2 1 0
1 3 1

0 1 2

f 3 72 14 8 0

1 1, 2 2, 3 4

When = 1: (A I)v1 = 0

1 1 0 v11
1
1 2 1 v 0 v 1
1

12

0 1 1 v13
1
rank of the matrix is 2
dimension of the null space is 1
If we let v11 =

v12 =
v13 =

1
v1 1

1
eigenvector

When = 2: (A 2I)v2 = 0

0 1 0 v21
1
1 1 1 v 0 v 0
2

22

0 1 0 v23
1
When = 4: (A 4I)v3 = 0

0 v31
2 1
1
1 1 1 v 0 v 2
3

32

0
1
1 2 v33

Eigenvectors and Adjugate

We can also obtain eigenvectors from the non-trivial columns
of adj(A I)
e.g.

1 1
A

1
1

0, 2

for = 0

1 1 1 1

adj( A 0I) adj

1
1
1
1

for = 2

1 1 1 1

adj( A 2I) adj

1
1
1

1
v1
1
1
v2
1

If repetitive eigenvalues are present, it may not be possible

to find eigenvectors from adj(A I)

Linear Independence of Eigenvectors

When a matrix A has two distinct eigenvalues, i and j, the two
corresponding eigenvectors vi and vj are linearly independent
Proof:

Assume v i kv j

i v i ki v j

Av i kAv j

if k 0

i v i k j v j

i j

Eigenvalues and Eigenvectors for

Symmetric and Asymmetric Matrices
If a matrix is symmetric
All eigenvalues are real, but they may not be
distinct
The matrix will have n distinct eigenvectors
(even if some eigenvalues are not distinct)

If a matrix is not symmetric

Eigenvalues may be complex numbers
The matrix may not have n linearly independent
eigenvectors

Example: Symmetric Matrix

1 1
A

1 1

Because A is symmetric and n = 2, we

can obtain two real eigenvalues and two
distinct eigenvectors

0, 2
eigenvectors

0 1 1 0
2

(Two real eigenvalues)

1
v1
1

1
v2
1

Two distinct eigenvectors form a complete basis set

Example: Asymmetric Matrix

3 4
A

1 1

Because A is NOT symmetric, we may have

complex eigenvalues and may not have two
distinct eigenvectors

0 1, 1

multiplicity = 2

(Two identical eigenvalues)

2
eigenvector v1
1
(Only one eigenvector)

Example: Symmetric Matrix

1 0 0
A 0 1 0
0 0 1

0 0 0

( A I ) 0 0 0
0 0 0

f 1 0 1
3

multiplicity = 3

1
0
0
v1 0 v 2 1 v 3 0
1
0
0

Although all eigenvalues are identical, we still have three

distinct eigenvectors

Orthogonality of Eigenvectors
In a n n symmetric matrix A, one can find n distinct
eigenvectors vi which are orthogonal to each other. These
eigenvectors can be normalized to give a complete
orthonormal basis set ui
Furthermore,

U u1 u 2 u3 ...u n

forms a orthonormal matrix

Properties of Orthonormal Matrix

If u1, u2, u3,.. un are n 1 orthonormal vectors
Let

U u1 u 2 u3 ...u n

u1T 1
T
1
u
2

1
U T u 3T

0
u T
n

U UI
T

U UI

0
=I

U U
T

for an orthonormal matrix

1

AU Au 1 Au 2 Au 3 ...Au n

1u1 2u 2 3u 3 ...nu n

u1T
T
u 2
U T u 3T

u T
n

1
0
0
0

UT AU D diagonal i

Example
1 1, 2 2, 3 4

2 1 0
A 1 3 1
0 1 2

1
v1 1
1

1
v 2 0
1

1
v 3 2
1

<v1 , v2> = 0, <v1 , v3> = 0, <v2 , v3> = 0

Normalize these eigenvectors:

1
1
1
1 u 1 0 u 1 2
1 2
u1
3

2
6
3
1
1
1

Example (Contd)
1

3
1
U
3
1

1
2
0
1
2

6
2
6
1

UT

1
3
1
2
1
6

1
3
0
2
6

1
3
1
2
1

orthonormal matrix

1 0 0
U T AU 0 2 0
0 0 4
U 1AU

U 1

1
3
1
2
1
6

1
3
0
2
6

1
3
1
2
1

System of Homogenous ODEs

dx1
a11x1 a12 x2 ... a1n xn x1 0 x10
dt
dx2
a21x1 a22 x2 ... a2 n xn x2 0 x20
dt

dxn
an1 x1 an 2 x2 ... ann xn
dt

dx
Ax
dt

xn 0 xn 0

initial condition: x0 x0

Homogeneous Linear ODEs

dx
Ax
dt
We can assume that the solution looks like

x ve

Therefore, we have:

vet Avet
v Av

A I v 0
All we need to do now is find eigenvalues and eigenvectors of A

General Solution
For each eigenvalue and eigenvector, we can find a homogenous
solution

x1 c1v1e1t
x 2 c2 v 2e2t

x n cn v n ent
General solution is a linear combination of all homogenous solutions

x c1v1e1t c2 v 2e2t c3 v3e3t ...

Example
dx A
2 x A xB
dt

x A 0 0

dxB
2 x A xB
dt

xB 0 3

dx
Ax
dt

k1 2
xB

xA

where

k1 1

2 1
xA
x A

xB

Find eigenvalue and eigenvector for A

For

1 0

2 1
A

2 1
1
v1
2

0,3

and

2 3

1
v2
1

Two homogenous solutions are

1 0
x1 e
2

and

1 3t
x 2 e
1

The general solution is x c1x1 c2 x 2

1 0 t
1 3t
x c1 e c2 e
2
1

Using Initial Conditions

Because the solution is
x1
1 3t
1 0 t
x c1 2e c2 1e


2

when t = 0

P = [v1,v2,v3...vn]

A matrix that contains

all eigenvectors

x1 0
1 1 1 c1
1
x 0 c1 2 c2 1 2 1 c
2


2
c1 1 1 x1 0
1

P
x0

c 2 1 x 0
2
2
1

Determine Coefficients
1 1 1
P
3 2 1
1

0
x0
3

c1
1
1
c P x 0 1

2
x1 1 e 3t
x
3t
2
e

what happens when

t 0
t ?

How Do Eigenvalues Affect the

General Solution?
x c1v1e1t c2 v 2e2t c3 v3e3t ...
If all are real and negative x decays with time
If any is real and positive x grows with time
If any is complex x is oscillatory in nature, its real part
determines whether the solution
decays or grows

Summary
To solve a system of linear, homogenous, and
first-order ODE, we only need to know
eigenvalues and eigenvectors of A
If the size of A is n n, we must have n
linearly independent eigenvectors. (Otherwise,
P does not exist)

Similar Matrices
If A and B are n n matrices, if there exists a non-singular matrix
P such that A = PBP-1. Then A and B are similar
If A and B are similar

A = PBP-1, B = P-1AP
det(A) = det(B)
rank(A) = rank(B)
A and B have the same eigenvalues and same characteristic
polynomial
A and B have different eigenvectors

Proof
f B det B I

det P A I P
det P det A I det P
det P 1AP P 1IP
1

det P 1 det P 1

det A I
f A

A and B have the same characteristic polynomial and eigenvalues

n

S1 aii trace of A

trace(A) = trace(B)

Sn det A

det(A) = det(B)

i 1

Similarity Transformation
d11 0 0

0 d 22
D

0
d
0
0
nn

P v1 , v 2 ,...v n
P-1AP = D
AP = PD

Av1, Av 2 ,...Av n d11v1, d22v 2 ,...dnnv n

Av1=d11v1
Av2=d22v2

Avn=dnnvn
If we choose vi as eigenvectors, and
d11, d22,...dnn as eigenvalues 1 2,.. n
Av1=1v1

Av2=2v2

Avn=nvn

Matrix Diagonalization
If we choose the eigenvalues of A as d11, d22,... dnn,
and eigenvector of A as v1, v2,....vn

Let P = [v1, v2,...vn], then

1 0 0 0

0
0
0
2
D
P 1AP
0 0 0

0 0 0 n
1
A PDP

Example
From the previous example:

2 1
A

2 1

1 1
P

2 1

0,3

1 1 1
P
3 2 1

0 0
P AP
D

0 2
1

or

A PDP

Solving System of Homogenous ODEs

dx
Ax
dt
We can get

let

A PDP1
x Pz

dz
PDP 1 (Pz )
P
dt

dz
Dz
dt

Because D is a diagonal matrix, the ODEs above can be solved easily

Then, we can get

x Pz

Example
2 x1
x1 4
Solve
x

x
4
10

2
2

1
x0
0

1
1
1 6, v1 2 8, v 2
1
2

6 0
1 1
,D
P

0
8
1
2

z1 6 0 z1
z 0 8 z
2
2

y Px

2
z0 P x0
1

z1 c1e6t
z 8t
2 c2e

c1 2
c 1
2

 x1
1 1 2e6t
x Pz 1 2 8t

e
2
or

x1
1 6t 1 8t
x 2 1e 2e

2
x1 2e6t e8t
x2 2e6t 2e8t

1t

x c1v1e c2 v 2e

2t

System of Nonhomogenous ODEs

dx1
a11x1 a12 x2 ... a1n xn r1 t x1 0 x10
dt
dx2
a21x1 a22 x2 ... a2 n xn r2 t x2 0 x20
dt

dxn
an1 x1 an 2 x2 ... ann xn rn t xn 0 xn 0
dt

dx
Ax rt
dt

initial condition: x0 x0

Example
Solve

let

x1 0 8 x1 4e2t

x 2 0 x
2 2t
2

A PDP1
x Pz

dz
Dz P 1rt
dt

dz
P PDP1 (Pz ) r t
dt
eigenvalues = 4, 4

2 2 1 0.25 0.5
P
P

0
.
25
0
.
5

1
1

P rt

4e 2 t

2t

0.25 0.5
0.25 0.5

e 2 t t
2t
e t

2t
4
0

t
e

dz
z 2t

dt 0 4 e t

e e t dt
e e e t dt

z1 c1e e
4t

z2 c1e

4t

4t

4t

4t

2t

4t

2t

dz1
4 z1 e 2t t
dt
dz2
4 z2 e 2t t
dt

x Pz

General Solution for

st
1

Order ODE

dz
z r t
dt

z t ce t et e t r t dt
e.g.

z1 2 z1 exp 2t
t

e 2t e 2t e 2t dt te 2t
0

z1 c1e2t te 2t

Algebraic and Geometric Multiplicity

Algebraic multiplicity (mi) = the multiplicity of eigenvalue i
Geometric multiplicity (gi) = the maximum number of linearly
independent eigenvectors associated
with each eigenvalue i

For a n n matrix, if (gi) = n then P-1AP = D

or

1 0 0 0
0

0
0
2

0 0 0

0
0
0

UTAU = D
(for symmetric matrices)

Jordan-Canonical Blocks (g = 1)
If A is a 6 6 matrix having the following eigenvalues (with
algebraic multiplicity and geometric multiplicity)

1 2, m1 1, g1 1; 2 3, m2 2, g 2 1; 3 4, m3 3, g3 1
We can define a Jordan-Canonical block J for matrix A:

2
0

0
J
0
0

0 0 0 0 0
3 1 0 0 0
0 3 0 0 0

0 0 4 1 0
0 0 0 4 1

0 0 0 0 4

Jordan-Canonical Blocks (g > 1)

A is a 5 5 matrix having the following eigenvalues (with
algebraic multiplicity and geometric multiplicity)

1 2, m1 4, g1 2
2 3, m2 1, g 2 1
2
0

J 0

0
0

1 0 0 0
2 0 0 0
0 2 1 0

0 0 2 0
0 0 0 3

Same eigenvalue is present in two different blocks

Jordan-Canonical Transformation
0
0
J n1 1
0
0
J n 2 2

P 1AP J 0
0
J n 3 3

.
.
.
0
0
0

0
0
0
0

.
.
0 J nr r

How to find a nonsingular matrix P that permits

such transformation?
Intuitively, we can let P = [ v1 v2 .. vi p1 p2]
eigenvectors

Example
Find J and P for the following matrix A:

1 1
A

1
3

f 2 trA det(A) 2 4 4 0

2,2

1 1
A I

1
1

2 1
J

0
2

1 1
adjA I

1 p12
P

1
p
22

Determine p12 and p22 to satisfy P 1AP J

1
v1
1

Example (Contd)
P 1AP J
AP PJ

1 1 1 p12
1 p12
1 3 1 p

1
p
22

22

p12 p22 1 2 p12

p22 1 p12

p12 3 p22 1 2 p22

p22 1 p12

2 1
0 2

let p12 = 0, p22 = 1

1 0
1 0
1 0 1 1 1 0 2 1
1
1

P
P
P AP

1
1

1
1
1
1
1
3
1
1
0
2

Example
4
3
5
A 1 0 3
1 2 1

2, 4, 4

4
3 0 18 18
7
for 2, adj A 2I adj 1 2 3 0 18
18
1 2 3 0 18
28
4
3 6
6 0
1
for 4, adj A 4I adj 1 4 3 6 6 0
1 2 3 6
6 0
(m = 2 )

1
v1 1
1

1
v 2 1
1

 2, 4, 4

We can let P = [v1 v2 p ]

A[v1 v2 p ] = [v1 v2 p ]J3()
2 0 0
Av1 v 2 p v1 v 2 p 0 4 1
0 0 4
Av 1 2 v1
Av 2 4 v 2
Ap v 2 4p

A 2I v1 0
A 4I v 2 0
A 4I p v 2

4
3 p1 1
1
1 4 3 p 1

2
1 2 3 p3 1

p1 1
p 0
2
p3 0

1
1
v1 1 v 2 1

1
1

General Formula
A I p
j

pi 1; i 1,2,..., (m j g j )

p0 v j
e.g.

1 2, m1 1, g1 1
2 1, m2 1, g 2 1
3 3, m3 3, g 3 1

p v1 v 2 v 3 p1 p 2
1 2

A 2I v1 0
A I v 2 0
A 3I v 3 0
A 3I p1 v 3
A 3I p 2 p1

2
0

J5 0

0
0

0 0 0 0
1 0 0 0
0 3 1 0

0 0 3 1
0 0 0 3

Example
dx
Ax ; x0 x 0
dt

1 1
0
A
x0

1 3
1

f 2 4 4 0 2,2

1
1 1
v1
adjA 2I

1
1 1
0
A 2I p v1 p
1

1 0
1 0
2 1
1
J
; P
; P

0
2
1
1
1
1

0
P z0
1

A PJP 1
let

x Pz
z1
2 1 z1
z 0 2 z

2
2
z2 2z2

z1 0 0
z 0 1
2

z2 c2 exp 2t

c2 1; z2 exp 2t
z1 2 z1 exp 2t

z1 c1 exp 2t t exp 2t

c1 0; z1 t exp 2t
0 te 2t te 2t

x1
1

Pz
x
1 1 2t 2t
2 t

e
te
e

Summary

Eigenvalue and eigenvectors

Symmetric and asymmetric matrices
Similarity transformation
Jordan-Canonical blocks
Solving systems of ODEs using similarity
transformation

Numerical Solution Methods for

Ordinary Differential Equations

Initial Value Problems (IVPs)

Constant-volume batch reactor
A

k1

dC A
k1C An
dt
dCB
k1C An k2CBm
dt
dCC
k 2CBm
dt

k2

at t = 0

C
C A C A0
CB CB 0
CC CC 0

First-order ODE-IVP
dy
y f t , y y y0 at t 0
dt
If f t , y f (t )
y

y0

dy f t dt

Integration can be done analytically or numerically

Forward Euler Method

y f t , y y y0 at t 0
h2
h3
yi 1 yi hyi
yi yi ...
2!
3!

Neglect higher order terms:

yi 1 yi hf ti , yi E h 2

Taylor series

y1 y0 hf t0 , y0

y2 y1 hf t1 , y1

yn yn 1 hf t n 1 , yn 1

y1 , y2 ,... yn can be calculated explicitly

Forward Euler Method

y f t , y y y0 at t 0

y1 y0 hy0

y2 y1 hy1

y0
f(t)

y1
h

y2
t1

t2

y3 y4
t3

t4

y5 y6
t5

t6

exact solution
numerical solution
t

Local Error and Global Error

h2
h3
yi 1 yi hyi
yi yi ...
2!
3!
Predominant local error =

Taylor series

h2
yi h 2
2!

Suppose we have m segments

h 2 mh2

b a h

yi
yi
yi h
global error =
2!
2!
j 1 2!

Example
y 20 y 7 exp 0.5t y0 5
Take h = 0.01

t0 = 0

y0 = 5

t1= 0.01 y1 = y0 + h f(t0, y0)

= 5 + (0.01)[(20)(5)+7exp(0.50)]
= 4.07
t2= 0.02 y2 = y1 + h f(t1, y1)
= 4.07 + (0.01)[(20)(4.07)+7exp(0.50.01)
= 3.32565

Higher-order ODE-IVPs
Consider

y f t , y, y y0 y0 , y0 0
h2
h3
yi 1 yi hyi
yi yi ...
2!
3!
y1 y0 hy0

y2 y1 hy1 this term is unknown

y0
f(t)

y1
h
0

y2
t1

t2

t3

t4

t5

t6

Decomposition of Higher Order ODEs

y yy y 2 2t 2

y0 2, y0 0

If we define a new variable

z y

z y

z0 y0

y z

y0 2

z yz y 2 2t 2

z0 0

Higher order IVPs can be decomposed into a set of first-order

ODEs

Euler Method for Solving Higher

Order ODEs
y f t , y, z y y0 at t t0
z g t , y, z z z0 at t t0

yi 1 yi hyi yi hf ti , yi , zi
zi 1 zi hzi zi hg ti , yi , zi
y1 y0 hf t0 , y0 , z0

z1 z0 hg t0 , y0 , z0

y2 y1 hf t1 , y1 , z1

z2 z1 hg t1 , y1 , z1

Example
y 0.05 y 0.15 y 0
y z

y0 1 , y0 0

y0 1

z 0.05z 0.15 y

z0 0

Take h = 0.5
t1 = 0.5 y1 = y0 + hy'0 = 1 + (0.5) 0 = 1
z1 = z0 + hz'0 = 0 + (0.5)[(0.05)(0) (0.15)(1)] = 0.075
t2 = 1.0 y2 = y1 + hy'1 = 1 + (0.5) (0.075) = 0.9625
z2 = z1 + hz'1 = 0.075 + 0.5[0.05 (0.075) 0.15 1]
= 0.15187

Note on Euler Method

Magnitude of local error is proportional to h2
Global error is proportional to h
Accuracy increases with a decrease in step size h (but
computation time also increases!)
How to determine appropriate step size h for accurate
solutions?

Error Analysis

Total error

Truncation error
Round-off error
Error due to numerical instability

truncation
error

round-off
error
hoptimum

Numerical Instability
numerically
unstable solution
y(t)

numerically
stable solution

exact solution
t
To avoid numerical instability Use a step size sufficiently small

How to Achieve Numerical Stability?

y y,
Exact solution
From numerical
solution

y0 y0

ye y0 exp t
y ye
d dy dye

y ye y ye
dt dt dt

If Euler's method
is used:

n1 n hf tn , n n h n 1 h n
n1
1
n

1 h 1

0 h 2

Backward Euler Method

y f t , y y y0 at t 0
h2
h3
yi 1 yi hyi1
yi yi ...
2!
3!

yi 1 yi hf ti 1 , yi 1 E h 2

backward Euler method

Implicit equations
Local error is the same as forward Euler method

Example
y t y,

y0 1

yi 1 yi hf ti 1 , yi 1 yi h ti 1 yi 1

1 ti 1 h yi 1 yi
1
yi 1
yi
1 ti 1 h

take h = 0.5
t0 = 0
y0 = 1
t1 = 0.5
y1 = 1/(1+0.5 0.5) 1 = 0.8
t2 = 1.0
y2 = 1/(1+1 0.5) 0.8 = 0.533

Error Analysis for Backward

Euler Method
y y,

y0 y0

y ye

i 1 i hf ti 1 , i 1 i h i 1
1 h i 1 i
i 1
1

i 1 h
1
1
1 h

criteria for numerical stability

y y 2 ,

y0 1

Forward Euler Method

y y 2 ,

y0 1

Backward Euler Method

Numerical Stability for ODEs

y1 f y1 , y2 y1 0 y10
y2 g y1 , y2 y2 0 y20
f
y
J 1
g
y1

Define a Jacobian matrix

for

y1 2 y1 3 y2
y2 5 y1 4 y2

Characteristic eq. of J:

f
y2

g
y2

Jacobian matrix

2 3
J

f
g f g g f
0

y1 y2 y1 y2 y1 y2
2

Estimate Maximum Step Size

hmax

Maximum allowable step size

max

Example:
1 y1 0
y1 0
y 200 102 y 5
2
2
f
y
J 1
g
y1

f
1
y2 0

g 200 102
y2

det

2,100

y10 1
y 2
20

200 102
hmax

0.02
100

Stiffness
When eigenvalues have different order of magnitude
y1 500.5 499.5 y1
y 499.5 500.5 y
2
2
2 1001 1000 0

1,1000

Analytical solution:
y1 1.5e t
y
t
1
.
5
e
2

y10 2
y 1
20

0.5e 1000t

0.5e 1000t

 1,1000

y1 1.5e t
y
t
1
.
5
e
2

0.5e 1000t

0.5e 1000t

region controlled by = 1

Maximum allowable step size

hmax

0.002
1000

Stiffness Ratio
SR

max
min

When SR < 100

When 100 < SR < 100,000
SR > 100,000

Not stiff
Classified as stiff
Very stiff

Summary

Explicit and implicit Euler method

Numerical stability
Major drawback is of Euler method is low accuracy
To increase accuracy, we may reduce the step size h,
but that also introduces round-off error and increase
computation time
Stiffness

2nd Order Runge-Kutta Method

y f t , y y y0 at t 0

yi 1 yi

ti1

f t, y dt
ti

If forward Euler method is used

ti 1

predicted yi 1

f t , y dt hyi

ti

yi 1 yi hyi

yi 1

hyi

yi
h

If trapezoidal rule is employed, then

ti1

ti

h
f t , y dt f ti , yi f ti 1 , yi 1
2
unknown

K1 hf ti , yi

K 2 hf ti h, yi K1

1
yi 1 yi K1 K 2
2
The global error is proportional to h2

yi K1

K1
yi
2

yi

yi

yi 1
h2 h2

K1 K 2
2

Example
y 0.4 y 0.2

y 0 at t 0

Take h = 0.1
t 0.1

K1 0.1 0.40 0.2 0.02

K 2 0.1 0.40 0.02 0.2 0.0192

1
y1 0 0.02 0.0192 0.0196
2
t 0.2

K1 0.1 0.40.0196 0.2 0.019216

K 2 0.1 0.40.0196 0.019216 0.2 0.018447

1
y2 0.0196 0.019216 0.018447 0.038431
2

4th Order Runge-Kutta Method (RK4)

If Simpsons 1/3 rule is employed, then
ti1

ti

h
f t , y dt f ti , yi 4 f ti 1 2 , yi 1 2 f ti 1 , yi 1
6
Why is the coefficient 1/6?
If we let

unknown

K1 hf ti , yi

K 2 hf ti h / 2, yi K1 / 2

K 3 hf ti h / 2, yi K 2 / 2
K 4 hf ti h, yi K 3

1
yi 1 yi K1 2 K 2 2 K 3 K 4
6
The global error is proportional to h4

Error Analysis for RK4

y y

Lets consider this very simple ODE

yi 1 yi exp h

exact solution

Numerical solution by RK4:

K1 hf ti , yi hyi

K 2 hf ti h / 2, yi K1 / 2 h 1 h / 2 yi

K 3 hf ti h / 2, yi K 2 / 2 h 1 h / 21 h / 2yi

K 4 hf ti h, yi K 3 h 1 h 1 h / 21 h / 2yi

h 2 h 3 h 4
1

yi 1 yi K1 2 K 2 2 K 3 K 4 1 h
yi
6
2
3!
4!

exp h O h5

Example
y y e t

y0 4

take h = 0.1

0.28988
0.1 3.85 e
0.29038
0.1 3.855 e
0.1 3.70962 e 0.28048

K1 0.1 4 e 0 0.3
K2
K3
K4

0.05

0.05

0.1

1
yi 1 yi K1 2 K 2 2 K 3 K 4 3.70983
6

Solving ODEs with RK4 Method

y f y, z y 0 y0
z g y, z z 0 z0
1
yi 1 yi K1 2 K 2 2 K 3 K 4
6
1
zi 1 zi L1 2 L2 2 L3 L4
6
K1 hf ti , yi , zi

K 2 hf ti h / 2, yi K1 / 2, zi L1 / 2

K 3 hf ti h / 2, yi K 2 / 2, zi L2 / 2
K 4 hf ti h, yi K 3 , zi L3

L1 hg ti , yi , zi

L2 hg ti h / 2, yi K1 / 2, zi L1 / 2

L3 hg ti h / 2, yi K 2 / 2, zi L2 / 2
L4 hg ti h, yi K 3 , zi L3

Example
y z
z 2tz 8 y

y(0) 12
z 0 0

K1 0.1 0 0

L1 0.1 2 0 0 8 12 9.6

take h = 0.1

ti 0, yi 12, zi 0

K 2 0.1 4.8 0.48

L2 0.1 2 0.05 4.8 8 12 9.648

K 3 0.1 4.824 0.4824

L3 0.1 2 0.05 4.824 8 11.76 9.45624

K 4 0.1 9.45624 0.945624

L4 0.1 2 0.1 9.45624 8 11.5176 9.403205

y0.1 12 1 6 0 2 0.4824 0.945624 11.521596

ti 0.05
ti 0.05

ti 0.1

Comparison of Forward Euler and RK4

For a given step size h
EFEM ah
1
4 c 3
ERK4 bh
h

EFEM = ah
ERK4 = bh4

Error and computation time when t = h or t = h/2

4th Order Runge-Kutta

Method

Forward Euler

Step size

h/2

h/2

Error

E/2

Eh3

Eh3/16

Time required

2T

4T

8T

Advantages of RK4 Methods

y 2t y

y0 1

EFEM ah

ERK 4 bh 4

Summary
Runge-Kutta method is better than Euler method in
terms of accuracy.
Global error for 2nd order Runge-Kutta method and
4th order Runge-Kutta method are proportional to h2
and h4, respectively.

Solving ODE-IVPs with MATLAB

y1 500.5 499.5 y1
y 499.5 500.5 y
2
2

function ydot = myode(t,y)

ydot =[-500.5*y(1)+499.5*y(2)
499.5*y(1)-500.5*y(2)];
save as myode.m

y10 2
y 1
20

ODE Solvers in the MATLAB

t0=0

starting point

tf =10

ending point

y0 =[2 1]

initial conditions

tspan =[t0 tf];

[t,y] = ode45('myode',tspan,y0);
name of the ODE
4th/5th order Runge-Kutta method

semilogx(t, y)
2
1.8

y1

1.6
1.4
1.2

y2

1
0.8
0.6
0.4
0.2
0
-5
10

-4

10

-3

10

-2

10

-1

10

10

10

Second Order ODE

y ( y 2 1) y y 0
y (0) 0.25, y(0) 0
Remember that a system of nonlinear differential equations can
always be expressed as a set of first order differential equations

y1 y
let

y1 0 y0 0

y2 y

y2 0 y0 0.25

y1 y1 (1 y22 ) y2
y2 y1

y1 0 0

y2 0 0.25

MATLAB Program
function ydot = vdpol(t,y)
ydot =[y(1).*(1-y(2).^2)-y(2);y(1)];
save as vdpol.m
t0=0;
tf=20;
tspan=[t0 tf];
y0=[0 0.25]'
[t,y]=ode45('vdpol',tspan,y0);

plot(t, y)
3

y
1

-1

y
-2

-3
0

10

12

14

16

18

20

Numerical Solution Methods for

Boundary Value Problems (BVPs)

Boundary Value Problems (BVPs)

Heat transfer equation
wire

d 2T
q
2
0
C p
dx
BC1

T TL x L

BC2

dT
0 x0
dx

T = TL

.
Where T is temperature, t is time, x is position and q
is the heat source

Dirichlet Type Boundary Conditions

Function values are given at both boundaries
y 2 y 3 y x
y (0) 1
y (1) 2
For example: constant temperature at both boundaries

T T1 x 0
T T2 x L

Neumann Type BCs

Derivatives are given at both boundaries
y 2 y 3 y x
y(0) 1
y(1) 2
For example: insulated surfaces or constant heat flux

dT
0 x0
dx
dT
1 x L
dx

Mixed BCs
y 2 y 3 y x
y(0) y (0) 1
y(1) y (1) 2
For example: heat flux depends on the temperature gradient

dT
k
h(T T1 ) x 0
dx
dT
k
h(T T 2 ) x L
dx

Solving BVPs with Shooting Method

y 2 y 3 y x
y (0) 1
y (1) 2
If we guess y(0) m
then we can treat the BVP as a IVP :

y 2 y 3 y x
y (0) 1
y(0) m

Use Euler method or Runge-Kutta

method to solve the ODE and get y(1)
Compare the calculated y*(1) with
the boundary condition y(1) = 2

Shooting Method
BC2
slope = m
BC1

Example
y 1 x / 5y x
y1 x 1 x / 5y2
y2 y1

y(1) 2, y(3) 1

y2 1 2, y2 3 1

function ydot = bvp(x,y)

ydot =[x+(1-x/5).*y(2);y(1)];
xspan=[1 3];
y0 =[-3.5 2]'
[x,y] = ode45('bvp',xspan,y0);
plot(x,y)

save as bvp.m

y2 (3) 4.811

5
4

y2 (1) 2

2
1

target

-1

y1 (1) 1.5

-2
1

1.2

1.4

1.6

1.8

2.2

2.4

2.6

2.8

y2 (1) 2

2
1.5
1
0.5
0

y2 (1) 0.453

-0.5

target

-1
-1.5
-2
-2.5

y1 (1) 3

-3
1

1.2

1.4

1.6

1.8

2.2

2.4

2.6

2.8

y2 (1) 2

target

-1

y2 (3) 1
-2

-3

y1 (1) 3.5
-4
1

1.2

1.4

1.6

1.8

2.2

2.4

2.6

2.8

How to Estimate the Next Guess?

S 2 S1
Next guess = S1
D R1
R2 R1

S1 : First guess
S2 : Second guess
R1 : First result
R2 : Second result
D : Desired value at the other boundary
3.00 1.50
1.000 4.811 3.500
1.5

0.453 0.4811

Finite Difference Method

Dirichlet-Type BCs
y 2 y 3 y 5 y(0) 1; y(1) 2
yi 1 yi 1
O h2
2h
yi 1 2 yi yi 1
2

yi
O
h
h2
yi

y 2 y0
y1
2h
y2 2 y1 y0

y1
h2

yn-1

y4
y
y2 3
y
y0 1

h
x 0 x 1 x2 x 3 x4

xn-1

Discretized Version of an ODE

at x1

y2 2 y1 y0
y2 y0
2
3 y1 5
2
h
2h

at x2

y3 2 y2 y1
y3 y1
2
3 y2 5
2
h
2h

at x3

y4 2 y3 y2
y4 y2
2
3 y3 5
2
h
2h

at xn-1

yn 2 yn1 yn2
yn yn 2
2
3 yn1 5
2
h
2h

 1 1
2
1 1
2 y0 2 3 y1 2 y2
h
h h
h

h
1 1
2
2 y1 2 3 y2
h
h
h

1 1
2 y2
h
h

Q
P

=5
1 1
2 y3
h h
2
2 3 y3
h

=5

1 1
2
1 1
2 yn 2 2 3 yn 1 2 yn
h
h h
h

=5

2
1 1
2 3 2
h

h h
2
1 1
2 2 3
h

h
h
1 1
2
h
h

1 1
2
h h
2
2 3

1 1
2
2 2 3
h
h
h

y1 1 1
y
5

h 2 h 0

3
5

1 1
yn 1 5 2 yn
h h

BC1

BC2

=5

MATLAB Program
h=0.01;
n=100;
Q=-2/h^2+3;
R=1/h^2-1/h;
P=1/h^2+1/h;
A=(diag(ones(n-1,1)*Q)+diag(ones(n-2,1)*R,1)+
diag(ones(n-2,1)*P,-1));
b=ones(n-1,1)*5;
b(1)=b(1)-P*1;
b(n-1)=b(n-1)-R*2;
y=A\b

Results
2

1.8

1.6

1.4

1.2

0.8
0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Finite Difference Method

Neumann-Type BCs
y 2 y 3 y 5 y(0) 1; y(1) 0
f i 2 4 f i 1 3 f i
O h2
2h
f 2 4 f1 3 f 0
1
f 0
2h

f i

f i 2 4 f i 1 3 f i
O h2
f i
2h
f n2 4 f n1 3 f n

0
fn
2h

3-point forward difference

BC1

3-point backward difference

BC2

Matrix Form
BC1
3

2h
1 1
2
h
h

2
h

2h
1 1
2

3
2
2

h h
h

2
1 1

2 3
2

h
h

y
0 1

y1 5

1 1
y 5

2

h h

1 1
2 1 1 yn 1
2 2 3 2
5
h
h
h
h h

1
2
3

h
2h
2h n 0

BC2

MATLAB Program
h=0.01;
n=100;
Q=-2/h^2+3;
R=1/h^2-1/h;
P=1/h^2+1/h;
A=diag(ones(n+1,1)*Q)+diag(ones(n,1)*R,1)+diag
(ones(n,1)*P,-1);
A(1,1)=-3/(2*h);A(1,2)=2/h;A(1,3)=-1/(2*h);
A(n+1,n+1)=3/(2*h);A(n+1,n)=-2/h;A(n+1,n1)=1/(2*h);
b=ones(n+1,1)*5;
b(1)=1;
b(n+1)=0;
y=A\b;

Result
3
2.9
2.8
2.7
2.6

2.5
2.4
2.3
2.2
2.1
2
0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Summary
Shooting method
convert BVPs into IVPs
Finite difference method
Dirichlet type
Neumann type