Introduction to Numerical Analysis

(Taylor Approximation)

MA 214, Spring 2023-24.

MA 214 - NA Spring 2023-24 1 / 42

Recall
Theorem: Let f be (n + 1)-times differentiable function on an open
interval containing the points a and x. Then there exists a number ζ
between a and x such that :

f (n+1) (ζ)
f(x) = Tn (x) +
(n + 1)!
where Tn is the Taylor’s polynomial of degree n for f at the point a and
the second term on the right hand side is the Remainder Term.
Pn f (k) (a)
Recall Tn (x) = (x − a)k , x ∈ R.
k=0 k!
Note that
T0 (x) = 1
MA 214 - NA Spring 2023-24 2 / 42
Mathematical Preliminaries: Taylor Polynomial (Contd.)
Taylor polynomial of degree 0 about the point a = 0 for the function
f (x) = ex
3

y = e x (blue), y = T 0 (x) (red)

2.5

1.5

0.5

0
−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1
x

Pn f (k) (a)
Recall Tn (x) = (x − a)k , x ∈ R. T0 (x) = 1
k=0 k!
MA 214 - NA Spring 2023-24 3 / 42
Mathematical Preliminaries: Taylor Polynomial (Contd.)
Taylor polynomial of degree 1 about the point a = 0 for the function
f(x) = ex
3

y = e x (blue), y = T 1 (x) (red)

2.5

1.5

0.5

0
−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1
x

Pn f (k) (a)
Recall Tn (x) = (x − a)k , x ∈ R. T1 (x) = 1 + x
k=0 k!
MA 214 - NA Spring 2023-24 4 / 42
Mathematical Preliminaries: Taylor Polynomial (Contd.)
Taylor polynomial of degree 2 about the point a = 0 for the function
f(x) = ex
3

y = e x (blue), y = T 2 (x) (red)

2.5

1.5

0.5

0
−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1
x

Pn f (k) (a) x2
Recall Tn (x) = (x − a)k , x ∈ R. T2 (x) = 1 + x +
k=0 k! 2!
MA 214 - NA Spring 2023-24 5 / 42
Mathematical Preliminaries: Taylor Polynomial (Contd.)
Taylor polynomial of degree 3 about the point a = 0 for the function
f(x) = ex
3

y = e x (blue), y = T 3 (x) (red)

2.5

1.5

0.5

0
−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1
x

Pn f (k) (a) x2 x3
Recall Tn (x) = (x − a)k , x ∈ R. T3 (x) = 1 + x + +
k=0 k! 2! 3!
MA 214 - NA Spring 2023-24 6 / 42
Mathematical Preliminaries: Taylor Polynomial (Contd.)
Taylor polynomial of degree 4 about the point a = 0 for the function
f(x) = ex
3

y = e x (blue), y = T 4 (x) (red)

2.5

1.5

0.5

0
−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1
x

x2 x3 x4
T4 (x) = 1 + x + + +
2! 3! 4!
MA 214 - NA Spring 2023-24 7 / 42
Taylor’s Approximation: Taylor’s Theorem (Contd.)

Example: The remainder term of the Taylor formula (with n = 4) for

ξ
f(x) = ex about a = 0 is R5 = e5! x5 , where ξ lies between x and 0.

MA 214 - NA Spring 2023-24 8 / 42

Taylor’s Approximation: Taylor’s Theorem (Contd.)

Example: The remainder term of the Taylor formula (with n = 4) for

ξ
f(x) = ex about a = 0 is R5 = e5! x5 , where ξ lies between x and 0.
Taylor Theorem ⇒ ex = T4 (x) + R5 .

MA 214 - NA Spring 2023-24 8 / 42

Taylor’s Approximation: Taylor’s Theorem (Contd.)

Example: The remainder term of the Taylor formula (with n = 4) for

ξ
f(x) = ex about a = 0 is R5 = e5! x5 , where ξ lies between x and 0.
Taylor Theorem ⇒ ex = T4 (x) + R5 .

x x2 x3 x4 eξ 5
⇒ e = 1+x+ + + + x
2! 3! 4! 5!

MA 214 - NA Spring 2023-24 8 / 42

Taylor’s Approximation: Taylor’s Theorem (Contd.)

Example: The remainder term of the Taylor formula (with n = 4) for

ξ
f(x) = ex about a = 0 is R5 = e5! x5 , where ξ lies between x and 0.
Taylor Theorem ⇒ ex = T4 (x) + R5 .

x x2 x3 x4 eξ 5
⇒ e = 1+x+ + + + x
2! 3! 4! 5!
Note that e ≈ 2.7183 and T4 (1) ≈ 2.7083

MA 214 - NA Spring 2023-24 8 / 42

Taylor’s Approximation: Taylor’s Theorem (Contd.)

Example: The remainder term of the Taylor formula (with n = 4) for

ξ
f(x) = ex about a = 0 is R5 = e5! x5 , where ξ lies between x and 0.
Taylor Theorem ⇒ ex = T4 (x) + R5 .

x x2 x3 x4 eξ 5
⇒ e = 1+x+ + + + x
2! 3! 4! 5!
Note that e ≈ 2.7183 and T4 (1) ≈ 2.7083 ⇒ e − T4 (1) ≈ 0.01.

MA 214 - NA Spring 2023-24 8 / 42

Taylor’s Approximation: Taylor’s Theorem (Contd.)

x x2 x3 x4 eξ 5
⇒ e = 1+x+ + + + x
2! 3! 4! 5!
Note that e ≈ 2.7183 and T4 (1) ≈ 2.7083 ⇒ e − T4 (1) ≈ 0.01.
For any x < 1,

MA 214 - NA Spring 2023-24 9 / 42

Taylor’s Approximation: Taylor’s Theorem (Contd.)

x x2 x3 x4 eξ 5
⇒ e = 1+x+ + + + x
2! 3! 4! 5!
Note that e ≈ 2.7183 and T4 (1) ≈ 2.7083 ⇒ e − T4 (1) ≈ 0.01.
For any x < 1,
eξ 5 e
|R5 | = x ≤ ≈ 0.0227.
5! 5!

MA 214 - NA Spring 2023-24 9 / 42

Taylor’s Approximation: Taylor’s Theorem (Contd.)

x x2 x3 x4 eξ 5
⇒ e = 1+x+ + + + x
2! 3! 4! 5!
Note that e ≈ 2.7183 and T4 (1) ≈ 2.7083 ⇒ e − T4 (1) ≈ 0.01.
For any x < 1,
eξ 5 e
|R5 | = x ≤ ≈ 0.0227.
5! 5!

0.01 ≈ |e − T4 (1)| ≤ 0.0227

MA 214 - NA Spring 2023-24 9 / 42

Taylor’s Approximation: Truncation Error

Definition (Truncation Error)

The remainder term involved in approximating f(x) by the Taylor’s
polynomial Tn (x) is also called the Truncation error.

MA 214 - NA Spring 2023-24 10 / 42

Taylor’s Approximation: Taylor’s Theorem (Contd.)
How Taylor’s Theorem is used?

MA 214 - NA Spring 2023-24 11 / 42

Taylor’s Approximation: Taylor’s Theorem (Contd.)
How Taylor’s Theorem is used?
Given:
a ∈ R and h ∈ R

MA 214 - NA Spring 2023-24 11 / 42

Taylor’s Approximation: Taylor’s Theorem (Contd.)
How Taylor’s Theorem is used?
Given:
a ∈ R and h ∈ R
An integer n

MA 214 - NA Spring 2023-24 11 / 42

Taylor’s Approximation: Taylor’s Theorem (Contd.)
How Taylor’s Theorem is used?
Given:
a ∈ R and h ∈ R
An integer n
′
f(a), f (a) , · · · , f (n) (a)

MA 214 - NA Spring 2023-24 11 / 42

Taylor’s Approximation: Taylor’s Theorem (Contd.)
How Taylor’s Theorem is used?
Given:
a ∈ R and h ∈ R
An integer n
′
f(a), f (a) , · · · , f (n) (a)

We want: f (a + h) ≈ ?

MA 214 - NA Spring 2023-24 11 / 42

Taylor’s Approximation: Taylor’s Theorem (Contd.)
How Taylor’s Theorem is used?
Given:
a ∈ R and h ∈ R
An integer n
′
f(a), f (a) , · · · , f (n) (a)

We want: f (a + h) ≈ ?
Taylor’s theorem can be used to get
f ′′ (a) 2 f (n) (a) n
f (a + h) ≈ f(a) + f ′ (a)h + h + ··· + h.
2! n!
MA 214 - NA Spring 2023-24 11 / 42
Taylor’s Approximation: Taylor’s Theorem (Contd.)

How to estimate the truncation error?

MA 214 - NA Spring 2023-24 12 / 42

Taylor’s Approximation: Taylor’s Theorem (Contd.)

How to estimate the truncation error?

f → (n + 1)-times continuously differentiable

MA 214 - NA Spring 2023-24 12 / 42

Taylor’s Approximation: Taylor’s Theorem (Contd.)

How to estimate the truncation error?

f → (n + 1)-times continuously differentiable

There exists an Mn+1 such that
|f (n+1) (ξ)| ≤ Mn+1 , for all ξ ∈ I := [α, β].

MA 214 - NA Spring 2023-24 12 / 42

Taylor’s Approximation: Taylor’s Theorem (Contd.)

How to estimate the truncation error?

f → (n + 1)-times continuously differentiable

There exists an Mn+1 such that
|f (n+1) (ξ)| ≤ Mn+1 , for all ξ ∈ I := [α, β].

Then, for fixed points a, x ∈ I

f (n+1) (ξ)
(x − a)n+1
(n + 1)!

MA 214 - NA Spring 2023-24 12 / 42

Taylor’s Approximation: Taylor’s Theorem (Contd.)

How to estimate the truncation error?

f → (n + 1)-times continuously differentiable

There exists an Mn+1 such that
|f (n+1) (ξ)| ≤ Mn+1 , for all ξ ∈ I := [α, β].

Then, for fixed points a, x ∈ I

f (n+1) (ξ) Mn+1

(x − a)n+1 ≤ |x − a|n+1 .
(n + 1)! (n + 1)!

MA 214 - NA Spring 2023-24 12 / 42

Taylor’s Approximation: Taylor’s Theorem (Contd.)

How to estimate the truncation error?

f → (n + 1)-times continuously differentiable

There exists an Mn+1 such that
|f (n+1) (ξ)| ≤ Mn+1 , for all ξ ∈ I := [α, β].

Then, for fixed points a, x ∈ I

f (n+1) (ξ) Mn+1

(x − a)n+1 ≤ |x − a|n+1 .
(n + 1)! (n + 1)!

We want the estimate on the RHS to be independent of x

MA 214 - NA Spring 2023-24 12 / 42
Taylor’s Approximation: Taylor’s Theorem (Contd.)

How to estimate the truncation error?

f (n+1) (ξ) Mn+1

(x − a)n+1 ≤ (β − α)n+1 , x ∈ I.
(n + 1)! (n + 1)!

MA 214 - NA Spring 2023-24 13 / 42

Taylor’s Approximation: Taylor’s Series

Definition (Taylor’s Series)

Let f be C∞ in a neighborhood of a point a. The power series
∞ (k)
X f (a)
(x − a)k
k!
k=0

is called the Taylor’s series of f about the point a.

MA 214 - NA Spring 2023-24 14 / 42

A function f : (a, b) → R is said to be real analytic if given any
x ∈ (a, b), there exists a neighborhood x ∈ U ⊂ (a, b) such that for any
point in U, the Taylor series expanded around x converges at all points
in U and gives the value of the function at that point

MA 214 - NA Spring 2023-24 15 / 42

A function f : (a, b) → R is said to be real analytic if given any
x ∈ (a, b), there exists a neighborhood x ∈ U ⊂ (a, b) such that for any
point in U, the Taylor series expanded around x converges at all points
in U and gives the value of the function at that point
X∞ (k)
f (a)
f(x) = (x − a)k .
k!
k=0

MA 214 - NA Spring 2023-24 15 / 42

Analytic Functions

All the important functions from calculus are real analytic. Polynomials,
sine, cosine, exponential function are all analytic functions. Composite
of two analytic functions is analytic. Sum, products and ratios of
analytic functions with nowhere vanishing denominator are analytic.
This gives us an abundance of analytic functions.

MA 214 - NA Spring 2023-24 16 / 42

Zero’s of analytic functions

An important feature of non-zero analytic functions on R is that their

zero’s form a discrete subset. If f is a non-zero analytic function on R,
then the zero set of f is discrete. That is, if a is a root of f, there exists
a neighborhood of a in which f has no other zero. Consequently if f and
g are two analytic functions which agree on a non-discrete set, they
coincide everywhere.

MA 214 - NA Spring 2023-24 17 / 42

Zero’s of analytic functions

An important feature of non-zero analytic functions on R is that their

zero’s form a discrete subset. If f is a non-zero analytic function on R,
then the zero set of f is discrete. That is, if a is a root of f, there exists
a neighborhood of a in which f has no other zero. Consequently if f and
g are two analytic functions which agree on a non-discrete set, they
coincide everywhere.
In particular, the zero set of any analytic function is either the entire real
line, or countable.

MA 214 - NA Spring 2023-24 17 / 42

Mathematical Preliminaries: Taylor’s Series (Contd.)
Example: Let f(x) = cos(x).

MA 214 - NA Spring 2023-24 18 / 42

Mathematical Preliminaries: Taylor’s Series (Contd.)
Example: Let f(x) = cos(x). The Taylor’s series about a is given by
cos(a)
cos(x) = cos(a) − sin(a)(x − a) − (x − a)2
2
sin(a) cos(a)
+ (x − a)3 + (x − a)4 + · · ·
6 24

MA 214 - NA Spring 2023-24 18 / 42

Mathematical Preliminaries: Taylor’s Series (Contd.)
Example: Let f(x) = cos(x). The Taylor’s series about a is given by
cos(a)
cos(x) = cos(a) − sin(a)(x − a) − (x − a)2
2
sin(a) cos(a)
+ (x − a)3 + (x − a)4 + · · ·
6 24
In particular, when a = 0,
x2 x4
cos(x) = 1 − + + ···
2 24

MA 214 - NA Spring 2023-24 18 / 42

Mathematical Preliminaries: Taylor’s Series (Contd.)
Example: Let f(x) = cos(x). The Taylor’s series about a is given by
cos(a)
cos(x) = cos(a) − sin(a)(x − a) − (x − a)2
2
sin(a) cos(a)
+ (x − a)3 + (x − a)4 + · · ·
6 24
In particular, when a = 0,
x2 x4
cos(x) = 1 − + + ···
2 24
That is, ∞
X (−1)k
cos(x) = x2k
(2k)!
k=0
MA 214 - NA Spring 2023-24 18 / 42
Taylor’s Approximation: Taylor’s Formula for Cosine Function

Taylor’s formula at a = 0 for n = 4

MA 214 - NA Spring 2023-24 19 / 42

Taylor’s Approximation: Taylor’s Formula for Cosine Function

Taylor’s formula at a = 0 for n = 4

x2 x4
cos(x) ≈ 1 − +
2 24

MA 214 - NA Spring 2023-24 19 / 42

Taylor’s Approximation: Taylor’s Formula for Cosine Function

Taylor’s formula at a = 0 for n = 4

x2 x4
cos(x) ≈ 1 − +
2 24
Remainder Term

(−1)3
cos(ξ) x6
6!
where ξ lies between 0 and x.

MA 214 - NA Spring 2023-24 19 / 42

Taylor’s Approximation: Taylor’s Formula for Cosine Function

Taylor’s formula at a = 0 for n = 4

x2 x4 (−1)3
cos(x) = 1 − + + cos(ξ) x6
2 24 6!
General Taylor’s formula at a = 0

X
m
(−1)k cos(ξ)
cos(x) = x2k + (−1)m+1 x2(m+1) .
(2k)! (2(m + 1))!
k=0

MA 214 - NA Spring 2023-24 20 / 42

Taylor’s Approximation: Taylor’s Formula for Cosine Function
The Taylor’s polynomial of degree n = 4 about a ̸= 0 is
cos(a) sin(a) cos(a)
T4 (x) = cos(a)−sin(a)(x−a)− (x−a)2 + (x−a)3 + (x−a)4 .
2 3! 5!

MA 214 - NA Spring 2023-24 21 / 42

Taylor’s Approximation: Taylor’s Formula for Cosine Function
The Taylor’s polynomial of degree n = 4 about a ̸= 0 is
cos(a) sin(a) cos(a)
T4 (x) = cos(a)−sin(a)(x−a)− (x−a)2 + (x−a)3 + (x−a)4 .
2 3! 5!
The remainder term is given by
sin(ξ)
− (x − a)5
5!
where ξ lies between a and x.

MA 214 - NA Spring 2023-24 21 / 42

Taylor’s Approximation: Taylor’s Formula for Cosine Function
The Taylor’s polynomial of degree n = 4 about a ̸= 0 is
cos(a) sin(a) cos(a)
T4 (x) = cos(a)−sin(a)(x−a)− (x−a)2 + (x−a)3 + (x−a)4 .
2 3! 5!
The remainder term is given by
sin(ξ)
− (x − a)5
5!
where ξ lies between a and x.
Therefore, Taylor’s theorem tells us that

MA 214 - NA Spring 2023-24 21 / 42

Taylor’s Approximation: Taylor’s Formula for Cosine Function
The Taylor’s polynomial of degree n = 4 about a ̸= 0 is
cos(a) sin(a) cos(a)
T4 (x) = cos(a)−sin(a)(x−a)− (x−a)2 + (x−a)3 + (x−a)4 .
2 3! 5!
The remainder term is given by
sin(ξ)
− (x − a)5
5!
where ξ lies between a and x.
Therefore, Taylor’s theorem tells us that given x ∈ R,

MA 214 - NA Spring 2023-24 21 / 42

Taylor’s Approximation: Taylor’s Formula for Cosine Function
The Taylor’s polynomial of degree n = 4 about a ̸= 0 is
cos(a) sin(a) cos(a)
T4 (x) = cos(a)−sin(a)(x−a)− (x−a)2 + (x−a)3 + (x−a)4 .
2 3! 5!
The remainder term is given by
sin(ξ)
− (x − a)5
5!
where ξ lies between a and x.
Therefore, Taylor’s theorem tells us that given x ∈ R, there exists a
point ξ between x and a

MA 214 - NA Spring 2023-24 21 / 42

Taylor’s Approximation: Taylor’s Formula for Cosine Function
The Taylor’s polynomial of degree n = 4 about a ̸= 0 is
cos(a) sin(a) cos(a)
T4 (x) = cos(a)−sin(a)(x−a)− (x−a)2 + (x−a)3 + (x−a)4 .
2 3! 5!
The remainder term is given by
sin(ξ)
− (x − a)5
5!
where ξ lies between a and x.
Therefore, Taylor’s theorem tells us that given x ∈ R, there exists a
point ξ between x and a such that
sin(ξ)
cos(x) = T4 (x) − (x − a)5 .
5!
MA 214 - NA Spring 2023-24 21 / 42
Caution 1

A function that is real analytic is neccessarily infinitely differentiable.

However the converse is not true.

MA 214 - NA Spring 2023-24 22 / 42

Caution 1

A function that is real analytic is neccessarily infinitely differentiable.

However the converse is not true. There exists (in abundance) C∞
functions which are not real analytic. The most famous example is

f(x) = 0 for x ≤ 0

and 1
f(x) = e− x
for x > 0.

Using this function one can construct other examples.

MA 214 - NA Spring 2023-24 22 / 42

Caution 2

Even if the function is real analytic, it may not be expandable as a

global power series around a point, for the entire domain.

MA 214 - NA Spring 2023-24 23 / 42

Caution 2

Even if the function is real analytic, it may not be expandable as a

global power series around a point, for the entire domain.For example,
1
the function 1+x 2 is real-analytic, but cannot be expanded as a power

series around 0 which converges at every point on R.

MA 214 - NA Spring 2023-24 23 / 42

Taylor’s theorem in more variables

We first set up some notation :

A multi-index is an n-tuple of nonnegative integers. Multi-indices are

generally denoted by the Greek letters α or β.
α = (α1 , α2 , · · · , αn ), β = (β1 , β2 , · · · , βn ), with αj , βj ∈ {0, 1, 2, ...}.

If α is a multi-index, we define |α| = α1 + α2 + · · · + αn ,

α! = α1 !α2 ! · · · αn !.

MA 214 - NA Spring 2023-24 24 / 42

xα = xα1 1 xα2 2 · · · xαn n (where x = (x1 , x2 , · · · , xn ) ∈ Rn ),
|α|
f
∂ α f = ∂1α1 ∂2α2 · · · ∂nαn f = ∂xα1 ∂x∂α2 ···∂xαm
1 2 n
The number |α| = α1 + · · · + · · · αn is called the order or degree of α.

If f is a function of class C k , the order of differentiation in a kth -order

partial derivative of f is immaterial. Thus, the generic kth -order partial
derivative of f can be written simply as ∂ α f with |α| = k.

MA 214 - NA Spring 2023-24 25 / 42

Taylor’s theorem in more variables

Suppose f : Rn → R is of class C k+1 on an open convex set S ⊂ Rn . If

a ∈ S and a + h ∈ S, then
X ∂ α f(a)
f(a + h) = hα + Ra,k (h)
α!
|α|≤k

where the remainder is given in Lagrange’s form by

X hα
Ra,k (h) = ∂ f(a + ch) for some c ∈ (0, 1).
α
α!
|α|=k+1

MA 214 - NA Spring 2023-24 26 / 42

Taylor’s theorem in 2 variables

Thus in 2 variables, Taylor’s theorem would read :

∂f(a, b) ∂f(a, b) 1 ∂ 2 f(a, b)

f(x,y)=f(a,b)+ (x − a) + (y − b) + 2
(x − a)2 +
∂x ∂y 2! ∂x
2 2
∂ f(a, b) 1 ∂ f(a, b)
(x − a)(y − b) + 2
(y − b)2 + Remainder Term
∂x∂y 2! ∂y

MA 214 - NA Spring 2023-24 27 / 42

Order of Convergence

MA 214 - NA Spring 2023-24 28 / 42

Orders of Convergence: Motivation
Example:
Sequence {n} Sequence {n2 }

MA 214 - NA Spring 2023-24 29 / 42

Orders of Convergence: Motivation
Example:
Sequence {n} Sequence {n2 }
Unbounded Unbounded

MA 214 - NA Spring 2023-24 29 / 42

Orders of Convergence: Motivation
Example:
Sequence {n} Sequence {n2 }
Unbounded Unbounded
Tends to ∞ as n → ∞ Tends to ∞ as n → ∞

MA 214 - NA Spring 2023-24 29 / 42

Orders of Convergence: Motivation
Example:
Sequence {n} Sequence {n2 }
Unbounded Unbounded
Tends to ∞ as n → ∞ Tends to ∞ as n → ∞

Which tends to ∞ faster?

MA 214 - NA Spring 2023-24 29 / 42

Orders of Convergence: Motivation
Example:
Sequence {n} Sequence {n2 }
Unbounded Unbounded
Tends to ∞ as n → ∞ Tends to ∞ as n → ∞

Which tends to ∞ faster?

Example:    
1 1
Sequence Sequence
n n2

MA 214 - NA Spring 2023-24 29 / 42

Orders of Convergence: Motivation
Example:
Sequence {n} Sequence {n2 }
Unbounded Unbounded
Tends to ∞ as n → ∞ Tends to ∞ as n → ∞

Which tends to ∞ faster?

Example:    
1 1
Sequence Sequence
n n2
Bounded Bounded

MA 214 - NA Spring 2023-24 29 / 42

Orders of Convergence: Motivation
Example:
Sequence {n} Sequence {n2 }
Unbounded Unbounded
Tends to ∞ as n → ∞ Tends to ∞ as n → ∞

Which tends to ∞ faster?

Example:    
1 1
Sequence Sequence
n n2
Bounded Bounded
Converges to 0 as n → ∞ Converges to 0 as n → ∞

MA 214 - NA Spring 2023-24 29 / 42

Orders of Convergence: Motivation
Example:
Sequence {n} Sequence {n2 }
Unbounded Unbounded
Tends to ∞ as n → ∞ Tends to ∞ as n → ∞

Which tends to ∞ faster?

Example:    
1 1
Sequence Sequence
n n2
Bounded Bounded
Converges to 0 as n → ∞ Converges to 0 as n → ∞
Which converges to 0 faster?
MA 214 - NA Spring 2023-24 29 / 42
Orders of Convergence: Motivation (contd.)

Given a convergent sequence.

MA 214 - NA Spring 2023-24 30 / 42

Orders of Convergence: Motivation (contd.)

Given a convergent sequence.

the order of convergence

MA 214 - NA Spring 2023-24 30 / 42

Orders of Convergence: Motivation (contd.)

Given a convergent sequence.

the order of convergence
is all about

MA 214 - NA Spring 2023-24 30 / 42

Orders of Convergence: Motivation (contd.)

Given a convergent sequence.

the order of convergence
is all about
studying the speed at which a sequence converges.

MA 214 - NA Spring 2023-24 30 / 42

Orders of Convergence: Motivation (contd.)

Given a convergent sequence.

the order of convergence
is all about
studying the speed at which a sequence converges.

Two approaches are

big O and little o notions; and
asymptotic rate of convergence.

MA 214 - NA Spring 2023-24 30 / 42

Orders of Convergence: Big Oh and Little oh

Big Oh and Little oh

By comparing the required sequence with a known sequence.

MA 214 - NA Spring 2023-24 31 / 42

Orders of Convergence: Big Oh and Little oh

Big Oh and Little oh

By comparing the required sequence with a known sequence.

MA 214 - NA Spring 2023-24 31 / 42

Orders of Convergence: Big Oh and Little oh

Big Oh and Little oh

By comparing the required sequence with a known sequence.

Introduced by

Edmund Landau Paul Bachmann

1877 − 1938 (Germany) 1837 − 1920 (Germany)
MA 214 - NA Spring 2023-24 31 / 42
Orders of Convergence: Big Oh and Little oh (contd.)

Definition (Big Oh)

Let {an } and {bn } be sequences of real numbers. Then

MA 214 - NA Spring 2023-24 32 / 42

Orders of Convergence: Big Oh and Little oh (contd.)

Definition (Big Oh)

Let {an } and {bn } be sequences of real numbers. Then Big Oh:
Notation: an = O(bn )

MA 214 - NA Spring 2023-24 32 / 42

Orders of Convergence: Big Oh and Little oh (contd.)

Definition (Big Oh)

Let {an } and {bn } be sequences of real numbers. Then Big Oh:
Notation: an = O(bn )
if there exists a real number C and a natural number N such that
|an | ≤ C |bn | for all n ≥ N.

MA 214 - NA Spring 2023-24 32 / 42

Orders of Convergence: Big Oh and Little oh (contd.)

Definition (Big Oh)

Let {an } and {bn } be sequences of real numbers. Then Big Oh:
Notation: an = O(bn )
if there exists a real number C and a natural number N such that
|an | ≤ C |bn | for all n ≥ N.

If bn ̸= 0 for every n,
 
an
an = O(bn ) ⇔ is bounded.
bn
MA 214 - NA Spring 2023-24 32 / 42
Orders of Convergence: Big Oh and Little oh (contd.)
Definition (Little oh)
Let {an } and {bn } be sequences of real numbers. Then

MA 214 - NA Spring 2023-24 33 / 42

Orders of Convergence: Big Oh and Little oh (contd.)
Definition (Little oh)
Let {an } and {bn } be sequences of real numbers. Then Little oh or
small oh:
Notation: an = o(bn )

MA 214 - NA Spring 2023-24 33 / 42

Orders of Convergence: Big Oh and Little oh (contd.)
Definition (Little oh)
Let {an } and {bn } be sequences of real numbers. Then Little oh or
small oh:
Notation: an = o(bn )

if for every ϵ > 0 there exists a natural number N such that

|an | ≤ ϵ |bn | for all n ≥ N.

MA 214 - NA Spring 2023-24 33 / 42

Orders of Convergence: Big Oh and Little oh (contd.)
Definition (Little oh)
Let {an } and {bn } be sequences of real numbers. Then Little oh or
small oh:
Notation: an = o(bn )

if for every ϵ > 0 there exists a natural number N such that

|an | ≤ ϵ |bn | for all n ≥ N.

If bn ̸= 0 for every n,
 
an
an = o(bn ) ⇔ converges to 0.
bn
MA 214 - NA Spring 2023-24 33 / 42
Orders of Convergence: Big Oh and Little oh (contd.)
Remark:
Let {an } and {bn } be two sequences.

MA 214 - NA Spring 2023-24 34 / 42

Orders of Convergence: Big Oh and Little oh (contd.)
Remark:
Let {an } and {bn } be two sequences.
Then an = o(bn ) =⇒ an = O(bn ).

MA 214 - NA Spring 2023-24 34 / 42

Orders of Convergence: Big Oh and Little oh (contd.)
Remark:
Let {an } and {bn } be two sequences.
Then an = o(bn ) =⇒ an = O(bn ).
The converse is not true.

MA 214 - NA Spring 2023-24 34 / 42

Orders of Convergence: Big Oh and Little oh (contd.)
Remark:
Let {an } and {bn } be two sequences.
Then an = o(bn ) =⇒ an = O(bn ).
The converse is not true.
Example: an = n and bn = 2n + 3

MA 214 - NA Spring 2023-24 34 / 42

Orders of Convergence: Big Oh and Little oh (contd.)
Remark:
Let {an } and {bn } be two sequences.
Then an = o(bn ) =⇒ an = O(bn ).
The converse is not true.
Example: an = n and bn = 2n + 3
Let an → 0 and bn → 0. Then

MA 214 - NA Spring 2023-24 34 / 42

Orders of Convergence: Big Oh and Little oh (contd.)
Remark:
Let {an } and {bn } be two sequences.
Then an = o(bn ) =⇒ an = O(bn ).
The converse is not true.
Example: an = n and bn = 2n + 3
Let an → 0 and bn → 0. Then
an = O(bn ) means the sequence {an } tends to 0 AT LEAST as fast as the
sequence {bn }

MA 214 - NA Spring 2023-24 34 / 42

Orders of Convergence: Big Oh and Little oh (contd.)
Remark:
Let {an } and {bn } be two sequences.
Then an = o(bn ) =⇒ an = O(bn ).
The converse is not true.
Example: an = n and bn = 2n + 3
Let an → 0 and bn → 0. Then
an = O(bn ) means the sequence {an } tends to 0 AT LEAST as fast as the
sequence {bn }

an = o(bn ) means the sequence {an } tends to 0 faster than the sequence
{bn }
MA 214 - NA Spring 2023-24 34 / 42
Orders of Convergence: Big Oh and Little oh (contd.)
The Big Oh and Little oh notations can be adapted for functions as follows.

MA 214 - NA Spring 2023-24 35 / 42

Orders of Convergence: Big Oh and Little oh (contd.)
The Big Oh and Little oh notations can be adapted for functions as follows.
Definition (Big Oh for Functions)
Let x0 ∈ R.

MA 214 - NA Spring 2023-24 35 / 42

Orders of Convergence: Big Oh and Little oh (contd.)
The Big Oh and Little oh notations can be adapted for functions as follows.
Definition (Big Oh for Functions)
Let x0 ∈ R.
Let f and g be continuous functions defined in an interval containing
x0 .

MA 214 - NA Spring 2023-24 35 / 42

Orders of Convergence: Big Oh and Little oh (contd.)
The Big Oh and Little oh notations can be adapted for functions as follows.
Definition (Big Oh for Functions)
Let x0 ∈ R.
Let f and g be continuous functions defined in an interval containing
x0 .
Then, the function f is said to be Big Oh of g as x → x0

MA 214 - NA Spring 2023-24 35 / 42

Orders of Convergence: Big Oh and Little oh (contd.)
The Big Oh and Little oh notations can be adapted for functions as follows.
Definition (Big Oh for Functions)
Let x0 ∈ R.
Let f and g be continuous functions defined in an interval containing
x0 .
Then, the function f is said to be Big Oh of g as x → x0


Notation: f(x) = O g(x)

MA 214 - NA Spring 2023-24 35 / 42

Orders of Convergence: Big Oh and Little oh (contd.)
The Big Oh and Little oh notations can be adapted for functions as follows.
Definition (Big Oh for Functions)
Let x0 ∈ R.
Let f and g be continuous functions defined in an interval containing
x0 .
Then, the function f is said to be Big Oh of g as x → x0


Notation: f(x) = O g(x)

if there exists a real number C and a real number δ such that

|f(x)| ≤ C |g(x)| whenever |x − x0 | ≤ δ.

MA 214 - NA Spring 2023-24 35 / 42
Orders of Convergence: Big Oh and Little oh (contd.)

Definition (Little oh for Functions)

Let x0 ∈ R.

MA 214 - NA Spring 2023-24 36 / 42

Orders of Convergence: Big Oh and Little oh (contd.)

Definition (Little oh for Functions)

Let x0 ∈ R.
Let f and g be continuous functions defined in an interval containing
x0 .

MA 214 - NA Spring 2023-24 36 / 42

Orders of Convergence: Big Oh and Little oh (contd.)

Definition (Little oh for Functions)

Let x0 ∈ R.
Let f and g be continuous functions defined in an interval containing
x0 .
Then, the function f is said to be Little Oh (or Small oh) of g as
x → x0 ,

MA 214 - NA Spring 2023-24 36 / 42

Orders of Convergence: Big Oh and Little oh (contd.)

Definition (Little oh for Functions)

Let x0 ∈ R.
Let f and g be continuous functions defined in an interval containing
x0 .
Then, the function f is said to be Little Oh (or Small oh) of g as
x → x0 ,


Notation: f(x) = o g(x)

MA 214 - NA Spring 2023-24 36 / 42

Orders of Convergence: Big Oh and Little oh (contd.)

Definition (Little oh for Functions)

Let x0 ∈ R.
Let f and g be continuous functions defined in an interval containing
x0 .
Then, the function f is said to be Little Oh (or Small oh) of g as
x → x0 ,


Notation: f(x) = o g(x)

if for every ϵ > 0 there exists a real number δ > 0 such that
|f(x)| ≤ ϵ |g(x)| whenever |x − x0 | ≤ δ.

MA 214 - NA Spring 2023-24 36 / 42

Orders of Convergence: Big Oh and Little oh (contd.)

Remark: Let f and g be two continuous functions.

MA 214 - NA Spring 2023-24 37 / 42

Orders of Convergence: Big Oh and Little oh (contd.)

Remark: Let f and g be two continuous functions.

Then f(x) = o(g(x)) =⇒ f(x) = O(g(x)).

MA 214 - NA Spring 2023-24 37 / 42

Orders of Convergence: Big Oh and Little oh (contd.)

Remark: Let f and g be two continuous functions.

Then f(x) = o(g(x)) =⇒ f(x) = O(g(x)).
The converse is not true.

MA 214 - NA Spring 2023-24 37 / 42

Orders of Convergence: Big Oh and Little oh (contd.)

Remark: Let f and g be two continuous functions.

Then f(x) = o(g(x)) =⇒ f(x) = O(g(x)).
The converse is not true.
Example: ???

MA 214 - NA Spring 2023-24 37 / 42

Orders of Convergence: Big Oh and Little oh (contd.)

Remark: Let f and g be two continuous functions.

Then f(x) = o(g(x)) =⇒ f(x) = O(g(x)).
The converse is not true.
Example: ???
Let f(x) → 0 and g(x) → 0 Then :

MA 214 - NA Spring 2023-24 37 / 42

Orders of Convergence: Big Oh and Little oh (contd.)

Remark: Let f and g be two continuous functions.

Then f(x) = o(g(x)) =⇒ f(x) = O(g(x)).
The converse is not true.
Example: ???
Let f(x) → 0 and g(x) → 0 Then :
f(x) = O(g(x)) means f(x) → 0 AT LEAST as fast as g(x) → 0 x → a

MA 214 - NA Spring 2023-24 37 / 42

Orders of Convergence: Big Oh and Little oh (contd.)

Remark: Let f and g be two continuous functions.

Then f(x) = o(g(x)) =⇒ f(x) = O(g(x)).
The converse is not true.
Example: ???
Let f(x) → 0 and g(x) → 0 Then :
f(x) = O(g(x)) means f(x) → 0 AT LEAST as fast as g(x) → 0 x → a

f(x) = o(g(x)) means f(x) → 0 faster than g(x) → 0 as x → a

MA 214 - NA Spring 2023-24 37 / 42

Likewise for functions defined on the real line, the function f is said to
be Big Oh of g as x → ∞

MA 214 - NA Spring 2023-24 38 / 42

Likewise for functions defined on the real line, the function f is said to
be Big Oh of g as x → ∞


Notation: f(x) = O g(x)

MA 214 - NA Spring 2023-24 38 / 42

Likewise for functions defined on the real line, the function f is said to
be Big Oh of g as x → ∞


Notation: f(x) = O g(x)

if there exists a real number C and a natural number N such that

|f(x)| ≤ C |g(x)| whenever x> N

MA 214 - NA Spring 2023-24 38 / 42

Analogously, function f is said to be Little Oh (or Small oh) of g as
x → ∞,

MA 214 - NA Spring 2023-24 39 / 42

Analogously, function f is said to be Little Oh (or Small oh) of g as
x → ∞,

Notation: f(x) = o g(x)

MA 214 - NA Spring 2023-24 39 / 42

Analogously, function f is said to be Little Oh (or Small oh) of g as
x → ∞,

Notation: f(x) = o g(x)

if for every ϵ > 0 there exists a natural number N such that

|f(x)| ≤ ϵ |g(x)| whenever x > N

MA 214 - NA Spring 2023-24 39 / 42

Example 1: For two monic polynomials f(x) and g(x), we have
f(x) = O(g(x)) as x → ∞ whenever deg f(x) ≤ deg g(x). Also
f(x) = o g(x)) as x → ∞ if and only if degf(x) < deg g(x).

MA 214 - NA Spring 2023-24 40 / 42

Example 1: For two monic polynomials f(x) and g(x), we have
f(x) = O(g(x)) as x → ∞ whenever deg f(x) ≤ deg g(x). Also
f(x) = o g(x)) as x → ∞ if and only if degf(x) < deg g(x).

Example 2: For any polynomial p(x), we have p(x) = o ex ) as x → ∞.

MA 214 - NA Spring 2023-24 40 / 42

Example 1: For two monic polynomials f(x) and g(x), we have
f(x) = O(g(x)) as x → ∞ whenever deg f(x) ≤ deg g(x). Also
f(x) = o g(x)) as x → ∞ if and only if degf(x) < deg g(x).

Example 2: For any polynomial p(x), we have p(x) = o ex ) as x → ∞.

A related example :

Example If f(x) = e−1/x on the positive real line, then f(x) = o xn ) as

x → 0 for any natural number n.

MA 214 - NA Spring 2023-24 40 / 42

 x2 x3
Since ex = 1 + x + 2! + 3! + · · · , we have :

ex = 1 + x + O(x2 ) as x → 0 and
x2
ex = 1 + x + 2! + O(x3 ) as x → 0.

MA 214 - NA Spring 2023-24 41 / 42

Interesting fact

Let π(x) be the prime-counting function defined to be the number of

primes less than or equal to x, for any real number x. The famous prime
number theorem in Number Theory states that π(x) x → 1 as x → ∞.
ln(x)

x
In particular, this shows that π(x) = O( ln(x) ) as x → ∞.

MA 214 - NA Spring 2023-24 42 / 42