Taller de Análisis Numérico

Convergence
Heetae Kim

Ingeniería Civil Industrial

Universidad de Talca

10 de Septiembre de 2018

Heetae Kim, Ingeniería Civil Industrial Taller de Análisis Numérico

Error Analysis for Iterative Methods

• The order of convergence of the functional iteration scheme.

• Rediscover Newton’s Method to obtain more rapid convergence.

• Accelerate the convergence of Newton’s Method in special circumstances.

Heetae Kim, Ingeniería Civil Industrial Taller de Análisis Numérico

Oder of Convergence

{p }
∞
• Suppose n n=0
is a sequence that converges to p, with pn ≠ p for all n.

• If positive constants λ and α exist with

pn+1 − p
lim α
= λ,
n→∞
pn − p

{p }
∞
then n n=0
converges to p of order α, with asymptotic error constant λ.

Heetae Kim, Ingeniería Civil Industrial Taller de Análisis Numérico

Oder of Convergence

pn+1 − p
lim α
= λ,
n→∞
pn − p

• In general, the sequence with a high order of convergence converges more

rapidly than the one with a lower order.

• Special order cases:

‣ If α = 1 (λ < 1), the sequence is linearly convergent.
‣ If 1 < α < 2 , the sequence is super linearly convergent.
‣ If α = 2, the sequence is quadratically convergent.
‣ If 2 < α < 3, the sequence is super quadratically convergent.
‣ If α = 3, the sequence is cubically convergent.

Heetae Kim, Ingeniería Civil Industrial Taller de Análisis Numérico

 Ejemplo

• {p } • {p }
∞ ∞
n n=0
is linearly convergent to 0 n n=0
is quadratic convergent to 0

pn+1 pn+1
with lim = 0.5 with lim 2
= 0.5
x→∞ pn x→∞
pn
2
pn ≈ 0.5 pn−1 pn ≈ 0.5 pn−1
≈ 0.5 0.5 pn−2 = (0.5)2 pn−2 2
2
4
≈ 0.5 0.5 pn−2 = (0.5) pn−2
3
Solutions of Equations in One
2 Variable
≈ 0.5 0.5 pn−3 = (0.5)3 pn−2 4
2 8
p0 speed of convergence of the sequences to≈00.5 | = 1. = (0.5) pn−3
3 7
≈ (0.5)
Table 2.7 illustrates the nrelative if | p 00.5 p0n−3
| = |p̃
n 2n
e 2.7 ≈ (0.5)2 −1 p0
Linear Convergence Quadratic Convergence
Sequence { p n }∞
n=0 Sequence { p̃ n }∞
n=0
n 2n −1
n (0.5) (0.5)
1 5.0000 × 10−1 5.0000 × 10−1
2 2.5000 × 10−1 1.2500 × 10−1
3 1.2500 × 10−1 7.8125 × 10−3
4 6.2500 × 10−2 3.0518 × 10−5
5 3.1250 × 10−2 4.6566 × 10−10
6 1.5625 × 10−2 1.0842 × 10−19
7 7.8125 × 10−3 5.8775 × 10−39 ( Cuando p0 = p0 = 1 )

Heetae Kim, Ingeniería Civil Industrial Taller de Análisis Numérico

 In which case the Fixed point converges linearly or quadratically?

Heetae Kim, Ingeniería Civil Industrial Taller de Análisis Numérico

Convergence of Fixed Point

pn+1 − p
lim
• x→∞ p − p
n

‣ pn+1 − p = g(pn) − g(p) = g′(ξn)(pn − p)

g(p)

g(pn)
where ξn is between pn and p.

‣ Since a b
pn+1 p pn
- converges to p, and converges to p,

- g′ is continuous on (a, b)

- therefore,

• Fixed-point methods of the form g(p) = p converges:

‣ If g′(p) ≠ 0, linear convergence with λ = |g′(p)|.

‣ If g′(p) = 0, higher-order convergence can occur.

Heetae Kim, Ingeniería Civil Industrial Taller de Análisis Numérico

Convergence of Fixed Point

pn+1 − p
lim
• x→∞ p − p
n

‣ pn+1 − p = g(pn) − g(p) = g′(ξn)(pn − p)

g(p)

g(pn)
where ξn is between pn and p.

‣ Since
pn+1 p pn
- converges to p, and converges to p,

- g′ is continuous on (a, b)

- therefore,

• Fixed-point methods of the form g(p) = p converges:

‣ If g′(p) ≠ 0, linear convergence with λ = |g′(p)|.

‣ If g′(p) = 0, higher-order convergence can occur.

Heetae Kim, Ingeniería Civil Industrial Taller de Análisis Numérico

Convergence of Fixed Point

pn+1 − p
lim y=x
• x→∞ p − p
n

‣ pn+1 − p = g(pn) − g(p) = g′(ξn)(pn − p)

g(p)

g(pn)
where ξn is between pn and p.

‣ Since
pn+1 p pn
- converges to p, and converges to p,

- g′ is continuous on (a, b)

- therefore,

• Fixed-point methods of the form g(p) = p converges:

‣ If g′(p) ≠ 0, linear convergence with λ = |g′(p)|.

‣ If g′(p) = 0, higher-order convergence can occur.

Heetae Kim, Ingeniería Civil Industrial Taller de Análisis Numérico

Convergence of Fixed Point

pn+1 − p
lim
• x→∞ p − p
n

‣ pn+1 − p = g(pn) − g(p)

g(p)

g(pn)
where ξn is between pn and p.

‣ Since
pn+1 p pn
- converges to p, and converges to p,

- g′ is continuous on (a, b)

- therefore,

• Fixed-point methods of the form g(p) = p converges:

‣ If g′(p) ≠ 0, linear convergence with λ = |g′(p)|.

‣ If g′(p) = 0, higher-order convergence can occur.

Heetae Kim, Ingeniería Civil Industrial Taller de Análisis Numérico

Convergence of Fixed Point

pn+1 − p
lim
• x→∞ p − p
n

‣ pn+1 − p = g(pn) − g(p) = g′(ξn)(pn − p)

g(p)

g(pn)
where ξn is between pn and p.

‣ Since
pn+1 p ξn pn
- converges to p, and converges to p,

- g′ is continuous on (a, b)

- therefore,

• Fixed-point methods of the form g(p) = p converges:

‣ If g′(p) ≠ 0, linear convergence with λ = |g′(p)|.

‣ If g′(p) = 0, higher-order convergence can occur.

Heetae Kim, Ingeniería Civil Industrial Taller de Análisis Numérico

Convergence of Fixed Point

pn+1 − p = lim g′(ξ )

lim
•
n
x→∞ p − p x→∞
n

‣ pn+1 − p = g(pn) − g(p) = g′(ξn)(pn − p)

pn+1 − p g(p)
= g′(ξ n )
pn − p
g(pn)
where ξn is between pn and p.

‣ Since
pn+1 p ξn pn
- converges to p, and converges to p,

- g′ is continuous on (a, b)

- therefore,

• Fixed-point methods of the form g(p) = p converges:

‣ If g′(p) ≠ 0, linear convergence with λ = |g′(p)|.

‣ If g′(p) = 0, higher-order convergence can occur.

Heetae Kim, Ingeniería Civil Industrial Taller de Análisis Numérico

Convergence of Fixed Point

pn+1 − p = lim g′(ξ )

lim
•
n
x→∞ p − p x→∞
n

‣ pn+1 − p = g(pn) − g(p) = g′(ξn)(pn − p)

pn+1 − p g(p)
= g′(ξ n )
pn − p
g(pn)
where ξn is between pn and p.

‣ Since a b
pn+1 p ξn pn
- { p } converges to p, and {ξ }
n
∞
n=0 n
∞
n=0
converges to p,

- g′ is continuous on (a, b)
- therefore, lim g′(ξ ) = g′( p)
x→∞ n

• Fixed-point methods of the form g(p) = p converges:

‣ If g′(p) ≠ 0, linear convergence with λ = |g′(p)|.

‣ If g′(p) = 0, higher-order convergence can occur.

Heetae Kim, Ingeniería Civil Industrial Taller de Análisis Numérico

Convergence of Fixed Point

pn+1 − p = lim g′(ξ )

lim = g′( p)
•
n
x→∞ p − p x→∞
n

‣ pn+1 − p = g(pn) − g(p) = g′(ξn)(pn − p)

pn+1 − p g(p)
= g′(ξ n )
pn − p
g(pn)
where ξn is between pn and p.

‣ Since a ξn b
pn+1 p pn
- { p } converges to p, and {ξ }
n
∞
n=0 n
∞
n=0
converges to p,

- g′ is continuous on (a, b)
- therefore, lim g′(ξ ) = g′( p)
x→∞ n

• Fixed-point methods of the form g(p) = p converges:

‣ If g′(p) ≠ 0, linear convergence with λ = |g′(p)|.

‣ If g′(p) = 0, higher-order convergence can occur.

Heetae Kim, Ingeniería Civil Industrial Taller de Análisis Numérico

Convergence of Fixed Point

pn+1 − p
lim = g′( p) = λ
x→∞ p − p
n

• Fixed-point methods of the form g(p) = p converges:

‣ If g′(p) ≠ 0, linear convergence with λ = |g′(p)|.

‣ If g′(p) = 0, higher-order convergence can occur.

Heetae Kim, Ingeniería Civil Industrial Taller de Análisis Numérico

Linear Convergence of Fixed Point

The condition for the existence of

the unique fixed point

• Conditions:
‣ g is continuous on C[a, b]
‣ g(x) ∈ [a, b], for all x ∈ [a, b].
‣ g′ is continuous on (a, b)
‣ a positive constant 0 < k < 1 exists with |g′(x)| ≤ k, for all x ∈ (a, b).

• If g′(p) ≠ 0 for any number p0 ≠ p in [a, b],

‣ then, the sequence pn = g(pn−1), for n ≥ 1, converges only linearly to the
unique fixed point p in [a, b]

• The unique fixed point iteration converges linearly!

Heetae Kim, Ingeniería Civil Industrial Taller de Análisis Numérico

Quadratic Convergence of Fixed Point

• Suppose:
‣ Fixed point condition satisfied.
‣ p is the solution of g(x)=x (p is the fixed point, g(p)=p).
‣ g′(p) = 0.
‣ g′′ is continuous on an interval containing p with | g′′(x) | < M.

• Then,
‣ there exist δ > 0 for p0 ∈ [p-δ, p+δ] and
‣ the sequence of pn = g(pn-1) for n ≧ 1 converges to p at least quadratically.

• Moreover for large n,

M
‣ pn+1 − p < pn − p 2

Heetae Kim, Ingeniería Civil Industrial Taller de Análisis Numérico

Quadratic Convergence of Fixed Point

• Expanding g(x) in a linear Taylor polynomial for x ∈ [p-δ, p+δ] gives,

where ξ lies between x and p,
g′′(ξ )
g(x) = g( p) + g′( p)(x − p) + (x − p)2 .
2

g′′(ξ )
• Since g(p) = p and g′(p) = 0, g(x) = p + (x − p)2
2

Heetae Kim, Ingeniería Civil Industrial Taller de Análisis Numérico

Quadratic Convergence of Fixed Point

• Expanding g(x) in a linear Taylor polynomial for x ∈ [p-δ, p+δ] gives,

where ξ lies between x and p,
g′′(ξ )
g(x) = g( p) + g′( p)(x − p) + (x − p)2 .
2

g′′(ξ )
• Since g(p) = p and g′(p) = 0, g(x) = p + (x − p)2
2
g′′(ξ n )
• When x = pn with ξn∈[pn, p] g( pn ) = p +
2
( pn − p)2

Heetae Kim, Ingeniería Civil Industrial Taller de Análisis Numérico

Quadratic Convergence of Fixed Point

• Expanding g(x) in a linear Taylor polynomial for x ∈ [p-δ, p+δ] gives,

where ξ lies between x and p,
g′′(ξ )
g(x) = g( p) + g′( p)(x − p) + (x − p)2 .
2

g′′(ξ )
• Since g(p) = p and g′(p) = 0, g(x) = p + (x − p)2
2
g′′(ξ n )
• When x = pn with ξn∈[pn, p] g( pn ) = p +
2
( pn − p)2

• And since pn+1 = g(pn) g′′(ξ n )

pn+1 = p + ( pn − p)2
(fixed point), 2

Heetae Kim, Ingeniería Civil Industrial Taller de Análisis Numérico

Quadratic Convergence of Fixed Point

• Expanding g(x) in a linear Taylor polynomial for x ∈ [p-δ, p+δ] gives,

where ξ lies between x and p,
g′′(ξ )
g(x) = g( p) + g′( p)(x − p) + (x − p)2 .
2

g′′(ξ )
• Since g(p) = p and g′(p) = 0, g(x) = p + (x − p)2
2
g′′(ξ n )
• When x = pn with ξn∈[pn, p] g( pn ) = p +
2
( pn − p)2

• And since pn+1 = g(pn) g′′(ξ n )

pn+1 = p + ( pn − p)2
(fixed point), 2
g′′(ξ n )
• Therefore, pn+1 − p = ( pn − p)2
2
pn+1 − p g′′(ξ n )
=
( pn − p) 2
2

pn+1 − p g′′(ξ )
• Let’s see where lim 2
=
2
go.
x→∞
( pn − p)

Heetae Kim, Ingeniería Civil Industrial Taller de Análisis Numérico

Quadratic Convergence of Fixed Point

• Since |g′(x)| ≤ k < 1 on [ p−δ, p+δ ] and g maps [ p−δ, p+δ ] into itself,
‣ from the Fixed-Point Theorem
{ }
∞
→ p converges to p.
n n=0

‣ ξn ∈ [ p, pn ] for each n,
{ }
∞
→ ξn also converges to p,
n=0

pn+1 − p g′′(ξ ) g′′( p)

• Therefore, lim = =
x→∞
( pn − p)
2
2 2

{ }
∞
• When g(p) = p and g′(p) = 0, the sequence pn n=0 converges
‣ If g″(p) ≠ 0, quadratically
at least quadratic
‣ If g″(p) = 0, with higher order

For a fixed point method to converge quadratically

we need to have both g(p) = p and g′(p) = 0.

Heetae Kim, Ingeniería Civil Industrial Taller de Análisis Numérico

Convergence of Fixed Point Method

• Suppose:
‣ g is continuous on C[a, b]
‣ g(x) ∈ [a, b], for all x ∈ [a, b].
‣ g′ is continuous on (a, b)
‣ a positive constant k < 1 exists with |g′(x)| ≤ k, for all x ∈ (a, b).
‣ g(p) = p

g(p) = p

No
g′(p) = 0 converges linearly

Yes

No
g″(p) = 0 converges quadratically

Yes

converges with even higher order

Heetae Kim, Ingeniería Civil Industrial Taller de Análisis Numérico

 • Newton’s Method is known to be fast. Why?

Heetae Kim, Ingeniería Civil Industrial Taller de Análisis Numérico

Convergence of Newton’s Method

• A fixed-point problem associated with a root-finding problem f(x)=0.

‣ p is the root, so f(p) = 0
‣ When we have the sequence pn = g( pn−1 ) for n ≧ 1,
with a differentiable function φ , we use
g(x) = x − φ (x) f (x)

• What we need for the quadratic convergence is g′(p) = 0.

• The differentiate of g(x), g′(x) = 1− φ ′(x) f (x) − φ (x) f (x).

g′( p) = 1− φ ′( p) f ( p) − φ ( p) f ′( p)
• When x = p
= 1− φ ′( p) ⋅0 − φ ( p) f ′( p)
= 1− φ ( p) f ′( p)

• To make g′(p) = 0, it should be that φ ( p) = 1/ f ′( p) .

Heetae Kim, Ingeniería Civil Industrial Taller de Análisis Numérico

Convergence of Newton’s Method

pn = g( pn−1 ) g(x) = x − φ (x) f (x) φ ( p) = 1/ f ′( p)

f ( p)
g( p) = p −
f ′( p)

f ( pn−1 )
pn = g( pn−1 ) = pn−1 −
f ′( pn−1 )

f ( pn−1 )
pn = pn−1 − ← simply a Newton’s Method!!!
f ′( pn−1 )

• If f(p) = 0 and f′(p) = 0, then for starting values sufficiently close to p,

Newton’s method will converge at least quadratically.

Heetae Kim, Ingeniería Civil Industrial Taller de Análisis Numérico

Remarks

• Suppose:
‣ g is continuous on C[a, b]
‣ g(x) ∈ [a, b], for all x ∈ [a, b].
‣ g′ is continuous on (a, b)
‣ a positive constant k < 1 exists with |g′(x)| ≤ k, for all x ∈ (a, b).
‣ g(p) = p

• If g′(p) ≠ 0 → converges linearly

• If g′(p) = 0 :
‣ If g″(p) ≠ 0, converges quadratically
‣ If g″(p) = 0, converges with even higher order

• If f(p) = 0 and f′(p) = 0, then for starting values sufficiently close to p,

Newton’s method will converge at least quadratically.

Heetae Kim, Ingeniería Civil Industrial Taller de Análisis Numérico