Lecture 5

Fixed point Methods

Course Website
https://sites.google.com/view/kporwal/teaching/mtl107
Fixed point iteration

The method discussed now have direct extensions to more

complicated problems, e.g., to systems of nonlinear equations and
to more functional equations.

Problem f (x) = 0 can be rewritten as

x = g (x). (⇤)
(There are many ways to do this.)
Given (*) we are looking for a fixed point, i.e., a point x ⇤ satisfying
g (x ⇤ ) = x ⇤
Algorithm:Fixed point iteration

Given a scalar function f (x). Select a function g (x) such that

f (x) = 0 () g (x) = x.

Then:
I Start from an initial guess x0 .
I For k =0,1,2,..... set

-0
x = g (x ),
k+1 k k = 0, 1, .....

until xk+1 satisfies some termination criterion

Ku }
→ ✗
*

84 )
="g( Lian )
It = k
" un =

E. •
✗
*
= gkÑ* )
Examples of fixed point iterations

Note: there are many ways to transform f(x)=0 into fixed point
form! Not all them ”good” in terms of convergence.
Options for fixed point iterations for

-
f (x)a
= xe x 1, x 2 [0, 1]

Di↵erent fixed point forms:

x
9,101)
-
.

x
" g1 (x) = e ,

☐é*
÷÷:*
"
* "
f- ""
" 1+x
¥ g2 (x) =
1 + ex
,
x
.

g3 (x) = x + 1 xe . (d)
= 0

f.
?⃝
 Numerical Methods for Computational Science and Engineering

Examples of fixed point iteration (cont.)

Fixed point iteration

Examples of fixed point iterations (cont.)

k xk+1 := g1 (xk ) xk+1 := g2 (xk ) xk+1 := g3 (xk )
0 0.500000000000000 0.500000000000000 0.500000000000000
1 0.606530659712633 0.566311003197218 0.675639364649936
2 0.545239211892605 0.567143165034862 0.347812678511202
3 0.579703094878068 0.567143290409781 0.855321409174107
4 0.560064627938902 0.567143290409784 -0.156505955383169
5 0.571172148977215 0.567143290409784 0.977326422747719
6 0.564862946980323 0.567143290409784 -0.619764251895580
7 0.568438047570066 0.567143290409784 0.713713087416146
8 0.566409452746921 0.567143290409784 0.256626649129847
9 0.567559634262242 0.567143290409784 0.924920676910549
10 0.566907212935471 0.567143290409784 -0.407422405542253

Figure 2: Result of di↵erent g

NumCSE, Lecture 3, Sept 26, 2013 18/32
 Numerical Methods for Computational Science and Engineering

Examples of fixed point iteration (const.)

Fixed point iteration

Examples of fixed point iterations (cont.)

k |xk x ⇤ | |xk x ⇤ | |xk x ⇤ |
0 0.067143290409784 0.067143290409784 0.067143290409784
1 0.039387369302849 0.000832287212566 0.108496074240152
2 0.021904078517179 0.000000125374922 0.219330611898582
3 0.012559804468284 0.000000000000003 0.288178118764323
4 0.007078662470882 0.000000000000000 0.723649245792953
5 0.004028858567431 0.000000000000000 0.410183132337935
6 0.002280343429460 0.000000000000000 1.186907542305364
7 0.001294757160282 0.000000000000000 0.146569797006362
8 0.000733837662863 0.000000000000000 0.310516641279937
9 0.000416343852458 0.000000000000000 0.357777386500765
10 0.000236077474313 0.000000000000000 0.974565695952037

Figure 3: Error with di↵erent g

NumCSE, Lecture 3, Sept 26, 2013 19/32

Some questions regarding the fixed point iteration

Suppose that we have somehow determined the continuous

function g 2 C [a, b]. Now let us consider the fixed point
iterationxk+1 = g (xk ). Obvious questions arise:
I Is there a fixed point x ⇤ in [a,b] ?
I If yes, is it unique ?

⇐ I Does the sequaence of iterates converge to a root x ⇤ ?.

I If yes, how fast ?
I If not, does this mean that no root exists ?
Fixed point theorem 9 :[a. b) → Cab]
If g 2 C [a, b] and a  g (x)  b for all x 2 [a, b], then there is a
- -

fixed point x ⇤ in the interval [a,b]. ⑤ ←④

A-
L
LL

÷ :÷÷
f. Ca ) 30
hlb)
.

.→ =*¥
so

x-D
gi 1 →

If, in addition, the derivatives g 0 exists and there is a constant

⇢ < 1 such that the derivatives 181<1
0
|g
a- (x)|  ⇢ 8x 2 (a, b),
then the fixed point x ⇤ is unique in this interval.
012 %
two fixed point

fl%-%'_I__I_#_I#
are
Assume
"'D -9%115
1%-0121=18
No contradiction
Convergence of the fixed point iteration

Does the sequence of iterates converge to a root x ⇤ ?

|xk+1 x ⇤ | = |g (xk ) g (x ⇤ )| = |g 0 (⇠)| · |xk x ⇤ |  ⇢|xk x ⇤|

with ⇠ 2 [xk , x ⇤ ].
This is a contraction if the factor ⇢ < 1. Thus,
-
9080k

÷
|xk+1
-
x ⇤ |  ⇢|xk x ⇤ |  ⇢2 |xk 1
¥
x ⇤ |  ...  ⇢k+1 |x0 x ⇤ |.

Since ⇢ < 1 then ⇢k ! 0 as k ! 1. G.b)

01, y t

) ) E f 1$ ☒
< III.
181*1--818
-

④ ④
 Numerical Methods for Computational Science and Engineering
Fixed point iteration
Convergence of fixed point iteration in 1D

Convergence of fixed point iterations in 1D

Vastly di↵erent behavior of di↵erent fixed point iterations:

g1 : linear convergence? g2 : quadratic convergence? g3 : no convergence?

 Geometric
eometric interpretationofoffixed
interpretation fixed point
point iteration
iteration
x0 x0 : start with startx0with
on the
x0 axis
on the x -axis
F (x
F (x0 ) 0 ): gogo parallel the
parallel to y-axis
to the to thetograph
y -axis of F ⌘ of
the graph g F ⌘g
x = F (x ) move parallel to the x-axis to
x1 =1 F (x0 )0 move parallel to the x -axis to the graph the graph y =y x= x
F (xF1(x
) 1 ) go parallel to the
go parallel toy-axis
the yto the to
-axis graph
the of F
graph of F
etc.

E, Lecture 3, Sept 26, 2013 2

Figure 4: Geometric Interpretation of fixed point iterations

Geometric Interpretation: Di↵erent Cases
Numerical Methods for Computational Science and Engineering
Fixed point iteration

IÉKL
left: at least linear convergence; right: divergence
Note: these are local scenarios.

NumCSE, Lecture 3, Sept 26, 2013 25/32

Geometric
Numerical MethodsInterpretation: Di↵erent Cases
for Computational Science and Engineering
Fixed point iteration

left: at least linear

Figureconvergence;
5: Convergenceright: divergence
and Divergence
Rate of convergence
-

Let x ⇤ be a fixed point of the iteration xk+1 = g (xk ) and

⇢ = |g 0 (x ⇤ )| with 0 < ⇢ < 1.
x0 sufficiently close to x ⇤ =) xk x ⇤ ⇡ g 0 (x ⇤ )(xk 1 x ⇤ ).

|xk x ⇤ | ⇡ ⇢|xk 1 x ⇤ | ⇡ .... ⇡ ⇢k |x0 x ⇤ |.

Definition: The rate rate of convergence is defined as rate =

log10 ⇢.

The rate us higher (faster convergence), the smaller ⇢.

About k = [1/rate] iteration steps are needed to reduce the error
by one order of magnitude.
Rate of convergence (Numerical example)

The rate of convergence for xk+1 = gi (xk ) is in the previous

scenarios for solving f (x) = xe x 1, with x 2 [0, 1].

Fixed point is x ⇤ = 0.5671432904.

g10 (x ⇤ ) = g1 (x ⇤ ) = x ⇤ = 0.5671432904.
0 < |g10 (x ⇤ )| < 1 ! convergence.
⇤ ⇤
g30 (x ⇤ ) = 1 x ⇤e x ex = 1.76.... ! divergence.

⇤ x⇤
1 x e f (x )⇤
g20 (x ⇤ ) = (e x ⇤ +1)2 = (e x ⇤ +1)2
= 0,
00 ⇤
g2 (x ) 6= 0 ! quadratic convergence (see next time: order of
convergence)
 Termination criteria
Termination criteria
Residual based termination: STOP convergent iteration {xk }, if
Residual based termination: STOP convergent iteration xk , if
|f|f(x
(xkk)|)|  ⌧⌧, ⌧⌧=ˆisprescribed
prescribedtolerance > 0 > 0.
tolerance
=)=)no no
guaranteed
guaranteed accuracy
accuracy
f f

x x

|f (xk )| small 6) |x x⇤ | small |f (xk )| small ) |x x⇤ | small

Figure 6: Convergence and Divergence

NumCSE, Lecture 3, Sept 26, 2013 29/32
Termination criteria (cont.)

Correction based termination: STOP convergent iteration xk , if

8
< ⌧abs
|xk+1 xk |  or (0.1)
:
⌧rel |xk |

⌧abs /⌧rel are prescribed absolute/relative tolerances > 0.

A posteriori termination criterion for linearly convergent iteration

with rate of convergence 0 < ⇢ < 1 :
⇢
|xk+1 x ⇤|  |xk+1 xk |
1 ⇢
Termination criteria (cont.)
Proof
Idea of the proof: Bound |xk+m xk+1 |, m > 1, independent
of m and let m ! 1. For m > 2

|xk+2 xk+1 | = |g (xk+1 ) g (xk )|  ⇢|xk+1 xk |

=
|xk+m
xk+1 | = |xk+m xk+m xk+m 1 + xk+m 1 .......
.. + xk+3 xk+2 + xk+2 | + |xk+2 xk+1 |
 |xk+m xk+m 1| + |xk+m 1 xk+m 2| + ..
..... + |xk+3 xk+2 | + |xk+2 xk+1 |
m 1 m 2
 (⇢ +⇢ + ..... + ⇢)|xk+1 xk |
m 2 m 3
 ⇢(⇢ +⇢ + .... + 1)|xk+1 xk |
1 ⇢ m 1 ⇢
 ⇢ |xk+1 xk | < |xk+1 xk |
1 ⇢ 1 ⇢
Let m ! 1 then xk+m ! x ⇤ (0.2)
Useful results from analysis

I Intermediate value theorem. If f 2 C [a, b] and s is a value

such that f (a)  s  f (b), then there exists a real number
c 2 [a, b] for which f (c) = s.
I Mean value theorem. If f 2 C [a, b] and f is di↵erential on the
open interval (a, b), then there exists a real number c 2 (a, b)
for which f 0 (c) = f (b)b fa(a) .
I Rolle’s theorem. If f 2 C [a, b] and f is di↵erentiable on (a, b),
and in addition f (a) = f (b) = 0, then there is a real number
c 2 (a, b) for which f 0 (c) = 0.