1 Newton’s (or Newton-Raphson) Method

Newton’s (or Newton-Raphson) method is one of the most powerful methods

for solving a root-finding problem.

In Newton’s method, it is assumed that f ∈ C 2 [a, b], p0 ∈ [a, b] be an

approximation to p such that f 0 (p0 ) 6= 0 and |p − p0 | is “small.” Consider first
Taylor polynomial at p0 ,

(p − p0 )2 00
f (p) = f (p0 ) + (p − p0 )f 0 (p0 ) + f (ξ (p))
2
where ξ(p) lies between p and p0 . Since f (p) = 0, this equation gives

(p − p0 )2 00
0 = f (p0 ) + (p − p0 )f 0 (p0 ) + f (ξ (p))
2
Since |p − p0 | is small, the term involving (p − p0 )2 is much smaller, so

0 ≈ f (p0 ) + (p − p0 )f 0 (p0 )

Solving for p gives

f (p0 )
p ≈ p0 − ≡ p1
f 0 (p0 )
This sets the stage for Newton’s method, which starts with an initial approxi-
∞
mation p0 and generates the sequence {pn }n=0 , by

f (pn−1 )
pn = pn−1 − for n ≥ 1
f 0 (pn−1 )

NOTE: Newton’s method cannot be continued if f 0 (pn−1 ) = 0 for some

n.

1
 Example: If f (x) = x2 − 6 and p0 = 1.Use Newton’s method to find p3 .

Solution: From the Newton’s formula:

f (p0 )
p1 = p0 −
f 0 (p0 )
f (1)
= 1− 0
f (1)

f 0 (x) = 2x so f 0 (1) = 2. Also, f (1) = −5. Hence, we have

−5 7
p1 = 1 − = = 3.5
2 2
Similarly, we have f 0 ( 72 ) = 7, f ( 72 ) = 25
4 and

f (p1 )
p2 = p1 −
f 0 (p1 )
7 f(7)
= − 0 27
2 f (2)
= 2.6071

Finally,

f (p2 )
p3 = p2 −
f 0 (p2 )
f (2.6071)
= 2.6071 − 0
f (2.6071)
0.7970
= 2.6071 −
5.2142
= 2.4542

√ √
NOTE: Exact solution is x2 − 6 = 0 ⇒ x1 = − 6 and x2 = 6 ≈
2.4494897

1.1 Convergence using Newton’s Method

Theorem: Let f ∈ C 2 [a, b]. If p ∈ (a, b) is such that f (p) = 0 and

0
f (p) 6= 0, then there exists a δ > 0 such that Newton’s method generates a
∞
sequence {pn }n=1 converging to p for any initial approximation p0 ∈ [p−δ, p+δ]
(pn → p, n → ∞) .

2
2 The Secant Method
Newton’s method is an extremely powerful technique, but it has a major weak-
ness: the need to know the value of the derivative of f at each approximation.

Now, we will consider a slight variation. By definition,

f (x) − f (pn−1 )
f 0 (pn−1 ) = lim
x→pn−1 x − pn−1
If pn−2 is close to pn−1 , then

f (pn−2 ) − f (pn−1 ) f (pn−1 ) − f (pn−2 )

f 0 (pn−1 ) ≈ =
pn−2 − pn−1 pn−1 − pn−2

Using this approximation for f 0 (pn−1 ) in Newton’s formula gives

f (pn−1 )(pn−1 − pn−2 )

pn = pn−1 −
f (pn−1 ) − f (pn−2 )
This technique is called the Secant method.

Example: Let f (x) = x2 − 6 and p0 = 3 and p1 = 2. Use Secant method

to find p3 .

Solution: Succeeding approximations are generated by the formula

f (pn−1 )(pn−1 − pn−2 )
pn = pn−1 − for n ≥ 2
f (pn−1 ) − f (pn−2 )
This gives
f (p1 )(p1 − p0 )
p2 = p1 −
f (p1 ) − f (p0 )
Since f (p1 ) = f (2) = −2 and f (p0 ) = f (3) = 3, we have

−2(−1)
p2 = 2 − = 2.4
−5
Similarly, f (p2 ) = f (2.4) = −0.24 and

f (p2 )(p2 − p1 )
p3 = p2 −
f (p2 ) − f (p1 )
(−0.24) (0.4)
= 2.4 −
−0.24 + 2
0.096
= 2.4 +
1.76
= 2.4545

3
3 Error Analysis for Iterative Methods
We will investigate the order of convergence of functional iteration schemes.

3.1 Order of Convergence

∞
Definition: Suppose {pn }n=0 is a sequence that converges to p, with pn 6= p
for all n. If positive constants λ and α exist with

|pn+1 − p|
lim α =λ
n→∞ |pn − p|
∞
then {pn }n=0 converges to p of order α, with asymptotic error constant λ.

In general, a sequence with a high order of convergence converges more

rapidly than a sequence with a lower order.

• If α = 1, the sequence is linearly convergent

• If α = 2, the sequence is quadratically convergent

1
Example: Show that the sequence pn = n2 , n ≥ 1 converges linearly to
p = 0.

Solution:
1
|pn+1 − 0| (n+1)2
lim = lim 1 =1
n→∞ |pn − 0| n→∞
n2
So pn → 0 linearly.

NOTE: Quadratically convergent sequences are expected to converge much

quicker than those that converge only linearly.
∞
Illustration: Suppose that {pn }n=0 is linearly convergent to 0 with

|pn+1 |
lim = 0.5
n→∞ |pn |

∞
and that {epn }n=0 is quadratically convergent to 0 with the same asymptotic
error constant,
|e
pn+1 |
lim 2 = 0.5
n→∞ |e pn |
For simplicity we assume that for each n we have

|pn+1 | |e
pn+1 |
≈ 0.5 and 2 ≈ 0.5
|pn | |e
pn |

4
For the linearly convergent scheme, this means that
2 n
|pn − 0| = |pn | ≈ 0.5 |pn−1 | ≈ (0.5) |pn−2 | ≈ ... ≈ (0.5) |p0 |

whereas the quadratically convergent procedure has

h i2
2 2 3 4
|e
pn − 0| = |e
pn | ≈ 0.5 |e
pn−1 | ≈ (0.5) 0.5 |e pn−2 | = (0.5) |e
pn−2 |
h i4
3 2 7 8
≈ (0.5) 0.5 |e pn−3 | = (0.5) |e
pn−3 |
2n −1 2n
≈ ... ≈ (0.5) |e
p0 |

If |p0 | = |e
p0 | = 1

n 2n −1
n Lineer Convergence: (0.5) → 0 Quadratic Convergence: (0.5) →0
1 0.5 0.5
2 0.25 0.125
3 0.125 0.0078125
4 0.0625 0.000030518
5 0.03125 4.6566x10−10
6 0.015625 1.0842x10−19
7 0.0078125 5.8775x10−39
−38
The quadratically convergent sequence is within 10 of 0 by the seventh
term. At least 126 terms are needed to ensure this accuracy for the linearly
convergent sequence.

Theorem: Let g ∈ C[a, b] be such that g(x) ∈ [a, b], for all x ∈ [a, b].
Suppose, in addition, that g 0 is continuous on (a, b) and a positive constant
k < 1 exists with
|g 0 (x)| ≤ k forall x ∈ (a, b)
If g 0 (p) 6= 0, then for any number p0 6= p in [a, b], the sequence

pn = g(pn−1 ) for n ≥ 1

converges only linearly to the unique fixed point p in [a, b]

Proof : We know from the Fixed-Point Theorem that the sequence con-
verges to p. Since g 0 exists on (a, b), we can apply the Mean Value Theorem to
g to show that for any n,

pn+1 − p = g(pn ) − g(p) = g 0 (ξn ) (pn − p)

where ξn is between pn and p. Since {pn }∞ n=0 converges to p, we also have

{ξn }∞
n=0 converging to p. Since g 0
is continuous on (a, b), we have

lim g 0 (ξn ) = g 0 (p)

n→∞

5
Thus
pn+1 − p |pn+1 − p|
lim = lim g 0 (ξn ) = g 0 (p) and lim = |g 0 (p)|
n→∞ pn − p n→∞ n→∞ |pn − p|

Hence, if g 0 (p) 6= 0, fixed-point iteration exhibits linear convergence with asymp-

totic error constant |g 0 (p)|.

NOTE: Theorem implies that higher-order convergence for fixed-point

methods of the form g(p) = p can occur only when g 0 (p) = 0

Theorem: Let p be a solution of the equation x = g(x). Suppose that

g 0 (p) = 0 and g 00 is continuous with |g 00 (x)| < M on an open interval I contain-
ing p. Then there exists a δ > 0 such that, for p0 ∈ [p − δ, p + δ], the sequence
defined by pn = g(pn−1 ), when n ≥ 1, converges at least quadratically to p.
Moreover, for sufficiently large values of n,
M 2
|pn+1 − p| < |pn − p|
2

NOTE: This theorem describes additional conditions that ensure the

quadratic convergence.

SUMMARY:
(i) For a fixed point method to converge quadratically we need to have both
g(p) = p and g 0 (p) = 0

(ii) If f (p) = 0 and f 0 (p) 6= 0, then for starting values sufficiently close to p,
Newton’s method will converge at least quadratically.

Proof : Consider the sequence

pn = g(pn−1 ) for n ≥ 1

for g in the form

g(x) = x − φ(x)f (x)
where φ is a differentiable function that will be chosen later. It is easy to see
that g(p) = p when f (p) = 0. Because

g 0 (x) = 1 − φ0 (x)f (x) − φ(x)f 0 (x)

and f (p) = 0, we have

g 0 (p) = 1 − φ0 (p)f (p) − φ(p)f 0 (p) = 1 − φ(p)f 0 (p)

Therefore, g 0 (p) = 0 if and only if φ(p) = 1/f 0 (p).

6
 If we let φ(x) = 1/f 0 (x), then we will ensure that φ(p) = 1/f 0 (p) and g is
quadratically convergent. Hence
f (pn−1 )
pn = g(pn−1 ) =pn−1 −
f 0 (pn−1 )
This, of course, is simply Newton’s method.

Example: Let f (x) = ex − x − 1. Show that Newton’s method with p0 = 1

converges to this zero but not quadratically.

Solution: We have

f (x) = ex − x − 1 and f 0 (x) = ex − 1

The first two terms generated by Newton’s method applied to f with p0 = 1 are
f (p0 ) e−2
p1 = p0 − =1− ≈ 0.58198
f 0 (p0 ) e−1
f (p1 ) 0.20760
p2 = p1 − 0
= 0.58198 − ≈ 0.31906
f (p1 ) 0.78957

n pn
0 1
1 0.58198
2 0.31906
3 0.16800
4 0.08635
5 0.04380
6 0.02206
7 0.01107
8 0.005545
9 2.7750x10−3
10 1.3881x10−3
11 6.9411x10−4
12 3.4703x10−4
13 1.7416x10−4
14 8.8041x10−5
15 4.2610x10−5
The sequence is clearly converging to 0, but not quadratically since f 0 (0) = 0.

Definition: The function f ∈ C m [a, b] has a zero of multiplicity m at p

in (a, b) if

f (p) = f 0 (p) = f 00 (p) = ... = f (m−1) (p) = 0, but f (m) (p) 6= 0

7
Example: f (x) = x3 − x4 has a zero of multiplicity 3 at x = 0. Because

f (0) = 0,
f 0 (x) = 3x2 − 4x3 ⇒ f 0 (0) = 0,
f 00 (x) = 6x − 12x2 ⇒ f 00 (0) = 0,
f 000 (x) = 6 − 24x ⇒ f 000 (0) = 6 6= 0.