See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/221989049

Newton’s method and its use in optimization

Article  in  European Journal of Operational Research · September 2007

DOI: 10.1016/j.ejor.2005.06.076 · Source: DBLP

CITATIONS READS

75 7,782

1 author:

Boris T. Polyak
Institute of Control Sciences
272 PUBLICATIONS   11,162 CITATIONS   

SEE PROFILE

All content following this page was uploaded by Boris T. Polyak on 19 March 2018.

The user has requested enhancement of the downloaded file.

 European Journal of Operational Research 181 (2007) 1086–1096
www.elsevier.com/locate/ejor

Newton’s method and its use in optimization

B.T. Polyak *

Institute for Control Science, Profsojuznaya 65, Moscow 117997, Russia

Received 30 September 2004; accepted 20 June 2005

Available online 25 April 2006

Abstract

Newton’s method is a basic tool in numerical analysis and numerous applications, including operations research and
data mining. We survey the history of the method, its main ideas, convergence results, modifications, its global behavior.
We focus on applications of the method for various classes of optimization problems, such as unconstrained minimization,
equality constrained problems, convex programming and interior point methods. Some extensions (non-smooth problems,
continuous analog, Smale’s results, etc.) are discussed briefly, while some others (e.g., versions of the method to achieve
global convergence) are addressed in more details.
 2006 Elsevier B.V. All rights reserved.

Keywords: Nonlinear programming; Newton’s method; Convergence; Global behavior; Interior-point methods

1. Introduction ern investigations and applications. Our goal is to

Newton’s method is one of the fundamental tools
in numerical analysis, operations research, optimi-
zation and control. It has numerous applications
in management science, industrial and financial
research, data mining. Its role in optimization can-
not be overestimated: the method is the basis for
the most effective procedures in linear and nonlinear
programming. For instance, the polynomial time
interior point algorithms in convex optimization
are based on Newton’s method.
The present paper surveys the key ideas of the
method in the historical perspective as well as mod-
*
Tel.: +7 495 334 8829; fax: +7 495 420 2016.
E-mail address: boris@ipu.rssi.ru
address the most challenging problems in this field
of research and to provide suitable references for a
reader.
The paper is organized as follows. Section 2 pre-
sents the basic idea of Newton’s method and the his-
tory of its development. Main mathematical results
on the convergence are addressed in Section 3. New-
ton’s method in its basic form possesses just local
convergence, its global behavior and modifications
to achieve global convergence are discussed in Sec-
tions 4 and 5. The case of underdetermined systems
is worth of special consideration (Section 6). In its
original form, Newton’s method is destined for solv-
ing equations. However, it can be easily tailored for
unconstrained (Section 7) and constrained (Section
8) optimization. Some extensions of the method
and directions for future research are described in
Section 9.

0377-2217/$ - see front matter  2006 Elsevier B.V. All rights reserved.
doi:10.1016/j.ejor.2005.06.076
 B.T. Polyak / European Journal of Operational Research 181 (2007) 1086–1096
2. Idea and history of the method
The basic idea of Newton’s method is very simple –
it is linearization. Suppose F : R1 ! R1 is a differen-
tiable function, and we are solving the equation
F ðxÞ ¼ 0.
Starting from an initial point x0 we can construct
the linear approximation of F(x) in the neighbor-
hood of x0 : F(x0 + h)  F(x0) + F 0 (x0)h and solve
the arising linear equation F(x0) + F 0 (x0)h = 0.
Thus, we arrive at recurrent method
1
xkþ1 ¼ xk  F 0 ðxk Þ F ðxk Þ; k ¼ 0; 1; . . .
This is the method proposed by Newton in 1669. To
be more precise, Newton dealt with polynomials
only; in the expression of F(x + h) he discarded
higher order terms in h. The method was illustrated
on the example F(x) = x3  2x  5 = 0. The start-
ing approximation for the root was x = 2. Then
F(2 + h) = h3 + 6h2 + 10h  1, neglecting higher
order terms Newton got the linear equation
10h  1 = 0. Thus the next approximation is
x = 2 + 0.1 = 2.1 and the process can be repeated
for this point. Fig. 1 demonstrates that the conver-
gence to the root of F(x) is very fast. It was J. Raph-
son, who proposed in 1690 the general form of
method (2) (i.e., F(x) was not assumed to be a poly-
nomial only and the notion of a derivative was
0.8
0.6
0.4
0.2
X0
0
–0.2
–0.4
–0.6
–0.8
–1
2 2.05
Fig. 1. Newton’s method.
1087
exploited), this is why the method is often called
the Newton–Raphson method.
The progress in development of the method is
linked with such famous mathematicians as Fourier,
Cauchy and others. For instance, Fourier proved in
1818 that the method converges quadratically in the
ð1Þ neighborhood of a root, while Cauchy (1829, 1847)
provided the multidimensional extension of (2) and
used the method to prove the existence of a root of
an equation. Important early contributions to the
investigation of the method are due to Fine [17]
and Bennet [3]; their papers are published in one
volume of Proceedings of National Academy of Sci-
ð2Þ ences USA in 1916. Fine proved the convergence in
the n-dimensional case with no assumption on the
existence of a solution. Bennet extended the result
for the infinite-dimensional case; this is a surprising
attempt because the foundations of the functional
analysis were not created at that moment. In 1948,
L.V. Kantorovich published a seminal paper [25]
where an extension of Newton’s method for func-
tional spaces was provided (Newton–Kantorovich
method). The results were also included in the survey
paper [26]; for further developments of the method
by Kantorovich, see [27–31].
The basic results on Newton’s method and
numerous references can be found in the books by
Ostrowski [47], Ortega and Rheinboldt [48], Kant-
orovich and Akilov [32,33]. A more recent bibliog-
raphy is available in the books [50,60,13], the
F(X)
X2 X1
2.1 2.15
1088 B.T. Polyak / European Journal of Operational Research 181 (2007) 1086–1096
survey paper [67] and on the special website,
devoted to Newton’s method [41].
3. Convergence results
Kantorovich [25] analyzes the same equation as
(1):
F ðxÞ ¼ 0;
but now F : X ! Y, where X, Y are Banach spaces.
The proposed method reads as (2)
1
xkþ1 ¼ xk  F 0 ðxk Þ F ðxk Þ; k ¼ 0; 1; . . . ;
0
where F (xk) is (Frechet) derivative of the nonlinear
operator F(x) at the point xk and F 0 (xk)1 – its in-
verse. The main convergence result from [25] looks
as follows.
Theorem 1. Suppose F is defined and twice continu-
ously differentiable on the ball B = {x : kx  x0k 6 r},
that the linear operator F 0 (x0) is invertible,
kF 0 (x0)1F(x0)k 6 g, kF 0 (x0)1F00 (x)k 6 K, x 2 B
and
pffiffiffiffiffiffiffiffiffiffiffiffiffiffi
1 1  1  2h
h ¼ Kg < ; r P g. ð5Þ
2 h
Then Eq. (3) has a solution x* 2 B, the process (4) is
well defined and converges to x* with a quadratic rate:
g 2k
kxk  x k 6 ð2hÞ .
h2k
The proof of the Theorem is simple enough; the
main novelty of Kantorovich’s contribution were
not technicalities, but the general problem formula-
tion and the use of the appropriate techniques of
functional analysis. Until Kantorovich’s works
[25–27] it was not understood that numerical analy-
sis should be considered in the framework of func-
tional analysis (note the title of [26]: ‘‘Functional
analysis and applied mathematics’’). Another pecu-
liarity of the above theorem is the lack of assump-
tion on the existence of a solution – the theorem is
not only a convergence result for a specific method,
but simultaneously an existence theorem for nonlin-
ear equations.
These properties of the approach ensured a wide
range of applications. Numerous nonlinear prob-
lems – nonlinear integral equations, ordinary and
partial differential equations, variational problems
– could be cast in the framework of (3) and method
(4) could be applied. Various examples of such
applications are presented in the monographs
[32,33]. Moreover, the Newton–Kantorovich
method as a tool for existence results was immedi-
ately used in the classical works of Kolmogorov,
Arnold and Moser (see, e.g., [1]) on KAM-theory
in mechanics. Actually many classical results in
the functional analysis are proved by use of New-
ton-like methods; the typical example is Ljusternik’s
ð3Þ theorem on tangent spaces, see the original paper
[38] or modern research [22].
Later Kantorovich [28,29] obtained another
proof of Theorem 1 and its versions, based on the
ð4Þ method of majorants. The idea is to compare itera-
tions (4) with scalar iterations, which in a sense maj-
orize them and which possess convergence
properties. This approach provides more flexibility;
the original proof is a particular case with a qua-
dratic majorant.
There exist numerous versions of Theorem 1,
which differ in assumptions and results. We mention
just one of them due to Mysovskikh [43].
Theorem 2. Suppose F is defined and twice continu-
ously differentiable on the ball B = {x : kx  x0k 6 r},
the linear operator F 0 (x) is invertible on B and
kF 0 (x)1k 6 b, kF00 (x)k 6 K, x 2 B, kF(x0)k 6 g and
X1  2n 1
h
h ¼ Kb2 g < 2; r P bg . ð7Þ
n¼0
2
ð6Þ Then Eq. (3) has a solution x* 2 B and the process (4)
converges to x* with quadratic rate:
k
bgðh=2Þ2 1
kxk  x k 6 k . ð8Þ
1  ðh=2Þ2
The difference to Theorem 1 is the assumption on
the invertibility of F 0 (x) on B (while in Theorem 1 it
was assumed to be invertible in the initial point x0
only) and the slighter assumption on h (h < 2
instead of h < 1/2). Other versions can be found in
the books [32,33,47,48,35,10].
Newton’s method (4) requires to calculate deriv-
atives F 0 (x) and to solve the linear equation at each
iteration; it can be a pretty hard computational bur-
den. The modified Newton method
1
xkþ1 ¼ xk  F 0 ðx0 Þ F ðxk Þ; k ¼ 0; 1; . . . ð9Þ
avoids this difficulty – the matrix F 0 (x0) is calculated
and inverted at the initial point only. The method
converges under the same conditions as (4), however
the price we pay for simplification is high – method
(9) converges linearly, not quadratically. There are
 B.T. Polyak / European Journal of Operational Research 181 (2007) 1086–1096
some tradeoffs – for instance, one can update the
derivatives after several iterations.
4. Global behavior
The critical condition for convergence is (5) or
(7). They mean that at the initial approximation
x0 the function kF(x0)k should be small enough, that
is x0 should be close to the solution. Thus, Newton’s
method is locally convergent. Very simple one-
dimensional examples demonstrate the lack of the
global convergence even for smooth monotone
F(x). There are many ways to modify the method
to achieve its global convergence (we will discuss
them later), but the problem of interest is the global
behavior of iterations. Obviously there are many
simple situations – say, there is a neighborhood S
of a solution such that x0 2 S implies convergence
to the solution (such a set is called basin of attrac-
tion) while trajectories starting outside Q do not
converge (e.g., tend to infinity). However, in the
case of non-uniqueness of a solution the structure
of basins of attractions may be very complicated
and exhibit the fractal nature. It was Cayley who
formulated this problem as early as 1879.
Let us consider Cayley’s example: solve the equa-
tion z3 = 1 by Newton’s method. Thus we take
F(z) = z3  1 and apply method (2):
z3  1 2zk 1 [
zkþ1 ¼ zk  k 2 ¼ þ 2.
3zk 3 3zk
1.5
0.5
–0.5
–1
1.5
–2
–2 –1.5 –1 –0.5
Fig. 2. Basin of attraction for x* = 1.
1089
It is worth to mention that we have formulated
Newton’s method in real spaces but it is as well
applicable in complex spaces; in the above example
we take zk 2 C. The equation has three roots
pffiffi
z1 ¼ 1; z2;3 ¼ 12  i 23;
and it is natural to expect that the entire plane C is
partitioned into three basins of attraction
S m ¼ fz0 : zk ! zm g; m ¼ 1; 2; 3
located around the corresponding roots. However,
the true picture is much more involved. First, there
is a single point z = 0 where the method is not de-
fined. It has three preimages – points pffiffiffiz0 such that
z1 = 0,pffiffithey
ffi are q, qð1=2  i 3=2Þ, where
q ¼ 1= 3 2. Again, each of them has three preimages
and so on. Thus there are 3k points z0 which are
mapped after k iterations into the point zk = 0,
and they generate a countable set of such initial
points, that the method is not applicable at some
iteration
S 0 ¼ fz0 : zk ¼ 0 for some kg.
It can be proved that for all points z0 62 S0 the meth-
od converges to one of the solutions (note that if
jzkj > 1 then jzk+1j < jzkj) and we have
3
C¼ Sm.
m¼0
0 0.5 1 1.5 2
1090 B.T. Polyak / European Journal of Operational Research 181 (2007) 1086–1096
1.5
0.5
–0.5
–1
–1.5
–2
–2 –1.5 –1
Fig. 3. Points with no convergence.
The sets Sm have a fractal structure: S0 is the bound-
ary of each Sm, m = 1, 2, 3 and in any neighborhood
of any point z 2 S0 there exist points from Sm,
m = 1, 2, 3. The set S1 is displayed in Fig. 2, the
set S0 – in Fig. 3. The sets of initial points which
possess no convergence (like the set S0) for itera-
tions of general rational maps were studied by Julia
[24] and are called now Julia sets. A lot of examples
linked with Newton’s method can be found in
numerous books on fractals [2,39], papers [12,49]
and web materials [23,14] (we mention just few of
the sources available). Some of these examples exhi-
bit much more complicated behavior than for Cay-
ley’s one. For instance, one can meet periodic or
chaotic trajectories of the iterations.
5. Overcoming the local nature of the method
In Cayley’s example Newton’s method converged
for almost all initial points (exceptions were the
countable number of points S0); the complex struc-
ture of basins of attraction was caused by the exis-
tence of several roots. However if (1) has a single
root, the method usually has local convergence
only. For instance, take F(x) = arctan x, which is
smooth, monotone and has the single root x* = 0.
It is easy to check that (2) converges if and only if
jx0j < x* (x* > 0 being a root of 2x = (1 + x2) arc-
tan x), for jx0j = x* we have periodic behavior of
iterations x0 = x1 = x2 = x3 =   , while for
jx0j > x* the iterations diverge: jxkj ! 1.
–0.5 0 0.5 1 1.5
There are several ways to modify the basic New-
ton method to achieve global convergence. The first
one is to introduce a regulated step-size to avoid too
large steps; this is the so called damped Newton’s
method:
1
xkþ1 ¼ xk  ak F 0 ðxk Þ F ðxk Þ; k ¼ 0; 1; . . . ; ð10Þ
where the step-size 0 < ak 6 1 is chosen to achieve
monotone decrease of kF(x)k, that is condition
kF(xk+1)k < kF(xk)k holds. We will discuss the par-
ticular algorithms for choosing ak later for minimi-
zation problems. The main goal in constructing
such algorithms is to preserve the balance between
convergence and rate of convergence, i.e., one
should take ak < 1 when xk is beyond a basin of
attraction of ‘‘pure Newton’s method’’ and switch
to ak = 1 inside this basin.
The second approach is the Levenberg-Marquardt
method [36,40]:
xkþ1 ¼ xk  ðak I þ F 0 ðxk ÞÞ1 F ðxk Þ; k ¼ 0; 1; . . .
ð11Þ
For ak = 0 the method converts into pure Newton’s
method, while for ak  1 it is close to a strongly
damped gradient method, which usually converges
globally. There are various strategies for adjusting
parameters ak, they are described, for instance, in
[48]. Method (11) works even when the pure New-
ton’s method does not – in a situation with the
degenerate operator F 0 (xk). As we shall see, for
 B.T. Polyak / European Journal of Operational Research 181 (2007) 1086–1096
minimization problems method (11) is very promis-
ing, because it does not assume the matrix F 0 (xk) to
be positive definite.
One more strategy is to modify Newton’s method
to prevent large steps; it can be done not by choos-
ing step-sizes as in the damped Newton’s method
(10), but introducing trust regions, where a linear
approximation of F(x) is valid. Such an approach
(originated in [19] and widely developed in [11]) will
be discussed later in the connection with optimiza-
tion problems.
6. Underdetermined systems
In the above analysis of Newton’s method it was
assumed that the linear operator F 0 (xk) is invertible.
However there are situations when this is definitely
wrong. For instance suppose that F : Rn ! Rm,
where m < n. That is we solve an underdetermined
system of m equations with n > m variables; of
course the rectangular matrix F 0 (xk) has no inverse.
Nevertheless an extension of Newton’s method for
this case can be provided, this is due to Graves
[21], who exploited the method to prove existence
of solutions of nonlinear mappings. We present
the result from [51] where emphasis on the method
is done with more accurate estimates.
Theorem 3. Suppose F : X ! Y is defined and dif-
ferentiable on the ball B = {x : kx  x0k 6 r}, its
derivative satisfies a Lipschitz condition on B
kF 0 ðxÞ  F 0 ðzÞk 6 Lkx  zk; x; z 2 B;
0
F (x) maps X onto Y and the following estimate holds:
kF 0 ðxÞ yk P lkyk; 8x 2 B; y 2 Y  ;
with l > 0 (star denotes conjugation). Introduce the
function
X
1 plane provided that the nominal matrix has all
k
H n ðtÞ ¼ t2
k¼n
and suppose that
Ll2 kF ðx0 Þk 2H 0 ðhÞ
h¼ < 1; q¼ 6 r.
2 Ll
Then the method
xkþ1 ¼ xk  y k ; F 0 ðxk Þy k ¼ F ðxk Þ;
1 min f ðxÞ;
ky k k 6 kF ðxk Þk; k ¼ 0; 1; . . .
l
is well defined and converges to a solution x* of the
equation F(x) = 0, kx*  x0k 6 q with the rate
1091
2H k ðhÞ
kxk  x k 6 . ð15Þ
Ll
Thus at each iteration of the method one should
solve the linear equation F 0 (xk)y = F(xk), where the
linear operator F 0 (xk) in general does not have the
inverse; however it maps X onto Y and this equation
has a solution (may be not a single one). Among the
solutions there exists the solution yk with the prop-
erty kykk 6 (1/l)kF(xk)k; this one is exploited in the
method. In the finite dimensional case (X = Rn,
Y = Rm, n > m) such solution is provided by the for-
mula yk = F 0 (xk)+F(xk), where A+ denotes the
pseudoinverse of the matrix A.
We consider an application of Theorem 3 to con-
vex analysis. The following result on convexity of
the nonlinear image of a small ball in Hilbert spaces
holds [52].
Theorem 4. Suppose X, Y are Hilbert spaces, F :
X ! Y is defined and differentiable on the ball B =
{x : kx  ak 6 r}, its derivative satisfies a Lipschitz
condition on B
kF 0 ðxÞ  F 0 ðzÞk 6 Lkx  zk; x; z 2 B;
F 0 (a) maps X onto Y and the following estimate holds:
kF 0 ðaÞ yk P lkyk; 8y 2 Y ; ð16Þ
with l > 0 and r < l/(2L). Then the image of the ball
B under the map F is convex, i.e., the set
S = {F(x) : x 2 B} is a convex set in Y.
This theorem has numerous applications in opti-
mization [52], linear algebra [53], optimal control
[54]. For instance, the pseudospectrum of a n · n
ð12Þ
matrix (a set of all eigenvalues of perturbed matrices
for perturbations bounded in Frobenius norm) hap-
pens to be the union of n convex sets in the complex
eigenvalues distinct and perturbations are small
enough. Another result is the convexity of the reach-
able set of a nonlinear system for L2 bounded con-
trols [54].
ð13Þ
7. Unconstrained optimization
Consider the simplest unconstrained minimiza-
tion problem in a Hilbert space H:
x 2 H. ð17Þ
ð14Þ
Assuming that f is twice differentiable, we can get
Newton’s method for minimization by two different
approaches.
1092 B.T. Polyak / European Journal of Operational Research 181 (2007) 1086–1096
First, the necessary (and sufficient for f convex)
condition for minimization is Fermat’s condition
rf ðxÞ ¼ 0;
that is, we should solve Eq. (3) with F(x) = $f(x).
Applying Newton’s method for this equation we
arrive at Newton’s method for minimization:
2 1
xkþ1 ¼ xk  ðr f ðxk ÞÞ rf ðxk Þ; k ¼ 0; 1; . . . ;
ð18Þ
where $2f denotes the second Frechet derivative (the
Hessian matrix in a finite dimensional case).
Second, we can approximate f(x) in the neighbor-
hood of a point xk by three terms of its Taylor
series:
f ðxk þ hÞ  fk ðhÞ
¼ f ðxk Þ þ ðrf ðxk Þ; hÞ þ 1
2
ðr2 f ðxk Þh; hÞ.
Then, minimizing the quadratic function fk(h) we
get the same method (18). Both these interpretations
would be exploited for constructing Newton meth-
ods for more general optimization problems.
The theorems on convergence of Newton’s
method for equations can be immediately adopted
to the unconstrained minimization case (just replace
F(x) by $f(x) and F 0 (x) by $2f(x)). The main feature
of the method – fast local convergence – remains
unchanged. However, there are some specific prop-
erties. The most important: Newton’s method in
its pure form does not distinguish minima, maxima
and saddle points – when started from a neighbor-
hood of a nonsingular critical point (i.e., a point
x* with $f(x*) = 0, $2f(x*) invertible) the method
converges to it, making no difference between mini-
mum, maximum or saddle points.
There are numerous ways to convert the method
into globally convergent one, they are similar to
modifications discussed in Section 5 (damped New-
ton, Levenberg-Marquardt, trust region). We con-
sider an important version due to Nesterov and
Nemirovski [44], in this version its complexity (the
number of iterations to achieve the desired accu-
racy) can be estimated. Nesterov and Nemirovski
introduce the class of self-concordant functions.
These are three times differentiable convex functions
defined on a convex set D  Rn, satisfying the
property
jr3 f ðxÞðh; h; hÞj 6 2ðr2 f ðxÞh; hÞ3=2 ;
8x 2 D; h 2 Rn .
The above formula includes the third and the sec-
ond derivatives of f and their action on a vector
h 2 Rn. In simpler form it can be presented as the
relation between the third and the second deriva-
tives of the scalar function u(t) = f(x + th):
ju000 ð0Þj 6 2ðu00 ð0ÞÞ
3=2
for all x 2 D, h 2 Rn. For instance, the function
f(x) = ln x, x > 0, x 2 R1 is self-concordant, while
the function f(x) = 1/x, x > 0 is not. For xk 2 D de-
fine Newton’s decrement
qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
dk ¼ rf ðxk ÞT ðr2 f ðxk ÞÞ1 rf ðxk Þ.
Now the Nesterov–Nemirovski version of the
damped Newton’s method reads
xkþ1 ¼ xk  ak ðr2 f ðxk ÞÞ1 rf ðxk Þ; k ¼ 0; 1; . . . ;
ð19Þ
1
ak ¼ 1; if dk 6 ; ð20Þ
4
1 1
ak ¼ ; if dk > . ð21Þ
ð1 þ dk Þ 4
Theorem 5. If f(x) is self-concordant and f(x) P f*,
"x 2 D then for the method (19)–(21) with x0 2 D,
e > 0 one has
f ðxk Þ  f  6 e
for
 
1
k ¼ c1 þ c2 log log þ c3 ðf ðx0 Þ  f  Þ. ð22Þ
e
Here c1, c2, c3 are some absolute constants.
The idea of the proof is simple enough. The min-
imization procedure consists of two stages. At Stage
1, method (19), (21) is applied, and the function
decreases monotonically: f(xk+1) 6 f(xk)  c, where
c is a positive constant. Obviously, this stage termi-
nates after a finite number of iterations (because f(x)
is bounded from below). At stage 2, the pure New-
ton’s method (19), (20) works and it converges with
the quadratic rate: 2dk+1 6 (2dk)2. Note that the
Newton decrement provides a convenient tool to
express the rate of convergence: there are no con-
stants like L, K, g in Theorems 1–3.
For equality (22) simple values of the constants
can be determined. Note that log log(1/e) is not large
even if e is small enough; for all reasonable e the fol-
lowing estimate holds
 B.T. Polyak / European Journal of Operational Research 181 (2007) 1086–1096
k ¼ 5 þ 11ðf ðx0 Þ  f  Þ.
However, numerous simulation results for various
types of optimization problems of different dimen-
sions [8] validate the following empirical formula
for the number of steps for method (19)–(21):
k ¼ 5 þ 0:6ðf ðx0 Þ  f  Þ.
The serious restriction of the above analysis was
the convexity assumption. Recently [45] another
version of Newton’s method has been proposed,
where global convergence and complexity results
were obtained for general smooth functions. Sup-
pose that f 2 C2,1, that is f is twice differentiable
on Rn and its second derivative satisfies a Lipschitz
condition with constant L on the set {x : f(x) 6
f(x0)}. The following version of Newton’s method
is considered in [45]:
xkþ1 ¼ arg min fk ðxÞ; h ¼ x  xk ;
x
fk ðxÞ ¼ f ðxk Þ þ ðrf ðxk Þ; hÞ
1 L
3
þ ðr2 f ðxk Þh; hÞ þ khk :
2 6
Thus at each iteration we solve the unconstrained
minimization problem with the same quadratic term
as in the pure Newton’s method, but regularized via
the cubic term. This problem looks hard and
non-convex, however it can be reduced to one-
dimensional convex optimization (see details in
[45]). Surprisingly, the proposed method has many
advantages if compared with pure Newton. Firstly,
it converges globally, for an arbitrary initial point.
Secondly, it does not converge to maximum points
or saddle points in contrast to Newton’s method.
Thirdly, its complexity for various classes of func-
tions can be estimated. There are implementable
versions of method (23), where the value L is not
known apriori.
8. Constrained optimization
We start with some particular cases of con-
strained optimization problems.
The first one is optimization subject to simple
constraints:
min f ðxÞ; x 2 Q;
where the set Q in a Hilbert space H is ‘‘simple’’ in
the sense that (24) with a quadratic function f can be
solved explicitly. For instance, Q may be a ball, a
linear subspace, etc. The extension of Newton’s
method for this case is based on the second interpre-
1093
tation of the method for unconstrained
minimization:
xkþ1 ¼ arg min fk ðxÞ;
x2Q
fk ðxÞ ¼ ðrf ðxk Þ; x  xk Þ ð25Þ
þ 12ðr2 f ðxk Þðx  xk Þ; ðx  xk ÞÞ:
The method converges under the same assumptions
as for the unconstrained case: if f is convex and
twice differentiable with Lipschitz second deriva-
tives on Q, Q is closed convex, f attains its minimum
on Q in a point x*, $2f(x*) > 0, then sequence (25)
converges locally to x* with a quadratic rate. This
result has been obtained in [37], see more details
and examples in [55,6].
Another simple situation is equality constrained
optimization:
min f ðxÞ; gðxÞ ¼ 0; ð26Þ
1
ð23Þ where f : X ! R , g : X ! Y; X, Y are Hilbert
spaces. If a solution x* of the problem exists and
is a regular point (g 0 (x*) maps X onto Y), then there
exists a Lagrange multiplier y* such that the pair x*,
y* is a stationary point of the Lagrangian
Lðx; yÞ ¼ f ðxÞ þ ðy; gðxÞÞ
that is the solution of the nonlinear equation
L0x ðx; yÞ ¼ 0; L0y ðx; yÞ ¼ 0. ð27Þ
Hence, we can apply Newton’s method for solving
this equation. Under natural assumptions it con-
verges locally to x*, y*, see rigorous results in
[56,6,5]. Various implementations of the method
can be also found in these references.
There are other versions of Newton’s method for
solving (26) which do not involve the dual variables
y. Historically, the first application of the Newton-
like method for finding the largest eigenvalue of a
matrix A = AT by reducing it to the constrained
optimization problem:
2
maxðAx; xÞ; kxk ¼ 1
is due to Rayleigh (1899). Later Kantorovich [27]
has proposed a pure Newton method for the above
problem. Another simple situation, where calcula-
tions can be simplified, is the case of linear con-
ð24Þ
straints g(x) = Cx  d. Then the method is
equivalent to Newton’s method for unconstrained
minimization of the restriction of f to the affine sub-
space Cx = d.
Now we proceed to applications of New-
ton’s method for convex constrained optimization
1094 B.T. Polyak / European Journal of Operational Research 181 (2007) 1086–1096
problems – the area, where the method plays the key
role in constructing the most effective optimization
algorithms. The basic scheme of interior-point meth-
ods looks as follows [44,66]. For the convex optimi-
zation problem
min f ðxÞ; x 2 Q;
with a convex self-concordant f and a convex
Q 2 Rn we construct a self-concordant barrier
F(x), defined on the interior of Q and growing to
infinity when a point approaches the boundary of
Q:
F : int Q ! R1 ; F ðxÞ ! 1 for x ! oQ.
Such barriers exist for numerous examples of the
constraints, for instance, if Q = {x : x P 0}, then
the logarithmic barrier has the desired properties:
Xn
F ðxÞ ¼  ln xi .
i¼1
Using the barriers, we take the function
fk ðxÞ ¼ tk f ðxÞ þ F ðxÞ
depending on the parameter tk > 0. Under natural
assumptions it can be proved that fk(x) has a mini-
mum point xk on int Q and f ðxk Þ ! f  (the minimal
value in (28)) when tk ! 1 (this is so called central
path). However, there is no need to obtain precise
values of xk , it suffices to make one step of the
damped Newton method (19)–(21) and then to vary
tk. There exists such way to adjust the parameters
ak, tk that the method yields polynomial-time com-
plexity, see details and rigorous results in
[44,46,4,8]. The theoretical result on polynomial-
time complexity is very important, however the
practical simulation results on implementation of
interior-point methods are not less important. They
demonstrate very high effectiveness of the method
for a broad spectrum of convex optimization prob-
lems – from linear programming to semi-definite pro-
gramming. For instance, the method is the
successful competitor to the classical simplex meth-
od for linear programs.
Similar ideas are exploited for non-convex con-
strained optimization problems, where so called
sequential quadratic programming (SQP) is the basis
for efficient algorithms, see, e.g., [11,20,7,66,61,34].
9. Some extensions
Newton’s method has numerous extensions, we
consider below just few of them.
• Relaxed smoothness assumptions. Newton’s
method can be applied in many situations where
equations (or functions to be minimized) are not
smooth enough. The simplest example is solving
Eq. (1) with piece-wise linear non-smooth convex
ð28Þ function F. Then, if a solution exists, method (2)
(naturally extended) finds it after a finite number
of iterations.
For the general case there exist numerous non-
smooth versions of Newton’s method, beginning
with the concept of semi-smoothness by Mifflin
[42]. Many researchers have contributed to the
area, see [9,34,57,58,62] and references therein;
the book [34] contains many basic ideas and his-
torical remarks. A related approach is based on
smoothing the equations, consult [18,9].
• Multiple roots. We analyzed Newton’s method in
a neighborhood of a simple root x*. In the case of
a multiple root Newton’s method either remains
convergent, but looses its fast rate of conver-
gence, or diverges. There is a modification of
the method, which preserves quadratic conver-
gence; it is due to Schroder (1870). We provide
it for the one-dimensional case (1):
1
xkþ1 ¼ xk  pF 0 ðxk Þ F ðxk Þ; k ¼ 0; 1; . . . ; ð29Þ
here p is the multiplicity of the root. Unfor-
tunately, we should know in advance this
multiplicity, moreover the situation in the multi-
dimensional case can be much more complicated.
• Higher-order methods. The local rate of conver-
gence of Newton’s method is fast enough; how-
ever, some researchers construct methods with
still higher rate of convergence. This problem
looks a bit artificial (actually very few iterations
of Newton’s method are required to obtain high
precision when we achieve its convergence
domain), so there is no need to accelerate the
method.
• Continuous version. Instead of discrete iterations
in Newton’s method (4) one can apply its contin-
uous analog
1
x_ ðtÞ ¼ F 0 ðxðtÞÞ F ðxðtÞÞ. ð30Þ
The simplest difference approximation of (30)
leads to the damped Newton method (10) with
constant step-size ak = a. As we know, the
damped Newton method can exhibit global con-
vergence, thus one can expect the same property
for the continuous version. Indeed, the global
convergence for (30) has been validated by Smale
[63], and its rate of convergence is analyzed in
 B.T. Polyak / European Journal of Operational Research 181 (2007) 1086–1096
[59]. Of course, the numerical value of continu-
ous methods is arguable, because their implemen-
tation demands their discretization.
• Data at one point. All results on the convergence
of Newton’s method include assumptions, which
are valid in some neighborhood of the solution or
in some prescribed ball. In contrast, Smale [64]
provides a convergence theorem based on data
available at the single initial point. However,
these data include bounds for all derivatives.
• Solving complementarity and equilibrium prob-
lems. There are many works where Newton’s
method is applied to problems, which cannot be
casted into equation solving, the typical examples
are complementarity, variational inequalities and
equilibrium-type problems. For references see,
e.g., [16,34].
• Implementation issues. We are unable to discuss
implementation issues of various versions of
Newton’s method, which are indeed important
for their practical application. Many details can
be found in the recent book [13], while codes of
corresponding algorithms can be downloaded
from [15]. For convex optimization problems
such issues are discussed in [8].
• Complexity. There exist very deep results on the
complexity of basic problems of numerical anal-
ysis (e.g., finding all roots of a polynomial), clo-
sely related to Newton’s method (some
modification of the method is usually proved to
achieve the best possible result). The interested
reader can consult [65].
References
[1] V.I. Arnold, Small denominators and problem of stability in
classical and celestial mechanics, Uspekhi Mathematicheskih
Nauk 18 (6) (1963) 91–192.
[2] M. Barnsley, Fractals Everywhere, Academic Press, London,
1993.
[3] A.A. Bennet, Newton’s method in general analysis, Proceed-
ings of National Academy of Sciences, USA 2 (10) (1916)
592–598.
[4] A. Ben-Tal, A. Nemirovski, Lectures on Modern Convex
Optimization: Analysis, Algorithms and Engineering Appli-
cations, SIAM, Philadelphia, 2001.
[5] D.P. Bertsekas, Constrained Optimization and Lagrange
Multiplier Method, Academic Press, New York, 1982.
[6] D.P. Bertsekas, Nonlinear Programming, Athena Scientific,
Belmont, 1999.
[7] L.T. Biegler, I.E. Grossmann, Part I: Retrospective on
optimization, Part II: Future perspective on optimization,
Preprints, Chemical Engineering Department, Carnegie-
Mellon University, 2004.
1095
[8] S. Boyd, L. Vandenberghe, Convex Optimization, Cam-
bridge University Press, Cambridge, 2004.
[9] X. Chen, Z. Nashed, L. Qi, Smoothing methods and
semismooth methods for nondifferentiable operator equa-
tions, SIAM Journal on Numerical Analysis 38 (2000) 1200–
1216.
[10] L. Collatz, Functional Analysis and Numerical Mathematics,
Academic Press, New York, 1966.
[11] A.B. Conn, N.I.M. Gould, Ph.L. Toint, Trust Region
Methods, SIAM, Philadelphia, 2000.
[12] J.H. Curry, L. Garnett, D. Sullivan, On the iteration of a
rational function: Computer experiments with Newton’s
method, Communications on Mathematical Physics 91
(1983) 267–277.
[13] P. Deulfhard, Newton Methods for Nonlinear Problems:
Affine Invariant and Adaptive Algorithms, Springer, Berlin,
2004.
[14] R.M. Dickau, Newton’s method. Available from: <http://
mathforum.org/advanced/robertd/newtons.html>.
[15] Electronic library ZIB. Available from: <http://www.zib.de/
SciSoft/NewtonLib>.
[16] F. Facchinei, J.S. Pang, Finite-Dimensional Variational
Inequalities and Complementarity Problems, Springer, New
York, 2003.
[17] H. Fine, On Newton’s method of approximation, Proceed-
ings of National Academy of Sciences, USA 2 (9) (1916)
546–552.
[18] M. Fukushima, L. Qi, Reformulation: Nonsmooth, Piece-
wise Smooth, Semismooth and Smoothing Methods, Klu-
wer, Dordrecht, 1999.
[19] S. Goldfeld, R. Quandt, H. Trotter, Maximization by
quadratic hill climbing, Econometrica 34 (1966) 541–551.
[20] P.E. Gill, W. Murray, M.A. Saunders, SNOPT: An SQP
algorithm for large-scale constrained optimization, SIAM
Journal on Optimization 12 (2002) 979–1006.
[21] L.M. Graves, Some mapping theorems, Duke Mathematical
Journal 17 (1950) 111–114.
[22] A.D. Ioffe, On the local surjection property, Nonlinear
Analysis: Theory, Methods and Applications 11 (5) (1987)
565–592.
[23] D.E. Joyce, Newton basins. Available from: <http://aleph0.
clarku.edu/djoyce/newton/newton.html>.
[24] G. Julia, Sur l’iteration des fonctions rationelles, Journal de
Mathematiques Pure et Applique 8 (1918) 47–245.
[25] L.V. Kantorovich, On Newton’s method for functional
equations, Doklady AN SSSR 59 (7) (1948) 1237–1240.
[26] L.V. Kantorovich, Functional analysis and applied mathe-
matics, Uspekhi Mathematicheskih Nauk 3 (1) (1948) 89–
185, Translated as N.B.S. Report 1509, Washington, DC,
1952.
[27] L.V. Kantorovich, On Newton’s method, Trudy Mathemat-
icheskogo Instituta Imeni V.A. Steklova 28 (1949) 104–144.
[28] L.V. Kantorovich, Principle of majorants and Newton’s
method, Doklady AN SSSR 76 (1) (1951) 17–20.
[29] L.V. Kantorovich, Some further applications of principle of
majorants, Doklady AN SSSR 80 (6) (1951) 849–852.
[30] L.V. Kantorovich, On approximate solution of functional
equations, Uspekhi Mathematicheskih Nauk 11 (6) (1956)
99–116.
[31] L.V. Kantorovich, Some further applications of Newton’s
method, Vestnik LGU, Seriya Matematika Mekhanika 0 (7)
(1957) 68–103.
1096 B.T. Polyak / European Journal of Operational Research 181 (2007) 1086–1096
[32] L.V. Kantorovich, G.P. Akilov, Functional Analysis in
Normed Spaces, Fizmatgiz, Moscow, 1959.
[33] L.V. Kantorovich, G.P. Akilov, Functional Analysis,
Nauka, Moscow, 1977.
[34] D. Klatte, B. Kummer, Nonsmooth Equations in Optimiza-
tion: Regularity, Calculus, Methods and Applications, Klu-
wer, Dordrecht, 2002.
[35] M.A. Krasnoselski, G.M. Vainikko, P.P. Zabreiko, Ya.B.
Rutitski, V.Ya. Stetsenko, Approximate Solution of Oper-
ator Equations, Nauka, Moscow, 1969, English translation:
Walter-Noordhoff, Groningen, 1972.
[36] K. Levenberg, A method for the solution of certain
nonlinear problems in least squares, Quarterly of Applied
Mathematics 2 (1944) 164–168.
[37] E.S. Levitin, B.T. Polyak, Constrained minimization meth-
ods, USSR Computational Mathematics and Mathematical
Physics 6 (5) (1966) 1–50.
[38] L.A. Ljusternik, On conditional extrema of functionals,
Mathematicheskij Sbornik 41 (3) (1934) 390–401.
[39] B. Mandelbrot, The Fractal Geometry of Nature, W.H.
Freeman, New York, 1983.
[40] D. Marquardt, An algorithm for least squares estimation of
nonlinear parameters, SIAM Journal of Applied Mathemat-
ics 11 (1963) 431–441.
[41] J.H. Mathews, Bibliography for Newton’s method. Avail-
able from: <http://math.fullerton.edu/mathews/newtons-
method/Newton’sMethodBib/Links/Newton’sMethodBib_
lnk_3.html>.
[42] R. Mifflin, Semismooth and semiconvex functions in con-
strained optimization, SIAM Journal on Control and
Optimization 15 (1977) 959–972.
[43] I.P. Mysovskikh, On convergence of L.V. Kantorovich’s
method for functional equations and its applications, Dok-
lady AN SSSR 70 (4) (1950) 565–568.
[44] Yu. Nesterov, A. Nemirovski, Interior-Point Polynomial
Algorithms in Convex Programming, SIAM, Philadelphia,
1994.
[45] Yu. Nesterov, B. Polyak, Cubic regularization of a Newton
scheme and its global performance, CORE Discussion
Papers 2003, No 41, Mathematical Programming, submitted
for publication.
[46] Yu. Nesterov, Introductory Lectures on Convex Program-
ming, Kluwer, Boston, 2004.
[47] A.M. Ostrowski, Solution of Equations and Systems of
Equations, Academic Press, Basel, 1960.
[48] J.M. Ortega, W.C. Rheinboldt, Iterative Solution of Non-
linear Equations in Several Variables, Academic Press, New
York-London, 1970.
[49] H.O. Peitgen, D. Saupe, F. Haeseler, Cayley’s problem and
Julia sets, Mathematical Intelligencer 6 (2) (1984) 11–20.
View publication stats
[50] H.O. Peitgen (Ed.), Newton’s Method and Dynamical
Systems, Springer, Berlin, 1989.
[51] B.T. Polyak, Gradient methods for solving equations and
inequalities, USSR Computational Mathematics and Math-
ematical Physics 4 (6) (1964) 17–32.
[52] B.T. Polyak, Convexity of nonlinear image of a small ball
with applications to optimization, Set-Valued Analysis 9 (1–
2) (2001) 159–168.
[53] B.T. Polyak, The convexity principle and its applications,
Bulletin of Brazilian Mathematical Society 34 (1) (2003) 59–
75.
[54] B.T. Polyak, Convexity of the reachable set of nonlinear
systems under L2 bounded controls, Dynamics of Continu-
ous, Discrete and Impulsive Systems 11 (2–3) (2004) 255–
268.
[55] B.T. Polyak, Introduction to Optimization, Optimization
Software, New York, 1987.
[56] B.T. Polyak, Iterative methods using Lagrange multipliers
for solving extremal problems with equality-type constraints,
USSR Computational Mathematics and Mathematical Phys-
ics 10 (5) (1970) 42–52.
[57] B.T. Polyak, Minimization of nonsmooth functionals, USSR
Computational Mathematics and Mathematical Physics 9 (3)
(1969) 14–29.
[58] L. Qi, J. Sun, A nonsmooth version of Newton’s method,
Mathematical Programming 58 (1993) 353–367.
[59] A.G. Ramm, Acceleration of convergence: A continuous
analog of the Newton’s method, Applicable Analysis 81 (4)
(2002) 1001–1004.
[60] W.C. Rheinboldt, Methods for Solving Systems of Nonlin-
ear Equations, SIAM, Philadelphia, 1998.
[61] S.M. Robinson, Perturbed Kuhn–Tucker points and rates of
convergence for a class of nonlinear programming algo-
rithms, Mathematical Programming 7 (1974) 1–16.
[62] S.M. Robinson, Newton’s method for a class of nonsmooth
functions, Set-Valued Analysis 2 (1994) 291–305.
[63] S. Smale, A convergent process of price adjustment and
global Newton’s methods, Journal of Mathematical Eco-
nomics 3 (1976) 107–120.
[64] S. Smale, Newton’s method estimates from data at one
point, in: R. Ewing, K. Gross, C. Martin (Eds.), The
Merging of Disciplines: New Directions in Pure, Applied and
Computational Mathematics, Springer, Berlin, 1986.
[65] S. Smale, Complexity theory and numerical analysis, Acta
Numerica 6 (1997) 523–551.
[66] S.J. Wright, Primal–Dual Interior-Point Methods, SIAM,
Philadelphia, 1997.
[67] T.J. Ypma, Historical development of the Newton–Raphson
method, SIAM Review 37 (4) (1995) 531–551.