Optimization Methods: Optimization using Calculus - Unconstrained Optimization

Module 2 Lecture Notes 3

Optimization of Functions of Multiple Variables: Unconstrained Optimization
Introduction
In the previous lectures we learnt how to determine the convexity and concavity of functions
of single and two variables. For functions of single and two variables we also learnt
determining stationary points and examining higher derivatives to check for convexity and
concavity, and tests were recommended to evaluate stationary points as local minima, local
maxima or points of inflection.
In this lecture functions of multiple variables, which are more difficult to be analyzed owing
to the difficulty in graphical representation and tedious calculations involved in mathematical
analysis, will be studied for unconstrained optimization. This is done with the aid of the
gradient vector and the Hessian matrix. Examples are discussed to show the implementation
of the technique.

Unconstrained optimization
If a convex function is to be minimized, the stationary point is the global minimum and
analysis is relatively straightforward as discussed earlier. A similar situation exists for
maximizing a concave variable function. The necessary and sufficient conditions for the
optimization of unconstrained function of several variables are given below.
Necessary condition
In case of multivariable functions a necessary condition for a stationary point of the function
f(X) is that each partial derivative is equal to zero. In other words, each element of the
gradient vector defined below must be equal to zero.
i.e. the gradient vector of f(X), x f at X=X*, defined as follows, must be equal to zero:
f

x ( *)
1

x ( *)
2
= 0
x f =

( *)

dxn

D Nagesh Kumar, IISc, Bangalore

M2L2

Optimization Methods: Optimization using Calculus - Unconstrained Optimization

The proof given for the theorem on necessary condition for single variable optimization can
be easily extended to prove the present condition.

Sufficient condition
For a stationary point X* to be an extreme point, the matrix of second partial derivatives
(Hessian matrix) of f(X) evaluated at X* must be:
(i)

positive definite when X* is a point of relative minimum, and

(ii)

negative definite when X* is a relative maximum point.

Proof (Formulation of the Hessian matrix)

The Taylors theorem with reminder after two terms gives us
n

f ( * + h) = f ( *) + hi
i =1

df
1 n n
2 f
( *) + hi h j
dxi
2! i =1 j =1
xi x j

=*+ h

0< <1
Since X* is a stationary point, the necessary condition gives
df
= 0,
dxi

i = 1,2,,n

Thus

f ( * + h) f ( *) =

1 n n
2 f
h
h
i j x x
2! i =1 j =1
i
j

=*+ h

For a minimization problem the left hand side of the above expression must be positive.
Since the second partial derivative is continuous in the neighborhood of X* the sign of
2 f xi x j = * + h is the same as the sign of 2 f xi x j = * . And hence
f ( * + h) f ( *) will be a relative minimum, if

f ( * + h) f ( *) =

1 n n
2 f
h
h
i j x x
2! i =1 j =1
i
j

=*

is positive. This can also be written in the matrix form as

Q = hT Hh

X=X*

where

H X=X*

2 f
=
xi x j

=*

is the matrix of second partial derivatives and is called the Hessian matrix of f(X).
D Nagesh Kumar, IISc, Bangalore

M2L2

Optimization Methods: Optimization using Calculus - Unconstrained Optimization

Q will be positive for all h if and only if H is positive definite at X=X*. i.e. the sufficient
condition for X* to be a relative minimum is that the Hessian matrix evaluated at the same
point is positive definite, which completes the proof for the minimization case. In a similar
manner, it can be proved that the Hessian matrix will be negative definite if X* is a point of
relative maximum.
A matrix A will be positive definite if all its eigenvalues are positive. i.e. all values of that
satisfy the equation

A I = 0
should be positive. Similarly, the matrix A will be negative definite if its eigenvalues are
negative. When some eigenvalues are positive and some are negative then matrix A is neither
positive definite or negative definite.
When all eigenvalues are negative for all possible values of X, then X* is a global maximum,
and when all eigenvalues are positive for all possible values of X, then X* is a global
minimum.
If some of the eigenvalues of the Hessian at X* are positive and some negative, or if some are
zero, the stationary point, X*, is neither a local maximum nor a local minimum.

Example
Analyze the function f ( x ) = x12 x22 x32 + 2 x1 x2 + 2 x1 x3 + 4 x1 5 x3 + 2 and classify the
stationary points as maxima, minima and points of inflection.
Solution
f

( *)

x1
2 x1 + 2 x2 + 2 x3 + 4 0
f

= 0
x f =
2 x2 + 2 x1
( *) =

x2
2 x + 2 x 5 0
3
1

f

(
*)

x3

Solving these simultaneous equations we get X*=[1/2, 1/2, -2]

2 f
2 f
2 f
=

2;
=

2;
= 2
x12
x2 2
x32
2 f
2 f
=
=2
x1x2 x2 x1

D Nagesh Kumar, IISc, Bangalore

M2L2

Optimization Methods: Optimization using Calculus - Unconstrained Optimization

2 f
2 f
=
=0
x2 x3 x3x2
2 f
2 f
=
=2
x3x1 x1x3
Hessian of f(X) is
2 f
H=

xi x j

2 2 2
H = 2 2 0
2 0 2

+2
I - H = 2
2
or

2
2
+2
0 =0
0
+2

( + 2)( + 2)( + 2) 2( + 2)(2) + 2(2)( + 2) = 0

( + 2)[ 2 + 4 + 4 4 + 4] = 0
( + 2)3 = 0
or 1 = 2 = 3 = 2
Since all eigenvalues are negative the function attains a maximum at the point
X*=[1/2, 1/2, -2]

D Nagesh Kumar, IISc, Bangalore

M2L2