Optimization Methods: Optimization using Calculus - Equality constraints 1

Module – 2 Lecture Notes – 4

Optimization of Functions of Multiple Variables subject to Equality Constraints

Introduction
In the previous lecture we learnt the optimization of functions of multiple variables studied
for unconstrained optimization. This is done with the aid of the gradient vector and the
Hessian matrix. In this lecture we will learn the optimization of functions of multiple
variables subjected to equality constraints using the method of constrained variation and the
method of Lagrange multipliers.

Constrained optimization
A function of multiple variables, f(x), is to be optimized subject to one or more equality
constraints of many variables. These equality constraints, gj(x), may or may not be linear. The
problem statement is as follows:
Maximize (or minimize) f(X), subject to gj(X) = 0, j = 1, 2, … , m
where
⎧ x1 ⎫
⎪x ⎪
⎪ ⎪
X = ⎨ 2⎬
⎪M⎪
⎩⎪ xn ⎭⎪
with the condition that m ≤ n ; or else if m > n then the problem becomes an over defined one
and there will be no solution. Of the many available methods, the method of constrained
variation and the method using Lagrange multipliers are discussed.

Solution by method of Constrained Variation

For the optimization problem defined above, let us consider a specific case with n = 2 and
m=1 before we proceed to find the necessary and sufficient conditions for a general problem
using Lagrange multipliers. The problem statement is as follows:
Minimize f(x1,x2), subject to g(x1,x2) = 0
For f(x1,x2) to have a minimum at a point X* = [x1*,x2*], a necessary condition is that the
total derivative of f(x1,x2) must be zero at [x1*,x2*].
∂f ∂f
df = dx1 + dx2 = 0 (1)
∂x1 ∂x2

D Nagesh Kumar, IISc, Bangalore M2L4

Optimization Methods: Optimization using Calculus - Equality constraints 2

Since g(x1*,x2*) = 0 at the minimum point, variations dx1 and dx2 about the point [x1*, x2*]
must be admissible variations, i.e. the point lies on the constraint:
g(x1* + dx1 , x2* + dx2) = 0 (2)
assuming dx1 and dx2 are small the Taylor series expansion of this gives us
∂g ∂g
g ( x1 * + dx1 , x2 * + dx2 ) = g ( x1*, x2 *) + (x1*,x 2 *) dx1 + (x1*,x 2 *) dx2 = 0 (3)
∂x1 ∂x2
or
∂g ∂g
dg = dx1 + dx2 = 0 at [x1*,x2*] (4)
∂x1 ∂x2
which is the condition that must be satisfied for all admissible variations.
Assuming ∂g / ∂x2 ≠ 0 (4) can be rewritten as

∂g / ∂x1
dx2 = − ( x1*, x2 *) dx1 (5)
∂g / ∂x2
which indicates that once variation along x1 (d x1) is chosen arbitrarily, the variation along x2
(d x2) is decided automatically to satisfy the condition for the admissible variation.
Substituting equation (5) in (1)we have:

⎛ ∂f ∂g / ∂x1 ∂f ⎞
df = ⎜ − ⎟ dx1 = 0 (6)
⎝ ∂x1 ∂g / ∂x2 ∂x2 ⎠ (x1 *, x 2 *)

The equation on the left hand side is called the constrained variation of f. Equation (5) has to
be satisfied for all dx1, hence we have

⎛ ∂f ∂g ∂f ∂g ⎞
⎜ − ⎟ =0 (7)
⎝ ∂x1 ∂x2 ∂x2 ∂x1 ⎠ (x1 *, x 2 *)

This gives us the necessary condition to have [x1*, x2*] as an extreme point (maximum or
minimum)

Solution by method of Lagrange multipliers

Continuing with the same specific case of the optimization problem with n = 2 and m = 1 we
define a quantity λ , called the Lagrange multiplier as

∂f / ∂x2
λ=− (8)
∂g / ∂x2 (x1 *, x 2 *)

Using this in (6)

D Nagesh Kumar, IISc, Bangalore M2L4

Optimization Methods: Optimization using Calculus - Equality constraints 3

⎛ ∂f ∂g ⎞
⎜ +λ ⎟ =0 (9)
⎝ ∂x1 ∂x1 ⎠ (x *, x *)
1 2

And (8) written as

⎛ ∂f ∂g ⎞
⎜ +λ ⎟ =0 (10)
⎝ ∂x2 ∂x2 ⎠ (x *, x *)
1 2

Also, the constraint equation has to be satisfied at the extreme point

g ( x1 , x2 ) ( x *, x *) = 0 (11)
1 2

Hence equations (9) to (11) represent the necessary conditions for the point [x1*, x2*] to be
an extreme point.
Note that λ could be expressed in terms of ∂g / ∂x1 as well and ∂g / ∂x1 has to be non-zero.

Thus, these necessary conditions require that at least one of the partial derivatives of g(x1, x2)
be non-zero at an extreme point.
The conditions given by equations (9) to (11) can also be generated by constructing a
function L, known as the Lagrangian function, as
L( x1 , x2 , λ ) = f ( x1 , x2 ) + λ g ( x1 , x2 ) (12)

Alternatively, treating L as a function of x1,x2 and λ , the necessary conditions for its
extremum are given by
∂L ∂f ∂g
( x1 , x2 , λ ) = ( x1 , x2 ) + λ ( x1 , x2 ) = 0
∂x1 ∂x1 ∂x1
∂L ∂f ∂g
( x1 , x2 , λ ) = ( x1 , x2 ) + λ ( x1 , x2 ) = 0 (13)
∂x2 ∂x2 ∂x2
∂L
( x1 , x2 , λ ) = g ( x1 , x2 ) = 0
∂λ
The necessary and sufficient conditions for a general problem are discussed next.
Necessary conditions for a general problem
For a general problem with n variables and m equality constraints the problem is defined as
shown earlier
Maximize (or minimize) f(X), subject to gj(X) = 0, j = 1, 2, … , m
⎧ x1 ⎫
⎪x ⎪
⎪ ⎪
where X = ⎨ 2⎬
⎪M⎪
⎩⎪ xn ⎭⎪

D Nagesh Kumar, IISc, Bangalore M2L4

Optimization Methods: Optimization using Calculus - Equality constraints 4

In this case the Lagrange function, L, will have one Lagrange multiplier λ j for each constraint

g j (X) as

L ( x1 , x2 ,..., xn , λ1 , λ2 ,..., λm ) = f ( X) + λ1 g1 ( X) + λ2 g 2 ( X) + ... + λm g m ( X) (14)

L is now a function of n + m unknowns, x1 , x2 ,..., xn , λ1 , λ2 ,..., λm , and the necessary conditions

for the problem defined above are given by

∂L ∂f m ∂g j
= ( X) + ∑ λ j ( X) = 0, i = 1, 2,..., n j = 1, 2,..., m
∂xi ∂xi j =1 ∂xi
(15)
∂L
= g j ( X) = 0, j = 1, 2,..., m
∂λ j

which represent n + m equations in terms of the n + m unknowns, xi and λ j . The solution to

this set of equations gives us

⎧ x1 * ⎫ ⎧ λ1 * ⎫
⎪ x *⎪ ⎪ λ *⎪
⎪ 2 ⎪ ⎪ ⎪
X = ⎨ ⎬ and λ* = ⎨ 2 ⎬ (16)
⎪ M ⎪ ⎪ M ⎪
⎪⎩ xn *⎪⎭ ⎩⎪λm *⎭⎪
The vector X corresponds to the relative constrained minimum of f(X) (subject to the
verification of sufficient conditions).
Sufficient conditions for a general problem
A sufficient condition for f(X) to have a relative minimum at X* is that each root of the
polynomial in ∈ , defined by the following determinant equation be positive.

L11 − ∈ L12 L L1n g11 g 21 L g m1

L21 L22 − ∈ L2 n g12 g 22 gm2
M O M M O M
Ln1 Ln 2 L Lnn − ∈ g1n g 2 n L g mn
=0 (17)
g11 g12 L g1n 0 L L 0
g 21 g 22 g2n M O M
M O M M M
g m1 gm2 L g mn 0 L L 0

D Nagesh Kumar, IISc, Bangalore M2L4

Optimization Methods: Optimization using Calculus - Equality constraints 5

where
∂2 L
Lij = ( X*, λ*), for i = 1, 2,..., n j = 1, 2,..., m
∂xi ∂x j
(18)
∂g p
g pq = ( X*), where p = 1, 2,..., m and q = 1, 2,..., n
∂xq

Similarly, a sufficient condition for f(X) to have a relative maximum at X* is that each root of
the polynomial in ∈ , defined by equation (17) be negative. If equation (17), on solving yields
roots, some of which are positive and others negative, then the point X* is neither a
maximum nor a minimum.

Example
Minimize
f ( X) = −3 x12 − 6 x1 x2 − 5 x22 + 7 x1 + 5 x2

Subject to x1 + x2 = 5

Solution
g1 ( X) = x1 + x2 − 5 = 0

L( x1 , x2 ,..., xn , λ1 , λ2 ,..., λm ) = f ( X) + λ1 g1 ( X) + λ2 g 2 ( X) + ... + λm g m ( X) with n = 2 and m = 1

L = −3 x12 − 6 x1 x2 − 5 x22 + 7 x1 + 5 x2 + λ1 ( x1 + x2 − 5)

∂L
= −6 x1 − 6 x2 + 7 + λ1 = 0
∂x1
1
=> x1 + x2 = (7 + λ1 )
6
1
=> 5 = (7 + λ1 )
6
or λ1 = 23

∂L
= −6 x1 − 10 x2 + 5 + λ1 = 0
∂x2
1
=> 3x1 + 5 x2 = (5 + λ1 )
2
1
=> 3( x1 + x2 ) + 2 x2 = (5 + λ1 )
2

D Nagesh Kumar, IISc, Bangalore M2L4

Optimization Methods: Optimization using Calculus - Equality constraints 6

−1
x2 =
2
11
and, x1 =
2

⎡ −1 11 ⎤
Hence X* = ⎢ , ⎥ ; λ* = [ 23]
⎣ 2 2⎦

⎛ L11 − ∈ L12 g11 ⎞

⎜ ⎟
⎜ L21 L22 − ∈ g 21 ⎟ = 0
⎜ g 0 ⎟⎠
⎝ 11 g12

∂ 2L
L11 = = −6
∂x12 ( X*,λ*)

∂ 2L
L12 = L21 = = −6
∂x1∂x2 ( X*,λ* )

∂ 2L
L22 = = −10
∂x22 ( X*,λ*)

∂g1
g11 = =1
∂x1 ( X*,λ* )

∂g1
g12 = g 21 = =1
∂x2 ( X*,λ* )

The determinant becomes

⎛ −6− ∈ −6 1⎞
⎜ ⎟
⎜ −6 −10− ∈ 1 ⎟ = 0
⎜ 1 0 ⎟⎠
⎝ 1

(−6− ∈)[−1] − (−6)[−1] + 1[−6 + 10+ ∈] = 0

or
=>∈= −2
Since ∈ is negative, X*, λ * correspond to a maximum.

D Nagesh Kumar, IISc, Bangalore M2L4