Theory of constrained
optimization
Nur Aini Masruroh
Scope
Lagrange function
Necessary and sufficient conditions for
equality constraints
Necessary and sufficient conditions for
inequality constraints
Necessary and sufficient conditions for
equality and inequality constraints
A general form for constrained
optimization problem
Minimize f(x)
Subject to hi(x) = 0 i = 1,,m
gj(x) 0 j = 1,,r
xT = [x1 x2 x3 xn] Rn
Question: does the theory of unconstrained
optimization apply for constrained problem?
Lagrange function
In general form
minimize f(x)
subject to hi(x) = 0 i = 1, , m
gj(x) 0 j = 1, , r
x = [x1 x2 xn]T
It can be converted into:
Minimize
m r
L( x, w, u ) f ( x) wi hi ( x) u j g j ( x)
i 1 j 1
L(x, w) is called Lagrange function and wi and uj are
called Lagrange multipliers
Lagrange function: remark
Function L transforms a constrained optimization
into an unconstrained one
Function L can be used to determine the first order
condition (FOC) necessary condition for optimal
solutions
Necessary and sufficient conditions for
equality constraints
In general, a problem with equality constraints can
be expressed as
Minimize f(x)
Subject to hi(x) = 0 i = 1,,m
xT = [x1 x2 x3 xn]
Define Lagrange function as
m
L( x, w) f ( x) wi hi ( x)
i
Necessary conditions of optimality
m
L f ( x) wi hi ( x) 0
i
L
hi ( x) 0 i 1,2,..., m
wi
Remarks:
1. If x* satisfies the conditions, then L(x*, w*) = f(x*) L gives the same
optimal solutions
2. x* is a stationary point for L. It is generally different from the stationary
point for f(x)
3. wi can be positive or negative
4. As necessary conditions require first order derivatives, they are also called
first order conditions (FOC)
Sufficient conditions of optimality
HL(x*,w*) is positive definite x* is a local minimum
HL(x*,w*) is negative definite x* is a local maximum
where HL(x*,w*) is second order derivatives over x and w.
2 h j
f wi hi nn
2
xi nm
H L ( x*, w*) T
j
h
0
x
i mn ( n m )( m n )
i 1,2,..., n
j 1,2,..., m
Note: as sufficient conditions require second order derivatives, they are
also called second order conditions (SOC)
Try the following example
1. Min f(x) = 4x12 + 5x22
Subject to h(x) = 2x1 + 3x2 6 = 0
2. Min f(x1, x2) = 4x12 + 5x22
Subject to h(x1, x2) = 2x1 + 3x2 7 = 0
Necessary and sufficient conditions for
inequality constraints
In general, a problem with inequality constraints can
be expressed as
Minimize f(x)
Subject to gi(x) 0 i = 1,,r
xT = [x1 x2 x3 xn]
What are the necessary conditions for this problem?
To convert inequality constraints into equalities, lets
introduce slack variables Si(i=1,,r) as follows
gi(x) + si2 = 0
Let L f ( x) ui ( g i ( x) si2 )
i
Apply the previous FOC conditions on L:
Lx f ( x) ui g i ( x) 0
i
Ls ui si 0
Lu g i ( x) si2 0
i 1,2,..., r
Relation between ui and gi(x)
Consider gi(x) + si2 = 0
If si=0, then gi(x) = 0 gi(x) is a binding (active) constraint
If si0, then gi(x) < 0 gi(x) is not binding (inactive)
For L(x*, u*, s*) = f(x*) ui[gi(x) + si2] = 0
When si 0,
ui= 0
gi(x) < 0
uigi(x) = 0
When si = 0,
gi(x) = 0
ui 0
Necessary and sufficient conditions of
optimality for general constrained
problems
General problem:
Minimize f(x)
Subject to hi(x) = 0 i = 1,,m
gi(x) 0 i = 1,,r
xT = [x1 x2 x3 xn]
m r
L( x, w, u ) f ( x) wi hi ( x) u j g j ( x)
i 1 j 1
General Karush-Kuhn-Tucker (KKT)
conditions (necessary)
1. Linear dependence of f, h, g
m r
f ( x) wi hi ( x) u j g j ( x) 0
i 1 j 1
2. Complementary conditions
uigj(x) = 0 j=1,,r
uj 0
3. Constraint feasibility
hi(x) = 0 i=1,,m
gj(x) 0 j=1,,r
Sufficient Conditions of Optimality
(SOC)
HL(x*, w*, u*) is positive definite x* is a
local minimum
HL(x*, w*, u*) is negative definite x* is a
local maximum
Where HL(x*, w*, u*) is Hessian matrix for L
with respect to x*, w*, and u*
Sufficient conditions for the uniqueness
of a local optimum
If f(x) is convex (concave) and the feasible
region is convex and a local minimum
(maximum) exists at x*,
1. x* is global minimum (maximum)
2. The KKT conditions are both necessary and
sufficient
The procedure for solving KKT equations
1. Assume no active inequality uj = 0
2. Solve the equations for x, the multiplier wi of
the equalities, and the multiplier uj of the
active inequality (for the first iteration there
are none)
3. If it satisfies all the constraints, STOP,
solution found. Otherwise go to step 4.
4. Set uj 0, then go to step 2.
Example
Minimize f = x12 + x22
Subject to g1 = 1 x1 0
g2 = x1 3 0
g3 = 2 x2 0
g4 = x2 4 0
