Lecture 4

Optimization within constraints

1
A simple unconstrained optimization
problem
Minimize the following function:
Solution:
f ( x1 x2)  025x12  x22 df  0 and df  0
dx1 dx2
or
 
 df 
Gradient of f is  dx   
 1   f  0
Defined as   0
 df   
 dx 

 2


Optimum:
X1=0
X2=0
2
 Minimize within a
linear equality constraint
Minimize  f ( x1 x2 )  025x12  x22
Subject to the constraint  ( x1 x2 )  0
Where  ( x1 x2 )  5  x1  x2

3
Defining the optimum mathematically

At the optimum we observe that this equation holds:

f  0
4
Solution using a LaGrange Function
( x1 x2 )  f ( x1 x2)  ( x1 x2)

This problem now has three variables: x1, x2, and 

At the optimum   0
x1
 0
x2
 0


This presents us with the need to solve three

equations in three unknowns.
5
 Numerical solution
LaGrangian Function:
( x1 x2 )  025x12  x22   (5  x1  x2 )

Solution is found by finding solution

to these three equations:
  05x    0
x1 1
Solution:
  2x    0
x2 2 x1  4
x2 1
  5 x  x  0
 1 2  2

6
Additional equality constraints
Minimize  f ( x1  x2 )
Subjectto  1 ( x1  x2 )  0
 2 ( x1  x2 )  0
 3 ( x1  x2 )  0
 f ( x1 x2 )  11 ( x1 x2 )  22 ( x1 x2 )  33 ( x1 x2 )
f  11  22  33  0
 
0 0
Solution x1 x2
  
0 0 0
1 2 3
7
 Inequality constraints
This region above and to the right of the line is not allowed
because it violates the constraint
We call this the “infeasible region”

G(x1, x2)<=0

This region below and to the left of the line is allowed

because it does not violate the constraint
We call this region the “feasible region”

Part of the x1,x2 plane is divided into the “feasible region” and the remaining
Is the “infeasible region” the solution can be on the constraint line or anywhere
Within the feasible region – but not in the infeasible region.

8
Optimization gets harder when we add
“inequality” constraints
Minimize  f (x)
Subject to   i (x)  0 i  1 2… N 
gi (x)  0 i  1 2… Ng
x  vector of real numbers dimension  N

By adding the less than or equal to constraint “g” we must follow more
elaborate procedures than simply finding where the gradient of the
Lagrangian is equal to zero.

9
The Karush-Kuhn-Tucker Conditions
N Ng
(x    )  f (x)   ii (x)   i gi (x)
i 1 i 1

The conditions for the optimum point x0   0   0 are


1. (x0   0   0 )  0 for i  1…N All partial derivatives equal zero
xi
2. i (x0 )  0 for i  1…N Restatement of constraint condition
3. gi (x0 )  0 for i  1…Ng
Restatement of constraint condition
 gi ( x )  0 
0 0
4. i
 for i  1…Ng Complementary slackness condition
i0  0 

10
Complementary slackness condition
applies independently to each
inequality constraint
i0  0
i0 gi (x0 )  0 Either: or

i  0 
0
gi ( x 0 )  0
or
both are true

i0  0 means that gi ( x 0 ) is non-binding

gi ( x 0 )  0 the constraint is binding
By “binding” we mean that the solution point is on the constraint, non-binding
means it is off the constraints itself but within the feasible space.

11
 Sample problem with an
inequality constraint
Minimize  f ( x1  x2 )  025 x12  x22
Subject to   ( x1  x2 )  5  x1  x2  0
g ( x1  x2 )  x1  02 x2  3  0

12
Solution to the sample with inequality
constraint
The Lagrangian for this problem is:
 f ( x1  x2 )  [ ( x1  x2 )]  [ g ( x1  x2 )]
 025 x12  x22   (5  x1  x2 )   ( x1  02 x2  3)

 05 x1      0
First condition of KKT gives: x1

 2 x2    02   0
x2

The second KKT condition gives

5  x1  x2  0

13
 Solution continued
The third KKT condition gives: x1  02 x2  3  0

The fourth condition gives:

 ( x1  02 x2  3)  0
 0

We now know how to test a solution for being optimum

but we still do not have a means of actually getting to the optimum.
In this exsample we simply test the fourth condition for both
Possibilities and the proper optimum should be apparent.

14
 Solution continued
If   0 , then by conditions 1 and 2
x1  4
x2  1
2
but
g ( x1  x2 ) x14  4  02(1)  3  12  0
x2 1

If   0 , then by conditions 2 and 3

x1  25
x2  25
by condition 1
  59375
  46875
and
g ( x1 x2 ) x1  x2 25  25  02(25)  3  0
All conditions are met.
To find the solution we had to test each possibility

15
16