EL6233 System

y Optimization
p
Methods
Z. P. Jiang

ECE Department
D t t Rm LC214
R
NYU-Poly. ext. 3646
eeweb.poly.edu/faculty/jiang
Spring 2008 Z.P. Jiang. No reproduction permitted without written permission of the author.

1/25/2010 ECE Dept at Poly 1

 Course Outline
An introductory course to practical Optimization
Methods

* linear and nonlinear programming

* calculus
l l off variations
i ti

* dynamic
y programming
p g g

* other optimization problems & methods

1/25/2010 ECE Dept at Poly 2

 Features of this Course

• Balance
B l b
between
t th
theory and
d applications
li ti

• Mix of academic examples and puzzles

• Basic homework vs. open-end

p projects
p j

• Independent reading vs. team work

• Numerical algorithms

1/25/2010 ECE Dept at Poly 3

 Motivational Examples of
O i i i
Optimization

! Linear Equations

" Linear Programming

# Least Squares Data Fitting

$ Nonlinear Programming
1/25/2010 ECE Dept at Poly 4
 !Linear Equations
Goal: a first pass to data fitting
Given three points: (2, 1), (3, 6) and (5, 4), find the
corresponding quadratic function

f (t ) ! x1 " x2t " x3t 2

with unknown xi , 1 # i # 3

1/25/2010 ECE Dept at Poly 5

Comment
An optimization problem usually involves
! Objective function

" Variables or unknowns

1/25/2010 ECE Dept at Poly 6

 "Linear Programming
bookshelf ( x1))”,, “cabinets
Manufacturing “bookshelf cabinets with
doors ( x2 )”, “cabinets with drawers ( x3 )”, and
“custom-designed cabinets ( x4 )”.

Objective:
j maximize the weekly
y revenue

1/25/2010 ECE Dept at Poly 7

 Cabinet Wood Labor Revenue

Bookshelf 10 2 100

With Doors 12 2 150

W. Drawers 25 8 200

Custom 20 12 400

Only 5000 units of wood and 1500 units of labor are available.
available
1/25/2010 ECE Dept at Poly 8
 LP Model
Solve the following linear programming problem
with constraints:

maximize P=100x1 " 150 x2 " 200 x3 " 400 x4

subject to 10 x1 " 12 x2 " 25 x3 " 20 x4 # 5000
2 x1 " 4 x2 " 8 x3 " 12 x4 # 1500
x1 , x2 , x3 , x4 $ 0.

1/25/2010 ECE Dept at Poly 9

Comment
Optimization problems often involve
# Constraints (on the unknowns)

1/25/2010 ECE Dept at Poly 10

 #Least Squares Data Fitting
Assume that some, or all, measurements are in
error and that there are more data than the
number of variables.

Goal: minimize the observation errors

%
ei ! mi ' x1 " x2ti " x3ti 2
&
1# i # N, N $ 3
1/25/2010 ECE Dept at Poly 11
 Formulation of the Problem
Mathematically, we want to solve

N N

,e ! , (*m ' % x " x t " x t &)+

2
2 2
minimize i i 1 2 i 3 i
x i !1 i !1

1/25/2010 ECE Dept at Poly 12

Comment
Optimization problems may be
$ Nonlinear (without constraints)

In this case, the problem is referred to as:

Unconstrained Nonlinear Programming

1/25/2010 ECE Dept at Poly 13

 $Nonlinear Programming
Electrical connections:

w1 w2
?

w3 w4

1/25/2010 ECE Dept at Poly 14

 NP Model
4
minimize
i i i P ! , wi
i !1

% xi ' x0 & " % yi ' y0 &

2 2
subject to wi = , 1# i # 4

% x1 '1& " % y1 ' 4& # 4

2 2

% x2 ' 9& " % y2 ' 5& # 1

2 2

2 # x3 # 4, ' 3 # y3 # '1
6 # x4 # 8,
8 ' 2 # y4 # 2 .
1/25/2010 ECE Dept at Poly 15
Comment
Optimization problems may be
% Nonlinear with constraints

In this case, the problem is referred to as:

Constrained Nonlinear Programming

1/25/2010 ECE Dept at Poly 16

 “Blood Diamond” Game
Context: 5 pirates robbed 100 diamonds and
decide to distribute these diamonds by the
rules:
• Draw lots to give each pirate a number out of
1, 2, 3, 4, 5.
• First No.
First, No 1 pirate proposes a distribution plan
for the group of fives to vote. Only when he
gets the majority
g j y of votes,, his p
proposal
p will be
approved.
Otherwise, he will be thrown into the sea.

1/25/2010 ECE Dept at Poly 17

 “Blood Diamond” (cont
(cont’d)
d)
• After No. 1 pirate dies, No.2 proposes his
distribution plan for the group of four to vote.
Only when he gets the majority of votes, his
proposal will be approved.
Otherwise, he too will be thrown into the sea.
• Continue this way for No. 3 and No. 4

1/25/2010 ECE Dept at Poly 18

 “Blood Diamond” (cont
(cont’d)
d)
Hypothesis:
All five pirates are equally smart and can make
their right judgement
judgement.

1/25/2010 ECE Dept at Poly 19

 “Blood Diamond” (cont
(cont’d)
d)
Problem:
What should No.1 pirate propose to maximize his
profit?

Solution at the end of this semester &

1/25/2010 ECE Dept at Poly 20

 Preliminaries

• Basic
B i N Notions
ti off Optimality
O ti lit

• Convexity

• Taylor Series

• The General Optimization

p Algorithm
g

• Rates of Convergence
1/25/2010 ECE Dept at Poly 21
 Basic Notions of Optimality
Consider an optimization problem

minimize f ( x) P ! f ( x) p
performance index
x-S
gi ( x) ! 0, i - I ./
subject
bj t tto 0 constraints
t i t
g j ( x) $ 0, j - J /1
It is worth noting that
maximize f ( x) ! ' minimize ' f ( x)
x-S x-S
1/25/2010 ECE Dept at Poly 22
 Feasibility
• A point satisfying all the constraints is said to
be feasible.
• Feasible set S: all feasible points.
p
• Active constraints: when g j ( x ) ! 0.
• Inactive constraints: when g j ( x ) 2 0.
0
• Boundary: The set of feasible points at which
at least
east one
o e inequality
equa ty iss active.
act e
• Interior points: All other points than boundary

1/25/2010 ECE Dept at Poly 23

 An Example
g1 ( x) ! x1 " 2 x2 " 3 x3 ' 6 ! 0
X3
g 2 ( x) ! x1 $ 0
Ax
g3 ( x) ! x2 $ 0
xC g 4 ( x) ! x3 $ 0.
Bx
X2

Points:
A = (0, 0, 2)
X1 B = (3, 0, 1)
C = (1, 1, 1).
1/25/2010 ECE Dept at Poly 24
 Minimizers
• Global minimizer x3 if f (x3 ) # f (x), 4x - S.
• Strict global minimizer if, additionally,

f (x3 ) 5 f (x), 4x - S, x 6 x3.

• Local minimizer if

f (x3 ) # f (x), 4x - S, such that | x ' x3 |5 7.

• Strict local minimizer if

f (x3 ) 5 f (x), ch that x 6 x3, | x ' x3 |5 7.

) 4x -S, ssuch
1/25/2010 ECE Dept at Poly 25
 Stationary Points
• For unconstrained problems, a stationary point
of f(x) is such that
3 8f 3
f '( x ) ! (x ) ! 0
8x
• A minimizer (or a maximizer) is a stationary
point, but not vice versa.

• Saddle point: a stationary point that is not

minimizer nor maximizer.
1/25/2010 ECE Dept at Poly 26
 Convexity
• Convex set S, if
9x " (1 ' 9) y - S for all 0 # 9 # 1.

• Convex function f, if

f (9x " (1 ' 9) y ) # 9f ( x) " (1 ' 9) f ( y )

4x, y - S and 0 # 9 # 1.

1/25/2010 ECE Dept at Poly 27

 Convex Programming Problem
For a convex set S and a convex function f over S,

min f ( x)
x-S

For example,
example such a set S can be defined via concav
concavee functions gi ,
: ;
S ! x - "n | gi ( x) $ 0 .

Special case of Convex Programming: Linear Programming,

where both the objective
objective and the constraints
constraints are linear.
linear
1/25/2010 ECE Dept at Poly 28
 An Important Result
3
A local minimizer x of a convex programming
problem is a global minimizer.
Moreover, if the cost function is strictly convex,
Moreover convex
3
then x is the unique global minimizer.

Proof. Prove by contradiction. See the book.

1/25/2010 ECE Dept at Poly 29

 Taylor Series
Goal: Approximate a (sufficiently smooth) function
by a polynomial around a point x0 .

1-variable:
1 2 pn (n)
f (x0 " p) ! f (x0 ) " pf '((x0 ) " p f "((x0 ) "#" f (x0 ) "#
2 n!
n-variables:
1 T 2
f (x0 " p) ! f (x0 ) " p <f (x0 ) " p < f (x0 ) p "#
T

2
1/25/2010 ECE Dept at Poly 30
 Notation:
otat o

Gradient <f ( x) :
T
= 8f ( x) 8f ( x) 8f ( x) >
<f ( x ) ! ? , ,# , @ -"
n

A 8x1 8x2 8xn B

Jacobian < f ( x ) :
2

= 8 2
f ( x ) > nC n
< f ( x) ! ?
2
@ -" .
? 8x 8x @
A i j B
1/25/2010 ECE Dept at Poly 31
 Example

f ( x) ! x " 3 x1 x2 ' x
2
1
2
2

Compute its gradient <f ( x) and

its Jacobian < f ( x ) at x ! (1, 1).
2

1/25/2010 ECE Dept at Poly 32

 Necessary Condition
A necessary condition for optimization is:

<f ( x0 ) ! 0.
0

1/25/2010 ECE Dept at Poly 33

 Sufficient Conditions
Sufficient conditions for minimization are:

< f ( x 0 ) ! 0,
0
< f ( x 0 ) 2 0.
2

1/25/2010 ECE Dept at Poly 34

Comment
nC n
A matrix M - " is said to be positive definite,
if 4x - " , x 6 0, it holds
n
o ds that
x Mx 2 0.
T

The eigenvalues of M are all D such that

det % D I ' M & ! 0.

1/25/2010 ECE Dept at Poly 35

 General Optimization Algorithm
Step 0: Specify some initial guess of the solution
x0
Step k (k=0,1,…): If x0 is optimal, STOP.
Otherwise determine an improved estimate of
Otherwise,
the solution: xk "1 ! xk " 9 k pk with step
e gt 9 k a
length andd sea
search d ect o pk
c direction

1/25/2010 ECE Dept at Poly 36

 Rates of Convergence
• Does an algorithm converge?

• How
H ffastt does
d it converge?
?

1/25/2010 ECE Dept at Poly 37

 Sequence : xk ; E x3 with rate r and rate constant C , if
• ek "1
lim r ! C 5 F, where ek $ xk ' x3
k EF
ek

• Linear convergence if r = 1.
1) 0 < C < 1, the sequence converges.
2) C > 1,
1 the sequence diverges.
diverges

• Superlinear convergence if r = 1 and C = 0.

• Quadratic convergence if r = 2.
1/25/2010 ECE Dept at Poly 38
 A Brief Introduction to Lagrange
M l i li M
Multiplier Method
h d
Lagrange multiplier theory essentially targets
at constrained optimization problems. For example,
max P ! f 0 % x & , x - " n
j to the constraint c ! f1 ( x).
subject )

To this end
end, introduce the augmented performance index
Pa ! f 0 % x & " DT % c ' f1 ( x) &
ith D the
with th vector
t off Lagrange
L multipliers.
lti li
1/25/2010 ECE Dept at Poly 39
 A Brief Introduction to Lagrange
M l i li M
Multiplier Method
h d
Then, necessary conditions for optimality are
8Pa
!0
8x
and
c ! f1 ( x).
For more details, wait for
the lectures on "nonlinear
nonlinear programming
programming".
1/25/2010 ECE Dept at Poly 40