1

(Introduction)

1.1 Optimization
optimization

.. 1940 optimization least square

newton method .. 1940 .. 1950
optimization (linear programming)
optimization
, ,

optimization
optimization
optimization
optimization

optimization modern
optimization method (nontraditional)
genetic algorithm, simulated annealing, particle swarm optimization, ant colony optimization,

EEE603 BP 1
neural network-based optimization fuzzy optimization modern method genetic algorithm
..1975 John Holland modern method simulated annealing
Kirkpatrick, Gelatt Vecchi particle swarm optimization ..1995
Kennedy Eberhart ant colony optimization ..1992 Marco Dorigo neural
network method optimization
neural network ..1985 Hopfield Tank optimization fuzzy
..1986 Rao

1.2 Optimization
1)
2)
3)
4)
5)
6)
7) pump turbine
8)
9)
10) plant
11)
12)

13) pipeline
14)
15)
16)
17)
18)

1.3 Optimization
optimization
optimization (quantity)
optimization
(mathematical model) optimization
(maximum) (minimum) (objective function)
(constraints)

EEE603 BP 2

optimization objective function constraints

1.3.1 (Design Variable)

(design variable)
, , ,
, ,

1.1 ( P )
P / A
( ) ( P ) ( A )

1.2
( )

1.3.2 (Objective Function)

(objective function)

maximum minimum Objective
function

J f (x) (1.1)

x x1 , x2 ,, xm T dimension m optimization
minimization problem
objective function

1.3 Objective function (time minimization)

tf

minimize J dt
t0
(1.2)

EEE603 BP 3
 t 0 t f

1.4 Objective function (energy minimization)

1/ 2
tf m 2
minimize J u i dt (1.3)
t i 1
0
u u1 , u 2 , , u m
T

1.5 Objective function (fuel minimization)

tf m

minimize J u (t ) dt
t 0 i 1
i (1.4)

u u1 , u 2 , , u m T

1.6 Objective function (least square minimization)

tf m tf

minimize J e dt e T edt2
i (1.5)
t 0 i 1 t0

e e1 , e 2 , , e m T

optimization objective function

optimization objective function multicriteria
multigoal optimization

1.7


tf
m

minimize J W1 u i (t ) W2 dt (1.6)
t0 i 1
objective function
(weighted value) W1
W2

1.6

EEE603 BP 4
1.3.3 (Constraints)
Constraints objective function constraints 2
external constraints internal constraints external constraints
material A material B
material A
material A external constraint
internal constraints internal constraint
material constraints
U min u (t ) U max (1.7)

1.8 Bang-Bang problem constraints 2 U min U max

to t1
20km/hr 100 km/hr objective function constraint

J minimize(t1 t 0 ) u (t )
20 km/hr
u
100 km/hr

constraints objective function constraints

objective function
optimization objective function constraints
constraints 2 equality constraints
(=) inequality constraints ( , )
constraints unconstrained problem
unconstrained problem constrained problem 4 5

1.3.4 (Quadratic Function)

optimization non-linear objective function

(linear
function)
(quadratic function) quadratic function (1.8)

EEE603 BP 5
 f ( x) x T Ax (1.8)
A (real symmetric matrix) x
(complex number) (1.8) Hermitian form
(1.9)

f ( x) x * Ax (1.9)
A Hermitian matrix x * conjugate
transpose A quadratic form hermitian form

Quadratic Form (Real) Hermitian Form (Complex)

a11 a12 a1n a11 a12 a1n
a a *
A 12 A 12

*
a1n ann a1n ann
1.1 quadratic form hermitian form

aij* complex conjugate aij quadratic form

A positive definite matrix positive definite matrix
A positive definite x T Ax 0 x
positive definite matrix 3
1) K1 K 2 positive definite matrix K 1 K 2 positive definite matrix
2) positive definite matrix K AT A positive definite matrix positive semidefinite
matrix
3) pivot eigenvalue positive definite matrix

quadratic form ( eigenvalue)

o x T Ax 0 x A Positive definite matrix (eigenvalue
A 0)
o x T Ax 0 x A Positive semi-definite matrix
(eigenvalue A 0)
o x T Ax 0 x A Negative definite matrix (eigenvalue
A 0)
o x T Ax 0 x A Negative semi-definite matrix
(eigenvalue A 0)
EEE603 BP 6
 o x T Ax 0 x T Ax 0 x A Indefinite matrix
1.9 Quadratic Form eigenvalue
1) Positive definite matrix
2 1
A
1 2

1 2
x x T Ax 0 x eigenvalue
3 4
A 1 3 eigenvalue 0
positive definite matrix

2) Positive semi-definite matrix

1 1
A
1 1
1 1
x x T Ax 0 x eigenvalue
1 1
A 0 2 eigenvalue 0
positive semi-definite matrix

3) Negative definite matrix

2 1
A
1 3
1 2
x x T Ax 0 x eigenvalue
3 4
A -1.4 -3.6 eigenvalue 0
negative definite matrix

4) Negative semi-definite matrix

1 1
A
1 1
1 1
x x T Ax 0 x eigenvalue
1 1
A 0 -2 eigenvalue 0
negative semi-definite matrix

5) Indefinite matrix
1 1
A
1 2

EEE603 BP 7
 1 1
x x T Ax 0 x x
1 1
2 2
x T Ax 0 x eigenvalue 0.6
1
A
1
-1.6 eigenvalue 0
indefinite matrix

1.10 positive definite matrix

1) u T Bu u12 2u1u 2 u 22 (u1 u 2 ) 2
1 1
B positive definite matrix B B
1 1
eigenvalue 0 2 B positive semi-definite matrix
u1 u2 (u1 u2 ) 2 quadratic function u T Bu 0 B
positive semi-definite matrix

2) u T Ku 2u12 2u1u 2 2u 22 u12 (u 2 u1 ) 2 2u 22

2 1
K positive definite matrix B B eigenvalue
1 2
1 3 B positive definite matrix u
( u ) u12 (u2 u1 ) 2 2u22 u T Ku 0

K positive definite matrix

3) u T Mu u12 6u1u2 u22 (u1 3u2 ) 2 8u22

1 3
M positive definite B B eigenvalue
3 1
-2 4 u u T Mu 0 u1 8 3 ,
u2 1

1.11 V ( x) 10 x12 4 x22 x32 2 x1 x2 2 x2 x3 4 x1 x3 quadratic

form quadratic form positive definite

V ( x) xT Px
10 1 2 x1
x1 x 2 x3 1 4 1 x 2
2 1 1 x3
eigenvalues P 0.4037, 3.9581 10.6382
quadratic form
V (x) positivie definite
EEE603 BP 8
 quadratic function (1.10)
(Gradient) quadratic from (1.11) (Hessian matrix)
(1.12)
1 T
f ( x) x Ax bT x c (1.10)
2
f ( x) Ax b (1.11)
2 f ( x) 2 f ( x)

x 2
1 x1x n
H ( x) 2 f ( x) A (1.12)
2
f ( x ) 2
f ( x)
x x
x n2
n 1
nonlinear taylor series expansion x *
(1.13)
1
f ( x ) f ( x * ) f ( x * )( x x * ) ( x x * ) T H ( x * )( x x * ) (1.13)
2!
x * first order term
(linear) second order term quadratic high order term
1.2

1.2 linearization taylor series expansion

EEE603 BP 9
1.4 Concave Functions Convex Functions
convex function

f (x1 (1 ) x2 ) f ( x1 ) (1 ) f ( x2 ) (1.14)
x1 x 2 x1 , x2 R 0 1
convex function hessian matrix f (x) convex function
hessian matrix f (x) positive definite positive semi-definite hessian matrix f (x)
negative definite negative semi-definite
concave function

Convex function Concave function

f (x)

f (x)

1.3: Convex function Concave function

Convex function
1)
2) eigenvalue hessian matrix 2 0
3) Hessian matrix positive-definite/semidefinite x

Concave function
1)
2) eigenvalue hessian matrix 2 0
3) Hessian matrix negative-definite/semidefinite x

1.5 Quadratic Convex Function Concave Function

f ( x1 , x 2 , x3 ) 3 x12 2 x 22 x32 2 x1 x 2 2 x1 x3 2 x 2 x3 6 x1 4 x 2 2 x3

EEE603 BP 10
Solution
6 x1 2 x2 2 x3 6

f ( x1 , x2 , x3 ) 4 x2 2 x1 2 x3 4
2x 2x 2x 2
3 1 2
6 2 2
f ( x1 , x2 , x3 ) 2 4 2
2 2 2
hessian matrix
1)
2) diagonal hessian matrix
3) determinant Hessian matrix
f 16 0

positive definite matrix f convex function

minimum point

1.5 (Duality)
Duality duality
Maximization Minimization
duality Linear Programming
duality
Maximize cT x
Ax b and x 0
dual Minimize yT b
yT A cT and y 0

x1 x2 xn
y1 a11 a12 a1n b1
y2 a21 a22 a2 n b2

ym am1 am 2 amn bm
c1 c2 cn

EEE603 BP 11
 1.6 x1 x2 objective function x1 x2 x1 0 , x2 0
Constraints x1 2 x2 4

4 x1 2 x2 12
x1 x2 1

Solution dual form

x1 x2
y1 1 2 4
y2 4 2 12
y3 1 1 1
1 1

x1 2 x2 4 column 1
x1 column 2 x 2 column

4 x1 2 x2 12 column 4 x1
column 2 x 2 column 12

x1 x2 1 column -1 x1
column 1 x 2 column 1

objective function x1 x2 column 1

x1 column 1 x 2 column cost
/

1.7 Duality
Maximize Z x1 2 x2 3 x3 4 x4
Subject to x1 2 x2 2 x3 3x4 20
2 x1 x2 3x3 2 x4 20
x1 , x2 , x3 , x4 0
Solution constraints 4 4 dual
Minimize W 20 y1 20 y2
Subject to y1 2 y2 1
2 y1 y2 2
2 y1 3 y2 3
3 y1 2 y2 4
EEE603 BP 12
 y1 , y2 0
x1 , x 2 , x3 , x 4 primal variable primal problem
y1 y2 dual variable dual problem primal problem dual problem

1) objective function primal problem dual
problem primal problem
objective function dual problem
2) primal problem dual problem
primal problem dual problem
3) objective function maximization primal problem minimization
dual problem
4) constraints dual problem primal problem
5) dual problem constraint primal problem
6) Dual dual problem primal problem

1.6 (Optimality)
optimization

Maximization Minimization
maximization problem
mimimization problem (-) objective function
minimum (-) minimization
(optimality)

maximize f ( x) minimize f ( x) (1.16)

1.6.1 Local Maxima/ Local Minima Global Maxima/ Global Minima

Lacal maxima Global minima 1.3
local maximum local maximum
global maximum
global minimum local minimum

1. f (x) local maximum x *

f ( x) f ( x* )
x x x * , 0

EEE603 BP 13
 2. f (x) local minimum x *
f ( x) f ( x* )
x x x * , 0

1.4 minima maxima

1.6.2 Necessary Conditions Sufficient Conditions local minima/ local maxima

f (x) a x b relative minimum x * a x* b
0 f ( x) f ( x * ) 0 x x * f (x)
x *

Taylor Series Expansion f ( x) f ( x * ) 0

1
f ( x ) f ( x * ) f ( x * )( x x * ) ( x x * ) T 2 f ( x * )( x x * )
2!

1
f ( x) f ( x * ) f ( x * )( x x * ) ( x x * ) T 2 f ( x * )( x x * )
2!

EEE603 BP 14
 1
f ( x) f ( x * ) 0 f ( x * )( x x * ) ( x x * ) T 2 f ( x * )( x x * ) 0
2!
2
1. f ( x * ) 0 weak minimum x * (Necessary condition)
x * local minimum f ( x * ) 0
f ( x * ) 0 local minimum x * x *
maximum saddle point

2. 2 f ( x * ) 0 strong minimum x * (Sufficient condition)

x * local minimum 2 f ( x * ) 0
2 f ( x * ) 0 local minimum x *

1. x* relative maximum
2. minimum maximum x*
f ( x* h) f ( x* )
lim m (positive) or m (negative) (1.25)
h 0 h
h m m

1.5

3. minimum maximum (endpoint)

h

EEE603 BP 15
4. minimum maximum
x 0 minimum maximum
stationary point

1.6 x 0

EEE603 BP 16
 2

(Linear Programming)

linear programming optimization objective

function constraints (linear function) constraints
equalities constraints inequalities constraints linear programming optimization
..1930 resources
2 resources
linear programming George B. Dantzig
linear programming simplex method ..1974
linear programming
linear programming
duality linear programming Kuhn
Tucker Charnes Cooper

linear programming simplex method

linear programming
Karmarkars method ..1984 Karmarkars method
simplex algorithm Dantzig

optimization linear programming objective function constraints

(linear functions) linear programming


nonlinear
linear programming
optimization linear programming
nonlinear linearization
linear programming linear programming
nonlinear optimization

EEE603 BP 1
2.1 Linear Programming
linear programming linear
programming

,

,

linear programming
linear programming
linear
programming linear programming

2.2 (Linear Programming Model)

Linear programming (LP) model 2 objective function constraints

2 (maximization
problem) (minimization problem) 2

2.2.1 Maximization problem

Objective function cT x c1 x1 c2 x2 ... cn xn
constraints
a11 x1 a12 x2 a1n xn b1
a21 x1 a22 x2 a2 n xn b2
( Ax b)

am1 x1 am 2 x2 amn xn bn
xi 0, i 1,2,..., n

Maximize cT x
Subject to: Ax b
x0

EEE603 BP 2
2.2.2 Minimization problem
Objective function y T b y1b1 y2b2 ... ymbm
constraints
y1a11 y 2 a 21 y m a m1 c1
y1a12 y 2 a 22 y m a m 2 c 2
( y T A cT )

y1a1n y 2 a 2 n y m a mn cn
yi 0, i 1,2,..., n

Minimize yT b
Subject to: y T A cT
y0

2.3 (Graphical method)

optimization graphical method maximum minimum
constraints
constraints
convex area
objective function convex area optimization graphical method

1) optimal
2) constraint
3) constraint (intersection)
(convex area)
4)
convex area
5) objective function optimal

optimization graphical method objective function 2

(feasible region)

EEE603 BP 3
 2.1 Maximize z 5 x1 4 x 2
constraints:
6 x1 4 x2 24 (1)
x1 2 x2 6 ( 2)
x1 x2 1 (3)
x2 2 ( 4)
x1 0 (5)
x2 0 (6)

Solution graphical method constraint

5 6x +4x =24
1 2

-x +x =1
4 x1=0 1 2
2

3
x

x =2
2
2
C D
E x +2x =6
1 B 1 2

A F x =0
0 2
0 1 2 3 4 5 6
x
1

constraint (1) 6 x1 4 x2 24
6 x1 4 x2 24

constraint (2) x1 2 x2 6
x1 2 x2 6

constraint (3) x1 x2 1
x1 x2 1

EEE603 BP 4
 constraint (4) x2 2
x2 2

constraint (5) x1 0
x1 0

constraint (6) x2 0
x2 0


constraint (intersection) (convex area)
ABCDEF ABCDEF objective function

A co-ordinate (0,0) cost z 5 0 4 0 0

B co-ordinate (0,1) cost z 5 0 4 1 4
C co-ordinate (1,2) cost z 5 1 4 2 13
D co-ordinate (2,2) cost z 5 2 4 2 18
E co-ordinate (3,1.5) cost z 5 3 4 1.5 21
F co-ordinate (4,0) cost z 5 4 4 0 20

x1 3 x2 1.5
E convex area
cost 21

2.2 Minimization z 3 x1 9 x2
constraints:
0.03 x1 0.01x2 0 (1)
0.21x1 0.3 x2 0 (2)
x1 x2 800 (3)
x1 0 (4)
x2 0 (5)

Solution graphical method constraint

EEE603 BP 5
 1500

0.03x -0.01x =0
1 2

1000

A
2
x

0.21x -0.3x =0
500 B 1 2

x +x =800
1 2

0
0 500 1000 1500
x
1

constraint (1) 0.03x1 0.01x2 0

0.03x1 0.01x2 0

constraint (2) 0.21x1 0.3x2 0
0.21x1 0.3x2 0

constraint (3) x1 x2 800
x1 x2 800

constraint (4) x1 0
x1 0

constraint (5) x2 0
x2 0


constraint (intersection)
A B objective function

A co-ordinate (200,600) cost z 0.3 200 0.9 600 6,000

B co-ordinate (470.6,329.4) cost z 4,376.4

EEE603 BP 6
 x1 470.6 x2 329.4 cost
B
4,376.4

2.4. Pivot Operation

Pivot operation minimum problem
maximum problem linear programming pivot operation
matrix
y1a11 yi ai1 ym am1 s1

y1a1 j yi aij ym amj sj (2.1)

y1a1n yi ain ym amn sn

pivot entry aij

y1a11 s j ai1 ym am1 s1

y1a1 j s j aij ym amj yi (2.2)

y1a1n s j ain ym amn sn

pivot (2.1) (2.2) pivot (pivot table)

s1 s j sn s1 yi sn
y1 a11 a1 j a1n y1 a11 a1 j a1n

yi ai1 aij ain sj ai1 aij ain

ym am1 amj amn ym am1 amj amn

EEE603 BP 7
 1
a ij
aij
ahj
a hj for h i
aij
aik
a ik for k j
aij
aik ahj
a hk ahk for k j and h i
aij

pivot entry aij aij

yi s j
1
pivot aij pivot ( aij ) entry
aij
aik
pivot ( a ik
pivot entry ) entry
aij
ahj
pivot ( a hj
pivot () entry ) entry
aij

pivot entry entry
aik ahj
pivot pivot ( a hk ahk
entry )
aij
2.5 inverse matrix A pivot operation

3 1 5
A 2 3 1
0 3 1
Solution

matrix A pivot table

pivot
entry 3 1 1
entry

EEE603 BP 8

pivot entry -11/3 2 2
entry

pivot entry -10/11 3 3

entry

s1 , s2 , s3
y1 , y 2 , y3

inverse matrix

0.6 1.4 1.6

1
A 0.2 0.3 0.7
0.6 0.9 1.1

1.4 Simplex method

Simplex method optimization simplex method
..1947 George Dantzig simplex method objective function
(feasible solution)
simplex method

simplex method 2

EEE603 BP 9
 1. Slack variable
2. Pivot operation

simplex method 2
objective function

2.5.1 Simplex Method Slack Variable

Maximization problem
maximization

x maximum objective function c T x constraint:

Ax b , x 0

constraint
constraint

Ax u b

Solution
x1 x2 xn u1 u2 um
constant
a11 a12 a1n 1 0
0 b1
a 21 a 22 a2n 0 1
0 b2

a m1 am2 a mn 0 0
1 bm
c1 c2 cn 0 0
0 0
Simplex Tableau maximization problem

objective function z c T x z c T x 0

Simplex Tableau maximization problem maximization

problem simplex method slack variable

1. simplex tableau

EEE603 BP 10
 2. Pivot column: column objective function row
pivot column column j
( tie) pivot
3. Pivot row: pivot row
pivot column
pivot row i
entry i
j pivot ( tie)
pivot
4. simplex tableau pivot
entry pivot column

5. objective function row
( slack variable)
3 objective function row

solution constant objective function row objective
function

Minimization problem
minimization

y minimum objective function y T b constraint:

y T A cT , y 0

constraint
constraint

y T A s T c T

Solution
y1 y2 yn s1 s2 sm
constant
a11 a12 a1n 1 0
0 c1
a 21 a 22 a2n 0 1
0 c2

a m1 am2 a mn 0 0
1 cm
b1 b2
bn 0 0
0 0
Simplex Tableau minimization problem

EEE603 BP 11
 objective function z y T b z y T b 0

Simplex Tableau minimization problem minimization problem

simplex method pivot maximization problem
simplex method slack variable

1. simplex tableau
2. Pivot column: column objective function row
pivot column column j
( tie) pivot
3. Pivot row: pivot row
pivot column
pivot row i
entry i
j pivot ( tie)
pivot
4. simplex tableau pivot
entry pivot column
5. objective function row ( slack variable)

3 objective function row

solution constant objective function row objective
function

2.6 slack variable

Maximize the function Z 3x1 5 x 2
constraint : x1 4
2 x2 12
3x1 2 x2 18 , x1 , x2 0
Solution
1. constraints linear equation slack variable u1 , u 2 , u 3
x1 u1 4
2 x2 u2 12
3 x1 2 x 2 u 3 18

2. initial simplex tableau

EEE603 BP 12
 Solution
x1 x2 u1 u2 u3
constant
1 0 1 0 0 4
0 2 0 1 0 12
3 2 0 0 1 18
-3 -5 0 0 0 0
3. maximum pivot column objective function
row -5 pivot element Solution Constant
pivot column 12/2 = 6
pivot
Solution
x1 x2 u1 u2 u3
constant
1 0 1 0 0 4
0 2 0 1 0 12
3 2 0 0 1 18
-3 -5 0 0 0 0

4. simplex tableau pivot

Solution
x1 x2 u1 u2 u3
constant
1 0 1 0 0 4
0 1 0 1/2 0 6
3 0 0 -1 1 6
-3 0 0 5/2 0 30

5. objective function row -3

pivot
Solution
x1 x2 u1 u2 u3
constant
1 0 1 0 0 4
0 1 0 1/2 0 6
3 0 0 -1 1 6
-3 0 0 5/2 0 30

6. simplex tableau pivot

x1 x2 u1 u2 u3 Solution

EEE603 BP 13
 constant
0 0 1 1/3 -1/3 2
0 1 0 1/2 0 6
1 0 0 -1/3 1/3 2
0 0 0 3/2 1 36
7. objective function row
36
x1 2 x2 6

minimum Graphical method

minimum x1 0 x 2 0

2.7 slack variable

Maximize the function Z 2 x1 3x2 x3
constraint : x1 2 x2 x3 6
2 x1 4 x2 x3 12
3 x1 3 x 2 x3 5, x1 , x 2 , x3 0
Solution
1. constraints linear equation slack variable
x1 2 x 2 x3 u1 6
2 x1 4 x 2 x3 u2 12
3 x1 3 x 2 x3 u3 5

2. initial simplex tableau

Solution
x1 x2 x3 u1 u2 u3
constant
1 -2 1 1 0 0 6
2 4 1 0 1 0 12
3 3 -1 0 0 1 5
-2 -3 -1 0 0 0 0

3. pivot column objective function row -3 pivot

element Solution Constant pivot column 5/3 =
1.667

EEE603 BP 14
 Solution
x1 x2 x3 u1 u2 u3
constant
1 -2 1 1 0 0 6
2 4 1 0 1 0 12
3 3 -1 0 0 1 5
-2 -3 -1 0 0 0 0

4. simplex tableau pivot

Solution
x1 x2 x3 u1 u2 u3
constant
3 0 1/3 1 0 2/3 28/3
-2 0 7/3 0 1 -4/3 16/3
1 1 -1/3 0 0 1/3 5/3
1 0 -2 0 0 1 5

5. objective function row -2

pivot
Solution
x1 x2 x3 u1 u2 u3
constant
3 0 1/3 1 0 2/3 28/3
-2 0 7/3 0 1 -4/3 16/3
1 1 -1/3 0 0 1/3 5/3
1 0 -2 0 0 1 5

6. simplex tableau pivot

Solution
x1 x2 x3 u1 u2 u3
constant
23/7 0 0 1 -1/7 6/7 180/21
-6/7 0 1 0 3/7 -4/7 16/7
5/7 1 0 0 1/7 1/7 17/7
-5/7 0 0 0 6/7 -1/7 67/7

7. objective function row -5/7

pivot

EEE603 BP 15
 Solution
x1 x2 x3 u1 u2 u3
constant
23/7 0 0 1 -1/7 6/7 180/21
-6/7 0 1 0 3/7 -4/7 16/7
5/7 1 0 0 1/7 1/7 17/7
-5/7 0 0 0 6/7 -1/7 67/7

8. simplex tableau pivot

Solution
x1 x2 x3 u1 u2 u3
constant
1 0 0 7/23 -1/23 6/23 60/23
0 0 1 6/23 9/23 -8/23 104/23
0 1 0 -5/23 4/23 -1/23 13/23
0 0 0 5/23 19/23 1/23 263/23

9. objective function row

263 / 23 11.435 x1 60 / 23 2.609 , x2 13 / 23 0.565 x3 104 / 23 4.522

2.8 slack variable

Minimize the function z 3 x1 9 x 2
constraint : 0.03x1 0.01x2 0
0.21x1 0.3x2 0
x1 x2 800 , x1 , x2 0
Solution
1. constraints linear equation slack variable
0.03x1 0.01x2 s1 0
0.21x1 0.3x2 s2 0
x1 x 2 s3 800

1. initial simplex tableau

Solution
x1 x2 s1 s2 s3
constant
-0.03 0.01 1 0 0 0

EEE603 BP 16
 -0.21 0.3 0 1 0 0
-1 -1 0 0 1 -800
-3 -9 0 0 0 0

2. pivot column objective function row -3 pivot

element Solution Constant pivot column
tie row 1 2 0 2 row pivot row
pivot entry (1,1)

Solution
x1 x2 s1 s2 s3
constant
-0.03 0.01 1 0 0 0
-0.21 0.3 0 1 0 0
-1 -1 0 0 1 -800
-3 -9 0 0 0 0

3. simplex tableau pivot

Solution
x1 x2 s1 s2 s3
constant
1 -0.33333 -33.3333 0 0 0
0 0.23 -7 1 0 0
0 -1.33333 -33.3333 0 1 -800
0 -10 -100 0 0 0

4. objective function row -10 -100 pivot column

-10 pivot row pivot row tie row 1 row 2
row 2 pivot row
Solution
x1 x2 s1 s2 s3
constant
1 -0.33333 -33.3333 0 0 0
0 0.23 -7 1 0 0
0 -1.33333 -33.3333 0 1 -800
0 -10 -100 0 0 0

EEE603 BP 17
5. simplex tableau pivot
Solution
x1 x2 s1 s2 s3
constant
1 0 -43.4783 1.449275 0 0
0 1 -30.4348 4.347826 0 0
0 0 -73.913 5.797101 1 -800
0 0 -404.348 43.47826 0 0

6. objective function row -404.348

pivot column pivot
row row 1 row 2 tie pivot row 3

pivot entry (3,3)
Solution
x1 x2 s1 s2 s3
constant
1 0 -43.4783 1.449275 0 0
0 1 -30.4348 4.347826 0 0
0 0 -73.913 5.797101 1 -800
0 0 -404.348 43.47826 0 0

7. simplex tableau pivot

Solution
x1 x2 s1 s2 s3 constant
1 0 0 -1.96078 -0.58824 470.5882
0 1 0 1.960784 -0.41176 329.4118
0 0 1 -0.07843 -0.01353 10.82353
0 0 0 11.76471 -5.47059 4376.471

8. objective function row

4,376.471 x1 470.5882 x2 329.4118

2.5.2 Simplex Method Pivot Operation

Simplex Method Slack Variable Simplex
tableau

EEE603 BP 18
Maximization problem
maximization

x maximum objective function c T x constraint:

Ax b , x 0

constraint
constraint

Ax u b u b Ax

x u maximum objective function c T x constraint:

u b Ax , x 0 , u 0

u slack variable constraint slack

variable

u Ax b

objective function z c T x z c T x 0

Simplex Tableau maximization
problem

Simplex Tableau maximization problem maximization

problem simplex method pivot

EEE603 BP 19
 1. simplex tableau ( slack variable s1 , s 2 ,, s m
u1 ,u 2 ,,u m )

x1 x2 ... xn -1
s1 a11 a12 ... a1n b1
s2 a21 a22 ... a2 n b2

sm a m1 am 2 ... a mn bm
c1 c2 ... cn 0
2. Pivot column: objective function column

3. Pivot row: pivot row solution coefficient pivot column

4. Update simplex tableau pivot operation
2 objective function ( ) solution (
)
objective function
5. maximal solution
cost maximal solution

Minimization problem
minimization

y minimum objective function y T b constraint:

y T A cT , y 0

constraint
constraint

y T A c T s T s T y T A c T

EEE603 BP 20
 y s minimum objective function y T b constraint:
s T y T A c T , y 0 , s 0

s slack variable constraint slack

variable

sT yT A cT
-1

objective function z y T b z y T b 0

Simplex Tableau minimization
problem

Simplex Tableau minimization problem minimization problem

simplex method pivot

1. simplex tableau
s1 s2 ... sn -1
y1 a11 a12 ... a1n b1
y2 a21 a22 ... a2 n b2

ym a m1 am 2 ... a mn bm
c1 c2 ... cn 0

2. Pivot row: objective function row

3. Pivot column: pivot column solution coefficient pivot
column

EEE603 BP 21
 4. Update simplex tableau pivot operation
2 objective function ( ) solution (
)
objective function
5. minimal solution
cost minimal solution

2.10 pivot operation

Minimize the function Z 5 x1 4 x2 6 x3 8 x4
Subject to : x1 2 x2 2 x3 4 x4 40
2 x1 x2 x3 2 x4 8
4 x1 2 x2 x3 x4 10 , x1 , x 2 , x3 , x 4 0
Solution
1. constraint minimize function
x1 2 x2 2 x3 4 x4 40
2 x1 x2 x3 2 x4 8
4 x1 2 x2 x3 x4 10

2. simplex tableau
s1 s2 s3 -1
x1 -1 -2 -4 5
x2 -2 1 2 -4
x3 -2 -1 -1 6
x4 -4 -2 1 -8
40 8 10 0

3. objective function column -8 pivot row row 4

pivot column solution coefficient pivot row
8/-2 = -4
pivot entry (4,2)
s1 s2 s3 -1
x1 -1 -2 -4 5
x2 -2 1 2 -4
x3 -2 -1 -1 6

EEE603 BP 22
 x4 -4 -2 1 -8
40 8 10 0

4. second simplex tableau pivot operation pivot

pivot
s1 x4 s3 -1
x1 3 -1 -5 13
x2 -4 0.5 2.5 -8
x3 0 -0.5 -1.5 10
s2 2 -0.5 -0.5 4
24 4 14 -32

5. third simplex tableau pivot operation

x2 x4 s3 -1
x1 0.75 -0.625 -3.125 7
s1 -0.25 -0.125 -0.625 2
x3 0 -0.5 -1.5 10
s2 0.5 -0.25 0.75 0
6 7 29 -80

6.

-80 x1 0 , x2 6,
x3 0 x4 7

2.11 pivot operation

Maximize the function Z 2 x1 x2
Subject to : 2 x1 x2 8
x1 2 x2 14
x1 x2 4 , x1 , x 2 0

EEE603 BP 23
Solution
1. simplex tableau
x1 x2 -1
s1 2 -1 8
s2 1 2 14
s3 -1 1 4
-2 -1 0

2. objective function row -2 solution coefficient

pivot column 8/2 = 4
pivot
x1 x2 -1
s1 2 -1 8
s2 1 2 14
s3 -1 1 4
-2 -1 0

3. second simplex tableau pivot operation pivot

pivot
s1 x2 -1
x1 0.5 -0.5 4
s2 -0.5 2.5 10
s3 0.5 0.5 8
1 -2 8

4. third simplex tableau pivot operation

s1 s2 -1
x1 0.4 0.2 6

EEE603 BP 24
 x2 -0.2 0.4 4
s3 0.6 -0.2 6
0.6 0.8 16

5.

16 x1 6 x2 4

2.6 MATLAB Solution

linear programming MATLAB

linear programming linprog linprog
minimum problem
A x b

min f x T
such that Aeq x beq (2.3)
lb x ub
x

2.12 MATLAB
Minimize the function 3 x1 2 x2 5 x3 , x 0
Subject to : x 2 2 x3 1
x1 x3 1
2 x1 3 x2 7 x3 5 , x1 , x 2 , x3 0
Solution
constraints (2.3)
x 2 2 x3 1
x1 x3 1
2 x1 3 x2 7 x3 5

MATLAB Code
1.
parameter
>>f = [3; -2; 5];
>>A=[0 1 -2; -1 0 -1; -2 3 -7];
>>b=[-1; -1; -5];
>>lb = zeros(size(b));

EEE603 BP 25
2. minimum function linprog initial condition
>>[x,fval] = linprog(f,A,b,[ ],[ ],lb);
3.
x = [0.0000; 0.6667; 1.0000]
fval = 3.6667

x minimize function x1 0, x 2 0.6667, x3 1

3.6667

2.13 MATLAB
Maximize the function x1 x2 2x3 , x 0
Subject to : x 2 2 x3 3
x1 3x3 2
2 x1 x2 x3 1 , x1 , x 2 , x3 0
Solution
MATLAB minimum problem
maximum problem linprog MATLAB
(Optimality) objective function maximum minimum
-1 objective function
x1 x2 2x3
MATLAB Code
1.
parameter
>> f=[-1; -1; -2];
>> A=[0 1 -2; -1 0 3; 2 1 1];
>> b=[3; 2; 1];
>> lb = zeros(size(b));
2. minimum function linprog initial condition
>> [x,fval] = linprog(f,A,b,[ ],[ ],lb);
3.
x = [0.0000; 0.3333; 0.6667]
fval = -1.6667

x maximize function x1 0, x 2 0.3333, x3 0.6667

1.6667 ( -1)

EEE603 BP 26
 3

(Single Variable Optimization)

single variable optimization

2 search
method approximation method single variable optimization 2
combination method search method
search method (unimodal function)
approximation method
(continuously differentiable function) approximation method

3.1 (Unimodal Function)

f (x) boundary L a b p
L unimodal function p
[a,p] [p,b]

3.1: Unimodal function

EEE603 BP 1
3.2 Search Methhod

Search method
objecctive functionn
search methhod siingle variablee 2 Gloden
G sectioon search meethod
Fibonaccci search meethod
(Initial innterval of unccertainty)
2



2


ooptimal
LLobal optimal
GGlobal optimaal
3.2.1 Golden Sectioon Search Meethod
Golden ssection searchh method
minimum maximum unimodal fuunction
golden ssection searchh 1953 Kiefer
golden section
(smaller interval) (larger interrval)

R R1 R2 (3.1)
R

R1
R2

R1 R2

3.2:: goldeen section

R2 R1
golden section (3.2)
R1 R

(3.1) (33.2)
2
R2 R2
1 (3.3)
R1 R1

EEE6603 BP 2
 R2
K (3.3)
R1

K2 K 1 (3.4)

(3.4) K 0.6180
6
goldenn section
seaarch method
K 0.6180
6

I n I 0 (0.6180) n 1 (3.5)
I n
n
I 0

n

3.3
lowerr boundary ( bl )
upper booundary ( bu ) I
I bu bl (3.6)

2
golden sectioon xu 2
xl
2
xl bu KI (3.7)
xu bl KI (3.8)

K 0.6180

f (x)

0.61800I

0.6180I

b1 x1 xu bu
3.3: Goldden section search
s methodd

2 minimization problem
f ( xl ) f ( xu )
xl f ( xu )

EEE6603 BP 3
 (tolerance)
golden section search method

1. f (x) , tolerance ( ), boundary bl ,1 , bu ,1

2. 2 ( x l ,1 xu ,1 ) (3.7) (3.8)
xl ,1 bu ,1 KI (3.9)
xu ,1 bl ,1 KI (3.10)
3. ( f ( xl ,1 ) f ( xu ,1 ) ) 2

Minimum problem
f ( xl , n ) f ( xu , n )
bl , n 1 bl , n (3.11)
bu , n 1 xu , n (3.12)
I n 1 bu , n 1 bl , n 1 (3.13)
xl , n 1 bu , n 1 KI n 1 (3.14)
xu , n 1 xl , n (3.15)
f ( xl , n ) f ( xu , n )
bl , n 1 xl , n (3.16)
bu , n 1 bu , n (3.17)
I n 1 bu , n 1 bl , n 1 (3.18)
xl , n 1 xu , n (3.19)
xu , n 1 bl , n 1 KI n 1 (3.20)
Maximum problem
f ( xl , n ) f ( xu , n ) (3.16) - (3.20)
f ( xl , n ) f ( xu , n ) (3.11) - (3.15)

4. tolerance ( ) bu , n bl , n

n n 1
3
5.

EEE603 BP 4
 x* min xl , n , xu , n x* maxxl , n , xu , n

f min min f ( xl , n ), f ( xu , n ) f max max f ( xl , n ), f ( xu , n )

3.1 maximum f ( x) x cos(x 2 ) [0.0 , 0.7] 1 10 4

Solution

plot maximum x 0.45

0.38 Golden section search

0.4

0.2

-0.2
f(x)

-0.4

-0.6

-0.8

-1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
x

1 ( n 1 )
bl ,1 0.0
bu ,1 0.7
I1 0.7

(3.9) (3.10)
xl ,1 0.7 (0.6180)(0.7) 0.2674 f ( xl ,1 ) f (0.2674) 0.2607
xu ,1 0.0 (0.6180)(0.7) 0.4326 f ( xu ,1 ) f (0.4326 ) 0.3600
f ( xu ,1 ) f ( xl ,1 )

EEE603 BP 5
 2 ( n 2 )
bl , 2 xl ,1 0.2674
bu , 2 bu ,1 0.7
I 2 bu , 2 bl , 2 0.7 0.2674 0.4326
xl , 2 xu ,1 0.4326
xu , 2 bl , 2 KI 2 0.2674 (0.6180)(0.4326) 0.5347
tolerance 0.4326
tolerance
Iteration bl bu xl xu f(xl) f(xu) I
1 0 0.7 0.2674 0.4326 0.260682 0.359963 0.7
2 0.2674 0.7 0.432653 0.534747 0.359973 0.333094 0.4326
3 0.2674 0.534747 0.369526 0.43262 0.336043 0.359967 0.267347
4 0.369526 0.534747 0.432641 0.471633 0.359971 0.361087 0.16522
5 0.432641 0.534747 0.471645 0.495742 0.361085 0.35518 0.102106
6 0.432641 0.495742 0.456745 0.471637 0.362113 0.361086 0.063102
7 0.432641 0.471637 0.447537 0.456741 0.361826 0.362113 0.038997
8 0.447537 0.471637 0.456744 0.462431 0.362113 0.361944 0.0241
9 0.447537 0.462431 0.453227 0.456742 0.362084 0.362113 0.014894
10 0.453227 0.462431 0.456743 0.458915 0.362113 0.362081 0.009204
11 0.453227 0.458915 0.4554 0.456742 0.362114 0.362113 0.005688
12 0.453227 0.456742 0.45457 0.455399 0.362107 0.362114 0.003515
13 0.45457 0.456742 0.4554 0.455912 0.362114 0.362115 0.002172
14 0.4554 0.456742 0.455912 0.456229 0.362115 0.362115 0.001343
15 0.4554 0.456229 0.455717 0.455912 0.362115 0.362115 0.00083
16 0.455717 0.456229 0.455912 0.456033 0.362115 0.362115 0.000513
17 0.455912 0.456229 0.456034 0.456108 0.362115 0.362115 0.000317
18 0.455912 0.456108 0.455987 0.456033 0.362115 0.362115 0.000196
19 0.455912 0.456033 0.455959 0.455987 0.362115 0.362115 0.000121
20 0.455959 0.456033 0.455987 0.456005 - - 7.48E-05

20
x* 0.4559 f max 0.3621

3.2 minimum f ( x) 0.65 [0.75 /(1 x 2 )] 0.65 x tan 1 (1 / x)

EEE603 BP 6
 [0.0 , 1.8540] golden section search method 1 10 4
Solution

plot minimum x 0.5

-0.32 Golden section search

-0.05

-0.1

-0.15
f(x)

-0.2

-0.25

-0.3

-0.35
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2
x
1 ( n 1 )
bl ,1 0.0
bu ,1 1.8540
I1 1.8540

(3.9) (3.10)
xl ,1 1.8540 (0.6180)(1.8540) 0.7082 f ( xl ,1 ) f (0.7082) 0.2889
xu ,1 0.0 (0.6180)(1.8540) 1.1458 f ( xu ,1 ) f (1.1458) 0.2087
f ( xu ,1 ) f ( xl ,1 )

2 ( n 2 )

bl , 2 bl ,1 0.0
bu , 2 xu ,1 1.1458
I 2 bu , 2 bl , 2 1.1458

EEE603 BP 7
 tolerance I 1.1854 10 4

xl , 2 bu , 2 KI 2 0.4377 f ( xl , 2 ) f (0.4377 ) 0.3089

xu , 2 xl ,1 0.7082 f ( xu , 2 ) f (0.7082) 0.2889
f ( xu , 2 ) f ( xl , 2 )
3 ( n 3 )
bl ,3 bl , 2 0.0
bu ,3 xu , 2 0.7082
I 3 bu ,3 bl ,3 0.7082
tolerance I 0.7082 10 4

xl ,3 bu ,3 KI 3 0.2705 f ( xl ,3 ) f (0.2705) 0.2786

xu ,3 xl , 2 0.4377 f ( xu ,3 ) f (0.4377 ) 0.3089
f ( xu ,3 ) f ( xl ,3 )
4 ( n 4 )
bl , 4 xl ,3 0.2705
bu , 4 bu ,3 0.7082
I 4 bu , 4 bl , 4 0.4377
tolerance I 0.4377 10 4

xl , 4 xu ,3 0.4377 f ( xl , 4 ) f (0.4377 ) 0.3089
xu , 4 bl , 4 KI 4 0.5410 f ( xu , 4 ) f (0.5410) 0.3082
f ( xu , 4 ) f ( xl , 4 )
5 ( n 5 )
bl ,5 bl , 4 0.2705
bu ,5 xu , 4 0.5410
I 5 bu ,5 bl ,5 0.2705
tolerance I 0.2705 10 4

xl ,5 bu ,5 KI 5 0.3738 f ( xl ,5 ) f (0.3738) 0.3028
xu ,5 xl , 4 0.4377 f ( xu ,5 ) f (0.4377 ) 0.3089
f ( xu ,5 ) f ( xl ,5 )
6 ( n 6 )
bl , 6 xl ,5 0.3738
bu , 6 bu ,5 0.5410
I 6 bu , 6 bl , 6 0.1672
tolerance I 0.1672 10 4

EEE603 BP 8
 xl , 6 xu ,5 0.4377 f ( xl , 6 ) f (0.4377 ) 0.3089
xu , 6 bl , 6 KI 6 0.4771 f ( xu , 6 ) f (0.4771) 0.3100
f ( xu , 6 ) f ( xl , 6 )
7 ( n 7 )
bl , 7 xl , 6 0.4377
bu , 7 bu , 6 0.5410
I 7 bu , 7 bl , 7 0.1033
tolerance I 0.1033 10 4

minimum

Iteration bl bu xl xu f(xl) f(xu) I

1 0 1.854 0.708228 1.145772 -0.28891 -0.20869 1.854
2 0 1.145772 0.437685 0.708087 -0.30893 -0.28893 1.145772
3 0 0.708087 0.270489 0.437598 -0.2786 -0.30893 0.708087
4 0.270489 0.708087 0.437652 0.540925 -0.30893 -0.30818 0.437598
5 0.270489 0.540925 0.373796 0.437618 -0.30279 -0.30893 0.270435
6 0.373796 0.540925 0.437639 0.477081 -0.30893 -0.31001 0.167129
7 0.437639 0.540925 0.477094 0.50147 -0.31001 -0.30979 0.103286
8 0.437639 0.50147 0.462022 0.477086 -0.30982 -0.31001 0.063831
9 0.462022 0.50147 0.477091 0.486401 -0.31001 -0.31 0.039447
10 0.462022 0.486401 0.471335 0.477088 -0.30997 -0.31001 0.024378
11 0.471335 0.486401 0.47709 0.480646 -0.31001 -0.31002 0.015066
12 0.47709 0.486401 0.480647 0.482844 -0.31002 -0.31002 0.009311
13 0.47709 0.482844 0.479288 0.480646 -0.31002 -0.31002 0.005754
14 0.479288 0.482844 0.480646 0.481486 -0.31002 -0.31002 0.003556
15 0.479288 0.481486 0.480127 0.480646 -0.31002 -0.31002 0.002198
16 0.480127 0.481486 0.480646 0.480967 -0.31002 -0.31002 0.001358
17 0.480646 0.481486 0.480967 0.481165 -0.31002 -0.31002 0.000839
18 0.480646 0.481165 0.480844 0.480967 -0.31002 -0.31002 0.000519
19 0.480646 0.480967 0.480769 0.480844 -0.31002 -0.31002 0.000321
20 0.480769 0.480967 0.480844 0.480891 -0.31002 -0.31002 0.000198
21 0.480769 0.480891 0.480816 0.480844 -0.31002 -0.31002 0.000122
22 0.480816 0.480891 0.480844 0.480862 - - 7.57E-05

EEE603 BP 9

22
x* 0.4808 f min 0.3100

3.2.2 Fibonacci Search Method

Fibonacci search method ..1966 Avriel Wilde fibonacci search
golden section search method K
tolerance K Fibonacci
search method K golden section search method fibonacci search
method iterations
fibonanci (3.21)
( 5 1) k 1 (1) k ( 5 1) k 1
Fk (3.21)
2k 1 5
(3.21)
F0 1 , F1 1 (3.22)
Fibonacci number
Fk Fk 2 Fk 1 k2 (3.23)

3.1 Fibonacci numbers

k Fk k Fk

0 1 10 89
1 1 11 144
2 2 12 233
3 3 13 377
4 5 14 610
5 8 15 987
6 13 16 1597
7 21 17 2584
8 34 18 4181
9 55 19 6765

EEE603 BP 10
 fibonacci search method Iterations
tolerance
1
Fk (3.24)

3.3 tolerance 0.0001

Fk 1000

3.1 Fk F16 1597 1000

16
iterations tolerance 3.2
3.2 Fibonacci method iterations

Number of
Tolerance
Iterations

0.00001 25
0.00002 24
0.00005 22
0.0001 20
0.0002 19
0.0005 17
0.001 16
0.002 14
0.005 12
0.01 11
0.02 9
0.05 7
0.10 6

EEE603 BP 11

fibonacci search method golden section search method K
golden section search method K 0.6180 K fibonacci search method

Fk 1
K (3.25)
Fk

K (3.8) (3.9)
F
xl bu k 1 I (3.26)
Fk
F
xu bl k 1 I (3.27)
Fk
Minimum problem
f ( xl , j ) f ( xu , j )
bl , j 1 bl , j (3.28)
bu , j 1 xu , j (3.29)
I j 1 bu , j 1 bl , j 1 (3.30)
Fn j 1
xl , j 1 bu , j 1 I j 1 (3.31)
Fn j

xu , j 1 xl , j (3.32)
f ( xl , j ) f ( xu , j )
bl , j 1 xl , j (3.33)
bu , j 1 bu , j (3.34)
I j 1 bu , j 1 bl , j 1 (3.35)
xl , j 1 xu , j (3.36)
Fn j 1
xu , j 1 bl , j 1 I n 1 (3.37)
Fn j

Maximum problem
f ( xl , j ) f ( xu , j ) (3.33)-(3.37)
f ( xl , j ) f ( xu , j ) (3.28)-(3.32)

3.4 maximum f ( x) x cos(x 2 ) [0.0 , 0.7] 1 10 4

Solution

EEE603 BP 12
 1 ( j 1)
1 10 4 tolerance K (iteration)
F20 tolerance 10946
20
Fk
F19
1
F20

bl ,1 0.0
bu ,1 0.7
I1 0.7

(3.26) (3.27)

6765
xl ,1 0.7 (0.7) 0.2673 f ( xl ,1 ) f (0.2674) 0.26063
10946
6765
xu ,1 0.0 (0.7) 0.4326 f ( xu ,1 ) f (0.4326) 0.35996
10946
f ( xl ,1 ) f ( xu ,1 )
2 ( j 2 )
bl , 2 xl ,1 0.2673
bu , 2 bu ,1 0.7
I 2 bu , 2 bl , 2 0.7 0.2673 0.4327

xl , 2 xu ,1 0.4326 f ( xl , 2 ) f (0.4326) 0.3600

4181
xu , 2 bl , 2 I 2 0.5348 f ( xu , 2 ) f (0.4326) 0.3331
6765
f ( xl , 2 ) f ( xu , 2 )

20

Iteration bl bu xl xu f(xl) f(xu) I

1 0 0.7 0.267376 0.432624 0.260661 0.359968 0.7
2 0.267376 0.7 0.432624 0.534752 0.359968 0.33309 0.432624
3 0.267376 0.534752 0.369505 0.432624 0.336031 0.359968 0.267376
4 0.369505 0.534752 0.432624 0.471633 0.359968 0.361086 0.165248
5 0.432624 0.534752 0.471633 0.495743 0.361086 0.35518 0.102129
6 0.432624 0.495743 0.456733 0.471633 0.362113 0.361086 0.063119

EEE603 BP 13
 7 0.432624 0.471633 0.447524 0.456733 0.361825 0.362113 0.03901
8 0.447524 0.471633 0.456733 0.462425 0.362113 0.361945 0.024109
9 0.447524 0.462425 0.453216 0.456733 0.362084 0.362113 0.0149
10 0.453216 0.462425 0.456733 0.458907 0.362113 0.362081 0.009209
11 0.453216 0.458907 0.45539 0.456733 0.362114 0.362113 0.005692
12 0.453216 0.456733 0.454559 0.45539 0.362107 0.362114 0.003517
13 0.454559 0.456733 0.45539 0.455902 0.362114 0.362115 0.002174
14 0.45539 0.456733 0.455902 0.456221 0.362115 0.362115 0.001343
15 0.45539 0.456221 0.45571 0.455902 0.362115 0.362115 0.000831
16 0.45571 0.456221 0.455902 0.45603 0.362115 0.362115 0.000512
17 0.455902 0.456221 0.45603 0.456094 0.362115 0.362115 0.00032
18 0.455902 0.456094 0.455966 0.45603 0.362115 0.362115 0.000192
19 0.455966 0.456094 0.45603 0.45603 0.362115 0.362115 0.000128
20 0.45603 0.456094 0.45603 0.456062 - - 6.40E-05

20
x* 0.4560 f max 0.3621

Golden Section Search Method

3.3 Approximation Method

approximation method objective function

quadratic polynomial 2 cubic polynomial 3
quadratic method

3.3.1 Quadratic Method

objective function quadratic function (2nd order
polynomial)
f ( x) ax 2 bx c (3.38)
minimum value x f ' ( x) 0
b
x* (3.39)
2a

EEE603 BP 14
 minimum a b
3 x1 , x2 x3 f1 , f 2 f3

x12 x1 1 a f1
2
x2 x2 1 b f 2 (3.40)
x32 x3 1 c f 3

(3.40) a b (3.39)

x*
x2
2
x32 f1 x32 x12 f 2 x12 x22 f 3 (3.41)
2( x2 x3 ) f1 ( x3 x1 ) f 2 ( x1 x2 ) f 3

3 h x2 3
x1 x2 h x3 x2 h (3.41)
h( f1 f 3 )
x* x2 (3.42)
2( f 3 2 f 2 f1 )

(3.41) (3.42) objective function

2

16
3.5 minimum f ( x) 2 x 2 1 x 5
x
Solution x1 1 , x3 5 x2 3
16
f ( x1 ) 2 x12 18
x1
16
f ( x2 ) 2 x22 23.33
x2
16
f ( x3 ) 2 x32 53.2
x3
step h 2 minimum (3.42)
2(18 53.2)
x* 3 1.5656
2(53.2 2(23.33) 18)

objective function 2 1.5656

x* 1.5874
100%
search method

3.4 Combination Method

EEE603 BP 15
Combination method search method approximation method
combination method
3 DSC method, Powells method Arithmetic mean method
3.4.1 Davies, Swann and Campey (DSC) method
DSC method approximation method search method
objective function 2 3

DSC method
1. x1
f ( x1 )
2. (direction) (step size)

x
3. 2 x2
f ( x2 ) x2 x1 x
f ( x2 ) f ( x1 ) f ( x1 ) f ( x2 )
f ( x1 ) f ( x2 )

4. step size 2 2x
3 x3 f ( x3 )
x3 x2 2x
5. 2 3 f ( x2 ), f ( x3 )
f ( x2 ) f ( x3 ) search 4
f ( x2 ) f ( x3 ) step size

6. 4 3
approximation method 3

3.4 DSC method
h 2
step size 2 3
3 step size 2 4
4
step size 5 4 2 3
4 5 4 3
2 3 5 (3.42)

EEE603 BP 16
 f (x)

x
3.4 : DSC interpoolation methodd

3.6 minimum f (xx) 4 x 2 12 x 9 step size 0.5

x0
Solution
1.
1 x1 0 f (xx1 ) f (0) 9
2. 2
x 2 0 0 .5 0.5 f (xx 2 ) f (0.5) 4
f ( x 2 ) f ( x1 ) step size 2h 1
3. 3
x3 0.5 1 1.5 f (xx3 ) f (0.5) 0
f ( x3 ) f ( x 2 ) step size 4h 2


4. 4
x 4 1.5 2 3.5 f (xx 4 ) f (3.5) 16
f ( x 4 ) f ( x3 ) step size 2h 1

5. 5
x5 3.5 1 2.5 f (xx5 ) f (2.5) 4
6. 4 4
3 0.5,1.5, 2.5 3 4, 0, 4
x2 1.5 step size h 1
(3.442)
h( f 1 f 3 )
x* x 2
2 ( f 3 2 f 2 f1 )

x* 1.5

EEE6603 BP 17
 16
3.7 minimum f ( x) 2 x 2 step size 0.01
x
x 1
Solution DSC method

point # x fx h
1 1 18 0.01
2 1.01 17.88178 0.02
3 1.03 17.65578 0.04
4 1.07 17.24307 0.08
5 1.15 16.55804 0.16
6 1.31 15.64594 0.32
7 1.63 15.12975 0.64
8 2.27 17.35426 0.32
9 1.95 15.81013 -

3 6 7 9 3 (3.42)
x* 1.6080 DSC method
x* 1.5874
DSC method

3.4.2 Powells method of Quadratic Interpolation

Powells method of quadratic interpolation objective function
x1 d x1 (current point) d (search
direction) Powells method 2 f ( x1 d )
3 tolerance

3 x1 d x1 ad , x1 bd , x1 cd
3
f a f ( x1 ad ) (3.43)
f b f ( x1 bd ) (3.44)
f c f ( x1 cd ) (3.45)
2
f ( ) f 0 f1 f 22 (3.46)

EEE603 BP 18

f a f ( a ) f 0 f1a f 2 a 2 (3.47)
f b f (b) f 0 f1b f 2b 2 (3.48)
f c f (c ) f 0 f1c f 2c 2 (3.49)

(3.47) (3.49) f 0 , f1 f 2
bc(c b) f a ac(a c) f b ab(b a) f c
f0 (3.50)
(a b)(b c)(c a)

(b 2 c 2 ) f a (c 2 a 2 ) fb (a 2 b 2 ) f c
f1 (3.51)
(a b)(b c)(c a)
(c b) f a (a c) f b (b a) f c
f2 (3.52)
(a b)(b c)(c a)

(3.46) f ' ( ) 0
f1
(3.53)
2 f2

(3.53) f ( ) 0 min
(3.51) (3.53)

min

0.5 (b 2 c 2 ) f a (c 2 a 2 ) f b (a 2 b 2 ) f c (3.54)
(b c) f a (c a) f b (a b) f c

f ( )
(b c) f a (c a) f b (a b) f c
0 (3.55)
(a b)(b c)(c a)

Powells method
1. step size h direction vector d unit vector
2. x1 , tolerance step size M
3. a x1 b x1 hd
f ( x1 ) f ( x1 hd ) c x1 hd

EEE603 BP 19
 4. f ( x1 ) f ( x1 hd ) c x1 2hd
5. min (3.54)
6. min
M min Md
M min (3.54)
M min
o tolerance min
o tolerance min
min (3.54)

3.7 minimum f ( x) x 3 6 x 2 4 x 8 0 x 1
x 0 M 0.3 step size = 0.01 tolerance = 0.001
Solution a, b, c a 0
h 0.01

Iteration # a b c fa fb fc xmin f min

1 0 0.01 0.02 -8 -8.0394 -8.0776 0.3317 -8.6302

2 0 0.01 0.3 -8 -8.0394 -8.6330 0.3172 -8.6332

3 0.3172 0.01 0.3 -8.6332 -8.0394 -8.6330 0.3094 -8.6336

4 0.3172 0.3094 0.3 -8.6332 -8.6336 -8.6330 0.3094 -8.6336

minimum f min 8.6336 xmin 0.3094

3.4.3 Arithmetic mean method

Arithmetic mean method
1. 3 (a,b,c) convex function f a fb f c x1 x2 x1 +
step size x3 2 x 2 x4 2 x3 , x5 2 x4 ,
abc
2. mean 3 convex function s
3

EEE603 BP 20
 2b c
3. s b s s
3
fb f s f fs
4. tolerance f ( s ) b
fs fs
b fb min( f s , fb ) 2 b b
convex function 2 tolerance

3.8 minimum f ( x) 4 x 2 12 x 9 step size 0.3,

x 0 tolerance 0.001
Solution
x1 0 f ( x1 ) 9
x2 0.3 f ( x2 ) 5.76
x3 0.6 f ( x3 ) 3.2
x4 1.2 f ( x4 ) 0.36
x5 2.4 f ( x5 ) 3.24

convex function a x3 0.6 , b x4 1.2 c x5 2.4 arithmetic mean

abc
s
3

0.6 1.2 2.4 4.2

s 1 .4 f ( s ) 0.04
3 3

tolerance

fb f s 0.36 0.04
8
fs 0.04

tolerance convex function s

f ( s ) f (b)
a 0.6 , b 1.4 , c 2.4
tolerance

Iteration # a b c s f(s)
1 0.6 1.2 2.4 1.4 0.04
2 1.2 1.4 2.4 1.666667 0.111111
3 1.2 1.4 1.666667 1.422222 0.024198
4 1.4 1.422222 1.666667 1.496296 5.49E-05

EEE603 BP 21
 5 1.422222 1.496296 1.666667 1.528395 0.003225
6 1.422222 1.496296 1.528395 1.482305 0.001253
7 1.482305 1.496296 1.528395 1.502332 2.18E-05
8 1.496296 1.502332 1.528395 1.509008 0.000325
9 1.496296 1.502332 1.509008 1.502545 2.59E-05
10 1.496296 1.502332 1.502545 1.500391 6.12E-07
11 1.496296 1.500391 1.502332 1.499673 4.27E-07
12 1.496296 1.499673 1.500391 1.498787 5.89E-06
13 1.498787 1.499673 1.500391 1.499617 5.87E-07
14 1.499617 1.499673 1.500391 1.499894 4.51E-08
15 1.499673 1.499894 1.500391 1.499986 7.78E-10
16 1.499894 1.499986 1.500391 1.50009 3.27E-08
17 1.499894 1.499986 1.50009 1.49999 3.94E-10
18 1.499986 1.49999 1.50009 1.500022 1.96E-09
19 1.499986 1.49999 1.500022 1.499999 1.30E-12
20 1.49999 1.499999 1.500022 1.500004 6.05E-11
21 1.49999 1.499999 1.500004 1.499998 1.94E-11
22 1.499998 1.499999 1.500004 1.5 5.52E-13
23 1.499999 1.5 1.500004 1.500001 6.05E-12
24 1.499999 1.5 1.500001 1.5 4.74E-13
25 1.499999 1.5 1.5 1.5 8.88E-15
26 1.499999 1.5 1.5 1.5 1.42E-14
27 1.5 1.5 1.5 1.5 4.97E-14
28 1.5 1.5 1.5 1.5 3.55E-15
29 1.5 1.5 1.5 1.5 0
30 1.5 1.5 1.5 1.5 0

minimum x 1.5

3.5 MATLAB Solution

MATLAB fminbnd search

method golden section search method
minimum problem maximum
problem -1 objective function
minimum problem
maximum problem

EEE603 BP 22
 3.9 minimum f ( x) x 2 (e x cos(x)) [1.75 , 3.25]
Solution
MATLAB Code
>> f=@(x)x^2*(exp(-x)+cos(pi*x));
>> [x,fval]=fminbnd(f,1.75,3.25)
x=
3.0668
fval =
-8.7609

x 3.0668 minimum -8.7609

3.10 maximum f ( x) x cos(x 2 ) [0.0 , 0.7]

1 10 4
Solution
maximum
minimum problem

f ( x) x cos(x 2 )
MATLAB Code
>> f=@(x)-x*cos(pi*(x^2));
>> [x,fval]=fminbnd(f,0,0.7,1e-4)
x=
0.4560
fval =
-0.3621

x 0.4560 maximum 0.3621 ( objective

function -1
-1 )

EEE603 BP 23
 4

(Unconstrained Multivariable Optimization)

3 search method (single variable functions)

single variable
multivariable functions Gradient method
Gradient method optimization multivariable

Gradient method 2 Steepest Descent
Method Newton-Rhapson Method multivariable functions
5

constrained problem
unconstrained problem

1. (constraints)
2. constrained problem
unconstrained problem
3. unconstrained problem constrained problem

4. unconstrained problem engineering

unconstrained problem 2
direct search method descent method direct search method objective
function partial derivative
nongradient method descent method first derivative
second derivative
descent method direct search method
descent methods gradient method Nongradient method Gradient
method

EEE603 BP 1
 Direct search methods Descent methods
- Random search method - Steepest descent (Cauchy) method
- Grid search method - Fletcher Reeves method
- Univariate methods - Newtons method
- Pattern search methods - Marquardt method
Powells method - Quasi Newton methods
- Simplex method Davidon - Fletcher - Powell method
Broyden - Fletcher - Goldfarb - Shanno method

4.1 Gradient method

Gradient x1 , x 2 , , x n
f / x1
f / x 2
f (4.1)


f / x n n1

Gradient
Gradient gradient
direction steepest ascent steepest ascent
steepest ascent

gradient steepest ascent gradient

steepest descent gradient
gradient

EEE603 BP 2
 search direction gradient

Gradient method (Iterative process) x k x k 1

x k 1 x k k d k (4.2)
d k = k (Search direction)
k = k (Step size)

Gradient method 2 first order method second order method

first partial derivative second partial derivative Steepest-
descent method first derivative Newton-
Raphson method second derivative

gradient method search direction d step size iteration

(diverge)
step size
step size objective function optimal

4.1.1 Steepest Descent Method

gradient minimum ..1847
Cauchy steepest descent method
steepest descent update optimal


objective function f x objective function
f x

EEE603 BP 3
 3

40 2

30
1
20
0

x2
10
0
-1
-10
4
2 4 -2
0 2
-2 0
-2 -3
x2 -4 -4 x1 -3 -2 -1 0 1 2 3
x1

2 x1 x2
x1 x2 contour
Gradient gradient
gradient

optimum
x k 1 x k k f x k (4.3)

objective function
1. x 0 tolerance k 0
k
2. Gradient objective function x
3. f ( x k ) x k

4. update x k 1 (4.3) k
5. k first derivative f x k 1 k (4.3)
x k 1
6. k k 1 2

4.1 minimum f ( x1 , x2 ) x12 x22 2 x1 4 x2 2 steepest descent

method f ( x10 , x20 ) (0,1) 0.0001
Solution

first derivative gradient

f f
2 x1 2 2 x2 4
x1 x2
2 2
f ( x k ) f ( x k )
Gradient f ( x )
k

x1 x2

EEE603 BP 4
 gradient x (0,1)
f f
2 2 f ( x 0 ) 8 2.8284
x1 x2

search direction (4.3)

d f ( x k ) d 0 (2,2)

step size first derivative f ( x 0 d 0 ) 0

0 2
f 0
1 2

2 1 2 22 4 1 2 2
2 2
0

0 .5

step size update

x11 x10 0 d 0 0 0.5(2) 1

x12 x20 0 d 0 1 0.5(2) 2

gradient x (1,2)
f f
0 0 f ( x1 ) 0
x1 x2

Iteration # x f x1, x2 g x1, x2 g x1, x2 d

0 2 2
1 1 -5 2 2.8284 2 0.5

1 0
2 2 -7 0 0 - -

x (1,2) cost -7

EEE603 BP 5
 4.2 minimum f ( x1 , x2 , x3 ) ( x1 4)4 ( x2 3)2 4( x3 5)4
f ( x10 , x20 , x30 ) (4,2,1) 0.001

Solution

first derivative gradient

f f f
4( x1 4)3 2( x2 3) 16( x3 5)3
x1 x2 x3

Iteration x f x1, x2 , x3 g x1, x2 , x3 g x1, x2 , x3 d

#
4 0 0
1 2 1025 2 1024 2 0.004

1 1024 1024
4 0 0
2 2 0.9843 1.9841 1.9841 1.9841 0.5

5.0641 0.0042 0.0042
4 0 0
3 3 8.06 10 6 0.0038 8.06 10 6 10.15
5.9 10 5
5.0620 0.004 0.004
4 0
4 2.9999 1.56 10 4 - -
1.2 106 2.56 10 4
5.0233 2.03 10 4

x (4, 2.9999, 5.0233) minimum 1.2 106

4.3 maximum f (x1, x2 ) 4x1 6x2 2x12 2x1 x2 2x22

f (x10 , x20 ) (1,1) 103

Solution
derivative
4 4 x1 2 x 2
g ( x1 , x2 )
6 2 x1 4 x2
4 2
h( x1 , x 2 )
2 4

EEE603 BP 6
 Iteration # x f x1, x2 , x3 g x1, x2 , x3 g x1, x2 , x3
1 2
1 1 -4 0 2 0.25

0.5 0
2 1 -4.5 1 1 0.25

0.5 0.5
3 1.25 -4.625 0 0.5 0.25

0.375 0
4 1.25 -4.6562 0.25 0.25 0.25

0.375 0.125
5 1.3125 -4.6641 0 0.125 0.25

0.3437 0
6 1.3125 -4.6660 0.0625 0.0625 0.25

0.3437 0.0312
7 1.3281 -4.6665 0 0.0312 0.2501

0.3359 0
8 1.3281 -4.6666 0.0156 0.0156 0.2501

0.3359 0.0078
9 1.3320 -4.6667 0 0.0078 0.2534

0.3340 0.0001
10 1.3320 -4.6667 0.0040 0.0040 0.2395

0.3340 0.0019
11 1.3330 -4.6667 0.0001 0.0019 0.2927

0.3334 0.0003
12 1.3330 -4.6667 0.0011 0.0011 0.2075

0.3335 0.0004
13 1.3332 -4.6667 0.0001 0.0004 -

x (0.3335,1.3332) maximum 4.6667

4.1.2 Newton-Raphson method

steepest descent method optimum gradient
objective function optimum

EEE603 BP 7
 second partial derivative objective function
(4.2) Taylor series expansion objective function f (x) x x
f ( x x) f ( x) T f ( x)x 0.5(x)T 2 f ( x)x (4.4)
optimum take derivative (4.4) x 0
g Hx 0 (4.5)
g f (x)
H 2 f ( x)

H nonsingular matrix inverse H

(4.5)
x H 1 g (4.6)

(4.6) search direction

x k 1 x k k ( H 1 ) k g k (4.7)
x k 1 x k k ( H 1 )f ( x k ) (4.8)

Newton-Raphson method
1. x 0 tolerance k 0
2. gradient g k x k g k x k
3. g k Hessian matrix ( H k ) ( H 1 ) k (Hessian matrix positive definite matrix
)
4. search direction
d k ( H 1 ) k g k (4.9)
5. x k 1 x k k d k d k (4.9) x k 1 x k k ( H 1 ) k g k
objective function
k k 1 2

4.4 minimum f ( x1 , x2 ) x1 x2 2 x12 2 x1 x2 x22

(x10 , x20 ) (0, 0) tolerance 10-6
Solution
f ( x1 , x2 ) x1 x2 2 x12 2 x1 x2 x22
First derivative

EEE603 BP 8
 ( x1 x2 2 x12 2 x1 x2 x22 )
f / x1 x1
g
f / x2 ( x1 x2 2 x1 2 x1 x2 x2 )
2 2

x2

1 4 x1 2 x2

1 2 x1 2 x2
1 4 x1 2 x 2 1
g0
1 2 x1 2 x 2 ( 0, 0) 1

second derivative
2 f / x12 2 f / x1x2
H 2
f / x2x1 2 f / x22
4 2

2 2
1 2 2 1 / 2 1 / 2
H 0-1
4 2 4 1 / 2 1

check Gradient
g0 12 1 2
2

Gradient tolerance
update
X1 X0 0 (H )0 g0
1

0 1 / 2 1 / 2 1

0 1 / 2 1 1


3 / 2

f (x11, x21 ) first derivative f (x11, x21 )

f ( x1 , x 2 ) - - 2 2 3 4
3 2 2 9 2

0 1 3 4 6 9
2 2
1
1
X 1 3 Gradient X1
2
1 4 x1 2 x 2 0
g1
1 2 x1 2 x 2 ( 1,3 / 2 ) 0

EEE603 BP 9
 Iteration # x f x1, x2 g x1, x2 g x1, x2 d
0 1 1
1 0 0 1 2 1 1

1 0
2 1.5 -1.25 0 0 - -

1
g1 0
Gradient X 1
3 / 2
minimum point

16
4.5 minimum f ( x) 2 x 2 x0 1
x
Solution
16
f ( x) 2 x 2
x
First derivative
df ( x ) 16
g 4x 2
dx x

Second derivative
d 2 f ( x) 32
H 2
4 3
dx x

Iteration # x f x1, x2 g x1, x2 g x1, x2 d

1 1 18 -12 12 12 1.7622
2 1.5874 15.1191 0 0 - -

4.6 maximum f (x1, x2 ) 4x1 6x2 2x12 2x1 x2 2x22

f (x10 , x20 ) (1,1) 103

Solution
derivative
4 4 x1 2 x 2
g ( x1 , x2 )
6 2 x1 4 x2
4 2
h( x1 , x 2 )
2 4

EEE603 BP 10
Iteration
x f x1, x2 , x3 g x1, x2 , x3 g x1, x2 , x3 d
#
1 2 0.6667
1 1 -4 0 2 0.3333 1

0.3333 0
2 1.3333 -4.6667 0 0 - -

x (0.3333,1.3333) maximum 4.6667

4.2 MATLAB Solution

unconstrained problem MATLAB minimum

unconstrained problem fminunc

4.7 minimum f ( x1 , x2 ) 100( x2 x12 ) 2 (1 x1 ) 2

1.2
X 1
1
Solution
MATLAB Code
Step#1 objective function M-file objfun.m
function f = objfun(x)
f = 100*(x(2)-(x(1)^2))^2 + (1-x(1))^2;
Step#2 minimum
x0=[-1.2;1];
f = objfun(x0)
options = optimset(LargeScale,off);
[x,fval]=fminunc(@objfun,x0,options)
f = 24.2000
Optimization terminated: relative infinity-norm of gradient less than options.TolFun.
x = 1.0000 1.0000
fval = 2.8336e-011
1

X minimum 2.8336 10-11

1

EEE603 BP 11
 5

(Constrained Multivariable Optimization)

optimization linear constrained 2 3 4

optimization nonlinear unconstrained optimization
nonlinear constraint
optimization constrained algorithm
unconstrained constrained 2 equality constraints
inequality constraints
equality constraints Jacobain method
Lagranges method inequality constraints Karush
Kuhn Tucker

5.1 (Constrained Problem)

constrained
constrained problem x f (x)
g i ( x) 0 , i 1,2,, m

hk ( x) 0 , k 1,2,, n

constrained 2
direct methods indirect methods

Direct Methods Indirect methods

- Random search method - Transformation of variables technique
- Heuristic search methods - Sequential unconstrained minimization technique
- Objective and constraint approximation method Interior penalty function method

EEE603 BP 1
- Methods of feasible directions Exterior penalty function method
- Generalized reduced gradient method Augmented Lagrange multiplier method

5.2

constrained
4
1) (optimum point)
minimum constrained problem unconstrained problem
minimum point necessary conditions sufficient conditions
f x*
0

2 f
J x* = positive definite
xi x j x*

2) Khun Tucker
necessary conditions gradient positive linear combination
gradient constraints
3) objective function local minima unconstrained 2
constrained problem minimum
4) objective function minimum unconstrained
constrained local minima

5.3 Jacobian Method

Jacobian method constrained f (x) 1

constrained g ( x ) 0

minimum z f (x) constrained g ( x ) 0

x ( x1 , x2 , , xn ) (5.1)
g ( g1 , g 2 , , g m )T (5.2)

EEE603 BP 2
 5.1:

gradient Taylors
theorem constrained
f ( x x) f ( x) f ( x)x o(x 2j ) (5.3)
g ( x x) g ( x) g ( x)x o(x 2j ) (5.4)
ox 2j high order ox 2j 0
(5.3)
(5.4)
f ( x ) f ( x )x (5.5)
g ( x ) g ( x)x (5.6)

constraints g ( x ) 0 g ( x) 0
(5.5) (5.6)

f ( x ) f ( x )x 0 (5.7)
g ( x)x 0 (5.8)
x ( y, z ) y ( y1 , y2 ,..., ym ) dependence variable
z ( z1 , z2 ,..., zn ) independence variable gradient
vector constrained
f ( y , z ) ( y f , z f ) (5.9)
g ( y, z ) ( y g , z g ) (5.10)

EEE603 BP 3
 Jacobian matrix (J) control matrix (C)
y g1

y g2
J yg (5.11)


g
y m
z g1

z g2
C z g (5.12)


g
z m
(5.7) (5.8)
f ( y, z ) y fy z fz (5.13)
Jy Cz (5.14)
Jacobian matrix nonsingular matrix (5.14)
y J 1Cz (5.15)
f ( y, z ) ( z f y fJ 1C )z (5.16)
(5.16) constrained gradient f z
c f ( y, z )
c f z f y fJ 1C (5.17)
c z

sensitivity constraints
optimal value optimal value constraints g ( x ) 0
f constraints g i ( x) g i
f sensitivity analysis nonlinear
programming optimal value

g i ( x) 0
f ( y , z ) y fy z fz
g ( x ) Jy Cz

g 0
y J 1g J 1Cz
f ( y , z )
f ( y , z ) y fJ 1 g c fz

EEE603 BP 4
 c f z f y f J 1 C
optimal point x0 ( y0 , z0 ) stationary constrained
gradient c f

f ( y 0 , z 0 ) y0 fJ 1g ( y 0 , z 0 ) (5.18)

f
y 0 fJ 1 (5.19)
g
(5.19) sensitivity coefficients
constraint g f

5.1 f X 0
f ( X ) x12 3 x22 5 x1 x32
g1 ( X ) x1 x3 2 x2 x22 11 0
g 2 ( X ) x12 2 x1 x2 x32 14 0

X 0 (1, 2, 3) x2 0.01 Y ( x1 , x3 ) Z x2
Solution
gradient dependent independent
f f
Y f ( , ) (2 x1 5 x32 , 10 x1 x3 )
x1 x3

f
Z f 6x2
x2

Jacobian matrix control matrix

g1 g1

x x3 x3 x1
J 1
g 2 g 2 2 x1 2 x2 2 x3
x x3
1

g1

x2 2 x2 2
C
g 2 2 x1
x
2

EEE603 BP 5

constraint gradient z x 2
1
1 3 1 6 2.83
J C
6 6 2 2.50

c f z f y f J 1C z
2.83
6(2) 47, 30 x2
2.50
46.01x2 0.4601

x 2 0.01 f 0.4601
dependent variable x1 x3
y J 1Cz

x2 0.01
x1 0.0283
J 1Cx2
x3 0.0250

f X 0 X f X 0
X 0 X 1 0.0283, 2 0.01, 3 0.025 0.9717, 2.01, 3.025
f X 0 58, f X 0 X 57.523

f X 0 X f X 0 0.477

x2 0.01 x1 x3
0.0283 0.0250
f 0.477 constraint gradient
f 0.4601
X
0

5.4 Lagrange Method

Lagrange method constrained problem

(first derivative) objective function optimize
f ( x1 , x2 ) equality constraint

g ( x1 , x2 ) 0 (5.20)

EEE603 BP 6
 x1 x2 x1 x2
x1 ( x2 ) (5.21)

(5.21) x2
F ( x2 ) f ( ( x2 ), x2 ) (5.22)
x2
h( x2 ) g ( ( x2 ), x2 ) 0 (5.23)

F f ( x1, x2 )
f f
dF dx1 dx2 (5.24)
x1 x2

(5.24) x1 x2
dF f f dx2
(5.25)
dx1 x1 x2 dx1
dF f dx1 f
(5.26)
dx2 x1 dx2 x2

(5.22) (5.23) minimum objective function

F ( x2 ) h( x2 )
dF ( x2 ) f f d ( x2 )
0 (5.27)
dx2 x2 x1 dx2
dh( x2 ) g g d ( x2 )
0 (5.28)
dx2 x2 x1 dx2

df df / dx1 dg
0 (5.29)
dx1 dg / dx1 dx1

f g 0 (5.30)
df
(5.31)
dg
Lagrange multiplier
Lagrange function (Lagrangean)
L ( x, ) f ( x ) g ( x ) (5.32)
necessary condition gradient Lagrangean
L ( x, ) 0 (5.33)

EEE603 BP 7
 5.2 minimize f ( x) x12 x22 x32
x1 3x2 x3 8
4 x1 2 x2 x3 2
x1 2 x2 x3 2
Solution
L( x, ) f ( x) g ( x)

L ( x, ) x12 x22 x32 1 ( x1 3 x2 x3 8) 2 ( 4 x1 2 x2 x3 2) 3 ( x1 2 x2 x3 2)
T
L L L L L L
L ( x , ) 0
x1 x2 x3 1 2 3
L
2 x1 1 42 3 0 (1)
x1
L
2 x2 31 22 23 0 (2)
x2
L
2 x3 1 2 3 0 (3)
x3
L
x1 3 x2 x3 8 (4)
1
L
4 x1 2 x2 x3 2 (5)
2
L
x1 2 x2 x3 2 (6)
1
(1) (6) x1 0, x2 2, x3 2, 1 2.40, 2 0.267, 3 1.333
f ( x) 8

5.4.1 (Lagrange Multiplier)

Lagrange multiplier objective function constraint
Lagrange multiplier
df
i (5.34)
dg i

5.3 minimize f ( x) x12 x22 x32

x1 3x2 x3 8
4 x1 2 x2 x3 2.1
x1 2 x2 x3 2
Solution
objective function 5.2

EEE603 BP 8
 4 x1 2 x2 x3 2 4 x1 2 x2 x3 2.1

f ( x) 8 0.1 2
8 0.1 0.267
8.0267

5.5 Karush Kuhn Tucker

Karush Kuhn Tucker constraints inequality constraints

Karush Kuhn Tucker minimum f ( x)
h( x) 0
(5.35)
g ( x) 0
slack constraint g ( x) 0

s 2 s12 , s22 ,, sm2 (5.36)

Lagrangean
L( x, s, ) f ( x) T h T ( g ( x) s 2 ) 0 (5.37)
T Lagrange multiplier
T KKT multiplier
(5.37) 1 x , s ,
L
f ( x) T h T g ( x) 0 (5.38)
x
L
2i si 0 i 1,2,, m (5.39)
x
L
( g ( x ) s 2 ) 0 (5.40)

L
h( x ) 0 (5.41)

(5.39) (5.40)
i g i ( x) 0 (5.42)
KKT
1. gradient Lagrange function (Lagrangean)
2. Constraints
h( x) 0 (equality constraint)

EEE603 BP 9
 g ( x) 0 (inequality constaint)
3. Complementary slackness slack variable s 0
4. Feasibility inequality constraint si2 0
5. Sign condition inequality multiplier 0
maximize KKT condition
Maximize f ( x)
subject to h( x) 0
g ( x) 0
f ( x) T h T g ( x) 0
KKT condition
i 0 , i 1,2,..., m
f ( x) T h T g ( x) 0
T g ( x) 0
h( x) 0
g ( x) 0
minimize KKT condition
Minimize f ( x)
subject to h( x) 0
g ( x) 0
f ( x ) h T g ( x ) 0
T

KKT condition
i 0 , i 1,2,..., m
f ( x) T h T g ( x) 0
T g ( x) 0
h( x) 0
g ( x) 0

5.4 minimize f ( x1 , x2 ) x12 x22 x1 x2 3x1

x1 , x2 0
Solution
KKT condition
1 , 2 T 0
f ( x ) 0
T x 0
x0

EEE603 BP 10
 f ( x1 , x2 ) x12 x22 x1 x2 3x1
2 x1 x2 1 3
x1 2 x2 2 0
1x1 2 x2 0
3 4
trial and error
1 0, x1 0 x2 0, 2 3
KKT condition f ( x) 0
1 0, x2 0 x1 3 / 2, 2 3 / 2
KKT condition f ( x) 2.25

5.6 MATLAB Solution

constraint problem MATLAB

fmincon

5.5 minimize f ( x) x13 6 x12 11x1 x3

x 0.1 0.1 3.0
T

x12 x22 x32 0

4 x12 x22 x32 0
x3 5 0
xi 0 i 1,2,3
Solution
MATLAB Code
Step#1 objective function M-file objfun.m
function f = objfun(x)
f = x(1)^3-6*x(1)^2+11*x(1)+x(3);
Step#2 M-file constraints.m
function [c,ceq]=constraints(x)
% Nonlinear inequality constraints
c=[x(1)^2+x(2)^2-x(3)^2;4-x(1)^2-x(3)^2;x(3)-5;-x(1);-x(2);- x(3)];
% Nonlinear equality contraints
ceq=[];
Step#3 minimum

EEE603 BP 11
 >> x0=[0.1 0.1 3.0];
>> f=objfun(x0)
f=
4.0410
>> [c ceq]=constraints(x0)
c=
-8.9800
-5.0100
-2.0000
-0.1000
-0.1000
-3.0000
ceq =
[]
options = optimset ('LargeScale', 'off');
[x, fval]=fmincon (@objfun, x0, [], [], [], [], [], [],
@constraints, options)
Optimization terminated: first-order optimality measure
less
than options. TolFun and maximum constraint violation is
less
than options.TolCon.
Active inequalities (to within options.TolCon = 1e-006):
lower upper ineqlin ineqnonlin
1
2
4
x=
0 1.4142 1.4142
fval =
1.4142
The values of constraints at optimum solution
c=
-0.0000
-0.0000

EEE603 BP 12
 -3.5858
0
-1.4142
-1.4142
ceq = []
0

x 1.4142 minimum 1.4142

1.4142

EEE603 BP 13