Fields of Optimization

Convex programming studies the case when

the objective function is convex
the constraints, if any, form a convex set.

Method of Optimization

Linear programming (LP)

the objective function f(x) is linear
the set of constraints is specified using only linear equalities and inequalities.
called a Polyhedron or a Polytope if it is bounded.

Quadratic programming
the
th objective
bj ti function to ha
have
e quadratic terms,
terms

while the set A must be specified with linear equalities and inequalities.

Geometric programming
objective and inequality constraints expressed as posynomials and equality
constraints as monomials can be transformed into a convex program.

Nonlinear programming
the objective
j
function or the constraints or both contain nonlinear parts.
p

This may or may not be a convex program. In general, the convexity of the pro
gram affects the difficulty of solving more than the linearity.

Stochastic programming
some of the constraints or parameters depend on random variables.

20 1 Basic Concepts : Optimization

20.1

20 1 Basic Concepts : Unconstrained Optimization

20.1

objective function
f (x,y) f ( x1 , x2 ,..., xn )
x,y

( x ,..., x )

control
t l variables
i bl
control
Optimize

optimization problem

function max. min.

function optimize

(objective function)

function

(maximize)


1

1 ..

optimization (optimal choice)

(control

( t l variables)
i bl )
objective
bj ti
function (max or min)

function
(minimize)



3

General Description of Optimization

20 1 Basic Concepts : Unconstrained Optimization

20.1

( x1 ,..., xn )

The function f(x) can be called,

an objective function, cost function, or energy function.
A feasible solution that minimizes (or maximizes, if that is the goal)
the objective function is called an optimal solution.

(constraint)



x1 x1 0
,,

Generally
when the feasible region or the objective function of the problem
not present convexity,

there may be several local minima and maxima,

where a local minimum x* is defined

x x*

f ( x* ) f ( x)

How can an Optimum be found?

One of Fermat's theorems states that
Optima of unconstrained problems are found at stationary
points,
points
where the 1st derivative or the gradient of the objective function is zero

More generally, they may be found at critical points,

where the 1st derivative or gradient of the objective function is zero or
undefined, or on the boundary of the choice set.

CONSTRAINED OPTIMIZATION
((LINEAR PROGRAMMING)
OG
G)

Optima of inequality-constrained
ineq alit constrained problems
bl
are instead
i t d
found on the boundary of the constrained set or
found by the gradient values using Lagrange multiplier method.

20 2 Linear Programming
20.2

20 2 Linear Programming
20.2

Mathematics Programming

linear
geometric solution
EX1 . Silvex Products J K 2 (
M1 M2) 2

M1: J 2
K 8
M2: J 5
K 2
J 40$ K 88$ J (x
) K (y) ..

optimization problem
(constraint) max or min objective
function z=f(x1,,xn)
Linear Programming (Linear Optimization) Mathematics Programming o
bjective function
z = f (x1,, xn) = ax1 ++ axn

linear equalities ( ) 3x1+ 4x2 36, x1 0

9

20 2 Linear Programming
20.2

10

20 2 Linear Programming
20.2
Fig.442 plot (0) (4)
O:z 0
A : z 40 12 480
B : z 40 10 88 5 840
C :: z 88 7.5 660

11

12

Linear Programming

20 2 Linear Programming
20.2

feasibility region
B (1) (2)

0 : x1 =0 x2 =0 z =0
(1)
C x1 =0 x2 = 60/8=7.5
(2)
A x2 =0 x1 = 60/5=12
B

2 x1 8 x2 60

(a)

5 x1 2 x2 60

(b)

Z = 40(10) + 88(5) = 840 J 10

K 5

4 (b); 20 x1 8 x2 240 (c)

(c) (a ); 18 x1 180
x1 10
i t d x1 10 in
instead
i (a)
x2 5
13

14

20 2 Linear Programming: Normal Form

20.2

20 2 Linear Programming :Normal form

20.2

linear slack variable

Normal Form Linear Programming

Maximize:

z f ( x ) c1 x1 c2 x2 cn xn

(1)

:
a11 x1
a21 x1

am1 x1

xi

15

a1n xn

a2 n xn

amn xn

b2

bm

b1

0 (i 1,..., n)
16

20 3 Simplex Method
20.3

20.3 Simplex Method

1. Normal Form
2. Simplex Table
3. basic variable non basic variable
non basic variable
0
feasible solution
3.1 Maximize: 1 Simplex Table
< 0 4
3.2 Minimize: 1 Simplex Table
> 0 4
4 Pivot
4.1 column pivot
4.2 row pivot
p
4.3 row operation pivot 0
3

basic feasible solution

basic feasible solution objective
function

maximize
z = 40 x1+ 88 x2
constraint
2 x1 + 8 x2 60
5 x1 + 2 x2 60
x1 0
x2 0
17

20.3 Simplex Method

18

20.3 Simplex Method

optimal solution augmented matrix

Normal Form objective function

z - 40 x1 + 88 x2
=0
2 x1 + 8 x2 + x3 = 60
5 x1 + 2 x2 + x4 = 60

x1

x2

x3

x4

1 40 88 0 0 0
T0 0
2
8 1 0 60
0
5
2 0 1 60

simplex table
2

xi 0

basic variable 0 x3 x4 T0
nonbasic variable basic variable

( i =1,2,3,4 )

Feasible
bl solution
l
for
f which
h h at least
l
n-m off variables
bl x1,.,xn are zero
From fig.422 we have n =2, m=4

19

20

20.3 Simplex Method

20.3 Simplex Method

optimal solution pivoting
basic feasible sol
solution
tion
z
z
( objective function )
step
p 1
Operation O1 : pivot column
1 column (4) 2
-40
40 2 pivot
i
column
l
( Maximize)
(
M i i )
Operation O2 : pivot row pivot c
, = 12))
olumn (( ((4)) 60/2= 30,60/5
(4) 3 pivot row
Operation O3: row operation pi
vot column
0
pivot row
(4)

basic feasible solution

nonbasic variable 0
basic feasible solution

z

x1

x2

x3

x4

1 40 88 0 0 0
2
8 1 0 60
T0 0
0
5
2 0 1 60

(4)

x3 row2 , x4 row3
21

20.3 Simplex Method

z

x1

x2

x3

22

20.3 Simplex Method

x4

Step 2 optimal -72 row 1

O1 O3
O1: column 3 ( -72 < 0)
O2: 36/7.2 = 5 60/2 = 30 7.2 pivot
() 3
O3: row operation
z x1 x2 x3
x4
b

8 480 Row1 8 Row3

1 0 72 0

T1 0 0 7.2 1 0.4 36 Row2 0.4 Row3

0 5
2 0
1 60
x1 x3 basic variable set non basic
variable x2 x4 0 basic feasible solution

0
10
4 840 Row1 10Row2
1 0
T2 0 0 7.2
1 0.4 36

1
1
2
Row2
0 5 0 36 0.9 50 Row3

7.2
23

24

20.3 Simplex Method

20 3 Simplex Method
20.3
x1, x2 basic x3, x4 non basic
set x3, x4 0 basic feasible solution T2

minimize z = f (x) (
maximize )
column pivot

row 1

pivot
i t

T2
row 1

z = f (10, 5) =(40)(10) + (88)(5) = 840

25

26

20.4 Simplex Method :

Simplex Difficulties

Degeneracy Difficulties in Starting

Degeneracy,

Problem #1: Degeneracy

g
y
degenerate feasible solution

1) Degeneracy ()
degenerate feasible solution feasible solution
0 (( n-m )) n

m
( x 0)
n = 4,

4 m = 2

0
n m = 2
solution

feasible solution
0
solve for optimum

Problem #2: Difficulties in Starting

Ex. the constraint condition, it leads to the violation of the constraint
condition

27

28

20 4 Simplex Method : Degeneracy

20.4

20 4 Simplex Method : Degeneracy

20.4

EX 1 Simplex Method, degenerate solution

. AB steel 2 I1 I2 3 R1 R2 R3 (
2 )

Solution x1 x2 I1, I2
maximize

29

30

20.4 Simplex Method :

Degeneracy Difficulties in Starting
Degeneracy,

20.4 Simplex Method : Degeneracy

normal form

31

32

20.4 Simplex Method :

Degeneracy Difficulties in Starting
Degeneracy,

20 4 Simplex Method : Degeneracy

20.4
z

x1

x2

x3

1 150 300
0
2
1
T0
0
1
1

0
1
0

x4

x5

x n = 5 m = 3 x
0 2 = n m
non degenerate feasible solution

0 0 0 0
1 0 0 16
0 1 0 8

0 0 1 3.5

x1, x2 nonbasic x3, x4, x5 basic

non basic variable

0
basic feasible solution

x1= 0 , x2= 0 , x3 = 16/1 = 16 , x4 = 8/1 = 8 , x5 = 3.5/1 = 3.5 ,
z = 0
O (0,0)
(0 0)
443
x n = 5 m = 3 x 0 2 =
5-3 ((n m)) non degenerate
g
feasible solution

Step 1 pivoting
O1: column pivot column 2 ( -150 < 0)
O2: row pivot 16/2 = 8, 8/1 = 8, 3.5/0

row 2 row 3
row 2 ( pivot = 2)
O3: row operation
p
simplex
p table T1

33

20.4 Simplex Method : Degeneracy

z
1
0
T1
0

x1

x2

0 225
2

1/ 2

x3

x4

34

20.4 Simplex Method :

Degeneracy Difficulties in Starting
Degeneracy,

x5 b

Step 2
pivoting
O1: column pivot column 3 ( -22.5 < 0)
O2: row pivot
16/1 = 16, 0/ = 0
pivot row 3
O3: row operation simplex table T2

0 0 1200 Row1 75Row2

1
0 0 16
1
1/ 2 1 0
0 Row3 Row2
2

0
0 1 3.5
Row4
75

x1, x4, x5 basic x2, x3 non basic

non
basic 0 basic feasible solution
x1= 16/2 = 8 , x2= 0 , x3 = 0 , x4= 0/1 = 0 , x5 = 3.5/1 = 3.5 , z = 1200
A: (8,0) 443
solution degenerate
0 3 n m

x4 = 0 A x4 non
basic
35

36

20.4 Simplex Method :

Degeneracy Difficulties in Starting
Degeneracy,
z
1
0
T2
0

x1

x2

x3 x4

x5

20.4 Simplex Method : Degeneracy

150 450 0 1200 Row1 450Row3

2 0 16 Row2 2Row3
2
0 1/ 2 1/ 2 1 0 0

2 1 3.5 Row4 2Row3

0 0
1
0
2

0
0

Row4
T3
z

x1

x2

1
0
T3
0

x3

x4

x5

0 150 150 1725 Row1 150 Row 4

2
0 2
9 Row 2 2 Row 4
1
R 3 Row
R 4
0 1 / 2 0 0 1 / 2 1.75 Row
2

0 0 1 2 1
3 .5

37

38

20.4 Simplex Method : Degeneracy

20 4 Simplex Method : Difficulties in Starting

20.4
Difficulties in Starting

basic feasible solution

start artificial variable

39

40

20 4 Simplex
20.4
Si l Method
M th d : Difficulties
Diffi lti iin St
Starting
ti

20 4 Simplex
20.4
Si l Method
M th d : Difficulties
Diffi lti iin St
Starting
ti
(7) (8) simplex table
z
1

0
T0
0

x1

x2

1
1

2
1
1

1
1
1

x3

x4

x5

b
0

1
2

0
0

1
0

0
1

41

20 4 Simplex
20.4
Si l Method
M th d : Difficulties
Diffi lti iin St
Starting
ti

42

20 4 Simplex
20.4
Si l Method
M th d : Difficulties
Diffi lti iin St
Starting
ti

z z Mx 6 2 x1 x 2 Mx 6 ( 2 M ) x1 (1

1
M ) x 2 Mx 3 M
2

(10 )

Mx6

slack variable

43

44

Simplex Table
(9)
(10)
M x (9);

x3 1 x1

1
x2 x6
2

1
z (2 M ) x1 (1 M ) x2 Mx3 M
2

z z Mx6 2 x1 x2 Mx6

1
x2 x3
2
x1 x2
x4

x1

1
x2 x6 )
2
1
Mx6 Mx3 M Mx1 Mx2
2
Mx3 M (1 x1

x1 x2

1
z z Mx6 2 x1 x2 ( Mx3 M Mx1 Mx2 )
2
1
z (2 M ) x1 (1 M ) x2 Mx3 M
2
1
z (2 M ) x1 (1 M ) x2 Mx3 M
2

x1

1
2

(8)

x5 4

x3 1 x1

Mx6 ((10))

(10)

1
x2 x6
2

1
x2 x3 x6 1
2

x1

x2

1 2 M 1 2 M M

1
1
0
2
T0
0
1

1
0

0
1
1 0

1
1
0
2

(9)

45

20 4 Simplex
20.4
Si l Method
M th d : Difficulties
Diffi lti iin St
Starting
ti
x1

x2

1 2 M 1 2 M M

1
1

1
0
2

T0
1
1 0
0
0
1
1 0

1
1

1
0
2

x4

x5

x6

T1 0

0 0 0 M

0 0 0
1

1 0 0
2
0 1 0
4

0 0 1
1

x5

x6

0 0 0 M

0 0 0
1

1 0 0
2
0 1 0
4

0 0 1
1

46

x3

x4

20 4 Simplex
20.4
Si l Method
M th d : Difficulties
Diffi lti iin St
Starting
ti

simplex table
z

x3

1 1
1
2 2
1
4 4
1

x1
0
1
0
0
0

x2

x3

2
1
2
1

2
3
2
0

x4

x5

x6 b
2 R1 R1 (2 M ) R2

1 R3 R3 R2

3 R4 R4 R2

0 R5 R5 R2

x1,x4, x5, x6 basic x2, x3 non basic

x2=0, x3 =0 x1 =1 ,row 2
x4 = 1,
1 row 3
x5 = 3, row 4
x6 = 0,
0 row 5
z = 2, row1
x1 =1, x2=0 A 444

1 1
1

x4, x5, x6 basic x1,x2, x3 non basic

Column 2 ( row 1) < 0 column pivot
1/1 < 2/1, < 4/1 row 2 row pivot
row operation
47

48

20 4 Simplex Method : Difficulties in Starting

20.4

20 4 Simplex
20.4
Si l Method
M th d : Difficulties
Diffi lti iin St
Starting
ti

5 7 Simplex Table
z

x1

x2

x3

x4

1 0 2
1

0 1 2
1
T2
0 0
2

3
0 0

x5

1 0 0 1

1 1 0 1

1 0 1 3

T3

1
2
1/ 2
1
2
1/ 2

3/ 2

0

0

x1

x2

3
2

x3

x4

3
2

3
4
3

x5
4
3
1
3
1
3
1

6 R1 R1 4 R4
3

1
2 R2 R2 R4
3

2 R3 R3 1 R4

2
3

4
3

1.5

3 /1 3

x1,x2, x4 basic x3, x5 non basic

x3=0, x5 =0 x1 =2 ,row 2
x4 = 2, row 3
x2 = (3x2)/3=2, row 4
z = 6, row1
x1 =2, x2=2 B 444, x3,x4, x5=0

column 3 3/2 pivot 4 row pivot Row

Operation

column 4 4/3 pivot row 3 row pivot

row operation

49

50

References

20 4 Simplex
20.4
Si l Method
M th d : Difficulties
Diffi lti iin St
Starting
ti
z

x1

4
3

3
2

1
2
1
2
1

3
4

x
3
2
1
2
1
3
3
4

Advanced Engineering Mathematics, 9th edition by

Erwin Kreyszig, John Wiley & Sons, Inc., 1999

Special Thanks to ..
for ppt of Simplex Method

MIT Math Lecture: Engineering Methods I - 31 - Method

s in Linear Programming

x1= 3, x 2= 1( C 444),
/ , 4=0,, x5=0
x3=3/2,x
fmax = f(3,1) =7

Prof. Gilbert Strang

http://video.google.com/videoplay?docid=75422656520210339
http://video.google.com/videoplay?docid 75422656520210339
5#

51

52