You are on page 1of 29

Introduction to Optimization

g
a rin
rjy ee
at ha in
ah tac eng
uw at il
G h iv
IT r B r C

i
, I ma fo
EL Ku ds
o
N ajib eth
Rajib Kumar Bhattacharjya

.R m
of ion
Pr zat

PT
Professor
i
im
Department of Civil Engineering
pt
O

IIT Guwahati
Email: rkbc@iitg.ernet.in

14 July 2021 R.K. Bhattacharjya/CE/IITG 1


Are you using optimization?

g
The word “optimization” may be very familiar or may be quite new to

a rin
rjy ee
you.

at ha in
ah tac eng
uw at il
G h iv
IT r B r C

i
, I ma fo
EL Ku ds
……. but whether you know about optimization or not, you are using

o
N ajib eth
optimization in many occasions of your day to day life …….

.R m
of ion
Pr zat

PT
i
im
pt
O
…………….Examples………………

7/14/2021 R.K. Bhattacharjya/CE/IITG 2


Optimization in real life

g
a rin
rjy ee
at ha in
ah tac eng
Newspaper
Cooking

uw at il
G h iv
hawker

IT r B r C

i
, I ma fo
EL Ku ds
o
N ajib eth
.R m
of ion
Pr zat

PT
i
im
pt
O

Forensic Ant colony


artist

7/14/2021 R.K. Bhattacharjya/CE/IITG 3


Nest
Food

7/14/2021
Nest
Food

O
pt
im
i
Pr zat
of ion
.R m
N ajib eth

R.K. Bhattacharjya/CE/IITG
PT o
EL Ku ds
, I ma fo
IT r B r C
G h iv
uw at il
ah tac eng
at ha in
i rjy ee
a rin
g
Nest
Food
Example
A farmer has 2400 m of fencing and wants to fence off a rectangular field that borders
a straight river. He needs no fence along the river. What are the dimensions of the
field that has the largest area?

g
a rin
rjy ee
Maximize 𝑓 = 𝑥𝑦

at ha in
1600

ah tac eng
x Subject to 𝑔 𝑥, 𝑦 = 𝑥 + 2𝑦 = 2400

uw at il
G h iv
IT r B r C

i
, I ma fo
400 400

EL Ku ds
1000

o
N ajib eth
y y

.R m
of ion
700 700

Pr zat

PT
2200

i
im
100 pt 100
O

RIVER

R.K. Bhattacharjya/CE/IITG
Example
A manufacturer needs to make a cylindrical can that will hold 1.5 liters of liquid.
Determine the dimensions of the can that will minimize the amount of material used in
its construction.

g
a rin
rjy ee
at ha in
ah tac eng
uw at il
G h iv
IT r B r C

i
, I ma fo
EL Ku ds
Minimize: A = 2πr2 + 2πrh

o
N ajib eth
.R m
of ion
Pr zat

PT
Constraint: πr2h = 1500
i
im
pt
O

1.5 liters = 1500 𝑐𝑚3


Dimension is in cm

R.K. Bhattacharjya/CE/IITG
Example
Objectives
Topology: Optimal connectivity of the
structure
Minimum cost of material: optimal cross

g
a rin
section of all the members

rjy ee
at ha in
ah tac eng
We will consider the second objective only

uw at il
G h iv
IT r B r C

i
, I ma fo
EL Ku ds
o
The design variables are the cross sectional area

N ajib eth
of the members, i.e. 𝐴1 to 𝐴7

.R m
of ion
Pr zat

PT
i
im
pt Using symmetry of the structure 𝐴7 = 𝐴1 , 𝐴6 = 𝐴2 , 𝐴5 = 𝐴3
O

You have only four design variables, i.e., 𝐴1 to 𝐴4

R.K. Bhattacharjya/CE/IITG
Optimization formulation
Objective

𝑀𝑖𝑛𝑖𝑚𝑖𝑧𝑒 𝑓 = 1.132𝐴1 𝑙 + 2𝐴2 𝑙 + 1.789𝐴3 𝑙 + 1.2𝐴4 𝑙

g
a rin
rjy ee
at ha in
ah tac eng
uw at il
What are the constraints?

G h iv
IT r B r C

i
, I ma fo
EL Ku ds
o
One essential constraint is non-negativity of design variables, i.e.

N ajib eth
.R m
𝐴1 , 𝐴2 , 𝐴3 , 𝐴4 ≥ 0

of ion
Pr zat

PT
i
im
pt
O

Is it complete now?

R.K. Bhattacharjya/CE/IITG
Optimization formulation
Member Force Member Force
𝐴𝐵 𝑃 𝐵𝐶 𝑃
− 𝑐𝑠𝑐𝜃 + 𝑐𝑠𝑐𝛼
2 2
𝐴𝐶 𝑃 𝐵𝐷 𝑃

g
a rin
+ 𝑐𝑜𝑡𝜃 − 𝑐𝑜𝑡𝜃 + 𝑐𝑜𝑡𝛼
2 2

rjy ee
at ha in
ah tac eng
uw at il
G h iv
IT r B r C

i
, I ma fo
EL Ku ds
o
First set of constraints Another constraint is buckling Another constraint may be the

N ajib eth
.R m
of compression members minimization of deflection at C

of ion
𝑃𝑐𝑠𝑐𝜃

Pr zat

PT
≤ 𝑆𝑦𝑐 𝑃 𝜋𝐸𝐴12

i
im
2𝐴1 ≤ pt 𝑃𝑙 0.566 0.500 2.236 2.700
2𝑠𝑖𝑛𝜃 1.281𝑙 2
O
+ + + ≤ 𝛿𝑚𝑎𝑥
𝑃𝑐𝑜𝑡𝜃 𝐸 𝐴1 𝐴2 𝐴3 𝐴4
≤ 𝑆𝑦𝑡
2𝐴2 𝑃 𝜋𝐸𝐴24
𝑐𝑜𝑡𝜃 + 𝑐𝑜𝑡𝛼 ≤
𝑃𝑐𝑠𝑐𝛼 2 5.76𝑙 2
≤ 𝑆𝑦𝑡
2𝐴3
𝑃
𝑐𝑜𝑡𝜃 + 𝑐𝑜𝑡𝛼 ≤ 𝑆𝑦𝑐 R.K. Bhattacharjya/CE/IITG
2𝐴4
Optimization formulation
𝑃𝑐𝑠𝑐𝜃
𝑆𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 𝑆𝑦𝑐 − ≥0
2𝐴1
𝑃𝑐𝑜𝑡𝜃
𝑆𝑦𝑡 − ≥0
2𝐴2

g
a rin
rjy ee
𝑃𝑐𝑠𝑐𝛼

at ha in
𝑆𝑦𝑡 − ≥0

ah tac eng
2𝐴3

uw at il
𝑃

G h iv
IT r B r C
𝑆𝑦𝑐 − 𝑐𝑜𝑡𝜃 + 𝑐𝑜𝑡𝛼 ≥ 0

i
, I ma fo
2𝐴4

EL Ku ds
o
N ajib eth
𝑀𝑖𝑛𝑖𝑚𝑖𝑧𝑒 𝑓 = 1.132𝐴1 𝑙 + 2𝐴2 𝑙 + 1.789𝐴3 𝑙 + 1.2𝐴4 𝑙 𝜋𝐸𝐴12 𝑃

.R m
− ≥0
1.281𝑙 2 2𝑠𝑖𝑛𝜃

of ion
Pr zat

PT
𝜋𝐸𝐴24 𝑃

i
im
pt − 𝑐𝑜𝑡𝜃 + 𝑐𝑜𝑡𝛼 ≥ 0
5.76𝑙 2 2
O

𝑃𝑙 0.566 0.500 2.236 2.700


𝛿𝑚𝑎𝑥 − + + + ≥0
𝐸 𝐴1 𝐴2 𝐴3 𝐴4

10 × 10−6 ≤ 𝐴1 , 𝐴2 , 𝐴3 , 𝐴4 ≤ 10 × 10−6
R.K. Bhattacharjya/CE/IITG
Traveling salesman problem

E
E A
A

g
a rin
rjy ee
E

at ha in
C

ah tac eng
A C

uw at il
G h iv
IT r B r C

i
, I ma fo
B D

EL Ku ds
C B D

o
N ajib eth
.R m
B

of ion
D Another solution: A B C D E
A possible solution: A B E D C

Pr zat

PT
i
im
pt
O

7/14/2021 R.K. Bhattacharjya/CE/IITG 11


Traveling salesman problem

g
a rin
rjy ee
at ha in
ah tac eng
Traveling

uw at il
G h iv
IT r B r C

i
salesman

, I ma fo
EL Ku ds
o
N ajib eth
problem traveling salesman problem

.R m
of ion
Pr zat

PT
i
im
pt
O

7/14/2021 R.K. Bhattacharjya/CE/IITG 12


What is Optimization?

• Optimization is the act of obtaining the best result under a given

g
a rin
rjy ee
circumstances.

at ha in
ah tac eng
uw at il
G h iv
IT r B r C

i
, I ma fo
• Optimization is the mathematical discipline which is concerned with

EL Ku ds
o
N ajib eth
finding the maxima and minima of functions, possibly subject to

.R m
of ion
constraints.

Pr zat

PT
i
im
pt
O

7/14/2021 R.K. Bhattacharjya/CE/IITG 13


Introduction to optimization
30

𝑓 = 𝑥−5 2
25
Minimum Equation of the line
20

15 How to find out the minimum of the function

g
a rin
rjy ee
10
𝑓′ = 2 × 𝑥 − 5 = 0

at ha in
ah tac eng
5

uw at il
G h iv
𝑥∗ = 5 Optimal solution

IT r B r C
0

i
, I ma fo
0 2 4 6 8 10

EL Ku ds
o
N ajib eth
.R m
of ion
30

Pr zat
𝑓 = 25 − 𝑥 2

PT
Equation of the line
i
im
25
pt
O
20
𝑓 ′ = 2𝑥 = 0
15

10 𝑥∗ = 0 Optimal solution
5
Maximum
0
-5 -3
7/14/2021 -1 1 3 5 14
Introduction to optimization
Equation of the surface
𝑓 𝑥, 𝑦 = − 𝑥 2 + 𝑦 2 + 4

g
a rin
rjy ee
at ha in
In this case, we can obtain the optimal

ah tac eng
solution by taking derivatives with respect

uw at il
G h iv
IT r B r C

i
to variable 𝑥 and 𝑦 and equating them to

, I ma fo
EL Ku ds
zero

o
N ajib eth
.R m
of ion
𝜕𝑓

Pr zat

PT
= −2𝑥 = 0 ⇒ 𝑥∗= 0
i
im
pt
O
𝜕𝑥

Optimal solution is 0,0 𝜕𝑓


= −2𝑦 = 0 ⇒ 𝑦∗= 0
𝜕𝑦

7/14/2021 15
7/14/2021
O
pt
im
i
Pr zat
of ion
.R m
N ajib eth
PT o
EL Ku ds
, I ma fo
IT r B r C
G h iv
Objective function is defined as

uw at il
ah tac eng
at ha in
i rjy ee
a rin
g
Minimization/Maximization f(x)
Single variable optimization

16
Single variable optimization

g
Stationary points

a rin
rjy ee
at ha in
For a continuous and differentiable function 𝑓(𝑥), a stationary point 𝑥 ∗ is a

ah tac eng
uw at il
point at which the slope of the function is zero, i.e.𝑓 ′ 𝑥 = 0 at 𝑥 = 𝑥 ∗ ,

G h iv
IT r B r C

i
, I ma fo
EL Ku ds
o
N ajib eth
.R m
of ion
Pr zat

PT
i
im
pt
O

Minima Maxima Inflection point

7/14/2021 R.K. Bhattacharjya/CE/IITG 17


Global minimum and maximum

A function is said to have a global or absolute


minimum at 𝑥 = 𝑥 ∗ if 𝑓(𝑥 ∗ ) ≤ 𝑓(𝑥) for all 𝑥 in the

g
a rin
rjy ee
domain over which 𝑓(𝑥) is defined.

at ha in
ah tac eng
uw at il
G h iv
IT r B r C

i
, I ma fo
EL Ku ds
o
N ajib eth
.R m
of ion
A function is said to have a global or absolute

Pr zat

PT
maximum at 𝑥 = 𝑥 ∗ if 𝑓(𝑥 ∗ ) ≥ 𝑓(𝑥) for all 𝑥 in
i
im
pt
O
the domain over which 𝑓(𝑥) is defined.

7/14/2021 R.K. Bhattacharjya/CE/IITG 18


7/14/2021
O
pt
im
i
Pr zat
of ion
.R m
N ajib eth
PT o
EL Ku ds
, I ma fo
f

IT r B r C
G h iv
uw at il
ah tac eng
at ha in
i rjy ee
a rin
g
X
Global optima
Introduction to optimization

Local optima

Local optima

Local optima
Local optima

19
Necessary and sufficient conditions for optimality

Necessary condition
If a function 𝑓(𝑥) is defined in the interval 𝑎 ≤ 𝑥 ≤ 𝑏 and has a relative minimum
at 𝑥 = 𝑥 ∗ , Where 𝑎 ≤ 𝑥 ∗ ≤ 𝑏 and if 𝑓 / 𝑥 exists as a finite number at 𝑥 = 𝑥 ∗ ,

g
a rin
rjy ee
then 𝑓 / 𝑥 ∗ = 0

at ha in
ah tac eng
Proof

uw at il
G h iv
IT r B r C

i
, I ma fo
𝑓 𝑥 ∗ + ℎ − 𝑓(𝑥 ∗ )

EL Ku ds
𝑓/ 𝑥∗ = lim

o

N ajib eth
ℎ→0

.R m
Since 𝑥 ∗ is a relative minimum 𝑓(𝑥 ∗ ) ≤ 𝑓 𝑥 ∗ + ℎ

of ion
Pr zat

PT
i
im
For all values of ℎ sufficiently close to zero, hence
pt
O

𝑓 𝑥 ∗ + ℎ − 𝑓(𝑥 ∗ )
≥0 if ℎ ≥ 0

𝑓 𝑥 ∗ + ℎ − 𝑓(𝑥 ∗ )
≤0 if ℎ ≤ 0

7/14/2021 R.K. Bhattacharjya/CE/IITG 20
Necessary and sufficient conditions for optimality
Thus
𝑓/ 𝑥∗ ≥ 0 If ℎ tends to zero through +ve value

g
𝑓/ 𝑥∗ ≤ 0

a rin
If ℎ tends to zero through -ve value

rjy ee
at ha in
ah tac eng
Thus only way to satisfy both the conditions is to have

uw at il
G h iv
IT r B r C

i
, I ma fo
EL Ku ds
o
N ajib eth
/ ∗
𝒇 𝒙 =𝟎

.R m
of ion
Pr zat

PT
i
im
Note: O
pt
• This theorem can be proved if 𝑥 ∗ is a relative maximum
• Derivative must exist at 𝑥 ∗
• The theorem does not say what happens if a minimum or maximum occurs at an end point of
the interval of the function
• It may be an inflection point also.
R.K. Bhattacharjya/CE/IITG
Sufficient conditions for optimality

Sufficient condition
Suppose at point 𝑥 ∗ , the first derivative is zero and first nonzero higher

g
a rin
rjy ee
derivative is denoted by 𝑛, then

at ha in
ah tac eng
uw at il
1. If 𝑛 is odd, 𝑥 ∗ is an inflection point

G h iv
IT r B r C

i
, I ma fo
𝑓′ 𝑥∗ = 0

EL Ku ds
o
N ajib eth
𝑓 ′′ 𝑥 ∗ = 0

.R m
2. If 𝑛 is even, 𝑥 ∗ is a local optimum

of ion
Pr zat

PT
𝑓3 𝑥∗ = 0
 If the derivative is positive, 𝑥∗
is a local minimum
i
im
pt
 If the derivative is negative, 𝑥 ∗ is a local maximum
O
𝑓4 𝑥∗ = 0

𝑓𝑛 𝑥∗ ≠ 0

7/14/2021 R.K. Bhattacharjya/CE/IITG 22


Sufficient conditions for optimality
Proof Apply Taylor’s series

ℎ2 ′′ ∗ ℎ𝑛−1 ℎ 𝑛
𝑓 𝑥 ∗ + ℎ = 𝑓 𝑥 ∗ + ℎ𝑓 ′ 𝑥 ∗ + 𝑓 𝑥 + ⋯+ 𝑓 𝑛−1 𝑥 ∗ + 𝑓 𝑛 𝑥 ∗

g
2! 𝑛−1 ! 𝑛!

a rin
rjy ee
at ha in
Since 𝑓 ′ 𝑥 ∗ = 𝑓 ′′ 𝑥 ∗ = ⋯ = 𝑓 𝑛−1 𝑥 ∗ = 0

ah tac eng
uw at il
G h iv
IT r B r C
ℎ𝑛 𝑛 ∗

i
, I ma fo
𝑓 𝑥∗ + ℎ − 𝑓 𝑥∗ = 𝑓 𝑥

EL Ku ds
𝑛!

o
N ajib eth
ℎ𝑛

.R m
When 𝑛 is even ≥0

of ion
𝑛!

Pr zat

PT
i
im
Thus if 𝑓 𝑛 𝑥 ∗ is positive
pt 𝑓 𝑥 ∗ + ℎ − 𝑓 𝑥 ∗ is positive Hence it is local minimum
O

Thus if 𝑓 𝑛 𝑥 ∗ negative 𝑓 𝑥 ∗ + ℎ − 𝑓 𝑥 ∗ is negative Hence it is local maximum

ℎ𝑛
When 𝑛 is odd, changes sign with the change in the sign of ℎ.
𝑛!

Hence it is an inflection point


R.K. Bhattacharjya/CE/IITG
Sufficient conditions for optimality

Take an example
𝑓 𝑥 = 𝑥 3 − 10𝑥 − 2𝑥 2 − 10

g
a rin
rjy ee
at ha in
𝑓 ′ 𝑥 = 3𝑥 2 − 10 − 4𝑥 = 0

ah tac eng
Apply necessary condition

uw at il
G h iv
IT r B r C

i
𝑥 ∗ = 2.61 𝑎𝑛𝑑 − 1.28

, I ma fo
Solving for 𝑥 These two points are stationary points

EL Ku ds
o
N ajib eth
.R m
of ion
Apply sufficient condition 𝑓 ′′ 𝑥 = 6𝑥 − 4

Pr zat

PT
i
im
pt
O
𝑓 ′′ 2.61 = 11.66 positive and n is odd 𝑓 ′′ −1.28 = −11.68 negative and n is odd

𝑥 ∗ = 2.61 is a minimum point 𝑥 ∗ = −1.28 is a maximum point

7/14/2021 R.K. Bhattacharjya/CE/IITG 24


Multivariable optimization without constraints

𝑥1
𝑥2
Minimize 𝑓(𝑋) Where 𝑋 = ⋮
𝑥𝑛

g
a rin
rjy ee
at ha in
ah tac eng
Necessary condition for optimality

uw at il
G h iv
IT r B r C

i
, I ma fo
If 𝑓(𝑋) has an extreme point (maximum or minimum) at 𝑋 = 𝑋 ∗ and if the first partial

EL Ku ds
o
N ajib eth
Derivatives of 𝑓 𝑋 exists at 𝑋 ∗ , then

.R m
of ion
Pr zat

PT
i
𝜕𝑓 𝑋 ∗ 𝜕𝑓 𝑋 ∗ im
𝜕𝑓 𝑋 ∗
pt
O

= =⋯= =0
𝜕𝑥1 𝜕𝑥2 𝜕𝑥𝑛

R.K. Bhattacharjya/CE/IITG
Multivariable optimization without constraints

Sufficient condition for optimality


The sufficient condition for a stationary point 𝑋 ∗ to be an extreme point is that the
matrix of second partial derivatives of 𝑓(𝑋) evaluated at 𝑋 ∗ is

g
a rin
rjy ee
at ha in
(1) positive definite when 𝑋 ∗ is a relative minimum

ah tac eng
uw at il
(2) negative definite when 𝑋 ∗ is a relative maximum

G h iv
IT r B r C

i
, I ma fo
(3) neither positive nor negative definite when 𝑋 ∗ is neither a minimum nor a maximum

EL Ku ds
o
N ajib eth
Proof Taylor series of two variable function

.R m
of ion
Pr zat

PT
𝜕𝑓 𝜕𝑓 1 𝜕2𝑓 𝜕2𝑓 𝜕2𝑓

i
2 2

im
𝑓 𝑥 + ∆𝑥, 𝑦 + ∆𝑦 = 𝑓 𝑥, 𝑦 + ∆𝑥 + ∆𝑦 + ∆𝑥 + 2∆𝑥∆𝑦 + ∆𝑦 +⋯
pt𝜕𝑥 𝜕𝑦 2! 𝜕𝑥 2 𝜕𝑥𝜕𝑦 𝜕𝑦 2
O

𝜕𝑓 𝜕2𝑓 𝜕2𝑓
𝜕𝑥 1 𝜕𝑥 2 𝜕𝑥𝜕𝑦 ∆𝑥
𝑓 𝑥 + ∆𝑥, 𝑦 + ∆𝑦 = 𝑓 𝑥, 𝑦 + ∆𝑥 ∆𝑦
𝜕𝑓 + 2!
∆𝑥 ∆𝑦 +⋯
𝜕2𝑓 𝜕 2 𝑓 ∆𝑦
𝜕𝑦 𝜕𝑥𝜕𝑦 𝜕𝑦 2
R.K. Bhattacharjya/CE/IITG
Multivariable optimization without constraints

1 𝑇
𝑓 𝑋 ∗ + ℎ = 𝑓 𝑋 ∗ + ℎ𝑇 𝛻𝑓 𝑋 ∗ + ℎ 𝑯ℎ + ⋯
2!
Another test
Since 𝑋 ∗ is a stationary point, the necessary

g
condition gives that 𝛻𝑓 𝑋 ∗ = 0

a rin
𝐴1 = 𝑎11

rjy ee
at ha in
Thus

ah tac eng
∗ ∗
1 𝑇 𝑎11 𝑎12
𝑓 𝑋 +ℎ −𝑓 𝑋 = ℎ 𝑯ℎ + ⋯ 𝐴2 = 𝑎

uw at il
G h iv
2! 21 𝑎22

IT r B r C

i
, I ma fo
𝑎11 𝑎12 𝑎13

EL Ku ds
Now, 𝑋 ∗ will be a minima, if ℎ𝑇 𝑯ℎ is positive

o
N ajib eth
𝐴3 = 𝑎21 𝑎22 𝑎23

.R m
𝑋 ∗ will be a maxima, if ℎ𝑇 𝑯ℎ is negative 𝑎31 𝑎32 𝑎33

of ion
Pr zat

PT
𝑎11 𝑎12 𝑎13 𝑎14 ⋯ 𝑎1𝑛
ℎ𝑇 𝑯ℎ will be positive if 𝑯 is a positive definite matrix

i
im
pt 𝑎21 𝑎22 𝑎23 𝑎24 ⋯ 𝑎2𝑛
𝐴𝑛 = ⋮
O
⋮ ⋮ ⋮ ⋯ ⋮
ℎ𝑇 𝐻ℎ will be negative if 𝑯 is a negative definite matrix 𝑎𝑛1 𝑎𝑛2 𝑎𝑛3 𝑎𝑛4 ⋯ 𝑎𝑛𝑛

A matrix 𝐻 will be positive definite if all the


eigenvalues are positive, i.e. all the 𝜆 values are  A matrix 𝑨 will be positive definite if any only if all the
positive which satisfies the following equation values 𝐴1 , 𝐴2 , 𝐴3 , …, 𝐴𝑛 are positive.
 The matrix will be negative definite is and only if the
𝑨 − 𝜆𝑰 = 0 R.K. Bhattacharjya/CE/IITG sign of 𝐴𝑗 is −1 𝑗 for 𝑗 = 1, 2, 3, … , 𝑛
Unimodal and duality principle

30

25

20 Optimal solution 𝑥 ∗ = 0

g
a rin
rjy ee
15

at ha in
Optimal solution 𝑥 ∗ = 0

ah tac eng
10

uw at il
G h iv
5

IT r B r C

i
, I ma fo
0

EL Ku ds
o
-5 -3 -1 1 3 5

N ajib eth
.R m
0

of ion
Pr zat

PT
-5 -3 -1 1 3 5
-5

i
im
pt
O
-10

-15 Minimization 𝑓 𝑥 = Maximization −𝑓 𝑥


-20

-25

-30
7/14/2021 R.K. Bhattacharjya/CE/IITG 28
7/14/2021
O
pt
im
i
Pr zat
of ion
.R m
N ajib eth

R.K. Bhattacharjya/CE/IITG
PT o
EL Ku ds
, I ma fo
IT r B r C
G h iv
uw at il
ah tac eng
at ha in
i rjy ee
a rin
g
29

You might also like