Engineering Optimization

Concepts and Applications

Fred van Keulen

Matthijs Langelaar
CLA H21.1
A.vanKeulen@tudelft.nl
Engineering Optimization Concepts and Applications

Recap / overview
Optimization problem
Negative null form

Special topics

Model

Linear / convex problems

Definition

Sensitivity analysis

Checking

Topology optimization

Solution methods
Unconstrained problems

Constrained problems

Optimality criteria

Optimality criteria

Optimization algorithms

Optimization algorithms

Engineering Optimization Concepts and Applications

Summary optimality conditions

Conditions for local minimum of unconstrained problem:

First Order Necessity Condition:

f 0

Second Order Sufficiency Condition:

H positive definite

For convex f in convex feasible domain:

condition for global minimum:

Sufficiency Condition:

Engineering Optimization Concepts and Applications

f 0

Stationary point nature summary

y T Hy i

Definiteness H

Nature x*

Positive d.

Minimum

Positive semi-d.

Valley

Indefinite

Saddlepoint

Negative semi-d.

Ridge

Negative d.

Maximum

Engineering Optimization Concepts and Applications

Complex eigenvalues?
Question: what is the nature of a stationary point when

H has complex eigenvalues?

Answer: this situation never occurs, because H is

symmetric by definition. Symmetric matrices have real

eigenvalues (spectral theory).

Engineering Optimization Concepts and Applications

Nature of stationary points

Nature of initial position depends on load (buckling):
F
2

l
k1

F 6, l 2
k1 10, k 2 9.5

k2

dz l l cos 1 cos 2
1
1
2
2
k11 k 2 2 Fdz
2
2
k11 Fl sin 1 cos 2

0
k 2 2 Fl sin 2 cos 1

0

0

Fl sin 1 sin
k1 Fl cos 01 cos 2
k1 2k 2

Fcrit min
,

0 sin k12sin
Fl
2
k 2 Fl cos 1 lcos l 2
Fl
Fcrit 4.75

Engineering Optimization Concepts and Applications

Nature of stationary points (2)

F 45
37
6.75

1
Engineering Optimization Concepts and Applications

Unconstrained optimization
algorithms
Single-variable methods
0th order (involving only f )
1st order (involving f and f )
2nd order (involving f, f and f )

Multiple variable methods

Engineering Optimization Concepts and Applications

Why optimization algorithms?

Optimality conditions often cannot be used:
Function not explicitly known (e.g. simulation)
Conditions cannot be solved analytically

Example:

f x1 x2 x`1 x1e x2 x2 e x1
2

Stationary points:

f
2 x1
x2

2
x

x
x
0
1
1 2 e
x
1

f 0

f 1 x x e x2 2 x e x1 0
1 2
2
x2

Engineering Optimization Concepts and Applications

0 order methods: pro/con

th

Strengths:
No derivatives needed
Work also for

discontinuous / nondifferentiable functions

Easy to program
Robust
Engineering Optimization Concepts and Applications

Weaknesses:
(Usually) less efficient

than higher order

methods (many function
evaluations)

Minimization with one variable

Why?
Simplest case: good starting point
Used in multi-variable methods during line search

Setting:

min f ( x)

Iterative process:

s.t. x x x

Model

Optimizer

x
Engineering Optimization Concepts and Applications

Termination criteria
Stop optimization iterations when:
Solution is sufficiently accurate (check optimality criteria)
Progress becomes too slow:

xk xk 1 x ,

f ( xk ) f ( xk 1 ) f

Maximum resources have been spent

The solution diverges
Cycling occurs

Engineering Optimization Concepts and Applications

xb

xa

Brute-force approach
Simple approach: exhaustive search
f

L0
n points:
Final interval size =

Ln

Disadvantage: rather inefficient

Engineering Optimization Concepts and Applications

2
Ln
L0
n 1
x

Basic strategy of 0th order

methods for single-variable case
1. Find interval [a0, b0] that contains the minimum

(bracketing)
2. Iteratively reduce the size of the interval [ak, bk]

(sectioning)
3. Approximate the minimum by the minimum of a simple

interpolation function over the interval [aN, bN]

Sectioning methods:
Dichotomous search
Fibonacci method
Golden section method
Engineering Optimization Concepts and Applications

Bracketing the minimum

f

[a0, b0]
x
x1

x2 = x1+

x3 = x2+

x4 = x3+2

Starting point x1, stepsize , expansion parameter : user-defined

Engineering Optimization Concepts and Applications

Unimodality
Bracketing and sectioning methods work best for unimodal

functions:
An unimodal function consists of exactly one monotonically
increasing and decreasing part

Engineering Optimization Concepts and Applications

Dichotomous search
Conceptually simple

idea:

Main Entry: dichotomous

Pronunciation: dI-'kt-&-m&s also d&Function: adjective
: dividing into two parts

Try to split interval in half in each step

L0

a0

L0/2

f f :
Engineering Optimization Concepts and Applications

<< L0

b0

Dichotomous search (2)

Interval size after 1 step (2 evaluations):

1
L1 L0
2

L0

Interval size after m steps (2m evaluations):

L0
1

Lm m 1 m
2
2

Proper choice for

Lideal
m

L0
m
2

Engineering Optimization Concepts and Applications

Lm Lideal
L0
m

10
10 10 2 m

Dichotomous search (3)

Example: m = 10

ideal
10

10

Lm

log
L0

L0
L0
10
2
1024

ideal
L10
L0

10 10240

Ideal
interval
reduction

Engineering Optimization Concepts and Applications

L0
10240

Sectioning - Fibonacci
Situation:

Fibonacci,
1180?-1250?

minimum
bracketed
between x1
and x3 :
x1

x2

Test new points and reduce interval

Optimal point placement?

Engineering Optimization Concepts and Applications

x4

x3

Optimal

sectioning

Fibonacci method: optimal sectioning method

Given:
Initial interval [a0, b0]
Predefined total number of evaluations N, or:
Desired final interval size

Engineering Optimization Concepts and Applications

Fibonacci sectioning - basic idea

Start at final interval and use symmetry and maximum interval reduction:

IN

IN-1 = 2IN
IN-2 = 3IN
IN-3 = 5IN
IN-4 = 8IN
IN-5 = 13IN

<< IN

I N j F j 1 I N

F j Fibonacci number

I N j 2 I N j 1 I N j

I k I k 2 I k 1

Engineering Optimization Concepts and Applications

Sectioning Golden Section

For large N, Fibonacci fraction

section ratio (0.618034):

Golden section method

uses this constant interval

reduction ratio

Engineering Optimization Concepts and Applications

converges to golden

FN 1
FN

Sectioning - Golden Section

Origin of golden section:

I1 I 2 I 3
I1 I1 2 I1

1 0 1, 2
2

I2 = I1

1 5

5 1

0.618034
2
Final interval: I N I
N
1
Engineering Optimization Concepts and Applications

I1

I2 = I1
I3 = I2

Comparison sectioning methods

Evaluations
10

Lm

log
L0

Ideal dichotomous
interval reduction

Golden
section
Fibonacci

Example:
reduction to 2% of
original interval:
N
Dichotomous

12

Golden section 9
Fibonacci
(Exhaustive

8
99)

Conclusion: Golden section simple and near-optimal

Engineering Optimization Concepts and Applications

Quadratic interpolation
Three points of the bracket define interpolating quadratic

function:

~
f ( x) ax 2 bx c
New point evaluated at

minimum of parabola:
For minimum: a > 0!

ai

ai+1

bii+1

~
b
f ' 2ax b 0 xnew
2a

Shift xnew when very close to existing point

Engineering Optimization Concepts and Applications

xnew

Unconstrained optimization
algorithms
Single-variable methods
0th order (involving only f )
1st order (involving f and f )
2nd order (involving f, f and f )

Multiple variable methods

Engineering Optimization Concepts and Applications

Cubic interpolation
Similar to quadratic interpolation, but with 2 points and

derivative information:

~
f ( x) ax 3 bx 2 cx d
ai

Engineering Optimization Concepts and Applications

bi

Bisection method
Optimality conditions: minimum at stationary point

Root finding of

Similar to sectioning methods, but uses derivative:

Engineering Optimization Concepts and Applications

Secant method
Also based on root finding of
Uses linear interpolation

Engineering Optimization Concepts and Applications

Unconstrained optimization
algorithms
Single-variable methods
0th order (involving only f )
1st order (involving f and f )
2nd order (involving f, f and f )

Multiple variable methods

Engineering Optimization Concepts and Applications

Newtons method
Again, root finding of

Basis: Taylor approximation of

f :

f '( x h) f '( x) f ''( x)h o(h 2 )

f ' ( x)
h
f " ( x)
New guess:

f ' ( xk )
xk 1 xk hk xk
f " ( xk )

Engineering Optimization Concepts and Applications

Linear
approximation

Newtons method
Best convergence of all methods:

f
xk

xk+2
xk
xk+1

Unless it diverges
Engineering Optimization Concepts and Applications

xk+1

xk+2

Summary single variable methods

Bracketing +
Dichotomous sectioning
Fibonacci sectioning
Golden ratio sectioning

0th order

Quadratic interpolation

In practice:
additional tricks
needed to deal
with:

Cubic interpolation
Bisection method

1st order

Multimodality

2nd order

fluctuations
Round-off
errors
Divergence

Secant method
Newton method

And many, many more!

Engineering Optimization Concepts and Applications

Strong