You are on page 1of 37

MAE 552 Heuristic Optimization

Lecture 3
January 28, 2002

Types of Optimization and Optimization


Algorithms
There are a many different types of optimization and even more
algorithms to perform it.
Why are there so many algorithms?
None of the traditional algorithms are robust.

For every different type of problem there is a different


solution method.
This can become a problem when a user only knows a limited
number of algorithms and tries to apply them to problems they
were not intended for.

Types of Optimization

Important Categories
Methods can be classified according to:

1. The type of problem being addressed


Convex vs. Multi-Modal
Continuous vs. Discrete
Constrained vs. Unconstrained
Linear, Quadratic and Nonlinear Programming

Analytical or Numerical

Important Categories
Methods can be classified according to:

2. The type of technique being used


Deterministic vs. Random
Derivative-based vs. Derivative free
Global vs. Local

Types of Problems: Convex versus Multimodal


1. Strictly Convex Problems have a single optima

Local Optima = Global Optima

Hessian matrix has positive eigenvalues


F(x)

x*

Types of Problems: Convex versus Multimodal


1. Non Convex Problems may have multiple local optima

F(x)

Types of Problems: Continuous vs. Discrete


1. In continuous problems x is defined over a range

xlxxu

Examples: length, weight, temperature etc.


F(x)

Types of Problems: Continuous vs. Discrete


1. In discrete problems x is defined only for specific values

xlxxu where x can take on only specific values

Examples: Material Type, routes taken, standardized sizing


F(x)

Types of Problems: Linear, Quadratic and Nonlinear F


1. Problems with Linear or Quadratic Objective Functions and
constraints are much easier to solve than those with general
nonlinear terms.
Examples:
Linear F=ax1 + bx2+ c
Quadratic F=F=ax1 + bx2+ cx22 +d
Nonlinear F=ax1 + bx22+ cx25 +x1x21/2

Types of Problems:Analytical or Numerical

Analytical Problems: Equations available to calculate


objective function and constraints

F(x)=ax1 + bx2+ c
g1(x)=x1+x20

Numerical Problems: Analysis codes used to determine


objective function and constraints

ANALYSIS

F(x)
g(x)

Type of Technique : Deterministic vs. Random


Deterministic Methods:

With a given set of initial conditions these algorithms always


follow the same path.

Xo

Examples: Enumerative Search, Steepest Descent, GRG,


Conjugate Gradient, Powells

Type of Technique : Deterministic vs. Random


Random Methods:

With a given set of initial conditions these algorithms follow


different paths with each run
Xo

Examples: Scatter Search, Genetic Algorithms, Simulated


Annealing

Type of Technique : Derivative-based/Derivative free


Derivative Based Methods:

First and Second Derivatives of the objective Function are


calculated either numerically of analytically
Examples: Steepest Descent, Newtons Method, Conjugate
Gradient, Varible Metric Method etc.
Derivative Free Methods:
Only F(X) information is used to guide the search
Examples: Powells Search, Random Search, SA, GA, etc.

Type of Technique :Local or Global Optimizer


Local Search Methods: Find best design within a portion of the
design space usually a convex space (neighborhood).
These methods include almost all of the traditional numerical
techniques zero-order and derivative-based.
Tend to get stuck in valleys of the design space
F(x)

Global
Opt.

Local Opt.

x*

Local Opt.

Type of Technique :Local or Global Optimizer


Global Search Methods: Find best design within entire design
space.
Tend to be robust but inefficient and expensive to run.

F(x)

Global
Opt.

Local Opt.

x*

Local Opt.

Choosing the Right Method


1. It is very important to match the optimization method to the
problem that you are solving.
2. Know the assumptions involved with each method.

Example: LP requires linear objective and constraints.

3. Know the relative efficiency of each proposed method.

Example: Using a Enumerative Search to solve a convex


problem will be very inefficient.

4. Some problems are intractable!!!

There are no methods that can guarantee a global optima


in finite time. Use methods that find good solutions fast.

Basics of problem Solving


Which algorithm is right for a problem is strongly dependent on
how the problem is mathematically described.
Three basic concepts common to every algorithmic approach to
problem solving
1. A representation of the problem
2. The objective
3. The evaluation function

Basics of problem Solving- Representation


The representation encodes alternative candidate solutions for
manipulation and defines the size of the search space.
For example in NLP typically the design variables are
represented as real numbers in n dimensions. Is this the only
possibility?

In the TSP the representation we have used is a permutation of


the natural numbers 1,2,3,.,n.
The size of the search space is not determined by the problem, it
is determined by your representation and the way that you handle
this encoding.

Basics of problem Solving- Representation


Your choice of representation will determine how you will be
able to manipulate the potential solutions
Alternative Representations for a NLP
{x1,x2}={5,6}

{x1,x2}={101,110}

10

X2

0
0

x1

10

Basics of problem Solving-Objective


The objective represents mathematically the task to be achieved

For the TSP the objective is typically to minimize the distance


covered by the salesman subject to the constraint of visiting each
city once and only once and returning the starting city.
mindist (legal trip)
For NLP an example might be:
min F=x1+x2 +x3

Basics of problem Solving-Evaluation Function


The evaluation function is typically a mapping from the space
of possible candidate solutions to a set of numbers , where each
element from the space of possible solutions is assigned a
number to indicate its quality.
The EF allows you to compare solutions in terms of quality.

Ordinal EFs allow you to rank the candidate solutions.


Current Candidate is the tenth best
Numerical evaluations allow you to rank the solutions and also
their degree of quality.
Current Candidate is 1.3 units better that the second best
candidate

Basics of problem Solving-Evaluation Function


Since it could be computationally expensive to determine a
numerical value for how good or bad a candidate it might be
good enough to know approximately how good a solution is.

Candidate 1
Eval. Function
Candidate 2

Superior
Candidate

Basics of problem Solving-Evaluation Function


For every real world problem the evaluation problem is chosen
by the designer.
It should of course indicate for instance that a solution that
meets the objective is better than one that does not.
It should also depend on factors such as the computational
complexity of the problem.
Often the objective function indicates a good evaluation
function.

Objective - Minimize Stress -> Evaluation Function -Stress

Basics of problem Solving-Evaluation Function


Other times you cannot derive a useful Evaluation Function
from the objective:
In the SAT problem the objective is to find a set of boolean
(TRUE,FALSE) variables that satisfies a logical statement
(makes it TRUE).

All wrong candidate solutions return FALSE which does not tell
you how to improve the solution.

Basics of problem Solving-Defining a Search


Problem
When you design an evaluation function you need to consider that
for many problems the only solutions of interest are the subset that are
Feasible (satisfy the constraints).
The feasible space can be defined as F where F S.
A search problem can then be defined as:
Given a search space S and its feasible part F S find x F such that
eval(x) eval(y)

for all y F
Note that the objective does not appear at all in the formulation!!
If your EF does not correspond with the objective you will
searching for the answer to the wrong problem.

Basics of problem Solving-Defining a Search


Problem
A point x that satisfies the condition is called a global solution.
Finding a global solution can be difficult and impossible to prove in
some cases.
It would be easier if we could limit the search to a smaller area of S.
This fact underlies many search techniques.

Basics of problem Solving-Neighborhood Search


If we concentrate on the area of S near to some point in the search
space we can more easily look in this neighborhood.

N(x)

N(x) of x is a set of all points in the search space that close to the
given point x.
N(x) ={y S: dist(x,y)}

Basics of problem Solving-Neighborhood Search


For a continuous NLP the Euclidean distance can be used to define a
neighborhood.
n

dist ( x, y )

2
(
x

y
)
i i
i 1

For the TSP a 2-swap neighborhood can be defined as all of the


candidates that would result from swapping two cities in a given
tour.
A solution x (a permutation of n=5 cities)
1-2-3-4-5 has n(n-1)/2 neighbors including
1-3-2-4-5 (swapping cities 2 and 3)
5-2-3-4-1 (swapping cities 1 and 5)
etc.

Basics of problem Solving-Neighborhood Search


Example: Quadratic Objective with no Constraints

F=x2+3

xc

Basics of problem Solving-Neighborhood Search

Min F=x2+3
xc

Step 1: Define a neighborhood around point xc.

N(x): xc- x xc +

Basics of problem Solving-Neighborhood Search

F=x2+3
x1
xc

Step 2: Sample a candidate solution from the neighborhood and


evaluate it.
if F(x1)>F(xc) reject point and choose another.

Basics of problem Solving-Neighborhood Search

F=x2+3
xc

x1

if F(x1)<F(xc) accept point and choose replace current point xc with


x1.

Basics of problem Solving-Neighborhood Search

F=x2+3

xc

Step 3: Create new neighborhood around xc and repeat process.

Basics of problem Solving-Neighborhood Search


Most realistic problems are considerably more difficult than a
quadratic bowl problem.
The evaluation function defines a response surface that describes
the topography of the search space with many hills and valleys.

Basics of problem Solving-Neighborhood Search


Finding the best peak or the lowest valley is like trying to navigate
a mountain range in the dark with only a small lamp.
Your decisions must be made using local information. You can
sample points in a local area and then decide where to walk next.

Basics of problem Solving-Neighborhood Search


If you decide to always go uphill then you will reach a peak but not
necessarily the highest peak.
You may need to walk downhill in order to eventually reach the
highest peak in the space.