Introduction to Non-Linear Optimization

Optimization in Engineering Design

Georgia Institute of Technology Systems Realization Laboratory 1

Nonlinear Optimization Note:

Unlike for linear problems, a global optimum for a nonlinear problem cannot be guaranteed, except for special cases, e.g., if you know the space is unimodal, or convex, or monotonicity exists. Two standard heuristics that most people use: 1) find local extrema starting from widely varying starting points of the variables and then pick the most extreme of these extrema (if they are not the same) 2) perturb a local extremum by taking a finite amplitude step away from it, and then see whether your routine returns you to a better point or "always" to the same one.

Question: How would you "automate" a search for a global extremum?

Optimization in Engineering Design

Georgia Institute of Technology Systems Realization Laboratory 2

Basic Steps in Nonlinear Optimization

In its simplest form, a numerical search procedure consists of four steps when applied to unconstrained minimization problems: 1) Selection of an initial design in the n-dimensional space, where n is the number of design variables 2) A procedure for the evaluation of the function (objective function) at a given point in the design space. 3) Comparison of the current design with all of the preceding designs. 4) A rational way to select a new design and repeat the process. Constrained minimization requires step for evaluation of constraints as well. Same applies for evaluating multiple objective functions.
Georgia Institute of Technology Systems Realization Laboratory 3

Optimization in Engineering Design

Nonlinear Optimization Process

Simplistically, most design tasks seek to find a perturbation to an existing design
which will lead to an improvement. Thus, we seek a new design which is the old design plus a change :

Xnew = Xold + d X
Optimization algorithms use the same formula, but apply a two step process: Xk = Xk-1 + a Sk You (the engineer) have to provide an initial design X0. The optimization will then determine a search direction S k that will improve the

design. Next question is how far we can move in direction Sk before we must find a new search direction. This is a one-dimensional search since we only have to determine

the value of the scalar a to improve the design as much as possible.

Optimization in Engineering Design

Georgia Institute of Technology Systems Realization Laboratory 4

A Good Algorithm

A good algorithm is (among others): Robust algorithm must be reliable for general design applications and (thus) must theoretically converge to the solution point starting from any given starting point. General Should not impose restrictions on the model's constraints and objective functions. Accurate Ability to converge to precise mathematical optimum point is important, though it may not be required in practice.

Easy to use by both experienced and inexperienced users. Should not have problem dependent tuning parameters. Efficient To be efficient, the number of repeated analyses should be kept to a minimum. Hence, an efficient algorithm has 1) a faster rate of convergence requiring fewer iterations, and 2) least number of calculations within one (design) iteration.

Note: Tradeoffs have to be made

Optimization in Engineering Design

Georgia Institute of Technology Systems Realization Laboratory 5

Zero and first order algorithms

You often must choose between algorithms which need only evaluations of the objective function or methods that also require the derivatives of that function.

Algorithms using derivatives are generally more powerful, but do not always compensate for the additional calculations of derivatives.
Note that you may not be able to compute the derivatives.

Optimization in Engineering Design

Georgia Institute of Technology Systems Realization Laboratory 6

Basic Descent Methods

Basic descent methods are the basic techniques for iteratively solving unconstrained minimization problems.

Important for practical situations because they offer the simplest and most direct alternatives for obtaining solutions.
Also good as a benchmark.

Optimization in Engineering Design

Georgia Institute of Technology Systems Realization Laboratory 7

General Basic Descent Method Algorithm

Basic steps: 1. start at an initial point; 2. determine according to a fixed rule a direction of movement; and

3.
4.

move in that direction to a (relative) minimum of the objective function on that line. At the new point, a new direction is determined and the same process is repeated. The primary difference between algorithms (steepest descent, Newton's method, etc) is the rule by which successive directions of movement are selected.
The process of determining the minimum point on a line is called line search.
Georgia Institute of Technology Systems Realization Laboratory 8

Optimization in Engineering Design