UCK 337 Introduction to Optimization

Lecture 1: Introduction

Assoc. Prof. N. Kemal Ure

Istanbul Technical University

ure@itu.edu.tr

March 1, 2022
Overview

1 Introduction

2 Problem Formulation

3 Examples

4 Types of Optimization Problems

5 Summary
Introduction
Introduction

▶ The main focus of this lecture is to give semi-formal answers to

following questions:

∎ What is optimization? Why do we need it?

∎ Is it useful for engineering?

∎ What kind of different optimization problems are out there?

▶ The objective is to give you a ”big picture” view of optimization and

motivate you for the rest of the class.
Problem Formulation
What is Optimization?

▶ Wikipedia: selection of a best element (with regard to some criteria)

from some set of available alternatives.

∎ Nice, but this is not useful. We need a formal (mathematical) description

▶ Mathematical Optimization (Programming)

∎ Set of elements (will be also called decision variables): x ∈ A

∎ Evaluate the cost of the element: f ∶ A → R

∎ Optimization Problem:
minimize f (x)
x∈A
Constrained and Unconstrained Optimization

▶ Generally A = Rn and we specify the constraints via functional

equalities and inequalities

▶ Unconstrained Problem: minimize f (x), f ∶ Rn → R

▶ Constrained Problem:

minimize f0 (x)
subject to fi (x) ≤ 0, i = 1, . . . , m

∎ Functions fi ∶ Rn → R represents constraints of the problem.

∎ We can embed the equality constraints by imposing fi (x) ≤ 0 and
−fi (x) ≤ 0 at the same time.

▶ Cool. But how is this connected to engineering?

Examples
Data Fitting

xi Experiment yi

▶ An experiment is conducted with inputs xi ∈ Rn . Outputs y i ∈ R are

observed, i = 1, . . . , ndata

▶ Your job is to find a linear function that explains the relationship

between x and y as good as possible

▶ Can we formulate this as an optimization problem? What is the set

of decisions? What is the cost function? What are the constraints?
Data Fitting

xi Experiment yi

▶ Let w ∈ Rn , so that y = wT x is a linear function. Here w is the

decision variable.
▶ The unconstrained formulation:
ndata
2
∑ [y − w x ]
i T i
minimize
i=1

▶ If we do not want the weights too large, we can add:

¿
Án 2
subject to ∥w∥2 = Á À∑ w ≤ R
i
i=1

∎ R is a positive number that constrains weights from growing too large

Spacecraft Control

▶ A satellite is orbiting the earth with

ability to control its orientation with
thrusters. The pitch angle of the
spacecraft is represented by θ ∈ [−π, π].
Let θ = [θ, θ̇].
▶ Spacecraft dynamics can be modeled as
a discrete time linear system

θk+1 = Aθk + Buk , k = 0, 1, . . . , tf

▶ Your job is to find a sequence of thruster commands uk such that

spacecraft is steered from θinit to θf inal in t = tf

▶ The fuel usage should be as low as possible. Thrusters can also exert
a limited amount of moment at per step.
Spaceraft Control

▶ Main idea: Objective function represents fuel usage, constraint

functions represent spacecraft dynamics and thruster limits

k=t −1
∑k=0 ∣uk ∣
f
minimize
subject to ∣uk ∣ ≤ umax , k = 0, . . . , tf − 1
θ0 − θinit = 0,
θtf − θf inal = 0,
θk+1 − Aθk − Buk = 0, k = 0, . . . , tf − 1

▶ Note that problem gets interesting when the dynamics are

time-variant A = Ak , B = Bk . We can handle that case too.

▶ Any optimal open loop linear control problem looks similar.

Design of a Wall Bracket

▶ Your job is to design a wall bracket that

supports W amount of load. The
material for the bars are fixed with
failure stress σa .
▶ You need to determine the cross
sectional areas A1 and A2 , such that the
total weight is minimized.
▶ Length of bars l1 and l2 can be
computed from geometry. Forces acting
bars F1 and F2 can be computed by
solving the statics equations
Design of a Wall Bracket

▶ Main idea: Cost function represents the weight, constraint functions

represent stress failure conditions and positivity of decision variables

minimize l1 A1 + l2 A2
subject to −A1 ≤ 0
−A2 ≤ 0,
A1 − σa ≤ 0,
F1

A2 − σa ≤ 0
F2

▶ Many other structural design problems can be formulated similarly

Example: Airline Crew Pairing

▶ Problem: We are given a list of flight nf schedules flights and a list

of np pairings.

∎ A pairing is a sequence of flights, each pairing can have different number

of flights.

∎ With each pairing there is an associated cost ci , i = 1, . . . , np

∎ We want to select a group of pairings such that every flight in our list is
covered and total cost is minimized.
Airline Crew Pairing

▶ Main Idea: Formulate each decision variable as a binary integer and

enforce the constraints so that each flight is covered at least once:

minimize c1 x1 + c2 x2 + . . . cn xn = cT x
subject to Ax ≥ 1
xi ∈ {0, 1}

▶ Here A is an nf × np matrix such that ai,j = 1 if flight i is included in

pairing j and ai,j = 0 otherwise.
Example: Airfoil Design

▶ Problem: We want to design an airfoil shape such that lift-to-drag

Cl /Cd is maximized

∎ Note that we simply cannot write an analytic expression that related the
shape parameters to aerodynamic parameters!

∎ We can use CFD though...

Airfoil Design

▶ Main Idea: We can simply model the CFD process as a black box,
nobody told us that our cost function f0 needs to be an explicit
formula

minimize −getLoDFromCFD(x)
subject to xmin ≤ x ≤ xmax

▶ Here getLoDFromCFD is a computer program, we do not have an

explicit formula for it.
More Examples

▶ We will be looking at many other examples throughout the class

∎ Airline Operations: Origin-Destination Pairs, Crew Scheduling
∎ Space Robotics: Shortest Path Problems, Descent Trajectory Generation
∎ Propulsion: Fuel Burning Rate, Nozzle Design
∎ Aircraft Design: Shape, structure and weight selection
∎ Avionics : Circuit Design, Sensor Design

▶ Good News: Almost any engineering problem can be cast as an

optimization problem!

▶ Bad News: We do not know how to efficiently solve the general

optimization problem. We need to understand the special cases we
can efficiently solve.
Types of Optimization Problems
Taxonomy of Optimization Problems

▶ The general problem is difficult for arbitrary f and fi

minimize f0 (x)
subject to fi (x) ≤ 0, i = 1, . . . , m

▶ There are certain exceptions which can be solved efficiently

∎ Least-squares problems
∎ Linear Programming
∎ Convex Optimization

▶ If you can formulate your problem as one of the above, you have
basically solved the problem

▶ Otherwise, you might need to develop new algorithms

Least-Squares Problems

minimize ∥Ax − b∥22

▶ Analytical solution exists! x∗ = (AT A)

−1
AT b

▶ There are efficient algorithms for computing the solution

▶ If A ∈ Rk×n , k ≥ n, runtime complexity is around n2 k

▶ It is easy to recognize least-squares problems

∎ Data Fitting is a least-squares problem!

∎ Just set A = [x1⊺ ; . . . ; xndata ⊺ ], b = [y 1 , . . . , y ndata ]⊺

Linear Programming

minimize c⊺ x
subject to a⊺i x ≤ bi , i = 1, . . . , m

▶ Cost and constraints are linear functions of the decision variable x

▶ No analytical solution
▶ There are efficient algorithms for computing the solution
▶ If x ∈ Rn , runtime complexity is around n2 m
▶ It is not as easy to recognize linear programming problems but there
are a few standard tricks
∎ Spacecraft Control is a linear programming problem! Cost function and
constraints are linear functions of uk
∎ Many other practical examples, such as airline scheduling.
Convex Optimization

▶ The general problem where f and fi are convex functions with

convex domains.

minimize f0 (x)
subject to fi (x) ≤ 0, i = 1, . . . , m

▶ Let θ1 , θ2 ≥ 0, θ1 + θ2 = 1
∎ Convex Function :f (θ1 x + θ2 y) ≤ θ1 f (x) + θ2 f (y)
∎ Convex Set: x, y ∈ A Ô⇒ θ1 x + θ2 y ∈ A
▶ Includes least-squares and linear programming as special cases
▶ Efficient polynomial time algorithms exist for finding global optimum,
even for nonlinear convex problems
▶ Can be difficult to recognize
∎ Wall Bracket design is a convex optimization problem!
∎ Many other important applications exist. Convex Optimization is at the
heart of modern applications.
Nonlinear Programming

▶ Back to the general form. f and fi are arbitrary (i.e. not convex)

minimize f0 (x)
subject to fi (x) ≤ 0, i = 1, . . . , m

▶ No general polynomial time algorithm exists for global optimum.

∎ Existing algorithms are either slow or get stuck in local optima
▶ Many challenging engineering problems are nonlinear and non-convex
∎ Airline crew pairing problem look linear, but it is not (because of integer
variables)
∎ Airfoil design problem is also non-convex (it is actually what we call a

black-box problem)

▶ There is still hope for small problem instances

∎ We have algorithms that can work with black-box problems
∎ We have convex/linear approximations to non-convex problems
Course Outline (Recap)

▶ Introduction (3 Weeks)
∎ Introduction to Optimization, Basics of Optimization

▶ Unconstrained Optimization (3 Weeks)

∎ 1D Search Methods, Gradient Methods, Newton’s Method

▶ Global Optimization (2 Weeks)

∎ Genetic Algorithms, Randomized Search, Cross-entropy methods

▶ Linear Optimization (2 Weeks)

∎ Linear Formulations, Simplex Method, Duality, Integer Programming

▶ Nonlinear Optimization and Case Studies (4 Weeks)

∎ Equality and Inequality Constraints, Convex Problems, Algorithms,
Multiobjective Optimization
Summary
Summary

▶ What we covered today:

∎ The formal definition of an optimization problem

∎ How most (if not all) engineering problems can be converted into
optimization problems

∎ The most important types of optimization problems we can address

▶ What is next?

∎ Reviewing mathematics and develop necessary and sufficient conditions

for optimality.