Introduction

Optimization Examples
Notation
Problem Classification
Modeling

Chapter 6
Introduction to Optimization

School of Electrical & Computer Engineering, AAiT

Addis Ababa University.

Class notes on: Mathematical Foundations in Engineering,

ECEG 6209

1 / 34
 Introduction
Optimization Examples
Notation
Problem Classification
Modeling

Outline

1 Introduction
Motivation
Definition
2 Optimization Examples
3 Notation
4 Problem Classification
Convex vs. non-convex
Continuous vs. discrete optimization
Constrained vs. unconstrained
Single- vs. Multi-objective optimization
5 Modeling

2 / 34
 Introduction
Optimization Examples
Motivation
Notation
Definition
Problem Classification
Modeling

Motivation

Engineering system design comprises selecting one or more

variables to meet a set of objectives.
A better design is obtained if an appropriate cost function can
be reduced.
The design is optimum when the cost is the lowest among all
feasible designs.

3 / 34
 Introduction
Optimization Examples
Motivation
Notation
Definition
Problem Classification
Modeling

Definition

Optimization is the act of obtaining the best result under

given circumstances.
Optimization can be defined as the process of finding the
conditions that give the maximum or minimum value of a
function.
Mathematical optimization: make something the best it can
possibly be

minimize cost (objective) function

by choosing variables
subject to constraints

4 / 34
 Introduction
Optimization Examples
Notation
Problem Classification
Modeling

Mathematical optimization spans a large number of fields

Communications
Control
Signal processing
Design (automotive, aerospace, biomechanical)
Circuit design
Economics, management, etc.

5 / 34
 Introduction
Optimization Examples
Notation
Problem Classification
Modeling

Example 1: Simple optimization problem

Problem statement:
Find the maximal and minimal value of f (x , y ) = 3x + 4y
subject to the following constraints:

x + 2y ≤ 14,
3x − y ≥ 0,
x − y ≤ 2;

6 / 34
 Introduction
Optimization Examples
Notation
Problem Classification
Modeling

Example 1: Simple optimization problem ...

Solution:
Rewrite:
x + 2y ≤ 14 y ≤ −0.5x + 7
3x − y ≥ 0 ⇒ y ≥ 3x
x −y ≤2 y≤ x − 2
Plot and identify the feasibility region.

7 / 34
 Introduction
Optimization Examples
Notation
Problem Classification
Modeling

Example 1: Simple optimization problem ...

Solution:
Evaluate f (x , y ) at corner points:
Corner 1 Corner 2 Corner 3
y = −0.5x + 7 y = −0.5x + 7 y = 3x
y = 3x y =x −2 y =x −2
intersection at intersection at intersection at
(2, 6) (6, 4) (−1, −3)
f (x , y ) = 30 f (x , y ) = 34 f (x , y ) = −15
Identify the minimal and the maximal from the corner
points
- The maximum of f (x , y ) = 34 occurs at (6, 4), and
- The minimum of f (x , y ) = −15 occurs at (−1, −3).

8 / 34
 Introduction
Optimization Examples
Notation
Problem Classification
Modeling

Problem 1: Student diet problem

Problem statement:
A student has a choice of breakfast menu (eggs, cereals, tarts)
and a limited ($10) budget to fulfill his/her nutrition needs
(1000 calories, 100 g protein) at minimum cost. Eggs provide
500 calories and 50g protein and cost $3.50; cereals provide
500 calories and 40g protein and cost $3; tarts provide 600
calories and 20g protein and cost $2. How does he/she choose
his/her breakfast mix?

9 / 34
 Introduction
Optimization Examples
Notation
Problem Classification
Modeling

Problem 1: Student diet problem..

Formulation:
Let x T = [x1 , x2 , x3 ] represent the quantities of eggs, cereals
and tarts chosen for breakfast. Then, the optimization
problem is mathematically formulated as follows:

min f = 3.5x1 + 3x2 + 2x3

x1 ,x2 ,x3

subject to:

500(x1 + x2 ) + 600x3 ≥ 1000,

50x1 + 40x2 + 20x3 ≥ 100,
3.5x1 + 3x2 + 2x3 ≤ 10;
x1 , x2 , x3 ∈ Z

10 / 34
 Introduction
Optimization Examples
Notation
Problem Classification
Modeling

Problem 2: Simplified manufacturing problem

Problem statement:
A manufacturer produces two products: tables and chairs.
Each table requires 10 kg of material and 5 units of labor, and
earns $7.50 in profit. Each chair requires 5 kg of material and
12 units of labor, and earns $5 in profit. A total of 60 kg of
material and 80 units of labor are available. Find the best
production mix to earn maximum profit.

11 / 34
 Introduction
Optimization Examples
Notation
Problem Classification
Modeling

Problem 2: Simplified manufacturing problem..

Formulation:
Let x T = [x1 , x2 ] represent the quantities of tables and chairs
to be manufactured. Then, the optimization problem is
mathematically formulated as follows:

max f = 7.5x1 + 5x2

x1 ,x2

subject to:

10x1 + 5x2 ≤ 60,

5x1 + 12x2 ≤ 80,
x1 , x2 ∈ Z

12 / 34
 Introduction
Optimization Examples
Notation
Problem Classification
Modeling

Problem 3: Shortest distance problem

Problem statement:
Find the shortest distance from a given point (x0 , y0 ) to a
given curve: y = f (x ).
Formulation:
The optimization problem is mathematically formulated to
minimize the Euclidian distance from the given point to the
curve:
min f = (x − x0 )2 + (y − y0 )2
x ,y

subject to:
y = f (x )

13 / 34
 Introduction
Optimization Examples
Notation
Problem Classification
Modeling

Problem 4: Data-fitting problem

Problem statement:
Given a set of N data points (xi , yi ), i = 1, . . . , N, fit a
polynomial of degree m to the data such that the mean
square error N 2
i=1 (yi − f (xi )) is minimized.
P

Formulation:
Let the polynomial be given as: f (x ) = a0 + a1 x + · · · + am x m
then, the unconstrained optimization problem is formulated as:
N
X
min f = (yi − a0 − a1 x − · · · − am x m )2
a0 ,··· ,am
i=1

14 / 34
 Introduction
Optimization Examples
Notation
Problem Classification
Modeling

Problem 5: Knapsack problem

Problem statement:
Given an assortment of n items, where each item i has a value
ci > 0, and a weight wi > 0, fill a knapsack of given capacity
(weight W ) so as to maximize the value of the included items.
Formulation:
Without loss of generality, we assume that W = 1. Let
xi ∈ {0, 1} denote the event that item i is selected for
inclusion in the sack; then, the knapsack problem is
formulated as:
n
X n
X
max f = ci xi subject to: wi xi ≤ 1
ai
i=1 i=1

15 / 34
 Introduction
Optimization Examples
Notation
Problem Classification
Modeling

Problem 6: Investment problem

Problem statement:
Given the stock prices pi and anticipated rates of return ri
associated with a group of investments, choose a mix of
securities to invest a sum of $1M in order to maximize return
on investment.
Formulation:
Let xi ∈ {0, 1} express the inclusion of security i in the mix,
then the investment problem is modeled as the knapsack
problem.

16 / 34
 Introduction
Optimization Examples
Notation
Problem Classification
Modeling

Problem 7: Transportation problem

Problem statement:
Goods are to be shipped from m supply points with capacities:
s1 , s2 , . . . , sm to distribution points with demands:
d1 , d2 , . . . , dn . Given the transportation cost cij for each of the
network routes, find the optimum quantities, xij , to be
shipped along those routes to minimize total shipment cost.
Formulation:
Let xij denote the quantity to be shipped from node i to node
j; then, the optimization problem is formulated as:
X
min f = cij xij
xij
i,j

subject to:
X X
xij = si , for i = 1, . . . , m; xij = dj , for j = 1, . . . , n; xij ≥ 0
j i
17 / 34
 Introduction
Optimization Examples
Notation
Problem Classification
Modeling

Problem 8: Shortest path problem

Problem statement:
Find the shortest path from node p to node q in a connected
graph (V , E ), where V denotes the vertices and E denotes
the edges.
Formulation:
Let eij denote the edge incident to both nodes i and j and let
f : E → R represent a real-valued weight function; further, let
P = (v1 , v2 , . . . , vn ) denote a path, where v1 = p, vn = q;
then, the unconstrained single-pair shortest path problem is
formulated as:
n−1
X
min f = ei,i+1
n
i=1

18 / 34
 Introduction
Optimization Examples
Notation
Problem Classification
Modeling

Problem 8: Shortest path problem..

Alternatively, let xij denote a variable associated with eij then, an
integer programming formulation of the shortest path problem is
given as: X
min f = eij xij
xij
i,j

subject to:

X X  1,
for i = p;
xij − xji = −1, for i = q;

 0,
j j otherwise.

Note: the shortest path problem is a well-known problem in graph

theory and algorithms, such as Dijkstra’s algorithm or
Bellman-Ford algorithm, are available to solve variants of the
problem.
19 / 34
 Introduction
Optimization Examples
Notation
Problem Classification
Modeling

Problem 9: Packet routing

Problem statement:
There are n data flows that must be routed from a source
node s to a destination node t following one of two possible
links, with capacity u1 = 1 and u2 = 2 respectively.

The company handling link 2 is 30% more expensive than the

company handling link 1. The table below specifies the
demands to be routed and the cost on link 1. 20 / 34
 Introduction
Optimization Examples
Notation
Problem Classification
Modeling

Problem 9: Packet routing..

Demand Required capacity (Mbps) Cost on link 1

1 0.3 200
2 0.2 200
3 0.4 250
4 0.1 150
5 0.2 200
6 0.2 200
7 0.5 700
8 0.1 150
9 0.1 150
10 0.6 900

Formulate a mathematical program to minimize the routing cost of

all the demands.
21 / 34
 Introduction
Optimization Examples
Notation
Problem Classification
Modeling

Problem 9: Packet routing..

Problem formulation:
Parameters:
i: index on the set of demands F = {1, ..., n};
ai : capacity used by demand i (∀i ∈ F );
ci : routing cost for demand i on link 1 (∀i ∈ F );
p: cost percentage difference between routing on link 2 and
link 1;
u1 : capacity installed on link 1;
u2 : capacity installed on link 2.
Optimization variables:
xi1 = 1 if packet i is routed on link 1, 0 otherwise (∀i ∈ F );
xi2 = 1 if packet i is routed on link 2, 0 otherwise (∀i ∈ F ).

22 / 34
 Introduction
Optimization Examples
Notation
Problem Classification
Modeling

Problem 9: Packet routing..

n
X
min f = (ci xi1 + (p + 1)ci xi2 )
xi1 ,xi2
i=1

subject to:

xi1 + xi2 = 1 ∀i ∈ F ,
X
ai xi1 ≤ u1 ,
i=1
X
ai xi2 ≤ u2 .
i=1

23 / 34
 Introduction
Optimization Examples
Notation
Problem Classification
Modeling

Problem 9: Packet routing..

Exercise:
1 Solve the above example using Microsoft Excel’s Solver
package.
2 How would you change the model to generalize to a situation
with m possible parallel links between s and t?

24 / 34
 Introduction
Optimization Examples
Notation
Problem Classification
Modeling

Problem 10: Power control in wireless networks

Problem statement:
In wireless system with L transmitter-receiver pairs
(communication links), a reliable communication is
established if signal-to-interference and noise ratios (SINRs)

Gii Pi
γi (P) = PL
σi + j6=i Gij Pj

exceeds target value γ0 . Find minimum power allocation that

meets SINR constraints.

25 / 34
 Introduction
Optimization Examples
Notation
Problem Classification
Modeling

Problem 10: Power control in wireless networks..

Problem formulation:
L
X
min f = Pi
Pi
i=1

subject to:

γi (P) ≥ γ0 , i = 1, . . . , L
0 ≤ Pi ≤ Pmax , i = 1, . . . , L.

can be formulated, solved, and analyzed as a linear program.

26 / 34
 Introduction
Optimization Examples
Notation
Problem Classification
Modeling

Notations
The following notation is used:
R denotes the set of real numbers;
Rn denotes the set of real n-vectors;
Rm×n denotes the set of real m × n matrices;
f : Rn → Rm denotes an Rm valued function defined over Rn ;
Z denotes integer vectors;
Small bold face letters such as x, y are used to represent
vectors or points in Rn ;
capital bold face letters such as A, B are used to represent
matrices;
Aq represents qth column of A; and,
I represents an identity matrix.
27 / 34
 Introduction
Convex vs. non-convex
Optimization Examples
Continuous vs. discrete optimization
Notation
Constrained vs. unconstrained
Problem Classification
Single- vs. Multi-objective optimization
Modeling

Standard form
Mathematical optimization

min f (x )
x ∈Rn
subject to :
ci (x ) = 0, i = 1, . . . , ne
dj (x ) ≥ 0, j = 1, . . . , ni

Variables: x ∈ Rn ; Objective function: f : Rn → R

Equality constraint functions: ci : Rn → R
Inequality constraint functions: dj : Rn → R
Feasible set: D = {x ∈ Rn | ci (x ) = 0, dj (x ) ≥ 0}
Different optimization algorithms are appropriate for different
problem types.
28 / 34
 Introduction
Convex vs. non-convex
Optimization Examples
Continuous vs. discrete optimization
Notation
Constrained vs. unconstrained
Problem Classification
Single- vs. Multi-objective optimization
Modeling

Convex vs. non-convex

Convex problems: Convex objective and constraint functions:

g(αx + βy ) ≤ αg(x ) + βg(y )

LP (linear programming): linear objective and constraints.

Common in management, finance, economics.
QP (quadratic programming): quadratic objective, linear
constraints. Often arise as algorithm subproblems.
29 / 34
 Introduction
Convex vs. non-convex
Optimization Examples
Continuous vs. discrete optimization
Notation
Constrained vs. unconstrained
Problem Classification
Single- vs. Multi-objective optimization
Modeling

Convex vs. non-convex..

NLP (nonlinear programming): the objective or some
constraints are general nonlinear functions. Common in the
physical sciences.
Convex: a unique optimum (local solution = global solution)
NLP: A global optimum is desired, but can be difficult to find.

Figure: Local and global solutions for a nonlinear objective function.

Local optimization algorithms can be used to find the global

optimum (from different starting points) for NLPs.
30 / 34
 Introduction
Convex vs. non-convex
Optimization Examples
Continuous vs. discrete optimization
Notation
Constrained vs. unconstrained
Problem Classification
Single- vs. Multi-objective optimization
Modeling

Continuous vs. discrete optimization

Discrete: The feasible set is finite

Always non-convex
Many problems are NP-hard
Sub-types: combinatorial optimization, integer programming
Example: How many warehouses should we build?
Continuous: The feasible set is uncountably infinite
Continuous problems are often much easier to solve because
derivative information can be exploited
Example: How thick should airplane wing skin be?
Discrete problems are often reformulated as a sequence of
continuous problems (e.g. branch and bound methods)

31 / 34
 Introduction
Convex vs. non-convex
Optimization Examples
Continuous vs. discrete optimization
Notation
Constrained vs. unconstrained
Problem Classification
Single- vs. Multi-objective optimization
Modeling

Constrained vs. unconstrained

Unconstrained problems (ne = ni = 0) are usually easier to

solve
Constrained problems are thus often reformulated as a
sequence of unconstrained problems (e.g. penalty methods)

32 / 34
 Introduction
Convex vs. non-convex
Optimization Examples
Continuous vs. discrete optimization
Notation
Constrained vs. unconstrained
Problem Classification
Single- vs. Multi-objective optimization
Modeling

Single-objective vs. Multi-objective optimization

We may want to optimize two competing objectives f1 and f2

(e.g. manufacturing cost and performance)
Pareto frontier: set of candidate solutions among which no
solution is better than any other solution in both objectives

These problems are often solved using evolutionary algorithms.

33 / 34
 Introduction
Optimization Examples
Notation
Problem Classification
Modeling

Modeling
Modeling: the process of identifying the objective, variables,
and constraints for a given problem.
The more abstract the problem, the more difficult modeling
becomes.

34 / 34