You are on page 1of 7

Topic and contents

School of Mathematics and Statistics


MATH3161/MATH5165 Optimization
Prof Jeya Jeyakumar

Topic 01 Optimization What is it?


1 Convex sets
Convex sets
Intersections
Extreme points
Convex combinations
Separating Hyperplane Theorem
2 Convex and concave functions
Convex functions
Concave functions
3 Global solutions
Convex programming problem
Concave programming problem
Linear programming
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 1 / 18
Optimization Modelling
Modelling
Denition (Optimization/Optimisation)
Optimization is nding the best way of doing something, subject to any
restrictions on what you can do.
Mathematical optimization is also known as
Mathematical (linear/nonlinear) programming
Optimal control
Meaning of best = concept of ordering
Business/Economics:
Maximize: prot, return, utility. . .
Minimize: cost, risk, . . .
Science/Biology/Chemistry/Physics:
Maximize: product, reaction rate, . . .
Minimize: energy, heat, . . .
Engineering
Maximize: strength, production, . . .
Minimize: cost, materials, time, . . .
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 2 / 18
Optimization Background
Background
Mathematical model
Variables
Objective
Constraints
Characterising optima Optimality conditions
What is an optimum/optimizer?
Formulae for identifying/characterizing optima
Derivatives I Newton (1642 1726), G W Leibniz (1646 1711)
Analytic methods for nding optima
Fermat (1601 1665), L Euler (1707 1783), Lagrange (1736 1813)
Finding optima = Numerical methods
Newtons method: I Newton (1642 1726), C F Gauss (1777 1855)
Linear programming: L Kantorovich (1912 1986), G B Dantzig (1914 2005)
Computer algorithms Linear algebra
Convexity R T Rockafellar (1938 )
Duality J von Neumann (1903 1957)
Maximum principle L S Pontryagin (1908-1988)
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 3 / 18
Optimization Variables
Variables
Decision variables: what can you change
Finite dimensional x R
n
, Number of variables n
Column vector
x =
_

_
x
1
x
2
.
.
.
x
n
_

_
Univariate: n = 1, x R
Multivariate: n = 2, 3, 4, . . . up to millions
Mathematical background: column vectors, symmetric matrices, matrix
operations
Discrete optimization: x
i
{0, 1}, x
i
N, x
i
Z
Matrix of variables X R
mm
=x R
m
2
Innite dimensional: the control
A function u C([a, b])
A function u L
1
([0, T])
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 4 / 18
Optimization Objective
Objective
A mathematical function of the variables quantifying the idea of best.
Finite dimensional: variables x R
n
, Objective function f : R
n
R
Linear f(x) = g
T
x, g R
n
xed
Ane f(x) = g
T
x + f
0
, g R
n
, f
0
R xed
Quadratic f(x) =
1
2
x
T
Gx + g
T
x + f
0
, G R
nn
, g R
n
, f
0
R
Innite dimensional: f : C([0, T] R, variables u C([0, T])
f(u) =
_
T
0
u(t)dt
Co-domain of objective function must be ordered (total order)
If , , R, then
and =
and =
Either or
If u, v R
n
, n 2,
u v v u 0 (v
i
u
i
) 0, i = 1, . . . , n
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 5 / 18
Optimization Constraints
Constraints
Constraints: Describe restrictions on the allowable values of variables
Constraint structure for variables x R
n
Equality constraints c
i
(x) = 0, i = 1, . . . , m
E
Inequality constraints c
i
(x) 0, i = m
E
+ 1, . . . , m
Feasible region R
n
= {x R
n
: c
i
(x) = 0, i = 1, . . . , m
E
;
c
i
(x) 0, i = m
E
+ 1, . . . , m}
Unconstrained problem = R
n
Standard formulation c
i
(x) 0 c
i
(x) 0
Algebraic structure of constraints
Simple bounds: x u
i
x
i
u
i
, i = 1, . . . , n
Linear constraints c
i
(x) = a
T
i
x b
i
Nonlinear constraints
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 6 / 18
Optimization Constraints
Constraint representation
Example (Constraint representation)
What feasible regions do the following constraints represent?
_
(x
1
a
1
)
2
+ (x
2
a
2
)
2
= r
_
(x
1
a
1
)
2
+ (x
2
a
2
)
2
r
_
(x
1
a
1
)
2
+ (x
2
a
2
)
2
r
x
2
= x
2
1
x
2
x
2
1
x
2
1
1 = 0
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 7 / 18
Optimization Standard formulation
Standard formulation
Denition (Standard formulation)
The standard formulation of a continuous nite dimensional optimization is
Minimize f(x)
x R
n
subject to c
i
(x) = 0, i = 1, . . . , m
E
;
c
i
(x) 0, i = m
E
+ 1, . . . , m
Conversions
Maximize to minimize: max

f(x) = min

f(x)
Constraint right-hand-side: c
i
(x) = b
i
c
i
(x) b
i
= 0
Less than or equal to inequalities: c
i
(x) 0 c
i
(x) 0
Strict inequality: c
i
(x) < 0 c
i
(x) + 0, some > 0
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 8 / 18
Optimization Standard formulation
A simplied farm planning problem
Example (Farm Planning)
Farmer Jack has 100 acres to devote to wheat and corn and wishes to plan
his planting to maximize the expected revenue. Jack has only $800 in
capital to apply to planting the crops, and it costs $5 to plant an acre of
wheat and $10 for an acre of corn. Their other activities leave the Jack
family only 150 days of labour to devote to the crops. Two days will be
required for each acre of wheat and one day for an acre of corn. If past
experience indicates a return of $40 from each acre of wheat and $30 from
each acre of corn, how many acres of each should be planted to maximize
his revenue?
Pose this as an optimization problem in standard form.
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 9 / 18
Optimization Standard formulation
Post Oce Parcel problem
Example (Post Oce Parcel Problem)
At one time the post oce regulations were that
the length plus the girth of a parcel must not exceed 1.8 metres.
What is the parcel of largest volume that could be sent through the post?
Pose this as an optimization problem in standard form. Assume that
The parcel has rectangular sides
The length of the parcel is the longest edge
The girth is the distance around the parcel perpendicular to the
length. For a rectangular box, girth is 2(height + depth).
x
1
x
2
x
3
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 10 / 18
Optimization Standard formulation
Standard formulation Example 2
Example (Standard formulation)
Maximize x
2
1
(x
2
1)
2
(x
2
3)
2
x
2
/2 on the set
= {x R
2
: x
2
x
2
1
, x
1
1, x
2
2 + x
1
, x
2
4 x
1
}
Write this problem in standard form and plot the feasible region .
What is the feasible region if the rst constraint becomes x
2
= x
2
1
?
2 1.5 1 0.5 0 0.5 1 1.5 2
0
0.5
1
1.5
2
2.5
3
3.5
4
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 11 / 18
Mathematical background Vector norms
Measures of size - norms
If R then its magnitude (or absolute value) || is
|| =
_
if 0;
if 0.
In R
n
there are several possible denitions norms denoted using || ||.
Denition (Vector norm)
A vector norm on R
n
is a function ||.|| from R
n
to R such that
1
x 0 for all x R
n
and x = 0 x = 0.
2
x + y x +y for all x, y R
n
. (Triangle inequality)
3
x = || x for all R, x R
n
.
Example (Vector norms)
The most widely used vector norms are
1-norm x
1
=

n
i=1
|x
i
|
2-norm x
2
=
_
n
i=1
|x
i
|
2
_1
2
= (x
T
x)
1
2
-norm or maximum norm x

= max
i=1,...,n
|x
i
|
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 12 / 18
Mathematical background Optima and optimizers
Local and global minima
Denition (Global minimum)
A point x

is a global minimizer of f(x) over R


n

f(x

) f(x) for all x . The global minimum is f(x

).
Denition (Strict global minimum)
A point x

is a strict global minimizer of f(x) over R


n

f(x

) < f(x) for all x , x = x

.
Denition (Local minimum)
A point x

is a local minimizer of f(x) over R


n

there exists a > 0 such that f(x

) f(x) for all x with


x x

. Then f(x

) is a local minimum.
Denition (Strict local minimum)
A point x

is a strict local minimizer of f(x) over R


n

there exists a > 0 such that f(x

) < f(x) for all x with


0 < x x

.
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 13 / 18
Mathematical background Optima and optimizers
Local and global maxima
Denition (Global maximum)
A point x

is a global maximizer of f(x) over R


n

f(x

) f(x) for all x . The global maximum is f(x

).
Denition (Strict global maximum)
A point x

is a strict global maximizer of f(x) over R


n

f(x

) > f(x) for all x , x = x

.
Denition (Local maximum)
A point x

is a local maximizer of f(x) over R


n

there exists a > 0 such that f(x

) f(x) for all x with


x x

. Then f(x

) is a local maximum.
Denition (Strict local maximum)
A point x

is a strict local maximizer of f(x) over R


n

there exists a > 0 such that f(x

) > f(x) for all x with


0 < x x

.
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 14 / 18
Mathematical background Optima and optimizers
Local and global minima and maxima
Example (Local and global minima and maxima)
= [0, 5], f(x) =
_

_
(x 1)
2
if x 0.5;
0.25 if 0.5 < x 1;
1.25 (x 2)
2
if 1 < x 2.5;
x 1.5 if 2.5 < x 3;
1.5 0.25 sin((x 3)) if 3 < x.
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
Local and global extrema
x
f(x)
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 15 / 18
Mathematical background Optima and optimizers
Local and global minima and maxima
Solution (Local and global minima and maxima)
Consider f(x) on the interval = [0, 5] and the points
0, 0.5, 2, 2.5, 3, 3.5, 4.5, 5
x
(a)
= 0 is strict local maximizer with f
(a)
= 1;
any point x
(b)
in the interval [0.5, 1] is a local and global minimizer
with f
(b)
= 0.25 (but not strict as adjacent points have the same
function value);
x
(c)
= 2 is a strict local maximizer with f
(c)
= 1.25;
x
(d)
= 2.5 is a strict local minimizer with f
(d)
= 1;
x
(e)
= 3 is a strict local maximizer with f
(e)
= 1.5;
x
(f)
= 3.5 is a strict local minimizer with f
(f)
= 1.25;
x
(g)
= 4.5 is a strict local and global maximizer with f
(g)
= 1.75;
x
(h)
= 5 is a strict local minimizer with f
(h)
= 1.5.
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 16 / 18
Mathematical background Existence
Existence
Denition (Extrema)
The global/local extrema of f over are all the global/local minima and
all the global/local maxima
Proposition (Existence of global extrema)
Let be a compact set and let f be continuous on . Then the global
extrema of f over exist.
Finite dimensional R
n
is compact is closed and bounded
Example (Existence)
Find the global extrema, if they exist, for the following problems
f(x) = e
x
on = [0, 1]
f(x) = e
x
on = [0, )
f(x) = sin x on = [0, 2)
f(x) = cos x on = [0, )
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 17 / 18
Mathematical background Relaxation
Relaxation
Proposition (Relaxation)
If f : R
n
R and

then
min
x
f(x) min
x

f(x)
Proof.
Let x


and f(x

) be the global minimizer and minimum of f over



.
As

, x


=x

. Thus min
x
f(x) f(x

).
If you make the feasible region larger then the minimum value of
objective function must not increase
If you make the feasible region larger then the maximum value of
objective function must not decrease
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 18 / 18
Mathematical background Derivatives
Gradients
Denition (Gradient)
Let f : R
n
R be continuously dierentiable.
The gradient f : R
n
R
n
of f at x is
f(x) =
_

_
f(x)
x1
f(x)
x2
.
.
.
f(x)
xn
_

_
The gradient is a column vector with n elements
The gradient f(x) of f at x is orthogonal to the contour
{x R
n
: f(x) = f(x)}
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 19 / 18
Mathematical background Derivatives
Hessians
Denition (Hessian)
Let f : R
n
R be continuously dierentiable. The Hessian

2
f : R
n
R
nn
of f at x is

2
f(x) =
_

2
f(x)
x
2
1

2
f(x)
x1x2


2
f(x)
x1xn

2
f(x)
x2x1

2
f(x)
x
2
2


2
f(x)
x2xn
.
.
.
.
.
.
.
.
.
.
.
.

2
f(x)
xnx1

2
f(x)
xnx2


2
f(x)
x
2
n
_

_
The Hessian
2
f(x) is an n by n matrix
If f is twice continuously dierentiable at x then

2
f(x)
x
i
x
j
=

2
f(x)
x
j
x
i
for all i = j
That is the Hessian matrix G =
2
f(x) is symmetric (G
T
= G)
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 20 / 18
Mathematical background Derivatives
Gradient and Hessian Example
Find the gradient and Hessian of
f(x) = 2x
2
1
3x
2
2
+ 4x
1
x
2
+ 2x
1
+ 6x
2
+ 8.
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 21 / 18
Mathematical background Derivatives
Gradient and Hessian Exercise
Example (Gradients and Hessians)
Let f(x) = x
2
1
+ (x
2
1)
2
(x
2
3)
2
+ x
2
/2 and
c
1
(x) = x
2
1
+ x
2
,
c
2
(x) = x
1
+ 1,
c
3
(x) = x
1
+ x
2
2,
c
4
(x) = x
1
x
2
+ 4.
For each function f(x), c
i
(x), i = 1, 2, 3, 4
Find the gradient and Hessian
Determine if the function is linear, quadratic or nonlinear
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 22 / 18
Mathematical background Derivatives
Example 1 Plots
2 1.5 1 0.5 0 0.5 1 1.5 2
0.4
0.1
0.6
1.1
1.6
2.1
2.6
3.1
3.6
x
1
x
2
x
1
2
+ (x
2
1)
2
(x
2
3)
2
+ x
2
/2
21
15
9
6
4 3
2.5
2
1.75 2
2.5
2
1.75
2
3
4
6
9
1.5
0.75
1
0.5
1.25
1.5
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 23 / 18
Mathematical background Derivatives
Linear and Quadratic functions
Example (Linear and Quadratic functions)
Let f
0
R, g R
n
and G R
nn
, G symmetric, be xed.
Find the gradient f(x) and Hessian
2
f(x) for the
Linear function f(x) = g
T
x + f
0
Quadratic function f(x) =
1
2
x
T
Gx + g
T
x + f
0
Solution
f(x) = g
T
x + f
0
=
n

i=1
g
i
x
i
+ f
0
f(x) = g, does not depend on x

2
f(x) = 0 R
nn
, n by n zero matrix
f(x) =
1
2
x
T
Gx + g
T
x + f
0
=
1
2
n

i=1
n

j=1
x
i
G
ij
x
j
+
n

i=1
g
i
x
i
+ f
0
f(x) = Gx + g

2
f(x) = G, does not depend on x
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 24 / 18
Mathematical background Positive denite matrices
Positive denite matrices Denition
Denition
A real square matrix A R
nn
is
positive denite x
T
Ax > 0 for all x R
n
, x = 0
positive semi-denite x
T
Ax 0 for all x R
n
negative denite x
T
Ax < 0 for all x R
n
, x = 0
negative semi-denite x
T
Ax 0 for all x R
n
indenite there exist x, y R
n
: x
T
Ax > 0 and y
T
Ay < 0
Generalization of nonnegative (order) to symmetric matrices:
A B AB 0 AB positive semidenite
Theoretical denition, not a practical test
A is negative denite A is positive denite
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 25 / 18
Mathematical background Positive denite matrices
Positive denite matrices Eigenvalues
A symmetric matrix A R
nn
has n real eigenvalues
i
, i = 1, . . . , n
there exists an orthogonal matrix Q (Q
T
Q = I) such that
A = QDQ
T
where D = diag(
1
, . . . ,
n
) and Q = [v
1
v
2
v
n
]
where v
i
is an eigenvector of A corresponding to eigenvalue
i
Determinant: det (A) =

n
i=1

i
Trace: trace (A) :=

n
i=1
a
ii
=

n
i=1

i
Proposition
A symmetric matrix A R
nn
is
positive denite
i
> 0 for all i = 1, . . . , n
positive semi-denite
i
0 for all i = 1, . . . , n
negative denite
i
< 0 for all i = 1, . . . , n
negative semi-denite
i
0 for all i = 1, . . . , n
indenite there exist i, j :
i
> 0 and
j
< 0
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 26 / 18
Mathematical background Positive denite matrices
Positive denite matrices Cholesky factorization
Proposition (Cholesky factorization)
A symmetric matrix A R
nn
is positive denite the Cholesky
factorization A = R
T
R exists, R a nonsingular upper triangular matrix
Proof.
= Suppose the Cholesky factorization A = R
T
R exists. Then
x
T
Ax = x
T
(R
T
R)x = (Rx)
T
(Rx) = y
T
y =
n

i=1
y
2
i
0
where y = Rx. Moreover, as R is nonsingular,
x
T
Ax = 0 y = 0 Rx = 0 x = 0.
= Text book on Linear Algebra, eg Golub and van Loan
Matlab chol
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 27 / 18