_
x
1
x
2
.
.
.
x
n
_
_
Univariate: n = 1, x R
Multivariate: n = 2, 3, 4, . . . up to millions
Mathematical background: column vectors, symmetric matrices, matrix
operations
Discrete optimization: x
i
{0, 1}, x
i
N, x
i
Z
Matrix of variables X R
mm
=x R
m
2
Innite dimensional: the control
A function u C([a, b])
A function u L
1
([0, T])
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 4 / 18
Optimization Objective
Objective
A mathematical function of the variables quantifying the idea of best.
Finite dimensional: variables x R
n
, Objective function f : R
n
R
Linear f(x) = g
T
x, g R
n
xed
Ane f(x) = g
T
x + f
0
, g R
n
, f
0
R xed
Quadratic f(x) =
1
2
x
T
Gx + g
T
x + f
0
, G R
nn
, g R
n
, f
0
R
Innite dimensional: f : C([0, T] R, variables u C([0, T])
f(u) =
_
T
0
u(t)dt
Codomain of objective function must be ordered (total order)
If , , R, then
and =
and =
Either or
If u, v R
n
, n 2,
u v v u 0 (v
i
u
i
) 0, i = 1, . . . , n
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 5 / 18
Optimization Constraints
Constraints
Constraints: Describe restrictions on the allowable values of variables
Constraint structure for variables x R
n
Equality constraints c
i
(x) = 0, i = 1, . . . , m
E
Inequality constraints c
i
(x) 0, i = m
E
+ 1, . . . , m
Feasible region R
n
= {x R
n
: c
i
(x) = 0, i = 1, . . . , m
E
;
c
i
(x) 0, i = m
E
+ 1, . . . , m}
Unconstrained problem = R
n
Standard formulation c
i
(x) 0 c
i
(x) 0
Algebraic structure of constraints
Simple bounds: x u
i
x
i
u
i
, i = 1, . . . , n
Linear constraints c
i
(x) = a
T
i
x b
i
Nonlinear constraints
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 6 / 18
Optimization Constraints
Constraint representation
Example (Constraint representation)
What feasible regions do the following constraints represent?
_
(x
1
a
1
)
2
+ (x
2
a
2
)
2
= r
_
(x
1
a
1
)
2
+ (x
2
a
2
)
2
r
_
(x
1
a
1
)
2
+ (x
2
a
2
)
2
r
x
2
= x
2
1
x
2
x
2
1
x
2
1
1 = 0
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 7 / 18
Optimization Standard formulation
Standard formulation
Denition (Standard formulation)
The standard formulation of a continuous nite dimensional optimization is
Minimize f(x)
x R
n
subject to c
i
(x) = 0, i = 1, . . . , m
E
;
c
i
(x) 0, i = m
E
+ 1, . . . , m
Conversions
Maximize to minimize: max
f(x) = min
f(x)
Constraint righthandside: c
i
(x) = b
i
c
i
(x) b
i
= 0
Less than or equal to inequalities: c
i
(x) 0 c
i
(x) 0
Strict inequality: c
i
(x) < 0 c
i
(x) + 0, some > 0
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 8 / 18
Optimization Standard formulation
A simplied farm planning problem
Example (Farm Planning)
Farmer Jack has 100 acres to devote to wheat and corn and wishes to plan
his planting to maximize the expected revenue. Jack has only $800 in
capital to apply to planting the crops, and it costs $5 to plant an acre of
wheat and $10 for an acre of corn. Their other activities leave the Jack
family only 150 days of labour to devote to the crops. Two days will be
required for each acre of wheat and one day for an acre of corn. If past
experience indicates a return of $40 from each acre of wheat and $30 from
each acre of corn, how many acres of each should be planted to maximize
his revenue?
Pose this as an optimization problem in standard form.
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 9 / 18
Optimization Standard formulation
Post Oce Parcel problem
Example (Post Oce Parcel Problem)
At one time the post oce regulations were that
the length plus the girth of a parcel must not exceed 1.8 metres.
What is the parcel of largest volume that could be sent through the post?
Pose this as an optimization problem in standard form. Assume that
The parcel has rectangular sides
The length of the parcel is the longest edge
The girth is the distance around the parcel perpendicular to the
length. For a rectangular box, girth is 2(height + depth).
x
1
x
2
x
3
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 10 / 18
Optimization Standard formulation
Standard formulation Example 2
Example (Standard formulation)
Maximize x
2
1
(x
2
1)
2
(x
2
3)
2
x
2
/2 on the set
= {x R
2
: x
2
x
2
1
, x
1
1, x
2
2 + x
1
, x
2
4 x
1
}
Write this problem in standard form and plot the feasible region .
What is the feasible region if the rst constraint becomes x
2
= x
2
1
?
2 1.5 1 0.5 0 0.5 1 1.5 2
0
0.5
1
1.5
2
2.5
3
3.5
4
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 11 / 18
Mathematical background Vector norms
Measures of size  norms
If R then its magnitude (or absolute value)  is
 =
_
if 0;
if 0.
In R
n
there are several possible denitions norms denoted using  .
Denition (Vector norm)
A vector norm on R
n
is a function . from R
n
to R such that
1
x 0 for all x R
n
and x = 0 x = 0.
2
x + y x +y for all x, y R
n
. (Triangle inequality)
3
x =  x for all R, x R
n
.
Example (Vector norms)
The most widely used vector norms are
1norm x
1
=
n
i=1
x
i

2norm x
2
=
_
n
i=1
x
i

2
_1
2
= (x
T
x)
1
2
norm or maximum norm x
= max
i=1,...,n
x
i

MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 12 / 18
Mathematical background Optima and optimizers
Local and global minima
Denition (Global minimum)
A point x
f(x
).
Denition (Strict global minimum)
A point x
f(x
.
Denition (Local minimum)
A point x
. Then f(x
) is a local minimum.
Denition (Strict local minimum)
A point x
.
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 13 / 18
Mathematical background Optima and optimizers
Local and global maxima
Denition (Global maximum)
A point x
f(x
).
Denition (Strict global maximum)
A point x
f(x
.
Denition (Local maximum)
A point x
. Then f(x
) is a local maximum.
Denition (Strict local maximum)
A point x
.
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 14 / 18
Mathematical background Optima and optimizers
Local and global minima and maxima
Example (Local and global minima and maxima)
= [0, 5], f(x) =
_
_
(x 1)
2
if x 0.5;
0.25 if 0.5 < x 1;
1.25 (x 2)
2
if 1 < x 2.5;
x 1.5 if 2.5 < x 3;
1.5 0.25 sin((x 3)) if 3 < x.
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
Local and global extrema
x
f(x)
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 15 / 18
Mathematical background Optima and optimizers
Local and global minima and maxima
Solution (Local and global minima and maxima)
Consider f(x) on the interval = [0, 5] and the points
0, 0.5, 2, 2.5, 3, 3.5, 4.5, 5
x
(a)
= 0 is strict local maximizer with f
(a)
= 1;
any point x
(b)
in the interval [0.5, 1] is a local and global minimizer
with f
(b)
= 0.25 (but not strict as adjacent points have the same
function value);
x
(c)
= 2 is a strict local maximizer with f
(c)
= 1.25;
x
(d)
= 2.5 is a strict local minimizer with f
(d)
= 1;
x
(e)
= 3 is a strict local maximizer with f
(e)
= 1.5;
x
(f)
= 3.5 is a strict local minimizer with f
(f)
= 1.25;
x
(g)
= 4.5 is a strict local and global maximizer with f
(g)
= 1.75;
x
(h)
= 5 is a strict local minimizer with f
(h)
= 1.5.
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 16 / 18
Mathematical background Existence
Existence
Denition (Extrema)
The global/local extrema of f over are all the global/local minima and
all the global/local maxima
Proposition (Existence of global extrema)
Let be a compact set and let f be continuous on . Then the global
extrema of f over exist.
Finite dimensional R
n
is compact is closed and bounded
Example (Existence)
Find the global extrema, if they exist, for the following problems
f(x) = e
x
on = [0, 1]
f(x) = e
x
on = [0, )
f(x) = sin x on = [0, 2)
f(x) = cos x on = [0, )
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 17 / 18
Mathematical background Relaxation
Relaxation
Proposition (Relaxation)
If f : R
n
R and
then
min
x
f(x) min
x
f(x)
Proof.
Let x
and f(x
=x
. Thus min
x
f(x) f(x
).
If you make the feasible region larger then the minimum value of
objective function must not increase
If you make the feasible region larger then the maximum value of
objective function must not decrease
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 18 / 18
Mathematical background Derivatives
Gradients
Denition (Gradient)
Let f : R
n
R be continuously dierentiable.
The gradient f : R
n
R
n
of f at x is
f(x) =
_
_
f(x)
x1
f(x)
x2
.
.
.
f(x)
xn
_
_
The gradient is a column vector with n elements
The gradient f(x) of f at x is orthogonal to the contour
{x R
n
: f(x) = f(x)}
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 19 / 18
Mathematical background Derivatives
Hessians
Denition (Hessian)
Let f : R
n
R be continuously dierentiable. The Hessian
2
f : R
n
R
nn
of f at x is
2
f(x) =
_
2
f(x)
x
2
1
2
f(x)
x1x2
2
f(x)
x1xn
2
f(x)
x2x1
2
f(x)
x
2
2
2
f(x)
x2xn
.
.
.
.
.
.
.
.
.
.
.
.
2
f(x)
xnx1
2
f(x)
xnx2
2
f(x)
x
2
n
_
_
The Hessian
2
f(x) is an n by n matrix
If f is twice continuously dierentiable at x then
2
f(x)
x
i
x
j
=
2
f(x)
x
j
x
i
for all i = j
That is the Hessian matrix G =
2
f(x) is symmetric (G
T
= G)
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 20 / 18
Mathematical background Derivatives
Gradient and Hessian Example
Find the gradient and Hessian of
f(x) = 2x
2
1
3x
2
2
+ 4x
1
x
2
+ 2x
1
+ 6x
2
+ 8.
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 21 / 18
Mathematical background Derivatives
Gradient and Hessian Exercise
Example (Gradients and Hessians)
Let f(x) = x
2
1
+ (x
2
1)
2
(x
2
3)
2
+ x
2
/2 and
c
1
(x) = x
2
1
+ x
2
,
c
2
(x) = x
1
+ 1,
c
3
(x) = x
1
+ x
2
2,
c
4
(x) = x
1
x
2
+ 4.
For each function f(x), c
i
(x), i = 1, 2, 3, 4
Find the gradient and Hessian
Determine if the function is linear, quadratic or nonlinear
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 22 / 18
Mathematical background Derivatives
Example 1 Plots
2 1.5 1 0.5 0 0.5 1 1.5 2
0.4
0.1
0.6
1.1
1.6
2.1
2.6
3.1
3.6
x
1
x
2
x
1
2
+ (x
2
1)
2
(x
2
3)
2
+ x
2
/2
21
15
9
6
4 3
2.5
2
1.75 2
2.5
2
1.75
2
3
4
6
9
1.5
0.75
1
0.5
1.25
1.5
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 23 / 18
Mathematical background Derivatives
Linear and Quadratic functions
Example (Linear and Quadratic functions)
Let f
0
R, g R
n
and G R
nn
, G symmetric, be xed.
Find the gradient f(x) and Hessian
2
f(x) for the
Linear function f(x) = g
T
x + f
0
Quadratic function f(x) =
1
2
x
T
Gx + g
T
x + f
0
Solution
f(x) = g
T
x + f
0
=
n
i=1
g
i
x
i
+ f
0
f(x) = g, does not depend on x
2
f(x) = 0 R
nn
, n by n zero matrix
f(x) =
1
2
x
T
Gx + g
T
x + f
0
=
1
2
n
i=1
n
j=1
x
i
G
ij
x
j
+
n
i=1
g
i
x
i
+ f
0
f(x) = Gx + g
2
f(x) = G, does not depend on x
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 24 / 18
Mathematical background Positive denite matrices
Positive denite matrices Denition
Denition
A real square matrix A R
nn
is
positive denite x
T
Ax > 0 for all x R
n
, x = 0
positive semidenite x
T
Ax 0 for all x R
n
negative denite x
T
Ax < 0 for all x R
n
, x = 0
negative semidenite x
T
Ax 0 for all x R
n
indenite there exist x, y R
n
: x
T
Ax > 0 and y
T
Ay < 0
Generalization of nonnegative (order) to symmetric matrices:
A B AB 0 AB positive semidenite
Theoretical denition, not a practical test
A is negative denite A is positive denite
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 25 / 18
Mathematical background Positive denite matrices
Positive denite matrices Eigenvalues
A symmetric matrix A R
nn
has n real eigenvalues
i
, i = 1, . . . , n
there exists an orthogonal matrix Q (Q
T
Q = I) such that
A = QDQ
T
where D = diag(
1
, . . . ,
n
) and Q = [v
1
v
2
v
n
]
where v
i
is an eigenvector of A corresponding to eigenvalue
i
Determinant: det (A) =
n
i=1
i
Trace: trace (A) :=
n
i=1
a
ii
=
n
i=1
i
Proposition
A symmetric matrix A R
nn
is
positive denite
i
> 0 for all i = 1, . . . , n
positive semidenite
i
0 for all i = 1, . . . , n
negative denite
i
< 0 for all i = 1, . . . , n
negative semidenite
i
0 for all i = 1, . . . , n
indenite there exist i, j :
i
> 0 and
j
< 0
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 26 / 18
Mathematical background Positive denite matrices
Positive denite matrices Cholesky factorization
Proposition (Cholesky factorization)
A symmetric matrix A R
nn
is positive denite the Cholesky
factorization A = R
T
R exists, R a nonsingular upper triangular matrix
Proof.
= Suppose the Cholesky factorization A = R
T
R exists. Then
x
T
Ax = x
T
(R
T
R)x = (Rx)
T
(Rx) = y
T
y =
n
i=1
y
2
i
0
where y = Rx. Moreover, as R is nonsingular,
x
T
Ax = 0 y = 0 Rx = 0 x = 0.
= Text book on Linear Algebra, eg Golub and van Loan
Matlab chol
MATH3161/MATH5165 (Optimization) H01 Optimization What is it? Session 1, 2014 27 / 18