1.

Constrained Optimization
In a constrained optimization problem, the objective function is either a cost function (to be minimized) or a
reward/utility function (to be maximized).
Now, let’s examine situations where a function is constrained. A constraint is a hard limit we place on a variable’s
value to prevent the objective function from going forever in certain directions:
2.1. Optimum’s Location
With nonlinear functions, the optima (maximum or minimum values) can either occur at the boundaries or between
them:
As we can see, the maximum or minimum values can occur in the interior or at boundaries.
In contrast, with linear functions, the maximum or minimum values occur only at boundaries:
3. Forms of Constrained Optimization Problems
The following formulation describes the general (canonical) form of a constrained minimization problem:
where   is the vector of decision variables;   is the objective function that needs to be
optimized;   for   and   for   are constraints that are
required to be satisfied.
If we need to maximize the objective function, e.g., the utilization efficiency of network resources, we can negate it to
make it fit the general form:
3.1. Types of Constrained Optimization Problems
Depending on the objective function and constraints, there are several types of constrained optimization problems.
Linear Programming (LP) covers the cases in which the objective function is linear and all constraints are
also linear. If all decision variables are integers, then we’ll have an Integer Linear Programming (ILP) problem. If
only some of the decision variables are integers (the remaining are continuous), that’s a Mixed Integer Linear
Programming (MILP) problem.
For example:
Nonlinear Programming (NLP) refers to the problems where the objective function or some constraints are
nonlinear. Similar to linear programming, there are Integer Nonlinear Programming (INLP) and Mixed Integer
Nonlinear Programming (MINLP).
Here’s an example of a problem with nonlinear constraints:
Finally, Quadratic Programming (QP) problems are those with linear constraints but the objective function is
quadratic. Similar to linear programming and nonlinear programming problems, we also have Integer Quadratic
Programming (IQP) and Mixed Integer Quadratic Programming (MIQP) problems.
For instance:
2. UnConstrained Optimization
Unconstrained optimization methods seek a local minimum (or a local maximum) in the absence of restrictions,
that is,
f (x) −→ min (x ∈ D)
for a real-valued function f : D −→ R defined on a nonempty subset D of Rn for a given n ∈ N. Unconstrained
optimization involves the theoretical study of optimality criteria and above all algorithmic methods for a wide variety of
problems.
In section 2.0 we have repeated — as essential basics — the well-known (first- and second-order) optimality
conditions for smooth real-valued functions. Often constraints complicate a given task but in some cases they simplify
it. Even though most optimization problems in ‘real life’ have restrictions to be satisfied, the study

of unconstrained problems is useful for two reasons: Firstly, they occur directly in some applications, so they are
important in their own right. Secondly, unconstrained problems often originate as a result of transformations of
constrained optimization problems. Some methods, for example, solve a general problem by converting it into a
sequence of unconstrained problems.
In section 3.1 elementary search and localization methods like the Nelder–Mead polytope method and Shor’s
ellipsoid method are treated. The first method is widely used in applications, as the effort is small. The results,
however, are in general rather poor. Shor’s ellipsoid method has attracted great attention, mainly because of its
applications to linear optimization in the context of interior-point methods. Often methods proceed by finding a
suitable direction and then minimizing along this direction (“line search”). This is treated in section 3.2. Trust region
methods, which we are going to cover in section 3.3, start with a given step size or an up- per bound for it and then
determine a suitable search direction. In section 3.4 the concept of conjugate directions is introduced. If the objective
function is quadratic, the resulting method terminates after a finite number of steps. The extension to
any differentiable function f : Rn −→ R goes back to Fletcher/Reeves (1964). Quasi-Newton methods in section 3.5
are based on the “least change secant update principle” by C. G. Broyden and are thus well motivated.
Due to the multitude of methods presented, this chapter might at times read a bit like a ‘cookbook’. To enliven it, we
will illustrate the most important methods with insightful examples, whose results will be given in tabular form. In
order to achieve comparability, we will choose one framework example which picks up on our considerations from
exercise 20 in chapter 2 (classification, support vector machines). To facilitate understanding, we content ourselves,
at first, with the discussion of an example with only twelve given points for the different methods. At the end of the
chapter we will consider a more serious problem, breast cancer diagnosis, and compile important results of the
different methods in a table for comparison.