This action might not be possible to undo. Are you sure you want to continue?
OPTIMIZATION IN SCILAB
This document has been written by Michaël Baudin and Vincent Couvert from the Scilab
Consortium and by Serge Steer from INRIA Paris  Rocquencourt.
© July 2010 The Scilab Consortium – Digiteo / INRIA. All rights reserved.
Abstract
In this document, we make an overview of optimization features in Scilab. The goal of this
document is to present all existing and nonexisting features, such that a user who wants to solve
a particular optimization problem can know what to look for. In the introduction, we analyse a
classiﬁcation of optimization problems. In the ﬁrst chapter, we analyse the ﬂagship of Scilab in
terms of nonlinear optimization: the optim function. We analyse its features, the management of
the cost function, the linear algebra and the management of the memory. Then we consider the
algorithms which are used behind optim, depending on the type of algorithm and the constraints.
In the remaining chapters, we present the algorithms available to solve quadratic problems, non
linear least squares problems, semideﬁnite programming, genetic algorithms, simulated annealing
and linear matrix inequalities. A chapter focus on optimization data ﬁles managed by Scilab,
especially MPS and SIF ﬁles. Some optimization features are available in the form of toolboxes,
the most important of which are the Quapro and CUTEr toolboxes. The ﬁnal chapter is devoted
to missing optimization features in Scilab.
Contents
Introduction 5
1 Nonlinear optimization 10
1.1 Mathematical point of view . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.2 Optimization features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.3 Optimization routines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.4 The cost function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.5 Linear algebra . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.6 Management of memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.7 QuasiNewton ”qn” without constraints : n1qn1 . . . . . . . . . . . . . . . . . . . 13
1.7.1 Management of the approximated Hessian matrix . . . . . . . . . . . . . . 15
1.7.2 Line search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
1.7.3 Initial Hessian matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
1.7.4 Termination criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
1.7.5 An example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
1.8 QuasiNewton ”qn” with bounds constraints : qnbd . . . . . . . . . . . . . . . . . 20
1.9 LBFGS ”gc” without constraints : n1qn3 . . . . . . . . . . . . . . . . . . . . . . . 20
1.10 LBFGS ”gc” with bounds constraints : gcbd . . . . . . . . . . . . . . . . . . . . . 21
1.11 Non smooth method without constraints : n1fc1 . . . . . . . . . . . . . . . . . . . 22
2 Quadratic optimization 23
2.1 Mathematical point of view . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.2 qpsolve . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.3 qp solve . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.4 Memory requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.5 Internal design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3 Nonlinear least square 26
3.1 Mathematical point of view . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.2 Scilab function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.3 Optimization routines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
4 Semideﬁnite programming 27
4.1 Mathematical point of view . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
4.2 Scilab function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
4.3 Optimization routines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
1
5 Genetic algorithms 28
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
5.2 Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
5.3 Support functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
5.3.1 Coding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
5.3.2 Crossover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
5.3.3 Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
5.3.4 Initialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
5.4 Solvers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
5.4.1 optim ga . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
5.4.2 optim moga, pareto ﬁlter . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
5.4.3 optim nsga . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
5.4.4 optim nsga2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
6 Simulated Annealing 35
6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
6.2 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
6.3 Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
6.4 Neighbor functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
6.5 Acceptance functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
6.6 Temperature laws . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
6.7 optim sa . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
7 LMITOOL: a Package for LMI Optimization in Scilab 41
7.1 Purpose . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
7.2 Function lmisolver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
7.2.1 Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
7.2.2 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
7.3 Function LMITOOL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
7.3.1 Noninteractive mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
7.3.2 Interactive mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
7.4 How lmisolver works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
7.5 Other versions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
8 Optimization data ﬁles 56
8.1 MPS ﬁles and the Quapro toolbox . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
8.2 SIF ﬁles and the CUTEr toolbox . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
9 Scilab Optimization Toolboxes 57
9.1 Quapro . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
9.1.1 Linear optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
9.1.2 Linear quadratic optimization . . . . . . . . . . . . . . . . . . . . . . . . . 58
9.2 CUTEr . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
9.3 The Unconstrained Optimization Problem Toolbox . . . . . . . . . . . . . . . . . 60
9.4 Other toolboxes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
10 Missing optimization features in Scilab 63
2
Conclusion 64
Bibliography 65
3
Copyright c 20082010  Consortium Scilab  Digiteo  Michael Baudin
Copyright c 20082009  Consortium Scilab  Digiteo  Vincent Couvert
Copyright c 20082009  INRIA  Serge Steer
This ﬁle must be used under the terms of the Creative Commons AttributionShareAlike 3.0
Unported License:
http://creativecommons.org/licenses/bysa/3.0
4
Introduction
This document aims at giving Scilab users a complete overview of optimization features in Scilab.
It is written for Scilab partners needs in OMD project (http://omd.lri.fr/tikiindex.php). The
core of this document is an analysis of current Scilab optimization features. In the ﬁnal part,
we give a short list of new features which would be interesting to ﬁnd in Scilab. Above all the
embedded functionalities of Scilab itself, some contributions (toolboxes) have been written to
improve Scilab capabilities. Many of these toolboxes are interfaces to optimization libraries, such
as FSQP for example.
In this document, we consider optimization problems in which we try to minimize a cost
function f(x) with or without constraints. These problems are partly illustrated in ﬁgure 1.
Several properties of the problem to solve may be taken into account by the numerical algorithms :
• The unknown may be a vector of real values or integer values.
• The number of unknowns may be small (from 1 to 10  100), medium (from 10 to 100  1
000) or large (from 1 000  10 000 and above), leading to dense or sparse linear systems.
• There may be one or several cost functions (multiobjective optimization).
• The cost function may be smooth or nonsmooth.
• There may be constraints or no constraints.
• The constraints may be bounds constraints, linear or nonlinear constraints.
• The cost function can be linear, quadratic or a general non linear function.
An overview of Scilab optimization tools is showed in ﬁgure 2.
In this document, we present the following optimization features of Scilab.
• nonlinear optimization with the optim function,
• quadratic optimization with the qpsolve function,
• nonlinear leastsquare optimization with the lsqrsolve function,
• semideﬁnite programming with the semidef function,
• genetic algorithms with the optim_ga function,
• simulated annealing with the optim_sa function,
• linear matrix inequalities with the lmisolver function,
5
>
1
0
0
U
n
k
n
o
w
n
s
1
0

1
0
0
U
n
k
n
o
w
n
s
1

1
0
U
n
k
n
o
w
n
s
N
o
n

l
i
n
e
a
r
O
b
j
e
c
t
i
v
e
Q
u
a
d
r
a
t
i
c
O
b
j
e
c
t
i
v
e
L
i
n
e
a
r
O
b
j
e
c
t
i
v
e
N
o
n

l
i
n
e
a
r
C
o
n
s
t
r
a
i
n
t
s
L
i
n
e
a
r
C
o
n
s
t
r
a
i
n
t
s
B
o
u
n
d
s
C
o
n
s
t
r
a
i
n
t
s
W
i
t
h
C
o
n
s
t
r
a
i
n
t
s
W
i
t
h
o
u
t
C
o
n
s
t
r
a
i
n
t
s
S
m
o
o
t
h
N
o
n
S
m
o
o
t
h
O
n
e
O
b
j
e
c
t
i
v
e
S
e
v
e
r
a
l
O
b
j
e
c
t
i
v
e
s
C
o
n
t
i
n
u
o
u
s
P
a
r
a
m
e
t
e
r
s
D
i
s
c
r
e
t
e
P
a
r
a
m
e
t
e
r
s
O
p
t
i
m
i
z
a
t
i
o
n
Figure 1: Classes of optimization problems
6
Figure 2: Scilab Optimization Tools
7
• reading of MPS and SIF ﬁles with the quapro and CUTEr toolboxes.
Scilab v5.2 provides the fminsearch function, which a derivativefree algorithm for small
problems. The fminsearch function is based on the simplex algorithm of Nelder and Mead (not
to be confused with Dantzig’s simplex for linear optimization). This unconstrained algorithm
does not require the gradient of the cost function. It is eﬃcient for small problems, i.e. up to
10 parameters and its memory requirement is only O(n). This algorithm is known to be able to
manage ”noisy” functions, i.e. situations where the cost function is the sum of a general nonlinear
function and a low magnitude function. The neldermead component provides three simplex
based algorithms which allow to solve unconstrained and nonlinearly constrained optimization
problems. It provides an object oriented access to the options. The fminsearch function is, in
fact, a specialized use of the neldermead component. This component is presented in in depth in
[2].
An analysis of optimization in Scilab, including performance tests, is presented in ”Optimiza
tion with Scilab, present and future” [3]. The following is the abstract of the paper :
”We present in this paper an overview of optimization algorithms available in theScilab soft
ware. We focus on the user’s point of view, that is, we have to minimize or maximize an objective
function and must ﬁnd a solver suitable for the problem. The aim of this paper is to give a simple
but accurate view of what problems can be solved by Scilab and what behavior can be expected
for those solvers. For each solver, we analyze the type of problems that it can solve as well as its
advantages and limitations. In order to compare the respective performances of the algorithms,
we use the CUTEr library, which is available in Scilab. Numerical experiments are presented,
which indicates that there is no cureforall solvers.”
Several existing optimization features are not presented in this document. We especially
mention the following tools.
• The fsqp toolbox provides an interface for a Sequential Quadratic Programming algorithm.
This algorithm is very eﬃcient but is not free (but is provided by the authors, free of charge,
for an academic use).
• Multiobjective optimization is available in Scilab with the genetic algorithm component.
The organization of this document is as following.
In the ﬁrst chapter, we analyse the ﬂagship of Scilab in terms of nonlinear optimization: the
optim function. This function allows to solve nonlinear optimization problems without constraints
or with bound constraints. It provides a QuasiNewton method, a Limited Memory BFGS algo
rithm and a bundle method for nonsmooth functions. We analyse its features, the management
of the cost function, the linear algebra and the management of the memory. Then we consider the
algorithms which are used behind optim, depending on the type of algorithm and the constraints.
In the second chapter we present the qpsolve and qp_solve functions which allows to solve
quadratic problems. We describe the solvers which are used, the memory requirements and the
internal design of the tool.
The chapter 3 and 4 brieﬂy present nonlinear least squares problems and semideﬁnite pro
gramming.
The chapter 5 focuses on genetic algorithms. We give a tutorial example of the optim_ga
function in the case of the Rastrigin function. We also analyse the support functions which allow
to conﬁgure the behavior of the algorithm and describe the algorithm which is used.
8
The simulated annealing is presented in chapter 6, which gives an overview of the algorithm
used in optim_sa. We present an example of use of this method and shows the convergence of
the algorithm. Then we analyse the support functions and present the neighbor functions, the
acceptance functions and the temperature laws. In the ﬁnal section, we analyse the structure of
the algorithm used in optim_sa.
The LMITOOL module is presented in chapter 7. This tool allows to solve linear matrix
inequalities. This chapter was written by Nikoukhah, Delebecque and Ghaoui. The syntax of the
lmisolver function is analysed and several examples are analysed in depth.
The chapter 8 focuses on optimization data ﬁles managed by Scilab, especially MPS and SIF
ﬁles.
Some optimization features are available in the form of toolboxes, the most important of which
are the Quapro, CUTEr and the Unconstrained Optimization Problems toolboxes. These modules
are presented in the chapter 9, along with other modules including the interface to CONMIN, to
FSQP, to LIPSOL, to LPSOLVE, to NEWUOA.
The chapter 10 is devoted to missing optimization features in Scilab.
9
Chapter 1
Nonlinear optimization
The goal of this chapter is to present the current features of the optim primitive Scilab. The
optim primitive allows to optimize a problem with a nonlinear objective without constraints or
with bound constraints.
In this chapter, we describe both the internal design of the optim primitive. We analyse in
detail the management of the cost function. The cost function and its gradient can be computed
using a Scilab function, a C function or a Fortran 77 function. The linear algebra components
are analysed, since they are used at many places in the algorithms. Since the management of
memory is a crucial feature of optimization solvers, the current behaviour of Scilab with respect
to memory is detailed here.
Three nonlinear solvers are connected to the optim primitive, namely, a BFGS QuasiNewton
solver, a LBFGS solver, and a NonDiﬀerentiable solver. In this chapter we analyse each solver
and present the following features :
• the reference articles or reports,
• the author,
• the management of memory,
• the linear algebra system, especially the algorithm name and if dense/sparse cases are taken
into account,
• the line search method.
The Scilab online help is a good entry point for this function.
1.1 Mathematical point of view
The problem of the non linear optimization is to ﬁnd the solution of
min
x
f(x)
with bounds constraints or without constraints and with f : R
n
→R the cost function.
10
1.2 Optimization features
Scilab oﬀers three nonlinear optimization methods:
• QuasiNewton method with BFGS formula without constraints or with bound constraints,
• QuasiNewton with limited memory BFGS (LBGFS) without constraints or with bound
constraints,
• Bundle method for non smooth functions (half derivable functions, nondiﬀerentiable prob
lem) without constraints.
Problems involving non linear constraints cannot be solved with the current optimization
methods implemented in Scilab. Non smooth problems with bounds constraints cannot be solved
with the methods currently implemented in Scilab.
1.3 Optimization routines
Nonlinear optimization in Scilab is based on a subset of the Modulopt library, developed at
Inria. The library which is used by optim was created by the Modulopt project at INRIA and
developped by Bonnans, Gilbert and Lemar´echal [5].
This section lists the routines used according to the optimization method used.
The following is the list of solvers currently available in Scilab, and the corresponding fortran
routine :
• ”qn” without constraints : a QuasiNewton method without constraints, n1qn1,
• ”qn” with bounds constraints : QuasiNewton method with bounds constraints, qnbd,
• ”gc” without constraints : a Limited memory BGFS methoud without constraints, n1qn3,
• ”gc” with bounds constraints : a Limited memory BGFS with bounds constraints, gcbd,
• ”nd” without constraints : a Non smooth method without constraints, n1fc1.
1.4 The cost function
The communication protocol used by optim is direct, that is, the cost function must be passed
as a callback argument to the ”optim” primitive. The cost function must compute the objective
and/or the gradient of the objective, depending on the input integer ﬂag ”ind”.
In the most simple usecase, the cost function is a Scilab function, with the following header :
[ f , g , i nd]= c o s t f ( x , i nd )
where ”x” is the current value of the unknown and ”ind” is the integer ﬂag which states if ”f”, ”g”
or both are to be computed.
The cost function is passed to the optimization solver as a callback, which is managed with
the fortran 77 callback system. In that case, the name of the routine to call back is declared as
”external” in the source code. The cost function may be provided in the following ways :
11
• the cost function is provided as a Scilab script,
• the cost function is provided as a C or fortran 77 compiled routine.
If the cost function is a C or fortran 77 source code, the cost function can be statically or
dynamically linked against Scilab. Indeed, Scilab dynamic link features, such as ilib for link for
example, can be used to provide a compiled cost function.
In the following paragraph, we analyse the very internal aspects of the management of the
cost function.
This switch is managed at the gateway level, in the sci f optim routine, with a ”if” statement :
• if the cost function is compiled, the ”foptim” symbol is passed,
• if not, the ”boptim” symbol is passed.
In the case where the cost function is a Scilab script, the ”boptim” routine performs the copy of
the input local arguments into Scilab internal memory, calls the script, and ﬁnally copy back the
output argument from Scilab internal memory into local output variables. In the case where the
cost function is compiled, the computation is based on function pointers, which are managed at
the C level in optimtable.c. The optimization function is conﬁgured by the ”setfoptim” C ser
vice, which takes as argument the name of the routine to callback. The services implemented in
AddFunctionInTable.c are used, especially the function ”AddFunctionInTable”, which takes the
name of the function as input argument and searches the corresponding function address, be it
in statically compiled libraries or in dynamically compiled libraries. This allows the optimization
solvers to callback dynamically linked libraries. These names and addresses are stored in the
hashmap FTab foptim, which maps function names to function pointer. The static ﬁeld foptim
fonc with type foptimf is then set to the address of the function to be called back. When the
optimization solver needs to compute the cost function, it calls the ”foptim”C void function which
in returns calls the compiled cost function associated to the conﬁgured address (*foptimfonc).
1.5 Linear algebra
The linear algebra which is used in the ”optim” primitive is dense. Generally, the linear algebra
is inlined and there is no use the BLAS API. This applies to all optimization methods, except
”gcbd”. This limits the performance of the optimization, because optimized libraries like ATLAS
cannot not used. There is only one exception : the LBFGS with bounds constraints routine gcbd
uses the ”dcopy” routine of the BLAS API.
1.6 Management of memory
The optimization solvers requires memory, especially to store the value of the cost function,
the gradient, the descent direction, but most importantly, the approximated Hessian of the cost
function.
Most of the memory is required by the approximation of the Hessian matrix. If the full
approximated Hessian is stored, as in the BFGS quasiNewton method, the amount of memory
is the square of the dimension of the problem O(n
2
), where n is the size of the unknown. When
12
a quasiNewton method with limited memory is used, only a given number m of vectors of size n
are stored.
This memory is allocated by Scilab, inside the stack and the storage area is passed to the
solvers as an input argument. This large storage memory is then split into pieces like a piece
of cake by each solver to store the data. The memory system used by the fortran solvers is the
fortran 77 ”assumedsizedummyarrays” mechanism, based on ”real arrayname(*)” statements.
The management of memory is very important for largescale problems, where n is from 100
to 1000. One main feature of one of the LBFGS algorithms is to limit the memory required.
More precisely, the following is a map from the algorithm to the memory required, as the number
of required double precision values.
• QuasiNewton BFGS ”qn” without constraints (n1qn1) : n(n + 13)/2,
• QuasiNewton BFGS ”qn” with bounds constraints (qnbd) : n(n + 1)/2 + 4n + 1,
• Limited Memory BFGS ”gc” without constraints (n1qn3) : 4n +m(2n + 1),
• Limited Memory BFGS ”gc” with bounds constraints (gcbd) : n(5 + 3nt) + 2nt + 1 with
nt = max(1, m/3),
• Non smooth method without constraints (n1fc1) : (n + 4)m/2 + (m+ 9) ∗ m+ 8 + 5n/2.
Note that n1fc1 requires an additionnal 2(m + 1) array of integers. Simplifying these array
sizes leads to the following map, which clearly shows why Limited Memory BFGS algorithms in
Scilab are more suitable for large problems. This explains why the name ”cg”was chosen: it refers
to the Conjugate Gradient method, which stores only one vector in memory. But the name ”cg”
is wrongly chosen and this is why we consistently use LBFGS to identify this algorithm.
• QuasiNewton ”qn” without constraints (n1qn1) : O(n
2
),
• QuasiNewton ”qn” with bounds constraints (qnbd) : O(n
2
),
• Limited Memory BFGS ”gc” without constraints (n1qn3) : O(n),
• Limited Memory BFGS ”gc” with bounds constraints (gcbd) : O(n),
• Non smooth method without constraints (n1fc1) : O(n).
That explains why LBFGS methods associated with the ”gc” option of the optim primitive
are recommended for largescale optimization. It is known that LBFGS convergence may be slow
for largescale problems (see [21], chap. 9).
1.7 QuasiNewton ”qn” without constraints : n1qn1
The author is C. Lemarechal, 1987. There is no reference report for this solver.
The following is the header for the n1qn1 routine :
13
subroutine n1qn1 (simul,n,x,f,g,var,eps,
1 mode,niter,nsim,imp,lp,zm,izs,rzs,dzs)
c!but
c minimisation d une fonction reguliere sans contraintes
c!origine
c c. lemarechal, inria, 1987
c Copyright INRIA
c!methode
c direction de descente calculee par une methode de quasinewton
c recherche lineaire de type wolfe
The following is a description of the arguments of this routine.
• simul : point d’entree au module de simulation (cf normes modulopt i). n1qn1 appelle
toujours simul avec indic = 4 ; le module de simulation doit se presenter sous la forme
subroutine simul(n,x, f, g, izs, rzs, dzs) et ˆetre declare en external dans le programme
appelant n1qn1.
• n (e) : nombre de variables dont depend f.
• x (es) : vecteur de dimension n ; en entree le point initial ; en sortie : le point ﬁnal calcule
par n1qn1.
• f (es) : scalaire ; en entree valeur de f en x (initial), en sortie valeur de f en x (ﬁnal).
• g (es) : vecteur de dimension n : en entree valeur du gradient en x (initial), en sortie valeur
du gradient en x (ﬁnal).
• var (e) : vecteur strictement positif de dimension n. amplitude de la modif souhaitee a la
premiere iteration sur x(i).une bonne valeur est 10% de la diﬀerence (en valeur absolue)
avec la coordonee x(i) optimale
• eps (es) : en entree scalaire deﬁnit la precision du test d’arret. Le programme considere
que la convergence est obtenue lorque il lui est impossible de diminuer f en attribuant ` a au
moins une coordonn´ee x(i) une variation superieure a eps*var(i). En sortie, eps contient le
carr´e de la norme du gradient en x (ﬁnal).
• mode (e) : deﬁnit l’approximation initiale du hessien
– =1 n1qn1 l’initialise luimeme
– =2 le hessien est fourni dans zm sous forme compressee (zm contient les colonnes de
la partie inferieure du hessien)
• niter (es) : en entree nombre maximal d’iterations : en sortie nombre d’iterations reellement
eﬀectuees.
• nsim (es) : en entree nombre maximal d’appels a simul (c’est a dire avec indic = 4). en
sortie le nombre de tels appels reellement faits.
• imp (e) : contrˆ ole les messages d’impression :
14
– = 0 rien n’est imprime
– = 1 impressions initiales et ﬁnales
– = 2 une impression par iteration (nombre d’iterations, nombre d’appels a simul, valeur
courante de f).
– >=3 informations supplementaires sur les recherches lineaires ; tres utile pour detecter
les erreurs dans le gradient.
• lp (e) : le numero du canal de sortie, i.e. les impressions commandees par imp sont faites
par write (lp, format).
• zm : memoire de travail pour n1qn1 de dimension n*(n+13)/2.
• izs,rzs,dzs memoires reservees au simulateur (cf doc)
The n1qn1 solver is an interface to the n1qn1a routine, which really implements the optimiza
tion method. The n1qn1a ﬁle counts approximately 300 lines. The n1qn1a routine is based on
the following routines :
• simul : computes the cost function,
• majour : probably (there is no comment) an update of the BFGS matrix.
Many algorithms are inlined, especially the line search and the linear algebra.
1.7.1 Management of the approximated Hessian matrix
The current BFGS implementation is based on a approximation of the Hessian [21], which is based
on Cholesky decomposition, i.e. the approximated Hessian matrix is decomposed as G = LDL
T
,
where D is a diagonal n ×n matrix and L is a lower triangular n ×n matrix with unit diagonal.
To compute the descent direction, the linear system Gd = LDL
T
d = −g is solved.
The memory requirements for this method is O(n
2
) because the approximated Hessian matrix
computed from the BFGS formula is stored in compressed form so that only the lower part of
the approximated Hessian matrix is stored. This is why this method is not recommended for
largescale problems (see [21], chap.9, introduction).
The approximated hessian H ∈ R
n×n
is stored as the vector h ∈ R
n
h
which has size n
h
=
n(n + 1)/2. The matrix is stored in factored form as following
h = (D
11
L
21
. . . L
n1
H
21
D
22
. . . L
n2
 . . . D
n−1n−1
L
nn−1
D
nn
) . (1.1)
Instead of a direct acces to the factors of D and L, integers algebra is necessary to access to the
data stored in the vector h.
The algorithm presented in ﬁgure 1.1 is used to set the diagonal terms the diagonal terms of D,
the diagonal matrix of the Cholesky decomposition of the approximated Hessian. The righthand
side
0.01c
max
v
2
i
of this initialization is analysed in the next section of this document.
15
k ←1
for i = 1 to n do
h(k) =
0.01c
max
v
2
i
k ←k +n + 1 −i
end for
Figure 1.1: Loop over the diagonal terms of the Cholesky decomposition of the approximated
Hessian
Solving the linear system of equations
The linear system of equations Gd = LDL
T
d = −g must be solved to computed the descent
direction d ∈ R
n
. This direction is computed by the following algorithm
• compute w so that Lw = −g,
• computed d so that DL
T
d = w.
This algorithm requires O(n
2
) operations.
1.7.2 Line search
The line search is based on the algorithms developped by Lemar´echal [26]. It uses a cubic inter
polation.
The Armijo condition for suﬃcient decrease is used in the following form
f(x
k
+αp
k
) −f(x
k
) ≤ c
1
α∇f
T
k
p
k
(1.2)
with c
1
= 0.1. The following fortran source code illustrates this condition
if (fbfa.le.0.10d+0*c*dga) go to 280
where fb = f(x
k
+ αp
k
), fa = f(x
k
), c = α and dga = ∇f
T
k
p
k
is the local directional derivative.
1.7.3 Initial Hessian matrix
Several modes are available to compute the initial Hessian matrix, depending on the value of the
mode variable
• if mode = 1, n1qn1 initializes the matrix by itself,
• if mode = 2, the hessian is provided in compressed form, where only the lower part of the
symetric hessian matrix is stored.
An additionnal mode=3 is provided but the feature is not clearly documented. In Scilab, the
n1qn1 routine is called with mode = 1 by default. In the case where a hotrestart is performed,
the mode = 3 is enabled.
If mode = 1 is chosen, the initial Hessian matrix H
0
is computed by scaling the identity matrix
H
0
= Iδ (1.3)
16
where δ ∈ R
n
is a nvector and I is the n ×n identity matrix. The scaling vector δ ∈ R
n
is based
on the gradient at the initial guess g
0
= g(x
0
) = ∇f(x
0
) ∈ R
n
and a scaling vector v ∈ R
n
, given
by the user
δ
i
=
0.01c
max
v
2
i
(1.4)
where c
max
> 0 is computed by
c
max
= max
_
1.0, max
i=1,n
(g
0
i
))
_
(1.5)
In the Scilab interface for optim, the scaling vector is set to 0.1 :
v
i
= 0.1, i = 1, n. (1.6)
1.7.4 Termination criteria
The following list of parameters are taken into account by the solver
• niter, the maximum number of iterations (default value is 100),
• nap, the maximum number of function evaluations (default value is 100),
• epsg, the minimum length of the search direction (default value is %eps ≈ 2.22e −16).
The other parameters epsf and epsx are not used. The termination condition is not based
on the gradient, as the name epsg would indicate.
The following is a list of termination conditions which are taken into account in the source
code.
• The iteration is greater than the maximum.
if (itr.gt.niter) go to 250
• The number of function evaluations is greater than the maximum.
if (nfun.ge.nsim) go to 250
• The directionnal derivative is positive, so that the direction d is not a descent direction for
f.
if (dga.ge.0.0d+0) go to 240
• The cost function set the indic ﬂag (the ind parameter) to 0, indicating that the optimization
must terminate.
call simul (indic,n,xb,fb,gb,izs,rzs,dzs)
[...]
go to 250
17
• The cost function set the indic ﬂag to a negative value indicating that the function cannot
be evaluated for the given x. The step is reduced by a factor 10, but gets below a limit so
that the algorithm terminates.
call simul (indic,n,xb,fb,gb,izs,rzs,dzs)
[...]
step=step/10.0d+0
[...]
if (stepbd.gt.steplb) goto 170
[...]
go to 250
• The Armijo condition is not satisﬁed and step size is below a limit during the line search.
if (fbfa.le.0.10d+0*c*dga) go to 280
[...]
if (step.gt.steplb) go to 270
• During the line search, a cubic interpolation is computed and the computed minimum is
associated with a zero step length.
if(c.eq.0.0d+0) goto 250
• During the line search, the step length is lesser than a computed limit.
if (stmin+step.le.steplb) go to 240
• The rank of the approximated Hessian matrix is lesser than n after the update of the
Cholesky factors.
if (ir.lt.n) go to 250
1.7.5 An example
The following script illustrates that the gradient may be very slow, but the algorithm continues.
This shows that the termination criteria is not based on the gradient, but on the length of the
step. The problem has two parameters so that n = 2. The cost function is the following
f(x) = x
p
1
+ x
p
2
(1.7)
where p ≥ 0 is an even integer. Here we choose p = 10. The gradient of the function is
g(x) = ∇f(x) = (px
p−1
1
, px
p−1
2
)
T
(1.8)
and the Hessian matrix is
H(x) =
_
p(p −1)x
p−2
1
0
0 p(p −1)x
p−2
2
_
(1.9)
18
The optimum of this optimization problem is at
x
= (0, 0)
T
. (1.10)
The following Scilab script deﬁnes the cost function, checks that the derivatives are correctly
computed and performs an optimization. At each iteration, the norm of the gradient of the cost
function is displayed so that one can see if the algorithm terminates when the gradient is small.
function [ f , g , i nd ] = myquadrati c ( x , i nd )
p = 10
i f i nd == 1  i nd == 2  i nd == 4 then
f = x ( 1) ˆp + x ( 2) ˆp ;
end
i f i nd == 1  i nd == 2  i nd == 4 then
g ( 1) = p ∗ x ( 1) ˆ( p−1)
g ( 2) = p ∗ x ( 2) ˆ( p−1)
end
i f i nd == 1 then
mprintf ( ” x=%e, f=%e,  g=%e\n”,norm( x ) , f , norm( g ) )
end
endfunction
function f = quadf ornumdi f f ( x )
f = myquadrati c ( x , 2 )
endfunction
x0 = [ −1. 2 1 . 0 ] ;
[ f , g ] = myquadrati c ( x0 , 4 ) ;
mprintf ( ”Computed f (x0) = %f\n”, f ) ;
mprintf ( ”Computed g(x0) = \n”) ; disp( g ’ ) ;
mprintf ( ”Expected g(x0) = \n”) ; disp( derivative ( quadf ornumdi f f , x0 ’ ) )
nap = 100
i t e r = 100
epsg = %eps
[ f opt , xopt , gradopt ] = optim ( myquadrati c , x0 , . . .
”ar” , nap , i t e r , epsg , imp = −1)
The script produces the following output.
−−>[ f opt , xopt , gradopt ] = optim ( myquadrati c , x0 , . . .
”ar” , nap , i t e r , epsg , imp = −1)
 x =1. 562050 e +000, f =7. 191736 e +000,  g  =5. 255790 e+001
 x =1. 473640 e +000, f =3. 415994 e +000,  g  =2. 502599 e+001
 x =1. 098367 e +000, f =2. 458198 e +000,  g  =2. 246752 e+001
 x =1. 013227 e +000, f =1. 092124 e +000,  g  =1. 082542 e+001
 x =9. 340864 e −001, f =4. 817592e −001,  g  =5. 182592 e+000
[ . . . ]
 x =1. 280564 e −002, f =7. 432396e −021,  g  =5. 817126 e−018
 x =1. 179966 e −002, f =3. 279551e −021,  g  =2. 785663 e−018
 x =1. 087526 e −002, f =1. 450507e −021,  g  =1. 336802 e−018
 x =1. 002237 e −002, f =6. 409611e −022,  g  =6. 409898 e−019
 x =9. 236694 e −003, f =2. 833319e −022,  g  =3. 074485 e−019
Norm of pr oj e c t e d gr adi ent l ower than 0. 3074485D−18.
19
gradopt =
1. 0D−18 ∗
0. 2332982 0. 2002412
xopt =
0. 0065865 0. 0064757
f opt =
2. 833D−22
One can see that the algorithm terminates when the gradient is extremely small g(x) ≈ 10
−18
.
The cost function is very near zero f(x) ≈ 10
−22
, but the solution is not accurate only up to the
3d digit.
This is a very diﬃcult test case for optimization solvers. The diﬃculty is because the function
is extremely ﬂat near the optimum. If the termination criteria was based on the gradient, the
algorithm would stop in the early iterations. Because this is not the case, the algorithm performs
signiﬁcant iterations which are associated with relatively large steps.
1.8 QuasiNewton ”qn” with bounds constraints : qnbd
The comments state that the reference report is an INRIA report by F. Bonnans [4]. The solver
qnbd is an interface to the zqnbd routine. The zqnbd routine is based on the following routines :
• calmaj : calls majour, which updates the BFGS matrix,
• proj : projects the current iterate into the bounds constraints,
• ajour : probably (there is no comment) an update of the BFGS matrix,
• rlbd : line search method with bound constraints,
• simul : computes the cost function
The rlbd routine is documented as using an extrapolation method to computed a range for the
optimal t parameter. The range is then reduced depending on the situation by :
• a dichotomy method,
• a linear interpolation,
• a cubic interpolation.
The stopping criteria is commented as ”an extension of the Wolfe criteria”. The linear algebra does
not use the BLAS API. It is inlined, so that connecting the BLAS may be diﬃcult. The memory
requirements for this method are O(n
2
), which shows why this method is not recommended for
largescale problems (see [21], chap.9, introduction).
1.9 LBFGS ”gc” without constraints : n1qn3
The comments in this solver are clearly written. The authors are Jean Charles Gilbert, Claude
Lemarechal, 1988. The BFGS update is based on the article [34]. The solver n1qn3 is an interface
to the n1qn3a routine. The architecture is clear and the source code is well commented. The
n1qn3a routine is based on the following routines :
20
• prosca : performs a vector x vector scalar product,
• simul : computes the cost function,
• nlis0 : line search based on Wolfe criteria, extrapolation and cubic interpolation,
• ctonb : copies array u into v,
• ddd2 : computed the descent direction by performing the product hxg.
The linear algebra is dense, which limits the feature to small size optimization problems. The
linear algebra does not use the BLAS API but is based on the prosca and ctonb routines. The
prosca routine is a call back input argument of the n1qn3 routine, connected to the fuclid routine.
This implements the scalar product, but without optimizaion. Connecting BLAS may be easy
for n1qn3. The algorithm is a limited memory BFGS method with m levels, so that the memory
cost is O(n). It is well suited for mediumscale problems, although convergence may be slow (see
[21], chap. 9, p.227).
1.10 LBFGS ”gc” with bounds constraints : gcbd
The author is F. Bonnans, 1985. There is no reference report for gcbd. The gcbd solver is an
interface to the zgcbd routine, which really implements the optimization method. The zgcbd
routine is based on the following routines :
• simul : computes the cost function
• proj : projects the current iterate into the bounds constraints,
• majysa : updates the vectors y = g(k + 1) −g(k), s = x(k + 1) −x(k), ys,
• bfgsd : updates the diagonal by Powell diagonal BFGS,
• shanph : scalse the diagonal by ShannoPhua method,
• majz : updates z,zs,
• relvar : computes the variables to relax by Bertsekas method,
• gcp : gradient conjugate method for Ax = b,
• dcopy : performs a vector copy (BLAS API),
• rlbd : line search method with bound constraints.
The linear algebra is dense. But zgcbd uses the ”dcopy” BLAS routine, which allows for some
optimizations. The algorithm is a limited memory BFGS method with m levels, so that the
memory cost is O(n). It is well suited for mediumscale problems, although the convergence may
be slow (see [21], chap. 9, p.227).
21
1.11 Non smooth method without constraints : n1fc1
This routine is probably due to Lemar´echal, who is an expert of this topic. References for this
algorithm include the ”Part II, Nonsmooth optimization” in [5], and the indepth presentation in
[18, 19].
The n1fc1 solver is an interface to the n1fc1a routine, which really implements the optimization
method. The n1fc1a routine is based on the following routines :
• simul : computes the cost function,
• fprf2 : computes the direction,
• frdf1 : reduction du faisceau
• nlis2 : line search,
• prosca : performs a vector x vector scalar product.
It is designed for functions which have a noncontinuous derivative (e.g. the objective function is
the maximum of several continously diﬀerentiable functions).
22
Chapter 2
Quadratic optimization
Quadratic problems can be solved with the qpsolve Scilab macro and the qp_solve Scilab prim
itive. In this chapter, we presents these two primitives, which are meant to be a replacement for
the former Quapro solver (which has be transformed into a Scilab toolbox). We especially analyse
the management of the linear algebra as well as the memory requirements of these solvers.
2.1 Mathematical point of view
This kind of optimization is the minimization of function f(x) with
f(x) =
1
2
x
T
Qx +p
T
x
under the constraints :
C
T
1
x = b
1
(2.1)
C
T
2
x ≥ b
2
(2.2)
(2.3)
2.2 qpsolve
The Scilab function qpsolve is a solver for quadratic problems when Q is symmetric positive
deﬁnite.
The qpsolve function is a Scilab macro which aims at providing the same interface (that is,
the same input/output arguments) as the quapro solver.
For more details about this solver, please refer to Scilab online help for qpsolve
The qpsolve Scilab macro is based on the work by Berwin A Turlach from The University of
Western Australia, Crawley [38]. The solver is implemented in Fortran 77. This routine uses the
Goldfarb/Idnani algorithm [11, 12].
The constraints matrix can be dense or sparse.
The qpsolve macro calls the qp_solve compiled primitive. The internal source code for
qpsolve manages the equality and inequality constraints so that it can be processed by the
qp_solve primitive.
23
2.3 qp solve
The qp_solve compiled primitive is an interface for the fortran 77 solver. The interface is im
plemented in the C source code sci qp solve.c. Depending on the constraints matrix, a dense or
sparse solver is used :
• If the constraints matrix C is dense, the qpgen2 fortran 77 routine is used. The qpgen2
routine is the original, unmodiﬁed algorithm which was written by Turlach (the original
name was solve.QP.f)
• If the constraints matrix C is a Scilab sparse matrix, the qpgen1sci routine is called. This
routine is a modiﬁcation of the original routine qpgen1, in order to adapt to the speciﬁc
Scilab sparse matrices storage.
2.4 Memory requirements
Suppose that n is the dimension of the quadratic matrix Q and m is the sum of the number of
equality constraints me and inequality constraints md. Then, the temporary work array which is
allocated in the primitive has the size
r = min(n, m), (2.4)
lw = 2n + r(r + 5)/2 + 2m+ 1. (2.5)
This temporary array is deallocated when the qpsolve primitive returns.
This formulae may be simpliﬁed in the following cases :
• if n m, that is the number of constraints m is negligible with respect to the number of
unknowns n, then the memory required is O(n),
• if m n, that is the number of unknowns n is negligible with respect to the number of
constraints m, then the memory required is O(m),
• if m = n, then the memory required is O(n
2
).
2.5 Internal design
The original Goldfarb/Idnani algorithm [11, 12] was designed to solve the following minimization
problem:
min
x
−d
T
x +
1
2
x
T
Dx
where
A
T
1
x = b
1
(2.6)
A
T
2
x >= b
2
(2.7)
where the matrix D is assumed to be symmetric positive deﬁnite. It was considered as a building
block for a Sequential Quadratic Programming solver. The original package provides two routines :
24
• solve.QP.f containing routine qpgen2 which implements the algorithm for dense matrices,
• solve.QP.compact.f containing routine qpgen1 which implements the algorithm for sparse
matrices.
25
Chapter 3
Nonlinear least square
3.1 Mathematical point of view
The problem of the non linear leastquare optimization is to ﬁnd the solution of
min
x
x
f(x)
2
with bounds constraints or without constraints and with f : R
n
→R
m
the cost function.
3.2 Scilab function
Scilab function called lsqrsolve is designed for the minimization of the sum of the squares of non
linear functions using a LevenbergMarquardt algorithm. For more details about this function,
please refer to Scilab online help
3.3 Optimization routines
Scilab lsqrsolve function is based on the routines lmdif and lmder of the library Minpack (Ar
gonne National Laboratory).
26
Chapter 4
Semideﬁnite programming
4.1 Mathematical point of view
This kind of optimization is the minimization of f(x) = c
T
x under the constraint:
F
0
+x
1
F
1
+... +x
m
F
m
≥ 0 (4.1)
or its dual problem, the maximization or −Trace(F
0
, Z) under the constraints:
Trace(F
i
, Z) = c
i
, i = 1, ..., m (4.2)
Z ≥ 0 (4.3)
4.2 Scilab function
The Scilab function called semidef is designed for this kind of optimization problems. For more
details about this function, please refer to Scilab online help
4.3 Optimization routines
Scilab semidef function is based on a routine from L. Vandenberghe and Stephen Boyd.
27
Chapter 5
Genetic algorithms
5.1 Introduction
Genetic algorithms are search algorithms based on the mechanics on natural selection and natural
genetics [17, 28]. Genetic algorithms have been introduced in Scilab v5 thanks to a work by Yann
Collette [9]. The solver is made of Scilab macros, which enables a highlevel programming model
for this optimization solver.
The problems solved by the current genetic algorithms in Scilab are the following :
• minimization of a cost function with bound constraints,
• multiobjective non linear minimization with bound constraints.
Genetic algorithms are diﬀerent from more normal optimization and search procedures in four
ways :
• GAs work with a coding of the parameter set, not the parameters themselves,
• GAs search from a population of points, not a single point,
• GAs use payoﬀ (objective function) information, not derivativess or other auxiliary knowl
edge,
• GAs use probabilistic transition rules, not deterministic rules.
A simple genetic algorithm that yields good results in many practical problems is composed
of three operators [17] :
• reproduction,
• crossover,
• mutation.
Many articles on this subject have been collected by Carlos A. Coello Coello on his website
[7]. A brief introduction to GAs is done in [43].
The GA macros are based on the ”parameters” Scilab module for the management of the
(many) optional parameters.
28
5.2 Example
In the current section, we give an example of the user of the GA algorithms.
The following is the deﬁnition of the Rastrigin function.
function Res = min_bd_rastrigin()
Res = [1 1]’;
endfunction
function Res = max_bd_rastrigin()
Res = [1 1]’;
endfunction
function Res = opti_rastrigin()
Res = [0 0]’;
endfunction
function y = rastrigin(x)
y = x(1)^2+x(2)^2cos(12*x(1))cos(18*x(2));
endfunction
This cost function is then deﬁned with the generic name ”f”. Other algorithmic parameters,
such as the size of the population, are deﬁned in the following sample Scilab script.
func = ’rastrigin’;
deff(’y=f(x)’,’y = ’+func+’(x)’);
PopSize = 100;
Proba_cross = 0.7;
Proba_mut = 0.1;
NbGen = 10;
NbCouples = 110;
Log = %T;
nb_disp = 10;
pressure = 0.05;
Genetic Algorithms require many settings, which are cleanly handled by the ”parameters”
module. This module provides the nit_param function, which returns a new, empty, set of pa
rameters. The add_param function allows to set individual named parameters, which are conﬁgure
with keyvalue pairs.
1 ga params = i ni t par am ( ) ;
2 // Parameters to adapt to the shape of the optimization problem
3 ga params = add param( ga params , ’minbound’ , eval ( ’ min bd ’+f unc+’ () ’ ) ) ;
4 ga params = add param( ga params , ’maxbound’ , eval ( ’max bd ’+f unc+’ () ’ ) ) ;
5 ga params = add param( ga params , ’ dimension ’ , 2 ) ;
6 ga params = add param( ga params , ’ beta ’ , 0 ) ;
7 ga params = add param( ga params , ’ delta ’ , 0 . 1 ) ;
8 // Parameters to f i ne tune the Genetic algorithm.
9 ga params = add param( ga params , ’ i ni t f unc ’ , i n i t g a d e f a u l t ) ;
10 ga params = add param( ga params , ’ crossover func ’ , c r o s s o ve r g a de f a ul t ) ;
11 ga params = add param( ga params , ’ mutation func ’ , mut at i on ga def aul t ) ;
12 ga params = add param( ga params , ’ codage func ’ , c odi ng ga i de nt i t y ) ;
29
13 ga params = add param( ga params , ’ sel ecti on func ’ , s e l e c t i o n g a e l i t i s t ) ;
14 ga params = add param( ga params , ’ nb couples ’ , NbCouples ) ;
15 ga params = add param( ga params , ’ pressure ’ , pr e s s ur e ) ;
The optim_ga function search a population solution of a singleobjective problem with bound
constraints.
1 [ pop opt , f obj pop opt , pop i ni t , f o bj po p i ni t ] = . . .
2 opti m ga ( f , PopSize , NbGen, Proba mut , Proba cross , Log , ga params ) ;
The following are the messages which are displayed in the Scilab console :
optim_ga: Initialization of the population
optim_ga: iteration 1 / 10  min / max value found = 1.682413 / 0.081632
optim_ga: iteration 2 / 10  min / max value found = 1.984184 / 0.853613
optim_ga: iteration 3 / 10  min / max value found = 1.984184 / 1.314217
optim_ga: iteration 4 / 10  min / max value found = 1.984543 / 1.513463
optim_ga: iteration 5 / 10  min / max value found = 1.998183 / 1.691332
optim_ga: iteration 6 / 10  min / max value found = 1.999551 / 1.871632
optim_ga: iteration 7 / 10  min / max value found = 1.999977 / 1.980356
optim_ga: iteration 8 / 10  min / max value found = 1.999979 / 1.994628
optim_ga: iteration 9 / 10  min / max value found = 1.999989 / 1.998123
optim_ga: iteration 10 / 10  min / max value found = 1.999989 / 1.999534
The initial and ﬁnal populations for this simulation are shown in 5.1.
The following script is a loop over the optimum individuals of the population.
1 pri ntf ( ’ Genetic Algorithm: %d points from pop opt\n’ , nb di sp ) ;
2 for i =1: nb di sp
3 pri ntf ( ’ Individual %d: x(1) = %f x(2) = %f −> f = %f\n’ , . . .
4 i , pop opt ( i ) ( 1 ) , pop opt ( i ) ( 2 ) , f obj pop opt ( i ) ) ;
5 end
The previous script make the following lines appear in the Scilab console.
Individual 1: x(1) = 0.000101 x(2) = 0.000252 > f = 1.999989
Individual 2: x(1) = 0.000118 x(2) = 0.000268 > f = 1.999987
Individual 3: x(1) = 0.000034 x(2) = 0.000335 > f = 1.999982
Individual 4: x(1) = 0.000497 x(2) = 0.000136 > f = 1.999979
Individual 5: x(1) = 0.000215 x(2) = 0.000351 > f = 1.999977
Individual 6: x(1) = 0.000519 x(2) = 0.000197 > f = 1.999974
Individual 7: x(1) = 0.000188 x(2) = 0.000409 > f = 1.999970
Individual 8: x(1) = 0.000193 x(2) = 0.000427 > f = 1.999968
Individual 9: x(1) = 0.000558 x(2) = 0.000260 > f = 1.999966
Individual 10: x(1) = 0.000235 x(2) = 0.000442 > f = 1.999964
5.3 Support functions
In this section, we analyze the GA services to conﬁgure a GA computation.
30
Figure 5.1: Optimum of the Rastrigin function – Initial population is in red, Optimum population
is accumulated on the blue dot
31
5.3.1 Coding
The following is the list of coding functions available in Scilab’s GA :
• coding_ga_binary : A function which performs conversion between binary and continuous
representation
• coding_ga_identity : A ”nooperation” conversion function
The user may conﬁgure the GA parameters so that the algorithm uses a customized coding
function.
5.3.2 Crossover
The crossover function is used when mates have been computed, based on the Wheel algorithm :
the crossover algorithm is a loop over the couples, which modiﬁes both elements of each couple.
The following is the list of crossover functions available in Scilab :
• crossover_ga_default : A crossover function for continuous variable functions.
• crossover_ga_binary : A crossover function for binary code
5.3.3 Selection
The selection function is used in the loop over the generations, when the new population is
computed by processing a selection over the individuals.
The following is the list of selection functions available in Scilab :
• selection_ga_random : A function which performs a random selection of individuals. We
select pop size individuals in the set of parents and childs individuals at random.
• selection_ga_elitist : An ’elitist’ selection function. We select the best individuals in
the set of parents and childs individuals.
5.3.4 Initialization
The initialization function returns a population as a list made of ”pop size”individuals. The Scilab
macro init_ga_default computes this population by performing a randomized discretization of
the domain deﬁned by the bounds as minimum and maximum arrays. This randomization is
based on the Scilab primitive rand.
5.4 Solvers
In this section, we analyze the 4 GA solvers which are available in Scilab :
• optim_ga : ﬂexible genetic algorithm
• optim_moga : multiobjective genetic algorithm
• optim_nsga : multiobjective Niched Sharing Genetic Algorithm
32
• optim_nsga2 : multiobjective Niched Sharing Genetic Algorithm version 2
While optim_ga is designed for one objective, the 3 other solvers are designed for multi
objective optimization.
5.4.1 optim ga
The Scilab macro optim_ga implements a Genetic Algorithm to ﬁnd the solution of an optimiza
tion problem with one objective function and bound constraints.
The following is an overview of the steps in the GA algorithm.
• processing of input arguments
In the case where the input cost function is a list, one deﬁnes the ”hidden” function _ga_f
which computes the cost function. If the input cost function is a regular Scilab function,
the ”hidden” function _ga_f simply encapsulate the input function.
• initialization
One computes the initial population with the init_func callback function (the default value
for init_func is init_ga_default)
• coding
One encodes the initial population with the codage_func callback function (default : coding_ga_identity)
• evolutionary algorithm as a loop over the generations
• decoding
One decodes the optimum population back to the original variable system
The loop over the generation is made of the following steps.
• reproduction : two list of children populations are computed, based on a randomized Wheel,
• crossover : the two populations are processed through the crossover_func callback function
(default : crossover_ga_default)
• mutation : the two populations are processed throught the mutation_func callback function
(default : mutation_ga_default)
• computation of cost functions : the _ga_f function is called to compute the ﬁtness for all
individuals of the two populations
• selection : the new generation is computed by processing the two populations through the
selection_func callback function (default : selection_ga_elitist)
5.4.2 optim moga, pareto ﬁlter
The optim_moga function is a multiobjective genetic algorithm. The method is based on [15].
The function pareto_filter extracts non dominated solution from a set.
33
5.4.3 optim nsga
The optim_nsga function is a multiobjective Niched Sharing Genetic Algorithm. The method is
based on [37].
5.4.4 optim nsga2
The function optim_nsga2 is a multiobjective Niched Sharing Genetic Algorithm. The method
is based on [14].
34
Chapter 6
Simulated Annealing
In this document, we describe the Simulated Annealing optimization methods, a new feature
available in Scilab v5 .
6.1 Introduction
Simulated annealing (SA) is a generic probabilistic metaalgorithm for the global optimization
problem, namely locating a good approximation to the global optimum of a given function in a
large search space. It is often used when the search space is discrete (e.g., all tours that visit a
given set of cities) [42].
Genetic algorithms have been introduced in Scilab v5 thanks to the work by Yann Collette
[9].
The current Simulated Annealing solver aims at ﬁnding the solution of
min
x
f(x)
with bounds constraints and with f : R
n
→R the cost function.
Reference books on the subject are [24, 25, 13].
6.2 Overview
The solver is made of Scilab macros, which enables a highlevel programming model for this opti
mization solver. The GA macros are based on the ”parameters”Scilab module for the management
of the (many) optional parameters.
To use the SA algorithm, one must perform the following steps :
• conﬁgure the parameters with calls to ”init param” and ”add param” especially,
– the neighbor function,
– the acceptance function,
– the temperature law,
• compute an initial temperature with a call to ”compute initial temp”
• ﬁnd an optimum by using the ”optim sa” solver
35
6.3 Example
The following example is extracted from the SA examples. The Rastrigin functin is used as an
example of a dimension 2 problem because it has many local optima but only one global optimum.
1 //
2 // Rastrigin function
3 //
4 function Res = mi n bd r as t r i gi n ( )
5 Res = [ −1 −1] ’ ;
6 endfunction
7 function Res = max bd r as t r i gi n ( )
8 Res = [ 1 1 ] ’ ;
9 endfunction
10 function Res = o p t i r a s t r i g i n ( )
11 Res = [ 0 0 ] ’ ;
12 endfunction
13 function y = r a s t r i g i n ( x)
14 y = x(1)ˆ2+x(2)ˆ2−cos (12∗x(1)) −cos (18∗x ( 2 ) ) ;
15 endfunction
16 //
17 // Set parameters
18 //
19 f unc = ’ rastri gi n ’ ;
20 Pr oba s t ar t = 0 . 8 ;
21 I t i nt e r n = 1000;
22 I t e xt e r n = 30;
23 I t Pr e = 100;
24 Min = eval ( ’ min bd ’+f unc+’ () ’ ) ;
25 Max = eval ( ’max bd ’+f unc+’ () ’ ) ;
26 x0 = (Max − Min ) . ∗ rand( si ze (Min , 1 ) , si ze ( Min , 2 ) ) + Min ;
27 def f ( ’y=f (x) ’ , ’y=’+f unc+’ (x) ’ ) ;
28 //
29 // Simulated Annealing with default parameters
30 //
31 pri ntf ( ’SA: geometrical decrease temperature law\n’ ) ;
32
33 sa params = i ni t par am ( ) ;
34 sa params = add param( sa params , ’ min delta ’ , −0. 1∗(Max−Min ) ) ;
35 sa params = add param( sa params , ’ max delta ’ , 0. 1∗(Max−Min ) ) ;
36 sa params = add param( sa params , ’ neigh func ’ , ne i gh f unc de f aul t ) ;
37 sa params = add param( sa params , ’ accept func ’ , ac c e pt f unc de f aul t ) ;
38 sa params = add param( sa params , ’ temp law’ , t emp l aw def aul t ) ;
39 sa params = add param( sa params , ’min bound’ , Min ) ;
40 sa params = add param( sa params , ’max bound’ ,Max) ;
41
42 T0 = comput e i ni t i al t emp ( x0 , f , Proba start , I t Pr e , sa params ) ;
43 pri ntf ( ’ I ni t i al temperature T0 = %f\n’ , T0 ) ;
44
45 [ x opt , f opt , s a mean l i s t , s a v a r l i s t , t e mp l i s t ] = . . .
36
46 opti m sa ( x0 , f , I t e xt e r n , I t i nt e r n , T0 , Log = %T, sa params ) ;
47
48 pri ntf ( ’ optimal solution : \n’ ) ; disp( x opt ) ;
49 pri ntf ( ’ value of the objective function = %f\n’ , f opt ) ;
50
51 s c f ( ) ;
52 drawlater ;
53 subplot ( 2 , 1 , 1 ) ;
54 xti t l e ( ’ Geometrical annealing ’ , ’ Iterati on ’ , ’Mean / Variance ’ ) ;
55 t = 1: length( s a me an l i s t ) ;
56 plot ( t , s a mean l i s t , ’ r ’ , t , s a v a r l i s t , ’ g ’ ) ;
57 l egend ( [ ’Mean’ , ’ Variance ’ ] ) ;
58 subplot ( 2 , 1 , 2 ) ;
59 xti t l e ( ’ Temperature evolution ’ , ’ Iterati on ’ , ’ Temperature ’ ) ;
60 for i =1: length( t )−1
61 plot ( [ t ( i ) t ( i +1)] , [ t e mp l i s t ( i ) t e mp l i s t ( i ) ] , ’k−’ ) ;
62 end
63 drawnow;
After some time, the following messages appear in the Scilab console.
optimal solution:
 0.0006975
 0.0000935
value of the objective function = 1.999963
The ﬁgure 6.1 presents the evolution of Mean, Variance and Temperature depending on the
iteration.
6.4 Neighbor functions
In the simulated annealing algorithm, a neighbour function is used in order to explore the domain
[43].
The prototype of a neighborhood function is the following :
1 function x nei gh = ne i gh f unc de f aul t ( x cur r ent , T, param)
where:
• x current represents the current point,
• T represents the current temperature,
• param is a list of parameters.
The following is a list of the neighbour functions available in the SA context :
• neigh_func_default : SA function which computes a neighbor of a given point. For
example, for a continuous vector, a neighbor will be produced by adding some noise to each
component of the vector. For a binary string, a neighbor will be produced by changing one
bit from 0 to 1 or from 1 to 0.
37
Figure 6.1: Convergence of the simulated annealing algorithm
• neigh_func_csa : The classical neighborhood relationship for the simulated annealing.
The neighbors distribution is a gaussian distribution which is more and more peaked as the
temperature decrease.
• neigh_func_fsa : The Fast Simulated Annealing neghborhood relationship. The corre
sponding distribution is a Cauchy distribution which is more and more peaked as the tem
perature decreases.
• neigh_func_vfsa : The Very Fast Simulated Annealing neighborhood relationship. This
distribution is more and more peaked as the temperature decreases.
6.5 Acceptance functions
There exist several kind of simulated annealing optimization methods:
• the Fast Simulated Annealing,
• the simulated annealing based on metropolishasting acceptance function,
• etc...
To implement these various simulated annealing optimization methods, you only need to
change the acceptance function. For common optimization, you need not to change the default
acceptance function.
The following is a list of acceptance functions available in Scilab SAs :
38
• accept_func_default : is the default acceptance function, based on the exponential func
tion
level = exp
_
−
F
neigh
−F
current
T
_
• accept_func_vfsa : is the Very Fast Simulated Annealing function, deﬁned by :
Level =
1
1 + exp
_
−
F
current
−F
neigh
T
_
6.6 Temperature laws
In the simulated annealing algorithm, a temperature law is used in a statistical criteria for the
update of the optimum [43]. If the new (neighbor) point improves the current optimum, the
update is done with the new point replacing the old optimum. If not, the update may still be
processed, provided that a statistical criteria is satisﬁed. The statistical law decreases while the
iterations are processed.
There are 5 temperature laws available in the SA context :
• temp_law_default : A SA function which computes the temperature of the next tempera
ture stage
• temp_law_csa : The classical temperature decrease law, the one for which the convergence
of the simulated annealing has been proven
• temp_law_fsa : The Szu and Hartley Fast simulated annealing
• temp_law_huang : The Huang temperature decrease law for the simulated annealing
• temp_law_vfsa : This function implements the Very Fast Simulated Annealing from L.
Ingber
6.7 optim sa
The optim_sa macro implements the simulated annealing solver. It allows to ﬁnd the solution of
an minimization problem with bound constraints.
It is based on an iterative update of two points :
• the current point is updated by taking into account the neighbour function and the accep
tance criterium,
• the best point is the point which achieved the minimum of the objective function over the
iterations.
While the current point is used internally to explore the domain, only the best point is returned
as the algorithm output.
The algorithm is based on the following steps, which include a main, external loop over the
temperature decreases, and an internal loop.
39
• processing of input arguments,
• initialization,
• loop over the number of temperature decreases.
For each iteration over the temperature decreases, the following steps are processed.
• loop over internal iterations, with constant temperature,
• if history is required by user, store the temperature, the x iterates, the values of f,
• update the temperature with the temperature law.
The internal loop allows to explore the domain and is based on the neighbour function. It is
based on the following steps.
• compute a neighbour of the current point,
• compute the objective function for that neighbour
• if the objective decreases or if the acceptance criterium is true, then overwrite the current
point with the neighbour
• if the cost of the best point is greater than the cost of the current point, overwrite the best
point by the current point.
40
Chapter 7
LMITOOL: a Package for LMI
Optimization in Scilab
R. Nikoukhah Ramine.Nikoukhah@inria.fr
F. Delebecque Francois.Delebecque@inria.fr
L. El Ghaoui ENSTA, 32, Bvd. Victor, 75739 Paris, France. Internet: elghaoui@ensta.fr.
Research supported in part by DRET under contract 92017BC14
This chapter describes a userfriendly Scilab package, and in particular its two main functions
lmisolver and lmitool for solving Linear Matrix Inequalities problems. This package uses Scilab
function semidef, an interface to the program Semideﬁnite Programming SP (Copyright c 1994
by Lieven Vandenberghe and Stephen Boyd) distributed with Scilab.
7.1 Purpose
Many problems in systems and control can be formulated as follows (see [6]):
Σ :
_
_
_
minimize f(X
1
, . . . , X
M
)
subject to
_
G
i
(X
1
, . . . , X
M
) = 0, i = 1, 2, ..., p,
H
j
(X
1
, . . . , X
M
) ≥ 0, j = 1, 2, .., q.
where
• X
1
, . . . , X
M
are unknown real matrices, referred to as the unknown matrices,
• f is a real linear scalar function of the entries of the unknown matrices X
1
, . . . , X
M
; it is
referred to as the objective function,
• G
i
’s are real matrices with entries which are aﬃne functions of the entries of the unknown
matrices, X
1
, . . . , X
M
; they are referred to as “Linear Matrix Equality” (LME) functions,
• H
j
’s are real symmetric matrices with entries which are aﬃne functions of the entries of the
unknown matrices X
1
, . . . , X
M
; they are referred to as “Linear Matrix Inequality” (LMI)
functions. (In this report, the V ≥ 0 stands for V positive semideﬁnite unless stated
otherwise).
41
The purpose of LMITOOL is to solve problem Σ in a userfriendly manner in Scilab, using the code
SP [23]. This code is intended for small and mediumsized problems (say, up to a few hundred
variables).
7.2 Function lmisolver
LMITOOL is built around the Scilab function lmisolver. This function computes the solution
X
1
, . . . , X
M
of problem Σ, given functions f, G
i
and H
j
. To solve Σ, user must provide an
evaluation function which “evaluates” f, G
i
and H
j
as a function the unknown matrices, as well
as an initial guess on the values of the unknown matrices. User can either invoke lmisolver
directly, by providing the necessary information in a special format or he can use the interactive
function lmitool described in Section 7.3.
7.2.1 Syntax
[XLISTF[,OPT]] = lmisolver(XLIST0,EVALFUNC[,options])
where
• XLIST0: a list structure including matrices and/or list of matrices. It contains initial guess
on the values of the unknown matrices. In general, the ith element of XLIST0 is the initial
guess on the value of the unknown matrix X
i
. In some cases however it is more convenient
to deﬁne one or more elements of XLIST0 to be lists (of unknown matrices) themselves. This
is a useful feature when the number of unknown matrices is not ﬁxed a priori (see Example
of Section 7.2.2).
The values of the matrices in XLIST0, if compatible with the LME functions, are used as
intial condition for the optimization algorithm; they are ignored otherwise. The size and
structure of XLIST0 are used to set up the problem and determine the size and structure of
the output XLISTF.
• EVALFUNC: a Scilab function called evaluation function (supplied by the user) which evalu
ates the LME, LMI and objective functions, given the values of the unknown matrices. The
syntax is:
[LME,LMI,OBJ]=EVALFUNC(XLIST)
where
– XLIST: a list, identical in size and structure to XLIST0.
– LME: a list of matrices containing values of the LME functions G
i
’s for X values in
XLIST. LME can be a matrix in case there is only one LME function to be evaluated
(instead of a list containing this matrix as unique element). It can also be a list of a
mixture of matrices and lists which in turn contain values of LME’s, and so on.
– LMI: a list of matrices containing the values of the LMI functions H
j
’s for X values
in XLIST. LMI can also be a matrix (in case there is only one LMI function to be
evaluated). It can also be a list of a mixture of matrices and lists which in turn contain
values of of LMI’s, and so on.
42
– OBJ: a scalar equal to the value of the objective function f for X values in XLIST.
If the Σ problem has no equality constraints then LME should be []. Similarly for LMI and
OBJ.
• options: a 5×1 vector containing optimization parameters Mbound, abstol, nu, maxiters,
and reltol, see manual page for semidef for details (Mbound is a multiplicative coeﬃcient
for M). This argument is optional, if omitted, default parameters are used.
• XLISTF: a list, identical in size and structure to XLIST0 containing the solution of the
problem (optimal values of the unknown matrices).
• OPT: a scalar corresponding to the optimal value of the minimization problem Σ.
7.2.2 Examples
Statefeedback with control saturation constraint
Consider the linear system
˙ x = Ax + Bu
where A is an n × n and B, an n × n
u
matrix. There exists a stabilizing state feedback K such
that for every initial condition x(0) with x(0) ≤ 1, the resulting control satisﬁes u(t) for all
t ≥ 0, if and only if there exist an n×n matrix Q and an n
u
×n matrix Y satisfying the equality
constraint
Q−Q
T
= 0
and the inequality constraints
Q ≥ 0
−AQ−QA
T
−BY −Y
T
B
T
> 0
_
u
2
max
I Y
Y
T
Q
_
≥ 0
in which case one such K can be constructed as K = Y Q
−1
.
To solve this problem using lmisolver, we ﬁrst need to construct the evaluation function.
function [LME,LMI,OBJ]=sf_sat_eval(XLIST)
[Q,Y]=XLIST(:)
LME=QQ’
LMI=list(A*QQ*A’B*YY’*B’,[umax^2*eye(Y*Y’),Y;Y’,Q],Qeye())
OBJ=[]
Note that OBJ=[] indicates that the problem considered is a feasibility problem, i.e., we are only
interested in ﬁnding a set of X’s that satisfy LME and LMI functions.
Assuming A, B and umax already exist in the environment, we can call lmisolver, and recon
struct the solution in Scilab, as follows:
43
> Q_init=zeros(A);
> Y_init=zeros(B’);
> XLIST0=list(Q_init,Y_init);
> XLIST=lmisolver(XLIST0,sf_sat_eval);
> [Q,Y]=XLIST(:)
These Scilab commands can of course be encapsulated in a Scilab function, say sf_sat. Then,
To solve this problem, all we need to do is type:
> [Q,Y]=sf_sat(A,B,umax)
We call sf_sat the solver function for this problem.
Control of jump linear systems
We are given a linear system
˙ x = A(r(t))x +B(r(t))u,
where A is n×n and B is n×n
u
. The scalar parameter r(t) is a continuoustime Markov process
taking values in a ﬁnite set {1, . . . , N}.
The transition probabilities of the process r are deﬁned by a “transition matrix” Π = (π
ij
),
where π
ij
’s are the transition probability rates from the ith mode to the jth. Such systems,
referred to as “jump linear systems”, can be used to model linear systems subject to failures.
We seek a statefeedback control law such that the resulting closedloop system is meansquare
stable. That is, for every initial condition x(0), the resulting trajectory of the closedloop system
satisﬁes lim
t→∞
Ex(t)
2
= 0.
The control law we look for is a modedependent linear statefeedback, i.e. it has the form
u(t) = K(r(t))x(t); K(i)’s are n
u
×n matrices (the unknowns of our control problem).
It can be shown that this problem has a solution if and only if there exist n × n matrices
Q(1), . . . , Q(N), and n
u
×n matrices Y (1), . . . , Y (N), such that
Q(i) −Q(i)
T
= 0,
TrQ(1) + . . . +TrQ(N) −1 = 0.
and
_
Q(i) Y (i)
T
Y (i) I
_
> 0,
−
_
A(i)Q(i) + Q(i)A(i)
T
+B(i)Y (i) + Y (i)
T
B(i)
T
+
N
j=1
π
ji
Q(j)
_
> 0, i = 1, . . . , N,
If such matrices exist, a stabilizing statefeedback is given by K(i) = Y (i)Q(i)
−1
, i = 1, . . . , N.
In the above problem, the data matrices are A(1), . . . , A(N), B(1), . . . , B(N) and the tran
sition matrix Π. The unknown matrices are Q(i)’s (which are symmetric n × n matrices) and
Y (i)’s (which are n
u
×n matrices). In this case, both the number of the data matrices and that
of the unknown matrices are apriori unknown.
The above problem is obviously a Σ problem. In this case, we can let XLIST be a list of two
lists: one representing the Q’s and the other, the Y ’s.
The evaluation function required for invoking lmisolver can be constructed as follows:
44
function [LME,LMI,OBJ]=jump_sf_eval(XLIST)
[Q,Y]=XLIST(:)
N=size(A); [n,nu]=size(B(1))
LME=list(); LMI1=list(); LMI2=list()
tr=0
for i=1:N
tr=tr+trace(Q(i))
LME(i)=Q(i)Q(i)’
LMI1(i)=[Q(i),Y(i)’;Y(i),eye(nu,nu)]
SUM=zeros(n,n)
for j=1:N
SUM=SUM+PI(j,i)*Q(j)
end
LMI2(i)= A(i)*Q(i)+Q(i)*A(i)’+B(i)*Y(i)+Y(i)’*B(i)’+SUM
end
LMI=list(LMI1,LMI2)
LME(N+1)=tr1
OBJ=[]
Note that LMI is also a list of lists containing the values of the LMI matrices. This is just a matter
of convenience.
Now, we can solve the problem in Scilab as follows (assuming lists A and B, and matrix PI
have already been deﬁned).
First we should initialize Q and Y.
> N=size(A); [n,nu]=size(B(1)); Q_init=list(); Y_init=list();
> for i=1:N, Q_init(i)=zeros(n,n);Y_init(i)=zeros(nu,n);end
Then, we can use lmisolver as follows:
> XLIST0=list(Q_init,Y_init)
> XLISTF=lmisolver(XLIST0,jump_sf_eval)
> [Q,Y]=XLISTF(:);
The above commands can be encapsulated in a solver function, say jump_sf, in which case
we simply need to type:
> [Q,Y]=jump_sf(A,B,PI)
to obtain the solution.
Descriptor Lyapunov inequalities
In the study of descriptor systems, it is sometimes necessary to ﬁnd (or ﬁnd out that it does not
exist) an n ×n matrix X satisfying
E
T
X = X
T
E ≥ 0
A
T
X + X
T
A +I ≤ 0
where E and A are n × n matrices such that E, A is a regular pencil. In this problem, which
clearly is a Σ problem, the LME functions play important role. The evaluation function can be
written as follows
45
function [LME,LMI,OBJ]=dscr_lyap_eval(XLIST)
X=XLIST(:)
LME=E’*XX’*E
LMI=list(A’*XX’*Aeye(),E’*X)
OBJ=[]
and the problem can be solved by (assuming E and A are already deﬁned)
> XLIST0=list(zeros(A))
> XLISTF=lmisolver(XLIST0,dscr_lyap_eval)
> X=XLISTF(:)
Mixed H
2
/H
∞
Control
Consider the linear system
˙ x = Ax +B
1
w +B
2
u
z
1
= C
1
x +D
11
w + D
12
u
z
2
= C
2
x +D
22
u
The mixed H
2
/H
∞
control problem consists in ﬁnding a stabilizing feedback which yields T
z
1
w
∞
<
γ and minimizes T
z
2
w
2
where T
z
1
w
∞
and T
z
2
w
2
denote respectively the closedloop transfer
functions from w to z
1
and z
2
. In [22], it is shown that the solution to this problem can be
expressed as K = LX
−1
where X and L are obtained from the problem of minimizing Trace(Y )
subject to:
X −X
T
= 0, Y −Y
T
= 0,
and
−
_
AX +B
2
L + (AX +B
2
L)
T
+B
1
B
T
1
XC
T
1
+L
T
D
T
12
+ B
1
D
T
11
C
1
X +D
12
L +D
11
B
T
1
−γ
2
I +D
11
D
T
11
_
> 0
_
Y C
2
X +D
22
L
(C
2
X +D
22
L)
T
X
_
> 0
To solve this problem with lmisolver, we deﬁne the evaluation function:
function [LME,LMI,OBJ]=h2hinf_eval(XLIST)
[X,Y,L]=XLIST(:)
LME=list(XX’,YY’);
LMI=list([A*X+B2*L+(A*X+B2*L)’+B1*B1’,X*C1’+L’*D12’+B1*D11’;...
(X*C1’+L’*D12’+B1*D11’)’,gamma^2*eye()+D11*D11’],...
[Y,C2*X+D22*L;(C2*X+D22*L)’,X])
OBJ=trace(Y);
and use it as follows:
> X_init=zeros(A); Y_init=zeros(C2*C2’); L_init=zeros(B2’)
> XLIST0=list(X_init,Y_init,L_init);
> XLISTF=lmisolver(XLIST0,h2hinf_eval);
> [X,Y,L]=XLISTF(:)
46
Descriptor Riccati equations
In Kalman ﬁltering for descriptor system
Ex(k + 1) = Ax(k) + u(k)
y(k + 1) = Cx(k + 1) + r(k)
where u and r are zeromean, white Gaussian noise sequences with covariance Q and R respec
tively, one needs to obtain the positive solution to the descriptor Riccati equation (see [33])
P = −
_
0 0 I
_
_
_
APA
T
+Q 0 E
0 R C
E
T
C
T
0
_
_
−1
_
_
0
0
I
_
_
.
It can be shown that this problem can be formulated as a Σ problem as follows: maximize
Trace(P) under constraints
P −P
T
= 0
and
_
_
APA
T
+Q 0 EP
0 R CP
P
T
E
T
P
T
C
T
P
_
_
≥ 0.
The evaluation function is:
function [LME,LMI,OBJ]=ric_dscr_eval(XLIST)
LME=PP’
LMI=[A*P*A’+Q,zeros(A*C’),E*P;zeros(C*A’),R,C*P;P*E’,P*C’,P]
OBJ=trace(P)
which can be used as follows (asuming E, A, C, Q and R are deﬁned and have compatible
sizes–note that E and A need not be square).
> P_init=zeros(A’*A)
> P=lmisolver(XLIST0,ric_dscr_eval)
Linear programming with equality constraints
Consider the following classical optimization problem
minimize e
T
x
subject to Ax + b ≥ 0,
Cx + d = 0.
where A and C are matrices and e, b and d are vectors with appropriate dimensions. Here the
sign ≥ is to be understood elementwise.
This problem can be formulated in LMITOOL as follows:
function [LME,LMI,OBJ]=linprog_eval(XLIST)
[x]=XLIST(:)
[m,n]=size(A)
47
LME=C*x+d
LMI=list()
tmp=A*x+b
for i=1:m
LMI(i)=tmp(i)
end
OBJ=e’*x
and solved in Scilab by (assuming A, C, e, b and d and an initial guess x0 exist in the environment):
> x=lmisolver(x0,linprog_eval)
Sylvester Equation
The problem of ﬁnding matrix X satisfying
AX +XB = C
or
AXB = C
where A and B are square matrices (of possibly diﬀerent sizes) is a wellknown problem. We refer
to the ﬁrst equation as the continuous Sylvester equation and the second, the discrete Sylvester
equation.
These two problems can easily be formulated as Σ problems as follows:
function [LME,LMI,OBJ]=sylvester_eval(XLIST)
[X]=XLIST(:)
if flag==’c’ then
LME=A*X+X*BC
else
LME=A*X*BC
end
LMI=[]
OBJ=[]
with a solver function such as:
function [X]=sylvester(A,B,C,flag)
[na,ma]=size(A);[nb,mb]=size(B);[nc,mc]=size(C);
if ma<>namb<>nbnc<>namc<>nb then error("invalid dimensions");end
XLISTF=lmisolver(zeros(nc,mc),sylvester_eval)
X=XLISTF(:)
Then, to solve the problem, all we need to do is to (assuming A, B and C are deﬁned)
> X=sylvester(A,B,C,’c’)
for the continuous problem and
> X=sylvester(A,B,C,’d’)
for the discrete problem.
48
7.3 Function LMITOOL
The purpose of LMITOOL is to automate most of the steps required before invoking lmisolver.
In particular, it generates a *.sci ﬁle including the solver function and the evaluation function
or at least their skeleton. The solver function is used to deﬁne the initial guess and to modify
optimization parameters (if needed).
lmitool can be invoked with zero, one or three arguments.
7.3.1 Noninteractive mode
lmitool can be invoked with three input arguments as follows:
Syntax
txt=lmitool(probname,varlist,datalist)
where
• probname: a string containing the name of the problem,
• xlist: a string containing the names of the unknown matrices (separated by commas if
there are more than one).
• dlist: a string containing the names of data matrices (separated by commas if there are
more than one).
• txt: a string providing information on what the user should do next.
In this mode, lmitool generates a ﬁle in the current directory. The name of this ﬁle is obtained
by adding “.sci” to the end of probname. This ﬁle is the skeleton of a solver function and the
corresponding evaluation function.
Example
Suppose we want to use lmitool to solve the problem presented in Section 7.2.2. Invoking
>txt=lmitool(’sf_sat’,’Q,Y’,’A,B,umax’)
yields the output
> txt =
! To solve your problem, you need to !
! !
!1 edit file /usr/home/DrScilab/sf_sat.sci !
! !
!2 load (and compile) your functions: !
! !
! getf(’/usr/home/DrScilab/sf_sat.sci’,’c’) !
49
! !
!3 Define A,B,umax and call sf_sat function: !
! !
! [Q,Y]=sf_sat(A,B,umax) !
! !
!To check the result, use [LME,LMI,OBJ]=sf_sat_eval(list(Q,Y)) !
and results in the creation of the ﬁle ’/usr/home/curdir/sf sat.sci’ with the following content:
function [Q,Y]=sf_sat(A,B,umax)
// Generated by lmitool on Tue Feb 07 10:30:35 MET 1995
Mbound = 1e3;
abstol = 1e10;
nu = 10;
maxiters = 100;
reltol = 1e10;
options=[Mbound,abstol,nu,maxiters,reltol];
///////////DEFINE INITIAL GUESS BELOW
Q_init=...
Y_init=...
///////////
XLIST0=list(Q_init,Y_init)
XLIST=lmisolver(XLIST0,sf_sat_eval,options)
[Q,Y]=XLIST(:)
/////////////////EVALUATION FUNCTION////////////////////////////
function [LME,LMI,OBJ]=sf_sat_eval(XLIST)
[Q,Y]=XLIST(:)
/////////////////DEFINE LME, LMI and OBJ BELOW
LME=...
LMI=...
OBJ=...
It is easy to see how a small amount of editing can do the rest!
7.3.2 Interactive mode
lmitool can be invoked with zero or one input argument as follows:
50
Syntax
txt=lmitool()
txt=lmitool(file)
where
• file: is a string giving the name of an existing “.sci” ﬁle generated by lmitool.
In this mode, lmitool is fully interactive. Using a succession of dialogue boxes, user can com
pletely deﬁne his problem. This mode is very easy to use and its operation completely self
explanatory. Invoking lmitool with one argument allows the user to start oﬀ with an existing
ﬁle. This mode is useful for modifying existing ﬁles or when the new problem is not too much
diﬀerent from a problem already treated by lmitool.
Example
Consider the following estimation problem
y = Hx + V w
where x is unknown to be estimated, y is known, w is a unitvariance zeromean Gaussian vector,
and
H ∈ Co{H(1), ..., H(N)} , V ∈ Co{V (1), ..., V (N)}
where Co denotes the convex hull and H(i) and V (i), i = 1, ..., N, are given matrices.
The objective is to ﬁnd L such that the estimate
ˆ x = Ly
is unbiased and the worst case estimation error variance E(x − ˆ x
2
) is minimized.
It can be shown that this problem can be formulated as a Σ problem as follows: minimize γ
subject to
I −LH(i) = 0, i = 1, ..., N,
X(i) −X(i)
T
= 0, i = 1, ..., N,
and
_
I (L(i)V (i))
T
L(i)V (i) X(i)
_
≥ 0, i = 1, ..., N,
γ −Trace(X(i)) ≥ 0, i = 1, ..., N.
To use lmitool for this problem, we invoke it as follows:
> lmitool()
This results is an interactive session which is partly illustrated in following ﬁgures.
51
7.4 How lmisolver works
The function lmisolver works essentially in four steps:
1. Initial setup. The sizes and structure of the initial guess are used to set up the problem,
and in particular the size of the unknown vector.
2. Elimination of equality constraints. Making repeated calls to the evaluation function,
lmisolver generates a canonical representation of the form
minimize ˜ c
T
z
subject to
˜
F
0
+z
1
˜
F
1
+· · · +z
˜ m
˜
F
˜ m
≥ 0, Az +b = 0,
where z contains the coeﬃcients of all matrix variables. This step uses extensively sparse
matrices to speed up the computation and reduce memory requirement.
3. Elimination of variables. Then, lmisolver eliminates the redundant variables. The equality
constraints are eliminated by computing the null space N of A and a solution z
0
(if any) of
Ax +b = 0. At this stage, all solutions of the equality constraints are parametrized by
z = Nx + z
0
,
where x is a vector containing the independent variables. The computation of N, z
0
is done
using sparse LU functions of Scilab.
Once the equality constraints are eliminated, the problem is reformulated as
minimize c
T
x
subject to F
0
+x
1
F
1
+· · · +x
m
F
m
≥ 0,
where c is a vector, and F
0
, . . . , F
m
are symmetric matrices, and x contains the indepen
dent elements in the matrix variables X
1
, . . . , X
M
. (If the F
i
’s are dependent, a column
compression is performed.)
Figure 7.1: This window must be edited to deﬁne problem name and the name of variables used.
52
Figure 7.2: For the example at hand the result of the editing should look something like this.
4. Optimization. Finally, lmisolver makes a call to the function semidef (an interface to SP
[23]). This phase is itself divided into a feasibility phase and a minimization phase (only
if the linear objective function is not empty). The feasibility phase is avoided if the initial
guess is found to be feasible.
The function semidef is called with the optimization parameters abstol, nu, maxiters,
reltol. The parameter M is set above the value
Mbnd*max(sum(abs([F0 ... Fm])))
For details about the optimization phase, and the meaning of the above optimization pa
rameters see manual page for semidef.
7.5 Other versions
LMITOOL is also available on Matlab. The Matlab version can be obtained by anonymous ftp from
ftp.ensta.fr under /pub/elghaoui/lmitool.
53
Figure 7.3: This is the skeleton of the solver function and the evaluation function generated by
LMITOOL using the names deﬁned previously.
54
Figure 7.4: After editing, we obtain.
Figure 7.5: A ﬁle is proposed in which the solver and evaluation functions are to be saved. You
can modify it if you want.
55
Chapter 8
Optimization data ﬁles
This section presents the optimization data ﬁles which can be used to conﬁgure a speciﬁc opti
mization problem in Scilab. The following is a (nonexhaustive) list of ASCII ﬁle formats often
used in optimization softwares :
• SIF : Standard Input Format [1, 30],
• GAMS : General Algebraic Modeling System [40, 16]
• AMPL : A Mathematical Programming Language [10, 39]
• MPS : Mathematical Programming System [27, 41]
but other ﬁle formats appeared in recent years, such as the XMLbased ﬁle format OSiL [35, 8, 36].
The following sections describe Scilab tools to manage optimization data ﬁles.
8.1 MPS ﬁles and the Quapro toolbox
The Quapro toolbox implements the readmps function, which reads a ﬁle containing description
of an LP problem given in MPS format and returns a tlist describing the optimization problem.
It is an interface with the program rdmps1.f of hopdm (J. Gondzio). For a description of the
variables, see the ﬁle rdmps1.f. MPS format is a standard ASCII medium for LP codes. MPS
format is described in more detail in Murtagh’s book [30].
8.2 SIF ﬁles and the CUTEr toolbox
The SIF ﬁle format can be processed with the CUTEr Scilab toolbox. Given a SIF [1] ﬁle the func
tion sifdecode generates associated Fortran routines RANGE.f, EXTER.f, ELFUN.f, GROUP.f
and if automatic diﬀerentiation is required ELFUND.f, GROUPD.f, EXTERA.f. An associated
data ﬁle named OUTSDIF.d and an Output messages ﬁle OUTMESS are also generated. All
these ﬁles are created in the directory whose path is given in Pathout. The sifdecode function
is based on the Sifdec code [20]. More precisely it results of an interface of SDLANC Fortran
procedure.
56
Chapter 9
Scilab Optimization Toolboxes
Some Scilab toolboxes are designed to solve optimization problems. In this chapter, we begin by
presenting the Quapro toolbox, which allows to solve linear and quadratic problems. Then we
outline other main optimization toolboxes.
9.1 Quapro
The Quapro toolbox was formely a Scilab builtin optimization tool. It has been transformed into
a toolbox for license reasons.
9.1.1 Linear optimization
Mathematical point of view
This kind of optimization is the minimization of function f(x) with
f(x) = p
T
x
under:
• no constraints
• inequality constraints (9.1)
• or inequality constraints and bound constraints ((9.1) & (9.2))
• or inequality constraints, bound constraints and equality constraints ((9.1) & (9.2) & (9.3)).
C ∗ x ≤ b (9.1)
ci ≤ x ≤ cs (9.2)
C
e
∗ x = b
e
(9.3)
57
Scilab function
Scilab function called linpro is designed for linear optimization programming. For more details
about this function, please refer to Scilab online help This function and associated routines
have been written by Cecilia Pola Mendez and Eduardo Casas Renteria from the University of
Cantabria. Please note that this function can not solve problems based on sparse matrices. For
this kind of problem, you can use a Scilab toolbox called LIPSOL that gives an equivalent of
linpro for sparse matrices. LIPSOL is available on Scilab web site
Optimization routines
Scilab linpro function is based on:
• some Fortran routines written by the authors of linpro
• some Fortran Blas routines
• some Fortran Scilab routines
• some Fortran Lapack routines
9.1.2 Linear quadratic optimization
Mathematical point of view
This kind of optimization is the minimization of function f(x) with
f(x) =
1
2
x
T
Qx +p
T
x
under:
• no constraints
• inequality constraints (9.1)
• or inequality constraints and bound constraints ((9.1) & (9.2))
• or inequality constraints, bound constraints and equality constraints ((9.1) & (9.2) & (9.3)).
Scilab function
Scilab functions called quapro (whatever Q is) and qld (when Q is positive deﬁnite) are designed
for linear optimization programming. For more details about these functions, please refer to
Scilab online help for quapro and Scilab online help for qld qld function and associated routine
have been written by K. Schittkowski from the University of Bayreuth, A.L. Tits and J.L. Zhou
from the University of Maryland. quapro function and associated routines have been written
by Cecilia Pola Mendez and Eduardo Casas Renteria from the University of Cantabria. Both
functions can not solve problems based on sparse matrices.
58
Optimization routines
Scilab quapro function is based on:
• some Fortran routines written by the authors of linpro
• some Fortran Blas routines
• some Fortran Scilab routines
• some Fortran Lapack routines
9.2 CUTEr
CUTEr is a versatile testing environment for optimization and linear algebra solvers [31]. This
toolbox is a scilab port by Serge Steer and Bruno Durand of the original Matlab toolbox.
A typical use start from problem selection using the scilab function sifselect. This gives
a vector of problem names corresponding to selection criteria [32]. The available problems are
located in the sif directory.
The sifbuild function can then be used to generate the fortran codes associated to a given
problem, to compile them and dynamically link it to Scilab. This will create a set of problem
relative functions, for example, ufn or ugr. This functions can be called to compute the objective
function or its gradient at a given point.
The sifoptim function automatically applies the optim function to a selected problem.
A Fortran compiler is mandatory to build problems.
This toolbox contains the following parts.
• Problem database
A set of testing problems coded in ”Standard Input Format” (SIF) is included in the sif/
subdirectory. This set comes from www.numerical.rl.ac.uk/cute/mastsif.html. The Scilab
function sifselect can be used to select some of this problems according to objective
function properties, contraints properties and regularity properties
• SIF format decoder
The Scilab function sifdecode can be used to generate the Fortran codes associated to a
given problem, while the Scilab function buildprob compiles and dynamically links these
fortran code with Scilab
• problem relative functions
The execution of the function buildprob adds a set of functions to Scilab. The ﬁrst one
is usetup for unconstrained or bounded problems or csetup for problems with general
contraints. These functions are to be called before any of the following to initialize the
problem relative data (only one problem can be run at a time). The other functions allow
to compute the objective, the gradient, the hessian values, ... of the problem at a given
point (see ufn, ugr, udh, ... for unconstrained or bounded problems or cfn, cgr, cdh, ...
for problems with general contraints)
59
• CUTEr and optim The Scilab function optim can be used together with CUTEr using either
the external function ucost or the driver function sifoptim.
The following is a list of references for the CUTEr toolbox :
• CUTEr toolbox on Scilab Toolbox center
• CUTEr website
9.3 The Unconstrained Optimization Problem Toolbox
The Unconstrained Optimization Problem Toolbox provides 35 unconstrained optimization prob
lems.
The goal of this toolbox is to provide unconstrained optimization problems in order to test
optimization algorithms.
The More, Garbow and Hillstrom collection of test functions [29] is widely used in testing
unconstrained optimization software. The code for these problems is available in Fortran from
the netlib software archives.
It provides the function value, the gradient, the function vector, the Jacobian and provides
the Hessian matrix for 18 problems. It provides the starting point for each problem, the optimum
function value and the optimum point x for many problems. Additionnally, it provides ﬁnite
diﬀerence routines for the gradient, the Jacobian and the Hessian matrix. The functions are
based on macros based functions : no compiler is required, which is an advantage over the CUTEr
toolbox. Finally, all function values, gradients, Jacobians and Hessians are tested.
This toolbox is available in ATOMS :
http://atoms.scilab.org/toolboxes/uncprb
and is manage under Scilab’s Forge :
http://forge.scilab.org/index.php/p/uncprb
To install it, type the following statement in Scilab v5.2 (or better).
1 at oms I ns t al l ( ’ uncprb ’ )
9.4 Other toolboxes
• Interface to CONMIN: An interface to the NASTRAN / NASA CONMIN optimization
program by Yann Collette. CONMIN can solve a nonlinear objective problem with non
linear constraints. CONMIN uses a twostep limited memory quasiNewtonlike Conjugate
Gradient. The CONMIN optimization method is currently used in NASTRAN (a profes
sionnal ﬁnite element tool) and the optimization part of NASTRAN (the CONMIN tool).
The CONMIN fortran program has been written by G. Vanderplaats (1973).
– CONMIN on Scilab Toolbox center
• Diﬀerential Evolution: random search of a global minimum by Helmut Jarausch. This
toolbox is based on a RainerStorn algorithm.
60
– Diﬀerential Evolution on Scilab Toolbox center
• FSQP Interface: interface for the Feasible Sequential Quadratic Programming library.
This toolbox is designed for nonlinear optimization with equality and inequality constraints.
FSQP is a commercial product.
– FSQP on Scilab Toolbox center
– FSQP website
• IPOPT interface: interface for Ipopt, which is based on an interior point method which
can handle equality and inequality nonlinear constraints. This solver can handle large scale
optimization problems. As open source software, the source code for Ipopt is provided
without charge. You are free to use it, also for commercial purposes. This ScilabIpopt
interface was based on the Matlab Mex Interface developed by Claas Michalik and Steinar
Hauan. This version only works on linux, scons and Scilab >=4.0. Tested with gcc 4.0.3.
Modiﬁcations to Scilab Interface made by Edson Cordeiro do Valle.
– Ipopt on Scilab Toolbox center
– Ipopt website
• Interface to LIPSOL: sparse linear problems with interior points method by H. Rubio Scola.
LIPSOL can minimize a linear objective with linear constraints and bound constraints. It
is based on a primaldual interior point method, which uses sparsematrix datastructure to
solve large, sparse, symmetric positive deﬁnite linear systems. LIPSOL is written by Yin
Zhang . The original Matlabbased code has been adapted to Scilab by H. Rubio Scola
(University of Rosario, Argentina). It is distributed freely under the terms of the GPL.
LIPSOL also uses the ORNL sparse Cholesky solver version 0.3 written by Esmond Ng and
Barry Peyton by H. Rubio Scola.
– LIPSOL on Scilab Toolbox center
– LIPSOL website
– LIPSOL User’s Guide
• LPSOLVE: an interface to lp solve. lp solve is a free mixed integer/binary linear program
ming solver with full source, examples and manuals. lp solve is under LGPL, the GNU lesser
general public license. lp solve uses the ’Simplex’ algorithm and sparse matrix methods for
pure LP problems.
– LPSOLVE toolbox on Scilab Toolbox center
– lp solve solver on Sourceforge
– lp solve on Geocities
– lp solve Yahoo Group
• NEWUOA: NEWUOA is a software developped by M.J.D. Powell for unconstrained op
timization without derivatives. The NEWUOA seeks the least value of a function F(x)
(x is a vector of dimension n ) when F(x) can be calculated for any vector of variables x
61
. The algorithm is iterative, a quadratic model being required at the beginning of each
iteration, which is used in a trust region procedure for adjusting the variables. When the
quadratic model is revised, the new model interpolates F at m points, the value m=2n+1
being recommended.
– NEWUOA toolbox on Scilab Toolbox center
– NEWUOA at INRIA Alpes
62
Chapter 10
Missing optimization features in Scilab
Several optimization features are missing in Scilab. Two classes of missing features are to analyse :
• features which are not available in Scilab, but which are available as toolboxes (see previous
section),
• features which are not available neither in Scilab, nor in toolboxes.
Here is a list of features which are not available in Scilab, but are available in toolboxes. These
features would be to include in Scilab.
• integer parameter with linear objective solver and sparse matrices : currently available in
LPSOLVE toolbox, based on the simplex method,
• linear objective with sparse matrices : currently available in LIPSOL, based on interior
points method,
• nonlinear objective and non linear constraints : currently available in interface to IPOPT
toolbox, based on interior point methods,
• nonlinear objective and non linear constraints : currently available in interface to CONMIN
toolbox, based on method of feasible directions,
Notice that IPOPT is a commercial product and CONMIN is a domainpublic library. Therefore
the only opensource, free, nonlinear solver with non linear constraints tool available with Scilab
is the interface to CONMIN.
Here is a list of features which are not available neither in Scilab, nor in toolboxes.
• quadratic objective solver with sparse objective matrix,
• simplex programming method (*),
• nonlinear objective with nonlinear constraints (*),
• nonlinear objective with nonlinear constraints problems based on sparse linear algebra,
• enabling/disabling of unknowns or constraints,
• customization of errors for constraints.
Functionalities marked with a (*) would be available in Scilab if the MODULOPT library
embedded in Scilab was updated.
63
Conclusion
Even if Scilab itself has lacks in optimization functionalities, all embedded functions are very
useful to begin with. After that, by downloading and installing some toolboxes, you can easily
improve your Scilab capabilities.
One of the questions we can ask is: “Why are these toolboxes not integrated in Scilab dis
tribution?”. The answer is often a problem of license. All GPL libraries can not be included in
Scilab since Scilab is not designed to become GPL.
64
Bibliography
[1] Nicholas I. M. Gould Andrew R. Conn and Philippe L. Toint. The sif reference document.
http://www.numerical.rl.ac.uk/lancelot/sif/sifhtml.html.
[2] Michael Baudin. Nelder mead user’s manual. http://wiki.scilab.org/The_NelderMead_
Component, 2009.
[3] Michael Baudin and Serge Steer. Optimization with scilab, present and future. To appear
in Proceedings Of 2009 International Workshop On OpenSource Software For Scientiﬁc
Computation (Ossc2009).
[4] Fr´ed´eric Bonnans. A variant of a projected variable metric method for bound constrained
optimization problems. Technical Report RR0242, INRIA  Rocquencourt, Octobre 1983.
[5] Joseph Fr´ed´eric Bonnans, JeanCharles Gilbert, Claude Lemar´echal, and Claudia A. Sagas
tiz´ abal. Numerical Optimization. Theoretical and Practical Aspects. Universitext. Springer
Verlag, November 2006. Nouvelle ´edition, revue et augment´eee. 490 pages.
[6] S. Boyd, L. El Ghaoui, E. Feron, and V. Balakrishnan. Linear Matrix Inequalities in System
and Control Theory, volume 15 of Studies in Applied Mathematics. SIAM, Philadelphia, PA,
June 1994.
[7] Carlos A. Coello Coello. List of references on evolutionary multiobjective optimization.
http://www.lania.mx/~ccoello/EMOObib.html.
[8] coin or.org. Optimization services instance language (osil). https://www.coinor.org/OS/
OSiL.html.
[9] Yann Collette. Personnal website. http://ycollette.free.fr.
[10] AMPL company. A modeling language for mathematical programming. http://en.
wikipedia.org/wiki/AMPL_programming_language.
[11] Goldfarb D. and Idnani A. Dual and primaldual methods for solving strictly convex
quadratic programs. Lecture Notes in Mathematics, SpringerVerlag, 909:226–239, 1982.
[12] Goldfarb D. and Idnani A. A numerically stable dual method for solving strictly convex
quadratic programs. Mathematical Programming, 27:1–33, 1982.
[13] Lawrence Davis. Genetic Algorithms and Simulated Annealing. Morgan Kaufmann Publishers
Inc., San Francisco, CA, USA, 1987.
65
[14] Kalyanmoy Deb, Samir Agrawal, Amrit Pratap, and T Meyarivan. A fast elitist non
dominated sorting genetic algorithm for multiobjective optimization: Nsgaii. pages 849–858.
Springer, 2000. http://www.lania.mx/%7Eccoello/deb00.ps.gz.
[15] Carlos M. Fonseca and Peter J. Fleming. Genetic algorithms for multiobjective optimization:
Formulationdiscussion and generalization. In Proceedings of the 5th International Conference
on Genetic Algorithms, pages 416–423, San Francisco, CA, USA, 1993. Morgan Kaufmann
Publishers Inc. http://www.lania.mx/%7Eccoello/fonseca93.ps.gz.
[16] gams.com. The general algebraic modeling system. http://www.gams.com/.
[17] David E. Goldberg. Genetic Algorithms in Search, Optimization & Machine Learning.
AddisonWesley, 1989.
[18] JeanBaptiste HiriartUrruty and Claude Lemar´echal. Convex Analysis and Minimization
Algorithms I: Fundamentals. Springer, October 1993.
[19] JeanBaptiste HiriartUrruty and Claude Lemar´echal. Convex Analysis and Minimization
Algorithms II: Advanced Theory and Bundle Methods. Springer, October 1993.
[20] hsl.rl.ac.uk. A lonesome sif decoder. http://hsl.rl.ac.uk/cuterwww/sifdec/doc.html.
[21] Stephen J. Wright Jorge Nocedal. Numerical Optimization. Springer, 1999.
[22] P.P. Khargonekar and M.A. Rotea. Mixed h2/h? control: a convex optimization approach.
Automatic Control, IEEE Transactions on, 36(7):824–837, Jul 1991.
[23] Vandenberghe L. and S. Boyd. Semideﬁnite programming. Internal Report, Stanford Uni
versity, 1994 (submitted to SIAM Review).
[24] P. J. M. Laarhoven and E. H. L. Aarts, editors. Simulated annealing: theory and applications.
Kluwer Academic Publishers, Norwell, MA, USA, 1987.
[25] P. J. M. van Laarhoven. Theoretical and Computational Aspects of Simulated Annealing.
Amsterdam, Netherlands : Centrum voor Wiskunde en Informatica, 1988.
[26] Claude Lemar´echal. A view of linesearches. Lectures Notes in Control and Information
Sciences, 30:59–78, 1981.
[27] lpsolve. Mps ﬁle format. http://lpsolve.sourceforge.net/5.5/mpsformat.htm.
[28] Zbigniew Michalewicz. Genetic Algorithms + Data Structures = Evolution Programs.
Springer, 1996.
[29] J. J. Mor´e, Burton S. Garbow, and Kenneth E. Hillstrom. Algorithm 566: Fortran subrou
tines for testing unconstrained optimization software [c5], [e4]. ACM Trans. Math. Softw.,
7(1):136–140, 1981.
[30] B. Murtagh. Advanced Linear Programming: Computation and Practice. McGrawHill, 1981.
[31] Philippe L. Toint Nicholas I.M. Gould, Dominique Orban. A constrained and unconstrained
testing environment, revisited. http://hsl.rl.ac.uk/cuterwww/.
66
[32] Philippe L. Toint Nicholas I.M. Gould, Dominique Orban. The cuter test problem set.
http://www.numerical.rl.ac.uk/cute/mastsif.html.
[33] R. Nikoukhah, A.S. Willsky, and B.C. Levy. Kalman ﬁltering and riccati equations for
descriptor systems. Automatic Control, IEEE Transactions on, 37(9):1325–1342, Sep 1992.
[34] Jorge Nocedal. Updating quasinewton matrices with limited storage. Mathematics of Com
putation, 35(151):773–782, 1980.
[35] optimizationservices.org. Optimization services. http://www.optimizationservices.org/.
[36] Jun Ma Robert Fourer and Kipp Martin. Osil: An instance language for optimization.
Computational Optimization and Applications, 2008.
[37] N. Srinivas and Kalyanmoy Deb. Multiobjective optimization using nondominated sorting
in genetic algorithms. Evolutionary Computation, 2:221–248, 1994. http://www.lania.mx/
%7Eccoello/deb95.ps.gz.
[38] Berwin A Turlach. Quadprog, (quadratic programming routines). http://www.maths.uwa.
edu.au/~berwin/software/quadprog.html.
[39] Wikipedia. Ampl (programming language). http://en.wikipedia.org/wiki/AMPL_
programming_language.
[40] Wikipedia. General algebraic modeling system. http://en.wikipedia.org/wiki/General_
Algebraic_Modeling_System.
[41] Wikipedia. Mps (format). http://en.wikipedia.org/wiki/MPS_(format).
[42] Wikipedia. Simulated annealing. http://en.wikipedia.org/wiki/Simulated_annealing.
[43] Patrick Siarry Yann Collette. Optimisation Multiobjectif. Eyrolles, Collection Algorithmes,
1999.
67
This document has been written by Michaël Baudin and Vincent Couvert from the Scilab Consortium and by Serge Steer from INRIA Paris  Rocquencourt. © July 2010 The Scilab Consortium – Digiteo / INRIA. All rights reserved.
Abstract In this document, we make an overview of optimization features in Scilab. The goal of this document is to present all existing and nonexisting features, such that a user who wants to solve a particular optimization problem can know what to look for. In the introduction, we analyse a classiﬁcation of optimization problems. In the ﬁrst chapter, we analyse the ﬂagship of Scilab in terms of nonlinear optimization: the optim function. We analyse its features, the management of the cost function, the linear algebra and the management of the memory. Then we consider the algorithms which are used behind optim, depending on the type of algorithm and the constraints. In the remaining chapters, we present the algorithms available to solve quadratic problems, nonlinear least squares problems, semideﬁnite programming, genetic algorithms, simulated annealing and linear matrix inequalities. A chapter focus on optimization data ﬁles managed by Scilab, especially MPS and SIF ﬁles. Some optimization features are available in the form of toolboxes, the most important of which are the Quapro and CUTEr toolboxes. The ﬁnal chapter is devoted to missing optimization features in Scilab.
. 1. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1. . . . . . . . . . . . . . . . . . . . . . . . . . .3 Optimization routines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7. . . . . . . . . . . . . . . . .2 Scilab function . . . . .5 Linear algebra .1 Management of the approximated Hessian matrix 1. . . . . . . . . . . 1. .7. . .3 Optimization routines . . . . . . . . . . . . . . . . . . . . . . . . . . . .1 Mathematical point of view . . . . 1. . . . . . . . . . . 2 Quadratic optimization 2. . . . . . . . . . . . . . . . . . 1. . . . . . . . . . . . . . . . . . . . . . 4 Semideﬁnite programming 4. . . .6 Management of memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1. . . .4 Termination criteria . . . .5 Internal design . . . . . . . . . . . . . . . .4 Memory requirements . . . . . . . . . 2. . . . . . . . . 2. . . . . . . . . . . . . . . . . . . . . .2 Line search . . . . . . . . . . . . . . . . . . . .2 qpsolve . . . . 4. . . . . . . . . . . . . . . . . . . . 3. . . . . . . . .2 Scilab function . . . . . . 3. . . . .7. . . 3 Nonlinear least square 3. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3 qp solve . . . . . . . . .7 QuasiNewton ”qn” without constraints : n1qn1 . . . . . . .1 Mathematical point of view . . . . . . . . .4 The cost function . . . .1 Mathematical point of view 2. . . . . . . . . . . . . . . . 4. . . . 1. . . . . . . . . . . .8 QuasiNewton ”qn” with bounds constraints : qnbd . . . . .11 Non smooth method without constraints : n1fc1 . . . . . 1. . . . . . . . . . 2. . . . . . . . . . . . . . . . . . .10 LBFGS ”gc” with bounds constraints : gcbd . . . . . . . . .9 LBFGS ”gc” without constraints : n1qn3 . . . . . . . 1. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3 Optimization routines . . . . . . . . 1. . . . . . . . . . . . . . 5 10 10 11 11 11 12 12 13 15 16 16 17 18 20 20 21 22 23 23 23 24 24 24 26 26 26 26 27 27 27 27 . . . . . . . 1. . . . . . . . . . . . . . .3 Initial Hessian matrix . . . . . . . . . . . . . . . . .2 Optimization features . . . 1. . . . . . . . . . . . . . . . . . . 1. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1. . . . . . . . . . . . . . . . . . . . . .5 An example . . . . . . . . . . .Contents Introduction 1 Nonlinear optimization 1. . .7. . . . . . . . . .1 Mathematical point of view . . . .7. . . . . . . . . . . . . . . . . . . . . 1 . . . . . . . . . . . . . .
2. . . . . . . . . . . . . . . . . 5. . . . . . . . . . . 7. . . . . . . . . . . .1 Introduction . . . . .1 Introduction . . . . . . . . . . . . . . . . . .3 optim nsga . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7. . . . . 7. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1 Syntax . . . . . . . .2 SIF ﬁles and the CUTEr toolbox . . . . . . . . .2 Interactive mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Scilab Optimization Toolboxes 9. 6. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2 Overview . . . . . . .5 Other versions . . . . . . . . . . . . . . . . . . . . .3. . . . . . . . . . . 6. . . . 5. . . . . . . . . . .4 How lmisolver works . . . . . . . . . . . . . . . . . . . .1. . . . . . . . . . . . . . 5. . . . . . . . . . . . . . . . . . . . . . . . . .4 Other toolboxes . . . 5. . .2 Example . . . . . . . 7. . . . . . . . . . . . . . . . . . . 5. . . . . . . . . . . . . 5.1 Quapro . . . . . . . . . .4 Solvers . . . . . . . . . . 9. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9. . . . . . . .1 Linear optimization . . . . . . . . . . . . . . . .2. .2 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2 Crossover . . . . . . . . . . . . . . . . pareto 5. . . . . . . . . . ﬁlter . . . . . . . . . . . . . .4 Initialization . . . . . . 9. . . . . . . .1 optim ga . . . .3 Example . . . . . . . . . . . . . . . . 8. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1 MPS ﬁles and the Quapro toolbox . . . . . . .3. . .3 Selection . . . . . . .4 optim nsga2 . .4 Neighbor functions . . . . . . . .5 Genetic algorithms 5. . . . . . . . . . . . . . . . . . . . .1 Coding . . . . . .2 Function lmisolver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1 Noninteractive mode . . . . . . . . .4. 7. . . . . . . . . . . . . . . . . . . . . .2 optim moga. . .4. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Simulated Annealing 6. . . .3. . . . . 7. . . . . . . . 7. . . . .3 Support functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3. . . . . . . . . . . . . . . . . . . . . . . . .5 Acceptance functions 6. . . . . . 7 LMITOOL: a Package for LMI Optimization in 7. . . . . . . . 5. . . . . . . . . . . . . . . . . . . . . . . .7 optim sa . . . .4. . . . . . . . . . . . .3. . . . . . . . . . . . . . . . Scilab . . . . 5. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3 Function LMITOOL . . . . 6. . . . . . . . . 5. . . . . .2 Linear quadratic optimization . . .1. . . . . . . . 8 Optimization data ﬁles 8. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1 Purpose . . . . . . . . . . . . . . . . . . . 6. . . . . . . . . . . . . . .3. . . . . . . . . .2 CUTEr . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 Missing optimization features in Scilab 2 . . . . . . . . . . . . . . . . . .4. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6 Temperature laws . . . . . . . . . . . . . 28 28 29 30 32 32 32 32 32 33 33 34 34 35 35 35 36 37 38 39 39 41 41 42 42 43 49 49 50 52 53 56 56 56 57 57 57 58 59 60 60 63 . . . . . . . . . . . . . . . . . . . . . . . . . . . 7. . . . . . . . . . . . . . . 9. . . . . . . . . . . 6. . . . .3 The Unconstrained Optimization Problem Toolbox 9. . . . . . . . . . . . . . . . . . . .
Conclusion Bibliography 64 65 3 .
Vincent Couvert Copyright c 20082009 .Serge Steer This ﬁle must be used under the terms of the Creative Commons AttributionShareAlike 3.Digiteo .Copyright c 20082010 .org/licenses/bysa/3.0 4 .Consortium Scilab .0 Unported License: http://creativecommons.INRIA .Digiteo .Consortium Scilab .Michael Baudin Copyright c 20082009 .
• There may be constraints or no constraints. Above all the embedded functionalities of Scilab itself. • The constraints may be bounds constraints. we consider optimization problems in which we try to minimize a cost function f (x) with or without constraints. Many of these toolboxes are interfaces to optimization libraries. • There may be one or several cost functions (multiobjective optimization).Introduction This document aims at giving Scilab users a complete overview of optimization features in Scilab. quadratic or a general non linear function. such as FSQP for example.php). • nonlinear leastsquare optimization with the lsqrsolve function. • quadratic optimization with the qpsolve function.fr/tikiindex. medium (from 10 to 100 .1 000) or large (from 1 000 . we give a short list of new features which would be interesting to ﬁnd in Scilab. An overview of Scilab optimization tools is showed in ﬁgure 2. linear or nonlinear constraints. we present the following optimization features of Scilab.10 000 and above). It is written for Scilab partners needs in OMD project (http://omd.lri. leading to dense or sparse linear systems. 5 . Several properties of the problem to solve may be taken into account by the numerical algorithms : • The unknown may be a vector of real values or integer values. • The number of unknowns may be small (from 1 to 10 . In this document. • linear matrix inequalities with the lmisolver function. The core of this document is an analysis of current Scilab optimization features.100). In this document. • semideﬁnite programming with the semidef function. some contributions (toolboxes) have been written to improve Scilab capabilities. • simulated annealing with the optim_sa function. In the ﬁnal part. • genetic algorithms with the optim_ga function. These problems are partly illustrated in ﬁgure 1. • The cost function can be linear. • The cost function may be smooth or nonsmooth. • nonlinear optimization with the optim function.
Discrete Parameters Several Objectives Non Smooth Without Constraints Bounds Constraints Figure 1: Classes of optimization problems 6 Continuous Parameters One Objective Smooth With Constraints Optimization Linear Constraints Linear Objective Quadratic Objective Nonlinear Constraints Nonlinear Objective 110 Unknowns 10100 Unknowns >100 Unknowns .
Figure 2: Scilab Optimization Tools 7 .
8 . This algorithm is very eﬃcient but is not free (but is provided by the authors. in fact. In the second chapter we present the qpsolve and qp_solve functions which allows to solve quadratic problems.e.• reading of MPS and SIF ﬁles with the quapro and CUTEr toolboxes.” Several existing optimization features are not presented in this document. Scilab v5. we analyse the ﬂagship of Scilab in terms of nonlinear optimization: the optim function. This algorithm is known to be able to manage ”noisy” functions. up to 10 parameters and its memory requirement is only O(n). This function allows to solve nonlinear optimization problems without constraints or with bound constraints. which a derivativefree algorithm for small problems. • The fsqp toolbox provides an interface for a Sequential Quadratic Programming algorithm. which indicates that there is no cureforall solvers. i. including performance tests. For each solver. a Limited Memory BFGS algorithm and a bundle method for nonsmooth functions. The fminsearch function is. i. free of charge. It provides a QuasiNewton method. The chapter 5 focuses on genetic algorithms. An analysis of optimization in Scilab. is presented in ”Optimization with Scilab. a specialized use of the neldermead component.2 provides the fminsearch function. We also analyse the support functions which allow to conﬁgure the behavior of the algorithm and describe the algorithm which is used. The chapter 3 and 4 brieﬂy present nonlinear least squares problems and semideﬁnite programming. The aim of this paper is to give a simple but accurate view of what problems can be solved by Scilab and what behavior can be expected for those solvers. • Multiobjective optimization is available in Scilab with the genetic algorithm component. In order to compare the respective performances of the algorithms. Then we consider the algorithms which are used behind optim. We describe the solvers which are used. It provides an object oriented access to the options. It is eﬃcient for small problems. that is. which is available in Scilab. The organization of this document is as following. The neldermead component provides three simplexbased algorithms which allow to solve unconstrained and nonlinearly constrained optimization problems. situations where the cost function is the sum of a general nonlinear function and a low magnitude function. We especially mention the following tools. We focus on the user’s point of view. we have to minimize or maximize an objective function and must ﬁnd a solver suitable for the problem. the linear algebra and the management of the memory.e. The following is the abstract of the paper : ”We present in this paper an overview of optimization algorithms available in theScilab software. Numerical experiments are presented. We give a tutorial example of the optim_ga function in the case of the Rastrigin function. we analyze the type of problems that it can solve as well as its advantages and limitations. the memory requirements and the internal design of the tool. depending on the type of algorithm and the constraints. present and future” [3]. We analyse its features. we use the CUTEr library. the management of the cost function. for an academic use). This unconstrained algorithm does not require the gradient of the cost function. The fminsearch function is based on the simplex algorithm of Nelder and Mead (not to be confused with Dantzig’s simplex for linear optimization). This component is presented in in depth in [2]. In the ﬁrst chapter.
The LMITOOL module is presented in chapter 7. 9 . which gives an overview of the algorithm used in optim_sa. the acceptance functions and the temperature laws. The chapter 8 focuses on optimization data ﬁles managed by Scilab. Delebecque and Ghaoui. The syntax of the lmisolver function is analysed and several examples are analysed in depth. to FSQP. we analyse the structure of the algorithm used in optim_sa. Some optimization features are available in the form of toolboxes. In the ﬁnal section. The chapter 10 is devoted to missing optimization features in Scilab. the most important of which are the Quapro. especially MPS and SIF ﬁles.The simulated annealing is presented in chapter 6. to LIPSOL. This chapter was written by Nikoukhah. to NEWUOA. Then we analyse the support functions and present the neighbor functions. CUTEr and the Unconstrained Optimization Problems toolboxes. These modules are presented in the chapter 9. This tool allows to solve linear matrix inequalities. along with other modules including the interface to CONMIN. to LPSOLVE. We present an example of use of this method and shows the convergence of the algorithm.
The linear algebra components are analysed. and a NonDiﬀerentiable solver. In this chapter we analyse each solver and present the following features : • the reference articles or reports.1 Mathematical point of view min f (x) x The problem of the non linear optimization is to ﬁnd the solution of with bounds constraints or without constraints and with f : Rn → R the cost function. a C function or a Fortran 77 function. We analyse in detail the management of the cost function. especially the algorithm name and if dense/sparse cases are taken into account. • the management of memory. In this chapter. since they are used at many places in the algorithms. 10 . the current behaviour of Scilab with respect to memory is detailed here. we describe both the internal design of the optim primitive. • the line search method. a BFGS QuasiNewton solver. • the author. The cost function and its gradient can be computed using a Scilab function. Three nonlinear solvers are connected to the optim primitive. The optim primitive allows to optimize a problem with a nonlinear objective without constraints or with bound constraints. The Scilab online help is a good entry point for this function. namely. a LBFGS solver. 1.Chapter 1 Nonlinear optimization The goal of this chapter is to present the current features of the optim primitive Scilab. • the linear algebra system. Since the management of memory is a crucial feature of optimization solvers.
In the most simple usecase. • ”qn” with bounds constraints : QuasiNewton method with bounds constraints. • Bundle method for non smooth functions (half derivable functions. i n d ]= c o s t f ( x . The library which is used by optim was created by the Modulopt project at INRIA and developped by Bonnans. qnbd. that is. with the following header : [ f . which is managed with the fortran 77 callback system. • ”gc” without constraints : a Limited memory BGFS methoud without constraints. ”g” or both are to be computed. n1fc1. g . n1qn1. the cost function must be passed as a callback argument to the ”optim” primitive. Problems involving non linear constraints cannot be solved with the current optimization methods implemented in Scilab. 1. i n d ) where ”x” is the current value of the unknown and ”ind” is the integer ﬂag which states if ”f”. 1. • QuasiNewton with limited memory BFGS (LBGFS) without constraints or with bound constraints. n1qn3. depending on the input integer ﬂag ”ind”.3 Optimization routines Nonlinear optimization in Scilab is based on a subset of the Modulopt library. nondiﬀerentiable problem) without constraints. the name of the routine to call back is declared as ”external” in the source code. gcbd. and the corresponding fortran routine : • ”qn” without constraints : a QuasiNewton method without constraints. developed at Inria. Gilbert and Lemar´chal [5]. In that case.4 The cost function The communication protocol used by optim is direct. e This section lists the routines used according to the optimization method used. the cost function is a Scilab function. Non smooth problems with bounds constraints cannot be solved with the methods currently implemented in Scilab. The cost function may be provided in the following ways : 11 . The following is the list of solvers currently available in Scilab. The cost function is passed to the optimization solver as a callback. • ”nd” without constraints : a Non smooth method without constraints. • ”gc” with bounds constraints : a Limited memory BGFS with bounds constraints.1.2 Optimization features Scilab oﬀers three nonlinear optimization methods: • QuasiNewton method with BFGS formula without constraints or with bound constraints. The cost function must compute the objective and/or the gradient of the objective.
but most importantly. 1. which are managed at the C level in optimtable. This applies to all optimization methods. In the case where the cost function is a Scilab script. it calls the ”foptim” C void function which in returns calls the compiled cost function associated to the conﬁgured address (*foptimfonc). the linear algebra is inlined and there is no use the BLAS API. can be used to provide a compiled cost function. calls the script. the computation is based on function pointers. 1. the descent direction. The optimization function is conﬁgured by the ”setfoptim” C service. Indeed. with a ”if” statement : • if the cost function is compiled. where n is the size of the unknown.c. If the full approximated Hessian is stored. such as ilib for link for example. When 12 . the amount of memory is the square of the dimension of the problem O(n2 ). Most of the memory is required by the approximation of the Hessian matrix. • if not. Scilab dynamic link features. When the optimization solver needs to compute the cost function. which maps function names to function pointer. be it in statically compiled libraries or in dynamically compiled libraries. There is only one exception : the LBFGS with bounds constraints routine gcbd uses the ”dcopy” routine of the BLAS API.6 Management of memory The optimization solvers requires memory. as in the BFGS quasiNewton method. which takes as argument the name of the routine to callback. the ”boptim” routine performs the copy of the input local arguments into Scilab internal memory. Generally. • the cost function is provided as a C or fortran 77 compiled routine. The services implemented in AddFunctionInTable. In the following paragraph. This allows the optimization solvers to callback dynamically linked libraries. especially to store the value of the cost function. In the case where the cost function is compiled. The static ﬁeld foptimfonc with type foptimf is then set to the address of the function to be called back. which takes the name of the function as input argument and searches the corresponding function address. These names and addresses are stored in the hashmap FTab foptim. in the sci f optim routine. except ”gcbd”. This limits the performance of the optimization. This switch is managed at the gateway level. the ”boptim” symbol is passed. especially the function ”AddFunctionInTable”. the cost function can be statically or dynamically linked against Scilab. If the cost function is a C or fortran 77 source code. and ﬁnally copy back the output argument from Scilab internal memory into local output variables.c are used. the ”foptim” symbol is passed. we analyse the very internal aspects of the management of the cost function.• the cost function is provided as a Scilab script.5 Linear algebra The linear algebra which is used in the ”optim” primitive is dense. the approximated Hessian of the cost function. because optimized libraries like ATLAS cannot not used. the gradient.
inside the stack and the storage area is passed to the solvers as an input argument.7 QuasiNewton ”qn” without constraints : n1qn1 The author is C. 1. The memory system used by the fortran solvers is the fortran 77 ”assumedsizedummyarrays” mechanism. The following is the header for the n1qn1 routine : 13 . There is no reference report for this solver. This explains why the name ”cg” was chosen: it refers to the Conjugate Gradient method. • QuasiNewton ”qn” with bounds constraints (qnbd) : O(n2 ). Lemarechal. 9). based on ”real arrayname(*)” statements. Simplifying these array sizes leads to the following map. as the number of required double precision values. • Limited Memory BFGS ”gc” with bounds constraints (gcbd) : O(n). where n is from 100 to 1000. This large storage memory is then split into pieces like a piece of cake by each solver to store the data. • Limited Memory BFGS ”gc” with bounds constraints (gcbd) : n(5 + 3nt) + 2nt + 1 with nt = max(1. 1987. One main feature of one of the LBFGS algorithms is to limit the memory required. • Limited Memory BFGS ”gc” without constraints (n1qn3) : 4n + m(2n + 1). More precisely. • Non smooth method without constraints (n1fc1) : O(n). • QuasiNewton BFGS ”qn” with bounds constraints (qnbd) : n(n + 1)/2 + 4n + 1. which stores only one vector in memory. This memory is allocated by Scilab. m/3). Note that n1fc1 requires an additionnal 2(m + 1) array of integers. which clearly shows why Limited Memory BFGS algorithms in Scilab are more suitable for large problems. • QuasiNewton ”qn” without constraints (n1qn1) : O(n2 ). the following is a map from the algorithm to the memory required. That explains why LBFGS methods associated with the ”gc” option of the optim primitive are recommended for largescale optimization. The management of memory is very important for largescale problems. only a given number m of vectors of size n are stored. chap. • Limited Memory BFGS ”gc” without constraints (n1qn3) : O(n). It is known that LBFGS convergence may be slow for largescale problems (see [21]. • Non smooth method without constraints (n1fc1) : (n + 4)m/2 + (m + 9) ∗ m + 8 + 5n/2.a quasiNewton method with limited memory is used. • QuasiNewton BFGS ”qn” without constraints (n1qn1) : n(n + 13)/2. But the name ”cg” is wrongly chosen and this is why we consistently use LBFGS to identify this algorithm.
Le programme considere que la convergence est obtenue lorque il lui est impossible de diminuer f en attribuant a au ` moins une coordonn´e x(i) une variation superieure a eps*var(i). inria. en entree le point initial . 1 mode.g. f.x. En sortie. en sortie valeur de f en x (ﬁnal).eps. 1987 c Copyright INRIA c!methode c direction de descente calculee par une methode de quasinewton c recherche lineaire de type wolfe The following is a description of the arguments of this routine. n1qn1 appelle toujours simul avec indic = 4 .f. eps contient le e carr´ de la norme du gradient en x (ﬁnal).imp. en sortie valeur du gradient en x (ﬁnal).izs.nsim. le module de simulation doit se presenter sous la forme subroutine simul(n.lp. lemarechal.une bonne valeur est 10% de la diﬀerence (en valeur absolue) avec la coordonee x(i) optimale • eps (es) : en entree scalaire deﬁnit la precision du test d’arret. • f (es) : scalaire . • var (e) : vecteur strictement positif de dimension n. izs. rzs.subroutine n1qn1 (simul.x.zm.niter.n.var. • imp (e) : contrˆle les messages d’impression : o 14 . en sortie le nombre de tels appels reellement faits.dzs) c!but c minimisation d une fonction reguliere sans contraintes c!origine c c. amplitude de la modif souhaitee a la premiere iteration sur x(i).rzs. en entree valeur de f en x (initial). • x (es) : vecteur de dimension n . e • mode (e) : deﬁnit l’approximation initiale du hessien – =1 n1qn1 l’initialise luimeme – =2 le hessien est fourni dans zm sous forme compressee (zm contient les colonnes de la partie inferieure du hessien) • niter (es) : en entree nombre maximal d’iterations : en sortie nombre d’iterations reellement eﬀectuees. g. • n (e) : nombre de variables dont depend f. dzs) et ˆtre declare en external dans le programme e appelant n1qn1. en sortie : le point ﬁnal calcule par n1qn1. • nsim (es) : en entree nombre maximal d’appels a simul (c’est a dire avec indic = 4). • g (es) : vecteur de dimension n : en entree valeur du gradient en x (initial). • simul : point d’entree au module de simulation (cf normes modulopt i).
– = 0 rien n’est imprime – = 1 impressions initiales et ﬁnales – = 2 une impression par iteration (nombre d’iterations, nombre d’appels a simul, valeur courante de f). – >=3 informations supplementaires sur les recherches lineaires ; tres utile pour detecter les erreurs dans le gradient. • lp (e) : le numero du canal de sortie, i.e. les impressions commandees par imp sont faites par write (lp, format). • zm : memoire de travail pour n1qn1 de dimension n*(n+13)/2. • izs,rzs,dzs memoires reservees au simulateur (cf doc) The n1qn1 solver is an interface to the n1qn1a routine, which really implements the optimization method. The n1qn1a ﬁle counts approximately 300 lines. The n1qn1a routine is based on the following routines : • simul : computes the cost function, • majour : probably (there is no comment) an update of the BFGS matrix. Many algorithms are inlined, especially the line search and the linear algebra.
1.7.1
Management of the approximated Hessian matrix
The current BFGS implementation is based on a approximation of the Hessian [21], which is based on Cholesky decomposition, i.e. the approximated Hessian matrix is decomposed as G = LDLT , where D is a diagonal n × n matrix and L is a lower triangular n × n matrix with unit diagonal. To compute the descent direction, the linear system Gd = LDLT d = −g is solved. The memory requirements for this method is O(n2 ) because the approximated Hessian matrix computed from the BFGS formula is stored in compressed form so that only the lower part of the approximated Hessian matrix is stored. This is why this method is not recommended for largescale problems (see [21], chap.9, introduction). The approximated hessian H ∈ Rn×n is stored as the vector h ∈ Rnh which has size nh = n(n + 1)/2. The matrix is stored in factored form as following h = (D11 L21 . . . Ln1 H21 D22 . . . Ln2  . . . Dn−1n−1 Lnn−1 Dnn ) . (1.1)
Instead of a direct acces to the factors of D and L, integers algebra is necessary to access to the data stored in the vector h. The algorithm presented in ﬁgure 1.1 is used to set the diagonal terms the diagonal terms of D, the diagonal matrix of the Cholesky decomposition of the approximated Hessian. The righthand side 0.01c2max of this initialization is analysed in the next section of this document. v
i
15
k←1 for i = 1 to n do h(k) = 0.01c2max vi k ←k+n+1−i end for Figure 1.1: Loop over the diagonal terms of the Cholesky decomposition of the approximated Hessian Solving the linear system of equations The linear system of equations Gd = LDLT d = −g must be solved to computed the descent direction d ∈ Rn . This direction is computed by the following algorithm • compute w so that Lw = −g, • computed d so that DLT d = w. This algorithm requires O(n2 ) operations.
1.7.2
Line search
The line search is based on the algorithms developped by Lemar´chal [26]. It uses a cubic intere polation. The Armijo condition for suﬃcient decrease is used in the following form
T f (xk + αpk ) − f (xk ) ≤ c1 α fk pk
(1.2)
with c1 = 0.1. The following fortran source code illustrates this condition if (fbfa.le.0.10d+0*c*dga) go to 280 where f b = f (xk + αpk ), f a = f (xk ), c = α and dga =
T fk pk is the local directional derivative.
1.7.3
Initial Hessian matrix
Several modes are available to compute the initial Hessian matrix, depending on the value of the mode variable • if mode = 1, n1qn1 initializes the matrix by itself, • if mode = 2, the hessian is provided in compressed form, where only the lower part of the symetric hessian matrix is stored. An additionnal mode=3 is provided but the feature is not clearly documented. In Scilab, the n1qn1 routine is called with mode = 1 by default. In the case where a hotrestart is performed, the mode = 3 is enabled. If mode = 1 is chosen, the initial Hessian matrix H 0 is computed by scaling the identity matrix H 0 = Iδ 16 (1.3)
where δ ∈ Rn is a nvector and I is the n × n identity matrix. The scaling vector δ ∈ Rn is based on the gradient at the initial guess g 0 = g(x0 ) = f (x0 ) ∈ Rn and a scaling vector v ∈ Rn , given by the user δi = where cmax > 0 is computed by
0 cmax = max 1.0, max(gi )) i=1,n
0.01cmax 2 vi
(1.4)
(1.5)
In the Scilab interface for optim, the scaling vector is set to 0.1 : vi = 0.1, i = 1, n. (1.6)
1.7.4
Termination criteria
The following list of parameters are taken into account by the solver • niter, the maximum number of iterations (default value is 100), • nap, the maximum number of function evaluations (default value is 100), • epsg, the minimum length of the search direction (default value is %eps ≈ 2.22e − 16). The other parameters epsf and epsx are not used. The termination condition is not based on the gradient, as the name epsg would indicate. The following is a list of termination conditions which are taken into account in the source code. • The iteration is greater than the maximum. if (itr.gt.niter) go to 250 • The number of function evaluations is greater than the maximum. if (nfun.ge.nsim) go to 250 • The directionnal derivative is positive, so that the direction d is not a descent direction for f. if (dga.ge.0.0d+0) go to 240 • The cost function set the indic ﬂag (the ind parameter) to 0, indicating that the optimization must terminate. call simul (indic,n,xb,fb,gb,izs,rzs,dzs) [...] go to 250 17
pxp−1 )T 1 2 (1.dzs) [.0. The step is reduced by a factor 10.5 An example The following script illustrates that the gradient may be very slow. The cost function is the following f (x) = xp + xp 1 2 where p ≥ 0 is an even integer.9) f (x) = (pxp−1 .steplb) goto 170 [.. if (ir.eq..steplb) go to 270 • During the line search. The problem has two parameters so that n = 2. a cubic interpolation is computed and the computed minimum is associated with a zero step length.gt. The gradient of the function is g(x) = and the Hessian matrix is H(x) = p(p − 1)xp−2 0 1 0 p(p − 1)xp−2 2 18 (1.izs. but on the length of the step.0d+0 [. call simul (indic.n.0.] go to 250 • The Armijo condition is not satisﬁed and step size is below a limit during the line search. the step length is lesser than a computed limit.gt..0d+0) goto 250 • During the line search.xb.• The cost function set the indic ﬂag to a negative value indicating that the function cannot be evaluated for the given x..] if (stepbd...fb.8) (1. Here we choose p = 10..lt.le.rzs.7.] step=step/10.. This shows that the termination criteria is not based on the gradient.le. if (stmin+step.10d+0*c*dga) go to 280 [. but the algorithm continues.steplb) go to 240 • The rank of the approximated Hessian matrix is lesser than n after the update of the Cholesky factors.gb.] if (step. if(c.n) go to 250 1. but gets below a limit so that the algorithm terminates.7) . if (fbfa.
. f =6.  g=%e\n” . 4 0 9 8 9 8 e −019  x  = 9 . [ f . 3 4 0 8 6 4 e −001 . g ] = myquadratic ( x0 . f =3. xopt . i t e r . nap . x0 ’ ) ) nap = 100 i t e r = 100 e p s g = %eps [ f o p t . 0 1 3 2 2 7 e +000 . 8 1 7 1 2 6 e −018  x  = 1 .norm( x ) .  g  = 6 . 4 ) .  g  = 3 . g . ”ar ” . mprintf ( ”Expected g(x0) = \n” ) .2 1 . 0 0 2 2 3 7 e −002 . 5 6 2 0 5 0 e +000 .279551 e −021 . 0 7 4 4 8 5 e −019 Norm o f p r o j e c t e d g r a d i e n t l o w e r than 0 . nap . 2 8 0 5 6 4 e −002 . f =7. g r a d o p t ] = optim ( myquadratic . 2 3 6 6 9 4 e −003 . .432396 e −021 . imp = −1)  x  = 1 . disp ( g ’ ) . 0 8 7 5 2 6 e −002 . checks that the derivatives are correctly computed and performs an optimization.092124 e +000 . 0 8 2 5 4 2 e +001  x  = 9 . . e p s g .  g  = 2 .  g  = 1 . 0)T . 2 5 5 7 9 0 e +001  x  = 1 .833319 e −022 . 1 8 2 5 9 2 e +000 [.  g  = 5 . f ) . imp = −1) The script produces the following output. −−>[ f o p t . disp ( derivative ( q u a d f o r n u m d i f f . the norm of the gradient of the cost function is displayed so that one can see if the algorithm terminates when the gradient is small. i t e r . g r a d o p t ] = optim ( myquadratic . f =1.409611 e −022 . f =3. f =7. 19 . f= %e. i n d ) p = 10 i f i n d == 1  i n d == 2  i n d == 4 then f = x(1)ˆp + x(2)ˆp . .  g  = 5 .  g  = 5 . 0 9 8 3 6 7 e +000 . .]  x  = 1 . f =4. x0 . 4 7 3 6 4 0 e +000 . ”ar ” ..  g  = 1 .817592 e −001 . xopt .458198 e +000 . 2 4 6 7 5 2 e +001  x  = 1 .415994 e +000 . function [ f . 2 ) endfunction x0 = [ −1.10) The following Scilab script deﬁnes the cost function. 3 3 6 8 0 2 e −018  x  = 1 . .  g  = 2 . f =1. 7 8 5 6 6 3 e −018  x  = 1 . x0 . norm( g ) ) end endfunction function f = q u a d f o r n u m d i f f ( x ) f = myquadratic ( x . At each iteration.. mprintf ( ”Computed f (x0) = %f\n” . 0 ] . 3 0 7 4 4 8 5D−18. f . mprintf ( ”Computed g(x0) = \n” ) .The optimum of this optimization problem is at x = (0. (1. end i f i n d == 1  i n d == 2  i n d == 4 then g ( 1 ) = p ∗ x ( 1 ) ˆ ( p−1) g ( 2 ) = p ∗ x ( 2 ) ˆ ( p−1) end i f i n d == 1 then mprintf ( ” x=%e. 1 7 9 9 6 6 e −002 . f =2.191736 e +000 . 5 0 2 5 9 9 e +001  x  = 1 . i n d ] = myquadratic ( x .  g  = 2 . e p s g . f =2.450507 e −021 .
0065865 fopt = 2 . • simul : computes the cost function The rlbd routine is documented as using an extrapolation method to computed a range for the optimal t parameter. It is inlined. 1988. The BFGS update is based on the article [34]. 1. Bonnans [4]. • ajour : probably (there is no comment) an update of the BFGS matrix. Because this is not the case. The range is then reduced depending on the situation by : • a dichotomy method. the algorithm would stop in the early iterations. Claude Lemarechal. chap. • rlbd : line search method with bound constraints. • a cubic interpolation. If the termination criteria was based on the gradient. The architecture is clear and the source code is well commented.8 QuasiNewton ”qn” with bounds constraints : qnbd The comments state that the reference report is an INRIA report by F. but the solution is not accurate only up to the 3d digit. so that connecting the BLAS may be diﬃcult.9 LBFGS ”gc” without constraints : n1qn3 The comments in this solver are clearly written.g r a d o pt = 1 . The stopping criteria is commented as ”an extension of the Wolfe criteria”. The zqnbd routine is based on the following routines : • calmaj : calls majour. 1. The n1qn3a routine is based on the following routines : 20 .2002412 0. The memory requirements for this method are O(n2 ). The solver qnbd is an interface to the zqnbd routine. 0D−18 ∗ 0. • proj : projects the current iterate into the bounds constraints. The authors are Jean Charles Gilbert. The linear algebra does not use the BLAS API. introduction). • a linear interpolation. The solver n1qn3 is an interface to the n1qn3a routine. which shows why this method is not recommended for largescale problems (see [21].0064757 One can see that the algorithm terminates when the gradient is extremely small g(x) ≈ 10−18 . The cost function is very near zero f (x) ≈ 10−22 .2332982 xopt = 0. 8 3 3D−22 0. the algorithm performs signiﬁcant iterations which are associated with relatively large steps. This is a very diﬃcult test case for optimization solvers. The diﬃculty is because the function is extremely ﬂat near the optimum. which updates the BFGS matrix.9.
The linear algebra is dense. chap. The linear algebra is dense. extrapolation and cubic interpolation. • nlis0 : line search based on Wolfe criteria. so that the memory cost is O(n). The zgcbd routine is based on the following routines : • simul : computes the cost function • proj : projects the current iterate into the bounds constraints. • relvar : computes the variables to relax by Bertsekas method. although convergence may be slow (see [21]. ys. • shanph : scalse the diagonal by ShannoPhua method. • bfgsd : updates the diagonal by Powell diagonal BFGS. Bonnans.• prosca : performs a vector x vector scalar product. but without optimizaion. connected to the fuclid routine. 9. so that the memory cost is O(n).227). s = x(k + 1) − x(k). 9. The algorithm is a limited memory BFGS method with m levels. 1. 1985. But zgcbd uses the ”dcopy” BLAS routine. The algorithm is a limited memory BFGS method with m levels.227). chap.10 LBFGS ”gc” with bounds constraints : gcbd The author is F. • simul : computes the cost function. This implements the scalar product. although the convergence may be slow (see [21]. The gcbd solver is an interface to the zgcbd routine. The prosca routine is a call back input argument of the n1qn3 routine. • ddd2 : computed the descent direction by performing the product hxg. The linear algebra does not use the BLAS API but is based on the prosca and ctonb routines. • gcp : gradient conjugate method for Ax = b. • majz : updates z. It is well suited for mediumscale problems. p. which limits the feature to small size optimization problems. • dcopy : performs a vector copy (BLAS API). • rlbd : line search method with bound constraints. which really implements the optimization method. There is no reference report for gcbd. p. It is well suited for mediumscale problems. which allows for some optimizations. • majysa : updates the vectors y = g(k + 1) − g(k). Connecting BLAS may be easy for n1qn3. • ctonb : copies array u into v.zs. 21 .
who is an expert of this topic. 22 . 19]. References for this e algorithm include the ”Part II.1. and the indepth presentation in [18. • prosca : performs a vector x vector scalar product. which really implements the optimization method.11 Non smooth method without constraints : n1fc1 This routine is probably due to Lemar´chal. The n1fc1 solver is an interface to the n1fc1a routine. the objective function is the maximum of several continously diﬀerentiable functions). The n1fc1a routine is based on the following routines : • simul : computes the cost function. Nonsmooth optimization” in [5]. • fprf2 : computes the direction. It is designed for functions which have a noncontinuous derivative (e.g. • frdf1 : reduction du faisceau • nlis2 : line search.
which are meant to be a replacement for the former Quapro solver (which has be transformed into a Scilab toolbox). we presents these two primitives. We especially analyse the management of the linear algebra as well as the memory requirements of these solvers.1) (2.Chapter 2 Quadratic optimization Quadratic problems can be solved with the qpsolve Scilab macro and the qp_solve Scilab primitive.2 qpsolve The Scilab function qpsolve is a solver for quadratic problems when Q is symmetric positive deﬁnite. For more details about this solver. This routine uses the Goldfarb/Idnani algorithm [11. The solver is implemented in Fortran 77. 2. The qpsolve macro calls the qp_solve compiled primitive. The internal source code for qpsolve manages the equality and inequality constraints so that it can be processed by the qp_solve primitive.3) 2.2) (2. Crawley [38]. The qpsolve function is a Scilab macro which aims at providing the same interface (that is. the same input/output arguments) as the quapro solver. 12]. In this chapter. The constraints matrix can be dense or sparse. please refer to Scilab online help for qpsolve The qpsolve Scilab macro is based on the work by Berwin A Turlach from The University of Western Australia. 23 .1 Mathematical point of view 1 f (x) = xT Qx + pT x 2 This kind of optimization is the minimization of function f (x) with under the constraints : T C 1 x = b1 T C 2 x ≥ b2 (2.
that is the number of unknowns n is negligible with respect to the number of constraints m.2. then the memory required is O(m). lw = 2n + r(r + 5)/2 + 2m + 1. in order to adapt to the speciﬁc Scilab sparse matrices storage. The qpgen2 routine is the original.QP. This formulae may be simpliﬁed in the following cases : • if n m. the qpgen2 fortran 77 routine is used. • if m n. This temporary array is deallocated when the qpsolve primitive returns. Depending on the constraints matrix. m). • if m = n.5 Internal design The original Goldfarb/Idnani algorithm [11. Then. The original package provides two routines : 24 . then the memory required is O(n). (2.3 qp solve The qp_solve compiled primitive is an interface for the fortran 77 solver.4 Memory requirements Suppose that n is the dimension of the quadratic matrix Q and m is the sum of the number of equality constraints me and inequality constraints md. 12] was designed to solve the following minimization problem: 1 min −dT x + xT Dx x 2 where A T x = b1 1 AT x >= b2 2 (2.4) (2.f) • If the constraints matrix C is a Scilab sparse matrix. the qpgen1sci routine is called.5) 2. It was considered as a building block for a Sequential Quadratic Programming solver. This routine is a modiﬁcation of the original routine qpgen1.7) where the matrix D is assumed to be symmetric positive deﬁnite. a dense or sparse solver is used : • If the constraints matrix C is dense.6) (2. then the memory required is O(n2 ).c. unmodiﬁed algorithm which was written by Turlach (the original name was solve. The interface is implemented in the C source code sci qp solve. that is the number of constraints m is negligible with respect to the number of unknowns n. the temporary work array which is allocated in the primitive has the size r = min(n. 2.
25 .compact. • solve.• solve.f containing routine qpgen2 which implements the algorithm for dense matrices.QP.f containing routine qpgen1 which implements the algorithm for sparse matrices.QP.
1 Mathematical point of view min x x The problem of the non linear leastquare optimization is to ﬁnd the solution of f (x)2 with bounds constraints or without constraints and with f : Rn → Rm the cost function. For more details about this function.2 Scilab function Scilab function called lsqrsolve is designed for the minimization of the sum of the squares of nonlinear functions using a LevenbergMarquardt algorithm.Chapter 3 Nonlinear least square 3. 3. 26 . please refer to Scilab online help 3.3 Optimization routines Scilab lsqrsolve function is based on the routines lmdif and lmder of the library Minpack (Argonne National Laboratory).
Vandenberghe and Stephen Boyd. m Z≥0 (4. 27 . the maximization or −T race(F0 .3) (4...2) (4.. please refer to Scilab online help 4.. Z) under the constraints: T race(Fi .3 Optimization routines Scilab semidef function is based on a routine from L.1 Mathematical point of view F0 + x1 F1 + . For more details about this function. Z) = ci . ..1) This kind of optimization is the minimization of f (x) = cT x under the constraint: 4. + xm Fm ≥ 0 or its dual problem.2 Scilab function The Scilab function called semidef is designed for this kind of optimization problems. i = 1.Chapter 4 Semideﬁnite programming 4.
Many articles on this subject have been collected by Carlos A. • GAs use payoﬀ (objective function) information. not deterministic rules. • multiobjective non linear minimization with bound constraints.Chapter 5 Genetic algorithms 5. The GA macros are based on the ”parameters” Scilab module for the management of the (many) optional parameters.1 Introduction Genetic algorithms are search algorithms based on the mechanics on natural selection and natural genetics [17. not a single point. • GAs search from a population of points. • GAs use probabilistic transition rules. • mutation. not the parameters themselves. 28]. Genetic algorithms are diﬀerent from more normal optimization and search procedures in four ways : • GAs work with a coding of the parameter set. Genetic algorithms have been introduced in Scilab v5 thanks to a work by Yann Collette [9]. The solver is made of Scilab macros. • crossover. Coello Coello on his website [7]. not derivativess or other auxiliary knowledge. 28 . which enables a highlevel programming model for this optimization solver. A brief introduction to GAs is done in [43]. A simple genetic algorithm that yields good results in many practical problems is composed of three operators [17] : • reproduction. The problems solved by the current genetic algorithms in Scilab are the following : • minimization of a cost function with bound constraints.
NbCouples = 110. endfunction function Res = max_bd_rastrigin() Res = [1 1]’. ’ crossover func ’ . params = add param ( ga params . 2 ) . m u t a t i o n g a d e f a u l t ) . ’ beta ’ . params = add param ( ga params . pressure = 0. ’ dimension ’ . params = add param ( ga params . which are conﬁgure with keyvalue pairs. params = add param ( ga params . Log = %T. 1 2 3 4 5 6 7 8 9 10 11 12 ga // ga ga ga ga ga // ga ga ga ga params = i n i t p a r a m ( ) . ’ mutation func ’ . are deﬁned in the following sample Scilab script. 0 . which are cleanly handled by the ”parameters” module. The add_param function allows to set individual named parameters. params = add param ( ga params . params = add param ( ga params . This module provides the nit_param function. nb_disp = 10. set of parameters. endfunction function Res = opti_rastrigin() Res = [0 0]’. Proba_cross = 0. such as the size of the population. Genetic Algorithms require many settings. endfunction function y = rastrigin(x) y = x(1)^2+x(2)^2cos(12*x(1))cos(18*x(2)). i n i t g a d e f a u l t ) . ’minbound ’ . ’ i n i t f u n c ’ .2 Example In the current section.05. Parameters to f i n e tune the Genetic algorithm .’y = ’+func+’(x)’). 0 ) . Other algorithmic parameters. NbGen = 10.7. ’ delta ’ . c o d i n g g a i d e n t i t y ) . 29 .1. PopSize = 100. params = add param ( ga params .5. c r o s s o v e r g a d e f a u l t ) . Proba_mut = 0. empty. params = add param ( ga params . endfunction This cost function is then deﬁned with the generic name ”f”. Parameters to adapt to the shape of the optimization problem params = add param ( ga params . The following is the deﬁnition of the Rastrigin function. ’ codage func ’ . function Res = min_bd_rastrigin() Res = [1 1]’. ’maxbound ’ . func = ’rastrigin’. which returns a new. we give an example of the user of the GA algorithms. eval ( ’ min bd ’+f u n c+’ ( ) ’ ) ) . eval ( ’max bd ’+f u n c+’ ( ) ’ ) ) . deff(’y=f(x)’. 1 ) .
Log .min / max value found = 1.min / max value found = 1.000252 > f = 1. s e l e c t i o n g a e l i t i s t ) .000558 x(2) = 0.3 Support functions In this section. 1 2 [ pop opt . .000409 > f = 1.999977 / 1.min / max value found = 1.000215 x(2) = 0.000101 x(2) = 0. Individual Individual Individual Individual Individual Individual Individual Individual Individual Individual 1: x(1) = 0.999982 4: x(1) = 0.000193 x(2) = 0.999968 9: x(1) = 0.994628 iteration 9 / 10 . The following are the messages which are displayed in the Scilab console : optim_ga: optim_ga: optim_ga: optim_ga: optim_ga: optim_ga: optim_ga: optim_ga: optim_ga: optim_ga: optim_ga: Initialization of the population iteration 1 / 10 .999964 5. P r o b a c r o s s .min / max value found = 1. .000442 > f = 1.314217 iteration 4 / 10 .984543 / 1.000519 x(2) = 0.min / max value found = 1. 4 i .999989 / 1. 5 end The previous script make the following lines appear in the Scilab console. f o b j p o p i n i t ] = .1. ’ nb couples ’ . f o b j p o p o p t .691332 iteration 6 / 10 .984184 / 0.000188 x(2) = 0.081632 iteration 2 / 10 .999966 10: x(1) = 0.998123 iteration 10 / 10 . 15 ga params = add param ( ga params .min / max value found = 1.000268 > f = 1.min / max value found = 1.min / max value found = 1.000235 x(2) = 0. The following script is a loop over the optimum individuals of the population. 30 .000136 > f = 1.999977 6: x(1) = 0. pop opt ( i ) ( 1 ) .853613 iteration 3 / 10 .999989 2: x(1) = 0. f o b j p o p o p t ( i ) ) .000351 > f = 1. ’ pressure ’ . PopSize .999551 / 1.13 ga params = add param ( ga params . The optim_ga function search a population solution of a singleobjective problem with bound constraints. Proba mut .999970 8: x(1) = 0.000427 > f = 1.min / max value found = 1.999534 The initial and ﬁnal populations for this simulation are shown in 5.682413 / 0. n b d i s p ) .000118 x(2) = 0. we analyze the GA services to conﬁgure a GA computation. ga params ) .000335 > f = 1.513463 iteration 5 / 10 .999989 / 1. 2 for i =1: n b d i s p 3 p r i n t f ( ’ Individual %d: x(1) = %f x(2) = %f −> f = %f\n ’ . 14 ga params = add param ( ga params .min / max value found = 1.999979 / 1. ’ s e l e c t i o n f u n c ’ . pop opt ( i ) ( 2 ) .871632 iteration 7 / 10 . p r e s s u r e ) . .000260 > f = 1.980356 iteration 8 / 10 . 1 p r i n t f ( ’ Genetic Algorithm : %d points from pop opt\n ’ . p o p i n i t . .000034 x(2) = 0. NbGen . optim ga ( f . NbCouples ) . .000497 x(2) = 0.000197 > f = 1.999979 5: x(1) = 0.998183 / 1.984184 / 1.999987 3: x(1) = 0.999974 7: x(1) = 0.
Optimum population is accumulated on the blue dot 31 .1: Optimum of the Rastrigin function – Initial population is in red.Figure 5.
3.2 Crossover The crossover function is used when mates have been computed.4 Initialization The initialization function returns a population as a list made of ”pop size” individuals. The Scilab macro init_ga_default computes this population by performing a randomized discretization of the domain deﬁned by the bounds as minimum and maximum arrays. The following is the list of selection functions available in Scilab : • selection_ga_random : A function which performs a random selection of individuals. we analyze the 4 GA solvers which are available in Scilab : • optim_ga : ﬂexible genetic algorithm • optim_moga : multiobjective genetic algorithm • optim_nsga : multiobjective Niched Sharing Genetic Algorithm 32 . • crossover_ga_binary : A crossover function for binary code 5. based on the Wheel algorithm : the crossover algorithm is a loop over the couples.3 Selection The selection function is used in the loop over the generations.4 Solvers In this section. 5. which modiﬁes both elements of each couple. 5. 5. We select pop size individuals in the set of parents and childs individuals at random. We select the best individuals in the set of parents and childs individuals. This randomization is based on the Scilab primitive rand.3.5. when the new population is computed by processing a selection over the individuals. The following is the list of crossover functions available in Scilab : • crossover_ga_default : A crossover function for continuous variable functions.1 Coding The following is the list of coding functions available in Scilab’s GA : • coding_ga_binary : A function which performs conversion between binary and continuous representation • coding_ga_identity : A ”nooperation” conversion function The user may conﬁgure the GA parameters so that the algorithm uses a customized coding function. • selection_ga_elitist : An ’elitist’ selection function.3.3.
the 3 other solvers are designed for multiobjective optimization. The following is an overview of the steps in the GA algorithm. 5. The method is based on [15]. • crossover : the two populations are processed through the crossover_func callback function (default : crossover_ga_default) • mutation : the two populations are processed throught the mutation_func callback function (default : mutation_ga_default) • computation of cost functions : the _ga_f function is called to compute the ﬁtness for all individuals of the two populations • selection : the new generation is computed by processing the two populations through the selection_func callback function (default : selection_ga_elitist) 5. 33 .4.1 optim ga The Scilab macro optim_ga implements a Genetic Algorithm to ﬁnd the solution of an optimization problem with one objective function and bound constraints.4.• optim_nsga2 : multiobjective Niched Sharing Genetic Algorithm version 2 While optim_ga is designed for one objective. • processing of input arguments In the case where the input cost function is a list. • initialization One computes the initial population with the init_func callback function (the default value for init_func is init_ga_default) • coding One encodes the initial population with the codage_func callback function (default : coding_ga_ide • evolutionary algorithm as a loop over the generations • decoding One decodes the optimum population back to the original variable system The loop over the generation is made of the following steps.2 optim moga. based on a randomized Wheel. the ”hidden” function _ga_f simply encapsulate the input function. If the input cost function is a regular Scilab function. The function pareto_filter extracts non dominated solution from a set. pareto ﬁlter The optim_moga function is a multiobjective genetic algorithm. one deﬁnes the ”hidden” function _ga_f which computes the cost function. • reproduction : two list of children populations are computed.
The method is based on [14].4 optim nsga2 The function optim_nsga2 is a multiobjective Niched Sharing Genetic Algorithm.3 optim nsga The optim_nsga function is a multiobjective Niched Sharing Genetic Algorithm.5.4.4. 5. The method is based on [37]. 34 .
namely locating a good approximation to the global optimum of a given function in a large search space. which enables a highlevel programming model for this optimization solver. The GA macros are based on the ”parameters” Scilab module for the management of the (many) optional parameters. To use the SA algorithm. 13]. we describe the Simulated Annealing optimization methods. Reference books on the subject are [24.Chapter 6 Simulated Annealing In this document. all tours that visit a given set of cities) [42]. 6.g. The current Simulated Annealing solver aims at ﬁnding the solution of min f (x) x with bounds constraints and with f : Rn → R the cost function. – the acceptance function. Genetic algorithms have been introduced in Scilab v5 thanks to the work by Yann Collette [9]. It is often used when the search space is discrete (e. • compute an initial temperature with a call to ”compute initial temp” • ﬁnd an optimum by using the ”optim sa” solver 35 .2 Overview The solver is made of Scilab macros. 6. 25. – the neighbor function. – the temperature law.1 Introduction Simulated annealing (SA) is a generic probabilistic metaalgorithm for the global optimization problem. a new feature available in Scilab v5 . one must perform the following steps : • conﬁgure the parameters with calls to ”init param” and ”add param” especially..
endfunction function y = r a s t r i g i n ( x ) y = x (1)ˆ2+ x (2)ˆ2 − cos ( 1 2 ∗ x (1)) − cos ( 1 8 ∗ x ( 2 ) ) . ’max bound ’ . ’ neigh func ’ . ’ accept func ’ . endfunction function Res = m a x b d r a s t r i g i n ( ) Res = [ 1 1 ] ’ . ’y=’+f u n c+’ (x) ’ ) . 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 // // Rastrigin function // function Res = m i n b d r a s t r i g i n ( ) Res = [−1 − 1 ] ’ . T0 = c o m p u t e i n i t i a l t e m p ( x0 .Max ) . It extern = 30. T0 ) . [ x opt . s i z e ( Min .6. I t i n t e r n = 1000. . . 36 . endfunction // // Set parameters // func = ’ rastrigin ’ . 1 ∗ ( Max−Min ) ) . It Pre = 100. 1 ) . ’ min delta ’ . t e m p l a w d e f a u l t ) . . . p r i n t f ( ’ I n i t i a l temperature T0 = %f\n ’ . // // Simulated Annealing with default parameters // p r i n t f ( ’SA: geometrical decrease temperature law\n ’ ) . param ( sa params param ( sa params param ( sa params param ( sa params param ( sa params param ( sa params param ( sa params . Proba start = 0 . Min ) . ’ max delta ’ . ’ temp law ’ . I t P r e . s a m e a n l i s t .3 Example The following example is extracted from the SA examples. 0 . . 2 ) ) + Min . endfunction function Res = o p t i r a s t r i g i n ( ) Res = [ 0 0 ] ’ . Min = eval ( ’ min bd ’+f u n c+’ ( ) ’ ) . s a v a r l i s t . The Rastrigin functin is used as an example of a dimension 2 problem because it has many local optima but only one global optimum. ’min bound ’ . sa params ) . f o p t . . t e m p l i s t ] = . x0 = (Max − Min ) . P r o b a s t a r t . . d e f f ( ’y=f (x) ’ . sa sa sa sa sa sa sa sa params params params params params params params params = = = = = = = = init add add add add add add add param ( ) . −0. . n e i g h f u n c d e f a u l t ) . ∗ rand ( s i z e ( Min . 8 . Max = eval ( ’max bd ’+f u n c+’ ( ) ’ ) . f .1∗(Max−Min ) ) . a c c e p t f u n c d e f a u l t ) .
a neighbour function is used in order to explore the domain [43]. for a continuous vector. optimal solution: . 37 . disp ( x o p t ) . ’k−’ ) . The prototype of a neighborhood function is the following : 1 function x n e i g h = n e i g h f u n c d e f a u l t ( x c u r r e n t . 1 ) . ’Temperature ’ ) . end drawnow . 1 . Log = %T.0. f o p t ) . a neighbor will be produced by adding some noise to each component of the vector. drawlater . s a m e a n l i s t . The following is a list of the neighbour functions available in the SA context : • neigh_func_default : SA function which computes a neighbor of a given point.46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 o p t i m s a ( x0 . p r i n t f ( ’ optimal solution : \n ’ ) . ’ Iteration ’ . subplot ( 2 .0006975 . ’ g ’ ) . ’Mean / Variance ’ ) . the following messages appear in the Scilab console. sa params ) . Variance and Temperature depending on the iteration. T. 2 ) . t .0. ’ Iteration ’ . plot ( t . x t i t l e ( ’ Geometrical annealing ’ . • T represents the current temperature. I t e x t e r n . s a v a r l i s t . [ t e m p l i s t ( i ) t e m p l i s t ( i ) ] . a neighbor will be produced by changing one bit from 0 to 1 or from 1 to 0. l e g e n d ( [ ’Mean ’ . 1 . ’ Variance ’ ] ) . param ) where: • x current represents the current point. for i =1: length ( t )−1 plot ( [ t ( i ) t ( i + 1 ) ] . For example. subplot ( 2 .4 Neighbor functions In the simulated annealing algorithm.999963 The ﬁgure 6. p r i n t f ( ’ value of the objective function = %f\n ’ . 6. x t i t l e ( ’Temperature evolution ’ . ’ r ’ . • param is a list of parameters.1 presents the evolution of Mean. I t i n t e r n . f .0000935 value of the objective function = 1. scf (). t = 1 : length ( s a m e a n l i s t ) . T0 . After some time. For a binary string.
The corresponding distribution is a Cauchy distribution which is more and more peaked as the temperature decreases. For common optimization. This distribution is more and more peaked as the temperature decreases.5 Acceptance functions There exist several kind of simulated annealing optimization methods: • the Fast Simulated Annealing.. • neigh_func_fsa : The Fast Simulated Annealing neghborhood relationship. To implement these various simulated annealing optimization methods. you need not to change the default acceptance function.1: Convergence of the simulated annealing algorithm • neigh_func_csa : The classical neighborhood relationship for the simulated annealing. • etc.. • the simulated annealing based on metropolishasting acceptance function. The neighbors distribution is a gaussian distribution which is more and more peaked as the temperature decrease. • neigh_func_vfsa : The Very Fast Simulated Annealing neighborhood relationship. 6. The following is a list of acceptance functions available in Scilab SAs : 38 . you only need to change the acceptance function.Figure 6.
which include a main. There are 5 temperature laws available in the SA context : • temp_law_default : A SA function which computes the temperature of the next temperature stage • temp_law_csa : The classical temperature decrease law. only the best point is returned as the algorithm output. the update may still be processed. If not. a temperature law is used in a statistical criteria for the update of the optimum [43]. based on the exponential function Fneigh − Fcurrent level = exp − T • accept_func_vfsa : is the Very Fast Simulated Annealing function.7 optim sa The optim_sa macro implements the simulated annealing solver. deﬁned by : Level = 1 + exp − 1 Fcurrent −Fneigh T 6. It is based on an iterative update of two points : • the current point is updated by taking into account the neighbour function and the acceptance criterium. and an internal loop. the one for which the convergence of the simulated annealing has been proven • temp_law_fsa : The Szu and Hartley Fast simulated annealing • temp_law_huang : The Huang temperature decrease law for the simulated annealing • temp_law_vfsa : This function implements the Very Fast Simulated Annealing from L. provided that a statistical criteria is satisﬁed. While the current point is used internally to explore the domain.6 Temperature laws In the simulated annealing algorithm. the update is done with the new point replacing the old optimum. Ingber 6. If the new (neighbor) point improves the current optimum. The algorithm is based on the following steps. 39 .• accept_func_default : is the default acceptance function. The statistical law decreases while the iterations are processed. It allows to ﬁnd the solution of an minimization problem with bound constraints. external loop over the temperature decreases. • the best point is the point which achieved the minimum of the objective function over the iterations.
the following steps are processed.• processing of input arguments. 40 . The internal loop allows to explore the domain and is based on the neighbour function. • loop over the number of temperature decreases. • initialization. store the temperature. • compute a neighbour of the current point. • if history is required by user. It is based on the following steps. • loop over internal iterations. • update the temperature with the temperature law. the x iterates. For each iteration over the temperature decreases. with constant temperature. the values of f. then overwrite the current point with the neighbour • if the cost of the best point is greater than the cost of the current point. overwrite the best point by the current point. • compute the objective function for that neighbour • if the objective decreases or if the acceptance criterium is true.
Delebecque@inria. . XM . 7.Chapter 7 LMITOOL: a Package for LMI Optimization in Scilab R.fr F. • f is a real linear scalar function of the entries of the unknown matrices X1 . . ... .fr. . • Gi ’s are real matrices with entries which are aﬃne functions of the entries of the unknown matrices. . .Nikoukhah@inria. . . . . . an interface to the program Semideﬁnite Programming SP (Copyright c 1994 by Lieven Vandenberghe and Stephen Boyd) distributed with Scilab. . This package uses Scilab function semidef. referred to as the unknown matrices. Delebecque Francois.1 Purpose Many problems in systems and control can be formulated as follows (see [6]): minimize f (X1 . XM . . . 41 . Victor. El Ghaoui ENSTA. . XM ) Gi (X1 . Internet: elghaoui@ensta. . Nikoukhah Ramine. . 75739 Paris. 2. p. they are referred to as “Linear Matrix Inequality” (LMI) functions. it is referred to as the objective function. France. . and in particular its two main functions lmisolver and lmitool for solving Linear Matrix Inequalities problems. . Bvd. . . i = 1. they are referred to as “Linear Matrix Equality” (LME) functions. 2. . 32. XM ) ≥ 0. . XM are unknown real matrices. . (In this report. Σ: subject to Hj (X1 .. XM .. where • X1 . the V ≥ 0 stands for V positive semideﬁnite unless stated otherwise). • Hj ’s are real symmetric matrices with entries which are aﬃne functions of the entries of the unknown matrices X1 .. q. Research supported in part by DRET under contract 92017BC14 This chapter describes a userfriendly Scilab package. . j = 1. . .fr L. . XM ) = 0. X1 . .
OBJ]=EVALFUNC(XLIST) where – XLIST: a list. user must provide an evaluation function which “evaluates” f .2.2 Function lmisolver LMITOOL is built around the Scilab function lmisolver. – LME: a list of matrices containing values of the LME functions Gi ’s for X values in XLIST. To solve Σ. Gi and Hj .The purpose of LMITOOL is to solve problem Σ in a userfriendly manner in Scilab. It can also be a list of a mixture of matrices and lists which in turn contain values of of LMI’s. It contains initial guess on the values of the unknown matrices. LME can be a matrix in case there is only one LME function to be evaluated (instead of a list containing this matrix as unique element). The size and structure of XLIST0 are used to set up the problem and determine the size and structure of the output XLISTF. are used as intial condition for the optimization algorithm. and so on.2).3. User can either invoke lmisolver directly. they are ignored otherwise. as well as an initial guess on the values of the unknown matrices. given functions f . • EVALFUNC: a Scilab function called evaluation function (supplied by the user) which evaluates the LME. – LMI: a list of matrices containing the values of the LMI functions Hj ’s for X values in XLIST. by providing the necessary information in a special format or he can use the interactive function lmitool described in Section 7. In general. This function computes the solution X1 . Gi and Hj as a function the unknown matrices. given the values of the unknown matrices. The values of the matrices in XLIST0. It can also be a list of a mixture of matrices and lists which in turn contain values of LME’s. XM of problem Σ. 7. . identical in size and structure to XLIST0. 42 .OPT]] = lmisolver(XLIST0. using the code SP [23]. LMI can also be a matrix (in case there is only one LMI function to be evaluated). This code is intended for small and mediumsized problems (say. In some cases however it is more convenient to deﬁne one or more elements of XLIST0 to be lists (of unknown matrices) themselves.2.options]) • XLIST0: a list structure including matrices and/or list of matrices. .EVALFUNC[. . This is a useful feature when the number of unknown matrices is not ﬁxed a priori (see Example of Section 7. The syntax is: [LME. 7. up to a few hundred variables). and so on.1 where Syntax [XLISTF[. the ith element of XLIST0 is the initial guess on the value of the unknown matrix Xi . LMI and objective functions. if compatible with the LME functions. .LMI.
we can call lmisolver.Y.2. This argument is optional. maxiters. • options: a 5×1 vector containing optimization parameters Mbound.– OBJ: a scalar equal to the value of the objective function f for X values in XLIST.OBJ]=sf_sat_eval(XLIST) [Q. as follows: 43 .e. i.[umax^2*eye(Y*Y’). if and only if there exist an n × n matrix Q and an nu × n matrix Y satisfying the equality constraint Q − QT = 0 and the inequality constraints Q ≥ 0 −AQ − QA − BY − Y B T > 0 u2 I Y max ≥ 0 YT Q T T in which case one such K can be constructed as K = Y Q−1 . • OPT: a scalar corresponding to the optimal value of the minimization problem Σ.Q]. default parameters are used. identical in size and structure to XLIST0 containing the solution of the problem (optimal values of the unknown matrices).2 Examples Statefeedback with control saturation constraint Consider the linear system x = Ax + Bu ˙ where A is an n × n and B. If the Σ problem has no equality constraints then LME should be []. and reconstruct the solution in Scilab. and reltol. 7.Y]=XLIST(:) LME=QQ’ LMI=list(A*QQ*A’B*YY’*B’.LMI. Assuming A.Qeye()) OBJ=[] Note that OBJ=[] indicates that the problem considered is a feasibility problem. There exists a stabilizing state feedback K such that for every initial condition x(0) with x(0) ≤ 1. an n × nu matrix. nu. abstol. function [LME. see manual page for semidef for details (Mbound is a multiplicative coeﬃcient for M). Similarly for LMI and OBJ. if omitted. the resulting control satisﬁes u(t) for all t ≥ 0.Y’. we are only interested in ﬁnding a set of X’s that satisfy LME and LMI functions. To solve this problem using lmisolver. B and umax already exist in the environment. • XLISTF: a list.. we ﬁrst need to construct the evaluation function.
Control of jump linear systems We are given a linear system x = A(r(t))x + B(r(t))u. It can be shown that this problem has a solution if and only if there exist n × n matrices Q(1). > 0. + TrQ(N ) − 1 = 0. . .B. i = 1. In this case. it has the form u(t) = K(r(t))x(t). . In this case. .e. B(N ) and the transition matrix Π. That is. i.umax) We call sf_sat the solver function for this problem. i = 1. Y (N ). . The transition probabilities of the process r are deﬁned by a “transition matrix” Π = (πij ). N . . K(i)’s are nu × n matrices (the unknowns of our control problem). ˙ where A is n × n and B is n × nu . and nu × n matrices Y (1). The control law we look for is a modedependent linear statefeedback. . [Q. A(N ). . . . In the above problem. . . XLIST0=list(Q_init. N }. . We seek a statefeedback control law such that the resulting closedloop system is meansquare stable. the resulting trajectory of the closedloop system satisﬁes limt→∞ E x(t) 2 = 0. referred to as “jump linear systems”. . say sf_sat. TrQ(1) + . The scalar parameter r(t) is a continuoustime Markov process taking values in a ﬁnite set {1. B(1). . N. XLIST=lmisolver(XLIST0. the Y ’s. all we need to do is type: > [Q. . . where πij ’s are the transition probability rates from the ith mode to the jth. such that Q(i) − Q(i)T = 0. − A(i)Q(i) + Q(i)A(i)T + B(i)Y (i) + Y (i)T B(i)T + j=1 πji Q(j) If such matrices exist. . . .Y]=sf_sat(A. . and Q(i) Y (i)T Y (i) I N > 0. the data matrices are A(1).sf_sat_eval). . The above problem is obviously a Σ problem. The evaluation function required for invoking lmisolver can be constructed as follows: 44 .Y]=XLIST(:) These Scilab commands can of course be encapsulated in a Scilab function. Then. Such systems. a stabilizing statefeedback is given by K(i) = Y (i)Q(i)−1 .Y_init). .> > > > > Q_init=zeros(A). both the number of the data matrices and that of the unknown matrices are apriori unknown. Q(N ). Y_init=zeros(B’). we can let XLIST be a list of two lists: one representing the Q’s and the other. . can be used to model linear systems subject to failures. . . . . for every initial condition x(0). The unknown matrices are Q(i)’s (which are symmetric n × n matrices) and Y (i)’s (which are nu × n matrices). . . To solve this problem.
n).nu)] SUM=zeros(n. This is just a matter of convenience. Descriptor Lyapunov inequalities In the study of descriptor systems. Q_init=list(). we can solve the problem in Scilab as follows (assuming lists A and B. and matrix PI have already been deﬁned).Y]=XLISTF(:). The above commands can be encapsulated in a solver function. The evaluation function can be written as follows 45 . First we should initialize Q and Y. we can use lmisolver as follows: > XLIST0=list(Q_init. [n.OBJ]=jump_sf_eval(XLIST) [Q.Y(i). > for i=1:N.Y(i)’. In this problem.B. the LME functions play important role. LMI1=list(). Now.Y]=jump_sf(A.nu]=size(B(1)) LME=list(). in which case we simply need to type: > [Q. A is a regular pencil.LMI2) LME(N+1)=tr1 OBJ=[] Note that LMI is also a list of lists containing the values of the LMI matrices. say jump_sf.PI) to obtain the solution.Y_init(i)=zeros(nu. LMI2=list() tr=0 for i=1:N tr=tr+trace(Q(i)) LME(i)=Q(i)Q(i)’ LMI1(i)=[Q(i).function [LME.Y_init) > XLISTF=lmisolver(XLIST0.jump_sf_eval) > [Q.i)*Q(j) end LMI2(i)= A(i)*Q(i)+Q(i)*A(i)’+B(i)*Y(i)+Y(i)’*B(i)’+SUM end LMI=list(LMI1.end Then.LMI.n) for j=1:N SUM=SUM+PI(j. [n. Q_init(i)=zeros(n.eye(nu.nu]=size(B(1)).Y]=XLIST(:) N=size(A). which clearly is a Σ problem. it is sometimes necessary to ﬁnd (or ﬁnd out that it does not exist) an n × n matrix X satisfying ET X = X T E ≥ 0 AT X + X T A + I ≤ 0 where E and A are n × n matrices such that E. Y_init=list().n). > N=size(A).
In [22]. we deﬁne the evaluation function: function [LME.L_init).LMI.C2*X+D22*L.. (X*C1’+L’*D12’+B1*D11’)’.Y.OBJ]=h2hinf_eval(XLIST) [X... L_init=zeros(B2’) XLIST0=list(X_init.YY’).h2hinf_eval).X*C1’+L’*D12’+B1*D11’.Y.L]=XLIST(:) LME=list(XX’. Y − Y T = 0.Y_init. XLISTF=lmisolver(XLIST0. and use it as follows: > > > > X_init=zeros(A). [Y. it is shown that the solution to this problem can be expressed as K = LX −1 where X and L are obtained from the problem of minimizing Trace(Y ) subject to: X − X T = 0.X]) OBJ=trace(Y)..E’*X) OBJ=[] and the problem can be solved by (assuming E and A are already deﬁned) > XLIST0=list(zeros(A)) > XLISTF=lmisolver(XLIST0.L]=XLISTF(:) 46 . and − T T T T AX + B2 L + (AX + B2 L)T + B1 B1 XC1 + LT D12 + B1 D11 T T C1 X + D12 L + D11 B1 −γ 2 I + D11 D11 > 0 > 0 Y C2 X + D22 L (C2 X + D22 L)T X To solve this problem with lmisolver.dscr_lyap_eval) > X=XLISTF(:) Mixed H2 /H∞ Control Consider the linear system x = Ax + B1 w + B2 u ˙ z1 = C1 x + D11 w + D12 u z2 = C2 x + D22 u The mixed H2 /H∞ control problem consists in ﬁnding a stabilizing feedback which yields Tz1 w ∞ < γ and minimizes Tz2 w 2 where Tz1 w ∞ and Tz2 w 2 denote respectively the closedloop transfer functions from w to z1 and z2 ..(C2*X+D22*L)’..OBJ]=dscr_lyap_eval(XLIST) X=XLIST(:) LME=E’*XX’*E LMI=list(A’*XX’*Aeye(). LMI=list([A*X+B2*L+(A*X+B2*L)’+B1*B1’. Y_init=zeros(C2*C2’).gamma^2*eye()+D11*D11’]. [X.LMI.function [LME.
A.P] OBJ=trace(P) which can be used as follows (asuming E. This problem can be formulated in LMITOOL as follows: function [LME. ET CT 0 I P =− 0 0 I It can be shown that this problem can be formulated as a Σ problem as follows: maximize Trace(P) under constraints P − PT = 0 and AP AT + Q 0 EP 0 R CP ≥ 0. white Gaussian noise sequences with covariance Q and R respectively. C. > P_init=zeros(A’*A) > P=lmisolver(XLIST0.C*P.zeros(C*A’). where A and C are matrices and e.P*C’.n]=size(A) 47 .E*P.P*E’.R.Descriptor Riccati equations In Kalman ﬁltering for descriptor system Ex(k + 1) = Ax(k) + u(k) y(k + 1) = Cx(k + 1) + r(k) where u and r are zeromean.LMI. Cx + d = 0.OBJ]=ric_dscr_eval(XLIST) LME=PP’ LMI=[A*P*A’+Q. Q and R are deﬁned and have compatible sizes–note that E and A need not be square).zeros(A*C’). b and d are vectors with appropriate dimensions. T T T T P E P C P The evaluation function is: function [LME.LMI. one needs to obtain the positive solution to the descriptor Riccati equation (see [33]) −1 AP AT + Q 0 E 0 0 R C 0 . Here the sign ≥ is to be understood elementwise.OBJ]=linprog_eval(XLIST) [x]=XLIST(:) [m.ric_dscr_eval) Linear programming with equality constraints Consider the following classical optimization problem minimize eT x subject to Ax + b ≥ 0.
’d’) for the discrete problem. the discrete Sylvester equation.mc]=size(C).C.C. if ma<>namb<>nbnc<>namc<>nb then error("invalid dimensions").linprog_eval) Sylvester Equation The problem of ﬁnding matrix X satisfying AX + XB = C or AXB = C where A and B are square matrices (of possibly diﬀerent sizes) is a wellknown problem. e.end XLISTF=lmisolver(zeros(nc. to solve the problem. These two problems can easily be formulated as Σ problems as follows: function [LME.ma]=size(A). all we need to do is to (assuming A.B. 48 .B.’c’) for the continuous problem and > X=sylvester(A.C.LME=C*x+d LMI=list() tmp=A*x+b for i=1:m LMI(i)=tmp(i) end OBJ=e’*x and solved in Scilab by (assuming A.[nb.flag) [na.mb]=size(B). b and d and an initial guess x0 exist in the environment): > x=lmisolver(x0.LMI.B. C.OBJ]=sylvester_eval(XLIST) [X]=XLIST(:) if flag==’c’ then LME=A*X+X*BC else LME=A*X*BC end LMI=[] OBJ=[] with a solver function such as: function [X]=sylvester(A. We refer to the ﬁrst equation as the continuous Sylvester equation and the second.[nc. B and C are deﬁned) > X=sylvester(A.sylvester_eval) X=XLISTF(:) Then.mc).
Y’.load (and compile) your functions: ! ! getf(’/usr/home/DrScilab/sf_sat. you need to ! !1.7.’A. Invoking >txt=lmitool(’sf_sat’. In this mode.’Q. • xlist: a string containing the names of the unknown matrices (separated by commas if there are more than one).B.sci’. The name of this ﬁle is obtained by adding “.’c’) 49 .3 Function LMITOOL The purpose of LMITOOL is to automate most of the steps required before invoking lmisolver.edit file /usr/home/DrScilab/sf_sat.3.sci” to the end of probname. 7.sci ! !2.varlist.2. lmitool generates a ﬁle in the current directory. • dlist: a string containing the names of data matrices (separated by commas if there are more than one). The solver function is used to deﬁne the initial guess and to modify optimization parameters (if needed).2.sci ﬁle including the solver function and the evaluation function or at least their skeleton.datalist) where • probname: a string containing the name of the problem.umax’) yields the output > txt = ! ! ! ! ! ! ! ! To solve your problem. • txt: a string providing information on what the user should do next.1 Noninteractive mode lmitool can be invoked with three input arguments as follows: Syntax txt=lmitool(probname. one or three arguments. it generates a *. In particular. lmitool can be invoked with zero. Example Suppose we want to use lmitool to solve the problem presented in Section 7. This ﬁle is the skeleton of a solver function and the corresponding evaluation function.
.. options=[Mbound.umax) ! !To check the result.umax and call sf_sat function: ! ! [Q.Y]=sf_sat(A. Y_init=.sci’ with the following content: function [Q.B....umax) // Generated by lmitool on Tue Feb 07 10:30:35 MET 1995 Mbound = 1e3.LMI.reltol].nu.Y]=XLIST(:) /////////////////DEFINE LME. LMI and OBJ BELOW LME=. It is easy to see how a small amount of editing can do the rest! 7. OBJ=.B.LMI.Y)) ! ! ! ! ! ! and results in the creation of the ﬁle ’/usr/home/curdir/sf sat.sf_sat_eval.3. reltol = 1e10.OBJ]=sf_sat_eval(list(Q. nu = 10.Define A..maxiters.B. ///////////DEFINE INITIAL GUESS BELOW Q_init=.2 Interactive mode lmitool can be invoked with zero or one input argument as follows: 50 .options) [Q. LMI=. abstol = 1e10..Y]=sf_sat(A. maxiters = 100...OBJ]=sf_sat_eval(XLIST) [Q.Y_init) XLIST=lmisolver(XLIST0. use [LME. /////////// XLIST0=list(Q_init.! !3..Y]=XLIST(:) /////////////////EVALUATION FUNCTION//////////////////////////// function [LME.abstol.
Syntax txt=lmitool() txt=lmitool(file) where • file: is a string giving the name of an existing “.. . The objective is to ﬁnd L such that the estimate x = Ly ˆ is unbiased and the worst case estimation error variance E( x − x 2 ) is minimized. H(N )} .. V (N )} where Co denotes the convex hull and H(i) and V (i). i = 1.. and I (L(i)V (i))T ≥ 0. To use lmitool for this problem..... Using a succession of dialogue boxes.. N.. L(i)V (i) X(i) γ − Trace(X(i)) ≥ 0. This mode is very easy to use and its operation completely self explanatory. y is known. N. Invoking lmitool with one argument allows the user to start oﬀ with an existing ﬁle.. This mode is useful for modifying existing ﬁles or when the new problem is not too much diﬀerent from a problem already treated by lmitool.sci” ﬁle generated by lmitool.. w is a unitvariance zeromean Gaussian vector. i = 1.. N.. we invoke it as follows: > lmitool() This results is an interactive session which is partly illustrated in following ﬁgures. .. N. user can completely deﬁne his problem. .. . .. V ∈ Co {V (1). i = 1. and H ∈ Co {H(1). X(i) − X(i)T = 0. lmitool is fully interactive. In this mode. N.. 51 . are given matrices. . ˆ It can be shown that this problem can be formulated as a Σ problem as follows: minimize γ subject to I − LH(i) = 0.. . i = 1.. i = 1. Example Consider the following estimation problem y = Hx + V w where x is unknown to be estimated...
and in particular the size of the unknown vector. . Az + b = 0. the problem is reformulated as minimize cT x subject to F0 + x1 F1 + · · · + xm Fm ≥ 0. . z0 is done using sparse LU functions of Scilab. . The computation of N. This step uses extensively sparse matrices to speed up the computation and reduce memory requirement. 2. where c is a vector. lmisolver eliminates the redundant variables. Then. a column compression is performed. 3. all solutions of the equality constraints are parametrized by z = N x + z0 . Elimination of variables. The equality constraints are eliminated by computing the null space N of A and a solution z0 (if any) of Ax + b = 0. The sizes and structure of the initial guess are used to set up the problem.1: This window must be edited to deﬁne problem name and the name of variables used. . ˜ ˜ where z contains the coeﬃcients of all matrix variables. Initial setup. XM . Once the equality constraints are eliminated. Fm are symmetric matrices. where x is a vector containing the independent variables. Elimination of equality constraints. Making repeated calls to the evaluation function. and x contains the independent elements in the matrix variables X1 .7. . .) Figure 7. . . At this stage. 52 . and F0 . lmisolver generates a canonical representation of the form minimize cT z ˜ ˜ ˜ ˜ subject to F0 + z1 F1 + · · · + zm Fm ≥ 0.4 How lmisolver works The function lmisolver works essentially in four steps: 1. (If the Fi ’s are dependent.
and the meaning of the above optimization parameters see manual page for semidef. Optimization.2: For the example at hand the result of the editing should look something like this. The Matlab version can be obtained by anonymous ftp from ftp.5 Other versions LMITOOL is also available on Matlab. The feasibility phase is avoided if the initial guess is found to be feasible.fr under /pub/elghaoui/lmitool.Figure 7. 4. This phase is itself divided into a feasibility phase and a minimization phase (only if the linear objective function is not empty). The function semidef is called with the optimization parameters abstol. reltol. Finally. 53 .ensta. The parameter M is set above the value Mbnd*max(sum(abs([F0 . maxiters. lmisolver makes a call to the function semidef (an interface to SP [23]). 7. Fm]))) For details about the optimization phase.. nu..
Figure 7. 54 .3: This is the skeleton of the solver function and the evaluation function generated by LMITOOL using the names deﬁned previously.
You can modify it if you want. Figure 7. 55 .Figure 7.5: A ﬁle is proposed in which the solver and evaluation functions are to be saved. we obtain.4: After editing.
2 SIF ﬁles and the CUTEr toolbox The SIF ﬁle format can be processed with the CUTEr Scilab toolbox.f.d and an Output messages ﬁle OUTMESS are also generated. GROUP. GROUPD. All these ﬁles are created in the directory whose path is given in Pathout. 41] but other ﬁle formats appeared in recent years.f. such as the XMLbased ﬁle format OSiL [35. MPS format is a standard ASCII medium for LP codes. 8. • GAMS : General Algebraic Modeling System [40.f and if automatic diﬀerentiation is required ELFUND.f.f. 30].f of hopdm (J.f. For a description of the variables. MPS format is described in more detail in Murtagh’s book [30]. The following sections describe Scilab tools to manage optimization data ﬁles. 16] • AMPL : A Mathematical Programming Language [10. The sifdecode function is based on the Sifdec code [20]. 8.f.1 MPS ﬁles and the Quapro toolbox The Quapro toolbox implements the readmps function. Gondzio). ELFUN. see the ﬁle rdmps1. An associated data ﬁle named OUTSDIF. 39] • MPS : Mathematical Programming System [27.f. 56 . 36]. 8. Given a SIF [1] ﬁle the function sifdecode generates associated Fortran routines RANGE. The following is a (nonexhaustive) list of ASCII ﬁle formats often used in optimization softwares : • SIF : Standard Input Format [1. which reads a ﬁle containing description of an LP problem given in MPS format and returns a tlist describing the optimization problem. EXTER. EXTERA.Chapter 8 Optimization data ﬁles This section presents the optimization data ﬁles which can be used to conﬁgure a speciﬁc optimization problem in Scilab. It is an interface with the program rdmps1. More precisely it results of an interface of SDLANC Fortran procedure.
Then we outline other main optimization toolboxes.1) (9.2) (9. It has been transformed into a toolbox for license reasons. In this chapter.Chapter 9 Scilab Optimization Toolboxes Some Scilab toolboxes are designed to solve optimization problems.1) & (9. 9. C ∗x≤b ci ≤ x ≤ cs C e ∗ x = be (9.2) & (9.3)). which allows to solve linear and quadratic problems.2)) • or inequality constraints.1) • or inequality constraints and bound constraints ((9.1 Quapro The Quapro toolbox was formely a Scilab builtin optimization tool.3) 57 .1 Linear optimization Mathematical point of view This kind of optimization is the minimization of function f (x) with f (x) = pT x under: • no constraints • inequality constraints (9. 9.1. bound constraints and equality constraints ((9.1) & (9. we begin by presenting the Quapro toolbox.
L. Zhou from the University of Maryland. Schittkowski from the University of Bayreuth.1) & (9.Scilab function Scilab function called linpro is designed for linear optimization programming.1. Scilab function Scilab functions called quapro (whatever Q is) and qld (when Q is positive deﬁnite) are designed for linear optimization programming. For more details about this function.1) • or inequality constraints and bound constraints ((9.1) & (9.3)). 58 . please refer to Scilab online help for quapro and Scilab online help for qld qld function and associated routine have been written by K. LIPSOL is available on Scilab web site Optimization routines Scilab linpro function is based on: • some Fortran routines written by the authors of linpro • some Fortran Blas routines • some Fortran Scilab routines • some Fortran Lapack routines 9.2 Linear quadratic optimization Mathematical point of view This kind of optimization is the minimization of function f (x) with 1 f (x) = xT Qx + pT x 2 under: • no constraints • inequality constraints (9.L. For this kind of problem. quapro function and associated routines have been written by Cecilia Pola Mendez and Eduardo Casas Renteria from the University of Cantabria. you can use a Scilab toolbox called LIPSOL that gives an equivalent of linpro for sparse matrices.2) & (9.2)) • or inequality constraints. Tits and J. Please note that this function can not solve problems based on sparse matrices. A. bound constraints and equality constraints ((9. For more details about these functions. please refer to Scilab online help This function and associated routines have been written by Cecilia Pola Mendez and Eduardo Casas Renteria from the University of Cantabria. Both functions can not solve problems based on sparse matrices.
ufn or ugr.uk/cute/mastsif. cdh. the hessian values. of the problem at a given point (see ufn.. This set comes from www.ac. for problems with general contraints) 59 . • Problem database A set of testing problems coded in ”Standard Input Format” (SIF) is included in the sif/ subdirectory. The available problems are located in the sif directory.. for example. . . cgr. contraints properties and regularity properties • SIF format decoder The Scilab function sifdecode can be used to generate the Fortran codes associated to a given problem. A typical use start from problem selection using the scilab function sifselect. while the Scilab function buildprob compiles and dynamically links these fortran code with Scilab • problem relative functions The execution of the function buildprob adds a set of functions to Scilab. udh. This gives a vector of problem names corresponding to selection criteria [32]. This toolbox is a scilab port by Serge Steer and Bruno Durand of the original Matlab toolbox.. ugr.rl. These functions are to be called before any of the following to initialize the problem relative data (only one problem can be run at a time).numerical. The sifbuild function can then be used to generate the fortran codes associated to a given problem. The sifoptim function automatically applies the optim function to a selected problem.2 CUTEr CUTEr is a versatile testing environment for optimization and linear algebra solvers [31]. to compile them and dynamically link it to Scilab. . A Fortran compiler is mandatory to build problems.html. The ﬁrst one is usetup for unconstrained or bounded problems or csetup for problems with general contraints. The Scilab function sifselect can be used to select some of this problems according to objective function properties. This toolbox contains the following parts. the gradient. This will create a set of problem relative functions. for unconstrained or bounded problems or cfn. This functions can be called to compute the objective function or its gradient at a given point..Optimization routines Scilab quapro function is based on: • some Fortran routines written by the authors of linpro • some Fortran Blas routines • some Fortran Scilab routines • some Fortran Lapack routines 9... The other functions allow to compute the objective.
scilab. The functions are based on macros based functions : no compiler is required. It provides the starting point for each problem. 60 . – CONMIN on Scilab Toolbox center • Diﬀerential Evolution: random search of a global minimum by Helmut Jarausch. Vanderplaats (1973). The CONMIN fortran program has been written by G. Additionnally.2 (or better). The CONMIN optimization method is currently used in NASTRAN (a professionnal ﬁnite element tool) and the optimization part of NASTRAN (the CONMIN tool). The More. which is an advantage over the CUTEr toolbox. Garbow and Hillstrom collection of test functions [29] is widely used in testing unconstrained optimization software. the optimum function value and the optimum point x for many problems.php/p/uncprb To install it. it provides ﬁnite diﬀerence routines for the gradient.org/toolboxes/uncprb and is manage under Scilab’s Forge : http://forge.3 The Unconstrained Optimization Problem Toolbox The Unconstrained Optimization Problem Toolbox provides 35 unconstrained optimization problems. The goal of this toolbox is to provide unconstrained optimization problems in order to test optimization algorithms. the gradient. the Jacobian and provides the Hessian matrix for 18 problems. 1 a t o m s I n s t a l l ( ’ uncprb ’ ) 9. CONMIN uses a twostep limited memory quasiNewtonlike Conjugate Gradient. gradients. the Jacobian and the Hessian matrix. type the following statement in Scilab v5. This toolbox is based on a RainerStorn algorithm. The code for these problems is available in Fortran from the netlib software archives. The following is a list of references for the CUTEr toolbox : • CUTEr toolbox on Scilab Toolbox center • CUTEr website 9. This toolbox is available in ATOMS : http://atoms.4 Other toolboxes • Interface to CONMIN: An interface to the NASTRAN / NASA CONMIN optimization program by Yann Collette.scilab. Finally. all function values. Jacobians and Hessians are tested.org/index. It provides the function value. CONMIN can solve a nonlinear objective problem with nonlinear constraints.• CUTEr and optim The Scilab function optim can be used together with CUTEr using either the external function ucost or the driver function sifoptim. the function vector.
– LPSOLVE toolbox on Scilab Toolbox center – lp solve solver on Sourceforge – lp solve on Geocities – lp solve Yahoo Group • NEWUOA: NEWUOA is a software developped by M. the GNU lesser general public license. Rubio Scola. Powell for unconstrained optimization without derivatives. lp solve uses the ’Simplex’ algorithm and sparse matrix methods for pure LP problems.3. lp solve is a free mixed integer/binary linear programming solver with full source. This toolbox is designed for nonlinear optimization with equality and inequality constraints. This solver can handle large scale optimization problems. The NEWUOA seeks the least value of a function F(x) (x is a vector of dimension n ) when F(x) can be calculated for any vector of variables x 61 . This version only works on linux. Modiﬁcations to Scilab Interface made by Edson Cordeiro do Valle. – FSQP on Scilab Toolbox center – FSQP website • IPOPT interface: interface for Ipopt.0. Rubio Scola.J. also for commercial purposes.D. Argentina). It is distributed freely under the terms of the GPL. examples and manuals. – Ipopt on Scilab Toolbox center – Ipopt website • Interface to LIPSOL: sparse linear problems with interior points method by H. This ScilabIpopt interface was based on the Matlab Mex Interface developed by Claas Michalik and Steinar Hauan. symmetric positive deﬁnite linear systems. You are free to use it. lp solve is under LGPL.0. Tested with gcc 4. LIPSOL can minimize a linear objective with linear constraints and bound constraints. – LIPSOL on Scilab Toolbox center – LIPSOL website – LIPSOL User’s Guide • LPSOLVE: an interface to lp solve.3 written by Esmond Ng and Barry Peyton by H. LIPSOL is written by Yin Zhang . Rubio Scola (University of Rosario. sparse. which is based on an interior point method which can handle equality and inequality nonlinear constraints. FSQP is a commercial product. As open source software. The original Matlabbased code has been adapted to Scilab by H. the source code for Ipopt is provided without charge. which uses sparsematrix datastructure to solve large.– Diﬀerential Evolution on Scilab Toolbox center • FSQP Interface: interface for the Feasible Sequential Quadratic Programming library. It is based on a primaldual interior point method. scons and Scilab >=4. LIPSOL also uses the ORNL sparse Cholesky solver version 0.
the new model interpolates F at m points. The algorithm is iterative. a quadratic model being required at the beginning of each iteration. – NEWUOA toolbox on Scilab Toolbox center – NEWUOA at INRIA Alpes 62 . which is used in a trust region procedure for adjusting the variables. When the quadratic model is revised.. the value m=2n+1 being recommended.
Notice that IPOPT is a commercial product and CONMIN is a domainpublic library. Here is a list of features which are not available neither in Scilab. based on interior point methods. • customization of errors for constraints. but which are available as toolboxes (see previous section). • nonlinear objective with nonlinear constraints problems based on sparse linear algebra. Therefore the only opensource. but are available in toolboxes. Functionalities marked with a (*) would be available in Scilab if the MODULOPT library embedded in Scilab was updated. These features would be to include in Scilab. nor in toolboxes. 63 . • features which are not available neither in Scilab. nonlinear solver with non linear constraints tool available with Scilab is the interface to CONMIN. • simplex programming method (*). based on method of feasible directions. nor in toolboxes. • quadratic objective solver with sparse objective matrix. • integer parameter with linear objective solver and sparse matrices : currently available in LPSOLVE toolbox. free. based on the simplex method. • enabling/disabling of unknowns or constraints. • nonlinear objective with nonlinear constraints (*). Two classes of missing features are to analyse : • features which are not available in Scilab.Chapter 10 Missing optimization features in Scilab Several optimization features are missing in Scilab. • nonlinear objective and non linear constraints : currently available in interface to CONMIN toolbox. based on interior points method. Here is a list of features which are not available in Scilab. • nonlinear objective and non linear constraints : currently available in interface to IPOPT toolbox. • linear objective with sparse matrices : currently available in LIPSOL.
by downloading and installing some toolboxes. you can easily improve your Scilab capabilities. One of the questions we can ask is: “Why are these toolboxes not integrated in Scilab distribution?”.Conclusion Even if Scilab itself has lacks in optimization functionalities. 64 . After that. The answer is often a problem of license. all embedded functions are very useful to begin with. All GPL libraries can not be included in Scilab since Scilab is not designed to become GPL.
volume 15 of Studies in Applied Mathematics. 909:226–239. 27:1–33.numerical. SIAM. E. and Idnani A. The sif reference document. Nelder mead user’s manual. revue et augment´ee.rl. November 2006. Toint. 490 pages. Conn and Philippe L.Bibliography [1] Nicholas I. and Claudia A. Optimization with scilab. Genetic Algorithms and Simulated Annealing. PA. https://www.uk/lancelot/sif/sifhtml. wikipedia. 2009. 1982. Boyd. [5] Joseph Fr´d´ric Bonnans. June 1994. Universitext. Personnal website. [9] Yann Collette. USA. present and future. Springer a Verlag. Dual and primaldual methods for solving strictly convex quadratic programs. [3] Michael Baudin and Serge Steer. Balakrishnan. http://en.lania. Linear Matrix Inequalities in System and Control Theory. Sagase e e tiz´bal. Morgan Kaufmann Publishers Inc. http://wiki. M. Coello Coello. Theoretical and Practical Aspects. San Francisco.org/wiki/AMPL_programming_language. http://www.html. http://www.html.. [2] Michael Baudin. Optimization services instance language (osil). Nouvelle ´dition. JeanCharles Gilbert. e e [6] S.org/OS/ OSiL. List of references on evolutionary multiobjective optimization.coinor.org. 65 .free. Numerical Optimization. Feron. CA.Rocquencourt. To appear in Proceedings Of 2009 International Workshop On OpenSource Software For Scientiﬁc Computation (Ossc2009). http://ycollette.mx/~ccoello/EMOObib. [11] Goldfarb D. and Idnani A. [13] Lawrence Davis.scilab.html. Technical Report RR0242. A variant of a projected variable metric method for bound constrained e e optimization problems. El Ghaoui. Lecture Notes in Mathematics. [7] Carlos A. Gould Andrew R.org/The_NelderMead_ Component. Claude Lemar´chal. Mathematical Programming. 1982. A numerically stable dual method for solving strictly convex quadratic programs. INRIA . Octobre 1983.fr. A modeling language for mathematical programming. 1987. [12] Goldfarb D. [10] AMPL company.ac. and V. L. SpringerVerlag. Philadelphia. [4] Fr´d´ric Bonnans. [8] coin or.
[26] Claude Lemar´chal.rl. A fast elitist nondominated sorting genetic algorithm for multiobjective optimization: Nsgaii. USA. pages 416–423. [15] Carlos M. pages 849–858. Genetic Algorithms + Data Structures = Evolution Programs. Hillstrom. 1996. [21] Stephen J. [18] JeanBaptiste HiriartUrruty and Claude Lemar´chal. M.uk.rl.lania. J. Aarts. 1999. Springer. Lectures Notes in Control and Information e Sciences. Murtagh. Rotea. [19] JeanBaptiste HiriartUrruty and Claude Lemar´chal. Semideﬁnite programming. Laarhoven and E. van Laarhoven. Toint Nicholas I. [29] J.uk/cuterwww/sifdec/doc. October 1993. Softw. Burton S. October 1993. L.rl. Mor´. http://www. Norwell.htm. [27] lpsolve.ac. Springer.net/5. [25] P. [24] P. Mixed h2/h? control: a convex optimization approach. http://lpsolve. Genetic Algorithms in Search.ps. Jul 1991. 66 . Algorithm 566: Fortran subroue tines for testing unconstrained optimization software [c5].gz. Stanford University. Samir Agrawal. 1993. http://hsl. Mps ﬁle format. 1981. Fleming. Dominique Orban. 1981. Simulated annealing: theory and applications. Genetic algorithms for multiobjective optimization: Formulationdiscussion and generalization. Springer. 36(7):824–837. Garbow.A.com/. The general algebraic modeling system. Springer. and Kenneth E. and T Meyarivan. [23] Vandenberghe L. J. [e4]. Fonseca and Peter J. Internal Report.ac.. [16] gams. A constrained and unconstrained testing environment. Kluwer Academic Publishers. 1981. 1988. IEEE Transactions on. ACM Trans. USA.lania. J. [31] Philippe L. revisited. and S. H.html. Boyd. McGrawHill. 30:59–78.M. Goldberg. Netherlands : Centrum voor Wiskunde en Informatica. Convex Analysis and Minimization e Algorithms II: Advanced Theory and Bundle Methods. [20] hsl.sourceforge. M.5/mpsformat. http://hsl. Advanced Linear Programming: Computation and Practice. 1994 (submitted to SIAM Review). Gould. Wright Jorge Nocedal. [28] Zbigniew Michalewicz. San Francisco. [22] P. Optimization & Machine Learning.com. A lonesome sif decoder. Amrit Pratap. Convex Analysis and Minimization e Algorithms I: Fundamentals.P.mx/%7Eccoello/fonseca93. AddisonWesley. 1987.uk/cuterwww/. 1989. [17] David E. CA. editors. Numerical Optimization. http://www. http://www. Math. Khargonekar and M.gams. [30] B.ac. 2000. Automatic Control. 7(1):136–140.ps. Theoretical and Computational Aspects of Simulated Annealing.[14] Kalyanmoy Deb.gz.mx/%7Eccoello/deb00. Morgan Kaufmann Publishers Inc. In Proceedings of the 5th International Conference on Genetic Algorithms. Amsterdam. A view of linesearches. Springer. MA.
maths. Multiobjective optimization using nondominated sorting in genetic algorithms.org/wiki/AMPL_ [40] Wikipedia. http://en. Kalman ﬁltering and riccati equations for descriptor systems. [43] Patrick Siarry Yann Collette. http://www. Nikoukhah. Mps (format). 67 . [36] Jun Ma Robert Fourer and Kipp Martin. http://www. [41] Wikipedia. Simulated annealing. Srinivas and Kalyanmoy Deb.numerical. programming_language.org/. http://en. 1999. [35] optimizationservices.org/wiki/General_ Algebraic_Modeling_System.[32] Philippe L.lania. Evolutionary Computation. Mathematics of Computation.uk/cute/mastsif. [42] Wikipedia.optimizationservices. Willsky.rl. Quadprog. Levy. [38] Berwin A Turlach.C. 2:221–248. [39] Wikipedia. 37(9):1325–1342.org/wiki/Simulated_annealing. General algebraic modeling system. A. [33] R.org. http://en. Optimisation Multiobjectif.wikipedia. (quadratic programming routines). http://www. Collection Algorithmes. Optimization services. Toint Nicholas I.M.uwa. and B. Ampl (programming language). Updating quasinewton matrices with limited storage. Computational Optimization and Applications.ac.wikipedia. The cuter test problem set. Automatic Control. Sep 1992. edu.org/wiki/MPS_(format). 1994. [37] N. http://en.S. Dominique Orban. 2008. Gould.html.html.wikipedia. 1980. Osil: An instance language for optimization.ps. Eyrolles. 35(151):773–782. http://www.au/~berwin/software/quadprog. IEEE Transactions on. [34] Jorge Nocedal.wikipedia.gz.mx/ %7Eccoello/deb95.