This action might not be possible to undo. Are you sure you want to continue?
MACHINE PERCEPTION
CZECH TECHNICAL
UNIVERSITY
R
E
S
E
A
R
C
H
R
E
P
O
R
T
I
S
S
N
1
2
1
3

2
3
6
5
Statistical Pattern Recognition
Toolbox for Matlab
User’s guide
Vojtˇech Franc and V´aclav Hlav´ aˇc
¦xfrancv,hlavac¦@cmp.felk.cvut.cz
CTU–CMP–2004–08
June 24, 2004
Available at
ftp://cmp.felk.cvut.cz/pub/cmp/articles/FrancTR200408.pdf
The authors were supported by the European Union under project
IST200135454 ECVision, the project IST200132184 ActIPret and
by the Czech Grant Agency under project GACR 102/03/0440 and
by the Austrian Ministry of Education under project CONEX GZ
45.535.
Research Reports of CMP, Czech Technical University in Prague, No. 8, 2004
Published by
Center for Machine Perception, Department of Cybernetics
Faculty of Electrical Engineering, Czech Technical University
Technick´a 2, 166 27 Prague 6, Czech Republic
fax +420 2 2435 7385, phone +420 2 2435 7637, www: http://cmp.felk.cvut.cz
Statistical Pattern Recognition Toolbox for Matlab
Vojtˇech Franc and V´aclav Hlav´ aˇc
June 24, 2004
Contents
1 Introduction 3
1.1 What is the Statistical Pattern Recognition Toolbox and how it has been
developed? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 The STPRtool purpose and its potential users . . . . . . . . . . . . . . 3
1.3 A digest of the implemented methods . . . . . . . . . . . . . . . . . . . 4
1.4 Relevant Matlab toolboxes of others . . . . . . . . . . . . . . . . . . . . 5
1.5 STPRtool document organization . . . . . . . . . . . . . . . . . . . . . 5
1.6 How to contribute to STPRtool? . . . . . . . . . . . . . . . . . . . . . 5
1.7 Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2 Linear Discriminant Function 8
2.1 Linear classiﬁer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.2 Linear separation of ﬁnite sets of vectors . . . . . . . . . . . . . . . . . 10
2.2.1 Perceptron algorithm . . . . . . . . . . . . . . . . . . . . . . . . 11
2.2.2 Kozinec’s algorithm . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.2.3 Multiclass linear classiﬁer . . . . . . . . . . . . . . . . . . . . . 14
2.3 Fisher Linear Discriminant . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.4 Generalized Anderson’s task . . . . . . . . . . . . . . . . . . . . . . . . 17
2.4.1 Original Anderson’s algorithm . . . . . . . . . . . . . . . . . . . 19
2.4.2 General algorithm framework . . . . . . . . . . . . . . . . . . . 21
2.4.3 εsolution using the Kozinec’s algorithm . . . . . . . . . . . . . 21
2.4.4 Generalized gradient optimization . . . . . . . . . . . . . . . . . 22
3 Feature extraction 25
3.1 Principal Component Analysis . . . . . . . . . . . . . . . . . . . . . . . 26
3.2 Linear Discriminant Analysis . . . . . . . . . . . . . . . . . . . . . . . 29
3.3 Kernel Principal Component Analysis . . . . . . . . . . . . . . . . . . . 31
3.4 Greedy kernel PCA algorithm . . . . . . . . . . . . . . . . . . . . . . . 32
3.5 Generalized Discriminant Analysis . . . . . . . . . . . . . . . . . . . . . 33
3.6 Combining feature extraction and linear classiﬁer . . . . . . . . . . . . 34
1
4 Density estimation and clustering 41
4.1 Gaussian distribution and Gaussian mixture model . . . . . . . . . . . 41
4.2 MaximumLikelihood estimation of GMM . . . . . . . . . . . . . . . . 43
4.2.1 Complete data . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
4.2.2 Incomplete data . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
4.3 Minimax estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
4.4 Probabilistic output of classiﬁer . . . . . . . . . . . . . . . . . . . . . . 48
4.5 Kmeans clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
5 Support vector and other kernel machines 57
5.1 Support vector classiﬁers . . . . . . . . . . . . . . . . . . . . . . . . . . 57
5.2 Binary Support Vector Machines . . . . . . . . . . . . . . . . . . . . . 58
5.3 OneAgainstAll decomposition for SVM . . . . . . . . . . . . . . . . . 63
5.4 OneAgainstOne decomposition for SVM . . . . . . . . . . . . . . . . . 64
5.5 Multiclass BSVM formulation . . . . . . . . . . . . . . . . . . . . . . . 66
5.6 Kernel Fisher Discriminant . . . . . . . . . . . . . . . . . . . . . . . . . 68
5.7 Preimage problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
5.8 Reduced set method . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
6 Miscellaneous 76
6.1 Data sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
6.2 Visualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
6.3 Bayesian classiﬁer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
6.4 Quadratic classiﬁer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
6.5 Knearest neighbors classiﬁer . . . . . . . . . . . . . . . . . . . . . . . 82
7 Examples of applications 85
7.1 Optical Character Recognition system . . . . . . . . . . . . . . . . . . 85
7.2 Image denoising . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
8 Conclusions and future planes 94
2
Chapter 1
Introduction
1.1 What is the Statistical Pattern Recognition Tool
box and how it has been developed?
The Statistical Pattern Recognition Toolbox (abbreviated STPRtool) is a collection of
pattern recognition (PR) methods implemented in Matlab.
The core of the STPRtool is comprised of statistical PR algorithms described in
the monograph Schlesinger, M.I., Hlav´ aˇc, V: Ten Lectures on Statistical and Structural
Pattern Recognition, Kluwer Academic Publishers, 2002 [26]. The STPRtool has been
developed since 1999. The ﬁrst version was a result of a diploma project of Vojtˇech
Franc [8]. Its author became a PhD student in CMP and has been extending the
STPRtool since. Currently, the STPRtool contains much wider collection of algorithms.
The STPRtool is available at
http://cmp.felk.cvut.cz/cmp/cmp software.html.
We tried to create concise and complete documentation of the STPRtool. This
document and associated help ﬁles in .html should give all information and parameters
the user needs when she/he likes to call appropriate method (typically a Matlab m
ﬁle). The intention was not to write another textbook. If the user does not know
the method then she/he will likely have to read the description of the method in a
referenced original paper or in a textbook.
1.2 The STPRtool purpose and its potential users
• The principal purpose of STPRtool is to provide basic statistical pattern recog
nition methods to (a) researchers, (b) teachers, and (c) students. The STPRtool
is likely to help both in research and teaching. It can be found useful by practi
tioners who design PR applications too.
3
• The STPRtool oﬀers a collection of the basic and the stateofthe art statistical
PR methods.
• The STPRtool contains demonstration programs, working examples and visual
ization tools which help to understand the PR methods. These teaching aids are
intended for students of the PR courses and their teachers who can easily create
examples and incorporate them into their lectures.
• The STPRtool is also intended to serve a designer of a new PR method by
providing tools for evaluation and comparison of PR methods. Many standard
and stateofthe art methods have been implemented in STPRtool.
• The STPRtool is open to contributions by others. If you develop a method
which you consider useful for the community then contact us. We are willing to
incorporate the method into the toolbox. We intend to moderate this process to
keep the toolbox consistent.
1.3 A digest of the implemented methods
This section should give the reader a quick overview of the methods implemented in
STPRtool.
• Analysis of linear discriminant function: Perceptron algorithm and multi
class modiﬁcation. Kozinec’s algorithm. Fisher Linear Discriminant. A collection
of known algorithms solving the Generalized Anderson’s Task.
• Feature extraction: Linear Discriminant Analysis. Principal Component Anal
ysis (PCA). Kernel PCA. Greedy Kernel PCA. Generalized Discriminant Analy
sis.
• Probability distribution estimation and clustering: Gaussian Mixture
Models. ExpectationMaximization algorithm. Minimax probability estimation.
Kmeans clustering.
• Support Vector and other Kernel Machines: Sequential Minimal Opti
mizer (SMO). Matlab Optimization toolbox based algorithms. Interface to the
SV M
light
software. Decomposition approaches to train the Multiclass SVM clas
siﬁers. Multiclass BSVM formulation trained by Kozinec’s algorithm, Mitchell
DemyanovMolozenov algorithm and Nearest Point Algorithm. Kernel Fisher
Discriminant.
4
1.4 Relevant Matlab toolboxes of others
Some researchers in statistical PR provide code of their methods. The Internet oﬀers
many software resources for Matlab. However, there are not many compact and well
documented packages comprised of PR methods. We found two useful packages which
are brieﬂy introduced below.
NETLAB is focused to the Neural Networks applied to PR problems. The classical
PR methods are included as well. This Matlab toolbox is closely related to the
popular PR textbook [3]. The documentation published in book [21] contains de
tailed description of the implemented methods and many working examples. The
toolbox includes: Probabilistic Principal Component Analysis, Generative Topo
graphic Mapping, methods based on the Bayesian inference, Gaussian processes,
Radial Basis Functions (RBF) networks and many others.
The toolbox can be downloaded from
http://www.ncrg.aston.ac.uk/netlab/.
PRTools is a Matlab Toolbox for Pattern Recognition [7]. Its implementation is
based on the object oriented programming principles supported by the Matlab
language. The toolbox contains a wide range of PR methods including: analysis
of linear and nonlinear classiﬁers, feature extraction, feature selection, methods
for combining classiﬁers and methods for clustering.
The toolbox can be downloaded from
http://www.ph.tn.tudelft.nl/prtools/.
1.5 STPRtool document organization
The document is divided into chapters describing the implemented methods. The last
chapter is devoted to more complex examples. Most chapters begin with deﬁnition of
the models to be analyzed (e.g., linear classiﬁer, probability densities, etc.) and the list
of the implemented methods. Individual sections have the following ﬁxed structure: (i)
deﬁnition of the problem to be solved, (ii) brief description and list of the implemented
methods, (iii) reference to the literature where the problem is treaded in details and
(iv) an example demonstrating how to solve the problem using the STPRtool.
1.6 How to contribute to STPRtool?
If you have an implementation which is consistent with the STPRtool spirit and you
want to put it into public domain through the STPRtool channel then send an email
to xfrancv@cmp.felk.cvut.cz.
5
Please prepare your code according to the pattern you ﬁnd in STPRtool implemen
tations. Provide piece of documentation too which will allow us to incorporate it to
STPRtool easily.
Of course, your method will be bound to your name in STPRtool.
1.7 Acknowledgements
First, we like to thank to Prof. M.I. Schlesinger from the Ukrainian Academy of
Sciences, Kiev, Ukraine who showed us the beauty of pattern recognition and gave us
many valuable advices.
Second, we like to thank for discussions to our colleagues in the Center for Machine
Perception, the Czech Technical University in Prague, our visitors, other STPRtool
users and students who used the toolbox in their laboratory exercises.
Third, thanks go to several Czech, EU and industrial grants which indirectly con
tributed to STPRtool development.
Forth, the STPRtool was rewritten and its documentation radically improved in
May and June 2004 with the ﬁnancial help of ECVision European Research Network
on Cognitive Computer Vision Systems (IST200135454) lead by D. Vernon.
6
Notation
Uppercase bold letters denote matrices, for instance A. Vectors are implicitly column
vectors. Vectors are denoted by lowercase bold italic letters, for instance x. The
concatenation of column vectors w ∈ R
n
, z ∈ R
m
is denoted as v = [w; z] where the
column vector v is (n + m)dimensional.
¸x x
) Dot product between vectors x and x
.
Σ Covariance matrix.
µ Mean vector (mathematical expectation).
S Scatter matrix.
A ⊆ R
n
ndimensional input space (space of observations).
¸ Set of c hidden states – class labels ¸ = ¦1, . . . , c¦.
T Feature space.
φ: A → T Mapping from input space A to the feature space T.
k: A A →R Kernel function (Mercer kernel).
T
XY
Labeled training set (more precisely multiset) T
XY
=
¦(x
1
, y
1
), . . . , (x
l
, y
l
)¦.
T
X
Unlabeled training set T
X
= ¦x
1
, . . . , x
l
¦.
q: A → ¸ Classiﬁcation rule.
f
y
: A →R Discriminant function associated with the class y ∈ ¸.
f: A →R Discriminant function of the binary classiﬁer.
[ [ Cardinality of a set.
  Euclidean norm.
det(()) Matrix determinant.
δ(i, j) Kronecker delta δ(i, j) = 1 for i = j and δ(i, j) = 0
otherwise.
_
Logical or.
7
Chapter 2
Linear Discriminant Function
The linear classiﬁcation rule q: A ⊆ R
n
→ ¸ = ¦1, 2, . . . , c¦ is composed of a set of
discriminant functions
f
y
(x) = ¸w
y
x) + b
y
, ∀y ∈ ¸ ,
which are linear with respect to both the input vector x ∈ R
n
and their parameter
vectors w ∈ R
n
. The scalars b
y
, ∀y ∈ ¸ introduce bias to the discriminant functions.
The input vector x ∈ R
n
is assigned to the class y ∈ ¸ its corresponding discriminant
function f
y
attains maximal value
y = argmax
y
∈Y
f
y
(x) = argmax
y
∈Y
(¸w
y
x) + b
y
) . (2.1)
In the particular binary case ¸ = ¦1, 2¦, the linear classiﬁer is represented by a
single discriminant function
f(x) = ¸w x) + b ,
given by the parameter vector w ∈ R
n
and bias b ∈ R. The input vector x ∈ R
n
is
assigned to class y ∈ ¦1, 2¦ as follows
q(x) =
_
1 if f(x) = ¸w x) + b ≥ 0 ,
2 if f(x) = ¸w x) + b < 0 .
(2.2)
The datatype used to describe the linear classiﬁer and the implementation of the
classiﬁer is described in Section 2.1. Following sections describe supervised learning
methods which determine the parameters of the linear classiﬁer based on available
knowledge. The implemented learning methods can be distinguished according to the
type of the training data:
I. The problem is described by a ﬁnite set T = ¦(x
1
, y
1
), . . . , (x
l
, y
l
)¦ containing pairs
of observations x
i
∈ R
n
and corresponding class labels y
i
∈ ¸. The methods
using this type of training data is described in Section 2.2 and Section 2.3.
8
II. The parameters of class conditional distributions are (completely or partially)
known. The Anderson’s task and its generalization are representatives of such
methods (see Section 2.4).
The summary of implemented methods is given in Table 2.1.
References: The linear discriminant function is treated throughout for instance in
books [26, 6].
Table 2.1: Implemented methods: Linear discriminant function
linclass Linear classiﬁer (linear machine).
andrerr Classiﬁcation error of the Generalized Anderson’s
task.
androrig Original method to solve the AndersonBahadur’s
task.
eanders Epsilonsolution of the Generalized Anderson’s task.
ganders Solves the Generalized Anderson’s task.
ggradander Gradient method to solve the Generalized Anderson’s
task.
ekozinec Kozinec’s algorithm for epsoptimal separating hyper
plane.
mperceptr Perceptron algorithm to train multiclass linear ma
chine.
perceptron Perceptron algorithm to train binary linear classiﬁer.
fld Fisher Linear Discriminant.
fldqp Fisher Linear Discriminant using Quadratic Program
ming.
demo linclass Demo on the algorithms learning linear classiﬁers.
demo anderson Demo on Generalized Anderson’s task.
2.1 Linear classiﬁer
The STPRtool uses a speciﬁc structure array to describe the binary (see Table 2.2)
and the multiclass (see Table 2.3) linear classiﬁer. The implementation of the linear
classiﬁer itself provides function linclass.
Example: Linear classiﬁer
The example shows training of the Fisher Linear Discriminant which is the classical
example of the linear classiﬁer. The Riply’s data set riply trn.mat is used for training.
The resulting linear classiﬁer is then evaluated on the testing data riply tst.mat.
9
Table 2.2: Datatype used to describe binary linear classiﬁer.
Binary linear classiﬁer (structure array):
.W [n 1] The normal vector w ∈ R
n
of the separating hyper
plane f(x) = ¸w x) + b.
.b [1 1] Bias b ∈ R of the hyperplane.
.fun = ’linclass’ Identiﬁes function associated with this data type.
Table 2.3: Datatype used to describe multiclass linear classiﬁer.
Multiclass linear classiﬁer (structure array):
.W [n c] Matrix of parameter vectors w
y
∈ R
n
, y = 1, . . . , c of
the linear discriminant functions f
y
(x) = ¸w
y
x) +b.
.b [c 1] Parameters b
y
∈ R, y = 1, . . . , c.
.fun = ’linclass’ Identiﬁes function associated with this data type.
trn = load(’riply_trn’); % load training data
tst = load(’riply_tst’); % load testing data
model = fld( trn ); % train FLD classifier
ypred = linclass( tst.X, model ); % classify testing data
cerror( ypred, tst.y ) % evaluate testing error
ans =
0.1080
2.2 Linear separation of ﬁnite sets of vectors
The input training data T = ¦(x
1
, y
1
), . . . , (x
l
, y
l
)¦ consists of pairs of observations
x
i
∈ R
n
and corresponding class labels y
i
∈ ¸ = ¦1, . . . , c¦. The implemented methods
solve the task of (i) training separating hyperplane for binary case c = 2, (ii) training
εoptimal hyperplane and (iii) training the multiclass linear classiﬁer. The deﬁnitions
of these tasks are given below.
The problem of training the binary (c = 2) linear classiﬁer (2.2) with zero training
error is equivalent to ﬁnding the hyperplane H = ¦x: ¸w x) +b = 0¦ which separates
the training vectors of the ﬁrst y = 1 and the second y = 2 class. The problem is
formally deﬁned as solving the set of linear inequalities
¸w x
i
) + b ≥ 0 , y
i
= 1 ,
¸w x
i
) + b < 0 , y
i
= 2 .
(2.3)
with respect to the vector w ∈ R
n
and the scalar b ∈ R. If the inequalities (2.3) have a
solution then the training data T is linearly separable. The Perceptron (Section 2.2.1)
10
and Kozinec’s (Section 2.2.2) algorithm are useful to train the binary linear classiﬁers
from the linearly separable data.
If the training data is linearly separable then the optimal separating hyperplane
can be deﬁned the solution of the following task
(w
∗
, b
∗
) = argmax
w,b
m(w, b)
= argmax
w,b
min
_
min
i∈I
1
¸w x
i
) + b
w
, min
i∈I
2
−
¸w x
i
) + b
w
_
,
(2.4)
where 1
1
= ¦i: y
i
= 1¦ and 1
2
= ¦i: y
i
= 2¦ are sets of indices. The optimal separating
hyperplane H
∗
= ¦x ∈ R
n
: ¸w
∗
x) + b
∗
= 0¦ separates training data T with maxi
mal margin m(w
∗
, b
∗
). This task is equivalent to training the linear Support Vector
Machines (see Section 5). The optimal separating hyperplane cannot be found exactly
except special cases. Therefore the numerical algorithms seeking the approximate so
lution are applied instead. The εoptimal solution is deﬁned as the vector w and scalar
b such that the inequality
m(w
∗
, b
∗
) −m(w, b) ≤ ε , (2.5)
holds. The parameter ε ≥ 0 deﬁnes closeness to the optimal solution in terms of the
margin. The εoptimal solution can be found by Kozinec’s algorithm (Section 2.2.2).
The problem of training the multiclass linear classiﬁer c > 2 with zero training
error is formally stated as the problem of solving the set of linear inequalities
¸w
y
i
x
i
) + b
y
i
> ¸w
y
x
i
) + b
y
, i = 1, . . . , l , y
i
,= y , (2.6)
with respect to the vectors w
y
∈ R
n
, y ∈ ¸ and scalars b
y
∈ R, y ∈ ¸. The task (2.6)
for linearly separable training data T can be solved by the modiﬁed Perceptron (Sec
tion 2.2.3) or Kozinec’s algorithm. An interactive demo on the algorithms separating
the ﬁnite sets of vectors by a hyperplane is implemented in demo linclass.
2.2.1 Perceptron algorithm
The input is a set T = ¦(x
1
, y
1
), . . . , (x
l
, y
l
)¦ of binary labeled y
i
∈ ¦1, 2¦ training
vectors x
i
∈ R
n
. The problem of training the separating hyperplane (2.3) can be
formally rewritten to a simpler form
¸v z
i
) > 0 , i = 1, . . . , l , (2.7)
where the vector v ∈ R
n+1
is constructed as
v = [w; b] , (2.8)
11
and transformed training data Z = ¦z
1
, . . . , z
l
¦ are deﬁned as
z
i
=
_
[x
i
; 1] if y
i
= 1 ,
−[x
i
; 1] if y
i
= 2 .
(2.9)
The problem of solving (2.7) with respect to the unknown vector v ∈ R
n+1
is equivalent
to the original task (2.3). The parameters (w, b) of the linear classiﬁer are obtained
from the found vector v by inverting the transformation (2.8).
The Perceptron algorithm is an iterative procedure which builds a series of vectors
v
(0)
, v
(1)
, . . . , v
(t)
until the set of inequalities (2.7) is satisﬁed. The initial vector v
(0)
can
be set arbitrarily (usually v = 0). The Novikoﬀ theorem ensures that the Perceptron
algorithm stops after ﬁnite number of iterations t if the training data are linearly
separable. Perceptron algorithm is implemented in function perceptron.
References: Analysis of the Perceptron algorithm including the Novikoﬀ’s proof of
convergence can be found in Chapter 5 of the book [26]. The Perceptron from the
neural network perspective is described in the book [3].
Example: Training binary linear classiﬁer with Perceptron
The example shows the application of the Perceptron algorithm to ﬁnd the binary
linear classiﬁer for synthetically generated 2dimensional training data. The generated
training data contain 50 labeled vectors which are linearly separable with margin 1.
The found classiﬁer is visualized as a separating hyperplane (line in this case) H =
¦x ∈ R
2
: ¸w x) + b = 0¦. See Figure 2.1.
data = genlsdata( 2, 50, 1); % generate training data
model = perceptron( data ); % call perceptron
figure;
ppatterns( data ); % plot training data
pline( model ); % plot separating hyperplane
2.2.2 Kozinec’s algorithm
The input is a set T = ¦(x
1
, y
1
), . . . , (x
l
, y
l
)¦ of binary labeled y
i
∈ ¦1, 2¦ training
vectors x
i
∈ R
n
. The Kozinec’s algorithm builds a series of vectors w
(0)
1
, w
(1)
1
, . . . , w
(t)
1
and w
(0)
2
, w
(1)
2
, . . . , w
(t)
2
which converge to the vector w
∗
1
and w
∗
2
respectively. The
vectors w
∗
1
and w
∗
2
are the solution of the following task
w
∗
1
, w
∗
2
= argmin
w
1
∈X
1
,w
2
∈X
2
w
1
−w
2
 ,
where A
1
stands for the convex hull of the training vectors of the ﬁrst class A
1
=
¦x
i
: y
i
= 1¦ and A
2
for the convex hull of the second class likewise. The vector w
∗
=
12
−55 −50 −45 −40 −35 −30
−55
−50
−45
−40
−35
−30
Figure 2.1: Linear classiﬁer trained by the Perceptron algorithm
w
∗
1
−w
∗
2
and the bias b
∗
=
1
2
(w
∗
2

2
−w
∗
1

2
) determine the optimal hyperplane (2.4)
separating the training data with the maximal margin.
The Kozinec’s algorithm is proven to converge to the vectors w
∗
1
and w
∗
2
in inﬁnite
number of iterations t = ∞. If the εoptimal optimality stopping condition is used
then the Kozinec’s algorithm converges in ﬁnite number of iterations. The Kozinec’s
algorithm can be also used to solve a simpler problem of ﬁnding the separating hyper
plane (2.3). Therefore the following two stopping conditions are implemented:
I. The separating hyperplane (2.3) is sought for (ε < 0). The Kozinec’s algorithm is
proven to converge in a ﬁnite number of iterations if the separating hyperplane
exists.
II. The εoptimal hyperplane (2.5) is sought for (ε ≥ 0). Notice that setting ε = 0
force the algorithm to seek the optimal hyperplane which is generally ensured to
be found in an inﬁnite number of iterations t = ∞.
The Kozinec’s algorithm which trains the binary linear classiﬁer is implemented in the
function ekozinec.
References: Analysis of the Kozinec’s algorithm including the proof of convergence
can be found in Chapter 5 of the book [26].
Example: Training εoptimal hyperplane by the Kozinec’s algorithm
The example shows application of the Kozinec’s algorithm to ﬁnd the (ε = 0.01)
optimal hyperplane for synthetically generated 2dimensional training data. The gen
erated training data contains 50 labeled vectors which are linearly separable with
13
margin 1. The found classiﬁer is visualized (Figure 2.2) as a separating hyperplane
H = ¦x ∈ R
2
: ¸w x) +b = 0¦ (straight line in this case). It could be veriﬁed that the
found hyperplane has margin model.margin which satisﬁes the εoptimality condition.
data = genlsdata(2,50,1); % generate data
% run ekozinec
model = ekozinec(data, struct(’eps’,0.01));
figure;
ppatterns(data); % plot data
pline(model); % plot found hyperplane
−10 −8 −6 −4 −2 0 2 4 6 8 10
−10
−8
−6
−4
−2
0
2
4
6
8
Figure 2.2: εoptimal hyperplane trained with the Kozinec’s algorithm.
2.2.3 Multiclass linear classiﬁer
The input training data T = ¦(x
1
, y
1
), . . . , (x
l
, y
l
)¦ consists of pairs of observations
x
i
∈ R
n
and corresponding class labels y
i
∈ ¸ = ¦1, . . . , c¦. The task is to design
the linear classiﬁer (2.1) which classiﬁes the training data T without error. The prob
lem of training (2.1) is equivalent to solving the set of linear inequalities (2.6). The
inequalities (2.6) can be formally transformed to a simpler set
¸v z
y
i
) > 0 , i = 1, . . . , l , y = ¸ ¸ ¦y
i
¦ , (2.10)
where the vector v ∈ R
(n+1)c
contains the originally optimized parameters
v = [w
1
; b
1
; w
2
; b
2
; . . . ; w
c
; b
c
] .
14
The set of vectors z
y
i
∈ R
(n+1)c
, i = 1, . . . , l, y ∈ ¸ ¸ ¦y
i
¦ is created from the training
set T such that
z
y
i
(j) =
⎧
⎨
⎩
[x
i
; 1] , for j = y
i
,
−[x
i
; 1] , for j = y ,
0 , otherwise.
(2.11)
where z
y
i
(j) stands for the jth slot between coordinates (n+1)(j −1)+1 and (n+1)j.
The described transformation is known as the Kesler’s construction.
The transformed task (2.10) can be solved by the Perceptron algorithm. The simple
form of the Perceptron updating rule allows to implement the transformation implicitly
without mapping the training data T into (n + 1)cdimensional space. The described
method is implemented in function mperceptron.
References: The Kesler’s construction is described in books [26, 6].
Example: Training multiclass linear classiﬁer by the Perceptron
The example shows application of the Perceptron rule to train the multiclass linear
classiﬁer for the synthetically generated 2dimensional training data pentagon.mat. A
user’s own data can be generated interactively using function createdata(’finite’,10)
(number 10 stands for number of classes). The found classiﬁer is visualized as its sep
arating surface which is known to be piecewise linear and the class regions should be
convex (see Figure 2.3).
data = load(’pentagon’); % load training data
model = mperceptron(data); % run training algorithm
figure;
ppatterns(data); % plot training data
pboundary(model); % plot decision boundary
2.3 Fisher Linear Discriminant
The input is a set T = ¦(x
1
, y
1
), . . . , (x
l
, y
l
)¦ of binary labeled y
i
∈ ¦1, 2¦ training
vectors x
i
∈ R
n
. Let 1
y
= ¦i: y
i
= y¦, y ∈ ¦1, 2¦ be sets of indices of training
vectors belonging to the ﬁrst y = 1 and the second y = 2 class, respectively. The class
separability in a direction w ∈ R
n
is deﬁned as
F(w) =
¸w S
B
w)
¸w S
W
w)
, (2.12)
where S
B
is the betweenclass scatter matrix
S
B
= (µ
1
−µ
2
)(µ
1
−µ
2
)
T
, µ
y
=
1
[1
y
[
i∈Iy
x
i
, y ∈ ¦1, 2¦ ,
15
−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1
−1
−0.8
−0.6
−0.4
−0.2
0
0.2
0.4
0.6
0.8
1
Figure 2.3: Multiclass linear classiﬁer trained by the Perceptron algorithm.
and S
W
is the within class scatter matrix deﬁned as
S
W
= S
1
+S
2
, S
y
=
i∈Yy
(x
i
−µ
y
)(x
i
−µ
y
)
T
, y ∈ ¦1, 2¦ .
In the case of the Fisher Linear Discriminant (FLD), the parameter vector w of the
linear discriminant function f(x) = ¸w x) + b is determined to maximize the class
separability criterion (2.12)
w = argmax
w
F(w
) = argmax
w
¸w
S
B
w
)
¸w
S
W
w
)
. (2.13)
The bias b of the linear rule must be determined based on another principle. The
STPRtool implementation, for instance, computes such b that the equality
¸w µ
1
) + b = −(¸w µ
2
) + b) ,
holds. The STPRtool contains two implementations of FLD:
I. The classical solution of the problem (2.13) using the matrix inversion
w = S
W
−1
(µ
1
−µ
2
) .
This case is implemented in the function fld.
16
II. The problem (2.13) can be reformulated as the quadratic programming (QP) task
w = argmin
w
¸w
S
W
w
) ,
subject to
¸w
(µ
1
−µ
2
)) = 2 .
The QP solver quadprog of the Matlab Optimization Toolbox is used in the
toolbox implementation. This method can be useful when the matrix inversion
is hard to compute. The method is implemented in the function fldqp.
References: The Fisher Linear Discriminant is described in Chapter 3 of the book [6]
or in the book [3].
Example: Fisher Linear Discriminant
The binary linear classiﬁer based on the Fisher Linear Discriminant is trained on
the Riply’s data set riply trn.mat. The QP formulation fldqp of FLD is used.
However, the function fldqp can be replaced by the standard approach implemented
in the function fld which would yield the same solution. The found linear classiﬁer is
visualized (see Figure 2.4) and evaluated on the testing data riply tst.mat.
trn = load(’riply_trn’); % load training data
model = fldqp(trn); % compute FLD
figure;
ppatterns(trn); pline(model); % plot data and solution
tst = load(’riply_tst’); % load testing data
ypred = linclass(tst.X,model); % classify testing data
cerror(ypred,tst.y) % compute testing error
ans =
0.1080
2.4 Generalized Anderson’s task
The classiﬁed object is described by the vector of observations x ∈ A ⊆ R
n
and a
binary hidden state y ∈ ¦1, 2¦. The class conditional distributions p
XY
(x[y), y ∈ ¦1, 2¦
are known to be multivariate Gaussian distributions. The parameters (µ
1
, Σ
1
) and
(µ
2
, Σ
2
) of these class distributions are unknown. However, it is known that the
parameters (µ
1
, Σ
1
) belong to a certain ﬁnite set of parameters ¦(µ
i
, Σ
i
): i ∈ 1
1
¦.
Similarly (µ
2
, Σ
2
) belong to a ﬁnite set ¦(µ
i
, Σ
i
): i ∈ 1
2
¦. Let q: A ⊆ R
n
→ ¦1, 2¦
17
−1.5 −1 −0.5 0 0.5 1
−0.2
0
0.2
0.4
0.6
0.8
1
1.2
Figure 2.4: Linear classiﬁer based on the Fisher Linear Discriminant.
be a binary linear classiﬁer (2.2) with discriminant function f(x) = ¸w x) + b. The
probability of misclassiﬁcation is deﬁned as
Err(w, b) = max
i∈I
1
∪I
2
ε(w, b, µ
i
, Σ
i
) ,
where ε(w, b, µ
i
, Σ
i
) is probability that the Gaussian random vector x with mean
vector µ
i
and the covariance matrix Σ
i
satisﬁes q(x) = 1 for i ∈ 1
2
or q(x) = 2 for
i ∈ 1
1
. In other words, it is the probability that the vector x will be misclassiﬁed by
the linear rule q.
The Generalized Anderson’s task (GAT) is to ﬁnd the parameters (w
∗
, b
∗
) of the
linear classiﬁer
q(x) =
_
1 , for f(x) = ¸w
∗
x) + b
∗
≥ 0 ,
2 , for f(x) = ¸w
∗
x) + b
∗
< 0 ,
such that the error Err(w
∗
, b
∗
) is minimal
(w
∗
, b
∗
) = argmin
w,b
Err(w, b) = argmin
w,b
max
i∈I
1
∪I
2
ε(w, b, µ
i
, Σ
i
) . (2.14)
The original Anderson’s task is a special case of (2.14) when [1
1
[ = 1 and [1
2
[ = 1.
The probability ε(w, b, µ
i
, Σ
i
) is proportional to the reciprocal of the Mahalanobis
distance r
i
between the (µ
i
, Σ
i
) and the nearest vector of the separating hyperplane
H = ¦x ∈ R
n
: ¸w x) + b = 0¦, i.e.,
r
i
= min
x∈H
¸(µ
i
−x) (Σ
i
)
−1
(µ
i
−x)) =
¸w µ
i
) + b
_
¸w Σ
i
w)
.
18
The exact relation between the probability ε(w, b, µ
i
, Σ
i
) and the corresponding Ma
halanobis distance r
i
is given by the integral
ε(w, b, µ
i
, Σ
i
) =
_
∞
r
i
1
√
2π
e
−
1
2
t
2
dt . (2.15)
The optimization problem (2.14) can be equivalently rewritten as
(w
∗
, b
∗
) = argmax
w,b
F(w, b) = argmax
w,b
min
i∈I
1
∪I
2
¸w µ
i
) + b
_
¸w Σ
i
w)
.
which is more suitable for optimization. The objective function F(w, b) is proven to
be convex in the region where the probability of misclassiﬁcation Err(w, b) is less than
0.5. However, the objective function F(w, b) is not diﬀerentiable.
The STPRtool contains implementations of the algorithm solving the original An
derson’s task as well as implementations of three diﬀerent approaches to solve the
Generalized Anderson’s task which are described bellow. An interactive demo on the
algorithms solving the Generalized Anderson’s task is implemented in demo anderson.
References: The original Anderson’s task was published in [1]. A detailed description
of the Generalized Anderson’s task and all the methods implemented in the STPRtool
is given in book [26].
2.4.1 Original Anderson’s algorithm
The algorithm is implemented in the function androrig. This algorithm solves the
original Anderson’s task deﬁned for two Gaussians only [1
1
[ = [1
2
[ = 1. In this case,
the optimal solution vector w ∈ R
n
is obtained by solving the following problem
w = ((1 −λ)Σ
1
+ λΣ
2
)
−1
(µ
1
−µ
2
) ,
1 −λ
λ
=
¸
¸w Σ
2
w)
¸w Σ
1
w)
,
where the scalar 0 < λ < 1 is unknown. The problem is solved by an iterative algorithm
which stops while the condition
¸
¸
¸
¸
γ −
1 −λ
λ
¸
¸
¸
¸
≤ ε , γ =
¸
¸w Σ
2
w)
¸w Σ
1
w)
,
is satisﬁed. The parameter ε deﬁnes the closeness to the optimal solution.
Example: Solving the original Anderson’s task
19
The original Anderson’s task is solved for the Riply’s data set riply trn.mat. The
parameters (µ
1
, Σ
1
) and (µ
2
, Σ
2
) of the Gaussian distribution of the ﬁrst and the
second class are obtained by the MaximumLikelihood estimation. The found linear
classiﬁer is visualized and the Gaussian distributions (see Figure 2.5). The classiﬁer is
validated on the testing set riply tst.mat.
trn = load(’riply_trn’); % load training data
distrib = mlcgmm(data); % estimate Gaussians
model = androrig(distrib); % solve Anderson’s task
figure;
pandr( model, distrib ); % plot solution
tst = load(’riply_tst’); % load testing data
ypred = linclass( tst.X, model ); % classify testing data
cerror( ypred, tst.y ) % evaluate error
ans =
0.1150
−0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
gauss 1
0.929
gauss 2
1.37
Figure 2.5: Solution of the original Anderson’s task.
20
2.4.2 General algorithm framework
The algorithm is implemented in the function ganders. This algorithm repeats the
following steps:
I. An improving direction (∆w, ∆b) in the parameter space is computed such that
moving along this direction increases the objective function F. The quadprog
function of the Matlab optimization toolbox is used to solve this subtask.
II. The optimal movement k along the direction (∆w, ∆b). The optimal movement k
is determined by 1Dsearch based on the cutting interval algorithm according to
the Fibonacci series. The number of interval cuttings of is an optional parameter
of the algorithm.
III. The new parameters are set
w
(t+1)
: = w
(t)
+ k∆w , b
(t+1)
: = b
(t)
+ k∆b .
The algorithm iterates until the minimal change in the objective function is less than
the prescribed ε, i.e., the condition
F(w
(t+1)
, b
(t+1)
) −F(w
(t)
, b
(t)
) > ε ,
is violated.
Example: Solving the Generalized Anderson’s task
The Generalized Anderson’s task is solved for the data mars.mat. Both the ﬁrst
and second class are described by three 2Ddimensional Gaussian distributions. The
found linear classiﬁer as well as the input Gaussian distributions are visualized (see
Figure 2.6).
distrib = load(’mars’); % load input Gaussians
model = ganders(distrib); % solve the GAT
figure;
pandr(model,distrib); % plot the solution
2.4.3 εsolution using the Kozinec’s algorithm
This algorithm is implemented in the function eanders. The εsolution is deﬁned as a
linear classiﬁer with parameters (w, b) such that
F(w, b) < ε
21
−5 −4 −3 −2 −1 0 1 2 3 4 5
−4
−3
−2
−1
0
1
2
3
4
5
gauss 1
0.0461
gauss 2
0.0483
gauss 3
0.0503
gauss 4
0.0421
gauss 5
0.047
gauss 6
0.0503
Figure 2.6: Solution of the Generalized Anderson’s task.
is satisﬁed. The prescribed ε ∈ (0, 0.5) is the desired upper bound on the probability
of misclassiﬁcation. The problem is transformed to the task of separation of two sets
of ellipsoids which is solved the Kozinec’s algorithm. The algorithm converges in ﬁnite
number of iterations if the given εsolution exists. Otherwise it can get stuck in an
endless loop.
Example: εsolution of the Generalized Anderson’s task
The εsolution of the generalized Anderson’s task is solved for the data mars.mat.
The desired maximal possible probability of misclassiﬁcation is set to 0.06 (it is 6%).
Both the ﬁrst and second class are described by three 2Ddimensional Gaussian distri
butions. The found linear classiﬁer and the input Gaussian distributions are visualized
(see Figure 2.7).
distrib = load(’mars’); % load Gaussians
options = struct(’err’,0.06’); % maximal desired error
model = eanders(distrib,options); % training
figure; pandr(model,distrib); % visualization
2.4.4 Generalized gradient optimization
This algorithm is implemented in the function ggradandr. The objective function
F(w, b) is convex but not diﬀerentiable thus the standard gradient optimization cannot
22
−5 −4 −3 −2 −1 0 1 2 3 4 5
−4
−3
−2
−1
0
1
2
3
4
5
gauss 1
0.0473
gauss 2
0.0496
gauss 3
0.0516
gauss 4
0.0432
gauss 5
0.0483
gauss 6
0.0516
Figure 2.7: εsolution of the Generalized Anderson’s task with maximal 0.06 probability
of misclassiﬁcation.
be used. However, the generalized gradient optimization method can be applied. The
algorithm repeats the following two steps:
I. The generalized gradient (∆w, ∆b) is computed.
II. The optimized parameters are updated
w
(t+1)
: = w
(t)
+ ∆w , b
(t+1)
= b
(t)
+ ∆b .
The algorithm iterates until the minimal change in the objective function is less than
the prescribed ε, i.e., the condition
F(w
(t+1)
, b
(t+1)
) −F(w
(t)
, b
(t)
) > ε ,
is violated. The maximal number of iterations can be used as the stopping condition
as well.
Example: Solving the Generalized Anderson’s task using the generalized
gradient optimization
The generalized Anderson’s task is solved for the data mars.mat. Both the ﬁrst
and second class are described by three 2Ddimensional Gaussian distributions. The
maximal number of iterations is limited to tmax=10000. The found linear classiﬁer as
well as the input Gaussian distributions are visualized (see Figure 2.8).
23
distrib = load(’mars’); % load Gaussians
options = struct(’tmax’,10000); % max # of iterations
model = ggradandr(distrib,options); % training
figure;
pandr( model, distrib ); % visualization
−5 −4 −3 −2 −1 0 1 2 3 4 5
−4
−3
−2
−1
0
1
2
3
4
5
gauss 1
0.0475
gauss 2
0.0498
gauss 3
0.0518
gauss 4
0.0434
gauss 5
0.0485
gauss 6
0.0518
Figure 2.8: Solution of the Generalized Anderson’s task after 10000 iterations of the
generalized gradient optimization algorithm.
24
Chapter 3
Feature extraction
The feature extraction step consists of mapping the input vector of observations x ∈ R
n
onto a new feature description z ∈ R
m
which is more suitable for given task. The linear
projection
z = W
T
x +b , (3.1)
is commonly used. The projection is given by the matrix W [n m] and bias vector
b ∈ R
m
. The Principal Component Analysis (PCA) (Section 3.1) is a representative of
the unsupervised learning method which yields the linear projection (3.1). The Linear
Discriminant Analysis (LDA) (Section 3.2) is a representative of the supervised learning
method yielding the linear projection. The linear data projection is implemented in
function linproj (See examples in Section 3.1 and Section 3.2). The data type used
to describe the linear data projection is deﬁned in Table 3.
The quadratic data mapping projects the input vector x ∈ R
n
onto the feature
space R
m
where m = n(n + 3)/2. The quadratic mapping φ: R
n
→R
m
is deﬁned as
φ(x) = [ x
1
, x
2
, . . . , x
n
,
x
1
x
1
, x
1
x
2
, . . . , x
1
x
n
,
x
2
x
2
, . . . , x
2
x
n
,
.
.
.
x
n
x
n
]
T
,
(3.2)
where x
i
, i = 1, . . . , n are entries of the vector x ∈ R
n
. The quadratic data mapping
is implemented in function qmap.
The kernel functions allow for nonlinear extensions of the linear feature extrac
tion methods. In this case, the input vectors x ∈ R
n
are mapped into a new higher
dimensional feature space T in which the linear methods are applied. This yields a
nonlinear (kernel) projection of data which has generally deﬁned as
z = A
T
k(x) +b , (3.3)
25
where A [l m] is a parameter matrix and b ∈ R
m
a parameter vector. The vector
k(x) = [k(x, x
1
), . . . , k(x, x
l
)]
T
is a vector of kernel functions centered in the training
data or their subset. The x ∈ R
n
stands for the input vector and z ∈ R
m
is the
vector of extracted features. The extracted features are projections of φ(x) onto m
vectors (or functions) from the feature space T. The Kernel PCA (Section 3.3) and
the Generalized Discriminant Analysis (GDA) (Section 3.5) are representatives of the
feature extraction methods yielding the kernel data projection (3.3). The kernel data
projection is implemented in the function kernelproj (See examples in Section 3.3
and Section 3.5). The data type used to describe the kernel data projection is deﬁned
in Table 3. The summary of implemented methods for feature extraction is given in
Table 3.1
Table 3.1: Implemented methods for feature extraction
linproj Linear data projection.
kerproj Kernel data projection.
qmap Quadratic data mapping.
lin2quad Merges linear rule and quadratic mapping.
lin2svm Merges linear rule and kernel projection.
lda Linear Discriminant Analysis.
pca Principal Component Analysis.
pcarec Computes reconstructed vector after PCA projection.
gda Generalized Discriminant Analysis.
greedykpca Greedy Kernel Principal Component Analysis.
kpca Kernel Principal Component Analysis.
kpcarec Reconstructs image after kernel PCA.
demo pcacomp Demo on image compression using PCA.
Table 3.2: Datatype used to describe linear data projection.
Linear data projection (structure array):
.W [n m] Projection matrix W = [w
1
, . . . , w
m
].
.b [m1] Bias vector b.
.fun = ’linproj’ Identiﬁes function associated with this data type.
3.1 Principal Component Analysis
Let T
X
= ¦x
1
, . . . , x
l
¦ be a set of training vectors from the ndimensional input space
R
n
. The set of vectors T
Z
= ¦z
1
, . . . , z
l
¦ is a lower dimensional representation of the
26
Table 3.3: Datatype used to describe kernel data projection.
Kernel data projection (structure array):
.Alpha [l m] Parameter matrix A [l m].
.b [m1] Bias vector b.
.sv.X [n l] Training vectors ¦x
1
, . . . , x
l
¦.
.options.ker [string] Kernel identiﬁer.
.options.arg [1 p] Kernel argument(s).
.fun = ’kernelproj’ Identiﬁes function associated with this data type.
input training vectors T
X
in the mdimensional space R
m
. The vectors T
Z
are obtained
by the linear orthonormal projection
z = W
T
x +b , (3.4)
where the matrix W [n m] and the vector b [m 1] are parameters of the projec
tion. The reconstructed vectors T
˜
X
= ¦ ˜ x
1
, . . . , ˜ x
l
¦ are computed by the linear back
projection
˜ x = W(z −b) , (3.5)
obtained by inverting (3.4). The mean square reconstruction error
ε
MS
(W, b) =
1
l
l
i=1
x
i
− ˜ x
i

2
, (3.6)
is a function of the parameters of the linear projections (3.4) and (3.5). The Principal
Component Analysis (PCA) is the linear orthonormal projection (3.4) which allows
for the minimal mean square reconstruction error (3.6) of the training data T
X
. The
parameters (W, b) of the linear projection are the solution of the optimization task
(W, b) = argmin
W
,b
ε
MS
(W
, b
) , (3.7)
subject to
¸w
i
w
j
) = δ(i, j) , ∀i, j ,
where w
i
, i = 1, . . . , m are column vectors of the matrix W = [w
1
, . . . , w
m
] and
δ(i, j) is the Kronecker delta function. The solution of the task (3.7) is the matrix
W = [w
1
, . . . , w
m
] containing the m eigenvectors of the sample covariance matrix
which have the largest eigen values. The vector b equals to W
T
µ, where µ is the
sample mean of the training data. The PCA is implemented in the function pca. A
demo showing the use of PCA for image compression is implemented in the script
demo pcacomp.
27
References: The PCA is treated throughout for instance in books [13, 3, 6].
Example: Principal Component Analysis
The input data are synthetically generated from the 2dimensional Gaussian dis
tribution. The PCA is applied to train the 1dimensional subspace to approximate
the input data with the minimal reconstruction error. The training data, the recon
structed data and the 1dimensional subspace given by the ﬁrst principal component
are visualized (see Figure 3.1).
X = gsamp([5;5],[1 0.8;0.8 1],100); % generate data
model = pca(X,1); % train PCA
Z = linproj(X,model); % lower dim. proj.
XR = pcarec(X,model); % reconstr. data
figure; hold on; axis equal; % visualization
h1 = ppatterns(X,’kx’);
h2 = ppatterns(XR,’bo’);
[dummy,mn] = min(Z);
[dummy,mx] = max(Z);
h3 = plot([XR(1,mn) XR(1,mx)],[XR(2,mn) XR(2,mx)],’r’);
legend([h1 h2 h3], ...
’Input vectors’,’Reconstructed’, ’PCA subspace’);
2 3 4 5 6 7 8
2.5
3
3.5
4
4.5
5
5.5
6
6.5
7
7.5
Input vectors
Reconstructed
PCA subspace
Figure 3.1: Example on the Principal Component Analysis.
28
3.2 Linear Discriminant Analysis
Let T
XY
= ¦(x
1
, y
1
), . . . , (x
l
, y
l
)¦, x
i
∈ R
n
, y ∈ ¸ = ¦1, 2, . . . , c¦ be labeled set
of training vectors. The withinclass S
W
and betweenclass S
B
scatter matrices are
described as
S
W
=
y∈Y
S
y
, S
y
=
i∈Iy
(x
i
−µ
i
)(x
i
−µ
i
)
T
,
S
B
=
y∈Y
[1
y
[(µ
y
−µ)(µ
y
−µ)
T
,
(3.8)
where the total mean vector µ and the class mean vectors µ
y
, y ∈ ¸ are deﬁned as
µ =
1
l
l
i=1
x
i
, µ
y
=
1
[1
y
[
i∈Iy
x
i
, y ∈ ¸ .
The goal of the Linear Discriminant Analysis (LDA) is to train the linear data projec
tion
z = W
T
x ,
such that the class separability criterion
F(W) =
det(
˜
S
B
)
det(
˜
S
W
)
=
det(S
B
)
det(S
W
)
,
is maximized. The
˜
S
B
is the betweenclass and
˜
S
W
is the withinclass scatter matrix
of projected data which is deﬁned similarly to (3.8).
References: The LDA is treated for instance in books [3, 6].
Example 1: Linear Discriminant Analysis
The LDA is applied to extract 2 features from the Iris data set iris.mat which
consists of the labeled 4dimensional data. The data after the feature extraction step
are visualized in Figure 3.2.
orig_data = load(’iris’); % load input data
model = lda(orig_data,2); % train LDA
ext_data = linproj(orig_data,model); % feature extraction
figure; ppatterns(ext_data); % plot ext. data
Example 2: PCA versus LDA
This example shows why the LDA is superior to the PCA when features good for
classiﬁcation are to be extracted. The synthetical data are generated from the Gaussian
mixture model. The LDA and PCA are trained on the generated data. The LDA and
29
−1.5 −1 −0.5 0 0.5 1 1.5
−0.5
−0.4
−0.3
−0.2
−0.1
0
0.1
0.2
0.3
0.4
0.5
Figure 3.2: Example shows 2dimensional data extracted from the originally 4
dimensional Iris data set using Linear Discriminant Analysis.
PCA directions are displayed as connection of the vectors projected to the trained LDA
and PCA subspaces and reprojected back to the input space (see Figure 3.3a). The
extracted data using the LDA and PCA model are displayed with the Gaussians ﬁtted
by the MaximumLikelihood (see Figure 3.3b).
% Generate data
distrib.Mean = [[5;4] [4;5]]; % mean vectors
distrib.Cov(:,:,1) = [1 0.9; 0.9 1]; % 1st covariance
distrib.Cov(:,:,2) = [1 0.9; 0.9 1]; % 2nd covariance
distrib.Prior = [0.5 0.5]; % Gaussian weights
data = gmmsamp(distrib,250); % sample data
lda_model = lda(data,1); % train LDA
lda_rec = pcarec(data.X,lda_model);
lda_data = linproj(data,lda_model);
pca_model = pca(data.X,1); % train PCA
pca_rec = pcarec(data.X,pca_model);
pca_data = linproj( data,pca_model);
figure; hold on; axis equal; % visualization
ppatterns(data);
30
h1 = plot(lda_rec(1,:),lda_rec(2,:),’r’);
h2 = plot(pca_rec(1,:),pca_rec(2,:),’b’);
legend([h1 h2],’LDA direction’,’PCA direction’);
figure; hold on;
subplot(2,1,1); title(’LDA’); ppatterns(lda_data);
pgauss(mlcgmm(lda_data));
subplot(2,1,2); title(’PCA’); ppatterns(pca_data);
pgauss(mlcgmm(pca_data));
3.3 Kernel Principal Component Analysis
The Kernel Principal Component Analysis (Kernel PCA) is the nonlinear extension of
the ordinary linear PCA. The input training vectors T
X
= ¦x
1
, . . . , x
l
¦, x
i
∈ A ⊆ R
n
are mapped by φ: A → T to a high dimensional feature space T. The linear PCA
is applied on the mapped data T
Φ
= ¦φ(x
1
), . . . , φ(x
l
)¦. The computation of the
principal components and the projection on these components can be expressed in
terms of dot products thus the kernel functions k: A A →R can be employed. The
kernel PCA trains the kernel data projection
z = A
T
k(x) +b , (3.9)
such that the reconstruction error
ε
KMS
(A, b) =
1
l
l
i=1
φ(x
i
) −
˜
φ(x
i
)
2
, (3.10)
is minimized. The reconstructed vector
˜
φ(x) is given as a linear combination of the
mapped data T
Φ
˜
φ(x) =
l
i=1
β
i
φ(x
i
) , β = A(z −b) . (3.11)
In contrast to the linear PCA, the explicit projection from the feature space T to the
input space A usually do not exist. The problem is to ﬁnd the vector x ∈ A its image
φ(x) ∈ T well approximates the reconstructed vector
˜
φ(x) ∈ T. This procedure is
implemented in the function kpcarec for the case of the Radial Basis Function (RBF)
kernel. The procedure consists of the following step:
I. Project input vector x
in
∈ R
n
onto its lower dimensional representation z ∈ R
m
using (3.9).
31
II. Computes vector x
out
∈ R
n
which is good preimage of the reconstructed vector
˜
φ(x
in
), such that
x
out
= argmin
x
φ(x) −
˜
φ(x
in
)
2
.
References: The kernel PCA is described at length in book [29, 30].
Example: Kernel Principal Component Analysis
The example shows using the kernel PCA for data denoising. The input data are
synthetically generated 2dimensional vectors which lie on a circle and are corrupted by
the Gaussian noise. The kernel PCA model with RBF kernel is trained. The function
kpcarec is used to ﬁnd preimages of the reconstructed data (3.11). The result is
visualized in Figure 3.4.
X = gencircledata([1;1],5,250,1); % generate circle data
options.ker = ’rbf’; % use RBF kernel
options.arg = 4; % kernel argument
options.new_dim = 2; % output dimension
model = kpca(X,options); % compute kernel PCA
XR = kpcarec(X,model); % compute reconstruced data
figure; % Visualization
h1 = ppatterns(X);
h2 = ppatterns(XR, ’+r’);
legend([h1 h2],’Input vectors’,’Reconstructed’);
3.4 Greedy kernel PCA algorithm
The Greedy Kernel Principal Analysis (Greedy kernel PCA) is an eﬃcient algorithm
to compute the ordinary kernel PCA. Let T
X
= ¦x
1
, . . . , x
l
¦, x
i
∈ R
n
be the set of
input training vectors. The goal is to train the kernel data projection
z = A
T
k
S
(x) +b , (3.12)
where A [d m] is the parameter matrix, b [m 1] is the parameter vector and
k
S
= [k(x, s
1
), . . . , k(x, s
l
)]
T
are the kernel functions centered in the vectors T
S
=
¦s
1
, . . . , s
d
¦. The vector set T
S
is a subset of training data T
X
. In contrast to the
ordinary kernel PCA, the subset T
S
does not contain all the training vectors T
X
thus
the complexity of the projection (3.12) is reduced compared to (3.9). The objective of
the Greedy kernel PCA is to minimize the reconstruction error (3.11) while the size d
of the subset T
S
is kept small. The Greedy kernel PCA algorithm is implemented in
the function greedykpca.
32
References: The implemented Greedy kernel PCA algorithm is described in [11].
Example: Greedy Kernel Principal Component Analysis
The example shows using kernel PCA for data denoising. The input data are
synthetically generated 2dimensional vectors which lie on a circle and are corrupted
with the Gaussian noise. The Greedy kernel PCA model with RBF kernel is trained.
The number of vectors deﬁning the kernel projection (3.12) is limited to 25. In contrast
the ordinary kernel PCA (see experiment in Section 3.3) requires all the 250 training
vectors. The function kpcarec is used to ﬁnd preimages of the reconstructed data
obtained by the kernel PCA projection. The training and reconstructed vectors are
visualized in Figure 3.5. The vector of the selected subset T
X
are visualized as well.
X = gencircledata([1;1],5,250,1); % generate training data
options.ker = ’rbf’; % use RBF kernel
options.arg = 4; % kernel argument
options.new_dim = 2; % output dimension
options.m = 25; % size of sel. subset
model = greedykpca(X,options); % run greedy algorithm
XR = kpcarec(X,model); % reconstructed vectors
figure; % visualization
h1 = ppatterns(X);
h2 = ppatterns(XR,’+r’);
h3 = ppatterns(model.sv.X,’ob’,12);
legend([h1 h2 h3], ...
’Training set’,’Reconstructed set’,’Selected subset’);
3.5 Generalized Discriminant Analysis
The Generalized Discriminant Analysis (GDA) is the nonlinear extension of the ordi
nary Linear Discriminant Analysis (LDA). The input training data T
XY
= ¦(x
1
, y
1
), . . . , (x
l
, y
l
)¦,
x
i
∈ A ⊆ R
n
, y ∈ ¸ = ¦1, . . . , c¦ are mapped by φ: A → T to a high dimen
sional feature space T. The ordinary LDA is applied on the mapped data T
ΦY
=
¦(φ(x
1
), y
1
), . . . , (φ(x
l
), y
l
)¦. The computation of the projection vectors and the pro
jection on these vectors can be expressed in terms of dot products thus the kernel
functions k: A A → R can be employed. The resulting kernel data projection is
deﬁned as
z = A
T
k(x) +b ,
33
where A [l m] is the matrix and b [m 1] the vector trained by the GDA. The
parameters (A, b) are trained to increase the betweenclass scatter and decrease the
withinclass scatter of the extracted data T
ZY
= ¦(z
1
, y
1
), . . . , (z
l
, y
l
)¦, z
i
∈ R
m
. The
GDA is implemented in the function gda.
References: The GDA was published in the paper [2].
Example: Generalized Discriminant Analysis
The GDA is trained on the Iris data set iris.mat which contains 3 classes of
4dimensional data. The RBF kernel with kernel width σ = 1 is used. Notice, that
setting σ too low leads to the perfect separability of the training data T
XY
but the kernel
projection is heavily overﬁtted. The extracted data ¸
ZY
are visualized in Figure 3.6.
orig_data = load(’iris’); % load data
options.ker = ’rbf’; % use RBF kernel
options.arg = 1; % kernel arg.
options.new_dim = 2; % output dimension
model = gda(orig_data,options); % train GDA
ext_data = kernelproj(orig_data,model); % feature ext.
figure; ppatterns(ext_data); % visualization
3.6 Combining feature extraction and linear classi
ﬁer
Let T
XY
= ¦(x
1
, y
1
), . . . , (x
l
, y
l
)¦ be a training set of observations x
i
∈ A ⊆ R
n
and
corresponding hidden states y
i
∈ ¸ = ¦1, . . . , c¦. Let φ: A →R
m
be a mapping
φ(x) = [φ
1
(x), . . . , φ
m
(x)]
T
deﬁning a nonlinear feature extraction step (e.g., quadratic data mapping or kernel
projection). The feature extraction step applied to the input data T
XY
yields extracted
data T
ΦY
= ¦(φ(x
1
), y
1
), . . . , (φ(x
l
), y
l
)¦. Let f(x) = ¸w φ(x)) + b be a linear
discriminant function found by an arbitrary training method on the extracted data
T
ΦY
. The function f can be seen as a nonlinear function of the input vectors x ∈ A,
as
f(x) = ¸w φ(x)) + b =
m
i=1
w
i
φ
i
(x) + b .
In the case of the quadratic data mapping (3.2) used as the feature extraction step,
the discriminant function can be written in the following form
f(x) = ¸x Ax) +¸x b) + c . (3.13)
34
The discriminant function (3.13) deﬁnes the quadratic classiﬁer described in Section 6.4.
Therefore the quadratic classiﬁer can be trained by (i) applying the quadratic data
mapping qmap, (ii) training a linear rule on the mapped data (e.g., perceptron, fld)
and (iii) combining the quadratic mapping and the linear rule. The combining is
implement in function lin2quad.
In the case of the kernel projection (3.3) applied as the feature extraction step, the
resulting discriminant function has the following form
f(x) = ¸α k(x)) + b . (3.14)
The discriminant function (3.14) deﬁnes the kernel (SVM type) classiﬁer described
in Section 5. This classiﬁer can be trained by (i) applying the kernel projection
kernelproj, (ii) training a linear rule on the extracted data and (iii) combing the kernel
projection and the linear rule. The combining is implemented in function lin2svm.
Example: Quadratic classiﬁer trained the Perceptron
This example shows how to train the quadratic classiﬁer using the Perceptron al
gorithm. The Perceptron algorithm produces the linear rule only. The input training
data are linearly nonseparable but become separable after the quadratic data map
ping is applied. Function lin2quad is used to compute the parameters of the quadratic
classiﬁer explicitly based on the found linear rule in the feature space and the mapping.
Figure 3.7 shows the decision boundary of the quadratic classiﬁer.
% load training data living in the input space
input_data = load(’vltava’);
% map data to the feature space
map_data = qmap(input_data);
% train linear rule in the feature sapce
lin_model = perceptron(map_data);
% compute parameters of the quadratic classifier
quad_model = lin2quad(lin_model);
% visualize the quadratic classifier
figure; ppatterns(input_data);
pboundary(quad_model);
Example: Kernel classiﬁer from the linear rule
This example shows how to train the kernel classiﬁer using arbitrary training algo
rithm for linear rule. The Fisher Linear Discriminant was used in this example. The
35
greedy kernel PCA algorithm was applied as the feature extraction step. The Riply’s
data set was used for training and evaluation. Figure 3.8 shows the found kernel clas
siﬁer, the training data and vectors used in the kernel expansion of the discriminant
function.
% load input data
trn = load(’riply_trn’);
% train kernel PCA by greedy algorithm
options = struct(’ker’,’rbf’,’arg’,1,’new_dim’,10);
kpca_model = greedykpca(trn.X,options);
% project data
map_trn = kernelproj(trn,kpca_model);
% train linear classifier
lin_model = fld(map_trn);
% combine linear rule and kernel PCA
kfd_model = lin2svm(kpca_model,lin_model);
% visualization
figure;
ppatterns(trn); pboundary(kfd_model);
ppatterns(kfd_model.sv.X,’ok’,13);
% evaluation on testing data
tst = load(’riply_tst’);
ypred = svmclass(tst.X,kfd_model);
cerror(ypred,tst.y)
ans =
0.0970
36
0 1 2 3 4 5 6 7 8
1
2
3
4
5
6
7
LDA direction
PCA direction
(a)
−2 −1.5 −1 −0.5 0 0.5 1 1.5 2
0
0.2
0.4
0.6
0.8
1
1.2
1.4
LDA
−4 −3 −2 −1 0 1 2 3 4 5
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
PCA
(b)
Figure 3.3: Comparison between PCA and LDA. Figure (a) shows input data and
found LDA (red) and PCA (blue) directions onto which the data are projected. Figure
(b) shows the class conditional Gaussians estimated from data projected data onto
LDA and PCA directions.
37
−8 −6 −4 −2 0 2 4 6 8 10
−8
−6
−4
−2
0
2
4
6
8
10
Input vectors
Reconstructed
Figure 3.4: Example on the Kernel Principal Component Analysis.
−8 −6 −4 −2 0 2 4 6 8 10
−6
−4
−2
0
2
4
6
8
Training set
Reconstructed set
Selected set
Figure 3.5: Example on the Greedy Kernel Principal Component Analysis.
38
−0.6 −0.5 −0.4 −0.3 −0.2 −0.1 0 0.1 0.2 0.3
−0.2
−0.15
−0.1
−0.05
0
0.05
0.1
0.15
Figure 3.6: Example shows 2dimensional data extracted from the originally 4
dimensional Iris data set using the Generalized Discriminant Analysis. In contrast
to LDA (see Figure 3.2 for comparison), the class clusters are more compact but at the
expense of a higher risk of overﬁtting.
−0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1
−0.6
−0.4
−0.2
0
0.2
0.4
0.6
0.8
Figure 3.7: Example shows quadratic classiﬁer found by the Perceptron algorithm on
the data mapped to the feature by the quadratic mapping.
39
−1.5 −1 −0.5 0 0.5 1
−0.2
0
0.2
0.4
0.6
0.8
1
1.2
Figure 3.8: Example shows the kernel classiﬁer found by the Fisher Linear Discriminant
algorithm on the data mapped to the feature by the kernel PCA. The greedy kernel
PCA was used therefore only a subset of training data was used to determine the kernel
projection.
40
Chapter 4
Density estimation and clustering
The STPRtool represents a probability distribution function P
X
(x), x ∈ R
n
as a
structure array which must contain ﬁeld .fun (see Table 4.2). The ﬁeld .fun speciﬁes
function which is used to evaluate P
X
(x) for given set of realization ¦x
1
, . . . , x
l
¦ of a
random vector. Let the structure model represent a given distribution function. The
Matlab command
y = feval(X,model.fun)
is used to evaluate y
i
= P
X
(x
i
). The matrix X [n l] contains column vectors
¦x
1
, . . . , x
l
¦. The vector y [1 l] contains values of the distribution function. In
particular, the Gaussian distribution and the Gaussian mixture models are described
in Section 4.1. The summary of implemented methods for dealing with probability
density estimation and clustering is given in Table 4.1
4.1 Gaussian distribution and Gaussian mixture model
The value of the multivariate Gaussian probability distribution function in the vector
x ∈ R
n
is given by
P
X
(x) =
1
(2π)
n
2
_
det(Σ)
exp(−
1
2
(x −µ)
T
Σ
−1
(x −µ)) , (4.1)
where µ [n 1] is the mean vector and Σ [n n] is the covariance matrix. The
STPRtool uses speciﬁc datatype to describe a set of Gaussians which can be labeled,
i.e., each pair (µ
i
, Σ
i
) is assigned to the class y
i
∈ ¸. The covariance matrix Σ is
always represented as the square matrix [n n], however, the shape of the matrix can
be indicated in the optimal ﬁeld .cov type (see Table 4.3). The datatype is described
in Table 4.4. The evaluation of the Gaussian distribution is implemented in the function
41
Table 4.1: Implemented methods: Density estimation and clustering
gmmsamp Generates sample from Gaussian mixture model.
gsamp Generates sample from Gaussian distribution.
kmeans Kmeans clustering algorithm.
pdfgauss Evaluates for multivariate Gaussian distribution.
pdfgauss Evaluates Gaussian mixture model.
sigmoid Evaluates sigmoid function.
emgmm ExpectationMaximization Algorithm for Gaussian
mixture model.
melgmm Maximizes Expectation of LogLikelihood for Gaus
sian mixture.
mlcgmm Maximal Likelihood estimation of Gaussian mixture
model.
mlsigmoid Fitting a sigmoid function using ML estimation.
mmgauss Minimax estimation of Gaussian distribution.
rsde Reduced Set Density Estimator.
demo emgmm Demo on ExpectationMaximization (EM) algorithm.
demo mmgauss Demo on minimax estimation for Gaussian.
demo svmpout Fitting a posteriori probability to SVM output.
Table 4.2: Datatype used to represent probability distribution function.
Probability Distribution Function (structure array):
.fun [string] The name of a function which evaluates probability
distribution y = feval( X, model.fun ).
pdfgauss. The random sampling from the Gaussian distribution is implemented in the
function gsamp.
The Gaussian Mixture Model (GMM) is weighted sum of the Gaussian distributions
P
X
(x) =
y∈Y
P
Y
(y)P
XY
(x[y) , (4.2)
where P
Y
(y), y ∈ ¸ = ¦1, . . . , c¦ are weights and P
XY
(x[y) is the Gaussian distribu
tion (4.1) given by parameters (µ
y
, Σ
y
). The datatype used to represent the GMM
is described in Table 4.5. The evaluation of the GMM is implemented in the function
pdfgmm. The random sampling from the GMM is implemented in the function gmmsamp.
Example: Sampling from the Gaussian distribution
The example shows how to represent the Gaussian distribution and how to sample
data it. The sampling from univariate and bivariate Gaussians is shown. The distribu
tion function of the univariate Gaussian is visualized and compared to the histogram
42
Table 4.3: Shape of covariance matrix.
Parameter cov type:
’full’ Full covariance matrix.
’diag’ Diagonal covariance matrix.
’spherical’ Spherical covariance matrix.
Table 4.4: Datatype used to represent the labeled set of Gaussians.
Labeled set of Gaussians (structure array):
.Mean [n k] Mean vectors ¦µ
1
, . . . , µ
k
¦.
.Cov [n n k] Covariance matrices ¦Σ
1
, . . . , Σ
k
¦.
.y [1 k] Labels ¦y
1
, . . . , y
k
¦ assigned to Gaussians.
.cov type [string] Optional ﬁeld which indicates shape of the covariance
matrix (see Table 4.3).
.fun = ’pdfgauss’ Identiﬁes function associated with this data type.
of the sample data (see Figure 4.1a). The isoline of the distribution function of the
bivariate Gaussian is visualized together with the sample data (see Figure 4.1b).
% univariate case
model = struct(’Mean’,1,’Cov’,2); % Gaussian parameters
figure; hold on;
% plot pdf. of the Gaussisan
plot([4:0.1:5],pdfgauss([4:0.1:5],model),’r’);
[Y,X] = hist(gsamp(model,500),10); % compute histogram
bar(X,Y/500); % plot histogram
% bivariate case
model.Mean = [1;1]; % Mean vector
model.Cov = [1 0.6; 0.6 1]; % Covariance matrix
figure; hold on;
ppatterns(gsamp(model,250)); % plot sampled data
pgauss(model); % plot shape
4.2 MaximumLikelihood estimation of GMM
Let the probability distribution
P
XY Θ
(x, y[θ) = P
XY Θ
(x[y, θ)P
Y Θ
(y[θ) ,
43
Table 4.5: Datatype used to represent the Gaussian Mixture Model.
Gaussian Mixture Model (structure array):
.Mean [n c] Mean vectors ¦µ
1
, . . . , µ
c
¦.
.Cov [n n c] Covariance matrices ¦Σ
1
, . . . , Σ
c
¦.
.Prior [c 1] Weights of Gaussians ¦P
K
(1), . . . , P
K
(c)¦.
.fun = ’pdfgmm’ Identiﬁes function associated with this data type.
−4 −3 −2 −1 0 1 2 3 4 5 6
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
−2 −1 0 1 2 3 4 5
−3
−2
−1
0
1
2
3
4
5
gauss 1
0.127
(a) (b)
Figure 4.1: Sampling from the univariate (a) and bivariate (b) Gaussian distribution.
be known up to parameters θ ∈ Θ. The vector of observations x ∈ R
n
and the hidden
state y ∈ ¸ = ¦1, . . . , c¦ are assumed to be realizations of random variables which
are independent and identically distributed according to the P
XY Θ
(x, y[θ). The con
ditional probability distribution P
XY Θ
(x[y, θ) is assumed to be Gaussian distribution.
The θ involves parameters (µ
y
, Σ
y
), y ∈ ¸ of Gaussian components and the values
of the discrete distribution P
Y Θ
(y[θ), y ∈ ¸. The marginal probability P
XΘ
(x[θ) is
the Gaussian mixture model (4.2). The MaximumLikelihood estimation of the pa
rameters θ of the GMM for the case of complete and incomplete data is described in
Section 4.2.1 and Section 4.2.2, respectively.
4.2.1 Complete data
The input is a set T
XY
= ¦(x
1
, y
1
), . . . , (x
l
, y
l
)¦ which contains both vectors of ob
servations x
i
∈ R
n
and corresponding hidden states y
i
∈ ¸. The pairs (x
i
, y
i
) are
assumed to samples from random variables which are independent and identically dis
tributed (i.i.d.) according to P
XY Θ
(x, y[θ). The logarithm of probability P(T
XY
[θ)
is referred to as the loglikelihood L(θ[T
XY
) of θ with respect to T
XY
. Thanks to the
44
i.i.d. assumption the loglikelihood can be factorized as
L(θ[T
XY
) = log P(T
XY
[θ) =
l
i=1
log P
XY Θ
(x
i
, y
i
[θ) .
The problem of MaximumLikelihood (ML) estimation from the complete data T
XY
is
deﬁned as
θ
∗
= argmax
θ∈Θ
L(θ[T
XY
) = argmax
θ∈Θ
l
i=1
log P
XY Θ
(x
i
, y
i
[θ) . (4.3)
The ML estimate of the parameters θ of the GMM from the complete data set T
XY
has an analytical solution. The computation of the ML estimate is implemented in the
function mlcgmm.
Example: MaximumLikelihood estimate from complete data
The parameters of the Gaussian mixture model is estimated from the complete
Riply’s data set riply trn. The data are binary labeled thus the GMM has two Gaus
sians components. The estimated GMM is visualized as: (i) shape of components of the
GMM (see Figure 4.2a) and (ii) contours of the distribution function (see Figure 4.2b)
.
data = load(’riply_trn’); % load labeled (complete) data
model = mlcgmm(data); % ML estimate of GMM
% visualization
figure; hold on;
ppatterns(data); pgauss(model);
figure; hold on;
ppatterns(data);
pgmm(model,struct(’visual’,’contour’));
4.2.2 Incomplete data
The input is a set T
X
= ¦x
1
, . . . , x
l
¦ which contains only vectors of observations
x
i
∈ R
n
without knowledge about the corresponding hidden states y
i
∈ ¸. The
logarithm of probability P(T
X
[θ) is referred to as the loglikelihood L(θ[T
X
) of θ with
respect to T
X
. Thanks to the i.i.d. assumption the loglikelihood can be factorized as
L(θ[T
X
) = log P(T
X
[θ) =
l
i=1
y∈Y
P
XY Θ
(x
i
[y
i
, θ)P
Y Θ
(y
i
[θ) .
45
The problem of MaximumLikelihood (ML) estimation from the incomplete data T
X
is
deﬁned as
θ
∗
= argmax
θ∈Θ
L(θ[T
X
) = argmax
θ∈Θ
l
i=1
y∈Y
P
XY Θ
(x
i
[y
i
, θ)P
Y Θ
(y
i
[θ) . (4.4)
The ML estimate of the parameters θ of the GMM from the incomplete data set
T
XY
does not have an analytical solution. The numerical optimization has to be used
instead.
The ExpectationMaximization (EM) algorithm is an iterative procedure for ML
estimation from the incomplete data. The EM algorithm builds a sequence of parameter
estimates
θ
(0)
, θ
(1)
, . . . , θ
(t)
,
such that the loglikelihood L(θ
(t)
[T
X
) monotonically increases, i.e.,
L(θ
(0)
[T
X
) < L(θ
(1)
[T
X
) < . . . < L(θ
(t)
[T
X
)
until a stationary point L(θ
(t−1)
[T
X
) = L(θ
(t)
[T
X
) is achieved. The EM algorithm is a
local optimization technique thus the estimated parameters depend on the initial guess
θ
(0)
. The random guess or Kmeans algorithm (see Section 4.5) are commonly used to
select the initial θ
(0)
. The EM algorithm for the GMM is implemented in the function
emgmm. An interactive demo on the EM algorithm for the GMM is implemented in
demo emgmm.
References: The EM algorithm (including convergence proofs) is described in book [26].
A nice description can be found in book [3]. The monograph [18] is entirely devoted
to the EM algorithm. The ﬁrst papers describing the EM are [25, 5].
Example: MaximumLikelihood estimate from incomplete data
The estimation of the parameters of the GMM is demonstrated on a synthetic
data. The ground truth model is the GMM with two univariate Gaussian components.
The ground truth model is sampled to obtain a training (incomplete) data. The EM
algorithm is applied to estimate the GMM parameters. The random initialization is
used by default. The ground truth model, sampled data and the estimate model are
visualized in Figure 4.3a. The plot of the loglikelihood function L(θ
(t)
[T
X
) with respect
to the number of iterations t is visualized in Figure 4.3b.
% ground truth Gaussian mixture model
true_model.Mean = [2 2];
true_model.Cov = [1 0.5];
true_model.Prior = [0.4 0.6];
sample = gmmsamp(true_model, 250);
46
% ML estimation by EM
options.ncomp = 2; % number of Gaussian components
options.verb = 1; % display progress info
estimated_model = emgmm(sample.X,options);
% visualization
figure;
ppatterns(sample.X);
h1 = pgmm(true_model,struct(’color’,’r’));
h2 = pgmm(estimated_model,struct(’color’,’b’));
legend([h1(1) h2(1)],’Ground truth’, ’ML estimation’);
figure; hold on;
xlabel(’iterations’); ylabel(’loglikelihood’);
plot( estimated_model.logL );
4.3 Minimax estimation
The input is a set T
X
= ¦x
1
, . . . , x
l
¦ contains vectors x
i
∈ R
n
which are realizations of
random variable distributed according to P
XΘ
(x[θ). The vectors T
X
are assumed to be
a realizations with high value of the distribution function P
XΘ
(x[θ). The P
XΘ
(x[θ)
is known up to the parameter θ ∈ Θ. The minimax estimation is deﬁned as
θ
∗
= argmax
θ∈Θ
min
x∈T
X
log P
XΘ
(x[θ) . (4.5)
If a procedure for the MaximumLikelihood estimate (global solution) of the distri
bution P
XΘ
(x[θ) exists then a numerical iterative algorithm which converges to the
optimal estimate θ
∗
can be constructed. The algorithm builds a series of estimates
θ
(0)
, θ
(1)
, . . . , θ
(t)
which converges to the optimal estimate θ
∗
. The algorithm converges
in a ﬁnite number of iterations to such an estimate θ that the condition
min
x∈T
X
log P
XΘ
(x[θ
∗
) − min
x∈T
X
log P
XΘ
(x[θ) < ε , (4.6)
holds for ε > 0 which deﬁnes closeness to the optimal solution θ
∗
. The inequality (4.6)
can be evaluated eﬃciently thus it is a natural choice for the stopping condition of
the algorithm. The STPRtool contains an implementation of the minimax algorithm
for the Gaussian distribution (4.1). The algorithm is implemented in the function
mmgauss. An interactive demo on the minimax estimation for the Gaussian distribution
is implemented in the function demo mmgauss.
References: The minimax algorithm for probability estimation is described and ana
lyzed in Chapter 8 of the book [26].
47
Example: Minimax estimation of the Gaussian distribution
The bivariate Gaussian distribution is described by three samples which are known
to have high value of the distribution function. The minimax algorithm is applied
to estimate the parameters of the Gaussian. In this particular case, the minimax
problem is equivalent to searching for the minimal enclosing ellipsoid. The found
solution is visualized as the isoline of the distribution function at the point P
XΘ
(x[θ)
(see Figure 4.4).
X = [[0;0] [1;0] [0;1]]; % points with high p.d.f.
model = mmgauss(X); % run minimax algorithm
% visualization
figure; ppatterns(X,’xr’,13);
pgauss(model, struct(’p’,exp(model.lower_bound)));
4.4 Probabilistic output of classiﬁer
In general, the discriminant function of an arbitrary classiﬁer does not have meaning of
probability (e.g., SVM, Perceptron). However, the probabilistic output of the classiﬁer
can help in postprocessing, for instance, in combining more classiﬁers together. Fitting
a sigmoid function to the classiﬁer output is a way how to solve this problem.
Let T
XY
= ¦(x
1
, y
1
), . . . , (x
l
, y
l
)¦ is the training set composed of the vectors x
i
∈
A ⊆ R
n
and the corresponding binary hidden states y
i
∈ ¸ = ¦1, 2¦. The training
set T
XY
is assumed to be identically and independently distributed from underlying
distribution. Let f: A ⊆ R
n
→R be a discriminant function trained from the data T
XY
by an arbitrary learning method. The aim is to estimate parameters of a posteriori
distribution P
Y FΘ
(y[f(x), θ) of the hidden state y given the value of the discriminant
function f(x). The distribution is modeled by a sigmoid function
P
Y FΘ
(1[f(x), θ) =
1
1 + exp(a
1
f(x) + a
2
)
,
which is determined by parameters θ = [a
1
, a
2
]
T
. The loglikelihood function is deﬁned
as
L(θ[T
XY
) =
l
i=1
log P
Y FΘ
(y
i
[f(x
i
), θ) .
The parameters θ = [a
1
, a
2
]
T
can be estimated by the maximumlikelihood method
θ = argmax
θ
L(θ
[T
XY
) = argmax
θ
l
i=1
log P
Y FΘ
(y
i
[f(x
i
), θ
) . (4.7)
48
The STPRtool provides an implementation mlsigmoid which uses the Matlab Opti
mization toolbox function fminunc to solve the task (4.7). The function produces the
parameters θ = [a
1
, a
2
]
T
which are stored in the datatype describing the sigmoid (see
Table 4.6). The evaluation of the sigmoid is implemented in function sigmoid.
Table 4.6: Datatype used to describe sigmoid function.
Sigmoid function (structure array):
.A [2 1] Parameters θ = [a
1
, a
2
]
T
of the sigmoid function.
.fun = ’sigmoid’ Identiﬁes function associated with this data type.
References: The method of ﬁtting the sigmoid function is described in [22].
Example: Probabilistic output for SVM
This example is implemented in the script demo svmpout. The example shows how
to train the probabilistic output for the SVM classiﬁer. The Riply’s data set riply trn
is used for training the SVM and the ML estimation of the sigmoid. The ML estimated
of Gaussian mixture model (GMM) of the SVM output is used for comparison. The
GMM allows to compute the a posteriori probability for comparison to the sigmoid
estimate. The example is implemented in the script demo svmpout. Figure 4.5a shows
found probabilistic models of a posteriori probability of the SVM output. Figure 4.5b
shows the decision boundary of SVM classiﬁer for illustration.
% load training data
data = load(’riply_trn’);
% train SVM
options = struct(’ker’,’rbf’,’arg’,1,’C’,10);
svm_model = smo(data,options);
% compute SVM output
[dummy,svm_output.X] = svmclass(data.X,svm_model);
svm_output.y = data.y;
% fit sigmoid function to svm output
sigmoid_model = mlsigmoid(svm_output);
% ML estimation of GMM model of SVM output
gmm_model = mlcgmm(svm_output);
% visulization
figure; hold on;
49
xlabel(’svm output f(x)’); ylabel(’p(y=1f(x))’);
% sigmoid model
fx = linspace(min(svm_output.X), max(svm_output.X), 200);
sigmoid_apost = sigmoid(fx,sigmoid_model);
hsigmoid = plot(fx,sigmoid_apost,’k’);
ppatterns(svm_output);
% GMM model
pcond = pdfgauss( fx, gmm_model);
gmm_apost = (pcond(1,:)*gmm_model.Prior(1))./...
(pcond(1,:)*gmm_model.Prior(1)+(pcond(2,:)*gmm_model.Prior(2)));
hgmm = plot(fx,gmm_apost,’g’);
hcomp = pgauss(gmm_model);
legend([hsigmoid,hgmm,hcomp],’P(y=1f(x)) MLSigmoid’,...
’P(y=1f(x)) MLGMM’,’P(f(x)y=1) MLGMM’,’P(f(x)y=2) MLGMM’);
% SVM decision boundary
figure; ppatterns(data); psvm(svm_model);
4.5 Kmeans clustering
The input is a set T
X
= ¦x
1
, . . . , x
l
¦ which contains vectors x
i
∈ R
n
. Let θ =
¦µ
1
, . . . , µ
c
¦, µ
i
∈ R
n
be a set of unknown cluster centers. Let the objective function
F(θ) be deﬁned as
F(θ) =
1
l
l
i=1
min
y=1,...,c
µ
y
−x
i

2
.
The small value of F(θ) indicates that the centers θ well describe the input set T
X
.
The task is to ﬁnd such centers θ which describe the set T
X
best, i.e., the task is to
solve
θ
∗
= argmin
θ
F(θ) = argmin
θ
1
l
l
i=1
min
y=1,...,c
µ
y
−x
i

2
.
The Kmeans
1
is an iterative procedure which iteratively builds a series θ
(0)
, θ
(1)
, . . . , θ
(t)
.
The Kmeans algorithm converges in a ﬁnite number of steps to the local optimum of
the objective function F(θ). The found solution depends on the initial guess θ
(0)
which
1
The K in the name Kmeans stands for the number of centers which is here denoted by c.
50
is usually selected randomly. The Kmeans algorithm is implemented in the function
kmeans.
References: The Kmeans algorithm is described in the books [26, 6, 3].
Example: Kmeans clustering
The Kmeans algorithm is used to ﬁnd 4 clusters in the Riply’s training set. The
solution is visualized as the cluster centers. The vectors classiﬁed to the corresponding
centers are distinguished by color (see Figure 4.6a). The plot of the objective function
F(θ
(t)
) with respect to the number of iterations is shown in Figure 4.6b.
data = load(’riply_trn’); % load data
[model,data.y] = kmeans(data.X, 4 ); % run kmeans
% visualization
figure; ppatterns(data);
ppatterns(model.X,’sk’,14);
pboundary( model );
figure; hold on;
xlabel(’t’); ylabel(’F’);
plot(model.MsErr);
51
−1.5 −1 −0.5 0 0.5 1
−0.2
0
0.2
0.4
0.6
0.8
1
1.2
gauss 1
0.981
gauss 2
1.45
(a)
−1.5 −1 −0.5 0 0.5 1
−0.2
0
0.2
0.4
0.6
0.8
1
1.2
(b)
Figure 4.2: Example of the MaximumLikelihood of the Gaussian Mixture Model from
the incomplete data. Figure (a) shows components of the GMM and Figure (b) shows
contour plot of the distribution function.
52
−5 −4 −3 −2 −1 0 1 2 3 4 5
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
Ground truth
ML estimation
(a)
0 5 10 15 20 25
−460
−458
−456
−454
−452
−450
−448
−446
−444
−442
iterations
l
o
g
−
l
i
k
e
l
i
h
o
o
d
(b)
Figure 4.3: Example of using the EM algorithm for ML estimation of the Gaussian
mixture model. Figure (a) shows ground truth and the estimated GMM and Figure(b)
shows the loglikelihood with respect to the number of iterations of the EM.
53
−0.4 −0.2 0 0.2 0.4 0.6 0.8 1
−0.4
−0.2
0
0.2
0.4
0.6
0.8
1
gauss 1
0.304
Figure 4.4: Example of the Minimax estimation of the Gaussian distribution.
54
−4 −3 −2 −1 0 1 2 3 4 5
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
svm output f(x)
p
(
y
=
1

f
(
x
)
)
P(y=1f(x)) ML−Sigmoid
P(y=1f(x)) ML−GMM
P(f(x)y=1) ML−GMM
P(f(x)y=2) ML−GMM
(a)
−1.5 −1 −0.5 0 0.5 1
−0.2
0
0.2
0.4
0.6
0.8
1
1.2
(b)
Figure 4.5: Example of ﬁtting a posteriori probability to the SVM output. Figure (a)
shows the sigmoid model and GMM model both ﬁtted by ML estimated. Figure (b)
shows the corresponding SVM classiﬁer.
55
−1.5 −1 −0.5 0 0.5 1
−0.2
0
0.2
0.4
0.6
0.8
1
1.2
(a)
0 2 4 6 8 10 12
0.04
0.06
0.08
0.1
0.12
0.14
0.16
t
F
(b)
Figure 4.6: Example of using the Kmeans algorithm to cluster the Riply’s data set.
Figure (a) shows found clusters and Figure (b) contains the plot of the objective func
tion F(θ
(t)
) with respect to the number of iterations.
56
Chapter 5
Support vector and other kernel
machines
5.1 Support vector classiﬁers
The binary support vector classiﬁer uses the discriminant function f: A ⊆ R
n
→R of
the following form
f(x) = ¸α k
S
(x)) + b . (5.1)
The k
S
(x) = [k(x, s
1
), . . . , k(x, s
d
)]
T
is the vector of evaluations of kernel functions
centered at the support vectors o = ¦s
1
, . . . , s
d
¦, s
i
∈ R
n
which are usually subset of
the training data. The α ∈ R
l
is a weight vector and b ∈ R is a bias. The binary
classiﬁcation rule q: A → ¸ = ¦1, 2¦ is deﬁned as
q(x) =
_
1 for f(x) ≥ 0 ,
2 for f(x) < 0 .
(5.2)
The datatype used by the STPRtool to represent the binary SVM classiﬁer is described
in Table 5.2.
The multiclass generalization involves a set of discriminant functions f
y
: A ⊆ R
n
→
R, y ∈ ¸ = ¦1, 2, . . . , c¦ deﬁned as
f
y
(x) = ¸α
y
k
S
(x)) + b
y
, y ∈ ¸ .
Let the matrix A = [α
1
, . . . , α
c
] be composed of all weight vectors and b = [b
1
, . . . , b
c
]
T
be a vector of all biases. The multiclass classiﬁcation rule q: A → ¸ = ¦1, 2, . . . , c¦ is
deﬁned as
q(x) = argmax
y∈Y
f
y
(x) . (5.3)
The datatype used by the STPRtool to represent the multiclass SVM classiﬁer is
described in Table 5.3. Both the binary and the multiclass SVM classiﬁers are imple
mented in the function svmclass.
57
The majority voting strategy is other commonly used method to implement the
multiclass SVM classiﬁer. Let q
j
: A ⊆ R
n
→ ¦y
1
j
, y
2
j
¦, j = 1, . . . , g be a set of g binary
SVM rules (5.2). The jth rule q
j
(x) classiﬁes the inputs x into class y
1
j
∈ ¸ or y
2
j
∈ ¸.
Let v(x) be a vector [c 1] of votes for classes ¸ when the input x is to be classiﬁed.
The vector v(x) = [v
1
(x), . . . , v
c
(x)]
T
is computed as:
Set v
y
= 0, ∀y ∈ ¸.
For j = 1, . . . , g do
v
y
(x) = v
y
(x) + 1 where y = q
j
(x).
end.
The majority votes based multiclass classiﬁer assigns the input x into such class y ∈ ¸
having the majority of votes
y = argmax
y
=1,...,g
v
y
(x) . (5.4)
The datatype used to represent the majority voting strategy for the multiclass SVM
classiﬁer is described in Table 5.4. The strategy (5.4) is implemented in the function
mvsvmclass. The implemented methods to deal with the SVM and related kernel
classiﬁers are enlisted in Table 5.1
References: Books describing Support Vector Machines are for example [31, 4, 30]
5.2 Binary Support Vector Machines
The input is a set T
XY
= ¦(x
1
, y
1
), . . . , (x
l
, y
l
)¦ of training vectors x
i
∈ R
n
and cor
responding hidden states y
i
∈ ¸ = ¦1, 2¦. The linear SVM aims to train the linear
discriminant function f(x) = ¸w x) + b of the binary classiﬁer
q(x) =
_
1 for f(x) ≥ 0 ,
2 for f(x) < 0 .
The training of the optimal parameters (w
∗
, b
∗
) is transformed to the following quadratic
programming task
(w
∗
, b
∗
) = argmin
w,b
1
2
w
2
+ C
l
i=1
ξ
p
i
, (5.5)
¸w x
i
) + b ≥ +1 −ξ
i
, i ∈ 1
1
,
¸w x
i
) + b ≤ −1 + ξ
i
, i ∈ 1
2
,
ξ
i
≥ 0 , i ∈ 1
1
∪ 1
2
.
58
Table 5.1: Implemented methods: Support vector and other kernel machines.
svmclass Support Vector Machines Classiﬁer.
mvsvmclass Majority voting multiclass SVM classiﬁer.
evalsvm Trains and evaluates Support Vector Machines classi
ﬁer.
bsvm2 Multiclass BSVM with L2soft margin.
oaasvm Multiclass SVM using OneAgainstAll decomposi
tion.
oaosvm Multiclass SVM using OneAgainstOne decomposi
tion.
smo Sequential Minimal Optimization for binary SVM
with L1soft margin.
svmlight Interface to SV M
light
software.
svmquadprog SVM trained by Matlab Optimization Toolbox.
diagker Returns diagonal of kernel matrix of given data.
kdist Computes distance between vectors in kernel space.
kernel Evaluates kernel function.
kfd Kernel Fisher Discriminant.
kperceptr Kernel Perceptron.
minball Minimal enclosing ball in kernel feature space.
rsrbf Reduced Set Method for RBF kernel expansion.
rbfpreimg RBF preimage by Schoelkopf’s ﬁxedpoint algorithm.
rbfpreimg2 RBF preimage problem by Gradient optimization.
rbfpreimg3 RBF preimage problem by KwokTsang’s algorithm.
demo svm Demo on Support Vector Machines.
The 1
1
= ¦i: y
i
= 1¦ and 1
2
= ¦i: y
i
= 2¦ are sets of indices. The C > 0 stands for the
regularization constant. The ξ
i
≥ 0, i ∈ 1
1
∪ 1
2
are slack variables used to relax the
inequalities for the case of nonseparable data. The linear p = 1 and quadratic p = 2
term for the slack variables ξ
i
is used. In the case of the parameter p = 1, the L1soft
margin SVM is trained and for p = 2 it is denoted as the L2soft margin SVM.
The training of the nonlinear (kernel) SVM with L1soft margin corresponds to
solving the following QP task
β
∗
= argmax
β
¸β 1) −
1
2
¸β Hβ) , (5.6)
subject to
¸β γ) = 1 ,
β ≥ 0 ,
β ≤ 1C .
59
Table 5.2: Datatype used to describe binary SVM classiﬁer.
Binary SVM classiﬁer (structure array):
.Alpha [d 1] Weight vector α.
.b [1 1] Bias b.
.sv.X [n d] Support vectors o = ¦s
1
, . . . , s
d
¦.
.options.ker [string] Kernel identiﬁer.
.options.arg [1 p] Kernel argument(s).
.fun = ’svmclass’ Identiﬁes function associated with this data type.
Table 5.3: Datatype used to describe multiclass SVM classiﬁer.
Multiclass SVM classiﬁer (structure array):
.Alpha [d c] Weight vectors A = [α
1
, . . . , α
c
].
.b [c 1] Vector of biased b = [b
1
, . . . , b
c
]
T
.
.sv.X [n d] Support vectors o = ¦s
1
, . . . , s
d
¦.
.options.ker [string] Kernel identiﬁer.
.options.arg [1 p] Kernel argument(s).
.fun = ’svmclass’ Identiﬁes function associated with this data type.
The 0 [l 1] is vector of zeros and the 1 [l 1] is vector of ones. The vector γ [l 1]
and the matrix H [l l] are constructed as follows
γ
i
=
_
1 if y
i
= 1 ,
−1 if y
i
= 2 ,
H
i,j
=
_
k(x
i
, x
j
) if y
i
= y
j
,
−k(x
i
, x
j
) if y
i
,= y
j
,
where k: A A → R is selected kernel function. The discriminant function (5.1) is
determined by the weight vector α, bias b and the set of support vectors o. The
set of support vectors is denoted as o = ¦x
i
: i ∈ 1
SV
¦, where the set of indices is
1
SV
= ¦i: β
i
> 0¦. The weight vectors are compute as
α
i
=
_
β
i
if y
i
= 1 ,
−β
i
if y
i
= 2 ,
for i ∈ 1
SV
. (5.7)
The bias b can be computed from the KarushKuhnTucker (KKT) conditions as the
following constrains
f(x
i
) = ¸α k
S
(x
i
)) + b = +1 for i ∈ ¦j: y
j
= 1, 0 < β
j
< C¦ ,
f(x
i
) = ¸α k
S
(x
i
)) + b = −1 for i ∈ ¦j: y
j
= 2, 0 < β
j
< C¦ ,
must hold for the optimal solution. The bias b is computed as an average over all the
constrains such that
b =
1
[1
b
[
i∈I
b
γ
i
−¸α k
S
(x
i
)) .
60
Table 5.4: Datatype used to describe the majority voting strategy for multiclass SVM
classiﬁer.
Majority voting SVM classiﬁer (structure array):
.Alpha [d g] Weight vectors A = [α
1
, . . . , α
g
].
.b [g 1] Vector of biased b = [b
1
, . . . , b
g
]
T
.
.bin y [2 g] Targets for the binary rules. The vector bin y(1,:)
contains [y
1
1
, y
1
2
, . . . , y
1
g
] and the bin y(2,:) contains
[y
2
1
, y
2
2
, . . . , y
2
g
].
.sv.X [n d] Support vectors o = ¦s
1
, . . . , s
d
¦.
.options.ker [string] Kernel identiﬁer.
.options.arg [1 p] Kernel argument(s).
.fun = ’mvsvmclass’ Identiﬁes function associated with this data type.
where 1
b
= ¦i: 0 < β
i
< C¦ are the indices of the vectors on the boundary.
The training of the nonlinear (kernel) SVM with L2soft margin corresponds to
solving the following QP task
β
∗
= argmax
β
¸β 1) −
1
2
¸β (H+
1
2C
E)β) , (5.8)
subject to
¸β γ) = 1 ,
β ≥ 0 ,
where E is the identity matrix. The set of support vectors is set to o = ¦x
i
: i ∈ 1
SV
¦
where the set of indices is 1
SV
= ¦i: β
i
> 0¦. The weight vector α is given by (5.7).
The bias b can be computed from the KKT conditions as the following constrains
f(x
i
) = ¸α k
S
(x
i
)) + b = +1 −
α
i
2C
for i ∈ ¦i: y
j
= 1, β
j
> 0¦ ,
f(x
i
) = ¸α k
S
(x
i
)) + b = −1 +
α
i
2C
for i ∈ ¦i: y
j
= 2, β
j
> 0¦ ,
must hold at the optimal solution. The bias b is computed as an average over all the
constrains thus
b =
1
[1
SV
[
i∈I
SV
γ
i
(1 −
α
i
2C
) −¸α k
S
(x
i
)) .
The binary SVM solvers implemented in the STPRtool are enlisted in Table 5.5.
An interactive demo on the binary SVM is implemented in demo svm.
References: The Sequential Minimal Optimizer (SMO) is described in the books [23,
27, 30]. The SV M
light
is described for instance in the book [27].
Example: Training binary SVM.
61
Table 5.5: Implemented binary SVM solvers.
Function description
smo Implementation of the Sequential Minimal Optimizer (SMO)
used to train the binary SVM with L1soft margin. It can
handle moderate size problems.
svmlight Matlab interface to SV M
light
software (Version: 5.00) which
trains the binary SVM with L1soft margin. The executable
ﬁle svm learn for the Linux OS must be installed. It can be
downloaded from http://svmlight.joachims.org/. The
SV M
light
is very powerful optimizer which can handle large
problems.
svmquadprog SVM solver based on the QP solver quadprog of the Mat
lab optimization toolbox. It trains both L1soft and L2soft
margin binary SVM. The function quadprog is a part of the
Matlab Optimization toolbox. It useful for small problems
only.
The binary SVM is trained on the Riply’s training set riply trn. The SMO solver
is used in the example but other solvers (svmlight, svmquadprog) can by used as well.
The trained SVM classiﬁer is evaluated on the testing data riply tst. The decision
boundary is visualized in Figure 5.1.
trn = load(’riply_trn’); % load training data
options.ker = ’rbf’; % use RBF kernel
options.arg = 1; % kernel argument
options.C = 10; % regularization constant
% train SVM classifier
model = smo(trn,options);
% model = svmlight(trn,options);
% model = svmquadprog(trn,options);
% visualization
figure;
ppatterns(trn); psvm(model);
tst = load(’riply_tst’); % load testing data
ypred = svmclass(tst.X,model); % classify data
cerror(ypred,tst.y) % compute error
62
ans =
0.0920
−1.5 −1 −0.5 0 0.5 1
−0.2
0
0.2
0.4
0.6
0.8
1
1.2
Figure 5.1: Example showing the binary SVM classiﬁer trained on the Riply’s data.
5.3 OneAgainstAll decomposition for SVM
The input is a set T
XY
= ¦(x
1
, y
1
), . . . , (x
l
, y
l
)¦ of training vectors x
i
∈ A ⊆ R
n
and corresponding hidden states y
i
∈ ¸ = ¦1, 2, . . . , c¦. The goal is to train the
multiclass rule q: A ⊆ R
n
→ ¸ deﬁned by (5.3). The problem can be solved by
the OneAgainstAll (OAA) decomposition. The OAA decomposition transforms the
multiclass problem into a series of c binary subtasks that can be trained by the binary
SVM. Let the training set T
y
XY
= ¦(x
1
, y
1
), . . . , (x
l
, y
l
)¦ contain the modiﬁed hidden
states deﬁned as
y
i
=
_
1 for y = y
i
,
2 for y ,= y
i
.
The discriminant functions
f
y
(x) = ¸α
y
k
S
(x)) + b
y
, y ∈ ¸ ,
are trained by the binary SVM solver from the set T
y
XY
, y ∈ ¸. The OAA decomposition
is implemented in the function oaasvm. The function allows to specify any binary SVM
63
solver (see Section 5.2) used to solve the binary problems. The result is the multiclass
SVM classiﬁer (5.3) represented by the datatype described in Table 5.3.
References: The OAA decomposition to train the multiclass SVM and comparison
to other methods is described for instance in the paper [12].
Example: Multiclass SVM using the OAA decomposition
The example shows the training of the multiclass SVM using the OAA decomposi
tion. The SMO binary solver is used to train the binary SVM subtasks. The training
data and the decision boundary is visualized in Figure 5.2.
data = load(’pentagon’); % load data
options.solver = ’smo’; % use SMO solver
options.ker = ’rbf’; % use RBF kernel
options.arg = 1; % kernel argument
options.C = 10; % regularization constant
model = oaasvm(data,options ); % training
% visualization
figure;
ppatterns(data);
ppatterns(model.sv.X,’ko’,12);
pboundary(model);
5.4 OneAgainstOne decomposition for SVM
The input is a set T
XY
= ¦(x
1
, y
1
), . . . , (x
l
, y
l
)¦ of training vectors x
i
∈ A ⊆ R
n
and
corresponding hidden states y
i
∈ ¸ = ¦1, 2, . . . , c¦. The goal is to train the multiclass
rule q: A ⊆ R
n
→ ¸ based on the majority voting strategy deﬁned by (5.4). The
oneagainstone (OAO) decomposition strategy can be used train such a classiﬁer. The
OAO decomposition transforms the multiclass problem into a series of g = c(c −1)/2
binary subtasks that can be trained by the binary SVM. Let the training set T
j
XY
=
¦(x
1
, y
1
), . . . , (x
l
j
, y
l
j
)¦ contain the training vectors x
i
∈ 1
j
= ¦i: y
i
= y
1
_
y
i
= y
2
¦
and the modiﬁed the hidden states deﬁned as
y
i
=
_
1 for y
1
j
= y
i
,
2 for y
2
j
= y
i
,
i ∈ 1
j
.
The training set T
j
XY
, j = 1, . . . , g is constructed for all g = c(c −1)/2 combinations of
classes y
1
j
∈ ¸ and y
2
j
∈ ¸ ¸¦y
1
j
¦. The binary SVM rules q
j
, j = 1, . . . , g are trained on
the data T
j
XY
. The OAO decomposition for multiclass SVM is implemented in function
64
−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1
−1
−0.8
−0.6
−0.4
−0.2
0
0.2
0.4
0.6
0.8
1
Figure 5.2: Example showing the multiclass SVM classiﬁer build by oneagainstall
decomposition.
oaosvm. The function oaosvm allows to specify an arbitrary binary SVM solver to train
the subtasks.
References: The OAO decomposition to train the multiclass SVM and comparison
to other methods is described for instance in the paper [12].
Example: Multiclass SVM using the OAO decomposition
The example shows the training of the multiclass SVM using the OAO decomposi
tion. The SMO binary solver is used to train the binary SVM subtasks. The training
data and the decision boundary is visualized in Figure 5.3.
data = load(’pentagon’); % load data
options.solver = ’smo’; % use SMO solver
options.ker = ’rbf’; % use RBF kernel
options.arg = 1; % kernel argument
options.C = 10; % regularization constant
model = oaosvm(data,options); % training
% visualization
figure;
ppatterns(data);
ppatterns(model.sv.X,’ko’,12);
pboundary(model);
65
−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1
−1
−0.8
−0.6
−0.4
−0.2
0
0.2
0.4
0.6
0.8
1
Figure 5.3: Example showing the multiclass SVM classiﬁer build by oneagainstone
decomposition.
5.5 Multiclass BSVM formulation
The input is a set T
XY
= ¦(x
1
, y
1
), . . . , (x
l
, y
l
)¦ of training vectors x
i
∈ A ⊆ R
n
and
corresponding hidden states y
i
∈ ¸ = ¦1, 2, . . . , c¦. The goal is to train the multiclass
rule q: A ⊆ R
n
→ ¸ deﬁned by (5.3). In the linear case, the parameters of the multi
class rule can be found by solving the multiclass BSVM formulation which is deﬁned
as
(W
∗
, b
∗
, ξ
∗
) = argmin
W,b
1
2
y∈Y
(w
y

2
+ b
2
y
) + C
i∈I
y∈Y\{y}
(ξ
i
)
p
. ¸¸ .
F(W,b,ξ)
, (5.9)
subject to
¸w
y
i
x
i
) + b
y
i
−(¸w
y
x
i
) + b
y
) ≥ 1 −ξ
y
i
, i ∈ 1 , y ∈ ¸ ¸ ¦y
i
¦ ,
ξ
y
i
≥ 0 , i ∈ 1 , y ∈ ¸ ¸ ¦y
i
¦ ,
where W = [w
1
, . . . , w
c
] is a matrix of normal vectors, b = [b
1
, . . . , b
c
]
T
is vector of
biases, ξ is a vector of slack variables and 1 = [1, . . . , l] is set of indices. The L1soft
margin is used for p = 1 and L2soft margin for p = 2. The letter B in BSVM stands
for the bias term added to the objective (5.9).
In the case of the L2soft margin p = 2, the optimization problem (5.9) can be
66
expressed in its dual form
A
∗
= argmax
A
i∈I
y∈Y\{y
i
}
α
y
i
−
1
2
i∈I
y∈Y\{y
i
}
j∈I
u∈Y\{y
j
}
α
y
i
α
u
j
h(y, u, i, j) , (5.10)
subject to
α
y
i
≥ 0 , i ∈ 1 , y ∈ ¸ ¸ ¦y
i
¦ ,
where A = [α
1
, . . . , α
c
] are optimized weight vectors. The function h: ¸¸11 →R
is deﬁned as
h(y, u, i, j) = (¸x
i
x
j
)+1)(δ(y
i
, y
j
)+δ(y, u)−δ(y
i
, u)−δ(y
j
, y))+
δ(y, u)δ(i, j)
2C
. (5.11)
The criterion (5.10) corresponds to the singleclass SVM criterion which is simple to
optimize. The multiclass rule (5.3) is composed of the set of discriminant functions
f
y
: A →R which are computed as
f
y
(x) =
i∈I
¸x
i
x)
u∈Y\{y
i
}
α
u
i
(δ(u, y
i
) −δ(u, y)) + b
y
, y ∈ ¸ , (5.12)
where the bias b
y
, y ∈ ¸ is given by
b
y
=
i∈I
y∈Y\{y
i
}
α
u
i
(δ(y, y
i
) −δ(y, u)) , y ∈ ¸ .
The nonlinear (kernel) classiﬁer is obtained by substituting the selected kernel k: A
A →R for the dot products ¸xx
) to (5.11) and (5.12). The training of the multiclass
BSVM classiﬁer with L2soft margin is implemented in the function bsvm2. The opti
mization task (5.10) can be solved by most the optimizers. The optimizers implemented
in the function bsvm2 are enlisted bellow:
MitchellDemyanovMalozemov (solver = ’mdm’).
Kozinec’s algorithm (solver = ’kozinec’).
Nearest Point Algorithm (solver = ’npa’).
The above mentioned algorithms are of iterative nature and converge to the optimal
solution of the criterion (5.9). The upper bound F
UB
and the lower bound F
LB
on the
optimal value F(W
∗
, b
∗
, ξ
∗
) of the criterion (5.9) can be computed. The bounds are
used in the algorithms to deﬁne the following two stopping conditions:
Relative tolerance:
F
UB
−F
LB
F
LB
+ 1
≤ ε
rel
.
67
Absolute tolerance: F
UB
≤ ε
abs
.
References: The multiclass BSVM formulation is described in paper [12]. The trans
formation of the multiclass BSVM criterion to the singleclass criterion (5.10) imple
mented in the STPRtool is published in paper [9]. The MitchellDemyanovMalozemov
and Nearest Point Algorithm are described in paper [14]. The Kozinec’s algorithm
based SVM solver is described in paper [10].
Example: Multiclass BSVM with L2soft margin.
The example shows the training of the multiclass BSVM using the NPA solver. The
other solvers can be used instead by setting the variable options.solver (other options
are ’kozinec’, ’mdm’). The training data and the decision boundary is visualized in
Figure 5.4.
data = load(’pentagon’); % load data
options.solver = ’npa’; % use NPA solver
options.ker = ’rbf’; % use RBF kernel
options.arg = 1; % kernel argument
options.C = 10; % regularization constant
model = bsvm2(data,options); % training
% visualization
figure;
ppatterns(data);
ppatterns(model.sv.X,’ko’,12);
pboundary(model);
5.6 Kernel Fisher Discriminant
The input is a set T
XY
= ¦(x
1
, y
1
), . . . , (x
l
, y
l
)¦ of training vectors x
i
∈ A ⊆ R
n
and
corresponding values of binary state y
i
∈ ¸ = ¦1, 2¦. The Kernel Fisher Discriminant
is the nonlinear extension of the linear FLD (see Section 2.3). The input training
vectors are assumed to be mapped into a new feature space T by a mapping function
φ: A → T. The ordinary FLD is applied on the mapped data training data T
ΦY
=
¦(φ(x
1
), y
1
), . . . , (φ(x
l
), y
l
)¦. The mapping is performed eﬃciently by means of kernel
functions k: A A → R being the dot products of the mapped vectors k(x, x
) =
¸φ(x) φ(x
)). The aim is to ﬁnd a direction ψ =
l
i=1
α
i
φ(x
i
) in the feature space
T given by weights α = [α
1
, . . . , α
l
]
T
such that the criterion
F(α) =
¸α Mα)
¸α (N+ µE)α)
=
¸α m)
2
¸α (N+ µE)α)
, (5.13)
68
−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1
−1
−0.8
−0.6
−0.4
−0.2
0
0.2
0.4
0.6
0.8
1
Figure 5.4: Example showing the multiclass BSVM classiﬁer with L2soft margin.
is maximized. The criterion (5.13) is the feature space counterpart of the crite
rion (2.13) deﬁned in the input space A. The vector m [l 1], matrices N [l l]
M [l l] are constructed as
m
y
=
1
[1
y
[
K1
y
, y ∈ ¸ , m = m
1
−m
2
,
N = KK
T
−
y∈Y
[1
y
[m
y
m
T
y
, M = (m
1
−m
2
)(m
1
−m
2
)
T
,
where the vector 1
y
has entries i ∈ 1
y
equal to and remaining entries zeros. The kernel
matrix K [l l] contains kernel evaluations K
i,j
= k(x
i
, x
j
). The diagonal matrix
µE in the denominator of the criterion (5.13) serves as the regularization term. The
discriminant function f(x) of the binary classiﬁer (5.2) is given by the found vector α
as
f(x) = ¸ψ φ(x)) + b = ¸α k(x)) + b , (5.14)
where the k(x) = [k(x
1
, x), . . . , k(x
l
, x)]
T
is vector of evaluations of the kernel. The
discriminant function (5.14) is known up to the bias b which is determined by the linear
SVM with L1soft margin is applied on the projected data T
ZY
= ¦(z
1
, y
1
), . . . , (z
l
, y
l
)¦.
The projections ¦z
1
, . . . , z
l
¦ are computed as
z
i
= ¸α k(x
i
)) .
The KFD training algorithm is implemented in the function kfd.
69
References: The KFD formulation is described in paper [19]. The detailed analysis
can be found in book [30].
Example: Kernel Fisher Discriminant
The binary classiﬁer is trained by the KFD on the Riply’s training set riply trn.mat.
The found classiﬁer is evaluated on the testing data riply tst.mat. The training data
and the found kernel classiﬁer is visualized in Figure 5.5. The classiﬁer is visualized as
the SVM classiﬁer, i.e., the isolines with f(x) = +1 and f(x) = −1 are visualized and
the support vectors (whole training set) are marked as well.
trn = load(’riply_trn’); % load training data
options.ker = ’rbf’; % use RBF kernel
options.arg = 1; % kernel argument
options.C = 10; % regularization of linear SVM
options.mu = 0.001; % regularization of KFD criterion
model = kfd(trn, options) % KFD training
% visualization
figure;
ppatterns(trn);
psvm(model);
% evaluation on testing data
tst = load(’riply_tst’);
ypred = svmclass( tst.X, model );
cerror( ypred, tst.y )
ans =
0.0960
5.7 Preimage problem
Let T
X
= ¦x
1
, . . . , x
l
¦ be a set of vector from the input space A ⊆ R
n
. The kernel
methods works with the data nonlinearly mapped φ: A → T into a new feature space
T. The mapping is performed implicitly by using the kernel functions k: A A → R
which are dot products of the input data mapped into the feature space T such that
k(x
a
, x
b
) = ¸φ(x
a
) φ(x
b
)). Let ψ ∈ T be given by the following kernel expansion
ψ =
l
i=1
α
i
φ(x
i
) , (5.15)
70
−1.5 −1 −0.5 0 0.5 1
−0.2
0
0.2
0.4
0.6
0.8
1
1.2
Figure 5.5: Example shows the binary classiﬁer trained based on the Kernel Fisher
Discriminant.
where α = [α
1
, . . . , α
l
]
T
is a weight vector of real coeﬃcients. The preimage problem
aims to ﬁnd an input vector x ∈ A, its feature space image φ(x) ∈ T would well
approximate the expansion point Ψ ∈ T given by (5.15). In other words, the pre
image problem solves mapping from the feature space T back to the input space A.
The preimage problem can be stated as the following optimization task
x = argmin
x
φ(x) −ψ
2
= argmin
x
φ(x
) −
l
i=1
α
i
φ(x
i
)
2
= argmin
x
k(x
, x
) −2
l
i=1
α
i
k(x
, x
i
) +
l
i=1
l
j=1
α
i
α
j
k(x
i
, x
j
) .
(5.16)
Let the Radial Basis Function (RBF) k(x
a
, x
b
) = exp(−0.5x
a
−x
b

2
/σ
2
) be assumed.
The preimage problem (5.16) for the particular case of the RBF kernel leads to the
following task
x = argmax
x
l
i=1
α
i
exp(−0.5x
−x
i

2
/σ
2
) . (5.17)
The STPRtool provides three optimization algorithms to solve the RBF preimage
problem (5.17) which are enlisted in Table 5.6. However, all the methods are guaranteed
to ﬁnd only a local optimum of the task (5.17).
71
Table 5.6: Methods to solve the RBF preimage problem implemented in the STPRtool.
rbfpreimg An iterative ﬁxed point algorithm proposed in [28].
rbfpreimg2 A standard gradient optimization method using the
Matlab optimization toolbox for 1D search along the
gradient.
rbfpreimg3 A method exploiting the correspondence between dis
tances in the input space and the feature space pro
posed in [16].
References: The preimage problem and the implemented algorithms for its solution
are described in [28, 16].
Example: Preimage problem
The example shows how to visualize a preimage of the data mean in the feature
induced by the RBF kernel. The result is visualized in Figure 5.6.
% load input data
data = load(’riply_trn’);
% define the mean in the feature space
num_data = size(data,2);
expansion.Alpha = ones(num_data,1)/num_data;
expansion.sv.X = data.X;
expansion.options.ker = ’rbf’;
expansion.options.arg = 1;
% compute preimage problem
x = rbfpreimg(expansion);
% visualization
figure;
ppatterns(data.X);
ppatterns(x,’or’,13);
72
−1.5 −1 −0.5 0 0.5 1
−0.2
0
0.2
0.4
0.6
0.8
1
1.2
Figure 5.6: Example shows training data and the preimage of their mean in the feature
space induced by the RBF kernel.
5.8 Reduced set method
The reduced set method is useful to simplify the kernel classiﬁer trained by the SVM
or other kernel machines. Let the function f: A ⊆ R
n
→R be given as
f(x) =
l
i=1
α
i
k(x
i
, x) + b = ¸ψ φ(x)) + b , (5.18)
where T
X
= ¦x
1
, . . . , x
l
¦ are vectors from R
n
, k: A A → R is a kernel function,
α = [α
1
, . . . , α
l
]
T
is a weight vector of real coeﬃcients and b ∈ R is a scalar. Let
φ: A → T be a mapping from the input space A to the feature space T associated
with kernel function such that k(x
a
, x
b
) = ¸φ(x
a
) φ(x
b
)). The function f(x) can be
expressed in the feature space T as the linear function f(x) = ¸ψ φ(x)) + b. The
reduced set method aims to ﬁnd a new function
˜
f(x) =
m
i=1
β
i
k(s
i
, x) + b = ¸φ(x)
m
i=1
β
i
φ(s
i
)
. ¸¸ .
˜
ψ
) + b , (5.19)
such that the expansion (5.19) is shorter, i.e., m < l, and well approximates the original
one (5.18). The weight vector β = [β
1
, . . . , β
m
]
T
and the vectors T
S
= ¦s
1
, . . . , s
m
¦,
s
i
∈ R
n
determine the reduced kernel expansion (5.19). The problem of ﬁnding the
73
reduced kernel expansion (5.19) can be stated as the optimization task
(β, T
S
) = argmin
β
,T
S

˜
ψ −ψ
2
= argmin
β
,T
S
_
_
_
_
_
m
i=1
β
i
φ(s
i
) −
l
i=1
α
i
φ(x
i
)
_
_
_
_
_
2
. (5.20)
Let the Radial Basis Function (RBF) k(x
a
, x
b
) = exp(−0.5x
a
−x
b

2
/σ
2
) be assumed.
In this case, the optimization task (5.20) can be solved by an iterative greedy algo
rithm. The algorithm converts the task (5.20) into a series of m preimage tasks (see
Section 5.7). The reduced set method for the RBF kernel is implemented in function
rsrbf.
References: The implemented reduced set method is described in [28].
Example: Reduced set method for SVM classiﬁer
The example shows how to use the reduced set method to simplify the SVM clas
siﬁer. The SVM classiﬁer is trained from the Riply’s training set riply trn/mat. The
trained kernel expansion (discriminant function) is composed of 94 support vectors.
The reduced expansion with 10 support vectors is found. Decision boundaries of the
original and the reduced SVM are visualized in Figure 5.7. The original and reduced
SVM are evaluated on the testing data riply tst.mat.
% load training data
trn = load(’riply_trn’);
% train SVM classifier
model = smo(trn,struct(’ker’,’rbf’,’arg’,1,’C’,10));
% compute the reduced rule with 10 support vectors
red_model = rsrbf(model,struct(’nsv’,10));
% visualize decision boundaries
figure; ppatterns(trn);
h1 = pboundary(model,struct(’line_style’,’r’));
h2 = pboundary(red_model,struct(’line_style’,’b’));
legend([h1(1) h2(1)],’Original SVM’,’Reduced SVM’);
% evaluate SVM classifiers
tst = load(’riply_tst’);
ypred = svmclass(tst.X,model);
err = cerror(ypred,tst.y);
red_ypred = svmclass(tst.X,model);
74
red_err = cerror(red_ypred,tst.y);
fprintf([’Original SVM: nsv = %d, err = %.4f\n’ ...
’Reduced SVM: nsv = %d, err = %.4f\n’], ...
model.nsv, err, red_model.nsv, red_err);
Original SVM: nsv = 94, err = 0.0960
Reduced SVM: nsv = 10, err = 0.0960
−1.5 −1 −0.5 0 0.5 1
−0.2
0
0.2
0.4
0.6
0.8
1
1.2
Original SVM
Reduced SVM
Figure 5.7: Example shows the decision boundaries of the SVM classiﬁer trained by
SMO and the approximated SVM classiﬁer found by the reduced set method. The
original decision rule involves 94 support vectors while the reduced one only 10 support
vectors.
75
Chapter 6
Miscellaneous
6.1 Data sets
The STPRtool provides several datasets and functions to generate synthetic data. The
datasets stored in the Matlab data ﬁle are enlisted in Table 6.1. The functions which
generate synthetic data and scripts for converting external datasets to the Matlab are
given in Table 6.2.
Table 6.1: Datasets stored in /data/ directory of the STPRtool.
/riply trn.mat Training part of the Riply’s data set [24].
/riply tst.mat Testing part of the Riply’s data set [24].
/iris.mat Well known Fisher’s data set.
/anderson task/*.mat Input data for the demo on the Generalized Ander
son’s task demo anderson.
/binary separable/*.mat Input data for the demo on training linear classiﬁers
from ﬁnite vector sets demo linclass.
/gmm samles/*.mat Input data for the demo on the Expectation
Maximization algorithm for the Gaussian mixture
model demo emgmm.
/multi separable/*.mat Linearly separable multiclass data in 2D.
/ocr data/*.mat Examples of handwritten numerals. A description is
given in Section 7.1.
/svm samples/*.mat Input data for the demo on the Support Vector Ma
chines demo svm.
76
Table 6.2: Data generation.
createdata GUI which allows to create interactively labeled vec
tors sets and labeled sets of Gaussian distributions. It
operates in 2dimensional space only.
genlsdata Generates linearly separable binary data with pre
scribed margin.
gmmsamp Generates data from the Gaussian mixture model.
gsamp Generates data from the multivariate Gaussian distri
bution.
gencircledata Generates data on circle corrupted by the Gaussian
noise.
usps2mat Converts U.S. Postal Service (USPS) database of
handwritten numerals to the Matlab ﬁle.
6.2 Visualization
The STPRtool provides tools for visualization of common models and data. The list
of functions for visualization is given in Table 6.3. Each function contains an example
of using.
Table 6.3: Functions for visualization.
pandr Visualizes solution of the Generalized Anderson’s
task.
pboundary Plots decision boundary of given classiﬁer in 2D.
pgauss Visualizes set of bivariate Gaussians.
pgmm Visualizes the bivariate Gaussian mixture model.
pkernelproj Plots isolines of kernel projection.
plane3 Plots plane in 3D.
pline Plots line in 2D.
ppatterns Visualizes patterns as points in the feature space. It
works for 1D, 2D and 3D.
psvm Plots decision boundary of binary SVM classiﬁer in
2D.
showim Displays images entered as column vectors.
77
6.3 Bayesian classiﬁer
The object under study is assumed to be described by a vector of observations x ∈ A
and a hidden state y ∈ ¸. The x and y are realizations of random variables with joint
probability distribution P
XY
(x, y). A decision rule q: A → T takes a decision d ∈ T
based on the observation x ∈ A. Let W: T¸ →R be a loss function which penalizes
the decision q(x) ∈ T when the true hidden state is y ∈ ¸. Let A ⊆ R
n
and the sets
¸ and T be ﬁnite. The Bayesian risk R(q) is an expectation of the value of the loss
function W when the decision rule q is applied, i.e.,
R(q) =
_
X
y∈Y
P
XY
(x, y)W(q(x), y) dx . (6.1)
The optimal rule q
∗
which minimizes the Bayesian risk (6.1) is referred to as the
Bayesian rule
q
∗
(x) = argmin
y
y∈Y
P
XY
(x, y)W(q(x), y) , ∀x ∈ A .
The STPRtool implements the Bayesian rule for two particular cases:
Minimization of misclassiﬁcation: The set of decisions T coincides to the set of
hidden states ¸ = ¦1, . . . , c¦. The 0/1loss function
W
0/1
(q(x), y) =
_
0 for q(x) = y ,
1 for q(x) ,= y .
(6.2)
is used. The Bayesian risk (6.1) with the 0/1loss function corresponds to the
expectation of misclassiﬁcation. The rule q: A → ¸ which minimizes the expec
tation of misclassiﬁcation is deﬁned as
q(x) = argmax
y∈Y
P
Y X
(y[x) ,
= argmax
y∈Y
P
XY
(x[y)P
Y
(y) .
(6.3)
Classiﬁcation with rejectoption: The set of decisions T is assumed to be T =
¸ ∪ ¦dont know¦. The loss function is deﬁned as
W
ε
(q(x), y) =
⎧
⎨
⎩
0 for q(x) = y ,
1 for q(x) ,= y ,
ε for q(x) = dont know ,
(6.4)
where ε is penalty for the decision dont know. The rule q: A → ¸ which mini
mizes the Bayesian risk with the loss function (6.4) is deﬁned as
q(x) =
⎧
⎨
⎩
argmax
y∈Y
P
XY
(x[y)P
Y
(y) if 1 −max
y∈Y
P
Y X
(y[x) < ε ,
dont know if 1 −max
y∈Y
P
Y X
(y[x) ≥ ε .
(6.5)
78
To apply the optimal classiﬁcation rules one has to know the classconditional
distributions P
XY
and a priory distribution P
Y
(or their estimates). The datatype
used by the STPRtool is described in Table 6.4. The Bayesian classiﬁers (6.3) and (6.5)
are implemented in function bayescls.
Table 6.4: Datatype used to describe the Bayesian classiﬁer.
Bayesian classiﬁer (structure array):
.Pclass [cell 1 c] Class conditional distributions P
XY
. The distribution
P
XY
(x[y) is given by Pclassy which uses the data
type described in Table 4.2.
.Prior [1 c] Discrete distribution P
Y
(y), y ∈ ¸.
.fun = ’bayescls’ Identiﬁes function associated with this data type.
References: The Bayesian decision making with applications to PR is treated in
details in book [26].
Example: Bayesian classiﬁer
The example shows how to design the plugin Bayesian classiﬁer for the Riply’s
training set riply trn.mat. The classconditional distributions P
XY
are modeled by
the Gaussian mixture models (GMM) with two components. The GMM are estimated
by the EM algorithm emgmm. The a priori probabilities P
Y
(y), y ∈ ¦1, 2¦ are estimated
by the relative occurrences (it corresponds to the ML estimate). The designed Bayesian
classiﬁer is evaluated on the testing data riply tst.mat.
% load input training data
trn = load(’riply_trn’);
inx1 = find(trn.y==1);
inx2 = find(trn.y==2);
% Estimation of classconditional distributions by EM
bayes_model.Pclass{1} = emgmm(trn.X(:,inx1),struct(’ncomp’,2));
bayes_model.Pclass{2} = emgmm(trn.X(:,inx2),struct(’ncomp’,2));
% Estimation of priors
n1 = length(inx1); n2 = length(inx2);
bayes_model.Prior = [n1 n2]/(n1+n2);
% Evaluation on testing data
tst = load(’riply_tst’);
ypred = bayescls(tst.X,bayes_model);
cerror(ypred,tst.y)
79
ans =
0.0900
The following example shows how to visualize the decision boundary of the found
Bayesian classiﬁer minimizing the misclassiﬁcation (solid black line). The Bayesian
classiﬁer splits the feature space into two regions corresponding to the ﬁrst class and
the second class. The decision boundary of the rejectoptions rule (dashed line) is
displayed for comparison. The rejectoption rule splits the feature space into three
regions corresponding to the ﬁrst class, the second class and the region in between
corresponding to the dont know decision. The visualization is given in Figure 6.1.
% Visualization
figure; hold on; ppatterns(trn);
bayes_model.fun = ’bayescls’;
pboundary(bayes_model);
% Penalization for don’t know decision
reject_model = bayes_model;
reject_model.eps = 0.1;
% Vislualization of rejetoption rule
pboundary(reject_model,struct(’line_style’,’k’));
6.4 Quadratic classiﬁer
The quadratic classiﬁcation rule q: A ⊆ R
n
→ ¸ = ¦1, 2, . . . , c¦ is composed of a set
of discriminant functions
f
y
(x) = ¸x A
y
x) +¸b
y
x) + c
y
, ∀y ∈ ¸ ,
which are quadratic with respect to the input vector x ∈ R
n
. The quadratic discrimi
nant function f
y
is determined by a matrix A
y
[n n], a vector b
y
[n 1] and a scalar
c
y
[1 1]. The input vector x ∈ R
n
is assigned to the class y ∈ ¸ its corresponding
discriminant function f
y
attains maximal value
y = argmax
y
∈Y
f
y
(x) = argmax
y
∈Y
(¸x A
y
x) +¸b
y
x) + c
y
) . (6.6)
In the particular binary case ¸ = ¦1, 2¦, the quadratic classiﬁer is represented by
a single discriminant function
f(x) = ¸x Ax) +¸b x) + c ,
80
−1.5 −1 −0.5 0 0.5 1
−0.2
0
0.2
0.4
0.6
0.8
1
1.2
Figure 6.1: Figure shows the decision boundary of the Bayesian classiﬁer (solid line)
and the decision boundary of the rejectoption rule with ε = 0.1 (dashed line). The
classconditional probabilities are modeled by the Gaussian mixture models with two
components estimated by the EM algorithm.
given by a matrix A [n n], a vector b [n 1] and a scalar c [1 1]. The input vector
x ∈ R
n
is assigned to class y ∈ ¦1, 2¦ as follows
q(x) =
_
1 if f(x) = ¸x Ax) +¸b x) + c ≥ 0 ,
2 if f(x) = ¸x Ax) +¸b x) + c < 0 .
(6.7)
The STPRtool uses a speciﬁc structure array to describe the binary (see Table 6.5)
and the multiclass (see Table 6.6) quadratic classiﬁer. The implementation of the
quadratic classiﬁer itself provides function quadclass.
Table 6.5: Datatype used to describe binary quadratic classiﬁer.
Binary linear quadratic (structure array):
.A [n n] Parameter matrix A.
.b [n 1] Parameter vector b.
.c [1 1] Parameter scalar c.
.fun = ’quadclass’ Identiﬁes function associated with this data type.
Example: Quadratic classiﬁer
The example shows how to train the quadratic classiﬁer using the Fisher Linear Dis
criminant and the quadratic data mapping. The classiﬁer is trained from the Riply’s
81
Table 6.6: Datatype used to describe multiclass quadratic classiﬁer.
Multiclass quadratic classiﬁer (structure array):
.A [n n c] Parameter matrices A
y
, y ∈ ¸.
.b [n c] Parameter vectors b
y
, y ∈ ¸.
.c [1 c] Parameter scalars c
y
, y ∈ ¸.
.fun = ’quadclass’ Identiﬁes function associated with this data type.
training set riply trn.mat. The found quadratic classiﬁer is evaluated on the test
ing data riply trn.mat. The boundary of the quadratic classiﬁer is visualized (see
Figure 6.2).
trn = load(’riply_trn’); % load input data
quad_data = qmap(trn); % quadratic mapping
lin_model = fld(quad_data); % train FLD
% make quadratic classifier
quad_model = lin2quad(lin_model);
% visualization
figure;
ppatterns(trn); pboundary(quad_model);
% evaluation
tst = load(’riply_tst’);
ypred = quadclass(tst.X,quad_model);
cerror(ypred,tst.y)
ans =
0.0940
6.5 Knearest neighbors classiﬁer
Let T
XY
= ¦(x
1
, y
1
), . . . , (x
l
, y
l
)¦ be a set of prototype vectors x
i
∈ A ⊆ R
n
and
corresponding hidden states y
i
∈ ¸ = ¦1, . . . , c¦. Let R
n
(x) = ¦x
: x −x
 ≤ r
2
¦ be
a ball centered in the vector x in which lie K prototype vectors x
i
, i ∈ ¦1, . . . , l¦, i.e.,
[¦x
i
: x
i
∈ R
n
(x)¦[ = K. The Knearest neighbor (KNN) classiﬁcation rule q: A → ¸
is deﬁned as
q(x) = argmax
y∈Y
v(x, y) , (6.8)
82
−1.5 −1 −0.5 0 0.5 1
−0.2
0
0.2
0.4
0.6
0.8
1
1.2
Figure 6.2: Figure shows the decision boundary of the quadratic classiﬁer.
where v(x, y) is number of prototype vectors x
i
with hidden state y
i
= y which lie in
the ball x
i
∈ R
n
(x). The classiﬁcation rule (6.8) is computationally demanding if the
set of prototype vectors is large. The datatype used by the STPRtool to represent the
Knearest neighbor classiﬁer is described in Table 6.7.
Table 6.7: Datatype used to describe the Knearest neighbor classiﬁer.
KNN classiﬁer (structure array):
.X [n l] Prototype vectors ¦x
1
, . . . , x
l
¦.
.y [1 l] Labels ¦y
1
, . . . , y
l
¦.
.fun = ’knnclass’ Identiﬁes function associated with this data type.
References: The Knearest neighbors classiﬁer is described in most books on PR, for
instance, in book [6].
Example: Knearest neighbor classiﬁer
The example shows how to create the (K = 8)NN classiﬁcation rule from the
Riply’s training data riply trn.mat. The classiﬁer is evaluated on the testing data
riply tst.mat. The decision boundary is visualized in Figure 6.3.
% load training data and setup 8NN rule
trn = load(’riply_trn’);
model = knnrule(trn,8);
83
% visualize decision boundary and training data
figure; ppatterns(trn); pboundary(model);
% evaluate classifier
tst = load(’riply_tst’);
ypred = knnclass(tst.X,model);
cerror(ypred,tst.y)
ans =
0.1160
−1.5 −1 −0.5 0 0.5 1
−0.2
0
0.2
0.4
0.6
0.8
1
1.2
Figure 6.3: Figure shows the decision boundary of the (K = 8)nearest neighbor
classiﬁer.
84
Chapter 7
Examples of applications
We intend to demonstrate how methods implemented in the STPRtool can be used to
solve a more complex applications. The applications selected are the optical charac
ter recognition (OCR) system based on the multiclass SVM (Section 7.1) and image
denoising system using the kernel PCA (Section 7.2).
7.1 Optical Character Recognition system
The use of the STPRtool is demonstrated on a simple Optical Character Recognition
(OCR) system for handwritten numerals from 0 to 9. The STPRtool provides GUI
which allows to use standard computer mouse as an input device to draw images of
numerals. The GUI is intended just for demonstration purposes and the scanner or
other device would be used in practice instead. The multiclass SVM are used as the
base classiﬁer of the numerals. A numeral to be classiﬁed is represented as a vector
x containing pixel values taken from the numeral image. The training stage of the
OCR system involves (i) collection of training examples of numerals and (ii) training
the multiclass SVM. To see the trained OCR system run the demo script demo ocr.
The GUI for drawing numerals is implemented in a function mpaper. The function
mpaper creates a form composed of 50 cells to which the numerals can be drawn (see
Figure 7.1a). The drawn numerals are normalized to size 16 16 pixels (diﬀerent
size can be speciﬁed). The function mpaper is designed to invoke a speciﬁed function
whenever the middle mouse button is pressed. In particular, the functions save chars
and ocr fun are called from the mpaper to process drawn numerals. The function
save chars saves drawn numerals in the stage of collecting training examples and the
function ocr fun is used to recognize the numerals if the trained OCR is applied.
The set of training examples has to be collected before the OCR is trained. The
function collect chars invokes the form for drawing numerals and allows to save the
numerals to a speciﬁed ﬁle. The STPRtool contains examples of numerals collected
85
from 6 diﬀerent persons. Each person drew 50 examples of each of numerals from
’0’ to ’9’. The numerals are stored in the directory /data/ocr numerals/. The ﬁle
name format name xx.mat (the character can be omitted from the name) is used.
The string name is name of the person who drew the particular numeral and xx stands
for a label assigned to the numeral. The labels from y = 1 to y = 9 corresponds to the
numerals from ’1’ to ’9’ and the label y = 10 is used for numeral ’0’. For example,
to collect examples of numeral ’1’ from person honza cech
collect_chars(’honza_cech1’)
was called. Figure 7.1a shows the form ﬁlled with the examples of the numeral ’1’.
The Figure 7.1b shows numerals after normalization which are saved to the speci
ﬁed ﬁle honza cech.mat. The images of normalized numerals are stored as vectors
to a matrix X of size [256 50]. The vector dimension n = 256 corresponds to
16 16 pixels of each image numeral and l = 50 is the maximal number of numerals
drawn to a single form. If the ﬁle name format name xx.mat is met then the function
mergesets(Directory,DataFile) can be used to merge all the ﬁle to a single one.
The resulting ﬁle contains a matrix X of all examples and a vector y containing labels
are assigned to examples according the number xx. The string Directory speciﬁes the
directory where the ﬁles name xx.mat are stored and the string FileName is a ﬁle name
to which the resulting training set is stored.
Having the labeled training set T
XY
= ¦(x
1
, y
1
), . . . , (x
l
, y
l
)¦ created the multiclass
SVM classiﬁer can be trained. The methods to train the multiclass SVM are described
in Chapter 5. In particular, the OneAgainstOne (OAO) decomposition was used as
it requires the shortest training time. However, if a speed of classiﬁcation is the major
objective the multiclass BSVM formulation is often a better option as it produce less
number of support vectors. The SVM has two free parameters which have to be tuned:
the regularization constant C and the kernel function k(x
a
, x
b
) with its argument.
The RBF kernel k(x
a
, x
b
) = exp(−0.5x
a
− x
b

2
/σ
2
) determined by the parameter
σ was used in the experiment. Therefore, the free SVM parameters are C and σ. A
common method to tune the free parameters is to use the crossvalidation to select the
best parameters from a preselected set Θ = ¦(C
1
, σ
1
), . . . , (C
k
, σ
k
)¦. The STPRtool
contains a function evalsvm which evaluates the set of SVM parameters Θ based on the
crossvalidation estimation of the classiﬁcation error. The function evalsvm returns the
SVM model its parameters (C
∗
, σ
∗
) turned out to be have the lowest crossvalidation
error over the given set Θ. The parameter tuning is implemented in the script tune ocr.
The parameters C = ∞ (data are separable) and σ = 5 were found to be the best.
After the parameters (C
∗
, σ
∗
) are tuned the SVM classiﬁer is trained using all
training data available. The training procedure for OCR classiﬁer is implemented in
the function train ocr. The directory /demo/ocr/models/ contains the SVM model
trained by OAO decomposition strategy (function oaosvm) and model trained based
on the multiclass BSVM formulation (function bsvm2).
86
The particular model used by OCR is speciﬁed in the function ocr fun which
implicitly uses the ﬁle ocrmodel.mat stored in the directory /demo/ocr/. To run the
resulting OCR type demo ocr or call
mpaper(struct(’fun’,’ocr_fun’))
If one intends to modify the classiﬁer or the way how the classiﬁcation results are
displayed then modify the function ocr fun. A snapshot of running OCR is shown in
Figure 7.2.
7.2 Image denoising
The aim is to train a model of a given class of images in order to restore images
corrupted by noise. The U.S. Postal Service (USPS) database of images of hand
written numerals [17] was used as the class of images to model. The additive Gaussian
noise was assumed as the source of image degradation. The kernel PCA was used to
model the image class. This method was proposed in [20] and further developed in [15].
The STPRtool contains a script which converts the USPS database to the Matlab
data ﬁle. It is implemented in the function usps2mat. The corruption of the USPS
images by the Gaussian noise is implemented in script make noisy data. The script
make noisy data produces the Matlab ﬁle usps noisy.mat its contains is summarized
in Table 7.1.
The aim is to train a model of images of all numerals at ones. Thus the unlabeled
training data set T
X
= ¦x
1
, . . . , x
l
¦ is used. The model should describe the nonlinear
subspace A
img
⊆ R
n
where the images live. It is assumed that the nonlinear subspace
A
img
forms a linear subspace T
img
of a new feature space T. A link between the
input space A and the feature space T is given by a mapping φ: A → T which is
implicitly determined by a selected kernel function k: A A →R. The kernel function
corresponds to the dot product in the feature space, i.e., k(x
a
, x
b
) = ¸φ(x
a
) φ(x
b
)).
The kernel PCA can be used to ﬁnd the basis vectors of the subspace T
img
from the
input training data T
X
. Having the kernel PCA model of the subspace T
img
it can be
used to map the images x ∈ A into the subspace A
img
. A noisy image ˆ x is assumed to
lie outside the subspace A
img
. The denoising procedure is based on mapping the noisy
image ˆ x into the subspace A
img
. The idea of using the kernel PCA for image denoising
is demonstrated in Figure 7.3. The example which produced the ﬁgure is implemented
in demo kpcadenois.
The subspace T
img
is modeled by the kernel PCA trained from the set T
X
mapped
into the feature space T by a function φ: A → T. Let T
Φ
= ¦φ(x
1
), . . . , φ(x
l
)¦ denote
the feature space representation of the training images T
X
. The kernel PCA computes
the lower dimensional representation T
Z
= ¦z
1
, . . . , z
l
¦, z
i
∈ R
m
of the mapped images
87
Table 7.1: The content of the ﬁle usps noisy.mat produced by the script
make noisy data.
Structure array trn containing the training data:
gnd X [256 7291] Training images of numerals represented as column
vectors. The input are 7291 grayscale images of size
16 16.
X [256 7291] Training images corrupted by the Gaussian noise with
signaltonoise ration SNR = 1 (it can be changed).
y [1 7291] Labels y ∈ ¸ = ¦1, . . . , 10¦ of training images.
Structure array tst containing the testing data:
gnd X [256 2007] Testing images of numerals represented as column vec
tors. The input are 2007 grayscale images of size
16 16.
X [256 2007] Training images corrupted by the Gaussian noise with
signaltonoise ration SNR = 1 (it can be changed).
y [1 2007] Labels y ∈ ¸ = ¦1, . . . , 10¦ of training images.
T
Φ
to minimize the mean square reconstruction error
ε
KMS
=
l
i=1
φ(x
i
) −
˜
φ(x
i
)
2
,
where
˜
φ(x) =
l
i=1
β
i
φ(x) , β = A(z −b) , z = A
T
k(x) +b . (7.1)
The vector k(x) = [k(x
1
, x), . . . , k(x
l
, x)]
T
contains kernel functions evaluated at the
training data T
X
. The matrix A [l m] and the vector b [m 1] are trained by the
kernel PCA algorithm. Given a projection
˜
φ(x) onto the kernel PCA subspace an
image x
∈ R
n
of the input space can be recovered by solving the preimage problem
x
= argmin
x
φ(x) −
˜
φ(x)
2
. (7.2)
The Radial Basis Function (RBF) kernel k(x
a
, x
b
) = exp(−0.5x
a
− x
b

2
/σ
2
) was
used here as the preimage problem (7.2) can be well solved for this case. The whole
procedure of reconstructing the corrupted image ˆ x ∈ R
n
involves (i) using (7.1) to
compute z which determines
˜
φ(x) and (ii) solving the preimage problem (7.2) which
yields the resulting x
. This procedure is implemented in function kpcarec.
In contrast to the linear PCA, the kernel PCA allows to model the nonlinearity
which is very likely in the case of images. However, the nonlinearity hidden in
88
the parameter σ of the used RBF kernel has to be tuned properly to prevent over
ﬁtting. Another parameter is the dimension m of the subspace kernel PCA subspace
to be trained. The best parameters (σ
∗
, m
∗
) was selected out of a prescribed set
Θ = ¦(σ
1
, m
1
), . . . , (σ
k
, m
k
)¦ such that the crossvalidation estimate of the mean square
error
ε
MS
(σ, m) =
_
X
P
X
(x)
_
x −x

2
_
dx ,
was minimal. The x stands for the ground truth image and x
denotes the image
restored from noisy one ˆ x using (7.1) and (7.2). The images are assumed to be gen
erated from distribution P
X
(x). Figure 7.4 shows the idea of tuning the parameters
of the kernel PCA. The linear PCA was used to solve the same task for comparison.
The same idea was used to tune the output dimension of the linear PCA. Having the
parameters tuned the resulting kernel PCA and linear PCA are trained on the whole
training set. The whole training procedure of the kernel PCA is implemented in the
script train kpca denois. The training for linear PCA is implemented in the script
train lin denois. The greedy kernel PCA Algorithm 3.4 is used for training the
model. It produces sparse model and can deal with large data sets. The results of
tuning the parameters are displayed in Figure 7.5. The ﬁgure was produced by a script
show denois tuning.
The function kpcarec is used to denoise images using the kernel PCA model. In
the case of the linear PCA, the function pcarec was used. An example of denoising of
the testing data of USPS is implemented in script show denoising. Figure 7.6 shows
examples of ground truth images, noisy images, images restored by the linear PCA and
kernel PCA.
89
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
Control: left_button − draw, middle_button − classify, right_button − erase.
(a)
(b)
Figure 7.1: Figure (a) shows handwritten training examples of numeral ’1’ and Figure
(b) shows corresponding normalized images of size 16 16 pixels.
90
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
Control: left_button − draw, middle_button − classify, right_button − erase.
(a)
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
1 2 3 4 5 6 7 8 9 0
4 0
5 3 1 9
6 2 8
(b)
Figure 7.2: A snapshot of running OCR system. Figure (a) shows handwritten nu
merals and Figure (b) shows recognized numerals.
91
−4 −3 −2 −1 0 1 2 3 4 5 6 7
−4
−3
−2
−1
0
1
2
3
4
5
6
7
Ground truth
Noisy examples
Corrupted
Reconstructed
Figure 7.3: Demonstration of idea of image denoising by kernel PCA.
Data
generator
model
model
Distortion
KPCA Preimage
problem
θ ∈ Θ
Error
e
r
r
o
r
θ
assessment
Figure 7.4: Schema of tuning parameters of kernel PCA model used for image restora
tion.
92
0 10 20 30 40 50 60 70 80 90 100
10
11
12
13
14
15
16
17
18
dim
ε
M
S
3.5 4 4.5 5 5.5 6 6.5 7 7.5 8
6
8
10
12
14
16
18
20
σ
ε
M
S
dim = 50
dim = 100
dim = 200
dim = 300
(a) (b)
Figure 7.5: Figure (a) shows plot of ε
MS
with respect to output dimension of lin
ear PCA. Figure(b) shows plot of ε
MS
with respect to output dimension and kernel
argument of the kernel PCA.
Ground truth
Numerals with added Gaussian noise
Linear PCA reconstruction
Kernel PCA reconstruction
Figure 7.6: The results of denosing the handwritten images of USPS database.
93
Chapter 8
Conclusions and future planes
The STPRtool as well as its documentation is being updating. Therefore we are grateful
for any comment, suggestion or other feedback which would help in the development
of the STPRtool.
There are many relevant methods and proposals which have not been implemented
yet. We welcome anyone who likes to contribute to the STPRtool in any way. Please
send us an email and your potential contribution.
The future planes involve the extensions of the toolbox by the following methods:
• Feature selection methods.
• AdaBoost and related learning methods.
• Eﬃcient optimizers to for large scale Support Vector Machines learning problems.
• Model selection methods for SVM.
• Probabilistic Principal Component Analysis (PPCA) and mixture of PPCA esti
mated by the ExpectationMaximization (EM) algorithm.
• Discrete probability models estimated by the EM algorithm.
94
Bibliography
[1] T.W. Anderson and R.R. Bahadur. Classiﬁcation into two multivariate normal
distributions with diﬀerrentia covariance matrices. Anals of Mathematical Statis
tics, 33:420–431, June 1962.
[2] G. Baudat and F. Anouar. Generalized discriminant analysis using a kernel ap
proach. Neural Computation, 12(10):2385–2404, 2000.
[3] C.M. Bishop. Neural Networks for Pattern Recognition. Clarendon Press, Oxford,
Great Britain, 3th edition, 1997.
[4] N. Cristianini and J. ShaweTaylor. Support Vector Machines. Cambridge Uni
versity Press, 2000.
[5] A.P. Dempster, N.M. Laird, and D.B. Rubin. Maximum likelihood from incom
plete data via the EM Algorithm. Journal of the Royal Statistical Society, 39:185–
197, 1977.
[6] R.O. Duda, P.E. Hart, and D.G. Stork. Pattern Classiﬁcation. John Wiley &
Sons, 2nd. edition, 2001.
[7] R.P.W. Duin. Prtools: A matlab toolbox for pattern recognition, 2000.
[8] V. Franc. Programov´e n´astroje pro rozpozn´ av´ an´ı (Pattern Recognition Program
ming Tools, In Czech). Master’s thesis,
ˇ
Cesk´e vysok´e uˇcen´ı technick´e, Fakulta
elektrotechnick´a, Katedra kybernetiky, February 2000.
[9] V. Franc and V. Hlav´aˇc. Multiclass support vector machine. In R. Kasturi,
D. Laurendeau, and Suen C., editors, 16th International Conference on Pattern
Recognition, volume 2, pages 236–239. IEEE Computer Society, 2002.
[10] V. Franc and V. Hlav´ aˇc. An iterative algorithm learning the maximal margin
classiﬁer. Pattern Recognition, 36(9):1985–1996, 2003.
95
[11] V. Franc and V. Hlav´ aˇc. Greedy algorithm for a training set reduction in the
kernel methods. In N. Petkov and M.A. Westenberg, editors, Computer Analysis
of Images and Patterns, pages 426–433, Berlin, Germany, 2003. Springer.
[12] C.W. Hsu and C.J. Lin. A comparison of methods for multiclass support vector
machins. IEEE Transactions on Neural Networks, 13(2), March 2002.
[13] I.T. Jollife. Principal Component Analysis. SpringerVerlag, New York, 1986.
[14] S.S. Keerthi, S.K. Shevade, C. Bhattacharya, and K.R.K. Murthy. A fast itera
tive nearest point algorithm for support vector machine classiﬁer design. IEEE
Transactions on Neural Networks, 11(1):124–136, January 2000.
[15] Kim Kwang, In, Franz Matthias, O., and Sch¨olkopf Bernhard. Kernel hebbian
algorithm for singlefram superresolution. In Leonardis Aleˇs and Bischof Horst,
editors, Statisical Learning in Computer Vision, ECCV Workshop. Springer, May
2004.
[16] J.T. Kwok and I.W. Tsang. The preimage problem in kernel methods. In Pro
ceedings of the Twentieth International Conference on Machine Learning (ICML
2003), pages 408–415, Washington, D.C., USA, August 2003.
[17] Y. LeCun, B. Boser, J.S. Denker, D. Henderson, R.E. Howard, W. Hubbard, and
L.J Jackel. Backpropagation applied to handwritten zip code recognition. Neural
Computation, 1:541–551, 1989.
[18] G. McLachlan and T. Krishnan. The EM Algorithm and Extensions. John Wiley
& Sons, New York, 1997.
[19] S. Mika, G. R¨ atsch, J. Weston, B. Sch¨olkpf, and K. M¨ uller. Fisher discriminant
analysis with kernel. In Y.H. Hu, J. Larsen, and S. Wilson, E. Douglas, editors,
Neural Networks for Signal Processing, pages 41–48. IEEE, 1999.
[20] S. Mika, B. Sch¨ olkopf, A. Smola, K.R. M¨ uller, M. Scholz, and G. R¨ atsch. Kernel
pca and denoising in feature spaces. In M.S. Kearns, S.A. Solla, and D.A. Cohn,
editors, Advances in Neural Information Processing Systems 11, pages 536 – 542,
Cambridge, MA, 1999. MIT Press.
[21] I.T. Nabney. NETLAB : algorithms for pattern recognition. Advances in pattern
recognition. Springer, London, 2002.
[22] J. Platt. Probabilities for sv machines. In A.J. Smola, P.J. Bartlett, B. Scholkopf,
and D. Schuurmans, editors, Advances in Large Margin Classiﬁers (Neural Infor
mation Processing Series). MIT Press, 2000.
96
[23] J.C. Platt. Sequential minimal optimizer: A fast algorithm for training support
vector machines. Technical Report MSRTR9814, Microsoft Research, Redmond,
1998.
[24] B.D. Riply. Neural networks and related methods for classiﬁcation (with discu
sion). J. Royal Statistical Soc. Series B, 56:409–456, 1994.
[25] M.I. Schlesinger. A connection between learning and selflearning in the pattern
recognition (in Russian). Kibernetika, 2:81–88, 1968.
[26] M.I. Schlesinger and V. Hlav´ aˇc. Ten lectures on statistical and structural pattern
recognition. Kluwer Academic Publishers, 2002.
[27] B. Sch¨olkopf, C. Burges, and A.J. Smola. Advances in Kernel Methods – Support
Vector Learning. MIT Press, Cambridge, 1999.
[28] B. Sch¨olkopf, P. Knirsch, and C. Smola, A. Burges. Fast approximation of support
vector kernel expansions, and an interpretation of clustering as approximation in
feature spaces. In P. Levi, M. Schanz, R.J. Ahler, and F. May, editors, Mus
tererkennung 199820. DAGM., pages 124–132, Berlin, Germany, 1998. Springer
Verlag.
[29] B. Scholkopf, A. Smola, and K.R. Muller. Nonlinear component analysis as a
kernel eigenvalue problem. Neural Computation, 10:1299–1319, 1998.
[30] B. Sch¨olkopf and A.J. Smola. Learning with Kernels. The MIT Press, MA, 2002.
[31] V. Vapnik. The nature of statistical learning theory. Springer Verlag, 1995.
97
Statistical Pattern Recognition Toolbox for Matlab
Vojtˇch Franc and V´clav Hlav´ˇ e a ac June 24, 2004
Contents
1 Introduction 1.1 What is the Statistical Pattern Recognition Toolbox and how it has been developed? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 The STPRtool purpose and its potential users . . . . . . . . . . . . . . 1.3 A digest of the implemented methods . . . . . . . . . . . . . . . . . . . 1.4 Relevant Matlab toolboxes of others . . . . . . . . . . . . . . . . . . . . 1.5 STPRtool document organization . . . . . . . . . . . . . . . . . . . . . 1.6 How to contribute to STPRtool? . . . . . . . . . . . . . . . . . . . . . 1.7 Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Linear Discriminant Function 2.1 Linear classiﬁer . . . . . . . . . . . . . . . . . . 2.2 Linear separation of ﬁnite sets of vectors . . . . 2.2.1 Perceptron algorithm . . . . . . . . . . . 2.2.2 Kozinec’s algorithm . . . . . . . . . . . . 2.2.3 Multiclass linear classiﬁer . . . . . . . . 2.3 Fisher Linear Discriminant . . . . . . . . . . . . 2.4 Generalized Anderson’s task . . . . . . . . . . . 2.4.1 Original Anderson’s algorithm . . . . . . 2.4.2 General algorithm framework . . . . . . 2.4.3 εsolution using the Kozinec’s algorithm 2.4.4 Generalized gradient optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 3 3 4 5 5 5 6 8 9 10 11 12 14 15 17 19 21 21 22 25 26 29 31 32 33 34
3 Feature extraction 3.1 Principal Component Analysis . . . . . . . . . . . 3.2 Linear Discriminant Analysis . . . . . . . . . . . 3.3 Kernel Principal Component Analysis . . . . . . . 3.4 Greedy kernel PCA algorithm . . . . . . . . . . . 3.5 Generalized Discriminant Analysis . . . . . . . . . 3.6 Combining feature extraction and linear classiﬁer
1
. 5 Support vector and other kernel machines 5. . . . . . . . . . . . . . . . . . . . . . . 4. . . . .3 Bayesian classiﬁer . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5. . . . . . . . . 6. . . . . . . . . . . . .8 Reduced set method . . . . . . . . .1 Data sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5 Kmeans clustering . . . . . . . . . . . . . . . . . .4 OneAgainstOne decomposition for SVM . . . . . . .2 Image denoising . . . . . . . . . . . . 5. . . . . .2. . .6 Kernel Fisher Discriminant . . . . 5. . .1 Support vector classiﬁers . . . . . . . . . . . . . 7. . . . . . 6 Miscellaneous 6. . . . . . . . . . . . . . . . 4. . . . . . 5. . . . . . . . . . . . . . . 41 41 43 44 45 47 48 50 57 57 58 63 64 66 68 70 73 76 76 77 78 80 82 85 85 87 94 7 Examples of applications 7. 8 Conclusions and future planes 2 . . . . . 6. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4. . . . . . . . . . . . . . . . . . . . . . . . . . 4. . . . . . . . . . . . . .2 Binary Support Vector Machines . . . . .7 Preimage problem . . . . . . . . . . .2 Incomplete data . . . . . . . . . .4 Probabilistic output of classiﬁer . . . . . . . . 6. . .4 Quadratic classiﬁer . . . . . . . . . .4 Density estimation and clustering 4. .1 Complete data . . .3 OneAgainstAll decomposition for SVM . . . . .5 Multiclass BSVM formulation . . . . . . . . . . . . . . . . . 5. . . . . . . . . . . . .3 Minimax estimation . . . . . . . . . . . . . . . . . . 6. . . . . . . . . . . .2 Visualization . . . . . . . . . . . . . . . . . . . . . . . . . . 5. . . . . . . . . . . . . . . . . . . . . . . . . . . .1 Gaussian distribution and Gaussian mixture model 4. . . . . . . . . . . . . . . . . . . 4. . . . . . . . . . . . . .5 Knearest neighbors classiﬁer . . .2. . . . . . . . . .1 Optical Character Recognition system . 5. . . . .2 MaximumLikelihood estimation of GMM . .
Chapter 1 Introduction
1.1 What is the Statistical Pattern Recognition Toolbox and how it has been developed?
The Statistical Pattern Recognition Toolbox (abbreviated STPRtool) is a collection of pattern recognition (PR) methods implemented in Matlab. The core of the STPRtool is comprised of statistical PR algorithms described in the monograph Schlesinger, M.I., Hlav´ˇ, V: Ten Lectures on Statistical and Structural ac Pattern Recognition, Kluwer Academic Publishers, 2002 [26]. The STPRtool has been e developed since 1999. The ﬁrst version was a result of a diploma project of Vojtˇch Franc [8]. Its author became a PhD student in CMP and has been extending the STPRtool since. Currently, the STPRtool contains much wider collection of algorithms. The STPRtool is available at http://cmp.felk.cvut.cz/cmp/cmp software.html. We tried to create concise and complete documentation of the STPRtool. This document and associated help ﬁles in .html should give all information and parameters the user needs when she/he likes to call appropriate method (typically a Matlab mﬁle). The intention was not to write another textbook. If the user does not know the method then she/he will likely have to read the description of the method in a referenced original paper or in a textbook.
1.2
The STPRtool purpose and its potential users
• The principal purpose of STPRtool is to provide basic statistical pattern recognition methods to (a) researchers, (b) teachers, and (c) students. The STPRtool is likely to help both in research and teaching. It can be found useful by practitioners who design PR applications too.
3
• The STPRtool oﬀers a collection of the basic and the stateofthe art statistical PR methods. • The STPRtool contains demonstration programs, working examples and visualization tools which help to understand the PR methods. These teaching aids are intended for students of the PR courses and their teachers who can easily create examples and incorporate them into their lectures. • The STPRtool is also intended to serve a designer of a new PR method by providing tools for evaluation and comparison of PR methods. Many standard and stateofthe art methods have been implemented in STPRtool. • The STPRtool is open to contributions by others. If you develop a method which you consider useful for the community then contact us. We are willing to incorporate the method into the toolbox. We intend to moderate this process to keep the toolbox consistent.
1.3
A digest of the implemented methods
This section should give the reader a quick overview of the methods implemented in STPRtool. • Analysis of linear discriminant function: Perceptron algorithm and multiclass modiﬁcation. Kozinec’s algorithm. Fisher Linear Discriminant. A collection of known algorithms solving the Generalized Anderson’s Task. • Feature extraction: Linear Discriminant Analysis. Principal Component Analysis (PCA). Kernel PCA. Greedy Kernel PCA. Generalized Discriminant Analysis. • Probability distribution estimation and clustering: Gaussian Mixture Models. ExpectationMaximization algorithm. Minimax probability estimation. Kmeans clustering. • Support Vector and other Kernel Machines: Sequential Minimal Optimizer (SMO). Matlab Optimization toolbox based algorithms. Interface to the SV M light software. Decomposition approaches to train the Multiclass SVM classiﬁers. Multiclass BSVM formulation trained by Kozinec’s algorithm, MitchellDemyanovMolozenov algorithm and Nearest Point Algorithm. Kernel Fisher Discriminant.
4
1.4
Relevant Matlab toolboxes of others
Some researchers in statistical PR provide code of their methods. The Internet oﬀers many software resources for Matlab. However, there are not many compact and well documented packages comprised of PR methods. We found two useful packages which are brieﬂy introduced below. NETLAB is focused to the Neural Networks applied to PR problems. The classical PR methods are included as well. This Matlab toolbox is closely related to the popular PR textbook [3]. The documentation published in book [21] contains detailed description of the implemented methods and many working examples. The toolbox includes: Probabilistic Principal Component Analysis, Generative Topographic Mapping, methods based on the Bayesian inference, Gaussian processes, Radial Basis Functions (RBF) networks and many others. The toolbox can be downloaded from http://www.ncrg.aston.ac.uk/netlab/. PRTools is a Matlab Toolbox for Pattern Recognition [7]. Its implementation is based on the object oriented programming principles supported by the Matlab language. The toolbox contains a wide range of PR methods including: analysis of linear and nonlinear classiﬁers, feature extraction, feature selection, methods for combining classiﬁers and methods for clustering. The toolbox can be downloaded from http://www.ph.tn.tudelft.nl/prtools/.
1.5
STPRtool document organization
The document is divided into chapters describing the implemented methods. The last chapter is devoted to more complex examples. Most chapters begin with deﬁnition of the models to be analyzed (e.g., linear classiﬁer, probability densities, etc.) and the list of the implemented methods. Individual sections have the following ﬁxed structure: (i) deﬁnition of the problem to be solved, (ii) brief description and list of the implemented methods, (iii) reference to the literature where the problem is treaded in details and (iv) an example demonstrating how to solve the problem using the STPRtool.
1.6
How to contribute to STPRtool?
If you have an implementation which is consistent with the STPRtool spirit and you want to put it into public domain through the STPRtool channel then send an email to xfrancv@cmp.felk.cvut.cz. 5
I. EU and industrial grants which indirectly contributed to STPRtool development. thanks go to several Czech. M. the STPRtool was rewritten and its documentation radically improved in May and June 2004 with the ﬁnancial help of ECVision European Research Network on Cognitive Computer Vision Systems (IST200135454) lead by D. we like to thank to Prof. our visitors. Third.Please prepare your code according to the pattern you ﬁnd in STPRtool implementations. Vernon. Provide piece of documentation too which will allow us to incorporate it to STPRtool easily. other STPRtool users and students who used the toolbox in their laboratory exercises. Schlesinger from the Ukrainian Academy of Sciences. 6 . Ukraine who showed us the beauty of pattern recognition and gave us many valuable advices. we like to thank for discussions to our colleagues in the Center for Machine Perception. Kiev. Forth.7 Acknowledgements First. Of course. 1. your method will be bound to your name in STPRtool. the Czech Technical University in Prague. Second.
Cardinality of a set. Matrix determinant. Kernel function (Mercer kernel). Mapping from input space X to the feature space F . Covariance matrix. c}. Mean vector (mathematical expectation). The concatenation of column vectors w ∈ Rn . . j) = 0 otherwise. Vectors are denoted by lowercase bold italic letters. for instance A. Set of c hidden states – class labels Y = {1. Vectors are implicitly column vectors. j) = 1 for i = j and δ(i. . z] where the column vector v is (n + m)dimensional. Euclidean norm. ndimensional input space (space of observations). . for instance x.Notation Uppercase bold letters denote matrices. 7 . . y1 ). . z ∈ Rm is denoted as v = [w. Kronecker delta δ(i. j) Dot product between vectors x and x . Scatter matrix. Unlabeled training set TX = {x1 . Labeled training set (more precisely multiset) TXY = {(x1 . . . Discriminant function of the binary classiﬁer. Logical or. (xl . Classiﬁcation rule. yl )}. . . . Feature space. Discriminant function associated with the class y ∈ Y. x·x Σ µ S X ⊆ Rn Y F φ: X → F k: X × X → R TXY TX q: X → Y fy : X → R f: X → R · · det(()·) δ(i. . . xl }.
the linear classiﬁer is represented by a single discriminant function f (x) = w · x + b . The problem is described by a ﬁnite set T = {(x1 . (xl . The input vector x ∈ Rn is assigned to the class y ∈ Y its corresponding discriminant function fy attains maximal value y = argmax fy (x) = argmax ( wy · x + by ) . Following sections describe supervised learning methods which determine the parameters of the linear classiﬁer based on available knowledge. The methods using this type of training data is described in Section 2. . y1 ). . . 2} as follows q(x) = 1 if f (x) = w · x + b ≥ 0 .3. The input vector x ∈ Rn is assigned to class y ∈ {1. The scalars by . given by the parameter vector w ∈ Rn and bias b ∈ R. ∀y ∈ Y . The implemented learning methods can be distinguished according to the type of the training data: I. 2 if f (x) = w · x + b < 0 .Chapter 2 Linear Discriminant Function The linear classiﬁcation rule q: X ⊆ Rn → Y = {1. c} is composed of a set of discriminant functions fy (x) = w y · x + by . 2. which are linear with respect to both the input vector x ∈ Rn and their parameter vectors w ∈ Rn . yl )} containing pairs of observations xi ∈ Rn and corresponding class labels yi ∈ Y.1) In the particular binary case Y = {1.2 and Section 2. . . (2. 8 . . ∀y ∈ Y introduce bias to the discriminant functions.2) The datatype used to describe the linear classiﬁer and the implementation of the classiﬁer is described in Section 2. . 2}. y ∈Y y ∈Y (2. .1.
demo anderson Demo on Generalized Anderson’s task. mperceptr Perceptron algorithm to train multiclass linear machine.4).1 Linear classiﬁer The STPRtool uses a speciﬁc structure array to describe the binary (see Table 2. ganders Solves the Generalized Anderson’s task. Example: Linear classiﬁer The example shows training of the Fisher Linear Discriminant which is the classical example of the linear classiﬁer. fldqp Fisher Linear Discriminant using Quadratic Programming.II. demo linclass Demo on the algorithms learning linear classiﬁers.3) linear classiﬁer. The parameters of class conditional distributions are (completely or partially) known. 2. References: The linear discriminant function is treated throughout for instance in books [26. 9 . The implementation of the linear classiﬁer itself provides function linclass. The Anderson’s task and its generalization are representatives of such methods (see Section 2. fld Fisher Linear Discriminant.1. andrerr Classiﬁcation error of the Generalized Anderson’s task.mat. The Riply’s data set riply trn. eanders Epsilonsolution of the Generalized Anderson’s task. ggradander Gradient method to solve the Generalized Anderson’s task.mat is used for training. androrig Original method to solve the AndersonBahadur’s task. Table 2. 6].2) and the multiclass (see Table 2. The summary of implemented methods is given in Table 2. The resulting linear classiﬁer is then evaluated on the testing data riply tst. ekozinec Kozinec’s algorithm for epsoptimal separating hyperplane. perceptron Perceptron algorithm to train binary linear classiﬁer.1: Implemented methods: Linear discriminant function linclass Linear classiﬁer (linear machine).
3) with respect to the vector w ∈ Rn and the scalar b ∈ R. .2 Linear separation of ﬁnite sets of vectors The input training data T = {(x1 . The implemented methods solve the task of (i) training separating hyperplane for binary case c = 2. tst.2: Datatype used to describe binary linear classiﬁer. y = 1. .1080 % % % % % load training data load testing data train FLD classifier classify testing data evaluate testing error 2.Table 2.3: Datatype used to describe multiclass linear classiﬁer. w · xi + b < 0 . y = 1. Multiclass linear classiﬁer (structure array): . . c. Table 2. . yi = 2 . y1 ). (ii) training εoptimal hyperplane and (iii) training the multiclass linear classiﬁer. The problem of training the binary (c = 2) linear classiﬁer (2.3) have a solution then the training data T is linearly separable.W [n × 1] The normal vector w ∈ Rn of the separating hyperplane f (x) = w · x + b. trn = load(’riply_trn’). . . . . (xl . cerror( ypred. . . yl )} consists of pairs of observations xi ∈ Rn and corresponding class labels yi ∈ Y = {1.b [1 × 1] Bias b ∈ R of the hyperplane.X. . The Perceptron (Section 2. tst = load(’riply_tst’).fun = ’linclass’ Identiﬁes function associated with this data type. model = fld( trn ).1) 10 . . c}. yi = 1 .y ) ans = 0. The deﬁnitions of these tasks are given below. (2.W [n × c] Matrix of parameter vectors w y ∈ Rn . .2. The problem is formally deﬁned as solving the set of linear inequalities w · xi + b ≥ 0 . c of the linear discriminant functions fy (x) = w y · x + b. .b [c × 1] Parameters by ∈ R. model ). ypred = linclass( tst. . . . Binary linear classiﬁer (structure array): . If the inequalities (2. . .fun = ’linclass’ Identiﬁes function associated with this data type.2) with zero training error is equivalent to ﬁnding the hyperplane H = {x: w · x + b = 0} which separates the training vectors of the ﬁrst y = 1 and the second y = 2 class. .
3) can be formally rewritten to a simpler form v · zi > 0 . yi = y .5) holds. The εoptimal solution can be found by Kozinec’s algorithm (Section 2.2.3) or Kozinec’s algorithm.6) for linearly separable training data T can be solved by the modiﬁed Perceptron (Section 2. This task is equivalent to training the linear Support Vector Machines (see Section 5).2). (2. .6) with respect to the vectors wy ∈ Rn . . yl )} of binary labeled yi ∈ {1. b] . The εoptimal solution is deﬁned as the vector w and scalar b such that the inequality m(w ∗ .b w. b) ≤ ε .4) where I1 = {i: yi = 1} and I2 = {i: yi = 2} are sets of indices. . Therefore the numerical algorithms seeking the approximate solution are applied instead.2. . The optimal separating hyperplane H∗ = {x ∈ Rn : w∗ · x + b∗ = 0} separates training data T with maximal margin m(w ∗ .2. . The optimal separating hyperplane cannot be found exactly except special cases. where the vector v ∈ Rn+1 is constructed as v = [w. An interactive demo on the algorithms separating the ﬁnite sets of vectors by a hyperplane is implemented in demo linclass. The task (2. .8) i = 1. min − w w i∈I2 . . (2. .2.and Kozinec’s (Section 2. . 11 (2. i = 1. The problem of training the separating hyperplane (2. . y ∈ Y. . (xl .2) algorithm are useful to train the binary linear classiﬁers from the linearly separable data. . l . y1). b∗ ). 2.b = argmax min min i∈I1 w · xi + b w · xi + b . b∗ ) = argmax m(w. The parameter ε ≥ 0 deﬁnes closeness to the optimal solution in terms of the margin. b) w. The problem of training the multiclass linear classiﬁer c > 2 with zero training error is formally stated as the problem of solving the set of linear inequalities w yi · xi + byi > wy · xi + by .1 Perceptron algorithm The input is a set T = {(x1 . l . (2. b∗ ) − m(w.7) . y ∈ Y and scalars by ∈ R. If the training data is linearly separable then the optimal separating hyperplane can be deﬁned the solution of the following task (w∗ . (2. 2} training vectors xi ∈ Rn .
w∗ = 1 2 argmin w1 ∈X1 .2. . % generate training data % call perceptron % plot training data % plot separating hyperplane 2. Example: Training binary linear classiﬁer with Perceptron The example shows the application of the Perceptron algorithm to ﬁnd the binary linear classiﬁer for synthetically generated 2dimensional training data. figure.7) is satisﬁed.3). The generated training data contain 50 labeled vectors which are linearly separable with margin 1. . v (1) . . −[xi .2 Kozinec’s algorithm The input is a set T = {(x1 . . w 2 . 2} training (0) (1) (t) vectors xi ∈ Rn . where X1 stands for the convex hull of the training vectors of the ﬁrst class X1 = {xi : yi = 1} and X2 for the convex hull of the second class likewise. w2 which converge to the vector w ∗ and w ∗ respectively. The parameters (w. 50.9) The problem of solving (2. b) of the linear classiﬁer are obtained from the found vector v by inverting the transformation (2. .1. (2. yl )} of binary labeled yi ∈ {1. w1 . The vector w∗ = 12 . (xl . The found classiﬁer is visualized as a separating hyperplane (line in this case) H = {x ∈ R2 : w · x + b = 0}. v (t) until the set of inequalities (2. y1).and transformed training data Z = {z 1 . . . The 1 2 vectors w∗ and w ∗ are the solution of the following task 1 2 w∗. ppatterns( data ). The Perceptron algorithm is an iterative procedure which builds a series of vectors (0) v . The Perceptron from the neural network perspective is described in the book [3]. . data = genlsdata( 2. . model = perceptron( data ).8). The Novikoﬀ theorem ensures that the Perceptron algorithm stops after ﬁnite number of iterations t if the training data are linearly separable. . . . 1] if yi = 2 . pline( model ). References: Analysis of the Perceptron algorithm including the Novikoﬀ’s proof of convergence can be found in Chapter 5 of the book [26]. . The initial vector v (0) can be set arbitrarily (usually v = 0).7) with respect to the unknown vector v ∈ Rn+1 is equivalent to the original task (2. . . z l } are deﬁned as zi = [xi . 1). The Kozinec’s algorithm builds a series of vectors w 1 . w 1 (0) (1) (t) and w 2 . . 1] if yi = 1 .w2 ∈X2 w1 − w2 . . See Figure 2. Perceptron algorithm is implemented in function perceptron. . . .
The Kozinec’s algorithm is proven to converge in a ﬁnite number of iterations if the separating hyperplane exists. References: Analysis of the Kozinec’s algorithm including the proof of convergence can be found in Chapter 5 of the book [26]. The Kozinec’s algorithm can be also used to solve a simpler problem of ﬁnding the separating hyperplane (2. The εoptimal hyperplane (2. The generated training data contains 50 labeled vectors which are linearly separable with 13 . II. Notice that setting ε = 0 force the algorithm to seek the optimal hyperplane which is generally ensured to be found in an inﬁnite number of iterations t = ∞. Example: Training εoptimal hyperplane by the Kozinec’s algorithm The example shows application of the Kozinec’s algorithm to ﬁnd the (ε = 0.4) 1 2 2 1 2 separating the training data with the maximal margin. The separating hyperplane (2. If the εoptimal optimality stopping condition is used then the Kozinec’s algorithm converges in ﬁnite number of iterations.3) is sought for (ε < 0).5) is sought for (ε ≥ 0).01)optimal hyperplane for synthetically generated 2dimensional training data. The Kozinec’s algorithm is proven to converge to the vectors w∗ and w ∗ in inﬁnite 1 2 number of iterations t = ∞. Therefore the following two stopping conditions are implemented: I.−30 −35 −40 −45 −50 −55 −55 −50 −45 −40 −35 −30 Figure 2. The Kozinec’s algorithm which trains the binary linear classiﬁer is implemented in the function ekozinec.3).1: Linear classiﬁer trained by the Perceptron algorithm w∗ − w ∗ and the bias b∗ = 1 ( w ∗ 2 − w∗ 2 ) determine the optimal hyperplane (2.
y1 ).1) which classiﬁes the training data T without error.3 Multiclass linear classiﬁer The input training data T = {(x1 . . % plot data pline(model). .margin 1. The problem of training (2. . % generate data % run ekozinec model = ekozinec(data. b1 . 14 .2: εoptimal hyperplane trained with the Kozinec’s algorithm.10) where the vector v ∈ R(n+1)c contains the originally optimized parameters v = [w1 . ppatterns(data). struct(’eps’. The found classiﬁer is visualized (Figure 2. w 2 . w c . 2. . . y = Y \ {yi } . It could be veriﬁed that the found hyperplane has margin model.6) can be formally transformed to a simpler set v · zy > 0 .margin which satisﬁes the εoptimality condition. .1).01)). . bc ] . . The task is to design the linear classiﬁer (2. figure. data = genlsdata(2. . c}.0. (2. (xl .1) is equivalent to solving the set of linear inequalities (2. . % plot found hyperplane 8 6 4 2 0 −2 −4 −6 −8 −10 −10 −8 −6 −4 −2 0 2 4 6 8 10 Figure 2. l .2) as a separating hyperplane H = {x ∈ R2 : w · x + b = 0} (straight line in this case). b2 . The inequalities (2.50.6). . . yl )} consists of pairs of observations xi ∈ Rn and corresponding class labels yi ∈ Y = {1. i i = 1. .2. . . .
mat.3). µy = 15 . for j = y . (2. 1] . The class separability in a direction w ∈ Rn is deﬁned as F (w) = w · SB w . Iy  i∈I y (2. The found classiﬁer is visualized as its separating surface which is known to be piecewise linear and the class regions should be convex (see Figure 2. The simple form of the Perceptron updating rule allows to implement the transformation implicitly without mapping the training data T into (n + 1)cdimensional space.10) (number 10 stands for number of classes). l. Example: Training multiclass linear classiﬁer by the Perceptron The example shows application of the Perceptron rule to train the multiclass linear classiﬁer for the synthetically generated 2dimensional training data pentagon. 2} be sets of indices of training vectors belonging to the ﬁrst y = 1 and the second y = 2 class. 6]. y i set T such that ⎧ ⎨ [xi . The transformed task (2. y1). y ∈ {1. y ∈ {1. Let Iy = {i: yi = y}. ppatterns(data).The set of vectors z y ∈ R(n+1)c . . . 1] . The described method is implemented in function mperceptron. model = mperceptron(data). w · SW w 1 xi . otherwise. . (xl . data = load(’pentagon’). % load training data % run training algorithm % plot training data % plot decision boundary 2. References: The Kesler’s construction is described in books [26. y −[xi . figure. . . 2} . . . i The described transformation is known as the Kesler’s construction. yl )} of binary labeled yi ∈ {1. ∈ Y \ {yi } is created from the training for j = yi . .10) can be solved by the Perceptron algorithm.12) where SB is the betweenclass scatter matrix SB = (µ1 − µ2 )(µ1 − µ2 )T . z i (j) = ⎩ 0. A user’s own data can be generated interactively using function createdata(’finite’.3 Fisher Linear Discriminant The input is a set T = {(x1 . pboundary(model). i = 1. respectively.11) where z y (j) stands for the jth slot between coordinates (n + 1)(j − 1) + 1 and (n + 1)j. 2} training vectors xi ∈ Rn .
the parameter vector w of the linear discriminant function f (x) = w · x + b is determined to maximize the class separability criterion (2. 16 .13) using the matrix inversion w = SW −1 (µ1 − µ2 ) . for instance. In the case of the Fisher Linear Discriminant (FLD). y ∈ {1.4 −0. holds.6 −0.8 1 Figure 2.4 0.12) w = argmax F (w ) = argmax w w w · SB w . computes such b that the equality w · µ1 + b = −( w · µ2 + b) .2 −0.8 −1 −1 −0. 2} .2 0 −0.2 0. The STPRtool contains two implementations of FLD: I. Sy = i∈Yy (xi − µy )(xi − µy )T . w · SW w (2. and SW is the within class scatter matrix deﬁned as SW = S1 + S2 .8 −0.8 0. The STPRtool implementation.3: Multiclass linear classiﬁer trained by the Perceptron algorithm.6 0.1 0.13) The bias b of the linear rule must be determined based on another principle.4 −0.6 0. The classical solution of the problem (2.2 0 0.6 −0.4 0. This case is implemented in the function fld.
The parameters (µ1 .4 Generalized Anderson’s task The classiﬁed object is described by the vector of observations x ∈ X ⊆ Rn and a binary hidden state y ∈ {1. ppatterns(trn). 2} are known to be multivariate Gaussian distributions. However.mat. tst = load(’riply_tst’). Σ1 ) belong to a certain ﬁnite set of parameters {(µi . 2}. Σ2 ) belong to a ﬁnite set {(µi . ypred = linclass(tst. 2} 17 . This method can be useful when the matrix inversion is hard to compute. Similarly (µ2 .II. model = fldqp(trn).13) can be reformulated as the quadratic programming (QP) task w = argmin w · SW w . trn = load(’riply_trn’).tst. figure. The method is implemented in the function fldqp. The QP solver quadprog of the Matlab Optimization Toolbox is used in the toolbox implementation.4) and evaluated on the testing data riply tst. The problem (2. Σ2 ) of these class distributions are unknown.mat.model). Example: Fisher Linear Discriminant The binary linear classiﬁer based on the Fisher Linear Discriminant is trained on the Riply’s data set riply trn. The class conditional distributions pXY (xy). References: The Fisher Linear Discriminant is described in Chapter 3 of the book [6] or in the book [3]. w subject to w · (µ1 − µ2 ) = 2 . Σ1 ) and (µ2 .X. y ∈ {1. Σi ): i ∈ I2 }.y) ans = 0. pline(model). The found linear classiﬁer is visualized (see Figure 2. it is known that the parameters (µ1 .1080 % load training data % compute FLD % % % % plot data and solution load testing data classify testing data compute testing error 2. However. Let q: X ⊆ Rn → {1. Σi ): i ∈ I1 }. the function fldqp can be replaced by the standard approach implemented in the function fld which would yield the same solution. cerror(ypred. The QP formulation fldqp of FLD is used.
e. i∈I1 ∪I2 where ε(w. b∗ ) is minimal (w ∗ . b. In other words. for f (x) = w∗ · x + b∗ ≥ 0 .14) The original Anderson’s task is a special case of (2. b) = argmin max ε(w.5 0 0. for f (x) = w∗ · x + b∗ < 0 . i. b.8 0.b w. b. w. b.2 −1.b i∈I1 ∪I2 (2. Σi ) . r i = min (µi − x) · (Σi )−1 (µi − x) = x∈H w · µi + b w · Σi w .14) when I1  = 1 and I2  = 1.4: Linear classiﬁer based on the Fisher Linear Discriminant. µi . µi .6 0. be a binary linear classiﬁer (2. Σi ) and the nearest vector of the separating hyperplane H = {x ∈ Rn : w · x + b = 0}. The probability of misclassiﬁcation is deﬁned as Err(w. q(x) = 2 . The probability ε(w..5 1 Figure 2. µi . it is the probability that the vector x will be misclassiﬁed by the linear rule q.2 1 0.2) with discriminant function f (x) = w · x + b. such that the error Err(w∗ . Σi ) is probability that the Gaussian random vector x with mean vector µi and the covariance matrix Σi satisﬁes q(x) = 1 for i ∈ I2 or q(x) = 2 for i ∈ I1 . b∗ ) = argmin Err(w.2 0 −0. b∗ ) of the linear classiﬁer 1 . Σi ) . b) = max ε(w. The Generalized Anderson’s task (GAT) is to ﬁnd the parameters (w ∗ . Σi ) is proportional to the reciprocal of the Mahalanobis distance r i between the (µi .4 0.5 −1 −0.1. µi . 18 .
1 Original Anderson’s algorithm The algorithm is implemented in the function androrig. Example: Solving the original Anderson’s task 19 .14) can be equivalently rewritten as (w ∗ . This algorithm solves the original Anderson’s task deﬁned for two Gaussians only I1  = I2  = 1. In this case. λ γ= w · Σ2 w . b) is not diﬀerentiable. w · Σ1 w is satisﬁed.b i∈I1 ∪I2 w · µi + b . the optimal solution vector w ∈ Rn is obtained by solving the following problem w = ((1 − λ)Σ1 + λΣ2 )−1 (µ1 − µ2 ) . The problem is solved by an iterative algorithm which stops while the condition γ− 1−λ ≤ε. References: The original Anderson’s task was published in [1]. b) is less than 0. The objective function F (w. w · Σ1 w where the scalar 0 < λ < 1 is unknown. The STPRtool contains implementations of the algorithm solving the original Anderson’s task as well as implementations of three diﬀerent approaches to solve the Generalized Anderson’s task which are described bellow.5.The exact relation between the probability ε(w. An interactive demo on the algorithms solving the Generalized Anderson’s task is implemented in demo anderson.4. However. b∗ ) = argmax F (w. The parameter ε deﬁnes the closeness to the optimal solution. b) is proven to be convex in the region where the probability of misclassiﬁcation Err(w. w · Σi w which is more suitable for optimization. b. 1−λ = λ w · Σ2 w .b w. A detailed description of the Generalized Anderson’s task and all the methods implemented in the STPRtool is given in book [26]. µi . b) = argmax min w. Σi ) = ∞ ri 1 2 1 √ e− 2 t dt .15) The optimization problem (2. the objective function F (w. b. 2. Σi ) and the corresponding Mahalanobis distance r i is given by the integral ε(w. 2π (2. µi .
8 0. % cerror( ypred.3 0.4 gauss 1 0.mat.X.2 0. The parameters (µ1 . tst.4 −0. % tst = load(’riply_tst’).6 0.1 −0.mat. The found linear classiﬁer is visualized and the Gaussian distributions (see Figure 2.4 0. 20 .929 0. model ).6 −0. Σ2 ) of the Gaussian distribution of the ﬁrst and the second class are obtained by the MaximumLikelihood estimation.7 gauss 2 0.5). Σ1 ) and (µ2 . pandr( model.The original Anderson’s task is solved for the Riply’s data set riply trn. % distrib = mlcgmm(data). distrib ).8 −0. % ypred = linclass( tst.9 1.1150 load training data estimate Gaussians solve Anderson’s task plot solution load testing data classify testing data evaluate error 0.37 0. trn = load(’riply_trn’).5: Solution of the original Anderson’s task.5 0. % figure.6 Figure 2. % model = androrig(distrib).y ) % ans = 0.2 0 0. The classiﬁer is validated on the testing set riply tst.2 0.
b) such that F (w. is violated. The optimal movement k along the direction (∆w. b) < ε 21 . ∆b) in the parameter space is computed such that moving along this direction increases the objective function F . b(t+1) ) − F (w (t) . III. distrib = load(’mars’). the condition F (w(t+1) .3 εsolution using the Kozinec’s algorithm This algorithm is implemented in the function eanders. The algorithm iterates until the minimal change in the objective function is less than the prescribed ε. Example: Solving the Generalized Anderson’s task The Generalized Anderson’s task is solved for the data mars. The number of interval cuttings of is an optional parameter of the algorithm. The εsolution is deﬁned as a linear classiﬁer with parameters (w. II. The new parameters are set w(t+1) : = w (t) + k∆w . This algorithm repeats the following steps: I. figure.mat. % load input Gaussians % solve the GAT % plot the solution 2. The found linear classiﬁer as well as the input Gaussian distributions are visualized (see Figure 2. b(t) ) > ε . The quadprog function of the Matlab optimization toolbox is used to solve this subtask.4.2. model = ganders(distrib). An improving direction (∆w.2 General algorithm framework The algorithm is implemented in the function ganders.e. i. ∆b). Both the ﬁrst and second class are described by three 2Ddimensional Gaussian distributions.distrib). pandr(model.6). b(t+1) : = b(t) + k∆b .. The optimal movement k is determined by 1Dsearch based on the cutting interval algorithm according to the Fibonacci series.4.
pandr(model. Example: εsolution of the Generalized Anderson’s task The εsolution of the generalized Anderson’s task is solved for the data mars.06’).7). The desired maximal possible probability of misclassiﬁcation is set to 0. The prescribed ε ∈ (0. = struct(’err’. The problem is transformed to the task of separation of two sets of ellipsoids which is solved the Kozinec’s algorithm. 0.5 0. The found linear classiﬁer and the input Gaussian distributions are visualized (see Figure 2. The objective function F (w.4 Generalized gradient optimization This algorithm is implemented in the function ggradandr.047 0. eanders(distrib.mat.0503 −1 −2 −3 −4 −5 −4 −3 −2 −1 0 1 2 3 4 5 Figure 2. distrib options model = figure.06 (it is 6%). Both the ﬁrst and second class are described by three 2Ddimensional Gaussian distributions. = load(’mars’). The algorithm converges in ﬁnite number of iterations if the given εsolution exists.options).0483 4 3 0.5) is the desired upper bound on the probability of misclassiﬁcation.0503 1 0 gauss 5 0.4.0421 gauss 6 gauss 4 0.0.0461 gauss 2 gauss 3 2 gauss 1 0.6: Solution of the Generalized Anderson’s task. is satisﬁed.distrib). % % % % load Gaussians maximal desired error training visualization 2. b) is convex but not diﬀerentiable thus the standard gradient optimization cannot 22 . Otherwise it can get stuck in an endless loop.
b(t) ) > ε . The optimized parameters are updated w(t+1) : = w (t) + ∆w .0473 0. The maximal number of iterations can be used as the stopping condition as well. be used. ∆b) is computed. i.7: εsolution of the Generalized Anderson’s task with maximal 0.0483 gauss 5 0 −1 gauss 6 −2 gauss 4 0.5 4 3 gauss 2 0. However. 23 .06 probability of misclassiﬁcation. Example: Solving the Generalized Anderson’s task using the generalized gradient optimization The generalized Anderson’s task is solved for the data mars. b(t+1) ) − F (w (t) . is violated. II. b(t+1) = b(t) + ∆b . Both the ﬁrst and second class are described by three 2Ddimensional Gaussian distributions. The algorithm repeats the following two steps: I.0516 gauss 3 2 1 0. the condition F (w(t+1) . The generalized gradient (∆w.mat.e.0496 gauss 1 0. The maximal number of iterations is limited to tmax=10000. The found linear classiﬁer as well as the input Gaussian distributions are visualized (see Figure 2. the generalized gradient optimization method can be applied.0516 −3 0.0432 −4 −5 −4 −3 −2 −1 0 1 2 3 4 5 Figure 2.8).. The algorithm iterates until the minimal change in the objective function is less than the prescribed ε.
distrib = load(’mars’). pandr( model. options = struct(’tmax’.0434 −4 −5 −4 −3 −2 −1 0 1 2 3 4 5 Figure 2. model = ggradandr(distrib. 24 . distrib ).8: Solution of the Generalized Anderson’s task after 10000 iterations of the generalized gradient optimization algorithm.0518 gauss 2 gauss 3 3 2 gauss 1 0.10000).options).0485 gauss 6 gauss 4 −2 −3 0.0475 1 0 gauss 5 0. figure. % load Gaussians % max # of iterations % training % visualization 5 4 0.0498 0.0518 −1 0.
x1 x1 .2). . . . The kernel functions allow for nonlinear extensions of the linear feature extraction methods. i = 1. . . (3. The projection is given by the matrix W [n × m] and bias vector b ∈ Rm . x2 . x2 x2 . The linear data projection is implemented in function linproj (See examples in Section 3. n are entries of the vector x ∈ Rn . x1 x2 . In this case. 25 (3.3) (3.1) is commonly used. . . where xi . The Linear Discriminant Analysis (LDA) (Section 3.Chapter 3 Feature extraction The feature extraction step consists of mapping the input vector of observations x ∈ Rn onto a new feature description z ∈ Rm which is more suitable for given task. . .1 and Section 3. The Principal Component Analysis (PCA) (Section 3. This yields a nonlinear (kernel) projection of data which has generally deﬁned as z = AT k(x) + b . . the input vectors x ∈ Rn are mapped into a new higher dimensional feature space F in which the linear methods are applied. . xn xn ]T . The quadratic data mapping is implemented in function qmap. .2) . The quadratic data mapping projects the input vector x ∈ Rn onto the feature space Rm where m = n(n + 3)/2. . x2 xn . . .1) is a representative of the unsupervised learning method which yields the linear projection (3. The quadratic mapping φ: Rn → Rm is deﬁned as φ(x) = [ x1 . The data type used to describe the linear data projection is deﬁned in Table 3.2) is a representative of the supervised learning method yielding the linear projection. . The linear projection z = WT x + b . . x1 xn . . .1). xn .
The kernel data projection is implemented in the function kernelproj (See examples in Section 3.1 Table 3. kpca Kernel Principal Component Analysis. The Kernel PCA (Section 3. . .where A [l × m] is a parameter matrix and b ∈ Rm a parameter vector.5) are representatives of the feature extraction methods yielding the kernel data projection (3. pcarec Computes reconstructed vector after PCA projection. .fun = ’linproj’ Identiﬁes function associated with this data type. . . .3) and the Generalized Discriminant Analysis (GDA) (Section 3. The set of vectors TZ = {z 1 . kpcarec Reconstructs image after kernel PCA. Linear data projection (structure array): . demo pcacomp Demo on image compression using PCA. The x ∈ Rn stands for the input vector and z ∈ Rm is the vector of extracted features. lin2quad Merges linear rule and quadratic mapping. The vector k(x) = [k(x.2: Datatype used to describe linear data projection.1 Principal Component Analysis Let TX = {x1 . . pca Principal Component Analysis.1: Implemented methods for feature extraction linproj Linear data projection. . lda Linear Discriminant Analysis.3 and Section 3. . qmap Quadratic data mapping. lin2svm Merges linear rule and kernel projection. The summary of implemented methods for feature extraction is given in Table 3. xl )]T is a vector of kernel functions centered in the training data or their subset. . xl } be a set of training vectors from the ndimensional input space Rn . .b [m × 1] Bias vector b. z l } is a lower dimensional representation of the 26 . . x1 ). The data type used to describe the kernel data projection is deﬁned in Table 3. gda Generalized Discriminant Analysis.5). Table 3.3). . .W [n × m] Projection matrix W = [w1 . . The extracted features are projections of φ(x) onto m vectors (or functions) from the feature space F . kerproj Kernel data projection. greedykpca Greedy Kernel Principal Component Analysis. . wm ]. . 3. . k(x.
4).options. W .arg [1 × p] Kernel argument(s). .b (3. . . The mean square reconstruction error 1 εM S (W. j) is the Kronecker delta function. w m ] containing the m eigenvectors of the sample covariance matrix which have the largest eigen values. (3.fun = ’kernelproj’ Identiﬁes function associated with this data type. . . j) . m are column vectors of the matrix W = [w1 . xl } are computed by the linear back ˜ projection ˜ x = W(z − b) . . . The parameters (W. The vectors TZ are obtained by the linear orthonormal projection z = WT x + b .sv. . . 27 .6) of the training data TX . The vector b equals to WT µ. xl }. . . .X [n × l] Training vectors {x1 . . b) of the linear projection are the solution of the optimization task (W. ∀i. i = 1. The Principal Component Analysis (PCA) is the linear orthonormal projection (3. The PCA is implemented in the function pca.5) obtained by inverting (3.4) where the matrix W [n × m] and the vector b [m × 1] are parameters of the projec˜ ˜ tion. .4) and (3. b) = argmin εM S (W . .ker [string] Kernel identiﬁer. b ) . (3. (3.7) subject to wi · wj = δ(i. The reconstructed vectors TX = {x1 . . . .b [m × 1] Bias vector b.6) is a function of the parameters of the linear projections (3. A demo showing the use of PCA for image compression is implemented in the script demo pcacomp. . input training vectors TX in the mdimensional space Rm . The solution of the task (3.Alpha [l × m] Parameter matrix A [l × m].Table 3. where µ is the sample mean of the training data. .7) is the matrix W = [w 1 .3: Datatype used to describe kernel data projection. .options.5).4) which allows for the minimal mean square reconstruction error (3. Kernel data projection (structure array): . . . wm ] and δ(i. j . . b) = l l ˜ xi − xi i=1 2 . where w i . .
model).’r’).mx)]. Z = linproj(X.References: The PCA is treated throughout for instance in books [13.. The PCA is applied to train the 1dimensional subspace to approximate the input data with the minimal reconstruction error. h2 = ppatterns(XR. [dummy.5 6 5.mn] = min(Z). . Example: Principal Component Analysis The input data are synthetically generated from the 2dimensional Gaussian distribution. h3 = plot([XR(1. % % % % generate data train PCA lower dim. 3.8 1].5 7 6. XR = pcarec(X. ’PCA subspace’).1: Example on the Principal Component Analysis.mn) XR(2. legend([h1 h2 h3].[1 0. proj.’bo’).5 5 4. 6]. model = pca(X. axis equal.mx)]. The training data.8..5 2 3 4 5 6 7 Input vectors Reconstructed PCA subspace 8 Figure 3.1).mn) XR(1. [dummy. 28 .0. reconstr.[XR(2. the reconstructed data and the 1dimensional subspace given by the ﬁrst principal component are visualized (see Figure 3.100). % visualization h1 = ppatterns(X.1). hold on. 7. X = gsamp([5.’Reconstructed’. data figure.5 3 2.’kx’). ’Input vectors’.mx] = max(Z).5].5 4 3.model).
. = ˜ det(SW ) det(SW ) ˜ ˜ is maximized. . model = lda(orig_data. y ∈ Y = {1. The LDA and PCA are trained on the generated data.8) SB = y∈Y Iy (µy − µ)(µy − µ)T . . . The withinclass SW and betweenclass SB scatter matrices are described as SW = y∈Y Sy . figure. (3. 6]. yl )}. The SB is the betweenclass and SW is the withinclass scatter matrix of projected data which is deﬁned similarly to (3. . . 2. c} be labeled set of training vectors. Sy = i∈Iy (xi − µi )(xi − µi )T . ppatterns(ext_data).model). i=1 µy = 1 xi . xi ∈ Rn . where the total mean vector µ and the class mean vectors µy .8). y ∈ Y are deﬁned as 1 µ= l l xi . The synthetical data are generated from the Gaussian mixture model.2 Linear Discriminant Analysis Let TXY = {(x1 .mat which consists of the labeled 4dimensional data. y ∈ Y .2). Example 1: Linear Discriminant Analysis The LDA is applied to extract 2 features from the Iris data set iris. such that the class separability criterion F (W) = ˜ det(SB ) det(SB ) . References: The LDA is treated for instance in books [3.2.3. The data after the feature extraction step are visualized in Figure 3. % % % % load input data train LDA feature extraction plot ext. . y1). . ext_data = linproj(orig_data. Iy  i∈I y The goal of the Linear Discriminant Analysis (LDA) is to train the linear data projection z = WT x . data Example 2: PCA versus LDA This example shows why the LDA is superior to the PCA when features good for classiﬁcation are to be extracted. The LDA and 29 . orig_data = load(’iris’). (xl .
1 −0.5 0.9 1].3b).5].4 −0.9 1]. distrib.:.5 1 1.5 0 0.5 0.Mean = [[5.1) = [1 0.Cov(:. lda_data = linproj(data.9.X.Prior = [0. data = gmmsamp(distrib.2 0.X. The extracted data using the LDA and PCA model are displayed with the Gaussians ﬁtted by the MaximumLikelihood (see Figure 3.250).5]].5 −1. ppatterns(data).lda_model).1 0 −0.:.5 −1 −0.X.3 −0.1).5 Figure 3.2 −0.pca_model). figure.1). 30 % % % % % mean vectors 1st covariance 2nd covariance Gaussian weights sample data % train LDA % train PCA % visualization . pca_model = pca(data. axis equal.9. pca_data = linproj( data. hold on.4 0.lda_model).2) = [1 0. lda_model = lda(data. pca_rec = pcarec(data.3 0. 0. PCA directions are displayed as connection of the vectors projected to the trained LDA and PCA subspaces and reprojected back to the input space (see Figure 3.Cov(:. 0.pca_model). distrib.0. % Generate data distrib.3a). distrib. lda_rec = pcarec(data.4] [4.2: Example shows 2dimensional data extracted from the originally 4dimensional Iris data set using Linear Discriminant Analysis.
xl }. . The input training vectors TX = {x1 . subplot(2.3 Kernel Principal Component Analysis The Kernel Principal Component Analysis (Kernel PCA) is the nonlinear extension of the ordinary linear PCA. .’PCA direction’). .:). ppatterns(pca_data).1). h2 = plot(pca_rec(1. The computation of the principal components and the projection on these components can be expressed in terms of dot products thus the kernel functions k: X × X → R can be employed. legend([h1 h2]. xi ∈ X ⊆ Rn are mapped by φ: X → F to a high dimensional feature space F . The kernel PCA trains the kernel data projection z = AT k(x) + b .:). ppatterns(lda_data).:).’LDA direction’. title(’PCA’).h1 = plot(lda_rec(1. The procedure consists of the following step: I.1.pca_rec(2. The reconstructed vector φ(x) is given as a linear combination of the mapped data TΦ l ˜ φ(x) = i=1 βi φ(xi ) . (3.9) ˜ φ(xi ) − φ(xi ) i=1 2 .2). subplot(2.9).1. φ(xl )}. This procedure is implemented in the function kpcarec for the case of the Radial Basis Function (RBF) kernel. figure. pgauss(mlcgmm(lda_data)). b) = l l (3. The problem is to ﬁnd the vector x ∈ X its image ˜ φ(x) ∈ F well approximates the reconstructed vector φ(x) ∈ F .’r’). The linear PCA is applied on the mapped data TΦ = {φ(x1 ). . (3. 31 .11) In contrast to the linear PCA. title(’LDA’). . the explicit projection from the feature space F to the input space X usually do not exist. . 3. . . such that the reconstruction error 1 εKM S (A.’b’). pgauss(mlcgmm(pca_data)). β = A(z − b) . Project input vector xin ∈ Rn onto its lower dimensional representation z ∈ Rm using (3. hold on.lda_rec(2.:).10) ˜ is minimized.
such that ˜ xout = argmin φ(x) − φ(xin ) 2 . sl )]T are the kernel functions centered in the vectors TS = {s1 . 32 . the subset TS does not contain all the training vectors TX thus the complexity of the projection (3.new_dim = 2. % generate circle data options.11) while the size d of the subset TS is kept small.arg = 4. .250. Let TX = {x1 . 3.9). The result is visualized in Figure 3.model).4 Greedy kernel PCA algorithm The Greedy Kernel Principal Analysis (Greedy kernel PCA) is an eﬃcient algorithm to compute the ordinary kernel PCA. .4.11). x References: The kernel PCA is described at length in book [29. Example: Kernel Principal Component Analysis The example shows using the kernel PCA for data denoising.1].’Reconstructed’). h2 = ppatterns(XR. The input data are synthetically generated 2dimensional vectors which lie on a circle and are corrupted by the Gaussian noise. 30]. The function kpcarec is used to ﬁnd preimages of the reconstructed data (3.12) where A [d × m] is the parameter matrix. (3. % Visualization h1 = ppatterns(X). . % compute reconstruced data figure. ’+r’). . % compute kernel PCA XR = kpcarec(X. The Greedy kernel PCA algorithm is implemented in the function greedykpca.1). . . sd }. .12) is reduced compared to (3. The goal is to train the kernel data projection z = AT kS (x) + b . . % output dimension model = kpca(X. k(x. % kernel argument options. .ker = ’rbf’. The kernel PCA model with RBF kernel is trained. xi ∈ Rn be the set of input training vectors.options). X = gencircledata([1. xl }. .5.II. s1 ). legend([h1 h2]. In contrast to the ordinary kernel PCA.’Input vectors’. Computes vector xout ∈ Rn which is good preimage of the reconstructed vector ˜ φ(xin ). . . The objective of the Greedy kernel PCA is to minimize the reconstruction error (3. b [m × 1] is the parameter vector and kS = [k(x. % use RBF kernel options. The vector set TS is a subset of training data TX .
.’Reconstructed set’. yl )}. % reconstructed vectors figure. In contrast the ordinary kernel PCA (see experiment in Section 3. Example: Greedy Kernel Principal Component Analysis The example shows using kernel PCA for data denoising. xi ∈ X ⊆ Rn .3) requires all the 250 training vectors. options.X.250. . y1). y1 ). yl )}. The input data are synthetically generated 2dimensional vectors which lie on a circle and are corrupted with the Gaussian noise. options. The function kpcarec is used to ﬁnd preimages of the reconstructed data obtained by the kernel PCA projection. .References: The implemented Greedy kernel PCA algorithm is described in [11]..5 Generalized Discriminant Analysis The Generalized Discriminant Analysis (GDA) is the nonlinear extension of the ordinary Linear Discriminant Analysis (LDA).1]. .ker = ’rbf’.sv. X = gencircledata([1.new_dim = 2.m = 25. % generate training data options. The Greedy kernel PCA model with RBF kernel is trained. (φ(xl ). options.’ob’. . The ordinary LDA is applied on the mapped data TΦY = {(φ(x1 ). h3 = ppatterns(model. .1). . The vector of the selected subset TX are visualized as well.12) is limited to 25. y ∈ Y = {1. . h2 = ppatterns(XR. The number of vectors deﬁning the kernel projection (3. subset run greedy algorithm XR = kpcarec(X. .12). c} are mapped by φ: X → F to a high dimensional feature space F .model). 3. model = greedykpca(X. . . legend([h1 h2 h3]. (xl . The resulting kernel data projection is deﬁned as z = AT k(x) + b . ’Training set’.’+r’). The input training data TXY = {(x1 .’Selected subset’). . 33 .5. The training and reconstructed vectors are visualized in Figure 3.arg = 4. . % % % % % use RBF kernel kernel argument output dimension size of sel.. % visualization h1 = ppatterns(X). The computation of the projection vectors and the projection on these vectors can be expressed in terms of dot products thus the kernel functions k: X × X → R can be employed.5.options).
. In the case of the quadratic data mapping (3.ker = ’rbf’. The RBF kernel with kernel width σ = 1 is used.where A [l × m] is the matrix and b [m × 1] the vector trained by the GDA. % % % % % % % load data use RBF kernel kernel arg. The function f can be seen as a nonlinear function of the input vectors x ∈ X . (xl . (z l . ppatterns(ext_data). . . . quadratic data mapping or kernel projection). The feature extraction step applied to the input data TXY yields extracted data TΦY = {(φ(x1 ).2) used as the feature extraction step. . options. visualization 3. b) are trained to increase the betweenclass scatter and decrease the withinclass scatter of the extracted data TZY = {(z 1 . that setting σ too low leads to the perfect separability of the training data TXY but the kernel projection is heavily overﬁtted.options). y1 ). as m f (x) = w · φ(x) + b = i=1 wiφi (x) + b . φm (x)]T deﬁning a nonlinear feature extraction step (e. . yl )}. (φ(xl ).new_dim = 2. yl )} be a training set of observations xi ∈ X ⊆ Rn and corresponding hidden states yi ∈ Y = {1. References: The GDA was published in the paper [2].g. . . Example: Generalized Discriminant Analysis The GDA is trained on the Iris data set iris. figure. ext_data = kernelproj(orig_data.6 Combining feature extraction and linear classiﬁer Let TXY = {(x1 . output dimension train GDA feature ext.arg = 1. The extracted data YZY are visualized in Figure 3. options. The GDA is implemented in the function gda. . . . The parameters (A. the discriminant function can be written in the following form f (x) = x · Ax + x · b + c . Let φ: X → Rm be a mapping φ(x) = [φ1 (x). . y1 ).model). c}. . . yl )}. Let f (x) = w · φ(x) + b be a linear discriminant function found by an arbitrary training method on the extracted data TΦY .6. . . options. model = gda(orig_data. z i ∈ Rm . . Notice.mat which contains 3 classes of 4dimensional data. . .. orig_data = load(’iris’). y1 ).13) . . 34 (3.
The combining is implement in function lin2quad. the resulting discriminant function has the following form f (x) = α · k(x) + b . This classiﬁer can be trained by (i) applying the kernel projection kernelproj. Function lin2quad is used to compute the parameters of the quadratic classiﬁer explicitly based on the found linear rule in the feature space and the mapping. ppatterns(input_data). % train linear rule in the feature sapce lin_model = perceptron(map_data).g. The Perceptron algorithm produces the linear rule only. The input training data are linearly nonseparable but become separable after the quadratic data mapping is applied. (ii) training a linear rule on the mapped data (e. (3. The 35 .14) deﬁnes the kernel (SVM type) classiﬁer described in Section 5.4..14) The discriminant function (3. % compute parameters of the quadratic classifier quad_model = lin2quad(lin_model). fld) and (iii) combining the quadratic mapping and the linear rule.The discriminant function (3. The combining is implemented in function lin2svm. In the case of the kernel projection (3.13) deﬁnes the quadratic classiﬁer described in Section 6. perceptron.3) applied as the feature extraction step. Example: Quadratic classiﬁer trained the Perceptron This example shows how to train the quadratic classiﬁer using the Perceptron algorithm. % load training data living in the input space input_data = load(’vltava’). pboundary(quad_model). Example: Kernel classiﬁer from the linear rule This example shows how to train the kernel classiﬁer using arbitrary training algorithm for linear rule. % visualize the quadratic classifier figure. (ii) training a linear rule on the extracted data and (iii) combing the kernel projection and the linear rule. Therefore the quadratic classiﬁer can be trained by (i) applying the quadratic data mapping qmap. % map data to the feature space map_data = qmap(input_data). Figure 3.7 shows the decision boundary of the quadratic classiﬁer. The Fisher Linear Discriminant was used in this example.
pboundary(kfd_model).13).options). % train kernel PCA by greedy algorithm options = struct(’ker’.8 shows the found kernel classiﬁer. cerror(ypred. kpca_model = greedykpca(trn.’new_dim’. % visualization figure.kfd_model).0970 36 . Figure 3. % train linear classifier lin_model = fld(map_trn).’ok’.tst.kpca_model).1.greedy kernel PCA algorithm was applied as the feature extraction step. % combine linear rule and kernel PCA kfd_model = lin2svm(kpca_model.’rbf’.sv.y) ans = 0.X.10). ypred = svmclass(tst.’arg’. ppatterns(trn). % load input data trn = load(’riply_trn’). ppatterns(kfd_model. % evaluation on testing data tst = load(’riply_tst’). % project data map_trn = kernelproj(trn. the training data and vectors used in the kernel expansion of the discriminant function.X.lin_model).X. The Riply’s data set was used for training and evaluation.
7
LDA direction PCA direction
6
5
4
3
2
1
0
1
2
3
4
5
6
7
8
(a)
LDA 1.4 1.2 1 0.8 0.6 0.4 0.2 0 −2 −1.5 −1 −0.5 0 0.5 1 1.5 2
PCA 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0 −4 −3 −2 −1 0 1 2 3 4 5
(b) Figure 3.3: Comparison between PCA and LDA. Figure (a) shows input data and found LDA (red) and PCA (blue) directions onto which the data are projected. Figure (b) shows the class conditional Gaussians estimated from data projected data onto LDA and PCA directions.
37
10 Input vectors Reconstructed 8
6
4
2
0
−2
−4
−6
−8 −8
−6
−4
−2
0
2
4
6
8
10
Figure 3.4: Example on the Kernel Principal Component Analysis.
8 Training set Reconstructed set Selected set 6
4
2
0
−2
−4
−6 −8
−6
−4
−2
0
2
4
6
8
10
Figure 3.5: Example on the Greedy Kernel Principal Component Analysis.
38
0.15
0.1
0.05
0
−0.05
−0.1
−0.15
−0.2 −0.6
−0.5
−0.4
−0.3
−0.2
−0.1
0
0.1
0.2
0.3
Figure 3.6: Example shows 2dimensional data extracted from the originally 4dimensional Iris data set using the Generalized Discriminant Analysis. In contrast to LDA (see Figure 3.2 for comparison), the class clusters are more compact but at the expense of a higher risk of overﬁtting.
0.8
0.6
0.4
0.2
0
−0.2
−0.4
−0.6 −0.8
−0.6
−0.4
−0.2
0
0.2
0.4
0.6
0.8
1
Figure 3.7: Example shows quadratic classiﬁer found by the Perceptron algorithm on the data mapped to the feature by the quadratic mapping.
39
2 0 −0. The greedy kernel PCA was used therefore only a subset of training data was used to determine the kernel projection.2 −1.5 0 0. 40 .5 −1 −0.4 0.1.8 0.8: Example shows the kernel classiﬁer found by the Fisher Linear Discriminant algorithm on the data mapped to the feature by the kernel PCA.2 1 0.5 1 Figure 3.6 0.
1 4. Let the structure model represent a given distribution function. . i. however. . . .fun (see Table 4.fun) is used to evaluate yi = PX (xi ).1) where µ [n × 1] is the mean vector and Σ [n × n] is the covariance matrix.fun speciﬁes function which is used to evaluate PX (x) for given set of realization {x1 .1.4. 2 det(Σ) (4. The Matlab command y = feval(X. . .2). the shape of the matrix can be indicated in the optimal ﬁeld .3). The covariance matrix Σ is always represented as the square matrix [n × n]. Σi) is assigned to the class yi ∈ Y. .model. In particular.e. The vector y [1 × l] contains values of the distribution function. The datatype is described in Table 4.Chapter 4 Density estimation and clustering The STPRtool represents a probability distribution function PX (x). The evaluation of the Gaussian distribution is implemented in the function 41 .cov type (see Table 4. the Gaussian distribution and the Gaussian mixture models are described in Section 4.1 Gaussian distribution and Gaussian mixture model The value of the multivariate Gaussian probability distribution function in the vector x ∈ Rn is given by PX (x) = 1 (2π) n 2 1 exp(− (x − µ)T Σ−1 (x − µ)) . . The STPRtool uses speciﬁc datatype to describe a set of Gaussians which can be labeled. xl }. x ∈ Rn as a structure array which must contain ﬁeld . each pair (µi . xl } of a random vector. The ﬁeld . The matrix X [n × l] contains column vectors {x1 . The summary of implemented methods for dealing with probability density estimation and clustering is given in Table 4..
model. Σy ). mmgauss Minimax estimation of Gaussian distribution.1: Implemented methods: Density estimation and clustering gmmsamp Generates sample from Gaussian mixture model. The datatype used to represent the GMM is described in Table 4. pdfgauss. c} are weights and PXY (xy) is the Gaussian distribution (4. emgmm ExpectationMaximization Algorithm for Gaussian mixture model. gsamp Generates sample from Gaussian distribution. kmeans Kmeans clustering algorithm. pdfgauss Evaluates Gaussian mixture model. The random sampling from the Gaussian distribution is implemented in the function gsamp.Table 4. demo mmgauss Demo on minimax estimation for Gaussian. y ∈ Y = {1. (4. . Probability Distribution Function (structure array): . .1) given by parameters (µy . . The Gaussian Mixture Model (GMM) is weighted sum of the Gaussian distributions PX (x) = y∈Y PY (y)PXY (xy) . The sampling from univariate and bivariate Gaussians is shown. demo svmpout Fitting a posteriori probability to SVM output.fun ). sigmoid Evaluates sigmoid function. rsde Reduced Set Density Estimator.2) where PY (y). pdfgauss Evaluates for multivariate Gaussian distribution. . Example: Sampling from the Gaussian distribution The example shows how to represent the Gaussian distribution and how to sample data it. melgmm Maximizes Expectation of LogLikelihood for Gaussian mixture.5.fun [string] The name of a function which evaluates probability distribution y = feval( X. mlsigmoid Fitting a sigmoid function using ML estimation. Table 4. The evaluation of the GMM is implemented in the function pdfgmm.2: Datatype used to represent probability distribution function. The distribution function of the univariate Gaussian is visualized and compared to the histogram 42 . demo emgmm Demo on ExpectationMaximization (EM) algorithm. The random sampling from the GMM is implemented in the function gmmsamp. mlcgmm Maximal Likelihood estimation of Gaussian mixture model.
model). 0. of the sample data (see Figure 4. % Mean vector % Covariance matrix % plot sampled data % plot shape 4. θ)PY Θ (yθ) .’r’). ’diag’ Diagonal covariance matrix. figure. . Table 4.X] = hist(gsamp(model.6.1:5]. µk }. . .Table 4. . hold on.1.cov type [string] Optional ﬁeld which indicates shape of the covariance matrix (see Table 4. yk } assigned to Gaussians.500). . The isoline of the distribution function of the bivariate Gaussian is visualized together with the sample data (see Figure 4.2 MaximumLikelihood estimation of GMM PXY Θ (x. % plot histogram % bivariate case model. . % univariate case model = struct(’Mean’. . .Cov [n × n × k] Covariance matrices {Σ1 . Parameter cov type: ’full’ Full covariance matrix.Y/500). . .4: Datatype used to represent the labeled set of Gaussians.Cov = [1 0. Labeled set of Gaussians (structure array): .250)).1]. model.1:5]. hold on.1b).3: Shape of covariance matrix.’Cov’. yθ) = PXY Θ (xy.3). [Y.pdfgauss([4:0.Mean [n × k] Mean vectors {µ1 .10). % Gaussian parameters figure.2). % plot pdf. 43 Let the probability distribution . pgauss(model). ppatterns(gsamp(model. Σk }. of the Gaussisan plot([4:0.fun = ’pdfgauss’ Identiﬁes function associated with this data type. .6 1]. . . . ’spherical’ Spherical covariance matrix. . % compute histogram bar(X. .y [1 × k] Labels {y1 .1a).Mean = [1.
.i. θ) is assumed to be Gaussian distribution. .2.fun = ’pdfgmm’ Identiﬁes function associated with this data type.1: Sampling from the univariate (a) and bivariate (b) Gaussian distribution. The logarithm of probability P (TXY θ) is referred to as the loglikelihood L(θTXY ) of θ with respect to TXY . 4. Σc }.05 −2 0 −4 −3 −2 −1 0 1 2 3 4 5 6 −3 −2 −1 0 1 2 3 4 5 (a) (b) Figure 4. . . .25 2 0.35 5 4 0.2. Gaussian Mixture Model (structure array): . y1).Cov [n × n × c] Covariance matrices {Σ1 .2. (xl . . The marginal probability PXΘ (xθ) is the Gaussian mixture model (4. 0. .1 −1 0. .Table 4. . be known up to parameters θ ∈ Θ. .5: Datatype used to represent the Gaussian Mixture Model. y ∈ Y of Gaussian components and the values of the discrete distribution PY Θ (yθ). µc }.1 and Section 4. The pairs (xi .1 Complete data The input is a set TXY = {(x1 .127 0 0. The θ involves parameters (µy . . The conditional probability distribution PXY Θ (xy.2 1 0. yθ).2. .Prior [c × 1] Weights of Gaussians {PK (1).Mean [n × c] Mean vectors {µ1 . yi) are assumed to samples from random variables which are independent and identically distributed (i. Σy ).2). . y ∈ Y. . .d. . .) according to PXY Θ (x. PK (c)}. . . . The vector of observations x ∈ Rn and the hidden state y ∈ Y = {1. c} are assumed to be realizations of random variables which are independent and identically distributed according to the PXY Θ (x. The MaximumLikelihood estimation of the parameters θ of the GMM for the case of complete and incomplete data is described in Section 4.3 3 0. Thanks to the 44 . . . yl )} which contains both vectors of observations xi ∈ Rn and corresponding hidden states yi ∈ Y. yθ). respectively. .15 gauss 1 0.
The data are binary labeled thus the GMM has two Gaussians components.struct(’visual’. yiθ) . 45 .i.’contour’)). Thanks to the i. pgmm(model. The problem of MaximumLikelihood (ML) estimation from the complete data TXY is deﬁned as l θ = argmax L(θTXY ) = argmax θ∈Θ θ∈Θ i=1 ∗ log PXY Θ (xi . figure. (4. ppatterns(data).2b) . % load labeled (complete) data model = mlcgmm(data). hold on.d. The estimated GMM is visualized as: (i) shape of components of the GMM (see Figure 4. 4. data = load(’riply_trn’). xl } which contains only vectors of observations xi ∈ Rn without knowledge about the corresponding hidden states yi ∈ Y.2. θ)PY Θ (yi θ) . pgauss(model). Example: MaximumLikelihood estimate from complete data The parameters of the Gaussian mixture model is estimated from the complete Riply’s data set riply trn. % ML estimate of GMM % visualization figure. assumption the loglikelihood can be factorized as l L(θTXY ) = log P (TXY θ) = i=1 log PXY Θ (xi . The logarithm of probability P (TX θ) is referred to as the loglikelihood L(θTX ) of θ with respect to TX .i.i. ppatterns(data). hold on. assumption the loglikelihood can be factorized as l L(θTX ) = log P (TX θ) = i=1 y∈Y PXY Θ (xi yi. . . .3) The ML estimate of the parameters θ of the GMM from the complete data set TXY has an analytical solution.2a) and (ii) contours of the distribution function (see Figure 4.2 Incomplete data The input is a set TX = {x1 . .d. The computation of the ML estimate is implemented in the function mlcgmm. yiθ) .
. The ground truth model. The random initialization is used by default. θ (t) . The random guess or Kmeans algorithm (see Section 4.5].Prior = [0. true_model. . The ground truth model is sampled to obtain a training (incomplete) data. 46 . such that the loglikelihood L(θ (t) TX ) monotonically increases.4) The ML estimate of the parameters θ of the GMM from the incomplete data set TXY does not have an analytical solution. % ground truth Gaussian mixture model true_model. The ground truth model is the GMM with two univariate Gaussian components.The problem of MaximumLikelihood (ML) estimation from the incomplete data TX is deﬁned as l θ = argmax L(θTX ) = argmax θ∈Θ θ∈Θ i=1 y∈Y ∗ PXY Θ (xi yi. The numerical optimization has to be used instead.5) are commonly used to select the initial θ (0) . References: The EM algorithm (including convergence proofs) is described in book [26]. The ﬁrst papers describing the EM are [25.3b. θ (1) .3a. 250). A nice description can be found in book [3]. θ)PY Θ (yi θ) . The EM algorithm for the GMM is implemented in the function emgmm. sampled data and the estimate model are visualized in Figure 4. true_model. .Mean = [2 2]. The ExpectationMaximization (EM) algorithm is an iterative procedure for ML estimation from the incomplete data. < L(θ (t) TX ) until a stationary point L(θ (t−1) TX ) = L(θ (t) TX ) is achieved.. The EM algorithm is a local optimization technique thus the estimated parameters depend on the initial guess θ (0) . L(θ (0) TX ) < L(θ (1) TX ) < .e. sample = gmmsamp(true_model.Cov = [1 0. . Example: MaximumLikelihood estimate from incomplete data The estimation of the parameters of the GMM is demonstrated on a synthetic data. The monograph [18] is entirely devoted to the EM algorithm. The plot of the loglikelihood function L(θ (t) TX ) with respect to the number of iterations t is visualized in Figure 4. An interactive demo on the EM algorithm for the GMM is implemented in demo emgmm. i. The EM algorithm is applied to estimate the GMM parameters.6]. . (4. The EM algorithm builds a sequence of parameter estimates θ (0) . . 5].4 0.
% display progress info estimated_model = emgmm(sample. The algorithm is implemented in the function mmgauss. . 47 . x∈TX (4.’r’)). θ (t) which converges to the optimal estimate θ ∗ .verb = 1.options). .% ML estimation by EM options. h2 = pgmm(estimated_model.ncomp = 2.1). ppatterns(sample.6) holds for ε > 0 which deﬁnes closeness to the optimal solution θ ∗ . ’ML estimation’).struct(’color’. θ (1) . .X). θ∈Θ x∈TX (4. ylabel(’loglikelihood’). xlabel(’iterations’).3 Minimax estimation The input is a set TX = {x1 . % visualization figure. The algorithm converges in a ﬁnite number of iterations to such an estimate θ that the condition x∈TX min log PXΘ (xθ ∗ ) − min log PXΘ (xθ) < ε .logL ). . figure. .6) can be evaluated eﬃciently thus it is a natural choice for the stopping condition of the algorithm. References: The minimax algorithm for probability estimation is described and analyzed in Chapter 8 of the book [26]. The PXΘ (xθ) is known up to the parameter θ ∈ Θ. . xl } contains vectors xi ∈ Rn which are realizations of random variable distributed according to PXΘ (xθ).struct(’color’. h1 = pgmm(true_model.5) If a procedure for the MaximumLikelihood estimate (global solution) of the distribution PXΘ (xθ) exists then a numerical iterative algorithm which converges to the optimal estimate θ ∗ can be constructed. The vectors TX are assumed to be a realizations with high value of the distribution function PXΘ (xθ). The inequality (4. 4. plot( estimated_model. hold on. The STPRtool contains an implementation of the minimax algorithm for the Gaussian distribution (4. % number of Gaussian components options. The minimax estimation is deﬁned as θ ∗ = argmax min log PXΘ (xθ) . .’Ground truth’. An interactive demo on the minimax estimation for the Gaussian distribution is implemented in the function demo mmgauss. The algorithm builds a series of estimates θ (0) . legend([h1(1) h2(1)].’b’)).X. .
struct(’p’. the discriminant function of an arbitrary classiﬁer does not have meaning of probability (e. The found solution is visualized as the isoline of the distribution function at the point PXΘ (xθ) (see Figure 4. for instance. .4). ppatterns(X. Let TXY = {(x1 . yl )} is the training set composed of the vectors xi ∈ X ⊆ Rn and the corresponding binary hidden states yi ∈ Y = {1.7) 48 .4 Probabilistic output of classiﬁer In general. The loglikelihood function is deﬁned as l L(θTXY ) = i=1 log PY F Θ (yif (xi ). Perceptron). 4. The distribution is modeled by a sigmoid function PY F Θ (1f (x). in combining more classiﬁers together. model = mmgauss(X). θ) = 1 . SVM. the minimax problem is equivalent to searching for the minimal enclosing ellipsoid. θ) of the hidden state y given the value of the discriminant function f (x). Let f : X ⊆ Rn → R be a discriminant function trained from the data TXY by an arbitrary learning method. X = [[0. The minimax algorithm is applied to estimate the parameters of the Gaussian. θ) .f. y1 ).’xr’. (xl . In this particular case. % run minimax algorithm % visualization figure. the probabilistic output of the classiﬁer can help in postprocessing. pgauss(model. The parameters θ = [a1 . . 2}. (4.0] [1.exp(model. % points with high p. 1 + exp(a1 f (x) + a2 ) which is determined by parameters θ = [a1 .Example: Minimax estimation of the Gaussian distribution The bivariate Gaussian distribution is described by three samples which are known to have high value of the distribution function. However. a2 ]T can be estimated by the maximumlikelihood method l θ = argmax L(θ TXY ) = argmax θ θ i=1 log PY F Θ (yi f (xi ).. θ ) .13). The aim is to estimate parameters of a posteriori distribution PY F Θ (yf (x). The training set TXY is assumed to be identically and independently distributed from underlying distribution. . . Fitting a sigmoid function to the classiﬁer output is a way how to solve this problem.d. a2 ]T .g.lower_bound))).1]].0] [0.
The function produces the parameters θ = [a1 . svm_model = smo(data.’arg’. The evaluation of the sigmoid is implemented in function sigmoid. Figure 4.1.7).y.’C’. The GMM allows to compute the a posteriori probability for comparison to the sigmoid estimate.The STPRtool provides an implementation mlsigmoid which uses the Matlab Optimization toolbox function fminunc to solve the task (4. The example shows how to train the probabilistic output for the SVM classiﬁer. Table 4.svm_model). % ML estimation of GMM model of SVM output gmm_model = mlcgmm(svm_output).X] = svmclass(data. Figure 4. 49 . hold on.5b shows the decision boundary of SVM classiﬁer for illustration. Sigmoid function (structure array): . a2 ]T which are stored in the datatype describing the sigmoid (see Table 4.A [2 × 1] Parameters θ = [a1 . a2 ]T of the sigmoid function.options).6: Datatype used to describe sigmoid function.10).y = data. % load training data data = load(’riply_trn’). % fit sigmoid function to svm output sigmoid_model = mlsigmoid(svm_output).’rbf’. The example is implemented in the script demo svmpout. svm_output. The Riply’s data set riply trn is used for training the SVM and the ML estimation of the sigmoid. . % train SVM options = struct(’ker’. % visulization figure. References: The method of ﬁtting the sigmoid function is described in [22].fun = ’sigmoid’ Identiﬁes function associated with this data type. % compute SVM output [dummy. Example: Probabilistic output for SVM This example is implemented in the script demo svmpout. The ML estimated of Gaussian mixture model (GMM) of the SVM output is used for comparison.X.svm_output.6).5a shows found probabilistic models of a posteriori probability of the SVM output.
% SVM decision boundary figure. gmm_apost = (pcond(1.sigmoid_model). The task is to ﬁnd such centers θ which describe the set TX best. . µi ∈ Rn be a set of unknown cluster centers. xl } which contains vectors xi ∈ Rn .X)..gmm_apost. θ (1) . hgmm = plot(fx. sigmoid_apost = sigmoid(fx. % sigmoid model fx = linspace(min(svm_output. ylabel(’p(y=1f(x))’). (pcond(1. θ (t) .e.Prior(2))). ..sigmoid_apost.. . Let the objective function F (θ) be deﬁned as l 1 F (θ) = min µy − xi 2 . the task is to solve l 1 ∗ min µy − xi 2 .’k’)..’P(f(x)y=2) MLGMM’). Let θ = {µ1 . θ = argmin F (θ) = argmin y=1.Prior(1)).. 200). . legend([hsigmoid. The Kmeans algorithm converges in a ﬁnite number of steps to the local optimum of the objective function F (θ).c l i=1 θ θ The Kmeans1 is an iterative procedure which iteratively builds a series θ (0) .. .’g’).:)*gmm_model. ppatterns(data). .Prior(1)+(pcond(2. . µc }.X). i. .:)*gmm_model. psvm(svm_model)..5 Kmeans clustering The input is a set TX = {x1 . ’P(y=1f(x)) MLGMM’.:)*gmm_model.c The small value of F (θ) indicates that the centers θ well describe the input set TX .. hsigmoid = plot(fx. hcomp = pgauss(gmm_model).. max(svm_output...’P(f(x)y=1) MLGMM’. gmm_model).hgmm. 4.xlabel(’svm output f(x)’). . . l i=1 y=1. % GMM model pcond = pdfgauss( fx. The found solution depends on the initial guess θ (0) which 1 The K in the name Kmeans stands for the number of centers which is here denoted by c.. ppatterns(svm_output)..’P(y=1f(x)) MLSigmoid’. 50 ./.hcomp]. .. .
% run kmeans % visualization figure. % load data [model. References: The Kmeans algorithm is described in the books [26. The vectors classiﬁed to the corresponding centers are distinguished by color (see Figure 4.data.MsErr). data = load(’riply_trn’).X.6b. pboundary( model ).6a).’sk’. figure. ppatterns(data). 6.X. 4 ). ylabel(’F’). Example: Kmeans clustering The Kmeans algorithm is used to ﬁnd 4 clusters in the Riply’s training set. hold on.y] = kmeans(data. xlabel(’t’).14). The Kmeans algorithm is implemented in the function kmeans. ppatterns(model. The solution is visualized as the cluster centers.is usually selected randomly. The plot of the objective function F (θ (t) ) with respect to the number of iterations is shown in Figure 4. 3]. plot(model. 51 .
2 0 −0. Figure (a) shows components of the GMM and Figure (b) shows contour plot of the distribution function.5 1 (a) 1.2: Example of the MaximumLikelihood of the Gaussian Mixture Model from the incomplete data.2 1 0.4 0.6 0.45 0.5 1 (b) Figure 4.2 0 −0.5 0 0.5 −1 −0.2 1 0.8 0.2 −1.4 gauss 1 0.5 −1 −0.8 gauss 2 0.2 −1.5 0 0. 52 .6 1.1.981 0.
53 .3 0.25 0.4 Ground truth ML estimation 0. Figure (a) shows ground truth and the estimated GMM and Figure(b) shows the loglikelihood with respect to the number of iterations of the EM.0.1 0.05 0 −5 −4 −3 −2 −1 0 1 2 3 4 5 (a) −442 −444 −446 −448 log−likelihood −450 −452 −454 −456 −458 −460 0 5 10 iterations 15 20 25 (b) Figure 4.3: Example of using the EM algorithm for ML estimation of the Gaussian mixture model.35 0.15 0.2 0.
4 −0.304 0.2 −0.2 0.6 0.4 0.2 0 0.4: Example of the Minimax estimation of the Gaussian distribution.1 0.8 0.8 1 Figure 4.4 −0. 54 .2 0 −0.4 gauss 1 0.6 0.
9 0. Figure (b) shows the corresponding SVM classiﬁer.4 0.3 0.2 0 −0.6 0.2 0.5: Example of ﬁtting a posteriori probability to the SVM output.2 1 0.1 P(y=1f(x)) ML−Sigmoid P(y=1f(x)) ML−GMM P(f(x)y=1) ML−GMM P(f(x)y=2) ML−GMM 0.7 0. Figure (a) shows the sigmoid model and GMM model both ﬁtted by ML estimated.8 0.5 1 (b) Figure 4.2 −1. 55 .5 −1 −0.1 0 −4 −3 −2 −1 0 1 svm output f(x) 2 3 4 5 (a) 1.6 p(y=1f(x)) 0.5 0.5 0 0.4 0.8 0.
1.1 F 0.14 0.16 0.6 0.06 0. Figure (a) shows found clusters and Figure (b) contains the plot of the objective function F (θ (t) ) with respect to the number of iterations.08 0.5 0 0.5 −1 −0.2 0 −0.6: Example of using the Kmeans algorithm to cluster the Riply’s data set.8 0. 56 .2 −1.04 0 2 4 6 t 8 10 12 (b) Figure 4.12 0.4 0.5 1 (a) 0.2 1 0.
. Both the binary and the multiclass SVM classiﬁers are implemented in the function svmclass.3. y∈Y. . . . . 2. y∈Y The datatype used by the STPRtool to represent the multiclass SVM classiﬁer is described in Table 5. .2) The datatype used by the STPRtool to represent the binary SVM classiﬁer is described in Table 5. αc ] be composed of all weight vectors and b = [b1 . . bc ]T be a vector of all biases.2. 2. . The binary classiﬁcation rule q: X → Y = {1. 57 .1) The kS (x) = [k(x. c} is deﬁned as (5. .3) q(x) = argmax fy (x) . . k(x. . . si ∈ Rn which are usually subset of the training data. sd }. The α ∈ Rl is a weight vector and b ∈ R is a bias. . 2 for f (x) < 0 . . 2} is deﬁned as q(x) = 1 for f (x) ≥ 0 . (5. y ∈ Y = {1. c} deﬁned as fy (x) = αy · kS (x) + by . sd )]T is the vector of evaluations of kernel functions centered at the support vectors S = {s1 . . . . . (5. . The multiclass classiﬁcation rule q: X → Y = {1. . . .Chapter 5 Support vector and other kernel machines 5. s1 ). The multiclass generalization involves a set of discriminant functions fy : X ⊆ Rn → R. . Let the matrix A = [α1 .1 Support vector classiﬁers The binary support vector classiﬁer uses the discriminant function f : X ⊆ Rn → R of the following form f (x) = α · kS (x) + b . .
. yj }. The majority votes based multiclass classiﬁer assigns the input x into such class y ∈ Y having the majority of votes y = argmax vy (x) . ∀y ∈ Y. (5. . The strategy (5.4) is implemented in the function mvsvmclass.The majority voting strategy is other commonly used method to implement the 1 2 multiclass SVM classiﬁer. . w · xi + b ≤ −1 + ξi .b i=1 i ∈ I1 . yl )} of training vectors xi ∈ Rn and corresponding hidden states yi ∈ Y = {1. The jth rule qj (x) classiﬁes the inputs x into class yj ∈ Y or yj ∈ Y. The implemented methods to deal with the SVM and related kernel classiﬁers are enlisted in Table 5.4) y =1.4. g do vy (x) = vy (x) + 1 where y = qj (x).5) 2 w.g The datatype used to represent the majority voting strategy for the multiclass SVM classiﬁer is described in Table 5. 2}.1 References: Books describing Support Vector Machines are for example [31. j = 1. end. vc (x)]T is computed as: Set vy = 0. i ∈ I2 .2 Binary Support Vector Machines The input is a set TXY = {(x1 . . 30] 5. Let qj : X ⊆ Rn → {yj . . . y1 ). The training of the optimal parameters (w ∗ . . w · xi + b ≥ +1 − ξi . i ∈ I1 ∪ I2 . b ) = argmin w + C ξip .. . Let v(x) be a vector [c × 1] of votes for classes Y when the input x is to be classiﬁed. For j = 1. The vector v(x) = [v1 (x). (5. . .. b∗ ) is transformed to the following quadratic programming task l 1 ∗ ∗ 2 (w .2). . 4. . . 2 for f (x) < 0 . . (xl .. 58 . .. . g be a set of g binary 1 2 SVM rules (5. ξi ≥ 0 . The linear SVM aims to train the linear discriminant function f (x) = w · x + b of the binary classiﬁer q(x) = 1 for f (x) ≥ 0 .
6) subject to β·γ = 1. rbfpreimg3 RBF preimage problem by KwokTsang’s algorithm. minball Minimal enclosing ball in kernel feature space. oaosvm Multiclass SVM using OneAgainstOne decomposition. The ξi ≥ 0. The training of the nonlinear (kernel) SVM with L1soft margin corresponds to solving the following QP task β ∗ = argmax β · 1 − β 1 β · Hβ . svmlight Interface to SV M light software. β ≥ 0. rbfpreimg2 RBF preimage problem by Gradient optimization. smo Sequential Minimal Optimization for binary SVM with L1soft margin. 59 . evalsvm Trains and evaluates Support Vector Machines classiﬁer. kfd Kernel Fisher Discriminant. 2 (5. kperceptr Kernel Perceptron. rbfpreimg RBF preimage by Schoelkopf’s ﬁxedpoint algorithm. diagker Returns diagonal of kernel matrix of given data. the L1soft margin SVM is trained and for p = 2 it is denoted as the L2soft margin SVM. The I1 = {i: yi = 1} and I2 = {i: yi = 2} are sets of indices. In the case of the parameter p = 1.Table 5. rsrbf Reduced Set Method for RBF kernel expansion. svmquadprog SVM trained by Matlab Optimization Toolbox. bsvm2 Multiclass BSVM with L2soft margin. oaasvm Multiclass SVM using OneAgainstAll decomposition. β ≤ 1C . kdist Computes distance between vectors in kernel space. i ∈ I1 ∪ I2 are slack variables used to relax the inequalities for the case of nonseparable data. mvsvmclass Majority voting multiclass SVM classiﬁer. svmclass Support Vector Machines Classiﬁer.1: Implemented methods: Support vector and other kernel machines. kernel Evaluates kernel function. demo svm Demo on Support Vector Machines. The C > 0 stands for the regularization constant. The linear p = 1 and quadratic p = 2 term for the slack variables ξi is used.
The 0 [l × 1] is vector of zeros and the 1 [l × 1] is vector of ones. b= Ib  i∈I b 60 . . bc ]T .1) is determined by the weight vector α. The discriminant function (5. sd }.7) The bias b can be computed from the KarushKuhnTucker (KKT) conditions as the following constrains f (xi ) = α · kS (xi ) + b = +1 f (xi ) = α · kS (xi ) + b = −1 for i ∈ {j: yj = 1. Hi. . .X [n × d] Support vectors S = {s1 . . . . .options. where the set of indices is ISV = {i: βi > 0}.3: Datatype used to describe multiclass SVM classiﬁer. . −βi if yi = 2 .fun = ’svmclass’ Identiﬁes function associated with this data type.Alpha [d × 1] Weight vector α.X [n × d] Support vectors S = {s1 .arg [1 × p] Kernel argument(s). The weight vectors are compute as αi = βi if yi = 1 . . . 0 < βj < C} . sd }. . .j = k(xi . 0 < βj < C} . Binary SVM classiﬁer (structure array): . xj ) if yi = yj . for i ∈ ISV . for i ∈ {j: yj = 2. .options. Multiclass SVM classiﬁer (structure array): . must hold for the optimal solution. The bias b is computed as an average over all the constrains such that 1 γi − α · kS (xi ) .b [c × 1] Vector of biased b = [b1 .options. .fun = ’svmclass’ Identiﬁes function associated with this data type.2: Datatype used to describe binary SVM classiﬁer.sv.ker [string] Kernel identiﬁer. . . The set of support vectors is denoted as S = {xi : i ∈ ISV }. xj ) if yi = yj . where k: X × X → R is selected kernel function. Table 5. .Table 5. .options. (5. . −k(xi .b [1 × 1] Bias b. .Alpha [d × c] Weight vectors A = [α1 . αc ].arg [1 × p] Kernel argument(s).sv. . . −1 if yi = 2 . . . bias b and the set of support vectors S. . The vector γ [l × 1] and the matrix H [l × l] are constructed as follows γi = 1 if yi = 1 . .ker [string] Kernel identiﬁer.
The bias b can be computed from the KKT conditions as the following constrains f (xi ) = α · kS (xi ) + b = +1 − αi 2C αi f (xi ) = α · kS (xi ) + b = −1 + 2C for i ∈ {i: yj = 1.4: Datatype used to describe the majority voting strategy for multiclass SVM classiﬁer.sv. 27. Majority voting SVM classiﬁer (structure array): . yg ] and the bin y(2.8) subject to β·γ = 1.bin y [2 × g] Targets for the binary rules. The bias b is computed as an average over all the constrains thus 1 αi b= ) − α · kS (xi ) . The vector bin y(1. . .:) 1 1 1 contains [y1 . The weight vector α is given by (5.5. . bg ]T . . . . must hold at the optimal solution. . Example: Training binary SVM. 61 . y2 . . y2 . β ≥ 0.arg [1 × p] Kernel argument(s). βj > 0} . The set of support vectors is set to S = {xi : i ∈ ISV } where the set of indices is ISV = {i: βi > 0}. . yg ]. . . . . . .:) contains 2 2 2 [y1 .options. . . .fun = ’mvsvmclass’ Identiﬁes function associated with this data type. γi (1 − ISV  i∈I 2C SV The binary SVM solvers implemented in the STPRtool are enlisted in Table 5. where Ib = {i: 0 < βi < C} are the indices of the vectors on the boundary. .options.Table 5.b [g × 1] Vector of biased b = [b1 . 2 2C (5. αg ]. The training of the nonlinear (kernel) SVM with L2soft margin corresponds to solving the following QP task β ∗ = argmax β · 1 − β 1 1 β · (H + E)β . . . 30]. An interactive demo on the binary SVM is implemented in demo svm. .Alpha [d × g] Weight vectors A = [α1 . . where E is the identity matrix. .ker [string] Kernel identiﬁer. sd }. . βj > 0} .7). The SV M light is described for instance in the book [27]. for i ∈ {i: yj = 2.X [n × d] Support vectors S = {s1 . . References: The Sequential Minimal Optimizer (SMO) is described in the books [23.
trn = load(’riply_trn’). options. The SMO solver is used in the example but other solvers (svmlight. svmquadprog) can by used as well. It can be downloaded from http://svmlight.X. The binary SVM is trained on the Riply’s training set riply trn.5: Implemented binary SVM solvers. SVM solver based on the QP solver quadprog of the Matlab optimization toolbox. % train model = % model = % model = % % % % load training data use RBF kernel kernel argument regularization constant SVM classifier smo(trn. description Implementation of the Sequential Minimal Optimizer (SMO) used to train the binary SVM with L1soft margin.C = 10.model).joachims. It can handle moderate size problems.ker = ’rbf’. Matlab interface to SV M light software (Version: 5. The trained SVM classiﬁer is evaluated on the testing data riply tst.options). cerror(ypred. % visualization figure.org/. The SV M light is very powerful optimizer which can handle large problems.arg = 1.options). svmlight(trn. ppatterns(trn). The function quadprog is a part of the Matlab Optimization toolbox. ypred = svmclass(tst.options).1. options.y) % load testing data % classify data % compute error 62 . tst = load(’riply_tst’).00) which trains the binary SVM with L1soft margin. psvm(model).tst. The decision boundary is visualized in Figure 5.Function smo svmlight svmquadprog Table 5. svmquadprog(trn. The executable ﬁle svm learn for the Linux OS must be installed. options. It useful for small problems only. It trains both L1soft and L2soft margin binary SVM.
yl )} of training vectors xi ∈ X ⊆ Rn and corresponding hidden states yi ∈ Y = {1. The problem can be solved by the OneAgainstAll (OAA) decomposition.0920 1.2 1 0.8 0. (xl . . The goal is to train the multiclass rule q: X ⊆ Rn → Y deﬁned by (5. y are trained by the binary SVM solver from the set TXY . y1 ). .2 0 −0. y∈Y. y ∈ Y. .3 OneAgainstAll decomposition for SVM The input is a set TXY = {(x1 .ans = 0.1: Example showing the binary SVM classiﬁer trained on the Riply’s data. . c}. .2 −1. . The function allows to specify any binary SVM 63 . The OAA decomposition transforms the multiclass problem into a series of c binary subtasks that can be trained by the binary y SVM.5 0 0. Let the training set TXY = {(x1 . The OAA decomposition is implemented in the function oaasvm. The discriminant functions fy (x) = αy · kS (x) + by . (xl . .4 0.6 0. .5 −1 −0.3).5 1 Figure 5. . . . 5. yi = 2 for y = yi . y1 ). . yl )} contain the modiﬁed hidden states deﬁned as 1 for y = yi . 2.
C = 10. i ∈ Ij . 2 2 for yj = yi .solver = ’smo’. g is constructed for all g = c(c − 1)/2 combinations of 1 2 1 classes yj ∈ Y and yj ∈ Y \ {yj }. % visualization figure. ppatterns(data).3) represented by the datatype described in Table 5.4 OneAgainstOne decomposition for SVM The input is a set TXY = {(x1 . . y1). pboundary(model). . (xl . . The OAO decomposition for multiclass SVM is implemented in function 64 .2. The goal is to train the multiclass rule q: X ⊆ Rn → Y based on the majority voting strategy deﬁned by (5. The OAO decomposition transforms the multiclass problem into a series of g = c(c − 1)/2 j binary subtasks that can be trained by the binary SVM. The oneagainstone (OAO) decomposition strategy can be used train such a classiﬁer.’ko’. ppatterns(model. . % % % % % % load data use SMO solver use RBF kernel kernel argument regularization constant training 5. j = 1. .sv.ker = ’rbf’. (xlj . . . . data = load(’pentagon’). The binary SVM rules qj . options. options.3.X. . j = 1. Example: Multiclass SVM using the OAA decomposition The example shows the training of the multiclass SVM using the OAA decomposition. c}. References: The OAA decomposition to train the multiclass SVM and comparison to other methods is described for instance in the paper [12]. options.4). ylj )} contain the training vectors xi ∈ I j = {i: yi = y 1 yi = y 2} and the modiﬁed the hidden states deﬁned as yi = 1 1 for yj = yi . . 2. . The result is the multiclass SVM classiﬁer (5.2) used to solve the binary problems. . y1 ).options ). g are trained on j the data TXY .arg = 1. j The training set TXY . . . . The training data and the decision boundary is visualized in Figure 5. model = oaasvm(data.12). . . Let the training set TXY = {(x1 . . options.solver (see Section 5. yl )} of training vectors xi ∈ X ⊆ Rn and corresponding hidden states yi ∈ Y = {1. . . The SMO binary solver is used to train the binary SVM subtasks.
2 0 0. options. The training data and the decision boundary is visualized in Figure 5.ker = ’rbf’. pboundary(model).2 0. References: The OAO decomposition to train the multiclass SVM and comparison to other methods is described for instance in the paper [12]. options.4 0.2 0 −0.arg = 1.8 1 Figure 5.3. oaosvm.2 −0.2: Example showing the multiclass SVM classiﬁer build by oneagainstall decomposition.8 −1 −1 −0. model = oaosvm(data. Example: Multiclass SVM using the OAO decomposition The example shows the training of the multiclass SVM using the OAO decomposition.6 −0. % visualization figure. data = load(’pentagon’). options.sv.4 −0.6 0.X.’ko’.options).6 −0. The SMO binary solver is used to train the binary SVM subtasks.C = 10. ppatterns(model.4 −0.solver = ’smo’.12). options. The function oaosvm allows to specify an arbitrary binary SVM solver to train the subtasks. ppatterns(data).8 0.1 0.6 0.8 −0. % % % % % % load data use SMO solver use RBF kernel kernel argument regularization constant training 65 .4 0.
.8 −0. where W = [w 1 .4 0.6 −0. . .2 −0. The goal is to train the multiclass rule q: X ⊆ Rn → Y deﬁned by (5.3).8 1 Figure 5. .4 −0. . bc ]T is vector of biases. . . In the linear case. . ξ∗ ) = argmin ( wy 2 + b2 ) + C (ξi )p .8 −1 −1 −0. b∗ . . .6 0. the optimization problem (5. (xl . .ξ) subject to w yi · xi + byi − ( w y · xi + by ) ≥ 1 − ξiy . yl )} of training vectors xi ∈ X ⊆ Rn and corresponding hidden states yi ∈ Y = {1.9) y 2 y∈Y W.2 0 −0. .b i∈I y∈Y\{y} F (W. 2. l] is set of indices.9).3: Example showing the multiclass SVM classiﬁer build by oneagainstone decomposition.4 −0. .5 Multiclass BSVM formulation The input is a set TXY = {(x1 . the parameters of the multiclass rule can be found by solving the multiclass BSVM formulation which is deﬁned as 1 (W∗ .6 0. . ξiy ≥ 0 . . . b = [b1 .6 −0.4 0.2 0. The letter B in BSVM stands for the bias term added to the objective (5. (5. . . y ∈ Y \ {yi } . . i ∈ I . The L1soft margin is used for p = 1 and L2soft margin for p = 2. wc ] is a matrix of normal vectors. y ∈ Y \ {yi } . i ∈ I . . 5.2 0 0.1 0.8 0. y1 ).9) can be 66 .b. ξ is a vector of slack variables and I = [1. c}. In the case of the L2soft margin p = 2.
i ∈ I . i∈I y∈Y\{yi } j∈I u∈Y\{yj } (5. y ∈ Y is given by by = i∈I y∈Y\{yi } u αi (δ(y. . .11) 2C The criterion (5. The nonlinear (kernel) classiﬁer is obtained by substituting the selected kernel k: X × X → R for the dot products x·x to (5. j) = ( xi ·xj +1)(δ(yi .12). Nearest Point Algorithm (solver = ’npa’). (5. (5. y∈Y. u)δ(i. u)) . i.10) can be solved by most the optimizers.9). αc ] are optimized weight vectors. The function h: Y×Y×I×I → R is deﬁned as h(y. FLB + 1 67 . y∈Y.9) can be computed. y))+ δ(y. u. The training of the multiclass BSVM classiﬁer with L2soft margin is implemented in the function bsvm2.expressed in its dual form A∗ = argmax A i∈I y∈Y\{yi } y αi − 1 2 y u αi αj h(y. . The optimizers implemented in the function bsvm2 are enlisted bellow: MitchellDemyanovMalozemov (solver = ’mdm’). yj )+δ(y. u. u)−δ(yi . The optimization task (5. y ∈ Y \ {yi} .12) where the bias by . where A = [α1 . ξ ∗ ) of the criterion (5. The upper bound FU B and the lower bound FLB on the optimal value F (W∗. b∗ . The above mentioned algorithms are of iterative nature and converge to the optimal solution of the criterion (5. j) . y)) + by .10) subject to y αi ≥ 0 . i. .10) corresponds to the singleclass SVM criterion which is simple to optimize.3) is composed of the set of discriminant functions fy : X → R which are computed as fy (x) = i∈I xi · x u∈Y\{yi } u αi (δ(u. Kozinec’s algorithm (solver = ’kozinec’). u)−δ(yj . yi) − δ(y. The bounds are used in the algorithms to deﬁne the following two stopping conditions: Relative tolerance: FU B − FLB ≤ εrel . j) . yi) − δ(u. The multiclass rule (5.11) and (5.
. The Kernel Fisher Discriminant is the nonlinear extension of the linear FLD (see Section 2. Example: Multiclass BSVM with L2soft margin. The other solvers can be used instead by setting the variable options. ppatterns(model. The mapping is performed eﬃciently by means of kernel functions k: X × X → R being the dot products of the mapped vectors k(x.ker = ’rbf’. . The aim is to ﬁnd a direction ψ = l αi φ(xi ) in the feature space i=1 F given by weights α = [α1 . options.Absolute tolerance: FU B ≤ εabs . y1 ).options).3). . α · (N + µE)α α · (N + µE)α 68 (5. The training data and the decision boundary is visualized in Figure 5.X. The MitchellDemyanovMalozemov and Nearest Point Algorithm are described in paper [14]. . . . . x ) = φ(x) · φ(x ) .solver (other options are ’kozinec’.4. y1). data = load(’pentagon’).arg = 1. % visualization figure.10) implemented in the STPRtool is published in paper [9]. options. αl ]T such that the criterion F (α) = α·m 2 α · Mα = . The transformation of the multiclass BSVM criterion to the singleclass criterion (5.’ko’. . ’mdm’). . References: The multiclass BSVM formulation is described in paper [12].sv. options. . ppatterns(data). (xl . The ordinary FLD is applied on the mapped data training data TΦY = {(φ(x1 ). The Kozinec’s algorithm based SVM solver is described in paper [10]. pboundary(model). % % % % % % load data use NPA solver use RBF kernel kernel argument regularization constant training 5. options. yl )}. . The input training vectors are assumed to be mapped into a new feature space F by a mapping function φ: X → F .C = 10. (φ(xl ).13) . The example shows the training of the multiclass BSVM using the NPA solver. yl )} of training vectors xi ∈ X ⊆ Rn and corresponding values of binary state yi ∈ Y = {1.12). 2}. . model = bsvm2(data.solver = ’npa’.6 Kernel Fisher Discriminant The input is a set TXY = {(x1 .
(zl . 69 .8 −0.8 −1 −1 −0. k(xl .2 0 0. The discriminant function f (x) of the binary classiﬁer (5. The kernel matrix K [l × l] contains kernel evaluations Ki. zl } are computed as zi = α · k(xi ) . .2 −0. .2 0 −0. The discriminant function (5. matrices N [l × l] M [l × l] are constructed as my = 1 K1y .13) is the feature space counterpart of the criterion (2.6 −0. . . .j = k(xi .6 0.6 −0.8 0. x)]T is vector of evaluations of the kernel.2) is given by the found vector α as f (x) = ψ · φ(x) + b = α · k(x) + b .4 −0. y1). M = (m1 − m2 )(m1 − m2 )T . . y where the vector 1y has entries i ∈ Iy equal to and remaining entries zeros. Iy my mT . .4 0.4 0. . yl )}. xj ). . m = m1 − m2 .1 0.14) where the k(x) = [k(x1 . .8 1 Figure 5. Iy  N = KKT − y∈Y y∈Y. is maximized.4: Example showing the multiclass BSVM classiﬁer with L2soft margin.14) is known up to the bias b which is determined by the linear SVM with L1soft margin is applied on the projected data TZY = {(z1 .6 0.13) deﬁned in the input space X . The diagonal matrix µE in the denominator of the criterion (5.13) serves as the regularization term. The projections {z1 .2 0.4 −0. (5. The KFD training algorithm is implemented in the function kfd. . x). The criterion (5. . The vector m [l × 1].
mu = 0.y ) ans = 0.arg = 1. i. the isolines with f (x) = +1 and f (x) = −1 are visualized and the support vectors (whole training set) are marked as well. options) % visualization figure. The classiﬁer is visualized as the SVM classiﬁer.. ppatterns(trn). . . Let ψ ∈ F be given by the following kernel expansion l ψ= i=1 αi φ(xi ) . xl } be a set of vector from the input space X ⊆ Rn .7 Preimage problem Let TX = {x1 . The detailed analysis can be found in book [30]. psvm(model). ypred = svmclass( tst. options. The mapping is performed implicitly by using the kernel functions k: X × X → R which are dot products of the input data mapped into the feature space F such that k(xa . options. . xb ) = φ(xa ) · φ(xb ) . 70 (5.5. The kernel methods works with the data nonlinearly mapped φ: X → F into a new feature space F .mat.15) . options. The training data and the found kernel classiﬁer is visualized in Figure 5.0960 % % % % % % load training data use RBF kernel kernel argument regularization of linear SVM regularization of KFD criterion KFD training 5.X.C = 10.001. trn = load(’riply_trn’). cerror( ypred. The found classiﬁer is evaluated on the testing data riply tst.ker = ’rbf’. .e. model ). options. model = kfd(trn.mat. % evaluation on testing data tst = load(’riply_tst’). Example: Kernel Fisher Discriminant The binary classiﬁer is trained by the KFD on the Riply’s training set riply trn.References: The KFD formulation is described in paper [19]. tst.
where α = [α1 .6.16) l l = argmin k(x . . αl ]T is a weight vector of real coeﬃcients.17) which are enlisted in Table 5.2 1 0. the preimage problem solves mapping from the feature space F back to the input space X .5 1 Figure 5.5 xa −xb 2 /σ 2 ) be assumed.17). its feature space image φ(x) ∈ F would well approximate the expansion point Ψ ∈ F given by (5.15). (5. .5: Example shows the binary classiﬁer trained based on the Kernel Fisher Discriminant.17) The STPRtool provides three optimization algorithms to solve the RBF preimage problem (5. The preimage problem can be stated as the following optimization task x = argmin φ(x) − ψ x l 2 = argmin φ(x ) − x i=1 αi φ(xi ) l 2 (5. xj ) .4 0. The preimage problem (5.2 0 −0. However.5 −1 −0. . x ) − 2 x i=1 αi k(x .16) for the particular case of the RBF kernel leads to the following task l x = argmax x i=1 αi exp(−0. xi ) + i=1 j=1 αi αj k(xi . The preimage problem aims to ﬁnd an input vector x ∈ X .1.2 −1. Let the Radial Basis Function (RBF) k(xa . all the methods are guaranteed to ﬁnd only a local optimum of the task (5.6 0.5 0 0. . 71 .8 0. xb ) = exp(−0. In other words.5 x − xi 2 /σ 2 ) .
6: Methods to solve the RBF preimage problem implemented in the STPRtool.2). % load input data data = load(’riply_trn’). rbfpreimg2 A standard gradient optimization method using the Matlab optimization toolbox for 1D search along the gradient. 16]. ppatterns(data.X = data.’or’.1)/num_data. ppatterns(x. The result is visualized in Figure 5.Alpha = ones(num_data.X).6.13). expansion. expansion. rbfpreimg3 A method exploiting the correspondence between distances in the input space and the feature space proposed in [16]. 72 . References: The preimage problem and the implemented algorithms for its solution are described in [28. % define the mean in the feature space num_data = size(data. expansion.arg = 1.ker = ’rbf’. % compute preimage problem x = rbfpreimg(expansion).options.sv. expansion.Table 5. rbfpreimg An iterative ﬁxed point algorithm proposed in [28]. % visualization figure.options. Example: Preimage problem The example shows how to visualize a preimage of the data mean in the feature induced by the RBF kernel.X.
8 Reduced set method The reduced set method is useful to simplify the kernel classiﬁer trained by the SVM or other kernel machines.19) is shorter. ˜ ψ (5. .18). The function f (x) can be expressed in the feature space F as the linear function f (x) = ψ · φ(x) + b. . sm }. βm ]T and the vectors TS = {s1 .19). 5.19) such that the expansion (5. Let φ: X → F be a mapping from the input space X to the feature space F associated with kernel function such that k(xa . . The problem of ﬁnding the 73 . . i. .2 1 0. (5. α = [α1 .8 0. si ∈ Rn determine the reduced kernel expansion (5.5 −1 −0.6: Example shows training data and the preimage of their mean in the feature space induced by the RBF kernel.5 0 0. The weight vector β = [β1 . .. m < l. . .2 0 −0. .e. x) + b = ψ · φ(x) + b . and well approximates the original one (5. Let the function f : X ⊆ Rn → R be given as l f (x) = i=1 αi k(xi .6 0. αl ]T is a weight vector of real coeﬃcients and b ∈ R is a scalar.5 1 Figure 5. .18) where TX = {x1 . x) + b = φ(x) · i=1 βi φ(si ) + b .1. . k: X × X → R is a kernel function. . xl } are vectors from Rn .4 0. The reduced set method aims to ﬁnd a new function m m ˜ f (x) = i=1 βi k(si . . . xb ) = φ(xa ) · φ(xb ) . .2 −1. .
model). the optimization task (5. % visualize decision boundaries figure. ppatterns(trn).X. % load training data trn = load(’riply_trn’). % evaluate SVM classifiers tst = load(’riply_tst’). err = cerror(ypred. In this case.19) can be stated as the optimization task m l 2 ˜ (β. % compute the reduced rule with 10 support vectors red_model = rsrbf(model. ypred = svmclass(tst.struct(’line_style’. The reduced expansion with 10 support vectors is found.’Reduced SVM’). The algorithm converts the task (5.mat.reduced kernel expansion (5. The SVM classiﬁer is trained from the Riply’s training set riply trn/mat.TS 2 = argmin β .7).’Original SVM’. 74 . The reduced set method for the RBF kernel is implemented in function rsrbf. References: The implemented reduced set method is described in [28]. (5.struct(’nsv’. % train SVM classifier model = smo(trn. Decision boundaries of the original and the reduced SVM are visualized in Figure 5.model).struct(’ker’.’r’)). xb ) = exp(−0. h1 = pboundary(model.1.20) can be solved by an iterative greedy algorithm.y).20) into a series of m preimage tasks (see Section 5. Example: Reduced set method for SVM classiﬁer The example shows how to use the reduced set method to simplify the SVM classiﬁer.X.tst.10)). h2 = pboundary(red_model.’arg’.’C’. The original and reduced SVM are evaluated on the testing data riply tst.TS i=1 βi φ(si ) − i=1 αi φ(xi ) .5 xa −xb 2 /σ 2 ) be assumed.20) Let the Radial Basis Function (RBF) k(xa . red_ypred = svmclass(tst. legend([h1(1) h2(1)]. The trained kernel expansion (discriminant function) is composed of 94 support vectors.’rbf’.’b’)).7.struct(’line_style’.10)). TS ) = argmin ψ − ψ β .
err = %.6 0.tst. The original decision rule involves 94 support vectors while the reduced one only 10 support vectors. Original SVM: nsv = 94.4f\n’ ..4 0.8 0.2 Original SVM Reduced SVM 1 0.7: Example shows the decision boundaries of the SVM classiﬁer trained by SMO and the approximated SVM classiﬁer found by the reduced set method.0960 1. err.5 1 Figure 5.5 −1 −0. model.2 0 −0.5 0 0.y).4f\n’]. ’Reduced SVM: nsv = %d.nsv.red_err = cerror(red_ypred.2 −1. err = 0.. red_err)..nsv.. 75 . . err = 0. red_model. err = %.0960 Reduced SVM: nsv = 10. fprintf([’Original SVM: nsv = %d.
Input data for the demo on the Generalized Anderson’s task demo anderson. Testing part of the Riply’s data set [24].Chapter 6 Miscellaneous 6.mat /riply tst. Well known Fisher’s data set. Input data for the demo on training linear classiﬁers from ﬁnite vector sets demo linclass. The datasets stored in the Matlab data ﬁle are enlisted in Table 6. Input data for the demo on the ExpectationMaximization algorithm for the Gaussian mixture model demo emgmm. The functions which generate synthetic data and scripts for converting external datasets to the Matlab are given in Table 6.mat /ocr data/*. Table 6.mat 76 .1 Data sets The STPRtool provides several datasets and functions to generate synthetic data.1: Datasets /riply trn.1.mat /svm samples/*. A description is given in Section 7. Examples of handwritten numerals.mat /binary separable/*.mat /iris. /multi separable/*.2. Training part of the Riply’s data set [24].mat stored in /data/ directory of the STPRtool. Linearly separable multiclass data in 2D.mat /anderson task/*. Input data for the demo on the Support Vector Machines demo svm.mat /gmm samles/*.1.
createdata genlsdata gmmsamp gsamp gencircledata usps2mat Table 6. pgauss Visualizes set of bivariate Gaussians. pboundary Plots decision boundary of given classiﬁer in 2D.3. ppatterns Visualizes patterns as points in the feature space. It works for 1D. It operates in 2dimensional space only. Generates data from the multivariate Gaussian distribution. 6. The list of functions for visualization is given in Table 6.2 Visualization The STPRtool provides tools for visualization of common models and data. pline Plots line in 2D. Postal Service (USPS) database of handwritten numerals to the Matlab ﬁle.3: Functions for visualization. Each function contains an example of using. showim Displays images entered as column vectors. pkernelproj Plots isolines of kernel projection. pandr 77 . Generates data from the Gaussian mixture model. Visualizes solution of the Generalized Anderson’s task. Table 6.2: Data generation. Generates linearly separable binary data with prescribed margin.S. GUI which allows to create interactively labeled vectors sets and labeled sets of Gaussian distributions. Converts U. 2D and 3D. Generates data on circle corrupted by the Gaussian noise. psvm Plots decision boundary of binary SVM classiﬁer in 2D. plane3 Plots plane in 3D. pgmm Visualizes the bivariate Gaussian mixture model.
The 0/1loss function W0/1 (q(x). y∈Y 78 . The rule q: X → Y which minimizes the expectation of misclassiﬁcation is deﬁned as q(x) = argmax PY X (yx) . The Bayesian risk (6. The x and y are realizations of random variables with joint probability distribution PXY (x.1) with the 0/1loss function corresponds to the expectation of misclassiﬁcation. where ε is penalty for the decision dont know.4) is deﬁned as ⎧ ⎨ argmax PXY (xy)PY (y) if 1 − max PY X (yx) < ε . . y). c}.4) Wε (q(x). y∈Y Classiﬁcation with rejectoption: The set of decisions D is assumed to be D = Y ∪ {dont know}. ∀x ∈ X . y) = ⎩ ε for q(x) = dont know . The Bayesian risk R(q) is an expectation of the value of the loss function W when the decision rule q is applied. y)W (q(x).3) = argmax PXY (xy)PY (y) . The loss function is deﬁned as ⎧ ⎨ 0 for q(x) = y .6. y) dx .. y∈Y (6. y)W (q(x). (6.1) is referred to as the Bayesian rule q ∗ (x) = argmin y y∈Y PXY (x. (6. y∈Y y∈Y (6. Let W : D × Y → R be a loss function which penalizes the decision q(x) ∈ D when the true hidden state is y ∈ Y. 1 for q(x) = y . 1 for q(x) = y . The STPRtool implements the Bayesian rule for two particular cases: Minimization of misclassiﬁcation: The set of decisions D coincides to the set of hidden states Y = {1. A decision rule q: X → D takes a decision d ∈ D based on the observation x ∈ X .e. y) . i.5) q(x) = ⎩ dont know if 1 − max PY X (yx) ≥ ε . . . The rule q: X → Y which minimizes the Bayesian risk with the loss function (6.2) is used.1) The optimal rule q ∗ which minimizes the Bayesian risk (6. (6. Let X ⊆ Rn and the sets Y and D be ﬁnite. R(q) = X y∈Y PXY (x. . y) = 0 for q(x) = y .3 Bayesian classiﬁer The object under study is assumed to be described by a vector of observations x ∈ X and a hidden state y ∈ Y.
Example: Bayesian classiﬁer The example shows how to design the plugin Bayesian classiﬁer for the Riply’s training set riply trn. The designed Bayesian classiﬁer is evaluated on the testing data riply tst.2)). y ∈ Y. .Pclass{1} = emgmm(trn. The datatype used by the STPRtool is described in Table 6. cerror(ypred.X(:. % Estimation of classconditional distributions by EM bayes_model. n2 = length(inx2). bayes_model. % Estimation of priors n1 = length(inx1).y==2).Prior [1 × c] Discrete distribution PY (y).y) 79 . The a priori probabilities PY (y).3) and (6. ypred = bayescls(tst.struct(’ncomp’.5) are implemented in function bayescls. inx1 = find(trn.struct(’ncomp’. Bayesian classiﬁer (structure array): .Pclass{2} = emgmm(trn.inx1).2.4.tst.fun = ’bayescls’ Identiﬁes function associated with this data type. The GMM are estimated by the EM algorithm emgmm.X. 2} are estimated by the relative occurrences (it corresponds to the ML estimate). The distribution PXY (xy) is given by Pclassy which uses the datatype described in Table 4. bayes_model.inx2). References: The Bayesian decision making with applications to PR is treated in details in book [26]. y ∈ {1. The classconditional distributions PXY are modeled by the Gaussian mixture models (GMM) with two components.Pclass [cell 1 × c] Class conditional distributions PXY .X(:. % load input training data trn = load(’riply_trn’). . % Evaluation on testing data tst = load(’riply_tst’). inx2 = find(trn.mat.4: Datatype used to describe the Bayesian classiﬁer.Prior = [n1 n2]/(n1+n2). Table 6. The Bayesian classiﬁers (6.2)).bayes_model).y==1).mat.To apply the optimal classiﬁcation rules one has to know the classconditional distributions PXY and a priory distribution PY (or their estimates).
% Vislualization of rejetoption rule pboundary(reject_model. The visualization is given in Figure 6. The decision boundary of the rejectoptions rule (dashed line) is displayed for comparison. The quadratic discriminant function fy is determined by a matrix Ay [n × n].fun = ’bayescls’. The input vector x ∈ Rn is assigned to the class y ∈ Y its corresponding discriminant function fy attains maximal value y = argmax fy (x) = argmax ( x · Ay x + by · x + cy ) . 2}. . . 6.0900 The following example shows how to visualize the decision boundary of the found Bayesian classiﬁer minimizing the misclassiﬁcation (solid black line).1. which are quadratic with respect to the input vector x ∈ Rn . the quadratic classiﬁer is represented by a single discriminant function f (x) = x · Ax + b · x + c . ∀y ∈ Y . pboundary(bayes_model). . bayes_model. % Visualization figure. c} is composed of a set of discriminant functions fy (x) = x · Ay x + by · x + cy .struct(’line_style’.ans = 0. 80 . .’k’)). 2. hold on. y ∈Y y ∈Y (6.1.4 Quadratic classiﬁer The quadratic classiﬁcation rule q: X ⊆ Rn → Y = {1. % Penalization for don’t know decision reject_model = bayes_model.6) In the particular binary case Y = {1. The rejectoption rule splits the feature space into three regions corresponding to the ﬁrst class. reject_model. the second class and the region in between corresponding to the dont know decision. ppatterns(trn).eps = 0. The Bayesian classiﬁer splits the feature space into two regions corresponding to the ﬁrst class and the second class. a vector by [n × 1] and a scalar cy [1 × 1].
The input vector x ∈ Rn is assigned to class y ∈ {1. . 2 if f (x) = x · Ax + b · x + c < 0 . (6.c [1 × 1] Parameter scalar c.8 0.6) quadratic classiﬁer. .5) and the multiclass (see Table 6. Example: Quadratic classiﬁer The example shows how to train the quadratic classiﬁer using the Fisher Linear Discriminant and the quadratic data mapping.1 (dashed line). a vector b [n × 1] and a scalar c [1 × 1].1. The classiﬁer is trained from the Riply’s 81 . The classconditional probabilities are modeled by the Gaussian mixture models with two components estimated by the EM algorithm.5: Datatype used to describe binary quadratic classiﬁer. given by a matrix A [n × n].2 1 0.5 1 Figure 6. Table 6. Binary linear quadratic (structure array): .2 0 −0.2 −1. 2} as follows q(x) = 1 if f (x) = x · Ax + b · x + c ≥ 0 .5 −1 −0.fun = ’quadclass’ Identiﬁes function associated with this data type. .5 0 0.b [n × 1] Parameter vector b.7) The STPRtool uses a speciﬁc structure array to describe the binary (see Table 6.4 0.1: Figure shows the decision boundary of the Bayesian classiﬁer (solid line) and the decision boundary of the rejectoption rule with ε = 0.6 0.A [n × n] Parameter matrix A. The implementation of the quadratic classiﬁer itself provides function quadclass.
.6: Datatype used to describe multiclass quadratic classiﬁer. trn = load(’riply_trn’). .quad_model). (xl . . . . . quad_data = qmap(trn). % make quadratic classifier quad_model = lin2quad(lin_model).A [n × n × c] Parameter matrices Ay .mat. y1 ). y ∈ Y. yl )} be a set of prototype vectors xi ∈ X ⊆ Rn and corresponding hidden states yi ∈ Y = {1. .X.8) y∈Y 82 . training set riply trn. .fun = ’quadclass’ Identiﬁes function associated with this data type. y) . cerror(ypred. % evaluation tst = load(’riply_tst’). l}.mat. The found quadratic classiﬁer is evaluated on the testing data riply trn.b [n × c] Parameter vectors by . Multiclass quadratic classiﬁer (structure array): .. . The Knearest neighbor (KNN) classiﬁcation rule q: X → Y is deﬁned as q(x) = argmax v(x. i. % visualization figure.tst. .0940 % load input data % quadratic mapping % train FLD 6. . Let Rn (x) = {x : x − x ≤ r 2 } be a ball centered in the vector x in which lie K prototype vectors xi . i ∈ {1. (6. ppatterns(trn). ypred = quadclass(tst.y) ans = 0.Table 6.5 Knearest neighbors classiﬁer Let TXY = {(x1 . y ∈ Y. c}. lin_model = fld(quad_data). . . .2). The boundary of the quadratic classiﬁer is visualized (see Figure 6.e. y ∈ Y. {xi : xi ∈ Rn (x)} = K. .c [1 × c] Parameter scalars cy . pboundary(quad_model).
4 0.fun = ’knnclass’ Identiﬁes function associated with this data type. KNN classiﬁer (structure array): . for instance.mat. The classiﬁer is evaluated on the testing data riply tst. .mat. y) is number of prototype vectors xi with hidden state yi = y which lie in the ball xi ∈ Rn (x).2: Figure shows the decision boundary of the quadratic classiﬁer.1. xl }.8 0.6 0. .2 −1.8). .5 0 0. .3. . . % load training data and setup 8NN rule trn = load(’riply_trn’). model = knnrule(trn.8) is computationally demanding if the set of prototype vectors is large. where v(x. in book [6]. References: The Knearest neighbors classiﬁer is described in most books on PR. .y [1 × l] Labels {y1 . . . The classiﬁcation rule (6.7. . The datatype used by the STPRtool to represent the Knearest neighbor classiﬁer is described in Table 6. 83 .X [n × l] Prototype vectors {x1 . yl }.7: Datatype used to describe the Knearest neighbor classiﬁer. Table 6.2 0 −0.5 1 Figure 6.5 −1 −0. The decision boundary is visualized in Figure 6. Example: Knearest neighbor classiﬁer The example shows how to create the (K = 8)NN classiﬁcation rule from the Riply’s training data riply trn.2 1 0.
ppatterns(trn). cerror(ypred.% visualize decision boundary and training data figure.y) ans = 0.tst. ypred = knnclass(tst.5 1 Figure 6.5 0 0.model).2 −1. pboundary(model).2 0 −0.8 0.2 1 0.5 −1 −0. 84 .1160 1.4 0.3: Figure shows the decision boundary of the (K = 8)nearest neighbor classiﬁer.X.6 0. % evaluate classifier tst = load(’riply_tst’).
To see the trained OCR system run the demo script demo ocr. The function mpaper is designed to invoke a speciﬁed function whenever the middle mouse button is pressed. The drawn numerals are normalized to size 16 × 16 pixels (diﬀerent size can be speciﬁed). 7. The applications selected are the optical character recognition (OCR) system based on the multiclass SVM (Section 7. The set of training examples has to be collected before the OCR is trained.2).1) and image denoising system using the kernel PCA (Section 7.1 Optical Character Recognition system The use of the STPRtool is demonstrated on a simple Optical Character Recognition (OCR) system for handwritten numerals from 0 to 9. The function mpaper creates a form composed of 50 cells to which the numerals can be drawn (see Figure 7. The function save chars saves drawn numerals in the stage of collecting training examples and the function ocr fun is used to recognize the numerals if the trained OCR is applied. The function collect chars invokes the form for drawing numerals and allows to save the numerals to a speciﬁed ﬁle. A numeral to be classiﬁed is represented as a vector x containing pixel values taken from the numeral image. The STPRtool provides GUI which allows to use standard computer mouse as an input device to draw images of numerals.1a). The training stage of the OCR system involves (i) collection of training examples of numerals and (ii) training the multiclass SVM. The GUI is intended just for demonstration purposes and the scanner or other device would be used in practice instead. The multiclass SVM are used as the base classiﬁer of the numerals. The GUI for drawing numerals is implemented in a function mpaper. The STPRtool contains examples of numerals collected 85 . the functions save chars and ocr fun are called from the mpaper to process drawn numerals. In particular.Chapter 7 Examples of applications We intend to demonstrate how methods implemented in the STPRtool can be used to solve a more complex applications.
The SVM has two free parameters which have to be tuned: the regularization constant C and the kernel function k(xa . . The directory /demo/ocr/models/ contains the SVM model trained by OAO decomposition strategy (function oaosvm) and model trained based on the multiclass BSVM formulation (function bsvm2). Having the labeled training set TXY = {(x1 . Figure 7. 86 . xb ) with its argument.5 xa − xb 2 /σ 2 ) determined by the parameter σ was used in the experiment. to collect examples of numeral ’1’ from person honza cech collect_chars(’honza_cech1’) was called. (Ck . σ1 ). The training procedure for OCR classiﬁer is implemented in the function train ocr. if a speed of classiﬁcation is the major objective the multiclass BSVM formulation is often a better option as it produce less number of support vectors. xb ) = exp(−0. The resulting ﬁle contains a matrix X of all examples and a vector y containing labels are assigned to examples according the number xx. . yl )} created the multiclass SVM classiﬁer can be trained. The RBF kernel k(xa . After the parameters (C ∗ . σ ∗ ) are tuned the SVM classiﬁer is trained using all training data available. The numerals are stored in the directory /data/ocr numerals/. .1a shows the form ﬁlled with the examples of the numeral ’1’.1b shows numerals after normalization which are saved to the speciﬁed ﬁle honza cech. (xl .from 6 diﬀerent persons. The function evalsvm returns the SVM model its parameters (C ∗ .DataFile) can be used to merge all the ﬁle to a single one.mat (the character can be omitted from the name) is used. y1 ). Therefore. the free SVM parameters are C and σ.mat are stored and the string FileName is a ﬁle name to which the resulting training set is stored. The string Directory speciﬁes the directory where the ﬁles name xx.mat is met then the function mergesets(Directory. The parameters C = ∞ (data are separable) and σ = 5 were found to be the best. Each person drew 50 examples of each of numerals from ’0’ to ’9’.mat. If the ﬁle name format name xx. σ ∗ ) turned out to be have the lowest crossvalidation error over the given set Θ. A common method to tune the free parameters is to use the crossvalidation to select the best parameters from a preselected set Θ = {(C1 . . σk )}. The ﬁle name format name xx. . The images of normalized numerals are stored as vectors to a matrix X of size [256 × 50]. For example. The vector dimension n = 256 corresponds to 16 × 16 pixels of each image numeral and l = 50 is the maximal number of numerals drawn to a single form. The string name is name of the person who drew the particular numeral and xx stands for a label assigned to the numeral. . The STPRtool contains a function evalsvm which evaluates the set of SVM parameters Θ based on the crossvalidation estimation of the classiﬁcation error. The parameter tuning is implemented in the script tune ocr. The methods to train the multiclass SVM are described in Chapter 5. . However. The Figure 7. the OneAgainstOne (OAO) decomposition was used as it requires the shortest training time. . The labels from y = 1 to y = 9 corresponds to the numerals from ’1’ to ’9’ and the label y = 10 is used for numeral ’0’. In particular.
.2. The model should describe the nonlinear subspace Ximg ⊆ Rn where the images live.mat its contains is summarized in Table 7. . The corruption of the USPS images by the Gaussian noise is implemented in script make noisy data.The particular model used by OCR is speciﬁed in the function ocr fun which implicitly uses the ﬁle ocrmodel.. 7. The U. A link between the input space X and the feature space F is given by a mapping φ: X → F which is implicitly determined by a selected kernel function k: X × X → R.3. . . To run the resulting OCR type demo ocr or call mpaper(struct(’fun’. The kernel function corresponds to the dot product in the feature space. The subspace Fimg is modeled by the kernel PCA trained from the set TX mapped into the feature space F by a function φ: X → F .e. The additive Gaussian noise was assumed as the source of image degradation. Having the kernel PCA model of the subspace Fimg it can be ˆ used to map the images x ∈ X into the subspace Ximg . . xb ) = φ(xa ) · φ(xb ) . φ(xl )} denote the feature space representation of the training images TX .2 Image denoising The aim is to train a model of a given class of images in order to restore images corrupted by noise. z i ∈ Rm of the mapped images 87 .1. . z l }.S. k(xa . The STPRtool contains a script which converts the USPS database to the Matlab data ﬁle. i. Postal Service (USPS) database of images of handwritten numerals [17] was used as the class of images to model. A snapshot of running OCR is shown in Figure 7. Thus the unlabeled training data set TX = {x1 . The script make noisy data produces the Matlab ﬁle usps noisy.mat stored in the directory /demo/ocr/. The idea of using the kernel PCA for image denoising is demonstrated in Figure 7. . The aim is to train a model of images of all numerals at ones.’ocr_fun’)) If one intends to modify the classiﬁer or the way how the classiﬁcation results are displayed then modify the function ocr fun. It is assumed that the nonlinear subspace Ximg forms a linear subspace Fimg of a new feature space F . A noisy image x is assumed to lie outside the subspace Ximg . . The denoising procedure is based on mapping the noisy ˆ image x into the subspace Ximg . The kernel PCA can be used to ﬁnd the basis vectors of the subspace Fimg from the input training data TX . . The example which produced the ﬁgure is implemented in demo kpcadenois. It is implemented in the function usps2mat. . . Let TΦ = {φ(x1 ). The kernel PCA computes the lower dimensional representation TZ = {z 1 . . This method was proposed in [20] and further developed in [15]. The kernel PCA was used to model the image class. xl } is used.
2) can be well solved for this case. (7. Structure array trn containing the training data: gnd X [256 × 7291] Training images of numerals represented as column vectors. xb ) = exp(−0. . However. . . (7. .5 xa − xb 2 /σ 2 ) was used here as the preimage problem (7. Structure array tst containing the testing data: gnd X [256 × 2007] Testing images of numerals represented as column vectors. . . The input are 2007 grayscale images of size 16 × 16.Table 7. y [1 × 7291] Labels y ∈ Y = {1.1: The content of the ﬁle usps noisy. the nonlinearity hidden in 88 . . x). Given a projection φ(x) onto the kernel PCA subspace an n image x ∈ R of the input space can be recovered by solving the preimage problem ˜ x = argmin φ(x) − φ(x) x 2 . X [256 × 7291] Training images corrupted by the Gaussian noise with signaltonoise ration SNR = 1 (it can be changed).mat produced by the script make noisy data.1) The vector k(x) = [k(x1 . y [1 × 2007] Labels y ∈ Y = {1. . . k(xl . x)]T contains kernel functions evaluated at the training data TX . In contrast to the linear PCA. 10} of training images. The input are 7291 grayscale images of size 16 × 16.1) to ˜ compute z which determines φ(x) and (ii) solving the preimage problem (7. i=1 β = A(z − b) . the kernel PCA allows to model the nonlinearity which is very likely in the case of images.2) The Radial Basis Function (RBF) kernel k(xa . . X [256 × 2007] Training images corrupted by the Gaussian noise with signaltonoise ration SNR = 1 (it can be changed). .2) which yields the resulting x . . TΦ to minimize the mean square reconstruction error l εKM S = i=1 ˜ φ(xi ) − φ(xi ) 2 . This procedure is implemented in function kpcarec. where ˜ φ(x) = l βi φ(x) . z = AT k(x) + b . 10} of training images. The whole ˆ procedure of reconstructing the corrupted image x ∈ Rn involves (i) using (7. The matrix A [l × m] and the vector b [m × 1] are trained by the ˜ kernel PCA algorithm.
. The whole training procedure of the kernel PCA is implemented in the script train kpca denois. The results of tuning the parameters are displayed in Figure 7. Figure 7. The linear PCA was used to solve the same task for comparison. the function pcarec was used.the parameter σ of the used RBF kernel has to be tuned properly to prevent overﬁtting. .1) and (7. images restored by the linear PCA and kernel PCA. The function kpcarec is used to denoise images using the kernel PCA model. The x stands for the ground truth image and x denotes the image ˆ restored from noisy one x using (7. The training for linear PCA is implemented in the script train lin denois. The same idea was used to tune the output dimension of the linear PCA. The greedy kernel PCA Algorithm 3. It produces sparse model and can deal with large data sets.2). (σk . noisy images. The images are assumed to be generated from distribution PX (x). Figure 7. Having the parameters tuned the resulting kernel PCA and linear PCA are trained on the whole training set. εM S (σ. The ﬁgure was produced by a script show denois tuning. mk )} such that the crossvalidation estimate of the mean square error PX (x) x − x 2 dx . In the case of the linear PCA. m1 ).6 shows examples of ground truth images. . Another parameter is the dimension m of the subspace kernel PCA subspace to be trained.4 is used for training the model. m∗ ) was selected out of a prescribed set Θ = {(σ1 . The best parameters (σ ∗ . m) = X was minimal.4 shows the idea of tuning the parameters of the kernel PCA. 89 . .5. An example of denoising of the testing data of USPS is implemented in script show denoising.
05 0 0 0. 0.2 0.1 0.2 0. middle_button − classify. 90 .35 0.1: Figure (a) shows handwritten training examples of numeral ’1’ and Figure (b) shows corresponding normalized images of size 16 × 16 pixels.Control: left_button − draw.1 0.6 0.15 0.25 0.9 1 (a) (b) Figure 7.7 0.45 0.4 0.5 0.5 0.4 0.3 0.3 0.8 0. right_button − erase.
45 0.3 0.4 0.2 0.5 0.3 0. Figure (a) shows handwritten numerals and Figure (b) shows recognized numerals.1 0. middle_button − classify. 0.2: A snapshot of running OCR system.Control: left_button − draw.1 0.15 0. 91 . right_button − erase.4 0.4 0 3 2 0.7 0.05 0 0 0.2 0.5 0.9 1 (a) 0.2 0.05 0 0 1 2 3 4 5 6 7 8 9 0 4 5 6 0.4 0.5 0.25 0.35 0.6 0.3 0.45 0.35 0.1 0.7 0.15 0.9 1 (b) Figure 7.5 0.2 0.8 1 9 8 0.8 0.25 0.3 0.1 0.6 0.
3: Demonstration of idea of image denoising by kernel PCA. Preimage problem error θ Data generator Error assessment 92 .7 6 5 4 3 2 1 0 −1 −2 −3 −4 −4 Ground truth Noisy examples Corrupted Reconstructed −3 −2 −1 0 1 2 3 4 5 6 7 Figure 7.4: Schema of tuning parameters of kernel PCA model used for image restoration. Distortion model KPCA model θ∈Θ Figure 7.
18 20 dim = 50 dim = 100 dim = 200 dim = 300 17 18 16 16 15 14 εMS 14 εMS 12 10 8 6 3. Ground truth Numerals with added Gaussian noise Linear PCA reconstruction Kernel PCA reconstruction Figure 7.5 σ 6 6.5 13 12 11 10 0 10 20 30 40 50 dim 60 70 80 90 100 4 4. 93 .5 7 7.6: The results of denosing the handwritten images of USPS database.5: Figure (a) shows plot of εM S with respect to output dimension of linear PCA. Figure(b) shows plot of εM S with respect to output dimension and kernel argument of the kernel PCA.5 5 5.5 8 (a) (b) Figure 7.
• Eﬃcient optimizers to for large scale Support Vector Machines learning problems. We welcome anyone who likes to contribute to the STPRtool in any way. Therefore we are grateful for any comment. There are many relevant methods and proposals which have not been implemented yet. suggestion or other feedback which would help in the development of the STPRtool. • Discrete probability models estimated by the EM algorithm. Please send us an email and your potential contribution. 94 . • AdaBoost and related learning methods. • Probabilistic Principal Component Analysis (PPCA) and mixture of PPCA estimated by the ExpectationMaximization (EM) algorithm. • Model selection methods for SVM.Chapter 8 Conclusions and future planes The STPRtool as well as its documentation is being updating. The future planes involve the extensions of the toolbox by the following methods: • Feature selection methods.
Franc and V. Neural Networks for Pattern Recognition. Hlav´ˇ. Classiﬁcation into two multivariate normal distributions with diﬀerrentia covariance matrices. and D.E. Hlav´ˇ.G. ac [10] V. editors. ShaweTaylor. [8] V. Kasturi. 2000.W. 1977. Anderson and R. Duin. Prtools: A matlab toolbox for pattern recognition.W. Bahadur. Laurendeau. and D. Stork. Anouar. In Czech). Multiclass support vector machine. Duda. Neural Computation. Great Britain. pages 236–239. 1997. Cambridge University Press. Hart. Baudat and F. Pattern Recognition. 2000. Support Vector Machines.P.P. Oxford. Journal of the Royal Statistical Society. P. volume 2. 33:420–431. [5] A.O. An iterative algorithm learning the maximal margin classiﬁer. [6] R. 2001. Fakulta e c ı e elektrotechnick´. February 2000. [2] G. Franc. Programov´ n´stroje pro rozpozn´v´n´ (Pattern Recognition Programe a a a ı ˇ e ming Tools. 95 . Laird. Bishop. 2003. Franc and V.R. ac D. and Suen C. edition. 2002. Rubin. N. a [9] V.. Katedra kybernetiky. Generalized discriminant analysis using a kernel approach. 12(10):2385–2404. Cesk´ vysok´ uˇen´ technick´. Anals of Mathematical Statistics. John Wiley & Sons. June 1962. Cristianini and J. 3th edition. Master’s thesis. Pattern Classiﬁcation. 39:185– 197. 36(9):1985–1996. [7] R. Maximum likelihood from incomplete data via the EM Algorithm. 2nd. In R. Clarendon Press. [3] C. Dempster. 2000. [4] N.M.B.M. IEEE Computer Society.Bibliography [1] T. 16th International Conference on Pattern Recognition.
Springer. 1989. January 2000. R. MA. pages 41–48. E. Henderson. Mika.[11] V. 96 .J Jackel. M¨ ller. In M. a o u [19] S. Murthy. ECCV Workshop. Douglas. March 2002. Statisical Learning in Computer Vision. [18] G. Keerthi. Springer. In N. Sch¨lkopf.J. In Leonardis Aleˇ and Bischof Horst. Hlav´ˇ. New York. Kernel o u a pca and denoising in feature spaces. Neural Networks for Signal Processing. M¨ ller.E. J.T. 2002. Solla. Smola. J.J. B.J.T. New York.C. editors. Smola. 1999. In Proceedings of the Twentieth International Conference on Machine Learning (ICML2003). [21] I. R¨tsch. Cambridge. A comparison of methods for multiclass support vector machins. Nabney. Scholkopf. W. Mika.. McLachlan and T. IEEE Transactions on Neural Networks.. Washington. Shevade. and Sch¨lkopf Bernhard. pages 536 – 542. K. Franz Matthias.K. Schuurmans. Jollife. John Wiley & Sons. Hubbard. editors.S. 1999. 11(1):124–136. and S. [12] C. [17] Y. In A. [14] S. Boser. MIT Press. Scholz. D. Cohn. In.R. and K. Hu. M. Springer. Krishnan.S. Probabilities for sv machines.K. 1986. A. [15] Kim Kwang. 1997. MIT Press. and D. editors. [13] I. and K. [20] S. P. Backpropagation applied to handwritten zip code recognition. Hsu and C. [22] J.H. NETLAB : algorithms for pattern recognition. C. SpringerVerlag. Kwok and I. 13(2). LeCun. Tsang. B. pages 426–433. Wilson. editors. A fast iterative nearest point algorithm for support vector machine classiﬁer design. Westenberg. The preimage problem in kernel methods. R¨tsch.T.A. Advances in Neural Information Processing Systems 11. Denker. B. Berlin. Greedy algorithm for a training set reduction in the ac kernel methods.R.S.A. and G. D. Bhattacharya. Howard. USA. pages 408–415. In Y. [16] J. Sch¨lkpf. Fisher discriminant analysis with kernel. G. S. 2003. Platt. 1:541–551. IEEE Transactions on Neural Networks.W. Advances in pattern recognition. and D. B. s editors. Bartlett. Lin. Larsen. Computer Analysis of Images and Patterns. The EM Algorithm and Extensions. Kearns. Neural Computation. Kernel hebbian o algorithm for singlefram superresolution. O. Weston.W. Advances in Large Margin Classiﬁers (Neural Information Processing Series). Principal Component Analysis. Germany. J. and L. May 2004. S. Petkov and M. IEEE. London. August 2003. Franc and V. 2000.A.
D. 1994. 10:1299–1319. Kluwer Academic Publishers. Smola. Hlav´ˇ.I. Neural Computation. SpringerVerlag. Schlesinger and V. 2002. Cambridge. Scholkopf. and A. Sequential minimal optimizer: A fast algorithm for training support vector machines. 2:81–88. Sch¨lkopf. Mustererkennung 199820. Series B. editors. Knirsch. May.J. J. Schanz.. Ahler. [30] B. Fast approximation of support o vector kernel expansions.J. Advances in Kernel Methods – Support o Vector Learning. 1998. M. and F. Germany. pages 124–132. Vapnik. and C. Riply.J. The MIT Press. Technical Report MSRTR9814. and an interpretation of clustering as approximation in feature spaces. Learning with Kernels. Smola. Ten lectures on statistical and structural pattern ac recognition. Sch¨lkopf and A. [27] B. Neural networks and related methods for classiﬁcation (with discusion). [29] B. Nonlinear component analysis as a kernel eigenvalue problem. 2002. Royal Statistical Soc. Redmond. Berlin.R. A. P. Levi. 97 . Kibernetika. Microsoft Research. 1999. o [31] V. Muller. A connection between learning and selflearning in the pattern recognition (in Russian). Burges. Burges. Smola. Schlesinger. [25] M.C. DAGM. Platt. The nature of statistical learning theory. 1998. [26] M. 1968. [28] B. and K. MIT Press. Springer Verlag. R. Smola. [24] B. Sch¨lkopf. A.I. C. MA. 1995.[23] J. In P. 56:409–456. 1998.
This action might not be possible to undo. Are you sure you want to continue?
We've moved you to where you read on your other device.
Get the full title to continue reading from where you left off, or restart the preview.