Welcome to Scribd. Sign in or start your free trial to enjoy unlimited e-books, audiobooks & documents.Find out more
Download
Standard view
Full view
of .
Look up keyword
Like this
0Activity
0 of .
Results for:
No results containing your search query
P. 1
Makalah

Makalah

Ratings: (0)|Views: 5|Likes:
Published by Black Xeros
Berisi tetntang pengetahuan
Berisi tetntang pengetahuan

More info:

Categories:Types, School Work
Published by: Black Xeros on Feb 14, 2013
Copyright:Attribution Non-commercial

Availability:

Read on Scribd mobile: iPhone, iPad and Android.
download as DOC, PDF, TXT or read online from Scribd
See more
See less

02/14/2013

pdf

text

original

 
Least Squares Support Vector Machines Classifiers :a multi two-spiral benchmark problem
Lukas, J.A.K Suykens, J. Vandewalle 
Dept. of Electrical Engineering ESAT-SISTAKatholieke Universiteit LeuvenKasteelpark Arenberg 10, B-3001 Leuven, Belgium.email : lukas.lukas@esat.kuleuven.ac.be
keywords : Support Vector Machines, least squares SVM, pattern recognition, classifier.
Abstract
Classification problems have arisen in many applications, attracting many researches to develop advanced classifier techniques.A method called Support Vector Machines (SVM) for pattern recognition and function estimation has been introduced by Vapnik (1995) in the framework of statistical learning theory. Since then there is a growing interest on this kernel method for itsinteresting features. In this paper, a least squares version (LS-SVM) is explained. LS-SVM expresses the training in terms of solving a set of linear equations instead of quadratic programming as for the standard SVM case. Iterative training algorithm for LS-SVM based on a conjugate gradient method is then applied. The LS-SVM is then able to solve large scale classification problems which is illustrated on a multi two-spiral benchmark problem.
INTRODUCTION
Support Vector Machines (SVM) for solving pattern recognition and nonlinear function estimation problems have been introduced in [7]. The idea of SVM is mapping the training data nonlinearly into a higher-dimensional featurespace, then construct a separating hyperplane with maximum margin there. This yields a nonlinear decision boundary in input space. By the use of a kernel function, either polynomial, splines, radial basis function (RBF) or multilayer perceptron, it is possible to compute the separating hyperplane without explicitly carrying out the mapinto the feature space. While classical Neural Networks techniques suffer from the existence of many local minima,SVM solutions are obtained from quadratic programming problems possessing a global solution.Recently, least squares (LS) versions of SVM have been investigated for classification [5] and function estimation[6]. In these LS-SVM formulations one computes the solution by solving a linear system instead of quadratic programming. This is due to the use of equality instead of inequality constraints in the problem formulation. In [1, 4]such linear systems have been called Karush-Kuhn-Tucker (KKT) systems and their numerical stability has beeninvestigated. This linear system can be efficiently solved by iterative methods such as conjugate gradient [2], andenables solving large scale classification problems. As an example we show the excellent performance on a multitwo-spiral benchmark problem, which is known to be a difficult test case for neural network classifiers [3].
LEAST SQUARES SUPPORT VECTOR MACHINES
Given a training set of N data points
 N 
x y
1
},{
=
, where
n
R x
is the
-th input pattern and
 R y
is the
-thoutput pattern, the classifier can be constructed using the support vector method in the form
+=
=
 N 
b x x K  y sign x y
1
),()(
α 
where
α 
are called support values and
b
is a constant. The
( )
,
 K 
is the kernel, which can be either 
( )
x x x x K 
=
,
(linear SVM);
( )
x x x x K 
)1(,
+=
(polynomial SVM of degree d);
 
( )
]tanh[,
θ κ 
+=
x x x x K 
(multilayer perceptron SVM), or 
( )
}/exp{,
222
σ 
x x x x K 
=
(RBFSVM), where
κ  
θ 
, and
σ 
are constants.For instance, the problem of classifying two classes is defined as
111)(1)(
=+=+++
 y yif  if  b xwb xw
ϕ ϕ 
This can also be written as
 N b xw y
,...,1,1])([
=+
ϕ 
where
( )
ϕ 
is a nonlinear function mapping of the input space to a higher dimensional space. LS-SVM classifiers\
=
+=
 N  LS ebw
ewwebw J 
122121,,
),,(min
γ  
subjects to the equality constraints
 N eb xw y
,...,1,1])([
==+
ϕ 
The Lagrangian is defined as
{ }
=
++=
 N  LS 
eb xw y J ebw L
1
1])([);,,(
ϕ α α 
with Lagrange multipliers
 R
α 
(called support values).The conditions for optimality are given by
=++= ======
==
01])([0 000)(0
11
 Le L N b L N w L
eb xw y e y x yw
ϕ γ  α α ϕ α 
α 
for 
 N 
,...,1
=
. After elimination of 
w
and
e
one obtains the solution
=+
v
b I  ZZ 
100
1
α γ  
with
];...;[],1;...;1[1],;...;[],)(;...;)([
1111
 N v N  N   N 
eee y y y x y x Z 
====
ϕ ϕ 
and
];...;[
1
 N 
α α α 
=
.Mercer’s condition is applied to the matrix
 ZZ 
=
with
),()()(
kl 
 x x K  y y  x x y y
==
ϕ ϕ 
The kernel parameters, i.e.
σ
for RBF kernel, can be optimally chosen by optimizing an upper bound on the VCdimension. The support values
α
are proportional to the errors at the data points in the LS-SVM case, while in thestandard SVM case many support values are typically equal to zero. When solving large linear systems, it becomesneeded to apply iterative methods [2].
 
BENCHMARK : MULTI TWO-SPIRAL PROBLEM
One of the well-known benchmark problems for assessing the quality of neural networks classifiers is two-spiral problem [3]. In [5] the excellent training and generalization performance of LS-SVM with RBF kernel on this problem has been shown. In the following a more complicated multi two-spiral classification problem is depicted inFigure 1. Given are 432 training data which consist of the two classes as indicated by '*' and 'o'. The LS-SVMclassifier using RBF kernel with
σ
= 1 and
γ 
= 50. The resulting classifier with support values
α
and bias term bobtained from the large scale algorithm is shown on Figure 1. Taking 432 support values one has nomisclassification on the training set, together with excellent generalization as is clear from the decision boundary between the black and white regions. The support value spectrum is depicted in the Figure 2, here the obtainedsupport values are sorted from largest to smallest.
FIGURE 1
. Multi two-spiral classification problem with 432 training data (class 1 and class 2 are indicated by ‘*’ and ‘o’). From black and white regions which determine the decision boundary between two classes is clearly shown the excellent generalization performance of the LS-SVM with RBF kernel.
FIGURE 2.
The spectrum of support values related to the classification problem in Figure 1. The support values
α
are sortedfrom the largest to the smallest value for the given training data set.

You're Reading a Free Preview

Download
scribd
/*********** DO NOT ALTER ANYTHING BELOW THIS LINE ! ************/ var s_code=s.t();if(s_code)document.write(s_code)//-->