Part II

Support Vector Machine Algorithms

Outline
 Some

variants of SVM  Relevant algorithms  Usage of the algorithms

SVM Algorithms


Famous SVM Implementations



LibSVM
http://www.csie.ntu.edu.tw/~cjlin/libsvm/

SVMSVM-Light
http://svmlight.joachims.org/

General Data Mining tools that contain SVM



Spider toolbox (matlab)

http://www.kyb.tuebingen.mpg.de/bs/people/spider

WEKA toolbox (java)

http://www.cs.waikato.ac.nz/~ml/weka/index.html

Variants in LibSVM


We use libSVM as an example

    

C-SVC NuNu-SVC One class SVM EpsilonEpsilon-SVR NuNu-SVR

C-SVC

The dual is

nunu-SVC

The dual is

One class-SVM class-

The dual is

EpsilonEpsilon-SVR

The dual is

nunu-SVR

The dual is

Commands of libSVM
 SVMtoy

: an Illustration tool  SVMtrain: the training algorithm  SVMpredict: the test algorithm  SVMscale: an algorithm for scalling

Usage of svm-train svmUsage: svmtrain [options] training_set_file [model_file] options:  -s svm_type : set type of SVM (default 0) 0 -- C-SVC 1 -- nu-SVC nu2 -- one-class SVM one3 -- epsilon-SVR epsilon4 -- nu-SVR nu -t kernel_type : set type of kernel function (default 2) 0 -- linear: u'*v 1 -- polynomial: (gamma*u'*v + coef0)^degree 2 -- radial basis function: exp(-gamma*|u-v|^2) exp(-gamma*|u3 -- sigmoid: tanh(gamma*u'*v + coef0)


     

-d degree : set degree in kernel function (default 3) -g gamma : set gamma in kernel function (default 1/k) -r coef0 : set coef0 in kernel function (default 0) -c cost : set the parameter C of C-SVC, epsilonCepsilonSVR, and nu-SVR (default 1) nu-n nu : set the parameter nu of nu-SVC, onenuoneclass SVM, and nu-SVR (default 0.5) nu-p epsilon : set the epsilon in loss function of epsilonepsilon-SVR (default 0.1)

     

-m cachesize : set cache memory size in MB (default 100) -e epsilon : set tolerance of termination criterion (default 0.001) -h shrinking: whether to use the shrinking heuristics, 0 or 1 (default 1) -b probability_estimates: whether to train a SVC or SVR model for probability estimates, 0 or 1 (default 0) -wi weight: set the parameter C of class i to weight*C, for C-SVC (default 1) -v n: n-fold cross validation mode n-

Example of svm-train(cont.) svmsvmtrain -s 0 -c 1000 -t 2 -g 0.5 -e 0.00001 data_file  Train a classifier with RBF kernel exp(-0.5|u-v|^2) and exp(-0.5|ustopping tolerance 0.00001  svmtrain -s 3 -p 0.1 -t 0 -c 10 data_file  Solve SVM regression with linear kernel u'v and C=10, and epsilon = 0.1 in the loss function.  svmtrain -s 0 -c 10 -w1 1 -w-1 5 data_file  Train a classifier with penalty 10 for class 1 and penalty 50 for class -1.  svmtrain -s 0 -c 500 -g 0.1 -v 5 data_file  Do five-fold cross validation for the classifier usingthe fiveparameters C = 500 and gamma = 0.1> svm-train -s 0 svmb 1 data_file


Usage of SVM-predict SVM Usage:



svmpredict [options] test_file model_file output_file

options: -b probability_estimates: whether to predict probability estimates, 0 or 1 (default 0); oneone-class SVM not supported yet svmtrain -c 100 -t 0 .\data\breast_cancer.train .\data\ svmpredict .\data\breast_cancer.test .\data\br .\data\ .\data\ east_cancer.train.model breast_cancer.res

 Examples
 

Usage of SVM-scale SVM Usage:

svmscale [-l lower] [-u upper] [-y [[[y_lower y_upper] [-s save_filename] [-r [[restore_filename] filename (default: lower = -1, upper = 1, no y scaling)  svmscale -l 0 -u 1 -s range datasetx\ datasetx\svmguide3> svmguide3.scale  svmscale -r range datasetx\svmguide3.t> datasetx\ svmguide3t.scale

Usage of SVM-light SVM Commands:

 

svm_learn svm_classify

Usage of SVM-light SVM  

svm_learn [options] example_file model_file Available options are: Learning options:

 

 

-z {c,r,p} - select between classification (c), regression (r), and preference ranking (p) (default classification) -c float - C: trade-off between training error and margin (default trade[avg. x*x]^-1) x*x]^-w [0..] - epsilon width of tube for regression (default 0.1) -j float - Cost: cost-factor, by which training errors on positive costexamples outweight errors on negative examples (default 1) -b [0,1] - use biased hyperplane (i.e. x*w+b0) instead of unbiased hyperplane (i.e. x*w0) (default 1) -i [0,1] - remove inconsistent training examples and retrain (default 0)

Usage of SVM-light SVM

Transduction options


-p [0..1] - fraction of unlabeled examples to be classified into the positive class (default is the ratio of positive and negative examples in the training data) -t int - type of kernel function: 0: linear (default) 1: polynomial (s a*b+c)^d 2: radial basis function exp(-gamma ||a-b||^2) 3: exp(||asigmoid tanh(s a*b + c) 4: user defined kernel from kernel.h -d int - parameter d in polynomial kernel -g float - parameter gamma in rbf kernel -s float - parameter s in sigmoid/poly kernel -r float - parameter c in sigmoid/poly kernel -u string - parameter of user defined kernel

Kernel options:


    

Example of SVM-Light SVM SVM-Light SVM-

considers the sparsity of data  Especially useful for Text Classification



svm_learn c 100 .\data\train.dat .\data\model.dat .\data\ .\data\ svm_classify .\data\test.dat .\data\model.dat .\ .\data\ .\data\ .\ data\ data\res.dat

Illustration of SVM in Spider

% Clear all variables/definitions

>> clear classes

% Generate 100 points from a toy data object and lets have a look to it.

>> d=gen(toy2d('uneven_gauss','l=200')); >> plot(d);

Illustration of SVM in Spider



% Create a C-SVM object with RBF kernel (sigma = 1) and C=Inf C-

>> s=svm({kernel('rbf',1)}); % We now train the SVM and obtain a trained object and the training result >> [r,a]=train(s,d); % We evaluate the training error and visualize the trained object >> loss(r) >> plot(a)