© All Rights Reserved

0 views

© All Rights Reserved

- Machine Learning
- Hasil Uji Normalitas
- MTH302(1)Wth Sol
- 10.1.1.104
- Handouts Shwartz
- Deep Residual Learning for Image Recognition (Summary)
- svm
- Development of Base Lining Methdologies in Singapore
- Introduction to Artificial Intelligence
- Format_ThesisProposal_Version_05_March2012.pdf
- Dimensionality Estimation, Manifold Learning and Function
- QT Spring 2012 Solved
- artificial-intelligence-for-executives-109066.pdf
- Neural Networks_vs_chaid Tree Ctp4 (2)
- Cluster Class
- QM Questions
- AI Report ICT Revised
- Facial Emotion Ranking Under Imbalanced Conditions
- Effectiveness Evaluation of Rule Based Classifiers for the Classification of Iris Data Set
- An Approach for Ids by Combining Svm and Ant Colony Algorithm

You are on page 1of 27

INTRODUCTION

TO

MACHINE

LEARNING

3RD EDITION

ETHEM ALPAYDIN

© The MIT Press, 2014

alpaydin@boun.edu.tr

http://www.cmpe.boun.edu.tr/~ethem/i2ml3e

CHAPTER 13:

KERNEL MACHINES

Kernel Machines

3

first

Define the discriminant in terms of support vectors

The use of kernel functions, application-specific

measures of similarity

No need to represent instances as vectors

Convex optimization problems with a unique solution

Optimal Separating Hyperplane

4

if C1

X x , r t where r

t

t t t 1 x

1 if x t

C2

find w and w0 such that

w T xt w0 1 for r t 1

w T xt w0 1 for r t 1

which can be rewritten as

r t w T xt w0 1

Margin

5

on either side

Distance of x to the hyperplane is w x w0

T t

w

r t w T xt w0

We require , t

w

1 2

2

Margin

6

min w subject to r t w T xt w 0 1, t

1 2

2

Lp w t r t w T xt w 0 1

N

1 2

2 t 1

w r w x w 0 t

N N

1 2 t t T t

2 t 1 t 1

Lp N

0 w t r t xt

w t 1

Lp N

0 t r t 0

w 0 t 1

7

Ld w w w T t r t xt w0 t r t t

1 T

2 t t t

w w t

1 T

2 t

r r x x t

1 t s t s t T s

2 t s t

subject to t r t 0 and t 0, t

t

Most αt are 0 and only a small number have αt >0; they are

the support vectors

8

Soft Margin Hyperplane

9

r t wT x t w0 1 t

Soft error

t

t

New primal is

1

2

2

Lp w C t t t t r t wT x t w0 1 t t t t

10

Hinge Loss

11

0 if y t r t 1

Lhinge(y , r )

t t

1 y t t

r otherwise

n-SVM

12

1 1

min w - n t

2

2 N t

subject to

r t w T xt w 0 t , t 0, 0

Ld r r x x

1 N t s t s t T s

2 t 1 s

subject to

1

t r

t t

0 ,0 t

,

N t

t

n

Kernel Trick

13

z = φ(x) g(z)=wTz

g(x)=wT φ(x)

The SVM solution

w t r t z t t r t φxt

t t

T t t

φx

t T

gx t r t K xt , x

t

Vectorial Kernels

14

Polynomials of degree q:

K x , x x x 1

t T t q

K x, y xT y 1

2

x1y1 x 2 y 2 12

1 2 x1y1 2 x 2 y 2 2 x1 x 2 y1y 2 x12 y12 x 22 y 22

x 1, 2 x1 , 2 x 2 , 2 x1 x 2 , x , x 2

1

2 T

2

Vectorial Kernels

15

Radial-basis functions:

xt x 2

K xt , x exp

2s 2

Defining kernels

16

Kernel “engineering”

Defining good measures of similarity

String kernels, graph kernels, image kernels, ...

Empirical kernel map: Define a set of templates mi

and score function s(x,mi)

(xt)=[s(xt,m1), s(xt,m2),..., s(xt,mM)]

and

K(x,xt)= (x)T (xt)

Multiple Kernel Learning

17

K x, y K1 x, y K 2 x, y

K x, y K x, y

1 2

m

K x, y i K i x, y

i 1

t s r t r s i K i xt , x s

1

Ld t

t 2 t s i

t i

t i

Multiclass Kernel Machines

18

1-vs-all

Pairwise separation

Error-Correcting Output Codes (section 17.5)

Single multiclass optimization

1 K

min w i C it

2

2 i 1 i t

subject to

w zt T xt w zt 0 w i T xt wi 0 2 it , i z t , it 0

SVM for Regression

19

f(x)=wTx+w0

Use the є-sensitive error function

if r t f xt

e r , f x t

t t 0

r f x t

otherwis e

min w C t t

1 2

2

t

r t w T x w0 t

w x w r

T

0

t

t

t , t 0

20

Kernel Regression

21

Kernel Machines for Ranking

22

but at least +1 unit margin.

Linear case:

1

min w i C it

2

2 t

subject to

w T xu w T xv 1 t , t : r u r v , it 0

One-Class Kernel Machines

23

min R 2 C t

t

subject to

x t a R 2 t , t 0

Ld x x r r x x

N

t t T s t s t s t T s

t t 1 s

subject to

0 t C , t 1

t

24

Large Margin Nearest Neighbor

25

D(xi, xj)=(xi-xj)TM(xi-xj)

For three instances i, j, and l, where i and j are of

the same class and l different, we require

D(xi, xl) > D(xi, xj)+1

and if this is not satisfied, we have a slack for the

difference and we learn M to minimize the sum of

such slacks over all i,j,l triples (j and l being one of k

neighbors of i, over all i)

Learning a Distance Measure

26

similar approach where M=LTL and learns L

Kernel Dimensionality Reduction

27

PCA on the

kernel matrix

(equal to

canonical PCA

with a linear

kernel)

Kernel LDA, CCA

- Machine LearningUploaded byAsim Arunava Sahoo
- Hasil Uji NormalitasUploaded bybasyev
- MTH302(1)Wth SolUploaded byshiny_star51
- 10.1.1.104Uploaded byMayo Finero
- Handouts ShwartzUploaded byDuy Nguyen
- Deep Residual Learning for Image Recognition (Summary)Uploaded byTomoki Tsuchida
- svmUploaded byAska Laveeska
- Development of Base Lining Methdologies in SingaporeUploaded byArief Ihsan
- Introduction to Artificial IntelligenceUploaded byJayant Chaudhari
- Format_ThesisProposal_Version_05_March2012.pdfUploaded byNadeem Anjum
- Dimensionality Estimation, Manifold Learning and FunctionUploaded byscribd202
- QT Spring 2012 SolvedUploaded byDee J Khan
- artificial-intelligence-for-executives-109066.pdfUploaded byMario Vladović
- Neural Networks_vs_chaid Tree Ctp4 (2)Uploaded byÁlvaro González Balaguer
- Cluster ClassUploaded byJitendra K Jha
- QM QuestionsUploaded byPriyanshu Kumar
- AI Report ICT RevisedUploaded byRhian Dennise Galanida Barcelona
- Facial Emotion Ranking Under Imbalanced ConditionsUploaded byeditor3854
- Effectiveness Evaluation of Rule Based Classifiers for the Classification of Iris Data SetUploaded byBONFRING
- An Approach for Ids by Combining Svm and Ant Colony AlgorithmUploaded byesatjournals
- ass2Uploaded byAditya Kumar
- Applications of Machine Learning-Mohammad JouhariUploaded byKaleab Tekle
- Syllabus Harvard Machine Learning AdvancedUploaded byjusticeUSA
- 5 S2 Bidang Keahlian JCM.pdfUploaded byMatahari Bhakti 'dida' Nendya
- kate ode lab 1 standard deviation practice - sheet1Uploaded byapi-440679457
- Deep Learning and Its ApplicationsUploaded byAman Agarwal
- immmmmmmmmmmmmmmmmmmmmmmsvqsvuqsctqcsq.pdfUploaded byRabia Almamalook
- MSc Thesis Nordin SahlaUploaded byPrashant Pawar
- L11 - Pattern Recognition PrinciplesUploaded byfivehours5
- sun2016.pdfUploaded bylubeck abraham huaman ponce

- Introduction to Artificial IntelligenceUploaded byLoges Waran
- CTS,CLS,CLR.pdfUploaded byLoges Waran
- MICAI2015_EFIM_High_Utility_Itemset_Mining.pdfUploaded byLoges Waran
- DATA vivaUploaded byLoges Waran
- WK4 - BitStuffingUploaded byLoges Waran
- Class and ObjectUploaded byLoges Waran
- New Text DocumentUploaded byLoges Waran
- Inter M Board E210882Uploaded byFrankmorel
- Tata Infotech netUploaded bygeethikachoudhary

- Unscented KF Using Agumeted State in the Presence of Additive NoiseUploaded byJang-Seong Park
- 1 DSP FundamentalsUploaded byshankar
- Fuzzy LogicUploaded byBobb Ketter
- IES - Electronics Engineering - Control System.pdfUploaded byRod S Pangantihon Jr.
- Contents Artificial IntelligenceUploaded byJulio Anthony Leonard
- 3183X_bibUploaded byCalin Campean
- Assignment IIR n FIRUploaded byWan Mohd Nazmin
- C05 Neural Networks and Deep LearningUploaded byangelvi
- non_movingUploaded bymsdraj
- chincymUploaded byAnonymous JIHJTWw4Th
- logic and distributed.pdfUploaded byAbhijith Sreekumar
- TEST YOUR COMMUNICATION KNOWLEDGE QUIZ.docxUploaded byKaren Taylor
- DCS - PLC comparación ABBUploaded byHenry Gómez Urquizo
- LOBSINGER - Cybernetic Theory and the Architecture of Performance - Cedric Price's Fun PalaceUploaded byKostas Mpaliotis
- Investigating DesignUploaded byMauricio Gomes de Barros
- 657_Lect_1 Interpersonal Commuincation skillUploaded byMegha Sharma
- chap1Uploaded bychandrakanth
- Daeroadmission@Gmail.comUploaded bysilverbyte
- Extended Essay AbstractUploaded byManav Shah
- 15.04.501_dpUploaded byRizal Haerul Akbar
- Design and Implementation of Fuzzy Logic Controller in PID Using LabVIEWUploaded bypriyam saikia
- Eye Gaze Human Computer Interface PDFUploaded byWolf
- business communication and skills for interviewUploaded byAnkit Gautam
- NonlinearSystems.pdfUploaded byWy Teay
- KNN.pptxUploaded byreverseengineer
- natural approachUploaded byapi-298520436
- Communication Skills ModuleUploaded byDeepen Sharma
- sistemas hybridosUploaded byFernando Burga Bustamante
- Language is Best Acquired Not LearnedUploaded byAlexandra Lexa
- CHAPTER 2 Sistem KendaliUploaded byRicky