© All Rights Reserved

3 views

© All Rights Reserved

- Chapter 1. Tensor Algebra
- Linear Algerbra.pdf
- Verslag Bachelor Stage Andres Definitief
- Remedy for Effective Cure of Diseases using Combined Neutrosophic Relational Maps
- mathgen-1859518523
- Canonical Correlation Analysis
- Jordan-canonical-form
- syllabus
- Algo Tracking
- Vectors & Geometry
- MoWalrand.pdf
- Vector Calculus+Shaw
- Maths Syalbus
- Assignment2 Ma 2014
- Linear Algebra
- Lecture 30
- lag11
- Upscsdfyllabdsffdsdus.in-uPSjhksfC ChkjhjjkhkSE Mains Mathhfgfematics Sylghfglabus
- 4310_HW5
- sec4_2.pdf

You are on page 1of 4

Laurenz Wiskott

first version 4 April 2006

last revision 12 May 2006

Contents

1 The

1.1

1.2

1.3

1.4

1.5

kernel trick

Nonlinear input-output functions . . . . . . . . . . . .

Inner product in feature space and the kernel function

Input-output function in terms of the kernel function .

Selecting the basis for the weight vector . . . . . . . .

Mercers theorem . . . . . . . . . . . . . . . . . . . . .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

1

1

2

3

3

3

1.1

If x denotes an input vector and k (x) with k = 1, ..., K is a set of fixed nonlinear scalar

functions in x, then

X

g(x) :=

wk k (x) = wT (x)

(1)

k

with real coefficients wk defines a nonlinear input-output function g(x). On the very right

I have used the more compact vector notation with w := (w1 , ..., wK )T and := (1 , ..., K )T .

Note, that not only x but also the k (x) can be regarded as vectors, which span a vector space

of all functions that can be expressed in the form of equation (1). This vector space is often

referred to as function space (D: Funktionenraum) and it has dimensionality K. Thus, is a

vector of vectors. For any concrete input vector x, (x) can be considered a simple vector

in a K-dimensional Euclidean vector space, which is often referred to as feature space (D:

Merkmalsraum). The process of going from x to (x) is referred to as a mapping (D: Abbildung)

from input space to feature space or as a nonlinear expansion (D: nichtlineare Erweiterung) (

Wiskott, 2006).

The dimensionality K of the feature space is usually much greater than the dimensionality of the input space. However, the manifold populated by expanded input vectors cannot

be higher-dimensional than the input-space. For instance, if the input space is two-dimensional and

feature space is three-dimensional, the two-dimensional manifold of input space gets embedded in

c 2006 Laurenz Wiskott

Currently at the Institute for Theoretical Biology at Humboldt-University Berlin and the Bernstein Center for

Computational Neuroscience Berlin, http://itb.biologie.hu-berlin.de/wiskott/.

the three-dimensional feature space. It can be folded and distorted, but you cannot make it truly

three-dimensional. Locally it will alway be two-dimensional (think of a towel that is rolled up).

For high-dimensional feature spaces, the mapping into and computations within feature

space can become computationally very expensive. However, some inner products in

particular feature spaces can be computed very efficiently without the need to actually

do the explicit mapping of the input vectors x into feature space.

1.2

and the feature space of polynomials

of degree two with the basis functions

1 := 1 ,

2 := 2 x1 , 3 := 2 x2 ,

4 := x21 , 5 := 2 x1 x2 , 6 := x22 .

(2)

(3)

(4)

Now, if we take the Euclidean inner product of the input vectors mapped into feature

space, we get

(x)T (

x) =

T

1, 2 x1 , 2 x2 , x21 , 2 x1 x2 , x22 1, 2 x

1 , 2 x

2 , x

21 , 2 x

1 x

2 , x

22

(5)

= 12 + 2x1 x

1 + 2x2 x

2 + x21 x

21 + 2x1 x2 x

1 x

2 + x22 x

22

2

(6)

= (1 + x1 x

1 + x2 x

2 )

(7)

(8)

= (1 + x x

) .

(Note that one would get the same result if one would drop the factors 2 in (3) and (4) but would

use a different inner product in feature space, namely z

z := z1 z1 +2z2 z2 +2z3 z3 +z4 z4 +2z5 z5 +z6 z6 .

Thus, there is some arbitrariness here in the way the nonlinear expansion and the inner product in

feature space are defined.)

Interestingly, this generalizes to arbitrarily high exponents, so that

(x)T (

x) = (1 + xT x

)d

(9)

is the inner product for a particular nonlinear expansion to polynomials of degree d. For d = 3 and

2 :=

1 := 1 ,

(10)

3 x1 , 3 := 3 x2 ,

4 :=

5 := 6 x1 x2 , 6 := 3 x22 ,

(11)

(x)T (

x) = (1 + xT x

)3 .

(14)

3 x21 ,

(12)

(13)

we get

Equation (9) offers a very efficient way of computing the inner product in feature

space, in particular for high degrees d. Because of the importance of the inner product between

two expanded input vectors we give it a special name and define the kernel function (D:

Kernfunktion)

k(x, x

) := (x)T (

x) .

(15)

Expressing everything in terms of inner products in feature space and using the kernel

function to efficiently compute these inner products is the kernel trick (D: Kern-Trick). It can

2

save a lot of computation and permits us to use arbitrarily high-dimensional feature spaces.

However, it requires that any computation in the feature space is formulated in terms of inner

products, including the nonlinear input-output function, as will be discussed in the next section.

The kernel function given above is just one example, there exist several others (M

uller et al.,

2001).

1.3

order to take advantage of the kernel trick, we have to rewrite (1). To do so we define w as a

linear combination of some expanded input vectors xj with j = 1, ..., J. We then get

X

w :=

j (xj )

(16)

j

g(x)

(1)

(16)

wT (x)

X

j (xj )T (x)

(17)

(18)

j

(15)

j k(xj , x) .

(19)

The number J of input vectors xj used to define w is usually taken to be much smaller

than the dimensionality K of the feature space, because otherwise the computational complexitiy of computing g(x) with the kernel trick (19) would be comparable to that of computing

it directly (1) and there would be no advantage in using the kernel trick. This, however, defines

a subspace of the feature space within which the optimization of g(x) takes place and one is back

again in not such a high-dimensional space. Thus, the kernel trick allows you to work in a potentially very high-dimensional space, but you have to choose a moderate-dimensional

subspace by selecting the basis for the weight vector before actually starting to optimize it. Selecting this basis can be an art in itself.

1.4

There are three possibilities to choose the basis for the weight vectors.

One simply takes all available data vectors x .

One selects a subset of the data vectors x with a good heuristics that predicts their usefulness.

One has a-priori ideas about what good basis vectors might be regardless of the data vectors.

1.5

Mercers theorem

It is interesting to note that since everything is done with the kernel function, it is not

even necessary to know the feature space and the inner product within it. It is only

required that the kernel function is such that a corresponding feature space and inner product

exists. There are criteria based on Mercers theorem that guarantee this existence, see (M

uller

et al., 2001). For instance, it is obvious that the kernel function must obey

k(x, x

) = k(

x, x)

(20)

k(x, x) 0

(21)

3

References

M

uller, K.-R., Mika, S., Ratsch, G., Tsuda, K., and Scholkopf, B. (2001). An introduction to

kernel-based learning algorithms. IEEE Trans. on Neural Networks, 12(2):181201.

Wiskott, L. (2006). Lecture notes on nonlinear expansion. http://itb.biologie.hu-berlin.de/

wiskott/Teaching/LectureNotes/NonlinearExpansion.pdf.

- Chapter 1. Tensor AlgebraUploaded byDiego Canales Aguilera
- Linear Algerbra.pdfUploaded byBob
- Verslag Bachelor Stage Andres DefinitiefUploaded byalamont
- Remedy for Effective Cure of Diseases using Combined Neutrosophic Relational MapsUploaded byAnonymous 0U9j6BLllB
- mathgen-1859518523Uploaded byVukhob
- Canonical Correlation AnalysisUploaded byBirat Sharma
- Jordan-canonical-formUploaded byVenkat Subramanian
- syllabusUploaded byakiscribd1
- Algo TrackingUploaded byNguyễn Trọng Linh
- Vectors & GeometryUploaded bysourav_cybermusic
- MoWalrand.pdfUploaded byfarnaz_2647334
- Vector Calculus+ShawUploaded byrichardfisica
- Maths SyalbusUploaded byrs81219952gmailcom
- Assignment2 Ma 2014Uploaded by96chandrahas
- Linear AlgebraUploaded bySoreanu Dumitru-Paul
- Lecture 30Uploaded bymatlab5903
- lag11Uploaded byKhmer Cham
- Upscsdfyllabdsffdsdus.in-uPSjhksfC ChkjhjjkhkSE Mains Mathhfgfematics SylghfglabusUploaded byShakthidhar Rao
- 4310_HW5Uploaded byGag Paf
- sec4_2.pdfUploaded bySceptic Granny
- Vaughn_Climenhaga Advanced_Linear_Algebra_I.pdfUploaded byIrving José
- Section a Introduction to Inner Product SpacesUploaded byAsk Bulls Bear
- Linear algebra part-5 (Eigen Values & Eigen Vectors).pdfUploaded bysakina ferdousi
- Assignment01-LinearAlgebraReviewUploaded bysohaib shujaat
- CH2.pdfUploaded bybihanjiang
- T6_VectorSpacesUploaded bySubham Mohapatra
- 10247Uploaded byMohit Kumar
- Assignment 1 StruUploaded byحاتم حاتم
- 1-s2.0-S0022030211001846-mainUploaded byPuspa Rini
- mastering linear agebraUploaded bytidak ada

- Pressure Drop & Flowrate-KarmanUploaded byAstri Ngent
- 68703279-Manhour-NSRP.xlsUploaded byvazzoleralex6884
- 101726935-Method-of-Heat-Exchanger-Sizing-Kern-Method.xlsxUploaded byvazzoleralex6884
- Header & Piping SizingUploaded bymedicbest
- NPSH Calculator For Process EngineerUploaded byTony
- TW Thermo-Kinetics.xlsUploaded byvazzoleralex6884
- Air Cooled Condenser Design SpreadsheetUploaded bySonal Gupta
- 4A104C Excel-template -Heat-exchanger-Design p1 Si UnitsUploaded byMustafa Alweshahi
- 414CC3_excel-template_prelim-shell-and-tube-heat-exchanger-design_si_units.xlsUploaded byvazzoleralex6884
- 48240888-ASME-B31-3-Calculator-V2Uploaded byep_alfi
- 20725830-Drain-a-Tank-Calculation.xlsUploaded byvazzoleralex6884
- Advanced_Excel_Formulas.xlsUploaded byLakiCamu
- A-AHPCorrect08.xlsUploaded byvazzoleralex6884
- 1252106057Bit HydraulicsTFA.xlsUploaded byvazzoleralex6884
- scrubber designUploaded bymayur1980
- Chemical Engg eBooks available at nominal chargesUploaded byinstrutech
- 23671371-Guide-to-Refinery-Process.xlsUploaded byvazzoleralex6884
- 2Phase ReliefUploaded byInggit Prillasari
- 7752137-Cost-Calculation.xlsUploaded byvazzoleralex6884
- SeparatorsUploaded byMascuud Baasha Caalim
- (EDITED-FINAL)Sludge - Storage CalculationUploaded byAmro Ahmad Ali
- 2 Phase DensityUploaded bydeyvishwarup
- _TOTAL CONDENSER ETHANOLDC_.xlsUploaded byvazzoleralex6884
- Sizing of separatorUploaded byHoney Tiwari
- data.xlsUploaded bycharro1623
- Vapour Liquid Equilibrium Data Collection.xlsUploaded byvazzoleralex6884
- TW ASME-ImpUploaded bycomienescu
- Velocity of SoundUploaded byWalter Rigamonti
- VolumesUploaded byintania66
- Steam Leak Calculation.xlsUploaded byvazzoleralex6884

- A New Foundation for PhysicsUploaded byDavid Thomson
- answersUploaded byapi-195130729
- Schrodinger’s Equation for the Hydrogen AtomUploaded byarafatsagar
- Equations of CirclesUploaded bymyanhaeyo
- PRM2Uploaded byada
- CHAPTER 1 Polar Coordinates and VectorUploaded byاحمد ظريف
- Amath 575 TalkUploaded bybeobeu
- Solutions 1Uploaded byGunawanSuhardjo
- Sparse MatUploaded byFabio Fernandez Garcia
- Planck EpochUploaded bybomansmith
- Lenz - Example of poincare anomaly.pdfUploaded byingjorgevera
- Parallel universeUploaded byNishant Madanagopal
- Gamma matricesUploaded by鄧新弘
- Convolution TableUploaded bybbutsch
- NPA Twin Paradox ReportUploaded byAnonymous 0U9j6BLllB
- 09+Particle+in+1-D+box.pdfUploaded bySigit Priatmoko
- Chapter 2: Mathematical Modelling of Translational Mechanical SystemUploaded byNoor Nadiah Mohd Azali
- QRG_ME.pdfUploaded byckvirtualize
- Physics 1p.pdfUploaded byRoshan Shetty
- QMath11 AbstractsUploaded byzeldaik
- Chemistry lectureUploaded byOuzamJatoi
- The Uncertainty PrincipleUploaded byruin_2832
- Makeenko Yu.M. Non-perturbative Methods in Gauge TheoryUploaded byanon_939189922
- Uniqueness of weak solutions for Lax Oleinik ForumlaUploaded byLori Hayes
- Lecture Slides for Chapter 6Uploaded bykstephens1990
- Solutions of Examp 00 Hall RichUploaded byRajesh Dhelia
- ME501 L1 Stress Oct 2018Uploaded byFiras Rockt
- Rotational DynamicsUploaded byAbqori Aula
- Quantum PhysicsUploaded byBilla Ajith
- Nondimensionalization of NS EquationUploaded byusama

## Much more than documents.

Discover everything Scribd has to offer, including books and audiobooks from major publishers.

Cancel anytime.