Uai-5

Basis unction Nethsnks

ladial
ANN may be

. The 3ubez Vae d Naintna

in Jeveral ways. Ca
arphoached tenepthons
multi Jayer
Recollkc Hha or eunsive fechnigue
apaton oa
as
Hhe
e viewed
appnorma hov) hOn-

C 9tochastie o
elasi }ng
he aoblen
se dve a hybnid
3 Hee, ing
pattenns y
donea nly sehanable
involves toskages:
mannen n0n-
gven e
amg
hht stage Eans into a
ne e

separable pa btetns o nned

ineanly he thans)
the cane
on whid sepanable hg
becor
aHennsinka d lneaely
heenem is used).
CCovens
Bhe
schabion by
stage com blebely (LS algenithm)
The 2 eshimatio9-

44ares
Jtast
elassilkcalionn
This hybntd apneach t battesn
4 hadial - hass unehion
deses:bed by usinga
CRBF) ntnenk
9tauceune COn3isks
3 ayens
The RoF
5. oure o de s)

3nbu 1ay en ceh sighing

Hidden unit (abpiies a n on- linean tsanseh.
the n 4 ace he
mtion hom
hid den 4ta
 The o Xayex sinea, designe uPP
to he a chvahon
the es ponge o he neBw onk
o the inpu layea
atte nn apphed
d in an
tna îne
6 NoTE The otbue laye
stage o Me
unsupenYibed
manna 9 n

O Ceduae.
hybard
transfoanmation
Satisses
hon 1m2a7
This pOcess o
two condirons
the Gaussian

t7ase d ôh
aAe
the one m (5)
REF netoqK
SViewed asa Keanel.

untcon
yattenns
On epanaDili

Cheotem

Covetis
Cast in.
Hatemnt :

hen broblem,
ifseahion
complex patBenn clas
n on neaafhy,
thgh dimen siona than tn
anable
to be
n eanly hat he
m oA Xely
ime nSion
a pace,
O
dens ay oula ted
pace ho
When we

Kecan hat classifreaien

setanable enn
have 2ineanl)
tt don a where 4ath

naunal) divides an inpu space ino negiong

Le X denote a ee of N hateny (yechn)

ach hich is assigned to
, N
and
clases X,
one o we
ble i a un fate
This diho komy 5 ossi oinb
the
the 4anates
in the amily
eas Pnose in cass
in the class X, nom
veckok
a
dene
fox eaeh arten z EX,
valued n e ien
se o heal -
ma de up
shown
,2, * m

51)
1 a weeton n
tha the a}tenn
9uppok
inengiona inut ace
Uun
m,-
in the
then mats he Porn
The veciog¢7)
inu Sp0Le in to
ns in the m- dimens ional

Conesbon ding un new shace

imensfon
 to (x) as a idden un cfion
We nele
as an im hok
Since
in a eed fon vand NN
hidden unit
o a
the et f
The 4cepanned by 19lomed
hidden enckons ()}

t the
xidaid
to be
A diahoomy , Xaf ETSB a
m -dimen
thesAE

shorabl
Vecto' tch ha
x E %2)

delned the yaion

The hyperplane

NTO(*)0
in the
Aa ¥nq sulate
descnibes
-space CJeatune 4pace)
nage ohy hypeplane
invexse im
The
' a )=
o
:

4wnface Chyeplare
deine the eharahng
the nu ace.
detisin bendan))
3 ceWi dex natual cless ofnopbings eblained
-ie ProdutlG
2 inean Con bînatron o
by n9
Coondrnates
aten ye clon
the
o ronden
narona yo ria ties

hen in O0
ig
Oa1,2

comeen n t 0
Whene K Bhe is equal to

vecronX and N a
inbut to expegs
he en
niyin t den

eus om).
on ogen o
enaies
pnoduc o}
An ah otder m ohomia
c a l e d

i,ig dimensionali ky
in p u t p a ce o
an
tox

Bh ene ane (aclea

mo
useA)

m onpmjals
n

Moncmial 7 poly nomial whre has en

NOTE
ghe fenm
Monomial t;
9
4 tramples ehe ye o ehanating unjaces
desenived in Ane hypeaplanes C 1 de
tde) hypen sfhees
fahional vatie tes) quad ices ( d

(a

seano ble diehotsmis

3 xamples o
5pon i dimenss
dsebof
4ehanable die hotomy

phenicaly
alnically
16 n ropability, Ehe epanation. aa s o
patens deponds
and
Ehe dichotomy cr0 sen
he n u spACE
pareans
he distnbu tion
bhe essle
dicho to mies o
Su hose ot a
gni pno bable
a.
X 2fat hav n e
robebily)
m

Lat P(N, m,) ( =

2 ( ) fon Nzm-
mo
YoP
C5)
Cempestn N-
binomiad Coeff cienB
integeriand
whe re he all
defned fon
ard hem selves

() CA-m)m
5),
nenpretaton o
Fon a graphical se?eing N:Am
Cmaliz khe 4gnation. b m,)
nobabililyP CA
the
and
loting va lues m'
ve1
he values into
tes cale
Nole: Nonmal a ion
&ange l o I.
 8
I. The above said o neVe od top Chanattenisties:

ahneshold fec & Ound

A =2.

Ehe valua P(2m ) l2 fn each

value of m
Ven

1 (5.5) atas the esseh ce o

atte nns
Cumalabive

othe Jac
that he
stament
isa
to the nobatility
connestondng will hesul in
distnibu}ion
inomial

jis
a
Jaia oin
thal (N-1)
theads.
0a jewea
Cm-
make the
dinmensibn o he
we

The highen PCN, mD

.
he clo
he n oba bility
hrdden bpae,

b e to unity.
basi ingnedieng
Covens
heoNem has
. hidden funeion
johmu lation of the
)hon-inean
*hee x
he input Vecton
defined b C2),
and i =
, A, * * * "'t
fthe
hdden (fea kune)
a mensicnality
cb high wil
theinu spae ,

corm aned
pace dimenstonal:y of the hidden
the
whene Va lue as51ghed
deleamintd the
i
space no.
hidden unitsD
he
to m Ce;
o Ex i The XoR noblem

J the XoR noblem, Hhene ane heoing ( þatten ns9

inu sþate.
C1, (1), (o,0) and (1,), in a 2D

(, The eunpatenns
of he XOB

Co,0)
noblem

O'D

0%

Decisio Maing Piag nam

ttenn elassiian
4isement i to Const nuct to
he na
o n hesbonse
noduceso)the
bnaay outpet
Bhal on C0,0) an
atlenn C'1)
the input he
i 9h4sbon se
eutu
( he binany Co)
ut atean (0,1) 0
in
lesest in the
o n ls tha ae
. hus, the

o he tamming dstance
nput ate,
in
an my
thal ane naxi mally abant.
ma to
 Aquen ce
s

2 The Hammn d:slane ana

4e 0

defind s he h0: of changes Pnom

Vie Veh a
Iand 00
Mamming dshance
both
The isdan a
whereas
he Hammín
one .

ol and lO ts
oth Junehions as
Gaussian hidden
ais o
Tefinea
-t,1), t =L)
exp -11x
C)
C ) = a x - - ) , t=[ ,
tunctoh
scond ticdd en
Fun chion PeCx)
Finst tid den
nu
patten O1353

0-36T3

367 8
CO) 0.1353

0 3618

C1,0 specifreation t h e tidden

Tatle: OR proteom a
the
funcionsson

examle1

in dim enAionalily
n thi omple,
he use
honlineaaly exem i}*d
noposed Hnte
i uffrient to
an hid don n cdon
o Gaussi
 aoblen tnto a2
tnansfonm the XOR
22 (Contd)
one
ineanly sepanabe
capaBy o a Sunlare
83 3chana bng eguence a
1 , Ka, N
Let
han dom pattenns (vecto ni). the
deine d
as
o m
Vatable
Sand
N be his seauence s .
Le that
tn
begen
uch
0 e ed ona.
langest has m, deghees
whene
sehanable , that the
deduce
We can

Then N= h 1S

Facbabilty (n+ln)
? C,m) -P
n 0,L3

delni non
neal he
this
innpre negative binomial disa:butron.

tha
he novatpility

This
distnibution
quals

th
uctess
a
Jon
Phece d e
the
Denhoui tnails.
Dequance 0
epealel
denoe hoba tbilrdas
e and
an d Jaibune
uLess
 2
pinemial dis tnibukion
The -R

efnad
CegpuenobebE) and
pi'no disarn
-Ve
K+n = "
the

e dseaso
(7E7,
n:

9a-
(* K)= nesult deicaibed
the
can
nobe distaibuion_
binomel
NOW the -e
ithana
15 Fus
unit
he hight
bym,
hn ed
- mebens
, and Haing ime a
to the

nnes ponds
tosses
Co Stqkence
N
Thus
Ene m, h Parlune vania ble N
Coin
. andom
van Coees5
a ain

espectaion

The edion ane

C:)
and
EN
m
And median N
vens theoMemn
A Conlaa
P.
-olows'
be slated
 The xpecled maxfmumho nandomy
ne
ane nea nly
assigned pattenns (vattyn9), thal 1
in aace o men.sionali
ty m
sehanable
equal o 2m
Paob km
The In enpolation
5 eh coun tened he

n CoveAs teolre the

fhe dimenson

toncept p incaeasin aten clasifn

hpnlneanly anable
taoblen o
UGe
hon-nean mappn
al50
We m4u deul honlinea3 teg
invol ves nea
Eransonn an ea3ies one

Pnoblem into

nenin with n
NN
Pd eiad an aep ut
Conden and
hiddan layen
Aingle
inpu layen,
ingle uhit
aen ch oen imfatian.
A Smgle o a nonlinean
idestgned t tnonm
The newonk the hiddn
Fhe input pae
rnapng nom inea mpp ng nom
ollowed
lo inu
hrdden
 Gimenso a the
denote e
mp
npue spae
Hne netwonk $tepaesents
a ma em
then, Ip oultput
md:m. inul- spae to e
Ehe
spae,
na n0:5)

mab as
he
think o
m T C T t .

tyerafate gha) ama)

al plok
- d i m e n sion
1 a fhe
T
Sface
The
the
outu
unKnowhn and

input
Soface T
he dm in att d
Genenely
cDn

the taning aata

usa

wih nahe he
s i tuBes
Con adune -Hn he
TAi a rebing Pno
data poinis in
opimzaHoN
n
Jmdace besed

s yhonymous
Cneerali7akier hase
debween the A cta pons
wth inten polation
 NOTE
nserton Semening didun
Intrpda tion:
( a - , )
( d
Le-) hawie

inear in tes polaron val

independent VU at
t h e -Jnet
on
v a l a s

ont uA LOL
valuts

be
roblam may
The in tenpo la-tion

taer s

oN d tom
Aven Ae and

R" 1 ..N
a | i = 19"
N eal hes d; e
(07e-

har
nd a unetin
Condilion
tnler bolakion
4aiss-he
F (x) diy 1,2, N

510)
15
Con tnain ed
inmg daka
pas
7ontg
haongh al)
 The hadial bas nction KEF
5 choosing a n o n tha
Consiss o equrvalee hëm

h&s the
N
ED
F) 49(l *-ail)
Plr- )r-,,N
whe e
o f N b:bran nction:
e
1
Ha
don otee

Eecrdean
usLually
Gra tuKen
N

be he entes
RFs
We geFa
.

Inse bng .mear ejnationS

multaneous
e g h i o e f r i e n h s )

tuhnown

W2

WN N

52)
 .
25 Ccontd.)

wheae 9( , ij =

Y 1)
desined

d,da,- dwJ ehonse)

Hsbonse
d T vec
. WN] ltnear weight
a Yecten

trainingamble
the
N with
matnix
N--N
Lt dan ole an

N
(5.14)

matnix.

ca llad inrenola-tion
matnix
This onm
Compact
he
5-12 n
Kewite C519)
and here me
h on 3Inqulan
A 1sumin9 that Can
solve
Ne
ho matni exs
weght ne os W
(545) fon the

W 4x (516)
matnix
thal the intenpolation
theonam has to
n on ingulan,
Miccheli
is
be dstussed.
 Micche lis Thecn m: In his the onem,

pn oved
he folowing peint
N be a
Let
oinls
intenpolation
matax d4,
the N-by-N
Then
hes t alamet

on Smgulan.
dang class of
Hdhelis theoaem Cove ns

KF
spme
. Multiquadries 2
)
=
( +C*) and nek
C 70

2 nye1se Mult.quadhig:

(n 4) Z 5-)

Uincion:
3aussian
3ome
) J a
0 and

neR (5-19)
 1

the in57 o e
hpn ingulan, the Poin sall e
dhn(dknd)
5:18 ane
n vEn
mul tuad nics o
Note tha a
tommon

519 shane
aausian aneions
eapenly ocalized Juneiors
in he

they ane
as 7
770
unge tha Pe) Pefitive definete
hese ase
Sn both of ane
a51+
quadnies
In tonfnask,
the mulki unbeun e5 7
be
ded
ce Ca) g e n valMes
nn cal n -
N-
end hs
RDE Netonks:
we
51o to 51,
eguatons
thno ugh
going in the onm
RDF netw onk
a
envision
cOn
illustnated
stauz tune, as
aayened

99 Layes: m
Thts con sisls
ndut layen: aimensiong
hene mo The
oune node»,
vecton x.
the inut
- liay
 80

autput
ehfoy y=F
WN

output lae
Hdden
Layen ge N) f4yeene
byen m)
nelwonk,
tased
KUF
an
snucune
ig intepolabon
theny
Hhe ame

Ths con i's

2 Lajen: h
Hidden denCDmputathon uniG
AS the zethe

no- of Smathe.
ea ch uni
toainingdanple, (N);
KBF
-matital descnibe a
= @Ch )
defnes the
The h intut doka oint
the ROF andx 1 the inpu
Centott o
ecton hom Soune nodes totidde
 5 hu leyen 3 Cons'sb oa ige
Ccmutaional unit
nesnjc tion on he
there ho
Nce tha
outut tayen
o a a u s s i a n

on the uge
Cormputa
-

whtoh each
in
as
he KEF, ayen the netwonk
un chion bhe hidden
uni n
-tiona
defined b
C P(X-)
- l ) , (Fa)

J1,2N

C the
measuNe bhe
5-20 raussidn-funcion with
Hidth bhe t
conden hidden uni ane
the Gnaus Sin
all
ene naly width
o
Comm on
a
a5si ened thot dis inguishes o he
an ame tn
Then the
0m Gnothen 15 the Cenka
hidden un:
 drscussed ma
labion no cedune
32 he inteno taning RBF
not e a good Eatgy o to new

tcause p
ne wonK
Meason:

data on Hhe o laoing m he taa ining

data tontz
hen he ho o h-
RM, *
e

is a n g e , thAn
ample
ases miseadng rantakons hesy
also nc no
.Ths leads lo

nieve
an
to het
tnained
NN
a a
atvalent t6
leanning
Design
outp ten
in tu
mapbtng)
dímensronal
(mulfi
hyen unface,
oinpet as a bnoblenn of data
ee * f
leaanmg givena
Hlente, hor
heconstnue
hypensunfae
*e panse.
may
inexse o
th at
sarà to
oin
hela ted
paoblems of tht
TwO
enmulatron
tach
each o l k e n f t h e
he othe.
Know]edge
ntiala Jull
heaihes shdied
as becn
he nob ]ems than he o the-
ne dea:l

nhasi
neater
anen
C a l l e d
dinecpno blem

nay
be wel-osed
36 APnoblem
-sed
 X and
34 AsS hme ha? t havehe- doma inf
alakd by &
ma pong
ahange Y

The netonsBnueken of maping

we-oed EX,
eboh
Exrsitnce: Ten eveny in?we
(a)
An
thene does xtstE

h e ne npu
ec
ions
toa any atn
Unae ness 6)J) Yf Y=t
, e X , e
have
Cordnuous
( ) c o n înuity Happing
#hene
extsd S = XE)
Any 70 hal
o 45 imtl
uch
Bat R whene
drstane beee

two gumen

Y
Vom atn A ange
tlustnation the maping inpaut)
 he 3 Condit ions
taR navieus
3 3 any o)
il-osid
the thobtm
(s nt 5absjred, data
ange
Fnoblen
il-posed infa a ehrHon.

Yfically
NOTES« ith small
amunT

2ataa
gehenal tha in
31 Examples o
0 n a r

dangna, well- Fon2a

pieiunes,
n4 edue
peech
data ak
3iessm s
an

noblem opkm
ing
C o n s i m u c i o n n

w
he-pllo
hynu)Ace
o
de
but hobem
invese

d
1- pos«d >e
i o l a Be
maynot txisE
CHitenten
not
eaypns may
The exiskencd i s t r nct wu outuh

C inpt in-f fn the

much
mh

hot
Thene may
( whcenjai nify
incneaks
tain hoe
7hesene
in put-
o u t math
the

at ion theoy
Regulani .
egulanizzation
is
4laement
here
A genanal in dhdes to solve

addna nfo

ll-esed roblen aermt ovodog

an
 the olkion 2
Regulanizaon &tab1lizs
hon-negative runCHn
an
au Xi lan
y means
d Soluh o
he
Ehat embeds Ine abo
inu-0Lt puB daka
Let the eo (taain"a,
he desnibed y i N

inpu 4ga
desintd nespone 50)
assumed
o
e
1
is
the output
Nobe that

apnoi 6n itted
Let F( YE ton
w
has

Weghi
nem F).
in vove s

KegdanitonTheo oy
5iKhone rs This tem
em;
dandEnon
(a) ton the (andaad enen

eagu1es

CF)
dest ned nespo
nse
eeen the
Cskon.ce.) whee i=,, N.

and he
actual en se
522)
-Fl]"
 This n tenm
c Regulan 3ng__ depereds on
tie gee mene
dereid by ,CF)
FO)

CF) = 1D C3.2
dierential chenatop
Dis
a hinea aigs
whene ablzn ince i
otlem,
negulanization

thesohuhion
to the ron-in
Rnd
Sm ooth
fmpe secl on
haKing

-fin
weh DFE) belongs
.
ace to
-h negularig ation

be minimigid
t
The
uantity
A.
theory
C)

L(621)
)isa
ve u a l hamben
whee negulasi5akion "Pirami
Tkhoneunchenal
4. e Can view he egulanigaion. pta.meb

as an inditatea o the buffieiency he gwnl

Bhat ai}y the ol tion
daka set a; xample,

he thoblem tu; no tont na ma.

f o,
an be deienminednconstndined)

non amples
D itely
iYA>
on e ohen rand,
toeciy
the 4o/ ton
eno gh
amodel
hepnegen.5
egualani 4ing G inluence
h he

cmplexty-ehayunedion
heguanizalfon
nal sote-ton s Contne led ty a ameter)

the1Eonoyun
edional:
héche Difenenbial
4 no te
Bion
The tniple ofn gulani za
tatd as n i m38s The
hal
Hhe unction )
tn defmed
Tkhonov unc ion CF)

,CF + C
whene ,CF)i the standand ennon
denm, , CF) 5 the egu laniging en m

and is the negu lani 3aton a a ma ten.

4 T mini m CF, e netd t Vning
kahule n t valnation ef
bhe difthental o

nay
The thé dt diffezentiol e a funcbional
defimed as &CF+Ph (55)
ds(E.4)
sred funedon ovecton *.
whene h a a )
(h)
must t e z no
Hon &
Cond: :

Y
526)
Fhchet
dstforenka
Civen a line as
A Eule- Lagmange Egiuaton: Can -d a
We
danenkal enaton P,
deknmint d ddjoin enato denoted
uniguely pain n chions
uch ha Jon any
C)and (X)which ase ufrei ently dfnen-
gnd
which ab»}y popen be undasy
biable
we cah Wite
conditim ,
DvCA) åx = Va) Duo) dr
x JIR

5.
.
 guaon 530 called Geen iden
St movides a math basts en deknng the
to
adjant eraten pkys a nole gm:lan
malaix atn pen
that a

heneraly,
u6) = DF()
DY)= ph)

-90
h oF) dx
d (r.1)
Ch, ODF 53)

hee Dis he adjei D.

T ranspoe.
R ine a r taans fematon (s)

caleulus
0me
Afeu
at

33

.93 is the Elen. Lagnnge quabiono

he Tikhonoy uncional (F).
 J0.
51 Guns Tunchion: C593) nepnesents a
in the aproima
antal dforential eqyatitn
unt 'on .
-thg
is knohn to
Theolution e
thrs euation
Consst he integmal aang jonmaBion othe

RHs o he Qabone

GC3 ) denobe a une ren

L
x and abhcan
on
4 ual
which both rectby
uneposes
eking,bu Joa d:ffenen
aname te
angtument)

actual vadues
oenaten L, we

inean dijfenental
a iven x , ) aksfes the
To
ipule ta hat

Conditiens
-jnchon
e, G) s a
F a
peunelany
he hescni bed
x
and 4atisfes
Condihons
the devaives
at he o inx,
Ecet ane al coh Hnuous
o G La, ) w-n deexmined by
deaivatives
the ho o teraton L
the 0n den o the
 unctien2
3 ith (x,) Censdene d as

, iJaisfes he hantial drcrka

e akon

Sangulest
53

ts Hhe deila

n chen
peskonetat t .
GCx,E) calk e
This thchon
dlforuhal
aRen unetm

a hole on a mea
The aneen's funchion ays
52 iman to th fon
that
dfenential eena ten
matnia equa.tion).
matni x (Jn
he inven e

dnote a cpnbnueus fun ttin

3Le )
the unc tien
Then

F (%,E) PCE)dz
L536
53 (conhd.)
solution fhe dfononl inl equation
(534)D
LF)=P)
e Gnen's nedron fin
whene CE)
opnaBsa L
the encan dijehen ial
solve he ulen -

Lagn ange eguaien 4

T

DD
L
&nd

LC5)
Using 4,
Naite

}-FoJ]
) S(¥-*9d¢
uama-Bion
hae
and
inbegaalien
The
nde o

ean inlenchanged
Zd-FXI6 (,xi)(541)

the
The mhi miz°"9 lul ion F(
a inca superpo sitio
hegulanidation eblew
N Cneen's unc h ens.
S4 etntd.)
the tentens of ha
he X; values netnasent
*gansion and

«hnusent Hhe
the eighs d- F:)/)
Crefcen ethe
the egulzakien nt iem
The 4luon Ene
N-dimensiona
u-shae t
es in an

Expansier Ccoien
55 ckminaiým
i 1 t o N (42)
Le d ; -Fo)]
eAitkn
5 -jellows:
Can e
5.41
(43)

at J, we
tvaluatng 5*

1,2, N
CX) =2 iG (%,)

New den tins ate h On intoduced

 G,M) G(,a).. G(8,ZN)
Gax) GCa)

, x)547)
n) G ( 4 ) .

(4s)
, Va
5-42 and y4
he wnite
Thun,
ma
(d-5)

5-41 a n 5 0 ,
F wen
Elma ig
G+AI) = 5 5)
dentily
Hhe N-by- N
whet
matl

Noe

Keca hal dueta?

Lis elf .adjeint
epenaten)
K aneen's un chien (,X) i» a

mme nic unctio

GM ) G(, y) Aall
an
53)
ens maBnx
tui len va

ymmt H mal*

(555)
ensuAne that
te
u1ently Jange
NCTE
cheese
rdoe n vent ble.
and
ve
definik
+I 1s
54)
C+I)"a
biin bhe keght
used
ton be desind hespense
specifed
ve tom valute t
and an anpaima
d
Veien AAmet
neauiai9 tion

Jeluien
the hegalanigabien
.
finaly, the
exan s i e n

Phobiem r

w:6 (x X) (5)
1

(x, xi) arcen's Juncion

entd..)
553 Safes tha
s tquivalenk
The Egularization atnca ch
Selktion
in
ems 4
the angien othe
aeens une don
a At he
sd in
Gheens funcHo ns
The no of the ho 4
examp
eual to
xhan Son 0es
nainin9
used the

hegulaaizAtion Newovds:
athnoximating
he egula nized
The ans.en 1neen's
innm
50 ( 5 5)
untion
G(YA)
cendened
a Xi y e
fanction shown
elow fer
une
a netwonk
At nuct

menta Ien
im ple

Aid den lajee

. N Gnecns aye
ayenn
unciohy
i: Kegud onization Nzornk
53 Centd-)

Laye
te the dmnsien mene
cken
cem s honmaas
i dden laye dinechy
Layen 2 hat anz
eenneitd

unilg nades
t a n pu
eens -uncHios

u u (ayen cons n
Layea ngle
trn7
unit

hnk
thpt e
L.neaaity 7 Th
a Meay w:gtec
h:darn un
he

x fenced

Can

Th's yte
et tnk
acttm m tdete
1calaaiza1on he *tnk

The -

Phe
Smes ha

ts+E defrhte

5th
irteelant
 Hidden l&y1 1a9
: RBF Nehosa

Clesinable
nekwenk
has thace
gulanizA}ion
This
aprximatea
nekuenK
a univesA
ny u i i vaniate
The hen foiwetly in ahyy
WonK
Con

E,
ontinutu
Suncion it hs es
oluion
9inse ths

xina tian
a ppme
He gula ai a tion
Cemput by
The Aphho
mal.
nelyenk

yabion uneona)
he anim
neguulm ihS he deriaticn
oplmality

takes

d)
6 henendged Kadal-Bass yndon o aaES
lhe tnaining otu
COne spon dence etuecn
The
unction
G(K,K) Je*
and Bae Gneens hsthosk
data
a heg:ulantzaton
N 0aues Value
i 1,8, tn N
odehending
Pe exensive
me he
thet
the templexity of-
this dificulh
v e ncorne
10
i duced
heyonk
a to 7
a uboplina

invols
cch
ng o an

Thi's apaae Aimensio

nelAce

oluhienin
a owA
ap*aeximâtke phhion ustng
ap?*0xtma
hon d one
Galenkin's
method.

a b e v e a i d

h e
ds
The
techni 4ue
,landand the
a t n o x i m a t i o n

method anik
basis
his
to on
Aun ng exoanded
hufon FCX)s

-(564)
Fo9

hene assemetd
to e
unc4ins
e asis
independent
in ealy

TyA:call .
b a s s u h e h S

i ho of
 Nith con sidenation

G( x-bil1) ,i-l,2
(5-68)
oCnest|i-*. i
whene the e
to e detenmined.

ay nefne
56+

F*C
GC% t)
FC)

the deen tminakion

to be adnesed is
The pmoblem
oweight wli= Lay*mi
fhe ne ncional&(F*)
defrned by
Cost
mm
G-)+
ECF)

(5 )
on the
RH an be exphessed
The s term
2

a the quaned
Eucrdean noi - Gl)
whete d h ,da, -.. dn7 (5+D
 G&CX1,) G,ta) --. G(, tm) A

= ** b). (53)
that d isN dmensithal as tefone.
Note
have dtfvan
bu he malntk G and
CN-by ) "m-by-i1
dimtn Sons

Dstng
DP (DE, DF*)t

/5.44)
m

m
Cy.t)

Whene in the e cond and thin d lines

weus the defnition df an od joint. opnatr

(535)
 IDFI*

The
m a i X
defined:

GnCba, bs)
GCta,t)
GEm on
the

min
im 3ation nesuë

the Ehe
Thus
edon
w
elads
56
Werght

anomete" aphaoac hes

Regulantzo. hron in hom)
m n mum-
vense L e a s t - s u anes dala-

Lhe seuelo tn

wtonve
dekezmined

Pheoyern
to
holukon
- i t n aoles
G d ' , 2 a0
endo fn vengE o6
whene 5+8)
( G'G) G
GT

Weghed Norm
vec ton
indindual elemet ohe n þul-
t apa opr1ate
to i}fea ent classe 5,
elon rlehal we ighted noa
m "the

to on5 de
hid deined
4uad en o
Cx) xce
GA Cronté)) Ih e cx 5

the fnpu
the
dImension 0
m
i
matnix,
and

yecton x ro Phe
ximaton

Ehe arni
e Waite
can
we
Nou,
duhon 6 a
Hegulanizad

intenpnetec)
may be
Weighted h ohm
The use oa
in wo a)s he
t s a n s o n m a t i o n t o

Appyin an afne
6pace.
npue äplae
Oniginal dimens/ona
the mo
s e d K i f f e n e n r a l

enenalZina

Bhe deiinikion o
tauation in
Cntenec at
operaton
RF CN K-t;
5.A Ciaussian matax C may R
noa m Heg htng
itr
ti and
exessed -tc e [x-t7
Gx- bile x

the invenpe maa1x2 s defmed

hene
 Ciaussian
te pte seng a muleivaaia te
C91) and covanian te
disnibuion th mean vbu C,

manix Z

Kecepti've rd
66
the heceptre
ivec cen te ti
tom a.nesen as, he
t; )
de rned
eld of G
non
) 6C x- t;lle)
con sYânt
ome ositive
hene
a is
b/ll x-6;lle)
necephve rel which
The
r
fnpu vechm
a b se ake a4RValwes , t a t e
G tlx -lc)
hed evel a
hon
aove ard delmibion,
Fnom the 7ke Covantane
0
Acenanios e7 tajning the shape,
identy n-luene
oh

On 0
ht ecephive eld
0 i en ta ti
 t e identiky matnik aho
hene
a common V*TAnte
O m ) whene Pe
A2 dag (T
h elemen of the.in pu
Vaaiane o the

and 2,.o
Vec tn x
matai,
By delnion,
npn -S
dagon
32
2isa+ dafnile matax

5-84)
=

iagonal maaix and

whene hotatron
malf*.

6 t h o h o nmg
n he
anc ize of
'shape
A dekenmtnes es

e ceptive eld
de fexnenes 8ienta.tion

(Kevesthd)
63 XoK Paobley O R xclusfve oR
mobem,
Considen aga
mtzae ence pon
joyedrievio usy sing.
hideenlayen
wth a nge
to the ame
a
utien
duce
Can o
REE t wonR
nole ing
ungipms olned a
ep -, 1,2

ane
where the centens t, and t

sumptions: i
shaamg wh'e
The oul-at ni uges i e I the noblem
h e ymmetny c
u3ed se ae a ing le
0tdah ni
e nly
only
Haight s to aetenmegd data-
a, bias b (e
indudes
unit
The oupu
2 independen} vaatabtey
valuts XOR
omtput
desihed
The
ave nan-
funcr on

Fixed in bu = +1

a)

ine
avssiah hngh
Nd uncion . Nehocake
naYS h:le
 Tre inu- 0ueput elg-ion new an is
ddme 2

the àoba table ge

To

whene desse ubpu

Cora
Table: 4ntul-outbub Tnansmaton

Combutedo XoR e

Jnbut

(0.o)

Le d a(-).j-t2a4

99)

4Aers
3
 5 84

0.13 59
0-36s 0 3618
5.1)
o.135936#8
0-364 0 3648

d 01] (5-1)
(593)
Since
gkae maa
ho han

Noe bha l G mote

4ta puin

1ova
detoamined

meters
dig.aly,we
ke the
hrs
546
ovenconme

mini mum noam

3oluhon.

-CGG)a"d
matx wth
a aRe
9 no
nv
.e n s e e9

in 13), twe
1/ T

D72
-

1:3509|
A12-12504
9392 12504
127 2 509 4202
14202 0 202
09202

L(5)
4 in

ling 541 an
i n a l l , u b s t ilu

- 2.5o12
- 2 5013

+29o
the KeFnetwoak
which (ompletes S t e i i ea o n

áme
ten ()
he hegula aizaiion
Stma ion o Qescn: d
nobemy
hean
eghession
honi
Cohsidex
a
dutput a
obseayable

model wrose in Vtckoh

by 0n
nestponse
in
Lime tebi

i de med i= 2 , N
Cx)+E
5.45
m o oth Cunve aund
hene (X) a

whi be
ample dnawn -nom
meen ma Y1iAn ta
10ie ees
 -7

K:i597)

ond cFhoi

necons tuct
( ) vey
The noble )j-
samb
the tarning ecl estiate o
e e
Le value
2
ome

0 T viby
iey the mih'mis
nabiom)
sion
aegre
(ho-linge
un ctien & ) E C - F e ] + 2 o

Ae to
to isstes
,
uts.ble
vale on
chose a
To

be Consrdched
the3ien measbnea

.Rouges
SDFOON measkned
dals
3nfidelity ' he

L-Fo)

vnage Sguaned Enten

RCA denoe he Ave nag quaned
Let
between wo uncioes
71 (Conkd..)
Ehe nogteeston une lion jC)
heppno xi'na-ingunctitn )
4)
I) -5
R() =

Vaue o thak i hi m S
m um 1
R) expnesee as
a lneas combi
CXx) be as:
Le obsena bies
Hhe grven
-nabio n o

ma t n ix
(5.101)
ACA y
Where
= ),H) )

Ldue,-

/5 102)
A)
aND RNN.

matai AA) Called

The N-g-
matY
he in e hce
contd)
We may now ReWaite5.49 as

5103)
-400
whene the N-by- veci6

3Jte).Jo

nhe
e W1ire 595
how
He can
-7 (5:10)

We oblain
5104 in 5103

15y

AP) THOD
NCT-AC) 4* e -

N
T ( 1 5)
AC) el1*
N
(NN)
idan) ty matn
o ) hote
exrected yalue
To ind the

th a
17 tenm on he HS u5105 5
Cunst ant
unatteced by xeelaloh.
naten.
 Contd
the ha te nm 5 310
he exoucta +ion e

he
The expechahon

E e a TU)aca)el
E 4Ceh leTA Tea) AC)E]}
tn E
FLeTa) AYE
o eigenva bes
106)
7um of emen
gum ragonal janae s
alas
a
The
ace of +arniple
waes

hs
he calan ise use
t mat
NOE algehna dales thet
exfst, then
Matnix ns
Compaple
Qman sfo
and
tnale o C
ho tnae o
Be an
=AC) NCA)
t n TaTeA
AC) eeT]}
t n ATC) AC))
 5

that he trae ofA A) AC

finallyneBing
the a AO),
4am s

(5103)

(5-0)
ERC e xhecte
vae ef

RC on
e e9ua bions de
Sumanan 7ng
sequlah paamee
dea to choese
The basit
bhe av aned eFUNon
tmnimye
RA).
data hel
the
over
dcnee tly nce
Ths cant be a ceomh shed nesion n hon
the (uh khown) eq
KA) invol ves

m oc be t, 5
ende,
cont&)
hoise 2nante is kro wn,
the as the
K0) (510)
use
the simare
abo minim3
roie of A it
optimun

NOTE I-A))*).
-

not Knew,
cmoss val daki
a
a n o t h e r
nchion(genenalzd

pny
expeched
thenefone ong eno ug
an
hesults
Gr0o & dalaset
& Vvaslable
the nose
when hom
tgnal
d'stinguish
to
KO
hoperies
4Apngxi.ma kion
cohe
mation
x í
LUniwexgal bie
0-)
be
An fn
te ana
5 on in eut
uch hal
unc bon
houn ded

aund
 denore the RBF ne bwoaks"'ami
Le nepnesented

Cong istmg o F

F) 2 (
an E TK

hee
thehem
a p A O x ' m a t o n

unvensal

Ehe d as:
s t a he
Then can
e
ntth ks
RBF
mafpin
- n-
output
with a
con
?inuous n o - o a } *

o any a n Re
heu Common wicl th
on ) an
enc u nc h o n Fx)naag

the
vch hat the
)
70
ne}wnk cloe to
RDF
Phe

n, P elLeo
noE
The Kenne
ANOTE the
lo sais.
hequined

a t

(han Pu qu aid)
glno nge
 Cuse o Vimnsipnali ty
class pbaoxima ting
The rempealy oj a
hnatib
"es
in
incneases exhonentially
untio ns
he e
n u
dimen ionaity
mo 97m0 t k n e mdex

Ymea sun
Sotoness n.dex Honunetian

NOTE on an a p x i ma
mo
nsna.n tels us hat
alily uged,
bell mans Cu odimensonna
vimation
techni techniue

dhe
appro
H4spectve cenetan, he ponentie

: 1s iaCy
ccuraey
nche5
ncheoses

Fa attam
Cnput domensionaliy).
naedied
a5
inCHeR

cUnse
1 scoate thi
uth he no
omplexl
un chon- i t c o m y e n s e l e s
jn ncHe ase in

in tbieiven
in
P 58,
is ilbs tbated
int o nhron
Thi he Congs tang
m m a. LP And
The ta be a ised by
sae har have

RBF netwnks
 und ionsA
nd Corhs nend74
wth the am nate o Con eng ence
hiaden shae
uhene m, ize of
O/Jm),

prox aon

Punct m spce
a ) Mulhlayeh
pordoprers
L2
to)2;fMY*)
i the
the whne PC)
whene F) 5 i g m e t d aetivat f o n

muldimensional

founien tnansfonn
fanertien
tinuncmon
ma
ame
Nehons
he
RF
LCR
Sebole spat of
fon hofe
F 2 exPC
hde
dentvati ves
ae integnabe

af the
the ne
o deniva }ives
n KB () n
nase H.h
75 is hgd to
apbnoxt ma. ttng
-funceio n ate o onwrgerve
he
tn den

ndebtndenl a me
ppmo i m a ting-hnetirng
The ac o 7dBr
nhwoKs
Fh ML
attainable s es
Asingly eonf hained
rmo
 hewinks ana ML:
Bf
ane exam p
Pmon ks and ML
yicey, FOF k eaona n d nesmks-

hon-negt 1yenea
unive s6l apbaxrmateng
Both ane

laen
Difteenes has a mgle
ng
trdden
hrd
RBF
ROfnebuoak
etMOn have
hhe aas
MLY
comme

Cnputakional odes
loyey
neuzonal medel hrdeen
eKS
in thase
nedas bposewth
B m u t a t e n

The arenen
ith
ane
dffinent
- i n c a h
and o p e
inDuA layen Ko
in
layen
3 . H d d e n

taye mear
cupayan
e al
nd
hidden
the
MLP.
S uni
hidden
h on-mea1 toch
ancbon oduc? oi
The o c v a tion o m b u r e s h e
nne
recn
inon M LD gra ic wegh
rd den uni
and
vechh
niukr ah
ativation
Juhe'n of Irhean rm

The K
halJ 0 Conlen
RGF hd
ir
ar

ink ve E09
ekren
ru)
3
na nt
 60
Conl nuel lobal a x mtians
5PL
lineanin puF- ou Fu
ma Ppin
onon
arexima Fion s
local
FBT tn Gtnuek
ma peings
ncnngen in put-oultou
G1auE'n etions
15 on-p -
Ke nnel e4ne5 the Cong tona
NCTrE do
esimate
techniue
- m n e

vataie
nandom yeween
axrechation
o nenlmearhela
tion
a

objaeive:
Ton
Yeniabks
and y.
handom e
a ain el sed

neqaessfon) n RO
nel hidden
layer
Ron ofhe
E n a n s e u n c o n

newcks

(in ReF hetwoak

Leanning Saalegies aeciated
wth the
h a
tneon wehls vove
netwonk
The
/a) the the
outhut unit) compaze
hen
hrdden
ime alk he
dfere nt
bion unctrohs
activa
m O n . m e 6 r

non-me1)
uni etion
vo/ve s(owly-
n's a
c va lon -jun
s a
ds htin se we e,
tne ul la yen
 The dt layer an K ne twmk

eTonm fenenta 0 an 2epaore

outhu lafe ns
helden an d
the orln3ation
4/n o Feclna,ues
K f ne-c1Dk
used
earnmg
ttis Kandum
aF
cenlenselec te d
FXed whcd e i n e
a
Assume RES
ed
te hièden unig
the actiaton Junelov andpm

ana
h0sen
Centexs
he
The to cakion o
he tnatning aa ta so
om

Te tom
o sihan
ngulaa - ye te ëNis
n,
ma the e
M
hea N-by-
3 is a
Otbnagonal ot?ices

U,ti
U

2, V,**

uch hat
uGV diag ( ,, )>
K min (M, N).
whene
0
7 7 : Og
he column yectoRs of D ane called let ginguar
the cotumn vecton V ane
veeton oGand
its night rngu lan Vec to x
cel-oganiaed
sukct-on f tenters
of
sg that consists
is aybnid len.ngece
to dlferent tag1s To ehma
sage
Se-oganitd lsarning centers of
KGF cbon
loutions fon
apepaiate
kyen
in a
idden

Cmlets the
leaantng 4ge the linen
Suerrviad esbimattng
the hehonk
design oj layer.
tre output
weig o The centexs of
enter
Suazvioed ele csen as
ce a
4ubervined
leanningp

Ehe k b
undengo Ar
used

ctR ea7rn.in
dune (a gen era-
Fea
a
ghadien desent
haough algo:thn)
the L MS
-lizadien. e
This
w i E h C l a n g a k i o h :

lepolatror thefolew
d strie
invelves
the cotmbimgd ue e
mebrod

ingedi eng
ma tn*

nut
neam
-

weg-hg
iagonal
ice in-lerpelatien
Kegulanizad
anams tr and
Selectien e a negelaniza ken
int stak etor