UNI

T-I
IIMachi
neLear
ningTechni
ques

I
NSTANCE-
BASEDLEARNI
NG:
k-Near
estNei
ghborLear
ning.

DECISIONTREELEARNING-Decisi
ontr
eelearni
ngalgor
it
hm,I
nduct
ivebias,I
nducti
veinfer
encewit
h
deci
siontr
ees,
Entr
opyandi
nformati
ontheory
, I
nfor
mati
ongai
n,I
D-3Algorit
hm,IssuesinDecisi
ont
ree
l
earni
ng.

SUPPORTVECTORMACHI NE:
Intr
oducti
on,
Typesofsupportvect
orker
nel–(Li
nearker
nel,
polynomi
al
ker
nel
,andGaussi
anker
nel
,Hyper
plane–(Deci
sionsur
face)
,Proper
ti
esofSVM,andIssuesinSVM.

BAYESIANLEARNING-Bayestheor
em,Conceptl
ear
ning,
Bay
esOpt
imal
Classi
fi
er,
Naï
veBay
escl
assi
fi
er,
Bayesi
anbel
iefnet
wor
ks,EM Algor
it
hm.

CLUSTERI NGANDITSTYPES: k-
meanscl ust
eri
ng,Hier
archicalCl
ust
ering,
par t
it
ioni
ngclust
eri
ng,Tr
aini
ng
andEvaluati
onofamodel,Lossfunct
ions,Eval
uat
ion,ConfusionMatr
ix,Datasetspl
itandCrossv
ali
dati
on,
Under
fit
tingandOver
fi
tt
ing,Feat
ureEngineer
ing.

I
NSTANCE-
BASEDLEARNI
NG:

(
alsoknownasmemor
y-basedl
ear
ningorl
azy
-lear
ning)

I
nmachinelearni
ng,
inst
ance-basedlear
ning(someti
mescal l
edmemor y-basedlear
ning)i
safamil
yof
l
earni
ngalgori
thmsthat
, i
nsteadofperf
ormingexpli
citgeneral
izat
ion,
compar enewpr obl
em i
nst
ances
wit
hinst
ancesseenintraini
ng,whichhavebeenstoredinmemor y
.

 Theyar
esomet
imesr
efer
redt
oasl
azyl
ear
ningmet
hodsbecauset
heydel
aypr
ocessi
ng
unt
ilanewi
nst
ancemustbecl
assi
fi
ed.

 I
tdoesn’
tat
temptt
obui
l
dt sa-
hemodel pri
oriatal
l
,butonl
yquer
iest
het
rai
ningdat
aon
demanddur
ingscor
ingf
oreachspeci
fi
ccase.

 Thenear
estnei
ghbor
sofani
nst
ancear
edef
inedi
nter
msofEucl
i
deandi
stance.

 Nomodeli
slear
ned.

 Thest
oredt
rai
ningi
nst
ancest
hemsel
vesr
epr
esentt
heknowl
edge

 Tr
aini
ngi
nst
ancesar
esear
chedf
ori
nst
ancet
hatmostcl
osel
yresembl
esnewi
nst
ance

 Exampl
esofi
nst
ance-
basedl
ear
ningal
gor
it
hmsar
ethek-
near
estnei
ghbor
sal
gor
it
hm,
ker
nel
machi
nesandRBFnet
wor
ks.

Whatar
etheadv
ant
agesanddi
sadv
ant
agesofi
nst
ance-
basedl
ear
ning?
Answer
:

Adv
ant
agesofi
nst
ance-
basedl
ear
ning:

 I
thastheabi
l
ityt
oadapttoprev
iousl
yunseendat
a,whi
chmeanst
hatonecanst
oreanew
i
nst
anceordroptheol
dinst
ance.

Di
sadv
ant
agesofi
nst
ance-
basedl
ear
ning:

 Cl
assi
fi
cat
ioncost
sar
ehi
gh.

 Lar
geamountofmemoryr
equi
redtostoret
hedat
a,andeachquer
yinv
olv
esst
art
ingt
he
i
denti
fi
cat
ionofal
ocal
modelf
rom scr
atch.

Di
ff
erencebet
weenmodel
-basedandI
nst
ance-
basedLear
ning

Usual
/Convent
ionalMachi
neLear
ning/
Model
-
I
nst
anceBasedLear
ning
basedLear
ning
Pr epar et hedat aformodel tr
aining.No
Prepar ethedataformodel trai
ning differ enceher e
Trainmodel f
rom tr
ainingdat atoest i
mat emodel Donott rai nmodel .Pat terndiscoverypostponed
paramet ersi.
e.di
scov erpatterns unt i
l scor ingquer yrecei ved
Storet hemodel i
nsuitablef orm Ther ei snomodel tost ore
Nogener alizati
onbef orescor ing.Onlygeneral
ize
Gener ali
zetherul
esi nform ofmodel ,evenbefor
e foreachscor i
ngi nstancei ndividuall
yasand
scoringinstanceisseen whenseen
Pr edictf orunseenscor inginstanceusing
Predictforunseenscor inginstanceusi ngmodel t rainingdat adirectly
Input / t
rainingdat amustbekeptsi nceeach
Cant hrowawayi nput/trai
ningdat aaftermodel quer yusespar torf ullsetoft raini
ng
tr
aining obser vations
Requi r
esaknownmodel f
or m Maynothav eexpl i
citmodel form
Stor ingt rainingdat agener all
yr equir
esmor e
Stori
ngmodel sgenerallyrequir
esl essstorage stor age
Scoringf ornewinstanceisgener al
lyfast Stor ingf ornewi nstancemaybesl ow

_
___
___
___
___
___
___
___
___
___
___
___
___
___
___
___
___
_

KNear
estNei
ghbor
s(KNN)Al
gor
it
hms:

TheK-nearestnei
ghbors(KNN)algor
it
hm i
sat y
peofsuper
visedmachi
nelear
ningal
gori
thms.KNNi s
ext
remelyeasytoimplementi
nit
smostbasicf
orm,andy
etperfor
msquit
ecomplexcl
assi
fi
cati
ontasks.

Theintuit
ionbehindtheKNN al gori
thm isoneofthesimplestofallthesupervisedmachi nelearning
al
gorit
hms.I tsi
mplycalculat
est hedistanceofanew dat
apoi nttoallothertr
ainingdatapoints.The
di
stancecanbeofanyt y
pee. gEuclideanorManhat
tanetc.ItthenselectstheK-nearestdatapoi nts,
whereKcanbeanyi nteger.Fi
nallyi
tassignsthedat
apointtotheclasst owhichthemaj ori
tyoftheK
dat
apoi nt
sbelong.
Exampl
e1:

Yourtaskistocl
assif
yanewdat apoi
ntwi t
h' X'
int
o" Bl
ue"classor"Red"class.Thecoordinat
eval
uesof
thedatapointar
ex=45andy =50.Supposet hevalueofKi s3.TheKNNal gor
it
hm start
sbycalcul
ating
thedi
stanceofpointXfr
om al
lthepoint
s.Itt
henf i
ndst he3nearestpoint
swi thl
eastdist
ancetopointX.
Thisi
sshowni nthefi
gur
ebelow.Thethreenearestpointshavebeenencircl
ed.

Thef i
nalstepoftheKNNal gori
thm i
st oassi
gnnew poi
nttothecl
asstowhichmaj
orit
yofthethr
ee
nearestpoi
ntsbelong.Fr
om t
hef i
gur
eabov ewecanseethatthet
woofthethr
eenear
estpoi
ntsbel
ong
totheclass"Red"whil
eonebelongstothecl
ass"Bl
ue"
.Theref
oret
henewdat
apointwi
llbecl
assi
fi
edas
"Red".

Exampl
e2:

Consi
deradat asetthatcont
ainst
wov
ari
abl
es:hei
ght(
cm)&wei
ght(
kg)
.Eachpoi
nti
scl
assi
fi
edas
nor
mal orunderwei
ght.

Basedontheabov
edat
a,y
ouneedt
ocl
assi
fyt
hef
oll
owi
ngsetasnor
malorunder
wei
ghtusi
ngt
heKNN
al
gori
thm.

Tof
indt
henear
estnei
ghbor
s,wewi
l
lcal
cul
atet
heEucl
i
deandi
stance.
TheEucl
i
deandi
stancebet
weent
wopoi
ntsi
nthepl
anewi
thcoor
dinat
es(
x,y
)and(
a,b)i
sgi
venby
:

Letuscal
cul
atet
heEucl
i
deandi
stancewi
tht
hehel
pofunknowndat
apoi
nts.

Thef
oll
owi
ngt
abl
eshowst
hecal
cul
atedEucl
i
deandi
stanceofunknowndat
apoi
ntsf
rom al
lpoi
nts.
Now,
wehav
eanewdat
apoi
nt(
x1,
y1)
,andweneedt
odet
ermi
nei
tscl
ass.

Looki
ngatt
henewdat
a,wecanconsi
dert
hel
astt
hreer
owsf
rom t
het
abl
e—K=3.

Sincethemaj
ori
tyofnei
ghbor
sar
ecl
assi
fi
edasnor
mal
aspert
heKNNal
gor
it
hm,
thedat
apoi
nt(
57,
170)
shouldbenor
mal.
Howdoy oudecidet hevalueofK( numberofneighbor s)inKNN?
Now, youunder standt heKNNal gorit
hm worki
ngmechani sm.Att hi
spoi nt,thequesti
onar i
sest hatHow
tochooset heopt i
malnumberofnei ghbors?Andwhatar eitsef f
ect sont heclassi
fi
er?Thenumberof
neighbor s(K)inKNNi sahy perparameterthatyouneedchooseatt het imeofmodelbui ldi
ng.Youcan
thinkofKasacont roll
ingvari
ableforthepredi
cti
onmodel .
Resear chhasshownt hatnooptimalnumberofnei ghborssui t
sallkindofdat asets.Eachdat asethasi t
's
ownr equirements.I nt hecaseofasmal lnumberofnei ghbors,thenoi sewi l
lhaveahi gherinfluenceon
ther esul
t ,andalar genumberofnei ghborsmakeitcomput ati
onall
yexpensi ve.Researchhasal soshown
thatasmal lamountofnei ghborsar emostf l
exi
blefitwhi chwi l
lhav elowbi asbuthighv arianceanda
l
ar genumberofnei ghborswillhaveasmoot herdecisionboundar ywhi chmeansl owerv ariancebut
higherbias.
Gener all
y, Datascient i
stschooseasanoddnumberi fthenumberofcl assesi seven.Youcanal socheck
bygener atingthemodelondi ffer
entv al
uesofkandcheckt hei
rper formance.Youcanal sot ryElbow
met hodher e.

3x3conf
usi
onmat
ri
x