Chuong 4 - Hoc Co Giam Sat - Adaboost PDF

KHOA CNG NGH THNG TIN I HC KHOA HC HU
BI GING HC MY
L Thuyt Nhn Dng

Pattern Recognition
TS. Nguyn ng Bnh Khoa Cng Ngh Thng Tin, i Hc Khoa Hc Hu URL: http://it.husc.edu.vn/gv/ndbinh/ Email: nguyendangbinh@gmail.com
Nguyn ng Bnh Professor Horst Cerjak, 19.12.2005
Chng 4 Hc c gim st & S phn lp
Trang 1
BI GING HC MY
Ni Dung Bi Ging
Chng 4 Hc C Gim St (Supervised Learning)
Trang 2
BI GING HC MY
Ni dung trnh by
Vn hc c gim st (hc c mu)
Bi ton Gii thiu phng php
Phn lp tuyn tnh

Phn tch tuyn tnh (Linear separability) Thut ton nhn thc (Perceptron)
B phn loi Vc t h tr (Support Vector Machine SVM

rng ca l (Wide margin) Hm gi (Cost function) Dng nguyn thy v i ngu (Primal and dual forms) Cc bin tr (Slack variables) Hm tn tht (Loss functions)
Nhn (Kernels) Logistic Regression Boosting v Hc trc tuyn (online learning) Ti u ha

Trang 3
BI GING HC MY
Vn hc c gim st
Bi ton: Qu trnh nhn dng c bt u bi qu trnh hc trn c s bit s phn lp ban u hoc thng tin v s phn lp tp cc i tng ca n. Qu trnh hc nhm xc nh s phn lp ng theo mi i tng ca khng gian (khi s phn t ca tp hc khng thay i) V d:
Bi ton phn lp tuyn tnh: bit dng phng trnh ng bin gii xT w + w = 0 , trong w l vc t tham s v 0 w0 l gi tr tham s T dng phng trnh phn lp qu trnh hc nhm xc nh s hi t ca cc tham s w, w0 vi mi x Bi ton nhn dng theo phng php xc sut vi phn b Gauss theo 2 tham s Ak (ma trn hip bin) v k (vc t trung bnh ) i vi mi lp tnh P(Ck | x)
Trang 4
BI GING HC MY
Gii thiu phng php

1. Hc c mu theo tham s Bi ton: Bit dng ca phng trnh phn lp ph thuc vo tham s. Vn hc l i tm s hi t ca tham s vi mi x
2. Hc c mu khng theo tham s Bi ton: Bit cc mu c trng cho cc lp. Vn hc l i tm du hiu c trng cho mi lp
Trang 5
BI GING HC MY
Phn lp nh phn
Cho tp d liu hun luyn vi v sao cho: , , hc b phn loi
Trang 6
BI GING HC MY
Phn tch tuyn tnh

Phn tch tuyn tnh
Khng phn tch tuyn tnh
Trang 7
BI GING HC MY
B phn loi tuyn tnh

B phn loi tuyn tnh c dng
Trong khng gian 2 chiu phn bit l mt ng thng w bnh thng l mt mt phng (plane), v b (bias) w c gi l vc t trng s (weight vector)
Nguyn ng Bnh Professor Horst Cerjak, 19.12.2005 Chng 4 Hc c gim st & S phn lp
Trang 8
BI GING HC MY
B phn loi tuyn tnh

B phn loi tuyn tnh c dng
Trong khng gian 3 chiu phn bit l mt mt phng i vi b phn loi K-NN n l cn thit thc hin d liu hun luyn Cho mt phn loi tuyn tnh, d liu hun luyn c s dng hc w v sau d liu hun luyn loi b i Ch w l cn thit phn loi d liu mi
Trang 9
BI GING HC MY
Nhc li: B phn loi nhn thc (Perceptron)

Cho d liu phn tch tuyn tnh xi c gn nhn thnh 2 loi yi = {-1, +1}, tm vc t trng s w sao cho hm phn bit phn tch thnh cc loi vi i = 1,,N Lm th no chng ta c th tm thy siu phng phn tch ny ? Thut ton Perceptron
B phn loi c dng Khi to w = 0 Chu trnh vng lp khi cc im d liu {xi, yi}
Nu xi l phn lp nhm (misclassified)
Cho ti khi tt c d liu l c phn lp ng

Trang 10
BI GING HC MY
V d trong khng gian 2 chiu

Khi to w = 0 Lp vi cc im d liu {xi, yi}
Nu xi l phn lp nhm (misclassified)
Cho ti khi tt c d liu c phn lp ng Trc khi cp nht Sau khi cp nht
Sau khi hi t
Trang 11
BI GING HC MY
V d perceptron
Nu d liu phn tch tuyn tnh, th thut ton hi t Hi t c th chm ng phn tch gn vi d liu hun luyn Chng ti mun c mt bin ln hn (larger margin) cho s tng qut
Trang 12
BI GING HC MY
Vc t trng s w tt nht l g?
Gii php cho khon cch bin ln nht: n nh nht trong nhiu v cc u vo
Trang 13
BI GING HC MY
My vc t h tr (SVM)
Trang 14
BI GING HC MY
SVM: Pht tho s lc

T v nh ngha cng mt mt phng, chng ta c quyn t do la chn chun ha Chn chun ha sao cho h tr dng (positive) v tr m (negative) tng ng. Khon cch l c cho bi Khon cch l (Margin) = khon cch ca cc mu gn nht t ng quyt nh/siu phng
Trang 15
i vi vc t cho vc t h
BI GING HC MY
SVM
Trang 16
BI GING HC MY
SVM Ti u
Hc SVM c th c xy dng nh mt ti u ha
vi
Hoc tng ng
vi
y l mt bi ton ti u ha bnh phng vi rng buc tuyn tnh v c ti thiu duy nht
Trang 17
BI GING HC MY
SVM Thut ton hnh hc

Tnh bao li ca cc im dng (positive), v bao li ca cc im m (negative) i vi mi cp im, mt trn bao li dng v mt trn bao li m, tnh khon cch l. Chn khon cch l ln nht
Trang 18
BI GING HC MY
SVM
Ch cn xem xt nhng im trn tp bao li (nhng im bn trong khng lin quan) phn tch Siu phng xc nh bi cc vc t h tr
Trang 19
BI GING HC MY
Pht biu nh l
Chng ta c bi ton ti u
vi
Gii php tnh w lun lun c th c vit nh sau:
Chng minh: (xem nh bi tp)
Do vy, bi ton ti u ha tng ng l

vi
Trang 20
BI GING HC MY
Cng thc nguyn thy v i ngu

N l s mu hun luyn, v d l s chiu ca vc t x Bi ton nguyn thy: cho
vi
Bi ton i ngu: cho

vi
phc tp ca gii php l O(d3) i vi bi ton nguyn thy, v O(N3) i vi bi ton i ngu Nu N<<d th s hu hiu hn gii/tm hn w Dng i ngu bao gm . Chng ta s tr li vi iu ny l ti sao y l mt u im khi chng ta xem xt v Nhn (Kernels)
Trang 21
BI GING HC MY
Cng thc nguyn thy v i ngu

Nguyn thy dng ca b phn loi:
i ngu dng ca b phn loi:
Trang 22
BI GING HC MY
My vc t h tr (SVM)
Trang 23
BI GING HC MY
Phn tch tuyn tnh: w tt nht l g?

Cc mu (im) c th c phn tch tuyn tnh nhng khon cch l rt hp
Nhng c th c gii php cho khon cch l ln l tt hn, mc d mt s rng buc l b vi phm Ni chung cn c s nh i (cn bng) gia khon cch l v s lng cc li trn d liu hun luyn
Trang 24
BI GING HC MY
Gii thiu bin b (slack) cho cc mu b phn lp nhm
Trang 25
BI GING HC MY
Gii php l mm (soft margin)

Bi ton ti u nguyn thy tr thnh
Vi rng buc Mi rng buc c th tha mn nu ln C l mt tham s iu ha (regularization)

C nh cho php rng buc c th b qua -> l ln C ln m rng buc cng kh c th b qua - > l hp C = thc thi tt c cc rng buc: l cng iu ny vn con l mt bi ton ti u bc hai v c mt ti thiu duy nht. Lu , ch c mt tham s C.
Trang 26
BI GING HC MY
V d: bi ton pht hin ngi i b

Mc tiu: pht hin (khoanh vng) v tr ng con ngi trong nh Pht hin khun mt s dng b phn loi vi ca s trt
Phng php: B pht hin da trn c tnh HOG
a bi ton pht hin i tng v bi ton phn lp nh phn Tr li cu hi: trong nh c cha i tng hay khng? V nu c th nm v tr no? Ch ra vng xc nh ta v kch thc ca i tng trong nh.
Trang 27
BI GING HC MY
Hun luyn d liu v cc c trng

D liu mu dng (positive): 1208 mu dng
D liu mu m: 1218 mu m (khi to)
Trang 28
BI GING HC MY
c tnh (features):
HOG( Histogram of oriented gradients)
nh
Cc hng chnh
Phn chia nh thnh li cc vng n v kch thc 8 x 8 im nh Mi vng n v c trch chn v biu din c tnh bi HOG Hng S chiu ca vc t c tnh = 16 x 8 (vng) x 8 (hng) = 1024
Trang 29
BI GING HC MY
Trang 30
BI GING HC MY
Cc mu trung bnh
Trang 31
BI GING HC MY
Thut ton
Hun luyn (Hc): Biu din mi ca s mu bng mt vc t c tnh HOG
vi
Hun luyn b phn loi bng thut ton SVM Kim chng (testing): s dng b phn loi hun luyn pht hin i tng t nh mi Dng b phn loi trt ca s pht hin i tng
Trang 32
BI GING HC MY
Dalal v Triggs, CVPR 2005

Trang 33
BI GING HC MY
M hnh hc
Slide t Deva Ramanan

Trang 34
BI GING HC MY
iu g lm trng s m c ngha?
wx>0 (w+ w- )x > 0 w+ x > w-x
M hnh ngi i b
>
M hnh nn
Trang 35
BI GING HC MY
Hm tn tht (loss functions)

Bi ton nguyn thy: cho
vi
Rng buc nh sau iu ny tng ng vi
c th c vit gn hn
ch ra phn dng. Do vy, bi ton ti u ha tng ng vi
Trang 36
BI GING HC MY
Hm tn tht (loss functions)
SVM dng hinge loss Mt s xp x n 0-1 loss

Trang 37
BI GING HC MY
Hinge loss vs. 0-1 loss
Hinge loss l cn trn ca 0-1 loss !
Trang 38
BI GING HC MY
Nhn (Kernels)
Phn tch tuyn tnh Cc hm c s Nhn cho SVM
Kernel trick Cc yu cu Cc hm c s xuyn tm (radial basis functions)
Nhn cho cc b phn loi khc
Trang 39
BI GING HC MY
Phng php Kernel: Mt cht v lch s

Minsky and Pappert (1969) ch ra hn ch ca perceptrons. Mng n ron(t gia 1980s) vt qua cc hn ch bng cch gn vo nhau nhiu n v tuyn tnh (multilayer neural networks). Gp hn ch ca tc v cc tiu a phng. Phng php Kernels (2000s) ni cc hm tuyn tnh nhng trong khng gian feature vi s chiu cao (high dimensional feature space.)
Trang 40
BI GING HC MY
Phng php Kernel: tng chnh

Bin i d liu vo khng gian nhiu chiu hn c th bin d liu thnh tch c tuyn tnh.
Trang 41
BI GING HC MY
X l d liu khng phn tch tuyn tnh

Gii thiu cc bin b (slack)
Phn loi tuyn tnh khng thch h p
Trang 42
BI GING HC MY
Gii php 1: s dng ta cc
D liu l phn tch tuyn tnh trong ta cc Phi tuyn trong khng gian gc ban u
Trang 43
BI GING HC MY
Gii php 2: nh x d liu vo khng gian nhiu chiu hn
D liu tuyn tnh trong khng gian 3 chiu (3D) iu ny c ngha rng bi ton c th c gii quyt bng mt b phn loi tuyn tnh
Trang 44
BI GING HC MY
Phng php Kernel: Lc bin i
Trang 45
BI GING HC MY
Phng php Kernel: nn ton hc
Trang 46
BI GING HC MY
B phn loi SVM trong mt khng gian c trng bin i
Hc b phn loi tuyn tnh trong w i vi
Trang 47
BI GING HC MY
B phn loi nguyn thy trong khng gian c trng bin i

B phn loi, vi :
Hc, cho vi rng buc - Mt nh x n t x vo (x) trong d liu c phn tch - Gii tm w trong khng gian s chiu cao - phc tp tnh ton ca gii php by gi l O(D3) ch khng phi l O(d3)
Trang 48
BI GING HC MY
B phn loi i ngu trong khng gian c trng bin i

B phn loi:
Hc: vi rng buc
- Lu rng (x) ch xy ra trong cp
Trang 49
BI GING HC MY
Php bin i c bit
Vit Thay th tt c cc ln xut hin ca dng i ngu

k l nhn (kernel) bi
Trang 50
BI GING HC MY
B phn loi i ngu vi nhn (kernels)

B phn loi:
Hc: vi rng buc Kernel Trick - B phn loi c th hc v p dng mt cch r rng m khng cn tnh ton (x) - Tt c nhng g cn thit l - phc tp vn cn O(N3)
Trang 51
BI GING HC MY
Nhn hp l (Valid Kernel)

Cho mt s hm ty , lm th no chng ta bit n tng ng vi tch v hng trong mt s khng gian? Mercer kernels: nu
Tnh i xng nh ngha dng, vi tt c ma trn Gram vi cc phn t th l nhn hp l (valid kernel) , K l
tha mn:
Trang 52
BI GING HC MY
Mt s v d v kernel
Kernel tuyn tnh Kernel a thc
Cha tt c cc iu kin a thc bc d
i vi d bt k
Kernel Gaussian
Khng gian c trng c s chiu v hn
Trang 53
BI GING HC MY
B phn loi SVM vi nhn Gaussian

Kch thc tp d liu
Trng s (c th bng 0)
Vc t h tr
Hm xuyn tm c s (RBF) SVM

Trang 54
BI GING HC MY
RBF Kernel SVM
D liu khng phn tch tuyn tnh trong khng gian c trng ban u
Trang 55
BI GING HC MY
Trang 56
BI GING HC MY
Gim C, cho rng l (mm) ln hn

Trang 57
BI GING HC MY
Trang 58
BI GING HC MY
Trang 59
BI GING HC MY
Gim sigma, tin ti b phn loi xm ging gn nht

Trang 60
BI GING HC MY
Trang 61
BI GING HC MY
ng dng: nhn dng ch s vit tay

Vc t c trng: mi nh l 28x28 im nh. Sp xp li nh l mt vc t x c 784 chiu Hun luyn: hc k=10 nh phn lp: 1 chng li phn cn li. Cc b phn loi SVM fk(x) Phn lp: Chn ln m c gi tr dng ln nht
Trang 62
BI GING HC MY
V d
Vit tay
Phn lp
Trang 63
BI GING HC MY
Kt lun v Kernel Trick

B phn loi co th hc vi khng gian cc c trng s chiu cao, m khng cn phi nh x cc im vo khng gian s chiu cao D liu c th phn tch tuyn tnh trong khng gian s chiu cao, nhng khng phn tch trong khng gian c trng ban u Kernels c th s dng cho mt SVM bi v dng tch v hng, nhng cng c th c s dng nhng ni khc m n khng gn lin vi SVM Chng ta s thy v d khc v Kernels sau ny trong l thuyt hi quy v hc khng c gim st
Trang 64
BI GING HC MY
Logistic Regression
Gaussian Nave Bayes Logistic Regression Hm Logistic v hm Sigmoid
Learning & Margin Logistic Regression Loss function Logistic Regression Learning
Trang 65
BI GING HC MY
Gaussian Nave Bayes

Xem xt bin boolean lin tc Gi s
gi s
Trang 66
BI GING HC MY
Li ti thiu c th l g?
Trng hp tt nht
Gi s iu kin c lp c tha mn Chng ta bit P(Y), P(X|Y) hon ho (v d d liu hun luyn v hn)
Trang 67
BI GING HC MY
Logistic Regression
Logistic regression (LR) tht s l mt phng php phn lp tng
Nave Bayes cho php tnh ton P(Y|X) bi hc P(Y) v P(X|Y) Ti sao khng hc P(Y|X) trc tip?
LR gii thiu mt s m rng ca phi tuyn tnh qua mt b phn loi tuyn tnh , bi s dng hm logistic (hoc sigmoid) B phn loi LR c nh ngha nh sau

Trang 68
BI GING HC MY
Xem xt hc f: X -> Y hoc P(Y|X),

X l mt vc t cc c trng gi tr thc, <X1 . Xn> Y l boolean Gi s rng tt c Xi l iu kin c lp cho Y M hnh P(Xi | Y = yk) l Gaussian N(ik,i) (khng phi ik) M hnh P(Y) l Bernoulli () iu hm rng dng ca P(Y|X)?
Tham s
Trang 69
BI GING HC MY
Dn xut dng ca P(Y|X) cho Xi lin tc
Trang 70
BI GING HC MY
Hm
Hm
Hm
Trang 71
BI GING HC MY
Hm
Hm Lut phn lp tuyn tnh
Hm
Trang 72
BI GING HC MY
Gii thch theo xc sut

Ngha rng (f(x)) l xc sut hu nghim m y = 1, tc l P(y = 1 | x) = (f (x)) Do , nu (f (x))> 0,5 th lp y = 1 c chn
Trang 73
BI GING HC MY
Hm logistic hoc hm sigmoid
Khi z i t n , (z) i t 0 n 1, hm squashing. Hnh dng ca sigmoid (v d hnh dng ging S:) (0) = 0.5, v nu th
Trang 74
BI GING HC MY
Trc gic: tai sao dng sigmod?

y, chn nh phn lp biu din bi ch khng phi l
Ti thiu bnh phng l ph hp ph hp vi y ph hp vi y Thch hp ca wx+b chi phi cc im xa hn Gy ra phn lp sai Thay v LR sigmod hi qui vi lp d liu
Trang 75
BI GING HC MY
Tng t trong 2 chiu
Trang 76
BI GING HC MY
Hc (learning)
Trong hi qui logistic ph hp vi hm sigmod s liu {xi, yi} bng cch gim thiu cc li phn loi
Trang 77
BI GING HC MY
Tnh cht ca l (bin)

Mt hm sigmod thin v l ln hn so vi b phn loi hm bc
Mc d vy, cn phi kim sot dc (gradient). Lm th no?

Trang 78
BI GING HC MY
c tnh kh nng (likelihood) ln nht

Gi s
Vit li gn hn Sau kh nng xy ra (gi s d liu c lp) l
V log likelihood m l
Trang 79
BI GING HC MY
Hm tn tht hi qui Logistic (Logistic Regression Loss function)

S dng k kiu sau cho
V vy trong c hai trng hp Gi s c lp, kh nng (likelihood) l V log likelihood m l l nh ngha hm tn tht (loss function)
Trang 80
Hc hi qui Logistic (Logistic Regression Learning)
BI GING HC MY
Hc c hnh thnh nh l bi ton ti tu
i vi cc im phn lp ng v l gn ti 0 i vi cc im phn lp sai v c th ln
l m (negative), l dng (positive),
Do vic ti u ha cc tham s gy bt li dn n phn lp sai (phn nhm lp). Cc qui chun (regularization) ngn chn over-fitting v duy tr l (bin)
Trang 81
BI GING HC MY
So snh hm chi ph ca SVM v LR

SVM
Logistic regression Lu :
C hai xp x 0-1 loss Hnh vi tim cn tng t S khc bit chnh l trn ca LR v khc 0 ngoi l SVM SVM cho gii php tha (sparse) i vi i
Trang 82
BI GING HC MY
AdaBoost
AdaBoost (Adaptive Boosting) l mt thut ton xy dng b phn loi mnh da trn s kt hp tuyn tnh cc b phn loi yu hn(x). N cung cp phng php la chn b phn loi yu v thit lp cc trng s
B phn loi mnh
B phn loi yu
Cp nht trng s t mu hun luyn (Boosting)

Y. Freund and R. Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences, 1997.
Trang 83
BI GING HC MY
Khng gian c tnh
AdaBoost
u vo
[Freund, Schapire 97]
- Tp d liu mu gn nhn - Phn b trng s i vi cc mu u ra
Thut ton for n=1 to N // N l s lng b phn loi yu - hun luyn b phn loi yu s dng mu v phn b trng s - tnh ton li - tnh ton trng s b phn loi - cp nht cc trng s mu end
Trang 84
BI GING HC MY
Tai sao n hot ng?

Thut ton AdaBoot thc hin ti u ha tham lam ca hm tn tht (loss function)
Trang 85
BI GING HC MY
T Boosting ngoi tuyn (offline) n trc tuyn (online)

Ngoi tuyn (Off-line)
u vo - tp mu hun luyn gn nhn
Trc tuyn (On-line)
[Freund, Schapire 97]
[Oza, Russel 01]
u vo - MT mu hun luyn c gn nhn
- trng s phn b trn cc mu
- b phn loi mnh c cp nht - khi to trng s quan trng
u ra
u ra
Thut ton For n=1 to N - hun luyn b phn loi yu s dng cc mu v phn b trng s
Thut ton For n=1 to N - cp nht b phn loi yu s dng cc mu v trng s quan trng
- tnh ton li - tnh ton tin cy - cp nht trng s End

- cp nht c tnh li - cp nht tin cy - cp nht trng s End

Trang 86
BI GING HC MY
Boosting cho la chn c trng

Mi c tnh tng ng vi mt b phn loi yu Cc c tnh
Sng con Haar-like Biu nh hng Nh phn cc b (LBP)
Tnh ton nhanh s dng cc cu trc d liu hu hiu

nh tch hp -integral images Cc biu c tnh tch hp
K. Tieu and P. Viola. Booting Image Retrival, CVPR 2000
F. Porikli. Integral histogram: A fast way to extract histograms in cnartesian spaces. CVPR 2005.
Trang 87
BI GING HC MY
B phn loi
Kt hp ca cc c tnh n gin s dng Boosting la chn c tnh/c trng
P. Viola and M. Jones. Rapid object detection using a boosted cascade of simple features. CVPR 2001.
Trang 88
BI GING HC MY
Boosting ngoi tuyn cho la chn c trng

Gii thiu B chn Selector
La chn mt c trng t c s d liu c trng cc b
Boosting c thc hin trn b chn (Selectors) ch khng phi trn b phn loi yu (weak classifiers) trc tip.
Trang 89
BI GING HC MY

set of training samples
inital weight distribution
straightforward with all training samples
Trang 90
BI GING HC MY

set of training samples hSelector1 h1,1 h1,2
. . .
h1,M
calculate weight
Trang 91
BI GING HC MY

set of training samples hSelector1 h1,1 h1,2 update weight distribution for training samples
. . .
h1,M
calculate weight
Trang 92
BI GING HC MY

set of training samples hSelector1 h1,1 h1,2 update weight distribution for training samples hSelector2
h2,1
h2,2 . . . h2,m update weight distribution for training samples
. . .
h1,M
h2,M
calculate weight
calculate weight
Trang 93
BI GING HC MY

set of training samples hSelector1 h1,1 h1,2 update weight distribution for training samples hSelector2 hSelectorT
h2,1
hT,1
h2,2 . . . h2,m update weight distribution for training samples
hT,2
. . .
. . .
hT,m
. . .
h1,M
h2,M
hT,M
calculate weight
calculate weight
calculate weight
Trang 94
BI GING HC MY

set of training samples hSelector1 h1,1 h1,2 update weight distribution for training samples hSelector2 hSelectorT
h2,1
hT,1
h2,2 . . . update weight distribution for training samples
hT,2
NOW GO ON-LINE
. . . h2,m . . . h1,M h2,M hT,M calculate weight calculate weight calculate weight
. . .
hT,m
final strong classifier hStrong
Trang 95
BI GING HC MY
Boosting trc tuyn cho la chn c trng
Trang 96
BI GING HC MY
Trang 97
BI GING HC MY
M hnh hc Boosting trc tuyn vi s gim st ca con ngi
Trang 98
BI GING HC MY
Trang 99
BI GING HC MY
Ti u Ha (Optimization)
Trang 100
BI GING HC MY
Chng ta thy nhiu hm chi ph
l nhng gii php duy nht ? Liu nhng gii php ph thuc vo im khi u ca mt thut ton ti u ha lp i lp li (chng hn nh gradient descent) Nu hm gi tr (cost function) l hm li (convex) th im ti u cc b l ti u ton cc (cung cp ti u ha trn mt tp li, m n l mt trong trng hp)
Trang 101
BI GING HC MY
Hm li (Convex functions)
D l min Rn Mt hm li l mt trong nhng tha mn i vi bt k x0 v x1 trong D
ng ni (x0, f(x0)) v (x1, f(x1)) nm trn hm th
Trang 102
BI GING HC MY
V d v hm li (convex)
Tng khng m ca mt hm li l li (A non-negative sum of convex functions is convex)

Trang 103
BI GING HC MY
Trang 104
BI GING HC MY
Thut ton gim nghing (Gradient descent or steepest descent)

Cch n gin nht hiu gradient descent l t v tr hin ti, ta i theo chiu gim ca o hm bc nht cho n khi khng th gim c na. Khi ta mt im ti u cc b.
ti thiu ha hm chi ph C(w) dng vng lp cp nht l tc hc Trong trng hp chng ti, hm tn tht l tng trn ton d liu hun luyn. Chng hn i vi LR
iu ny c ngha rng mt cp nht lp i lp li bao gm i qua cc d liu hun luyn vi mt bn cp nht cho mi im u im l cho mt lng ln d liu, iu ny c th c thc hin tng im mt.
Trang 105
BI GING HC MY
Gradient Descent
Trang 106
BI GING HC MY
Thut ton gim nghing cho LR

Ti thiu ha s dng gradient descent cho lut cp nht [xem nh bi tp] Lu :
iu ny tng t nhng khng ging ht nhau vi lut cp nht ca perceptron
C mt gii phapr duy nht cho w Trong thc t phng php Newton hiu qu hn c s dng ti thiu ha LR C th bi ton vi w tr thnh v hn cho d liu phn tch tuyn tnh
Trang 107
BI GING HC MY
Gim nghing cho SVM

u tin, bi ton ti u ha c vit li nh l trung bnh
Vi = 2/(NC) ln n t l chung ca bi ton v
Bi v hinge loss khng kh vi, mt tiu dc (subgradient) c tnh
Trang 108
BI GING HC MY
Sub-gradient for hinge loss
Trang 109
BI GING HC MY
Thut ton gim tiu nghing cho SVM
Cp nht lp i lp li l:
l tc hc Sau , mi ln lp t bao gm chu trnh qua d liu hun luyn vi cc cp nht:
Trong thut ton Pegasos t l hc c thit lp t = 1/t

Trang 110

Chuong 4 - Hoc Co Giam Sat - Adaboost PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Chuong 4 - Hoc Co Giam Sat - Adaboost PDF

Uploaded by

Copyright:

Available Formats

KHOA CNG NGH THNG TIN I HC KHOA HC HU

L Thuyt Nhn Dng

Nguyn ng Bnh Professor Horst Cerjak, 19.12.2005

Chng 4 Hc c gim st & S phn lp

KHOA CNG NGH THNG TIN I HC KHOA HC HU

Chng 4 Hc C Gim St (Supervised Learning)

Nguyn ng Bnh Professor Horst Cerjak, 19.12.2005

Chng 4 Hc c gim st & S phn lp

KHOA CNG NGH THNG TIN I HC KHOA HC HU

Phn lp tuyn tnh

B phn loi Vc t h tr (Support Vector Machine SVM

Nhn (Kernels) Logistic Regression Boosting v Hc trc tuyn (online learning) Ti u ha

Nguyn ng Bnh Professor Horst Cerjak, 19.12.2005

KHOA CNG NGH THNG TIN I HC KHOA HC HU

Nguyn ng Bnh Professor Horst Cerjak, 19.12.2005

KHOA CNG NGH THNG TIN I HC KHOA HC HU

Gii thiu phng php

Nguyn ng Bnh Professor Horst Cerjak, 19.12.2005

Chng 4 Hc c gim st & S phn lp

KHOA CNG NGH THNG TIN I HC KHOA HC HU

Nguyn ng Bnh Professor Horst Cerjak, 19.12.2005

Chng 4 Hc c gim st & S phn lp

KHOA CNG NGH THNG TIN I HC KHOA HC HU

Phn tch tuyn tnh

Khng phn tch tuyn tnh

Nguyn ng Bnh Professor Horst Cerjak, 19.12.2005

Chng 4 Hc c gim st & S phn lp

KHOA CNG NGH THNG TIN I HC KHOA HC HU

B phn loi tuyn tnh

KHOA CNG NGH THNG TIN I HC KHOA HC HU

B phn loi tuyn tnh

KHOA CNG NGH THNG TIN I HC KHOA HC HU

Nhc li: B phn loi nhn thc (Perceptron)

Cho ti khi tt c d liu l c phn lp ng

KHOA CNG NGH THNG TIN I HC KHOA HC HU

V d trong khng gian 2 chiu

KHOA CNG NGH THNG TIN I HC KHOA HC HU

KHOA CNG NGH THNG TIN I HC KHOA HC HU

KHOA CNG NGH THNG TIN I HC KHOA HC HU

Nguyn ng Bnh Professor Horst Cerjak, 19.12.2005

Chng 4 Hc c gim st & S phn lp

KHOA CNG NGH THNG TIN I HC KHOA HC HU

SVM: Pht tho s lc

KHOA CNG NGH THNG TIN I HC KHOA HC HU

Nguyn ng Bnh Professor Horst Cerjak, 19.12.2005

Chng 4 Hc c gim st & S phn lp

KHOA CNG NGH THNG TIN I HC KHOA HC HU

Nguyn ng Bnh Professor Horst Cerjak, 19.12.2005

Chng 4 Hc c gim st & S phn lp

KHOA CNG NGH THNG TIN I HC KHOA HC HU

SVM Thut ton hnh hc

Nguyn ng Bnh Professor Horst Cerjak, 19.12.2005

Chng 4 Hc c gim st & S phn lp

KHOA CNG NGH THNG TIN I HC KHOA HC HU

Nguyn ng Bnh Professor Horst Cerjak, 19.12.2005

KHOA CNG NGH THNG TIN I HC KHOA HC HU

Gii php tnh w lun lun c th c vit nh sau:

Chng minh: (xem nh bi tp)

Do vy, bi ton ti u ha tng ng l

Nguyn ng Bnh Professor Horst Cerjak, 19.12.2005

Chng 4 Hc c gim st & S phn lp

KHOA CNG NGH THNG TIN I HC KHOA HC HU

Cng thc nguyn thy v i ngu