8 MaxEnt Handout

Ni dung
multinomial logistic regression
Gii thiu
Classification and Prediction
M hnh hi quy logistic a lp

c lng tham s
Hiu chnh tham s
V d
Phn loi nh
Phn loi hoa Iris
M hnh hi quy logistic a lp vi hm c trng

Hm c trng
M hnh hi quy logistic vi hm c trng
Bi tp
L Hng Phng
<phuonglh@gmail.com>
Vietnam National University of Hanoi
Hanoi University of Science
October 2014
L Hng Phng
(HUS, VNU)
Multinomial Logistic Regression
October 2014
1 / 66
Ni dung
L Hng Phng
(HUS, VNU)
October 2014
2 / 66
Gii thiu
Khi bi ton phn loi c nhiu lp, ta c th m rng m hnh

hi quy logistic nh phn trn cho trng hp a lp.

c lng tham s
Hiu chnh tham s
M hnh hi quy logistic a lp cn c gi l m hnh entropy

cc i (maximum entropymaxent), mt dng ca m hnh
logtuyn tnh.
V d
Phn loi nh
Phn loi hoa Iris

Hm c trng
Bi tp
L Hng Phng
(HUS, VNU)
October 2014
3 / 66
L Hng Phng
(HUS, VNU)
October 2014
4 / 66
Ni dung
M hnh entropy cc i c pht minh nhiu ln, trong nhiu lnh

vc khc nhau:
Trong l thuyt xc sut, di cc tn m hnh entropy cc i, m
hnh logtuyn tnh, trng ngu nhin Markov v h hm m ;
Gii thiu

c lng tham s
Hiu chnh tham s
V d
Phn loi nh
Phn loi hoa Iris

Hm c trng
Bi tp
Trong thng k ton hc di tn hi quy logistic;

Trong c hc thng k v vt l, di cc tn phn phi Gibbs,
phn phi Boltzmann;
Trong cc mng n-ron di tn my Boltzmann v hm kch hot
softmax.
L Hng Phng
(HUS, VNU)
October 2014
5 / 66
1
exp(kT x),
Z
(1)
trong Z l s hng chun ho m bo phn phi xc sut:

Z=
K
X
k=1
L Hng Phng
(HUS, VNU)
P (y = k| x; k ) =
(HUS, VNU)
October 2014
6 / 66
Xc sut i tng x thuc lp k {1, 2, . . . , K} c m hnh bi:

P (y = k| x; k ) =
L Hng Phng
K
X
exp(kT x).
(2)
k=1
October 2014
7 / 66
Tham s k = (k0 , k1 , . . . , kD )T l mt vc-t tham s D + 1

chiu ng vi lp k.
Mi lp k c mt vc-t tham s k ng vi D + 1 c trng (c
trng th 0 c c nh l n v).
Ta c ma trn tham s ca
10
20

K0
L Hng Phng
(HUS, VNU)
m hnh:
11
21
K1
1D
2D
.

KD
October 2014
8 / 66
Ni dung
V iu kin chun ho
K
X
P (y = k| x; k ) = 1,
Gii thiu

c lng tham s
Hiu chnh tham s
V d
Phn loi nh
Phn loi hoa Iris

Hm c trng
Bi tp
k=1
nn ta ch cn c lng (K 1) vc-t tham s k .

Do , vc-t tham s ca m hnh c (K 1) (D + 1) chiu.
L Hng Phng
(HUS, VNU)
October 2014
9 / 66
c lng tham s
Trung bnh ca log-hp l ca tp d liu hun luyn l:

N
1 X
log P (yi | xi ; ).
() =
N
October 2014
10 / 66
c lng cc tham s ca m hnh, ta cn tm cc tiu ho

hm mc tiu sau:
(4)
J() = () + R(),
trong R() l s hng hiu chnh dng trnh hin tng qu

khp v tng chnh xc ca m hnh.
Mc tiu ca vic hiu chnh l lm trn m hnh, pht cc
tham s ln.
Tham s 0 dng iu khin tnh cn bng ca m hnh
trong vic ph hp vi d liu quan st v vic hiu chnh.
i=1
(HUS, VNU)
(HUS, VNU)
c lng tham s
Cng thc tnh xc sut i tng x thuc lp y trong m hnh

entropy cc i:

P
D
x
exp
j=0 yj j
.
P
P (y| x; ) = P
(3)
D
K
x
exp
j
kj
j=0
k=1
L Hng Phng
L Hng Phng
October 2014
11 / 66
L Hng Phng
(HUS, VNU)
October 2014
12 / 66
Ni dung
Hiu chnh dng L1

Nu s dng hiu chnh dng L1 th hm mc tiu l
Gii thiu

c lng tham s
Hiu chnh tham s
J1 () = () +
Ch rng hm mc tiu J1 khng phi l hm li nn nghim ti u

cc b c th khng phi l nghim ti u ton cc.
V d
Phn loi nh
Phn loi hoa Iris

Hm c trng
Bi tp
(HUS, VNU)
October 2014
13 / 66
Hiu chnh dng L2
X 2
j .
J2 () = () +
2
(6)
j=1
D thy hm mc tiu J2 l hm li nn ta c th dng cc thut ton

ti u li tm tham s ti u ca m hnh.
(HUS, VNU)
L Hng Phng
(HUS, VNU)
October 2014
14 / 66
Hiu chnh dng L2
Hiu chnh dng L2 l mt hm ton phng, hm mc tiu l:
L Hng Phng
(5)
|j |.
j=1
L Hng Phng
D
X
October 2014
15 / 66
Kiu hiu chnh L2 tng ng vi vic gi nh rng cc tham

s j tun theo phn phi chun vi trung bnh = 0 v phng
sai 2 .
Do , nu mt tham s j cng xa gi tr trung bnh 0 th xc
sut ca n cng nh (t l vi lch chun ).
Ta c:
1
j2
P (j ) = q
exp 2
2j
2j2
L Hng Phng
(HUS, VNU)
October 2014
16 / 66
Hiu chnh dng L2
Hiu chnh dng L2
Theo cng thc Bayes:
Nu vit di dng cc log xc sut:
P (|(xi , yi )) P ((xi , yi )|)P (),
N
log P (|{xi , yi }N
i=1 ) = log P ({xi , yi }i=1 |)+
!
D
2
1
X
j
log q
exp 2
+ c,
2 2
2
j
j=1
trong P () l xc sut tin nghim ca tham s.

Nu gi nh cc tham s j l c lp th ta c
P () =
D
Y
vi c l mt hng s.
P (j ).
T , hm mc tiu J s c dng
j=1
J() = () +
D
X
j2
j=1
2j2
v y chnh l dng hiu chnh L2 .

L Hng Phng
(HUS, VNU)
October 2014
17 / 66
L Hng Phng
(HUS, VNU)
c lng tham s
c lng tham s
Ta c
Do
kj
N
1 X
log P (yi | xi ; )
() =
N
i=1
(
!)
N
K
X
1 X T
=
yi xi log
exp(kT xi )
.
N
i=1
N
1 X
()
=
kj
N
1
=
N
L Hng Phng
i=1
(
N
X
i=1
k=1
(yi = k) xij PK
T
kj
k=1 exp(k xi )
(HUS, VNU)
1
=
N
!)
K
X
= exp(kT xi ) xij ,
)
(
N
exp(kT xi )
1 X
xij
() =
(yi = k) xij PK
T
kj
N
k=1 exp(k xi )
k=1
K
X
(yTi xi )
log
exp(kT xi )
kj
kj
k=1
exp(kT xi )
18 / 66
nn
T
(
K
X
October 2014
exp(kT xi )
k=1
October 2014
!)
19 / 66
i=1
N
X
i=1
N
1 X
(yi = k) xij
P (y = k| xi ; ) xij .
N
i=1
L Hng Phng
(HUS, VNU)
October 2014
20 / 66
c lng tham s
c lng tham s
Vi m hnh entropy cc i hiu chnh dng L2 th o hm ring
ca hm mc tiu J2 () ng vi tham s kj l
N
N
1 X
1 X
(yi = k) xij
P (y = k| xi ; ) xij
() =
kj
N
N
i=1
i=1
Nhn xt:
P
i lng N1 N
i=1 (yi = k) xij l k vng mu ca c trng th
j trn mu hun luyn ng vi lp k;
P
i lng N1 N
i=1 P (y = k| xi ; ) xij l k vng ca c trng th
j ng vi m hnh P (y = k| x; ).
L Hng Phng
(HUS, VNU)
October 2014
21 / 66
c lng tham s
J2 ()
=
kj
!
N
N
X
X
1
1
(yi = k) xij
P (y = k| xi ; ) xij +kj .
N
N
i=1
i=1
c lng vc-t tham s , ta cn gii h phng trnh
J2 () = 0, j = 0, 1, 2, . . . , D.
kj
L Hng Phng
(HUS, VNU)
October 2014
22 / 66
c lng tham s: Phng php thang lp
C nhiu thut ton c dng c lng tham s ca m hnh

entropy cc i.
Hai phng php chnh:
phng php thang lp
phng php ti u
Thut ton GIS (Generalized Iterative Scaling)

J. N. Darroch and D. Ratcliff, Generalized iterative scaling for
log-linear models, Annals of Mathematical Statistics, vol. 43, no. 5,
pp. 14701480, 1972.
Thut ton IIS (Improved Iterative Scaling)

S. D. Pietra, V. D. Pietra, and J. Lafferty, Inducing features of
random fields, IEEE PAMI, vol. 19, no. 4, pp. 380393, 1997.
J. Lafferty, A. McCallum, and F. Pereira, Conditional random
fields: Probabilistic models for segmenting and labeling sequence
data, in ICML, 2001, pp. 282289.
Thut ton SCGIS (Sequential Conditional Generalized Iterative

Scaling)
J. Goodman, Sequential conditional generalized iterative scaling,
in Proceedings of ACL, 2002, pp. 916.
L Hng Phng
(HUS, VNU)
October 2014
23 / 66
L Hng Phng
(HUS, VNU)
October 2014
24 / 66
c lng tham s: Phng php ti u
c lng tham s
Phng php gradient bc mt: phng php gim gradient,

phng php gradient lin hp;
Phng php gradient bc hai : phng php Newton v cc
phng php ta-Newton:
Ch rng cc phng php ti u cng s dng cc th tc lp

tm chui {(n) }
n=1 hi t ti gi tr ti u ca tham s.
tm hiu c th v cc phng php hun luyn m hnh
entropy cc i, xem thm cc ti liu:
thut ton BFGS J. Kazama and J. Tsujii, Evaluation and

extension of maximum entropy models with inequality constraints,
in EMNLP, 2003.
thut ton L-BFGS J. Nocedal and S. J. Wright, Numerical
Optimization, 2nd ed. New York: Springer, 2006.
thut ton OWL-QN (Orthant-wise Limited-memory
Quasi-Newton) G. Andrew and J. Gao, Scalable training of
l1 -regularized log-linear models, in ICML, 2007, pp. 3340.
thut ton Newton ct C.-J. Lin, R. C. Weng, and S. S. Keerthi,
Trust region Newton methods for large-scale logistic regression, in
Proceedings of the 24th ICML, Corvallis, OR, 2007.
L Hng Phng
(HUS, VNU)
October 2014
25 / 66
c lng tham s: Mt s nhn xt
J. Gao, G. Andrew, M. Johnson, and K. Toutanova, A comparative

study of parameter estimation methods for statistical natural
language learning, in Proceedings of ACL, 2007, pp. 824831.
M. Collins, A. Globerson, T. Koo, X. Carreras, and P. L. Bartlett,
Exponentiated gradient algorithms for conditional random fields
and max-margin Markov networks, The Journal of Machine
Learning Research (JMLR), vol. 9, pp. 17751822, 2008.
F.-L. Huang, C.-J. Hsieh, K.-W. Chang, and C.-J. Lin, Iterative
scaling and coordinate descent methods for maximum entropy, in
Proceedings of ACL-IJCNLP, 2009, pp. 285288.
R. Malouf, A comparison of algorithms for maximum entropy
parameter estimation, in CONLL, 2002.
L Hng Phng
(HUS, VNU)
October 2014
26 / 66
Cc thut ton ti u c tc v hiu qu cao hn cc thut

ton thang lp.
Trong nhiu trng hp, cc phng php L-BFGS v gradient lin

hp l tt hn gim gradient ngu nhin trong nhiu trng hp.
Phng php thang lp cp nht mi thnh phn j ca ti mt

thi im, nn chi ph ti mi bc lp l nh nhng s bc lp
l ln.
Nu s lng tham s l tng i nh th L-BFGS cho kt qu

tt, cn vi cc bi ton c s chiu ln th phng php gradient
lin hp thng cho kt qu tt.
Ngc li, phng php (ta) Newton c chi ph cao ti mi bc

lp v phi tnh ng (xp x) Hessian ca hm mc tiu nhng c
tc hi t nhanh.
Cc phng php gradient lin hp v L-BFGS cng c th tn

dng c cc thut ton tnh ton song song tt hn.
L Hng Phng
(HUS, VNU)
October 2014
27 / 66
L Hng Phng
(HUS, VNU)
October 2014
28 / 66
M hnh entropy cc i hiu chnh dng L2 thng cho kt qu

cao hn mt cht m hnh entropy cc i hiu chnh dng L1 .
Vi dng hiu chnh L2 , o hm ca cc s hng hiu chnh

j 0 khi j 0.
Tuy nhin, dng chun ho L1 c hiu qu gn tng t m li

c tc hun luyn nhanh hn nhiu so vi dng hiu chnh L2 .
Tc ng ca s hng hiu chnh gim dn nu j nh.
L Hng Phng
(HUS, VNU)
October 2014
29 / 66
Tc ng ca cc s hng hiu chnh l khng i, khng ph

thuc vo mc ln nh ca j .
Do , dng chun ho L1 sinh m hnh tha, theo ngha s cho
kt qu c lng trong c nhiu tham s j = 0.
V vy, dng hiu chnh L1 cn c dng lm phng php chn
cc c trng.
(HUS, VNU)
L Hng Phng
(HUS, VNU)
October 2014
30 / 66
October 2014
32 / 66
Ni dung
Vi dng hiu chnh L1 , o hm ca cc s hng hiu chnh l

sign(j ) {, } tr khi j = 0.
L Hng Phng
T , dng hiu chnh L2 lm cc tham s thng l nh, xp x

0, nhng khng bng 0.
October 2014
31 / 66
Gii thiu

c lng tham s
Hiu chnh tham s
V d
Phn loi nh
Phn loi hoa Iris

Hm c trng
Bi tp
L Hng Phng
(HUS, VNU)
Ni dung
Phn loi nh
Tp nh c cung cp bi nhm nghin cu th gic my tnh
ca i hc Massachusetts, Hoa K.
Gii thiu

c lng tham s
Hiu chnh tham s
Tp d liu gm 210 nh dng hun luyn m hnh v 2100 nh

dng kim tra chnh xc ca m hnh.
Mi nh c phn vo mt trong 7 lp sau: mt gch (brickface),
bu tri (sky), l cy (foliage), xi-mng (cement), ca s
(window), ng i (path) v c (grass).
V d
Phn loi nh
Phn loi hoa Iris
Mi lp c 30 mu hun luyn v 300 mu kim tra.

Hm c trng
Bi tp
L Hng Phng
(HUS, VNU)
Cc mu nh c trch ra t 7 bc nh ngoi tri v c phn

on bng tay to phn loi cho tng im nh. Mi mu nh
l mt vng im nh kch thc 3 3.
October 2014
33 / 66
Phn loi nh
L Hng Phng
(HUS, VNU)
October 2014
34 / 66
Phn loi nh
Mi mu c 19 c trng l cc s thc:
hedge-mean: o tng phn ca cc im nh k nhau theo

chiu dc. c s dng pht hin on nm ngang;
hedge-sd : xem c trng 8;
region-centroid-col : ch s ct ca im nh trung tm ca vng;
region-centroid-row : ch s hng ca im nh trung tm ca vng;
region-pixel-count: s im nh ca vng, y bng 9;
10
intensity-mean: gi tr trung bnh trong vng ca (R + G + B)/3;
short-line-density-5 : kt qu ca mt thut ton trch on thng,

l s on thng di 5 (hng bt k) vi tng phn thp,
nh hn hoc bng 5, i qua vng nh;
11
rawred-mean: gi tr trung bnh trong vng ca gi tr R;
12
rawblue-mean: gi tr trung bnh trong vng ca gi tr G;
13
rawgreen-mean: gi tr trung bnh trong vng ca gi tr G;
14
exred-mean: o mu tha: (2R (G + B));
15
exblue-mean: o mu xanh da tri tha : (2B (G + R));
16
exgreen-mean: o mu xanh l cy tha: (2G (R + B));
17
value-mean: bin i phi tuyn 3 d ca RGB.
18
saturation-mean: xem c trng 17;
19
hue-mean: xem c trng 17.
short-line-density-2 : ging nh short-line-density-5 nhng m s

on thng c tng phn cao, ln hn hoc bng 5;
vedge-mean: o tng phn ca cc im nh nm k nhau
theo chiu ngang trong vng. C 6 im nh, gi tr trung bnh v
lch chun cho trc. c trng ny c s dng pht hin
cnh dc.
vegde-sd : xem c trng 6;
L Hng Phng
(HUS, VNU)
October 2014
35 / 66
L Hng Phng
(HUS, VNU)
October 2014
36 / 66
Phn loi nh
Phn loi nh
brickface
sky
foliage
cement
window
path
grass
brickface
sky
foliage
cement
window
path
grass
0
1
2
3
4
5
6
7
8
9
0.0282
0.0197
0.0007
0.2541
0.0047
0.0002
0.1484
0.0497
0.0118
0.1309
0.0635
0.0125
0.039
0.5715
0.0007
0.0015
0.3321
0.327
0.2038
0.1823
0.1531
0.0068
0.0613
1.3779
0.0076
0.0037
0.3887
0.2637
0.3646
0.0351
0.0025
0.0137
0.0045
0.0226
0.029
0.0062
0.1627
0.0627
0.0244
0.0579
0.1958
0.0154
0.0718
1.7625
0.0179
0.0051
0.1284
0.1015
0.3072
0.122
0.2794
0.0037
0.1286
2.5145
0.0013
0.0095
0.234
0.1441
0.0579
0.0833
0.0318
0.0073
0.0488
0.2858
0.0006
0
0.0347
0.0928
0.0522
0.0149
10
11
12
13
14
15
16
17
18
19
0.0905
0.3326
0.0122
0.592
1.2693
0.2351
1.5044
0.0026
0.0036
0.0435
0.0828
0.0878
0.0536
0.2826
0.5118
0.0876
0.5994
0.0501
0.0181
0.1154
0.1482
0.445
0.1626
0.163
0.8904
0.0431
0.9336
0.1919
0.6181
1.1184
0.03
0.1565
0.1011
0.1677
0.3797
0.2133
0.593
0.1542
0.0386
0.0238
0.0013
0.0414
0.0288
0.0087
0.1203
0.0903
0.03
0.1771
0.4281
0.3079
0.0564
0.1107
0.0924
0.034
0.163
0.108
0.271
0.0146
0.1235
0.3985
0.0709
0.0257
0.1011
0.3394
0.2896
0.516
0.8055
0.1066
0.0134
0.277
chnh xc ca m hnh:
Trn tp hun luyn: 96.66%
Trn tp kim tra: 93.09%
L Hng Phng
(HUS, VNU)
October 2014
37 / 66
Ni dung
1
2
L Hng Phng
October 2014
38 / 66
Iris
Tp d liu v hoa Iris1 ni ting trong lnh vc nhn dng.
Gii thiu
Xut hin trong bi bo ca Ronald Fisher nm 1936, ngy nay

vn c dng thng xuyn.

c lng tham s
Hiu chnh tham s
V d
Phn loi nh
Phn loi hoa Iris

Hm c trng
Bi tp
Tp hun luyn: 130 mu, tp kim tra: 20 mu
1
L Hng Phng
(HUS, VNU)
(HUS, VNU)
October 2014
39 / 66
c trng
Lp
Setosa
Versicolour
Virginica
di ca l i
rng ca l i
di ca cnh hoa
rng ca cnh hoa
http://archive.ics.uci.edu/ml/datasets/Iris
L Hng Phng
(HUS, VNU)
October 2014
40 / 66
Phn loi hoa Iris
Phn loi hoa Iris
L-BFGS, khng s dng hiu chnh tham s, chnh xc ca m

hnh trn tp kim tra l 100% v trn tp hun luyn l 98.46%
Khi s dng hiu chnh tham s th s bc lp v chnh xc

ca m hnh ng vi cc tham s c cho trong bng sau:
Cc tham s c c lng nh sau:

Lp
setosa
versicolor
virginica
L Hng Phng
0
3.281
17.056
20.336
(HUS, VNU)
1
5.633
1.582
4.051
2
16.834
5.558
11.275
3
26.748
9.068
17.680
1.0
2.0
3.0
4.0
5.0
6.0
4
12.396
1.777
14.172
October 2014
41 / 66
Phn loi hoa Iris
L Hng Phng
(HUS, VNU)
S bc
30
17
19
16
14
20
KT
100%
100%
100%
100%
100%
100%
HL
97.69%
96.92%
97.69%
96.92%
96.15%
96.15%
October 2014
42 / 66
Phn loi hoa Iris
Ta thy m hnh c hiu chnh t chnh xc cao nht l

97.69% khi = 3.0.
Cc tham s ca m hnh khi l:
Lp
0
1
2
setosa
0.236
0.546
1.279
versicolor
0.244
0.560
1.279
virginica
0.507 0.954 1.083
3
1.813
1.813
1.810
Ta thy khi dng phng php hiu chnh L2 , cc tham s ca m

hnh c gi tr tuyt i b hn nhiu gi tr tuyt i ca cc
tham s trong m hnh khng hiu chnh; ng thi cc tham s
phn b xung quanh gi tr 0, ph hp vi kho st l thuyt.2 .
4
0.847
0.847
1.337
S bc lp ca thut ton ti u L-BFGS cng ph thuc vo s

hng hiu chnh .
Vi cc tham s nh, vic hiu chnh cng gip gim thiu kh nng trn s
khi ci t m hnh entropy cc i.
L Hng Phng
(HUS, VNU)
October 2014
43 / 66
L Hng Phng
(HUS, VNU)
October 2014
44 / 66
So snh chnh xc
So snh chnh xc
chnh xc ca mt s m hnh phn loi trn tp d liu Iris:

M hnh
Chun mt chiu (dng ring j )
Chun mt chiu (dng ring j )
GDA
MaxEnt L-BFGS
MaxEnt L-BFGS, L2
(HUS, VNU)
HL
100.00%
85.00%
100.00%
100.00%
100.00%
95.38%
87.69%
97.69%
98.46%
97.69%
October 2014
Cc c trng trong tp d liu l kch thc ca cc phn t t

nhin (l v cnh hoa) nn thng tun theo phn phi chun.
M hnh GDA vi gi nh d liu phn phi chun t ra m hnh
ho d liu tt.
45 / 66
October 2014
46 / 66
1.0
0.6
Density
0.2
0.0
0.0
2.0
Sepal Length
L Hng Phng
0.8
0.3
0.2
0.1
Density
(HUS, VNU)
L i, cnh hoa v phn phi chun
0.4
L Hng Phng
0.4
L Hng Phng
KT
Ta thy m hnh MaxEnt cho kt qu tt nht trn c tp kim

tra v tp hun luyn.
(HUS, VNU)
2.5
3.0
3.5
4.0
4.5
Sepal Width
October 2014
47 / 66
L Hng Phng
(HUS, VNU)
October 2014
48 / 66
0.3
0.0
0.00
0.1
0.2
Density
0.15
0.10
0.05
Density
0.20
0.4
0.25
0.5
Petal Length
L Hng Phng
(HUS, VNU)
Petal Width
October 2014
49 / 66
Ni dung
L Hng Phng
(HUS, VNU)
October 2014
50 / 66
October 2014
52 / 66
Ni dung
Gii thiu
Gii thiu

c lng tham s
Hiu chnh tham s

c lng tham s
Hiu chnh tham s
V d
Phn loi nh
Phn loi hoa Iris
V d
Phn loi nh
Phn loi hoa Iris

Hm c trng

Hm c trng
Bi tp
Bi tp
L Hng Phng
(HUS, VNU)
October 2014
51 / 66
L Hng Phng
(HUS, VNU)
Hm c trng
Hm c trng
Ta tng qut ho m hnh entropy cc i mc trc vi vic s

dng cc hm c trng.
Gi s X v Y l cc bin ngu nhin xc nh tng ng trn cc

tp X v Y.
Vic s dng hm c trng cho php biu din ngn gn tp d

liu quan st (cc i tng xi v lp yi ) v tng qut ho m
hnh.
ngn gn k hiu, vi (x, y) (X , Y), ta vit P (X = x |Y = y)

n gin l P (x |y).
Ta nh ngha hm c trng f nh sau:
f : X Y RD .
L Hng Phng
(HUS, VNU)
October 2014
53 / 66
Hm c trng
L Hng Phng
(HUS, VNU)
October 2014
54 / 66
Hm c trng
Vi mi (x, y) (X , Y), f (x, y) l mt vc-t D chiu ng vi D

c trng:
Tng qut, mi hm c trng c nh ngha bi:

fj (x, y) = Aa (x)Bb (y),
f (x, y) = (f1 (x, y), f2 (x, y), . . . , fD (x, y)).

Cc hm c trng thnh phn fj (x, y) R, tuy nhin trong m
hnh entropy cc i chng thng nhn gi tr nh phn thng
qua mt hm ch s no ca x v y.
L Hng Phng
(HUS, VNU)
October 2014
55 / 66
trong ch s di a nh s mt tp hm xc nh trn x, ch
s di b nh s mt tp hm xc nh trn y.
Nu cc hm ny l hm nh phn xc nh vic c hay khng c
mt tnh cht no ca x v y th tch Aa (x)Bb (y) l mt dng
hi logic.
L Hng Phng
(HUS, VNU)
October 2014
56 / 66
Hm c trng
Hm c trng
Ta c th xc nh mi thng tin hu ch cho vic phn loi bng

cc hm c trng tng ng xc nh trn cc lp y v cc thuc
tnh xi ca x.
Cc hm ny khng nht thit phi c lp nhau.
V d, nu x l mt t, ta c th xy dng cc hm c trng
khai thc cc thng tin ca x nh:
Cc gi tr ca hm c trng ny thng c trch rt t ng

t cc mu c trng tng ng.
Trong cc m hnh entropy cc i ng dng trong hc my, s
chiu D ca mi vc-t c trng l ln, c th t hng trm ngn
ti hng triu c trng.
Ta k hiu RD l vc-t tham s ca m hnh.
A1 (x) = (x bt u bng mt ch ci in hoa)

A2 (x) = (x bt u bng T)
A3 (x) = (x l Thomson)
A4 (x) = (x c 7 ch ci)
L Hng Phng
(HUS, VNU)
October 2014
57 / 66
Ni dung

c lng tham s
Hiu chnh tham s
October 2014
58 / 66
Xc sut ca mi lp c xc nh bi
Gii thiu
(HUS, VNU)
L Hng Phng
P (y| x; ) = P
exp(T f (x, y))

.
T
y Y exp( f (x, y ))
(7)
Mu s ca xc sut ny chnh l s hng chun ho

X
Z() =
exp(T f (x, y )),
V d
Phn loi nh
Phn loi hoa Iris
y Y
m bo phn phi xc sut:

X
P (y| x; ) = 1, x X .

Hm c trng
yY
Bi tp
L Hng Phng
(HUS, VNU)
October 2014
59 / 66
L Hng Phng
(HUS, VNU)
October 2014
60 / 66
Cho trc mu hun luyn (xi , yi ), i = 1, 2, . . . , N , trung bnh log-hp

l ca d liu l:
Ta c, j = 1, 2, . . . , D:
N
1 X
log P (yi | xi ; )
() =
N
i=1
N
X
X
1
T f (xi , yi ) log
=
exp(T f (xi , y)) .
N
i=1
L Hng Phng
(HUS, VNU)
yY
October 2014
61 / 66
1 X
X
1
() =
fj (xi , yi ) P
exp( T f (xi , y))
T
j
N i=1
yY exp( f (xi , y)) j
yY
N
X
1 X
1
=
exp( T f (xi , y))fj (xi , y)
fj (xi , yi ) P
T
N i=1
yY exp( f (xi , y)) yY
N
X
exp( T f (xi , y))
1 X
P
fj (xi , y)
fj (xi , yi )
=
T
N i=1
yY exp( f (xi , y))
yY
N
X
1 X
=
fj (xi , yi )
P (y| xi ; )fj (xi , y)
N i=1
yY
L Hng Phng
(HUS, VNU)
October 2014
62 / 66
Ta thy:
1 PN
Khi p dng m hnh entropy cc i vi dng hiu chnh L2 , ta c

hm mc tiu cn cc tiu ho v cc o hm ring ca n l:
N
X
X
1
T f (xi , yi ) log
exp(T f (xi , y)) + T
J2 () =
N
2
i=1
yY
N
X
X
1
fj (xi , yi )
J2 () =
P (y| xi ; )fj (xi , y) + j .
j
N
b
i=1 fj (xi , yi ) = E[fj (x, y)] l k vng mu ca c trng th j
N
trn tp hun luyn;
1 PN P
i=1
yY P (y| xi ; )fj (xi , y) = E[fj (x, y)] l gi tr k vng
N
ca c trng th j theo phn phi xc sut ca m hnh.
Nh vy, vic tm gn vi vic gii h phng trnh:
b j (x, y)] = E[fj (x, y)], j = 1, 2, . . . , D.

E[f
i=1
Ni cch khc, ta cn tm m hnh trong k vng ca mi c trng

j khp vi gi tr ca n trn tp hun luyn.
L Hng Phng
(HUS, VNU)
October 2014
63 / 66
L Hng Phng
(HUS, VNU)
yY
October 2014
64 / 66
Ni dung
Bi tp
Ci t cc thut ton c lng tham s ca m hnh hi quy logistic
a lp:
Gii thiu

c lng tham s
Hiu chnh tham s
V d
Phn loi nh
Phn loi hoa Iris

Hm c trng
Bi tp
(HUS, VNU)
Thut ton gim gradient theo lot
Thut ton gim gradient ngu nhin
Thut ton Newton
Chy cc thut ton trn cc d liu th nghim v thng bo kt qu.
L Hng Phng
October 2014
65 / 66
L Hng Phng
(HUS, VNU)
October 2014
66 / 66

8 MaxEnt Handout

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

8 MaxEnt Handout

Uploaded by

Copyright:

Available Formats

Ni dung

multinomial logistic regression

Classification and Prediction

M hnh hi quy logistic a lp

M hnh hi quy logistic a lp vi hm c trng

Multinomial Logistic Regression

Multinomial Logistic Regression

M hnh hi quy logistic a lp

Khi bi ton phn loi c nhiu lp, ta c th m rng m hnh

M hnh hi quy logistic a lp

M hnh hi quy logistic a lp cn c gi l m hnh entropy

M hnh hi quy logistic a lp vi hm c trng

Multinomial Logistic Regression

Multinomial Logistic Regression

M hnh hi quy logistic a lp

M hnh entropy cc i c pht minh nhiu ln, trong nhiu lnh

M hnh hi quy logistic a lp

M hnh hi quy logistic a lp vi hm c trng

Trong thng k ton hc di tn hi quy logistic;

Multinomial Logistic Regression

M hnh hi quy logistic a lp

trong Z l s hng chun ho m bo phn phi xc sut:

Multinomial Logistic Regression

M hnh hi quy logistic a lp

Xc sut i tng x thuc lp k {1, 2, . . . , K} c m hnh bi:

Multinomial Logistic Regression

Tham s k = (k0 , k1 , . . . , kD )T l mt vc-t tham s D + 1

Multinomial Logistic Regression

M hnh hi quy logistic a lp

M hnh hi quy logistic a lp

M hnh hi quy logistic a lp vi hm c trng

nn ta ch cn c lng (K 1) vc-t tham s k .

Multinomial Logistic Regression

Trung bnh ca log-hp l ca tp d liu hun luyn l:

Multinomial Logistic Regression

Multinomial Logistic Regression

c lng cc tham s ca m hnh, ta cn tm cc tiu ho

trong R() l s hng hiu chnh dng trnh hin tng qu

Cng thc tnh xc sut i tng x thuc lp y trong m hnh

Multinomial Logistic Regression

Hiu chnh dng L1

M hnh hi quy logistic a lp

Ch rng hm mc tiu J1 khng phi l hm li nn nghim ti u

M hnh hi quy logistic a lp vi hm c trng

Multinomial Logistic Regression

Hiu chnh dng L2

D thy hm mc tiu J2 l hm li nn ta c th dng cc thut ton

Multinomial Logistic Regression

Hiu chnh dng L2

Hiu chnh dng L2 l mt hm ton phng, hm mc tiu l:

Multinomial Logistic Regression

Kiu hiu chnh L2 tng ng vi vic gi nh rng cc tham

Multinomial Logistic Regression

Hiu chnh dng L2

Hiu chnh dng L2

Theo cng thc Bayes:

Nu vit di dng cc log xc sut:

P (|(xi , yi )) P ((xi , yi )|)P (),

trong P () l xc sut tin nghim ca tham s.

v y chnh l dng hiu chnh L2 .

Multinomial Logistic Regression

Multinomial Logistic Regression

Multinomial Logistic Regression

Multinomial Logistic Regression