Professional Documents
Culture Documents
Boltzmann Machine Learning With Latent Maximum Entropy Principle
Boltzmann Machine Learning With Latent Maximum Entropy Principle
!"#
$%&
' $() +* ,-
%#.0/213
4.5
st]WH<«L5Q0]Wuå¸ÛWÜMÜaÝM¬5
uKQY <TaEM 5u]aOZOPI]auOPQZ]mQZIpOPQYIO³IpO X:T0VTaNZ³Eu]mQZLQZ EL © «ªCENZIKH<IK<uKLM[EuKLQT
"_L]0N]WHE¤
EL@]iJ>IpupR<]aOYL4`
TaH<]WN]W0LQYNZIp0Tv`ELuO[>OP< 0]WH0]W<ENYT>]a `TM Jm]WNZIp]aEupL`LHETaQYIpHEQY <Lr5TM<uKLQYLr`E]WQZ]<[)L
]aH)OYTa0LQYIpL@OÂLTmJaLNZupRUNYL@OSQZNYI®QZIKJML$]aH<`)ENYL4JaL4HMQ
]M5E¤ QY <L$TM<OYLNZJaL4`rIKH_5Ta0EupL5QZL`E]m¨ Q]
]aH<`^{½LQZ E¨tL0IpOZOP¤
N]mQYL¸0Tv`ELuOVNZTa LIpHE© ¼`ELJaL4uKTM_L@`jXÑ EL¸]auqQZLNZH<]mQZIKJML IpHEn`E]mQ]EÖXÑ <]WQIpO4.[ J0¬®yÕâV$^LuKLDQ ¬U]aH<`
ENZIKH_5IpEuKL^LENYTM_T>OPLM[ ǻ
[#IOU]fHETaH¤â<]aNZ]aLQYNZIp)]WE¤ t¨ ¬#`LH<TWQYL
QY EL&`ELH<OYIqQZIKL@OTaM V ß]aH<(` -NZL4OY_L@®QYIpJaL4uKRM[M]aH<`
E¨ NZTM]M _]aOYL4`TaHUU]mQ EIpHE]OYL5QTaVVL@]mQZENYL@OIpH0QY EL$`E]mQ] upL56Q ¨tMt¨ ¬`L4HETWQZLÂQY <i¨ L 5TMH<t¨ `%IK QYIpTaH<]au_`L4H<OYi¨ Iq QSRTW6V Ci¨ MIKJMt¨ LM(H [
Iå LaOP UIKL4H>Q$OPQZ]mQZIpOPQYIO4[ÔL@]W¶upL4]WNZHEL4NZO4[<TMN<]MOPIO&VEH<®¤ QY <LH $¬ -* g¬ ¬^ ELNZ.L e¬ i¬ ¬5
QYIpTaH_OZ¬5¹XÑ EIOQZL4 EHEIÐ>ELrL4Ta0L4O³<]WN]W0LQYNZIpU© <LH£^L e¡ km|}A:gM| ced Ò$IKJMLHVL4]WQYENZL4E O ¢ >£W£W£ ¢9¤
[_OYL45IKVRvIKH<
HEL@5L4OZOZ]WNZIKupR³ <]iJMLÑQYT³]aEENZTi¦IKU]mQZL^QY ELENYIpH<IKE¨ upLa ¹ª£IpO QY <Lf<NYTM_L4NPQZIKL@Or^L¸^Ta<up`yuKIp¶aL¹QYTÏ0 ]WQZ ÙIKHÙQY EL£`E]WQZ]<[
]WHrL5¦vQYL4H<OPIpTaHQYT
°>]iRvHEL4O4²aU]W¦vIp³<½L4H>QYNZTavR «ª¬ENZIKHE¤ OYLupL4®Q#]ãSTaIpHMQ#ENZTa_]WEIpuKIKQSRÃ0Tv`EL9u M¥VNZTa-QZ ELÂOP<]M5L^TWV_]auKu
5IpEupLQZ <]mQL5¦EupIpIqQZuKRÃIpH<5TMNYTaN]mQZL4O8up]WQYLH>Q%Ji]aNYI]W<uKL@OhIpHQY <L `IOSQZNYIpEQZIKTMH<NO ¦¡TmJaLqN ÀQYT0U]m¦Ip0IK4LQZ ELÑãSTaIpH>Q
LH>QYNZTavR
VTaNZ³<up]WQYIpTaHh[m]WH_`ÃQY EL4NYL4vR$L¦vQYLH_`EOhQZ EL^TaNZIKMIKH<]auaENZIpH<5IpEupL
QYTº]aOYL4O&Â ELNZL³`E]WQZ]Ta0TaHEL4HMQO&]aNYL0IOYOYIKH<<
XÑ EL
NYL¤
OYEuqQZIKH<&ENZIKH_5IpEuKLIpO%`IK¯L4NYL4H>Q8VNZTa¾_TaQY ³U]W¦vIp³<¾upIp¶aLupIK¤ § ¨ Ô$ ¬ {¨
©7ª « i¨ ¬uKTM[ t¨ g¬ U¨ ¬ ¨ Ìi¬
ETvT`rL@OSQZIKU]WQYIpTaH]WH<`)OSQ]WH<`<]WN`U]m¦Ip³EL4H>QYNZTavRa[vEQ
TWVäQZLHºR>IpLu`EO_LQPQZLNÑL4OPQYIpU]mQYL@OIpHQZ EL$ENYL@OPL4H<5L&TWV8 EI`E`L4H OYEãSL45QÂQYTQY <L5TaH_OSQZNZ]aIKH>QZO
Jm]WNZIp]aEupL4OÑ]WH<`)upIK0IKQYL@`rQZNZ]aIKH<IKHEU`E]WQZ]<
ÕÓHQZ EIpO:<]a_L4N:^L`L4TMH<OPQYN]mQYLQY EL © «ª¹ENYIpH<IKEupLvRÃ`L5¤ ©7ª « ¢9¬ i¨ 9¬ i¨ G ¬ U¨ e¬ Z $¯ t¨ ²¬ © - ¢C¬ i¨ @¬ U¨ ³ g¬ t¨ % ¬
NZIKJvIpHE
H<LÙ]auKMTaNZIqQZ EUOVTaNÂ^TauKQYU]aHEHºU]M EIKH<L<]aNZ]a¤
L5QZLN0L4OPQYIpU]mQZIKTMHh[]WH<`¼OP <Tm ETm]¹NZTa<<OSQ0]WH<`NZ]aEI`uKR ® °^±
´ §Ì £K£W£ µ£[ ¡]WH< ` ÙHETaQÑIpH<`L4_L4H<`L4HMQ ¨ ÛM¬
5TMHvJaLNZaL4H>QhHEL£Jm]WNZI]WH>QhTWVQZ ELª]WupaTMNYIKQY E-]aH_L^`L5¤
JaL4uKTM© _L@`jÁENL¦_L4NYIp0LH>QZOOP ETmÏQZ <]mQ^L4OPQYIp0]WQYIpTaH_]aOYL4`  EL4NYL i¨ ¬5$à&LNZL t¨ ¬ÂIO&QZ ELL0EIpNYI]au:`IOSQZNYIpE¤
TaH «ªtaL4HELN]WupuKR³R>IpLu`EO_LQPQYL4NNZL4OYEuKQZOQZ <]WH0U]m¦Ip³E QYIpTaHTmJMLNÑQZ ELTa_OPL4NYJML4``E± ]mQ]E[<]aH<` D`L4HETWQZL4O
QY EL³OYL5QTWV
upIK¶MLupIK ETvT`ºL4OPQYIp0]WQYIpTaHh[_]WNYQYI5Eu]WNZuKR0Â EL4HIKHVL4NYNZIpHE
<Ip`v¤ Ta_OPL4NYJML4O` ¡Jm]auKEL@OÕÓH>QY<IqQZIKJMLupRa[QZ EL5TMH<OSQZNZ]aIKH>QOÂOPL4IqVR
`L4H)<HEIqQOÑVNZTaÀOYU]Wupuh]W0TaEH>QOÑTWV#`E]mQ]E QY _]mQ
^L$NZL4Ð>EIpNZLQZ ELÃL5¦L4®Q]mQZIKTMH<OÂTW± GV ¢9¬ ¨ ¸¬^IpHºQZ EL5TM¤
XÑ EL«NZLU]aIKH<`ELNUTWVÃQZ EL«<]a_L4NIpOrTaNZM]aHEIK4L4`Ï]aO0VTMuKupTm
O4 EupL5QZL
0T`Lu%QZTrU]mQ «QY <LIpNL4<IKNZIp4]Wu%L5¦L4®Q]mQZIKTMH<O&TMH
á%IKNOPQ4[hIpHtL4®QZIKTMHtÛº^L
IpH>QYNZT`<5LQZ EL0up]WQYLH>QU]m¦Ip³E QY <LIKH<Ta0EupL5QZLÃ`<]mQZ] r[vQZ]W¶vIpHE0IKH>QZT0]MTaEH>QÑQY ELOPQYNZ<5¤
LH>QZNYTM>RºENZIKH<IK<uKLM[]aH<`QY <LH«TMQYupIKH<L
]UMLHEL4NZ]aujQYN]WIpHEIpHE QY<NYLTWVQZ EL0`LLH<`ELH<LTaV#QY EL0EH<Ta<OYLNZJaL@`5Ta0TaHEL4H>Q
]WupaTMNYIKQY <'VTaNQZ EIOENZIKH<IK<uKLUIpHnvL45QYIpTaH£ÝE)ÁH_5L0QY <L4OYL ÙTa¶ H U
ENZLupIK0IpH<]WNZIpL4O³]aNYLrIKHn<up]M5La[#^LQZ ELHENYL@OPL4H>Q³QY <LrU]WIpH G
¨ HEVTaNYQYEH<]WQYL4uKRM[
QY <LNZL¸IOrH<TÏOYIK0EupLtOPTMuKQZIKTMH×VTaN M¥¹IpH
5TMH>QYNZIKEEQYIpTaHfTWVQZ EIOÃ<]WLNPÞ×])HEL4·QZNZ]aIKHEIpHE]WupaTMNYIKQY E ÌM[ ÛM¬®%à
Tm^LJaL4N4[i]aTvT`]WEENZTi¦IKU]WQYIpTaH]aHLTMQZ]aIKHEL@`
VTaNTMuqQZU]WH<H
U]a <IKHELÂL@OSQZIKU]WQYIpTaHvÞ×IKHvL@®QZIKTMH _L4®¤ vRNZL4OPQYNZIp5QYIpHEQY EL0T`LuhQYT0 <]iJaL]aHºL¦vTaH<LH>QYI]WuÔVTMNY
QYIpTaH«Ú5TM0<]WNZL4O^QZ ELHEL¾<]aNZ]aLQYL4NÂTaQZIK0Ip4]mQZIKTMHº<NYTa¤
5L@`ENZLQYT&OPQZ]aH<`E]aNZ`ENZTvL4`<NYL@O[@]WH<`vL@®QZIKTMH &5Ta0<]aNYL@O ¨
¤
¬ ¹T·º L5¦x » ¬W¼ ¬ ¢ ¬ i¨ 5¬ ¾
QY <L³aL4HELN]WupIp4]mQZIKTMH_L4NPVTMNYU]WH_5LTaV#QY EL
HEL§L@OSQZIKU]mQZIKTMH ³· ¸ i
ENZIKH_5IpEuKLºQYTtOPQZ]WH_`E]WN`£U]m¦Ip³EupIK¶MLupIK <T>T`jtsfL5TaHE¤ 6½
5up<`LÃQZ EL<]a_L4NÂÂIqQZ ]ENZIpL5V#`IpOZ5_OYOYIKTMHºIpH«vL45QYIpTa H v  EL4NYL¿¹.·
À ª « L¦vOÁ9 ¤¬K¼ ¬ ¢ ª ¬ i¨ ¬Ã t¨ ¬IpO]H<TaNY¤
U]WupIK4IKH<r5TMH<OSQ]WH>QÂQY _]mQL4H<OYENYL@½OT « M· ¨t ¬J ¨t ¬=3Ìa
XÑ EIO&NZL4OPQYNZI®QYIpTaH<NYTmJvI`L4O
]VNYL4L³<]aNZ]a0L5QYL4N ¬VTMNL@]a
VL4]WQYENZLUVEH_®QYIpTaH ¢9¬Sf^Rf]M`TaQZIKH<¹OP_ Å] ÄP½ upTaa¤åupIpHEL4]aN Æ
NZL4OPQYNZIp5QYIpTaHh[IKQÂQYENZH<OÑTMQÑQY <]WQ
L]aHrVTMNY
Eu]mQYL]EN]a5¤
!#"$%&'(")*,+-
./0&
/132-% 454.+ QYI]au%]auKMTaNZIqQZ E»VTaNÃ]WEENZTi¦IKU]WQYLupRºOZ]mQZIpOPVRvIKH<QY <L © «ª
%4672-#'8!946
72:
)&02-&0;%2:
+<*2=<>20+9
6+??@04645;%AA+%<
:?@ ) B#+% +C
2>/1DD2>+C1E&0-+F7<-;2:
-+%4G>HI
JK0+A?-2>/L
ENZIKH_5IpEuKLM
2-040!M2>+73
;4N#0& A04=+9
52-&
2>%+OK2:<A+
P")-QC40RTSU
Ç È y8TÆ y E z E#Ê Æ Ê Æ È½ z ÈÇ_ÊÓ w | ÊÇ
É
4.
% 'HV-*;%45;72>13A2-&
&82>M
# 5HW-X B2>&
=<-2-0+9
2-45& 0+9
N+
045=AC040!72-+7Y45A.HV>Z>H[2-8BD2:
.+L
HV'0+%&0E\V]^0+%<E2-+%,_I0Y`-aa-b@c)>ed+9
=fg2:*04h
D2>
-+
Ç ÈgË z Ê qÆ y Ì |½ÇÂÈ y6z
z$
\iJ&QC 1O
T2-jR3k8l-m@nc=4E+088O
2-%8BD2:
*
%A<>2/L X:Tº`L4NYIpJaL]rENZ]M®QZIp4]WuhQZNZ]aIKH<IKHEº]WupaTaNZIKQY E»VTaNuKTMW¤âuKIpHEL4]aN
0+9
8Rpo%>p
%G%;#4504[-H#
46%2-# 8!@")G45A%/12>4545;%Ag
%2>
0T`LuO[^© LUL5¦EupTaIKQQY <LUVTaupuKTmÂIpHEIpH>QYIp0]WQYL)5TaH<HEL45QYIpTaH
45AE45-5
q>H)82>45+72>%D2-8BD2>
+3
0&%+%r9;%T4q28?2>/L
¨ L5QS© ^LLH «ª]aH<`U]m¦Ip³EupIK¶MLupIK <T>T`¡L@OSQZIKU]mQZIKTMH
2-R)s0A\tsC;
1,
e2-jRg`-a-ab@cgHV>.2>+(2>
5
A
g
A>?
&02-4545&82>M04h
D2>
>4GHV-e
%4$-%0R ª¬5
1(E& F7% 323
dv G4;<
5km?dv
l- 6H!I 70J 3$'8K?
?9 : ;L
M= <>
NO!
?
PH!<#Q? #?4 61"&@
!#L!%9 M$'
<3&) A(?!B!%
*++R3C,-
PEQ1&.D#D#?&1/0L upTaHEMLNÑ`LLH<`<OÑTaH EEQÑN]mQZ ELN^TaHrQY EL$ØE¦L@`r5TMH<OPQZ]WH>QO
VNZTa QY EL¼ENYL4¨ JvIKTM<O½ IKQYLN]mQZIKTMH ¨ }~ XÑ EIO¹0L@]WH<O«QY <]WQ
U]m¦IpIpIpHE >¬OYEãSL45QQYT Úa½ ¬ÂIKQY -NZL4OY_L@®QQYT IpO
!%
P
*E?D
*L+
*5!S=R#<.
*
\] # @ ^TMUUVWT_V`(%?ED (ETM 9UXWVaYLb?
GH!#,I8
E Z #??L&@8Q[9
? H!%A! HETm]¸5TaHvJML5¦tTaQZIK0Ip4]mQZIKTMHENZTa<uKL4±ÂIKQY ¼uKIpHEL@]WN05½ TaHE¤
OP¨ QYN]WIpHMQ+¨ Ov CXÑ ELMLHEL4NZ]auKIpL@`£IqQZLN]mQZIKJMLOZ]auKIpHEt]WupaTMNYIKQY E
!UH!I 3FT_
VD
*!-
+!3,c
?L #&. 1D
+?&@!49@
[9MD#\i&@?d9
&.>
l/0\i 1&@&.
Z5eA<3/0
#&1" [!%$'+&@R&j: G/0 !L&m
P$'$'#
*&n(E+Q",k!N70\f" 3,gTh LU: #9"V ! Ò$ÕP<¬ ]WNZNYT ³LQ]WuåÌ@%Í aÛa¬jTMN#IK0ENZTmJaL@`IKQYL4NZ]WQYIpJaLÑOY4]WuK¤
IpHE]WupaTaNZIKQY E ¨ ÕPÕPE¬ ¨¨*v L4uKu}Ap] ~ IpL5QYN]ºL5Q³]Wuå)Ì4Ía7Í W¬&]WHfL
<OYL4`ºQYT0U]m¦Ip0IK4OL x ¬^JMLNZRU:L 5IpLH>QYupRa
e"opqZ 3
k?Xsr4t
t
u%f
X áENZTaßQY EL@OPLUTM<OPL4NYJm]WQYIp½ TaH<½ O4[jLU4]WHtNYL@5TmJMLNVL4]aOYIpEuKL0upTaa¤
XѨ+v EIOÑ5TaH<HEL45QYIpTaHº]auKupTm
O<OQYTL5¦EupTaIKQÑ]aHºª¿]WupaTMNYIKQY E upIKHEL@]WN%0Tv`ELuO8vRÃ<OYIKHE$]WH]WupaTaNZIKQY EÙQY <]WQTa
EIKH<L4O8ª
ÂIKQY HEL4OPQYL@`)IqQZLN]mQYIpJaLOZ]auKIpHE
QYT]aupEu]mQYL$QZ EL¿OPQYL4h
© ¹Lª¡0<EOPNYQYIpL4H<NLIKEQup]WLauå-ÑÌ4ÕâQr%Í IWO¬8IpQY0T_Ø<TMH<NP`Q]W<3H> #QU"!%QY+TR L0OPTMEuKE <QY]MIpOPTaIpH<LMO[^QZT <TmQYÑ <L ¤ * 6¼9Ec}F_;_km| @7: l
LJMLN@[QY <]WQêÂIKupuTaHEupRØ<H<`¸]WuKQYL4NYH<]WQYIpJaLVL@]aOYIK<uKL0OPTMuKE¤
QYIpTaH_O[# EIpupLQY <L © ¹ª·]WH<` © ª§<NYIpH<5IpEupL4O
ÂIKupuÑ`Iq¯ÔLN ]aupE:u]mQY¥³LrÒ$IqQIKO³JML5H<NYNZL}~H> Q[hL5VTM¦NL4L45]MQZ]m ¹QZIKVTML@{H ]mQY <}ANY~3L ¢9¬PIK[ QY £´ "NZL4OYÌ _>L@£W£K®£WQ'µnQZT [
U]WNZ¶aL@`uKRIpH«QZ EL³VL@]aOYIK<uKL0OPTMuKEQYIpTaH<O&QY EL4RENZL5VLN@sfL
Ipuq¤
up<OSQZNZ]WQYL$QY <IpO
`EIpOPQYIpH<®QZIKTMH_L4uKTm }A~ >RÔ¥ ½ ¬
X:TVTaNZ³<up]WQYL³]aH¹ª]auKMTaNZIqQZ E»VTaN&upL4]aNYH<IKHEruKTMW¤âuKIpHEL4]aN
0¨ T`LuO[Ø<NZOPQn`L4Ta0TMOYL¼QZ EL×upTaW¤âupIK¶MLupIK ETvT`DVEH<®QZIKTMH
½
¬}A~ $¯ i¨ ¬ © - ¢ ¬ i¨ @¬ ·? P U¨ ³ 7¬ t¨ % ¬ ¨ >¬
w ¬^IpH>QYT ® ° ±
OYEãSL@®QÑQYT © ª « ¢ ¬ ¨i ¬@³· ¨t ¬g ¨U ¬ ¨ ÚM¬ XÑ v<O4[8ª¤âÕP<NYTmJvI`L4Oé ]WH¸L5¯ÔL45QYIpJaL00L4]W¨H_O&QZT)Ø<H<`¸VL4]W¤
OY© IKEupLºOYTaupQZIK¨TMH<OQZTQY EL «ªDENZIKH<IK<uKLM sfLrHETaQYLrQY <]WQ
t¨ I ¬ © - ¢9¬ ¨t ¬@ · * ¨tM ¬) ¨U ¬8 ´ §Ì£K£W£ µ E
]aENZIqQZLH Ì4ÍMÍMÚM¬0 <]aO)OYEaML4OPQYL4`y]OYIK0Ipup]aN)]auKMTaNZIqQZ E[
e¯
® ° ± ÅÂ]WupIpLLQ#H>4Q0`uKL4IQY`N TtHE¨ Ì4OYTaÍML5Q%QPÍaOYQZÍMEIKH<¬&E{ <Eup]M¡RO$]&]aupOYTaT)ÌHvJaEIKL4HCNYNYTMM_ªLT>H<OPL@¤âL#ÕP`¹ÔE]a#NYHfTvà
TaTm]aVS^:uKMLTaJMNZTMLIqQZNYN@ EL['NY^L@Lº5L4LÐ> <H>E]iQZIpuKJMJ>RML ¤ [
ÕâQIOU5NZIqQZIp4]Wu^QYT£NZL4]auKIpLºQY <]WQ0QY ELHEL4"TaH<OPQYN]WIpHEL@`nTaE¤
QYIp0IK@]mQZIKTMHENZ¨ TaEupL¢IpH © L0U¨ ]̺IpO³_ nL4]MOPIpLN
QY <]aH VTa<H<` ^ÆÙÌ$QYTULTMNYLÃL¯L@®QYIpJaLÃIpH0]aHvRr]MOPL@O ¬
U]m¦IpIpIpHE Ì@¬UOYEãSL@®QQZT ÛM¬VTaNºuKTMW¤âuKIpHEL@]WNU0¨ T`LuO[ sfL$4]WHHETmCL¦v<uKTMIqQQY ELê¤âÕPU]a© uKMTaNZIqQZ EDQZT
`L4JaL4uKTM]
L4]a<OYL
QZ ELUNZIKM MQà <]aH<`tOPI`L0TWVQY ELTaH<OPQYN]WIpH>QZO ÚM¬&H<T EN]a5QYI]Wuh]aEENZTi¦IKU]mQZIKTMHUQYT0QZ EL ¹ª¼<NYIpH<5IpEupLa
l 6n9Ec}F_;_km|@7:l
*
= E ¥#\ <T>T>OPL$N]WH<`TMIpHEIqQZIp]auhJm]WupEL4O^VTMN
s %¥
ÂÅ EHUª¤åÕPQYTTaHvJaL4NYMLH<La[iQYTÃTMQZ]aIKH
VL4]MOPIpEupL
¥
½
Y 1 YJ
h 0 E ¥\^]aupEu]mQYL$QZ ELLH>QYNZTavRUTaV a
½ I³·
E 6
0 %¥ ÂÅ LL4]WQtQY ELC]WTmJaLÏOSQZL<OtOYLJaL4NZ]au
QYIp0L4O&QYTºENYT`_5L³]OYL5Q$TWV`IOPQYIpH<®QVL4]aOYIpEuKL]aH<`I`E]mQZL4O4
"! # #
Z 1 ZL
\ ETvTMOYL#QZ ELVL@]aOYIKEupL4]WH<`I`E]WQYL#QZ <]mQ]a EIpLJML4OÔQZ EL EIpa <L4OPQ
LH>QZNYTM>RM á%IKMENZLÌa¥^TauKQYU]aHEHUU]a EIpHEL
0T`Luå¥:QY ELHET`L4OJ·]WNZL
XÑ EIOÂupL4]a`<O^QYTU]HEL¾L4OPQYIp© U]mQYIpTaHºQYL@ EHEIÐ>ELÃQY <]WQ
LÃÂIpuKu Ta_OPL4NYJML4`r]aH<`rQY ELHET`L@OE¾]WNZLÃEHETM<OPL4NYJML4`j
5TM0<]WNZL
QZTOPQZ]aH<`E]WN` ªLupTmÁHELÃ]WE<]aNYL4H>Q^5TM¤
EupIp4]mQZIKTMHh[ÑØ<NOSQ@[ÂIpOQY <]WQº^L«HELL@`ÏQYT4]Wu5Eu]mQZLQY <L«LH¤
QY<NYL@*O ¢ = ¨i q¬ = [ ¢ > ¨i q¬ > p[ ¢ >@? ¨t q¬ > ? [
QYNZTa<IKL@OTWVÑQY <Lº]aH<`I`E]mQZLU0Tv`ELuOENZT`<5L@`tvRfª¤åÕPj ]W<uKL@OÃIKHfQY <LUT`L4u})v_L@5IKØ_]auKupRa[:LTaH<OYI`LN$QY <LUVL4]W¤
à
Tm^LJMLN@[WIKQQYENZH<OTaQQY <]WQL`THETWQ^HELL@`QYT³4]Wu5Eu]mQZL V TaNfBÌ ¨i AD C(EG¨i F1AH,· ]aH< `§BÌ AJ IKEJLMA w
[Â ELNZL
LH>QZNYTMEIKL@OL¦v<uKI5IKQYupRC_L@]W_OPL¸TaHELf]WH×NZL4TmJaL4NUQY EL¸LH¤ ¬ >£W£K£W .M -£K£W£W 2j¬®O N
TaQYL0QZ <]mQQZ EL0VL4]W¤
QYNZTavRTWV <3 #"!%+R upTaW¤âupIKHEL@]WN0T`L4upOOPIp0EuKRU]MO]vRvENZTv`¤ *
QY<NYL@O#]WNZL^]auKuEIpH<]aNYRM[a]WH<`³QY <LNZL5VTaNZL^LÂ]aH
NZL<NYL@OPL4HMQ%QY <L
<5Q
TWVNYEH<HEIKH<0ª¤âÕPUQZT0TaHvJaL4NYMLH<La OPQYNZ<®QZENYL&TWVjQY <L&upTaW¤âupIKHEL@]WN0T`L4u_vRU]aN]W< h[>]MOOY ETmÂH
$
;<km;cec}9k % ı< ¥ H! <3 #"!%+R3F %,
x ¨ ¥ ¥4¬
IpHá%IpaENZL0Ìa
¨q§ ¨ ³· ¬ ? w ¨ ½ ¥¬e ¨q§ ¨ M· ¬z§ ¨ ¥C ¥@½ ¬ X ½ Ò$IpJaLHU]ÃOYL4Ð>ELH_5LÂTWVTa<OYLNZJaL@`
`<]mQZ] i¨ -£K£W£W
8 ¬5[a^L
VTaNZ³<up]WQYL$QY EL © «ªÏENZIKH<IK<uKL]MO ±
XÑ EL4NYLVTaNZLa[]mQ
]V½ L@]aOYIK<uKLOYTaup¨ QYIpTaH ¥a[½ L _½ ]iJaL]WupNZL4]a`ER UP } ]Wª ¦ § ¨ ¸$¬ § ¨ ³_¬ z}§ ¨
¬ ¨ Í>¬
]aupEu]mQYL@`¸QY ELºLH>QYNZTavRaJ[ ¨lx ¥ ¥½ ¬5[IKHtQZ ELº OPQYL4£TWV
ª¤åÕPÔ OYEãSL45QÂQYT
¬ Z = i¨ ¬
½ ½
X:T`ENZ]i½]5upL4]WN`IOPQYIpH<®QZIKTMH£¨ _LQSL4LH © «ª§]aH<`¸ © ª
[ ª « = i¨ J
]aOZOYE0LÖQY _]mQQZ ELÙQYLNZ § ¥ ¥¬nVNZTa \TaNZTaupup]aNYRÀÌ
IOº© TaH<OPQZ]aHMQ)]a5NZTMOZOU`IK¯L4NYL4H>¨ Qr½ VL4]M½ OPIpEupL¹OYTaupQYIpTaH<O4DXÑ EL4H ® °
e¯
ª
[j EI ¸U]W¦vIp0IK4L4GO w ¥@¬5[hÂIKupu ETvT>OPL³QY ELU0T`Lu
±
i¨ J ¬ t¨ ¬ t¨ M ¬
QY _]mQ0 <]aO¨ upTmL@OSQLH>QYNZTavRa[#½ <LNZL4]aO © ¹ª
[Â EI U]m¦IK¤ ª
« >
Z
e¯ >#QRTS >
0IK4L4AO § ¨ ³·@¬®[%ÂIKupu ET>¨ OPLU])0Tv`ELu#QZ <]mQ <]MOÃuKL@]aOPQuKIp¶aL¤ ® ° ± <UWV
upIK ETvT`j Á&V5TMENOPLMM[ § ¥C ¥4¬ÂIKupuHETWQ© LTaH<OPQZ]a© H>Q$IpH ª « > ? i¨ J ¬ -#QRXS > ? U¨ ¬
EN]a5QYI5L³]aH<`)QY EL5TM_]W½ NZIpOYTa½ HL5QS^LLH¸ ªÖ]aH<` «ª
VTaNYÌ AZC[E\FA],¸]WH_`¸YÌ A^I_E^L`Abw
WU V
IOÑHETWQ&OYTOPQYN]WIpa >QYVTaNZ^]aNZ`j[>EQÂQZ EIOÂL5¦E]W0© EupLÃ`TvL4OÂ EIpa© E¤
upIKM MQQZ ELIpN`IK¯L4NYL4H<5LM ¬CXÑ <L0Ve]a®QQZ <]mQ «ª-]WH<`t ª  EL4NYL t¨ g¬ i¨ >£W£W£K . >£W£K£W 2 ¬^]WH<` i¨ $¬ 8
]WNZL«`IK¯L4NYL4H>QºN]WIOPL@OQY ELfÐM<L4OPQYIpTaH×TWVÂ EI ×0L5QZ ET`×IpO
QY <L
0TMOPQL5¯ÔL45QYIpJaL
 EL4HUIKHEVLNZNYIpHE]Ã0T`Lu<VNYTM3OZ]W0EupL
>]WIpHUL$]aH]WEEupR0ª¤âÕPQYTØ<H<`]VL4]aOYIpEuKL&±uKTMW¤âuKIpHEL4]aN
`E]WQZ]E 0T`LuånX8T¸L5¦L4QYLºQY <L)ª½¨ OSQZLh[]Wu5<up]WQYLQZ ELºVL4]mQZENZL
L5¦L45QZ]mQZIKTMH<OÂ]a45TMNZ`IpHE
QYT >¬
T z ÊPÔ%EÊSÇ Æ (Ç 'tÇ z
)p|3 ÆÆ |½# w Ê qÆ y
&
v<Ê Æ Ê Æ È = }A~S Ì 8 9 =9
5
9¼
\TaH_OPI`LN^]MNZ]aE EI]Wu_0Tv`ELuÂIqQZ+ *µEIpH<]WNZR0HET`L4OQZ]a¶>¤
IpHECTaH-Jm]WupEL4OIpH ¶ Ü 4Ì ¹M &OZOYE0LtQY <]WQ¹]W0¨ TaHE¼QY <L4OYL }A~ Ì
S
8
9 ¨ ¬
U³ 9
HET`L@OQZ ELNZLf]aNY-L ,·Ta<OYLNZJaL@`CHET`L4O ¨ -£K£W£W'/© .<¬®[ > 5
9¼
-#QRTS >
VTaNÌYAZC[EFaA],f]aH<`¹ÌbAI_E^L`Abw
WU V
OYLNZJaL4`Ï`E]WQZ]t <]MOQY¶ ELVTM. NYTWV] ,-`Ip0LH<OYIpTaH<]auÂJaL45QYTMN
¨i >£W£W£K . N ¬ Ü 4Ì ¹ Ò$IKJMLHº]aHºTM<OYLNZJaL4`ºOPL@Ð>ELH<L X:TL5¦L4QYLQY ELOSQZL^LQY <LHVTMNY
Eup]WQYLQZ EL³OYIK0EupLN
TW"6V9 57,Ô¶ ¤Ó`Ip. L4H<OYIKTMH<]WuÃJaL@®QZTaN¿O i¨ -£K£W£W
8 ¬5[Â ELNZL U]m¨ ¦Ip³EL4¨ H>QYNZTavRUENZTaEupLÂIKQY upIKHEL@]WNÂ5TMH<OPQYN]WIpHMQO[]MO
Ü 4Ì ¹ VTM; N :^ Ì -£K£W£K<5[ÑL¸]mQPQZL0QQZTnIpHVLN)] IpH >¬Ñ]aH<` Úa¬
u]mQYL4H>Q
U]m¦Ip³E¢LH>QZNYTM>Rf0T`LuQY <]WQ
U]mQ EL4O³L5¦L4®¤ UP } ]Wª ¦ § ¨ ¸$¬ § ¨ ³_¬ z}§ ¨
¬ ¨ Ì4Ü>¬
QZ]WQYIpTaH<O^TMHVL4]WQYENZL4OÑ`LØ<HEL@`rL5QS^LL4HrLJMLNZR0<]WIpNÑTWV8Jm]WNZIK¤
OYEãSL@®QÂQZT ¨¬ }~ \^]WNZupT
L@OSQZIKU]mQZIKTMH ¨ &¶vupLRLQ&]Wuå^Ì@ÍaÎ>Úa¬®[v0L4]aHrØ_Lu`rQZ EL5¤
ª « = t
TaNZR ¨ Ë]WELH¨ ]M`.ÅÂT`NZIKMELM[<Ì4ÍMÍaÎ>¬8TMNMLHEL4NZ]auKIpL@`³LupIKLV
S
¨¬ }~
=
ENZTa<]aM]WQYIpTaH st]WIpH>ÂNZIpa >Q8L5Q]Wuå%ÛaÜaÜaÝ<[sfLupuKIpHE$]WH<`X:L h[
ª « > t S
> ÛWÜMÜaÝ<[ L@`E`I`E]0L5Q&]au}ÛaÜaÜMÛM¬Â <LH * IOÂu]WNZaLa
ut
 EL4NYL ¨t ¬{ ¨i >£W£K£W .M -£K£W£K 2h¬® ÕÓH½QZ EIpO X:Tn5Ta0<]aNYLª¤åÕPQYTnOPQZ]aH<`E]aNZ`Ï^TauKQYU]aHEHC0]M EIpHEL
]MOPLM[@QY ELENYTM<]W<IKupIqQSRÃ`IOSQZNYIpEQZIKTMHVTMNhQY <L^Ta0EupL5QYL^`E]mQ] L4OPQYIpU]mQZIKTMHQYL@ EHEIÐM<L4O4[ÔØ<NZOPQ$TaH<OYI`LNQY EL0`L4NYIpJm]mQYIpTaH¹TWV
0T`Lu84]WH)_LÂNZIKQPQYL4H ]Ã`IpNYL@®Qª"]W<ENYT>]a h8ÕÓH0OPQZ]aH<`E]aNZ`ª£[mMIKJMLH
QY ELÂ<NYL¤
JvIKTM<O<]WN]W0L5QZLNO }A~ [jTaH<L
OYTaupJaL4O
VTaN$HELD<]WN]W0L5QZLNO
fvR0]W¦IK0IpIpHE&QZ ELÂ]W¦IpuKI]WNZR xCVEH_®QYIpTaHÂIKQY NYL@OPL45Q
L
t¨ ¬ t¨ ¬ QYT
Ì £ ª® ® ª ®
¨uKTM ¨ ¹ ¬z
Ì
Û 5 9 ¼ -#QRXS WUWV | ¨tM 9 ¬
8
 EL4NYL
IpOrQY EL * ^* OPRv00L5QYNZI
\|
U]mQZNYIK¦)TaV <]WN]W0L5QZLN¨ O
5TMNYNZL4OYTaH<`IpHE0QZTQY <L³VL@]mQY<NYL@O
5<up]WQYL4`rvRvR0<OYIKH< N
L4ÑQYTMHh² O^0L5QZ ET`TMNQY <LEIOYL4®QZIKTMH PO QYLU]aH<`0]ÃaLH<LN]WupIK4L4`³ª ]WupaTaNZIKQY E· _]aO%QYTL
<OYL4`IpH
0L5QZ ET`rQZTOPTMuKJML&VTMkN ¡ }~_=# q IpH QY< IpO]aOYLaXÑ EL³OPQZ]WH_`E]WN`]WEENZTM]M rIOÑQYTr<OPL]0MNZ]M`IpLH>Q
]aOZ5L4H>QQYT¹]aEENZTi¦IKU]mQZLupR¸OYTaupJaL0QZ ELº OPQYL4h¹à
Tm^LJMLN@[
Y
Q <LÃOPQYL4)OYIpLÃHELL@`EOQYT0LÃTaH>QYNZTaupupL4`UQZT
L4H<OYENYL]
0TaHETa¤
P
q
§ 8
k
¨ q
© *
ª
=
« :
¬ Y
Q TMHEIÃIK0ENZTmJaL4L4H>Q^IpH x0
&('
*) +,)-
ª¤åÕPr <]aO&`IpOPQYIpH<5Q&]a`Jm]WH>Q]WaL@O^TmJaLNÑQZ ELOPQZ]WH_`E]WN`rMNZ]W¤
"!# %$
k `IpLH>Q]MOYLH>Q
ª ]WEENZTM]M h«á%IKNOPQ4[#ª¤âÕPt5TM0EuKLQYL4uKR
/. B
`
01325476 8:9;6=<?>
A@CB?DE@FHG
+ 6 8# - 9<
B:DE@ JI
1
\ILKNM,O QP ,c
]iJaTMIp`<OÃQY ELº<OYLTWVÂQZEHEIpHE¹<]aNZ]a0L5QYL4NZOÂ EIKupLºOPQYIpupuM<]WNY¤
0SRUTWV + 6 8 # -
@
]WH>QZLLIpHEU0TaH<TWQYTMHEIÃIK0ENZTmJaL4L4H>Q4ÑTaNZLTmJMLN@[L <]iJML
*§4¨=©qª*«=¬:
V Ta<H<`tQY _]mQª¤âÕP£5TMH>JMLNZaL@O$Ve]aOPQYLN³QY <]aHnaN]a`EIKL4HMQ]aOP¤
k
X!# %$
&('
) +Y,Z
5
L4H>Q^ªn>á%IpaENZLÛOY ETm
O#QZ EL$NYL@OPEuKQTaV8]³OYIK0EupL&L¦vLNZIK¤
k 0
LH>Q^QY <]WQÑTa0<]WNZL4OQY ELNZ]WQYLTWV:TaHvJaL4NYMLH<LÂTWV8 OPQYL4
B
/.
`
0 1[2 4 6 89\6=<:>
A@CB:DE@ F]G
+ 6 8:# _ 9^<
B:DE@ JI
1
\ I`KNM,O Yc P
TaEQYIpIp4]WQYIpTaH¾QYL@ EHEIÐ>EL4O¹TaH§]×OPU]auKu³^TMuqQZU]WHEH§U]W¤
0SRUTWV + 6 8 # Z
@
EIpHELÂIqQZ ¹Ø<JML
JvIOPIpEupL
H<Tv`EL4OÃ]WH<`«QY ENZLL EI`E`L4H¸HET`L@O
A*§4¨=©qª+F«=¬
k
\Ta0<]aNYIpHEϪ¤âÕPר QZTQZ EL£aN]a`EIKL4HMQ]aOZ5L4HMQª ]WupaTa¤
EX!# %$
&('
*Y Z Y,a
Z
N IqQZ E ENZTaTMOYL4`¹IKH ¶vuKL4R¹L5Q]au}Ì4ÍMÎMÚM¬®[j^L
Ø_H<`¸QY <]WQ
k
ª ¤åÕP¸TaEQZ]WIpH<OOYE<OPQZ]aHMQZIp]auKupRVe]MOSQZLN³TaHvJaL4NYMLH<LaUá%IKa¤
Bb.
`
A@CB:DE@ FcG
B?DE@ JI ILK]M O \ c P @
ENZLtÛ]WuOPTÏOP <Tm
OQZ <]mQ<OYIKHECOYLJaL4NZ]au&ÕPÖIqQZLN]mQYIpTaH_OrIpH
QY <LIpHEHEL4NÃuKTvTa8[ <[jRvIKL4up`<O$Ve]a¨ OPQYLNTaHvJaL4NYMLH<LQY <]aH
^0 132 4 6 89\6=<:> _d6 8:# 9^< 1 0SRUTeV Z 6 8 # a
8
f
gih[j7fkHV>Ek f3lmh3nof3p2>+7¿k
QZ]a¶vIKHE]OYIKHEMuKL0ÕP¸OSQZLhh[ "̨  EIp £TaNZNYL@OPTaH<`<O
QZT
ÅÂIpL4uKL4N4² OÑENZTaTMOYL4`r]auKMTaNZIqQZ E ÅÂIKL4upLNÌ@ÍaÍaÍ>¬P¬5
à
L4NYL IpOrQY EL * `Ip0LH<OYIKTMH<]WuÃJML4®QZTaNºÂIqQZ Ù]Wupu̹LupL5¤ sfL$H<TWQYLQY _]mQÂIKHºENZLJvIKTM<OTMNY¶Ô[^R>NZHEL ¨ Ì4ÍMÍMÛM¬# <]MO<NYTa¤
0LH>QZO4[]aH<` 4 IpO³QY EL * *Ip`ELH>QYIKQSR£U]mQYNZIK¦ÔnXÑ <L TMOYL4`Ï]£OPL@ÐM<LH>QYI]WuÂEÔ`E]WQYL«]WupaTMNYIKQY <¢VTaN0QY <L«µOPQYL4
NZL4Ð>EIpNYL@`0L5¦_L@®Q]mQYIpTaH_O4]WHUL$4]Wu5Eu]mQZL4`0vRU`IKNZL45QLHv¤ IpHÏ]t^TauKQY40]aHEHÏ0]M EIpHELº_]WN]W0L5QZLN0L4OPQYIpU]mQYIpTaHC]WupaTa¤
rq s
−280
L5QPQZLN
L@OSQZIKU]WQYL4O^TaV:Jm]WNZIKTM<O^EH<`L4NYupRvIKH<0T`L4upgO ¥<OPL@`
QYT¸aL4HELN]mQZLUQY EL)`E]mQ]E«sfL)L@]aOYENZLUQY EL)Ð><]WupIqQSRfTWV&]aH
L4OPQYIpU]mQZNL vRr4]Wu5Eu]mQZIKHE
QY <L D¨t? !!- L
P(Q VNYTMQY <L
EMIS:K=4
−290
5TMNYNZL45QU0]aNYMIKH_]WuÂ`IOSQZNYIpEQZIKTMH ¥ ¬QYTtQZ ELL4OPQYIpU]mQYL@`
−300
U]WNZaIpH<]Wu`IOPQYNZIKEEQYIpTa*H i¨ ¬:TaH
QY <L
R3!3 %K? `E]WQZ]Ã5TM¤
EMIS:K=1
−310
−320
TaHEL4HMTQ
−330
−340
® °
−350
%á IKMENZLÛ¥Ã\TaHvJaL4NYMLH<LLJm]Wup<]WQYIpTaHh¥ÂupTaa¤åupIK¶MLupIK <T>T`JMLNY¤
0 50 100 150 200 250 300
XÑ ELaTM]au<IOQZT³0IpHEIK0IpLQY ELÃ5NZTMOZOL4H>QYNZTavR
L5QS^LL4HUQY <L
OY<OjIKQYLN]mQZIKTMHh[iOPTMuKI`Ã5ENZJaL`L4HETWQZL4Ohª¤âÕP&ÂIKQY ³IE<[4`ETWQP¤ U]WNZaIpH<]Wu#`IOPQYNZIKEEQYIpTaHtTWV^QY ELUL@OSQZIKU]WQYL4`¸0T`Lpu ]aH<`
QYL@`¹ENYJML
`L4HETWQZL4Oê¤åÕP«ÂIqQZ f²
ÌM[ ]aH<`«`E]MOP EL@`¹ENYJML QY <L¹5TMNYNZL45Q0U]aNYMIKH<]aGu Mt¨ ¥W §ß5NZTMOZO0LH>QYi¨ NZTavRnTaVLNZTtIpO
`L4HETWQZL4OÑaN]a`EIKL4HMQ
]MOYLH>Q4 TaEQZ]WIpHEL@`rTMHEuKRºÂ ELH ¬^0]WQZ EL@gO ¥ ¬®
sfL5TMH<OPI`L4N8]ÂOPL4NYIpL4OhTWVL5¦LNZIK0LH>QOjÂIqQZ `IK¯L4NYL4HMQ:T`v¤
LuOÂ]WH_© `r`IK¯L4NYL4H>QÂOZ]W© 0EupLÃOPIpL@O^QYT
QYL4OPQÑQY ELÃNZTaE_OSQZHEL4OZO^TWV
^RvNYHELM² O)]WupaTMNYIKQY <+NZL4Ð>EIpNZL4O)]u]WNZaL«HvE
_L4NTWV³IKQYL4NZ]W¤ TWQY «ªÙ]WH<`¹ ªÖQYTºOY<]aNZOYLQYN]WIpHEIpHEr`E]WQZ]<ÕÓH«_]WNY¤
QYIpTaH_O¹IpH§QY <LÏ OPQYL4§QYTÙLH<OYENZL£QY _]mQ£]×U]m¦IK
E IpO QYI5<up]aN4[^LÃ<OPL@`ºQY ELÃVTMuKupTmÂIKH<L¦_L4NYIp0LH>QZ]auh`L4OYIKMHh
]a <IKL4JaL4`h#Á&QZ ELNZÂIpOYLÂQZ EL&0TaH<TWQYTMHEI&5TMHvJaLNZaL4H<5LÑENZTaE¤
LNYQSR0]WH0L&JvIpTau]mQYL@`VTaN#QZ ELOYL4Ð>EL4H>QYI]WuEEÔ`E]WQYL4O EL
<NYTa¤ ÌaÂá%IK¦t¨t ]) Q]WNZaL5Q³^TMuqQZU]WHEHtU]a <IKHELU0T`L4[u M¥ i¨ ,¬
TMOYL4O48ÕÓHTMEN%4]aOYLa[iª¤âÕPÃ<OYL4O%]Â<]aNZ]auKupLuMEÔ`E]mQZLENZT5L¤ ¥ ¬5
`ENZL^QY <]WQ]iJaTMIp`EO8QY <IpO`EI 5<uqQSRM×OYL4Ð>EL4H>QYI]Wu]WupaTMNYIKQY E Û
Ò$¨t L4HELN]m QZL¢] OZ]W0EupL¢TWV¡Ta¨t <OYLNZJaL@`!`E]WQZ]
QY _]mQU]WIpHMQ]WIpH<OºQY ELf0TaHETaQYTMHEIpf5TMHvJaLNZaL4¨ H<5L¹ENYTM_L4NPQSR
]aH¸]aupOYTºLU]a EIpLJML4`¸]WupTaHErQY EL0upIKHEL@OÃTWV \TaupupIKH<O$LQ]au} -£K£W£K 8 ¬Ñ]MTaN`IKH<³QZA
T M¥ ¬® ±
ÛWÜMÜMÛM¬® ÝE¢ ÅÂ<H0ª¤âÕPÃQYTÃMLHEL4NZ]WQYL^³EuKQYIpEupLÑVL4]aOYIpEuKLÂOYTaupQYIpTaH_O
w z Ç | &<Ê Æ È ¿Ë Ë Å y MEÊ |½%ÊPÇ Æ
e x xHz {
vROPQZ]WNYQYIpHE&VNYTMÌ4ÜaÜ&NZ]aH<`TM¾IpHEIKQYI]Wu<]aNZ]a0L5QYL4NZO &
ªJMLHr]MOYOYE0IpHEQZ <]mQÑTaHEL$ <]MO]aHrL5¯ÔL45QYIpJaLÃ]WupaTMNYIKQY E½VTMN Ú
vL4uKL@®© Q0QY <L0]W¦IK
ELH>QZNYTM>Rn4]WH<`EIp`E]WQYL f]MO
upTv4]Wu:<]WN]W0L5QZLN
TMQYIp0IK@]mQZIKTMHh[<QZ ELNZL³NZLU]aIKH<OÂQY EL
IpOZOP<L QZ EL «ª¾L@OSQZIKU]mQZLa[%]WH<`¹QY© ELUU]W¦vIp³<ßuKIp¶aL4uKIp ETvT`
TWVTaEIpHEÂIKQY «
EuKQYIpEuKL³uKT]au8U]m¦IpU]EÂX8TºOPL4uKL@®Q
VNZTa
−2.4
0.2
−2.45
−2.55
Cross entropy
Log likelihood
0.16
−2.6
0.14
−2.65
−2.75
0.1 LME
−2.8
−2.85 0.08
1 2 3 1 2 3
10 10 10 10 10 10
%á IKMENZLÃÝ<¥#
© JaL4NZ]aaL$uKTMW¤âuKIp¶aL4uKIp ETvT`TaV8QY <L © ªÏL4OPQYIpU]mQZL %á IKMENZLÃÚE¥&JaL4NZ]aaL&NY© T>OYOLH>QYNZTavR0L5QS^LLHrQY EL$© QZNYELÃ`IOS¤
Sample size Sample size
á%IKMENZ$L ÂOP ETm
OQZ <]mQ © «© ªTMQZ]aIKHEL@`Ã]ÂOPIpaHEIKØ_4]WH>QYupR&upTmL4N
4.5
5NZTMOZO^LH>QYNZTavR0QZ <]WH ª
4.4
LME
4.3
4.2
Entropy
4.1 0.32
4 0.3
MLE
3.9 0.28
MLE
3.8 0.26
3.7 0.24
Cross entropy
3.6 0.22
1 2 3
10 10 10
%á IKMENZLJ_¥:
JMLN]WMLLH>QZNYTM>RTaV<QZ EL
© ª¸L4OPQYIpU]mQZL^JMLNOP_O
Sample size
0.2
0.16
LME
0.14
%á IKMENZL4O×Ý¡]WH<`À½Ø<NOSQÖOY ETmQZ <]mQ×QZ EL·]iJaL4NZ]aaLÖupTaa¤
upIK¶MLupIK ETvT`EO0© ]WH_`]iJaL4NZ]aaLU© L4HMQZNYTMEIpL4OTWV
QZ EL0Tv`ELuO
<NYTa¤
0.12
1 2 3
10 10 10
á%IKMENZ*L <¥&JaL4NZ]aaL&NYT>OYOLH>QYNZTavR0L5QS^LLHrQY EL$QZNYELÃ`IOS¤
Sample size
! 5
£:\TMuKupIKH_O[gÅ h _]WEIpNYL0]WH_`
8IKHEMLN ¨ ÛaÜaÜ>Ûa¬® © TaMIpOP¤
0.27
MLE
QYI
NZLMNYL@OYOYIpTaHh[a`E]a^¨ TvTMOPQ]aH<`^NYL4aU]WHU`IOPQZ]WH_5L4O4 U
Ht
0.26
0.25
°_ v ]aNYNZT Ï]aH<` v
Å ]mQ EuKIK¯ ¨ Ì4Í WÛa¬5yÒ$LHEL4NZ]auKIpL@`IKQP¤
0.24
"!#
U[?
&@
*+D#_²E
N
*H!%
P+D%! [ u [8Ì mÜW¤ZÌ MÎMÜE
0.23
LME
$
-7 -
Ô v Lupup]s IpL5QZNZ]<[ ³ v L4uKu]s IpL5QYN]&]aH<`³°_ © ]W¯L4NPQSR ¨ Ì4ÍaÍ W¬5
0.22
&% ('
á%IKMENZAL ¥&JaL4NZ]aaL&NYT>OYOLH>QYNZTavR0L5QS^LLHrQY EL$QZNYELÃ`IOS¤
Sample size
U]mQZL4OÑIpHª¦_L4NYIp0LH>Q
ÝE v L0<OPQYLN@[ © ]aIKN`t]WH<` v hÂÅ EEIpH ¨ Ì4Í W¬5««]m¦IK¤ %
³<±upIp¶aLupIp ETvTv`tL@OSQZIKU]WQYIpTaHtVNZTa¢IKH_5Ta0EupL5QZLr`E]WQZ]«JvI]
N
sfLf _]iJaL¹IpH>QYNZT`<5L@`y]¼HEL¿IKHEVLNZLH<L¹QZL4 EHEIÐ>ELa[QY <L
u]mQYL4H>Q&U]m¦IK
E"LH>QYNZTavRr<NYIpH<5IpEupLa[]WH<`<OYL4`IKQ
QZTr`L5¤
- $' ?&l(E$'
?
P+? [ t [hÌMÌ4Ý i¤ZÌMÌ@Ú E
6- %
Ô © ]W<NYIKQY4LH ¨ Ì@ÍaÍ>Úa¬®-XÑ ELfª¤Ó]WupaTMNYIKQY EVTaNMNZ]aE EIK¤
NZIKJML]H<L×upTv4]Wu<]WN]W0LQYLNTaQZIK0Ip4]WQYIpTaH0L5QZ ET`j[Eª¤ ]au$]MOYOYT5I]mQZIKTMH×T`L4upOºÂIKQY y0IOYOYIpHE¼`E]WQZ]< ?&l(E$'
X
ÕPÔ[
]aH<`y]aupOYTÏ]¼HEL4¿OYLupL4®QZIKTMHÖQZL4 EHEIÐ>ELa[몤⪤åÕPj[
VTaN
TMuqQZU]WH<H£0]M EIpHEL4O4XÑ EL4OYLUENZIKH<IK<uKL@ORvIKL4up`tH<L 98 ²³
?
PH!%
NX 1?
IEQ"!%H! [ 8[hÌ4Í<Ì5¤SÛWÜ<Ìa
4-
7
XÑ EL4NYL¼NYL40]aIKH·OYLJMLN]Wu]iJaLHvEL@OQYTÙL5¦vQYL4H<`§QZ EIO¹^TaNZ¶Ô XÃvLPãSHETm
OY¶>I ¨ Ì4ÍaÎ M¬5 à&IKM ELNTMNZ`L4N«^TauKQYU]av HEHÙU]W¤
EIpHEL4O4 - $'? m 3
P\]?7?!<3? ?&l(E$'
Pq [
°_ LHE¶MLN@[
TMNYLaL4HELN]Wup¨uK RM[Ô>R«]auK upTmÂIpHEº<IK H< ]aNYR>¤âJm]WupEL4`¨ VL@]mQZENYL@O$TWV :
ª`h
0LNZIp4]WH)ÕÓH<OPQYIKQYQZLTWVi vROPIO4[ÝaÎ W¤ >ÜaÝ<
QY <LUVTaNZ ¢ ¨Y¬ ª £W£W£ VTaN
-£K£W£
? ¨[] P
<]WQY yIp
H ¬®[Lt5TMH<OPQYNZ<®Q)T`L4upOºQZ <]mQ¨ ]WNZL
á^$vTMQY <LRa[ v $E ><ENYU]aH<O«]WH_`¾³$Ò$ ET`EOYI ¨ ÛWÜaÜMÝM¬5
L4OZOYLH>QYI]WupuKR ÄY EIKM ELNY¤âTaN`L0N Æ×^TauKQYU]aHEHDU]a <IKHEL@O vL5¤ ÂÅ L4aEu]WNZIK4L4`CMNYL4L4`RCIpTaNYQZ]aH<5LfOY]a0EuKIpHE_ I?K??D# %!
ãSHETm
OY¶vI#Ì4ÍaÎ M¬5 m 3$'?L±%9<3?%&.?
P+? 9D# %!#!%q ²³Q"!%
3&m!
vG
L@`E]]aH<`.Å &]W¶m]aHET ¨ Ì@ÍaÍMÎM¬® v LQYL4NY0IpHEIpOPQYI&]aHEHEL@]WuK¤
$ ;+ - '
uOYT<[ÂQZ EL¹QZNZ]aIKH<IKHEÏ]WH<`yL4OPQYIpU]mQYIpTaHyLQY ET`EO4]WHyOPQYIpuKu
LIK0ENZTmJaL@`rvRrQZ]a¶vupIKH<
QZ EL OYIK
EuqQ]WHEL4Ta<OYupRaÕÓHQZ EIpO N IpHEUª¢]WupaTMNYIKQY E - $'? -
*\]7"! [ 8[Û aÛi¤SÛWÎ>Û
"N
<]a_L4N4[>vRUNZ]aH<`Ta0upR <T>T>OPIpHE³`EIq¯ÔLNZLH>QÂOSQ]WNYQYIpHETaIpHMQO[ ! 6-<-
£#st]WIpHvÂNYIpa >Q4[XðM]M]W¶v¶aTMup]«]aH<`³s×IpuKuOP¶vR ¨ ÛWÜaÜMÝM¬5
^LQZ]a¶aLUQZ ELVL@]aOYIK<uKLº© uKTMW¤âuKIpHEL4]aN0T`Lu^ÂIqQZ nU]m¦Ip³E X:NYL4L5¤â<]aOYL4`rNYL4<]WN]W0L5QZLNZIK@]mQZIKTMHUVNZ]a0L^TaNZ¶0VTaN
]WH_]WupRvOYIO
LH>QZNYTM>RJm]WupEL&]MOQZ EL «ª£L4OPQYIpU]mQZLa#XÑ EIOENZT5L@`ENZL
IpO TWVLupIKLVENYTM<]W>]mQZIKTMHÏ]aH<`CNYL4up]WQYL@`Ï]auKMTaNZIqQZ EUO ±VcV]V
5TM0EQZ]WQYIpTaH<]auKupR¹L¦_L4H<OPIpJaLMXÑ >_OIKQ
IO^TaNYQY v EIpuKL0QZT Y?L!3XG±%9<3?%&.?
P+?ÄYEL #?%Q [ 8[:ÌaÌiÛWÜm¤ÌaÌ <
`L4JaLupTa]aH]WH<]auKTMaTM<Oj`L5QZLNZ0IKHEIOPQYI]WHEH<L4]WupIpHE
«ª#¤Óª¤ Ô#st]WHE_[ v vE<NYU]WH_O]aH<`
<]aT ¨ ÛWÜMÜMÛM¬®¼XÑ <L
%/ -
ÕP¸]WupaTaNZIKQY E"QYT]aQYTM0]WQYI]auKupR)© Ø<H<`¹¨ VL4]aOYIpEuKLU]m¦Ip³E u]mQYL4H>Q
0]W¦IK
ELH>QZNYTM>RU<NYIpH<5IpEupLa#U]WHv<OZ5NZIpQ
>=
LH>QZNYTM>R)upTaW¤âupIKHEL@]WN
0T`L4u8VTMN ¹ª G&L4`E]º]WH<` N&]W¶m]aHET
rt
d âd>kmd>A%gadvo Ì4ÍW¤ÓÚaÜE
9-5%
v &&¶vupLRa[Ò&à&IKH>QYTMH¾]aH<`¾XÃ&vLSãSHETm
OY¶vI ¨ Ì@ÍaÎMÚM¬®" _° L4`EIp`I]E[%s§:á<NYL4LU]WH8[%]aH<`
:s¸L4IpOZO ¨ ÛWÜaÜ>Ûa¬5\TaHE¤
upL4]WNZHEIpHE£]WupaTMNYIK¨QY EVTaNr^TauKQY4U]WHEHCU]a EIpHEL@O ³ PO QYL NZup<IKL®VºQZIKHEE)NZTaV<NZ]aLML0]WLQYH<IpTaLH·NZaR¹]Wupa]aTMENYEIKQYNZ <Ti¦v0IpUO4 ]mQZIKOPTMEH<<O]aIKH<QPQZ`¸L4`DaLH<QYT LN]W±VcupIK4V]L4V `
t
t