Professional Documents
Culture Documents
Deep Learing Book PDF
Deep Learing Book PDF
!#"%$'&)(*,+.-!"/$0132406575701(,+'-98:$;!$<>=?$<1$@2:A9BDCECEF6-*6*G3H@IJ6(*KMLON0 P,$C75RQT(*S IUI?$CWV
XY[Z]\;^;_a`cbed.fW_Wghghikjl^]m_Wgh_n\befMop4_Wq>\rbhseZ_Wj.stvuwxuzy9\;{lg|vm)_Egh_n\befMo9}*~n; fMo6lkRp4beik'_
}
m_n\;jl}1R@
~E|[
6}!a t
ikkU\Z_Eshse_a4j1ik'_Ebeghis[.}1
;6sM\rse_
6shbe_W_7sn} \;k_WZ}l4mT
;; ~
}>a1t
a pY[m41u4} m> dr_Eb4 o>\lghgh_W_
¡.},~r¢r£'¤;_Ebekikj9} _7beZ]\;j.
¥ .\;jlj*} ¦_Ed
j6{!§¨)be_Wgh_n\befMo9© \rshdsn©befEi¦d
^;Zikj>\;}>k^;d;bhq1b7¨)l{lkikikghk¦ol\;_WZ_Eikshj se_
© _n 9}l ¦\lgM¨)ªlbegOsn© ^;Z]*© 1_
)bhbn} ©>\j> #l« kk_Ebn}¬@©v[4_E1bM\;*4_7s[®dbe6gW¯1shbeikfM gd°cseol_4shbM\;1_W± }
q1beikj1^
_Ebn}*~n;
¤ ©
u®o1_fWd
j6'_Ebe^;_WjlfE_ºd°{>\f7.|»q1bed;q>\;^
\seikd
j¼k_n\rbejlikjl^#ikg\;jl\; W_n
gh²]d³c\;´Wg4µW¶rse·>d]¸;_Eµ
½ ¹ q1U\ikjºfEd
ZZd
jq1ol_Wj1d
Z_Wj1d
jd;{lgh_Ebe
_n{6q1bM\fEseisei¦d;jl_EbegW© \j6
1j>1_EghikbM\{lk_]{_Wo>\n6ikd;beg
d;°{>\f76q bed
qfn\j#{_]\n'd
i¦1_W3ikseoshbeikfM6g
seol\sD\be_
bM\rbe_W¾_7½1qd;gh_n¾ikjºgh_Ebeikd
1gse_Ef7o1jlikfn\;%q1l{lkikfn\rsei¦d;jlgW©,u®olikgaq>\q_Eb^
ik'_Egaghd
Z_
d °c\sej6old
gh\;_R shsebeolikfMd;6begWgD}>o>\\nj>'_dgh¿!1_E^
be^
g_E_EgOse½ _nql¦\;sejlo>\\sesDikd
ghj1_Wg)fWd;dj>°6|Ào.d]bM1se_EolbD_Ed
]q ®seikdZbei¦W© \seikd
jÁZ_Eseo1d61g
\rbe_®\
\j.sM\;^
_Ed
lg*°Jdbcjl_W bM\;.j1_Es9shbM\ikjlikjl^ ©rYÀsikggho1dnjseol\s,Zd
gOs)[fW¦\;ghghikfn\k±
gh_WfEd
j>6|»d;bM1_7bDZ_7seold61gR\be_ikZq1bM\fEseikfn\;c°Jdb
U\rbe^
_]jl_E1bM\;%j1_Es[d;be6gW©t°J_E
Z_Eseo1d61g\rbe_aq1bed
qd;gh_nseol\s)1dDjld;sol\r
_4seol_Wgh_akikZisM\seikd
j1gW©
 ÃlÄÅÆ!ÇaÈDÉ
Ê,Å!ËeÇÄ
241ÌWÍ>Î!C701Î*6Ïl.5E=U01Á=?K< $;C7ÐÁÎ,0 Î*(!I? C]!$;(!CE I!$5eÑ)01C7ÍÁI?$; C7*=U!Ï# IUÏ10 CE=¦5EÒ!ÓÔP,$;Ì (*K7$
=¦5
=JK
Ì0 *Ì$Î5E(*6I?I?Ð3K7=UÓÎ!I?$ -,Ìr01ÓÎ*(5E657=?0 * IUI?к$ÕÌ=U$;l5;-9 *GP,$;Ì (*K7$=¦5
0 Ö×57$;ÁÑ)01C7ÍK;A
Ø0.Ñ4$< $;C;-'Ï1$r5M5E=U*Ï:=U5570Ñ40 CEÍÑ4$I?IÀ-6 *GKM01Ó$57=?Ó$;K570@Ñ40 CEÍÙ.5 IUI[- Ì K7$$;ÓÚÓ0 CE$R06Ö
66C755EÒ*6ºK7Ì=U$;*Ìr$1A1ÛD$;K7=UÏ1!=?!ÏÙ69G57CW6=?!=?!Ï]]!$5eÑ)01C7Í(*K7=U!ÏÙP* ÌWÍ>Î!CE0 ÎCE$;Ül(!=?C7$
K
Ó Í>=U!Ï
Ó >Ð@K7$$;Ó=U!Ï1IUÐ@6CEP!=U57CW6CEÐDÌWÒ!01=?Ì$;KKM(9ÌWÒ] K/57Ò!$)l(*ÓÙP,$C®6*G@5eÐ>Î,$;Kv06Ö!!0G$
K-
I?'Ð1$CWK->I?$;6CE!=?!ÏCE657$
K-5ECE =U*=U!Ï *G¾57$;KM5DK7$r5WK-* *GKM0ÖÝ01CM5EÒ%A*ÞRÒ!$
KM$]ÌWÒ*0 =JÌr$;KaÌ;6P,$
ÌrCE=¦5E=?Ì;6I[-Ð $5/5EÒ!$CE$=JK/!0ÖÝ0l01IUÎ*C70>06ÖCE$;Ì=UÎ,$ÖÝ01CvG$
Ìr=JG=U*Ïa57Ò!$;ÓßP9$
Ì (*KM$57Ò!$;Ð@ C7$I? C7Ï1$I?Ð
Î!C701P!I?$Óà *GG!.5W:G!$Î,$*G$;l5;A Ø
0.Ñ)$;< $C
-
57Ò!$;C7$a C7$RÒ!$;(!C7=JKM57=JÌK69GK70 Ó$R(!*G!$CEIUÐ>=?!Ï
57Ò!$;0 CEÐ57Ò*65
Ì ¾Ò!$;IUÎÏ1(!=?G!$@Î!CW Ìr57=U57=?0 !$;CR570Ó6Í1$:P,$r5757$;C
ÌWÒ!0 =JÌr$
KA
áhÙ5EÒ!$Râ*CWKe5K7$;Ìn5E=U01P9$;IU0.ÑãÑ)$R=?15EC70G(9Ìr$aKe5W6*G*6CWGÙP*1ÌWÍ>Î!C701Î*6Ïl.5E=U01]6*GG=JK7Ì(*K7K
>(!ÓÙP,$CD06ÖK7=UÓÎ!I?$]Ò!$;(!C7=JKM57=JÌK01C5EC7=JÌWÍKaÖÝ0 CD=UÓÎ!CE0.<>=U*Ï=¦5WKÎ,$C7ÖÝ0 CEÓ *Ìr$1A*äå$Ù!$æ>5
G=?KEÌr(9K7K=JKEKM(!$
K]0 ÖaÌ0 >< $;C7Ï1$*Ì$ Aäå$57Ò!$;çG$
K7ÌC7=?P9$#ÖÝ$Ñéè7ÌI?1K7K7=?Ì;6IJê¾K7$;Ì0 *GL[01CEG$;C
!0 LOI?=U!$
6C01Î57=?Ó=Uë
.57=?0 T5E$;ÌWÒ!!=JÜl(!$;Kº6*GìK7Ò!0.Ñí5EÒ*.55EÒ!$=?C¾ Î!Î!I?=?Ì;.5E=U01ç5E0z!$(*CE I
!$r5eÑ40 CEͼ57CW6=?!=?!ϼ=JK< $;C7ÐzI?=?Ó=U57$
Gc-RG$;K7Î!=U57$#Ó6>ÐãÌI? =UÓK5E0¼57Ò*$ÁÌr0 l5ECE C7Ðz=?ì57Ò*$
IU=U57$;CE657(!CE$ Aî=U96I?IUÐ1-Ñ)$Î!CE$;K7$l5ÁÖÝ$;ÑïK7$;Ì0 *GL[01CEG$;CÙÓ$r5EÒ!0G!K57Ò9.5ºG0¼ Ì;Ìr$;IU$;CE657$
IU$
6CE!=U*Ï=?Ìr$C75E =UÌ1KM$
KA
ð ñÙò%ó®Æ*Ä
Ë7ÄôõóÄ
È÷öxò/Äò%Æ*ó®ø7ËeùcóÅ!ËeÇÄ
ÞRÒ!$CE$6CE$KM$;< $CW6I Î!Î!CE01 ÌWÒ*$;K
570Á6(!570 Ó657=JÌÓ1ÌWÒ!=?!$IU$
6CE!=U*Ï*-cP!(5ÓÙ(*ÌWÒú06Ö457Ò*$
KM(*Ì;Ìr$
K7KMÖÝ(!I>6Î!Î!CE011ÌWÒ!$;K/Ì6ÙP9$4Ì.5E$Ï10 CE=Uë;$;G Kû6üEý þ.ÿ Wý Wþ Wý.ü 9ÿ lû ;þ nA'ÞRÒ*$
COST FUNCTION
Desired
Output
Output
D0, D1,...Dp
Parameters M(Z,W)
W
LEARNING
MACHINE
Input
Z0, Z1,... Zp
$;Ül(*.5E=U01*Ka6CE$:0 P5W6=?!$;G C
, ., G,
. .
G
.,! . + ., #"
., $ , 5 &
.$ + J , % .
ÞRÒ!$]6P,0.< $:$;Ül(*657=?0 *KaÌ;66IJKM0P9$ÑaCE=U5M57$;=UÓ.5EC7=UæÖÝ0 CE'Ó C
' # . )(X
. .
. G *
. +.
G
. + ,. + . 5 -
ÞRÒ!$KM=?ÓÎ!IU$
Ke5aI?$; C7!=?!'Ï ÝÓ=?!=UÓ=?ë;657=?0 4Î!CE0Ìr$
G(!CE$:=?KM(*ÌWÒKM$5M5E=U!Ï=JK457Ò!$Ï1CE6L
G=U$;l5
G$;KEÌr$;15 IUÏ10 CE=¦5EÒ!ÓÑaÒ!$;C7$ =JKa=U57$;CE657=?< $I?Ð PG e(*Ke5E$;G K4ÖÝ01IUI?0.ÑDK C
/.% /. F F10 5 %
áh 5EÒ!$KM=?ÓÎ!IU$
Ke5Ì1KM$1- 0 =?K@3K7Ì;6IJ6C@Ìr01*Ke5W6l5;AQ#01C7$KM01Î!Ò!=JKe5E=?Ì;.5E$;GÁÎ*C70Ìr$
G(!CE$;KD(*K7$
<' C7=J6P*IU$ 0 AáhT065EÒ!$CÓ$r5EÒ!0G!K 0 5W6Í $
K]5EÒ!$3ÖÝ0 CEÓ 06Ö@ G=J6Ï10 * I)Ó657CE=¦æc-®0 C=JK
$;KM57=?Ó657$¾06ÖR57Ò!$¾=U><1$CWKM$Ø
$;KEKM=J6¼Ó.5EC7=Uæú06Öa57Ò*$ºÌr0lKe5ÖÝ(!*Ìn5E=U01 »K7$;Ì0 *G¼G!$CE=U<..5E=U<1$
Ó657CE=¦æ K7(*ÌWÒz1K]=?¼57Ò!$ $ÑR5E0 ç *G D(*1KM=UL
$ÑR5E0 úÓ$r57Ò*0>G*KG$
K7ÌC7=?P,$;GúIJ.57$;CÙ=?
57Ò!$ÌWÒ* Î57$;C;SA Î!C701Î9$;CÌWÒ!0 =JÌr$06Ö 0 =JK@=?ÓÎ,0 C75E l5@ *GÁÑa=?IUIP,$G=JK7Ì(*K7K7$;G .5@IU$;!Ï65EÒ
I?657$C
A
a
ò Æ!óÊ,Å!ËMÊóø
Æ9ËMÊ
241ÌWÍ>Î!C701Î*6Ïl.5E=U01Ì63P,$< $CEÐK7IU0.ÑÎ96C757=JÌr(!IJ6CEIUÐÖÝ01CaÓÙ(!IU57=?IJ'Ð $CE$;Gº!$r5eÑ40 CEÍK4ÑaÒ!$CE$
57Ò!$#Ìr0lKe5KM(*CMÖ»1Ìr$3=?K5eÐ>Î!=JÌ6I?I?Ðå*0 LhÜl(* GCW.5E=?Ì -v*0 LhÌr01l<1$ræc-v *GzÒ*=UÏ1ÒTG=?Ó$*K7=?0 * I
Ña=¦5EÒ Ó6>ÐI?0>Ì;6I®Ó=U*=UÓ36*G 2.0 C 9.5@CE$Ï =?0 9KAÞRÒ!$CE$=?K:!0¾ÖÝ0 CEÓÙ(!IJ5E0¾Ï1(*6CW6l5E$$
57Ò*6:5 /5EÒ!$a!$r5eÑ40 CEÍÑa=?IUI*Ì0 >< $;C7Ï1$5E0@:Ï10l0GKM01IU(!57=?0 %"- E ®Ìr01>< $CEÏ $;*Ìr$)=?KK7Ña=UÖ×5;-601C
Ìr01l<1$CEÏ $;*Ìr$a$;< $0Ì;Ìr(!CWK6546I?I[AlØ
0.Ñ)$;< $C
-6=?57Ò*=?K4KM$
Ìn5E=U01Ñ)$DG=?KEÌr(9K7K4>(!ÓÙP,$C
06Ö5EC7=JÌWÍK5EÒ*.5Ì ÁÏ CE$;657I?о=?ÓÎ!C70.<1$]5EÒ!$ÌWÒ96*Ì$;KD06Öâ**G!=U!ÏÏ10>0>G KM01IU(!57=?0 ÁÑaÒ!=?I?$
6IJKM0
G!$;ÌrCE$;1KM=?!Ïa5EÒ!$)Ì0 >< $;C7Ï1$*Ì$®57=?Ó$)0 Ö×57$ÙPlÐ@01CEG$;CEK%06Ö*Ó6Ï1!=U57(*G$1A
QÁ0 CE$G$r5W6=?IU$
G
57Ò!$;0 CE$r57=JÌ I e(*KM57=Uâ9Ì657=?0 *KRÑa=?IUI/P9$Ï1=U<1$3=UIJ.57$;CKM$
Ìn5E=U01*KA
!"$#&%(')*,+!-/.0%12')3"034
45º$;1ÌWÒx=¦5E$CW.57=?0 /-$;Ül(*657=?0 CE$;Ül(!=?C7$
KzÌ0 ÓÎ!I?$r57$#Î9 KEK57Ò*C701(!Ï ÒT57Ò*$Á$l57=?CE$
G!.5W K7$r5=?#0 CWG$Ca5E0ºÌr01ÓÎ!(57$]57Ò!$¾6ý 5 ü7ý;û
0 Ca5EC7(*$]Ï CW G=?$l5
A*ÞRÒ!=JK=JKaC7$ÖÝ$CEC7$
G¾570º1K
P*.5WÌWÒI?$; C7*=U!ÏK7=U9Ìr$6$l5E=UCE$èMP*65EÌWÒ*ê:0 ÖG!65E:ÓÙ(9Ke5P,$Ìr01*K7=?G$;C7$
GÙP,$rÖÝ01C7$aÑ4$=?Ï Òl5WK
6CE$ (!ÎG!.5E$;GcA
I¦5E$( CE*.5E( =U<1$I?Ð -R0 *$ Ì (*K7$úKe5E0>ÌWÒ9 KM57=JÌ Ý01!I?=U!$ ¾IU$
6CE!=U*ÏçÑaÒ!$;C7$å
Wÿ >û a$ræ! ÓÎ*IU$ 3
; ==JKÌWÒ!01K7$ Ý$1A Ï9A!CW6*G01ÓIU"Ð )ÖÝCE0 Ó 5EÒ!$57CW6=?!=?!ÏKM$5
.5$
ÌWÒ
=¦5E$( CW.57=?0 .nXA
[ÿ ý 06Ö,57Ò!$57CE(!$aÏ CW G=?$l5=JK®57Ò!$;Ìr01ÓÎ*(57$
GP9 K7$;G0 57Ò*$$CEC701C
(
. 7 < /%. F10 5 %X
2)$
Ì (*KM$57Ò!=JK/$
Ke5E=UÓ657$0 Ö57Ò!$)Ï CW G=?$l5/=JK%*0 =JKMÐ1-5EÒ!$Ñ4$=?Ï Òl5EK/Ó'ÐD!0 5vÓ0.< $Î!C7$
Ìr=JKM$;IUÐ
G0.Ña57Ò*$)Ï1CE1G=U$;l5v.5$; ÌWÒÙ=¦5E$CW.57=?0 /A DK/Ñ4$)K7Ò*6I?I>KM$;$ -;57Ò!=JK:èM!01=?K7$;ê65v$
ÌWÒ]=U57$;CE657=?0
Ì6 P9$ G<. 15W6Ï1$0 (9KSA ?l570ÌWÒ*1Ke5E=?ÌÙIU$
6CE!=U*Ϻ=JK
Ï1$!$;CE IUI?Ð357Ò!$Î!CE$rÖÝ$CECE$;GÓ$r5EÒ!0GÁÖÝ01C
P* K7=?Ì:P*1ÌWÍ>Î!C701Î*6Ïl.5E=U01ÖÝ0 CR5EÒ!$@ÖÝ0 I?IU0.Ña=?!Ï57Ò!CE$$CE$;1KM01*DK C
89 #!:3;!4<%1=&>?!"A@B%12')3C"3C4
A ?l5E0>ÌWÒ9 KM57=JÌI?$;6CE!=?!Ï=?K4(*K7(*6I?I?Ð : D @Ö»1Ke5E$C457Ò963P*.5WÌWÒºI?$;6CE!=?!Ï*A
E A ?l5E0>ÌWÒ9 KM57=JÌDI?$; C7*=U!Ï6IJK700 Ö×57$;3CE$;K7(!IU5EKR=?¾P,$r5757$;C
KM01IU(!57=?0 *K;A
KM=?ÓÎ!IU$3Ì1KM$¾ÑaÒ!$CE$¾5ECE =U*=U!ÏåKM$50 Ö
KM=?ë$ =JKÙ=?* G!< $C757$;l57I?ÐåÌ0 ÓÎ90lKM$
G¼06Ö
=?G$;l57=JÌ6IvÌr0 Î*=U$
Ka06ÖK7$r5
Ña=U57Ò K7 ÓÎ!IU$
KA < $;CE Ï =?!Ï57Ò!$]Ï1CE1G=?$l50.< $;C6I?<I
Î*.5757$CE*KÙÏ =?< $;K]57Ò!$¾$ræ! Ìr5KE6Ó$CE$;K7(!I¦5 KÌr0 ÓÎ!(!57=?!Ï#57Ò*$ºÏ CW G!=U$;15P* K7$;Gú01' e(*Ke5
57Ò!$3â*CWKe@5 AÞRÒ>(*K;-P*.5WÌWÒçÏ1CE1G=U$;l5G$
K7Ì$l5=?KÑR KM57$ÖÝ(!IaP,$;Ì (*K7$¾=U5C7$
Ìr01ÓÎ*(57$
K
57Ò!$KE6Ó$Ül(*6l5E=¦5eÐ 57=?Ó$
KP,$rÖÝ0 CE$0 *$Î*6CW6Ó$r5E$C@(!ÎG!.5E$ ABD 5EÒ!$065EÒ!$CÒ96*Gc-
Ke5E0>ÌWÒ9 KM57=JÌÁÏ CW G!=U$;153Ña=UI?IKM$;$åzÖÝ(!I?I$Î,0>ÌWÒ K =¦5E$CW.57=?0 9K5EÒ!C701(!Ï Ò LOI?0 !Ï
57CW6=?!=U*ÏK7$r5
A%áhúÎ!CW Ìn5E=?Ì$ -c$æ ÓÎ!IU$
K@CE C7$;IUÐÁ Î!Î,$;6C@Ó0 CE$57Ò* ú0 *Ì$=?¼#G!.5W K7$r5
-
P!(5:57Ò!$;C7$Ù C7$Ù(*KM(96I?IUÐ#ÌrI?(*KM57$;CEK
06ÖÎ*65M57$;C79K5EÒ*.5@6CE$]< $;C7ÐKM=?Ó=UIJ6C
A9î*0 CD$ræ!6ÓÎ!I?$Ù=?
Î!Ò!0 *$Ó$ÌrIJ KEKM=Uâ9Ì657=?0 %- IUIc0 Ö/57Ò*$:Î9.5M5E$CE*K4ÖÝ0 CR5EÒ!$@Î!Ò!01!$Ó$ 2:2ÙÑa=?IU<I »Ò!0 Î,$rÖÝ(!I?I?$Ð
Ìr0 l5W6=?]ÓÙ(*ÌWÒ06Ö!5EÒ!$RK7 Ó$4=?ÖÝ0 CEÓ.57=?0 /A;áO5=JK/5EÒ!=JKvCE$;G!(!*G! *ÌrÐ:57Ò9.5Ì6Ó6Í $)P*.5WÌWÒ
IU$
6CE!=U*ÏÓ(*ÌWÒKMI?0.Ñ)$;C457Ò* ¾01L[I?=?!$ A
?l570ÌWÒ*1Ke5E=?Ì
IU$
6CE!=U*Ï I?K70Ù0 Ö×57$¾C7$
KM(*I¦5WK)=?3P9$5M57$;CaKM01IU(5E=U01*K4P9$
Ì (*KM$:06Ö/57Ò!$@!01=?K7$
=U5EÒ!$:(!Î,G*.57$
K1A 01!I?=U!$
6C)!$r5eÑ40 CEÍK(*K7(*6I?I?ÐÒ*'<1$ÓÙ(*I¦5E=UÎ!I?$:IU0Ì I*Ó=?!=UÓÙ06Ö/G/= V,$;CML
=U!ÏG$Î5EÒ*K;AlÞRÒ!$DÏ 0l6I!0 Öc57CW6=?!=U*Ï=?K570]I?0Ì657$01!$06Öc5EÒ!$;K7$Ó=?!=UÓ*Al2R.5WÌWÒIU$
6CE!=?!Ï
Ña=UI?I%G=JK7Ì0.< $C57Ò!$@Ó=?!=UÓ(!Ó0 Ö/ÑaÒ*.5E$<1$C4P* K7=U¾57Ò*$DÑ4$=?Ï Òl5EKR C7$D=U!=U57=J6I?I?ÐÎ!IJ Ìr$
GcA>áh
Ke5E0>ÌWÒ9 KM57=JÌIU$
6CE!=U*Ï*-c5EÒ!$!01=?K7$Î!CE$;K7$l5]=Uú5EÒ!$(!ÎG!657$;KÌ åCE$;K7(!IU5Ù=?ú57Ò!$Ñ4$=?Ï Òl5WK
e(!ÓÎ!=U*Ïå=?l570 57Ò!$P*1KM=?ç0 Ö: !065EÒ!$C
-vÎ,01KEK7=UP!I?ÐçG$;$Î,$C
-®IU0Ì IRÓ=?!=?ÓÙ(!ÓAÞRÒ!=JKÒ*1K
P9$;$#G$Ó01*Ke5ECE657$
Gº=?#Ìr$C75E =UK7=UÓÎ!I?=¦â9$;GÌ1KM$
K "!- [A
?l570ÌWÒ*1Ke5E=?ÌDIU$
6CE!=?!Ï=JKa6IJK70(*K7$rÖÝ(*I%ÑaÒ!$¾57Ò*$DÖÝ(**Ìn5E=U013P,$=?!ÏÓ0G$I?$;G¾=?KaÌWÒ96!Ï L
=U!Ïã0.< $;C57=?Ó$ -RçÜl(!=U57$ Ì0 ÓÓ0 KEÌr$;*6CE=U0ç=U=?*G(9Ke5EC7=J6I:6Î!Î!I?=JÌ.5E=U01*KÑaÒ*$CE$Á57Ò*$
G!.5WG=JKe5EC7=?P!(5E=U01ÁÌWÒ*6!Ï1$;KRÏ1CE1G(*6I?I?Ð0.< $;Ca57=?Ó$ Ý$1A Ï9A9G(!$]570Ñ4$; C
6*G¾5E$;6CD06Ö®57Ò*$
Ó1ÌWÒ!=?!$;
K nAcáOÖ)57Ò!$I?$; C7!=?!ϾÓ1ÌWÒ!=?!$G0>$;K@!0 5G!$r57$
Ìn5]69GÁÖÝ01IUI?0.Ñß57Ò*$ÌWÒ*6!Ï1$=U5=?K
=UÓÎ,01KEKM=?P!I?$
5E0I?$; C7º57Ò!$G!65EÎ!CE0 Î,$CEIUÐ69GIJ6CEÏ $:Ï $*$CW6I?=Uë
.57=?0 $;C7CE0 CWKÑa=?I?IcC7$
KM(!IU5;A
äx=¦5EÒzP*.5WÌWÒúIU$
6CE!=U*Ï*-%ÌWÒ* !Ï $
KÏ 0(!*G!$r57$
Ìn5E$;Gz *GåÑ)$01P5E =U¼CW.5EÒ!$CP*1GúC7$
KM(*I¦5WK
KM=?*Ìr$¾Ñ)$6CE$I?=UÍ1$I?Ðå570ú'< $CW6Ï1$0.< $;CÙKM$;< $;CE ICE(!IU$
K-ÑaÒ!$;C7$
K0 LOIU=?!$¾IU$
6CE!=U*Ï å=UÖ
0 Î,$CW.57$
G#Î!CE0 Î,$CEIUÐ ÀKM$;$ÙP,$I?0.ÑÚ=UåKM$
Ìn5E=U01 A!Q( Ña=?IUIv57CW ÌWÍ35EÒ!$ÌWÒ96!Ï1$;K
*GÁÐ>=?$IJG
Ï 0>0G¾ Î!Î!CE0'æ>=?Ó.5E=U01ºCE$;K7(!I¦5WKA
Û
$
KMÎ!=U57$
57Ò!$: G<. 15W6Ï1$;K®06Ö/KM570ÌWÒ*1Ke5E=?ÌRI?$; C7*=U!Ï9-65EÒ!$CE$
6CE$
KM57=?IUI,CE$; K70 9K®ÑaÒlÐ0 *$
Ó=?Ï Òl5
Ì0 *K7=?G!$CR(*K7=U!ÏP*65EÌWÒI?$;6CE!=?!Ï C
8 9 #!!3;:4&%1=<>+!-/@B%12')3"034
1A&)01*G=U57=?0 *KR0 Ö®Ì0 >< $;C7Ï1$*Ì$D C7$:Ñ)$;IUI/(!*G$;CEKM570>0GcA
E ARQ 6>Ð:1ÌÌ$I?$CW.57=?0 @5E$;ÌWÒ!*=?Ül(!$
K Ý$1A Ï9A
Ì0 P e(*Ï1.5E$Ï1CE1G=U$;l
5 01!IUÐ:0 Î!L
$;CE657$@=?3P9.5EÌWÒ3I?$;6CE!=?!Ï*A
*ARÞRÒ*$0 CE$r5E=?Ì;6I *6I?ÐKM=JK0 Öa57Ò!$Ñ4$=?Ï Òl5G!Ðl96Ó=?Ì;KÙ *G¼Ìr01l<1$CEÏ $;*Ìr$
CW.5E$;Ka C7$@KM=?ÓÎ!I?$C
A
ÞRÒ!$;K7$º G!<' l5E6Ï1$;KKM57$;Ó÷ÖÝCE0 Ó÷57Ò*$ºK7 Ó$!0 =JK7$57Ò*65ÙÓ6Í1$ºKe5E0ÌWÒ* KM57=JÌIU$
6CE!=?!Ï
G<.6l5W6Ï $;0 (*K;A.ÞRÒ!=JK®!0 =JKM$1-.ÑaÒ!=JÌWÒ=JKK70@ÌC7=U57=JÌ I>ÖÝ0 C®â*9G=U*ÏP9$5M57$;CI?0>Ì;6IÓ=?!=UÓ I?K70
Î!C7$;< $;15WK@ÖÝ(!I?I4Ì0 >< $;C7Ï1$*Ì$Ù5E0#57Ò*$ºÓ=U*=UÓ(!ÓAváh*KM57$
Gú0 ÖaÌ0 >< $;C7Ï1=U*Ϻ5E0#57Ò!$¾$ræ! Ìr5
Ó=?!=?ÓÙ(!Ó-5EÒ!$Ìr01l<1$CEÏ $;*Ìr$
KM5E IUIJK40 (5G(*$D5E057Ò!$@Ñ4$=?Ï Òl5 9(*Ìn5E(*.5E=U01*KA!ÞRÒ*$@K7=?ë$:06Ö
57Ò!$ *(*Ìn5E(*.5E=U01*KÙG!$Î,$*G¼01ú57Ò!$¾G$Ï1C7$;$06Ö!01=?K7$06ÖR57Ò*$¾KM570ÌWÒ* KM57=JÌ(!ÎG!.5E$;K;AÞRÒ*$
<' C7=J69Ìr$º0 Ö
5EÒ!$ 9(*Ìn5E(*.5E=U01*K6CE0 (!9Gú57Ò!$I?0Ì6I4Ó=U*=UÓ(!Ó =JKÎ!C701Î901CM5E=U01*6I570å57Ò*$
IU$
6CE!=U*Ï3CW.5E$ 0 E **- E (- & O0A ?>0#=?å01CEG!$C@570#CE$;G!(*Ìr$57Ò*$ *(*Ìr57(*657=?0 *KÑ4$Ì å$;=¦5EÒ!$C
G$;ÌC7$
K7G$ À6!*$;6UI v57Ò*$IU$
6CE!=?!ÏCE657$R0 CÒ*'< $a61G!6Î5E=U<1$aP*.5WÌWÒKM=?ë$1A6áh57Ò!$;0 CEÐ *-
- &!- " =U54=JKRKMÒ!0.Ña57Ò*65)5EÒ!$D01Î57=?Ó6I6!*$;6I?=?!ÏKEÌWÒ!$;G!(!IU$:06Öc5EÒ!$:IU$
6CE!=U*ÏÙCW.57$D=?K
06Ö57Ò!$@ÖÝ01C7Ó
0 .
% E
ÑaÒ!$CE$ .®=?K57Ò!$D>(!ÓÙP,$C)06ÖcÎ*.5757$;C7*KÎ!CE$;K7$l57$
G6*G =JK]Ì0 *KM5E l5;A áhÎ*CE1Ìn57=JÌr$1-65EÒ!=?K
Ó'кP9$@5E0l0Ö»1Ke5
!0 57Ò!$;CÓ$r5EÒ!0G570C7$;Ó0.< $a!01=?K7$=JK5E0Ù(*K7$ºè7Ó=?!=UL[P9.5EÌWÒ*$;KEê!-.5EÒ*.54=?K;-lKM5E CM5)Ña=¦5EÒ
#KMÓ6I?IP9.5EÌWÒ¼K7=Uë;$º69G =?*ÌC7$
K7$57Ò!$ºK7=?ë$º1K@57CW6=?!=?!Ï#Î!CE0>Ì$$
G!KAvQ6I?IU$;CÙG=JK7Ì(*KEKM$
K
0 !$Ó$r5EÒ!0G ÖÝ01C]G0 =?!Ͼ57Ò!=JK E " 6*G¼BDC7C " G!=?KEÌr(*KEK7$;K:57Ò*=?K:ÖÝ01CIU=?!$; CÎ!CE0 P!I?$ÓK;A
Ø0.Ñ4$< $;C;-lG$;Ì=?G!=U!ÏÙ57Ò!$:CW.57$:.5RÑaÒ!=JÌWÒ570=U9ÌrCE$; K7$a5EÒ!$DP9.5EÌWÒ¾KM=?ë$:6*GºÑaÒ!=?ÌWÒº=U!Î*(5EK
570=U*ÌIU(9G$@=U¾57Ò!$K7Ó6I?I,P9.5EÌWÒ*$;K4=?Ka1KRG=¦ÕÌ(!I¦51K4G!$r57$;C7Ó=?!=?!Ï57Ò!$:Î!CE0 Î,$CRIU$
6CE!=?!Ï
CE657$ A A V$;Ìr57=?< $I?Ð357Ò!$KM=?ë$06Ö5EÒ!$ÙI?$; C7!=?!ÏCW.5E$Ù=? KM570ÌWÒ* KM57=JÌI?$; C7!=?!ϺÌ0 CEC7$
KMÎ,0 9G!K
57057Ò*$:CE$;K7Î,$;Ìn5E=U<1$KM=?ë$0 Ö/57Ò*$Ó=?!=%P9.5EÌWÒ/A
065E$º6IJK7057Ò*65]57Ò!$¾Î!CE0 P!I?$Ó÷06ÖaC7$;Ó0.<l=?!Ï5EÒ!$!01=?K7$=?ú57Ò!$3G!.5W#Ó'Ð P,$IU$
K7K
ÌrCE=¦5E=?Ì;6Ic5EÒ*60 *$@57Ò!=?!ÍKaP,$;Ì (*K7$06Ö®Ï $*$CW6I?=Uë
.57=?0 /A9BD< $C757CW6=?!=?!ÏÓ'к0>Ì;Ìr(!C
IU01!Ï
P9$ÖÝ0 CE$D5EÒ!$!01=?K7$@C7$;Ï =?Ó$@=JKa$<1$3C7$
ÌWÒ!$
GcA
!0 57Ò!$;CD G!<' l5E6Ï1$06ÖP*65EÌWÒ5ECE =U!=?!Ï=JKa5EÒ*.5:0 !$=?KD6P!I?$5E0º(*K7$ÙK7$;Ìr01*G0 CWG$;C
Ó$57Ò!0G!K5E0åK7Î,$$;Gç57Ò*$3I?$; C7*=U!ÏåÎ!CE0Ìr$;KEK;A ?>$
Ìr01*Gz01CEG$;CÓ$57Ò!0G!KK7Î9$;$;GãIU$
6CE!=?!Ï
PlÐ $;KM57=?Ó657=?!Ï!065 e(9Ke55EÒ!$Ï CW G=?$l5@P!(5Ù I?K70º5EÒ!$Ìr(*C7<..5E(!CE$0 Ö5EÒ!$Ìr0lKe5]K7(!CMÖ»1Ìr$1A
8:=U<1$ú57Ò*$ºÌr(!CE<..5E(!C7$1-%0 *$ºÌ å$
Ke5E=UÓ657$5EÒ!$º6Î*Î!C70'æ=?Ó.57$I?0Ì.5E=U01ú06ÖR57Ò!$3 Ìr57(* I
Ó=?!=?ÓÙ(!ÓA
Û
$
KMÎ!=U57$57Ò!$
1G<.6l5E Ï $
K06ÖP*.5WÌWÒ(!ÎG!.5E$;K;-1KM570ÌWÒ* KM57=JÌRI?$;6CE!=?!Ï=JKKM57=?IUI906Ö×57$;57Ò*$
Î!C7$ÖÝ$CEC7$
GÓ$r5EÒ!0GºÎ* CM5E=?Ì(!IJ6CEIUÐÑaÒ!$;¾G$
6I?=U!ÏÑa=U57Òº<1$CEÐI? C7Ï1$DG!65EKM$5EK4P9$
Ì (*KM$:=U5
=?KK7=?ÓÎ*IUÐÓ(*ÌWÒ3Ö» KM57$;C;A
* "3C4 C%
.0%1
$r5eÑ40 CEÍK]I?$; C7ú5EÒ!$ºÖ»1Ke5E$;KM5ÙÖÝCE0 Ó 57Ò*$¾Ó01KM5(*!$ræÎ,$;Ìn5E$;GãK7 ÓÎ*IU$1AÞRÒ!$;C7$ÖÝ0 CE$ -v=¦5=?K
G<>=JK7 P!IU$570 ÌWÒ*0l0lKM$3ÁKE6ÓÎ!I?$¾.5$;1ÌWÒ¼=¦5E$CW.5E=U01¼57Ò*65=JK]5EÒ!$3Ó0lKe5(!Ö» Ó=?I?=? CÙ5E0
57Ò!$úKMÐKM57$;Ó3A
0657$1-457Ò!=JK#6Î!Î*IU=?$;K¾0 !I?Ðì570ìKM570ÌWÒ* KM57=JÌÁI?$; C7*=U!ÏTKM=?*Ì$ 5EÒ!$ 01CEG!$C306Ö
=U!Î*(5Î!CE$;K7$l5E657=?0 ç=JK=?C7CE$I?$<.6l5ÖÝ0 CP*65EÌWÒ,+.A)B
Ö:Ìr01(!CWKM$1-v5EÒ!$CE$=?K!0¼K7=UÓÎ!I?$Ñ4'Ð
570åÍl*0.Ñ ÑaÒ!=?ÌWÒç=U!Î*(5EK C7$¾=UÖÝ01C7Ó657=?0 zCE=?ÌWÒ/-vÒ!0.Ñ4$<1$C
-v < $CEмKM=?ÓÎ!I?$º5EC7=JÌWÍå57Ò*65
ÌrCE(*G$I?Ð =?ÓÎ!I?$Ó$l5EK]57Ò!=JK=JG$;=JK5E0ÁKM=?ÓÎ!I?Ð ÌWÒ!0>01K7$K7(*ÌÌ$;KEKM=?< $$ræ! ÓÎ*IU$
K:57Ò*656CE$
ÖÝC701Ó þ.ÿ rü 4ÌrIJ KEKM$
KKM=?*Ì$57CW6=?!=?!Ï#$ræ!6ÓÎ!I?$;K]P9$;IU01!Ï =?!Ï5705EÒ!$ºKE6Ó$ÌrIJ KEKÑa=?IUI
Ó0lKe5aI?=UÍ1$I?оÌ0 l5E =U3KM=?Ó=UIJ6Ca=?ÖÝ0 CEÓ.5E=U01%A
C&%
.%1," "
*
3C>&'
2"&3 <3;%13;
1A?Òl( $Á57Ò!$ 57CW6=?!=U*ÏzKM$5¾KM0z57Ò*65¾KM(9ÌÌr$
K7K7=?< $57CW6=?!=?!Ïz$æ ÓÎ!IU$
K
*$< $;C ÝCW6CE$I?"Ð P,$I?0 !Ï5E057Ò!$]KE6Ó$ÌrIJ KEKA
E A C7$
KM$;l5=?!Î!(5$ræ! ÓÎ*IU$
KÙ5EÒ*.5Î!CE0G(*Ìr$ I? C7Ï1$º$CECE0 CÙÓ01C7$ºÖÝC7$L
Ül(!$;l57I?Ð57Ò* $ræ!6ÓÎ!I?$;K45EÒ*.5Î!CE0G(*Ì$@K7Ó6I?I%$CEC701C;A
Xu®ol_dbM1_Ebvikj:olikfMo^;bM\
ik_Wj.seg\rbe_gh1ZZ_nDikj@{l\sefMoZ]\WD{_\r¿!_WfEse_n:{.
bed
1j>1d¿Ù_Ebhbed;b
i°seol_7be_ai¦g\:ghik^;jliª>fn\j.sbM\jl^;_ad;°%^;bM\;1ik_Wj.s
\kl_WgW©
Ø0.Ñ4$< $;C;-®01!$ÁÓÙ(9Ke5ºP,$ Ì;6CE$rÖÝ(!IÑaÒ*$xÎ,$C757(!CEP!=?!ϼ57Ò*$Á!0 CEÓ6IaÖÝCE$;Ül(!$;*Ìr=?$;K06Ö
=U!Î*(5$ræ!6ÓÎ!I?$;K:P,$;Ì (*K7$5EÒ!=JKÌWÒ*6!Ï1$;K
57Ò!$CE$IJ.5E=U<1$Ù=?ÓÎ,0 C75E *Ìr$57Ò9.5@57Ò!$!$5eÑ)01C7Í
Î!I?1Ìr$
K/01ÙG/= V,$;C7$;15$ræ!6ÓÎ!I?$;K;A'ÞRÒ!=?KvÓ'Ð@0 C/Ó'Ð@!0 5vP,$4G$
KM=?CW6P!I?$ A
î!0 C$ræ! ÓÎ*IU$1- >ÿ
D *ÿ ý ¦ÿ Eþ #þ ý hý D Oý6ÿ *ÿ >û kÿ ü ,DWý þ6ÿ ý Àü
P,$;Ì;6(*K7$01(57I?=?$CWK
=U!Î*(54<1$;Ìr570 C46CE$
Î,01K7=¦5E=U<1$ ->6I?I90 Öc57Ò!$:(!ÎG!.5E$;K406Ö%Ñ4$=?Ï Òl5WK5EÒ*.5)ÖÝ$$;Gº=Ul5E0]*0>G!$DÑa=?IUI
P9$57Ò!$KE6Ó$K7=?Ï »=ÀA $ AcK7=UÏ10 Q;nA
K:¾C7$
KM(!IU5;-957Ò*$;K7$Ñ4$=?Ï Òl5EK:Ì 01!IUÐ# IUI®G$;ÌC7$
K7$
0 C
IUI%=?*ÌC7$
K7$ Eû ü)ÖÝ01C
Ï =?< $;=U!Î*(5DÎ*65M5E$CE%A*ÞRÒ>(*K;-*=UÖÑ4$=?Ï Òl5<1$;Ìn5E0 CÓ(*KM5
< $CEÐK7IU0.ÑA
áhº5EÒ!$]6P,0.< $@$ræ! ÓÎ*IU$1->57Ò!$=?!Î!(5WKaÑ4$CE$: IUI%Î,01K7=U57=?< $ A!Ø
0.Ñ)$;< $;C;->=?3Ï1$!$;CE IÀ- lÐ
KMÒ!=UÖ×506Ö)57Ò!$'< $;CE Ï $]=?!Î!(5]'ÑR'Ð3ÖÝC701Óë$CE0ºÑa=?I?IP!=?1KD57Ò!$(!ÎG!.5E$;K@=? ¾Î*6C757=JÌr(*I? C
G=UCE$;Ìr57=?0 Á *G357Ò>(*KDKMI?0.Ñ G!0.Ña#IU$
6CE!=U*Ï*A*ÞRÒ!$;C7$ÖÝ0 CE$ -!=U5D=JKÏ10>0>G35E0ºK7Ò!=¦Ö×5D57Ò!$Ù=U!Î*(5EK
KM05EÒ*.5D57Ò!$Ù'<1$CW6Ï1$:0.<1$CR5EÒ!$57CW6=?!=?!ϺK7$r5
=JK
ÌIU0lKM$@5E0ë$;C709AÁÞRÒ!=JKÒ!$;(!C7=JKM57=JÌ]KMÒ*0 (!IJG
P9$º Î!Î!I?=U$
G¼.5 IUI)I?'Ð1$CWK@ÑaÒ!=JÌWÒ¼Ó$
6*K]57Ò*65ÙÑ4$Ñ4 l557Ò!$º'<1$CW6Ï1$0 Ö45EÒ!$ !
06Ö4¾!0G$570P,$ÌIU0lKM$Ù570ë$;C70¾P9$
Ì (*KM$57Ò*$;K7$01(57Î*(5EK C7$Ù57Ò!$=?!Î!(5WK:570357Ò!$!$æ>5
<' C7=J6P*IU$
*G , =JK57Ò!$ Ì0 ÓÎ901!$l5406Öc57Ò*$ 57CW6=?!=?!ÏÙ$æ!6ÓÎ!I?$ $A ?Ì IU=?!ÏÙK7Î9$;$;G!K
y
ω2
Lines of
ω1 ω2 constant E
ω1
z1 z2
KL-
Expansion
Covariance
Equalization
ÑaÒ!01K7$@0 (5EÎ!(5EK
6CE$: IUÑR'ÐK)Î,01K7=U57=?< $6*GK70ÓÙ(*KM5aÒ*'< $Ó$;6¾57Ò9.5=JKaÎ90lKM=U57=?< $1A
"04
&" 9
1A?ÐlÓÓ$r5EC7=J̺K7=?Ï Ó0 =JG!K]K7(*ÌWÒz KÙÒ>ÐlÎ,$CEP,0 I?=?Ì5W6!Ï1$l5Ù0 Ö×57$ãÌr0 ><1$CEÏ $Ö» KM57$;C
5EÒ*635EÒ!$]Ke5W6*G*6CWGI?0 Ï =JKM57=JÌ
ÖÝ(**Ìn5E=U01%A
E A CE$;Ìr01ÓÓ$;*G$
GzKM=?Ï Ó01=?G - )=J6K C X5!( "- 5W6!Ò
FV <A ?>=?*Ì$57Ò*$
5W6!ÒÁÖÝ(!*Ìn5E=U01Á=JKDK70 Ó$r5E=UÓ$
K
Ìr01ÓÎ!(5E657=?0 * IUI?Ð3$ræÎ9$;*K7=U<1$ -6Á6Î*Î!C70'æ=?Ó.L
5E=U0106Öv=U5P>кCW.57=?006Ö®Î901IUÐ>!01Ó=J6IJKRÌ 3P,$(*K7$;G¾=U9Ke5E$; G%A
*A ?0 Ó$r5E=UÓ$;K=U5=JKÒ!$I?ÎÖÝ(!I>5E0:1G!GÙ:K7Ó IUIlI?=U!$
6C57$;C7Ó-'$ A Ï*A x5E6*0Ò 7
K70 K4570'<10 =JG 965
KMÎ,065WKA
1.5
1
1
0.8
0.5
0.6
-3 -2 -1 1 2 3
0.4
-0.5
0.2 -1
-1.5
-6 -4 -2 2 4 6
(a) (b)
ÞRÒ!$
Ìr01*KM5E6l5WK=?57Ò!$DC7$
Ìr01ÓÓ$;*G$;GºKM=?Ï Ó0 =JGÏ =?< $;6P,0.< $aÒ*'<1$P,$$¾ÌWÒ!01K7$K70
57Ò*65;- Wþÿ Àü7ý .ü Wþ¾ÿ »K7$$Î!CE$<>=?0 (*KG!=?KEÌr(*KEK7=U01 n-*57Ò!$]<. C7=J6*Ì$
06Ö5EÒ!$0 (!57Î!(5WK@Ña=UI?I6IJK703P,$ÌrI?01K7$Ù5E0 P,$;Ì;6(*K7$57Ò!$$6V,$
Ìn5E=U<1$Ïl6=? 0 Ö57Ò*$KM=?Ï Ó0 =JG
=?K
C701(!Ï Ò!I?Ð]0.<1$CR=U5EK(9KM$ÖÝ(!IvCW6*Ï $ Aáh#Î96C757=JÌr(!IJ6C
->57Ò!=JK:KM=?Ï Ó0 =JG3Ò* Ka57Ò!$]Î*C701Î9$;CM5E=U$
K
» 1 - ÝP a57Ò!$KM$
Ìr0 9G#G$CE=U<.657=?< $=JK
Ó.æ=?ÓÙ(!Ó .5 1-9 *G »Ì R57Ò*$
4$ V$;Ìn5E=U<1$Ï1 =U¾=?KÌIU0lKM$:5EB0 1A
BD!$06Ö:57Ò!$#Î,065E$l57=J6IaÎ!CE0 P*IU$;ÓKÑa=U57Òx(9KM=?!ϼKMÐ>ÓÓ$r5EC7=JÌKM=?Ï Ó0 =JG!K=JK57Ò9.557Ò*$
$CEC701C4K7(!C7Ö» Ìr$@Ì ºP,,$ 5 ü,.5a!$; C457Ò!$@0 CE=?Ï =?%Aî!0 C45EÒ!=?KRCE$;1KM01º=U5a=?KRÏ10l0G570'< 01=?G
=U!=U57=J6I?=?ë=?!ϾÑa=U57Òå< $;C7Ð#K7Ó6I?IÑ4$=?Ï Òl5WKAc24$;Ì;6(*K7$Ù0 Ö57Ò!$KE.5E(!CE657=?0 Á0 Ö57Ò*$K7=?Ï Ó0 =JG!K;-
57Ò!$$CEC701CDK7(!C7Ö» Ìr$=JKD I?K70 965DÖ» C:ÖÝC701Ó 5EÒ!$0 CE=UÏ1=U/A DG!G=?!Ï3¾KMÓ IUI®I?=U!$
6CD57$;C7Ó 5E0
57Ò!$]K7=UÏ1Ó0 =JG¾Ì;6KM01Ó$57=?Ó$;KRÒ!$;IUÎ#'< 01=?G57Ò!$ ,.5CE$Ï =?0 9KA
&"034 2')4<%
!.*C%1
há ÌrIJ KEK7=¦â9Ì;.5E=U01]Î!CE0 P!I?$ÓK;-;5W6CEÏ $5v<.6I?(!$;K® C7$5eÐlÎ*=?Ì;6I?IUÐP!=?* C7Ð Ý$1A Ï9A 3 L
- 7 Q=PnA'&)01ÓL
Ó01ºÑa=JK7G!0 Ó Ó=UÏ1Òl5RK7$$;Óõ570KM(*Ï Ï $
Ke5)57Ò*65)5EÒ!$:5E C7Ï1$r5)<. IU(!$
K4P9$K7$r5a.5457Ò!$@<.6I?(!$:06Ö
57Ò!$]K7=UÏ1Ó0 =JG R Ka1KMÐ>ÓÎ5E0657$
KA!Ø
0.Ñ)$;< $;C;-157Ò!=JKaÒ*1KaKM$;< $;CE IG!CE'ÑaP*1ÌWÍKA
îv=?CWKe5
-;=?*KM5E6P*=UI?=¦5E=U$
K®Ì6ÙC7$
KM(!IU5;A.ÞRÒ!$)57CW6=?!=?!Ï
Î*C70Ìr$
K7K/Ña=UI?Il57CEÐ@570:GC7=?< $57Ò!$40 (!57Î!(5
KaÌrI?01K7$@ KRÎ,01KEKM=?P!I?$D5E057Ò!$:5W6CEÏ $r5a<.6I?(!$
K-ÑaÒ!=JÌWÒ#Ì ¾01!IUкP9$ ÌWÒ*=U$;< $;G3 K7Ð>ÓÎ570 57=UL
Ì6I?I?Ð A DK)ÙC7$
KM(*I¦5
- 5EÒ!$
Ñ4$=?Ï Òl5EK Ý01(57Î!(!5R *G$<1$Ò!=JG!G$; C7$DGC7=?< $;570I? C7Ï1$C *G
I? C7Ï1$C<' IU(*$;KÑaÒ!$CE$)57Ò!$K7=UÏ1Ó0 =JGÙG$;C7=?<..57=?< $)=?KÌIU0lKM$)570@ë;$CE0*A'ÞRÒ!$R< $CEÐIJ6CEÏ $Ñ4$=?Ï Òl5WK
=U*ÌC7$
K7$5EÒ!$DÏ1CE1G=?$l5EK;- Ò!0.Ñ4$<1$C
-65EÒ!$;K7$DÏ1CE1G=?$l5EK)6CE$57Ò*$ºÓ(!IU57=?Î!IU=?$;G¾P>Ð6º$æÎ90 L
!$l57=J6I?I?ÐK7Ó6I?I9K7=UÏ1Ó01=?GG!$CE=U<..5E=U<1$ Ý$æ!Ìr$Î!5ÑaÒ!$¾@5eÑa=?KM57=?!Ï5E$CEÓF=JK G*G$;G5E0]57Ò*$
KM=?Ï Ó0 =JG 4Î!CE0G(*Ì=U!ÏÑ4$=?Ï Òl5(!ÎG!.5E$]ÌrI?01K7$@570ë;$CE0*"A
K
C7$
KM(!IU5;-!57Ò!$Ñ4$=?Ï Òl5EKÓ'Ð
P9$
Ìr0 Ó$]KM57(*ÌWÍA
?>$;Ì0 *Gc-,ÑaÒ!$;#57Ò!$0 (!57Î!(5WKDKE.5E(!CW.57$1-*5EÒ!$Ù*$r5eÑ40 CE;Ï1=U<1$;K*0º=?*G=JÌ.5E=U01Á06Ö4Ìr0 !L
â9G$9Ìr$]I?$<1$I[A*äxÒ!$;Á6#=?!Î!(5
Î9.5M5E$CEÖ»6I?I?K
!$; C
G$
Ìr=JKM=?0 P,0 (!9G!6CEÐ5EÒ!$]0 (!57Î!(5
ÌrIJ KEK=?K(**Ìr$;CM5W6=?%AáeG$; IUI?Ðz5EÒ!=JKKMÒ*0 (!IJGTP9$#CE$ *$
Ìn5E$;GT=UT5EÒ!$#!$r5eÑ40 CEÍzP>Ðã ã01(5ML
Î!(5@<' IU(*$]57Ò9.5D=JKD=UÁP,$r5eÑ4$$;#57Ò*$]5eÑ40Î90lK7K7=?P!IU$Ù5E C7Ï1$r5
<.6I?(!$
K-,=ÀA $ A,!0 5D!$
6C
$;=¦5EÒ!$C
K7ÐlÓÎ5E065E$ A'Ø0.Ñ4$< $;C;-I? C7Ï1$Ñ)$;=UÏ1Òl5EK/57$9G570
ÖÝ0 CWÌr$46I?I101(57Î!(!5EKv570
57Ò!$)5E =UIJK/0 Ö57Ò!$RK7=UÏ L
Ó01=?GCE$Ïl6CWGI?$;KEK®06Ö,57Ò!$D(!*Ìr$;CM5W6=?l5eÐ A ÞRÒ>(*K;-65EÒ!$!$5eÑ)01C7ÍÓ'ÐÎ!CE$;G=JÌn5)@ÑaCE0 *Ï]ÌrIJ KEK
Ña=¦5EÒ!0 (5:Ï =?<>=U!Ͼ6>о=?*G=JÌ657=?0 Á0 Ö=¦5WKI?0.ÑÚÌr01â9G$;*Ìr$]=?#5EÒ!$ÙCE$;K7(!IU5;A"/6CEÏ $Ñ4$=?Ï Òl5WK
57Ò*65DK7657(!CW.5E$:57Ò!$]*0>G!$;KaÓ6Í1$@=¦5
=?ÓÎ90lK7K7=?P!IU$@5E0ºG/= V,$;C7$;l57=J.57$P,$r5eÑ4$$;35eÐlÎ*=?Ì;6I/ *G
!0 l5eÐ>Î!=JÌ6Ic$æ!6ÓÎ!I?$;K;A
KM01IU(5E=U01ç5E0ú57Ò!$
KM$#Î*C701P!IU$;ÓK=?K570¼KM$557Ò!$Á5E6CEÏ $5<' IU(*$;K5E0åP,$ÁÑa=U57Ò!=?ì57Ò*$
CE !Ï $0 Öc57Ò!$:KM=?Ï Ó01=?Gc-1CE657Ò!$;C5EÒ*6º655EÒ!$:1KMÐ>ÓÎ5E0657=JÌa<.6I?(!$
KA!&46CE$ÓÙ(9Ke54P9$D5E Í $/-
Ò!0.Ñ)$;< $;C;->5E0=U9KM(!CE$5EÒ*.55EÒ!$]!0G$Ù=?K
!065
CE$;KM57CE=JÌn57$
G357001!I?Ð57Ò*$ÙI?=U!$
6CÎ96C75
06Ö®57Ò*$
KM=?Ï Ó0 =JGcQA ?>$5M57=?!Ï:57Ò*$5W6CEÏ $r5®<.6I?(!$;Kv5E0
5EÒ!$RÎ901=Ul506Ö!5EÒ!$RÓ.æ=UÓ(!ÓÚK7$;Ìr01*GÙG!$CE=U<..5E=U<1$
0 ú57Ò*$3K7=UÏ1Ó0 =JG =JKÙ57Ò!$¾P9$
Ke5Ñ4'Ð 570#5W6Í1$º G<. 15W6Ï1$06Öa57Ò*$º!01!IU=?!$
6CE=¦5eÐåÑa=¦5EÒ!0 (!5
K7657(!CW.5E=U!Ï 57Ò*$ÁKM=?Ï Ó0 =JGcAÞRÒ!=JK=JK6!0 57Ò!$;CC7$
K70 ç57Ò!$ÁK7=UÏ1Ó0 =JGç=?xîv=?Ï (*C7$ PT=JK
Ï 0>0G#ÌWÒ!0 =JÌr$1A*áO5
Ò*1KÓ.æ=?ÓÙ(!ÓK7$;Ì0 *GG$;C7=?<..57=?< $.5 ]ÑaÒ*=?ÌWÒÁÌ0 CEC7$
KMÎ,0 *G¾5E057Ò*$
P!=U96CEÐ5E C7Ï1$r5a<.6I?(!$;K45eÐ>Î!=JÌ Ic=U#ÌrIJ KEK7=¦â9Ì;.5E=U01ºÎ!CE0 P!I?$ÓK;A
2')4&%(
&)Ò!0>0lKM$R5W6CEÏ $r5<. IU(!$
K.557Ò!$Î,0 =?l506Ö,57Ò*$Ó.æ=UÓ(!ÓõKM$
Ìr0 9GG!$CE=U<..5E=U<1$a0 57Ò*$
K7=UÏ1Ó0 =JG¾K70 K4570'<10 =JGºKE.57(*CE657=?!ÏÙ5EÒ!$0 (!57Î!(5D(!!=U5EK;A
t s[ikgOseikjl^Ôse_EbeZ ikgõ\ghZ]\;kìkikjl_W\bõse_EbeZ \;l1_W sedÔseo1_jld61_íd
seql sn}ú_
© ^1©
åsM\;j1o
©
3" "0!." "034 C%%("4&;
+ EF
F
J ,!
5 %
ÞRÒl(9K-'5E0:=?*K7(!CE$5EÒ*.5®57Ò*$ C7$R6Î!Î*C70'æ=?Ó657$;IUÐ 457Ò!$RÑ4$=?Ï Òl5EK®K7Ò!0 (*I?GP9$RCW6*G!0 ÓIUÐ
GCE'Ña¾ÖÝCE0 ÓíG=JKe5EC7=?P!(5E=U01Ña=¦5EÒ3Ó$
63ë$CE06*G3Ke5W6*G*6CWG¾G$;<>=?657=?0 3Ï =?< $3P>Ð
+ 7F
%"
ÑaÒ!$CE$ Ô=?K45EÒ!$>(!ÓÙP,$Ca0 Ö=?!Î!(!5EKR5E057Ò!$(!*=¦5
A
3" "!.0" "3C4 %1"04<;
DK7K7(!Ó=U*Ï5EÒ*.5 C
?
I ;
$ 7
C *
U
= !
Ï E
C 6
7
5
$ K4K7Ò!01(!I?GºP9$@Î!CE0 Î,0 C757=?0 96I957057Ò!$KEÜ1(96CE$
CE0l0 540 Ö%57Ò*$
>(!ÓP9$;Ca06Ö®=U*Î!(5EKa57057Ò!$(*!=¦5
Ñ4$=?Ï Òl5WK@=U¼I?0.Ñ)$;CI?'Ð1$CWKKMÒ*0 (!IJG 5eÐlÎ*=?Ì;6I?IUÐ P9$IJ6CEÏ $;C:57Ò* ¼=Uå57Ò*$
Ò*=UÏ1Ò!$CaIJ'Ð $;CEK
B
57Ò!$;CR57CE=?ÌWÍK)ÖÝ0 Ca=?ÓÎ!C70.<>=?!Ï57Ò!$]Ì0 >< $;C7Ï1$*Ì$
=?*ÌrI?(*G$ C
%(3;*
Q#01Ó$l57(!Ó
. 7 < 0 ( + 7 /%. 6
065E$º5EÒ*.55EÒ!=JKKM$50 Ö
CE(!IU$
K=?K$
K7Ðú570¼Ìr01ÓÎ*(57$#69GçKM57CW6=?Ï Òl5MÖÝ01C7ÑR6CWGå570ú=?ÓL
Î!IU$;Ó$l5;Acäå$KM=?ÓÎ!IUÐ Ò*'< $570Í $;$Î 57CW ÌWÍ#06Ö46ú G!G=U57=?0 96I®< $
Ìn5701C@=U AÜ,8A * -C,57Ò*$
'< $CW6Ï1$;G:Ï CW G!=U$;15 ' A.ÞRÒ!$)!0 CEÓ 06Ö5EÒ!=?K<1$;Ìn5E0 C/57Ò!$;Ìr01l57CE0 IJKc57Ò*$4K7=Uë;$0 Ö57Ò!$4IU$
6CE!=?!Ï
CE657$ »K7$$ AÜ,8A - %rA ÞRÒ!$
IUÏ10 CE=¦5EÒ!ÓÚÖÝ01IUI?0.ÑK5EÒ!$
K7=UÓÎ!I?$=?15E(!=U57=?0 C1Ö»6C4'Ñ4'ÐÖÝCE0 Ó57Ò*$
Ó=?!=?ÓÙ(!Ó »I? C7Ï1$ÙG=JKe5W6*Ì$
=U5DÎ!CE0Ìr$$
G!KD=U P!=UÏ3Ke5E$Î*K:69G ÌrI?01K7$5E057Ò*$Ó=?!=UÓ(!Ó
=¦5
!!$; I?K45EÒ!$I?$;6CE!=?!ÏCE657$ ×ÖÝ01CR57Ò!$;0 CE$r57=JÌ I%G$r5W6=?I?K
KM$;$ E * UrA
9 "!. +!"0"!C*C36"0<3C # "04
&" 9# 3C"0
IU57Ò!01(!Ï Ò¼Ó01KM5K7ÐKe5E$ÓK(*K7$!0G$;KÙP* K7$;Gå01zG065ÙÎ*C70G(*Ìr5EK] *G¼K7=UÏ1Ó01=?G*K-%Ó lÐ
0657Ò*$CR5eÐ>Î9$
KR06Ö(!!=U5EK Ý01CaI?'Ð1$CWK#Ì ¾P,$(*K7$;GcA" Ìr01ÓÓ013 I¦5E$CE*.5E=U<1$D=JK457Ò*$:CW G!=? I
P* K7=?KÖÝ(!*Ìr57=?0 ÀN2Rî !$5eÑ)01C7Í'»K7$$ (- E &!- "!- E UáhºN2Rîå!$r5eÑ40 CEÍK- 57Ò!$:G065RÎ!CE0>GL
(*Ìn50 Ö95EÒ!$aÑ)$;=UÏ1Òl5 *G=U*Î!(5< $
Ìn5E0 C®=?KC7$;Î!I?1Ìr$
GÙÑa=U57ÒGA(*ÌIU=JG$
6G=JKe5W6*Ì$4P,$r5eÑ4$$;
L<M7N ¹*¹ d
j6'_Ebe^;_WjlfW_d°/seo1_>dna©>pR beikjl^@seol_Rªlj>\,gOsM\;^;_Dd°%k_n\bej1i¦j1^seol_
\n'_EbM\;^;_>dnTikg
\qlq bedr½ ikZ]\se_Ek]d;jl_a ikZ_Wjlghikd
jl\;sedn\rbM1g®seol_)ZikjlikZ:1ZR\;jlÙisikg\D^;d.d6\;q1q1bedn½ ikZ]\|
seikd;jd;°seo1_aZikjlikZ:1Z_Wik^;_Wj6
\¦1_1ibe_WfEseikd;jd;°cseo1_ _Wghghi¦\j9©
57Ò!$D=U!Î*(54 *GÑ)$;=UÏ1Òl5)6*G5EÒ!$DK7=?Ï Ó0 =JG=?KC7$;Î!IJ Ìr$
GP>Ð $ræÎ901!$l5E=? IÀAlÞRÒ!$0 (!57Î!(5
Ìn5E=U<>=U5eÐ=JKÌr01ÓÎ*(57$
Gc-!$ A Ï*A>ÖÝ01Ca0 !$@01(57Î!(!5;-*1K
J
, $ræÎ
F E ,F F
, F
, Kv+
ÑaÒ!$CE$ , , =JK5EÒ!$
Ó$
6 »KM5E *G! CEGG$;<l=J.5E=U01 ®0 Öc57Ò!$ [L[57Òº8@ (*K7K7=J6%AlÞRÒ!$;K7$(!!=U5EK
Ì6 C7$;Î!I?1Ìr$0 CDÌ0>$ræ=?KM5DÑa=U57Ò 57Ò!$KM5E69G!6CWG#(!!=U5EK@ *G57Ò*$ÐÁ6CE$](*K7(* IUI?Ð357CW6=?!$;G#P>Ð
ω ω
ωmin b) ωmin
a)
ω
ωc ωmin
E(ω) E(ω)
η > 2 ηopt dE/dω
η > ηopt
ω
ωc ωmin
dE(ωc)
dω
ω ω ∆ω
ωmin ωmin
i
c) d)
iki
L<M7N ¹*¹ M
b
\ ¦
i E
_ 6
j )
s W
_ h
g W
f W
_ .
j
s J
°
d R
b
i !
¿ E
_ e
b W
_ .
j
s k
n
_
\ e
b 1
j i¦j1^@bM\se_WgW©
äxÒ*.5=JK57Ò!$#01Î57=?Ó6Ia<.6I?(!$06ÖD57Ò!$#I?$; C7!=?!ÏåCE657$ 0 (Á"%$r5(9Kâ*CEKM5Ìr01*KM=JG$;C
57Ò!$ÙÌ; K7$@=U rLhG=?Ó$*K7=U01%A DK7K7(!Ó=U*Ï57Ò*65: Ì;6P,$Ù6Î!Î*C70'æ=?Ó657$
GP>оÜl(*1GCE657=JÌ
ÖÝ(!*Ìn5E=U01%- 0 ( Ì;6#P,$G$;C7=?< $
GPlÐ3â*CWKM5D$ræÎ* *G=?!Ï =? Þ'Ð>IU01C
K7$CE=U$
K
6P,0 (!5
57Ò*$
Ìr(!CEC7$;l5RÑ4$=?Ï Òl5
- DC
< & 7 F &
- 7 E F F 6F & F - 7 565D54
E
ÑaÒ!$CE$]Ñ4$](*K7$]57Ò*$K7Ò!0 C757Ò* *G
K A,áOÖ =JK
Ül(*1GCE657=JÌ@57Ò*$K7$;Ìr01*G
0 CWG$C]G$;C7=?<..5E=U<1$=JK]Ì0 *KM5E 15] *G 57Ò!$Ò!=?Ï Ò*$C]01CEG$;C:57$CEÓK<.6!=JK7Ò%A/ÛDW= V$CE$l57=J.5E=U*Ï
P90 57Ò#KM=JG$
KRÑa=¦5EÒ3CE$;K7Î,$;Ìn5a5E&0 5EÒ!$Ï =?< $
K
& &
- 7 F
- 6F & - 5 EXE
F
?>$r5757=?!BÏ ,7.#6*G#*0657=?!Ï5EÒ*.5 & ,7. -9Ñ)$6CE$]IU$Ö×5:6Ö×57$CDC7$
6C7L
CE !Ï =?!ÏÑa=¦5EÒ +
F D
,7.& F
F
- 5
-
E
&)0 ÓÎ*6CE=?!Ïå57Ò!=JKÑa=U57Òì57Ò*$Á(!ÎG!.5E$Á$;Ül(*657=?0 E n-Ñ4$â**GT5EÒ*.5¾Ñ)$ÁÌ;6TC7$
ÌWÒì
Ó=?!=?ÓÙ(!Ó =U01!$Ke5E$Î=¦Ö
0 (<
F & + 5 E
F
®$CEÒ*6Î*K® $; K7=?$CÑR'Ð]57001P5E =U5EÒ!=?K)K7 Ó$aC7$
KM(!IU5=JK=?IUI?(*KM57CW.57$
G=Uî=?Ï (!CE$ & »=UU= nA
ÞRÒ!$¼P90 5M5E0 Ó Ï1CE Î!Ò Î*IU0 5EK357Ò*$¼Ï CW G=?$l5#06Ö 1K#ãÖÝ(!*Ìn5E=U01 06&Ö A ?>=?*Ìr$ =?K
Ü1(9 GCW.5E=?Ì -v5EÒ!$#Ï CW G!=U$;15=JKKM=?ÓÎ!IUÐã KM57CW6=?Ï Òl5IU=?!$#Ña=U57ÒT<' IU(*$3ë;$CE0å6557Ò!$#Ó=?!=¦L
ÓÙ(!Óõ6*G
.55EÒ!$
Ì(!C7CE$l5Ñ4$=?Ï Òl5 A DF 6F =JKK7=UÓÎ!I?Ð]5EÒ!$
K7IU01Î9$R06Ö5EÒ!=?K
IU=?!$]69Gº=JKÌr01ÓÎ!(57$
G3(*KM=?!Ï57Ò*$]Ke5W6*G! CEG¾KMI?0 Î,$@ÖÝ0 CEÓÙ(*I?
F F & F ,/. F 5 E "
?>0 I?<>=U!ÏÖÝ01:C ,7.º5EÒ!$Ï1=U<1$;K4$;Ül(*657=?0 E nA
äxÒ!=UI?$#5EÒ!$ÁI?$;6CE!=?!ϼCW.57$5EÒ*.5¾Ï =?< $;KÖ» KM57$;KM5¾Ì0 >< $;C7Ï1$*Ì$¾=JK 0 ( -57Ò*$ÁI? C7Ï1$;KM5
IU$
6CE!=U*ÏTCW.5E$å57Ò*65 Ì6 P,$¼(*K7$;G Ña=¦5EÒ!0 (!5 Ì (*KM=?!ÏìG=?< $CEÏ $;*Ìr$ú=JK » I?K70xKM$;$¼îv=?Ï6L
(!C7$'& ÝU= eG
0 + E 0 (
5 E &
áO1Ö =?K@!0 5$ræ! Ìr57I?ÐÁÜ1(9 GCW.5E=?ÌÙ57Ò!$; 5EÒ!$Ò!=?Ï Ò!$;C@0 CWG$CD57$;C7ÓK@=? $
Ü1(9.57=?0 E :6CE$
!065Î!CE$;Ì=?K7$I?мë$;C70ú6*G E Ù=?K01!I?Ðz6ì6Î!Î!CE0'æ=UÓ657=?0 %AáhTK7(*ÌWÒì Ì; K7$ -®=U5Ó'Ð
5E6Í1$]ÓÙ(*I¦5E=UÎ!I?$Ù=U57$;CE657=?0 *K570¾IU0Ì657$5EÒ!$Ó=?!=?ÓÙ(!Ó $;< $#ÑaÒ*$Á(*K7=?!Ï 0 ( -9Ò!0.Ñ4$<1$C
-
Ìr0 ><1$CEÏ $9Ìr$DÌ;6Ke5E=UI?I%P,$]Ül(!=¦5E$@Ö» KM5;A
áh ÓÙ(*I¦5E=UÎ!I?$G=UÓ$;*KM=?0 9K-cG$57$CEÓ=U*=U!Ï 0 (a=?K@¾P!=¦5@Ó0 CE$G=UÕÌr(!IU5P9$
Ì (*KM$57Ò*$
C7=?Ï Òl5
K7=?G$0 Ö E 4=?KÓ657CE=¦æ +DÑaÒ!$;C7$ =JK
Ì;6I?IU$
Gº5EÒ!$]Ø$
K7K7=J6ÑaÒ!01K7$]Ìr01ÓÎ90 L
!$l5EK
6CE$DÏ1=U<1$3PlÐ
,! ,DF E X(
Ña=¦5EÒ
¼-* *G $
Ü1(96I5705EÒ!$@570 5E6I/l(*ÓÙP,$Ca06Ö®Ñ4$=?Ï Òl5EK;A
=JK
Ó$;1KM(*C7$406Ö957Ò!$aÌr(*C7<..5E(!CE$)0 Ö A.áh5eÑ)0@G=?Ó$*K7=U01*K;-'57Ò!$RI?=U!$
K®06ÖÌr0 9Ke5W6l5
E
ωmin,2
\ {
ωmin,1 ω1 ν1
L<M7N ¹!¹ y9ikjl_Wgd°cfWd
j1gOsM\;j.s©
ÑaÒ!$CE!$ Ií=?K:5EÒ!$>(!ÓÙP,$C@0 Ö57CW6=?!=?!Ï< $
Ìn5701CEK;AÞRÒ!$Ø$;KEK7=? =? 5EÒ!=JKÌ K7$57(!CE*K@0 (!5
57Ò!$P,$@57Ò!$]KE6Ó$ K45EÒ!$]Ìr0.<. C7=J6*Ì$
Ó.5EC7=Uæº0 Öv5EÒ!$=U*Î!(5EK;-
I J + 5 E -
ÞRÒl(9K-!$
ÌWÒ$=?Ï $;><' IU(*$:0 Ö =JK6IJK70Ó$
K7(!C7$0 Öv5EÒ!$ÙÌ0.<' C7=J69Ìr$:0 CK7Î!C7$
G¾06Ö®57Ò*$
=U!Î*(5EK IU01!Ï57Ò!$]Ì0 CEC7$
KMÎ,0 9G=U*ÏÙ$=?Ï $;*G=?C7$
Ìn5E=U0131KaKMÒ!0.Ña3=?3î=?Ï (!CE$'*!A
x2
x1
$;Ül(*.5E=U01¾Ña=?I?I/Ìr01l<1$CEÏ $1A
Ø0.Ñ G0>$;KÙ57Ò!=JKÒ!$I?ÎzÑa=U57ÒãÌWÒ*0l0lKM=?!Ï#5EÒ!$ºI?$; C7*=U!Ï CE657$;K ºáeG$; IUI?ÐúÑ)$ÑR6l5G=¦Ö×L
ÖÝ$CE$l5ºI?$;6CE!=?!ÏúCE657$
K6I?0 !Ïú5EÒ!$ G/= V$CE$l5º$=?Ï $;*G=?C7$
Ìn5E=U01*KAÞRÒ*=?Kº=?KK7=?ÓÎ*IU$Á=UÖ57Ò*$
$=?Ï $9G=UCE$;Ìr57=?0 *K:6CE$ÙI?=?!$;G (!Î Ña=U57Ò 57Ò!$Ì0l01CEG!=U*657$.æ$;KD06Ö5EÒ!$Ñ)$;=UÏ1Ò15WKA,áhúKM(*ÌWÒ
Ì K7$ ->5EÒ!$Ñ4$=?Ï Òl5EKR6CE$@(!*Ì0 (!Î!I?$;G369G¾Ñ4$Ì 31K7K7=UÏ1º$
ÌWÒ3Ñ)$;=UÏ1Ò15a=U5EKR0.Ña3IU$
6CE!=?!Ï
CE657$
P9 K7$;G0157Ò*$:Ì0 CEC7$
KMÎ,0 *G!=U!ÏÙ$=?Ï $><.6I?(!$1AlØ
0.Ñ)$;< $C
- =UÖ%5EÒ!$DÑ4$=?Ï Òl5WK) C7$:Ìr0 (*Î!IU$
G
57Ò!$;Ñ)$aÓ(*Ke5â*CEKM5CE065E657$ K7(*ÌWÒ57Ò*65 =?KG=? Ï 01*6I[-.=ÀA $ A 57Ò!$DÌr0>0 CWG=?*.5E$R6æ$;KI?=U*$
(!ÎÑa=U57Ò5EÒ!$
$;=UÏ1$*G!=UCE$;Ìr57=?0 *K »K7$$Dî=?Ï (!CE$ (.P nA>ÞRÒ!=JK=JK57Ò!$DÎ!(!CEÎ90lKM$D06Ö/G=J6Ï10 * IU=?ë=?!Ï
57Ò!$]Ø
$;KEKM=J6G=JK7Ì(*K7K7$;G¾$; C7I?=?$C
A
"%$r5 P,$@57Ò!$CE065W.57=?0 ¾Ó.57CE=Uæ¾KM(9ÌWÒº5EÒ*.5
+
#
ÑaÒ!$CE$ =?KG!=? Ï 0 96Ic6*G + 9A!ÞRÒ!$]Ìr0lKe5RÖÝ(!*Ìr57=?0 357Ò*$ÁÌ ¾P,$ÑaCE=¦5757$#1K
0 E+
# *
=Uì0 CWG$;C5E0z'< 01=?GãG=U<1$CEÏ $;*Ìr$1-ÑaÒ*$CE$ + =JK57Ò!$#IJ6CEÏ $
Ke5$;=UÏ1$><.6I?(!$¾0 Ö zAî!01C
Ö» KM57$;KM5
Ìr01>< $CEÏ $;*Ìr$DÑ)$@Ò*'<1$
0 ( # -
+ 5
áOÖ /, .3=JKIU0 5
KMÓ IUI?$CR5EÒ*6 + 57Ò!$;Ì0 >< $;C7Ï1$*Ì$DÑa=?I?I%P9$<1$CEкK7IU0.Ñ 6I?0 !Ï57Ò*$
,7.ÙG=?C7$
Ìn5E=U01%A;áhÙÖ» Ìr5;-.Ìr01l<1$CEÏ $;*Ìr$5E=UÓ$4=?KÎ!CE0 Î,0 C757=?0 96I65E05EÒ!$aÌr0 9G=¦5E=U01]>(!ÓÙP,$C
0. %1
@B"03%12' A%( & ' îv=?Ï (*C7$ G=JKMÎ!IJ'ÐKåK7$r50 Ö $æ!6ÓÎ!I?$;KGCW'ÑazÖÝCE0 Ó 5eÑ40
@8 6(*KEKM=J6G=?KM57CE=?P!(57$
GÙÌI?1K7K7$;KvÌr$l5E$CE$;G]65 ÝL A - L A * *G A - A * nA'ÞRÒ!$)$=?Ï $><.6I?(!$
K
1.4
1.2
0.8
y 0.6
0.4
ω2
0.2
ω0 ω1
−0.2
−0.4
χ0 χ1
−0.6
−0.8
−1.2
−1.4
−1.4−1.2 −1 −0.8−0.6−0.4−0.2 0 0.2 0.4 0.6 0.8 1 1.2 1.4
u®d]fW¦\ghgh_Wga bM\Wj°Ubed
ZÚ^'\lghghi¦\;j¾1ikgOshbei|
l{L 1M/N seik¹ d;Ojl9g¹ fW_Wj.se_Ebe_W\s |À © £ } |À6© ¤ )\j> 6© £1} © ¤7©
îv=?Ï (*C7'$ E K7Ò!0.ÑK:5EÒ!$ºKE6Ó$$ræ!6ÓÎ!I?$(*K7=U*ÏÁKe5E0>ÌWÒ9 KM57=JÌ=?*KM57$
Gå06ÖaP*65EÌWÒúÓ0>G!$
IU$
6CE!=U*Ï*A6Ø$;C7$1- @I?$;6CE!=?!Ï@CE657$R06Ö 0 5 E =JK(*K7$;GcAlBD*$Ì6K7$$R5EÒ*.55EÒ!$a57CWPe$
Ìn5E0 CEÐ
=?KÓÙ(9ÌWÒç!01=?K7=U$;CÙ5EÒ*6ã=UãP*.5WÌWÒçÓ0G$KM=?*Ìr$30 *IUÐz ç$;KM57=?Ó.5E$¾0 Ö57Ò!$3Ï CW G!=U$;15=?K
(*KM$
Gã65$; ÌWÒT=U57$CW.5E=U01%AÞRÒ!$#Ìr0lKe5=JKÎ!I?065M5E$;Gì K ÖÝ(!*Ìr57=?0 ì06Ö:$Î,0ÌWÒ%A<ì$Î,0ÌWÒ
Ò!$CE$@=?KK7=?ÓÎ*IUкG!$râ*!$
G GK =?!Î!(!5
Î!CE$;K7$l5E657=?0 *K4ÑaÒ!=JÌWÒ%->ÖÝ0 CKM570ÌWÒ*1Ke5E=?Ì:I?$; C7!=?!Ï9-
Weight space Weight space
2 2
1.8 1.8
1.6 1.6
1.4 1.4
1.2 1.2
1 1
0.8 0.8
0.6 0.6
0.4 0.4
0.2 0.2
0 0
−1 −0.8−0.6−0.4−0.2 0 0.2 0.4 0.6 0.8 1 −1 −0.8−0.6−0.4−0.2 0 0.2 0.4 0.6 0.8 1
Log MSE (dB) Log MSE (dB)
0 0
−5 −5
−10 −10
−15 −15
−20 −20
0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10
epochs epochs
\
{
L<M7N ¹POO>¹ _Wik^
o.s:shbM\ O_EfEsed;bh \j>#_Ebhbed;b@fW1be'_1 beikjl^k_n\bej1i¦j1^º°?d;b \
í~ ¡º\;jl {
z¢ ¡6©
Ìr0 CECE$;K7Î901*G!Kc5E0 Ñ)$;=UÏ1Ò15(!ÎG!.5E$;K;A.áhÙP*65EÌWÒ%-.6$Î,0ÌWÒÌr01C7CE$;K7Î,0 *G!K%570:0 !$4Ñ4$=?Ï Òl5
(!Î,G*.57$1A
1.8
1.8 1.6
1.4
1.6
1.2
1
1.4
0.8
1.2 0.6
0.4
1 0.2
0
0.8
−0.2
0.6 −0.4
−0.6
0.4 −0.8
−1
0.2
−1.2
0 −1.4
−1 −0.8−0.6−0.4−0.2 0 0.2 0.4 0.6 0.8 1 −2 −1.6 −1.2 −0.8 −0.4 0 0.4 0.8 1.2 1.6 2 2.4
−5 −5
−10
batch −10
−15 −15
−20 −20
0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10
epochs epochs
¢6©
6*G E P!=J K7$;K;A*ÞRÒ!$ Ìn5E=U<.657=?0 3ÖÝ(!*Ìr57=?0 #=JK 1 5 $( c5E !Ò % E nA9ÞRÒ*$57CW6=?!=?!Ï
KM$5:Ìr01l5E6=?*K $æ ÓÎ!IU$
KaÖÝCE0 Óï$;1ÌWÒ#06Ö E ÌI?1K7K7$;K;A924065EÒÁÌrIJ KEKM$
K
C7$Ù8@ (*K7K7=J6ÁG=JKeL
57CE=UP!(!57$;GºÑa=¦5EÒ3KM5E *G! CEGG!$<>=?657=?0 A A&)IJ KEK DÒ*1K)ÙÓ$;6¾06Ö%
L D *GºÌI?1K7K E Ò9 K4
Ó$
6¾06Ö 7 1A>Þ6CEÏ $5)<.6I?(!$
K4 C7$
#L DÖÝ0 CRÌrIJ KEK : *G 7
ÖÝ0 CaÌrIJ KEK E Aî=UÏ1(!C7!$ KMÒ*0.ÑK
57Ò!$]KM570ÌWÒ*1Ke5E=?ÌD57CWP e$
Ìn5E0 CEÐÖÝ0 CR57Ò*$$ræ!6ÓÎ!I?$ A
y
ω3
ω1
ω2
ω0
L<M7N ¹PO,¹ u®ol_aZikjlχikZ]\*ZDlseik¦\n'_Ebjl_EsÀ®dbe!©
3 * ')!3> &'
2"0<3C !3 9 ?'-'):' *C'->! % ')!3C>:'
!"0<3
% #""0% 9
ä $#Ì ç(*K7$¾57Ò!$CE$;K7(!I¦5WK06Ö5EÒ!$Î!C7$;<>=U01(*KK7$;Ìr57=?0 ç570Be (*KM57=UÖÝÐçKM$;< $;CE I)0 Ö57Ò!$35EC7=JÌWÍK
G=?KEÌr(9K7K7$;G¾$; C7I?=U$;C;A
* ')! =%
%1!3 >')
% "3 * #!2')" .%1
ÞRÒ!$DCE$;1KM01ÖÝ0 CR5EÒ!$6P,0.< $
57CE=?ÌWÍ=JK457Ò*65Ù!01!ë$;C70Ó$;63=?º5EÒ!$@=U!Î*(5a<' C7=J6P*IU$
K
rÌ CE$;.5E$;K$5 ü Uý.ü[û
$=?Ï $;l<. IU(!$1A*ÞRÒ!=JKÓ$; *KR5EÒ!$ÙÌr01*G=U57=?0 #>(!ÓP9$;CÑa=UI?I/P,$]IJ6CEÏ $1-
=ÀA $ A.5EÒ!$
Ì01KM5K7(!C7Ö» Ì$aÑa=UI?I!P,$Ke5E$$Î=?KM01Ó$RG!=UCE$;Ìr57=?0 *K *GKMÒ96I?IU0.Ñã=U065EÒ!$CWKK70@57Ò*65
áOÖ:5EÒ!$Á=?!Î!(53<.6CE=? P!IU$
K6CE$ÁÌr01C7CE$IJ.5E$;Gc-®5EÒ!=?KºÑa=UI?I
*065ºÓ6Í1$57Ò!$ $CEC701CKM(!C7Ö» Ì$
KMÎ!Ò*$CE=?Ì;6I[-P!(5=U5Ña=UI?I%Î,01KEKM=?P!I?ÐC7$
G(*Ì$:=U5EK$;ÌÌ$l57CE=JÌr=U5eÐ A
&)0 CEC7$;I?657$;G=U!Î*(5a<' C7=J6P*IU$
K)(9KM(* IUI?ÐÌ (*K7$D5EÒ!$@$=?Ï $;l<1$;Ìr570 CWK0 Ö 570P,$:CE065W.5E$;G
'Ñ4'Ð:ÖÝCE0 Ó 57Ò*$Ìr0>0 CWG=?*.5E$).æ$
1K Àî=UÏ1(!C7$ (6
<1$CWKM(*K ('P %5EÒ>(*K®Ñ)$;=UÏ1Òl5(!ÎG!657$;K C7$4!0 5
G$;Ì0 (!Î!I?$;G%A*Û
$
Ìr01(!Î!I?$;G3Ñ)$;=UÏ1Òl5EKRÓ6Í1$D5EÒ!$Áè70 !$@I?$;6CE!=?!ÏCE657$@Î,$CÑ4$=?Ï Òl5EêÓ$r5EÒ!0G
0 Î5E=UÓ6I[-5EÒl(9K-Ñ4$@Ò*'< $:57Ò*$@ÖÝ0 I?IU0.Ña=?!Ï57CE=?ÌWSÍ C
%( :'-')%1.02%A% "3 * #!2')" .%1
0.ÑõKM(!Î*Î90lKM$5EÒ*.557Ò!$º=U!Î*(5<.6CE=J6P!I?$;K0 Ö#!$(*C701¼Ò*'< $P,$$;çG$;Ì0 CEC7$;I?657$
Gc-
57Ò!$]Ø
$;KEKM=J63ÖÝ0 CR5EÒ!=JK!$;(!C701¾=JKR5EÒ!$#G=J6Ï10 * I%6*G3=U5EKa$;=UÏ1$><.6I?(!$;KRÎ,0 =?l5
6I?0 !Ï57Ò*$
Ìr0>0 CWG=?*.5E$º.æ$;K;A/áhTKM(9ÌWÒç Ì; K7$57Ò!$3Ï CW G!=U$;15=?K!06557Ò!$3P,$;KM5G$
K7Ì$l5G=?CE$;Ìn5E=U01
KÌ6¼P,$KM$;$z=?çî=UÏ ('P%A 45Ù5EÒ!$3Î901=Ul5 ®-®6ã6CEC70.ÑõKMÒ*0.ÑK57Ò9.5Ï1CE1G=?$l5G0>$;K
!065@Î901=Ul5@570.ÑR6CWG!K57Ò!$Ó=U*=UÓ(!ÓA%Ø0.Ñ4$< $;C;-,=UÖÑ)$=?*Ke5E$;1G 1K7K7=UÏ1Á$;1ÌWÒÁÑ4$=?Ï Òl5:=U5EK
0.Ña I?$;6CE!=?!Ï3CE657'$ Ý$;Ül(* I57Ò!$=?l<1$CWKM$06Ö57Ò!$Ìr01C7CE$;K7Î,0 *G=?!Ͼ$=?Ï $><.6I?(!$ a5EÒ!$å57Ò*$
G$;KEÌr$;15G=?CE$;Ìn5E=U01¾Ña=?I?IcP9$=?º5EÒ!$]G=?C7$
Ìn5E=U01º06Ö57Ò*$:0 57Ò!$;C6CEC70.Ñì5EÒ*.5aÎ,0 =?l5EKaG=?C7$
Ìn5EIUÐ
570.ÑR6CWG!K)57Ò!$Ó=?!=UÓ(!Ó'C
# % % !'-!% .%12')3C"3C4 ')2% > &' %1!- %1"4&;
åøMó (>Ë7Êó®ø lò/ÊÇÄDÈ ÇÆ*Èò/ÆTÇ Å!Ë õËeùcóÅ!ËeÇ)Ä òcÅ ÇaÈ?
áhÙ5EÒ!$4ÖÝ0 I?IU0.Ña=?!Ï:Ñ)$RÑa=?IUIP!CE=?$*Ð=?l57CE0>G!(*Ìr$457Ò*$
$ÑR5E0 %-6Ì0 Pe(!Ïl.5E$)Ï1CE1G=U$;l5;-68@6(*KEKML
$ÑR5E0 %-l"%$;< $>P,$CEÏQÁ6CWÜl(*6CWG>569GÙ5EÒ!$ D(*1KM=UL $ÑR5E0 @»2Rî8 ?"®Ó$r5EÒ!0G@»K7$$
I?K70
- - *- " *nA
Ó01C7$@5EÒ*6#ÖÝ$;Ñ <. C7=J6P!I?$;K;A ?>=?*Ìr$]57Ò!$]$;C7CE0 C4ÖÝ(!9Ìn57=?0 #=JK=?#Ï $;!$CW6Ic*0 LhÜl(* GCW.5E=?Ì -
57Ò!,$ $;ÑR5701 6I?Ï 01C7=U57Ò*Ó=U =¦5WK@0 CE=UÏ1=U* I/ÖÝ01C7Ó=JK:!065@(*KE6P!I?$ÖÝ0 C@Ï $;!$CW6I!$(*CE I*$r5ML
Ñ)01C7ÍI?$; C7!=?!Ï9AlØ
0.Ñ)$;< $C4=U5Ï =?< $;K4Ï10l0Gº=U*K7=?Ï Òl5EK)ÖÝ0 CG!$< $;IU01Î!=?!ÏÙÓ0 CE$KM01Î!Ò!=JKe5E=?Ì;.5E$;G
6I?Ï 0 CE=U57Ò!ÓK;->1KaG=JK7Ì(*K7K7$;G¾=?¾5EÒ!$@ÖÝ0 I?I?0.Ña=U!Ï9A
U
ω
Λ-½ Θ′
ΘΛ -½
ω ω
Network
U Λ -½ Θ Network
input input
$ & $ + & $ E
& $ + + & $ +
0 C ®0 IJ6Í *G3Na=UP*=U$;C7$
$ & &$ F & + $ + ; + $ 5
$ + $ +
ÞRÑ)0G=?C7$
Ìn5E=U01*K $ 69G $ + 6CE$@G!$râ*!$
G KaÌr01Pe(!Ïl.57$:=UÖ
+ $ +
$
=ÀA $ AÌr0 Qe(!Ï1657$ºG=?CE$;Ìn5E=U01*KÙ C7$01CM5EÒ!0 Ï10 * IG=?C7$
Ìn57=?0 9K]=?¼57Ò!$3KMÎ9 Ìr$0 Ö
6¼=JG$l5E=¦5eÐ
Ø$;KEK7=? ]Ó.57CE=UæÀKM$;$)î=UÏ1(!CE$: (XnA$CEÐ@=UÓÎ,0 C75E6l5%ÖÝ01CÌ0 >< $;C7Ï1$*Ì$=UÙP90 57ÒÌWÒ*0 =JÌr$;K/=?K
ρκ−1
ωκ
ρκ
ÞRÒ!$
Î,01K7=¦5E=U<1$
G$â*!=U57$:$;KM57=?Ó657$0 Öc57Ò!$:=?l<1$CWKM$Ø
$;KEKM=J6=JK4G0 !$:G=?C7$
Ìn57I?ÐÑa=¦5EÒ!0 (!5
C7$
Ü1(*=UCE=U*ÏÓ657CE=¦æ3=U><1$CWKM=?0 #69G¾P>Ð30 !I?Ð3(*KM=?!ÏÏ1CE1G=U$;l5=UÖÝ01C7Ó657=?0 %A I?Ï 0 CE=U57Ò!Ó=UL
Ì6I?I?Ð#57Ò!=JKÙÌ;6åP9$G!$;KEÌrCE=UP,$;Gú K:ÖÝ01IUI?0.ÑDK C< Dâ9CEKM5Ù3Î90lKM=U57=?< $G$â*!=U57$Ó.5EC7=Uæ =?K
ÌWÒ!01K7$%-$1A Ï9A *- E )5EÒ!$35EÒ!$]KM$
6CWÌWÒ¾G=?CE$;Ìn5E=U01¾=JKK7$r5R570
%. %. & /%. %4
®@I?=?!$
K7$;6CWÌWÒÙ=JKÎ,$C7ÖÝ0 CEÓ$;G6I?0 !Ï ,- ÑaÒ!=JÌWÒÏ =?< $
K57Ò!$(*Î,G!657$RÖÝ01C57Ò!$
Î*6CW6Ó$r5E$CWK
.5a57=?Ó$'.
%. /. F F10 %. S/%. 45
îv=?* IUI?Ð 57Ò*$#$;KM57=?Ó657$#0 ÖD5EÒ!$Á=?l<1$CWKM$Ø$
K7K7=J6T=?K(*Î,G!657$
GcAR&)01ÓÎ96CE$;Gç570ú57Ò*$
$ÑR5E0 º IUÏ10 CE=¦5EÒ!ÓÚ5EÒ!$ D(9 K7=¦L $;ÑR570 Î!Î!CE01 ÌWÒ0 *IUÐ!$$
G!KÏ1CE1G=?$l5=?ÖÝ0 CEÓ.57=?0 /A
ÞRÒ!$
Ó01KM5)K7(*ÌÌ$;KEKeÖÝ(*;I D(*1KM=UL
$ÑR5E0 6I?Ï 01C7=U57Ò*Óà=JK5EÒ!$
24C70.ÐG$;LhîvI?$r5WÌWÒ!$C7Le8:0 IJG>Ö»6CEPL
?>Ò*6*!0 »2Rîv8 ?"Ó$57Ò!0GcA1ÞRÒ!$a(!ÎG!.5E$aC7(!I?$4ÖÝ0 C®5EÒ!$a$;KM57=?Ó.57$a0 Ö*5EÒ!$a=U><1$CWKM$aØ$
K7K7=J6
=?K
/%. < /. F
7 + + + + F
+ +7 +
57Ò!$]Ø
$;KEKM=J6=JK6Î!Î*C70'æ=?Ó657$
GP>Ð5EÒ!$ÙKEÜ1(96CE$D0 Öv5EÒ!$ l Ìr01P!=J6 ÀKM$;$]6IJKM0K7$;Ìr57=?0 (>A E
ÖÝ0 CÖÝ(!C757Ò!$;C
G=JK7Ì(*K7K7=?0
J
+
+
& -5 &
%
ÞRÒ!$:"/$<1$>P9$;C7ÏQÁ6CWÜl(*6CWG>5Ó$r5EÒ!0G=JK4IU=?Í $
57Ò!$8@6(*KEKML
$ÑR5701º6P,0.< $1- P!(!54=U54Ò9 K4
C7$;Ï (!IJ6CE=Uë
.5E=U01ÙÎ* CE Ó$57$;C ¾5EÒ*.5)Î!C7$;< $;15WK®=¦5ÖÝC701ÓP!I?0.Ña=?!Ï@(!Î%- =UÖ%KM01Ó$a$=?Ï $><.6I?(!$
K
6CE$@K7Ó6I?I +
J
+
7 & -
X(
ÑaÒ!$CE$ G$*0657$
K#5EÒ!$ã(*!=¦5eÐ Ó.5EC7=UæcADÞRÒ!$x8@6(9K7K $ÑR5E0 ÚÓ$57Ò!0GÚ=JKÁ<. IU=JGßÖÝ01C
Ü1(9 GCW.5E=?Ì:Ìr0lKe54ÖÝ(!*Ìr57=?0 *KaÒ*0.Ñ)$;< $C4KM=?Ó=UIJ6CRÎ!CE0Ìr$;G!(!C7$@6IJK70Ñ40 CEÍKÑa=U57ÒH@(!I?IUP*1ÌWÍlL
"%$=?P!I?$CÌ01KM5a *G3=?KaÌ;6I?IU$
G D.5E(!CE I/8:CE1G=?$l5 »K7$$$ A Ï*A - - E *nA
Æ*Ë7Ê
Å>Ç ÊÇ DÉÅ>òíÅ ò ò 1>ËeóÄ ËMÄ WÇÆ óÅ!ËeÇ)Ä ËMÄ
ÚÉDøhÅ!ËMøMó ò%ÆzÄ
òc6Å úÇÆ
!"3C"0%
" B%(')%(3 %
ä $]Ì;63ÑaC7=U57$@5EÒ!$ l L[57Ò3IU=?!$06Ö57Ò*$]Ø$;KEK7=?
$
& % & 7 $ F &
$
ÑaÒ!$CE$ $
65D565-
DX
65D5654
=?KD< $
Ìn5E0 C0 Öë$CE01K
6*G#01!I?о01!$&Ù65
57Ò*$ÙÍlLÀ5EÒ
Î90lKM=U57=?0 %AÞRÒ!=JKDÌ;6ÁP,$Ù=?ÓÎ!I?$Ó$l57$
GÁÑa=U57ÒåKM=?ÓÎ!I?$ÙCE$;Ìr=?Î,$XC DÌr01ÓÎ!(57$Ù57Ò!$Ù570 5E6I
C7$
KM(!IU5EK469GG=?<l=JG$DPlÐ >A>Û
(*$
5E0Ù>(!Ó$CE=?Ì;6I9$CEC701CEK=?5EÒ!=JK)Ì0 ÓÎ!(5W.57=?0 ¾K7ÌWÒ!$;Ó$57Ò*$
C7$
KM(!IU57=?!ÏåØ$;KEK7=? zÓ=UÏ1Ò15!065P,$¾Î,$C7ÖÝ$;Ìr57I?ÐúK7Ð>ÓÓ$57CE=?Ì Aváhç57Ò!=JKÌ1KM$=U5KMÒ!01(!IJGzP,$
[ün/ÿ T Eþ@1KaG$;KEÌrCE=?P9$
G¾P,$I?0.ÑA
*C!'-%! 0" :3 ') "
!"0<3 > &'A%
!* A% <3 :3 9
@ % #<1% 3 %('-4 2'
*C2' 9 =!.04<&'-"0
KEKM(!Ó=?!ÏÓ$;6KEÜ1(96CE$;GºÌ01KM5aÖÝ(!*Ìr57=?0
& E J F
; +1 F
; *
57Ò!$;¾5EÒ!$Ï CW G!=U$;15a=JK
& F F
% +
-
J %
6*Gº57Ò!$]Ø
$;KEKM=J6¾ÖÝ0 I?I?0.ÑKa K
J
+
7 J F
; + F
5 )"
K7=?ÓÎ*IU=UÖÝÐ>=U!Ï Î!Î!CE0'æ>=?Ó.5E=U0106Öv57Ò!$Ø$
K7K7=J6º=JK457Ò*$@KEÜl(*6CE$:06Ö/5EÒ!$'11Ìr01P!=? ÑaÒ!=JÌWÒ
=?KÎ,01K7=¦5E=U<1$KM$;Ó=ULhG$râ**=¦5E$Ó657CE=¦æ¾06ÖG!=UÓ$9KM=?0 C%
J
+
)$"
ÑaÒ!$CE$5EÒ!$aKM$
Ìr01*G5E$CEÓ ÖÝC701Ó AÜ,8A #" cÑR K®GCE0 Î!Î,$;G%A'ÞRÒ*=?K®=JK$;Ül(!=?<' IU$;l5v5E0@1K7K7(!Ó=?!Ï
57Ò*6557Ò!$!$r5eÑ40 CEÍ=JK
I?=?!$; CaÖÝ(!*Ìr57=?0 Á0 Ö5EÒ!$]Î* CE Ó$r57$;CEK A
Ï16=?57Ò*=?K
=?K
C7$
G=?IUÐ
=UÓÎ!I?$Ó$l5E$;G#ÖÝ01C5EÒ!$ >LÀ5EÒ Ìr01IU(!ÓÁ0 Ö57Ò!$ 11Ìr01P!=? C!ÖÝ01CD IUIv57CW6=?!=?!ÏÎ*65M57$;C79K-
Ñ)$RÖÝ01C7ÑR6CWGÙÎ*C701Î*6Ïl.57$1-'57Ò!$; E ®KM$557Ò*$
Ìr57=?<l=U5eÐ06Ö,57Ò!$01(57Î*(5)(!*=¦5WK570 6*G0 *IUÐ
57Ò!$ lLÀ5EÒ¼0 (5EÎ!(5]5E0 1- ]P*1ÌWÍlÎ*C701Î*6Ïl.57=?0 úKM57$μ=JK5W6Í1$z69Gå57Ò!$Ï1CE1G=?$l5]=?K
ÌÌ(!ÓÙ(*I?657$;G%A
+ ! C') :4&2"034 %( &3 9 9 %(')" #!!" #<%1
"%$r5(9KÌr0 9KM=JG$C:Ó(!I¦5E=¦LOIJ'Ð $CK7ÐKe5E$ÓÚÑa=U57ÒK70 Ó$4ÖÝ(!*Ìn5E=U01*6I*P!IU0ÌWÍKÑa=¦5EÒ , =?!Î!(5WK-
0 (5EÎ!(5EKa *G ÷Î* CE Ó$57$;CEK)06Ö%5EÒ!$:ÖÝ0 CEÓ
r2A 0.Ñ 1K7K7(!Ó$DÑ4$:Íl*$Ñ
DF @F -cÑaÒ!=?ÌWÒå=JK Ó.57CE=UæA%ÞRÒ*$å=¦5]=JKKe5ECE =UÏ1Òl5DÖÝ01C7ÑR6CWG570#Ì0 ÓÎ!(5E$
57Ò!=JKaÓ.5EC7=Uæ
6F + 6F F )" E
F F 7 F5
z
f( )
y
L<M7N ¹PO *¹ ®\;fM6q1bed
ql\;^
\seikjl^@seo1_a1i¦\;^;d
jl\; W_ ghghi¦\;j*¯>ghik^
Zd;iU g ¦ _7°?s \;jlm4g eb i¦^;o.s 7©
*C"3C4 %'- 9 *6,<>% A%1"0:3 !3 9 #&%16:'
7: F
)"-
(*KM=?!Ïå0 !I?Ðú5eÑ40 Ï CW G!=U$;15Ìr01ÓÎ*(5E657=?0 *'K »65Î,0 =?l5 69G 7 íCE$;K7Î9$
Ìn57=?< $;IU"Ð r-
ÑaÒ!=?ÌWÒ#Ì;63P9$CE$;1G=UI?ÐÌr01ÓÎ!(57$
G3Ña=¦5EÒ3P9 ÌWÍ>Î!CE0 Î z=JKaKMÓ6I?I/Ì0 *KM5E 1-5 nA
ÞRÒ!=?K)Ó$r5EÒ!0GºÌ P,$: Î!Î!I?=U$
G570Ìr01ÓÎ*(57$D57Ò!$:Î!CE=U9Ìr=?Î*6I,$=?Ï $;l<1$;Ìr570 C)6*Gº$=?Ï $!L
<' IU(*$06Ö ÷P>Ð5EÒ!$Î90.Ñ4$CRÓ$r5EÒ!0GcA*24к=U57$;CE657=?!Ï6*G3K7$r5M5E=U*Ï
/. 7 < %.
#&
/.%
57Ò!$< $;Ìr5701C /%. Ña=UI?IÌ0 >< $;C7Ï1$:5705EÒ!$ÙIJ6CEÏ $
Ke5$;=UÏ1$>< $
Ìn5701Ca06Ö *G %. :5E057Ò*$
Ìr0 CECE$;K7Î901*G=?!Ïú$=?Ï $><.6I?(!$ E *4- 4- [1A ?>$$ I?K70 ÖÝ0 C36x$<1$xÓ0 CE$Á Ì;Ìr(!CW.5E$
Ó$57Ò!0G¾57Ò9.5 RG0>$;KR!0 5(*K7$Dâ9!=¦5E$]G/= V,$;C7$;*Ìr$
K6*G E 4Ò* KaK7=?Ó=?IJ6CÌr01ÓÎ!IU$æ=¦5eÐ1A
xÄ
ó®ø lË ¼ÇÅ
ò ò1>ËeóÄ ËMÄ ÚÉDøhÅ!Ë
.øeóò/ÆçÄòcÅ åÇÆ
Oá 5=?K=Ul57$;C7$
Ke5E=U*Ï@570](!9G$CWKe5W6*GÒ!0.ÑìK70 Ó$a06Ö,57Ò!$57CE=?ÌWÍKK7Ò!0.ÑaÎ!C7$;<>=U01(*KMI?ÐÙ=?*(*$*Ì$
0 57Ò!$DØ$;KEK7=? %-.=ÀA $ A1Ò!0.ÑãG!0l$
K®57Ò*$
Ø$
K7K7=? ÌWÒ96!Ï1$4Ña=U57Ò CEÌWÒ*=¦5E$;Ìn5E(!CE$R *GG$r5W6=?I?K06Ö
57Ò!$=UÓÎ!I?$Ó$;15W.5E=U01%A,Þ)Ð>Î!=JÌ6I?I?Ð -!5EÒ!$$=?Ï $><.6I?(!$ÙG!=?KM57CE=UP*(57=?0 Á0 Ö57Ò!$Ø$
K7K7=? ÁI?0l01ÍK
IU=?Í $5EÒ!$]01!$]KMÍ1$r5WÌWÒ!$;G3=? î=UÏ1(!C7$ E C*ÖÝ$;ÑÚKMÓ IUI%$;=UÏ1$><.6I?(!$;K;-!Ó6>оÓ$
G=?(!Ó 0 !$
K
6*GÖÝ$;Ñ< $;C7ÐIJ6CEÏ $
01!$;K;A äå$:Ña=UI?I!0.Ñ6CEÏ (!$57Ò*65)5EÒ!$ ?ý.üÀû ÿ¦û :5.ý ! Ña=?IUIcÌ;6(*K7$
57Ò!$@5EC701(!P!I?$:=?357Ò!$@5ECE =U*=U!ÏÎ!CE0Ìr$
K7K4P,$;Ì (*K7$ E !- E E
Log10 Eigenvalue
−2
−2.5
−3
−3.5
−4
−4.5
−5
−5.5
−6
0 100 200 300 400 500 600 700 800
Eigenvalue order
/ik^
_Ej ;\;k1_ghq_EfEshbelZikj]\R£U\W._7bghol\be_n:_Wik^
o.segj1_Es[d;be ¢;¡~r¢;¤;£~n
hsL<bMM7\N ikjl¹P_nO *¹ d;j
¢;@o>\j>6beikshse_Ej1ik^
isegW©
57Ò!$3I?1Ke5IJ'Ð $C
A ÞRÒ!=?KQV,$
Ìn5EKÙ57Ò*$3I?$; C7*=U!Ï KMÎ,$$
Gç69GçÌ zÎ!CE0.<>=?G$36ã=U!Ï1C7$
G=?$l5
5703$ræÎ!IJ6=?#57Ò!$K7IU0.ÑßI?$; C7!=?!Ͼ=U IU0.Ñ4$CDI?'Ð1$CWK
69G57Ò!$Ö» KM5 »K70 Ó$r5E=UÓ$01KEÌr=?IUIJ.5E=U!Ï
IU$
6CE!=U*Ï=U5EÒ!$
IJ KM5I?'Ð1$C
AXì5EC7=JÌWÍ]5E0ÙÌr01ÓÎ,$9K7657$a5EÒ!=?K)G=/V,$;C7$;154K7Ì;6I?$a06ÖcIU$
6CE!=?!Ï=?K
570(*K7$
5EÒ!$@=U>< $;CEK7$DG!=? Ï 0 96IcØ$
K7K7=? 5E0Ì0 l57CE0 I,57Ò*$:I?$; C7*=U!ÏCE657$ »K7$$6IJK70K7$;Ìn5E=U01
&!A
Dø Ë7Äô
4òvÊÇÄ
È zÆ*Èò%
Æ òcÅ Ça?È Å
Ç É
øhÅ!Ë7øeó ò/Æ
òc6Å úÇÆ
2)$ÖÝ0 CE$ÙÑ4$Ì0 *Ì$l57CW.5E$Ù=? 5EÒ!=JK:K7$;Ìr57=?0 0 Ò*0.Ñ 57035E =UI?0 C@KM$
Ìr01*G#0 CWG$;C
5E$;ÌWÒ!!=JÜl(!$;K
ÖÝ0 C57CW6=?!=?!ÏDIJ6CEÏ $)!$r5eÑ40 CEÍK-
I?$r5(*Kvâ*CWKe5CE$Î,$;65®K70 Ó$4CE657Ò!$;CÎ,$;KEK7=UÓ=JKe5E=?ÌÖ» Ìr5EK6P,0 (!5
6Î!Î!I?Ð>=U*ÏÌrIJ KEK7=?Ì;6I9K7$;Ì0 *G0 CWG$C)Ó$57Ò!0G!K;AÞ/$
ÌWÒ!!=JÜl(!$;K)(*K7=U!ÏÙÖÝ(!I?IØ
$;KEKM=J6º=UÖÝ01C7Ó6L
57=?0 À8@6(9K7K4L $;ÑR570 /-!"%$<1$>P,$CEÏ6LhQÁ6CWÜl(*6CWG>5) *G¾2Rî8 ?"4Ì630 *IUÐ Î!Î!I?Ð570< $;C7Ð
KMÓ6I?IR!$r5eÑ40 CEÍK]5ECE =U*$;Gz=?TP*.5WÌWÒçÓ0G$ -®Ò!0.Ñ4$<1$C57Ò!0lKM$KMÓ IUIR!$5eÑ)01C7ÍK6CE$º!0 5
57Ò!$01!$;K@5EÒ*.5]*$$;GúK7Î9$;$;G=?!Ï#(!Îú5EÒ!$Ó01KM5;A/QÁ01KM5ÙKM$
Ìr0 9G 01CEG$;CÓ$r57Ò*0>G*&K »Ì0 P e(L
Ï1.5E$Ï CW G=?$l5
-%24î8 ?-A;A#A :CE$;Ül(!=?CE$I?=?!$rLhKM$
6CWÌWÒå *GúÌ 5EÒ!$CE$rÖÝ01C7$!0 5]P9$º(*KM$
G
=Uú5EÒ!$ºKM570ÌWÒ* KM57=JÌÓ0G$ A/Q 6>Ð 0 Ö45EÒ!$5EC7=JÌWÍKG=?KEÌr(9K7K7$;GåÎ!CE$<>=?0 (*K7IUÐå6Î!Î*IUÐ 0 !I?ÐÁ5E0
P*.5WÌWÒ#IU$
6CE!=U*Ï*A9î*C701Ó 01(!C
$æÎ9$;C7=?$*Ì$ÙÑ4$]Í>!0.Ñ 57Ò*65:Ì;6CE$rÖÝ(!I?I?к5E(!!$;G Ke5E0ÌWÒ* KM57=JÌ
Ï CW G=?$l5G$;KEÌr$;15=JKÒ*6CWGç570úP9$
.501ãIJ6CEÏ $ÌrIJ KEKM=Uâ9Ì;.57=?0 ãÎ!CE0 P!I?$ÓK;A®î*0 CK7Ó6I?IU$;C
Î!C701P!I?$ÓKa5EÒ*.5@C7$
Ül(!=UCE$Ù1ÌÌr(*CE657$CE$;6IULO<' IU(*$;G0 (!57Î!(5WK:IU=?Í $]=?ÁÖÝ(**Ìn5E=U01 6Î!Î*C70'æ=?Ó6L
57=?0 ì0 CÌ0 l57CE0 IRÎ!CE0 P!I?$ÓK;-Ñ)$#K7$$5EÒ*.5ºÌ0 P e(!Ïl.5E$3Ï1CE1G=?$l'5 »Ña=¦5EÒ ®0 IJ6ÍlLON=UP!=?$CE$
Number of Eigenvalues
20
19
18
17
16
15
14
13
12
11 Big killers
10
9
8
7
6
5
4
3
2
1
0
0 2 4 6 8 10 12 14 16
Eigenvalue magnitude
/ik^
_Ej ;\;k1_ghq_EfEshbelZikj]\R£U\W._7bghol\be_n:_Wik^
o.segj1_Es[d;be ;
¢ ¡~r¢;¤;£~n
hsL<bMM7\N ikjl¹_n 9¹ d;j
¢;@o>\j>6beikshse_Ej1ik^
isegW©
Learning 1.8
rates: 1.6
η0 = 0.12
η1 = 0.03
1.4
η2 = 0.02 1.2
Hessian
0.8
largest
eigenvalue: 0.6
0.2
0
−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1
Learning
rate (batch): −5
−15
−20
0 1 2 3 4 5 6 7 8 9 10
epochs
seo be_W_aq>\rbM\;Z_Ese_7beg ¢
®_Wik^;o6segW}9~a{li¦\g 7©
Weight space
2
Learning 1.8
rates: 1.6
η0 = 0.76
η1 = 0.18
1.4
η2 = 0.12 1.2
Hessian
0.8
largest
eigenvalue: 0.6
0.2
0
−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1
Learning
rate (batch): −5
−15
−20
0 1 2 3 4 5 6 7 8 9 10
epochs
seo be_W_aq>\rbM\;Z_Ese_7beg ¢
®_Wik^;o6segW}9~a{li¦\g 7©
%(' % C 9 ä $CE$Î,$;65a57Ò!$CE$;K7(!IU506Ö®0 (!CG=JKEÌr(*KEKM=?0 =?#KM$
Ìn57=?0 (A ""C!Ke5W6C757=?!Ï
ÖÝC701ÓíCE *G0 Óí=U!=U57=J6I/< $;Ìr5701C -!57Ò!$=U57$;CE657=?0
.
Ña=UI?I*$;< $l5E(*6I?IUÐÌr01l<1$CEÏ $4570]57Ò!$Î*C7=?*Ìr=?Î* I*$=?Ï $;>< $;Ìr5701C »0 C4@< $;Ìr5701C=U5EÒ!$Î!CE=?*Ìr=?Î*6I
$=?Ï $9KMÎ*1Ìr$ ) *G
Ña=?I?I/Ìr01l<1$CEÏ $D57057Ò*$]Ìr0 CECE$;K7Î901*G=?!Ï$;=UÏ1$><.6I?(!$ 4- OA
.0&' !3"0<3
!065EÒ!$C@Ó$r5EÒ!0GÁÓ6Í1$;KD(*KM$06Ö5EÒ!$Ö» Ìn5:57Ò9.5KMÓ6I?IÎ9$;CM5E(!CML
P*.5E=U01*KR06Ö57Ò!$Ï1CE1G=?$l5R I?K70IU$
Gº57057Ò*$Î!C7=?*Ì=UÎ* Ic$=?Ï $;l<1$;Ìr570 CR0 Ö
.
% 7 F
#&
ÑaÒ!$CE$ #=?K@K7Ó6I?I!Ìr0 9Ke5W6l5;A1BD!$R=U57$;CE657=?0 06Ö*5EÒ!=JKÎ!CE0>Ì$;G(*C7$4C7$
Ül(!=UCE$;Kv5eÑ40:ÖÝ0 CEÑ4 CEG
6*Gº5eÑ)0P*1ÌWÍ>Ñ4 CEGºÎ!C701Î*6Ïl.5E=U01ºKe5E$Î*KaÖÝ0 Ca$;1ÌWÒ¾Î9.5M5E$CE¾=?357Ò!$@5ECE =U*=U!ÏK7$r5
A
3.0"03%
*2"&3 &> ÞRÒ!$ÖÝ0 I?IU0.Ña=?!ÏCE(!I?$ÙÓ6Í1$;KD(*KM$06Ö5EÒ!$CE(!!*=U!Ï3'<lL
$CW6Ï $D57001P5E =U¾57Ò*$I? C7Ï1$;KM54$;=UÏ1$><.6I?(!$@06Ö/5EÒ!$]'< $;CE Ï $:Ø$;KEK7=? ¾<1$CEÐÖ» KM5
. F 7
7 F 5 #&
Þ/0K7(!ÓÓ C7=?ë$1->57Ò!$$;=UÏ1$><.6I?(!$ 2.< $
Ìn5701C4Ì0 ÓÎ!(5W.5E=U01*6K C
AaCE *G0 Óí< $;Ìr5701C4=JKÌWÒ!0lKM$;ÖÝ01C=U*=¦5E=? IU=?ë;657=?0 306Ö -
E Aa6=?!Î!(5
Î9.5M5E$CE=?K
Î!C7$
KM$;l57$;G¾Ña=U57ÒÁG$
KM=?C7$
G30 (5EÎ!(5;-,ÖÝ0 CEÑ4 CEG36*G3P*1ÌWÍ>Ñ4 CEG
Î!CE0 Î* Ï1.5E=U01%-Ke5E$Î3=?KÎ9$;CMÖÝ01C7Ó$;G369G5EÒ!$Ï CW G=?$l5WK a C7$@KM570 CE$;G%-
!A
=?Ka1G!G$
Gº5E057Ò!$]Ì(!C7CE$l5aÑ4$=?Ï Òl5a< $;Ìr5701C -
AaÖÝ01C7ÑR6CWG6*G#P9 ÌWÍ>Ñ4 CEG3Î!CE0 Î96Ï1657=?0 Ke5E$Î =?K
Î9$;CMÖÝ01C7Ó$;GÁÑa=¦5EÒ#57Ò!$Î,$C757(!CEP9$
G
Ñ)$;=UÏ1Òl5R<1$;Ìr570 Ca *G¾57Ò!$Ï1CE1G=U$;l5EK R C7$KM570 CE$;G%-
"A457Ò!$G!W= V$CE$*Ì$ 1 F ;@=?KÌ0 ÓÎ!(5E$;Gå6*G 57Ò*$CE(!!*=U!Ï#'<1$CW6Ï $Ù06Ö
57Ò!$$;=UÏ1$>< $
Ìn5E0 CR=?KR(*Î,G!657$
Gc-
&!ARÑ)$@I?0>0 Î3ÖÝC701Ó E O-L & (*15E=UIvC7$
K70 * P!IUÐKM5E P!IU$@CE$;K7(!IU5=?KR01P5E =U!$
GÖÝ01C -
(>A457Ò!$01Î57=?Ó6I%I?$; C7*=U!ÏCW.57$@=JK457Ò!$;#Ï =?< $;¾ K
0 ( 5
áhî=?Ï (!CE$ E Ñ4$aKM$;$457Ò!$R$<10 I?(57=?0 06Ö957Ò!$R$=?Ï $;><' IU(*$41KDÖÝ(!9Ìn57=?0 06Ö957Ò!$R>(!ÓP9$;C06Ö
Î*.5757$CEÎ!CE$;K7$l5E657=?0 *K,ÖÝ01CvR!$;(!CW6I6!$5eÑ)01C7ÍD=U]Ò*6*G!ÑaC7=U5M5E$]ÌWÒ* CE1Ìn57$;CC7$
Ìr0 Ï1!=U57=?0
5E K7ÍAáh Î!CW Ìr57=JÌr$Ñ)$ G*6Î5@57Ò*$I?$; ÍÁKM=?ë$06Ö)57Ò!$CE(!!!=?!Ï'< $;CE Ï $]=? 01CEG!$C:5703Ï $5
ÖÝ$Ñ4$C *(*Ìr57(*657=?0 *@K À K6IJK70 =?*G=JÌ657$;Gã0 ç57Ò!$â9Ï (!CE$ rAáhã57Ò*$¾â9Ï (!CE$Ñ)$KM$;$357Ò*65
.Ö×57$;C]ÖÝ$;Ñ)$;CÙ5EÒ*6 Î*65M5E$CEzÎ!CE$;K7$l5E657=?0 *K@5EÒ!$¾Ì0 CEC7$
Ìn5Ù01CEG!$C]0 ÖÓ Ï !=U57(9G$ÖÝ01C
57Ò!$¾$=?Ï $><.6I?(!$1-%=ÀA $5EÒ!$I?$; C7!=?!ÏÁCW.5E$=?KÙC7$
ÌWÒ!$
GcAvî*C701ÓÔ5EÒ!$º$æ>Î,$CE=?Ó$;l5EK]Ñ4$º I?K70
0 P*K7$CE< $D57Ò*65a57Ò!$ 9(*Ìn5E(*.5E=U01*KR06Ö57Ò!$]'<1$CW6Ï1$DØ$
K7K7=J630.< $C)57CW6=?!=?!Ï6CE$KMÓ6I?I[A
80
70
60
50
eigenvalue
40
30
γ=0.003
γ=0.01
20
10
0
0 50 100 150 200 250 300 350 400
/'d;k1seikd
j#d°seo1_]_Wik^
_Wj6;\;kl_Ù\;g:\°J1jlfEseikd
j#d°seol_]j6lZ:{_7bd;°q>\rshse_Ebejq1be_Egh_Wj1|
MsL<\rM7seN ikd
¹jl g@,¹ °?d;b]\gho>\rbe_n#®_Wik^
o.s]j1_Es[d;be3iseo塺¦\W._EbegW} ;£
;¤3fWd
j1jl_Wf7sei¦d;jlg\j>¼~r¢;;¤º°Ube_W_
ql\bM\Z_Ese_EbegW©lu®ol_RshbM\;ikj1i¦j1^@gh_EsfWd;jlghikgOseg)d°®~n;
o>\j>6beikshse_Ej1ik^;iksegW©
áh î=UÏ1(!C7$ E "º6*G E &Ñ4$KM5E CM5:Ña=¦5EÒÁ5EÒ!$KE6Ó$=U!=U57=J6IÌ0 *G=U57=?0 9K-6*G Î9$;CMÖÝ01C7Ó
â!æ$;G3>(!ÓÙP,$C0 Ö$;Î90ÌWÒ*KÑa=¦5EÒ3I?$; C7*=U!ÏCW.5E$;KÌ0 ÓÎ!(5E$;G3P>кÓÙ(*I¦5E=UÎ!I?Ð>=U*Ï57Ò!$]Î*C7$L
G=?Ìr57$
GI?$; C7*=U!Ï:CW.57$4P>ÐÙ:Î!CE$;G$â*!$;GÌr01*KM5E6l5
A6&)Ò*0l0lKM=?!Ï@Ìr01*Ke5W6l5»=ÀA $ A (*KM=?!Ï:57Ò*$
Î!C7$
G=JÌn57$
G0 Î5E=UÓ I/CW.57$a6I?ÑR'Ð>KÏ =?< $;KC7$
KM=JG(* I%$CEC701CEKRÑaÒ*=?ÌWÒ C7$<1$CEоÌIU0lKM$@5E057Ò*$
$CEC701C ÌWÒ!=?$<1$;GP>Ð57Ò!$DP9$
Ke5RÌWÒ!0 =JÌr$0 Ö57Ò!$:Ìr0 9Ke5W6l5;A1áh065EÒ!$C)Ñ)01CEG!K;-.5EÒ!$ºè7Î!C7$
G=JÌn57$
G
0 Î5E=UÓ6IcCW.5E$;ê=JKR0 Î5E=UÓ6I/$!01(!Ï Ò%A
2.5
2
MEAN SQUARED ERROR
1.5
1 epoch
2 epochs
1
3 epochs
4 epochs
5 epochs
0.5
0
0 0.250.50.75 1 1.251.51.75 2 2.252.52.75 3 3.253.53.75 4
LEARNING RATE
PREDICTED OPTIMAL LEARNING RATE
_W\;jzg .>\be_Wz_7bhbed;b\g\°J1jlfEseikd
jçd;°seo1_ºbM\rsei¦d {_EsÀ®_W_Wjzk_n\rbejlikjl^ÁbM\se_¾\;jl
qL<beM7_nN 1¹ik fEse*¹_W d;q1seikZ]\;k_n\rbejlikjl^ºbM\se_°?d;bÙ\°JlkÁfWd;jljl_EfEse_nÁjl_EsÀ®d;be ¤;£ #; ~n 7©u®o1_
shbM\ikjlikjl^@gh_7s)fEd
jlghikgOsegd;°/;
:ol\;j>6beishse_Wj1ik^;iksegW©
2.5
2
MEAN SQUARED ERROR
1.5
1 epoch
1
2 epochs
3 epochs
4 epochs
0.5
5 epochs
0
0 0.25 0.5 0.75 1 1.25 1.5 1.75 2 2.25 2.5 2.75 3
LEARNING RATE
PREDICTED OPTIMAL LEARNING RATE
_W\;jzg .>\be_Wz_7bhbed;b\g\°J1jlfEseikd
jçd;°seo1_ºbM\rsei¦d {_EsÀ®_W_Wjzk_n\rbejlikjl^ÁbM\se_¾\;jl
qL<beM7_nN 1¹ik fEse*_W¹ åd
q sei¦Z]\)k_n\rbejlikjl^bM\rse_°Jd;b\gho>\rbe_n _Wik^
o.sj1_Es[d;beÁiseoz¡3¦\n'_Ebeg ~W'¢r£
~n¡ ;¤ Á
¢ #£
å~n
ú~n 7} £
;¤ kd.fn\ ÙfWd;jljl_EfEseikd
jlg\;jlã~r¢;;¤°?be_E_ºql\bM\Z_Ese_Ebeg
ghol\be_nÙ®_Wik^
o.seg 7©lu®ol_RshbM\i¦j1ikjl^gh_EsfWd;jlghikgOsegd;°~n;
@ol\;j>6beishse_Wj i¦^;isegW©
Â Ë >ÊcÉ?1>ËeÇÄïóÄ
È åÇÄ
Êcø7É?>ËeÇÄ
ÌÌ0 CWG=?!ϼ570ã57Ò*$ CE$;Ì0 ÓÓ$*G!657=?0 *KºÓ$l57=?0 !$
G6P,0.< $1-)çÎ!CE1Ìn5E=¦5E=U01!$CºÖ» Ìr=?!ÏT
ÓÙ(!IU57=UL[IJ'Ð $;CR!$(!CW6Ic*$r5a57CW6=?!=?!ÏÎ!CE0 P!I?$ÓÑ40 (*I?G¾Ï 057Ò*C701(!Ï Ò¾57Ò!$@ÖÝ01IUI?0.Ña=?!ÏKM57$;Î*KDC
KMÒ>( $@5EÒ!$$ræ!6ÓÎ!I?$;K
Ìr$;15E$CR5EÒ!$=U*Î!(5
<.6CE=J6P!I?$;K4P>кK7(!P57CW Ìr57=?!Ï57Ò!$Ó$
6
!0 CEÓ6I?=Uë;$:57Ò!$=?!Î!(5<. C7=J6P!I?$:570Ke5W6*G*6CWG¾G$;<>=?657=?0 306Ö
=¦Ö®Î,01KEKM=?P!I?$ -*G$
Ìr01C7CE$IJ.5E$57Ò*$=U!Î*(5
<.6CE=? P!I?$;K;A
Î!=JÌWͺ!$r5eÑ40 CEÍÑa=U57Ò357Ò!$]K7=?Ï Ó0 =JGÖÝ(**Ìn5E=U01K7Ò!0.Ña3=U3â*Ï1(!CE$
KM$5a57Ò!$@5W6CEÏ $54<.6I?(!$
KaÑa=¦5EÒ!=?¾5EÒ!$CW6!Ï1$D0 Öv5EÒ!$]KM=?Ï Ó01=?Gc->5eÐ>Î!=JÌ IUI?Ð 7 @6*G¾
L A
=U*=¦5E=? IU=?ë$@5EÒ!$Ñ)$;=UÏ1Òl5EK4570CW6*G!0 Ó<.6I?(!$
Ka KRÎ!CE$;KEÌrCE=UP,$;G¾P> Ð &*A
ÞRÒ!$@Î!C7$ÖÝ$CEC7$
GºÓ$57Ò!0G¾ÖÝ0 CR5ECE =U!=?!Ï57Ò*$!$r5eÑ40 CEÍKMÒ*0 (!IJGºP,$Î!=JÌWÍ $
G¾1K4ÖÝ0 I?IU0.ÑKDC
=¦Ö957Ò!$)57CW6=?!=?!Ï@KM$5=JKI? C7Ï1:$ ÝÓ0 CE$)57Ò*
ÖÝ$ÑzÒ>(!*G!C7$
GÙKE6ÓÎ!I?$;
K %69G]CE$;G(!9G!6l5;-
6*G=UÖ57Ò*$]5E1KMÍ3=JK
ÌrIJ KEK7=¦â9Ì;.5E=U01%-*(*K7$ÙKM570ÌWÒ*1Ke5E=?ÌÏ1CE1G=?$l5Ña=¦5EÒ Ì C7$ÖÝ(!I%5E(!!=?!Ï*-
0 Ca(*K7$:57Ò!$]KM570ÌWÒ*1Ke5E=?Ì@G=J6Ï10 * Ic"%$<1$>P,$CEÏQÁ CEÜl(* CEG54Ó$r5EÒ!0GcA
=¦Ö57Ò*$@57CW6=?!=U*ÏK7$r5a=JKR!065a5E0l0IJ6CEÏ $ ->01Ca=¦Öv57Ò!$@5W K7Í=?KRCE$Ï1C7$
K7K7=?0 %-l(*K7$@Ì0 P e(*Ï1.5E$
Ï CW G=?$l5
A
&)I?1K7K7=?Ì;6IcK7$;Ìr01*G>LO0 CWG$C4Ó$r5EÒ!0G!K6CE$@=UÓÎ!CW Ìr57=JÌ Ic=U#6I?Ó01KM56I?Ic(*KM$ÖÝ(!I/Ì; K7$;K;A
ÞRÒ!$D*0 LOIU=?!$
6CRGÐ>*6Ó=JÌK)06ÖKM570ÌWÒ*1Ke5E=?Ì
Ï CW G=?$l5RG$;KEÌr$;15R=?ºÓÙ(*I¦5E=¦LOI?'Ð1$C4!$(*CE I
!$r5eÑ40 CEÍK-6Î96C757=JÌr(!IJ6CEIUÐ K=¦5RÎ9$;CM5W6=?*K5E0ÙÏ1$!$;CE IU=?ë;657=?0 %-.=JK)KM57=?I?I*Ö» CÖÝCE0 Ó P9$;=U!ÏÑ)$;IUI
(!*G$;CEKM570>0GcA1QÁ0 CE$a57Ò!$;0 CE$r5E=?Ì;6I!Ñ40 CEÍ *GKMÐKM57$;Ó657=JÌa$ræÎ,$CE=UÓ$;15W6I9Ñ)01C7Í=?K!$$
G$;GcA
8 3C ,.% 9 4<%
%(3; A "A "A 2:A HA LhNA6Q¼A6Ï1CE657$ÖÝ(!IUI?ÐÙ1ÌWÍl*0.ÑaIU$
GÏ $4ÓÙ(5E(*6I
$ræ!ÌWÒ*6*Ï $:Ï CW6l5EK)ÖÝCE0 Ó :Û :
Ûà6*$G ?!î)A
ò Eò/Æ!ò/Ä
ÊòC
~
©
©!t)Z]\rbeiש4)_W1bM\k_n\bej1i¦j1^ikjºgOshbelfEse be_nq>\rbM\;Z_Ese_7bRghq>\fW_Wg jl\se bM\;,beik_WZ]\jljli¦\j
^bM\
i¦_Ej6sn©YÀj i¦fMo>\_Wv )© d
W_Ebn} ikfMo>\;_E/YM©*
d;bM1\;j*},\j>u®old;Z]\;ga`_7seghf7o1_
}*_n1ised;begW}
}'
d
klZ_ }'ql\;^;_R~r¢
.©.u®o1_ YOu
`cbe _Wgh
gW}9
~n
6©
"!$#%&
'()# +*,#"
- .0/21-'3%+
¢6©
©vt)Z]\beiש R\rse1bM\;^;bM\;1ik_Wj.s®dbe6g5_ 4fWik_Wj.se i¦j k_n\bej1i¦j1^1©
768# %9':
'()# }
~W ¢ 7¯ ¢;¡6~ ; ¢
}%~n;
¤6©
©am©
®\shseiseiש*ibegOsh|c\;jl@gh_WfWd;j> |»dbM1_EbvZ_Eseold6 gc°Jd;bvk_n\bej1ikjl^1¯._Es[_W_Wj]gOse_W_Wq_EgOs® _WghfW_Wj.s
\j>Ùjl_Esed;=j < gZ_Eseold6!© }>£1¯k~E£1~ ;l~ }c~n;'¢.©
£1©
©>v_Wf7
_Eb)\;j>A @D©>y*_r lj*>©YÀZ
q1?be68dr6# ik%jl^D9se':o1
_a'()fW# d
j6'_7be^
_Wj1fW_d;°%{>\f76{ bed
q>\^'\rsei¦d;jk_n\bej1i¦j1^
iseo¾gh_WfWd
jl3d61_7bZ_Eseold1gW©YÀj¾pa\n6i¦3u,d;1be_EseE6.} _Ed;¿be_E ikj.sed
j*},\j>¾u,_Ebhbe_Wj1fW_
6_ Oj1dngh i×}l_n1ised;begW} }
ql\;^;_Wg)¢;R ;.'.©!y,\Wbe_W*j1fWB_ #"
% be k{>\. l0ZÚ#B!Ct'ghD2gh&d.fWEi¦\F seGH_WGIgW}968~n#
2¤;J ©
5'()#2K-'MLN#"5 O/2%+%7/P
DQ#"#
¡6©
)© ©.ikghold;q9© SB
Q'(TM#-UV!$# M*
'('3XWY
-#-. 2K'((# ©, ¦\be_Wjl1d;j`cbe_WghgW}1)½6°JdbM*}
~W
¡6©
©y%©ndshsed
*©64jlkikjl_/\¦^;d;beiseolZg,\;jlRgOsed.f7ol\;gOseikf\qlq bedr½ ikZ]\seikd;jlgW©;Y[japa\n6i¦a1\;\
*}W_n1ised;bn}
}
®Z \Z: KP{ beiU [\^
_
}*Q~n;K
.]¤6©Ku®^ol_a)_E
,sed
>j'(Y[T?jl# gOseiUse` se_a_EFH F_Ebebdik_WcAgW}># ®\;UZDBDQ{1#be9di¦1
^;'0_a'4DQAj1i¦
_EbeTghi':s[# e`% be2_Egh-gW'(K© '('3af
6©apD©1!©1vbed.d
Zo1_n\
Ù\;jlÙpD©ly*dn®_
© 1sei¦;\bei¦\{lk_R°J1jlfEseikd
jÙikj.se_Ebeqd
¦\seikd;j\;jl\;l\;q seik'_
¤ © j1_Es[©*y/d;©*be6gW1© j668seik# j1%_:9P\ j> /Qt
1 ©9-'3© %+ }_W¢6ik^
¯ '_Ej>¢.~*;6©
¡
¡.d
}/Z~Wql
¤;se¤ ik© jl^]gh_WfWd;j>d;bM1_7b1_Ebeik;\seik'_Wg4i¦j¾l_W_W |
1d;bh\rbM3jl_7s[®dbe6gW¯,tbe_W6ik_Ea© 2B
2
H
5'((# 2X#
>'(T?# U }~W
6©u,d
©
\ )ql©!qpR_n\r\bnbe©'_Wjº\j>1© © d.d6 .©a)d;se_:d;j¦_W\bejlikj1^ÙbM\rse_:ghfMol_n lk_Wg4°?d;bagOsed.fMo>\gOsei¦f:d
q1sei|
*,#"
--K./21-'3%+ =*,#"
- K. >#B!C'D20E FGHG 68# "!"P
}1q>\;^;_Wg£'" ;.£'¤6}! \;j \se_Wd }
~W¤ C© @D ®t©ry9}9_n~W
l¤;j9© © LNd#"be^'5\ j¬
-# \;Q1J°?Z] \;)#j12j92© '3YH
9H9-Q'(K-
"._3
-# 2P
5'()# 2 '\ -
QK.O%#"
~W C© @D5 :©;f y9©v_r`/ o11pTj9© se ol_E_Eghjli¦_EgWbM}1\4¦ikj1n\rik'se_Eikd
beghjÙis \;_jl`:©ljl_E_Es sÀ® d©beD 1 be_Wikgh_ ik^
j`cgO\shbebMikg\rse4_W^
Y ik7_W}9gW~W©
YÀ¤
j:6m© ©'`c°J_Eik°?_Ebn} ©.6f7o be_Ese_Ebn}
_W1ised;bn}
YK>B
2 "!$# %
'()# &*B#"
--K./Q1 -'3%+ \# }'ql\;^;_Wg¡;;R¤ ;
'¡.}
¢;¢6©C@D~W
©6;y9 _n© lj9}6YM©.¬\;j.se_Ebn}.\;j>Ù© t
© 6d
k¦\ ©%6_WfWd;j>d;bM _Eb®q bed
q_Ebhseik_Wg®d°,_7bhbed;bgh bh°×\;fE_WgW©cY[j
YK >
Q "!$# %
'()# *,#"
--K.&/21-'3%+ # } 1\j \rse_Wd1}1 ®t}>~n; ~;©
¢ C© @D©1dy9be^'_r\ j1¬j9}1\;`1S©°?Z]@D\;©lj1 j9ikZ]© \bM!}l\j>)©1`_n\bekZ: shse_Ebn©%t1sed
Z]\rseikfRk_n\bej1i¦j1^:bM\se_4Z]\½ ikZi¦W\|
P>'(T?# U } ¯ ¡;¢
"¡ ;6¡;; }c~n;
6©
"!
¢;¡6© © k¦_7bn© 1q_Ebe6i¦gh_WÁk_n\rbejlikjl^ºd
jÁ¦\be^;_be_n ljll\;j.s
shbM\;ikjlikj1^¾gh_EsegW© 2'3 P
'()#P
}>£ ~7¯k~n¡"; ¢;¡6}c~W
6©
"!
¢ © ©#S © P
\d.d6#B!6\;jlSB
)=©/Q11 ©r-'3pa%0\rbe '_Wj9 ©;>\gOsk_n\rbejlikjl^ikj
j1_Es[d;be6g,d° ¦d.fW\;k |Ýse1jl_naq1bed.fE_Wghghikjl^
1jlisegW© },~;¯ ¢;¤ ~ ; ¢;£1}%~W
¤
6©
¢;6©a
© 1bM\sM\6S©B
_:? 68#%O
9S9
J':
5'()a#f ©`%olpTseo1_WghikgW}l)jlik'_EbeghisÀd°%u,d;6'd1}*~n
¢6©
¢¤ ©a
© bM\sM\6}c¬@© |Àm© #l« kk_Ebn}/t
© !ik_Wo1_
}%\j>#©ct)Z]\rbeiשt41\;q sei¦
_d
j1|»kikjl_]k_n\bej1ikjl^ikj
fMo>\jl^;i¦j1^_Wj66ibed
jlZ_Ej6segW©úY[j ikfMo>\;_Ea )© d
W_Ebn} ikf7ol\;_WRYM©v
d;bM1\;j9}®\;jlúu®old;Z]\;g
; © ©l)©>)bhbn© 1P
%+)
5
P0 .H# K'D%+ !$# 7/2':#"
DS
-'()
+
2K. ©/`/o1pTseol_WghikgW})be_W^
d;j
bM\;1l\se_aY[j1gOseise1se_
}!~n;'¡.©
6~
© © )©D)bhbn© m_WZdr6i¦j1^¼jld;i¦gh_ ikj d;j1|»kikjl_ gh_n\befMolghikjl^ç\
1\;q sei¦
_ {l\sefMo ghikW_EgW©éY[j
ikfMo>\_Wv )© d;W_Ebn} ikf7ol\;_W/Y7©*
dbMl\j9}9\;jl3u®old;Z]\;ga`_7seghf7o1_
}*_n1ised;begW}
P
&
}.'d;¦1Z_ }
q>\^
_¢;
¢6©6u®o1_ YOu `%be_EghgW}l~n;'.©
¢6© © 1©y%
©>)"bh!$bn# ©* m%_W
^
'()1# ¦+\be*iknB\#"se
ikd; j-Kikj.0se/2ol1_-gh'3_Wk%+_WfE seikd
jÙd;°*bM\
i¦\;!{>\ghi¦gv°Jlj1fEseikd
j]fW_Wj.se_EbegW©
}! 7¯
R; '¢;6}c~n;'¡6©
; ©68©#t%O©>9`_W':\
be'()kZ:# shse_E bn©v>\;gOs_E½1\;f7s)ZDlseikqlkikfn\seikd;j{.Ùseo1_aol_Wghghi¦\j9© >
?68# %9':
'()# }
6¯k~W£.$ ;>~
}c~W
£1©
£1© © ©,`cbe_WghgW})©`©,¦\;jlj1_Ebh.}!©9t
©,u,_W1'd
kgh..}9\;j> ©,u4© /_Eshse_Ebekikjl^ © O% )
-
©# ®\Z:{ beiU ^
_]4j1ik'_Ebeghis[`%be_WghgW}
®W\
5Z: 9{ :beiU ^
_
} %6jl ^
=¦\D2j>A*
}*~n'>;¤
#B!¤6© /P
5)2'( ?
*B#-.
%0%+K.
¡6©apD©61\;\
!}._n1isedbn© Z KP,[\
2K.&
Q'(T?# -U0_aEFHFb cA# UBDQ#9I
''D2T
©Áu®o1_)_Esed
j#YÀjlgOseise1se_6_Ebeik_WgW}v ®\;ZD{1bei¦1^;_)jlik'_Ebeghis[3`cbe_WghgW}v ®\;Z@|
£ ~
©
© \qljlik© /Q':
'(K-'()
-
J[\
2. =D2# 1 ik¦_7.}l)_E @dbe!}!~W
¤6©
£'¢6©t© \ik{_W×} u4© \;jl\;n\W®\ } © ikj.sed
j*}.¬@©1 o1i¦;\;j1d1}6\j>¬@©61© y9\;j1^1©c`%old;jl_WZ_be_WfEd
^;|
£
©
P© /2 i¦. _E^
_E
beikjl*,fM#"
}1t
--© K¬a. d
Zd61\ }1\;jlu4© _Wgh
_WgW©/6sed.fMo>\gOseikfa j>\ZikfWg®d;°¦_W\bejlikj1^:iseo
Zd;Z_Wj.selZ ikjjl_E1bM\;*jl_7s[®dbe6gW© }>¢
6¯ £;£.¢
$¡ ;.£;£''.}%~n
£1©
£;£1© © © @®\jl^\j>©ctZ]\beiשu®ol_5_ 4#SfE i¦P_Ejl
fE\#B!Y\;*?j>Dº1 se-ol)
5_:Obe d;{llgOsej1_Wghg
d;°j>\rse1bM\^;bM\;1ik_Wj.s
_WghfW_Wj.sk_n\rbejlikjl^Dbelk_
©%Y[j ikf7ol\;_W*Y7©1
dbMl\j9} i¦fMo>\_W91©l¬a_W\bejlgW}l\;jl \bM\@t©1 d;¦¦\6}
_W1ised;begW}
P
K
2 $!$# %
'()# *,#"
- .&/21 '3%0 }1
d
klZ_a~n6© u®ol_ YOu
`cbe_WghgW}9~n
;¤ ©