Deep Leanning- Nos Bp Sawng8

PA

Binaystep funchiono E00A be a threshel-

ale clatnfier I. whether or nol
he heulon Sheuld he oueHynled baled on yaluc
fo 4he ncas rand fo(mation-
XO
F) o X 0

,-8inoyshep
funchion Cunb e
Wu Achvahon
funhen. gh.le
then tode
def binas Step(x) Creuh binau
Uuneier

uuhen h s e aSe
ekesn L
M-ulirle clans.in
hetal-Vuiade
binas-Ctef (C)_
binas ee - )
6lP (S,°).
The grudien ot +he etep fonli ov i4 Teso
whih Csel a hind sante fo he baektycpuahon
fAecels.e. if Caltulahe +he detivuliue
(x) w i4h o E (ovu ou
+obe
( ) =ofov ali x
ywueli n- aue Caltul a t oupdae the wushl-
 aln.
Deepleauin Nate B4 Casvagta
DATE
PAGE

nd hial duhiag 4he baukplergation PMotes

Ne4rdiaol oy hefoncion iA zee,
he weihl ad biales don' da

Sigmoid foncon - T is noo-leaeal

uHUaie fondHon. jq moid taNtorM 4he
lats blw he Tane andL.

Sqntax-

-0-L
2 3

MPor nu as hP
def ligmoid- foncion
(n)_

SIgMeid tunchHon(+),
Sigmeid-fundHen (-22) b994
u i uooth -shaed funen_
Oud r Cotintuoul diffeieahiasle
The deis u a h fondHen Couu du-
o be (Sismaid(x)*(- Siamoid (x)).
F qmai ) * 1-Siqmoi A))
Deelens ina, N o e,: Sawagr xdn
DATE
PAOE

The radienl- }alue aue n i i lenl- For san

3 m 3 hu- the hrah H mut ette

t 0 h t y remon. Thi i p t y thal- for
Va reaber than 3or leH than - 3 , w
w e V tu Smay radital= Vadue auachu
he Aleueork no- raily 1taauiuy
S19moeid funcHen 1 o l - SqmMetmc
aeund 7 o
Tanh 6 Taneb-(hypebolie ):= the tanh
4vachon
ve Cimtlau o theligmed
FoncHon
dFeelue td Hhal- E1 M Mehic.
the u Vauluej
a o u u he a u n
The Tau
Intu-
thds Coule i from- to l Thus the
in be t +he

to thenex>ayT uut nol- alda

ame el - e

Tounh) t1+e-a ))-L e+e-

tunh) a& Sigmald (an) -

 Dee lecsni Nate by Santagya Dain
DATE
PAGE
Phentode
def tanhfuncion (X)):
7=2|1 nAexp (-2*x) -
ehusn 2

anh-fonct ion (oC) t4n hfunchion (-1)_

D 2
o.H6

A Ae Mepadnas anh fondha)

Siuiet Simeid FonHo)
+anh fondion t l a i n u s u s Cd
oi ffes eti ahe at a Peio
tanh'(ns 1- tanh 20)
+h uu'eul- at he tanh FonHon.

3 3

the a eus t a t h tvneH a4 4

+eepe Co nLfauel +o M e iameid

Tan'h refere oU he (ymoid

+0nei on Sinte E A 2elo cenlere ane
the
adien)a aUe nol- tmehd o
MUse ina Cetin direeian
 Deey lelnj a-Nahu vy Saw an_
DATE
PAGE

ReLU -
he ReLu Shandsor ReefieduneasUnil-
he M ain ttud Uaujaqe a A the Relu Fvncion
&U 4her activaHon feniond í thal- \Joe

noaLHvae etl the hêulous a he (aue

0ul
Thiy Laud 4Aal- h e h e e n s wi
be deaciyaed if +he duhp- 4 h e ntas
raus Corm aho 4 leHthan o

MAK8,X) ) = X>=0
F)

tor e neahu íntw- values h

h e neulo
mddid 2eto , h e s
Snle Oul a Certain
dou nal- u ae v e - ,
he Relu Foncioy
No. of heusonu ale acrueked
hen
LoYe touputahonall7 e fidenl
far
Couupan eel o he siamcid a2d
tanh foncHo
Pythen Lede,
det elu-funchon C )

tusn
se

eluFuacron (F, Jdu-fonchon -

(7,0)
 Deel lexiui Noes B Cawega Dai PAGE
DATE

he aradlieal- a, +he Relu Foncion.

-2

eetiu lide gradieul Value i

odueto his rsau ' dusuina
he bek PA epas aho Puece
he weighl-: aud biadd for lome
ausont as rne updhed.thy
Can CTeal deaA_neulou
Can u i dh
nw e-auHvjeA h 14 +aken
Cay ca b4 h e Lea Relu Fonho.

) eek Relu fphoued uesusíon

heReLu fvALhion A LL aw ha
the ruAienl- s o gcxo
u del deaeHvae me heulo
h e espou
Leaky Relo u defia Fo
audreH hi lehlem. nd Fead mdeiniue
he Re:u Funuhon a ofor negakve
7 yuuuy,
we depine i aj extcea nel*Sall
ne Couonea 4

70
 Deeglecuminq Nohs O4% laWaqyn Di0ATE
PAQE

BLkíng hiy SAlal Madi filatlonhe

Coe b e a non 7ejo vallue .

Heule uLe woi ho louerey enlouues

d ed haulans fn thol Men o-

def leayYelu-funchan )

lun b.01 X

se
elu

leyYeufuncHoy+ )
rely-Foac on - .

Paameteki Jeel ReLU-TAe Paia meterised Re Lu ,

h e hame su4IH 10trodules a ew
PcuAMeky a a Sle pe n eneakve Pal
fa aa
he fotion_
9yntar f() X
= ax
Dee ltang Nobu 4Caaayn rdn
OATS

n Patanehetited Relu Foebion a i

au trainade fala Mcher+he nehoyh
ato leasu{ the Vulue o a for Fuuey
wd

ExPonenHa neas n-- ules a le

uwefor den the hegar Nalue
4 dehined a
X7z0
a (ey-1 X2O
PHen de-
def eluFontion X, a)

3uuna np. txe() -1)

e

lu-funtion , o1)
i -
-5 D
- O o913
 Die leuuing Neku Sawafya n '

4). f+max functions

-Safmay funisn it atlen:
duuhed a a lonuhin aHon al
MlHRMe igmeids we kn talgmeid l n
Valaublo 0 aud L. hich Can he treued
a tAehahiuHies data tein)- heloua to a
ania cla
iopSetmay funion
Can he
led or muhdau lakhiHCcui c _PAablemi
funia tum the Malalsi Fora
daatein bel euan e o eah ini viduud claH

=Eep (Xi
Smax lki)=
P n tode-
daf SoftLap funch au ( )
2 nP eke ()
z z . luml)

S cenayfondhon Lo 12 3:11)

aayC 0. 6%0 6.1 boo

u le busliu a nehoor FY aMeh Clas

hl m The cuu- lcuer weuld ha d

Man huladc he No e clau i n
he a4 o r inate it4u haue
h c t e elanu 4hUMe UAulA be hree: neul ou

20
 Dee leasi g Neu 98 Sawa4a D»ia
PAGE

Sack PMapaaahion (o deep leasniaq,-iEi4an

A190. Hhaul-PA ofaqefel4he e r o from ow to C(N.
Baik Abeqe PAetagion S h e eMantea
huel netuwrk Taining 1E he wethe
Dfin tninathe dhe weihi
h e a l n a basee on he emov t e
u n e s in he pruious epeuh (ie
tahen)
PAupe uniny he wu{n
elew u Ho redluue eror
kethe ulodel Jeliahde by tooresdia
fH enualizion.
Bakpepasalioy is a Shot]-
focm Fos baukwand Pepaqahou e Gro

auiiu a Nesa NekooriA. thu methed

hul Lwlale theradien- a a
fonioy with Upee to a l +Ae WeinhE
the nekoork
Sige ikasbieo =
LnPat Vele (%1
OFpuFNe
Lnt-Vele2 (x2-
-Vue 3 x m
al
Vlut
Aes- fonel

8ek PAopaqatien. PAalen auinue uoH the

Vulue a log funetion t euehe to 7so ov
n e by Zeo. 4 butpul valueeatke o
 Dee lezsuuing Nas B4 avuyn l n DATT

eeu e a t u l Vallue.
Nod fin he bukylepagahon PAolen
un final foun

Lelonu'deied An banpie

, WIE H NLeo4
0.30D
AO 0,61
0.D

G N:0oH 0 0.99

heth w i , w * i +b *1
- 1 k 0 + 0 . 20 | t 0 3 0.3*S

Siamaid AHvaisn funehon i4 w u

Cuufu
32£4a9 2
Gukh T4073Tt
T+e

Same wuu Repuul Sep to Cuulal-e buth

ourh O 466&93 76.

el-0 w bath +w, t0uk b+ b1 xl

y0.S93269942+ 04taK 666 336 t06 t

Oul-b +e- hto of 6 r967

1te
 Deeelecctwing Nohes 0Sauya ain
DATE
PAGE

tme p a - Repea Ste Ho Caltalae oul02

bul 6.719164&
APrly
o s fonchon -

s e
2{{raugel- oult ut)
Eo (tusgt -outo,2
D: 7 3 60)2
027461lo63

Co 0-023Ldo26.

ttoku i+Eo2'=
2463+|l0
The Oauk wa pu

pdae +he win-yig qradien)-

desten
o 4ha- t h lauwe he reicl-
D u t t o be later o the Achual but
Theiub minimyy, dheeor For
eLuh Ouut neuson Ond -he newote
a a whole

AT6utp-later.

owAide w

Eto kd asuhial deaiue a

w
 Deeleuni nNoy 04. Sauvaqya DATE h
PAOE

l a n nd- dreudy diffesetiahe ay hav

to find Some sduton bl Etatl w
So

Etokd a0uto ne-6

out0 a Deto)
Eto takjg-010u-o)2 t (taguko2-0h
Etotal
oulol
tae-o - OutoI) =-lo.o1- .9i3ot)
0-4413 ro

Jouro) Odo -6uto)

a ncto

.913607(-0.36rot)
1866 (Taz

Det o Oud-hi *60thL+53t

Dul-h1 t w -)+ o +0
w Octh
0.S 3 26 4412

EtoalL b.29136 fo7 to.16661bo1& 0 S3267112

w 008216o

daethe weiqh=
TO deATeaye +he emor ,e then Strau-+hy
Malue ho M4he CwMenl wei9h)
 Dee lecunq Nobe 9CawA uin

-h EFoal

0Y bS0.02160L4 0 3 116

leumi aq sae ea)

ane was-Repeal- Ske o upaube..
o.o6k6 166

ATHiddenlaug T6 le&ides weighi

Pasuli al delj vahv u JEto kal w.7E w
Etaa
outh) w

Etobd Eo E02
outh olh

Eo1E aget-0)
ure 1+-hefo

hero wtouuthi tw ou-b b

Fo heo
urhi Souto1 ahto 0uth_
to Jhu-
DuFhi
O 13644B62 640
 Eo2
0ut hi
- 001ou1 19

CD,ol9DyT111) 0.034s
Etobal1 Oo 311Y2
o3o6

DuthF heth

0uth Luth -ecthu)

a hth
0.13 26911 -0-3247?24)
6.2413 oo }o1

( hth o 1 * t w3ia tb*1

hth)

b0 36 3So 306 0 24I3 0o 71 0 0

0 0 004 36S68

UP&ute WL

wi
1 0. 0 0 0ol348 =0.1417801

Same voasz!= fetud &r o (alula-e pda

w 0-1955 I43
w .2443 i
T 0.219So22 9.
 Decr lecusuiag aloft 0 Cakyayy d DATE
PAGF

tinu haue duhed au at 0ce toemit

we
when we fed Forwatd he 0.oand o1
qtd0pnau e o r 0n *me
heto ove was o.2163l o1
attey+his fiTs Tound e
hak Mergtio
the o tel m r y now doon o
0.291o2192

E n l - nok deem k e Muu, b -

afer eeuk e h s Mam l0,o oo

Plywme to 000oo 3lo6(

Chao Ruleß
The hao Kule alo o
ind Ae detui vatue a Coupotibe fitnehon
The hin aule lauA

Fs)-ftgti}zy-
e u u how o diffelutiabe
Coupodie funehion.
A funhou 14 CauLpotibe if you lan
usute ita f « ) in o4he Lsords.
ir sA a funuh o wthin a tunion
Ora funtievn dftvndtan
 Dee leruni Nok O9sauwagyn ri n
PAO

for ¬yenpte o(x) 14 (onepo tibe betowse

fw w f ( ) Cos(n) r (n) =
tos(n ) fg) daforskion
within funhon f ov 4-ythe a a fuacioan

So we Col th Lohe funebion a

f the ule funnehion

heneialized chedn Rale-

we Can entsaure h e ehain rule

buyaud 4he nivatahe Caa.
CAde he Cuse uheue xe RM d u t R
wuich Nean thel the nocfUncHen f Mapt
M 10u Fo nGutpuul- , uekile due funtion_
ueiuesh n fala Fo PMedure on astr h_
for fe L +he 9eneiwli znd Chn de
Shal-e

7h) Ah Judo

dhJu

lan ao viJaaline 4he w0Aukine afe Cheui

de by a
ree diava

X2

Ppruu- 4 Chuin Rulu by e diaqray.

Rpredeuukahon a Paw L Dmulhlaqes prcdehey-
Pesertsen itsa oqe AAN E Sinqle
laqur heusel Nekuroa k
AfexLeptoch dakel a Vechov «eAal - Valueed
L s alLalahes o Lntu Louuhin ahon
thesc ofl3 hen ouutpu-a 1f redl
sqrecher than &ome thresheld_and-
theuwiie

ou ven n fuls (a, XA)he oubt olx Xn

b the peuteprn

wheae w? i a e a l -Yalued lovitanh oY weiqhl

h defemire s the eruhnbuHon e înfl- K}
t o the Pestepthtn but
 DATE
PAGE

o A hrekhol) ha he wchi ed
Ccuthinekon a 1 d Le Wnn my
Cuyay io crd or +he puuepeh
o outi a

Somel me , the pOLe e n fonchon 14

Wu te n u

O m iw.)

wheae

t-othewije
wuh

Ou

(n
Peuleaoh.
Repiafahena Pewei Pulp ser

the deicn Scataie Yepndade

 0ATE
PAOE

Se- aininq Exanwlec md he deiGon

SuaLc eprei +hal clajkf Them
Lenetl
b) Se rain Egnp that id noj- ineasc
SeriahC
X X ale he feulepho 'npuis
foiti Eampics adue india b
heati b "-

The Pe4lepoe\ Can be Vitd

Prebekiska a
hytpan detikosn ustae 1o A dimersional
Spaie ikaie

The Peuuqhun auhcs a i for insami inp en

&ue Side ct a hysoplan md ouheia-
fo1wantd on the ethei Cide
A
Sinpe Puulep rn Can be wd o pnesenl-
Pnr bcelan uncion. >

N-woTk aroten Ho
A - gap S o t o IHemeulany boelkan

Ahen wi a NuuooY & feticNens

Lu- aum Toue =

+ ,faise = -L

we louideu 2 intu-Aud Picepmows

eh - iA Connued o althe 4
Puulptwns with Ceiic wwei{A
hebas( ) Euk Paup ben 14-2
Tie eah Pleden ui fir CLL i
he
 DATE
PAGE

wighe
Deuh_ +heje e hent is Lonntuht o

0uhpul PeuLepyon b weiqh)-s

uwh hecd o be \uned )
h eh e duulu- peucextn )
Ntoov

-iN(The 1-(

bial-2

Cine t nsiues w =-L

Dyhd Ge

his Nekook Coubad n 3 lager_

he aqer lenkininq e in fui-
Cilee he in fal l«qer_

he Mialle laqey Lerheun w dhe 4

Puuyon i Cuuied hdien layer
he fid tujer Couedui_ one Ocultul-
huson i4 Cullcd 4he otukp lager
he

the ilden laqr aue duustud b

h h ha hq-
Dashedte
try wuhl-
 DArE
PA

we Clin tuil- +hi Nteovk CAn bt i

to lpiem)e) c booan Fonion
nul Seprie o ne)

c h PLen heiy i n dhe idile «qe

fi oudy for a G i C inpui-(and

ho tu o putepens fire Foc he lawe,

Mo 1H S WoYk
b an Exa npie the XoR
Xon fFouic

L-
L-Co beHe, dias tuui o dhe hetusu
whi >w

XoR ha y

i nalH in ne todtgi Ceui cÁihioi

eial ouh
o
OY ne cO he 4 foibde nfu and lan be

-e- e ive Guuf or thu-

 DATE
PAGE

Theorer An noo Can funcLion n P n frLt la be prelentee

EXall hy n Neluoyk eehron CenLeini
hidern
n Can rinq Pecephon
Leo we -Seus hov Cou e - f u no

Sigmei Neuhen The Baiic Bud)dinq

BloKs the peep Neulal ekcosks
fe Cailed
the sfimoid neuson.
Sigmed Neuton aye cfmi|e to
Pe ns: bul th asve ughl7
Modi ficd Suuh. that-+he oukpul Rom
the Simeii Neuien 3 Muih imoeter
than the Ster funchion al ouh
Dom pesulephen

1 h &i gmeid Neuien

In 4he Pea{e ptson model Gusufi
deided bya hYeshold wo he Medel
wd re if the weighbe eum id
9reker than the hoeShe)d._
Sothruhaidleq w th_
ie b the pultptron i4 Vau Hatuih
Souetine
1 See the hawh thvecheldisn-
Lsith an éxamie
 0ATE
PAGE

onghe Lonsider the desie makínq (Moes

f a peiudenuhether he he wcutd uke
to pwuhase a as r n0bicd n au
uneCn X Salauu cm b
Cetti he hrech01d blu'o) -o
e 02 the u t d e y the puuptron
Me4e w{ look Uke

Salasf Canht}
Ata

20

Cales io thosand

tree Le arn See Phl- He Peusan wfhb

Salay wthen So k an al- by a uy.
whule ala iA qreuher then o k (an b1 4_
CaSo Vay d, thu- a Peuio0
with aa 49.9k Can no- Pdheuje Ci_
Cu end So k Can bu au.
the mad Chaugr In the ÍnkLl- h a Puicphen_
Cun Soneine lause the uuul- te
Coelh Fli a o L
to 00deni thid Leblen)
hh nnodut a nee tme a auiRuial hesen
SigmoidOeuien
Culled e
 DATE
PAGE

n SIqme'd Neus,en a (mau change n he

O dud Cutuse a Jma Changen
he Outptl- as Crcosel o h e cteppe
uttl ho4h ofuf and
tn meid tUnedion
utpulai YCel Neutionf tta
he iqmeid
B o +o Av uuhich i4
Dcal Vceiue b)u
Smeothe Y dha) bina 10u

heae ae Mang fonchiovi w # the

ChetatteHu a an 's Shapes Cuwe
Knewn a Simaidfuncons

h e mos)- Commo uied fvnchicn is h

T+e-tWT+5)

St mei d Neuse Reprejeneeon

we no lenger See a thaue toaui ion a-
he thruho!d b . he outfl- fo M +he
i moi Neulenin0 o or L
iNteadE i a Teal Value blo O-
whih Ccn be ntet pye-ed a pAchahlk