Neuial netuworki and dep learning (pe D)CigcsE23)

Atitiual Menal metworu

fundamental tente
Al cuual netuwovlu aue thore itomnatim oueing eystenms, w l i t
ase lonutuuted and iuplem entd to mo del tthe human rain
The mam ebjeutive e tle menal netwov veueanh i to dvelep
a
tmputatiomal deuite for node lling te rain to peuform
vaious onputational tauu at a tastei vate ttan tte radi tien al
uttma
41tiual leual natworl AAl N' pesse lanqe numba e
1nte tonnetted rocelsing demenu talled nede 0Y unit a
nenans, w i t
uuall osaat in pala llel and ake
eni7uneol
a vequlak aa tlitutunl AMA's Leletive behavioui is
chanatt eited by Liv alilty to leasn, ntall
an enealite
trainina pattuM v data imila to tlat
el a uman brain
(*
W

hrelittutuue s iple ANN

i
i Net iuput
a autiv ation of Le
uE neuu ens X1 and x, t.e,tte
Output e iput inau
ww weigluts
The output tte dutjut mulen
Y Can be obtaned y
aphinq autwatiens oves te net input v.e

fyn)
cutut funetion ( met input talulate d)
at n euhons teanal Tate s calleol te attivation.
Bielegital lemal networ tman bram tonuiru ot to
neukoms ap ximalely

Me AOm

uleul
tetl ody
Cema)
Dends

Stluematie diaanam al aieloqfral neuken

The tiue lmuict o3 main pasu
Soma 07 (ell Lod4 w e te (ell nutleuu t lotated
2Dendaitu wleue tLe muwe tu Lenneted 4o tte Lell
Ledy
-

3 Anm - w i tanie tle i p u u u t e nun

wugl
output
W2
output AT

(Xn
wn roek ng
element

Matlematial modet
et axtiiúal neuken
emine loqy rlaui onthiM Letween ielegical and atiiial neuc-
iologiLal meum Artitiual muwm
Cell ewnm
Den dait Wlgt ov întsLonnetEien
Soma
Net input
o utput

n ,w t 1i W t - -- * l n wn

)Bram v Lompute Lernpaiion beween biolegical neuson

nmn
cun d Arti fiual
Speed t y de time o AAN I nanoelondu BN i¢ millisetomds
rou e i n : AN s fasda with paallel opesatiens
(3
Sae and tovnple ufy k more for tle brain
n lotatioms
tevage tapaulty ANN Ctovel fomatim menon
Tolerante ANN a no fault tolenanLe Las tolekan te

Cnt netLenim. n ANN t t vex imple .

tvolutom e leual Metuwovl

Yea Neal i m eh eaniption
wetwor
1143 Mc(ullocle and MeCulloc and The angemeat e newnn4
t t s emon Pitts n tLi lase îs a
lombinaticn
olege funticns . Uuique
featme eLis newaern is

1149 Hel network tte Lon ept e tashold

LftuD muOn ale attwe
Simultaneouly ,tomnettion
Strenq& ic inaeajed
t158
Peaepn ran ttese te
ogenbLatl weigut on tle
la62 tonnetbi on
Lock, Minyadjuted att tan Le
Papeat
160 Aaine tidaw and
leigut ae
aduuteel to vedulue
tte ditteente Llw the etil
to tle ot and d
eaived otp
MsE ob tained .
A12 onen elt-|
totenen pule are tlutehed togetle
0vjaniting to
ealne Ma p etan a fired olp neunen
1182 tlopliela Jehn 4top{feld Baed m red e
eigtite TLeze
18 uetwon and Tan
net lan aluo att al alcouatw=
19e
Mem0ny nety
11
86 Baukpropagation Lumelhast. Mululayel wi EL eLnoi lali
Hton and to
metwo L
pagnted 4<em o lp
wiiamM iaden unit
1189 Co untet- mila to Lohemen le
.

asniug
aation ottus 4er a l unity i a
networ
u ti atlas laye
1494 Adative Por e t t Linay and analeq
110 Le man Le
Gmes Vaued iputs
Theey LAT
t49 Ladial Lt as romLead luemblu baiu
a
popa4aticn
funuton net and Lowe
wit gauian funtin
Nto loj nitim essential for tanantu
FulluiTna
remituen
Baie tmodels of
ati fiial mewnal networ
The medrl ofAN ase p eitfed t e 3 baie entities
tThe modeli naptie intaonneiti OnL
Thetrainim
2The ov
Leasntng mlu adoptrd tw7 undati ng
7 andd
adjuting tle onneution weit
3 T i aitrv ation funti one
Comn eutie
Theae ane in 4ypes

Singte lay feed fonoasd netwc,l

p lay olp layes
w
(X
Wi

w n
utput neukms
w o

(Xn Wn m (
*uu ale
lonnect ed
rocesing nodes wit& vaaio
n te
tul ting to suiL o outpuu bne pu noa Input
and
output ase Lin ed witl, eatl othe. layes it fommed
taling rocesuina elemenu and lmbining it witt o te
ro tesia elements
ti Multi- taye feed fFo
nwaid ne twort
dden yes olp lay e
wuy

ub outnat
Ewwms hewno

(Xn
tomed y tnt eatonnetbin o Sewenal
t and layen. lay esbl
output is tatled idden layeA. +lidaen
is iuteAnal tv the metwo lay e
(1) Feed bacl
euworl Single ode wit f own feedlai
output

teedbak
2 non eulom iu the Dutput lay eh [s an Iuut to a n ed e
in te (ame layel receeding layes l
tall ed feed fenvasd
mewr
1 output ae divettetd
bat a luput to tle t e fhg
elemeotu ln the ame laytk ettedinq layes Í calle d
feedba t network

,J tLe outputtale aiteted bait lo

tt input of t,, Same
Ayes tlen it is (ateal fee dbak
leuient networu ane metworu wit fer cd all
Tetwovlu i t leted Loop
Competitive ne

- E >Au
Red wegt e -E. Tit net ts called Maralet
iw le La ruAnt metuoort

WI
(*
( 2

Wnm
Sle Layes netuwo wito fe ed batu lom oubput Can e dive eted
to oLeing element iue, o to u Proteiug edem ent[ e ti

)Multi layel Aeunent uetuodL

Wnt
o LeAna alement output be
tan
diveute d basb to the nodu iu tle
e eedinq layes, formimq a multilayes eununt netwo
ProLiug element output can le dirnuted to oeniw elemnt
Tte to stes meing element m t e tame layet

i latal r t i o n ttutue on -ten tu-otf -Sumound)

a protesing menon rteivs two Aitteuent dauu tpu
e s itatony" i u t {om neanly rocesing tlement 4
iuliiton' elementu torn more distan tly letated mteuing
element

2 eani
AMnt A wo hinds leaunm
a a a n m t leauaiuq : Updatu t e Lonnecbion wefgLt

ti Stnuue learnig fousu on tle tlange tu nlw suuuliue.

AAN Can be qenesally lauited t o thare tateserieu
Aupeawised lealni
Neuna
X metwo
fut (Atual elp)

ero mal
L0-4) enkatoY CReied outpud)
Sinals
leainiug is petomeo wit t e elo of teailesk
v e t o r aleng witl lauaet vertor Is lalled rainiug taiy
Dui trainfug,the p Veutor Ts reentd to the Aulo, witl
Ault In an
Ipveitoy
Lm pased wit duid olp veutoy ,ten eoi iqnal i

eneaated by nfw
t is auumed ttat te lorrett
taset' ofp vauu ake loown
fov e atl
iuput natein .
t) Vuaup eauised Leaknis
AA

Ciuput) Atual oubpub

leaniua it thomed thout te etp
p u t veutors dmilal type
a
teatl
ae
ouptd ittout tte use
otraiuing data
tatLile dituv eina all ttye eatues, tte netwv
undugoy
clange f it Najamtbeu. TLit Tpecs is rall ed *

u i
fy ovqan iting
enaut duteu wiu lge tomed .

() Leimtonement Leaníug
AlcturalL
MetuonA >y
Ct) Atual output)

inal R
mals enekator tAtinforteMent
itnal)
leaniua baed tn tte titic iomati en is tall ed »etufertemea"
teaniuy and Be edLail (ent
calle ed einhenement Hmal
hwiseal leasniuy te toytet tuqet ol values ale
nown fov eatl fuput pattun leintonenment leanina ,leus
omatin mi7t be available.
8. Autivation funttienc
b Daentity FunitiontC«) =a for aU z
The op hene vemaiu the Same au iuput >X

8 t a y step funtion fx)=S f n26 tLaelheld

f) Lo if <0

Bipolas tep fun ctiontx){I If ^26

ivy oidal tundtim: lidel ued t Raukropajation ets

ta) ina imaid funtien: losistie timwid funetion or uniFelas
imeid tunction
fCa)
A Ste0nes panametei
fC) =A fC)f1-f()J
Yange o to l

) ipelar imeid tuntim f(a) =- 2

+e

4a)= [itf0«)]l1- f(a)]

-ASA-3 A=tA = Ti is tlet ely velated to lypesleliz
tangnt funetion

Lamp tunction fCx) =( if >) t)

o 1f <o +

+
Tmpos taot tuminoleqteu ot ANN
haiguu leiyu lomtains t e lutomm.ati on abo ut e lp imal.
eiwti ake vep ryented in tema o matti
hleigut matu fe auo Lalled te lemnettin matrin

. W2m

wi weugut om pro teting lement i to roceusinq element J

Bias ias induded by adaina Lenponent ne=1 to tle
t ut vettor X 1 e iuput Vettov x (1, X, X2, -.- . Xi,
=
>Xn)
i a is owidened Ule anettas wei gut fe oj =
j
in nw
(x Y

(x wi
Wn

(
3 TCret.eld Trehetd value l ued in te ativation tunetien
A Lempaiton made b[s tte taluulatdnet
met.input and te
teshsld to obtain te networt
eutput
Cnet) =S et6 Thehetd
f net
4 eaning llate 2t is used to Conlh»l the aM Ount of

weit adjut ment at eatlu step of ainíug.

= o tp , deteanni ne Yate o Leanma at catl tep

Momentum Fattoy: CeeAg ente is made atu if a

wwow.estum autov is qdd ed lo the weigtut up dation. o t es
Mome ntum helps tle net u easonaly luge weigut adjutmenu
until the tovetions ane iatle tame fenual direution tov Seuul

atlea
Ciatlane lauaretu f4 i« uued to Len bet ue daspee o h i l a i

ttuu t unj4 2t
euired or pattens to loe auimed to te Same
ed i adaptive reonante tony CART) netuwo

S o to
Motat o
Ativation unit Xi iuput inal
1Autivatiom eh unft [= f )
j h l ü h tom tonneuticm frm unit Xt to unit j
oj iay auting on unit j . Bfa Las a Lonutant attivatiom

W l u g t mati, l ={wi
i Met put to unit j ven by fimi jt w
lzll l om of magiitude veutor x
OjTrelotd fov adav atin 0}muAbn j
S
STniing iuput vettor.s =CsS.S, Sn)
T Traiag output vator, T= Ct, th tj,**n)
X uput vutor, X Cx,AL,. Ai .An)
=

Aw clange u wáglt given y Aw= w Lnw- wij Cold)

keauniy vate; it enolu t e amuunt w e i q t

adjutment at eath Aep o traiuing.
M. Cutloth-fitly Nuson

2As wually calted al M-P Kleunen.

->M-P meukon ase tonn et ted by diveeted weiguted patl
Akivation oh NP Aeuon t binau
hteigtut auoúated wstl Ue lorm Munit elion
a may be
 12
ntitatey( weiqht fe posituve) uliLitoy lweigt tu n eaatie)
Al exitatony tomneted wti gut en teing uto a utiulas
ntnon will ave lanme weints

Ariteiline
The ativati on-fun tion is dehined as

fCyi)=

Antm
For uLiton o be alkolute, etlaeLetd wi tl tle ativ ation
funtiom ho ulA Aatist tle fotlersing Londiti on
O>nw-P
<The
TLe output wil fire f it veteive
Say t" oi move utí
i u lout no fli itony iuput wlue
e
taletq
w >
tt-1) w.
M- euan has no
Datiulas rai uiua algritt
analyis Las to be
afommed tu
dtumine le vauu al
te weti and te austd.
Atetuons_

Pte an netwo ds twme undes ingte- laye fe ed fo nv aid

-
metwo ru and ase auo talled inu ple pes
teptuong
TLe ehtertuen newo tonaictu et taee unit, armety, Senu
nit Ctput unit) auouato unl4
, Lidden unlt), eponse unitlou
unit)
I h e Aenon unity ae Lewnetted to auo d atet unit i t fiala

e i t haiug value 0 oY t, wlit ale aMmed at randLorm

TLe iuay altvaton tunution fs wed iu Senoy unit and
amouat uni t
TLe unit khas a n auivatim o o - 1 . TLe biuas
tsose
ste wit fhed tliuhotd 0 1 ued as aito atuow -fev alieia tok 1e
uhut maly uat axe kent t m tle auoùatet unit to the
TNone unut ase my binay .

T h e ottput tle uepton etarol ft qiven y

fC)
w e fCyin) 1u aztiwatiw tuntion and i dutiued a
fyiu)-(I if yiu>o

T e w eiah witl be
adjuted on te ns o Ule leasning
sule f an eun has ouuned fpr a patrul Las thainig nattesn it
wtnew)= wjlold) + dt»ji
Llnw) = blold) t «t
Dultput oet dutput uire
ined weigt utput
V alue o 01-1 at
yand w ()

wn

Sen ey tunut wn

Seryor 1id tn
*vet entmg ay
ttean Assouatet unit Aeupente unît
Oninal peaLeptuem weworl
etepton leainiug Lule
is t l a
u Cale
o tle peates trom leanniug mule, le learrmg mal
iffean Le betwen t e duied and auual rspemse a ne m .

Le wat updatim
i t then wtnew) = wleld) + at a ld>ltaing yate)
ewCnw) =wlold)
Arit eutine :

w
wi
wn

Xu
gle elaui fitation peteptugn metwol
7Le qoal o te peuestin net it to
anifty te îput patteun
a embe o net a uermeh to atiullal
a lau.
Flowaut tor traiwing mes ite euwtput)
Stant

uitialite weiguu

Set al oto 1)

(A)
 (A) S
for
ALe
ea

Aeiv ale theput unit aj=si|

callate tle net iuput yi
y autivation iotaiu
fyi)
No
y=t

wtnuw)= WLeld )+ diai| wjtnuw)=wield) |

blnew)- bLtd) +at bCnw)=Llold)

wut
changes/

ex stuum traiuiug slaonttmw tor iugle outut a u :

Slep o: Aializ e tle weti and e bia .Ao initiaize ti
teanniua vate klo<d£1). tor imptiüty « is sSet to
eep L pea form ste -6 u i l tle fimal
ttot ping tend?tion
falee. îs
sle2Pea-formAAeps 3 lor eatk taning ais mdialed by s:t
ep2 Tewput layea tontaiig luput unitu ir apsled witH
deotity aetivation funitiem
 =
Stepy Caluulate e output a tLe metwodL To do ev, fint obtalu
t e met iuput
b+ ajw
wene " te mumb P 1um
1u tle iuput lasu Tien
ativation oves lue net imput talulated to oltaiu te output

Step leist and dias adfutmunt lenpae tle value of tl

autual output and deirrd outp ut
, ten wilneuw)= wjlodd) + dt ai
LCne)= Llold) +«t
ee we ave wCnew)= wlold)
binew)= blold)
Step6 Tyaiu te netwovL until tLene it no wei7at thanqe
TLs 1s t e toppía n d t i or tor t e netuwovt 24 i on ditin
is not met, ten Atart aqain trom Step 2

Metworlt anitetune fov e p bon neluwo di foy Seweal oulput

alses (

Wht
w

wn

w
wam
eteturm traiing algentlm bv Maltisle output lauu
Stepo itiatite the weigtute, bíauu and leasníng vate tuitally

Siepl: eeet fov ctoppinq ondition; H it fale eato m step

9tep Peatorm steN 3- for easl Lipolas ov Lina traiuna
Vutor air s't
St s Set ativ ation o eal input unit i=1to n

o eal output unit j-rto r x,

Step: Calutate output rone
e met iuput is calulatid as

+ j
Ten autivationt ae pled oves tle net iput to talulateu
Output eiponge
-fCym)=
siepC Mate adjut ment iu weirty And a for i1to M and
- to n
3 ftYf,ten wj Lnew) wij(otd) +«tjnj|
=

ftne) = bflotd)+ dtj

ue we have
wtno) wijtotdl)
Lne) bftela)
step Test for tle Stoptin Lenditiom i e, if Lue is no
thange in
wit tln tlop tte braiuing motes, ee tart agaiu
frem step
e eptuom Netuoo Tuting Algon thnm
sAepo: Ihe juitial wegtttu to Ge uu ed heue a e taben from
e
a n g alaonthm
stepL: For eat iuput vettor X to be tlassifie dpeafom Step 3
slep et ativatjons o tle
iuput unit
Steps obtain tle m e o output unit
wi

fCyiu)

TLe lenditien fov

to
Sepasatinq te veupnse em ngien et Noritive
reion o teo [s

Te
etendton tov Sepanating tte reupoo t r
vegB on ot teo
t 0 to
to
im cnegatre is
e
lndYtronu alove aie atatd for
wit twD
iugle layen puuftn neluo l
fuput neunon and one output nenen and on ia e

Sigmid Nunon
t e imoidl neusn. mall dange in t e
a
iupuut onty tauuu a
emall dange în tle
output as onosed to the stepped output
meid nunont ase i u l a to epons, but auetty tliatty
Tmedisied u t tat tte
outrut Om te i
moid newAOm is mutt
STmoottet tan tte stp tunttonal
output hom eceptron Tese
ane many unutiond
witt tLe tlasautuisti o an "s" shaped cuve
Lnown as gmoid funtticnd The mett
Lommony uued funttion s
he logistic funttion

IteCwh+L)

wi meid leunen
epvegentation (togis tic funetin
le no Lermges see a Ahaap traition at te tautsld b The output
veal vaue
tOTn te zmotd munen is not o dI Jutead ,lt is a

between o-1 olich tan be tuteaprebed a a poabii

feed -fonwaxd lewal Netuoovls

e Tmoduu ase lalled feed fo wad be cale uomation flD uws

taoug tte funttion belmg evauated e n n thoug TLe

Ite media te tomputationu wed to define f and finally to tLe
output y Tee ane no eedbatb Lenneution m wil ou tsuu
e model ale -fed bath t o isksel} e n teedfomo and MeLal
me turovu ane entend ed to inilude feedbath Lennetns, tL ale
talled veuAeot mual netiworlL
A fe etowand nual metwort it a
bioloqally iwivnd
tlaui filation 'algeithm Pt tonict a umbu ot imple neunan-
t e roceuing urit or
anited 'n layess. tvesy unit m layu a

COnnected wit atl unitu mthe preuious

laye atA enteu at tle
lut and paises taoug, tLe netwo r, laya y la e , untiL t
hives at tle output
Duaina momal speaation, a t is wen it
aut as a laui hex, tleae I n -fe
codb atl blw layeus TLi i
wy tey ae calle d heed fono an d metuuua
ne luDrlu

output
Hu dden output
iput lau Lae Laye
tt foTo asd Neual ae tuo o
Pep Lntation po we Feed fonwasd Nnal ne tworle:
Lepvuentatien powe is rulatid to alility ot nunal nelwo A to atim a

p e taelu to aAtiu lau 1utante and ueate wel deh ined attuuhate
a

deithon youndakies for ttat dau

al degins wit MP
Neuen wlithis vey
euon hi t veay imAle idea insified m odel ef a

ohummation eh a l pu btig
AAtan tawleld or
a

otleawise 1t doesn't
given fun ttion tte mau
a

i1gess
ht 2D
a
anuluis witl iuputr -for
amninma ting on funetien anc
tLen mo
viug onto 3D
anaui i& 3
iupul
o fun dti m
71t n2= 2ai2
==l
Co C Art a2tag =0
Co) C)

CIy0,0) A
(o10) Cluo) een) CrO
Foy Senalaum îma 20- 13
Coordinatn stm aUne For &epau atim 'm 3p toovdimale
Auird. Meunom will 6re ytem Manea
resuired .N euLen
tor pint iqhts to tte line will Are dor all
point aove tLe
plane
Mene, MP hleuhon tarn be wed to
vepeent any boel ean fun1tion
wih 1s Uwe ary leaaale *o we can See
r a stit demaseatin
Tle wste ad e a adual deùnen
bounday wit anutG ug
stiqtt ekove tle depaatinq beundaya 1 and
just Le louo auo .
 Prollems 2
mplement AMD ftunction uling MeCult ot h -ittu neuOn (tah e
oway olata)
Conuides tle tnith table o ANO unition
A2

Milul lotl-pit nuum, only analuis is

asume tle
eiug pfomed tene,
wei altu be Wj=l aad w = .
uith Uue aumed
wuquu, te met put îs t alulabed tov fou
iuput For upu
C) yu= m wt1 w, = 1Xl+ 1x1 =
Cuo)
Co) i = (Wjt niuw, =
OXlt1Xl = 1

w
For an AUD funution, te output îr igu if Lotl le input axe hi
For tiit londition, tte net
iuput ft taluulated at 2 . tleneerloated
en
Hi net fupul.te tlres eld is set te, f the
taeleld value ie
aAeates ttan oi eual to then e meuOn fives, else it doeun't
e(o te taold value is let
aluo Le o&lained
eual to 2 .lo- 2) TL tan
y
nw
e n = 2 w=l lenitatoy wigutu) and =olno inLi Liton
ucigtt)
Th,e output of ntuuL On Y15 flyiu) =f
wlue '2' pmstn t tLe tLatalstd value
)1 yu 2
2
 22
find the weigt requied to
puto m the
fellowt tlati fication
uwiua tenLeptuon newovt Tle vettors Cl l
t1) and (-t,t, 1,-1)
ase etomiug to Le das CAo hawe taaget value 1) ,veetos
Ct -1) and C1Ht) ase mot Leloming to tte dau l o Laue
tanget value i). -Asunme leauniug rate ai
and iuitial wiglE

The tnut, talle for the iven

veitor iven au
put
Target (t)

tw=W2 ws =Wy L*o and

=
=

te leasniug rate
tiehetd o o 2, D tle =
=! Sinue tte
autrvation tunition is
f #>0*2
1£in£0 2
if -0.2
yle net iuput is
iven
d te weiztt Uang ane

w Cae) uwj Uslo)+dt xi

=

wee y fCyiu) 6 Cn ew) Llold) tat

=

e wi(nuw)= wilola)
blnw) = blotd
and A= t

TLe baiwing p&ome and te

weigtuts axe
 W AW2 Awa AuWu Ab wi w W3 wy

- 1
O

- - -2 2O
CRo l-2
O 1 3
- - 3 O -

3
4 2 02
- 2 2 O 2
O

Eroul-3
2 O O 0O - 2 O2 0
-t 2 O 2 0 2 0
-2
- 2 O2
- I - O 2 2 0 2C

Thus lu tie tlid epoch all Le

laluLate d outp uty Le LoTne
Aang et and e nauo Las equal to
lnwenqed.

w22

W y 2

Ale tioot andiiutune