Probability ond rondomn

Ver ia ble

P.sgl: clessi fied as de ter

romdo
AAondom Cngredckable Conna t
Vaue at bime
Predic& éhe vawe
certutmy
consid eI
A.ia(t) is gene ruked ftom an
Oscrllakor Wikh Poa ftequency
stubilry. outpH is random
let enoltiple people
-speec Speck same senten ce
n mcend We are captUring
Wavo form wave forn Wile be
echifferent, bU& shke Gog le
pekon
Cem Un derstumd 6
bce of,
bz robabiliy and
ndom variable
 Math tosle availa ble for
rumdom Phenomenoy
) Bandon veria ble
ex Dies Expermen
tukes
20
each
40

Fx (9)
Fx C15) = PéX<Is) -

Non Decreasi n1
 GOUssiam bn
Ray dom Varylne
fx (x)

reul
Lide

PDF

CG)is difderey, halion

Uniform Distnbukio

eVerg has
patince .
 E(X-m)
e-rxe ) ebe relefed,
Unco
Unohre ney
PK-8;
Mak'y: =
Cx) represent
on
ean
other
Matriy -n
X) Var (X
Cov(X,X) CoYKN, Z qre
BVsO. he
)coVdndmce
N.BVs Cov(X,,x)=
E are
ECX) is
X2)
then
COV theyno
Cov(X,
E
 RVs
conside fha gon does
requis (xe,
X)
greser zero
stronger tracked
C, hni'te
Cx
=
qre
Vélup
mVerse
fUnCion
relofion
(Foy X2)=cov
correlati'on, eigen =o
dot-Produc
bs
ofod i's
ce sUbs de es kors alwe
den is i'es
Cov(,
somi Valw vector,
Vec mmakrix
exis
distribuh'on
demce cor fric
()Symme C
indepen eam aon
/ Eigen
aieign
dqpenPropes
qmakrix
m bcz Posihive
Ej then
mon- Sealerle
Va
in ’id
() (i) (iv)
F
'y
 iinea
ly
indepevcme identea,
eigen museecl to
ortho9
oa
Diangralizafiompattixare
Nabyix is to
Need
tneorer?
limik
Cenfrel RVs
ITD and
Know
Vecko}
pondlet) :
which
this
indeponde
Distribte
valve
eijen UNxN Valwe boz
Vec&or tavtd
Veckors
re xist Cinde
egen
ANxN
= e orthogoal cnder
eiaen
always TID:
other.
U S& To
) )
 pckure
pixels
of
soB face And
ferke 3
fuking Same
: kbis
ot We kribußed
dis
nderstund friends
ghoka. ce valves),
hab C(recan
255
Verian phobo
are
say
Lefls
scy
We pixel care know cam
Know 0o) pixel
We and
3 egch (csing also e CIre take
NowAne
letis (300 say mecn
alD |S are NowVarble
3 SUm6
Now"We
bo
Pho kvee
be
Phokos phoko
Now Letis
we
Dther,
Phobos
cbou.b khe forstk SO
 S s

all
mean

observa biom
Cim d
be
So if Wo fulke more Rs
d Variamce Wil! be
bggey and
Problewn biger.
libe.
lets ora lirod id

TIP has sume istibvbon

which cn Oniform or Gass/ag
or ony ofher, bUt will
be
Galssic DstribU
ton with
’ this is called cedel Aioit
fheorern,
 Con Vex opkiizutian
xI) ) kinizabion is necessy
MI tecknfues : e
1

Dinear
regression ogsthc Iegre ssion,
SVM to
Jenerl
fenckionith nknow
difrlt
But fos khe class t prstlems
ML Coyí Vex fnckio: one
Cam fim d sol with Kess compleriy
conver seß
Seb c is .con Vex
Point

if khis holdstrve then c is

ConVex seb

pormts iy Can dd dty a ne.

selohgs foc.
 Convex Non convex
Cxampls

e). he

(aley:)':
in circal
eg3, Atine subspAc i the set st
to
Aa=b
where 0
eg 4. ntersec ior) oex seks
is also Conve
poikive Slmdefioi te
Matrices
 A fimckiom
Jonaj Conyer sol
demole d
oy¢ DA) 8

ola)4(1- o)f

G-f wi be
Concave
Convex funt kioy have only
One glo bal imt'lm
) Gsb oydey con dito for
Con Vexiy
ine everg polt
his line shod l bo bolow
fine ti'or f

, t)
 cecoid lays,
Somidfn ieFosI
f(a1,
) cero
For
eAm
ple s
hessiqy
varigble.fimckio,
Gso
Con
Vea
ve 2
condhiom Ahen
concave
funnCkioy.
qeßive of
For
posHve
semidefiite
p gyeaker fa)=a?a
= ne
be
mdst
Hessiam
order fn1te.
Semide is
fa)
hessiamdervekhive
one
dsece)
if
id
)
 fa)

CODVex

a) ba tc
b by
) (
f) Hessidn
SO fimction is bokn conves
Conçcve
 t 9perVised learmim9
t ainng phuse
Teshing ghos o
’Apoblerd ot pre dic Ging te
khe
erice o6 falk (Ho me) n city
s0ze (fe) price ( )
2
)40o
|S3
1
bobh
la ble
boßh
sperviso d
qre gVem
 Wriße. Cam we
&he on bdsod Then )
(=1,2
--
byted epesen be Ca Sek
he hatcomsdey Uslet )
hola
limear cs me qss
mode Ghe Rinear selakion
is
tpUt khe prCe
plot)eroblen
o -)
estn sIze2
1mg stefs Two
given
Ukpu date& inpuß Theare
reSsi0r limeat
 hala) .

hoa)
hoCa)

hoCa)
cpproch ko solve
=) one of 8he
aoR o,
this probley formekhod!
Qeast s qUare

# Leust sqcdre
JJ(Go,oi)
1
obyec kive ffn
 20 (=| looto,a)-9]
ey nimhnizing J(o, 0,) )ne get
Gdifseremtiaking J(an On)
equi lizinA
two eq") two Varna ble
dJCoo, O)

Ty taking
tiake Jos, @)
hoCa) Dot O,

y L

ho Co)4
1
 )

3 0)(oelo t
(aslor)(

13
-3
A

(uy

3
:) fen geb S

J(es, e) - 2 iy-xc|| 2

=-LxtCy-xo)

x(y -xo) =
T

7’thi's
1
x'y
forg,
Soloton

Pseudo Inverse
)f s mon sqUare mcbrn'x, We
can tqke fsúedo roVese
 ) Ir genet l we can consi do
ho e)
() +

-) Tn gene ral
a Wewe cdn Wrife

he)

lc)

=)for Xx to be mvertble

fos Breek We Use siogU (or

Valwe decowpgition (sVD)
 :) We solve lhe ohom by
Osig tS

t Solve he sno prohogo

e by sig gtotgblicice
Predicfel error
pProdch
We consides ba2

nere, is khe error bol

Ween hor)
Now, cons der as q remoo
riable
) AssUme E S gussjny disth.
buded wik
vanionce
ne ca wrih
Now
hala))
c)
 are g/ Ven

y als o
We hqve pdf of )
erpe)?

z, to 36

is aloo GaUss jcrE

262

) ITD
every independett ideyhal
dsfrbuted
ft Max inn Um Cikeliheo d
estia kioY of o (MIE )
) proble is formefobod
s follons
-) we he
obye kive fUnch on LCo)
Over a!
(m)
Llo)- p ( .

Joist prob. oÑy )

(2)

problem is o And o
Such hat ik
L(o)
Lco) ’ hughood tonckeo
 haVo qssOnne
-) Now, We
rid wi

dent

to gon P(") is
ndepesdest
maxhmizakion is beKwee
1his

conpd ep has Som e

Con)g Brys om Bhab

ku ke
-) so Bu log (L(o))
 loy (Lco)) ’ Ravge beomes
co, o)

ex
a(9)-d)) 262

6o maxlm) ze
We have Bo

YMIn/ ze

) id we C(SSUe
262 2

,)
CS J(oo0i) is Ls proah
-) We Brg
kry Ro ge R
svch khat probab
of o's
geffing yyy
Com) becomes

Now, we know bhet for

unear regress/OV

MAP Maxi g0Um a po3tereiorg E

(MAP)
esbima ion o o (Vecbot)
-) One can betber esBmake
Prior sno wle dge qbovf
We Use
nkmown(o) slso should
his glo
9ve bo tter f fo the give
dat
 ) This mekho is reterred qs
reglarizioy im machine lea
rjm this lso Cked in avoidib

) foy egression robVemg ikis

relerred as Ridye vegrocsi0

-) for MLECiag likelhosd

es Giyn"), We mag

L(o)=

’ for MAf khe problem iis formle

ked by considering he bjec
bive foncki on

Lco) = P((//,2y ) () ()

- e C o ) . re)
()
 odsp

o)7

-(0)
).
bw d

(o4
X

(Ceo P
)

(o)4.
(*h)d
 To find thct we need
fho knowledoe
AssUme
Ossi) dis tibuted Wif
& are inde peyde
NCO e) &
P(o)
222

Now,
Plo) ,

V21Tz2 2

bcz hey are iderc

 T

92L3) liol|

Sh
 (o)).

'mizeis7 of this yield s MAp

ostimaioy
nen,ce
LCo) =2 m
,)+A|e
2

Now

2
-2x'(y- xo)

x ytxxo
Hon

o( xx+T)= xTy
er
 MAP is beßko fham MLE
becquse for MLE We
bt
cam be Joqssich vorub le

We take
ho'
opolznommial regression
e9. ho Ca) = Oo fo,

Cy
holu
)

# ibe rabive ane Shod fo

get so lwfio
jtidtert descent
 ) nere tne obec k ve fun is
me
Jío)
Me are stumed
sbtuined terufvk
as follow sS

K+|
K)?
K
Jo))
k’is iterkve nUber
starb wiBh assUmim
inihal Purmekers valwe
2 |.e

Note ’ J J must be di ferentable

JCo)
Vealee
We have
1SSUe
 sole kioy
oight aive
ds
ams er Ha6
depends
oiial Valwes

2m

2(y-(oto,
2

ikerbe
Smell
v veloe
then We sag, he algo convérjes
 +1dso))/
do
d(J(0-)) J(0.)4
XS dc) (Oot J(Oots)=
J
ion series
ex lor Tay
eiriabl
)
on only (tor J
(o)
o =
reke
Qearnimgsre Gstep
/
VJo)
where
VJoo
dien kic has fo
 Gst order CPpreximation
JCoots) dJco))
do

J(o) J(ao) 1 dJ (o) co- ao)

do
JCo)o?
JCo) J(as) + dJoo- o)
do
2C2) x(0-2
= 4+ 4(0-2)
4- 0-4

let d

J (o+s) J(o) d dJ
 S-- d

do
# stecpast gadiend descen
d
 Pakjey
Pienig
Ve
nenf malig
bike Vs
dg vscal
fwo dyim9 clussi
Vsed becan ogisic -)
for
fio binarg
regreSS|
N logistic fot
istigck only fake can
Vectoy
foy
dechnique.
suporvisod alsoa is46is ))
regreSSION
logistic
) Remdom Variqble logis tic
regreSSI

follows Bornali distrito

Modol sod here

hala) 9(o) oTa

fos lnar reHessio n

ho C) Qo f@ f -

Oho «) (o os

Uheee) c (s, )
 os) ukes Vale
So ho*) efueeg
so ho()efresey
some khrg.
fhe
(Usna MLL)
ho) So )

fhe
=|- ho (a)
We (m wite
-holx
We corsider he
=) for MLE
CIS
LCO)
(2)
L(o)
troblem is formed
btain hut
(oo, a, a) which L(o)
=) As cll y)
y are depo don f
We (an WYiBe
L(O)

-e)

[ico))

CAo)
) -lo) caled bmag cross
entrog.
 is called oS bmq y
CrosS qs
there re
two di'strib0 fon

true esthrnabed
) let Us obtin he o veehwes
by csimg gradie n desce (e)
=) $o Use GD We have to
d feren fiae

) consder, 9(z) -! hge)

whese
 d (9))
dz
(1+

ge) %z) (I-s))

(halz)).
hea)- hae)
(0co))
Lo):
log/-ho
y) (1-8))
hoa) (-haa)
()
hoas-
ha(o))
 ()

ha)