agisHe Keaession

classification paobtenm'

datoset:
Study bous Play hOtUMS o/PCPass/ Fail)

2 Fail

3 fail
3 PasS
OutlieS 1 Pass

we Cannot pexkocm eg&ession ,we need to-

into Pass (
pexko& classtiktcalion Foil

Output>1,but wehave o6 h6eshold

tuwo cohditons only yso.S =O

Best fit Line

.5 -- Best fit Line

6 8 10 12 13 14
Fail (Butin Seality itis fass)
ohy we use loqisic Reg&cssion when we can
Solve alassipicaios psoblem using
Linea Regaession

and
due to outliers best it line gets change
6esults wi be wsong

we cannot &emove outies always.

Theeshold
n togistic
wil
can't
'we squash
not
be changed, once
"c
change the line.
he line. e
aed.

Squash("cut")
best tline using sigmoid
AnctHon.

he(ox)=Go+61

Sigmold tunckion output wi

6ange betn 0 and 1
So, even if a perSon Study to67hrs
yos5
he model wi shooFa
n we Can
e cannot ased ineO Sing
Kegsession in 4his Ype og
Psoblem statement
change and

we connot &emovNE OutHies alwaYS.

Theeshold can't be changed, once aed
we squash ("cus +he lin wbe
nogistic
winot change the line

best ktlinc
Squash "cut")(
using Sigmoid

o uncHon.

hex)=6o+610
Sigmod kuncion > output wil
&ange betn 0 and l.
Pamofo Funchion:

Sigmoid kuncon beto&t

Ceate abest itlime
2 Squashing >
Sigmoid unchon

nea Kegsession Costkuncion

SCeo,6)
heco
MSE
heox)=eo+01
GH&adient desent
Conveo kunction
one qlobal minima)

LogistC Reg&essioncos4 umelion

Steps:
Cfeaie best itline
apply squashing3 using siqmoid unclion
 S(e0,e1) = he o)
sigmoid uncion
he(o) s(e0+e1)
et tihaicniMAaABot
Aet Oo+e10
heC)= (R)
hec)
be()
1+ ot81)

be()=
e0+610()

but, aket applying sigmoid junction, cost

kunction coilbetom non-conven unction
and bavc a chancc5 +o gct loca minima

e No convea
unction
e
change theCost unchon to solve thè convexity
p&oblenm

Lg loss Cost Funclion

1oq he(x)
cost uncion

conveacunclion
he (a) =E

+ ®o+61)

cost(heox) -ylog(he(a)- (1-)log(-be

'0, cos+ (he(o) )

This will nevC, aiNe local

Cost (he(o),
minima

1og(be))

)1og(1-ha(a))
valuc
reath
 minimige cost uncion (€6,e1)
by changtmg
e0, 61

Convetgence Algosithm

Repeat Convegence
J

3(eo, G)

o O andjal
take hscshold =0:5
by dekault
using ROC ond ToC CUxv€, we can
deine+h&eshold.
 xkesmance Meeics

Conusion matviot
2 Acc acy

3. Pecision
Reca
5. P-Beta Scoe

datase*

Olp P-y(nodelpfedictE

Conkusion mali Aclal valuke

co8Eect
Oweong
peediction
pfediction
3 2
pfedic+ed O
value (G)
 Actua value
TP Tue Posittve
TP FP Co6sect
TN: TEue Megakve
match
TO
AOL us&ong FP: False Positive

P&edicted valuAe match F: false NegaiHve

Accwxacy

reis
AccwKacy TP+TN
TP+ PP +FN+T

acCwe acy 3t1

3+2+1+1
S7,
dataset Binary classillccatton

lcoo datapointHs

oo Imbalanced daaset
(o)

dumb model 9we get 0 accwaCy

is 307.,it is not sukktcierst

the accwracy
odel is not good To OVeCome this psoblem
e Can use Kecali and Paecision
 Psecision

Peecision TP
Actual TP+ PP

Outok all the actual values

howmany akecossecHy
edieF T pedicted.

O aim is to seduce False fositve (FP)

mail spam o6 ham

TP: actuas spam and psedicted spam

FN actua 6to0 d)
spam and p&edicted>hoxm
(notgcod)
Peedicted
FP: actual> haxm and aetua
Cmailis not spam)
Spam
ceiHcal p&oblem)
on
ocus Seducing FP.

Diabe4es o ot diabees
Actual diabeles and Psedicied- Not
diabetes
Ceittca p8oblem Seduce PN
 Recall
Oat o 'ai 4hepfedicted values
Recall TPTP+FO
howmany aURE cwKAentY
Psedicted.

Tommo6ow the stock market ts going to

C&ash
TwopointS ConsuAmtrS FN

Companies RP
(can Aake cextain
decision)

Actua fot FPcompaies cautake aclion

Te company aeHon ?Sale shar es

and discounted p<ice)
o
pedicted
TN
Consume

-Beta Scose

F-Beta Scose (1+B2) Psecision xRecall

2.Psecision
+ Recall
O FP and FN oke both impostant

PI Scose =2.PxR
P+R

PP is mose impostant
B 0:5
han P1o

f-O:5 scose =(1+0.25) PxR

O:25xP+8

3 1 F >>FP, (FPis
B2
less impo stant 4han Fru),

F2scose = (+4)PxR
4x P+R
In-depth Insights
Logistic Regression
 papergrid
Date:

Clasalea
Classikicaion_psoblem, it just /kethe
eOhleM ECept hat the values
a
ake on oni small numbef o e
Segsessionadel
nooUOant topdedict
discsele Nalues

yEO,1 O Ve class
Binaey class 1 +Ve elagS
Hypohesis Kepseserstatio

iu iively
ages %an o
,
it doesh't omake seDselo8
0
smalles thao obeo we
be(ox)totakeNalues

To jioxbis le's changethe fosm kooaohat yéO1

tosatisky {osOus hypo¥besis he)
0sbe(oc)
Tbis accomplished
uncion by_plugging r
e' ftohe logistic

Ou Aew os uses 46e sigmoro Runchoo

alsocalled
he "Logis¥cPuncion".

be (o) g(eox)
ga)
funetioo q*), shocmD besc,
mapsanydea oumbee8 tothbe
0,1) iotesval, making
it usekul o6
4eansfosmingan
Obita6y-Nalued tunetioninto uncion a betteašuited
{o classiticatio

be(o0 awill giveusthe

p&obability*hatou6 output isl,
OuYput:1
thusbe()= 0.3
 1-PCy=olot;e)
Date.
I
papergrid

beCo) PCye1|xe)=
PCyolxie) +PCy»|lx;0)

Oecisioo Rounda&y

do ocde8Ho ge ous disceete O08classiyieG¥ON, We

HSonslae4he outori o hc hypožhesiS uneonds(o11ocas.
COS

be(ox) O:5
beot)o: 0
Tbe way ousogistickuocior g
behaves is %at uben ds
inpu is gseavefhan of equal to 3e€0,its outpt is
9EcateS hao o6 equal 40 05

>o:5 when2>0
Remembes:
0,eIg /2 Cor)=

o, e 0g()
g(o)0
SoioEinput 40 is e heo +hat means

heo) 9(®') 05 uheo 9 0

Tbe decision
afea usbe
se
bypobesis uncion
o
boundas tsthe line
ql + +hot.sepe.sates
CSeated by ou&
*e
 papergrid
Date:

Yi 5+-)X+0.270
5- O
153
o his Cgse Ou6 decision boundoafy tsa stsaiqbt
yectical ine_placcd on thegsapbwhes =8 add
evesyh{ng *o he lef q{ that denotes wbie
evefyhing fothe sigbt depotes q0.

again,he înpit to
eed o 6esiomoid functioo
6e tineaf, could be atunctioo
g)
ot
Ceg.ex
a
desesibes
doesot

eiccle(e.q eo+ 6:02+ 2o) o&any shape ojt

data
ouE

Cost RuncHon

wecanootuSe hesame cost {unctHoohat weuSe o

lineas segsessioo because he
Logistic Rinction uii
oause be outpkidobe wavy coausing niang local
opma
loothee wosds, Hoilloof be acorveo uncdion

Cost uoction lo8 logisCjunction:

S(O) m i1 cost(he ( y)
cost (he(oxt),u)=-Hog (be(od)iya
cost (he (oc)) -loq-he(0 ity=o

ngto) heta)
 papergrid
Date:

=0
cost (beCo)1y) ba)E
ye0 and beo)->1
cos+(beo)1)0,i
COS (he(atd,y) ie and be(c)->o

ous cosSect answe 'y is0 hen 6eC0S uocon wi

6e 0i ou6 bypothesis_uncion Y
also oukouis 0. Ou
ypohesis appfoaches ,hcohe cost{uncion wi
Qppsoac6 ioginitg

ou coSRec answG& 'yisI,+hen the cost{uncion

pinbe o ous
i bypohesis {anckion opeist. {ous
bypethesis appcoaches 0,hen keGost unctioo coi
aposoach intinit

Nove that oeii0g the cost uncHoo to his way qua&atee

hat SC)is Coove Jo logishc_kAalion. Segkess io0

iro pllged (oetunclion and G7sadientOescent

we caD comp&eSS OuS GOSt KunCHion's woconditional caseS

oto o0e case

Cost(he(x)ig) =-y loq(he(o))- (1-y):log(-he(x)

Thus, uwheo we substitute yal io abv eg,

a cos+(he(o)) ylog (ho(o0)_
similafiy, weaet anožhes Tesm_Cwhen ye0.

'wEcanweideouSeotE CoSt uocioD as

olocOS
 papergrid
Date:

SCe)-1 tog(hetod)) +(1-i)-leg1-haloz)

A vectosiged implementationis

SCO)= logb-Clu log(l-

Gadient Desccot:

Giencea {osm o gadient deseentis

RepeatR

we cOaO WOSkout he deSivaivG pa€ using calculusto

et

Repeat
\.o
halo)-yi
mi1
2
Notice
tnlioea
updaie
Segsessio0:
al
e
hat hisalgos ithm is idenical to he one awe used
St have do simulta0eously
valuesin the1o. he)
isdittesent

A Yectosized inmplemenaton is:

e - x(a(xe)-

Advanceod Oplimalfon

waot mina Se).

CoStunc¥ion S(O):
e,wehave c0dehot can compute
Gaiven
Se)
S(8) tasjao
 egrid
Date:
I
Optiomigawoo olgosithnms:

GSadient desceo
Goojugae gsadient
RGS
LBRGIS.

odvartages O No neced to manuatly pick.ibyt

(2oden jasies ¥han gsadientdescen
dis adv" mofe comple doimplemen*

Mollclasg_elassiLcaions

Nocw wewill appsoacb #he class'caioo ofoata usbeo cwehave

MoEC6ao tw0 Co¥eg0&ieS
Insteao of y=Co,1 wc wiil eopand ousde{iOHoo sohat

Sioce y=lo,1, Jwedivideoufosoblem tnto

he index stasts at0) binasy_ elassikicaion otl (+ 6cz

600c, wepfeoi of 16epobabeliHy thatpe oblems;in

'y' isa
raemehc membec ogone Of 0uS clasSeS.

be) P(y:olgx;)
ha(o)= PCyzilox;e

he'(ox)=_PCyzolx;6
pSediction ma«ha (ox)
wea&c basically
lumpiog a he
one class andhen
choosi0g
othess into o single 5econdclass, uwe dothis &epeated
 papergrid
luio9 6i0as Date: I

logisic
usehe egeessiontoeach
bypothesis
Asedietiog.
hat case,aod *6us
ekusned thie btghest
NalueaSOu

ne-Vs-alone-vs-eal)

AA-

o
-
Class
class2
class 3:x
:
A

heloc)=P(y=ilx;e) Ci=2,3)

To summasige
Tsain a logistic segsession class ikiee helo«)jo8 each

classile& to psedict the psobaly hat qei

a
OD new input ox,to make a p$ediokion piok the class

hat m acimizes
mao be()
 papergrid
Date:
II

KequlaiQatM

De+A

(undeettHngo8 (accukade) Covexftt)

highbias) Chiahyaxiance)

wC bavevoo maoyeatureS,hen easOEd bypohesis

may it Saining set vesy wet but ail to geeesale
gencsalige ooco eoamples

Tbese afetwo main oplioosto

add&ESS heissue Oj
oveSiiog

Reduccheoumbec
of_{eauces:
manuglly selec¥ wbich {eotues
do keep
use a
model selechon
algoeitbro

2. Requlacization
keep al thejeaukes
O
,butseduce
pasa meieks 6
heoag oitude
Regulaization wasks wellgobeg
lot o slighkly usekul uebavea
ceseukce.LeakueeS.

oe 6oveoverHA0g
{om
can seduce the uweiqht ou,byupohesis
that someok he fungkon1
eams inOudG
 papergrid
Date: I

Lundtio cosoy byincseasing

heis cost
Egwe
wgoted fo make he
kollowing unoton mose
9uadKaNio:

doithout elinminale
actually
geHing eio o fhese featues
changing *beofm ofcut
oue cOSt unchon. bypdthesis, C oe
COn iosfeod modig

mine
2m 2
2 (hao)-yi +1000 e+looo.e
Now io08de& {08be cost_LuncHo0 toget
closett0 2ek0
we oil bave to eoluce be valueS O O3 and
toneot
This uwil in dun gseatlySeduce the
in
ReYO
N
oue bupothesis ungio0
Nalues okSZA

As a&esut,we see that oenbypOesis

(aeapk)
fo0ks ke a
quadeako kunclonbutiHs he data beHe duetoeodea
Small idems eg0s anod 4«t,

i/
6i3e House ,Sige qf House
0o+1x+62t

Suppose ue penalige aod make 3,84 eally small.

mi6 1 (ba(i)-qi*+
2
10vo2 lovoet

pink akaph showg Hhis complele eaualion:2

 papergrid
Date:

ta parom.
all Oou heYa paamele
CouCould alsoSegulaeige
i a single sunmaions aS

mio belori)-yi+2J
e 2 il

The 2 oslamb da,is be Regulaei2otHO0 paranmele.

deYeemioes bow much 4he coss o Ow heta paramele

UsingheabovecostAncioo with becocSa Summgion

we CGO smoo¥h heoutpet o oue bypohesis {uocionso
eduee oveSLHOg

lambdo t
is choosen to be too laxge, may smooth out
he luockoo doo much and cause undeAiing.

equlauRed mea ÅeqkesSien

Gsadieot Descent
we wil1 g8adient descent {uocion to sepogYe
moodify
out Go_16om he
Sesf O hc_poramcYers because do we
o0t wan to AeOalize eo.

Repeat
o Go-X. 2 (baoci)-qi.

j ba(od)-i\

iESh2,s
 papergrid
1
Date:

fbe dekm eper{osmS oU seguloeigatoo oith some

S
sanipulaton OUG update sule ca0also be sepsesenled

mi
aluays willbeless ¥haD

otutiely we can seetas Eeducingthealueo

someamount OD eveSy update
Gb b
se2nd fexm is exoacHy sameas it wa befose

Nosmal Bquokoni

To add thecquaHon ishe same

fegulaxizaion,
in
as
, we adod anotbe&dexm
inside
ou,oiginal eocept hat
he paanheses
xTx+2LXy
whese L=
aluays
O 3/O

1
down the
Lisam aeio with O atthe topleftt and ps
evesywhcse else
diagonal, with O's

I XP*).
shold hove dimension(nt)
intuitivelu ,
this is

ideoYity mo¥tiot (hough ayeaor nof

the
theèade
dentity A.
muplied
*no), ih a sinoe eealno.
ineleding

Howevel,when we add
Imx, 4hen X 'Xis non-OVeSH6le
hen X'X+*L 6ecomes invesihle.
the deem A:L
 papergrid
Date:

ion
Keqularigeod oqistte Keqkess

CoStuno¥ioo
GOS uOCion {OSlogishc
Segsession GS
laq(1-8bec
SO)=2i.loq(ha(ox))+(q).

G Ca0 Segulorige this cquadion 6yaddiagaderm

Ho he eod

3Ce) -L logba (oxi)) (I).l0g(1-ba(oxi

20 je

meansto emplieitly caxclude hebiasesm,

Go. i.C. heevectos isindereo8om oto n (holdling Ot
values Bo though en)
god his Sum eotplicitly skips Go by Sunning Som
Ido D Skipping
Thus, when compukog he cquaion,uwe should
coninuously updaiehsuofolousing equgkions

Grsadieot Ocscent

Sepeat

o eo-o:
M, (baC)-yi).

- i alg

ei heCcxi)-i)a+

ste) (12