Professional Documents
Culture Documents
Design of SVM
Design of SVM
May 5, 2022
1 / 20
SVM
w t Xi + w0 > 0
w .Xi + w0 > 0 if Xiw1
w .Xi + w0 < 0 if Xiw2
g (X ) = w .Xi + w0 ≥ b for good generalization
The distance of point ‘X’ from the hyperplane ‘H’ which is
represented as g(X) can be calculated as r = g (x)/||w ||
which is nothing but (W .Xi + w0 )/||w || .
2 / 20
So, we must ensure that
(W .Xi + w0 )/||W || ≥ b ⇒ 1
(W .Xi + w0 ) ≥ b.||W ||
W .Xi + w0 ≥ 1 if Xi ω1
W .Xi + w0 ≤ −1 if Xi ω2 ⇒ 2
3 / 20
From 2, we can establish an uniform criteria as follows Yi is the class label
whose values is ±1
+1 for class ω1
-1 for class ω2
4 / 20
5 / 20
From 1 ,
In order to maximize the margin ‘b’, ||w || has to be minimized and at
the same time wo has to be maximized .
For minimization of ||w || , let’s consider the other constraints.
yi (w .Xi + w0 ) = 1
Because it is constraint optimisation problem, it can be converted into
un-constraint problem by using the lagrangian multiplier.
6 / 20
Minimization of ||W || is same as minimization of φ(W ) -
function of W
φ(W ) = W t .W
φ(w ) = 12 .W .W (dot product)
7 / 20
It can be written using unconstraint optimization problem as follows
Pn
L(||W ||, wo ) = 12 ||W ||2 – i=1 λ[yi (W .Xi + wo )–1]
minimize ||w || .
1 2
2 ||W || is objective function
[yi (W .Xi + wo )–1] is a constraint
8 / 20
We can define the lagrangian of the form,
minimize ||wo || .
maximize ||w || .
αi is Lagrangian multiplier
L(w , wo ) = 12 (w .w )–
P
αi [yi (w .Xi + wo )–1]
9 / 20
L(W , wo ) = 12 (W .W )– P αi [yi (W Xi + w
P
Po )–1]
L(W , wo ) = 21 (W .W )– αi yi (W Xi ) − αi yi wo + αi
P
∂L P
∂wo =− αi yi = 0
∂L Pn
∂wo = i=1 αi yi = 0 ⇐ one of the constraints
10 / 20
L(W , wo ) = 12 (W .W )– αi yi (W .Xi ) − αi yi wo + αi
P P P
∂L P
∂w = W − αi yi Xi = 0
Pn
W = i=1 αi yi Xi
where P
W .Xi = αj yj Xj and αi yi wo ) = 0
11 / 20
1 Pn Pn P
= 2 i=1 αi αj yi yj (Xi .Xj )– i=1 αi αj yi yj (Xi .Xj ) + αi
Pn 1 Pn
L(W , wo ) = i=1 αi − 2 i=1 αi αj yi yj (Xi .Xj )
12 / 20
αi - lagrangian multipliers are always positive
αi ≥ 0
Pn
i=1 αi yi =0
13 / 20
g (z) = W .Z P
+ wo
g (z) = sign( ni=1 αi yi Xi .Z + wo )
14 / 20
The steps of SVM design to estimate W and wo
Pn
W = i=1 αi yi Xi
and wo
wo = 12 [min
P P
∀yi =+1 αi yi (Xi .Xj ) + max ∀yi =−1 αi yi (Xi .Xj )]
wo = 21 [min
P P
∀yi =+1 W .Xi + max ∀yi =−1 W .Xi ]
g (Z ) = W .ZP+ wo
g (Z ) = sign ni=1 αi yi Xi .Z + wo
15 / 20
16 / 20
but in implementation,
w t Xi + wo = 1
w .Xi + wo = 1
wo = 1 − (w .X )
17 / 20
Additional slides
g(x,y) = 0 or g(x,y) = c
L(x, y , λ) = f (x, y )–λg (x, y )
18 / 20
Additional slides
Here
f (x, y ) = f (||w ||, wo ) ⇒ objective function
g (x, y ) = yi (w .X + wo ) = 1 ⇒ constraint
we can define the lagrangian of the form ,
Pn
L(||w ||, wo ) = 12 ||w ||2 – i=1 λ[yi (w .Xi + wo )–1]
minimize ||w || .
maximize ||wo ||
||w ||2 is objective function
[yi (w .Xi + wo )–1] is the constraint
19 / 20
THANK YOU
20 / 20