Formulation of the Primal Problem and Its Transformation to the

Dual in SVM
Diego Canales by ChatGPG

Primal Problem
The primal problem for a Soft Margin Support Vector Machine (SVM) can be formulated as follows:
N
1 X
min ∥w∥2 + C ξi
w,b,ξ 2 i=1
(1)
subject to yi (wT xi + b) ≥ 1 − ξi ,
ξi ≥ 0, ∀i.
Here, w represents the weight vector, b is the bias, and ξi are the slack variables that allow for misclassification
of data points.

Lagrangian Formulation
To transform the primal problem into its dual, we introduce the Lagrangian by adding Lagrange multipliers
for each constraint. We have two sets of constraints, hence two sets of Lagrange multipliers: αi for the
constraint yi (wT xi + b) ≥ 1 − ξi and µi for the constraint ξi ≥ 0.
The complete Lagrangian is given by:
N N N
1 X X X
L(w, b, ξ, α, µ) = ∥w∥2 + C ξi − αi [yi (wT xi + b) − 1 + ξi ] − µi ξi . (2)
2 i=1 i=1 i=1

Deriving the KKT Conditions

The Karush-Kuhn-Tucker (KKT) conditions are necessary for optimality. To find the optimal solution, we
differentiate the Lagrangian with respect to the primal variables (w, b, and ξi ) and set these derivatives to
zero.

Partial Derivatives of the Lagrangian

The partial derivatives of the Lagrangian with respect to the primal variables are:
1. With respect to w:
N
X
∇w L = w − αi yi xi = 0 (3)
i=1
which gives us the relationship:
N
X
w= αi yi xi . (4)
i=1
2. With respect to b:
N
∂L X
=− αi yi = 0 (5)
∂b i=1

1
implying the dual feasibility condition for α:
N
X
αi yi = 0. (6)
i=1

3. With respect to ξi :
∂L
= C − αi − µi = 0 (7)
∂ξi
establishing the relationship between αi , µi , and the penalty parameter C:

αi + µi = C. (8)

The complementary slackness conditions are:

αi (yi (wT xi + b) − 1 + ξi ) = 0 (9)

µi ξi = 0 (10)
αi ≥ 0 (11)
µi ≥ 0. (12)

Substitution into the Lagrangian

We substitute the relationships found for w and ξi back into the Lagrangian. By doing this, we eliminate
the primal variables and
PNconvert the Lagrangian into a function solely of the dual variables α and µ.
Substituting w = i=1 αi yi xi and using the conditions αi + µi = C, the Lagrangian becomes:
N N N N N
1 XX X X X
L(α) = αi αj yi yj xTi xj − αi yi yj xTi xj + αi − µi ξi . (13)
2 i=1 j=1 i=1 i=1 i=1

Since µi ξi = 0 for all i, and considering the dual feasibility and complementary slackness conditions, the
terms involving µi can be omitted, leading to the simplified Lagrangian solely in terms of α.

Dual Problem Formulation

The dual problem is obtained by maximizing the Lagrangian with respect to α, under the constraints derived
from the KKT conditions. The dual problem is formulated as:
N N N
X 1 XX
max L(α) = αi − αi αj yi yj xTi xj (14)
α
i=1
2 i=1 j=1

subject to:
0 ≤ αi ≤ C, ∀i, (15)
N
X
αi yi = 0. (16)
i=1

Maximizing this Lagrangian subject to these constraints leads us to the optimal values of αi , which in
turn allow us to find the optimal w and b for the primal problem. The non-zero αi correspond to the support
vectors.