EE657: PRML Jan-Apr 2019

Practice Problems

1. Show that mH (2N ) ≤ mH (N )2 , where mH (N ) is the maximum number of

dichotomies on N points.

2. Let H = {h1 , h2 , ...hM } with some finite M . Show that dvc (H) ≤ log2 M .

3. The monotonically increasing hypothesis set is

H = {h | x1 ≥ x2 =⇒ h(x1 ) ≥ h(x2 )},

where x1 ≥ x2 if and only if the inequality is satisfied for every component.

(a) Give an example of a monotonic classifier in two dimensions, clearly

showing the +1 and −1 regions.

(b) Compute mH (N) and hence the VC dimension. [Hint: Consider a set of N
points generated by first choosing one point, and then generating the next
point by increasing the first component and decreasing the second compo-
nent until N points are obtained.]

4. In this problem, we will consider Ω = R. That is, x is a one dimensional vari-

able. For a hypothesis set,
( D
!)
X
H = hc | hc (x) = sign ci x i ,
i=0

prove that the VC dimension of H is exactly (D+1) by showing that

(a) There are (D+1) points which are shattered by H.

(b) There are no (D+2) points which are shattered by H.

5. Consider a data set with two data points x± ∈ Rd having class ±1 respectively.
Compute the optimal (maximum margin) hyperplane (b? , w? ) and its margin.
 6. Consider a data set with three data points in R2 :
   
0 0 −1
X= 0 −1  , Y =  −1 
−2 0 +1
Compute the optimal hyperplane (b? , w? ) and its margin.
7. Data matrix containing four data points in R2 and target values, together with
separability constraints are given below. The inequality on a particular row is
the separability constraint for the corresponding data point in that row.
   
0 0 −1 −b ≥1
 2 2   −1  −(2w1 + 2w2 + b) ≥ 1
X=  Y = 
 2 0   +1  2w1 + b ≥1
3 0 +1 3w1 + b ≥1
Compute the optimal hyperplane (b? , w? ) and its margin.

8. Solve dual optimization problem for the data given in problem 7. Use the
following steps.
(a) Show that the dual optimization problem is to minimize
L(α) = 4α22 + 2α32 + 92 α42 − 4α2 α3 − 6α2 α4 + 6α3 α4 − α1 − α2 − α3 − α4
subject to the constraints
α1 + α2 = α3 + α4 ;
α1 , α2 , α3 , α4 ≥ 0
(b) Use the equality constraint to replace α1 in L(α) to get
L(α) = 4α22 + 2α32 + 29 α42 − 4α2 α3 − 6α2 α4 + 6α3 α4 − 2α3 − 2α4
(c) Fix α3 , α4 ≥ 0 and minimize L(α) in (b) with respect to α2 to show that
α3 3α4 α3 α4
α2 = + and α1 = α3 + α4 − α2 = +
2 4 2 4
Are these valid solutions for α1 and α2 ?
(d) Use the expressions in (c) to reduce the problem to minimizing
L(α) = α32 + 94 α42 + 3α3 α4 − 2α3 − 2α4
subject to α3 , α4 ≥ 0. Show that the minimum is attained when α3 = 1
and α4 = 0. What are α1 , α2 ?
9. Let Ω = Rd . Show that K(u, θ) = uT θ satisfies Mercer’s conditions.
10. Let Ω = Rd . Let f : Ω → R be a real valued function. Show that
K(u, θ) = f (u)f (θ) is a positive definite kernel.