Professional Documents
Culture Documents
Feature Embedding Clustering Using POCS-based Clustering Algorithm
Feature Embedding Clustering Using POCS-based Clustering Algorithm
Abstract: An application of the POCS-based clustering algorithm (POCS stands for Projection Onto Convex Set), a novel
clustering technique, for feature embedding clustering problems is proposed in this paper. The POCS-based clustering
algorithm applies the POCS’s convergence property to clustering problems and has shown competitive performance when
compared with that of other classical clustering schemes in terms of clustering error and execution speed. Specifically,
the POCS-based clustering algorithm treats each data point as a convex set and applies a parallel projection operation
from every cluster prototype to corresponding data members in order to minimize the objective function and update the
prototypes. The experimental results on the synthetic embedding datasets extracted from the 5 Celebrity Faces and MNIST
datasets show that the POCS-based clustering algorithm can perform with favorable results when compared with those
of other classical clustering schemes such as the K-Means and Fuzzy C-Means algorithms in feature embedding clustering
problems.
2.1 Convex Set where 𝑃𝑐𝑖 is the projection of 𝑥𝑘 onto convex set 𝑐𝑖 and
𝑤𝑖 is the weight of importance of the projection. Note
Convex set has been one of the most classical and that 𝑥𝑘 represents the 𝑘 𝑡ℎ projection of the initial point
powerful tools in the optimization theory [10]. A set of 𝑥0 . The projection continues until convergence or when
data points is called a convex set if it has the following the terminating condition is satisfied. The main
property: given a non-empty set 𝐴 which is the subset of advantages of the parallel mode of the POCS method
a Hilbert space 𝐻, 𝐴 ⊆ 𝐻 is called convex, for ∀𝑥1 , 𝑥2 ∈ include the computational efficiency and the improved
𝐴 and ∀λ ∈ [0, 1], if the following holds true: execution time.
𝑥 ≔ λ𝑥1 + (1 − λ)𝑥2 ∈ 𝐴 (1) When the sets are disjoint, the parallel form of the POCS
method converges to a point that minimizes the weighted
Note that if λ = 1, x = 𝑥1 , and if λ = 0, x = 𝑥2 . For any sum of the squares of distances to the sets, which can be
value of 0 ≤ λ ≤ 1 and x ∈ 𝐴, x lies on the line segment expressed as:
joining 𝑥1 and 𝑥2 when the set is convex.
𝑛
2
2.2 Projection onto Convex Set 𝑑 = ∑ 𝑤𝑖 ‖𝑥 ∗ − 𝑃𝑐𝑖 (𝑦 ∗ )‖ (5)
𝑖=1
The concept of the projection of a point to a plane is
utilized to solve many optimization problems such as where 𝑥 ∗ is the convergence point such that the distance
finding a point on the plane that has the minimum 𝑑 defined by (5) is minimized. A graphical illustration of
distance from the center of projection. For a given point the convergence of the parallel POCS method is
x ∉ 𝐴, the projection of x onto 𝐴 is the unique point y ∈ presented in Fig. 2.
𝐴 such that the distance between x and y is minimum. If
x ∈ 𝐴, the projection of x onto 𝐴 is x. The optimization 2.4 The POCS-based Clustering Algorithm
task can be expressed as:
For disjoint convex sets, the parallel mode of the POCS
𝑦 = 𝑎𝑟𝑔𝑚𝑖𝑛‖𝑥 − 𝑦 ∗ ‖2 (2) method converges to a point that minimizes the weighted
sum of the squared distances. This property has been
where 𝑦 ∗ is all the points on the convex set 𝐴 . A applied to clustering problems and the POCS-based
graphical illustration of the projection of a point onto a clustering algorithm has been proposed in our
convex set is shown in Fig. 1. preliminary work [11]. The POCS-based clustering
algorithm considers each data point as a convex set and
2.3 Parallel Projection onto Convex Sets all data points in the cluster as disjoint convex sets. The
objective function of the algorithm is defined as:
In the parallel mode of the POCS method, a point is
𝐶 𝑛
projected to all convex sets concurrently. All the 2
projections are combined convexly with corresponding 𝐽 = 𝑎𝑟𝑔𝑚𝑖𝑛 ∑ ∑ 𝑤𝑖 ‖𝑥𝑗 − 𝑃𝑐𝑖 (𝑥𝑗 )‖ (6)
weight values to solve the minimization problem. For a 𝑗 𝑖=1
‖𝑥𝑗 − 𝑑𝑖 ‖
𝑤𝑖 = 𝑛
(7)
∑𝑝=1‖𝑥𝑗 − 𝑑𝑝 ‖
with a constraint:
𝑛
∑ 𝑤𝑖 = 1 (8)
𝑖=1
K-Means++ POCS-based
Face 111.2±0.3 111.3±0.7
MNIST 6,804.6±12.8 6,836.9±27.6
K-Means++ POCS-based
Face 4.3±0.6 4.1±0.9
MNIST 916.9±301.8 771.2±269.6
Table 3. Comparison between the POCS-based Table 5. Comparison of the K-Means, FCM, and POCS-
clustering and the K-Means++ algorithms in terms of based clustering algorithms in terms of execution time
classification accuracy on different feature embedding (ms) on different feature embedding sets.
sets.
K-Means FCM POCS-based
K-Means++ POCS-based
Face 11.8±4.1 45.3±6.2 4.5±2.3
Face 99.36±1.7 99.53±1.5
MNIST 282.1±30.8 1,484.9±40.2 615.9±243.7
MNIST 64.6±2.2 63.9±1.8