6772 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL.

30, 2021

Joint Learning of Latent Similarity and Local

Embedding for Multi-View Clustering
Aiping Huang , Member, IEEE, Weiling Chen , Member, IEEE, Tiesong Zhao , Senior Member, IEEE,
and Chang Wen Chen , Fellow, IEEE

Abstract— Spectral clustering has been an attractive topic

in the field of computer vision due to the extensive growth
of applications, such as image segmentation, clustering and
representation. In this problem, the construction of the simi-
larity matrix is a vital element affecting clustering performance.
In this paper, we propose a multi-view joint learning (MVJL)
framework to achieve both a reliable similarity matrix and
a latent low-dimensional embedding. Specifically, the similarity
matrix to be learned is represented as a convex hull of similarity
matrices from different views, where the nuclear norm is imposed
to capture the principal information of multiple views and
improve robustness against noise/outliers. Moreover, an effective
low-dimensional representation is obtained by applying local
embedding on the similarity matrix, which preserves the local
intrinsic structure of data through dimensionality reduction.
With these techniques, we formulate the MVJL as a joint
optimization problem and derive its mathematical solution with
the alternating direction method of multipliers strategy and
the proximal gradient descent method. The solution, which Fig. 1. Application examples of multi-view clustering in computer vision.
consists of a similarity matrix and a low-dimensional rep-
resentation, is ultimately integrated with spectral clustering
or K-means for multi-view clustering. Extensive experimental
results on real-world datasets demonstrate that MVJL achieves perspectives [1]–[3], and these data are called multi-view
superior clustering performance over other state-of-the-art data. Relative to traditional single-view clustering methods,
methods. multi-view ones access more features and structural informa-
Index Terms— Multi-view clustering, joint learning, feature tion hidden in multi-view data, and have been widely used in
representation, local embedding, nuclear norm. computer vision [4], [5]. As illustrated in Fig. 1, a multi-view
clustering method was utilized to estimate pseudo labels of
unlabeled training data for person re-identification [6], [7];
I. I NTRODUCTION
An efficient and robust image co-segmentation algorithm

R ECENTLY, most visual data have multiple modalities or

are represented by heterogeneous features from different
Manuscript received October 8, 2020; revised April 14, 2021; accepted
June 27, 2021. Date of publication July 26, 2021; date of current version
July 30, 2021. This work was supported in part by the National Natural
Science Foundation of China under Grant 61901119 and Grant 62001116 and
in part by the Natural Science Foundation of Fujian Province under
Grant 2019J01222 and Grant 2020J01466. The associate editor coordinating
the review of this manuscript and approving it for publication was Prof.
Kui Jia. (Corresponding author: Tiesong Zhao.)
Aiping Huang and Weiling Chen are with the Fujian Key Laboratory
for Intelligent Processing and Wireless Transmission of Media Information,
College of Physics and Information Engineering, Fuzhou University, Fuzhou
350108, China (e-mail: sxxhap@163.com; weiling.chen@fzu.edu.cn).
Tiesong Zhao is with the Fujian Key Laboratory for Intelligent Processing
and Wireless Transmission of Media Information, College of Physics and
Information Engineering, Fuzhou University, Fuzhou 350108, China, and
also with the Peng Cheng Laboratory, Shenzhen 518055, China (e-mail:
t.zhao@fzu.edu.cn).
Chang Wen Chen was with the Department of Computer Science and
Engineering, University at Buffalo, The State University of New York,
Buffalo, NY 14228 USA. He is now with the Department of Computing,
The Hong Kong Polytechnic University, Hong Kong (e-mail: chencw@
buffalo.edu).
Digital Object Identifier 10.1109/TIP.2021.3096086
1941-0042 © 2021 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See https://www.ieee.org/publications/rights/index.html for more information.
was designed by coupling with multi-view clustering [8];
More accurate image annotation and retrieval have benefited
from multi-view clustering methods [9], [10]. In addition,
a multi-view clustering method based on bipartite match-
ing has been used for video summarization [11]. The past
decade has witnessed a boom in multi-view clustering meth-
ods based on subspaces [12], [13], matrix factorization [14],
multi-kernel [15] and spectra [16].
Among them, multi-view spectral clustering demonstrates
promising performance. It contains two main steps: similarity
matrix construction and clustering implementation. The first
step is of utmost importance because the performance of the
spectral clustering algorithm is highly dependent on the quality
of the similarity matrix. Although existing similarity matrix
construction methods have improved multi-view spectral clus-
tering performance, most of them suffer from the following
two problems. First, the reliability of their similarity matrices
may be impaired by noise and outliers embedded in raw
data; thus, the ultimate clustering performance can be affected.
Second, they may fail to exploit the local intrinsic structure

Authorized licensed use limited to: University of Technology Sydney. Downloaded on November 10,2023 at 12:08:03 UTC from IEEE Xplore. Restrictions apply.
HUANG et al.: JOINT LEARNING OF LATENT SIMILARITY
Fig. 2. Framework for proposed method. We first extract feature matrices of datasets from m views and use them to construct similarity matrices. With the
local embedding technique and nuclear norm constraint, the fused similarity matrix W and latent low-dimensional representation Y are generated by MVJL,
and corresponding two strategies MVJL-S and MVJL-K are designed for the final clustering results.
embedded in similarity matrices, which makes them difficult
to learn an effective feature representation for multi-view
clustering.
In this paper, we propose an MVJL method that integrates
latent similarity learning and local embedding technique into
a unified framework. As demonstrated in Fig. 2, we extract
multi-view features, construct a similarity matrix for each
view, and optimize an objective function to learn a reliable
similarity matrix and an effective low-dimensional represen-
tation. Both the similarity matrix and the low-dimensional
representation can characterize the underlying clustering struc-
ture of multi-view data. By integrating them with spectral
clustering or K-means, two clustering strategies, MVJL-S
and MVJL-K, are designed for the final results. The major
contributions of this paper are summarized as follows:
• The fused similarity matrix based on nuclear norm. The
similarity matrix is iteratively updated until convergence.
With the nuclear norm constraint, the MVJL model can
capture the principle information of multiple views to
enhance its robustness to noise.
• Local embedding for low-dimensional representation. By
searching an effective low-dimensional representation,
the MVJL model leverages the local similarity infor-
mation of samples to reveal the underlying clustering
structure.
• The MVJL-S and MVJL-K strategies for multi-view clus-
tering. The MVJL model learns a latent similarity matrix
and a low-dimensional embedding to further design two
multi-view clustering strategies: MVJL-S and MVJL-K.
Experimental results validate their superior performance
over other popular methods.
The rest of this paper is arranged as follows. Section II
reviews related works to this paper. The proposed frame-
work and corresponding iterative optimization algorithm are
developed in Section III. Section IV presents the experimental
results and analysis. Conclusions are drawn in Section V.
Authorized licensed use limited to: University of Technology Sydney. Downloaded on November 10,2023 at 12:08:03 UTC from IEEE Xplore. Restrictions apply.
6773
II. R ELATED W ORK
Thanks to ubiquitous multi-view data, multi-view cluster-
ing [17]–[20] has been extensively investigated during the past
decade. It improves clustering performance by capturing the
rich information of multiple cues. Many existing multi-view
clustering methods are derived from spectral clustering due
to their potential performance and well-defined mathematical
framework. According to the construction of the similarity
matrix, the existing spectral-based multi-view clustering meth-
ods can be mainly divided into two categories.
The first type employs tools such as Gaussian kernels to
construct similarity matrices of different views, and then fuses
them to learn a shared similarity matrix of all views. For
example, early work [21] proposed a co-training multi-view
spectral clustering method in which the eigenvectors from one
view are adopted as predictor functions to update the graph of
the other view. In [22], the similarities of eigenvectors learned
from different views were further enhanced by enforcing these
eigenvectors towards a specific common consensus. Recent
work [23] minimized disagreement between different views
and constrained the rank of the Laplacian matrix to learn a
consensus similarity graph. [24] integrated graph completion
and consensus representation learning into a joint framework
for incomplete multi-view clustering.
The second type is self-representation-based subspace learn-
ing methods, which utilize source data as a dictionary to learn
a self-representation matrix for the subsequent construction
of similarity matrix. For example, [25] proposed a latent
multi-view subspace clustering model that jointly learns an
underlying representation and performs data reconstruction
within a unified framework. This method was further general-
ized in [26] by combining it with a deep neural network. [27]
adopted the Hilbert Schmidt Independence Criterion (HSIC)
and intact space learning to construct an intactness-aware
similarity matrix. [15] automatically assigned ideal weight for
each view to learn similarity information without additional
6774
parameters. [28] proposed a unified model to jointly learn the
kernel representation tensor and affinity matrix.
Although the above similarity matrix construction methods
have improved clustering performance, most of them suffer
from the impact of noise and outliers embedded in raw
data, or ignore the local manifold structure of the similarity
matrix, which affects further improvement of the ultimate
performance. To this end, an increasing number of multi-view
spectral clustering approaches coupled with manifold learning
have been proposed. For example, [29] presented a non-
negative matrix factorization framework, where consensus
manifold regularization was incorporated to maintain the local
geometrical structure of all views. [30] utilized adaptive neigh-
borhoods to perform multi-view clustering/classification and
local manifold structure learning simultaneously. Inspired by
this, [31] employed Laplacian embedding on a pre-learned
data representation to preserve the local manifold structure for
obtaining a shared similarity matrix for spectral clustering.
Recently, low-rank matrix learning has been widely stud-
ied [32]–[34] and benefits many multi-view learning methods.
For example, [35] utilized a low-rank decomposition to learn a
shared consensus similarity matrix. [36] performed symmetric
matrix factorization with a low-rank constraint to characterize
the data underlying clustering structure. In [37], the low-rank
property was encoded by the Tucker decomposition to capture
high-order correlations among views. In addition, low-rank
matrix learning can also be employed for noise/outlier process-
ing. For example, [38] developed a cross-view low-rank
analysis model for outlier detection. In [39], a low-rank
constraint was incorporated into marginalized denoising to
boost the model robustness of its proposed model. [13] utilized
a low-rank constraint to guarantee robustness to outliers and
sample-specific corruptions.
In this work, we integrate low-rank constraint and manifold
learning into a joint learning model for a reliable similarity
matrix and a latent low-dimensional representation. Different
from the above low-rank-constraint-based methods, we employ
the nuclear norm as a low-rank constraint to learn a reliable
similarity matrix, so as to capture the principal information
from multi-view data and enhance its robustness against
noise/outliers. The proposed model also differs from the exist-
ing spectral-based multi-view clustering methods in the way
to addresses the local structure embedded in multi-view data.
We employ local embedding on the shared similarity matrix
to learn a low-dimensional representation, thereby excavating
the local manifold structure embedded in multi-view data
for performance improvement. Both the similarity matrix and
the low-dimensional representation characterize the cluster-
ing structure of multi-view data. Therefore, two strategies,
MVJL-S and MVJL-K, are designed for the final cluster-
ing results by integrating them with spectral clustering or
K-means. To the best of our knowledge, MVJL is an early
attempt for multi-view learning that can produce two different
clustering strategies by joint learning.
III. M ULTI -V IEW J OINT L EARNING FOR C LUSTERING
Real-world datasets contain various representations in mul-
tiple views. On one hand, similarity matrix can be regarded as
Authorized licensed use limited to: University of Technology Sydney. Downloaded on November 10,2023 at 12:08:03 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 30, 2021
TABLE I
S UMMARY OF THE K EY N OTATIONS U SED IN THE PAPER
a representation of data, and constructing a reliable similarity
matrix is beneficial for multi-view clustering. On the other
hand, the local structure information in sample space is more
capable of reflecting the relationships among samples. Towards
this end, we employed low-rank constraint and manifold learn-
ing to formulate an MVJL framework for a reliable similarity
matrix and a latent low-dimensional embedding. The details of
the framework are elaborated in this section. Table I presents
key notations used in the paper.
A. Problem Formulation
For a given multi-view dataset {X(i) }m i=1 with m views and
X(i) ∈ Rdi ×n . MVJL aims to capture a shared low-dimensional
embedding Y = [y1 , · · · , yn ] ∈ Rk×n with k  di (i =
1, 2, · · · , m) by preserving sufficient local structure informa-
tion of samples, and simultaneously learn a fused similarity
matrix W = [Wi j ]n×n for characterizing the similarities
among samples, which can be abstracted in the following
canonical form:
 2
1  
m
 
min W − αi W(i)  + γ W∗
α,W,Y 2  
i=1 F
β 
n n
+ yi − y j 22 Wi j
4
j =1 i=1
T
s.t. α ≥ 0, α 1 = 1, (Y) = 0 (1)
where W(i) ∈ Rn×n is the similarity matrix obtained by X(i)
by the Gaussian kernel, and α = [α1 , · · · , αm ]T is a weighted
vector to balance the significance of different views’ similarity
matrices. (Y) = 0 is a certain constraint imposed on Y, and
β, γ are two nonnegative hyperparameters.
In Equation (1), the first term ensures learning a shared
similarity matrix for different views with a minimal fitting
error. By incorporating the idea of principal component analy-
sis, the second term adopts the nuclear norm to enforce the
shared similarity matrix to be of low-rank. This not only
facilitates learning a compact form to compress the important
information into a small proportion of components, but also
benefits to improve robustness to noise and outliers. The last
term aims to preserve sufficient local structure information of
HUANG et al.: JOINT LEARNING OF LATENT SIMILARITY 6775

samples to learn a latent low-dimensional representation. The An intermediate variable θ is introduced to separate Z and α,
motivation behind this term is to preserve the inter-sample and Equation (5) can be represented as the form of
similarity of original data after projection to a low-dimensional 1
subspace. Therefore, the similarity matrix W is embedded min vec(W) − Zα2F s.t. α = θ, α ≥ 0, α T 1 = 1. (6)
α,θ 2
as a penalty factor for the sample pairs violating this rule.
The weighted vector α can be regarded as a regularization This is equivalent to minimizing the following ADMM
parameter for the multi-objective optimization problem, where problem:
α ≥ 0 and α T 1 = 1. 1
As we know Lρ (α, θ, y) = vec(W) − Zθ 2F + y T (α − θ )
2

n 
n ρ
+ α − θ 2F s.t. α ≥ 0, α T 1 = 1, (7)
yi − y j 22 Wi j 2
i=1 j =1 which can be solved by the iterative updating rules:
 n  n
= (yiT yi − 2yiT y j + yTj y j )Wi j α (t +1) = arg min Lρ (α, θ (t ) , y(t ) ), (8)
α
i=1 j =1
θ (t +1) = arg min Lρ (α (t +1) , θ, y(t ) ), (9)
= 2T r (YDYT ) − 2T r (YWYT ) = 2T r (YLW YT ) (2) θ
y(t +1) = y(t ) + ρ(α (t +1) − θ (t +1)), (10)
where LW = D − W is the Laplacian matrix and D

n where t denotes the number of iterations.
is a diagonal matrix with Dii = Wi j . To ensure the
j =1
Eliminating constant terms of Lρ (α, θ, y), the optimization
existence of the regularized solution for minimizing Equa- problem with respect to α is written as
tion (2), the constraint YDYT = I is imposed on Y to ρ
remove an arbitrary scaling factor in the embedding. Denote min yT (α − θ ) + α − θ 2F s.t. α ≥ 0, α T 1 = 1. (11)
α 2

n
di ×n
By specific algebra formulation, it is further simplified as
X= [X(1); X(2); · · · ∈R ; X(m)]
. The low-dimensional
i=1
 2
representation Y can be represented by a linear projection ρ y

n
min α−θ +  s.t. α ≥ 0, α T 1 = 1, (12)
di ×k α 2 ρ F
P ∈ Ri=1 with Y = P T X. Consequently, the MVJL
problem is further represented as which can be solved with a closed-form solution by the method
 2 proposed in [40].
1  
m
 (i)  β The optimization problem of updating θ is formulated as
min W − αi W  + T r (P T XLW XT P)
α,W,P 2   2 1 ρ
i=1 F min vec(W) − Zθ 2F + y T (α − θ ) + α − θ 2F . (13)
θ 2 2
+ γ W∗ s.t. α ≥ 0, α T 1 = 1, P T XDXT P = I. (3)
This problem is unconstrained and the optimal solution is
∂ L (α,θ,y)
attained at ρ ∂θ = 0. Taking the derivative of Lρ (α, θ, y)
B. Alternating Optimization Algorithm
with respect to θ , we have
According to the objective function in Equation (3),
∂Lρ (α, θ, y)
we simultaneously seek a latent low-dimensional represen- = −ZT vec(W)+ZT Zθ −y + ρ(θ − α). (14)
tation and a fused similarity matrix for multi-view data. ∂θ
Since this function is not jointly convex for all the variables, ∂ Lρ (α,θ,y)
Set ∂θ = 0. Then, the optimal solution is
we develop an efficient alternating optimization algorithm that
separates the multi-variable optimization problem into several θ = (ZT Z + ρI)−1 (ZT vec(W) + y + ρα). (15)
solvable subproblems.
2) Updating P With Fixed α and W: The subproblem of
1) Updating α With Fixed W and P: The subproblem of
updating P aims to solve the minimization problem
updating α is equivalent to
 2 arg min T r (P T XLW XT P) s.t. P T XDXT P = I. (16)
1  
m P
 (i) 
min W − αi W  s.t. α ≥ 0, α T 1 = 1. (4)
α 2  Using the Lagrangian multiplier method, λ ∈ Rk×k is
i=1 F
introduced to construct a Lagrangian function as
Denote vec(W) ∈ Rnn×1 as the vectorization of W. Then
the subproblem above is equivalently transformed as L(P, λ) = T r (P T XLW XT P) − T r (λT (P T XDXT P − I)).
(17)
1
min vec(W) − Zα2F s.t. α ≥ 0, α T 1 = 1, (5) As we know, its optimal value is attained at ∂ L∂P
(P,λ)
= 0
α 2
∂ L(P,λ)
and ∂λ = 0. Thus, we obtain
where Z = [vec(W(1) ), . . . , vec(W(m) )] ∈ Rnn×m .
We adopt the alternating direction method of multipli- ∂L(P, λ)
= 2XLW XT P − XDXT P(λ + λT ) = 0 (18)
ers (ADMM) strategy to settle the above optimization problem. ∂P

Authorized licensed use limited to: University of Technology Sydney. Downloaded on November 10,2023 at 12:08:03 UTC from IEEE Xplore. Restrictions apply.
6776 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 30, 2021

and Algorithm 1 Algorithm for MVJL

∂L(P, λ)
= P T XDXT P − I = 0. (19)
∂λ
Based on Karush-Kuhn-Tucher (KKT) conditions, we know
that λ = [λi j ]k×k is a nonnegative diagonal matrix. Therefore,
XLW XT P = XDXT Pλ (20)
implies that for any column pi (1 ≤ i ≤ k) of P,
XLW XT pi = λii XDXT pi (21)
holds. This illustrates that pi is a generalized eigenvector of
matrices XLW XT and XDXT , and λii is the corresponding
generalized eigenvalue.
3) Updating W With Fixed α and P: Directly solving W is
a tough task, as it is restricted by a non-differentiable nuclear
norm. To this end, the proximal gradient descent method
is employed to update W. Denote the objective function
J (W) = f (W) + g(W), where a differentiable component
 2
1  
m
  β
f (W) = W − αi W(i)  + T r (P T XLW XT P)
2   2
i=1 F
and a non-differentiable component g(W) = γ ||W|||∗. The
proximal gradient descent method reads the iterative rule as
1
W(t +1) = Proxg (W(t ) − ∇ f (W(t ))),
1/L
(22)
L
where L is a constant. As we know, Proxg (W(t ) −
1/L
(t ) The complexity of computing P is O(d 3 + nd 2 ), where
L ∇ f (W )) is equal to the value of
1
m
 2 d = di . Because the number m of views is much lower
1  (t ) 1 
(t )  γ
arg min W − (W − ∇ f (W )) + ||W||∗ , (23) i=1
than d and n, the overall computational cost of MVJL is
W 2 L F L
O(n 3 + nd 2 + d 3 ).
which has a closed form solution, read as
W(t +1) = U γ /L V
T
, (24) IV. E XPERIMENTS
where γ /L is a soft thresholding, i.e., γ /L = max In this section, comprehensive experiments on real-world
{0, − γL I}. Herein, U VT is a singular value decomposition datasets are conducted to evaluate the performance of the
of W(t ) − L1 ∇ f (W(t ) ) with proposed methods and comparison algorithms.

m
β
[∇ f (W(t ))]i j = [W(t ) − αi W(i) ]i j + ||yi −y j ||22 . (25) A. Test Datasets
4
i=1
Eight publicly available multi-view benchmarks are
Thus, all variables have been updated. employed to provide a fair testing platform for all compared
Summarizing the aforementioned analysis of optimal solu- methods.
tions to all subproblems, the algorithm for MVJL is presented ALOI1 is a collection of color images with small objects
in Algorithm 1. Once the optimal W and Y are available, that were taken at various viewing angles, illumination direc-
the subsequent clustering strategies MVJL-S and MVJL-K, tions, illumination colors, and object orientations. For each
which respectively integrate with spectral clustering or image, RGB color histograms, HSV color histograms, color
K-means, are conducted for the final results. similarity and Haralick texture features are extracted as four
It is difficult to generally prove the convergence for our views.
algorithm. Fortunately, most sub-optimization problems have Caltech1012 is a database containing 101 categories of
closed-form solutions in each iteration, and empirical evidence images. Following [41], twenty classes totaling 2,386 samples
on real-world data suggests that the proposed algorithm has are selected to evaluate the performance of our algorithm. Six
stable convergence behavior, as shown in Section IV-F2. different features are extracted as views: 48-D Gabor features,
The computational complexity of the proposed MVJL 40-D wavelet moments features, 254-D CENTRIST features,
method relies on those of updating optimization vari-
ables α, W and P. To update α, the complexity is O(mn 2 ). 1 http://aloi.science.uva.nl/
The W-subproblem leads to a complexity of O(n 3 ). 2 http://www.vision.caltech.edu/Image Datasets/Caltech101/

Authorized licensed use limited to: University of Technology Sydney. Downloaded on November 10,2023 at 12:08:03 UTC from IEEE Xplore. Restrictions apply.
HUANG et al.: JOINT LEARNING OF LATENT SIMILARITY 6777

TABLE II
A B RIEF C HARACTERIZATION OF A LL T ESTED D ATA S ETS

Fig. 3. Sample images from ALOI, Caltech101, NUS-WIDE and MSRC-v1 datasets.

1984-D HOG features, 512-D GIST features, and 928-D local
binary pattern (LBP) features.
HW3 consists of handwritten numerals from 0 to 9 digit
classes with a total of 2,000 patterns. In the dataset, six public
features are available. They are 76-D Fourier coefficients of
the character shapes (FOU), 216-D profile correlations (FAC),
64-D Karhunen-love coefficients (KAR), 240-D pixel averages
in 2 × 3 windows (PIX), 47-D Zernike moment (ZER) and
6-D morphological (MOR) features.
MNIST4 is a handwritten digit collection from which
10,000 samples are selected for testing, along with three views
of features produced by IsoProjection with 30 dimensions, lin-
ear discriminant analysis with 9 dimensions and neighborhood
preserving embedding with 30 dimensions.
NUS-WIDE5 is a web image dataset. We select a subset
containing 1,600 images of 8 categories in our experiments.
Six available features for each image are provided: 64-D
color histogram, 144-D color correlogram, 75-D edge direction
histogram, 128-D wavelet texture, 225-D blockwise color
moments, and 500-D bag-of-words.
MSRC-v16 dataset contains 210 images in 7 classes com-
posed of trees, buildings, airplanes, cows, faces, cars and
bicycles. We extract five visual features from each image:
3 http://archive.ics.uci.edu/ml/datasets/Multiple+Features
4 http://yann.lecun.com/exdb/mnist/
5 https://lms.comp.nus.edu.sg/research/NUS-WIDE.htm
6 https://www.microsoft.com/en-us/research/project/image-understanding/
CMT with 24 dimensions, HOG with 576 dimensions, GIST
with 512 dimensions, CENTRIST with 254 dimensions, and
LBP with 256 dimensions.
ORL7 is a face image dataset containing 40 distinct sub-
jects. Each subject has 10 different face images taken under
different lighting conditions, facial expressions and facial
details (such as glasses or no glasses).
YouTube8 is a video dataset, from which 2,000 samples are
selected. Each sample is described by six views consisting of
audio features (mfcc, volume stream, and spectrogram stream)
and visual features (cuboid histogram, hist motion estimate and
hog features).
The important statistics of the aforementioned datasets are
summarized in Table II and several sample images from the
ALOI, Caltech101, NUS-WIDE and MSRC-v1 datasets are
shown in Fig. 3.
B. Comparison Algorithms and Parameter Settings
To evaluate the effectiveness of the proposed algorithms,
we compare them with the following seven state-of-the-art
clustering approaches.
K-Means is a classical clustering algorithm that serves
as a benchmark of various unsupervised learning methods.
7 http://www.uk.research.att.com/facedatabase.html
8 http://archive.ics.uci.edu/ml/datasets/YouTube+Multiview+Video+
Games+Dataset

Authorized licensed use limited to: University of Technology Sydney. Downloaded on November 10,2023 at 12:08:03 UTC from IEEE Xplore. Restrictions apply.
6778
It is inclined to cluster data into spherical distributions and
is sensitive to initial values.
MVC [42] proposed two kinds of unsupervised learning
schemes (multi-view EM algorithm and agglomerative algo-
rithm) for text data with a co-training idea, and proved the
superiority against their single-view counterparts in the context
of web categorization.
MVCC [43] attempted to explore a latent representation
matrix of each view and then used them to derive a shared
consensus representation for all views. This goal was reached
through concept decomposition with a local manifold structure
regularization.
SwMC [44] is a totally self-weighted multi-view clustering
method. It leveraged multiple graphs to assign a reasonable
weight for each view according to the importance of the view
and directly achieved final clustering results without further
postprocessing.
MvDMF [45] is a deep matrix factorization-based
multi-view clustering model. It introduced graph regularizers
to guide the consensus representation learning in the final layer
such that most shared geometric structures across multiple
graphs could be preserved.
MLAN [30] is a multi-view learning model with adaptive
neighbors. Implementing multi-view learning in tandem with
local structure embedding, the researchers obtained an opti-
mal graph with a certain number of connected components
corresponding to exact clustering assignments.
MVKSC [46] unified two or more kernel spectral cluster-
ing (KSC) models into a joint framework. Meanwhile, a cou-
pling term was added to exploit complementary information
of multiple views to improve the clustering performance.
MSC-IAS [27] first utilized multi-view information to learn
an intact space, constructed the intactness-aware similarity
matrix in the space by HSIC, and employed spectral clus-
tering on the obtained similarity matrix to perform the final
clustering.
MvSCN [47] encapsulated deep learning for multi-view
spectral clustering, where SiameseNet and QR decomposition
were respectively employed to design embedding networks
and an orthogonal layer.
Several algorithmic parameters must be set in advance.
For the proposed MVJL method, the embedded dimension
number is set as the number of clusters, the regularized
parameter β is taken as 1 and the weight coefficient γ for local
embedding is fixed as 0.1. Regarding all multi-view clustering
algorithms for comparison, we implement them through the
source codes provided by the authors and adopt their default
settings if feasible. For MVCC, we set the regularization
parameter α = 100 and two trade-off coefficients β = 100
and γ = 10. For SwMC, we set the penalty coefficient λ for
the sum of the c smallest eigenvalues to 1. Regarding MVKSC,
the regularization parameters and kernel parameters are set as
1 and 0.1, respectively. For MLAN, the number of adaptive
neighbors and the maximal number of iterations are fixed as
9 and 30, respectively. Regarding MvDMF, the parameter γ
is set as 0.5, β is fixed to 0.01 and the number of nearest
neighbors k is taken as 5. For MSC-IAS, its parameters
are fixed as follows: λ2 = 0.1, dimension of intact space
Authorized licensed use limited to: University of Technology Sydney. Downloaded on November 10,2023 at 12:08:03 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 30, 2021
d = 500 and nearest neighbor number k = 3. For MvSCN,
the balance parameter λ and neighborhood size k are fixed as 1
and 14, respectively. In addition to the specific parameters, all
comparison algorithms need to know the number of clusters
in advance, whereas MVC and K-means do not face the
parameter selection issue. Moreover, since MVC can deal with
only two views, we take the first two views of each test
dataset as input. For K-means, we concatenate feature vectors
of different views together for the all-view clustering setting.
C. Evaluation Metrics
The clustering performance is evaluated by comparing the
obtained cluster labels with the ground-truth. Four well-known
metrics including clustering accuracy (ACC) [48], normalized
mutual information (NMI) [49], purity (Purity), and adjusted
rand index (ARI) [50] are employed for quantitative com-
parison. These four metrics favor different properties, which
facilitates the comprehensive evaluation of clustering results.
For each metric, a higher value is preferable to indicate better
performance. For each experiment, the mean value along
with the standard deviation are recorded by repeating each
algorithm 30 times with random initializations.
D. Experimental Results
In this subsection, we compare the proposed clustering
strategies MVJL-K and MVJL-S with seven other algorithms
in terms of the aforementioned four widely used metrics.
The detailed clustering results on eight datasets are reported
in Tables III and IV, where the numbers in the parentheses
are the standard deviation. It should be noted that running
MvSCN on MSRC-v1 and ORL will encounter matrix inver-
sion errors. Therefore, we report only the performance of ten
other methods on these two datasets. For ease of comparison,
we highlight the best results in red boldface and the second
best in blue boldface. From the two tables, the following
observations can be made:
First, multi-view methods are evidently more promising than
the single-view baseline, as evidenced by the fact that the
maximum performance improvement provided by multi-view
algorithms can be achieved 54.3%. This suggests that utilizing
various multi-view clustering models to explore the hidden
complementary information contained in multi-view data can
effectively improve clustering performance.
Second, the compared algorithms show respective merits in
various scenarios. MVKSC does not perform well on most
datasets, which may be caused by its requirement of different
parameter settings in different datasets. However, for a fair
comparison, we set all parameters as a uniform fixed value in
this work because of the addition of new test datasets. MVC is
designed especially for document clustering and different EM
applications in the model differ in specific tasks. For image
clustering, MVC can also achieve acceptable performance.
MVCC provides a flexible fusion strategy. Its clustering
performance is competitive with MVC. MSC-IAS is a sub-
space clustering model that can avoid information loss caused
by insufficient views. By recovering an intact space from
HUANG et al.: JOINT LEARNING OF LATENT SIMILARITY 6779

TABLE III
A C OMPARISON OF ACC AND NMI OF D IFFERENT M ULTI -V IEW C LUSTERING A LGORITHMS . T HE B EST R ESULTS A RE H IGHLIGHTED IN R ED B OLDFACE
AND THE S ECOND B EST A RE M ARKED W ITH B LUE B OLDFACE . (H IGHER M EANS B ETTER )

TABLE IV
A C OMPARISON OF P URITY AND ARI OF D IFFERENT M ULTI -V IEW C LUSTERING A LGORITHMS . T HE B EST R ESULTS A RE H IGHLIGHTED IN R ED
B OLDFACE AND THE S ECOND B EST A RE M ARKED W ITH B LUE B OLDFACE . (H IGHER M EANS B ETTER )

multi-view data, MSC-IAS improves the clustering perfor- the weight for each view without additional parameters.
mance compared with MVC and MVCC. MLAN and SwMC SwMC achieves good performance on Caltech101, HW and
are parameter-free methods. They can automatically allocate MSRC-v1. However, its performance degrades severely on

Authorized licensed use limited to: University of Technology Sydney. Downloaded on November 10,2023 at 12:08:03 UTC from IEEE Xplore. Restrictions apply.
6780
Fig. 4. Experimental visualizations of the learned similarity matrix in the four sampled datasets.
ALOI, NUS-WIDE and YouTube. In general, MLAN performs
promisingly on most datasets but is not as robust as SwMC.
MvDMF and MvSCN are deep-learning-based methods. Due
to the lack of large numbers of well-defined training data,
these methods cannot embody their highlighted advantages in
multi-view clustering on the test datasets.
Third, the two proposed clustering strategies MVJL-K and
MVJL-S have unique characteristics. For example, the ARI
performance improvement of MVJL-S over MVJL-K can
achieve up to 7.4% on HW, while MVJL-K improves ACC
performance by 6.7% compared with MVJL-S on Caltech101.
This demonstrates the respective merits of the low-rank struc-
ture and low-dimensional representation in different scenarios.
Certainly, there are some cases in which our methods do
not perform well. For example, the images in NUS-WIDE
cannot be well clustered by MVJL-S. This may be attributed
to the challenge of the dataset or the extracted features
being unfavorable to the estimation of similarity matrices.
Fortunately, the employment of local embedding can well
preserve the points’ similarity relationship and maximize the
smoothness with respect to the intrinsic manifold structure
of the dataset in low embedding subspace. This can alle-
viate the unreliability of similarity estimation to an extent,
which promotes MVJL-K to achieve the best performance
on NUS-WIDE. Overall, the proposed MVJL-S and MVJL-K
outperform the compared algorithms in terms of performance
and robustness, which indicates that MVJL is a promis-
ing method for multi-view clustering. The superiority of
MVJL is that the multi-view clustering performance can be
effectively improved by utilizing the interactions of learning
an optimal low-rank structure, assigning a reasonable view
weight and finding a low-dimensional representation in a joint
framework.
E. Visualization Analysis
Fig. 4 presents the visualizations of the fused similarity
matrix on four datasets. Herein, we randomly select a subset
containing 2,000 samples from MNIST for better visualization.
Evidently, more values in yellow colors in the diagonal blocks
means a better similarity matrix. In this sense, the similarity
matrices learned on HW, MNIST and MSRC-v1 outperform
those learned on NUS-WIDE. The possible reasons that the
outline of the similarity matrix is not obvious on NUS-WIDE
lie in the challenge of the dataset itself or the extracted features
that are unfavorable to the estimation of similarity matrices.
Authorized licensed use limited to: University of Technology Sydney. Downloaded on November 10,2023 at 12:08:03 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 30, 2021
Fig. 5 presents an intuitive scatter diagram to illustrate
the clustering performance of all algorithms on a subset
of MNIST with 2,000 samples. First, we stack the feature
vectors of 3 views together, and employ t-distributed stochastic
neighbor embedding (t-SNE) [51] to project the original
high-dimensional data onto a 2-D space. Then, the mapped
2-D data are colored with the cluster labels obtained by
different methods and ground truth. Evidently, a better per-
formance should be closer to the ground truth. In this sense,
the visualization effects coincide with the clustering results
shown in Tables III and IV. The proposed methods can
effectively group similar samples into the same clusters,
which further validates the effectiveness and superiority of our
model.
F. Model Discussions
1) Runtime Analysis: In Table V, we present the running
time of the state-of-the-art methods on all test datasets.
Due to the matrix inversion exceptions of running MvSCN
on MSRC-v1 and ORL, the running times of MvSCN
on these two datasets are not recorded. From the table,
the single-view method (i.e., K-means) often performs more
efficiently than multi-view methods. The reason is clear:
K-means simply concatenates feature vectors of different
views together for all-view clustering. By contrast, multi-view
clustering methods need to address the multi-view data
fusion problem. Among the multi-view methods, MVC is
almost the fastest, while MvDMF consumes the longest
time on most datasets. Our MVJL-K and MVJL-S perform
stably and exhibit a good running speed compared with
the competitors. Considering its superior clustering perfor-
mance, its time consumption is acceptable for real-world
applications.
2) Convergence Analysis: To validate the convergence prop-
erty of our method, we compute the objective function values
on four test datasets, respectively. The corresponding curves
are presented in Fig. 6. From the figure, the objective function
values decrease with increasing iteration number and then
achieve convergence. Although our method requires tens of
iterations to converge, its computational complexity is not
high due to its simple model instead of a deep network. This
suggests that the proposed algorithm has stable convergence
behavior.
Fig. 7 shows the evolution of view weights on four test
datasets. At the beginning, we treat each view equally and
HUANG et al.: JOINT LEARNING OF LATENT SIMILARITY 6781

Fig. 5. Experimental visualizations of different multi-view clustering methods on a subset of MNIST with 2,000 samples.

TABLE V
A C OMPARISON OF T IME C ONSUMPTION FOR A LL A LGORITHMS ( IN S ECONDS )

Fig. 6. Convergence curves of MVJL on four test datasets.

initialize αi = m1 , where αi is the i -th view weight and m is 3) Parameter Sensitivity: To investigate the variations of
the number of views. From Fig. 7, the weight curves rapidly MVJL under different settings, parameter sensitivity analysis
converge after a limited number of iterations, which also is conducted in this subsection, as shown in Figs. 8 and 9.
indicates the stable convergence of the proposed algorithm. The accuracy variation curves of MVJL-K and MVJL-S are

Authorized licensed use limited to: University of Technology Sydney. Downloaded on November 10,2023 at 12:08:03 UTC from IEEE Xplore. Restrictions apply.
6782 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 30, 2021

Fig. 7. The evolution of view weights on four test datasets.

Fig. 8. Performance of MVJL with varied regularization coefficient β ranging from {1 × 10−5 , 1 × 10−4 , · · · , 1 × 104 }.

Fig. 9. Performance of MVJL with varied regularization coefficient γ ranging from {1 × 10−5 , 1 × 10−4 , · · · , 1 × 102 }.

marked with the solid and dotted lines, respectively. In MVJL, the optimal parameters on each test dataset, and then select
there are two parameters β and γ to be tuned. In the parameter an appropriate uniform value for each parameter to execute
sensitivity check, we employ a grid searching strategy to find multi-view clustering on all datasets.

Authorized licensed use limited to: University of Technology Sydney. Downloaded on November 10,2023 at 12:08:03 UTC from IEEE Xplore. Restrictions apply.
HUANG et al.: JOINT LEARNING OF LATENT SIMILARITY
Fig. 8 explicitly shows the performance variation curves
on eight datasets, where the regularized parameter β ranges
from {1 × 10−5 , 1 × 10−4 , · · · , 1 × 104 }. The best clustering
results of MVJL are attained when β ∈ [10−5 , 1] in most
cases. Specifically, MVJL achieves acceptable performance
on almost all datasets when β is 1. By contrast, the perfor-
mance of MVJL-K decreases sharply when β is 10, which
occurs because this thresholding value leads to inaccurate
generalized eigenvalues and their corresponding eigenvectors.
Furthermore, MVJL-S performs more robustly than MVJL-K.
The potential reason may be that the K-Means method is very
sensitive to initialization values. Overall, both MVJL-K and
MVJL-S achieve promising performance over a wide range.
The parametric sensitivity of MVJL with respect to γ is
analyzed in Fig. 9, where the weight coefficient γ for local
embedding ranges from {1×10−5 , 1×10−4 , · · · , 1×102 }. It is
worth mentioning that a larger γ may lead to learning a trivial
low-rank representation solution. On the tested datasets, this
phenomenon occurs when γ exceeds 1 × 103 . Nevertheless,
both MVJL-K and MVJL-S are stable as the parameter γ
differs in the selected wide range of values, which also
suggests the robustness of the proposed method.
V. C ONCLUSION
In this paper, a principled joint learning method named
MVJL was proposed for multi-view clustering. In MVJL,
a similarity matrix and a low-dimensional representation
shared by all views were jointly learned. This can effectively
enhance the robustness to noise and outliers as well as facil-
itate investigations into the underlying clustering structure of
multi-view data. Moreover, an effective alternating optimiza-
tion with two clustering strategies was proposed to ensure
a high-quality clustering performance. Extensive experiments
performed on eight datasets verified the clear superiority of
MVJL. Our model is able to flexibly investigate the comple-
mentarity among multiple views for clustering.
R EFERENCES
[1] G. Chao and S. Sun, “Semi-supervised multi-view maximum entropy
discrimination with expectation Laplacian regularization,” Inf. Fusion,
vol. 45, pp. 296–306, Jan. 2019.
[2] P. Zhu, Q. Hu, Q. Hu, C. Zhang, and Z. Feng, “Multi-
view label embedding,” Pattern Recognit., vol. 84, pp. 126–135,
Dec. 2018.
[3] S. Yang, L. Li, S. Wang, W. Zhang, Q. Huang, and Q. Tian, “Skele-
tonNet: A hybrid network with a skeleton-embedding process for multi-
view image representation learning,” IEEE Trans. Multimedia, vol. 21,
no. 11, pp. 2916–2929, Nov. 2019.
[4] F. Fang, L. Li, H. Zhu, and J.-H. Lim, “Combining faster R-CNN and
model-driven clustering for elongated object detection,” IEEE Trans.
Image Process., vol. 29, pp. 2052–2065, 2020.
[5] L. Yang, C. Shen, Q. Hu, L. Jing, and Y. Li, “Adaptive sample-level
graph combination for partial multiview clustering,” IEEE Trans. Image
Process., vol. 29, pp. 2780–2794, 2020.
[6] X. Xin, J. Wang, R. Xie, S. Zhou, W. Huang, and N. Zheng, “Semi-
supervised person re-identification using multi-view clustering,” Pattern
Recognit., vol. 88, pp. 285–297, Apr. 2019.
[7] W. Li, X. Zhu, and S. Gong, “Person re-identification by deep joint
learning of multi-loss classification,” in Proc. 26th Int. Joint Conf. Artif.
Intell., Aug. 2017, pp. 2194–2200.
[8] Z. Tao, H. Liu, H. Fu, and Y. Fu, “Multi-view saliency-guided clustering
for image cosegmentation,” IEEE Trans. Image Process., vol. 28, no. 9,
pp. 4634–4645, Sep. 2019.
Authorized licensed use limited to: University of Technology Sydney. Downloaded on November 10,2023 at 12:08:03 UTC from IEEE Xplore. Restrictions apply.
6783
[9] Y. Gong, Q. Ke, M. Isard, and S. Lazebnik, “A multi-view embedding
space for modeling internet images, tags, and their semantics,” Int. J.
Comput. Vis., vol. 106, no. 2, pp. 210–233, 2014.
[10] M. Chi, P. Zhang, Y. Zhao, R. Feng, and X. Xue, “Web image retrieval
reranking with multi-view clustering,” in Proc. 18th Int. Conf. World
Wide Web (WWW), 2009, pp. 1189–1190.
[11] Y. Fu, Y. Guo, Y. Zhu, F. Liu, C. Song, and Z.-H. Zhou, “Multi-
view video summarization,” IEEE Trans. Multimedia, vol. 12, no. 7,
pp. 717–729, Nov. 2010.
[12] Z. Kang et al., “Partition level multiview subspace clustering,” Neural
Netw., vol. 122, pp. 279–288, Feb. 2020.
[13] W. Zhu, J. Lu, and J. Zhou, “Structured general and specific multi-view
subspace clustering,” Pattern Recognit., vol. 93, pp. 392–403, Sep. 2019.
[14] Y. Yang, F. Shen, Z. Huang, H. T. Shen, and X. Li, “Discrete nonnegative
spectral clustering,” IEEE Trans. Knowl. Data Eng., vol. 29, no. 9,
pp. 1834–1845, Sep. 2017.
[15] S. Huang, Z. Kang, I. W. Tsang, and Z. Xu, “Auto-weighted multi-view
clustering via kernelized graph learning,” Pattern Recognit., vol. 88,
pp. 174–184, Apr. 2019.
[16] Z. Hu, F. Nie, R. Wang, and X. Li, “Multi-view spectral clustering via
integrating nonnegative embedding and spectral embedding,” Inf. Fusion,
vol. 55, pp. 251–259, Mar. 2020.
[17] J. Wu, Z. Lin, and H. Zha, “Essential tensor learning for multi-view
spectral clustering,” IEEE Trans. Image Process., vol. 28, no. 12,
pp. 5910–5922, Dec. 2019.
[18] M.-M. Cheng, L. Jing, and M. K. Ng, “Tensor-based low-dimensional
representation learning for multi-view clustering,” IEEE Trans. Image
Process., vol. 28, no. 5, pp. 2399–2414, May 2019.
[19] Z. Yang, Q. Xu, W. Zhang, X. Cao, and Q. Huang, “Split multiplicative
multi-view subspace clustering,” IEEE Trans. Image Process., vol. 28,
no. 10, pp. 5147–5160, Oct. 2019.
[20] X. Peng, Z. Huang, J. Lv, H. Zhu, and J. T. Zhou, “Multi-view clustering
without parameter selection,” in Proc. 36th Int. Conf. Mach. Learn.,
2019, pp. 5092–5101.
[21] A. Kumar and H. Daume, “A co-training approach for multi-view
spectral clustering,” in Proc. 28th Int. Conf. Mach. Learn., vol. 2011,
pp. 393–400.
[22] A. Kumar, P. Rai, and H. Daume, “Co-regularized multi-view spec-
tral clustering,” in Proc. Adv. Neural Inf. Process. Syst., 2011,
pp. 1413–1421.
[23] K. Zhan, F. Nie, J. Wang, and Y. Yang, “Multiview consensus graph
clustering,” IEEE Trans. Image Process., vol. 28, no. 3, pp. 1261–1270,
Mar. 2019.
[24] J. Wen et al., “Adaptive graph completion based incomplete multi-view
clustering,” IEEE Trans. Multimedia, early access, Aug. 3, 2020, doi:
10.1109/TMM.2020.3013408.
[25] C. Zhang, Q. Hu, H. Fu, P. Zhu, and X. Cao, “Latent multi-view
subspace clustering,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit.
(CVPR), Jul. 2017, pp. 4279–4287.
[26] C. Zhang et al., “Generalized latent multi-view subspace clustering,”
IEEE Trans. Pattern Anal. Mach. Intell., vol. 42, no. 1, pp. 86–99,
Jan. 2020.
[27] X. Wang, Z. Lei, X. Guo, C. Zhang, H. Shi, and S. Z. Li, “Multi-view
subspace clustering with intactness-aware similarity,” Pattern Recognit.,
vol. 88, pp. 50–63, Apr. 2019.
[28] Y. Chen, X. Xiao, and Y. Zhou, “Jointly learning kernel representation
tensor and affinity matrix for multi-view clustering,” IEEE Trans.
Multimedia, vol. 22, no. 8, pp. 1985–1997, Aug. 2020.
[29] L. Zong, X. Zhang, L. Zhao, H. Yu, and Q. Zho, “Multi-view cluster-
ing via multi-manifold regularized non-negative matrix factorization,”
Neural Netw., vol. 88, pp. 74–89, Apr. 2017.
[30] F. Nie, G. Cai, J. Li, and X. Li, “Auto-weighted multi-view learning for
image clustering and semi-supervised classification,” IEEE Trans. Image
Process., vol. 27, no. 3, pp. 1501–1511, Sep. 2017.
[31] D. Xie, Q. Gao, Q. Wang, X. Zhang, and X. Gao, “Adaptive latent
similarity learning for multi-view clustering,” Neural Netw., vol. 121,
pp. 409–418, Jan. 2020.
[32] R. Xia, Y. Pan, L. Du, and J. Yin, “Robust multi-view spectral clustering
via low-rank and sparse decomposition,” in Proc. 28th AAAI Conf. Artif.
Intell. (AAAI), 2014, pp. 2149–2155.
[33] H. Xu, X. Zhang, W. Xia, Q. Gao, and X. Gao, “Low-rank tensor
constrained co-regularized multi-view spectral clustering,” Neural Netw.,
vol. 132, pp. 245–252, Dec. 2020.
[34] J. Wu, X. Xie, L. Nie, Z. Lin, and H. Zha, “Unified graph and low-rank
tensor learning for multi-view clustering,” in Proc. 34th AAAI Conf.
Artif. Intell. (AAAI), 2020, pp. 6388–6395.
6784
[35] Z. Tao, H. Liu, S. Li, Z. Ding, and Y. Fu, “From ensemble clustering
to multi-view clustering,” in Proc. 26th Int. Joint Conf. Artif. Intell.,
Aug. 2017, pp. 2843–2849.
[36] Y. Wang, L. Wu, X. Lin, and J. Gao, “Multiview spectral clustering
via structured low-rank matrix factorization,” IEEE Trans. Neural Netw.
Learn. Syst., vol. 29, no. 10, pp. 4833–4843, Oct. 2018.
[37] Y. Chen, X. Xiao, and Y. Zhou, “Multi-view clustering via simulta-
neously learning graph regularized low-rank tensor representation and
affinity matrix,” in Proc. IEEE Int. Conf. Multimedia Expo (ICME),
Jul. 2019, pp. 1348–1353.
[38] S. Li, M. Shao, and Y. Fu, “Multi-view low-rank analysis with applica-
tions to outlier detection,” ACM Trans. Knowl. Discovery Data, vol. 12,
no. 3, pp. 32-1–32-22, 2018.
[39] Z. Tao, H. Liu, S. Li, Z. Ding, and Y. Fu, “Marginalized multiview
ensemble clustering,” IEEE Trans. Neural Netw. Learn. Syst., vol. 31,
no. 2, pp. 600–611, Feb. 2020.
[40] F. Nie, X. Wang, and H. Huang, “Clustering and projected clustering
with adaptive neighbors,” in Proc. 20th ACM SIGKDD Int. Conf. Knowl.
Discovery Data Mining, Aug. 2014, pp. 977–986.
[41] Y. Li, F. Nie, H. Huang, and J. Huang, “Large-scale multi-view spectral
clustering via bipartite graph,” in Proc. 29th AAAI Conf. Artif. Intell.
(AAAI), 2015, pp. 2750–2756.
[42] S. Bickel and T. Scheffer, “Multi-view clustering,” in Proc. 4th IEEE
Int. Conf. Data Mining (ICDM), Nov. 2004, pp. 19–26.
[43] H. Wang, Y. Yang, and T. Li, “Multi-view clustering via
concept factorization with local manifold regularization,” in
Proc. IEEE 16th Int. Conf. Data Mining (ICDM), Dec. 2016,
pp. 1245–1250.
[44] F. Nie, J. Li, and X. Li, “Self-weighted multiview clustering with
multiple graphs,” in Proc. 26th Int. Joint Conf. Artif. Intell., Aug. 2017,
pp. 2564–2570.
[45] H. Zhao, Z. Ding, and Y. Fu, “Multi-view clustering via deep matrix
factorization,” in Proc. 31st AAAI Conf. Artif. Intell. (AAAI), 2017,
pp. 2921–2927.
[46] L. Houthuys, R. Langone, and J. A. K. Suykens, “Multi-
view kernel spectral clustering,” Inf. Fusion, vol. 44, pp. 46–56,
Nov. 2018.
[47] Z. Huang, J. T. Zhou, X. Peng, C. Zhang, H. Zhu, and J. Lv, “Multi-
view spectral clustering network,” in Proc. 28th Int. Joint Conf. Artif.
Intell., Aug. 2019, pp. 2563–2569.
[48] L. Lovasz and M. D. Plummer, Matching Theory. Providence, RI, USA:
American Mathematical Society, 2009.
[49] A. Strehl and J. Ghosh, “Cluster ensembles—A knowledge reuse frame-
work for combining partitionings,” Proc. 18th AAAI Conf. Artif. Intell.
(AAAI), vol. 3, 2002, pp. 583–617.
[50] L. Hubert and P. Arabie, “Comparing partitions,” J. Classification, vol. 2,
no. 1, pp. 193–218, 1985.
[51] L. van der Maaten and G. Hinton, “Visualizing data using t-SNE,”
J. Mach. Learn. Res., vol. 9, pp. 2579–2605, Nov. 2008.
Aiping Huang (Member, IEEE) received the B.S.
degree in mathematics and applied mathematics
from Putian University, Putian, China, in 2011, and
the M.S. degree in basic mathematics from Minnan
Normal University, Zhangzhou, China, in 2014. She
is currently pursuing the Ph.D. degree with the
College of Physics and Information Engineering,
Fuzhou University. She worked as a Lecturer with
the School of Information Science and Technology,
Xiamen University Tan Kah Kee College, from
2014 to 2018. Her research interests include com-
puter vision, image processing, machine learning, and data mining.
Authorized licensed use limited to: University of Technology Sydney. Downloaded on November 10,2023 at 12:08:03 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 30, 2021
Weiling Chen (Member, IEEE) received the
B.S. and Ph.D. degrees in communication engi-
neering from Xiamen University, Xiamen, China,
in 2013 and 2018, respectively. She is currently a
Lecturer with the College of Physics and Informa-
tion Engineering, Fuzhou University, China. From
September 2016 to December 2016, she held a vis-
iting position with the School of Computer Science
and Engineering, Nanyang Technological University,
Singapore. Her current research interests include
image quality assessment, image compression, and
underwater acoustic communication.
Tiesong Zhao (Senior Member, IEEE) received the
B.S. degree in electrical engineering from the Uni-
versity of Science and Technology of China, Hefei,
China, in 2006, and the Ph.D. degree in computer
science from the City University of Hong Kong,
Hong Kong, in 2011. He worked as a Research
Associate with the Department of Computer Sci-
ence, City University of Hong Kong, from 2011 to
2012, a Postdoctoral Fellow with the Department
of Electrical and Computer Engineering, University
of Waterloo, from 2012 to 2013, and a Research
Scientist with the Ubiquitous Multimedia Laboratory, University at Buffalo,
The State University of New York, Buffalo, from 2014 to 2015. He is currently
a Minjiang Distinguished Professor with the College of Physics and Informa-
tion Engineering, Fuzhou University, China. His research interests include
multimedia signal processing, coding, quality assessment, and transmission.
Due to his contributions in video coding and transmission, he received the
Fujian Science and Technology Award for Young Scholars in 2017. He has
also been serving as an Associate Editor for IET Electronics Letters since
2019.
Chang Wen Chen (Fellow, IEEE) received the
B.S. degree from the University of Science and
Technology of China in 1983, the M.S.E.E. degree
from the University of Southern California in 1986,
and the Ph.D. degree from the University of Illinois
at Urbana–Champaign in 1992.
He is currently a Chair Professor of visual com-
puting at The Hong Kong Polytechnic University.
Previously, he has been an Empire Innovation Pro-
fessor of Computer Science and Engineering with
the University at Buffalo, The State University of
New York, from 2008 to 2021. He also served as the Dean of the School of
Science and Engineering, The Chinese University of Hong Kong, Shenzhen,
from 2017 to 2020. He was the Allen Henry Endow Chair Professor with the
Florida Institute of Technology from 2003 to 2007. He was with the Faculty
of Electrical and Computer Engineering, University of Missouri, Columbia,
from 1996 to 2003, and the Faculty of Electrical and Computer Engineering,
University of Rochester, from 1992 to 1996. His research has been supported
by NSF, DARPA, Air Force, NASA, Whitaker Foundation, Microsoft, Intel,
Kodak, Huawei, and Technicolor.
Prof. Chen has been a fellow of SPIE since 2007. He received nine
best paper awards or best student paper awards. He has also received
several research and professional achievement awards, including the Sigma
Xi Excellence in Graduate Research Mentoring Award in 2003, the Alexander
von Humboldt Research Award in 2009, the University at Buffalo Exceptional
Scholar-Sustained Achievement Award in 2012, The State University of
New York System Chancellors Award for Excellence in Scholarship and
Creative Activities in 2016, and the Distinguished ECE Alumni Award from
the University of Illinois at Urbana–Champaign in 2019. He has served as
the Conference Chair for several major IEEE, ACM, and SPIE conferences
related to multimedia video communications and signal processing. From
January 2014 to December 2016, he has been the Editor-in-Chief of IEEE
T RANSACTIONS ON M ULTIMEDIA. From January 2006 to December 2009,
he has also served as the Editor-in-Chief for IEEE T RANSACTIONS ON
C IRCUITS AND S YSTEMS FOR V IDEO T ECHNOLOGY. He has been an
Editor for several other major IEEE T RANSACTIONS and journals, including
the P ROCEEDINGS OF IEEE, IEEE J OURNAL OF S ELECTED A REAS IN
C OMMUNICATIONS, and IEEE J OURNAL OF E MERGING AND S ELECTED
T OPICS IN C IRCUITS AND S YSTEMS .