Semi-Supervised Kernel-Based Temporal Clustering
Rodrigo Araujo Mohamed S. Kamel
Department of Electrical and
Computer Engineering
University of Waterloo
Waterloo, Ontario N2L 3G1
Email: raraujo@uwaterloo.ca
Abstract—In this paper, we adapt two existing methods to per-
form semi-supervised temporal clustering: Aligned Cluster Anal-
ysis (ACA), a temporal clustering algorithm, and Constrained
Spectral Clustering, a semi-supervised clustering algorithm. In
the first method, we add side information in the form of pairwise
constraints to its objective function, and in the second, we add a
temporal search to its framework. We also extend both methods
by propagating the constraints throughout the whole similarity
matrix. In order to validate the advantage of the proposed semi-
supervised methods to temporal clustering, we evaluate them in
comparison to their original versions as well as another semi-
supervised temporal cluster on three temporal datasets. The
results show that the proposed methods are competitive and
provide good improvement over the unsupervised approaches.
Keywords—Semi-supervised clustering; Temporal segmentation;
Kernel k-means.
I. I NTRODUCTION
Temporal clustering (TC) can be defined as the factor-
ization of multiple time series into a set of non-overlapping
segments that belong to k temporal clusters [1]. Temporal
clustering is similar to normal clustering, which also requires
a similarity measure, a clustering algorithm, and an evaluation
criterion; however, the temporal nature of the data requires
special treatment at one or more of these components. The
two major ways to handle time series is to either modify
existing static data clustering algorithms to handle time, or
convert the time series into a form that allows static algorithms
to work on. The former approach relies, in most cases, on
modifying the similarity measure to an appropriate measure
of time series, for example, dynamic time warping (DTW).
The latter maps the time series into a different representation
or domain that embeds the temporal information, such as
Wavelets, Fourier, and Haar transform [2]; or into a number of
model parameters and then apply the conventional static data
clustering algorithm.
Clustering time series is a tool that can be applied in many
problems such as data mining, visualization, and segmenta-
tion. The problem of segmentation is specially important in
applications like human motion analysis, audio-visual emotion
analysis, animal behaviour analysis, etc. Solving the problem
of segmentation using a clustering approach requires modelling
the temporal variability of the segments in a time series. As
a result, simply applying some of the traditional time series
clustering techniques such as the ones reviewed in [3] may not
produce satisfactory results. Most of the temporal clustering
Department of Electrical and
Computer Engineering
University of Waterloo
Waterloo, Ontario N2L 3G1
Email: mkamel@uwaterloo.ca
techniques assume that the time series are already segmented,
and this is the main difference between the traditional time
series clustering techniques and what is being defined as TC.
One technique that uses a clustering approach to segment the
data is Aligned Cluster Analysis (ACA) [4], which frames this
problem as an energy-based temporal clustering and solves it
via dynamic programming.
Similar to the general clustering problem, temporal seg-
mentation using clustering may not have satisfactory results
due to being a totally unsupervised approach. In some situa-
tions, prior high-level knowledge is known about the clusters,
or some labeled data is available. This information can be
used to aid the clustering algorithm, and this class of learning
is known as semi-supervised clustering.
The most common way to add supervisory information
is the use of two pairwise constraints: must-link and cannot-
link. In must-link constraints, two points must be in the same
cluster, and in cannot-link constraints, two points should not
be in the same cluster. This type of constraint is a more
general form of supervisory information compared to labeled
data, and it can be expanded by using transitive closure on
the must-link relations. Some popular unsupervised algorithms
that have been adapted into a semi-supervised framework are
Constrained K-means Clustering with Background Knowledge
[5], Semi-supervised Kernel Mean Shift Clustering [6], Semi-
supervised Kernel K-means [7], and Constrained Spectral
Clustering [8]. However, all these methods are not designed
to handle temporal data.
In this work, we investigate developing semi-supervised
temporal clustering methods. First, we present Semi-
Supervised Aligned Clustering Analysis (SSACA), an adap-
tation of the successful temporal clustering method, Aligned
Cluster Analysis (ACA) [4], where we add side information
in the form of pairwise constraints. Second, we change Con-
strained Spectral Clustering, which is a semi-supervised clus-
tering method, to a temporal semi-supervised method using the
same framework. We also add to both methods, an exhaustive
constraint propagation algorithm to improve their performance.
We test our methods on both synthetic and real-world datasets.
II. R ELATED WORK
Segmentation and clustering of time series are problems
in various fields including video segmentation [9], facial ex-
pression interpretation [4], animal behavior recognition [10],
human-motion analysis [11], stock market analysis, etc. How-
ever, an in-depth understanding of this problem is necessary
to differentiate the techniques used to solve it.
It is noteworthy that the TC problem addressed in this paper
is different from clustering time series, which refers to the
problem of clustering time series when they are already seg-
mented. The proposed approach performs temporal segmenta-
tion and groups similar clusters simultaneously. Another way
of doing temporal segmentation is by change-point detection
[12] [13]. Change-point detection consists of an analysis of
changes on the distribution of the points within a window
of temporal observations in order to locate their boundaries.
However, this technique detects only local boundaries and does
not provide a global model for temporal events. HACA [11]
and ACA [4] solve this problem by minimizing the errors
across various segments for each k cluster.
In terms of semi-supervised methods, we have identified
two approaches that add supervision to temporal clustering.
[14] proposes Temporal-Driven Constrained K-Means (TDCK-
Means), an extension of k-means that incorporates a temporal-
aware dissimilarity measure, which combines Euclidean dis-
tance in the multidimensional space with the distance between
timestamps. TDCK-means makes an assumption that adjacent
observations of the same entity should be assigned to the same
cluster, and based on this assumption it creates a contiguity
penalty function that penalizes contiguous observations that
are not assigned to the same cluster. Therefore, external
information knowledge is embedded to the algorithm based on
this notion that similar observations should be close in time.
2
Essentially, they have must-link soft constrain between all pairs
ψ(X[si ,si+1 ) ) − zc
=
of observations belonging to the same entity.
[15] proposes a system that summarizes a user’s daily
activities, such as sleeping, walking, and working by com-
bining data from body sensors, GPS, computer monitoring
data, videos, and audio. The paper describes a semi-supervised
temporal clustering algorithm to group large amounts of
multimodal data into different activities. Similar to [14], the
constraints are used to penalize non-smooth changes (over
time) on the assigned clusters. In this method, there is no
control over the granularity of the temporal term, and there
is also no robust metric between time series.
III. M ODEL D ESCRIPTION
A. Aligned Clustering Analysis (ACA)
Aligned Cluster Analysis (ACA) [4] is an extension of the
kernel k-means clustering algorithm. ACA combines kernel
k-means with Dynamic Time Alignment Kernel (DTAK), an
alternative to DTW (Dynamic Time Warping).
Kernel k-means is a partitioning clustering algorithm,
where n data points are divided into k disjoint partitions. It
minimizes an objective function that represents the within-
cluster and finds the partitions of the data that is a local
optimum of the following energy function:
k X
X n
2 2
Jkkm (G) = gci kφ(xi ) − zc k = kX − ZGkF ,
c=1 i=1
|
2
{z } (1)
distφ (xi , zc )
T
s.t.G 1k = 1n ,
where xi ∈ Rd (see notation1 ) is a vector representing the ith
data point, φ(.) is a nonlinear mapping function, and zc ∈ Rd
represents the centroid of the data points in cluster c. Matrix
G ∈ {0, 1}k×n indicates, in a binary format, the cluster to
which xi belongs, and Z ∈ Rd×n is the actual value of the
means.
ACA considers distances between segments, which can
even be of different lengths, instead of distances between
points. The chosen distance for ACA is Dynamic Time Align-
ment Kernel (DTAK), which is an extension of DTW. DTAK,
unlike DTW, satisfies the triangle inequality. In DTAK, the
distance between two sequences, X = [x1 , . . . , xn ] ∈ Rd×n
and Y = [y1 , . . . , yn ] ∈ Rd×n can be computed recursively
(details on [11]).
The goal of ACA is to decompose a segment X =
[x1 , . . . , xn ] ∈ Rd×n into m disjoint segments, where each
segment belongs to a single k cluster. Each segment is
constrained by a maximum length nmax , which controls the
temporal granularity of the segmentation. The segments begin
at position si and end at si+1 − 1, such that ni = si+1 − si ≤
nmax . An indicator matrix G ∈ {0, 1}k×m assigns each
segment to a cluster; gci = 1 if Zi belongs to cluster c.
ACA achieve temporal clustering by minimizing:
m
k X
X
Jaca (G, s) = gci
c=1 i=1
| {z }
dist2ψ (Yi , zc ) (2)
k[ψ(Y1 , . . . , ψ(Ym ) − ,
2
ZGkF
T
s.t.G 1k = 1m and si+1 − si ∈ [1, nmax ],
where G ∈ {0, 1}k×m is a cluster indicator matrix, and s ∈
Rm+1 is the segment vector Y = X[si ,si+1 ) . In the case of
ACA, the dist2ψ (Yi , zc ) is the squared distance between the
ith segment and the center of cluster c in the nonlinear mapped
feature space represented by ψ(.).
B. Kernel-Based Semi-Supervised Clustering
[7] shows that the objective function of semi-supervised
clustering based on Hidden Markov Random Fields (HMRF),
with squared Euclidian distance and a certain class of con-
straint penalty function, can be expressed as a special case
of weighted kernel k-means. They also point out the the-
oretical equivalence between weighted kernel k-means and
graph clustering, which unifies several vector-based and graph-
based approaches. One implication of this unification is that a
single algorithm for semi-supervised clustering make several
objective functions are included in the same framework as
special cases.
The semi-supervised clustering objective of HMRF can be
1 Bold capital letters denote a matrix X, bold lower-case letters a column
vector x
expressed as where M is the set of must-link constraints, C is the set of
cannot-link constraints, wij is the penalty cost for violating a
k X
n
X 2 constraint xi and xj , and gi refers to the cluster label of xi .
JHM RF (Z, G) = gci kxi − zc k Similar to the objective function of HMRF k-means, we have
c=1 i=1
X X (3) the same three terms: S, must-link, and cannot-link.
− wij + wij ,
Because the constraints are held in the segment level,
xi ,xj ∈M xi ,xj ∈C
gi =gj gi =gj we have two kernel matrices: K and T. K is the frame
kernel matrix, K = φ(X)T φ(X) ∈ Rn×n , and each entry
where M is the set of must-link constraints, C is the set of kij defines the similarity between two frames, xi and xj .
cannot-link constraints, wij is the penalty cost for violating a T = [τij ]m×m ∈ Rm×m is the segment kernel matrix that
constraint xi and xj , and gi refers to the cluster label of xi . represents the similarity of the segments using the distance
There are three terms in this objective function. The first term DTAK. To use the same framework, the kernel matrix should
is the unsupervised k-means term of the objective function. be constructed as T = S + W, where S represents the
2
Note that the distance kxi − zc k can be represented as a similarity matrix in the segment level, and W is the weight
matrix of pairwise squared Euclidean distances among the data matrix.
points (proof in [7]). We refer later to this distance matrix as S. The optimization of SSACA’s objective function is a non-
The second term is based on the must-link constraints, which convex problem. In ACA, the method adopted is a dynamic
state that, for every must-link xi and xj (if xi and xj are in the programming (DP)-based algorithm, which has a complexity
same cluster, i.e., they satisfied that constraint), the objective O(n2 nmax ) to examine all possible segmentations. In SSACA,
function is rewarded by subtracting some pre-specified weight. we adapt the DP-based search to a semi-supervised framework,
Similarly, the third term in the objective function states that, for where we incorporate the pairwise constraints into the algo-
every xi , xj that is a cannot-link, if they are in the same cluster, rithm. We call the new algorithm SS DPSearch (Algorithm
they have violated that constraint, so the objective function is 1). SS DPSearch optimizes SSACA w.r.t. G and s, as well
penalized by some pre-specified penalty weight. Later, we refer as rewarding or penalizing the distance between segments
to the second and third term of the function as W. τ (X[i,v] , Y˙j ), according to the given pairwise constraints.
[7] also shows that, for the equivalence of the HMRF k-
means and the weighted kernel k-means to hold, it is necessary Algorithm 1 SS DPSearch
to construct a certain kernel matrix and set weights in a parameter: nmax , k, nml , ncl
certain way. A kernel matrix K should have two components input: G ∈ {0, 1}k×ṁ , ṡ ∈ R(ṁ+1) , K ∈ Rn×n , T ∈ Rm×m ,
K = S + W. S is the similarity matrix and comes from the M ∈ Znml ×2 , C ∈ Zncl ×2
unsupervised term, while W is the constraint matrix. This output: G ∈ {0, 1}k×m , ṡ ∈ R(m+1)
matrix W has a pre-specified wij weight for must-link and 1: headTail = getHeadTails(M, C);
−wij for cannot-link, and zero otherwise. Thus, this objective 2: for v = 1 to n do
function is mathematically equivalent to the weighted kernel k- 3: J(v) ← ∞;
means objective function. In other words, we can run weighted 4: if v ≥ headTail(:,1) and v < headTail(:,2) then
kernel k-means to decrease the objective function. 5: continue;
6: end if
It may be the case that K is not positive semidefinite, a
7: if isTail(v) then
requirement for the kernel k-means to converge. This problem
8: for j = 1 to ṁ do
can be avoided by performing diagonal shifting [7], which does
9: Retrive directly from T(X[i,v] , Y˙j );
not change the global optimal solution. However, in practice,
10: end for
we observe that diagonal shifting is not necessary in order to
11: c∗ ← arg minc distψ (X[i,v] , z˙c ); J ←
guarantee convergence.
distψ (X[i,v] , żc∗ );
∗
12: J([i, v]) ← J, g[i,v] ← e∗c , i∗[i,v] ← i;
C. Semi-Supervised Aligned Clustering Analysis (SSACA) 13: else
14: for nv = 1 to min(nmax , v) do
In SSACA, we use of the equivalence of ACA and
15: { Same as DPSearch [11]}
weighted kernel k-means to incorporate the kernel k-means
16: end for
semi-supervised framework into ACA. Therefore, we incorpo-
17: end if
rate the penalty terms, which are derived from the must-link
18: end for
and cannot-link, into the objective function. Thus, we can write
{Perform backward segmentation [11]}
the SSACA objective function as

As parameters, the algorithm receives the length constraint

k X
m

2 nmax and the number k of clusters. As inputs, the algorithm
requires an initial indicator matrix G ∈ {0, 1}k×m , that
X

Jssaca (G, s) = gci
ψ(X[si ,si+1 ) ) − zc

c=1 i=1
| {z } assigns segments to a cluster, an initial segmentation ṡ ∈
dist2ψ (Yi , zc ) (4) R(ṁ+1) , and the frame kernel matrix K ∈ Rn×n . Additionally
X X to the original approach inputs, there are pairs of must-links
− wij + wij ,
∈ Znml ×2 , cannot-links ∈ Zncl ×2 , and T ∈ Rm×m , which is
xi ,xj ∈M xi ,xj ∈C
gi =gj gi =gj the segment kernel matrix.
 1) Exhaustive and Efficient Constraint Propagation: The
pairwise constraints are used to adjust the similarity matrix
for the kernel k-means clustering algorithm by penalizing
or rewarding constrained segments. This technique, however,
affects only the constrained segment similarities. To make the
propagation of constraints more efficient, we borrow the idea
of exhaustive and efficient constraint propagation (E 2 CP) from
Lu and Ip [8] and adapt it to our framework. The rationale
behind this method is to spread the effects of the constraints
throughout the whole similarity matrix S.
E 2 CP tackles the problem of constraint propagation by
decomposing it into sets of label propagation subproblems.
Given the dataset X = [x1 , . . . , xnz ] ∈ Rd×nz , a set of must-
link M, and a set of cannot-link C, we can represent all the
pairwise constraints in a single matrix W = Zij N ×N , where
Wij = 1 if (xi , xj ) ∈ M, Wij = −1 if (xi , xj ) ∈ C, and
Wij = 0 otherwise.
Each j-th column of W.j can be seen as a two-class semi-
supervised learning problem, where the positive class (Wij >
0) represents the segments that should be in the same cluster
and the negative class (Wij < 0) represents the segments
that should not be in the same cluster. If (Wij ) = 0, xi xj
are not constrained. Each column and row are solved by label
propagation in parallel [16] to ensure that all the segments will
be affected by the propagation. The algorithm can be described
as follows:
1) Create the similarity matrix T or a symmetric k-NN
graph.
−1 −1
2) Create the matrix L̄ = D 2 T D 2 , where D is a
diagonal matrix with its (i,i)-element equal to the sum
of the i-th row of T .
3) Iterate Fv (t + 1) = αL̄Fv (t) + (1 − α)W for ver-
tical constraint propagation until convergence, where
Fv (t) ∈ F and α is a parameter between 0 and 1.
4) Iterate Fh (t + 1) = αFh (t)L̄ + (1 − α)Fv∗ for
horizontal constraint propagation until convergence,
where Fh (t) ∈ F and Fv∗ is the limit of {Fv(t) }.
5) Output F ∗ = Fh∗ as the final representation of the
pairwise constraints, where Fh∗ is the limit of {Fh(t) }.
Intuitively, the algorithm receives information from its
neighbor at each iteration, and the parameter α controls the
relative amount of information passed from the neighbors. The
final label of the segments is set to be the cluster from which it
has received the most information during the iteration process.
Without loss of generality, [16] shows that {F (t)} can
be calculated in a closed form. The output F ∗ represents
an exhaustive set of pairwise constraints with the associated
confidence scores |F ∗ |. Now, we can adjust the similarities in
T with the output scores of F ∗ as described in Equation 5.
This resulting similarity matrix T̃ has the same properties as
the kernel matrix specified in the kernel k-means framework;
it is both nonnegative and symmetric [8].
1 − (1 − Fij∗ )(1 − Wij ), Fij∗ ≥ 0

T˜ij = (5)
(1 + Fij∗ )Wij , Fij∗ < 0
In order to improve the performance of SSACA, we add
the exhaustive propagation (EP) to its algorithm, and we have
SSACA+EP. Algorithm 2 shows SSACA+EP.
Algorithm 2 SSACA + EP (Exhaustive propagation)
input: S ∈ Rn×n : input frame kernel matrix, T ∈ Rn×n : in-
put segment kernel matrix, W ∈ Rn×n : constraint penalty,
k: number of clusters, M: set of must-link constraints, C:
set of cannot-link constraints, ṡ: initial segmentation.
output: G ∈ {0, 1}k×n : Final partitioning of the points, s:
Final segmentation.
1: Propagate the constraints F ∗ ← W
2: Form the matrix T̃ according to Equation 5.
3: Diagonal-shift T̃ by adding σI to guarantee positive
definiteness of T̃ .
4: Get initial clusters G(0) using constraints.
5: Return G, s = SSDPSearch(G(0) , ṡ, S, T̃ , M , C, k).
D. Semi-Supervised Temporal Spectral Clustering (SSTSC)
Spectral clustering is a graph clustering algorithm that
uses the eigenvalues of the similarities to reduce their di-
mensionality before clustering them. As explained in III-B,
there is an equivalence between weighted kernel k-means and
graph clustering. In order to use the same methodology that
we use for SSACA and to comply with the semi-supervised
kernel k-means framework, we need to form a kernel matrix
in the K = S + W format. The output T˜ij of the exhaustive
propagation method satisfies that requirement. To perform
spectral clustering, we need to find the k largest nontrivial
eigenvector of T̃ to derive the low dimension feature vector.
Next, we perform the temporal clustering search, which adds
the temporal aspect to this approach. The first steps are
described below, and the complete algorithm is depicted in
Algorithm 3.
1) Find k largest nontrivial eigenvectors v1 . . . vk of
−1 −1
D̃ 2 T̃ D̃ 2 , where D̃ is a diagonal matrix with its
(i,i)-element equal to the sum of the i-th row of T̃ ;
2) Form E = [v1 , . . . , vk ], and normalize each row to
the unit length. The rows of E represent the low-
dimensional features of segment X[si ,si+1 ] ;
3) Create a kernel with the new feature vector Ei (i =
1, . . . , m).
Algorithm 3 SS Temporal Spectral Clustering + EP
input: S ∈ Rn×n : input frame kernel matrix, T ∈ Rn×n : in-
put segment kernel matrix, W ∈ Rn×n : constraint penalty,
k: number of clusters, M: set of must-link constraints, C:
set of cannot-link constraints, ṡ: initial segmentation.
output: G ∈ {0, 1}k×n : Final partitioning of the points, s:
Final segmentation.
1: Propagate the constraints F ∗ ← W .
2: Form the matrix T̃ according to Equation 5.
3: Diagonal-shift T̃ by adding σI to guarantee positive
definiteness of T̃ .
4: Get initial clusters G(0) using constraints.
5: Create a new kernel T̂ with the new feature vector Ei (i =
1, . . . , m).
6: Return G, s = SSDPSearch(G(0) , ṡ, S, T̂ , M , C, k).
TABLE I. AVERAGE ACCURACY RESULTS OF SOME TEMPORAL - BASED TABLE II. AVERAGE ACCURACY RESULTS ON HIGH VARIANCE
k- MEANS METHODS . SEGMENTS (V IDEO FEATURES ).

Average Accuracy Average Accuracy

TDCK-means TCK-means Constrained Temp-driven Simple Arousal Valence Power Expectancy
[14] k-means K-means K-means SSACA+EP 0.75 ± 0.09 0.73 ± 0.09 0.76 ± 0.12 0.83 ± 0.08
0.38 ± 0.05 0.36 ± 0.05 0.34 ± 0.04 0.30 ± 0.04 0.28 ± 0.02 SSTSC+EP 0.84 ± 0.10 0.77 ± 0.09 0.82 ± 0.09 0.92 ± 0.07
SSACA 0.75 ± 0.11 0.74 ± 0.09 0.76 ± 0.12 0.83 ± 0.08
TDCK [14] 0.52 ± 0.05 0.53 ± 0.06 0.46 ± 0.09 0.39 ± 0.04
ACA [4] 0.42 ± 0.02 0.54 ± 0.04 0.39 ± 0.04 0.42 ± 0.02
SC 0.37 ± 0.00 0.44 ± 0.00 0.41± 0.00 0.40 ± 0.00
IV. E XPERIMENTS
We demonstrate the feasibility of the proposed methods on TABLE III. AVERAGE ACCURACY RESULTS ON HIGH VARIANCE
three datasets: a synthetic time series, a motion capture dataset, SEGMENTS (AUDIO FEATURES ).
and an audio-visual emotion dataset. The accuracy evaluation
Average Accuracy
criterion is the same as in [17]. In all experiments, we compare Arousal Valence Power Expectancy
the proposed methods (SSACA, SSACA+EP, and SSTSC+EP) SSACA+EP 0.78 ± 0.08 0.78 ± 0.08 0.81 ± 0.14 0.92 ± 0.06
SSTSC+EP 0.76 ± 0.12 0.76 ± 0.08 0.83 ± 0.12 0.93 ± 0.08
to their baseline unsupervised approach (ACA), a version of SSACA 0.74 ± 0.10 0.74 ± 0.10 0.74 ± 0.13 0.84 ± 0.08
spectral clustering (SC) modified to handle time, as described TDCK [14] 0.50 ± 0.06 0.50 ± 0.08 0.45 ± 0.11 0.39 ± 0.04
in [11], and the recently proposed semi-supervised tempo- ACA [4] 0.47 ± 0.03 0.42 ± 0.03 0.73 ± 0.11 0.49 ± 0.02
SC 0.44 ± 0.00 0.39 ± 0.00 0.70 ± 0.00 0.45 ± 0.00
ral cluster, Temporal Driven Constrained k-means (TDCK)
[14]. Compared to other methods such as simple k-means,
Temporal-driven k-means, Constrained k-means, and Temporal
Constrained k-means, TDCK had the best results as can be sensible, happy, angry, and sad. The dataset is labeled for
attested by our own experiments in Table I. arousal, valence, power, and expectancy.
We perform the experiments on a combined video of four
A. Synthetic Dataset characters, so we have a longer conversation and a higher
The synthetic dataset is a randomly generated time series variability. We categorize the continuous values of the affective
created by sampling 2D gaussian distributions. It contains 78 dimensions, which range from [-1, +1], in 6 categories, and
segments (m = 78), four clusters (k = 4), and approximately we observe the results of visual and audio features on arousal,
4400 frames. For this experiment, we analyze the average valence, power, and expectancy. The audio features consist of
accuracy at different numbers of random constraints over 30 1871 features, including 25 energy and spectral related low-
runs. We use a Gaussian kernel with σ = 1 for all the level descriptors (LLD) x 42 functionals, 6 voicing related
kernel-based methods (all methods, except TDCK [14]), and LLD x 32 functionals, and 10 voiced/unvoiced durational
nmax = 17, which control the granularity of the ACA-based features. Details for LLD and functionals can be found in
methods. Specifically for the methods that uses exhaustive [18]. For visual features, we extract Local Binary Patterns
propagation, i.e SSACA+EP and SSTSC+EP, we use α = 0.01 (LBP), based on the approach described in [19]. We apply
and α = 0.4 respectively as the constraints propagation rate. approximately 5% of the total number of possible constraints.
For the TDCK algorithm the weights given for the time-aware
dissimilarity are γt = 0.25 and γd = 1, and the parameters for C. CMU Motion Capture Dataset
the penalty function are β = 0.3 and δ = 3.
Carnegie Mellon University Motion Capture Dataset (MO-
CAP) [20] is a human motion dataset. We perform the exper-
Synthe6c  Dataset   iments on fourteen sequences of subject 86, which contain
0.8   roughly ten actions (e.g., walking, punching, kicking). We
0.75  
0.7  
use the same setup as in [11]. The range of frames in the
0.65   experiment is from 160 to 300, the nmax is set to 60, and
SSACA+EP  
0.6   the number of constraints are based on 20% of all possible
Accuracy  

0.55   SSTSC+EP  
0.5   SSACA  
constraints.
0.45   TDCK  
0.4  
0.35  
ACA   V. R ESULTS AND D ISCUSSIONS
SC  
0.3  
0.25  
In the first experiment (Figure 1), we evaluate the methods
0   10   20   30   40   50   on different numbers of constraints. Note that the proposed
Number  of  Constraint  Pairs   semi-supervised methods increase consistently as the number

Fig. 1. Accuracy of synthetic data using different numbers of constraints. TABLE IV. ACCURACY OF 14 SEQUENCES OF SUBJECT 86 FROM THE
MOCAP DATASET.
Average over 14 sequences
B. Audio-visual Emotion Dataset Accuracy
SSACA+EP 0.92 ± 0.08
Audio-Visual Emotion Challenge (AVEC) [18] is an audio- SSTSC+EP 0.92 ± 0.07
visual emotion recognition dataset created for the emotion SSACA 0.92 ± 0.07
recognition challenge (AVEC 2012). The dataset consists of TDCK [14] 0.49 ± 0.05
ACA [4] 0.87 ± 0.09
conversations among several participants and four stereotyped SC 0.74 ± 0.13
characters. Each character has a specific emotion stereotype:
of constraints gets higher. Furthermore, after the number of
constraints reach over 40 pairs, the exhaustive propagation
method SSACA+EP starts to differentiate from its counter part
(SSACA). The single process of keeping the compatibilities
correct gets very expensive in high number of constraints,
therefore we stopped at 50 pairs.
Granted that the TDCK algorithm was originally created
for multi-entity problems, and in this case we are using it in
a single entity scenario, it can be said that this algorithm may
be under used; however, it is one of the few approaches that
is temporal and semi-supervised, and it is still fair to make
comparisons with our approaches. Differently from TDCK,
our method consider distances between segments instead of
distances between points, which captures better the temporal
dynamic of the data. Also, the pairwise constraints are directed
applied to the affected pairs, instead of using them as a unique
general assumption applied to all the data. These aspects
favourite the proposed methods over TDCK.
Tables II and III show the results of the complex problem of
audio-visual emotion analysis. On both cases, SSTSC+EP and
SSACA+EP, which have exhaustive propagation, performed
similar or better than the regular propagation used in SSACA.
The high complex nature of visual features makes the spectral-
based approach (SSTSC+EP) a more suitable method. For
the audio features, the advantage of SSTSC+EP is more
evident when we analyze the power and expectancy emotion
dimensions. In this type of application, emotions are recurrent,
and may be further apart, which causes TDCK to perform
poorly due to its assumption that benefits closer clusters.
In the human motion segmentation dataset, we also ob-
served improvements of the proposed methods compared to
the baselines (Table IV). However, differences between the
semi-supervised methods are not very visible due to the very
low number of segments.
In general, the methods show substantial improvements
with minimal addition of side information compared to ex-
clusively unsupervised methods. In addition, it can be seen
that some specific problems, such as audio-visual emotion
analysis is more suitable for SSTSC due to the spectral nature
of the algorithm, specially in the case of power and expectancy
dimensions. Simpler cases such as the synthetic dataset is
more suitable for SSACA. Moreover, the use of exhaustive
propagation can enhance the accuracy of the proposed methods
in situations with a higher number of constraints and segments.
VI. C ONCLUSION
In this paper, we propose two semi-supervised temporal
clustering methods. First, we transform a kernel-based unsu-
pervised temporal clustering into a semi-supervised temporal
clustering by adding some constraints in the form of must-links
and cannot-links. Then, we transform a constrained spectral
clustering into a temporal semi-supervised clustering. We
evaluate our methods on a synthetic dataset and on two distinct
complex problems. Results show substantial improvements
compared to the original unsupervised methods and other semi-
supervised method with minimal supervision.
R EFERENCES
[1] M. Hoai and F. De la Torre, “Maximum margin temporal clustering,”
in Proceedings of the Fifteenth International Conference on Artificial
Intelligence and Statistics (AISTATS-12), vol. 22, 2012, pp. 520–528.
[2] T. Fu, “A review on time series data mining,” Engineering Applications
of Artificial Intelligence, vol. 24, no. 1, pp. 164 – 181, 2011.
[3] T. W. Liao, “Clustering of time series data – a survey,” Pattern
Recognition, vol. 38, no. 11, pp. 1857 – 1874, 2005.
[4] F. Zhou, F. De la Torre, and J. F. Cohn, “Unsupervised discovery of
facial events,” in Computer Vision and Pattern Recognition (CVPR),
2010 IEEE Conference on, June 2010, pp. 2574–2581.
[5] K. Wagstaff, C. Cardie, S. Rogers, and S. Schrödl, “Constrained k-
means clustering with background knowledge,” in Proceedings of the
Eighteenth International Conference on Machine Learning, ser. ICML
’01. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc.,
2001, pp. 577–584.
[6] S. Anand, S. Mittal, O. Tuzel, and P. Meer, “Semi-supervised kernel
mean shift clustering,” Pattern Analysis and Machine Intelligence, IEEE
Transactions on, vol. PP, no. 99, pp. 1–1, 2013.
[7] B. Kulis, S. Basu, I. S. Dhillon, and R. J. Mooney, “Semi-supervised
graph clustering: a kernel approach,” Machine Learning, vol. 74, no. 1,
pp. 1–22, 2009.
[8] Z. Lu and H. H. S. Ip, “Constrained spectral clustering via exhaustive
and efficient constraint propagation,” in Proceedings of the 11th Euro-
pean Conference on Computer Vision: Part VI, ser. ECCV’10. Berlin,
Heidelberg: Springer-Verlag, 2010, pp. 1–14.
[9] S. Arifin and P. Cheung, “Affective Level Video Segmentation by
Utilizing the Pleasure-Arousal-Dominance Information,” Multimedia,
IEEE Transactions on, vol. 10, no. 7, pp. 1325–1341, 2008.
[10] S. Oh, J. Rehg, T. Balch, and F. Dellaert, “Learning and inferring motion
patterns using parametric segmental switching linear dynamic systems,”
International Journal of Computer Vision, vol. 77, no. 1-3, pp. 103–124,
2008.
[11] F. Zhou, F. De la Torre, and J. K. Hodgins, “Hierarchical aligned cluster
analysis for temporal clustering of human motion,” IEEE Transactions
Pattern Analysis and Machine Intelligence (PAMI), vol. 35, no. 3, pp.
582–596, 2013.
[12] X. Xuan and K. Murphy, “Modeling changing dependency structure
in multivariate time series,” in Proceedings of the 24th International
Conference on Machine Learning, ser. ICML ’07. New York, NY,
USA: ACM, 2007, pp. 1055–1062.
[13] Z. Harchaoui, E. Moulines, and F. R. Bach, “Kernel change-point
analysis,” in Advances in Neural Information Processing Systems 21,
D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou, Eds. Curran
Associates, Inc., 2009, pp. 609–616.
[14] M. Rizoiu, J. Velcin, and S. Lallich, “Structuring typical evolutions
using temporal-driven constrained clustering,” in Tools with Artificial
Intelligence (ICTAI), 2012 IEEE 24th International Conference on,
vol. 1, 2012, pp. 610–617.
[15] F. De la Torre and C. Agell, “Multimodal diaries,” in Multimedia and
Expo, 2007 IEEE International Conference on, July 2007, pp. 839–842.
[16] D. Zhou, O. Bousquet, T. N. Lal, J. Weston, and B. Schlkopf, “Learning
with local and global consistency,” in Advances in Neural Information
Processing Systems 16. MIT Press, 2004, pp. 321–328.
[17] R. Araujo and M. Kamel, “A semi-supervised temporal clustering
method for facial emotion analysis,” in Multimedia and Expo Workshops
(ICMEW), 2014 IEEE International Conference on, July 2014, p. to
appear.
[18] B. Schuller, M. Valstar, R. Cowie, and M. Pantic, “Avec 2012: The
continuous audio/visual emotion challenge - an introduction,” in Pro-
ceedings of the 14th ACM International Conference on Multimodal
Interaction, ser. ICMI ’12. New York, NY, USA: ACM, 2012, pp.
361–362.
[19] A. Sayedelahl, R. Araujo, and M. Kamel, “Audio-visual feature-decision
level fusion for spontaneous emotion estimation in speech conversa-
tions,” in Multimedia and Expo Workshops, 2013 IEEE International
Conference on, July 2013, pp. 1–6.
[20] M. Shell. (2012) Carnegie mellon university motion capture database.
[Online]. Available: http:// mocap.cs.cmu.edu