1898
28, 2021
Feature Fusion for Multimodal Emotion Recognition
Based on Deep Canonical Correlation Analysis
Ke Zhang, Yuanqing Li , Jingyu Wang , Member, IEEE, Zhen Wang , and Xuelong Li , Fellow, IEEE
Abstract—Fusion of multimodal features is a momentous prob-
lem for video emotion recognition. As the development of deep
learning, directly fusing feature matrixes of each mode through
neural networks at feature level becomes mainstream method.
However, unlike unimodal issues, for multimodal analysis, finding
the correlations between different modal is as important as dis-
covering effective unimodal features. To make up the deficiency in
unearthing the intrinsic relationships between multimodal, a novel
modularized multimodal emotion recognition model based on deep
canonical correlation analysis (MERDCCA) is proposed in this
letter. In MERDCCA, four utterances are gathered as a new group
and each utterance contains text, audio and visual information as
multimodal input. Gated recurrent unit layers are used to extract
the unimodal features. Deep canonical correlation analysis based
on encoder-decoder network is designed to extract cross-modal
correlations by maximizing the relevance between multimodal. The
experiments on two public datasets show that MERDCCA achieves
the better results.
Index Terms—Deep canonical correlation analysis, gated
recurrent unit, multimodal emotion recognition.
I. INTRODUCTION
ULTIMODAL emotion recognition researches have al-
M ways been the coexistences of opportunities and chal-
lenges [1]. As soon as the multimodal emotion recognition is
mentioned, it is always closely related to our daily life [2] and this
is also the reason why people’s daily habits become the hot spots
for researchers to seek new inspirations [3]. Visual [4], audio [5],
text [6], electroencephalogram [7] and many other expression
Manuscript received July 1, 2021; revised August 29, 2021; accepted August
30, 2021. Date of publication September 14, 2021; date of current version
September 29, 2021. This work was supported in part by the Basic Research
Strengthening Program of China under Grant 2020-JCJQ-ZD-015-00-02; in part
by the National Natural Science Foundation for Distinguished Young Scholars
under Grant 62025602; and in part by the National Natural Science Foundation
of China under Grants U1803263, 61871470, 11931015, and 61502391. The
associate editor coordinating the review of this manuscript and approving it for
publication was Prof. Xun Cao. (Corresponding author: Yuanqing Li.)
Ke Zhang and Yuanqing Li are with the National Key Laboratory of Aerospace
Flight Dynamics and School of Astronautics, Northwestern Polytechnical
University, Xi’an, Shaanxi 710072, China (e-mail: zhangke@nwpu.edu.cn;
YUANQINGLI@MAIL.NWPU.EDU.CN).
Jingyu Wang is with the School of Artificial Intelligence, Optics and Electron-
ics (iOPEN), School of Astronautics, Northwestern Polytechnical University,
Xi’an, Shaanxi 710072, China (e-mail: jywang@nwpu.edu.cn).
Zhen Wang and Xuelong Li are with the School of Artificial Intelli-
gence, Optics and Electronics (iOPEN), Northwestern Polytechnical Uni-
versity, Xi’an, Shaanxi 710072, China (e-mail: zhenwang0@gmail.com;
xuelong_li@nwpu.edu.cn).
Digital Object Identifier 10.1109/LSP.2021.3112314
1070-9908 © 2021 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: Amrita Vishwa Vidyapeetham Chennai Campus. Downloaded on January 20,2023 at 16:14:23 UTC from IEEE Xplore. Restrictions apply.
IEEE SIGNAL PROCESSING LETTERS, VOL.
signals [8] of human can be collected to carried out emotion
recognition. However, it is obviously that human, especially
human emotions, is very complicated [9]. There are a great
many ways for people to express their emotions [10] according
to the culture traditions and living habits [11]. These complex-
ity and uncertainty make multimodal emotion recognition get
more and more attention than unimodal emotion recognition
in recent years for its wider application scenarios and market
prospects [12]. Followed by these advantages, the difficulties
of extracting and fusing features from heterogeneous modals
becomes the main topics and obstacles for multimodal emotion
recognition.
At present, most multimodal emotion recognition methods
choose to extract notable features within unimodal before next
fusion. Audio features are integrated into contextualized word
embeddings through a parallel bidirectional language model
inspired by word embeddings [13]. Fuzzy logic Convolutional
Neural Network(CNN) is utilized to process tri-modal data [14].
Audio and visual information are fused at feature level by a
belief network based CNN [15] and three separate networks
are designed to capture features of trimodal parallelly [16].
These work all process multimodal inputs meanwhile by directly
calculating the feature matrices through deep neural networks
at feature level or decision level.
Unlike the work previously mentioned which make great
efforts on discovering the emotion related features of uni-
modal [17], some recent works transit their focuses on the
nonlinear relationship between modals and its influences on
feature selections. Tang et al. [18] design a deep neural network
to fuse multi-scale features for defocus blur detection. Implicit
fusion via joint feature representation learning is verified fea-
sible through single shared network [19] and cross-view local
structure [20]. Deep transfer networks are used by Shu et al. [21]
to mitigate the insufficient image data from text domain and by
Tang et al. [22] to transfer information across heterogeneous
domains. Wen et al. [23] proposes a context-gated convolution
neural network to discover the cross-modal information under
both word-aligned and un-aligned conditions. Chen et al. [24]
uses two separate CNNs to process the audio and visual infor-
mation and fuses them by calculating the feature correlations.
Chen et al. [25] also puts their key research point on digging the
implicit relation between audio and visual modal that all features
are mixed up and re-clustered by k-means. These new feature
groups then compute the cross-modal relationship according to
the emotion tags. Both of these former two work maximize the
correlation between heterogeneous modals by using canonical
correlation analysis, which is designed to reflect the overall
correlation between two groups of indicators. However, both
 ZHANG et al.: FEATURE FUSION FOR MULTIMODAL EMOTION RECOGNITION BASED ON DEEP CANONICAL CORRELATION ANALYSIS
Fig. 1. The Structure of the proposed MERDCCA.
of them ignore to find the internal effective features within each
unimodal.
Motivated by the issues mentioned above, this letter proposes
a modularized multimodal video emotion recognition model
based on deep canonical correlation analysis (MERDCCA).
To the best of our knowledge, this is the first time that both
the unimodal and cross-modal features are given consideration
with using audio, text and visual as trimodal inputs. The detail
of MERDCCA will be introduced in following sections. The
main contributions of this letter are listed as below: (1) a novel
modularized multimodal emotion recognition model based on
deep canonical correlation analysis is proposed to improve
the multimodal emotion recognition performance. (2) Deep
Canonical Correlation Analysis(DCCA) is applied to generate
integrated variables of multimodal and maximize their correla-
tions. (3) Four related utterances are united as a new group and
the encoder-decoder system with optimized loss function are
used to retain unimodal features and subsequent cross-modal
correlations at the same time.
II. PROPOSED METHOD
Screening and amplifying related features are the first and
most important step for following effective multimodal emotion
recognition [26]. Extracting features from unimodal or cross-
modal are popular options in recent years. In fact, these two
aspects have their own irreplaceable advantages in emotion
recognition. Unimodal features can better filter out the most
relevant information for emotion recognition from the other
noises. Cross-modal features can better explore the interaction
between modals, because as a complex human activity, emotion
is often a comprehensive expression of simultaneous interpreting
of various channel information. Unimodal features are unable
to express all effective information. However, traditional and
recent methods either put research points on the internal feature
extraction of unimodal, either focus on the features related to
cross-modal that there is almost no method that takes both into
account. In order to make up for this defect and make a discussion
in this area of research, this letter proposes a modularized
multimodal video emotion recognition model based on deep
canonical correlation analysis which is shown in Fig. 1. From
Fig. 1, there are mainly three modules in MERDCCA: unimodal
Authorized licensed use limited to: Amrita Vishwa Vidyapeetham Chennai Campus. Downloaded on January 20,2023 at 16:14:23 UTC from IEEE Xplore. Restrictions apply.
1899
feature extraction, correlation analysis and classification. The
loss function is specially designed as three parts to ensure the
constraining and optimizing of the effective features selection
for both unimodal and cross-modal simultaneously. The details
will be explained in the following subsections.
A. Unimodal Feature Extraction
This module is the first part of the MERDCCA which is
responsible for extracting unimodal features which are related
to emotion recognition.
Firstly, to make full use of context information, video seg-
ments are preprocessed that every four utterances are pack-
aged into a new input group and the third utterance is the
target waiting to be recognized as shown in Fig. 1. In new
group, the context of both sides of interlocutors are recorded
and the emotion persistence and contagious of the target are
potentially exposed. Let GA (t), GT (t) and GV (t) represent
the audio, text and visual features of the grouped utterances
including the target at timestamp t. Secondly, three parallel
←−−→
sub-nets consist of bidirectional gated recurrent unit (GRU ) are
designed to fused the tri-modal information of the new group
and attention mechanism(Attention) are combined to better ex-
tract context information and unimodal features. The processed
audio(Af usion ), text(Tf usion ) and visual(Vf usion ) features are
achieved as below.
←−−→
Af usion = GRU (Attention(GA (t))) (1)
←−−→
Tf usion = GRU (Attention(GT (t))) (2)
←−−→
Vf usion = GRU (Attention(GV (t))) (3)
B. Correlation Analysis
To compute the cross-modal correlations, we employ deep
canonical correlation analysis(DCCA) [27] to maximize the
dependence between representative vectors. DCCA is an under-
standing of cross covariance matrix and a multivariate statistical
analysis method that uses the correlation between comprehen-
sive variable pairs to reflect the overall correlation between two
groups of indicators. It is a data analysis and dimension reduction
method similar to Principal Component Analysis(PCA) [28].
Different from PCA, DCCA can reduce the dimension of two
 1900
spaces and provide the same heterogeneous representation. In
order to utilize DCCA to compute the cross-modal correlations,
extracted features of three modals are first fused into two chan-
nels according to the DCCA input dimension limitation. These
two channels are acquired by fusing text modal with audio and
visual separately through two GRU layers and attention mecha-
nism is added. The reason that we use text modal as the medium
of the bimodal fusion to help choose the key point of bimodal
features during training is that according to [29], compared to the
other two modals, text contains the most comprehensive content
of conversations and has less chance of misunderstanding.
After this step, fused bimodal information are input to the
next encoder-decoder system which is composed of two group
of fully connected layers with symmetrical number of nodes.
These two groups of layers correspond to former two bimodal
information channels respectively just as shown in Fig. 1. The
output of the encoder are represented as (H1 , H2 ) which are two
comprehensive variable pairs of the original bimodal features
after dimension reduction by the encoder. We extract the cross-
modal features from these two pairs by using DCCA to maximize
the correlation between them. The detailed calculation formulas
are as below.
←−−→
F usionT A = GRU (Attention(Tf usion , Af usion )) (4)
←−−→
F usionT V = GRU (Attention(Tf usion , Vf usion )) (5)
(H1 , H2 ) = Encoder(F usionT A , F usionT V ) (6)
where F usionT A and F usionT V are bimodal features fused
after GRU layers. In DCCA, to using the correlation coefficient
to calculate correlations for high-dimensional data, (H1 , H2 )
has to be processed as below.
H1 = aT H1 , H2 = bT H2 (7)
The goal of DCCA is to find the expected value (a∗ , b∗ ) of
(a, b) to maximize the corr(H1 , H2 ), which is the correlation
coefficient of (H1 , H2 ).
(a∗ , b∗ ) = arg max corr(H1 , H2 ) (8)
(a,b)
cov(H1 , H2 )
corr(H1 , H2 ) =   (9)
D(H1 ) D(H1 )
where cov(H1 , H2 ) is the covariance of H1 and H2 , and D(H1 ),
D(H1 ) are the variances of them, respectively. To optimize
this method, the formula above can be rewrite into following
constrained optimization formulation:
arg max aT SH1 H2 b
(a,b)
s.t. aT SH1 H1 a = 1, bT SH2 H2 b = 1 (10)
To obtain the maximal value of the target and corresponding
dimensional reduction vector (a, b), singular value decomposi-
tion (SVD) is used to carry out the optimization. Then, com-
prehensive variable pairs (H1 , H2 ) are reconstructed through
decoder which is mirror symmetry with encoder network. The
reconstituted output of decoder is represented as (RT A , RT V ).
(RT A , RT V ) = Decoder(H1 , H2 ) (11)
Authorized licensed use limited to: Amrita Vishwa Vidyapeetham Chennai Campus. Downloaded on January 20,2023 at 16:14:23 UTC from IEEE Xplore. Restrictions apply.
IEEE SIGNAL PROCESSING LETTERS, VOL. 28, 2021
TABLE I
DATASETS SPLIT
C. Classification and Loss Function
The reconstructed pair (RT A , RT V ) is finally fused again
through a GRU layer following with a fully connected layer
to complete the classification. To give considerations on both
unimodal feature extractions as well as cross-modal correlation
extraction, the final loss function Loss are composed of three
parts in our method, namely
Loss = lossDCCA + lossrec + lossclass (12)
where lossDCCA is the DCCA loss which is used to max-
imize the cross-modal features by measuring the correlation
between comprehensive variable pairs (H1 , H2 ). Larger the
correlation, smaller the loss. lossrec is the reconstruction dif-
ference using Euclidean distance between (RT A , RT V ) and
(F usionT A , F usionT V ). It is intended to maintain the ex-
tracted unimodal features from the first module. lossclass is the
cross-entropy loss of the final classification.
III. EXPERIMENTAL RESULTS AND ANALYSIS
A. Experimental Settings
IEMOCAP [30] and CMU-MOSI [31] are chosen to carry
out experiments based on our model. Tri-modal information of
both datasets are all recorded on utterance level. The former is
labeled with one of six emotion labels, which are happy, sad,
neutral, angry, excited, and frustrated, while the latter is tagged
as a value between −3 to +3 that higher the value, the more
positive the emotion with the neutral boundary as 0. The split of
datasets are shown in Table I.
To evaluate the performances of the experimental results, the
accuracy and macro-average F-score are calculated as below.
precision · recall
Fβ = (1 + β 2 ) · (13)
(β 2 · precision) + recall
where β represents the weight between precision and recall. In
this letter, β is set as 1 meaning precision and recall have same
weight. P-value is calculated through paired T-test between our
method and baselines as below to evaluate the significance with
the significance level α set as 0.05 [32].
d − d0
t= √ (14)
Sd / n
df = n − 1 (15)
where d and d0 represent the sample mean of differences and the
hypothesized population mean difference respectively, Sd is the
standard deviation of differences, df is the degree of freedom.
←−−→
The GRU layer in our method is set as 1 with 1024 hidden nodes
for unimodal feature extraction in the first module and 512 for
the others. The hidden nodes of the encoder-decoder system are
set as 512, 256, 2, 256 and 512 separately. The drop probably
 ZHANG et al.: FEATURE FUSION FOR MULTIMODAL EMOTION RECOGNITION BASED ON DEEP CANONICAL CORRELATION ANALYSIS
Fig. 2. Features of CMU-MOSI dataset.
Fig. 3. Features of IEMOCAP dataset.
TABLE II
EXPERIMENTAL RESULTS ON CMU-MOSI
TABLE III
EXPERIMENTAL RESULTS ON IEMOCAP
is set as 0.45 and the learning rate is 0.0015 with the optimizer
weight decay as 0.0005.
B. Results and Discussion
The experimental results on datasets are shown in Figure
and Figure respectively. For CMU-MOSI, sentiment analysis
is carried out and utterances are classified into positive and
negative. The trimodal information of text, audio and visual of
the first module is shown in Fig. 2(a). The value of (H1 , H2 )
after correlation calculation of the cross-modal through DCCA
in the second module is shown in Fig. 2(b). For IEMOCAP,
emotion recognition is done and six emotion types are classified.
Fig. 3(a) dipict the unimodal features and Fig. 3(b) shows the
corresponding cross-modal value of each emotion type after
DCCA.
The final recognition performances of our method are com-
pared with several baselines and the results of two datasets are
shown in Table II and Table III separately. In these tables, Acc.
means accuracy and bold font denotes the best performances.
Authorized licensed use limited to: Amrita Vishwa Vidyapeetham Chennai Campus. Downloaded on January 20,2023 at 16:14:23 UTC from IEEE Xplore. Restrictions apply.
1901
From Fig. 2(b) and Fig. 3(b), it can be seen clearly that
the distribution characteristics of each unimodal is maintained,
however, they are preliminary re-clustered while maximizing
cross-modal correlations through the second module using
DCCA. The addition of reconstruction loss and the DCCA loss
in loss function part enable enables our method to retain and
utilize both unimodal and cross-modal features at the same time
which is also one of the most significant differences with other
methods.
In Table II, our model achieves the best performance on
CMU-MOSI dataset and about 1% higher than the state-of-the-
art. The reason is that the method in [29] only consider the
cross-modal features based on attention mechanism, although
they found that text is the most effective modal as a pivot for
other modalities. [33] and [34] focus on multimodal information
fusion, but all fusions are completed through attention based
deep network directly that both unimodal and cross-modal fea-
tures are not specifically extracted. [35] does not give attention
on cross-modal features that it receives the best results by taking
context from the other related utterances into account and fusing
trimodal information through hierarchical module. The results
of [33], [34] and [35] are higher that its of [29] obviously. It
is because these three methods put their emphases on unimodal
features extraction and on the contrary, [29] only cares about
cross-modal features. However, unimodal information actually
plays an essential role in multimodal emotion recognition that
cannot be ignored. As a result, our method achieves better
overall recognition accuracy than baselines by considering both
unimodal and cross-modal aspects. In Table III, our method
obtains the best accuracy in sad, excited and average categories.
We are also very close to the best result in frustrated type. The
distance of the results in happy type is because happy and excited
are considered to be the same in method of [38]. View-specific
and cross-view features are both considered in [36], however, the
definition of view in this method is confined to unimodal from
different time nodes instead of cross-modal. Method in [37] put
most efforts on mining contextual relations that trimodal infor-
mation are fused by cascading at the very first step. The lack of
effective features screening making the accuracy of this method
turns out to be the lowest. Our model achieves the best results
on CMU-MOSI and most categories on IEMOCAP because we
unite related utterances as new groups to grasp the context firstly.
Secondly, inner unimodal features are screened and cross-modal
correlations are maximized under specific emotions. Finally, loss
function is optimized to maintain the extracted unimodal and
cross-modal features.
IV. CONCLUSION
In this letter, we propose a novel modularized multimodal
emotion recognition model based on deep canonical correlation
analysis. Both unimodal features extraction and cross-modal
correlations are placed in the same important position. Effective
unimodal features are retained while maximizing the subsequent
cross-modal correlations and related contextual information are
also considered to improve the performances. The experimen-
tal results on two public datasets prove the validity of our
method.
 1902
REFERENCES
[1] P. Jiang, B. Wan, Q. Wang, and J. Wu, “Fast and efficient facial expression
recognition using a gabor convolutional network,” IEEE Signal Process.
Lett., vol. 27, pp. 1954–1958, 2020.
[2] M. Hu, Q. Chu, X. Wang, L. He, and F. Ren, “A two-stage spatiotemporal
attention convolution network for continuous dimensional emotion recog-
nition from facial video,” IEEE Signal Process. Lett., vol. 28, pp. 698–702,
2021.
[3] Y. Tian, J. Cheng, Y. Li, and S. Wang, “Secondary information aware
facial expression recognition,” IEEE Signal Process. Lett., vol. 26, no. 12,
pp. 1753–1757, Dec. 2019.
[4] P. Jiang, G. Liu, Q. Wang, and J. Wu, “Accurate and reliable facial
expression recognition using advanced softmax loss with fixed weights,”
IEEE Signal Process. Lett., vol. 27, pp. 725–729, 2020.
[5] G. Kim, H. Lee, B.-K. Kim, S.-H. Oh, and S.-Y. Lee, “Unpaired
speech enhancement by acoustic and adversarial supervision for speech
recognition,” IEEE Signal Process. Lett., vol. 26, no. 1, pp. 159–163,
Jan. 2019.
[6] S. Yang, Y. Wang, and L. Xie, “Adversarial feature learning and unsuper-
vised clustering based speech synthesis for found data with acoustic and
textual noise,” IEEE Signal Process. Lett., vol. 27, pp. 1730–1734, 2020.
[7] R. Nawaz, K. H. Cheah, H. Nisar, and V. V. Yap, “Comparison of different
feature extraction methods for eeg-based emotion recognition,” Biocybern.
Biomed. Eng., vol. 40, no. 3, pp. 910–926, 2020.
[8] C. Gouveia, A. Tomé, F. Barros, S. C. Soares, J. Vieira, and P. Pinho, “Study
on the usage feasibility of continuous-wave radar for emotion recognition,”
Biomed. Signal Process. Control, vol. 58, 2020, Art. no. 101835.
[9] Y. Jiang, W. Li, M. S. Hossain, M. Chen, A. Alelaiwi, and M. Al-
Hammadi, “A snapshot research and implementation of multimodal infor-
mation fusion for data-driven emotion recognition,” Inf. Fusion, vol. 53,
pp. 209–221, 2020.
[10] V. Rajan, A. Brutti, and A. Cavallaro, “Conflictnet: End-to-end learning
for speech-based conflict intensity estimation,” IEEE Signal Process. Lett.,
vol. 26, no. 11, pp. 1668–1672, Nov. 2019.
[11] Z. Li, D. Wu, F. Nie, R. Wang, Z. Sun, and X. Li, “Multi-view clus-
tering based on invisible weights,” IEEE Signal Process. Lett., vol. 28,
pp. 1051–1055, 2021.
[12] X. Li, M. Chen, and F. Nie, “Locality adaptive discriminant analysis,” in
Proc. Int. Joint Conf. Artif. Intell., 2017, pp. 2201–2207.
[13] S.-Y. Tseng, S. Narayanan, and P. Georgiou, “Multimodal embeddings
from language models for emotion recognition in the wild,” IEEE Signal
Process. Lett., vol. 28, pp. 608–612, 2021.
[14] T.-L. Nguyen, S. Kavuri, and M. Lee, “A multimodal convolutional neuro-
fuzzy network for emotion understanding of movie clips,” Neural Netw.,
vol. 118, pp. 208–219, 2019.
[15] D. Nguyen, K. Nguyen, S. Sridharan, D. Dean, and C. Fookes, “Deep
spatio-temporal feature fusion with compact bilinear pooling for multi-
modal emotion recognition,” Comput. Vis. Image Understanding, vol. 174,
pp. 33–42, 2018.
[16] P. Tzirakis, J. Chen, S. Zafeiriou, and B. Schuller, “End-to-end multi-
modal affect recognition in real-world environments,” Inf. Fusion, vol. 68,
pp. 46–53, 2021.
[17] J. Wang, Z. Ma, F. Nie, and X. Li, “Progressive self-supervised clustering
with novel category discovery,” IEEE Trans. Cybern., early access, 2021,
doi: 10.1109/TCYB.2021.3069836
[18] C. Tang, X. Zhu, X. Liu, L. Wang, and A. Zomaya, “Defusionnet: De-
focus blur detection via recurrently fusing and refining multi-scale deep
features,” in Proc. IEEE/CVF Conf. Comput. Vision Pattern Recognit.,
2019, pp. 2700–2709.
[19] Y. Wang, F. Sun, M. Lu, and A. Yao, “Learning deep multimodal feature
representation with asymmetric multi-layer fusion,” in Proc. 28th ACM
Int. Conf. Multimedia, 2020, pp. 3902–3910.
Authorized licensed use limited to: Amrita Vishwa Vidyapeetham Chennai Campus. Downloaded on January 20,2023 at 16:14:23 UTC from IEEE Xplore. Restrictions apply.
IEEE SIGNAL PROCESSING LETTERS, VOL. 28, 2021
[20] C. Tang et al., “Cross-view locality preserved diversity and consensus
learning for multi-view unsupervised feature selection,” IEEE Trans.
Knowl. Data Eng., early access, 2021, doi: 10.1109/TKDE.2020.3048678
[21] X. Shu, G.-J. Qi, J. Tang, and J. Wang, “Weakly-shared deep transfer
networks for heterogeneous-domain knowledge propagation,” in Proc.
23rd ACM Int. Conf. Multimedia, 2015, pp. 35–44.
[22] J. Tang, X. Shu, Z. Li, G.-J. Qi, and J. Wang, “Generalized deep transfer
networks for knowledge propagation in heterogeneous domains,” ACM
Trans. Multimedia Comput. Commun. Appl., vol. 12, no. 4, pp. 1–22, 2016.
[23] H. Wen, S. You, and Y. Fu, “Cross-modal context-gated convolution
for multi-modal sentiment analysis,” Pattern Recognit. Lett., vol. 146,
pp. 252–259, 2021.
[24] C. Guanghui and Z. Xiaoping, “Multi-modal emotion recognition by
fusing correlation features of speech-visual,” IEEE Signal Process. Lett.,
vol. 28, pp. 533–537, 2021.
[25] L. Chen, K. Wang, M. Wu, W. Pedrycz, and K. Hirota, “K-means
clustering-based kernel canonical correlation analysis for multimodal
emotion recognition,” IFAC-PapersOnLine, vol. 53, no. 2, pp. 10 250–
10254, 2020.
[26] X. Li, M. Chen, F. Nie, and Q. Wang, “A multiview-based parameter free
framework for group detection,” in Proc. AAAI Conf. Artif. Intell., vol. 31,
no. 1, 2017, pp. 4147–4153.
[27] C. Guo and D. Wu, “Discriminative sparse generalized canonical cor-
relation analysis (DSGCCA),” in Proc. Chin. Automat. Congr., 2019,
pp. 1959–1964.
[28] Z. Wang, F. Nie, C. Zhang, R. Wang, and X. Li, “Capped (p)-norm LDA for
outliers robust dimension reduction,” IEEE Signal Process. Lett., vol. 27,
pp. 1315–1319, 2020.
[29] D. Gkoumas, Q. Li, C. Lioma, Y. Yu, and D. Song, “What makes the
difference? an empirical comparison of fusion strategies for multimodal
language analysis,” Inf. Fusion, vol. 66, no. 6, pp. 184–197, 2021.
[30] C. Busso et al., “Iemocap: Interactive emotional dyadic motion capture
database,” Lang. Resour. Eval., vol. 42, no. 4, p. 335, 2008.
[31] A. Zadeh, R. Zellers, E. Pincus, and L.-P. Morency, “Mosi: Multimodal
corpus of sentiment intensity and subjectivity analysis in online opinion
videos,” 2016, arXiv:1606.06259.
[32] K. Zhang, Y. Li, J. Wang, E. Cambria, and X. Li, “Real-time video
emotion recognition based on reinforcement learning and domain knowl-
edge,” IEEE Trans. Circuits Syst. Video Technol., early access, 2021, doi:
10.1109/TCSVT.2021.3072412
[33] M. G. Huddar, S. S. Sannakki, and S. R. Vijay, “Attention-based multi-
modal contextual fusion for sentiment and emotion classification using
bidirectional LSTM,” Multimedia Tools Appl., vol. 80, no. 9, pp. 13059–
13076, 2021.
[34] R. Cao, C. Ye, and H. Zhou, “Multimodel sentiment analysis with self-
attention,” in Proc. Future Technol. Conf., 2021, vol. 1, pp. 16–26.
[35] N. Majumder, D. Hazarika, A. Gelbukh, E. Cambria, and S. Poria,
“Multimodal sentiment analysis using hierarchical fusion with context
modeling,” Knowl.-Based Syst., vol. 161, pp. 124–133, 2018.
[36] A. Zadeh, P. P. Liang, N. Mazumder, S. Poria, E. Cambria, and L.-P.
Morency, “Memory fusion network for multi-view sequential learning,”
in Proc. AAAI Conf. Artif. Intell., 2018, pp. 5634–5641.
[37] D. Hazarika, S. Poria, R. Mihalcea, E. Cambria, and R. Zimmermann,
“Icon: Interactive conversational memory network for multimodal emotion
detection,” in Proc. Conf. Empirical Methods Natural Lang. Process.,
2018, pp. 2594–2604.
[38] S. Yildirim, Y. Kaya, and F. Kılıç, “A modified feature selection method
based on metaheuristic algorithms for speech emotion recognition,” Appl.
Acoust., vol. 173, 2021, Art. no. 107721.