Spontaneous Facial Expression Recognition by

Heterogeneous Convolutional Networks

Xianlin Peng1 , Lei Li1 , Xiaoyi Feng1 , and Jianping Fan2
1 School of Electronics and Information, Northwestern Polytechnical University
e-mail: 397448497@qq.com
2 School of Information Science and Technology, Northwest University
e-mail: pjy@nwu.edu.cn

Abstract— Spontaneous facial expression achieves much at-
tention recently as it has potential applications in the field of
computer vision and pattern recognition. Although the convolu-
tional networks have been applied for recognizing acted facial
expressions and obtained the state-of-the-art performance, the
performance of recognizing spontaneous facial expressions still
needs to be improved. In this paper, a heterogeneous deep
model is presented to recognize spontaneous expressions. The
deep model consists of two types of convolutional networks with
different architectures. To leverage the acted data, these two
deep sub-networks are pre-trained over acted data and then
transferred to the spontaneous data. Experiments have shown the
advantages of the proposed method on the dataset of spontaneous
facial expression.
Keywords— Spontaneous facial expression, Deep learning, Con-
volutional networks, Heterogeneous architecture
I. I NTRODUCTION
Facial expression recognition (FER) is a popular topic in
the fields of computer vision and pattern recognition [1]. For
last three decades, the FER technologies have been applied
to many scenes, such as news analysis and human-computer
interface. An amount of approaches have been presented in the
applications and various visual features for expressions have
been proposed to describe the diversity of expressions. Since
the facial expressions refer to the race, culture, personality,
etc., classifying facial expressions into basic types is still
challenging.
Existing approaches of recognizing facial expressions can be
classified into two categories. The first category concentrates
on the six basic emotions, which are happiness, sadness,
surprise, anger, disgust and fear. Many approaches are de-
voted to recognize these basic expressions over conventional
datasets, such as the MMI facial expression database [2]
and the Japanese Female Facial Expression (JAFFE) database
[3]. The MMI has more than 2,000 images or frames of
expressions and more than 500 expression images of 50
persons. The Japanese Female Facial Expression (JAFFE)
database [3] contains 213 expression images of 10 Japanese
females. The second category focuses on extracting fine-
grained description for facial expression, such as action units
(AUs) [4], neutral [5], [6], [7], [8], wink [6], [7], sleepy
[6], talk [7], scream [8] . The most famous dataset, namely
Cohn-Kanade database [4], contains many action units and
combinations of action units based on the Facial action coding
system (FACS) proposed by Ekman. All the above approaches
refer to the acted expressions, however, the spontaneous facial
expressions are very different from the acted ones, which are
acquired in a highly-controlled environment and mimic the
spontaneous expressions. Therefore, the approaches learned
from the acted expressions suffer from low-performance in
real-world applications.
Many approaches have made their efforts to improve the
generalization of recognizing expressions. Shan et al. [9]
trained their methods on across-datasets. After extracting local
binary pattern (LBP) features from the Cohn-Kanade database,
the Support Vector Machine (SVM) algorithm is then tested
on the MMI database and the JAFFE database, respectively.
On the other hand, the deep learning techniques are utilized
to recognize the acted expressions as the deep features can
learn robust features of expressions. A 3Dconvolutional neural
networks (CNNs) [10] is used to recognize dynamic expres-
sions with strong spatial structural constraints of the action
parts towards the video data. In [11], the conventional deep
models, i.e., CNNs and deep neural networks (DNNs), were
jointly used to learn the appearance and geometric features for
acted expressions. The above deep models can achieve good
performance on video based expression data and focus on the
acted data.
Recently, a spontaneous image dataset leveraging web im-
ages have been constructed [12]. Since these spontaneous-
expression images were labeled by a large amount of web
users and have large diversity, it is difficult to recognize the ba-
sic expressions even with deep models (verified in subsequent
experiments). In this paper, a novel facial expression method
with deep models is proposed to address the problem of
spontaneous FER. As the variations in spontaneous-expression
images are greatly large, a heterogeneous architecture is used
to describe the variations of spontaneous expressions. In the
heterogeneous model, the CNNs based on VGG-Face [13]
and ResNet [14] are employed to automatically extract the
visual features. These two models are then learned jointly with
shared classification layer. To leverage the acted expressions,
we utilize the acted data from Cohn-Kanade database [4] to
pre-train the deep model.
The rest of this paper is organized as follows. The proposed
method is presented in Section II. Then we discuss the
experimental results for algorithm evaluation in Section III.

978-1-5386-3148-5/17/$31.00 © 2017 IEEE 70

 1 Face
Alignment
Fig. 1. The flowchart of recognizing the spontaneous expressions.
At last, Section IV describes our conclusions.
II. P ROPOSED M ETHOD
Fig. 1 illustrates the framework of our proposed deep
recognition method. Given a still image, the feature vector
x = (x1 , x2 , ..., xd )T can be automatically learned by the deep
framework from the aligned images and then classified into six
basic types c = {1, 2, ..., 6}.
A. Face Alignment
Although the web images have been cropped by the Viola-
Jones face detection algorithm, the face images need to be
aligned to reduce the influence of expression-unaware facial
regions. In this paper, we use an effective method to align the
facial images, which are shown in Fig. 2.
Eye
Localization
Original
Image
Five keypoints
Localization
Aligned
Image
Automatic
Alignment
Fig. 2. The face alignment for expressions.
First, an accurate eye localization method [15] is used to
detect the positions of eyes for active shape model (ASM).
The conventional projection algorithm is employed to obtain
the candidate position. Then the positions of eyes and gray
values are used as the features and further determined to be the
final position by a SVM model. The SVM model was trained
on the face image dataset, namely YALEB dataset [16]. Using
the fast learning based model, the initial points for ASM model
can be obtained.
Second, the ASM model is used to detect the five keypoints
(landmarks) in facial regions, i.e., the centers of eyes, the the
tip of the nose and the corners of the mouth. Based on the
trained model used in [17], we merely select five-points ASM
model for facial expression in this context. With two initial
978-1-5386-3148-5/17/$31.00 © 2017 IEEE
2 Heterogeneous 3 Automatic
Convolutional Networks Recognition
points, the ASM model predicts the accurate positions for the
facial images.
Finally, all the images in the spontaneous dataset are aligned
to the first images with the five landmarks. The alignment of
ith image is calculated by
( )T
s · cosθ · xik − s · sinθ · yik − tix
z′ i = · zi
s · sinθ · xik + s · cosθ · yik − tiy (1)
zi = (xi1 , yi1 , . . . , xik , yik , . . . , xi5 , yi5 )T
where ti = (tix , tiy )T represents the translation vector of
shape zi along the axis x and y. θ can be calculated by the eye
positions of two images. From Fig. 2, it can be seen that the
aligned image has less regions than the original image. Before
performing the convolution operations, the raw images would
be aligned and cropped to the size 224 × 224 for extracting
the features.
B. Heterogeneous Convolutional Networks
The heterogeneous convolutional networks (HCN) contains
two heterogeneous layers: (1) VGG-Face based layers; (2)
ResNet based layers. The Fig. 3 shows our proposed archi-
tecture, which can extract the deep features automatically.
VGG-Face. Conventional VGG-Face model [13] is trained on
the data with size of 1 million images. Thus, we propose a new
expression recognition model based on pre-trained model in
this work. The VGG-face model is initially designed for face
recognition [13]. We utilize the deep model to pre-train our
model and change the final layer for expression recognition
problem.
To adapt the VGG-face model for our facial expression
recognition problem, we fine-tune the model using the con-
structed natural expression dataset. As in the problem of facial
expression recognition, we have six classes (i.e., six basic
expressions), we change the output of the last full connected
layer from 2622 to 6 (2622 is the number of the subjects used
for face recognition). After making these changes, we retrain
the deep model by freezing certain layers in the CNN model
and learn the weights and biases of the unfrozen layers. To
select which layers we should learn we repeat the experiment
with different frozen layers.
ResNet. Standard ResNet model [14] is trained on the data
of ImageNet [] and used for image classification. So, we
propose a new expression recognition model based on pre-
trained model in this work.
The standard ResNet mostly has 3×3 filters in each convolu-
tional layers. Following the configure for convolutional layers
71
 VGG-face model
濷 濷
224灤224 濷 濷
Conv Layer ReLU
Heterogeneous Convolutional Networks
Resnet model
Input aligned
face image
濷 濷
Conv Layer
Conv Layer
Fig. 3. The architecture of heterogeneous convolutional networks.
[], the downsampling is performed directly by convolutional
layers that have a stride of 2. At the end of convolutional
layers, the global average pooling layer is used to reduce the
dimensionality of features. In the standard, a 1000-way fully-
connected (FC) layer with softmax is employed to predict the
classes while we use 6-way FC layer to recognize expressions.
The total number of weighted layers is 50 in this context.
In convolutional layers, the shortcut connections between
each two layers are inserted for obtaining the residual rep-
resentations. Through the convolution without downsampling
in these two adjacent layers, the identity shortcuts can be
directly employed and the identity mapping is performed. In
the residual addition, the element-wise addition is performed
on two feature maps, channel by channel.
C. Classification and Learning
As shown in Fig. 3, the common classification layer with
fully-connections are used to fuse the predictions from two
heterogeneous networks. The Softmax function is employed
to predict the probability of recognizing expressions for two
deep models. It is defined by
exp(wcT v)
yc = ∑C (2)
T
c=1 exp(wc v)
where yc is the predicted probability of cth expression, and
v denotes the output feature vector of last pooling layer.
The MAX operation is performed on the outputs of two
deep models and the larger probability is used to predict the
expressions.
When learning the parameters of deep models, we employ
pre-trained models to avoid over-fitting before fine tuning the
deep models for FER problem. Two deep models, i.e., VGG-
Face based CNN and ResNet based CNN, are pre-trained on
the Cohn-Kanade database [4]. Then we fin tune the models
in the spontaneous dataset. The logloss function is used to
978-1-5386-3148-5/17/$31.00 © 2017 IEEE
Prob
Global
濷 Pooling
濷 ‫ܺܣܯ‬
angry
Pool disgust
FC
fear
happy
sadness
surprise
Global
濷 Pooling
濷
Conv Layer Prob
FC
learn the parameters by the stochastic gradient decent (SGD)
algorithm.
In SGD procedure for parameter learning, the momentum
is set to 0.9 and weight decay 0.0005. The stopping criterion
for SGD is set to 10−4 for iterations. The learning rate is set
to 10−3 in the beginning and will be multiplied with damping
factor 0.5 when all mini-batches are traversed and re-allocated
randomly.
III. E XPERIMENTS
A. Experimental Setup
In our experiment, the spontaneous facial expression
database [12] with 1648 images is employed to evaluate
the performance. Totally, all images are classified into six
basic categories (i.e., angry, disgust, fear, happy, sadness,
surprise) and about 300 images can be used for each kind
of expressions. In this dataset, the spontaneous expressions
are obtained from different persons in diverse cultures and
countries. The images are of different resolutions and most of
them are frontal images.
To assess the effectiveness of our proposed deep model
on the spontaneous expression database, we conduct exper-
iments evaluating the performance of the proposed model
and comparing the results against those of using traditional
shallow models and deep models. Since the Gabor feature and
LBP feature [18] have been used in FER problem, these two
approaches are used as the shallow models. The conventional
VGG-Face model and ResNet model are used as the deep
models.
The recognition accuracy is utilized to evaluate all ap-
proaches on the spontaneous dataset. The entire dataset is
randomly splitted into the training set and test set. In each
category of expressions, the spontaneous images are assigned
the training set and test set with fixed ratio of 90% : 10%.
72
 Table 1. The recognition accuracy of all approaches for comparison on the spontaneous-expression dataset.
Approach Anger Disgust
Gabor+KNN 0.567 0.408
LBP+SVM 0.700 0.400
AU+ KNN 0.721 0.440
DCNN [12] 0.672 0.754
ResNet [14] 0.654 0.552
Our proposed method (HCN) 0.723 0.792
B. Experimental Results
The recognition rates obtained using different methods are
shown in Table 1. As one can expect, the performance of
traditional LBP and Gabor features is much lower than that
of the learned features with the deep model. This confirms
the relative sensitivity of traditional LBP and Gabor feature
to diverse expressions. In most configures, the deep method
outperforms the shallow models, while in some cases the
shallow models are competitive with deep models. That might
be due to the insufficient samples for large variations in these
expressions.
The conventional CNNs, i.e., VGG-Face [12] and ResNet
[14], are used to compare with our proposed method. As
shown in Table 1, our proposed method outperform other
two deep models. It is worth noting that the ResNet cannot
outperform the VGG-Face in most configures for the problem
of FER. Although the ResNet achieved better performance
than VGG-series models for image classification, the less
number of samples in the problem of FER still limit the
learning its ability compared to ImageNet dataset. Through the
combination of two heterogeneous datasets, the performance
can be improved for recognizing the spontaneous expressions.
IV. C ONCLUSION
A heterogeneous deep architecture is developed to spon-
taneous FER. In the heterogeneous model, the CNNs based
on VGG-Face and ResNet are employed to automatically
extract the visual features. These two models are then learned
jointly with shared classification layer. To leverage the acted
expressions, we utilize the acted data to pre-train the deep
model. Experimental results assessed the effectiveness of our
proposed approach which outperformed state-of-the-art meth-
ods.
R EFERENCES
[1] S.Z. Li and A.K. Jain, Handbook of Face Recognition, Springer
Publishing Company, Incorporated, 2nd edition, 2011.
[2] M. Pantic, M. Valstar, R. Rademaker, and L. Maat, “Web-based database
for facial expression analysis,” in IEEE International Conference on
Multimedia and Expo (ICME), 2005, pp. 1–5.
[3] M. Kamachi, M. Lyons, and J. Gyoba, “The japanese female facial
expression (jaffe) database,” 1998, http://www.kasrl.org/jaffe.html.
[4] T. Kanade, J. F. Cohn, and Y. Tian, “Comprehensive database for
facial expression analysis,” in Fourth IEEE International Conference
on Automatic Face and Gesture Recognition (FG), 2000, pp. 46–53.
978-1-5386-3148-5/17/$31.00 © 2017 IEEE
Spontaneous Expression
Fear Happy Sadness Surprise Average
0.300 0.667 0.533 0.308 0.464
0.575 0.733 0.633 0.508 0.593
0.564 0.701 0.633 0.617 0.613
0.755 0.983 0.908 0.936 0.834
0.738 0.750 0.892 0.823 0.788
0.762 0.971 0.935 0.942 0.874
[5] M. Lyons, S. Akamatsu, M. Kamachi, and J. Gyoba, “Coding facial
expressions with gabor wavelets,” in IEEE International Conference on
Automatic Face and Gesture Recognition (FG), 1998, pp. 200–205.
[6] P. N. Belhumeur, J. P. Hespanha, and D. Kriegman, “Eigenfaces vs.
fisherfaces: recognition using class specific linear projection,” IEEE
Transactions on Pattern Analysis & Machine Intelligence, vol. 19, no.
7, pp. 711–720, 2013.
[7] T. Sim, S. Baker, and M. Bsat, “The cmu pose, illumination, and
expression (pie) database,” in IEEE International Conference on
Automatic Face and Gesture Recognition (FG), 2002, pp. 46 – 51.
[8] A. M. Martinez, “The ar face database,” Cvc Technical Report, vol. 24,
1998.
[9] C. Shan, S. Gong, and P.W. Mcowan, “Facial expression recognition
based on local binary patterns: A comprehensive study,” Image & Vision
Computing, vol. 27, no. 6, pp. 803–816, 2009.
[10] M. Liu, S. Li, S. Shan, R. Wang, and X. Chen, “Deeply learning
deformable facial action parts model for dynamic expression analysis,”
in Asian Conference on Computer Vision (ACCV), 2014, pp. 143–157.
[11] H. Jung, S. Lee, J. Yim, S. Park, and J. Kim, “Joint fine-tuning in deep
neural networks for facial expression recognition,” in IEEE International
Conference on Computer Vision (ICCV), 2015, pp. 2983–2991.
[12] X. Peng, Z. Xia, L. Li, and X. Feng, “Towards facial expression
recognition in the wild: A new database and deep recognition system,”
in IEEE Conference on Computer Vision and Pattern Recognition
Workshops (CVPRW), 2016, pp. 1544–1550.
[13] O. M. Parkhi, A. Vedaldi, and A. Zisserman, “Deep face recognition,”
in British Machine Vision Conference (BMVC), 2015.
[14] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for
image recognition,” in IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), 2016, pp. 770–778.
[15] Z. Xia, W. Zhang, F. Tan, X. Feng, and A. Hadid, “An accurate eye
localization approach for smart embedded system,” in International
Conference on Image Processing Theory (IPTA), 2016, pp. 1–5.
[16] Kuang Chih Lee, Jeffrey Ho, and David J Kriegman, “Acquiring
linear subspaces for face recognition under variable lighting,” IEEE
Transactions on Pattern Analysis & Machine Intelligence, vol. 27, no.
5, pp. 684–98, 2005.
[17] Z. Xia, X. Feng, J. Peng, X. Peng, and G. Zhao, “Spontaneous micro-
expression spotting via geometric deformation modeling,” Computer
Vision & Image Understanding, vol. 147, no. C, pp. 87–94, 2016.
[18] T. Ojala, M. Pietikainen, and T. Maenpaa, “Multiresolution gray-scale
and rotation invariant texture classification with local binary patterns,”
IEEE Transactions on Pattern Analysis & Machine Intelligence, vol. 24,
no. 7, pp. 971–987, 2002.
73