International Journal of Advances in Electronics and Computer Science, ISSN: 2393-2835 Volume-5, Issue-3, Mar.

-2018
http://iraj.in
FACIAL EXPRESSION RECOGNITION USING CNN: A SURVEY
1
RAVICHANDRA GINNE, 2KRUPA JARIWALA
1
M.Tech student, Department of Computer Engineering, SVNIT, Surat
2
Assistant Professor, Department of Computer Engineering, SVNIT, Surat
E-mail: 1ravichandraginne@yahoo.com, 2knj@coed.svnit.ac.in

Abstract - Facial expression recognition (FER) has become an active research area that finds a lot of applications in areas
like human-computer interfaces, human emotion analysis, psychological analysis, medical diagnosis etc. Popular methods
used for this purpose are based on geometry and appearance. Deep convolutional neural networks (CNN) have shown to
outperform traditional methods in various visual recognition tasks including Facial Expression Recognition. Even though
efforts are made to improve the accuracy of FER systems using CNN, for practical applications existing methods might not
be sufficient. This study includes a generic review of FER systems using CNN and their strengths and limitations which help
us understand and improve the FER systems further.

Keywords - CNN, FER, feature map, ReLU, MLP, BN.

I. INTRODUCTION weights using back propagation and optimization of

People have been trying to build artificial intelligent
(AI) systems that are equivalent to humans since a
long time. The increased availability of
computational power and data for training have
helped for developing machine learning techniques
and fast learning machines. Among all, deep learning
is widely considered as a promising technique to
make intelligent machines. Facial expression
recognition is the process of identifying human
emotion based on facial expressions. While humans
are naturally capable of understanding the emotions,
it stayed as a challenging task for machines. Facial
expression recognition process contains features
extraction and classification as shown in Fig. 1.
errors.
II. CONVOLUTION NEURAL NETWORKS
CNN are biologically-inspired variants of multi-layer
perceptron (MLP) networks. They use an architecture
which is particularly well suitable to classify images.
The connections between the layers and the weights
associated with some form subsampling results in
features that are invariant to translation for
classifying images. Their architecture makes
convolutional networks fast to train.
A. Architecture of CNN
A CNN contains input layer, multiple hidden layers
and an output layer. Multiple Convolutional layers,
sub-sampling (pooling) layers, normalization layers
and fully connected layers are treated as hidden
layers. Fig.2 shows the architecture of a 5-layered
CNN.

Fig.1 Facial Expression Recognition System

Facial Expression Recognition system is a

classification task. Classifier takes a set of features
that are retrieved from the input image as input.
These features describe the facial expression.
Choosing a good feature set, efficient learning
technique and a diversified database for training are
the important factors in classification. Feature set
should contain.
CNN is a combination of deep learning technology
and artificial neural networks. A massive
development in deep learning and applying CNN for
Fig. 2Architecture of a 5-layered CNN [4]
a classification problem has attained a great success
[1,2,3]. The success comes from the fact that feature
extraction and classification can be performed  In convolutional layer, input image is convolved
simultaneously on them. Critical features are with kernel(s) to produce feature map(s) or
extracted by deep learning methods by updating the activation map(s).
 Pooling reduces the dimensionality of each
feature map but retains the most important

Facial Expression Recognition using CNN: A Survey

13
International Journal of Advances in Electronics and Computer Science, ISSN: 2393-2835 Volume-5, Issue-3, Mar.-2018
http://iraj.in
information. It partitions the image into non-
overlapping regions, and each region is
subsampled (down-sampled) by a non-linear
function such as maximum, minimum or
average.Max pooling is used as the most
commondown-sampling functionthatgives
maximum value for each region as output.
 In Rectified Linear Units (ReLU) layer an
activation function f(x) = max (0, x) is applied
element wise. ReLU introduces non-linearity in
the network. Other functions used to increase
non-linearity are hyperbolic tangent, sigmoid etc.
 The fully connected layer of CNN is located after
several convolutional and pooling layers andit is
a traditional multi-layer perceptron (MLP). All
neurons in this layer are fully connected to all
activations in the previous layer.
 In the loss layer, different loss functions suitable
for different tasks are used. A Softmax loss
function is used for classification of an image
into multiple classes.

III. CNN FOR FACIAL EXPRESSION

RECOGNITION

In recent years various architectures and models of

CNN are created and used for facial expression
recognition. This research includes a summery review
of researches related to facial expression recognition
using CNN.

A. Deep CNN for FER

AyşegülUçar [5] proposed a CNN model with 10
layers for FER as shown in Fig.3. In their
architecture, a convolutional layer with kernel size 5
* 5, stride 1 and pad 2 was applied. Next a max
pooling layer with kernel size 3 * 3, stride 2 and pad
1 is applied. This process is repeated 3 times with
different strides and pads as shown in Fig. 3.
Followed by a convolutional layer with kernel size 2
* 2, stride 1 and pad 0 is applied which is followed by
another convolution layer of kernel size 1 * 1, stride 1
and pad 0. Finally a fully connected layer is added to Fig.3 Pipeline of proposed CNN [5]
the network.
B. Baseline CNN structure for FER
Their model was evaluated on images from JAFFE Minchul Shin et al. [8] have looked into four network
database [6] and CK database [7]. In first evaluation models that are known to perform well in facial
seven facial expressions of various images from the expression recognition to analyse the most efficient
JAFFE database were used for training. In the second network structure. The effect of techniques for pre-
evaluation Cohn-Kanadedatabase which contains processing the input image on the performance was
images of all races was employed for six expressions. also investigated.
They resized the images to 16x16 and the proposed The first network that was looked into was Tang’s
model was trained for 30 epochs with a batch size of CNN structure [9]. It contains a layer for input
40. They used 0.001 as the learning rate for first transformation and three convolutional and pooling
eleven epochs, 0.002 for 13-29 epochs and 0.00001 layers and a fully connected two-layer perceptron at
for the last epoch. Their results shows that their the end.
proposed method outperforms traditional geometric
and appearance based methods with high accuracy.

Facial Expression Recognition using CNN: A Survey

14
International Journal of Advances in Electronics and Computer Science, ISSN: 2393-2835
http://iraj.in
The second network is Yu’s structure [2] which
contains five convolutional layers, three stochastic
pooling layers, and three fully connected layers. The
network has two convolutional layers prior to
pooling, except for the first layer.
The third network that is investigated is Kahou’s
structure [3] which contains three convolutional and
pooling layers followed by a MLP of two layers.
The last one is the Caffe-Image Net structure [10]. It
was designed for the classification of images taken
from the ImageNet dataset into 1000 classes. But the
output nodes were reduced to seven in Baseline CNN
approach. Every convolutional and fully connected
layers of all the four networks is applied with a ReLU
layer and a dropout layer.
Five test sets (FET – 2013, SFEW2.0, CK+, KDEF
and Jaffe) are chosen to perform tests with the four
network structures. For the pre-processing of input
image it was found that the Histogram equalization
method shows the most reliable performance for all
the four networks. It was alsoobserved that Tang’s
network could achieve reasonably high accuracy
forhistogram equalized images compared to the other
network models. Based on this observation they
suggested Tang’s simple network along with
histogram equalization as the baseline model for
carrying out further research.
C. FER with CNN ensemble
Kaung Liu et al. [11] have proposed a model
consisting of many subnets that are structured
differently. Each of these subnets are separately
trained on a training set and combined together.The
output layers are removed and the layers before the
last layer are concatenated together. Finally, this
connected network is trained to output the final
expression labels.
They evaluated their model using Facial Expression
Recognition 2013 (FER – 2013) dataset. It
containsgrayscale image of faces of size 48 x 48
pixels. They divided the dataset intoan 80 % training
set and 20 % validation set. They trained the subnets
separately. Each of the subnets achieved a different
accuracy on the dataset. By combining and averaging
the outputs of CNNs of different structures their
network reports betterment in performance when
compared to single CNN structure.
D. Stacked Deep Convolutional Auto-Encoders for
Emotion Recognition from Facial Expressions
Ariel Ruiz-Garcia et al. [12] have studied the effect of
reducing the number of convolutional layers and pre-
training the Deep CNN as a Stacked Convolutional
Auto-Encoder (SCAE) in a greed layer-wise
unsupervised fashion for emotion recognition using
facial expression. They incorporated Batch
Normalization (BN) for both convolutional and fully
connected layers in their model to accelerate training
and improve classification performance.
Facial Expression Recognition using CNN: A Survey
15
Volume-5, Issue-3, Mar.-2018
In the SCAE emotion recognition model, each
convolutional layer and its subsequent layers: BN,
ReLU and Max Pooling are treated as a single block
and an Auto-Encoder is created for each one of these
blocks. The first Auto-Encoder learns to reconstruct
raw pixel data. The second Auto-Encoder learns to
reconstruct the output of the first encoder and so on.
Finally, the fully connected layer is trained to
associate the output of the last convolutional encoder
with its corresponding label.
Their CNN with BN and the SCAE emotion
recognizers are trained and tested using KDEF [13]
dataset. Applying the pre-training technique to give
weights to the CNN using Auto-Encoders increased
their model’s performance to 92.53% and
dramatically reduced the training time.
OBSERVATIONS
From the above study it is clear that even though
there are numerous approaches for facial expression
recognition still models are being developed
continuously. The reason for this is the accuracy.
Researches are continuously trying to improve the
accuracy of the FER by proposing various
architectures and model. They also adopted some of
the other techniques in their architectures to improve
the accuracy as discussed in Section 1V to account
for the problem of accuracy. Efforts are done to
reduce the training time for better performance.
Ensemble CNNs are used to improve the accuracy of
facial expression recognition.
CONCLUSIONS
This paper includes a study of some of the facial
expression recognition systems based on CNNs.
Different architectures, approaches, requirements,
databases for training/testing images and their
performance have been studied here. Each method
has its own strengths and limitations. This study helps
to understand different kinds of models for facial
expression recognition and to develop new CNN
architectures for better performance and accuracy.
REFERENCES
[1] Ian J. Goodfellow et al. “Challenges in representation
learning: A report on three machine learning contests”, in
Neural Information Processing, ICONIP, 2013, pp. 117-124.
[2] Z. Yu, C. Zhang, "Image based static facial expression
recognition with multiple deep network
learning", Proceedings of the 2015 ACM on International
Conference on Multimodal Interaction, pp. 435-442, 2015,
November.
[3] S. E. Kahou, C. Pal, X. Bouthillier, P. Froumenty, Glehre,
R. Memisevic, M. Mirza, "Combining modality specific
deep neural networks for emotion recognition in
video", Proceedings of the 15th ACM on International
conference on multimodal interaction, pp. 543-550, 2013,
December.
[4] B. Fasel, "Head-pose invariant facial expression recognition
using convolutional neural networks," Proceedings of the
 International Journal of Advances in Electronics and Computer Science, ISSN: 2393-2835
http://iraj.in
Fourth IEEE International Conference on Multimodal
Interfaces, 2002, pp. 529-534.
[5] A. Uçar, "Deep Convolutional Neural Networks for facial
expression recognition," 2017 IEEE International
Conference on INnovations in Intelligent SysTems and
Applications (INISTA), Gdynia, 2017, pp. 371-375.
[6] M. Lyons, J. Budynek, S. Akamatsu, "Automatic
classification of single facial images", IEEE Trans Pattern
Anal Mach Intell., vol. 21, no. 12, pp. 1357-1362, 1999.
[7] T. Kanade, J. F. Cohn, T. Yingli, "Comprehensive database
for facial expression analysis", in: IEEE 4th international
conference on automatic face and gesture recognition, pp.
46-53, 2000.
[8] M. Shin, M. Kim and D. S. Kwon, "Baseline CNN structure
analysis for facial expression recognition," 2016 25th IEEE
International Symposium on Robot and Human Interactive
Communication (RO-MAN), New York, NY, 2016, pp. 724-
729.
[9] Y. Tang, "Deep learning using support vector
machines", CoRR abs/1306.0239, 2013.
Volume-5, Issue-3, Mar.-2018
[10] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R.
Girshick, T. Darrell, "Caffe: Convolutional architecture for
fast feature embedding", Proceedings of the ACM
International Conference on Multimedia, pp. 675-678, 2014,
November.
[11] K. Liu, M. Zhang and Z. Pan, "Facial Expression
Recognition with CNN Ensemble," 2016 International
Conference on Cyberworlds (CW), Chongqing, 2016, pp.
163-166.
[12] A. Ruiz-Garcia, M. Elshaw, A. Altahhan and V. Palade,
"Stacked deep convolutional auto-encoders for emotion
recognition from facial expressions", 2017 International
Joint Conference on Neural Networks (IJCNN), Anchorage,
AK, 2017, pp. 1586-1593.
[13] D. Lundqvist, A. Flykt, A. Ohman, The Karolinska Directed
Emotional Faces — KDEF CD ROM from Department of
Clinical Neuroscience Psycology section, Karolinska
Institute, pp. 3-5, 1998.



Facial Expression Recognition using CNN: A Survey

16