Robust Facial Emotion Detection using CNN
Architecture
Aiman Shahid,
Department of Computer Science
University of Engineering and
Technology
Taxila, Pakistan
18-cs-21@students.uettaxila.edu.pk
Abstract— Emotions are a natural and significant
element of human behavior that influence how we
communicate. For a greater understanding of human
behavior, precise analysis, and interpretation of the
emotional content of human facial expressions are
required. AI has gained popularity in almost every
field. Deep learning methods have been investigated as
a series of strategies for achieving robustness and
scalability on new types of data. Deep learning is used
in this article to recognize human emotions through
facial expressions. We have built a novel CNN model
that can predict human expression from an image.
CK+ and FER2013 dataset was used to experiment
with and train a deep convolutional network in this
case. We also applied augmentation techniques like the
K-fold method while working with the CK+ dataset.
Furthermore, we attained an accuracy rate of 89
percent with CK+ and 69 percent with the FER2013
dataset which was far better than many systems. Using
our CNN model, we recognized six different emotions.
When tested with different people under varying
ambient and light conditions, our work has shown
positive outcomes. Results were evaluated using the
confusion matrix and roc curve.
Keywords— facial emotion recognition, deep learning-
based facial emotion recognition, classification, CNN, k-
fold technique.
I. INTRODUCTION
Facial expressions are key components of human
communication because they help us perceive
others' intentions. People use facial expressions and
verbal tones to infer other people's emotional states,
such as joy, sadness, and aggression. Verbal
components transmit one-third of human
communication, whereas nonverbal components
convey two-thirds, according to different research
[1,2]. Facial expressions are one of the most
important nonverbal components in social
communication because they contain emotional
significance. As a result, it's not a bit surprising that
face emotion research has gotten a lot of attention
in recent decades, with applications ranging from
Hamad Rizwan
Department of Computer Science,
University of Engineering and
Technology
Taxila, Pakistan
18-cs-40@students.uettaxila.edu.pk
visual processing sciences to affective computing
and computer animations [2].
Automatic facial emotion recognition (FER) (An
expanded variant of the acronym FER is used in
each study, such as facial emotion recognition and
facial expression recognition.) is a popular topic.
Because this study focuses on the broad
characteristics of facial emotion recognition, the
word FER refers to facial emotion recognition in
this work.) With the fast growth of artificial
intelligence approaches, such as human-computer
interface (HCI) [3,4], virtual reality (VR) [5,6],
augmented reality (AR) [6, advanced driver
assistant systems (ADASs) [7], and entertainment
[8,9], has also been expanding recently. Although
many sensors for FER inputs such as
electromyography (EMG), electrocardiogram
(ECG), electroencephalograph (EEG), and camera
may be used, a camera has been the most successful
sensor that detects since it gives the most useful
information for FER and does not require
maintenance.
In this paper, we present a deep learning-based
emotion detection model based on facial
expressions from photographs. Unlike previous
applications, the system maintains high accuracy
outcomes in a variety of environments and models.
In our experiment, we developed a novel CNN
model. In our experiment, we used two datasets:
FER2013 and CK+. The CNN model was then
trained using both datasets. Furthermore, the
accuracy of the produced CNN's was checked by
comparing them to test data.
II. LITERATURE REVIEW
Deep learning has addressed issues in a variety of
fields [9], while previous techniques involved
manually gathering data and making decisions
based on a set of rules. The sensory mistake is
highly likely to occur in the earlier approaches, and
decisions are made based on a human-built set of
rules, which may or may not be always right.
Researchers have used a variety of approaches to
classify human face expression, including support
vector machines [10], principal component analysis
[11], and convolutional neural networks [12, 13].
The majority of real-time emotion detection
research has focused on visuals.
When the studies in the literature on facial emotion
identification with computer vision techniques are
evaluated over the last several years, they indicate a
considerable increase in the development of
computer technology.
Even though there are several applications for facial
recognition and detection. Breuer and Kimmel [14]
employed visual approaches to interpret a CNN
model for emotion identification that was trained
with a range of datasets. They put CNN to the test
on datasets for face emotion detection as well as
several facial emotion recognition applications.
Jung et al. [15] devised a method that combined
two forms of CNNs. One of them is the
identification of geometric aspects of facial parts,
while the other is feature extraction from visual
data.
Liu et colleagues [16] categorize distinct facial
expressions using the FER2013 dataset and a two-
layer CNN. They also compared it to four other
existing models, finding that the suggested model had
a test accuracy of 49.8%. S Suresh et al [17] created a
Deep Neural Network-based sign language
recognition system that distinguishes six distinct sign
languages (DNN). When two models with different
optimizers (Adam and SGD) are compared, it is
discovered that the model with the Adam optimizer is
more accurate. K. Bouaziz et al [18] have exhibited
an analytics approach that incorporates picture
recognition tools and methodologies. The suggested
model uses CNN architecture to classify different
types of handwriting. Automatic facial expression
recognition was achieved in [19] by extracting
features using wavelets modification and classifying
emotions using the K-nearest neighbors (KNN)
algorithm.
Because a person's face often has many traits,
scientists used principal component analysis (PCA)
to identify facial features.
Krithika et al. [20] employed the Voila Jones
algorithm and local binary patterns (LBP) to
recognize a face, facial expression, eye, and head
movement to determine the learner's interest,
boredom, and so on during learning. In comparison to
other modern systems, machine learning and deep
learning algorithms be more efficient in terms of
implementation and outcomes [21].
CNN can not only recognize the face and facial
expressions, but it can also be fine-tuned to detect
certain areas of the face rather than the entire face.
Furthermore, on numerous datasets, including FER-
2013, CK+, FERG, and JAFFE [22] [23] [24], the
CNN model outperforms previous approaches.
Even though there has been a lot of work done on
emotion recognition, it has all been focused on
discovering the fundamental emotion, but the
complex emotions that humans confront during the
learning process have been neglected.
III. PROBLEM STATEMENT
In the fields of computer vision and artificial
intelligence, facial emotion recognition plays a
critical role. Emotion is one of the fundamental
expressions of human beings. It is an integral part
of our nonverbal communication. It helps in
determining a person's ideas, behavior, and feelings.
Based on facial expressions, we provide the
findings of recognizing five emotional states
(neutral, Happy, sad, anger, and fear). To predict
the grayscale picture of a person’s face, which
emotion the facial expression conveys. Our
evaluation metric will be the accuracy for each
emotion (fraction of correctly classified images)
[12], supplemented by confusion matrix which
highlights which emotions are better recognized
than others. Our system would be to understand a
face and its characteristics and then make a
weighted assumption of the identity of the person.
 Input: 48 by 48 grayscale image of a face
 Output: Emotion conveyed by facial
expression
 IV. REFERENCES
[1] Mehrabian A. Communication without words.
InCommunication theory 2017 Sep 4 (pp. 193-200).
Routledge.
[2] Kaulard, K., Cunningham, D.W., Bülthoff, H.H. and
Wallraven, C., 2012. The MPI facial expression database—a
validated database of emotional and conversational facial
expressions. PloS one, 7(3), p.e32321.
[3] Sandoval, Francisco, Alberto Prieto, Joan Cabestany, and
Manuel Graña, eds. Computational and Ambient
Intelligence: 9th International Work-Conference on Artificial
Neural Networks, IWANN 2007, San Sebastián, Spain, June
20-22, 2007, Proceedings. Vol. 4507. Springer, 2007.
[4] Bartneck, C. and Lyons, M.J., 2007, July. HCI and the face:
Towards an art of the soluble. In International Conference on
Human-computer Interaction (pp. 20-29). Springer, Berlin,
Heidelberg.
[5] Hickson, S., Dufour, N., Sud, A., Kwatra, V. and Essa, I.,
2019, January. Eyemotion: Classifying facial expressions in
VR using eye-tracking cameras. In 2019 IEEE Winter
Conference on Applications of Computer Vision (WACV)
(pp. 1626-1635). IEEE.
[6] Chen, C. H., Lee, I. J., & Lin, L. Y. (2015). Augmented
reality-based self-facial modeling to promote the emotional
expression and social skills of adolescents with autism
spectrum disorders. Research in developmental disabilities,
36, 396-403.
[7] Assari, M. A., & Rahmati, M. (2011, November). Driver
drowsiness detection using face expression recognition. In
2011 IEEE International Conference on Signal and Image
Processing Applications (ICSIPA) (pp. 337-341). IEEE.
[12] Chen, X., Yang, X., Wang, M., & Zou, J. (2017, May).
Convolution neural network for automatic facial expression
recognition. In 2017 International conference on applied
system innovation (ICASI) (pp. 814-817). IEEE.
[13] Alizadeh, S., and A. Fazel. "Convolutional neural networks
for facial expression recognition. arXiv 2017." arXiv preprint
arXiv:1704.06756 (2017).
[14] Breuer, Ran, and Ron Kimmel. "A deep learning perspective
on the origin of facial expressions." arXiv preprint
arXiv:1705.01842 (2017).
[15] Jung H, Lee S, Yim J, Park S, Kim J. Joint fine-tuning in
deep neural networks for facial expression recognition.
InProceedings of the IEEE international conference on
computer vision 2015 (pp. 2983-2991).
[16] Modi, Shrey, and Mohammed Husain Bohara. "Facial
Emotion Recognition using Convolution Neural Network." In
2021 5th International Conference on Intelligent Computing
and Control Systems (ICICCS), pp. 1339-1344. IEEE, 2021.
[17] Solanki, Shivangi. "DEEP CONVOLUTIONAL NEURAL
NETWORKS FOR FACIAL EMOTION RECOGNITION."
Turkish Journal of Physiotherapy and Rehabilitation 32: 3.
[18] Zhou, F., Fan, W., Sheng, Q. and Tao, M., 2018, July. Ship
detection based on deep convolutional neural networks for
polsar images. In IGARSS 2018-2018 IEEE International
Geoscience and Remote Sensing Symposium (pp. 681-684).
IEEE.
[19] Ou, J., 2012. Classification algorithms research on facial
expression recognition. Physics Procedia, 25, pp.1241-1244.
[8] Zhan, C., Li, W., Ogunbona, P., & Safaei, F. (2008). A real-
time facial expression recognition system for online games.
International Journal of Computer Games Technology, 2008.
[9] Semwal, V. B., Mondal, K., & Nandi, G. C. (2017). Robust
and accurate feature selection for humanoid push recovery
and classification: deep learning approach. Neural
Computing and Applications, 28(3), 565-574.
[10] Chen, L., Zhou, C. and Shen, L., 2012. Facial expression
recognition based on SVM in E-learning. Ieri Procedia, 2,
pp.781-787.
[11] Ren, X.D., Guo, H.N., He, G.C., Xu, X., Di, C. and Li, S.H.,
2016, June. Convolutional neural network based on principal
component analysis initialization for image classification. In
2016 IEEE first international conference on data science in
cyberspace (DSC) (pp. 329-334). IEEE.
[20] Krithika, L.B. and GG, L.P., 2016. Student emotion
recognition system (SERS) for e-learning improvement based
on learner concentration metric. Procedia Computer Science,
85, pp.767-776.
[21] Revina, I.M. and Emmanuel, W.S., 2018. A survey on human
face expression recognition techniques. Journal of King Saud
University-Computer and Information Sciences.
[22] Minaee, Shervin, Mehdi Minaei, and Amirali Abdolrashidi.
"Deep-emotion: Facial expression recognition using
attentional convolutional network." Sensors 21, no. 9 (2021):
3046.
[23] Sun, A., Li, Y.J., Huang, Y.M. and Li, Q., 2017, September.
Using facial expression to detect emotion in e-learning
system: A deep learning method. In International Symposium
on Emerging Technologies for Education (pp. 446-455).
Springer, Cham.
[24] Zadeh, M.M.T., Imani, M. and Majidi, B., 2019, February.
Fast facial emotion recognition using convolutional neural
networks and Gabor filters. In 2019 5th Conference on
Knowledge Based Engineering and Innovation (KBEI) (pp.
577-581). IEEE.