See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/372230358

Emotion recognition for an E-Learning platform using Deep Learning : A

Comparison of Different Approaches

Article · July 2023

CITATIONS READS
0 4

4 authors:

Mohammed Kodad Achraf Zbaida

Université Hassan II de Casablanca Université Hassan II de Casablanca
2 PUBLICATIONS 0 CITATIONS 2 PUBLICATIONS 0 CITATIONS

SEE PROFILE SEE PROFILE

Youssfi Mohamed Abdelmajid Bousselham

Université Hassan II de Casablanca Université Hassan II de Casablanca
190 PUBLICATIONS 827 CITATIONS 17 PUBLICATIONS 106 CITATIONS

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Semantic Web Service Discovery and Composition Framework View project

Contribution to the Development of A Dynamic Circulation Map using the Multi-Agent Approach View project

All content following this page was uploaded by Mohammed Kodad on 09 July 2023.

The user has requested enhancement of the downloaded file.

 Emotion recognition for an E-Learning platform
using Deep Learning : A Comparison of Different
Approaches

KODAD Mohammed ZBAIDA Achraf YOUSSFI Mohamed

Informatique, Intelligence Artificielle et
Cyber Sécurité (L2IAS)
ENSET Mohammedia, Université
Hassan II de Casablanca
Mohammedia, Morocco
kodad.mohammed.me@gmail.com
Informatique, Intelligence Artificielle et
Cyber Sécurité (L2IAS)
ENSET Mohammedia, Université
Hassan II de Casablanca
Mohammedia, Morocco
zbaida.achraf@gmail.com
Informatique, Intelligence Artificielle et
Cyber Sécurité (L2IAS)
ENSET Mohammedia, Université
Hassan II de Casablanca
Mohammedia, Morocco
m.youssfi@enset-media.ac.ma

BOUSSELHAM Abdelmajid
Informatique, Intelligence Artificielle et
Cyber Sécurité (L2IAS)
ENSET Mohammedia, Université
Hassan II de Casablanca
Mohammedia, Morocco
bousselham@enset-media.ac.ma

Abstract— This abstract provides a brief summary of
utilizing deep learning techniques to recognize emotions
through facial expressions. Deep learning models, specifically
Convolutional Neural Networks (CNNs), have gained
significant popularity in accurately analyzing and
understanding emotions from facial images.
By training these models on extensive datasets of labeled
facial expression images, they can effectively learn and extract
crucial features. CNNs excel in capturing spatial details from
facial images.
The application of deep learning-based emotion recognition
extends to various domains, including human-computer
interaction, healthcare, and entertainment. Real-time emotion
detection enables personalized interventions, adaptive content
delivery, and the creation of emotionally captivating
experiences, particularly in the context of E-Learning.
However, challenges remain, such as the limited availability
of diverse and well-annotated datasets and the need to account
for variations in facial expressions across individuals and
cultures.
Nevertheless, the integration of deep learning techniques
for emotion recognition has the potential to revolutionize
human-computer interaction, enhance user experiences, and
foster more empathetic and adaptable technologies across
different fields. Continuous research and advancements in
deep learning approaches are expected to further refine the
accuracy and reliability of emotion recognition systems based
on facial expressions.
Keywords— emotion recognition, e-learning, facial
expression, online machine learning, real-time, Deep Learning,
DataSet
I. INTRODUCTION
TDeep learning has emerged as a powerful technology
that can help machines understand complex patterns in data.
One area where deep learning shows promise is in
recognizing emotions, especially by analyzing facial
expressions. Emotion recognition is important in how
computers interact with humans, and it can greatly improve
e-learning platforms.
E-learning platforms have become popular for their
flexibility and accessibility in education. However, they
often struggle to understand and respond to learners'
emotions, which affects the effectiveness of personalized
and engaging learning experiences.
To address this, researchers are using deep learning
techniques to build emotion recognition systems. These
systems analyze facial expressions using specialized neural
networks like CNNs and RNNs to accurately identify and
classify emotions in real-time. This opens up exciting
possibilities for integrating emotion recognition technology
into e-learning platforms, making learning more
personalized and adaptive.
This paper explores the potential of using deep learning
and facial expression analysis for emotion recognition in
e-learning platforms. We will discuss why emotion
recognition is important for improving human-computer
interaction and its specific benefits in e-learning. We will
also explain the methods and techniques used in deep
learning systems, focusing on how CNNs analyze facial
expressions.
By integrating deep learning-based emotion recognition
into e-learning platforms, we can gain real-time insights into
learners' emotions, allowing for personalized support and
customized content. Deep learning models can also create
engaging learning materials that make the learning
experience more enjoyable.
However, while integrating emotion recognition
technology has many benefits, it's important to balance its
use with respect for privacy and user autonomy. This paper
will also discuss the ethical considerations and challenges
associated with using emotion recognition in e-learning
platforms.

1
 In summary, this paper highlights the potential of deep
learning techniques for emotion recognition and their
application in e-learning platforms. By using facial
expressions to understand learners' emotions, we can
transform the way people interact with online education,
creating a more empathetic, adaptable, and effective
learning environment.
II. RESEARCH PROBLEM:
In e-learning, the way teachers and students
communicate is usually through written messages, which
makes it difficult to express and understand emotions.
Unlike in traditional classrooms, where students can show
their feelings through their words and actions, e-learning
lacks this ability.
Emotions are important in learning because they affect
how motivated and engaged students are, as well as how
well they remember information. That's why it's essential to
include emotions in e-learning systems to make the learning
experience better and improve results. By finding ways for
students to express their emotions and for the system to
recognize those emotions, e-learning can become more like
face-to-face learning and help students feel more connected
to what they're learning.
Researchers have looked into different ways to
recognize emotions, such as by analyzing speech, body
signals, and facial expressions. Facial expressions, in
particular, have shown promise for recognizing emotions in
e-learning. However, most studies have focused on
analyzing pre-recorded data and not on recognizing
emotions in real-time during online learning.
One challenge in e-learning is figuring out if students are
satisfied and engaged. How students feel about their
learning experience affects how motivated they are and how
well they do. But the usual ways of measuring satisfaction,
like feedback forms or surveys, can be subjective and may
not capture the true emotions of students. That's why it's
important to develop more accurate and objective ways to
measure how satisfied and engaged students are in
e-learning.
There's also a lack of research on recognizing emotions
in real-time using facial expressions in e-learning. This is an
area that deserves more attention. Developing a system that
can recognize emotions in real-time using facial expressions
would greatly improve the e-learning experience. This study
aims to fill this research gap by creating an online system
that uses machine learning to recognize emotions from
facial expressions in real-time. The system will be tested
with a group of students in an e-learning environment to see
how well it measures their satisfaction and engagement.
III. RELATED WORKS
BThe study [1] describes an algorithm for real-time
emotion recognition using virtual markers, facial landmarks,
and EEG signals. The study focused on physically disabled
individuals and children with Autism. The algorithm used
CNN and LSTM classifiers to classify six facial emotions
and EEG signals. The study involved fifty-five
undergraduate students for facial emotion recognition and
nineteen for collecting EEG signals. Virtual markers were
placed on the subject's face, and the markers were tracked
using an optical flow algorithm. The distance between the
center of the subject's face and each marker position was
used as a feature for facial expression classification, while
the fourteen signals collected from the EEG signal reader
were used for emotional classification.
The article [2] discusses the importance of facial
recognition in various applications, such as security, identity
verification, and database management systems. The article
presents a deep learning algorithm for accurate facial
recognition and identification, using haar cascade detection
and a convolutional neural network model. The proposed
work includes three objectives : face detection, recognition,
and emotion classification, using OpenCV, Python
programming, and a dataset. An experiment was conducted
to identify the emotions of multiple students, and the results
demonstrate the efficacy of the face analysis system. Finally,
the accuracy of the automatic face detection and recognition
is measured.
They proposed [3] a Convolutional Neural Network
(CNN) based LeNet architecture for facial expression
recognition. First of all, they merged 3 datasets (JAFFE,
KDEF and our custom dataset). Then they trained the LeNet
architecture for emotion states classification. In this study,
they achieved accuracy of 96.43% and validation accuracy
of 91.81% for classification of 7 different emotions through
facial expressions.
They present [4] an approach of Facial Expression
Recognition (FER) using Convolutional Neural Networks
(CNN). This model created using CNN can be used to detect
facial expressions in real time. The system can be used for
analysis of emotions while users watch movie trailers or
video lectures.
They discuss [5] how computer-animated agents and
robots can add a social dimension to human-computer
interaction. Real-time face-to-face communication requires
relying on sensory-rich perceptual primitives rather than
slow symbolic inference processes due to the high level of
uncertainty at this time scale. The system presented in the
paper detects frontal faces and codes them with respect to 7
dimensions in real time. It employs boosting techniques and
SVM classifiers to enhance performance and has been tested
on a dataset of posed facial expressions. The system's
outputs change smoothly over time, providing a potentially
valuable representation to code facial expression dynamics
in a fully automatic and unobtrusive manner.
They compared [6] five different methods for real-time
emotion recognition from facial images, specifically for the
four basic emotions of happiness, sadness, anger, and fear.
Three of the approaches are based on convolutional neural
networks (CNN) and two are conventional methods using
Histogram of Oriented Gradients (HOG) features. The
approaches compared are : AlexNet CNN, Affdex CNN,
FER-CNN, SVM using HOG features, and MLP artificial
neural network using HOG features. The paper presents the
results of testing these methods in real-time on a group of
eight volunteers.
This paper [7] presents an advanced deep learning
technique for emotion prediction through facial expression
2
analysis. The proposed approach employs a two-stage
convolutional neural network (CNN) model. The first CNN
predicts the primary emotion of the input image as happy or
sad, while the second CNN predicts the secondary emotion.
The model was trained on FER2013 and JAFFE datasets
and achieved superior results compared to existing
state-of-the-art methods for emotion prediction from facial
expressions.
This paper [8] addresses the challenging task of
real-time emotion recognition through facial expression in
live video using an automatic facial feature tracker for face
localization and feature extraction. The extracted facial
features are fed into a Support Vector Machine classifier to
infer emotions. The paper presents the results of
experiments evaluating the accuracy of the approach for
various scenarios, including person-dependent and
person-independent recognition. The findings show that the
proposed method is effective in achieving fully automatic
and unobtrusive expression recognition in live video. The
paper concludes by discussing the significance of the
research for affective and intelligent man-machine interfaces
and suggesting possible future improvements.
This paper [9] focuses on the importance of analyzing
users' facial expressions to improve the interaction between
humans and machines. The paper proposes a method for
extracting facial features and recognizing the user's
emotional state that is robust to facial expression variations
among different users. The method extracts facial animation
parameters (FAPs) and uses a novel neuro fuzzy system to
analyze FAP variations both at the discrete emotional space
and in the 2D continuous activation-evaluation space. The
system can further learn and adapt to specific users' facial
expression characteristics using clustering analysis. The
paper reports the experimental results of emotionally
expressive datasets, indicating the good performance and
potential of the proposed approach.
The aim of this study [10] is to develop predictive
models that can classify emotions in real-time from videos
of workshop participants engaging with an educational
robot. We combine the two best generalizing models
(Inception-v3 and ResNet-34) to achieve better prediction
accuracy. To test our approach, we apply the models to
video data and analyze the predicted emotions based on the
participants' gender, activities, and tasks. Statistical analysis
reveals that female participants are more likely to show
emotions in almost all activity types, and happiness is the
most frequently predicted emotion for all activity types,
regardless of gender. Additionally, programming is the
activity type where the analyzed emotions were the most
frequent. These findings highlight the potential of using
facial expressions to improve teaching practices and
understand student engagement.
IV. FOUNDATIONAL CONCEPTS
A. Machine Learning (ML) vs. Deep Learning: A Brief
Comparison
Machine Learning (ML) and Deep Learning are both
subfields of artificial intelligence (AI) that focus on training
algorithms to learn patterns and make predictions or
decisions. While they share similarities, there are key
differences between ML and Deep Learning.
1) Machine Learning :
Machine Learning involves training algorithms to analyze
and interpret data, and make predictions or decisions based
on patterns and statistical models. ML algorithms learn from
labeled data and use features derived from that data to make
predictions on new, unseen examples. ML algorithms can be
broadly categorized into supervised, unsupervised, and
reinforcement learning.
Supervised Learning : In supervised learning, algorithms are
trained using labeled data, where each data point is
associated with a corresponding label or outcome. The goal
is to learn a mapping function that can predict labels for
new, unseen data accurately. Examples of supervised
learning algorithms include linear regression, decision trees,
and support vector machines.
Unsupervised Learning : Unsupervised learning involves
training algorithms on unlabeled data, without any
predefined labels or outcomes. The algorithms learn to
identify patterns, similarities, and structures within the data.
Clustering algorithms and dimensionality reduction
techniques are common examples of unsupervised learning.
Reinforcement Learning : Reinforcement learning involves
training algorithms to make decisions or take actions in an
environment to maximize a cumulative reward signal. The
algorithms learn through trial and error, receiving feedback
from the environment based on their actions. Reinforcement
learning has been successful in applications such as game
playing and robotics.
2) Deep Learning :
Deep Learning is a subset of ML that focuses on training
deep neural networks with multiple layers to automatically
learn hierarchical representations of data. Deep Learning
algorithms are inspired by the structure and function of the
human brain, and they excel at capturing complex patterns
and relationships in large-scale datasets. Deep neural
networks consist of interconnected layers of artificial
neurons (nodes), with each layer extracting increasingly
abstract features from the input data.
Deep Learning architectures, such as Convolutional Neural
Networks (CNNs) for image recognition and Recurrent
Neural Networks (RNNs) for sequential data, have achieved
remarkable performance in various domains, including
computer vision, natural language processing, and speech
recognition. Deep Learning algorithms often require
substantial amounts of labeled training data and powerful
computational resources for training due to their complex
architectures.
3) Key Differences:
Representation and Feature Engineering : In traditional ML,
domain experts often need to manually engineer features
from raw data. Deep Learning, on the other hand,
automatically learns hierarchical representations from the
raw input data, eliminating the need for extensive feature
engineering.
3
Performance and Scalability: Deep Learning algorithms can
achieve state-of-the-art performance in certain tasks when
trained on large amounts of data. ML algorithms may be
more suitable for smaller datasets or when interpretability of
the model is critical.
Computational Requirements: Deep Learning algorithms
require significant computational resources, such as
powerful GPUs, due to their complex architectures. ML
algorithms can often be trained on more modest hardware.
In summary, Machine Learning focuses on training
algorithms to learn patterns and make predictions based on
features derived from data, while Deep Learning utilizes
deep neural networks to automatically learn hierarchical
representations of data. The choice between ML and Deep
Learning depends on the task, available data, computational
resources, and the desired level of interpretability.
B. Recognition of emotion based on facial expressions
Recognition of emotion based on facial expressions is a
fascinating field that focuses on developing methods and
technologies to accurately detect and interpret human
emotions by analyzing facial movements and expressions.
Human faces are incredibly expressive, conveying a wide
range of emotions through subtle changes in muscle
movements, such as smiles, frowns, and raised eyebrows.
Researchers and scientists in this field explore various
approaches to recognize and understand emotions based on
facial expressions. They investigate the underlying
mechanisms of facial expressions, the relationship between
specific facial movements and emotional states, and the
patterns that indicate different emotions.
The aim is to develop computer vision and machine
learning algorithms that can automatically detect and
classify facial expressions to accurately recognize emotions
like happiness, sadness, anger, surprise, fear, and disgust.
These algorithms learn from large datasets of labeled facial
expressions, training models to identify the unique features
and patterns associated with each emotion.
The applications of emotion recognition based on facial
expressions are wide-ranging. In psychology, it can aid in
understanding human behavior, studying emotional
disorders, and improving therapy techniques. In
human-computer interaction, it enables more natural and
empathetic interactions between humans and machines. In
fields like virtual reality and gaming, it enhances immersion
and user experience. Additionally, it has potential
applications in areas like market research, customer
feedback analysis, and security systems.
Researchers employ various techniques in emotion
recognition, including feature extraction from facial
landmarks, deep learning models such as convolutional
neural networks (CNNs), and multimodal approaches that
combine facial expressions with other modalities like voice
and physiological signals.
The ultimate goal is to develop sophisticated systems
that can accurately interpret and respond to human emotions
in real-time, enabling machines to understand and adapt to
human emotional states. This advancement can
revolutionize numerous industries and significantly enhance
our interaction with technology and each other.
Fig. 1. The steps of emotion recognition using facial expression
V. MATERIALS AND METHODS
In this study, we will compare different ways of recognizing
emotions from facial expressions to see which one works
the best. We will look at different techniques and algorithms
used in this field and analyze how well they perform. Based
on this analysis, we will create our own special Neural
Network (NN) that can recognize emotions accurately.
To make our NN better, we will consider things like how we
prepare the data, the methods we use to pick out important
features, and the type of structure we give to the network.
We will also fine-tune the model using advanced methods
and check that it works well in different situations.
By creating our own NN based on what we learn from the
study, we hope to improve the technology used to recognize
emotions. We want to make it more accurate and faster,
which can be useful in many areas like computers
interacting with people, E-Learning, virtual reality, and
healthcare.
In summary, this study involves comparing different
methods of recognizing emotions from facial expressions
and using that knowledge to create our own specialized
Neural Network. Our goal is to improve how well
computers can understand emotions from faces and make
the technology more reliable and effective in the future.
A. Data Set Description
For our research, we will utilize the dataset available at
[DataSET] as the primary dataset to train and test our
models. This dataset contains folders representing
different facial expressions, including Surprise, Anger,
Happiness, Sad, Neutral, Disgust, and Fear.
The dataset is divided into two main folders, Training
and Testing, to facilitate model configuration for
end-users. The training set comprises a total of 28,079
samples, while the testing set contains 7,178 samples.
Each sample consists of grayscale images of faces with
dimensions of 48x48 pixels. The dataset ensures that the
faces are registered automatically, resulting in a more or
4
 less centered face occupying a similar amount of space
in each image.
It's important to note that this dataset was obtained from
the "Challenges in Representation Learning: Facial
Expression Recognition Challenge" competition. The
dataset was prepared by Pierre-Luc Carrier and Aaron
Courville as part of their ongoing research project. They
generously provided a preliminary version of their
dataset to the workshop organizers for use in this
contest.
B. VGG16 Model
The VGG16 [11] model is a convolutional neural
network (CNN) architecture that has been widely used in
various computer vision tasks, including emotion
recognition based on facial expressions. It was
developed by the Visual Geometry Group (VGG) at the
University of Oxford.
The VGG16 model is characterized by its deep structure,
consisting of 16 layers, including 13 convolutional
layers and 3 fully connected layers. The architecture of
VGG16 is known for its simplicity and uniformity,
where the convolutional layers have a small receptive
field (3x3) and are stacked one after another. This design
choice allows the model to learn intricate features by
capturing local patterns in the early layers and then
gradually capturing more complex patterns as the depth
increases.
The input to the VGG16 model is typically an image,
and the network performs a series of convolutional
operations to extract features from the input image. Each
convolutional layer is followed by a rectified linear unit
(ReLU) activation function, which introduces
non-linearity into the model. The intermediate outputs
from the convolutional layers are downsampled using
max-pooling layers to reduce spatial dimensions while
preserving the most salient features.
After the convolutional layers, the extracted features are
flattened and passed through a series of fully connected
layers. These fully connected layers further process the
features and eventually produce the output predictions.
In the case of emotion recognition, the output layer of
the VGG16 model is typically configured to have
multiple units corresponding to different emotion classes
(e.g., happy, sad, angry, etc.). The final output is
obtained using a softmax activation function, which
produces a probability distribution over the different
emotion classes.
To train the VGG16 model for emotion recognition, a
labeled dataset of facial expression images is used. The
model is optimized using methods such as stochastic
gradient descent (SGD) or Adam optimizer, and the loss
function used is typically categorical cross-entropy,
which measures the difference between the predicted
probabilities and the true labels. The model is trained on
the labeled dataset iteratively, adjusting the network's
weights to minimize the loss and improve its prediction
accuracy.
The VGG16 model has demonstrated strong
performance in various computer vision tasks, including
emotion recognition based on facial expressions. Its deep
architecture allows it to learn complex patterns and
features from images, enabling accurate recognition of
different emotions. However, it's worth noting that the
VGG16 model can be computationally intensive and
may require substantial computational resources for
training and inference, particularly when dealing with
large-scale datasets.
In summary, the VGG16 model is a powerful CNN
architecture commonly employed in emotion recognition
based on facial expressions. Its deep structure, along
with its ability to learn intricate features, makes it
suitable for capturing meaningful representations from
images and accurately predicting different emotions.
C. VGG19 Model
The VGG19 [12] model is an extension of the VGG16
model and is also a convolutional neural network (CNN)
architecture commonly utilized in various computer
vision tasks, including emotion recognition based on
facial expressions.
Similar to VGG16, VGG19 was developed by the Visual
Geometry Group (VGG) at the University of Oxford. It
is called VGG19 because it consists of 19 layers,
including 16 convolutional layers and 3 fully connected
layers. The architecture of VGG19 shares similarities
with VGG16, but it has a deeper structure, allowing it to
capture more complex patterns and features from
images.
VGG19 follows a similar design principle to VGG16,
where it utilizes small receptive fields (3x3) in its
convolutional layers stacked on top of each other. This
design choice allows the network to learn rich and
detailed features by applying multiple convolutional
operations sequentially. The convolutional layers are
typically followed by rectified linear unit (ReLU)
activation functions to introduce non-linearity into the
model.
Max-pooling layers are used after each set of
convolutional layers to downsample the feature maps,
reducing spatial dimensions while preserving the most
salient information. The intermediate feature maps are
then passed through a series of fully connected layers,
which further process the extracted features and generate
the final output predictions.
In the context of emotion recognition based on facial
expressions, the VGG19 model's input is an image of a
face, and it learns to extract discriminative features from
facial regions to distinguish between different emotions.
The output layer of VGG19 is configured to have
multiple units representing the possible emotion classes.
5
 A softmax activation function is commonly used to
produce a probability distribution over the emotion
classes, allowing the model to make predictions about
the dominant emotion in the input facial expression.
Training the VGG19 model for emotion recognition
typically involves using a labeled dataset of facial
expression images. The model is optimized using
algorithms such as stochastic gradient descent (SGD) or
Adam optimizer, and the categorical cross-entropy loss
function is commonly employed to measure the
discrepancy between the predicted probabilities and the
true emotion labels. Through iterative training, the
VGG19 model adjusts its weights to minimize the loss
and improve its ability to accurately classify emotions.
The VGG19 model's increased depth compared to
VGG16 enables it to capture more intricate and nuanced
features, potentially leading to improved performance in
emotion recognition tasks. However, it's important to
note that the additional depth also increases the model's
complexity and computational requirements, demanding
more computational resources during training and
inference.
In summary, the VGG19 model is an extension of the
VGG16 architecture widely utilized in emotion
recognition tasks based on facial expressions. Its deeper
structure enables it to capture more complex patterns,
allowing for better discrimination among different
emotions. By leveraging the convolutional and fully
connected layers, the VGG19 model can effectively
extract features from facial images and provide accurate
predictions for various emotion classes.
D. ResNet50V2 Model
The ResNet50V2 [13] model is a convolutional neural
network (CNN) architecture that has been widely
employed in various computer vision tasks, including
emotion recognition based on facial expressions.
ResNet50V2 is an extension of the original ResNet
architecture introduced by Microsoft Research. The "50"
in the name refers to the number of layers in the
network, indicating its depth. The "V2" denotes that it is
an updated version of the model with improved
performance and efficiency.
One of the key features of the ResNet architecture is the
introduction of residual connections or skip connections.
These connections allow the network to bypass some
layers and directly propagate the input or intermediate
activations to subsequent layers. This mitigates the
problem of vanishing gradients, enabling the network to
learn more effectively, especially when dealing with
very deep architectures.
The ResNet50V2 model consists of a series of
convolutional layers, followed by global average pooling
and fully connected layers. It incorporates residual
blocks, which are composed of multiple convolutional
layers, batch normalization, and activation functions.
The residual connections within the blocks facilitate the
flow of information and improve gradient flow during
training.
In the context of emotion recognition based on facial
expressions, the ResNet50V2 model takes an input
image of a face and processes it through the layers to
extract discriminative features. These features capture
the unique characteristics of facial expressions
associated with different emotions.
The output layer of the ResNet50V2 model is typically
configured to have multiple units corresponding to
different emotion classes. The final activation function,
often softmax, generates a probability distribution over
these emotion classes, enabling the model to make
predictions about the dominant emotion exhibited in the
facial expression.
Training the ResNet50V2 model for emotion recognition
involves using a labeled dataset of facial expression
images. The model's weights are optimized using
algorithms such as stochastic gradient descent (SGD) or
Adam optimizer. The choice of loss function, such as
categorical cross-entropy, helps measure the
dissimilarity between predicted probabilities and the true
emotion labels. Through iterative training, the model
adjusts its weights to minimize the loss and improve its
ability to accurately classify emotions.
The ResNet50V2 architecture has shown remarkable
performance in various computer vision tasks due to its
deep structure, residual connections, and efficient
training. These attributes make it capable of capturing
complex visual patterns and effectively recognizing
emotions based on facial expressions.
In summary, the ResNet50V2 model is a deep CNN
architecture with residual connections, designed for
tasks such as emotion recognition based on facial
expressions. Its ability to learn intricate features, along
with improved gradient flow through residual
connections, allows it to effectively capture and classify
different emotions. By leveraging its layers and
connections, the ResNet50V2 model demonstrates
strong performance in recognizing emotions from facial
expressions.
E. EfficientNetB0 Model
The EfficientNetB0 [14] model is a convolutional neural
network (CNN) architecture that has gained attention for
its efficiency and excellent performance in various
computer vision tasks, including emotion recognition
based on facial expressions.
EfficientNetB0 belongs to a family of models known as
EfficientNets, which were designed by combining
principles from neural architecture search and model
scaling. These models achieve high accuracy while
6
maintaining computational efficiency, making them
suitable for resource-constrained environments.
The EfficientNetB0 architecture follows a compound
scaling method, which uniformly scales the network's
depth, width, and resolution. This scaling allows the
model to achieve a good balance between model
capacity and computational efficiency. The "B0" in the
name signifies the base configuration of the EfficientNet
family, where "B0" represents the smallest and least
computationally expensive variant.
EfficientNetB0 consists of multiple stacked layers of
depth-wise separable convolutions, which reduce the
F. EfficientNetB7 Model
number of parameters and computational cost while
maintaining model performance. These depth-wise
separable convolutions split the standard convolutional
operation into two separate steps: a depth-wise
convolution, which processes each input channel
separately, and a point-wise convolution, which
combines the output of the depth-wise convolution
across channels.
The EfficientNetB0 model also incorporates other
techniques such as batch normalization, activation
functions, and skip connections. These techniques aid in
improving the learning process, increasing model
accuracy, and facilitating gradient flow during training.
In the context of emotion recognition based on facial
expressions, the EfficientNetB0 model takes an input
image of a face and passes it through its layers to extract
meaningful features. These features capture the relevant
patterns and expressions associated with different
emotions.
The output layer of the EfficientNetB0 model is
typically configured to have multiple units
corresponding to the different emotion classes. The final
activation function, often softmax, produces a
probability distribution over these emotion classes,
enabling the model to make predictions about the
dominant emotion displayed in the facial expression.
Training the EfficientNetB0 model for emotion
recognition involves using a labeled dataset of facial
expression images. The model's weights are optimized
using algorithms such as stochastic gradient descent
(SGD) or Adam optimizer. The choice of a suitable loss
function, such as categorical cross-entropy, helps
measure the dissimilarity between the predicted
probabilities and the true emotion labels. Through
iterative training, the model adjusts its weights to
minimize the loss and improve its ability to accurately
classify emotions.
The EfficientNetB0 model's efficiency and performance
make it well-suited for emotion recognition based on
facial expressions. Its ability to capture important
features while being computationally efficient allows for
accurate emotion classification, even in resource-limited
settings.
In summary, the EfficientNetB0 model is a highly
efficient CNN architecture that achieves excellent
performance in emotion recognition tasks based on
facial expressions. Its compound scaling approach,
depth-wise separable convolutions, and other techniques
contribute to its efficiency and accuracy. By leveraging
these features, the EfficientNetB0 model demonstrates
strong performance in accurately recognizing emotions
from facial expressions.
EfficientNetB7 is a deep learning model and a part of the
EfficientNet family, which is a series of convolutional
neural networks (CNNs) designed to achieve
state-of-the-art performance with significantly fewer
parameters compared to other models. The
EfficientNetB7 model is the largest and most powerful
variant in the EfficientNet series.
The main idea behind EfficientNet models is compound
scaling, which involves scaling the depth, width, and
resolution of the network in a balanced manner. This
allows EfficientNetB7 to achieve better performance and
efficiency by effectively utilizing computational
resources.
Specifically, the EfficientNetB7 model has the following
characteristics:
● Depth: EfficientNetB7 has a deep network
architecture with a large number of layers, allowing
it to capture complex patterns and features from the
input data.
● Width: It has a significantly larger number of
channels or filters in each layer compared to
smaller variants, which enables it to learn more
expressive representations.
● Resolution: The input images to EfficientNetB7 are
of higher resolution, which allows the model to
capture fine-grained details and improve
recognition accuracy.
EfficientNetB7 has been pre-trained on large-scale
datasets, such as ImageNet, using techniques like
transfer learning. As a result, it has learned to recognize
a wide range of features from different images. This
pre-training makes it a powerful feature extractor that
can be fine-tuned on specific tasks or datasets with
relatively few additional training samples.
Due to its efficiency and high performance,
EfficientNetB7 is commonly used in various computer
vision tasks, such as image classification, object
detection, and semantic segmentation, where it
consistently achieves top-tier results. However, it should
be noted that EfficientNetB7 might require significant
7
 computational resources, especially during training, due
to its large size.
G. Stochastic Gradient Descent (SGD)
Stochastic Gradient Descent (SGD) is an iterative
optimization algorithm commonly used in machine
learning for training models, including neural networks,
to minimize a given loss function and find the optimal
set of parameters. It is particularly effective when
dealing with large datasets.
The name "stochastic" refers to the fact that the
algorithm operates on randomly selected subsets of the
training data, known as mini-batches, instead of the
entire dataset. This random sampling introduces a level
of randomness into the optimization process and allows
the algorithm to escape local minima more easily.
Here is a step-by-step explanation of how SGD works:
H. Stochastic Gradient Descent – Adam Method
1. Initialize Parameters: The algorithm starts by
initializing the model parameters with random
values. These parameters are the variables that the
model will learn during the training process.
2. Select Mini-Batch: SGD randomly selects a
mini-batch of training examples from the dataset.
The size of the mini-batch is typically chosen
based on computational constraints and can range
from a few samples to a few hundred samples.
3. Compute Gradient: The selected mini-batch is used
to compute the gradient of the loss function with
respect to the model parameters. The gradient
represents the direction and magnitude of the
steepest ascent or descent in the loss function's
landscape. It indicates how the parameters should
be adjusted to minimize the loss.
4. Update Parameters: The parameters are updated by
taking a small step in the opposite direction of the
gradient. This step is controlled by a learning rate,
which determines the size of the update. A smaller
learning rate leads to slower convergence, while a
larger learning rate may cause overshooting and
instability.
5. Repeat: Steps 2-4 are repeated for a fixed number
of iterations or until a stopping criterion is met.
The entire dataset is usually processed multiple
times, with each pass over the data known as an
epoch. The order in which the mini-batches are
processed can either be randomly shuffled or
preserved in their original order.
The key advantage of SGD is its efficiency in processing
large datasets. Since it operates on mini-batches, it
requires less memory and computational resources
compared to batch gradient descent, where the entire
dataset is used in each iteration. The stochastic nature of
SGD also adds a regularizing effect, helping the model
generalize better and avoid overfitting.
However, SGD introduces some noise due to the
randomness in mini-batch selection, which can cause the
optimization process to be more erratic. To address this,
various modifications of SGD have been proposed, such
as momentum, adaptive learning rates (e.g., AdaGrad,
RMSprop, Adam), and learning rate schedules.
In summary, Stochastic Gradient Descent is an iterative
optimization algorithm that updates model parameters
based on randomly selected mini-batches of training
data. It efficiently handles large datasets and helps
models converge to an optimal solution. By iteratively
adjusting the parameters in the direction of steepest
descent, SGD enables the training of complex machine
learning models.
The Adam optimizer is an extension of the stochastic
gradient descent (SGD) algorithm that combines
elements of both the AdaGrad and RMSprop
optimization techniques. It is widely used in training
deep neural networks due to its effectiveness in finding
good solutions and its adaptive learning rate capabilities.
Here is a deeper explanation of the Adam optimizer:
1. Initialization: The Adam optimizer initializes two
variables, namely the first moment estimate (often
called the "mean") and the second moment
estimate (often called the "variance"). These
variables are initialized as vectors of zeros with the
same dimensions as the model's parameters.
2. Computing Gradients: During each iteration of the
optimization process, a mini-batch of training
examples is randomly sampled. The gradients of
the loss function with respect to the model
parameters are computed using backpropagation.
3. Updating the First Moment Estimate: The Adam
optimizer updates the first moment estimate by
calculating the exponentially decaying average of
the gradients. This step helps capture the overall
trend of the gradients over time.
4. Updating the Second Moment Estimate: The
second moment estimate is updated by calculating
the exponentially decaying average of the squared
gradients. This step helps capture the magnitudes
of the gradients and acts as a form of adaptive
learning rate adjustment.
5. Bias Correction: In the early iterations of training,
the estimates of the first and second moments can
be biased towards zero due to their initialization as
zero vectors. To address this, bias correction is
applied to the first and second moment estimates to
make them unbiased.
8

6. Learning Rate Scaling: The Adam optimizer scales
the gradients by dividing them by the square root
of the second moment estimate. This scaling allows
for adaptive learning rates, where the learning rate
is automatically adjusted based on the magnitudes
of the gradients.
7. Parameter Update: Finally, the model parameters
are updated by subtracting the scaled gradients,
which are divided by the square root of the second
moment estimate, multiplied by the learning rate.
This step effectively moves the parameters in the
direction that minimizes the loss function.
The Adam optimizer's adaptive learning rate mechanism
makes it less sensitive to the choice of an initial learning
rate and helps achieve faster convergence. It combines
the benefits of AdaGrad, which adjusts the learning rate
for each parameter individually, and RMSprop, which
performs adaptive learning rate scaling. Additionally, the
bias correction step ensures that the estimates of the first
and second moments are accurate, particularly during the
early stages of training.
In summary, the Adam optimizer is an adaptive
optimization algorithm that updates model parameters by
maintaining estimates of the first and second moments of
the gradients. It incorporates adaptive learning rates,
making it robust and efficient in training deep neural
networks. By combining the strengths of AdaGrad and
RMSprop, the Adam optimizer provides an effective
approach for minimizing the loss function during the
training process.
VI. EXPERIMENTAL STUDY
The main objective of the party is to carry out an
experimental study that focuses on comparing the
performance of five different algorithms used for emotion
recognition based on facial expressions. The diagram
provided below illustrates the sequential steps involved in
this study, starting from the importation of data and
progressing through the various stages until the testing of
the best model and parameters identified through our
comprehensive comparison.
Fig. 2. The pipeline of our experiment
A. Experimental Setup
Scaling and resizing an image involves adjusting its size
while preserving its aspect ratio or changing the aspect
ratio as desired. The process typically involves two
steps: scaling and resizing.
The pixel values of an image typically range from 0 to
255, representing the intensity of each pixel. Scaling the
image by dividing it by 255 transforms the pixel values
to a range between 0 and 1. This normalization is often
performed to ensure that the pixel values are within a
consistent and standardized range, which can be
beneficial for various image processing algorithms and
models.
In the context of facial expression recognition, resizing
an image to (48, 48) is commonly done to preprocess
facial images and prepare them as input for emotion
recognition models. The dimensions of 48x48 pixels
have been widely adopted in facial expression datasets
and models. This size is typically sufficient to capture
important facial features while keeping the
computational requirements manageable.
Fig. 3. Data importation
To load a pre-trained model means to import and initialize a
pre-trained deep learning model that has been trained on a
large dataset. The pre-trained model has already learned
meaningful representations of the data it was trained on,
making it capable of performing specific tasks, such as
image classification.
A pre-trained model on ImageNet refers to a deep learning
model that has been trained on the large-scale ImageNet
dataset. The ImageNet dataset is a widely used benchmark
in computer vision, consisting of millions of labeled images
belonging to thousands of different classes or categories.
Fig. 4. Load the pre-trained models
9
 The term "freezing" refers to the process of preventing
the weights and parameters of specific layers in a
pre-trained model from being updated or trained further
during the fine-tuning or transfer learning process.
When we load a pre-trained model, all the layers in the
model have already been trained on a large dataset to
learn meaningful representations. However, in some
cases, we may want to fine-tune the model for a specific
task using your own dataset.
By freezing layers, we keep the learned representations
intact, especially in the early layers that capture basic
features and patterns. This is useful because the
pre-trained model has been trained on a dataset similar
to our task, and we want to leverage the pre-existing
knowledge while fine-tuning on your specific data.
hyperparameter space using different strategies to
find the best combination of hyperparameters.
● Objective Functions: You can define an objective
function that quantifies the performance of your
model based on specific metrics, such as accuracy,
loss, or any custom evaluation metric. The tuner
uses this objective function to guide the search
process and optimize the hyperparameters.
● Early Stopping: Keras Tuner supports early
stopping, which allows you to stop the search
process if the performance of the model plateaus or
worsens. This helps save computational resources
by terminating the search early if no further
improvements are observed.
● Results Analysis: Keras Tuner provides utilities to
analyze and visualize the results of the
hyperparameter search, such as the best
hyperparameters found, performance metrics
across different trials, and search history.

By using Keras Tuner, you can automate the process of

Fig. 5. Freezing layers
Keras Tuner is a hyperparameter optimization library for
Keras, a popular deep learning framework. It provides a
convenient and efficient way to search and tune
hyperparameters to improve the performance of your
deep learning models.
finding the best hyperparameters for your deep learning
models. This helps you save time and resources by
optimizing your models' performance and generalization
without the need for manual trial and error.
Overall, Keras Tuner simplifies the hyperparameter
optimization process by providing an interface to define
search spaces, select search algorithms, define objective
functions, and analyze the results. It empowers you to
find the optimal hyperparameters for your deep learning
models efficiently.

Hyperparameters are parameters that are not learned

during the training process but are set prior to training.
They control various aspects of the model, such as the
learning rate, the number of layers, the number of units
in each layer, the activation functions, the dropout rate,
and more. Tuning these hyperparameters can have a
significant impact on the performance and generalization
of your model.

Keras Tuner offers different search algorithms and

strategies to efficiently explore the hyperparameter space
and find optimal or near-optimal combinations. Here are
some key components and features of Keras Tuner:
Fig. 6. Keras-tuner_1
● Search Spaces: Keras Tuner allows you to define
the search space for each hyperparameter. You can
specify the range, values, or distributions that the
tuner should consider when searching for the best
hyperparameters.

● Tuners: Keras Tuner provides different tuners, such

as RandomSearch, Hyperband, and Bayesian
Optimization-based tuners like
BayesianOptimization and TPE (Tree-structured
Parzen Estimators). These tuners explore the

10
 transforming the output of the final layer into
meaningful class probabilities, enabling reliable and
interpretable predictions.

Fig. 9. Best model

Here is the graph of the Loss and the accuracy:

Fig. 7. Keras-tuner_2

Fig. 8. Early Stopping

B. Results and decision

The optimal model consists of the following
components:

● VGG16: A convolutional neural network

architecture known for its deep layers and excellent
performance in image classification tasks.

● Flatten Layer: This layer is responsible for

transforming the multi-dimensional output of the
VGG16 model into a one-dimensional vector,
enabling compatibility with fully connected layers.

● Dense Layer with 1000 Units: A fully connected

layer with 1000 units. Each unit is connected to
every neuron in the previous layer, allowing for
complex feature extraction and representation.

● Output Layer (Dense Layer): This layer serves as

the final layer of the model and contains 6 units,
corresponding to the 6 classes in the classification
task. It utilizes the softmax activation function.

To summarize, the optimal model is built upon the
VGG16 architecture, which serves as the foundational
backbone. It is complemented by a flatten layer to
reshape the output, a dense layer with 1000 units to
capture intricate patterns, and an output layer with
softmax activation for effective multiclass classification.
The softmax function plays a pivotal role in
Fig. 10. The graphical representation depicting the loss and accuracy
metrics
The confusion matrix provides a comprehensive view of
the model's performance by showing the distribution of
correct and incorrect predictions across different classes.
It helps in understanding the types of errors made by the
model, such as false positives and false negatives.

11

Fig. 11. Confusion matrix
VII. CONCLUSION
In conclusion, this research study aimed to compare the
performance of five different algorithms (VGG16, VGG19,
RESNET50V2, EfficientNETB0, and EfficientB7) for
recognizing emotions based on facial expressions. The study
followed a step-by-step process, starting from importing the
data and going through different stages until finding the best
model and settings through our comparison.
The results of this study provide valuable insights into how
well these algorithms performed in recognizing emotions.
By comparing their accuracy, confusion matrix, we were
able to determine which algorithm worked the best for
recognizing emotions based on facial expressions.
These findings contribute to improving emotion recognition
technology and can be useful in various areas like e-learning
platforms or computer interactions. By using the best model
and settings identified in this study, we can enhance the
effectiveness and efficiency of emotion recognition systems,
leading to better user experiences and tailored support.
It's important to mention that further research can be done to
explore other algorithms or improve the existing ones. Also,
when using these algorithms in real-life applications, we
should consider factors like computational resources,
complexity, and finding a balance between accuracy and
efficiency.
Overall, this research provides a foundation for future
developments in emotion recognition based on facial
expressions. It highlights the importance of selecting the
most suitable algorithm to achieve accurate and reliable
results. By using the findings of this study, we can advance
emotion recognition technology and improve
human-computer interactions in various fields.
VIII. LIMITATIONS AND FUTURE WORKS
While the tested algorithms showed promise, we
encountered several limitations that we aim to address in our
future works. Below, we outline these limitations and
discuss our objectives for future research.
1. Limitations:
● Limited Accuracy: The tested algorithms for
emotion recognition based on facial expressions
did not provide highly accurate results. Further
improvements are needed to enhance their
performance.
● Dataset Limitations: The dataset used for training
and evaluation may have had limitations, such as
being small or not representing emotions well. This
could have affected the accuracy of the algorithms.
● Algorithm Suitability: The five algorithms tested
may not cover all possible options. Exploring more
algorithms could lead to better results.
● Resource Requirements: The tested algorithms may
require powerful computers with high-end graphics
cards and processors. Access to such resources
could be a challenge in real-life applications.
2. Future Works:
● Improved Algorithm Development: More research
should focus on developing better algorithms for
emotion recognition based on facial expressions.
Exploring new approaches, like considering time or
attention, could improve accuracy.
● Enhanced Dataset Collection: Gathering larger and
more diverse datasets that represent different
emotions and people can improve algorithm
accuracy.
● Real-life Testing and Validation: It is important to
test and validate the algorithms in real-life
situations, such as in e-learning platforms. This can
provide feedback for improvements and
adjustments.
● Hybrid Approaches: Combining multiple
algorithms or using ensemble techniques could
lead to better accuracy in emotion recognition.
● Personalized CNN Configuration: Creating a
custom Convolutional Neural Network (CNN) with
personalized settings specifically for emotion
recognition could improve accuracy and
adaptability to different datasets.
By addressing these limitations and exploring future
research directions, we can improve the accuracy and
practical application of emotion recognition technology.
This will enhance human-computer interactions and provide
better user experiences in various fields.
12
 View publication stats

FIGURE TABLE [10] Wisal Hashim Abdulsalam, Rafah Shihab Alhamdani, and
Mohammed Najm Abdullah “Facial Emotion Recognition from
Videos Using Deep Convolutional Neural Networks” [CrossRef]
[11] Philipp Michel and Rana El Kaliouby “Real time facial expression
Figure Figure title Page recognition in video using support vector machines” [CrossRef]
number Number [12] David Dukić and Ana Sovic Krzic “Real-Time Facial Expression
Recognition Using Deep Learning with Application in the Active
Figure 1 The steps of emotion 4 Classroom Environment”. [CrossRef]
recognition using facial [13] VGG16 [CrossRef]
expression [14] VGG19 [CrossRef]
[15] ResNet50V2 [CrossRef]
Figure 2 The pipeline of our 9 [16] EfficientNetB0 [CrossRef]
experiment [17] EfficientNetB7 [CrossRef]
[18] Adam optimizer [CrossRef]
Figure 3 Data importation 9

Figure 4 Load the pre-trained models 9

Figure 5 Freezing layers 10

Figure 6 Keras-tuner_1 10

Figure 7 Keras-tuner_2 11

Figure 8 Early Stopping 11

Figure 9 Best model 11

Figure 10 The graphical representation 11

depicting the loss and
accuracy metrics

Figure 11 Confusion matrix 12

REFERENCES

[1] Ayvaz, Uğur; Gürüler, Hüseyin; Devrim, Mehmet Osman “USE OF

FACIAL EMOTION RECOGNITION IN E-LEARNING
SYSTEMS” [CrossRef]
[2] D'Errico Francesca, Paciello Marinella, De Carolis Bernardina,
Vattanid Alessandro Palestra Giuseppe, Anzivino Giuseppe Cognitive
Emotions in E-Learning Processes and Their Potential Relationship
with Students’ Academic Adjustment [CrossRef]
[3] Hassouneh Aya, A.M. Mutawa, and M. Murugappan “Development
of a Real-Time Emotion Recognition System Using Facial
Expressions and EEG based on machine learning and deep neural
network” [CrossRef]
[4] Shaik Asif Hussain and Ahlam Salim Abdallah Al Balushi “A real
time face emotion classification and recognition using deep learning
model” [CrossRef]
[5] Mehmet Akif Ozdemir, Berkay Elagoz, Aysegul Alaybeyoglu, Reza
Sadighzadeh and Aydin Akan “Real Time Emotion Recognition from
Facial Expressions Using CNN Architecture” [CrossRef]
[6] Isha Talegaonkar, Kalyani Joshi, Shreya Valunj, Rucha Kohok and
Anagha Kulkarni “Real Time Facial Expression Recognition using
Deep Learning” [CrossRef]
[7] Marian Stewart Bartlett, Gwen Littlewort, Ian Fasel and Javier R.
Movellan “Real Time Face Detection and Facial Expression
Recognition: Development and Applications to Human Computer
Interaction.” [CrossRef]
[8] Aneta Kartali, Miloš Roglić, Marko Barjaktarović, Milica
Đurić-Jovičić and Milica M. Janković “Real-time Algorithms for
Facial Emotion Recognition: A Comparison of Different
Approaches”. [CrossRef]
[9] Ati Jain, Hare Ram Sah “Student’s Feedback by emotion and speech
recognition through Deep Learning”. [CrossRef]

13