Real-time student emotion and performance

analysis.

Pramodini Metgud
Navya Dayanand Naik Sukrutha M S
pramodinimetgud@gmail.com
navyanaik292@gmail.com sukruthams1@gmail.com
Anita S Prasad
Department of Electronics and
Communication
JSS Science and Technology University
Mysuru, India
anith.sp@sjce.ac.in

978-1-6654-9781-7/22/$31.00 ©2022 IEEE

Abstract –Emotional state of the learner in any course plays
a major role. By analyzing the facial emotion accurately,
student performance can be improved. The proposed system
gives an accurate prediction of emotions with reference to
the data set FER-2013. Accuracy of around 85.72% is given
by using the EfficientNet-B0 algorithm. After facial
expression detection, face recognition is performed to fetch
the student information. The student information along with
performance is sent to concerned faculty and parents.
Keywords— Real-time facial emotion recognition,
EfficientNet-B0, Student data.
I. INTRODUCTION
The face is the most expressive part of a person [4].
Facial emotion recognition identifies a person’s emotion
from their facial expression. All human emotions can be
expressed as a combination of 7 basic emotions (anger, fear,
disgust, sadness, surprise, happiness, and neutral).
Emotion recognition and analysis are of great importance
in medical practice, education, and social practices. In the
field of education, the mental health of the student is very
important for their academic excellence [5]. The traditional
way of acquiring the emotion of students is by manual
analysis of the student’s marks and observations from the
feedback forms.
II. RELATED WORK
Emotions have a significant impact on students' learning
processes, according to recent studies [8][9][10][11]. The
research that is now available seems to indicate that positive
emotions affect students' attention, motivation, and self-
regulation of learning, which in turn influences learning.
Negative emotions, on the other hand, have an impact on
student’s performance and achievement, which also affects
learning. For instance, guilt and worry lower motivation and
interest. Different emotions may elicit various non-task-
related cognitions, and they may vary in the degree of
intensity and permanence of these cognitions. As a result,
both positive and negative emotions play a crucial part in
how information is stored and retrieved during learning.
According to [11], the activation of emotions encourages the
activation of association data in long-term memory, which
makes recall easier.
Many researchers are interested in improving the
accuracy of facial emotion recognition. Imane Lasri [1]
proposed a system to recognize the student’s emotions using
deep learning techniques like CNN. The data set used is
FER-2013. The aim was to help the teacher modify their
presentation according to the result.
Mingxing Tan performed a systematic study on model
scaling to improve the accuracy of the Convolution Neural
Networks (CNN). Based on their observation, a new method
of scaling uniformly scales all dimensions of depth /width
/resolution using a highly effective compound coefficient.
Further, they used neural architecture search to design a new
baseline network and scale it up to obtain a family of models,
called EfficientNets [2].
Research has shown that emotion plays a very important
role in students’ life both personally and academically.
Academically, students experience a large variety of self-
referenced and social emotions like pride, shame, peer
pressure, and anxiety [6]. Hence there is a need for regular
analysis of students’ emotions.
Naman Gupta [7] proposed the automatic attendance
system to overcome the drawbacks faced by using the
traditional system of marking attendance, in which a teacher
has to manually mark attendance, which is both time-
consuming and prone to mistakes.
In this paper, the implementation of emotion recognition
is done for the analysis of student emotions based on their
facial expression using EfficientNet-B0, EfficientNet can be
considered a group of convolution neural network models,
and gives better results than Machine Learning algorithms.
The project is divided into three main parts: face emotion
recognition, student data acquisition, and compiling and
sending the required data to the concerned teacher.
III. PROPOSED APPROACH
The main purpose of the project is to analyze the emotion
of the students twice every semester, once at the beginning, a
few weeks after school/college starts, and again just before
the final exams. The results and data obtained will be shared
with the concerned faculty and parents, if the student
displays any negative emotions like anger, sorrow, and fear.
The project can be divided into three sections or less. The
first section is student emotion analysis which is done using
EfficientNet-B0, and the FER-2013 data set. The second
section is student face recognition, this information is used to
find all the data related to the students. In the third section,
all the data obtained in the first two sections will be sent to
the concerned faculty and the student’s parents.
Proposed Algorithm
Step 1: Image captured from the webcam.
Step 2: Analysis of emotions using EfficientNetB0.
Step 3: If: Indication of any negative emotion.
Facial detection.
Send notification to concerned faculty and parents.
Else: Exit
Step 4: End
A. Student emotion detection
The popularity of using CNNs for image analysis has
grown over the past few years, largely due to the impressive
results that have been obtained. In contrast, CNN requires a
large amount of training data before it can begin to work.
Using CNNs trained on millions of images has been used to
solve this problem, but it is time-consuming and requires a
huge amount of data. Using a pre-trained model avoids the
need for a large amount of data and maximizes processing
resources. The most common problem is when the intended
data collection is too small for it to be useful. There may be
instances in which overfitting is an issue, and data
augmentation is not always enough to solve it. This study
proposes a transfer learning method that uses
EfficientNetB0, an emotion recognition system, to focus on
salient face areas.
Data set
The data set used is FER-2013. The data set
consists of 48x48 pixel grayscale images of the faces of
people. The faces are automatically registered in order
that the face is more or less centered and occupies
about an equivalent amount of space in each image . The
training set consists of 28,709 grayscale images and the test
set consists of 3,589 grayscale images. It contains 7
emotions (anger, happy, sad, fear, surprised, neutral, and
disgust).
Proposed approach
In this paper, we suggest that the EfficientNetB0
Convolutional Neural Network model be used as the basis
for transferring learning. Real-world photographs of facial
expressions were used to test the suggested approach. In this
study, EfficientnetB0 was used to transfer learning. While
creating a data generator for images, there are a few things
to keep in mind (Rescale by 255, split data 20 percent,
rotation by 5, shift height and weight by 20 percent, zoom
20 percent) Datasets were divided into three sections after
being cleaned up: training, testing, and validation. Target
picture size 128x128 has been used for all three datasets.
For all three datasets, batch size 64 has been used. Select
seven different label types (such as angry, disgusted, fearful,
happy, neutral, sad, and surprised). The training set has
22968 photos, whereas the validation set contains 5741, and
the test collection contains 7178. The EfficientNetB0 and
imagenet weight were then loaded using transfer learning.
Adam optimizer and categorical cross-entropy loss function
are used to construct the model. At 10, 50, and 100 epochs,
the model has been trained. Weight should be used for
testing when training is complete. A Haarcascade frontal
face classifier is used to recognize facial characteristics and
determine emotional state from seven categories using
OpenCV.
a)EfficientNet-B0 model
One can think of EfficientNet as a collection of
convolutional neural networks. Despite its complexity,
however, it still proves to be more efficient than most of its
predecessors. In the EfficientNet model family, there are
eight models from B0 through B7, each specifying a model
with increased accuracy and more parameters There are
more than one million photos in the ImageNet collection
that was used to train EfficientNet-B0. The network can
identify over 1000 item types such as keyboards, mice, and
other animals. A broad variety of pictures are represented by
more complex feature representations in the network. The
network accepts images with a resolution of 224 by 224
pixels.[2]
EfficientNet Architecture
 In this section, we will discuss the EfficientNet-B0
architecture in detail.
 In B0, which is a mobile-sized architecture, there are 11M
trainable parameters.
 Let's have a look at table 1 to see how the new architecture
appears.
Table 1: EfficientNet-B0 network
In the architecture, 7 inverted residual blocks are used, but
each is set differently. Additionally, these blocks use
squeeze & excitation blocks along with swish activation.
Swish Activation
Swish is the product of a linear and a sigmoid activation.
Swish(x) = x * sigmoid(x)
Inverted Residual Block
In MobileNet's residual block, convolution is based on
depthwise separable convolution, which employs depthwise
convolution first and pointwise convolution after that. The
result is a reduction in the number of parameters that can be
trained. The inverted residual block results in skip
connections linking narrow layers, while larger ones are left
between skip connections.
Squeeze and Excitation Block
When generating an output feature map, CNN weights the
convolutional layers equally Rather than treating each
channel equally, the squeeze and excitation (SE) block gives
each channel more weight. The SE square gives an yield of
shape (1 x 1 x channels) that characterizes the weightage for
each channel, and the astounding portion is that the neural
organize can learn this weightage by itself like other
parameters, which may be a enormous advantage.
EfficientNet’ MBConv Block
To begin with, the MBConv piece requires information and
the moment, the MBConv block's contentions. The final
layer yields the information. Qualities such as input and
yield channels, development and press proportion, and so on
are all portion of a square contention which will be used
inside an MBConv square.
b)Transfer Learning
A model that has been learned for one task may be
transferred to another similar activity via transfer learning.
Deep learning networks with multiple parameters are
expensive and time-consuming to compute. These networks
are not overfitted because of the volume of data needed to
train them. As a result, it is typical for researchers to invest a
lot of time in training for cutting-edge models. The premise
that a state-of-the-art model is trained using a lot of
resources, hence the benefits of such investments should be
realized many times over, gave rise to the concept of
transfer learning. The best thing is that transfer learning
allows you to reuse all or part of your model! In this
manner, we can avoid training the entire model.
c) Loss Function Categorical Cross Entropy
For categorical classification, cross-entropy loss contributed
by training data point 𝒊,(𝒙𝒊 , 𝒚𝒊), is simply the "negative log-
likelihood (NLL)":
since the ground truth probability is one for the correct label
𝒚𝒊 and zero for every other label.
d) Adam Optimizer (Adaptive Moment Estimation
It is a gradient descent optimization algorithm. This strategy
works well when tackling large problems with lots of data or
parameters. It is efficient and requires minimal storage. It
makes intuitive sense to combine the "gradient descent" and
"RMSP" algorithms.
The estimations of the first moment (mean) and second
moment (uncentered variance) of the gradients are denoted
by the letters mt and vt, respectively. Researchers of Adam
have found that because mt and vt are initialised as vectors
of zeros, they are biassed toward zero, particularly during
initial time steps and especially when decay rates are slow
(such that t1 and t2 are almost 1).
Proposed Algorithm for emotion classification
Step 1: Load FER2013 Dataset.
Step 2: Preprocess the image by defining the image data
generator.
Step 3: load the dataset of the training set, validation set,
and test set using target image size 128x128 and batch size
64.
Step 4: Select class label into 7 categories including {0:
"Angry", 1: "Disgusted", 2: "Fearful", 3: "Happy", 4:
"Neutral", 5: "Sad", 6: "Surprised"}
Step 5: Divide the images into training, testing, and
validation set
Step 6: Apply the EfficientNetB0 model to train the
network.
Step 7: load the weight of ImageNet.
Step 8: Use transfer learning to select a related predictive
modeling problem with an abundance of data.
Step 9: Compile the model using the Adam optimizer.
Step 10: Calculate the loss using categorical cross-entropy.
Step 11: Train the model for 10,50, and 100 epochs to detect
facial emotions.
Step 12: Test the model using OpenCV using a haarcascade
frontalface classifier to detect facial features and predict the
emotion from seven categories.
Step 13: Obtain accurate expression detection results.
The validation accuracy obtained in this method after
running 20 epochs is at 85.72% and the loss is at 18% as
shown in Fig 1. The model reaches an accuracy of 85.72%
by the end of the 7th epoch. The loss function is minimum
for this model at 18% at the end of the 10th epoch. If 20
epochs are run all the parameters will be met.
Fig 1. Training and validation accuracy.
B. Student face recognition
In this section, the input used is from the first section, the
same input real-time image captured, is used for face
recognition. In a class of 200 students, the teacher cannot
remember all the names. So along with the emotion, the
student’s name should also be sent to the teacher to make
his/her job easy.
In this method, two databases are created, a student
database and an unknown database. The student database
contains the images of students of a class. The unknown
database contains the data which is taken from a camera,
which is installed in the class. Face recognition and emotion
detection algorithms both will be applied.
Student database creation
The student database is created by storing the images of
the students of a class, along with their names. A google
spreadsheet is also created with the names, addresses,
CGPAs, etc. of the students of that class
With the help of the images, which are stored in the
student database, facial recognition can be performed to
identify the student who is showing negative emotions.
Face detection
The detection of the faces is done by identifying the 68
landmarks present on a person’s face. Based on these
landmarks, the Haar cascade or Viola and Jones algorithm
[8] will be used. It will create a face bounding box.
Face recognition
The face recognition library is used to recognize the
student’s face. Face recognition is considered the world’s
simplest face recognition library. It has built-in dlib’s state-
of-the-art face recognition built with deep learning. The
model has an accuracy of 99.38% on labeled faces.
 Face recognition is implemented in this project as shown
in Fig 2. The student database has labeled images of the
students of a class. After the student is identified, the data
obtained from the emotion detection of the student is shared
with the teacher. This is discussed in the next section.
name, address, backlogs, CGPA, etc. of the students. Once
the student is identified, all the data present in the sheet
along with the emotion analysis data is sent to the concerned
faculty and that student’s parents. Fig 3 is an example of
student information collected and stored by school/college.

Fig 3. Student data.

IV. RESULTS
The following results were obtained from this project. Fig 4.
Shows real-time student emotion analysis. The graph is
showing negative emotions like fear and anger. Fig 5. is the
face recognition done for a student showing negative
emotions, Fig 6. shows all the data obtained is sent as an
email to the concerned faculty and the student’s parents.

Fig. 2. Face recognition system

Proposed Algorithm for face recognition

Step 1: Start.
Step 2: Creating the student database.
Step 3: Camera to capture the faces of students.
Step 4: Face detection.
Step 5: Face recognition by comparing images taken from
the camera with the student database. Fig 4. Real-time results of a student showing negative
emotions
Step 6: IF: face present in the student database.
Go to step 7.
ELSE: go back to step 2.
Step 7: Save and send the data.
Step 8: End.

C. Sharing of data
In this section, the data obtained from the before two
sections will be sent to the concerned faculty and the Fig 5. Face recognition of the student.
student’s parents. In the student database which was created,
a google sheet which contains all the data related to
students. The google sheet contains information like the

Fig 6. Student data was sent to the concerned faculty.
V. CONCLUSION
This paper proposes a method for real-time student
emotion analysis, face recognition, and sharing the student
information with concerned faculty. The model used in this
project gives 85.72% accuracy for emotion detection on the
FER-2013 database, 99.83% accuracy for student face
recognition on the labeled student database. The notification
helps teacher and parents to improve the student’s
performance. This will definitely help to improve the
learning environment.
This framework helps to monitor the student’s
emotional state. This approach will also help teachers and
parents to be aware of the academic performance. In the
future, we can work on 3D facial emotion analysis because
they give better result as it considers spontaneous change in
the face.
[4] R. G. Harper, A. N. Wiens, and J. D. Matarazzo, Nonverbal
communication: the state of the art. New York: Wiley, 1978.
[5] G. H. Bower, “Mood and Memory American Psychologist 36:
129– 148,” CrossRef Google Sch., 1981.
[6] R. Pekrun, T. Goetz, W. Titz, and R. P. Perry, “Academic
Emotions in Students’ Self-Regulated Learning and
Achievement: A Program of Qualitative and Quantitative
Research,” Educ. Psychol., vol. 37, no. 2, pp. 91–105, Jun.
2002.
[7] Naman Gupta, Purushottam Sharma, Vikas Deep, “Automatic
Attendance System Using OpenCV”
[8] M. Feidakis, T. Daradoumis, and S. Caballe, “Endowing e-
Learning Systems with Emotion Awareness,” in 2011
Third International Conference on Intelligent Networking and
Collaborative Systems, 2011, pp. 68–75.
[9] J. M. Harley, S. P. Lajoie, C. Frasson, and N. C. Hall, “An
Integrated Emotion-Aware Framework for Intelligent Tutoring
Systems,” in Artificial Intelligence in Education, 2015, pp. 616–
619.
[10] F. Benmarrakchi, J. E. Kafi, A. Elhore, and S. Haie, “Exploring
the use of the ICT in supporting dyslexic students’ preferred
learning styles : A
[11] preliminary evaluation,” Educ. Inf. Technol., pp. 1–19, Oct.
2016.
[12] G. H. Bower, “Mood and Memory American Psychologist 36:
129– 148,” CrossRef Google Sch., 1981.

REFERENCES
[1] Imane Lasri, Anouar Riad Solh, “Facial Emotion Recognition of
 
Students using Convolution Neural Network” in 2019 Third
International Conference on Intelligent Computing in Data
Sciences (ICDS), 2019.
[2] Mingxing Tan, Quoc Le, “ EfficientNet: Rethinking Model
Scaling for Convolution Neural Networks” Proceedings of the
36th International Conference on Machine Learning, PMLR
97:6105-6114, 2019.
[3] Viola, M. J. Jones, and Paul, “Robust real-time face detection”
in International journal of computer vision 57.2 (2004), 2004.