Stress Detection through Facial Expressions

E. Shreya K. Joy Princy Sayeeda Juvana Khan

Department of Computer Science
Engineering
G.Naryamamma Institute of
Technology and Science
shreyaetooru@gmail.com
Department of Computer Science
Engineering
G.Naryamamma Institute of
Technology and Science
joyprincykore@gmail.com
Department of Computer Science
Engineering
G.Naryamamma Institute of
Technology and Science
sayeedajuvanakhan@gmail.com

T. Manasa
Department of Computer Science
Engineering
G.Naryamamma Institute of
Technology and Science
tandamanasa@gmail.com

Abstract—Mental health problems are detrimental to
a students' health. They can affect many areas of
students' lives, reducing their quality of life, academic
achievement, physical health, and satisfaction with the
college experience, and negatively impacting
relationships with friends and family members. These
issues can also have long-term consequences for students,
affecting their future employment, earning potential, and
overall health. Hence it is very important to identify the
stress early on to prevent the consequences. Early
detection can help them from falling into depression and
can suggest possible solutions that can relieve their stress
or to consult a doctor. Unfortunately, an automatic,
continuous, and unobtrusive and accurate early stress
detection method does not exist in real time. The existing
framework detects stress of an individual from facial
expressions, facial cues(stress value) using a web camera.
We are enhancing the existing framework for stress
detection in real time, we record face expressions, facial
cues by using a web camera and heart rate by using a
real sense camera. Multiple people’s data is being
collected at the same time. The information gathered
above is integrated to create a model that predicts
whether the students' feel stressed or not. A GUI is
created to improve user accessibility.
change. In moderate amounts, stress can be motivating
and even beneficial, as it helps us stay alert and
respond effectively to threats or challenges. However,
when stress becomes chronic or overwhelming, it can
have detrimental effects on our well-being and overall
quality of life.
Types of Stress: There are mainly two types of
stress, feels in daily lifestyle eventually which
characteristics are explained below: Eustress (Good
Stress)
● Motivates, focuses energy.
● Is short-term.
● Is perceived as within our coping abilities.
● Feels exciting.
● Improves performance.
● Causes anxiety or concern.
● Can be short- or long-term.
● Is perceived as outside of our coping abilities.
● Feels unpleasant.
● Decreases performance.

I. INTRODUCTION ● Can lead to mental and physical problems.

In this fast-paced world we inhabit, we often
encounter a prevalent force that influences our well-
being and ability to cope with life's challenges. This
omnipresent factor can manifest as a response to
various external and internal pressures, shaping our
thoughts, emotions, and behaviors. It can either fuel
our motivation or hinder our progress, impacting our
physical and mental health. As we navigate through
the complexities of daily life, understanding and
effectively managing this force is essential for
maintaining balance and resilience. Whether it's the
pressure of meeting deadlines, maintaining
relationships, or coping with unexpected challenges,
this force called “Stress” can affect us physically,
mentally, and emotionally. Stress is a natural response
of our body to situations that require adaptation or
Physiological Stress Physiological Stress responses
starts from the body reaction with the realization of
stressor presence, which is followed by sending signals
to the brain, and then to specific sympathetic and
hormonal responses to eliminate, reduce or cope with
the stress. Some of the common symptoms will be seen
like increased heart rate, increasing sugar and fat
levels, decreasing intestinal movements, etc. Stress can
be predicted by using physiological features, like.
● Facial expressions
● Changing breathing rate (respiration rate)
● Increases blood flow in forehead, around the eyes
● Electric conductivity on the skin surface
Physiological sensors play a key role in the world.
Analyzing sensor data is a very big task to the data

XXX-X-XXXX-XXXX-X/XX/$XX.00 ©20XX IEEE

analysts, but it provides an objective measure to
physiological signals. In comparison with other
resources like emotional (facial expressions or oral
statements) physiological signals offer substantial
advantages. Sensors can be used to read the signals
which help to analyze them and use for the research.
Today many sensors are available in the market which
can be wired/wireless and contact/non- contact. For
example, in the health domain like heart rate monitor,
human body temperature sensor, fitness sensor, etc.
The requirements of sensors for health monitoring
systems are that they should be portable, consume less
power, user friendly and so on [13]. 6 Depth sensing
techniques are an emerging technology. Depth
imaginaries image channel in which each pixel relates
to a distance between the image plane and the
corresponding object in the RGB image. So many
depth sensors are available in the market like
Microsoft Kinect V1 & V2, Intel RealSense D415 &
D435, Azure Kinect, Orbbec Astra (PRO), Asus Xtion
2, PrimeSense Carmine 1.08, etc.
Electrocardiography: Electrocardiography (ECG or
EKG) is a non-invasive medical test that records the
electrical activity of the heart over time. It helps
diagnose various heart conditions and irregularities by
measuring the heart's rhythm and identifying potential
abnormalities.
Electrooculography: Electrooculography (EOG) is
a technique used to measure and record the electrical
potentials generated by eye movements. It is
commonly utilized in medical and 7 research settings
to assess eye movements and analyze their correlation
with neurological conditions.
Blood volume pressure: It refers to the force
exerted by the blood against the walls of blood vessels,
including arteries and veins. Monitoring blood
pressure is crucial for assessing cardiovascular health
and identifying conditions like hypertension that may
pose risks to overall well-being.
Electrodermal activity: EDA is a measure of the
skin's electrical conductance, which changes with
sweat gland activity and emotional arousal. Q Sensor
is a wearable device that utilizes EDA to monitor
stress levels and emotional responses, providing
valuable insights into an individual's physiological
reactions in various situations.
Kinect V1: Also known as Kinect for Xbox 360,
was the first version of Microsoft's motion sensing
input device. It featured a 640x480 resolution camera,
depth sensor, and microphone array, primarily
designed for gaming.
Kinect V2: Also known as Kinect for Xbox One
and Kinect for Windows v2, was the improved version
of Kinect with enhanced capabilities. It featured a
1080p resolution camera, improved depth sensing,
higher accuracy, and enhanced skeletal tracking,
offering better performance and wider application
possibilities.
Intel RealSense D415: It is a depth-sensing camera
that provides accurate and detailed 3D depth
perception. With a wide field of view and high
resolution, it is ideal for applications such as robotics,
3D scanning, augmented reality, and facial
recognition.
Intel RealSense D435 is an advanced depth-sensing
camera with a wider field of view and higher depth
resolution compared to the D415. It enables precise
and immersive 3D perception, making it ideal for
applications in robotics, virtual reality, 3D scanning,
and gesture-based interactions.
II. EXISTING SYSTEM
Traditional system to detect stress is finding the
symptoms like a person having pains in head, chest,
stomach or muscles, digestive problems, reproductive
issues, changes to heart rate and blood pressure. In this
method stress symptoms can be a reason for some
other diseases and most of the symptoms can be
faked/misled. So, in this system there is a chance of a
mislead of the symptoms and accuracy of this is very
poor. Now-a-days, wearable sensors are used to create
smart watches which helps to detect stress by using
pulse rate. Still, this gives accurate values but must be
worn every time to monitor the stress of the human
which is uncomfortable sometimes. Sweat sensor is
another traditional way of detecting stress levels but
there is a possibility of misled results. So existing
systems may be wearable or have symptoms which can
lead to some misleading results. But the stress
detection before it causes high damage to human needs
prevention.
III. PROPOSED SYSTEM
In the proposed system, non-wearable sensors are
used to detect stress without bothering the person and
without misleading with faked symptoms. This system
not only uses facial 9 expressions but also facial cues
and heart rate from non-wearable sensors like System
Camera, Intel Realsense D435 Camera which has
depth sensing technology. The proposed system would
help in continuous monitoring of the subject so that
exact predictions can be found. The proposed System
uses Machine Learning algorithms to classify stress.
Image Processing is used at the initial stage for
detection, the student’s image is given by the browser
which serves as input . A single GUI is made to detect
a person through facial recognition and detect facial
cues and find heart rate.
IV. METHODOLOGY
Using web camera, we will be identifying the
person (student). We will identify the facial pattern of
the person in front of the camera and from those facial
patterns we will consider eyebrows, eyes and lips
values and normalize those to get the expression using equipment named Microsoft Kinect Sensor V1 and by
a pre-trained deep learning model. The model is loaded implementing methods such as SVM and Random
and utilized to classify the detected facial expressions. forest classifiers are used to evaluate the classification
This model has been trained on a dataset of facial performance of the selected features, by taking
expressions to identify the emotions. The facial region parameters such as Extracted 1421 features based on
is pre-processed, and the pre-processed image is then four phases of a normal respiration cycle. The
passed through the pre trained model to obtain the conclusion is that Recognition accuracy for
predicted emotion probabilities. If the predicted recognizing psychological stress, physical stress, and
emotion label falls into the 10 categories of “scared”, relaxing states shows that the framework is effective.
“sad”, or “angry”, it is considered as a stressed In the paper by Kempfle [11], he 13 examines the
emotion. Otherwise, it is considered as a non-stressed impact of the most prominent parameters to estimate a
emotion. Using the web camera, we also focus on a person’s respiration rate from depth imaging using the
specific area of the face such as the forehead, which is equipment Kinect V2 as depth sensor and Kinect sdk
known to show changes related to the heart rate. We 2.0. The methods applied in this experiment are Noise
then analyze the shades of green in that area calculates filters, Signal-to-noise ratio, box-plotting. The
the average level of intensity. Using a simple formula, parameters taken into consideration are distance to the
the heart rate of the person is estimated. user, region of interest, respiration rate, and sampling
frequency. The results show that the distance and ROI
size matter significantly for the signal quality. The
V. LITERATURE SURVEY introduction of the paper by Soleimani [16] is,
identifying facial cues which are helpful for Stress
Detection, the equipment used here is Web Camera,
Stress definition is still debated. Many researchers
the methods applied here are Identifying the facial
have worked to find human stress levels in different
Expression using the Facial cues and face_utils library.
environments with the help of depth cameras and
So, the conclusion is that eight real valued and seven
managed to achieve a lot. Whether the person is in
binary parameters were successfully extracted from 97
stress or not cannot be detected so easily without
subjects (467 facial images) for seven different
considering many features like for example if we
expressions (neutral, angry, disgust, fear, sad, happy,
consider only emotional features which has the
and surprised).
possibility of getting manipulated results but if we
consider physiological features like respiration rate and
facial cues along with emotional features then that
result will be quite accurate. It is hard to detect stress
with less features. Some of the researchers use sensors VI. OBJECTIVES
like contact / non-contact devices which helps to take
these physiological features. Many studies have been
These days stress has been a major problem for any
conducted in this field. The paper by A.L.Fielding [3]
age group of people. Irrespective of the age everyone
tells about Intel RealSense D415 depth camera which
is experiencing stress in a different way. This stress is
has been investigated for measuring the respiratory
leading to many health problems, especially people
motion of the body for cancer patients. The equipment
who are facing mental challenges. So, it is important to
used here is Intel RealSense D415 Camera. The
monitor stress in real time and treat it properly. Hence
methods applied in this experiment are precision of
a system is proposed to detect stress using facial
Depth Measurement-s and Camera System Latency
expressions, facial cues, and their breathing patterns as
Measurement and the parameters considered are
indications to identify whether the person is in stress or
Latency of the camera was measured. The concluded
not.
results shown are that the camera data showed good
results when compared with a NDI Polaris Spectra
system. The paper by Massaroni [12] tells about
System consisting of a laptop’s built-in RGB camera
and the recording of the chest movements of a subject,
changing waveform indicating respiratory pattern. The
equipment used is the laptop’s built-in RGB camera
algorithm for post-processing of acquired video data.
Few methods used are Absolute error, standard error,
percentage error and Bland–Altman analysis. This
paper considered a few Parameters Both males and
females seated in front of the webcam, wearing slim-fit
and loose-fit shirts. The conclusion is the measuring Fig. 1. The architecture of the proposed model
system shows better performance on females. In the
paper by Yuhao [15] non-contact devices are used to The GUI interface will utilize computer vision to
identify human stress and the relationship between detect and track multiple human subjects in real-time.
RSPS and stress recognition is analyzed using Image processing methods will be employed to
analyze facial expressions, body postures, and
gestures. The system is designed to handle a variable
number of people within the camera's field of view.
Techniques like object detection and deep learning will
be applied to identify and label individual persons. The
GUI will display bounding boxes around each detected
person to assist with identification. Advanced neural
networks will be implemented to differentiate between
stress-related facial expressions and normal ones. The
system may also 15 utilize infrared or thermal imaging
to capture physiological signals associated with stress.
Tracking algorithms will ensure that the GUI
maintains accurate identification as people move
within the camera frame. The GUI interface will
provide real-time stress level assessments for each
detected individual.
Our GUI will use advanced facial recognition
algorithms to classify various facial expressions
accurately. The system will identify common facial
expressions associated with stress, such as frowns,
furrowed brows, and clenched jaws. It will also
recognize positive expressions like smiles, indicating
moments of relaxation. GUI interface will display real-
time updates of detected facial expressions alongside
heart rate data. It will provide insights into how
different expressions correlate with changes in heart
rate, aiding stress analysis. The system will consider
multiple facial cues, including eye movements, mouth
shapes, and forehead tension, for accurate
classification. The GUI offers stress management
techniques based on an individual's unique facial
expression patterns. It aims to create an innovative
GUI interface that seamlessly integrates facial
expression classification and heart rate analysis to
provide users with an insightful and reliable stress
management tool.
Identifying stress values is the core focus of the GUI
interface, which aims to create an innovative and
reliable tool for stress assessment through a camera-
based approach using heart rate and facial expressions
analysis. The GUI interface utilizes advanced
computer vision and machine learning techniques to
detect and classify facial expressions associated with
stress, such as frowns, furrowed brows, and tense
facial muscles. Additionally, the system analyzes heart
rate data obtained through the camera to capture
physiological changes indicative of stress, providing a
comprehensive stress assessment. By combining real-
time video feeds and heart rate analysis, the GUI offers
continuous monitoring of stress levels, helping
individuals understand their stress patterns and take
appropriate measures for stress management. Machine
learning models are 16 trained to recognize individual
variations in facial expressions, enhancing the
accuracy of stress identification personalized to each
user. To ensure real-time feedback, the GUI interface
instantly displays stress values, enabling users to
manage stress effectively and make necessary
adjustments to their daily routine. Advanced feature
extraction techniques are employed to capture subtle
changes in facial expressions, aiding in the
identification of stress-related cues. The GUI offers
visual feedback with bounding boxes around detected
facial expressions and real-time heart rate readings,
providing a clear and user-friendly interface. The GUI
interface may integrate with wearable devices to
provide continuous heart rate monitoring for improved
stress analysis. The interface is adaptable and
accessible to users of all technical backgrounds,
making it user-friendly and easily navigable. In
addition to identifying stress values, the GUI may offer
recommendations for stress relief techniques based on
stress profiles. Heart rate analysis and facial expression
classification complement each other, creating a
comprehensive stress assessment tool. The project's
GUI interface strives to be at the forefront of stress
identification technology, offering users a
comprehensive tool for stress assessment and
management through heart rate and facial expression
analysis
We aimed to create an efficient stress detection system
by applying algorithms on combined data from facial
expressions and heart rate obtained through a camera.
Leveraging Python libraries and advanced computer
vision techniques, real-time video feeds are processed
and facial features, such as frowns, furrowed brows,
are extracted to assess stress related expressions.
Concurrently, Python libraries are utilized to analyze
heart rate data obtained from the camera's monitoring
capabilities. To achieve a comprehensive stress
assessment, we integrated the facial expression and
heart rate data, effectively combining physiological
and behavioral indicators of stress. Through Python-
based machine learning algorithms, we developed a
predictive model to classify stress levels based on the
combined data. For stress detection , several Python
libraries that are commonly used in computer vision,
machine learning, and data analysis tasks are
employed. These libraries provided us with the
necessary tools and algorithms to process facial
expressions and heart rate data obtained through the
camera and perform stress classification.
VII. UML DIAGRAM
The use case diagram is used to identify the
primary elements and processes that form the System.
The primary elements are termed as "actors" and the
processes are called "use cases." The use case diagram
shows which actors interact with each use case.
 8. Stress Categorization:
- Categorize stress levels as "Low," "Moderate," or
"High" based on the calculated stress value thresholds.

9. Real-Time Visualization:
- Display the detected emotions, stress value, stress
level, and heart rate on the video frame.
- Continuously update these visualizations as new
video frames are processed.

10. Average Stress Calculation:

- Calculate and display the average stress value over
Fig 2. Use case Diagram
the entire assessment period.
- Provide users with insights into their stress levels
during the session.
VIII. ALGORITHM
11. Stress Management Tips:
Here's an overview of the steps involved: - Offer stress management tips, including deep
breathing, walking, counting, and stretching.
1. Initialization:
- Enhance user experience by providing practical ways
- Import necessary libraries and initialize the user
to alleviate stress.
interface using Tkinter.
- Load required modules and resources, including
12. Graphical Representation:
facial landmarks, emotion recognition model, and
- Plot stress values over time using a red dot plot,
image assets.
visualizing fluctuations in stress levels.
- Display the stress level classification in graphical
2. User Interface:
form for enhanced understanding.
- Create a graphical user interface (GUI) with a title,
background image, and a "Stress Detection" button.
13. Completion and Clean-Up:
- Define the layout and positioning of interface
- Terminate the stress detection process when the user
elements using the Tkinter library.
presses the 'q' key.
- Close video capture and release resources to ensure
3. Stress Detection:
proper clean-up.
- Upon clicking the "Stress Detection" button, initiate
Convolutional Neural Network [CNN]: CNN is the
the stress detection process.
most well-known image recognition and satisfaction
- Access the webcam to capture real-time video frames
algorithm. o It is the key innovations that led to the
of the user's face.
deep neural network renaissance in computer vision,
which is a subset of machine learning.
4. Facial Landmarks:
- Employ the dlib library to detect facial landmarks, Using CNNs for stress detection through facial
including eyebrows and lip points. expressions involves training a model to analyze facial
- Utilize these landmarks to measure eye and lip images and predict stress levels based on the detected
distances, crucial for stress assessment. features.
A typical CNN consists of a combination of
5. Emotion Recognition:
convolutional, pooling and dense layers [Ref Fig 2]
- Deploy a pre-trained emotion recognition model to
classify emotions from facial expressions.
- Determine whether the detected emotions are
indicative of stress, such as anger, sadness, or fear.

6. Heart Rate Calculation:

- Calculate the user's heart rate using color signals
from the video stream.
- Extract the green channel and measure mean intensity
within a designated region of interest (ROI).

7. Stress Level Calculation:

- Calculate normalized stress values based on eye
distance, lip distance, and heart rate.
- Combine these values to generate a comprehensive
stress value.

Fig 3. Convolutional Neural Network Different Types Layers
IX. IMPLEMENTATION AND TESTING
In the provided code, classifying facial expressions
is implemented using a pre-trained deep learning
model. The code utilizes the "emotion_classifier"
loaded from a trained model file
("_mini_XCEPTION.102-0.66.hdf5"). This model has
been previously trained on a large dataset containing
facial expression images associated with different
emotions. 23 During the real-time video processing,
the code captures facial regions of interest from the
camera feed. These regions are then pre-processed and
resized to match the input size expected by the
"emotion_classifier" model. The pre-processed facial
region is converted into a numpy array and expanded
to create a batch of size one for prediction. The
"emotion_classifier" model, being a deep learning-
based model, employs a convolutional neural network
(CNN) architecture that has learned to recognize and
classify facial expressions. The model takes the pre-
processed facial region as input and outputs a
probability distribution across various emotion classes,
such as "angry," "happy," "sad," "surprised," etc. The
emotion with the highest probability is considered the
predicted facial expression . For example, if the model
predicts a high probability for "happy," it means the
individual's facial expression is classified as happy.
Similarly, if the model predicts a high probability for
"angry," the facial expression is classified as angry. To
distinguish stress-related expressions, the code then
checks if the predicted emotion falls into the categories
of "scared," "sad," or "angry." If the emotion matches
any of these categories, the code labels the facial
expression as "Stressed." Otherwise, the expression is
classified as "Not Stressed." The classification result is
then displayed on the GUI interface, providing real-
time feedback on the person's facial expression and
whether they are displaying signs of stress. In
summary, classifying facial expressions in the
provided code is achieved through a deep learning-
based model ("emotion_classifier") trained on facial
expression datasets, and it plays a crucial role in
determining stress levels during the stress detection
process.
We have identified the stress value through facial
cues and heart rate using a combination of facial
feature analysis and physiological heart rate
calculations.
Facial Cue Analysis: We utilized facial landmarks
detection to extract important facial cues, such as the
distance between eyebrows and the distance between
the upper and lower lips. These facial cues serve as 24
indicators of stress-related facial expressions. The
distances are normalized by dividing them by the
maximum distance observed during the analysis. This
normalization ensures that the values are scaled
appropriately and helps in comparing different
individuals' facial cues.
Here, a combination of algorithms is applied for stress
detection through facial expressions and heart rate
analysis.This model processes real-time video frames
and predicts the most probable emotion associated
with the facial expression. For heart rate calculation,
signal processing techniques are applied to analyze
color signals from the video feed, focusing on the
green channel. A region of interest (ROI), typically the
forehead area, is selected to estimate the individual's
average heart rate using image processing methods.
Next, normalized values of facial cues (such as
distances between facial features) and heart rate are
combined using weighted averaging. The weights
assigned to each metric reflect their significance in
stress detection. By computing the stress value, stress
levels are determined, and predefined thresholds
categorize them as "Low," "Moderate," or "High."
Overall, this integrated approach of deep learning,
signal processing, and weighted averaging enables
real-time stress assessment based on facial expressions
and heart rate data obtained through the camera feed.
X. DESCRIPTION OF TECHNOLOGY USED
A. TKINTER
Tkinter is the standard GUI library for Python.
Python when combined with Tkinter provides a fast
and easy way to create GUI applications. Tkinter
provides a powerful object oriented interface to the Tk
GUI toolkit. Creating a GUI application using Tkinter
is an easy task. All you need to do is perform the
following steps
• Import the Tkinter module.
• Create the GUI application main window.
• Add one or more of the above-mentioned widgets
to the GUI application.
 • Enter the main event loop to act against each It is a pre-trained model file used for facial
event triggered by the user. landmark detection. It is specifically designed to detect
68 facial landmarks or key points on a human face.
Facial landmark detection is an important task in
B. CV2 (OPENCV) computer vision that involves identifying key points on
a face, such as the corners of the eyes, the tip of the
OpenCV is the huge open-source library for
nose, the corners of the mouth, and so on. These
computer vision, machine learning, and image
landmarks serve as reference points for various facial
processing and now it plays a major role in real-time
analysis tasks, including face alignment, expression
operation which is very important in today’s systems.
recognition, and facial pose estimation. The
By using it, one can process images and videos to
`shape_predictor_68_face_landmarks.dat` file is
identify objects, faces, or even handwriting of a
trained using a machine learning algorithm, often
human. When integrated with various libraries, such as
based on shape regression, on a large dataset of
NumPy, python can process the OpenCV array
annotated facial images. The model learns to predict
structure for analysis. To Identify image pattern and its
the coordinates of the 68 facial landmarks given an
various features we use vector space and perform
input face image. 29 Once loaded, the model can be
mathematical operations on these features. The cv2
used to estimate the positions of these facial landmarks
package is extensively used for video processing and
on a given face image. The landmarks are typically
computer vision tasks in the program. It provides
represented as (x, y) coordinates, allowing for further
functions for capturing video frames from a webcam
analysis and manipulation of specific facial regions.
(cap.read()), resizing frames (imutils.resize()),
The `shape_predictor_68_face_landmarks.dat` file is
converting frames to grayscale (cv2.cvtColor()),
commonly used in conjunction with libraries or
drawing contours and text on frames
frameworks like dlib, OpenCV, or Python's
(cv2.drawContours() and cv2.putText()), and
face_recognition library to perform facial landmark
displaying frames (cv2.imshow()). OpenCV is crucial
detection tasks. Other pre-trained models available for
for face detection, facial landmark estimation, and
facial landmark detection, including models with a
drawing visual feedback on the video frames.
different number of landmarks (e.g., 5, 68, or even
C. MINI_XCEPTION more), depending on the specific requirements of the
application.
The term "mini_XCEPTION" refers to a specific
deep learning architecture used for facial expression E. EB DISTANCE
recognition. It is a modified version of the Xception
The EBDIST function calculates the Euclidean
(Extreme Inception) 28 architecture, which is a
distance between the eyebrow landmarks of a detected
convolutional neural network (CNN) model originally
face. The term "EB" stands for "eyebrow" in this
introduced for image classification tasks. The
context. The ebdist function takes two parameters, leye
mini_XCEPTION architecture is designed to
and reye, which represent the landmarks corresponding
efficiently capture and analyze facial features to
to the left eyebrow and the right eyebrow, respectively.
recognize different facial expressions. It is specifically
These landmarks are obtained from the facial
tailored for the task of facial expression recognition,
landmarks detected by the shape_predictor function
which involves categorizing facial images into various
from the dlib package. By calculating the Euclidean
emotion classes such as happy, sad, angry, etc. The
distance between the left eyebrow and right eyebrow
mini_XCEPTION architecture typically consists of
landmarks, the ebdist function quantifies the separation
multiple convolutional layers, followed by batch
or length between the eyebrows. This distance can
normalization and activation functions. It employs a
provide an indication of certain facial features or
combination of depth wise separable convolutions and
expressions, which is relevant for stress detection in
pointwise convolutions to reduce the computational
the program. The calculated eyebrow distance is then
complexity while maintaining expressive power. This
appended to the points list, which keeps track of the
helps in efficient feature extraction from facial images
eyebrow distances over time or frames. These
and enables the model to learn discriminative features
distances are used as a factor in stress level calculation,
related to different facial expressions. In the provided
along with other factors such as lip distance and heart
program, the mini_XCEPTION architecture is loaded
rate. Overall, the ebdist function helps in measuring
from a pretrained model file named
and monitoring the eyebrow distance as a feature for
"_mini_XCEPTION.102-0.66. hdf5 ''. This pre-trained
analyzing facial expressions and determining stress
model has been trained on a large dataset of labeled
levels in the provided program.
facial expression images, allowing it to recognize and
classify facial expressions accurately. By utilizing the F. FACE_RECOGNITION
mini_XCEPTION architecture, the program is able to
Face-Recognition library built using the “dlib”
leverage the learned representations and predictions of
library, which is built using C++ Language. Face-
the model to detect and analyze facial expressions in
Recognition library is easy to import. It has some
real-time video frames.
amazing methods. Recognize and manipulate faces
D. SHAPE PREDICTOR from Python or from the command line with the
world's simplest face recognition library. Built using person, and whether the person is stressed or not based
dlib's state-of-the-art face recognition built with deep on the facial expressions is shown in the message box.
learning. The model has an accuracy of 99.38% on the
Labeled Faces in the Wild benchmark.
G. FACE_UTILS
This is an open source wrapper library for the most
common face detection models. It also provides
multiple face utilities such as face cropping. Features:
Face Detection: By creating an object of the face
detection model, you want you can use it to detect the
faces in images.I have also implemented outputs forms
such as: 1: (x,y,w,h) and 2: (top,right,bottom,left) Face
cropping: Ability to crop the face from the image
given the detector result. You can also zoom in and out
from the face.

Fig 5. Face Expression 1 GUI

XI. DATASETS DESCRIPTION In the fig 5 the person’s face is recognized by the
camera. Then, the expression of the person is evaluated
The model relies on pre-trained models and does and the lip distance and eyebrow distance is calculated.
not require a specific dataset during runtime. The Additionally, heart rate is also calculated based on
model loads a pre-trained model from the file changes in the intensity of the green color channel in a
"_mini_XCEPTION.102-0.66. hdf5" using the specific region of interest (ROI) on the face.
load_model function. This model is expected to Evaluating the person’s stress value, it shows whether
contain the learned weights and architecture necessary the person is stressed or not.
for facial expression recognition. The dataset used to
train this pre-trained model would have been utilized
during the training phase, which is typically done
separately. The dataset used for training facial
expression recognition models could vary, and there
are several publicly available datasets commonly used
for this purpose, such as CK+, FER2013, or RAF-DB.
(dataset for mini x)(ibug).

XII. RESULTS ANALYSIS

The user interface gives us the results for the facial

recognition, facial expression, and facial cues. The
Face recognition and face expression outputs are
stored in a csv formatted file. The output of the facial
cues is shown on the GUI screen itself.
Fig 6. Face Expression 2 GUI
In the fig 6 the person’s face is more relaxed as the
person seems to be smiling. In this picture , the
eyebrow distance, lip distance and heart rate are
calculated and evaluated accordingly. It evaluated
accordingly and hence gave the conclusion that the
person is “Not Stressed”.

XIII. CONCLUSIONS AND FUTURE WORK

In this project, features like heart rate, facial cues

Fig 4. Face Recognition GUI
In face recognition and facial pattern recognition, once
the face is detected, the expression of the specific
and facial expressions are used for detecting stress. In
traditional systems, the data collected by people
perceptive like themselves, physiologist experts and
external observers and then classified the person as
stress or normal, which is a biased classification. In
this project to avoid the data bias, data collected from
web cameras for all features like face emotions and
facial cues is used. Facial expressions, Facial cues and
Heart Rate calculation, we use web cameras and for
future enhancements, modifications should be done
such that all the values are taken using a single camera.
If all the values can be collected from a single camera,
integration can be achieved. Need to apply the
different types of classification algorithms to increase
the accuracy and compare for analysis. Heart Rate
calculation is based on image processing techniques
applied to the video frames. Specifically, the heart rate
is estimated using changes in the intensity of the green
color channel in a specific region of interest (ROI) on
the face. The heart rate calculation does not rely on a
specific module or package but is implemented using
mathematical operations and the OpenCV library for
video processing.
REFERENCES
[1] Bolhari, A. Rezaeian, J. Bolhari, and S. Bairamzadeh,
“Occupational Stress Level among Information
Technology Professionals in Iran,” International Journal
of Information and Electronics Engineering, vol. 2,
issue 5, September 2012.
[2] B Padmaja et al., “Tree Net analysis of human stress
behaviour using sociomobile data,” Journal of big data,
vol. 3, issue 1, pp. 1-24, 2016.
[3] Fielding, Andrew & Pandey, Ajay & Jonmohamadi,
Yaqub & Via, Riccard Weber, Damien & Lomax,
Antony & Fattori, Giovanni. (2020). Preliminary Study
of the Intel RealSense D415 Camera for Monitoring
Respiratory Like Motion of an Irregular Surface. IEEE
Sensors Journal. PP. 1-1. 10.1109/JSEN.2020.2993264.
[4] G.-C. Enrique, V. Osmani, and O. Mayora, “Automatic
stress Detection in working environment from
smartphones’ accelerometer data: A first step,” IEEE
Journal of Biomedical and Health Informatics, vol.20,
issue 4, pp. 1053-1060, 2016.
[5] Gimpel et al., “my Stress: Unobtrusive smartphone -
based Stress detection,” in Proc. the 23rd European
Conference on Information Systems (ECIS 2015),
Münster, Germany, May 2015.
[6] H. Lu et al., “Stress Sense: Detecting stress in
unconstrained acoustic environments using
smartphones,” in Proc. ACM Conference on Ubiquitous
Computing, pp. 351- 360, 2012
[7] https://
biomedicalengineeringonline.biomedcentral.com/
articles/10.1186/1475https://biomedical-engineering-
online.biomedcentral.com/articles/10.1186/1475-925X-
8- 16925X-8-16
[8] ttps://www.hopkinsmedicine.org/health/conditions-
anddiseases/
vitalsignsbodytemperaturepulsecanraterespirationratebl
oodpressure#:~:text=Normal%20respiration%20rates
%20for%20an,to%2016%20breaths%20per%20minute.
[9] https://www.researchgate.net/publication/
305846036_Stress_and_anxiety_detection_using_facial
_cancues_from_videos
[10] J.-H. Kim et al., “The Impacts of physical exercise on
stress coping and well-being in university students in
the context of leisure,” Journal on Health, vol. 6, 2014.
[11] Kempfle, Jochen & Van Laerhoven, Kristof. (2018).
Respiration Rate Estimation with Depth Cameras: An
Evaluation of Parameters. 1-10.
10.1145/3266157.3266208.
[12] Massaroni C, Lo Presti D, Formica D, Silvestri S,
Schena E. Non-Contact Monitoring of Breathing Pattern
and Respiratory Rate via RGB Signal Measurement.
Sensors.j2019; 19(12):2758.
https://doi.org/10.3390/s19122758
[13] P. Kostopoulos, A. I. Kyritsis, M. Deriaz, and D.
Konstantas, “Stress detection using smart phone data,”
eHealth 3600 International Summit on eHealth,
Budapest, Hungary, June 14-16, pp. 340-351,2016.
[14] Rebeca M. Torrente-Rodríguez, Jiaobing Tu, Yiran
Yang, Jihong Min, Minqiang l llWang, Yu Song, You
Yu, Changhao Xu, Cui Ye, Waguih William IsHak, Wei
Gao. Investigation of Cortisol Dynamics in Human
Sweat Using a Graphene-Based Wireless mHealth
System. Matter, 2020; DOI: 10.1016/j.matt.2020.01.021
[15] Shan, Yuhao & Li, Shigang & Chen, T. (2020).
Respiratory signal and human stress: non-contact
detection of stress with a low-cost depth sensing
camera. International Journal of Machine Learning and
Cybernetics.
[16] Soleimani, Manuchehr & Shipley, Rebecca & Smith,
Nic & Mitchell, Cathryn. (2009). Medical imaging and
physiological modelling: Linking physics and biology.
Biomedical engineering online. 8. 1.
10.1186/1475925X-8-1. [17] V. Bhelkar, D. K. Shedge,
"Different types of wearable sensors and health
monitoringfsystems: A survey", Proc. 2nd Int. Conf.
Appl. Theoretical Comput. Commun.fTechnol., pp. 43-
48,2016.