Emotion Detecting Robot

DEPT. OF ECE, Emotion detecting robot Feb 21, 2024 1

 OVERVIEW
• INTRODUCTION
• LITERATURE SURVEY
• MOTIVATION
• OBJECTIVE
• PROPOSED SYSTEM
• HARDWARE ARCHITECTURE
• SOFTWARE IMPLEMENTATION
• RESULTS
• CONCLUSION
• PUBLICATIONS
DEPT. OF ECE, Emotion detecting robot
• REFERENCES
2 Feb 21, 2024
 INTRODUCTION
• A personal assistant, also known as voice-based artificial intelligent system is one of
the hot topics in the current world.
• They are programmed to listen to human’s verbal command and responds to them
which makes it a human-computer device interaction.
• They can provide voice interaction, music playback, making to-do lists, setting
alarms and other real-time information.
• Helpful for physically challenged people, elders and people with mental disorders.
• Currently available systems does not have the feature to give emotional assistance.
• Here a system is proposed which additionally serves to be an emotional assistant
who can recognize emotions in people and respond to people accordingly.

DEPT. OF ECE, Emotion detecting robot

3 Feb 21, 2024
 INTRODUCTION

• Emotion recognition is the process of identifying human emotion.

• Emotions are reflected in voice, hand and body gestures,and mainly
through facial expressions
• Emotion detection helps to provide human computer interaction.
• Understanding emotions and knowing how to react to people’s
expressions greatly enriches the interaction.
• Emotions of a person are recognized using his/her behavioral
characteristics.
• Voice, handwriting, facial expressions, brain signals (EEG), heart
signals (ECG) etc.
DEPT. OF ECE, Emotion detecting robot Feb 21, 2024
4
 INTRODUCTION

• Need for emotion detection?

People with mental disorders are often
exposed to severe emotional stress during
daily work.
Emotional intelligence refers to the ability to
identify and regulate our own emotions

DEPT. OF ECE, Emotion detecting robot 5 Feb 21, 2024

 LITERATURE SURVEY
PAPER TECHNOLOGY AUTHOR YEAR
Emotion Detection and  DWT Charvi Jain 2018
Characterization using • Image into wavelets
Facial Features [10] • Wavelet coefficient spectrum-signal data points.
• A data vector Scaling of wavelets creating respective data
vectors
 SVM
• Hyperplane
• Kernel trick

Facial Emotion  Shallow CNN Sachin Saj 2019

Recognition Using Shallow Tk
CNN[11]

Facial expression  Convolution operation on the input, followed by 4 residual Bin Li 2021
recognition via ResNet-50 blocks, and finally performed full connection operation
DEPT. OF ECE, Emotion detecting robot 6 Feb 21, 2024
 LITERATURE SURVEY
PAPER FEATURES AUTHOR YEAR
Olivia, the smart home o Face detection Anisha Sachdeva 2019
virtual assistant o Home automation [6]

A smart virtual assistant o Numerical data such as

about covid 19 number of covid 19
cases ,number of deaths. Yusuf Ugurlu 2020
o Frequently asked questions [8]
about covid 19

Visual travel assistant for o Tour guide

visually impaired people o Details of places such as
opening hours, rating, N S Priya 2020
location [7]
o Interacts with the user

DEPT. OF ECE, Emotion detecting robot 7 Feb 21, 2024

 LITERATURE SURVEY
PAPER FEATURES AUTHOR YEAR
Smart robotic personal o Wheel chair for physically
assistant vehicle challenged
o Voice controlled robot Dhiraj Sunehra
vehicle 2021
o Articulates the text from a [9]
captured image and present
the equivalent audio

Speech Emotion Analysis o Accelerates the convergence

of Netizens Based on speed of network Lei yang 2021
Bidirectional LSTM and o Improved robustness [10]
PGCDBN

Desktop based smart voice o Text to speech

assistant using python o Speech to text Jesudoss 2022
integrated with Arduino o Language processing [12]
o Voice recognition
DEPT. OF ECE, Emotion detecting robot 8 Feb 21, 2024
 OBJECTIVE

• To setup a system which can monitor its surroundings and detect peoples emotions.
• To respond to the user according to detected emotions.

DEPT. OF ECE, Emotion detecting robot 9 Feb 21, 2024

 PROPOSED SYSTEM

10 Feb 21, 2024

DEPT. OF ECE, Emotion detecting robot
 BASIC WORKING

System
In System
Camera Emotion Response to
wakeup analyzes
input Detection the user
mode the input

DEPT. OF ECE, Emotion detecting robot 11

Feb 21, 2024
 BASIC WORKING

• The system basically works in two modes.

Sleep mode
Wakeup mode
• When the system is in wake up mode, it constantly monitors its surrounding
and the people around.
• If it gets a command from the user , it does the necessary action in response.
• This system will also have the ability to do emotional analysis of people
around and talk to them as a companion does to comfort them.
• The system can also be put in sleep mode through voice command.

DEPT. OF ECE, Emotion detecting robot 12 Feb 21, 2024

 HARDWARE ARCHITECTURE
PI POWER
AUDIO
CAMERA SPEAKER DISTRIBUTION
AMPLIFIER

MOTOR
CONTROLLER
RASPBERRY PI

RGB LED(EYES) MOTOR

DEPT. OF ECE, Emotion detecting robot 13

Feb 21, 2024
 HARDWARE ARCHITECTURE

• The input is taken from camera.

• Two RGB OLED displays serves as eyes.
• Motors are attached to limited movement.
• Motors are driven by a motor driver.
• Raspberry pi serves as the processor.
• Speaker and an audio amplifier is connected to the raspberry pi.
• A power supply of 5 V drives the circuit

DEPT. OF ECE, Emotion detecting robot Feb 21, 2024

14
 HARDWARE COMPONENTS
Component Specification

Micro processor Raspberry pi 4

Motor driver DRv8833 12 V

Camera PICAM

Motor 6V BO Motor

Speaker PWM amplified speaker

DEPT. OF ECE, Emotion detecting robot 15 Feb 21, 2024

 HARDWARE COMPONENTS
Raspberry pi 4 model b
• This product’s key features include a
• high-performance 64-bit quad-core processor,
• dual-display support at resolutions up to 4K via a pair of
micro-HDMI ports,
• 8GB of RAM, dual-band 2.4/5.0 GHz wireless LAN,
• Bluetooth 5.0, Gigabit Ethernet, USB 3.0, and PoE
capability
OLED 128X64 Display
• High contrast
• This display is made of 128x64 individual white OLED
pixels.
DEPT. OF ECE, Emotion detecting robot 16 Feb 21, 2024
 HARDWARE COMPONENTS

BO Motor
• Bo motor (Battery Operated) lightweight DC geared motor which gives good
torque and rpm at lower voltages.
• This motor can run at approximately 200 rpm when driven by a single Li-Ion cell.
DRv8833 Motor driver
• DRV8833 is a dual H-bridge motor driver IC that can be used for bidirectional
control of two brushed DC motors at 2.7 V to 10.8 V.

DEPT. OF ECE, Emotion detecting robot

17 Feb 21, 2024
 SOFTWARE FLOWCHART

Analyze Fetch Face Feature extraction

environment frame detection

NO
classifier Display
emotion

Response as
voice

Face
Dataset Feature extraction
detection

DEPT. OF ECE, Emotion detecting robot Feb 21, 2024 18

 EXPLANATION

• The system continuously analyzes the environment.

• If any face is detected, relevant features are extracted.
• If there is no face detection happening, then the system continuous to fetch
frame.
• The extracted features are fed to the classifier.
• The classifier uses an algorithm is already trained using a standard dataset.
• The classifier compares the input with trained parameters and thus determine and
classify the associated emotion.
• After emotion classification, the output is displayed on the OLED display.
DEPT. OF ECE, Emotion detecting robot 19 Feb 21, 2024
• The system is programmed to produce a voice output according to detected
emotion.
 EMOTION RECOGNITION
USING RESNET 50

DEPT. OF ECE, Emotion detecting robot 20 Feb 21, 2024

 RESNET 50

o RESNET 50 is a 50 layer convolutional neural network.

o It consists of
• A 7*7 kernel convolution
• 4 residual blocks
• Fully connected layers using softmax activation function.

DEPT. OF ECE, Emotion detecting robot 21 Feb 21, 2024

 RESNET 50
• Convolution operation on the input, followed by 4 residual blocks, and finally
performed full connection operation to achieve classification tasks.
• Convolution and pooling is the process of feature engineering, local
amplification and local feature extraction,
• The FC layer can be thought of as feature weighting.
• The structure of the FC layer is usually a way to quickly learn the nonlinear
combinations of advanced attributes generated by the convolutional layer.
• Fully connected (FC) layer usually appears at the end of the CNN to summarize
the features of the previous layers
• The FC layer will learn a possible nonlinear function.

DEPT. OF ECE, Emotion detecting robot 22 Feb 21, 2024

 METHODOLOGY OF RESNET 50

• First, the image, is converted into a form suitable for multilevel perceptron
• It is flattened into column vectors and fed back to the feed forward neural network.
• The flattened data is then applied to each iteration of the training.
• The model has the ability to distinguish between the major features in the image and
some low-level features.
• Then classify them through classification techniques such as Softmax.
• Here it will give output as the classification results of the seven expressions

DEPT. OF ECE, Emotion detecting robot Feb 21, 2024 23

 EMOTION RECOGNITION
Training data Live data

Feature
extraction Preprocessing

Promisi no Extract
ng selected
feature features

yes
Add to selected
feature list

Training with
Classification
resnet 50
DEPT. OF ECE, Emotion detecting robot Feb 21, 2024
24
 SOFTWARE REQUIREMENTS

• Python
Python is a high-level, general-purpose programming language
• Thonny IDE
Thonny is an integrated development environment for Python
• Arduino IDE
The Arduino Software (IDE) makes it easy to write code and upload it to the board offline.
• VNC Viewer
VNC is remote access and control software for an unrivalled mix of Windows, Mac, UNIX
and Linux computers with a simple peer-to-peer architecture.

DEPT. OF ECE, Emotion detecting robot 25

Feb 21, 2024
 SOFTWARE REQUIREMENTS

• Advanced IP scanner
Advanced IP Scanner allows network admins to have access to network scanning,
retrieving IP and MAC addresses.
• PuTTY
PuTTY is a free implementation of SSH and Telnet for Windows and Unix platforms,
along with an xterm terminal emulator.

DEPT. OF ECE, Emotion detecting robot 26 Feb 21, 2024

 DATASET

• Face Emotion Recognition(FER) 2013 dataset.

• FER 2013 contains approximately 30,000 facial images of different expressions.
• It can be divided into 7 types: 0=angry, 1=disgust, 2=fear, 3=happy, 4=sad, 5=surprise,
6=neutral.

Feb 21, 2024

DEPT. OF ECE, Emotion detecting robot 27
 HARDWARE DEVELOPMENT

DEPT. OF ECE, Emotion detecting robot

Feb 21, 2024
28
 HARDWARE STRUCTURE

INDICATOR LED

OLED DISPLAY

DEPT. OF ECE, Emotion detecting robot Feb 21, 2024

29
 RESULTS

DEPT. OF ECE, Emotion detecting robot Feb 21, 2024

30
 EMOTION DETECTION

DEPT. OF ECE, Emotion detecting robot 31 Feb 21, 2024

 RESULTS
RESNET 50

DEPT. OF ECE, Emotion detecting robot

DEPT. OF ECE, Multipurpose AI Robot 32 Feb 21, 2024

 VOICE OUTPUT

DEPT. OF ECE, Emotion detecting robot 33 Feb 21, 2024

 FUTURE SCOPE

• Can add a feature for displaying the percentage of emotions at the end of the day.
• Can add an extended feature to detect depression.
• Can add fetures helpful for physically and mentally disabled people and make it as a
multi functional robot.

DEPT. OF ECE, Emotion detecting robot 34 Feb 21, 2024

 CONCLUSION
• To give comfort to the mentally depressed people and physically disabled people, a
system which can do emotion detection is designed and developed.
• Emotion recognition based on RESNET 50 model is done and obtained accuracy as
86%.
• Hardware architecture and working model is proposed and developed.
• Hardware software integration and complete system is developed aand deployed.

DEPT. OF ECE, Emotion detecting robot

35
Feb 21, 2024
 THANK YOU

DEPT. OF ECE, Emotion detecting robot 36 Feb 21, 2024