Human Emotion Detection with Speech Recognition
Using Mel-frequency Cepstral Coefficient and CNN
Akshat Kumar Tiwari
SRM institute of science and SRM institute of science and
technology, SRM university technology, SRM university
at7989@srmist.edu.in
Abstract— Speech is one of the most common and effective
modes of communication, as one not only gets the message but
also intent, motive and emotions of the individual communicating.
It’s not only the facial expression but also the signals generated by
one’s speech that helps us in recognizing the emotion of the
individual. The main hurdle that emotion recognition systems face
is the fact that people have varied emotional response, when
exposed to the same stimuli. We have used dataset from --. It
is also observed that using cnn along with mfcc improves the
accuracy of the model.
Keywords—Speech Emotion Recognition, Recursive Neural
Network, Long short term memory, K Neural Network
I. INTRODUCTION
Deep learning behaves like our nervous system. When it is
when presented with a problem to solve it processes the data
it already has and builds a solution upon it. The data already
present to the system is called as training data. In other
words deep learning models computer to process
information and compute results like the human brain but
with much more computational power.
Since the rapid advancement in the field of AI and
specially deep learning, it is being widely used in various
fields nowadays like medical treatment [1], detecting frauds,
and detecting emotions etc. Speech emotion recognition is
the process of categorizing and classifying emotions. It has
varied field of applications and can be used to upscale and
optimize the already existing business operations, it also has
applications that help in resolving social issues like suicide
prevention and much more.
Multiple studies uses Support Vector Machine (SVM) for
emotion detection but they were found that they were not as
optimal as other deep learning algorithms [2]. We have used
Convolutional Neural Network(CNN) in our model which
yields in better result than compared to SVM. For prediction we
are per-training the data and constructing a training set and
testing set and testing set, as our prediction obtains an optimum
node such that the predicted node provides the satisfactory
output. We can look at CNN as a model that feeds ahead
artificial network in which joining sequence among its nodes is
motivated by presenting an animal visual-cortex, its dimension
is bounded by the depth of the structure, also the size of the
filter and zero-padding.
Based on existing emotion recognition research, feature
extraction is used for speech recognition both in the
implementation of emotion recognition, feature extraction
used is Mel-frequency Cepstral Coefficient. This is because
Authorized licensed use limited to: SRM Institute of Science and Technology. Downloaded on February 15,2023 at 06:36:35 UTC from IEEE Xplore. Restrictions apply.
Anika Pandey Dr Aruna
Anguvel
SRM institute of science and
ap7481@srmist.edu.in technology, SRM university
this method is considered the most appropriate for human
frequency modeling. Therefore the authors use this method for
feature extraction. The dialect that will be used in this research
is the dialect of Makassar people as voice data. The author
chose this dialect because of its unique characteristics and
different dialects from other regions. This rough-speeching
dialect marks the identity of a strong and mighty person.
II. Literature survey
a. Machine learning-based speech emotion recognition
system[2] this paper provides a block diagram of speech
emotion recognition system. It also proposes the
methodology of breaking the speech signal into
segments of small time intervals, which makes it easier
to extract features.
b. Speech emotion recognition using Deep Learning [3] this
project uses deep learning models to detect emotions
and attempt to classify them according to audio signals.
IEMOCAP dataset is used for detection of emotions. It
uses SER to gauge the emotional state of the driver and
comes handy to prevent any accidents.
c. Emotion detection in dialog systems: applications
strategies and challenges [4] , this paper has two
conclusions. First is to label audio signals as angry and
the second one is to gauge how satisfied a customer. It
uses 21 hours of recorded data where users are
reporting phone connectivity issues, the actors then
choose to rate their experience based on a scale of 1-5.
Finally the models labels the data as “garbage”,
“angry”, “not angry”, “don’t know”.
d. Emotion detection: a technology overview [5] in this
paper we explore various factors that stimulate
emotions and emotion recognition with voice is delt in
here. It describes three technologies vokaturi which is
run on a computer, beyond verbal which is a cloud
based service and is used in the project to send api
request and get results it needs at least 13 seconds of
voice audio.
e. Speech emotion recognition using feqture and word
embedding [6] this paper describes categorical speech
recognition. It tells us that we can combine text and
voice functions to improve the accuracy of the model.
f.
 g. A review on emotion detection and classification
using speech [7] this paper demonstrates all of the
techniques we can use for feature extraction which
when coupled with certain algorithms can yield in
improved efficiency.
III. METHOD
We have used cnn for out model and its architecture
diagram of the system is given below.-
Training and testing model: we fetch a training data to the
system consisting of experimental label and weight training
is also provided for the network. An audio is taken as an
input which is then normalized in order to train the CNN,
the purpose being that the impact of presentation sequence
of the examples don’t affect the training performance. The
outcome we get from the process is that we it acquires the
best result with this learning data. It fetches the system with
energy along with pitch. The network weights trained gives
the determined emotion. The output is represented in a
numerical value each corresponding to the expressed
emotions.
The emotion that are detected here are happy, angry, sad
and garbage.
Algorithm used;
1. Sample audio is given as input
2. Waveform is plotted from the input
3. Using librosa we extract the mfcc(mel frequency
cepstral coefficient)
4. Then the data is divided into training and testing
data after constructing a CNN model and its layers
to train the dataset.
5. Then we predict the emotions
Authorized licensed use limited to: SRM Institute of Science and Technology. Downloaded on February 15,2023 at 06:36:35 UTC from IEEE Xplore. Restrictions apply.
Dataset is collected from Kaggle – Ravdess dataset. It has
1440 files and 60 actors and 1440 trials. The ravedess
consists of 24 professional voices and lexically matched
sentences in the north American accent which are happy
sad and angry and labeled as garbage. Every expression
is generatedin2 levels of emotional intensity (light, bold),
with a neutral expression. Every file out of 1440 files has
an unique filename. The filename holds a 7-part numerical
identifier (e.g., 03-02-05-01-02-02-11.wav). They
constitutethe evoking features.
MFCC for feature extraction:
It is one of the methods used for feature extraction. . The
human auditory system is assumed to process speech
signals non linearly and measured on a mel-frequency
scale. In speech recognition, Mel-frequency cepstrum
represents the short-run spectral strength of the greeting
frame using the linear transform cosine of the log freedom
spectrum on a non-linear mel-frequency scale [8].
Modules of CNN
In our CNN module we have four important layers;
1. Convolution layer: Identifies salient regions at
intervals, length utterances that are variable and
depicts the feature map sequence.
2. Activation layer: it is a non linear activation layer
function that is used as customary to the
convolution layer outputs, we have used corrected
linear unit (ReLU) for this.
3. Max pooling layer: this layer enables options with
the maximum value to the dense layers. It helps to
keep the variable length inputs to a fixed sized
feature array.
4. Dense layer