2020 International Conference for Emerging Technology (INCET)
Belgaum, India. Jun 5-7, 2020
Sarcasm Detection using Genetic Optimization on
LSTM with CNN
Darkunde Mayur Ashok1, Agrawal Nidhi Ghanshyam2, Sayed Saniya Salim3, Dungarpur Burhanuddin Mazahir4, Bhushan S.Thakare5
1,2,3,4,5
Computer Engineering, Sinhgad Academy of Engineering, Pune, India
1
mayur08darkunde@gmail.com, 2agrawal.nidhi6@gmail.com, 3saniyasayed514@gmail.com, 4burhandungarpur10@gmail.com,
5
bhushan.thakare86@gmail.com
Abstract—The challenging problem of 21st Century is to
detect sarcasm in vivid data available on a large scale. Over 20
years of study in this field, the past 10 years have shown a
significant progress not only in semantic features, but also an
upward trend has also been observed in the various machine-
learning approaches to analyze and process the data. To enlist
a few, theories of sarcasm, it’s syntactical and semantic
properties; lexical features have been an area of interest for
almost all of them. In this paper, we propose a unique deep
neural network model whose Bidirectional LSTM undergo
Hyper parameters optimization using genetic algorithm
followed by a Convolution Neural Network for sarcasm
detection. We put forward the results in a robust way, which
may result in a better future work in this field.
Keywords— Genetic; Optimization; Deep learning; Natural
language processing; LSTM; CNN;
I. INTRODUCTION
st
21 Century, the Information Age, showed a surge in the
generation of enormous amount of data which directly
affected on opinionated textual data over the Internet through
various social media platforms. Sentiment Analysis is used
for the analysis of opinionated text for a better understanding
of the actual motive of the person concerned. Sarcasm is
identified as a sentiment where, the literal meaning differs
from what is being said. Therefore, the challenging problem
has led to rise in the study of understanding sarcasm and
detecting it in text in the area of sentiment analysis.
The research is considered as one of the latest in the area
of natural language processing, with time not more than 20
years, but, the need of dynamic sarcasm detection and its
application for the betterment of digital society of 21st
century is already discovered. To begin with, there exist
various viewpoints to the particular problem. Firstly -
psycholinguistic, the study of establishing relationship
between natural language and human perspective. E.g The
existence of context based information in sarcastic utterances
[1]. Sarcasm is categorized as sub-form of humor from the
point of view of humor [2]. While, considering the medical
aspect, it is examining the understanding of sarcasm of a
person with physically damaged brain affecting the thinking
and uttering process [3]. Convolutional Neural Networks
(CNNs) have shown good accuracy over the state of the art
traditional models. Genetic algorithm are usually used to
construct a CNN architecture automatically [4] and in [5]
real time application for image classification with
automatically generated CNN architecture with the help of it.
Embedding layers are used like a perfect launch pad.
Embedding layer provides words to vector conversion and
for computational purpose numbers are always easy to
understand from a machines perspective. This paper presents
978-1-7281-6221-8/20/$31.00 ©2020 IEEE
Authorized licensed use limited to: Middlesex University. Downloaded on August 04,2020 at 05:44:08 UTC from IEEE Xplore. Restrictions apply.
a unique approach of parameter optimization of LSTM by
genetic algorithm followed by a CNN model.
The dynamic detection of sarcasm, at present, is needed,
as sarcasm plays an important part in everyday aspect.
Several studies concentrate on detecting sarcasm by using
various data sets, features and algorithms. Abundant study
done in this field has majorly utilized Twitter as dataset.
II. SARCASM: KEY ASPECTS
‘Sarcasm’ has been in the core of human emotions right
from the beginning of the mankind, humans have used it as a
sly weapon to intimidate a person in a jolly way. The
contrast of what been said and what really implies is not the
same always which usually differs the way it is interpreted.
A. Sarcasm in Human Emotion
Sarcasm for humans is a form of humor. It has been
gradually being used by people to reflect something negative
in positive way regarding people who are generally at a
higher order. In recent years, it became a necessity to extract
the actual emotion and opinion of people on social media
platforms as it provides a uniform platform to all kinds of
people. Sarcasm plays crucial role in sentiment analysis as
sarcasm is basically expressing emotion in well framed
sentence. For example, “Cannibalism can solve both of the
world hunger and overpopulation problem” which generally
implies humans should kill each other.Thus identification of
emotions behind these comments is major task in sentiment
analysis. In this digital world, Social media is kind of human
voice on various human issues. Human emotions such as
anger, fear, disgust, joy, surprise and sadness are indentified
from textual data in researches [6]. Researchers had done
most of the work by using training model. This model is
trained by annotating the large dataset with basic emotions.
SenticNet, wordNet this approaches are commonly used to
find the emotion in web based content. In recent research
work millions of emoji data is used train the model and
detection sentiment, emotion and sarcasm is done in superior
way [7].
B. Dataset Analysis
We observe that, through all data set taken into account
by the researchers, the dataset popular among them is
Twitter. The breakpoint to this reason is because the
properties of Twitter, such as word limit, which is not
available on any other social media platform.
One of the features amongst many which diverts the
researcher’s interest from other social media platform in
Twitter is #(hashtag). A hashtag is a unique strategy for
categorization of messages on Twitter. The string following
the hashtag represents or depicts the central idea of the tweet.
1
Therefore, users with similar ideas adopt the same hashtag.
For instance, consider the recent “Plastic Ban Movement”
where the hashtag-#saynotoplastic was adopted by millions
across the globe. Amazon Customer Reviews have
contributed to over a hundred million reviews and
experience, which provides wide spectrum of data for
Natural Language Processing (NLP). Amazon Alexa a
popularly house automation IOT device currently booming
in the market presents fascinating data to the researchers. For
eg. A person saying “Alexa you would be great swimmer”.
That generally implies sarcasm, besides form the plain
textual data; human tonal clues are present in this kind of
dataset which would leads to a better hindsight in NLP.
Emoticons that were neglected earlier are now becoming a
hot topic for recognizing the different sentiments associated
with them.
III. RELATED WORK : SARCASM DETECTION
Techniques used in sarcasm detection could be
considered the central part of the processing. Once the data
has been established, it is the technique that has to be clever
enough to derive a model that fits the data. Based on the
types of approaches, we can broadly classify them in two
categories. (1) Rule-based, (2) Learning algorithms and
features based. Here rule-based technique is based on
evidences of sarcasm whereas the other technique is
dependent upon the parameters used to classify sarcasm and
learning algorithm using those parameters as features.
A. Rule-based approach
These rule-based approaches are commonly known as 9-
rule based approaches. Hao [8] used this 9-rule approach for
various expressions and emotions in sentences. Important
rule of sarcasm is applied by Bharti et al. [9] that negative
statement spoken in positive way shows the chances of
sentence to be sarcastic high. For example “I like the oil
stains on your oversized shirt” Here sentence actual meaning
is negative but it is spoken in positive way. Rule based
algorithm for sarcasm detection were proposed by Bharti et
al. [10] PBLGA(Parsing Based Lexical Generation
Algorithm) that takes input as a sentence and split the
sentence in different phrases such as noun, adjective, verb
and adverb. After processing of the phrases the output is in
form of lexicons where the lexicons are categorized into
positive, negative, positive situation and negative situation
using different classifiers namely support vector machine,
naïve bayes, maximum entropy and decision tree. Use of
parse trees is done by Reyes et al. [11] for identifying irony
in twitter dataset, mainly statistical based method is
implemented.
B. Supervised Learning
Basak et al. [12] suggested the need for classification of
online shaming tweets not just into positive and negative but
into Abusive, Comparison, Passing judgment, Religious,
Sarcasm, Whataboutery. Thus six SVM’s were trained for
above classification. Rajeshwari and shantibala et al. [13]
proposed a system to recognized the sarcastic emotion of the
individual with use of Multinomial Naïve Bayes(MNB) and
also to identify type of sarcastic emotion using SVM model.
MNB basically models the word counts and regulated the
underlying calculation to deal with in. SVM here is further
used for classifying sarcastic tweets in different type as
Authorized licensed use limited to: Middlesex University. Downloaded on August 04,2020 at 05:44:08 UTC from IEEE Xplore. Restrictions apply.
Depression, Brooding, Polite and Maniac. Amazon alexa
presents a tough challenge for Natural Language Processing
as the data gathered varies in diversity. Pandey et al. [14]
worked on Amazon Alexa dataset which was processed
using the SentiWordNet 3.0 and the further processing was
done with the help of TextBlob to remove noise and
unwanted data.
C. Semi-supervised
Rajeshwari and ShantiBala [15] came forward with SASI
(Semi-supervised Algorithm for Sarcasm Identification) that
implements both unsupervised and supervised approaches.
The unsupervised approaches are used to find labeled
quantities and cluster into different classes. Supervised
machine learning approaches uses large labeled data set to
train the classifier. The classification of the various types of
sarcastic emotions of the individual users is performed with
SVM.
D. Hybrid Approach
It is the combination of more than one machine learning
classifier, which is bind together to become mixed classifier.
Author Amir et al. [16] form CUECNN model which is for
sarcasm detection using word and user embedding features.
Devamanyu Hazarika et al. [17] came up with the hybride
approach as CASCADE (ContextuAl SarCasm DEtector).
This model generally adapts a hybrid approach of content
and context-driven sarcasm detection. The above mentioned
model performs significantly better than the CUECNN
model proposed by Amir et al. [16].
E. Neural Network
Neural Network forms the base of deep learning, a
subfield of machine learning where algorithms are inspired
from structure of human brain. Neural Network takes input
data to find pattern and then it predicts the output for new set
of similar data. Neural units connect to many others units
and form a network. Each unit may have a function which
combines the values of all its inputs together.
1) Artificial Neural Network(ANN):
ANN is based on biological Neural Network that
replicates the processing of human brain to build algorithms
and processes information. ANN works by building directed
graphs where nodes are formed by artificial neurons, these
neurons are functions which take various weighted input and
the sum is passed though non liner function known as
Activation function. The connection between neurons is
shown by the weighted edge in graph and this various layers
of nodes feeds information forward to process. ANN deals
with large amount of associated processing units and
produce meaningful results.
2) Convolution and Recurrent Neural Network:
Le Hoang Son et al. [18] implied convolution neural
network and soft attention-based bidirectional long short-
term memory in their studies. They used two dataset of
Twitter as balanced and unbalanced, balanced dataset has
20,000 annotated tweet that include hash tags e.g. #sarcasm,
#sarcastic, etc and unbalanced dataset 15,000 sarcastic and
25,000 non-sarcastic tweet. 39,000 tweets are used for
training dataset containing tweets. Furthermore author used
eight layers for this hybrid approach. Humor a form of
amusement for human beings but is a complex aspect and
contains many forms. Jaiswal et al. [19] proposed a work for
humor detection and classification of its type such as
Sarcasm, Pun, Irony and Exaggeration using various
classification algorithms.
P.K.Mandal and R.Mahto [20] worked on news headline
dataset for sarcasm detection. News headline implied perfect
meaning, minimal grammatical error, unique sentences and
clean data. Natural Language Toolkit (NLTK) was used for
data pre-processing and to carry out operations such as
tokenization and lemmatization. The dataset consisted of
26,709 news headlines where 11,725 were sarcastic records.
The first layer of the proposed model is an embedding layer
to encode words into vectors which was followed by a CNN-
LSTM . Hia and Shimada [21] proposed a method that
classifies text into sarcastic or non-sarcastic using an RNN
model. They used two types of features to train their RNN
Model 1.)Words in sentences 2.)The role pair relation vector.
Firstly, input words in a sentence were inserted into the RNN
model. Identification of role pairs in the sentence by
obtaining a list of role pairs from a corpus. Construct of the
role pair relation vector and input the vector to the RNN
model took place in the last phase. If role pairs were not
present in the sentence similar relation vectors were assigned
to similar sentences.
IV. PROPOSED APPROACH
The deep learning architecture proposed in this paper
consist of BERT as a embedding layer , LSTM layer whose
hyper parameters, window size and number and hidden units
have been optimized using a genetic algorithm followed by a
Convolutional Neural Network.
Fig. 1. System Arctitecture of proposed model
Authorized licensed use limited to: Middlesex University. Downloaded on August 04,2020 at 05:44:08 UTC from IEEE Xplore. Restrictions apply.
A. Data pre-processing
The data present on the twitter is diverse in its own
nature like the human population we have limited our self to
only English language. The tweets are acquired using
#hastags such as #Sarcasm, #sarcasm, #Humor. After
acquisition of the data is pre-processed by eliminating URLs,
hashtags, person tagged, emojis and conversion of text into
lowercase. NLTK is used to process the raw data and
perform tokenization operation. The words are then reduced
to their root words’ using the Potter Stemmer the operation is
called as Stemming.
B. Input Layer
The input layer acts as the first layer in the architecture
where the processed tweets are fed to the network.
C. Embedding layer
The second layer in the architecture is the embedding
layer which converts the input text into real time vector
representation. The conversion of words into vector
representation is beneficial as computer understand numbers
for clearly than a human language. BERT has been used as
an embedding layer, as it uses Transformer to understand the
contextual relation between words related to each other in
the text being processed. It is been trained on a large corpus
of Wikipedia which results in better performance.
D. LSTM layer
The third layer is the interconnected cells of the LSTM;
they act as a base architecture of Recurrent Neural Network.
The main function performed by the LSTM layer is to decide
which data to remember and which data to forget via the
forget gate as the internal architecture of LSTM is a gated
architecture which consist of 3 gates input, output and forget
gate.
E. Genetic Optimization
The LSTM cell consists of various parameters which are
needed to be fine tuned to get the optimal results.
Optimization with the help of Genetic algorithm is done to
obtain the optimal hyper parameters, window size and
number of hidden units of the LSTM layer. The overall
optimization reduces the Root Mean Square Error (RMSE)
which indirectly increases the accuracy of the model.
Fig. 2. Genetic Algorithm
F. Convolutional Neural Network
The fourth layer in the architecture is the Convolutional
Neural Network. CNN have been used as a unique approach
to solve the real world application using deep learning. The
idea of CNN followed by LSTM is; CNN has the capability
to search over a large space while LSTM have the capability
to search over time. The optimized LSTM layer provide
good set of input to the CNN to increase the efficiency of the
model to detect sarcasm in a contextual way due to BERT
embedding layer.
V. DISCUSSION
The proposed approach would perform better that the
state of the art models, as they rely mostly on the lexical
features. BERT provides a way to interpret sarcasm in a
contextual way. Contextualized information provides a better
hindsight to get in-depth understanding of text. A CNN
network provides an accuracy of 85-88%. A LSTM-CNN
model with word embedding provides an accuracy of 90-
92%. The proposed model would give accuracy around 93-
95% due to the parameter optimization, BERT as embedding
layer and reduced RMSE error.
VI. CONCLUSION
The field of Sarcasm detection has seen a significant
growth in past few years, due to increase in usage of social
media sites and big data boom. Sarcasm detection remains
the problem where many ways have been explored to reach
the goal state but only partial fulfillment is achievable. The
traditional techniques/algorithm performs well with small
amount of data but is not able to crunch large volume of
data. The work done provides a way to understand sarcasm
using deep learning model. The proposed deep learning
model for sarcasm detection has been explained. The future
work of the model would be to detect sarcasm in real time
data.
REFERENCES
[1] R. G. W., “On the Psycholinguistics of Sarcasm,” Journal of
Experimental Psychology, Volume 115, pp. 315, 1986.
[2] R. A. Martin, P. Puhlik-Doris, G. Larsen, J. Gray ve K. Weir,
“Individual differences in uses of humor and their relation to
psychological well-being: Development of the Humor Styles
Questionnaire,” Journal of Research in Personality, Volume 37, no. 1,
pp. 48-75, 2003.
[3] S. McDonald ve S. Pearce, “Clinical insights into pragmatic theory:
frontal lobe deficits and sarcasm,” Brain and Language, Volume 53,
no. 1, pp. 81-104, 1996.
[4] Lingxi Xie, Alan Yuille, “ Genetic CNN ” The IEEE International
Conference on Computer Vision (ICCV), 2017, pp. 1379-1388
Authorized licensed use limited to: Middlesex University. Downloaded on August 04,2020 at 05:44:08 UTC from IEEE Xplore. Restrictions apply.
[5] Yanan Sun, Bing Xue, Mengjie Zhang, Gary G. Yen, “Automatically
designing CNN architectures using genetic algorithm for image
classification” in IEEE Transactions on Cybernetics, 2020
[6] C. Strapparava and R. Mihalcea, “Learning to Identify Emotions in
Text”, Proceedings of the 2008 ACM Symposium on Applied
Computing, pp. 1556-1560, 2008.
[7] Jayashree Subramanian, Varun Sridharan, Kai Shu, and Huan Liu,
“Exploiting Emojis for Sarcasm Detection”, Springer Nature
Switzerland AG 2019, pp. 70–80,2019.
[8] Hao, T. V., “Detecting Ironic Intent in creative Comparisons.” ECAI,
Vol. 215, 765–770, 2010.
[9] S. K. Bharti, K. S. Babu, S. K. Jena, “Parsing-based Sarcasm
Sentiment Recognition in Twitter Data,” in Proceedings of the 2015
IEEE/ACM International Conference on Advances in Social
Networks Analysis and Mining , 2015
[10] S.K. Bharti , R. Pradhan , K.S. Babu ,S.K. Jena,” Trends in Social
Network Analysis”, , Lecture Notes in Social Networks,2017,pp 51-
76
[11] A. Reyes, P. Rosso ve T. Veale, “A multidimensional approach for
detecting irony in Twitter,” Language Resources and Evaluation,
2012
[12] Rajesh Basak, Shamik Sural , Niloy Ganguly, ve Soumya K. Ghosh,
“Online Public Shaming on Twitter: Detection, Analysis, and
Mitigation”,2019
[13] Rajeswari K and ShanthiBala P, “sarcasm detection using machine
learning techniques ”, International Journal of Recent Scientific
Research, Vol. 9, Issue, 4(L), pp. 26368-26372, April, 2018
[14] Avinash Chandra Pandey, Saksham Raj Seth and Mahima Varshney,
“Sarcasm Detection of Amazon Alexa Sample Set”Advances in
Signal Processing and Communication, Lecture Notes in Electrical
Engineering 526, 2019.
[15] K.Rajeswari and P.ShanthiBala, “Recognization of Sarcastic
Emotions of Individuals on Social Network ”, International Journal
of Pure and Applied Mathematics, Vol. 118 No. 7, pp. 253-25, 2018
[16] S. Amir, B. C. Wallace, H. Lyu, P. Carvalho , M. J. Silva, “Modelling
Context with User Embeddings for Sarcasm Detection in Social
Media,” in Proceedings of The 20th SIGNLL Conference on
Computational Natural Language Learning, 2016.
[17] D. Hazarika, S. Poria, S. Gorantla, E. Cambria, R. Zimmermann, R.
Mihalcea, “CASCADE: Contextual Sarcasm Detection in Online
Discussion Forum” in Proceedings of the 27th International
Conference on Computational Linguistics, pages 1837–1848, Santa
Fe, New Mexico, USA, August 20-26, 2018.
[18] Le Hoang Son, Akshi Kumar , Saurabh Raj Sangwan , Anshika
Arora, Anand Nayyar , ve Mohamed Abdel-Basset, “Sarcasm
Detection Using Soft Attention-Based Bidirectional Long Short-Term
Memory Model With Convolution Network”, Special section on
emerging trends, issues and challenges in underwater acoustic sensor
networks,2019
[19] Arunima Jaiswal, Monika, Anshu Mathur, Prachi, Sheena Mattu,
“Automatic Humour Detection in Tweets using Soft Computing
Paradigms”, International Conference on Machine Learning, Big
Data, Cloud and Parallel Computing (Com-IT-Con),2019
[20] Paul K.Mandal and Rakeshkumar Mahto, “Deep CNN-LSTM with
Word Embeddings for News Headline Sarcasm Detection”, 16th
International Conference on Information Technology-New
Generations (ITNG),2019.
[21] Hiai, S., & Shimada, K. (2019). Sarcasm Detection Using RNN with
Relation Vector. International Journal of Data Warehousing and
Mining,15(4),66–78