Comparative Study of Service-based Sentiment Analysis

of Social Networking Sites Fanatical Contents

Abstract. The proliferation of mobile web services (MWS) for sentiment anal-
ysis makes it hard to identify the best MWS for sentiment analysis of social
networking sites' fanatical contents. This paper carries out a comparative study
of service-based sentiment analysis of social networking sites' fanatical con-
tents. This is achieved by cleaning, transformation, and reduction of fanatical
contents from the publicly available social media dataset, and multiple MWS
are selected for comparison using the application programming interface (API)
key of the MWS. To evaluate the service-based sentiment analysis, standard
measures such as accuracy, precision, recall, and f-measures of sentiment result
for each MWS are used. The result shows that Dandelion SA performs better in
terms of accuracy (72.5%) and recall (76.9%), while Wingify SA performs bet-
ter in terms of precision (88.6%) and f-measure (75.5%), though AlchemyAPI
offers the most crucial elements in analyzing sentiments such as emotion, rele-
vance score, and sentiment type. The outcomes of this paper will benefit the
sentiment analysis service developers, sentiment analysis service requesters as
well as other researchers in the social media fanatical content domain.

Keywords: sentiment analysis; mobile web service; social media; fanatical con-
tents

1 Introduction

In modern society, social networking sites (SNS) have become an integral platform
for interaction between people across the world. SNS like Twitter and Facebook are
widely used for sharing and accessing all sorts of content such as personal points of
view, an emotion about certain topics, etc. [1]. These SNS has undoubtedly generated
a massive amount of data in the form of tweets, status updates, posts, reviews, and
comments which contain a lot of fanatical content. Fanatical contents are normally
shared by individuals with extreme political or religious views, particularly those who
advocate illegal, violent, or other extreme action [2].
One of the global scenarios of fanatism is about how religion has been widely
blamed for violence [3]. With the rise of issues on fanatism, there need to control the
rate of users expressing or disseminating fanatical rage on SNA. It is also important to
counter this kind of threat to maintaining the wellbeing of societies across the globe.
As such, Natural Language Processing (NLP) techniques such as sentiment analysis
are used to identify fanatical contents from the SNS to create awareness among socie-
ty as it has established quite an active online presence nowadays [4]. SA deals with
many challenges through NLP because it is a difficult task. It has been proven in the
study made by [2], that it is extremely challenging to study the text effectively at an
2

arbitrary level of detail as outlined in the definition, and not to forget, most of the data
gathered are subjective.
A lightweight software system designed to support interoperable machine-to-
machine interaction across a network with an interface described using adaptable
protocols, RESTful description, smaller message formats popularly known as Mobile
Web Service (MWS) are now used for SA [5], [6]. There are various choices of
MWS, each has its specialty that help researchers in identifying sentiment [7]. How-
ever, not all MWS offer the same functions. Some may just allow a user to check the
sentiment polarity of a text without being able to view the confidence score. Thus, in
this case, it should be stressed that SA should not only aim to achieve sentiment po-
larity from data but also to discover the most accurate result generated by the MWS.
The proliferation of MWS for sentiment analysis makes it hard to identify the best
MWS for sentiment analysis of SNS fanatical contents.
Based on the existing research, not many researchers have studied MWS for SA. In
rare cases, [7] compared 15 different web services specialized in sentiment analysis of
three types of datasets which is from large movie dataset, Twitter dataset, and Ama-
zon product review dataset. However, the comparative analysis of MWS for SA fo-
cused more on two datasets (movie and product review) instead of MWS for SA of
fanatical contents in SNS. Knowing that SNS is widely used nowadays, the idea of
extracting data from it is acceptable as it gives different accuracy results when tested
in different MWS. Therefore, the focus of this research is to see how accurate each
MWS is in generating sentiment results by comparing its mean square error. For the
least mean square error, a value that is closer to zero will indicates a better result [8].
In this study, data is gathered from two social networking sites (Twitter and Face-
book) for service-based sentiment analysis to identify fanatics-related content. multi-
ple MWS designs for SA such as MonkeyLearn, Semantria, ParallelDots, Text2Data,
etc are selected for comparison using the application programming interface (API)
key of the MWS. Standard measures such as accuracy, precision, recall, and f-
measures, are used to evaluate the service-based sentiment analysis so as to find the
best MWS for analyzing sentiment on social networking sites' fanatical contents.
As a contribution, this paper presents an extensive and deeper critical evaluation of
the main functionalities of six SA-related MWS. This research will help in proving
the best MWS in terms of giving an accurate result of SA. Moreover, it will enable
other researchers to have sufficient information on the capabilities of these MWS and
helps them to choose the most suitable one to be used for their specific purposes espe-
cially in analyzing sentiment related to serious issues like fanatism so as to create
awareness, stimulate big techs and policymakers to control this issue from wide-
spreading around the world. The remaining of the paper is organized as follows. In
Section 2, the related works are discussed. In Section 3, the methodology is presented.
The results and discussions are presented in Section 4. Finally, the conclusions and
future recommendations are given in Section 5.
 3

2 Related Work

This section reviews various works related to this research. These include social me-
dia fanatical content, sentiment analysis, and mobile web service.

2.1 Fanatical Contents Sentiment Analysis

Over the years, social networking sites such as Twitter, Facebook, and other dis-
cussion forums are abused by fanatics, radicals, extremists, propagandists, etc. to
share and promote hate speech, offensive comments, religious ambiguity with decep-
tive intentions, manipulated political and social views [1]. Researchers from different
domains e.g. sociology, data science, and software engineering, etc. are continually
creating tools to help individuals, societies, big techs, and policymakers battle and
conquer these issues of manipulation, radicalization, etc. from wide-spreading around
the world [4]. Subsequently, SNS sentiment analysis and automatic detection of fanat-
ical contents is a significant issue for peace and security. Sentiment Analysis (SA),
also called Opinion Mining aims to analyze the sentiments, opinions, attitudes, and
emotions of individuals towards subjects, products, individuals, organizations, and
services [9]–[11]. SA involves the classification of sentiment text into classes such as
positive, negative, and neutral.
There are numerous researches related to sentiment analysis of social networking
sites' fanatical contents. For example, [6]–[8] highlighted the use of lexicon-based and
machine learning approaches. A piece of a text message is represented as a bag of
words in a lexicon-based approach. Following this representation of the message, all
positive and negative words or phrases within the message are assigned to either dic-
tionary-based sentiment values or corpus-based sentiment values. Moreover, [2], [4]
experimented with models such as the unigram model, a feature-based model, and a
tree kernel-based model to classify sentiment analysis on Twitter data where combin-
ing prior word polarity with their Part of Speech (POS) tags play an important role in
a classification task. It’s also observed that POS can be assigned to an input sequence
word based on POS tag from relevant words in relevant training sequence identified
using global semantic information.

2.2 Web Service-based Sentiment Analysis

Several authors have recognized the importance of web service-based sentiment anal-
ysis [7], [8], [13], [14]. These software systems designed to support interoperable
machine-to-machine interaction known as Mobile Web Service (MWS) are now used
for SA and the capabilities of MWS in examining the accurate result in analyzing
sentiment is the key performance indicator for determining the quality of service.
According to [7], document level SA was proven as the most used feature among all
levels of SA. In this features, the various task is provided including three different
classifications of sentiment which is positive, negative, and neutral. Compared to the
other features that mostly provide only positive and negative classes, the document
level SA deals with tagging individual documents to find sentiment polarities with
4

three different classes. Therefore, in this research, only MWS that offers document-
level SA and polarity will be considered.
Several researchers have used different techniques in evaluating the capability of
MWS in terms of accuracy and polarity of sentiment. For example [7], [8], [14] focus
on analyzing the capabilities of MWS to classify and score various texts with regards
to the sentiments in it. most of the MWS uses various categorization of SA techniques
which are machine learning approach and lexicon-based approach, dictionary-based
lexicon approach, and corpus-based lexicon approach. The corpus-based approach in
[7], [8], [14] mainly relies on a sentiment lexicon and has statistical and semantic
techniques which measure a perceived meaning of an object that led to a result of
sentiment analysis. These researches focused on three different datasets which were
from large movie reviews, Twitter, and Amazon product reviews.
Despite the many contributions of [1]–[3], [7], [8], [12]–[14], the comparative
analysis of MWS for SA focused more on very few MWS that does not consider sen-
timent polarity of a text. Given that SNS like Twitter and Facebook are widely used
for sharing fanatical contents by individuals (fanatics, radicals, extremists, propagan-
dists, etc.) with extreme political or religious views, particularly those who advocate
illegal, violent, or other extreme action [1], [2], there is a clear lack of focus on fanat-
ical contents and only focused on one dataset from SNS which is Twitter, while the
other two datasets are from movie and product review. Because of these shortcom-
ings, the idea of a comparative study of service-based sentiment analysis of social
networking sites' fanatical contents is very motivating and important. The focus of
this research is to see how accurate each MWS is in generating sentiment results.

3 Methodology

The methodology used in conducting the comparative study of service-based senti-

ment analysis of social networking sites' fanatical contents comprises of four stages
(data collection, data pre-processing, service-based sentiment analysis, and compari-
son of service-based sentiment analysis). The four stages are explained in Section 3.1,
3.2, 3.3, and 3.4 respectively.

3.1 Data collection

Datasets from social networking sites (Twitter and Facebook) provide a large data for
service-based sentiment analysis of fanatical contents, therefore, tweets and posts
gathered from Twitter [15] and Facebook [16] are used as our benchmark data. Con-
cerning the first collection of datasets (from Twitter), the tweets contain a lot of opin-
ions that were expressed by different users. The second collection of datasets (from
Facebook) did not show much difference from the first collection except for the long-
er length of sentences. Both of it contains neutral opinion as well as positive and neg-
ative opinion. However, to assess the performance of MWS, only a few documents
from the whole datasets will be used. All the datasets were extracted based on the list
of keywords that are associated with fanatical content.
 5

3.2 Data pre-processing

Pre-processing of dataset helps in preparing the raw data that has been gathered for
the service-based sentiment analysis of social networking site fanatical contents.
These involve text cleaning, white space removal, expanding abbreviation, stemming,
stopping words removal, negation handling, etc. [3], [15], [16]. The processed dataset
is further used for feature extraction to compute the polarity of the sentence which is
useful in determining opinion. The RStudio provides an ability to pre-process data.
Therefore, RStudio is used for cleaning, transformation, and reduction of fanatical
contents from the publicly available social media dataset.

3.3 Service-based sentiment analysis

Several MWS for sentiment analysis are chosen for the comparative study of service-
based sentiment analysis of social networking sites' fanatical contents [7], [8], [14].
Considering that the document level SA deals with tagging individual documents to
find sentiment polarities with three different classes (positive, negative, and neutral),
ten MWS (DeepAI SA, AlchemyAPI, Lymbix SA, Musicmetric, Twinword SA, Se-
mantria, ParallelDots SA, Dandelion SA, Wingify, and Sentimetrix) are selected from
the programmableweb.com. Popularity, ease of use, and the maximum amount of
data that each MWS can handle are used as the basis for selecting the ten MWS for
sentiment analysis.
Mobile Web Service (MWS) bringing about convenience and simplicity in access-
ing web service through mobile devices. The sentiment analysis service provider pub-
lishes its services to the local registry. A service request is sent to the local registry.
The communication between the request and the response is carried out through API.
Lightweight protocols such as kXML2, kSOAP2, are adapted to overcome the signifi-
cant overhead of the original SOAP implementation. Figure 1 shows the overview of
MWS communication architecture. The MWS is called through the API after authen-
tication and the dataset is parsed after which the MWS functionality is applied to
obtain the SA result.
6

Fig. 1. Overview of MWS communication architecture.

3.4 Comparison of Service-based sentiment analysis

The performance of each MWS is measured by accuracy, precision, recall, and f-

measure. Accuracy is the level of closeness between sentiment polarity obtained from
the MWS and the actual value of the sentiment polarity, the accuracy is calculated
using Equation 1. Precision is used to measure the level of correctness of the senti-
ment analysis results generated by each MWS, the precision is calculated using Equa-
tion 2. The recall is used to measure the ability of each MWS to find sentiment polari-
ties with three different classes (positive, negative, and neutral) within the dataset, the
recall is calculated using Equation 3. F-measure is the harmonic mean or weighted
average of the precision and recall. The F-measure is calculated using Equation 4.
Four classifications are used to measure the performance of each MWS. These are
True Positive (TP), False Positive (FP), False Negative (FN), and True Negative
(TN).

𝑇𝑃+𝑇𝑁
Accuracy = (1)
𝑇𝑃+𝑇𝑁+𝐹𝑃+𝐹𝑁
𝑇𝑃
Precision = (2)
(𝑇𝑃+𝐹𝑃)
𝑇𝑃
Recall = (3)
(𝑇𝑃+𝐹𝑁)
Precision ∗ Recall
F-Measure = 2 ∗ (4)
Precision + Recall

4 Results and Discussions

The performance of each MWS is obtained after experimental evaluation. The evalua-
tion results are based on four criteria (accuracy, precision, recall, and f-measure).
Table 1 presents the results of the comparative study of service-based sentiment anal-
ysis of social networking sites fanatical contents. The MWS with good accuracy are
Dandelion SA, Twinword SA, AlchemyAPI, and Lymbix SA which has an accuracy
of 72.5%, 69.2%, 63.4%, and 62.8% respectively.
 7

Looking into the overall result for each MWS in Table 1 below, the MWS with a
good precision are Wingify, Sentimetrix, Musicmetric, and Dandelion SA which has a
precision of 88.6%, 82.4%, 75.2%, and 67.3% respectively. The MWS with a good
recall are Dandelion SA, Twinword SA, Lymbix SA, and Semantria which has a re-
call of 76.9%, 70.5%, 68.2%, and 67.6% respectively. The MWS with good accuracy
are Wingify, Dandelion SA, AlchemyAPI, and Sentimetrix which have an accuracy of
75.5%, 71.8%, 65.9%, and 65.5% respectively.

Table 1. The performance of each mobile web service for SA.

MWS Accuracy Precision Recall F-measure
DeepAI SA 0.614 0.560 0.597 0.578
AlchemyAPI 0.634 0.672 0.647 0.659
Lymbix SA 0.628 0.615 0.682 0.647
Musicmetric 0.527 0.752 0.531 0.622
Twinword SA 0.692 0.591 0.705 0.643
Semantria 0.619 0.571 0.676 0.619
ParallelDots SA 0.562 0.510 0.564 0.536
Dandelion SA 0.725 0.673 0.769 0.718
Wingify 0.591 0.886 0.658 0.755
Sentimetrix 0.571 0.824 0.543 0.655

Dandelion SA presents the highest accuracy result with 72.5%. It also computes the
highest results in recall as shown in Figure 2. Dandelion SA might seem the best
MWS to classify positive text. This is because it gives the highest amount of positive
sentiment among all MWS. Moreover, the amount of negative classified sentiment
from Dandelion SA is the lowest. Wingify SA presents the highest precision result
with 88.6%. It also computes the highest results in f-measure. Dandelion SA might
seem the best MWS to classify both positive text and negative text. This is because it
provided a better level of correctness in terms of positive sentiment and negative sen-
timents among all the MWS.
AlchemyAPI gave a comparable number for all the four criteria (accuracy, preci-
sion, recall, and f-measure) as shown in Figure 1 and Figure 2. It portrays a balance in
terms of reviews of positive (precision and recall) and f-measures. Though it’s limited
when compared with other MWS such as Dandelion SA, Wingify. On average,
Twinword SA, and Sentimetrix also gives a good result during testing as it appears as
the second-best group of MWS (after Wingify and Dandelion SA) in terms of the four
evaluation criteria. While ParallelDots SA and DeepAI SA perform slightly less than
their counterpart in terms of service-based sentiment analysis of social networking
sites fanatical contents.
Based on the analysis of the results, it is noticeable that all the MWS do offer a
slightly higher than average result for an accurate polarity (positive, negative, or neu-
tral) especially for service-based sentiment analysis of social networking sites fanati-
cal contents. Contrary to what has been stated in [2], [7], [8], [14], it is proven that
ParallelDots SA and Twinword SA are the most widely used MWS that classify both
short and long text. While ParallelDots SA and Wingify show the best accuracy and
8

precision respectively. However, when working with larger text, all the MWS have
shown no problem in performing sentiment analysis. Despite the ironic nature of fa-
natical content in social networking sites, which could be regarded as very difficult to
detect, it is noteworthy the fact that all the MWS can perform reasonably well. And
lastly, it is essential to say that artificial intelligence-based MWS like DeepAI SA
might hold the key for service-based sentiment analysis of social networking sites'
fanatical contents given the trajectory of AI successes.

Fig. 2. The accuracy and recall of the service-based SA.

Fig. 3. The precision and f-measure of the service-based SA.

5 Conclusion and future work

This presented a comparative study of service-based sentiment analysis of social

networking sites fanatical contents. To ease the selection of the most suitable MWS
for fanatical contents sentiment analysis, ten MWS are selected for comparison using
the application programming interface (API) key of the MWS, while the datasets from
social networking sites (Twitter and Facebook) provide a large data for service-based
sentiment analysis of fanatical contents is used as our benchmark data. Despite the
ironic nature of fanatical content in social networking sites, which could be regarded
as very difficult to detect, overall, the results demonstrate that Dandelion SA has the
 9

highest accuracy (72.5%) and recall (76.9%) results. While Wingify SA presents the
highest precision (88.6%) and f-measure (75.5%) result. The outcomes of this paper
will benefit the sentiment analysis service developers, sentiment analysis service re-
questers as well as other researchers in the social media fanatical content domain. The
future work aims at exploiting the non-functional correlation between the MWS for
sentiment analysis service developers to further improve the precision and accuracy as
close to 100% as possible. Moreover, it is essential to explore artificial intelligence-
based MWS like DeepAI SA which might hold the key for service-based sentiment
analysis of social networking sites' fanatical contents given the trajectory of AI suc-
cesses.

References

[1] R. T. Adek, Bustami, and M. Ula, “Systematics Review on the Application of Social
Media Analytics for Detecting Radical and Extremist Group,” IOP Conf. Ser. Mater.
Sci. Eng., vol. 1071, no. 1, p. 012029, 2021.
[2] M. Asif, A. Ishtiaq, H. Ahmad, H. Aljuaid, and J. Shah, “Telematics and Informatics
Sentiment analysis of extremism in social media from textual information,” Telemat.
Informatics, vol. 48, no. October 2019, p. 101345, 2020.
[3] P. Karampelas, T. Bourlai, C. Surveillance, Y. Wei, and L. Singh, “Detecting users
who share extremist content on twitter,” in Surveillance in Action, Springer, 2018, pp.
351–368.
[4] U. Kursuncu et al., “Modeling islamist extremist communications on social media
using contextual dimensions: Religion, ideology, and hate,” Proc. ACM Human-
Computer Interact., vol. 3, no. CSCW, 2019.
[5] S. Garba, R. Mohamad, and N. A. Saadon, “Web Service Discovery Approaches for
Dynamic Mobile Environment,” Int. J. E-Services Mob. Appl., vol. 13, no. 4, pp. 16–
38, 2019.
[6] S. Garba, R. Mohamad, and N. A. Saadon, “Search Space Reduction Approach for
Self-adaptive Web Service Discovery in Dynamic Mobile Environment,” in Advances
in Intelligent Systems and Computing, vol. 1073, 2020, pp. 1111–1121.
[7] J. Serrano-guerrero, J. A. Olivas, F. P. Romero, and E. Herrera-viedma, “Sentiment
analysis : A review and comparative analysis of web services,” Inf. Sci. (Ny)., vol. 311,
pp. 18–38, 2015.
[8] A. B. M. N. Basmmi, S. Abd Halim, and N. A. Saadon, “Comparison of Web Services
for Sentiment Analysis in Social Networking Sites,” in IOP Conference Series:
Materials Science and Engineering, 2020, vol. 884, no. 1, p. 12063.
[9] M. Alam, F. Abid, C. Guangpei, and L. V. Yunrong, “Social media sentiment analysis
through parallel dilated convolutional neural network for smart city applications,”
Comput. Commun., vol. 154, pp. 129–137, Mar. 2020.
[10] A. Sawarn, Ankit, and M. Gupta, “Comparative Analysis of Bagging and Boosting
Algorithms for Sentiment Analysis,” in Procedia Computer Science, 2020, vol. 173,
pp. 210–215.
[11] I. Atoum, “A novel framework for measuring software quality-in-use based on
10

semantic similarity and sentiment analysis of software reviews,” J. King Saud Univ. -
Comput. Inf. Sci., vol. 32, no. 1, pp. 113–125, Jan. 2020.
[12] E. Ferrara, “Contagion dynamics of extremist propaganda in social networks,” Inf. Sci.
(Ny)., vol. 418–419, pp. 1–12, 2017.
[13] H. U. Khan and D. Peacock, “Possible effects of emoticon and emoji on sentiment
analysis web services of work organisations,” Int. J. Work Organ. Emot., vol. 10, no.
2, pp. 130–161, 2019.
[14] S. Gao, J. Hao, and Y. Fu, “The application and comparison of web services for
sentiment analysis in tourism,” in 12th International Conference on Service Systems
and Service Management (ICSSSM), 2015, pp. 1–6.
[15] A. Pak and P. Paroubek, “Twitter as a corpus for sentiment analysis and opinion
mining,” Proc. 7th Int. Conf. Lang. Resour. Eval. Lr. 2010, pp. 1320–1326, 2010.
[16] A. Ortigosa, J. M. Martín, and R. M. Carro, “Sentiment analysis in Facebook and its
application to e-learning,” Comput. Human Behav., vol. 31, no. 1, pp. 527–541, 2014.