MULTILELEVL CONTEXT PYRAMID

NETWORK FOR SENTIMENTAL ANALYSIS

Karre Ranadheer Pulipaka.Akhil Manda Vamshi krishna
Dept. of CSE Dept. of CSE Dept. of CSE
SR University SR University SR University
Warangal, India Warangal, India Warangal, India
2103a51166@gmail.com 2103a51143@gmail.com 2103a51438@gmail.com

Ambadi Sai Kudikala Sai charan

Dept. of CSE Dept. of CSE
SR University SR University
Warangal, India Warangal, India
2103a51546@gmail.com 2103a51470@gmail.com

Abstract—This study describes a way for refining and
visualizing vast amounts of collective intelligence data using a
multilevel sentiment network to increase intuitive and semantic
understanding. The semantic interpretation technique aims to
limit network learning by establishing a stable network
topology that serves as a guiding framework for users. Given
the ubiquity of emotional expression through visual content
such as images and short videos on social media, determining
the mood of such content is gaining popularity. As a result, this
study focuses on sentiment analysis, which encompasses data
categorization, pre-processing techniques, text representations,
learning models, and practical applications. The proposed
module was evaluated using a number of attributes and deep
learning classification algorithms already presented in the
literature..
Keywords—Multilevelsentimentnetwork,semantic
interpretation,pre-processing methods,algorithms.
I. INTRODUCTION
According to research, the sentiment transmitted by visuals
has a substantial influence on visual perception. [1].
Affective material, as opposed to non-emotional indications
within visuals, is more likely to capture viewers' attention.
Viewers have a better understanding of affective
stimuli,which leads to greater engagement with the emotional
aspects of the content[2]. Content that evokes strong feelings
in the audience is more likely to draw in viewers than
content that doesn't. Affective cues are better understood by
viewers, which increases their involvement with the
content's emotional elements. [3], Opinion mining, user
behavior prediction, emotive image retrieval from photos,
gaming scene modeling, and more applications require this.
Because the sentiment elicited and the surrounding items
interact, visual sentiment analysis is more difficult to solve
than many classic visual tasks in image content
understanding. The following are these two primary
challenges:
 Per Interpreting social network photo data requires an
understanding of how objects are perceived at
different scales. The model needs to be able to
perform multi-scale analysis because of the various
item sizes and placements in these pictures in order to
provide an accurate interpretation.
• Different objects in photographs can trigger different
feelings to varying degrees of emotional expression.
 Objects with low emotional expressiveness provide
emotional information through low-level properties.
Conversely, objects with complex semantic meaning,
such as individuals and the objects that belong to
them, usually provoke stronger emotional responses.
[4].
Our review article will focus on the most significant field
of multimodal SA, which works on visual and textual
information posted on social media platforms. Many
people are more inclined to use these data to express
themselves on these platforms. The visual and textual
characteristics will be integrated to establish the general
attitudes conveyed in the postings, which will then be
categorised as positive, negative or neutral.
II. LITERATURE REVIEW
Academic and industrial interest in image sentiment
recognition has been growing. With a focus on sentiment
region detection and visual attention mechanisms,
researchers have produced a number of important and
influential papers on image sentiment recognition. The
evolution of picture sentiment recognition is reviewed in this
part from a number of angles that are directly relevant to our
work.
a. Image Sentiment Recognition:
The study of sentiment polarity sparked by visual content
is known as image sentiment recognition. Based on
technology developments, research in this area can be
divided into three categories: low-level feature methods,
semantic-level feature methods, and high-level feature
methods.
The goal of low-level feature approaches is to use
psychology and image processing theories to correlate
images with sentiment categories. Based on psychological
tests, Wang et al. introduced three fuzzy histograms for each
emotional element. Drawing inspiration from psychology
and art theories, Machajdik and Hanbury integrated low-level
aspects such as color, texture, composition, faces, and skins.
The principles-of-art-based elements that Zhao et al.
proposed center on harmony, balance, and emphasis. Sartori
et al. investigated how color combinations affect emotions
and used art theory to create features and algorithms. Low-
level characteristics are effective, but their capacity to collect
features could describe the sentiment-interfering factors, the
semantic gap between low-level features and high-level
sentiments consistently existed due to the complexity,
fuzziness, and globalism of visual sentiment[5].
The pyramid network based on backbone convolution
network, feature fusion, sentiment important attention,
sentiment relation attention, and classification are the five
elements of our suggested model for picture sentiment
identification that are depicted in Figure 1.
. convolution and pooling. We secondly construct
b. Visual Attention Mechanism:
The visual attention mechanism of the human eye has
been identified by psychologists, and it suggests that
individuals tend to focus on specific regions of the visual
field that capture their interest. This subjective zone often has
a greater impact on human emotion than the objective zone.
Based on this idea, numerous studies have extracted
sentiment areas from photos using deep learning models with
attention processes.
Song et al. suggested using visual attention techniques to
pinpoint sentiment regions in pictures. Wu et al. developed a
multipartition method to identify sentiment-relevant regions.
Yadav and Vishwakarma introduced a residual attention
model with trunk and mask branches that influence the
relevance of different visual regions. Important areas were
determined. [6].
c. Sentiment Region Detection:
Not all of the image's information is useful for sentiment
analysis. Certain areas are more likely than others to provide
significant emotional information, drawing attention and
evoking strong feelings. You and your colleagues examined
how local regions affect the identification of visual sentiment
and focused on the local area that is relevant to human
emotional reaction. Yang and colleagues presented a
technique that uses object and mood scores to automatically
identify locations that are effective. To automatically identify
sentiment regions and focus on important sentiment
components, Xiong et al. created a region-based convolution
neural network. Zheng and colleagues took use of the distinct
roles played by small areas in visual sentiment analysis with
reference to the overall picture. In addition to locating
sentiment regions from a global and local view, image
sentiment is closely related to different levels of visual
features. Rao et al. proposed a multilevel region-based
convolution neural network to utilize different levels of
sentiment regions. The great majority of the above research
focuses on discovering sentiment regions. However, the
semantic relationship between regions is critical for
sentiment representation. Zhang et al. designed a novel
model exploring the relationship between the image
sentiment and semantic object combination in an image. Ten,
they proposed a multilevel correlation analysis model of
sentiment regions to exploit the effects of the interactions
between sentiment regions [7].
III. PROPOSED METHOD
Image Sentiment Recognition Model:
This section presents the proposed model ASRIR
for picture sentiment recognition. An overview of
the importance of the attention-based sentiment
region and the relationship analysis network
Sentiment important attention and sentiment
relational attention are utilized to improve
performance. Using the pyramid network as input,
area features, including multilayer semantic
information, are first extracted from an image.
The pyramidal characteristics reflect the regions
of the image through the architecture of
the significant attention and relation attention
method based on the pyramidal feature to
comprehend the significance of different regions
and the relationships among them. After
extracting the attention weights from multilevel
discriminative representations, these
representations are merged and fed into a fully
connected layer for sentiment recognition. [8].
A. Region Feature Extraction.
The network, modeled like CNN, is very good at
extracting visual features. Consequently, we study the feature
maps that a convolutional neural network extracts and try to
establish a connection between emotion polarity and visual
attributes. The commonly used model for image processing is
called reset architecture, and it greatly enhances the results of
many different tasks, including object identification, picture
segmentation, and image classification. Our core
architecture, which produces image representation without
sacrificing generality, is ResNet. The ImageNet image
recognition dataset, which includes over 15 million annotated
photos from 22,000 distinct categories, is used to pretrained
the ResNet50 network [41]. From the convolutional layer, we
extract the feature F ∈ Rh×w×c for image x, where h and w
are the height and width of, in which h and w are the height
and width of the feature map, and c is the number of
channels[9].
B. Sentiment Important Attention:
Image area plays a major role in sentiment expression.
However, different portions of an image have varying
contributions to the sentiment polarity prediction. Using a
global visual feature straight out of a convolution network to
predict sentiment could not function well due to the
irrelevant regions. We generate sentiment maps based on
spatial dimensions since the feature map has both spatial and
channel dimensions. much attention to selecting the semantic
features and emphasizing the sentiment-related sections of
the channel of different channels. The architecture of
sentiment important attention. We introduce the detail of
spatial wise important attention and channel wise importan
attention in the following.
Image area plays a significant role in the expression of
sentiment. However, different regions of an image have
varying contributions to sentiment polarity prediction.
Sentiment prediction with a global visual feature directly out
of a convolution network is challenging due to the irrelevant
regions. We generate sentiment maps based on the spatial
dimensions because the feature map includes both channel
and spatial dimensions. extreme caution in selecting the
semantic characteristics and emphasizing the sentiment-
related regions of the channel
The sentiment essential attention illustrated in Figure 3 is
divided into two branches: the spatially wise sentiment
important attention and the channel wise sentiment
important attention.
The distribution of spatial attention over all visual regions is
produced via a sigmoid function in conjunction with two 1 ×
1 convolution layers.
.
C. Sentiment Relation Attention.
In addition to the importance of each individual
zone, understanding the visual sentiment of a
picture also vitally depends on how those regions
interact with one another. To explore the semantic
relationship between regions, we design sentiment
relation attention, which consists of channelwise
sentiment relation attention for analyzing the
semantic relation between channels and
spatialwise sentiment relation attention for
examining the relationship between spatial
regions. The architecture of sentiment relation
attention is shown in Figure 4. The following
describes the specifics of the channel- and space-
related attention. Spatially-wise relation attention
is used to investigate the relationship between
many regions.
. The region feature vector is obtained by one 1 × 1
convolution layer and one reshape operation[10].
Figure 4: Spatial wise sentiment relation attention and
channel wise sentiment relation attention are the two
branches that make up the sentiment relation attention
depiction.
The multilayer pyramid network's feature maps are
designated as F1, F2, and F3. The pyramidal features
improve image sentiment detection by focusing on small
differences in image regions from different sizes and
locating samples on different levels.
IV. EXPERIMENTAL RESULTS
1) Datasets:
Six benchmark datasets for image sentiment recognition—
Abstract, IAPSa, Artphoto, TwitterLDL, FlickrLDL, and FI
(Flickr and Instagram)—are used in our investigations. The
eight sentiment categories of the Mikels emotion model are
used to annotate these datasets. Table 1 displays the datasets'
statistics, and the following is a quick synopsis of them.
2) Abstract:
228 photos with a combination of color and texture make up
the dataset [2]. Approximately 230 individuals annotated
these photos by choosing the most appropriate emotional
categories. The category with the most votes without any
indeterminism is the ground truth..
3) IAPSa:
The dataset is a portion of the 1,182 picture, multi-content
International Afective Picture System (IAPS), an emotional
image collection that is frequently utilized in studies on
emotion and attention. From the IAPS dataset, IAPSa
chooses 209 negative and 186 positive photos, labeling them
with eight sentiment categories.
4) Implementation Details:
We use Kera's framework together with TensorFlow to
implement the suggested model. ResNet101, the main
network, underwent pretrained training on ImageNet.
Adaptive moment estimation (Adam) was used to train the
models over a 100 epoch GPU run. FI, Twittered, and Flared
are the large datasets where the batch size was set to 64. The
learning rate was initialized to 0.0001 and reduced by 10
every 10 epochs. in the small-scale datasets, Abstract,
IAPSa, and Art photo, the batch size was set to 32. The
learning rate was initialized to 0.001, and the decay strategy
is the same as above. The training images were resized to
256 × 256 and randomly cropped into a 224× 224 sub-
images. We employed data augmentation techniques, such
as random horizontal fipping and random cutout. Tese
preprocess methods can help avoid overftting problems and
improve generalization ability. The image data was
normalized to zero and one before inputting the network. In
the test stage, we resized the image and randomly cropped it
into a subimage. We run the model on the dataset three
times and calculate the average result as the recognition
performance [11].
categories and the identification of eight categories. Eight
sentiment categories are reduced to two, with the labels
"positive" for amusement, amazement, contentment, and
enthusiasm and "negative" for wrath, disgust, fear, and
sadness. Examining the binary sentiment recognition
outcomes, we juxtapose our model against three widely-used
CNN-like networks. In order to confirm the effectiveness of
the suggested model, we examine the confusion matrices of
eight sentiment categories. Table 2 displays the outcome for
the Art photo dataset. We use precision, recall, F1, and
accuracy as our four evaluation metrics. Table 2 illustrates
that our suggested method's accuracy is 80.86%. Our
method with relation attention for sentiment recognition
improves the performance by 6.41% for accuracy compared
to the baseline model ResNet101. This indicates the relation
attention can model the relationship between local regions.
Additionally, our method with important attention improves
the performance by 5.80% for accuracy, indicating the
effectiveness of sentiment important mechanism. In terms of
the four metrics of each sentiment category, our method
with sentiment essential attention and relation attention
performs nearly better than any other way. The results for
two categories on the dataset FI are displayed in Table 3.
Overall, our proposed method consistently outperforms
others. In particular, the F1 score for the positive and
negative categories can achieve 91.77% and 91.21%, which
improves the backbone network ResNet101 by 6.86% and
4.08%. Recalling the outcome of the Art photo dataset,
sentiment important attention and relation attention-
recognizing models outperform baseline models overall,
further highlighting the benefits of sentiment significant
attention and sentiment relation attention[12].
The impact of parameter b in our suggested approach on the
FI dataset is seen in Figure 7. −P represents the positive
category's performance, and −N represents the negative
category's performance.
focus can enhance picture sentiment recognition
performance and produce more accurate recognition results
for each sentiment category.
.
V. CONCLUSION
In this paper, we investigate the problem of image sentiment
recognition. In response to the observation that local regions
have varying relevance for sentiment response and that the
interaction between regions significantly contributes to
visual sentiment, we provide a system to automatically
analyze the importance and relationship of regions on
multilevel deep feature maps. We extract the
multidimensional sentiment regions based on the backbone
network ResNet101 by merging the multilevel features
using the pyramid network. Considering the intricacy of the
emotion-evoking areas, we design the sentiment relation
attention and the sentiment important attention to evaluate
the regions for image sentiment detection. Experiments on
several popular datasets have produced results that
demonstrate the proposed framework can beat other state-of-
the-art methods and attain excellent
ACKNOWLEDGMENT
We are grateful to SR university Management for their
assistance with this endeavor.
REFERENCES
[1] Fan, S.; Shen, Z.; Jiang, M.; Koenig, B.L.; Xu, J.; Kankanhalli, M.S.;
Zhao, Q. Emotional attention: A study of image sentiment and visual
attention. In Proceedings of the IEEE Conference on Computer Vision
and Pattern Recognition, Salt Lake City, UT, USA, 18–22 June 2018.
[2] Brosch, T.; Pourtois, G.; Sander, D. The perception and categorisation
of emotional stimuli: A review. Cogn. Emot. 2010, 24, 377–400.
[CrossRef]
[3] Ortis, A.; Farinella, G.M.; Battiato, S. Survey on visual sentiment
analysis. IET Image Process. 2020, 14, 1440–1456. [CrossRef]
[4] Article Multi-Level Context Pyramid Network for Visual Sentiment
Analysis Haochun Ou, Chunmei Qing *, Xiangmin Xu and Jianxiu
Jin.
[5] Attention-Based Sentiment Region Importance and Relationship
Analysis for Image Sentiment Recognition Shanliang Yang , 1 Linlin
Xing , 1 Zheng Chang , 1 and Yongming Li 2.
[6] E. Ragusa, T. Apicella, C. Gianoglio, R. Zunino, and P. Gastaldo,
Design and Deployment of an Image Polarity Detector with Visual
Attention, Cognitive Computation, Springer, Berlin, Germany, 2021.
[7] J. Zhang, X. Liu, M. Chen, Q. Ye, and Z. Wang, “Image Sentiment
Classifcation via Multi-Level Sentiment Region Correlation
Analysis,” Neurocomputing, vol. 469, pp. 221–233, 2021.
[8] J. A. Mikels, B. L. Fredrickson, G. R. Larkin, C. M. Lindberg, S. J.
Maglio, and P. A. Reuter-lorenz, “Emotional category data on images
from the international afective picture system,” Behavior Research
Methods, vol. 37, no. 4, pp. 626–630, 2005.
[9] J. Machajdik and A. Hanbury, “Afective image classifcation using
features inspired by psychology and art theory,” in Proceedings of the
18th International Conference on Multimedea2010, pp. 83–92, IEEE,
Firenze, Italypp, October 2010.
[10] L. Wu, M. Qi, M. Jian, and H. Zhang, “Visual Visual Sentiment
Analysis by Combining Global and Local Informationentiment
analysis by combining global and local information,” Neural
Processing Letters, vol. 51, no. 3, pp. 2063–2075, 2020.
[11] S. Zhao, X. Yao, J. Yang et al., “Afective Image Content Analysis:
Two Decades Review and New Perspectives,” IEEE Transaction on
Pattern Analysis and Machine Intelligence, vol. 44, no. 10, pp. 6729–
6751, 2021.
[12] J. Yang, M. Sun, and X. Sun, Learning Visual Sentiment Distributions
via Augmented Conditional Probability Neural Network, pp. 224–
230, Springer, Berlin, Germany, 2017.