Neural Computing and Applications

https://doi.org/10.1007/s00521-022-07790-5 (0123456789().,-volV)(0123456789().
,- volV)

ORIGINAL ARTICLE

Multimodal tweet classification in disaster response systems using

transformer-based bidirectional attention model
Rani Koshy1,2 • Sivasankar Elango1

Received: 12 March 2022 / Accepted: 6 September 2022

 The Author(s), under exclusive licence to Springer-Verlag London Ltd., part of Springer Nature 2022

Abstract
The goal of this research is to use social media to gain situational awareness in the wake of a crisis. With the developments
in information and communication technologies, social media became the de facto norm for gathering and disseminating
information. We present a method for classifying informative tweets from the massive volume of user tweets on social
media. Once the informative tweets have been found, emergency responders can use them to gain situational awareness so
that recovery actions can be carried out efficiently. The majority of previous research has focused on either text data or
images in tweets. A thorough review of the literature illustrates that text and image carry complementary information. The
proposed method is a deep learning framework which utilizes multiple input modalities, specifically text and image from a
user-generated tweet. We mainly focused to devise an improved multimodal fusion strategy. The proposed system has a
transformer-based image and text models. The main building blocks include fine-tuned RoBERTa model for text, Vision
Transformer model for image, biLSTM and attention mechanism. We put forward a multiplicative fusion strategy for
image and text inputs. Extensive experiments have been done on various network architectures with seven datasets
spanning different types of disasters, including wildfire, hurricane, earth-quake and flood. Several state-of-the-art
approaches were surpassed by our system. It showed good accuracy in the range of 94–98%. The results showed that
identifying the interaction between multiple related modalities will enhance the quality of a deep learning classifier.

Keywords Disaster tweet classification  Multimodal data fusion  BiLSTM  Attention  RoBERTa  Vision transformer

1 Introduction disaster happened. The unpredictability of the emergency

Disasters can happen any time, anywhere. They cause
massive destruction and are sometimes uncontrollable.
Human-made disasters, such as industrial explosions,
armed conflicts and riots result knowingly or unknowingly
by humans. Natural disasters like earthquakes, drought etc.,
occur due to natural phenomena. In whatever form it is,
lives and livelihood of people, the environment and the
economy are proportionally affected by the intensity of the
& Rani Koshy
406320002@nitt.edu
Sivasankar Elango
sivasankar@nitt.edu
1
Department of Computer Science and Engineering, National
Institute of Technology, Trichy, Tamilnadu 620015, India
2
Department of Computer Science and Engineering, College
of Engineering, Trivandrum, Kerala 695016, India
makes things worse. This is the scenario where disaster
response systems play a vital role. Quick response and
rehabilitation is the key to the success of a disaster man-
agement system. Delayed service means no service.
Crisis informatics or disaster informatics [36] is an area
of study exploiting information and technology in different
phases of disasters and other emergencies. It focuses on
interconnecting people, technology, organizations and
information during crises/disasters. It views the citizenry as
a powerful, collectively intelligent and self-organizing
force that can play a transformational and indispensable
role in emergency management.
In disaster informatics, data is the driving force. Social
media sites have grown in popularity as a result of
advances in information and communication technology.
They can be considered as a significant source of real-time
information. In today’s world, the first to report on an issue
is always social media [30].

123

Physical sensors, human sensors, and social media users
are all data sources for social media analysis [37]. In the
proposed system, we focus on data supplied by social
media users, particularly the general public. Early warning
and event detection, situational awareness, resource col-
lection and dissemination and post-disaster analysis are
some ways that social media can be used in disaster
informatics.
In the proposed work, we concentrate on getting situa-
tional awareness from the available data. For this, we have
to identify whether the social media content is informative
or not informative. Situational awareness is when you are
aware of what is going on around you [11] or—in a tech-
nical manner—as ‘‘the perception of the elements in the
environment within a volume of time and space, the
comprehension of their meaning and the projection of their
status in the near future’’ [10].
Social media analysis has numerous obstacles in this era
of data deluge. The data on social media is overflowing
with noisy data. When a crisis occurs, social media
becomes overburdened with messages, making it impossi-
ble for rescuers to keep track of them. It’s challenging to
pick out messages that require immediate attention from
the massive volume of texts. The nonstandard terminology
and brevity of social media material also make it difficult
to find important information. In addition to textual mate-
rial, users can contribute images and videos from the crisis
region, which can improve situational awareness. The
studies [2, 40–42, 50, 51] say that single modality analysis
is not sufficient for getting good results.
In the past, the majority of social media research
focused on text-only or image-only data. However, when
image and text data are combined, they reveal comple-
mentary information. Multimodal data analysis is an active
open research area. Using various modalities provides
more contextual information, allowing more robust learn-
ing. Thus it necessitates an information processing system
which can automatically identify disaster-relevant tweets
by considering both text and image. Figure 1 displays some
tweets from recent emergency situations that include text
and related images.
This research aims to develop a classification system
that considers both the text and the associated images to
determine whether a tweet is informative or not. Identify-
ing disaster-relevant information is an essential require-
ment for authorities to deliver efficient and on-time
recovery operations.
Multimodality can be dealt with a variety of ways,
including integrating various models at the output
level(referred to as decision-level or late fusion) or at the
feature level (referred to as feature-level or early fusion)
[14]. Late Fusion: Fusion takes place at the decision-
making level. For each input modal, there will be a
123
Neural Computing and Applications
different classification model. Individual classification
model decisions are merged in some way to arrive at a final
judgement. Early Fusion: Here, fusion is taking place at the
feature level. Different features will be merged in some
manner and subsequently processed.
We explored a variety of text and image models, as well
as experiments, to better understand how text and image
features interact. For text and image processing, we used
cutting-edge transformer-based models. To identify the
interaction between different input modalities, we put
forward an early fusion strategy.
We performed extensive experiments with CrisisMMD
[2] which is a real-world dataset, collected from Twitter
during 7 major natural disasters occurred at different parts
of the world. The experimental setup is the same for all the
experiments to get comparable results. We tested our sys-
tem in two scenarios to confirm its viability in real-world
use. In-domain Classification: The same dataset’s frag-
ments are used for both training and testing. Cross-domain
Classification: The classifier is tested on a different dataset
after being trained on one. The results showed that the
proposed approach is more robust than baseline models and
single-modality systems.
Even though the proposed approach showed good
results, there are still chances for improvement. With
greater hardware resources, we can expand the system’s
capacity. Deep learning architectures need a lot of
resources, including greater memory and computing power.
The experiments are constrained by the resources available.
The proposed system helps to improve the situational
awareness for disaster response and recovery operations,
which benefits people’s quality of life. Our findings are
useful to research disciplines that require diverse inputs
from several modalities, such as fake news detection,
question-answering systems and so on.
The contributions of this research are listed below:
• An extensive comparative analysis of various textual,
visual and multimodal systems for classification task in
in-domain and cross domain scenarios is performed.
• Transformer-based image and text processing models
are utilized for multimodal fusion systems.
• A robust deep learning neural network architecture is
proposed that has a novel multi-modal feature fusion
layer and some of the most recent deep learning
techniques such as BiLSTM, ViT model and RoBERTa
model as the building blocks. The proposed model can
exploit the hidden interaction between multiple modal-
ities for the automatic detection of informative tweets
for a disaster response system. Later, the tweet can be
used for disaster recovery and mitigation.
The arrangement of the paper is as follows: Sect. 2 gives a
quick summary of some of the most noteworthy disaster-
Neural Computing and Applications
Fig. 1 Users’ post during disasters
related works. The necessary background details for
building up the multimodal fusion system are elaborated in
Sect. 3. Section 4 illustrates the approach and architecture
of the proposed system. Section 5 lays forth the experi-
mental setup and assesses the outcomes. Finally, Sect. 6
gives a conclusion and prospects of the system.
2 Related works
In this section, we discuss previous research initiatives that
have been relevant to our work. There are text-only anal-
yses, image-only analyses and multimodal analyses. In
some multimodal systems, inputs from multiple sources
have been taken. E.g., different sensors (temperature sen-
sors, humidity sensors), data from weather departments
(wind speed, rainfall level) etc. In the proposed approach,
we take multiple inputs of different modalities from the
tweet. It provides the advantage of quick and immediate
data availability in the event of a calamity. There is no
need to use data from other sources. Several studies have
shown the utility of Twitter text and images in disaster
recovery operations.
2.1 Text-based classification
Sreenivasulu et al. [25] presented a work with random
forest classifier for identifying damage-related tweets.
They utilized lexical features, syntactic features and fre-
quency of the words related to damage assessment. Linear
regression and support vector regression are used to weigh
different features. They achieved accuracy up to 94% on
datasets of earthquakes in Italy, Chily and India floods. In S
Madichetty et al. [29] a stacked Convolutional Neural
Network (CNN) is proposed for recognising tweets con-
veying resource requirement and availability. Crisis word
embedding is used here. They concatenated the output of a
K-nearest neighbour (KNN) classifier and a CNN classifier.
The concatenated output is finally classified by a support
vector machine (SVM) classifier. They achieved accuracy
in the range 67–77% on datasets for different disasters.
Snyder et al. [45] put forward an interactive learning
123

framework for situational awareness classification. The
user can iteratively interact with the system. Whenever the
classification goes wrong, the user corrects the classifier.
Only the tweet text is considered here. As the building
blocks, they employed Word2Vec embedding, CNN, and
Long Short Term Memory (LSTM). This system is inte-
grated with the SMART2.0 toolkit.
To aid victims who require medical assistance, Sreeni-
vasulu et al. designed a majority voting-based ensemble
classifier which can identify medical resource tweets [28].
They achieved 82.4% accuracy. Zahra et al. [52] used a
random classifier with linguistic characteristics to identify
eye-witness messages. They defined features exclusive to a
direct eye-witness, such as words indicating perceptual
senses, first-person pronouns and adjectives.
In Ghafarian et al. [13] an approach based on a lin-
guistic concept known as ‘distributional hypothesis’ is
proposed. A tweet is modelled as a distribution of words.
An SVM classifier predicts a label for a distribution. They
showed that the idea is superior than the bag-of-words
(BOW) model. They tested their system with several
datasets and achieved an accuracy of (74–80)%. Kejriwal
et al. [19] proposed a system to detect disaster-related
urgent messages using minimally-supervised approach.
They trained the system with labelled and unlabelled
tweets. This approach is suitable for adapting to a new
crisis.
2.2 Image-based classification
Alam et al. [1] developed the Image4Act, a deep neural
network framework for image classification. They utilized
social media images posted during a disaster to get the
situational awareness. They fine-tuned VGG16 and
achieved 67% accuracy on relevancy classification.
Alam et al. [3] implemented image classification for
damage level assessment. The images from disaster areas
will be very messy and difficult to understand, even for
humans. They implemented real-time image capturing,
deduplication and information extraction. For image clas-
sification and deduplication, they used the VGG16 model
and the perceptual hashing technique, respectively. Their
results showed that by utilizing social media imagery,
emergency responders can deliver relief efforts effectively.
Chaudhuri et al. [7] developed a CNN model to classify
images having human body parts out of the debris. With
this system, the emergency responders can get information
about trapped survivors. Their system was suited for a
smart city environment. They achieved an accuracy of
83.2%.
Valdez and Godmalin [49] proposed a lightweight CNN
with two classification heads for identifying the type and
intensity level of natural calamity. They used a fine-tuned
123
Neural Computing and Applications
MobileNetV3 with feed forward neural network (FFN)
added for classification. They developed a dataset for
natural disasters (wildfire, flood, earthquake and volcanic
eruption) and disaster intensity levels (severe, moderate
and insignificant). They achieved an accuracy of 96.8% for
disaster type and 93.2% for intensity level.
Kyrkou and Theocharides [21] developed ERnet, a
computationally efficient CNN model with residual con-
nections for aerial image classification suitable for
unmanned aerial vehicles (UAV). They introduced an
aerial image dataset, including images of fire, flooding,
ruined buildings and traffic accidents. They achieved an
accuracy of 90.1% with a reduced memory requirement.
2.3 Multimodal approach
Rizk et al. [39] developed a multimodal framework with
two stages suitable for energy-constrained devices to pro-
cess social media tweet texts and images. Level 1 classi-
fiers process image and text. These classifiers’ decisions
are combined and used to train level 2 classifier. They
attained an accuracy of 92.43%.
Mouzannar et al. [32] developed a multimodal deep
learning model to identify damage-related information. For
image processing, pretrained Inception model is used, and
for text processing, CNN-based neural network is used.
Finally, the text features and image features are combined
and given to the FFN classifier. They achieved an accuracy
of 92.62%.
Mohanty et al. [31] conducted a case study of Hurricane
Irma. For a relevancy classifier, they proposed a multi-
modal technique. They used four different classifiers: a
classifier to identify tweets with geospatial attributes, an
image classification model, a user authenticity classifica-
tion model and a text classification model to find tweets
that were just about Hurricane Irma. The results of the four
models are combined. If the score exceeds a predetermined
level, the social media post is classified as disaster relevant.
They used a decision-level fusion method.
Another multimodal approach was proposed by Kumar
et al. [20] to classify disaster-related informative content.
The feature vector generated by LSTM and VGG16 is
concatenated and further passed through an FFN. They
achieved F1-score ranging from 0.61 to 0.92 for various
datasets.
Madichetty et al. [26] proposed a multimodal approach
utilizing BERT language model and DenseNet image
model to analyze tweet text and associated image. They
implemented a late fusion approach. The output probability
vectors from the text model and image model are averaged.
This value is taken for prediction.
Ofli et al. [34] proposed a deep learning neural network
that realizes a multimodal approach with image
Neural Computing and Applications
classification using VGG16 and text classification using
word2vec and CNN. Image feature vector and text feature
vector are concatenated and given to an FFN for final
classification. They achieved an accuracy of 78.4%.
Existing multimodal fusion approaches are not promis-
ing for handling complex multimodal and high-dimen-
sional data. This is due to the fact that:
• Existing systems consider only a single modality input
and learn the pattern in that modality. They cannot
identify relationships across different input modalities.
• The existing systems are unable to prioritise different
features in the order of importance. They value all
features equally.
As a solution, we require a framework for classification
techniques based on feature-fusion that identify crucial
latent associations in input data from different modalities.
It must place varying degrees of emphasis on various ele-
ments (e.g., different words, different regions of images,
etc.) depending on their importance.
3 Preliminaries
This section provides an overview of the key components
of the proposed system. The proposed system constitutes a
text preprocessing module, RoBERTa (Robustly Opti-
mized BERT Pretraining Approach) text model, ViT
(Vision Transformer) image model, BiLSTM (Bi-direc-
tional Long Short Term Memory) and attention module.
3.1 Text preprocessing
Text is an unstructured form of data. Text preprocessing
cleans up the raw data, making it more precise and man-
ageable for the text processing system [15]. Typically, a
social media tweet will be informal and contains several
redundant information. To mitigate these issues, prepro-
cessing is done. From the extensive experiments we had
done, we found that preprocessed text gives better perfor-
mance [44]. Common text preprocessing methods includes:
• Expanding contractions
• Changing the case of the letters to lower case
• Removing punctuations
• Removing stopwords
• Removing URLs
• Stemming and lemmatization
• Removing extra spaces
3.2 Word embedding
Word embedding is a distributed learned representation for
text in the form of real-valued vector. The words having
the same meaning will have a representation that is more or
less similar. This is one of the key breakthroughs in deep
learning techniques for Natural Language Processing
(NLP).
3.2.1 RoBERTa
BERT (Bi-directional Encoder Representations from
Transformers), introduced by Google Brain in 2018 [8], is
a milestone in the realm of NLP. It is pretrained on
BookCorpus and Wikipedia. It outperformed several NLP
systems. The basic building block of BERT is transformer,
which is an improvement over traditional encoder-decoder
systems. Later, several improvements over BERT in terms
of performance and training speed were proposed, such as
XLNet, RoBERTa, DistilBERT etc. RoBERTa, introduced
by Facebook, outperformed BERT in over 20 NLP tasks on
GLUE benchmark datasets.
In the proposed approach, RoBERTa [22] model is used
to generate word embedding. We experimented with sev-
eral word embedding techniques, and the results are shown
in the Table 2. RoBERTa showed promising results.
RoBERTa is a transformer-based NLP model. It generates
dynamic word embedding.
3.3 Fine-tuned RoBERTa
Fine-tuning is an approach to transfer learning. In a real-
world scenario, it is difficult to hold the assumption that the
training dataset and the test dataset follow the same dis-
tribution and have the same statistical property. This con-
straint can be overcome by transfer learning. In [6] transfer
learning is described as a technique to transfer and apply
the knowledge acquired from one task to perform another
task.
We can employ a pretrained model for a related but
different task through fine-tuning. Then, the model will
start with the learned weights of the pretrained model and
later be trained to learn the task-specific features. Hence,
the system gets a good starting point rather than a ran-
domized state. Fine-tuning is particularly useful where the
labelled dataset is scarce. The benefits of fine-tuning are:
• Overfitting is less likely to occur.
• The model converges faster with reduced computational
resources and time.
The advantage of fine-tuning comes from the fact that the
pretrained model has a deep contextual understanding.
With fine-tuning, the knowledge acquired by the pre-
123

trained model can be considered for solving the current
problem.
In [48], the authors have used the technique of fine-
tuning to utilize the ALBERT language model for cyber-
bullying analysis of social media data. Their system has
achieved state-of-art results with an F1 score of 95%. In
[33], the authors used a fine-tuned BERT model for the
sentiment analysis of the Indonesian user reviews about
mobile apps. Their experiments showed overfitting beha-
viour when the model was trained from scratch. They got
state-of-the-art results with Indo-BERT-Base, a BERT
variant pretrained in the Indonesian language. In [35] the
authors implemented an NLP system for multilingual
grammatical error correction. The results showed that fine-
tuning produces better models with a relatively smaller
dataset and reduced computational power consumption.
In our use case, a properly labelled dataset is lacking at
the onset of a disaster. We used the pre-trained RoBERTa
model, which is later fine-tuned for our task-specific
dataset.
The key characteristics of RoBERTa are:
• It generates contextualized word embedding.
• It takes into account the bidirectional transformer
concept so that it can utilize past and future contexts.
• Pretrained with 10 times more data and 8 times larger
batches than BERT.
• Byte-pair encoding is used for tokenization. So, it can
recognize rare words and out-of-vocabulary tokens.
• In BERT, randomly chosen 15% masks used in the
pretraining is fixed for the entire process. In RoBERTa,
masks are dynamically changed during the training
time.
3.4 BiLSTM
The basic building block of Bi-directional Long Short
Term Memory (BiLSTM) is Long Short Term Memory
(LSTM). LSTM was introduced as a replacement to
Recurrent Neural Network (RNN) to solve the problem of
vanishing gradient.
LSTM has three blocks: forget gate, input gate and
output gate [17]. Other than the addition of gating mech-
anism, the concept of LSTM is similar to RNN. The gating
mechanism helps to remove irrelevant parts and retain
relevant parts from the previous states. Consequently,
LSTM is able to beat the vanishing gradient (i.e. evading
information) problem.
Figure 2 shows a high-level architecture of LSTM.
Equations 1 to 7 show the mathematical formulae for the
forget gate, input gate and output gate. ft ; it and ot represent
forget gate, input gate and output gate at time step t.
Wf ; Wi ; Wo ; bf ; bi and bo indicate the weight and bias
123
Neural Computing and Applications
parameters of forget gate, input gate and output gate of a
single cell in LSTM. r; , þ and k denote the sigmoid
function, element-wise multiplication, element-wise addi-
tion and concatenation operation respectively. xt denotes
the input data at time t. ht is the hidden state at time t. C~t
and Ct represent the candidate cell state and the final cell
state respectively at time t.
The forget gate determines what information is to be
removed and what information is to be retained from the
previous states, the input gate determines what information
is to be taken into consideration from the current inputs,
and the output gate determines what information is to be
passed to the next time step. An LSTM cell has two states:
hidden state and cell state. All these things are manifested
with the help of sigmoid and tanh functions.
I ¼ ht1 k xt ð1Þ
ft ¼ rðWf  I þ bf Þ ð2Þ
it ¼ rðWi  I þ bi Þ ð3Þ
C~t ¼ tanhðWc  I þ bc Þ ð4Þ
Ct ¼ ft  Ct1 þ it  C~t ð5Þ
ot ¼ rðWo  I þ bo Þ ð6Þ
ht ¼ ot  tanhðCt Þ ð7Þ
BiLSTM is a sequence processing neural network model. It
is the improved variant of RNN and LSTM. It is capable of
solving the vanishing and exploding gradient problem.
BiLSTM is made up of two LSTMs. One LSTM pro-
cesses text in the forward direction (from left to right),
while the other processes text in the backward direction
(from right to left). A high-level view of BiLSTM is shown
in Fig. 3.
wt: word embedding of tth word
wft: representation of wt by forward LSTM
wbt: representation of wt by backward LSTM
In the forward direction, the output of forward LSTM is
determined by the current input word vector and the pre-
ceding hidden vector. Similarly, in the backward direction,
the output of backward LSTM is determined by the current
input word vector and the prior hidden vector. The output
of forward and backward LSTMs are then concatenated to
produce the final feature vector. As a result, the available
contextual information is improved. This architecture
materializes the idea that the meaning of a word is deter-
mined by the words that come before and after it. Hence,
BiLSTM is thought to generate a feature representation that
captures more information. This leads to improved
learning.
Neural Computing and Applications
Fig. 2 LSTM
Fig. 3 BiLSTM
3.5 Image models
Several studies have shown that utilizing tweet text along
with the associated images produces better results
[20, 26, 31, 32, 34, 39]. We can think of the image as a
supplementary data to the information in the tweet text.
Accordingly, if we can extract image features efficiently,
the deep learning system’s capability will be increased. It
will boost the amount of contextual data available.
We need image features that meaningfully and precisely
represent the information encompassed in the image.
Convolutional Neural Network (CNN) is the most popular
deep learning algorithm for image feature extraction. CNN
has predicated on the premise that image understanding at a
local level is sufficient. CNN comprises convolutional
layers and pooling layers. Convolutional layers are capable
of extracting complicated features that accurately charac-
terise an image. Downsampling is accomplished via pool-
ing layers. Thereby, CNN is resistant to scale, rotation,
distortion and occlusion.
ImageNet, CIFAR and MNIST are some of the promi-
nent image datasets publicly available. Using these data-
sets, some pretrained image models were developed by
researchers which showed better performance. VGG16,
ResNet152, InceptionV3, EfficientNet and DenseNet are
some of them.
CNN’s inability to understand how the features relate to
one another is their pitfall. To capture long-range depen-
dencies, large convolutional kernels are required. However,
the computing efficiency decreases. Later, the concept of
transformer became popular among researchers, and it
proved effective in retaining long-term dependency and
parallel processing. The transformer-based models are built
around the attention mechanism. With the attention layer,
global data is accessible by the transformer to generate the
token embeddings. Section 3.6 discusses the importance of
attention mechanism.
3.5.1 Vision transformer
Vision Transformer, or ViT, is an image processing model
with a transformer-like architecture [9]. It replaces con-
volutions with transformer. Multi-head self-attention layer
is the key component in the transformer module. In addi-
tion, the transformer has a multi-layer perceptron with
GELU activation function and normalization. Some resid-
ual connections are also incorporated to improve long-
range dependency. The residual connections help the free
flow of data bypass the non-linear activation func-
tions. Figure 4 illustrates the architecture of Vision
Transformer as depicted by Dosovitskiy et al. [9].
123

Transformer is a sequence processing algorithm. So, the
image is split into fixed-sized patches, and these patches
are flattened to vectors. As a result, the image is trans-
formed into a sequence of tokens. These tokens, along with
positional embedding, is input to the transformer encoder.
CNN models use pixel-array, whereas ViT uses a series of
visual tokens.
3.6 Attention mechanism
Attention mechanism [4] is one of the most amazing
breakthroughs in both computer vision and NLP in the past
decade. This is the cornerstone in all transformer-based
systems, such as BERT, XLNet, ViT, CoATNet etc.
As the term indicates, the attention mechanism makes
the model concentrate on the most relevant parts of the
available data while ignoring the other parts. It mimics the
most important cognitive process of the human brain.
It was introduced in the paper [4] by Bahdanau et al. in
2014. Initially, it was used for neural machine translation,
but later its strength in other deep learning models was also
successfully proved.
The attention concept is materialized by a feed forward
neural network (FFN). The basic idea behind the attention
mechanism is to give varying weights to different parts of
the data. The FFN generates an attention score for each of
the encoded input vectors, with the most relevant vectors
being attributed the highest attention weights. The attention
score indicates the relevance of each input element with
respect to the task. This score is then normalized by a
softmax operation. Later, the context vector of the input
data is formed by combining (usually summation) the
attention-weighted input vectors. The following equations
show the entire process.
expðv  ht Þ
at ¼ P ð8Þ
t expðv  ht Þ
X tion from the tweet text
S¼ at :ht ð9Þ
t features from the associated image
ht : Hidden vector of input word at time t
at : Normalized attention weight for ht
S : Attention-weighted context vector for the sentence
v is a trainable parameter which is randomly initialized
and jointly learned by the system to identify the most
attentive word. Over time, several variants of attention
mechanism according to the score calculation and context
vector generation came out.
123
Neural Computing and Applications
4 Proposed system
4.1 Overview
We present a neural network architecture that considers
both text and image to provide improved precision in
obtaining situational awareness from the massive volume
of social media conversational data in the context of a
disaster.
The problem statement can be defined as follows:
Given a social media tweet Z ¼ fx; yg containing both
textual information, x, and visual information, y, we need
to determine whether Z has situational information or not.
We can consider this as a classification task. This can be
formulated mathematically as Eq. 10.
a^ ¼ argmax pðajx; y; hÞ ð10Þ
a2A
x: text
y: image
h: parameters
A: set of classes
a^ : predicted class
Later the tweet containing situational content can be
used by disaster management organisations to accelerate
the recovery operations.
We need a multimodal joint representation, /ðx; yÞ, that
robustly exploits the interaction between textual features
and image features. Using this collective representation,
equation 10 can be learned efficiently.
A high-level view of the overall system architecture is
depicted in the Fig. 5.
Major modules in the proposed system are:
• Text Feature Extraction Module: extracts the informa-
• Image Feature Extraction Module: extracts image
• Fusion Module: combines the features given by the
feature extraction modules. In the proposed system, we
used the early fusion technique.
• Classification Module: Finally, a feed forward neural
network (FFN) is employed for the classification part.
Figure 6 shows the detailed architecture of the proposed
system.
The proposed system achieved superior performance
over state-of-the-art systems.
Neural Computing and Applications
Fig. 4 Vision transformer
Fig. 5 Overall system
architecture
4.2 Feature extraction modules
This section describes the methodology used for extracting
features from different input modalities.
We built two feature extraction modules: one for text
and one for images. Later, a fusion module is used to get
the joint representation of extracted features. To obtain the
final classification result, the fusion module’s output is
routed through an FFN. We used fine-tuned ViT model for
visual feature extraction and RoBERTa model for text
feature extraction.
4.2.1 Textual feature extraction: BiLSTM neural network
with fine-tuned RoBERTa
The text features are extracted using a BiLSTM neural
network together with fine-tuned RoBERTa model.
Tweet text is undergoing a series of transformations to
get fine-grained features. As the first step, text prepro-
cessing is done.
We have used text normalization procedures, including:
• removal of duplicate tweets, stopwords, punctuations,
symbols, retweet tags, HTML tags, URLs and
emoticons.
• conversion to lower case letters
• replacement of contractions: Typically, some word
contractions are used in social media tweets. We had
made an attempt to replace these contractions with the
actual word.
The preprocessed text is further undergone the word
embedding process.
1. Word embedding by RoBERTa model
In the proposed approach, word embedding is
achieved with fine-tuned RoBERTa.
Ft ¼ \ft1 ; ft2 ; . . .; ftn [
Ft represents the tweet text vector after RoBERTa
embedding, where n denotes the number of words in a
tweet text. We used n=60. In the time of a crisis, people
usually send brief tweets. Each fti is a vector of 768
dimension.
fti 2 Rdw
dw denotes the dimension of word vector, which is
123

Fig. 6 Proposed system architecture
of size 768 generated by RoBERTa.
The RoBERTa embeddings are processed further
using BiLSTM.
4.2.2 Visual feature extraction
In the proposed approach, we are using Vision Transformer
(ViT) [9], a transformer-based image classification model.
The authors of [9] claim that ViT achieved 4 times better
results when compared with the state-of-the-art CNN
models.
We have experimented with CNN models and ViT. The
results are shown in the Table 3. ViT delivered a good
performance.
In the proposed approach, we used the ViT_b32 model.
It features 12 layers of transformer encoders, and thus 12
attention heads. Transformer can take only 1d input.
Therefore, the image, X, is converted from the dimension
of H  W  C to a sequence of flattened 2d patches, Xp
having the dimension of N  ðP2  CÞ [9].
123
Neural Computing and Applications
X 2 RHWC ð11Þ
2
CÞ
Xp 2 RNðP ð12Þ
HW
N¼ ð13Þ
P2
H: Height of the image
W: Width of the image
(H, W): the resolution of the image
C: the no. of channels
(P, P) : the resolution of the patch
N: the number of patches
Let D be the transformer hidden vector dimension. Hence,
the 2d image patches, Xp , are converted to patch embed-
ding vectors of dimension D, RND . Similar to BERT, a
trainable class token, CLS, is prepended to the embedded
patches. Hence, there are N?1 patches. In addition to this,
since the transformer does not have any indication of token
position, some positional embeddings, Epos , are also
attached with the patch embeddings.
Epos 2 RðNþ1ÞD
The hidden vector size in ViT_b32 is 768. [9] can be
referred for further details on ViT model.
In the proposed system, we take the output of the
penultimate layer of the ViT. Since this is a classification
task, we are capturing only the output vector of the CLS
token, i.e. the vector of size 768 corresponding to the extra
learnable class embedding entity. The final embedded
vector I 2 R768 is passed to the multimodal interaction
layer.
4.3 Multimodal fusion module
Multimodal fusion is the process of combining data from
many input modalities into a single unit. It exploits the
complementarity of heterogeneous inputs.
We propose a multiplicative fusion approach with a
BiLSTM followed by attention mechanism.
4.3.1 Multimodal interaction layer
This layer combines inputs of multiple modalities.
In the proposed system, early fusion with multiplicative
merging is done. Early fusion can be considered as gen-
erating fine quality features.
Both feature extraction module generates feature vectors
of dimension 768. To reduce the dimensionality of feature
vectors, a nonlinear mapping is done by a feed forward
neural network. Then merging is done by multiplying each
Neural Computing and Applications

pair of elements from both feature vectors. Equations 14– 4.4 Classification module
16 shows the fusion operations implemented.
0
ð14Þ The contextual feature vector, C, is fed to a fully connected
T ¼ f ðT  W0 þ b0 Þ
FFN having a softmax activation function. The classifier is
0
I ¼ f ðI  W1 þ b1 Þ ð15Þ defined as follows:

TI ¼ T  I
0 0 y~ ¼ argmax pðyjCÞ ð20Þ
y2Y
768
T; I 2 R
0 0
pðyjCÞ ¼ softmaxðW  C þ bÞ ð21Þ
T ; I 2 Ru
u
ð16Þ where W is the weight matrix, b is the bias vector, Y is the
du
TI 2 Rd
set of classes, y~ is the predicted label for the feature vector
W0 ; W1 2 Ru768 C, i.e. informative or not_informative.
b0 ; b1 2 Ru The whole model is trained end-to-end with a supervised
learning procedure.
where, T is the tweet text, I is the image, u is the dimension
0 0
of nonlinearly-mapped feature vectors, T and I are the
mapped text and image feature vectors, respectively. 
denotes the outerproduct. W 0 ; W 1 ; b0 and b1 are weight and
bias vectors.
The weight matrix and bias vectors are jointly learned in
the network. Later, TI is passed to BiLSTM with an
attention layer.

4.3.2 Attention layer

In the proposed approach, we use the idea of attention put

forward by Bahdanau [4]. It is robust and globally draws
the relative importance of input words with respect to the
output word.
We apply BiLSTM with attention to the fused text and
image vector to extract the multimodal contextual vector.
Let M; \m1 ; m2 ; . . .; mn [ ; be the biLSTM output of
the fused feature vector. The alignment score
A; \a1 ; a2 ; . . .an [ ; for M is calculated by applying the
tanh function with a feed forward neural network. W
indicates the weight parameters, and b indicates bias
parameters. The network jointly learns W and b. The a
values in Eq. 18 symbolizes the importance of each input
feature.
Hence, the final feature vector, C, is calculated by the
weighted sum of all feature elements as in Eq. 19.
ai ¼ tanhðW T mi þ bÞ ð17Þ
expðai Þ
ai ¼ Pn ð18Þ
j¼1 expðaj Þ

X
n
C¼ ai  m i ð19Þ
i¼1

Now, C represents the fused contextual fine-grained feature

5 Experimental setup
vector for the tweet.
Figure 7 shows the BiLSTM with attention layer.
The execution setup and environment used in our experi-
Next is the classification module.
ments are described in this section.

123
 Neural Computing and Applications

The results obtained and the performance of the proposed

system will be discussed in the next section.

5.1 Dataset

Since our aim is multimodal data analysis, we focused on

Fig. 7 BiLSTM with attention module
To verify the feasibility of the proposed system for
practical deployment, we performed the experiments in two
scenarios.
• In-domain Classification: The training and testing is
done on fragments of the same dataset. We had done a
comparative analysis of in-domain text-only, image-
only and multimodal analysis. Tables 2, 3, 4, 5, 6, 7, 8,
9 and 10 show the results obtained.
• Cross-domain Classification: In in-domain classifica-
tion, training and testing dataset have the same
characteristics and statistical distribution. But, this is
not possible in real-world situations. Any catastrophic
event’s early stages are marked by a dearth of labelled
data about the event itself. Using a similar previous
event to train the classifier is a practical reasonable
solution to this problem. We performed cross-domain
experiments with our proposed model and the system
performed well. Table 11 shows the cross-domain
results.
multimodal datasets. We used the dataset CrisisMMD
published in [2]. This dataset was used extensively in
several state-of-the-art models [12, 20, 25–27].
CrisisMMD gives a bimodal dataset consisting of text
and associated images scrapped from Twitter for seven
disasters including wildfire, earthquake and floods occurred
across different parts of the world in 2017. So, the dataset
has different linguistic and semantic characteristics. The
tweets have one of two labels: informative or non-
informative.
We collected the dataset and performed data cleaning
procedures and data augmentation techniques to make the
dataset meaningful and balanced. Our model was trained
and validated on all these seven datasets. Table 1 shows the
distribution of data in the dataset.
5.2 Hyperparameters
We used NVIDIA GeForce RTX 3060 with 12GB dedi-
cated RAM and 16 GB shared RAM.
The models are built with the TensorFlow platform
using Python programming language. We used fine-tuned
RoBERTa model and fine-tuned ViT model. The choice of
text and image feature extractors is based on our experi-
mental results. RoBERTa and ViT showed better perfor-
mance. The details are shown in Tables 2 and 3.
The sequence length used for text is 60. The image size
is 224x224x3. The optimizer used is ADAM with a
learning rate of 1e-5 and a decay of 1e-6. We used a
stratified train-test split of 80:20 and a 5-fold cross-vali-
dation. The classification layer is a feed forward neural
network having two levels of 1024 and 512 neurons and a
final layer of 3 neurons. The activation functions used in

Table 1 Distribution of data in

Sl. no. Disaster name Tweet annotation Period
different disasters
Informative Non_informative
Text Image Text Image

1 Hurricane Irma 3544 2208 960 2296 2017 Sep 6 to Sep 9

2 Hurricane Harvey 3332 2457 1102 1977 2017 Sep 6 to Sep 9
3 Hurricane Maria 2842 2231 1714 2325 2017 Sep 6 to Sep 9
4 California Wildfires 1253 985 336 604 2017 Sep 6 to Sep 9
5 Mexico Earthquake 1031 841 349 539 2017 Sep 6 to Sep 9
6 Iraq-Iran Earthquake 493 400 104 197 2017 Sep 6 to Sep 9
7 Srilanka Floods 367 252 655 770 2017 Sep 6 to Sep 9

123
Neural Computing and Applications

Table 2 Results of various Text

Sl. no. Model Accuracy Precision Recall F1-measure ROC-AUC score
classification Models
1 RF with TF-IDF 89.67 90.0 90.0 90.0 .9618
2 SVM with TF-IDF 89.32 89.0 89.0 89.0 .9689
3 Gaussian NB 75.0 80.0 75.0 74.0 .7514
4 Stacking Ensemble with tf-idf 90.16 90.0 90.0 90.0 .9652
5 Majority Voting Ensemble 90.67 91.0 91.0 91.0 .9675
6 GLOVE with FFN 88.07 88.0 88.0 88.0 .9540
7 Word2Vec with FFN 88.75 89.0 89.0 89.0 .9594
8 BERT with FFN 90.48 91.0 90.0 90.0 .9639
9 RoBERTa with FFN 92.10 92.0 91.0 91.0 .9680
10 XLNET with FFN 91.14 91.0 91.0 91.0 .9619
11 Proposed System 97.27 97.0 97.0 97.0 .9938

Table 3 Results of various

Sl. no. Model Accuracy Precision Recall F1-measure ROC-AUC score
Image Classification Models
1 VGG16 77.47 77.0 77.0 77.0 .8550
2 VGG19 73.96 74.0 74.0 74.0 .8202
3 InceptionV3 80.51 80.0 80.0 80.0 .8833
4 ResNet50 80.56 81.0 81.0 81.0 .8884
5 DenseNet121 81.59 82.0 82.0 82.0 .8957
6 EfficientNetV2 77.10 77.0 77.0 77.0 .8559
7 ViT 82.20 82.0 82.0 82.0 .9003
8 Proposed system 97.27 97.0 97.0 97.0 .9938

the classification layers are relu and softmax functions. The
batch size used is 8. All the details of the proposed system
are given in Sect. 4. To beat the overfitting problem, we
used dropout regularization of 0.2 and an early stopping
mechanism for validation accuracy.
5.3 Baseline models
To exemplify the effectiveness of our system, we compared
the system with certain baseline models for text classifi-
cation, image classification and multimodal classification.
5.3.1 Text-only classification
We built tweet classification models for tweet text only.
• Gaussian Naive Bayes [38]: Gaussian Naive Bayes is a
variant of the Naive Bayes classifier. It is a simple
algorithm but has good predictive power. It is com-
monly applied for text classification. We implemented a
Naive Bayes’ classifier for tweet text classification with
TF-IDF vectorization.
• SVM [25]: We implemented a SVM (Support Vector
Machine) classifier for tweet text classification with TF-
IDF vectorization. SVM is also proved to be successful
in several text classification tasks. The algorithm tries to
learn the hyper-plane that clearly separates the classes
with a good kernel function. But choosing the right
kernel is not an easy task.
• Random Forest [25]: We have built an RF classifier for
text classification along with TF-IDF vectorization. RF
is a highly successful machine learning algorithm. It is
based on a collection of decision trees, known as
’forest’, learned using the ‘bagging’ method. One
peculiarity of RF is that it can measure the relative
importance of each feature on prediction. By randomly
selecting data samples and features, multiple decision
trees are built, and either the average or the maximum
vote of the results is taken.
• Stacking Ensemble [24, 29]: Multiple heterogeneous
base learners are simultaneously trained. The meta
learner is trained using the base learner’s predictions as
features. An ensemble learner can outperform any
single base model. We implemented a stacking ensem-
ble with decision tree, random forest, KNN and
XGBoost as base learners and logistic regression as
the meta learner.
• Majority Voting Ensemble [28]: Multiple base models
are built, and their predictions are combined. Finally,
the system predicts the class with the most votes.
Ideally, the system will be better than any single model
used. We implemented a majority voting classifier with

123
 Neural Computing and Applications

AdaBoost, XGBoost, Random Forest and SVM as the is implemented.They have done a late fusion strategy
contributing models. with an additive operation.
• Deep learning models: We implemented some deep • Abhinav Kumar (2020) [20]: A multimodal informa-
learning models with the following techniques as word tive tweet classification is done with LSTM and fine-
embedding methods. tuned VGG16 with a concatenative feature fusion.
• Gautam et al. (2019) (mean probability) [12]:
– GLOVE [23]
ResNet50 for image classification and
– Word2Vec [45]
BiLSTM,CNN?GLOVE for text classification are
– BERT [23, 26]
used. The late fusion strategy is done by averaging
– RoBERTa
the class probabilities.
– XLNet
• Gautam et al. (2019) (custom decision policy) [12]:
ResNet50 for image classification and
5.4 Image-only analysis BiLSTM,CNN?GLOVE for text classification are
used. The class probabilities are averaged and given
We built tweet classification models based on image clas- to a classification system with ReLU and softmax
sification using the following pre-trained image models. functions.
• Gautam et al. (2019) (logistic regression decision
• VGG16 [1, 3, 20, 27, 43]
policy) [12]: Image feature extraction is done with
• VGG19 [12]
ResNet50 and text feature extraction is done with
• InceptionV3 [32, 46]
BiLSTM, CNN with GLOVE. Later, the features are
• ResNet50 [12, 16]
concatenated and used for classification.
• DenseNet [18, 26]
• EfficientNetV2 [47]
• ViT [9]
6 Evaluation metrics
5.5 Multimodal analysis
Accuracy, macro-average precision, macro-average recall,
macro-average F1-Score and ROC-AUC score are used as
We experimented with various multimodal fusion tech-
the evaluation metrics for our classification model.
niques and compared the proposed approach with some
existing systems. We carried out both in-domain and cross- TP þ TN
Accuracy ¼ ð22Þ
domain classifications. TP þ TN þ FP þ FN
TP
• Additive fusion Different inputs are fused by addition Precision ¼ ð23Þ
operation. This is suitable for applications that are not TP þ FP
strongly affected by the joint values of the inputs. TP
Recall ¼ ð24Þ
• Concatenative fusion Multiple inputs are fused by TP þ FN
concatenating each other. The argument in support of 2  Precision  Recall
concatenation is that the inputs are not at all modified or F1  Score ¼ ð25Þ
Precision þ Recall
at least limited to some extent. So, the naturality of the
inputs can be preserved. But it cannot capture the where TP: True Positive, TN: True Negative, FP: False
interaction between multiple modalities. Positive, FN: False Negative
• Averaging The inputs are averaged for getting the We have used macro-averaged precision, recall and F1-
fused vector. Score to give equal importance to all the classes. It is the
• Multiplicative fusion Multiple inputs are fused by arithmetic mean of all distinct classes’ respective values.
multiplication operation. Multiplicative fusion is good So, we will get an average measurement per class.
Pn
at learning interaction between multiple modalities. Precisioni
Macro  averaged Precision ¼ i¼1 ð26Þ
• Sreenivasulu et al. (2021) [26] : The authors have n
implemented a multimodal tweet classification model Pn
Recalli
with fine-tuned BERT and DenseNet. They have done a Macro  averaged Recall ¼ i¼1 ð27Þ
n
late fusion strategy with an averaging operation. Pn
• S. Madichetty et al. (2020) [27]: (Multimodal additive F1  Scorei
Macro  averaged F1  Score ¼ i¼1
fusion with VGG16 and CNN) A multimodal tweet n
classification model with a CNN and fine-tuned VGG16 ð28Þ

123
Neural Computing and Applications

Table 4 Results of multimodal fusion strategies on California Wild-

where n is the number of classes.
fires dataset
ROC-AUC score Area Under Curve is the metric to
indicate the capability of a classifier to discriminate Model no. Acc. Precision Recall F1- ROC-AUC score
score
between classes. It is the summary of the Receiver Oper-
ating Characteristic curve. The value can be between 0 and M1 82.0 70.0 74.0 72.0 –
1. A value of 0 means that the model’s prediction is 100% M2 65.0 – – 64.0 –
incorrect, while a value of 1 indicates that the model’s M3 – 74.0 73.0 73.0 –
prediction is 100% correct. The model with an AUC score M4 79.2 – – –
between 0.9 and 1 is considered superior. M5 75.0 – – –
M6 75.3 – – –
M7 95.26 95.0 95.0 95.0 .9905
7 Results and discussion M8 95.26 95.0 95.0 95.0 .9877
M9 97.53 98.0 98.0 98.0 .9930
We tested our system with the benchmark multimodal M10 97.13 97.0 97.0 97.0 .9942
dataset, CrisisMMD. We have done extensive experiments
for text-only analysis, image-only analysis and multimodal
analysis. We combined datasets for the seven disasters in
the CrisisMMD and utilised them as a single dataset for
Table 5 Results of multimodal fusion strategies on Hurricane Harvey
text-only and image-only classification. The experimental Dataset
results for in-domain classification are displayed in
Tables 2, 3, 4, 5, 6, 7, 8, 9 and 10. Bold values in the Model no. Acc. Precision Recall F1- ROC-AUC score
score
tables indicate the best performance metric values obtained
in our experiments. Results for cross-domain classification M1 84.0 68.0 79.0 71.0 –
are shown in Table 11. Results revealed that the proposed M2 77.70 – – 77.60 –
system surpasses existing systems in terms of the afore- M3 – 83.0 82.0 82.0 –
mentioned evaluation metrics by a significant margin. M4 76.0 – – –
M5 73.5 – –
7.1 Text-only classification M6 79.2 – – –
M7 93.91 94.0 94.0 94.0 .9916
We implemented text classification using Naive Bayes, M8 96.01 96.0 96.0 96.0 .9923
SVM, Random Forest, Stacking Ensemble, Majority Vot- M9 94.61 95.0 95.0 95.0 .9904
ing Ensemble and ANN algorithms. We assessed the M10 97.13 97.0 97.0 97.0 .9942
strength of these algorithms with different text vectoriza-
tion methods including TF-IDF, GLOVE, Word2Vec,
BERT, XLNet and RoBERTa. Data augmentation was
done to get balanced class distribution in the dataset. The
Table 6 Results of multimodal fusion strategies on Hurricane Irma
results are shown in Table 2. ROC-AUC curves of the text dataset
classification models are shown in Fig. 8. Ensemble
Model no. Acc. Precision Recall F1- ROC-AUC score
methods and dynamic word embedding techniques give
score
good results. Individually, RoBERTa based FFN system
produces better results. RoBERTa has also been proved to M1 81.0 68.0 75.0 71.0 –
be more robust than other word-embedding systems in M2 73.82 – – 73.55 –
several investigations [5]. RoBERTa is a transformer-based M3 – 81.0 81.0 81.0 –
pretrained model for text processing. It has the capability M4 73.39 – – –
of bidirectional textual processing. It is able to generate M5 71.4 – –
deep contextualized dynamic word embedding because of M6 80.2 – – –
its rigorous and improved training procedures. It is trained M7 95.99 96.0 96.0 96.0 .9928
with dynamic masks and tremendous training data. We M8 95.92 96.0 96.0 96.0 .9929
utilized RoBERTa embeddings in the proposed approach. M9 93.44 94.0 93.0 93.0 .9772
Proposed approach has a gain of 5% in accuracy and 6% in M10 94.72 95.0 95.0 95.0 .9886
F1-score over RoBERTa only model. This exemplifies the
importance of multimodal systems in data analysis.

123
 Neural Computing and Applications

Table 7 Results of multimodal fusion strategies on Hurricane Maria Table 10 Results of multimodal fusion strategies on Srilanka Floods
dataset Dataset
Model no. Acc. Precision Recall F1- ROC-AUC score Model no. Acc. Precision Recall F1- ROC-AUC score
score score

M1 80.0 77.0 80.0 78.0 – M1 89.0 91.0 86.0 88.0 –

M2 72.96 – – 72.84 – M2 91.78 – – 92.00 –
M3 – 83.0 83.0 83.0 - M3 – 94.0 94.0 94.0 –
M4 74.79 – – – M4 72.6 – – –
M5 57.7 – – M5 70.8 – –
M6 79.4 – – – M6 74.14 – – –
M7 92.42 92.0 92.0 92.0 .9746 M7 96.84 97 97 97 .9924
M8 92.50 93.0 92.0 92.0 .9801 M8 94.47 94.0 94.0 94.0 .9892
M9 92.11 92.0 92.0 92.0 .9767 M9 97.23 97.0 97.0 97.0 .9971
M10 98.24 98.0 98.0 98.0 .9962 M10 97.23 97.0 97.0 97.0 .9928

Table 8 Results of multimodal fusion strategies on Iraq-Iran Earth- Table 11 F1 scores of cross domain classification with proposed
quake dataset system; D1: Hurricane Irma, D2: Hurricane Maria, D3: Hurricane
Harvey, D4: California Wildfire, D5: Mexico Earthquake, D6: Sri
Model no. Acc. Precision Recall F1- ROC-AUC score Lanka Floods, D7: Iraq-Iran Earthquake
score
Train set Test set
M1 83.0 74.0 63.0 66.0 –
D1 D2 D3 D4 D5 D6 D7
M2 68.18 – – 67 –
M3 – 79.0 79.0 79.0 – D1 95.0 84.0 88.0 81.0 78.0 90.0 82.0
M4 73.5 – – – D2 73.0 98.0 87.0 80.0 76.0 85.0 79.0
M5 80.2 – – D3 76.0 84.0 97.0 82.0 76.0 90.0 81.0
M6 75.2 – – – D4 74.0 76.0 81.0 97.0 79.0 85.0 78.0
M7 93.94 94.0 94.0 94.0 .9889 D5 74.0 72.0 74.0 72.0 98.0 70.0 79.0
M8 96.46 97 96.0 96.0 .9720 D6 77.0 78.0 81.0 73.0 80.0 97.0 82.0
M9 94.95 95.0 95.0 95.0 .9917 D7 70.0 78.0 80.0 78.0 75.0 69.0 98.0
M10 97.98 98.0 98.0 98.0 .9833

7.2 Image-only classification

Table 9 Results of multimodal fusion strategies on Mexico Earth- We implemented tweet classification with the pretrained
quake Dataset image classification models, including VGG16, VGG19,
Model no. Acc. Precision Recall F1- ROC-AUC score InceptionV3, ResNet50, EfficientNet, DenseNet and a
score transformer-based vision model, ViT. Each of the models
differs in computational complexity. Table 3 shows the
M1 83.0 76.0 81.0 78.0 –
results of various image classification models. ROC-AUC
M2 74.29 – – 74.25 –
curves are also shown in Fig. 9. The authors of ViT
M3 – 73.0 72.0 72.0 –
claimed that ViT achieved 88.55% accuracy in ImageNet
M4 77.3 – – –
1k dataset. We implemented a ViT base-32 based image
M5 74.6 – –
classification model.
M6 77.9 – – – In our experiments, ViT shows better results. It achieved
M7 92.76 93.0 93.0 93.0 .9772 an accuracy gain in the range 0–8% and F1-score gain in
M8 92.52 93.0 93.0 93.0 .9768 the range 0–8% over other models. The strength of ViT is
M9 93.22 93 93 93 .979 that it is a transformer-based model. The embedded image
M10 97.90 98 98 98 .9930 patches are output to a transformer encoder.
ViT enjoys all the benefits of transformers. It has some
residual connections to gain long-range dependency. The

123
Neural Computing and Applications
Fig. 8 ROC AUC curves for text only classification
multihead self-attention layer in transformer enables
information to be embedded globally over the entire image.
CNN, on the other hand, is based on local filters, which
results in poor performance in capturing useful patterns
globally. Also, transformers introduce better parallelization
than CNN.
Proposed approach uses ViT for visual feature extrac-
tion. It shows a clear margin on all performance metrics
over ViT only model, which clearly emphasizes the
necessity of multimodal systems.
7.3 Multimodal classification
If either the text or the image in a tweet is informative, the
tweet is deemed informative. This assumption avoids the
information loss.
In addition to the proposed method, we implemented
additive, concatenative, and averaged fusion strategies
together with RoBERTa and ViT. Proposed method has
multiplicative fusion.
Our experiments included in-domain and cross-domain
scenarios.
• In-domain classification We performed a comparative
analysis of various fusion methods and existing sys-
tems. Details are given in Sect. 5.5.
Table 4, 5, 6, 7, 8, 9 and 10 and Fig. 11 shows the per-
formance metrics of different multimodal systems tested on
seven datasets including Hurricanes—Harvey, Irma, Maria,
Earthquakes—Iraq_Iran, Mexico, Srilanka Floods and
California Wildfires. Figure 10 shows the ROC-AUC
curves for different fusion strategies tested on Hurricane
123

Fig. 9 ROC AUC curves for image only classification
Maria dataset. The naming convention for models used in
the tables and figures is as follows:
Fig. 10 ROC AUC curves for different fusion approaches on
Hurricane Maria dataset
123
Neural Computing and Applications
– M1: Sreenivasulu et al. [26] (Multimodal additive
fusion with fine-tuned BERT and DenseNet)
– M2: Madichetty et al. [27] (Multimodal additive fusion
with VGG16 and CNN)
– M3: Kumar [20]
– M4: Gautam et al. [12] (Mean Probability)
– M5: Gautam et al. [12] (Custom Decision Policy)
– M6: Gautam et al. [12] (Logistic Regression Decision
Policy)
– M7: Additive fusion with RoBERTa and ViT
– M8: Concatenative fusion with RoBERTa and ViT
– M9: Averaged fusion with RoBERTa and ViT
– M10: Proposed System
In all the experiments, the proposed approach shows a
consistent performance).
The proposed approach is a multimodal data fusion
system for tweet classification which takes tweet text and
the associated image as the input. Text and image act as
complementary to each other. Each of the input modals is
processed in the context of the other one. Therefore, we
will get more contextual information. We used RoBERTa
for text feature extraction and ViT for visual feature
extraction.
We tested additive, concatenative, averaged and multi-
plicative fusion strategies. In concatenation-based fusion,
no interaction between different input modals is explored.
In additive and averaged fusion, joint value of features
Neural Computing and Applications
Fig. 11 Performance metric graphs of different multimodal systems
on CrisisMMD dataset
cannot be completely explored. In multiplicative fusion, all
the possible hidden associations between vectors of dif-
ferent modalities may be unearthed. It can effectively
capture the interaction between input modals and identify
matching elements in a better way.
We applied an early fusion strategy that performs mul-
tiplication between each pair of elements. In early fusion,
feature-wise fusion is taking place. Together with this
fusion method, we added BiLSTM and attention mecha-
nism also. BiLSTM helps to analyze context in both
directions. With the help of BiLSTM and attention layer,
we are able to get highly contextualized fine-grained fea-
tures prioritized with respect to the downstream task. We
were able to achieve greater performance with this
architecture.
• Cross-domain classification We carried out the cross
domain experiments with the proposed system for every
possible pair of CrisisMMD dataset. As our focus is to
devise a multimodal fusion technique rather than
domain adaptation, we carried out less comparative
analysis in this case. We reported the F1-score in
Table 11. Non-diagonal entries display the findings
from cross domain experiments. Since F1-score is the
harmonic mean of precision and recall, it can reflect
both metrics as well. The proposed system performed
well in the cross-domain scenario also. This demon-
strates the potential of the proposed system to gener-
alize. The system offers a reliable framework for cross-
domain applications. In light of this, we concluded that
our suggested system offers a feasible, real-world
solution for tweet classification in crisis management.
8 Conclusion
We presented a neural network architecture for informative
content classification from the user-generated posts avail-
able on social media for disaster management systems. The
classification system makes use of both text and image
data. To achieve superior results, NLP, image processing,
and deep learning approaches are integrated in a productive
way. We have implemented a multimodal feature-wise
fusion in the proposed system. RoBERTa is used for text
feature extraction, while ViT is used for image feature
extraction. The system is evaluated for in-domain and cross
domain scenarios on seven multimodal disaster datasets,
including hurricanes, earthquakes, wildfires and floods.
The test results proved that the proposed system is robust.
It attained a good accuracy gain over baselines.
Our findings are helpful in research disciplines that
require diverse inputs from several modalities, such as fake
news detection, question-answering systems and so on. The
proposed methodology can be applied to data from any
social media site; even though we experimented with
Twitter data,
Analysis of user-generated content has several open
research issues. Code mixing and transliteration are
prevalent in social media posts. We intend to examine the
impact of code mixing and transliteration on disaster tweet
classification systems in the future.
Data availability The datasets used and analyzed during the current
study are available online: https://crisisnlp.qcri.org/crisismmd.
Declarations
Competing interests The authors declare that they have no conflict of
interest.
123
 Neural Computing and Applications

References 23. Madichetty S, Muthukumarasamy S (2020) Detection of situa-

tional information from twitter during disaster using deep learn-
ing models. Sādhanā 45(1):1–13
1. Alam F, Imran M, Ofli F (2017) Image4act: online social media
24. Madichetty S, Sridevi M (2019) Disaster damage assessment
image processing for disaster response. In: Proceedings of the
from the tweets using the combination of statistical features and
2017 IEEE/ACM international conference on advances in social
informative words. Soc Netw Anal Min 9(1):1–11
networks analysis and mining 2017, pp 601–604
25. Madichetty S, Sridevi M (2021) A novel method for identifying
2. Alam F, Ofli F, Imran M (2018) Crisismmd: multimodal twitter
the damage assessment tweets during disaster. Future Gener
datasets from natural disasters. In: Twelfth international AAAI
Comput Syst 116:440–454
conference on web and social media
26. Madichetty S, Muthukumarasamy S, Jayadev P (2021a) Multi-
3. Alam F, Ofli F, Imran M (2018) Processing social media images
modal classification of twitter data during disasters for humani-
by combining human and machine computing during crises. Int J
tarian response. J Ambient Intell Hum Comput
Hum Comput Interact 34(4):311–327
12(11):10223–10237
4. Bahdanau D, Cho K, Bengio Y (2014) Neural machine translation
27. Madichetty S et al (2020a) Classifying informative and non-in-
by jointly learning to align and translate. arXiv preprint arXiv:
formative tweets from the twitter by adapting image features
1409.0473
during disaster. Multimed Tools Appl 79(39):28,901-28,923
5. Balakrishnan V, Shi Z, Law CL et al (2021) A deep learning
28. Madichetty S et al (2020) Identification of medical resource
approach in predicting products’ sentiment ratings: a comparative
tweets using majority voting-based ensemble during disaster. Soc
analysis. J Supercomput 2021:1–21
Netw Anal Min 10(1):1–18
6. Bengio Y, Courville A, Vincent P (2013) Representation learn-
29. Madichetty S et al (2021) A stacked convolutional neural network
ing: a review and new perspectives. IEEE Trans Pattern Anal
for detecting the resource tweets during a disaster. Multimed
Mach Intell 35(8):1798–1828
Tools Appl 80(3):3927–3949
7. Chaudhuri N, Bose I (2019) Application of image analytics for
30. Martinez-Rojas M, del Carmen Pardo-Ferreira M, Rubio-Romero
disaster response in smart cities. In: Proceedings of the 52nd
JC (2018) Twitter as a tool for the management and analysis of
Hawaii international conference on system sciences
emergency situations: a systematic literature review. Int J Inf
8. Devlin J, Chang MW, Lee K et al (2018) Bert: pre-training of
Manag 43:196–208
deep bidirectional transformers for language understanding.
31. Mohanty SD, Biggers B, Sayedahmed S et al (2021) A multi-
arXiv preprint arXiv:1810.04805
modal approach towards mining social media data during natural
9. Dosovitskiy A, Beyer L, Kolesnikov A et al (2020) An image is
disasters—a case study of hurricane irma. Int J Disaster Risk
worth 16x16 words: transformers for image recognition at scale.
Reduct 54(102):032
arXiv preprint arXiv:2010.11929
32. Mouzannar H, Rizk Y, Awad M (2018) Damage identification in
10. Endsley MR (1995) Toward a theory of situation awareness in
social media posts using multimodal deep learning. In: ISCRAM
dynamic systems. Hum Factors 37(1):32–64
33. Nugroho KS, Sukmadewa AY, Wuswilahaken DWH et al (2021)
11. Flin R, O’connor P, Crichton M (2017) Safety at the sharp end: a
Bert fine-tuning for sentiment analysis on Indonesian mobile apps
guide to non-technical skills. CRC Press, Boca Raton
reviews. In: 6th international conference on sustainable infor-
12. Gautam AK, Misra L, Kumar A et al (2019) Multimodal analysis
mation engineering and technology 2021, pp 258–264
of disaster tweets. In: 2019 IEEE fifth international conference on
34. Ofli F, Alam F, Imran M (2020) Analysis of social media data
multimedia Big Data (BigMM). IEEE, pp 94–103
using multimodal deep learning for disaster response. arXiv
13. Ghafarian SH, Yazdi HS (2020) Identifying crisis-related infor-
preprint arXiv:2004.11838
mative tweets using learning on distributions. Inf Process Manag
35. Pajak K, Pajak D (2022) Multilingual fine-tuning for grammatical
57(2):102145
error correction. Expert Syst Appl 200(116):948
14. Gunes H, Piccardi M (2008) Automatic temporal segment
36. Palen L, Anderson KM, Mark G et al (2010) A vision for tech-
detection and affect recognition from face and body display.
nology-mediated support for public participation & assistance in
IEEE Trans Syst Man Cybern Part B (Cybernetics) 39(1):64–84
mass emergencies & disasters. ACM-BCS Vis Comput Sci
15. Han J, Pei J, Kamber M (2011) Data mining: concepts and
2010:1–12
techniques. Elsevier, Amsterdam
37. Phengsuwan J, Shah T, Thekkummal NB et al (2021) Use of
16. He K, Zhang X, Ren S et al (2016) Deep residual learning for
social media data in disaster management: a survey. Future
image recognition. In: Proceedings of the IEEE conference on
Internet 13(2):46
computer vision and pattern recognition, pp 770–778
38. Pourebrahim N, Sultana S, Edwards J et al (2019) Understanding
17. Hochreiter S, Schmidhuber J (1997) Long short-term memory.
communication dynamics on twitter during natural disasters: a
Neural Comput 9(8):1735–1780
case study of hurricane sandy. Int J Disaster Risk Reduct
18. Huang G, Liu Z, Van Der Maaten L et al (2017) Densely con-
37(101):176
nected convolutional networks. In: Proceedings of the IEEE
39. Rizk Y, Jomaa HS, Awad M et al (2019) A computationally
conference on computer vision and pattern recognition,
efficient multi-modal classification approach of disaster-related
pp 4700–4708
twitter images. In: Proceedings of the 34th ACM/SIGAPP sym-
19. Kejriwal M, Zhou P (2020) On detecting urgency in short crisis
posium on applied computing, pp 2050–2059
messages using minimal supervision and transfer learning. Soc
40. Shah R, Zimmermann R (2017) Multimodal analysis of user-
Netw Anal Min 10(1):1–12
generated multimedia content. Springer, Berlin
20. Kumar A, Singh JP, Dwivedi YK et al (2020) A deep multi-
41. Shah RR, Yu Y, Zimmermann R (2014) Advisor: personalized
modal neural network for informative twitter content classifica-
video soundtrack recommendation by late fusion with heuristic
tion during emergencies. Ann Oper Res 2020:1–32
rankings. In: Proceedings of the 22nd ACM international con-
21. Kyrkou C, Theocharides T (2019) Deep-learning-based aerial
ference on multimedia, pp 607–616
image classification for emergency response applications using
42. Shah RR, Mahata D, Choudhary V et al (2018) Multimodal
unmanned aerial vehicles. In: CVPR workshops, pp 517–525
semantics and affective computing from multimedia content. In:
22. Liu Y, Ott M, Goyal N et al (2019) Roberta: a robustly optimized
Intelligent multidimensional data and image processing. IGI
bert pretraining approach. arXiv preprint arXiv:1907.11692
Global, pp 359–382

123
Neural Computing and Applications
43. Simonyan K, Zisserman A (2014) Very deep convolutional net-
works for large-scale image recognition. arXiv preprint arXiv:
1409.1556
44. Singh T, Kumari M (2016) Role of text pre-processing in twitter
sentiment analysis. Procedia Comput Sci 89:549–554
45. Snyder LS, Lin YS, Karimzadeh M et al (2019) Interactive
learning for identifying relevant tweets to support real-time sit-
uational awareness. IEEE Trans Vis Comput Graph
26(1):558–568
46. Szegedy C, Liu W, Jia Y et al (2015) Going deeper with con-
volutions. In: Proceedings of the IEEE conference on computer
vision and pattern recognition, pp 1–9
47. Tan M, Le Q (2019) Efficientnet: rethinking model scaling for
convolutional neural networks. In: International conference on
machine learning, PMLR, pp 6105–6114
48. Tripathy JK, Chakkaravarthy SS, Satapathy SC et al (2020)
Albert-based fine-tuning model for cyberbullying analysis. Mul-
timed Syst 2020:1–9
49. Valdez DB, Godmalin RAG (2021) A deep learning approach of
recognizing natural disasters on images using convolutional
neural network and transfer learning. In: Proceedings of the
international conference on artificial intelligence and its appli-
cations, pp 1–7
50. Yu Y, Tang S, Aizawa K et al (2018) Category-based deep cca
for fine-grained venue discovery from multimodal data. IEEE
Trans Neural Netw Learn Syst 30(4):1250–1258
51. Yu Y, Tang S, Raposo F et al (2019) Deep cross-modal corre-
lation learning for audio and lyrics in music retrieval. ACM Trans
Multimed Comput Commun Appl 15(1):1–16
52. Zahra K, Imran M, Ostermann FO (2020) Automatic identifica-
tion of eyewitness messages on twitter during disasters. Inf
Process Manag 57(1):102,107
Publisher’s Note Springer Nature remains neutral with regard to
jurisdictional claims in published maps and institutional affiliations.
Springer Nature or its licensor holds exclusive rights to this article
under a publishing agreement with the author(s) or other rightsh-
older(s); author self-archiving of the accepted manuscript version of
this article is solely governed by the terms of such publishing
agreement and applicable law.
123