Neurocomputing 445 (2021) 121–133

Contents lists available at ScienceDirect

Neurocomputing
journal homepage: www.elsevier.com/locate/neucom

A comparative study of language transformers for video question

answering
Zekun Yang a,⇑, Noa Garcia b, Chenhui Chu b, Mayu Otani c, Yuta Nakashima b, Haruo Takemura a
a
Graduate School of Information Science and Technology, Osaka University, Japan
b
Institute of Datability Science, Osaka University, Japan
c
CyberAgent, Inc., Japan

a r t i c l e i n f o a b s t r a c t

Article history: With the goal of correctly answering questions about images or videos, visual question answering (VQA)
Received 29 June 2020 has quickly developed in recent years. However, current VQA systems mainly focus on answering ques-
Revised 23 November 2020 tions about a single image and face many challenges in answering video-based questions. VQA in video
Accepted 22 February 2021
not only has to understand the evolution between video frames but also requires a certain understanding
Available online 10 March 2021
Communicated by Zidong Wang
of corresponding subtitles. In this paper, we propose a language Transformer-based video question
answering model to encode the complex semantics from video clips. Different from previous models
which represent visual features by recurrent neural networks, our model encodes visual concept
Keywords:
Video question answering
sequences with a pre-trained language Transformer. We investigate the performance of our model using
Language representation four language Transformers over two different datasets. The results demonstrate outstanding improve-
Language transformers ments compared to previous work.
Ó 2021 Elsevier B.V. All rights reserved.

1. Introduction related to this environment; and text VQA [15,37,31], which

A visual question answering (VQA) system aims to answer
questions related to images, which is an interesting but challeng-
ing task for an intelligent system. It usually takes an image and a
related question as input and predicts a correct answer as output
[46,1]. During question answering, semantics in both language
(the question) and visual (the image) content is retrieved and rep-
resented by high-dimensional vectors as visual and language fea-
tures. Attention mechanisms are also adopted to find
corresponding parts in the image that are helpful to answer the
question [46,47,28]. VQA has been quickly developing in recent
years, and many strong attempts have been made [13,3,36,1] to
create better VQA systems.
There are many branches of VQA tasks, including general VQA
[1,13,3], which needs to answer a question related to an image;
video question answering (video-QA) [38,54,18], which addresses
question answering in video sequences; embodied question
answering (embodied-QA) [6,43,51], where an agent is set in a ran-
dom place in 3D environment and need to answer questions
⇑ Corresponding author.
E-mail addresses: yang.zekun@lab.ime.cmc.osaka-u.ac.jp (Z. Yang), noagar-
cia@ids.osaka-u.ac.jp (N. Garcia), chu@ids.osaka-u.ac.jp (C. Chu), otani_mayu@cy-
beragent.co.jp (M. Otani), n-yuta@ids.osaka-u.ac.jp (Y. Nakashima), takemura@ime.
cmc.osaka-u.ac.jp (H. Takemura).
focuses on answering questions related to the texts in the image.
We believe video-QA can be a proxy of embodied-QA and text-
QA tasks because video is a media that has multiple modalities,
i.e., a sequence of frames, audio tracks, and speech contents (or
subtitles). Video-QA systems need to combine these modalities in
videos, and try to understand what is happening in order to find
the correct answer.
Recently, some techniques have been proposed to handle video-
QA tasks. The inputs of video-QA vary by task, e.g. video only
[10,52], video and subtitle [22,17,23], video and audio [29], etc.
Video and subtitles contain rich information and have been widely
applied in many similar tasks [22,17,23], so we take the video
frame sequences, the corresponding subtitles, a question, and can-
didate answers as inputs, as shown in Fig. 1.
Video-QA systems face two main challenges: First, the visual
input of video-QA for each question is a sequence of video frames
instead of a single image, making it necessary for the system to
understand the evolution of the video [56,32]. Second, understand-
ing subtitles is also of importance, because they contain the tran-
scription of what the characters are saying and convey
complementary information that is not shown in video frames
[22,18,11].
In video-QA models, visual and language elements are usually
handled by individual flows [38,22,18]. The visual flow retrieves
and processes the semantics from video frames, and the language

https://doi.org/10.1016/j.neucom.2021.02.092
0925-2312/Ó 2021 Elsevier B.V. All rights reserved.
Z. Yang, N. Garcia, C. Chu et al.
Fig. 1. Overview of our work. We use variants of Transformers to encode the
complex semantics from video clips in video question answering task.
flow obtains and processes the semantics from subtitles in parallel.
Recurrent neural networks (RNNs), especially long short-term
memory (LSTM) networks [14], are used in each flow to encode
those elements. As the inputs of video-QA tasks often contain long
sequences, RNN-based networks may have difficulty grasping the
semantics in them and can thus be a bottleneck to the improve-
ment of accuracy in the video-QA task.
In the natural language processing community, Transformer
[40], a novel architecture that aims at solving sequence-to-
sequence tasks while handling long-range dependencies, has been
proposed and used extensively [2,9,25,7]. Transformers rely on
self-attention to compute representations of the input and output
without using RNNs. All of them have been reported to outperform
RNNs in several natural language processing tasks.
We propose to improve video-QA by capturing the visual and
language semantics from video clips more accurately. Different
from previous models which represent visual features by RNNs,
our model encodes visual elements in the input with a pre-
trained language Transformer [40]. For this, we follow [22] and
adopt a visual concept detector [35] to represent each detected
concept with a corresponding word/phrase. We fine-tune the net-
work in an end-to-end manner (including the pre-trained Trans-
former) for the video-QA task.
One key question about our new video-QA model is which
Transformer should be used for video-QA tasks among the different
pre-trained variants. To explore this, we refer to a reading compre-
hension task called SQuAD [34] because video-QA needs to under-
stand the subtitles to predict the correct answer, which is similar
to reading comprehension. We assume that variants of Transform-
ers that work well for SQuAD also perform well for video-QA tasks.
The SQuAD leaderboard1 shows that BERT [8], XLNet [48], RoBERTa
[26], and ALBERT [21] were the best performers at the time of this
research. Therefore, we select them and compare their performances
in our experiments.
Our contribution is threefold: i) We propose an improved model
for video-QA. Our model uses a pre-trained language Transformer
to encode a sequence of visual concepts besides subtitles. ii) We
evaluate our model on four commonly applied Transformers and
two video-QA datasets, and find that BERT works better than other
variants of Transformers. iii) We give discussions on how different
pre-training strategies and configurations of Transformers affect
the video-QA performance.
1
https://rajpurkar.github.io/SQuAD-explorer/.
122
Neurocomputing 445 (2021) 121–133
The remainder of this paper is arranged as follows: Section 2
introduces related work on video-based question answering and
language representations respectively. Section 3 gives a brief intro-
duction to the Transformers we studied. Section 4 describes our
proposed model in detail. Section 5 presents the experimental
evaluation. Section 6 gives discussions about the results, and Sec-
tion 7 concludes the paper.
2. Related work
2.1. Video-based question answering
In the past few years, some video-QA related work is proposed
to discuss and handle its technical aspects. For example, Wu et al.
[45] present a passage retrieval algorithm to extend the text ques-
tion answering task to videos; Zhao et al. [54] propose the context-
aware question understanding network to deal with video-QA with
several rounds of dialogues; Wu et al. [44] present a cross-
language video-QA model that handles the English questions, and
finds answers in Chinese videos.
Different video-QA tasks may handle the combinations of input
modalities in different ways. For example, in [10,52], video frames
themselves are taken as input to answer related questions. The
tasks in [22,17] take video frames and subtitles as input. Moreover,
video frames and audio are used as inputs in [29] to deal with
questions related to speeches. Our target task involves sequences
of video frames and corresponding subtitles.
Representation of video is also one of the main concerns in
video-QA tasks, and there are a number of techniques related even
only to the input combination of the target task. For instance, Lei
et al. [22], Kim et al. [16], and Zeng et al. [53] mainly use LSTM
to represent the visual (sequences of frames) and language (subti-
tles) features. Lei et al. [23] adopt BERT for language representation
and LSTM for visual representation. Garcia et al. [11] use object
detectors to detect features in video frames and represent the
sequences of frames by bag-of-concept, while they use BERT for
subtitle representation. Owing to its rich capability, we used BERT
to represent both visual and language features in [49].
This work shares a similar idea to Zeng et al. [53] because both
of us are handling visual elements linguistically. Our work is dis-
tinct from Zeng et al. [53] for two reasons: 1) We use a sequence
of words/phrases to represent a sequence of frames, which is much
simpler than the sequences adopted in [53]. 2) Besides taking
visual concept features, we also include the subtitles that contain
the transcription of what the characters are saying, which acts
complementarily to the video frames.
This work is also different from our previous work [49], in
which we used BERT for visual and language representations. We
were not clear about which Transformer performs the best in
video-QA task, and why it performs the best before carrying on this
project. This work not only shows the performance of other three
Transformers in video-QA task, but also empirically shows how dif-
ferent pre-training strategies and configurations of Transformers
affect the video-QA performance.
2.2. Language representation
In question answering models, language representations are
used to convert the words or sentences into vectors for further
steps. Language representation methods handle the language fea-
tures at word level or sentence level and have been quickly devel-
oped in recent years. For example, GloVe [33] is presented to
leverage statistical information at the word level by training only
on non-zero elements in a word-word co-occurrence matrix.
Skip-thoughts vectors [19] generate language representations at
Z. Yang, N. Garcia, C. Chu et al.
sentence level, providing a generic, distributed encoder to recon-
struct the surrounding sentences of an encoded passage. Besides
these methods, many Transformers have been proposed for sen-
tence feature representation. For example, BERT [8] is proposed
to give deep bidirectional representations of unlabeled text; XLNet
[48] enables bidirectional context learning by maximizing the
expected likelihood over all permutations of the factorization
order; RoBERTa [26] is an improved recipe for training BERT mod-
els; and ALBERT [21] lowers the memory consumption and
increases the training speed of BERT. Some other language repre-
sentations [30,4,39,27] have also been proposed, aiming at repre-
senting language features in words or sentences better.
3. Introduction to transformers
Transformers [40] are novel architectures that rely on self-
attention to compute representations of the input and output
without using RNNs. They transform one sequence into another
by reducing the complexity of the algorithm to constant level, so
that longer sequences can be processed more easily. To represent
language features better, many variants of Transformers have been
proposed in recent years.
In Transformers, the complex language features are converted
into vectors by input representations so that they can be handled
easily in computer systems. For a given input word token
sequence, the input representation is the combination of the corre-
sponding token embeddings, segment embeddings, and position
embeddings. Token embeddings denote the embeddings for the
input tokens, segment embeddings denote the sentence that each
token belongs to (e.g. A: the former sentence; B: the latter sen-
tence), and position embeddings denote the position of each token
within the input sequence. In token embeddings, the first token in
every sentence is a [CLS] mark, which is used to obtain the output
in classification tasks. Then, the former sentence follows. The [SEP]
token is added between the former and latter sentences and at the
end of the latter sentence to indicate separation between different
sentences. The latter sentence then goes between two [SEP] tokens.
We investigate the best performing Transformers on the SQuAD
dataset [34] at this moment, which are BERT [8], XLNet [48],
RoBERTa [26], and ALBERT [21]. These Transformers are mainly
distinguished from each other by their pre-training settings. We
use Transformers to model the semantics captured from video
clips, including visual concept features and subtitles. We introduce
each studied Transformer first, and then we evaluate their perfor-
mances via experiments.
3.1. BERT
BERT is a language representation model designed to extract
pre-trained deep bidirectional representations [8]. It uses bidirec-
tional Transformers [40], meaning every word attends to the con-
text of both sides in every layer of the network. Pre-trained BERT
representations can be fine-tuned to achieve state-of-the-art per-
formance in a wide range of tasks [8,25,12].
To obtain a deep bidirectional representation, BERT uses
masked language modeling during its pre-training, which masks
some of the input tokens at random, and then predicts those
masked tokens. To understand the relations between two sen-
tences, a binarized next sentence prediction is also taken in the
pre-training to give predictions of whether a certain sentence is
the next sentence of another sentence.
3.2. XLNet
XLNet is a generalized method that leverages both auto-
regressive language modeling and auto-encoding [48]. It integrates
123
Neurocomputing 445 (2021) 121–133
the segment recurrence and relative encoding scheme of
Transformer-XL [5] into pre-training and improves the perfor-
mance for tasks with longer text sequences. Experiments show
that XLNet also has good performance on a wide range of tasks
[48,7,55].
Instead of masked language modeling, XLNet uses permutation
language modeling during pre-training, which learns to utilize con-
textual features from all positions to capture bidirectional
contexts.
3.3. RoBERTa
RoBERTa is an improved recipe for training BERT models by
measuring the impact of hyper-parameters and data size [26].
Compared with standard BERT, pre-trained RoBERTa model is cre-
ated by longer training with bigger batches over more data.
RoBERTa can match or exceed the performance of BERT and other
similar methods [26].
During the pre-training, dynamic masking is used. Unlike BERT
which performs masking during data pre-processing, RoBERTa
generates the masking pattern when a sequence is fed into the
model.
3.4. ALBERT
ALBERT is a language representation method [21] proposed to
reduce memory consumption and training time. It has fewer
parameters than BERT-large and achieves significantly better per-
formance [21].
ALBERT employs factorized embedding parameterization and
cross-layer parameter sharing to reduce the parameters. The larger
embeddings are decomposed into two smaller matrices and are
projected into a lower dimensional embedding space. Then, the
lower dimensional space is projected to the hidden layers. More-
over, all parameters are shared across layers to improve the effi-
ciencies of parameters.
ALBERT uses sentence order prediction instead of next sentence
prediction in BERT, which focuses on modeling the tie between dif-
ferent sentences. This setup forces the model to learn finer-grained
distinctions about discourse-level coherence properties. Conse-
quently, ALBERT improves the performance for multi-sentence
encoding tasks.
4. The proposed model
We propose a video-QA model to answer multiple choice video
questions, which is illustrated in Fig. 2. In our model, two flows for
visual and language predictions work respectively at first. Then,
their results are summed to obtain the joint answer prediction.
In the visual flow, we represent the visual semantics from each
video frame as a set of objects and attributes that appear in the
scene, which is called visual concept features. In the language flow,
the language semantics is extracted from the subtitles. In each
flow, visual concept features or subtitles are handled along with
the question and each candidate answer with the same variant of
language Transformer but with different instances.
4.1. Visual representations
Recent work [22,50] found that using detected object labels as
input has comparable or better performance than using features
from convolutional neural networks directly in image captioning
and video-QA tasks. Thus, we use visual concept features, which
are the labels of the detected objects using Faster-RCNN [35], to
represent the content of a video scene semantically. Here, the
Z. Yang, N. Garcia, C. Chu et al. Neurocomputing 445 (2021) 121–133

Fig. 2. Proposed model for answering multiple choice video questions. Note that V, Q, and S denote visual concept, question, and subtitle, respectively.

visual concept features contain both objects and attributes, which

are shown in the format of adjective + noun or just noun, such
as man, black hair, and blue shirt. Fig. 3 shows an example of such
features.
We extract visual concept features from each video frame using
Faster-RCNN [35] pre-trained on the Visual Genome dataset [20],
as in [1]. The video frames are extracted at 3 fps. In every extracted
frame, the visual concept features are represented by correspond-
ing nouns or phrases. We aggregate the visual concepts from all the
frames and remove the duplicated ones to obtain unique visual
concept features from a whole scene. Then, the unique visual con-
cept features v , the question q, and each candidate answer item ai
(i = 0, 1, 2, 3, 4) are concatenated and rearranged into a single string
ci . Each rearranged string is tokenized to obtain the sequence T ci .
ci ¼ ½v ; q; ai  ð1Þ
Fig. 3. Example of visual concept features detected using Faster-RCNN.
T ci ¼ tokenizeðci Þ ð2Þ
Here, the concatenation of v and q, [v ; q], is set as the former
sentence for the Transformer, and [ai ] is set as the latter sentence. is tokenized to form the sequence of tokens T wi , which is fed into
When ci is longer than the maximum number of tokens L, the num- the Transformer to obtain V wi . Then, V 0wi is fed into a fully con-
bers of token in the former ([v ; q]) and latter ([ai ]) sentences are nected layer to obtain the language flow representation Rwi for
measured. The last token(s) will be truncated from the longer sen- answer ai , where F w is a trainable parameter.
tence until the number of words in T ci is no more than L.
wi ¼ ½s; q; ai  ð5Þ
Next, T ci is fed into the language Transformers to obtain an out-
put V ci , a matrix containing the vector representation of each word T wi ¼ tokenizeðwi Þ ð6Þ
in the input sentence. The output vector corresponding to the [CLS]
token, V 0ci , is fed into a fully connected layer to obtain the visual V wi ¼ Transformerw ðT wi Þ ð7Þ
flow representation Rci for answer ai , where F c is a trainable
parameter. Rwi ¼ F w V 0wi ð8Þ
V ci ¼ Transformerc ðT ci Þ ð3Þ
4.3. Prediction
Rci ¼ F c V 0ci ð4Þ
Finally, the representations of visual and language flows for
each question are summed to obtain Rpi . Softmax is used to convert
4.2. Language representations
the summed vector into the answer scores Rf :

Similarly, in the language flow, we concatenate the subtitles s, Rpi ¼ Rci þ Rwi ð9Þ
the question q, and the candidate answer items ai (i = 0, 1, 2, 3,
4) to form the string wi . The concatenation of s and q, [s; q], is set Rp ¼ ½Rp0 ; Rp1 ; Rp2 ; Rp3 ; Rp4  ð10Þ
as the former sentence, and [ai ] is set as the latter sentence. The
last token(s) will be truncated from the longer sentence until the Rf ¼ softmaxðRp Þ ð11Þ
number of words in T wi is no more than L. The rearranged string
124
Z. Yang, N. Garcia, C. Chu et al.
The answer with the maximum score is selected as the final
predicted answer ap with
ap ¼ argmaxðRf Þ
5. Experiments
In our experiments, the learning rate is set to 2  105 , the
number of epochs is set to 10, the training batch size is set to 8
and the inferring (i.e. validation and test) batch size is set as 16.
The detailed information about different pre-trained Transformer
models we use is shown in Table 1. L, the maximum number of
tokens per sequence, can be 512 at most. To save memory, L is
set to 128 in our training. Adam optimizer is used with its weight
decay setting as 1  105 . Cross-entropy loss is applied in our
prediction.
Datasets: Two video-QA datasets are adopted in our experi-
ments: the TVQA dataset [22] and the Pororo dataset [18]. The
TVQA dataset is based on six TV shows with 152,500 question–an-
swer pairs (Q/A pairs) from 21,800 clips, while the Pororo dataset is
based on a children’s cartoon video series called Pororo with 8834
Q/A pairs from 171 episodes. In both datasets, each question is in
multiple-choice style with one correct answer out of five candidate
answers. For each question, subtitles corresponding to each video
scene are also provided. To correctly answer these questions,
video-QA systems need to achieve a joint understanding to both
visual and language features. The TVQA dataset provides two
modes for prediction: one is using time stamp annotated elements,
whose inputs are visual and language features corresponding to
the question; another is using full length elements, which takes
all the visual and language features as inputs. The subtitles in each
video clip contain several rounds of dialogues. The Pororo dataset
provides only full-length elements for prediction. The subtitles
are composed of narrations and a few rounds of dialogues. Besides
the videos and subtitles, descriptions about the video scenes are
also given. To compare our results with those of previous models,
we report the results on TVQA [22], STAGE [23], and MDAM [17].
Input Sequence: To test the performance of Transformers, the
input tokens containing visual and language features are rear-
ranged in three different ways:
1Þ V=S þ Q þ A
2Þ V=S þ : þ Q þ A
3Þ V=S þ ½SEP þ Q þ A
where V represents visual concepts, which corresponds to v in Sec-
tion 4, S represents subtitles, which corresponds to s, Q represents
question, which corresponds to q, and A represents answer, which
combines all ai . V/S indicates that both visual concepts and subtitles
are taken in visual and language flows. Ablation studies are con-
ducted by removing either visual concepts (S + Q + A) or subtitles
(V + Q + A). We report the highest accuracies of each Transformer
along with its corresponding word rearrangements.
Table 1
Detailed information about different pre-trained Transformer models we use.
Transformer Model name Layers
BERT bert-base-uncased 12
XLNet xlnet-base-cased 12
RoBERTa roberta-base 12
ALBERT albert-base-v1 12
Neurocomputing 445 (2021) 121–133
5.1. Results on the TVQA dataset
Results with time stamp annotated elements: Results on the
ð12Þ TVQA dataset with time stamp annotated elements are presented
in Table 2. The TVQA dataset has no answer label in the test set,
so we submit the prediction of the test set to the test server. As
the number of total submissions to the test server is limited, we
split 15,253 Q/A pairs from the training set to form a test* set,
while the validation set is kept the same. For comparison, we
report results of the TVQA model [22], which uses a shared LSTM
for both visual and language representations and two unique
LSTMs for joint-modeling, the results from Thomas et al. [42],
which takes the TVQA model’s framework, but uses either shared
LSTM and BERT for comparison and enlarges the output dimension
of the unique LSTM to 600; and STAGE [23], which uses LSTM for
visual and BERT for language representation. We also report some
results on the official test set.
From the results of the test set, when we use V/S + Q + A as
inputs, the accuracies of question answering using Transformers
are all better than the TVQA model, where GloVe + LSTM is used.
They are also better than STAGE, except RoBERTa is adopted. When
we use V + Q + A as inputs, the validation accuracy of RoBERTa is
17.04% lower than TVQA baseline and is 17.40% lower than Tho-
mas et al. using GloVe + LSTM. Among these Transformers, BERT
performs the best in answer predictions. Results show that when
we use both visual and subtitle representations, our model using
BERT obtains an accuracy up to 4.23% higher than that obtained
with TVQA baseline and up to 2.48% higher than that with STAGE.
It is also seen that the results of using V + Q + A is about 20%
lower than using S + Q + A, while the results of using S + Q + A is
close to using V/S + Q + A, we think the reason lies in the bias in
the TVQA dataset. Thomas et al. [42] has proved that the TVQA
dataset tends to concentrate on the information in the subtitles
while suppress the information in the video during training. This
means that a stronger natural language processing framework will
contribute more to the improvement of accuracy. In Table 2, both
LSTM-based and Transformer-based methods are natural language
processing frameworks, making the prediction of subtitle-based
questions better.
BERT is the worst performing Transformer among the four vari-
ants in SQuAD but is the best in our experiments on the TVQA data-
set. To study the reason why BERT performs the best here, we
consider the different pre-training settings in different Transform-
ers, which mainly refers to the language modeling strategy in each
Transformer.
ð13Þ The language modeling strategies in different Transformers
influence how they predict the answer according to visual and lan-
guage features. When modeling language features, Transformers
temporarily cover some words in the input sentence and try to pre-
dict them based on the nearby contexts. In our case, most of
answers are implied in the given context (V/S). When we use BERT,
the masked language modeling masks some words in the sen-
tences randomly and uses the contexts on both sides to predict
that word during pre-training. In XLNet, when the sentence is
short, there are fewer permutations for words, giving only limited
information for the permutation language modeling to predict the
words while keeping its long memory. Our model takes no more
Hidden size Attention heads Parameters
768 12 110 M
768 12 110 M
768 12 125 M
768 12 11 M
125
Z. Yang, N. Garcia, C. Chu et al. Neurocomputing 445 (2021) 121–133

Table 2
Accuracy (in %) of proposed model and baselines on TVQA dataset with time stamp annotations. Note that one only has limited chances to submit their results to the test server
for evaluation; thus, we only show representative results of our model.

Input Name Model Rearrangements Val Test* Test

V+Q+A TVQA [56] GloVe + LSTM – 45.03 – 45.44
Thomas [42] GloVe + LSTM – 45.39 – –
Thomas [42] BERT – 43.44 – –
Ours BERT V+.+Q + A 48.95 49.23 –
Ours XLNet V+.+Q + A 48.17 48.53 –
Ours RoBERTa V+Q+A 27.99 27.66 –
Ours ALBERT V+Q+A 47.01 47.53 –
S+Q+A TVQA [56] GloVe + LSTM – 65.15 – 66.36
Thomas [42] GloVe + LSTM – 66.07 – –
Thomas [42] BERT – 68.30 – –
Ours BERT S+[SEP]+Q + A 70.65 70.22 –
Ours XLNet S+.+Q + A 68.21 68.22 –
Ours RoBERTa S+.+Q + A 69.32 68.92 –
Ours ALBERT S+[SEP]+Q + A 67.53 66.57 –
V/S + Q + A TVQA [56] GloVe + LSTM – 67.70 – 68.48
STAGE [23] LSTM + BERT – 70.50 – 70.23
Ours BERT V/S+.+Q + A 72.41 72.23 72.71
Ours XLNet V/S+[SEP]+Q + A 70.28 70.33 70.64
Ours RoBERTa V/S+.+Q + A 69.09 69.11 68.82
Ours ALBERT V/S+.+Q + A 70.32 69.65 70.35

than 128 tokens in training, which is only a quarter of that in the
experiments with XLNet (512). The information for word predic-
tion becomes limited and the performance is not as good as BERT’s.
When we use RoBERTa, the masking pattern is generated when a
sequence is fed into the model. This means RoBERTa also needs
to fine-tune the masks and fit the context structure while training.
Because SQuAD is a reading comprehension task, the input in
SQuAD is a paragraph with several sentences, the tie between dif-
ferent sentences is strong, and the masks fit the context structure
well. However, the subtitles in our task are mainly dialogues, the
tie between sentences is weaker than between paragraphs, and
the masks cannot fit the context structure as well as SQuAD, mak-
ing the performance of RoBERTa insufficient. In ALBERT, large
embeddings are factorized into smaller ones, which breaks some
word embeddings and cannot lead to a better prediction compared
with BERT.
When we use V + Q + A, the accuracy of RoBERTa is much lower
than that of other Transformers. We believe this is also because of
the dynamic masking in RoBERTa. When we use RoBERTa, it needs
to know how to predict the words in the masks during training. By
using subtitles and questions, dynamic masking can learn from a
certain structure of sentence, so the accuracy of answer prediction
is only slightly lower than that of BERT. However, when we use
visual concept features, which are just word sequences, the
dynamic masking cannot learn from a certain structure of sentence
and gives many wrong predictions.
We also consider the function of next sentence prediction in our
task, as it is designed to find the next sentence (A) according to the
previous sentences (V/S + Q), and hence, might be responsible for
the drop of accuracy in V + Q + A when we use RoBERTa. We find
that only BERT has such a mechanism (XLNet and RoBERTa do
not), while ALBERT has improved next sentence prediction to sen-
tence order prediction. Previous work [48,26,21] has shown that
next sentence prediction is almost ineffective because it lacks the
difficulty compared with language modeling. To check the results,
the performance of XLNet is close to that of BERT, meaning that
next sentence prediction is not the main reason for such a drop.
5.1.1. Results with full-length elements
Besides using time stamp annotated elements, we also want to
test the performance of our model under full-length visual concept
features and subtitles. We use full-length elements (i.e. visual con-
cepts and subtitles without time stamp annotations) instead of the
time stamp annotated elements in the TVQA dataset. The maxi-
mum number of tokens per input, L, is set as 128. Before inputting
the data, it is necessary to do some pre-processing because full
length elements contain many tokens and some of them may be
truncated before being conveyed into our model directly. Thus,
we use an 80-token, 10-step sliding window (i.e. There are at max-
imum 80 tokens in each segment, and it will move by 10 tokens to
obtain the next segment) in the language flow to select the tokens
that best match the question and five candidate answers. To make
our model concentrate on the main components of the question
and answers, we concatenate the questions and five candidate
answers together and remove the stop words from them. The
cosine similarity of the TF-IDF index between the tokens in the
window and question–answer pair is used as a metric for selecting
the input tokens.
To implement this idea, we first generate a vocabulary of
approximately 44,000 words with all the tokens in the TVQA train-
ing set that appear at least 5 times. Then, we segment the tok-
enized subtitles into 80-token segments and compute the cosine
similarity between the TF-IDF representations of each segment
and the question. Finally, we select the segment with the highest
cosine similarity as the input subtitle. The results of each model
are listed in Table 3.
From the table, it can be seen that the best test accuracy of our
model is 0.02% lower than that of the TVQA baseline. There may be
two main reasons for this. First, full-length elements contain too
many words to be covered in the embeddings. Our model embeds
the visual concepts/subtitles, the question, and candidate answers
altogether and truncates the remaining words when the input
sequence is longer than L. However, in [22,23,16], the questions,
answers, subtitles, and visual features are embedded indepen-
dently, enabling the tokenized input to contain more words and
convey more information. Second, our work does not use attention
to find the corresponding part of the visual/subtitle elements
related to the question. In the TVQA model [22], context matching
modules are used to build context-aware vectors, which is helpful
for prediction. In STAGE [23], guided attention is applied to match
the words in questions and answers to the visual concepts and sub-
titles. In PAMN [16], dual memory embedding is adopted to enable
the pinpointing of different temporal parts for each module. In
HERO [24], cross modal Transformer is taken to fuse QA pair with
local textual context, then the QA pairs are fused with global video
context in temporal Transformer module. We would like to explore

126
Z. Yang, N. Garcia, C. Chu et al.
Table 3
Accuracy (in %) of proposed model and baselines using full length elements on TVQA dataset.
Input Name Model
TVQA [56] LSTM
PAMN [16] LSTM
STAGE [23] LSTM + BERT
V/S + Q + A HERO [24] Transformers
Proposed BERT
Proposed XLNet
Proposed RoBERTa
Proposed ALBERT
attention mechanisms to continue improving the prediction in full-
length elements.
5.2. Results on the Pororo dataset
Results on the Pororo dataset are listed in Table 4. For a fair
comparison, we only use the video scenes and subtitles, and we
do not use the video scene descriptions provided. As STAGE only
performs experiments on the TVQA dataset, yet their pre-trained
BERT model is not available for the Pororo dataset, we compare
our model against the MDAM [17] model and TVQA [22] frame-
work (i.e. we use the model provided by TVQA but change the
dataset to Pororo). We obtain an accuracy up to 6.73% higher than
that of MDAM and up to 11.26% higher than that of TVQA. As the
Pororo dataset has no time stamp annotations, we use full length
elements directly. Note that we do not report a comparison with
[18], as it also uses the video scene descriptions to find a correct
answer.
From the results, when we use both visual and language fea-
tures, the accuracies of using Transformers are improved compared
with those using LSTM. Among these Transformers, ALBERT has the
best accuracy on the Pororo test set. In detail, it increases the accu-
racy by 6.73% compared with MDAM [17]. We also find that when
we use RoBERTa, the accuracy of using V + Q + A is lower than using
other Transformer, and the accuracy of using V/S + Q + A drops by
14.13% approximately compared with using S + Q + A in the test
set. We believe the reason for such a drop also lies in the structure
of visual concept features.
We find that the best accuracy of V + Q + A on the Pororo test set
using Transformers is still 0.82% lower than MDAM [17], and the
validation accuracies of BERT, RoBERTa, and ALBERT when using
V/S + Q + A are lower than those using S + Q + A. We think the rea-
son lies in the representation method of visual features.
In MDAM [17], the visual features are directly retrieved as ten-
sors from ResNet-152 pre-trained on the ImageNet set, while in
our proposed model, the visual features are first retrieved as ten-
sors, then classified, and finally represented as words from
Faster-RCNN pre-trained on the Visual Genome set. Because Visual
Genome is a real-world based set, while Pororo is a cartoon based
dataset, there might be some classification errors in our visual con-
cept features, making them not sufficiently accurate to give correct
answer predictions.
To illustrate this clearly, we show some detected features from
TVQA and Pororo frames in Fig. 4, where the features in the top line
are from the TVQA dataset, and those in the bottom line are from
the Pororo dataset. From the figure, it can be seen that even if
the scene contains more features, Faster-RCNN detects features
in the TVQA dataset accurately. In the Pororo dataset, it detects
the photo album pages (bottom left) as window, and the cartoon
figure (bottom right) as yellow toy. From the figure, our Faster-
RCNN cannot retrieve cartoon features as accurately as real-
world features, which should be responsible for the drop in valida-
tion accuracy when using V/S + Q + A.
127
Neurocomputing 445 (2021) 121–133
Rearrangements Val Test
– 64.42 66.46
– – 66.77
– 68.56 69.67
– – 71.25
V/S+.+Q + A 65.07 66.44
V/S+[SEP]+Q + A 61.40 61.68
V/S+.+Q + A 63.86 64.23
V/S+.+Q + A 63.69 63.56
6. Discussions
6.1. Training time and inferring time
We want to compare which model runs faster during the train-
ing and inferring. Thus, we report the training time per epoch and
estimate the inferring time per question by dividing the total time
needed in inferring and the total number of questions to infer. Our
experiments are performed on a computer with a Core i7 8700 K
CPU (3.70 GHz), 32G RAM, and an Nvidia TITAN RTX GPU. We keep
other configurations the same while only change the model we
use. The training time per epoch and the inferring time per ques-
tion with different Transformers on the TVQA dataset are listed
in Table 5.
From the table, we can see that all the Transformer-based mod-
els need more time for training and inferring compared with TVQA
baseline [22]. This means the Transformers is more time-
consuming and memory-consuming. Also, the training time of
using V + Q + A is a bit shorter than S + Q + A, while the inferring
time of using V + Q + A is no shorter than S + Q + A, except XLNet is
used. Usually, the best validation accuracy can be obtained within
3 epochs in both flows in our model and TVQA model.
6.2. Attention weights in different flows
We would like to take a deeper look into the mechanisms of
answer predictions using Transformers. We select BERT, the best-
performed Transformer on the TVQA dataset, and visualize its
attention weights via BERTviz [41]. We pick up a question as an
example and visualize the attention weights of the visual and lan-
guage flows at layer 1, 7 and 12 in Fig. 5 respectively. The answer to
this question can be found in the language flow. Note that the
height of the full attention visualization is very large, so we only
put the focus (i.e. where the attention assembles) here. From the
figure, the attention weights tend to assemble at the [CLS] mark
at layer 1 of both flows, meaning BERT starts by [CLS] for sentence
classification. They tend to assemble at the [SEP] mark in layer 7 of
both flows, implying BERT is trying to find the correspondence of
the former sentence (S/V + Q) and the latter sentence (A). However,
the attention weights tend to assemble in different ways the two
flows in layer 12: they tend to assemble at the punctuation in
the language flow while still at the [SEP] mark at the visual flow.
This implies that BERT have found the correspondence of sentences
and tend to know how the sentence is structured in the language
flow, but in the visual flow, it did not find correspondence, and is
still trying to find it.
6.3. Evaluation with different sequence lengths
We also explore how different sequence lengths influence the
video-QA accuracy. We calculate the maximum number of words
(Max), the minimum number of words (Min), the average number
of words (Avg) and the percentage of sequences with more than
Z. Yang, N. Garcia, C. Chu et al. Neurocomputing 445 (2021) 121–133

Table 4
Accuracy (in %) of proposed model and baselines on Pororo dataset.

Input Name Model (Rearrangements) Val Test

V+Q+A MDAM [17] LSTM – 42.10
TVQA [22] GloVe + LSTM 34.87 33.77
Ours BERT (V + Q + A) 40.75 40.03
Ours XLNet (V + Q + A) 38.14 41.28
Ours RoBERTa (V + Q + A) 31.70 35.25
Ours ALBERT (V+.+Q + A) 37.83 38.93
S+Q+A MDAM [17] LSTM – 42.50
TVQA [22] GloVe + LSTM 37.60 33.90
Ours BERT (S+[SEP]+Q + A) 52.51 54.16
Ours XLNet (S + Q + A) 50.61 52.24
Ours RoBERTa (S + Q + A) 53.07 54.01
Ours ALBERT (S+.+Q + A) 52.04 53.64
V/S + Q + A MDAM [17] LSTM – 48.90
TVQA [22] GloVe + LSTM 37.78 42.53
Ours BERT (V/S+.+Q + A) 52.35 55.26
Ours XLNet (V/S + Q + A) 51.38 54.89
Ours RoBERTa (V/S+[SEP]+Q + A) 37.17 39.88
Ours ALBERT (V/S+.+Q + A) 51.48 55.63

Fig. 4. Examples of detected features in TVQA and Pororo datasets. The top line shows frames from the TVQA dataset, and the bottom line shows frames from the Pororo
dataset.

Table 5 more than 128 words in the language flow. We also find that the
Training time (in h:mm, per epoch) and inferring time (in ms, per question) for percentage of sequences with more than 512 words in both flows
different Transformers on the TVQA dataset.
is less than 1%.
Input Model Training time Inferring time We evaluate our model with three different values of L (128,
TVQA [22] V+Q+A 2:14 5.70 256, and 512) on the TVQA test* set with time stamp annotations
BERT 2:57 21.29 using different Transformers. The results are listed in Table 7.
XLNet 4:13 30.46 From the table, when L increases, the test* accuracies increased
RoBERTa 2:50 18.21
when using BERT, XLNet, and ALBERT. The reason is that L implies
ALBERT 2:42 17.49
TVQA [22] S+Q+A 2:16 5.70 the amount of information conveyed into the network. When we
BERT 2:59 21.36 increase L, the information conveyed into the network increases,
XLNet 4:15 30.46 making the network predicts better. However, when we use
RoBERTa 2:52 17.95 RoBERTa, the accuracy drops by 2.64% when we increase L from
ALBERT 2:45 17.36
TVQA [22] V/S + Q + A 4:15 10.09
128 to 256 and 2.58% when we increase L from 128 to 512. Accord-
BERT 5:27 32.69 ing to the statistics and previous results, about 45.11% of
XLNet 7:45 54.77 sequences have more than 128 words in the visual flow, which is
RoBERTa 5:22 30.01 approximately three times that in the language flow. As RoBERTa
ALBERT 4:55 33.87
is sensitive to word sequence (i.e. the visual semantics), when
we increase L to 256, the increasing length of word sequence
may play an important role in the drop of accuracy. As only
128, 256, and 512 words with time stamp annotations for visual 3.94% of sequences have more than 256 words in the visual flow,
and language sequences of the TVQA test* set, as shown in Table 6. which is slightly higher than the percentage in the language flow
From the table, 45.11% of the sequences have more than 128 (3.30%), when we increase L to 512, the accuracy slightly increases
words in the visual flow, and more than 15% of the sequences have compared with that when L is 256.

128
Z. Yang, N. Garcia, C. Chu et al. Neurocomputing 445 (2021) 121–133

Fig. 5. Attention weights of the BERT model in the visual and language flow. (a) Visual flow, Layer 1, (b) Visual flow, Layer 7, (c) Visual flow, Layer 12, (d) Language flow, Layer
1, (e) Language flow, Layer 7, (f) Language flow, Layer 12. Note that the height of the full attention visualization is very large, so we only put the focus (i.e. where the attention
focuses) here.

Table 6
Statistics of the input sequences on the TVQA test* set.

Max Min Avg >128 >256 >512

Visual flow 527 20 135.73 45.11% 3.94% 0.01%
Language flow 684 18 95.58 15.30% 3.30% 0.75%

Table 7
Accuracy results (in %) on the TVQA test* set using different L values (128, 256, and 512).

BERT XLNet RoBERTa ALBERT

128 72.23 70.33 69.11 69.65
256 72.68 72.45 66.47 70.20
512 72.79 72.98 66.53 70.45

From the results above, even if the accuracy of XLNet is 2.54%
lower than that of BERT when L = 128, when we increase L to
512, the performance of XLNet is slightly higher than that of BERT,
meaning XLNet is more suitable for longer sequences. For RoBERTa,
the performance might be influenced by the visual concept fea-
tures, which are composed of words or phrases instead of a whole
sentence. For ALBERT, the performance is stable regardless of add-
ing the length of word sequence and the change in L.
Overall, we believe that there is not a specific Transformer suit-
able for all kinds of tasks. It depends on detailed requirements to
select the proper language Transformer to help the model perform
better.

129
Z. Yang, N. Garcia, C. Chu et al. Neurocomputing 445 (2021) 121–133

6.4. Qualitative results sets in Fig. 6. The bounding boxes or the words in blue are hints to
the correct predictions, which are annotated by us manually. To
Finally, we show some examples of successful and unsuccessful categorize by dataset, questions (1) to (4) are from the TVQA data-
predictions of different Transformers on the TVQA and Pororo data- set, and questions (5) and (6) are from the Pororo dataset. To cat-

Fig. 6. Successful and unsuccessful predictions on TVQA and Pororo datasets. (1) to (4) are examples from the TVQA dataset, (5) and (6) are examples from the Pororo dataset.
Note that the bounding boxes or words in blue are hints to the answer, which are annotated by us manually.

130
Z. Yang, N. Garcia, C. Chu et al.
egorize by type, questions (1), (3), and (5) are related to the visual
features, while questions (2), (4), and (6) are related to the subti-
tles. Note that the subtitles in (4), (5), and (6) are long, so we cut
off some unrelated parts. As the original subtitles in the Pororo
dataset have no punctuation, we also add some necessary punctu-
ation to make them easy to read.
From the figure, when we use BERT, XLNet, and ALBERT, our
model gets six, five, and four correct predictions, respectively.
When we use RoBERTa, our model only gets two correct predic-
tions. We now discuss each case in detail.
In question (1), the scene is set in a park, and a couple is sitting
on a bench. When using BERT, XLNet, and ALBERT, our model cap-
tures the visual features related to park and bench in the video
scene, so it gives correct predictions; RoBERTa and the TVQA base-
line cannot deal with the video scene correctly and hence give
incorrect predictions. In question (2), the answer can be found
from the subtitles, which explain the difference between comics
and comic books. This question spans approximately 12 s and is
quite challenging. All of the four Transformers give correct predic-
tions according to the subtitles, but the TVQA baseline fails to give
a correct prediction. In question (3), beer bottles are on the desk
when they are talking in the video frame, BERT, XLNet, and
RoBERTa can predict the answer correctly, but ALBERT and the
TVQA baseline give wrong predictions. In question (4), Chandler
says he is wiped, and another person says they should get going
(leave). When BERT, XLNet, and the TVQA baseline are adopted,
they correspond get going and leave and give correct predictions,
while RoBERTa and ALBERT cannot correspond them, thereby gen-
erating incorrect predictions. In question (5), although the subtitle
is saying Hold on. I will come save you, it is difficult to know Who is
to be saved tongtong. When looking at the video frames, we find it
is Tutu. To answer this question correctly, the proposed model
needs to understand the relationship between characters, which
is not considered in our model. We believe BERT, XLNet, and
ALBERT make use of the previous sentence (Tu tu, ugh) to find
the answer. In question (6), the subtitle says they almost won some
luck, so the answer is they are lucky. BERT and ALBERT can connect
these two phrases and predict well, while XLNet and RoBERTa can-
not connect them and hence give wrong predictions.
From these examples, our model is able to solve both visual and
language related questions that cannot be solved by LSTM. The rea-
son for this may be that language Transformers that we adopt use a
self-attention bidirectional structure, making every word attend to
its context on both sides, while in LSTM, the follow-up words in a
long sentence may have a weak attendance to their far previous
words. However, when the answer is not explicit in the video
frames or the subtitles or requires outside knowledge sources,
our model gives bad predictions.
7. Conclusion
In this paper, we proposed an improved model for video-QA
tasks. We used language representations for visual concept fea-
tures and subtitles based on Transformers to capture semantics
in both video scenes and subtitles more accurately. Experiments
were conducted to test the performance of our model by consider-
ing different Transformers, different input arrangements, subtitles
with/without time stamp annotations, and different maximum
lengths with our model. Results showed that BERT performed the
best on the TVQA dataset, which improved the accuracy by
4.23%, while ALBERT performed the best on the Pororo dataset,
which improved the accuracy by 6.73% compared with previous
models. We also found that the accuracy is different when we
use different Transformers on the same dataset. The reason behind
131
Neurocomputing 445 (2021) 121–133
such differences is related to the characteristics of our task and the
different pre-training settings in different Transformers. As future
work, we want to improve the visual feature representations so
that the accuracy can be further improved.
Declaration of Competing Interest
The authors declare that they have no known competing finan-
cial interests or personal relationships that could have appeared to
influence the work reported in this paper.
CRediT authorship contribution statement
Zekun Yang: Methodology, Software, Investigation, Writing -
original draft. Noa Garcia: Investigation, Methodology, Writing -
review & editing. Chenhui Chu: Methodology, Writing - review &
editing. Mayu Otani: Methodology, Writing - review & editing.
Yuta Nakashima: Supervision, Writing - review & editing. Haruo
Takemura: Supervision.
Declaration of Competing Interest
The authors declare that they have no known competing finan-
cial interests or personal relationships that could have appeared
to influence the work reported in this paper.
Acknowledgements
This work was supported by JSPS KAKENHI No. 18H03264,
China Scholarship Council and ACT-I, JST.
References
[1] P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, L. Zhang,
Bottom-up and top-down attention for image captioning and visual question
answering, in: Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, 2018, pp. 6077–6086.
[2] A. Burns, R. Tan, K. Saenko, S. Sclaroff, B.A. Plummer, Language features matter:
effective language representations for vision-language tasks, in: Proceedings of
the IEEE International Conference on Computer Vision, 2019, pp. 7474–7483.
[3] R. Cadene, H. Ben-Younes, M. Cord, N. Thome, Murel: multimodal relational
reasoning for visual question answering, Proc. CVPR (2019) 1989–1998.
[4] A. Conneau, D. Kiela, H. Schwenk, L. Barrault, A. Bordes, Supervised learning of
universal sentence representations from natural language inference data, in:
Proceedings of the 2017 Conference on Empirical Methods in Natural Language
Processing, 2017, pp. 670–680.
[5] Z. Dai, Z. Yang, Y. Yang, J.G. Carbonell, Q. Le, R. Salakhutdinov, Transformer-xl:
attentive language models beyond a fixed-length context, in: Proceedings of
the 57th Annual Meeting of the Association for Computational Linguistics,
2019, pp. 2978–2988.
[6] A. Das, S. Datta, G. Gkioxari, S. Lee, D. Parikh, D. Batra, Embodied question
answering, in: Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition Workshops, 2018, pp. 2054–2063.
[7] P. Dasigi, N.F. Liu, A. Marasovic, N.A. Smith, M. Gardner, Quoref: a reading
comprehension dataset with questions requiring coreferential reasoning, in:
Proceedings of the 2019 Conference on Empirical Methods in Natural Language
Processing and the 9th International Joint Conference on Natural Language
Processing (EMNLP-IJCNLP), 2019, pp. 5927–5934.
[8] J. Devlin, M.W. Chang, K. Lee, K. Toutanova, BERT: pre-training of deep
bidirectional transformers for language understanding, in: Proceedings of the
2019 Conference of the North American Chapter of the Association for
Computational Linguistics: Human Language Technologies, 2019, pp. 4171–
4186.
[9] M.A. Di Gangi, M. Negri, M. Turchi, Adapting transformer to end-to-end spoken
language translation, in: INTERSPEECH 2019, International Speech
Communication Association (ISCA), 2019, pp. 1133–1137..
[10] J. Gao, R. Ge, K. Chen, R. Nevatia, Motion-appearance co-memory networks for
video question answering, in: Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition, 2018, pp. 6576–6585.
[11] N. Garcia, M. Otani, C. Chu, Y. Nakashima, KnowIT VQA: answering knowledge-
based questions about videos, in: Proceedings of the Thirty-Fourth AAAI
Conference on Artificial Intelligence, 2020.
Z. Yang, N. Garcia, C. Chu et al.
[12] L. Gong, D. He, Z. Li, T. Qin, L. Wang, T. Liu, Efficient training of BERT by
progressively stacking, International Conference on Machine Learning (2019)
2337–2346.
[13] Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, D. Parikh, Making the V in VQA
matter: Elevating the role of image understanding in Visual Question
Answering, in: Conference on Computer Vision and Pattern Recognition
(CVPR), 2017..
[14] S. Hochreiter, J. Schmidhuber, Long short-term memory, Neural Comput. 9
(1997) 1735–1780.
[15] R. Hu, A. Singh, T. Darrell, M. Rohrbach, Iterative answer prediction with
pointer-augmented multimodal transformers for textvqa, in: Proceedings of
the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020,
pp. 9992–10002.
[16] J. Kim, M. Ma, K. Kim, S. Kim, C.D. Yoo, Progressive attention memory network
for movie story question answering, in: Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition, 2019, pp. 8337–8346.
[17] K.M. Kim, S.H. Choi, J.H. Kim, B.T. Zhang, Multimodal dual attention memory
for video story question answering, in: Proceedings of the European
Conference on Computer Vision (ECCV), 2018, pp. 673–688.
[18] K.M. Kim, M.O. Heo, S.H. Choi, B.T. Zhang, Deepstory: video story qa by deep
embedded memory networks, in: Proceedings of the 26th International Joint
Conference on Artificial Intelligence, AAAI Press, 2017, pp. 2016–2022.
[19] R. Kiros, Y. Zhu, R.R. Salakhutdinov, R. Zemel, R. Urtasun, A. Torralba, S. Fidler,
Skip-thought vectors, Advances in Neural Information Processing Systems
(2015) 3294–3302.
[20] R. Krishna, K. Hata, F. Ren, L. Fei-Fei, J. Carlos Niebles, Dense-captioning events
in videos, in: Proceedings of the IEEE International Conference on Computer
Vision, 2017, pp. 706–715.
[21] Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, R. Soricut, Albert: a lite bert
for self-supervised learning of language representations, in: International
Conference on Learning Representations, 2019.
[22] J. Lei, L. Yu, M. Bansal, T. Berg, Tvqa: localized, compositional video question
answering, in: Proceedings of the 2018 Conference on Empirical Methods in
Natural Language Processing, 2018, pp. 1369–1379.
[23] J. Lei, L. Yu, T.L. Berg, M. Bansal, Tvqa+: spatio-temporal grounding for video
question answering, 2019. arXiv preprint arXiv:1904.11574..
[24] L. Li, Y.C. Chen, Y. Cheng, Z. Gan, L. Yu, J. Liu, Hero: hierarchical encoder for
video+ language omni-representation pre-training, 2020. arXiv preprint
arXiv:2005.00200..
[25] Z. Li, X. Ding, T. Liu, Story ending prediction by transferable bert, in:
Proceedings of the 28th International Joint Conference on Artificial
Intelligence, AAAI Press, 2019, pp. 1800–1806.
[26] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L.
Zettlemoyer, V. Stoyanov, Roberta: a rbustly optimized bert pretraining
approach, 2019. arXiv preprint arXiv:1907.11692..
[27] J. Lu, D. Batra, D. Parikh, S. Lee, Vilbert: pretraining task-agnostic visiolinguistic
representations for vision-and-language tasks, Adv. Neural Inf. Process. Syst.
(2019) 13–23.
[28] P. Lu, L. Ji, W. Zhang, N. Duan, M. Zhou, J. Wang, R-vqa: learning visual relation
facts with semantic attention for visual question answering, in: Proceedings of
the 24th ACM SIGKDD International Conference on Knowledge Discovery &
Data Mining, ACM, 2018, pp. 1880–1889.
[29] D. Mamgai, S. Brodiya, R. Yadav, M. Dua, An improved automated question
answering system from lecture videos, in: Proceedings of 2nd International
Conference on Communication, Computing and Networking, Springer, 2019,
pp. 653–659.
[30] B. McCann, J. Bradbury, C. Xiong, R. Socher, Learned in translation:
contextualized word vectors, Adv. Neural Inf. Process. Syst. (2017) 6294–6305.
[31] A. Mishra, S. Shekhar, A.K. Singh, A. Chakraborty, Ocr-vqa: visual question
answering by reading text in images, in: 2019 International Conference on
Document Analysis and Recognition (ICDAR), IEEE, 2019, pp. 947–952.
[32] J. Mun, P. Hongsuck Seo, I. Jung, B. Han, Marioqa: answering questions by
watching gameplay videos, Proc. ICCV (2017) 2867–2875.
[33] J. Pennington, R. Socher, C. Manning, Glove: global vectors for word
representation, in: Proceedings of the 2014 Conference on Empirical
Methods in Natural Language Processing, 2014, pp. 1532–1543.
[34] P. Rajpurkar, R. Jia, P. Liang, Know what you don’t know: Unanswerable
questions for squad, in: Proceedings of the 56th Annual Meeting of the
Association for Computational Linguistics (Volume 2: Short Papers), 2018, pp.
784–789..
[35] S. Ren, K. He, R. Girshick, J. Sun, Faster r-cnn: towards real-time object
detection with region proposal networks, Adv. Neural Inf. Process. Syst. (2015)
91–99.
[36] A. Singh, V. Goswami, V. Natarajan, Y. Jiang, X. Chen, M. Shah, M. Rohrbach, D.
Batra, D. Parikh, Pythia-a platform for vision & language research, in: SysML
Workshop, NeurIPS, 2018..
[37] A. Singh, V. Natarajan, M. Shah, Y. Jiang, X. Chen, D. Batra, D. Parikh, M.
Rohrbach, Towards vqa models that can read, in: Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition, 2019, pp. 8317–8326.
[38] M. Tapaswi, Y. Zhu, R. Stiefelhagen, A. Torralba, R. Urtasun, S. Fidler, Movieqa:
understanding stories in movies through question-answering, in: Proceedings
of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp.
4631–4640.
[39] J. Turian, L. Ratinov, Y. Bengio, Word representations: a simple and general
method for semi-supervised learning, in: Proceedings of the 48th Annual
132
Neurocomputing 445 (2021) 121–133
Meeting of the Association for Computational Linguistics, Association for
Computational Linguistics, 2010, pp. 384–394.
[40] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N. Gomez, Ł. Kaiser, I.
Polosukhin, Attention is all you need, Proc. NIPS (2017) 5998–6008.
[41] J. Vig, A multiscale visualization of attention in the transformer model, 2019.
arXiv preprint arXiv:1906.05714 URL:https://arxiv.org/abs/1906.05714..
[42] T. Winterbottom, S. Xiao, A. McLean, N. Al Moubayed, On modality bias in the
tvqa dataset, 2020..
[43] Y. Wu, L. Jiang, Y. Yang, Revisiting embodiedqa: a simple baseline and beyond,
IEEE Trans. Image Process. 29 (2020) 3984–3992.
[44] Y.C. Wu, C.H. Chang, Y.S. Lee, Clvq: cross-language video question/answering
system, in: IEEE Sixth International Symposium on Multimedia Software
Engineering, IEEE, 2014, pp. 294–301.
[45] Y.C. Wu, J.C. Yang, A robust passage retrieval algorithm for video question
answering, IEEE Trans. Circuits Syst. Video Technol. 18 (2008) 1411–1421.
[46] H. Xu, K. Saenko, Ask, attend and answer: exploring question-guided spatial
attention for visual question answering, in: European Conference on Computer
Vision, Springer, 2016, pp. 451–466.
[47] K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhudinov, R. Zemel, Y. Bengio,
Show, attend and tell: neural image caption generation with visual attention,
Proc. ICML (2015) 2048–2057.
[48] Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R.R. Salakhutdinov, Q.V. Le, Xlnet:
generalized autoregressive pretraining for language understanding, Advances
in Neural Information Processing Systems (2019) 5754–5764.
[49] Z. Yang, N. Garcia, C. Chu, M. Otani, Y. Nakashima, H. Takemura, Bert
representations for video question answering, The IEEE Winter Conference on
Applications of Computer Vision (2020) 1556–1565.
[50] X. Yin, V. Ordonez, Obj2text: generating visually descriptive language from
object layouts, in: Proceedings of the 2017 Conference on Empirical Methods
in Natural Language Processing, 2017, pp. 177–187.
[51] L. Yu, X. Chen, G. Gkioxari, M. Bansal, T.L. Berg, D. Batra, Multi-target embodied
question answering, in: Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition, 2019, pp. 6309–6318.
[52] Y. Yu, H. Ko, J. Choi, G. Kim, End-to-end concept word detection for video
captioning, retrieval, and question answering, in: Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition, 2017, pp. 3165–3173.
[53] K.H. Zeng, T.H. Chen, C.Y. Chuang, Y.H. Liao, J.C. Niebles, M. Sun, Leveraging
video descriptions to learn video question answering, in: Thirty-First AAAI
Conference on Artificial Intelligence, 2017.
[54] Z. Zhao, X. Jiang, D. Cai, J. Xiao, X. He, S. Pu, Multi-turn video question
answering via multi-stream hierarchical attention context network, IJCAI
(2018) 3690–3696.
[55] W. Zhong, J. Xu, D. Tang, Z. Xu, N. Duan, M. Zhou, J. Wang, J. Yin, Reasoning over
semantic-level graph for fact checking, 2019. arXiv preprint arXiv:1909.03745..
[56] L. Zhu, Z. Xu, Y. Yang, A.G. Hauptmann, Uncovering the temporal context for
video question answering, Int. J. Comput. Vision 124 (2017) 409–421.
Zekun Yang received his B.S. in Computer Science and
Technology from Xidian University in 2015, and M.S. in
Circuit and System from Lanzhou University in 2018,
respectively. He is now a Ph.D. student at Graduate
School of Information Science and Technology, Osaka
University. His research interests are computer vision
and natural language processing.
Noa Garcia received her B.S. degree in Telecommuni-
cation Engineering from Universitat Politècnica de Cat-
alunya, Spain, in 2012, and her Ph.D degree in Computer
Science from Aston University, United Kingdom, in
2019. She is currently a researcher at Osaka University,
Japan. Her research interests include applications on
high-level visual understanding at the intersection of
computer vision, natural language processing, and
machine learning.
Z. Yang, N. Garcia, C. Chu et al.
Chenhui Chu received his B.S. in Software Engineering
from Chongqing University in 2008, and M.S., and Ph.D.
in Informatics from Kyoto University in 2012 and 2015,
respectively. He is currently a research assistant pro-
fessor at Osaka University. His research interests center
on natural language processing, particularly machine
translation and multimodal machine learning.
Mayu Otani received the B.S. degree from Kyoto
University in 2013, and M.S, and Ph.D in engineering
from Nara Institute of Science and Technology in 2015
and 2018. She is currently an Research Scientist at
CyberAgent, Inc. Her research interests include video
understanding and multimodal machine learning.
133
Neurocomputing 445 (2021) 121–133
Yuta Nakashima received the B.E. and M.E. degrees in
communication engineering and the Ph.D. degree in
engineering from Osaka University, Osaka, Japan, in
2006, 2008, and 2012, respectively. From 2012 to 2016,
he was an Assistant Professor at the Nara Institute of
Science and Technology. He is currently an Associate
Professor at the Institute for Datability Science, Osaka
University. His research interests include computer
vision and machine learning and their applications. His
main research includes video content analysis using
machine learning approaches. He is a member of ACM,
IEICE, and IPSJ.
Haruo Takemura received his B.E. M.E. and Ph.D.
degree from Osaka University in 1982, 1984, and 1987,
respectively. He is currently a Professor at Cybermedia
Center, Osaka University since 2001. His research
interests include Interactive Computer Graphics,
Human–Computer Interaction, and Mixed Reality. He is
a member of IEEE, ACM, IPSJ, and IEICE.