A Machine-Learning Pipeline for Semantic-Aware and
Contexts-Rich Video Description Method
Y. Aun
Faculty of Information and Communication Technology,
Universiti Tunku Abdul Rahman, Malaysia
aunyc@utar.edu.my
M. L. Gan*
Faculty of Information and Communication Technology,
Universiti Tunku Abdul Rahman, Malaysia
ganml@utar.edu.my
ABSTRACT
Video description (VD) methods use machine learning to auto-
matically generate sentences to describe video contents. Global-
description based VD (gVD) methods generates global description
to provide the big picture of video scenes but they lack finer grain
entities information. Meanwhile, modern entity-based VD (eVD) use
deep learning to train ML models like object model (YOLOv3), hu-
man activity model (CNN), location tracking (DeepSORT) to resolve
individual entity that made up the complete sentences. However,
existing eVD are limited in the types of supported entities; thus,
resulting in eVD generating sentences that contexts-deprived and
incomplete to clearly describe video scenes. In addition, the entities
resolved by eVD are isolated since they are inferred from different
ML models; resulting in sentences that are not semantically cohe-
sive; contextually and grammatically. In this paper, a two-stages ML
pipeline (teVD) is proposed for a holistic and semantic-aware VD
sentence generation. Firstly, a ML pipeline is designed to aggregate
several high performing ML models for resolving fine grain entities
to improve the accuracy of resolved entities. Second, the compo-
nents in the entities set are ‘stitched’ together using an entity trim-
ming method to (1) remove shadow entities and (2) to re-arrange
entities based on linguistic rules to generate video descriptions that
are context-aware and less ambiguous. The experimental results
showed that teVD successfully improved the quality of generated
sentences in short videos; achieving BLEU score of 48.01 and ME-
TEOR score of 32.80 on MSVD dataset.
KEYWORDS
Semantic, NLP, auto captioning, CNN, activity recognition
ACM Reference Format:
Y. Aun, J. Y. M. Khaw, M. L. Gan*, and L. T. Tin. 2022. A Machine-Learning
Pipeline for Semantic-Aware and Contexts-Rich Video Description Method.
In 2022 4th International Conference on Pattern Recognition and Intelligent
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full citation
on the first page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or a
fee. Request permissions from permissions@acm.org.
PRIS 2022, July 29–31, 2022, Wuhan, China
© 2022 Association for Computing Machinery.
ACM ISBN 978-1-4503-9608-0/22/07. . . $15.00
https://doi.org/10.1145/3549179.3549182
14
J. Y. M. Khaw
Faculty of Information and Communication Technology,
Universiti Tunku Abdul Rahman, Malaysia
khawym@utar.edu.my
L. T. Tin
Faculty of Information and Communication Technology,
Universiti Tunku Abdul Rahman, Malaysia
avocafe1989@gmail.com
Systems (PRIS 2022), July 29–31, 2022, Wuhan, China. ACM, New York, NY,
USA, 9 pages. https://doi.org/10.1145/3549179.3549182
1 INTRODUCTION
Video captioning or video description (VD) is the automated pro-
cess of textual/natural language sentences generation that describe
the video content. VD is often seen as a composite product of mul-
tiple machine learning models that is used to resolve entities in the
generated sentences. Recently, VD has been gaining momentum
courtesy of the proliferation of deep learning technique and models
that are more accurate, robust and diverse (more classes). For exam-
ple; object and actions in short videos can be accurately classified
using YOLOv3 and any activity models trained with ResNet, VG-
GNet, Inception and GoogleNet respectively. VD takes videos inputs;
pass them through a series of ML models to make predictions and
resolve the entities needed in forming informative sentences that
describes the video content. The outcome of VD is well received
in niche domains like decoding surveillance videos, generate more
relatable YouTube’s description that can attract views and to aid
under-privileged community in ‘seeing’ the videos.
Generally, there are two types of VD; namely global-description
based (gVD) and entity-based (eVD). gVD is useful to ‘describe’
video with multiple subjects, actions and scenes collectively to give
a high-level overview of events. However, for its simplicity; gVD
skims on finer grain details that deprived the sentence from second-
level contexts information. The ‘second-level’ information here
refers to micro-level information; for example, ‘Alex’ in the ‘User’
class, or ‘Sunny’ in the ‘Weather’ class. gVD generalize video de-
scription well, and is capable to ‘caption’ a wider range of scenes;
but the generated sentences can be overly vague and generic. Con-
sider the following two sentences: “A man is walking a dog in the
park”, “A young man (named Bob) wearing a blue dotted t-shirt is
walking a dog in the park happily”. The first sentence is generated
by using most of the current video description method. It is not
wrong, but not informative enough. The second sentence is the
ideal sentence that is generated with rich semantic context. On
the other spectrum, eVD is designed to resolve more contexts to
generate rich sentences. eVD leverages on modern ML architecture
and make predictions using ML models to ‘fill up the slot’ in the sen-
tence’s schema. eVD is useful to paint a scene specific description;
but their accuracy is determined by the choice of ML models used.
However, combining predictions from several ML models means
PRIS 2022, July 29–31, 2022, Wuhan, China
entities are resolved in isolated manner; resulting in sentences that
lack coherency and disjointed.
In this paper, we proposed a semantic-aware video description
method that keep the contexts richness from eVD while improving
the coherency among entities in forming grammatically correct
sentences. A ML pipeline is designed to aggregate state-of-the-
art pretrained ML models to resolve entities in short videos. This
project is motivated by the flexibility of pipelining the models as
components can always be changed with a better one or more can
be added. Having this flexibility allows for semantically richer sen-
tences to be crafted easily instead of training special purpose deep
neural network for video captioning. We added an ‘entity trimming’
method to better model better human-human, human-object and
object-object interaction / relationship to improve the clarity and
structure of the generated sentences. The outcome of this work is
a video captioning method that resolve not only more contexts;
but more useful contexts from short videos for semantic aware
sentence generation. From the experimental results, the proposed
work achieved a BLEU score of 48.01 and METEOR score of 32.80
on MSVD benchmarking dataset.
2 RELATED WORKS
The components in video; in all its complexity can be described
modularly based on individual elements. Video description (DV)
uses applied ML to automatically describe the events based on
video contents. Two types of ML models are the most widely used;
specifically, Object Detection and Recognition and Natural Lan-
guage Processing (NLP). Object detection and recognition has been
a popular field for quite some time. Object recognition is a task to
recognise / identify objects in digital images. There’s a difference
between object detection and object recognition. Object detection
is to locate the object location and enclose it with a bounding box.
Object recognition on the other hand classify detected object. Due
to the popularity of deep learning, currently there are few popular
models for object detection and recognition. For examples, a few
are: Region-Based Convolutional Neural Network (R-CNN), You
Only Look Once (YOLO), Single Shot Detector (SSD). In this paper,
YOLOv3 is used for object detection and recognition [1]. Mean-
while, Natural Language Processing is the application / task for
computer to understand human language. NLP has a wide array
of implementation such as language translation application (e.g.
Google Translate), personal assistant application (e.g. Siri, Cortana).
In video description field, NLP technology can help us better formu-
late generated sentence so that it turns out grammatically correct
and makes sense. Besides that, NLP is also used on entity trimming
applications to generate linguistically correct sentence.
2.1 Human Focused Video Description
In [2], high- level features (HLFs) video frames are initially iden-
tified. Features such as objects, movements (e.g. action, emotion)
and properties (e.g. gender) are defined under HLFs whereby the
main subject is a human. According to the author’s results in the
2007 and 2008 TREC video summarisation task dataset, the follow-
ing categories are identified as human related information: actions,
emotions, age, gender, body parts, grouping, identity and dressing.
15
Yichiet Aun et al.
Among these features, dressing, emotions (e.g. happy, sad, fear-
ful, surprised, angry, disgusted) and gender are most noticeable by
humans. Non HLFs such as organic/inorganic objects, landmarks,
colours, and environment were also considered. Of these features,
location was highlighted as important. Template-based methods
utilizing sentence generation (SVO triplets) with simple NLG were
implemented in [3]. Subsequently the HLFs were extracted from the
video frame, and part-of-speech (POS) tag is created and associated
to the HLFs. Singular, plural, present tense, NN – noun, NNS – noun,
VB – verb, VBZ – verb were among the tags associate. Thereafter
Anon, 2019 proposed the POS tagger from NLTK toolkit. The HLFs
information are extracted frame wise where the samples spans up
to eight frames. The Hidden Markov Model (HMM) was used to
train the activity recognition model. This work provides insights
for the semantics required to understand and form an informational
sentence. The recognized sentences could span for few events in
a single video. However, the approach is event localized and does
not form any correlation with surrounding events. There is also no
context understanding between events.
2.2 Sequence To Sequence – Video To Text
In this paper by [4], the author uses deep learning network for video
description. The work focuses on open-domain video description
which are sensitive to temporal structure. Specifically, Long Short
Term Memory (LSTM) network which is a variant of Recurrent
Neural Network (RNN) was used. Due to the diverse set of objects,
actions, scene and their attributes, it makes it hard to generate
description for open domain videos. Therefore, video clips paired
with sentence description from large datasets are used to feed into
learning network to train the model. The author states that LSTMs
were chosen because of it has a great success on task such as speech
recognition and machine translation, which have a sequence-to-
sequence nature. Therefore, video and language also inherit the
properties above, so it suits well for LSTMs training. Raw frames in
RGB and optical flow format are feed into CNN networks, which
then the features output act as the input for first LSTM layer. The
stacked LSTM in the paper consist of two layers, the first layer for
visual feature inputs, the second for language models (with given
text input) and hidden representation of video sequence from the
first layer (see Figure 1). Each of the LSTM consist of 1000 hidden
units. Raw frames that feed into CNN generates a fixed length vector.
Optical flow images are also used due to flow CNN models proved
to provide good results for activity recognition. The features vectors
then feed into decoder to generate sequence of words. According
to the author, although RNN can be used to decode the vector, it is
believed that the long-term dependencies of video sequence may
lead to an inferior result. Therefore, LSTM were deployed.
2.3 Dense-Captioning Events In Videos
The work by [5] states that they are able to perform video descrip-
tion by using dense video captioning. The method proposed is
claimed to be able to identify all events happen in the video in a
single pass. Therefore, multiple sentence description is generated
describing different events happen at different timespan in a video.
In addition, the model proposed also uses context from other sur-
rounding events to caption each event. It claims that it was able to
A Machine-Learning Pipeline for Semantic-Aware and Contexts-Rich Video Description Method
Figure 1: Overall system design view of S2VT implementa-
tion.
use context from past and future sequence to generate caption for
an event. Also, the model is able to perform on videos as long as 10
minutes. To enable description of multiple event in a long video,
action localization is needed. The model uses action proposal and
social human tracking method to achieve action localization. The
input video frames are U = {Ut } where t ∈ {0, . . . ,T − 1}, repre-
sents frames in temporal order. The output for this model is a set of
sentences si = (tstart , tend , {vj }), where tstart indicates the sentence
start time and tend for end time and vj ∈ V, where vj is a set of
words with different length with vocabulary set V. To start with the
pipeline, the author claims that their model starts with extracting
C3D frame features, then feeding it through action proposal module
which generate a set of proposals: P = {(ti_start , ti_end , scorei , hi )},
where scorei represents the proposal score. Scores higher than the
set threshold will have their hidden representation hi , to be used
as the input for LSTM captioning module. To capture video context
from surrounding events, the author groups all of the events into
two categories relative to current event. The two groups are events
that has already occurred (past) and events that are yet to occur
(future). For a video event with a hidden representation of h , and
a start and end time of [ti_start , ti_end ] exported from the proposal
module, the past and future representation can be calculated by:
past 1 Õ
l t jend < tiend ai j h j
h i
hi = past
z j,i
f utur e 1 recognize the overall / globally active event. The followed entity
l t jend < tiend ai j h j
Õ h i
hi = (2)
z f utur e j,i
where h is the hidden representation of other proposed events. ai j
is the attention used j to determine how relevant event j is to event
i. Z is the normalization. ai j can be calculated as follows:
w i = w a hi + ba
ai j = w i hi
Where w i is the annotation vector, which can be calculated from
learnt weights w a and bias ba . Lastly, the hidden representations
are then concatenated (hi_past , hi , hi_f utur e ) and fed into LSTM to
generate sentences.
PRIS 2022, July 29–31, 2022, Wuhan, China
2.4 Automated Textual Descriptions For A
Wide Range Of Video Events With 48
Human Actions
In [6], the authors demonstrated video textual description accord-
ing to human actions by a hybrid method. The method consists of a
description generator as well as an action classifier. The description
uses rule-based method that infers entities to an action, while the
bag-of-features deduces the actual action on display. Fusion engine,
visual processing, action recognition, description generator and
event description are the core elements of the proposed method.
Initially, a moving object (e.g. human, animal, non-organic matter)
is classified with visual processing. Human classification will have
additional feature description such as body part and pose. Next,
the tracked objects are fused to become entities using the fusion
engine. Low level object features are then extracted with the event
descriptor. Subsequently descriptions based on entity, relationship
or the overall picture is done through the abstract descriptor. The
action classifier is capable to deduce up to 48 human actions. Fi-
nally, a hypothesis is derived by the description generator based
on the data obtained by the event descriptor. Corresponding to
the hypothesis, textual descriptions are produced according to the
actions, objects and entities. This method however is self-limiting
as the results are only able to describe with short, simple verbs
without semantics comprehension. Refer to figure 2
3 MACHINE LEARNING PIPELINE FOR
SEMANTIC-AWARE VIDEO DESCRIPTION
We designed a ML pipeline that aggregates 3 ML models for entity
resolution. The predictions from ML are then fed into sentence
generator to build a semantic aware sentence describing videos
events in fine details. The components in the pipeline are shown in
figure 3
Firstly, the video frames in the video will be extracted and passed
into different components in the pipeline for context extraction.
The object detector based on You-Only-Look-Once v3 (YOLOv3)
model detect the position of human and objects in the frame. At the
same time video frames sampled at 10 frames interval are passed
(1) into scene recognition module for the recognition of the scene in
the video frame. Also, action recognition model is also used to
(mainly detected human) will then further be processed for different
process of feature extraction. Extraction including: face detection,
face / identity recognition, gender detection, age detection, emotion
recognition, clothing colours and pattern detection, crowd counting,
and localized action recognition (details discussed below). After
that, all the captured features are being aggregated and filtered. A
preliminary sentence will then be generated based on a fill-in-the-
blanks method using a predefined template. The sentence will then
(3)
be passed into next stage where language masking model will be
applied to perform entity trimming from all the detected objects
in the scene. Lastly, before forming its shape, the sentence then
passed into grammar checking model to correct any grammar error
and output and display it.
16
PRIS 2022, July 29–31, 2022, Wuhan, China
Figure 2: The processing pipeline of the system (with 48 Human Actions).
Figure 3: Proposed system design.
4 SUPPORTED ENTITIES
The proposed pipeline attempts to resolve object types, human
(identity), scenes and activity that forms the core entities in the gen-
erated sentences. This section discusses the individual ML models
used to make these predictions.
4.1 Videos Frames
Video frames are captured and input using OpenCV function
cv2.VideoCapture(video), where video is the file path to the video file.
The frames are sampled at their original frame rate and resolution.
Frame can be sampled at a lower rate for faster performance. The
inputted frame will consist of three channels which are Red, Green
and Blue (RGB) channels. Frames will be extracted into a temporary
folder as individual images in Joint Photographic Experts Group
(JPEG) format for easier processing on latter stages.
4.2 Object And Human Detection
In this module, You-Only-Look-Once version 3 (YOLOv3) model
was used to perform detection [1]. YOLO is using CNN (specifically
Darknet-53) as its deep neural network for object detection. YOLO
model achieves great performance with good accuracy in detecting
and classifying objects. As said by the authors, YOLOv3 has an accu-
racy on par with Single-Shot Detector (SSD) but is three times faster.
For each frame, output will be the bounding box coordinates for the
detected item, the item label, and their corresponding confidence
level. The confidence level for detection is set at 0.4 to avoid too
many unnecessary detections. As this project is human-oriented,
each detected human details will be saved separately for tracking
and other purposes (discussed below). The details are outputted in
the format: (frame, id, bbox_x, bbox_y, width, height, conf, x, y, z)
17
Yichiet Aun et al.
where, frame indicates detected human frames, id indicates track-
ing id but since tracking haven’t initiated, -1 is put for padding,
bbox_x and bbox_y indicates the top-left corner coordinates of the
bounding box, width and height represent the width and height of
the bounding box, conf indicates the confidence of detected human,
and x, y, z are used for 3D tracking where in this case - 1 are used
for padding. Also, the number of detected humans for each frame
will be recorded and calculated with simple max occurrence to get
a rough figure (simple crowd counting) where how many humans
are included in the scene.
4.3 Scene Recognition
For scene recognition, places365-CNNs is used as the model to
predict scene classes [7]. The model is trained on 10 million scene
photographs with almost 5000 images for each category of 400+.
Modified version of CNN (Places-CNN) was used in the training
process (end-to-end network). After detection, information such as
environment type (indoor / outdoor), scene category, scene attribute
(what are the attribute of the scene that leads to classification)
and activation map can be acquired. To ensure good performance,
frames are sampled in 10 frames interval where each detection
are stored in array. After that, result with the max occurrence are
used as the final result. As output, the environment type (indoor /
outdoor) and scene category were acquired.
4.4 Action Recognition
A series of video frames are processed into RGB frames and optical
flow frames (by applying TV-L1 optical flow algorithm). For most
of the video we acquire nowadays, the aspect ratio is usually at
16:9 compared to 4:3 at the olden days. However, for frame input
A Machine-Learning Pipeline for Semantic-Aware and Contexts-Rich Video Description Method
into action recognition model, frame size must be in 224x224 pixels,
which is 1:1 in ratio. Therefore, video frames are resized and cropped
in the centre with original ratio maintained. It can be safely assumed
that most of the video will have its most important information
focused at the centre of a scene. So, we have high chance that we
won’t lost important information by cropping off the frames. Since
most of the video have ratio of 16:9, the height is always fixed to
244 pixels and the width of the video will be adjusted accordingly.
Finally, the frame will be cropped into 224x224 pixels from the
centre. Cropped frame will be pre-processed before used for action
recognition. To process RGB frames, pixel values are normalized
between -1 and 1 for both all three RGB channels. Each frame will
then append into arrays where ultimately the array will have a size
of (batch_size, frame_count, frame_width, frame_height, channels),
where frame_count represent the total number of frames extracted
from the video frame_width and frame_height represents the frame
size (width, height), which in this case is 224x224 and channels
represent number of frame channels where in this case is 3 for RGB
channels. On the other hand, to process optical flow stream, frames
are converted into grayscale image first. Then optical flow stream is
calculated using TV-L1 optical flow algorithm. The pixels value will
then be truncated between -20 and 20 to eliminate large movement.
After that, pixel values are normalized between -1 and 1 for both
channels. The resulting array will have the same value as RGB array
except for channels which is only 2. Both arrays will then be fed
into Two-Stream Inflated 3D ConvNet (I3D) [8] model for action
recognition. As output, the detected action class will be output
along with its confidence level. The action with highest confidence
level was chosen as the occurring event and this information will
be passed on to the next stage.
4.5 Human And Object Tracking
In this module, DeepSORT [9] model was used to track human loca-
tion. The model will be able to track every different individual even
when occlusions happen in between. The model is improved on top
of Simple Online and Realtime Tracking (SORT) model. It uses the
detection info obtained earlier by the YOLOv3 model. As a result,
all humans will be tracked and with their respective bounding box
coordinates. The results are outputted in the format as followed:
(frame, id, bbox_x, bbox_y, width, height, conf, x, y, z) where, frame
indicates detected human frames, id indicates tracking id of human
(each individuals are associated with an ID), bbox_x and bbox_y
indicates the top-left corner coordinates of the bounding box, width
and height represent the width and height of the bounding box,
conf indicates the confidence of detected human, and x, y, z are
used for 3D tracking where in this case - 1 are used for padding.
For convenience of processing in further stages, the detection info
is saved as NumPy array file format (.npy). Currently, DeepSORT
is only trained on tracking humans’ position. If objects are needed
to be tracked, the model needs to be retrained with the particular
object features.
5 FEATURE EXTRACTION
After acquired each human tracking bounding box, each unique
entity can be segmented / crop out to go through feature extraction
18
PRIS 2022, July 29–31, 2022, Wuhan, China
process. In this paper, we demonstrate the use of applied ML using
face features and clothing.
5.1 Face Feature Extractions
Features extraction methods such as face recognition (identity
recognition) [10], gender and age detection [11], emotion detection
are used to achieve entity smoothing. All the mentioned meth-
ods are using dlib (King, 2019) as a library / toolkit to perform
recognition. The module uses a pre-compiled library by [10] for
ease of development. For face recognition, previously acquired /
saved face encodings will be used for face comparison to compare
with unknown faces. If there are known faces detected (confidence
threshold: 0.4, based on face distance, lower distance have higher
confidence level), the detected label with highest confidence level
will be used to resolve the entity identity. If there are more than
one detected identity along the sequence, label with the highest
frequency is selected as the identity. This is to cater with issue
such as mis-identification due to angle or image resolution issue.
In case of no known face detected, gender and age of the person
will be calculated and resolve the entity into a more meaningful
context. Also, all detected face will go through the pipeline of emo-
tion recognition to recognize current individual facial expression
per-frame-basis. Emotion classes available are: [angry, happy, sad,
neutral, disgust, fear, surprise]. All detection will be recorded and
calculate the max occurrence. To improve the detection success
rate for the features mentioned above, face detection is employed
to detect the exact location and bounding boxes of the faces. In
case of no faces detected, the pipeline carries on without carrying
out feature extraction mentioned above. The detected faces will be
cropped out according to coordinates for feature extractions.
5.2 Clothes / apparel recognition
This module can be separated into two parts: clothes / apparel de-
tection and clothes / apparel attribute recognition. For the first part,
a custom YOLOv3 detector was trained on ModaNet dataset [12] by
[13]. ModaNet contains around 55000 street fashion images with
polygon annotations. The dataset contains 13 meta fashion cate-
gories / classes. Given the images of detected humans, the custom
YOLOv3 detector detects the category of apparel worn by the hu-
man, bounding box coordinates and labels are outputted and saved
in array as a result. In the meantime, detected apparel will have
their frame cropped according to their bounding box and saved
as JPEG images for further processing. On part two, the frames of
detected apparel are input into clothes attribute detector [14] to
generate detections of fashion attributes. The fashion attributes
detector is trained on DeepFashion dataset [15]. The dataset con-
tains over 800000 diverse fashion images. The fashion attribute
detector is trained on a subset of 300000 clothes images with 1000
number of clothing attributes. Each detected attribute is associated
with clothes category in python dictionary manner. Besides that,
clothes base colour is also computed. The cropped image will be
gone through K-Means clustering (grouping the pixels of the frame,
three clusters are set) to get the major colour from the frame (in
RGB format). The colour space in RGB is then converted into Lab
(CIELAB) colour space for easier comparison. Finally colour dis-
tance is computed using delta-E (specifically deltaE_cie76) against
PRIS 2022, July 29–31, 2022, Wuhan, China
predefined colour palette containing 22 distinguish colour. Colour
with the shortest delta-E distance will be chosen as the base colour
of detected apparel. Same as before, each detected colour is saved
in the same python dictionary structure. With all detected clothes
category, attribute and colour saved in a dictionary array, each
clothes category attribute and colour will be calculated with max
occurrence method. Finally, a clean list of clothes categories with its
attribute and colour will be outputted (in format: {clothes_category,
attribute, colour}).
5.3 Sentence generation, entity trimming and
grammar correction
Features / semantic context generated will then be fused and fil-
tered to generate dictionaries describing global scene also localized
human attributes. All the detected and captured context will be
arranged in a fill-in-the-blanks manner based on a pre-defined
template to generate a preliminary text. The generated sentence
contains basic and important context from the detection, it is still
far from perfection. Language masking model is used to perform
entity trimming on the objects / entity detected. The model used is
based on a deep learning model: Bidirectional Encoder Representa-
tions from Transformers (BERT) [16], which is being packaged into
a library called FitBERT [17] for ease of use. BERT is a state-of-the-
art model which is widely used in NLP applications to perform task
such as next sentence prediction, masked language model (predict-
ing masked token) and so on. This round, in this project, language
masking model is used to predict the best and most suitable en-
tity to be insert at any point of the sentence. Most of the crucial
information such as subject, action, scene and so on is provided
in the masked sentence to aid in entity prediction / trimming. For
example, from a list of detected objects: [car, handphone, cat, dog,
glass, ball], a masked sentence of: “A man is talking on a ***mask***
in a football stadium”, where ***mask*** is the masked entity to
be predict. After prediction, ideal sentence would be: “A man is
talking on the handphone in a football stadium”. After performing
entity trimming, the last part of the pipeline is to perform grammar
correction on the sentence to ensure the correctness of the sentence.
The method used is calling the unofficial Application Programming
Interface (API) [18] of a popular grammar checking tool: Gram-
marly [19]. Different from all of the previous module, the API only
supports on JavaScript, therefore, NodeJS was used as an external
link to python main program. The generated sentence is packed and
sent to Grammarly’s server, where the corrected sentence will be
returned as a result. Finally sentences / description for the scene are
generated. This mainly split into two parts: the first one would be
a global / overview sentence describing the scene as a whole (with
less details), the second one would be multiple localized sentence
describing each and every human detected with their attributes
and actions attached. Figures 4, 5, 6 and 7 show some experimental
results.
6 EXPERIMENTAL SETUP
We train and implement the ML pipeline using on Intel i5, 32GB
DDR3 ram and Nvidia 2080Ti. Standard libraries related to com-
puter vision like Python 3.6.9 (language), OpenCV 4.1.0, CUDA
toolkit, TensorFlow, Keras are used. The model is evaluated using
19
Yichiet Aun et al.
Bilingual Evaluation Understudy (BLEU) and Metric for Evalua-
tion of Translation with Explicit Ordering (METEOR) scores. BLEU
score is one of the popular metrics that used to measure the quality
or accurateness of computer-generated text [20]. The metric cal-
culates the overlapping of words ranging from unigrams (single
word) up to n-grams (multiple or n consecutive words), between
the predicted sentence and one or more ground truth / reference
sentences. High-scoring description in BLEU requires generated
sentence matches the length and word of the ground truth. This
project uses BLEU-4 as evaluation metric which scores are calcu-
lated up to 4-grams. BLEU can be calculated as shown in Table 1(a).
In the equation, lr is the length of the referenced corpus and lC is
the length of the candidate sentence, N is the total number of n-
grams used, w n represents the positive weights for each gram, and
pn is the geometric average of the modified n-gram precisions. On
the other hand, METEOR score is a metric that computes how well
a generated sentence aligned to the referenced sentence [21]. ME-
TEOR actually is a metric proposed to curb the weaknesses of BLEU.
What METEOR differences from BLEU is that METEOR uses lexical
database of a language (WordNet by default), and matches word by
semantic matching instead of exact matching. METEOR accounts
for exact word match, synonym matching, stemmed word matching
as well as paraphrase matching. The dataset chosen for evaluation is
Microsoft Research Video Description Corpus (MSVD). The dataset
is one of the popular videos in the wild dataset recording up to 1970
YouTube video clips. The annotations are hand-labelled by humans
(∼40 annotations per clip) which each sample ranges between 10 to
25 seconds displaying one major event. 30 videos would be chosen
by random to perform evaluation with both chosen metrics. The
obtained score would then be compared with past works to evaluate
the performance of the project. The proposed method generates
two types of descriptions, an overview / global sentence describ-
ing the main event and multiple localized sentences describing
individual human sentences. Therefore, calculation for both BLEU
and METEOR scores are shown in Table 1(b). From the equation
above, average global detection score (Averaд) is the summation of
each detection score (nth G scor e ) divided by the number of samples
(N ) evaluated. On the other side, average localize average score
are calculated as shown in Table 1(c). For the first for each video
sample, each generated localized sentence score (lscor e ) is summed
together and divided by the number of sentences generated on
the sample (Nl ) to get the lscor e for the sample. Then the average
localized detection score (Averaдelscor e ) is the summation of each
detection score (nth Lscor e ) divided by the number of samples (N )
evaluated. Finally, an overall score is calculated as in Table 1(d).
7 PERFORMANCE EVALUATION
The proposed model is evaluated against various models developed
over the period of time [22]. Table 2 shows the comparison with
methods / models benchmarking on MSVD dataset. From Table
1, the proposed pipeline achieved BLEU scores etter and surpass
many of the compared models and methods but does not perform
well in METEOR metric. This can be reasoned by a few hypotheses.
The score component is observed in Table 3. Firstly, both BLEU
and METEOR scores are better in global prediction. This suggest
that our model does captures global and main event context fairly
A Machine-Learning Pipeline for Semantic-Aware and Contexts-Rich Video Description Method PRIS 2022, July 29–31, 2022, Wuhan, China

Figure 4: Screenshot of an output from proposed system

Figure 5: Screenshot of an output from proposed system

Figure 6: Screenshot of an output from proposed system.

Figure 7: Screenshot of an output from proposed system

20
PRIS 2022, July 29–31, 2022, Wuhan, China Yichiet Aun et al.

Table 1: The equations for BLUE and METEOR score

(a) (b)
N n=1 n
thG
ÍN
lr AveraдeG scor e = scor e
loдBLU E = min(1 − , 0) + w n loдpn
Í
lc N
n=1
(c) (d)
Í Nl Aver aдe G scor e +Aver aдe L scor e
Lscor e = n=1 n
thl
scor e Overall Score = 2
Nl
n=1 n
ÍN th L
AveraдeLscor e = N
scor e

Table 2: Comparison of benchmarking result with other works on MSVD dataset

Techniques/Models/Methods BLEU Scores METEOR Scores

S2VT - 29.8
h-RNN 49.9 32.6
MM-VDN 37.6 29.0
Glove+Deep Fusion Ensble 42.1 31.4
S2FT - 29.9
HRNE 43.8 33.1
GRU-RCN 43.3 31.6
LSTM-E 45.3 31.0
SCN-LSTM 51.1 33.5
LSTM-TSA 52.8 33.5
TDDF 45.8 33.3
BAE 42.5 32.4
PickNet 46.1 33.1
M3 -IC 52.8 33.3
RecNetlocal 52.3 34.1
TSA-ED 51.7 34.0
GRU-EVE 47.9 35.0
Current Project 48.01 32.80

Table 3: Breakdown of score obtained from each component

Global Prediction Score Localized Prediction Score

BLEU 50.55 45.47
METEOR 34.39 31.30

well. However, when it comes to localized prediction, the scores
plummeted quite some bit. This may be due to that it still poses a
challenge to detect and describe events in a finer level. Also, this
may as well suggest that the reference sentences do not meet the
context anticipation of the inferred sentences. This is due to that the
annotations in the dataset mostly contains limited context (without
apparel, emotions, scene and so on). Besides that, the MSVD datasets
does not only contains annotations describing human actions but
also non-human objects, such as animals and vehicles. Therefore, for
those video / sample with no human involved, the system performs
miserably with score of zero as there are no sentence generated.
In fact, if video scores with non-human scenes are excluded, the
BLEU and METEOR scores are actually quite good with scores of
62.01 and 43.58 respectively.
8 FINDINGS AND CONCLUSION
Currently, available video description method tends to have short
and concise sentence, some have semantics information involved
must most lacks of it. This project aims to introduce a method and
pipeline of video description by incorporating semantic context
which also includes audio information to provide a richer and more
accurate video description. The pipeline aims to resolve entity for
better information conveying using slot filling approach. Up until
this stage, context that can be resolved in this project are: actions,
identity (age, gender, name), emotions, objects, apparel and its at-
tributes and also scene. The score achieved for BLEU and METEOR
are 48.01 and 32.80 respectively. Despite the achievement, there
are a lot of problem and challenges encountered during the de-
velopment of project. The extraction of semantics is affected by
occlusion of entities or objects, clarity of videos, limitation of some

21
A Machine-Learning Pipeline for Semantic-Aware and Contexts-Rich Video Description Method
feature’s extraction methods that will propagate down the caption-
ing pipeline. In conclusion, this paper explores the possibilities of
pipelining models in order to achieve semantic and context rich
detection to generate descriptions for a video. Also, methods such
as language masking, human identity recognition provides ways
to perform entity smoothing and trimming. Going forward, new
modules can be added in to further enhance the accuracy or con-
text richness of the pipeline; such as, dense video auto captioning,
human intention prediction, audio information extraction, speech
and emotion recognition and information extraction from video
subtitles.
REFERENCES
[1] J. Redmon and A. Farhadi, "Yolov3: An Incremental Improvement," in arXiv
preprint, 2018.
[2] M. U. G. Khan, L. Zhang and G. Y., "Human Focused Video Description," in Pro-
ceedings of the IEEE International Conference on Computer Vision Workshops
(ICCV Workshops), 2011.
[3] A. Gatt and E. Reiter, "SimpleNLG: A Realisation Engine for Practical Applica-
tions.," in Proceedings of the 12th European Workshop on Natural Language
Generation, ENLG ’09, Stroudsburg, PA, USA, 2009.
[4] S. Venugopalan, M. Rohrbach, R. Mooney, T. Darrell and K. Saenko, "Sequence to
Sequence Video to Text.," in Proceeding ICCV, 2015.
[5] R. Krishna, K. Hata, F. Ren, L. Fei-Fei and J. C. Niebles, "Dense-captioning Events
in Videos," in arXiv preprint arXiv:1705.00754, 2017.
[6] P. Hanckmann, K. Schutte and G. J. Burghouts, "Automated textual descriptions
for a wide range of video events with 48 human actions," in Proceedings of the
European Conference on Computer Vision Workshops and Demonstrations, 2012.
[7] B. Zhou, A. Lapedriza, A. Khosla, A. .Oliva and A. Torralba, "Places: A 10 million
image database for scene recognition," in IEEE TPAMI, 2017.
[8] J. Carreira and A. Zisserman, "Quo vadis, action recognition? a new model and
the kinetics dataset," in Computer Vision and Pattern Recognition (CVPR), 2017.
22
PRIS 2022, July 29–31, 2022, Wuhan, China
[9] N. Wojke, A. Bewley and D. Paulus, "Simple online and realtime tracking with a
deep association metric," in CoRR abs/1703.07402, 2017.
[10] A. Geitgey, "face-recognition 1.3.0," 20 February 2020. [Online]. Available: https:
//pypi.org/project/face-recognition/. [Accessed 20 November 2020].
[11] G. Levi and T. Hassner, "Age and gender classification using convolutional neural
networks," in IEEE Conference on Computer Vision and Pattern Recognition
Workshop (CVPRW), 2015.
[12] S. Zheng, F. Yang, M. H. Kiapour and R. Piramuthu, "Modanet: A large-scale
street fashion dataset with polygon annotations," in ACM Multimedia, 2018.
[13] Simaiden, "Clothing detection using YOLOv3, RetinaNet, Faster RCNN in
ModaNet and DeepFashion2 dataset," 12 January 2020. [Online]. Available:
https://github.com/simaiden/Clothing-Detection. [Accessed 20 November 2020].
[14] X. Liu, J. Li, J. Wang and Z. Liu, "MMFashion: An Open-Source Toolbox for Visual
Fashion Analysis," arXiv preprint arXiv:2005.08847, 2020.
[15] Z. Liu, P. Luo, S. Qiu, X. Wang and X. Tang, "Deepfashion: Powering robust
clothes recognition and retrieval with rich annotations," in CVPR, 2016.
[16] J. Devlin, M. W. Chang, K. Lee and K. B. Toutanova, "Bert: Pre-training of deep
bidirectional transformers for language under- standing," arXiv preprint arXiv:
1810.04805, 2018.
[17] R. Biljana, S. Aneta, Q. Jenkins, J. and S. Havens, "fitbert 0.9.0," 22 May 2020.
[Online]. Available: https://pypi.org/project/fitbert/. [Accessed 20 November
2020].
[18] Stewartmcgown, "Unofficial Grammarly API Client," 9 September 2019. [On-
line]. Available: https://github.com/stewartmcgown/grammarly-api. [Accessed
20 November 2020].
[19] M. Lytvyn, A. Shevchenko and D. Lider, "Grammarly," 2020. [Online]. Available:
https://www.grammarly.com. [Accessed 20 November 2020].
[20] K. Papineni, S. Roukos, T. Ward and W.-J. Zhu, "BLEU: A method for automatic
evaluation of machine translation," IBM Research Report RC22176 (W0109-022),
2001.
[21] S. Banerjee and A. Lavie, "METEOR: An automatic metric for MT evaluation with
improved correlation with human judgments," in ACL Workshop on Intrinsic and
Extrinsic Evaluation Measures for Machine Translation and/or Sum- marization,
2005.
[22] N. Aafaq, A. Mian, W. Liu, S. Z. Gilani and M. Shah, "Video description: A survey
of methods, datasets and evaluation metrics," arXiv preprint arXiv:1806.00186,
2018.