IMAGE CAPTIONING

MINI PROJECT REPORT

Submitted by

MAGESHKUMAR S (212221060155)
MOKESH M (212221060176)
SANJAY K (212221060240)

in partial fulfillment for the award of the

degree of

BACHELOR OF ENGINEERING

IN

ELECTRONICS AND COMMUNICATION ENGINEERING

SAVEETHA ENGINEERING COLLEGE

(AUTONOMOUS)
AFFILIATED TO ANNA UNIVERSITY THANDALAM,
CHENNAI-600 025
JUNE 2023

i
 SAVEETHA ENGINEERING COLLEGE (AUTONOMOUS),
CHENNAI
ANNA UNIVERSITY, CHENNAI – 600 025

BONAFIDE CERTIFICATE

Certified that this project report “IMAGE CAPTIONING” is the Bonafide

work of “MAGESH KUMAR S (212221060155), MOKESH M

(212221060176), SANJAY K (212221060240)” who carried out the project

work under my/our supervision.

SIGNATURE SIGNATURE

Dr. C. Sheeba Joice, M.E., M.B.A, PhD Dr. N. Sugitha,M.E.,PhD

HEAD OF THE DEPARTMENT SUPERVISOR
ELECTRONICS AND COMMUNICATION Associate Professor
ENGINEERING ELECTRONICS AND
COMMUNICATION
ENGINEERING

Submitted for the project viva-voice examination held on_____________________

INTERNAL EXAMINER EXTERNAL EXAMINER

ii
 MINI PROJECT APPROVAL SHEET

The Mini project sheet “IMAGE CAPTIONING” submitted by

‘MOKESH M (212221060176), SANJAY K (212221060240),
MAGESH KUMAR S (212221060155) ’ is approved for submission,
as partial requirement for the award of the Degree of Bachelor of
Engineering in Electronics and Communication, Anna University
during the academic year 2022- 2023.

Submitted for the University Project Viva Voice examination held on

.

INTERNAL EXAMINER EXTERNAL EXAMINER

iii
 ABSTRACT

The aim for image captioning is to automatically create the description

of an image using any natural language. With the development of deep
learning, means of using it to understand image content and generate
descriptive text. The objective of our project is to learn the concepts
of a CNN and LSTM model and build a working model of Image caption
generator by implementing CNN with LSTM . A convolutional neural
networks (CNN) is employed to extract image features as the coding
layer, the long short-term memory (LSTM)-attend is used to decode
the multilayer dense attention model, and the description text is
generated. . The image will be converted into a multi-feature dataset,
characterizing its distinctive features. The analysis will be carried out
on the popular Flickr8K dataset. The experimental results in the field
of general images validate the model’s good ability to understand
images and generating text. Finally, this paper highlights some open
challenges in the image caption task.

iv
 ACKNOWLEDGMENT

We convey our sincere thanks to Dr.N.M.Veeraiyan - President(SMET)

and Chancellor-SIMATS, Saveetha Amaravathi University, Dr.S.Rajesh,
Director - Saveetha Engineering College and Dr. V. Saveetha Rajesh –
Director, Saveetha Medical College and Hospital for providing us with the
facilities for the completion of our project. We are grateful to our Principal,
Dr.N.Duraipandian M.E,Ph.D., for his continuous support and
encouragement in carrying out our project work. We are deeply indebted to
our beloved Head of the Department, Dr.Sheeba Joice, M.E., M.B.A, Ph.D
Department of Electronics and Communication, for giving us the
opportunity to display our professional skills through this project.

We are greatly thankful to our Project Coordinator, Dr.S. Asha, M.Tech,

Ph.D and our Project Guide Dr.M. Vanitha, M.E., Ph.D for their
valuable guidance and motivation which helped to complete our project on
time.

We thank all our teaching and non- teaching faculty members of the
Department of Electronics and Communication for their passionate
support, for helping us to identify our mistakes and also for the
appreciation they gave us. We heartily thank our library staff and the
management for their extensive support in providing the resources and
information that helped us to complete the project successfully. Also, we
would like to record our deepest gratitude to our parents for their constant
encouragement and support, which motivated us a lot to complete our
project work.
v
 TABLE OF CONTENTS

CHAPTER TITLE PAGE

NO. NO.

ABSTRACT iv

LIST OF FIGURES viii

LIST OF ABBREVIATIONS x

1. INTRODUCTION 1

1.1 OVERVIEW 1
1.2 LSTM 1
1.3 CNN MODEL 2
1.3.1 ARCHTECTURE OF CNN 3
1.4 RNN MODEL 3
1.5 ARCHITECTURE OF LTSM 4
1.6 BACKEND 4

2. LITERATURE SURVEY 5

3. REQUIREMENT ANALYSIS 7

3.1 HARDWARE REQUIREMENT 7

3.1.1 GPU 7
3.1.2 TPU 8
3.2 SOFTWARE REQUIREMENT 8
3.2.1 TENSORFLOW 8
3.2.2 KERAS 8

4. SYSTEM ANALYSIS AND DESIGN 9

4.1 EXISTING SYSTEM 9

4.2 DRAWBACKS OF CAPTIONBOT 9
4.3 PROPOSED SYSTEM 9
vi
 4.4 UML DIAGRAMS 10
4.4.1 USE CASE DIAGRAM 10
4.4.2 ACTIVITY DIAGRAM 11
4.4.3 ARCHITECTURE FOR IMAGE
CAPTIONING 12
5 SYSTEM IMPLEMENTATION 13
5.1 IMPORTING THE PACKAGES 13
5.2 GETTING AND PERFORMING 13
DATA CLEANING
5.2.1 LOAD DOC 13
5.2.2 ALL IMG CAPTION 13
5.2.3 CLEANING TEXT 14
5.2.4 TEXT VOCABULARY 14
5.2.5 SAVE DESCRIPTION 14
5.3 EXTRACTING FEATURE 15
VECTOR FROM ALL IMAGES
5.4 LOADING DATASET FOR 15
TRAINING THE MODEL
5.5 TOKENIZING THE VOCABULARY 15
5.6 CREATE DATA GENERATOR 16

6 PERFORMANCE EVALUATION 17
6.1 TESTING THE MODEL 17
6.2 RESULT 17

7 CONCLUSION AND FUTURE

ENHANCEMENT 18
7.1 CONCLUSION 18
7.2 FUTURE ENHANCEMENT 18

vii
 APPENDIX PAGE NO

APPENDIX 1 SOURCE CODE 19

APPENDIX 2 SCREENSHOT 20

viii
 LIST OF FIGURES

FIGURE NO. NAME PAGE NO.

1.3.1 ARCHITECTURE OF CNN 3

1.3 ARCHITECTURE OF LTSM 4

4.4.1 USE CASE DIAGRAM 10

4.4.2 ACTIVITY DIAGRAM 11

4.4.3 ARCHITECTURE DIAGRAM FOR IMAGE 12

CAPTIONING

6.2 RESULT 17

ix
 LIST OF ABBREVIATIONS

CNN – Convolutional Neural Network

RNN – Recurrent Neural Network

LSTM – Long Short Term Memory

MLP - Multilayer Perceptron

x
 CHAPTER 1

INTRODUCTION

1.3 OVERVIEW
Image captioning automatically generating natural description
according to the content observed in an image, is an important part
of scene understanding, which combines the knowledge of computer
vision and natural language processing.

1.2 LSTM

LSTM stands for Long short term memory, they are a type of RNN
which is well suited for sequence prediction problems. Based on the
previous text, we can predict what the next word will be. LSTM can
carry out relevant information throughout the processing of inputs
and with a forget gate, it discards non-relevant information. LSTM
with a soft attention as the decoder which selectively focuses the
attention over a certain part of an image to predict the next sentences.
LSTM describes from input image, by processing already extracted
feature maps.The LSTM as the language model for the decoder to
decode the vector into a sentence.

ResNet50 has a reliable initialization for object recognition and

allows reducing training time. For any image from the training set,
we get the output vector representation from the last convolution
layer. This vector is fed to the LSTM input. The length of the
description may differ, the model should know where to start and
stop. To do this, we add two tokens < START > and < END >, which

1
are the beginning and end of each sign. The network for generating
the captions will have to capture the words between these
tokens.The LSTM model learns to predict the next word St in the
commentary based on the vector of visual features and the previous
t-1 words. LSTM provides the next state vector ht and next word.
The context vector zt is a concatenation of the feature vector and
one hot vector of word representation.

1.3 CNN MODEL

Convolutional Neural networks are specialized deep neural networks which can
process the data that has input shape like a 2D matrix. Images are easily
represented as a 2D matrix and CNN is very useful in working with images. CNN
is basically used for image classifications and identifying if an image is a bird, a
plane or Superman, etc. Convolutional Neural Network (CNN) layers are used
for feature extraction on input data.

CNN model uses Loopy filter. This filter will detect the features of
the given data and generates a feature map. The advantage of CNN
model is that it is location invariant ie) it can detect the feature even
if it is located in any other location on the feature map. It scans
images from left to right and top to bottom to pull out important
features from the image and combines the feature to classify images.

2
1.3.1 ARCHITECTURE OF CNN

1.4 RNN MODEL

RNNs are used in deep learning and in the development of models

that simulate the activity of neurons in the human brain. They are
especially powerful in use cases in which context is critical to
predicting an outcome. RNN use feedback loops to process a
sequence of data that informs the final output, which can also be a
sequence of data. They use feedback loops allow information to
persist; the effect is often described as memory.

3
1.5 ARCHITECTURE OF LSTM

1.6 BACKEND

For the image caption generator, we will be using the Flickr8K and
Flickr text dataset. Flickr8K contains 8091 images. Flickr text
contains text files and caption of images.

4
 CHAPTER 2
LITERATURE SURVEY

2.1 LITERATURE SURVEY

Image captioning requires recognizing the important objects, there

attributes, and there relationship in an image. It also needs to
generate syntactically and semantically correct sentences. Deep-
learning-based techniques are capable of handling the complexity
and challenges of image captioning. In this survey, we aim to present
a comprehensive review of existing deep-learning-based image
captioning technique. Describing the content of an image is a
fundamental problem in artificial intelligence that connects
computer vision and natural language processing.

Convolutional Neural Networks (CNN) are biologically-inspired

variants of Multi Layered Perceptrons. It is comprised of one or
more convolution layers (often with a subsampling step) and then
followed by one or more fully connected layers as in a standard
multilayer neural network. The architecture of a CNN is designed
to take advantage of the 2D structure of an input image (or other 2D
input such as a speech signal). This is achieved with local
connections and tied weights followed by some form of pooling
which results in translation invariant features. Another benefit of

5
CNNs is that they are easier to train and have many fewer
parameters than fully connected networks with the same number of
hidden units. CNN have been widely used and studied for image
tasks, and are currently state-of-the art for object recognition and
detection.

Recurrent Neural Networks (RNNs) are models that have shown

great promise in many NLP tasks. The concept of RNNs is to make
use of sequential information. In a traditional neural network we
assume that all inputs (and outputs) are independent of each other.
But for many tasks that’s not effective. If you want to predict the
next word in a sentence you have to know which words came before
it. RNNs are called recurrent because they perform the same task
for every element of a sequence, with the output being depended on
the previous computations. Alternatively RNNs can be thought of as
networks that have a “memory” which captures information about
what has been calculated so far.

Our approach is to infer these alignments and use them to learn a generative
model of descriptions. We develop a deep neural network model that infers the
alignment between segments of sentences and the region of the image that they
describe. We introduce a Recurrent Neural Network architecture that takes
aninput image and generates its description in text. Our experiments show that
the generated sentences produce sensible qualitative predictions.

6
 CHAPTER 3

REQUIREMENT ANALYSIS

3.1 HARDWARE REQUIREMENT

The science and methodology behind deep learning have been in

existence for decades. In recent years, however, there has been a
significant acceleration in the utilization of deep learning due to an
increasing abundance of digital data and the involvement of the
powerful hardware.

3.1.1 GPU

A Graphical Processing Unit is a specialized electronic circuit designed to rapidly

manipulate and alter memory to accelerate the creation of images in a frame
buffer intended for output to a display device. These are used in embedded
system, mobile phones, personal computer, workstation and game consoles.
Compared to CPU, the performance of matrix multiplication on Graphics
Processing Unit is significantly better. With GPU computing resources, all the
deep learning tools mentioned achieve much higher speedup when compared to
their CPU-only versions. GPUs have become the platform of choice for training
large, complex Neural Network based systems because of their ability to
accelerate the systems. There is no single software tool that can consistently
outperform others, however, which implies that there exist some opportunities to
further optimize the performance.

7
3.1.2 TPU

Tensor Processing Unit is an AI accelerator application-specific integrated circuit

developed by google specifically for neural network machine learning. The goal
was to run whole inference models in the TPU to reduce I/O between the TPU
and the host CPU.

3.2 SOFTWARE REQUIREMENT

3.2.1 TENSORFLOW

TensorFlow is a free and open-source software library for machine

learning. It is a symbolic math library based in dataflow and

differentiable programming. TensorFlow is designed for remarkable

flexibility, portability, and high efficiency of equipped hardware.

3.2.2 KERAS

Keras is an open-source software library that provides a python interface for

artificial neural networks. It acts an interface for the TensorFlow library. Keras
allows for easy and fast prototyping . Keras runs seamlessly on CPU and GPU. It
was developed with a focus on enabling fast experimentation. The
ImageDataGenerator class provided by the Keras API is nothing but an
implementation of generator function in Python.

8
 CHAPTER 4

SYSTEM ANALYSIS AND DESIGN

4.1 EXISTING SYSTEM

Microsoft have published an image captioning experiment on the

internet “CaptionBot”. The idea is that you upload a photo to the
service, and it tries to automatically generate a caption. It accurately
detects what’s was on the display. The computer vision API
identifies the components of the photo, it mixes that with data from
the Bing image search API, and runs any faces it spots through their
emotion API. This analyses human facial expressions to detect
anger, contempt, fear, happiness, sadness or surprise.

4.2 DRAWBACKS OF CAPTIONBOT

Sometime the output is not accurate. It detects the unwanted feature.

It covers only small percentage of the visual concept. Sometime it
fails to describe the image.

4.3 PROPOSED SYSTEM

Image caption generator is a task that involves computer vision and

natural language processing concepts to recognize the context of an
image and describe them in a natural language like English. The
concepts of a CNN and LSTM model and build a working model of
Image caption generator by implementing CNN with LSTM.

9
4.4 UML DIAGRAMS

4.4.1 USE CASE DIAGRAM

10
4.4.2 ACTIVITY DIAGRAM

11
4.4.3 ARCHITECTURE DIAGRAM FOR IMAGE

CAPTIONING

12
 CHAPTER 5

SYSTEM IMPLEMENTATION

5.1 IMPORTING THE PACKAGES

Here we need to import all the necessary packages. The packages

are numpy, string, image, dump, xception, os, keras, embedding,
LSTM.

5.2 GETTING AND PERFORMING DATA CLEANING

The main text file which contains all image captions is

Flickr8k.token in our Flickr_8k_text folder. The format of our
file is image and caption separated by a new line (“\n”). Each
image has 5 captions and we can see that #(0 to 5)number is
assigned for each caption. Here we will define 5 functions
Load doc , All img caption , Cleaning text , Text vocabulary ,
Save description.

5.2.1 LOAD DOC

For loading the document file and reading the contents inside the file
into a string.

5.2.2 ALL IMG CAPTION

This function will create a descriptions dictionary that maps images

with a list of 5 captions.

13
5.2.3 CLEANING TEXT

This function takes all descriptions and performs data cleaning. This
is an important step when we work with textual data, according to
our goal, we decide what type of cleaning we want to perform on

the text. In our case, we will be removing punctuations, converting

all text to lowercase and removing words that contain numbers.

5.2.4 TEXT VOCABULARY

This is a simple function that will separate all the unique words and
create the vocabulary from all the descriptions.

5.2.5 SAVE DESCRIPTION

This function will create a list of all the descriptions that have been
pre-processed and store them into a file. We will create a

descriptions.txt file to store all the captions.

5.3 EXTRACTING THE FEATURE VECTOR FROM ALL

IMAGES
This technique is also called transfer learning, we don’t have to do
everything on our own, we use the pre-trained model that have been
already trained on large datasets and extract the features from these
models and use them for our tasks. We are using the Xception model
which has been trained on imagenet dataset that had 1000

14
different classes to classify. We can directly import this model from
the keras.applications . Make sure you are connected to the internet
as the weights get automatically downloaded. Since the Xception
model was originally built for imagenet, we will do little changes
for integrating with our model. One thing to notice is that the
Xception model takes 299*299*3 image size as input. We will
remove the last classification layer and get the 2048 feature vector.
model = Xception( include_top=False, pooling=’avg’ ) .The
function extract_features() will extract features for all images and
we will map image names with their respective feature array. Then
we will dump the features dictionary into a “features.p” pickle file.

5.4 LOADING DATASET FOR TRAINING THE MODEL

In Flickr8ktest folder, we have Flickr8k.trainImages.txt file that

contains a list of 6000 image names that we will use for training.For
loading the training dataset, we need more functions:

• Load photos– This will load the text file in a string and will
return the list of image names.

• Load clean descriptions – This function will create a dictionary

that contains captions for each photo from the list of photos. We
also append the <start> and <end> identifier for each caption.
We need this so that our LSTM model can identify the starting
and ending of the caption.

• Load features – This function will give us the dictionary for

image names and their feature vector which we have previously
extracted from the Xception model.
15
5.5 TOKENIZING THE VOCABULARY

Computers don’t understand English words, for computers, we will

have to represent them with numbers. So, we will map each word of
the vocabulary with a unique index value. Keras library provides us
with the tokenizer function that we will use to create tokens from
our vocabulary and save them to a “tokenizer.p” pickle file.

5.6 CREATE DATA GENERATOR

Here we need to train our model on 6000 images and each image
will contain 2048 length feature vector and caption is also
represented as numbers. This amount of data for 6000 images is not
possible to hold into memory so we will be using a generator method
that will yield batches.

X1(FEATURE) X2(TEXT) Y(WORD TO

PREDICT)
FEATURE START TWO
FEATURE START,TWO DOGS
FEATURE START,TWO,DOGS DRINK
FEATURE START,TWO,DOGS,DRINKS WATER
FEATURE END
START,TWO,DOGS,DRINKS,WATER

16
 CHAPTER 6

PERFORMANCE EVALUATION

6.1 TESTING THE MODEL

The model has been trained, now, we will make a separate file
testing_caption_generator.py which will load the model and
generate predictions. The predictions contain the max length of
index values so we will use the same tokenizer.p pickle file to get
the words from their index values. we encode all the test images and
save them in the file “encoded_test_images.pkl”.

6.2 RESULT

The above project was done and description for image came
successfully.

17
 CHAPTER 7

CONCLUSION AND FUTURE ENHANCEMENT

7.1 CONCLUSION

We have implemented a CNN-RNN model by building an image

caption generator. Some key points to note are that our model
depends on the data, so, it cannot predict the words that are out of its
vocabulary. This model is capable to autonomously view an image
and generate a reasonable description in natural language with
reasonable accuracy and naturalness. There is still work going on
for alternating Pre-Trained Photo Models to improve the feature
extraction of the model. And also planning to improve and achieve
better performance by using word vectors on a much larger corpus
of data.

7.2 FUTURE ENHANCEMENT

Future work is to develop segmentation-based visual explanation methods

and to compare them with state-of-the-art approaches like Grad-Cam.
There is still work going on for re- ranking.

18
 APPENDIX 1

SOURCE CODE

from transformers import VisionEncoderDecoderModel, ViTFeatureExtractor,

AutoTokenizer
import torch
from PIL import Image
model = VisionEncoderDecoderModel.from_pretrained("nlpconnect/vit-gpt2-
image-captioning")
feature_extractor = ViTFeatureExtractor.from_pretrained("nlpconnect/vit-gpt2-
image-captioning")
tokenizer = AutoTokenizer.from_pretrained("nlpconnect/vit-gpt2-image-
captioning")
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)
max_length = 16
num_beams = 4
gen_kwargs = {"max_length": max_length, "num_beams": num_beams}
def predict_step(image_paths):
images = []
for image_path in image_paths:
i_image = Image.open(image_path)
if i_image.mode != "RGB":
i_image = i_image.convert(mode="RGB")
images.append(i_image)
pixel_values=feature_extractor(images=images,
return_tensors="pt").pixel_values
pixel_values = pixel_values.to(device)
output_ids = model.generate(pixel_values, **gen_kwargs)
preds = tokenizer.batch_decode(output_ids, skip_special_tokens=True)
preds = [pred.strip() for pred in preds]
return preds
predict_step(['sample2.jpg']) # ['a woman in a hospital bed with a woman in a
hospital bed']

19
 APPENDIX 2
SCREENSHOT

20
REFERENCES

1. Anderson P, Fernando B, Johnson M, Gould S (2016) Spice:

semantic propositional image caption evaluation

2. Wu Y, et al. (2016) Google’s neural machine translation system:

Bridging the gap between human and machine translation.

3. Vinyals O, Toshev A, Bengio S, Erhan D (2014) Show and tell :

neural image caption generator.

4. Schmidhuber J (2015) Deep learning in neural networks: An

overview of Neural Networks.

5. Hodosh M, Young P, Hockenmaier J (2013) Framing image

description.

6. Biswas R (2019) Diverse image caption generation and

automated human judgement through active learning.

7. Dudley JJ, Kristensson PO (2018) A review of user interface

design for interactive machine learning.

8. Gunning D, Aha D (2019) DARPA’s explainable artificial

intelligence (XAI) program.
21