See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/357977459

Gastrointestinal image classification based on VGG16 and transfer learning

Conference Paper · December 2021

DOI: 10.1109/ICISAT54145.2021.9678481

CITATIONS READS
0 256

3 authors:

Amina Benkessirat Nadjia Benblidia

Saad Dahlab University Blida1 University
3 PUBLICATIONS   2 CITATIONS    61 PUBLICATIONS   171 CITATIONS   

SEE PROFILE SEE PROFILE

Azeddine Beghdadi
Université Sorbonne Paris Nord
340 PUBLICATIONS   3,612 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Image Compression oriented restoration View project

marketing View project

All content following this page was uploaded by Amina Benkessirat on 17 February 2022.

The user has requested enhancement of the downloaded file.

 Gastrointestinal image classification based on
VGG16 and transfer learning
Benkessirat Amina Benblidia Nadjia
LRDSI Laboratory LRDSI Laboratory
Blida 1 university Blida 1 university
Blida, Algeria Blida, Algeria
aminabks-gsi@outlook.com benblidia@yahoo.com
Abstract—Investigational procedures and medical diagnosis
can be greatly improved by opting for detecting automatically
abnormalities and anatomical landmarks in medical images.
However, this remains a challenging task and still unexplored
field. This paper aims to investigate the capabilities of a pre-
trained deep convolutional neural network VGG-16 model for
images categorization with transfer learning containing anatom-
ical landmarks, pathological finding and endoscopic procedures.
Data augmentation is also performed to highlight the importance
of data size for deep models. The accuracies achieved before and
after data augmentation are 96.9% and 98.8% respectively.
Index Terms—VGG-16, Transfer Learning, classification, cat-
egorization, data augmentation, endoscopy
I. I NTRODUCTION
When healthy cells in the lining of the gastrointestinal (GI)
tract grow out of the normal control; something abnormal
happens such as rectal bleeding, inflammation or mass for-
mation [1]. When abnormalities are detected early, cure is
often possible [2]. Therefore, to save the patient life, we need
to find the affected cells in early stage. Every year, roughly
2.8 million new cases of stomach, colorectal, and esophageal
cancer are diagnosed, with 1.8 million deaths [3]. The gold
standard for GI tract investigation is endoscopic examination,
requiring both high-tech equipment and highly skilled staff.
The interior of the GI tract is examined in real time video
using high quality digital endoscopes in this process, and all
normal and abnormal finding must be reported [3], what could
be time consuming task for the physician who is supposed
to visually analyse all the sequences. To low significantly
this cost of investigational procedures, machine learning (ML)
algorithms could provide an automatic classification system.
To low significantly this cost of investigational procedures,
machine learning (ML) algorithms could provide an automatic
classification system.
Tremendous efforts have been put into the field of medical
image classification. Even if the images are acquired differ-
ently, which gives different image nature. Furthermore, differ-
ent types of deterioration, such as blur, noise, and contrast
flaws, influence them [4]. Since traditional medical image
classification methods are based on visual features as well
as their combination [5], such as colour shape and texture,
978-1-6654-7824-3/21/$31.00 ©2021 IEEE
Beghdadi Azeddine
L2TI laboratory, Institute of Galilee
Sorbonne Paris North University
Paris, France
azeddine.beghdadi@univ-paris13.fr
classification performances are strongly affected by intrusive
visual elements. Furthermore, visual features fall into low-
level feature space, which lack the ability to represent high-
level problem domain concepts, and are rather poor in their
ability to generalize. Consequently, traditional medical image
classification methods have been unable to meet the needs of
real medical application and to deal with the medical images
problems. Deep learning (DL) techniques are now used to
solve difficult task and to improve the performance of existing
traditional techniques; and makes it possible for computers to
learn extremely complicated mathematical models for data rep-
resentation. The computational model is a classical Artificial
Neural Network (ANN) [6], [7] with a lot of hidden layers of
neurons, the huge number of layers inspires the name DL [6].
DL techniques are characterized by a strong generalization
ability, when DL model is trained for a particular task, he
will be able to perform accurately this task, using a variety
of testing data [6]. Convolutional neural networks (CNNs) are
the most popular image processing models to date [7]. The
dataset used in our work (Kvasir) became accessible in the
fall of 2017 as part of MediaEval’s Medical Multimedia Chal-
lenge, a benchmarking project that assigns challenges to the
scientific community [3]. The main contribution of this study is
maximizing the performance of MediaEval challenge methods,
by implementing a transfer learning CNN for the automated
classification of each Kvasir classes. The advantage of our
solution is achieving high performance without requiring hand-
crafted attribute extraction. Also, to to maximize accuracy and
avoid overfitting, emphasis is given to data augmentation. The
remainder of this paper is organized as follows. The proposed
solution is presented in Section 3 after a review of relevant
work in Section 2. Section 4 describes the experimental results
on the Kvasir dataset, and section 5 closes with a broad
conclusion as well as research recommendations.
II. R ELATED WORK
As part of the MediaEval’s challenge, researchers devel-
oped efficient systems, in terms of categorization rate and
execution time; this makes their systems reliable. Agrawal
et al [8] performed 3 classification models. Based on the
baseline features provided with the dataset and extracted
features by Inception-v3, and VGGNet. The first approach uses
the baseline features and extracted features by Inception-V3.
The second approach uses the baseline features and extracted
features by VGGNet. The third one combines the features
extracted by both networks as well as the baseline features.
In all three cases the classification was carried out using
SVM model, and they reached a classification accuracy of
95.9%, 95.3% and 96.1% respectively. The disadvantage is
that the feature extraction step is necessary and additional step
compared to the DL techniques. Pogorelov et al. [9] performed
3 different classification approaches. Classification based on
global features (GF). Classification using CNN. Classification
using transfer learning in DL (TFL). In the first approach,
the GF were extracted from Lire open source software. The
classification was performed by Random Forrest (RF) and
Logistic Model Tree (LMT) from Weka. Two different variants
were carried out: one uses all extracted GF, and the second
uses only two GF. Secondly, two different models built from
scratch. The first with 3 layers and the second with 6 convo-
lution layers. The activation function used is ReLU for the
convolutional layer and Maxpooling for the pooling layer.
In all layers, a dropout of 0.5 was included. The last two
dense layers for classification were made with the activation
functions ReLU and sigmoid respectively. The two networks
were trained for 200 epochs using Adam optimizer. In the
third approach is based on transfer learning by re-training and
fine-tuning the pre-trained Inception v3 model. They reached
a classification accuracy of 96.4%, 95.9% and 95.7% for the
three approach respectively. Naqvi et al. [10] used six baseline
provided features in addition to two extracted textures features.
The model was tested carrying out 10-fold cross validation.
It has been noticed that some features perform very poorly.
Hence they have been removed. Using logistic regression and
discriminant kernel analysis, they formed a separate model
for each feature. The overall technique is then applied for
predictions. They then used K-means clustering to come up
with a reduced dataset representing the entire distribution.
They reached a classification accuracy of 94.2%. The requiring
of features extractions step is considered as disadvantage.
Petscharnig et al. [11] proposed a CNN architecture based on
GoogLeNet and already existing CNN models. The prominent
architectural feature of their proposal is the inception module.
They reached a classification accuracy of 93.9%. Liu et al.
[12] developed a two stages based model. The first to learn
data representation, it is based on bidirectional marginal Fisher
analysis (BMFA). The second based on SVM for images cat-
egorization. They reached a classification accuracy of 92.6%.
Apart from the methods in the context of the challenge, Cogan
et al. [13] proposes an automatic preprocessing framwork to
filter, applied edge removal, color mapping and scaling and
contrast enhancement, for each image in the dataset. They
perform data augmentation before training different predefined
neural networks architectures. Using Inception-v4, Inception-
ResNet-v2, and NASNet the obtained accuracies were 98.45%,
98.48%, and 97.35%, respectively. Theire work seems orig-
inal among existing literature, incorporing a preprocessing
framwork. However, th contribution of their proposal is not
justified, since they haven’t cited the accuracies result before
performing preprocessing.
All these methods have shown their effectiveness and have
achieved acceptable classification rates. In the following, we
present a flexible model that outperforms the MediaEval
challenge methods.
III. CNN M ODEL AND A RCHITECTURE
Neural network, particularly Convolutional Neural Network
(CNN) are one of the most important technologies for digital
image analysis. with all its modules, including image classi-
fication. CNN is modelled typically as the mammalian visual
cortex, consequently it is successfully applied to vision recog-
nition tasks [6], [7]. Classifying WCE images is a challenging
task, facing several factors produce negative changes in the
images, such as specular reflections (light saturation), uneven
distribution of pixels (vignetting), blurred areas and dark areas
[14], [15]. CNN have the ability to learn features directly from
the input image dataset. In what follows we define the general
structure of a CNN.
A. Convolutional Neural Network Model
CNNs are similar to ANNs in that they are constituted
of neuron layers that optimise automatically via learning. It
has been shown that CNNs are strongly discriminative for
real-world pattern recognition, and they have the ability to
learn both the global and local structures of images [14], [15].
They are one of the strongest learning algorithms to recognize
image information and have shown outstanding success in the
segmentation, classification, detection and retrieval of relevant
tasks [15]. Similar to standard ANNs, CNNs are multilayer
networks, except that these layers are not simple perceptrons
[6] [16]. The CNN layers fall into four categories [7] [6] [16]
[17]: convolutional layers, activation layers, pooling layers,
and fully connected layers.
1) Convolutional layers: The convolution operations are
done with the so called convolutional kernels (or filters) [6].
Useful features extraction is performed by convolutional op-
erations, from locally correlated data points, while preserving
spatial relationship between pixels [15]. The convolution result
serves as input to the activation layer. 1 demonstrates the
structure of a CNN.
2) Activation layers: They consist of a non-linear pro-
cessing unit, called the activation function [15]. This non-
linear unit integrates non-linearity in the feature space and
contributes to the abstractions learning, and thus enables the
learning the differences in the images, at the semantic level.
The activation result serves as input to the pooling layer.
3) Activation layers: They consist of a non-linear pro-
cessing unit, called the activation function [15]. This non-
linear unit integrates non-linearity in the feature space and
contributes to the abstractions learning, and thus enables the
learning the differences in the images, at the semantic level.
The activation result serves as input to the pooling layer.
 4) Pooling layers: This layer performs a subsampling, to
summarize the results and to makes the input invariant to
geometrical distortions [16] [15]. The principle is to calculate
one output value v for a nxn grid from the activation map,
where v is the maximum (max-pooling) or average (average-
pooling) value of that grid in the activation map [6]. The
popular VGG-16 network considers the pooling layer as part
of convolutional layer.
5) Fully connected layers: Usually, used at the end of CNN
dedicated to perform classification [15]. This layer serves to
represent compactly the input signal (e.g images) [6].
6) Other layers: There exist other regulatory layers than
the mapping layers mentioned above(batch normalization,
dropout...).
These layers are incorporated to optimize CNN performance
[6] [15]. Batch normalization is incorporated to address the
problems due to the internal covariance shift (a change in
the distribution of hidden units’ values) within feature-maps
[6] [15]. Dropout is incorporated to address the overfitting
problem, by introducing regularization within the network, to
improve generalization by randomly skipping some units or
connections with a certain probability. Figure 1 demonstrates
the structure of a CNN model.
Fig. 1. Structure of a CNN
B. Transfer learning in CNN
To train a deep CNN, huge labelled training data is required,
to ensure a high classification performance. Nevertheless,
medical images acquisition and annotation is a challenging
task. When such difficulties arise, the usage of CNN attributes
such as VGG-16 [18], via transfer learning has been proven
to be efficient for image analysis [18]. The usage of CNN
attributes such as VGG-16 [18], via transfer learning has been
proven to be efficient for image analysis [18]. VGG-16 is
a pre-trained networks on ImageNet dataset [18], and has
a remarkable feature extraction’s capability which involves
a high image classification rate. Figure 2 demonstrates the
structure of VGG-16 model.
All hidden layers are equipped with the Rectified Linear
Unit (ReLU) function, which computes the function f (x) =
max(x, 0). As mentioned above the first convolutional layer
receives an image of size 224x224. The convolutional layers
use a stride of 1 pixel. The max pooling layers use a stride
of 2 pixels (for spatial pooling). Three fully connected layers
follow the set of convolutional layers. Their channel sizes are
Fig. 2. Structure of VGG-16 model
4096, 4096 and 1000 respectively. Finally, a softmax layer to
perform classification. VGG-16 is used efficiently, because of
its generalization ability of other dataset [18].
C. Proposed Architecture
Only the last three layers of VGG-16 were fine-tuned in our
effort. This decision has been motivated by the observations
below [19]:
• The first layers of a pre-trained CNN contain information
about edge and color.
• The later layers offer information about the classes’
specifics.
Learning of VGG-16 has been done on more than 1 million
images, and has been destined to categorize them into 1000
classes [18]. The last three VGG-16’s layers perform classifi-
cation, considering these 1000 classes. consequently, these are
the layers that need to be fine-tuned for a new categorization
task [19] [20]. So the concept is to take all the layers as they
are, except the three last layers. Then, configure three new
layers, namely fully connected layer, a softmax layer, and a
classification output layer, to perform a new classification task.
The size of the fully connected layer must be similar to the
number of class to be distinguished [20]. The size value in
our current work is 8, which corresponds to the number of
classes, as described in the next section.
In the training step, we used the RMSprop optimizer, the
momentum was 0.9n the initial learning rate was 1e-1 and the
end learning rate was 1e-4. This to achieve the best accuracy
and to minimise the binary cross entropy loss function, which
is given by E:
M
1 X
E=− [ym ×log(hô (xm )+(1−ym )×log(1−hô (xm ))]
M m=1
(1)
where, M is the number of training instances, ym is the target
label for training instance m, xm is the input for training
instance m, hô is the model with neural network weights ô.
In a second experiments serie, to alleviate the overfitting
problem caused by the small training set size, we carried out
dataset augmentation. This was performed by reflecting images C. Results
at random in both the vertical and horizontal axes. In effect,
The classification results on the Kvasir dataset are presented
the advantage of mirroring is to help learning the anatomical
in this section. So, we illustrate the comparison of the pre-
structure viewed from different perspectives [21].
trained VGG-16 model with the results corresponding to the
D. Motivation for Choosing VGG-16 MediaEval’s challenge on Kvasir dataset. Before performing
The choice of the pre-trained VGG-16 network, compared data augmentation our model reached out 96.9% accuracy. Fig-
to the AlexNet and InceptionNet networks was motivated by ure 4 depists the confusion matrix comparing the inter-class
[22]: performance of the second serie of experiemnt. Where the
diagonal represents correct labels. The results of our proposals
• VGG-16 is deeper than AlexNnet, so that he it has a
are competitive with MediaEval’s challenge results as shown in
better feature learning.
Table I, especially after performing data augmentation (serie
• VGG-16 has a stronger generalization ability than Incep-
2). However, this comparison is not very fair, because the
tionNet since it is simpler.
sets used to test and validate the model are not the same.
IV. E XPERIMENTS AND R ESULTS Nonetheless, the proposed models seem performant on each
We start this section by introducing the Kvasir dataset. Then provided metrics.
we discuss the results.
TABLE I
A. Kvasir Dataset P ERFORMANCE OF OUR MODELS AND M EDIA E VAL’ S CHALLENGE
MODELS
Kvasir dataset [3], [23] contains images from inside the
gastrointestinal (GI) tract, collected using endoscopic equip- Author Year Acc Pr R F1
ment at Vestre Viken Health Trust in Norway.This dataset was This study (Serie 1) 2021 96.9 84.9 84.8 83.2
made accessible in the fall of 2017 as part of MediaEval’s This study (Serie 2) 2021 98.8 87.3 88.6 87.3
Agrawal et al [8] 2017 96.1 84.7 85.2 84.7
Medical Multimedia Challenge, a benchmarking program that Pogorelov et al. [9] 2017 95.7 82.9 82.9 82.6
assigns challenges to the scientific community. [3]. The im- Naqvi et al. [10] 2017 94.2 76.7 77.2 76.7
age resolution varies from 720x576 up to 1920x1072 pixels. Petscharnig et al. [11] 2017 93.9 75.5 75.5 75.5
Liu et al. [12] 2017 92.6 70.3 70.3 70.3
Three major anatomical landmarks, three clinically significant
discoveries, and two endoscopic procedures are all represented
in the images. images have been annotated and verified by According to the confusion matrix, we can observe that
specialists. On some of the images, the left-most quarter of our proposal performs considerably well for Polyp and Dyed-
the image is devoted to annotations and image information, Resection-Margins. We can also see that the Normal z-line nzl
while the anatomical perspective takes up the remaining three and Esophagitis es classes cause the most confusion. We can
quarters. A green box in the bottom left corner of several justify this by the observation in [3]. Because both classes
photos depicts the endoscope’s position in the GI tract. Angle are captured from a similar anatomical location, except that
of view, size, luminance, zoom, and center point are all Esophagitis belongs to an unhealthy area and Normal z-line
different. In Figure 3, an image of each class in the collection to a healthy area. Because both classes are captured from a
is chosen at random. similar anatomical location, except that Esophagitis belongs
1) Anatomical landmarks: An aberrant characteristic of the to an unhealthy area and Normal z-line to a healthy area.
GI system that can be seen using an endoscopy. They also be Furthermore, Tabel I shows that our solution outperforms the
typical pathological sites such as ulcers or inflammation. The MediaEval’s challenge results.
anatomical landmarks included in Kvasit collection are Z-line,
pylorus and cecum. V. C ONCLUSION
2) Pathological findings: A characteristic of the gastroin-
testinal tract that is aberrant. Endoscopically, a damage or The use of a pre-trained VGG-16 deep CNN model with
alteration in the normal mucosa can be seen. The discovery transfer learning for image classification from inside the
could be a symptom of a developing sickness or a precursor. gastrointestinal tract is examined in this work. VGG-16 model
The pathologies ulcerative colitis , polyps, and esophagitis are with transfer learning was successful in providing a 98.8%
included in the Kvasir dataset. classification rate. Our proposed model has demonstrated per-
3) Endoscopic Procedures: Two set of images related to formance which is competitive to the MediaEval’s challenge
removal of lesions are provided, namely dyed and lifted polyp results. Our next challenge would be to improve the accuracy
and dyed resection margins. as close to 100 %. Our aim is to investigate the confusion
between the normal z line nzl and the esophagitises classes
B. Evaluation Criteria to reduce the error rate. Future works will be dedicated to
To validate the VGG-16 model’s efficiency using transfer investigating image quality enhancement to show the image
learning, the Accuracy (Acc), Precision (P r), Recall (R), and quality impact on deep classification, starting with exploiting
F1 score are used. These metrics are used widely in medical the preprocessing framework presented in [13]. Furthermore,
imaging [24]. another pre-trained models with transfer learning for the
 Fig. 3. An image of each class of the Kvasir dataset
Fig. 4. Confusion matrix demonstrating interclass accuracy.
classification task can be explored, or more deeper version
of VGG network.
R EFERENCES
[1] H. Q. Ontario et al., “Colon capsule endoscopy for the detection of col-
orectal polyps: an evidence-based analysis,” Ontario health technology
assessment series, vol. 15, no. 14, p. 1, 2015.
[2] C. E. Board. (2020) Colorectal-cancer: statistics. [Online]. Available:
https://www.cancer.net/cancer-types/colorectal-cancer/statistics
[3] K. Pogorelov, K. R. Randel, C. Griwodz, S. L. Eskeland, T. de Lange,
D. Johansen, C. Spampinato, D.-T. Dang-Nguyen, M. Lux, P. T. Schmidt
et al., “Kvasir: A multi-class image dataset for computer aided gastroin-
testinal disease detection,” in Proceedings of the 8th ACM on Multimedia
Systems Conference, 2017, pp. 164–169.
[4] Z. Al-Ameen, G. Sulong, and M. G. M. Johar, “Enhancing the contrast
of ct medical images by employing a novel image size dependent
normalization technique,” International journal of Bio-science and bio-
technology, vol. 4, no. 3, pp. 63–68, 2012.
[5] Z. Lai and H. Deng, “Medical image classification based on deep
features extracted by deep model and statistic feature fusion with
multilayer perceptron,” Computational intelligence and neuroscience,
vol. 2018, 2018.
[6] F. Altaf, S. M. Islam, N. Akhtar, and N. K. Janjua, “Going deep
in medical image analysis: Concepts, methods, challenges, and future
directions,” IEEE Access, vol. 7, pp. 99 540–99 572, 2019.
[7] G. Litjens, T. Kooi, B. E. Bejnordi, A. A. A. Setio, F. Ciompi,
M. Ghafoorian, J. A. Van Der Laak, B. Van Ginneken, and C. I. Sánchez,
“A survey on deep learning in medical image analysis,” Medical image
analysis, vol. 42, pp. 60–88, 2017.
[8] T. Agrawal, R. Gupta, S. Sahu, and C. Y. Espy-Wilson, “Scl-umd at the
medico task-mediaeval 2017: Transfer learning based classification of
medical images.” in MediaEval, 2017.
[9] K. Pogorelov, K. R. Randel, C. Griwodz, S. L. Eskeland, T. de Lange,
D. Johansen, C. Spampinato, D.-T. Dang-Nguyen, M. Lux, P. T. Schmidt
View publication stats
et al., “Kvasir: A multi-class image dataset for computer aided gastroin-
testinal disease detection,” in Proceedings of the 8th ACM on Multimedia
Systems Conference, 2017, pp. 164–169.
[10] S. S. A. Naqvi, S. Nadeem, M. Zaid, and M. A. Tahir, “Ensemble of
texture features for finding abnormalities in the gastro-intestinal tract.”
in MediaEval, 2017.
[11] S. Petscharnig, K. Schöffmann, and M. Lux, “An inception-like cnn
architecture for gi disease and anatomical landmark classification.” in
MediaEval, 2017.
[12] Y. Liu, Z. Gu, and W. K. Cheung, “Hkbu at mediaeval 2017 medico:
Medical multimedia task,” 2017.
[13] T. Cogan, M. Cogan, and L. Tamil, “Mapgi: Accurate identification of
anatomical landmarks and diseased tissue in gastrointestinal tract using
deep learning,” Computers in biology and medicine, vol. 111, p. 103351,
2019.
[14] B. Sdiri, F. A. Cheikh, K. Dragusha, and A. Beghdadi, “Comparative
study of endoscopic image enhancement techniques,” in 2015 Colour
and Visual Computing Symposium (CVCS). IEEE, 2015, pp. 1–5.
[15] A. Khan, A. Sohail, U. Zahoora, and A. S. Qureshi, “A survey of the
recent architectures of deep convolutional neural networks,” Artificial
Intelligence Review, vol. 53, no. 8, pp. 5455–5516, 2020.
[16] I. Goodfellow, Y. Bengio, A. Courville, and Y. Bengio, Deep learning.
MIT press Cambridge, 2016, vol. 1, no. 2.
[17] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification
with deep convolutional neural networks,” Advances in neural informa-
tion processing systems, vol. 25, pp. 1097–1105, 2012.
[18] K. Simonyan and A. Zisserman, “Very deep convolutional networks for
large-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.
[19] J. Yosinski, J. Clune, Y. Bengio, and H. Lipson, “How transferable are
features in deep neural networks?” in Advances in neural information
processing systems, 2014, pp. 3320–3328.
[20] S. Lu, Z. Lu, and Y.-D. Zhang, “Pathological brain detection based on
alexnet and transfer learning,” Journal of computational science, vol. 30,
pp. 41–47, 2019.
[21] M. Kim, J. Zuallaert, and W. De Neve, “Towards novel methods for
effective transfer learning and unsupervised deep learning for medical
image analysis,” in Doctoral Consortium (DCBIOSTEC 2017), 2017,
pp. 32–39.
[22] Q. Guan, Y. Wang, B. Ping, D. Li, J. Du, Y. Qin, H. Lu, X. Wan,
and J. Xiang, “Deep convolutional neural network vgg-16 model for
differential diagnosing of papillary thyroid carcinomas in cytological
images: a pilot study,” Journal of Cancer, vol. 10, no. 20, p. 4876,
2019.
[23] simula, “The kvasir dataset,” ps://datasets.simula.no/kvasir/, 2020.
[24] A. Benkessirat and N. Benblidia, “Fundamentals of feature selection:
An overview and comparison,” in 2019 IEEE/ACS 16th International
Conference on Computer Systems and Applications (AICCSA). IEEE,
2019, pp. 1–6.