2022 International Conference on Trends in Quantum Computing and Emerging Business Technologies (TQCEBT)
CHRIST (Deemed to be University), Pune Lavasa Campus, India. Oct 14-15, 2022
An effective Approach for Pneumonia Detection
using Convolution Vision Transformer
2022 International Conference on Trends in Quantum Computing and Emerging Business Technologies (TQCEBT) | 978-1-6654-5361-5/22/$31.00 ©2022 IEEE | DOI: 10.1109/TQCEBT54229.2022.10041662
Somya Gupta
CHRIST (Deemed to be University),
India
somya.gupta@msds.christuniversity.in
Parul Joshi
CHRIST (Deemed to be University),
India
parul.joshi@msds.christuniversity.in
Abstract – Early detection of pneumonia in patients
through effective medical imaging may enable timely remedial
measures and reduce the severity of the infection. There is an
increase in cases among new-borns, teenagers and also people
with health issues in recent years. The COVID-19 pandemic
also revealed the major impact pneumonia had on the lungs
and the consequences of delayed detection. The presence of the
infection in the lungs is examined through images of Chest X-
ray, however, for an early diagnosis of the infection, this paper
proposes an automated model as a more effective alternative.
Convolutional Vision Transformer (CVT) which gives an
accuracy of 97.13%, and is a robust combination of
Convolution and Vision Transformer (ViT), is suggested in this
paper as a potential model to detect pneumonia early in
patients.
Keywords - Chest X-ray, Pneumonia, Detection, Vision
Transformation, Convolution Neural Network, Convolutional
Vision Transformer, Deep Learning.
I. INTRODUCTION
Countries in the developing stage have registered an
increase in the cases of pneumonia. Excess pollution, poor
living conditions, rise in population and scarcity of medical
infrastructure has contributed to the increasing rate of
infection. Overcoming these setbacks is possible by looking
at how technology can be used on a wide scale to detect the
condition before it reaches a point of severity. Patients
diagnosed early will be able receive specialized treatments.
Due to the simple fact that pneumonia can be detected only
by images of a chest X- ray, there is a setback as assessing
the images is a challenging task and comes with high health
risk. [1]
Increasing the chance of survival for pneumonia patients
is possible through timely examination The method of
diagnosing the infection with chest X-ray images is risky and
subjected to several health concerns. An automated
algorithm is required which will improve the quality of
healthcare and help in better, more accurate diagnosis of the
infection. This algorithm will prove to be both time and life-
saving for the stakeholders [2]. Among the growing
technological advancements, computer vision is a precise
way to identify infection with the help of deep learning
frameworks.
978-1-6654-5361-5/22/$31.00 ©2022 IEEE
Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY CALICUT. Downloaded on August 26,2023 at 16:52:48 UTC from IEEE Xplore. Restrictions apply.
Andrea Rodrigues
CHRIST (Deemed to be University),
India
andrea.rodrigues@christuniversity.in
Jossy George
CHRIST (Deemed to be University),
India
frjossy@christuniversity.in
As per the findings of studies in the health sector, y
Convolutional Neural Networks (CNN), was instrumental in
examination of chest X-ray images for the identification of
the infection. However, the accuracy obtained with it is not
satisfactory. The other deep learning models used for the
same purpose are VGG-19 and ViT. ViT is an upcoming
new model for image processing which is still in the process
of development. ViT has lately surpassed CNN’s
performance for image classification, however, the expense
of pre-training is still exorbitant given the huge outside
datasets [3].
A combination of Vision Transformer along with
convolutional and pooling layers has been employed in this
study to examine chest X-Ray images. The CVT system
combines two widely used architectures to overcome some
important limitations of each approach on its own, namely
Convolutional Neural Networks (CNNs) and Transformer-
based models. By leveraging both techniques, the Vision
System can outperform existing architectures, especially at
low data levels while achieving similar performance with
respect to the large dataset regime. This process occurs
without compromising on accuracy or speed [4].
II. LITERATURE REVIEW
Debaditya Shome et al., [5] described the
Vision Transformer (ViT) which supports deep learning for
COVID -19 prediction from the images that are based on
chest X-ray. A total of 30 K collection images of chest X-
rays were gathered and used. The data set for the study
composed of mixing different open-source data sets A very
good score of accuracy that reached to 98% and 99% was
noted. A self-operating pneumonia detection model works
effectively in early prediction of pneumonia in many places
as told by Khushal Tyagi et al., [6] They implied 3 different
models named Convolutional Neural Network (CNN),
VGG16, and Visual Transformer (ViT) and compared the
outcomes of all models. The outcomes suggested that ViT
can recognize pneumonia with 96.45% accuracy. This model
can be used to prevent unfortunate situations in far off
places.
As explained in Boyuan Wang et al., [7] the paper shows
how early detection of COVID - 19 through deep learning
could help in curbing the spread of the virus to some extent
and also decrease hospital costs. Swim Transformer (ST) is
1
 chosen as the second network to make a model for the same.
Later this model is compared with Vision Transformer (ViT)
and Convolutional Neural Network (CNN) and it
outshone both the other models. When it was pre - trained on
a big data and send to many mid - size or small image
identification benchmarks, Vision Transformer (ViT) attains
excellent outcomes as compared with Convolutional Neural
Networks (CNN) rather requiring substantially lesser
estimated resources to train Showed in their paper, by
Alexey Dosovitskiy et al., [8]
Sangjoon Park et al., [9] described the situation that the
images of chest x-ray were taken yearly in hospitals is not
suitable to be used as there is less of manual labelling by
experienced people, mainly in underprivileged areas. They
demonstrated that a deep learning structure that takes help of
knowledge processing self-monitored learning and training,
that is proof that the functionality of the real model
developed with a fewer amount of labels may eventually be
better with increased unlabelled information. Testing
solutions show that the suggested model attains better
robustness in contrast to the existing surroundings and can
be applied easily to varied diagnosis of infections and
viruses, including tuberculosis (Tb), COVID-19, and
pneumothorax. As explained by Tathagat Banerjee et al.,
[10] various types of pneumonia have same and difficult
symptoms. In the given paper, they assessed the problem and
attempted to devise an architecture that used Unsupervised
Generative Transfusion Network, also known as (UGTN).
This architecture is drafted with the help of state-of-the Art
Transformers (AT) and eight layer’s decoders and encoders
to arrive at an accuracy that amounted to 92 and 81% which
is equal to and in some cases higher than the predecessor
networks. Finally, the researchers suggest a firm
Unsupervised Generative Transfusion Network (UTGN)
architecture which is of an alternative style send family with
the feature of Generative Adversarial Networks (GAN).
Xiaohong Gao et al., [11] described vision transformer
architectures built upon attention models and are scalable
when compared with CNN based models. In comparison
with CNN based model DenseNet for COVID-CT, ViT
model appears to perform better with 76.6% accuracy
whereas DenseNet realized accuracy of 73.7%. While chest
CT images are in 3D volume, it is a natural approach to
process these data in 3D form. As addressed previously, the
lesioned regions are proportionally small compared to the
whole volume, which might constitute the main reason that
3D based systems perform far worse than 2D based models.
Koushik Sivarama Krishnan and Karthik Sivarama Krishnan
[12] explained that the lungs of the people infected with
pneumonia swell up as the infection increases. Examining
chest X-rays is thus a way to identify pneumonia. In the
paper they have employed an automated and effective
technique for distinguishing COVID-19 from the conditions
of Pneumonia, X-rays of the chest, and lung opacity. If a big
version of Vision Transformer (ViT) is used with a better
and bigger dataset it is possible to reach a better and higher
performance metrics.
Nam Nguyen and J. Morris Chang [13] proposed a better
model for detection of pneumonia which is an amalgamation
of model-centric and data-centric approaches. They begin
by, formulating pre-training focused on data for very scarce
data conditions of the search dataset. After that, they showed
two hybrids which is also a mixed model employing Dense
2
Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY CALICUT. Downloaded on August 26,2023 at 16:52:48 UTC from IEEE Xplore. Restrictions apply.
Associative Memory (Modern Hopfield Network) and the
Convolution-Attention Neural Architectures which preserves
the self-attention from the Vision Transformer (ViT). Their
approach is able to result in a more effective development in
comparison to the Conventional baseline approach. The
better working model in their study was able to achieve R2 =
0.85 ± 0.05 and Pearson correlation coefficient ρ = 0.92 ±
0.02 in geographic extent & R2 = 0.72 ± 0.09, ρ = 0.85 ±
0.06 in blur prediction. A deep learning-based framework is
described by Mohamad Mahmoud Al Rahhal et al., [14] for
the detection of COVID-19 through Computed Tomography
(CT) and X-ray images. The ViT model was employed to
depict the pipeline which used a Siamese encoder. The
Siamese encoder was used to enable the distillation and class
token They used arouse Convolution at varied rates to enable
more deep features from multi-scale feature maps. Increasing
the dataset was possible with the help of data augmentation
which generated adversarial examples that enhanced the
performance and the result of the model.
III. PROPOSED FRAMEWORK
A. Dataset
The dataset for this study is procured from Kaggle and it
consists of 5856 images which are divided into two sets:
train and test, each of which have two categories “Normal”
and “Pneumonia” as shown in Fig. 1. In the training set, the
number of Chest X-ray images of patients who do not have
pneumonia is 1346 and for patients who have Pneumonia is
3883 X-rays. In the test set, the division is of 234 X-rays
indicating no pneumonia and 390 having pneumonia.
Fig. 1. Two Categories of the Dataset
1) Data resizing and Augmentation
Fig. 2 shows that the dataset is not distributed in a
balanced way This imbalance may result in biased
prediction or it may produce overfitting problems. Data
augmentation was applied on the training set to remove this
unevenness.
Data Augmentation is a technique which helps in
producing synthetic data from the already existing data. So,
the data is altered using various transformations such as
scaling, cropping or rotation, but the meaning of the data
remains intact. This technique helps in increasing the size of
the dataset as well as balancing the imbalanced class by
adding the synthetic data. Hence this technique has been
used to deal with the problem of imbalance data.
Following augmentations have been applied on the data:
x Randomly rotated some training images from
training by two degrees.
 x Randomly zoomed some training images by 2%
x Randomly flipped images horizontally.
Fig 2: Imbalanced Dataset
2) Methodology
The model presented in this paper is a combination of
two types of algorithms - a) Convolutional Neural Networks
and b) Vision transformer.
In this model too the Vision Transformer is used along
with some changes. Convolution layers are added in the
algorithm of Vision Transformers.
The pipeline or the flow of the model is depicted in Fig.
3. This flowchart gives a basic understanding of the model
from the initial process of data aggregation till deployment.
Fig. 3: Structure of Convolutional Vision Transformer
B. Convolutional Neural Networks
CNN are deep learning models which consist of feed
forward networks and different layers such as convolutional
layers and pooling layers. Each of these layers have their
own functionality and they form the architecture of CNN
which is widely used for image recognition and
classification.
3
Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY CALICUT. Downloaded on August 26,2023 at 16:52:48 UTC from IEEE Xplore. Restrictions apply.
x Convolutional Layers – In this layer, filters are
applied to extract information about various features
of the image. The output of this layer is termed as
feature map and it provides information about the
edges and corners of the image.
x Pooling Layer – The main purpose of this layer is to
decrease the dimension or size of the feature map by
summarizing the features and converting it into a
smaller size matrix with significant or relevant
features.
x Fully Connected Layer – This is the last layer of the
architecture and, as the name suggests, it consists of
fully connected layers in which all neurons are
linked to each other. Then the output of pooling
layer is flattened and passed on and then the
classification process commences.
C. Vision Transformer
The concept of transformers was originally taken from
Natural Language Processing. Transformers have been used
for years in the NLP domain and have attained popularity for
their simplicity and high performance.
Vision Transformers model was introduced in a research
paper titled “An image is Worth 16*16 Words: Transformers
for Image Recognition at Scale.” A total of twelve scholars
from the Google Research Brain Team including Neil
Houlsby and Alexey Dosovitskiy developed this model.
The input image of ViT is broken into patches of the
same size. The size of patch is chosen randomly according to
the trial and test method. According to the patch size and the
image size, number of patches are calculated based on the
given formula:
2
§ image _ size ·
number _ of _ patches ¨ ¸
© patch _ size ¹
These patches, also known as tokens are transformed in 2
Dimension by flattening them into a vector format. Then a
layer of position embedding is added to the patches to secure
positional information [5]. Finally, these tokens pass through
the transformer encoder consisting of three layers:
x Multi -Head Self Attention Layer (MSP)
x Layer Norm (LN)
x Multi – Layer Perceptron (MLP) Layer
The following equation has been used for positional
embedding of the patches:
2i
d mod el
PE( pos ,2 i ) sin( pos / 10000 )
2i
d mod el
PE( pos ,2 i 1) cos( pos / 10000 )
D. Convolution Vision Transformers (CVT)
The model proposed in this paper consist of a vision
transformer along with the added convolutions.
 Instead of adding the full architecture of CNN, only the
convolution layer and pooling layer is added before the
positional embedding layer of the transformer. The image
patches are first passed on to the convolutional and pooling
layers and only then it is embedded, followed by the
transformer encoder and finally the output (as shown in Fig.
4)

Fig. 5: Model Accuracy and Loss (overfitting results)

Fig. 4: Structure of Convolution Vision Transformer

The convolutions have been placed after the patches with

the intention of obtaining important features from each of
them instead of passing the whole image in the convolutions
and then dividing the obtained output in the patches. This
setup helps in obtaining the importance of each patch, which
will then pass on to the transformer encoder and will help the
model learn the features efficiently, even on smaller dataset.
This setup is notably also giving the best model accuracy.
IV. EXPERIMENTAL SETUP
After setting up of the algorithm, the model was tried and Fig. 6: Model Accuracy and Loss (without overfitting)
tested with different parameters and each time the accuracy
and loss graph was plotted to determine the overfitting. Fig.
5 depicts the initial stage of the model which performed Table 1. shows the initial and final values of the
extremely well on the training set, but proved to not be as parameters which were being modified to achieve the final
efficient on the testing set indicating the presence of biasness model.
and model overfitting. Hence, to remove it, the model was
executed several times before the final model was achieved TABLE I. DIFFERENT PARAMETERS OF THE MODEL
which resulted in promising findings and showed no Parameters Initial Value Final Value
overfitting (as shown in Fig. 6).
Learning Rate 0.0001 0.00001
No. of Epoch 75 120
Batch Size 128 256
No. of heads 4 4
Patch Size 9 6
Projection Dimension 64 64
Image Size 90 72
No. transformer layers 8 8
Number of Patches 100 144

4
Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY CALICUT. Downloaded on August 26,2023 at 16:52:48 UTC from IEEE Xplore. Restrictions apply.
 V. RESULTS AND DISCUSSION
After performing training processes and also testing the
algorithm, the model’s efficiency can be stated based on
four performance metrics namely Precision, Recall, F1 score
and Accuracy.
x Accuracy– It gives the extent to which the model is
able make correct predictions. The formula for
accuracy is:
Correct _ predictions
Accuracy
Total _ predictions
x Precision – It estimates how many values are really
positive out of all the values or classes predicted as
positive by the model.
x Recall – It is an estimate which states that out of all
the positives in the data, how many is the model
able to predict correctly.
x F1 Score – F1 score is another evaluation metric
which is used to assess the model’s performance.
Usually, the F1 score is used for evaluating any
model built on an imbalanced dataset instead of
precision and recall. This is because F1 score is
attained when precision and recall are combined,
hence biasness is eliminated from this evaluator.
Precision u Recall
2u
Precision  Recall
Firstly, the training and testing accuracy obtained by the
model is 97.13% and 93.43% respectively which indicates
that the model is performing very well on both training and
testing data. Both the accuracies have been plotted and
shown in Fig. 7.
Fig. 7: Train and Test Accuracy
Confusion matrix is plotted for the evaluation and it is
shown below in the Fig. 8.
5
Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY CALICUT. Downloaded on August 26,2023 at 16:52:48 UTC from IEEE Xplore. Restrictions apply.
NORMAL PNEUMONIA
Fig. 8: Confusion Matrix of the Model
Confusion Matrix is essential as it aids in computing the
False Positive (FP), True Positive (TP), False Negative (FN),
and True Negative of the prediction where,
FP indicates that the prediction is positive but in reality,
it is false.
TP means that the in reality and in prediction it is
positive.
FN indicates that the prediction is negative and same is
the case in reality
TN indicates that the prediction showed negative but in
reality it is found to be true.
Here the two classes defined are ‘Pneumonia’ and
‘Normal’ instead of Positive and Negative. The following
can be inferred from the plotted confusion matrix:
x 368 images have been correctly predicted as
Normal.
x 22 images have been predicted as normal but it
actually belongs to Pneumonia class.
x 215 images predicted correctly as Pneumonia
images.
x 19 images incorrectly predicted as Pneumonia.
In the end to get the value of precision, recall and F1
score, classification report has been employed. The report
obtained for the model is shown in Table 2.
TABLE II. F1 SCORE, RECALL AND PRECISION
Here, 0 refers to the ‘Pneumonia’ class and 1 refers to the
‘Normal’ class.
The F1 score for both the classes: 0 and 1 is 95% and
91%, respectively, which is a very good score and it
 indicates that the model is able to predict and deal with both
the classes effectively.
Table 3 depicts the comparison of the model with the
other existing models. And it can be seen that CVT
outperforms all the models in all the evaluation metric. Also,
as mentioned before CVT can even perform better with less
data and therefore it requires less computational power,
which makes CVT better than others.
TABLE III. COMPARISON ACROSS MODELS
VI. CONCLUSION
Employing the Convolution Vision Transformer (CVT)
to examine chest X-Ray images was proposed to arrive at a
timely detection of pneumonia. CVT is a combination of
Vision Transformer along with convolutional and pooling
layers. This hybrid model not only outperforms both CNN
and CVT, but it also it overcomes the limitation of both the
approaches. The final train and test accuracy obtained by the
model is 97.13 % and 93.34 % respectively. Pneumonia is an
extremely critical health issue and early detection can be life
saving for some patients. This model will help in faster and
more accurate examination of pneumonia and will help the
health care sector to function effectively.
REFERENCES
[1] R. Kundu, R. Das, Z. W. Geem , G. T. Han and R. Sarkar,
"Pneumonia detection in chest X-ray images using an ensemble of
deep learning models," Plos one, no. 1, 2021.
[2] B. Almaslukh, "A Lightweight Deep Learning-Based Pneumonia
Detection Approach for Energy-Efficient Medical Systems," Wireless
Communications and Mobile Computing, p. 14, 2021.
6
Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY CALICUT. Downloaded on August 26,2023 at 16:52:48 UTC from IEEE Xplore. Restrictions apply.
[3] S. d’Ascoli, H. Touvron, M. L. Leavitt, A. S. Morcos, G. Biroli and
L. Sagun, "Convit: Improving vision transformers with soft
convolutional inductive biases," International Conference on Machine
Learning, no. 1, pp. 2286-2296, 2021.
[4] A. Razzaq, "Marktechpost.com," 20 july 2021. [Online].
[5] D. Shome, T. Kar, S. N. Mohanty, P. Tiwari, K. Muhammad, A.
AlTameem and A. K. J. Saudagar, "Covid-transformer: Interpretable
covid-19 detection using vision transformer for healthcare.,"
International Journal of Environmental Research and Public Health,
no. 1, 2021.
[6] K. Tyagi, G. Pathak, R. Nijhawan and A. Mittal, "Detecting
Pneumonia using Vision Transformer and comparing with other
techniques.," 5th International Conference on Electronics,
Communication and Aerospace Technology (ICECA), pp. 12-16,
2021.
[7] B. Wang, D. Zhang and Z. Tian, "STCovidNet: Automatic Detection
Model of Novel Coronavirus Pneumonia Based on Swin
Transformer.," no. 1, 2022.
[8] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai,
T. Unterthiner and N. Houlsby, "COVID-Transformer: Interpretable
COVID-19 Detection Using Vision Transformer for Healthcare," no.
1, 2020.
[9] S. Park, G. Kim, Y. Oh, J. B. Seo, S. M. Lee, J. H. Kim and J. C. Ye,
"AI can evolve without labels: self-evolving vision transformer for
chest X-ray diagnosis through knowledge distillation.," 2022.
[10] T. Banerjee, S. Karthikeyan, A. Sharma, K. Charvi and S. Raman,
"Attention-Based Discrimination of Mycoplasma Pneumonia,"
Proceedings of International Conference on Computational
Intelligence and Data Engineering , no. 1, pp. 29-41, 2022.
[11] X. Gao, Y. Qian and A. Gao, "COVID-VIT: Classification of Covid-
19 from CT chest images based on vision transformer models," p. 6,
2021.
[12] K. Sivarama Krishnan and K. Sivarama Krishnan, "Vision
Transformer based COVID-19 Detection using Chest X-rays," p. 5,
2021.
[13] N. Nguyen and J. M. Chang, "COVID-19 Pneumonia Severity
Prediction using Hybrid Convolution-Attention Neural
Architectures," 2021.
[14] M. M. Al Rahhal, Y. Bazi, R. M. Jomaa, A. AlShibli, N. Alajlan, M.
L. Mekhalfi and F. Melgani , "COVID-19 Detection in CT/X-ray
Imagery Using Vision Transformers," Journal of Personalized
Medicine, 2022.