Early Detection of Oral Cancer using Image Preprocessing

and Computational Techniques

A PROJECT REPORT
Submitted to

ASSAM DON BOSCO UNIVERSITY

By

JANOO PERIWAL - DC2020BTE0094

SONU MONI RABHA - DC2020BTE0169
MOHD UMAIR ALI SIDDIQUE - DC2020BTE0178

in partial fulfillment for completion

of Mini Project of
BACHELOR OF TECHNOLOGY
IN
COMPUTER SCIENCE AND ENGINEERING

DEPARTMENT OF COMPUTER SCIENCE & ENGINEERING

SCHOOL OF TECHNOLOGY ASSAM DON BOSCO UNIVERSITY
AZARA, GUWAHATI 781017, ASSAM, INDIA.
BATCH (2020- 2024)
EXAMINATION CERTIFICATE

This is to certify that Janoo Periwal (DC2020BTE0094), Sonu Moni Rabha

(DC2020BTE0169), Mohd. Umair Ali Siddique (DC2020BTE0178)of the Department of
Computer Science & Engineering has carried out the Project Work in a manner satisfactory to warrant
its acceptance and also defended it successfully.
I wish them all the success in their future endeavors.

Examiners:

1. Internal Examiner:

2. Internal Examiner:
ACKNOWLEDGEMENT

We take this opportunity to express our profound gratitude and deep regards to our guide
Mrs.Nupur Choudhury for her exemplary guidance, monitoring and constant encouragement
throughout the course of this mini project. The help and guidance given by him from time to
time shall carry us a long way in the journey of life on which we are about to embark.

We also take this opportunity to express a deep sense of gratitude to Assistant Professor Mrs.
Nupur Choudhury and Mrs. Smriti Priya Medhi, Project Coordinator, for their cordial
support, valuable information and guidance, which helped us in completing this task through
various stages.

We also thank our HOD, Dr. BOBBY SHARMA and ASSAM DON BOSCO UNIVERSITY
for giving us this opportunity to do our project to explore and increase our knowledge in the field
of Computer Science.

ABSTRACT
This study aimed to develop and evaluate state-of-the-art convolutional neural network (CNN) models
for the detection of oral cancer using biopsy histopathological images. The developed models VGG16,
VGG19, InceptionV3, AlexNet, ResNet50 showed high sensitivity and specificity for detecting oral
cancer, with the testing accuracy of 84%, 82%, 67%, 76%, 42% respectively. The study demonstrated
the effectiveness of image preprocessing techniques in improving the performance of the CNN models.
These findings suggest that the use of advanced computational techniques in conjunction with
histopathological images can significantly improve the detection of oral cancer in clinical practice,
potentially leading to better patient outcomes and a reduction in healthcare costs. The study underscores
the potential of deep learning and image processing methods to revolutionize cancer detection and
diagnosis.

Keywords : Convolutional neural network (CNN) models, Oral cancer, Image preprocessing
techniques, Deep learning
 TABLE OF CONTENTS

ACKNOWLEDGEMENTS
ABSTRACT
LIST OF TABLES
LIST OF FIGURES
ABBREVIATIONS

CHAPTER 1 INTRODUCTION
1.1 Overview of Oral Squamous Carcinoma and its Detection 1
1.2 Objective of the work 2
1.3 Domain of the work 3
1.4 Problem Definition 4
1.5 Background of the research 4
1.6 Motivation of the work 4
CHAPTER 2 LITERATURE SURVEY
2.1 Study of the existing systems 5
CHAPTER 3 FEASIBILITY STUDY
3.1 Technical Feasibility 7
3.2 Economical Feasibility 8
3.3 Operational Feasibility 8
3.2 Schedulic Feasibility 8
CHAPTER 4 SYSTEM DESIGN
4.1 Study Design 10
4.2 Workflow 11
4.3 Use case Diagram 11
CHAPTER 5 IMPLEMENTATION
5.1 Dataset description 12
5.2 Image pre-processing for pre-trained model 13
5.3 Image Segmentation 14
5.4 Random Forest Classifier 14
5.5 Graphic user Interface 15
CHAPTER 6 RESULT
6.1 Model evaluation matrix 17
6.2 Reviewing the CNN model performance individually
6
 6.2.1 VGG 16 18
6.2.2 VGG 19 19
6.2.3 Inception V3 19
6.2.4 ResNet 50 20
6.2.5 AlexNet 21
6.3 Segmentation Result 22
6.4 Performance Analysis 24

CHAPTER 7 CONCLUSION 25

REFERENCES 26

LIST OF TABLES
 Table Title Page

Table No. 1 Literature Review 5

Table No. 2 Software Requirements 7

Table No. 3 Hardware Requirements 7

Table No. 4 Estimation for Cocomo Model 9

Table No. 5 Results with CNN model 24

8
 LIST OF FIGURES

Figure Title Page

Figure No.1 Work Breakdown Structure 8

Figure No. 2 Gantt Chart 9

Figure No. 3 System of study design 10

Figure No. 4 Workflow of the approach 11

Figure No. 5 Use Case Diagram 11

Figure No. 6(A) Normal OSCC Figure 12

Figure No.6(B) Histopathological Images 12

Figure No. 7 Bar graph for dataset splitting 13

Figure No. 8 Random Forest Classifier 15

Figure No. 9(A) User Interface 16

Figure No.9(B) Browsing Window 16

Figure No.10(A) Uploaded Image 16

Figure No.10(B) Dropdown menu of mode 16

Figure No.11 Screenshot displaying prediction 16

Figure No.12 Architecture of Neural Networks of VGG 16 18

Figure No.13(A) Accuracy graph for VGG 16 18

Figure No. 13(B) Prediction for VGG 16 18

Figure No. 14 Architecture of Neural Networks of VGG 19 19

Figure No. 15(A) Accuracy graph for VGG 19 19

Figure No. 15(B) Prediction for VGG 19 19

Figure No. 16 Architecture of Neural Networks of Inception V3 20

Figure No. 17(A) Accuracy graph for Inception V3 20

Figure No. 17(B) Prediction for Inception V3 20

Figure No. 18 Architecture of Neural Networks of ResNet 50 21

Figure No. 19(A) Accuracy graph for ResNet 50 21

Figure No. 19(B) Prediction for ResNet 50 21

Figure No. 20 Architecture of Neural Networks of AlexNet 22

Figure No. 21(A) Accuracy graph for AlexNet 22

Figure No. 21(B) Prediction for AlexNet 22

Figure No. 22(A) Normal Image 23

Figure No. 22(B) Segmented Image 23

Figure No. 23(A) Normal Image 23

Figure No. 23(B) Segmented Image 23

Figure No. 24 Layer Summary of VGG 16 for segmentation 23

10
 ABBREVIATIONS

OSCC - Oral Squamous Cell Carcinoma

MRI - Magnetic Resonance Imaging
PET - Positron Emission Tomography
H&E - Hematoxylin and Eosin
CNN - Convolutional Nueral Network
LOGREG - Logistic Regression
RF - Random Forest
NN - Neural Network
SVM - Support Vector Machine
ANN - Artificial Neural Network
GPU - Graphic Processing Unit
AUC - Area Under Curve
TP - True Positive
TN - True Negative
FP - False Positive
FN - False Negative
VGG - Visual Geometry Group

Chapter 1
 Introduction

The present chapter describes an overview of Oral Squamous Carcinoma worldwide, objective
of the work, domain of the work, problem definition, background of research, motivation of
work.

1.1 Overview of Oral Squamous Carcinoma & it’s detection

Oral squamous carcinoma, also known as oral cancer, is a malignant tumour that develops in
the tissues of the mouth or throat. It is the most common type of head and neck cancer and can
occur in various areas, including the lips, gums, tongue, cheeks, floor of the mouth, and tonsils.
According to the World Health Organization (WHO), oral squamous carcinoma accounts for
over 90% of all oral cavity and oropharyngeal cancers. It is estimated that around 350,000 new
cases of oral cancer are diagnosed worldwide each year, and the incidence rate is increasing in
many countries. In the United States, the American Cancer Society estimates that in 2022, there
will be around 54,130 new cases of oral and oropharyngeal cancer and approximately 10,850
deaths from these cancers.[33] The five-year survival rate for oral squamous carcinoma is around
60%, although this can vary depending on the stage of the cancer at the time of diagnosis and
the individual's overall health. Risk factors for oral squamous carcinoma include tobacco use,
heavy alcohol consumption, a diet low in fruits and vegetables, exposure to the human
papillomavirus (HPV), and a family history of the disease. Prevention measures for oral
squamous carcinoma include avoiding tobacco and alcohol use, practicing good oral hygiene,
consuming a healthy diet, and receiving the HPV vaccine. Early detection through regular oral
exams and screenings can also improve the chances of successful treatment and survival. [30,31,32]

There are numerous technical methods for detecting cancer, such as:
Imaging technologies: Imaging tools like X-rays, CT scans, MRIs, and PET scans can be used
to see inside the body to find tumours or other abnormal growths.
Biopsy: During a biopsy, a small sample of tissue is taken from the body and examined under
a microscope to see if cancer is present.
Molecular diagnostic tests: To identify the precise type of cancer and how it is expected to
behave, these tests examine the genetic and molecular properties of cancer cells.
Liquid biopsy: This more recent sort of diagnostic procedure involves searching for cancerous
cells or DNA shards that cancer cells have shed in blood samples.
Artificial intelligence and machine learning: These technologies can be used to analyse medical
imaging and other diagnostic data to identify patterns that may indicate the presence of cancer.
Telemedicine: Telemedicine technologies can be used to remotely connect patients with
oncologists and other cancer specialists, allowing for more timely and efficient diagnosis and
treatment.
Overall, a combination of these and other technologies can be used to detect cancer at its
earliest stages, which can increase the chances of successful treatment and improve outcomes
for patients.

2
Traditional machine learning techniques do have certain limitations, despite the fact that they
have shown some encouraging findings in the detection and diagnosis of cancer.
Here are a few examples:
Low Data Availability: The restricted availability of high-quality data is one of the major
drawbacks of traditional machine learning in cancer detection. Even though a lot of data is
created in the field of cancer research, a lot of it is of low quality, lacking in detail, or
inconsistent, making it challenging to develop precise and trustworthy models.
Problem with Feature Extraction: Expert feature extraction is necessary for traditional machine
learning algorithms, but it is time-consuming, labor-intensive, and requires specialised
knowledge. It can be difficult to choose the proper set of characteristics that can effectively
describe cancer characteristics and distinguish between various types of cancer.
Overfitting: When traditional machine learning algorithms overfit the training set, they perform
poorly when applied to fresh data. As an algorithm gets too complicated, it starts to pick up
noise in the data instead of the underlying patterns.
Lack of Interpretability: Due to their opaque decision-making, traditional machine learning
algorithms are frequently referred to as "black boxes." In the field of medicine, it can be crucial
to be able to understand how the algorithm arrived at a specific prediction or diagnosis.
Restricted Scalability: Because the amount of data being generated for cancer diagnosis is
always growing, traditional machine learning methods may not scale well to huge datasets.
Although conventional machine learning techniques have demonstrated considerable potential
in cancer detection.

1.2 Objective of the work

The goal of this project is to :

● Detect and classify oral cancer (aka oral cavity cancer - A growth that occurs in the
mouth caused due to uncontrolled cell division.) from microscopic images (having 100x
magnification and 400-500x magnification) by machine learning methods.
● Study the present state of art technique, analysis and comparison on benchmark dataset
for the oral cancer detection from F-measure, sensitivity, specificity, accuracy,
precision points of view.
● Develop a user-friendly interface that will provide an interface for the machine learning
model for the prediction of cancerous and non-cancerous cells.

1.3 Domain of the work

To complete this project the following domains needed :

1. Machine learning. : Machine learning (ML) has been used extensively for cancer
prediction because it has shown promising results in detecting patterns and predicting
outcomes based on large and complex data sets. Early Detection: One of the most
significant benefits of using machine learning in cancer prediction is the potential for

3
 early detection. ML algorithms can analyze a large amount of data and detect patterns
that can indicate the presence of cancer, sometimes before physical symptoms appear.
Early detection is crucial in the treatment of cancer as it increases the chances of
successful treatment.
Machine learning is being used for cancer prediction because it has the potential to
improve early detection, provide personalized treatment recommendations, analyze big
data more accurately and efficiently, and improve the accuracy of cancer diagnosis.
These benefits can lead to better patient outcomes and contribute to the fight against
cancer.

2. Data Science : Data science, particularly machine learning, image processing, natural
language processing, and data visualization, can be useful for oral cancer detection.
These techniques can help analyze large and complex datasets, identify risk factors,
detect abnormalities, and develop personalized treatment plans. Machine learning can
be used to develop predictive models, while image processing can be used to analyze
oral cavity images. Natural language processing can extract and analyze data from
patient medical records, and data visualization can present complex data in a clear and
concise manner.

3. Bio-Technology : biotechnology can be used to develop digital pathology tools and

image analysis software that can analyze histopathological images of tissue samples to
identify cancerous cells. These tools can provide faster and more accurate diagnosis,
leading to better treatment outcomes. Additionally, by integrating histopathological
image data with other cancer-related data, biotechnology can provide a more
comprehensive understanding of cancer, leading to personalized treatment plans.

4. Application Development : A user interface always makes it easy for the

implementation and making predictions using algorithm.

1.4 Problem Definition

Oral cancers comprise a heterogeneous group of cancers, most commonly squamous cell
carcinomas (SCC), that typically arise from the epithelial lining of the oral cavity, sinonasal
tract, pharynx, larynx, and salivary glands. The global incidence of cancers of the oral cavity
is estimated to be 377 713 new cases and 177 757 deaths in 2020. The Surveillance,
Epidemiology, and End Results (SEER) 2018 report shows 4 out of 12 people can’t survive
oral cancer. Tobacco-related cancer is estimated to contribute 3.7 lakh cases, which is 27.1
percent of the total cancer burden in 2020 and highest in the north-eastern region of the country.

4
It is known that early detection of cancer is vital for a successful diagnosis and prognosis of
the disease. Moreover, due to the lack of education that can empower them to make decisions
to protect and improve their own health, they remain at risk of contracting the disease. Our
aim is to address this problem at an early stage so that an accurate diagnosis can be made at the
correct time for an effective line of treatment.

1.5 Background of the research

Due to exponentially growth of oral cancers in the current scanario, the need of it’s early
detection is quite challenging for the pathologists, doctors, researchers etc. Ultimately
computational techniques can easily replace the manual process of detecting the cancer. The
reliability of image preprocessing and computational techniques has greatly affected the
method of detection and also giving some prominent results. Using the above mentioned
techniques have greater reliability but still the process is challenging from the collection of
biopsy samples manually then giving H&E solutions and then comes the most time taking task
that is analyzing the histopathological images manually for the carcinoma and classifying
weather the Oral Squamous Cell is having carcinoma or not having the carcinoma.

1.6 Motivation of the work

Cancer is the leading cause of death worldwide. The National Cancer Registry Programme
declares that, in India, oral cancer is the most common cancer among men (16.1 percent of all
cancers) and the second most common cancer among women (10.4 percent of all cancers).
Healthcare costs will need to be reduced if we are to treat more people. The area of diagnosis
offers significant potential for cost reduction while improving accuracy through the use of
technology. Early detection of cancers has been a challenging task for pathologist, scientist,
researchers or doctors even after the computational techniques has been giving reliable results
due to manually examining the histopathological images for carcinoma classification.
Hence giving the region of interests using the segmentation technique of the machine learning
for the images and also providing a comparative analysis between the CNN models for the
same image giving different result can be a game changer for the early detection of cancer
using image preprocessing and computational techniques.
Using the application, it will be easier for the user to detect cancerous and non-cancerous cells.
Chapter 2

Literature Survey

2.1 Study of the existing systems

At the current scenario there are lots of research already done and still going on Early Cancer
Detection for specific regions example Breast, Skin, Oral Cavity, Lungs using Machine
Learning techniques.
Many approaches were made to help the oral onco-pathologist to screen the histopathological
section into normal OSF (oral submucous fibrosis) without Dysplasia and OSF with

5
Dysplasia.Staining the biopsy sections with H&E(Hematoxylin and eosin stain).[1] Obtaining
autofluorescence images from the patients with oral lesions and finding unique regions of
interest (ROI) sites corresponding to normal and confirmed neoplastic areas.[6]
There are some scanning softwares too developed where scans were done with axial orientation
of the mandibular occlusal plane. Scans were performed on 1 of 3 CT scanners (Highlight,
LightSpeed, or CTI Systems; General Electric Medical Systems, Milwaukee, Wis) and for
obtaining the images use of a bone algorithm was made.[5]

Table No. 1 Literature Review

S.No. YOP Author Name Dataset Domain of Work Technique Used Accuracy

1. 2022 Maximilian TCGA HNSC Machine learning models Classifiers- 88% - NN, 89%
Leitheiser et. al.1 predict the primary sites random forest SVM &
of head and neck (RF), neural LOGREG, and
squamous cell carcinoma network (NN), 83% -RF model
metastases based on DNA elastic net
methylation penalized
logistic
regression
(LOGREG),
and support
vector machine
(SVM)

2. 2022 O T Jones, et.al.3 Munich Artificial intelligence and ANN, SVM 85.30%
University machine learning
Hospital and algorithms for early
Eindhoven detection of skin cancer
hospital, and in community and
SkinVision user primary care settings
database of
smartphone
images,Data
from the
National Health
Interview
Survey

3. 2022 Yi-Chung Lin et.al 6 Binding DB Identifying Drug Targets Deep Neural Not Mentioned
of Oral Squamous Cell Network
Carcinoma through a
Systems Biology Method
and Genome-Wide
Microarray Data for Drug
Discovery by Deep
Learning and Drug
Design Specifications

4. 2022 M. Muthu Rama Not Mentioned Diagnosis of lymph node HE stained overall
Krishnan et.al. 10 metastasis in head and lymph node, accuracy,
neck squamous cell CNN sensitivity, and
carcinoma using deep specificity of the

6
 learning diagnostic
model reached
86%, 100%, and
75.9%

5. 2022 P.S. Hiremath et.al Not Mentioned Development and Logistic Not Mentioned
11 validation of machine regression,
learning-based risk random forest,
prediction models of oral support vector
squamous cell carcinoma machine with
using salivary the radial basis
autoantibody biomarkers function kernel,
eXtreme
Gradient
Boosting
(XGBoost), and
a stacking
model.

6. 2022 Dinesh V Rajput CLINICALLY Predicting clinical Random Forest 84.58% using
et.al 12 DATA FROM outcomes of radiotherapy (RF), Kernel RF.KSVM-
MEDICALS for head and neck Support Vector 95.12%,
squamous cell carcinoma Machine 77.55%,
patients using machine (KSVM), and 98.61%, and
learning algorithms XGBoost . 92.25%

7. 2022 Anuradha. K 13. NOT Histopathologic Oral AlexNet in the accuracy

MENTIONED Cancer Prediction Using convolutional 97.66% and
Oral Squamous Cell neural network 90.06%
Carcinoma Biopsy
Empowered with Transfer
Learning

8. 2020 Rahman, et.al.14 Ayursundra Study of morphological Medical Image 100%

Healthcare Pvt. and textural features for Processing, Oral
Ltd. and Dr B. classification Biopsy Images,
Borooah Cancer of oral squamous cell Cell
Institute carcinoma by traditional segmentation,
machine Cell
learning techniques classification

10. 2020 Krishnan et.al.15 Computer Science, Medical 100

Computer Vision and Image
Pattern Processing,
Recognition,Oral Oral Biopsy
Biopsy Images, Cell
segmentation,
Cell
classification

Chapter 3

Feasibility Study

7
3.1 Technical Feasibility

3.1.1 Software requirement

Table No. 2 Software Requirements

S.No Software Uses Versions

1 Python Programming A tool for development program which 3.10.7

language performs data manipulation

2 Tensorflow An open source artificial intelligence library 2.9.0

used for classification,
Perception,Understanding

3 Numpy A library for python programming language, 1.23.4

use for efficient computation on array

4 Jupyter notebook The Jupyter Notebook is a web-based v7.0.0

interactive computing platform. The
notebook combines live code, equations,
narrative text, visualizations, interactive
dashboards, and other media.

5 labelme Labelme is an open-source annotation tool 5.11

written in Python to support manual image
polygonal annotation for object
segmentation.

6 Keras Keras is an API used in creating the deep 2.9.0

learning models and enabling them
accessible to the devices.

7 ImageJ Labeling tool used for the masking of images 1.41

and easier finding the region of interest.

Table No. 3 Hardware Requirements

S.No Hardware Uses Versions

1 Processor A processor or processing unit is an electrical Intel i5 or higher , AMD

component (digital circuit) that performs Ryzen 5 or higher
operations on an external data source, usually
memory or some other data stream

2 GPU A graphics processing unit (GPU) is a specialized Nvidia Geforce Gtx 1650
electronic circuit designed to manipulate and alter or higher,
memory to accelerate the creation of images in a Radeon 5000 or higher
frame buffer intended for output to a display
device.

3.2 Economic Feasibility

8
The software required for designing and implementing in the this project are readily
available. As such, there is no economical constrains on the software for this project.

3.3 Operational Feasibility

Operational feasibility is to measure of how well a proposed system solve the problems, and
takes the advantage of the opportunities identified during the scope definition and how it
satisfies the requirement identified in the requirement analysis phase of system development.
The project is operational feasible as the user has background of doing the research project
and some knowledge for the techniques used in the project.

3.4 Schedulic Feasibility

3.4.1 Work Breakdown Structure

Figure No. 1 Work Breakdown Structure

3.4.2 Gantt Chart

9
 Figure No. 2 Gantt Chart
3.4.3 Cocomo Model

We have determined our projects fit the characteristics of Semi- Detached mode. We estimated
our project will have 3000 delivered source instructions, therefore using formula, we can
estimate :

Table No. 4 Estimation for Cocomo Model

*KLOC= Kilo Line of Code

Effort = A*(KLOC) B Productivity = KLOC/Effort
=3.0(1.72) 1.12 = 3/6.72
=5.8 person-month =0.45

Time =C(Effort) D Average staff size = Effort/Time

=2.5(5.8) 0.35 =6.72/5.9
=5.1months = 1.14persons

Chapter 4

10
 System Design

4.1 Study Design

For the early detection initially a biopsy sample dataset was gathered from a paper [13], the
workflow as shown in Figure 1 took two approaches - first approach is to preprocess the dataset
and predict the cancerous and non-cancerous cell by training, testing and validating the dataset
on various CNN models. Second approach is to label the cancerous histopathological images
using a software and extracting the features using classifiers based on which a segmented image
will be generated showing the region of interest.
Output of the two approaches will be displayed through a user interface having a window to
display image, buttons to browse the system and panel for showing the prediction.

Figure No. 3 System of study design

4.2 Workflow

11
 Figure No. 4 Workflow for the approach

4.3 Use Case Diagram

Figure No. 5 Use Case Diagram

Chapter 5

12
 Implementation
5.1 Dataset description

Previously published dataset was an image repository containing a total of 1224 images, which
are categorised into two sets based on their resolution. The first set includes 89
histopathological images of normal oral cavity epithelium and 439 images of Oral Squamous
Cell Carcinoma (OSCC) captured at 100x magnification. The second set comprises 201 images
of normal oral cavity epithelium and 495 histopathological images of OSCC captured at 400x
magnification. These images were obtained using a Leica ICC50 HD microscope from H&E
stained tissue slides, which were collected, prepared, and labelled by medical experts from 230
patients.

B
Figure No.6 (A) Normal OSCC, Figure No. 6 (B) Histopathological images

Set1 (100x magnified) images were used in both the approaches which is containing a total of
762 images, categorised into two sets i.e. Normal & OSCC. The splitting of dataset as shown

13
in Figure 1 was done using “train_test_split” library in the ratio of 70:15:15 for training, testing
and validation respectively.

Figure No. 7 Bar graph for dataset splitting

Data Source: https://data.mendeley.com/datasets/ftmp4cvtmb/1

5.2 Image Pre-Processing for Pre-Trained models

To pre-process the images we performed rotation upto the range of +15 degree (chosen
experimentally), sheared the images form -0.2 to +0.2, zoomed in the range of 80% to 120%,
horizontally flipped the images, shifted the width range upto maximum 10%, shifted the height
range upto maximum of 10%.The input of any dimension in a batch of 32 and classifying it
into Cancerous or Non Cancerous tissue with 244x244x3 dimension which depicts the image
matrix with 244 rows, 244 columns and 3 represents number of colour channels (red, blue,
green) in the image.The training, testing and validating are in the ratio of 70%, 15% and 15%
respectively.
After the image is pre-processed the images are fitted into classifier trained with the mentioned
dataset, taking the input of any size and dimension and classifying it into Cancerous or Non
Cancerous tissue with 244x244x3 dimension which depicts the image matrix with 244 rows,
244 columns and 3 represents number of colour channels (red, blue, green) in the image.
Now the results were shown through a user-interface which had four buttons (Load image,
View segmented image, Choose model, View output), a 500x500 window to display the image
and a highlighted label to display the classified result.

5.3 Image Segmentation

14
In this report, we present our approach to segmenting images of oral tissues into regions of
interest (ROIs) using a combination of traditional image processing techniques and machine
learning algorithms.
We used a dataset of oral tissue images collected from patients diagnosed with oral cancer. The
images were preprocessed by converting them to grayscale and resizing them to a standard size
of 256x256 pixels. We then applied a combination of traditional image processing techniques
and machine learning algorithms to segment the images into ROIs.

Our feature extraction pipeline included the following steps:

● Gabor filtering using various orientations, frequencies, and phases to extract texture
features from the images
● Edge detection using the Canny, Roberts, Sobel, Scharr, and Prewitt filters to
identify boundaries between different regions in the images
● Gaussian and median filtering to smooth the images and reduce noise
● Laplacian filtering to enhance edges and edges of edges in the images

5.4 Random Forest Classifier

RandomForestClassifier is a machine learning algorithm that uses an ensemble of decision

trees to perform classification. It works by creating multiple decision trees on random subsets
of the training data, and then combining their predictions to obtain a final output.
The algorithm first randomly selects a subset of the features, and then builds a decision tree
using a random subset of the training data. This process is repeated multiple times, with each
decision tree being trained on a different subset of the data and features. The final output is
then determined by combining the predictions of all the decision trees.
RandomForestClassifier is often used in situations where there are a large number of features
and a relatively small number of training examples, as it is less prone to overfitting than other
classification algorithms. It is also effective in handling noisy data, as the averaging of multiple
decision trees can help to reduce the impact of individual noisy examples.

15
 Figure No. 8 Random Forest Classifier

In medical diagnosis, RandomForestClassifier can be used to classify different types of cells

or tissues based on their features, as demonstrated in the example you provided. It can also be
used to classify patients into different categories based on their symptoms or test results, and
to predict the likelihood of a patient developing a particular disease based on their medical
history and other factors.
Overall, RandomForestClassifier is a powerful and flexible classification algorithm that can be
used in a wide range of applications, including medical diagnosis.
We used a Random Forest classifier to train a model on the segmented ROIs. The model was
trained using 80% of the data and tested on the remaining 20%.

5.5 Graphic User Interface

The user interface is designed for the user to directly upload the images and get a predicted
output whether the image contains cancer(cancerous) or not(non-cancerous) using a python
library tkinter(8.6).
The UI contain essential components like buttons to browse, window to see the image,
dropdown menu for the selection of desired CNN model, panel ot show the predicted result as
a message or text, cross button to close the software, minimisation button to minimize the
software and a maximization button to screen the software at full screen as you can see in the
Figure no. (9-11).

16
 A B

Figure No. 9 (A) User interface, Figure No. 9 (B) Browsing window

A B

Figure No. 10 (A) Uploaded image, Figure No.10 (B) Dropdown menu of mode

Figure No. 11 Screenshot displaying prediction

The frontend of the graphic user interface was developed using the python library “Tkinter
(8.6)” in which we have defined functions for the browsing, segment model, preprocessing of
image (resizing, horizontal flipping, shearing, etc.), drop down option, window etc. whereas
the predicted results are stored in jupyter notebook localhost server temporarily.

Chapter 6

17
 Results

6.1 Model Evaluation Metrics

For comparative analysis of models, metrics which are considered are : accuracy, recall,
precision, loss, auc score for each set (training, testing and validation)[40]
Accuracy: It is the number of correctly categorized testing samples in the complete testing set.
In other words, it is the likelihood that the model correctly predicts the patient's status (positive
or negative).[37]

(1)

Sensitivity: The number of testing samples correctly categorized as positive in all samples with
the ground truth positive is referred to as sensitivity. The true positive rate (TPR) is another
name for it. In other words, it is the likelihood that a patient with a condition will be projected
by the model to have the disease.

(2)

Precision: It measures how well a model predicts a certain category.

AUC: If we take two data points from two different classes, AUC metrics is used for identifying
that this percent(%) of chances is there that the model will correctly segregate or classify the
two different classes

Accuracy Comparison: For visualization the accuracy while testing and validating of the results
with a help of graph or pie-charts.

Loss Comparison: For visualizing the loss while testing and validating of the results with a
help of graph or pie charts.

6.2 Reviewing the CNN model performance individually

18
6.2.1 VGG 16

The VGG16 architecture consists of 16 layers of convolutional neural networks, which are
designed to learn and extract relevant features from images. This architecture has been proven
to be very effective in recognizing patterns and features in images and classifying them
correctly.

Figure No. 12 Architecture of Neural Networks of VGG-16

In oral cancer detection, VGG16 can be used to analyze images of the oral cavity and classify
them into normal or abnormal tissue. By training the model on a large dataset of oral cavity
images, the VGG16 algorithm can learn to recognize patterns and features that are indicative
of oral cancer.[29]

A
B

Figure No. 13 (A) Accuracy graph for VGG 16 , Figure No. 13 (B) Prediction for VGG 16
\
6.2.2 VGG 19

The VGG19 architecture consists of 19 layers of convolutional neural networks,

which makes it more complex than VGG16. The additional layers allow it to
extract more complex features from images, which can improve its accuracy in
recognizing patterns and features in images.

19
 Figure No. 14 Architecture of Neural Networks of VGG-19

In oral cancer detection, VGG19 can be used to analyze images of the oral cavity and classify
them into normal or abnormal tissue. By training the model on a large dataset of oral cavity
images, the VGG19 algorithm can learn to recognize patterns and features that are indicative
of oral cancer with high accuracy.[28]

A B

Figure No. 15 (A) Accuracy graph for VGG 19 Figure No. 15 (B) Prediction for VGG 19

6.2.3 Inception V3

The Inception V3 architecture is designed to extract features from images of

varying sizes, allowing it to accurately classify images with different resolutions.
This is particularly useful in oral cancer detection as medical images of the oral
cavity can vary in resolution and quality.

20
 Figure No. 16 Architecture of Neural Networks of Inception V3

In oral cancer detection, Inception v3 can be used to analyze medical images of the oral cavity
and classify them into normal or abnormal tissue. By training the model on a large dataset of
oral cavity images, the Inception v3 algorithm can learn to recognize patterns and features that
are indicative of oral cancer.[26]

A B

Figure No. 17 (A) Accuracy graph for Inception V3, Figure No.17 (B) Prediction for Inception V3

6.2.4 ResNet 50

The ResNet 50 architecture consists of 50 layers of convolutional neural networks, which

allows it to extract very complex features from images. This is particularly useful in oral cancer
detection as medical images of the oral cavity can contain complex patterns and features that
can be indicative of cancerous growth.

21
 Figure No.18
Architecture of
Neural Networks of
ResNet 50

In oral cancer
detection, ResNet
50 can be used to
analyze medical
images of the oral
cavity and classify
them into normal
or abnormal tissue. By training the model on a large dataset of oral cavity images, the ResNet
50 algorithm can learn to recognize patterns and features that are indicative of oral cancer with
high accuracy.[27]

A B

Figure No. 19 (A) Accuracy graph for ResNet 50,Figure No. 19 (B) Prediction for ResNet 50

6.2.5 AlexNet

22
While AlexNet is not as deep as some of the more recent architectures, it is still capable of
extracting important features from images, especially in cases where the dataset is not large
enough to require the use of a more complex architecture. Additionally, AlexNet can be faster
to train and execute than some of the more complex architectures, making it a useful option
when computational resources are limited.

Figure No. 20 Architecture of Neural Networks of AlexNet

By fine-tuning the pre-trained model on a smaller dataset of oral cavity images, the model can
learn to recognize patterns and features that are specific to oral cancer with high accuracy,
making it a useful tool for oral cancer detection. Using transfer learning with AlexNet can also
be computationally efficient, as the pre-trained model can reduce the training time and
computational resources required to train the model on the smaller dataset of oral cavity
images.[25]

A B

Figure No. 21 (A) Accuracy graph for AlexNet, Figure No. 21 (B) Prediction for AlexNet

6.3. Segmentation Results

Our segmentation method includes two models one of which is having traditional methods to
extract features using filters like gabor , canny edge, sobel, schar etc, the other model on the
other hand uses VGG16 model as a feature extractor.Figure 24 show the architecture of our
VGG16 model. Both models use Random forest classifiers for the classification.The first model
performs well after we train and test on a single image, giving an accuracy more than 90%. But
the model fails to train on more than 6 numbers of images. Figure 22 shows the segmentation
result of our first model. Our second model showed a good segmentation result after training
multiple images.on training dataset we have achieved an accuracy of 74%. Figure 23 shows
an example of the segmentation result of our second model results for one of the images in the
dataset. How ever this model fails to differentiate white background and ROI.

23
 Figure No 22 (A) Normal Image Figure No 22 (B) Segmented Image

A B

Figure No 23 (A) Normal Image Figure No 23 (B) Segmented Image

Figure 24 Layer summary of VGG16 for feature extraction

6.4 Performance Analysis

After the preprocessing of images and training-testing them with the CNN models we can
observe the following output on the basis of CNN model performance .Table 5 shows about
the accuracy, recall, precision, loss(%) of the mentioned models.

Table 5 Results with CNN models

24
S.No Model Accuracy Sensitivity(in Precision Loss AUC
(in %) %) (in %) (in %) (in %)

1. VGG 16 91.73 90.76 100 23.33 97.58

2. VGG 19 88.82 88.37 98.28 28.19 96.07

3. Inception V3 71.43 71.43 95.51 55.21 83.31

4. ResNet 50 76.98 94.74 78.95 49.90 81.66

5. AlexNet 42.86 27.37 89.66 84.62 68.78

Chapter 7

Conclusion

25
India is considered in the list of having the highest occurrence of oral cancer across the globe
and especially in North-East India. In this paper we have studied the features which are
essential for detecting the cancerous cell from a histopathological image and tested the images
with same features on CNN models for getting a prediction with higher accuracy.It can be
observed that the same augmented dataset is giving very different results for different CNN
models where VGG16 shows the highest AUC score of 0.9758, sensitivity of 0.9076, precision
score of 0.1 and testing accuracy of 0.9173 out of other mentioned model in section 6.2. Figure
13 shows the accuracy graph (A) which depicts that after 5 epochs an effective result can be
achieved. Making early detection possible with the help of this approach we also find a
loophole in the field of biotechnology that is interface for better navigation.As per the part of
our research we also developed a graphical user interface for the pathologists, doctor or any
particular user who wants to check for the cancer form the histopathological image explained
in section 5.5.
However, our pipeline during the segmentation section has some limitations. First Both
methods use the Random forest model as the classifier; it requires a significant amount of
computational resources as the Random forest model utilizes CPU instead of GPU. Second, the
pipeline may not generalize well to images with different lighting conditions or contrast levels.
Future work could explore the use of deep learning algorithms to address these limitations.

REFERENCES

26
[1] M. M. R. Krishnan, V. Venkatraghavan, U. R. Acharya, M. Pal, R. R. Paul, L. C. Min, et al.,
"Automated oral cancer identification using histopathological images: a hybrid feature extraction
paradigm," Micron, vol. 43, no. 2-3, pp. 352-364, 2012.

[2] M. M. R. Krishnan, P. Shah, M. Pal, C. Chakraborty, R. R. Paul, J. Chatterjee, and A. K. Ray,

"Structural markers for normal oral mucosa and oral sub-mucous fibrosis," Micron, vol. 41, no. 4, pp.
312-320, 2010.

[3] M. Muthu Rama Krishnan, Mousumi Pal, Ranjan Rashmi Paul, Ajoy Kumar Ray, Jyotirmoy
Chatterjee, Chandan Chakraborty, et al., "Automated classification of cells in sub-epithelial
connective tissue of oral sub-mucous fibrosis—An Support vector machine (SVM) based approach,"
Computers in Biology and Medicine, vol. 39, no. 12, pp. 1096-1104, Dec. 2009.

[4] D. V. Rajput and J. V. Tupkari, "Early detection of oral cancer: PAP and AgNOR staining in brush
biopsies," Journal of Oral and Maxillofacial Pathology, vol. 14, no. 2, p. 52, 2010.

[5] J. M. Brockenbrough, G. J. Petruzzelli, and L. Lomasney, "DentaScan as an accurate method of

predicting mandibular invasion in patients with squamous cell carcinoma of the oral cavity," Archives
of Otolaryngology–Head & Neck Surgery, vol. 129, no. 1, pp. 113-117, 2003.

[6] D. Roblyer, C. Kurachi, V. Stepanek, M. D. Williams, A. K. El-Naggar, J. J. Lee, et al., "Objective

Detection and Delineation of Oral Neoplasia Using Autofluorescence ImagingAutofluorescence
Imaging of Oral Neoplasia," Cancer Prevention Research, vol. 2, no. 5, pp. 423-431, 2009.

[7] M. Leitheiser, D. Capper, P. Seegerer, A. Lehmann, U. Schüller, K. R. Müller, et al., "Machine

learning models predict the primary sites of head and neck squamous cell carcinoma metastases based
on DNA methylation," The Journal of Pathology, vol. 256, no. 4, pp. 378-387, 2022.

[8] O. T. Jones, R. N. Matin, M. van der Schaar, K. P. Bhayankaram, C. K. I. Ranmuthu, M. S. Islam, et

al., "Artificial intelligence and machine learning algorithms for early detection of skin cancer in
community and primary care settings: a systematic review," The Lancet Digital Health, vol. 4, no. 6,
pp. e466-e476, 2022.

[9] Y. C. Lin and B. S. Chen, "Identifying Drug Targets of Oral Squamous Cell Carcinoma through a
Systems Biology Method and Genome-Wide Microarray Data for Drug Discovery by Deep Learning
and Drug Design Specifications," International Journal of Molecular Sciences, vol. 23, no. 18, p.
10409, 2022.

[10] H. Tang, G. Li, C. Liu, D. Huang, X. Zhang, Y. Qiu, and Y. Liu, "Diagnosis of lymph node metastasis
in head and neck squamous cell carcinoma using deep learning," Laryngoscope Investigative
Otolaryngology, vol. 7, no. 1, pp. 161-169, 2022.

[11] Y. C. Wang, P. C. Hsueh, C. C. Wu, and Y. J. Tseng, "Machine Learning Based Risk Prediction
Models for Oral Squamous Cell Carcinoma Using Salivary Biomarkers," Studies in Health
Technology and Informatics, vol. 281, pp. 498-499, 2021.

[12] T. Gangil, A. B. Shahabuddin, B. Dinesh Rao, K. Palanisamy, B. Chakrabarti, and K. Sharan,

"Predicting clinical outcomes of radiotherapy for head and neck squamous cell carcinoma patients
using machine learning algorithms," Journal of Big Data, vol. 9, no. 1, pp. 1-19, 2022.

[13] A. U. Rahman, A. Alqahtani, N. Aldhafferi, M. U. Nasir, M. F. Khan, M. A. Khan, and A. Mosavi,

"Histopathologic oral cancer prediction using oral squamous cell carcinoma biopsy empowered with
transfer learning," Sensors, vol. 22, no. 10, pp. 3833, 2022.

[14] T. Y. Rahman, L. B. Mahanta, H. Choudhury, A. K. Das, and J. D. Sarma, "Study of morphological

and textural features for classification of oral squamous cell carcinoma by traditional machine learning
techniques," Cancer Reports, vol. 3, no. 6, p. e1293, 2020.

[15] M. M. R. Krishnan, M. Pal, S. K. Bomminayuni, C. Chakraborty, R. R. Paul, J. Chatterjee, and A. K.

Ray, "Automated classification of cells in sub-epithelial connective tissue of oral sub-mucous

27
 fibrosis—An SVM based approach," Computers in Biology and Medicine, vol. 39, no. 12, pp. 1096-
1104, 2009.

[16] P. S. Hiremath, Y. H. Iranna, and J. D. Pujari, "Classification of squamous cell carcinoma based on
color and textural features in microscopic images of esophagus tissues," Journal of Computer Science,
vol. 3, no. 7, pp. 566-573, 2007.

[17] D. V. Rajput and J. V. Tupkari, "Early detection of oral cancer: PAP and AgNOR staining in brush
biopsies," Journal of Oral and Maxillofacial Pathology, vol. 14, no. 2, p. 52, 2010.

[18] A. Anuradha, B. Kiran Kumar Naik, G. Vijay Srinivas, R. S. Devi, and H. K. Puneet, "Incidence of
mast cells in oral squamous cell carcinoma: a short study," Journal of Oncology, 2014.

[19] T. Y. Rahman, L. B. Mahanta, A. K. Das, and J. D. Sarma, "Automated oral squamous cell carcinoma
identification using shape, texture and color features of whole image strips," Tissue and Cell, vol. 63,
p. 101322, 2020.

[20] V. Borse, A. N. Konwar, and P. Buragohain, "Oral cancer diagnosis and perspectives in India,"
Sensors International, vol. 1, p. 100046, 2020.

[21] B. Ilhan, K. Lin, P. Guneri, and P. Wilder-Smith, "Improving oral cancer outcomes with imaging and
artificial intelligence," Journal of Dental Research, vol. 99, no. 3, pp. 241-248, 2020.

[22] N. Anitha and K. Jamberi, "Diagnosis, and Prognosis of Oral Cancer using classification algorithm
with Data Mining Techniques," Int. J. Data Min. Tech. Appl, vol. 6, pp. 59-61, 2017.

[23] R. A. Welikala, P. Remagnino, J. H. Lim, C. S. Chan, S. Rajendran, T. G. Kallarakkal, et al.,

"Automated detection and classification of oral lesions using deep learning for early detection of oral
cancer," IEEE Access, vol. 8, pp. 132677-132693, 2020.

[24] M. Abdel-Basset, V. Chang, and R. Mohamed, "HSMA_WOA: A hybrid novel Slime mould
algorithm with whale optimization algorithm for tackling the image segmentation problem of chest
X-ray images," Applied Soft Computing, vol. 95, p. 106642, 2020.

[25] D. Cascio, V. Taormina, and G. Raso, "Deep CNN for IIF images classification in autoimmune
diagnostics," Applied Sciences, vol. 9, no. 8, p. 1618, 2019.

[26] S. Ramaneswaran, K. Srinivasan, P. D. R. Vincent, and C. Y. Chang, "Hybrid inception v3 XGBoost

model for acute lymphoblastic leukemia classification," Computational and Mathematical Methods in
Medicine, vol. 2021, pp. 1-10, 2021.

[27] M. N. S. Jahromi, P. Buch-Cardona, E. Avots, K. Nasrollahi, S. Escalera, T. B. Moeslund, and G.

Anbarjafari, "Privacy-constrained biometric system for non-cooperative users," Entropy, vol. 21, no.
11, p. 1033, 2019.

[28] M. Lagunas and E. Garces, "Transfer learning for illustration classification," arXiv preprint
arXiv:1806.02682, 2018.

[29] M. Loukadakis, J. Cano, and M. O'Boyle, "Accelerating deep neural networks on low power
heterogeneous architectures," in Proceedings of the International Conference on High Performance
Computing & Simulation (HPCS), 2018, pp. 112-118.

[30] National Institute of Dental and Craniofacial Research. (n.d.). Oral Cancer.
https://www.nidcr.nih.gov/health-info/oral-cancer

[31] Oral Cavity & Oropharyngeal Cancer Key Statistics 2021.

https://www.cancer.org/cancer/oral-cavity-and-oropharyngeal-cancer/about/key-statistics.html

[32] HPV and Oropharyngeal Cancer | Centers for Disease Control and Prevention.
https://www.cdc.gov/cancer/hpv/basic_info/hpv_oropharyngeal.htm

28
[33] Cancer of the Oral Cavity and Pharynx - Cancer Stat Facts.
https://seer.cancer.gov/statfacts/html/oralcav.html

[34] Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning," Nature, vol. 521, no. 7553, pp. 436-444, 2015.

[35] I. Goodfellow et al., "Generative adversarial nets," in Advances in Neural Information Processing
Systems (NIPS), Curran Associates, Inc., Red Hook, NY, USA, 2014, pp. 2672-2680.

[36] Z. C. Lipton, J. Berkowitz, and C. Elkan, "A critical review of recurrent neural networks for sequence
learning," arXiv preprint arXiv:1506.00019, 2015.

[37] D. Silver et al., "Mastering the game of Go with deep neural networks and tree search," Nature, vol.
529, no. 7587, pp. 484-489, 2016.

[38] G. E. Hinton and R. R. Salakhutdinov, "Reducing the dimensionality of data with neural networks,"
Science, vol. 313, no. 5786, pp. 504-507, 2006.

[39] M. Holleman, "Convolutional Neural Network on the iPhone with VGGNet,”

https://machinethink.net/blog/convolutional-nueral-networks-on-the-iphone-with-vggnet/.

[40] Z. Vujović, "Classification model evaluation metrics," International Journal of Advanced Computer
Science and Applications, vol. 12, no. 6, pp. 599-606, 2021.

29