NAME: MITHUN KUMAR

REG. NO.: 20MID0025

COURSE: INTELLIGENT DATABASE SYSTEMS
GUIDED BY: PROF. POORNIMA N.
SLOT: E1+TE1
SCHOOL OF COMPUTER SCIENCE AND ENGINEERING
WINTER SEMESTER 2022-23
DIGITAL ASSIGNMENT-1

Title
A Robust Traffic Sign Recognition System using Deep
Convolutional Neural Network & Keras
1.Abstract

Traffic congestion and road safety are major concerns in many cities in
India and other countries. One way to improve road safety is by ensuring
that drivers accurately recognize and obey traffic signs. However, due to
factors such as sign wear and tear, bad weather, and human error,
drivers may not always be able to accurately identify traffic signs. This
can lead to dangerous driving situations, such as running red lights or
failing to yield.

To address this problem, a traffic sign classifier system can be

developed. This system will use computer vision and machine learning
algorithms to automatically identify and classify traffic signs in real-time.
The classifier will be trained on a large dataset of traffic sign images,
German traffic datasets, to ensure high accuracy in recognizing a variety
of traffic signs.

The goal of this project is to build a traffic sign classifier system that can
accurately recognize traffic signs in real-world driving conditions and
help improve road safety in India and other countries.
 1. Introduction

A traffic sign classifier system using computer vision and machine learning can
improve road safety by accurately recognizing traffic signs. The classifier can be
used in self-driving vehicles, navigation systems, and other driver assistance
technologies to provide real-time information about traffic signs and road
conditions. The goal is to reduce the risk of accidents caused by human error
in recognizing traffic signs and improve road safety.

2. Related research
https://www.sciencedirect.com/science/article/abs/pii/S026288560
2001567

Literature survey 1:
"Traffic sign recognition and analysis for intelligent vehicles" by A. de
la Escalera, J. Ma Armingol, and M. Mata presents an extensive
literature survey of traffic sign recognition systems for intelligent
vehicles.

The authors begin by introducing the concept of intelligent vehicles

and the importance of traffic sign recognition for their safe and
efficient operation. They then review the historical development of
traffic sign recognition systems, starting with rule-based and
template-based approaches and progressing to more sophisticated
machine learning algorithms, including neural networks and support
vector machines.

The authors go on to review the various image processing techniques

that have been used in traffic sign recognition systems, including
color-based segmentation, edge detection, and feature extraction.
They also discuss the challenges associated with traffic sign
recognition, such as variability in lighting conditions, occlusion, and
different perspectives.
The study then presents a comprehensive overview of the different
datasets that have been used to evaluate traffic sign recognition
systems, including the German Traffic Sign Recognition Benchmark
(GTSRB) and the LISA Traffic Sign Dataset. The authors also discuss
the different performance metrics that have been used to evaluate
these systems, such as accuracy, precision, recall, and F1-score.

Finally, the authors present a summary of the current state-of-the-art

in traffic sign recognition for intelligent vehicles, highlighting recent
advances in deep learning approaches and the potential of using
multiple sensors for improved performance.

Overall, the literature survey presented in this paper provides a

comprehensive overview of the different approaches to traffic sign
recognition for intelligent vehicles and highlights the challenges and
opportunities associated with this important area of research.

Research objective:
The research objective of the "Traffic sign recognition and analysis for
intelligent vehicles" by A. de la Escalera, J. Ma Armingol, and M.
Mata is to provide a comprehensive review of the state-of-the-art in
traffic sign recognition systems for intelligent vehicles. The objective
is to present a detailed survey of the different image processing
techniques, machine learning algorithms, and datasets that have been
used for traffic sign recognition, as well as the performance metrics
used to evaluate these systems. The authors also aim to identify the
challenges associated with traffic sign recognition and highlight the
potential of using multiple sensors for improved performance.
Ultimately, the goal of this research is to provide insights and
recommendations for the development of more accurate and efficient
traffic sign recognition systems for intelligent vehicles.
https://ieeexplore.ieee.org/abstract/document/4279118

Literature survey 2:
The research "Automatic Detection and Classification of Traffic Signs"
by Carlos Filipe Paulo and Paulo Lobato Correia published in the
IEEE explores the state-of-the-art in traffic sign detection and
classification systems. The authors present an overview of different
approaches and techniques used for automatic traffic sign
recognition, including shape-based methods, color-based methods,
and hybrid methods that combine both shape and color features. They
also provide an overview of the challenges involved in traffic sign
recognition, including variations in lighting, viewpoint, and occlusion.

The study presents a comparative study of different feature extraction

techniques and machine learning algorithms for traffic sign
recognition. The authors also evaluate the performance of their
proposed system on two public datasets, comparing their results with
those of other state-of-the-art systems.

Furthermore, the paper addresses the issue of real-time traffic sign

recognition, highlighting the importance of fast processing times for
practical applications. The authors propose a hybrid approach that
combines both shape and color features and uses a hierarchical
classification scheme to achieve real-time performance.

Overall, the objective of this literature survey is to provide a

comprehensive overview of the techniques and challenges involved in
traffic sign detection and classification, and to propose a new hybrid
approach for real-time traffic sign recognition.

Research objective:
The research objective of the "Automatic Detection and Classification
of Traffic Signs" by Carlos Filipe Paulo and Paulo Lobato Correia
published in IEEE is to investigate different approaches and
techniques for automatic traffic sign recognition, with a focus on real-
time performance. The objective is to propose a new hybrid approach
that combines both shape and color features and uses a hierarchical
classification scheme to improve the accuracy and efficiency of traffic
sign recognition systems.

The authors aim to present a comparative study of different feature

extraction techniques and machine learning algorithms for traffic sign
recognition, evaluating their performance on public datasets and
comparing their results with those of other state-of-the-art systems.
They also aim to identify the challenges involved in traffic sign
recognition, such as variations in lighting, viewpoint, and occlusion,
and propose solutions to address these challenges.

Ultimately, the goal of this research is to propose a new traffic sign

recognition system that can accurately and efficiently detect and
classify traffic signs in real-time, improving the safety and efficiency
of intelligent transportation systems.

https://ieeexplore.ieee.org/abstract/document/9478172
Literature survey 3:
In the research "Traffic Signs Classification using Convolutional
Neural Networks: A review" published in IEEE, Palak and A.L. Sangal
provide an overview and evaluation of the use of Convolutional
Neural Networks (CNNs) for traffic sign recognition. They review
several research studies that have used CNNs to address the
challenges of traffic sign classification, such as variations in lighting,
weather conditions, and camera angles.

The authors highlight the importance of traffic sign classification for

intelligent transportation systems and present a comprehensive
review of the current state-of-the-art techniques for traffic sign
recognition. They provide an overview of the different types of CNN
architectures and explain how they have been adapted for traffic sign
recognition. They also describe the various pre-processing techniques
that have been used to enhance the performance of CNNs for traffic
sign recognition.
In addition, the paper presents a comparative analysis of the
performance of different CNN architectures and pre-processing
techniques. The authors evaluate the performance of various CNN
architectures on different datasets and compare the results with those
of other state-of-the-art approaches. They also identify the limitations
and challenges of CNNs for traffic sign recognition and propose future
research directions to overcome these challenges.

Overall, the paper provides a comprehensive review of the use of

CNNs for traffic sign recognition and serves as a valuable resource for
researchers working in this field. The authors' review and evaluation
of the current state-of-the-art techniques, as well as their
identification of future research directions, provide useful insights for
improving the accuracy and efficiency of traffic sign recognition
systems.

Research objective:
The objective of this wor "Traffic Signs Classification using
Convolutional Neural Networks: A review" published in IEEE, by
Palak and A.L. Sangal is to provide a comprehensive overview and
evaluation of the use of Convolutional Neural Networks (CNNs) for
traffic sign recognition. The research objective of this paper is to
review and compare the different CNN architectures and pre-
processing techniques that have been used in the field of traffic sign
recognition, in order to identify the most effective approaches and
highlight the limitations and challenges of CNNs for this task. The
paper aims to provide insights for improving the accuracy and
efficiency of traffic sign recognition systems, and to identify future
research directions to address the current challenges in this field.
Ultimately, the objective of this research is to contribute to the
development of intelligent transportation systems that can accurately
and efficiently recognize and interpret traffic signs in real-world
scenarios.
https://ieeexplore.ieee.org/abstract/document/8310023

Literature survey 4:
The study in "Simultaneous Traffic Sign Detection and Boundary
Estimation Using Convolutional Neural Network" published in IEEE,
by Hee Seok Lee and Kang Kim, proposes a novel method for
simultaneous traffic sign detection and boundary estimation using a
Convolutional Neural Network (CNN). The authors present a CNN-
based approach that can detect traffic signs in an image and
simultaneously estimate their boundaries with high accuracy. The
proposed method is designed to work in real-time and is
computationally efficient, making it suitable for practical applications.

The literature survey presented in the paper highlights the existing

methods and approaches for traffic sign detection and boundary
estimation, including rule-based and machine learning-based
methods. The authors discuss the limitations of these methods, such
as their reliance on handcrafted features, and their inability to detect
and estimate the boundaries of traffic signs simultaneously. They then
present a detailed description of the proposed method, which consists
of a CNN that can classify and regress the traffic sign labels and their
boundary coordinates. The proposed method is evaluated on several
benchmark datasets, and the results show that it outperforms existing
methods in terms of accuracy and efficiency.

The research objective of this paper is to develop a real-time and

efficient method for simultaneous traffic sign detection and boundary
estimation using a CNN. The authors aim to address the limitations of
existing methods and demonstrate the advantages of their proposed
approach. The paper aims to contribute to the development of
intelligent transportation systems that can accurately detect and
estimate the boundaries of traffic signs in real-world scenarios, which
is crucial for improving road safety and reducing the likelihood of
accidents.
Research objective:
The research objective of "Simultaneous Traffic Sign Detection and
Boundary Estimation Using Convolutional Neural Network" by Hee
Seok Lee and Kang Kim is to propose a method for simultaneously
detecting traffic signs and estimating their boundaries in images
captured by a camera installed in a vehicle, using a deep learning
approach based on convolutional neural networks. The proposed
method aims to improve the accuracy and speed of traffic sign
detection and boundary estimation, which are essential tasks for
advanced driver assistance systems and autonomous driving. The
paper also compares the proposed method with other state-of-the-art
methods and evaluates its performance in terms of accuracy and
computational efficiency.

https://ascelibrary.org/doi/abs/10.1061/(ASCE)CP.1943-
5487.0000491

Literature survey 5:
The paper "Evaluation of Multiclass Traffic Sign Detection and
Classification Methods for U.S. Roadway Asset Inventory
Management" by Vahid Balali and Mani Golparvar-Fard aims to
review and evaluate the performance of various traffic sign detection
and classification methods for the purpose of roadway asset inventory
management in the United States. The authors provide a detailed
analysis of the existing traffic sign datasets and evaluate several state-
of-the-art methods for traffic sign detection and classification,
including the use of traditional computer vision techniques and deep
learning-based approaches. The paper also examines the impact of
various factors such as image resolution, lighting conditions, and sign
occlusion on the performance of these methods. The research
objective is to provide a comprehensive evaluation of different traffic
sign detection and classification methods and identify the most
effective approaches for roadway asset inventory management. The
findings of this study can help transportation agencies and
researchers to make informed decisions on the selection and
implementation of traffic sign detection and classification methods
for effective roadway asset management.
Research objective:
The research objective of "Evaluation of Multiclass Traffic Sign
Detection and Classification Methods for U.S. Roadway Asset
Inventory Management" by Vahid Balali and Mani Golparvar-Fard is
to compare and evaluate different multiclass detection and
classification methods for traffic sign inventory management on U.S.
roadways, with the aim of improving the accuracy and efficiency of
the asset inventory system. The study aims to address the challenges
of detecting and classifying multiple types of traffic signs, including
variations in appearance due to different lighting and weather
conditions, and to propose a reliable and efficient solution for
maintaining an accurate inventory of road signs.

3. Proposed work (Architecture Diagram)

CNN
Keras layers:

4. System model flowchart diagram

Input 1
 Input 2

Input 3

5. Results

• To accurately classify traffic signs in real-world scenarios.

• To improve the robustness of the classifier to various imaging
conditions, such as changes in lighting and weather.
• To evaluate the impact of different training strategies, such as transfer
learning, on the performance of the classifier.
➔Model was able to predict the right class for each of the 5 new test images.
Test Accuracy = 98.0%. In all cases, the model was very certain (80% - 100%)
6. Conclusion
We have been able to reach a very high accuracy rate. We can observe
that the models saturate after nearly 10 epochs, so we can save some
computational resources and reduce the number of epochs to 10. We
can also try other pre-processing techniques to further improve the
model's accuracy.. We can further improve on the model using
hierarchical CNNs to first identify broader groups (like speed signs) and
then have CNNs to classify finer features (such as the actual speed limit)
This model will only work on input examples where the traffic signs are
cantered in the middle of the image. It does not have the capability to
detect signs in the image corners.

Future work
I. Online Learning: Another challenge in TSR is that traffic sign
designs can change over time, as new signs are introduced or
existing signs are modified. Online learning techniques can be
used to update a CNN model with new data as it becomes
available, allowing the model to adapt to changes in traffic sign
designs.

II. Multi-Task Learning: In addition to recognizing traffic signs,

a CNN model could be trained to perform other related tasks,
such as detecting pedestrians, vehicles, or other objects in the
scene. Multi-task learning techniques can be used to train a
single model to perform multiple tasks simultaneously,
potentially improving overall accuracy and reducing
computation time.

III. Explainable AI: As CNNs become more widely used in safety-

critical applications like TSR, there is a growing need for models
that are transparent and explainable. Techniques such as
 attention mechanisms and saliency maps can be used to
highlight the regions of the image that the model is focusing on,
providing insight into how the model is making its predictions.

7. Limitations in the proposed work

Limited Dataset: The performance of CNN models heavily

depends on the quality and quantity of the training data. While the
German Traffic Sign Recognition Benchmark (GTSRB) dataset is a
widely used dataset for TSR research, it is still a relatively small
dataset compared to other image classification tasks. This can limit
the ability of CNN models to generalize to new, unseen data.

Computational Requirements: CNN models for TSR can be

computationally expensive to train and deploy, especially for large
datasets and complex models. This can limit their scalability and
practicality for real-time applications.

Interpretability: CNN models are often considered as "black-

box" models, meaning that it can be difficult to understand how
they make predictions. This can limit their interpretability and
transparency, which may be important for certain applications,
such as legal or ethical considerations.

Limited Contextual Information: CNN models for TSR

typically only consider the traffic sign itself and do not take into
account contextual information, such as the surrounding road
environment or the behaviour of other road users. This can limit
their ability to make accurate predictions in complex traffic
scenarios.
10.References
1. "Traffic sign recognition and analysis for intelligent vehicles" by A. de la
Escalera, J. Ma Armingol, and M. Mata.
2. "Automatic Detection and Classification of Traffic Signs" by Carlos Filipe
Paulo and Paulo Lobato Correia.
3. "Traffic Signs Classification using Convolutional Neural Networks: A review"
published in IEEE, Palak and A.L. Sangal
4. "Simultaneous Traffic Sign Detection and Boundary Estimation Using
Convolutional Neural Network" published in IEEE, by Hee Seok Lee and Kang
Kim.
5. "Evaluation of Multiclass Traffic Sign Detection and Classification Methods
for U.S. Roadway Asset Inventory Management" by Vahid Balali and Mani
Golparvar-Fard.

Reference Links:
https://www.sciencedirect.com/science/article/abs/pii/S026288560
2001567
https://ieeexplore.ieee.org/abstract/document/4279118
https://ieeexplore.ieee.org/abstract/document/9478172
https://ieeexplore.ieee.org/abstract/document/8310023
https://ascelibrary.org/doi/abs/10.1061/(ASCE)CP.1943-
5487.0000491