Deep Learning Approach for U.S.
Traffic Sign Recognition
Emmanuel B. Nuakoh, Kaushik Roy, Xiaohong Yuan, Albert Esterline
Department of Computer Science
North Carolina A&T State University
1601 E Market St
Greensboro, NC 27411
ebnuakoh@aggies.ncat.edu; {kroy, xhyuan, esterlin}@ncat.edu
ABSTRACT
Advanced Driver Assistant Systems (ADAS) have seen massive
improvements in recent years; from detecting pedestrians, road
lanes, traffic signs and signals, and vehicles to recognizing and
tracking traffic signs. Traffic sign recognition systems are used to
detect and classify the traffic signs. This research is focused on
the classification aspect of the ADAS; i.e. identifying the class a
traffic sign belongs to. Most of the current ADAS that use the
U.S. traffic signs are limited to speed limit signs recognition only.
This work seeks to expand the corridors of U.S. traffic signs
recognition to cover all the publicly available classes. The
research adopts the VGGNet architecture modified to classify
U.S. traffic signs provided by the LISA TS benchmark. The
original VGGNet was used to classify The German Traffic Sign
Recognition Benchmark and reported an accuracy of 98.7%. This
work recorded a validation accuracy of 99.04%.
CCS Concepts
• Computing methodologies➝Supervised learning by
classification; Neural networks
Keywords
Traffic Sign Recognition; Convolutional Neural Network;
Deep Learning; Deep Neural Networks; Computer Vision
1. INTRODUCTION
Traffic sign recognition systems (TSRS) are concerned with two
activities; detecting and recognizing traffic signs in an image
frame [1]. Detecting or localizing the traffic sign includes finding
the color and shape in an image frame. Recognizing a sign has to
do with classifying the detected sign into a given type of traffic
signs. This work is focused on the recognition aspect of the TSRS.
Several researches have been conducted in the area of traffic sign
recognition using publicly available datasets, such as The Belgian
Traffic Sign Classification (BTSC) dataset [2], the German
Traffic Sign Recognition and Detection Benchmark (GTSRB and
GTSDB) [3], the Croatian traffic sign dataset (rMASTIF) [4], and
the Tsinghua-Tencent 100 K benchmark [5]. However, limited
research has been conducted on US traffic signs [6].
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. Copyrights
for components of this work owned by others than ACM must be
honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior
specific permission and/or a fee. Request permissions from
Permissions@acm.org.
ICDLT 2019, July 5–7, 2019, Xiamen, China
© 2019 Association for Computing Machinery.
ACM ISBN 978-1-4503-7160-5/19/07…$15.00
https://doi.org/10.1145/3342999.3343016
47
Several methods have shown success in TSRS. Convolutional
Neural Networks (CNN) is one of the methods that has yielded
high success in classifying traffic signs [5]. These deep learning
methodologies have been tested exhaustively on European traffic
signs but have not yet been adopted for classifying U.S. traffic
signs. This work is an application of the VGGNet proposed by [7]
to classify U.S. traffic signs beyond speed limit signs to
incorporate all signs contained in the LISA TS dataset [8].
1.1 Problem Statement and Hypothesis
The application of TSRS in ADAS has seen some commercial
success with limited functionality [4]. The limitation is not only
linked to the number of supported traffic signs, but also the areas
of the road network where they are most effective. The lack of
standardized datasets for training and testing models pose another
problem in this area of research, making it quite difficult to
compare the performance of different models.
Unbalanced traffic sign datasets also present another layer of
difficulty for developing TSRSs. Machine learning depends on
huge amount of data for training the algorithm to perform better
on unseen data, however, most traffic sign datasets are porous and
need to be augmented in other ways to balance them.
1.2 Contribution
The main contribution of this work is to use an existing deep
learning architecture and modify it to classify the U.S. traffic
signs presented in LISA TS benchmark. Limited work has been
done using U.S. datasets and no known work has been reported
using a deep learning technique such as CNN for classifying this
dataset. This research is an initial step to develop a robust CNN
for classifying the whole U.S. traffic sign dataset in the LISA TS
benchmark.
2. LITERATURE REVIEW
Traffic sign recognition research has gained a lot of traction in
recent years. Various methods with wide applicability for image
recognition have proven that they can be used to recognize traffic
signs as well [9].
Research into TSRS was boosted since several of these datasets
are commonly used to evaluate the performance of computer
vision algorithms for traffic sign detection and recognition. Deep
learning for image recognition has seen wide applicability in
biometrics, medical diagnostics, text categorization, speech
recognition and traffic sign recognition, just to name a few.
According to [13] conventional machine learning has limitation
with processing the raw data. These are representation-learning
based classifications. Each layer of representation signifies the
presence or absence of parts of objects such as edges and motif.
Subsequent layers detect objects as combinations of identified
parts.
2.1 Traffic Sign Recognition with Deep
Learning
[10] developed a committee of CNN and a multilayer perceptron
(MLP) that were trained on HOG features (HOG3). They trained
the best architecture initialized with uniformly random
distribution weights with a hyperbolic tangent activation function.
The classification had a combined classification rate (CCR) of
98.98%, outperforming humans in some instances as reported by
[11] in the GTSRB competition. In the same report, [12] placed
second with 98.97% accuracy using a multi-layer ConvNets with
more sophisticated non-linearities. An improvement to [10] using
CNN with data augmentation and jittering and achieved an
accuracy of 99.46% [13].
[14] proposed a Hinge Loss Stochastic Gradient Descent
(HLSGD) cost function method for training CNNs. This function
is similar to Support Vector Machines (SVM) hinge loss and
performed faster than the Stochastic Gradient Descent (SGD),
which is preferred for training CNN. They achieved an accuracy
of 99.65% beating the state-of-the-art approach of 99.46%
reported in [13, 15].
[4] developed OneCNN, a convolutional neural network, inspired
by [12]. They developed a single network that is deeper and more
complex but less computationally costly, and used it to classify
multiple datasets, particularly, GTSRB, BTSC, and introduced a
new dataset called rMASTIF. They achieved an accuracy of
99.11% as against the state-of-the-art of 99.65% for the GTSRB
[14]; 98.17% for BTSC against the state-of-the-art [15] with
98.77% and; recorded 99.53% for rMASTIF. Their work was only
related to the classification aspect of the TSRS as does this
research.
2.2 Traffic Sign Recognition with Traditional
Machine Learning
[16] used SVM with Gaussian kernels to recognize traffic signs
from blobs which have been categorized into shape classes. To
test the effect of occlusion on recognition, an occlusion mask was
placed on the images. Small-, medium- and largesized masks
reported 93.24%, 67.85% and 44.90% probabilities of
successfully recognizing the signs respectively. An observation
made was that a large-sized occlusion mask placed in the middle
pictoram’s inner area, showed the worst performance during
recognition. [17] applied an SVM for classifying traffic signs on
the GTSRB dataset. They combined local binary pattern (LBP),
HOG and GABOR for feature extraction. For LBP alone, the
performance was 93.36%, GABOR alone recorded a performance
of 93.90% and HOG alone recorded 94.56%. A combination of all
three yielded the best performance of 97.04%. HOG and GABOR
together had a performance of 97.00%; close to the performance
of all three combined. Their proposed algorithm was ninth overall
compared to the results obtained in the 2011 GTSRB competition;
however, it had the best performance for “Other Prohibitions”,
and “Mandatory”, categories scoring 99.86%, and 99.83%,
respectively. EBLearn 2LConvNet, and CNN HOG3 were the
previous best performers in those categories scoring 99.80%, and
97.89, respectively.
3. METHODOLOGY AND EXPERIMENTS
3.1 Approach
CNN has made a lot of stride in the image recognition space in
recent times. It gained recognition in the traffic sign recognition
space during “The German Traffic Sign Recognition Benchmark”
48
(GTSRB) competition, where some of the best methods presented
used CNN for classification [18].
There is limited research on traffic sign detection and recognition
using US traffic signs. [6] used an R-CNN algorithm to detect US
traffic signs and showed some good result on LISA-TS Extension
dataset [19]. The classification was based on the speed limit super
class alone.
Our research tries to extend the boundaries of the current research
to include all the signs in the dataset by replicating work done by
[12] adopted for classifying the U.S. traffic signs dataset.
3.2 Experimental Setup
The experimental setup consists of two main parts. Firstly,
preparing the data, that spans all the tools used to extract and
separate traffic signs into their respective classes. Secondly, the
extracted data is fed into a deep neural network for classification.
The LISA TS dataset is a publicly available traffic sign dataset for
the United States [8]. This dataset offers the researchers, who are
interested in classifying U.S. traffic signs, an avenue to train and
test their models. The LISA TS dataset was selected for this work
because there is a limited work has been done on U.S. traffic
signs.
3.2.1 Preparing the dataset.
The LISA TS dataset comes with a set of Python tools for
extracting traffic sign images from annotated frames. Since this
research is concerned with the classification aspect of TSRS, these
tools were useful. There are 47 classes of traffic signs that are
named using the convention: “0000”, “0001”, …, “0045” and
“0046”. An 80/20 split sets for Training and Testing were
obtained using the Python script provided as “split1.csv” and
“split2.csv”, respectively. Traffic sign images under each split set
were cropped and copied to their respective class folder. These
class folders were then manually copied into two new folders
called “Training” and “Testing” as training and test sets,
respectively.
3.2.2 Exploring the dataset.
The LISA TS dataset consists of 6610 frames with 7855
annotations [8]. The sizes of the traffic sign images range from as
little as 6×6 to 167×168 pixels. The cameras used for image
collection had resolutions ranging from 640×480 to 1024×522
pixels with certain images being greyscale while others were color
images.
The extracted signs were split into training and test sets with
80/20 split ratio respectively. Fig. 1 shows the distribution of
traffic signs for each class. The most populated classes are
“SpeedLimit40”, “doNotPass”, and “rampSpeedAdvisory50” in
an order of decreasing magnitude. The least occurring traffic signs
in the dataset are “rampSpeedAdvisory20”, “speedLimitUrdbl”,
“thruMergeLeft”, and “turnLeft”.
Fig. 2 shows samples of traffic sign images in each class. The
images shown are the first images in that class. It should be noted
that several of these signs have numbers showing speed limits of
some sort. This raises concern for the performance of the model
especially with the limited samples in each class.

Fig. 1. Training Data Distribution per Class for Each Traffic
Sign
3.2.3 Deep Neural Network (DNN) Model
Architecture.
The VGG Layer has a batch normalization layer for faster and
better training, a param ReLu layer for solving dead linear
rectifier issue during training, convolution layer with parametric
relay activation using Xavier Scheme for weights and biases
initialization, fully connected layer with fully connected dense
layers also using Xavier Scheme for weights and biases
initializations, and a max pooling operation as its basic elements.
The spatial transformer layer consists of a localization and an
affine transformation layer. This layer consists of a 5x5
Convolution filter, followed by a 3x3 Convolution filter, a 1x1
Convolution Filter, 128 Dense layer, 64 Dense layer and 6
Identity transformers.
The VGG Network (Fig. 3) is made up of 2 back to back
Convolutions of 2x2 kernel size and a stride of 2. The layer has 1
pooling layer followed by a dropout layer. The model is made up
of 4 VGG layers. The first layer extracts 32 feature maps from an
input layer of 32x32x3 features while the second layer extracts 64
feature maps from a 16x16x32 input, with the third layer
extracting 128 feature maps from an 8x8x64 input and the fourth,
256 feature maps from an input of 4x4x128. The model also has 3
Fully Connected layers with the 1024 hidden layers that map to
512 hidden layers that further map to the 47 classes in the dataset.
Figure 2. Sample Image in Each Class
Model building params used include: Xavier Initialization as a
tuning methodology; early stopping to restore previous checkpoint
if accuracy gains in testing does not meet current requirements;
batch size of 500, learning rate of 5e-5, regularization factor of
1e-5 were chosen as best choice for hyper params after searching
over 500 Epochs; Adam Optimizer was used for optimization.
This implementation is a modified version of VGGNet1. 8 GPUs
1 https://github.com/vamsiramakrishnan/TrafficSignRecognition
49
each of memory size of 4GB, installed on a single system were
used to allow multiprocessing of the computations during training.
4. RESULTS
The original research recorded an accuracy of 98.7% on the
GTSRB dataset. This work recorded a training accuracy of
99.02% and a validation accuracy of 99.04% on LISA TS dataset.
The model was trained over 600 epochs. The confusion matrices
of the 600th epoch are presented in Fig. 4 for normalized and non-
normalized samples. 15 out of the 1567 images were wrongly
classified. One image in the “keepRight” class causes a confusion
to the model as “zoneAhead45” 1% of the time. One
“pedestrainCrossing” sign amounting to 1% of the class was
misclassified as “stop”. “rampSpeedAdvisoryUrdbl” causes
confusion to the model by classifying as “school” 100% of the
time; only one sign is in this class. One “rightLaneMustTurn” sign
making up 6% of the class misclassifies as “stop”.
“speedLimit25” is wrongly classified 2% of the time as
“speedLimitUrdbl”, which stands for unreadable speed limits.
“speedLimit55” is misclassified as “truckSpeedLimit55” 100% of
the time. “speedLimit65” is misclassified 6% of the time as
“stop”. “speedLimitUrdbl” represents speed limit signs that were
unreadable to be correctly classified by human. This class was
classified 8% of the time as “speedLimit35” and 4% of the time as
“speedLimit50”. Finally, “yield” misclassified 8% of the time as
“stop”.
Fig. 5 shows plots of accuracy (left) and loss (right) during
training and validation respectively. It was noted that the model
could have performed better if it had been trained a little longer.
The model does not overfit since the validation accuracy is less
than the training accuracy throughout the whole process. The
model shows a loss of 7.03% during training and 8.8% during
validation. The original VGGNet showed a loss of 7.5% during
validation, no value was recorded for training.
5. CONCLUSION AND FUTURE WORK
ADAS helps drivers with the perception in a complex
environment with a lot of background noise. A computer is rather
trained to pick up things that may escape the human eye. This
research seeks to investigate using the VGGNet to recognize U.S.
traffic signs. Previous work done with the same dataset, but with
the speed limits signs only scored an accuracy of 95.7%, a little
under the 97% reported by [6]. This research shows an overall
accuracy of 99.04% during validation.
Figure. 3. VGGNet Architecture1
With the LISA TS dataset being limited and porous, only training
and validation were performed. In future, the LISA TS extension
would be employed for testing the model. Future work would be
extended to detect traffic signs in video frames in real time.
Temporal Convolutional Networks (TCN) would be investigated
whether it can be used to improve the speed of the model. It has
been used by [20] for detecting human action from video frames
but no known work has been done using TCN for traffic sign
recognition.
The dataset would be experimented on modified LeNet, AlexNet,
and CUDA ConvNet. The GTSRB and BTSC would be used to
test the model performance and compare the algorithm to other
standardized algorithms.
Figure 4. Confusion Matrix of Model after 600 Epochs with
and without Normalization
Figure 5. Plot of Accuracy and Loss during Training and
Validation
6. REFERENCES
[1] Escalera, A. D. L., Moreno, L., Salichs, M. A., and
Armingol, J. M. 1997. Road traffic sign detection and
classification.
[2] Timofte, R., Zimmermann, K., and Van Gool, L. 2014.
“Multi-View Traffic Sign Detection, Recognition, and 3D
Localization”. Machine vision and applications, 25(3), 633-
647.
[3] Stallkamp, J., Schlipsing, M., Salmen, J., and Igel, C. 2011.
“The German traffic sign recognition benchmark: a multi-
class classification competition”. In Neural Networks
(IJCNN), The 2011 International Joint Conference on (pp.
1453-1460). IEEE.
[4] Jurišić, F., Filković, I., and Kalafatić, Z. 2015. “Multiple-
Dataset Traffic Sign Classification with OneCNN”. In
Pattern Recognition (ACPR), (2015, November) 3rd IAPR
Asian Conference on (pp. 614-618). IEEE.
[5] Zhu, Z., Liang, D., Zhang, S., Huang, X., Li, B., and Hu, S.
2016. “Traffic-Sign Detection and Classification in The
Wild”. In Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition (pp. 2110-2118).
[6] Li, Y., Møgelmose, A., and Trivedi, M. M. 2016. “Pushing
the “Speed Limit”: High-Accuracy US Traffic Sign
Recognition with Convolutional Neural Networks”. IEEE
Transactions on Intelligent Vehicles, 1(2), 167-176.
50
[7] Simonyan, K., and Zisserman, 2015. “A. Very deep
Convolutional Networks for Large-Scale Image
Recognition”. In International Conference on Learning
Representations (ICLR).
[8] Møgelmose, A., Trivedi, M. M., and Moeslund, T. B. 2012.
"Vision based Traffic Sign Detection and Analysis for
Intelligent Driver Assistance Systems: Perspectives and
Survey," IEEE Transactions on Intelligent Transportation
Systems.
[9] Mathias, M., Timofte, R., Benenson, R., and Van Gool, L.
2013. “Traffic Sign Recognition—How Far Are We from
The Solution?”. In Neural Networks (IJCNN), The 2013
International Joint Conference on (pp. 1-8). IEEE.
[10] Cireşan, D., Meier, U., Masci, J. and Schmidhuber, J. 2011.
"A committee of neural networks for traffic sign
classification," The 2011 International Joint Conference on
Neural Networks, San Jose, CA, 2011, pp. 1918-1921.
[11] Stallkamp, J., Schlipsing, M., Salmen, J., & Igel, C. (2011,
July). The German Traffic Sign Recognition Benchmark: A
multi-class classification competition. In IJCNN 6, 7.
[12] Sermanet, P., and LeCun, Y. 2011. “Traffic Sign Recognition
with Multi-Scale Convolutional Networks”. In Neural
Networks (IJCNN), The 2011 International Joint Conference
on (pp. 2809-2813). IEEE.
[13] CireşAn, D., Meier, U., Masci, J., and Schmidhuber, J. 2012.
Multi-column deep neural network for traffic sign
classification. Neural networks, 32, 333-338.
[14] Jin, J., Fu, K., and Zhang, C. 2014. "Traffic Sign Recognition
with Hinge Loss Trained Convolutional Neural Networks".
In IEEE Transactions on Intelligent Transportation Systems,
vol. 15, no. 5, pp. 1991-2000, Oct. 2014.
[15] Zhu, Y., Wang, X., Yao, C., and Bai, X. 2013. Traffic sign
classification using two-layer image representation. In IEEE
International Conference on Image Processing (pp. 3755-
3759). (2013, September). IEEE.
[16] Maldonado-Bascón, S., Lafuente-Arroyo, S., Gil-Jimenez, P.,
Gómez-Moreno, H., and LópezFerreras, F. 2007. “Road-sign
Detection and Recognition Based on Support Vector
Machines”. IEEE transactions on intelligent transportation
systems, 8(2), 264-278.
[17] Berkaya, S. K., Gunduz, H., Ozsen, O., Akinlar, C., and
Gunal, S. 2016. “On circular traffic sign detection and
recognition”. Expert Systems with Applications, 48, 67-75.
[18] Stallkamp, J., Schlipsing, M., Salmen, J., and Igel, C. 2012.
“Man vs. computer: Benchmarking Machine Learning
Algorithms for Traffic Sign Recognition”. Neural Networks,
Volume 32, 2012, Pages 323-332.
[19] Møgelmose, A., Liu, D., and Trivedi, M. M., 2014. “Traffic
Sign Detection for U.S. Roads: Remaining Challenges and a
Case for Tracking,” IEEE Intelligent Transportation Systems
Conference, (ITSC2014), Oct. 2014.
[20] Lea, C., Flynn, M. D., Vidal, R., Reiter, A., and Hager, G. D.
2017. "Temporal Convolutional Networks for Action
Segmentation and Detection," 2017 IEEE Conference on
Computer Vision and Pattern Recognition (CVPR),
Honolulu, HI, 2017, pp. 1003-1012.