Real-time pothole detection system on vehicle using
improved YOLOv5 in Malaysia
Au Yang Her Weng Kean Yew
School of Engineering and School of Engineering and
Physical Sciences Physical Sciences
IECON 2022 – 48th Annual Conference of the IEEE Industrial Electronics Society | 978-1-6654-8025-3/22/$31.00 ©2022 IEEE | DOI: 10.1109/IECON49645.2022.9968423
Heriot-Watt University Heriot-Watt University
Malaysia Malaysia
Putrajaya, Malaysia Putrajaya, Malaysia
w.yew@hw.ac.uk w.yew@hw.ac.uk
Abstract— Commonly in Malaysia, the road surfaces deteriorate
over time which results to potholes. With the increasing number
of potholes on the road, it has become a road hazard, potentially
harming the safety of the Malaysian citizens. Potholes generally
will cause drivers to skew away from their original direction
which may result in vehicle damages and/or road accidents.
Computer vision has improved significantly in the past decade
to improve the technologies that require image processing, such
as pothole detection. However, it has several limitations in
pothole detection as potholes have inconsistent shapes, making
it difficult to obtain an accurate prediction. Another limitation
is the speed of the detection algorithm is not able to predict the
pothole in real time. In this paper, the use of computer vision
technology on vehicles is presented as a form of solution to detect
the potholes in real time. A deep learning model based on
Convolutional Neural Networks, YOLOv5 is found to improve
the accuracy of the prediction as compared to past results. The
findings on the trained YOLOv5 model have a mAP@0.5 of 80.8
%, 82.2 % and 82.5 % on the YOLOv5m6, YOLOv5s6 and
YOLOv5n6 respectively. The trained YOLOv5n6 model was
chosen to be the main image processing model for its robust
performance to size ratio. The results were able to demonstrate
the actual deployment of the trained YOLOv5n6 model on the
road in real time with distance estimation.
Keywords—computer vision, pothole detection, convolutional
neural network
I. INTRODUCTION
. Technologies have improved human lives by several fold
over the past decade until humans are almost capable to
developing an autonomous navigation system. Although
multiple advancements in technology had been invented, there
is one flaw in technology that still bothers people till now:
pothole on road surfaces. Potholes vary in size and shape
caused by the ground water expanding and contracting. Thus,
potholes that have severe depression can cause vehicles to
skew away from the original direction, causing drivers to lose
control, resulting in vehicle damages and/or road accidents.
This problem is more prevalent in lesser developed countries
and can be found quite commonly in Malaysia. With the help
from Waze, an application for Malaysian road traffic
navigation, reported a whopping 52,295 potholes in the state
Selangor and city Kuala Lumpur alone. Besides that, the
statistics show that the total deaths caused from pothole
accidents from the year 2000 to 2011 is 840 and amounts to
11.2% of all the road deaths based on road defects [1].
Many local drivers have expressed their concerns on
potholes, where even the Malaysian Minister Khairy
Jamaluddin, who had met an accident in December 2020,
where he tripped over a pothole while cycling and had ended
Authorized licensed use limited to: Vasavi College of Engineering. Downloaded on September 22,2023 at 02:07:12 UTC from IEEE Xplore. Restrictions apply.
Pang Jia Yew Melissa Chong Jia Ying
School of Engineering and School of Engineering and
Physical Sciences Physical Sciences
Heriot-Watt University Heriot-Watt University
Malaysia Malaysia
Putrajaya, Malaysia Putrajaya, Malaysia
j.pang@hw.ac.uk w.yew@hw.ac.uk
up in a hospital for recovery treatments [2]. Within a few days
later, on the 3rd of January 2021, a 75-year-old motorcyclist
had met fatality after crashing into a pothole in Kuala Lumpur
where he flew off his motorcycle and landed 30m away from
his motorcycle [3]. With these cases, a method would be
needed to assist drivers to quickly detect potholes on road
surfaces.
Several research have been done for pothole detection.
Authors in [4] used popular semantic segmentation techniques
such as Mask-RCNN and U-Net algorithms and they yielded
an accuracy of 80.00% and 41.67% and respectively. This
shows that Mask RCNN significantly outperformed U-Net,
with especially poor performance by U-Net on images with
multiple potholes and long shot images. On the other hand,
authors from [5] have developed a stereo vision-based pothole
detection system based on a novel disparity transformation
algorithm and disparity map modelling algorithm and yielded
an accuracy of 99.64%. Disparity map modelling prevents
disparity errors of larger than one pixel which provides higher
accuracy than point cloud modelling However, this research
results in a further need for segmentation of road surfaces as
not all surfaces of the road can be treated as quadratic. The
pothole detection model in paper [6] was constructed using
“you only look once” (YOLO) algorithm based on
convolutional neural network (CNN). In this study, a pothole
image was first converted to a single channel using grey scale
to minimize loss in quality of the image and was then fed into
the YOLO algorithm. Overall, the model showed a mean
average precision (mAP) of 77.86%. On the other hand,
authors in [7] proposed a pothole detection system using a
black-box camera. The proposed algorithm consists of 3 steps:
pre-processing (cropping, greyscaling, and thresholding),
candidate extraction (detecting lanes to remove unwanted
noise) and cascaded detection (distinguish true or false
detected potholes) was able to obtain a mAP of 71%.
Although extensive research has been conducted on various
methodologies as well as the accuracy of the pothole
detection, these literatures were lacking in terms of the speed
of the detection in real time.
In our paper, the pothole detection system will be constructed
using the YOLOv5 model to build a pothole recognition
methodology. After training the model using the existing
datasets, the optimal model of the YOLOv5 is selected to be
deployed on the Nvidia Jetson Xavier X which is located on
the vehicle. The images of the moving vehicle from the Intel
Realsense D435i Camera will be given as input, then the
system will be able to detect the potholes and its distance as
the vehicle is moving in real time. The remainder of our paper
is structured as follows; Section II provides a description of
the YOLOv5 segmentation framework. Our methodology is
presented in Section III, with results and discussion in Section
IV. Finally, a summary of the primary findings is presented in
Section V.
II. FRAMEWORK
A. Model development
In this paper the CNN (Convolutional Neural Network)
model that will be used is You Only Look Once version 5
(YOLOv5) by Ultralytics where their model can be found in
GitHub. YOLOv5 is based off of Pytorch machine learning
library which can be executed in Linux-based OS using
Python 3.6 and above. The schematic architecture of
YOLOv5 can be seen in Figure 1.
Figure 1: Schematic Architecture of YOLOv5
In the initial stages of YOLOv5, it requires a Backbone, a
feature extraction network that is used to encode the
network’s input into a certain feature extraction. In the use
case of YOLOv5, it uses the DarkNet as its backbone where
it displays information of the classified images and prints the
top 10 classes of the images as it loads the configuration file
and weights. This information then gets passed to the Path
Aggregation Network (PANet) which is the neck of
YOLOv5. PANet framework first shortens the information
path and enhances the feature pyramid by creating bottom-up
path augmentation [8]. Then it recovers the broken
information path between all feature levels through the
adaptive feature pooling; it aggregates features from all
feature levels while avoiding expeditiously assigned results
[8]. At the end of PANet it captures different views of each
proposed layers by augment mask prediction [8]. Figure 2
shows the sequence for PANet framework.
Figure 2: PANet Framework: (a) Feature Pyramid Network
(FPN). (b) Bottom-up Path Augmentation. (c) Adaptive
Feature Pooling. (d) Box Branch. (e) Fully connected Fusion
At the output side of YOLOv5, it displays the class
approximation by the convolutional layers with its bounding
boxes on the detected objects.
Authorized licensed use limited to: Vasavi College of Engineering. Downloaded on September 22,2023 at 02:07:12 UTC from IEEE Xplore. Restrictions apply.
In addition, YOLOv5 has 5 different models to choose from
with YOLOv5x6 (extra-large) being the most accurate model
but requires more graphical computational cost and being the
biggest in size while YOLOv5n6 (nano) is the least accurate
but requires less graphical computational cost and smallest in
size. The statistics and performances of each model can be
found in Figures 3 & 4. These model weights were pretrained
using COCO dataset that contains 5000 images of 80 classes
of identification with 300 epochs of training cycles.
Figure 3: YOLOv5 Models and their Performances[9]
Figure 4: YOLOv5 Graphical Computational
Performances[9]
B. Image Dataset for Labelling
The pothole image datasets were taken from RTK, Kaggle,
Pothole-600, Roboflow and PotDataset datasets which
amounts to a total of 3784 images of potholes. For the best
outcome and accuracy, the images were split into a ratio of
7:3 where the train folder contains 70 % of the entire dataset
collection and the test folder contains 30 % of the dataset
collection.
III. METHODOLOGY
A. Model training
Due to the NVIDIA Jetson Xavier NX’s 384 CUDA cores, in
this paper, the following three models of YOLOv5:
YOLOv5m6, YOLOv5s6 and YOLOv5n6 were trained. This
is because they provide a balance of performance to
computational cost ratio making it more efficient for
deployment. Google Colab Pro was used for the model
training with GPU ‘Tesla P100-PCIE-16GB’ and RAM of
32GB.
B. Hardware integration with YOLOv5 Table 1 - Tabulation of Trained Models Pothole Detection
The hardware architecture is shown in Figure 5. An Intel Performances
Realsense D435i camera will be used for its stereoscopic Pothole Detection Performances
functions, which allows for 3D calculations and perspective
on distance estimation. The camera is connected to an Nvidia Models Precision (%) Recall (%) mAP@0.5 (%) mAP@0.5:0.95 (%)
Jetson Xavier NX supercomputer which holds and operates
the AI algorithm in detecting potholes and the distance of the YOLOv5m6 84.2 71.3 80.8 53.9
pothole from the vehicle. The Nvidia Jetson Xavier NX
YOLOv5s6 80.7 75.4 82.2 54.7
consumes 30 W of power with specs of 8 GB RAM and 384
CUDA cores (GPU) which are crucial for the use of AI YOLOv5n6 81.9 74.6 82.5 53.8
computations.

From Table 1, the trained m6 model of the YOLOv5 has the

Figure 5: Pothole detection hardware architecture
highest precision while surprisingly the trained n6 model has
the highest mAP@0.5%. In terms of graphical computational
cost, the trained n6 model requires the least according to
Figures 3 & 4. Thus, the n6 model provides the best precision
to performance ratio and is selected as the use case for the
pothole detection. To demonstrate the performance of the
trained YOLOv5n6 model, Google Colab was used with the
trained model The image size was set to 1152 pixels and the
‘conf’ or confidence/threshold was set to 0.30 or 30%. The
results are shown in Figure 7.

The Nvidia Jetson Xavier NX is loaded with JetPack 4.4 SDK

as specified in their “Getting Started with Jetson Xavier NX
Developer Kit” site from Nvidia [10]. The home directory of
the Nvidia Jetson Xavier NX will house the libraries and
dependencies for the YOLOv5 to run: numpy, pandas, scipy,
scikit-image, matplotlib, seaborn, opencv-python, torch,
torch-vision and cython .

Once the configuration for YOLOv5 is done, the drivers for

the Intel Realsense D435i camera is installed for the camera
to function with YOLOv5. The folder that contains the shell
script for the installation of RealSense SDK was downloaded
into the Nvidia Jetson Xavier NX and ran in the terminal. The
flow chart of the proposed YOLOv5 Instance Segmentation
is shown in Figure 6. (a)

Figure 6: Flow chart of the proposed YOLOv5 Instance

Segmentation

IV. RESULT & DISCUSSION (b)

A. Model training performance Figure 7: Results of The Detection in

The following show the validation results by their precision,
recall rate and mean average precision over IoU (mAP@0.5
& mAP@0.5:0.95) upon finishing the model training. Since
there were three models that were trained; YOLOv5m6,
YOLOv5s6 and YOLOv5n6, their results are shown in Table
1.
Google Colab (a) Before and (b) After
B. Model deployment on hardware
After getting the results on the performance of the trained
model, the model would then be deployed into the Nvidia
Jetson Xavier NX to see how it performs on the device itself.

Authorized licensed use limited to: Vasavi College of Engineering. Downloaded on September 22,2023 at 02:07:12 UTC from IEEE Xplore. Restrictions apply.
The Nvidia Jetson Xavier Nx and the Intel Realsense D435i
camera were installed on-board a vehicle; a Myvi car and
were powered through a car inverter with an output power of
220W as can be seen in Figure 8. The vehicle was deployed
and tested in the neighbourhood of Casawood Cybersouth,
Malaysia.

(c)

Figure 8: Pothole detection hardware architecture

The trained model n6 was executed in the directory of the

YOLOv5 folder as the necessary files are located there. After
executing the code in terminal, the image processing
commences, and the terminal shows the inference time for
each frame that it processes which showed an average of
0.040 s or 40 ms per frame which amounts to about ~25 FPS
(Frames Per Second). Figures 9 (a) – (d) shows the model
detection using the on-board Intel Realsense D435i camera
with its distance measurements of the pothole.

(d)
Figure 9(a)-(d): Results of the Pothole Detection System
on Vehicle in real-time

While the YOLOv5n6 was running, the distance accuracy

measurements were taken down for future reference on
distance estimation. The model when using the Intel Real
Sense D435i camera, has an accurate detection at a distance
of 1.5 m onwards whereas a distortion between 0-1 m. In a
brightly lit condition, the model has an accurate detection up
to 15 m whereas in low light condition the accuracy of the
detection is valid up to 12 m.
(a)
However, there were some imperfections observed on the
YOLOv5n6 model during the pothole detection. As shown in
Figure 9 below the model detected manhole as pothole. The
manhole detected can be accepted as manholes can also cause
accidents like potholes. The detection also accounts for the
severity and depth of the pothole in the images i.e. how wide
the potholes are, dark colour characteristics of potholes which
would perceive the depth of the pothole and so on as the
image datasets used had severe depression and wide surface
areas of potholes. If the manhole is flushed to the surface of
the road the model would not detect it as a pothole.

(b)

Authorized licensed use limited to: Vasavi College of Engineering. Downloaded on September 22,2023 at 02:07:12 UTC from IEEE Xplore. Restrictions apply.

Figure 10: Manhole detected of the
Pothole Detection System
In addition, the model also detected certain parts of trees as
potholes and even detected shadows as potholes. The
detected trees and shadows are based on the angle at which
the camera was mounted and the intensity of light which may
off-balance the white balance of the camera. This off-balance
could make the trees and shadows look like potholes to the
model as shown in Figure 11. Overall, the model had done a
good job at detecting potholes with its mAP@0.5 of 82.5%.
Figure 11: Part of trees detected of the Pothole Detection
System
V. CONCLUSION
This paper primarily shows the work process done on the
development of pothole detection system. The pothole
detection methodology was based off of YOLOv5 CNN
model that has included its pre-processing backbone DarkNet
and its neck PANet which essentially entraps the object
detection and use convolutional methods to achieve the image
processing model. The findings on the trained YOLOv5
model have a mAP@0.5 of 80.8 %, 82.2 % and 82.5 % on the
YOLOv5m6, YOLOv5s6 and YOLOv5n6 respectively.
Authorized licensed use limited to: Vasavi College of Engineering. Downloaded on September 22,2023 at 02:07:12 UTC from IEEE Xplore. Restrictions apply.
The trained YOLOv5n6 model was chosen to be the main
image processing model for its robust performance to size
ratio which perfectly suited the mobility on deploying it and
lack of GPU cores in the Nvidia Jetson Xavier NX. Following
that, the detection model was then integrated with the Intel
Realsense D435i Stereo Camera to provide images as the
input. The devices on-board the vehicle were installed and
tested in a household area called Casawood Cybersouth
located in Dengkil, Malaysia.
The results were able to demonstrate the actual deployment
of the trained YOLOv5n6 model on the road in real time with
distance estimation. However, the results also showed some
minor errors as it detects manholes, trees and shadows as
potholes which is due to the resemblances with vast images
of potholes in the image dataset. It is recommended that
further research can be done to enhance the image processing
particularly during night-time and increase detection
performance on long distance object detection.
ACKNOWLEDGMENT
This work is part of the ‘Road Obstacles and Pedestrian
Awareness System’ (ROPAS) project and funded by the
Empower Grant Scheme by Heriot-Watt University
Malaysia.
REFERENCES
[1] Y. Darma, M. R. Karim, and S. Abdullah, “An analysis of
Malaysia road traffic death distribution by road environment,”
Sadhana - Academy Proceedings in Engineering Sciences, vol. 42,
no. 9, pp. 1605–1615, Sep. 2017, doi: 10.1007/s12046-017-0694-
9.
[2] “Malaysia’s Minister Khairy Jamaluddin injures from fall after
bicycle hits pothole,” The Straits Times, Dec. 28, 2020. Accessed:
May 31, 2022. [Online]. Available:
https://www.straitstimes.com/asia/se-asia/malaysias-minister-
khairy-jamaluddin-injured-from-fall-after-bicycle-hits-pothole
[3] K. Perimbanayagam, “75-year-old man killed after crashing into
pothole,” New Straits Times, Jan. 03, 2021. Accessed: May 31,
2022. [Online]. Available:
https://www.nst.com.my/news/nation/2021/01/654169/75-year-
old-man-killed-after-crashing-pothole
[4] S. Thiruppathiraj, U. Kumar, and S. Buchke, “Automatic pothole
classification and segmentation using android smartphone sensors
and camera images with machine learning techniques,” in 2020
IEEE REGION 10 CONFERENCE (TENCON), Nov. 2020, pp.
1386–1391. doi: 10.1109/TENCON50793.2020.9293883.
[5] R. Fan, U. Ozgunalp, B. Hosking, M. Liu, and I. Pitas, “Pothole
Detection Based on Disparity Transformation and Road Surface
Modeling,” IEEE Transactions on Image Processing, vol. 29, pp.
897–908, 2020, doi: 10.1109/TIP.2019.2933750.
[6] J. W. Baek and K. Chung, “Pothole classification model using
edge detection in road image,” Applied Sciences (Switzerland),
vol. 10, no. 19. MDPI AG, Oct. 01, 2020. doi:
10.3390/APP10196662.
[7] Y. Jo and S. Ryu, “Pothole detection system using a black-box
camera,” Sensors (Switzerland), vol. 15, no. 11, pp. 29316–29331,
Nov. 2015, doi: 10.3390/s151129316.
[8] S. Liu, L. Qi, H. Qin, J. Shi, and J. Jia, “Path Aggregation Network
for Instance Segmentation,” Mar. 2018.
[9] GitHub, “GitHub - ultralytics/yolov5.”
https://github.com/ultralytics/yolov5 (accessed May 31, 2022).
[10] NVIDIA Developer, “Getting Started With Jetson Xavier NX
Developer Kit.”
https://developer.nvidia.com/embedded/learn/get-started-jetson-
xavier-nx-devkit (accessed May 31, 2022).