SPATIAL ROAD INTELLIGENCE MAPPING: ROAD CONDITION

DETECTION USING COMPUTER VISION

st nd rd th
1 Gilang Islamay Putra, 2 Izzuddin, 3 Muhammad Hanan Azhar Jihaannuriy, 4
Najib Ridho Sandika, 5thRizky Azriel Fahrezi.
1) Department of Mathematics, Faculty of Science and Technology, UIN Syarif Hidayatullah Jakarta,
South Tangerang, Indonesia {islamay.putra19@mhs.uinjkt.ac.id, zuddin.ahmd19@mhs.uinjkt.ac.id,
hanan.azhar19@mhs.uinjkt.ac.id, najib.ridho20@mhs.uinjkt.ac.id, rizky.azriel20@mhs.uinjkt.ac.id}

Abstract
One of the right steps to detect road defects and potholes is to use artificial intelligence to detect them. In recent
years, computer vision systems have proven that they can be used to automatically detect road damage when
integrated into computing devices. The various types of road defects in the real world require more creativity
and innovation than previous research. This journal uses computer vision to detect road defects or potholes by
using transfer learning, specifically You Only Look Once Representation (YOLOR) algorithm. Based on a data
set containing 665 images, mean Average Precision (mAP) of 52% was obtained and testing area for detection
is in Ciputat, Tangerang Selatan. Through this model and deployment, the infrastructure development can be
more optimal from the context of time and be more right on target.
Keywords: Computer Vision, Road Intelligence, Transfer Learning.

1. Introduction

Traffic accidents around the world are still

in the news almost every day. According to the
World Health Organization (WHO), an estimated
1.3 million people lost their lives because of road
accidents. Road accident deaths are dominated by
children and adolescents heading toward adulthood
aged 5-29 years [1]. Clearly this cannot be
underestimated. Not only do they take lives, but
traffic accidents are also an economic problem due
to medical payments and reduced productivity due
to hospitalization. In fact, the cost of these
From these accidents, 698 lives have been lost and
accidents is equivalent to 3% of gross domestic
1171 injured. The cause of most traffic accidents in
product (GDP) in most countries [1].
Indonesia is the result of human error and facilities
Traffic accidents are caused by many
are in second place. Furthermore, infrastructure
factors. Some of them include reckless or
ranked third as a cause of accidents.
unfocused driving, exceeding the speed limit, not
wearing a helmet when riding a motorcycle or seat Infrastructure as a factor supporting the safety and
belt when driving a car, and mistakes made by comfort of transportation traffic should be built and
other drivers. In addition, there is one other maintained as well as possible. Roads are one of
contributing factor to accidents that is no less them. It is common knowledge that people often
significant, namely poor road conditions. In still find damaged or potholed roads. Of course, this
Indonesia itself, traffic accidents are still quite incident greatly disrupts the smooth flow of traffic
common [2]. According to data from the National and the safety of motorists. This is especially the
consisting of 43 cases of collisions, 19 cases of case in areas that are difficult to reach by the
overturned vehicles, and 3 vehicles on fire in the government or areas that are often passed by large
range 2007-2016 vehicles, such as trucks. In addition, weather
conditions or poor-quality materials can worsen the
condition of already damaged roads. The
government is working on the best solution to
traffic accidents by tracking which roads are
damaged or have potholes [3].
 Through the Ministry of Public Works and
Housing (PUPR), the government plans to create a
tool that can automatically detect damaged roads or
potholes using artificial intelligence (AI). With the
help of AI, the work to detect early areas or
locations of road damage can be done effectively
and efficiently so that it can reduce the level of
traffic accidents.
2. Related Work
Work using computer vision so far has
always relied on the utilization of powerful real
time computing methods with a very good level of
intelligence accuracy. However, in terms of road
defect detection, most of the techniques were
originally designed to assist road inspectors with
image processing techniques (scanners). It has long
been recognized that the computer vision approach
is one of the best technologies to provide asset
management tool implementation. Notably, the
biggest obstacle to detecting and classifying road
defects is consistently achieving a high level of
classification accuracy.
The lack of road defect datasets used
results in the lack of a more comprehensive
approach. The problem has been addressed in
recent years but remains an open issue due to the
unbalanced types of asphalt structural defects
found. Despite the problem, the dataset is often
used in many studies using artificial intelligence
methods such as using simple image features and
machine models to identify road defects or
potholes. As more image sets become larger, DL-
based methods are often used to classify road
defects. Several studies aiming to improve road
defect detection and classification achieved
excellent results by utilizing the capabilities of the
architecture.
Advances in deep learning techniques
have made highly efficient and resource-limited
applications of object detection systems on mobile
devices and have been utilized in road defect
acquisition and detection systems. Such
advancements have been possible due to the use of
single-stage detectors such as YOLO. These
detectors can produce high-accuracy detections, but
they are time-consuming, making them unsuitable
for use on resource-constrained devices. In previous
road damage detection research studies, often only
produce output models without further application
stages such as being applied to a Graphical User
Interface (GUI) which in this case can be used to
provide updates on which roads have road damage
or potholes.
3. Method and Experimental Details
3.1. Data
In this research, the data is obtained from Kaggle
(https://www.kaggle.com/datasets/chitholian/annota
ted-potholes-dataset). In the The total of the data is
665 with 80% of them are for training the model
and the rest is for testing the model.
Data sample
Data is already labeled and can be accessed in
the .xml file. The label is only ‘pothole’ so that
researchers can focus only on potholes and know
which part(s) of the road is damaged.
3.2. Computer Vision
Computer vision is a field of Artificial Intelligence
(AI) that enables computers and systems to derive
meaningful information from digital images,
videos, and other visual inputs – and take actions or
make recommendations based on that information.
Computer vision needs lots of data. It runs analyses
of data over and over until it discerns distinctions
and ultimately recognizes the image. For example,
to train a computer to recognize potholes, it needs
to be a few vast quantities of pothole images and
street-related images to learn the differences and
recognize a pothole. Essential technology is used to
accomplish this, machine learning. Machine
learning uses algorithmic models that enable a
computer to teach itself about the context of visual
data. If enough data is fed through the model, the
computer will “look” at the data and teach itself to
tell one image from another. Algorithms enable
machine learning to learn by itself, rather than
someone programming it to recognize an image [4].
Transfer learning is the improvement of learning in
a new task through the transfer of knowledge from
a related task that has already been learned.
Common machine learning algorithms, in contrast,
traditionally address isolated tasks. Transfer
learning attempts to change this by developing
methods to transfer knowledge learned in one or
more source tasks and use it to improve learning in
related target tasks. Techniques that enable
knowledge transfer represent progress toward
making machine learning as efficient as human
learning [5].
YOLOR or You Only Learn One
Representation is a state-of-the-art machine
learning algorithm for object detection. YOLOR is
proposed as a “unified network to encode implicit
knowledge and explicit knowledge together”.
YOLOR using an approach to combine explicit
knowledge, defined as learning based on given data
and input, with implicit knowledge learned
subconsciously. Therefore, the concept of YOLOR
is based on encoding implicit and explicit
knowledge together, similar to how mammalian
brains process implicit and explicit knowledge in
conjunction with each other [6]. It, specifically
YOLOv3, is used to train the data to get a good
model that will be able to detect real potholes on
the road.
3.3. Model Evaluation
Researchers used mean average precision
(mAP) to evaluate the model. The mean average
precision compares the ground-truth bounding the
box to the detected box and returns a score. The
higher the score the more accurate the model is in
its detection. This journal focuses on pothole
reports in an application that will be seen by both
government and civilian users, for the government
it is hoped that it will be able to help in dealing
with existing road condition problems.
4. Result and Discussion
4.1. Experiment Setting and Preprocessing
For the early stage of the research, unzip
and gather the data based on its group. As the data
is scattered with annotated images which train and
test parts are separated with .json file, unzip the
data to get all the files. Then, training and testing
data are listed in .json file. By automating the
separation process, data has already been splitted to
training with proportion of 80% and the rest is for
testing.
4.2. Results and Analysis
After splitted the data with the total of 532
for training and 133 images with the corresponding
labels for testing, train the YOLOv3 model with 30
epochs. As the limited computing power, we have
been already achieved 52% mAP.
4.3. Application
After the model was trained and got the
desired result of evaluation, Researchers deploy the
model into a GUI. Researchers are able to detect
potholes automatically around their area without
the inconvenience of writing them down one by
one. Also the user is able to see in which road
there’s plenty of potholes to minimize the
possibility of an accident happening and contribute
by submitting picture of a potholes. Government
can too use this to get information around which
road there’s potholes to fix and improve the road
infrastructure.
For further research, gyroscope can be
used to detect the severity of a potholes rather than
pointing them each potholes at a time. Researchers
can also use heatmap to convey to the user
regarding the severity of the road itself.
A GUI is a system that uses design
principles that conform to the model-view-
controller software pattern. So, the representation
of internal and external information that is
displayed to the user. Users will only see which
functions can be used without needing to
understand the code input to process the command.
That way, interaction becomes easier. Developers
can change the appearance of the interface as freely
as possible, the important thing is that the
functionality remains the same.
The significance and benefits for the
public works and spatial planning department is
that researchers can inform the public works
department regarding potholes in the road that are
dangerous for road users and here researchers also
create an application prototype that can be used as
an application to report related points Hopefully the
community can help in reporting
For further development, from just a
prototype application, researchers hope that it can
be an effective and targeted application where the
application can function as a useful application for
equitable development in this country.
 7. References
[1]. World Health Organization “Road Traffic
Injuries”, 2022
[2] A. Dhiman and R. Klette, “Pothole Detection
Using Computer Vision and Learning”, 2019
[3] M. Famely, “Mengenal TCM, Teknologi
Perbaikan Jalan Berlubang di Indonesia,” Kompas,
2022
[4]. IBM “What is computer vision?”
[5]. L. Torrey and J. Shavlik “Transfer Learning”,
2009
[6]. V. Mell, “YOLOR - You Only Learn One
Representation”, viso.ai, 2022

5. Conclusion

In this journal, researchers apply computer

vision to detect road defects or potholes by using
transfer learning specifically YOLOR, with this
kind of approach researchers find ease to
preprocessing even though it takes heavy load for
common comercial computers however the result is
quite promising and effective. Talking about the
application of this research, researchers hope that in
the future governments in related fields can apply it
to the intended area so that they can take action in
the form of road repairs appropriately with the
future goal of better road infrastructure and the
level of traffic accidents caused by road damage or
potholes can be significantly reduced. While
technically, gyroscopes can be used to detect the
severity of a plot holes rather than pointing them at
each plot holes at a time. Researchers can also use
heatmap to convey to the user regarding the
severity of the road itself.

6. Acknowledgements
It is unfeasible to finish the project report
thoroughly without input and encouragement from
other people. First and foremost, researchers would
like to extend our sincere appreciation to all the
personages who have helped the researchers in this
endeavor. Without their active guidance, help,
cooperation, and encouragement, the researcher
would not have made headway in the project.
Ineffable indebted to Mr.Taufik Edy Sutanto,
M.Sc.Tech., Ph.D. as our lecturer who has guided
us during this research. Researchers would like to
express our gratitude to the UIN Jakarta
Mathematics Student Association for providing us
with information and supporting us in this
competition