2019 IEEE 9th International Conference on System Engineering and Technology (ICSET), 7 October 2019, Shah Alam, Malaysia
Watershed-based Real-time Image Processing
for Multi-Potholes Detection on Asphalt Road
Tran Duc Chung
Computing Fundamental Department Faculty of Engineering Technology & Built Environment
FPT University
Hoa Lac Hi-Tech Park, Hanoi, Vietnam
chungtd6@fe.edu.vn
Abstract—This paper presents a real-time watershed-based
algorithm for detecting multiple potholes on asphalt road surface.
The algorithm uses (i) inverted binary in combination with Otsu
thresholding techniques to find the optimal threshold value of
the image in an inverted color space; then (ii) morphological
technique with open, then close kernels to filter small noises and
bold pothole edges on the image; and (iii) distance transform
for finding markers on the pre-watershed-phase image before
applying the watershed algorithm. As a result, the algorithm
achieves real-time processing speed of approximately 33.1 frame-
per-second (fps). Based on the tested images, it is evident that
the algorithm can be used for detecting effectively potholes with
different sizes and structures on three types of road surfaces
namely smooth, aged, and degraded ones.
Index Terms—multi-pothole, detection, watershed, real-time,
automobile
I. I NTRODUCTION
In recent years, the development of automotive industries
has resulted in the mass deployment of automobiles on roads
in both developing and developed countries. Although the
quality of automobiles nowadays has been improved enor-
mously, there are still low-cost models that operate on roads of
developing countries. The reason is because expensive models
are only attracted by the top earners in a region, while the mid-
range citizens can only afford low-cost automobiles as a mean
of comfort transportation. In order to support automobiles in
reducing accidents [1]–[4] by hitting potholes on roads, this
research aims to developed an improved real-time watershed-
based pothole detection algorithm. The algorithm supports
fully automated detection of various types of pothole shapes
and sizes on different road surfaces such as smooth, aged, and
degraded ones.
The remaining of this paper is organized as follows: Section
II details related researches in pothole detection in recent
years. Section III presents the proposed algorithm. Section IV
discusses results and presents the analysis. Finally, Section V
presents conclusions and some suggestions for future works.
II. R ELATED W ORKS
The development and broaden usage of automobile for
transportation has helped to boost up the research activities in
pothole detection recently [5]. In order to support automobile
to operate effectively and safely on different road conditions,
978-1-7281-0758-5/19/$31.00 ©2019 IEEE 268
M. K. A. Ahamed Khan
UCSI University
Kuala Lumpur, Malaysia
mohamedkhan@ucsiuniversity.edu.my
pothole identification is currently of focus. Typically, on mid-
range automobile, pothole detection using a single camera
is used. This is because using dual camera (stereo cameras)
[5], [6] or laser-based object identification technique results in
much more expensive automobile which is then un-affordable
for mid-range people. Thus, the majority of automobiles in this
range are using single camera for detection of front objects.
In [5], for road fitting purpose, an approximation technique
using bi-square weighted robust least-squares was presented.
Pothole can be detected by its size and depth, however, its
limitation when using stereo cameras is that it requires cameras
calibration before use. In addition, the pothole detection alone
took approximately 1.6s to perform which is not yet suitable
for real-time application.
In [7], a support vector machine (SVM)-based crowd-
sourced system was developed for localizing potholes on
multi-lane environments. The data for detecting potholes came
from accelerometer sensor data embedded on the vehicle. The
main disadvantage of this approach is that it has detection
accuracy of about 90% while processing time for applying the
algorithm was not discussed. Pothole detection using Wi-Fi
system locating at roadsides was proposed in [8]. However, the
challenges with system running Wi-Fi is at its low reliability
and point-to-point instead of mesh connection [9]–[11] while
vehicle is moving on the road at high speed. In [12], a
cost-effective cloud-based pothole detection algorithm was
proposed. Different from the aforementioned researches, the
authors proposed the use of ultrasonic sound for identifying
potholes and humps and their depths and heights on road.
However, the main drawback of the system is that the pro-
cessing time of the algorithm was not discussed. In addition,
the ultrasonic sound sensor has limitation in detection range,
typically within a couple of meters, thus it may not be effective
in detecting the potholes in advanced, especially when vehicle
is moving at high speed.
In [13], a pothole detection technique utilizing dark region,
round shape and rugged texture was developed. The developed
system was running in parallel. It used super pixel, wavelet
energy field and differential techniques for detecting the pot-
holes on road. As a result, it is possible to achieve 10-15
fps processing speed with 2 Megapixel captured images using
core i7 Windows system. The disadvantage of this technique
 2019 IEEE 9th International Conference on System Engineering and Technology (ICSET), 7 October 2019, Shah Alam, Malaysia
is at the reduced processing speed from 30fps to half, this
means real-time processing is not yet achievable using the
technique. One of the problem with using deep learning
[14] or heuristic [15] approaches like Context-Aware Deep
Network (DesnowNet) [16] Generative Adversarial Networks
(ArtGAN) [17] and convolutional neural network (CNN) [18]
is the processing time usually long which is not suitable for
real-time application. Even though the detection accuracy with
the approach can be high compared to the SVM approach,
the accuracy of the system depends on the sample training
images and this grows with the various types of potholes
detectable by the system in several environments like wet, dry
and shady. There are various application of pothole detection
including protecting and improving safety for drivers and pas-
sengers, identifying and locating potholes for road repair and
maintenance [2], [19]–[21], especially on highways. In order
to improve processing time which is the main limitation of
image-based pothole detection, usually algorithms developed
in digital signal processor (DSP) was used [6], [22]. In this
algorithm, pothole can be detected by using disparity values
of the background scene to that of road surface. For improving
mobility of pothole detection device, a mobile, Android-
based pothole detection algorithm was implemented [19]. This
technique has significantly lower in accuracy when detecting
potholes at turning junction where vehicle speed is too slow for
a realized acceleration fluctuation and when pothole appeared
at multi-lane street compared to normal straight road.
III. P ROPOSED A LGORITHM
In this work, the following algorithm (see Fig. 1) is pro-
posed for pothole detection.
As seen from Fig. 1, the processing algorithm goes through
the following steps. Firstly the image in RGB format is loaded
Start
Morphology with
Read RGB Image
Open Kernel
Find Foreground Image
Convert to Morphology with
Gray Scale Close Kernel
Find Unknown Part
Create ROI, Find Background Image Find Markers using
Fill Image within ROI by Dilation Connected Components
Threshold Image
Calculate Distance
Transform
Add 1 to Markers
Highlight Found
Potholes with Red
End
Fig. 1. The Propose Algorithm
269
into memory, then converted to gray scale for two purposes:
(i) ease of processing, and (ii) saving processing time. An
region of interest (ROI) is then created, this limits the region
of focus into a certain area. The reason is that not all portion
of the captured images are useful for the analysis. After ROI
is created, it is filled up by the original image within the
specified ROI. The image is then thresholded with maximum
value of 255 while using inverted binary and Otsu thresholding
techniques [23]. The inverted binary thresholding technique
helps to invert the image pixel from white to black and
vice-versa. The Otsu technique is used to find the optimal
thresholding values given the image.
The image is further processed by applying morphological
technique with firstly open kernel to remove small white
noises, then close kernel to bold the remaining white spaces
which are mainly considered as potholes. The image back-
ground is then calculated by dilation. The number of iterations
for these two sub-steps are 1 and 2 respectively. This is to keep
the processing time of the algorithm optimal.
A distance transform of the image is created which will
be used to calculate the markers before applying watershed-
algorithm. Foreground image and unknown image portion
are then calculated. In order to find markers for applying
watershed algorithm, the connected components technique is
used. It is then added with 1 so that all background image’s
pixels are not 0, it is 1 instead. Any unknown region is
then marked as 0. Finally, watershed algorithm is applied on
original image and markers. The last step is to highlight the
found potholes with red color.
IV. R ESULTS AND A NALYSIS
In this research, at first, an image of potholes on an asphalt
road having resolution of 640 x 480 pixels is used. The
algorithm is run on a computer system with an Intel core
i5 5255U and 4 GB DDR3 memory at bus of 1600 MHz.
The potholes image is displayed in the Fig. 2 where ROI is
highlighted in red.
0
100
200
300
400
0 100 200 300 400 500 600
Fig. 2. Image with ROI
 2019 IEEE 9th International Conference on System Engineering and Technology (ICSET), 7 October 2019, Shah Alam, Malaysia

0 0

100 100

200 200

300 300

400 400

0 100 200 300 400 500 600 0 100 200 300 400 500 600
Fig. 3. Thresholded Image in ROI Fig. 5. Estimated Foreground Image

0 0

100 100

200 200

300 300

400 400

0 100 200 300 400 500 600 0 100 200 300 400 500 600
Fig. 4. Estimated Background Image
Fig. 6. Distance Transform Image

After converting to gray scale and applying the aforemen- 0

tioned thresholding steps, the thresholded image is presented
in Fig. 3. As seen from Fig. 3, the image has many noises
shown in white space dots. However, there are two main 100
white regions that can represent the locations where potholes
appear. Following the thresholded image are the background
and foreground images shown in Figs. 4 and 5. From the 200
background image, the pothole regions are better highlighted
in white. While for foreground image, the smaller part of the
potholes are also highlighted in white. 300
In preparation for applying watershed algorithm, distance
transform is calculated. Based on Fig. 6, it can be seen that
the largest distant area is located on the top of the figure while 400
the smaller distant area is located right at the center of the
image. In these locations, there are more changes in image
intensities with similar change values.
0 100 200 300 400 500 600
After processing the image, the result is plotted in Fig. 7.
From Fig. 7, it is seen that the red curves determine the Fig. 7. Processed Image with Identified Potholes

270
 2019 IEEE 9th International Conference on System Engineering and Technology (ICSET), 7 October 2019, Shah Alam, Malaysia

0 0

100 100

200 200

300 300

400 400

0 100 200 300 400 500 600 0 100 200 300 400 500 600
Fig. 8. Identified Potholes on Smooth Road Surface Fig. 9. Identified Potholes on Aged Road Surface

0
detected potholes. Here, both potholes are detected correctly.
The pothole on the top was found and its boundary is very
close to that can be observed using eyes. This is because of 100
the close to uniform intensity distribution of the pothole region
on the image. While the pothole below was identified by a
group of nearby potholes. The reason is that this pothole has 200
several white regions inside its area on the image. These white
regions are considered as noises that interrupt the detection of
the potholes using the algorithm. 300
In order to verify the effectiveness of the proposed algo-
rithm, images with different potholes sizes and structures were
tested. As a results, Figs 8, 9, and 10 present the detected
400
potholes on smooth, aged, and degraded road surfaces. It is
seen from Fig. 8 that the deep pothole in the middle of the
road was well recognized and its surrounding was detected.
0 100 200 300 400 500 600
In Fig. 9, on an aged road surface, one main pothole with
Fig. 10. Identified Potholes on Degraded Road Surface
significant size was detected in the middle of the image while
other multiple smaller potholes were captured as well. These
potholes are located scattered from the central pothole and speed is slightly below 30.0 fps. This is because the last image
have much smaller sizes compared to the main one. This is contains much more number of potholes as compared to the
the significant evidence of the aged road. others. The more potholes to be detected, the more time taken
Another type of degraded road surface was tested in Fig. 10. to process the image. However, situation like the last image
In this figure, since the road was degraded and there was no will rarely meet in typical road surface which is regularly
maintenance, multiple potholes with significantly large sizes maintained. The algorithm’s average processing speed is 33.1
were detected. Detection of this type of road can help to fps which meets the requirement for real-time application.
provide early warning to the driver for driving ahead carefully.
Here it should be noted that the detected potholes are within TABLE I
the specified ROI as in Fig. 2. This ROI can be adjusted based AVERAGE P ROCESSING S PEED
on the real testing environment and it shall vary from one setup
to another. Fig. 1 Fig. 2 Fig. 3 Fig. 4
Processing Speed (fps) 34.7 37.4 33.6 26.4
For benchmarking purpose, the processing speed of the
proposed algorithm was measured as the mean of the average
speeds when running the algorithm ten (10) times with Figs.
7, 8, 9, and 10. The summary of the processing speed is shown V. C ONCLUSIONS AND F UTURE W ORKS
in Table I. In general, the processing speed of the first three In conclusion, this paper has presented a real-time
images are well above 30.0 fps while for the last image, the watershed-based asphalt road multi-pothole detection algo-

271
 2019 IEEE 9th International Conference on System Engineering and Technology (ICSET), 7 October 2019, Shah Alam, Malaysia
rithm for automobiles using image sensor. The algorithm is [15] W. H. Lim and N. A. M. Isa, “Particle swarm optimization with
able to detect potholes with different sizes and structures while dual-level task allocation,” Engineering Applications of Artificial
Intelligence, vol. 38, pp. 88 – 110, 2015. [Online]. Available:
having an average real-time processing speed of up to 33.1 fps. http://www.sciencedirect.com/science/article/pii/S0952197614002620
It is evident that the algorithm is effective in detecting the [16] Y. Liu, D. Jaw, S. Huang, and J. Hwang, “Desnownet: Context-
potholes on smooth, aged and degraded road surfaces. Future aware deep network for snow removal,” IEEE Transactions on Image
Processing, vol. 27, no. 6, pp. 3064–3073, June 2018.
works will include testing of the algorithm on other types [17] W. R. Tan, C. S. Chan, H. E. Aguirre, and K. Tanaka, “Artgan: Artwork
of road surface such as wet, rocky ones and improving its synthesis with conditional categorical gans,” in 2017 IEEE International
processing speed further with higher resolution image before Conference on Image Processing (ICIP), Sep. 2017, pp. 3760–3764.
[18] V. Pereira, S. Tamura, S. Hayamizu, and H. Fukai, “A deep learning-
testing it on real environment. based approach for road pothole detection in timor leste,” in 2018
IEEE International Conference on Service Operations and Logistics,
ACKNOWLEDGMENT and Informatics (SOLI), July 2018, pp. 279–284.
[19] A. Mednis, G. Strazdins, R. Zviedris, G. Kanonirs, and L. Selavo, “Real
The authors would thank FPT University, Hanoi, Vietnam time pothole detection using android smartphones with accelerometers,”
for supporting this research. in 2011 International Conference on Distributed Computing in Sensor
Systems and Workshops (DCOSS), June 2011, pp. 1–6.
R EFERENCES [20] A. Akagic, E. Buza, and S. Omanovic, “Pothole detection: An efficient
vision based method using rgb color space image segmentation,” in
[1] R. Fan, U. Ozgunalp, B. Hosking, M. Liu, and I. Pitas, “Pothole 2017 40th International Convention on Information and Communication
detection based on disparity transformation and road surface modeling,” Technology, Electronics and Microelectronics (MIPRO), May 2017, pp.
IEEE Transactions on Image Processing, pp. 1–1, 2019. 1104–1109.
[2] K. Vigneshwar and B. H. Kumar, “Detection and counting of pothole [21] I. Schiopu, J. P. Saarinen, L. Kettunen, and I. Tabus, “Pothole detection
using image processing techniques,” in 2016 IEEE International Confer- and tracking in car video sequence,” in 2016 39th International Confer-
ence on Computational Intelligence and Computing Research (ICCIC), ence on Telecommunications and Signal Processing (TSP), June 2016,
Dec 2016, pp. 1–4. pp. 701–706.
[3] S. Hegde, H. V. Mekali, and G. Varaprasad, “Pothole detection and inter [22] A. Mikhailiuk and N. Dahnoun, “Real-time pothole detection on
vehicular communication,” in 2014 IEEE International Conference on tms320c6678 dsp,” in 2016 IEEE International Conference on Imaging
Vehicular Electronics and Safety, Dec 2014, pp. 84–87. Systems and Techniques (IST), Oct 2016, pp. 123–128.
[4] Likun Xia, Tran Duc Chung, and K. A. A. Kassim, “An automobile [23] B. Xue, C. Yu, Y. Wang, M. Song, S. Li, L. Wang, H. Chen, and
detection algorithm development for automated emergency braking C. Chang, “A subpixel target detection approach to hyperspectral image
system,” in 2014 51st ACM/EDAC/IEEE Design Automation Conference classification,” IEEE Transactions on Geoscience and Remote Sensing,
(DAC), June 2014, pp. 1–6. vol. 55, no. 9, pp. 5093–5114, Sep. 2017.
[5] Y. Li, C. Papachristou, and D. Weyer, “Road pothole detection system
based on stereo vision,” in NAECON 2018 - IEEE National Aerospace
and Electronics Conference, July 2018, pp. 292–297.
[6] C. K. Chan, Y. Gao, Z. Zhang, and N. Dahnoun, “Implementation
and evaluation of a pothole detection system on ti c6678 digital signal
processor,” in 2014 6th European Embedded Design in Education and
Research Conference (EDERC), Sep. 2014, pp. 297–301.
[7] A. Fox, B. V. K. V. Kumar, J. Chen, and F. Bai, “Multi-lane pothole
detection from crowdsourced undersampled vehicle sensor data,” IEEE
Transactions on Mobile Computing, vol. 16, no. 12, pp. 3417–3430, Dec
2017.
[8] S. S. Rode, S. Vijay, P. Goyal, P. Kulkarni, and K. Arya, “Pothole
detection and warning system: Infrastructure support and system design,”
in 2009 International Conference on Electronic Computer Technology,
Feb 2009, pp. 286–290.
[9] T. D. Chung, R. Ibrahim, V. S. Asirvadam, N. Saad, and S. M. Hassan,
Wireless HART: Advanced EWMA Filter Design for Industrial Wireless
Networked Control Systems, 1st ed. Taylor & Francis Group, LLC,
2017.
[10] C. D. Tran, R. Ibrahim, V. S. Asirvadam, N. Saad, and
H. S. Miya, “Internal model control for industrial wireless
plant using wirelesshart hardware-in-the-loop simulator,” ISA
Transactions, vol. 75, pp. 236 – 246, 2018. [Online]. Available:
http://www.sciencedirect.com/science/article/pii/S0019057816308424
[11] T. D. Chung, R. B. Ibrahim, V. S. Asirvadam, N. B. Saad, and
S. M. Hassan, “Adopting EWMA filter on a fast sampling wired link
contention in WirelessHART control system,” IEEE Transactions on
Instrumentation and Measurement, vol. 65, no. 4, pp. 836–845, April
2016.
[12] R. Madli, S. Hebbar, P. Pattar, and V. Golla, “Automatic detection
and notification of potholes and humps on roads to aid drivers,” IEEE
Sensors Journal, vol. 15, no. 8, pp. 4313–4318, Aug 2015.
[13] S. Lee, S. Kim, K. E. An, S. Ryu, and D. Seo, “Image processing-
based pothole detecting system for driving environment,” in 2018 IEEE
International Conference on Consumer Electronics (ICCE), Jan 2018,
pp. 1–2.
[14] T. Tsai, W. Cheng, C. You, M. Hu, A. W. Tsui, and H. Chi, “Learning
and recognition of on-premise signs from weakly labeled street view
images,” IEEE Transactions on Image Processing, vol. 23, no. 3, pp.
1047–1059, March 2014.

272