Vol. 5, No.

9 September 2014 ISSN 2079-8407

Journal of Emerging Trends in Computing and Information Sciences
©2009-2013 CIS Journal. All rights reserved.

http://www.cisjournal.org

System and Method for Detecting Potholes based on Video Data

1
Taehyeong Kim*, 2 Seung-Ki Ryu
1
Senior Researcher, Korea Institute of Civil Engineering and Building Technology, Korea (*corresponding author)
2
Research Fellows, Korea Institute of Civil Engineering and Building Technology, Korea

ABSTRACT
Pavement distress detection such as cracks, potholes, etc., mostly performed manually, is a labor-intensive and time-
consuming process. Several efforts have been made for developing a technology which can automatically detect and
recognize potholes, thus may contribute to improvement of survey efficiency and pavement quality through prior
investigation and immediate action. The purpose of this study is to develop a system and method for detecting potholes
based on video data. In this study, we have proposed a system that collects video data through a newly designed optical
device mounted on a vehicle and a method that detects potholes in the collected video data using Y signals of input frames.

Keywords: Pothole, Video data, Detecting, Y signals

1. INTRODUCTION potholes in the collected video data using Y signals of

As one type of pavement distresses, a pothole is
defined as a bowl-shaped depression in the pavement
surface and minimum plan dimension is 150 mm [1].
Potholes can generate damages such as flat tire and wheel
damage, impact and damage on the lower part of a
vehicle, sudden braking and steering wheel operation, and
vehicle collision and major accidents.
Also, potholes are important clues indicating
structural defects of the asphalt road, and accurately
detecting these potholes is one of important tasks for
determining proper strategies of asphalt-surfaced
pavement maintenance and rehabilitation. However,
pothole detection mostly performed manually is a labor-
intensive and time-consuming. Thus, several efforts have
been made for developing a technology which can
automatically detect and recognize potholes, which may
contribute to improvement of survey efficiency and
pavement quality through prior investigation and
immediate action.
As shown in Kim and Ryu [2], existing methods
for pothole detection can be divided into vibration-based
methods by Yu and Yu, De Zoysa et al., Erikson et al. and
Mednis et al. [3–6], 3D reconstruction-based methods by
Wang, Chang et al., Hou et al., Li et al., Staniek, and
Moazzam et al. [7–13], and vision-based methods by
Koch and Brilakis, Jog et al., Lokeshwor et al., Koch et
al., Buza et al., and Lokeshwor et al. [14–19].
Although the vision-based methods are cost-
effective as compared to 3D laser scanner methods, it may
be difficult to accurately detect a pothole by these
methods due to the distorted signal generated by noise
since they detect a pothole through analysis of the
collected image and video data.
Thus, the purpose of this study is to develop a
system and method for effectively detecting potholes
based on video data. In this study, we developed a system
that collects video data through a newly designed optical
device mounted on a vehicle and a method that detect
input frames.
2. RELATED WORKS
Vision-based methods include 2D image-based
approaches such as Koch and Brilakis [14] and Buza et al.
[18] and video-based approaches as proposed by Jog et al.
[15], Lokeshwor et al. [16] and Koch et al. [17]. In this
paper, the main focus of the study is on video-based
approaches.
Jog et al. proposed a new approach based on 2D
recognition and 3D reconstruction to detect and measure
width, number, and depth of potholes using a monocular
camera mounted on the rear of a car [15]. The method of
2D recognition built upon the previous work of Koch and
Brilakis [14], and the method of 3D sparse reconstruction
built upon the previous work of Golparvar-Fard et al.
[20]. Also, 3D dense reconstruction and mesh modeling
were based on the dense 3D point cloud reconstruction of
Golparvar-Fard et al. [20] and the poison surface
reconstruction approach. They used several actual
potholes using a Canon Vixia HD camera (20fps at
20mph) to validate the proposed method.
Lokeshwor et al. proposed a method for
automated detection and assessment of potholes, cracks
and patches from video clips of Indian highways [16]. In
the proposed method, using DFS (Distress Frames
Selection) algorithm, captured video clips are segmented
automatically into two different types of frames as
follows.
- Frames with distress
- Frames without distress
Then, database of frames with distress is
processed with CDDMC (Critical Distress Detection,
Measurement and Classification) algorithm which
consists of image enhancement, image segmentation,
visual properties extraction, detection and classification
by decision logic, and quantification. Also, the decision
logic for potholes, cracks and patches was developed by

703
 Vol. 5, No. 9 September 2014 ISSN 2079-8407
Journal of Emerging Trends in Computing and Information Sciences
©2009-2013 CIS Journal. All rights reserved.

http://www.cisjournal.org
three main distinctive visual properties of these distresses 3. POTHOLE DETECTION SYSTEM
as follows. In this study, we develop a system that collects
video data through a newly designed optical device
- The visual texture (standard deviation), mounted on a vehicle and detects a pothole from the
- The shape (circularity), collected video data using our proposed algorithm.
- The dimension (average width).
3.1 System
A database of 1275 video frames with distress Fig 1 show the concept of pothole detection
was selected randomly from the obtained results after system. This system includes an optical device and a
applying DFS algorithm to various video clips. As results, pothole detection algorithm.
an overall accuracy of 97% with 95% precision and 81%
recall in detecting frames with potholes, overall accuracy
of 94% with 93% precision and 98% recall in detecting
frames with cracks, and overall accuracy of 90% with
8.5% precision and 19% recall in detecting frames with
patches. If few objects such as bleedings, manholes, black
colored road markings and discoloration spots appear very
similar to potholes, cracks and patches, the proposed
method could not deliver high accuracy.

The method by Koch and Brilakis [14] was

limited to single frames and therefore cannot determine
the magnitude of potholes in the frame of video-based
pavement assessment. In order to complement and
improve the previous method, Koch et al. presented an
enhanced pothole-recognition method that updates the
texture signature for intact pavement regions and utilize
vision tracking to track detected potholes over a sequence
of frames [17]. The proposed method was implemented in
MATLAB and tested on 39 pavement video containing Fig 1: The concept of pothole detection system
10,180 frames. The resulting total recognition precision
and recall were 75% and 84%, respectively. 3.2 Optical Device
Consequently, compared with the previous method, the The newly designed optical device for collecting
texture-comparison performance was increased by 53%, pothole video data is divided into three parts, light/optical
and the computation time was reduced by 57%. They part, video part, and GPS control/total control part, as
assumed that only one pothole enters the viewport at a shown in Fig 2.
time, and therefore additional work is needed for
considering multiple potholes in the viewport.

Table 1 shows the summary of video-based

approaches for detecting potholes.

Table 1: Vision-based approached for detecting potholes

Authors Equipment Experiments
Jog et al. Canon Vixia A few series of video
[14] HD camera stream (1440*810, 20fps)

Lokeshwor ARRB NSV 1275 video frames

et al. [15] and a digital consisting of 283
camera monochromatic images
(1280*960) by ARRB
NSV and 992 video Fig 2: Design of the optical device for collecting video
images(640*480) by a data
digital camera in India
Koch et al. Robot vehicle 39 video clips Also, Fig 3 show the drawing of the optical
[13] with a rear (640*480, 30fps) device that was designed to easily be mounted in a vehicle.
camera (HP It has several functions such as collecting and storing
Elite vided data of potholes, communicating by Wi-Fi, and
Autofocus gathering location information by GPS.
webcam)

704
 Vol. 5, No. 9 September 2014 ISSN 2079-8407
Journal of Emerging Trends in Computing and Information Sciences
©2009-2013 CIS Journal. All rights reserved.

http://www.cisjournal.org

Fig 3: Drawing of the optical device

Table 2 shows the detailed specification of the

optical device.

Table 2: Specification of the optical device

Item Specification
Housing - Plastic
Size - 110(W)*40(L)*110(H), 1.5L
Range - The inside lane, left and right lanes
Resolution - 1280*720, 60fps
Camera - 6 glasses
module - CMOS fixed type
- The diameter of lenses: 12mm Fig 4: Process of pothole detection algorithm
CPU - More than 3000DMIPS
Storage - Two storage spaces for safety 4.1 Noise Removal of Original Image
GPS - Antenna: As a basic process, noise removal of original
25mm(W)x25mm(L) image is conducted to the signal processing of all images.
- Backup Battery In this study, we simplify an image using Gaussian Blur.
Fig 5 show before and after images by noise removal of
Power - Using the power of a vehicle
original image.
- Holding secondary power unit

The optical device can collect video data of the

inside line as well as left and right lanes. The lenses of
camera module consist of 6 glasses and the diameter of
the lenses is 12mm. The storage part has two storage
space for safety. The optical device uses the power of a
vehicle, and also it has a secondary power unit.

4. POTHOLE DETECTION METHOD

We developed a method that detect a pothole
from the video data using brightness information of Y
(gray) in the consecutive flames. Fig 4 shows the process
of the proposed detection algorithm. In this process, small
crack parts are not under consideration.

(a) Before

705
 Vol. 5, No. 9 September 2014 ISSN 2079-8407
Journal of Emerging Trends in Computing and Information Sciences
©2009-2013 CIS Journal. All rights reserved.

http://www.cisjournal.org
by shape and size information. Fig 7 show before and
after images by simplification of video by binarization.

(b) After
Fig 5: Before and after images by noise removal of
original image
(a) Before
4.2 Adjustment of Brightness by Region
In this step, we adjust brightness to improve the
contrast of an image. Consequently, the difference of
brightness between pothole and background may become
large. Adaptive histogram equalization is applied to this
step.

Fig 6 show before and after images by

adjustment of brightness by region.

(b) After
Fig 7: Before and after images by simplification of video
by binarization

4.4 Noise Removal of the Binary Image

After binarization, still, there may be small
noises in an image. Thus, we need to remove the noise of
the binary image. Fig 7 show before and after images by
noise removal of the binary image.
(a) Before

(a) Before
(b) After
Fig 6: Before and after images by adjustment of
brightness by region

4.3 Simplification of Video by Binarization

Binarization is needed to simplify an image in
this step, and we simplify several objects within an image

706
 Vol. 5, No. 9 September 2014
Journal of Emerging Trends in Computing and Information Sciences
©2009-2013 CIS Journal. All rights reserved.
http://www.cisjournal.org
(b) After
Fig 8: Before and after images by noise removal of the
binary image
4.5 Extraction of the Edge of Segmented Object
In this step, the edge of segmented object on the
binary image is extracted, as shown in Fig 9.
Fig 9: Example of the edge extraction of segmented
object
4.6 Selection and Square Zoning for Objects
In this step, we select an object that has the
extracted edge and make a square zone for the object as
ROI (Region of Interest), as shown in Fig 10.
Fig 10: Example of square zoning of ROI (red color)
4.7 Return of Pothole Area Information
Finally, we get information of the detected
pothole within the image, as shown in Fig 11.
ISSN 2079-8407
Fig 11: Example of pothole area detection
Also, based on above process, we developed a
program that can detect a pothole and display a square
zoning of ROI on the original video while playing the
video as shown in Fig 12.
Fig 12: Pothole detection program
5. CONCLUSION
The purpose of this study was to develop a
system and method for effectively detecting potholes
based on video data. In this study, we developed a system
that can collects video data through a newly designed
optical device mounted on a vehicle and a method that can
detect potholes in the collected video data using Y signals
of input frames. Also, based on above method, we
developed a program that can detect a pothole and display
a square zoning of ROI on the original video while
playing the video.
In this study, the newly designed and made
optical device can be easily mounted on a vehicle and
collect pothole video data effectively. However, for
commercialization, there is a need to develop the
integrated terminal equipment that include an optical
device and image processer. Also, the improvement of the
performance of pothole detection by the proposed
algorithm is still a future area of research.
ACKNOWLEDGEMENTS
This research was supported by a grant from a
Strategic Research Project (Development of Pothole-Free
Smart Quality Terminal, 2014-0219) funded by the Korea
Institute of Civil Engineering and Building Technology.
707
 Vol. 5, No. 9 September 2014
Journal of Emerging Trends in Computing and Information Sciences
©2009-2013 CIS Journal. All rights reserved.
http://www.cisjournal.org
REFERENCES
[1] J. S. Miller, and W. Y. Bellinger, Distress
identification manual for the long-term pavement
performance program, FHWA-RD-03-031, Federal
Highway Administration, Washington, DC, USA
(2003).
[2] T. Kim and S. Ryu, Review and analysis of pothole
detection methods, Journal of Emerging Trends in
Computing and Information Sciences, Vol. 5, No. 8,
(2014), ?-?.
[3] B. X. Yu, and X. Yu, Vibration-based system for
pavement condition evaluation, In Proceedings of
the 9th International Conference on Applications of
Advanced Technology in Transportation (2006),
183-189.
[4] K. De Zoysa, C. Keppitiyagama, G. P. Seneviratne,
and W.W.A.T. Shihan, A public transport system
based sensor network for road surface condition
monitoring, In Proceedings of Workshop on
Networked Systems for Developing Regions
(2007), 1-6.
[5] J. Erikson, L. Girod, and B. Hull, The pothole
patrol: using a mobile sensor network for road
surface monitoring, In Proceedings of the 6th
International Conference on Mobile Systems,
Applications, and Services (2008), 29-39.
[6] A. Mednis, G. Strazdins, R. Zviedris, G. Kanonirs,
and L. Selavo, Real time pothole detection using
Android smartphones with accelerometers, In
Proceedings of the International Conference on
Distributed Computing in Sensor Systems and
Workshops (2001), 1-6.
[7] K. C. P. Wang, Challenges and feasibility for
comprehensive automated survey of pavement
conditions, In Proceedings of 8th International
Conference on Applications of Advanced
Technologies in Transportation Engineering (2004),
531-536.
[8] K.T. Chang, J. R. Chang, and J. K. Liu, Detection
of pavement distress using 3D laser scanning
technology, In Proceedings of the ASCE
International Conference on Computing in Civil
Engineering (2005), 1-11.
[9] Z. Hou, K. C. P. Wang, and W. Gong,
Experimentation of 3D pavement imaging through
stereovision, In Proceedings of the International
Conference on Transportation Engineering (2007),
376-381.
[10] Q. Li, M. Yao, X. Yao, and B. Xu, A real-time 3D
scanning system for pavement distortion
inspection, Measurement Science and Technology,
Vol. 21, No. 1, (2009), 15702-15709.
ISSN 2079-8407
[11] M. Staniek, Stereo vision techniques in the road
pavement evaluation, In Proceedings of the
XXVIII International Baltic Road Conference
(2013), 1-5.
[12] D. Joubert, A. Tyatyantsi, J. Mphahlehle, and V.
Manchidi, Pothole tagging system, In Proceedings
of the 4th Robotics and Mechatronics Conference
of South Africa(2011), 1-4.
[13] I. Moazzam, K. Kamal, S. Mathavan, S. Usman,
and M. Rahman, Metrology and visualization of
potholes using the Microsoft Kinect sensor, In
Proceedings of the 16th International IEEE Annual
Conference on Intelligent Transportation
Systems(2013), 1284-1291.
[14] C. Koch, and I. Brilakis, Pothole detection in
asphalt pavement images, Advanced Engineering
Informatics, Vol. 25 (2011), 507-515.
[15] G. M. Jog, C. Koch, M. Golparvar-Fard, and I.
Brilakis, Pothole properties measurement through
visual 2D recognition and 3D reconstruction, In
Proceedings of the ASCE International Conference
on Computing in Civil Engineering (2012), 553-
560.
[16] H. Lokeshwor, L. K. Das, and S. K. Sud, Method
for automated assessment of potholes, cracks and
patches from road surface video clips, Procedia –
Social and Behavioral Sciences, Vol. 104 (2013),
312-321.
[17] Koch, C., G. M. Jog, and I. Brilakis, Pothole
detection with image processing and spectral
clustering, Journal of Computing in Civil
Engineering, Vol. 27, No. 4, (2013), 370-378.
[18] Buza, E., S. Omanovic, and A. Huseinnovic, Stereo
vision techniques in the road pavement evaluation,
In Proceedings of the 2nd International Conference
on Information Technology and Computer
Networks, (2013), 48-53.
[19] Lokeshwor, H., L. K. Das, and S. Goel, Robust
method for automated segmentation of frames
with/without distress from road surface video clips,
Journal of Transportation Engineering, Vol. 140
(2014), 31-41.
[20] M. Golparvar-Fard, F. Pena-Mora, S. Savarese,
Automated model-based progress monitoring using
unordered daily construction photographs and IFC
based building information models, Journal of
Computing in Civil Engineering (2014),
10.1061/(ASCE)CP.1943-5487.0000205.
708
 Vol. 5, No. 9 September 2014 ISSN 2079-8407
Journal of Emerging Trends in Computing and Information Sciences
©2009-2013 CIS Journal. All rights reserved.

http://www.cisjournal.org
AUTHOR PROFILES
Taehyeong Kim received the degree in transportation Seung-Ki Ryu received the degree in electrical
engineering at the University of Maryland in the U.S. engineering at the ChungBuk National University in
Currently; he is a senior researcher at Korea Institute of Korea. Currently, he is a research fellow at Korea Institute
Civil Engineering and Building Technology. His research of Civil Engineering and Building Technology. His
interest covers intelligent transportation systems, research interest covers intelligent transportation systems,
information technology, optimization, paratransit, information technology, ubiquitous city, construction IT
logistics, and simulation. convergence and logistics.

709