Proceedings of the Fourth International Conference on Trends in Electronics and Informatics (ICOEI 2020)
IEEE Xplore Part Number: CFP20J32-ART; ISBN: 978-1-7281-5518-0
Real-Time Garbage, Potholes and Manholes
Monitoring System using Deep Learning
Techniques
Shahzia Sayyad Shaily Parmar
Department of Computer Department of Computer
Engineering Engineering
Shah & Anchor Kutchhi Shah & Anchor Kutchhi
Engineering College Engineering College
Mumbai, Maharashtra Mumbai, Maharashtra
shahzia.sayyad@sakec.ac.in shaily.parmar@sakec.ac.in
Abstract— A major challenge in urban cities is waste
management, as the pace of urbanization is growing rapidly
sustainable urban development strategies are therefore
required. One of the main concerns with our environment
has been improper garbage containment, unrepaired
potholes, open manholes, and stagnant water which are
harmful to the well-being of the residents. S ince the idea of
smart cities is very trendy these days and without a smart
waste management system, smart cities can't be complete.
Traditional manual monitoring is a cumbersome process
and utilizes more effort, time, and cost which can easily be
assisted with present technologies. Monitoring such tasks
using IoT is one such solution, but incorporating IoT deals
with the use of sensitive electronic devices which are difficult
to maintain and hence add on some additional costs. Hence
this paper describes the development of a system using deep
learning techniques which can cut down huge costs in
effectively monitoring the surroundings and lend a hel ping
hand for a better, safer present and future.
Keywords - IoT, Faster-RCNN, potholes, garbage, manholes
I. Introduction
The concept of a city integrates the use of
modern technologies to optimize various city operations
and services. Street cleanliness is one of the primary tasks
of urban civilization. Cleanliness, management of
potholes and manholes represents the spiritual outlook
and humanistic atmosphere of a city. However, sites of
widespread debris, open manholes, unrepaired potholes ,
and stagnant water are common nowadays in many
urbanizing areas. The Zurburg 2002 report states that in
2010 the amount of waste generated was 20,000 tons per
day, and it was estimated that the amount will be no less
than around 47,000 tons per day by 2025. With the
existing methods of collecting and disposal, it is nearly
impossible to manage such an amount of waste in the
future as around 30% of waste end up on the roads and
public places due to ineffective disposing and collecting
978-1-7281-5518-0/20/$31.00 ©2020 IEEE
Authorized licensed use limited to: Western Sydney University. Downloaded on August 16,2020 at 12:38:40 UTC from IEEE Xplore. Restrictions apply.
Mrinalini Jadhav Karan Khadayate
Department of Computer Department of Computer
Engineering Engineering
Shah & Anchor Kutchhi Shah & Anchor Kutchhi
Engineering College Engineering College
Mumbai, Maharashtra Mumbai, Maharashtra
mrinalini.jadhav@sakec.ac.in karan.khadayate@sakec.ac.in
methods. An article published in ‘The Times of India'
dated 02/08/2018 states that "1,600 metric tons of garbage
is generated every day in the city". An article in
'www.dailymail.co.uk' dated 28/03/2017 states "30 deaths
take place daily due to potholes". An article in
'https://www.mid-day.com' dated 29/09/2016 states "A
21-year-old person died after his bike hit a pothole". In
the traditional system, manual monitoring every area
proves to be a cumbersome procedure that utilizes a lot of
effort, time, and cost. Sometimes the daily activities of
localities become difficult due to odor, infections , and
many other factors caused by such scenarios. At such
times contacting or complaining to the respective
authorities seems to be the only approach. But these
facilities aren’t available easily and instantly which may
worsen situations. Thus, there is a need to design an
automated system which will assist the concerned
authorities to efficiently tackle the problem of garbage,
open manholes, and potholes. In this paper, a real-time
application is proposed to fill the requirements using
advanced techniques of Artificial Intelligence and image
processing to automate the monitoring process.
II. Literature Survey
Varioue research works that have implemented
automated garbage, manholes and potholes monitoring
systems and discussed this below. This will help one gain
a comprehensive idea of all the different types of systems
proposed or developed and can help one develop an
efficient system based on previously proposed or
developed systems.
A. Anitha [1] developed an Internet of Things
(IoT) based system that consists of a sensor over the trash
container which will distinguish the absolute degree of
trash inside it. At the point when the trash will arrive at
the greatest level, a notice will be sent to the corporation's
826
 Proceedings of the Fourth International Conference on Trends in Electronics and Informatics (ICOEI 2020)
IEEE Xplore Part Number: CFP20J32-ART; ISBN: 978-1-7281-5518-0
office, at that point the workers can take further activities
to exhaust the container. The workers can check the status
of the containers on their cell phones. Ultrasonic sen sors
are being utilized in the system to check the degree of
trash in the dustbins.
Mustafa M.R et al.[2] also proposed an IoT
based system using ultrasonic sensors that measures the
trash level, an Advanced RISC Machine (ARM)
microcontroller that controls system activity while
everything will be associated with Thing Speak. This
system enables the waste administration to screen,
dependent on the degree of the trash inside the dustbin.
The system shows the status of various four sorts of trash;
local waste, paper, glass, and plastic on a Liquid Crystal
Display (LCD) screen.
Liu Ying et al. [3] developed a system that
automatically detects and identifies garbage directly using
a front-end embedded monitoring module which uses the
improved You Only Look Once Version 2 (YOLOv2)
network model to do garbage detection and recognition,
and manages thousands of monitoring front-ends through
narrow-band Internet of Things and background server.
Means of target box dimension clustering and
classification network pre-training are used to improve
the YOLOv2 model performance. The network becomes
lightweight after replacing the feature extraction network
by YOLOv2.
Hiroya Maeda et al. [4] launched a system with
a large-scale road damage dataset. This dataset is
composed of 9,053 road damage images captured with a
Smartphone installed on a car, with 15,435 instances of
road surface damage included in these road images. In
each image, they annotated the bounding box representing
the location and type of damage. They also used a state-
of-the-art object detection method using convolution
neural networks to train the damage detection model with
the dataset, and compared the accuracy and runtime speed
on both, using a web server and a Smartphone. Finally,
the paper demonstrates the type of damage that can be
classified into eight types with high accuracy by applying
the proposed object detection method.
[5] proposed the installation of high-resolution
cameras on vehicles to collect the street images. Mobile
edge servers are used to store and extract street image
information temporarily. This street data is transmitted to
the cloud data center for analysis through city networks.
At the same time, Faster Region-Convolutional Neural
Network (Faster R-CNN) is used to identify the street
garbage categories and count the number of garbage.
Finally, the results are incorporated into the street
cleanliness calculation framework to ultimately visualize
the street cleanliness levels, which provides convenience
for city managers to arrange clean-up personnel
effectively.
Chandraprabha R. et al. [6] developed a system
that uses Wireless Sensor Networks (WSN), consisting of
sensor nodes made out of controller, memory, transceiver,
978-1-7281-5518-0/20/$31.00 ©2020 IEEE
Authorized licensed use limited to: Western Sydney University. Downloaded on August 16,2020 at 12:38:40 UTC from IEEE Xplore. Restrictions apply.
and battery. The maximum threshold levels are set for
each sensor. The sensors monitor the changing conditions
when drainage gets blocked or sewage water floods or
manhole covers are open. When a specific sensor arrives
at the particular threshold level, the respective value of
the sensor will be sent to the microcontroller. The
microcontroller sends the signal and the precise area of
the drainage through IoT to the managing station and a
programmed mail is sent by Raspberry Pi. This alarms the
individual in-charge to take the necessary actions in
regards to the issue happening inside the drainage.
[7] developed a system that uses sensors (for
example, accelerometers present in smartphones) which is
capable of automatically detecting potholes in real-time
and also monitor the road traffic conditions. Various
Artificial Intelligence (AI) procedures like Support
Vector Machines (SVM) based on K-means clustering are
applied to the information acquired from such sensors to
estimate street/traffic conditions.
Seung-Ki Ryu et al.[8] proposed a pothole
detection method based on two-dimensional (2D) images.
This system is applied to ITS (Intelligent Transportation
System) service and road management system. Survey
vehicles in Korea were used to collect 2D road images.
The system gave promising results and the information
extracted using the proposed method was used in
determining the maintenance of the road management
system and in taking immediate action for their repair and
maintenance. It also provides alert information of
potholes to drivers as one of ITS services.
Arul Ananth T S et al. [9] proposed a system that
uses various types of sensors like waft, level, and gas
sensors by interfacing with ARM7. This system is
evolved to monitor the water level, water waft, and gases.
If the drainage gets blocked and water overflows this
blockage could be recognized by the sensor and will
display the message at the 16x2 LCD, then this
information is dispatched via Global System for Mobile
Communication (GSM) to the nearby municipality
provider for further corrective movement.
On surveying the papers one would notice the
evolution from using sensors installed in vehicles to using
smartphones, using IoT based systems, detection based
on threshold value to using artificial intelligence and
machine learning approaches, post-processing to real-time
detection of garbage, manhole or potholes. However, each
system is designed for detecting either garbage, manhole,
or pothole; they are unable to detect all the three problems
in a single system. To enable a single system to detect
garbage, manholes, and potholes have developed a system
using a combination of Image Processing and Deep
Learning techniques and have introduced an improved
technique to reach the desired goal.
An online survey is conducted through Google
Forms in which it gains more information about the
hygiene problems faced by residents in our localities. In a
day, about 165 responses were submitted. 80% of
827
 Proceedings of the Fourth International Conference on Trends in Electronics and Informatics (ICOEI 2020)
IEEE Xplore Part Number: CFP20J32-ART; ISBN: 978-1-7281-5518-0
responses were in agreement about the presence of public
bins nearby. Despite the availability, 75% of responses
agreed that there is a presence of overflowing bins and
widespread garbage in some areas. 51% of responses
were with the fact of potholes being kept unrepaired for
as long as 3 months. It is also found that 49.7% of
manholes were left open. Moreover, 74% of respondents
voted for the development of a mobile application which
could help them to tackle such problems efficiently.
III. Proposed System
Fig. 1. Flowchart
The overall idea is to develop an alert system.
An application is created in which users can upload
photos of untidy locations, manholes, and potholes. The
images will be uploaded along with the location and
timestamp. An Image Classifier based on the Faster R-
CNN model is built for validation of the images for the
occurrence of debris, potholes, and manholes and if
found, those details are then stored in a database, which
will then be used to alert the concerned authorities. If left
unchecked, a counter will be kept for each image-location
pair which will indicate the delay of necessary actions.
Prolonged delay will increase the severity (and hence
priority) of that location.
IV. Methodology
A. Training the Image Classifier
Preparing Dataset
The dataset was created by downloading 70 images from
Google Images for each class i.e. Garbage, Potholes , and
Manholes. These images were then divided as 70% for
training and 30% for testing.
978-1-7281-5518-0/20/$31.00 ©2020 IEEE
Authorized licensed use limited to: Western Sydney University. Downloaded on August 16,2020 at 12:38:40 UTC from IEEE Xplore. Restrictions apply.
Labeling Images
The downloaded images were then labeled using the
LabelImg Tool[11]. This tool generated an Extensible
Markup Language (XML) file for each image stating the
Xmin, Ymin, Xmax and Ymax coordinates along with the
Class Label for each class occurrence in that image.
Fig. 2. Label Image
Training the image classifier
A pre-trained Faster-RCNN model was downloaded from
Common Objects in Context(COCO) Trained Models[12]
repository. This model is trained to detect 90 classes on
the MS COCO dataset. The Tensorflow Application
Program Interface API[13] was used for seamless training
and testing of the Image Classifier with Tensorflow
library in Python language. TensorBoard was used to
precisely monitor the training process. Tensorboard
provides a graphical representation of the following losses
against the step number:
● Region Proposal Network (RPN) Loss
RPN Loss [10], LRP N calculated by Eq. (1) is to monitor
the training of the RPN. It is the sum of two components,
the classification loss, LR1 shown in Eq. (2) which
measures the inaccuracy of RPN in deciding whether an
anchor consists of an object or not and the localization
loss, LR2 as shown in Eq. (3) which measures the
inaccuracy of RPN in deciding the anchor coordinates.
= + ∗ (1)
= ∑ ( , ∗)
(2)
= ∑ ∗
∗ ( , ∗
) (3)
Here, i is the anchor index in mini-batch, Ncls stands for
the batch size, Nreg is the number of anchor boxes, Pi is the
probability that anchor i has an object, Pi * stands for the
ground truth of anchor i containing an object (i.e. value is
1 if ith anchor contains an object, 0 otherwise), T i
represents the vector of parameterized coordinates (i.e.
Xmin , Ymin , Width and Height of the anchor), Ti * is the
828
 Proceedings of the Fourth International Conference on Trends in Electronics and Informatics (ICOEI 2020)
IEEE Xplore Part Number: CFP20J32-ART; ISBN: 978-1-7281-5518-0

ground truth box associated with that positive anchor, TABLE 4. BOX-CLASSIFIER LOCALISATION LOSS
is the balancing factor, Lcls is the log loss of 2 classes (LC2 )
(object and non-object) and Lreg is a smooth L1 loss Step Loss
function. 23210 0.070
While training the classifier, the RPN losses were 35000 0.040
observed as follows: 45000 0.010
64030 0.005
TABLE 1. RPN CLASSIFICATION LOSS (LR1 )
Step Loss The total loss (sum of LR and LCN ) was around 2.0 when
25220 0.008 the training initiated. The training process was terminated
35410 0.010 when the total loss dropped around 0.02. It took
45180 0.007 approximately 65,000 iterations to reach this milestone.
62490 0.001
B. Back -end implementation
TABLE 2. RPN LOCALISATION LOSS (LR2 )
Step Loss Web Server
26940 0.047 Flask Web Framework is used to develop a HyperText
35000 0.018 Transfer Protocol (HTTP) server. This server is used to
45000 0.003 provide the following services:
62580 0.002 ● User management and authorization
● Accept images from the users along with the
● Classifier Network Loss Geolocation and Timestamp to store them in a
The classifier network loss [14], LCN is used to measure database.
● Feed the images to the Image Classifier for
the inaccuracy of the classifier network. It is calculated
classification and store the classification details in
similarly as the RPN loss, i.e. it has 2 components,
classification loss LC1 shown in Eq. 4 and regression loss the database.
● Assist the users to track their complaints.
LC2 . The main difference is that in RPN, there are only 2
● Provide a summary of the complaints in a graphical
classes (object and non-object), but in the classifier
network there are multiple classes (equal to the class form to the concerned authorities.
labels for which the model is being trained). The
classification component is cross -entropy loss and is Database System
calculated as: Relational Database Management System (RDBMS) was
used to store the user and complaint details on the server.
∑
[ ][ ] This database helps the concerned authorities to monitor
= (∑ )
[ ][ ] the cleanliness of the locality by storing the Geo-Location
(4) and time stamp of the complaints.
C. Front-end implementation
Where, Nsamples is the number of samples, i stands for the The front end is an Android App which consists of the
sample number, j stands for the index of class labels, following modules:
x[i][j] stands for the probability that the sample i belongs
to class j and Ci is the class index for which sample i Login & Register Activity
shows maximum belongingness. Using this module, users can sign up and log in to use the
The regression loss LC2 is calculated in a similar way to app for posting complaints. A minimum 8 character
LR2 except, now the regression coefficients are class - password is used for authentication of the users.
specific, i.e. for each object class, the coefficient of
regressions is calculated. Home Activity
In this module, the user can capture images using the
TABLE 3. BOX-CLASSIFIER CLASSIFICATION camera and upload them on the Web Server. The
LOSS (LC1 ) Geographic location of the user and Time Stamp of the
Step Loss capture action is noted and sent along with the image data
23410 0.14 and user's ID in the complaint body. The image data is
35000 0.100 encoded into Base-64 format for a seamless transfer over
46640 0.090 the Internet.
61990 0.020

978-1-7281-5518-0/20/$31.00 ©2020 IEEE 829

Authorized licensed use limited to: Western Sydney University. Downloaded on August 16,2020 at 12:38:40 UTC from IEEE Xplore. Restrictions apply.
 Proceedings of the Fourth International Conference on Trends in Electronics and Informatics (ICOEI 2020)
IEEE Xplore Part Number: CFP20J32-ART; ISBN: 978-1-7281-5518-0

image classifier for classification and uploads the

classification results back to the server. Hardware
isolation of this module from the Web Server helps in
reducing the computational load on both modules and
thus, enhances performance and user experience of the
system.
V. Result

Fig. 3. Home Page

History Activity
Through this activity, users can view a list of all
complaints they have registered before.
Fig. 5. Garbage Detected
Details Activity
This activity helps the users to keep track of the selected
complaint. It shows the status of the complaints to the
user along with the classification results of the image
submitted with that complaint in graphical form.

Fig. 6. Pothole Detected

Fig. 4. Details Page

Admin Panel
This section is used by officials of the concerned
authorities to monitor the complaints.

D. Image classification Fig. 7. Pothole, Manhole, and Garbage Detected

A script is written in python which regularly queries the As shown in figures 5, 6 and 7, the garbage, manhole and
Web Server after fixed intervals for complaints that pothole objects identified are marked in rectangular boxes
haven't been filtered yet. For each of such complaints, the and the value on the rectangle represents an accuracy of
script downloads the corresponding image feeds it to the

978-1-7281-5518-0/20/$31.00 ©2020 IEEE 830

Authorized licensed use limited to: Western Sydney University. Downloaded on August 16,2020 at 12:38:40 UTC from IEEE Xplore. Restrictions apply.
 Proceedings of the Fourth International Conference on Trends in Electronics and Informatics (ICOEI 2020)
IEEE Xplore Part Number: CFP20J32-ART; ISBN: 978-1-7281-5518-0

determining whether the object is a garbage category,

[14].Object Detection and Classification using R-CNNs [Online].
manhole category or pothole category.
Available: http://www.telesens.co/2018/03/11/object-detection-and-clas
sification-using-r-cnns/
VI. Conclusion

The challenges embedded in the garbage,

potholes, and manhole management as detailed in the
introduction leading to the development of the system is
significant. To address these challenges, using the
mentioned methodologies to implement a real-time
scalable and reliable system and lend a helping hand in
solving global problems. Hence, the system shows how
advancements in technologies can be used to solve a
variety of such issues easily.

ACKNOWLEDGM ENT

The research was supported by Shah & Anchor

Kutchhi Engineering College, Mumbai. We express our
gratitude to the Management and Principal, Dr. Bhavesh
Patel for providing resources that we required in this
research. We also thank our HoD Prof. Uday Bhave for
his immense support.
REFERENCES

[1]. Anitha, A. (2017, November). Garbage monitoring system using

IoT . In 14th ICSET-2017, IOP Conf. Series: Materials Science and
Engineering (Vol. 263, p. 042027).
[2]. Mustafa, M. R., & Azir, K. K. (2017). Smart bin: internet-of-things
garbage monitoring system. In MATEC Web of Conferences (Vol. 140,
p. 01030). EDP Sciences.
[3]. Liu, Y., Ge, Z., Lv, G., & Wang, S. (2018, August). Research on
Automatic Garbage Detection System Based on Deep Learning and
Narrowband Internet of T hings. In Journal of Physics: Conference
Series (Vol. 1069, No. 1, p. 012032). IOP Publishing.
[4]. Maeda, H., Sekimoto, Y., Seto, T ., Kashiyama, T ., & Omata, H.
(2018). Road damage detection using deep neural networks with images
captured through a smartphone. arXiv preprint arXiv:1801.09454.
[5]. Zhang, P., Zhao, Q., Gao, J., Li, W., & Lu, J. (2019). Urban Street
Cleanliness Assessment Using Mobile Edge Computing and Deep
Learning. IEEE Access, 7, 63550-63563.
[6] Chandraprabha R, Ashwini C.V, Dharani M, Harshitha G, Kruti
Mohan. "Smart Real T ime Manhole Monitoring System" ,Volume:06,
Issue:07,July 2019,e-ISSN:2395-0056 ,p-ISSN:2395-0072.
[7]. Akinwande, V., Adewole, K., Bello, O., & Akintola, A. (2015).
Automatic and real-time pothole detection and traffic monitoring system
using smartphone technology. In International Conference on Computer
Science Research and Innovations (CoSRI 2015).
[8]. Ryu, S. K., Kim, T ., & Kim, Y. R. (2015). Image-based pothole
detection system for IT S service and road management
system. Mathematical Problems in Engineering, 2015.
[9]. Arul Ananth T S, Ramya Laxmi G, Renuka K, Karthik K. "Smart
Sensors and Arm Based Drainage Monitoring System", Digital Object
Identifier:10.35940/ijitee.K1183.09811S19, September 2019, Volume-8,
ISSN:2278-3075.
[10]. Shaoqing Ren, Kaiming He, Ross Girshick, Jian Sun “ Faster R-
CNN: T owards Real-T ime Object Detection with Region Proposal
Networks” (Submitted on 4 Jun 2015 (v1), last revised 6 Jan 2016 (v3))
arXiv:1506.01497
[11]. LabelImg T ool [Online]. Available: https://github.com/tzutalin/lab
elImg
[12]. COCO T rained Models [Online]. Available: https://github.com/ten
sorflow/models/blob/master/research/object_detection/g3doc/detection_
model_zoo.md
[13]. T ensorFlow Models API [Online]. Available: https://github.com/
tensorflow/models/tree/master/research/object_detection/legacy

978-1-7281-5518-0/20/$31.00 ©2020 IEEE 831

Authorized licensed use limited to: Western Sydney University. Downloaded on August 16,2020 at 12:38:40 UTC from IEEE Xplore. Restrictions apply.