Cloud Computed Machine Learning Based
Real-Time Litter Detection using Micro-UAV
Surveillance
Ashley Chung, Dong Young Kim, Ethan Kwok,
Michael Ryan, Erika Tan
NJ Governor’s School of Engineering and Technology
Piscataway, New Jersey
Abstract—Litter can remain undetected and uncollected for
extended periods of time, leading to detrimental consequences
on the environment. Solutions to mitigating these effects focus on
severe legal action directed towards offenders or litter collection
events, all of which are not automated. Therefore, to reduce
the amount of manual labor required for current solutions, this
project aims to implement an automated micro-unmanned aerial
vehicle (UAV) capable of real time litter detection from UAV
surveillance footage. Performances of five different algorithms
(two classifiers and three detectors) were compared after training
them on various public images of litter on the Google Cloud
Platform to determine the strongest models to utilize in the
ensemble method. Out of the two ensemble models tested, one
being a custom-built ensemble and the other being a bootstrap
aggregating (bagging) ensemble, performance of the bagging
ensemble demonstrates a significant improvement in performance
over any individual model.
I. I NTRODUCTION
Undetected litter is a ubiquitous problem that has negative
implications on quality of life, the environment, and the
economy. When municipal solid waste is improperly disposed,
it can become a hindrance not only in public areas, where they
are most commonly found, but also in animal habitats. For
example, litter containing harmful chemicals may contaminate
the surrounding water and air, causing further habitat alteration
by depleting levels of oxygen and light. In turn, this impedes
the ability of habitats to support life, leading to a decline
in species diversity [1]. The ingestion and entanglement of
various improperly disposed contaminants also poses a consid-
erable threat to wildlife. Unfortunately, unless these biohazards
are identified and removed, they will take many years to
decompose and will continue to disrupt the ecosystem [2].
In addition, up to 90% of forest fires are commonly started by
another commonly disposed item: cigarettes [3]. These fires
can not only cause damage to the forest ecosystem, but also
cost millions of dollars in reparations.
Currently, most solutions to this problem are not automated;
rather, they revolve around legislation or community efforts to
manually remove litter. For instance, the punishment for minor
infractions is a fine and an order to clean litter or complete
community service [4]. Additionally, many organizations, such
as Keep America Beautiful, work closely to maintain public
areas manually [5]. While these solutions have tried to reduce
1
978-1-5386-9374-2/18/$31.00 ©2018 IEEE
Authorized licensed use limited to: Robert Gordon University. Downloaded on May 28,2021 at 05:38:47 UTC from IEEE Xplore. Restrictions apply.
Ryan Gamadia
Corresponding Author
Rutgers School of Engineering
Piscataway, New Jersey
the detrimental effects of littering, they are not feasible long
term solutions, as human surveillance is both time-consuming
and dependent on manual labor. Thus, a more effective long
term solution would be an automated object detection system
capable of detecting and locating litter solely from real-time
transmission of visual remote UAV video.
II. E XPERIMENTAL P ROCEDURE
A. Dataset Preparation
The images used to train and test the models were pooled
together from two main sources: Trashnet Dataset from Gary
Thung and Mindy Yang’s final project for Stanford University
CS 229 Machine Learning course and various litter images
from Google Images. From the Trashnet Dataset, 2342 of
2527 litter images with a white background and resolution of
512x384 were selected as useful for the final dataset. To further
supplement the dataset, the tool Google Images Download was
utilized to gather 566 additional litter images from various
sources [6]. This ensured that the model would learn in a
more versatile manner rather than solely detecting litter on
white background without much noise. After forming the final
dataset of 2825 images, the images were separated into eight
classes: bottle, can, cardboard, container, cup, paper, scrap, and
wrapper. For training and testing, while the classifiers could
use only the Trashnet dataset because they needed images with
only one object, the detectors were able to utilize the final
dataset. To generate the necessary annotations for detectors,
the tool LabelImg was utilized to save the annotations in an
XML file using PASCAL visual object class (VOC) format
[7]. Both classifiers and detectors trained and tested with a
80:20 training to testing ratio split.
B. Convolutional Neural Network (CNN)
To implement the CNN model, transfer learning was used
to further train the VGG-16 pre-trained model for Keras with
additional layers. Due to its ability to apply kernels onto image
vectors of multi-channeled images, the VGG-16 model was
able to recognize import features in any image. Four more
layers were added to adapt the VGG-16 for litter classification:
a Flatten layer, two Dense layers, and a Dropout layer. The
Flatten layer converted all of the pooled images taken from the
dataset into a continuous one-dimensional vector. The Dense
layers conducted a linear operation on the given input vector.
The Dropout layer prevented overfitting the neural network
on the litter dataset through a random elimination of units and
connections between layers. Furthermore, the learning rate was
increased to 1 ∗ 10−3 with the ADAM optimizer for the first
60 iterations, while it was later decreased to 1 ∗ 10−4 and with
the SGD optimizer for the next 180 iterations.
C. Support Vector Machine (SVM)
The SVM with HOG model was mainly developed with
Scikit-Learn and Scikit-Image. The training process started
with the feature extraction from the training images. The HOG
function from Scikit-Image extracted and saved the histograms
from each image. An 8x8 cell, 1x1 block, and 12 orienta-
tion bins were used as parameters for the feature descriptor,
resulting in a vector size of 3600 per image. These values
were chosen to maintain the useful features while also saving
computation time. After saving all the feature descriptors into
separate files, the SVC module with a linear kernel from
Scikit-Learn was used to train and save the necessary classifier.
The testing process followed a similar structure; images in
the the testing process were passed through the same feature
extraction process as the training step.
D. Single Shot Multibox Detector (SSD) and Region-based
Fully Convolutional Network (R-FCN)
The SSD and R-FCN models were built using the Ten-
sorFlow Object Detection API [8]. Transfer learning was
used to instantiate the new R-FCN model with some of
the previously trained layers of the Resnet-101 network and
the SSD model with previously trained layers from Googles
InceptionV2. Both models were previously trained on the
Microsoft Common Objects in Context (MSCOCO) Dataset
using over 220,000 annotated images of general objects from
people to planes. In order to use the TensorFlow Object
Detection API, a simple script was written in Python to convert
the images and image annotations from PASCAL-VOC format
to TFRecord files.
After preliminary testing, it was clear that the hyperparame-
ters for both models needed tuning. During initial training, the
loss value fluctuated instead of steadily decreasing. To mitigate
this issue, the learning rate of both models was decreased to
make the training more gradual and consistent. The R-FCN
learning rate was changed from 3 ∗ 10−4 to 6 ∗ 10−6 , and the
SSD learning rate was changed from 4 ∗ 10−4 to 1 ∗ 10−4 . The
rate of decay of the learning rate was also slightly increased,
so that the models would learn quickly at the beginning of
the training, but would slow down as the model progressed,
making changes more gradually. For the R-FCN, this was
achieved by dividing the learning rate by ten at around every
80,000 steps. For the SSD, this was accomplished by changing
the learning rate decay factor from 0.95 to 0.9. After these
small adjustments the models were ready for training on the
final dataset.
2
978-1-5386-9374-2/18/$31.00 ©2018 IEEE
Authorized licensed use limited to: Robert Gordon University. Downloaded on May 28,2021 at 05:38:47 UTC from IEEE Xplore. Restrictions apply.
E. You Only Look Once (YOLO)
The YOLO model was mainly implemented with DarkFlow,
an open source adaptation of the Darknet library for Ten-
sorFlow. The algorithm was generated with the architecture
and weights of both YOLO and tiny YOLO, a compressed
version of YOLO. In order to customize the algorithm for litter
detection, the output layer was changed to have eight possible
litter classifications and 65 filters. After initial testing, param-
eters, including epochs, batch size, and learning rate, were
tuned to trade off bias with variance. It was found that YOLO
had a tendency to poorly detect smaller objects and generally
needed to have a greater number of iterations. By changing the
learning rate and batch size hyperparameters, the YOLO was
able to converge to a global optimum. Ultimately, the YOLO
model was finalized with a learning rate of 1 ∗ 10−5 and a
batch size of 16.
F. Custom Ensemble
A custom ensemble method was designed and implemented
using the two best detectors and best classifier to extrapolate
a more accurate prediction. The custom ensemble first ran the
first detector to propose its predictions for possible locations
of litter in the image. Afterwards, the custom ensemble ran
the second detector on each proposed subregion from the
first model. The second detector shrunk the predictions of
the first detector model and detected objects missed by the
first detector. Finally, the custom ensemble ran the classifier
model on each subregion proposed by the second detector
model to determine if the smallest predictions were accurate
or if the prediction of second detector was too small. If all of
the bounding boxes proposed by the second model matched
the classes assigned by the classifier, then the predictions of
the second model were selected in place of the predictions of
the first model. Otherwise, the predictions of the first detector
were accepted. In this way, the various models used a system
of checks and balances to produce a more accurate final
prediction.
G. Bagging Ensemble
The bagging ensemble was built with the two best per-
forming detectors. Predictions from each detector were aver-
aged together to determine the final prediction. Due to the
possibility of detecting multiple objects in one image, the
ensemble method needed to determine which bounding boxes
corresponded to the same object. This step was critical to
ensure the ensemble would not ruin predictions with multiple
objects by averaging the box to a point in the image that
likely does not contain any object at all. In order to match the
boxes together, the Intersection over Union (IoU) values were
generated for every combination of boxes. The bounding boxes
were considered to be correlated and were averaged together
only if the IoU score exceeded 50%. Otherwise the predictions
were considered separate entities. If a bounding box did not
have any other boxes correlated to it, the prediction would be
added to the final prediction only if the original model had an
exceedingly high confidence value [9].
H. Micro-UAV Integration an F1 score of 0. However, the SVM saw successes with
To control and receive surveillance footage from the micro- cups and papers due to both types of objects having a distinct
UAV, a third party, open-source project called Hack-a-Drone shape, which was very advantageous for the SVM since it took
was implemented due to the limited access to the built-in in HOG as its feature descriptor. Unfortunately, the feature
software [10]. The buffered image files were piped from the descriptor caused problems for the SVM as well. Many bottles
project, which was written in Java, to a separate Python script and containers had a similar shape, which lead to the SVM
through a localhost socket for compatibility with the machine confusing the two classes many times, as shown by the F1
learning models developed with Python. Despite the slight scores.
speed loss due to the indirect connection, this method saved TABLE 2. AP S CORES OF D ETECTORS AND E NSEMBLES
time from recreating the drone communication protocol from
scratch. In order to minimize the delay in the video feed, a SSD R-FCN YOLO Custom Bagging
multithreading approach was utilized to capture drone footage Bottle 0.62 0.71 0.56 0.56 0.77
in one thread while processing the footage in another. Can 0.70 0.55 0.48 0.67 0.72
III. R ESULTS AND A NALYSIS Cardboard 0.52 0.73 0.79 0.49 0.78
Container 0.71 0.68 0.55 0.71 0.83
Different evaluation metrics were used to determine the
Cup 0.12 0.00 0.00 0.09 0.11
effectiveness of each classifier and detector. F1 score was used
Paper 0.80 0.74 0.84 0.78 0.85
to assess the classifiers to balance the weights of precision
and recall. Average Precision (AP) score was used to evaluate Scrap 0.67 0.56 0.00 0.67 0.67
the detectors because the IoU values needed to be taken into Wrapper 0.32 0.22 0.00 0.20 0.41
account. Average 0.56 0.53 0.40 0.52 0.64

TABLE 1. F1 S CORES OF C LASSIFIERS

CNN SVM C. SSD
Bottle 0.681 0.647 The SSD produced the best results out of the three detector
Can 0.489 0.510 models, returning an mAP score of 0.56. The top results of
Cardboard 0.588 0.689 the SSD may be attributed to its longest training time out of
Container 0.442 0.419 the three detector models. Moreover, bottles, cans, cardboards,
Cup 0.000 0.889 containers, and papers had the most images, resulting in
Paper 0.000 0.746 a thoroughly trained model. The lack of cup images also
Scrap 0.154 0.000 impacted the SSD, producing an AP score of merely 0.12.
Wrapper 0.282 0.643
D. R-FCN
Average 0.330 0.623
The R-FCN performed nearly as well as the SSD, delivering
an mAP score of 0.53. R-FCN’s best performances were seen
A. CNN with bottles, cardboards, and papers. This is consistent with
While the CNN was able to achieve a low training loss of the fact that there was a higher number of images for those
0.1598, the average F1 score of the test set showed that the three classes compared to others. The R-FCN received an AP
model had overfitted the training dataset. Two methods were score of 0.00 on cup images largely due to the lack of cup
brought in to relieve overfitting: the creation of a validation training images available.
subset and the increase of the learning rate. Hyperparameter
E. YOLO
tuning resulted in a model that demonstrated a noticeable
improvement, with an average F1 score of 0.330. The F1 The tiny YOLO algorithm performed poorly with an mAP
scores of 0.0 regarding the cup and scrap classes may have score of 0.08. As a result, a second YOLO model was
resulted from the fact that there was an unequal distribution implemented using a more powerful version of YOLO with
of image data. There were fewer images of cups and scraps more convolutional layers. The updated YOLO model saw
to train on compared to other types of litter such as bottles; significant improvement, as it returned an mAP score of 0.40.
thus, there was not as much information that the models could YOLO’s lower performance was mostly attributed due to its
learn from. As a result, the model rarely classified litter as a speed. A faster detection algorithm resulted in a trade off
cup or a scrap. between speed and accuracy. Moreover, the YOLO model
had a considerable demand for data, as demonstrated by
B. SVM its poor performances with cups, scraps, and wrappers. The
As seen in Table 1, the SVM produced an average F1 architecture of a YOLO model also caused some difficulty
score of 0.623, markedly higher than the F1 score of the regarding small object detection. This issue was undoubtedly
CNN. Similar to the CNN, the SVM performed poorly when present in the same aforementioned classes, thus significantly
classifying scraps due to the lack of training data, returning weakening YOLO’s average mAP score.

3
978-1-5386-9374-2/18/$31.00 ©2018 IEEE

Authorized licensed use limited to: Robert Gordon University. Downloaded on May 28,2021 at 05:38:47 UTC from IEEE Xplore. Restrictions apply.
F. Custom Ensemble
From the results of the classifiers and detectors, the custom
ensemble was assembled with the SSD, R-FCN, and SVM
models. Unexpectedly, the custom ensemble delivered an mAP
score of 0.52, performing marginally worse than the two
best detector models, SSD and R-FCN. Observing individual
results revealed the main reason for the unexpectedly worse
performance: the second detector (R-FCN) focused too closely
on an object causing the classifier (SVM) to inaccurately
recognize an object from a very miniscule bounding box. Fur-
thermore, the SVM was not robust enough in some categories
to be the final classifier for the ensemble, as demonstrated
by its F1 score of 0.623. In addition, the Trashnet dataset
contained many images of only specific parts of litter, such
as only the upper half of a water bottle. This may have
confused the model when shrinking the image, as the second
detector model often split the object into pieces and the
classifier would still detect each piece as the correct object of
interest. Ultimately, the checks and balances system between
the models in the custom ensemble did not show an overall
improvement.
G. Bagging Ensemble
The bagging ensemble was built with the SSD and R-FCN
and returned an mAP score of 0.64, showing an improvement
of approximately 0.086 over the SSD, the best individual
detector. The bagging ensemble performed successfully, unlike
the custom ensemble, due to its nature of averaging the
results, which balanced inaccuracies in the predictions of
individual models. This method also made fewer false positive
predictions because the boxes produced by each individual
model needed extremely high confidence scores to overwrite
the other model if both models did not agree. Since the
bagging ensemble demonstrated the strongest results of all
other ensemble methods and individual models, it was selected
for use with the micro-UAV.
IV. C ONCLUSION
The use of robust machine learning algorithms alongside
with computer vision techniques has great potential to affect
environment monitoring and maintenance. This paper shows
that a combination of individual models into an ensemble
performs far better with regards to litter detection than any
single algorithm. While the complex custom ensemble did
not create any noticeable improvements compared to the
individual algorithms, the more simplistic bagging ensemble
was able to significantly improve the accuracy of the combined
strong models. Future directions include an increased dataset
4
978-1-5386-9374-2/18/$31.00 ©2018 IEEE
Authorized licensed use limited to: Robert Gordon University. Downloaded on May 28,2021 at 05:38:47 UTC from IEEE Xplore. Restrictions apply.
and higher-resolution camera to further train and enhance the
models. Continuation of this research could result in the de-
ployment of drones to public areas. With further developments
such as the installation of a GPS and automated piloting
system, the micro-UAVs could report the precise location and
type of litter to organizations to help communities stay clean
with minimal human supervision. The design presented in this
paper demonstrates the versatility and extensibility to larger
areas of land.
ACKNOWLEDGEMENTS
The authors of this paper would like to acknowledge project
mentor Ryan Gamadia and residential teaching assistant (RTA)
Pragya Hooda for their guidance and involvement throughout
the research process, in addition to Andrew Page and Google
for their contribution of Google Cloud Platform credits. More-
over, special gratitude towards program director Ilene Rosen,
associate program director Jean Patrick Antoine, research
coordinator Brian Lai, and head RTA Nicholas Ferraro for
making it possible to conduct high level research at the New
Jersey Governors School of Engineering and Technology (NJ
GSET). Special thanks to Anthony Yang, Shantanu Laghate,
and Jennifer He for supplying preliminary resources and aid.
Finally, much gratitude to the sponsors of NJ GSET for
their continued participation and support: Rutgers University,
Rutgers School of Engineering, the State of New Jersey,
Silverline Windows, Lockheed Martin, Rubiks, and the alumni
of NJ GSET.
R EFERENCES
[1] Impacts of Mismanaged Trash, EPA, 23-May-2017. [Online]. Available:
https://www.epa.gov/trash-free-waters/impacts-mismanaged-trash.
[2] R. Ahmad, Detrimental Effects of Littering, EcoMENA, 15-May-2018.
[Online]. Available: https://www.ecomena.org/littering/.
[3] Wildfire Causes and Evaluations (U.S. National Park
Service), National Parks Service. [Online]. Available:
https://www.nps.gov/articles/wildfire-causes-and-evaluations.htm.
[4] States with Littering Penalties, National Conference of State Legis-
latures. [Online]. Available: http://www.ncsl.org/research/environment-
and-natural-resources/states-with-littering-penalties.aspx.
[5] What We Do, Keep America Beautiful. [Online]. Available:
https://www.kab.org/about-us/what-we-do.
[6] Vasa, H., Google Images Download, (2015), GitHub repository,
https://github.com/hardikvasa/google-images-download
[7] Tzutalin, D., LabelImg,(2015), GitHub repository,
https://github.com/tzutalin/labelImg
[8] Kaiser, L., Tensorflow Models, (2016), Github repository,
https://github.com/tensorflow/models/
[9] Arhnbom, M., Ensemble-ObjDet, (2017), GitHub repository,
https://github.com/ahrnbom/ensemble-objdet
[10] Berlin, N. Hutting, J. and Runge R., Hack-a-Drone, (2017), GitHub
repository, https://github.com/Ordina-JTech/hack-a-drone/