Seminar Title 2019-20

Chapter 1

INTRODUCTION
Detecting the objects which are stationary or rather are kept at one particular location for
a long duration without disturbing are determined as abandoned objects. Detecting it by
processing the video through various operations and obtaining a well-defined object with its
accurate specifications and alerting about it to the user controller is the main objective of
Abandoned Object Detection Systems. Discriminating between humans and non-human in order
to avoid false alarms can be done through learning algorithms. Reduction of false alarm rate is
done by feeding the system with information to help avoid non-emergency cases and also by
learning algorithms.

Figure 1.1: Abandoned Object in Airport

An object which may be living or non-living, if it is static for longer time and no one is
attending it then it can be an ‘Abandoned Object’. Generally, security systems like CCTV are
used to detect these abandoned objects at public places. Here the quality and the effectiveness of

Dept of CS & E MIT-Thandavapura Page 1

Seminar Title 2019-20

human feature detection are not up to the mark. A lot of research is in process in this area.
Complexity and probability issues are the important factors for the implementation. Previous
work shows implementation of algorithms to minimize noise but it was not as remarkable as the
noise was still creating difficulties in obtaining the output. In the research work the noise
reduction is done by Gaussian Blur algorithm-method. Also, not only the object detection but
also an effective security alert system is important. So, this proposed system is developed using
modern tools and techniques. An Android application is also developed so that the security alert
or object detection information should be given to the respective person via E-Mail.

Figure 1.2: Attended v/s Unattended object

When a solitary bag is identified, the system considers its history to find out the possible
bag locations. After capturing the successive frames from the video, the system keeps a watch on
the bag locations, whose occurrence in or disappearance from the sense calculate the status of the
bag, and decides the suitable action. This system can be used by security guards so that it will

Dept of CS & E MIT-Thandavapura Page 2

Seminar Title 2019-20

help them to analyze critical situations and take action accordingly. If a person is on railway
station, was moving but after some time he sleeps, in this case previous system may consider
human also as an abandoned object so results are ambiguous. So, in such cases using the
algorithms like RGB to HSV conversion, skin color thresholding etc we can differentiate the
person and object. Classification of abandoned object is complex task. Results may get confusing
because of the wrong object Identification problem. The main purpose of human body motion
analysis is to identify human motion with its behavior form the background image in video
sequence of CCTV and has been used in technologies like intelligence control, human computer
interaction motion analysis. So, identify the human feature with good accuracy is the motivation
behind implementation of the proposed system. Image processing methods are used in this
proposed work along with modified algorithms to get the desired results.

Chapter 2

LITERATURE SURVEY

Dept of CS & E MIT-Thandavapura Page 3

Seminar Title 2019-20

2.1 Paper Study

2.1.1 A Novel Algorithm for Abndoned Object Detection

Published on 2017

Weilong Huang, Lizuo Jin, Yi Luo, Yawei Li, Tong Cui

Abandoned object detection is a very crucial problem in many computer vision

task.This paper propose a novel algorithm based on changes detection and blob
seperation.These algorithms include detection and seperation strategies.

Methodology: The proposed model is based on the background modeling and change
detection. The pipe line of abandoned object detection and object seperation is described
in the figure shown below.

Figure 2.1: Pipeline of Abandoned Object Detection

Steps followed in this methodologies are:

Dept of CS & E MIT-Thandavapura Page 4

Seminar Title 2019-20

 Background modelling: Deals with whether the pixel identified is the background
or not. Firstly, the frame is captured and then model is used as classifier in each
frame for each pixel.

(a) Initialize a background model B0 i,j for each pixel, where 0 ≤ i ≤ m−1,

0 ≤ j ≤ n−1.

(b) For each pixel in current frame Iti,j, if Iti,j ∈ B0 i,j, then (Iti,j is classified as a
background pixel, otherwise it is identified as a foreground pixel.

(c) For each newly classified background pixel Iti,j, use it as a new train sample to
update B0 i,j.

(d) For t = t + 1, go to step (b). Where B0 i,j is regarded as a mixture of Gaussian

distribution in RGB color space. In the model, pixel is represented by its own
color. A learning rate λ ∈ [0,1] is applied to update B0 i,j in step (c) with B0 i,j
and (1−λ)Iti,j.

 Change Detection: Image subtraction is direct method to detect the change

between two images.But change detection is always influenced by noise, cluster
and so on. Histogramoforiented gradients (HOG) describes the distribution of
image gradients on different orientations. Correlation and different between two
images and binary map are defined by the formulae,

 Blob Seperation: For the detection binary map, each blob Blobt(i) works as a
candidate of abandoned objects and followed by its properties, i.e. mass center
Ct(i), size St(i) and bounding box Rt(i).

Dept of CS & E MIT-Thandavapura Page 5

Seminar Title 2019-20

Conclusion: In this paper, we propose a novel detection algorithm based on change

detection and blob separation. HOG descriptors and smoothed frame are calculated
among all the positions as representations. Correlation maps are calculated sequentially
with all the frames, with the correlation map, the blob detection method is used to find the
candidates. Our proposed approach suits more practical scenarios in which objects
located near each other.

2.1.2 Abandoned Object Detection based on Tachograph Video

Published on 2016

He Ma, Tiantian Zhao, Jinghan Li, Wei Qian

In this paper, a novel abandoned object detection based on tachograph is

proposed. Firstly, we use the Harris-SIFT features and particle filter to achieve object
tracking. After that, the VIBE algorithm is applied to detect the abandoned object with
background modeling frame by frame, using both background subtraction method and
frame difference method.

Methodology: Here, the methodology involves the object tracking and abandoned object
detection; which inturn has sub steps involved.

 Object Tracking: In this section, both the particle filter and Harris-SIFT features
are used to achieve moving object tracking. First of all, the particle filter is
applied to predict the candidate area in the current frame. Furthermore, the Harris-
SIFT feature descriptor, extracted from the candidate area, is constructed. Finally,

Dept of CS & E MIT-Thandavapura Page 6

Seminar Title 2019-20

the vector is updated according to the matching between object model and the
feature points in the candidate area during the tracking process.

 Abandoned Object Detection: At the beginning with, we use VIBE algorithm to

generate the background model frame by frame according to the target vehicle
obtained from object tracking. Then, the background subtraction method is used
to detect the moving foreground region. The frame difference method is then
applied between the foreground region of the previous frame and the foreground
region of current frame. Finally, we apply the OTSU algorithm to detect the
frames during the duration when the object is abandoned.

Conclusion: In this paper a novel method was presented to detect vehicle abandoned
object. The detection algorithm was divided into three parts, at first using SIFT feature
and particle filter to achieve object tracking, after that, both VIBE algorithm,
backgroundsubtractionmethodandframedifferencemethod were used to obtain the object
area. Finally the OTSU algorithm is applied to detect the frames during the duration
when the object is abandoned. The experimental results illustrate the proposed algorithm
is accurate and robust under different and complex conditions.

2.1.3 Using RGB-D Sensors for the Detection of Abandoned Luggage

Published on 2017

Dept of CS & E MIT-Thandavapura Page 7

Seminar Title 2019-20

M.Ajami,B.Lang

In this paper we present a novel approach for the detection of abandoned luggage
in an intelligent video surveillance system. Thenoveltyof this approach is in combining
the data of 2 sensors, the RGB sensor and the depth sensor, to achieve a robust
foreground segmentation. The novelty also lies in the use of the RGB sensor to extract
the features of the suspected abandoned piece of luggage and verify them in the following
frames. The tracking algorithm of the depth sensor is used to detect users and therefore
eliminate persons from the list of suspected objects and reduce the amount of objects to
be observed and analysed.

Methodology: The methodology of automatically acknowledging and verifying an object

as an abandoned luggage can be divided into two steps.

 Object Segmentation: The region of interest (ROI) which contains an object that
ought to be observed (suspicious objects) has to be identified in each new frame
over a specific period of time. Thereafter, the location and the features for each
suspected object in a frame are saved for later verification in the next step.

 Feature Detection and Matching: A continuous analysisis applied, in order to

detect whether the suspicious objects detected in step one were moved, had
contact with humans, or are still in the field of vision over time.

Both steps make use of the RGB- and depth-data of the Xtion PRO LIVE sensor
(Xtion PRO LIVE is equipped with an infra-red sensor, adaptive depth detection
technology, and a RGB sensor), to increase the robustness of the whole system and to
avoid false positive outputs. If an object was detected as suspicious, an alarm will guide
the attention of the security personnel to the specific scene, where the potential piece of

Dept of CS & E MIT-Thandavapura Page 8

Seminar Title 2019-20

unattended luggage is to be found. To explain the used methods and algorithms, the two
steps of this feature are described in detail.

Conclusion: The main challenge for the detection of the unattended luggage is isolating
the luggage from the background and the dynamicobjects, since background subtraction
techniques are influenced by changes in dynamic scenery over time. Through the usage of
two different sensor types, three background models for each sensor and a coordinated
update mechanism for each background model, a very robust system was created. The
usage of SURF features and the storage of the exact position of an object make it possible
to track unattended luggage even in crowded situations. The suggested system is a
reliable method for detecting unattended luggage over along period of time and is able to
cope with lighting changes, and crowded scenes, independent from the shape and colour
of potentially dangerous objects. The usage of a system like this, could improve the
reaction times of the security personnel, who can in turn initiate actions to save lives,
which are threatened by potential explosive devices hidden inside these pieces of
luggage.

2.1.4 A Real Time Abandoned Object Detection Hardware Approach

Published on 2017

Kethaki shet-Talathi, S.V.Khobragade

Dept of CS & E MIT-Thandavapura Page 9

Seminar Title 2019-20

Most of th ebomb blast was done with the help of abandoned objects, so effective
and efficient detection and real time localization of abandoned objects is very important
to prevent attacks. This paper presents an effective hardware implementation approach or
abandoned object detection in video surveillance. We had combine long term and short
term background model for foreground extraction. Change detection is done with the help
of fuzzy clustering using log ratio and mean ratio operators. In proposed system SVM
classifier is used to classify detected static object and its location is traced by GPS,
further alert process is handled by embebded module. Overall communication is done
through Internet of things.

Methodology: This system is proposed for real time abandoned object detection and its
addressing using IOT for enhancing our public security with th ehelp of PIC controller
and alarm system.

Figure 2.2: Block Diagram

 Image Acquisition: We are considering the live video as an inputto the proposed
system. In this module, we are going to capture the video through the camera anf
it will be given as an input to the system.

 Static Object Detection: We had used log operator for detecting change. Change
detection is nothing but process odf recognising difference between two images.
The logarithmic operator is characterised by enhancing the low-intensity pixels
while weakening the pixels in the areas of high intensity. Clustering involves the

Dept of CS & E MIT-Thandavapura Page 10

Seminar Title 2019-20

task of dividing dta points into homogeneous classes or clusters so that items are
in different classes are as dissimilar as possible.

 Classification of Detected Object: With the help of State Vector Machine (SVM)
classifier we classify different object based on their feature because abandoned
objects may have different abitarary shape and color.

 Raising Alarm and Display Results: Whenever abandoned object is detected by th

esystem,then the alarm will be turn on its location will have tracked by GPS
module and feed to embedded module for displaying on LCD and furture
communicataion through IOT.

Conclusion: As accuracy of abounded object detection is completely depends on binary

difference images more focus needs to be given on it. More features can be extracted in
order to reduce false alarm rate. During the implementation of our project we studied
different algorithm like sSVM. Fuzzy clustering and veila johns. We also studied
different image morphological operations. We also observed that with the use of SVM
Classifier accuracy of classification increased by large extent.

Chapter 3

METHODOLOGY

3.1 System Architecture

Dept of CS & E MIT-Thandavapura Page 11

Seminar Title 2019-20

This system is to be able to identify abandoned object in crowded places using an

accurate and reliable method e.g. market and airport where it is not easy to monitor number
of cameras for the security guards. In the existing methodology if the duration of no
movement by a human is high then he is considered to be an abandoned. Hence in the
proposed system human feature detection is carried along with abandoned object detection to
specify whether the detected object is human or an abandoned object.

The proposed system is based on the following algorithms:

(1) Blurring,

(2) Thresholding

(3) Detection of blob

(4) Motion detection

(5) Conversion of RGB to HSV

Dept of CS & E MIT-Thandavapura Page 12

Seminar Title 2019-20

Figure 3.1: System Architecture

Dept of CS & E MIT-Thandavapura Page 13

Seminar Title 2019-20

3.2 Algorithm Details

3.2.1 Blurring

Blurring of an image is usually used to reduce image noise and reduce detail.
Image quality is degraded when the blurring downgrades the high frequency components
of an Image. A color blur is possible through the output of blurring an image by a color
function which is used in graphics software. For enhancement of image structures at
various scales like scalespace representation and scale-space implementation, color
smoothing is used.

Figure 3.2: Blurring of image

Algorithm:

1. Obtain the Image from the video

2. Formulas for extracting the color values of each pixel

sumBlue = sumBlue +col & 0xff;

sumGreen = sumGreen +(col >> 8) & 0xff;

sumRed = sumRed +(col >> 16) & 0xff;

3.2.2 Thresholding

Dept of CS & E MIT-Thandavapura Page 14

Seminar Title 2019-20

Thresholding is method used to remove an intended object or target object from

its background image by allocating an value of intensity T(threshold) for every pixel such
as each pixel is either categorized as an background point or a object point. In the image
processing, the background image pixel’s gray level and current object pixel’s gray level
gets vary. Generally intensity is the easiest property which is shared by pixel in particular
region. So thresholding is normally used for the segmenting light and dark regions. Here
image is converted in to binary image by changing all of the pixels below certain
threshold value to ZERO and all above threshold value to ONE.

Figure 3.3: Thresholding

Algorithm:

1. Calculate the average value of red green and blue

2. If average value is less than threshold value then set white colour to the pixel, which is
required feature of the image.

3. Else make it as black

3.2.3 Detection of blob

Dept of CS & E MIT-Thandavapura Page 15

Seminar Title 2019-20

Blob detection refers to mathematical methods that are intended for identifying
the images with difference in image features such as contrast, brightness etc., If the
properties of digital image either constant or vary within a range it is considered to be a
blob; The points present in the blob can be calculated if they are having some common
properties. Further processing requires the interested regions which alert the presence of
entities in object tracking and is obtained by blob detection.

Figure 3.4: Blob Identification

Algorithm:

1. Group the white pixel by scanning the first line of the image
2. If white pixel found the repeat step 1 for every line to obtained a collection of blobs
3. The area of the blob get calculated by counting number of pixels
4. The extreme values of x and y get calculated by bounding box; and calculate the center
point.

3.2.3 Motion Detection

Dept of CS & E MIT-Thandavapura Page 16
Seminar Title 2019-20

The process of detection of changes in the location of an entity in reference to its

environment is called motion detection. Motion detection is identified by the difference
between in location of the object from current frame and previous frame. Motion
Detection can be executed by calculating the Center of Gravity (COG) of the object in the
captured frame. Motion Detection is done to recognize whether the object in the frame is
static or moving.

Figure 3.5: Motion Detection

Algorithm:

1. Calculate Center of gravity

2. Calculate the Cx and Cy by averaging the minimum and maximum values x and y
coordinates respectively
3. Extract Cx and Cy from previous frames

Dept of CS & E MIT-Thandavapura Page 17

Seminar Title 2019-20

4. Compare and analyze the values of x and y coordinates

5. Motion detected.

3.2.3 Conversion of RGB to HSV

If any person is moving from one place to other, but after some time he stops and
sleeps then system may recognize person as abandoned object. To avoid such a situation,
human detection is important. Here this proposed system will detect the human being as
non abandoned object which provides us with accurate results. The RGB model stands
for Red Green Blue, in which three colors are added together in different ways to
reproduce more number of colors. Hue, Saturation, Value or HSV is a color model that
represents colors in terms of their shade and brightness.HSV is represented in a number
of models, the most common being the cylindrical representation.

Figure 3.6: RGB to HSV

Dept of CS & E MIT-Thandavapura Page 18

Seminar Title 2019-20

Algorithm:

1. Crop the grabbed image

2. Extract the RGB value of each pixel.

3. Analyse the max and min value from the RGB value

4. Set V=max

5. By checking the V value compute H and S values

6. If V=0 then H=S=0.

7. Apply skin colour thresholding using the HSV value.

8. Hence Detect Object if Human or NOT.

Dept of CS & E MIT-Thandavapura Page 19

Seminar Title 2019-20

3.3 Security Alert System

Figure 3.7: Security alert system

After detecting the abandoned object it is to be informed to the security in charge for
attending that object so as to take further immediate action on it if it is dangerous. For example,
if more than 100 CCTVs are mounted at railway station for surveillance system. It is not possible
to keep a track of every CCTV. For tracking every CCTV more man power is required for
monitoring the CCTV, it increases the costing. Solution for this situation is sending SMS or
email to security guard. Security guard will get image and location of the abandoned object
through email/SMS. He will directly approach to the object and will take appropriate action. It
reduces Man-power and Cost which makes the system more efficient and reliable. After
differentiating the non living and living object system will get actual abandoned object, In case
of reporting to the security guard steps are

1. Raise the alarm

2. Save the updated image on the server

3. Send that saved image to the security guard on his android phone

Dept of CS & E MIT-Thandavapura Page 20

Seminar Title 2019-20

4. Send information via Email

3.4 Implementation Details

The proposed system is implemented using following Software and Hardware:

Table 3.1: Hardware and software requirement

Dept of CS & E MIT-Thandavapura Page 21

Seminar Title 2019-20

Chapter 4

Result Analysis

After the implementation of the complete system the desired results are achieved,
which are shown in the following screenshot. Here the abandoned objects are captured and
shown by RED rectangle and humans in some scenarios are shown by BLUE rectangle. The
results achieved for the different scenarios mentioned as Scenario-1 to scenario-10. Scenario’s
considered are as follows:

SCENARIO No SCENARIO DESCRIPTION

S1 Environment is a closed room, object is kept
on a lower level, clean background, object is
smaller in size.
S2 Environment is a closed room, object is kept
on a certain height. Human also appeared
beside the object, human features detected.

S3 Environment is a corridor staircase, object is

placed at a distance from the focus of the
camera, human appeared sitting on the stairs
(marked in blue rectangle)
S4 Environment is a table, smaller object, human
hand appeared and detected (marked in blue)
S5 Environment is a closed room with medium
light intensity, smaller object, human appeared
but not detected.

Table 4.1: Scenario for System Testing

Dept of CS & E MIT-Thandavapura Page 22

Seminar Title 2019-20

Figure 4.1: Scenario 1 Figure 4.2: Scenario 2

Figure 4.3: Scenario 3 Figure 4.4: Scenario 4

Figure 4.5: Scenario 5

Dept of CS & E MIT-Thandavapura Page 23

Seminar Title 2019-20

Following table will provide the details of all 10 scenarios. In each scenario the
Thresholding, Background count (BG Count) which is difference between background and
foreground, Buffer BG Count which is difference between current background and buffered
background, Blob size are considered with different values.

Scenario T BGC BBGC BS Time(s) Human

No Feature
Detection
(%)
S1 30 5 10 20 34 NA
S2 50 5 10 10 77 3.518
S3 40 7 10 10 53.8 Detected
and shown
by blue
rectangle
S4 40 15 15 20 43 Not
detected
clearly
S5 42 6 10 11 43 5.137

Table 4.2: Result Analysis

In Table 4.2, T - Thresholding
BGC - BG Count
BBGC - Buffered BG Count
BS - Blob Size
Time - Time required for detection (Sec)
Thresholding, Background count (BG Count) which is difference between background
and foreground, Buffer BG Count which is difference between current background and buffered
background, Blob size are considered with different values. In some scenarios as human is not
there in the actual scenario so human feature calculation is not applicable. Here we can see that
in Scenario-2 (S2) the human feature % is calculated as 3.518% and in scenario-5 (S2) is
5.137%.

Chapter 5

Generalization and Technical Limitations

Dept of CS & E MIT-Thandavapura Page 24
Seminar Title 2019-20

5.1 Advantages

 Abandoned object detection is very useful in the crowed areas specially where the area is
security alerted.
 Abandoned object is tracked through out in the streaming so that the object is not left out.
In case of emergency, abandoned object picture and its location is sent to the security in
charge, so that it is attended.
 An abandoned object might not always be a bomb or an explosive, it might also be a
luggage lost. Tracking of such luggage helps the person to get it back soon and avoid
theft.
 Human detection reduces the false alarming of abandoned object found.
 It is less expensive as the security alert is through smart phone application or email.
 Reduces the personnel requirement.

5.2 Drawbacks

 Minute detection is failed.

 Detection area is restricted to the area of camera coverage.
 Human bombs are undetected.
 False alarming can waste lot of time and energy.

Conclusion

Dept of CS & E MIT-Thandavapura Page 25

Seminar Title 2019-20

The abandoned object and human feature detection system is proposed along with
security using Android application. This proposed system is based on algorithms like Blurring,
Thresholding, Detection of blob, Motion detection and conversion of RGB to HSV. System
works for the real time environment and capturing of live streaming videos. The goal of this
project is to distinguish human’s features from arbitrary objects and raising security alarm.
Updated Image will be sent to Security person with Smart phone having Android Application
and also via E-Mail. This system may be more time efficient than the older systems. In future the
system can be made more robust and accuracy with image quality may be enhanced.

References

Dept of CS & E MIT-Thandavapura Page 26

Seminar Title 2019-20

[1] Kevin, Lin, Shen-Chi Chen, Chu-Song Chen, Daw-Tung Lin, and Yi-Ping Hung,
“Abandoned Object Detection via Temporal Consistency Modeling and Back-Tracing
Verification for Visual Surveillance” IEEE Transactions on Information Forensics and
Security, 2015.
[2] Weilong Huang, Lizuo Jin, Yi Luo “A Novel Algorithm for Abandoned Object Detection”
August 2017, International Conference on (ICA)
[3] He Ma, Tiantian Zhao, Jinghan Li, Wei Qian “Abandoned Object Detection Based on
Tachograph Videos” 2016 IEEE
[4] K. Muchtar, C. Y. Lin, and C. H. Yeh. Grabcut-based abandoned object detection. In
IEEE International Workshop on Multimedia Signal Processing, 2014.

Dept of CS & E MIT-Thandavapura Page 27