Balaji PPT Final

"All progress is born of inquiry.
Doubt is
often better than overconfidence, for it leads
to inquiry, and inquiry leads to invention"
- Hudson Maxim
Welcome to
Pre-Ph.D.
presentation
Tuesday, December 24, 2019
ROAD TRAFFIC SURVEILLANCE SYSTEM TO CLASSIFY THE VEHICLES ON INDIAN ROADS 1
Road Traffic Surveillance System to classify
the Vehicles on Indian Roads using Motion
based Segmentation
R.Balaji Ganesh
Reg. No.: 10489
Ph.D., Scholar – Part Time
(External Category B2)
Guide:
Dr.S.Appavu alias Balamurugan
Professor and Head
K.L.N. College of Information Technology
Madurai - 630612
2
Tuesday, December 24, ROAD TRAFFIC SURVEILLANCE SYSTEM TO CLASSIFY THE VEHICLES ON INDIAN
2019 ROADS
About Me...
 Presently working as Assistant Professor (CSE) in
Ramco Institute of Technology, Rajapalayam
 Received Masters in Technology (CIT) in

Manonmaniam Sundaranar University in the year
2011
 Received Bachelors in Engineering (CSE) in PSR

Engineering College in the year 2006
 Currently pursuing Ph.D. in Information

Communication Engineering (Part Time External
Category B2) Date of Registration: 18.06.2013
Tuesday, December 24, ROAD TRAFFIC SURVEILLANCE SYSTEM TO CLASSIFY THE VEHICLES ON INDIAN ROADS 3
2019
AGENDA
Motivation
Introduction
Literature Review
Proposed Solutions
Results and Discussions
Performance
Conclusion
List of Publications
2019
Motivation
 With an ever-increasing vehicular traffic on urban cities roads, the

significance of Intelligent Transportation System (ITS) is inevitable.
 Today, ITS is benefitting most from video-based surveillance systems

by extracting and analyzing information useful for traffic planning and
security with diverse applications including vehicle counting, vehicle
tracking, vehicle trajectory, vehicle classification, vehicle velocity,
queue length, license plate recognition, traffic density, traffic lane
change etc.
 However, developing a reliable and efficient video-based vehicle

detection system is quite challenging and is a growing field of research.
A promising video-based detection system must handle environment
dynamics efficiently. It must be adaptive to changes in scene
illumination and weather conditions.
 Jittering camera or noise contamination due to wind are practical issues

being faced in vehicle detection
ROAD TRAFFIC SURVEILLANCE SYSTEM TO CLASSIFY THE VEHICLES ON INDIAN
ROADS
2019
 Vehicle shadows under sunlight are also quite
challenging to address as long shadows cause
occlusion problems and thus incorrect classification
in many cases.
 Similarly, at night time headlights and low illumination

poses accurate detection problems. Therefore,
detection of moving vehicles under such scenarios is
an important yet demanding task
2019
Tuesday, December 24, 2019ROAD TRAFFIC SURVEILLANCE SYSTEM TO CLASSIFY THE VEHICLES ON INDIAN ROADS 7
INTRODUCTION
 Traffic problem is an important issue happening in many cities in India.

There are many important causes of the traffic problem.
 The number of people moving into an urban area has grown

substantially, leading to a dramatic increase in the number of vehicles.
However, roadway capacity has grown relatively slow and become
insufficient.
 This causes an imbalance between the numbers of vehicles and roads,

resulting in road traffic congestion, especially in large cities.
 A video sequence of road can be processed and analyzed to

detect and count vehicles.
 Further information, such as the speed of a vehicle or traffic

density, can also be calculated by using computer vision.
2019
Who can benefit from this???
 This would directly benefits to two groups of people:
 Road users
 Traffic administrations, Traffic Cops
 If road users know the real-time traffic information, they can then use
the information to choose the best way for traveling and can avoid
traffic congestion.
 On the other hand, traffic administrations can utilize the traffic

information in their traffic control systems, resulting to a better traffic
management.
2019
Computer Vision and Traffic Surveillance Systems
 Nowadays, computer vision - an analysis and interpretation of
images and videos captured by a digital camera - has gained
more popularity and been used in many fields including industry,
robotics, medicine, etc.
 Computer vision can also be applied for solving traffic and

transportation problems.
 Traffic surveillance systems aid commuters, provides valuable

data for traffic cops, traffic infrastructure managers, enforces
laws, encourage safe driving. Despite all these things the
leverage of software systems for traffic management has
emerged.
2019
Why need classification of Vehicles
 Efficient and robust localization of vehicles from an
image sequence (video) can lead to semantic results,
such as ‘car is stopped,’ ‘Bus is moving faster than
Truck.’ However, such information can be more
relevant if we not only can detect vehicles but also
can classify them.
 Information such as class of a vehicle can be useful

for intelligent transportation systems
 Object detection and tracking has achieved good

accuracy in recent years.
 However, the same cannot be said about object

classification.
2019
Why Traffic Video and Why not Traffic Image
 Object recognition in case of still images has
the problem of dealing with the clutter in the
scene and a large number of classes.
 Object recognition in video sequences has the
benefit of using background segmentation to
remove clutter
 Frames obtained from video surveillance
cameras are generally of low resolution.
 Incase of traffic video surveillance, the vehicles
cover very small areas of these images, making
the classification problem challenging.
Global Features for Vehicle Classification
 Vehicle classes such as cars and vans are
difficult to differentiate as they have similar
sizes.
 Therefore, classification techniques that use
global features such as size and shape of the
detected blob do not yield satisfactory results.

Challenges in Vehicle Classification
 However, developing a reliable and efficient
video-based vehicle detection system is quite
challenging and is a growing field of research.
 A promising video-based detection system must
handle environment dynamics efficiently
 It must be adaptive to changes in scene
illumination and weather conditions.
 Vehicle shadows under sunlight are also quite
challenging to address as long shadows cause
occlusion problems and thus incorrect
classification in many cases.
Challenges in Vehicle Classification
 Similarly, at night time headlights and low
illumination poses accurate detection problems.
 Therefore, detection of moving vehicles under
such scenarios is an important yet demanding
task
 This research primarily focuses on
contemporary moving vehicle detection and
segmentation techniques, leaving out camera
calibration approaches and vehicle tracking
methods.

Categorization of Vehicle
Detection and Segmentation
Approaches

Categorization of Vehicle Detection
 With advancement in image processing and
computer vision techniques, much of the
research has been done in the field of moving
object’s regions of change detection among
multiple captured image sequences
 vehicle detection and segmentation techniques
approaches are classified based on
 Background Subtraction
 Feature Extraction-based
 Motion-based

Background Subtraction Methods
 In this method, moving objects are extracted as
‘foreground’ from each frame by taking an absolute
difference between the current frame and the reference
frame called ‘Background’ frame or model
 The essence of the method lies in the accurate
estimation of Background model for which both non-
adaptive and adaptive modelling techniques are
available. (Xie, L., Zhu, G., Wang, Y., Xu, 2005)
 Since non-adaptive methods suffer from a change in
illumination and climate conditions, adaptive modelling
is preferred

Background Subtraction Methods - Limitations
 Despite Temporal median or Moving Average

method, BS methods require large memory
buffers for its computation (Lo, B.P.L., Velastin,
S.A., 2011, Monnet, A., et al.:, 2003)
 Threshold value Th is non-adaptive and is same
for all pixels in frame.

Vehicular Feature Based Methods
 This method segment moving objects from
background image by detecting vehicle’s
inherent visual features like its colour, edges,
contour, texture or body part such as head
lights (Guo, D., Fraichard, T. et al., 2000)
 Since the method does not require a vehicle in
motion, it can detect stationary vehicle as well.
 A trainable system for certain class of vehicle
detection without using motion, tracking to
handcrafted models in unconstrained, cluttered
scenes was a breakthrough (Papageorgiou,
C.P., Poggio, T., 2008)
Motion Based Methods
 The vehicular motion observed from a static
camera seems as pixels in the image to be
moving
 In optical flow, movement of each pixel is
calculated by measuring temporal changes of
the pixel, and their correlation in an image
sequence
 vector field of this motion is referred as Optical
flow

 Motion based vehicle detection methods trace
these flow vectors in 2-D which are produced
due to vehicle motion velocity vectors in an
image sequence
 This approach can even detect independently
moving vehicles from the camera.
 With the use of specialized hardware and GPUs
this Motion based methods yields good results

 Frame differencing is another motion-based
detection approach which involves subtracting
two consecutive frames by pixel in an image
series to obtain moving object pixels
 Subsequently the moving foreground area is
calculated by setting a threshold
 Variants of the approach with
(Zhao, et al., 2006)
 Three-Frame-Differencing
 Multi-Frame-Differencing

 It cannot cope with noise, abrupt illumination
changes, or periodic movement in the
background such as trees
 Zhang et al (2010), uses two step algorithm for
adaptive background update and motion
histogram-based vehicle segmentation
 In adaptive background update, lighting
changes of the scene evolve the background.

Night Time and
Complex Weather Conditions

Night-Time Traffic Surveillance
 The methods above discusses can be efficiently
applied to daytime traffic scenes with stationary
and unchanged lighting conditions
 Spatial-temporal difference features are no
longer reliable when vehicles stop or move
slowly in congested traffic areas, and vehicles
may be falsely detected as background regions
and missed.
 In poorly illuminated or nighttime conditions,
background scenes are substantially affected
by the lighting effect of moving vehicles.
 For night time surveillance videos, foreground
objects that can be detected are usually
headlights, auxiliary lights, or reflections of
lights of the vehicles.
 Locating and pairing headlights are important
for nighttime surveillance
 The Steps in Night-time Traffic Surveillance are
 Headlight segmentation and detection
 Headlight pairing
 Vehicle tracking
 Vehicle counting and detection.
 Zhang et al. (2012), proposes a system to
detect and track vehicles using an effective bi-
directional head light reasoning algorithm.
 The accuracy rate of headlight detection was
95.2%, the vehicle tracking rate was only 88.2%
 However, it is difficult to segment out the entire
vehicles at night
 Headlights are extracted based on the analysis
of the light attenuation model.

In Complex Weather Conditions
 Chen, Z., Ellis, T., (2012) proposes improved
version of the background subtraction method
to alleviate the negative impacts from gradual
changes.
 Finally level set and Kalman filter and support
vector machine are utilized to improve the
accuracy of vehicle classification

Problem Statement
 The main goal of this research work is to
implement an efficient method for recognizing
vehicles in Indian conditions.
 There are some challenges in these conditions
that makes
 vehicle recognition in Indian roads is much
harder.

Problem Statement
The two important challenges in vehicle recognition to apply for Indian scenario
Row1: Complex road conditions (non lane system, heavy loaded vehicles, un-
structured vehicle parking system and obstacles on roads respectively)
Row2: Intraclass variation among vehicles, here Truck is shown as an example.

• In India, most of the cases traffic system is non-
lane based.
• Road conditions are more varied and traffic is
unstructured, there is lack of discipline and
overloaded vehicle movement is quite natural.
• In India, vehicles are parked frequently by the
sides of the roads. There is no separate system
for vehicle parking management.
• Roads are not only occupied with vehicles, so
many obstacles on roads create disturbance to
the traffic
• Pedestrians do not have separate ways for
their movement in most of the cases.
• Shapes of the vehicles have a key role in
recognition, there is high intra-class variance
among Indian vehicles.
• It creates the chances for miss recognition.
• With in same vehicle class there are large
variety's and models.

Proposed Solutions
 Solution I
 A Robust Framework to detect moving vehicles
in different Road conditions
 Solution II
 Road Traffic Surveillance System to Classify the
Vehicles on Indian Roads in Different Climatic
Conditions

Solution I
 The system tries to solve the problem of counting and classifying
the vehicles in Indian road conditions.
 The system uses color image based foreground moving object
detection by preserving the color and model of the moving
vehicles.
 The color image based background subtraction technique is
supported by cascaded linear regression.
 The system also uses HoG for contour creation and extraction
followed by morphological dilation to connect the missing pixels in
the vehicle object.

Pre processing of Video
 Extracted frames obtained from the videos are
given as the input to Laplacian of Gaussian
filtering
 Boundary coefficient is extracted for each row
and column of the frame.
 Then, the normalized coefficient is extracted
based on the fitness value, and the logarithmic
value is calculated for the frame.
 After preprocessing, contrast stretching and
histogram equalization process are applied in
the enhancement stage
System Design
 The system performs in real time, such as with moving
casted shadows on sunny days, headlight reflections on
the road, rainy days, and traffic jams, using only a single
standard camera.
 A robust adaptive multilayer segmentation strategy is
proposed which detects foreground pixels
corresponding to moving and stopped vehicles, from
noisy images (because of compression)
 The proposed approach generates an adaptive
threshold as a combination of luminance and
chromaticity disparity maps between the frames

Steps in the Approach
 Step1: Read Video Input
 Step2: Preprocess Video (Frame selection,
Noise Removal, Histogram Equalization using
bilateral filtering)
 Step3: Frame Blurring to select ROI & Image
Enhancement
 Step4: Background Subtraction Using
Multilayer Background Subtractions Model

 Step5: Image In-painting using MoG to Separate
Background & Foreground Moving Objects
 Step6: Cascaded Regression to Improve Detection
Accuracy of Foreground Objects
 Step7: Contour Creation of Blob Detection to the
Moving objects Frames
 Step8: Count the Number of Blobs Result in
Number of Vehicles From Image Frames
 Step9: Finally the counts of the vehicles in each
frame to be saved in Log file (debug.log)

Foreground Mask and Edge
 All detection techniques are based on

modeling the background of the image
 In defining the background it is assumed that
the stationary objects could vary in color and
intensity over time
 In addition to processing in real time, systems
need to be able to adapt to these changes.

Multi-layer Background Subtraction Algorithm
 Both adaptive and Non-adaptive methods

require large buffer memory for computation
 It also suffers from local minima problem
 To address these issues, new multilayer
segmentation architecture is proposed which
fuses different image cues, and combines
bottom-up and top-down strategies to solve
global/local illumination changes, and to obtain
conditional background model learning.

Steps in the ML BS Algorithm
 In this algorithm, pixel vectors are formed each
for luminance and chrominance.
 High level noises are removed in the pixel
vectors using bilateral filtering.
 Intensity levels of the pixels are compared for
the shadow, white, foreground, vehicle masks.
 The shadow and background masks are
combined for improving the efficiency of the
algorithm.

Steps in the ML BS Algorithm
 Sobel edge detection algorithm is applied to
retain the edges of the highlighted vehicle
objects in the image.
 Watershed segmentation is used to identify
and fill the foreground, vehicle and white blobs
in the foreground image.
 Finally mask is reinforced in the direction the
shadow.
 Background mask is updated and the mask is
returned to perform subtraction

(Pseudo
Code)
Multi Layer
Background
Subtraction
Algorithm

Multi Layer Background Subtraction Algorithm
(Pseudo Language)
Crop highlight regions in Mask

Watershed (tf): fill foreground & highlight & white blob in
between as foreground
Reinforce Mask from shadow directions
Update B, Σl, Σc
return Mask

Contour Creation
 The purpose of contour drawing is to
emphasize the mass and volume of the vehicle
object rather than the detail.
 Contour is a cross-section of the three-
dimensional graph of the function f(x, y)
parallel to the x, y plane
 The focus is on the outlined shape of the
vehicle object and not the minor details

Contour Creation
contours, hierarchy = cv2.findContours(fg_mask) cv2.RETR_EXTERNAL,
cv2.CHAIN_APPROX_SIMPLE)
log.debug("Found %d vehicle contours.", len(contours))
matches = [ ]
for (i, contour) in enumerate(contours):
(x, y, w, h) = cv2.boundingRect(contour)
contour_valid = (w >= MIN_CONTOUR_WIDTH) and (h >=
MIN_CONTOUR_HEIGHT)
log.debug("Contour #%d: pos=(x=%d, y=%d) size=(w=%d, h=%d)
valid=%s", i, x, y, w, h, contour_valid)
if not contour_valid:
continue
centroid = get_centroid(x, y, w, h)
matches.append(((x, y, w, h), centroid))

 Once the background subtracted image is
obtained, vehicle contours needs to be found.
 Contours centroid can also be calculated and
contours can be matched to keep track of the
vehicle.
 Contour’s minimum height and width is set to
be static.
 In this work,
 minimum height = 20315 pixels (4.3 meters)
minimum length = 59055 pixels (12.5 meters)

Cascaded Regression to improve Vehicle Detection
 Generating a set of initial bounding boxes

(Contours) is the first stage in our approach.
 Reducing the set of possible boxes at an early
stage is motivated by the fact that it is not
feasible to score all sub-windows of an image
 The initial set of bounding boxes are composed
into Super-pixel windows (i.e. all pixels in a
superpixel are most likely uniform in, say, color
and texture)
 Superpixels have been shown to be strong
cues about object boundaries
Cascaded Regression results (Day Time)

Cascaded Regression results (Night Time)

Vehicle Detection
 In this work, adaptive SVM is combined with
HoG to improve the vehicle classification
 An adaptive SVM classifier is trained with HoG
Features to detect the Vehicle object

Why HoG?
 A class of objects such as a vehicle vary so
much in color.
 Structural cues like shape give a more robust
representation.
 Gradients of specific directions captures some
notion of shape.
 To allow for some variability in shapes
Histogram Oriented Gradients has been used
in this work

HoG Feature Extraction
 The idea of HOG is instead of using each
individual gradient direction of each individual
pixel of an image, pixels are grouped into small
cells.
 For each cell, all the gradient directions are
computed and grouped into a number of
orientation bins.
 The gradient magnitude in each sample is
summed up and the stronger gradients
contribute more weight to their bins, and
effects of small random orientations due to
noise is reduced.
HoG Feature Extraction
 This histogram gives a picture of the dominant
orientation of that cell.
 Doing this for all cells gives a representation of
the structure of the image.
 The HOG features keep the representation of
an object distinct but also allow for some
variations in shape.

Classifier Training
 Linear Support Vector Machine classifier
technique is used in this work
 As SVM is capable of learning high
dimensional space with a few of training data
 The basic concept of SVM is to search an
optimal separating in hyperplane, whereas it
can separate two classes

Boundary to select the Hyper plane
 The Separating Boundary chosen is
𝑁
𝑓 𝑥 = ෍ 𝛼𝑖 𝑦𝑖 𝐾 𝑥𝑖 , 𝑥 + 𝑏
𝑖=1
N is number of training data sample

0 ≤ 𝛼𝑖 ≤ 𝐶, 𝑖 = 1,2, … … 𝑁 are non-negative
parameter learned from the data
C is the penalization misclassification cost
parameter used in the training data sample

Radial Basis Function
 A radial basis function (RBF) kernel has been
chosen, in which the adjustable parameter 𝜎
has major role for the kernel in the separating
boundary to function correctly
𝑥𝑖 − 𝑥 2
𝐾 𝑥𝑖 , 𝑥 = exp(− )
2𝜎 2

Process Flow

Logical Architecture of the System
Noise Removal
Input Video Color Deduce the type of Multi-layer
using Bi-lateral
Sequence from Histogramming video using color BS model
filtering, Histogram
Camera to Select Frames Dispersion
Equalization
Count the Centroid

Morphology and Train the
vehicles and calculation,
Cascaded Contour creation background
update the log Track the
regression model
file vehicle

Composition of Video datasets
No. of videos captured in different environments
and their contribution to the Training Data set
Input % in the
Video Scenario Videos Training Data
(200) set
Single Way at Normal Scenario 50 25%
Double way at Normal Scenario 60 30%
At Snow Days 20 10%

At Winter Season 20 10%
Night Time Video 50 25%

Results and Discussions
 Out of the 200 video we have 140 video data
sets are captured during day time.
 In the Sunny sequence, the main difficulty
comes from the detection of dark vehicles that
project shadows.
 There may be some dark vehicles that do not
have sufficient gradient features to identify that
they are vehicles.

Results for Day time video
(a) Input video frame (b) Multi-cue color background

subtracted image (c) Vehicles Identified and counted in
Two-way roads in Highway roads and City roads in India

Results – Night Time video
 Out of the 200 video we have 20 video data
sets captured during night time.
 Reflection Suppression, Shadow removal &
Head light detection were applied in order to
detect the moving vehicles.
 The main difficulty in the Nightfall sequence is
correctly segmenting the headlight reflections
on the road with respect to the vehicle.

Head light pairing
Head light pairing using distribution of Head lights in the

neighboring pixels using distribution of Headlights on
Intensity, Reflection Intensity and Reflection Suppressed
Map

Reflection Suppressed Map

Head light counting and Pairing in Night Time
No. of Headlights Detected

using the Distribution of
Intensity, Intensity Map and
Reflection Suppressed
Map

Rainy Day / Snow Time
 Optimized SVM classifier is used to classify the
pixel (snow and rain streak pixel)
 The moving objects are detected using bi-
lateral filtering.
 To eliminate non-vehicles, histogram of every
detected rectangle’s background model is
calculated.
 A vehicle or non-vehicle decision is made
based on the intensity distribution of the
background model for the corresponding
rectangle
The results of the algorithm which removes the rain streaks from the input
image and identified the moving vehicles.

Results – Snow Time
The results of the algorithm which removes the snow streaks from the
input image and identify the moving vehicles.

Inferences
 We can measure the sensitivity of background
subtraction methods in detecting low contrast
targets against the learned background.
 The lower the threshold values that are needed
for detecting foregrounds, the more sensitive
the background subtraction approach will be,
and hence the better the results

Inferences
 An image of each of the four videos used for
this test:
1) a nightfall with diffuse ambient light in which
vehicles have headlights turned on
2) a sunny day where vehicles project shadows
3) a rainy day in which the sun appears and
causes vehicles to project shadows

Inferences
 The segmentation and Vehicle counting
approach is compared with other vision-based
recent alternatives
 Gaussian Mixture Model foreground detection
 Histogram of Gradients feature detection
 Detection based on adaptive motion histogram

 Precision is the ratio of vehicle detected correctly to the
total number of detections in the scene i.e.
𝑡𝑝
𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 =
𝑡𝑝 + 𝑓𝑝
 Recall is the ratio of the number of vehicle correctly
detected to the actual number of vehicle in the scene.
𝑡𝑝
𝑅𝑒𝑐𝑎𝑙𝑙 =
𝑡𝑝 + 𝑓𝑛
where, tp is true-positive, fp is false-positive and fn is
false-negative.

 The two different scenarios under which the
algorithms are tested are:
 Traffic Density: The detection accuracy
depends on the density of vehicles on the road
inside the coverage area. It can be low or high.
 Illumination Conditions: Lighting affects vision
based object detection algorithms.

Impact of Traffic Density
Algorithm Low (<4 vehicles) High (> 8 vehicles)
Precision Recall (%) Precision (%) Recall (%)

(%)
Mixture of 85.71 80.0 82.75 77.41
Gaussians
Histogram 92.0 90.19 86.79 88.49
Oriented
Gradients
SVM 85.71 88.88 81.35 84.21

Classification

Impact of Illumination Conditions
Algorithm Afternoon Evening
Precision (%) Recall (%) Precision (%) Recall (%)
Mixture of 87.17 77.27 89.47 82.92

Gaussians
Histogram 84.50 86.95 88.23 90.90

Oriented
Gradients
SVM 89.55 90.85 93.75 92.30
Classification

Solution II
Road Traffic Surveillance System to Classify the
Vehicles on Indian Roads in Different Climatic
Conditions

Introduction
 To realize road traffic flow surveillance under
various environments which contain poor
visibility conditions, this work gives the solution
to extract the required information from
surveillance video under different weather
condition like day, night and rain
 This system will dynamically choose the
respective algorithm based on identified nature
of the weather

Introduction
 This system is based on image segmentation using a
 Laplacian of Gaussian edge detector (LoG);
 morphological filtering of the edge map objects and
classification into small, medium and large vehicles
on the basis of size;
 nearest centroid minimum distance classifier.
 This approach can be used for both stationary and fast
moving traffic in contrast to motion detection based
approaches.
 First main focus to identify the environment from
captured frames. Then, system will automatically
choose best algorithm based on identified the nature of
the weather
Block Diagram
Environment
Pre-processing
Identification
Vehicle Count
Vehicle Detection
& Classification

Environment Identification Module
 The Environment Identification Module is used
to identify the type of the video as Daytime or
Night time
 A frame representing the type of video is
extracted from the video
 Top half of the Image is cropped as it mostly
contains the pixels related to the Environment
 The color model has been converted from RGB
to HSV, since luminance is the dominant
feature to identify the type of video

 Luminance is not available directly in RGB
color model. So Image is converted from RGB
to HSV model to manipulate Luminance.
 Histogram is calculated for H and V
components

Night Time Image – Hue, Value Histograms

Day Time Image – Hue, Value Histograms

Classifying the type of video
 From the H-Histogram, the amount of pixels
that falls in red and yellow color hue range are
determined as 𝑛ℎ
 From the V-Histogram, the amount of pixels in
the image that are light or bright are calculated
as 𝑛𝑣
 If 𝑛ℎ < 288 && 𝑛𝑣 > 150 the image is classified
as Day time Image
 If 𝑛ℎ > 288 && 𝑛𝑣 < 150 the image is
classified as Night time Image

Pre-processing Module
 Preprocessing will be done based on identified
environment to make sure frames are
enhanced with more information for further
processing.

Rain Fall environment
 Preprocessing the image and aim to remove visual
effects of rain (i.e.) separate the rain layer and de-
rained image layer from the rain image
 The single frame image extraction uses dictionary
learning algorithm that focus on different weather
condition
 The mutual exclusive property is applied on image
frames to remove patches that has large discriminative
codes
 Such separation of code imply non linear composition
between two images for providing accurate results

Night Vision Environment
 New incremental histogram equalization approach is
developed in which the image histogram is equalized
at each intensity level starting from the maximum
intensity found in the original image upto 255, but in
which, for each step the histogram equalized output
image of the previous step used as input.
 This system is achieved good output image quality
even for dark video frames enabling a viewer at the
client’s end to see under poor visibility conditions.

Un processed Night video frames

Histogram processed video frames

Overview of the core system
 The video frames are converted into grey scale
to simplify and facilitate subsequent
processing.
 Negatives of the image is calculated followed
by Laplacian of Gaussian (LoG) edge detection
method and mathematical morphology
 The size dimensions of the vehicles are also
extracted and used as inputs to the classifiers
at the classification stage.

Negative Transformation
 For each grey frame extracted from the video,
its negative is computed.
 This is to ensure that as much relevant edge
detail as possible is extracted in the
segmentation stage, thus minimizing spurious
edge discontinuities

Image and its Negative Image

ROI Mask Modeling
 One of the extracted frames is used to model a
Region of Interest (ROI) polygon.
 This polygon is ultimately used to mask the
processed binary frames so as to limit the
counting and classification of vehicles to those
found only within the region of interest.
 The size of this polygon is chosen empirically
such that the vehicle intra-class variations are
minimized

ROI Mask Modelling – Free flow traffic

Improvement of Segmentation performance
 The segmentation performance is improved by
compensating for non-uniform illumination of the
scene using the morphological top-hat
transformation prior to the segmentation stage.
 This is computed as
 𝑔 𝑥, 𝑦 = 𝑓 𝑥, 𝑦 − 𝑓 𝑥, 𝑦 𝑜𝑏 𝑥, 𝑦
 𝑔 𝑥, 𝑦 is the uniform background frame
 𝑓 𝑥, 𝑦 is the input frame
 𝑓 𝑥, 𝑦 𝑜𝑏 𝑥, 𝑦 is the morphological opening of
𝑓 𝑥, 𝑦 using a structuring element, 𝑜𝑏 𝑥, 𝑦
Image Smoothing and Blurring
 The uniform background frame is smoothed using a
median filter to remove random noise
 Then aggressively blurred using a Gaussian filter so as
render ‘noise’ edges into the background and therefore
reduce the chances of their detection.
 This also minimizes the effects of shadows in the traffic
scene.
 The blurred frame is contrast enhanced linearly so as
to emphasize the edges while preserving the mean
intensity values of the frames.

Image Segmentation
 To extract objects in both the frame and its negative,
the Laplacian of a Gaussian (LoG) edge detection
method is used due to its excellent edge detection
properties and relative simplicity
 The LoG of a two dimensional image is computed as

2 2 2 𝑥 +𝑦 2 2
𝑥 + 𝑦 − 2𝜎 −
𝛻 2 𝐺 𝑥, 𝑦 = 𝑒 2𝜎 2
𝜎4
 Where 𝛻 2 is a Laplacian operating on the Gaussian

smoothed image 𝐺 𝑥, 𝑦 and 𝜎 is the standard
deviation of the image pixel intensities
 The proposed approach infers the fact that
shadows are semi-transparent
 Therefore by appropriately enhancing and
segmenting the frames, shadow effects can be
greatly reduced.
 In this way, the complex and often ineffective
shadow removal algorithms are avoided

Summation – Positive and Negative Frames
 After segmentation, the positive and negative
frames are added to eliminate double counts
 It also ensures that maximum objects are
detected as possible.
 This addition is possible since the frame and its
negative are spatially registered and therefore
the objects which occur simultaneously in both
the frame and its negative reinforce each other.
 The output of the summer give the complete
edge map, and therefore the binary image of
the frame
Post-processing and Feature Extraction
 The obtained binary frame is then subjected to morphological
filtering
 The segmented binary frame is closed so as to eliminate any
spurious disjoints between connected components
 Holes in the connected components are filled to ensure that true
sizes of objects are used in subsequent stages
 The processed frame is masked using the modeled ROI mask so
as to limit the counting and classification to the objects found in
the region of interest only
 Objects that are not of interest such as roadside buildings and
vehicles moving on lanes that are not of interest
 Irrelevant small objects within the ROI such as pedestrians are
deleted using a morphological opening operation.

 The morphological opening is done after masking the
frame so as to ensure that unwanted objects on the
border of the region of interest are deleted as well.
 The consequence of this processing is that the shapes
of the vehicles are not preserved, and therefore,
cannot be used for classification
 Instead, areas of the bounding boxes of the resulting
connected components are extracted and used as
inputs of the nearest centroid minimum distance
classifier which assigns the vehicles appropriate class
labels

 The resulting connected components in the
fully processed frame represent vehicles on the
road at that time. These components are
counted to give the total number of vehicles on
the given section of the road at the given time

a b

 It should be noted that, object 5 in Fig. (b)
consists of two vehicles which are represented
as one vehicle as can be seen in Fig. (a).
 This is due to the fact that the car is occluded
from the camera’s view by the larger vehicle in
front of it.
 This scenario highlights the importance of
proper installation of cameras meant for traffic
management systems.

Vehicle Classification
 A Euclidean distance based nearest centroid minimum distance classifier
is used to classify the vehicles into three classes on the basis of their
dimensions: small, medium and large vehicles.
 For both convenience and practical reasons, five-fold cross-validation

technique was used.
 Using this method, the dataset of the extracted vehicles is split randomly
into five approximately equal subsets for cross validation.
 Each subset contains all the three classes, but not necessarily in equal
portions.
 At each of the five validation trials, one subset is used for testing while the
other four are used for training.
 Classification accuracies for the five trials are averaged to obtain the
classification accuracy of the algorithm for a given dataset.

 In order to use the nearest centroid minimum distance classifier,
the feature vectors of the vehicles present in the four training
subsets are averaged for each class at each trial.
 Therefore, in the training set, each class is represented by its
mean vector
 To classify a given unlabeled vehicle, the Euclidean distance
between its feature vector and each of the vectors representing
the three classes is calculated.

Then the vehicle is assigned to the class of the nearest centroid. This
can be simplified by evaluating the decision functions of all the three
classes for this classifier as
1 𝑇𝑇
𝑑𝑗 𝑥 = 𝑥 𝑚𝑗 − 𝑚𝑗 𝑚𝑗 𝑗 = 1,2,3 … … .
2
Where
• dj(x) is the decision function of class wj,
• x is the unknown feature vector
• mj is the mean vector representing class wj;
• x is assigned to class wj if dj(x) yields the largest numerical value.

EXPERIMENTAL RESULTS AND DISCUSSIONS
 Video data set was collected using a 12MP camera

mounted above the road under which the subject
vehicles passed.
 In order to assess the performance of the system
under various illumination levels across the day,
 the data was collected at 0630hrs before the sun is
up;
 1230hrs when the sun is overhead and the shadows
are negligible, and
 1630hrs when both reflections from the road surface
and shadows are strongest
 Each collection period lasted 20 minutes, resulting in at
least 36000 frames each time
 The three datasets from the three time periods
of the day were added to form an overall
dataset and then 5-fold cross-validated
(5 subsets)
 Data was also collected from a traffic scene
that involved very slow moving traffic so as to
assess the performance of the proposed
system on such traffic scenes or on stationary
ones

Summary of the Results
Time of Total No. Total No. Total No. Total No. of No. of Small No. of No. of large Detection Classification
the Day of of correct of wrong vehicles vehicles Medium vehicle used Accuracy Accuracy
Vehicles detections detections used for used for vehicles for
classification classification used for classification
classification
0630 220 209 11 216 132 56 28 95.0% 81.7%
hrs
1230 306 291 15 298 182 65 51 95.1% 88.0%
hrs
1630 438 425 13 438 280 78 80 97.0% 93.8%
hrs
Overall 964 925 39 952 594 199 159 96.0% 89.4%
Slow 246 202 44 224 114 73 37 82.1% 83.8%
moving
traffic

COLUMN BAR CHAR BETWEEN TOTAL NUMBER OF VEHICLES
AND VEHICLES DETECTED
1000
900
800
700
600
500
Total No. of Vehicles
400
Total No. of correct detections
300
200
100
0
0630 1230 1630 Overall Slow
hrs hrs hrs moving
traffic

Inference
 The relatively low camera position was the main cause
of detection errors since it was difficult to ‘see’ the
spaces between the vehicles on the same lane when
the vehicles involved were very close together
 For small vehicles, due to their size, their entire tops
were visible while this was not true for the other
classes where only the fronts and some part of the tops
were visible; resulting in the usage of different
dimensions for different vehicles in classification.
 This greatly reduced the ability of the classifier to
distinguish between small and medium vehicles as is
evident from the following confusion matrices

CONFUSION MATRIX FOR TEST SUBSET 1
Predicted Class Labels

Subset 1 Small vehicle Medium Large Vehicle
Vehicle
Small vehicle 117 1 0
Actual Class Medium 8 23 8
Label vehicle
Large vehicle 0 0 31
Classification accuracy 91.0%


Vehicle
Label vehicle


Vehicle
Label vehicle


Vehicle
Label vehicle


Vehicle
Label vehicle
Overall classification accuracy is

mean (91.0%, 87.2%, 89.0%, 88.0%, 91.8%) = 89.4%

Inference
 As regards to the classification accuracies
related to the times of the day, it may seem
that they improve as the day matures
 The reason behind this could be the number of
training samples and the accuracy with which
the data was collected.
 For the morning dataset, for example, the set
was much smaller than any of the other two.
 The regions of interest (ROI) masks influenced
the classification accuracy

CONCLUSION
 This paper attempted to solve the problem of
vehicle detection, counting and classification in
natural traffic scenes using video surveillance
systems.
 A robust system irrespective of different poor
weather conditions is built
 Various preprocessing techniques to identify
the nature of the weather and dynamically
choose the suitable algorithm is also presented

 It is inferred that slow moving or stationary
traffic is the main problem facing traffic
management authorities in most towns in India
 In vehicle detection algorithm, vehicles were
simultaneously extracted from the traffic video
data frames and their negatives using the
Laplacian of a Gaussian edge detector.
 Edge linking was achieved through
mathematical morphology and summation of
the positive and negative edge maps

 It was also noted that over segmentation
occurred for very large trucks with cabins and
their trailers being detected as separate
vehicles
 The position of the camera can be raised to be
high enough with respect to ROI mask to
overcome over segmentation

 This work could be extended to incorporate the
specifications of the cameras into the algorithm
 It is also noted after trying different cameras
with varying resolution has an implicit
relationship with the sizes of the required
structuring elements and the appropriate size
of the region of interest

Acknowledgement
 This work is supported in part by the Science
and Engineering Research Board under
Department of Science and Technology (DST-
SERB), Government of India.
 The authors wish to acknowledge the financial
support from the Department of Science and
Technology (DST-SERB), New Delhi, India for
the Major Research Project entitled
‘Development of Intelligent Traffic Analysis
Platform based on the Analysis of Trajectory
Big Data’ vide reference F.No :
ECR/2017/000733.
Publications in Journals
[1] Appavu alias Balamurugan S, Balaji Ganesh R, “Road Traffic Surveillance System
to classify the vehicles on Indian Roads in different climatic conditions”, Mobile
Information Systems, IOS Press, United States, December 2017 (SCI Journal,Impact
factor:0.849) available online at
http://iospress.us/articles/2018/03/IOS78IND2017.php (SCI, UGC Approved)
[2] Appavu alias Balamurugan S, Balaji Ganesh R, et al. “Detection and Pairing of
Vehicle Headlight in Night Scenes”, International Journal of Research in Engineering,
IT and Social Sciences, Vol.3, Issue 9, Sep.2013, ISSN No. 22500588 (UGC Approved
Journal List No. 42301, May 2018) (Citations:2)
[3] Appavu alias Balamurugan S, Balaji Ganesh R, “A Robust framework to detect

moving vehicles in different road conditions in India“ has been accepted for
publication in JOURNAL OF THEORETICAL AND APPLIED INFORMATION
TECHNOLOGY (E-ISSN 1817-3195 / ISSN 1992-8645) (Scopus Indexed, UGC
approved Journal List)
[4] Appavu alias Balamurugan S, Balaji Ganesh R, “An Intelligent Video Surveillance
Framework with Big Data Management for Indian Road Traffic System”, International
Journal of Computer Applications, Foundations of Computer Science, Newyork, Vol.
123, No.10, August 2015 ISSN No.0975-8887.DOI: 10.5120/ijca2015905519,
(Citations: 2)
Papers presented in Conferences held abroad
Appavu alias Balamurugan S, Balaji Ganesh R et al. “Indian Road Traffic
Surveillance System using Blob Tracking for ATIS Applications” Proceedings
of International Conference on Secure Knowledge Management in Big Data,
BITS Pilani, Dubai Campus, December 2014. DOI:
10.13140/RG.2.2.11865.21606

Papers presented in IEEE International Conference
Balaji Ganesh R et al. “Vision based system for detecting vehicles from videos in
different weather conditions” in Proceedings of IEEE International Conference on
Electronics, Communication and Aerospace Technologies during April 2017
(DOI: 10.1109/ICECA.2017.8203649)

Major References
[1] Bhandarkar S. M. and Luo, X., (2005), Fast and Robust Background Updating for Real- time Traffic Surveillance and
Monitoring, Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition
CVPR’05, San Diego, USA, June 25-25, IEEE Computer Society, pp. 55 – 55
[2] Chang, F. L., (2006), Object Tracking, Shadow Removal and Collision Event Detection for Traffic Surveillance System, MSc
Thesis, National Cheng Kung University, Taiwan
[3]Liu, Y. and Payeur, P. , (2003), Vision-based Detection of Activity for Traffic Control, Proc. IEEE Canadian Conference on
Electrical and Computing Engineering, Montréal,Canada, 2, May 4-7, pp. 1347 – 1350.,
http://dx.doi.org/10.1109/CCECE.2003.1226149
[4] Kumar, P., Ranganath, S., Weimin, H. and Sengupa, K., (2005), Framework for Real-time Behavior Interpretation from
Traffic Video, IEEE Transactions on Intelligent Transportation Systems, 6(1), pp. 43-53.,
http://dx.doi.org/10.1109/TITS.2004.838219
[5] Hsieh, Jun-Wei, et al. "Automatic traffic surveillance system for vehicle tracking and classification." IEEE Transactions on
Intelligent Transportation Systems 7.2 (2006): 175-187
[6]Pang, C.C.C., Lam, W.W.L. and Yung, N.H.C., (2007), A Method for Vehicle Count in the Presence of Multiple-Vehicle
Occlusions in Traffic Images, IEEE Transactions on Intelligent Transportation Systems, 8(3),pp. 441-459.,
http://dx.doi.org/10.1109/TITS.2007.902647
[7] Wang, Guolin, Deyun Xiao, and Jason Gu. "Review on vehicle detection based on video for traffic surveillance." Automation
and Logistics, 2008. ICAL 2008. IEEE International Conference on. IEEE, 2008.
[8] Rosin, P. and Ioannidis, E., (2003), Evaluation of Global Image Thresholding for Change Detection, Pattern Recognition
Letters, 24(14), pp. 2345-2356., http://dx.doi.org/10.1016/S0167-8655(03)00060-6
[9] Elhabian, S. Y., El-Sayed, K. M. and Ahmed, S. H., (2008), Moving Object Detection in Spatial Domain using Background
Removal Techniques – State-of-Art, Recent Patents on Computer Science, 1(1), pp. 32-54., http://dx.doi.org/10.2174/10032

[10] A. Haselhoff and A. Kummert, "A vehicle detection system based on Haar and Triangle features," 2009 IEEE Intelligent Vehicles
Symposium, Xi'an, 2009, pp. 261-266.
[11] Roh, H. J., Datla, S. and Sharma, S., (2013), Effect of Snow, Temperature and Their Interaction on Highway Truck Traffic,
Journal of Transportation Technologies, 3, pp. 24-38., http://dx.doi.org/10.4236/jtts.2013.31003
[12] Zhou, J., Guo, D. and Zhang, D., (2007), Moving Vehicle Detection for Automatic Traffic Monitoring, IEEE Transactions on
Vehicular Technology, 56(1), pp. 51-59., http://dx.doi.org/10.1109/TVT.2006.883735
[13] State of the Art Evaluation of Traffic Detection and Monitoring Systems , (2007), test report Texas Transportation Institute for
Arizona Department of Transportation, available at: http://www.azdot.gov/tpd/atrc/publications/project_reports/pdf/az627(1).pdf
[14] R. C. Gonzalez and R. E. Woods, Digital image procesing, third edition, Upper Saddle River, New Jersey: Pearson Education
Inc., 2008.
[15] P. K. Mishra, M. Athiq, A. Nandoriya and S. Chaudhuri, "Video-based vehicle detection and classification in heterogeneous
traffic conditions using a novel kernel classifier," IETE Journal of Research, vol. 59, no. 5, pp. 541-550, 2013
[16] G. Dougherty, Pattern Recognition and Classification, An Introduction, New York: Springer Science+Business Media, 2013.
[17] A. R. Webb and K. D. Copsey, Statistical pattern recognition, Third Edition, Chichester, West Sussex, UK: John Wiley & Sons,
Ltd, 2011.
[18] R. O. Duda, P. E. Hart and D. G. Stork, Pattern Classification, Second Edition, John Wiley & Sons, 2000.
[19] M. Pancharatnam and D. U. J. Sonnadara, "Vehicle counting and classification from a traffic scene," in Proceedings of the 26th
National IT Conference, Colombo, Sri Lanka.
[20] G. Zhang, R. P. Avery and Y. Wang, "A Video-based vehicle detection and classification system for real-time traffic data
collection using uncalibrated video cameras," Department of Civil and Environmental Engineering, University of Washington,
Washington, 2006.

[21] E. Ince, "Measuring traffic flow and classifying vehicle types: a surveillance video based approach," Turkish Journal of Electrical
Engineering and Computer Science, vol. 19, no. 4, 2011.
[22] Kanistras, Konstantinos, et al. "Survey of unmanned aerial vehicles (uavs) for traffic monitoring." Handbook of Unmanned Aerial
Vehicles. Springer Netherlands, 2015. 2643-2666.
[23] Xu, Zheng, et al. "Semantic based representing and organizing surveillance big data using video structural description
technology." Journal of Systems and Software 102 (2015): 217-225.


Balaji PPT Final

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Balaji PPT Final

Uploaded by

Copyright:

Available Formats

"All progress is born of inquiry.

 Received Masters in Technology (CIT) in

 Received Bachelors in Engineering (CSE) in PSR

 Currently pursuing Ph.D. in Information

 With an ever-increasing vehicular traffic on urban cities roads, the

 Today, ITS is benefitting most from video-based surveillance systems

 However, developing a reliable and efficient video-based vehicle

 Jittering camera or noise contamination due to wind are practical issues

 Similarly, at night time headlights and low illumination

 Traffic problem is an important issue happening in many cities in India.

 The number of people moving into an urban area has grown

 This causes an imbalance between the numbers of vehicles and roads,

 A video sequence of road can be processed and analyzed to

 Further information, such as the speed of a vehicle or traffic

 On the other hand, traffic administrations can utilize the traffic

 Computer vision can also be applied for solving traffic and

 Traffic surveillance systems aid commuters, provides valuable

 Information such as class of a vehicle can be useful

 Object detection and tracking has achieved good

 However, the same cannot be said about object

Tuesday, December 24, 2019

Tuesday, December 24, 2019

Tuesday, December 24, 2019

Tuesday, December 24, 2019

Tuesday, December 24, 2019

 Despite Temporal median or Moving Average

Tuesday, December 24, 2019

Tuesday, December 24, 2019

Tuesday, December 24, 2019

Tuesday, December 24, 2019

Tuesday, December 24, 2019

Tuesday, December 24, 2019

Tuesday, December 24, 2019

Tuesday, December 24, 2019

Tuesday, December 24, 2019

Tuesday, December 24, 2019

Tuesday, December 24, 2019

Tuesday, December 24, 2019

Tuesday, December 24, 2019

Tuesday, December 24, 2019

Tuesday, December 24, 2019

Tuesday, December 24, 2019

 All detection techniques are based on

Tuesday, December 24, 2019

 Both adaptive and Non-adaptive methods

Tuesday, December 24, 2019

Tuesday, December 24, 2019

Tuesday, December 24, 2019

Tuesday, December 24, 2019

Crop highlight regions in Mask

Tuesday, December 24, 2019

Tuesday, December 24, 2019

Tuesday, December 24, 2019

Tuesday, December 24, 2019

 Generating a set of initial bounding boxes

Tuesday, December 24, 2019

Tuesday, December 24, 2019

Tuesday, December 24, 2019

Tuesday, December 24, 2019

Tuesday, December 24, 2019

Tuesday, December 24, 2019

N is number of training data sample