22/05/2021

Motion Detection and Tracking

• Motion detection
• Approaches of motion detection
• Moving object tracking
Computer Vision
Chapter 6: Motion detection and Tracking

1 2

Motion Detection Approaches to Motion Detection

• Motion detection: Action of sensing physical movement in a give • Optical Flow
area
• Motion can be detected by measuring change in speed or vector ‒ Compute motion within region or the frame as a whole
of an object
• Goals of motion detection
‒ Identify moving objects
‒ Detection of unusual activity patterns
‒ Computing trajectories of moving objects

• Applications of motion detection • Change detection

‒ Indoor/outdoor security ‒ Detect objects within a scene
‒ Real time crime detection
‒ Track object across a number of frames
‒ Traffic monitoring
‒ Many intelligent video analysis systems are based on motion detection

3 4
 22/05/2021

Optical flow vs. motion field Estimating optical flow

• Optical flow does not always correspond to motion field
• Assume the image intensity I is constant

Time = t Time = t+dt

 x, y 
• Optical flow is an approximation of the motion field  x  dx, y  dy 
• The error is small at points with high spatial gradient
under some simplifying assumptions 𝐼 (𝑥, 𝑦, 𝑡) ≈ 𝐼 (𝑥 + 𝛿𝑥, 𝑦 + 𝛿𝑦, 𝑡 + 𝛿𝑡)

Optical flow constraint Lucas-Kanade algorithm

𝐼(𝑥, 𝑦, 𝑡) ≈ 𝐼(𝑥 + 𝛿𝑥, 𝑦 + 𝛿𝑦, 𝑡 + 𝛿𝑡)
Assume a single velocity for all pixels within an image patch
𝜕𝐼 𝜕𝐼 𝜕𝐼
𝐼(𝑥, 𝑦, 𝑡) ≈ 𝐼(𝑥, 𝑦, 𝑡) + 𝛿𝑥 + 𝛿𝑦 + 𝛿𝑡
𝜕𝑥 𝜕𝑦 𝜕𝑡 𝐸(𝑢, 𝑣) = (𝐼 (𝑥, 𝑦)𝑢 + 𝐼 (𝑥, 𝑦)𝑣 + 𝐼 )
, ∈

𝜕𝐼 𝛿𝑥 𝜕𝐼 𝛿𝑦 𝜕𝐼 Minimizing 𝐴𝐮 − 𝐛
+ + = 0, and let 𝛿𝑡 → 0
𝜕𝑥 𝛿𝑡 𝜕𝑦 𝛿𝑡 𝜕𝑡 𝐮 = (𝐴 𝐴) 𝐴 𝐛
𝐼 𝑢+𝐼 𝑣+𝐼 =0
𝐼 𝑢 + 𝐼 𝑣 = −𝐼 , 𝐼 𝐼 𝐼 𝐼 𝐼
𝑢 𝑢
𝐼 𝐼 = −𝐼 , ∇𝐼 𝐮 = 𝐛; 𝐴𝐮 = 𝐛 =−
𝑣 𝑣
𝐸(𝑢, 𝑣) = (𝐼 𝑢 + 𝐼 𝑣 + 𝐼 ) Hessian 𝐼 𝐼 𝐼 𝐼 𝐼

( ∇𝐼 ⋅ ∇𝐼 ) →
𝑢 =− ∇𝐼 ⋅ 𝐼

LHS: sum of the 2x2 outer product of the gradient vector

 22/05/2021

Matrix form Matrix form

𝐴Δ𝐮 = 𝐛
𝐸(𝐮 + Δ𝐮) = 𝐼 (𝐱 + 𝐮 + Δ𝐮) − 𝐼 (𝐱 )
𝐴= 𝐉 (𝐱 + 𝐮)𝐉 (𝐱 + 𝐮)
≈ 𝐼 (𝐱 + 𝐮) + 𝐉 (𝐱 + 𝐮)Δ𝐮 − 𝐼 (𝐱 )
𝐛=− 𝑒 𝐉 (𝑥 + 𝑢)
= 𝐉 (𝐱 + 𝐮)Δ𝐮 + 𝑒
𝐼 𝐼 𝐼 𝐼 𝐼
𝜕𝐼 𝜕𝐼
𝐉 (𝐱 + 𝐮) ≈ ∇𝐼 (𝐱 + 𝐮) = , (𝐱 + 𝐮) 𝐴= , 𝐛=−
𝜕𝑥 𝜕𝑦
𝐼 𝐼 𝐼 𝐼 𝐼
𝑒 = 𝐼 (𝐱 + 𝐮) − 𝐼 (𝐱 )
𝜕𝐼 𝜕𝐼
𝐉 (𝐱 + 𝐮) ≈ ∇𝐼 (𝐱 + 𝐮) = , (𝐱 + 𝐮)
𝜕𝑥 𝜕𝑦
𝑒 = 𝐼 (𝐱 + 𝐮) − 𝐼 (𝐱 )

𝐉 (𝐱 + 𝐮) ≈ 𝐉 (𝐱 )

for computational efficiency

Computing gradients in X-Y-T The aperture problem

y 𝐼 𝐼
1
𝐼 = [(𝐼 , , + 𝐼 , , +𝐼 , , +𝐼 , , ) Let 𝐴 = ∇𝐼 ⋅ ∇𝐼 , and 𝐛 = −
4𝛿𝑥 time 𝐼 𝐼
−(𝐼 , , + 𝐼 , , +𝐼, , +𝐼, , )]
• Algorithm: At each pixel compute u by solving 𝐴𝐮 = 𝐛
j+1 • A is singular if all gradient vectors point in the
likewise for Iy and It
k+1 same direction
k ‒ e.g., along an edge
x ‒ of course, trivially singular if the summation is over a
i i+1 single pixel or there is no texture
‒ i.e., only normal flow is available (aperture problem)
• Corners and textured areas are OK
 22/05/2021

Error functions Horn-Schunck algorithm

• Robust error function • Global method with smoothness constraint to solve
𝑥
𝐸(𝐮) = 𝜌(𝐼(𝐱 + 𝐮) − 𝐼(𝐱)), 𝜌(𝑥) = aperture problem
1 + 𝑥 /𝑎
• Spatially varying weights • Minimize a global energy functional with calculus of
variations
𝐸(𝐮) = 𝑤 (𝐱 )𝑤 (𝐱 + 𝐮) 𝐼(𝐱 + 𝐮) − 𝐼(𝐱 )
𝐸= 𝐼 𝑢+𝐼 𝑣+𝐼 +𝛼 ∇𝑢 + ∇𝑣 𝑑𝑥𝑑𝑦
• Bias and gain: images taken with different exposure
𝜕𝐿 𝜕 𝜕𝐿 𝜕 𝜕𝐿
− − =0
𝐼(𝐱 + 𝐮) = (1 + 𝛼)𝐼(𝐱) + 𝛽, 𝛼 is the gain and 𝛽 is the bias 𝜕𝑢 𝜕𝑥 𝜕𝑢 𝜕𝑦 𝜕𝑢
𝜕𝐿 𝜕 𝜕𝐿 𝜕 𝜕𝐿
𝐸(𝐮) = 𝐼(𝐱 + 𝐮) − (1 + 𝛼)𝐼(𝐱 ) − 𝛽 − − =0
𝜕𝑣 𝜕𝑥 𝜕𝑣 𝜕𝑦 𝜕𝑣
where 𝐿 is the integrand of the energy function
• Correlation (and normalized cross correlation)
E (u)   I ( xi ) I (x i  u)
i

Horn-Schunck algorithm Horn-Schunck algorithm

(u , v )
𝐼 (𝐼 𝑢 + 𝐼 𝑣 + 𝐼 ) − 𝛼 Δ𝑢 = 0 • Iterative scheme v
𝐼 (𝐼 𝑢 + 𝐼 𝑣 + 𝐼 ) − 𝛼 Δ𝑣 = 0
𝜕 𝜕
where Δ = + is the Laplace operator
𝜕𝑥 𝜕𝑦

Δ𝑢(𝑥, 𝑦) = 𝑢̄ (𝑥, 𝑦) − 𝑢(𝑥, 𝑦)

(𝐼 + 𝛼 )𝑢 + 𝐼 𝐼 𝑣 = 𝛼 𝑢̄ − 𝐼 𝐼
𝐼 𝐼 𝑢 + (𝐼 + 𝛼 )𝑣 = 𝛼 𝑣̄ − 𝐼 𝐼
• Yields high density flow
• Fill in missing information in the homogenous
regions
• More sensitive to noise than local methods
 22/05/2021

Background Subtraction
• Uses a reference background image for
comparison purposes.
• Current image (containing target object) is b. Same scene later
a. Original scene
compared to reference image pixel by pixel.
• Places where there are differences are detected
and classified as moving objects.

Motivation: simple difference of two images

shows moving objects

Subtraction of scene a from scene b Subtracted image with threshold of 100

17 18

Static Scene Object Detection and Approaches to Background Modeling

Tracking
• Background Subtraction
• Model the background and subtract to obtain
object mask • Statistical Methods
• Filter to remove noise (e.g., Gaussian Mixture Model, Stauffer and Grimson 2000)
• Group adjacent pixels to obtain objects
• Track objects between frames to develop
Background Subtraction:
trajectories
1. Construct a background image B as average of few
images
2. For each actual frame I, classify individual pixels as
foreground if |B-I| > T (threshold)
3. Clean noisy pixels

19 20
 22/05/2021

Background Subtraction

Background Image Current Image

21 22

Gaussian Mixture Model

Statistical Methods

Block 44x42 Pixel 172x165

• Pixel statistics: average and standard deviation of Temple1 - RGB Distribution of Pixel 172x165

color and gray level values 200

150

100

B
50

• Gaussian Mixture Model Temple1 - RG Distribution of Pixel 172x165

0
250

200
200
250

250 150

‒ Model the color values of a particular pixel as a mixture of

150

G 100 100
R

Gaussians 200

‒ Multiple adaptive Gaussians are necessary to cope with 150

acquisition noise, lighting changes, etc.

G
100

‒ Pixel values that do not fit the background distributions

R-G-B Distribution
(Mahalanobis distance) are considered foreground 50

0
0 50 100 150 200 250

R-G Distribution
R

23 24
 22/05/2021

Detection of Moving Objects Based on Dynamic Distribution Learning and Outlier

Local Variation Detection
(1) f (t )  mean (t  1)
 C1 Detect Outlier
std (t  1)
For each block location (x,y) in the video plane
• Consider texture vectors in a symmetric window [t-W, f (t )  mean (t  1)
(2)  C2 Switch to a nominal state
t+W] at time t std (t  1)
• Compute the covariance matrix
• Motion measure is defined as (3) mean (t )  u  mean (t  1)  (1  u )  f (t )
the largest eigenvalue of the covariance matrix Update the estimates of mean and standard
deviation only when the outliers are not
(4)
std (t )   (t )2 detected

(5)  2 (t )  u   2 (t  1)  (1  u )  ( f (t )  mean (t  1)) 2

25 26

Moving object tracking Model based tracking

• Mathematical model of objects’ motions:
• Model of object motion
‒ position, velocity (speed, direction), acceleration
• Kalman filter
• Can predict objects’ positions
• Mean Shift
Motion
Model

Predict
Update? Location

Verify
Location

28
 22/05/2021

Simple Motion Model Prediction

• Can predict position at time t knowing
• Newton’s laws
‒ Position
1
st   s  ut  a t
0
2 ‒ Velocity

2 ‒ Acceleration
• s = position • At t=0
• u = velocity
• a = acceleration
‒ all vector quantities
‒ measured in image co-ordinates

29 30

Uncertainty Verification
• If some error in a - Da or u - Du • Is the object at the predicted location?
• Then error in predicted position - Ds ‒ Matching
• How to decide if object is found
‒ Search area
• Where to look for object

1
Ds t   s0  Dut  Da t 2
2

31 32
 22/05/2021

Object Matching Search Area

• Compare • Uncertainty in knowledge of model parameters
‒ A small bitmap derived from the object vs. ‒ Limited accuracy of measurement
‒ Small regions of the image ‒ Values might change between measurements
• Matching? • Define an area in which object could be
‒ Measure differences ‒ Centred on predicted location, s  Ds

33 34

Update the Model Kalman Filter

Black Box
System

• Is the object at the predicted location? Error Sources

• Yes External Controls System

System State
‒ No change to model (desired but not
known)
• No Observed
Measurements
Optimal Estimate
of System State
Measuring
‒ Model needs updating Devices Estimator

‒ Kalman filter is a solution Measurement

Error Sources
• Mathematically rigorous methods of using uncertain
measurements to update a model
• Problem formulation
‒ System state cannot be measured directly
‒ Need to estimate “optimally” from measurementsh

35 36
 22/05/2021

Kalman Filter Kalman filter

• Recursive data processing algorithm • Matrix description of system state, model and
• Generates optimal estimate of desired quantities measurement
given the set of measurements • Progressive method
• Optimal?
‒ For linear system and white Gaussian errors, Kalman
prediction
filter is “best” estimate based on all previous
measurements correction
‒ For non-linear system optimality is ‘qualified’ • Proper dealing with noise
• Recursive?
‒ Doesn’t need to store all previous measurements and
reprocess all data each time step

37

Kalman filter assumptions

Kalman filter
• Linear system
• Advantages of using KF in particle tracking ‒ System parameters are linear
• Progressive method function of parameters at some
previous time
‒ No large matrices has to be inverted
‒ Measurements are linear function
• Proper dealing with system noise of parameters
• Track finding and track fitting • White Gaussian noise
 KF is the
• Detection of outliers ‒ White: uncorrelated in time
optimal filter
• Merging track from different segments ‒ Gaussian: noise amplitude
 22/05/2021

Kalman filter Mathematically

• Relates
‒ Measurements y[k]
• e.g. positions How do observations relate to model?
‒ System state x[k]
• Position, velocity of object, etc
yk   Hk xk   nk 
‒ Observation matrix H[k]
• Relates system state to measurements  xk 
 
 xk  1 0 0 0  xk 
‒ Evolution matrix A[k]
 yk  0 0 1 0  yk   nk 

• Relates state of system between epochs
‒ Measurement noise n[k]    
 
 y k 
‒ Process noise v[k]

41 42

Prediction of System State Prediction of Observation

• Relates system states at epochs k and k+1
• From predicted system state, estimate what
observation should occur:
xˆ k  1 | k   Ak xk | k   vk 
 xˆk  1 | k  1 T 0 0  xk | k  yˆ k  1 | k   Hk xˆ k  1 | k   nk 
 xˆk  1 | k   0 1 0 0   xk | k   xˆk  1 | k 
   vk   
 yˆ k  1 | k  0 0 1 T   yk | k   xˆk  1 | k  1 0 0 0  xˆk  1 | k 
ˆ      yˆ k  1 | k   0 0 1 0  yˆ k  1 | k   nk 
 y k  1 | k   0 0 0 1  y k | k     
ˆ 
 y k  1 | k 
43 44
 22/05/2021

Updating the Model Updating the Model

• Predict/estimate a measurement yˆ k  1 | k  xˆ k  1 | k  1  xˆ k  1 | k   Gk  1Dyk  1 | k 

• Make a measurement Dyk  1 | k   yk  1  yˆ k  1 | k 
yk  1
• Predict state of model
xˆ k  1 | k 
• G is Kalman Gain
• How does the new measurement ‒ Derived from A, H, v, n.
contribute to updating the model?

G  Ck | k HT HC k | k  HT  n 1
Ck  1 | k   Ck | k   GHCk | k 
C  system covariance

45 46

Example Condensation Tracking

• So far considered single motions
• Tracking two corners of a minimum bounding box
• What if movements change?
• Matching using colour
‒ Bouncing ball
• Image differencing to locate target
‒ Human movements
• Use multiple models
‒ plus a model selection mechanism

47 48
 22/05/2021

Tracking Using Hidden Markov

Selection and Tracking Models
• Occur simultaneously • Hidden Markov model describes
• Maintain ‒ States occupied by a system
‒ A distribution of likely object positions plus weights ‒ Possible transitions between states
• Predict ‒ Probabilities of transitions
‒ Select N samples, predict locations ‒ How to recognise a transition
• Verify
‒ Match predicted locations against image
‒ Update distributions

49 50

Mean-shift
Intuitive Description
• The mean-shift algorithm is an efficient approach to Region of
interest
tracking objects whose appearance is defined by Center of

histograms. (not limited to only color) mass

• Motivation – to track non-rigid objects, (like a walking

person), it is hard to specify an explicit 2D parametric
motion model.

• Appearances of non-rigid objects can sometimes be

modeled with color distributions Mean Shift
vector
Objective : Find the densest region
Distribution of identical billiard balls
Stolen from: www.wisdom.weizmann.ac.il/~deniss/vision_spring04/files/mean_shift/mean_shift.ppt
 22/05/2021

Mean Shift Vector Mean Shift Vector

• Given:
Data points and approximate location of the mean of
this data:

• Task:
Estimate the exact location of the mean of the data
by determining the shift vector from the initial mean.

Mean-Shift Object Tracking Mean-Shift Object Tracking

Target Representation PDF Representation

Represent the Target Model Target Candidate

Choose a (centered at 0)
Choose a model by its (centered at y)
reference
feature space PDF in the
target model 0.35 0.3

feature space 0.3

0.25
0.25

0.2

Probability

Probability
0.2
0.15
0.15
0.1
0.1

0.05 0.05

0 0
1 2 3 . . . m 1 2 3 . . . m
0.35 color color
0.3

0.25
Quantized  m
 m
Probability

q  qu u 1..m q p  y    pu  y u 1..m p

0.2

Color Space 0.15

u 1 u 1
0.1
u 1 u 1
0.05

0
1 2 3 . . . m

color
Similarity f  y   f  q, p  y  
Function:   Q is the target histogram,
P is the object histogram
Stolen from: www.cs.wustl.edu/~pless/559/lectures/lecture22_tracking.ppt Stolen from: www.cs.wustl.edu/~pless/559/lectures/lecture22_tracking.ppt (depends on location y)
 22/05/2021

Mean-Shift Object Tracking

Target Localization Algorithm Mean Shift
• Mean-Shift in tracking task:
 track the motion of a cluster of interesting features.
Start from
the position Search in the Find best • 1. choose the feature distribution to represent an object (e.g.,
of the model model’s candidate by
in the current neighborhood maximizing a color + texture),
frame in next frame similarity func.
• 2. start the mean-shift window over the feature distribution
generated by the object
• 3. finally compute the chosen feature distribution over the next
video frame
‒ Starting from the current window location, the mean-shift algorithm
will find the new peak or mode of the feature distribution, which
(presumably) is centered over the object that produced the color and
    texture in the first place.
q p  y f  p  y  , q 
Stolen from: www.cs.wustl.edu/~pless/559/lectures/lecture22_tracking.ppt
‒ In this way, the mean-shift window tracks the movement of the object
frame by frame.

Thank you for

your attention!