2D TARGET TRACKING

USING KALMAN FILTER

2006-2007

PROJECT REPORT

SUBMITTED TO THE FACULTY OF

ELECTRONICS AND COMMUNICATION ENGINEERING

NATIONAL INSTITUTE OF TECHNOLOGY, WARANGAL (A.P)

(DEEMED UNIVERSITY)

BACHELOR OF TECHNOLOGY
IN
ELECTRONICS AND COMMUNICATION ENGINEERING

SUBMITTED BY

AMIT KUMAR KARNA (04403)

DEVENDER BUDHWAR (04411)
SAHIL SANDHU (04455)

Under the Guidance of

Dr. T. Kishor Kumar

DEPARTMENT OF ELECTRONICS AND COMMUNICATION

ENGINEERING
NATIONAL INSTITUTE OF TECHNOLOGY
(DEEMED UNIVERSITY)
WARANGAL-506004(A.P)

1
 Acknowledgements

We would like to express our sincere thanks to our

faculty, Dr. T. Kishor Kumar, Department of Electronics
and Communication Engineering, National Institute of
Technology, Warangal for his constant encouragement,
splendid and gracious guidance throughout our work. He
has been a constant source of inspiration and helped us
in each stage. We express our deep gratitude to him for
such encouragement and kindly cooperation.

Amit Kumar Karna (4403)

Devender Budhwar (4411)
Sahil Sandhu (4455)

2
 CONTENTS

Abstract
Introduction
Kalman Filter
State-space Representation
Underlying Dynamic System Model
Code
Results
Conclusions
References

3
 2D TARGET TRACKING USING KALMAN FILTER
Sahil Sandhu Devender Budhwar
¾ B.Tech (E.C.E.) ¾ B.Tech (E.C.E.)
National Institute of National Institute of
Technology Warangal Technology Warangal
sahilsandhu@gmail.com dev.budhwar@gmail.com
Abstract
It is now quite common in the recursive
approaches for motion estimation, to find
applications of the Kalman filtering
technique both in time and frequency
domains. In the block-based approach, very
few approaches are available of this
technique to refine the estimation of motion
vectors resulting from fast algorithms.. This
paper proposes an object motion estimation
which uses the Kalman filtering technique
to improve the motion estimates resulting
from both the three step algorithm and
Kalman application.
1.0 Introduction
In the field of motion estimation for video
coding many techniques have been applied. It is
now quite common to see the Kalman filtering
technique and some of its extensions to be used
for the estimation of motion within image
sequences.
Particularly in the pixel-recursive approaches,
which suit very much the Kalman formulation,
one finds various ways of applying this
estimation technique both in the time and
frequency domains. On a very general
perspective, we find use of Kalman filter (KF),
which implies linear state-space representations
and the extended Kalman filter (EKF), which
uses the linearised expressions of non-linear
state- space formulations. Moreover, the parallel
extended Kalman filter (PEKF) which consists of
a parallel bank of EKF’s, is often encountered in
practice.
In the block-based motion-compensated
prediction approaches, the most common
procedure is the block- matching technique.
4
Amit Kumar Karna
¾ B.Tech (E.C.E.)
National Institute of
Technology Warangal
karnamit@gmail.com
Given a macro block in the current frame, the
objective is to determine the block in a reference
frame (one of the past for forward motion
estimation or one of the futures for backward
motion estimation) in a certain search area that
“best” matches the current macro block
according to a specific criterion. In most coding
systems, the “best” match is found using the
mean absolute error (MAE) criterion. There are
several well known algorithms that perform the
block matching motion estimation, among them
being the full search algorithm (FSA) that
determines the motion vector of a macro block
by computing the MAE at each location in the
search area. This is the simplest method, it
provides the best performance, but at a very high
computational cost.
To reduce these computational requirements,
several heuristic search strategies have been
developed, as for example the two-dimensional
logarithmic search, the parallel one-dimensional
search, etc [3-51. These are often referred to as
fast search algorithms. In fast algorithms the
procedures applied for motion estimation are of
significantly lower complexity, but yield a
suboptimal solution in the sense that they may
not avoid the convergence of the MAE cost
function to a local minimum instead of the global
one. Lately, some new fast strategies as well as
motion estimation have been proposed. But, very
few applications are available of Kalman
filtering for the estimation of motion vectors.
This paper proposes a motion estimation using
Kalman filtering to improve the motion vector
estimates resulting from both the conventional
three step algorithm (TSA) based Kalman filter
proposed in Section 2 introduces the state-space
representation for the motion vector. The
corresponding Kalman equations are:

2.0 The Kalman Filter
The Kalman filter is a recursive estimator. This
means that only the estimated state from the
previous time step and the current measurement
are needed to compute the estimate for the
current state. In contrast to batch estimation
techniques, no history of observations and/or
estimates is required. It is unusual in being
purely a time domain filter; most filters (for
example, a low-pass filter) are formulated in the
frequency domain and then transformed back to
the time domain for implementation.
The state of the filter is represented by two
variables:
, the estimate of the state at time k;
, the error covariance matrix (a measure of
the estimated accuracy of the state estimate).
The Kalman filter has two distinct phases:
Predict and Update. The predict phase uses the
estimate from the previous timestep to produce
an estimate of the current state. In the update
phase, measurement information from the
current timestep is used to refine this prediction
to arrive at a new, (hopefully) more accurate
estimate.
Predict
(predicted state)
(predicted estimate covariance)
Update
(innovation or measurement residual)
(innovation (or residual) covariance)
5
(optimal Kalman gain)
(updated state estimate)
(updated estimate covariance)
The formula for the updated estimate covariance
above is only valid for the optimal Kalman gain.
Usage of other gain values require a more
complex formula found in the derivations
section.
Invariants
If the model is accurate, and the values for
and accurately reflect the distribution of the
initial state values, then the following invariants
are preserved: all estimates have mean error zero
where E[ξ] is the expected value of ξ, and
covariance matrices accurately reflect the
covariance of estimates
The Kalman filter can be regarded as an adaptive
low-pass infinite impulse response Digital filter,
with cut-off frequency depending on the ratio
between the process- and measurement (or
observation) noise, as well as the estimate
covariance. Frequency response is, however,
rarely of interest when designing state estimators
such as the Kalman Filter, whereas for Digital
Filters, frequency response is usually of primary
concern. For the Kalman Filter, the important
goal is how accurate the filter is, and this is most
often decided based on empirical Monte Carlo
simulations, where "truth" (the true state) is
known.
In the extended Kalman filter (EKF) the state
transition and observation models need not be
linear functions of the state but may instead be
(differentiable) functions.
The function f can be used to compute the
predicted state from the previous estimate and
similarly the function h can be used to compute
the predicted measurement from the predicted
state. However, f and h cannot be applied to the
covariance directly. Instead a matrix of partial
derivatives (the Jacobian) is computed.
At each timestep the Jacobian is evaluated with
current predicted states. These matrices can be
used in the Kalman filter equations. This process
essentially linearizes the non-linear function
around the current estimate.
This results in the following extended Kalman
filter equations:
Predict
Update
Where the state transition and observation
matrices are defined to be the following
Jacobians
Relationship to recursive Bayesian estimation
The true state is assumed to be an unobserved
Markov process, and the measurements are the
observed states of a hidden Markov model.
Because of the Markov assumption, the true state
is conditionally independent of all earlier states
given the immediately previous state.
Similarly the measurement at the k-th timestep is
dependent only upon the current state and is
conditionally independent of all other states
given the current state.
Using these assumptions the probability
distribution over all states of the HMM can be
written simply as:
6
However, when the Kalman filter is used to
estimate the state x, the probability distribution
of interest is that associated with the current
states conditioned on the measurements up to the
current timestep. (This is achieved by
marginalizing out the previous states and
dividing by the probability of the measurement
set.)
This leads to the predict and update steps of the
Kalman filter written probabilistically. The
probability distribution associated with the
predicted state is product of the probability
distribution associated with the transition from
the (k − 1)th timestep to the kth and the
probability distribution associated with the
previous state, with the true state at (k − 1)
integrated out.
The measurement set up to time t is
The probability distribution of updated is
proportional to the product of the measurement
likelihood and the predicted state.
The denominator
is a normalization term.
The remaining probability density functions are
Note that the PDF at the previous timestep is
inductively assumed to be the estimated state and
covariance. This is justified because, as an
optimal estimator, the Kalman filter makes best
use of the measurements, therefore the PDF for
given the measurements is the Kalman
filter estimate.
3.0 State-space representation
The scanning in a frame is from the top left to
the bottom right. The motion vector of a
macroblock can be predicted from that of its left
spatial neighbour. The measured motion vectors
are obtained through a conventional three step
procedure.
In the same manner as in, the intraframe motion
estimation process is mlodelled through an auto-
regressive model which produces the state-space
equations.
From this macroblock-based representation
how do we define the 8x8-block: based
representation? Each 16x16-block yields a
zig-zag sequence of four 8x8 blocks:
This corresponds to a conventional pixel
decimation for block matching in an 8x8-bock.
The motion vector of these sub-blocks is defined
as V' = (vx,vy), where v, and v, denote the
horizontal and vertical components, respectively.
These two components are assumed independent.
The motion vector of an 8x8-block can be
predicted (through KF) from the one of the
previous 8x8-block according to the time index k
of the zig-zag order using the state equation:
V(k+ 1 )=F(k)V(k) +W(k)
where W(k)is the state noise vector, i.e. the
prediction error with covariance matrix Q. The
state noise components w,, w, are assumed
independent and Gaussian distributed with
zero-imean and same variance q. The matrix
F(k) is the transition matrix which describes
the dynamic behaviour of the motion vector
from one 8x8-block to the next:
It is observed that the measured motion
vectors are actually obtained from the TSA
run on a 16x16-block basis that yields the zig-
zag sequence of four measured motion vectors
Y(k) with same values on the 8x8-block basis.
This means that the Kalman filter has four
measurements instead of one to adjust the
7
motion estimate V(k/k) when the assumption
of smooth changes is not strictly valid. As a
result, it is expected that we have a better
motion estimate for the 8x8-motion
compensation procedure.
4.0 Underlying dynamic system model
Kalman filters are based on linear dynamical
systems discretised in the time domain. They are
modelled on a Markov chain built on linear
operators perturbed by Gaussian noise. The state
of the system is represented as a vector of real
numbers. At each discrete time increment, a
linear operator is applied to the state to generate
the new state, with some noise mixed in, and
optionally some information from the controls on
the system if they are known. Then, another
linear operator mixed with more noise generates
the visible outputs from the hidden state. The
Kalman filter may be regarded as analogous to
the hidden Markov model, with the key
difference that the hidden state variables are
continuous (as opposed to being discrete in the
hidden Markov model). Additionally, the hidden
Markov model can represent an arbitrary
distribution for the next value of the state
variables, in contrast to the Gaussian noise
model that is used for the Kalman filter. There is
a strong duality between the equations of the
Kalman Filter and those of the hidden Markov
model. A review of this and other models is
given in Roweis and Ghahramani (1999).
In order to use the Kalman filter to estimate the
internal state of a process given only a sequence
of noisy observations, one must model the
process in accordance with the framework of the
Kalman filter. This means specifying the
matrices Fk, Hk, Qk, Rk, and sometimes Bk for
each time-step k as described below.
Fig. Model underlying Kalman filter
Model underlying the Kalman filter. Circles are
vectors, squares are matrices, and stars represent
Gaussian noise with the associated covariance
matrix at the lower right. The Kalman filter imshow(Im)
model assumes the true state at time k is evolved Imwork = double(Im);
from the state at (k − 1) according to: %extract ball
[cc(i),cr(i),radius,flag] =
Where, extractball(Imwork,Imback,i);%,fig1,fig2,fig3,fi
Fk is the state transition model which is applied g15,i);
to the previous state xk−1; if flag==0
Bk is the control-input model which is applied to continue
the control vector uk; end
wk is the process noise which is assumed to be hold on
drawn from a zero mean multivariate normal for c = -0.9*radius: radius/20 : 0.9*radius
distribution with covariance Qk. r = sqrt(radius^2-c^2);
plot(cc(i)+c,cr(i)+r,'g.')
At time k an observation (or measurement) zk of plot(cc(i)+c,cr(i)-r,'g.')
the true state xk is made according to end
%Slow motion!
where Hk is the observation model which maps pause(0.02)
the true state space into the observed space and end
vk is the observation noise which is assumed to figure
be zero mean Gaussian white noise with plot(cr,'g*')
covariance Rk. hold on
plot(cc,'r*')
The initial state, and the noise vectors at each
step {x0, w1, ..., wk, v1 ... vk} are all assumed to <extractball.m>
be mutually independent.
Many real dynamical systems do not exactly fit % extracts the center (cc,cr) and radius of the
this model; however, because the Kalman filter largest blob
is designed to operate in the presence of noise, function
an approximate fit is often good enough for the [cc,cr,radius,flag]=extractball(Imwork,Imback,in
filter to be very useful. Variations on the Kalman dex)%,fig1,fig2,fig3,fig15,index)
filter described below allow richer and more cc = 0;
sophisticated models. cr = 0;
radius = 0;
flag = 0;
5.0 Code [MR,MC,Dim] = size(Imback);
% subtract background & select pixels with a
big difference
<Detect.m>
fore = zeros(MR,MC); %image
subtracktion
%detect fore = (abs(Imwork(:,:,1)-Imback(:,:,1)) > 10) ..
clear,clc | (abs(Imwork(:,:,2) - Imback(:,:,2)) > 10) ...
% compute the background image | (abs(Imwork(:,:,3) - Imback(:,:,3)) > 10);
Imzero = zeros(240,320,3); % Morphology Operation erode to remove
for i = 1:5 small noise
Im{i} = foremm = bwmorph(fore,'erode',2); %2 time
double(imread(['DATA/',int2str(i),'.jpg'])); % select largest object
Imzero = Im{i}+Imzero; labeled = bwlabel(foremm,4);
end stats = regionprops(labeled,['basic']);%basic
Imback = Imzero/5; mohem nist
[MR,MC,Dim] = size(Imback); [N,W] = size(stats);
% loop over all images if N < 1
for i = 1 : 60 return
% load image end
Im = (imread(['DATA/',int2str(i), '.jpg']));

8
 kfinit=0;
% do bubble sort (large to small) on regions in x=zeros(100,4);
case there are more than 1
id = zeros(N); % loop over all images
for i = 1 : N for i = 1 : 60
id(i) = i; % load image
end Im = (imread(['DATA/',int2str(i), '.jpg']));
for i = 1 : N-1 imshow(Im)
for j = i+1 : N imshow(Im)
Imwork = double(Im);
if stats(i).Area < stats(j).Area
tmp = stats(i);
%extract ball
stats(i) = stats(j);
[cc(i),cr(i),radius,flag] =
stats(j) = tmp; extractball(Imwork,Imback,i);
tmp = id(i); if flag==0
id(i) = id(j); continue
id(j) = tmp; end
end hold on
end for c = -1*radius: radius/20 : 1*radius
end r = sqrt(radius^2-c^2);
% make sure that there is at least 1 big region plot(cc(i)+c,cr(i)+r,'g.')
if stats(1).Area < 100 plot(cc(i)+c,cr(i)-r,'g.')
return end
end % Kalman update
selected = (labeled==id(1)); i
% get center of mass and radius of largest if kfinit==0
centroid = stats(1).Centroid; xp = [MC/2,MR/2,0,0]'
radius = sqrt(stats(1).Area/pi); else
cc = centroid(1); xp=A*x(i-1,:)' + Bu
cr = centroid(2); end
flag = 1; kfinit=1;
return PP = A*P*A' + Q
K = PP*H'*inv(H*PP*H'+R)
x(i,:) = (xp + K*([cc(i),cr(i)]' - H*xp))';
<kalman.m> x(i,:)
[cc(i),cr(i)]
clear,clc P = (eye(4)-K*H)*PP
% compute the background image
Imzero = zeros(240,320,3); hold on
for i = 1:5 for c = -1*radius: radius/20 : 1*radius
Im{i} = r = sqrt(radius^2-c^2);
double(imread(['DATA/',int2str(i),'.jpg'])); plot(x(i,1)+c,x(i,2)+r,'r.')
Imzero = Im{i}+Imzero; plot(x(i,1)+c,x(i,2)-r,'r.')
end end
Imback = Imzero/5; pause(0.3)
[MR,MC,Dim] = size(Imback); end

% Kalman filter initialization % show positions

R=[[0.2845,0.0045]',[0.0045,0.0455]']; figure
H=[[1,0]',[0,1]',[0,0]',[0,0]']; plot(cc,'r*')
Q=0.01*eye(4); hold on
P = 100*eye(4); plot(cr,'g*')
dt=1; %end
A=[[1,0,0,0]',[0,1,0,0]',[dt,0,1,0]',[0,dt,0,1]']; %estimate image noise (R) from stationary ball
g = 6; % pixels^2/time step posn = [cc(55:60)',cr(55:60)'];
Bu = [0,0,0,g]'; mp = mean(posn);

9
 diffp = posn - ones(6,1)*mp;
Rnew = (diffp'*diffp)/5;
6.0 RESULTS
For comparison purposes, the Full Search
Algorithm (FSA), the Three Step Algorithm
(TSA), the 16x16-block based Kalman Filter
(16x16-KF) presented in [22] and the proposed
8x8-block based Kalman Filter (8x8-KF) are run
for different classes of video sequences. We use
50 frames of the following sequences: ‘Alkistis’,
‘Carphone’, ‘Foreman’ and a sub-sampled
sequence of ‘News’.
It is observed that all the above listed techniques
can be qualified as sub-optimal techniques. Even
the full search algorithm does not give the real
motion. Hence, in any case the results expected
are very close in terms of average PSNR (see
Table 1).
As in, the algorithm using the Kalman filtering
on a 16x16-block basis provides a greater
average peak signal to noise ratio than the one
resulting from the conventional three step
algorithm. As expected the 8x8- block based
proposed approach results in an even greater
PSNR better than both the TSA and 16x16-KF.
For the particular state-space representation
used, when the real motion corroborate the
assumptions made regarding, only translational
motion with very smooth changes, the 8x8-block
based motion estimation using Kalman filter is
even better than the one resulting from the full
search algorithm, as one can see from the
average PSNR values and the curves in Figure 1.
The computational complexity of each of
the above stated algorithms is given in Table 2.
The complexity is evaluated in terms of number
of operations per block (NOPB). The
formulation of the complexity uses N=16 and
p=7, where N defines the size of a NxN-block
and p the maximum displacement within the
search area. The full search requires (2p+1)’
search locations and the three step 25.
The Kalman filter is usually of high
complexity. But for our particular application,
that uses an intentionally simple state-space
model for the motion, it is observed from Table 2
that the Kalman filter implementation for
refining the motion estimates resulting from the
TSA, does not significantly increase the
10
computational complexity. Indeed, due to the
basic formulation of the motion, the Kalman
filter equations can be largely simplified and thus
reduced to their scalar form. Subsequently, any
expensive matrix calculation is avoided.
7.0 Conclusion
The above presented results are encouraging, in
the sense that with the appropriate state model
and a priori assumptions close to the real motion
vector behaviour, we are able through Kalman
filtering to have a greater PSNR than the other
techniques for any frame of the sequence. It is
evident that a Kalman filter using such simple
formulation as in this paper is more suitable for
class A image sequences. At this point, there are
few questions I alternatives that may be of
interest:
-Shall we consider sub-pixel accuracy?
-Shall we leave the motion components
independency assumption and consider the real
correlation that exists between the displacement
in x- and y-direction?
-Shall we envisage more elaborate state-space
representation for the motion that will imply
implementation of extended Kalman filters and
parallel Kalman filters?
Any of the above consideration is feasible.
Regarding the usefulness of implementing such
approaches, this will be a matter of trade-off
among the resulting motion estimates, the
computational complexity and the real visual
quality of the images.
References
[1] H. G. Musmann, P. Pirsch and H.-J. Grallert,
‘Advances in picture coding’, Proc. IEEE,
Vol. 73, No. 4, pp. 523-548, 1985.
[2] I. Aksu, F. Jldiz and J. B. Burl, ‘A
comparison of the performance of image
motion operating on low signal to noise ratio
images’, Proc. of the 34th Midwest Symposium
on Circuits and Systems, Vol. 2, pp. 1124-1
128, NY 1992.
[3] A. Murat Tekalp, ‘Digital video processing’,
Prentice-Hall, 1995.
[4] Internet resources