Under the Guidance of



We would like to express our sincere thanks to our faculty, Dr. T. Kishor Kumar, Department of Electronics and Communication Engineering, National Institute of Technology, Warangal for his constant encouragement, splendid and gracious guidance throughout our work. He has been a constant source of inspiration and helped us in each stage. We express our deep gratitude to him for such encouragement and kindly cooperation.

Amit Kumar Karna (4403) Devender Budhwar (4411) Sahil Sandhu (4455)



Abstract Introduction Kalman Filter State-space Representation Underlying Dynamic System Model Code Results Conclusions References


Sahil Sandhu ¾ B.Tech (E.C.E.) National Institute of Technology Warangal sahilsandhu@gmail.com Devender Budhwar ¾ B.Tech (E.C.E.) National Institute of Technology Warangal dev.budhwar@gmail.com Amit Kumar Karna ¾ B.Tech (E.C.E.) National Institute of Technology Warangal karnamit@gmail.com

It is now quite common in the recursive approaches for motion estimation, to find applications of the Kalman filtering technique both in time and frequency domains. In the block-based approach, very few approaches are available of this technique to refine the estimation of motion vectors resulting from fast algorithms.. This paper proposes an object motion estimation which uses the Kalman filtering technique to improve the motion estimates resulting from both the three step algorithm and Kalman application.

Given a macro block in the current frame, the objective is to determine the block in a reference frame (one of the past for forward motion estimation or one of the futures for backward motion estimation) in a certain search area that “best” matches the current macro block according to a specific criterion. In most coding systems, the “best” match is found using the mean absolute error (MAE) criterion. There are several well known algorithms that perform the block matching motion estimation, among them being the full search algorithm (FSA) that determines the motion vector of a macro block by computing the MAE at each location in the search area. This is the simplest method, it provides the best performance, but at a very high computational cost. To reduce these computational requirements, several heuristic search strategies have been developed, as for example the two-dimensional logarithmic search, the parallel one-dimensional search, etc [3-51. These are often referred to as fast search algorithms. In fast algorithms the procedures applied for motion estimation are of significantly lower complexity, but yield a suboptimal solution in the sense that they may not avoid the convergence of the MAE cost function to a local minimum instead of the global one. Lately, some new fast strategies as well as motion estimation have been proposed. But, very few applications are available of Kalman filtering for the estimation of motion vectors. This paper proposes a motion estimation using Kalman filtering to improve the motion vector estimates resulting from both the conventional three step algorithm (TSA) based Kalman filter proposed in Section 2 introduces the state-space representation for the motion vector. The corresponding Kalman equations are:

1.0 Introduction
In the field of motion estimation for video coding many techniques have been applied. It is now quite common to see the Kalman filtering technique and some of its extensions to be used for the estimation of motion within image sequences. Particularly in the pixel-recursive approaches, which suit very much the Kalman formulation, one finds various ways of applying this estimation technique both in the time and frequency domains. On a very general perspective, we find use of Kalman filter (KF), which implies linear state-space representations and the extended Kalman filter (EKF), which uses the linearised expressions of non-linear state- space formulations. Moreover, the parallel extended Kalman filter (PEKF) which consists of a parallel bank of EKF’s, is often encountered in practice. In the block-based motion-compensated prediction approaches, the most common procedure is the block- matching technique.


(optimal Kalman gain) (updated state estimate) (updated estimate covariance) The formula for the updated estimate covariance above is only valid for the optimal Kalman gain. Usage of other gain values require a more complex formula found in the derivations section. Invariants If the model is accurate, and the values for and accurately reflect the distribution of the initial state values, then the following invariants are preserved: all estimates have mean error zero

2.0 The Kalman Filter
The Kalman filter is a recursive estimator. This means that only the estimated state from the previous time step and the current measurement are needed to compute the estimate for the current state. In contrast to batch estimation techniques, no history of observations and/or estimates is required. It is unusual in being purely a time domain filter; most filters (for example, a low-pass filter) are formulated in the frequency domain and then transformed back to the time domain for implementation. The state of the filter is represented by two variables: , the estimate of the state at time k; , the error covariance matrix (a measure of the estimated accuracy of the state estimate). The Kalman filter has two distinct phases: Predict and Update. The predict phase uses the estimate from the previous timestep to produce an estimate of the current state. In the update phase, measurement information from the current timestep is used to refine this prediction to arrive at a new, (hopefully) more accurate estimate. Predict (predicted state) (predicted estimate covariance) Update (innovation or measurement residual) (innovation (or residual) covariance)

where E[ξ] is the expected value of ξ, and covariance matrices accurately reflect the covariance of estimates

The Kalman filter can be regarded as an adaptive low-pass infinite impulse response Digital filter, with cut-off frequency depending on the ratio between the process- and measurement (or observation) noise, as well as the estimate covariance. Frequency response is, however, rarely of interest when designing state estimators such as the Kalman Filter, whereas for Digital Filters, frequency response is usually of primary concern. For the Kalman Filter, the important goal is how accurate the filter is, and this is most often decided based on empirical Monte Carlo simulations, where "truth" (the true state) is known. In the extended Kalman filter (EKF) the state transition and observation models need not be linear functions of the state but may instead be (differentiable) functions.

The function f can be used to compute the predicted state from the previous estimate and similarly the function h can be used to compute the predicted measurement from the predicted state. However, f and h cannot be applied to the covariance directly. Instead a matrix of partial derivatives (the Jacobian) is computed.


At each timestep the Jacobian is evaluated with current predicted states. These matrices can be used in the Kalman filter equations. This process essentially linearizes the non-linear function around the current estimate. This results in the following extended Kalman filter equations: Predict


Where the state transition and observation matrices are defined to be the following Jacobians

However, when the Kalman filter is used to estimate the state x, the probability distribution of interest is that associated with the current states conditioned on the measurements up to the current timestep. (This is achieved by marginalizing out the previous states and dividing by the probability of the measurement set.) This leads to the predict and update steps of the Kalman filter written probabilistically. The probability distribution associated with the predicted state is product of the probability distribution associated with the transition from the (k − 1)th timestep to the kth and the probability distribution associated with the previous state, with the true state at (k − 1) integrated out.

The measurement set up to time t is The probability distribution of updated is proportional to the product of the measurement likelihood and the predicted state.

Relationship to recursive Bayesian estimation The true state is assumed to be an unobserved Markov process, and the measurements are the observed states of a hidden Markov model.

The denominator is a normalization term. The remaining probability density functions are

Because of the Markov assumption, the true state is conditionally independent of all earlier states given the immediately previous state. Similarly the measurement at the k-th timestep is dependent only upon the current state and is conditionally independent of all other states given the current state. Using these assumptions the probability distribution over all states of the HMM can be written simply as:

Note that the PDF at the previous timestep is inductively assumed to be the estimated state and covariance. This is justified because, as an optimal estimator, the Kalman filter makes best use of the measurements, therefore the PDF for given the measurements filter estimate. is the Kalman

3.0 State-space representation
The scanning in a frame is from the top left to the bottom right. The motion vector of a macroblock can be predicted from that of its left spatial neighbour. The measured motion vectors


are obtained through a conventional three step procedure. In the same manner as in, the intraframe motion estimation process is mlodelled through an autoregressive model which produces the state-space equations. From this macroblock-based representation how do we define the 8x8-block: based representation? Each 16x16-block yields a zig-zag sequence of four 8x8 blocks:

motion estimate V(k/k) when the assumption of smooth changes is not strictly valid. As a result, it is expected that we have a better motion estimate for the 8x8-motion compensation procedure.

4.0 Underlying dynamic system model
Kalman filters are based on linear dynamical systems discretised in the time domain. They are modelled on a Markov chain built on linear operators perturbed by Gaussian noise. The state of the system is represented as a vector of real numbers. At each discrete time increment, a linear operator is applied to the state to generate the new state, with some noise mixed in, and optionally some information from the controls on the system if they are known. Then, another linear operator mixed with more noise generates the visible outputs from the hidden state. The Kalman filter may be regarded as analogous to the hidden Markov model, with the key difference that the hidden state variables are continuous (as opposed to being discrete in the hidden Markov model). Additionally, the hidden Markov model can represent an arbitrary distribution for the next value of the state variables, in contrast to the Gaussian noise model that is used for the Kalman filter. There is a strong duality between the equations of the Kalman Filter and those of the hidden Markov model. A review of this and other models is given in Roweis and Ghahramani (1999). In order to use the Kalman filter to estimate the internal state of a process given only a sequence of noisy observations, one must model the process in accordance with the framework of the Kalman filter. This means specifying the matrices Fk, Hk, Qk, Rk, and sometimes Bk for each time-step k as described below.

This corresponds to a conventional pixel decimation for block matching in an 8x8-bock. The motion vector of these sub-blocks is defined as V' = (vx,vy), where v, and v, denote the horizontal and vertical components, respectively. These two components are assumed independent. The motion vector of an 8x8-block can be predicted (through KF) from the one of the previous 8x8-block according to the time index k of the zig-zag order using the state equation: V(k+ 1 )=F(k)V(k) +W(k) where W(k)is the state noise vector, i.e. the prediction error with covariance matrix Q. The state noise components w,, w, are assumed independent and Gaussian distributed with zero-imean and same variance q. The matrix F(k) is the transition matrix which describes the dynamic behaviour of the motion vector from one 8x8-block to the next: It is observed that the measured motion vectors are actually obtained from the TSA run on a 16x16-block basis that yields the zigzag sequence of four measured motion vectors Y(k) with same values on the 8x8-block basis. This means that the Kalman filter has four measurements instead of one to adjust the

Fig. Model underlying Kalman filter Model underlying the Kalman filter. Circles are vectors, squares are matrices, and stars represent Gaussian noise with the associated covariance


matrix at the lower right. The Kalman filter model assumes the true state at time k is evolved from the state at (k − 1) according to: Where, Fk is the state transition model which is applied to the previous state xk−1; Bk is the control-input model which is applied to the control vector uk; wk is the process noise which is assumed to be drawn from a zero mean multivariate normal distribution with covariance Qk. At time k an observation (or measurement) zk of the true state xk is made according to where Hk is the observation model which maps the true state space into the observed space and vk is the observation noise which is assumed to be zero mean Gaussian white noise with covariance Rk. The initial state, and the noise vectors at each step {x0, w1, ..., wk, v1 ... vk} are all assumed to be mutually independent. Many real dynamical systems do not exactly fit this model; however, because the Kalman filter is designed to operate in the presence of noise, an approximate fit is often good enough for the filter to be very useful. Variations on the Kalman filter described below allow richer and more sophisticated models.

imshow(Im) Imwork = double(Im); %extract ball [cc(i),cr(i),radius,flag] = extractball(Imwork,Imback,i);%,fig1,fig2,fig3,fi g15,i); if flag==0 continue end hold on for c = -0.9*radius: radius/20 : 0.9*radius r = sqrt(radius^2-c^2); plot(cc(i)+c,cr(i)+r,'g.') plot(cc(i)+c,cr(i)-r,'g.') end %Slow motion! pause(0.02) end figure plot(cr,'g*') hold on plot(cc,'r*')

% extracts the center (cc,cr) and radius of the largest blob function [cc,cr,radius,flag]=extractball(Imwork,Imback,in dex)%,fig1,fig2,fig3,fig15,index) cc = 0; cr = 0; radius = 0; flag = 0; [MR,MC,Dim] = size(Imback); % subtract background & select pixels with a big difference fore = zeros(MR,MC); %image subtracktion fore = (abs(Imwork(:,:,1)-Imback(:,:,1)) > 10) .. | (abs(Imwork(:,:,2) - Imback(:,:,2)) > 10) ... | (abs(Imwork(:,:,3) - Imback(:,:,3)) > 10); % Morphology Operation erode to remove small noise foremm = bwmorph(fore,'erode',2); %2 time % select largest object labeled = bwlabel(foremm,4); stats = regionprops(labeled,['basic']);%basic mohem nist [N,W] = size(stats); if N < 1 return end

5.0 Code <Detect.m>
%detect clear,clc % compute the background image Imzero = zeros(240,320,3); for i = 1:5 Im{i} double(imread(['DATA/',int2str(i),'.jpg'])); Imzero = Im{i}+Imzero; end Imback = Imzero/5; [MR,MC,Dim] = size(Imback); % loop over all images for i = 1 : 60 % load image Im = (imread(['DATA/',int2str(i), '.jpg']));



% do bubble sort (large to small) on regions in case there are more than 1 id = zeros(N); for i = 1 : N id(i) = i; end for i = 1 : N-1 for j = i+1 : N if stats(i).Area < stats(j).Area tmp = stats(i); stats(i) = stats(j); stats(j) = tmp; tmp = id(i); id(i) = id(j); id(j) = tmp; end end end % make sure that there is at least 1 big region if stats(1).Area < 100 return end selected = (labeled==id(1)); % get center of mass and radius of largest centroid = stats(1).Centroid; radius = sqrt(stats(1).Area/pi); cc = centroid(1); cr = centroid(2); flag = 1; return

kfinit=0; x=zeros(100,4); % loop over all images for i = 1 : 60 % load image Im = (imread(['DATA/',int2str(i), '.jpg'])); imshow(Im) imshow(Im) Imwork = double(Im); %extract ball [cc(i),cr(i),radius,flag] = extractball(Imwork,Imback,i); if flag==0 continue end hold on for c = -1*radius: radius/20 : 1*radius r = sqrt(radius^2-c^2); plot(cc(i)+c,cr(i)+r,'g.') plot(cc(i)+c,cr(i)-r,'g.') end % Kalman update i if kfinit==0 xp = [MC/2,MR/2,0,0]' else xp=A*x(i-1,:)' + Bu end kfinit=1; PP = A*P*A' + Q K = PP*H'*inv(H*PP*H'+R) x(i,:) = (xp + K*([cc(i),cr(i)]' - H*xp))'; x(i,:) [cc(i),cr(i)] P = (eye(4)-K*H)*PP hold on for c = -1*radius: radius/20 : 1*radius r = sqrt(radius^2-c^2); plot(x(i,1)+c,x(i,2)+r,'r.') plot(x(i,1)+c,x(i,2)-r,'r.') end pause(0.3) end % show positions figure plot(cc,'r*') hold on plot(cr,'g*') %end %estimate image noise (R) from stationary ball posn = [cc(55:60)',cr(55:60)']; mp = mean(posn);

clear,clc % compute the background image Imzero = zeros(240,320,3); for i = 1:5 Im{i} = double(imread(['DATA/',int2str(i),'.jpg'])); Imzero = Im{i}+Imzero; end Imback = Imzero/5; [MR,MC,Dim] = size(Imback); % Kalman filter initialization R=[[0.2845,0.0045]',[0.0045,0.0455]']; H=[[1,0]',[0,1]',[0,0]',[0,0]']; Q=0.01*eye(4); P = 100*eye(4); dt=1; A=[[1,0,0,0]',[0,1,0,0]',[dt,0,1,0]',[0,dt,0,1]']; g = 6; % pixels^2/time step Bu = [0,0,0,g]';


diffp = posn - ones(6,1)*mp; Rnew = (diffp'*diffp)/5;

computational complexity. Indeed, due to the basic formulation of the motion, the Kalman filter equations can be largely simplified and thus reduced to their scalar form. Subsequently, any expensive matrix calculation is avoided.

For comparison purposes, the Full Search Algorithm (FSA), the Three Step Algorithm (TSA), the 16x16-block based Kalman Filter (16x16-KF) presented in [22] and the proposed 8x8-block based Kalman Filter (8x8-KF) are run for different classes of video sequences. We use 50 frames of the following sequences: ‘Alkistis’, ‘Carphone’, ‘Foreman’ and a sub-sampled sequence of ‘News’. It is observed that all the above listed techniques can be qualified as sub-optimal techniques. Even the full search algorithm does not give the real motion. Hence, in any case the results expected are very close in terms of average PSNR (see Table 1). As in, the algorithm using the Kalman filtering on a 16x16-block basis provides a greater average peak signal to noise ratio than the one resulting from the conventional three step algorithm. As expected the 8x8- block based proposed approach results in an even greater PSNR better than both the TSA and 16x16-KF. For the particular state-space representation used, when the real motion corroborate the assumptions made regarding, only translational motion with very smooth changes, the 8x8-block based motion estimation using Kalman filter is even better than the one resulting from the full search algorithm, as one can see from the average PSNR values and the curves in Figure 1. The computational complexity of each of the above stated algorithms is given in Table 2. The complexity is evaluated in terms of number of operations per block (NOPB). The formulation of the complexity uses N=16 and p=7, where N defines the size of a NxN-block and p the maximum displacement within the search area. The full search requires (2p+1)’ search locations and the three step 25. The Kalman filter is usually of high complexity. But for our particular application, that uses an intentionally simple state-space model for the motion, it is observed from Table 2 that the Kalman filter implementation for refining the motion estimates resulting from the TSA, does not significantly increase the

7.0 Conclusion
The above presented results are encouraging, in the sense that with the appropriate state model and a priori assumptions close to the real motion vector behaviour, we are able through Kalman filtering to have a greater PSNR than the other techniques for any frame of the sequence. It is evident that a Kalman filter using such simple formulation as in this paper is more suitable for class A image sequences. At this point, there are few questions I alternatives that may be of interest: -Shall we consider sub-pixel accuracy? -Shall we leave the motion components independency assumption and consider the real correlation that exists between the displacement in x- and y-direction? -Shall we envisage more elaborate state-space representation for the motion that will imply implementation of extended Kalman filters and parallel Kalman filters? Any of the above consideration is feasible. Regarding the usefulness of implementing such approaches, this will be a matter of trade-off among the resulting motion estimates, the computational complexity and the real visual quality of the images.

[1] H. G. Musmann, P. Pirsch and H.-J. Grallert,
‘Advances in picture coding’, Proc. IEEE, Vol. 73, No. 4, pp. 523-548, 1985. [2] I. Aksu, F. Jldiz and J. B. Burl, ‘A comparison of the performance of image motion operating on low signal to noise ratio images’, Proc. of the 34th Midwest Symposium on Circuits and Systems, Vol. 2, pp. 1124-1 128, NY 1992. [3] A. Murat Tekalp, ‘Digital video processing’, Prentice-Hall, 1995.

[4] Internet resources


Sign up to vote on this title
UsefulNot useful