Proeeedings of the 2009 IEEE

International Con ference on Meehatronies and Automation

August 9 - 12, Changehun, China

Moving Object Tracking based on Mobile Robot Vision

Lin Rui, Du Zhijiang and Sun Lining
State Key Laboratory of Robotics and System
Harbin Institute of Technology
Harbin.Heilongjiang Province, China
{linrui,duzjOl,lnsun }@hit.edu.cn

Abstract-The paper describes a robotic application that
tracks a moving objcct by utilizing a mobile robot with capacity
of avoiding obstacles. Real-time tracking algorithm is based on
mobile robot vision using adaptive color matching and Kalman
filter. Adaptive color matching is used to obtain motion vectors
of moving object in the robot coordinate system . It can adjust
color matching threshold adaptively to reduce the influence of
lighting variations in the scene. Kalman filter is applied to design
a view window. View window, which contains the next position
of the moving object estimated by Kalman filter, is determined
on image plane to reduce the image processing area. Color
matching threshold can adjust itself adaptively in view window.
A virtual targets based algorithm is also presented for real-time
obstacle avoidance in this paper. Experimental results show
that the tracking algorithm can adapt to lighting variations
and has good tracking precision. It can also avoid obstacles
smoothly while tracking moving object. Furthermore, it can be
implemented in real time.
Index Terms-Virtual targets.View window.Real-time track-
ing.Avoiding obstacles
I. I NTROD UCTIO N
For many robot domains, such as robot soccer, the problem
of tracking objects and avoiding obstacles has received a good
deal of attention over the last few years. There are many
techniques for the problem. Many methods are proposed to
track moving objects in [1]-[4]. Tracking algorithm using
vision should adapt to the complex variations of lighting and
backgrounds , as well as be of low computational complexity.
Such methods have some limits in a dynamic environment
dynamic world. The key technique of tracking moving object
is to identify moving object from the surrounding objects
in image sequences, and then calculate the motion vector
of moving object. Adaptive color matching is carried out to
reduce influence of lighting variations and to attain motion
vector of moving object relative to mobile robot, which is
used in obstacles avoidance algorithm. A view window is
designed by Kalman filter on image plane synchronously. It
is applied to limit possible search space for a moving object
in image sequences and to attain the best matching results
for searching moving object.
While tracking moving object, the mobile robot should
have real-time obstacle avoidance capacity. We propose vir-
tual targets based algorithm to avoid collision in dynamical
environment with stationary and moving obstacles. Fuzzy
control is used to obtain a set of virtual targets to avoid
obstacles. Strategies for formulating the fuzzy rule sets for
motion and obstacle avoidance have been proposed. Then we
applied Virtual Force Method to calculate the control values:
linear velocity v and angular velocity w.Then the mobile
robot moves toward the virtual target directly. The method
can fulfill the requirements of tracking an object and avoiding
obstacles simultaneously in real time.Fig.1 shows the whole
algorithm architecture.
Camera Laser Sensor
Image Sequences Obstacles Detecting

while the camera is moving with mobile robot or the back-

ground changes a lot. The mobile robot should have capacity Jl Jl
of avoiding obstacles while tracking moving objects . Aiming Adap tive co lor ma tching Fuzzy Con trol
at the problem of obstacles avoidance, several traditional Motion Vector of Moving Objec t Virtual Targets
control methods have been applied, like Potential Field [5]
and Virtual Force Field [6]. But they may not work well in an
unknown and dynamical environment. In addition, intelligent
Jl lr Jl
Kalman Filter Virtua l Force Field
control methods such as fuzzy control and artificial neural
network [7] have been developed greatly. View Window Control Values

In this paper, we propose a real-time tracking algorithm

for a moving object using mobile robot vision. It is based Jl
on adaptive color matching and Kalman filter. We focus Robot Con tro l Onboard Computer
on the problem of robot vision for a robot operating in a I
* This work was supported by National High Technology Research and
Development Program of China (2005AA404290) Fig. 1. The whole algorithm architecture.

978-1-4244-2693-5/09/$25.00 ©2009 IEEE 3625

 II. TRACKING ALGORITHM
A. Adaptive Color Matching
While an object is moving in the dynamic environment at
higher speed, color of moving object on image plane may
change a lot because of lighting variations in the scene. In
addition to lighting variations, there are additional problems
of non-diffuse lighting causing specula reflections, shadows,
and large variations in color from the sun-side to the shade-
side. In order to reduce influence of lighting variations,
we utilize lighting-compensating algorithm in image pre-
processing. The lighting-compensating algorithm is partially
carried out in view window region of moving object only. As
for color features of moving object, we adopt YCrCb color
space to reduce the influence of illumination. We use Cr and
Cb values for color matching. The YCrCb color values can
be obtained from RGB color values by equation
(1)
Where, Md is conversion matrix; M T is constant matrix;
M p c is the RGB value matrix of image after lighting com-
pensating.
We select an initial label for image pixels classification.
The combination of initial-labeling and adaptive color match-
ing provides the plasticity for lighting variations. To label
pixels according to whether they belong to moving object,
we utilize an initial-labeling scheme. Essentially, we examine
each pixel p on image plane and estimate the likelihood of
features of each pixel. After an initial-labeling, the image will
be transferred to binary image in order to reduce calculation
cost. The white speckles appear because there are some noise
and other objects with similar color in the scenes. These
regions of speckles may be not small and therefore are not
easily filtered. Such speckles are common in the indoor
scenes, which will influence the search space of moving
object if they are not filtered precisely.
TABLE I
THE FEATURE USED TO FILTER REGIONS
Feature Description
Area Number of pixels in region
Shape Ratio of width to height
Color uniformity The color uniformity in region
Possible position The location relative to the robot estimated by
Kalman filter
In order to reduce the computational cost, we filter the
small regions whose pixel numbers are smaller than thresh-
old, such as the regions of single speckle. After coarse filter-
ing, we define some features of moving object to filter out the
noise of these regions. We set weight coefficients for these
likelihood features. Table 1 lists the 4 features we use to filter
noise regions. Shape is the ratio of width to height. As for a
ball, the ratio is approximately equal one. Area is the number
3626
of pixel in the moving region. It is not constant and has
certain relation with distance of moving object relative to the
mobile robot. Color uniformity represents color uniformity
level of the whole region of moving ball. Possible position
means the possible pixel position of moving ball centroid
estimated by Kalman filter. Here possible position is used to
limit the search space on the image plane. We assume that
each feature is independent and set theses features different
weight coefficients. The features describe the likelihood of
bonding box dimensions, locations of the regions compared
to some expectation, and area of regions. We assume that
the features can be matched probabilistically and have a
Gaussian uncertainty distribution. Lastly, results of matching
probability are given in (2).
M
r, == L a.S, j == I ...N (2)
i=l
Where P, is the character of the region being matched;
S, represents the region features; siqma; is the weight
coefficient. N is the number of regions to be matched after
coarse filtering. M is the number of likelihood features and
here M is 4. Every possible region after coarse filtering has a
matching value Pj . If the maximum matching value is bigger
than matching threshold which is set by user in advance, the
region with maximum matching value is reserved and other
regions are filtered out. Otherwise, the mobile robot does not
find moving object and begins to implement task of searching
moving object.
After threshold adapting and region searching, the moving
object can be recognized on image plane. The motion vector
of object can be calculated in the robot coordinate system.
After calculating the centroid vector M k of region on image
plane, we can obtain the position of moving object relative
to mobile robot by
(3)
Where, Wk represents the position of moving object ,
which the mobile robot should move to, in the 3D robot co-
ordinate system; C- 1 R; 1 R;; 1 p- 1 is the conversion matrix,
which converts the image coordinates into the 3D robot coor-
dinate system. The conversion matrix includes some system
parameters such as focal length of camera, the rolling angle
() c and pitching angle O:c of camera, converting distance from
the image coordinate to the origin of 3D robot coordinate
system and etc. Wk is used as measure matrix by Kalman
filter. It is also applied to judge its position relation with
obstacles and mobile robot in the next section. The position
relation is taken as an input value of fuzzy control.
B. View Window Designed by Kalman Filter
In this paper, we utilize Kalman filter to estimate the
next position and velocity of moving object relative to the
mobile robot and to set a view window on image plane. In
other words, we design Kalman filter to reduce the space
where moving object is searched for, and to attain its motion
vector W Zk from W k at the next frame time. We assume
that the velocity of moving object relative to mobile robot
remains changeless in the course of image processing at every
frame time. This is equivalent to encoding a tracking history
into state parameters of Kalman filter and estimating motion
vector of moving object relative to the robot. In order to
adjust the color matching threshold, we define a view window
according to the resulting estimates. View window is mainly
used for updating the matching threshold. The robot tracks
the centroid of moving object in the robot coordinate system.
The center of view window is possible position of moving
object centroid on image plane although there are position
error and velocity error. The size of moving object on image
plane changes along with the distance from moving object
relative to the mobile robot. View window will change with
the size of moving object on image and the velocity of it to
ensure including the region of moving object. We can use (4)
to calculate the coordinate of moving object relative to the
mobile robot.
(4)
The center of view window is set by M v w . Its size is
determined by the distance and velocity of moving object
relative to the mobile robot. It is a possible region of moving
object at the next frame time. It is used to reduce the
space of image sequences where the lighting configuration
is re-counted. We search for Ym , which represents the Y
value of the most pixels in view window. As we know,
moving object in the possible area is the largest region in
view window. Y values in view window mainly present the
lighting configurations of moving object. Cr and Cb color
configuration have a certain non-linear relation with Y value.
We can adjust the Cr and Cb color matching thresholds by
equation:
M N
(5)
Ym == most(L L Yi*width+j)
j
[Crth Cbth]T == f(Ym ) (6)
Where, Ym represents the Y value whose pixels are
the most in view window; Crth and Cb th are the color
matching thresholds at the next frame time; Then adaptive
color matching evolves from the current frame to the next,
capturing the changes in the image size of an object as it
moves.
3627
III. OBSTACLES AVOIDANCE ALGORITHM
A. Virtual Targets Derived from Fuzzy Control
While tracking moving object, the mobile robot should
have real-time obstacle avoidance capacity. In order to avoid
collision with obstacles, the mobile robot system needs to
judge the direction in which the robot should move and to
generate proper instructions to low-level drivers, by consid-
ering the obstacle-free distances and their time differences.
Virtual target are applied to be the next temporary targets
mobile robot will move toward while tracking moving ob-
ject. To obtain effective virtual targets, the system should
take all account of the parameters of obstacles surrounding
mobile robot. Fuzzy control provides a useful mechanism
for judging a situation correctly and generating a proper
control instruction. The developed rule base for virtual targets
consists of 108 rules. These rules are classified into the
following 4 categories. These are the number of obstacle
clusters, direction of real target relative to mobile robot,
obstacle avoidance layers and position relation of target and
obstacle clusters.
We set 2 meters as an available maximum range limit for
detecting obstacles. obsti(d s , as, de, a e , d m , am) is applied
to describe contour parameters of obstacle i . Where, ds ,
as, de, a e , d m and am are the initial, terminal, and nearest
distance and angle parameters of obstacle i relative to mobile
robot in laser polar coordinate system.
J d;,i + dL - 2d e,i d s ,j cos (O:e,i - O:s,j) < dt h (7)
Where, dth is a safe threshold of desired size for mobile
robot to pass. No is the final number of obstacle clusters after
combination. No is fuzzitied and expressed by the fuzzy sets:
N Z , N Land N M, where they stand for zero, less and more.
The membership function is depicted in Fig. 2(a).
The obstacles exert strong effects on motion of mobile
robot when they are in the immediate vicinity of the robot,
and weak effects when they are further away. So layered
obstacle avoidance policy is applied according to the distance
between obstacles and mobile robot. The mobile robot's work
area is divided into three layers, exigent obstacle avoidance
layer, normal obstacle avoidance layer and directly advancing
layer according to the radius H; and R n .
The distributing area of obstacle clusters surrounding
mobile robot is categorized into three memberships: LE( ==
exigent obstacle avoidance layer), LN (== normal obstacle
avoidance layer), LD (== directly advancing layer), as shown
in Fig.2 (b).
The direction of real target relative to mobile robot is
fuzzified and expressed by the fuzzy sets: FL, FR, BL and
BR ,where they stand for front-left, front-right, back-left
and back-right, respectively. The target may be between two
obstacle clusters, or in front of some obstacle cluster, or
 1.0

(a) The number of obstacle clusters

LE LN LD
1.0

(a) Virtual target is the same as the position of real

target

(b) The obstacle avoidance layers

:xk -20 0

(c) The obstacle avoidance layers

20 t ill

<rlX>.
-180 -135 -90 -45 0 45 90

(d) The direction of moving object relative to mobile robot

135 180 deg
(b) Virtual target to detour obstacle cluster

Fig. 3. Virtual target derived from fuzzy control

at the back of some obstacle cluster. It is fuzzified and

1.0 expressed by the fuzzy sets: PN, PF, and PB respectively.
Their membership functions are depicted in Fig.2 (c) and
(d).
The virtual target as final output of fuzzy control is updated
at every control interval time. The membership functions of
output variables dv and (tv are depicted in Fig. 2 (e) and
(e) The position between obstacle clusters and moving (f) respectively. Its coordinate is described as (dv, (tv ) in the
object
laser polar coordinate system. Then it can be converted into
4 1.0 Pv(x v , Yv, (tv ) in 20 world coordinate system by:

¥illfJftljjj)I.
- 180 -150 -120 -90 -60 -30 0 30 60

(I) The angle virtual target relative to mobile robot

90 120 150 180 dcg
Xv = Xr + dv cos(Or + (tv -
Yv = Yr + dv sin(Or + (tv -
7f

7f
/ 2)
/ 2)
(8)
(9)

Where, P; (x r , Yr , (tr) is present coordinate of mobile

Fig. 2. Fuzzy membership functions of input and output variables robot in 2D world coordinate system. Some instances of
virtual targets are derived from fuzzy control as shown in
Fig. 3.

3628
B. "Local Minimum Traps" of Virtual Targets
One problem of obstacle avoidance based on virtual targets
is the possibility for the mobile robot to get "trapped". This
situation may occur when the virtual target is the same as
the current position of mobile robot. Then the robot will not
move any more. Traps can be created by a variety of different
obstacle configurations, and different types of traps can be
distinguished easily. This section presents a comprehensive
set of rules to recover from the trap condition.
In an ideal, trap-states may be discovered by simply
monitoring the speed of the robot. If caught in a trap, the
robots speed will become zero as the robot converges to the
equilibrium position. In a dynamic system, however, the robot
may stop to avoid moving obstacles in the exigent obstacle
avoiding layer and will either oscillate or run in a closed
loop. Therefore, it is practical to monitor the position of
virtual targets relative to mobile robot. Firstly, our method for
trap-state detection is comparing virtual target with current
position of mobile robot. If the virtual target is the same as
the current position or F t is a zero vector, the mobile robot is
very likely about to get trapped . Second, the Virtual Target-
To-Robot directions will be recorded. If the direction angles
vibrate around zero and the robot' linear velocity is equal to
zero, the robot is very likely about to get trapped. Third, if
the robot's direction of travel is more than a certain degree
Co ( Co > 360°) set beforehand, the robot is also very likely
about to get trapped. Co > 360° indicates that the robot has
traveled at least one full loop.
Once detected, there are several ways to remedy the
situation . On these trap condition, a random angle etr and
a random distance d; will be added to the angle value etr
and distance value d; of virtual target respectively. et r and
d; should satisfy :
letr I < Cn and letr I =1= 0 (10)
Cd < d; < 0 (I I)
Where, Co and Cd are constants . The recovery algorithm
is computationally simple, and does not reduce the overall
performance (if evoked only temporarily), even if the mobile
robot would not be trapped.
The recovery algorithm is computationally simple, and
does not reduce the overall performance (if evoked only
temporarily), even if the mobile robot would not be trapped.
IV. CONTROL VALUES SENT TO ONBOARD
COMPUTER
In order to control mobile robot to track moving object,
two control values must be sent to its onboard computer
to control the two tracking motors: linear velocity v and
angular velocity w. The virtual target (dv , et v) is the final
output of fuzzy control. It is also a temporary target mobile
robot advances toward at the current interval time . The final
3629
Fig. 4. Research platform HR-l and Coordinate systems
control values for linear velocity t/ and angular velocity w
can be calculated based on Virtual Force Method by
(12)
(13)
Where, F et is force constant; Me is coefficient matrix; F t
is an attracting force vector. It is generated by the virtual
target point P; (xv ,Yv), whose coordinates are outputs of the
fuzzy control. At any time during tracking moving object, Ft
pulls the mobile robot toward the virtual target. The repulsive
force vector F; generated by obstacle clusters is omitted. So
the mobile robot can keep on moving toward virtual targets
to detour obstacles while tracking moving object.
V. EXPERIMENTAL RESULTS
This section presents some experimental results which
illustrate the operational characteristics of the proposed
method. Fig. 4 shows our robot research platform and robot
coordinate systems. The camera and Laser sensor are fixed
up on the top of mobile robot. We set a frame interval to be
III 5 seconds, since the current implementation of the method
requires such a processing time per frame for a video image
(320x240 pixels with 24 bits) . The maximum linear velocity
v and angular velocity w of the mobile robot are set to be
700 mm/sec and 50 degree/sec, respectively . We evaluated
the suggested methods under realistic tracking conditions.
The initial model of moving ball can be made manually by
user or automatically by some identification module . Then
the mobile robot begins to track the moving orange ball.
As shown in Fig.5, the green rectangle is the region
detected by adaptive color matching at the current frame
time . The red rectangle around the moving ball is the region
estimated from the previous frame by Kalman filter. It is
also taken as a view window. While the ball moves at lower
speed relative to the mobile robot, the green rectangle almost
overlaps with the red rectangle on image plane . When the ball
moves at higher speed, the red rectangle has bigger size than

Fig. 5. Experiment results
the green rectangle . And the centers of two rectangles have
a little deviation because of position error and velocity error.
When the moving ball is partly obstructed by obstacles or an
obstacle, the mobile robot can track it well. The algorithm
is able to adapt and continue to recognize the moving ball
despite lighting variations. Although there are some objects
with similar color and shape in the scene, the robot can also
track the moving ball precisely. While tracking the moving
ball, the mobile robot can detour the stationary and moving
obstacles in the environment almost smoothly.
VI. CONCLUSIONS
In this paper, we have proposed a technique for real time
tracking moving object and avoiding obstacles algorithm used
in robot vision domains, such as robot soccer. The key to
tracking algorithm resides: an adaptive color matching to
calculate the present motion vector of moving object and the
Kalman filter used to predict the next possible position, to set
a view window used for adjusting color matching threshold.
The new method has a simple design and is implemented
in real time with very low computational complexity. It is
also robust to lighting variations. The obstacle avoidance
algorithm uses fuzzy control to judge the status of the
obstacles and to generate proper virtual targets for mobile
robot to advance toward. Then the mobile robot can fulfill
avoiding obstacles smoothly while tracking moving object.
The experimental results have shown that the algorithm has
good tracking precision.
Our future work will focus on extending tracking algorithm
to work better for more practical and complex applications.
3630
For example, predicting the pOSItIOn of moving ball when
the moving ball is completely obstructed by obstacles or an
obstacle and recognizing the moving ball from other similar
balls which are moving nearly are currently in progress.
REFERENCES
[1] K. Takaya, R. Malhotra, "Tracking Moving Objects in a Video Sequence
by the Neural Network Trained for Motion Vectors," in 2005 IEEE
Pacific Rim Conference on Communications, Computers and signal
Processing, 2005, pp. 153-156.
[2] D. Jang, H. Choi, "Moving Object Tracking Using Active Models," in
1998 International Conference on Image Processing, Chicago, 1998, pp.
648-652 .
[3] B. Browning, M. Veloso, "Real-Time, Adaptive Color-based Robot
Vision," in 2005 IEEEIRSJ International Conference on Intelligent
Robots and Systems, 2005, pp. 3871-3876.
[4] S. Pei, W. Kuo, and W. Huang, "Tracking moving objects in image
sequences using 1-0 trajectory filter," IEEE Signal Processing Letters,
vol. 13, January. 2006, pp. 13-16.
[5] Khatib 0, " Real time obstacle avoidance for manipulators and mobile
robots," The International Journal of Robotics Research, vol. 5, no. 1,
1986, pp. 90-98.
[6] Borenstein J, Koren Y, " Real-time obstacle avoidance for fast mobile
robots," IEEE Transactions on Systems, Man, and Cybernetics, vol. 19,
September. 1989, pp. 1179-1187.
[7] Liu X., Peng L., Li J., Xu Y., "Obstacle avoidance using fuzzy neural
networks," in Proceedings of the 1998 International Symposium on
Underwater Technology, Tokyo, 1998, pp.282-286.
[8] C. Chen, C. Cheng, D. Page, A. Koschan, and M. Abidi, "A Moving Ob-
ject Tracked by A Mobile Robot with Real-Time Obstacles Avoidance
Capacity," in the 18th International Conference on Pattern Recognition,
Hong Kong, 2006, pp. 1091-1094.
[9] B. A. G, P. Ioannis, "Prediction and tracking of moving objects in image
sequence," IEEE Transactions on Image Processing, vol. 9, August.
2000, pp. 1441-1445.
[10] Y. Cho, J. Lee, " High Speed Tracking Algorithm of Moving Object,"
Industrial Electronics, vol. 3, June. 2001, pp. 1563-1568.
[11] P.Lanvin, J-C.Noyer, and M.Benjelloun, "Object detection and tracking
using the particle filtering," 2003. Conference Record of the Thirty-
Seventh Asilomar Conference on Signals, Systems and Computers, vol.
2, November. 2003, pp. 1595-1599.
[12] E. Malis, E. Marchand, "Experiments with robust estimation techniques
in real-time robot vision," in Proceedings of the 2006 International
Conference on Intelligent Robots and Systems, Beijing, 2006, pp. 223-
228.
[13] F.Belkhouche, B.Belkhouche, "On the tracking and interception of a
moving object by a wheeled mobile robot," Robotics, Automation and
Mechatronics , vol. I, December. 2004, pp. 130-135.
[14] D.Hsu, J.Leu,S.Chen, W. Chang, W. Fang, "Using shape feature
matching to track moving objects in image sequences," Multimedia and
Expo, vol. 2, Aug. 2000, pp. 785-788.
[15] Lei B., Li w., " A Fuzzy Behaviours Fusion Algorithm for Mobile
Robot Real-time Path Planning in Unknown Environment," In the 2007
IEEE International Conference on Integration Technology, Shenzhen,
2007, pp. 173-178.
[16] Foka A F, Trahanias P E, " Predictive control of robot velocity to avoid
obstacles in dynamic environments," In the 2003 IEEE International
Conference on Intelligent Robots and Systems, Las Vegas, 2003, pp.
370-375.
[17] X. Chen and Y. Li, "Smooth Formation Navigation of Multiple
Mobile Robots for Avoiding Moving Obstacles", International Journal
of Control, Automation , and Systems, Vol. 4, No.4,2006,pp.466-479 .