Proceedings of the 2009 IEEE
International Con ference on Mechatronics and Automation
August 9 - 12, Changchun, China
Vision Based Environment Recognition and
Movement Control for Autonomous Wheeled
Mobile Robots
Gen'ichi Yasuda
Department ofHuman and Computer Intelligence
Nagasaki Institute ofApplied Science
536 Aba-machi, Nagasaki 851-0193, Japan
E-mail: YASUDA_Genichi @NiAS.ac.jp
Abstract-This paper presents an experimental study on
environment recognition and movement control for multiagent
robotic soccer using wheeled mobile robots. A color image
segmentation method based on YUV color information of each
pixel of the color image is applied to the object recognition. The
color image segmentation is performed using the tables that
transform each value in a pixel to membership in several color
classes, where the lower and higher threshold values of Y, U, and
V are determined and described for the sample color markers.
The position of objects on the field is efficiently and accurately
calculated using a monocular camera. A common image
processing method is used to detect landmarks such as
corners/edges in the environment. A movement control algorithm
is implemented based on the distance and orientation to the target.
Experimental results of real-time movement control are also
presented.
Index Terms- Mobile robots, image processing, object
recognition, environment recognition, movement control
I. INTRODUCTIO N
In recent years, there have been many studies on vision-based
motion control in various robotic applications. We are now
developing a multiagent robotic system for soccer game [I]. In
robotic soccer, each of robots must know its surroundings such
as the position of the ball and goals when the robots carry out
cooperative tasks. Furthermore a robot must keep its own
position to estimate the arrangement of objects and share it
among its teammates. The reactiveness ofsoccer robots requires
a vision system with a high processing cycle time.
In this paper, we focus on object recognition, environment
recognition, and movement control for small mobile robots
using only a monocular camera system with the limited viewing
angle. For object recognition, fast color image segmentation is
performed based on YUV color information of each pixel ofthe
color image. The position of an orange ball on the field is
efficiently and accurately calculated using a monocular camera.
Then, suitable environment recognition and movement control
methods for real-time navigation of mobile robots are described
with experimental results.
978-1-4244-2693-5/09/$25.00 ©2009 IEEE 1083
Bin Ge
Liao Ning Key Lab ofIntelligent Information Processing
Dalian University
Economic & Technical Development Zone, Dalian 116622, China
E-mail: gebin58@263.net
II. CONTROL SYSTEM ARCHITECTURE
The robot system is constructed of a control unit (1.5GHz
Pentium 4 CPU) and a mobile base unit. The hardware
configuration is shown in Fig.l. The control unit is equipped
with an image processing board (Hitachi IP5005) and executes
image processing to detect and localize objects in the field. It
works for recognizing the circumstances around the robot,
planning its behaviors, transmitting commands and
communicating with other robots. The robot mounts a
monocular video camera to observe the field or workspace,
target object, and all robots, and sends NTSC video signal to the
image processing board, which converts the video signal to 256
by 220 color digital image, and extracts and measures objects in
every 33msec. The image processing software is implemented
using Visual C++ and the software library of IP5005 under
Windows NT4.0.
CCD Camera
Color image process ing board VHF
(Hitachi IP5005)
Wireless modernt
o
D o
W ireless
LAN
~~-I
Control unit (PC) Mobile base unit
Fig. 1 Hardware configuration of the robot system.
As the mobile base unit, an autonomous mobile robot
platform for experimentation applications in indoor flat-floored
environments (Applied AI Systems, LABO-l) was used. A
view of the robot is shown in Fig.2. The robot body is
rectangular shaped, approximately 310 mm wide and 380 mm
long. The chassis is 30 mm high and rides on a set of four hard
rubber wheels. Two driving motor and wheel sets are located on
both sides, one places near each comer of the vehicle. Wheel
base is approximately 21 em. Wheel diameters are 11 cm. Each
of four motors is controlled individually and drives a single
wheel. A differential drive mechanism with 4 wheels permits
the robot to tum smoothly within a small turning radius.
 result is determined. This point and the optical centre of the
camera define a viewing ray. Its intersection with the ground
plane yields an estimate ofthe object's distance and its direction
in camera coordinates. With the help of the intrinsic camera
parameters pixel coordinates are converted into image
coordinates. The 3-D point corresponding to the observed
imag e point is calculated by the formula specifying the
relationship between the ground plane and the image plane. The
focal length ofthe camera, its height, the angle of inclination are
also used as parameters. The point given in cam era coordinates
can easily be expressed in the coordinate system of the robot .
Fig. 3 shows the relationship between the ground plane and the
image plane .

Fig. 2 View of experimental mobile robot.

The control software was implemented on a

microcontroller-based control system (Motorola MC68332),
H
with the interface circuits for the internal and external sensors.
The mobile robot has 8 infrared sensors: four at the front, one on
each side, and two at the back. The robot can use the infrared
sensors for obstacle avoidance. The CPU communicates with
the control unit via wireless modem (RS-232C interface, y
9600bps). It acts as the controller of the primitive actions (a) Side view
ordered by the control unit, and sends sensor data when the
Screen
control unit requests them.
y
III. COLOR BASED OBJ ECT RECOGNITION

The color image segmentation algorithm [2][3] enables a d

pixel to be classified into one of a number of color classes based D
on color space transformation. After the color segmentation, the
regions are binarized, and dilation and erosion operators are
applied using the 4-neighborhood algorithm, in order to remove
small disturbances and make the regions more compact. Then (b) Bottom view Object
the regions are separated based on color into separate lists using
Fig. 3 Relationship between ground plane and image plane.
a 4-neighborhood labeling algorithm. Next, they are filtered and
sorted by area size so that the smaller noisy regions are removed
and the larger regions are chosen by high level processing The distances estimated by the monocular camera system are
algorithms. The centroid of each region is calculated using a formulated as follows:
library function of IP5005. The system can classify each pixel
and find regions of up to 8 colors, and outputs their centroid and y = HeF - h tan fJ) (I)
area at 30 Hz on general purpose commodity hardware; the FtanfJ+h
image processing time was shortened by pre fetching the next
frame image during processing the current image and by D = dey cos fJ + H sin fJ) (2)
restricting the processing area. Position and orientation
recognition of several objects in the field can be simultaneously F
executed within 33 ms using the library functions provided by where
IP5005. H: height of the camera (=121 mm)
Besides recognizing relevant objects, further tasks are to
F : focal length of the camera (=8mm)
localize the recognized objects and to perform the
self-localization of the robot on the field. The location of an fJ : angle of inclination (=8°)
object is calculated using the formula of a monocular camera h : Y coordinate in image plane
system [4]. All relevant objects are located on the ground of the d : X coordinate in image plane
field. The restriction onto the ground provides an one-to-one
correspondence between a point on the ground plane and its Experiments of object localization were conducted. The
observation in the image plane. In order to estimate the location positions of the orange golf ball in front of the robot were
of an object the lowest point in the corresponding segmentation

1084
depicted in Fig. 4. The result of the experiment is shown in
TABLE I. The row Ax- shows the average estimation errors in x
coordinate, and the row ~y shows those in y coordinate. The
estimation errors are shown in [mm]. A variation of
measurement is sufficiently small for movement control as
shown in Fig. 5.
-250 250
IV. E NVIRONMENT R ECOGNITION
Image processing software has been implemented using the
same control unit for environment recognition in an indoor
experimental space. The flowchart is shown in Fig. 7. First, the
edges surrounding the various objects are amplified to increase
the quality ofthe image. A Laplacian edge detection operation is
performed to accentuate the objects in the image. In binarization,
an appropriate threshold value is set to preserve the important
features using a histogram of the source image, on the
assumption that the two important peaks in the histogram
represent the wall and the floor. Erosion and dilation are used to
remove the enhanced noise pixels. Thinning and shrinking
operations are performed to reduce the image objects and find
their skeleton. An example sequence of images obtained by
image processing procedures is shown in Fig. 8.

Fig. 4 Positions of object.

TABLE I
EXPERIMENTAL RESULT OF OBJECTLOCALIZATION
PI P2 P3
Ax- 18.89 11.02 21.55
~y 7.73 0.14 6.23

611

.
610
609 ;~
608
607
X [mm]606
605
604
~
•
....
... . .! ••-

•
603
o 0.2 0.4 0.6 0.8
Y[mm]

Fig. 5 A variation of measurements of object location.

For robotic soccer, experiments of recognition of an orange
ball were conducted using the image processing system. Fig. 6
shows an example of ball detection.
Fig. 6 Experimental result of orange ball detection: (a)
binarization, (b) centroid calculation.
Fig. 7 Flowchart of image processing for environment
recognition.
After the thinning and shrinking operations, the ordinary
Hough transform is performed to represent the straight lines in
the binary image. The feature points , such as convex or concave
comer, are obtained among their intersections. The boundary
lines between the floor and the wall are discriminated using the
straight lines obtained by Hough transform. Then the movable
area in front of the robot is determined. The distance to the
comer point is calculated on the assumption that the point is on
the floor. In this step, the distance information obtained by some
range finders such as ultrasonic range sensors can be used. The
directions of the target and the obstacle or wall should be
considered in movement control of the mobile robot. For
navigation tasks, a fuzzy logic based sensor fusion algorithm is
used to determine the desired direction of the robot [5].

1085
 control part, behavior control part, and planning part. Real-time
control part executes direct sensor and motor control including
the infrared sensors data acquisition and encoder based position
and velocity controls on the onboard computer in the mobile
base unit. Simple reflex actions brought about by infrared
sensors or bumpers are also executed with the multitask
operating system. Behavior control part monitors the status of
the objects in the environment and commands from the upper
level and selects a suitable primitive action at a time Primitive
actions include wall following, obstacle avoidance, and target
(I) original (2) Laplacian filtering following. Planning part manages the environment map and
human interface, and generates a behavior plan composed of
primitive actions.
For vision based control, the software in the control unit
calculates the following at the control period of 33 ms. First, it
determines the desired translational and rotational velocities
using the position and orientation errors to the goal position.
The motion control algorithm must allow a general
nonholonomic mobile robot to accurately reach a target point
with a desired orientation in a dynamic environment. The
point-to-point (PTP) control method is used to control the
robots to follow transient subgoals. Fig. 9 shows the robot
kinematics for the point-to-point (PTP) control method to
(3) binarization (4) erosion control the robots to follow transient subgoals. The following
feedback control law was adopted:
v=kpp
OJ=kaa+kpfJ (3)

((J=a+B)
where v: the tangent velocity of the robot, OJ : the angular
velocity of the robot, p: the distance between the centre of the
robot's wheel axle and the goal position, a: the angle between
the forward direction of the robot and the vector connecting the
centre of the wheel axle with the goal position, B: the pose
(5) dilation (6) thinning error.
The control parameterskp , k a, kpwere determined [6] so
that the closed control system is locally exponentially stable as
follows:
k p > 0 , ka - k p > 0 , k p < 0 (4)
Furthermore, the parameters were adjusted so that the robot
can be smoothly guided to face the goal [7][8].

Ye;=Y/
f3
(7) shrinking (8) Hough transform goal XG=X/
p
Fig. 8 Sequence of images obtained by image processing
procedures.

v. EXPERIMENTS OF MOV EMENT CONTROL

Control software for autonomous navigation of the wheeled
mobile robot was implemented in the control unit. Overall Fig. 9 Kinematics and movement control of a mobile robot.
control software for the mobile robot is composed of real-time

1086
 Then it calculates the desired angular velocities of the right
and left wheels by a simple, invertible linear transformation.
Finally, it sends 8-bit control data (4-bit data for each motor) to
the onboard computer with wireless modem. The onboard
computer executes multitasking processing with preemptive
task switching at a rate of 128 Hz (approximately every 8
msec). The kernel performs the system-level functions such as
sensor reading and motor control just before the task is switched.
The onboard computer responds to the commands from the host
quickly. The closed loop motor controller is implemented in
software and called by the Period Interrupt Timer interrupt
service routine. Speed and acceleration are measured as the
(a)
values over the interrupt period (1/128 s). Speed profile
parameters and PID coefficients for position control are
configured in software. Thus the angular velocity control loop is
executed on the onboard computer and run every 8 ms to
compensate for differences between desired and estimated
velocity of the wheels. The flowchart of command generation
for movement control executed in the control unit is shown in
Fig. 10, where the turn angle of the robot towards the target a
is calculated in terms of robot coordinates.

(b)
Fig. 11 Example of movement trajectory in the face of the
straight line ofa wall: (a) Hough transform, (b) real image.

send duty cycle data to robot

by serial communication

NO

YES (a)
send stop command to robot

Fig. 10 Flowchart of generation of command for

movement control in the control unit.

For vision based navigation control , experiments of primitive

actions in indoor environments such as wall following, obstacle
avoidance, and goal seeking were conducted.
Examples of movement traje ctory in the face of the straight
edge line and in the face of the corner of a wall are shown with
their Hough transform images in Fig. 11 and Fig. 12,
respectively. Examples of movement trajectory in the face of
(b)
the straight edge line and in the face of the corner of a
rectangular obstacle are shown with their Hough transform Fig. 12 Example of movement trajectory in the face of the
images in Fig. 13 and Fig. 14, respectively.
comer ofa wall: (a) Hough transform, (b) real image.

1087
 VI. CONCLUSIONS
The color image segmentation method has been implemented
on general purpose, inexpensive hardware, and applied to object
recognition for robotic soccer by wheeled mobile robots. The
accuracy of object localization was satisfactory for movement
control in robotic soccer.
Movement control using the localization method has been
implemented. The feedback control law determines the
translational and rotational velocities using the position and
orientation errors to the goal position. Experiments show that
the robot can be smoothly guided to navigate to the goal without
colliding with obstacles.
(a) We are planning to develop a method to construct an
environment map by integrating the visual information of
teammates as the future work. Using the map, the robot can
know the situation outside its view, and perform advanced
behaviors such as role sharing and effective pass work.

REF ERENCES

(b)
Fig. 13 Example of movement trajectory in the front of an
obstacle: (a) Hough transform, (b) real image.
[I] G. Yasuda, and H. Takai, "A microcontroller based architecture for locally
intelligent robot agents", The Japanese Society for Artificial Intelligence,
SIC-challenge Technical Report, no. 4, pp. 11-15,1999.
[2] 1. Bruce, T. Balch, and M. Veloso, " Fast and inexpensive color image
segmentation for interactive robots", Proceedings of IROS'OO, pp.
2061-2066,2000.
[3] G. Yasuda, and B. Gc, " Position control of mobile robots using global
vision with fast color image segmentation", Proceedings of the 3rd
China-Japan Mechatronics Symposium , pp. 191-196,2002.
[4] T. Kubota, H. Hashimoto, and F. Harashima, "Path searching for mobile
robot by local planning", Journal ofRobotics Society ofJapan , vol. 7, no.
4, pp. 267-274 , Aug. 1989 (in Japanese).
[5] G. Yasuda, "Distributed Implementation of Communicating Process
Architectures for Autonomous Mobile Robots", IFAC Distributed
Computer Control Systems 1997, Pergamon, pp. 67-72,1997.
[6] R. Siegwart and I. Nourbakhsh, Introduction to Autonomous Mobile
Robots, MIT Press, Cambridge, MA, 2004.
[7] K. I lan, and M. VcIoso, " Reactive visual control of multiple
non-holonomic robotic agents", Proceedings ofICRA '98, 1998.
[8] Y. Kanayama , Y. Kimura, F. Miyazaki and T. Noguchi, "A stable tracking
control method for a non-holonomic mobile robot", Proceedings of
IROS '91, pp. 1236-1241, 1991.

(a)

(b)
Fig. 14 Example of movement trajectory in the face of the
comer of an obstacle: (a) Hough transform , (b) real image.

1088