A REPORT ON

ROBOT MANEUVERING USING VISION

BASED OBSTACLE DETECTION AND
ULTRASONIC BASED COLLISION AVOIDANCE
SIR PADAMPAT SINGHANIA UNIVERSITY
Submitted By:
Saurabh Kumar
Enrolment No 09EC000630
(saurabhkumar.spsu@gmail.com)
ABSTRACT
The congestion on roads is increasing day by day due to rapid increase in the number of vehicles. The
congestion coupled with drivers inherent limitations has resulted in fatal road accidents during bygone years.
In 1985 a report based on British and American crash data found drivers error, intoxication and other human
factors contributing wholly or partly to about 93% of crashes [1].
A rigorous amount of research is being carried around the globe to look into the issues. For example the
ASH program in the United States by National Automated Highway System consortium, Integrated Multimodal
Transport System in Japan by Toyota, La Route Automatisee by French government [10] etc. have been done
to make roads safer. The key idea that came out of the research is the concept of Intelligent Transport Systems
(IST).
Studies on Intelligent Transport Systems suggest that in order to reduce the number of accidents some mea-
sures may be adopted for example: prohibition for driving by alcoholic driver, improved road designs, regular
vehicle maintenance, array of interconnected sensor etc. may help in checking the number of accidents.
But all these solutions have the human as a driver, which is always the main cause of accidents. So if a
system is afxed with vehicle which assist the driver during driving may reduce the chances of accidents. To
reach such a target the rst stage is to make a robot which can maneuver itself.
To make a robot guide itself without assistance from a human being always a challenging task. An au-
tonomous robot, also known as robotic or informally as driver less or self-driving, is an autonomous robot
capable of fullling the human transportation capabilities of a traditional car. As an autonomous robot, it is ca-
pable of sensing its environment and navigating on its own [2]. To maneuver autonomously a robot has to sense
the world with techniques such as ultrasonic ranging, computer vision etc. So that advanced control systems
interpret the information to identify appropriate navigation paths, as well as obstacles and relevant signage.
2
Acknowledgments
I would like to thank Mr. Phoolpreet for his valuable guidance and Dr. A.K Gupta for his suggestion. Also I
would like to give thanks to my brother Rahul Kothari who helped me in making the L
A
T
E
X document for this
report.
3
Contents
1 Introduction 6
2 Optical Flow 8
2.1 Estimation of Optical ow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
3 Time To Contact (T
c
) 11
4 Collision Detection and Compute Steering 13
4.1 Object Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
4.1.1 Static Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
4.1.2 Moving Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
4.2 Compute Steering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
5 EXPERIMENTAL RESULTS 16
5.1 System Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
5.2 Result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
5.3 Comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
6 FUTURE WORK 18
7 Conclusion 18
4
List of Figure
Fig 1 Over-View of system.........................................................................................................................7
Fig 4.1.2: Sketch for Temporal Derivative.................................................................................................14
Fig 4.2 Body Control Automation.............................................................................................................15
Figures 5.2.1 to 5.2.7 (Test Result)............................................................................................................16-17
5
1 Introduction
Computer vision for ITS (Intelligent Transport Systems) is an active research area [8]. One of the key issues of
ITS is the ability to avoid obstacles, another well-researched problem [9]. In the vision-based navigation work
reported in the past, obstacle avoidance is carried out using ultrasonic sensors. These sensors take over the
control of the robot as long as obstacles are detected in the vicinity. The control is then handed back to vision-
based processing once the obstacles are no longer a factor. While it is expedient to use ultrasonic sensors in such
a manner, we believe it is empirically more interesting to use vision-based processing for obstacle avoidance
also.
Since the primary focus of machine intelligence and advanced robotics is to capture human faculties in a
computer and since humans use vision for obstacle avoidance while navigating, we are evidently interested in
doing the same in a robot. This requires a method to perceive them.
In this article we address the problem of obstacle sensing, obstacle avoidance through their motion (optical
ow) in an image sequence and by use of ultrasonic sensor. Since the perceived motion can be caused either
by the obstacle itself or by the motion of the camera (which is the motion of the robot in the case of a camera
xed on it) therefore vision data is used to detect the obstacle, as the image motion provides powerful cues for
under-standing scene structure.
Divergence of image ow (the sum of image ow derivatives in two perpendicular directions) is theoretically
not affected by camera rotation, so it gives a measure of scene structure for a moving observer and it can be
used by a moving observer to navigate about an environment.
The robot system described here uses ow divergence to steer around obstacles while it attempts to achieve
a goal (which for now is simply to drive straight ahead). When the obstacle avoidance is insufcient to avoid
collision, the ultrasonic data warn the robot of the impending collision. The robot stops, turns, and resumes
wandering straight ahead in the new direction. Robots uses time-to-contact to estimates (derived from optical
ow divergence) to detect imminent head-on collision. This is combined with an optical ow and ultrasonic
data to steer the robot down. That is, the robot attempts to move straight ahead, but adjusts its heading to
maintain clearance on either of its sides.
When the robot drive itself the central time-to-contact (T
c
) predictions warn the robot of the impending
collision. The robot stops, turns, and resumes wandering, following a new path. In above approach real-time
intelligent behavior by using minimalist visually derived representations.
The representations contain only task-relevant information (i.e., relevant to obstacle avoidance) and the in-
formation is represented in two-dimensional (2-D) image coordinates only. The control algorithms directly use
observable image information represented in the 2-D image sequence; a three-dimensional (3-D) reconstruction
is not required [3]. Similarly, 2-D interpretations based on environmental assumptions have enabled impressive
collision avoidance capabilities with other robot.
6
Compute Optical
Flow
Update and Predict
Tc
Compute Steering &
Collision Detection
Camera Input Robot
Fig 1 Over-View of system
In the rest of this paper, in Section II we briey explain the method used for computing optical ow. The
method used for time to contact estimation is described in Section III . The system collision detection and
steering logic is explained in Section IV and the experimental results obtained are presented in Section V.
Finally the conclusions and the discussions are given in Section VII.
7
2 Optical Flow
Optical ow (or optic ow) is the pattern of apparent motion of objects, surfaces, and edges in a visual scene
caused by the relative motion between an observer (an eye or a camera) and the scene [4].
2.1 Estimation of Optical ow
The optical ow methods try to calculate the motion between two image frames which are taken at times t and
t +t at every pixel position.
For a 2D dimensional image at time t +t a pixel at location (x, y, t) with intensity I(x, y, t) would have
moved by (x, y, t) between the two image frames, and the following image constraint equation can be
given:
I(x, y, t) = I(x +x, y +y, t +t). (1)
Assuming the movement to be small, the image constraint at I(x, y, t) with Taylor series can be developed
to get:
I(x +x, y +y, t +t) = I(x, y, t) +
I
x
x +
I
y
y +
I
t
t. (2)
This is only possible when
I
x
x +
I
y
y +
I
t
t = 0. (3)
Or in other words (after dividing throughout by t and taking t 0)
I
x
V
x
+
I
y
V
y
+
I
t
= 0. (4)
Here V
x
, V
y
are the x and y components of the velocity or optical ow of I(x, y, t) and
I
x
,
I
y
and
I
t
are the
derivatives of the image at I(x, y, t) in the corresponding directions. In short form equation(4) can be written
as
I
x
V
x
+I
y
V
y
=I
t
, (5)
which is same as
I V =I
t
, (6)
where is the gradient operator. This is an equation in two unknowns and cannot be solved as such. This is
known as the aaperture problem aof the optical ow algorithms. To nd the optical ow another set of equations
is needed, given by some additional constraint. All optical ow methods introduce additional conditions for
estimating the actual ow. Horn Schunck method [5] can be used for determining V
x
, V
y
. To nd the optical
ow another set of equation is needed, given by some additional constraint. All optical ow methods introduce
additional conditions for estimating the actual ow.Thus to determine the two perpendicular components of
8
optical ow some additional constrains have to be introduced [5]. Commonly opaque objects of nite size
undergoing rigid motion or deformation. So in this case nearby points on the objects also have similar velocities
or motion and discontinuity in the ow pattern can be expected where one object occlude other.
Minimum square method of the gradient of the optical ow is another way to express the additional constrain
_
V
x
x
_
2
+
_
V
y
y
_
2
+
_
V
y
x
_
2
+
_
V
y
y
_
2
(7)
Another measure of the smoothness of the optical ow eld is the sum of the square of the Laplacians of the
x and y components of the ow.
The ow is formulated as a global energy functional which is then sought to be minimized. This function is
given for two-dimensional image streams as
E =
_ _
_
[I
x
v
x
+I
y
v
y
+I
t
]
2
+
2
_
||v
x
||
2
+||v
y
||
2
_
_
dxdy (8)
where I
x
, I
y
and I
t
are the derivatives of the image intensity values along the x, y and time dimensions
respectively.V
x
,V
y
are the two perpendicular component of optical ow vector, and the parameter is a regu-
larization constant. Larger values of lead to a smoother ow.
This functional can be minimized by solving the associated EulerLagrange equations,giving
I
x
(I
x
v
x
+I
y
v
y
+I
t
)
2
_

2
x
2
+

2
y
2
_
v
x
= 0 (9)
I
y
(I
x
v
x
+I
y
v
y
+I
t
)
2
_

2
x
2
+

2
y
2
_
v
y
= 0 (10)
where subscripts again denote partial differentiation and =

2
x
2
+

2
y
2
denotes the Laplace operator.
In practice the Laplacian is approximated numerically using nite differences, and may be written V
x
(x, y) =

V
x
(x, y) V
x
(x, y) where

V
x
(x, y) is a weighted average of calculated in a neighborhood around the pixel at lo-
cation (x,y). Using this notation the above equation system may be written
_
I
2
x
+
2
_
v
x
+I
x
I
y
v
y
=
2
v
x
I
x
I
t
(11)
I
x
I
y
v
x
+
_
I
2
y
+
2
_
v
y
=
2
u
y
I
y
I
t
(12)
which is linear in V
x
and V
y
and may be solved for each pixel in the image. However, since the solution
depends on the neighboring values of the ow eld, it must be repeated once the neighbors have been updated.
The following iterative scheme is derived:
v
k+1
x
= v
k
x

I
x
_
I
x
v
k
x
+I
y
v
k
y
+I
t
_

2
+I
2
x
+I
2
y
(13)
v
k+1
y
= v
k
y

I
y
_
I
x
v
k
x
+I
y
v
k
y
+I
t
_

2
+I
2
x
+I
2
y
(14)
9
where the superscript k+1 denotes the next iteration, which is to be calculated and k is the last calculated
result. This is in essence the Jacobi method applied to the large, sparse system arising when solving for all
pixels simultaneously.
10
3 Time To Contact (T
c
)
Let the translation velocity for the environments W= (W
x
, W
y
, W
z
) and the angular velocity = (
x
,
y
,
z
)
then the moment R is dene as

R =W R (15)
In whatever follows r = (x, y) is the image world coordinate while R = (X, Y, Z) is the real world coordi-
nates.
In other words the above equation can be written as
_
_
_

Z
_
_
_=
_
_
_
W
x
W
y
W
z
_
_
_
_
_
_

y
Z
z
Y

z
X
x
Z

x
Y
y
X
_
_
_ (16)
Assume image plane lies at f = 1, then x =
X
Z
and y =
Y
Z
, after differentiation one gets
x =

XZ

ZX
Z
2
(17)
y =

YZ

ZY
Z
2
(18)
Substitute

X,

Y, &,

Z in Eq-16 above one gets the following equations for the x and y component of the
velocity.
_
V
x
V
y
_
=
_

X

Y
_
=
1
Z
_
1 0 x
0 1 y
_
_

_
W
x
W
y
W
z
_

_+
_
xy
_
1+x
2
_
y
1+y
2
xy x
_
_

z
_

_ (19)
Divergence of the ow eld is computed to estimate time to contact (T
c
) [6], [7] Thus the equations for the
components of optical ow due to general camera motion (arbitrary translation and rotation) in a stationary
environment can be written as:
V
x
=
1
Z
(T
x
+xT
z
) +
_
xy
x

_
1+x
2
_

y
+y
z

(20)
V
y
=
1
Z
(T
y
+yT
z
) +
__
1+y
2
_

x
xy
x
+x
z

, (21)
where Z is the depth of the object in the environment relative to the camera, and (T
x
, T
y
, T
z
) and (
x
,
y
,
z
)
are the translational and rotational motion of the environment relative to the camera. The divergence of optical
ow eld [parametrized by image coordinates (x, y)] is dened by
(V
x
, V
y
) =
V
x
x
+
V
y
y
. (22)
Also
11
V
x
x
=

x
(T
x
+xT
z
) +T
z
y
x
2x
y
(23)
V
y
y
=

x
(T
y
+yT
z
) +T
z
2y
x
x
y
, (24)
where = 1/Z, at (x, y) = (0, 0). So that
(V
x
, V
y
) =

x
(T
x
) +2T
z
+

x
(T
y
), (25)
or
(V
x
, V
y
) = 2T
z
. (26)
The gradient of the imaged surface is perpendicular to the transverse velocity. So time to contact is:
T
c
=
Z
T
z
=
2
(V
x
, V
y
)
(27)
12
4 Collision Detection and Compute Steering
4.1 Object Detection
To detect object rst the motion information is extracted from the sequences using temporal derivatives. For
camera, sequential images (video frames), 1 , 5 ,9 and 13 , are digitized and stored and the temporal derivative
is computed as
In order to detect the objects the vision data alone is not adequate as when the image difference is taken(for
optical ow computation, as shown in gure ) in to account then all the slow moving objects having very small
amount of optical ow values so to nd these objects ultrasonic sensor is used.Thus to detect the objects both
vision and ultrasonic data are used.
The main task is to avoid obstacles while achieving the mobility goals. In order to detect obstacle robot
divided obstacle in two parts
1. Static Objects
2. Moving Objects
4.1.1 Static Objects
To detect static objects, a pair of Ultrasonic sensor is used i.e. ultrasonic sensor continuously emitting ultrasonic
waves around 40 KHz, and when these ultrasonic pulse beam strike on an object it reect back to ultrasonic
sensor .By calculating the to and fro time (for the ultrasonic pulse) , obstacle can be detected along with the
information of time to contact (T
c
) .
4.1.2 Moving Objects
To nd out the moving objects mainly the vision data is used. To do so rst step is to capture four frame[F1,
F5, F9, F13] from the given video stream (g 4.1.2) after that Temporal Derivative I
0
and I
1
is calculated as
I
0
= F
5
F
4
(28)
I
1
= F
10
F
7
(29)
Then the next step is to threshold the optical ow values, thresholding is done to remove the unwanted
objects also as in most scenarios, the vast majority of the scenes will be stationary with a small percentage of
the objects in motion. So, Simply compute a grid of motion vectors across the scene the moving object can be
found out by thresholding the optical ow values.
13
- -
Optical flow
Computation
Thresholding
F1 F2 F3 F4 F5 F6 F7 F8 F9 F10
Input Video
Stream
Camera
Optical Flow
Component (Vx, Vy)
Fig 4.1.2: Sketch for Temporal Derivative
4.2 Compute Steering
Ideally, the robot uses the vision data to identify the direction in which it has to travel. The robots behavioral
goal is simply to move forward, steering away from obstacle. The robot steers smoothly to the new desired
heading with saturated visual feedback controls. The desired heading attempts to avoid obstacle is made by
choosing a desired heading angle
q
.
To nd out the desire heading a simple look table is used.

q
= k, (30)
where is calculate by the aid of look table and k is a constant which can be calibrated experimentally. Thus
when there is obstacle then robot turn at
q
and travel a distance d (where the distance is directly proportional
to time to contact). After traveling a distance d robot turn again at angle -
q
and then continue their path. This
process is shown in below state diagram.
14
Initialization
Optical Flow
Values
Tc
Obstacles
Heading Angle
Continue
Current Path
Velocity
Reduction
Distance
calculation
Turn
Travel Given
Distance
Reduce
Velocity
Stop
Tc<threshold Tc>=threshold
Fig 4.2 Body Control Automation
15
5 EXPERIMENTAL RESULTS
5.1 System Description
In our experiment, we use a custom made mobile robot which provide odometric measurements and digital
video stream(a web cam is xed on the front of the robot).We need to synchronized odometry data and video
stream (otherwise the detection will be incorrect). To give an overview of our experimental results(for object
detection), we will use a video where cars are moving on an express highway.
5.2 Result
Figures 5.2.1 to 5.2.7 show the video sequence on which we superimposed the contour of the detected objects,
respectively. The horizontal line in the image corresponds to the horizon.
Below some experimental results are displayed for obstacle detection. Here the frames were captured at the
rate of 15 frames per second.
Here the bounding box shows the near by obstacles
Fig 5.2.1:
Fig 5.2.2:
Fig 5.2.3:
Fig 5.2.4:
16
Fig 5.2.5: Fig 5.2.6:
Fig 5.2.7:
5.3 Comments
The model that we described here is not tested completely as a whole system mean that the different modules of
robot like obstacle detection (using vision data), obstacle avoidance (using ultrasonic data) are not synchronize
completely in that robot but as an independent system these module(obstacle detection, obstacle avoidance)
gives good result (as shown above).
17
6 FUTURE WORK
In future work, our main priority is synchronize the vision data with the ultrasonic data so that robot can
maneuver for a long time, also we will explore new models of motion estimation. Improvements can be made
in motion estimation with the use of optical remote sensing technology (i.e use LIDAR sensor for 3D imaging)
. We are currently researching this aspect and we expect to submit these results in future articles.
7 Conclusion
As with all such systems dealing with higher-level robotic intelligence, the performance can never be expected
to be completely foolproof. The best that one can do is to devise appropriate automatic error correction and
detection strategies. To briey discuss the various failure modes of our system, the vision-based collision
avoidance capability depends obviously on the visual contrast between the obstacles nearby things and thus
whenever this is not obstacle is not detected properly.
We proposed here a way of detecting obstacles in a mobile robot environment by motion estimation estima-
tion from an image sequence. The originality in this method is that we detect obstacle based on the motion in
the image sequences.
Firstly, we extract a optical ow of the environment and then we separate the moving obstacles and static
obstacle from the environment by trying to t the optical ow model to the observed video stream.
18
References
[1] Interactive Highway Safety Design Model: Accident Predictive Module by H. Lum and J.A Reagan
http://www.fhwa.dot.gov/publications/publicroads/95winter/p95wi14.cfm
[2] What were driving at by Sebastian Thrun, Software Engineer.
http://googleblog.blogspot.com.au/2010/10/what-were-driving-at.html
[3] D. Raviv and M. Herman, Visual Servoing from 2-D Image Cues, In Active Perception, Y. Aloimonos,
ed., Lawrence Erlbaum Associates, Hillsdale, NJ, pp. 191-226, 1993.
[4] Practical Image Processing and video Processing Using Matlab by OGE Marques IEEE Press Chapter 22
page no.562.
[5] Determining Optical Flow Berthold K.P Horn and Brian G. Schunck Articial Intelligence Laboratory,
Massachusetts Institute Of Technology, M A 02139, USA.
[6] Real-Time Obstacle Avoidance Using Central Flow Divergence, and Peripheral Flow IEEE TRANSAC-
TIONS ON ROBOTICS AND AUTOMATION, VOL. 14, NO. 1, FEBRUARY1998.
[7] G. S. Young, T. H. Hong, M. Herman, and J. C. S. Yang, New visual invariants for obstacle detection
using optical ow induced from general motion, in Proc. IEEE Workshop Appl. Computer. Vision, Palm
Springs, CA, Nov. 30Dec. 2, 1992, pp. 100109.
[8] E. Dicksmanns, The development of machine vision for road vehicles in the last decade, 2002
[9] J. Borenstein and Y. Koren, Histogramic in-motion mapping for mobile robot obstacle avoidance, IEEE
Trans. Robot. Automat., vol. 7, pp. 535539, Aug. 1991.
[10] Automated Highway Systems Chapter 2 http://scholar.lib.vt.edu/theses/available/etd-
5414132139711101/unrestricted/ch2.pdf
19