2010 International Coriference on System Science and Engineering
Vision-Based Side Vehicle Detection from a
Moving Vehicle
Wen-Chung Chang
Department ofElectrical Engineering
National Taipei University of Technology
Taipei, Taiwan, R.o.C
e-mail: wchang@ee.ntut.edu.tw
Abstract-In this paper, a side vehicle detection system with
real-time on-board vision is proposed. The system employs camera
vision to detect side moving vehicles, and provides necessary
warning to drivers and passengers when the vehicle is making lane
change. Based on SURF, feature detection and comparison are
applied to find the same features between two consecutive images
and calculate the feature vectors. Further image analysis is
performed based on the feature vectors to determine whether there
are side moving vehicles. If any side moving vehicle is detected, the
system will apply inverse perspective projection technique
estimate the positions of surrounding vehicles in Cartesian space.
Then, the system will generate on-screen signs of safety, warning,
or danger to inform drivers and passengers based on the estimated
distance to side moving vehicles. The approach has been
successfully validated in real traffic environments by performing
experiments with two CCD cameras mounted on top of side mirrors
of a roadway moving vehicle.
Keywords-Computer vision, Intelligent transportation systems,
Real-time vision, Side vehicle detection.
I. INTRODUCTION
In recent years, many car manufacturers have invested lots
of manpower and resources on developing Intelligent
Transportation Systems (ITS) such as auto navigation, traffic
flow surveillance and driver-assist system to improve the safety
and efficiency of transportation. As for the driver-assist system,
it covers the lane departure warning system, the front vehicle
collision warning system, and the auxiliary night vision system.
In reality, when drivers and passengers are opening the doors to
get off the vehicle, the only available resource for making
judgment is the side mirrors. But the side vision blind spot and
human negligence are usually the causes of many collisions
and car accidents. To reduce the occurrence of car accidents,
this paper focuses on side vehicle detection when the vehicle is
making lane change. A number of researchers have
demonstrated side vehicle detection that uses laser, radar or
ultrasonic as primary detection sensor. However, laser sensors
and radar sensors are too expensive and ultrasonic sensors are
easily interfered with other ultrasonic sources. Therefore,
camera vision is apparently a popular sensor for vehicle
detection. Many papers use different sensors to detect side
vehicle and the goal is to provide information of surrounding
vehicles to ensure safe driving. Song et al. [1] used multiple
ultrasonic sensors installed in the vehicle side to detect side
vehicles. Wender and Dietmayer [2] performed sensor fusion
of a laser scanner and a camera vision sensor. The laser
978-1-4244-6474-61101$26.00 © 2010 IEEE
Authorized licensed use limited to: UNIVERSITY OF SHARJAH. Downloaded on November 24,2020 at 04:34:35 UTC from IEEE Xplore. Restrictions apply.
Kuo-Jung Hsu
Department ofElectrical Engineering
National Taipei University of Technology
Taipei, Taiwan, R.o.C
e-mail: kuojung.hsu66@gmail.com
scanner estimates the distance as well as the contour
information of observed objects. In addition, video-based
vehicle detection utilizes pattern recognition algorithm to
classifY the object's sides based on contour information. Kato
et al. [3] also combined different sensors to detect vehicles, and
used camera vision to detect lane marking, then the vehicle
would utilize radar sensor to estimate the distance of vehicle.
Kim and Chon [4] used pure camera vision to detect vehicles.
To perform comer detection and find the matched feature
to
between two consecutive images, one needs to analyze the
feature vector to locate moving vehicles. Chang and Cho [5]
presented a real-time vision-based vehicle detection system
employing an online boosting algorithm. In [6], a system
employing both parts-based detection and decision tree ideas
was proposed to detect side vehicles. Miyoshi et al. [7]
presented a temporal median filter to detect the latest
background images. The technique was applied to the video
image obtained from a hand-held camera and detect the moving
objects.
In this paper, a method is proposed for detecting side
moving vehicles, providing distance information and warning
figure to drivers and passengers in a moving vehicle. This
paper is organized as follows. Section II describes the overall
system architecture. Section III illustrates the distance
estimation by inverse perspective projection technique. The
side vehicle detection technique is presented in Section IV.
Section V shows the experimental results of the side vehicle
detection. Final concluding remarks are shown in Section VI.
II. SYSTEM ARCHITECTURE
For the description of side vehicle detection architecture,
one first needs to understand the blind spot area. There is a
side vision blind spot in the vehicle side. If there is a side
moving vehicle in the blind spot area at the same time when the
driver or passenger is opening the doors to get off the vehicle
or is making lane change, a collision or car accident may occur
as illustrated in Fig. 1. The arrowhead shows the direction of
traffic and the green area can be viewed by the both side
mirrors, and the yellow area is the blind spot area. Therefore,
one cannot see it by both side mirrors.
In this paper, a side vehicle detection approach is proposed
with two CCD cameras mounted onto a roadway vehicle as
shown in Fig. 2. The CCD cameras face the rear of the vehicle.
In this experimental system, the height of the camera is 1.2
- 553 - ICSSE 2010
 2010 International Coriference on System Science and Engineering

meters and the pan angles, (J\ and (J2' between the camera and
the vehicle are both 15 degrees.
Fig. 4 shows a flow chart of the proposed side vehicle
detection process. There are three major stages constituting
the process: detecting features, comparing features, and
calculating feature vectors. First stage is to detect SURF
features from previous and current images. Second stage
refers first stage result to relate current features from previous
image. Third stage will calculate the feature vectors based on
further image analysis to determine whether there are side
moving vehicles. Finally, the side moving vehicle region in
the image is used to estimate the distance between our
experimental vehicle and side moving vehicle based on
inverse perspective projection technique.

CCD camera
r(- '\ CCD camera

� 4
�.
,/
;

�"'.

Fig. 4. Flow chart for the proposed side vehicle detection process.
(a) Bird's eye view of the vehicle.

III. VEHICLE DISTANCE ESTIMATION

In this section, we will introduce the method for distance

(b) Lateral view of the vehicle.
Fig. 2. CCO cameras are mounted on the side mirrors of a vehicle.
The captured images are shown in Fig. 3, where the side
moving objects includes a vehicle and a motorcycle.
estimation. For information about the distance estimation
method, please refer to [8]. It utilizes inverse perspective
projection technique to estimate the distance between our
experimental vehicle and side moving vehicle. When the
distance is calculated, we can show distance information on
monitor and show warning figure based on different distances.
A. CCD model and real world model
First, one needs to establish coordinate systems for the
CCD camera and the world space, respectively, as shown in Fig.
5. The origin of the CCD camera coordinate system
(Xe, Ye,Ze ) is located at the optical center of the camera with
optical axis along the Ze axis. The orientation of the world
coordinate frame is chosen as the orientation of the CCD
camera frame after rotating about Xe axis for 90 degrees. As
with the origin Ow of the world coordinate system, attached to
the moving vehicle, is at height h below the optical center
along the Ye axis.

(a) A side vehicle (b) A side motorcycle

Fig. 3. Images captured by the left CCO camera.

- 554 - IeSSE 2010

Authorized licensed use limited to: UNIVERSITY OF SHARJAH. Downloaded on November 24,2020 at 04:34:35 UTC from IEEE Xplore. Restrictions apply.
 2010 International Coriference on System Science and Engineering

(2)
i=O j=O

where I� (x, y) is defined as the sum of the rectangular region

h from the top left comer to the bottom right point (x, y). One
can calculate I� (x,y) of every point's location, and then use

. Jr. . . . . . . . . .
the results to calculate the feature points. As shown in Fig 6,
if one wants to calculate the sum of the values inside a
rectangular area, only the four points (a,h,c,d) are needed
..----.....
.. yw
in addition to the results from previous calculation. Thus,
L= A-B -C+ D (3)
Xw
Fig. 5. The CCO camera and world coordinate systems.
where A, B, C, D are the integral image for the four points a, b,
c, d, respectively, and L is the sum of the values inside the
B. Estimation of distance to side vehicles rectangular area.
To detect side moving vehicles in the images based on our o

side vehicle detection system, we need to refer the position of

the bottom of the side moving vehicle in the image to
determine the estimated distance between the side moving
vehicles and our experimental vehicle. Based on the coordinate
system and perspective projection model of the camera, the
distance can be determined by the following equations. d b

f ·h
Yw =f . x.-f tanB
Yi cosB-Yi smB----'-
' ---=--­
-
X tanB+ f (1) c a
i
X -f tanB
Xw =Yw i Fig. 6. Illustration of integral image.
X tanB+ f
i
B. Road detection

where f is the focal length of the CCD camera, (X ' Y ) is the In order to reduce the computation load of system, one
i i only needs to do detection computation for the region where
coordinate of the side vehicle in the image plane.
the vehicle is driving through. Hence, one needs to outline the
road range, and exclude unnecessary non-road background
image on both sides for the purpose of reducing the
IV. SIDE VEHICLE DETECTION
computation load of system. As illustrated in Fig. 7, one can
Lateral vehicle image is captured by image detector, and the obtain the desired picture by using edge detection, and make
range between the roadway edge and the vehicle's side is the use of the results to get road edge line (green straight line), and
processing area. The SURF algorithm is employed to capture then use RANSAC [27] method to find out the nearest linear
the feature points. One can find out the matched feature point equation that matches those points.
pair by comparing two successive images, and then use the
matched feature point pair to calculate the feature vector.
Based on the feature vector, one can judge if there is a vehicle
in the side area. The Inverse Perspective Projection is used to
calculate detected information, and then the coordinates on the
image can be transferred to the real world coordinates relative
to the vehicle. Based on the determined distance, different
warning figures are shown on screen for drivers.
A. Integral image
The advantage of using integral image is that it can
accelerate the calculation. The expression of integral image is
as follows.

Fig.7. Edge detection on a lateral image

- 555 - IeSSE 2010

Authorized licensed use limited to: UNIVERSITY OF SHARJAH. Downloaded on November 24,2020 at 04:34:35 UTC from IEEE Xplore. Restrictions apply.
 2010 International Coriference on System Science and Engineering
The outlines of RANSAC to yield a linear equation is as
follows.
1) Taking out the points by random from all points as needed
for calculating linear equation.
2) Calculating a linear equation.
3) Setting up a threshold value for possible errors.
4) Putting all the other points into the linear equation to
calculate separate error value.
5) If error value is smaller than threshold value, then one
should be added into the count.
Repeat the steps above to find out the linear equation that
have most counts, i.e., the linear equation that best match all
the points.
C. SURF
[ ]
SURF uses Hessian Matrix H(x,y) to calculate feature
points.
Lxx (x, y) Lxy(x, y)
H(x, y) Lxy(x, y) Lyy(x, y)
= (9)
where Lxx (x, y) is convolution of the Gaussian second order
in the image that includes (x, y). The mask in Fig. 8(a) is
used while calculating Lyy(x, y) and the mask in Fig. 8(b) is
used while calculating Lxx (x, y), and the mask in Fig. 8(c) is
used for calculating Lxy(x, y).
The feature points location are select by relying on the
determinant of the Hessian. The local maxima of the
determinant are determined to find out the locations.
1 1
1 1
-I
1 "Lldx
(a) yy (b) xx (c) xy
Fig. 8. Similar Gaussian second orders
D. Compare Feature Points
One needs to compare the feature points of two images in
order to determine which feature points on one image match to
the other image. SURF comparison method is employed to
locate the matched feature points. First, find out the feature
points on the first image, and based on which match
calculation is performed against another image. The
description table is built based on SURF and the information
about feature points' surroundings. Considering the
calculation speed, only 4 x 4 dimension is used to produce
descriptive values of the feature points. The values in
description table are produced by calculation through Haar
wavelet filter. Please refer Fig. 9 for more information about
Authorized licensed use limited to: UNIVERSITY OF SHARJAH. Downloaded on November 24,2020 at 04:34:35 UTC from IEEE Xplore. Restrictions apply.
the calculation method. Each feature point is surrounded by
four areas, which are upper right, bottom right, upper left and
bottom left, respectively. The size of each area is a 5x5 square.
First, we have to do calculation for one of the four areas and
we will get several singular values of dx and dy, and se�eral
Idxl and Idyl.
Secondly, summing up all of these values will yield , I dx
Idy, Ildxl, IIdyl , respectively. Lastly, combining
these values will constitute a four dimensional vector
(Idx, Idy, Ildxl, Ildyl)·
From Fig. 9 we can see that SURF quantifies these four
summed values. The element with value larger than a certain
threshold is set to be one, whereas the element with value
smaller than a certain threshold is set to be zero. Thus, the
resulting vector is a four dimensional one with values one and
zero. Fig. 10 illustrates the SURF descriptor for three
different image-intensity patterns.
m dx
Fig. 9. Statistic Haar wavelet filter in dx and dy response
DUD
L b�
"Ldx
1 "Ldy
"Lldy
Fig. 10. Feature description table constituted
E. Detect the side vehicles
To detect the sideway moving vehicles, one needs to
understand the sideway static background and the movement
model of the sideway moving objects. From Fig. 1 1, we can
find that when watching the rear area from the moving vehicle,
the relative moving direction of the static background object is
backward, and the object on the image will gradually become
smaller and finally disappear from the center of the screen. As
for the relative moving direction of sideway moving vehicle,
its speed verifies according to the speed of the experimental
vehicle. When the sideway moving vehicle (Vs) is slower than
the experimental vehicle (Vo), the moving direction of the
former is also backward. When the speed of sideway moving
- 556 - IeSSE 2010
 2010 International Coriference on System Science and Engineering

vehicle (Vs) is the same as the experimental vehicle (Vo), the

relative movement is static. When the sideway moving
vehicle (Vs) is faster than the experimental vehicle (Vo), the
relative moving direction is forward. Based on this
characteristic, we can find out the forward feature vector, and
regard it as moving vehicles.

Fig. 13. Locate matched moving pixels.

Fig. 14. Detect moving pixels of side vehicles.

Fig. II. Schematic figure of establishing background.

V. EXPERIMENTAL RESULTS
Safety Warning Danger
The proposed method has been effectively implemented
using two CCD cameras mounted onto a roadway vehicle.
Distance
The captured images from side CCD cameras are transferred
to the computer in 320x240 BMP format. Fig. 12 illustrates a
sequence of three images that are captured when the
experimental vehicle is moving on the road.

Fig. 12. Illustration of the sequence images.

The feature points were obtained from two successive
images, and we compared the feature points to find out the
feature point pair shown in Fig. 13. Then, we calculated the
feature vector based on the location of the feature point pair,
and marked the forward direction of feature vector, which is
shown in Fig. 14. After that, we calculate the distance
between the sideway moving vehicle and the experimental
vehicle. Meanwhile, different warning signs as shown in Fig.
15 were given based on different length of the distance for
drivers to know if it is safe to change the traffic lane, and in
this way to ensure driving safety.
Fig. IS. The different warning figures shown on screen.
Table 1 and Fig. 16 show the experimental results that are
generated from experimental platform directly. It provides the
distance information and warning figures on the monitor. The
detection rate in the table denotes the ratio the system detects
the existence of a moving vehicle on the sideway in the image,
while false alarm ratio denotes the system incorrectly detects
that there is a moving vehicle on the sideway and causes a
false alarm.
Percentage Total frame Detection frame

Detection rate 83.93% 56 frame 47 frame

False alarm ratio 9.88% 81 frame 8 frame

Table I. The expenmental results.

- 557 - IeSSE 2010

Authorized licensed use limited to: UNIVERSITY OF SHARJAH. Downloaded on November 24,2020 at 04:34:35 UTC from IEEE Xplore. Restrictions apply.
 2010 International Coriference on System Science and Engineering
Fig. 16. The experimental platform for side vehicle detection.
VI. CONCLUSION
In this paper, a method of side vehicle detection is
presented which has been effectively implemented in a
roadway environment for detecting side moving vehicles. A
vision-based approach is proposed for detecting side moving
vehicles even in the blind spot area. The detection system can
be operated at real-time rate to provide drivers and passengers
with surrounding traffic information. Any moving side
vehicles, including motorcycles, can be effectively detected.
- 558 -
Authorized licensed use limited to: UNIVERSITY OF SHARJAH. Downloaded on November 24,2020 at 04:34:35 UTC from IEEE Xplore. Restrictions apply.
Therefore, the proposed system is capable of ensuring safety
driving and reducing accidents caused by lateral collision.
ACKNOWLEDGMENT
This research was supported by the National Science
Council of Taiwan, R.O.C. under grant NSC 98-2221-E-027-
065-MY3.
REFERENCES
[I ] K.-T. Song, C.-H. Chen and C.-H. Chiu Huang, "Design and
Experimental Study of an Ultrasonic Sensor System for Lateral Collision
Avoidance at Low Speeds," Proceedings of the 2004 IEEE Intelligent
Vehicles Symposium, Parma, italy,pp. 647-652, Jun. 2004.
[2 ] S. Wender and K. Dietmayer, "3D vehicle detection using a laser
scanner and a video camera," Congress of lET Intelligent Transport
Systems, Vol. 2,No. 2,pp. 105-112,Aug. 2008.
[3] T. Kato, Y. Ninomiya and I. Masaki, "An Obstacle Detection Method by
Fusion of Radar and Motion Stereo," Conference of IEEE Transactions
on Intelligent Transportation Systems, Vol. 3,No. 3,Sep. 2002.
[4] Z.-W. Kim and T.-E. Cohn, "Pseudoreal-Time Activity Detection for
Railroad Grade-Crossing Safety," Conference of IEEE Transactions on
Intelligent Transportation Systems, Vol. 5,No. 4, Dec. 2004.
[5] W.-C. Chang and C.-W. Cho, "On-Line Boosting for Vehicle
Detection," IEEE Transactions on Systems, Man, and Cybernetics, Part
B: Cybernetics, 40(3), June 2010.
[6] W.-C. Chang and Chih-Wei Cho, "Real-Time Side Vehicle Tracking
Using Parts-Based Boosting," In Proceedings of the 2008 IEEE
International Conference on Systems, Man and Cybernetics, Singapore,
Oct. 12-15 2008.
[7] M. Miyoshi, J.-K. Tan and S. Ishikawa, "Extracting Moving Objects
from a Video by Sequential Background Detection Employing a Local
Correlation Map," International Conference of the 2008 IEEE on
Systems, Man and Cybernetics, 2008.
[8] W.-C. Chang and T.-C. Tsao, "Omni-Directional Lane Detection and
Vehicle Motion Estimation with Multi-Camera Vision," In Proceedings
of the 2009 CACS International Automatic Control Conference, Taipei,
Taiwan,Nov. 27-29,2009.
IeSSE 2010