P m d l n g r d the 2004 lEEE

-
Internatlonal Confennu an Rabotlsr (L Automatron
New Orleans, LA April 2004

An interactive driver assistance system monitoring

the scene in and out of the vehicle
Lars Petersson’, Luke Fletcher+, Nick Barnest, Alexander Zelinskys
National ICT Australia Limited*$ Research School of Information Sciences and Engineering+§
Locked Bag 8001, Canberra ACT 2601. The Australian National University, Canberra ACT 0200.
Email: lars.petersson@nicta.com.au*luke@syseng.anu.edu.aut nick.barnes@nicta.com.autalex@syseng.anu.edu.au§

Abstract-This paper presents a framework for interactive is approaching the cars in front too rapidly, or helshe may try
driver assislance systems including techniques for fast speed to change lane when there is a car in the blindspot.
sign detection and elassilication, car detection and tracking, and How can the system within the car perceive whether the
lane departure warning. In addition, the driver’s actions are
monitored. The integrated system uses information extracted driver is already aware of a particular event? An eye gaze,
from the road scene (speed signs, position within the lane, relative and head pose tracking system is used to detect where the
position to other cars, etc.) together with information about the driver is looking. This way, we can ensure a warning is issued
driver’s slate such as eye gaze and head pose, lo issue adequate if the attention of the driver was directed elsewhere.
warnings. A touch screen monitor presents relevant information
and allows the driver to interact with the system. The research
is focused around robust on-line algorithms. Initial results of on-
line speed sign detection and car tracking are presented in the
context of a driver assistance system.

I. INTRODUCTION
Improved safety is key technological goal in road vehicles
today. One way to achieve this is by creating systems within
the vehicle that support the driver in reacting to changing
road conditions. Our research is focused on driver assistance
systems: systems that assist the driver in controlling the car,
but keep the driver in the loop. Impressive work in this and
related areas has been performed by Thorpe [ I ] [21 [31 ,
Dickmanns [4] [5] [6] and Broggi [7] [E]. Their work deals
mostly with the sensing aspect of driver assistance which is
essential to create robust and reliable systems. An interesting
research area is, however, how to handle the information flow
generated. Depending on the context, information has different
significance, e.g. how are warnings most efficiently conveyed?
In this paper, we present a framework for interactive driver
assistance systems in which we can perform research on
new technologies relevant to vehicle safety. It is centered
around a touch screen monitor mounted next to the driver
with a backbone of computing power and sensordactuators.
We demonstrate techniques for fast speed sign detection and
classification, car detection and tracking, and lane departure
warning. In addition, the driver’s actions are monitored by the
use of a faceLAB system from Seeing Machines [9]. Only
robust on-line algorithms are used.
An important task for a driver assistance system is sign
recognition. Signs giving information relevant to the local
conditions and appear clearly in the environment, however,
a driver may not notice a sign, due to distractions or a lack
of concentration. In this case it is helpful to make them aware
of the information that they have missed. The same is m e for
monitoring the position of other vehicles. Perhaps the driver
Fig. 1. The driver’s seat showing the touch screen monitor, active Stereo
vision head looking out the windscreen and a passive Stem p c nmnitwing
the driver.
Section II discusses driver assistance systems in general,
outlining the purpose and demands of a driver assistance
system. A comparison is made to a human co-pilot to highlight
the important aspects. In Section IU the software framework
is described with a generic model of the system. Section N
shows the method of interfacing to the driver. Preliminary
work on a system for fast visual speed sign recognition is
presented in Section V. Section VI presents a method for
obstacle detection and tracking which is applied to detecting
and tracking other vehicles. A lane tracker is shown in
Section VI-D. The integration of the different subsystems is
presented in Section W, followed by conclusions and future
work.
11. DRIVERASSISTANCESYSTEMS
A Driver Assistance System (DAS) may perform activities
like relieving the driver of distracting routine activities, wam
about upcoming situations and possibly take control of the car
if an accident is imminent. Depending on the task to be per-
formed, a DAS must have appropriate levels of competencies

0-7803-8232-3/04/$17.00 @ZOO4 IEEE 3475

 ~
in a number of areas. If we, consider the DAS to be a human
co-pilot it is easier to pick out the important aspects. To be
of assistance, the CO-pilotneed to be aware of what is going
on outside of the car, e.g., are there any pedestrians in sight.
. where are they going, how is the road turning, etc. Moreover.
we would like our CO-pilotto warn us if we have not noticed
an upcoming situation. That means that not only should the
CO-pilotbe aware of what is going on outside of the car, but
also what is happening inside, i.e., the driver’s responses. In
addition, our CO-pilotmust -know where the vehicle is going,
how fast, if we are braking, accelerating etc. to make good
decisions. Good decisions are a result-of good reasoning. A
successful driver-co-pilot team requires good communication.
The CO-pilot must not be inuusive or present the driver with
too much information. Finally, if the co-pilot notices that the
driver does not respond to a situation that will result in an
accident, he must be.able to take control over the car.
Returning to our non-human co-pilot, the DAS, we can
..
condense the above to the-followinf key competencies:
Traffic situation monitoring
..
Driver’s state monitoring
Vehicle state monitoring
..
Communication with the driver
Vehicle control
Reasoning system
The~firstthree collect information which t h e b A S can use to
analyse the current situation. The fourth, Communication with
the driver, provides both input to the DAS and output to the
driver. E.g., the driver can specify an overall goal, orthe DAS
can give information to the driver. .Vehicle contml is necessary
if it is expected that the DAS should be able to perform any
semi- or fully autonomous maneuvers. A reasoning system
may range from adirect mapping from an input to an output,
to a complex system using the latest advances in artificjal
intelligence. The level of competence in each category is
dependent on the specific task to be solved.
Finally, with a human co-pilot, the DAS should possess the
-
following behavior:
Intuitive. The behavior of the DAS must make immediate
sense in the context of the standard driving task.
Nonintrusive. Must not distract or disrupt the driver
unless it is necessary.
. Overridable. The driver has ultimate control and can
refuse assistance.
III. THE SOFTWARE FRAMEWORK
.’. Figure 2 shows the generic model .of our driver assistive
system. A clienhserver based system architecture has been
designed where each of the principle information sources, such
as the road scene. the driver or the vehicle are represented
by an information sewer. The information servers are generic
across the driver assistance systems, a particular driver assis-
tance system is implemented by a simple “DAS logic” client
that encapsulated the desired ;behaviour of the DAS system.
These “DAS logic” clients effectively perform the role of cross
referencing and auditing the data from the information sources.
~
3476
W
Fig. 2. Sub-cornponenor of the Driver Assistive System.
The clienVserver suucture is implemented using CORBA.
This gives clean well defined interfaces between modules and
distributed computing becomes less complex. Computer vision
algorithms in general tend to consume a lot of computing
power and so it is necessary to run the DAS on several com-
puters. Moreover, developing algorithms for driver assistance
systems requires many diverse skills so a modular approach
is necessary to allow a large team to work on it at the same
time.
A. Driver state
The driver’s. head position .and eye gaze from the Face-
LAB system are combined with the other sensors directed
at the driver to estimate the driver’s state. The driver’s
state is organised on three levels: a time-stamped log of
raw events such as brake pedal depressions; an interface
to inquire.about the driver’s current gaze direction; and a
simple behaviour interpreter. The .behaviour interpreter just
monitors the logged events and the vehicle state to form simple
conclusions such as: driver^ intending to accelerate”, ”Driver
intending stop/slow”, ”Driver intending changing lanes” i n d
”Driver loqking at X (using the Seeing Machines environment
segmentation tool). Also the interface can, be queried for
”time since” inquiries about the driver gaze direction. Here
a direction’andconfidence is given and the engine returns the
time since the driver looked in that direction by searching the
FaceLAB log file.
FaceLAB is a driver monitoring system developed by
Seeingmachines [9] in conjunction with ANU and Volvo
Technological Development. It uses a passive stereo pair of
cameras mounted on the dashboard to capture video images
of the driver’s head. These images are processed in real-
time to determine the 3D pose of the persons face ( f l m m ,
fl deg) as well as the eye gaze direction (f3deg), blink rates
and eye closure. Clinical trials show head position and eye
closure are key indicators for the detection of fatigue in drivers
[IO].When augmented with information about the vehicle and
traffic situation additional inferences can be made. Land and
Lee [ 1 I ] investigated the relevance of gaze direction relative
to road curvature. Apostoloff et. al. [I21 showed a correlation
between eye gaze direction and road curvature, particularly
when monitoring oncoming traffic.
B. Vehicle sfuie
The state of the vehicle is monitored using a 3-axis ac-
celerometer, 3-axis gyro and a GPS. There is also a poten-
tiometer on the steering shaft to measure the current steering
angle. These vehicle state sensors will be combined with an
Ackerman steering model and an extended Kalman filter to
provide a robust estimate of the vehicle motion, between video
image frames.
IV. USERINTERFACE
It is important that information gathered, and decisions
made by the driver assistance system are conveyed to the
driver in an effective manner. Although many warnings and
selections can be made using switches, there is often a need for
more context and richer ways of interaction. The user interface
must not add to the driving task, but rather make it easier by
selecting the appropriate information in an intelligent way. As
mentioned in [13], information display issues that will affect
the system’s safety, usability and acceptance include
. maddiry (auditory, visual, tactile etc)
formai (text, map, tone, voice)
time (stan time, duration, frequency)
We have chosen to use a touch screen monitor as the
main interface as it provides means for the driver to do
advanced selections in the information flow, and context based
information can be displayed. This choice is in line with what
many of the car manufacturers have chosen. The particular
touch screen we have chosen is a high contrast colour 12” LCD
monitor. model “El0 Entuitive 1266V from Elo Graphics.
It comes as a panel mount device with a serial interface.
It is mounted next to the driver in a position suitable for
ergonomical interaction, where the driver can interact at a
glance. A head-up-display (HUD) displaying information in
the same field of view as the road has many advantages as
there is no need for the driver to change gaze direction or
the focal distance. However, currently available H U B suffer
from poor contrast and do not work well under some lighting
conditions.
Multimodal interfaces have a tendency to introduce a lower
workload on the user [14]. We are researching the possibilities
of adding, for example, auditory warning systems. A design
process for natural warning sounds is presented in [I51 which
introduces methods for ussociabiliiy and sound imagery. As-
sociability represents the required effort to associate sounds
to their assigned alert function meaning. An associable sound
requires less effort and fewer cognitive resources. Sound
imagery is used to develop sound images, which by its acoustic
characteristics has a particular meaning to someone without
prior training in a certain context.
V. SPEED SIGN RECOGNITION
Road sign recognition research has been around since the
mid 1980’s. A typical approach applies normalised cmss
correlation directly to several places in the traffic scene image.
Some more recent approaches separate detection and classifi-
cation steps [16]. This is particularly used when a large numbeI
3477
of sign types are to be classified. We argue that this can be
an effective means of managing computation for even a small
number of sign types if a detection stage is available that has
low computational cost, facilitating real-time operation.
We propose a new method of fast sign detection that is
applicable to signs with a circular feature, a significant subset
of signs: the fast radial symmetry detector [17]. All Australian
speed signs have a red circle on a white background with black
numbers. We are able to eliminate the vast majority of false
positives by considering only radially symmetric regions, that
are stable across several images, have a high count of pixels
in ratio to the radius.
Cross correlation can then be applied to the small number of
candidates. For cross-correlation, scale is generally a problem,
typically requiring multiple templates at different resolutions.
However, from the radius returned from the fast radial symme-
try detector we know the approximate scale of the template.
Fig. 3. A candidate sign detected by the fast radial symmetry detector at the
size if appears in the image. and close-up. The outer circle and numbers are
n m . Despite itr consistent appearance as a small image, it contains few
pixels that could be said to be r e d b k k , or even white.
There is much possible variation in the appearance of a
speed sign in an image. Throughout the day, and at night,
lighting conditions vary enormously. A sign may be well lit
by direct sunlight, headlights, it may be completely in shadow
even on a bright day, or heavy rain may blur the image of the
sign. Ideally, signs have clear colour contrast, but over time
they become faded, but still be clear to drivers. Although signs
appear by the road edge, this may be far from the car on a
multi-lane highway - to the left or right, or very close on a
single lane exit ramp. While signs are generally a standard
distance above the ground, they can also appear on temporary
roadworks signs at ground level. Thus, it is not simple to
restrict the possible viewing positions of a sign within the
image. By modelling the road [I81 or the sky, it may be
possible to dictate parts of the image where a sign cannot
appear, but road modelling has its own computational expense.
and colour-based methods are not robust.
However, the roadway is well structured. Under Australian
law the appearance of speed signs is highly restricted: they
must be of a particular size, and be a white sign with black
numbers surrounded by a red circle. Unless the sign has been
tampered with signs will appear approximately orthogonally
to the road direction. Finally, signs are always placed to be
easily visible, so the driver can easily see them without having
 ~
to look away from the road.
.Our algorithm searches the image for near circular features.
A legal speed sign must have a red circle around it, and the
signs almost always appear orthogonal to the road. Provided
our camera points in the direction of vehicle motion, the
surface of all signs will be parallel to the image plane of
the camera. On a rapidly curving road it may be that the sign
only appears parallel to the image plane briefly, but this will
be when the vehicle is close to the sign, so it will appear large
in the image. If we are processing images at > 30Hz.and we
are able to recognise a sign reliably from only a small number
of frames then generally we are safe to assume that the sign
is parallel to the image plane.
Three further restrictions on the radial symmetry detector
can reduce the number of candidates. Firstly, .we are only
interested in circles of a particular size range. A circle with
a small number of pixels as its radius may well constitute a
speed sign, however, there will noi be enough pixels present to
discern what the sign says, and so there is no point in funher
processing. we should wait until the sign is close enough to
be recognised. Further, in normal driving conditions a sign
will never appear closer to the camera than several metres.
Given a camera of approximately known focal length, we can
impose an upper limit on the possible radius of circles that we
are interested in. In our system these limits were empirically
derived based on sample road images.
A. Initial results on sign recognition
For evaluation purposes, the sign recognition system has
been run over several image sequences taken from the research
vehicle. The sequences come from cameras in a binocular head
mounted approximately in the position of the rear-view mirror.
All images used in the experiments were taken of signs on
public roads around Canberra. Two sample signs from those
used are shown in Figure 4. There is great variation in the
appearance of the signs, including apparent scale, lighting, and
the deterioration of the sign due to weather conditions.
Fig. 4. Sample imager with speed signs present. The quality of sign and
the lighting vaied within ow sequences, along with the scale of the sngn that
appeared. Also. more than one sign may appear in a single imge.
Classification using the fast radial symmetry detector has
proved highly sucessful, with only 10% false positives over
many varied scenes, and a total of 1107 input images.
The performance of the classifier on the candidates pro-
duced by the detection phase was encouraging. Requiring
temporal consistancy of classification for several frames could
3476
improve the results further, both rejecting invalid candidates,
and reducing the number of misclassifications.
VI. OBSTACLE DETECTION A N D TRACKING
Reliable obstacle detection and tracking is a challenging
problem in automotive research due to the diverse operat-
ing enviropment. Large variations in lighting, apparent ob-
stacle size and relative velocity all must be accomodated.
A moving sensor and unknown object appearance preclude
many classic segmentation techniques such as background
subtraction or appearance based feature detection. Progress
has tended along two avenues: superior sensing techniques
and constraining the problem. Superior sensors include laser
range finders, millimetre wave radars or large baseline stereo
cameras [19][20][7]. Constrained problem solutions have used
significant assumptions about the road scene, e.g., flat roads,
featureless road surfaces or looking for car-like objects only
[21]. Our technique combines several sensor data processing
techniques and some weak assumptions about obstacles (such
as a consistent size and location over time) to develop a robust
detection and tracking system. The strength of the system is
the ease of adding additional information sources (like better
sensors, or image processing algorithms).
A. System overview
The obstacle detection and tracking engine provides the “ob-
stacle state object”. This object can be queried to provide a list
of known obstacles. Each obstacle is returned with a location,
size and trajectory estimate and associated confidence.
The Obstacle detection and tracking engine has three phases
of operation: detection, distillation and tracking. Figure 5
shows these phases which operate concurrently detecting new
obstacles while tracking previously detected obstacles. The
first phase uses a set of “bonom up”, whole image techniques
(stereo disparity, optical flow, colour consistency) to search the
image for likely obstacles. The second phase uses the particle
filter based “Distillation algorithm” [22] [ 121.
Fig. 5 . Phases: 1. Obstacle detection. 2. Obstacle distillation, 3. Obstacle
tracking
B. Obstacle Detection
The obstacle detection phase is based on the coarse segmen-
tation of potential obstacles from a stereo template correlation-
based disparity map and image gradient-based optical flow
data. As mentioned in Frank et. al. [23] the range of optical
flow values and disparities encountered in the road scene is
large. Disparities and image motion in a single instance can
range from 0 at the horizon to over 64 pixels in the near field.
Frank et. al. limited the vehicle speed in their experiments to
satisfy the gradient based optical flow estimation constraint of
image motion of less than 2 pixels per frame. They mention
future work could use Gaussian image pyramids to enhance
the dynamic range possible in the flow estimation. We have
adopted a image pyramid technique both in the optical flow
and disparity map estimation. For optical flow we implement
a method similar to Simocelli [24]. Optical flow is computed
for the coarsest images then the result used to warp the next
higher image resolution to maintain an acceptably small image
motion at each level. The penalty for using a coarse to fine
approach is the integration of any errors.
Using an image pyramid and calculating the disparity for
each image resolution with the same size correlation window
means the correlation window is effectively halving for each
image resolution going from course to fine. This property
matches large objects at large disparities and smaller objects at
small disparities (i.e., on the horizon). Also, by not warping
the image between resolutions we avoid the propagation of
errors between image resolutions. At high resolutions we are
interested in finding distant objects with small disparities,
where larger objects such as close vehicles are recovered at the
coarse image resolutions. Coarse resolution images can only
resolve disparities to half the accuracy of the next higher image
resolution, but as this works in opposition to the property of
disparity estimates deteriorating as distances increase the effect
on the resultant disparity map is acceptable.
The stereo data is further processed by removing the road
surface using the V-Disparity technique [251. The disparity
of the road is assumed to be dominant making a contour in
the histogram. The road surface can then be removed from
the disparity image. Potential obstacles are extracted using
basic constraints of a minimum height and maximum height.
Figure 6 shows the resultant segmented road scene. Several
false positives exist, but this is acceptahle as the desired result
is a low number of false negatives (missed obstacles) at the
expense of some false positives (phantom obstacles) which
will be filtered in the second phase of the system.
Fig. 6. Road scene with obstacles identified with bounding boxer.
3479
Fig. 7. Distillation: a visual cue processing fmnework
C. Obstacle Distillation
In this phase the potential obstacles identified above are
“distilled” using the “Distillation algorithm” into consistently
detected obstacles. The “Distillation algorithm” (Figure 7) is
the combination of a particle filter with an intelligent cue
processing system. The cue processing system changes the
rate different sensor data is incorporated into the panicle filter
based on how well the sensors are performing. Stereo data
may be disrupted by a momentary occlusion of one camera,
in this case the information from this cue is ignored in favour
of other cues which are unaffected.
Sets of particles representing each obstacle candidate are in-
jected into the state-space in a Gaussian distribution around the
potential obstacle’s detected location. Stereo disparity, optical
flow and colour consistency cues are again used to evaluate
the potential obstacles, this time however, only the projected
locations of the particles are evaluated not the whole image.
Over several iterations of the filter, particles representing
unsubstantiated obstacles dissipate. Found obstacles are rep-
resented by clusters of particles which remain consistent over
time. Each cluster of particles surviving a minimum number
of iterations is checked against a Gaussian distribution at its
centroid. If the Gaussian distribution adequately describes the
cluster an extended Kalman filter based tracker is initialised
(phase three). Figure 8 illustrates clusters of panicles detected
representing obstacles to he replaced with Kalman filters.
POb..lll.
....
b
Fig. 8. Obstacle dirtillation: Uni-modal clusters in the parrick filler are
extracted far tracking.
D. Obstacle Tracking
In the third phase of the system an extended Kalman
filter and template correlation is used to track each obstacle
independently. Using the obstacle location extracted from
the previous phase, a uniqueness detector identifies good
image correlation templates from each stereo camera. This
collection of templates is then used lo track the obstacle. The
 correlation templates are tracked independently in the image
using normalised cross correlation. The collection of templates
per obstacle is then evaluated for the mean shift in the image
and high correlation. Templates tracking inconsistently or
unreliablely are discarded. The remaining templates are used
to estimate the new location of the obstacle fed to the extended
Kalman filter. The extended Kalman filter tracks the location of
the vehicle in the 3D road coordinate system using a constant
velocity motion model. The size of the vehicle estimated in
the previous phase is assumed to be constant. Eventually
the obstacle tracked is lost: either overtaken, obscured, too
far in front of the vehicle to be seen, or any other random
failure. This condition is identified by either no reliable image
templates remaining or by a divergence in the covariance
matrix of the filter. In either case the system will discard the
extended Kalman filter and, as a precaution, inject a cluster
of particles at the final location of the object back into the
particle filter in the above phase of the system.
Figure 9 shows the output of the obstacle tracking engine.
Each car is tracked using an independent extended Kalman
filter and image template correlation. Later in this image
sequence the centre car is lost due to a template tracking
failure (only one template is tracking reliably at this stage),
then the second phase of the system quickly detects the vehicle
again and tracks using a new filter and new image templates.
Also in this sequence the car on the far right is obscured
: by an overtaking vehicle. Again this obstacle is lost, and the
overtaking vehicle detected and tracked instead.
Fig. 9. Output of obstacle tracker. Rectangles indicale obstacle bounding
boxes, ‘+’ indicate correlation template locations. ‘*’ indicate centroid of
obstacle.
The lane tracking engine, which also is based on the
Distillation tracking algorithm [22], has been developed by
Apostoloff et al. and is decrihed in full in [12]. In this sub-
system visual cues such as edges and road colour consistency
are combined with cues based on physical world constraints
such as vanishing points and plausible road shapes, to distill
a winning hypothesis of the vehicle’s position with respect to
the road. Lane tracking estimates the road width, lateral offset
of vehicle from road centerline and the yaw of the vehicle
with respect to the road centerline (see Figure IO).
3480
.
Fig. 10. Road model used far the particle filter. The dark shaded region
is wed as the non road boundary in the mlour cues while the Light shaded
region is the mad region. Note that the figure is exaggerated for clarity.
VII. INTEGRATION
The driver-DAS interface has been designed with concepts
of “intuitiveness”, “noninuusiveness” and “ovemdability” in
mind. above uses standard road signage colours to convey the
importance of information. Critical issues appear red, warnings
yellow, and affirmative information in green.
In the DAS lane position is shown as lines overlayed on
the camera image. Lane departure is shown as as green to
red tinting over the left or right side of the image (based on
which lane boundary is being crossed). The intensity of the
colour represents the perceived seriousness of the violation.
Intentional lane departures are detected by the driver using
the indicators. Acknowledgement of the warning can be made
by the driver by looking at or touching the touch-screen.
Tracked obstacles are highlighted on the touch-screen, again
a colour coding system represents the DAS’s perceived es-
timate of the threat. Close fast moving obstacles have red
boundaries. distant vehicles green. Colour coding is adjusted
based on vehicle speed, i.e. at lOOkm/h vehicles 10 metres
away are red, at 5 0 k d vehicles I O metres away are yellow.
Warnings are based on driver attention. If there is an obstacle
that is perceived a high threat yet the driver is detected as
looking in that direction, no warning is given. If the driver
head pose and eye gaze has not been in the direction of the
obstacle a warning is given.
Regarding the speed warning system, it is reasonable to
assume that if the driver is aware of the current speed limit
they should have adjusted their speed to a safe level. Thus,
if a system can monitor speed signs, and inform the driver if
they are currently driving too fast (or too slow), then the driver
will only gain information that they don’t know. (Alternatively,
the driver may have decided not to react to the information,
in which case it may not be a bad thing to inform them
anyway). Speed signs are continously being monitored and
when a strong candidate appears it is shown in the interface
together with its interpretation. Thus, the driver can tell what
information the warning was based on.
Circumstances can change quickly, so instead of relying on
the driver to remember to look at the screen, several auditory
warnings are planned. Warnings will consist of tones based
on the threat level and will only occur when sustained high
level threats have been ignored (such as prolonged speeding,
or lane departure).
 It is planned to integrate these waminglinformation sys-
terns further to driver assistive systems with higher
behavioural functions. These second generation systems will
attempt to interact more like a human co-pilot. Instead of the
monitoring lane position these systems would look
for trends and deviations in the behaviour of the driver and
road environment. A draft of the gui is shown in Figure 11.
Output from the speed sign detection algorithm (Section V)
and the obstacle detection algorithm (Section VI) is shown.
The user interface has to be a comnromise between orovidine
clear, instantaneous messages and suffient information for the
- DD. 369k3702.
driver to re-evaluate the decision made by the system ( e.g.,
the image of the speedsign that may have been misread).
Fig. 11. A draft of the gui showing output from the speed sign detection
algorithm at the top. and carlobslade detection at Ihe bottoni.
VIII. CONCLUSION & FUTURE WORK
A framework for interactive driver assistance systems was
presented. Techniques for fast speed sign detection and classi-
fication, car detection and tracking, and lane tracking were
included. The driver assistance framework includes means
for monitoring the driver’s actions, and the integrated system
uses information extracted from the road scene together with
information about the driver’s state such as eye gaze and
head pose to issue adequate warnings. A touch screen monitor
was used to present relevant information and allow the driver
to interact with the system. The research focuses around
robust and reliable algorithms for extracting information from
the road scene, adequate warning mechanisms and a proper
integration of subsystems.
Future work includes further research on appropriate warn-
ings, such as different auditory signals depending on the level
of threat. It is planned to study the behavior of the driver
and vehicle more closely and to detect when deviations from
the normal or expected occurs. The area of driver assistance
systems is enormous, and the presented framework allows
research to be carried out easily in this domain.
Ix. ACKNOWLEDGEMENT
Thanks to Leanne Matuszyk for helping out with experi-
ments, pictures and the G n , Thanks a[so to Gareth Loy for
proof reading.
R E F ER EN CE S
[I] R. Aufrere, J. Gawdy, C. Meru, C. Thorpe, C.E. Wang, and T. Yata.
‘Perception for collision avoidance and autonomous driving:’ Mecho-
nonics, vol. 13, no. 10,pp. 1149-1161. December 2003.
[21 C. Mem, S. McNeil, and C. ‘I%orpe, “Side collision waming systems for
transit buses,” in I V 2003, IEEE Intelligent Vehicle Sjmposium, October
2000, paper accepted for publication.
3481
[3] L. zhao and C. “stem and neural network-based pedestrian
detection,’’ IEEE Tmnsonions on lnlelllgenl Tmnsponation Sy3Iems.
vol. 1. no. 3, pp. 148 -154. September 2000.
E, D. and A, zapp, .AAuton,anous high speed road vehicle
guidance by computer vision,” in Pmc. IFAC. 1987, pp. 221-226.
[51 E. D. ~icLmanns.“Vertebrate-type vision for autonomous vehicles:’ in
P m . Sympesium on Biologically Inspired Systems. December 2000.
[61 _, .Anexpectation.based, multi.faeal, saccadic (ems)vision
for vehicle guidance:’ in Pmc. Intemrionol Symposium on Robodes
Rcsemh. Salt Lake cik October l999.
171 A. Broggi. M. Bertozri, and A. Fascioli. ”Self-calibration of a stereo
..
vision s w e m for automotive aoolications.” Seoul. Korea. May 2001.
[8] M. Benomi. A. Bmggi, M. CarleUi. A. Fascioli. T. Gmf, P. Grisleli,
and M. Meinecke, “Lr pedestrian detection for advanced driver assis-
tance systems,” in Pmc. Ponem Recognition Symposium,Magdeburg,
Gennany, September 2003.
[9] Seeing Machines, “Facelab face and eye tracking system:’
hrrp:wnw.seeingrr?chines.enm, 2001.
[IO] N. L. Haworth, T. Trig@. and E. M. Grey, “Driver fatigue: Concpets,
measurement and crash countermeasures,’’ Federal Office of Road Safety
Contract Report 72 by Human Fanors Group. Depmment of Psychol-
ogy, Monash Universily, Tech. Rep.. 1988.
[ I l l M. L a d and D. Lee. “where we lwk when we steer:’ Nolure, no. 369,
pp. 742-744, lune 1994.
1121 N. Apostoloff and A. &tinsky, “Wsion in and out of vehicles: iotegrated
driver and road scene monitoring,” in Lrprrirnrntol RoOorics Vlll. rer.
Advanced Robotics Series, B. Siciliano and P. Dario, Eds Springer-
Verlag, 2002.
[ 131 M. Cellario. “Human-centered intelligent vehicles: Toward multimodal
interface integration,” lnrelligenr Tmnsponarion Swems. pp. 78-81.
July 2001.
[I41 Y. Liu and T. Dingus, ‘Development of human factors gudielines for
advanced !raveler information systems (atis) and commercial vehicle
operations (WO): Human facton evaluation of the effectiveness of multi-
modality displays io advanced Vdveler information systems:’ Federal
Highway Adminishation, Washington D.C., Tech. Rep. FHWA-RD-96-
150. July 1999.
1151 P. Ulfvenpn, “Design of natural warning sounds in human-machine
syrtems:’ PhD. dissertation, Depamnent of Industrial Economics and
Management Series, Royal Institute of Technology, Stockholm, Sweden.
December 2003.
[I61 P. Paclik, I. Novovicicova, P. Somol, and P. Pudil, “Road sign clasification
using the laplace kernel classifie2’ Podern Recngnifion Letters, vol. 21.
pp. 1165-1173, 2oW.
[17] G. Loy and A. &hsky, “Fast radial symmetry for detecting points of
interest:‘lEEE Tronr Porrcm Amlysir and Mochine Inrelligence. vol. 25.
no. 8, pp. 959-973. Aug. 2003.
1181. R. Labawde, D. Auben. and 1.-P. Tarel,“Real time obstacle detection
.
in st-vision on non Rat mad geomehy through ’’
[I91 1. Robem and P. Cake, “Obstacle detection for a mining vehicle using
a 2d laser,” in Pmc. Auslmlion Conference on Roborics and Auromalion
(ACR4). Melbounn. Aurfralia. August 2000, pp. 185-190.
1201 G.Bmoker and H. D m t - W h i t e . “Millimeuewave radar:’ Melbourne.
AusValia. 2001.
1211 E Dellam. D. Pomerleau, and C. Thorpe, “Model-based car tracking
integrated with a may 1w8,
~ 2 2 1G. b y , L. Retcher. N. A ~ O S ~ O I O and
~ ~ . A. Zelinsky, adaptive
fusion architecture for target hacking:’ in Pmc. The 51h Inlemalionol
Conference on Auromlic Foee ond Gesrure Reco~nition,Washington
DC. May 2002.
[23] U. F&e and S. Heinrich, “Fast obstacle detection for urban naffic
situations:’ vol. 3. no. 3, pp. 173-181, September 2002.
[24] E. P. Shoncelli. Boyesion Multi-scok Differential oplicd Flow. Aca-
demic Press. 1999, vol. 2. ch. 14, pp. 297422.
.1251. R. Labawde. D. Aubert. and J.-p. Tare1 “Real time obstacle detection
in stere&ision on non &mad I geomehy through ”v-disparity” repre-
senlation,” in Pmc. IEEE ltIIdigm1 Vehicle Symposium. France. lune
2002.