2004 IEEE IntelligentVehicles Symposium
University of Parma
Parma, Italy June 14-17,2004
Multi-Target Multi-Object Tracking,
Sensor Fusion of Radar and Infrared
Rainer Mobus,'>*
'Research & Technology, RICIAA
DaimlerChrysler AG, 70546 Stuttgart, Germany
email {rainer.moebus,
u1i.u. kolbe} @daimlerchrysler.com
Abstract- This paper presents algorithms and techniques
for single-sensor tracking and multi-sensor fusion of infrared
and radar data. Multiple model filtering and data associa-
tion techniques are presented and results are shown for all
presented algorithms.
I. INTRODUCTION
Many driver assistance systems heavily rely on external
vehicle sensors. One product that is already available on
the market is the so-called Adaptive Cruise Control, ACC,
that makes use of a 77GHz radar sensor. The maximum ac-
celeration is constrained to 1 2 , the maximum deceleration
is limited to-32. A considerable disadvantage of current
77GHz ACC systems is their limited angular sensor range.
The longitudinal range is very good and goes up to 150
Fig. 1. Sensor ranges of 77GHz radar and infrared laser:
meters, in best cases beyond. The lateral range, however,
is limited to f 3 . 4 degrees. For assistance systems beyond
ACC the 77GHz radar sensor .might not be sufficient.
For improved ACC functionality, lane change assistants,
inner-city assisting systems or collision warning / collision
mitigation systems a rather complete representation of the
traffic scene ahead of the car is needed and the reliability
of the sensor information should be as high as possible.
Another sensor suitable for object recognition is the infrared
sensor, scanning an area of about f 2 0 degrees up to 80
meters. Figure 1 shows the ranges of the 77GHz radar
sensor'and the infrared scanner, the infrared sensor scanning
a large area uncovered by the radar. To exploit the advan-
tages of each technology without being affected by their
shortcomings all sensor information can be fused in a sensor
fusion step. Complementary regions add up to a complete
range ahead of the car, in overlapping regions redundant
information is obtained increasing the reliability of the
o-7ao3.~a31o-91o41$2o.ooo 2004 IEEE
Uli Kolbe'
*Automatic Control Laboratory
ETH Zentrum - ETL, CH-8092 Zurich, Switzerland
email moebus@control.ee.ethz.ch
detected objects. Figure 2 shows a 7'7GHz standard ACC
sensor and the infrared sensor used for the multi-sensor
fusion in this paper. To show the motivation and necessity
of tracking and fusion algorithms figure 3 shows a bird view
of all infrared and radar measurements obtained during a
20 second test drive. The graph shows the trajectory of
the test vehicle as a solid line and the obtained sensor
measurements as points. For the sake of readablity the
lateral position of the measurements is exaggerated by a
factor of 20. These data show that a powerful tracking and
powerful data association and fusion algorithms are needed
to reject clutter and noise and to estimate the dynamics of
the preceding vehicles reliably.
Fig. 2. Pictures of a 77GHz radar sensor and an infrared sensor for
automotive use.
11. OUTLINE OF THIS PAPER
This paper presents algorithms and techniques for the
single-sensor tracking and the multi-sensor fusion. Chapter
three is dedicated to single-sensor tracking and to results
of radar tracking and infrared tracking with the interacting
multiple model approach. Chapter four covers sensor fusion
and data association techniques, especially the probabilistic
data association. The fifth chapter presents results of multi-
ple model filtering applied to the sensor fusion of infrared
and radar data. The last chapter gives an outlook.
111. SINGLESENSOR
TRACKING
An overview of the operations performed in the single-
sensor tracking is shown in figure 4. Different sensors
generate sensor specific measurements or targets. In the
first operation (gating, clustering) a pre-selection of these
targets takes place. New targets are compared to existing
732

150 -100 -50 0 so 100 150
lateral position [m]
Fig. 3. Bird view of a 20 second test drive. The trajectory of the test
vehicle as well as all infrared and radar measurentents are shown.
Fig. 4. Flow chart of the operations perjormed in the single-sensor
tracking step.
objects. If a target is likely to be a new measurement for an
already existing object, the target is labeled. In some cases
an object creates more than one measurement. In this case
measurements are clustered and labeled accordingly. In this
gating step some unlikely measurements can be discarded,
e.g. standing objects that might not be of interest in certain
applications [ 2 ] , [4]. After this step the data association
takes place. The task is to associate the validated targets
to existing objects. After this operation, all associated
measurements are used to update the state estimation of the
existing objects. If a target could not be associated to an
existing object, a new object is created. If no measurement
was assigned to an existing object, this object is killed.
Objects might also be merged or split here. Output of the
single-sensor tracking is a list of objects for each sensor.
Attributes of this list are e.g. distance, lateral distance,
relative lateral velocity and relative longitudinal velocity.
A. Tracking with 77GHz Radar Sensor
In [SI and [7] the single-sensor radar tracking was pre-
sented with the so-called interacting multiple model filter
approach. The idea of the IMM filter is briefly explained
by figure 5. Two Kalman filters with different models or
parameter sets are running in parallel and interact according
*'(k I r)
Fig. 5. Flow char! of the IMMfilter principles.
200
Fig. 6. Results of longitudinal radar tracking using the IMM filter
approach.
to their likelihood. In each time step the actual state vector
is predicted and innovated using different hypotheses. The
resulting state vector is a linear combination of the two
hypotheses. The weighting is according to the likelihood of
the hypotheses. For further detail the reader is referred to
[5], [6], [SI. Results obtained by this approach applied to the
longitudinal tracking of 77GHz radar data is shown in figure
6. Model 1 was chosen as a constant velocity model, model
2 as a constant acceleration model. The results presented
in figure 6 show relative velocity and relative acceleration
corresponding to a traffic scene where a car is cruising
straight ahead of the test vehicle. First the relative velocity
is decreasing considerably, remains constant for a while
and then increases again. The graph shows raw relative
velocity measurements of the radar sensor, the tracked
relative velocity and the estimated relative acceleration.
The segments with changing relative velocity correspond
to longitudinal maneuvers of the test vehicle or the car
ahead. The model probabilities graph shows that in case of
these maneuvers the probability of the constant acceleration
733
 5 3
32
Fig. I . Results of lateral infrared laser tracking using the IMM filter
approarh.
model is very high. The constant velocity model however
dominates in case of a rather steady state behavior of the
two cars.
B. Tracking with Infrared Sensor
Results of the IMM filtering applied to the lateral tracking
of infrared data are shown in figure 7. Model 1 is chosen
here as a constant lateral distance model and model 2 as
a constant lateral velocity model. The results presented in
figure 7 correspond to a traffic scene where a car is cruising
ahead of the test vehicle, then performs a lane-change to the
left, another lane-change back to the lane of the test vehicle
and finally leaves this lane to the right. The graphs show
the raw data and the tracked data of the lateral position,
the estimated lateral velocity and the corresponding model
probabilities. The model probabilities graph shows that the
probability of the constant lateral velocity model gets very
high in case of lane-changing maneuvers.
IV. SENSORFUSION
The multi-sensor fusion is set up in the same man-
ner as the single-sensor tracking. The multi-sensor fusion,
however, uses the output of the single-sensor tracking as
input Iist. An overview of the operations performed in this
sensor fusion step is shown in figure 8. Comparing figures
4 and 8 shows that the principles and techniques of the
single-sensor tracking and the sensor fusion step are very
similar. The advantage of this separation of the tracking
and sensor fusion step is modularity, complexity reduction
and reusability of the algorithms. These advantages are
given priority over one large sensor fusion step on the
measurement level.
A. Probabilistic Data Association (PDA)
In a multi-target, multi-object environment two typical
situations are depicted in figure 9. These two traffic scenes
are real-world examples and were obtained during a test
drive on a public highway. The figures show laser and radar
734
Fig. 8 . Flow chart of the operations perfonon,ied in the multi-sensorfusion
step.
Fig. 9. Bird view of 111'0 rrafic scenes to illitstrate the data association
problern.
objects tracked with the presented tracking algorithms. The
laser objects appear as thin black boxes, the radar objects as
thick black crosses. The fusion objects are depicted as thick
boxes. The lane information obtained by a vision system is
also depicted as dashed lines. The lines moving away from
the small car symbol indicate the ranges of the radar and the
infrared sensor. In the left scene at about 65m two fusion
objects, two laser objects and two radar objects can be seen.
In the right scene at about 40m a similar configuration
is shown with two fusion objects, two radar objects and
one laser object. In the fusion algorithm it now has to be
decided which measurements (radar and laser objects) are
associated to which existing objects (fusion objects). For
most objects shown in these two scenes this step is rather
straightforward, for the two particular configurations just
mentioned, however, it is not. Various approaches exist to
perform this data association operation 111, [21, [31, [41.
Most of these data association algorithms make use of the
so called gating.
The probability density function of the Kalman filter resid-
uals is given by
where 7(k) is the corresponding residual vector, given by
J(k) = [?(k)-H(k)i(klk- l)]. (2)
The corresponding residual covariance matrix S evaluates
to
S ( k ) = H ( k ) P ( k \ k - l)H(k)T + R ( k ) , (3)
with H ( k ) describing the observation matrix and R the
measurement noise. A measurement is declared valid if
the normalized residual fulfills the following thresholding
condition.
.\/7(k)TS(klk- 1)-17(k) 5 c. (4)
Statistically more than 99.8% of all valid measurements lie
within the gate defined by choosing c = 3. The probabilistic
data association approach uses all measurements that are
valid according to this gating. All valid measurements are
then combined to one single residual. The weighting is
according to the likelihood values of the corresponding
measurements. The combined residual vector of m valid
measurements is calculated as follows.
2(klk) = a ( k p - 1 ) +K ( k ) 3 ( k ) . (7)
The updated covariance matrix P(klk) is the standard
Kalman filter covariance update plus a correction term that
accounts for additional uncertainty due to the fact that the
different residuals are combined to one but are unequally
weighted. The PDA Kalman filter covariance update is
The probabilistic data association is clearly a suboptimal
data association technique. The advantages over other (op-
timal) data association techniques are low complexity and
the fact that all measurements with a certain minimum
likelihood value are considered in the innovation step. Only
very unlikely measurements are discarded. For the specific
problem of object tracking for automotive applications these
characteristics seem appropriate. Looking at the two traffic
scenes in figure 9 it is reasonable to use the radar objects
between the fusion objects for the update of both fusion
objects. Discarding them might implicate loss of informa-
tion. Associating them to just one fusion object might be
wrong and creating a new fusion object is not reasonable
conceming the little lateral distance of the existing fusion
objects.
735
B. Probabilistc Data Association with IMM
The data association operations always take place in
between the prediction and innovation step of the dynamic
Kalman filter process [l]. If IMM filtering is applied, this
has to be considered and the PDA has to be adapted. With
n filters running in parallel, first a combined covariance
matrix Pc is calculated according to the predicted model
probabilities
n
Pc(klk- 1) = &u'(k(k- 1)P"klk- l ) , (9)
i= 1
with
,ui(klk- I ) , i = I...n (10)
being the predicted model probabilities. For each measure-
ment 7i(k) the corresponding residual covariance matrix
S i ( k ) = Hj(k)Pc(k(k-l)Hj(k)T +Rj(k) (11)
can be calculated. This matrix Si is then used to perform the
gating according to (4) to decide whether the measurement
7i(k) is declared valid or not. The combined residual accord-
ing to (5) and (6) is then used to perform the innovation
step separately for all IZ parallel Kalman filters within the
IMM filter.
V. SENSORFUSION
OF 77GHz RADARAND IR
In this chapter results of single-sensor tracking and sensor
fusion are presented, obtained by the algorithms described
above. In figure 10 results of both infrared and radar single-
sensor tracking and of the sensor fusion are shown. The
results correspond to the same traffic scene described for
figure 7.
The first graph shows the tracked lateral distance of radar
and infrared single-sensor tracking. The result of the sensor-
fusion step is also shown in the same subfigure. It can
be seen, that the car is always in range of the infrared
sensor whereas the radar tracking looses the object when
it leaves the sensor's range. The following two graphs
show longitudinal distance and relative velocities of both
infrared and radar. The results of single-sensor tracking
and sensor fusion are depicted. The next graph shows the
relative longitudinal acceleration obtained from the sensor
fusion step. The last graph shows the longitudinal model
probabilities of the steady state model (constant relative
velocity) and the maneuver model (constant relative accel-
eration model). As expected, the maneuver model prevails
when the acceleration is considerably high. Figure 1 1 zooms
in the interval of about 15s-25s of both lateral position
estimation and relative velocity estimation. In the lateral
position graph it can be seen that the fusion result follows
the infrared tracking closer than the radar tracking. This
is due to the smaller measurement noise given to the
infrared signal and to the PDAF based residual calculation.
The same reasoning applies to the relative velocity graph
where the sensor fusion signal follows the radar data closer.
 IOW

Boo -

800 -

-
L
E
700-

2 600-

g
f 500-
10 15 eo 25 so
'8,
E 400-
-0

300 -

200 -

loo -

L 8 I I ' 1 3 , I

-200 -150 -100 60 0 50 100 150 2

lateral position [mf

Fig. 12. Bird v i m of a 20 second test drive. All infrared and radar
ineasurenients are shown as well as the trajectory of the test vehicle and
two fusion objects.

presented algorithms are shown. The fusion object track

Fig. 10. Results of longitudinal and lateral sensor fusion of radar and
infrared data using IMM filtering and probabilistic data association.
close to the trajectory of the test vehicle corresponds to
the data shown in figure 7 , l O and 1 1 . The second track
belongs to another vehicle cruising 011 the left side of the
test vehicle. Comparing the trajectories of the fusion objects
with the sensor measurements, it can be seen that the phase
delay between measurements and fused objects is rather
moderate. Another obvious thing is that the lateral position
estimation is dominated by the infrared sensor.

VI. SUMMARY AND CONCLUSION

In this paper algorithms for single-sensor tracking and
15 w multi-sensor fusion were presented. The results show that
fusing radar data with infrared data considerably increases
-1.5 detection range, reliability and accuracy of the object
tracking. This is mandatory for further development of
c driver assistance systems. Using multiple model filtering
E
2 for sensor fusion applications helps to capture the dynamics
3 of maneuvering objects while still achieving smooth object
tracking for not maneuvering objects. This is important
when safety and comfort systems have to make use of
the same sensor information. Comfort systems generally
require smoothly filtered data whereas for safety systems it
is crucial to capture maneuvers of other road users as fast
Fig. 11. Close look at lateral distance and relative velocity estimation as possible. All presented algorithms are tested in real-time
offgure 10. on standard PC sytems.
VII. OUTLOOK
To demonstrate the capability of the presented algorithms, In further research the presented data association al-
figure 1 2 shows the same data as figure 3. Additionally gorithms will be compared to optimal data association
the tracked trajectory of two fusion objects obtained by the algorithms. The results of the presented tracktng algorithms

736
will be compared to reference data obtained by vehicle-to-
vehicle communication.
Further the effects of multiple model estimation upon
optimal longitudinal control as presented in [7] will be
examined in more detail.
REFERENCES
[l] A. Gelb, “Applied Optimal Estimation”, MIT Press 1974.
[2] Y. Bar-Shalom, T. E. Fortmann, “Tracking and Data Association” ,
Academic Press INC, San Diego, New York 1988.
[3] Y. Bar-Shalom, W. Dale Blair, “Multitarget-Multisensor Tracking:
Applications and Advances Volume III”, Artech House, Boston,
London, 2000.
737
[4] S. S. Black”, “Multiple Target Tracking with Radar Applications”,
Artech House, 1986.
[SI A.P. Blom, Y. Bar-Shalom, “The Interacting Multiple Model Algo-
rithm for Systems with Markovian Switching Coefficients”, IEEE
Transactions on Automatic Control, Vol 33, No 8, 1988.
[6] Xiao-Rong Li, Y. Bar-Shalom, “Multiple Model Estimation with
Variable Structure” IEEE Transactions on Automatic Control, Vol
41, No 4, 1996.
[7] Rainer Mobus, Mato Baotic, Manfred M o d , ”Multi-Object Adap-
tive Cruise Control”, Hybrid Systems Computation and Control
Conference HSCC 03, Prague, Czech Republic, April 2003.
[8] Rainer Mobus, Annin Joos, Uli Kolbe, “Multi-Target Multi-Object
Radartracking”, IEEE Intelligent Vehicles Conference, Columbus,
Ohio, USA, June 2003.