Lecture Outline

Target Tracking: Lecture 7

Multiple Sensor Tracking Issues
Multiple Sensor Tracking Architectures
Umut Orguner Multiple Sensor Tracking Problems
umut@metu.edu.tr
Track Association
room: EZ-12 tel: 4425
Track Fusion
Department of Electrical & Electronics Engineering
Middle East Technical University
Ankara, Turkey

1 / 37 2 / 37

Multi Sensor Architectures Multi Sensor Architectures

Figures taken from: M.E. Liggins and Kuo-Chu Chang

“Distributed Fusion Architectures, Algorithms, and Performance within a Network-Centric Architecture,”
Ch.17, Handbook of Multisensor Data Fusion: Theory and Practice, Taylor & Francis, Second Edition, 2009.

Figures taken from: M.E. Liggins and Kuo-Chu Chang

“Distributed Fusion Architectures, Algorithms, and Performance within a Network-Centric Architecture,”
Ch.17, Handbook of Multisensor Data Fusion: Theory and Practice, Taylor & Francis, Second Edition, 2009.

3 / 37 4 / 37
Multi Sensor Architectures: Centralized Multi Sensor Architectures: Hierarchical without Memory

Figures taken from: M.E. Liggins and Kuo-Chu Chang

“Distributed Fusion Architectures, Algorithms, and Performance within a Network-Centric Architecture,”
Ch.17, Handbook of Multisensor Data Fusion: Theory and Practice, Taylor & Francis, Second Edition, 2009.

Figures taken from: M.E. Liggins and Kuo-Chu Chang

“Distributed Fusion Architectures, Algorithms, and Performance within a Network-Centric Architecture,”
Ch.17, Handbook of Multisensor Data Fusion: Theory and Practice, Taylor & Francis, Second Edition, 2009.

5 / 37 6 / 37

Multi Sensor Architectures: Hierarchical with Memory Multi Sensor Architectures: Hierarchical with Feedback
without Memory

Figures taken from: M.E. Liggins and Kuo-Chu Chang

“Distributed Fusion Architectures, Algorithms, and Performance within a Network-Centric Architecture,” Figures taken from: M.E. Liggins and Kuo-Chu Chang
Ch.17, Handbook of Multisensor Data Fusion: Theory and Practice, Taylor & Francis, Second Edition, 2009. “Distributed Fusion Architectures, Algorithms, and Performance within a Network-Centric Architecture,”
Ch.17, Handbook of Multisensor Data Fusion: Theory and Practice, Taylor & Francis, Second Edition, 2009.

7 / 37 8 / 37
Multi Sensor Architectures: Hierarchical with Feedback Multi Sensor Architectures: Decentralized without Memory
with Memory

Figures taken from: M.E. Liggins and Kuo-Chu Chang

Figures taken from: M.E. Liggins and Kuo-Chu Chang “Distributed Fusion Architectures, Algorithms, and Performance within a Network-Centric Architecture,”
“Distributed Fusion Architectures, Algorithms, and Performance within a Network-Centric Architecture,” Ch.17, Handbook of Multisensor Data Fusion: Theory and Practice, Taylor & Francis, Second Edition, 2009.
Ch.17, Handbook of Multisensor Data Fusion: Theory and Practice, Taylor & Francis, Second Edition, 2009.

9 / 37 10 / 37

Multi Sensor Architectures: Decentralized with Memory Multi Sensor Architectures: Pros & Cons

The traditional centralized architecture gives optimal

performance but
Requires high bandwidth communications.
Requires powerful processing resources at the fusion center.
There is a single point of failure and hence reliability is low.
For distributed architectures
Communications can be reduced significantly by
communicating tracks less often.
Computational resources can be distributed to different nodes
Higher survivability.
It is a necessity for legacy systems e.g. radars sometimes might
not supply raw data.
Figures taken from: M.E. Liggins and Kuo-Chu Chang
“Distributed Fusion Architectures, Algorithms, and Performance within a Network-Centric Architecture,”
Ch.17, Handbook of Multisensor Data Fusion: Theory and Practice, Taylor & Francis, Second Edition, 2009.

11 / 37 12 / 37
Problems in Multi Sensor TT Correlation

Registration: Coordinates (both time and space) of different Suppose the fusion center have the
Centralized Case
sensors or fusion agents must be aligned. prediction x̂k|k−1 in both cases.
yk1
Bias: Even if the coordinate axes are aligned, due to the
Centralized Case
transformations, biases can result. These have to be compensated. Fusion
x̂k|k  1     1 
Center
Correlation: Even if the sensors are independently collecting data, yk C1 ek
2 = xk +
processed information to be fused can be correlated. yk2
yk C2 e2k
Rumor propagation: The same information can travel in loops in Decentralized Case where e1k and e2k are independent.
the fusion network to produce fake information making the overall
yk1 x̂1k|k
system overconfident. This is actually a special case of correlation. Tracker-1 Decentralized Case
Out of sequence measurements: Due to delayed communications " #  
Fusion
x̂1k|k
 1 
between local agents, sometimes measurements belonging to a target Center
x̂k|k I x̃k
= xk −
whose more recent measurement has already been processed, might x̂2k|k I x̃2k
Tracker-2
yk2 x̂2k|k
arrive to a fusion center.

13 / 37 14 / 37

Correlation Correlation
Define x̃ik , xk − x̂ik|k , then
Suppose the target follows the dynamics
x̃ik =xk − Ax̂ik−1|k−1 − Kki Ci A(xk−1 − x̂ik−1|k−1 ) − Kki Ci wk − Kki eik
xk = Axk−1 + wk =Axk−1 + wk − Ax̂ik−1|k−1 − Kki Ci A(xk−1 − x̂ik−1|k−1 )
and the ith sensor measurement is given as − Kki Ci wk − Kki eik
=(I − Kki Ci )Ax̃ik−1 + (I − Kki Ci )wk − Kki eik
yki = Ci xk + eik
Hence
Then with the KF equations
x̃ik =(I − Kki Ci )Ax̃ik−1 + (I − Kki Ci )wk − Kki eik
x̂ik|k =Ax̂ik−1|k−1 + Kki (yki − Ci Ax̂ik−1|k−1 )
x̃jk =(I − Kkj Cj )Ax̃jk−1 + (I − Kkj Cj )wk − Kkj ejk
=Ax̂ik−1|k−1 + Kki Ci (xk − Ax̂ik−1|k−1 ) + Kki eik
=Ax̂ik−1|k−1 + Kki Ci A(xk−1 − x̂ik−1|k−1 ) + Kki Ci wk + Kki eik We can calculate the correlation matrix Σij i jT
k , E(x̃k x̃k ) as

Σij i ij T j T i j
k = (I − Kk Ci )AΣk−1 A (I − Kk Cj ) + (I − Kk Ci )Q(I − Kk Cj )
T

15 / 37 16 / 37
Correlation Correlation Illustration
Assuming that Σij0 = 0, we can calculate the correlation between
the estimation errors of the local trackers recursively as Maneuvers make this problem more dominant and visible.
120

Σij ij j j σa = 0.3m/s2 100

i T T i T
k = (I − Kk Ci )AΣk−1 A (I − Kk Cj ) + (I − Kk Ci )Q(I − Kk Cj )
2200

2000
80
1800

Position Errors (m)

1600
60
1400

y (m)
1200
40

This necessitates that the fusion center knows the individual 1000

Kalman gains Kki and Kkj of the local trackers which is not
800
20

600

400
0

very practical. 200

800 1000 1200 1400 1600 1800
x (m)
2000 2200 2400 2600 2800
120
0 20 40 60 80
time (s)
100 120 140 160

σa = 0.3m/s2

Assuming that the errors are independent is not a good idea 2200

2000
100

either. 1800

1600
80

Position Errors (m)

1400

Neglecting the correlation makes the resulting estimates 60

y (m)
1200

overconfident i.e., very small covariances meaning that too

1000

40
800

small gates and smaller Kalman gains. 600

400
20

Σij
200

When Q = 0, k = 0, i.e., no correlation when no process 800 1000 1200 1400 1600 1800
x (m)
2000 2200 2400 2600 2800
0
0 20 40 60 80
time (s)
100 120 140 160

noise.
17 / 37 18 / 37

Rumor Propagation Rumor Propagation

Hierarchical Case: Rumor always flows to the fusion center Decentralized case: Rumor propagates everywhere.

Figures taken from: M.E. Liggins and Kuo-Chu Chang

Figures taken from: M.E. Liggins and Kuo-Chu Chang “Distributed Fusion Architectures, Algorithms, and Performance within a Network-Centric Architecture,”
“Distributed Fusion Architectures, Algorithms, and Performance within a Network-Centric Architecture,” Ch.17, Handbook of Multisensor Data Fusion: Theory and Practice, Taylor & Francis, Second Edition, 2009.
Ch.17, Handbook of Multisensor Data Fusion: Theory and Practice, Taylor & Francis, Second Edition, 2009.

19 / 37 20 / 37
Track Association: Testing Track Association
Test for Track Association [Bar-Shalom (1995)]:
What about the cross covariance Σij
k?
Two estimates x̂ik|k , x̂jk|k
and the covariances Σik|k , Σjk|k are
given from ith and jth local systems. Simple method is to set it Σij
k = 0.

We calculate the difference vector ∆ij It can be calculated using Kalman gains if they are transmitted to
k
the fusion center.
∆ij i j
k , x̂k|k − x̂k|k Approximation for cross covariance from [Bar-Shalom (1995)]:
The following cross-covariance approximation was proposed:
Then we calculate covariance Γij ij ijT
k , E(∆k ∆k ) as
 . 1
Σij j
i 2
k ≈ ρ Σ k|k . ∗ Σ k|k
Γij i j ij ijT
k = Σk|k + Σk|k − Σk − Σk
where multiplication and power operations are to be done
Then test statistics Dkij calculated as element-wise. For negative numbers, square root must be taken on
the absolute value and sign must be kept.
Dkij = ∆ijT ij −1 ij ij
k (Γk ) ∆k ≶ γk
The value of ρ must be adjusted experimentally. ρ = 0.4 was
suggested for 2D tracking.
can be used for checking track association.
21 / 37 22 / 37

Track Association Track Association

Method proposed by [Blackman (1999)] for two local agents
Suppose local agent i and j have NTi and NTj tracks Method proposed by [Blackman (1999)] for two local agents
respectively. Define the quantities
A track association hypothesis θk between local agents i and βT : target density (number of targets/state-space volume)
Pi∈j : probability that local agent i has a track in the common field of
j can be represented as a NTi × NTj -size binary matrix
view with local agent j given that there is a target there.
Z = [zmn ∈ {0, 1}] such that βFi T : False track density of the tracker of local agent i (same unit as
 βT ).
1, If track
 m of local agent i
is associated with Then the probability of a track association hypothesis is given by
zmn = track n of local agent j

0, otherwise j
i j
Y
i NN NN
P (θk ) ∝ (βN A)
A (β
N A)
A βT Pi∈j Pj∈i N (x̂m n mn
k|k − x̂k|k ; 0, Γk )
{m,n|zmn =1}
Note that the constraints
i j
NT
X NT
X where
i i j j
zmn ≤ 1 ∀n and zmn ≤ 1 ∀m βN A = βT Pi∈j (1 − Pj∈i ) + βF T and βN A = βT Pj∈i (1 − Pi∈j ) + βF T
PNTi PNT j PNTi PNT j
m=1 n=1 i i j j
NN A , NT − m=1 n=1 zmn and NN A , NT − m=1 n=1 zmn

must be satisfied for a valid track association hypothesis.

23 / 37 24 / 37
 Track Association Track Association: Assignment Problem

Method proposed by [Blackman (1999)] for two local agents Method proposed by [Blackman (1999)] for two local agents
j
i
j βT Pi∈j Pj∈i N (x̂m n mn
k|k − x̂k|k ; 0, Γk )
Y
i NN A NN A
P (θk ) ∝ (βN A) (βN A) βT Pi∈j Pj∈i N (x̂m n mn
k|k − x̂k|k ; 0, Γk ) i i
X
log P (θk ) = NN A log βN A + log j
+C
{m,n|zmn =1} βN
{m,n|zmn =1} A
j j
Divide by the constant (βN NT
A) Form the assignment matrix:

i
Y βT Pi∈j Pj∈i N (x̂m n mn
k|k − x̂k|k ; 0, Γk ) Aij T1j T2j T3j T4j NA1 NA2 NA3
i NN A
P (θk ) ∝ (βN A) j
βN T1i `11 `12 `13 `14 i
log βN A × ×
{m,n|zmn =1} A
T2i `21 `22 `23 `24 × i
log βN A ×
Maximizing this probability is equivalent to maximizing log of it. T3i `31 `32 `33 `34 × × i
log βN A

βT Pi∈j Pj∈i N (x̂m −x̂n ;0,Γmn

k )
X βT Pi∈j Pj∈i N (x̂m n mn
k|k − x̂k|k ; 0, Γk ) where `mn , log k|k k|k
.
i i j
log P (θk ) = NN A log βN A + log j
+C βN A

{m,n|zmn =1}
βN A Then, use auction(Aij ) to get track association decisions.

25 / 37 26 / 37

Track Association Track Fusion: Independence Assumption

Track association for more than two local agents. Once we associate two tracks, we have to fuse them to obtain a
One way is to solve multi dimensional assignment problem. fused track. This is called as track fusion.
The simpler way is to do the so-called sequential pairwise Consider the track fusion at point A assuming tCR = tCT = t.
track association.
Suppose we have NL local agents whose tracks need to be fused. Independence assumption gives
Then, we order the local agents according to some criteria e.g. −1 −1 −1
accuracy, priority, etc. (ΣA
t ) =(ΣB
t ) + (ΣC
t )
−1 A B −1 B C −1 C
(ΣA
t ) x̂t =(Σt ) x̂t + (Σt ) x̂t
Tracks of Tracks of Tracks of
Local Agent 2 Local Agent 3 Local Agent NL
This is simplistic and expected to
give very bad results here.
Track Track Track
Tracks of Final
Association & Association & Association &
Local Agent 1
Track Fusion Track Fusion Track Fusion
Fused Tracks This is also called as naive fusion.

27 / 37 28 / 37
Track Fusion: Optimal Solution Track Fusion: Channel Filter
Consider the track fusion at point A assuming tCR = tCT = t.
Consider the track fusion at point A assuming tCR = tCT = t. Channel filter, equivalent measurements, tracklets etc.
Optimal solution −1 −1 −1 −1
(ΣA
t ) =(ΣB
t ) + (ΣC
t ) − (ΣD
t|t−td )
(ΣA −1
= I − (ΣC −1 CB
∆−1 B −1
∆−1 BC



t ) t ) Σt B + I − (Σt ) Σt
−1 A B −1 B C −1 C −1 D
C (ΣA D
t ) x̂t =(Σt ) x̂t + (Σt ) x̂t − (Σt|t−td ) x̂t|t−td
∆−1
−1 A −1 B −1 BC

−1 C
(ΣA (ΣC CB B


t ) x̂t = I− t ) Σt B tx̂ + I − (Σt ) Σ t ∆C x̂t
One can define ẑtC and ZtC , which are called equivalent
where measurements or tracklets in the literature, as
C −1 CB B −1 BC
∆B , ΣB BC
t − Σt (Σt ) Σt ∆C , ΣC CB
t − Σt (Σt ) Σt (ZtC )−1 ,(ΣC
t )
−1
− (ΣD
t|t−td )
−1

(ZtC )−1 ẑtC ,(ΣC −1 C D −1 D

t ) x̂t − (Σt|t−td ) x̂t|t−td
This is very difficult to compute in
a scalable way for variable networks Transmitting these quantities
(no fixed-structure). instead from a local agent, one can
use the independent track fusion
formulas.
29 / 37 30 / 37

Track Fusion Track Fusion: LEA Algorithm

Illustration of Correlation Independent Schemes. Largest ellipsoid algorithm or safe fusion: Suppose we have local
estimates x̂B B C C
t , Σt and x̂t , Σt .
−1
z T (ΣB
t ) z=1 Find SVD of ΣB T
t = U1 Λ 1 U1
−1/2
Define the transformation T1 = Λ1 U1T
Transform ΣC C T
t with T1 and define PC = T1 Σt T1 .
−1
z T (ΣC
t ) z=1
Find SVD of PC = U2 Λ2 U2T .
Define the transformation T2 = U2T T1
Transform x̂B B C C
t , Σt and x̂t , Σt with T2 .

ẑtB = T2 x̂B
t and ẑtC = T2 x̂C
t
−1
z T (ΣA
t ) z=1
Largest Ellipsoid
Algorithm ZtB = T2 ΣB T
t T2 = Inx and ZtC = T2 ΣC T
t T2 = Λ2
Covariance Intersection
...

31 / 37 32 / 37
Track Fusion: LEA Algorithm Track Fusion: CI

Covariance intersection
Largest ellipsoid algorithm continued:
Define
...
−1 −1 −1
Define set of indices I = {i|1 ≤ i ≤ nx , [Λ2 ]ii < 1} (ΣA
t (w)) = w(ΣB
t ) + (1 − w)(ΣC
t )
Find vector ẑtA and covariance ZtA as
Find

B i = j, i 6∈ I
[Zt ]ii w∗ = arg min |ΣA
(
[ẑtB ]i t (w)|

i 6∈ I
[ẑtA ]i , , [ZtA ]ij , [ZtC ]ii i = j, i ∈ I w∈[0,1]
[ẑtC ]i i∈I 
0 i 6= j

using optimization.
Find fused estimate and covariance Then the fused estimate and covariance are given as
−1
x̂A −1 A
ΣA −1 A −T (ΣA
t ) =w∗ (ΣB
t )
−1
+ (1 − w∗ )(ΣC
t )
−1
t =T2 ẑt t =T2 Zt T2 .
−1 A ∗ B −1 B ∗ C −1 C
(ΣA
t ) x̂t =w (Σt ) x̂t + (1 − w )(Σt ) x̂T

33 / 37 34 / 37

Track Fusion References

According to the recent work (recommended)

K. C. Chang, Chee-Yee Chong and S. Mori, “On scalable M. E. Liggins II, Chee-Yee Chong, I. Kadar, M. G. Alford, V. Vannicola and
V. Thomopoulos, “Distributed fusion architectures and algorithms for target
distributed sensor fusion,” Proceedings of 11th
tracking,” Proceedings of the IEEE, vol.85, no.1, pp. 95-107, Jan. 1997.
International Conference on Information Fusion, Jul.
2008. M. E. Liggins and Kuo-Chu Chang, “Distributed fusion architectures,
algorithms, and performance within a network-centric architecture,” Ch.17,
channel filter seems to be the best algorithm for track fusion in Handbook of Multisensor Data Fusion: Theory and Practice, Taylor & Francis,
terms of Second Edition, 2009.
scalability; K. C. Chang, Chee-Yee Chong and S. Mori, “On scalable distributed sensor
estimation errors; fusion,” Proceedings of 11th International Conference on Information Fusion,
Jul. 2008.
and memory.

35 / 37 36 / 37
References

S. Blackman and R. Popoli, Design and Analysis of Modern Tracking Systems.

Norwood, MA: Artech House, 1999.
Y. Bar-Shalom and X. R. Li, Multitarget-Multisensor Tracking: Principles,
Techniques. Storrs, CT: YBS Publishing, 1995.

37 / 37