Professional Documents
Culture Documents
Ba-Ngu Vo http://ba-ngu.vo-au.com
ISPL
Intelligent Sensing & Perception Lab
https://ispl-research.com/ IEEE Signal Processing Society, 2022
Introduction
1950’s-1960’s: Kalman filter - Bode & Shannon, Zadeh & Ragazzini, Levinson,
Swerling, Stratonovich, Kalman ... Schmidt’s Implementation – Apollo program
1990’s--: Particle Filter for non-linear filtering - Gordon, Salmond & Smith
State Estimation
Multi-object State Space Model
PHD filter
Beyond PHD filtering
Labelled Random Finite Set
Generalized Labelled Multi-Bernoulli Filter
Some Demonstrations
State Estimation
State Space Model of a Dynamical System
observation space ℤ
zk
zk-1
observation vector
state space 𝕏
State Estimation
state transition
xk System ID
xk-1 state vector Control
𝑥0 𝑥1 𝑥2 𝑥3 𝑥4 𝑥𝑘
State space
0 1 2 3 4 … k time
Complexity per time step increases with time - not suitable for online
Smooth over a fix-length moving window - fixed complexity per time step
Fixed complexity per time step - suitable for online, widely used
Multi-Object System
What if the state & observation are not vectors but finite sets?
observation space ℤ
observations of the
states
state space 𝕏
state transition
Xk
3 vectors
5 vectors
Xk-1
Theoretical considerations
▪ Can’t treat a (random) finite set as if it were a (random) vector
▪ PDF of random finite set is not the same as PDF of random vector
▪ Mahler’s Finite Set Statistics (FISST) provides a suitable notion of density & integration
Observation equation
likelihood
x z
Zk = Qk(Xk) Kk
misdetection
x
gk(Zk|Xk)
clutter
Observation likelihood
x0 state space 𝕏
S
Probability Hypothesis Density Filter
PHD filter: Propagates PHD forward in time
state space
PHD PHD
vk-1(xk-1|Z1:k-1) prediction
vk|k-1(xk|Z1:k-1) update
vk(xk|Z1:k)
R. Mahler, “Multitarget Bayes filtering via first-order multitarget moments,” IEEE Trans. AES, 39(4):1152–1178, 2003
▪ Approximate by a Poisson RFS density that minimizes KL-div from multi-object filtering density
PHD PHD
vk-1(xk-1|Z1:k-1) prediction vk|k-1(xk|Z1:k-1) update
vk(xk|Z1:k)
R. Mahler, “PHD filters of higher order in target number,” IEEE Trans. AES,43(4):1523–1543, 2007
vk-1 vk
▪ Approximate by an IID cluster density that minimizes KL-div from multi-object filtering density
▪ Gaussian mixture implementation (cubic in no. detections)
Vo et. al. “Analytic implementations of the Cardinalized PHD Filter," IEEE Trans. SP, 55(7-II): 3553-3567, 2007
Beyond PHD Filtering
Recall: Multi-object Bayes filter
prediction
pk-1(Xk-1|Z1:k-1) pk|k-1(Xk|Z1:k-1) data-update pk(Xk|Z1:k)
e.g. measurement ={z1}, prior = p({x1, x2}), detection probability: PD(x1) = 0.9, PD(x2) = 0.1.
𝑝({𝑥1 , 𝑥2 }|{𝑧1 }) ∝ 92𝑔(𝑧1 |𝑥1 )𝑝({𝑥1 , 𝑥2 }) + 𝑔(𝑧1 |𝑥2 )𝑝({𝑥1 , 𝑥2 }) + 9𝜅(𝑧1 )𝑝({𝑥1 , 𝑥2 })
filtering density x1 detected & x2 missed x1 missed & x2 detected x1 & x2 missed
Suppose we truncate the last 2 terms: 𝑓(𝑥1 , 𝑥2 |{𝑧1 }) ≜ 𝑔(𝑧1 |𝑥1 )𝑝({𝑥1 , 𝑥2 })
𝑓(𝑎, 𝑏|{𝑧1 }) = 𝑔(𝑧1 |𝑎)𝑝({𝑎, 𝑏}) ≠ 𝑓(𝑏, 𝑎|{𝑧1 }) = 𝑔(𝑧1 |𝑏)𝑝({𝑎, 𝑏})
f is not symmetric in its arguments, i.e. not a function of the set {a,b}
▪ Even if the truncated sum is a function of sets, what’s the approximation error?
Beyond PHD Filtering
𝑋0 𝑋1 𝑋2 𝑋3 𝑋4 𝑋𝑘
State space
𝑋0:𝑘
?
multi-object trajectory
trajectories
0 1 2 3 4 … k
Labeled Random Finite Set
multi-object trajectory = history of labeled multi-object states
X1 X4 Xk
State space
X 0:k
labeled
trajectories
0 1 2 3 4 … k time
labeled multi-object state X = {x1 xj xn} multi-object state augmented with distinct labels
Goodman et. al. Mathematics of Data Fusion. probability density of point processes, 1997
Mahler, Statistical Multisource-Multitarget Information Fusion, Artech House, 2007
Labeled Random Finite Set
multi-object trajectory = history of labeled multi-object states
X1 X4 Xk
State space
X 0:k
labeled
trajectories
0 1 2 3 4 … k time
Labeled RFS - 2 birds with one stone: provides trajectories & closure under truncation
Vo & Vo "Labeled RFSs and Multi-Object Conjugate Priors," IEEE Trans. SP, 61(13): 3460-3475, 2013
Labeled Random Finite Set
What about set of unlabeled trajectories?
𝑘
Trajectory space 𝕋 = ⨄𝑡=1 ℕ × 𝕏𝑡 , unlabeled multi-object trajectory = finite subset of 𝕋
e.g. a
3 trajectory
distinct elements from the trajectory space
1 1 1 k = 1, 3 objects, multi-object state = 1.0 Physically nonsensical:
,
1.0 1.0 , 1.0 counts different segments of 1 trajectory, as separate trajectories
2.0 2.0 k = 2, 2 objects, multi-object state = 2.0
inconsistent cardinalities with multi-object state history
3.0 k = 3, 1 object, multi-object state = 3.0
Numerically Infeasible:
▪ Need to estimate trajectories over entire duration, complexity per time step explodes with time
▪ Can’t estimate trajectories over shorter window – no means to connect trajectories from different windows
▪ Necessary to truncate posterior, but no guarantee truncated “density” is a function of sets
▪ PHD approximation? complexity per time step still grows with time – won’t work in real applications.
Labeled Random Finite Set
Recall: PHD = instantaneous expected no. objects per unit (hyper) volume
PHD (intensity function) of an RFS
PHD peaks: highest expected No. points per unit volume,
most likely locations of objects
state space 𝕏
𝑘
▪ “PHD” on trajectory space 𝕋 = ⨄𝑡=1 ℕ × 𝕏𝑡 has different units of measurements for different 𝑘
▪ How can we locate the n* highest peaks if they have different units?
Generalized Labeled Multi-Bernoulli (GLMB) Filter
GLMB filter: Multi-object Analogue of Kalman Smoother/Filter
Closure under Bayes recursion - Analytic solutions to:
▪ Multi-object Bayes Posterior Recursion (Smoothing-while-filtering)
Scenario
▪ surveillance region: 64km by 36km
▪ objects can appear anywhere, appearing rate ~ 200-3000/frame (unknown to filter)
▪ mean clutter density 200 km-2 (460800 per frame) uniform,
▪ detection probability 0.88
▪ linear Gaussian state & observation noise sigma of 0.2ms-2 & 5m
M. Beard et. al. "A Solution for Large-scale Multi-object Tracking," IEEE Trans. SP, 68:2754–2769, 2020.
Generalized Labeled Multi-Bernoulli (GLMB) Filter
labels of X p ( x)
xX
( )
▪ Weight Normalization
B.-T. Vo, et. al. "Labeled Random Finite Sets and Multi-Object Conjugate Priors," IEEE Trans. SP, 61(13): 3460-3475, 2013.
Generalized Labeled Multi-Bernoulli (GLMB) Filter
Labeled Multi-Object Density Approximations
Truncation of GLMB
▪ Any truncation of a GLMB is a GLMB – closure under truncation
▪ L1-norm of truncation error can be computed analytically
▪ Minimum L1-norm truncation error: truncate smallest weights
Vo et. al. "Labeled RFS and the Bayes Multi-Target Tracking Filter," IEEE Trans. SP, 62(24):6554-6567, 2014
Filter implementation:
▪ K-shortest path prediction & ranked assignment update (cubic in no. detections)
▪ Gibbs sampling + Joint Prediction & Update (linear in no. detections)
Vo et. al. "Labeled RFS and the Bayes Multi-Target Tracking Filter," IEEE Trans. SP, 62(24):6554-6567, 2014.
Vo et. al. “An Efficient Implementation of the GLMB Filter,” IEEE Trans. SP, 65(8):1975-1987, 2017.
Smoother implementation:
▪ Multi-dimensional assignment (NP-Hard): Gibbs sampling (linear in total no. of time steps)
▪ On-line: smooth over fixed-length window and link trajectories with the same labels
Vo & Vo “A Multi-Scan Labeled RFS Model for Multi-object State Estimation,” IEEE Trans. SP, 67(19):4948-4963, 2019
Demonstration: Multi-object Smoothing
Cell-microscopy – posterior multi-scan statistics
▪ Duration 1000mins, sampling period D=10mins
▪ Clutter 0.3/scan, Scenario:
▪ Detection probability 0.33 (very low) ▪ min 001: 4 cells appear, live for 100mins
▪ Observation noise sigma 0.3mm ▪ min 200: 4 cells appear, live for 200mins
▪ Dynamic noise sigma 0.01mm/D2 ▪ min 500: 4 cells appear, live for 400mins
Vo & Vo “A Multi-Scan Labeled RFS Model for Multi-object State Estimation,” IEEE Trans. SP, 67(19):4948-4963, 2019
Demonstration: Multi-Sensor Fusion
3D (state + extent) tracking from multiple camera views
▪ 4 Kinect sensors placed high up and facing inwards in each corner
▪ YOLO detector produces 2D detections of bounding boxes in pixel space
▪ Detections are noisy and subject to false positives/negatives
Ong et. al. “A Bayesian Filter for Multi-view 3D Multi-object Tracking with Occlusion Handling,” IEEE Trans. PAMI, 2020.
Demonstration: Multi-Sensor Fusion
Tracking output from fused and filtered detections (colours denote estimated identities)
Ong et. al. “A Bayesian Filter for Multi-view 3D Multi-object Tracking with Occlusion Handling,” IEEE Trans. PAMI, 2020.
Demonstration: Multi-Sensor Fusion
Tracking multiple jumping/falling people in 3-D
Ong et. al. “A Bayesian Filter for Multi-view 3D Multi-object Tracking with Occlusion Handling,” IEEE Trans. PAMI, 2020.
Demonstration: Multi-Sensor Fusion
Provides tracks in 3D instead of ground-plane tracks as in existing methods
No retraining when reconfiguring cameras
Ong et. al. “A Bayesian Filter for Multi-view 3D Multi-object Tracking with Occlusion Handling,” IEEE Trans. PAMI, 2020.
Demonstration: Autonomous Driving
Autonomous Driving: SLAM + multi-object filtering
H. Deusch et. al. “The Labeled Multi-Bernoulli SLAM filter,” IEEE Signal Proc. Letters, 22(10), 2015.
Demonstration: Autonomous Driving
Autonomous Driving: SLAM + multi-object filtering
Thank You!