Spectral Compressive Sensing

Marco F. Duarte

Portions are joint work with:

Richard G. Baraniuk
(Rice University)

Hamid Dadkhahi
(UMass Amherst)

Karsten Fyhn
(Aalborg University)

Spectral Compressive Sensing

Compressive sensing applied to frequency-sparse signals

linear
measurements

frequency-sparse
signal

Fourier components

[E. Cands, J. Romberg, T. Tao; D. Donoho]

Spectral Compressive Sensing

Compressive sensing applied to frequency-sparse signals

=
frequency-sparse
signal

linear
measurements

nonzero
DFT coefficients

DFT Basis for

frequency-sparse signals

Frequency-Sparse Signals
and the DFT Basis
1 h j2f /N j22f /N
e(f ) = p
e
e
. . . ej2(N
N

1)f /N

Normalized approx. error

Frequency-Sparse Signals
and the DFT Basis
1

Integral frequencies
Arbitrary frequencies

0.8
0.6
0.4
0.2
0
0

10

20
30
40
Approximation sparsity K

Signal is sum of 10 sinusoids

50

Compressive Sensing for

Frequency-Sparse Signals
60

Average SNR, dB

50

Root MUSIC on M signal samples

Standard IHT via DFT (Best)

40
30
20
10
0
100

200
300
400
Number of measurements M

500

Compressive Sensing for

Frequency-Sparse Signals
60

Average SNR, dB

50

Root MUSIC on M signal samples

Standard IHT via DFT (Average)
Standard IHT via DFT (Best/Worst)

40
30
20
10
0
100

200
300
400
Number of measurements M

500

The Redundant DFT Frame

The Redundant DFT Frame

, c = 10

The Redundant DFT Frame

, c = 10

The Redundant DFT Frame

, c = 10

Recovery
1
algorithms
operate similarly
0.5
to matched
filtering:
( )

Dirichlet Kernel

0
0
1

0.02

0.04

0.06

0.08

0.1

0.12

The Redundant DFT Frame

, c = 10

Sparse approximation
algorithms fail

[Cands, Needell, Eldar, Randall 2011]

Normalized approx. error

Sparse Approximation of
Frequency-Sparse Signals
3

Standard sparse approx. via DFT Basis

Standard sparse approx. via DFT Frame

0
0

10

20
30
40
Approximation sparsity K

50

Signal is sum of 10 sinusoids at arbitrary frequencies

Structured Sparse Signals

A K-sparse signal lives on A K-structured sparse
the collection of K-dim
signal lives on a particular
(reduced) collection of
subspaces aligned with
K-dimensional canonical
coordinate axes
subspaces

RN

RN

[Baraniuk, Cevher, Duarte, Hegde 2010]

Structured Restricted
Isometry Property (SRIP)
Preserve the structure only between sparse signals that
follow the structure model
Random (iid Gaussian, Rademacher) matrix has the SRIP
with high probability if

RN

mK K-dim planes
[Blumensath, Davies; Lu, Do]

RM

x1
x2

Leveraging Structure in Recovery

Many state-of-the-art sparse recovery algorithms
(greedy and optimization solvers) rely on
[Daubechies, Defrise, and DeMol;
thresholding

Nowak, Figueiredo, and Wright;

Tropp and Needell; Blumensath and Davies...]

RN

Thresholding provides the

best approximation of
x within
K
K

Structured Recovery Algorithms

Modify existing approaches (optimization or greedy-based)
to obtain structure-aware recovery algorithms:
replace the thresholding step in IHT, CoSaMP, SP, ... with a
best structured sparse approximation step
that finds the closest point within union of subspaces

RN

Greedy structure-aware recovery

algorithms inherit guarantees
of generic counterparts
(even
though feasible set may be
K
nonconvex)

Structured Frequency-Sparse Signals

R

A K-structured frequencysparse signal x consists of K

sinusoids that are mutually
incoherent:

if

If x is K-structured frequency-sparse, then there exists a

K-sparse vector such that
and the nonzeros
in are spaced apart from each other (band exclusion).

( )

1
0.5
0
0
1

0.02

0.04

0.06

0.08

0.1

0.12

Structured Frequency-Sparse Signals

If x is K-structured frequency-sparse, then there exists a
K-sparse vector such that
and the nonzeros
in are spaced apart from each other.

( )

1
0.5
0
0

0.02

0.04

0.06

0.08

0.1

0.12

( ) > 0.1

1
Preserve the structure
only between sparse signals that
follow the structured sparsity model
0.5

Random (iid Gaussian,

Bernoulli) matrix has the
0
0.02
0.04
0.06
0.08
0.1
structured RIP 0with
high
probability
if 0.12

Structured Sparse Approximation

Algorithm 1:

RN

Integer Program
Inputs:

Signal vector x
Target sparsity K
Redundancy factor c
Maximum coherence

Output:

Approximation vector
Compute coefficients:

( )

1
0.5
0
0

Solve support:
0.02

0.04

0.06

0.08

0.1

0.12

( ) > 0.1

Mask coefficients:

0.5[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 1 1 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
0
0

0.02

0.04

0.06

0.08

0.1

0.12

Return

Recovery with Structured Sparsity

Theorem:
Assume we obtain noisy CS measurements of a
signal
. If has the structured RIP
with
, then the output of the structured
IHT algorithm obeys

CS recovery
error

signal K-term
structured sparse approximation error

In words, instance optimality based on

structured sparse approximation
[Baraniuk, Cevher, Duarte, Hegde 2010]

noise

Structured Sparse Approximation

Algorithm 2:

RN

Inhibition Heuristic
Inputs:

Signal vector x
Target sparsity K
Redundancy factor c
Maximum coherence

Output:

Approximation vector

( )

1
0.5
0
0

0.02

0.04

0.06

0.08

0.1

0.12

Compute coefficients:
Initialize:
While is nonzero and
Find max abs entry
Copy entry
Inhibit coherent entries

( ) > 0.1

1
0.5

Return

of

Structured Sparse Approximation

DFT Frame + Thresholding

equivalent to
Maximum Likelihood Estimate
of amplitudes and frequencies
for frequency-sparse signal
via Periodogram

frequencies
amplitudes

Widely-studied problem:
Line spectral estimation

Structured Sparse Approximation

Algorithm 3:

Line Spectral Estimation

Inputs:
Signal vector x
Target sparsity K
Output:
Parameter estimates
Signal estimate

x
K

MUSIC
Root MUSIC
ESPRIT
PHD
...

Normalized approx. error

Sparse Approximation of
Frequency-Sparse Signals
3

Standard sparse approx. via DFT Basis

Standard sparse approx. via DFT Frame

0
0

10

20
30
40
Approximation sparsity K

50

Signal is sum of 10 sinusoids at arbitrary frequencies

Normalized approx. error

Sparse Approximation of
Frequency-Sparse Signals
3

Standard sparse approx. via DFT Basis

Standard sparse approx. via DFT Frame
Structured sparse approx. via Alg. 1
Structured sparse approx. via Alg. 2
Structured sparse approx. via Alg. 3

0
0

10

20
30
40
Approximation sparsity K

50

Signal is sum of 10 sinusoids at arbitrary frequencies

Structured CS: Performance

60

Average SNR, dB

50

Root MUSIC on M signal samples

Standard IHT via DFT (Average)
Standard IHT via DFT (Best/Worst)

40
30
20
10
0
100

200
300
400
Number of measurements M

500

Structured CS: Performance

60

Average SNR, dB

50
40

SIHT via Alg. 1

SIHT via Alg. 2
SIHT via Root MUSIC
Root MUSIC on M signal samples
Standard IHT via DFT (Average)
Standard IHT via DFT (Best/Worst)

30
20
10
0
100

200
300
400
Number of measurements M

500

From Recovery of Sparse Signals

To Line Spectral Estimation

Mean Freq. Est. Error, Hz

Can read indices of nonzero DFTF coefficients to obtain

frequencies of frequency-sparse signal components
Equivalence: accurate recovery = accurate estimation?
Algorithms: Alg. 3 essentially combines legacy line
spectral estimation with CS recovery algorithms
500

Structured sparsity
Multitaper
Root MUSIC
Standard Sparsity

400
300
200
100
0

100

200
300
400
Number of measurements M

How to change
signal model
to further
improve
performance?

500

Interpolating the Projections

(Dirichlet Kernel)
[m3G; v 1.77; Prn:27/08/2012; 14:35] P.12 (1-19)

M.F. Duarte, R.G. Baraniuk / Appl. Comput. Harmon. Anal. ()

Main lobe of Dirichlet

kernel can be well

approximated by a
quadratic polynomial
(parabola)
Three samples
around peak are
required for
interpolation

d sinusoids frequency response (Dirichlet kernel) in the main lobe by a quadratic function. The Dirichlet kernels maxi75 radians/second. The maximum/peak of the quadratic least-squares fit to three samples of a redundant DFT of the signal
! = 1.9775, i.e., it is an accurate estimate of the sinusoids frequency with precision to four
und its peak is located at

From Discrete to Continuous Models

Both the DFT basis and the DFT frame can be conceived
as samplings from an infinite set of signals e(f)
for a discrete set of values for the frequency

1 h j2f /N j22f /N
e(f ) = p
e
e
. . . ej2(N
N

1)f /N

Since the signal vector e(f) varies smoothly in each entry

as a function of f, we can represent the signal set as a
one-dimensional nonlinear manifold:

N
f

Parameter space

e(3) ...
e(2)
e(1)
e(0)

From Discrete to Continuous Models

For computational reasons, we wish to design methods
that allow us to interpolate the manifold from the
samples obtained in the DFT basis/frame to increase the
resolution of the frequency estimates.
An interpolation-based compressive line spectral
estimation algorithm obtains projection values for sets of
manifold samples and interpolates around peak on the
rest of the manifold to get frequency estimate

N
f

Parameter space

e(3) ...
e(2)
e(1)
e(0)

Interpolating the Manifold:

Polar Interpolation
All points in manifold have
e(3) ...
e(2)
e(1)
e(0)

equal norm; distance b/w

samples is uniform
Manifold must be contained
within unit Euclidean ball
(hypersphere)
Project signal estimates
into hypersphere
Find closest point in
manifold by interpolating
from closest samples with
polar coordinates
Integrate band exclusion to
get Band-Excluding
Interpolating SP (BISP)

Interpolating the Manifold:

Polar Interpolation
In BISP, find closest point
e(f0-1/c)
e(f0)
e(f0+1/c)

in manifold by interpolating
from closest samples with
polar coordinates:

Map back from manifold to

0

N
f

frequency estimates
(parameter space)

Akin to Continuous Basis Pursuit (CBP)

[Ekanadham, Tranchina, and Simoncelli 2011]

2 B0 (S n ), i = i+1

Fig. 1. Frequency estimation performance in noise-less case.

Fig. 1. Frequency estimation performance in noise-less case.

1022

ent(9) and (6)

ng
10
ent

stisti-

10 11
10
nthesis, `1 -analysis,

Average
Average
costcost
in frequency
in frequency
estimation
estimation

Average
Averagecost
costininfrequency
frequencyestimation
estimation

Compressive Line Spectral Estimation:

Sn}
103
Performance
Evaluation ` -analysis
S n(9)
} and (6)
ng
103
1

-analysis
``11-synthesis
`1 -synthesis
SIHT
SIHT
SDP
SDP
BOMP
BOMP
CBP
CBP
BISP
BISP

102
102
101
101
100
100
10 1
10 1
10 2
0
10 2
0

nthesis,
gorithm`[15]
on the
1 -analysis,
gorithm
the
In[15]
theon
BISP
+encies.
1
4
10
+encies.
1
4
the10
BISP
ms withIncoherence
ms with coherence
5
10
15
20
a range of subsam5
10[dB]
15
20
SNR
7
a range
of the
subsamts
to verify
10level
7
SNR [dB]
20
40
60
80
100
10
tsestimation.
to verify the
level
We set
20Fig. 2. Frequency
40
60 performance
80
100case.
estimation
in
noisy
estimation.
We nuset
he
result of the
Number of measurements (M)
Fig. 2.
Frequency
estimation performance
noisy case.
Number
of measurements
(M)Liao in
he result
the nu1.
In theofnoiseless
BOMP
[Fannjiang
and
2012]
N = 100, K = 4,
+1
Noiseless
Noisy
1.
In
the
noiseless
lar interpolation alSDP: Atomic
Norm Minimization
+1
Fig.
1.
Frequency
estimation
performance
in noise-less
case.
c
=
5,
=
0.2
Hz
`1 -analysis
9.5245
8.8222
Noiseless
Noisy
interpolation
alolara given
estimation
[Tang,
Rhaskar,
Shah,
Recht
2012]
Fig. 1. Frequency estimation performance in noise-less case.
-synthesis 9.5245
2.9082
2.7340
`1 -analysis
8.8222
estimation
elo aofgiven
approximation
`SIHT
1 -synthesis

0.2628
2.9082

0.1499
2.7340

sis,
sis,
the
the
ISP
ISP
nce
nce
amamvel
vel
set
set
nu-

Fig. 1. Frequency estimation

in noise-less
case.
Compressive
Lineperformance
Spectral
Estimation:
Fig. 1. Frequency estimation performance in noise-less case.
Performance Evaluation (Noise)

Average
Average
costcost
in frequency
in frequency
estimation
estimation

+1
+1

Number of measurements (M)

Number of measurements (M)

103
103
102
102
101
101
100
100
10 1
10 1
10 2
0
10 2
0

`1 -analysis
-analysis
``11-synthesis
`1 -synthesis
SIHT
SIHT
SDP
SDP
BOMP
BOMP
CBP
CBP
BISP
BISP

5
5

10
SNR10[dB]

N = 100, K = 4, M =SNR
50, [dB]
c = 5,

15
15

20
20

= 0.2 Hz

Fig. 2. Frequency estimation performance in noisy case.

Fig. 2. Frequency estimation performance in noisy case.

15
20
SNR10[dB]
SNR [dB] Estimation:
Compressive Line Spectral
Fig. 2. Frequency
estimation performance
in noisy case.
Computational
Expense
Fig. 2. Frequency estimation performance in noisy case.

Noiseless
`1 -analysis
9.5245
Time
(seconds) Noiseless
-synthesis 9.5245
2.9082
`1 -analysis
0.2628
`SIHT
2.9082
1 -synthesis
SDP
8.2355
SIHT
0.2628
BOMP
0.0141
SDP
8.2355
CBP
46.9645
BOMP
0.0141
BISP
5.4265
CBP
46.9645
BISP
5.4265

Noisy
8.8222
Noisy
2.7340
8.8222
0.1499
2.7340
9.9796
0.1499
0.0101
9.9796
40.3477
0.0101
1.4060
40.3477
1.4060

Table 1. Average computation times in seconds.

Table 1. Average computation times in seconds.

Conclusions
Spectral CS provides significant improvements on
frequency-sparse signal recovery
address coherent dictionaries via structured sparsity
simple-to-implement modifications to recovery algs
can leverage decades of work on spectral estimation
robust to model mismatch, presence of noise

Compressive line spectral estimation:

recovery via parametric dictionaries provides
compressive parameter estimation
dictionary elements as samples from manifold models
from dictionaries to manifolds via interpolation techniques
from recovery to parameter estimation from compressive
measurements
localization, bearing estimation, radar imaging, ...
http://www.ecs.umass.edu/~mduarte

mduarte@ecs.umass.edu