A Tutorial On Compressive Sensing

A Tutorial on Compressive Sensing
Simon Foucart
Drexel University / University of Georgia
CIMPA13
New Trends in Applied Harmonic Analysis
Mar del Plata, Argentina, 5-16 August 2013
This minicourse acts as an invitation to the elegant theory of
Compressive Sensing. It aims at giving a solid overview of the
fundamental mathematical aspects. Its content is partly based
on a recent textbook coauthored with Holger Rauhut:
Part 1: The Standard Problem and First Algorithms
The lecture introduces the question of sparse recovery, establishes

its theoretical limits, and presents an algorithm achieving these
limits in an idealized situation. In order to treat more realistic
situations, other algorithms are necessary. Basis Pursuit (that is,
`1 -minimization) and Orthogonal Matching Pursuit make their first
appearance. Their success is proved using the concept of
coherence of a matrix.
Keywords
Keywords
I Sparsity
Essential
Keywords
I Sparsity
Essential
I Randomness
Nothing better so far
(measurement process)
Keywords
I Sparsity
Essential
I Randomness
Nothing better so far
(measurement process)
I Optimization
Preferred, but competitive alternatives are available
(reconstruction process)
The Standard Compressive Sensing Problem
x : unknown signal of interest in KN


y : measurement vector in Km

y : measurement vector in Km with m N,

s : sparsity of x


s : sparsity of x = card j ∈ {1, . . . , N} : xj 6= 0 .


Find concrete sensing/recovery protocols,


Find concrete sensing/recovery protocols, i.e., find
I measurement matrices A : x ∈ KN 7→ y ∈ Km


I measurement matrices A : x ∈ KN →
7 y ∈ Km
I m
reconstruction maps ∆ : y ∈ K → 7 x ∈ KN


7 y ∈ Km
I m
such that
∆(Ax) = x for any s-sparse vector x ∈ KN .


7 y ∈ Km
I m
such that
In realistic situations, two issues to consider:


7 y ∈ Km
I m
such that
In realistic situations, two issues to consider:
Stability: x not sparse but compressible,
Robustness: measurement error in y = Ax + e.
A Selection of Applications
I Magnetic resonance imaging
Figure: Left: traditional MRI reconstruction; Right: compressive

sensing reconstruction (courtesy of M. Lustig and S. Vasanawala)
I Sampling theory
ï2
ï4
ï6
ï0.5 ï0.4 ï0.3 ï0.2 ï0.1 0 0.1 0.2 0.3 0.4 0.5
Figure: Time-domain signal with 16 samples.

I Sampling theory
I Error correction
I Sampling theory
I Error correction
I and many more...
`0 -Minimization
Since
N
X N
X
kxkpp := |xj |p −→ 1{xj 6=0} ,
p→0
j=1 j=1
`0 -Minimization
Since
N
X N
X
kxkpp := |xj |p −→ 1{xj 6=0} ,
p→0
j=1 j=1
the notation kxk0 [sic] has become usual for
kxk0 := card(supp(x)), where supp(x) := {j ∈ [N] : xj 6= 0}.

`0 -Minimization
Since
N
X N
X
kxkpp := |xj |p −→ 1{xj 6=0} ,
p→0
j=1 j=1
For an s-sparse x ∈ KN , observe the equivalence of

`0 -Minimization
Since
N
X N
X
kxkpp := |xj |p −→ 1{xj 6=0} ,
p→0
j=1 j=1

I x is the unique s-sparse solution of Az = y with y = Ax,
`0 -Minimization
Since
N
X N
X
kxkpp := |xj |p −→ 1{xj 6=0} ,
p→0
j=1 j=1

I x can be reconstructed as the unique solution of
(P0 ) minimize kzk0 subject to Az = y.

z∈KN
`0 -Minimization
Since
N
X N
X
kxkpp := |xj |p −→ 1{xj 6=0} ,
p→0
j=1 j=1

I x can be reconstructed as the unique solution of

z∈KN
This is a combinatorial problem, NP-hard in general.

Minimal Number of Measurements
Given A ∈ Km×N , the following are equivalent:

1. Every s-sparse x is the unique s-sparse solution of Az = Ax,


2. ker A ∩ {z ∈ KN : kzk0 ≤ 2s} = {0},

2. ker A ∩ {z ∈ KN : kzk0 ≤ 2s} = {0},
3. For any S ⊂ [N] with card(S) ≤ 2s, the matrix AS is injective,

2. ker A ∩ {z ∈ KN : kzk0 ≤ 2s} = {0},
4. Every set of 2s columns of A is linearly independent.

2. ker A ∩ {z ∈ KN : kzk0 ≤ 2s} = {0},
As a consequence, exact recovery of every s-sparse vector forces
m ≥ 2s.

2. ker A ∩ {z ∈ KN : kzk0 ≤ 2s} = {0},
As a consequence, exact recovery of every s-sparse vector forces
m ≥ 2s.
This can be achieved using partial Vandermonde matrices.

Exact s-Sparse Recovery from 2s Fourier Measurements
Identify an s-sparse x ∈ CN with a function x on {0, 1, . . . , N − 1}
with support S, card(S) = s.
with support S, card(S) = s. Consider the 2s Fourier coefficients
N−1
X
x̂(j) = x(k)e −i2πjk/N , 0 ≤ j ≤ 2s − 1.
k=0
N−1
X
x̂(j) = x(k)e −i2πjk/N , 0 ≤ j ≤ 2s − 1.
k=0
Consider a trigonometric polynomial vanishing exactly on S, i.e.,

Y
1 − e −i2πk/N e i2πt/N .

p(t) :=
k∈S
N−1
X
x̂(j) = x(k)e −i2πjk/N , 0 ≤ j ≤ 2s − 1.
k=0

Y

p(t) :=
k∈S
Since p · x ≡ 0, discrete convolution gives

N−1
X
0 = (p̂ ∗ x̂)(j) = p̂(k)x̂(j − k), 0 ≤ j ≤ N − 1.
k=0
N−1
X
x̂(j) = x(k)e −i2πjk/N , 0 ≤ j ≤ 2s − 1.
k=0

Y

p(t) :=
k∈S

N−1
X
0 = (p̂ ∗ x̂)(j) = p̂(k)x̂(j − k), 0 ≤ j ≤ N − 1.
k=0
Note that p̂(0) = 1 and that p̂(k) = 0 for k > s.

N−1
X
x̂(j) = x(k)e −i2πjk/N , 0 ≤ j ≤ 2s − 1.
k=0

Y

p(t) :=
k∈S

N−1
X
0 = (p̂ ∗ x̂)(j) = p̂(k)x̂(j − k), 0 ≤ j ≤ N − 1.
k=0
Note that p̂(0) = 1 and that p̂(k) = 0 for k > s. The equations
s, . . . , 2s − 1 translate into a Toeplitz system with unknowns
p̂(1), . . . , p̂(s).
N−1
X
x̂(j) = x(k)e −i2πjk/N , 0 ≤ j ≤ 2s − 1.
k=0

Y

p(t) :=
k∈S

N−1
X
0 = (p̂ ∗ x̂)(j) = p̂(k)x̂(j − k), 0 ≤ j ≤ N − 1.
k=0
Note that p̂(0) = 1 and that p̂(k) = 0 for k > s. The equations
s, . . . , 2s − 1 translate into a Toeplitz system with unknowns
p̂(1), . . . , p̂(s). This determines p̂, hence p, then S, and finally x.
Optimization and Greedy Strategies
`1 -Minimization (Basis Pursuit)
Replace (P0 ) by

z∈KN
Replace (P0 ) by

z∈KN
I Geometric intuition
Replace (P0 ) by

z∈KN
I Unique `1 -minimizers are at most m-sparse (when K = R)
Replace (P0 ) by

z∈KN
I Convex optimization program, hence solvable in practice
Replace (P0 ) by

z∈KN
I In the real setting, recast as the linear optimization program
N
X
minimize cj subject to Az = y and − cj ≤ zj ≤ cj .
c,z∈RN
j=1
Replace (P0 ) by

z∈KN
I In the real setting, recast as the linear optimization program
N
X
minimize cj subject to Az = y and − cj ≤ zj ≤ cj .
c,z∈RN
j=1
I In the complex setting, recast as a second order cone program

Basis Pursuit — Null Space Property
∆1 (Ax) = x for every vector x supported on S if and only if

(NSP) kuS k1 < kuS k1 , all u ∈ ker A \ {0}.

For real measurement matrices,

For real measurement matrices, real and complex NSPs read


X X
|uj | < |u` |, all u ∈ kerR A \ {0},
j∈S `∈S

X X
|uj | < |u` |, all u ∈ kerR A \ {0},
j∈S `∈S
Xq Xq
2 2
vj + wj < v`2 + w`2 , all (v, w) ∈ (kerR A)2 \ {0}.
j∈S `∈S

X X
|uj | < |u` |, all u ∈ kerR A \ {0},
j∈S `∈S
Xq Xq
2 2
vj + wj < v`2 + w`2 , all (v, w) ∈ (kerR A)2 \ {0}.
j∈S `∈S
Real and complex NSPs are in fact equivalent.

Orthogonal Matching Pursuit
Starting with S 0 = ∅ and x0 = 0, iterate
(OMP1 ) S n+1 = S n ∪ j n+1 := argmax |(A∗ (y − Axn ))j | ,

j∈[N]
n+1
= argmin ky − Azk2 , supp(z) ⊆ S n+1 .

(OMP2 ) x
z∈CN

j∈[N]
n+1

(OMP2 ) x
z∈CN
I The norm of the residual decreases according to
2
ky − Axn+1 k22 ≤ ky − Axn k22 − (A∗ (y − Axn ))j n+1 .


j∈[N]
n+1

(OMP2 ) x
z∈CN
I The norm of the residual decreases according to
2
ky − Axn+1 k22 ≤ ky − Axn k22 − (A∗ (y − Axn ))j n+1 .

I Every vector x6=0 supported on S, card(S) = s, is recovered

from y = Ax after at most s iterations of OMP if and only if
AS is injective and
(ERC) max |(A∗ r)j | > max |(A∗ r)` |

j∈S `∈S

for all r6=0 ∈ Az, supp(z) ⊆ S .
First Recovery Guarantees
Coherence
Coherence
For a matrix with `2 -normalized columns a1 , . . . , aN , define
µ := max |hai , aj i| .
i6=j
Coherence
i6=j
As a rule, the smaller the coherence, the better.

Coherence
i6=j

However, the Welch bound reads
s
N −m
µ≥ .
m(N − 1)
Coherence
i6=j

s
N −m
µ≥ .
m(N − 1)
I Welch bound achieved at and only at equiangular tight frames

Coherence
i6=j

s
N −m
µ≥ .
m(N − 1)
I Welch bound achieved at and only at equiangular tight frames

√
I Deterministic matrices with coherence µ ≤ c/ m exist
Recovery Conditions using Coherence
I Every s-sparse x ∈ CN is recovered from y = Ax ∈ Cm via at

most s iterations of OMP provided
1
µ< .
2s − 1

1
µ< .
2s − 1
I Every s-sparse x ∈ CN is recovered from y = Ax ∈ Cm via
Basis Pursuit provided
1
µ< .
2s − 1

1
µ< .
2s − 1
I Every s-sparse x ∈ CN is recovered from y = Ax ∈ Cm via
Basis Pursuit provided
1
µ< .
2s − 1
I In fact, the Exact Recovery Condition can be rephrased as
kA†S AS k1→1 < 1,
and this implies the Null Space Property.

Summary
Summary
The coherence conditions for s-sparse recovery are of the type

c
µ≤ ,
s
Summary

c
µ≤ ,
s
while the Welch bound (for N ≥ 2m, say) reads
c
µ≥ √ .
m
Summary

c
µ≤ ,
s
c
µ≥ √ .
m
Therefore, arguments based on coherence necessitate
m ≥ c s 2.
Summary

c
µ≤ ,
s
c
µ≥ √ .
m
m ≥ c s 2.
This is far from the ideal linear scaling of m in s...

Summary

c
µ≤ ,
s
c
µ≥ √ .
m
m ≥ c s 2.
This is far from the ideal linear scaling of m in s...

Next, we will introduce new tools to break this quadratic barrier
(with random matrices).

A Tutorial On Compressive Sensing

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

A Tutorial On Compressive Sensing

Uploaded by

Copyright:

Available Formats

A Tutorial on Compressive Sensing

The lecture introduces the question of sparse recovery, establishes

x : unknown signal of interest in KN

x : unknown signal of interest in KN

x : unknown signal of interest in KN

x : unknown signal of interest in KN

x : unknown signal of interest in KN

x : unknown signal of interest in KN

x : unknown signal of interest in KN

x : unknown signal of interest in KN

x : unknown signal of interest in KN

x : unknown signal of interest in KN

x : unknown signal of interest in KN

Figure: Left: traditional MRI reconstruction; Right: compressive

Figure: Time-domain signal with 16 samples.

the notation kxk0 [sic] has become usual for

kxk0 := card(supp(x)), where supp(x) := {j ∈ [N] : xj 6= 0}.

the notation kxk0 [sic] has become usual for

kxk0 := card(supp(x)), where supp(x) := {j ∈ [N] : xj 6= 0}.

For an s-sparse x ∈ KN , observe the equivalence of

the notation kxk0 [sic] has become usual for

kxk0 := card(supp(x)), where supp(x) := {j ∈ [N] : xj 6= 0}.

For an s-sparse x ∈ KN , observe the equivalence of

the notation kxk0 [sic] has become usual for

kxk0 := card(supp(x)), where supp(x) := {j ∈ [N] : xj 6= 0}.

For an s-sparse x ∈ KN , observe the equivalence of

(P0 ) minimize kzk0 subject to Az = y.

the notation kxk0 [sic] has become usual for

kxk0 := card(supp(x)), where supp(x) := {j ∈ [N] : xj 6= 0}.

For an s-sparse x ∈ KN , observe the equivalence of

(P0 ) minimize kzk0 subject to Az = y.

This is a combinatorial problem, NP-hard in general.

Given A ∈ Km×N , the following are equivalent:

Given A ∈ Km×N , the following are equivalent:

1. Every s-sparse x is the unique s-sparse solution of Az = Ax,

Given A ∈ Km×N , the following are equivalent:

1. Every s-sparse x is the unique s-sparse solution of Az = Ax,

Given A ∈ Km×N , the following are equivalent:

1. Every s-sparse x is the unique s-sparse solution of Az = Ax,

Given A ∈ Km×N , the following are equivalent:

1. Every s-sparse x is the unique s-sparse solution of Az = Ax,

Given A ∈ Km×N , the following are equivalent:

1. Every s-sparse x is the unique s-sparse solution of Az = Ax,

As a consequence, exact recovery of every s-sparse vector forces

Given A ∈ Km×N , the following are equivalent:

1. Every s-sparse x is the unique s-sparse solution of Az = Ax,

As a consequence, exact recovery of every s-sparse vector forces

This can be achieved using partial Vandermonde matrices.

Consider a trigonometric polynomial vanishing exactly on S, i.e.,

Consider a trigonometric polynomial vanishing exactly on S, i.e.,

Since p · x ≡ 0, discrete convolution gives

Consider a trigonometric polynomial vanishing exactly on S, i.e.,

Since p · x ≡ 0, discrete convolution gives

Note that p̂(0) = 1 and that p̂(k) = 0 for k > s.

Consider a trigonometric polynomial vanishing exactly on S, i.e.,

Since p · x ≡ 0, discrete convolution gives

Consider a trigonometric polynomial vanishing exactly on S, i.e.,

Since p · x ≡ 0, discrete convolution gives

(P1 ) minimize kzk1 subject to Az = y.

(P1 ) minimize kzk1 subject to Az = y.

(P1 ) minimize kzk1 subject to Az = y.

(P1 ) minimize kzk1 subject to Az = y.

(P1 ) minimize kzk1 subject to Az = y.