Compressed Sensing

Lecture 4
Recap: Coherence and RIP
Alexander Jung

111

Recap
Consider the SLM z = Ax + w
We already learned some important quantities associated with CS-matrix A:
The spark spark(A): Minimum number of linearly dependent columns
of A; yields necessary and sufficient condition in the noiseless case for
k k0 based recovery.
The RIP constant s : Smallest number s such that
(1 s )kxk22 kAxk22 (1 + s )kxk22 for all x Xs

RIP yields sufficient conditions for successful recovery in the noisy case.

 T

The coherence (A) , maxi6=j  A A i,j : Computationally cheap

alternative to the RIP. Coherence yields sufficient conditions for

recovery that are weaker than those based on RIP.

211

A Quiz
Consider SLM z = Ax + w with kxk0 1 and CS-matrix
!
0 1 1/2
A=
1 0 1/ 2

What is spark(A), 2 , and (A) ?

Is it possible to perfectly recover any 1-sparse vector x from z?

311

Coherence of a Bernoulli Matrix

mp
Consider a random matrix A {1/ m, 1/ m}
with i.i.d. zero
mean entries

By construction, the columns of A are normalized, i.e., kal k2 = 1

The coherence (A) is a RV since A is random
What is the probability that (A) > 0 for a given 0 ?
Two step analysis:
 T

Upper bound the probability P{ A A i,j  > 0 } for fixed i 6= j

 T

Apply union bound over i, j to bound P{maxi,j  A A i,j  > 0 }
411

Coherence of a Bernoulli Matrix - Step 1

mp
Consider a random matrix A {1/ m, 1/ m}
with i.i.d. zero
mean entries

 T

Upper bound the probability P{ A A i,j  > 0 } for fixed i 6= j

Use shorthand Ti,j :=

T
A A

i,j and note that

P{|Ti,j | > 0 } = P{Ti,j > 0 } + P{Ti,j < 0 }

The RV Ti,j is a sum of bounded i.i.d. RVs:
Ti,j =

m
X

l=1

cl with cl := al,i al,j {1/m, 1/m}

Using Markov inequality, one can show that

P{Ti,j > 0 } exp(m02 /2)

511

Coherence of a Bernoulli Matrix - Step 2

Result of Step 1: For any i 6= j we have P{|Ti,j | > 0 } 2 exp(m02 /2)
For coherence (A) we need to control maxi6=j |Ti,j |
For two events A, B, the union bound is P{A B} P{A} + P{B}
Applying union bound over all i 6= j (there are (p 1)p of them):
P{(A) > 0 } = P{max |Ti,j | > 0 }
i6=j

union bound

Step 1

(p 1)p max P{|Ti,j | > 0 }

i6=j

(p 1)p 2 exp(m02 /2)

2p2 exp(m02 /2)

611

When is the RIP satisfied?

Testing RIP of a given matrix NP-hard
One approach to RIP is using probability theory
This approach is constructive: gives explicit constructions for RIP-matrices
However, it is important to verify if there exists at least one matrix with RIP
RIP requires a minimum number m of measurements:
If the matrix A Rmp satisfies RIP of order 2s with 2s 1/2
then it must hold that


2s
m
log p/s
10

711

Packing Sets
Consider the metric space Rm endowed with metric kz1 z2 k2
The open ball with center zc and radius > 0 is defined as
B(zc , ) := {z Rm : kzc zk2 < }
A packing set D = {z1 , . . . , zD } Rm with small radius r and large
radius R is a set of D points such that
for any different z, z D: kz z k2 2r
for any z D: kzk2 R

Thus, the D small balls {B(z, r)}zD are disjoint

Moreover, the small balls are contained in the big ball B(0, r + R)
811

Packing Sets (ctd.)

The volume V m () of m-dimensional ball B(zc , ) Rm is given by
V m () = cm m
where the constant cm does not depend on radius

Consider packing set of size D with small radius r and large radius R
Since D disjoint balls with radius r contained in a ball with radius R + r:
DV m (r) V m (R + r)

Dcm rm cm (R + r)m
D (1 + R/r)m

log D
m
log(1 + R/r)
911

Packing Set Construction

Let us construct a packing set by applying a CS matrix A Rmp to a set
of sparse vectors E Xs
There is a set E Xs of size D = (p/s)s/2 such that
for any different x, x

E : kx x k2

for any x E : kxk2 s

s/2

Now consider the set D := AE = {Ax1 , . . . , AxD } Rm

If A satisfies RIP of order 2s with 2s 1/2, then D is packing set with
p

r = s/4 and R = 3/2 s

Thus,

log D
m
= (s/2) log(p/s)/ log(1 + 24)
log(1 + R/r)
1011

Thank you!

1111