This action might not be possible to undo. Are you sure you want to continue?

Olga V. Holtz

UC Berkeley & TU Berlin

Stanford

January 2009

History and Introduction Main idea Constructions Approximation theory

**Compressed Sensing: History
**

Compressed Sensing (CS)

People involved,

(Right to left: J. Claerbout, B. Logan, D. Donoho, E. Candés, T. Tao and R. DeVore)

History and Introduction Main idea Constructions Approximation theory

Compressed Sensing: Introduction Old-fashioned Thinking Compressed Sensing (CS) (CS camera at Rice) Collect data at grid points For n pixels. take n observations History and Introduction Main idea Constructions Approximation theory Takes only O(n1/4 log5 (n)) random measurements instead of n .

Ωπ] Shannon-Nyquist Uniform time sampling with spacing h ≤ 1/Ω gives exact reconstructon A/D converters: sample and quantize Problem: if Ω is very large.Traditional signal processing Model signals as band-limited functions x(t) Support of x is contained in [−Ωπ. one cannot build circuits to sample at the desired rate History and Introduction Main idea Constructions Approximation theory .

Signal processing using CS Compressive sensing seeks a way out of this dilemma Two new components: New model classes for signals: signals are sparse in some representation system (basis/frame) New meaning of samples: sample is a linear functional applied to the signal History and Introduction Main idea Constructions Approximation theory .

Signal processing using CS Compressive sensing seeks a way out of this dilemma Two new components: New model classes for signals: signals are sparse in some representation system (basis/frame) New meaning of samples: sample is a linear functional applied to the signal Given x ∈ IRn with n large. ask m non-adaptive questions about x Question means inner product v · x with v ∈ IRn means sample Such sampling is described by an m × n linear system Φx = y History and Introduction Main idea Constructions Approximation theory .

. a basis) Issue: in many problems. History and Introduction Main idea Constructions Approximation theory .Structure of signals With no additional information on x cannot say anything But we are interested in those x that have structure Typically x can be represented by sparse linear combinations of certain building blocks (e.g. we do not know the basis Here we assume the basis is known (for now) Ansatz: look for k -sparse solutions: x ∈ Σk that is # supp(x) ≤ k .

Moreover. The solution is in general not unique.Sparsest Solutions of Linear equations Find a sparsest solution of linear system (P0 ) min{ x 0 : Φx = b. this problem is NP-Hard History and Introduction Main idea Constructions Approximation theory . x ∈ IRn } where x 0 = number of nonzeros of x and Φ ∈ IRm×n with m < n.

and Saunders (1999)] Solving (P1 ) in polynomial time Can be solved by linear programming: min s.Basis Pursuit Main idea: Use the convex relaxation (P1 ) min{ x 1 : Φx = b. x ∈ IRn } Basis Pursuit [Chen. Donoho.t. 1y T Φx = b −y ≤ x ≤ y History and Introduction Main idea Constructions Approximation theory .

.Sparse Recovery and Mutual Incoherence Mutual incoherence: T M(Φ) = max|φi φj | i=j where Φ = [φ1 . . φn ] ∈ IRm×n and φi 2 = 1. M(Φ) Then the solution of (P1 ) is equal to the solution of (P0 ). Theorem (Elad and Bruckstein (2002)) Suppose that for the sparsest solution x we have √ 1 ( 2 − 2) x 0< . History and Introduction Main idea Constructions Approximation theory .

φn ] ∈ IRm×n . . Romberg. Candès (2008)) √ If δ2k < 2 − 1. Tao (2006)]: Let δk be the smallest number such that (1 − δk ) x 2 2 ≤ Φx 2 2 ≤ (1 + δk ) x 2 2 for all k -sparse vectors x ∈ IRn where Φ = [φ1 . Theorem (E. then for all k -sparse vectors x such that Φx = b. the solution of (P1 ) is equal to the solution of (P0 ).Sparse Recovery and RIP Restricted Isometry Property of Order k [Candès. . J. History and Introduction Main idea Constructions Approximation theory .

Mangasarian’s approach. where x is the solution of (P1 ) and ˆ ˆ σk (x )1 = min x − z z∈Σk 1. then ˆ x −x 2 ˆ ≤ C0 k −1/2 σk (x )1 + C1 . Other Heuristics: Orthogonal Matching Pursuit. and Saunders (1999)] Theorem (E. J. Donoho. Candès (2008)) ˆ Suppose that the matrix Φ is given and b = Φx + e where √ e 2 ≤ . If δ2k < 2 − 1. etc. History and Introduction Main idea Constructions Approximation theory .Approximate Recovery and RIP Basis Pursuit De-Noising (BPDN): (P1 ) min{ x 1 : Φx − b 2 ≤ } [Chen. Bilinear formulation.

History and Introduction Main idea Constructions Approximation theory .End of Part I.

Particular case: there is a probabilistic construction of 1 matrices Φ of size m × n with entries {± √m } satisfying RIP of order k with the above bound.Construction of CS Matrices Good compressive sensing (CS) matrices: Known Result for Random matrices Known reconstruction bounds for matrices with entries drawn at random from various probability distributions: k ≤ Cm/ log(n/m). History and Introduction Main idea Constructions Approximation theory . Bernoulli and other classical matrix ensembles. Speciﬁc recipes include Gaussian.

δ) ⊂ Ω s. there is a set Ω0 (x. δ) 2 Gaussian. ) Suppose Φ = Φ(ω) is a collection of random m × n matrices Property PO(δ): the collection is said to have CMI if.t. Bernoulli and many other families have this property History and Introduction Main idea Constructions Approximation theory . for each x ∈ IRn . (1 − δ) x 2 ≤ Φx 2 ≤ (1 + δ) x and (Ω(x. δ)c ) ≤ C0 e−c0 mδ 2. ω ∈ Ω(x.Probabilistic Construction of CS Matrices Introduce the Concentration of Measure Inequality (CMI) property on a probability space (Ω.

ε) satisﬁes 2 2 +ln C (Ωc ) ≤ C0 (#X )2 e−c0 nε = e2 ln(#X )−c0 mε 0 0 If m ≥ (2 ln(#X ) + ln C0 )/c0 ε2 . there is a Lipschitz function Φ : IRn → IRm s. a set X of points in IRn such that #X =: σ > σ0 = O(ln m/ε2 ). History and Introduction Main idea Constructions Approximation theory .x ∈X Ω(x − x . then the set Ω0 := ∩x.t. (1 − ε) u − v 2 ≤ Φ(u) − Φ(v ) 2 ≤ (1 + ε) u − v 2.Property PO and the JL Lemma Johnson-Lindenstrauss Lemma. If X is a set of points and m > c ln(#X )ε−2 with c sufﬁciently large. then the measure is < 1 hence we get the JL lemma. 1). Given ε ∈ (0.

then a random draw of Gaussian will satisfy RIP with respect to all of these bases simultaneously Basic reason why this works Pr Φ(ω)x 2 2 − x 2 2 ≥ε x 2 2 ≤ 2e−mc0 (ε) . 0 < ε < 1. History and Introduction Main idea Constructions Approximation theory .The JL Lemma and RIP If k ≤ cm/ ln(n/m) and Φ satisﬁes JL. the probability that a Gaussian emsemble satisﬁes RIP is 1 − Ce−cm If we have a collection of O(ecm ) bases. then we have RIP of order k For c sufﬁciently small.

Holy grail: k ≤ Cm/ log(n/m) – achievable for random matrices but not yet in deterministic constructions. using structured matrices such as Toeplitz. Subgoal: deterministic polynomial time algorithm for constructing good CS matrices.Deterministic Construction of CS Matrices Difﬁculties in Deterministic Case [DeVore]: Proposes a deterministic constructions of order √ k ≤ C m log n/ log(n/m) which is still far from probabilistic results. Xu]. Hassibi. generalized Vandermonde matrices. Very recent ideas Find deterministic constructions with better bounds using bipartite expander graphs [Indyk. History and Introduction Main idea Constructions Approximation theory . cyclic.

To extract x / approximation to x.Connections with approximation theory Cohen. Best k -term approximation error: σk (x)X := inf z∈Σk x −z X. Encoder-decoder viewpoint The matrix Φ serves as an encoder producing y = Φx. use a decoder ∆ (not necessarily linear). Thus ∆(y ) = ∆(Φx) approximates x. Dahmen. History and Introduction Main idea Constructions Approximation theory . JAMS. DeVore [2009]: Compressed sensing and best k -term approximation.

History and Introduction Main idea Constructions Approximation theory . Cσk (K )X . where inf sup x − ∆(Φx) . for a given class K . En (K )X ≤ En (K )X := σk (K )X := sup σk (X ).t. x ∈ Σk =⇒ ∆(Φx) = x. (Φ.t.∆) x∈K x∈K A pair (Φ.Performance of encoders-decoders Performance of encoder-decoder pairs Ask for the largest value of k s. ∆) is called instance-optimal of order k with constant C for the space X is x − ∆(Φx) X ≤ Cσk (x)X for all x ∈ X with a constant C independent of k and n. Ask for the largest value of k s..

e.Connection with Gelfand widths Gelfand widths For K a compact set in X and m ∈ IN.. Let K ⊂ IRn be symmetric. History and Introduction Main idea Constructions Approximation theory 1 ≤ m ≤ n. K = −K . If X ⊆ IRn is any normed space. the Gelfand width of K of order m is d m (K )X := inf codimY ≤m sup{ x X : x ∈ K ∩ Y }. then d m (K )X ≤ Em (K )X ≤ C0 d m (K )X . . and satisfy K + K ⊂ C0 K for some C0 . i. Basic result Lemma.

n. log(n/m) }. n. q. p) where q Ψ(m. Garnaev. n. p) := min{1. n1−1/q m−1/2 } Ψ(m. Kashin (1977. q. The necessary number of measurements k satisﬁes k ≤ c0 m/ log(n/m). 1/q−1/p 1/q−1/2 . History and Introduction Main idea Constructions Approximation theory . 1. q.1984)] C1 Ψ(m. n. p) ≤ d m (U( n )) ≤ C2 Ψ(m. 2) := min{1.Orders of Gelfand widths Orders of Gelfand widths of q balls Theorem [Gluskin. m Corollary.

the matrix Φ∗ ΦT is positive deﬁnite. the following are equivalent: There is a decoder ∆ s. For any T as above. T History and Introduction Main idea Constructions Approximation theory . Σ2k ∩ N = {0}. Uniqueness of recovery Lemma.Instance optimality and the null space of Φ Denote N := N (Φ) :={x : Φx = 0}.t. the matrix ΦT has rank 2k . ∆(Φx) = x for all x ∈ Σk . For any set T with #T = 2k . For an m × n matrix Φ and for 2k ≤ m.

η ∈ N.Approximate recovery Approximation to accuracy σk Theorem [Cohen.t. Given an m × n matrix Φ. DeVore (2009)]. . A necessary condition is that η X ≤ C · σ2k (η)X . x − ∆(Φx) X ≤ Cσk (x)X is that η X ≤ C/2 · σ2k (η)X . This gives rise to the null space property (in X of order 2k ): η X ≤ C · σ2k (η)X . a sufﬁcient condition that there exists a decoder ∆ s. Dahmen. η ∈ N. a norm · X and a value of k . History and Introduction Main idea Constructions Approximation theory η ∈ N.

k ∈ IN and Φ is an encoding matrix. Conversely. History and Introduction Main idea Constructions Approximation theory . then there exists a decoder ∆ so that x − ∆(Φx) X ≤ Cσk (x)X . the validity of the above condition for some decoder ∆ implies that Φ has the null space property in X of order 2k with constant C. n Suppose that X is an lp space. Dahmen. DeVore (2009)].The null space property Approximation and the null space property Corollary [Cohen. If Φ has the null space property in X of order 2k with constant C/2.

2−1−( 2+1)δ History and Introduction Main idea Constructions Approximation theory . Then (Φ. revisited RIP and good encoder-decoder pairs Theorem [Candès-Romberg-Tao (2006)]. Deﬁne the decoder ∆ by ∆(y ) := argminΦz=y z 1 . Let Φ be any matrix √ with satisﬁes the RIP of order 3k with δ3k ≤ δ < ( 2 − 1)2 /3.One of the main results. ∆) satisﬁes x − ∆(Φx) in X = 1 with C = X ≤ Cσk (x)X √ √ 2 2+2−(2 2−2)δ √ √ .

History and Introduction Main idea Constructions Approximation theory .The End.

Sign up to vote on this title

UsefulNot usefulCompressive Sampling or Compressed Sensing

Compressive Sampling or Compressed Sensing

- image reconstruction using compressive sensing
- 05967912
- 1-s2.0-S002437950900055X-main
- Advanced Control Notes
- TEST ON ENGINEERING MATHEMATICS.docx
- l_11
- lag15
- 0912 ch01
- Or Assignment_Group 15
- Linear Algebra - Solved Assignments - Fall 2006 Semester
- Exam Practice math 2107
- MKlein MAE239 TermPaper 20Mar2013
- [IP-1] AHP
- Ddd
- International Journal of Mathematics and Statistics Invention (IJMSI)
- Libro
- 20. Eng-An Interpretive Structural Model-Jocob P. George
- cha7
- LR
- MODUL A SET 1
- 24_numerics
- listofprogramC++
- Mktg & Biz Plan -Griffiths-3
- mm-40
- Chapter 4
- R Exercises
- Tech Review
- Ch 9. C.Y LEE & NNGOS
- 12 programs.docx
- Basics of Matlab-1.doc
- CS