0 Up votes0 Down votes

3 views184 pagesJan 24, 2018

© © All Rights Reserved

PDF, TXT or read online from Scribd

© All Rights Reserved

3 views

© All Rights Reserved

- Solution Manual for Discrete Fourier Analysis and Wavelets 1st Ed - Allen Broughton, Kurt Bryan
- M1 Formula Sheet
- mca details
- Quantum Mechanics
- Dpp Maths & Computer-new
- Lecture Notes on Linear System Theory
- Machine Learning
- Me471s03 Laplace Examp
- IIT Guhati Note Continuum Mechanics
- Form-finding of Tensegrity Structures Subjected to Geometrical Constraints
- M1H
- Notes 5
- Chen 2018
- I year syllabus 2013-14.pdf
- Paper 7
- Clustering
- PS4_Soln
- mathgen-1898293085
- decoupling in multivariable systems.pdf
- A Geometric Review of Linear Algebra

You are on page 1of 184

Based upon textbook with Jelena Kovačević

Includes slides by Andrea Ridolfi and Amina Chebira

Outline

1 Introduction

Why a tutorial on teaching?

Overview

2 Basic Principles in Teaching SP with Geometry

Unified view

Hilbert space tools—Part I: Basics through projections

Basic results I: Best approximation

Hilbert space tools—Part II: Bases through discrete-time systems

3 Example Lecture: Sampling and Interpolation

Motivation

Classical View and Historical Notes

Operator View

Basis Expansion View

Extensions

Summary

4 Teaching Materials

Textbooks

Supplementary materials

5 Wrap up

Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 2 / 96

Introduction Why a tutorial on teaching?

Teaching should not remain static

Applications change (and we should want them to)

◮ Signal processing thinking should be applied broadly

◮ Global Fourier techniques relatively less important than in the past

◮ Classical DSP architectures relative less important than in the past

◮ Likely to use high-level programming languages

◮ Different base of knowledge

◮ Biology, economics, social sciences, . . .

◮ Knowledge grows, time in school does not

◮ Must be willing to cull details to convey big picture

◮ Should teach what is most reusable and generalizable

Introduction Why a tutorial on teaching?

Teaching should not remain static

Applications change (and we should want them to)

◮ Signal processing thinking should be applied broadly

◮ Global Fourier techniques relatively less important than in the past

◮ Classical DSP architectures relative less important than in the past

◮ Likely to use high-level programming languages

◮ Different base of knowledge

◮ Biology, economics, social sciences, . . .

◮ Knowledge grows, time in school does not

◮ Must be willing to cull details to convey big picture

◮ Should teach what is most reusable and generalizable

Introduction Why a tutorial on teaching?

Teaching should not remain static

Applications change (and we should want them to)

◮ Signal processing thinking should be applied broadly

◮ Global Fourier techniques relatively less important than in the past

◮ Classical DSP architectures relative less important than in the past

◮ Likely to use high-level programming languages

◮ Different base of knowledge

◮ Biology, economics, social sciences, . . .

◮ Knowledge grows, time in school does not

◮ Must be willing to cull details to convey big picture

◮ Should teach what is most reusable and generalizable

Introduction Why a tutorial on teaching?

Teaching should not remain static

Applications change (and we should want them to)

◮ Signal processing thinking should be applied broadly

◮ Global Fourier techniques relatively less important than in the past

◮ Classical DSP architectures relative less important than in the past

◮ Likely to use high-level programming languages

◮ Different base of knowledge

◮ Biology, economics, social sciences, . . .

◮ Knowledge grows, time in school does not

◮ Must be willing to cull details to convey big picture

◮ Should teach what is most reusable and generalizable

Introduction Overview

Overview

Goals of the tutorial:

See that geometric notions unify (simplify!) signal processing

Learn/review basics of Hilbert space view

See Hilbert space view in action

Learn about textbooks Foundations of Signal Processing and

Fourier and Wavelet Signal Processing

Developing unified view of signal processing

Hilbert space tools—Part I: Basics through projections

◮ A few key results (best approximation and the projection theorem)

Hilbert space tools—Part II: Bases through discrete-time systems

Example lecture: Sampling made easy

Textbooks and related materials

Introduction Overview

Overview

Goals of the tutorial:

See that geometric notions unify (simplify!) signal processing

Learn/review basics of Hilbert space view

See Hilbert space view in action

Learn about textbooks Foundations of Signal Processing and

Fourier and Wavelet Signal Processing

Developing unified view of signal processing

Hilbert space tools—Part I: Basics through projections

◮ A few key results (best approximation and the projection theorem)

Hilbert space tools—Part II: Bases through discrete-time systems

Example lecture: Sampling made easy

Textbooks and related materials

Introduction Overview

Overview

Goals of the tutorial:

See that geometric notions unify (simplify!) signal processing

Learn/review basics of Hilbert space view

See Hilbert space view in action

Learn about textbooks Foundations of Signal Processing and

Fourier and Wavelet Signal Processing

Developing unified view of signal processing

Hilbert space tools—Part I: Basics through projections

◮ A few key results (best approximation and the projection theorem)

Hilbert space tools—Part II: Bases through discrete-time systems

Example lecture: Sampling made easy

Textbooks and related materials

Introduction Overview

Goals of the tutorial:

See that geometric notions unify (simplify!) signal processing

Learn/review basics of Hilbert space view

See Hilbert space view in action

Learn about textbooks Foundations of Signal Processing and

Fourier and Wavelet Signal Processing

Developing unified view of signal processing

Hilbert space tools—Part I: Basics through projections

◮ A few key results (best approximation and the projection theorem)

Hilbert space tools—Part II: Bases through discrete-time systems

Example lecture: Sampling made easy

Textbooks and related materials

Basic Principles in Teaching SP with Geometry Unified view

Unifying principles

Signal processing has various dichotomies

continuous time vs. discrete time

infinite intervals vs. finite intervals

periodic vs. aperiodic

deterministic vs. stochastic

Example payoffs:

Unified understanding of best approximation (projection theorem)

Unified understanding of Fourier domains

Unified understanding of signal expansions (including sampling)

Basic Principles in Teaching SP with Geometry Unified view

Unifying principles

Signal processing has various dichotomies

continuous time vs. discrete time

infinite intervals vs. finite intervals

periodic vs. aperiodic

deterministic vs. stochastic

Example payoffs:

Unified understanding of best approximation (projection theorem)

Unified understanding of Fourier domains

Unified understanding of signal expansions (including sampling)

Basic Principles in Teaching SP with Geometry Unified view

Unifying principles

Signal processing has various dichotomies

continuous time vs. discrete time

infinite intervals vs. finite intervals

periodic vs. aperiodic

deterministic vs. stochastic

Example payoffs:

Unified understanding of best approximation (projection theorem)

Unified understanding of Fourier domains

Unified understanding of signal expansions (including sampling)

Basic Principles in Teaching SP with Geometry Unified view

Signal processing has various dichotomies

continuous time vs. discrete time

infinite intervals vs. finite intervals

periodic vs. aperiodic

deterministic vs. stochastic

Unified understanding of best approximation (projection theorem)

Unified understanding of Fourier domains

Unified understanding of signal expansions (including sampling)

Basic Principles in Teaching SP with Geometry Unified view

Examples of Hilbert spaces:

finite-dimensional vectors (basic linear algebra)

sequences on {. . . , −1, 0, 1, . . .} (discrete-time signals)

sequences on {0, 1, . . . , N − 1} (N-periodic discrete-time signals)

functions on (−∞, ∞) (continuous-time signals)

functions on [0, T ] (T -periodic continuous-time signals)

scalar random variables

With framework in place, can go farther, faster

Leverage “real world” geometric intuition

Basic Principles in Teaching SP with Geometry Unified view

Examples of Hilbert spaces:

finite-dimensional vectors (basic linear algebra)

sequences on {. . . , −1, 0, 1, . . .} (discrete-time signals)

sequences on {0, 1, . . . , N − 1} (N-periodic discrete-time signals)

functions on (−∞, ∞) (continuous-time signals)

functions on [0, T ] (T -periodic continuous-time signals)

scalar random variables

With framework in place, can go farther, faster

Leverage “real world” geometric intuition

Basic Principles in Teaching SP with Geometry Unified view

Mathematical rigor

Everything should be made as simple as possible, but no simpler.

– Common paraphrasing of Albert Einstein

Make everything as simple as possible without being wrong.

– Our variant for teaching

Teach the difference among

◮ rigorously true, with elementary justification

◮ rigorously true, justification not elementary (e.g., Poisson sum formula)

◮ convenient and related to rigorous statements (e.g., uses of Dirac delta)

function . . . was Lebesgue but not Riemann integrable, then I would not

fly in it.

– Richard W. Hamming

Basic Principles in Teaching SP with Geometry Unified view

Mathematical rigor

Everything should be made as simple as possible, but no simpler.

– Common paraphrasing of Albert Einstein

Make everything as simple as possible without being wrong.

– Our variant for teaching

Teach the difference among

◮ rigorously true, with elementary justification

◮ rigorously true, justification not elementary (e.g., Poisson sum formula)

◮ convenient and related to rigorous statements (e.g., uses of Dirac delta)

function . . . was Lebesgue but not Riemann integrable, then I would not

fly in it.

– Richard W. Hamming

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Vector spaces

A vector space generalizes easily beyond the R2 Euclidean plane

Axioms

A vector space over a field of scalars C (or R) is a set of vectors V together

with operations

◮ vector addition: V × V → V

◮ scalar multiplication: C × V → V

that satisfy the following axioms:

1. x +y =y +x

2. (x + y ) + z = x + (y + z)

3. ∃ 0 ∈ V s.t. x + 0 = x for all x ∈ V

4. α(x + y ) = αx + αy

5. (α + β)x = αx + βx

6. (αβ)x = α(βx)

7. 0x = 0 and 1x = x

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Vector spaces

Examples

CN : complex (column) vectors of length N

(write as infinite column vector)

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Vector spaces

Key notions

Subspace

◮ S ⊆ V is a subspace when it is closed under vector addition and scalar

multiplication:

⋆ For all x, y ∈ S, x + y ∈ S

⋆ For all x ∈ S and α ∈ C, αx ∈ S

Span

◮ S: set of vectors (could be infinite)

◮ span(S) = set of all finite linear combinations of vectors in S:

( N−1 )

X

S = αk ϕk | αk ∈ C, ϕk ∈ S and N ∈ N

k=0

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Vector spaces

Key notions

Subspace

◮ S ⊆ V is a subspace when it is closed under vector addition and scalar

multiplication:

⋆ For all x, y ∈ S, x + y ∈ S

⋆ For all x ∈ S and α ∈ C, αx ∈ S

Span

◮ S: set of vectors (could be infinite)

◮ span(S) = set of all finite linear combinations of vectors in S:

( N−1 )

X

S = αk ϕk | αk ∈ C, ϕk ∈ S and N ∈ N

k=0

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Vector spaces

Key notions

Linear independence

N−1

◮ S = {ϕk }k=0 is linearly independent when:

N−1

X

αk ϕk = 0 only when αk = 0 for all k

k=0

Dimension

◮ dim(V ) = N if V contains a linearly independent set with N vectors and every

set with N + 1 or more vectors is linearly dependent

◮ V is infinite dimensional if no such finite N exists

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Vector spaces

Key notions

Linear independence

N−1

◮ S = {ϕk }k=0 is linearly independent when:

N−1

X

αk ϕk = 0 only when αk = 0 for all k

k=0

Dimension

◮ dim(V ) = N if V contains a linearly independent set with N vectors and every

set with N + 1 or more vectors is linearly dependent

◮ V is infinite dimensional if no such finite N exists

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Inner products

Inner products generalize angles (especially right angles) and orientation

An inner product on vector space V is a function h·, ·i : V × V → C

satisfying

1 Distributivity: hx + y , zi = hx, zi + hy , zi

2 Linearity in the first argument: hαx, y i = αhx, y i

3 Hermitian symmetry: hx, y i∗ = hy , xi

4 Positive definiteness: hx, xi ≥ 0 and hx, xi = 0 if and only if x = 0

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Inner products

Examples

N−1

X

On CN : hx, y i = xn yn∗ = y ∗ x

n=0

X

On CZ : hx, y i = xn yn∗ = y ∗ x

n∈Z

Z ∞

On CR : hx, y i = x(t)y ∗ (t) dt

−∞

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Drawn in R2 and true in general:

hx, y i = x1 y1 + x2 y2

= kxk ky k cos α

= product of 2-norms times the cos

of the angle between the vectors

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Drawn in R2 and true in general:

hx, y i = x1 y1 + x2 y2

= kxk ky k cos α

= product of 2-norms times the cos

of the angle between the vectors

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Orthogonality

Let S = {ϕi }i ∈I be a set of vectors

Definition (Orthogonality)

x and y are orthogonal when hx, y i = 0 written x ⊥ y

S0 ⊥ S1

Right angles (perpendicularity) extends beyond Euclidean geometry

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Norm

Norms generalize length in ordinary Euclidean space

Definition (Norm)

A norm on V is a function k·k : V → R satisfying

1 Positive definiteness: kxk ≥ 0 and kxk = 0 if and only if x = 0

2 Positive scalability: kαxk = |α| kxk

3 Triangle inequality: kx + y k ≤ kxk + ky k with equality if and only if y = αx

p

kxk = hx, xi

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Norm

Norms generalize length in ordinary Euclidean space

Definition (Norm)

A norm on V is a function k·k : V → R satisfying

1 Positive definiteness: kxk ≥ 0 and kxk = 0 if and only if x = 0

2 Positive scalability: kαxk = |α| kxk

3 Triangle inequality: kx + y k ≤ kxk + ky k with equality if and only if y = αx

p

kxk = hx, xi

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

p

Any inner product induces a norm: kxk = hx, xi

Examples

N−1

!1/2

p X

N 2

On C : kxk = hx, xi = |xn |

n=0

!1/2

p X

On CZ : kxk = hx, xi = |xn |2

n∈Z

Z 1/2

p ∞

On CR : kxk = hx, xi = |x(t)|2 dt

−∞

p

On C-valued random variables: kxk = hx, xi = E |x|2

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Properties

Pythagorean theorem

◮ x ⊥y ⇒ kx + y k2 = kxk2 + ky k2

X
2 X

◮ {xk }k∈K orthogonal ⇒

xk

=

kxk k2

k∈K k∈K

x +y

y

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Properties

Parallelogram Law

kx + y k2 + kx − y k2 = 2(kxk2 + ky k2 )

x

x +y

y y

x −y

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Properties

Cauchy–Schwarz inequality

|hx, y i| ≤ kxk ky k

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Examples

P 1/p

N−1

On CN : kxkp = n=0 |xn |

p

, p ∈ [1, ∞)

P p

1/p

On CZ : kxkp = n∈Z |xn | , p ∈ [1, ∞)

kxk∞ = sup|xn |

n∈Z

R 1/p

∞

On CR : kxkp = −∞

|x(t)|p dt , p ∈ [1, ∞)

t∈R

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

p

Geometry of ℓ : Unit balls

x1

kxk∞ = 1

kxk4 = 1

kxk2 = 1

kxk1 = 1

kxk1/2 = 1

x0

Valid norm (and convex unit ball) for p ≥ 1; ordinary geometry for p = 2

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

A normed vector space is a set satisfying axioms of a vector space where the

norm is finite

!1/2

X

2

kxk = |xn | < ∞

n∈Z

Z ∞ 1/2

kxk = |x(t)|2 dt < ∞

−∞

No harm in considering only functions with finitely-many discontinuities

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

A normed vector space is a set satisfying axioms of a vector space where the

norm is finite

!1/2

X

2

kxk = |xn | < ∞

n∈Z

Z ∞ 1/2

kxk = |x(t)|2 dt < ∞

−∞

No harm in considering only functions with finitely-many discontinuities

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Definition

A sequence of vectors x0 , x1 , . . . in a normed vector space V is said to converge

to v ∈ V when limk→∞ kv − xk k = 0, or for any ε > 0, there exists Kε such that

kv − xk k < ε for all k > Kε .

Example

For k ∈ Z+ , let

1, for t ∈ [0, 1/k ];

xk (t) =

0, otherwise.

v (t) = 0 for all t. Then for p ∈ [1, ∞),

Z ∞ 1/p 1/p

1 k→∞

kv − xk kp = |v (t) − xk (t)|p dt = −→ 0,

−∞ k

Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 24 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Definition

A sequence of vectors x0 , x1 , . . . in a normed vector space V is said to converge

to v ∈ V when limk→∞ kv − xk k = 0, or for any ε > 0, there exists Kε such that

kv − xk k < ε for all k > Kε .

Example

For k ∈ Z+ , let

1, for t ∈ [0, 1/k ];

xk (t) =

0, otherwise.

v (t) = 0 for all t. Then for p ∈ [1, ∞),

Z ∞ 1/p 1/p

1 k→∞

kv − xk kp = |v (t) − xk (t)|p dt = −→ 0,

−∞ k

Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 24 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Definition

A sequence of vectors x0 , x1 , . . . in a normed vector space V is said to converge

to v ∈ V when limk→∞ kv − xk k = 0, or for any ε > 0, there exists Kε such that

kv − xk k < ε for all k > Kε .

Example

For k ∈ Z+ , let

1, for t ∈ [0, 1/k ];

xk (t) =

0, otherwise.

v (t) = 0 for all t. Then for p ∈ [1, ∞),

Z ∞ 1/p 1/p

1 k→∞

kv − xk kp = |v (t) − xk (t)|p dt = −→ 0,

−∞ k

Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 24 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Definitions

A sequence {xn } is a Cauchy sequence in a normed space when for any

ε > 0, there exists kε such that kxk − xm k < ε for all k, m > kε

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Hilbert spaces

Examples

Q is not a complete space

X∞

1 π2

◮ → ∈ R, ∈

/Q

n=1

n2 6

X∞

1

◮ → e ∈ R, ∈

/Q

n=0

n!

e

5 8 65

1 2 2 3 24

R is a complete space

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Hilbert spaces

Examples

Q is not a complete space

X∞

1 π2

◮ → ∈ R, ∈

/Q

n=1

n2 6

X∞

1

◮ → e ∈ R, ∈

/Q

n=0

n!

e

5 8 65

1 2 2 3 24

R is a complete space

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Hilbert spaces

Examples

All finite dimensional spaces are complete

◮ ℓ2 (Z) and L2 (R) are Hilbert spaces

C q ([a, b]), functions on [a, b] with q continuous derivatives, are not complete

except for q = 0 under L∞ norm

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Hilbert spaces

Examples

All finite dimensional spaces are complete

◮ ℓ2 (Z) and L2 (R) are Hilbert spaces

C q ([a, b]), functions on [a, b] with q continuous derivatives, are not complete

except for q = 0 under L∞ norm

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Hilbert spaces

Examples

All finite dimensional spaces are complete

◮ ℓ2 (Z) and L2 (R) are Hilbert spaces

C q ([a, b]), functions on [a, b] with q continuous derivatives, are not complete

except for q = 0 under L∞ norm

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Summary on spaces

Vector spaces

Normed vector spaces

Hilbert

spaces

• ℓ1 (Z)

•Q N

•C N • ℓ∞ (Z)

• C ([a, b]) • ℓ2 (Z) • L1 (R)

• L2 (R) • L∞ (R)

• (C 1 ([a, b]), k · k∞ ) • (V , d)

• ℓ0 (Z)

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Linear operators

Linear operators generalize matrices

Definitions

A : H0 → H1 is a linear operator when for all x, y ∈ H0 , α ∈ C:

1 Additivity: A(x + y ) = Ax + Ay

2 Scalability: A(αx) = α(Ax)

Null space (subspace of H0 ): N (A) = {x ∈ H0 , Ax = 0}

Range space (subspace of H1 ): R(A) = {Ax ∈ H1 , x ∈ H0 }

Operator norm: kAk = sup kAxk

kxk=1

Inverse: Bounded B : H1 → H0 inverse of bounded A if and only if:

BAx = x, for every x ∈ H0

ABy = y , for every y ∈ H1

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Linear operators

Linear operators generalize matrices

Definitions

A : H0 → H1 is a linear operator when for all x, y ∈ H0 , α ∈ C:

1 Additivity: A(x + y ) = Ax + Ay

2 Scalability: A(αx) = α(Ax)

Null space (subspace of H0 ): N (A) = {x ∈ H0 , Ax = 0}

Range space (subspace of H1 ): R(A) = {Ax ∈ H1 , x ∈ H0 }

Operator norm: kAk = sup kAxk

kxk=1

Inverse: Bounded B : H1 → H0 inverse of bounded A if and only if:

BAx = x, for every x ∈ H0

ABy = y , for every y ∈ H1

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Linear operators

Linear operators generalize matrices

Definitions

A : H0 → H1 is a linear operator when for all x, y ∈ H0 , α ∈ C:

1 Additivity: A(x + y ) = Ax + Ay

2 Scalability: A(αx) = α(Ax)

Null space (subspace of H0 ): N (A) = {x ∈ H0 , Ax = 0}

Range space (subspace of H1 ): R(A) = {Ax ∈ H1 , x ∈ H0 }

Operator norm: kAk = sup kAxk

kxk=1

Inverse: Bounded B : H1 → H0 inverse of bounded A if and only if:

BAx = x, for every x ∈ H0

ABy = y , for every y ∈ H1

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Linear operators

Linear operators generalize matrices

A : H0 → H1 is a linear operator when for all x, y ∈ H0 , α ∈ C:

1 Additivity: A(x + y ) = Ax + Ay

2 Scalability: A(αx) = α(Ax)

Null space (subspace of H0 ): N (A) = {x ∈ H0 , Ax = 0}

Range space (subspace of H1 ): R(A) = {Ax ∈ H1 , x ∈ H0 }

Operator norm: kAk = sup kAxk

kxk=1

Inverse: Bounded B : H1 → H0 inverse of bounded A if and only if:

BAx = x, for every x ∈ H0

ABy = y , for every y ∈ H1

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Adjoint operators

Adjoint generalizes Hermitian transposition of matrices

A∗ : H1 → H0 is the adjoint of A : H0 → H1 when

hAx, y iH1 = hx, A∗ y iH0 for every x ∈ H0 , y ∈ H1

If A = A∗ , A is self-adjoint or Hermitian

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

2 y2 = 1

8 y3

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Adjoint operators

Let A : H0 → H1 be a bounded linear operator

1 A∗ exists and is unique

2 (A∗ )∗ = A

3 AA∗ and A∗ A are self-adjoint

4 kA∗ k = kAk

5 If A is invertible, (A−1 )∗ = (A∗ )−1

6 If B : H0 → H1 is bounded, (A + B)∗ = A∗ + B ∗

7 If B : H1 → H2 is bounded, (BA)∗ = A∗ B ∗

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Z k+1/2

A : L2 (R) → ℓ2 (Z) (Ax)k = x(t)dt

k−1/2

Z !

X X n+1/2 XZ n+1/2

hAx, y iℓ2 = (Ax)n yn∗ = x(t) dt yn∗ = x(t)yn∗ dt

n∈Z n∈Z n−1/2 n∈Z n−1/2

XZ n+1/2 Z ∞

= x ∗ (t) dt =

x(t)b x ∗ (t) dt = hx, b

x(t)b x iL2 = hx, A∗ y iL2

n∈Z n−1/2 −∞

x(t) yn = (Ax)n

10 10

5 5

t n

5 10 15 5 10 15

−5 −5

−10 −10

b x(t), yn , b

x (t)

10 10

5 5

t t

5 10 15 5 10 15

−5 −5

−10 −10

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Unitary operators

A bounded linear operator A : H0 → H1 is unitary when:

1 A is invertible

2 A preserves inner products: hAx, Ay iH1 = hx, y iH0 for every x, y ∈ H0

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Unitary operators

A bounded linear operator A : H0 → H1 is unitary when:

1 A is invertible

2 A preserves inner products: hAx, Ay iH1 = hx, y iH0 for every x, y ∈ H0

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Projection operators

P is idempotent when P 2 = P

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Projection operators

P is idempotent when P 2 = P

Theorem

If A : H0 → H1 , B : H1 → H0 bounded and A is a left inverse of B, then BA is

a projection operator. If B = A∗ then, BA = A∗ A is an orthogonal projection

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation

x•

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation

x•

x ∈ S that is closest to x

Nearest point problem: Find b

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation

x•

•

•′ b

x

x

x ∈ S that is closest to x

Nearest point problem: Find b

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation

x•

•

•′ b

x

x

x ∈ S that is closest to x

Nearest point problem: Find b

Solution uniquely determined by x − b

x⊥X

◮ Circle must touch S in one point, radius ⊥ tangent

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation

x

•

•′ b

x

x

x ∈ S that is closest to x

Nearest point problem: Find b

Solution uniquely determined by x − b

x⊥X

◮ Circle must touch S in one point, radius ⊥ tangent

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation

x ∈ S that is closest to x

Find b

x = argminkx − sk

b

s∈S

•

•′ b

x

x

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation

Let S be a closed subspace of Hilbert space H and let x ∈ H.

Existence: There exists xb ∈ S such that kx − b

x k ≤ kx − sk for all s ∈ S

Orthogonality: x − b

x ⊥ S is necessary and sufficient to determine b

x

Uniqueness: b

x is unique

Linearity: b

x = Px where P is a linear operator

Self-adjointness: P = P ∗

All “nearest vector in a subspace” problems in Hilbert spaces are the same

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation

Let S be a closed subspace of Hilbert space H and let x ∈ H.

Existence: There exists xb ∈ S such that kx − b

x k ≤ kx − sk for all s ∈ S

Orthogonality: x − b

x ⊥ S is necessary and sufficient to determine b

x

Uniqueness: b

x is unique

Linearity: b

x = Px where P is a linear operator

Self-adjointness: P = P ∗

All “nearest vector in a subspace” problems in Hilbert spaces are the same

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation

Let S be a closed subspace of Hilbert space H and let x ∈ H.

Existence: There exists xb ∈ S such that kx − b

x k ≤ kx − sk for all s ∈ S

Orthogonality: x − b

x ⊥ S is necessary and sufficient to determine b

x

Uniqueness: b

x is unique

Linearity: b

x = Px where P is a linear operator

Self-adjointness: P = P ∗

All “nearest vector in a subspace” problems in Hilbert spaces are the same

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation

Z 1 2 1

0 = hx(t) − b

x (t), 1it = cos( 32 πt) − (a0 + a1 t) 1 dt = − − a0 − a1

0 3π 2

Z 1 4 + 6π 1 1

0 = hx(t) − b

x (t), tit = cos( 32 πt) − (a0 + a1 t) t dt = − − a0 − a1

0 9π 2 2 3

1 1

0.8 0.8

0.6 x(t) 0.6

0.4 0.4

0.2 0.2

0

−0.2

x (t)

b 0

−0.2

−0.4 −0.4

−0.6 −0.6

−0.8 −0.8

−1 −1

0 0.2 0.4

t 0.6 0.8 1 0 0.2 0.4

t 0.6 0.8 1

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation

Find the constant c that minimizes E (x − c)2

Note:

◮ Expected square is a Hilbert space norm

◮ Constants are a closed subspace in vector space of random variables

◮ c determined uniquely by E[ (x − c) αc ] = 0 for all α ∈ R

◮ c = E[ x ]

Alternative:

◮ Expand into quadratic function of c and minimize with calculus

◮ Not too difficult, but lacks insight

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation

Find the constant c that minimizes E (x − c)2

Note:

◮ Expected square is a Hilbert space norm

◮ Constants are a closed subspace in vector space of random variables

◮ c determined uniquely by E[ (x − c) αc ] = 0 for all α ∈ R

◮ c = E[ x ]

Alternative:

◮ Expand into quadratic function of c and minimize with calculus

◮ Not too difficult, but lacks insight

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation

Find the constant c that minimizes E (x − c)2

Note:

◮ Expected square is a Hilbert space norm

◮ Constants are a closed subspace in vector space of random variables

◮ c determined uniquely by E[ (x − c) αc ] = 0 for all α ∈ R

◮ c = E[ x ]

Alternative:

◮ Expand into quadratic function of c and minimize with calculus

◮ Not too difficult, but lacks insight

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation

Find the constant c that minimizes E (x − c)2

Note:

◮ Expected square is a Hilbert space norm

◮ Constants are a closed subspace in vector space of random variables

◮ c determined uniquely by E[ (x − c) αc ] = 0 for all α ∈ R

◮ c = E[ x ]

◮ Expand into quadratic function of c and minimize with calculus

◮ Not too difficult, but lacks insight

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation

Find the constant c that minimizes E (x − c)2

Note:

◮ Expected square is a Hilbert space norm

◮ Constants are a closed subspace in vector space of random variables

◮ c determined uniquely by E[ (x − c) αc ] = 0 for all α ∈ R

◮ c = E[ x ]

◮ Expand into quadratic function of c and minimize with calculus

◮ Not too difficult, but lacks insight

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation

Consider: Jointly wide-sense stationary

y b

x

discrete-time stochastic processes x and y h

Find the linear −

shift-invariant

filter h that x

+ e

minimizes E |xn − b xn |2 where b x =h∗y

Note:

◮ Expected squared absolute value is a Hilbert space norm

◮ LSI filtering puts b

xn in a closed subspace

Solution: Use orthogonality (extended for processes)

◮ h determined uniquely by relation between cross- and autocorrelations:

cx,bx,k = abx,k , k ∈Z

Cx,y (e jω )

◮ DTFT-domain version: H(e jω ) = , ω∈R

Ay (e jω )

Alternative:

◮ Expand into quadratic function of h and minimize with calculus

◮ Messy (!)—especially in complex case—and lacks insight

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation

Consider: Jointly wide-sense stationary

y b

x

discrete-time stochastic processes x and y h

Find the linear −

shift-invariant

filter h that x

+ e

minimizes E |xn − b xn |2 where b x =h∗y

Note:

◮ Expected squared absolute value is a Hilbert space norm

◮ LSI filtering puts b

xn in a closed subspace

Solution: Use orthogonality (extended for processes)

◮ h determined uniquely by relation between cross- and autocorrelations:

cx,bx,k = abx,k , k ∈Z

Cx,y (e jω )

◮ DTFT-domain version: H(e jω ) = , ω∈R

Ay (e jω )

Alternative:

◮ Expand into quadratic function of h and minimize with calculus

◮ Messy (!)—especially in complex case—and lacks insight

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation

Consider: Jointly wide-sense stationary

y b

x

discrete-time stochastic processes x and y h

Find the linear −

shift-invariant

filter h that x

+ e

minimizes E |xn − b xn |2 where b x =h∗y

Note:

◮ Expected squared absolute value is a Hilbert space norm

◮ LSI filtering puts b

xn in a closed subspace

Solution: Use orthogonality (extended for processes)

◮ h determined uniquely by relation between cross- and autocorrelations:

cx,bx,k = abx,k , k ∈Z

Cx,y (e jω )

◮ DTFT-domain version: H(e jω ) = , ω∈R

Ay (e jω )

Alternative:

◮ Expand into quadratic function of h and minimize with calculus

◮ Messy (!)—especially in complex case—and lacks insight

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation

Consider: Jointly wide-sense stationary

y b

x

discrete-time stochastic processes x and y h

Find the linear −

shift-invariant

filter h that x

+ e

minimizes E |xn − b xn |2 where b x =h∗y

Note:

◮ Expected squared absolute value is a Hilbert space norm

◮ LSI filtering puts b

xn in a closed subspace

Solution: Use orthogonality (extended for processes)

◮ h determined uniquely by relation between cross- and autocorrelations:

cx,bx,k = abx,k , k ∈Z

◮ DTFT-domain version: H(e jω ) = , ω∈R

Ay (e jω )

Alternative:

◮ Expand into quadratic function of h and minimize with calculus

◮ Messy (!)—especially in complex case—and lacks insight

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation

Local averaging

Z k+ 21

2 2

A : L (R) → ℓ (Z) (Ax)k = x(t)dt

k− 12

has adjoint A∗ : ℓ2 (Z) → L2 (R) that produces staircase function

x(t) yn = (Ax)n

10 10

5 5

5 10 15t 5 10 15n

−5 −5

−10 −10 x(t)

10

5

x (t) = (A∗ Ax)(t)

b x(t), yn , b

x (t)

t

10 10 5 10 15

5 5 −5

5 10 15t 5 10 15t −10

−5 −5

−10 −10

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation

1 2

(−π, π) (π, π)

2

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation

(−π, π) (π, π)

3

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation

1 2

(−π, π) (π, π)

2

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

Bases

Definition (Basis)

Φ = {ϕk }k∈K ⊂ V is a basis when

1 Φ is linearly independent and

2 Φ is complete in V : V = span(Φ)

X

Expansion formula: for any x ∈ V , x= αk ϕk

k∈K

{αk }k∈K : is unique

αk : expansion coefficients

Example

The standard

basis for RN T

ek = 0 0 · · · 0 1 0 0 ··· 0 , k = 0, . . . , N − 1

N−1

X

any x ∈ RN , x= xk ek

k=0

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

Bases

Definition (Basis)

Φ = {ϕk }k∈K ⊂ V is a basis when

1 Φ is linearly independent and

2 Φ is complete in V : V = span(Φ)

X

Expansion formula: for any x ∈ V , x= αk ϕk

k∈K

{αk }k∈K : is unique

αk : expansion coefficients

Example

The standard

basis for RN T

ek = 0 0 · · · 0 1 0 0 ··· 0 , k = 0, . . . , N − 1

N−1

X

any x ∈ RN , x= xk ek

k=0

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

Bases

Definition (Basis)

Φ = {ϕk }k∈K ⊂ V is a basis when

1 Φ is linearly independent and

2 Φ is complete in V : V = span(Φ)

X

Expansion formula: for any x ∈ V , x= αk ϕk

k∈K

{αk }k∈K : is unique

αk : expansion coefficients

Example

The standard

basis for RN T

ek = 0 0 · · · 0 1 0 0 ··· 0 , k = 0, . . . , N − 1

N−1

X

any x ∈ RN , x= xk ek

k=0

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

Bases

Examples

α1 α1 ϕ1

x x

ϕ

e1 ϕ1

ϕ1 ϕ1

ϕ0

ϕ0 ϕ0

α0 α0 ϕ0 ϕ

e0

Orthonormal basis Biorthogonal basis . . . with dual basis

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

Synthesis operator

X

◮ Φ : ℓ2 (K) → H Φα = αk ϕk

k∈K

X X

hΦα, y i = αk ϕk , y = αk hy , ϕk i∗

k∈K k∈K

Analysis operator

◮ Φ∗ : H → ℓ2 (K) (Φ∗ x)k = hx, ϕk i, k ∈ K

Note that the analysis operator is the adjoint of the synthesis operator

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

Synthesis operator

X

◮ Φ : ℓ2 (K) → H Φα = αk ϕk

k∈K

X X

hΦα, y i = αk ϕk , y = αk hy , ϕk i∗

k∈K k∈K

Analysis operator

◮ Φ∗ : H → ℓ2 (K) (Φ∗ x)k = hx, ϕk i, k ∈ K

Note that the analysis operator is the adjoint of the synthesis operator

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

Synthesis operator

X

◮ Φ : ℓ2 (K) → H Φα = αk ϕk

k∈K

X X

hΦα, y i = αk ϕk , y = αk hy , ϕk i∗

k∈K k∈K

Analysis operator

◮ Φ∗ : H → ℓ2 (K) (Φ∗ x)k = hx, ϕk i, k ∈ K

Note that the analysis operator is the adjoint of the synthesis operator

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

Orthonormal bases

Φ = {ϕk }k∈K ⊂ H is an orthonormal basis for H when

1 Φ is a basis for H and

2 Φ is an orthonormal set

hϕi , ϕk i = δi −k for all i, k ∈ K

then Φ is an orthogonal basis for H

If we also have kϕk k = 1, then Φ is an orthonormal basis

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

Orthonormal bases

Φ = {ϕk }k∈K ⊂ H is an orthonormal basis for H when

1 Φ is a basis for H and

2 Φ is an orthonormal set

hϕi , ϕk i = δi −k for all i, k ∈ K

then Φ is an orthogonal basis for H

If we also have kϕk k = 1, then Φ is an orthonormal basis

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

Orthonormal bases

Φ = {ϕk }k∈K ⊂ H is an orthonormal basis for H when

1 Φ is a basis for H and

2 Φ is an orthonormal set

hϕi , ϕk i = δi −k for all i, k ∈ K

then Φ is an orthogonal basis for H

If we also have kϕk k = 1, then Φ is an orthonormal basis

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

Φ = {ϕk }k∈K orthonormal basis for H, then for any x ∈ H:

αk = hx, ϕk i for k ∈ K, or α = Φ∗ x, and α is unique

X

Synthesis: x = hx, ϕk iϕk = Φα = ΦΦ∗ x

k∈K

Example

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

Φ = {ϕk }k∈K orthonormal basis for H, then for any x ∈ H:

αk = hx, ϕk i for k ∈ K, or α = Φ∗ x, and α is unique

X

Synthesis: x = hx, ϕk iϕk = Φα = ΦΦ∗ x

k∈K

Example

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

Φ = {ϕk }k∈K orthonormal basis for H, then for any x ∈ H:

αk = hx, ϕk i for k ∈ K, or α = Φ∗ x, and α is unique

X

Synthesis: x = hx, ϕk iϕk = Φα = ΦΦ∗ x

k∈K

Example

ϕ2 = e2 x

hx, ϕ1 i

ϕ1 = e1

ϕ0 = e0

hx, ϕ0 i

x01

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

Φ = {ϕk }k∈K orthonormal basis for H

X

kxk2 = |hx, ϕk i|2 = kΦ∗ xk2 = kαk2

k∈K

In general:

hx, y i = hΦ∗ x, Φ∗ y i = hα, βi

where αk = hx, ϕk i, βk = hy , ϕk i

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

Φ = {ϕk }k∈K orthonormal basis for H

X

kxk2 = |hx, ϕk i|2 = kΦ∗ xk2 = kαk2

k∈K

In general:

hx, y i = hΦ∗ x, Φ∗ y i = hα, βi

where αk = hx, ϕk i, βk = hy , ϕk i

y

β

Φ∗

x

θ Φ

θ α

H ℓ2 (K)

Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 52 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

Φ = {ϕk }k∈K orthonormal basis for H

X

kxk2 = |hx, ϕk i|2 = kΦ∗ xk2 = kαk2

k∈K

In general:

hx, y i = hΦ∗ x, Φ∗ y i = hα, βi

where αk = hx, ϕk i, βk = hy , ϕk i

y

β

Φ∗

x

θ Φ

θ α

H ℓ2 (K) isometry

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

Theorem

Φ = {ϕk }k∈I ⊂ H, I ⊂ K

X

PI x = hx, ϕk iϕk = ΦI Φ∗I x

k∈I

M

H = S{k} where S{k} = span(ϕk )

k∈K

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

Theorem

Φ = {ϕk }k∈I ⊂ H, I ⊂ K

X

PI x = hx, ϕk iϕk = ΦI Φ∗I x

k∈I

M

H = S{k} where S{k} = span(ϕk )

k∈K

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

Definition

Biorthogonal pairs of bases

e = {ϕ

Φ = {ϕk }k∈K ⊂ H and Φ ek }k∈K ⊂ H is a biorthogonal pair of bases

when

1 e are both bases for H

Φ and Φ

2 e are biorthogonal: hϕi , ϕ

Φ and Φ ek i = δi −k for all i, k ∈ K

e are interchangeable

Roles of Φ and Φ

Example

1 0 1 1 0 1

ϕ0 = 1, ϕ1 = 1, ϕ2 = 1, Φ = 1 1 1

0 1 1 0 1 1

0 −1 1 0 1 −1

−1

e0 = 1 ,

ϕ e1 = 1 ,

ϕ e2 = −1,

ϕ Φ = −1 1 0

−1 0 1 1 −1 1

Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 54 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

Definition

Biorthogonal pairs of bases

e = {ϕ

Φ = {ϕk }k∈K ⊂ H and Φ ek }k∈K ⊂ H is a biorthogonal pair of bases

when

1 e are both bases for H

Φ and Φ

2 e are biorthogonal: hϕi , ϕ

Φ and Φ ek i = δi −k for all i, k ∈ K

e are interchangeable

Roles of Φ and Φ

Example

1 0 1 1 0 1

ϕ0 = 1, ϕ1 = 1, ϕ2 = 1, Φ = 1 1 1

0 1 1 0 1 1

0 −1 1 0 1 −1

−1

e0 = 1 ,

ϕ e1 = 1 ,

ϕ e2 = −1,

ϕ Φ = −1 1 0

−1 0 1 1 −1 1

Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 54 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

Definition

Biorthogonal pairs of bases

e = {ϕ

Φ = {ϕk }k∈K ⊂ H and Φ ek }k∈K ⊂ H is a biorthogonal pair of bases

when

1 e are both bases for H

Φ and Φ

2 e are biorthogonal: hϕi , ϕ

Φ and Φ ek i = δi −k for all i, k ∈ K

e are interchangeable

Roles of Φ and Φ

Example

1 0 1 1 0 1

ϕ0 = 1, ϕ1 = 1, ϕ2 = 1, Φ = 1 1 1

0 1 1 0 1 1

0 −1 1 0 1 −1

−1

e0 = 1 ,

ϕ e1 = 1 ,

ϕ e2 = −1,

ϕ Φ = −1 1 0

−1 0 1 1 −1 1

Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 54 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

Theorem

e = {ϕ

Φ = {ϕk }k∈K , Φ ek }k∈K biorthogonal pair of bases for H

Any x ∈ H has expansion coefficients

αk = hx, ϕ e ∗x

ek i for k ∈ K, or α = Φ

X

Synthesis: x = hx, ϕ

ek iϕk = Φα = ΦΦ e ∗x

k∈K

X

Also x = hx, ϕk iϕ

ek

k∈K

e1 ϕ1

ϕ

ϕ0

ϕ

e0

Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 55 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

Theorem

e = {ϕ

Φ = {ϕk }k∈K , Φ ek }k∈K biorthogonal pair of bases for H

Any x ∈ H has expansion coefficients

αk = hx, ϕ e ∗x

ek i for k ∈ K, or α = Φ

X

Synthesis: x = hx, ϕ

ek iϕk = Φα = ΦΦ e ∗x

k∈K

X

Also x = hx, ϕk iϕ

ek

k∈K

e1 ϕ1

ϕ

ϕ0

ϕ

e0

Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 55 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

Theorem

e = {ϕ

Φ = {ϕk }k∈K , Φ ek }k∈K biorthogonal pair of bases for H

Any x ∈ H has expansion coefficients

αk = hx, ϕ e ∗x

ek i for k ∈ K, or α = Φ

X

Synthesis: x = hx, ϕ

ek iϕk = Φα = ΦΦ e ∗x

k∈K

X

Also x = hx, ϕk iϕ

ek

k∈K

e1 ϕ1

ϕ

ϕ0

ϕ

e0

Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 55 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

X

x = hx, ϕk iϕk = ΦΦ∗ x without {ϕk }k∈J being linearly independent

k∈J

Example

R2

1 0 −1

Given Φ =

ϕ2 = e2 x 0 1 −1

hx, ϕ1 i

ϕ0 = e0

ϕ1 = e1 {ϕ0 , ϕ1 , ϕ2 } is a frame for R2

hx, ϕ0 i

x01

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

Assume x = Φα = Ψβ

Then β = Ψ∗ x = Ψ∗ Φα

CΦ,Ψ : ℓ2 (K) → ℓ2 (K) s.t. CΦ,Ψ = Ψ∗ Φ

As a matrix

.. .. ..

. . .

· · · hϕ−1 , ψ−1 i hϕ0 , ψ−1 i hϕ1 , ψ−1 i · · ·

CΦ,Ψ = · · · hϕ−1 , ψ0 i hϕ0 , ψ0 i hϕ1 , ψ0 i · · ·

· · · hϕ−1 , ψ1 i hϕ0 , ψ1 i hϕ1 , ψ1 i · · ·

.. .. ..

. . .

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

Assume x = Φα = Ψβ

Then β = Ψ∗ x = Ψ∗ Φα

CΦ,Ψ : ℓ2 (K) → ℓ2 (K) s.t. CΦ,Ψ = Ψ∗ Φ

As a matrix

.. .. ..

. . .

· · · hϕ−1 , ψ−1 i hϕ0 , ψ−1 i hϕ1 , ψ−1 i · · ·

CΦ,Ψ = · · · hϕ−1 , ψ0 i hϕ0 , ψ0 i hϕ1 , ψ0 i · · ·

· · · hϕ−1 , ψ1 i hϕ0 , ψ1 i hϕ1 , ψ1 i · · ·

.. .. ..

. . .

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

Assume x = Φα = Ψβ

Then β = Ψ∗ x = Ψ∗ Φα

CΦ,Ψ : ℓ2 (K) → ℓ2 (K) s.t. CΦ,Ψ = Ψ∗ Φ

As a matrix

.. .. ..

. . .

· · · hϕ−1 , ψ−1 i hϕ0 , ψ−1 i hϕ1 , ψ−1 i · · ·

CΦ,Ψ = · · · hϕ−1 , ψ0 i hϕ0 , ψ0 i hϕ1 , ψ0 i · · ·

· · · hϕ−1 , ψ1 i hϕ0 , ψ1 i hϕ1 , ψ1 i · · ·

.. .. ..

. . .

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

Assume x = Φα = Ψβ

Then β = Ψ∗ x = Ψ∗ Φα

CΦ,Ψ : ℓ2 (K) → ℓ2 (K) s.t. CΦ,Ψ = Ψ∗ Φ

.. .. ..

. . .

· · · hϕ−1 , ψ−1 i hϕ0 , ψ−1 i hϕ1 , ψ−1 i · · ·

CΦ,Ψ = · · · hϕ−1 , ψ0 i hϕ0 , ψ0 i hϕ1 , ψ0 i · · ·

· · · hϕ−1 , ψ1 i hϕ0 , ψ1 i hϕ1 , ψ1 i · · ·

.. .. ..

. . .

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

Let y = Ax with A : H → H

How are expansion coefficients of x and y related?

◮ {ϕk }k∈K orthonormal basis of H

◮ x = Φα, y = Φβ

Matrix representation allows computation of A directly on coefficient

sequences

Γ : ℓ2 (K) → ℓ2 (K) s.t. β = Γα

As a matrix:

.. .. ..

. . .

· · · hAϕ−1 , ϕ−1 i hAϕ0 , ϕ−1 i hAϕ1 , ϕ−1 i · · ·

Γ = · · · hAϕ−1 , ϕ0 i hAϕ0 , ϕ0 i hAϕ1 , ϕ0 i · · ·

· · · hAϕ−1 , ϕ1 i hAϕ0 , ϕ1 i hAϕ1 , ϕ1 i · · ·

.. .. ..

. . .

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

Let y = Ax with A : H → H

How are expansion coefficients of x and y related?

◮ {ϕk }k∈K orthonormal basis of H

◮ x = Φα, y = Φβ

Matrix representation allows computation of A directly on coefficient

sequences

Γ : ℓ2 (K) → ℓ2 (K) s.t. β = Γα

As a matrix:

.. .. ..

. . .

· · · hAϕ−1 , ϕ−1 i hAϕ0 , ϕ−1 i hAϕ1 , ϕ−1 i · · ·

Γ = · · · hAϕ−1 , ϕ0 i hAϕ0 , ϕ0 i hAϕ1 , ϕ0 i · · ·

· · · hAϕ−1 , ϕ1 i hAϕ0 , ϕ1 i hAϕ1 , ϕ1 i · · ·

.. .. ..

. . .

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

Let y = Ax with A : H → H

How are expansion coefficients of x and y related?

◮ {ϕk }k∈K orthonormal basis of H

◮ x = Φα, y = Φβ

Matrix representation allows computation of A directly on coefficient

sequences

Γ : ℓ2 (K) → ℓ2 (K) s.t. β = Γα

As a matrix:

.. .. ..

. . .

· · · hAϕ−1 , ϕ−1 i hAϕ0 , ϕ−1 i hAϕ1 , ϕ−1 i · · ·

Γ = · · · hAϕ−1 , ϕ0 i hAϕ0 , ϕ0 i hAϕ1 , ϕ0 i · · ·

· · · hAϕ−1 , ϕ1 i hAϕ0 , ϕ1 i hAϕ1 , ϕ1 i · · ·

.. .. ..

. . .

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

H H

A

x • • y

Φ∗ Φ Φ∗ Φ

α • • β

Γ

2

ℓ (K) ℓ2 (K)

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

H H

A

x • • y

Φ∗ Φ Φ∗ Φ

α • • β

Γ

2

ℓ (K) ℓ2 (K)

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

Example

Z 2(ℓ+1)

1

Let A : H0 → H1 , y (t) = Ax(t) = 2 x(τ ) dτ

2ℓ

for 2ℓ ≤ t < 2(ℓ + 1), ℓ∈Z

H0 : piecewise-constant, finite-energy functions with breakpoints at integers

H1 : piecewise-constant, finite-energy functions with breakpoints at even

integers

1, for t ∈ I ;

Given χI (t) =

0, otherwise

Let Φ = {ϕk (t)}k∈Z = {χ[k,k+1) (t)}k∈Z

Ψ = {ψi (t)}i ∈Z = { √12 χ[2i ,2(i +1))(t)}i ∈Z

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

Example (Cont.)

Z 2

1 1 √1 √1

Aϕ0 (t) = 2 χ[0,2) (t) ⇒ hAϕ0 , ψ0 i = 2 2 dτ = 2

0

.. .. .. .. .. .. ..

. . . . . . .

· · · 1 1 0 0 0 0 · · ·

Then Γ = √12

· · · 0 0 1 1 0 0 · · ·

· · · 0 0 0 0 1 1 · · ·

.. .. .. .. .. .. ..

. . . . . . .

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

Discrete-time systems

respect to the standard basis

.. .. .. .. ..

.. .. .. ..

. . . . . . . . .

y−2 . . . h0 h−1 h−2 h−3 h−4 . . . x−2

y−1 . . . h1 h0 h−1 h−2 h−3 . . . x−1

y =

y 0 = . . .

h2 h1 h0 h−1 h−2 . . .

x0 = Hx

y1 . . . h3 h2 h1 h0 h−1

. . . 1

x

y2 . . . . . .

h4 h3 h2 h1 h0 x2

.. .. .. .. .. .. .. .. .

..

. . . . . . . .

| {z }

H

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

Discrete-time systems

.. .. .. .. .. .

. . . . . ..

.

.. h0∗ h1∗ h2∗ h3∗ h4∗

. ..

.. ∗

h−1 h0∗

h1∗ ∗

h2 h3 ∗ .

. .

H =

∗

.

. ∗

h−2 ∗

h−1 h0∗ h1∗ h2∗ . .

. . . .

. ∗

h−3 ∗ ∗

h−2 h−1 h0 h1 ∗ ∗ .

..

∗

h−4 ∗

h−3 ∗

h−2 ∗

h−1 h0∗ .

. .. .. .. .. .

.. . . . . ..

∗

Adjoint of filtering by hn is filtering by h−n

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

y0 h0 hN−1 hN−2 . . . h1 x0

y1 h1 h0 hN−1 . . . h2 x1

y2 h2 h h0 ...

h3 x2

y = = 1 = Hx

.. .. .. .. .. .. ..

. . . . . . .

yN−1 hN−1 hN−2 hN−3 . . . h0 xN−1

| {z }

H

◮ Eigenvectors (-sequences, -signals) lead to diagonal representation of H

◮ Fourier transform follows logically from the class of operators

◮ Convolution theorem follows logically from the definition of the Fourier

transform

Example Lecture: Sampling and Interpolation Motivation

Sampling and interpolation bridge the analog and digital worlds

Sampling: discrete-time sequence from a continuous-time function

Interpolation: continuous-time function from a discrete-time sequence

Doing all computation in discrete time is the essence of digital signal processing:

yn wn

x(t) Sampling DT processing Interpolation v (t)

y (t) v (t)

xn Interpolation CT channel Sampling b

xn

Example Lecture: Sampling and Interpolation Motivation

◮ Don’t/can’t sample at one point

◮ Causal, non-ideal filters

◮ Multichannel, time-interleaved

◮ Digital photography

◮ Computational imaging (magnetic resonance, space-from-time, ultrasound,

computed tomography, synthetic aperture radar, . . . )

◮ Reflection seismology, acoustic tomography, . . .

1 Understand classical sampling as a special case of a Hilbert space theory

2 Gain a generalizable understanding of sampling

Example Lecture: Sampling and Interpolation Motivation

◮ Don’t/can’t sample at one point

◮ Causal, non-ideal filters

◮ Multichannel, time-interleaved

◮ Digital photography

◮ Computational imaging (magnetic resonance, space-from-time, ultrasound,

computed tomography, synthetic aperture radar, . . . )

◮ Reflection seismology, acoustic tomography, . . .

1 Understand classical sampling as a special case of a Hilbert space theory

2 Gain a generalizable understanding of sampling

Example Lecture: Sampling and Interpolation Motivation

◮ Don’t/can’t sample at one point

◮ Causal, non-ideal filters

◮ Multichannel, time-interleaved

◮ Digital photography

◮ Computational imaging (magnetic resonance, space-from-time, ultrasound,

computed tomography, synthetic aperture radar, . . . )

◮ Reflection seismology, acoustic tomography, . . .

1 Understand classical sampling as a special case of a Hilbert space theory

2 Gain a generalizable understanding of sampling

Example Lecture: Sampling and Interpolation Motivation

◮ Don’t/can’t sample at one point

◮ Causal, non-ideal filters

◮ Multichannel, time-interleaved

◮ Digital photography

◮ Computational imaging (magnetic resonance, space-from-time, ultrasound,

computed tomography, synthetic aperture radar, . . . )

◮ Reflection seismology, acoustic tomography, . . .

1 Understand classical sampling as a special case of a Hilbert space theory

2 Gain a generalizable understanding of sampling

Example Lecture: Sampling and Interpolation Classical View and Historical Notes

Bandwidth

A sequence x is bandlimited when there exists ω0 ∈ [0, 2π) such that the

discrete-time Fourier transform X satisfies

0.5 æ æ

0.5 0.5

æ

ææ æææ æææææ

æ æ æ ææ æ ææ æ æ

æ ææ æ æ æ

æ æ æ æ æ æ æ æ æ æ

æ æ æ æ æ

æ æ æ æ æææææ æ æ æ æææ

æ

æ

æ æ æ æ ææææ

æ æ æ æ æ æ æ ææ

æ æ æ æ æ æ æ æ ææ ææ

æ æ ææ æ æ æ æ æ æ æ

ææ ææ æææææ

æ æ ææ

æ æ

æ æ ææ æ

æ ææ

ææ æ

æ

-0.5 -0.5 -0.5

ω0 = π ω0 = π/2 ω0 = π/4

Example Lecture: Sampling and Interpolation Classical View and Historical Notes

Bandwidth

A function x is bandlimited when there exists ω0 ∈ [0, ∞) such that the Fourier

transform X satisfies

The set of sequences in ℓ2 (Z) with bandwidth at most ω0 and the set of functions

in L2 (R) with bandwidth at most ω0 are denoted BL[−ω0 /2, ω0 /2]

A bandlimited set is always a subspace

◮ Subspace is closed in Hilbert space ℓ2 (Z) or L2 (R)

Example Lecture: Sampling and Interpolation Classical View and Historical Notes

Bandwidth

A function x is bandlimited when there exists ω0 ∈ [0, ∞) such that the Fourier

transform X satisfies

The set of sequences in ℓ2 (Z) with bandwidth at most ω0 and the set of functions

in L2 (R) with bandwidth at most ω0 are denoted BL[−ω0 /2, ω0 /2]

A bandlimited set is always a subspace

◮ Subspace is closed in Hilbert space ℓ2 (Z) or L2 (R)

Example Lecture: Sampling and Interpolation Classical View and Historical Notes

Bandwidth

A function x is bandlimited when there exists ω0 ∈ [0, ∞) such that the Fourier

transform X satisfies

The set of sequences in ℓ2 (Z) with bandwidth at most ω0 and the set of functions

in L2 (R) with bandwidth at most ω0 are denoted BL[−ω0 /2, ω0 /2]

A bandlimited set is always a subspace

◮ Subspace is closed in Hilbert space ℓ2 (Z) or L2 (R)

Example Lecture: Sampling and Interpolation Classical View and Historical Notes

Classical Sampling

6 0;

(sin t)/t, for t =

Recall: sinc(t) =

1, for t = 0

Let x be a function and let T > 0. Define

X

b

x (t) = x(nT ) sinc( Tπ (t − nT )).

n∈Z

x = x.

Need two samples per cycle of fastest component

Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 69 / 96

Example Lecture: Sampling and Interpolation Classical View and Historical Notes

Classical Sampling

6 0;

(sin t)/t, for t =

Recall: sinc(t) =

1, for t = 0

Let x be a function and let T > 0. Define

X

b

x (t) = x(nT ) sinc( Tπ (t − nT )).

n∈Z

x = x.

Need two samples per cycle of fastest component

Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 69 / 96

Example Lecture: Sampling and Interpolation Classical View and Historical Notes

History

Many names:

◮ Shannon sampling theorem

◮ Nyquist–Shannon sampling theorem

◮ Nyquist–Shannon–Kotel’nikov sampling theorem

◮ Whittaker–Shannon–Kotel’nikov sampling theorem

◮ Whittaker–Nyquist–Shannon–Kotel’nikov sampling theorem

Well-known people associated with sampling (but less often so):

◮ Cauchy (1841) – apparently not true

◮ Borel (1897)

◮ de la Vallée Poussin (1908)

◮ E. T. Whittaker (1915)

◮ J. M. Whittaker (1927)

◮ Gabor (1946)

Less-known people, almost lost to history:

◮ Ogura (1920)

◮ Küpfmüller (Küpfmüller filter)

◮ Raabe [PhD 1939] (assistant to Küpfmüller)

◮ Someya (1949)

◮ Weston (1949)

Best approach: use no names?

◮ Cardinal Theorem of interpolation theory

Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 70 / 96

Example Lecture: Sampling and Interpolation Classical View and Historical Notes

History

Many names:

◮ Shannon sampling theorem

◮ Nyquist–Shannon sampling theorem

◮ Nyquist–Shannon–Kotel’nikov sampling theorem

◮ Whittaker–Shannon–Kotel’nikov sampling theorem

◮ Whittaker–Nyquist–Shannon–Kotel’nikov sampling theorem

Well-known people associated with sampling (but less often so):

◮ Cauchy (1841) – apparently not true

◮ Borel (1897)

◮ de la Vallée Poussin (1908)

◮ E. T. Whittaker (1915)

◮ J. M. Whittaker (1927)

◮ Gabor (1946)

Less-known people, almost lost to history:

◮ Ogura (1920)

◮ Küpfmüller (Küpfmüller filter)

◮ Raabe [PhD 1939] (assistant to Küpfmüller)

◮ Someya (1949)

◮ Weston (1949)

Best approach: use no names?

◮ Cardinal Theorem of interpolation theory

Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 70 / 96

Example Lecture: Sampling and Interpolation Classical View and Historical Notes

History

Many names:

◮ Shannon sampling theorem

◮ Nyquist–Shannon sampling theorem

◮ Nyquist–Shannon–Kotel’nikov sampling theorem

◮ Whittaker–Shannon–Kotel’nikov sampling theorem

◮ Whittaker–Nyquist–Shannon–Kotel’nikov sampling theorem

Well-known people associated with sampling (but less often so):

◮ Cauchy (1841) – apparently not true

◮ Borel (1897)

◮ de la Vallée Poussin (1908)

◮ E. T. Whittaker (1915)

◮ J. M. Whittaker (1927)

◮ Gabor (1946)

Less-known people, almost lost to history:

◮ Ogura (1920)

◮ Küpfmüller (Küpfmüller filter)

◮ Raabe [PhD 1939] (assistant to Küpfmüller)

◮ Someya (1949)

◮ Weston (1949)

Best approach: use no names?

◮ Cardinal Theorem of interpolation theory

Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 70 / 96

Example Lecture: Sampling and Interpolation Classical View and Historical Notes

Many names:

◮ Shannon sampling theorem

◮ Nyquist–Shannon sampling theorem

◮ Nyquist–Shannon–Kotel’nikov sampling theorem

◮ Whittaker–Shannon–Kotel’nikov sampling theorem

◮ Whittaker–Nyquist–Shannon–Kotel’nikov sampling theorem

Well-known people associated with sampling (but less often so):

◮ Cauchy (1841) – apparently not true

◮ Borel (1897)

◮ de la Vallée Poussin (1908)

◮ E. T. Whittaker (1915)

◮ J. M. Whittaker (1927)

◮ Gabor (1946)

Less-known people, almost lost to history:

◮ Ogura (1920)

◮ Küpfmüller (Küpfmüller filter)

◮ Raabe [PhD 1939] (assistant to Küpfmüller)

◮ Someya (1949)

◮ Weston (1949)

Best approach: use no names?

◮ Cardinal Theorem of interpolation theory

Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 70 / 96

Example Lecture: Sampling and Interpolation Classical View and Historical Notes

P

Let b

x (t) = n∈Z x(nT ) g (t − nT ) [will deduce that g should be sinc]

Example Lecture: Sampling and Interpolation Classical View and Historical Notes

P

Let b

x (t) = n∈Z x(nT ) g (t − nT ) [will deduce that g should be sinc]

R∞

Since x(nT ) g (t − nT ) = −∞ x(τ ) g (t − τ ) δ(τ − nT ) dτ ,

Z ∞ X

b

x (t) = g (t − τ ) x(τ ) δ(τ − nT ) dτ

−∞ n∈Z

Example Lecture: Sampling and Interpolation Classical View and Historical Notes

P

Let b

x (t) = n∈Z x(nT ) g (t − nT ) [will deduce that g should be sinc]

R∞

Since x(nT ) g (t − nT ) = −∞ x(τ ) g (t − τ ) δ(τ − nT ) dτ ,

Z ∞ X

b

x (t) = g (t − τ ) x(τ ) δ(τ − nT ) dτ

−∞ n∈Z

X

FT 2π X 2π

δ(t − nT ) ←→ δ ω− k

T T

n∈Z k∈Z

Example Lecture: Sampling and Interpolation Classical View and Historical Notes

P

Let b

x (t) = n∈Z x(nT ) g (t − nT ) [will deduce that g should be sinc]

R∞

Since x(nT ) g (t − nT ) = −∞ x(τ ) g (t − τ ) δ(τ − nT ) dτ ,

Z ∞ X

b

x (t) = g (t − τ ) x(τ ) δ(τ − nT ) dτ

−∞ n∈Z

| {z }

h(τ )

X

FT 2π X 2π

δ(t − nT ) ←→ δ ω− k

T T

n∈Z k∈Z

X

b (ω) = G (ω) 1

X X ω−

2π

k

T T

k∈Z

| {z }

H(ω)

Example Lecture: Sampling and Interpolation Classical View and Historical Notes

X

b (ω) = 1 G (ω)

X X ω−

2π

k

T T

k∈Z

Reconstruction X

b (ω) = X (ω) for all ω?

How can we have X

◮ x ∈ BL[−π/T , π/T ] implies replicas do not overlap

T , for |ω| < π/T ;

◮ G (ω) = selects “desired” replica with correct gain

0, for t = 0

Example Lecture: Sampling and Interpolation Classical View and Historical Notes

Dissatisfaction

Mathematical rigor of derivation:

X

FT 2π X 2π

δ(t − nT ) ←→ δ ω− k

T T

n∈Z k∈Z

X

b

x (t) = x(nT ) sinc( Tπ (t − nT ))

n∈Z

◮ Very slow decay of terms makes truncation accuracy poor

Technological implementability:

◮ Point measurements difficult to approximate physically

◮ Causality of reconstruction

Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 73 / 96

Example Lecture: Sampling and Interpolation Classical View and Historical Notes

Dissatisfaction

Mathematical rigor of derivation:

X

FT 2π X 2π

δ(t − nT ) ←→ δ ω− k

T T

n∈Z k∈Z

X

b

x (t) = x(nT ) sinc( Tπ (t − nT ))

n∈Z

◮ Very slow decay of terms makes truncation accuracy poor

Technological implementability:

◮ Point measurements difficult to approximate physically

◮ Causality of reconstruction

Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 73 / 96

Example Lecture: Sampling and Interpolation Classical View and Historical Notes

Dissatisfaction

Mathematical rigor of derivation:

X

FT 2π X 2π

δ(t − nT ) ←→ δ ω− k

T T

n∈Z k∈Z

X

b

x (t) = x(nT ) sinc( Tπ (t − nT ))

n∈Z

◮ Very slow decay of terms makes truncation accuracy poor

Technological implementability:

◮ Point measurements difficult to approximate physically

◮ Causality of reconstruction

Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 73 / 96

Example Lecture: Sampling and Interpolation Operator View

For a fixed positive T and interpolation postfilter g (t), let Φ : ℓ2 (Z) → L2 (R) be

given by X

(Φ y )(t) = yn √1T g (t/T − n), t∈R

n∈Z

Φ

y g (t) b

x

T

P

For simplicity, we will consider only T = 1: (Φ y )(t) = n∈Z yn g (t − n)

Example Lecture: Sampling and Interpolation Operator View

Reconstruction space

Range of interpolation operator has special form

A subspace W ⊂ L2 (R) is a shift-invariant subspace with respect to shift T ∈ R+

when x(t) ∈ W implies x(t − kT ) ∈ W for every integer k. In addition,

w ∈ L2 (R) is called a generator of W when W = span({w (t − kT )}k∈Z).

Example Lecture: Sampling and Interpolation Operator View

For a fixed positive T and sampling prefilter g ∗ (−t), let Φ∗ : L2 (R) → ℓ2 (Z) be

given by

(Φ∗ x)n = hx(t), √1T g (t/T − n)it , n∈Z

Φ∗

T

x g ∗ (−t) y

Example Lecture: Sampling and Interpolation Operator View

Theorem

Sampling and interpolation operators are adjoints

X

hΦ∗ x, y i = hx(t), g (t − n)it yn∗

n∈Z

X Z ∞

= x(t) g ∗ (t − n) dt yn∗

n∈Z −∞

Z !

∞ X

∗

= x(t) g (t − n) yn∗ dt

−∞ n∈Z

Z !∗

∞ X

= x(t) yn g (t − n) dt

−∞ n∈Z

= hx, Φ y i

Example Lecture: Sampling and Interpolation Operator View

Theorem

Sampling and interpolation operators are adjoints

X

hΦ∗ x, y i = hx(t), g (t − n)it yn∗

n∈Z

X Z ∞

= x(t) g ∗ (t − n) dt yn∗

n∈Z −∞

Z !

∞ X

∗

= x(t) g (t − n) yn∗ dt

−∞ n∈Z

Z !∗

∞ X

= x(t) yn g (t − n) dt

−∞ n∈Z

= hx, Φ y i

Example Lecture: Sampling and Interpolation Operator View

x = Φ Φ∗ x

Sampling followed by interpolation: b

T

yn

x(t) g ∗ (−t) T

g (t) b

x (t)

b

x is best approximation of x within shift-invariant subspace generated by g if

P = Φ Φ∗ is an orthogonal projection operator

P is automatically self-adjoint: P ∗ = (Φ Φ∗ )∗ = P

Need P idempotent: P 2 = Φ Φ∗ Φ Φ∗ = P

Example Lecture: Sampling and Interpolation Operator View

x = Φ Φ∗ x

Sampling followed by interpolation: b

T

yn

x(t) g ∗ (−t) T

g (t) b

x (t)

b

x is best approximation of x within shift-invariant subspace generated by g if

P = Φ Φ∗ is an orthogonal projection operator

P is automatically self-adjoint: P ∗ = (Φ Φ∗ )∗ = P

Need P idempotent: P 2 = Φ Φ∗ Φ Φ∗ = P

Example Lecture: Sampling and Interpolation Operator View

x = Φ Φ∗ x

Sampling followed by interpolation: b

T

yn

x(t) g ∗ (−t) T

g (t) b

x (t)

b

x is best approximation of x within shift-invariant subspace generated by g if

P = Φ Φ∗ is an orthogonal projection operator

P is automatically self-adjoint: P ∗ = (Φ Φ∗ )∗ = P

Need P idempotent: P 2 = Φ Φ∗ Φ Φ∗ = P

Example Lecture: Sampling and Interpolation Operator View

x = Φ Φ∗ x

Sampling followed by interpolation: b

T

yn

x(t) g ∗ (−t) T

g (t) b

x (t)

x is best approximation of x within shift-invariant subspace generated by g if

P = Φ Φ∗ is an orthogonal projection operator

P is automatically self-adjoint: P ∗ = (Φ Φ∗ )∗ = P

Need P idempotent: P 2 = Φ Φ∗ Φ Φ∗ = P

Example Lecture: Sampling and Interpolation Operator View

Interpolation followed by sampling: yb = Φ∗ Φ y

b

x (t) T

yn

T

g (t) g ∗ (−t) ybn

Example Lecture: Sampling and Interpolation Operator View

Interpolation followed by sampling: yb = Φ∗ Φ y

b

x (t) T

yn

T

g (t) g ∗ (−t) ybn

Example Lecture: Sampling and Interpolation Operator View

Interpolation followed by sampling: yb = Φ∗ Φ y

b

x (t) T

yn

T

g (t) g ∗ (−t) ybn

Example Lecture: Sampling and Interpolation Operator View

Theorem

Let g be orthogonal to its integer shifts: hg (t − n), g (t)it = δn . The system

T

yn

x(t) g ∗ (−t) T

g (t) b

x (t)

yields b

x = P x where P is the orthogonal projection operator onto the

shift-invariant subspace S generated by g .

Corollaries:

If x ∈ S, then x is recovered exactly from samples y

If x 6∈ S, then b

x is the best approximation of x in S

Example Lecture: Sampling and Interpolation Operator View

Theorem

Let g be orthogonal to its integer shifts: hg (t − n), g (t)it = δn . The system

T

yn

x(t) g ∗ (−t) T

g (t) b

x (t)

yields b

x = P x where P is the orthogonal projection operator onto the

shift-invariant subspace S generated by g .

Corollaries:

If x ∈ S, then x is recovered exactly from samples y

If x 6∈ S, then b

x is the best approximation of x in S

Example Lecture: Sampling and Interpolation Operator View

T

yn

x(t) g ∗ (−t) T

g (t) b

x (t)

◮ Immediately, orthogonal projection property holds

g ∗ (−t) = g (t)

Example Lecture: Sampling and Interpolation Operator View

T

yn

x(t) g ∗ (−t) T

g (t) b

x (t)

◮ Immediately, orthogonal projection property holds

g ∗ (−t) = g (t)

Example Lecture: Sampling and Interpolation Operator View

For a fixed positive N and sampling filter g , let Φ∗ : ℓ2 (Z) → ℓ2 (Z) be given by

Φ∗

∗ y

x g−n N

For a fixed positive N and interpolation filter g , let Φ : ℓ2 (Z) → ℓ2 (Z) be given by

X

(Φ y )n = yk gn−kN , n∈Z

k∈Z

Φ

y N gn b

x

Example Lecture: Sampling and Interpolation Operator View

A subspace W ⊂ ℓ2 (Z) is a shift-invariant subspace with respect to shift L ∈ Z+

when xn ∈ W implies xn−kL ∈ W for every integer k. In addition, w ∈ ℓ2 (Z) is

called a generator of W when W = span({wn−kL }k∈Z ).

Theorem

Let g be orthogonal to its shifts by multiples of N: hgn−kN , gn in = δk . The system

∗

yn

xn g−n N N gn b

xn

yields b

x = P x where P is the orthogonal projection operator onto the

shift-invariant subspace S generated by g with shift N.

Example Lecture: Sampling and Interpolation Operator View

T

yn

x(t) ge(t) T

g (t) b

x (t)

e ∗ , interpolation operator Φ

Sampling operator Φ

Example Lecture: Sampling and Interpolation Operator View

T

yn

x(t) ge(t) T

g (t) b

x (t)

e ∗ , interpolation operator Φ

Sampling operator Φ

e ∗ generally not self-adjoint, but can still be a projection operator

ΦΦ

e ∗ Φ = I for an oblique projection

Check Φ

Example Lecture: Sampling and Interpolation Operator View

T

yn

x(t) ge(t) T

g (t) b

x (t)

e ∗ , interpolation operator Φ

Sampling operator Φ

e ∗ generally not self-adjoint, but can still be a projection operator

ΦΦ

e ∗ Φ = I for an oblique projection

Check Φ

e

S

·

S

•x

•

b

x

Example Lecture: Sampling and Interpolation Basis Expansion View

Synthesis operator associated with basis {ϕk }k∈K for H

X

◮ Φ : ℓ2 (K) → H Φα = αk ϕk

k∈K

Analysis operator associated with basis {ϕk }k∈K for H

◮ Φ∗ : H → ℓ2 (K) (Φ∗ x)k = hx, ϕk i, k ∈ K

P

When {ϕk }k∈K is an orthonormal set, k∈K hx, ϕk iϕk is an orthogonal

projection

Special case of a shift-variant space and a basis obtained from shifts of

function:

ϕk = g (t − k)

Example Lecture: Sampling and Interpolation Basis Expansion View

Synthesis operator associated with basis {ϕk }k∈K for H

X

◮ Φ : ℓ2 (K) → H Φα = αk ϕk

k∈K

Analysis operator associated with basis {ϕk }k∈K for H

◮ Φ∗ : H → ℓ2 (K) (Φ∗ x)k = hx, ϕk i, k ∈ K

P

When {ϕk }k∈K is an orthonormal set, k∈K hx, ϕk iϕk is an orthogonal

projection

Special case of a shift-variant space and a basis obtained from shifts of

function:

ϕk = g (t − k)

Example Lecture: Sampling and Interpolation Extensions

Variations

Multichannel sampling

2T y0,n

x(t) ge0 (t)

2T y1,n

ge1 (t)

Periodic nonuniform sampling (time-interleaved ADC)

Many inverse problems have linear forward models, perhaps not shift-invariant

Similar subspace geometry holds

Example Lecture: Sampling and Interpolation Extensions

Variations

Multichannel sampling

2T y0,n

x(t) ge0 (t)

2T y1,n

ge1 (t)

Periodic nonuniform sampling (time-interleaved ADC)

Many inverse problems have linear forward models, perhaps not shift-invariant

Similar subspace geometry holds

Example Lecture: Sampling and Interpolation Extensions

Beyond subspaces

A subset W ⊂ L2 (R) has rate of innovation ρ with respect to generator g if any

x ∈ W can be written as X

x = αk g (t − tk )

k∈Z

where

#{tk in [−T /2, T /2]} ρ

lim sup =

T →∞ T 2

W is not a subspace

For some g , exact recovery from uniform samples at rate ρ is possible

Many classes of techniques

Example Lecture: Sampling and Interpolation Summary

Summary

Adjoints

◮ Time reversal between sampling and interpolation

Subspaces

◮ Shift-invariant, range of interpolator Φ

◮ Null space of sampler Φ∗

Projection

◮ Φ Φ∗ always self adjoint

◮ Φ∗ Φ = I implies Φ Φ∗ is a projection operator

◮ Together, orthogonal projection operator, best approximation

Basis expansions

◮ Sampling produces analysis coefficients for basis expansion

◮ Interpolation synthesizes from expansion coefficients

Teaching Materials Textbooks

Textbooks

Two books:

M. Vetterli, J. Kovačević, and V. K. Goyal,

Foundations of Signal Processing

Fourier and Wavelet Signal Processing

a single volume and with some variations in titles) since

2010 at

http://www.fourierandwavelets.org

hyperlinks, no exercises with solutions or exercises

Teaching Materials Textbooks

Textbooks

Foundations of Signal Processing

1 On Rainbows and Spectra

2 From Euclid to Hilbert

3 Sequences and Discrete-Time Systems

4 Functions and Continuous-Time Systems

5 Sampling and Interpolation

6 Approximation and Compression

7 Localization and Uncertainty

Features:

About 640 pages illustrated with more than 200 figures

More than 200 exercises (more than 30 with solutions within the text)

Solutions manual for instructors

Summary tables, guides to further reading, historical notes

Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 90 / 96

Teaching Materials Textbooks

Textbooks

Fourier and Wavelet Signal Processing

1 Filter Banks: Building Blocks of Time-Frequency Expansions

2 Local Fourier Bases on Sequences

3 Wavelet Bases on Sequences

4 Local Fourier and Wavelet Frames on Sequences

5 Local Fourier Transforms, Frames and Bases on Functions

6 Wavelet Bases, Frames and Transforms on Functions

7 Approximation, Estimation, and Compression

Teaching Materials Textbooks

Prerequisites

Mathematical maturity

◮ Mechanical use of calculus not enough

◮ Sophistication to read and write precise mathematical statements needed

(or could be learned here)

Linear algebra

◮ Basic facility with matrix algebra very useful

◮ Abstract view built carefully within the book

Probability

◮ Basic background (e.g., first half of Introduction to Probability by Bertsekas

and Tsitsiklis) needed (else all stochastic material could be skipped)

Signals and systems

◮ Basic background (e.g., Signals and Systems by Oppenheim and Willsky)

helpful but not necessary

Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 92 / 96

Teaching Materials Supplementary materials

Solutions manual

Convolution of Derivative and Primitive

Let h and x be differentiable functions, and let

Z t Z t

h(1) (t) = h(τ ) dτ and x (1) (t) = x(τ ) dτ

−∞ −∞

integration by parts.

Teaching Materials Supplementary materials

Solutions manual

From the definition of convolution, (4.35),

Z ∞

(h ∗ x)(t) = x(τ )h(t − τ ) dτ.

−∞

involving h(1) and x ′ . With the associations

we obtain

u ′ (τ ) = x ′ (τ ) and v (τ ) = −h(1) (t − τ ).

Z ∞

t=∞

(h ∗ x)(t) = −x(τ ) h(1) (t − τ )t=−∞ + h(1) (t − τ ) x ′ (τ ) dτ. (1)

−∞

τ →±∞

Teaching Materials Supplementary materials

Mathematica figures

|X10 (e jω )| |X100 (e jω )|

2 2

Π Π 3Π

Π

ω Π Π 3Π

Π

ω

4 2 4 4 2 4

|X1000 (e jω )| |X1000 (e jω )|

2

Π Π 3Π

Π

ω 9Π Π ω

4 2 4 20 2

illustrating the Gibbs phenomenon. Shown are

|XN (e jω )| from (3.84) with different N. Observe how

oscillations narrow from (a) to (c), but their amplitude

remains√ constant (the topmost grid line in every plot),

1.089 2.

Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 95 / 96

Wrap up

- Solution Manual for Discrete Fourier Analysis and Wavelets 1st Ed - Allen Broughton, Kurt BryanUploaded byMasoomeh Akbarzadeh
- M1 Formula SheetUploaded byabdiraheem
- mca detailsUploaded byapi-271838818
- Quantum MechanicsUploaded byriemannadi
- Dpp Maths & Computer-newUploaded byMuhammad Atif
- Lecture Notes on Linear System TheoryUploaded bykkd3
- Machine LearningUploaded byManab Chetia
- Me471s03 Laplace ExampUploaded byHasen Bebba
- IIT Guhati Note Continuum MechanicsUploaded byRahul Karna
- Form-finding of Tensegrity Structures Subjected to Geometrical ConstraintsUploaded byAndrésGonzález
- M1HUploaded byapi-3824743
- Notes 5Uploaded bynvhanh
- Chen 2018Uploaded bySurenaHamid
- I year syllabus 2013-14.pdfUploaded bykavyankhs
- Paper 7Uploaded byRakesh Lin
- ClusteringUploaded byMAHESH
- PS4_SolnUploaded bybharat
- mathgen-1898293085Uploaded bya1134454
- decoupling in multivariable systems.pdfUploaded byBhautik Daxini
- A Geometric Review of Linear AlgebraUploaded byfengw89
- Curriculum for Bachelor of EngineeringUploaded byNaozu Tachi
- Voice LeadingUploaded byvicher2020
- Iiserb Mm1 NotesUploaded bySayan Kunti
- ppt_002Uploaded byLam Wong
- Please Rea4Uploaded bybeckybirdie

- nlipovac_prostor_i_mjestoUploaded byAleksandra Prastalo
- Fuzzy ConceptsUploaded byRodolfo Quijada
- Ponnusamy Sugawa, Vuorinen. (Eds.) Quasiconformal Mappings and Their Applications. IWQCMA05 (Draft, Madras, 2006)(ISBN 8173198071)(358s)_MCcUploaded bydmkrp
- Henri PoincaréUploaded byscarllatti
- Musical SymmetryUploaded bysparrowjakazz
- Graphs and Matrices in Geometry Fiedler.pdfUploaded byFatima Petilla
- Albert Einstein on Space-time -- Britannica Online EncyclopediaUploaded bybskienhvqy
- Hyperbolic Geometry-JAMES W. CANNON, WILLIAM J. FLOYD, RICHARD KENYON, AND WALTER R. PARRYUploaded byluciferlight
- Treatise of Plane Geometry Through Geometric Algebra - R G CalvetUploaded byw2006y
- Geometric Group Theory, Volume 1Uploaded byfelipeciji
- Differential Geometry III Lecture Notes - A. KovalevUploaded bytomgillespie
- Grass Mann Algebra BookUploaded bywoodbrent7
- tmp_2477-FIE-book-high-res1260069406Uploaded bysafariore
- Geometrical probUploaded byRogelio Lazo Arjona
- Vol_59_Num_2_Spring_2000Uploaded byOvidiu Sso Sso
- Spatial Questions_ Cultural Topologies and Social Spatialisation - Rob Shields (1)Uploaded byPedro Porbén
- Generic Framework for Image GeometryUploaded bydavidrojasv
- Topic 9 Three Dimensional Shapes (3D Solids).pdfUploaded bynickzen
- Relativity Without TearsUploaded byahsbon
- 3. CAD Dr.C.K.shene Geometric ConceptsUploaded byDineshNewalkar
- Course Outline 201 NYC 05 ScienceUploaded byprideaux74
- 2nd Year Course OutlineUploaded byMiliyon Tilahun
- Visibility Graph AlgorithmUploaded byKha Trấn Ác
- geodesiaUploaded byDarwins Cortes
- ch7Uploaded byyashwanthr3
- Tensors Poor ManUploaded byAlan Turing
- Henri Poincare and Relativity T - A. a. LogunovUploaded byDavid Granero
- commsys-2-2012-6Uploaded byJoseph John
- BookReview - Classical Mechaniscs - Rao 818Uploaded bycarmo-neto
- On the Role of Differential Geometry in Signal ProcessingUploaded byshaktishenoy

## Much more than documents.

Discover everything Scribd has to offer, including books and audiobooks from major publishers.

Cancel anytime.