You are on page 1of 184

# Teaching Signal Processing with Geometry

## Martin Vetterli and Vivek Goyal

Based upon textbook with Jelena Kovačević
Includes slides by Andrea Ridolfi and Amina Chebira

## August 27, 2012

Outline
1 Introduction
Why a tutorial on teaching?
Overview
2 Basic Principles in Teaching SP with Geometry
Unified view
Hilbert space tools—Part I: Basics through projections
Basic results I: Best approximation
Hilbert space tools—Part II: Bases through discrete-time systems
3 Example Lecture: Sampling and Interpolation
Motivation
Classical View and Historical Notes
Operator View
Basis Expansion View
Extensions
Summary
4 Teaching Materials
Textbooks
Supplementary materials
5 Wrap up
Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 2 / 96
Introduction Why a tutorial on teaching?

Teaching should not remain static
Applications change (and we should want them to)
◮ Signal processing thinking should be applied broadly
◮ Global Fourier techniques relatively less important than in the past

## Computing platforms change (and we should want them to)

◮ Classical DSP architectures relative less important than in the past
◮ Likely to use high-level programming languages

## Students change (and we should want them to)

◮ Different base of knowledge
◮ Biology, economics, social sciences, . . .

## Eternal challenge of the educator

◮ Knowledge grows, time in school does not
◮ Must be willing to cull details to convey big picture
◮ Should teach what is most reusable and generalizable

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 3 / 96

Introduction Why a tutorial on teaching?

Teaching should not remain static
Applications change (and we should want them to)
◮ Signal processing thinking should be applied broadly
◮ Global Fourier techniques relatively less important than in the past

## Computing platforms change (and we should want them to)

◮ Classical DSP architectures relative less important than in the past
◮ Likely to use high-level programming languages

## Students change (and we should want them to)

◮ Different base of knowledge
◮ Biology, economics, social sciences, . . .

## Eternal challenge of the educator

◮ Knowledge grows, time in school does not
◮ Must be willing to cull details to convey big picture
◮ Should teach what is most reusable and generalizable

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 3 / 96

Introduction Why a tutorial on teaching?

Teaching should not remain static
Applications change (and we should want them to)
◮ Signal processing thinking should be applied broadly
◮ Global Fourier techniques relatively less important than in the past

## Computing platforms change (and we should want them to)

◮ Classical DSP architectures relative less important than in the past
◮ Likely to use high-level programming languages

## Students change (and we should want them to)

◮ Different base of knowledge
◮ Biology, economics, social sciences, . . .

## Eternal challenge of the educator

◮ Knowledge grows, time in school does not
◮ Must be willing to cull details to convey big picture
◮ Should teach what is most reusable and generalizable

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 3 / 96

Introduction Why a tutorial on teaching?

Teaching should not remain static
Applications change (and we should want them to)
◮ Signal processing thinking should be applied broadly
◮ Global Fourier techniques relatively less important than in the past

## Computing platforms change (and we should want them to)

◮ Classical DSP architectures relative less important than in the past
◮ Likely to use high-level programming languages

## Students change (and we should want them to)

◮ Different base of knowledge
◮ Biology, economics, social sciences, . . .

## Eternal challenge of the educator

◮ Knowledge grows, time in school does not
◮ Must be willing to cull details to convey big picture
◮ Should teach what is most reusable and generalizable

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 3 / 96

Introduction Overview

Overview
Goals of the tutorial:
See that geometric notions unify (simplify!) signal processing
Learn/review basics of Hilbert space view
See Hilbert space view in action
Learn about textbooks Foundations of Signal Processing and
Fourier and Wavelet Signal Processing

## Structure of the tutorial:

Developing unified view of signal processing
Hilbert space tools—Part I: Basics through projections
◮ A few key results (best approximation and the projection theorem)
Hilbert space tools—Part II: Bases through discrete-time systems
Textbooks and related materials

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 4 / 96

Introduction Overview

Overview
Goals of the tutorial:
See that geometric notions unify (simplify!) signal processing
Learn/review basics of Hilbert space view
See Hilbert space view in action
Learn about textbooks Foundations of Signal Processing and
Fourier and Wavelet Signal Processing

## Structure of the tutorial:

Developing unified view of signal processing
Hilbert space tools—Part I: Basics through projections
◮ A few key results (best approximation and the projection theorem)
Hilbert space tools—Part II: Bases through discrete-time systems
Textbooks and related materials

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 4 / 96

Introduction Overview

Overview
Goals of the tutorial:
See that geometric notions unify (simplify!) signal processing
Learn/review basics of Hilbert space view
See Hilbert space view in action
Learn about textbooks Foundations of Signal Processing and
Fourier and Wavelet Signal Processing

## Structure of the tutorial:

Developing unified view of signal processing
Hilbert space tools—Part I: Basics through projections
◮ A few key results (best approximation and the projection theorem)
Hilbert space tools—Part II: Bases through discrete-time systems
Textbooks and related materials

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 4 / 96

Introduction Overview

Overview
Goals of the tutorial:
See that geometric notions unify (simplify!) signal processing
Learn/review basics of Hilbert space view
See Hilbert space view in action
Learn about textbooks Foundations of Signal Processing and
Fourier and Wavelet Signal Processing

## Structure of the tutorial:

Developing unified view of signal processing
Hilbert space tools—Part I: Basics through projections
◮ A few key results (best approximation and the projection theorem)
Hilbert space tools—Part II: Bases through discrete-time systems
Textbooks and related materials

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 4 / 96

Basic Principles in Teaching SP with Geometry Unified view

Unifying principles
Signal processing has various dichotomies
continuous time vs. discrete time
infinite intervals vs. finite intervals
periodic vs. aperiodic
deterministic vs. stochastic

## Each can placed in a common framework featuring geometry

Example payoffs:
Unified understanding of best approximation (projection theorem)
Unified understanding of Fourier domains
Unified understanding of signal expansions (including sampling)

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 5 / 96

Basic Principles in Teaching SP with Geometry Unified view

Unifying principles
Signal processing has various dichotomies
continuous time vs. discrete time
infinite intervals vs. finite intervals
periodic vs. aperiodic
deterministic vs. stochastic

## Each can placed in a common framework featuring geometry

Example payoffs:
Unified understanding of best approximation (projection theorem)
Unified understanding of Fourier domains
Unified understanding of signal expansions (including sampling)

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 5 / 96

Basic Principles in Teaching SP with Geometry Unified view

Unifying principles
Signal processing has various dichotomies
continuous time vs. discrete time
infinite intervals vs. finite intervals
periodic vs. aperiodic
deterministic vs. stochastic

## Each can placed in a common framework featuring geometry

Example payoffs:
Unified understanding of best approximation (projection theorem)
Unified understanding of Fourier domains
Unified understanding of signal expansions (including sampling)

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 5 / 96

Basic Principles in Teaching SP with Geometry Unified view

Unifying principles
Signal processing has various dichotomies
continuous time vs. discrete time
infinite intervals vs. finite intervals
periodic vs. aperiodic
deterministic vs. stochastic

## Each can placed in a common framework featuring geometry

Example payoffs:
Unified understanding of best approximation (projection theorem)
Unified understanding of Fourier domains
Unified understanding of signal expansions (including sampling)

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 5 / 96

Basic Principles in Teaching SP with Geometry Unified view

## Unifying framework: Hilbert spaces

Examples of Hilbert spaces:
finite-dimensional vectors (basic linear algebra)
sequences on {. . . , −1, 0, 1, . . .} (discrete-time signals)
sequences on {0, 1, . . . , N − 1} (N-periodic discrete-time signals)
functions on (−∞, ∞) (continuous-time signals)
functions on [0, T ] (T -periodic continuous-time signals)
scalar random variables

## More abstraction. More mathematics. More difficult?

With framework in place, can go farther, faster
Leverage “real world” geometric intuition

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 6 / 96

Basic Principles in Teaching SP with Geometry Unified view

## Unifying framework: Hilbert spaces

Examples of Hilbert spaces:
finite-dimensional vectors (basic linear algebra)
sequences on {. . . , −1, 0, 1, . . .} (discrete-time signals)
sequences on {0, 1, . . . , N − 1} (N-periodic discrete-time signals)
functions on (−∞, ∞) (continuous-time signals)
functions on [0, T ] (T -periodic continuous-time signals)
scalar random variables

## More abstraction. More mathematics. More difficult?

With framework in place, can go farther, faster
Leverage “real world” geometric intuition

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 6 / 96

Basic Principles in Teaching SP with Geometry Unified view

Mathematical rigor
Everything should be made as simple as possible, but no simpler.
– Common paraphrasing of Albert Einstein
Make everything as simple as possible without being wrong.
– Our variant for teaching

## Correct intuitions are separate from functional analysis details

Teach the difference among
◮ rigorously true, with elementary justification
◮ rigorously true, justification not elementary (e.g., Poisson sum formula)
◮ convenient and related to rigorous statements (e.g., uses of Dirac delta)

## . . . if whether an airplane would fly or not depended on whether some

function . . . was Lebesgue but not Riemann integrable, then I would not
fly in it.
– Richard W. Hamming

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 7 / 96

Basic Principles in Teaching SP with Geometry Unified view

Mathematical rigor
Everything should be made as simple as possible, but no simpler.
– Common paraphrasing of Albert Einstein
Make everything as simple as possible without being wrong.
– Our variant for teaching

## Correct intuitions are separate from functional analysis details

Teach the difference among
◮ rigorously true, with elementary justification
◮ rigorously true, justification not elementary (e.g., Poisson sum formula)
◮ convenient and related to rigorous statements (e.g., uses of Dirac delta)

## . . . if whether an airplane would fly or not depended on whether some

function . . . was Lebesgue but not Riemann integrable, then I would not
fly in it.
– Richard W. Hamming

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 7 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Vector spaces
A vector space generalizes easily beyond the R2 Euclidean plane

Axioms
A vector space over a field of scalars C (or R) is a set of vectors V together
with operations
◮ vector addition: V × V → V
◮ scalar multiplication: C × V → V
that satisfy the following axioms:
1. x +y =y +x
2. (x + y ) + z = x + (y + z)
3. ∃ 0 ∈ V s.t. x + 0 = x for all x ∈ V
4. α(x + y ) = αx + αy
5. (α + β)x = αx + βx
6. (αβ)x = α(βx)
7. 0x = 0 and 1x = x

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 8 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Vector spaces

Examples
CN : complex (column) vectors of length N

## CZ : sequences – discrete-time signals

(write as infinite column vector)

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 9 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Vector spaces

Key notions
Subspace
◮ S ⊆ V is a subspace when it is closed under vector addition and scalar
multiplication:
⋆ For all x, y ∈ S, x + y ∈ S
⋆ For all x ∈ S and α ∈ C, αx ∈ S

Span
◮ S: set of vectors (could be infinite)
◮ span(S) = set of all finite linear combinations of vectors in S:
( N−1 )
X
S = αk ϕk | αk ∈ C, ϕk ∈ S and N ∈ N
k=0

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 10 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Vector spaces

Key notions
Subspace
◮ S ⊆ V is a subspace when it is closed under vector addition and scalar
multiplication:
⋆ For all x, y ∈ S, x + y ∈ S
⋆ For all x ∈ S and α ∈ C, αx ∈ S

Span
◮ S: set of vectors (could be infinite)
◮ span(S) = set of all finite linear combinations of vectors in S:
( N−1 )
X
S = αk ϕk | αk ∈ C, ϕk ∈ S and N ∈ N
k=0

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 10 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Vector spaces

Key notions
Linear independence
N−1
◮ S = {ϕk }k=0 is linearly independent when:
N−1
X
αk ϕk = 0 only when αk = 0 for all k
k=0

## ◮ If S is infinite, we need every finite subset to be linearly independent

Dimension
◮ dim(V ) = N if V contains a linearly independent set with N vectors and every
set with N + 1 or more vectors is linearly dependent
◮ V is infinite dimensional if no such finite N exists

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 11 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Vector spaces

Key notions
Linear independence
N−1
◮ S = {ϕk }k=0 is linearly independent when:
N−1
X
αk ϕk = 0 only when αk = 0 for all k
k=0

## ◮ If S is infinite, we need every finite subset to be linearly independent

Dimension
◮ dim(V ) = N if V contains a linearly independent set with N vectors and every
set with N + 1 or more vectors is linearly dependent
◮ V is infinite dimensional if no such finite N exists

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 11 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Inner products
Inner products generalize angles (especially right angles) and orientation

## Definition (Inner product)

An inner product on vector space V is a function h·, ·i : V × V → C
satisfying
1 Distributivity: hx + y , zi = hx, zi + hy , zi
2 Linearity in the first argument: hαx, y i = αhx, y i
3 Hermitian symmetry: hx, y i∗ = hy , xi
4 Positive definiteness: hx, xi ≥ 0 and hx, xi = 0 if and only if x = 0

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 12 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Inner products

Examples
N−1
X
On CN : hx, y i = xn yn∗ = y ∗ x
n=0

X
On CZ : hx, y i = xn yn∗ = y ∗ x
n∈Z

Z ∞
On CR : hx, y i = x(t)y ∗ (t) dt
−∞

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 13 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

## Geometry in inner product spaces

Drawn in R2 and true in general:
hx, y i = x1 y1 + x2 y2
= kxk ky k cos α
= product of 2-norms times the cos
of the angle between the vectors

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 14 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

## Geometry in inner product spaces

Drawn in R2 and true in general:
hx, y i = x1 y1 + x2 y2
= kxk ky k cos α
= product of 2-norms times the cos
of the angle between the vectors

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 14 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Orthogonality
Let S = {ϕi }i ∈I be a set of vectors

Definition (Orthogonality)
x and y are orthogonal when hx, y i = 0 written x ⊥ y

## S0 and S1 are orthogonal when every s0 ∈ S0 is orthogonal to S1 , written

S0 ⊥ S1
Right angles (perpendicularity) extends beyond Euclidean geometry

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 15 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Norm
Norms generalize length in ordinary Euclidean space

Definition (Norm)
A norm on V is a function k·k : V → R satisfying
1 Positive definiteness: kxk ≥ 0 and kxk = 0 if and only if x = 0
2 Positive scalability: kαxk = |α| kxk
3 Triangle inequality: kx + y k ≤ kxk + ky k with equality if and only if y = αx

p
kxk = hx, xi

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 16 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Norm
Norms generalize length in ordinary Euclidean space

Definition (Norm)
A norm on V is a function k·k : V → R satisfying
1 Positive definiteness: kxk ≥ 0 and kxk = 0 if and only if x = 0
2 Positive scalability: kαxk = |α| kxk
3 Triangle inequality: kx + y k ≤ kxk + ky k with equality if and only if y = αx

p
kxk = hx, xi

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 16 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

## Norms induced by inner products

p
Any inner product induces a norm: kxk = hx, xi

Examples
N−1
!1/2
p X
N 2
On C : kxk = hx, xi = |xn |
n=0

!1/2
p X
On CZ : kxk = hx, xi = |xn |2
n∈Z

Z 1/2
p ∞
On CR : kxk = hx, xi = |x(t)|2 dt
−∞

p  
On C-valued random variables: kxk = hx, xi = E |x|2

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 17 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

## Norms induced by inner products

Properties
Pythagorean theorem

◮ x ⊥y ⇒ kx + y k2 = kxk2 + ky k2

X 2 X
◮ {xk }k∈K orthogonal ⇒
xk
=
kxk k2
k∈K k∈K

x +y
y

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 18 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

## Norms induced by inner products

Properties
Parallelogram Law
kx + y k2 + kx − y k2 = 2(kxk2 + ky k2 )

x
x +y
y y
x −y

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 19 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

## Norms induced by inner products

Properties
Cauchy–Schwarz inequality
|hx, y i| ≤ kxk ky k

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 20 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

## Norms not necessarily induced by inner products

Examples
P 1/p
N−1
On CN : kxkp = n=0 |xn |
p
, p ∈ [1, ∞)

P p
1/p
On CZ : kxkp = n∈Z |xn | , p ∈ [1, ∞)

kxk∞ = sup|xn |
n∈Z

R 1/p

On CR : kxkp = −∞
|x(t)|p dt , p ∈ [1, ∞)

t∈R

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 21 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

p
Geometry of ℓ : Unit balls

x1

kxk∞ = 1
kxk4 = 1
kxk2 = 1
kxk1 = 1
kxk1/2 = 1

x0

Valid norm (and convex unit ball) for p ≥ 1; ordinary geometry for p = 2

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 22 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

## Normed vector spaces

A normed vector space is a set satisfying axioms of a vector space where the
norm is finite

!1/2
X
2
kxk = |xn | < ∞
n∈Z

## L2 (R): square-integrable functions (“finite-energy continuous-time signals”)

Z ∞ 1/2
kxk = |x(t)|2 dt < ∞
−∞

## x and y are the same when kx − y k = 0

No harm in considering only functions with finitely-many discontinuities

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 23 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

## Normed vector spaces

A normed vector space is a set satisfying axioms of a vector space where the
norm is finite

!1/2
X
2
kxk = |xn | < ∞
n∈Z

## L2 (R): square-integrable functions (“finite-energy continuous-time signals”)

Z ∞ 1/2
kxk = |x(t)|2 dt < ∞
−∞

## x and y are the same when kx − y k = 0

No harm in considering only functions with finitely-many discontinuities

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 23 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

## Hilbert spaces: Convergence

Definition
A sequence of vectors x0 , x1 , . . . in a normed vector space V is said to converge
to v ∈ V when limk→∞ kv − xk k = 0, or for any ε > 0, there exists Kε such that
kv − xk k < ε for all k > Kε .

## Choice of the norm in V is key

Example
For k ∈ Z+ , let 
1, for t ∈ [0, 1/k ];
xk (t) =
0, otherwise.
v (t) = 0 for all t. Then for p ∈ [1, ∞),
Z ∞ 1/p  1/p
1 k→∞
kv − xk kp = |v (t) − xk (t)|p dt = −→ 0,
−∞ k

## For p = ∞: kv − xk k∞ = 1 for all k

Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 24 / 96
Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

## Hilbert spaces: Convergence

Definition
A sequence of vectors x0 , x1 , . . . in a normed vector space V is said to converge
to v ∈ V when limk→∞ kv − xk k = 0, or for any ε > 0, there exists Kε such that
kv − xk k < ε for all k > Kε .

## Choice of the norm in V is key

Example
For k ∈ Z+ , let 
1, for t ∈ [0, 1/k ];
xk (t) =
0, otherwise.
v (t) = 0 for all t. Then for p ∈ [1, ∞),
Z ∞ 1/p  1/p
1 k→∞
kv − xk kp = |v (t) − xk (t)|p dt = −→ 0,
−∞ k

## For p = ∞: kv − xk k∞ = 1 for all k

Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 24 / 96
Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

## Hilbert spaces: Convergence

Definition
A sequence of vectors x0 , x1 , . . . in a normed vector space V is said to converge
to v ∈ V when limk→∞ kv − xk k = 0, or for any ε > 0, there exists Kε such that
kv − xk k < ε for all k > Kε .

## Choice of the norm in V is key

Example
For k ∈ Z+ , let 
1, for t ∈ [0, 1/k ];
xk (t) =
0, otherwise.
v (t) = 0 for all t. Then for p ∈ [1, ∞),
Z ∞ 1/p  1/p
1 k→∞
kv − xk kp = |v (t) − xk (t)|p dt = −→ 0,
−∞ k

## For p = ∞: kv − xk k∞ = 1 for all k

Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 24 / 96
Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

## Hilbert spaces: Completeness

Definitions
A sequence {xn } is a Cauchy sequence in a normed space when for any
ε > 0, there exists kε such that kxk − xm k < ε for all k, m > kε

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 25 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Hilbert spaces

Examples
Q is not a complete space
X∞
1 π2
◮ → ∈ R, ∈
/Q
n=1
n2 6

X∞
1
◮ → e ∈ R, ∈
/Q
n=0
n!
e

5 8 65
1 2 2 3 24
R is a complete space

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 26 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Hilbert spaces

Examples
Q is not a complete space
X∞
1 π2
◮ → ∈ R, ∈
/Q
n=1
n2 6

X∞
1
◮ → e ∈ R, ∈
/Q
n=0
n!
e

5 8 65
1 2 2 3 24
R is a complete space

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 26 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Hilbert spaces

Examples
All finite dimensional spaces are complete

## ℓp (Z) and Lp (R) are complete

◮ ℓ2 (Z) and L2 (R) are Hilbert spaces

C q ([a, b]), functions on [a, b] with q continuous derivatives, are not complete
except for q = 0 under L∞ norm

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 27 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Hilbert spaces

Examples
All finite dimensional spaces are complete

## ℓp (Z) and Lp (R) are complete

◮ ℓ2 (Z) and L2 (R) are Hilbert spaces

C q ([a, b]), functions on [a, b] with q continuous derivatives, are not complete
except for q = 0 under L∞ norm

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 27 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Hilbert spaces

Examples
All finite dimensional spaces are complete

## ℓp (Z) and Lp (R) are complete

◮ ℓ2 (Z) and L2 (R) are Hilbert spaces

C q ([a, b]), functions on [a, b] with q continuous derivatives, are not complete
except for q = 0 under L∞ norm

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 27 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Summary on spaces

Vector spaces
Normed vector spaces

## Inner product spaces Banach spaces

Hilbert
spaces
• ℓ1 (Z)
•Q N
•C N • ℓ∞ (Z)
• C ([a, b]) • ℓ2 (Z) • L1 (R)
• L2 (R) • L∞ (R)

• (C 1 ([a, b]), k · k∞ ) • (V , d)
• ℓ0 (Z)

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 28 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Linear operators
Linear operators generalize matrices

Definitions
A : H0 → H1 is a linear operator when for all x, y ∈ H0 , α ∈ C:
1 Additivity: A(x + y ) = Ax + Ay
2 Scalability: A(αx) = α(Ax)
Null space (subspace of H0 ): N (A) = {x ∈ H0 , Ax = 0}
Range space (subspace of H1 ): R(A) = {Ax ∈ H1 , x ∈ H0 }
Operator norm: kAk = sup kAxk
kxk=1

## A is bounded when: kAk < ∞

Inverse: Bounded B : H1 → H0 inverse of bounded A if and only if:
BAx = x, for every x ∈ H0
ABy = y , for every y ∈ H1

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 29 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Linear operators
Linear operators generalize matrices

Definitions
A : H0 → H1 is a linear operator when for all x, y ∈ H0 , α ∈ C:
1 Additivity: A(x + y ) = Ax + Ay
2 Scalability: A(αx) = α(Ax)
Null space (subspace of H0 ): N (A) = {x ∈ H0 , Ax = 0}
Range space (subspace of H1 ): R(A) = {Ax ∈ H1 , x ∈ H0 }
Operator norm: kAk = sup kAxk
kxk=1

## A is bounded when: kAk < ∞

Inverse: Bounded B : H1 → H0 inverse of bounded A if and only if:
BAx = x, for every x ∈ H0
ABy = y , for every y ∈ H1

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 29 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Linear operators
Linear operators generalize matrices

Definitions
A : H0 → H1 is a linear operator when for all x, y ∈ H0 , α ∈ C:
1 Additivity: A(x + y ) = Ax + Ay
2 Scalability: A(αx) = α(Ax)
Null space (subspace of H0 ): N (A) = {x ∈ H0 , Ax = 0}
Range space (subspace of H1 ): R(A) = {Ax ∈ H1 , x ∈ H0 }
Operator norm: kAk = sup kAxk
kxk=1

## A is bounded when: kAk < ∞

Inverse: Bounded B : H1 → H0 inverse of bounded A if and only if:
BAx = x, for every x ∈ H0
ABy = y , for every y ∈ H1

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 29 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Linear operators
Linear operators generalize matrices

Definitions
A : H0 → H1 is a linear operator when for all x, y ∈ H0 , α ∈ C:
1 Additivity: A(x + y ) = Ax + Ay
2 Scalability: A(αx) = α(Ax)
Null space (subspace of H0 ): N (A) = {x ∈ H0 , Ax = 0}
Range space (subspace of H1 ): R(A) = {Ax ∈ H1 , x ∈ H0 }
Operator norm: kAk = sup kAxk
kxk=1

## A is bounded when: kAk < ∞

Inverse: Bounded B : H1 → H0 inverse of bounded A if and only if:
BAx = x, for every x ∈ H0
ABy = y , for every y ∈ H1

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 29 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 30 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Adjoint generalizes Hermitian transposition of matrices

A∗ : H1 → H0 is the adjoint of A : H0 → H1 when
hAx, y iH1 = hx, A∗ y iH0 for every x ∈ H0 , y ∈ H1

If A = A∗ , A is self-adjoint or Hermitian

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 31 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

2 y2 = 1
8 y3

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 32 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Let A : H0 → H1 be a bounded linear operator
1 A∗ exists and is unique
2 (A∗ )∗ = A
3 AA∗ and A∗ A are self-adjoint
4 kA∗ k = kAk
5 If A is invertible, (A−1 )∗ = (A∗ )−1
6 If B : H0 → H1 is bounded, (A + B)∗ = A∗ + B ∗
7 If B : H1 → H2 is bounded, (BA)∗ = A∗ B ∗

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 33 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Z k+1/2
A : L2 (R) → ℓ2 (Z) (Ax)k = x(t)dt
k−1/2

Z !
X X n+1/2 XZ n+1/2
hAx, y iℓ2 = (Ax)n yn∗ = x(t) dt yn∗ = x(t)yn∗ dt
n∈Z n∈Z n−1/2 n∈Z n−1/2

XZ n+1/2 Z ∞
= x ∗ (t) dt =
x(t)b x ∗ (t) dt = hx, b
x(t)b x iL2 = hx, A∗ y iL2
n∈Z n−1/2 −∞

x(t) yn = (Ax)n
10 10
5 5

t n
5 10 15 5 10 15
−5 −5

−10 −10

b x(t), yn , b
x (t)
10 10
5 5

t t
5 10 15 5 10 15
−5 −5

−10 −10

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 34 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Unitary operators

## Definition (Unitary operators)

A bounded linear operator A : H0 → H1 is unitary when:
1 A is invertible
2 A preserves inner products: hAx, Ay iH1 = hx, y iH0 for every x, y ∈ H0

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 35 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Unitary operators

## Definition (Unitary operators)

A bounded linear operator A : H0 → H1 is unitary when:
1 A is invertible
2 A preserves inner products: hAx, Ay iH1 = hx, y iH0 for every x, y ∈ H0

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 35 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Projection operators

## Definition (Projection, orthogonal projection, oblique projection)

P is idempotent when P 2 = P

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 36 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections

Projection operators

## Definition (Projection, orthogonal projection, oblique projection)

P is idempotent when P 2 = P

## An oblique projection operator is not self adjoint

Theorem
If A : H0 → H1 , B : H1 → H0 bounded and A is a left inverse of B, then BA is
a projection operator. If B = A∗ then, BA = A∗ A is an orthogonal projection

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 36 / 96

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation

x•

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 37 / 96

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation

## S is a line in Euclidean space

x•

x ∈ S that is closest to x
Nearest point problem: Find b

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 37 / 96

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation

## S is a line in Euclidean space

x•

•′ b
x
x

x ∈ S that is closest to x
Nearest point problem: Find b

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 37 / 96

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation

## S is a line in Euclidean space

x•

•′ b
x
x

x ∈ S that is closest to x
Nearest point problem: Find b
Solution uniquely determined by x − b
x⊥X
◮ Circle must touch S in one point, radius ⊥ tangent

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 37 / 96

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation

## S is a line in Euclidean space

x

•′ b
x
x

x ∈ S that is closest to x
Nearest point problem: Find b
Solution uniquely determined by x − b
x⊥X
◮ Circle must touch S in one point, radius ⊥ tangent

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 37 / 96

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation

## Best approximation problem:

x ∈ S that is closest to x
Find b
x = argminkx − sk
b
s∈S

•′ b
x
x

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 38 / 96

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation

## Theorem (Projection theorem)

Let S be a closed subspace of Hilbert space H and let x ∈ H.
Existence: There exists xb ∈ S such that kx − b
x k ≤ kx − sk for all s ∈ S

Orthogonality: x − b
x ⊥ S is necessary and sufficient to determine b
x

Uniqueness: b
x is unique

Linearity: b
x = Px where P is a linear operator

## Idempotency: P(Px) = Px for all x ∈ H

All “nearest vector in a subspace” problems in Hilbert spaces are the same

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 39 / 96

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation

## Theorem (Projection theorem)

Let S be a closed subspace of Hilbert space H and let x ∈ H.
Existence: There exists xb ∈ S such that kx − b
x k ≤ kx − sk for all s ∈ S

Orthogonality: x − b
x ⊥ S is necessary and sufficient to determine b
x

Uniqueness: b
x is unique

Linearity: b
x = Px where P is a linear operator

## Idempotency: P(Px) = Px for all x ∈ H

All “nearest vector in a subspace” problems in Hilbert spaces are the same

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 39 / 96

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation

## Theorem (Projection theorem)

Let S be a closed subspace of Hilbert space H and let x ∈ H.
Existence: There exists xb ∈ S such that kx − b
x k ≤ kx − sk for all s ∈ S

Orthogonality: x − b
x ⊥ S is necessary and sufficient to determine b
x

Uniqueness: b
x is unique

Linearity: b
x = Px where P is a linear operator

## Idempotency: P(Px) = Px for all x ∈ H

All “nearest vector in a subspace” problems in Hilbert spaces are the same

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 39 / 96

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 40 / 96

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 40 / 96

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation

## Solution: Use orthogonality

Z 1  2 1
0 = hx(t) − b
x (t), 1it = cos( 32 πt) − (a0 + a1 t) 1 dt = − − a0 − a1
0 3π 2
Z 1  4 + 6π 1 1
0 = hx(t) − b
x (t), tit = cos( 32 πt) − (a0 + a1 t) t dt = − − a0 − a1
0 9π 2 2 3

1 1
0.8 0.8
0.6 x(t) 0.6
0.4 0.4
0.2 0.2
0
−0.2
x (t)
b 0
−0.2
−0.4 −0.4
−0.6 −0.6
−0.8 −0.8
−1 −1

0 0.2 0.4
t 0.6 0.8 1 0 0.2 0.4
t 0.6 0.8 1

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 40 / 96

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation

## Consider: Real-valued random variable x

 
Find the constant c that minimizes E (x − c)2
Note:
◮ Expected square is a Hilbert space norm
◮ Constants are a closed subspace in vector space of random variables

## Solution: Use orthogonality

◮ c determined uniquely by E[ (x − c) αc ] = 0 for all α ∈ R
◮ c = E[ x ]

Alternative:
◮ Expand into quadratic function of c and minimize with calculus
◮ Not too difficult, but lacks insight

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 41 / 96

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation

## Consider: Real-valued random variable x

 
Find the constant c that minimizes E (x − c)2
Note:
◮ Expected square is a Hilbert space norm
◮ Constants are a closed subspace in vector space of random variables

## Solution: Use orthogonality

◮ c determined uniquely by E[ (x − c) αc ] = 0 for all α ∈ R
◮ c = E[ x ]

Alternative:
◮ Expand into quadratic function of c and minimize with calculus
◮ Not too difficult, but lacks insight

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 41 / 96

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation

## Consider: Real-valued random variable x

 
Find the constant c that minimizes E (x − c)2
Note:
◮ Expected square is a Hilbert space norm
◮ Constants are a closed subspace in vector space of random variables

## Solution: Use orthogonality

◮ c determined uniquely by E[ (x − c) αc ] = 0 for all α ∈ R
◮ c = E[ x ]

Alternative:
◮ Expand into quadratic function of c and minimize with calculus
◮ Not too difficult, but lacks insight

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 41 / 96

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation

## Consider: Real-valued random variable x

 
Find the constant c that minimizes E (x − c)2
Note:
◮ Expected square is a Hilbert space norm
◮ Constants are a closed subspace in vector space of random variables

## Solution: Use orthogonality

◮ c determined uniquely by E[ (x − c) αc ] = 0 for all α ∈ R
◮ c = E[ x ]

Alternative:
◮ Expand into quadratic function of c and minimize with calculus
◮ Not too difficult, but lacks insight

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 41 / 96

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation

## Consider: Real-valued random variable x

 
Find the constant c that minimizes E (x − c)2
Note:
◮ Expected square is a Hilbert space norm
◮ Constants are a closed subspace in vector space of random variables

## Solution: Use orthogonality

◮ c determined uniquely by E[ (x − c) αc ] = 0 for all α ∈ R
◮ c = E[ x ]

Alternative:
◮ Expand into quadratic function of c and minimize with calculus
◮ Not too difficult, but lacks insight

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 41 / 96

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation

## Example 3: Wiener filter

Consider: Jointly wide-sense stationary
y b
x
discrete-time stochastic processes x and y h
Find the linear −
 shift-invariant
 filter h that x
+ e
minimizes E |xn − b xn |2 where b x =h∗y
Note:
◮ Expected squared absolute value is a Hilbert space norm
◮ LSI filtering puts b
xn in a closed subspace
Solution: Use orthogonality (extended for processes)
◮ h determined uniquely by relation between cross- and autocorrelations:

cx,bx,k = abx,k , k ∈Z

Cx,y (e jω )
◮ DTFT-domain version: H(e jω ) = , ω∈R
Ay (e jω )
Alternative:
◮ Expand into quadratic function of h and minimize with calculus
◮ Messy (!)—especially in complex case—and lacks insight

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 42 / 96

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation

## Example 3: Wiener filter

Consider: Jointly wide-sense stationary
y b
x
discrete-time stochastic processes x and y h
Find the linear −
 shift-invariant
 filter h that x
+ e
minimizes E |xn − b xn |2 where b x =h∗y
Note:
◮ Expected squared absolute value is a Hilbert space norm
◮ LSI filtering puts b
xn in a closed subspace
Solution: Use orthogonality (extended for processes)
◮ h determined uniquely by relation between cross- and autocorrelations:

cx,bx,k = abx,k , k ∈Z

Cx,y (e jω )
◮ DTFT-domain version: H(e jω ) = , ω∈R
Ay (e jω )
Alternative:
◮ Expand into quadratic function of h and minimize with calculus
◮ Messy (!)—especially in complex case—and lacks insight

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 42 / 96

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation

## Example 3: Wiener filter

Consider: Jointly wide-sense stationary
y b
x
discrete-time stochastic processes x and y h
Find the linear −
 shift-invariant
 filter h that x
+ e
minimizes E |xn − b xn |2 where b x =h∗y
Note:
◮ Expected squared absolute value is a Hilbert space norm
◮ LSI filtering puts b
xn in a closed subspace
Solution: Use orthogonality (extended for processes)
◮ h determined uniquely by relation between cross- and autocorrelations:

cx,bx,k = abx,k , k ∈Z

Cx,y (e jω )
◮ DTFT-domain version: H(e jω ) = , ω∈R
Ay (e jω )
Alternative:
◮ Expand into quadratic function of h and minimize with calculus
◮ Messy (!)—especially in complex case—and lacks insight

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 42 / 96

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation

## Example 3: Wiener filter

Consider: Jointly wide-sense stationary
y b
x
discrete-time stochastic processes x and y h
Find the linear −
 shift-invariant
 filter h that x
+ e
minimizes E |xn − b xn |2 where b x =h∗y
Note:
◮ Expected squared absolute value is a Hilbert space norm
◮ LSI filtering puts b
xn in a closed subspace
Solution: Use orthogonality (extended for processes)
◮ h determined uniquely by relation between cross- and autocorrelations:

cx,bx,k = abx,k , k ∈Z

Cx,y (e jω )
◮ DTFT-domain version: H(e jω ) = , ω∈R
Ay (e jω )
Alternative:
◮ Expand into quadratic function of h and minimize with calculus
◮ Messy (!)—especially in complex case—and lacks insight

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 42 / 96

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation

## Example 4: Best piecewise-constant approximation

Local averaging
Z k+ 21
2 2
A : L (R) → ℓ (Z) (Ax)k = x(t)dt
k− 12
has adjoint A∗ : ℓ2 (Z) → L2 (R) that produces staircase function

## AA∗ is identity, so A∗ A is orthogonal projection

x(t) yn = (Ax)n
10 10
5 5
5 10 15t 5 10 15n
−5 −5
−10 −10 x(t)
10
5
x (t) = (A∗ Ax)(t)
b x(t), yn , b
x (t)
t
10 10 5 10 15
5 5 −5
5 10 15t 5 10 15t −10
−5 −5
−10 −10

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 43 / 96

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation

1 2

(−π, π) (π, π)
2

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 44 / 96

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation

(−π, π) (π, π)
3

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 45 / 96

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation

1 2

(−π, π) (π, π)
2

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 46 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

Bases

Definition (Basis)
Φ = {ϕk }k∈K ⊂ V is a basis when
1 Φ is linearly independent and
2 Φ is complete in V : V = span(Φ)
X
Expansion formula: for any x ∈ V , x= αk ϕk
k∈K
{αk }k∈K : is unique
αk : expansion coefficients

Example
The standard
 basis for RN T
ek = 0 0 · · · 0 1 0 0 ··· 0 , k = 0, . . . , N − 1
N−1
X
any x ∈ RN , x= xk ek
k=0

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 47 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

Bases

Definition (Basis)
Φ = {ϕk }k∈K ⊂ V is a basis when
1 Φ is linearly independent and
2 Φ is complete in V : V = span(Φ)
X
Expansion formula: for any x ∈ V , x= αk ϕk
k∈K
{αk }k∈K : is unique
αk : expansion coefficients

Example
The standard
 basis for RN T
ek = 0 0 · · · 0 1 0 0 ··· 0 , k = 0, . . . , N − 1
N−1
X
any x ∈ RN , x= xk ek
k=0

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 47 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

Bases

Definition (Basis)
Φ = {ϕk }k∈K ⊂ V is a basis when
1 Φ is linearly independent and
2 Φ is complete in V : V = span(Φ)
X
Expansion formula: for any x ∈ V , x= αk ϕk
k∈K
{αk }k∈K : is unique
αk : expansion coefficients

Example
The standard
 basis for RN T
ek = 0 0 · · · 0 1 0 0 ··· 0 , k = 0, . . . , N − 1
N−1
X
any x ∈ RN , x= xk ek
k=0

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 47 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

Bases

Examples

α1 α1 ϕ1
x x
ϕ
e1 ϕ1
ϕ1 ϕ1
ϕ0
ϕ0 ϕ0
α0 α0 ϕ0 ϕ
e0
Orthonormal basis Biorthogonal basis . . . with dual basis

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 48 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

## Definition (Basis synthesis operator)

Synthesis operator
X
◮ Φ : ℓ2 (K) → H Φα = αk ϕk
k∈K

## ◮ Adjoint: Let α ∈ ℓ2 (Z) and y ∈ H

X  X
hΦα, y i = αk ϕk , y = αk hy , ϕk i∗
k∈K k∈K

## Definition (Basis analysis operator)

Analysis operator
◮ Φ∗ : H → ℓ2 (K) (Φ∗ x)k = hx, ϕk i, k ∈ K

Note that the analysis operator is the adjoint of the synthesis operator

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 49 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

## Definition (Basis synthesis operator)

Synthesis operator
X
◮ Φ : ℓ2 (K) → H Φα = αk ϕk
k∈K

## ◮ Adjoint: Let α ∈ ℓ2 (Z) and y ∈ H

X  X
hΦα, y i = αk ϕk , y = αk hy , ϕk i∗
k∈K k∈K

## Definition (Basis analysis operator)

Analysis operator
◮ Φ∗ : H → ℓ2 (K) (Φ∗ x)k = hx, ϕk i, k ∈ K

Note that the analysis operator is the adjoint of the synthesis operator

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 49 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

## Definition (Basis synthesis operator)

Synthesis operator
X
◮ Φ : ℓ2 (K) → H Φα = αk ϕk
k∈K

## ◮ Adjoint: Let α ∈ ℓ2 (Z) and y ∈ H

X  X
hΦα, y i = αk ϕk , y = αk hy , ϕk i∗
k∈K k∈K

## Definition (Basis analysis operator)

Analysis operator
◮ Φ∗ : H → ℓ2 (K) (Φ∗ x)k = hx, ϕk i, k ∈ K

Note that the analysis operator is the adjoint of the synthesis operator

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 49 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

Orthonormal bases

## Definition (Orthonormal basis)

Φ = {ϕk }k∈K ⊂ H is an orthonormal basis for H when
1 Φ is a basis for H and
2 Φ is an orthonormal set
hϕi , ϕk i = δi −k for all i, k ∈ K

## If span(Φ) = H and Φ is an orthogonal set,

then Φ is an orthogonal basis for H
If we also have kϕk k = 1, then Φ is an orthonormal basis

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 50 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

Orthonormal bases

## Definition (Orthonormal basis)

Φ = {ϕk }k∈K ⊂ H is an orthonormal basis for H when
1 Φ is a basis for H and
2 Φ is an orthonormal set
hϕi , ϕk i = δi −k for all i, k ∈ K

## If span(Φ) = H and Φ is an orthogonal set,

then Φ is an orthogonal basis for H
If we also have kϕk k = 1, then Φ is an orthonormal basis

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 50 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

Orthonormal bases

## Definition (Orthonormal basis)

Φ = {ϕk }k∈K ⊂ H is an orthonormal basis for H when
1 Φ is a basis for H and
2 Φ is an orthonormal set
hϕi , ϕk i = δi −k for all i, k ∈ K

## If span(Φ) = H and Φ is an orthogonal set,

then Φ is an orthogonal basis for H
If we also have kϕk k = 1, then Φ is an orthonormal basis

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 50 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

## Definition (Orthonormal basis expansions)

Φ = {ϕk }k∈K orthonormal basis for H, then for any x ∈ H:
αk = hx, ϕk i for k ∈ K, or α = Φ∗ x, and α is unique
X
Synthesis: x = hx, ϕk iϕk = Φα = ΦΦ∗ x
k∈K

Example

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 51 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

## Definition (Orthonormal basis expansions)

Φ = {ϕk }k∈K orthonormal basis for H, then for any x ∈ H:
αk = hx, ϕk i for k ∈ K, or α = Φ∗ x, and α is unique
X
Synthesis: x = hx, ϕk iϕk = Φα = ΦΦ∗ x
k∈K

Example

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 51 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

## Definition (Orthonormal basis expansions)

Φ = {ϕk }k∈K orthonormal basis for H, then for any x ∈ H:
αk = hx, ϕk i for k ∈ K, or α = Φ∗ x, and α is unique
X
Synthesis: x = hx, ϕk iϕk = Φα = ΦΦ∗ x
k∈K

Example

ϕ2 = e2 x
hx, ϕ1 i
ϕ1 = e1
ϕ0 = e0

hx, ϕ0 i
x01

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 51 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

## Theorem (Parseval’s equalities)

Φ = {ϕk }k∈K orthonormal basis for H
X
kxk2 = |hx, ϕk i|2 = kΦ∗ xk2 = kαk2
k∈K

In general:
hx, y i = hΦ∗ x, Φ∗ y i = hα, βi
where αk = hx, ϕk i, βk = hy , ϕk i

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 52 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

## Theorem (Parseval’s equalities)

Φ = {ϕk }k∈K orthonormal basis for H
X
kxk2 = |hx, ϕk i|2 = kΦ∗ xk2 = kαk2
k∈K

In general:
hx, y i = hΦ∗ x, Φ∗ y i = hα, βi
where αk = hx, ϕk i, βk = hy , ϕk i

y
β
Φ∗
x
θ Φ
θ α
H ℓ2 (K)
Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 52 / 96
Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

## Theorem (Parseval’s equalities)

Φ = {ϕk }k∈K orthonormal basis for H
X
kxk2 = |hx, ϕk i|2 = kΦ∗ xk2 = kαk2
k∈K

In general:
hx, y i = hΦ∗ x, Φ∗ y i = hα, βi
where αk = hx, ϕk i, βk = hy , ϕk i

y
β
Φ∗
x
θ Φ
θ α
H ℓ2 (K) isometry

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 52 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

## Orthogonal projection and decomposition

Theorem
Φ = {ϕk }k∈I ⊂ H, I ⊂ K
X
PI x = hx, ϕk iϕk = ΦI Φ∗I x
k∈I

## Φ induces an orthogonal decomposition

M
H = S{k} where S{k} = span(ϕk )
k∈K

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 53 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

## Orthogonal projection and decomposition

Theorem
Φ = {ϕk }k∈I ⊂ H, I ⊂ K
X
PI x = hx, ϕk iϕk = ΦI Φ∗I x
k∈I

## Φ induces an orthogonal decomposition

M
H = S{k} where S{k} = span(ϕk )
k∈K

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 53 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

## Biorthogonal pairs of bases

Definition
Biorthogonal pairs of bases
e = {ϕ
Φ = {ϕk }k∈K ⊂ H and Φ ek }k∈K ⊂ H is a biorthogonal pair of bases
when
1 e are both bases for H
Φ and Φ
2 e are biorthogonal: hϕi , ϕ
Φ and Φ ek i = δi −k for all i, k ∈ K

e are interchangeable
Roles of Φ and Φ

Example
       
1 0 1 1 0 1
ϕ0 = 1, ϕ1 = 1, ϕ2 = 1, Φ = 1 1 1
0 1 1 0 1 1
       
0 −1 1 0 1 −1
−1
e0 =  1 ,
ϕ e1 =  1 ,
ϕ e2 = −1,
ϕ Φ = −1 1 0
−1 0 1 1 −1 1
Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 54 / 96
Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

## Biorthogonal pairs of bases

Definition
Biorthogonal pairs of bases
e = {ϕ
Φ = {ϕk }k∈K ⊂ H and Φ ek }k∈K ⊂ H is a biorthogonal pair of bases
when
1 e are both bases for H
Φ and Φ
2 e are biorthogonal: hϕi , ϕ
Φ and Φ ek i = δi −k for all i, k ∈ K

e are interchangeable
Roles of Φ and Φ

Example
       
1 0 1 1 0 1
ϕ0 = 1, ϕ1 = 1, ϕ2 = 1, Φ = 1 1 1
0 1 1 0 1 1
       
0 −1 1 0 1 −1
−1
e0 =  1 ,
ϕ e1 =  1 ,
ϕ e2 = −1,
ϕ Φ = −1 1 0
−1 0 1 1 −1 1
Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 54 / 96
Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

## Biorthogonal pairs of bases

Definition
Biorthogonal pairs of bases
e = {ϕ
Φ = {ϕk }k∈K ⊂ H and Φ ek }k∈K ⊂ H is a biorthogonal pair of bases
when
1 e are both bases for H
Φ and Φ
2 e are biorthogonal: hϕi , ϕ
Φ and Φ ek i = δi −k for all i, k ∈ K

e are interchangeable
Roles of Φ and Φ

Example
       
1 0 1 1 0 1
ϕ0 = 1, ϕ1 = 1, ϕ2 = 1, Φ = 1 1 1
0 1 1 0 1 1
       
0 −1 1 0 1 −1
−1
e0 =  1 ,
ϕ e1 =  1 ,
ϕ e2 = −1,
ϕ Φ = −1 1 0
−1 0 1 1 −1 1
Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 54 / 96
Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

## Biorthogonal basis expansion

Theorem
e = {ϕ
Φ = {ϕk }k∈K , Φ ek }k∈K biorthogonal pair of bases for H
Any x ∈ H has expansion coefficients
αk = hx, ϕ e ∗x
ek i for k ∈ K, or α = Φ
X
Synthesis: x = hx, ϕ
ek iϕk = Φα = ΦΦ e ∗x
k∈K
X
Also x = hx, ϕk iϕ
ek
k∈K

e1 ϕ1
ϕ

ϕ0

ϕ
e0
Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 55 / 96
Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

## Biorthogonal basis expansion

Theorem
e = {ϕ
Φ = {ϕk }k∈K , Φ ek }k∈K biorthogonal pair of bases for H
Any x ∈ H has expansion coefficients
αk = hx, ϕ e ∗x
ek i for k ∈ K, or α = Φ
X
Synthesis: x = hx, ϕ
ek iϕk = Φα = ΦΦ e ∗x
k∈K
X
Also x = hx, ϕk iϕ
ek
k∈K

e1 ϕ1
ϕ

ϕ0

ϕ
e0
Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 55 / 96
Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

## Biorthogonal basis expansion

Theorem
e = {ϕ
Φ = {ϕk }k∈K , Φ ek }k∈K biorthogonal pair of bases for H
Any x ∈ H has expansion coefficients
αk = hx, ϕ e ∗x
ek i for k ∈ K, or α = Φ
X
Synthesis: x = hx, ϕ
ek iϕk = Φα = ΦΦ e ∗x
k∈K
X
Also x = hx, ϕk iϕ
ek
k∈K

e1 ϕ1
ϕ

ϕ0

ϕ
e0
Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 55 / 96
Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

## Frames: Overcomplete representations

X
x = hx, ϕk iϕk = ΦΦ∗ x without {ϕk }k∈J being linearly independent
k∈J

Example

R2

 
1 0 −1
Given Φ =
ϕ2 = e2 x 0 1 −1
hx, ϕ1 i

ϕ0 = e0
ϕ1 = e1 {ϕ0 , ϕ1 , ϕ2 } is a frame for R2
hx, ϕ0 i
x01

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 56 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

## How are the expansion coefficients in two orthonormal bases related?

Assume x = Φα = Ψβ

Then β = Ψ∗ x = Ψ∗ Φα

## Change of basis from Φ to Ψ that maps α to β is the operator

CΦ,Ψ : ℓ2 (K) → ℓ2 (K) s.t. CΦ,Ψ = Ψ∗ Φ

As a matrix
 
.. .. ..
 . . . 
· · · hϕ−1 , ψ−1 i hϕ0 , ψ−1 i hϕ1 , ψ−1 i · · ·
 
 
CΦ,Ψ = · · · hϕ−1 , ψ0 i hϕ0 , ψ0 i hϕ1 , ψ0 i · · ·
 
· · · hϕ−1 , ψ1 i hϕ0 , ψ1 i hϕ1 , ψ1 i · · ·
 
.. .. ..
. . .

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 57 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

## How are the expansion coefficients in two orthonormal bases related?

Assume x = Φα = Ψβ

Then β = Ψ∗ x = Ψ∗ Φα

## Change of basis from Φ to Ψ that maps α to β is the operator

CΦ,Ψ : ℓ2 (K) → ℓ2 (K) s.t. CΦ,Ψ = Ψ∗ Φ

As a matrix
 
.. .. ..
 . . . 
· · · hϕ−1 , ψ−1 i hϕ0 , ψ−1 i hϕ1 , ψ−1 i · · ·
 
 
CΦ,Ψ = · · · hϕ−1 , ψ0 i hϕ0 , ψ0 i hϕ1 , ψ0 i · · ·
 
· · · hϕ−1 , ψ1 i hϕ0 , ψ1 i hϕ1 , ψ1 i · · ·
 
.. .. ..
. . .

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 57 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

## How are the expansion coefficients in two orthonormal bases related?

Assume x = Φα = Ψβ

Then β = Ψ∗ x = Ψ∗ Φα

## Change of basis from Φ to Ψ that maps α to β is the operator

CΦ,Ψ : ℓ2 (K) → ℓ2 (K) s.t. CΦ,Ψ = Ψ∗ Φ

As a matrix
 
.. .. ..
 . . . 
· · · hϕ−1 , ψ−1 i hϕ0 , ψ−1 i hϕ1 , ψ−1 i · · ·
 
 
CΦ,Ψ = · · · hϕ−1 , ψ0 i hϕ0 , ψ0 i hϕ1 , ψ0 i · · ·
 
· · · hϕ−1 , ψ1 i hϕ0 , ψ1 i hϕ1 , ψ1 i · · ·
 
.. .. ..
. . .

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 57 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

## How are the expansion coefficients in two orthonormal bases related?

Assume x = Φα = Ψβ

Then β = Ψ∗ x = Ψ∗ Φα

## Change of basis from Φ to Ψ that maps α to β is the operator

CΦ,Ψ : ℓ2 (K) → ℓ2 (K) s.t. CΦ,Ψ = Ψ∗ Φ

As a matrix
 
.. .. ..
 . . . 
· · · hϕ−1 , ψ−1 i hϕ0 , ψ−1 i hϕ1 , ψ−1 i · · ·
 
 
CΦ,Ψ = · · · hϕ−1 , ψ0 i hϕ0 , ψ0 i hϕ1 , ψ0 i · · ·
 
· · · hϕ−1 , ψ1 i hϕ0 , ψ1 i hϕ1 , ψ1 i · · ·
 
.. .. ..
. . .

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 57 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

## Matrix representation of operator: Orthonormal basis

Let y = Ax with A : H → H
How are expansion coefficients of x and y related?
◮ {ϕk }k∈K orthonormal basis of H
◮ x = Φα, y = Φβ
Matrix representation allows computation of A directly on coefficient
sequences
Γ : ℓ2 (K) → ℓ2 (K) s.t. β = Γα
As a matrix:  
.. .. ..
 . . . 
· · · hAϕ−1 , ϕ−1 i hAϕ0 , ϕ−1 i hAϕ1 , ϕ−1 i · · ·
 
 
Γ = · · · hAϕ−1 , ϕ0 i hAϕ0 , ϕ0 i hAϕ1 , ϕ0 i · · ·
 
· · · hAϕ−1 , ϕ1 i hAϕ0 , ϕ1 i hAϕ1 , ϕ1 i · · ·
 
.. .. ..
. . .

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 58 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

## Matrix representation of operator: Orthonormal basis

Let y = Ax with A : H → H
How are expansion coefficients of x and y related?
◮ {ϕk }k∈K orthonormal basis of H
◮ x = Φα, y = Φβ
Matrix representation allows computation of A directly on coefficient
sequences
Γ : ℓ2 (K) → ℓ2 (K) s.t. β = Γα
As a matrix:  
.. .. ..
 . . . 
· · · hAϕ−1 , ϕ−1 i hAϕ0 , ϕ−1 i hAϕ1 , ϕ−1 i · · ·
 
 
Γ = · · · hAϕ−1 , ϕ0 i hAϕ0 , ϕ0 i hAϕ1 , ϕ0 i · · ·
 
· · · hAϕ−1 , ϕ1 i hAϕ0 , ϕ1 i hAϕ1 , ϕ1 i · · ·
 
.. .. ..
. . .

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 58 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

## Matrix representation of operator: Orthonormal basis

Let y = Ax with A : H → H
How are expansion coefficients of x and y related?
◮ {ϕk }k∈K orthonormal basis of H
◮ x = Φα, y = Φβ
Matrix representation allows computation of A directly on coefficient
sequences
Γ : ℓ2 (K) → ℓ2 (K) s.t. β = Γα
As a matrix:  
.. .. ..
 . . . 
· · · hAϕ−1 , ϕ−1 i hAϕ0 , ϕ−1 i hAϕ1 , ϕ−1 i · · ·
 
 
Γ = · · · hAϕ−1 , ϕ0 i hAϕ0 , ϕ0 i hAϕ1 , ϕ0 i · · ·
 
· · · hAϕ−1 , ϕ1 i hAϕ0 , ϕ1 i hAϕ1 , ϕ1 i · · ·
 
.. .. ..
. . .

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 58 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

H H
A
x • • y

Φ∗ Φ Φ∗ Φ

α • • β
Γ
2
ℓ (K) ℓ2 (K)

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 59 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

H H
A
x • • y

Φ∗ Φ Φ∗ Φ

α • • β
Γ
2
ℓ (K) ℓ2 (K)

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 59 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

## Example: Averaging Operator I

Example
Z 2(ℓ+1)
1
Let A : H0 → H1 , y (t) = Ax(t) = 2 x(τ ) dτ
2ℓ
for 2ℓ ≤ t < 2(ℓ + 1), ℓ∈Z
H0 : piecewise-constant, finite-energy functions with breakpoints at integers
H1 : piecewise-constant, finite-energy functions with breakpoints at even
integers

1, for t ∈ I ;
Given χI (t) =
0, otherwise
Let Φ = {ϕk (t)}k∈Z = {χ[k,k+1) (t)}k∈Z
Ψ = {ψi (t)}i ∈Z = { √12 χ[2i ,2(i +1))(t)}i ∈Z

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 60 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

## Example: Averaging Operator II

Example (Cont.)
Z 2
1 1 √1 √1
Aϕ0 (t) = 2 χ[0,2) (t) ⇒ hAϕ0 , ψ0 i = 2 2 dτ = 2
0
 
.. .. .. .. .. .. ..
. . . . . . .
 
· · · 1 1 0 0 0 0 · · ·
 
Then Γ = √12 
· · · 0 0 1 1 0 0 · · ·
· · · 0 0 0 0 1 1 · · ·
 
.. .. .. .. .. .. ..
. . . . . . .

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 61 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

Discrete-time systems

## A linear system A : ℓ2 (Z) → ℓ2 (Z) has a matrix representation H with

respect to the standard basis

## For a linear shift-invariant (LSI) system, the matrix H is Toeplitz:

  .. .. .. .. ..  
.. .. .. ..
. . . . . . . . .
    
y−2  . . . h0 h−1 h−2 h−3 h−4 . . .  x−2 
    
y−1  . . . h1 h0 h−1 h−2 h−3 . . .  x−1 
    
y = 
 y 0  = . . .
  h2 h1 h0 h−1 h−2 . . .  
  x0  = Hx
 y1  . . . h3 h2 h1 h0 h−1 
. . .  1 
 x
   
 y2  . . . . . .  
   h4 h3 h2 h1 h0   x2 
.. .. .. .. .. .. .. .. .
..
. . . . . . . .
| {z }
H

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 62 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

Discrete-time systems

## Matrix representation of A∗ is H ∗ [Note: using orthonormal basis]

 
.. .. .. .. .. .
 . . . . . .. 
. 
 .. h0∗ h1∗ h2∗ h3∗ h4∗ 
 
. .. 
 .. ∗
h−1 h0∗
h1∗ ∗
h2 h3 ∗ .
 
. . 
H = 

 .
. ∗
h−2 ∗
h−1 h0∗ h1∗ h2∗ . .

 
. . . . 
 . ∗
h−3 ∗ ∗
h−2 h−1 h0 h1 ∗ ∗ .
 
 .. 
 ∗
h−4 ∗
h−3 ∗
h−2 ∗
h−1 h0∗ .
 
. .. .. .. .. .
.. . . . . ..

Adjoint of filtering by hn is filtering by h−n

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 63 / 96

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems

## For an N-periodic setting, the matrix H is circulant:

    
y0 h0 hN−1 hN−2 . . . h1 x0
 y1   h1 h0 hN−1 . . . h2   x1 
    
 y2   h2 h h0 ... 
h3   x2 
y =   =  1   = Hx
 ..   .. .. .. .. ..   .. 
 .   . . . . .  . 
yN−1 hN−1 hN−2 hN−3 . . . h0 xN−1
| {z }
H

## Equally well in both cases (and continuous time as well):

◮ Eigenvectors (-sequences, -signals) lead to diagonal representation of H
◮ Fourier transform follows logically from the class of operators
◮ Convolution theorem follows logically from the definition of the Fourier
transform

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 64 / 96

Example Lecture: Sampling and Interpolation Motivation

## Sampling and Interpolation

Sampling and interpolation bridge the analog and digital worlds
Sampling: discrete-time sequence from a continuous-time function
Interpolation: continuous-time function from a discrete-time sequence

Doing all computation in discrete time is the essence of digital signal processing:

yn wn
x(t) Sampling DT processing Interpolation v (t)

## Interpolation followed by sampling occurs in digital communication:

y (t) v (t)
xn Interpolation CT channel Sampling b
xn

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 65 / 96

Example Lecture: Sampling and Interpolation Motivation

## Real-world sampling not pure mathematical idealization

◮ Don’t/can’t sample at one point
◮ Causal, non-ideal filters

## Many practical architectures different from classical structure

◮ Multichannel, time-interleaved

## Most information acquisition is intimately related to sampling

◮ Digital photography
◮ Computational imaging (magnetic resonance, space-from-time, ultrasound,
computed tomography, synthetic aperture radar, . . . )
◮ Reflection seismology, acoustic tomography, . . .

## Goals from this lecture:

1 Understand classical sampling as a special case of a Hilbert space theory
2 Gain a generalizable understanding of sampling

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 66 / 96

Example Lecture: Sampling and Interpolation Motivation

## Real-world sampling not pure mathematical idealization

◮ Don’t/can’t sample at one point
◮ Causal, non-ideal filters

## Many practical architectures different from classical structure

◮ Multichannel, time-interleaved

## Most information acquisition is intimately related to sampling

◮ Digital photography
◮ Computational imaging (magnetic resonance, space-from-time, ultrasound,
computed tomography, synthetic aperture radar, . . . )
◮ Reflection seismology, acoustic tomography, . . .

## Goals from this lecture:

1 Understand classical sampling as a special case of a Hilbert space theory
2 Gain a generalizable understanding of sampling

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 66 / 96

Example Lecture: Sampling and Interpolation Motivation

## Real-world sampling not pure mathematical idealization

◮ Don’t/can’t sample at one point
◮ Causal, non-ideal filters

## Many practical architectures different from classical structure

◮ Multichannel, time-interleaved

## Most information acquisition is intimately related to sampling

◮ Digital photography
◮ Computational imaging (magnetic resonance, space-from-time, ultrasound,
computed tomography, synthetic aperture radar, . . . )
◮ Reflection seismology, acoustic tomography, . . .

## Goals from this lecture:

1 Understand classical sampling as a special case of a Hilbert space theory
2 Gain a generalizable understanding of sampling

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 66 / 96

Example Lecture: Sampling and Interpolation Motivation

## Real-world sampling not pure mathematical idealization

◮ Don’t/can’t sample at one point
◮ Causal, non-ideal filters

## Many practical architectures different from classical structure

◮ Multichannel, time-interleaved

## Most information acquisition is intimately related to sampling

◮ Digital photography
◮ Computational imaging (magnetic resonance, space-from-time, ultrasound,
computed tomography, synthetic aperture radar, . . . )
◮ Reflection seismology, acoustic tomography, . . .

## Goals from this lecture:

1 Understand classical sampling as a special case of a Hilbert space theory
2 Gain a generalizable understanding of sampling

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 66 / 96

Example Lecture: Sampling and Interpolation Classical View and Historical Notes

Bandwidth

## Definition (Bandwidth of sequence)

A sequence x is bandlimited when there exists ω0 ∈ [0, 2π) such that the
discrete-time Fourier transform X satisfies

## The smallest such ω0 is called the bandwidth of x.

0.5 æ æ
0.5 0.5
æ
ææ æææ æææææ
æ æ æ ææ æ ææ æ æ
æ ææ æ æ æ
æ æ æ æ æ æ æ æ æ æ
æ æ æ æ æ
æ æ æ æ æææææ æ æ æ æææ
æ
æ
æ æ æ æ ææææ
æ æ æ æ æ æ æ ææ
æ æ æ æ æ æ æ æ ææ ææ
æ æ ææ æ æ æ æ æ æ æ
ææ ææ æææææ
æ æ ææ
æ æ
æ æ ææ æ
æ ææ
ææ æ
æ
-0.5 -0.5 -0.5

ω0 = π ω0 = π/2 ω0 = π/4

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 67 / 96

Example Lecture: Sampling and Interpolation Classical View and Historical Notes

Bandwidth

## Definition (Bandwidth of function)

A function x is bandlimited when there exists ω0 ∈ [0, ∞) such that the Fourier
transform X satisfies

## Definition (Bandlimited sets)

The set of sequences in ℓ2 (Z) with bandwidth at most ω0 and the set of functions
in L2 (R) with bandwidth at most ω0 are denoted BL[−ω0 /2, ω0 /2]

## If ω0 < ω1 then BL[−ω0 /2, ω0 /2] ⊂ BL[−ω1 /2, ω1 /2]

A bandlimited set is always a subspace
◮ Subspace is closed in Hilbert space ℓ2 (Z) or L2 (R)

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 68 / 96

Example Lecture: Sampling and Interpolation Classical View and Historical Notes

Bandwidth

## Definition (Bandwidth of function)

A function x is bandlimited when there exists ω0 ∈ [0, ∞) such that the Fourier
transform X satisfies

## Definition (Bandlimited sets)

The set of sequences in ℓ2 (Z) with bandwidth at most ω0 and the set of functions
in L2 (R) with bandwidth at most ω0 are denoted BL[−ω0 /2, ω0 /2]

## If ω0 < ω1 then BL[−ω0 /2, ω0 /2] ⊂ BL[−ω1 /2, ω1 /2]

A bandlimited set is always a subspace
◮ Subspace is closed in Hilbert space ℓ2 (Z) or L2 (R)

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 68 / 96

Example Lecture: Sampling and Interpolation Classical View and Historical Notes

Bandwidth

## Definition (Bandwidth of function)

A function x is bandlimited when there exists ω0 ∈ [0, ∞) such that the Fourier
transform X satisfies

## Definition (Bandlimited sets)

The set of sequences in ℓ2 (Z) with bandwidth at most ω0 and the set of functions
in L2 (R) with bandwidth at most ω0 are denoted BL[−ω0 /2, ω0 /2]

## If ω0 < ω1 then BL[−ω0 /2, ω0 /2] ⊂ BL[−ω1 /2, ω1 /2]

A bandlimited set is always a subspace
◮ Subspace is closed in Hilbert space ℓ2 (Z) or L2 (R)

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 68 / 96

Example Lecture: Sampling and Interpolation Classical View and Historical Notes

Classical Sampling

6 0;
(sin t)/t, for t =
Recall: sinc(t) =
1, for t = 0

## Theorem (Sampling theorem)

Let x be a function and let T > 0. Define
X
b
x (t) = x(nT ) sinc( Tπ (t − nT )).
n∈Z

x = x.

## Easier in cycles/sec rather than radians/sec:

Need two samples per cycle of fastest component
Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 69 / 96
Example Lecture: Sampling and Interpolation Classical View and Historical Notes

Classical Sampling

6 0;
(sin t)/t, for t =
Recall: sinc(t) =
1, for t = 0

## Theorem (Sampling theorem)

Let x be a function and let T > 0. Define
X
b
x (t) = x(nT ) sinc( Tπ (t − nT )).
n∈Z

x = x.

## Easier in cycles/sec rather than radians/sec:

Need two samples per cycle of fastest component
Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 69 / 96
Example Lecture: Sampling and Interpolation Classical View and Historical Notes

History
Many names:
◮ Shannon sampling theorem
◮ Nyquist–Shannon sampling theorem
◮ Nyquist–Shannon–Kotel’nikov sampling theorem
◮ Whittaker–Shannon–Kotel’nikov sampling theorem
◮ Whittaker–Nyquist–Shannon–Kotel’nikov sampling theorem
Well-known people associated with sampling (but less often so):
◮ Cauchy (1841) – apparently not true
◮ Borel (1897)
◮ de la Vallée Poussin (1908)
◮ E. T. Whittaker (1915)
◮ J. M. Whittaker (1927)
◮ Gabor (1946)
Less-known people, almost lost to history:
◮ Ogura (1920)
◮ Küpfmüller (Küpfmüller filter)
◮ Raabe [PhD 1939] (assistant to Küpfmüller)
◮ Someya (1949)
◮ Weston (1949)
Best approach: use no names?
◮ Cardinal Theorem of interpolation theory
Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 70 / 96
Example Lecture: Sampling and Interpolation Classical View and Historical Notes

History
Many names:
◮ Shannon sampling theorem
◮ Nyquist–Shannon sampling theorem
◮ Nyquist–Shannon–Kotel’nikov sampling theorem
◮ Whittaker–Shannon–Kotel’nikov sampling theorem
◮ Whittaker–Nyquist–Shannon–Kotel’nikov sampling theorem
Well-known people associated with sampling (but less often so):
◮ Cauchy (1841) – apparently not true
◮ Borel (1897)
◮ de la Vallée Poussin (1908)
◮ E. T. Whittaker (1915)
◮ J. M. Whittaker (1927)
◮ Gabor (1946)
Less-known people, almost lost to history:
◮ Ogura (1920)
◮ Küpfmüller (Küpfmüller filter)
◮ Raabe [PhD 1939] (assistant to Küpfmüller)
◮ Someya (1949)
◮ Weston (1949)
Best approach: use no names?
◮ Cardinal Theorem of interpolation theory
Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 70 / 96
Example Lecture: Sampling and Interpolation Classical View and Historical Notes

History
Many names:
◮ Shannon sampling theorem
◮ Nyquist–Shannon sampling theorem
◮ Nyquist–Shannon–Kotel’nikov sampling theorem
◮ Whittaker–Shannon–Kotel’nikov sampling theorem
◮ Whittaker–Nyquist–Shannon–Kotel’nikov sampling theorem
Well-known people associated with sampling (but less often so):
◮ Cauchy (1841) – apparently not true
◮ Borel (1897)
◮ de la Vallée Poussin (1908)
◮ E. T. Whittaker (1915)
◮ J. M. Whittaker (1927)
◮ Gabor (1946)
Less-known people, almost lost to history:
◮ Ogura (1920)
◮ Küpfmüller (Küpfmüller filter)
◮ Raabe [PhD 1939] (assistant to Küpfmüller)
◮ Someya (1949)
◮ Weston (1949)
Best approach: use no names?
◮ Cardinal Theorem of interpolation theory
Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 70 / 96
Example Lecture: Sampling and Interpolation Classical View and Historical Notes

History
Many names:
◮ Shannon sampling theorem
◮ Nyquist–Shannon sampling theorem
◮ Nyquist–Shannon–Kotel’nikov sampling theorem
◮ Whittaker–Shannon–Kotel’nikov sampling theorem
◮ Whittaker–Nyquist–Shannon–Kotel’nikov sampling theorem
Well-known people associated with sampling (but less often so):
◮ Cauchy (1841) – apparently not true
◮ Borel (1897)
◮ de la Vallée Poussin (1908)
◮ E. T. Whittaker (1915)
◮ J. M. Whittaker (1927)
◮ Gabor (1946)
Less-known people, almost lost to history:
◮ Ogura (1920)
◮ Küpfmüller (Küpfmüller filter)
◮ Raabe [PhD 1939] (assistant to Küpfmüller)
◮ Someya (1949)
◮ Weston (1949)
Best approach: use no names?
◮ Cardinal Theorem of interpolation theory
Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 70 / 96
Example Lecture: Sampling and Interpolation Classical View and Historical Notes

## Sampling theorem: Conventional justification

P
Let b
x (t) = n∈Z x(nT ) g (t − nT ) [will deduce that g should be sinc]

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 71 / 96

Example Lecture: Sampling and Interpolation Classical View and Historical Notes

## Sampling theorem: Conventional justification

P
Let b
x (t) = n∈Z x(nT ) g (t − nT ) [will deduce that g should be sinc]
R∞
Since x(nT ) g (t − nT ) = −∞ x(τ ) g (t − τ ) δ(τ − nT ) dτ ,
Z ∞ X
b
x (t) = g (t − τ ) x(τ ) δ(τ − nT ) dτ
−∞ n∈Z

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 71 / 96

Example Lecture: Sampling and Interpolation Classical View and Historical Notes

## Sampling theorem: Conventional justification

P
Let b
x (t) = n∈Z x(nT ) g (t − nT ) [will deduce that g should be sinc]
R∞
Since x(nT ) g (t − nT ) = −∞ x(τ ) g (t − τ ) δ(τ − nT ) dτ ,
Z ∞ X
b
x (t) = g (t − τ ) x(τ ) δ(τ − nT ) dτ
−∞ n∈Z

## Recall: Fourier transform of Dirac comb

X  
FT 2π X 2π
δ(t − nT ) ←→ δ ω− k
T T
n∈Z k∈Z

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 71 / 96

Example Lecture: Sampling and Interpolation Classical View and Historical Notes

## Sampling theorem: Conventional justification

P
Let b
x (t) = n∈Z x(nT ) g (t − nT ) [will deduce that g should be sinc]
R∞
Since x(nT ) g (t − nT ) = −∞ x(τ ) g (t − τ ) δ(τ − nT ) dτ ,
Z ∞ X
b
x (t) = g (t − τ ) x(τ ) δ(τ − nT ) dτ
−∞ n∈Z
| {z }
h(τ )

## Recall: Fourier transform of Dirac comb

X  
FT 2π X 2π
δ(t − nT ) ←→ δ ω− k
T T
n∈Z k∈Z

X  
b (ω) = G (ω) 1
X X ω−

k
T T
k∈Z
| {z }
H(ω)

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 71 / 96

Example Lecture: Sampling and Interpolation Classical View and Historical Notes

X  
b (ω) = 1 G (ω)
X X ω−

k
T T
k∈Z

## b has “spectral replication”

Reconstruction X
b (ω) = X (ω) for all ω?
How can we have X
◮ x ∈ BL[−π/T , π/T ] implies replicas do not overlap

T , for |ω| < π/T ;
◮ G (ω) = selects “desired” replica with correct gain
0, for t = 0

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 72 / 96

Example Lecture: Sampling and Interpolation Classical View and Historical Notes

Dissatisfaction
Mathematical rigor of derivation:
X  
FT 2π X 2π
δ(t − nT ) ←→ δ ω− k
T T
n∈Z k∈Z

## Mathematical/technological plausibility of use:

X
b
x (t) = x(nT ) sinc( Tπ (t − nT ))
n∈Z

## ◮ At each t, reconstruction is an infinite sum

◮ Very slow decay of terms makes truncation accuracy poor
Technological implementability:
◮ Point measurements difficult to approximate physically
◮ Causality of reconstruction
Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 73 / 96
Example Lecture: Sampling and Interpolation Classical View and Historical Notes

Dissatisfaction
Mathematical rigor of derivation:
X  
FT 2π X 2π
δ(t − nT ) ←→ δ ω− k
T T
n∈Z k∈Z

## Mathematical/technological plausibility of use:

X
b
x (t) = x(nT ) sinc( Tπ (t − nT ))
n∈Z

## ◮ At each t, reconstruction is an infinite sum

◮ Very slow decay of terms makes truncation accuracy poor
Technological implementability:
◮ Point measurements difficult to approximate physically
◮ Causality of reconstruction
Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 73 / 96
Example Lecture: Sampling and Interpolation Classical View and Historical Notes

Dissatisfaction
Mathematical rigor of derivation:
X  
FT 2π X 2π
δ(t − nT ) ←→ δ ω− k
T T
n∈Z k∈Z

## Mathematical/technological plausibility of use:

X
b
x (t) = x(nT ) sinc( Tπ (t − nT ))
n∈Z

## ◮ At each t, reconstruction is an infinite sum

◮ Very slow decay of terms makes truncation accuracy poor
Technological implementability:
◮ Point measurements difficult to approximate physically
◮ Causality of reconstruction
Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 73 / 96
Example Lecture: Sampling and Interpolation Operator View

## Definition (Interpolation operator)

For a fixed positive T and interpolation postfilter g (t), let Φ : ℓ2 (Z) → L2 (R) be
given by X
(Φ y )(t) = yn √1T g (t/T − n), t∈R
n∈Z

Φ
y g (t) b
x
T

## Generalizes sinc interpolation

P
For simplicity, we will consider only T = 1: (Φ y )(t) = n∈Z yn g (t − n)

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 74 / 96

Example Lecture: Sampling and Interpolation Operator View

Reconstruction space
Range of interpolation operator has special form

## Definition (Shift-invariant subspace of L2 (R))

A subspace W ⊂ L2 (R) is a shift-invariant subspace with respect to shift T ∈ R+
when x(t) ∈ W implies x(t − kT ) ∈ W for every integer k. In addition,
w ∈ L2 (R) is called a generator of W when W = span({w (t − kT )}k∈Z).

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 75 / 96

Example Lecture: Sampling and Interpolation Operator View

## Definition (Sampling operator)

For a fixed positive T and sampling prefilter g ∗ (−t), let Φ∗ : L2 (R) → ℓ2 (Z) be
given by
(Φ∗ x)n = hx(t), √1T g (t/T − n)it , n∈Z

Φ∗
T
x g ∗ (−t) y

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 76 / 96

Example Lecture: Sampling and Interpolation Operator View

## Adjoint relationship between sampling and interpolation

Theorem
Sampling and interpolation operators are adjoints

## Let x ∈ L2 (R) and y ∈ ℓ2 (Z)

X
hΦ∗ x, y i = hx(t), g (t − n)it yn∗
n∈Z
X Z ∞ 
= x(t) g ∗ (t − n) dt yn∗
n∈Z −∞
Z !
∞ X

= x(t) g (t − n) yn∗ dt
−∞ n∈Z
Z !∗
∞ X
= x(t) yn g (t − n) dt
−∞ n∈Z
= hx, Φ y i

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 77 / 96

Example Lecture: Sampling and Interpolation Operator View

## Adjoint relationship between sampling and interpolation

Theorem
Sampling and interpolation operators are adjoints

## Let x ∈ L2 (R) and y ∈ ℓ2 (Z)

X
hΦ∗ x, y i = hx(t), g (t − n)it yn∗
n∈Z
X Z ∞ 
= x(t) g ∗ (t − n) dt yn∗
n∈Z −∞
Z !
∞ X

= x(t) g (t − n) yn∗ dt
−∞ n∈Z
Z !∗
∞ X
= x(t) yn g (t − n) dt
−∞ n∈Z
= hx, Φ y i

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 77 / 96

Example Lecture: Sampling and Interpolation Operator View

## Relationships between sampling and interpolation

x = Φ Φ∗ x
Sampling followed by interpolation: b

T
yn
x(t) g ∗ (−t) T
g (t) b
x (t)

b
x is best approximation of x within shift-invariant subspace generated by g if
P = Φ Φ∗ is an orthogonal projection operator

P is automatically self-adjoint: P ∗ = (Φ Φ∗ )∗ = P

Need P idempotent: P 2 = Φ Φ∗ Φ Φ∗ = P

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 78 / 96

Example Lecture: Sampling and Interpolation Operator View

## Relationships between sampling and interpolation

x = Φ Φ∗ x
Sampling followed by interpolation: b

T
yn
x(t) g ∗ (−t) T
g (t) b
x (t)

b
x is best approximation of x within shift-invariant subspace generated by g if
P = Φ Φ∗ is an orthogonal projection operator

P is automatically self-adjoint: P ∗ = (Φ Φ∗ )∗ = P

Need P idempotent: P 2 = Φ Φ∗ Φ Φ∗ = P

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 78 / 96

Example Lecture: Sampling and Interpolation Operator View

## Relationships between sampling and interpolation

x = Φ Φ∗ x
Sampling followed by interpolation: b

T
yn
x(t) g ∗ (−t) T
g (t) b
x (t)

b
x is best approximation of x within shift-invariant subspace generated by g if
P = Φ Φ∗ is an orthogonal projection operator

P is automatically self-adjoint: P ∗ = (Φ Φ∗ )∗ = P

Need P idempotent: P 2 = Φ Φ∗ Φ Φ∗ = P

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 78 / 96

Example Lecture: Sampling and Interpolation Operator View

## Relationships between sampling and interpolation

x = Φ Φ∗ x
Sampling followed by interpolation: b

T
yn
x(t) g ∗ (−t) T
g (t) b
x (t)

b
x is best approximation of x within shift-invariant subspace generated by g if
P = Φ Φ∗ is an orthogonal projection operator

P is automatically self-adjoint: P ∗ = (Φ Φ∗ )∗ = P

Need P idempotent: P 2 = Φ Φ∗ Φ Φ∗ = P

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 78 / 96

Example Lecture: Sampling and Interpolation Operator View

## Relationships between sampling and interpolation

Interpolation followed by sampling: yb = Φ∗ Φ y

b
x (t) T
yn
T
g (t) g ∗ (−t) ybn

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 79 / 96

Example Lecture: Sampling and Interpolation Operator View

## Relationships between sampling and interpolation

Interpolation followed by sampling: yb = Φ∗ Φ y

b
x (t) T
yn
T
g (t) g ∗ (−t) ybn

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 79 / 96

Example Lecture: Sampling and Interpolation Operator View

## Relationships between sampling and interpolation

Interpolation followed by sampling: yb = Φ∗ Φ y

b
x (t) T
yn
T
g (t) g ∗ (−t) ybn

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 79 / 96

Example Lecture: Sampling and Interpolation Operator View

## Sampling for shift-invariant subspaces

Theorem
Let g be orthogonal to its integer shifts: hg (t − n), g (t)it = δn . The system

T
yn
x(t) g ∗ (−t) T
g (t) b
x (t)

yields b
x = P x where P is the orthogonal projection operator onto the
shift-invariant subspace S generated by g .

Corollaries:
If x ∈ S, then x is recovered exactly from samples y

If x 6∈ S, then b
x is the best approximation of x in S

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 80 / 96

Example Lecture: Sampling and Interpolation Operator View

## Sampling for shift-invariant subspaces

Theorem
Let g be orthogonal to its integer shifts: hg (t − n), g (t)it = δn . The system

T
yn
x(t) g ∗ (−t) T
g (t) b
x (t)

yields b
x = P x where P is the orthogonal projection operator onto the
shift-invariant subspace S generated by g .

Corollaries:
If x ∈ S, then x is recovered exactly from samples y

If x 6∈ S, then b
x is the best approximation of x in S

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 80 / 96

Example Lecture: Sampling and Interpolation Operator View

T
yn
x(t) g ∗ (−t) T
g (t) b
x (t)

## sinc(πt) is orthogonal to its integer shifts

◮ Immediately, orthogonal projection property holds

g ∗ (−t) = g (t)

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 81 / 96

Example Lecture: Sampling and Interpolation Operator View

T
yn
x(t) g ∗ (−t) T
g (t) b
x (t)

## sinc(πt) is orthogonal to its integer shifts

◮ Immediately, orthogonal projection property holds

g ∗ (−t) = g (t)

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 81 / 96

Example Lecture: Sampling and Interpolation Operator View

## Definition (Sampling operator)

For a fixed positive N and sampling filter g , let Φ∗ : ℓ2 (Z) → ℓ2 (Z) be given by

Φ∗
∗ y
x g−n N

## Definition (Interpolation operator)

For a fixed positive N and interpolation filter g , let Φ : ℓ2 (Z) → ℓ2 (Z) be given by
X
(Φ y )n = yk gn−kN , n∈Z
k∈Z

Φ
y N gn b
x

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 82 / 96

Example Lecture: Sampling and Interpolation Operator View

## Definition (Shift-invariant subspace of ℓ2 (Z))

A subspace W ⊂ ℓ2 (Z) is a shift-invariant subspace with respect to shift L ∈ Z+
when xn ∈ W implies xn−kL ∈ W for every integer k. In addition, w ∈ ℓ2 (Z) is
called a generator of W when W = span({wn−kL }k∈Z ).

Theorem
Let g be orthogonal to its shifts by multiples of N: hgn−kN , gn in = δk . The system

yn
xn g−n N N gn b
xn

yields b
x = P x where P is the orthogonal projection operator onto the
shift-invariant subspace S generated by g with shift N.

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 83 / 96

Example Lecture: Sampling and Interpolation Operator View

## Beyond orthogonal case

T
yn
x(t) ge(t) T
g (t) b
x (t)

e ∗ , interpolation operator Φ
Sampling operator Φ

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 84 / 96

Example Lecture: Sampling and Interpolation Operator View

## Beyond orthogonal case

T
yn
x(t) ge(t) T
g (t) b
x (t)

e ∗ , interpolation operator Φ
Sampling operator Φ
e ∗ generally not self-adjoint, but can still be a projection operator
ΦΦ
e ∗ Φ = I for an oblique projection
Check Φ

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 84 / 96

Example Lecture: Sampling and Interpolation Operator View

## Beyond orthogonal case

T
yn
x(t) ge(t) T
g (t) b
x (t)

e ∗ , interpolation operator Φ
Sampling operator Φ
e ∗ generally not self-adjoint, but can still be a projection operator
ΦΦ
e ∗ Φ = I for an oblique projection
Check Φ
e
S
·
S
•x

b
x

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 84 / 96

Example Lecture: Sampling and Interpolation Basis Expansion View

## Definition (Basis synthesis operator)

Synthesis operator associated with basis {ϕk }k∈K for H
X
◮ Φ : ℓ2 (K) → H Φα = αk ϕk
k∈K

## Definition (Basis analysis operator)

Analysis operator associated with basis {ϕk }k∈K for H
◮ Φ∗ : H → ℓ2 (K) (Φ∗ x)k = hx, ϕk i, k ∈ K
P
When {ϕk }k∈K is an orthonormal set, k∈K hx, ϕk iϕk is an orthogonal
projection
Special case of a shift-variant space and a basis obtained from shifts of
function:
ϕk = g (t − k)

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 85 / 96

Example Lecture: Sampling and Interpolation Basis Expansion View

## Definition (Basis synthesis operator)

Synthesis operator associated with basis {ϕk }k∈K for H
X
◮ Φ : ℓ2 (K) → H Φα = αk ϕk
k∈K

## Definition (Basis analysis operator)

Analysis operator associated with basis {ϕk }k∈K for H
◮ Φ∗ : H → ℓ2 (K) (Φ∗ x)k = hx, ϕk i, k ∈ K
P
When {ϕk }k∈K is an orthonormal set, k∈K hx, ϕk iϕk is an orthogonal
projection
Special case of a shift-variant space and a basis obtained from shifts of
function:
ϕk = g (t − k)

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 85 / 96

Example Lecture: Sampling and Interpolation Extensions

Variations
Multichannel sampling

2T y0,n
x(t) ge0 (t)

2T y1,n
ge1 (t)

## Sample signal and derivatives

Many inverse problems have linear forward models, perhaps not shift-invariant
Similar subspace geometry holds

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 86 / 96

Example Lecture: Sampling and Interpolation Extensions

Variations
Multichannel sampling

2T y0,n
x(t) ge0 (t)

2T y1,n
ge1 (t)

## Sample signal and derivatives

Many inverse problems have linear forward models, perhaps not shift-invariant
Similar subspace geometry holds

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 86 / 96

Example Lecture: Sampling and Interpolation Extensions

Beyond subspaces

## Definition (Semilinear signal model)

A subset W ⊂ L2 (R) has rate of innovation ρ with respect to generator g if any
x ∈ W can be written as X
x = αk g (t − tk )
k∈Z

where
#{tk in [−T /2, T /2]} ρ
lim sup =
T →∞ T 2

W is not a subspace
For some g , exact recovery from uniform samples at rate ρ is possible
Many classes of techniques

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 87 / 96

Example Lecture: Sampling and Interpolation Summary

Summary

◮ Time reversal between sampling and interpolation

Subspaces
◮ Shift-invariant, range of interpolator Φ
◮ Null space of sampler Φ∗

Projection
◮ Φ Φ∗ always self adjoint
◮ Φ∗ Φ = I implies Φ Φ∗ is a projection operator
◮ Together, orthogonal projection operator, best approximation

Basis expansions
◮ Sampling produces analysis coefficients for basis expansion
◮ Interpolation synthesizes from expansion coefficients

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 88 / 96

Teaching Materials Textbooks

Textbooks

Two books:
M. Vetterli, J. Kovačević, and V. K. Goyal,
Foundations of Signal Processing

## J. Kovačević, V. K. Goyal, and M. Vetterli,

Fourier and Wavelet Signal Processing

## Manuscripts distributed in draft form online (originally as

a single volume and with some variations in titles) since
2010 at
http://www.fourierandwavelets.org

## Free, online versions have gray scale images, no PDF

hyperlinks, no exercises with solutions or exercises

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 89 / 96

Teaching Materials Textbooks

Textbooks
Foundations of Signal Processing
1 On Rainbows and Spectra
2 From Euclid to Hilbert
3 Sequences and Discrete-Time Systems
4 Functions and Continuous-Time Systems
5 Sampling and Interpolation
6 Approximation and Compression
7 Localization and Uncertainty

Features:
About 640 pages illustrated with more than 200 figures
More than 200 exercises (more than 30 with solutions within the text)
Solutions manual for instructors
Summary tables, guides to further reading, historical notes
Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 90 / 96
Teaching Materials Textbooks

Textbooks
Fourier and Wavelet Signal Processing
1 Filter Banks: Building Blocks of Time-Frequency Expansions
2 Local Fourier Bases on Sequences
3 Wavelet Bases on Sequences
4 Local Fourier and Wavelet Frames on Sequences
5 Local Fourier Transforms, Frames and Bases on Functions
6 Wavelet Bases, Frames and Transforms on Functions
7 Approximation, Estimation, and Compression

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 91 / 96

Teaching Materials Textbooks

Prerequisites

## Textbook is a mostly self-contained treatment

Mathematical maturity
◮ Mechanical use of calculus not enough
◮ Sophistication to read and write precise mathematical statements needed
(or could be learned here)
Linear algebra
◮ Basic facility with matrix algebra very useful
◮ Abstract view built carefully within the book
Probability
◮ Basic background (e.g., first half of Introduction to Probability by Bertsekas
and Tsitsiklis) needed (else all stochastic material could be skipped)
Signals and systems
◮ Basic background (e.g., Signals and Systems by Oppenheim and Willsky)
Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 92 / 96
Teaching Materials Supplementary materials

Solutions manual
Convolution of Derivative and Primitive
Let h and x be differentiable functions, and let
Z t Z t
h(1) (t) = h(τ ) dτ and x (1) (t) = x(τ ) dτ
−∞ −∞

## be their primitives. Give a sufficient condition for h ∗ x = h(1) ∗ x ′ based on

integration by parts.

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 93 / 96

Teaching Materials Supplementary materials

Solutions manual
From the definition of convolution, (4.35),
Z ∞
(h ∗ x)(t) = x(τ )h(t − τ ) dτ.
−∞

## We wish to apply definite integration by parts, (2.204b), to get to a form

involving h(1) and x ′ . With the associations

## u(τ ) = x(τ ) and v ′ (τ ) = h(t − τ ),

we obtain

u ′ (τ ) = x ′ (τ ) and v (τ ) = −h(1) (t − τ ).

## Substituting these into (2.204b) gives

Z ∞
t=∞
(h ∗ x)(t) = −x(τ ) h(1) (t − τ ) t=−∞ + h(1) (t − τ ) x ′ (τ ) dτ. (1)
−∞

τ →±∞

## Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 94 / 96

Teaching Materials Supplementary materials

Mathematica figures
|X10 (e jω )| |X100 (e jω )|
2 2

Π Π 3Π
Π
ω Π Π 3Π
Π
ω
4 2 4 4 2 4

## (a) N = 10 (b) N = 100

|X1000 (e jω )| |X1000 (e jω )|
2

Π Π 3Π
Π
ω 9Π Π ω
4 2 4 20 2

## Figure: Truncated DTFT of the sinc sequence,

illustrating the Gibbs phenomenon. Shown are
|XN (e jω )| from (3.84) with different N. Observe how
oscillations narrow from (a) to (c), but their amplitude
remains√ constant (the topmost grid line in every plot),
1.089 2.
Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 95 / 96
Wrap up