You are on page 1of 27

# Sampled representations

## • How to store and compute with continuous functions?

• Common scheme for representation: samples
write down the function’s values at many points

## Sampling and reconstruction

CS 4620 Lecture 13

## [FvDFH fig.14.14b / Wolberg]

Cornell CS4620 Fall 2008 Lecture 13 • 1 Cornell CS4620 Fall 2008 Lecture 13 • 2

Reconstruction Filtering

## • Making samples back into a continuous function • Processing done on a function

for output (need realizable method) can be executed in continuous form (e.g. analog circuit)
for analysis or processing (need mathematical method) but can also be executed using sampled representation
amounts to “guessing” what the function did in between
• Simple example: smoothing by averaging
[FvDFH fig.14.14b / Wolberg]

Cornell CS4620 Fall 2008 Lecture 13 • 3 Cornell CS4620 Fall 2008 Lecture 13 • 4
Roots of sampling Sampling in digital audio

## • Nyquist 1928; Shannon 1949 • Recording: sound to analog to samples to disc

famous results in information theory
• Playback: disc to samples to analog to sound again
• 1940s: first practical uses in telecommunications how can we be sure we are filling in the gaps correctly?

## • 1960s: first digital audio systems

• 1970s: commercialization of digital audio
• 1982: introduction of the Compact Disc
the first high-profile consumer application

## • This is why all the terminology has a communications or audio

“flavor”
early applications are 1D; for us 2D (images) is important

Cornell CS4620 Fall 2008 Lecture 13 • 5 Cornell CS4620 Fall 2008 Lecture 13 • 6

Undersampling Undersampling

• What if we “missed” things between the samples? • What if we “missed” things between the samples?
• Simple example: undersampling a sine wave • Simple example: undersampling a sine wave
unsurprising result: information is lost unsurprising result: information is lost
surprising result: indistinguishable from lower frequency surprising result: indistinguishable from lower frequency
also was always indistinguishable from higher frequencies also was always indistinguishable from higher frequencies
aliasing: signals “traveling in disguise” as other frequencies aliasing: signals “traveling in disguise” as other frequencies

Cornell CS4620 Fall 2008 Lecture 13 • 7 Cornell CS4620 Fall 2008 Lecture 13 • 7
Undersampling Undersampling

• What if we “missed” things between the samples? • What if we “missed” things between the samples?
• Simple example: undersampling a sine wave • Simple example: undersampling a sine wave
unsurprising result: information is lost unsurprising result: information is lost
surprising result: indistinguishable from lower frequency surprising result: indistinguishable from lower frequency
also was always indistinguishable from higher frequencies also was always indistinguishable from higher frequencies
aliasing: signals “traveling in disguise” as other frequencies aliasing: signals “traveling in disguise” as other frequencies

Cornell CS4620 Fall 2008 Lecture 13 • 7 Cornell CS4620 Fall 2008 Lecture 13 • 7

Undersampling Undersampling

• What if we “missed” things between the samples? • What if we “missed” things between the samples?
• Simple example: undersampling a sine wave • Simple example: undersampling a sine wave
unsurprising result: information is lost unsurprising result: information is lost
surprising result: indistinguishable from lower frequency surprising result: indistinguishable from lower frequency
also was always indistinguishable from higher frequencies also was always indistinguishable from higher frequencies
aliasing: signals “traveling in disguise” as other frequencies aliasing: signals “traveling in disguise” as other frequencies

Cornell CS4620 Fall 2008 Lecture 13 • 7 Cornell CS4620 Fall 2008 Lecture 13 • 7
Preventing aliasing Preventing aliasing

## • Introduce lowpass filters: • Introduce lowpass filters:

remove high frequencies leaving only safe, low frequencies remove high frequencies leaving only safe, low frequencies
choose lowest frequency in reconstruction (disambiguate) choose lowest frequency in reconstruction (disambiguate)

Cornell CS4620 Fall 2008 Lecture 13 • 8 Cornell CS4620 Fall 2008 Lecture 13 • 8

## Linear filtering: a key idea Convolution warm-up

• Transformations on signals; e.g.: • basic idea: define a new function by averaging over a sliding window
bass/treble controls on stereo
• a simple example to start off: smoothing
blurring/sharpening operations in image editing
smoothing/noise reduction in tracking

• Key properties
linearity: filter(f + g) = filter(f) + filter(g)
shift invariance: behavior invariant to shifting the input
• delaying an audio signal
• sliding an image around

## • Can be modeled mathematically by convolution

Cornell CS4620 Fall 2008 Lecture 13 • 9 Cornell CS4620 Fall 2008 Lecture 13 • 10
Convolution warm-up Convolution warm-up

• basic idea: define a new function by averaging over a sliding window • basic idea: define a new function by averaging over a sliding window
• a simple example to start off: smoothing • a simple example to start off: smoothing

Cornell CS4620 Fall 2008 Lecture 13 • 10 Cornell CS4620 Fall 2008 Lecture 13 • 10

## Convolution warm-up Convolution warm-up

• basic idea: define a new function by averaging over a sliding window • basic idea: define a new function by averaging over a sliding window
• a simple example to start off: smoothing • a simple example to start off: smoothing

Cornell CS4620 Fall 2008 Lecture 13 • 10 Cornell CS4620 Fall 2008 Lecture 13 • 10
Convolution warm-up Discrete convolution

## • Convolution: same idea but with weighted average

each sample gets its own weight (normally zero far away)

## • This is all convolution is: it is a moving weighted average

Cornell CS4620 Fall 2008 Lecture 13 • 11 Cornell CS4620 Fall 2008 Lecture 13 • 12

## Filters Convolution and filtering

• Sequence of weights a[j] is called a filter • Can express sliding average as convolution with a box filter
• Filter is nonzero over its region of support • abox = […, 0, 1, 1, 1, 1, 1, 0, …]
usually centered on zero: support radius r

## • Filter is normalized so that it sums to 1.0

this makes for a weighted average, not just any
old weighted sum

## • Most filters are symmetric about 0

since for images we usually want to treat
left and right the same

a box filter

Cornell CS4620 Fall 2008 Lecture 13 • 13 Cornell CS4620 Fall 2008 Lecture 13 • 14
Example: box and step Example: box and step

Cornell CS4620 Fall 2008 Lecture 13 • 15 Cornell CS4620 Fall 2008 Lecture 13 • 15

## Example: box and step Example: box and step

Cornell CS4620 Fall 2008 Lecture 13 • 15 Cornell CS4620 Fall 2008 Lecture 13 • 15
Example: box and step Convolution and filtering

## • Convolution applies with any sequence of weights

• Example: bell curve (gaussian-like) […, 1, 4, 6, 4, 1, …]/16

Cornell CS4620 Fall 2008 Lecture 13 • 15 Cornell CS4620 Fall 2008 Lecture 13 • 16

## • Convolution applies with any sequence of weights

• Example: bell curve (gaussian-like) […, 1, 4, 6, 4, 1, …]/16

Cornell CS4620 Fall 2008 Lecture 13 • 16 Cornell CS4620 Fall 2008 Lecture 13 • 17
Discrete convolution Discrete filtering in 2D

## • Notation: • Same equation, one more index

• Convolution is a multiplication-like operation
commutative
associative
now the filter is a rectangle you slide around over a grid of numbers
scalars factor out • Commonly applied to images
identity: unit impulse e = […, 0, 0, 1, 0, 0, …] blurring (using box, using gaussian, …)
sharpening (impulse minus blur)

## • Conceptually no distinction between filter and signal • Usefulness of associativity

often apply several filters one after another: (((a * b1) * b2) * b3)
this is equivalent to applying one filter: a * (b1 * b2 * b3)

Cornell CS4620 Fall 2008 Lecture 13 • 18 Cornell CS4620 Fall 2008 Lecture 13 • 19

And in pseudocode…

[Philip Greenspun]
original!!!!|!!!!box blur sharpened!!!!|!!!!gaussian blur

Cornell CS4620 Fall 2008 Lecture 13 • 20 Cornell CS4620 Fall 2008 Lecture 13 • 21
Optimization: separable filters

## • definition: a2(x,y) is separable if it can be written as:

[Philip Greenspun] this is a useful property for filters because it allows factoring:
original!!!!|!!!!box blur sharpened!!!!|!!!!gaussian blur

Cornell CS4620 Fall 2008 Lecture 13 • 21 Cornell CS4620 Fall 2008 Lecture 13 • 22

## second, convolve with this

first, convolve with this first, convolve with this

Cornell CS4620 Fall 2008 Lecture 13 • 23 Cornell CS4620 Fall 2008 Lecture 13 • 23
Continuous convolution: warm-up Continuous convolution: warm-up

• Can apply sliding-window average to a continuous function just as • Can apply sliding-window average to a continuous function just as
well well
output is continuous output is continuous
integration replaces summation integration replaces summation

Cornell CS4620 Fall 2008 Lecture 13 • 24 Cornell CS4620 Fall 2008 Lecture 13 • 24

## Continuous convolution: warm-up Continuous convolution: warm-up

• Can apply sliding-window average to a continuous function just as • Can apply sliding-window average to a continuous function just as
well well
output is continuous output is continuous
integration replaces summation integration replaces summation

Cornell CS4620 Fall 2008 Lecture 13 • 24 Cornell CS4620 Fall 2008 Lecture 13 • 24
Continuous convolution: warm-up Continuous convolution

• Can apply sliding-window average to a continuous function just as • Sliding average expressed mathematically:
well
output is continuous
integration replaces summation note difference in normalization (only for box)

## weighting is now by a function

weighted integral is like weighted average
again bounds are set by support of f(x)

Cornell CS4620 Fall 2008 Lecture 13 • 24 Cornell CS4620 Fall 2008 Lecture 13 • 25

## One more convolution Continuous-discrete convolution

• Continuous–discrete convolution

## used for reconstruction and resampling

Cornell CS4620 Fall 2008 Lecture 13 • 26 Cornell CS4620 Fall 2008 Lecture 13 • 27
Continuous-discrete convolution Continuous-discrete convolution

Cornell CS4620 Fall 2008 Lecture 13 • 27 Cornell CS4620 Fall 2008 Lecture 13 • 27

## Continuous-discrete convolution Continuous-discrete convolution

Cornell CS4620 Fall 2008 Lecture 13 • 27 Cornell CS4620 Fall 2008 Lecture 13 • 27
Resampling Resampling

## • Changing the sample rate • Reconstruction creates a continuous function

in images, this is enlarging and reducing forget its origins, go ahead and sample it

## • Creating more samples:

increasing the sample rate
“upsampling”
“enlarging”

## • Ending up with fewer samples:

decreasing the sample rate
“downsampling”
“reducing”

Cornell CS4620 Fall 2008 Lecture 13 • 28 Cornell CS4620 Fall 2008 Lecture 13 • 29

Resampling Resampling

## • Reconstruction creates a continuous function • Reconstruction creates a continuous function

forget its origins, go ahead and sample it forget its origins, go ahead and sample it

Cornell CS4620 Fall 2008 Lecture 13 • 29 Cornell CS4620 Fall 2008 Lecture 13 • 29
Resampling Resampling

## • Reconstruction creates a continuous function • Reconstruction creates a continuous function

forget its origins, go ahead and sample it forget its origins, go ahead and sample it

Cornell CS4620 Fall 2008 Lecture 13 • 29 Cornell CS4620 Fall 2008 Lecture 13 • 29

## loop over nearby pixels,

average using filter weight
looks like discrete filter,
but offsets are not integers
and filter is continuous
remember placement of filter
relative to grid is variable

Cornell CS4620 Fall 2008 Lecture 13 • 30 Cornell CS4620 Fall 2008 Lecture 13 • 31
Cont.–disc. convolution in 2D Separable filters for resampling

## • just as in filtering, separable filters are useful

separability in this context is a statement about a continuous filter, rather than a
discrete one:

## • resample in two passes, one resampling each row and one

resampling each column
• intermediate storage required: product of one dimension of src. and
the other dimension of dest.
• same yucky details about boundary conditions

Cornell CS4620 Fall 2008 Lecture 13 • 32 Cornell CS4620 Fall 2008 Lecture 13 • 33

A gallery of filters

• Box filter
Simple and cheap

• Tent filter
[Philip Greenspun]

Linear interpolation

• Gaussian filter
Very smooth antialiasing filter

• B-spline cubic
Very smooth

## two-stage resampling using a • Catmull-rom cubic

separable filter Interpolating

• Mitchell-Netravali cubic
Good for image upsampling

Cornell CS4620 Fall 2008 Lecture 13 • 34 Cornell CS4620 Fall 2008 Lecture 13 • 35
Box filter Tent filter

Cornell CS4620 Fall 2008 Lecture 13 • 36 Cornell CS4620 Fall 2008 Lecture 13 • 37

## Gaussian filter B-Spline cubic

−3(1 − |x|)3 + 3(1 − |x|)2 + 3(1 − |x|) + 1 −1 ≤ x ≤ 1,
1
fB (x) = (2 − |x|)3 1 ≤ |x| ≤ 2,
6

0 otherwise.

Cornell CS4620 Fall 2008 Lecture 13 • 38 Cornell CS4620 Fall 2008 Lecture 13 • 39
Catmull-Rom cubic Michell-Netravali cubic

−3(1 − |x|)3 + 4(1 − |x|)2 + (1 − |x|) −1 ≤ x ≤ 1, 1 2
1 fM (x) = fB (x) + fC (x)
fC (x) = (2 − |x|)3 − (2 − |x|)2 1 ≤ |x| ≤ 2, 3  3
2
  −21(1 − |x|)3 + 27(1 − |x|)2 + 9(1 − |x|) + 1 −1 ≤ x ≤ 1,
0 otherwise. 1 
= 7(2 − |x|)3 − 6(2 − |x|)2 1 ≤ |x| ≤ 2,
18 

0 otherwise.

Cornell CS4620 Fall 2008 Lecture 13 • 40 Cornell CS4620 Fall 2008 Lecture 13 • 41

## Effects of reconstruction filters Properties of filters

• For some filters, the reconstruction process winds up implementing a • Degree of continuity
simple algorithm
• Impulse response
• Box filter (radius 0.5): nearest neighbor sampling
• Interpolating or no
box always catches exactly one input point
it is the input point nearest the output point • Ringing, or overshoot
so output[i, j] = input[round(x(i)), round(y(j))]
x(i) computes the position of the output coordinate i on the input grid

## • Tent filter (radius 1): linear interpolation

tent catches exactly 2 input points
weights are a and (1 – a)
result is straight-line interpolation from one point to the next

## interpolating filter used for reconstruction

Cornell CS4620 Fall 2008 Lecture 13 • 42 Cornell CS4620 Fall 2008 Lecture 13 • 43
Ringing, overshoot, ripples Constructing 2D filters

## • Overshoot • Separable filters (most common approach)

caused by
negative filter
values

• Ripples
constant in,
non-const. out
ripple free when:

Cornell CS4620 Fall 2008 Lecture 13 • 44 Cornell CS4620 Fall 2008 Lecture 13 • 45

## Yucky details Yucky details

• What about near the edge? • What about near the edge?
the filter window falls off the edge of the image the filter window falls off the edge of the image
need to extrapolate need to extrapolate
methods: methods:
• clip filter (black) • clip filter (black)
• wrap around • wrap around
• copy edge • copy edge
• reflect across edge • reflect across edge
• vary filter near edge • vary filter near edge
[Philip Greenspun]

[Philip Greenspun]
Cornell CS4620 Fall 2008 Lecture 13 • 46 Cornell CS4620 Fall 2008 Lecture 13 • 46
Yucky details Yucky details

• What about near the edge? • What about near the edge?
the filter window falls off the edge of the image the filter window falls off the edge of the image
need to extrapolate need to extrapolate
methods: methods:
• clip filter (black) • clip filter (black)
• wrap around • wrap around
• copy edge • copy edge
• reflect across edge • reflect across edge
• vary filter near edge • vary filter near edge

[Philip Greenspun]

[Philip Greenspun]
Cornell CS4620 Fall 2008 Lecture 13 • 46 Cornell CS4620 Fall 2008 Lecture 13 • 46

## Yucky details Yucky details

• What about near the edge? • What about near the edge?
the filter window falls off the edge of the image the filter window falls off the edge of the image
need to extrapolate need to extrapolate
methods: methods:
• clip filter (black) • clip filter (black)
• wrap around • wrap around
• copy edge • copy edge
• reflect across edge • reflect across edge
• vary filter near edge • vary filter near edge
[Philip Greenspun]

[Philip Greenspun]
Cornell CS4620 Fall 2008 Lecture 13 • 46 Cornell CS4620 Fall 2008 Lecture 13 • 46
Yucky details Yucky details

• What about near the edge? • What about near the edge?
the filter window falls off the edge of the image the filter window falls off the edge of the image
need to extrapolate need to extrapolate
methods: methods:
• clip filter (black) • clip filter (black)
• wrap around • wrap around
• copy edge • copy edge
• reflect across edge • reflect across edge
• vary filter near edge • vary filter near edge

[Philip Greenspun]

[Philip Greenspun]
Cornell CS4620 Fall 2008 Lecture 13 • 46 Cornell CS4620 Fall 2008 Lecture 13 • 46

## Yucky details Yucky details

• What about near the edge? • What about near the edge?
the filter window falls off the edge of the image the filter window falls off the edge of the image
need to extrapolate need to extrapolate
methods: methods:
• clip filter (black) • clip filter (black)
• wrap around • wrap around
• copy edge • copy edge
• reflect across edge • reflect across edge
• vary filter near edge • vary filter near edge
[Philip Greenspun]

[Philip Greenspun]
Cornell CS4620 Fall 2008 Lecture 13 • 46 Cornell CS4620 Fall 2008 Lecture 13 • 46
Yucky details Yucky details

• What about near the edge? • What about near the edge?
the filter window falls off the edge of the image the filter window falls off the edge of the image
need to extrapolate need to extrapolate
methods: methods:
• clip filter (black) • clip filter (black)
• wrap around • wrap around
• copy edge • copy edge
• reflect across edge • reflect across edge
• vary filter near edge • vary filter near edge

[Philip Greenspun]

[Philip Greenspun]
Cornell CS4620 Fall 2008 Lecture 13 • 46 Cornell CS4620 Fall 2008 Lecture 13 • 46

## • What about near the edge? • Very common operation

the filter window falls off the edge of the image devices have differing resolutions
need to extrapolate applications have different memory/quality tradeoffs
methods:
• Also very commonly done poorly
• clip filter (black)
• wrap around • Simple approach: drop/replicate pixels
• copy edge
• reflect across edge • Correct approach: use resampling
• vary filter near edge
[Philip Greenspun]

Cornell CS4620 Fall 2008 Lecture 13 • 46 Cornell CS4620 Fall 2008 Lecture 13 • 47
Resampling example Resampling example

Cornell CS4620 Fall 2008 Lecture 13 • 48 Cornell CS4620 Fall 2008 Lecture 13 • 48

## Resampling example Resampling example

Cornell CS4620 Fall 2008 Lecture 13 • 48 Cornell CS4620 Fall 2008 Lecture 13 • 48
Resampling example Resampling example

Cornell CS4620 Fall 2008 Lecture 13 • 48 Cornell CS4620 Fall 2008 Lecture 13 • 48

## Resampling example Resampling example

Cornell CS4620 Fall 2008 Lecture 13 • 48 Cornell CS4620 Fall 2008 Lecture 13 • 48
Resampling example Resampling example

Cornell CS4620 Fall 2008 Lecture 13 • 48 Cornell CS4620 Fall 2008 Lecture 13 • 48

## • Very common operation

devices have differing resolutions

## • Also very commonly done poorly

• Simple approach: drop/replicate pixels
• Correct approach: use resampling

## 1000 pixel width [Philip Greenspun]

Cornell CS4620 Fall 2008 Lecture 13 • 49 Cornell CS4620 Fall 2008 Lecture 13 • 50
[Philip Greenspun]

[Philip Greenspun]
by dropping pixels gaussian filter
box reconstruction filter bicubic reconstruction filter

## 250 pixel width 4000 pixel width

Cornell CS4620 Fall 2008 Lecture 13 • 51 Cornell CS4620 Fall 2008 Lecture 13 • 52

Types of artifacts

• Garden variety
what we saw in this natural image
fine features become jagged or sparkle

• Moiré patterns

## [Hearn & Baker cover]

600ppi scan of a color halftone image

Cornell CS4620 Fall 2008 Lecture 13 • 53 Cornell CS4620 Fall 2008 Lecture 13 • 54
Types of artifacts

• Garden variety
what we saw in this natural image
fine features become jagged or sparkle

• Moiré patterns
caused by repetitive patterns in input
produce large-scale artifacts; highly visible

• These artifacts are aliasing just like in the audio example earlier

## [Hearn & Baker cover]

• How do I know what filter is best at preventing aliasing?