Image Interpolation

Introduction Interpolation Techniques

What is image interpolation? (D-A conversion) Why do we need it? 1D zero-order, first-order, third-order 2D = two sequential 1D (divide-and-conquer) Directional(Adaptive) interpolation* Digital zooming (resolution enhancement) Image inpainting (error concealment) Geometric transformations (where your imagination can fly)
EE465: Introduction to Digital Image Processing 1

Interpolation Applications

Introduction
What is image interpolation?
An image f(x,y) tells us the intensity values at the integral lattice locations, i.e., when x and y are both integers Image interpolation refers to the guess of intensity values at missing locations, i.e., x and y can be arbitrary Note that it is just a guess (Note that all sensors have finite sampling distance)
EE465: Introduction to Digital Image Processing 2

A Sentimental Comment
Havent we just learned from discrete sampling (A-D conversion)? Yes, image interpolation is about D-A conversion Recall the gap between biological vision and artificial vision systems
Digital: camera + computer Analog: retina + brain
EE465: Introduction to Digital Image Processing 3

Engineering Motivations
Why do we need image interpolation?
We want BIG images
When we see a video clip on a PC, we like to see it in the full screen mode

We want GOOD images

If some block of an image gets damaged during the transmission, we want to repair it

We want COOL images

Manipulate images digitally can render fancy artistic effects as we often see in movies
EE465: Introduction to Digital Image Processing 4

Scenario I: Resolution Enhancement

Low-Res. High-Res.

EE465: Introduction to Digital Image Processing

Scenario II: Image Inpainting

Non-damaged

Damaged
6

EE465: Introduction to Digital Image Processing

Scenario III: Image Warping

You will work on a digital fun mirror in CA#3

EE465: Introduction to Digital Image Processing 7

Image Interpolation
Introduction Interpolation Techniques
What is image interpolation? Why do we need it? 1D linear interpolation (elementary algebra) 2D = 2 sequential 1D (divide-and-conquer) Directional(adaptive) interpolation* Digital zooming (resolution enhancement) Image inpainting (error concealment) Geometric transformations
EE465: Introduction to Digital Image Processing 8

Interpolation Applications

1D Zero-order (Replication)
f(n) n

f(x) x

EE465: Introduction to Digital Image Processing

1D First-order Interpolation (Linear)

f(n) n

f(x) x

EE465: Introduction to Digital Image Processing

10

Linear Interpolation Formula

Heuristic: the closer to a pixel, the higher weight is assigned Principle: line fitting to polynomial fitting (analytical formula) f(n) f(n+a) f(n+1)

1-a

f(n+a)=(1-a) f(n)+a f(n+1), 0<a<1 Note: when a=0.5, we simply have the average of two

EE465: Introduction to Digital Image Processing

11

Numerical Examples
f(n)=[0,120,180,120,0] Interpolate at 1/2-pixel f(x)=[0,60,120,150,180,150,120,60,0], x=n/2 Interpolate at 1/3-pixel f(x)=[0,20,40,60,80,100,120,130,140,150,160,170,180,], x=n/6

EE465: Introduction to Digital Image Processing

12

1D Third-order Interpolation (Cubic)*

f(n) n f(x) x Cubic spline fitting
http://en.wikipedia.org/wiki/Spline_interpolation
EE465: Introduction to Digital Image Processing 13

From 1D to 2D
Engineers wisdom: divide and conquer 2D interpolation can be decomposed into two sequential 1D interpolations. The ordering does not matter (row-column = column-row) Such separable implementation is not optimal but enjoys low computational complexity If you dont know how to solve a problem, there must be a related but easier problem you know how to solve. See if you can reduce the problem to the easier one. - rephrased from G. Polyas How to Solve It
EE465: Introduction to Digital Image Processing 14

Graphical Interpretation of Interpolation at Half-pel

row

column

f(m,n)

g(m,n)

EE465: Introduction to Digital Image Processing

15

Numerical Examples
a c b d first-order

zero-order a a c c a a c c b b d d b b d d

a (a+c)/2 c

(a+b)/2b (a+b+c+d)/4(b+d)/2 (c+d)/2 d

EE465: Introduction to Digital Image Processing

16

Numerical Examples (Cont)

Col n row m X(m,n) a b Y 1-a Col n+1 X(m,n+1)

1-b row m+1 X(m+1,n) X(m+1,n+1)

Q: what is the interpolated value at Y? Ans.: (1-a)(1-b)X(m,n)+(1-a)bX(m+1,n) +a(1-b)X(m,n+1)+abX(m+1,n+1)

EE465: Introduction to Digital Image Processing 17

Bicubic Interpolation*

http://en.wikipedia.org/wiki/Bicubic_interpolation
EE465: Introduction to Digital Image Processing 18

Limitation with bilinear/bicubic

Edge blurring Jagged artifacts
Edge blurring Jagged artifacts

Z X X
EE465: Introduction to Digital Image Processing

Z
19

Edge-Sensitive Interpolation
Step 1: interpolate the missing pixels along the diagonal a x c d b Since |a-c|=|b-d| x has equal probability of being black or white

black or white?

Step 2: interpolate the other half missing pixels a d c Please refer to esi1.m and esi2.m in interp.zip
EE465: Introduction to Digital Image Processing 20

Since |a-c|>|b-d| x=(b+d)/2=black

NEDI Step 1: Interpolate diagonal pixels*

1
4
Implementation: -Formulate LS estimation problem with pixels at low resolution and solve {a1,a2,a3,a4} -Use {a1,a2,a3,a4} to interpolate the pixel 0 at the high resolution

1 0 2

Please refer to sri1.m in interp.zip

EE565 Advanced Image Processing Copyright Xin Li 21

NEDI Step 2: Interpolate the Other Half*

1
Implementation: -Formulate LS estimation problem with pixels at low resolution and solve {a1,a2,a3,a4} -Use {a1,a2,a3,a4} to interpolate the pixel 0 at the high resolution

2
2

1 0 4

Please refer to sri2.m in interp.zip

EE565 Advanced Image Processing Copyright Xin Li 22

Geometric Duality

Step-I

Step-I after rotation of 45-degree

Step-II

EE465: Introduction to Digital Image Processing

23

Image Interpolation
Introduction Interpolation Techniques
1D zero-order, first-order, third-order 2D zero-order, first-order, third-order Directional interpolation*

Interpolation Applications
Digital zooming (resolution enhancement) Image inpainting (error concealment) Geometric transformations (where your imagination can fly)
EE465: Introduction to Digital Image Processing 24

Pixel Replication

low-resolution image (100100) high-resolution image (400400)

EE465: Introduction to Digital Image Processing 25

Bilinear Interpolation

low-resolution image (100100)

high-resolution image (400400)

EE465: Introduction to Digital Image Processing 26

Bicubic Interpolation

low-resolution image (100100)

high-resolution image (400400)

EE465: Introduction to Digital Image Processing 27

Edge-Directed Interpolation (Li&Orchard2000)

low-resolution image (100100)

EE465: Introduction to Digital Image Processing

high-resolution image (400400)

28

(Color-Filter-Array Interpolation)

Image Demosaicing

Bayer Pattern

EE465: Introduction to Digital Image Processing

29

Image Example

Ad-hoc CFA Interpolation

Advanced CFA Interpolation

30

EE465: Introduction to Digital Image Processing

Error Concealment*

damaged

interpolated
EE465: Introduction to Digital Image Processing 31

Image Inpainting*

EE465: Introduction to Digital Image Processing

32

Image Mosaicing*

EE465: Introduction to Digital Image Processing

33

Geometric Transformation

In the virtual space, you can have any kind of apple you want! MATLAB functions: griddata, interp2, maketform, imtransform
EE465: Introduction to Digital Image Processing 34

Basic Principle
(x,y) (x,y) is a geometric transformation We are given pixel values at (x,y) and want to interpolate the unknown values at (x,y) Usually (x,y) are not integers and therefore we can use linear interpolation to guess their values
MATLAB implementation: z=interp2(x,y,z,x,y,method);
EE465: Introduction to Digital Image Processing 35

Rotation
y y x x

x' cos y ' = sin

sin x cos y
36

EE465: Introduction to Digital Image Processing

MATLAB Example
z=imread('cameraman.tif'); % original coordinates [x,y]=meshgrid(1:256,1:256); % new coordinates a=2; for i=1:256;for j=1:256; x1(i,j)=a*x(i,j); y1(i,j=y(i,j)/a; end;end % Do the interpolation z1=interp2(x,y,z,x1,y1,'cubic');
EE465: Introduction to Digital Image Processing 37

Rotation Example

=3o

EE465: Introduction to Digital Image Processing

38

Scale
a=1/2

x' a 0 x y ' = 0 1 / a y
EE465: Introduction to Digital Image Processing 39

Affine Transform

square

parallelogram
x' a11 a12 x d x y + d y ' = a 21 a22 y

EE465: Introduction to Digital Image Processing

40

Affine Transform Example

x' .5 1 x 0 y ' = .5 2 y + 1

EE465: Introduction to Digital Image Processing

41

Shear

square

parallelogram
x' 1 0 x d x y ' = s 1 y + d y

EE465: Introduction to Digital Image Processing

42

Shear Example

x ' 1 0 x 0 y ' = .5 1 y + 1
EE465: Introduction to Digital Image Processing 43

Projective Transform
B A D B A C

C D

square

x '=

a1 x + a2 y + a3 a7 x + a8 y + 1

quadrilateral

a4 x + a5 y + a6 y' = a7 x + a8 y + 1
EE465: Introduction to Digital Image Processing 44

Projective Transform Example

[ 0 0; 1 0; 1 1; 0 1]

[-4 2; -8 -3; -3 -5; 6 3]

45

EE465: Introduction to Digital Image Processing

Polar Transform

r= x +y
2

= tan

y x

EE465: Introduction to Digital Image Processing

46

Iris Image Unwrapping

EE465: Introduction to Digital Image Processing

47

Use Your Imagination

r -> sqrt(r)

http://astronomy.swin.edu.au/~pbourke/projection/imagewarp/
EE465: Introduction to Digital Image Processing 48

Free Form Deformation

Seung-Yong Lee et al., Image Metamorphosis Using Snakes and Free-Form Deformations,SIGGRAPH1985, Pages 439-448
EE465: Introduction to Digital Image Processing 49

Application into Image Metamorphosis

EE465: Introduction to Digital Image Processing

50

Summary of Image Interpolation

A fundamental tool in digital processing of images: bridging the continuous world and the discrete world Wide applications from consumer electronics to biomedical imaging Remains a hot topic after the IT bubbles break
EE465: Introduction to Digital Image Processing 51