Image Processing,

Retrieval, and Analysis (I)

Prof. Christian Bauckhage
Outline
Lecture 21

Recap

Artistic Warps
Superpositions

Tiling Effects
Pixelize
Slices
Radial Tiles

Mesh Warping
Mappings between Quadrilaterals

Summary
Recap

fisheye / lens effect

I enlarge a part of an image as if it was viewed through a
magnifying glass

example
Recap

approach (1D)
I we consider a warp functionT to the effect

g̃(x̃) = g(T(x))

I letting x > 0 be the distance to the center of

the (virtual) magnifying glass, we require that

T(x) ' x, if x is large

and

T(x) < x, if x is small

Recap

approach (1D)
I that is

either add a quantity to x which is negative for small x and equals

0 for large x

T(x) = δ+ (x) + x

or multiply a quantity to x which is less than 1 for small x and

equals 1 for large x

T(x) = δ· (x) · x
Recap

examples of lenses (I)

I different functions δ+ (r, rmax ) on a 400 × 400 image
I in each example, rmax = 150

   
r r2
original δ+ = −r · 1 − rmax δ+ = −r · 1 − 2
rmax
Recap

examples of lenses (II)

I different functions δ+ (r, rmax ) on a 400 × 400 image
I in each example, rmax = 150

√2 −r2
   
rmax r2
δ+ = −r · rmax δ+ = −r · tanh r
rmax δ+ = −r · exp − 21 2
rmax
Recap

practical computation (2D)

I understand T as a central force

T = T(µ, x) = T(r, x)

where

r=µ−x
Recap

practical computation (2D)

I understand T as a central force

T = T(µ, x) = T(r, x)

where

r=µ−x

⇔ understand T as a distance dependent displacement

in the direction of the source of the force

T(µ, x) = x + ∆(µ − x)
Recap

example

I here, we applied
(µ−x)2
g̃(x) = g x + e−


2σ2 · (µ − x)

where µ = (200, 150) and σ = 150

Recap

waves effect
I transform the image such that it appears as if on the
bottom of a swimming pool whose water is in motion

example
Recap

approach (1D)


I as usual, we are interested in g̃(x̃) = g T(x)
I we choose

T(x) = x + α sin(νx − φ)

where the parameters ν, α, and φ denote

the frequency
the amplitude
the phase
of the wave
Recap

practical computation (2D)

I again, we follow the idea of a central force
I we choose

r
T(r, x) = x + α sin νkrk − φ ·
krk
= x + ∆(r, α, ν, φ)
Recap

swirl effect
I twist an image about a point µ just as if you would put an
immersion blender into it

example
Recap

principle
I each pixel x of the images is being displaced by a vector ∆

∆
r

x x
r ϕ′ r
ϕ
µ µ
Recap

practical computation (2D)

I accordingly, we may again understand the effect to be
caused by a central force

T(r, x) = x + ∆(r)

where we choose
 
  
cos ϕ(r) + δ+ krk − cos ϕ(r)
∆(r) = krk ·  
  
+
sin ϕ(r) + δ krk − sin ϕ(r)
Artistic Warps

observation
I the idea of understanding effects to be caused by a central
force allows for an easy extension towards multiple centers
Artistic Warps

observation
I the idea of understanding effects to be caused by a central
force allows for an easy extension towards multiple centers
I to this end, we make use of the principle of superposition
Artistic Warps

observation
I the idea of understanding effects to be caused by a central
force allows for an easy extension towards multiple centers
I to this end, we make use of the principle of superposition
I for instance, if we are given n centers µ1 , µ2 , . . . , µn of
forces, we can define more general transformation
functions
X
n
T(r1 , r2 , . . . , rn , x) = x + ∆(ri , π1 , π2 , . . . , πm )
i=1

where the π1 , π2 , . . . , πm denote possible further

parameters of the function ∆
Artistic Warps

example (I)
Artistic Warps

example (II)
Artistic Warps

example (III)
Artistic Warps

explanation
I the examples show superimposed fisheye, waves,
and swirl effects

I the centers µi of the individual transformations are

     
128 400 640
µ1 = , µ2 = , and µ3 =
160 220 30

in all three cases

Tiling Effects

pixelize
I artificially reduce the resolution of an image

example

Original 10 × 10
Tiling Effects

example

20 × 20 40 × 40
Tiling Effects

approach
I combine several pixels into superpixels
Tiling Effects

approach
I combine several pixels into superpixels

I set the intensity of a superpixel to the mean

of the intensities of the pixels it comprises
Tiling Effects

approach
I combine several pixels into superpixels

I set the intensity of a superpixel to the mean

of the intensities of the pixels it comprises

I incorporate the parameters

tilesizeX and tilesizeY
Tiling Effects

“naı̈ve” python implementation

y = 0
while y < rmax:
x = 0
while x < cmax:
meancolor = 0.0
for y2 in range(tileziseY):
for x2 in range(tilesizeX):
meancolor += gOrig[y+y2,x+x2]

meancolor /= tilesizeX * tileziseY

for y2 in range(tileziseY):
for x2 in range(tilesizeX):
gWarp[y+y2,x+x2] = int(meancolor)

x += tilesizeX

y += tilesizeY
Tiling Effects

slices
I divide an image into many tiny mosaic pieces and slightly
shift their positions

example

tilesizeX = 20, tilesizeY = 20 tilesizeX = 20, tilesizeY = 20

dXmax = 10, dYmax = 10 dXmax = 20, dYmax = 20
Tiling Effects

example

tilesizeX = 40, tilesizeY = 40 tilesizeX = 40, tilesizeY = 40

dXmax = 10, dYmax = 10 dXmax = 20, dYmax = 20
Tiling Effects

approach
I implement loops similar to the pixelize effect

I choose parameters such as

tilesizeX, tilesizeY, dXmax, dYmax
Tiling Effects

“naı̈ve” python implementation

import numpy as np

y = 0
while y < rmax:
x = 0
while x < cmax:

dx = np.random.randint(-dXmax, dXmax)
dy = np.random.randint(-dYmax, dYmax)

for y2 in range(tileziseY):
for x2 in range(tilesizeX):
if y+y2+dy >= 0 and y+y2+dy < rmax and \\
x+x2+dx >= 0 and x+x2+xy < cmax:
g_warp[y+y2+dy,x+x2+dx] = g_orig[y+y2,x+x2]

x += tilesizeX

y += tilesizeY
Tiling Effects

variation
I punch out confetti and slightly shift them

example

r = 20, dXmax = 0, dYmax = 0 r = 20, dXmax = 15, dYmax = 15

Tiling Effects

example

r = 10, dXmax = 15, dYmax = 15 r = 30, dXmax = 20, dYmax = 20

Tiling Effects

radial tiles
I transit to polar coordinates and compute tiles along the
radial axis

example

original δ = 10
Tiling Effects

example

δ = 20 δ = 40
Tiling Effects

“naı̈ve” python implementation

import numpy as np

for y in range(rmax):
for x in range(cmax):
r = np.sqrt((x-px)**2 + (y-py)**2)
phi = np.atan2(y-py, x-px)

r = r - ((int)r % (int)delta)

u = r * np.cos(phi) + px
v = r * np.sin(phi) + py

gWarp[y,x] = resample(gOrig[u,v])
Mesh Warping

motivation (I)
Mesh Warping

motivation (II)
Mesh Warping

linear mappings between intervals

I assume an interval [a, b] where a, b ∈ R and a < b
Mesh Warping

linear mappings between intervals

I assume an interval [a, b] where a, b ∈ R and a < b

I if [a, b] is linearly mapped to [c, d], c < d . . .

Mesh Warping

linear mappings between intervals

I assume an interval [a, b] where a, b ∈ R and a < b

I if [a, b] is linearly mapped to [c, d], c < d . . .

I . . . then all x ∈ [a, b] are mapped to x 0 ∈ [c, d]

Mesh Warping

linear mappings between intervals

I assume an interval [a, b] where a, b ∈ R and a < b

I if [a, b] is linearly mapped to [c, d], c < d . . .

I . . . then all x ∈ [a, b] are mapped to x 0 ∈ [c, d]

I we have
x−a
x0 = (d − c) + c
b−a
Mesh Warping

bilinear mappings between quadrilaterals

x11

xb xd

1 x
x10
x01
v xa
u
xc

x00
u 1
Mesh Warping

bilinear mappings
I we have
u−0
xa = (x10 − x00 ) + x00
1−0
1
= u(x10 − x00 ) + x00 (1)
v u
u−0
xb = (x11 − x01 ) + x01
1−0 u 1

= u(x11 − x01 ) + x01 (2)

x11

xb xd
and
x
x10

v−0 x01
xa
x= (xb − xa ) + xa xc
1−0 x00
= v(xb − xa ) + xa (3)
Mesh Warping

bilinear mappings
I plugging (1) and (2) into (3) yields

x =uv(x11 + x00 − x10 − x01 )+

u(x10 − x00 ) + v(x01 − x00 ) + x00 1

v u

u 1

x11

xb xd

x
x10
x01
xa
xc

x00
Mesh Warping

bilinear mappings
I plugging (1) and (2) into (3) yields

x =uv(x11 + x00 − x10 − x01 )+

u(x10 − x00 ) + v(x01 − x00 ) + x00 1

v u
I in matrix form
 
 uv
u 1
  
x a b c d 
u

= x11
y e f g h v
xb xd
1 x
x10
x01
xa
xc

x00
Mesh Warping

bilinear mappings
I for the coefficients a, b, . . . , h we have

a = x11 + x00 − x10 − x01

b = x10 − x00
c = x01 − x00
d = x00
e = y11 + y00 − y10 − y01
f = y10 − y00
g = y01 − y00
h = y00
Mesh Warping

note:

I again, we have a geometric coordinate transformation

I that is, a warp x = T(u)

Mesh Warping

properties of bilinear mappings

I bilinear maps T have several interesting characteristics:
I T become an affine map once a = e = 0
Mesh Warping

properties of bilinear mappings

I bilinear maps T have several interesting characteristics:
I T become an affine map once a = e = 0

I horizontal and vertical lines are mapped onto lines

Mesh Warping

properties of bilinear mappings

I bilinear maps T have several interesting characteristics:
I T become an affine map once a = e = 0

I horizontal and vertical lines are mapped onto lines

I ratios of distance between equidistant points are preserved

Mesh Warping

properties of bilinear mappings

I bilinear maps T have several interesting characteristics:
I T become an affine map once a = e = 0

I horizontal and vertical lines are mapped onto lines

I ratios of distance between equidistant points are preserved

I diagonal lines are mapped onto curves

Mesh Warping

properties of bilinear mappings

original bilinear warp

Mesh Warping

properties of bilinear mappings

I the composition of two bilinear maps is not
another bilinear map but a biquadratic map
Mesh Warping

properties of bilinear mappings

I the composition of two bilinear maps is not
another bilinear map but a biquadratic map

I the inverse mapping T −1 , too, is not bilinear

Mesh Warping

properties of bilinear mappings

I the composition of two bilinear maps is not
another bilinear map but a biquadratic map

I the inverse mapping T −1 , too, is not bilinear

I we can see this from considering

x = auv + bu + cv + d
= v(au + c) + bu + d
x − bu − d
⇒v=
au + c
Mesh Warping

properties of bilinear mappings

I plugging v into

y = euv + fu + gv + h

yields
x − bu − d x − bu − d
y =eu + fu + g +h
au + c au + c
⇒ 0 =eu(x − bu − d) + fu(au + c)+
g(x − bu − d) + (h − y)(au + c)
⇔ 0 =eux − ebu2 − eud + fau2 + fcu + gx − gbu − gd+
hau + hc − yau − yc
⇔ 0 =(fa − eb)u2 + (ex − ed + fc − gb + ha − ya)u+
gx − gd + hc − yc
Mesh Warping

properties of bilinear mappings

I using the substitutions
A = af − be
B = ex − ay − de + cf − bg + ah
C = gy − cy + ch − dg
it follows that
Au2 + Bu + C = 0
Mesh Warping

properties of bilinear mappings

I using the substitutions
A = af − be
B = ex − ay − de + cf − bg + ah
C = gy − cy + ch − dg
it follows that
Au2 + Bu + C = 0

I for u we therefore have

r 
B B 2 C
u=− ± −
2A 2A A
Mesh Warping

properties of bilinear mappings

I using the substitutions
A = af − be
B = ex − ay − de + cf − bg + ah
C = gy − cy + ch − dg
it follows that
Au2 + Bu + C = 0

I for u we therefore have

r 
B B 2 C
u=− ± −
2A 2A A

I i.e. T −1 is not even unique!

Mesh Warping

bilinear mappings in image processing

I bilinear mappings T are defined through a set of 8
parameters {a, b, . . . , h}
Mesh Warping

bilinear mappings in image processing

I bilinear mappings T are defined through a set of 8
parameters {a, b, . . . , h}

⇔ if an image is supposed to be warped using a bilinear

transformation, these parameters need to be provided
or have to be determined from data
Mesh Warping

bilinear mappings in image processing

I bilinear mappings T are defined through a set of 8
parameters {a, b, . . . , h}

⇔ if an image is supposed to be warped using a bilinear

transformation, these parameters need to be provided
or have to be determined from data

I the latter will require 8 equations

Mesh Warping

bilinear mappings in image processing

I bilinear mappings T are defined through a set of 8
parameters {a, b, . . . , h}

⇔ if an image is supposed to be warped using a bilinear

transformation, these parameters need to be provided
or have to be determined from data

I the latter will require 8 equations

I these 8 equations were available, if for 4 points ui in the

source image we would know 4 corresponding points xi
in the destination image
Mesh Warping

bilinear mappings in image processing

I we could then derive

x1 = au1 v1 + bu1 + cv1 + d

..
.
x4 = au4 v4 + bu4 + cv4 + d
y1 = eu1 v1 + fu1 + gv1 + h
..
.
y4 = eu4 v4 + fu4 + gv4 + h

I this need for 4 point correspondences is the reason why

we consider mappings between arbitrary quadrilaterals!
Summary

we now know about

I several “artistic” image effects

I bilinear mappings and their properties