Part 06

Visual Information Processing: Computational
photography
Ji Hui
National University of Singapore
August 12, 2021
Ji Hui (National University of Singapore) Visual Information Processing: Computational photography August 12, 2021 1 / 26
Gradient domain editing: Image blending
Gradient domain editing: Image stitching
Plain stitching gradient domain stitching
Vector field and image gradient
Continues v.s. Discrete
Vector function : R2 ! R2 , each point associated w/. a vector
G(x, y) = [u(x, y), v(x, y)]>

∂I
Continues case → derivative → [0 1 − 1]∗ I
∂x
Discrete case → Finite differences ∂I
→ [0 1 − 1] ∗ I
T
∂y
Image gradient G = [u, v]> = [Ix , Iy ]> = [ @x
@I @I >
, @y ]
Image
I(x,y)
Ix Iy
Image gradient
Calculating image gradients rI:
@I
! [1, 1] ⌦ I
@x
@I
! [1, 1]> ⌦ I
@y
where ⌦ denotes convolution

Image gradient is a vector field, but not all vector fields are gradient field
of an image
Image gradient field [u, v] = [ @x , @y ] satisfies
@I @I >
@u @v
=0
@y @x
since
@2I @2I
= .
@x@y @y@x
Gradient domain editing
Why editing images in gradient domain, not directly copy-and-paste in
pixel domain Intuition – hole filling
Intuition: Continuity in brightness for brightness consistency when filling
hole
• 1D:
x1 x2
• 2D:
Gradient domain
Seamlessly pasteediting onto
Why editing images in gradient domain, not directly copy-and-paste in

- Add a linear function so that the boundary condition is respected
pixel domain
- Gradient error is equally distributed all over Ω in order to respect the
Intuition: filling
boundary details via blending gradients
condition
2D case
Membrane Interpolation for filling hole
⌦: a close subset of S with

boundary @⌦
f ⇤ : known function over S\⌦
f : unknown function over ⌦
Intuition – hole filling

Membrane interpolation
ZZ
min krf k22 ,
f ⌦
subject to Dirichlet boundary

x1 x2
conditions: f |@⌦ = f ⇤ |@⌦
Calculus of variation
when we minimize g(x) over R, we seek g 0 (x) = 0, and when we
minimize the objective over function, Variational method come to help.
Consider an integral of the form
Z
J(f ) = F (x, f, fx )dx,
⌦
The stationary point (a function) then satisfies the so-called

Euler-Lagrange equation:
@F d @F
=0
@f dx @fx
The proof is based on Calculus of Variation. Consider the function
(✏) = J[f + ✏⌘], 8⌘
The minimium implies that (0) = 0. which gives 0
Z
@F d @F
⌘(x)( )dx = 0, 8⌘.
@f dx @fx
Euler-Lagrange equation for Membrane interpolation
In our case:
F = krf k22 = (fx2 + fy2 )
Then, we have
@F d @F d @F
=0
@f dx @fx dy @fy
The minimizer has zero Laplacian
@F @fx2 + fy2
= =0
@f @f
d @F d @(fx2 + fy2 ) d @2f
= = 2fx = 2 2
dx @fx dx @fx dx @x
2 2
d @F d @(f x + f y ) d @2f
= = 2fy = 2 2
dy @fx dy @fy dy @y
Then, we have
@2f @2f
f= + = 0.
@x2 @y 2
Smooth hole filling
Minimizing ZZ
min krf k22 , f |@⌦ = f ⇤ |@⌦
f ⌦
The minimizer satifies
f = 0, over ⌦, subject to f |@⌦ = f ⇤ |@⌦
Discretization: forward difference

@f @f
⇡ fx+1,y fx,y ⇡ fx,y+1 fx,y
@x @y
or backward difference:
@f @f
⇡ fx,y fx 1,y ⇡ fx,y fx,y 1
@x @y
In addition, we have
f (x, y) = fx+1,y + fx 1,y + fx,y+1 + fx,y 1 4fx,y = 0.
Image editing in gradient domain
Consider a vector field G = (u, v) (pasted gradient from source) in a

bounded region ⌦:
@g @g
u= , v= .
@x @y
Find the values of f in ⌦ that minimize
ZZ
min krf Gk2
f ⌦
subject to f |@⌦ = f ⇤ |@⌦ .

Possion equation
In our case, F = krf Gk22

The minimizer satisfies
@2f @2f @u @v
f divG = 0 $ + =0
@x2 @y 2 @x @y
The minimizer is defined by
f = divG, subject to f |@⌦ = f ⇤ |@⌦ .
The discretization
@f @f
⇡ fx+1,y fx,y ⇡ fx,y+1 fx,y
@x @y
and
divG = ux,y ux 1,y + vx,y vx,y 1.
Poisson editing
Solving the Poisson equation: solving a linear system
Ax = b.
which can be done by minimizing min kAx bk22 , solved by greatest

descent method
Another issue for image blending: blending boundary
Seam Carving for minimizing boundary error
Intuition
Human vision more sensitive to edges – so try remove content from
smoother areas
Basic Idea: remove unimportant pixels from the image
For example, pixels with less magnitude
@f @f
E(rf (x, y)) = | |+| |
@x @y
Finding seam: s⇤ = mins E(s)
Poisson editing for image relighting
Consider a image g. let d denote desired brightness of pixel value at a
region ⌦, and rg desired image gradients different from rf .
Possion editing: Minimizing
ZZ
min krf rgk22 + µkf dk⌦
f
input relighting
HDR imaging
Dynamic range
Dynamic range
The ratio between the maximum (white) and minimum (black) measuable
light intensity
The real world is high dynamic range
about 5-10 million values to store all brightnesses in real world
typical camera provides limited dynamic range
typical 8-bit provide only 256 values
Dynamic range and exposure
Over-exposure
over-exposure
Under-exposure
Under exposure
Image photometric calibration
Photometric: How pixel values relate to radiance amounts in the world
Imaging system response function
g: log E ! Z,
where E denote received radiance energy from CCD sensor during

exposure time, and Z denote pixel value recorded in image
HDR imaging
Recovering full dynamic range of scene is all about g 1
.
Filling up the curve in g, via multiple exposures.
Finding the inverse of g which exists as g is monotonic and smooth.
Remark
Camera response function g is non-linear and unknown, its unique inverse
exists as g is monotonic and smooth for all cameras.
HDR imaging via multiple exposure
Capture many images with different exposure settings.
Shutter speed, F/stop, ISO
Apply photometric calibration to each camera.
Combine the calibrated images, e.g., using averaging weighted by
exposures
Mathematical formulation
Response function
Z = g(log E), E =R· t ! log E = log R + log( t),
where E denotes iradiance exposure and R denotes radiance.
Inverse response function
1
g (z) = log R + log( t)
Consider n images with different exposure time, we have
1
log Ri + log( tj ) = g (Zi,j ),
for i-th pixel in j-th image.
Solving the model with smooth contraint to estimate g 1
X zX
max
00
min k log Ri + log( tj ) g 1 (Zi,j )k22 + k(g 1
) (z)k22
i,j zmin
Composite radiance map using a weighted averaging scheme

P 1
j !(Zi,j )(g (Zi,j ) log tj
log Ri = P ,
! (Zi,j )
where ! emphasizes higher pixel values and larger gradients.
Display a HDR image using tone mapping
Tone mapping: mapping an HDR image to a regular image

Part 06

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Part 06

Uploaded by

Copyright:

Available Formats

Visual Information Processing: Computational

National University of Singapore

August 12, 2021

Plain stitching gradient domain stitching

G(x, y) = [u(x, y), v(x, y)]>

where ⌦ denotes convolution

Why editing images in gradient domain, not directly copy-and-paste in

⌦: a close subset of S with

Intuition – hole filling

subject to Dirichlet boundary

The stationary point (a function) then satisfies the so-called

The proof is based on Calculus of Variation. Consider the function

(✏) = J[f + ✏⌘], 8⌘

The minimium implies that (0) = 0. which gives 0

f = 0, over ⌦, subject to f |@⌦ = f ⇤ |@⌦

Discretization: forward difference

f (x, y) = fx+1,y + fx 1,y + fx,y+1 + fx,y 1 4fx,y = 0.

Consider a vector field G = (u, v) (pasted gradient from source) in a

subject to f |@⌦ = f ⇤ |@⌦ .

In our case, F = krf Gk22

The minimizer is defined by

f = divG, subject to f |@⌦ = f ⇤ |@⌦ .

Solving the Poisson equation: solving a linear system

which can be done by minimizing min kAx bk22 , solved by greatest

where E denote received radiance energy from CCD sensor during

Composite radiance map using a weighted averaging scheme

Tone mapping: mapping an HDR image to a regular image

You might also like