PROCESAMIENTO DIGITAL DE

IMAGENES Y VISION ARTIFICIAL

Semana 15 – sesión 1
Imaging Stitching
Objetivos de la sesión

• Conectar dos imagenes

Image Stitching

1. Align the images over each other

camera pan ↔ translation on cylinder
2. Blend the images together

Image Stitching
Image stitching

1. Take pictures on a tripod (or handheld)

2. Warp images to spherical coordinates
3. Extract features
4. Align neighboring pairs using RANSAC
5. Write out list of neighboring translations
6. Correct for drift
7. Read in warped images and blend them
8. Crop the result and import into a viewer
Matching features

What do we do about the “bad” matches?

RAndom SAmple Consensus

Select one match, count inliers

RAndom SAmple Consensus

Select one match, count inliers

Least squares fit

Find “average” translation vector

Assembling the panorama

Stitch pairs together, blend, then crop

Problem: Drift

Error accumulation
small (vertical) errors accumulate over time
apply correction so that sum = 0 (for 360° pan.)
 Problem: Drift
(x1,y1)

(xn,yn)

Solution copy of first

add another copy of first image at the end image

this gives a constraint: yn = y1

there are a bunch of ways to solve this problem
add displacement of (y1 – yn)/(n -1) to each image after the first
compute a global warp: y’ = y + ax
run a big optimization problem, incorporating this constraint
best solution, but more complicated
known as “bundle adjustment”
Full-view (360° spherical) panoramas
Full-view Panorama

+ +
+ +
Texture Mapped Model
Global alignment

•Pairwise matching → global energy function that involves

all of the per-image pose parameters
•Local adjustments to reduce blurring due to local mis-
registration

•Panorama recogition
Bundle Adjustment

Developed in photogrammetry in 50´s , for general

structure from motion (Szeliski and Kang 1994)
Pairwise alignment
Bundle Adjustment
N features, appears in M images
Bundle Adjustment

Given initial set of {Rj, Kj}, refine them together

Predicted location of feature I Observed location of feature I

in image k in image k

Xik depends on xij, error-in-variable

Overweighted for feature observed many times
Bundle Adjustment

True bundle adjustment to estimate camera pose and 3d

points

Disadvantages
A lot of variables to solve
Slow convergence
Bundle Adjustment

Sparse matrix techniques

Levenberg-Marquardt

http://www.ics.forth.gr/~lourakis/sba/
Non-linear least squares
Linear approximation of residual
allows quadratic approximation of sum-
e0 − J
of-squares

(e0 − J) (e0 − J)

T

2J T J − 2J T e0 = 0
Minimization corresponds to finding zeros of
derivative

( )
  = J J J e0
T -1 T

N
Levenberg-Marquardt: extra term to deal with singular N
(decrease/increase l if success/failure to descent)

(extra term = descent term)

N' = J T J + λdiag(J T J)
 Bundle Adjustment
 D(mij , P̂i (M̂ j ))
m n
Jacobian of has sparse
2
block structure
i =1 j =1

cameras independent of other cameras,

points independent of other points

P1 P2 P3 M U1
im.pts.
view 1
U2 W
J= N = JT J =
U3

WT V
12xm 3xn
(in general Needed for non-linear minimization
much larger)
Bundle Adjustment
Eliminate dependence of camera/motion parameters on
structure parameters
Note in general 3n >> 11m

U-WV-1WT
 I − WV   N =
−1

0 I 

Allows much more efficient

computations WT V
e.g. 100 views,10000 points,
solve 1000x1000, not 30000x30000 11xm 3xn

Often still band diagonal

use sparse linear algebra algorithms
Up Vector Selection

Keep the upright direction at the final stitched

image

Multiply each roation by a global rotation Rg

Up Vector Selection

rg1 is the smallest eigenvector of the scatter or moment

matrix spanned by the individual camera rotation
x-vectors
Full rotation matrix Rg
Parallax Removal

Blurry or ghosting
Unmodeled radial distortion
3D parallax: failure to rotate the camera around its optical center
Small scene motion, large scale scene motion

Different approaches for them

Parallax Removal

Based on bundle adjustment, compute 3D point location,

then reproject it to images

Deghosting a mosaic with motion parallax (Shum and Szeliski 2000) c 2000 IEEE: (a) composite with
parallax; (b) after a single deghosting step (patch size 32); (c) after multiple steps (sizes 32, 16 and 8)
Cierre