Course : Applied Projective Geometry

Effective Period : September 2015

Estimation – 2D Projective
Transformations

Session 06-07
Estimation – 2D Projective
Transformations
 Acknowledgement

These slides have been adapted from:

Richard Hartley, Andrew Zisserman (2004).

Multiple View Geometry in Computer Vision.
Cambridge University Press. ISBN: 0-521-54051-
8.

Chapter 4
 Learning Objectives

LO 1 : Explain basics of projective geometry

LO 2 : Demonstrate the algorithms of projective

geometry
 Contents

Estimation – 2D Projective Transformations

• The Direct Linear Transformation (DLT) Algorithm

• Different Cost Functions
• Transformation Invariance and Normalization
 Parameter estimation

• 2D homography
Given a set of (xi,xi’), compute H (xi’=Hxi)
• 3D to 2D camera projection
Given a set of (Xi,xi), compute P (xi=PXi)
• Fundamental matrix
Given a set of (xi,xi’), compute F (xi’TFxi=0)
• Trifocal tensor
Given a set of (xi,xi’,xi”), compute T
 Number of measurements
required
• At least as many independent
equations as degrees of freedom
required
 x   h11 h12 h13   x 
• Example:     
λ  y  h21 h22 h23   y 
x'  Hx
 1   h31 h32 h33   1 
2 independent equations / point
8 degrees of freedom

4x2≥8
 Approximate solutions

• Minimal solution
4 points yield an exact solution for H
• More points
– No exact solution, because measurements are inexact
(“noise”)
– Search for “best” according to some cost function
– Algebraic or geometric/statistical cost
 Gold Standard algorithm
• Cost function that is optimal for some assumptions
• Computational algorithm that minimizes it is called
“Gold Standard” algorithm
• Other algorithms can then be compared to it
 Direct Linear Transformation
(DLT)
 h 1T x 
x i  Hx i  T i
xi   xi, yi, wi  Hx i   h 2 x i 
T

xi  Hx i  0  3T 
 yh 3 T x  wh 2 T x   h x i 
 i T i i i
3T
xi  Hx i  wih x i  xih x i 
 1

 2T 1T 
 xih x i  yih x i 
 
 0T 
 wi x i
T

yi x iT
  h 
1

 T
 2
 T
 wi x i 0 T
 xix i  h   0
 yix iT  T T   3
 xi x i 0  h 
Ai h  0
 Direct Linear Transformation
(DLT)
• Equations are linear in h
Ai h  0
• Only 2 out of 3 are linearly independent
(indeed, 2 eq/pt)
 0T  wix iT yix iT  h1   h 1

 T
 2  0 T
 wix i T
yix i  2 
T

 wix i  xix i  h   0  h   0
T T
0 T
 yix iT  T T   3  wix i
T
0 T
 xix i  3 
 xi x i 0  h  h 

(only drop third row if wi’≠0) x A
i i
1
 y A
i i
2
 w 
i i 0
A 3

• Holds for any homogeneous

representation, e.g. (xi’,yi’,1)
 Direct Linear Transformation
(DLT)
• Solving for H  A1 
A 
Ah  0  2 h  0
A3 
 
size A is 8x9 or 12x9, but rank 8  A 4 

Trivial solution is h=09T is not interesting

1-D null-space yields solution of interest
pick for example the one with h  1
 Direct Linear Transformation
(DLT)  A1 
• Over-determined solution A 
 2 h  0
 
Ah  0  
A n 
No exact solution because of inexact measurement
i.e. “noise”

Find approximate solution

- Additional constraint needed to avoid 0, e.g. h 1
- Ah  0 not possible, so minimize Ah
 DLT algorithm

Objective
Given n≥4 2D to 2D point correspondences {xi↔xi’}, determine
the 2D homography matrix H such that xi’=Hxi
Algorithm
(i) For each correspondence xi ↔xi’ compute Ai. Usually only two first
rows needed.
(ii) Assemble n 2x9 matrices Ai into a single 2nx9 matrix A
(iii) Obtain SVD of A. Solution for h is last column of V
(iv) Determine H from h
 Inhomogeneous solution
Since h can only be computed up to scale, ~
pick hj=1, e.g. h9=1, and solve for 8-vector h

 0 0 0  xi wi '  yi wi '  wi wi ' xi yi ' yi yi ' ~   wi yi ' 

x w ' h   
 i i yi wi ' wi wi ' 0 0 0 xi xi ' yi xi ' w x '
 i i 

Solve using Gaussian elimination (4 points) or using linear

least-squares (more than 4 points)
However, if h9=0 this approach fails
also poor results if h9 close to zero
Therefore, not recommended
Note h9=H33=0 if origin is mapped to infinity 0 
l T Hx 0   0 0 1 H 0  0
1
 Degenerate configurations
x1 x1
x4 x4 x1
x2 H? H’? x4
x2
x2
x3 x3
(case A) (case B) x3

Constraints: x i  Hx i  0 i=1,2,3,4

Define: H *  x 4 l T
Then,  
H* x i  x4 l T x i  0, i  1,2,3
H*x 4  x   l x   kx 
4
T
4 4

H* is rank-1 matrix and thus not a homography

If H is unique solution, then no homography mapping xi→xi’(case B)

*

If further solution H exist, then also αH +βH (case A) *

(2-D null-space in stead of 1-D null-space)

 Solutions from lines, etc.
2D homographies from 2D lines

li  H T li Ah  0
Minimum of 4 lines

3D Homographies (15 dof)

Minimum of 5 points or 5 planes

2D affinities (6 dof)

Minimum of 3 points or lines

Conic provides 5 constraints

Mixed configurations?
 Cost functions
• Algebraic distance
• Geometric distance
• Reprojection error

• Comparison
• Geometric interpretation
• Sampson error
 Algebraic distance
DLT minimizes Ah
e  Ah residual vector
ei partial vector for each (xi↔xi’)
algebraic error vector
2
 0  wix T
 yix  T T
d alg  x i , Hx i   ei  
2 2 i i
h
 wix i  xix 
T T T
0 i

algebraic distance
d alg  x1 , x 2   a12  a22 where a   a1 , a2 , a3  T  x1  x 2
2

 d alg  xi , Hxi    ei

2 2 2 2
 Ah  e
i i
Not geometrically/statistically meaningfull, but given good normalization it works fine and is very fast (use
for initialization)
 Geometric distance
x measured coordinates

x̂ estimated coordinates

x true coordinates

d(.,.) Euclidean distance (in image)

Error in one image
Ĥ  argmin  d x i , Hx i   2 e.g. calibration pattern

H i

Symmetric transfer error


Ĥ  argmin  d x i , H xi  d  x i , Hx i 
-1
 2 2

H i

Reprojection error
Ĥ, x̂ , x̂   argmin  d  x , x̂ 
i i
H, x̂ i , x̂ i
i i
2
  
 d x i , x̂ i  2

i
subject to x̂ i  Ĥx̂ i
Reprojection error

 -1

d x, H x  d  x , Hx 
2 2

d  x, x̂   d  x , x̂ 
2 2
 Comparison of geometric
and algebraic distances
Error in one image
xi   xi, yi, wi  x̂i   xˆi, yˆ i, wˆ i   Hx
T T

 h 1

 yiwˆ i  wi yˆ i   0T  wix iT yix i  2 
T 

A i h  ei     T T
h 
 wixˆi  xiwˆ i   wix i 0T  xix i  3 
h 
d alg  x i , x̂i    yiwˆ i  wi yˆ i    wixˆi  xiwˆ i 
2 2 2

2

d  x i , x̂ i    yi / wi  yˆ i / wˆ i    xˆi / wˆ i  xi / wi 
2

2 1/ 2

 d alg  x i , x̂ i  / wiwˆ i
wi  1 wˆ i  h 3 x,iexcept for affinities
typical, but not

For affinities DLT can minimize geometric distance

Possibility for iterative algorithm

 Geometric interpretation of
reprojection error
Estimating homography~fit surface ν
to H
points X=(x,y,x’,y’) T
in 4

xi  Hx irepresents
 0 2 quadrics in  4
(quadratic in X)
  xi  xˆi    yi  yˆ i    xi  xˆi    yi  yˆ i 
2 2 2 2 2
X i  X̂ i

 d  x i , x̂ i   d  xi , x̂i 
2 2

d  x i , x̂ i   d  x i , x̂ i   d   X i , H 
2 2 2

Analog to conic fitting

d alg  x , C   x T Cx
2

d   x, C
2
 Sampson error

2
X̂
Vector that minimizes the geometric error
measurement
X  X̂
is the closest point on the variety to the

Sampson error: 1st order approximation of

νH
between algebraic and geometric error
X
X̂
Ah  CH  X   0

CH  X  δ X   CH  X  
CH
X
δX δ X  X̂  X  
CH X̂  0
CH CH
CH  X   δX  0 Jδ X  e with J 
X X
Find the vector that minimizes
δX subject to
δX Jδ X  e
Find the vector that minimizes
δX subject to
δX Jδ X  e

Use Lagrange multipliers:

minimize

δ X δ X - 2λ Jδ X  e   0
T
derivatives

2δ X - 2λ T J  0 T  δX  J T λ
2 Jδ X  e   0  JJ T λ  e  0
 λ   JJ T  e
1

 δ X   J  JJ T

T 1
e

X̂  X  δ X δX
2
 
 δ TX δ X  e T JJ T
1
e
 Sampson error
between algebraic and geometric error

2
X̂
Vector that minimizes the geometric error
measurement
X  X̂
is the closest point on the variety to the

Sampson error: 1st order approximation of νH X̂ X

Ah  CH  X   0

CH  X  δ X   CH  X  
CH
X
δX δ X  X̂  X CH X̂  0 
CH
CH  X   δX  0 Jδ X  e
X
Find the vector that minimizes
δX subject to
δX Jδ X  e

δX
2
 δ δ  e JJ
T
X X
T
 
T 1
e (Sampson error)
 Sampson approximation

A few points
δX
2
 e JJ
T
 
T 1
e
(i) For a 2D homography X=(x,y,x’,y’)
(ii) e  C  X is the algebraic error vector
H
(iii) CH is a 2x4 matrix, e.g.
J
X J   
(iv) Similar to algebraic error
11 w
i 
x i h 2  yix i h 3 / x   wih21  yih31
T T
in fact,
2
same as Mahalanobis distancee  eTe
(v) Sampson error independent of linear e 2 T
JJ
reparametrization (cancels out in between e and J)
(vi) Must be summed for all points

  
(vii) Close to geometric error, but muchefewer T 1
T unknowns
JJ e
 Statistical cost function and
Maximum Likelihood Estimation
• Optimal cost function related to noise model
• Assume zero-mean isotropic Gaussian noise (assume
outliers removed)
1  d  x, x  2 /  2 2 
Pr  x   2
e
2 πσ
Error in one image
1  xi ,Hx i  2 /  2 2 
Pr   xi  | H    2
e d
i 2 πσ
1
log Pr   x i  | H     d  x i , Hx i   constant
 2
2
2σ
Maximum Likelihood Estimate

 d x i , Hx i
  2
 Statistical cost function and
Maximum Likelihood Estimation
• Optimal cost function related to noise model
• Assume zero-mean isotropic Gaussian noise (assume
outliers removed)

Pr  x  
1
e 
 d  x, x  2 / 2 2 
2
2 πσ
Error in both images
1  d  x i , x i  2  d  xi ,Hx i  2  /  2 2 
Pr   x i  | H    2
e 
i 2 πσ

Maximum Likelihood Estimate

 d  x i , x̂ i   d  xi , x̂i 
2 2
 Mahalanobis distance
• General Gaussian case
Measurement 2X with covariance matrix Σ
  X  X   1  X  X 
T
XX 

Error in two images (independent)

2 2
X  X   X   X  

Varying covariances
X
2 2
i  Xi i
 Xi  Xi i
i
 Invariance to transforms ?

~
x  Tx ~~
x   Hx ~
x   Hx
~
x   Tx  ~
Tx   HTx
-1 ~
? ~ x  T HTx
H  T HT
-1

will result change?

for which algorithms? for which transformations?
 Non-invariance of DLT

Given x i  xi and H computed by DLT,

and ~
x i  Tx i , ~
x i  Tx i

Does the DLT algorithm applied to ~

xi  ~
xi
~
yield H  THT -1?
Effect of change of coordinates on algebraic error
~ ~ ~~
 
ei  x i  Hx i  Txi  THT-1 Tx i  T*  xi  Hx i   T*ei

for similarities
sR t   R 0
T    T  s  T
*

so  0 1   - t R s 

~~
A i h   e~1 , e~2   sR  e1 , e2   s A i h
T T

d alg  xi , Hxi   sd alg  ~ ~~

x i , Hx i 
 Non-invariance of DLT

Given x i  xi and H computed by DLT,

and ~
x i  Tx i , ~
x i  Tx i

Does the DLT algorithm applied to ~

xi  ~
xi
~
yield H  THT -1?

minimize  d alg  x i , Hx i  subject to H  1

 2

 
i
~ ~~ 2
 minimize  d alg xi , Hx i subject to H  1

 
i
~ ~~ 2 ~
 minimize  d alg xi , Hx i subject to H  1
i
 Invariance of geometric error

Given x i  xi and H,

~ ~ ~ ~ ~
and x i  x i , x i  Tx i , x i  T x i , H  THT -1
  
Assume T’ is a similarity transformations
d~ ~
x i , H~   
x i  d Tx i , THT-1Tx i  d  Tx i , THx i 
 sd  x i , Hx i 
 Normalizing transformations
• Since DLT is not invariant,
what is a good choice of coordinates?
e.g.
– Translate centroid to origin
– Scale to a 2 average distance to the origin
– Independently on both images
1
w  h
Or
0 w / 2
Tnorm 
 0 
w  h h / 2
 0 0 1 
 Importance of normalization
 h1 
0 0 0  xi  yi  1 yixi yi yi yi  2 
x  h   0
 i yi 1 0 0 0  xixi  xi yi  xi  3 
h 
~102 ~102 1 ~102 ~102 1 ~104 ~104 ~102

orders of magnitude difference!

 Normalized DLT algorithm
Objective
Given n≥4 2D to 2D point correspondences {xi↔xi’}, determine
the 2D homography matrix H such that xi’=Hxi
Algorithm
(i) Normalize points ~
xi  Tnorm x i , ~
x i  Tnorm
 x i
(ii) Apply DLT algorithm to ~
xi  ~ x i ,
(iii) Denormalize solution ~
-1 HTnorm
H  Tnorm
 Iterative minimization methods
Required to minimize geometric error
(i) Often slower than DLT
(ii) Require initialization
(iii) No guaranteed convergence, local minima
(iv) Stopping criterion required
Therefore, careful implementation required:
(v) Cost function
(vi) Parameterization (minimal or not)
(vii) Cost function ( parameters )
(viii) Initialization
(ix) Iterations
 Parameterization

Parameters should cover complete space

and allow efficient estimation of cost

– Minimal or over-parameterized? e.g. 8 or 9

(minimal often more complex, also cost surface)
(good algorithms can deal with over-
parameterization)
(sometimes also local parameterization)
– Parametrization can also be used to restrict
transformation to particular class, e.g. affine
 Function specifications

(i) Measurement vector XN with covariance Σ

(ii) Set of parameters represented by vector P N
(iii) Mapping f : M →N. Range of mapping is surface S
representing allowable measurements
(iv) Cost function: squared Mahalanobis distance

X  f  P   X  f  P    1  X  f  P  
2 T

Goal is to achieve  , or get as close as

possible in terms of Mahalanobis distance
f  P  X
Error in one image

d 
 i ix  , Hx  2

f : h   Hx1 , Hx 2 ,..., Hx n 
X  f  h
Symmetric transfer error
 -1

 d x i , H xi  d  xi , Hxi 
2 2

f : h   H -1x1 , H -1x 2 ,..., H -1xn , Hx1 , Hx 2 ,..., Hx n 

X  f  h
Reprojection error
 i i
d  x , x̂  2
 d  x 
i , x̂ 
i  2

f :  h, x̂1 , x̂ 2 ,..., x̂ n    x̂1 , x̂ 2 ,..., x̂ n 

X  f  h
 Initialization

• Typically, use linear solution

• If outliers, use robust algorithm

• Alternative, sample parameter space

 Iteration methods

Many algorithms exist

• Newton’s method
• Levenberg-Marquardt

• Powell’s method
• Simplex method
 Gold Standard algorithm

Objective
Given n≥4 2D to 2D point correspondences {xi↔xi’}, determine
the Maximum Likelyhood Estimation of H
(this also implies computing optimal xi’=Hxi)
Algorithm
(i) Initialization: compute an initial estimate using normalized DLT or
RANSAC
(ii) Geometric minimization of -Either Sampson error:
● Minimize the Sampson error
● Minimize using Levenberg-Marquardt over 9 entries of h
or Gold Standard error:
● compute initial estimate for optimal {xi}
● minimize cost over {H,x 1,x2,…,xn}
● if many points, use d  x i ,method
x̂ i   d  x i , x̂i 
2 2
sparse
 Robust estimation

• What if set of matches contains gross outliers?

 RANSAC

Objective
Robust fit of model to data set S which contains outliers
Algorithm
(i) Randomly select a sample of s data points from S and instantiate the
model from this subset.
(ii) Determine the set of data points Si which are within a distance
threshold t of the model. The set Si is the consensus set of samples
and defines the inliers of S.
(iii) If the subset of Si is greater than some threshold T, re-estimate the
model using all the points in Si and terminate
(iv) If the size of Si is less than T, select a new subset and repeat the
above.
(v) After N trials the largest consensus set Si is selected, and the model is
re-estimated using all the points in the subset Si
 Distance threshold

Choose t so probability for inlier is α (e.g. 0.95)2

• Often empirically d
 2
• Zero-mean
m Gaussian noise σ then follows
distribution with m=codimension of model
(dimension+codimension=dimension space)

Codimension Model t2
1 l,F 3.84σ2
2 H,P 5.99σ2
3 T 7.81σ2
 How many samples?

Choose N so that, with probability p, at least one

random sample is free from outliers. e.g. p=0.99

1  1  e 
s N
 1 p

N  log 1  p  / log 1  1  e 
s

proportion of outliers e
s 5% 10% 20% 25% 30% 40% 50%
2 2 3 5 6 7 11 17
3 3 4 7 9 11 19 35
4 3 5 9 13 17 34 72
5 4 6 12 17 26 57 146
6 4 7 16 24 37 97 293
7 4 8 20 33 54 163 588
8 5 9 26 44 78 272 1177
 Acceptable consensus set?
• Typically, terminate when inlier ratio reaches expected
 
T  1 e n
ratio of inliers
 Adaptively determining
the number of samples

e is often unknown a priori, so pick worst case, e.g. 50%, and

adapt if more inliers are found, e.g. 80% would yield e=0.2

– N=∞, sample_count =0
– While N >sample_count repeat
• Choose a sample and count the number of inliers
• Set e=1-(number of inliers)/(total number of points)
• Recompute N from e
 N  log1  p  / log 1  1  e s 
• Increment the sample_count by 1
– Terminate
 Robust Maximum Likelyhood
Estimation
Previous MLE algorithm considers fixed set of inliers

Better, robust cost function (reclassifies)

e 2 e 2  t 2 inlier
R   ρ d i  with ρ e    2 2 2
i t e  t outlier
 Other robust algorithms
• RANSAC maximizes number of inliers
• LMedS minimizes median error

• Not recommended: case deletion, iterative least-

squares, etc.
Automatic computation of H
Objective
Compute homography between two images
Algorithm
(i) Interest points: Compute interest points in each image
(ii) Putative correspondences: Compute a set of interest point matches
based on some similarity measure
(iii) RANSAC robust estimation: Repeat for N samples
(a) Select 4 correspondences and compute H
(b) Calculate the distance d for each putative match
(c) Compute the number of inliers consistent with H (d<t)
Choose H with most inliers
(iv) Optimal estimation: re-estimate H from all inliers by minimizing ML
cost function with Levenberg-Marquardt
(v) Guided matching: Determine more matches using prediction by
computed H
Optionally iterate last two steps until convergence
 Determine putative
correspondences
• Compare interest points
Similarity measure:
– SAD, SSD, ZNCC on small neighborhood

• If motion is limited, only consider interest points with

similar coordinates

• More advanced approaches exist, based on

invariance…
Example: robust
computation
Interest points
(500/image)

Putative correspondences (268)

Outliers (117)

Inliers (151)

Final inliers (262)

 References

Richard Hartley, Andrew Zisserman (2004). Multiple

View Geometry in Computer Vision. Cambridge
University Press. ISBN: 0-521-54051-8.