Geometric Estimation with

Local Affine Frames and Free-form

Surfaces
Dissertation
zur Erlangung des akademischen Grades
Doktor der Ingenieurwissenschaften
(Dr.-Ing.)
der Technischen Fakultt
der Christian-Albrechts-Universitt zu Kiel

Kevin Kser

Kiel
2008

1. Gutachter

Prof. Dr.-Ing. Reinhard Koch

2. Gutachter

Prof. Dr.-Ing. Joachim Denzler

3. Gutachter

Prof. Marc Pollefeys

Datum der mndlichen Prfung

20.03.2009

Berichte aus der Informatik

Kevin Kser

Geometric Estimation with Local Affine Frames

and Free-form Surfaces

Shaker Verlag
Aachen 2009

Bibliographic information published by the Deutsche Nationalbibliothek

The Deutsche Nationalbibliothek lists this publication in the Deutsche
Nationalbibliografie; detailed bibliographic data are available in the Internet
at http://dnb.d-nb.de.
Zugl.: Kiel, Univ., Diss., 2009

Copyright Shaker Verlag 2009

All rights reserved. No part of this publication may be reproduced, stored in a
retrieval system, or transmitted, in any form or by any means, electronic,
mechanical, photocopying, recording or otherwise, without the prior permission
of the publishers.
Printed in Germany.

ISBN 978-3-8322-8250-9
ISSN 0945-0807
Shaker Verlag GmbH P.O. BOX 101818 D-52018 Aachen
Phone: 0049/2407/9596-0 Telefax: 0049/2407/9596-9
Internet: www.shaker.de e-mail: info@shaker.de

Inspiration is needed in geometry, just as much as

in poetry.

Alexander Sergeyevich Pushkin

Acknowledgements
This thesis would not have been possible without the support of many people. First of
all I would like to thank Reinhard Koch for giving me the opportunity to work in the
eld of computer vision and advising me during the last years. I am grateful to him and
the current and past members of his group for the outstanding working athmosphere and
team spirit that made it a pleasure to work. I would also like to thank Marc Pollefeys and
Joachim Denzler for being in my thesis committee.
Furthermore, I am grateful to my parents Carmen and Rolf Kser for always supporting
me on my way, for my education, and for making academic studies possible. The largest
thanks are owed to my wife Andrea who constantly encouraged me and supported me
during the last years and tolerated the evenings and weekends that I spent on working for
this thesis.
Also, I have to thank my former colleagues Jan-Michael Frahm for many insights into
3D reconstruction when I became a Ph.D. student, Oliver Granert, who introduced me
into the eld of invariant image features, and Christian Beder for very fruitful discussions
and cooperation in the nal year of this thesis. I would also like to thank my colleagues
Anne Sedlazeck, Arne Petersen, Bogumil Bartczak, Sandro Esquivel, Ingo Schiller, Kristine
Haase, Daniel Jung, Anatol Frick and Falko Kellner for proof-reading the thesis, good
software collaboration and many discussions and other activities during the last years.
The last points are also true for Felik Woelk and Jan-Friso Evers-Senne, whom I thank
not only for the constant eorts to improve my physical tness after work but also for
the exciting cooporation on the Tracking-Framework. Last but not least I want to thank
Renate Staecker and Torge Storm for their constant organisational, administrative and
technical support.

II

Contents
1 Introduction
1.1

Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2 Basic Concepts
2.1

Camera Model and Geometry . . . . . . . . . . . . . . . . . . . . . . . . .

2.1.1

Projective Geometry and Notation . . . . . . . . . . . . . . . . . .

2.1.2

Geometric Objects . . . . . . . . . . . . . . . . . . . . . . . . . . .

2.1.3

Rotation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2.1.4

9
12

Photometric Image Creation . . . . . . . . . . . . . . . . . . . . . . . . . .

13

Plenoptic Function . . . . . . . . . . . . . . . . . . . . . . . . . . .

13

2.2.2

Piecewise Continuous Surfaces . . . . . . . . . . . . . . . . . . . . .

13

2.2.3

Camera Hardware, CCD . . . . . . . . . . . . . . . . . . . . . . . .

14

2.2.4

Lens Eects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

14

2.2.5
2.3

. . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Pose . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2.2.1

2.2

Camera Model

2.1.5

Brightness Models

15

. . . . . . . . . . . . . . . . . . . . . . . . . . .

Relations between Local Regions in a Scene . . . . . . . . . . . . . . . . .

16

2.3.1

Displacement, Euclidean, Similarity, Ane Transform . . . . . . . .

17

2.3.2

General Homography . . . . . . . . . . . . . . . . . . . . . . . . . .

19

2.3.3

Perspectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

21

2.3.4

Homography induced by a Scene Plane . . . . . . . . . . . . . . . .

22

2.3.5

Innite Homography and Conjugate Rotation . . . . . . . . . . . .

23

3 Primitives in the Literature

25

3.1

Points . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

26

3.2

Lines and Planes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

27

III

CONTENTS

3.3
3.4
3.5
3.6
3.7

Conics, Quadrics and Convex Hull Regions

Intensities . . . . . . . . . . . . . . . . . .
Local Curves and Lines . . . . . . . . . . .
Local Regions . . . . . . . . . . . . . . . .
Summary and Relation to this Thesis . . .

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

4.1 Robust Local Image Features . . . . . . . . . . . . . . . . . .

4.1.1 Regions of Interest . . . . . . . . . . . . . . . . . . . .
4.1.2 Local Ane Frame . . . . . . . . . . . . . . . . . . . .
4.1.3 Descriptors . . . . . . . . . . . . . . . . . . . . . . . .
4.1.4 Matching Strategies . . . . . . . . . . . . . . . . . . . .
4.2 The LAF Correspondence Constraint . . . . . . . . . . . . . .
4.2.1 Concatenation of Local Ane Frames . . . . . . . . . .
4.2.2 Warp Constraints . . . . . . . . . . . . . . . . . . . . .
4.2.3 Physically Motivated Interpretation . . . . . . . . . . .
4.2.4 Triangle Decomposition . . . . . . . . . . . . . . . . .
4.2.5 Renement and Upgrade from Simpler Features . . . .
4.3 Alignment in Scale Space . . . . . . . . . . . . . . . . . . . . .
4.3.1 Related Work on Gradient-based Alignment . . . . . .
4.3.2 Parametric Image Alignment with Uncertainty . . . . .
4.3.3 Evaluation of the Optimization . . . . . . . . . . . . .
4.3.4 Summary on Gradient-based Optimization . . . . . . .
4.3.5 Practical Optimization for LAF correspondences . . . .
4.4 LAF Uncertainty . . . . . . . . . . . . . . . . . . . . . . . . .
4.4.1 Obtaining and Representing Uncertainty . . . . . . . .
4.4.2 Empiric Covariance . . . . . . . . . . . . . . . . . . . .
4.4.3 Incidence and Outlier Detection . . . . . . . . . . . . .
4.4.4 Maximum-Likelihood Estimation . . . . . . . . . . . .
4.4.5 Evaluation: Measuring and Finite Area Approximation
4.5 Relation to other Primitives . . . . . . . . . . . . . . . . . . .
4.5.1 Triple of Points . . . . . . . . . . . . . . . . . . . . . .
4.5.2 Conic Correspondence . . . . . . . . . . . . . . . . . .
4.5.3 Curves . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

4 Dierential Constraints

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

28
29
29
30
33
35

36
37
37
41
42
42
42
44
45
46
49
49
51
53
58
62
64
64
65
67
68
70
71
75
75
75
78
79

CONTENTS
5 Applications: Geometric Estimation

5.1 General Homography . . . . . . . . . . . . . . . . . . . . . . . . .

5.1.1 Previous Work on Homography Estimation . . . . . . . . .
5.1.2 Obtaining a General Homography
from Two Feature Correspondences . . . . . . . . . . . . .
5.1.3 Generalizing to n Correspondences . . . . . . . . . . . . .
5.1.4 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.2 Conjugate Rotation . . . . . . . . . . . . . . . . . . . . . . . . . .
5.2.1 Previous Work . . . . . . . . . . . . . . . . . . . . . . . .
5.2.2 A Minimal Parameterization . . . . . . . . . . . . . . . . .
5.2.3 Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.2.4 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.2.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.3 Triangulation and Normal Estimation . . . . . . . . . . . . . . . .
5.3.1 Previous Work . . . . . . . . . . . . . . . . . . . . . . . .
5.3.2 Patchlet Estimation . . . . . . . . . . . . . . . . . . . . . .
5.3.3 Maximum Likelihood Estimation . . . . . . . . . . . . . .
5.3.4 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.3.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.4 Pose Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.4.1 Perspectivity . . . . . . . . . . . . . . . . . . . . . . . . .
5.4.2 Previous Work . . . . . . . . . . . . . . . . . . . . . . . .
5.4.3 Pose Estimation from a LAF correspondence . . . . . . .
5.4.4 Optimization, Tracking, Maximum Likelihood Estimation .
5.4.5 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.4.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6 Free-form Surface Models

6.1 Oine Modeling . . . . . . . . . . . . . . . . . . . .

6.1.1 Structure from Motion from Spherical Images
6.1.2 Bundle Adjustment . . . . . . . . . . . . . . .
6.1.3 Dense 3D Reconstruction . . . . . . . . . . . .
6.2 Initialization using a Descriptor Database . . . . . .
6.2.1 Previous Work on View Registration . . . . .

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

81

. . . . . 83
. . . . . 83
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

83
89
90
95
95
96
101
103
107
107
108
109
114
114
116
116
118
118
119
122
123
127
127
131

134
135
141
142
143
143

VI

CONTENTS

6.2.2 Scene Database . . . . . . .

6.3 Tracking Free-form Surface Models
6.3.1 Spherical Camera . . . . . .
6.3.2 Camera Tracking . . . . . .
6.3.3 System Evaluation . . . . .
6.3.4 Discussion . . . . . . . . . .

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

145
150
151
154
158
168

7 Conclusion

171

A Analysis

175

B Probability Theory

181

7.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171

7.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174

A.1 Taylor Series . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175

A.2 Homographies in P1: Rational Functions in R1 . . . . . . . . . . . . . . . 177
B.1 Basic Concepts . . . . . . . . . . . . . . . .
B.1.1 Cumulative Distribution and Density
B.1.2 Moments . . . . . . . . . . . . . . . .
B.1.3 Mahalanobis Distance . . . . . . . .
B.2 Distributions . . . . . . . . . . . . . . . . .
B.2.1 Normal Distribution . . . . . . . . .
B.2.2 2 distribution . . . . . . . . . . . .
B.3 Statistical Testing . . . . . . . . . . . . . . .
B.3.1 Incidence Test . . . . . . . . . . . . .
B.4 Uncertainty Propagation . . . . . . . . . . .
B.4.1 Linear Error Propagation . . . . . .
B.4.2 Monte Carlo Methods . . . . . . . .
B.4.3 Unscented Transform . . . . . . . . .

.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.

C.1 Robust Estimation . . . . . . . . . . . . . . . . .

C.1.1 Observations and Uncertainty . . . . . . .
C.1.2 Least Squares and Covariance Estimation
C.1.3 Covariance Estimation . . . . . . . . . . .
C.1.4 Newton-like methods . . . . . . . . . . . .

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

C Robust Estimation

.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.

181
181
181
183
184
184
185
186
186
187
187
187
188

189

189
189
190
190
191

CONTENTS

VII

C.1.5 Gross Errors and Breakdown Point . . . . . . . . . . . . . . . . . . 192

C.1.6 Robust Error Functions . . . . . . . . . . . . . . . . . . . . . . . . 193
C.1.7 RANSAC-like methods . . . . . . . . . . . . . . . . . . . . . . . . . 194

D Source Code

197

D.1 Conjugate Rotation Parameterization . . . . . . . . . . . . . . . . . . . . . 197

VIII

CONTENTS

Acronyms

List of Abbreviations
AC

Absolute Conic

CAD
CCD

Computer Aided Design

Charged Coupled Device

DIAC
DLT
DOF
DoG

Dual Image of the Absolute Conic

Direct Linear Transformation
Degrees of Freedom
Dierence of Gaussians

EBR
EKF

Edge-based Region
Extended Kalman Filter

FoV

Field of View

GPU

Graphics Processing Unit

IAC
IBCA
IBR

Image of the Absolute Conic

Image Brightness Constancy Assumption
Intensity-based Region

LAF
LDA

Local Ane Frame

Linear Discriminant Analysis

MAD
MDA
MLE
MSER

Mean Absolute Dierence

Multiple Discriminant Analysis
Maximum Likelihood Estimation
Maximally Stable Extremal Region

NCC

Normalized Cross Correlation

PCA
pdf

Principal Component Analysis

probability density function

Acronyms
PLI
PMD

Pre-image of the line at innity

Photonic Mixer Device

RANSAC Random Sampling Consensus

SAD
SfM
SIFT
SLAM
SSD
SVD

Sum of Absolute Dierences

Structure from Motion
Scale Invariant Feature Transform
Simultaneous Localization and Mapping
Sum of Squared Dierences
Singular Value Decomposition

ToF

Time of Flight

VRML

Virtual Reality Modeling Language

Symbols and Notation

To improve readability and for clarity of the equations, the following fonts, styles, and
symbols have been used:
R
C
Pn
s
x
x

the set of the real numbers

the set of the complex numbers
the projective space representing vectors of
Euclidean vector from
element from

Pn

and the ideal points

Rn

Matrix acting on elements from

Inn
0nn

0n

l
[x]

I [x]
H [x]
f
x

Rn

a scalar number (real-valued if not denoted otherwise)

sup [f]

Matrix acting on elements from

nn
nn

identity matrix
zero matrix

zero vector of dimension

plane at innity
line at innity

33

cross product matrix of

for all
there exist(s)

I 's grey value at x

H acting on x
derivative of f with respect to x,
supremum of f [x] for x [a; b]

image

function

[a;b]

det [M]
trace [M]

Rn
Pn

determinant of square matrix

trace of square matrix

evaluated at

equality up to scale, collinearity

euc [x]
hom [x]
vec [M]
dim [x]
exp [x]

homogeneous to euclidean mapping

euclidean to homogeneous mapping
vectorization of matrix, stacking rows on top of one another
dimension of

exponential function of

Throughout the document, boldface italic serif letters

XI

will always denote Euclidean

XII

Acronyms

vectors while boldface upright serif letters x denote homogeneous vectors. Matrices appear
as capital letters, where those acting on Euclidean vectors are denoted by italic font without
serifs (as A), while matrices acting on homogeneous vectors are denoted by upright serif
letters (as A). Functions are indicated by typewriter font T with the argument in square
brackets [arg], while matrices are indicated by round brackets.