Camera Pose

Estimation
and RANSAC

Srikumar
Ramalingam

Review

Pose
Camera Pose Estimation and RANSAC
Estimation

RANSAC
Srikumar Ramalingam

School of Computing
University of Utah
 Presentation Outline

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam

Review 1 Review
Pose
Estimation

RANSAC
2 Pose Estimation

3 RANSAC
 Camera Models and Projection (Reminder)

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam

Review

Pose
Estimation

RANSAC

Let the optical center be the origin of the camera.

Let (X m , Y m , Z m ) be the coordinates of a 3D point Q,
relative to the world system.
Let the 2D pixel be denoted by q(u, v , 1)T .
 Camera Models and Projection (Reminder)

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam
Projection of 3D point on the image:

Xm
 
Review  
u  
 Ym 
Pose
 v ∼

R −Rt
Estimation K 0  
0T 1  Zm 
RANSAC 1
1

The following 3 × 3 matrix is the camera matrix:

 
ku f 0 ku x0
K =  0 kv f kv y0 
0 0 1
 Projection Matrix (Reminder)

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam
The projection matrix that maps 3D points to 2D image is
Review
given by:
Pose  
Estimation
R −Rt
P= K 0
RANSAC 0T 1


P= KR −KRt

P = KR I −t
 What is Camera Calibration?

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam

Review

Pose
Estimation
The task refers to the problem of computing the
RANSAC calibration matrix K.
In other words, we compute the focal length, principal
point, and aspect ratio in the camera matrix.
 Forward Projection

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam

Review

Pose
Estimation

RANSAC

Xm
 
 
u
 Ym 
 v  ∼ KR I −t  
 Zm 
1
1
 Backward Projection

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam

Review

Pose
Estimation

RANSAC

Q ∼ K−1 q

 
u
Q ∼ K−1  v 
1
 Presentation Outline

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam

Review 1 Review
Pose
Estimation

RANSAC
2 Pose Estimation

3 RANSAC
 What is pose estimation?

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam The problem of determining the position and orientation of the
camera relative to the object (or vice-versa).
Review

Pose
Estimation

RANSAC

Left: Camera Image, Right: 3D model of the world

 What is pose estimation?

Camera Pose
Estimation
and RANSAC
The problem of determining the position and orientation of the
Srikumar
Ramalingam camera relative to the object (or vice-versa).
Review

Pose
Estimation

RANSAC

We use the correspondences between 2D image pixels (and

thus camera rays) and 3D object points (from the world) to
compute the pose.
 Pose Estimation

Camera Pose
Estimation
and RANSAC

Srikumar We consider that the camera is calibrated, i.e. we know its

Ramalingam
calibration matrix K.
Review
We are given three 2D image to 3D object
Pose
Estimation correspondences. Let the 3 2D points be given by:
RANSAC      
u1 u2 u3
q1 =  v1  q2 =  v2  q3 =  v3  .
1 1 1

Let the 3 3D points be given by:

Qm m m
1 , Q2 , Q3
 Input and Unknowns

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam

Review

Pose
Given qi , Qm
i , i = {1, 2, 3}, and K in the following equation:
Estimation

qi = KR I −t Qm


RANSAC
i , i = {1, 2, 3}

Our goal is to compute the rotation matrix R and the

translation t.
 Pairwise Distance Computation

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam
Given the three 3D points Qm i , i = {1, 2, 3} we compute
Review the 3 pairwise distances d12 , d23 , and d31 as follows:
Pose
Estimation

RANSAC
dij = dist(Qm m
i , Qj )

dist(Qm m
i , Qj ) =
q
(Xim − Xjm )2 + (Yim − Yjm )2 + (Zim − Zjm )2
 World frame to Camera frame

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam
Let the three 3D points Qm i , i = {1, 2, 3} be denoted by
c
Qi , i = {1, 2, 3} in the camera coordinate system.
Review

Pose
Estimation
In other words, we have Qci = RQm
i − Rt.
RANSAC Here Qm c
i ’s are known variables and Qi ’s are unknowns.
It is easy to observe the following since the distance
between two points do not change when we transform
them from one coordinate frame to another:

dist(Qm m c c
i , Qj ) = dist(Qi , Qj )
 Reformulation of Pose Estimation

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam

Review

Pose
Estimation

RANSAC

We can compute Qci as follows:

Qci ∼ K−1 qi

Qci = λi K−1 qi
Here λi is an unknown scalar that determines the distance of
the 3D point Qci from the optical center along the ray OQci .
 Reformulation of Pose Estimation

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam

Review

Pose
Estimation

RANSAC

Qci = λi K−1 qi
We simplify the notations, let us denote K−1 qi as follows:
 
Xi
K−1 qi =  Yi  (1)
Zi
 Reformulation of Pose Estimation

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam

Review

Pose
Estimation

RANSAC

 
Xi
Qci = λi  Yi 
Zi
The pose estimation can be seen as the computation of the
unknown λi parameters.
 Reformulation of Pose Estimation

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam

Review

Pose
Estimation

RANSAC

dist(Qci , Qcj ) = dist(Qm m

i , Qj ) = dij , ∀i, j = {1, 2, 3}, i 6= j

q
(λi Xi − λj Xj )2 + (λi Yi − λj Yj )2 + (λi Zi − λj Zj )2 = dij
 Reformulation of Pose Estimation

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam

Review

Pose
Estimation

RANSAC

(λ1 X1 − λ2 X2 )2 + (λ1 Y1 − λ2 Y2 )2 + (λ1 Z1 − λ2 Z2 )2 = d12

2

(λ2 X2 − λ3 X3 )2 + (λ2 Y3 − λ3 Y3 )2 + (λ2 Z2 − λ3 Z3 )2 = d23

2

(λ3 X3 − λ1 X1 )2 + (λ3 Y3 − λ1 Y1 )2 + (λ3 Z3 − λ1 Z1 )2 = d31

2

We have 3 quadratic equations and 3 unknowns.

 Reformulation of Pose Estimation

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam

Review
(λ1 X1 − λ2 X2 )2 + (λ1 Y1 − λ2 Y2 )2 + (λ1 Z1 − λ2 Z2 )2 = d12
2

Pose (λ2 X2 − λ3 X3 )2 + (λ2 Y3 − λ3 Y3 )2 + (λ2 Z2 − λ3 Z3 )2 = d23

2
Estimation

RANSAC
(λ3 X3 − λ1 X1 )2 + (λ3 Y3 − λ1 Y1 )2 + (λ3 Z3 − λ1 Z1 )2 = d31
2

We have 3 quadratic equations and 3 unknowns.

We can have a total of 23 possible solutions for the three
parameters (λ1 , λ2 , λ3 ).
Several numerical methods exist to solve the polynomial
system of equations.
 How to identify a unique solution?

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam

Review
Out of the 8 solutions, only one will be the correct
Pose
Estimation solution.
RANSAC
In some of the solutions, the 3D point will be behind the
camera.
Using additional point correspondence, we can identify the
correct solution.
 Computing the Pose

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam

Review
We remind you the relation between Qci and Qm
i :
Pose c m
Estimation Qi = RQi − Rt.
RANSAC
We are given Qm c
i and we have computed Qi .
From three 3D-to-3D point correspondences we can
compute the transformation parameters (R, t) using
Horn’s method.
 Presentation Outline

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam

Review 1 Review
Pose
Estimation

RANSAC
2 Pose Estimation

3 RANSAC
 Matching Images

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam

Review

Pose
Estimation

RANSAC
 Matching Images

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam

Review

Pose
Estimation

RANSAC

We match keypoints from left and right images.

 Matching Images

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam

Review

Pose
Estimation

RANSAC

We match keypoints from left and right images.

 Matching Images

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam

Review

Pose
Estimation

RANSAC

We match keypoints from left and right images.

 Matching Images

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam

Review

Pose
Estimation

RANSAC

We match keypoints from left and right images.

 Matching Images

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam

Review

Pose
Estimation

RANSAC

We match keypoints from left and right images.

 Matching Images

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam

Review

Pose
Estimation

RANSAC

We match keypoints from left and right images.

One of the matches is incorrect!
 Matching Images

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam

Review

Pose
Estimation

RANSAC

We match keypoints from left and right images.

One of the matches is incorrect!
In a general image matching problem, we can have 100’s
of incorrect matches.
 Outliers and Inliers

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam

Review

Pose
Estimation

RANSAC

We match keypoints from left and right images.

 Outliers and Inliers

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam

Review

Pose
Estimation

RANSAC

We match keypoints from left and right images.

 Robustness

Camera Pose
Estimation
and RANSAC Lets consider a simpler example linear regression.
Srikumar
Ramalingam

Review

Pose
Estimation

RANSAC

How can we fix this?

Slide: Noah Snavely
 Idea

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam

Review
Given a hypothesized line.
Pose
Estimation Count the number of points that agree with the line, i.e.,
RANSAC points within a small distance of the line.
For all possible lines, select the one with the largest
number of inliers.
Slide: Noah Snavely
 Counting Inliers

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam

Review

Pose
Estimation

RANSAC

Slide: Noah Snavely

 Counting Inliers

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam

Review

Pose
Estimation

RANSAC

3 inliers
Slide: Noah Snavely
 Counting Inliers

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam

Review

Pose
Estimation

RANSAC

20 inliers!
Slide: Noah Snavely
 How do we find the best line?

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam

Review

Pose
Estimation Unlike least-squares, no simple closed-form solution
RANSAC Hypothesize-and-test
Try out many lines, keep the best one
Which lines?
 Translations

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam

Review

Pose
Estimation

RANSAC

Slide: Noah Snavely

 RANdom SAmple Consensus

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam

Review

Pose
Estimation

RANSAC

Slide: Noah Snavely

 RANdom SAmple Consensus

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam

Review

Pose
Estimation

RANSAC

Slide: Noah Snavely

 RANdom SAmple Consensus

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam

Review

Pose
Estimation

RANSAC

Slide: Noah Snavely

 RANdom SAmple Consensus

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam

Review

Pose
Estimation

RANSAC

Slide: Noah Snavely

 RANdom SAmple Consensus

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam

Review

Pose
Estimation

RANSAC

Slide: Noah Snavely

 RANdom SAmple Consensus

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam

Review

Pose
Estimation

RANSAC

Slide: Noah Snavely

 RANdom SAmple Consensus

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam

Review

Pose
Estimation

RANSAC

Slide: Noah Snavely

 RANdom SAmple Consensus

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam

Review

Pose
Estimation

RANSAC

Slide: Noah Snavely

 RANdom SAmple Consensus

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam

Review

Pose
Estimation

RANSAC

Slide: Noah Snavely

 RANdom SAmple Consensus

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam

Review

Pose
Estimation

RANSAC

Slide: Noah Snavely

 RANSAC

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam
Idea:
Review

Pose
All the inliers will agree with each other on the translation
Estimation vector; the (hopefully small) number of outliers will
RANSAC (hopefully) disagree with each other
RANSAC only has guarantees if there are ≤ 50% outliers
All good matches are alike; every bad match is bad in its
own way - Alyosha Efros, CMU
Slide: Noah Snavely
 Pose Estimation

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam

Review

Pose
Estimation

RANSAC

Inliers?
 Pose Estimation

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam

Review

Pose
Estimation

RANSAC

Inliers?
 Pose Estimation

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam

Review

Pose
Estimation

RANSAC

Inliers?
 Pose Estimation

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam

Review

Pose
Estimation

RANSAC

Inliers?
 RANSAC

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam
Inlier threshold related to the amount of noise we expect
Review in inliers
Pose Often model noise as Gaussian with some standard
Estimation
deviation (e.g., 3 pixels)
RANSAC
Number of rounds related to the percentage of outliers
we expect, and the probability of success we would like to
guarantee
Suppose there are 20% outliers, and we want to find the
correct answer with 99% probability
How many rounds do we need?
 Sample size

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam

Review

Pose
Estimation

RANSAC

How do we generate a hypothesis?

Slide: Noah Snavely
 General Version - RANSAC

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam
1 Randomly choose s samples
Review Typically s = minimum sample size that lets you fit a
Pose model
Estimation

RANSAC
2 Fit a model (e.g., line) to those samples
3 Count the number of inliers that approximately fit the
model
4 Repeat N times
5 Choose the model that has the largest set of inliers
Slide: Noah Snavely
 How many rounds?

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam

Review

Pose
Estimation

RANSAC

If we have to choose s samples each time

with an outlier ratio e
and we want the right answer with probability p
Slide: M. Pollefeys
 Acknowledgments

Camera Pose
Estimation
and RANSAC

Srikumar
Ramalingam

Review Some presentation slides are adapted from the following

Pose
Estimation
materials:
RANSAC Peter Sturm, Some lecture notes on geometric computer
vision (available online).
Kristen Grauman’s computer vision lecture slides
Noah Snavely’s computer vision lecture slides