Camera Calibration

What is Camera Calibration?

Primarily, finding the quantities internal to the camera that affect the imaging process
Position of image center in the image
It is typically not at (width/2, height/2) of image

Focal length Different scaling factors for row pixels and column pixels Skew factor Lens distortion (pin-cushion effect)

Motivation
Good calibration is important when we need to
Reconstruct a world model: Virtual L.A. project Interact with the world
Robot, hand-eye coordination
Image plane

Evaluation of position of a square for 2 focal lengths (red and blue projection geometry)

We see a square of known size

Scaling of Rows and Columns in Image

Camera pixels are not necessarily square Camera output may be analog (NTSC) Image may be obtained by digitizing card
A/D converter samples NTSC signal
Camera NTSC signal Digitizing Monitor display

CCD/ CMOS

Compound Lens Imaging

Inexpensive single lens systems distort image at its periphery Compound lenses may be used to reduce chromatic effects and pin-cushion effects
Camera Image plane
Nodal Point

Principal planes

Image plane for equivalent pinhole camera is not camera image plane

f C
Center of Projection

Variety of Techniques
VERY large literature on the subject Work of Roger Tsai influential Linear algebra method described here
Can be used as initialization for iterative non linear methods.

Some interesting methods use vanishing points

Camera and Calibration Target

Calibration Procedure
Calibration target: 2 planes at right angle with checkerboard patterns (Tsai grid)
We know positions of pattern corners only with respect to a coordinate system of the target We position camera in front of target and find images of corners We obtain equations that describe imaging and contain internal parameters of camera
As a side benefit, we find position and orientation of camera with respect to target (camera pose)

Image Processing of Image of Target

Canny edge detection Straight line fitting to detected linked edges Intersecting the lines to obtain the image corners Matching image corners and 3D target checkerboard corners
By counting if whole target is visible in image

We get pairs (image point)--(world point)

( xi , yi ) ( X i , Yi , Z i )

Central Projection
If world and image points are represented by homogeneous vectors, central projection is a linear transformation:

xs xi = f zs ys yi = f zs

u f v = 0 w 0

0 f 0

xs 0 0 ys 0 0 zs 1 0 1
y x

xi = u / w, yi = v / w

center of projection

Image plane Scene point (xs , ys , zs ) Image point (xi , yi , f) z

C f

Transformation From Lengths to Pixels

Transformation uses: image center (x0, y0) scaling factors kx and ky y0 ypix y
Image center Image point (xi , yi , f)

x xpix

x0

xs xi = f zs ys yi = f zs
u ' x 0 v' = 0 y w' 0 0 x0 y0 1 xs 0 ys 0 with zs 0 1

x pix y pix

xs + zs x0 = k x xi + x0 = f k x zs y s + z s y0 = k y yi + y0 = f k y zs
x pix = u ' / w'
then

x = f kx y = f ky

y pix = v' / w'

Internal Camera Parameters

u ' x v' = 0 w' 0
x s 0 y 0 0

xs s x0 0 x pix = u ' / w' x = f kx ys y y0 0 with zs y pix = v' / w' y = f ky 0 1 0 1 x0 0 x s x0 1 0 0 0 y0 0 = 0 y y0 0 1 0 0 = K [I 3 | 03 ] 1 0 0 0 1 0 0 1 0

x and y focal lengths in pixels x0 and y0 coordinates of image center in pixels Added parameter s is skew parameter K is called calibration matrix. Five degrees of freedom. K is a 3x3 upper triangular matrix

From Camera Coordinates to World Coordinates y

C x Image plane Image point (xi , yi , f) Z

Y (xs , ys , zs ) M (Xs , Ys , Z s ) O

From Camera Coordinates to World Coordinates 2

x i j C X k z I O M (Xs , Ys , Z s ) in world cordinate system (xs , ys , zs ) in image cordinate system Z

CO = T translation vector J K CM = CO + OM xS i + y S j + zS k = Tx i + Ty j + Tz k + X S I + YS J + Z S K

xS = Tx + X S I.i + YS J.i + Z S K.i

xS Tx I.i y = T + I.j S y z S Tz I.k K.i X S J.j K.j YS J.k K.k Z S J.i

Homogeneous Coordinates
xS Tx I.i J.i K.i X S y = T + I.j J.j K.j Y S y S z S Tz I.k J.k K.k Z S xS I.i y I.j S = z S I.k 1 0 Tx X S Ty YS J.k K.k Tz Z S 0 0 1 1 J.i J.j K.i K.j xS y S = R z S 0 T 3 1 X S T YS Z 1 S 1

From Camera Coordinates to World Coordinates 3

x i j C J X k z I O M (Xs , Ys , Z s ) (xs , ys , zs ) Z

CM = OM - OC

xS i + yS j + z S k = ( X S - X C ) I + (YS - YC ) J + ( Z S - Z C ) K

xS = ( X S - X C ) I.i + (YS - YC ) J.i + ( Z S - Z C ) K.i

~ x cam = R (X - C)

~ (T = -R C)

~ C is vector OC expressed in world coordinate system

Homogeneous Coordinates 2
~ Here we use - R C instead of T

xS y S = R z S 0 T 3 1

XS ~ - R C YS 1 ZS 1

Linear Transformation from World Coordinates to Pixels

Combine camera projection and coordinate transformation matrices into a single matrix P
u' v ' = K [I 3 w' xs y | 0 3 ] s zs 1 XS ~ - R C YS 1 Z S 1

xS y S = R z S 0 T 3 1

XS ~ - R C YS 1 ZS 1

u' v' = K [I | 0 ] R 3 3 T 0 3 w' XS u' Y v' = P S ZS w' 1

x = PX

Properties of Matrix P
Further simplification of P:
x = K [I 3 R | 0 3 ] T 0 3 ~ - R C X 1

[I 3

R | 0 3 ] T 0 3

~ x = K R I3 | -C X ~ P = K R I3 | - C

~ - R C ~ ~ = R - R C = R I3 | -C 1

] [

P has 11 degrees of freedom: ~ 5 from triangular calibration matrix K, 3 from R and 3 from C P is a fairly general 3 x 4 matrix left 3x3 submatrix KR is non-singular

Calibration
1. Estimate matrix P using scene points and their images 2. Estimate the interior parameters and the exterior parameters
~ P = K R I3 | - C

Left 3x3 submatrix of P is product of uppertriangular matrix and orthogonal matrix

Finding Camera Translation

Find homogeneous coordinates of C in the scene C is the null vector of matrix P
P C = 0:
P = K R I3

~ | -C

1 0 0 0 1 0 0 0 1

X c 0 X c Yc = 0 Yc Z c 0 Zc 1 0

Find null vector C of P using SVD

C is the unit singular vector of P corresponding to the smallest singular value (the last column of V, where P = U D VT is the SVD of P)

Finding Camera Orientation and Internal Parameters

Left 3x3 submatrix M of P is of form M=K R
K is an upper triangular matrix R is an orthogonal matrix

Any non-singular square matrix M can be decomposed into the product of an uppertriangular matrix K and an orthogonal matrix R using the RQ factorization
Similar to QR factorization but order of 2 matrices is reversed

RQ Factorization of M
1 0 0 c' 0 s ' c' ' s' ' 0 R x = 0 c s , R y = 0 1 0 , R z = s ' ' c' ' 0 0 s c s ' 0 c' 0 0 1

m33 m32 , s= 2 Compute c = 2 2 1/ 2 2 ( m32 + m33 ) (m32 + m33 )1/ 2

Multiply M by Rx. The resulting term at (3,2) is zero because of the values selected for c and s Multiply the resulting matrix by Ry, after selecting c and s so that the resulting term at position (3,1) is set to zero Multiply the resulting matrix by Rz, after selecting c and s so that the resulting term at position (2,1) is set to zero

M RxR yR z = K M = K R TRTRT = K R z y x

Computing Matrix P
Use corresponding image and scene points
3D points Xi in world coordinate system Images xi of Xi in image

Write xi = P Xi for all i Similar problem to finding projectivity matrix H (i.e. homography) in homework

Improved Computation of P
xi = P Xi involves homogeneous coordinates, thus xi and P Xi just have to be proportional: x i P Xi = 0 Let p1T, p2T, p3T be the 3 row vectors of P
v'i p T Xi w'i p T Xi 3 2 T T xi P X i = w'i p1 Xi u 'i p 3 Xi T u'i p T X i v'i p1 X i 2

T p 1 X i T P X i = p 2 X i p T X i 3

0T 4 w 'i X T i v 'i X T i

w'i X T i 0T 4 u 'i X T i

v 'i X T p 1 i T u'i X i p 2 = 0 0T p 3 4

p1 p2 is a 12 1 vector p3

Improved Computation of P, contd

Third row can be obtained from sum of ui times first row - vi times second row
0T 4 w'i X T i v 'i X T i w'i X T i 0T 4 u 'i X T i v 'i X T p 1 i u 'i X T p 2 = 0 i 0T p 3 4

So we get 2 independent equations in 11 unknowns (ignoring scale) With 6 point correspondences, we get enough equations to compute matrix P Ap=0

Solving A p = 0
Linear system A p = 0 When possible, have at least 5 times as many equations as unknowns (28 points) Minimize || A p || with the constraint || p || = 1
P is the unit singular vector of A corresponding to the smallest singular value (the last column of V, where A = U D V T is the SVD of A)

Called Direct Linear Transformation (DLT)

Improving P Solution with Nonlinear Minimization

Find p using DLT Use as initialization for nonlinear 2 minimization of d ( x i , PX i )
i

Use Levenberg-Marquardt iterative minimization

Radial Distortion
We have assumed that lines are imaged as lines Not quite true for real lenses
Significant error for cheap optics and for short focal lengths

Scene

Image

Corrected image

Radial Distortion Modeling

In pixel cordinates the correction is written
xc x0 = L(r ) ( x x0 )
(x0, y0) (x, y) (xc, yc)

yc y0 = L ( r ) ( y y0 ) with r 2 = ( x x0 ) 2 + ( y y0 ) 2 L(r ) = 1 + 1 r + 2 r 2 + ...

Distorted line And ideal line: (x, y) (x, y)

Minimize f ( 1 , 2 ) = ( x'i xci ) 2 + ( y 'i yci ) 2 i using lines known to be straight (x,y) is radial projection of (x,y) on straight line

References
Multiple View Geometry in Computer Vision, R. Hartley and A. Zisserman, Cambridge University Press, 2000, pp. 138-183 Three-Dimensional Computer Vision: A Geometric Approach, O. Faugeras, MIT Press, 1996, pp. 33-68 A Versatile Camera Calibration Technique for 3D Machine Vision, R. Y. Tsai, IEEE J. Robotics & Automation, RA-3, No. 4, August 1987, pp. 323-344