Camera Matrix

16-385 Computer Vision (Kris Kitani)

Carnegie Mellon University
2D to 2D Transform
(last session)
3D object

3D to 2D Transform
(today)

2D to 2D Transform
(last session)
A camera is a mapping between

the 3D world

and

a 2D image
 A camera is a mapping between
the 3D world and a 2D image

x = PX
2D image camera 3D world
point matrix point

What do you think the dimensions are?

 x = PX
2 3
2 3 2 3 X
X p1 p2 p3 p4 6 Y 7
4 Y 5 = 4 p5 p6 p7 p8 56 7
4 Z 5
Z p9 p10 p11 p12
1
homogeneous Camera homogeneous
image matrix world point
3x1 3x4 4x1
 The pinhole camera
y
image plane
X
x
x

camera center principal axis

z
z=f
principal point

What is the equation for image coordinate x (in terms of X)?

y
What is the equation for image coordinate x (in terms of X)?

image plane

X
Y
?
z
f

Z
y

image plane

X
fY Y
Z

z
f

> >
[X Y Z] 7! [f X/Z f Y /Z]
 Pinhole camera geometry
y
image plane
X
x
x

camera center principal axis

z
z=f
principal point

What is the camera matrix P for a pinhole camera model?

x = PX
 Relationship from similar triangles…
> >
[X Y Z] 7! [f X/Z f Y /Z]

generic camera model 2 3

2 3 2 3 X
X p1 p 2 p3 p 4 6 Y 7
4 Y 5 = 4 p5 p 6 p7 p 8 5 6 7
4 Z 5
Z p9 p10 p11 p12
1

What does the pinhole camera model look like?

2 3
? ? ? ?
P=4 ? ? ? ? 5
? ? ? ?
 Relationship from similar triangles…
> >
[X Y Z] 7! [f X/Z f Y /Z]

generic camera model 2 3

2 3 2 3 X
X p1 p 2 p3 p 4 6 Y 7
4 Y 5 = 4 p5 p 6 p7 p 8 5 6 7
4 Z 5
Z p9 p10 p11 p12
1

What does the pinhole camera model look like?

2 3
f 0 0 0
P=4 0 f 0 0 5
0 0 1 0
 2 3
f 0 0 0
P=4 0 f 0 0 5
0 0 1 0

Camera origin and image origin might be different

CCD array

camera coordinate
p system

image coordinate system

 CCD array

camera coordinate
p system

image coordinate system

2 3
f 0 px 0
P=4 0 f py 0 5
0 0 1 0

Accounts for different origins

 In general, the camera and image sensor
have different coordinate systems

X world point

Oimage

x
image point
Ocamera
In general, there are three different coordinate systems…

X world point

Oimage

Oworld
x
image point
Ocamera

so you need the know the transformations between them

Can be decomposed into two matrices
2 32 3
f 0 px 1 0 0 0
P=4 0 f py 5 4 0 1 0 0 5
0 0 1 0 0 1 0
(3 x 3) (3 x 4)

P = K[I|0]
2 3
f 0 px
K=4 0 f py 5 calibration matrix

0 0 1
 Assumes that the camera and world
share the same coordinate system
2 32 3
f 0 px 1 0 0 0
P=4 0 f py 5 4 0 1 0 0 5
0 0 1 0 0 1 0

What if they are different?

How do we align them?
yc
Camera xc yw
coordinate
system
zc xw

World zw
coordinate
system
 Assumes that the camera and world
share the same coordinate system
2 32 3
f 0 px 1 0 0 0
P=4 0 f py 5 4 0 1 0 0 5
0 0 1 0 0 1 0

What if they are different?

How do we align them?
yc
Camera xc yw
coordinate
system
zc xw

Rt World zw
coordinate
3R rotation and translation to align axis system
 yc
Xw
Camera xc yw
coordinate
system
zc xw

World zw
coordinate
system
 yc
Xw
Camera xc yw
coordinate
system
C zc xw

World zw
Coordinate of the
coordinate
camera center in the
system
world coordinate frame
 Xc

yc
Xw
Camera xc yw
coordinate
system
C zc xw

World zw
Coordinate of the
coordinate
camera center in the (X w C) system
world coordinate frame

(X w C)
Translate
 Why aren’t
Xc the points
aligned?

yc
Xw
Camera xc yw
coordinate
system
C zc xw

World zw
coordinate
system

(X w C)
Translate
What happens to points after alignment?

Rotate Xc
yc R Xw
Camera xc yw
coordinate
system
C zc xw

World zw
coordinate
system

R(X w C)
Rotate Translate
In inhomogeneous coordinates:

X c = R(X w C)
Optionally in homogeneous coordinates:
2 3 2 3
Xc  Xw
6 Yc 7 6 Yw 7
6 7= R RC 6 7
4 Zc 5 0 1 4 Zw 5
1 (4 ⇥ 4) 1

General mapping of a pinhole camera

P = KR[I| C]
 Quiz

What is the meaning of each matrix of the camera matrix

decomposition?

P = KR[I| C]

3x3
intrinsics
 Quiz

What is the meaning of each matrix of the camera matrix

decomposition?

P = KR[I| C]

3x3 3x3
intrinsics 3D rotation
 Quiz

What is the meaning of each matrix of the camera matrix

decomposition?

P = KR[I| C]

3x3 3x3 3x3

intrinsics 3D rotation identity
 Quiz

What is the meaning of each matrix of the camera matrix

decomposition?

P = KR[I| C]

3x3 3x3 3x3 3x1

intrinsics 3D rotation identity 3D translation
General mapping of a pinhole camera

P = KR[I| C]
(translate first then rotate)

Another way to write the mapping

P = K[R|t]
where

t= RC
(rotate first then translate)
 Quiz

The camera matrix relates what two quantities?

 Quiz

The camera matrix relates what two quantities?

x = PX
 Quiz

The camera matrix relates what two quantities?

x = PX
3D points to 2D image points
 Quiz

The camera matrix relates what two quantities?

x = PX
3D points to 2D image points

The camera matrix can be decomposed into?

 Quiz

The camera matrix relates what two quantities?

x = PX
3D points to 2D image points

The camera matrix can be decomposed into?

P = K[R|t]
 Quiz

The camera matrix relates what two quantities?

x = PX
3D points to 2D image points

The camera matrix can be decomposed into?

P = K[R|t]
intrinsic and extrinsic parameters
 Generalized pinhole camera model

P = K[R|t]
2 32 3
f 0 px r1 r2 r3 t1
P=4 0 f py 5 4 r4 r5 r6 t2 5
0 0 1 r7 r8 r9 t3
intrinsic parameters extrinsic parameters

2 3 2 3
r1 r2 r3 t1
R = 4 r4 r5 r6 5 t = 4 t2 5
r7 r8 r9 t3
3D rotation 3D translation
 Why do we need P?

to properly relate world points to image points

(by taking into account different coordinate systems)