Lab Course Kinect Programming For Computer Vision: Transformations

Lab Course Kinect Programming for Computer Vision
Transformations and Camera Calibration!

Loren Schwarz (schwarz@in.tum.de)!
May 25, 2011!
Lecture Outline"
! ! Part I: Basics of Transformations ! ! Part II: Basics of Cameras !
Kinect Lab Course - Transformations and Cameras!
2 !
Outline Part I"

2D Transformations! Homogeneous Coordinates in 2D! Hierarchy of Transformations in 2D! Hierarchy of Transformations in 3D!
3 !
2D Transformations"
The basic building blocks of transformations are: ! Scaling (isotropic, non-isotropic)! Rotation (around one rotation axis)! Translation (motion along the two axes)! Transformations are dened by their action on point coordinates.! ! Notation:!
p = (x, y) =
original point!
x y
p = (x , y ) =

transformed point!
x y
4 !
2D Transformations Scaling"
Isotropic scaling: same scaling factor in both spatial dimensions!
x y
s 0
0 s
x y
5 !
2D Transformations Scaling"
Non-isotropic scaling: different scaling factors in the two spatial dimensions!
x y
sx 0
0 sy
x y
6 !
2D Transformations Rotation"
"
x y
cos sin
sin cos
x y
7 !
2D Transformations Rotation"
p= x y = r cos() r sin()
y!
r!
x!
8 !
2D Transformations Translation"
x y
x y
tx ty
9 !
2D Transformations"
Have you noticed we have two types of representations for the basic transformations seen so far?! ! Scaling, Rotation: !Matrix-vector product" ! x sx 0 x x cos sin = = ! y 0 sy y y sin cos ! Translation: !Sum of vectors" " x x tx = + " y y ty " " How to achieve a matrix-vector product form for all types of transformations?!
x y
10 !
Outline Part I"

11 !
Homogeneous Coordinates in 2D"

How to achieve a matrix-vector product form for all types of transformations?! Write a 2D point in homogeneous coordinates:!
! Then, translation can easily be expressed as a matrix-vector product:!
x p = y R3 1
We will see more properties of homogeneous representation later.!
x 1 y = 0 1 0
0 1 0
x + tx tx x ty y = y + ty 1 1 1
12 !

Rotation in homogeneous representation:!
x cos y = sin 1 0
sin cos 0
0 x 0 y 1 1

Combined transformations (here: rotation and translation):!
x cos y = sin 1 0
sin cos 0
xcos ysin + tx tx x ty y = xsin + ycos + ty 1 1 1
13 !
2D Projective Space "

! When dealing with points represented in homogeneous coordinates, we are working in a Projective space.! Points in 2D Euclidean space are represented by Cartesian coordinates!
p=
x y
R2
Points in 2D Projective space are represented by Homogeneous coordinates:!
x p = y P2 1
Kinect Lab Course - Transformations and Cameras! 14 !
2D Projective Space "

y! y! p = (x,y)! x! x! z = 1! z!
Projective Plane!
k(x,y,1)! p = (x,y,1)!
Point in Euclidean 2D space!
Point in Projective 2D space. All points on the orange line represent the same point. A homogeneous vector in 2D projective space represents an equivalence class of points.!
15 !
Points and Lines in 2D Projective Space"

Properties of points" A 2D point (x, y) has the homogeneous representation The homogeneous vector For any non-zero k,
(x, y, 1). !
x y (x, y, w) with w0 represents the point ( , , 1).! w w
(x, y, w) and k(x, y, w) represent the same point.!
16 !

Properties of lines" A line
ax + by + c = 0
has the homogeneous representation if and only if
l = (a, b, c) .!
A point
lies on a line
x l = 0 .!
Two lines
and l intersect in the point
x = l ! l
.!
The line joining the two points
and x is
l = x x.!
17 !

Special points and lines" Points with 3rd coordinate zero
(x, y, 0)are called points at innity.!

= (0, 0, 1) .!
All points at innity lie on the line at innity l
l (x, y, 0) = (0, 0, 1) (x, y, 0) = 0

Parallel lines intersect at innity.!
In projective space, all lines intersect in a point, even parallel lines.!
18 !
Outline Part I"

19 !
Hierarchy of Transformations in 2D"

Classication of transformations in terms of quantities or properties that are invariant (left unchanged) by the transformation! Hierarchy: Starting from most specialized type of transformation, successively remove invariants to get more general types of transformations !
! Most general:! Projective transformations!
!
!
!
Afne transformations!
!
!
!
Similarity transformations!
!
!
! Most specialized:! Euclidean transformations!
Containment relation from general to specialized, e.g.: afne transformations have all properties of projective transformations, plus additional ones.!
20 !
Hierarchy of Transformations in 2D: Euclidean Transformations"

Transformation:!
Concise notation for transformation matrix:!
x cos y = sin 1 0
sin cos 0
tx x ty y 1 1
R R = I det(R) = 1
Teuclidean =
R 0
t 1
Degrees of freedom: !
3 (1 rotation, 2 translation) !
Invariants: length, area, angles!

21 !
Hierarchy of Transformations in 2D: Similarity Transformations"

Transformation:!
x scos y = ssin 1 0
sR 0 t 1
ssin scos 0
tx x ty y 1 1
R R = I
Tsimilarity =
det(R) = 1
4 (1 rotation, 2 translation, 1 isotropic scaling) !
Invariants: ratio of lengths, angles, parallel lines!

22 !
Hierarchy of Transformations in 2D: Afne Transformations"

Transformation:!
x a11 y = a21 1 0
A 0
a12 a22 0
t 1
tx x ty y 1 1
A
non-singular!
Tanity =
6 (2 translation, 4 entries of A) !
Invariants: parallel lines, ratios of areas, line at innity !

23 !
Hierarchy of Transformations in 2D: Afne Transformations"

Afne transformations geometrically consist of two components:!
Rotation!
Non-isotropic Scaling!
A = R()R()
1 0
0 2
R()
24 !
Hierarchy of Transformations in 2D: Projective Transformations"

Transformation:!
x a11 y = a21 1 v1
A v
a12 a22 v2
t v
tx x ty y v 1
A
non-singular!
Tprojectivity =
8 (9 arbitrary matrix entries, up to scale) !
Invariants: Intersection points, tangency, cross-ratios!

25 !
Hierarchy of Transformations in 2D Summary"

Projective
"8 DOF!
! ! !
Afne " "6 DOF!
Tprojectivity =
A v A 0 sR 0
R 0
t v t 1 t 1
t 1
26 !
! ! !
Similarity "4 DOF!
Tanity =
Tsimilarity =
! ! !
Euclidean "3 DOF!
Teuclidean =
Action of Transformations in 2D on Points at Innity"

Afne Transformations:!
a11 a21 0
a12 a22 0
tx x a11 x + a12 y ty y = a21 x + a22 y 1 0 0
Point at innity, Mapped by afne e.g. intersection ! transformation to another of blue lines! point at innity!
27 !
Action of Transformations in 2D on Points at Innity"

Projective Transformations:!
a11 a21 v1
a12 a22 v2
tx x a11 x + a12 y ty y = a21 x + a22 y v 0 v1 x + v2 y

Point at innity, e.g. intersection ! of blue lines!
Mapped by projective transformation to a nite point (vanishing point)!
Only projective transformations can make points at innity nite points.!

Outline Part I"

29 !

How to achieve a matrix-vector product form for all types of transformations?! Write a 3D point in homogeneous coordinates:!
! 3D projective space: properties analogous to 2D homogeneous representation! All types of transformations (scaling, translation, rotation, etc.) can be written as a matrix-vector product! Several transformations can be composed into one transformation matrix!
x y R4 p= z 1
30 !
Hierarchy of Transformations in 3D (1): Euclidean Transformations"

Transformation:!
x r11 y r21 = z r31 1 0
r12 r22 r32 0
r13 r23 r33 0
Teuclidean =
R 0
t 1
tx x ty y tz z 1 1
R R = I det(R) = 1
6 (3 rotation, 3 translation) !
Invariants: volume, angles!

31 !
Hierarchy of Transformations in 3D (2): Similarity Transformations"

Transformation:!
x sr11 y sr21 = z sr31 1 0
sr12 sr22 sr32 0
sr13 sr23 sr33 0
Tsimilarity =
sR 0
t 1
tx x ty y z tz 1 1
R R = I det(R) = 1
7 (3 rotation, 3 translation, 1 scaling) !
Invariants: angles!
32 !
Hierarchy of Transformations in 3D (3): Afne Transformations"

Transformation:!
x a11 y a21 = z a31 1 0
a12 a22 a32 0
a13 a23 a33 0
Tanity =
A 0
t 1
tx x ty y tz z 1 1
non-singular!
12 (3 translation, 9 entries of A) !
Invariants: parallel planes, ratios of volumes, plane at innity !

33 !
Hierarchy of Transformations in 3D (4) Projective Transformations"

Transformation:!
x a11 y a21 = z a31 1 v1
a12 a22 a32 v2
a13 a23 a33 v3
Tprojectivity =
A v
t v
tx x ty y tz z v 1
A
non-singular!
15 (16 arbitrary matrix entries, up to scale) !
Invariants: Intersection points, tangency!

34 !
Hierarchy of Transformations in 3D Summary"

Projective
"15 DOF!
! ! !
Afne " "12 DOF!
Tprojectivity =
A v A 0 sR 0
R 0
t v t 1 t 1
t 1
35 !
! ! !
Similarity "7 DOF!
Tanity =
Tsimilarity =
! ! !
Euclidean "6 DOF!
Teuclidean =
Lecture Outline"
! ! Part I: Basics of Transformations ! ! Part II: Basics of Cameras !
36 !
Pinhole Camera Model"

Mapping between 3D world and 2D image by central projection!
37 !

In homogeneous coordinates!
! fX $ ' f # & ) # fY & = ) # # Z & ) & " % (

*! 0 # , 0 ,# ,# 1 0 +# "
X Y Z 1
$ & & & & %

38 !

Mapping between 3D world and 2D image by central projection!
! fX $ ' f # & ) # fY & = ) # # Z & ) & " % (
! $ *' *# X & 0 ,) 1 ,# Y & 1 0 , ,) # & ,) ,# Z & 1 0 + 1 +( " 1 %

39 !

Origin of image plane is not necessarily at the camera center!
principal point (perpendicular intersection point of principal axis and image plane)
40 !

Origin of image plane is not necessarily at the camera center!
! fX + Zp x # # fY + Zpx # # Z "
$ ' f & ) &=) & ) & ) % (
px f py 1
*! 0 # , 0 ,# ,# 0 ,# +"
X Y Z 1
$ & & & & %
! f # K =# # # "
px $ & py & & 1 & %
intrinsic camera parameters matrix
41 !

Camera center is not necessarily at the world coordinate system origin!
camera center in world coordinates

! ! ! X cam = R X-C
Points with a tilde are in non-homogeneous notation!
" ! R !RC X cam = $ $ 0 1 #
% 'X ' &
3D point in camera coordinates projection to image plane from camera coordinates projection to image plane from world coordinates
x = K [ I | 0 ] X cam
! x = KR "I | !C$ X # %

Camera center is not necessarily at the world coordinate system origin!
! x = KR "I | !C$ X # %
x = PX
! P = KR "I | !C$ # %
projection matrix, 9 DOF
! f # K =# # # "
px $ & py & & 1 & %
! R, C
extrinsic parameters, each 3 DOF
intrinsic parameters matrix, 3 DOF
CCD Camera Model"

Camera coordinates are mapped to pixels of the CCD chip!
mx , my
number of pixels per unit distance in camera coordinates
! m # x K =# my # # " ! ! # x K =# !y # # "
1 x0 y0 1
$! f px &# f py &# &# 1 &# %" $ & & 4 DOF & & %
10 DOF
$ & & & & %
! P = KR "I | !C$ # %
44 !
Finite Projective Camera Model"

Both axes in camera coordinates need not have identical scaling!
skew parameter
! ! s x # K =# !y # # "
x0 $ & y0 & & 1 & %
5 DOF
! P = KR "I | !C$ # %
11 DOF
45 !
Camera Calibration"
Relating the ideal camera model to the properties of an actual physical device and determining the position and orientation of the camera w.r.t. the world coordinate frame1.!
! Estimating the entries of! K, R, C

Camera models with increasing generality:!
pinhole camera ! CCD camera ! nite projective camera ! 9 DOF! !10 DOF! !11 DOF!
! ! s x # K =# !y # # "
x0 $ & y0 & & 1 & %
! P = KR "I | !C$ # %
In addition: lens distortion parameters!
http://www.dis.uniroma1.it/~iocchi/stereo/calib.html!
46 !
Camera Calibration"
Checkerboard pattern captured from multiple views (the checker sizes in the real world are known)
Semi-automatic measurements to find sizes in each of the images
Linear and non-linear optimization for estimating camera parameters MATLAB calibration toolbox: http://www.vision.caltech.edu/bouguetj/calib_doc!
Images from http://www.vision.caltech.edu/bouguetj/calib_doc!
47 !
Camera Calibration"
sdfsdf!
Images from http://www.vision.caltech.edu/bouguetj/calib_doc!
48 !
Stereo Calibration"
Find the spatial relation between two cameras observing the same scene (stereo setup)! For each orientation of the calibration pattern, capture corresponding left and right camera images! Calibrate each camera separately, as before! Perform stereo calibration using the result of separate calibration as an input!
49 !
Calibrating the Kinect"

Depth and RGB images do not match exactly, there is an offset! Stereo calibration for getting both images from the same camera perspective!
RGB Image!
Depth Image!
50 !
Calibrating the Kinect"

Kinect has two cameras with standard optics!
RGB camera! Infrared (IR) camera!
Infrared image is used internally for computing depth image, but is accessible through the API! Checkerboard pattern is not visible in depth images, but visible in IR images!
Infrared! Projector! RGB! Infrared! Camera! Camera!
RGB Image!
Infrared Image!
Relating RGB and Depth Images"

Stereo calibration gives R, t and .!
Step 1: Back-project every 2D point from depth image into 3D space! Step 2: Apply rigid transformation between two cameras to 3D points! Step 3: Project every transformed 3D point into the RGB image!
Back-projection using! Relative transformation! between rigidly connected cameras!

IR"
R, t
RGB"
3D Points! Projection using!
52 !

Step 1: Back-project every 2D point from depth image into 3D space! !
x ToF data: depth d in meters for every pixel location ! = (x, y) Desired data: 3D coordinates X = (X, Y, Z) for every pixel !
Depth Image!
3D Point Cloud!
53 !

Step 1: Back-project every 2D point from depth image into 3D space! ! Apply inverse of IR camera projection to obtain 3D points:!
x fx y = 0 0 w
0 fy 0
cx cy 1
(x cx )Z X= fx
X 0 fx X + cx Z Y = fy Y + cy Z 0 Z 0 Z W
(y cy )Z Y = fy
Inverse relation for X and Y!


Step 1: Back-project every 2D point from depth image into 3D space! ! For every depth pixel at location x = (x, y) with depth d, ! compute a 3D point X = (X, Y, Z) with coordinates: !
Intrinsics of the IR (depth) camera!


Step 2: Apply rigid transformation between two cameras to 3D points! Step 3: Project every transformed 3D point into the RGB image! ! ! ! are the coordinates of the projection of X into the RGB image! look up color corrsponding to a given 3D point!
56 !
Assignment 3"
Extend your application such that it can read out Kinect IR images! Extend your application for saving IR and RGB images to bitmap les! Familiarize yourself with the Matlab Camera Calibration Toolbox http://www.vision.caltech.edu/bouguetj/calib_doc/!
Read the examples (especially examples 1 and 5)! Download the toolbox! Try out the examples!
Perform stereo-calibration for your Kinect device!

Note that you cannot extract IR and RGB images simultaneously!
" Outlook Assignment 4:" Implement a 3D point cloud visualization for kinect depth data! Add a feature to color the 3D points with the colors from the RGB camera!
57 !

Lab Course Kinect Programming For Computer Vision: Transformations

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Lab Course Kinect Programming For Computer Vision: Transformations

Uploaded by

Copyright:

Available Formats

Lab Course Kinect Programming for Computer Vision

Transformations and Camera Calibration!

! ! Part I: Basics of Transformations ! ! Part II: Basics of Cameras !

Kinect Lab Course - Transformations and Cameras!

Outline Part I"

Kinect Lab Course - Transformations and Cameras!

Kinect Lab Course - Transformations and Cameras!

Kinect Lab Course - Transformations and Cameras!

Kinect Lab Course - Transformations and Cameras!

Kinect Lab Course - Transformations and Cameras!

Kinect Lab Course - Transformations and Cameras!

Kinect Lab Course - Transformations and Cameras!

Outline Part I"

Kinect Lab Course - Transformations and Cameras!

Homogeneous Coordinates in 2D"

! Then, translation can easily be expressed as a matrix-vector product:!

We will see more properties of homogeneous representation later.!

Kinect Lab Course - Transformations and Cameras!

Homogeneous Coordinates in 2D"

Combined transformations (here: rotation and translation):!

xcos ysin + tx tx x ty y = xsin + ycos + ty 1 1 1

Kinect Lab Course - Transformations and Cameras!

2D Projective Space "

Points in 2D Projective space are represented by Homogeneous coordinates:!

2D Projective Space "

Point in Euclidean 2D space!

Kinect Lab Course - Transformations and Cameras!

Points and Lines in 2D Projective Space"

x y (x, y, w) with w0 represents the point ( , , 1).! w w

(x, y, w) and k(x, y, w) represent the same point.!

Kinect Lab Course - Transformations and Cameras!

Points and Lines in 2D Projective Space"

has the homogeneous representation if and only if

and l intersect in the point

The line joining the two points

Kinect Lab Course - Transformations and Cameras!

Points and Lines in 2D Projective Space"

(x, y, 0)are called points at innity.!

All points at innity lie on the line at innity l

l (x, y, 0) = (0, 0, 1) (x, y, 0) = 0

In projective space, all lines intersect in a point, even parallel lines.!

Kinect Lab Course - Transformations and Cameras!

Outline Part I"

Kinect Lab Course - Transformations and Cameras!

Hierarchy of Transformations in 2D"

! Most specialized:! Euclidean transformations!

Kinect Lab Course - Transformations and Cameras!

Hierarchy of Transformations in 2D: Euclidean Transformations"

Concise notation for transformation matrix:!

Invariants: length, area, angles!

Kinect Lab Course - Transformations and Cameras!

Hierarchy of Transformations in 2D: Similarity Transformations"

Concise notation for transformation matrix:!

Invariants: ratio of lengths, angles, parallel lines!

Kinect Lab Course - Transformations and Cameras!

Hierarchy of Transformations in 2D: Afne Transformations"

Concise notation for transformation matrix:!

Invariants: parallel lines, ratios of areas, line at innity !

Kinect Lab Course - Transformations and Cameras!

Hierarchy of Transformations in 2D: Afne Transformations"

Kinect Lab Course - Transformations and Cameras!

Hierarchy of Transformations in 2D: Projective Transformations"

! $ ' # X & 0 ,) 1 ,# Y & 1 0 , ,) # & ,) ,# Z & 1 0 + 1 +( " 1 %