07 ShapefX

Shape from X
Videos: 2021_CV1_VO6.1_Stereo, 2021_CV1_VO6.2_Epipolar,

2021_CV1_VO6.3_PhotStereo
Readings:
Szeliski: Chapter 11 (11.1,11.2,11.3,11.4)

Chapter 12 (12.1,12.2)
Some Slides adapted from Univ. of Washington

http://www.cs.washington.edu/education/courses/cse576/08sp/
Some Slides adapted from Cornell
http://www.cs.cornell.edu/courses/cs4670/2018sp/
Horst Bischof
Professor Horst Cerjak, 19.12.2005 Computer Vision I SS21
1
Shape from X
or
How to get 3D Information
Horst Bischof
2
Ill-posed
• In trying to extract 3d structure from 2d images, vision is
an ill-posed problem.
Horst Bischof
3
Ill-posed
Horst Bischof
4
Ill-posed
- An image isn’t enough to disambiguate the many

possible 3d worlds that could have produced it.
Horst Bischof
5
Ill-posed
Horst Bischof
6
Horst Bischof
7
Depth Cues
Size of known objects

Occlusion of Objects
Color change…far away is blueish
Motion…Slow motion if far away
Stereo
Focus
Texture
Vergence
Horst Bischof
8
Shape from x
Profile (3D Scanning)
Shape from Shading
Shape from Shadow
Structured Light
Shape from Focus
Photometric Stereo
Interferometry
Stereopsis
Shape from Motion
Tomography (CAT)
Horst Bischof
9
Shape from Stereo
Horst Bischof
10
Why Stereo Vision?
2D images project 3D points into 2D:
P
Q
P’=Q’
O
• 3D Points on the same viewing line have the

same 2D image:
– 2D imaging results in depth information loss
Horst Bischof
11
Stereo Vision
Refers to the ability of:
The ability to infer information on the 3D

structure and distance of a scene from two
or more images taken from different viewpoints.
Horst Bischof
12
Stereograms
• Invented by Sir Charles Wheatstone, 1838
Horst Bischof
13
Horst Bischof
14
Horst Bischof
15
Random Dot Stereograms
1960 to proof that only Stereo information is necessary

[Julez59]
Horst Bischof
16
Remember: Pinhole Camera
From Th. Svoboda
Horst Bischof
17
camera
Perspective projection world
y
z
y’
Cartesian coordinates:
We have, by similar triangles, that
(x, y, z) -> (f x/z, f y/z, -f)
Ignore the third coordinate, and get
x y
(x, y,z) → ( f ,f )
z z
Horst Bischof
 18
Other projection models:
Orthographic projection
( x, y , z ) → ( x, y )
Horst Bischof
19
Other projection models:
Weak perspective
• Issue
– perspective effects, but not
over the scale of individual
objects
– collect points into a group
at about the same depth,
then divide each point by
the depth of its group
– Adv: easy
– Disadv: only approximate
 fx fy 
( x, y, z ) →  , 
 z0 z0 
Horst Bischof
20
Recap: Camera calibration
Y
Y’
Z O
O’ X’ X
Z’
Horst Bischof
21
Final perspective projection
Camera extrinsics: where your camera is relative
to the world. Changes if you move the camera
Camera intrinsics: how

your camera handles pixel.
Changes if you change
your camera
Horst Bischof
22
Camera calibration
• Need to estimate P
• How many parameters does P have?
– Size of P : 3 x 4
– But:
– P can only be known upto a scale
– 3*4 - 1 = 11 parameters
Horst Bischof
23
Recovering Depth Information:
P
Q
P’1 P’2=Q’2
Q’1
O2
O1
Depth can be recovered with two images and triangulation.
Horst Bischof
24
Stereo Vision Problems:
Correspondence Problem:
– Determining which pixel on the left corresponds to which pixel on
the right.
Reconstruction Problem (Triangulation):

– Given a number of correspondence pairs and camera geometry
information, find location and 3D structure of the observed objects.
Horst Bischof
25
Triangulation
• Suppose we have two cameras
– Calibrated: parameters known
• And a pair of corresponding pixels
• Find 3D location of point!
(x1,y1)
(x2,y2)
Horst Bischof
26
Binocular stereo
• Given two calibrated cameras
– Find pairs of corresponding pixels
– Use corresponding image locations to set
up equations on world coordinates
– Solve!
Horst Bischof
27
Binocular stereo
• General case: cameras can be arbitrary locations and
orientations
Horst Bischof
28
Binocular stereo
• Special case: cameras are parallel to each other and
translated along X axis
X axis
Z axis
Horst Bischof
29
Stereo head
Kinect / depth cameras
Horst Bischof
30
A simple stereo system
P
Z
x xr
f l
pl pr Disparity:
Ol Or
T
T is the stereo baseline

d measures the difference in retinal position between corresponding points
Horst Bischof
31
A simple stereo system
P
Z
x xr
f l
pl pr
Ol Or •Depth is inversely
proportional to disparity
T
Horst Bischof
32
Disparity
Images Disparity
Position 1
d1
Position 2
d2
Position 3
d3
Horst Bischof
33
The math
Horst Bischof
34
Perspective projection in rectified
cameras
• Without loss of generality, assume origin is at pinhole
of 1st camera
Horst Bischof
35
cameras
of 1st camera
Horst Bischof
36
cameras
of 1st camera
Horst Bischof
37
cameras
of 1st camera
Horst Bischof
38
cameras
of 1st camera
Horst Bischof
39
cameras
of 1st camera
Horst Bischof
40
cameras
of 1st camera
X coordinate differs by tx/Z
Horst Bischof
Professor
Y coordinate is the same!
Horst Cerjak, 19.12.2005 Computer Vision I SS21
41
cameras
• X coordinate differs by tx/Z
• That is, difference in X coordinate is inversely
proportional to depth
• Difference in X coordinate is called disparity
• Translation between cameras (tx) is called baseline
• disparity = baseline / depth
Horst Bischof
42
The disparity image
• For pixel (x,y) in one image, only need to know
disparity to get correspondence
• Create an image with color at (x,y) = disparity
right image
disparity
left image
Horst Bischof
43
cameras
• For rectified cameras, correspondence problem is easier

• Only requires searching along a particular row.
Horst Bischof
44
Finding Correspondences:
P Q
P’1 P’2 Q’2

Q’1
O2
O1
Horst Bischof
45
Finding Correspondences:
Horst Bischof
46
NCC - Normalized Cross Correlation
• Lighting and color change pixel intensities
• Example: increase brightness / contrast
• 𝐼 ′ = 𝛼𝐼 + 𝛽
• Subtract patch mean: invariance to 𝛽
• Divide by norm of vector: invariance to 𝛼
• 𝑥′ = 𝑥− < 𝑥 >
𝑥′
• 𝑥′′ =
||𝑥′||
• similarity = 𝑥 ′′ ⋅ 𝑦′′
Horst Bischof
47
Cross-correlation of neighborhood
translate so that mean is zero
Horst Bischof
48
left image band
right image band
1
0.5 cross
correlation
Horst Bischof
Professor Horst Cerjak, 19.12.2005 x Computer Vision I SS21
49
target region
left image band

right image band
1
0.5 cross
correlation
Horst Bischof
Professor Horst Cerjak, 19.12.2005 x Computer Vision I SS21
50
Results with window search
Data
Window-based matching Ground truth
Horst Bischof
51
Hierarchical Matching
32 32 Coarse to fine Matching:
Stereo
32
32 – Start at a certain Resolution

matching
– Calculate Depth
left(n) right(n) depth(n)
64 64 – Use Calculated Depth as

Approximation for Finer
Stereo Resolution
64
64
matching
– Repeat till the full resolution
left(n-1) right(n-1) depth(n-1) has been reached
512 512
Stereo
512
512
matching
left(0) right(0) depth(0)
Horst Bischof
52
Dense 3D Models
Horst Bischof
53
cameras
• For rectified cameras, correspondence problem is

easier
Horst Bischof
54
Rectifying cameras
• Given two images from two cameras with known

relationship, can we rectify them?
Horst Bischof
55
Rectifying cameras
• Can we rotate / translate cameras?

– Do we need to know the 3D structure of the
world to do this?
Horst Bischof
56
Rotating cameras
• Assume K is identity
• Assume coordinate system at camera pinhole
Horst Bischof
57
Rotating cameras
What happens if the camera is rotated?
Horst Bischof
58
Rotating cameras
• What happens if the camera is rotated?
Rotation matrix
Homogenous coordinates of Homogenous coordinates of

mapped pixel original pixel
• No need to know the 3D structure

Horst Bischof
59
Rotating cameras
Horst Bischof
60
Rectifying cameras
Horst Bischof
61
Rectifying cameras
Horst Bischof
62
Rectifying cameras
Horst Bischof
63
Rectifying cameras
Horst Bischof
64
cameras
• For rectified cameras, correspondence problem is easier

Horst Bischof
65
cameras
What about non-

rectified
cameras?
Is there an
• For rectified cameras, correspondence problem is
easier equivalent?
Horst Bischof
66
Epipolar constraint
• Reduces 2D search problem to search along a

particular line: epipolar line
Horst Bischof
67
Epipolar constraint
True in general!
• Given pixel (x,y) in one image, corresponding pixel in the
other image must lie on a line
• Line function of (x,y) and parameters of camera
• These lines are called epipolar line
Horst Bischof
68
Epipolar geometry
Horst Bischof
69
Epipolar geometry - why?
• For a single camera, pixel in image plane must
correspond to point somewhere along a ray
Horst Bischof
70
Epipolar geometry
Given an image point in one view, where is the
corresponding point in the other view?
?
epipolar line
C /
epipole C
baseline
• A point in one view “generates” an epipolar line in the other
view
• The corresponding point lies on this line
Horst Bischof
71
Epipolar line
Epipolar constraint
• Reduces correspondence problem to 1D search along an
epipolar line
Horst Bischof
72
Epipolar lines
Horst Bischof
73
Epipolar lines
Horst Bischof
74
Epipolar lines
Epipole
Horst Bischof
75
Epipolar geometry continued
Epipolar geometry is a consequence of the coplanarity of the camera
centres and scene point
x x/
C C/
The camera centres, corresponding points and scene point lie

in a single plane, known as the epipolar plane
Horst Bischof
76
Nomenclature
X
l/ right epipolar line
left epipolar line x
x/
e e/
C C/
• The epipolar line l/ is the image of the ray through x
• The epipole e is the point of intersection of the line joining the camera centres
with the image plane
this line is the baseline for a stereo rig, and
the translation vector for a moving camera
• The epipole is the image of the centre of the other camera: e = PC/ , e/ = P/C
Horst Bischof
77
The epipolar pencil
X
e e /
baseline
As the position of the 3D point X varies, the epipolar planes “rotate” about
the baseline. This family of planes is known as an epipolar pencil (a pencil is
a one parameter family).
All epipolar lines intersect at the epipole.
Horst Bischof
78
The epipolar pencil
X
e e /
baseline
As the position of the 3D point X varies, the epipolar planes “rotate” about
the baseline. This family of planes is known as an epipolar pencil (a pencil is
a one parameter family).
All epipolar lines intersect at the epipole.
Horst Bischof
79
Epipolar geometry - the math
Homogenous Homogenous
coordinates of point in coordinates of point in
image 2 image 1
Essential
matrix
Horst Bischof
80
Epipolar geometry - the math
• We assumed that intrinsic parameters K are identity

• What if they are not?
Fundamental matrix
Horst Bischof
81
Fundamental matrix
Fundamental matrix
Horst Bischof
82
Estimating F
• If we don’t know K1, K2, R, or t, can we estimate F for

two images?
• Yes, given enough correspondences

Horst Bischof
83
Estimating F – 8-point algorithm
• The fundamental matrix F is defined by

x' Fx = 0
for any pair of matches x and x’ in two images.
 f11 f12 f13 
• Let x=(u,v,1)T and x’=(u’,v’,1)T, F =  f 21 f 22 f 23 
 f 31 f 32 f 33 
each match gives a linear equation
uu' f11 + vu' f12 + u ' f13 + uv' f 21 + vv' f 22 + v' f 23 + uf31 + vf 32 + f 33 = 0
Horst Bischof
84
8-point algorithm
• Pros: it is linear, easy to implement and fast

• Cons: susceptible to noise
• Degenerate: if points are on same plane
• Normalized 8-point algorithm: Hartley

– Position origin at centroid of image points
– Rescale coordinates so that center to
farthest point is sqrt (2)
Horst Bischof
85
Active stereo with structured light
Li Zhang’s one-shot stereo
camera 1 camera 1
projector projector
camera 2
• Project “structured” light patterns onto the object
– simplifies the correspondence problem
Horst Bischof
86
Microsoft Kinect
Horst Bischof
87
Laser scanning
Digital Michelangelo Project

http://graphics.stanford.edu/projects/mich/
• Optical triangulation
– Project a single stripe of laser light
– Scan it across the surface of the object
– This is a very precise version of structured light scanning
Horst Bischof
Source: S. Seitz 88
Laser scanned models
The Digital Michelangelo Project, Levoy et al.

Horst Bischof
Source: S. Seitz 89

Horst Bischof
Source: S. Seitz 90

Horst Bischof
Source: S. Seitz 91
Photometric stereo
Horst Bischof
92
Radiance
• Pixels measure radiance
This pixel
Measures radiance along this ray
Horst Bischof
93
Where do the rays come from?
• Rays from the

light source
“reflect” off a
surface and
reach camera
• Reflection:
Surface absorbs
light energy and
radiates it back
Horst Bischof
94
Light rays interacting with a surface
• Light of radiance 𝐿𝑖 comes from light source at an
incoming direction 𝜃𝑖
• It sends out a ray of radiance 𝐿𝑟 in the outgoing
direction 𝜃𝑟
• How does 𝐿𝑟 relate to 𝐿𝑖 ?
• N is surface normal
• L is direction of light, making 𝜃𝑖
with normal
𝜃𝑖 𝜃 • V is viewing direction, making 𝜃𝑟
𝑟 with normal
Horst Bischof
95
• N is surface normal
• L is direction of light, making 𝜃𝑖
with normal
𝜃𝑖 𝜃 • V is viewing direction, making 𝜃𝑟
𝑟 with normal
Output radiance
along V 𝐿𝑟 = 𝜌 𝜃𝑖 , 𝜃𝑟 𝐿𝑖 cos 𝜃𝑖 Incoming
irradiance along L
Bi-directional reflectance function (BRDF)
Horst Bischof
96
𝜃𝑖 𝜃
𝐿𝑟 = 𝜌 𝜃𝑖 , 𝜃𝑟 𝐿𝑖 cos 𝜃𝑖
𝑟
• Special case 1: Perfect mirror

– 𝜌 𝜃𝑖 , 𝜃𝑟 = 0 unless 𝜃𝑖 = 𝜃𝑟
• Special case 2: Matte surface
– 𝜌 𝜃𝑖 , 𝜃𝑟 = 𝜌0 (constant)
Horst Bischof
97
Special case 1: Perfect mirror
• 𝜌 𝜃𝑖 , 𝜃𝑟 = 0 unless 𝜃𝑖 = 𝜃𝑟
• Also called “Specular surfaces”
• Reflects light in a single, particular direction
Horst Bischof
98
Special case 2: Matte surface
• 𝜌 𝜃𝑖 , 𝜃𝑟 = 𝜌0
• Also called “Lambertian surfaces”
• Reflected light is independent of viewing direction
Horst Bischof
99
Lambertian surfaces
• For a lambertian surface:
𝜃𝑖 𝜃
• 𝜌 is called albedo 𝑟
– Think of this as paint

– High albedo: white colored surface
– Low albedo: black surface
– Varies from point to point
Horst Bischof
100
Lambertian surfaces
• Assume the light is directional: all

rays from light source are parallel
– Equivalent to a light source
infinitely far away
𝜃𝑖 𝜃
𝑟
• All pixels get light from the same
direction L and of the same
intensity Li
Horst Bischof
101
Lambertian surfaces
Intrinsic Image
Decomposition
Shading image
Reflectance
image
Horst Bischof
102
Reconstructing Lambertian surfaces
• Equation is a constraint on albedo and normals

• Can we solve for albedo and normals?
Horst Bischof
103
Solution 1: Recovery from a single image
• Step 1: Intrinsic image decomposition
– Reflectance image 𝜌 𝑥, 𝑦
– Shading image
– Decomposition relies on priors
on reflectance image
• What kind of priors?
– Reflectance image captures
the “paint” on an object
surface
– Surfaces tend to be of uniform
color with sharp edges when
color changes
Images from Barron et al, TPAMI 13

Horst Bischof
104
Solution 1: Recovery from a single image
• Step 2: Decompose shading image into illumination
and normals
– Called Shape-From-Shading
– Relies on priors on shape: shapes are
smooth
Far
Horst Bischof
Near 105
Limitations
• Big problems
– doesn’t work for shiny things, semi-translucent
things
– shadows, inter-reflections
• Smaller problems
– camera and lights have to be distant
– calibration requirements
• measure light source directions, intensities
• camera response function
• Newer work addresses some of these issues
Horst Bischof
106
Solution 2: Recovery from multiple
images
• Represents an equation in the albedo and normals

• Multiple images give constraints on albedo and normals
• Called Photometric Stereo
Image credit: Wikipedia

Horst Bischof
107
Photometric stereo
N
L3 L2
L1
V
Can write this as a matrix equation:
Horst Bischof
108
Solving the equations
Horst Bischof
109
More than three lights
• Get better results by using more lights
Least squares solution:
Solve for N, kd as before
What’s the size of LLT?

Horst Bischof
110
Photometric Stereo
What results can you get?
Input Normals (RGB Normals (vectors) Shaded 3D Textured 3D

(1 of 12) colormap) rendering rendering
Horst Bischof
111
Results…
from Athos Georghiades

http://cvc.yale.edu/people/Athos.html
Horst Bischof
112
Shape from Focus
Shape from Focus:
Move object until in focus
How to measure focus?
Horst Bischof
113
Form aus „Schärfe“ -
Shape-from-Focus
Microscopy
Horst Bischof
114
Shape from Texture
Interpretation of Scene?
Depth Perception by Size of
Squares
Texture contains 3D
Information
Horst Bischof
115

07 ShapefX

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

07 ShapefX

Uploaded by

Copyright:

Available Formats

Shape from X

Videos: 2021_CV1_VO6.1_Stereo, 2021_CV1_VO6.2_Epipolar,

Szeliski: Chapter 11 (11.1,11.2,11.3,11.4)

Some Slides adapted from Univ. of Washington

How to get 3D Information

- An image isn’t enough to disambiguate the many

Size of known objects

• 3D Points on the same viewing line have the

The ability to infer information on the 3D

1960 to proof that only Stereo information is necessary

From Th. Svoboda

Camera intrinsics: how

Depth can be recovered with two images and triangulation.

Reconstruction Problem (Triangulation):

Kinect / depth cameras

T is the stereo baseline

• disparity = baseline / depth

• For rectified cameras, correspondence problem is easier

P’1 P’2 Q’2

translate so that mean is zero

left image band

Window-based matching Ground truth

32 – Start at a certain Resolution

64 64 – Use Calculated Depth as

• For rectified cameras, correspondence problem is

• Given two images from two cameras with known

• Can we rotate / translate cameras?

What happens if the camera is rotated?

• What happens if the camera is rotated?

Homogenous coordinates of Homogenous coordinates of

• No need to know the 3D structure

• For rectified cameras, correspondence problem is easier

What about non-

• Reduces 2D search problem to search along a

The camera centres, corresponding points and scene point lie

• We assumed that intrinsic parameters K are identity

• If we don’t know K1, K2, R, or t, can we estimate F for

• Yes, given enough correspondences

• Pros: it is linear, easy to implement and fast

• Normalized 8-point algorithm: Hartley

Li Zhang’s one-shot stereo

Digital Michelangelo Project

The Digital Michelangelo Project, Levoy et al.

The Digital Michelangelo Project, Levoy et al.

The Digital Michelangelo Project, Levoy et al.

• Rays from the

Bi-directional reflectance function (BRDF)

• Special case 1: Perfect mirror

• For a lambertian surface:

– Think of this as paint

• Assume the light is directional: all

• Equation is a constraint on albedo and normals

Images from Barron et al, TPAMI 13

• Newer work addresses some of these issues

• Represents an equation in the albedo and normals

Image credit: Wikipedia

Can write this as a matrix equation:

Least squares solution:

Solve for N, kd as before

What’s the size of LLT?

Input Normals (RGB Normals (vectors) Shaded 3D Textured 3D

from Athos Georghiades