ROBOT VISION

Lecture Notes




로봇 비젼
강의 노트





Course No. 15678
Credits Hours 3





Instructor:
Professor
Choi, Tae-Sun

Course outline:

The principles of the machine/robot vision are introduced. It covers image formation,
pattern classification, motion and optical effect for object recognition. In addition, the
design technology of the robot vision system with optical device is studied.

Prerequisite: Digital Signal Processing, Image Processing.

Textbook and References: 1. Robot Vision, B. K. P. Horn, MIT Press.
2. Computer Vision, Dana Ballard and Christopher
Brown, Prentice Hall.
Course Schedule
1. Image Formation & Image Sensing
2. Binary Images: Geometrical Properties
3. Binary Images: Topological Properties
4. Regions & Image Segmentation
5. Image Processing: Continuous Images
6. Image Processing: Discrete Images
7. Edge & Edge Finding
8. Lightness & Color
9. Reflectance Map: Photometric Stereo
10. Reflectance: Shape from Shading
11. Motion Field & Optical Flow
12. Photogrammetry & Stereo
13. Pattern Classification
14. Polyhedral Objects
15. Extended Gaussian Images
16. Passive Navigation & Structure from Motion

1
1. Introduction
I. Human vision system

Figure 1—1
• Rod : more sensitive to light
Scotopic vision
Slender receptor

• Cone : shorter & thicker in structure
Photopic vision : R.G.B
6.5 million

• Fovea : Density of cones is greatest
The region of sharpest photopic vision Camera visual system

2

Figure 1—2
i. Photometric information
brightness : (ex: bright green, dark green)
color (ex, R. G. B)
ii. Geometric information
shape

Figure 1—3

2
NTSC : US standard for TV picture and audio coding and transmission
(timing calculation)
For one frame : 512×512 = 256k Byte
For RGB : 3
For real time : 30 frames/sec
256k × 3 × 30 = 22.5M Byte/sec
180 M bit/sec
180 Mbps T1 1.544 Mbps
Compression ratio :
180
120
1.544

Wavelet compression
MPEG I, II, IV
Compression JPEG
Fractal
H.261
II. Robot vision
RV
Computer graphics
Image processing
compressio
3

Figure 1—4


Figure 1—5

4
“Scene description”
Under constrained problems : Inverse problem
for example 2 equations
3 unknown variables are given
1. Image analysis :a detailed but undigested description
2. Scene analysis : more parsimonious, structured descriptions suitable for
decision marking


The apparent brightness of a surface depends on three factors
1. Microstructure
2. Distribution of incident light
3. Orientation of the surface w. r. t light source & observer


5

Figure 1—6
“Lambertian Surface”
One that appears equally bright from all viewing directions and reflects all incident light,
absorbing none

6
2. Image Formation & Image sensing
What determines where(Shape-Geometric) the image of some point will appear?
What determines how(Brightness, Color – Photonetiz) bright the image of some surface
will be?

I. Two Aspects of Image Formation
i. Perspective projection

Figure 2—1



7
ii. Orthographic Projection
2 2
2 2
0
' ' '
(magnification)
x y f
m
z
x y
δ δ
δ δ
+
= =
+

In case the depth range of a scene is small relative to the distance z
0

Figure 2—2

\
|
=
=
my y
mx x
'
'
if m=1

\
|
=
=
y y
x x
'
'

=> This orthographic projection can be modeled by rays parallel to the optical axis
(rather than one passing through the origin)

(a) (b)
Figure 2—3


I
p
E
δ
δ
= [watt/m]
Where δP is the power of the radiant energy falling on the
8

II. Brightness
1. Image Brightness : Irradiance
2
[ / ]
P
E watt m
I
δ
δ
=
2. Scene Brightness :
L=
2
2
[ / ]
P
L watt m
I W
δ
δ δ
= where δ
2
P is the power emitted by the infinitesimal surface patch
of area (δI) into an infinitesimal solid angle (δw)
2
, where is area, and is Distance
A
SolidAngle A D
D

Ex 1 Hemisphere
w δ
R
2
2
4 1
Solid angle = = 2
2
Whole sphere = 4
R
R
π
π
π

Figure 2—4
Ex 2 Small patch
9

Figure 2—5
Lens

Figure 2—6



cos

( / cos )2
Image Irradiance
P
E
I
I
Solid angle
f
δ
δ
δ α
α
=
=

0
2

=
'
cos

( / cos )
Scene Radiance
P
L
O w
Solid angle
z
δ
δ δ
δ θ
α
=

Transfer Function: ( )
L
f
E
δ
δ
= i
10
f(·) : BRDF(Bidirectional Reflectance Distribution Function) in Ch 10
Ex3) Sphere/Radius 2m

Figure 2—7
III. Lenses


Figure 2—8

i. Lens formula
1 1 1
' f z z
= + 2–1
' '
2
s z z
R D

= 2–2
Solve (2-1) & (2-2) to calculate R – blur circle radius
11
1 1 1
( )
2
DS
R
f s z
= − −
R = 0 at focused point P′
ii. Depth of field (DOF)
The range of distances over which objects are focused “sufficiently well” in the sense
that thediameter of the blur circle is less than the resolution of the imaging device.
The larger the lens aperture => the less the DOF

Figure 2—9

iii. Vignetting
In a simple lens, all the rays that enter the front surface of the lens end up being focused in the image

Figure 2—10
12
In a compound lens, some of the rays that pass through the first lens may be occluded
by portions of the second lens, and so on … Vignetting is a reduction in light-gathering
power with increasing inclination of light rays with respect to the optical axis




Figure 2—11
iv. Aberration
Points on the optical axis may be quite well focused, which those in a corner of the
image are smeared out
13
IV. Image sensing
i. Color sensing
a) Human eye

Figure 2—12
b) CCD sense
R, G, B filter
R(λ) : 1 ( at Red color)
14
0 (otherwise)
G(λ) : 1 (at Green color)
0 (otherwise)
B(λ) : 1 (at Blue color)
0 (otherwise)
→ UV filter filtered out UV ray
ii. Randomness and Noise
a) Random variable [R.V]
Consider an experiment H with sample description space Ω the elements on points of
Ω,δ, are the random out comes of H
If to every δ, we assign a real number X(δ) The real establish a correspondence rule
between δ and R the real time, such a rule subject to certain constraints is called a
random variable

15
Figure 2—13
R.V.
Probability density function (p.d.f) p(x)
Where p(x) ≥ 0 for all x, therefore: ( ) 1 p x dx

−∞
=


mean :
( ) xp x dx µ

−∞
=

: first moment
variance :
2 2 2
( ) ( ) ( ) : second moment x p x dx x p x dx δ µ
∞ ∞
−∞ −∞
= − =
∫ ∫


Cumulative probability distribution (CDF) P(x)
( ) ( )
x
P x p x dt
−∞
=


Two R.V.

S V R
X X X
. .
2 1
+ =

P(X
1
) & P(X
2
) what P(x) = ?
Solution

16
Figure 2—14

Given x
2
, x
1
: x-x
2

1 1 1 1 2
( ) ( ) P x x P x x x δ δ ⋅ = −

Now, x
2
can take on a range of values P
2
(x
2
).δ(x
2
)
To find the prob. that x lies between x and x+δx,
1 2 2 2 2
1 2 2 2 2
1 2
1 2
( ) ( ) ( )
( ) ( ) ( )
( ) ( )

P x x P x x xP x dx
P x P x x P x dx
P x t P t dt
P P
δ δ

−∞

−∞

−∞
⋅ = −
∴ = −
= −
= ∗


∫ ∫

For multiple
RV's
:

=
=
N
i
i
x
N
x
1
1

Mean value :
µ
µ
=
N
N

Variance :
2 2
2
N
N N
σ σ
=
Standard Deviation :
N
σ


b) Gaussian (Normal)
p.d.f :
2
1
2
1
( )
2
x
p x e
µ
σ
πσ
− | |

|
\ ¹
=

17


c) Poisson (m>0)
p.d.f : ( )
!
n
m
m
p k e
n

= ⋅
where n arrivals in a time interval T for some m
mean = m variance = σ
2


Ex Let X and Y be independent r.v.'s with ( ) ( )
x
x
p x e u x

= and
1
( ) [ ( 1) ( 1)]
2
y
p y u y u y = + − − and Let Z X Y ≡ + , what is the p.d.f of Z?

(a) (b)
Figure 2—15

18
( )
1
( ) ( ) * ( ) ( ) ( )
(a) 1 ; ( ) 0
1
(b) 1 1 ; ( )
2
(c) z 1 ; (
z x y x y
z
z
z y
z
z
p z p x p y p z y p y dy
z p z
z p z e dy
p z

−∞
− −

= = −
< − =
− ≤ < =



1
( )
1
1
)
2
z y
e dy
− −

=




Figure 2—16
19
3. Binary images : Geometric properties
I. Binary Images

Figure 3—1
1. Area of the object 1 sin cos 0
sin cos
x y
x y θ θ ρ
ρ ρ
θ θ
+ = ⇒ − + =


( , )
I
A b x y dxdy =
∫ ∫
; zero-th Moment
2. Position of the object : center of area
1
( , ) x xb x y dxdy
A
=
∫ ∫
: First moment
1
( , ) y b x y dxdy
A
=
∫ ∫

3. Orientation of the object
(ρ,θ)
2
( , )
I
E r b x y dxdy =
∫ ∫
: second moment

20
II. Simple Geometrical Properties

Figure 3—2
2 2 2
0 0
2 2
2 2 2 2
( ) ( )
( ( sin cos )) ( ( cos sin ))
( ) 2 ( sin cos ) 2 ( cos sin )
r x x y y
x s y s
x y x y s x y s
ρ θ θ ρ θ θ
ρ ρ θ θ θ θ
= − + −
= − − + + − +
= + + + − − + +

To find the shortest distance from (x, y) to (x
0
, y
0
) differentiate with respect to S and set
equal to zero
2
( )
0
d r
ds
=
θ θ sin cos y x S + =
2 2
0
2
(sin cos ) sin ( cos sin ) cos
sin sin sin cos
sin (sin cos )
x x x x y
x y
y
θ θ ρ θ θ θ θ
θ ρ θ θ θ
θ θ θ ρ
− = + + − +
= + −
= − +

) cos sin ( cos
0
ρ θ θ θ + − − = − y x y y
2
0
2
0
2
) ( ) ( y y x x r − + − = ∴

2
) cos sin ( ρ θ θ + − = y x

21

If, r=0 , ; 0 cos sin = + − ρ θ θ y x
; 0 cos sin ), , ( = + − ρ θ θ y x y x At

Finally,
2
( , ) E r b x y dxdy =
∫ ∫


2
( sin cos ) ( , )
I
x y b x y dxdy θ θ ρ = − +
∫ ∫

Differentiating w.r.t ρ and setting the result to zero lead to
2( sin cos ) ( , ) 0
I
dE
x y b x y dxdy
d
θ θ ρ
ρ
= − + =
∫ ∫

0 ) cos sin ( = + − ρ θ θ y x A
Let ' ' x x x x x x − = ⇒ − =
y y y − = '
θ θ ρ θ θ
ρ θ θ
cos ' sin ' cos sin
0 ) cos ) ' ( sin ) ' (
y x y x
y y x x
− = + −
= + − − −

and so

2
2 2
( ' sin ' cos ) ( , )
' sin ( , ) ( 2 ' ') cos sin
I
I I
x y b x y dxdy
x b x y dxdy x y
θ θ
θ θ θ

= + −
∫ ∫
∫ ∫ ∫ ∫

2 2
2 2
( , ) ' cos ( , )
sin sin cos cos
I
b x y dxdy y b x y dxdy
a b c
θ
θ θ θ θ
+
= − +
∫ ∫

where,
22

2
2
( ') ( , ) ' '
2 ( ' ') ( , ) ' '
( ') ( , ) ' '
I
I
I
a x b x y dx dy
b x y b x y dx dy
c y b x y dx dy
=
= ⋅
=
∫ ∫
∫ ∫
∫ ∫

Now
θ θ 2 sin
2
1
2 cos ) (
2
1
) (
2
1
b c a c a E − − − + =

Differentiating w.r.t θ and setting the result to zero, we have
c a
b

= θ
2
tan
unless 0 ; b a c = =

\
|
+ − =

=


θ θ ρ
θ
cos sin
tan
2
1
1
y x
c a
b


III. Projections

Figure 3—3
23
∫ ∫
= = dx y x b x V dy y x b x h ) , ( ) ( , ) , ( ) (
Area :
( , ) ( )
( )
A b x y dxdy h x dx
v y dy
= =
=
∫ ∫ ∫


Position :

∫ ∫ ∫
=
= =
dy y yv
A
y
dx x xh
A
dxdy y x xb
A
x
) (
1
) (
1
) , (
1

Orientation :
c a
b

=
−1
tan
2
1
θ
( , ) ρ θ θ θ ρ cos sin y x + − =
2 2
( , ) ( )
I
x b x y dxdy x h x dx =
∫ ∫ ∫

2 2
( , ) ( )
I
y b x y dxdy y v x dy =
∫ ∫ ∫

∫ ∫ ∫ ∫ ∫
− − = dy y h y x v x dt t d t dxdy y x xyb
I
) (
2
1
) (
2
1
) ( ) , (
2 2 2


Diagonal projection (θ=45°)

Figure 3—4
24
Suppose that θ is 45°
∫ ∫
+ − = = ds s t s t b dxdy y x b t d ) (
2
1
), (
2
1
( ) , ( ) (

Now consider that
2 2
2 2
1
( ) ( , ) ( )
2
1 1
( ) ( , )
2 2
I I
I
x y b x y dxdy t d t dt
x xy y b x y dxdy
+ =
= + +
∫ ∫ ∫ ∫
∫ ∫

so,
2 2 2
1 1
( , ) ( ) ( ) ( )
2 2
I
xyb x y dxdy t d t dt x h x dx y v y dy = − −
∫ ∫ ∫ ∫ ∫


IV. Discrete Binary Images
(reference of book figure 3-8)

area
∑∑
= =
=
N
i
M
j
ij
b A
1 1

position ) , ( j i
∑∑
∑∑
=
=
i j
ij
i j
ij
jb
A
j
ib
A
i
1
1


orientation ( , ) ρ θ
∑∑
∑∑
∑∑
i j
ij
i j
ij
i j
ij
ijb
b j
b i
2
2


25



V. Run-Length coding

Figure 3—5
Where r
ik
is the k-th run of the i-th line and the first run in each row is a run of zeros

2
.......... ,
6
1
2
1
4
1
2 2
i
i i
n
i
m
k
i
n
i
i k i
m
r r r r r A
i
+ + + = =
∑∑ ∑
= = =

position (center of area) : ) , ( j i
26
∑ ∑

= =
=
= ⋅ =
=
n
i
j
n
i
i
k
m
k
i i
jv
A
j h i
A
i
r h
i
1 1
2
2
1
1
..........
1
,

j
v


1. Find the first horizontal differences of the image data


Figure 3—6
2. The first difference of the vertical projection can be computed from the
projection of the first
27
horizontal differences
⇒ count the number of circles subtracted by the number of triangles.
3. The vertical projection v
j
can be found by summing the result from left to right




Orientation ( , ) ρ θ
j
j i
j t
t i j
ij
j
j
i
i
v j h i d t ijb
v j
h i
∑ ∑ ∑ ∑∑


− − =
2 2 2
2
2
2
1
2
1

Where, ) (
2
1
j i t + =
d
t
can be obtained in a way similar to that used to obtain the vertical projection
28
4. Binary Image : Topological properties
Jordan curve theorem
A simple closed curve separate the image into two simply connected regions

Figure 4—1
4 Connectedness – only edge-adjacent cells are considered neighbors.

Figure 4—2


29
Ex 1

Figure 4—3
4 objects
` 2 backgrounds
No closed curve ⇒ contradiction(by Jordan curve theorem)

8 connectedness – Corner-adjacent cells are considered neighbors, too.

Figure 4—4
6 connectedness

30
Figure 4—5


Figure 4—6

Figure 4—7
i. A sequential Labeling Algorithm
Recursive Labeling
1. Choose a point where b
ij
=1 and assign a label to this point and into neighbors
31

Figure 4—8

2. Next, label all the neighbors of these neighbors ⇒ one component will have
been labeled completely

3. Find new places to start a labeling operation whenever an unlabeled point is
found where
b
ij
=1
4. Try every cell in this scan

Sequential Labeling

32

Figure 4—9

1. if A is zero, there is nothing to do.

Figure 4—10
2. if A is one, if D has been labeled, simply copy that label and move on if not [if
one B orc is labeled, copy that label else, choose a new label for A]
(go to step 2)
33
II. Local counting and iterative modification
i. Local counting

Figure 4—11

Horizontal N
Vertical N Total Length = 2N
34

Figure 4—12
The overestimated rate : 1 : 2 2 : 2 = N N
Considering all slopes overestimated average ratio :
4
π

Overestimated average rate :
π
4


-Euler number-
1. No. of Bodies – No. of Holes
2. No. of upstream convexities – No. of upstream concavities

convexities(+1) concavities(-1)
Figure 4—13
35


Figure 4—14
Body hole
1. body - hole : E = 1- 2 = -1 2 - 0 = 2 1 - 0 = 1 1 - 1 = 0 1 - 0 = 1 1 - 0 = 1
2. convexity - concavity : 1- 2 = -1 2 – 0 = 2 1 – 0 = 1 1 – 1 = 0 1 – 0 = 1 1 - 0 = 1




ii. The additive set property

Figure 4—15
36

⇒This permits to split an image up into smaller pieces and obtain an overall answer by
combining the
results of operations performed on the pieces

Figure 4—16
1 2 2 2 1
) ( ) ( ) ( ) ( ) ( ) ( e E d E c E b E a E e d c b a E + + + + = ∪ ∪ ∪ ∪
[ ( ) ( ) ( ) ( )] E a b E b c E c d E d e − + + + ∩ ∩ ∩ ∩
1-0 = 1 2-0 =2 3-0 =3 2-0 =2
(1 2 2 2 1) (1 2 3 2) = 0 = + + + + − + + +
right (body = 1, hole 1) 1-1 =0
convexities =2
1(body) –
37
concavities =2

Ex1) hand

Figure 4—17
Body : 1 Hole : 0 1-0 =1
Convexities : 5 , Concavities : 4 5-4 =1
Euler number = 1


iii. Iterative modification
Instead of adding up the outputs of Local operators in the previous section, the new
binary image determined by the iterative method can be used as input to another cycle
of computation this process, called iterative modification is useful because it allows us
convexit
concavit
38
to incrementally changes an image that is difficult to process into one that might
succumb to the methods already discussed

Euler Differential E*
2. E* = - ( 1 – 1 = 0 ) + ( 1- 0 ) = 1
Part pattern Current pattern
2


3
39

4

5

* *
E E
new
− =
40
5. Regions & image segmentation

I. 5.1 Thresholding Methods

•Image segmentation : An image segmentation is the partition of an image into a set of
non-overlapping regions whose union is the entire image


Figure 5—1
Detection of discontinuities : points, lines, and edges

Figure 5—2
41

Figure 5—3

=
= + ⋅ ⋅ ⋅ ⋅ ⋅ + + =
q
i
i i q q
z w z w z w z w R
1
2 2 1 1


A mark used for detecting isolated points

Figure 5—4
i. Point detection
The detection of isolated points in an image is straightforward
If |R| > T when T is threshold, it’s an isolated point
ii. Line detection
The next level of complexity involves the detection of lines in an image. Consider
the four masks, as shown below


42

Figure 5—5
• If at a certain point in the image, |R
i
|>|R
j
| for all j≠i that point is said to be
more likely associated with a line in the direction of mask i, for example, if
|R
1
|>|R
j
| , for j=2,3,4 the line is more likely associated with a horizontal line.

iii. Edge detection
The idea underlying most edge detection technologies is the computation of a local
derivative operator

Figure 5—6
43
Gradient operator
(
(
(
(
¸
(

¸





= ∇
y
f
x
f
f

2
2
|
|
¹
|

\
|


+ |
¹
|

\
|


= ∇
y
f
x
f
f

Figure 5—7
Laplacian Operator
2
2
2
2
2
y
f
x
f
f


+


= ∇
) ( 4
8 6 4 2 5
2
z z z z z f + + + − = ∇ in discrete form

Figure 5—8

44
2
2
5 2 4 6 8
Cross Section of f
f = 4 (z ) z z z z

∇ − + + +


II. Thresholding
• Thresholding is one of the most important approaches to image segmentation

Figure 5—9
1 if ( , )
( , )
0 otherwise
f x y T
g x y
< ¦
=
´
¹


III. Region-oriented segmentation
i. Basic Formulation

Figure 5—10

45
Let R represent the entire image region and R
i
the partitioned region for n=1,2,………n
(a) R Ri
n
i
=
=

1
(entire image region)
(b)
i
R is a connected region, i=1,2,……n
(c)
i j
R R ∩ = φ for all i & j , i ≠j

\
|

=
=
φ
φ
φ
1 1
2 1
5 1
R R
R R
R R




(d) ) (
i
R P = True for i=1,2,…….n

Figure 5—11
ii. Region Growing by Pixel Aggregation
“Region Growing” is a procedure that groups pixels or sub-regions into larger regions.
The simplest of these approaches is pixel aggregation, which starts with a set of “seed”
points and from these grows regions by appending to each seed
Point those neighboring pixels that have similar properties



46
Ex 1

Figure 5—12
iii. Region splitting and Merging
Spitting
Let R represent the entire image region and select a predicate P as discussed in the
previous section For a square image, one approach for segmenting R is to subdivide it
successively into smaller and smaller quadrant regions so that, for any region R
i
,
P(R
i
) = TRUE that is, if P(R
i
or R) = FALSE, divide the image into quadrants

Figure 5—13
Merging
If only splitting were used, the final partition likely would contain adjacent regions
With identical properties, this draw back may be remedied by allowing merging as well
as splitting two adjacent regions R
j
&R
k
are merged only if P(R
j
∪R
k
)=TRUE

Figure 5—14
47
• The preceding discussion may be summarized by the following procedure in
which, at any step, we split into four disjointed quadrants any region R
i
where
P(R
i
)= FALSE

Figure 5—15
a) merge any adjacent regions R
j
& R
k
for which P(R
j
∪R
k
) =TRUE ;
b) stop when no further merging and splitting is possible
Ex 2

Figure 5—16
Merging FALSE R P
TRUE R R R P FALSE R P
FALSE R P FALSE R P
FALSE R P TRUE R P
TRUE R P FALSE R P
→ +
+ +
= +
= =
= =
) (
) ( ) (
) ( ) (
) ( ) (
) ( ) (
4
41 32 23 3
43 2
34 1
21
∪ ∪


Figure 5—17
48
6. Image Processing : Continuous images
I. Linear, Shift-Invariant Systems
Spatial domain → Point Spread Function (PSF)
Frequency domain → Modulation Transfer Function (MTF)

Figure 6—1
• Defocused image ( g ) is a processed version of the focused image ( f )
Double the brightness of ideal image and the defocused image. If the image system
moves, f & g moves the same amount.

Figure 6—2

Linearity : 1 2 1 2 ( , ) ( , ) ( , ) ( , ) f x y f x y g x y g x y α β α β + → +
Shift : ( , ) ( , ) f x a y b g x a y b − − → − −
49
II. Convolution
( , ) ( , ) ( , ) f x y h x y g x y → →
( , ) ( , ) ( , ) g x y f x y h d d
g f h
ε η ε η ε η
∞ ∞
−∞ −∞
= − −
→ = ⊗
∫ ∫

Linear
( ) ( ) 1( , ) 2 ( , ) ? f x y f h g x y α β + → → =
[ ] 1 2 ( , ) ( , ) ( , ) ( , ) g x y f x y f x y h d d α ε η β ε η ε η ε η
∞ ∞
−∞ −∞
= − − + − − ⋅
∫ ∫

1 2 ( , ) ( , ) g x y g x y α β = +
Shift invariant
( , ) ( , ) ( , ) f x a y b h x y g x y − − → →
( , ) ( , ) ( , ) g x y f x a y b h d d ε η ε η
∞ ∞
−∞ −∞
= − −
∫ ∫

Impulse response
( , ) ( , ) ( , ) f x y h x y h x y → →
( , ) ( , ) ( , ) g x y f x y h d d ε η ε η ε η
∞ ∞
−∞ −∞
= − −
∫ ∫

and ( , ) ( , ) g x y h x y =

Only possible if
Origin
( , )
0 otherwise
f x y
∞ ¦
=
´
¹

This function is called unit impulse ( , ) x y δ or Dirac delta function
( , ) 1 x y dxdy δ
∞ ∞
−∞ −∞
=
∫ ∫


50
Shifting Property : ( , ) ( , ) (0, 0) x y h x y dxdy h δ
∞ ∞
−∞ −∞
=
∫ ∫

and ( , ) ( , ) ( , ) x y h d d h x y δ ε η ε η ε η
∞ ∞
−∞ −∞
− − ⋅ =
∫ ∫

=> ( , ) h x y : Impulse Response of the system

Figure 6—3
( , ) ( , ) k x y ε η δ ε η − − : scaled delta function

( , ) ( , ) ( , ) f x y f x y d d ε η δ ε η ε η
∞ ∞
−∞ −∞
= ⋅ − −
∫ ∫

( , ) ( , ) ( , ) g x y f x y h x y d d ε η ε η
∞ ∞
−∞ −∞
= − − ⋅
∫ ∫


b a ⊗ = ( , ) ( , ) a x y b x y d d ε η ε η
∞ ∞
−∞ −∞
− − ⋅
∫ ∫
<= Let , x ε α − = y η β − =
= ( , ) ( , ) a b x y d d b a α β α β α β
∞ ∞
−∞ −∞
⋅ − − = ⊗
∫ ∫


Similarly, ( ) ( ) a b c a b c ⊗ ⊗ = ⊗ ⊗


51
Cascade
1 1 2 ( )
1 2
f h f h h
f h h
⊗ ⊗ ⊗
→ → →
=> 1 2 ( ) f h h ⊗ ⊗

1 2 f h h g → ⊗ →
III. MTF
An Eigen function of a system is a function that is reproduced with at most a change in
amplitude.
( )
jwt jwt
e A w e → →

2 ( )
( , )
i ux vy
f x y e
π +
→ =

{ } 2 ( ) ( )
( , ) ( , )
i u x v y
g x y e h d d
π ε η
ε η ε η
∞ ∞
− + −
−∞ −∞
= ⋅
∫ ∫

=
2 ( ) 2 ( )
( , )
i ux vy i u v
e e h d d
π π ε η
ε η ε η
∞ ∞
+ − +
−∞ −∞

∫ ∫

=
2 ( )
( , )
i ux vy
e A u v
π +


IV. Fourier Transform
Represent the signal as an infinite weighted sum of an infinite number of sinusoids (u: angular frequency)
( ) ( )
iux
F u f x e dx


−∞
=


Note:
cos sin 1
ik
e k i k i = + = −
52
Arbitrary function => Single Analytic Expression
Spatial Domain(x)=> Frequency Domain(u) (Frequency Spectrum F(u))
Inverse Fourier Transform (IFT)
( ) ( )
1
2
iux
f x F u e du
π

−∞
=


( )
2
1
( , ) ( , )
4
i ux vy
f x y F u v e dudv
π
∞ ∞
+ +
−∞ −∞
=
∫ ∫


V. The Fourier Transform of Convolution
Let g f h = ⊗
Now,

( ) ( )
( ) ( )
( ) ( )
( )
( ) ( )
( ) ( )
2
2
2 2
2 2 '


' '

i ux
i ux
i u x i u
i u i ux
G u g x e dx
f h x e d dx
f e d h x e dx
f e d h x e dx
F u H u
π
π
π τ π τ
π τ π
τ τ τ
τ τ τ
τ τ
∞ ∞

−∞ −∞
∞ ∞

−∞ −∞
∞ ∞
− − −
−∞ −∞
∞ ∞
− −
−∞ −∞
=
= −
(
( = −
¸ ¸ ¸ ¸
( ( =
¸ ¸ ¸ ¸
=
∫ ∫
∫ ∫
∫ ∫
∫ ∫

Convolution in spatial domain Multiplication in frequency domain
Spatial Domain (x) Frequency Domain(u)
g f h G FH
g fh G F H
= ⊗ ↔ =
= ↔ = ⊗


53
So, we can find g(x) by Fourier transform

Figure 6—4

Spatial Domain (x) Frequency Domain (u)
Linearity
( ) ( )
1 2
c f x c g x + ( ) ( )
1 2
c F u c G u +
Scaling
( ) f ax 1 u
F
a a
| |
|
\ ¹

Shifting
( )
0
f x x − ( )
0
2 i ux
e F u
π −

Symmetry
( ) F x ( ) f u −
Conjugation
( ) f x

( ) F u


Convolution
( ) ( ) f x g x ⊗ ( ) ( ) F u G u
Differentiation
( )
n
n
d f x
dx

( ) ( ) 2
n
i u F u π



54
VI. Generalized Functions and Unit Impulses
• The integral of the one-dimensional unit impulse is the unit step function,
( ) ( )
x
t dt u x δ
−∞
=

, where ( ) u x = 1, for 0; x ≻
1
2
= , for 0; x =
0 = , for 0 x ≺ .
What is the Fourier transform of the unit impulse?
We have
( )
( , ) 1
i ux vy
x y e dxdy δ
∞ ∞
− +
−∞ −∞
=
∫ ∫
, as can be seen by substituting 0 x = and 0 y =
into
( ) i ux vy
e
− +
, using the sifting property of the unit impulse. Alternatively, we can use
( )
0
lim ( , )
i ux vy
x y e dxdy ε
ε
δ
∞ ∞
− +

−∞ −∞
∫ ∫
, or
0
1 1
lim
2 2
iux ivy
e dx e dy
ε ε
ε
ε ε
ε ε
− −

− −
∫ ∫
,
That is
0
sin sin
lim 1
u v
u v
ε
ε ε
ε ε

= .
VII. Convergence Factors and the Unit Impulse
We want a smoothed function of f(x)
( ) ( ) ( ) g x f x h x = ∗
Let us use a Gaussian kernel

Figure 6—5
55
( )
2
2
1 1
exp
2 2
x
h x
σ πσ
(
= −
(
¸ ¸

Then
( ) ( )
2
2
1
exp 2
2
H u u π σ
(
= −
(
¸ ¸

( ) ( ) ( ) G u F u H u =
H(u) attenuates high frequencies in F(u) (Low-pass Filter)!
VIII. Partial Derivatives
What is the F.T. of
( , ) x y f
x


and
f
x


?
2 ( ) i ux vy
f
e dxdy
x
π
∞ ∞
− +
−∞ −∞


∫ ∫
=
2 2 ux vy
f
e dx e dy
x
π π
∞ ∞
− −
−∞ −∞
(


(

¸ ¸
∫ ∫

2 ux
f
e dx
x
π


−∞



=
2 2
( , ) ( 2 ) ( , )
iux iux
f x y e iu f x y e dx
π π
π


− −
−∞
−∞
− − ⋅


=
2 2
( , ) (2 ) ( , )
iux iux
f x y e iu f x y e dx
π π
π


− −
−∞
−∞
+ ⋅



We can’t proceed unless, lim ( , ) 0
x
f x y
→±∞
=
In that case,
2 ( )
2 ( , )
i ux vy
iu f x y e dxdy
π
π
∞ ∞
− +
−∞ −∞
∫ ∫
=(2 ) ( , ) iu F u v π ⋅

( , )
( , )
2 ( , )
2 ( , )
x y
x y
f
F iu F u v
x
f
F iv F u v
x
π
π
∂ ¦ ¹
∴ = ⋅
´ `

¹ )
∂ ¦ ¹
= ⋅
´ `

¹ )

56
IX. Rotational Symmetry and Isotropic operators
• Rotationally symmetric operators are particularly attractive because the treat
image features in the same way, no matter what their orientation is.
• Circular symmetry

Figure 6—6
Let us introduce polar coordinates in both spatial and frequency domain

Figure 6—7
cos x r φ = and sin y r φ = cos( ) ux vy r ρ φ α + = ⋅ −
cos u ρ α = ⋅ and sin v ρ α =

57
{ }
2 ( )
( , ) ( , )
i ux vy
F f x y f x y e dxdy
π
∞ ∞
− +
−∞ −∞
=
∫ ∫

2 cos( )
0 0
( )
i r
r f r e rdrd
π φ α
φ
∞ ∞
− ⋅ −
= ⋅
∫ ∫

Zeroth order Vessel function
(Hankel Transform) => 0 ( ) 2 ( ) (2 ) r F f r J r rdr ρ ρ π π ρ

−∞
=


0
0
( ) 2 ( ) (2 ) fr r f J r d ρ π ρ π ρ ρ ρ

=


where
2
cos( )
0
0
1
( )
2
i x w
J x e d
π
θ
θ
π
− ⋅ −
=



X. Blurring

2 2
2
1
2
2
1
( , )
2
x y
h x y e
σ
πσ
+

=
. . F T ⇓
2 2
2
1
2 ( ) 2
2
1
( , )
2
x y
i ux vy
H u v e e dxdy
π
σ
πσ
+ ∞ ∞

− +
−∞ −∞
= ⋅
∫ ∫

2 2
1 1
2 2 2 2
1 1
2 2
x y
iux ivy
e e dx e e dy
π π σ σ
πσ πσ
| | | | ∞ ∞
− −
| |
− −
\ ¹ \ ¹
−∞ −∞
= ⋅ ⋅
∫ ∫


2
1
2 2
1
2
x
iux
e e dx
π σ
πσ
| | ∞

|

\ ¹
−∞


2 2
1 1
2 2
cos(2 ) sin(2 )
x x
e ux dx i e ux dx
σ σ
π π
| | | | ∞ ∞
− −
| |
\ ¹ \ ¹
−∞ −∞
= − ⋅ ⋅
∫ ∫


58
The second integral is odd so, integral over symmetric region is zero.

And,
2
2 2
1
2
2
1
cos(2 ) ,
2
x u
a
e ux dx e a
a
π
σ
π
π
σ
| | ∞
− − |
\ ¹
−∞
= ⋅ =


2
2 2 2
1
1
(2 )
2
2
cos(2 ) 2
x
u
e ux dx e
π σ
σ
π π σ
| | ∞

− ⋅ ⋅ |
\ ¹
−∞
= ⋅ ⋅



2 2 2 2 2 2
1 1
(2 ) (2 )
2 2
1 1
( , ) 2 2
2 2
u v
H u v e e
π σ π σ
πσ πσ
π σ π σ
− ⋅ ⋅ − ⋅ ⋅
= ⋅ ⋅ ⋅ ⋅ ⋅
⋅ ⋅


2 2 2 2
1
(2 ) ( )
2
u v
e
π σ − ⋅ + ⋅
=
Which is also rotationally symmetric. { } F G a u s s i a n G a u s s i a n =

Figure 6—8: Low Pass Filtering
XI. Restoration and Enhancement
( , ) f x y → ( , ) '( , ) ( , ) h x y h x y f x y → →
The cascade of the two systems is the identity system.
1
'( , ) min ,
( , )
H u v A
H u v
| |
=
|
|
\ ¹
,
Where, A is the maximum gain. Or more elegantly, we can use something like
59
2 2
( , )
'( , )
( , )
H u v
H u v
H u v B
=
+
,
where
1
(2 ) B
is the maximum gain, if ( , ) H u v is real.

XII. Correlation

Figure 6—9
Cross correlation of ( , ) a x y and ( , ) b x y is defined as
( , ) ( , )* ( , )
( , ) ( , )
ab x y a x y b x y
a x y b d d
φ
ε η ε η ε η
∞ ∞
−∞ −∞
=
= − − ⋅
∫ ∫

If ( , ) a x y = ( , ) b x y , the result is called autocorrelation
( , ) .( ( , ) ( , )) ab ab ab x y Symmetric x y x y φ φ φ → − − =
(0, 0) ( , ) aa aa x y φ φ ≥ for all ( , ) x y
2
(0, 0) ( , ) ( , ) ( , ) aa a a d d a d d φ ε η ε η ε η ε η ε η
∞ ∞ ∞ ∞
−∞ −∞ −∞ −∞
= ⋅ =
∫ ∫ ∫ ∫



60
XIII. 6.13 Optimal Filtering

Figure 6—10
E =
{ }
2
( , ) ( , ) o x y d x y dxdy
∞ ∞
−∞ −∞

∫ ∫

( , ) ( , ) ( , ) i x y b x y n x y = +
( , ) ( , ) ( , ) o x y i x y h x y = ⊗

So,
2 2
( , ) 2 ( , ) ( , ) ( , ) E o x y o x y d x y d x y dxdy
∞ ∞
−∞ −∞
= − ⋅ +
∫ ∫

{ }
2
2
( , ) ( , ) ( , ) o x y i x y h x y = ⊗
( , ) ( , ) ( , ) ( , ) i x y h d d i x y h d d ε η ε η ε η α β α β α β
∞ ∞ ∞ ∞
−∞ −∞ −∞ −∞
= − − ⋅ × − − ⋅
∫ ∫ ∫ ∫


Therefore,
2
( , ) ( , ) ( , ) ( , ) o dxdy h h d d d d i x y i x y dxdy ε η α β ε η α β ε η α β
∞ ∞ ∞ ∞ ∞ ∞ ∞ ∞
−∞ −∞ −∞ −∞ −∞ −∞ −∞ −∞
= ⋅ − − ⋅ − −
∫ ∫ ∫ ∫ ∫ ∫ ∫ ∫


x A x A
y B y B
α α
β β
− = ⇒ = +
− = ⇒ = +

61
⇒ ( ( ), ( ) ( , ) i A B i A B dAdB ε α η β
∞ ∞
−∞ −∞
= − − − − ⋅
∫ ∫

( , ) ii φ ε α η β = − − (Autocorrelation)
62
7. Image Processing – Discrete Image
I. Sampling Theorem
To convert a continuous distribution of brightness to a digital image, it necessary to
extract the brightness at each point arranged at a period. Thus is called sampling.

Assume an image as a one-dimensional function for simplicity.
Let the brightness at position x i.e. pixel value of pixel at x, be fix.

Figure 7—1

Figure 7—2
63
Suppose a function that is composed by infinite numbers of Dirac’s delta functions
assigned at an interval T. This is called comb function defined as follows.
A simple digital image from ( ) f x , denoted ( ) T f x , is expressed as ( ) f x multiplied by
the comb function ( ) T comb x , i.e.

( ) ( ) ( ) T T f x f x comb x = ⋅
{ } { } { } ( ) ( ) ( ) ( ) ( ) ( ) T T FT f x v FT f x v FT comb x v = ⊗
and, { }
1
1
( ) ( ) ( ) T
T
FT comb x v comb V
T
=
{ } { }
1
( ) ( ) ( ) ( ) T FT f x v FT f x v
T
= ⊗
1
( )
T
comb V

That’s a convolution of comb function?
We start from “convolution with delta function”
( ) ( ) ( ) ( ) ( ) ( ) ( ) f t t f y t y dy f t t t dy f t δ δ δ
∞ ∞
−∞ −∞
∗ = ⋅ − = ⋅ − =
∫ ∫

i.e., convolution of a function and delta function is equal to original function itself.
Since a comb function is a sequence of the delta function arranged at constant period,
convolution of a function and comb function is an arrangement of duplication of the
original function at a constant period.

64
The F.T. of ( ) T f x , which is the sampled version of ( ) f x at period T is an infinite
sequence of duplicating of { } ( ) T FT f x which is the F.T. of the original function ( ) f x
at period
1
T
.

Figure 7—3
If the period of comb function in the frequency domain is sufficiently large, adjacent
{ } ( ) FT f x ’s do not overlap. In this case, the F.T. of the original function { } ( ) FT f x ,
can be separated and extracted i.e. no information of the brightness distribution of the
original image is lost by sampling. However, if the internal of the comb function is the
frequency domain is small, adjacent { } ( ) FT f x (v)’s overlap. In this case, the original
{ } ( ) FT f x can’t be separated and a faulty function will be extracted. This effect is
called aliasing.
65
{ } { } { } ( ) ( ) ( ) ( ) ( ) ( ) T T FT f x v FT f x v FT comb x v ⇒ = ⊗

Figure 7—4
Since the support of { } ( ) FT f x is in the range between ~ c c V V − , the period has to be at
least 2 c V to avoid overlapping of { } ( ) FT f x ’s since, T is the sampling period.
1
T

denote the number of sampling per unit length, i.e. sampling rate consequently, the
original brightness distribution can be reconstructed by a sampled digital image of the
sampling rate is more than twice the maximum frequency contained in the original
distribution. This theorem is called sampling theorem.



66
Discrete Fourier Transform (DFT)

1 1
2 ( )
2 ( )
0 0
0 0
1
( , ) ( , ) ( , )
m n
M N
i u v
i ux vy
M N
m n
F u v f x y e dxdy f m n e
MN
π
π
∞ ∞
− −
− +
− +
= =
= = ⋅
∑∑
∫ ∫

(u=0~M-1, v=0~N-1)
Inv. D.F.T.
1 1
2 ( )
0 0
( , )
m n
M N
i u v
M N
m n
F u v e
π
− −
+ +
= =
= ⋅
∑∑
(m=0~M-1, n=0~N-1)

• f(m,n) ≜ Input Signal Amplitude (real or complex) at sample (m, n)
T ≜ Sampling interval
s f ≜ Sampling rate
1
( )
T
= , samples/second
M,N = Numbers of Spatial samples = Number of frequency samples (integer)

• Magnitude
1
2 2
2
( , ) ( , ) ( , ) F u v R u v I u v ( = +
¸ ¸

Phase Angle
1
( , )
( , ) tan
( , )
I u v
u v
R u v
φ

=
Power Spectrum
2
2 2
( , ) ( , ) ( , ) ( , ) P u v F u v R u v I u v = = +


1 1
0 0
1
(0, 0) ( , )
M N
m n
F f m n
MN
− −
= =
=
∑∑
⇒ Average of ( , ) f m n or D.C. component.
If ( , ) f x y is real, its F.T. is conjugate complex
( , ) ( , )
( , ) ( , )
F u v F u v
F u v F u v


= − −
= − −
⇒ Spectrum of D.F.T. is symmetric.

67
• Relationship between samples in the spatial and frequency domains.

1
u
M x
=
⋅ ∆

1
v
N y
=
⋅ ∆


• Translation
0 0
0 0
2 ( )
( , ) ( , )
u v
i x y
M N
f x y e F u u v v
π − ⋅ ⋅ + ⋅
⋅ ⇔ − −
0 0
0 0
2 ( )
( , ) ( , )
ux vy
i
M N
f x x y y F u v e
π − ⋅ +
− − ⇔

Case : when 0 0 ,
2 2
M N
u v = =
{ }
2 ( )
( )
2 2
cos ( ) ( 1)
x y
i
i x y x y
e e x y
π
π
π
⋅ +
+ +
= = + = −
( , )( 1) ( , )
2 2
x y
M N
f x y F u v
+
⇒ − ⇔ − −

And similarly,
( , ) ( , )( 1)
2 2
u v
M N
f x y F u v
+
− − ⇒ −

Rotation
cos
sin
x r
y r
θ
θ
= ⋅
= ⋅

cos
sin
u w
v w
φ
φ
= ⋅
= ⋅


Then, ( , ) f x y and ( , ) F u v become ( , ) f r θ and ( , ) F w φ
68
0 0 ( , ) ( , ) f r F w θ θ φ θ + ⇔ +

⇒ Rotating ( , ) f x y by an angle 0 θ rotates ( , ) F u v by the same angle, and vice versa.

Periodicity ( , ) ( , ) ( , ) ( , ) F u v F u M v F u v N F u M v N = + = + = + +
The inverse transform is also periodic.
( , ) ( , ) ( , ) ( , ) f x y f x M y f x y N f x M y N = + = + = + +

Separability
1 1
2 2
0 0
1 1
( , )
y x
M M
j v j u
N M
x y
F u v e e
M N
π π
− −
− −
= =
= ⋅
∑ ∑

1
2
0
1
( , )
x
M
j u
M
x
F u v e
M
π


=
=

, (where
1
2
0
1
( , )
y
M
j v
N
y
F u v e
N
π


=
=

)
( , ) ( , ) ( , ) f x y F x v F u v → →


• Convolution
1 1
0 0
1
( , ) ( , ) ( , ) ( , )
M N
m n
f x y h x y f m n h x m y n
MN
− −
= =
⊗ = ⋅ − −
∑∑


f h F H
f h F H
⊗ ⇒ ⋅
⋅ ⇒ ⊗
Pr ( . .) ove H W →
1-D
Row transform
1-D
column transform
69
8. Edge & Edge detection
I. Edges in Images
• An Edge is a boundary between two figures with relatively distinct grey-level
properties.
• Edges are curves in the image where rapid changes occur in brightness or in the
spatial derivatives of brightness.
• Change of brightness can be occurred at
• Change of surface orientation
• Different surfaces
• One object occludes another
• A boundary between light and shadow falling on a single surface
II. Differential Operators

Figure 8—1
70

Figure 8—2
sin cos 0 x y θ θ ρ − + =
( ) ( )
1 2 1
E , B B -B u( sin cos ) x y x y θ θ ρ = + − +
If t > 0 B
1
t < 0 B
2

t = 0 ½(B
1
+ B
2
)
( ) ( )
( ) 1
t
u t x dx
x dx
δ
δ
−∞

−∞
=
=




Figure 8—3
) cos sin ( ) ( cos
) cos sin ( ) ( sin
2 1
2 1
ρ θ θ δ θ
ρ θ θ δ θ
+ − − − =


+ − − =


y x B B
y
E
y x B B
x
E


71
squared gradient
2
2 1
2
2
)) cos sin ( ) (( ρ θ θ δ + − − =
|
|
¹
|

\
|


+ |
¹
|

\
|


y x B B
y
E
x
E

Laplacian of E(x,y)
2 2
2
2 2
2
2
1 2 2
2
2
1 2 2
2 '
1 2
sin ( ) '( sin cos )
cos ( ) '( sin cos )
( ) ( sin cos )
xx
yy
E E
E
x y
E
E B B x y
x
E
E B B a y
y
E B B x y
θ δ θ θ ρ
θ δ θ θ ρ
δ θ θ ρ
∂ ∂
∇ = +
∂ ∂

= = − − +


= = − − +

∇ = − − +


III. Discrete Approximations
• Discrete form of Laplacian
E
x
= E( i+1 , j ) – E( i , j )
E
y
= E( i , j+1 ) – E( i , j )

E
xx
= E( i – 1 , j ) – 2E( i , j ) + E( i + 1 , j )
E
xx
+ E
yy
= (E
i-1 j
+ E
ij-1
+ E
i-1 j
+ E
i j+1
) – 4E
ij



72
IV. 8.4 Local operators and noise
bb
nn
H
φ
φ
ρ
+
⋅ − =
1
1
2 '
-------------ⓐ
ρ
2
= u
2
+ v
2


suppose that
2
2
2
& N
S
nn bb
= = φ
ρ
φ
Then
2 2 2
2 2
'
N S
S
H
ρ
ρ
+
− = - ⓐ
at low frequency -ρ
2
: Laplacian operation
at high frequency -
2
2
N
S
<< ∞ small gain Noise can be reduced !
Derivation of ⓐ

Figure 8—4
''
' '' 2

1 1
( )
1
( )
i bb nb
ii bb bn nb nn
bb
bb nn
nn
bb
H
SNR
H H
α
ϕ ϕ ϕ
ϕ ϕ ϕ ϕ ϕ
ϕ
ϕ ϕ
ϕ
ϕ
+
= =
+ + +
=
+
= =
+
= ⋅ ℑ ∇

73
For example, if the optimal filter is a gaussian

Figure 8—5
2
2 2
2
2
2
1
) , (
σ
πσ
y x
e y x h
+

=
) (
2
1
2 2 2
) 0 , (
v u
e u H
+ −
=
σ



74
9. Lightness & color

Figure 9—1
b´(x,y) = e´(x,y) - r´(x,y)
• Note that r´(x,y) is constant within a patch and sharp discontinuities at edges
between patches and e´(x,y) varies smoothly

Figure 9—2
Task : separate r´(x,y) & e´(x,y)
• Take the logarithm of image brightness b´(x,y)
log b´(x,y) = log r´(x,y) +log e´(x,y)
b(x,y) = r(x,y) +e(x,y)

2
b(x,y) =∇
2
r(x,y) +∇
2
e(x,y)

Figure 9—3
75
• cut slow slope by a threshold

Figure 9—4
T(∇
2
b(x,y) = ∇
2
r(x,y)
|∇
2
b(x,y)| > T then t(x,y) = ∇
2
b(x,y) else t(x,y) = 0
• recover r(x,y)

Figure 9—5
∫ ∫
∇ =
= ∇
dxdy y x r y x l
y x t y x l
) , ( ) , ( (
) , ( ) , (
2
2

Fourier Transform :
) (
1
) (
) ( ) (
)] , ( [ )] , ( [
2
2
2
e T L
T L
y x t y x l
ρ
ρ
ρ ρ ρ
− = ⇒
= − ⇒
ℑ = ∇ ℑ

2
1
) ( ..
) ( ) (
ρ
ρ ρ
− =
⋅ =
p G where
T G

ρ
π
+ = ) log(
2
1
) ( r r g
IFT : I(x,y) = g(r) * t(x,y)
76

Figure 9—6
I(x,y) = r(x,y) = reflectance
e(x,y) = b(x,y) – I(x,y)


Figure 9—7



77
. . .
) , ( ) , ( ) , (
) , ( ) , ( ) , (
5 5 5
4 4 4
3 3 3
2 2 2
1 1 1
e r b
e r b
e r b
y x e y x r y x b
y x e y x r y x b
⋅ =
⋅ =
⋅ =
=
=

) (
74
37
:
2 1
2 2
1 1
2
1
1
e e
e r
e r
b
b
R ≅ = = - equation 9.1

52
26
2
1
=
r
r
- equation 9.2
3
2
3
2
5
2
5
2
2
95
38
112
28
:
r
r
b
b
r
r
b
b
R
= =
= =  


Least square error 11 variables 16 equations
0 ≤ reflectance ≤ 1
16
equations
78
10. Reflectance Map : photometric stereo
(using one camera)

I. Radiometry
irradiance
A
p
E
δ
δ
= geometric stereo (using one camera)
radiance
W A
p
L
δ δ
δ
2
=

solid angle = Area of a spherical surface patch / (radius of the sphere)
2


Figure 10—1
2
2
1
4
2
Solid angle 2
R
R
π
π

= =

Figure 10—2
79

Figure 10—3

2 2
1
) 1 , , (
ˆ
q p
q p
n
+ +
− −
=

II. Image Formation

Figure 10—4
80
solid angle =
2
) cos / (
cos
α
θ δ
z
O

solid angle =
2
) cos / (
cos
α
α
S
SI


2
cos
cos
|
¹
|

\
|
=
S
z
I
O
θ
α
δ
δ
--------- equation 10.2.1
The solid angle subtended by the lens

α
α π
α
α
α
π
3
2
2
2
cos
4 ) cos / (
cos
2
|
¹
|

\
|
=
|
¹
|

\
|
= Ω
z z


Thus the power of the light originating on the patch and passing through the lens is
θ δ δ cos Ω ⋅ = O L P
where L is the radiance of the surface in the direction toward the Lens

θ α
α π
δ
δ
δ
δ
θ α
α π
δ δ
cos cos
4
cos cos
4
3
2
3
2
|
¹
|

\
|
= =
|
¹
|

\
|
=
z I
O
L
I
P
E
z
O L p

where E is the irradiance of the image at the patch under consideration

substituting for I O δ δ , we finally obtain
81

α
α π
α
α π
4
2
4
2
cos
4
cos
4
|
¹
|

\
|
⋅ =
∝ ∴
|
¹
|

\
|
=
s
L E
L E
s
L E


III. Bidirectional reflectance distribution function (BRDF)

Figure 10—5

Figure 10—6
82
) , (
i i
E φ θ δ ; irradiance
The amount of light falling on the surface from the direction ) , (
i i
φ θ
) , (
e e L
φ θ δ : radiance

The brightness of the surface as seen from the direction
) , (
e e
φ θ
BRDF :
1
cos sin ) , ; , (
) , (
) , (
) , ; , (
2
0
=
=
∫ ∫

π
π
π
φ θ θ θ φ θ φ θ
φ θ δ
φ θ δ
φ θ φ θ
e e e e e e i i
i i
e e
e e i i
d d
E
L
f


IV. Extended Light sources

Figure 10—7
solid angle w δ
2
) tan (
.
ce dis
area The
= The area =
i i
δφ θδθ sin
83
(proof :
2
0
sin 2
i i i
sw
π
π
π
θ δθ δϕ π

= =
∫ ∫ ∫ ∫

) , (
i i
E φ θ : Radiance per unit solid angle coming from the direction ) , (
i i
φ θ
The radiance from the patch under consideration

i i i i i i i
E E δφ δθ θ φ θ δω φ θ sin ) , ( ) , ( =
Total irradiance of the surface is
2
0
0
( , ) sin cos
i i i i i i
E E d d
π
π
π
θ ϕ θ θ θ ϕ

=
∫ ∫

where
i
θ cos accounts for the foreshortening of the surface as seen from the direction
) , (
i i
φ θ
The radiance of the surface,
∫ ∫
= ) , ( ) , , ( ) , (
i i e e i i e e
E j f L φ θ φ θ φ θ φ θ

i i i i
d d φ θ θ θ cos sin

V. Surface Reflectance properties
• Ideal Lambertian surface
One that appears equally bright from all viewing directions and reflects all
incident light, absorbing none
84
BRDF ) , , (
e e i i
j f φ θ φ θ constant

2
0
2
0
( , , , ) sin cos 1
2 sin cos 1
1
1
i i
e e e e e e
e e e
f j d d
f d
f
f
π
π
π
π
θ φ θ ϕ θ θ θ ϕ
π θ θ θ
π
π

=
=
=
∴ =
∫ ∫



sine BRDF is constant for a Lambertian surface
)
1
(
1
cos sin ) , (
π π
φ θ θ θ φ θ
= ⋅ = ⋅ =
⋅ =
∫ ∫
f E E f
d d E f L
o o
i i i i i i


where the irradiance is E
o

• Ideal specular Reflector

Figure 10—8

85
2
0
( , ; , ) sin cos 1
sin cos 1
1
sin cos
i i e e e e e e
i i
i i
f d d
k
k
π
π
π
θ ϕ θ ϕ θ θ θ ϕ
θ θ
θ θ

=
⇒ =
⇒ =
∫ ∫


In this case,
i i
i e i e e e i i
j f
θ θ
π φ θ δ θ θ δ φ θ φ θ
cos sin
) ( ) ( ) , , ( − − − =


Determine the radiance of a specular reflecting surface under an extended source where
the irradiance is E
o

2
0
( ) ( )
( , )
sin cos
( , ) sin cos
( , )
( , )
e i e i
e e
i i
i i i i i i
e e
i i
L
E d d
E
E
π
π
π
δ θ θ δ ϕ ϕ π
θ ϕ
θ θ
θ ϕ θ θ θ ϕ
θ ϕ π
θ ϕ

− − −
=
⋅ ⋅
= −
=
∫ ∫


VI. Surface Brightness

Figure 10—9
86
i
E L θ
π
cos
1
=

•Light source : a “sky” of uniform radiance E
2
0
2
0
2
0
1
sin cos
sin cos
cos 2
[1 1]
2 2
i i i i
i i i i
i
L E d d
E
d d
E
E E
π
π
π
π
π
π
π
θ θ θ ϕ
π
θ θ θ ϕ
π
θ


= ⋅
=
− (
= = + =
(
¸ ¸
∫ ∫
∫ ∫

The radiance of the patch ≡The radiance of the source

VII. Surface orientation

Figure 10—10

87
T
y y y
T
x x x
y x
q r
p r
q p r r n
) , , 0 (
) , 0 , (
) 1 , , (
δ δ
δ δ
=
=
− − = × =


The unit surface normal nˆ
2 2
1
) 1 , , (
ˆ
q p
q p
n
n
n
T
+ +
− −
= =

Figure 10—11

T
v ) 1 , 0 , 0 ( ˆ ≡
assuming α is very small

2 2 2 2
2 2
1
1
) 1 , 0 , 0 (
1
) 1 , , (
ˆ ˆ cos
1
) 1 , , (
ˆ
q p q p
q p
v n
q p
q p
n
T
+ +
= ⋅
+ +
− −
= =
+ +
− −
=
θ




88
VIII. The reflectance Map

Figure 10—12
) , ( ˆ
) , ( ˆ
2 2
1 1
q p R s
q p R s





i
E L θ
π
cos
1
= for 0 ≥
i
θ

2 2 2 2 2 2 2 2
1 1
1
1
) 1 , , (
1
) 1 , , (
ˆ ˆ cos
s x
s s
s s
T
s s
i
q p q p
q q p p
q p
q p
q p
q p
s n
+ + + +
+ +
=
+ +
− −

+ +
− −
= ⋅ = θ
= R(p,q) ; Reflectance Map
c
q p q p
q q p p
q p R
s s
s s
=
+ + + +
+ +
=
2 2 2 2
1 1
1
) , ( - equation 10.9.1

Figure 10—13
89
2 2 2
1
2
1
2 2
2
2 2 2
1
2
1
1 1
1
1 1
1
) , (
1 1
1
) , (
s s
s s
s s
s s
q p q p
q q p p
q p R
q p q p
q q p p
q p R
+ + + +
+ +
=
+ + + +
+ +
=


IX. Shading in image

Consider a smoothly curved object the image of such an object will have spatial
variations in brightness due to the fact that surface patches with different orientations
appear with different brightness the variation of brightness is called shading

) , ( cos
cos
4
4
2
q p R L E
f
d
L E
i

|
|
¹
|

\
|
⋅ =
θ α α
α
π


after normalizing E(x,y) = R(p,q)
ρ : reflectance factor where 1 0 ≤ ≤ ρ

n s y x E where
n s p y x E
ˆ ˆ ) , ( , 1
) ˆ ˆ ( ) , (
⋅ = =
⋅ ⋅ =
ρ


Ex1) Consider a sphere with a Lambertian surface illuminated by a point source at
essentially the same place as the viewer in the case of cylinder
90

Figure 10—14

In this case,
2 2
1
) 1 , , (
ˆ
) 1 , 0 , 0 ( ˆ
) 1 , 0 , 0 ( ˆ
q p
q p
n
v
s
+ +
− −
=
=
=

From equation 10.9.1 - s n q p R ˆ ˆ ) , ( ⋅ =
2 2
2 2 2 2
1
1
1 1
1
q p
q P q p
q q p p
s s
s s
+ +
=
+ + + +
+ +
=

0
2
0
0
2
1
2 2 2
) (
) 2 ( )) ( (
2
1
z z
z z y
d
d
q
z z
x
x y x r
d
d
p
y
z
x
z


− = =

− = − + − = =


Finally, E(x,y) = R(p,q)
91
2
2 2
2
0
2 2
2 2
1
) (
1
1
1
1
r
y x
z z
y x
q p
+
− =

+
+
=
+ +
=

( x , y ) = ( 0 , 0 )
E( x, y) maximum = 0
x
2
+y
2
=r
2

E(x,y) minimum = 0

X. Photometric stereo

Figure 10—15

92
2
2
2
2
2 2
2 2
2 2
2
1
2
1
2 2
1 1
1 1
1 1
1
) , (
1 1
1
) , (
s s
s s
s s
s s
q P q p
q q p p
q p R E
q P q p
q q p p
q p R E
+ + + +
+ +
= =
+ + + +
+ +
= =


Ex2) R(p, q) : linear and independent
2
2 2
2
1
1 1
1
1
) , (
1
) , (
r
q q p p
q p R
r
q q p p
q p R
+ +
=
+ +
=

then
2 1 2 1
1 2
2
2 2 1
2
1
) 1 ( ) 1 (
p q q p
q r E q r E
p

− − −
=

2 1 2 1
2 1
2
1 1 1
2
2
) 1 ( ) 1 (
p q q p
p r E p r E
q

− − −
=

Ex3) R(p,q) : nonlinear

Figure 10—16
93
XI. Albedo : reflectance factor :
) ˆ ˆ ( n s E
i i
⋅ = ρ
where
2 2
1
) 1 , , (
ˆ
i i
T
i i
i
q p
q p
s
+ +
− −
= for i=1, 2, 3, …………

2 2
1
) 1 , , (
ˆ
q p
q p
n
T
+ +
− −
=

Figure 10—17

Unknown variables p, q, ρ
At least 3 equations



94
11. Reflectance Map : shape from shading
I. Recovering Shape from Shading

Figure 11—1
i. Picture
One more information => strong constraint smoothness neighboring patches of the
surface => similar orientation assumption of smoothness : strong constraint

ii. Linear reflectance map

Figure 11—2
95
2 2
0
2 2
0 0
sin cos tan
b a
b
b a
a
a
b
+
=
+
= = θ θ θ

Figure 11—3

: sin cos ) ( θ θ θ q p m + = directional derivative
)) , ( (
1
sin cos ) (
1
2 2 2 2
0 0 0
y x E f
b a b a
bq ap
q m

+
=
+
+
=
+ = ⇒ θ θ θ


S can be recovered from E(x,y) using above curve

Figure 11—4
96
δξ δ m z = (small change a long the characteristic curve)
)) , ( (
1
1
2 2
y x E f
b a
z
m

+
= =
δξ
δ

θ ξ ξ
θ ξ ξ
sin ) (
cos ) (
0
0
+ =
+ =
y y
x x where

0
0
1
0
2 2
0
( ) ( )
1
( ( , ))
z z m d
z f E x y d
a b
ξ
ξ
ξ θ ξ
ξ

= +
= +
+




iii. Rotationally symmetric reflectance maps

Figure 11—5


97
2 2
2 2
sin
cos
tan
q p
q
q p
p
p
q
s
s
s
+
=
+
=
=
θ
θ
θ


The slope
s s s
q p m θ θ θ sin cos ) ( + =

2 2
q p + =
)) , ( (
1
y x E f

=
The change in Z
δξ δξ θ δ
δξ δξ θ δ
δξ δξ δ
2 2
2 2
1 2 2
sin
cos
)) , ( (
q p
q
y
q p
p
x
y x E f q p m z
s
s
+
= =
+
= =
⋅ + = =


To simplify δz , we could take a step of length δξ
2 2
1
q p +
rather than δξ
δξ
δξ
δ
)) , ( (
) (
1
2 2
2 2 2 2
y x E f
q p
q p q p z

=
+ =
+ × + =

δx = pδξ - equation 11.1.1
δy = qδξ

98
We need to determine p&q at the new point in order to continue the solution
y x t r w y x E ate Differenti
q p f y x E
& , , ) , (
) ( ) , (
2 2
+ =

2 2
'
) (
q p S where
x
S
S f
x
S
S
E
x
E
E
x
+ =


=





=


=

y
z
q
x
z
p
x
q
q
x
p
p
x
s


=


=


+


=


, , 2 2

y x
z
q
x
z
p
∂ ∂

+


=
2
2
2
2 2

2 2
2
2( ) '( )
2( ) '( )
x
y
E pr qs f s
E ps qt f s
p p z z
p x x y
q y x x y
δ δ δ δ
= + |

= +
\
∂ ∂ ∂ ∂
= + = +
∂ ∂ ∂ ∂ ∂
- equation 11.1.2
y s x r δ δ + =
y t x s q δ δ δ + =

in our case δξ δ δξ δ q y p x = &
δξ δ
δξ δ
⋅ + =
⋅ + =
) (
) (
qt ps q
qs pr p

From (2)
δξ δ
δξ δ
' 2
' 2
f
E
q
f
E
p
y
x
=
=


As δξ → 0 , we obtain the differential equations
99

\
|
= =
= + = = =

' 2
,
' 2
)) , ( ( , ,
1 2 2
f
E
q
f
E
p
y x E f q p z q y p x
y
x
ɺ ɺ
ɺ ɺ ɺ


(x, y, z, p & q) solvable
5 Differential Eqs
one more time by differentiating
)) , ( (
' 2
,
' 2
. . , ,
1
y x E f z
f
E
q y
f
E
p x
t r w q y p x
y
x

=
= = = =
= =
ɺ
ɺ ɺ ɺ ɺ ɺ ɺ
ɺ ɺ ξ


iv. General Case

R(p,q) = f(ap + bq)
R(p,q) = f(P
2
+ q
2
)
R(p,q) = arbitarary


y s x r
y
y x
z
x
x
z
p
y q x p z
δ δ
δ δ δ
δ δ δ
+ =
∂ ∂

+


=
+ =
2
2
2
- equation 11.1.3

y t x s
y
y
z
x
y x
z
q
δ δ
δ δ δ
+ =


+
∂ ∂

=
2
2 2


100
|
|
¹
|

\
|
⋅ =
|
|
¹
|

\
|
|
|
¹
|

\
|
=
|
|
¹
|

\
|
y
x
H
y
x
t s
s r
q
p
δ
δ
δ
δ
δ
δ


where H is the Hessian Matrix
2
2
2
2
y
z
x
z
t r


+


= + : Laplacian
Differentiating E(x,y) w.r.t x & y

x
q
q
E
x
p
p
E
x
E
E
x





+





=


=

S R r R
S
y x
z
R
x
z
R
q p
q p
⋅ + ⋅ =
∂ ∂

⋅ +


⋅ =
2
2
2

t R S R E
q p y
⋅ + ⋅ =
|
|
¹
|

\
|
⋅ =
|
|
¹
|

\
|
q
p
y
x
R
R
H
E
E
- equation 11.1.4
from equation 11.1.3 & equation 11.1.4
Let δξ
δ
δ

|
|
¹
|

\
|
=
|
|
¹
|

\
|
q
p
R
R
y
x


q
p
R y
R x
=
=
ɺ
ɺ

Then δξ
δ
δ
δ
δ
|
|
¹
|

\
|
=
|
|
¹
|

\
|
⋅ =
|
|
¹
|

\
|
⋅ =
|
|
¹
|

\
|
y
x
q
p
E
E
R
R
H
y
x
H
q
p

101

\
|
= =
+ = + = ⋅ + ⋅ =
+ = =
=
y x
q p
q p q
p
E q E p
qR pR y q x p z
y
q
x
p
z
qR pR z R y
R x
ɺ ɺ
ɺ ɺ ɺ
ɺ ɺ
ɺ
,
,
δξ
δ
δξ
δ
δξ
δ

5unknown variables : x,y,z, p & q

Figure 11—6

II. 11.3 Singular points

Figure 11—7
) , ( ) , (
0 0
q p R q p R < for all ) , ( ) , (
0 0
q p q p ≠

Figure 11—8
102
0 , 0
0 , 0
= = = =
= =
q p
q p
R y R x
R R
ɺ ɺ

) (
2
1
) , (
2 2
q p q p R + = - equation 11.3.1
(p,q) = (0,0) – singular point
) 2 (
2
1
2 2
0
cy bxy ax z z + + + =

Thus by ax
dx
dz
p + = =
& cy bx
dy
dz
q + = =
substituting p & q in equation 11.3.1
] ) ( ) [(
2
1
) (
2
1
) , (
2 2 2 2
cy bx by ac q p q p R + + + = + =

2 2 2 2 2 2
) (
2
1
) ( ) (
2
1
y c b bxy c a x b a + + + + + =

The brightness gradient is
y c b bx c a E
by c a x b a E
y
x
) ( ) (
) ( ) (
2 2
2 2
+ + + =
+ + + =


Differentiating again,
(
(
(
(
¸
(

¸

+ =
+ =
+ =
2 2
2 2
) (
c b E
b c a E
b a E
yy
xy
xx
3 equations, 3 unknowns
103
III. Stereo graphic projection

Figure 11—9

¸

= +
+
=
+
=
+
2 2 2
2 2
2 2
) (
2
R b a
R
q p
a
b
R
q f
a R
b

solve the above equations

2 2 2 2
1 1
2
,
1 1
2
q p
q
g
q p
p
f
+ + +
=
+ + +
=
conversely
2 2 2 2
4
4
,
4
4
g f
g
q
g f
f
p
− −
=
− −
=


104
IV. 11.7 Relaxation methods
characteristic strip expansion


E(x , y) = R(p , q)
= R
s
(f,g)
Minimize

∫ ∫
+ + + = dxdy gy gx fy f e
s
) ( ) (
2 2 2 2

A low, Minimize

∫ ∫
− = dxdy g f R y x E e
s i
2
)) , ( ) , ( (
overall, minimize
i s
e e λ +
if brightness measurement are very accurate,
λis large
if brightness measurement are very noisy,
λis small

105
minimize
∫ ∫
dxdy g g f f g f F
y x y x I
) , , , , , (
= It’s a problem in the calculus of variations.(see appendix)
The corresponding Euler equations are


0
0
=






=






y x
y x
g g g
f f f
F
y
F
x
F
F
y
F
x
F

where F
f
is the partial derivative of F with respect to F
2 2 2 2 2
)) , ( ) , ( ( ) ( ) ( g f R y x E g g f f F
s y x y x
⋅ + + + + = λ
The Euler equations for this problem yield

yy f
xx x f
s
s f
f F
y
f f
x
F
x
f
R
g f R y x E F
y
x
2
2 ) 2 (
)) , ( ) , ( ( 2
=


= ⋅


=




− − = λ


f f f yy xx
F F
y
F
x
f f
y x
=


+


= + ) ( 2

f
R
g f R y x E
s
s


− − = )) , ( ) , ( ( λ

g
R
g f R y x E g
s
s


− − = ∇ )) , ( ) , ( (
2
λ (2 equations, 2 unknowns(f,g))
Application to photometric stereo consider n images,
106

∫ ∫

∫ ∫
=
− + + + − =
n
i
i i I i y x y I
dxdy g f R y x E dxdy g g f fx e
1
2 2 2 2 2
)) , ( ) , ( ( )) ( ) (( λ
where E
i
(x,y) is the brightness measured in the i th image and R
i
is the corresponding
reflectance map
The Euler equations for this problem yield



=
=


− − = ∇


− − = ∇
n
i
i
i i i
i
i i
n
i
i
g
R
g f R y x E g
f
R
g f R y x E f
1
2
1
2
)) , ( ) , ( (
)) , ( ) , ( (
λ
λ


V. 11.8 Recovering Depth from a needle diagram

Figure 11—10



107

Figure 11—11

0 0
( , )
0 0
( , )
( , ) ( , ) ( )
x y
x y
z x y z x y pdx qdy = + +



Minimize the error
dxdy p z p z
I
p x
) ) ( ) ((
2 2
− + −
∫ ∫

where p and q are the given estimates of the components of the gradient while
y x
z z &
are the partial derivatives of the best-fit surface
Minimize
∫ ∫
dxdy z z z F
y x I
) , , (

The Euler equations is
0 =






y x
z z z
F
y
F
x
F

108

so that from
2 2
) ( ) ( q z p z F
y x
− + − =
0
) ( 2 ) ( 2
=



⋅ + −


⋅ =


+


=
q z
y
p z
x
F
y
F
x
F
y x
z z z
y x

) (
0 ) ( 2 ) ( 2
2
y x
y x yy xx
y yy x xx
q p z
q p z z
q z p z
+ = ∇
+ = +
= − + −


109
12. Motion field & Optical flow

Figure 12—1

• Motion field : a purely geometric concept
• Optical flow : Motion of brightness patterns
observed when a camera is moving relative to the objects being imaged

I. Motion Field

Figure 12—2
110

0
0
0
0
, &
ˆ '
i
r r
i
t t
i
v v
r r
f r z
δ δ
δ δ
= =
=

- equation 12.1.1

Differentiate - equation 12.1.1
2
0
0 0 0 0
2
0
0
0 0
0
) ˆ (
) ˆ ( ) ˆ (
) ˆ (
) ˆ ( ) ˆ (
'
1
z r
r z v v z r
z r
r z
dt
dr
dt
dr
z r
dt
r d
f
i

⋅ − ⋅
=

⋅ − ⋅
= ⋅

2
0
0 0
) ˆ (
ˆ ) (
'
1
z r
z v r
v
f
i

× ×
= ⋅ - equation 12.1.2

II. Optical flow

Figure 12—3
111
Ex1)

Figure 12—4

E(x, y, t) ≡ E(x + uδt , y + vδt , t + δt) where u(x, y) & v(x, y) are the x & y components
of the optical flow vector

For a small time interval t δ , if brightness varies smoothly with x, y, &t,
) , , ( ) , , ( ) , , ( t y x E
t
E
t
y
E
y
x
E
x t y x E t t t v y t u x E ≡ + + + + = + + + ρ
α
α
δ
α
α
δ
α
α
δ δ δ δ
where ρ contains second and higher order terms in δx , δy , and δt

0 = ⋅


+ ⋅


+ ⋅


t
t
t
E
t
y
y
E
t
x
x
E
δ
δ
δ
δ
δ
δ

0 = + +
t y x
E v E u E (Optical flow constraint equation)
u, v : two variables
one equation :
⇒ one more constraint needed

III. Smoothness of the optical flow
Usually, the motion field varies smoothly minimize a measure of departure from
smoothness
112
∫ ∫
+ + + = dxdy vy vx uy ux e
s
)) ( ) ((
2 2 2 2

Also, minimize the error in the optical flow constraint eq.
∫ ∫ ∫ ∫
+ + = − =
+ +
y dxd E v E u E dxdy t y x E t y x E e
t y x att t at c
2 2
) ( ] ) , , ( ) , , ( [
δ

Hence, minimize
c s
e e λ + ,
Minimize
c s
e e λ +
Where
∫ ∫
+ + + = dxdy vy vx uy ux e
s
)) ( ) ((
2 2 2 2


∫ ∫
+ + = dxdy E v E u E e
t y x c
2
) (
λ: a parameter that weights the error in the image motion equation relative to the
departure
from smoothness
λ=> Large
=> small
Minimize
∫ ∫
dxdy v v u u v u F
y x y x
) , , , , , (

2 2 2 2 2
) ( ) ( ) (
t y x y x Y X
E v E u E v v u u F + + + + + + = λ
Euler equations are

0
0
=






=






y x
y x
v v v
u u u
F
y
F
x
F
F
y
F
x
F


113
yy y u
xx x u
x t y x u
u u
x
F
y
u u
x
F
x
E E v E u E F
y
x
2 ) 2 (
2 ) 2 (
) ( 2
=


=


=


=


+ + = λ


u u u E E v E u E
yy xx x t y x
2
) ( ∇ = + = + + λ
Similarly,
0
) (
2
= + +
∇ = + +
t y x
y t y x
E v E u E
v E E v E u E λ


IV. Boundary conditions
*Natural boundary conditions (see appendix)
ds
dx
F
ds
dy
F
ds
dx
F
ds
dy
F
y x
y x
v v
u u
=
=

where S denotes arc length along the boundary curve
T
ds
dx
ds
dy
n |
¹
|

\
|
− = , ˆ is a unit vector
perpendicular to the boundary
Rewriting the above conditions.
0 ˆ ) , (
0 ˆ ) , (
= ⋅
= ⋅
n F F
n F F
y x
y x
v v
u u

114
in our case,
2 2 2 2 2
) ( ) ( ) (
t y x y x y x
E v E u E v v u u F + + + + + + = λ
0 ˆ ) , (
0 ˆ ) , (
= ⋅
= ⋅
n v v
n u u
y x
y x


V. Discrete case
From
c s
e e e λ + =
dxdy E v E u E dxdy v v u u
t y x y x y x
2 2 2 2 2
) ( )) ( ) (( + + + + + + =
∫ ∫ ∫ ∫
λ
) (
ij
i j
ij
c s e λ + =
∑∑


) ) ( ) ( ) ( ) ((
4
1
2
, 1 ,
2
, , 1
2
, 1 ,
2
, , 1 , j i j i j i j i j i j i j i j i j i
v v v v u u u u s − + − + − + − =
+ + + +

2
) (
t ij y ij x ij
E v E u E c + + =
Differentiating w.r.t
kl kl
v u & yields

,
) ( 2 ) ( 2
, ) ( 2 ) ( 2
y t kl y kl x kl kl
kl
x t kl y kl x kl kl
kl
E E v E u E v v
v
e
E E v E u E u u
u
e
+ + + − =


+ + + − =


λ
λ

where v u & are average of u & v

115
0
0
=


=


kl
kl
v
e
u
e
solve this

t y kl x kl x y kl y x
t x kl y x kl y kl y x
E E v E u E E v E E
E E v E E u E u E E
λ λ λ λ
λ λ λ λ
− + + − = + +
− − + + = + +
) 1 ( )) ( 1 (
, ) 1 ( )) ( 1 (
2 2 2
2 2 2



Iterative scheme is
.
) ( 1
,
) ( 1
2 2
1
2 2
1
y
y x
t
n
kl y
n
kl x n
kl
n
kl
x
y x
t
n
kl y
n
kl x n
kl
n
kl
E
E E
E v E u E
v v
E
E E
E v E u E
u u
+ +
+ +
− =
+ +
+ +
− =
+
+
λ
λ


Figure 12—5
116
13. Photogrammetry & Stereo
I. Disparity between the two images.

Figure 13—1
Disparity : D x x
r l
= −
' '

) 3 (
) 2 ( ) (
2
) 1 ( ) (
2
' '
'
'
− = =
− ←

=
− ←
+
=
z
y
f
y
f
y
line red
z
b
x
f
x
line blue
z
b
x
f
x
r l
r
l

solve (1) (2) & (3)
f
z
b
x x
r l
= − ) (
' '
: disparity
' ' ' '
' '
' '
' '
,
2 / ) (
,
2 / ) (
r l r l
r l
r l
r l
x x
bf
z
x x
y y
b y
x x
x x
b x

=


=


=
117
D x x
z
r l
1 1
' '
=



II. Photogrammetry

Figure 13—2

• absolute orientation
) , , ( known r r
r l


(
(
(
¸
(

¸

r
r
r
z
y
x

0
r r R r
l r
+ =

(
(
(
¸
(

¸

33 32 31
23 22 21
13 12 11
r r r
r r r
r r r
component
rotaional

(
(
(
¸
(

¸

34
24
14
r
r
r
component
nal translatio


x
r
118
where R is a 3*3 orthonormal matrix
I R R
T
= ⋅

r l l l
r l l l
r l l l
z r z r y r x r
y r z r y r x r
x r z r y r x r
= + + +
= + + +
= + + +
34 33 32 31
24 23 22 21
14 13 12 11


3 equations => 12 unknowns

• Orthonormality
I R R
T
= ⋅
1
1
1
2
33
2
32
2
31
2
23
2
22
2
21
2
13
2
12
2
11
= + +
= + +
= + +
r r r
r r r
r r r


0
0
0
13 33 12 32 11 31
33 23 32 22 31 21
23 13 22 12 21 11
= + +
= + +
= + +
r r r r r r
r r r r r r
r r r r r r



Figure 13—3

119
1 2 1 2 r r l l
r r r r − = −
3+7 = 10 equations
12 unknown variables

P&Q : 10+3 = 13 equations

Figure 13—4

0
r r R r
l r
+ ⋅ = - equation 13.3.1

(
(
(
¸
(

¸

=
(
(
(
¸
(

¸

=
34
24
14
0
33 32 31
23 22 21
13 12 11
r
r
r
r
r r r
r r r
r r r
R

III. Relative orientation
r l
r r , ; not known
r l
z z , unknowns
The projection points ) , ( & ) , (
' ' ' '
r r l l
y x y x known determine the ratios of x and y to z
using
120
r
r r
r
r r
l
l l
l
l l
z
y
f
y
z
x
f
x
z
y
f
y
z
x
f
x
= =
= =
' '
' '
&
&
- - equation 13.4.1

12 + 2*(n)
substitute B into A

l
r
r l
l l
l
l
r l l l
z
f
f
z
x r z r
f
z
y r
f
z
x r
z
f
x r z r y r x r
) ( ) ' (
'
14 13 12
'
11
14 13 12 11
= + + +
= + + +



= = + + +
= = + + +
= = + + +
l
r
l
l l
r
l
r
r
l
l l
r
l
r
r
l
l l
z
z
f
z
f
r f r y r x r
s
y
z
z
y
z
f
r f r y r x r
s
x
z
z
x
z
f
r f r y r x r
34 33
'
32
'
31
'
24 23
'
22
'
21
'
14 13
'
12
'
11
- equation 13.3.2

12 + 2n ; # of unknowns for n points
7 + 3n ; # of equations
7 from orthonormality & distance length

121
I R R
T
= ⋅
(
(
(
¸
(

¸

1 0 0
0 1 0
0 0 1

j i if r r
r
j i
i
≠ =
=
0
1
2

∴12 + 2n = 7 + 3n
n = 5
n ≥5

IV. 13.5 Using a known relative orientation
s
f
z
l
= ; constant s x
f
z
x x
l
l
l l
⋅ = =
/ '
s y y
l l
⋅ =
'
s f z
l
⋅ =
where s is a scaling factor
from equation 13.3.2
w cs r s f r y r x r z
v bs r s f r y r x r y
u as r s f r y r x r x
l l r
l l r
l l r
+ = + + + =
+ = + + + =
+ = + + + =
34 33
'
32
'
31
24 23
'
22
'
21
14 13
'
12
'
11
) (
) (
) (

w cs
v bs
z
y
f
y
w cs
u as
z
x
f
x
r
r r
r
r r
+
+
= =
+
+
= =
'
'





122
V. Computing depth
Once IR , ) , ( ), , ( ,
' ' ' '
0 r r l l
y x y x r known, The depth ) , (
r l
z z can be computed
as follows

Figure 13—5

From equation 13.3.2, multiply by
f
z
l

r l
l l
r
r
l
l l
r
r
l
l l
z r z r
f
y
r
f
x
r
z
f
y
r z r
f
y
r
f
x
r
z
f
x
r z r
f
y
r
f
x
r
= + + +
= + + +
= + + +
34 33
'
32
'
31
'
24 23
'
22
'
21
'
14 13
'
12
'
11
) (
) (
) (


From any two equations
Compute
r l
z z &
Then compute
r
r r T
r r r l
l
l l T
l l l l
z
f
y
f
x
z y x r
z
f
y
f
x
z y x r
|
|
¹
|

\
|
= =
|
|
¹
|

\
|
= =
1 , , ) , , (
1 , , ) , , (
' '
' '


123
VI. 13.7 exterior orientation

Figure 13—6
0
r r R r
a c
+ ⋅ =

c a a a
c a a a
c a a a
z r z r y r x r
y r z r y r x r
x r z r y r x r
= + + +
= + + +
= + + +
34 33 32 31
24 23 22 21
14 13 12 11


I R R
T
= ⋅ (orthonormality)

c
c
c
c
z
y
f
y
z
x
f
x
= =
'
&
'

34 33 32 31
24 23 22 21
34 33 32 31
14 13 12 11
'
'
r z r y r x r
r z r y r x r
z
y
f
y
r z r y r x r
r z r y r x r
z
x
f
x
a a a
a a a
c
c
a a a
a a a
c
c
+ + +
+ + +
= =
+ + +
+ + +
= =
- equation 13.7.1

# of equations = 2n +6
# of unknowns = 12
2n+6 = 12 => n=3 (at least we need three equations)
124
VII. Interior Orientation
• scaling error
• translation error
• skewing error
• shearing error

An affine transformation


23 22 21
13 12 11
) ( ) ( '
) ( ) ( '
a
z
y
a
z
x
a y
a
z
y
a
z
x
a x
c
c
c
c
c
c
c
c
+ + =
+ + =


from equation 13.7.1

34 33 32 31
24 23 22 21
34 33 32 31
14 13 12 11
'
'
s z s y s x s
s z s y s x s
f
y
s z s y s x s
s z s y s x s
f
x
a a a
a a a
a a a
a a a
+ + +
+ + +
=
+ + +
+ + +
=


ij
s
R : not orthonormal
I R R
ij ij
s
T
s
≠ ⋅ 12 unknowns
#of equations = 2n
∴ 2n=12 ; n=6
125
VIII. Finding conjugate points
i. Gray-Level Matching
• (reference figure 3-1)
z
b
x
f
x
z
b
x
f
x
r l 2
&
2
' ' −
=
+
=
at a matched point
) , ( ) , (
' ' ' '
r r r l l l
y x E y x E =
) (
) ' ,
2
( ) ' ,
2
( ~
' '
y y y
y
z
b
x
f E y
z
b
x
f E E
r l
r l
= ≅

=
+


Let
z
bf
y x d
z
x
f
x
= ≅ ) ' , ' ( &
'


) ' ), ' , ' (
2
1
' ( ) ' ), ' , ' (
2
1
' ( y y x d x E y y x d x E
r l
− = +
criterion : minimize
i s
e e e λ + =
where
∫ ∫
∇ ⋅ ∇ = ' ' ) (
2
dy dx d e
s


∫ ∫
− = ' ' ) (
2
dy dx E E e
r l i

(solution) using Euler equation
0
' '
' '
=






y x
d d
F
y
F
x
Fd
where
126
( )
]
' '
[
2
1
()] () [ 2
]
'
()
)
2
1
( )
2
1
(
'
()
()][ () [ 2
)] ' ), ' , ' (
2
1
' ( ) ' ), ' , ' (
2
1
' ( [ ) (
2 2
x
E
x
E
E E
x
E
x
E
E E E Fd
y y x d x E y y x d x E d F
r l
r l
r l
r l
r l


+


− =


− −


− − =
− − + + ∇ ⋅ ∇ =
λ
λ
λ


]
'
()
'
()
[
2
1
()] () [ ) (
) ( 2
' '
2 2
2 2
' '
x
E
x
E
E E d
d F
y
F
x
r l
r l
dy dx


+


− = ∇ ∇
∇ ∇ =


+


λ ∵

127
14. Pattern classification
Reference – “ Pattern Classification & Scene Analysis”
Dudda & Hart (Wiley Interscience pub. C)

1. Bayes Decision Theory
Fundamental statistical approach to the problem of pattern classification
• Maximum likelihood estimation : parameters fixed but unknown
• Baysian estimation : parameters, random variables having some known a prior
distribution

(A) Maximum likelihood estimation

Figure 14—1

Given ; 1. Model of system
2. pata : outcome of some type of probabilistic experiment.
Goal : Estimate unknown model parameters from data
Let n i
i
. ,......... 2 , 1 , = λ be n I.I.D. observations on a random variable X drawn from
) : ( θ x f
x

128
The joint lpdf of
n
X X X ,........ ,
2 1
is
1 x
f

Where ) ; .. ,......... (
1
θ
n
x x L is called the likelihood function (LF)
The maximum likelihood estimation (MLE) θ
ˆ
is that value that maximizes the LF, that
is
) ; .. ,......... ( )
ˆ
; .. ,......... (
1 1
θ θ
n n
x x L x x L ≥

Ex1) Assume ) , ( ;
2
σ µ N X where σis known. Compute the MLE of the mean μ
<solution> The LF for h realizations of X is on a
|
¹
|

\
|
− −
|
|
¹
|

\
|
=

=
h
i
i
n
x X L
0
2
2
2
) (
2
1
exp
2
1
) ; ( µ
σ
πσ
µ

since the log function is monotonic, the maximum of L(x ; μ) is also that of log L(x ; μ)

Hence

=
− − − =
n
i
i
x
n
x L
1
2
2
2
) (
2
1
) 2 log(
2
) ; ( log µ
σ
πσ µ
Set 0
) ; ( log
=


u
x L µ

This yields

=
= −

n
i
i
x
0
2
0 ) (
1
µ

=
=
n
i
i
x
n
1
1
ˆ µ

Ex2) Consider the normal p.d.f.,
129

=
− − =
n
i
x
x x f
1
2
2
2
2
) ) (
2
1
exp(
2
1
) , , ( µ
σ
πσ
σ µ
The log likelihood function is


=
− − − − =
n
i
i
x n
n
j x x L
1
2
2
2
2 1
) (
2
1
log 2 log
2
) , ,....... ( log µ
σ
σ π σ µ
Now set 0 , 0 =


=


σ µ
L L

Obtain the simultaneous equations



=
=
= − + −
= −
n
i
i
n
i
i
x
n
x
1
2
3
1
0 ) (
1
0 ) (
µ
σ σ
µ


Hence

=
=
n
i
i
X
n
1
1
ˆ µ


=
− =
n
i
i
x
n
1
2 2
) ˆ (
1
ˆ µ σ

(B) Baysian Estimation
Suppose that an classifier to separate two kinds of lumber, ash and birch


Figure 14—2

130
A prior : • probability p(w
1
) : the next piece is ash
• probability p(w
2
) : the next piece is birch

Decision Rule : Decide w
1
if p(w
1
) > p(w
2
)
Decide w
2
otherwise
Bay’s Rule ;


) (
) ( ) | (
) | (
x p
w p w x p
X w p
j j
j

=
where

=
⋅ =
r
j
j j
w p w x p x p
1
) ( ) | ( ) (

Bayes Decision Rule for minimizing the probability of error ;
Decide w
1
if p(w
1
|x) > p(w
2
|x)
Decide w
2
otherwise

Decide w
1
if P(x| w
1
)p(w
1
) > p(x| w
2
)p(w
2
)
Decide w
2
otherwise

If equally likely, p(w
1
)=p(w
2
), then
131
Decide w
1
if p(x| w
1
) > p(x| w
2
)
Decide w
2
if otherwise

Figure 14—3

Probability error : p(error)
( ) ( , )
( | ) ( )
p error p error x dx
p error x p x dx

−∞

−∞
=
=



where p(error|x)

\
|
1 2
2 1
) | (
) | (
w decide we if x w p
w decide we if x w p


Figure 14—4
132
(B-1) Generalized Form (for Bayes classifier)
Let } .......... {
1 s
w w = Ω be the finite set of S states of nature and ) ........ {
1 a
A α α = be the
finite set of possible actions.


Let ) | (
j i
w α λ be the loss incurred for taking action
i
α when the state of nature is
j
w
Let the feature vector X be a d-component vector-valued r.v
Then
) (
) ( ) | (
) | (
x p
w p w x p
x w p
j j
j
=
Where

=
=
s
j
j j
w p w x p X p
1
) ( ) | ( ) (
The expected loss associated with taking action
i
α is
) | ( ) | ( ) | (
1

=
⋅ =
s
j
j j i i
X w p w X R α λ α

Target: to find a Bayes decision rule against p(w
j
) that minimizes the overall risk
The overall risk is given by


⋅ = x d x p x x R R ) ( ) | ) ( (α
clearly, if ) (x α is chosen so that ) | ) ( ( x x R α is as small as possible, for every X , then
the overall risk will be minimized Bayes decision rule for the general form; to minimize
the overall risk.
133
1.Compute the conditional risk

=
⋅ =
s
j
j j i i
X w p w X R
1
) | ( ) | ( ) | ( α λ α
for i=1,………..a


2.Select the action
i
α for which ) | ( X R
i
α is minimum
3.Obtain the resulting minimum overall risk called the Bayes risk
B.2 Two category classification
Let ) | (
j i ij
w α λ λ = the loss incurred for deciding w
i
when the true state of nature is w
j
.
) | ( ) | ( ) | (
) | ( ) | ( ) | (
2 22 1 21 2
2 12 1 11 1
X w p X w p X R
X w p X w p X R
λ λ α
λ λ α
+ =
+ =
- equation 14.1

The decision rule :
Decide w
1
if ) | ( ) | (
2 1
X R X R α α < - equation 14.2
Decide w
2
otherwise


If we substitute 1 into 2
We have
) | ( ) ( ) | ( ) (
) | ( ) | ( ) | ( ) | (
1 11 21 2 22 12
2 22 1 21 2 12 1 11
X w p X w p
X w p X w p X w p X w p
λ λ λ λ
λ λ λ λ
− < − ⇒
+ < +

Hence likelyhood ratio
134

Figure 14—5

<Minimum – error – rate classification>
Symmetrical on zero-one loss function
The loss ) | (
j i
w α λ = 0 i=j for i,j=1,……c
1 i≠j
all errors are equally costly
The conditional risk is

=
⋅ =
c
j
j j i i
X w p w X R
1
) | ( ) | ( ) | ( α λ α

) | ( 1
) | (
X w p
X w p
i
c
i j
j
− =
=




To minimize the average prob of error, we should select the i that maximizes the a
posterior prob
) | ( X w p
i
In other words, for minimum error rate j
Decide
i
w if ) | ( ) | ( X w p X w p
j i
> for all , i j ≠
B-3 The multicategory classification
135
A pattern classifier

Figure 14—6

), ( X g
i
for i=1,………c ; discriminant function
The classification is said to assign a feature vector X to class
j
w
If ) ( ) ( X g X g
j i
> for all i j ≠
) | ( ) ( X w p X g
i i
= ; a posterior prob


=
=
c
j
j j
j i
w p w X p
w p w X p
1
) ( ) | (
) ( ) | (
by Bayes rule

) ( log ) | ( log
) ( ) | (
i i
i i
w p w X p
w p w X p
+ ⇒



Decision Rule
1. Divide the feature space into C division regions, R
1
,………..R
c

2. If ) ( ) ( X g X g
j i
> for all j≠i then X is in R
I

136
In other words, assign X to w
I


Ex) three category classification

Probability error calculation
P(error X
o
) ) ( ) (
0 0
X P X error P ⋅ =
) ( ) ( ) ( ) (
3 3 0 2 2 0
w P w x P w P w X P + =
Total error
P(error)


∞ −
= dx x error P ) , (

Sign up to vote on this title
UsefulNot useful