by to related
image in induces , velocity has point Object
z r
r r
r r
r
v
r
v
v v
=
= =
f
dt
d
dt
d
P
i
i
i
i
2 Cases Where this Assumption
Clearly is not Valid
(a) (b)
(a) A smooth sphere is rotating
under constant illumination.
Thus the optical flow field
is zero, but the motion field
is not.
(b) A fixed sphere is
illuminated by a moving
sourcethe shading of the
image changes. Thus the
motion field is zero, but the
optical flow field is not.
What is Meant by Apparent
Motion of Brightness Pattern?
The apparent motion of brightness patterns is an awkward concept. It is not
easy to decide which point P' on a contour C' of constant brightness in the
second image corresponds to a particular point P on the corresponding
contour C in the first image.
The aperture problem
Aperture Problem
(a) Line feature observed through a small aperture at time t.
(b) At time t+ot the feature has moved to a new position. It is not possible
to determine exactly where each point has moved. From local image
measurements only the flow component perpendicular to the line
feature can be computed.
Normal flow: Component of flow perpendicular to line feature.
(a) (b)
Brightness Constancy Equation
Let P be a moving point in 3D:
At time t, P has coords (X(t),Y(t),Z(t))
Let p=(x(t),y(t)) be the coords. of its image at
time t.
Let E(x(t),y(t),t) be the brightness at p at time t.
Brightness Constancy Assumption:
As P moves over time, E(x(t),y(t),t) remains
constant.
Brightness Constraint Equation
( ) ( ) ( ) flow. optical of components the , , , and irradiance the be , , Let y x v y x u t y x E
( ) ( )
expansion Taylor
, , , , t y x E t t t v y t u x E = + + + o o o
( ) ( )
0 limit taking and by dividing
, , , ,
= +
c
c
+
c
c
+
c
c
+
t t
t y x E e
t
E
t
y
E
y
x
E
x t y x E
o o
o o o
0
derivative total the of expansion the is which
0
=
=
c
c
+
c
c
+
c
c
dt
dE
t
E
dt
dy
y
E
dt
dx
x
E
short: 0 = + +
t y x
E v E u E
Brightness Constancy Equation
Taking derivative wrt time:
Brightness Constancy Equation
Let
(Frame spatial gradient)
(optical flow)
and
(derivative across frames)
Brightness Constancy Equation
Becomes:
v
x
v
y
r E
The OF is CONSTRAINED to be on a line !
E
t
/r E
Interpretation
Values of (u, v) satisfying
the constraint equation lie
on a straight line in velocity
space. A local measurement
only provides this constraint
line (aperture problem).
( ) ( )
( )
( )
T
y x
T
y x
t y x
n
E E
E E
E v u E E
,
Let
, ,
flow Normal
=
=
n
u
( )
T
y x
t y
y x
t x
n
E E
E E
E E
E E


.

\

+
= =
2 2 2
, n n u u
Optical flow equation
Barber Pole illusion
http://www.sandlotscience.com/Ambiguous/barberpole.htm
Solving the aperture problem
How to get more equations for a pixel?
Basic idea: impose additional constraints
most common is to assume that the flow field is smooth locally
one method: pretend the pixels neighbors have the same (u,v)
If we use a 5x5 window, that gives us 25 equations per pixel!
Constant flow
Prob: we have more equations than unknowns
The summations are over all pixels in the K x K
window
Solution: solve least squares problem
minimum least squares solution given by solution (in
d) of:
Taking a closer look at (A
T
A)
This is the same matrix we used for corner detection!
Taking a closer look at (A
T
A)
The matrix for corner detection:
is singular (not invertible) when det(A
T
A) = 0
One e.v. = 0 > no corner, just an edge
Two e.v. = 0 > no corner, homogeneous region
Aperture
Problem !
But det(A
T
A) = [
i
= 0 > one or both e.v. are 0
Edge
large gradients, all the same
large
1
, small
2
Low texture region
gradients have small magnitude
small
1
, small
2
High textured region
gradients are different, large magnitudes
large
1
, large
2
An improvement
NOTE:
The assumption of constant OF is more likely
to be wrong as we move away from the point of
interest (the center point of Q)
Use weights to control
the influence of the
points: the farther
from p, the less weight
Solving for v with weights:
Let W be a diagonal matrix with weights
Multiply both sides of Av = b by W:
W A v = W b
Multiply both sides of WAv = Wb by (WA)
T
:
A
T
WWA v = A
T
WWb
A
T
W
2
A is square (2x2):
(A
T
W
2
A)
1
exists if det(A
T
W
2
A) = 0
Assuming that (A
T
W
2
A)
1
does exists:
(A
T
W
2
A)
1
(A
T
W
2
A) v = (A
T
W
2
A)
1
A
T
W
2
b
v = (A
T
W
2
A)
1
A
T
W
2
b
Observation
This is a two image problem BUT
Can measure sensitivity by just looking at one of the
images!
This tells us which pixels are easy to track, which are
hard
very useful later on when we do feature tracking...
Revisiting the small motion assumption
Is this motion small enough?
Probably notits much larger than one pixel (2
nd
order
terms dominate)
How might we solve this problem?
Iterative Refinement
Iterative LukasKanade Algorithm
1. Estimate velocity at each pixel by solving LucasKanade
equations
2. Warp H towards I using the estimated flow field
 use image warping techniques
3. Repeat until convergence
Reduce the resolution!
image I image H
Gaussian pyramid of image H Gaussian pyramid of image I
image I image H
u=10 pixels
u=5 pixels
u=2.5 pixels
u=1.25 pixels
Coarsetofine optical flow
estimation
image I image J
Gaussian pyramid of image H Gaussian pyramid of image I
image I image H
Coarsetofine optical flow
estimation
run iterative LK
run iterative LK
warp & upsample
.
.
.
Optical flow result
Additional Constraints
Additional constraints are necessary to estimate optical flow, for
example, constraints on size of derivatives, or parametric models of the
velocity field.
Horn and Schunck (1981): global smoothness term
This approach is called regularization.
Solve by means of calculus of variation.
( ) ( ) smoothness from departure :
2 2 2 2
dy dx v v u u e
y x
D
y x s
+ + + =
}}
( ) equation constraint flow optical in error :
2
}}
+ + =
D
t y x c
dy dx E v E u E e
( )
( ) ( ) min
of gradient the denote , Let
2
2
2
2
2
V + V + + V
= V
}}
dy dx v u E E
A A A A
t
T
y x
u
Discrete implementation leads to
iterative equations
Geometric interpretation
( )
( )
line. constraint
the toward direction in the adjustment
an minus , , neighbors the of
values the of average the is point a at
, value new the flow, optical the
estimating for scheme iterative In the
v u
v u
( )
( )
y
y x
t
n
y
n
x
n n
x
y x
t
n
y
n
x
n n
E
E E
E v E u E
v v
E
E E
E v E u E
u u
v u v u
2 2
1
2 2
1
1
1
and of averages local denotes ,
+ +
+ +
=
+ +
+ +
=
+
+
O e
+ V
x
x u x x min , ,
2
2
t E t E W
t
( ) ( ) ( ) ( )  
2
2
2
2
2 2
2
2
2
2
2
v u E v E v E u E u
E
E E
x y y x x y y x t
T
V + V + +
+ V
+ + V
}}
o
o
o
u
Uras et al. (1988): Use constraints on secondorder derivatives
( )
( ) ( )
( ) ( )
( )
( )


.

\

= 
.

\

(
=
V
t E
t E
v
u
t E t E
t E t E
dt
t E d
ty
tx
yy xy
xy xx
,
,
, ,
, ,
0
,
x
x
x x
x x
x
( ) ( ) ( ) ( ) ( ) ( )
( ) ( ) ( )
( ) b u
x x b
x x x x
2
1
2
1
1 1
, ,
, , , diag , , , Denote
W A A W A
E E
W W W E E A
T T
T
n t t
T
n
T
n
=
=
= V V =
Classification of Optical Flow
Techniques
Gradientbased methods
Frequencydomain methods
Correlation methods
3 Computational Stages
1. Prefiltering or smoothing with
lowpass/bandpass filters to enhance signalto
noise ratio
2. Extraction of basic measurements (e.g.,
spatiotemporal derivatives, spatiotemporal
frequencies, local correlation surfaces)
3. Integration of these measurements, to produce 2D
image flow using smoothness assumptions
Energybased Methods
Adelson Berger (1985), Watson Ahumada (1985), Heeger (1988):
Fourier transform of a translating 2D pattern:
All the energy lies on a plane through the origin in frequency space
Local energy is extracted using velocitytuned filters (for example,
Gaborenergy filters)
Motion is found by fitting the best plane in frequency space
Fleet Jepson (1990): Phasebased Technique
Assumption that phase is preserved (as opposed to amplitude)
Velocity tuned band pass filters have complexvalued outputs
( ) ( ) ( )
t y x y x t y x
w v w u w w w E F w w w E F + + = o 0 , , , ,
( ) ( )
( ) w t x i
e w t x w t x R
, ,
, , , ,

=
0 or 0
dt
d
phase the and amplitude the with
= + + =
t y x
v u   


Correlationbased Methods
Anandan (1987), Singh (1990)
1. Find displacement (dx, dy) which maximizes cross correlation
or minimizes sum of squared differences (SSD)
2. Smooth the correlation outputs
( ) ( ) ( ) ( )
+
=
+
=
=
n
n i
n
n j
dy j dx i E j i E j i W dy dx CC , , , ,
2 1
( ) ( ) ( ) ( ) ( )
+
=
+
=
=
n
n i
n
n j
dy j dx i E j i E j i W dy dx SSD
2
2 1
, , , ,
A Pattern of Hajime Ouchi
Bias in Flow Estimation
Symmetric noise in spatial and temporal derivatives
Notation: oA=AA', where A is the estimate, A' the actual value and oA the
error
( ) ( )
( )
( )
( ) ( )
noise spatial of deviation standard
ts measuremen of number
' ' ' ' of value expected
solution LS
form matrix in
1
2
1
s
T
s
T T
t t y y x x
n
E E n E
E E E
E E
E E v E E u E E
i i i i i i
o
o
o
o o o
u u u u
b u
b u
=
=
=
= +
Underestimation in length
Bias in direction: more underestimation in direction of fewer
measurements
coplanar ' and , or coplanar, ' ' , 'C, m m t R C m C m C
( )
  ( ) 0 '
constraint epipolar 0 '
=
=
m m t
m m t
R
R
T
T
  R E E
T
= = t m m , 0 '
  m t m t
=
(
(
(
=
(
(
(
0
0
0
: Def
1 2
1 3
2 3
3
2
1
m m
m m
m m
m
m
m
Epipolar Constraint for Discrete Motions
lines. epipolar ing correspond the are and
' m m
l l
'
m
m m t m e
m
m
E l
E R R l
=
= = =
'
'
' ' '
: ' , 0 or 0 ' : constraint Epipolar
' '
m m m m
m m
E l l E
T T
= = =
points lie on their corresponding epipolar lines.
The epipole lies on all epipolar lines
. 0 or , ' 0 ' = = E E
T T
e m m e
12 21 11 22 3
32 11 12 31 2
31 22 21 32 1
E E E E e
E E E E e
E E E E e
=
=
=
  , 0
1
, , or 0 line a Consider =
(
(
(
= + + y
x
c b a c by ax
    . 1 , , and , , with 0 or
T T
T
y x c b a l l = = = m m
, and points wo through t goes line a If
2 1
m m
. or 0 and 0 then
2 1 2 1
m m m m = = = l l l
T T
Sources:
Horn (1986)
J. L. Barron, D. J. Fleet, S. S. Beauchemin (1994). Systems and
Experiment. Performance of Optical Flow Techniques. IJCV 12(1):43
77. Available at http://www.cs.queesu.ca/home/fleet/
research/Projects/flowCompare.html
http://www.cfar.umd.edu/~fer/postscript/ouchipapernew.ps.gz (paper
on Ouchi illusion)
http://www.cfar.umd.edu./ftp/TRs/CVLReports1999/TR4080
fermueller.ps.gz (paper on statistical bias)
http://www.cis.upenn.edu/~beau/home.html
http://www.isi.uu.nl/people/michael/of.html (code for optical flow
estimation techniques)
Much more than documents.
Discover everything Scribd has to offer, including books and audiobooks from major publishers.
Cancel anytime.