Lucas-Kanade Optical Flow

Computer Vision 16-385

Carnegie Mellon University (Kris Kitani)
 Ix u + Iy v + It = 0

@I @I dx dy @I
Ix = Iy = u= v= It =
@x @y dt dt @t
spatial derivative optical flow temporal derivative

How can we use the brightness constancy equation to

estimate the optical flow?
 unknown

Ix u + Iy v + It = 0

known

We need at least ____ equations to solve for 2 unknowns.

 unknown

Ix u + Iy v + It = 0

known

Where do we get more equations (constraints)?

 Where do we get more equations (constraints)?

Ix u + Iy v + It = 0
Assume that the surrounding patch (say 5x5) has
constant flow
Assumptions:
Flow is locally smooth
Neighboring pixels have same displacement

Using a 5 x 5 image patch, gives us 25 equations

Assumptions:
Flow is locally smooth
Neighboring pixels have same displacement

Using a 5 x 5 image patch, gives us 25 equations

Ix (p1 )u + Iy (p1 )v = It (p1 )

Ix (p2 )u + Iy (p2 )v = It (p2 )
..
.
Ix (p25 )u + Iy (p25 )v = It (p25 )
Assumptions:
Flow is locally smooth
Neighboring pixels have same displacement

Using a 5 x 5 image patch, gives us 25 equations

2 3 2 3
Ix (p1 ) Iy (p1 ) It (p1 )
6 Ix (p2 ) Iy (p2 ) 7  6 It (p2 ) 7
6 7 u 6 7
6 .. .. 7 = 6 . 7
4 . . 5 v 4 .
. 5
Ix (p25 ) Iy (p25 ) It (p25 )

Matrix form
Assumptions:
Flow is locally smooth
Neighboring pixels have same displacement

Using a 5 x 5 image patch, gives us 25 equations

2 3 2 3
Ix (p1 ) Iy (p1 ) It (p1 )
6 Ix (p2 ) Iy (p2 ) 7  6 It (p2 ) 7
6 7 u 6 7
6 .. .. 7 = 6 . 7
4 . . 5 v 4 .
. 5
Ix (p25 ) Iy (p25 ) It (p25 )

A
25 ⇥ 2
x
2⇥1
b
25 ⇥ 1

How many equations? How many unknowns? How do we solve this?

 Least squares approximation
2 > >
x̂ = arg min ||Ax b|| is equivalent to solving A Ax̂ = A b
x
 Least squares approximation
2 > >
x̂ = arg min ||Ax b|| is equivalent to solving A Ax̂ = A b
x

To obtain the least squares solution solve:

> x̂ >
A A A b
2 P P 3 2 P 3
Ix Ix Ix Iy  Ix It
4 p2P
P p2P
P 5 u 4 p2P
P 5
=
Iy Ix Iy Iy v Iy It
p2P p2P p2P

where the summation is over each pixel p in patch P

> 1 >
x = (A A) A b
 Least squares approximation
2 > >
x̂ = arg min ||Ax b|| is equivalent to solving A Ax̂ = A b
x

To obtain the least squares solution solve:

> x̂ >
A A A b
2 P P 3 2 P 3
Ix Ix Ix Iy  Ix It
4 p2P
P p2P
P 5 u 4 p2P
P 5
=
Iy Ix Iy Iy v Iy It
p2P p2P p2P

where the summation is over each pixel p in patch P

Sometimes called ‘Lucas-Kanade Optical Flow’


(special case of the LK method with a translational warp model)
 When is this solvable?

> >
A Ax̂ = A b

ATA should be invertible

ATA should not be too small

λ1 and λ2 should not be too small

ATA should be well conditioned

λ1/λ2 should not be too large (λ1=larger eigenvalue)
Where have you seen this before?

2 P P 3
Ix Ix Ix Iy
> 4 p2P
P p2P
P 5
A A = Iy Ix Iy Iy
p2P p2P
Where have you seen this before?

2 P P 3
Ix Ix Ix Iy
4 p2P
P p2P
P 5
Iy Ix Iy Iy
p2P p2P

Harris Corner Detector!

 Implications
• Corners are when λ1, λ2 are big; this is also when
Lucas-Kanade optical flow works best

• Corners are regions with two different directions of

gradient (at least)

• Corners are good places to compute flow!

What happens when you have no ‘corners’?

 You want to compute optical flow.
What happens if the image patch contains only a line?
 small visible
image patch

In which direction is the line moving?

 small visible
image patch

In which direction is the line moving?

Want patches with different gradients to
the avoid aperture problem
Want patches with different gradients to
the avoid aperture problem
 H(x,y) = y I(x,y)
x x
1 1 1 1 - - - -
2 2 2 2
optical flow: (1,1) - 1 1 1
3 3 3 3 - 2 2 2
4 4 4 4 - 3 3 3
5 5 5 5 - 4 4 4
y y

Ix u + Iy v + It = 0
Compute gradients Solution:

Ix (3, 3) = 0
Iy (3, 3) = 1 v=1
It (3, 3) = I(3, 3) H(3, 3) = 1

We recover the v of the optical flow but not the u.

This is the aperture problem.
 Note:
The Lucas-Kanade method is a general
framework for image matching

(not only optical flow)

Applied to optical flow:

Assume that the flow over a small image patch is constant
 Horn-Shunck! Lucas-Kanade!
Optical Flow 
 Optical Flow!
(1981) (1981)

‘smooth’ flow ‘constant’ flow

Constraint: Contraint:
Flow of neighboring pixels Flow over a small patch
should be smooth should be the same
Horn-Schunck
16-385 Computer Vision
Carnegie Mellon University (Kris Kitani)
 Horn-Schunck ! Lucas-Kanade!
Optical Flow (1981) Optical Flow (1981)

brightness constancy
method of differences
small motion

‘smooth’ flow! ‘constant’ flow!

(flow can vary from pixel to pixel) (flow is constant for all pixels)

global method local method

(dense) (sparse)
 Optical Flow
(Problem definition)

0
I(x, y, t) I(x, y, t )

Estimate the motion

(flow) between these
two consecutive images
How is this different from estimating a 2D transform?
 Key Assumptions
(unique to optical flow)

Color Constancy!
(Brightness constancy for intensity images)

Implication: allows for pixel to pixel comparison

(not image features)

Small Motion!
(pixels only move a little bit)

Implication: linearization of the brightness

constancy constraint
 Approach

0
I(x, y, t) I(x, y, t )

Look for nearby pixels with the same color

(small motion) (color constancy)
 Brightness constancy
(x + u t, y + v t)

(x, y) (x, y)

I(x, y, t) I(x, y, t + t)

Optical flow (velocities): (u, v) Displacement: ( x, y) = (u t, v t)

I(x + u t, y + v t, t + t) = I(x, y, t)
For a really small time step…
Optical Flow Constraint equation
I(x + u t, y + v t, t + t) = I(x, y, t)

@I @I @I
I(x, y, t) + x+ y+ t = I(x, y, t)
@x @y @t

@I dx @I dy @I
+ + =0
@x dt @y dt @t

Ix u + Iy v + It = 0
Solution lies on a straight line
u

Ix u + Iy v + It = 0

The solution cannot be determined uniquely with

a single constraint (a single pixel)
Where can we get an additional constraint?
 Horn-Schunck ! Lucas-Kanade!
Optical Flow (1981) Optical Flow (1981)

brightness constancy
method of differences
small motion

‘smooth’ flow! ‘constant’ flow!

(flow can vary from pixel to pixel) (flow is constant for all pixels)

global method local method

 Smoothness

most objects in the world are rigid or

deform elastically !
moving together coherently

we expect optical flow fields to be smooth

 Key idea
(of Horn-Schunck optical flow)

Enforce
brightness constancy

Enforce
smooth flow field

to compute optical flow

Enforce brightness
constancy

Ix u + Iy v + It = 0

For every pixel,

 2
min Ix uij + Iy vij + It
u,v
Enforce brightness
constancy

Ix u + Iy v + It = 0

For every pixel,

 2
min Ix uij + Iy vij + It
u,v
lazy notation for Ix (i, j)
 displacement of object

Find the optical flow such that it satisfies:

Ix Iy It

+ =

u changes these v changes these

Enforce smooth flow field
ui,j+1

ui 1,j uij ui+1,j min(ui,j ui+1,j ) 2

u

ui,j 1

u-component of flow
 Which flow field is more likely?

X X
2 2
(uij ui+1,j ) ? (uij ui+1,j )
ij ij
 Which flow field is more likely?

less likely more likely

 optical flow objective function
 2
Optical flow constraint Ed (i, j) = Ix uij + Iy vij + It


Smoothness constraint
1
Es (i, j) = (uij ui+1,j )2 + (uij ui,j+1 )2 + (vij vi+1,j )2 + (vij vi,j+1 )2
4
i, j + 1 i, j + 1 i, j + 1 i, j + 1

(uij ui+1,j ) (uij ui,j+1 ) (vij vi+1,j ) (vij vi,j+1 )

i 1, j ij i + 1, j i 1, j ij i + 1, j i 1, j ij i + 1, j i 1, j ij i + 1, j

i, j 1 i, j 1 i, j 1 i, j 1
Horn-Schunck optical flow
X ⇢
min Ed (i, j) + Es (i, j)
u, v
ij
where
 2
Ed (i, j) = Ix uij + Iy vij + It

and

1
Es (i, j) = (uij ui+1,j )2 + (uij ui,j+1 )2 + (vij vi+1,j )2 + (vij vi,j+1 )2
4
 How do we solve this
minimization problem?

X ⇢
min Ed (i, j) + Es (i, j)
u, v
ij
 How do we solve this
minimization problem?

X ⇢
min Ed (i, j) + Es (i, j)
u, v
ij
Compute partial derivative, derive update equations
(gradient decent)
 Compute the partial derivatives of this huge sum!
(   )
X 1 2
(uij ui+1,j )2 + (uij ui,j+1 )2 + (vij vi+1,j )2 + (vij vi,j+1 )2 + Ix uij + Iy vij + It
ij
4
 Compute the partial derivatives of this huge sum!
(   )
X 1 2
(uij ui+1,j )2 + (uij ui,j+1 )2 + (vij vi+1,j )2 + (vij vi,j+1 )2 + Ix uij + Iy vij + It
ij
4

it’s not so bad…

@E
= 2(ukl ūkl ) + 2 (Ix ukl + Iy vkl + It )Ix
@ukl

how many terms depend on k and l?

 Compute the partial derivatives of this huge sum!
(   )
X 1 2
(uij ui+1,j )2 + (uij ui,j+1 )2 + (vij vi+1,j )2 + (vij vi,j+1 )2 + Ix uij + Iy vij + It
ij
4

it’s not so bad…

@E
= 2(ukl ūkl ) + 2 (Ix ukl + Iy vkl + It )Ix
@ukl

how many terms depend on k and l?

 Compute the partial derivatives of this huge sum!
(   )
X 1 2
(uij ui+1,j )2 + (uij ui,j+1 )2 + (vij vi+1,j )2 + (vij vi,j+1 )2 + Ix uij + Iy vij + It
ij
4

i, j + 1

(uij ui+1,j )
(u2ij 2uij ui+1,j + u2i+1,j ) (u2ij 2uij ui,j+1 + u2i,j+1 )
i 1, j ij i + 1, j

(variable will appear four times in sum)

i, j 1

@E
= 2(ukl ūkl ) + 2 (Ix ukl + Iy vkl + It )Ix
@ukl
@E
= 2(vkl v̄kl ) + 2 (Ix ukl + Iy vkl + It )Iy
@vkl
⇢
1
local average ūij = ui+1,j + ui 1,j + ui,j+1 + ui,j 1
4
@E
= 2(ukl ūkl ) + 2 (Ix ukl + Iy vkl + It )Ix
@ukl
@E
= 2(vkl v̄kl ) + 2 (Ix ukl + Iy vkl + It )Iy
@vkl

Where are the extrema of E?

@E
= 2(ukl ūkl ) + 2 (Ix ukl + Iy vkl + It )Ix
@ukl
@E
= 2(vkl v̄kl ) + 2 (Ix ukl + Iy vkl + It )Iy
@vkl

Where are the extrema of E?

(set derivatives to zero and solve for unknowns u and v)

2
(1 + Ix )ukl + Ix Iy vkl = ūkl Ix It

Ix Iy ukl + (1 + Iy2 )vkl = v̄kl Iy It

 @E
= 2(ukl ūkl ) + 2 (Ix ukl + Iy vkl + It )Ix
@ukl
@E
= 2(vkl v̄kl ) + 2 (Ix ukl + Iy vkl + It )Iy
@vkl

Where are the extrema of E?

(set derivatives to zero and solve for unknowns u and v)

2
(1 + Ix )ukl + Ix Iy vkl = ūkl Ix It

Ix Iy ukl + (1 + Iy2 )vkl = v̄kl Iy It

this is a linear system Ax = b how do you solve this?

ok, take a step back, why are we doing all this math?
 We are solving for the optical flow (u,v) given
two constraints
(   )
X 1 2
(uij ui+1,j )2 + (uij ui,j+1 )2 + (vij vi+1,j )2 + (vij vi,j+1 )2 + Ix uij + Iy vij + It
ij
4

smoothness brightness constancy

We need the math to minimize this

(back to the math)
 Partial derivatives of Horn-Schunck objective function E:

@E
= 2(ukl ūkl ) + 2 (Ix ukl + Iy vkl + It )Ix
@ukl
@E
= 2(vkl v̄kl ) + 2 (Ix ukl + Iy vkl + It )Iy
@vkl

Where are the extrema of E?

(set derivatives to zero and solve for unknowns u and v)

2
(1 + Ix )ukl + Ix Iy vkl = ūkl Ix It

Ix Iy ukl + (1 + Iy2 )vkl = v̄kl Iy It

Ax = b how do you solve this?

 2
(1 + Ix )ukl + Ix Iy vkl = ūkl Ix It
2
Ix Iy ukl + (1 + Iy )vkl = v̄kl Iy It

1 adjA
Recall x=A b= b
det A
 2
(1 + Ix )ukl + Ix Iy vkl = ūkl Ix It
2
Ix Iy ukl + (1 + Iy )vkl = v̄kl Iy It

1 adjA
Recall x=A b= b
det A

Same as the linear system:

2 2 2
{1 + (Ix + Iy )}ukl = (1 + Ix )ūkl Ix Iy v̄kl Ix It

(determinant of A)
2 2 2
{1 + (Ix + Iy )}vkl = (1 + Iy )v̄kl Ix Iy ūkl Iy It
(determinant of A)
{1 + (Ix2 + Iy2 )}ukl = (1 + Ix2 )ūkl Ix Iy v̄kl Ix It
(determinant of A)

2 2 2
{1 + (Ix + Iy )}vkl = (1 + Iy )v̄kl Ix Iy ūkl Iy It
(determinant of A)

Rearrange to get update equations:

Ix ūkl + Iy v̄kl + It
ûkl = ūkl 2 2
I x
new old 1 + (Ix + Iy )
value average

Ix ūkl + Iy v̄kl + It
v̂kl = v̄kl 2 2
I y
1 + (Ix + Iy )
check for missing lambda
ok, take a step back, why did we do all this math?
 We are solving for the optical flow (u,v) given
two constraints
(   )
X 1 2
(uij ui+1,j )2 + (uij ui,j+1 )2 + (vij vi+1,j )2 + (vij vi,j+1 )2 + Ix uij + Iy vij + It
ij
4

smoothness brightness constancy

We needed the math to minimize this

(now to the algorithm)
 Horn-Schunck Optical Flow
Algorithm
1. Precompute image gradients! Iy Ix
2. Precompute temporal gradients! It
3. Initialize flow field! u=0
v=0
4. While not converged



Compute flow field updates for each pixel:

Ix ūkl + Iy v̄kl + It Ix ūkl + Iy v̄kl + It

ûkl = ūkl 2 2
Ix v̂kl = v̄kl 2 2
Iy
1 + (Ix + Iy ) 1 + (Ix + Iy )