Pratik Jain

YouTube Video
Hamburg Taxi Sequence

Pratik Jain 2
Horn and Schunck Optical Flow
Brightness Constancy Assumption
𝑓 𝑥, 𝑦, 𝑡 = 𝑓(𝑥 + 𝑑𝑥, 𝑦 + 𝑑𝑦, 𝑡 + 𝑑𝑡)

Frame 𝑡 Frame 𝑡 + 𝑑𝑡

𝑦 + 𝑑𝑦

𝑥 𝑥 + 𝑑𝑥
Pratik Jain 3
 𝑓 𝑥, 𝑦, 𝑡 = 𝑓(𝑥 + 𝑑𝑥, 𝑦 + 𝑑𝑦, 𝑡 + 𝑑𝑡)

Using Taylor Series

𝜕𝑓 𝜕𝑓 𝜕𝑓
𝑓 𝑥, 𝑦, 𝑡 = 𝑓 𝑥, 𝑦, 𝑡 + 𝑑𝑥 + 𝑑𝑦 + 𝑑𝑡 + ⋯ ℎ𝑖𝑔ℎ𝑒𝑟 𝑜𝑟𝑑𝑒𝑟 𝑡𝑒𝑟𝑚𝑠
𝜕𝑥 𝜕𝑦 𝜕𝑡

𝜕𝑓 𝜕𝑓 𝜕𝑓
0= 𝑑𝑥 + 𝑑𝑦 + 𝑑𝑡
𝜕𝑥 𝜕𝑦 𝜕𝑡

𝑑𝑥 𝑑𝑦 𝑑𝑡
𝑓𝑥 + 𝑓𝑦 + 𝑓𝑡 =0
𝑑𝑡 𝑑𝑡 𝑑𝑡

Pratik Jain 4
 𝑣
𝑑𝑥 𝑑𝑦 𝑑𝑡
𝑓𝑥 + 𝑓𝑦 + 𝑓𝑡 =0
𝑑𝑡 𝑑𝑡 𝑑𝑡
𝑓𝑡
−
𝑓𝑦
𝑓𝑥 𝑢 + 𝑓𝑦 𝑣 + 𝑓𝑡 = 0

𝑓𝑦 𝑣 = −𝑓𝑥 𝑢 − 𝑓𝑡

𝑓𝑥 𝑓𝑡 𝑓𝑥 𝑢
𝑣=− 𝑢− −
𝑓𝑦
𝑓𝑦 𝑓𝑦

Pratik Jain 5
Smoothness Constraint

Frame 𝑡 Frame 𝑡 + 𝑑𝑡

𝑢 𝑣

𝑓 𝑥, 𝑦, 𝑡 𝑓(𝑥 + 𝑑𝑥, 𝑦 + 𝑑𝑦, 𝑡 + 𝑑𝑡)

𝑢 𝑥, 𝑦 𝑣 𝑥, 𝑦
Pratik Jain 6
 𝑢 𝑣

𝑢𝑥 𝑢𝑦 𝑣𝑥 𝑣𝑦

Pratik Jain 7
Cost Function
𝑢𝑥 𝑢𝑦 𝑣𝑥 𝑣𝑦

𝐸 = ඵ(𝑓𝑥 𝑢 + 𝑓𝑦 𝑣 + 𝑓𝑡 )2 + 𝜆 𝑢𝑥2 + 𝑢𝑦2 + 𝑣𝑥2 + 𝑣𝑦2 𝑑𝑥𝑑𝑦

𝑥, 𝑦

Functional

We want to find 2 functions u and v such that the above functional is minimum.

Pratik Jain 8
Functional Example

 Consider 2 points
 Cost = find the curve that has minimum length
 Here we can see that we want to find a function which
is having the minimum length

Pratik Jain 9
Cost Function
𝑢𝑥 𝑢𝑦 𝑣𝑥 𝑣𝑦

𝐸 = ඵ(𝑓𝑥 𝑢 + 𝑓𝑦 𝑣 + 𝑓𝑡 )2 + 𝜆 𝑢𝑥2 + 𝑢𝑦2 + 𝑣𝑥2 + 𝑣𝑦2 𝑑𝑥𝑑𝑦

𝑥, 𝑦

Functional

We want to find 2 functions u and v such that the above functional is minimum.

Pratik Jain 10
 Minimize the cost
𝐸 = ඵ(𝑓𝑥 𝑢 + 𝑓𝑦 𝑣 + 𝑓𝑡 )2 + 𝜆 𝑢𝑥2 + 𝑢𝑦2 + 𝑣𝑥2 + 𝑣𝑦2 𝑑𝑥𝑑𝑦
𝑥, 𝑦
Euler-Lagrange equations
𝑑𝑢𝑥 𝑑𝑢𝑦 𝑑𝑣𝑥 𝑑𝑣𝑦
𝐸𝑢 − − =0 𝐸𝑣 − − =0
𝑑𝑥 𝑑𝑦 𝑑𝑥 𝑑𝑦

2(𝑓𝑥 𝑢 + 𝑓𝑦 𝑣 + 𝑓𝑡 )𝑓𝑥 − 2𝜆 𝑢𝑥𝑥 + 𝑢𝑦𝑦 = 0 2(𝑓𝑥 𝑢 + 𝑓𝑦 𝑣 + 𝑓𝑡 )𝑓𝑦 − 2𝜆 𝑣𝑥𝑥 + 𝑣𝑦𝑦 = 0

(𝑓𝑥 𝑢 + 𝑓𝑦 𝑣 + 𝑓𝑡 )𝑓𝑥 − 𝜆 ∆2 𝑢 = 0 (𝑓𝑥 𝑢 + 𝑓𝑦 𝑣 + 𝑓𝑡 )𝑓𝑦 − 𝜆 ∆2 𝑣 = 0

Pratik Jain 11
 Masks
Frame 𝑡 Frame 𝑡 + 𝑑𝑡
−1 1 −1 −1 −1 −1
Frame 𝑡 1 1
−1 1 −1 −1
𝑓𝑥1 𝑓𝑦1 𝑓𝑡1
−1 1 −1 −1 1 1
Frame 𝑡 + 𝑑𝑡
−1 1 1 1 1 1
𝑓𝑥2 𝑓𝑦2 𝑓𝑡2
𝑓𝑥 mask 𝑓𝑦 mask 𝑓𝑡 mask
𝑓 𝑥, 𝑦, 𝑡 𝑓(𝑥 + 𝑑𝑥, 𝑦 + 𝑑𝑦, 𝑡 + 𝑑𝑡)
𝑓𝑥 = 0.5 ((Frame 𝑡)* 𝑓𝑥1 + ( Frame 𝑡 + 𝑑𝑡)* 𝑓𝑥2 )
𝑓𝑦 = 0.5 ((Frame 𝑡)* 𝑓𝑦1 + ( Frame 𝑡 + 𝑑𝑡)* 𝑓𝑦2 )
𝑓𝑡 = (Frame 𝑡)*𝑓𝑡1 +( Frame 𝑡 + 𝑑𝑡)*𝑓𝑡2
Pratik Jain 12
Laplacian Mask
(𝑓𝑥 𝑢 + 𝑓𝑦 𝑣 + 𝑓𝑡 )𝑓𝑥 − 𝜆 ∆2 𝑢 = 0 (𝑓𝑥 𝑢 + 𝑓𝑦 𝑣 + 𝑓𝑡 )𝑓𝑦 − 𝜆 ∆2 𝑣 = 0

1 1 1
12 6 12
1 1 ∆2 𝑢 = (𝑢𝑎𝑣𝑔 − 𝑢)
−1
6 6
1 1 1 ∆2 𝑣 = (𝑣𝑎𝑣𝑔 − 𝑣)
12 6 12

(𝑓𝑥 𝑢 + 𝑓𝑦 𝑣 + 𝑓𝑡 )𝑓𝑥 − 𝜆 𝑢𝑎𝑣𝑔 − 𝑢 = 0 (𝑓𝑥 𝑢 + 𝑓𝑦 𝑣 + 𝑓𝑡 )𝑓𝑦 − 𝜆 𝑣𝑎𝑣𝑔 − 𝑣 = 0

Pratik Jain 13
Solving the equations
(𝑓𝑥 𝑢 + 𝑓𝑦 𝑣 + 𝑓𝑡 )𝑓𝑥 − 𝜆 𝑢𝑎𝑣𝑔 − 𝑢 = 0 (𝑓𝑥 𝑢 + 𝑓𝑦 𝑣 + 𝑓𝑡 )𝑓𝑦 − 𝜆 𝑣𝑎𝑣𝑔 − 𝑣 = 0

𝑓𝑥 2 𝑢 + 𝑓𝑥 𝑓𝑦 𝑣 + 𝑓𝑥 𝑓𝑡 − 𝜆𝑢𝑎𝑣𝑔 + 𝜆𝑢 = 0 (𝑓𝑥 𝑓𝑦 𝑢 + 𝑓𝑦 2 𝑣 + 𝑓𝑦 𝑓𝑡 ) − 𝜆𝑣𝑎𝑣𝑔 + 𝜆𝑣 = 0

2
(𝑓𝑥 + 𝜆)𝑢 + 𝑓𝑥 𝑓𝑦 𝑣 = −𝑓𝑡 𝑓𝑥 + 𝜆𝑢𝑎𝑣𝑔 𝑓𝑥 𝑓𝑦 𝑢 + (𝑓𝑦 2 + 𝜆)𝑣 = −𝑓𝑡 𝑓𝑦 + 𝜆𝑣𝑎𝑣𝑔
Cramers Rule

(𝑓𝑥 2 + 𝜆) 𝑓𝑥 𝑓𝑦 −𝑓𝑡 𝑓𝑥 + 𝜆𝑢𝑎𝑣𝑔 𝑓𝑥 𝑓𝑦 (𝑓𝑥 2 + 𝜆) −𝑓𝑡 𝑓𝑥 + 𝜆𝑢𝑎𝑣𝑔

𝐷= 𝐷𝑢 = 𝐷𝑣 =
𝑓𝑥 𝑓𝑦 2
(𝑓𝑦 + 𝜆) −𝑓𝑡 𝑓𝑦 + 𝜆𝑣𝑎𝑣𝑔 (𝑓𝑦 2 + 𝜆) 𝑓𝑥 𝑓𝑦 −𝑓𝑡 𝑓𝑦 + 𝜆𝑣𝑎𝑣𝑔

Pratik Jain 14
 (𝑓𝑥 2 + 𝜆) 𝑓𝑥 𝑓𝑦 −𝑓𝑡 𝑓𝑥 + 𝜆𝑢𝑎𝑣𝑔 𝑓𝑥 𝑓𝑦 (𝑓𝑥 2 + 𝜆) −𝑓𝑡 𝑓𝑥 + 𝜆𝑢𝑎𝑣𝑔
𝐷= 𝐷𝑢 = 𝐷𝑣 =
𝑓𝑥 𝑓𝑦 (𝑓𝑦 2 + 𝜆) −𝑓𝑡 𝑓𝑦 + 𝜆𝑣𝑎𝑣𝑔 (𝑓𝑦 2 + 𝜆) 𝑓𝑥 𝑓𝑦 −𝑓𝑡 𝑓𝑦 + 𝜆𝑣𝑎𝑣𝑔

𝐷 = (𝑓𝑥 2 + 𝜆)(𝑓𝑦 2 + 𝜆) − (𝑓𝑥 𝑓𝑦 )2 =≫ 𝑓𝑥2 𝑓𝑦2 + 𝜆𝑓𝑦2 + 𝜆𝑓𝑥2 + 𝜆2 − (𝑓𝑥 𝑓𝑦 )2 =≫ 𝜆(𝑓𝑦2 +𝑓𝑥2 + 𝜆)

𝐷𝑢 = (−𝑓𝑡 𝑓𝑥 + 𝜆𝑢𝑎𝑣𝑔 )(𝑓𝑦 2 + 𝜆) − −𝑓𝑡 𝑓𝑦 + 𝜆𝑣𝑎𝑣𝑔 𝑓𝑥 𝑓𝑦 =≫ (−𝑓𝑡 𝑓𝑥 𝑓𝑦 2 − 𝑓𝑡 𝑓𝑥 𝜆 + 𝑓𝑦 2 𝜆𝑢𝑎𝑣𝑔 + 𝜆2 𝑢𝑎𝑣𝑔 + 𝑓𝑡 𝑓𝑥 𝑓𝑦 2 − 𝜆𝑣𝑎𝑣𝑔 𝑓𝑥 𝑓𝑦 )

𝐷𝑣 = (−𝑓𝑡 𝑓𝑦 + 𝜆𝑣𝑎𝑣𝑔 )(𝑓𝑥 2 + 𝜆) − −𝑓𝑡 𝑓𝑥 + 𝜆𝑢𝑎𝑣𝑔 𝑓𝑥 𝑓𝑦 =≫ (−𝑓𝑡 𝑓𝑥 2 𝑓𝑦 − 𝑓𝑡 𝑓𝑦 𝜆 + 𝑓𝑥 2 𝜆𝑣𝑎𝑣𝑔 + 𝜆2 𝑣𝑎𝑣𝑔 + 𝑓𝑡 𝑓𝑥 2 𝑓𝑦 − 𝜆𝑢𝑎𝑣𝑔 𝑓𝑥 𝑓𝑦 )

2 2
𝐷𝑢 (−𝑓𝑡 𝑓𝑥 𝜆 + 𝑓𝑦 𝜆𝑢𝑎𝑣𝑔 + 𝜆2 𝑢𝑎𝑣𝑔 − 𝜆𝑣𝑎𝑣𝑔 𝑓𝑥 𝑓𝑦 ) 𝐷𝑣 (−𝑓𝑡 𝑓𝑦 𝜆 + 𝑓𝑥 𝜆𝑣𝑎𝑣𝑔 + 𝜆2 𝑣𝑎𝑣𝑔 − 𝜆𝑢𝑎𝑣𝑔 𝑓𝑥 𝑓𝑦 )
𝑢= = 𝑣= =
𝐷 𝜆(𝑓𝑦2 +𝑓𝑥2 + 𝜆) 𝐷 𝜆(𝑓𝑦2 +𝑓𝑥2 + 𝜆)
2 2
𝐷𝑢 (−𝑓𝑡 𝑓𝑥 + 𝑓𝑦 𝑢𝑎𝑣𝑔 + 𝜆𝑢𝑎𝑣𝑔 − 𝑣𝑎𝑣𝑔 𝑓𝑥 𝑓𝑦 ) 𝐷𝑣 (−𝑓𝑡 𝑓𝑦 + 𝑓𝑥 𝑣𝑎𝑣𝑔 + 𝜆𝑣𝑎𝑣𝑔 − 𝑢𝑎𝑣𝑔 𝑓𝑥 𝑓𝑦 )
𝑢= = 𝑣= =
𝐷 (𝑓𝑦2 +𝑓𝑥2 + 𝜆) 𝐷 (𝑓𝑦2 +𝑓𝑥2 + 𝜆)

Pratik Jain 15
 2 2
𝐷𝑢 (−𝑓𝑡 𝑓𝑥 + 𝑓𝑦 𝑢𝑎𝑣𝑔 + 𝜆𝑢𝑎𝑣𝑔 − 𝑣𝑎𝑣𝑔 𝑓𝑥 𝑓𝑦 ) 𝐷𝑣 (−𝑓𝑡 𝑓𝑦 + 𝑓𝑥 𝑣𝑎𝑣𝑔 + 𝜆𝑣𝑎𝑣𝑔 − 𝑢𝑎𝑣𝑔 𝑓𝑥 𝑓𝑦 )
𝑢= = 𝑣= =
𝐷 (𝑓𝑦2 +𝑓𝑥2 + 𝜆) 𝐷 (𝑓𝑦2 +𝑓𝑥2 + 𝜆)

2 2 2
2 2 2
𝐷𝑢 (𝑓𝑦 𝑢𝑎𝑣𝑔 + 𝑓𝑥 𝑢𝑎𝑣𝑔 + 𝜆𝑢𝑎𝑣𝑔 − 𝑓𝑥 𝑢𝑎𝑣𝑔 − 𝑣𝑎𝑣𝑔 𝑓𝑥 𝑓𝑦 − 𝑓𝑡 𝑓𝑥 ) 𝐷𝑣 (𝑓𝑦 𝑢𝑎𝑣𝑔 + 𝑓𝑥 𝑣𝑎𝑣𝑔 + 𝜆𝑣𝑎𝑣𝑔 − 𝑓𝑦 𝑢𝑎𝑣𝑔 − 𝑢𝑎𝑣𝑔 𝑓𝑥 𝑓𝑦 − 𝑓𝑡 𝑓𝑦 )
𝑢= = 𝑣= =
𝐷 (𝑓𝑦2 +𝑓𝑥2 + 𝜆) 𝐷 (𝑓𝑦2 +𝑓𝑥2 + 𝜆)

2 2
𝐷𝑢 𝑢𝑎𝑣𝑔 𝑓𝑦 2 + 𝑓𝑥 2 + 𝜆 − 𝑓𝑥 (𝑓𝑥 𝑢𝑎𝑣𝑔 + 𝑓𝑦 𝑣𝑎𝑣𝑔 + 𝑓𝑡 ) 𝐷𝑣 𝑣𝑎𝑣𝑔 𝑓𝑦 + 𝑓𝑥 + 𝜆 − 𝑓𝑦 (𝑓𝑥 𝑢𝑎𝑣𝑔 + 𝑓𝑦 𝑣𝑎𝑣𝑔 + 𝑓𝑡 )
𝑢= = 𝑣= =
𝐷 (𝑓𝑦2 +𝑓𝑥2 + 𝜆) 𝐷 (𝑓𝑦2 +𝑓𝑥2 + 𝜆)

𝑃 𝑃
𝑢 = 𝑢𝑎𝑣𝑔 − 𝑓𝑥 𝑣 = 𝑣𝑎𝑣𝑔 − 𝑓𝑦
𝐷 𝐷

𝑃 = (𝑓𝑥 𝑢𝑎𝑣𝑔 + 𝑓𝑦 𝑣𝑎𝑣𝑔 + 𝑓𝑡 ) 𝐷 = (𝑓𝑦2 + 𝑓𝑥2 + 𝜆)

Pratik Jain 16
Algorithm
 Initialize u and v as 0 matrices 𝑃 = (𝑓𝑥 𝑢𝑎𝑣𝑔 + 𝑓𝑦 𝑣𝑎𝑣𝑔 + 𝑓𝑡 )
 Compute𝑓𝑥 , 𝑓𝑦 𝑎𝑛𝑑 𝑓𝑡 images using the masks 𝐷 = (𝑓𝑦2 + 𝑓𝑥2 + 𝜆)
𝑃 𝑃
 Update 𝑢 = 𝑢𝑎𝑣𝑔 − 𝑓𝑥 , 𝑣 = 𝑣𝑎𝑣𝑔 − 𝑓𝑦
𝐷 𝐷
 Repeat the above steps until some error measure is satisfied

Pratik Jain 17
Synthetic Images

Image from
Prof. Mubarak
Shah’s Slides

Pratik Jain 18
Results

Image from
Prof. Mubarak
𝜆=4
Shah’s Slides
Pratik Jain 19
Hamburg Taxi Sequence

40 iterations 𝜆=4

Pratik Jain 20
Hamburg Taxi Sequence

40 iterations 𝜆=4

Pratik Jain 21