Optical Flow algorithm:

Author: Vivekkumar P
Mentor: Prof.Manaswita Bose, Energy science & Engg, IITB.

Optical flow algorithms are used to compute the velocity field between the
sequence of images. Let us assume that, the brightness of an image at a point
(x, y) and at time (t) can be denoted as I(x, y, t). The image intensity at a point
with a displacement of δx and δy in both x and y direction at time t + δt can
be represented as I(x + δx, y + δy, t + δt) The brightness constancy assumption
states that, the image intensity remains constant over time. Based on this,

I(x, y, t) = I(x + δx, y + δy, t + δt) (1)

From Taylor’s series expansion,
∂I ∂I ∂I
I(x, y, t) = I(x, y, t) + δx + δy + δt + ∆ (2)
∂x ∂y ∂t
Here ∆ represents higher order terms. After neglecting the higher order
derivatives, and dividing by δt, we have
∂I δx ∂I δy ∂I
+ + =0 (3)
∂x δt ∂y δt ∂t
When δt → 0,
∂I dx ∂I dy ∂I
+ + =0 (4)
∂x dt ∂y dt ∂t
∂I ∂I ∂I
u+ v+ =0 (5)
∂x ∂y ∂t
Here u and v are the velocity components in x and y directions respectively.

∇I.~v + It = 0 (6)
This is said to be Optical flow constraint equation, ref: 6 . Still the system is
under-constrained. To solve u and v components we require additional equation.
To solve this Horn and Schunck assumed it as an error minimization problem.
Also assumed that the optical flow is smooth across the pixels.
The total error to be minimized be,

2 = b 2 + α2 s 2 (7)
Where,
b = Ix u + Iy v + It (8)

2s = (ū − u)2 + (v̄ − v)2 (9)

1
 Where, b is the occurrence of error in the optical flow constraint equation. s
is the estimate of smoothness across the pixels. Substitute b and s in equation
7. We have,

2 = (Ix u + Iy v + It )2 + α2 [(ū − u)2 + (v̄ − v)2 ] (10)

Here, α controls the weight of the smoothness terms compared to optical flow
constraint equation. Differentiate equation 10 w.r.t u and v,

∂2
= 2Ix (Ix u + Iy v + It ) − 2α2 (ū − u) (11)
∂u
∂2
= 2Iy (Ix u + Iy v + It ) − 2α2 (v̄ − v) (12)
∂v
Setting equations 11 and 12 equals to zero we arrives,

Ix (Ix u + Iy v + It ) − α2 (ū − u) = 0 (13)

Iy (Ix u + Iy v + It ) − α2 (v̄ − v) = 0 (14)

Ix2 u + Ix Iy v + Ix It − α2 ū + α2 u = 0 (15)

Iy Ix u + Iy2 v + Iy It − α2 v̄ + α2 v = 0 (16)
Separate the u and v terms from the above equations,

(α2 + Ix2 )u + Ix Iy v = α2 ū − Ix It (17)

Ix Iy u + (α2 + Iy2 )v = α2 v̄ − Iy It (18)

Rearrange the above equation in matrix form,
 2
α + Ix2 Ix Iy
   2 
u α ū − Ix It
= 2 (19)
Ix Iy α2 + Iy2 v α v̄ − Iy It
   2 −1  2
α + Ix2 Ix Iy

u α ū − Ix It
= (20)
v Ix Iy α2 + Iy2 α2 v̄ − Iy It
 2
α + Iy2 −Ix Iy
   2 
u 1 α ū − Ix It
= 2 2 (21)
v α (α + Ix2 + Iy2 ) −Ix Iy α2 + Ix2 α2 v̄ − Iy It
The solution is
  " Ix ū+Iy v̄+It #
u ū − Ix α2 +I 2 +I 2
= x y
Ix ū+Iy v̄+It (22)
v v̄ − Iy 2 2 2
α +Ix +Iy

2
 The iterative form of the above equations,
Ix ūn + Iy v̄ n + It
un+1 = ūn − Ix (23)
α2 + Ix2 + Iy2

n+1 n Ix ūn + Iy v̄ n + It
v = v̄ − Iy (24)
α2 + Ix2 + Iy2
To find the values of Ix , Iy , It between the two consecutive images we use
the following equations,
Using the forward differencing scheme,
1
Ix ≈ (Ii,j+1,k −Ii,j,k +Ii+1,j+1,k −Ii+1,j,k +Ii,j+1,k+1 −Ii,j,k+1 +Ii+1,j+1,k+1 −Ii+1,j,k+1 )
4
(25)

1
Iy ≈ (Ii+1,j,k −Ii,j,k +Ii+1,j+1,k −Ii,j+1,k +Ii+1,j,k+1 −Ii,j,k+1 +Ii+1,j+1,k+1 −Ii,j+1,k+1 )
4
(26)

1
It ≈ (Ii,j,k+1 −Ii,j,k +Ii+1,j,k+1 −Ii+1,j,k +Ii,j+1,k+1 −Ii,j+1,k +Ii+1,j+1,k+1 −Ii+1,j+1,k )
4
(27)
To find the local average ū and v̄,

1 1
ūn ≈ (uni−1,j −uni+1,j +uni,j−1 −uni,j+1 )+ (uni−1,j−1 −uni+1,j−1 +uni−1,j+1 −uni+1,j+1 )
6 12
(28)

1 n 1 n
v̄ n ≈ (vi−1,j n
− vi+1,j n
+ vi,j−1 n
− vi,j+1 ) + (vi−1,j−1 n
− vi+1,j−1 n
+ vi−1,j+1 n
− vi+1,j+1 )
6 12
(29)
The iterations should satisfy the below convergence criteria σ and it can be
defined as,
s
1 X n+1 n+1
(ui,j − uni,j )2 + (vi,j n )2 < σ
− vi,j (30)
N ij

3
 Algorithm: Horn–Schunck optical flow
• → Input: Image sequence I(x, y), Weighted parameter (α), maximum iter-
ation (Nmax ) and convergence criteria (σ)
• → Compute Ix , Iy and It
• → Initialize u = 0, v = 0 and n = 0
• → while n < Nmax do Convergence criteria > σ
– Compute ū and v̄
– Compute,

n+1 n Ix ūn + Iy v̄ n + It
u = ū − Ix
α2 + Ix2 + Iy2

n+1 n Ix ūn + Iy v̄ n + It
v = v̄ − Iy
α2 + Ix2 + Iy2
– Compute and check convergence criteria
q P
1 n+1 n 2 n+1 n 2
N ij (ui,j − ui,j ) + (vi,j − vi,j )

– Change n → n + 1
• → end
Reference:

Berthold K.P.Horn and Brain.G.Schunck Determining optical flow, Artificial

Intelligence laboratory, MIT, 1980.