Global Interpretation of Optical Flow Field: A Least-Squares Approach

Haibo Li Department of Electrical Engineering, Linkoping University, S-581 83 Linkoping, Sweden e-mail: haibo@isy.liu.se
Abstract
T h i s paper presents a Least-Squares approach t o globally interpret optical flow field. Using the derivatives of optical flow field, we can obtain a linear and global constraint equation f o r the translation motion. T h u s the translation motion can be robustly and simply recovered using a L S method. Once the translation motion is recovered, the rotation can be immediately recovered only using optical flow field. Therefore, the recovered rotation motion i s more reliable.
We can obtain the 3D motion parameters by nonlinear optimization of equation(3). Due to the fact that no local structure parameters are dealt with, this approach is a global fashion to interpret the 3D motion. But the fact that the constraint equation (3) is nonlinear implies that multiple solution may emerge. They correspond to ambiguous three-dimensional interpretations. The second approach is using the first and second spatial derivatives of the optical flow field to interpret 3D motion. Longuet-Higgins and Prazdny[2] showed that using up to second order spatial derivatives of the optical flow field, rigid motion and local orientation of a curved surface patch could, in principle, be determined. Lately Waxman[8], Subbarao[7], et a1 further extend this approach. Closed-form solutions for planar surfaces and curved surfaces are obtained. But there are some problems with this approach. Only the motion of the planar surface can be globally interpreted. In contrast, for general curved surfaces they only are interpreted locally[8]. Another problem is that the nonlinear relation between the 3D motion (and/or structure) and the spatiotemporal derivatives of the optical flow field is still there. This will result in an ambiguous solution in the interpretation of the 3D motion field. Another approach is the least-squares one[3]. In this approach the whole optical flow field is used. The algorithm takes the abundance of available data into account and is robust enough to allow numerical iniplementation. But its application is generally limited to restricted cases. For example, it is only suitable to the case when the scene depth is available, or translation only case, rotation only case etc. For the general cases no such an appi,oach as the least-squares approach is given due to the nonlinear nature of equations( 1)(2). In this paper a least-squares approach to globally int,erpret the optical flow field is proposed. Using the third-order and second-order flows we can obtain a linear constraint on the translation motion. Therefore the translation can be firstly recovered from a global optical flow field with a LS approach. Once the translation motion is recovered we can immediately recover the rotation motion and structure parameters. Since

Introduction

Motion Understanding has received considerable attention lately[lO]. One of the important tasks in motion understanding is to recover the three-dimensional motion and structure of a moving object from a sequence of its images. The most common approach to achieve the task is based on two phases: computation of an optical flow field and interpretation of this field. Much literature on the computation of the optical flow field has been created[lO]. The optical flow fields recovered by the existing techniques, however, are noisy and partially incorrect, especially near occlusion or motion boundaries[9]. This will bring a serious difficulty to the interpretation of such flow fields. In this paper I linit myself in the interpretation of the optical flow field. As we know the optical flow field induced by a rigid body motion is

\\There the 3 0 rigid motion is represented by a rctation R = [R,,Ry,RZ]t followed by a translation T = [Vx,,Kit, and 2 is the scene depth. The optiV , cal flow field is denoted ( U , v ) . When the optical flow field ( U , v ) is available, several schemes for recovering motion and structure parameters have been suggested[l][2][3]. The first approach is to reduce equations( l)(2) into the following nonlinear constraint equation by eliminating the depth 2 from the above equations
U

- XYRX

- (1

+ x2)R, + wR,
v,

xv, -

668

0-8186-2910-X/92 Q 1992 IEEE $3.00

in this new approach the nonlinearity has been eliminated, and the whole optical flow field is used in a LS form. Therefore more robust interpretation of motion can be expected. In this paper we also give a simple inethod to estimate F O E from the global optical flow field. Different from the existing approach to estimate F O E , our approach is able t o compute F O E from the optical flow field partially induced by the rotation motion without derotation.

Optical Flow Field Induced by a Rigid Body

The well-known equations of optical flow field inhave been shown as flow field is denoted

It is seen from equations (8)(9) that in the case of the planar patch (6) the optical flow field is given exactly by second-order polynomials in the image coordinates. Therefore we can draw a conclusion that the optical flow field produced by a moving planar surface is an exact second-order field. This conclusion was originally obtained by Longuet-Higgins and Prazdny[2], and lately was further obtained by Waxman, Subbarao, et a1[8][7].

Optical Flow Field of A Curved Surface

( F O E ) is defined as
(4)
Yj

For a curved patch, if we want to represent the depth 2 in the spatial coordinates X and Y , the second-order as well as higher-order polynomials in the spatial coordinates will be involved. For the simplicity, let us examine the simplest curved surface: a
quadric surface.

v v vz

(5)

A quadric surface can be described in the spatial coordinates by

2=20

If V, = 0, the F O E will correspond to the infinity point on the image plane. The estimation of the F O E Crom an optical flow field is also a problem of practical interest [5][6]. Equations (1)(2) define an instantaneous optical flow field. Clearly an optical flow field is not only determined by both the translation and rotation of the observer but also influenced by the shape of an object tlirough the depth 2.In the following we will examine how the optical flow field is influenced by the shape of a moving object.

1 1 +px + q Y + -c x x x + - c y Y Y + c x y X Y 2 2

Equation 10) may be approximated in the image coordinates y p = Z - l =Z,l(l-pz-qy--k,,z2

(10)

1 2

- ?k,,Y2

- k,,zy

Optical Flow Field of A Planar Surface

+ 03(z,Y)

(11)

Consider first the planar surface patch. It can be described exactly in both spatial and image coordinate systems as[8]

where 03(z, y) denotes third- and higher-order polynomials in the image coordinates. The normalized curvatures is defined by

(L,a,,) kyy,

= Z O ( C X X , C Y Y ,C X Y )

(12)

Z=Zo+pX+qY

(6)

where p and q are the slopes of the plane. X , Y are the spatial coordinates. Equation(6) can be further rewritten in the image coordinate system

2-1 = Z,l(l-

pz

+ qy)

(7)

Inserting equation (11) into equations (1)(2), it is seen that the optical Row field induced by a curved patch is a higher-order flow (higher than the second order). This conclusion is very intuitive. Since the optical flow field induced by a planar surface is an exact second-order field, for any curved surface its optical flow field is certain to be a higher-order flow.

Substituting equation (7) into equations (1)(2) we have

T h e Recovery of t h e Translation Motion Parameters

1
U=-(-& 20

-a2,)+(--V,--qV,)y-(az
20
20

Since the optical flow field induced by curved patches is a higher-order flow we should make full use of this fact. Carefully examining the optical flow field (1)(2) it is found that there is no contribution from the rotation motion t o the higher-order flow. This fact implies that there exists only information about the translation and the scene shape in the higher-order flow. Therefore we can remove the rotation components from the optical flow field by means of a highorder differential operation.

669

Let p = and take second- and third-order differeiitia,l of equations(l)(2). This will yield the following equalities

0 constraint equa-

-- z 3 x j 0

asv

a3v -xj
ay3

aZv + 3asu f + ( 3 7 + Y ax ax

2-

a3v

- y-)asu = 0 (21)
ax3

ax3

-3 y j

a3U

ay

+ (3-a y 2

a2u

- 2-

a3v
ay3

+ y-)asu ay3

= 0 (22)

- = x0--+3v,-

a U 3

a3U - -BY3
Rliere xo = xV, - V,.
aZv a22

ax3

ax3
d3P ay3

d3P

a2P
ax2

(14)

20-

Therefore we immediately obtain a least-squares solution to the FOE. This is a simple and global method to estimate the FOE from the optical flow field. The most essential feature of this method is that it is able to estimate the FOE from the optical flow field induced partially by rotation motion without derotation.

6
-

yo-

d2P
ax2

The Recovery of the Rotation Motion Parameters

Once the translation motion is recovered we immediately obtain the following constraint for the rotation motion:
(U

a3v - -aX3

d3P
YO- ax3

- (XY%

- (1

+ X 2 ) f l y + Y%))Yo
X%))XO

where yo = yV, - Vy. Inserting( l G ) ( 18) into (14) and (13)(15) into (17) we have

= (U - ((1
( Y O - (1 XY

+ y2)% - xyay -

(23)

After simple manipulation we have

--

asU a3v a3U v, + -v + (3-aa 22 2v + z-ax3 - y-)V, 3 ax3 ax3 aX

d3,u

= 0 (19) = 0 (20)

+ Y2)X0)% + (XYXO - (1 + X2)YO)fiy + (xxo + YY0)Qz = uyo (24)

2x 10

-vz ay3

d3v

-v Y + ( 3 7 - 2 ay3 + y-)V, ay3 ay ay3

a2U

d3u

a3v

a3U

We see from equations (19)(20) that these equations are still valid when both sides of these equations are multiplied by a scale factor. This illustrates that we can not recover three translation components simultaneously, we only recover the direction of translation[. It is worth noticing that the constraint equations (19)(20) for the translation motion are linear and global. The linearity and globality of the constraints are very important from a practical point of view. Due to the linearity nature we could develop a fast algorithm to recover the translation motion. It is particularly important for the use of optical flow information that the translation motion is globally recoverable from the optical flow field. This is because the optical flow field itself is noisy and the noise in the measured optical flow is further accentuated by differentiation. If we locally use the noisy derivatives of optical flow to recover the motion parameters, then the recovered motion parameters will be unreliable. But when these noisy derivatives of optical flow are utilized globally, the inherent noise will be suppressed. Thus reliable motion parameters can be expected. This statement has been proven by our quantitative experiments. Therefore using the higher-order derivatives of
optical flow is not necessarily a draw-back, the key lies i n how t o use these noisy derivatives, locally or globally.

Obviously, the constraint equation (24) is linear for the rotation motion parameters. Therefore the rotation motion can be robustly recovered from the global optical flow field by a Least-Squares method. It is noteworthy that the recovery of the rotation motion only use the original optical flow field, no derivative of optical flow is involved. Therefore the estimated rotation parameters are more reliable.

The Recovery of the Scene Structure

Once the motion parameters are fully recovered we can c o m p t e the scene structure with the following methods: The first method is to recover the depth 2 directly from equations(l)(2). With the depth map Z(x, y) we can compute any scene structure parameters. The second method is to recover the scene structure from the derivatives of the parameter p. Since the motion parameters have been recovered we can immediately obtain the xo and yo. Then we have

1 a3u d3P - - -ay3

- a3p

ax3

- yo ax3

xo ay3 1 asv --

As a by-product of the recovery of the translation motion, the FOE can be easily obtained. Using the

Making use of the corresponding relationship between the parameter p and the scene parameter, we have
cyy

= 2--1 aZu

x o ay2

(29)

Clearly we are incapable to interpret the optical flow field induced by a moving planer surface using a least-squares method. The recovery of motion parameters of a moving plane is surprisingly difficult! This is beyond my imagination. The similar conclusion had been drawn by Longuet-Higgins in 1980[2].

References
For the recovery of the other structure parameters we can utilize similar correspondancies.
[l] Tsai and Huang, Uniqueness and estimation of three-dimensional motion parameters of rigid objects with curved surfaces, IEEE Trans.PAM1, ~01.6,No.1, ppl3-16, 1984.

8
8.1

Discussion
Now we discuss some special cases.

The quadric patch case

we further extend 0 ( x 3 ) in the equation (11) to t,liird-order term as

0 ( ~ 3 ) =

[2] Longuet-Higgins and Prazdny, The interpretation of a moving retinal image, Proceedings of the Royal Society of London B 208, pp385-397, 1980. [3] Bruss and Horn, Passive navigation, CVGIP 21, pp3-20, 1983. [4] Aloimonos, Weiss and Bandopadhay, Active vision, Int. Journal Comp. Vision. 1988 [5] R.Jain, Direct computation of the Focus of Expansion, IEEE Trans.PAM1, vol 5, No.1, 1983.

~ ( 3 )q

X 4 )

(31)

where

1 1 1 F(3) = --(-krrpx3+ -kyvqy3 2 20 2 1 1 ( p Y y P k r y q ) x Y 2 ( + m q kZ,P)X2Y

(32)

Inserting equation (11) into equations (l)(2) we have

[6] Negahdaripour and Horn, A direct method for locating the Focus of Expansion, CVGIP vol46, No.3 1989. [7] Subbarao, Interpretation of visual motion: A computational study, Ph.D. thesis, University of Maryland, Dept. of Computer Science, 1986. [8] Waxman and Wohn, Image flow theory: A framework for 3D inference from time-varying imagery, Advances in Computer Vision, ed. by Brown, 1988. [9] Verri and Poggio, (Against quantitative optical flow, Proc. 1st ICCV, pp171-180, 1987.

- - - --Cxx(K - P K )
ax3

1 2 U 1 - - - ---CYYqK ay3 2 asv 1 - - - -fxxPVy 8x3 asv 1 - = --Cyy(V*-qVg) ay3 2

a3U

(33) (34) (35) (36)

a3

Therefore for a special curved surface - a quadric surface the third-order optical flow field exists. So the translation motion can be, in principle, recovered from t,lie linear constraint8 equations (19)(20). Obviously, this conclusion is also valid for the more general curved surfaces as their third-order optical flow fields are certrain to exist. 8 . 2 The planer surface case Forin the equations (1)(2) we know that the optical flow field induced by a moving planer surface is an exact second-order field. So we have

[lo] Aggarwal and Nandhakumar, On the computation of motion from sequence of images: A review, Proceedings of the IEEE, pp.917-935, 1988.
[ll] Haibo Li, Direct computation of Time-toCollision from the spatio-temporal derivatives of images, to be submitted. [12] Haibo Li, Time-to-Impact from optical flow field, t o be submitted. [13] Haibo Li, Direct Location of the Focus of Expansion from the spatio-temporal derivatives of images, to be submitted.

a3u - -- 0

ay3

a -3 v -

ax3
a3v

O 0

- --

BY3

67 1