IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 18, NO.
10, OCTOBER 1996 1029
Analyzing Looming Motion Components From
Their Spatiotemporal Spectral Signature
Phillipe Burlina and Rama Chellappa
Abstract-This paper addresses the use of spatiotemporal transform
methods applied to the analysis of dynamic image sequences and the
characterization of image motion. Image motion including a divergent
component (resulting from a looming camera component) is analyzed
in the spatiotemporal Mellin Transform (MT) domain, resulting in the
separation of the spectrum into two parts: a structural term
corresponding to the spatial MT of the static image, and a kinematic
term depending on Time-to-Collision (a motion support). We examine
potential applications of this property for the recovery of image motion
from integral image brightness measurements and the computation of
Time-To-Collisionusing spatiotemporal MT analysis.
Index Terms-Motion analysis, frequency domain analysis, time-to-
collision, Mellin transforms, spectral structure.
4) --
I INTRODUCTION
THIS paper examines the spectral signature associated with spatio-
temporal observations of image motion that includes a divergent
motion component. Divergent motion arises from a non-zero
looming (translation along the optical axis) component of the
observer’s motion. Looming detection mechanisms are essential to
many visually controlled tasks [131.
Previous works in image motion analysis based on spatiotem-
poral spectral approaches [SI, [71, [91, [111have addressed the case
of image motion characterized by velocity vectors constant in
space across the surface patch of interest at any time instant, and
possibly varying in time. This type of motion is usually assumed
locally, but if satisfied across the entire image, it will have resulted
from 2D camera translation parallel to the image plane and at constant
depth. This purely translational image motion (i.e. parallel trans-
lation) yields a 3D spectrum with two multiplicative terms [5]
corresponding to:
1) the 2D spectrum of the stationary image, and
2) a spectral component depending on the actual motion law.
These results have been applied to motion flow estimation, stabili-
zation and independent motion detection. More details on these
results are given in Section 2, which formalizes the problem at hand.
This work addresses the case of motion involving a nonzero
looming camera translation component, for which the previous
spectral approach is incomplete. Such camera motion introduces a
divergent flow in the image motion field, which is inherently more
difficult to interpret in the frequency domain: divergent motion
gives rise to image trajectories with non-zero temporal derivatives
of arbitrary order. We show in Section 3 that this type of motion is
appropriately analyzed in the spatiotemporal Mellin transform
(MT) domain (defined in 141). This transformation yields the
desired separability into kinematic and structural elements: The
spatiotemporal MT of the full dynamic image can be factored into
the known spatial MT of the original image at rest, and a motion
MT support depending on Time-to-Collision (TTC) through a
The authors are with the Center for Automation Research and the Department o f
Electrical Engineering, A.V. Williams Building, room 441 7 , University of
Mayland, College Park, M D 20742. E-mail: [burlina, ramal@cfar.umd.edu.
Manuscript received Dec. 13, 1994; revised Apr. 19, 1996. Recommended
for acceptance by S.Peleg.
For information on obtaining reprints of this article, please send e-mail to:
transpami@computer.org,and reference IEEECS Log Number P96046.
0162-8828/96$05.0001996 IEEE
hypergeometric function. These results can be used for the design
of TTC operators based on integral brightness measurements. Spe-
cial cases of MT parameters are studied in Section 4, as each point
in the Mellin parameter space leads to a different operator.
2 MOTIVATIONS
AND PREVIOUS WORK
We are interested in the following problem: given image point
trajectories,
r(t)= f(r, t), r(O)= ro
and under the assumption of image point brightness constancy inside
a temporal window of length A T
I(r (t),t) = Io(r(O)), t E AT (1)
we seek an appropriate integral transformation TII(.,.)l(ss, s,)
achieving the factorization of the spatiotemporal intensity function
I(r, t ) in the transform domain into a term depending only on the
structure of the image at rest TIIo(r)](s,) and a term M(s,, s,)
depending on the actual motion parameters, i.e.,
Such a transformation exists only in specific cases of rigid camera
motion. For more insight into this matter, the reader is referred to
[14], in which a related problem is investigated for the case of spatial
transforms applied to the design of invariant descriptors.
The specific case composed of the Special Euclidean group
SE(2) of image transformations can be analyzed using simple
Fourier operators. Consider for a moment the case of 2D translations.
fl,
Let R.),I(r, t), lo(ro),I (k, I&k) respectively denote the Fourier
operator, the dynamic image intensity as a function of the image
position r and time t, the still image at time 0 and their respective
Fourier transforms. If all image points undergo an exact two-
dimensional uniform translation, i.e., r(t) = ro+ Vt, and if the brightness
constancy constraint in (1) is satisfied, then the spatiotemporal
Fourier transform of the dynamic image intensity satisfies
I (k,fl = I&) &f+ V Tk) (2)
From (2), the spectrum of an image generated by a uniform parallel
translation admits an impulsive plane whose slope is determined
by the image velocity vector. Based on this observation, [91 used a
least square estimation of the slope of this plane from the output
of Gabor bandpass filters. Similar spatiotemporal energy filters are
studied in [ll]. In [21, this geometric result is used to justify a
multi-scale incremental motion estimation/stabilization procedure.
More recently, the situation of pure two-dimensional translation
with polynomial motion laws was studied in [5]: The Fourier
transform of the image intensity was derived for time-polynomial
and space-constant velocities and geometrically interpreted for up
to third order motions. Higher-order derivatives result in a power
spread outside of the original impulsive plane. Polynomial and
periodic laws were also examined in [31, where the image motion
estimation is converted into a problem of phase estimation for
polynomial phase or phase modulated signals. Similar results
carry over to the 2D rotational motion case.
OF DIVERGENT
3 MELLINTRANSFORMS IMAGERY
3.1 Assumptions
Consider the more general case of arbitrary camera motion in-
cluding a non-zero looming component. Let P : [ X , Y, Zl be a point
in 3-space projected in the image at r so that
r = P(P) + r,
where P(P) = [fc X / Z , f c Y/ZIT (fc and r, are the focal length and
image center position). Under an arbitrary camera motion, i.e.,
1030 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 18, NO. 10, OCTOBER 1996
P= - o x P - V , image point trajectories satisfy
with I,+,. the Focus of Expansion (FOE), and the rotational terms
given by
The image sequence can be appropriately warped so as to cancel
out the additive contribution of the rotational flow r,. The authors
have reported a method to achieve this derotation procedure, and
the interested reader is referred to [17].When this is accomplished,
(3)yields
or equivalently s ( Z ( r - rfoc))=0. Without loss of generality,
image point positions are now measured with respect to the FOE,
assumed tracked, i.e., ~f,, = 0. We therefore proceed with the alge-
braic measure Y only. Then we have & ( Y Z) = 0.This in turn yields
the following time-trajectory for r(t) for divergent points: If the
observer's looming motion follows a general polynomial law (i.e.,
Z'"' = 0, n > k for some order k, the exponent in parenthesis denoting
the order of the time-derivative), we have
r(0) Z ( 0 ) = Y ( t ) Z ( t )
and using the notation a"] = for all Z # 0 this yields
The special case of looming with a uniform law (or uniform
looming), i.e., Z'n) = 0, n > 1, yields
-1
where z ( t ) = - (a'l'(t)) = - Z / Z (respectively, zo) represents the
TTC at time t (resp. time 0). The TTC will be assumed spatially
constant across the image intensity profile under analysis. For later
use, note that TTC simply satisfies i ( t )= - 1and .(E) = zo - .
At this point, it is worthwhile to draw the reader's attention to
the following observation. Consider the simpler uniform looming
case, for which d(rZ) = 0 yields Y ( ~ Z) + nu("-')Z = 0 , which
finally leads to Y ( ~=) (-l)"n!($)" Y. Therefore, looming motion
components, even when uniform, generate image trajectories with
non-zero temporal derivatives of arbitrary order. As a result, the
frequency domain interpretation of uniform looming motion is
inherently more complex than that of parallel translation. Time-
polynomial laws for the image velocity cannot completely describe
this type of motion.
We now turn to the Mellin transform. The MT of a real function
f ( x ) with support on the positive real line is defined as a general
integral transformation with kernel equal to xs-',s E C [4]
.Z[f(x)] (s) = Jl-=f(x)Pdx
with s such that 1,< Xe(s) < 4.Previous applications of MT's have
mostly related to object recognition for the design of invariant
descriptors [15]. We consider here the application of a 2D MT to
divergent dynamic imagery, where the MT is applied along the
time and radial directions. Note that along the angular dimension,
the choice of the actual transform operator is not significant since
the angular coordinate remains invariant under looming. Any
arbitrary transformation is suitable along that dimension, and the
angular component participation is omitted in the following
derivations.
3.2 MT of a Complete Polynomial Divergent Sequence
Consider the case of polynomial looming observed from the time
origin to completion zo. We show next that the MT applied to this
spatiotemporal input yields the desired factorization into struc-
tural and kinematic components. This case reveals a simple ex-
pression for the motion support. Taking the 2D MT along the time
and radial directions on a dynamic radial intensity profile result-
ing from polynomial looming, and using (5),we get for t E [O, %]
(4)
Letting ,U = r(1 + ck t=l
J'l(n)
7t ' ) , this in turn yields
aqo)
+
=~M[I,(~)](z,~NM[(~ ~ t Y ( z I j
,=1
(5) The application of the MT on spatiotemporal radial observations
of dynamic imagery generated by looming motion with polyno-
mial laws results in the desired factorization of the spectrum into a
motion support and a structural MT component. In the case of
uniform looming, this specializes to
q I ( L t ) ] ( z r , z , )= q,
(Y)](zr)7:P(1 -ZrrZ*) (6)
with Xe(z,) < 1, Re(z,) > 0. The derivation follows immediately from
the definitions of the MT and the Beta function fl.,.)with support
continued over complex variables.
3.3 MT of an Incomplete Divergent Sequence
Consider the Gauss hypergeometric function series F2,1(., .; .;.) (see [4]),
xi=,Wx'
with'
defined as: F2,,(a,b; c; x ) = (a), T(a + n)/T(a).
=
itself We first quote a result from [lo] needed in this section:Assume I x I < 1,
Re(zJ > Re(zz)> 0, then
Consider the more general case, where the looming motion is
known only up to current time instant E, i.e., only part of the total
looming sequence I(r, t ) is observed, for t E [0, t] where t < q,. This
case is relevant for the possible extraction of TTC (zo or equiva-
lently de)). Call J&) the indicator function on the set [0, E]. We
seek the spatiotemporal MT of the function I(r, t)J&). In this case,
we obtain the desired factorization with, in the case of uniform loom-
ing, the motion support Y(z, ZJ expressed in terms of F2,1(., .; .;.) and
TTC. Indeed, if t < TO, we have
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 18, NO. 10, OCTOBER 1996
where the recursive property r(z + 1) = z T(z) was used. The
Gauss-Kummer identity [lo] applied on (8) yields the limit (6)
found in the previous section when E + zO.
4 APPLICATION
TO SPECIFIC MELLINPARAMETERS
Equation (8) leads to various interpretations and relationships
between motion support and TTC, some of which are used and
examined here. In the following, the Mellin analysis of incom-
plete sequences is applied to the derivation of integral spatio-
temporal TTC operators. Various Mellin parameters subspaces
are considered. One possibility consists of evaluating (8) for
integral or rational values of the Mellin parameters, leading to
arbitrary space-time image moments. Another entails consid-
ering spaces leading to special spectral interpretations. These
possibilities are explored next.
4.1 Integer and Rational Mellin Parameters
The case of positive integer Mellin variables, i.e., (zr, zf) E .Y?,
yields expressions involving arbitrary-order spatiotemporal
moments of the intensity function. Letting zt = k E N and z, =
m E A( in (9) leads to
Mm,k= N I I ( r , t ) ] ( z r= m,z, = k )
= j?t jOwdr I ( r , t)rm-'tk-'
( n+ m - (e)n
= s ( m, -I)!
L 2
n=o k+n n!
where for simplicity the known Mellin transform of the image
at rest evaluated at z,= m is denoted by S,, and corresponds to
the ( m - 1)st order spatial moment of the original image at rest,
i.e., N [ I , ( r ) ] ( m )= S,. In particular, we have
1031
from which various TTC expressions may be derived from zero
order spatial and low time-order spatiotemporal moments, as for
examp1e
where TTC may be derived numerically. Conversely, moments of
zero order along time and arbitrary order along space lead to the
following expressions for any m 2 1:
The particular cases m = 2 and m = 3 yield polynomial equations in
z,, leading to
The use of moments for computing TTC is illustrated on
synthetically generated intensity profiles taken along the radial
direction and resulting from looming camera motion. This in-
put could be obtained, for instance, from a radially-sampled
image or a polar retina. (similar to the retina described in 1161).
Fig. 1 shows radial profiles at various time instants for t = 0,
0.1, ., 0.8 by time increments of 0.1. In these figures, time flows
(and the intensity diverges) from top to bottom, while the spatial
origin of the image lies at the left of the profile. Without loss of
generality, time is normalized, i.e., z0 = 1. Alternatively, this
dynamic radial profile can itself be thought of as an image to
which the 2D Mellin transform is applied. The transformation
is applied to a partial section of the image, from the top (time
origin) to some E < 1 = zo. The application of moments in the
determination of zo is illustrated for lower order moments, i.e.,
M1,l (represented by a full line), M,,, (dots), M3,1(dashes) and
(dash-dots) (the actual TTC is displayed as the bold line).
The resulting calculated TTC is shown in Fig. 2. For and
Ml,2, the TTC is derived by using a Newton-Raphson method.
One aspect of the calculation consists of accounting for bound-
ary effects in the calculation of the MT. To address this prob-
lem, we use a prominent feature and compute the moment on a
bounded spatial profile up to that feature. We then use a pro-
cedure interleaving the moment-based computation of TTC
and the feature tracking using a Kalman filtering scheme.
Fig. 1. Radial intensity I (r, t).
1032 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 18, NO. 10, OCTOBER 1996

In general, the motion support introduces a phase shift in the time-

space Mellin transform. Using this fact, d.)may be expressed as

for f # 0 and a # 0. Fig. 3 shows TTC calculated from the phase

operator: TTC is plotted versus elapsed time, obtained from the
calculated phase of the MT at f = 0.02, 0.04, 0.06, 0.08 (as shown
from upper left corner and clockwise). This result is obtained from
the radial intensity profile I&) and the corresponding dynamic
sequence I(r, t ) shown in Fig. 1. Iff = 0 and a = n E N +then this
reduces to a case of spatiotemporal moments leading to

Fig. 2. TTC computed vs. actual.

(14)
Similar TTC operators can be derived from rational Mellin
parameters, either implicitly or explicitly, such as

1
2s&
2
; 1 I
z0 = 2'2 .
-4& ' M$2 0x 08

G(&+&qj?
~

1 ~

2'2 2 06 0.6

which result from basic properties of the hypergeometric function. 04 0.4

The use of spatiotemporal rational moments should nevertheless I1 2 02

be avoided when min(Xe(z,), Re(z,)) I1/ 2 . This constraint is justified 0

0 02 04 06 08 0' 0.2 0.4 06 0.8
in a subsequent discussion paragraph.
1 1
4.2 A Spectral Interpretation
08 0.8
Particular spectral interpretations arise when choosing Mellin
parameters along planes defined by ( z , Z J = (a,+ j 2 n fi, a,+ j 2 n ft).
06 0.6

U4 0.4
An example of such a case is reported here. Using the Binomial
0.2 02
formula, it may be shown that F(z,, 2,; z,; z ) = (1 - z)-"".Therefore,
0
on planes defined by z,= zt + 1, ( 8 )becomes '0 02 04 06 OX 0 0.2 04 06 0.8

34[W)J,(f)](Z! + 1 , z t ) = ~ [ I f l ( 4 ] +1)[+T
(z, z;' (10) Fig. 3. TTC from phase

The last equation may be the basis for further moment expres- 4.3 Discussion
sions. We use it first for the more general case of Mellin trans-
Some observations are in order concerning the application of spa-
forms evaluated along particular subspaces yielding spectral
tiotemporal MT as a tool for divergent motion analysis. Because of
interpretations and given by (zr, Z J = ((a+ 1) + j 2 n f, a + j 2 n fl. its reliance on MT, this method has limitations resembling those
In this case ( 8 ) leads to involved in the use of global operators. These are limitations
similar to those of moments (such as boundary effects, memory
effects for lower-order temporal moments, redundancy and
greater noise sensitivity for higher-order moments, see 111). The
behavior of the above phase- and magnitude-based TTC operators
can be shown to closely depend on the selected value of a. For
smaller values of a the performance of these operators degrades.
Similarly, faster "convergence" results from use of higher-order
moments along with greater noise sensitivity. One explanation
lies in the general form of the equivalent Parseval's formula for
using the notation 3 4 [ . l ( a + j 2 II f, b + j 2 rI fl = N7a,b[.1(fl
and the Mellin transform reported in [SI: For -1 < a < 1/2, values of
n/l[.l(a + j 2 n fl = N_?,[.](fl. If a = 0, the motion support adds the transformed function closer to the time origin are overem-
only a phase component to the overall spectral distribution. phasized in the resulting Mellin magnitude spectrum. The
The motion support otherwise alters the spectrum and d.)may converse is true for a > 1/2. For this reason, the use of rational
be recovered from the magnitude spectrum moments for the determination of sshould not be carried out in
cases where min(Re(z,), Re(z,)) < 1/2.See also [11 for a discussion
of related properties of spatial moments. Limitations due to
specific assumptions made by this method should also be further
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE. VOL. 18, NO. 10, OCTOBER 1996 1033

considered. One of these assumptions, the brightness constancy [I51 Y. Sheng and H. Arsenault, “Experimentson Pattern Recognition
assumption in U), is shared by most spatiotemporal filtering ap- Using Invariant Fourier-Mellin Descriptors,” J. Optical Society of
proaches (see for instance [ 5 ] ,[6], [9]). The success of this method America, vol. 3, pp. 771-776, June 1986.
[I61 M. Tistarelli and G. Sandini, ”On the Advantages of Polar and
also depends on the ability to carefully track the FOE and rotational Log-Polar Mapping for Direct Estimation of Time-to-Impact from
motion, which further raises the problem of sensitivity to these pa- Optical Flow,” IEEE Trans. Pattern Analysis and Machine Intelligence,
rameters. More on this may be found in [17]. This method offers vol. 15, pp. 401410, Apr. 1993.
some notable advantages: It relies on integral information which is [I71 Y. S . Yao, P. Burlina, and R. Chellappa, ”Electronic Image Sta-
more robust to noise than differential information. These operators bilization Using Multiple Visual Cues,” Proc. Int‘l Conf. Image
allow sharing of resources for both recognition and motion analysis Processing, pp. 191-194, Crystal City, Va., Oct. 1995.
purposes. Finally, the delicate numerical implementation of the MT
is offset by the possibility of real-time analog MT computation using
optical Fourier processing methods [121. Lastly, we wish to draw the
reader’s attention to the relationship between the MT and other re-
lated filtering schemes to ”processes” observed or proposed in bio-
logical vision: The literature contains ample studies of these and the
interested reader is referred to [61.

5 CONCLUSION
We extended spatiotemporal spectral approaches of image motion
recovery to types of motion including camera looming. We showed
that the factorization of the spatiotemporal spectrum is obtained in
the MT domain. The expression of the MT motion support is derived
for incomplete divergent sequences. The latter relationship involves
a hypergeometric function which can be specialized to particular
values of the Mellin parameters, each yielding an operator for com-
puting the TTC. The properties of these operators are discussed.

REFERENCES
Y. Abu-Mostafa and D. Psaltis, ”Recognitive Aspects of Moment
Invariants,” IEEE Trans. Pattern Analysis and Machine Intelligence,
vol. 6, pp. 698-706, NOV.1984.
P. Burt, R. Hingorani, and R. Kolczynski, ”Mechanisms for
Isolating Components Patterns in the Sequential Analysis of
Multiple Motion,” Proc. IEEE Workshop on Visual Motion
(Princeton,NJ), pp. 187-193, Oct. 1991.
W. Chen, G. Giannakis, and N. Nandhakumar, ”Spatio-Temporal
Approach for Time-Varying Image Motion Estimation,” Proc. Int’l
Conf. Image Processing, pp. 232-236, Austin, Tex., Nov. 1994.
S. Colombo and J. Lavoine, Transformations de Laplace et de Mellin;
Formulaires, Mode d’lltilisation, Gauthier-Villars,Paris, 1972.
G. Cortellazzo and M. Balanza, ”Frequency Domain Analysis of
Translations with Piecewise Cubic Trajectories,” IEEE Trans. Pattern
Analysis and Machine Intelligence, vol. 15, pp. 411416, Apr. 1993.
M. Eckert and G. Buchsbaum, ”Efficient Coding of Natural Time
Varying Images in the Early Visual System,” Philosophical Trans.,
the Royal Society London, Series B, vol. 339, pp. 385-395,1993.
D. Fleet and A. Jepson, ”Stability of Phase Information,” IEEE
Trans. on Pattern Analysis and Machine Intelligence, vol. 15,
pp. 1,253-1,269, Dec. 1993.
S. Goh, The Mellin Transform: Theory and Digital Implementation,
PhD dissertation, Purdue Univ., 1985.
D.J. Heeger, ”Optical Flow Using Spatiotemporal Filters,” Int’l J .
Computer Vision, vol. 1, pp. 279-302,1988.
K. Iwasaki, H. Kimura, S.Shimomura, and M. Yoshida, From
Gauss to Painleni, Vieweg und Sohn, Braunschweig, 1991.
R. Jasinschi, ”Intrinsic Constraints in Space-Time Filtering: A New
Approach to Representing Uncertainty in Low-Level Vision,” IEEE
Trans. Pattern Analysis and Machine Intelligence, vol. 14, pp. 353-366,
Mar. 1992.
N.Konforti, D. Mendlovic, and E. Marom, ”Real Time Optical
Generation of Circular or Mellin Radial-Harmonic Filters,” J. Optical
Society of Amevira, vol. 7, pp. 225-230, Feb. 1990.
R. Nelson and J. Aloimonos, ”Obstacle Avoidance Using Flow
Field Divergence,” I E E E Trans. Pattern Analysis and Machine Intel-
ligence, vol. 2, pp. 1,102-1,106, Oct. 1989.
J. Segman, J. Rubinstein, and Y. Zeevi, “The Canonical Coordi-
nates Method for Pattern Deformation: Theoretical and Compu-
tational Considerations,” l E E E Trans. Pattern Analysis and Machine
Intelligence, vol. 14, pp. 1,171-1,183, Dec. 1992.