1114

IEEE TRANSACTIONS ON PAITERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 16, NO. I 1, NOVEMBER 1994

Correspondence
Computation of Surface Geometry and
Segmentation Using Covariance Techniques
Jens Berkmann and Terry Caelli
Abstract-In this correspondence, the application of covariance techniques to surface representationof 3-D objects is discussed and such ways
of computing surface geometry are compared with traditional methods
using Differential Geometry. It is shown how the covariance method
provides surface descriptors that are invariant to rigid motions without
explicitly using surface parameterizations or derivatives. Analogous covariance operators for both the Gauss and Weingarten maps are defined
and a range image segmentation technique is presented that labels pixels
as jump or crease discontinuities or planar, parabolic or curved region
types.
Index Terms- Differential Geometry, covariance, Gauss map, image
segmentation, jump map, crease map, region segmentation.

I. INTRODUCTION
Object recognition systems based upon range data typically compute surface shape interms of surface normals or curvatures as
developed from Differential Geometry (see, for example [l], [2],
[3], [5]). However, problems do occur with implementing these descriptors, notably, the need for surface parameterization and adequate
smoothness to support the use of Differential operators. Further to
this, such curvature measures are not necessarily robust under additive
noise as they involve nonlinear combinations of partial derivatives.
For these reasons we have further explored an altemate signal
processing basis for computing local shape measures4ovariance (as
recently used by Liang and Todhunter [4])--and show how invariant
surface shape operators can be developed with this statistic.
The covariance approach dispenses with surface parameterization
and provides invariant descriptions of shape via the eigenvalues of covariance matrices of different orders. The aim of this correspondence
is to further investigate the properties of the covariance approach
and to propose definitions of the Gauss and the Weingarten maps
derived from covariance matrix operations. Finally, we propose a new
range image segmentation technique using this covariance approach
to segment the surface into planar, parabolic, and curved regions as
well as jump and crease discontinuities.

where gz = (xz,yt, 2,) correspond to the projection plane (I,y)

and depth (2) values at position i; gm= 1/n E:=, g, to the mean
position vector; n to the total number of pixels in the neighborhood
of gz used to compute (1).
The eigenvalues and eigenvectors of CI determine three orthogonal
vectors two of which define a plane whose orientation is such that it
minimizes, in the least square sense, the (squared distance) orthogonal
projections of all the points (in the neighborhood window) onto the
plane. Liang and Todhunter [4] originally proposed this plane as a
reasonable approximation to the surface tangent plane and, so long
as the two eigenvectors are chosen to preserve the sidedness of
the surface points, it can be viewed as an analogue to this. That
is, the eigenvectors corresponding to the two largest eigenvalues
(XI and XZ) form this tangent plane, C1 plane, if sidedness is
preserved, otherwise the plane must be formed by (XZ and Xz). This
sidedness-preserving or topology preserving principle, can be seen
as a replacement of surface parameterization. It is important to note
that the eigenvalues of CI are already invariant to rigid motions.
An analogous definition of the second fundamental form follows.
We define, about a point 2 on a surface, a two-dimensional covariance matrix in the following manner:

with the two dimensional vectors -*y defined by the projection:

Y = 82

--2

. (E* - %
)proj,
I

where ;c2is in a small neighborhood of %I. We project the difference

vector that points from 5 to gt onto the tangent plane as determined by (1) and weight the resulting two-dimensional vector by
distance s, , which measures the orthogonal distance from the tangent
plane (determined from C I )to point ;c, . Given both tangent vectors
tl and t2 and the normal vector 14 at point 5 derived from (1) we
may write (3) more explicitly:

st

We can then define the quadratic form

IIC = J

APPROACH
11. COVARIANCE
Following Liang and Todhunter [4] we define the local (first order)
surface covariance matrix ( C I )as:

Manuscript received November 23, 1992; revised December 1, 1993. This

project was funded in part by grants from the Gottlieb Daimler and Karl
Benz Foundation and the Australian Research Committee. Recommended for
acceptance by Editor-in-Chief A. K. Jain.
J. Berkmann was with the Computer Science Department, the University
of Melboume, Parkville, Vic. 3052, Australia. He is now with the Institute
for Communications, Technical University of Munich, D-80290 Munich,
Germany.
T. Caelli is with the Computer Science Department, the University of
Melbourne, Parkville, Vic. 3052, Australia; e-mail: tmc@vis.mu.oz.au.
IEEE Log Number 9405786.

(3)

.CII .c

(5)

as a second fundamental form (Weingarten map) based on covariance methods with the defined covariance matrix according to
(2) and a chosen unit vector in the tangent plane. Analogous to
classical computations of surface geometry, we define the principal
directions as those directions that are given by the eigenvectors of
this covariance matrix. In contrast, Liang and Todhunter [4] estimated
principal directions by calculating the covariance matrix of a given
normal map. They selected the two eigenvectors of the covariance
matrix corresponding to the two largest eigenvalues as estimations
of principal directions.
An altemate way of computing such principle directions, since they
lie on the tangent plane anyway, is to calculate the covariance matrix
from the projections of the normal vectors, within a neighbourhood of
a point, onto the tangent plane and define the principal directions as

0162-8828/94$04.00 0 1994 IEEE

--

1115

TRANSACTIONS ON PAITERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 16, NO. 11, NOVEMBER 1994

STEP 3: Region segmentation. In this stage pixels neither being

labeled as jump nor crease are labeled as planar, parabolic (developable) or curved. In order to do so the covariance matrix CI
according to (1) is calculated in a 7 x 7 pixel neighborhood at each
pixel of the range image. The eigenvectors of this covariance matrix
are, again, used as input for the next stage, which calculates the
covariance matrix C, according to (6) from the projected normal
vectors in a 7 x 7 pixel neighborhood where only pixels that have
not been labeled so far as jump or crease are taken into account. Two
thresholds for each of the two eigenvalues of this covariance matrix
i= 1
are applied. Pixels with smaller values of both eigenvalues than the
thresholds are labeled as planar. Pixels with a smaller value of the
with the two dimensional vectors vi defined by the projection:
smaller eigenvalue than the threshold but with values of the larger
(7) value beyond the corresponding threshold are labeled as parabolic.
Pixels not meeting above conditions are assigned the label curved.
We have obtained experimental results for the proposed segmenwhere &,i2 are the estimated tangent vectors obtained from (1) and
assigned to the point (I, y, z ) &tthe center of the current window, tation technique using many synthetic range images: 2-D arrays. The
with 14% being the normal vector obtained by (1) at a position range images were not filtered or preprocessed in any way. However,
(I,, yz,zz)in the neighborhood of (2, y, 2). The two eigenvalues we have computed surface Mean and Gaussian curvatures using the
of this covariance matrix provide information about the intrinsic biquadratic surface algebraic form proposed in [l] and the jumpdimensionality of the distribution of the projected normal vectors and crease-edge detection technique presented in [5]. Together these
on the tangent plane, analogous to the Gauss map defined above region and boundary detection methods are termed the Smoothed
(see (5)). The normal vectors in the neighborhood of a locally flat Differential Geometry, or SDG method.
Here, the biquadratic surface fitting technique originally labels the
surface are, after projection onto the tangent plane, simply mapped
into one single point, namely the point the tangent plane is assigned range image into its eight surface types based on the sign maps of
to. As a result both eigenvalues will be zero (0-dimensional). In the Gaussian and Mean curvature after setting two symmetric thresholds
case of a locally developable (parabolic) surface the projected normal h,,,, and H,,,, respectively. In order to compare this method to
vectors are mapped onto a straight line. Therefore one eigenvalue our approach, pixels with elliptic or hyperbolic labels were joined and
will be very small (1-dimensional). Finally, a locally curved surface assigned the joint label curved irrespective of their Mean curvature
manifests itself in two nonzero eigenvalues (2-dimensional) due to label. In addition, pixels with labels ridge or valley were assigned the
the distribution of the projected normal vectors on the tangent plane. label parabolic. The planar label corresponded to absolute values of
The eigenvaluesof all these covariance matrices are invariant to rigid Gaussian and Mean curvature that were both smaller than the K,,,,
motions.
and H,,,, thresholds. These thresholds K,,,, and H,,,, were set
manually in order to achieve best discrimination.
The jump-detection technique we used is similar to the method
111. REGIONAND DISCONTINUITY
DETECTION
described in [5]. The jump-edge magnitude M3,, is computed as
We have used CI in (1) and C, in (6) for range image segmentation
the maximum difference in depth between a point and its neighbors
into planar, parabolic and curved regions as well as jump- and
in a 5 x 5 window. This is formulated for a point (z,y) as follows:
crease-edges. The procedure is as follows.
STEP I: Jump-edge detection. The covariance matrix cr,accordMJUmp(z,
Y) = max{lz(z, y) - z ( z k, Y I ) [ : -2 I k,1 I 2)
ing to (l), is calculated in a 5 x 5 pixel neighborhood at each
(8)
pixel of the range image. The eigenvalues and eigenvectors of this
4.
covariancematrix are evaluated using the fast Jacobi-rotation routine.
The maximal eigenvalue of this covariance matrix CI is utilized as where z represents the depth value. Pixels with values of the edge
a jump-edge detector. Pixels with values of the maximal eigenvalue magnitude bigger than a certain threshold are assigned the label jump.
The crease-detectiontechnique we used for comparison purposes is
larger than a certain threshold are labeled as jump in the sense that
only one direction or orientation is present and that a curve-model is entirely due to [5]. The crease-edge magnitude M,,,,,, is computed
more appropriate than a surface-model. The eigenvectors calculated as the maximum angular difference between adjacent unit surface
normals in a 3 x 3 pixel neighborhood. This is formally given as:
in this stage are used for the next step.
STEP 2: Crease-edge detection. The covariance matrix C,, acM c r e a s e =max{arccos [E(I,
Y ) ~
cording to (6), is calculated in a 5 x 5 pixel neighborhood at each
. E(. k,y Z)] : -1 5 k,1 5 l}
(9)
pixel of the range image. The eigenvalues and eigenvectors of this
covariance matrix are evaluated using the fast Jacobi-rotation routine.
The maximal eigenvalue of this covariance matrix C, is utilized as where 3 denotes a unit surface normal obtained from estimates of the
a crease-edge detector. Pixels not already labeled as jump and with first partial derivatives. The crease-edge magnitude is thresholded.
values of the maximal eigenvalue larger than a certain threshold are Pixels that have not been labeled as jump are labeled as crease if the
labeled as crease. That is, creases differ from jumps in so far as they computed crease-edge magnitude lies beyond the threshold.
In the final segmentation stage the edge maps are superimposed
have a range of surface normal variations in one direction (except
onto
the region map (planar, parabolic, curved) given above.
for comers). It is important to note that the projection of the normal
We have analyzed fifteen range images using both methods (SDG
vectors onto the estimated tangent plane can be dispensed with at this
stage of crease detection, since only the major variance is taken into and covariance) and an example of the resultant segmentation is
account and this is due to the abrupt changes of the normal vectors at shown in Fig. 1. The covariance region segmentation technique

the eigenvectors of this two-dimensional covariance matrix. In addition, this definition enables us to establish a segmentation technique
for planar, parabolic, and curved regions based on the observation
of the characteristic clustering of the projected normals onto the
tangent plane.
Consequently, we have an analogous operator to the Gauss map
of classical differential geometry. In this way we define the twodimensional covariance matrix:

creases. The projection onto the tangent plane plays a more important
role in the following stage of region segmentation.

In [5]a 3 x 3 neighborhood was chosen.

1116

IEEE TRANSACTIONS ON PATIERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 16, NO. 11, NOVEMBER 1994

method and image, which gave best results in terms of the entropy
measures before comparing them.

IV. DISCUSSION

The aim of this paper was to extend the covariance technique

introduced by Liang and Todhunter [4] and to develop an analogous
definition of surface descriptors without using Calculus. Furthermore,
we have defined an analogous operator to the classical Weingarten
map (second fundamental form) and the Gauss map on the basis of
covariance matrices. The eigenvalues of the covariance matrices are
invariant to rigid motion as well, since the computation operates only
on the tangent plane. In addition, we have shown how the covariance
method treats discontinuities in a very natural manner.
Finally, since covariance methods do not rely on the definition
of a surface in terms of a surface parameterization, the spectrum
of the covariance matrix (the full set of eigenvalues) provides us
with a type of smooth transformation between lines, surfaces and
volumes. One nonzero eigenvalue defines merely linear components,
while a solution of three equal eigenvalues corresponds to data of
uniform density in a volume about a point-somewhat analogous to
a local fractile dimensionality. Further to this, covariance methods
provide ideal ways of treating signals embedded in additive Gaussian
or white noise and so may provide further integration of geometry
with principles of signal processing.
However, it should be remembered that what we have investigated
here is an analogy to Differential Geometry and we do not claim
to have a complete alternative to the rich set of surface properties
derived by that approach. Rather, we have been able to derive
some simple surface properties based on a standard signal processing
measure that does not need surface parameterization nor the Calculus.

Fig. 1. Firsr row: Original range image. Second row: Combined jump-and
crease-edge maps for SDG (left) and covariance (right) methods. Jumps
have been assigned the brighter color.Third row: Final segmentation for SDG
(left) and Covariance (right) methods. Planar, parabolic, curved regions and
discontinuities are shown, where the brightness increases in the above order
(planar corresponding to black). For reasons of better visibility the jump and
crease labels were joined (white).
exhibited, even without any pre- or postprocessing, quite acceptable
regionness and smoothness results. The thresholds for both methods
were chosen by observing the results from a wide range of thresholds
and selecting results that optimized, visually, the fit of surface parts
to the known [5] surface geometry types: jumps, creases, planar,
parabolic and curved. However, to provide an objective measure of
the achieved degree of regionness, we computed an average entropy
(uncertainty) measure as follows. A 7 x 7 window was moved over
the full image and, at every pixel, the relative frequencies of labels [5]
were computed. From these values probabilities and the associated
information statistic H = - p, log p, was computed. The average
of this measure constituted the average entropy measure: Ha,,. We
found that for our examples the average entropy Ha,,, of the reference
method (final segmentation) was in between 1.4 and 2.5 times larger
than the average entropy of the covariance approach (see Fig. I).
A final note should be made on the choice of the thresholds.
Automatic determination of thresholds, in particular, the thresholds
for the discontinuity detection can be accomplished by using, for
example, histogram information or even expert knowledge-though
this has not become, as yet, a standard procedure in the literature. For
this reason and for reasons of a fair comparison of our method with
already established methods, we selected threshold values, for each

ACKNOWLEDGMENT
The authors wish to thank S. Berkmann for helping with the
simulations.
REFERENCES
P. J. Besl and R. C. Jain, Invariantsurface characteristicsfor 3D object
recognition in range hages. Computer Vision, Graphics and Image
Processing, vol. 33, i o . 1, pp. 33-80, 1986.
Ting-Jun Fan, Gerard Medioni, and Ramakant Nevatia, Recognizing
3-D objects using surface descriptions, IEEE Trans. Pattern Anal.
Machine Intell., vol. 11, no. 11, pp. 1140-1156, 1989.
Ani1 K. Jain and Richard Hoffman, Evidence-B;lsed Recognition of
3-D Objects, IEEE Trans. Pattern Anal. Machine Intell., vol. 10, no.
6, pp. 783-801, 1988.
P. Liang and J. S. Todhunter, Representation and recognition of
surface shapes in range images,Computer Vision, Graphics and Image
Processing, vol. 52, no. 10, pp. 78-109, 1990.
Naokazu Yokoya and Martin D. Levine, Range image segmentation
based on differentialgeometry: A hybrid approach,IEEE Trans. Pattern
Anal. Machine Intell., vol. 11, no. 6, pp. 643449, 1989.