Combinatorial Surface Integration

Roberto Fraile and Edwin R. Hancock Department of Computer Science The University of York Heslington, York YO10 5DD, UK

Graph-Spectral surface integration techniques construct an integration path assuming that the surface contains a path along which the integration error is minimal. This paper presents a generalisation that uses Minimum Spanning Trees of the weighted grid graph of surface normals, which scales with no need for surface segmentation. The problem of choosing an integration path is reduced to defining a local weight function. The method is assessed at weighting human face surface normals with geometric and information-theoretic functions of local support.

as different paths through the same location, at the cost of computational complexity, and a more sophisticated error model. In this work we are solely concerned with a single height estimate for each surface location. Surface patches with an integration path over which surface integration is exact, are small and rare, because the error due to measurement noise propagates along the path. To overcome this problem, we propose using an integration tree in a way that generalises the notion of integration path. A first requirement for such a scheme is that every two points are connected by a path, so that height increments can be estimated at every location of the grid. A second requirement is that there is only one path between each two points, so that height estimates are unique. Considering the surface locations as a grid graph, a Spanning Tree fulfils this requirement. Furthermore, if a weight is assigned to each edge of the grid graph, a Minimum Spanning Tree of the weighted graph has the property of having minimal total weight. A Minimum Spanning Tree exists so long as the weight graph is connected [2]. The surface is reconstructed by integrating along the paths of the tree, using the surface normals. Thus it is possible to choose both the weight function and the optimisation procedure independently. In this case the optimisation algorithm is a Minimum Spanning Tree algorithm. The weights are calculated locally from the surface normals. The choice of local weight function used to construct the weight matrix determines the performance of the method and its robustness against errors in the data, and is independent of the rest of the algorithm. First the process of surface integration, which is independent of the weight function, is described. An optimal integration tree, attaining minimum total weight, is constructed as the Minimum Spanning Tree of the weight matrix. Then four weight functions are defined and finally we proceed to illustrate and discuss the performance of the weight functions on surface normals of human faces. 1

1 Introduction
Surface integration is used to reconstruct surface heights from gradients produced by shape-from-shading algorithm [1,5,12,13]. An integration path is constructed between every two locations of the surface, or a subset of it, and the surface normals along discrete locations of that path are used to estimate the height increments along that integration path. Most approaches enforce a global integrability constraint on the surface and minimise an energy functional. Alternatively, recent work by Robles-Kelly [3, 9] uses graph-spectral methods [10, 11] to segment the data and build up an integration path on which simple numerical integration allows the reconstruction of single surface estimates, assuming that the height estimates along that integration path are perfect. The weight matrix is constructed using an estimate of the sectional curvature, and the spectral structure of the weight matrix is used to segment the surface into patches. Each segment is then integrated using a single integration path, and heights are merged by minimising the average height difference along boundaries. Further than considering a single height estimate, a belief propagation method [7] could be used to combine multiple estimates from different sources of information, such

2 Integration over trees
In order to propagate heigh estimates along a surface, we are using paths that connect each location of the surface to a root location. Once this integration tree is defined, integration can be performed by using the trapezium rule on each pair of consecutive locations along the integration path. The gradient values at each location and the fact that the grid is equidistant are enough to calculate the height difference up to a multiplying constant for the entire surface. A weight matrix can be defined to represent the error expected in the estimates for a pair of adjacent locations. Given a weight matrix for the lattice graph of surface locations, calculating the integration tree is a matter of finding the tree that spans over all surface locations, and attains the minimum weight. The integration tree can thus be computed by a Minimum Spanning Tree algorithm such as Prim’s algorithm, whose complexity is quadratic on the number of nodes, more efficient algorithms exist [2]. Figure 1 illustrates the concept of integration trees for a face. Using an integration tree instead of a single path reduces the path length and, because paths can branch, paths with lower weight can be followed, thus reducing integration error.

Logstar : An information-theoretic weight function. The total description length of the field of surface normals is minimised by reducing the number of bits required to incrementally encode surface normals along the integration paths. The resulting integration tree helps describe the surface in a minimal amount of bits, under certain constraints. The underlying assumption is that better compression is a sign of a better model, where the model in this case is the integration tree. Principal : An integration tree whose branches follow the minor curvature direction can be expected to yield a lower integration error. Since the principal curvature directions are an intrinsic property of the surface being reconstructed, they can be estimated by minimising the total difference between adjacent surface normals. The weight functions Logstar and Principal are now described in more detail.


Description length

3 The weight function
The problem of defining an integration tree has been reduced to defining a weight function for each edge of the graph. Consider the data as a labelled grid graph, in which the labels are the surface normals, and that we need to define weights from which a Minimum Spanning Tree will be derived. A local weight function needs to be defined for each edge of the graph, represented by a pair of surface normals (u, v). Four local weight functions are considered, which are chosen to represent a variety of assumptions on the source of the data. Random: A uniformly random weight function. This weight function makes no use of any information contained in the data. Note that this weight function produces a seemingly random integration tree, as shown in Figure 1. Let Norm denote the weight function that simply assigns the Euclidean distance between the unit normal vectors at each node of the edge. Norm(u, v) = v − u

The weight matrix of the graph can be defined according to a description length criterion. The problem of finding an integration path can be seen as a matter of finding the best model that fits the surface normals under certain conditions. The Minimum Description Length principle [6, 8] states that, if one can describe the data using the model, as a binary string, and there is an algorithm that enumerates all the possible models, one should choose the model that describes the data in a more compact form. This description needs to take into account • The length, in bits, of the description of the model. • The length, in bits, of the description of the data, given the model. Let d be the surface normals, and let M denote the models we are interested in. Considering the model as a function that constructs exactly the data from the error, (thus representing the regularities in the data), d = M (e) = U (me) (me denoting the concatenation of the binary strings m and e), where e is whatever is needed to reconstruct the field of surface normals by incrementally encoding normal directions along the integration tree. In general, M can be represented by a binary program m for a Universal prefixfree Turing machine U [6]. We can expect e to describe small vector differences, so we can represent it as the surface normal at the root of the tree, f , and the list of surface normal increments for each tree edge, (g1 , g2 , . . . , gr ). d = U (mcf g1 g2 · · · gr ) 2

This function uses the information in the surface normals as unit vectors in the Euclidean space. This is effectively a least squares method, which is optimal when the surface normal differences are drawn from a Gaussian distribution.

y 10 15 20 25 30 10 15

y 20 25 30 10 15 5 5

y 20 25 30 10 15 5

y 20 25 30

where c is a constant-size program that converts the initial value f and the increments gi into absolute values. Let Length(b) denote the length in bits of the binary string b. So long as we do not take the structural complexity of the tree M into account, we could consider Length(m) and Length(f ) to be constant. They will be much smaller than the differences in Length(g1 g2 · · · gn ). Thus we can use the sum of the lengths of the vector differences as weight function: Length(g1 ) + Length(g2 ) + · · · + Length(gn ) and thus define our MDL encoding as a function of the difference gi for each i. In this case we shall choose the log encoding of positive integers as described in [4] which is a compact encoding under the assumption that the values are close to zero. The specific encoding used for the increments is of limited relevance, so long as it is a reasonable way to compress small positive integers. Thus we can define: Logstar (u, v) = Length(log (Natural(v − u))) where Natural is a function that maps fixed point numbers into positive integers suitable to be encoded into binary strings using log . The underlying idea is that we are effectively compressing the data without loss of information, using the integration tree as a model.


5 10 15

5 10 15

5 10 15

5 10 15

Since e, d1 and d2 are normalised, the value of α−1 is in 2 the interval [1, ∞). A large value corresponds to a direction aligned with the major curvature direction, and a value 1 corresponds to the minor curvature direction.



Minor principal curvature direction

A different way to assign weights to pair of adjacent locations is to use the proximity of the edge’s direction to the minor Principal Curvature direction. An integration path that follows the minor Principal Curvature direction is bound to convey the least error in terms of integration. In smooth surfaces the minor Principal Curvature direction provides either a closed path or a path that ends at a singularity. The Principal Curvature directions can be estimated from the second derivatives of the surface function. The weight of an edge is a measure of the deviation from the minor principal curvature direction, calculated as follows. Let d1 and d2 be the normalised principal curvature directions, which are the eigenvectors of the Hessian. Let e be the normalised direction of the edge, which can be represented in the basis (d1 , d2 ). e = (α1 , α2 ) d1 d2

Figure 1 shows the integration trees for each of the four different methods. The integration tree of the Random weight function contains no information on the surface characteristics. The Norm weight function produces integration paths with a greater bias against large changes in surface normals (The implementation used to produce the figure also masks out regions for which surface normals are not defined). When the description length weight function Logstar is used in the weight matrix, the Minimum Spanning Tree provides the integration tree with lowest total weight, which in effect is an upper bound of the minimum description length of the surface normals, using the integration tree as a model. The integration tree for the Principal weight function favours paths along the minor principal curvature direction, as expected.

4 Experiments
This surface integration method has been applied to a dataset consisting of surface heights of ten different faces. Each face was given as both a 256 × 256 array of surface heights, and as a 256 × 256 field of surface normals, and each dataset was subsampled as required for illustration purposes. An estimate of the standard deviation of the difference of the reconstructed surface and the ground truth was used as a measure of the error. Four methods have been assessed: Random, Norm, Logstar and Principal . In most cases the method Norm performed best, as shown in Figure 2. The lowest error was achieved by the method Norm. This is due to the robustness against outliers appearing near the occluding contour, where the slope of the surface normals is of great magnitude and prone to errors. 3

x 20 25 30

x 20 25 30

x 20 25 30

x 20 25 30

(a) Random

(b) Norm

(c) Logstar

(d) Principal

Figure 1. Integration trees calculated from surface normals using each of the local weight functions.

Integration trees

The weight is the inverse of the value of |α2 |, which happens to be: Principal (u, v, e) = |α2 |


|d11 d22 − d21 d12 | |d11 e2 − d21 e1 |

5 Conclusion
5 0

10 0 −10 −20

−10 −15 −20 −25

60 50 40 30 20 10 y 0 0 10 x 20 30 50 40 60

50 40 30 40 20 10 y 20 10 x 30 50 60

(a) Logstar . The error is 0.38.

(b) Norm . The error is 0.26.

Figure 2. Example reconstructions of face 10.

We have presented a surface integration method from gradient fields that integrates along paths of a spanning tree. A combinatorial algorithm is used to minimise the total weight of an integration tree. The problem of path-based surface integration, under the assumption that a perfect path exists, has thus been simplified to the problem of defining an appropriate weight function of local support. We have tested this approach with information-theoretic and geometry-based weight functions. The lack of need for segmentation suggests that this method can be scaled up to datasets of larger size, and to surface normal models of greater complexity.

[1] A. Agrawal and R. Chellappa. An algebraic approach to surface reconstruction from gradient fields. In Proceedings of ICCV 2005, pages 23–114, 2005. [2] A. Gibbons. Algorithmic Graph Theory. Cambridge University Press, 1985. [3] A. graph-spectral approach to shape-from shading. A. robles-kelly and e.r. hancock. IEEE Transactions on Image Processing, 13(7):912–926, July 2004. [4] P. D. Gr¨ nwald. The Minimum Description Length Principle u and Reasoning Under Uncertainty. PhD thesis, CWI, 1998. [5] R. Klette and K. Schl¨ ns. Height data from gradient fields. u In Proceedings of SPIE, number 2908, pages 204–215, 1996. [6] M. Li and P. Vit´ nyi. An Introduction to Kolmogorov Coma plexity and Its Applications. Graduate Texts in Computer Science. Springer, 2nd edition, 1997. [7] N. Petrovic, I. Cohen, B. J. Frey, R. Koetter, and T. S. Huang. Enforcing integrability for surface reconstruction algorithms using belief propagation in graphical models. In CVPR 2001 Conference Proceedings, volume 1, page 743, 2001. [8] J. Rissanen. Encyclopedia of Statistical Sciences, volume 5, chapter Minimum-Description-Length Principle, pages 523–527. Wiley New York, 1983. [9] A. Robles-Kelly. Graph-spectral methods for Computer Vision. PhD thesis, The University of York, September 2003. [10] S. S. and B. K.L. Quantitative measures of change based on feature organization: Eigenvalues and eigenvectors. Computer Vision and Image Understanding, 71(1):110–136, 1998. [11] J. Shi and J. Malik. Normalized cuts and image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(8):888–905, August 2000. [12] P. L. Worthington and E. R. Hancock. New constraints on data-closeness and needle map consistency for shape-fromshading. PAMI, 21(12):1250–1267, December 1999. [13] R. Zhang, P.-S. Tsai, J. Cryer, and M. Shah. Shape-fromshading: a survey. PAMI, 21(8):690–706, August 1999.

Figure 3. Ground truth (left) and reconstruction (right) of face 10.

The Norm weight function also performs better when Gaussian noise is added to the surface normals. Incidentally, both Logstar and Norm are implemented as information-theoretic codes, and only differ in the actual encoding used. The method Norm uses a very expensive encoding that uses a very high number of bits to encode large differences (n2 bits to encode the number n), As a result, great differences in surface normals yield an expensive encoding. It can be expected that filtering outliers from the data, prior to surface integration, will produce similar good reconstructions using both methods. In most cases method Principal worked second to Norm, thus favouring a geometric interpretation of the local weight function. Overall, the experiments seem to be dominated by the presence of outliers in the surface normals, which usually are located near the occluding contour, as shown in Figure 2. The method that assigns a greater weight to the outliers, Norm is the one that yields better results. Figure 3 shows an example of good reconstruction of the surface. Note that in all cases a single integration tree has been used to integrate over the entire surface, thus avoiding the use of segmentation. 4