TSINGHUA SCIENCE A N D TECHNOLOGY ISSN 1007-0214 11/38 pp66-70 V o l u m e 14, N u m b e r SI, June 2009

3-D Shape Measurement of Complex Objects by Combining Color-Coded Fringe and Neural Networks
QIN Dahui LI Zhongwei

W A N G Congjun ( i ^ ) , SHI Yusheng ( 5 t - M " )

State K e y L a b o r a t o r y of Material Processing and Die and M o u l d Technology, H u a z h o n g University of Science and Technology, W u h a n 430074, China

Abstract:

A new approach for three dimensional (3-D) shape measurement was proposed based on color-

coded fringe and neural networks. By applying the phase-shift technique to fringe projection, point clouds were generated with high spatial resolution and limited accuracy. The picture element correspondence problem was solved by using projected color-coded fringes with different orientations. Once the high accurate corresponding points were decided, high precision dense 3-D points cloud was calculated by the well trained net. High spatial resolution can be obtained by the phase-shift technique and high accuracy 3-D object point coordinates are achieved by the well trained net, which is not dependent on the camera model and will work for any type of camera. Some experiments verify the performance of this method.
Key words:

three dimensional (3-D) shape measurement; color-coded fringe; neural networks; correspon-

dence problem

Introduction
The automatic, non-contact three dimensional (3-D) measurement of object surfaces is one of the most important applications of digital close-range photogrammetry and computer vision, and is becoming more and more important in industrial application such as design, manufacturing, and automatic error detection ' . Many
[1 2]

still slow for 3-D coordinates acquirement. The projection of a laser line or a grid of coded light is more efficient than the projection of laser spot. For line laser range scan and coded light scan measurement, spatial resolution of image information acquired by chargecouple device (CCD) camera is low, moreover, all images need to be thresholded, so it tends to be sensitive to the circumstances. The phase-shifting technique not only can obtain high spatial resolution but also is insensitive to the circumstances. Besides, it is highly robust. So fringe projection based on phase shift is adopted. A fringe projection system consisting of one C C D camera and a projector shown in Fig. l a is based on an active pattern projection because the pattern geometry of the projection unit must be calibrated. A fringe projection system consisting of at least two C C D cameras and a projector shown in F i g . l b is based on a passive pattern projection because calibration and orientation of the projector is not required . The calibration of the
[6]

methods based on laser pot, laser line , coded light ,

[3] [4]

and phase-shift

[5]

have been developed within the last

few years. These sensor systems are based on triangulation and structured light. The scanning of a surface with a light spot is a simple method of surface 3-D measurement, which provides a non-contact and fast measurement compared to conventional coordinate measurement machines, but is
Received: 2008-11-09; revised: 2009-02-23 * Supported by the Eleventh Five-Year Pre-Research Project of China * * To whom correspondence should be addressed. E-mail: qindahui@gmail.com; Tel: 86-13098820919

QIN Dahui

(Jl^-kJ^)

et al:

3-D Shape

Measurement

of Complex

Objects

by Combining

67

projector is very complicated. The accuracy of the calibration is lower than the calibration of a camera. Most of the projectors use non sinusoidal patterns to project the phase shift images. The phase error is unable to be compensated . A passive pattern projec[7]

color-coded fringe projection. A three-layer BP neural network (BPNN) is well trained to establish the mapping relationship between the homologous image point coordinates and corresponding 3-D coordinates of the measured object. Then, correspondence between homologous points can be performed by a special approach for phase determination
[12]

tion adopted in this paper can eliminate systematical error sources like non-sinusoidal higher precision.
Projector Projector

wave-forms

and

. The obvious advan-

avoid their disadvantages mentioned above to gain

tage of this approach is that high spatial resolution can be obtained by the phase-shifting technique using fewer fringe images and that high accurate 3-D object point coordinates are achieved by the well trained net which is not dependent on the camera model and works for any type of camera in 3-D shape measurement for very complex objects.

1
(a) A c t i v e t r i a n g u l a t i o n Fig. 1 (b) P a s s i v e t r i a n g u l a t i o n T w o t y p e s o f s t r u c t u r e d light s y s t e m

Principle

B P neural network (BPNN) is a multilayer feedforward network with backpropagation algorithm. The typical architecture of the three layers B P N N with a hidden layer is shown in Fig. 2.
Input layer Hidden layer Output layer

Camera calibration is a critical step for reconstruction of object surface, and its goal is to establish coordinate mapping between the global 3-D-coordinates of a point and the 2-D-coordinates of the projected ima g e ^ . Classical camera calibration approach based on perspective projection
10

model

involves

full-scale

nonlinear optimization^' \ and requires a lot of good initial guess values. Therefore, it is very computingintensive. Tsai's two-step calibration
[11]

is a representa-

tive for the two-step type calibration techniques which involves a direct solution of some camera parameters and an iterative solution for the other parameters. Twostep approach has significant improvement, but it is still cumbersome with decreased accuracy. Neural networks is a well-known approach to approximate a nonlinear system without an explicit physical model, and it can use two image coordinates (x y\
f

Fig. 2

Typical architecture of a three layers B P N N

The neural network has m inputs and neurons in its output layer, these numbers were picked by a particular application. The hidden layer has r neurons. This number was picked by guesswork and experience. The wy ( z - 1 , 2, 3, r; j = 1, 2, 3, m) is the connection (scalar weight) from the No. neuron in the input layer to the No. j neuron in the hidden layer. The wjd (k = 1, 2, 3, n; i = 1, 2, 3, r) is a connection (scalar weight) from the z-th neuron in the hidden layer to the -th neuron in the output layer. The bj is a scalar bias to the y'-th neuron of the input layer and b is a scalar bias
t

x"y")

of

homologous (corresponding) image points and corresponding object 3-D world coordinates point coordinates (XYZ) to train net so as to establish the relationship between the global 3-D-coordiantes of a point and stereo-pair images of homologous image point projection for two C C D cameras. Therefore, it eliminates camera mathematic model and does not calculate explicit values of extrinsic or intrinsic camera parameters. Therefore, the cameras need not to be calibrated actually. In this paper, a method is proposed based on stereovision using back propagation (BP) neural network and

to the z-th neuron of the hidden layer. Neural networks are commonly adjusted or trained, so that a particular input leads to a specific target output. The network is adjusted, based on a comparison of the output and the target, until the network output matches the target. Generally speaking, the least

68

Tsinghua

Science

and Technology,

June 2 0 0 9 , 1 4 ( S 1 ) : 6 6 - 7 0

mean square (LMS) error algorithm is the performance function for supervised training in BP neural network. Assume that there are training samples input/target pairs [X Y ] (i = 1,2,... , N). Here X is an input to neth t l

is a smoothing parameter which can be determined by a cross-validation procedure. A trained three-layer B P neural network with enough sigmoid function neurons in the hidden layer can approximate any function with any precision paper, the image coordinates (x y\
f [14]

work, and Yj is the corresponding target output. As each input is applied to the network, the network output is compared to the target. The error is calculated as the difference between the target output and the network output. The goal is to minimize the average of the sum of these errors. mse =

. In this

x"y") 3-D

of homolocoordinate

gous image points can be put into a one-to-one correspondence with the correspondent (XYZ) of an object and obviously it is a complex

nonlinear mapping relations between image coordi-

V
N

Y-Y

nates and correspondent 3-D coordinate (XYZ) because

(1)

of various lens distortions. So stereo-pair images and correspondent 3-D coordinate (XYZ) can be used to train networks. The input vector is the image stereopair point coordinates (x'y , x"y"),
f

where mse is the mean square error. The L M S algorithm adjusts the weights and biases of the B P network so as to minimize this mse. The traditional B P neural network suffers from the over-fitting problem and the convergence to global minimum
[13]

and then the corre-

spondent 3-D coordinate (XYZ) is obtained by the output layer. To obtain the 3-D coordinate (XYZ) of an object point, the homologous points of stereo-pair images are needed. So correspondence is crucial for 3-D shape measurement. Because there is no explicit camera model parameters, epipolar condition is unable to be used. Correspondence is solved by color-coded fringe projection based four-step phase shift technology. The whole process for correspondence is shown in Fig. 3. The periods of the phase image in Fig. 3 are only for

is

not

always

achieved. Marzban et a l .

proposed a method to

avoid the over-fitting and local minima problem and improve the generalization ability of neural network models, adding a weight-decay or regularization term in the estimation process. Therefore, a regularization or weight-decay term is often added to Eq. (1) E = K - K + r . m s w
1
N

(2)

where msw is the sum of squares of all weights, and

Fig. 3

W h o l e p r o c e s s for c o r r e s p o n d e n c e

QIN Dahui

(Jl^-kJ^)

et al:

3-D Shape

Measurement

of Complex

Objects

by Combining

69

expatiating correspondence algorithm, and are not the actual measurement. The first 24-bits true color image projected includes three frequencies in the horizontal direction, in which each pixel values are divided into R, G, component of three primary colors, forming three solid color images with sinusoidal intensity distribution to represent three different frequencies. The period numbers of R, G, solid color images are 59, 64, and 70. The other three 24-bits true color images are shifted 90 degrees in turn. Then, four-step phase shift technology is used to obtain four sawtooth images from sinusoidal images b y E q . (3). () = a r c t a n ^ ^ ()- ()
3

Experiments

A 400 m m 300 m m high precise calibration board with 99 circle marks is used to obtain sample data sets for neural network training. To ensure the precision of the neural network, twenty groups of calibration images are captured from different height. One calibration image is shown in Fig. 4. Then the training sample sets can be extracted from the images by using an automatic circle-center-extraction algorithm. Totally 99x20 sample points are collected, and fifteen groups sample data are picked up as training samples and the other five groups are used for test. Once the neural network is trained, it can be employed for 3-D reconstruction.

(-

2 K

(
L

(3)
are the same

where (),

(),
2

(),
3

and 7 ( )
4

solid color image sinusoidal intensity respectively. Then, the sawtooth image with 5 periods is obtained based on the heterodyne principle using two images with 59 and 64 p e r i o d s
[15]

. The sawtooth image with 6

periods is obtained based the same principle using two images with 64 and 70 periods. For last two sawtooth images, the heterodyne principle is used to solve the problem of phase unwrapping again, and finally the image of phase unambiguous over the field of view in the horizontal direction is obtained. For 24-bits true color images projected with three frequencies in the vertical direction, the image of phase unambiguous over the field of view in the vertical direction is obtained by the same process. So correspondence is solved by a pair of absolute phase values in two directions perpendicular to each other.
Fig. 4 Calibration image

To verify the performance of the method, a complex metal part is measured with the trained network. Eight groups color fringe patterns are captured during the measurement process. One group vertical images are shown in Fig. 5. The measurement result is shown in Fig. 6, and the local enlarging graph shows that the system can reconstruct complex 3-D model with very smooth details.

Fig. 5

One group vertical images

70

Tsinghua

Science

and Technology,

June 2 0 0 9 , 1 4 ( S 1 ) : 6 6 - 7 0

Fig. 6

M e a s u r e m e n t results W o r k s h o p for D i g i t a l I m a g e P r o c e s s i n g a n d Graphics. 01denburg,Germany, 1993: 103-114. Computer

Conclusions
[5]

A new approach for 3-D shape measurement is proposed based on color-coded fringe and neural networks. A three-layer enhanced BP neural network is used to train the stereo vision application system to calculate 3-D coordinates. The training set for neural network consists of a variety of stereo-pair images and corresponding 3-D world coordinates. Two different directions fringe patterns are used to solve the correspondence problem. Once the high accurate homologous points are obtained, high precision dense 3-D points cloud can be calculated by the well trained networks. The obvious advantage of this approach is that high spatial resolution can be obtained using fewer fringe images by the phase-shifting technique and high accuracy 3-D object point coordinates are achieved by the well trained networks. A complex metal part is measured with the trained networks, and the experiment result shows that the system can reconstruct complex 3D model with very smooth details. References
[1] Chen F, B r o w n G W, Song M. Overview of three-

G u R u o w e i , Y o s h i z a w a T, O t a n i Y. O n e - s t e p p h a s e shift 3 D surface p r o f i l o m e t r y w i t h g r a t i n g p r o j e c t i o n . Optics Lasers in Engineering, 1994, 2 1 : 6 1 - 7 5 . and

[6]

L u h m a n n T, R o b s o n S, K y l e S, H a r l e y I. C l o s e R a n g e P h o togrammetry: Principles, Thechniques and Applications. N e w Jersey: John Wiley and Sons, 2006.

[7]

B e r g m a n n D . N e w a p p r o a c h for a u t o m a t i c surface r e c o n s t r u c t i o n w i t h c o d e d light. SPIE, 1 9 9 5 , 2 5 7 2 : 1-9.

[8]

T o m i o E . A c a m e r a c a l i b r a t i o n t e c h n i q u e u s i n g t h r e e sets o f p a r a l l e l lines. Machine 159-167. Vision and Applications, 1990,13:

[9]

B r o w n D C. C l o s e - r a n g e c a m e r a calibration. metricEng, 1971, 37: 855-866.

Photogram-

[10] F a i g W. C a l i b r a t i o n o f c l o s e - r a n g e p h o t o g r a m m e t r y s y s t e m s : M a t h e m a t i c a l f o r m u l a t i o n . Photogrammetric Remote Sensing, 1975, 41: 1479-1486. Eng.

[11] Tsai R. A v e r s a t i l e c a m e r a c a l i b r a t i o n t e c h n i q u e for h i g h a c c u r a c y 3 - D m a c h i n e v i s i o n m e t r o l o g y u s i n g off-the-shelf T V c a m e r a s a n d l e n s e s . IEEE Automation, Journal on Robotics and

1987, 3(4): 323-344. Proc.

[12] T h e s i n g J. N e w a p p r o a c h e s for p h a s e d e t e r m i n a t i o n . SPIE, 1998, 3478: 133-136.

dimensional shape m e a s u r e m e n t using optical Opt. Eng., [2] 2 0 0 0 , 3 9 ( 1 ) : 10-22.

methods.

[13] M a r z b a n C, W i t t A . A B a y e s i a n n e u r a l n e t w o r k for s e v e r e h a i l size p r e d i c t i o n . Weather 610. [14] T a n g Y, C h e n W, S u X , et al. N e u r a l n e t w o r k a p p l i e d t o r e c o n s t r u c t i o n o f c o m p l e x o b j e c t s b a s e d o n fringe p r o j e c t i o n . Forecast, 2 0 0 1 , 16(5): 600-

L i Z h o n g w e i , Shi Y u s h e n g , W a n g C o n g j u n , et al. A c c u r a t e c a l i b r a t i o n m e t h o d for a s t r u c t u r e d light s y s t e m . Opt. 2008, 47(5): 053604. Eng.,

[3]

Forest

J,

Salvi

J. A

review

of

laser

scanning

three-

Optics

Communication,

2007, 278: 274-278.

d i m e n s i o n a l digitizers, intelligent r o b o t s a n d s y s t e m s . In: I E E E / R S J International Conference on Intelligent Robots and System. E P F L Lausanne, Switzerland, 2002: 73-78, [4] K r a t t e n t h a l e r W, M a y e r J, D u w e P. 3 D - s u r f a c e m e a s u r e m e n t w i t h c o d e d light a p p r o a c h . In: F o u r t h I n t e r n a t i o n a l

[15] R e i c h C, R i t t e r R, T h e s i n g J. W h i t e light h e t e r o d y n e p r i n ciple for 3 - D m e a s u r e m e n t . Proc. 242. SPIE, 1997, 3100: 236-