This action might not be possible to undo. Are you sure you want to continue?
3-D Shape Measurement of Complex Objects by Combining Color-Coded Fringe and Neural Networks
QIN Dahui LI Zhongwei
W A N G Congjun ( i ^ ) , SHI Yusheng ( 5 t - M " )
State K e y L a b o r a t o r y of Material Processing and Die and M o u l d Technology, H u a z h o n g University of Science and Technology, W u h a n 430074, China
A new approach for three dimensional (3-D) shape measurement was proposed based on color-
coded fringe and neural networks. By applying the phase-shift technique to fringe projection, point clouds were generated with high spatial resolution and limited accuracy. The picture element correspondence problem was solved by using projected color-coded fringes with different orientations. Once the high accurate corresponding points were decided, high precision dense 3-D points cloud was calculated by the well trained net. High spatial resolution can be obtained by the phase-shift technique and high accuracy 3-D object point coordinates are achieved by the well trained net, which is not dependent on the camera model and will work for any type of camera. Some experiments verify the performance of this method.
three dimensional (3-D) shape measurement; color-coded fringe; neural networks; correspon-
The automatic, non-contact three dimensional (3-D) measurement of object surfaces is one of the most important applications of digital close-range photogrammetry and computer vision, and is becoming more and more important in industrial application such as design, manufacturing, and automatic error detection ' . Many
still slow for 3-D coordinates acquirement. The projection of a laser line or a grid of coded light is more efficient than the projection of laser spot. For line laser range scan and coded light scan measurement, spatial resolution of image information acquired by chargecouple device (CCD) camera is low, moreover, all images need to be thresholded, so it tends to be sensitive to the circumstances. The phase-shifting technique not only can obtain high spatial resolution but also is insensitive to the circumstances. Besides, it is highly robust. So fringe projection based on phase shift is adopted. A fringe projection system consisting of one C C D camera and a projector shown in Fig. l a is based on an active pattern projection because the pattern geometry of the projection unit must be calibrated. A fringe projection system consisting of at least two C C D cameras and a projector shown in F i g . l b is based on a passive pattern projection because calibration and orientation of the projector is not required . The calibration of the
methods based on laser pot, laser line , coded light ,
have been developed within the last
few years. These sensor systems are based on triangulation and structured light. The scanning of a surface with a light spot is a simple method of surface 3-D measurement, which provides a non-contact and fast measurement compared to conventional coordinate measurement machines, but is
Received: 2008-11-09; revised: 2009-02-23 * Supported by the Eleventh Five-Year Pre-Research Project of China * * To whom correspondence should be addressed. E-mail: email@example.com; Tel: 86-13098820919
but it is still cumbersome with decreased accuracy. based on a comparison of the output and the target. these numbers were picked by a particular application. Therefore. 67 projector is very complicated. and it can use two image coordinates (x y\ f x m Fig. a method is proposed based on stereovision using back propagation (BP) neural network and to the z-th neuron of the hidden layer. 3. The typical architecture of the three layers B P N N with a hidden layer is shown in Fig. i = 1. Therefore. A three-layer BP neural network (BPNN) is well trained to establish the mapping relationship between the homologous image point coordinates and corresponding 3-D coordinates of the measured object. it is very computingintensive. 3. Input layer Hidden layer Output layer Camera calibration is a critical step for reconstruction of object surface. The hidden layer has r neurons. 2. The obvious advan- avoid their disadvantages mentioned above to gain tage of this approach is that high spatial resolution can be obtained by the phase-shifting technique using fewer fringe images and that high accurate 3-D object point coordinates are achieved by the well trained net which is not dependent on the camera model and works for any type of camera in 3-D shape measurement for very complex objects. the cameras need not to be calibrated actually. Therefore. The wy ( z . so that a particular input leads to a specific target output. m) is the connection (scalar weight) from the No. 2. until the network output matches the target. it eliminates camera mathematic model and does not calculate explicit values of extrinsic or intrinsic camera parameters. Neural networks are commonly adjusted or trained. Projector Projector wave-forms and . The wjd (k = 1. Classical camera calibration approach based on perspective projection 10 model involves full-scale nonlinear optimization^' \ and requires a lot of good initial guess values. 1 (b) P a s s i v e t r i a n g u l a t i o n T w o t y p e s o f s t r u c t u r e d light s y s t e m Principle B P neural network (BPNN) is a multilayer feedforward network with backpropagation algorithm. j neuron in the hidden layer. j = 1. 2 Typical architecture of a three layers B P N N The neural network has m inputs and η neurons in its output layer. Twostep approach has significant improvement. 2. r) is a connection (scalar weight) from the z-th neuron in the hidden layer to the £-th neuron in the output layer. The accuracy of the calibration is lower than the calibration of a camera. Neural networks is a well-known approach to approximate a nonlinear system without an explicit physical model. Tsai's two-step calibration  is a representa- tive for the two-step type calibration techniques which involves a direct solution of some camera parameters and an iterative solution for the other parameters. 3. A passive pattern projec color-coded fringe projection. This number was picked by guesswork and experience. 1 (a) A c t i v e t r i a n g u l a t i o n Fig. Then. correspondence between homologous points can be performed by a special approach for phase determination  tion adopted in this paper can eliminate systematical error sources like non-sinusoidal higher precision. r. The bj is a scalar bias to the y'-th neuron of the input layer and b is a scalar bias t x"y") of homologous (corresponding) image points and corresponding object 3-D world coordinates point coordinates (XYZ) to train net so as to establish the relationship between the global 3-D-coordiantes of a point and stereo-pair images of homologous image point projection for two C C D cameras. In this paper. ζ neuron in the input layer to the No.QIN Dahui (Jl^-kJ^) et al: 3-D Shape Measurement of Complex Objects by Combining . 2.1 . n. 2. Generally speaking. 3. The phase error is unable to be compensated . The network is adjusted. Most of the projectors use non sinusoidal patterns to project the phase shift images. and its goal is to establish coordinate mapping between the global 3-D-coordinates of a point and the 2-D-coordinates of the projected ima g e ^ . the least .
. Marzban et a l . Here X is an input to neth t l is a smoothing parameter which can be determined by a cross-validation procedure. The L M S algorithm adjusts the weights and biases of the B P network so as to minimize this mse. In this x"y") 3-D of homolocoordinate gous image points can be put into a one-to-one correspondence with the correspondent (XYZ) of an object and obviously it is a complex nonlinear mapping relations between image coordi- —V N 1 Y-Y - nates and correspondent 3-D coordinate (XYZ) because (1) of various lens distortions. Therefore. and Yj is the corresponding target output. A trained three-layer B P neural network with enough sigmoid function neurons in the hidden layer can approximate any function with any precision paper.. m s w 1 N - (2) where msw is the sum of squares of all weights.7 0 mean square (LMS) error algorithm is the performance function for supervised training in BP neural network. the network output is compared to the target.. 3 are only for is not always achieved. So stereo-pair images and correspondent 3-D coordinate (XYZ) can be used to train networks. The whole process for correspondence is shown in Fig. Correspondence is solved by color-coded fringe projection based four-step phase shift technology. epipolar condition is unable to be used. 3.. So correspondence is crucial for 3-D shape measurement. the homologous points of stereo-pair images are needed. 3 W h o l e p r o c e s s for c o r r e s p o n d e n c e . (1) E =— Υ K . f where mse is the mean square error. As each input is applied to the network. and τ Fig. June 2 0 0 9 . the image coordinates (x y\ f  work. The traditional B P neural network suffers from the over-fitting problem and the convergence to global minimum  and then the corre- spondent 3-D coordinate (XYZ) is obtained by the output layer.2.K + r . N). Because there is no explicit camera model parameters. x"y"). 1 4 ( S 1 ) : 6 6 . The periods of the phase image in Fig. proposed a method to avoid the over-fitting and local minima problem and improve the generalization ability of neural network models. The error is calculated as the difference between the target output and the network output.68 Tsinghua Science and Technology. mse = . The goal is to minimize the average of the sum of these errors. Assume that there are Ν training samples input/target pairs [X Y ] (i = 1. a regularization or weight-decay term is often added to Eq. The input vector is the image stereopair point coordinates (x'y . adding a weight-decay or regularization term in the estimation process. To obtain the 3-D coordinate (XYZ) of an object point.
G. 5. 4. the sawtooth image with 5 periods is obtained based on the heterodyne principle using two images with 59 and 64 p e r i o d s  . Then. 5 One group vertical images . One calibration image is shown in Fig. 3 and 7 ( χ ) 4 solid color image sinusoidal intensity respectively. 69 expatiating correspondence algorithm. For last two sawtooth images. Once the neural network is trained. the image of phase unambiguous over the field of view in the vertical direction is obtained by the same process. it can be employed for 3-D reconstruction. a complex metal part is measured with the trained network. 4 Calibration image To verify the performance of the method. The period numbers of R.QIN Dahui (Jl^-kJ^) et al: 3-D Shape Measurement of Complex Objects by Combining . The first 24-bits true color image projected includes three frequencies in the horizontal direction. 2 Ι (χ). in which each pixel values are divided into R. Β component of three primary colors. φ(χ) = a r c t a n ^ ^ Ι (χ)-Ι (χ) λ 3 2 Experiments A 400 m m χ 300 m m high precise calibration board with 99 circle marks is used to obtain sample data sets for neural network training. One group vertical images are shown in Fig. Fig. The other three 24-bits true color images are shifted 90 degrees in turn. So correspondence is solved by a pair of absolute phase values in two directions perpendicular to each other. G. λ Ι (χ). 6. Β solid color images are 59. and finally the image of phase unambiguous over the field of view in the horizontal direction is obtained. and fifteen groups sample data are picked up as training samples and the other five groups are used for test. The sawtooth image with 6 periods is obtained based the same principle using two images with 64 and 70 periods. and 70. the heterodyne principle is used to solve the problem of phase unwrapping again. and the local enlarging graph shows that the system can reconstruct complex 3-D model with very smooth details. Eight groups color fringe patterns are captured during the measurement process. twenty groups of calibration images are captured from different height. The measurement result is shown in Fig. Τ (χΛ- Τ 2 K ^ (χΛ L (3) are the same where Ι (χ). forming three solid color images with sinusoidal intensity distribution to represent three different frequencies. (3). 64. To ensure the precision of the neural network. Fig. Totally 99x20 sample points are collected. four-step phase shift technology is used to obtain four sawtooth images from sinusoidal images b y E q . and are not the actual measurement. For 24-bits true color images projected with three frequencies in the vertical direction. Then the training sample sets can be extracted from the images by using an automatic circle-center-extraction algorithm. Then.
242. Eng. E P F L Lausanne. A c c u r a t e c a l i b r a t i o n m e t h o d for a s t r u c t u r e d light s y s t e m . Two different directions fringe patterns are used to solve the correspondence problem.  2 0 0 0 . Journal on Robotics and 1987. Photogrammetric Remote Sensing. A complex metal part is measured with the trained networks.  K r a t t e n t h a l e r W. and  L u h m a n n T. d i m e n s i o n a l digitizers.r a n g e c a m e r a calibration. O n e . R i t t e r R. methods. Once the high accurate homologous points are obtained. N e w Jersey: John Wiley and Sons. A three-layer enhanced BP neural network is used to train the stereo vision application system to calculate 3-D coordinates.  T a n g Y. intelligent r o b o t s a n d s y s t e m s . 2002: 73-78. A c a m e r a c a l i b r a t i o n t e c h n i q u e u s i n g t h r e e sets o f p a r a l l e l lines. Thechniques and Applications. Optics Lasers in Engineering. SPIE. 1997. 2 0 0 1 .. Eng. 3478: 133-136. Weather 610. 47(5): 053604. SPIE. A review of laser scanning three- Optics Communication. 3 D .D m a c h i n e v i s i o n m e t r o l o g y u s i n g off-the-shelf T V c a m e r a s a n d l e n s e s .7 0 Fig. 1998. Song M. 278: 274-278. N e w a p p r o a c h for a u t o m a t i c surface r e c o n s t r u c t i o n w i t h c o d e d light. SPIE. 2 5 7 2 : 1-9. Switzerland. H a r l e y I. 1993: 103-114. 3100: 236- . C a l i b r a t i o n o f c l o s e .  T h e s i n g J. Machine 159-167. N e u r a l n e t w o r k a p p l i e d t o r e c o n s t r u c t i o n o f c o m p l e x o b j e c t s b a s e d o n fringe p r o j e c t i o n . Proc. June 2 0 0 9 . R o b s o n S. In: F o u r t h I n t e r n a t i o n a l  R e i c h C. 1971. Overview of three- G u R u o w e i . 3(4): 323-344.D m e a s u r e m e n t . T h e s i n g J. 3 9 ( 1 ) : 10-22. S u X . W i t t A . Forecast. 1 4 ( S 1 ) : 6 6 .70 Tsinghua Science and Technology. The training set for neural network consists of a variety of stereo-pair images and corresponding 3-D world coordinates..s u r f a c e m e a s u r e m e n t w i t h c o d e d light a p p r o a c h . 01denburg. D u w e Η P.  Forest J. Salvi J. 2008. In: I E E E / R S J International Conference on Intelligent Robots and System. et al. 1994. O t a n i Y. A B a y e s i a n n e u r a l n e t w o r k for s e v e r e h a i l size p r e d i c t i o n . A v e r s a t i l e c a m e r a c a l i b r a t i o n t e c h n i q u e for h i g h a c c u r a c y 3 .7 5 . C l o s e R a n g e P h o togrammetry: Principles. K y l e S. high precision dense 3-D points cloud can be calculated by the well trained networks. 2007.  B e r g m a n n D . metricEng. dimensional shape m e a s u r e m e n t using optical Opt. 2 1 : 6 1 . C l o s e . Computer 3 Conclusions  A new approach for 3-D shape measurement is proposed based on color-coded fringe and neural networks.  M a r z b a n C. Opt. The obvious advantage of this approach is that high spatial resolution can be obtained using fewer fringe images by the phase-shifting technique and high accuracy 3-D object point coordinates are achieved by the well trained networks. and the experiment result shows that the system can reconstruct complex 3D model with very smooth details. Shi Y u s h e n g . 1975. 41: 1479-1486. 2006.Germany. W a n g C o n g j u n .13:  B r o w n D C. 1990. Photogram-  F a i g W. Vision and Applications. 37: 855-866. et al. C h e n W. IEEE Automation.  T o m i o E . 6 M e a s u r e m e n t results W o r k s h o p for D i g i t a l I m a g e P r o c e s s i n g a n d Graphics. B r o w n G W. M a y e r Κ J. Y o s h i z a w a T. N e w a p p r o a c h e s for p h a s e d e t e r m i n a t i o n .  Tsai R. Eng.r a n g e p h o t o g r a m m e t r y s y s t e m s : M a t h e m a t i c a l f o r m u l a t i o n . W h i t e light h e t e r o d y n e p r i n ciple for 3 . Proc. 16(5): 600- L i Z h o n g w e i . 1 9 9 5 . References  Chen F.s t e p p h a s e shift 3 D surface p r o f i l o m e t r y w i t h g r a t i n g p r o j e c t i o n .