FACE RECOGNITION SYSTEM BASED ON FRONT-END FACIAL FEATURE EXTRACTION USING FPGA

-----------------------------------

Pao-Chung Chao
Digitally signed by Pao-Chung Chao DN: CN = Pao-Chung Chao, C = TW, O = De La Salle University, OU = MS-ECE Reason: I am the author of this document Date: 2009.11.12 15:28:07 +08'00'

A Thesis Proposal Presented to the Faculty of the College of Engineering De La Salle University -----------------------------------In Partial Fulfillment of The Requirements for the Degree of Master of Science in Electronics and Communications Engineering ------------------------------------By PAO-CHUNG, CHAO November 2009

De La Salle University, Manila

Face Recognition System, 2009

Abstract
Face recognition has received significant attention in recent years. Many researches have been done in the domain of face detection and recognition. However, there are only few works being implemented on hardware and put into test in real environment. In this research, face detection and facial feature extraction are developed based on a hardware using FPGA. Well-known algorithms in moving object detection, face detection, poses estimation, illumination normalization, facial feature extraction and recognition, are selected and implemented. The performance and accuracy of the system are experimented using software simulation and hardware implementation. For face recognition, one image per person in face database is the basic requirement in the system. Finally, the system will be put into test in a real environment using a video camera.

1

De La Salle University, Manila

Face Recognition System, 2009

TABLE OF CONTENTS Chapter I: Introduction...........................................................................................................5 1.1 Background of the Study ...........................................................................................5 1.2 Problem Statement.....................................................................................................6 1.3 Objectives of the Study ..............................................................................................7 1.3.1 General Objectives..........................................................................................7 1.3.2 Special Objectives ...........................................................................................7 1.4 Significance of the Study ...........................................................................................8 1.5 Scope and Delimitations ............................................................................................9 1.5.1 Hardware Requirement..................................................................................9 1.5.2 Software Requirement....................................................................................9 1.5.3 Prospect of the System....................................................................................9 Chapter II: Literature Review..............................................................................................10 2.1 Reviews of Face Recognition Technology ..............................................................10 2.1.1 Issues on Face Recognition...........................................................................11 2.1.2 Face Detection ...............................................................................................12 2.1.3 Facial Feature Extraction and Recognition................................................18 2.1.4 Face Database................................................................................................21
2

De La Salle University, Manila

Face Recognition System, 2009

2.2 A Review of Hardware Based Face Recognition ...................................................22 Chapter III: Theoretical Framework...................................................................................29 3.1 System Architecture.................................................................................................29 3.2 Image Processing Unit .............................................................................................30 3.3 Face Recognition Unit..............................................................................................31 Chapter IV: Methodology......................................................................................................32 4.1 Development .............................................................................................................32 4.1.1 Image Processing Unit ..................................................................................32 4.1.2 Face Recognition Unit...................................................................................37 4.1.3 Peripheral Interfacing ..................................................................................38 4.2 Experimentation and Evaluation ...........................................................................38 4.2.1 Software Simulation......................................................................................39 4.2.2 Hardware Evaluation ...................................................................................39 Chapter V: Summary.............................................................................................................40 References...............................................................................................................................41

3

De La Salle University, Manila LIST OF SYMBOLS FPGA (Field Programmable Gate Array) RFID (Radio-Frequency Identification) PDMs (Point Distribution Models) PCA (Principal Component Analysis) LDA (Linear Discriminant Analysis) FA (Factor Analysis) SVM (Support Vector Machine) LDA/FLD (Fisher Discriminant Analysis) ICA (Independent Component Analysis) PDBNN (Probabilistic Decision-Based Neural Networks) GA (Genetic Algorithm) HMM (Hidden Markov Model)

Face Recognition System, 2009

CNN-SOM (Convolution Neural Network Self Organizing Map) ANN (Artificial Neural Networks) mLBP (Modified Local Binary Pattern) PCNN (Principal Component Neural Network) ZISC (Zero Instruction Set Computer) DSP (Digital Signal Processor) LUT (Look Up Table)

4

De La Salle University, Manila

Face Recognition System, 2009

Chapter I: Introduction
1.1 Background of the Study Due to many reasons, such as authority management and security issue since the 9/11 terrorist attacks, a sophisticated security system become more important in our daily life, especially for person identification. Nowadays, there are various ways to identify a person, which could be classified into two categories, such as biometric and non-biometric methods [10]. On the one hand, non-biometric identification system uses a serious numbers of personal ID and password to identify a person, such as ID cards (student/citizen), credit card, system password, and other ID used to represent a person. However, non-biometric ID could be forged or taken by other people, which system can not verify the validity of the ID holder. On the other hand, there are some successful biometric recognition systems, using iris, finger print, voice, palm geometry, signature, and facial features to identify a person, which these systems have high accuracy in recognition [12]. However, some of the biometric recognition systems, such as iris and finger print scanning, are intrusive identification, which the systems need the cooperation of users and the equipments needed are usually expensive [12]. On the contrary, in a face recognition system, cameras are installed at a surveillance place, which the system can capture all the objects in real time without being noticed. As a result, face recognition system have received significant attention.
5

De La Salle University, Manila

Face Recognition System, 2009

Moreover, many researches have been done in face detection and recognition [10][11][12] in recent years. In these papers, the results show high accuracy and excellent performance. However, most of the researches are based on certain preconditions, and only few works have been practiced in real environment due to the performance issue. In this research, software and hardware based face recognition techniques are studied. The face recognition system is developed based on hardware implementation and put into test in real environment. 1.2 Problem Statement In the past few years, face recognition has attracted great interest by many researchers. There are many well-known algorithms are created in the recent years [12]. In face recognition, the simple process of face recognition system should go through image data retrieval, face detection, facial feature extraction and face recognition. However, some researches focus on a part of face recognition system, such as face detection, face recognition, or algorithms dealing with certain issues. Moreover, in real environment, images are from nature scenes instead of from pre-collected images, which there are many variant factors, such as complex background noise, lightening condition, poses variation, and other factors [10], under the nature scenes; and most of the researches only deal with certain issues. Furthermore, most experiments are by software simulation on a computer. However, for image sequences, the image processing tasks are computational expensive [11], which the
6

De La Salle University, Manila

Face Recognition System, 2009

implementation can not be put into a real environment to test. As a result, a hardware implementation is conducted in this research. 1.3 Objectives of the Study 1.3.1 General Objectives The main objective is to design a face recognition system, which an image processing unit is implemented based on a hardware using FPGA. In addition, the system will be put into test in a real environment. 1.3.2 Special Objectives 1.3.2.1 Accelerate Computational Tasks FPGA Hardware Implementation Most of the computational tasks in this system are implemented on Altera FPGA (Field Programmable Gate Array) to advance the performance required in real time recognition. Well-known algorithms Implementation Efficient, accurate, and feasible algorithms are adapted and implemented in this system. 1.3.2.2 Reduce laborious tasks Use one image per person to reduce data collection and training time
7

De La Salle University, Manila

Face Recognition System, 2009

Using one image per person is required in this system, which many algorithms require various training data to reach the accuracy [13]. The advantage of using one image per person is not only to reduce the data collective issue, but also to reduce training time while building the face database. 1.3.2.3 Design a Flexible Architecture of Recognition System Algorithms replaceable In face recognition, new algorithms are presented every year. In this research, a flexible architecture is designed in order to adapt new algorithms. Hardware software co-design A hardware and software co-design (Co-SW/HW) is implemented in this research. The advantage of Co-SW/HW is to exploit the synergism of hardware and the flexibility of software [14]. 1.4 Significance of the Study Nowadays, non-biometric identification systems are commonly used in schools and buildings security. For example, students need an ID card, RFID (Radio-frequency identification) card, to pass the entrance gate in De La Salle University. However, the ID card could be used by other people, which is not the person registered to the card. To deal with this, security guards use their bare eyes to compare a picture on the screen with the person. In this occasion, this system
8

De La Salle University, Manila

Face Recognition System, 2009

could help them to do the work. Also, this system can easily be integrated to a present face database in the school due to the system only need one image per person in the database. 1.5 Scope and Delimitations In this research, some hardware equipments and software tools are needed to do the experiment. In addition, there are some limitations for the system functions. 1.5.1 Hardware Requirement In this research, the key equipments used are computers, an Altera NIOS StratixII FPGA kit, an image CMOS sensor, and a monitor. 1.5.2 Software Requirement Some software programming tools are needed in this research, which are a C/C++ compiler IDE, Quartus II design software, Nios II IDE and ModelSim. 1.5.3 Prospect of the System This system does not expect to solve all the issues in face recognition, such as extreme facial expression, wearing on the face, great age discrepancy, extreme lightening condition, and without frontal face information. In the system, front-end facial feature extraction is based on a FPGA hardware implementation, and face recognition is software based, which the recognition unit is implemented on either an embedded system on FPGA or PC system, which depends on the capacity of the FPGA used.
9

De La Salle University, Manila

Face Recognition System, 2009

Chapter II: Literature Review
2.1 Reviews of Face Recognition Technology There are some researchers have done some reviews in the domain of face recognition technology. In a review by Andrea F. Abate et al. (2007) [12], the paper studies the recent face recognition trend in 2D imagery and 3D model based algorithms. In the article, the discussion includes most important 2D and 3D face model databases, see subsection 2.1.4, issues on face recognition, see subsection 2.1.1, old and new methods in face recognition, and experimental results. Next, in a survey by W. ZHAO et al. (2003) [10], the paper conducts a contemporary survey of still-based and video-based face recognition research, see subsection 2.1.3. In the paper, some available commercial face recognition systems are listed. Besides from the general issues [12] in the face recognition, the psychophysics and neuroscience issues relevant to face recognition are also discussed in this paper; for example, is face recognition a dedicated process, is face perception the result of holistic, and ranking of significance of facial features etc. The methods of key steps, such as face detection and feature extraction, prior to recognition are also discussed. Finally, the protocols of evaluation of face recognition are described. Next, in a survey by Erik Hjelmas and Boon Kee Low (2000) [11], the paper presents a comprehensive and critical survey of face detection algorithms, which are the first step in the
10

De La Salle University, Manila

Face Recognition System, 2009

face recognition systems. The technical approach and performance of the algorithms are discussed in this paper, see subsection 2.1.2. Next, in a survey by J. Kittler el al. (2005) [16], this paper studies 3D imaging, modeling and recognition approaches. The review presents 3D face modeling, 3D to 3D and 3D to 2D registration, 3D based recognition and 3D assisted 2D based recognition, see subsection 2.1.3.3. Finally, in the subsection 2.2, some researches in face recognition based on hardware implementation are presented. 2.1.1 Issues on Face Recognition Illumination issue Illumination variation could be caused by environment lightening condition, reflectance properties of skin, which different race has different complexion, and configurations of a camera, such as white balance and sensitivity. Because automatic face recognition can be seen as a pattern recognition issue [12], the result will be upset due to illumination changes patterns of a image dramatically in face recognition system. Pose issue Variation of poses, such as head pans and tilts, could influence the result of recognition. Some researches use multi-view approach, which need different
11

De La Salle University, Manila

Face Recognition System, 2009

poses images as training images. Another approach is 3D face recognition, which is more robust to pose variation [12][16]. Occlusion issue Occlusion problem can be seen as expression issue. For some algorithms, such as appearance-based approach, occlusion of eyes and mouth will dramatically influence the accuracy of recognition. One way to deal with this issue is to use local approach, which divide the face into different parts and find the best match [12]. Other issues There are some other issues, such as the wearing on the face, complex background, and the aging issue, which could make the recognition more complicated. 2.1.2 Face Detection 2.1.2.1 Local/Feature Based Low-level analysis In human vision, some visual features, such as edges, color, and motion are derived in our inner retina, prior to high-level visual activities in the brain.

12

De La Salle University, Manila

Face Recognition System, 2009

Edges In early face detection, some approaches use edge features within the head outline to identify the face by using shape and position information of the face. In an edge-detection-based approach, edges need to be labeled and matched to a face model in order to verify correct face detections. Gray-levels The dark color within a face outline can also be used as features. Facial features, such as eyebrows, pupils, and lips appear generally darker than other facial regions. The extraction of dark features is implemented by low-level gray-scale thresholding. Color In face detection, many researched use skin color to detect the face. There are several skin color models being built by researchers. These models are in various color formats, such RGB, HSI, YIQ, HSV, YUV, and other color formats. Motion For video sequences source, the moving objects could be located
13

De La Salle University, Manila

Face Recognition System, 2009

as possible face region. Then the region is put into analysis by searching the face features. Generalized measure Facial features are in symmetry, which could be used as criteria to verify face detection. The symmetry operator and symmetry magnitude map are built as model to verify the measurement. Feature analysis The results generated from low-level analysis are likely to be ambiguous. For example, after finding a possible face region by using a skin color model or motion, the detected region is still needed to be verified. Feature searching Feature searching is based on the prominent facial features, eyes, nose, and mouth. A pair of eyes is the most commonly used in facial feature searching. After finding a possible eye pair, then the algorithm moves on to search for a nose or a mouth. In addition, face on top of shoulder also implies that a small area on top of a larger area could be a head.

14

De La Salle University, Manila

Face Recognition System, 2009

Constellation analysis In face detection, some algorithms are able to deal with missing features and problems due to translation, rotation, scale, poses, eyeglasses, and complex backgrounds. For the different poses, the probabilistic face models based on multiple face appearance have also been proposed, which faces are classified under different viewpoints. A Bayesian network, reported by Yow and Cipolla in 100 lab scene images, is able to cope with small variations in scale, orientation, viewpoint, the presence of eyeglasses, and missing features. Active shape models The low edge contrast of features makes the edge detection process problematic due to the lightening condition or skin color blending with background color. For facial feature extraction, active shape model will interact with local image features, such as edges and illumination, and gradually change to take the shape of the feature. Snakes The algorithm, active contours or snakes, are commonly used to form a shape of boundary. The evolution of a snake is achieved by
15

De La Salle University, Manila

Face Recognition System, 2009

minimizing an energy function, Esnake, denoted as Esnake = Einternal + Eexternal ,where Einternal, Eexternal are the internal and external energy functions, respectively. Deformable templates A deformable template is using 11 parameters based on its salient features. The mechanism involves the external energy, valley, edge, peak, and image brightness (Ev, Ee, Ep, Ei ) given by E = Ev + Ee + Ep + Ei + Einternal
E int ernal = k K1 k k 1 1 ( xe − x c ) 2 + 2 ( p1 + {r + b}) 2 + 2 ( p 2 + {r + b}) 2 + 3 (b − 2r ) 2 2 2 2 2 2 2

where the energy k1, k2, k3 control the course and the deformation of the template. Point distribution models(PDMs) The contour of PDM is discrete labeled points in a set. These points are parameterized over training by using different objects of sizes and poses. The model denoted as:
x = x + Pv , where x represents a point on the PDM, x is the mean

value of that point in the training set, P = [p1 p2 … pt ] is the matrix of the t most significant variation vectors of the covariance of deviations, and v is the weight vector for each mode.
16

De La Salle University, Manila

Face Recognition System, 2009

2.1.2.2 Holistic/Image Based Linear subspace methods Subspaces of a human face can be extracted from the overall image space. There are several approaches use subspaces to represent a human face. The algorithms include principal component analysis (PCA), linear discriminant analysis (LDA), and factor analysis (FA). Some examples of eigenfaces, PCA, can be seen in Fig. 2.1.2.2-1

Fig. 2.1.2.2-1 The number below each image indicates the principal component number, ordered according to eigenvalues.

17

De La Salle University, Manila

Face Recognition System, 2009

Neural networks In pattern recognition including face detection and recognition, neural networks have become popular in recent years. Besides simple MLP network, there are other architectures proposed by researchers, such as modular architectures, committee–ensemble classification, complex learning algorithms, auto-associative and compression networks, and networks evolved or pruned with genetic algorithms. Statistical approaches Different from other approaches, statistical approaches, such as systems based on information theory, a support vector machine and Bayes’ decision rule, are also proposed in face detection and recognition. Each statistical approach provides different training and learning algorithms. 2.1.3 Facial Feature Extraction and Recognition 2.1.3.1 Categorization of Still Face Recognition Techniques 2.1.3.1.1 Holistic/Image Based Principal-component analysis (PCA) Eigenfaces Direct application of PCA [Craw and Cameron 1996; Kirby and Sirovich 1990; Turk and Pentland 1991] Probabilistic eigenfaces Two-class problem with prob. measure [Moghaddam and Pentland 1997] Fisherfaces/subspace LDA
18

De La Salle University, Manila

Face Recognition System, 2009

FLD on eigenspace [Belhumeur et al. 1997; Swets and Weng 1996b; Zhao et al. 1998] SVM Two-class problem based on SVM [Phillips 1998] Evolution pursuit Enhanced GA learning [Liu and Wechsler 2000a] Feature lines Point-to-line distance based [Li and Lu 1999] ICA ICA-based feature analysis [Bartlett et al. 1998] LDA/FLD LDA/FLD on raw image [Etemad and Chellappa 1997] PDBNN Probabilistic decision based NN [Lin et al. 1997] 2.1.3.1.2 Local/Feature Based Pure geometry methods Earlier methods [Kanade 1973; Kelly 1970]; recent methods [Cox et al. 1996; Manjunath et al. 1992] Dynamic link architecture Graph matching methods [Okada et al. 1998; Wiskott et al. 1997] Hidden Markov model HMM methods [Nefian and Hayes 1998; Samaria 1994; Samaria and Young 1994] Convolution Neural Network SOM learning based CNN methods [Lawrence et al. 1997] 3.1.3.1.3 Hybrid method Modular eigenfaces Eigenfaces and eigenmodules [Pentland et al. 1994] Hybrid LFA Local feature method [Penev and Atick 1996] Shape-normalized Flexible appearance models [Lanitis et al. 1995] Component-based Face region and components [Huang et al. 2003]
19

De La Salle University, Manila

Face Recognition System, 2009

2.1.3.2 Categorization of Video-based Face Recognition Techniques Still-image methods Basic methods [Turk and Pentland 1991; Lin et al. 1997; Moghaddam and Pentland 1997; Okada et al. 1998; Penev and Atick 1996; Wechsler et al. 1997; Wiskott et al. 1997] Tracking-enhanced [Edwards et al. 1998; McKenna and Gong 1997, 1998; Steffens et al. 1998] Multimodal methods Video- and audio-based [Bigun et al. 1998; Choudhury et al. 1999] Spatiotemporal methods Feature trajectory-based [Li and Chellappa 2001; Li et al. 2001a] Video-to video methods [Zhou et al. 2003]

2.1.3.3 3D Face Recognition In 3D face recognition, the first step is to construct a 3D model by 3D sensing for facial biometrics. The visual reconstruction techniques are in two categories, such as active and passive sensing, which passive sensing reconstructs facial appearance directly from the images or video. After receiving the row facial information, the 3D face models, such as simple models, biomechanical models, and morphable models, are used to build to 3D face database. The automatic face registration is 3D to 3D registration or 3D to 2D registration. In 3D Recognition, recognition is using 3D shape only, 3D shape and texture, or 3D shape assisted 2D recognition.

20

De La Salle University, Manila

Face Recognition System, 2009

2.1.4 Face Database Refer to papers [10][12] for the details of 2D/3D face databases. Table 2.1.4-1 Important 2D face databases Name RGB/ gray Size No. of people Ima ges/ pers on AR Face Database* RGB 576x 126 768 Richard’s MIT database CVL Database RGB RGB 70M 67F 6 p,o Yes 82M 74F 7 p,e Yes http://www.lrv.fri.un i-lj.si/facedb.html 576 p,i Yes http://cvc.yale.edu/p rojects/yalefacesB/y alefacesB.html The Yale Face Database* Gray Scale 320x 15 243 14M 1F PIE Database* RGB 640x 68 486 FERET* Gray RGB 256x 30,000 384 ~60 8 p,i,e,i/ o,t Yes p,i,e Yes 11 i,e Yes http://cvc.yale.edu/p rojects/ yalefaces/yalefaces. html http://www.ri.cmu.e du/projects/project_ 418.html http://www.itl.nist.g ov/iad/humanid/ feret/ 108M 6F The Yale Face Database B* Gray Scale 640x 10 480 26 i,e,o,t Yes http://rvl1.ecn.purdu e.edu/~aleix/aleix_f ace_DB.html Condit ons avail able URL

480x 154 640

640x 114 480

The ‘*’ points out most used databases. Image variations are indicated by (i) illumination, (p) pose, (e) expression, (o) occlusion, (i/o) indoor/outdoor conditions and (t) time delay.
21

De La Salle University, Manila

Face Recognition System, 2009

2.2 A Review of Hardware Based Face Recognition There are many researches related to face recognition based on software implementation. However, only a few face recognitions based on hardware implementation. In recent years, FPGA-based face recognition becomes popular for researchers due to FPGA’s high performance and reconfigured ability. In a paper by T. Nakano et al. (2003) [17], coarse region segmentation based on FPGA is used to detect face regions in an image. The video rate with 64×64 pixels is achieved by using the FPGA implementation. A template matching based on dynamic-link architecture is performed on the PC system. The architecture can be seen in Fig. 2.2-1

Fig. 2.2-1 Face/object recognition system Next, in a research by Xiitoguaiig Li and Shawlti Areibi (2004) [18], face recognition is using Artificial Neural Networks (ANN) based on hardware update module (HUM). Both a processor and HUM are implemented on a FPGA chip. The results only include the FPGA usage
22

De La Salle University, Manila

Face Recognition System, 2009

consumed by the system and the time consumed by feed forward calculation, backward calculation, and updating. Next, in a research by Y M. Mustafah et al. (2007) [19], a smart camera is developed based on PicoBlaze microcontroller using FPGA to extract region of interest (ROI). Then the high resolution face image is sent to client PC to perform the face recognition task. In this research a HW/SW co-design architecture is implemented. Fig.2.2-2 shows the system architecture and Fig.2.2-3 shows face extraction in different resolutions.

Fig. 2.2-2 System Architecture

Fig. 2.2-3 Overall Scene (a), ROI extracted from scene with resolution of 7MP (b), 5 MP(c), 3MP (d), IMP (e) and VGA (f).
23

De La Salle University, Manila

Face Recognition System, 2009

Next, in a research by Mohan A.R. et al. (2008) [20], Principal Component Neural Network (PCNN) based on FPGA in face recognition system is designed. Result presents real-time video surveillance and high-speed access control. Next, in a research by I. Sajid, M. M. Ahmed (2008) [21], the paper presents a fixed point technique with software hardware co-design due to the floating point operations in Eigen values algorithms are costly and complex in terms of hardware. The system architecture can be seen in Fig 2.2-4. The result shows that the implementation of householder (HH) algorithm saves power in the cost of losing precision no more than .008 percent in the diagonal values of computed matrix.

Fig. 2.2-4 High level description

24

De La Salle University, Manila

Face Recognition System, 2009

Next, in a book by Ginhac Dominique et al. (2007) [22], in chapter eight, three different hardware platforms, such as FPGA, zero instruction set computer (ZISC) chips(Fig), and digital signal processor (DSP) TMS320C62, dedicated to face recognition are presented. The result shows the success rate of face tracking and identifying is 92%, 85%, and 98.2% respectively in FPGA, ZISC, and DSP. Processing speeds are respectively 14, 25, and 4.8 images/s of in image size 288 x 352.

Fig. 2.2-5 Implementation based on FPGA

Fig. 2.2-6 Implementation based on ZISC Chip

Fig. 2.2-7 Implementation based on DSP
25

De La Salle University, Manila

Face Recognition System, 2009

Next, in a research by P. Rauschert et al. (2002) [23], a template matching in frequency domain based on FPGA face recognition is presented in this paper. The hardware structure can be seen in Fig. 2.2-8. The result shows that the identification rate is around 90% and the utilization of FPGA is 37.84% logic device and 96.875% of block RAMs on Xilinx FPGA XCV1000.

Fig. 2.2-8 Identification Flow Next, in a research by Nasim Shams et al. (2006) [24], face recognition system based on Daubechies wavelets algorithm is implemented on a 300K gates Spartan-Il FPGA, as can be seen in Fig. 2.2-9. The result show that the system processes in high frame per second.

Fig. 2.2-9 Floorplan on XC2S300E-6PQ20 using Xilinx PACE.
26

De La Salle University, Manila

Face Recognition System, 2009

Next, in a research by Gonzalo Carvajal et al. (2007) [25], an analog-VLSI neural network for face recognition based on subspace methods is proposed in this paper. As can be seen in Fig. 2.2-10, the dimensionality reduction is based on principal components analysis (PCA) and Linear Discriminant Analysis (LDA), whose coefficients can be either programmed or learned on-chip.

Fig. 2.2-10 System Architecture Next, in a research by Rajkiran Gottumukkal and Vijayan K. Asari (2003) [26], a fast parallel architecture, as can be seen in Fig 2.2-11, based on Composite PCA face recognition is proposed in this paper. The system works on both feature extraction and classification. The result shows that the system is able to identify a person from a database of 110 images of 10 individuals in approximately 4 ms.

Fig 2.2-11 Block diagram of (a) PE1, (b) PE2.
27

De La Salle University, Manila

Face Recognition System, 2009

Next, in a research by G. F. Zaki et al. (2007) [14], a hardware/software co-design face recognition system on an FPGA is proposed. As can be seen in Fig. 2.2.12, the system uses hardware acceleration module and a processor on FPGA. In this research, face detection is in a unified simple background. Features Extraction is using PCA by implementing hardware acceleration module. The result shows that proposed architecture is 1.7 times faster compared to the system implemented without HW acceleration and the recognition efficiency achieves 85.3%.

Fig. 2.2.12 System Block Diagram To sum up, only a few face recognition systems have been developed based on hardware implementation. The works based on hardware studied in this paper, some of the works [20][21][23][24] implement only with still images input instead of video image from the nature scene. In addition, the researches are conducted under different input source image. Also, some of the results didn’t present the accuracy of the recognition.

28

De La Salle University, Manila

Face Recognition System, 2009

Chapter III: Theoretical Framework
3.1 System Architecture In this research, face recognition system is composed of three units, which are input, process, and output unit. The process unit deals with the image processing, face recognition and data transmission, as can be seen in Fig. 3-1. A face image, row data received from an image CMOS sensor, input to the system, which the image processing unit analyzes the input data and outputs the result, facial feature vectors, to the next stage. Next, the face recognition unit identifies the person by comparing the input vectors with the facial feature vectors in the face database, and finally outputs a possible identity to a monitor.

Fig. 3-1 System Architecture Block Diagram The process unit is implemented based on Altera StratixII FPGA Develop Kit. The Image COMS Sensor is connected to the I/O pin, which the image sensor interface is implemented on FPGA.

29

De La Salle University, Manila 3.2 Image Processing Unit

Face Recognition System, 2009

The major computational tasks are in the image processing unit. The unit includes six procedures, moving object filter, face detection, frontal face estimation, facial feature localization, illumination normalization and facial feature extraction, as can be seen in Fig.3.2. Taking the advantage of video sequences, moving object detection and segmentation are developed to extract foreground objects from video images. After extracting foreground objects, complex background noises are eliminated. Therefore, less efforts are needed in the next phase. For face detection, foreground objects are to be analyzed to find face regions. Once the face regions are found, face pose estimation is executed to find frontal faces needed in the next procedure. Next, key region of the faces is extracted and put into illumination normalization. Finally, facial features are extracted and transformed for recognition.

Fig. 3-2 Image Processing Unit Block Diagram

30

De La Salle University, Manila 3.3 Face Recognition Unit

Face Recognition System, 2009

In Face recognition unit, there are two data storages and an identification unit, as can be seen in Fig.3-3. The facial feature data will be stored in a face database, and the recent matched face vector will be put in the look up table (LUT). Once the image process unit outputs test face features. The face identification unit first compares the test image with the data in the LUT, which is to shorten the searching time in the database, before comparing the data in the database. There is pre-computed threshold to verify the value of closest matching between test image and the images in database or LUT.

Fig. 3-3 Face Recognition Unit Block Diagram

31

De La Salle University, Manila

Face Recognition System, 2009

Chapter IV: Methodology
4.1 Development In this research, the system is implemented by software and hardware co-design. The image processing unit, NIOS MCU, and other peripheral interfaces are developed using VHDL hardware language based on FPGA. As for the face recognition, applications are developed using Altera’s software development IDE environment, which the applications are running in an embedded OS system on NIOS MCU. 4.1.1 Image Processing Unit Image processing unit includes six procedures, moving object filter, face detection, frontal face estimation, facial feature localization, illumination normalization, and facial feature extraction. 4.1.1.1 Moving Object Filter For the video sequence image, the foreground objects can be abstracted before detecting the facial regions. To do so, we can eliminate the background noise and narrow the range of the processing area. There is variety of object segmentation algorithms, based on motion-based and spatio-temporal [2]. However, there are advantages and disadvantages of each algorithm [3], which each method deals with different issues, such as illumination, complex background, blending, and other issues.
32

De La Salle University, Manila

Face Recognition System, 2009

In order to deal with the globe illumination changes, A Kalman filter based background updating algorithm [1] is selected to do the work in this research. The algorithm is robust to gradual and sharp illumination changes, which the result can be seen in Fig 4.1.1.1-1.

Fig 4.1.1.1-1 Three background images and one foreground result 4.1.1.2 Face Detection For face detection, two approaches, color-based and feature-based face detection, are studied in this research. There are many researches using color-based algorithm to detect the color skin of faces. However, the skin-color models are varied due to nature conditions [4], such as the lighting condition, the white-balance of cameras, and the skin color of various race etc. Component-based face detection [5] is developed in this research thanks to the moving object extraction, which eliminates complex background in an image. The component-based face detection uses the facial features and its geometry, see Fig 4.1.1.2-1, to verify the candidate face region.

33

De La Salle University, Manila

Face Recognition System, 2009

Fig 4.1.1.2-1 Schematic template and geometry relations 4.1.1.3 Frontal Face Estimation Once face region is located, the face poses is estimated to search a frontal face. Besides from learning-based face pose estimation, an asymmetry and geometry based method [6] is proposed due to human face is composed in certain geometry. As a result, different face pose geometry model can be built beforehand, as can be seen in Fig 4.1.1.3-1.

Fig 4.1.1.3-1 Face Pose Model

34

De La Salle University, Manila

Face Recognition System, 2009

4.1.1.4 Facial Feature Localization After detecting a face, a key region of the face is extracted from the face image. An extraction method is using the geometry of common facial proportion of eyes and mouth, see Fig 4.1.1.4-1.

Fig 4.1.1.4-1 Facial Feature Localization 4.1.1.5 Illumination Normalization In many face recognition systems, accuracy drop sharply because of varying illumination condition. In this research, illumination normalization is proposed instead of using variety of training images to adapt illumination. A modified local binary pattern (mLBP) is developed to compensate illumination. The mLBP preprocessing has the high recognition rate in the eigenspace-based face recognition [7]. A result is showing in Fig. 4.1.1.5-1.

35

De La Salle University, Manila

Face Recognition System, 2009

A. Original Face

B. M+LBP

Fig. 4.1.1.5-1 A. Original image from the Yale B database B. Image pre-processed by mLBP 4.1.1.6 Facial Feature Extraction Eigenface based, so-called Principal Component Analysis (PCA), recognition have been improved effectively in face recognition by many researchers. In addition, a modular PCA algorithm has been implemented successfully using FPGA with multi-lane architecture [8]. In this research, modular PCA-based recognition and mLBP illumination Normalization collaborate to do the job of one image per person face recognition.

L ⎛L ⎞ Eq. 4.1.1.6-1. I ijk (m, n) = I i ⎜ ( j − 1) + m, (k − 1) + n ⎟, ∀i, j , k , m, n N ⎝N ⎠ where i:1 to M ; M: number of images in training set. j,k: 1 to N; N2 : number of sub-images. m,n: 1 to L/N Eq. 4.1.1.6-2. A =
1 MN 2

∑ ∑ ∑I
i =1 j =1 k =1

M

N

N

ijk

Eq. 4.1.1.6-3. Wtest , jkr = E rT ( I test , jk − A), ∀j , k , r
36

De La Salle University, Manila

Face Recognition System, 2009

The size of face image L×L are divided into N2 smaller blocks, so the size of sub-image would be L2 / N2 . Those sub-images Ii represent as a function of the original image, see Eq. 4.1.1.6-1. The mean image of all the test sub-images is computed as Eq. 4.1.1.6-2. The weight vector of test image is computed for each of the sub-image using the eigenvectors as Eq. 4.1.1.6-3.

4.1.2 Face Recognition Unit
Every test image from image processing unit will compare with face images in the look up table before going through the face database.

4.1.2.1 Face Identification
Dijk , see Eq. 4.2.1 -1, represent the distances between the test image’s weight vector and the weight vectors of the images in the database. The minimum value of Di , see Eq. 4.2.1 -2, means the ith face image in the database resembles closest to the test face image. Eq. 4.2.1 -1 Dijk = Eq. 4.2.1 -2 Di = 1 M' ∑ Wtest , jkr − Wijkr , ∀i, j, k M ' r =1

1 N2

∑ ∑D
j =1 k =1

N

N

ijk

, ∀i

A pre-defined threshold value is compared with the minimum distance, which if the minimum distance exceeds the threshold value the test image is rejected as face image.

37

De La Salle University, Manila

Face Recognition System, 2009

4.1.2.1 Look Up Table
In order to shorten the searching time from the face database, recent hit of ten test images will be stored in a look up table (LUT). The LUT stores the distance of hit images and indexes linking to the face database.

4.1.2.3 Face Database
One frontal face image for each person is required for the system. The Di and personal ID of training face images will be stored in the face database.

4.1.3 Peripheral Interfacing 4.1.3.1 Interface to Image CMOS Sensor
Row data are received from image COMS sensor in order to reduce the computational tasks. An interface of a sensor, likely USB, I2C, CAN bus, depends on manufacturers’ models, which the interface of the sensor is implemented.

4.1.3.2 Interface to Monitor
VGA interface is developed for the communication between FPGA and a monitor.

4.2 Experimentation and Evaluation
In this research, algorithms being selected or created will be put into test before doing hardware evaluation. In early experiment, software programs are created for each procedure to see the effect. Next, hardware evaluation will be implemented on a FPGA.
38

De La Salle University, Manila

Face Recognition System, 2009

4.2.1 Software Simulation
The proposed system will be put into test on a PC by means of software programming before conducting a hardware experiment. For software programming, C/C++ programming language is used to implement the algorithms. In this stage, the accuracy of recognition is emphasized instead of the system performance.

4.2.2 Hardware Evaluation
For hardware evaluation, Altera NIOS StratixII FPGA kit [9] (see Fig. 4.2.2-1), a monitor, and an image CMOS sensor are needed to do the experiment. For hardware programming language, VHDL language and Altera QuartusII are used to do the hardware simulation. The conditions of the system evaluation should include real scene experiment, accuracy of recognition, and performance of recognition. The criteria of these conditions such as variations of the source image, recognition rate, processing frames per second are measured.

Fig. 4.2.2-1 Altera NIOS StratixII
39

De La Salle University, Manila

Face Recognition System, 2009

Chapter V: Summary

In face recognition system, face detection and facial feature extraction are key steps before into face identification. For video-based face recognition, foreground and background objects can be separated by detecting moving objects, which eliminating complex background facilitates tasks of face detection and recognition. However, in face recognition, there are some issues, such as illumination and pose variation, should be resolved to improve the accuracy. To deal with these problems, several algorithms are proposed in this research, such as illumination normalization and pose estimation. In addition, the face recognition in this system requires only one image per person in face database, which gathering multiple images of various poses or illumination is laborious work. Furthermore, a flexible architecture is designed in order to integrate these algorithms selected in this system. The system will be simulated by software programming and implemented on hardware using FPGA. Finally, the system will be put into test in real environment.

40

De La Salle University, Manila

Face Recognition System, 2009

References
[1] Stefano Messelodi, Carla Maria Modena, Nicola Segata, and Michele Zanin (2005), Kalman filter based background updating algorithm robust to sharp illumination changes, Lecture notes in computer science, 2005 - Springer [2] Dengsheng Zhang and Guojun Lu (2001),Segmentation of Moving Objects in Image Sequence: A Review, Circuits, Systems, and Signal Processing, 2001 - Springer [3] D. Gutchess, M. Trajkovi´c, E. Cohen-Solal, D. Lyons (2001), A. K. Jain†, A Background Model Initialization Algorithm for Video Surveillance, Conference on Computer Vision, Vancouver, Canada, 2001 – ieeecomputersociety [4] Shinjiro Kawato and Jun Ohya (2000), Automatic Skin-color Distribution Extraction for Face Detection and Tracking, 2000 IEEE [5] Kyoung-Mi Lee (2007), Component-based face detection and verification, 2007 Elsevier B.V. [6] Yuxiao Hu, Longbin Chen, Yi Zhou, Hongjiang Zhang (2004), Estimating Face Pose by Facial Asymmetry and Geometry, 2004 IEEE [7] Javier Ruiz-del-Solar, Julio Quinteros (2008),Illumination compensation and normalization in eigenspace-based face recognition: A comparative study of different pre-processing approaches, 2008 Elsevier B.V. [8] Rajkiran Gottumukkal, Hau T. Ngo, Vijayan K. Asari (2005),Multi-lane architecture for eigenface based real-time face recognition, 2005 Elsevier B.V [9] ALTERA, Nios Development Board Reference Manual, Stratix II Edition, Nov 1, 2009 http://www.altera.com [10] W. ZHAO, R. CHELLAPPA, P. J. PHILLIPS and A. ROSENFELD (2003), Face Recognition: A Literature Survey, ACM Computing Surveys, Vol. 35, No. 4, December 2003,

41

De La Salle University, Manila pp. 399–458.

Face Recognition System, 2009

[11] Erik Hjelmas (2001), Face Detection: A Survey, 2001 by Academic Press [12] Andrea F. Abate, Michele Nappi, Daniel Riccio, Gabriele Sabatino (2007), 2D and 3D face recognition: A survey, 2007 Elsevier B.V. [13] Xiaoyang Tan, Songcan Chen, Zhi-Hu Zhou, Fuyan Zhang (2006), Face recognition from a single image per person:Asurvey, 2006 Elsevier B.V. [14] G. F. Zaki, R. A. Girgis, W. W. Moussa, W. R. Gabran (2007), Using the Hardware/Software Co-design Methodology to Implement an Embedded Face Recognition/Verification System on an FPGA, 2007 IEEE [15] Elham Bagherian, Rahmita Wirza O.K. Rahmat (2008), Facial feature extraction for face recognition: a review, 2008 IEEE [16] J. Kittler, A. Hilton, M. Hamouz, J. Illingworth (2005), 3D Assisted Face Recognition: A Survey of 3D Imaging, Modelling and Recognition Approaches, 2005 IEEE [17] T. Nakano, T. Morie, and A. Iwata (2003), A Face/Object Recognition System Using FPGA Implementation of Coarse Region Segmentation, 2003 SICE [18] Xiitoguaiig Li and Shawlti Areibi (2004), A Hardware/Software Go-design Approach for Face Recognition, 2004 IEEE [19] Y M. Mustafah, A. W. Azman, A. Bigdeli, B. C. Lovell (2007), AN AUTOMATED FACE RECOGNITION SYSTEM FOR INTELLIGENCE SURVEILLANCE: SMART CAMERA RECOGNIZING FACES IN THE CROWD, 2007 IEEE [20] Mohan A.R. ,N. Sudha and Pramod K. Meher (2008), An Embedded Face Recognition System on A VLSI Array Architecture and its FPGA Implementation, 2008 IEEE [21] I. Sajid, M. M. Ahmed, I. Taj, M. Humayun, and F. Hameed (2008), Design of High Performance FPGA Based Face Recognition System, PIERS Proceedings, Cambridge, USA,
42

De La Salle University, Manila July 2-6, 2008

Face Recognition System, 2009

[22] Ginhac Dominique, Yang Fan and Paindavoine Michel (2007), Design, Implementation and Evaluation of Hardware Vision Systems dedicated to Real-Time Face Recognition, Face Recognition, Book edited by: Kresimir Delac and Mislav Grgic, ISBN 978-3-902613-03-5, pp.558, I-Tech, Vienna, Austria, June 2007 [23] P. Ravschert, A. Kummert, M. Krips and Y. Klimets (2002), Face Recognition By Means of Template Matching in Frequency Domain – A Hardware Based Approach, 2002 IEEE [24] Nasim Shams, Iraj Hosseini, Mohammad Sadegh Sadri, Ehsan Azarnasab (2006), LOW COST FPGA-BASED HIGHLY ACCURATE FACE RECOGNITION SYSTEM USING COMBINED WAVELETS WITH SUBSPACE METHODS, 2006 IEEE [25] Gonzalo Carvajal, Waldo Valenzuela and Miguel Figueroa (2007), Subspace-Based Face Recognition in Analog VLSI, Nov 1, 2009, books.nips.cc [26] Rajkiran Gottumukkal, Vijayan K. Asari (2003), System Level Design of Real Time Face Recognition Architecture Based on Composite PCA, 2003 ACM

43

Sign up to vote on this title
UsefulNot useful