Professional Documents
Culture Documents
-----------------------------------
De La Salle University
------------------------------------
In Partial Fulfillment of
Master of Science in
-------------------------------------
By
PAO-CHUNG, CHAO
November 2009
De La Salle University, Manila Face Recognition System, 2009
Abstract
Face recognition has received significant attention in recent years. Many researches
have been done in the domain of face detection and recognition. However, there are only
few works being implemented on hardware and put into test in real environment. In this
research, face detection and facial feature extraction are developed based on a hardware
using FPGA. Well-known algorithms in moving object detection, face detection, poses
selected and implemented. The performance and accuracy of the system are experimented
using software simulation and hardware implementation. For face recognition, one image
per person in face database is the basic requirement in the system. Finally, the system will
1
De La Salle University, Manila Face Recognition System, 2009
TABLE OF CONTENTS
Chapter I: Introduction...........................................................................................................5
2
De La Salle University, Manila Face Recognition System, 2009
Chapter V: Summary.............................................................................................................40
References...............................................................................................................................41
3
De La Salle University, Manila Face Recognition System, 2009
LIST OF SYMBOLS
FPGA (Field Programmable Gate Array)
FA (Factor Analysis)
GA (Genetic Algorithm)
4
De La Salle University, Manila Face Recognition System, 2009
Chapter I: Introduction
Due to many reasons, such as authority management and security issue since the 9/11
terrorist attacks, a sophisticated security system become more important in our daily life,
especially for person identification. Nowadays, there are various ways to identify a person,
which could be classified into two categories, such as biometric and non-biometric methods [10].
On the one hand, non-biometric identification system uses a serious numbers of personal ID and
password to identify a person, such as ID cards (student/citizen), credit card, system password,
and other ID used to represent a person. However, non-biometric ID could be forged or taken by
other people, which system can not verify the validity of the ID holder.
On the other hand, there are some successful biometric recognition systems, using iris,
finger print, voice, palm geometry, signature, and facial features to identify a person, which these
systems have high accuracy in recognition [12]. However, some of the biometric recognition
systems, such as iris and finger print scanning, are intrusive identification, which the systems
need the cooperation of users and the equipments needed are usually expensive [12]. On the
contrary, in a face recognition system, cameras are installed at a surveillance place, which the
system can capture all the objects in real time without being noticed. As a result, face recognition
5
De La Salle University, Manila Face Recognition System, 2009
Moreover, many researches have been done in face detection and recognition [10][11][12]
in recent years. In these papers, the results show high accuracy and excellent performance.
However, most of the researches are based on certain preconditions, and only few works have
been practiced in real environment due to the performance issue. In this research, software and
hardware based face recognition techniques are studied. The face recognition system is
developed based on hardware implementation and put into test in real environment.
In the past few years, face recognition has attracted great interest by many researchers.
There are many well-known algorithms are created in the recent years [12]. In face recognition,
the simple process of face recognition system should go through image data retrieval, face
detection, facial feature extraction and face recognition. However, some researches focus on a
part of face recognition system, such as face detection, face recognition, or algorithms dealing
with certain issues. Moreover, in real environment, images are from nature scenes instead of
from pre-collected images, which there are many variant factors, such as complex background
noise, lightening condition, poses variation, and other factors [10], under the nature scenes; and
image sequences, the image processing tasks are computational expensive [11], which the
6
De La Salle University, Manila Face Recognition System, 2009
implementation can not be put into a real environment to test. As a result, a hardware
The main objective is to design a face recognition system, which an image processing
unit is implemented based on a hardware using FPGA. In addition, the system will be put
this system.
z Use one image per person to reduce data collection and training time
7
De La Salle University, Manila Face Recognition System, 2009
Using one image per person is required in this system, which many
algorithms require various training data to reach the accuracy [13]. The advantage
of using one image per person is not only to reduce the data collective issue, but
z Algorithms replaceable
In face recognition, new algorithms are presented every year. In this research,
buildings security. For example, students need an ID card, RFID (Radio-frequency identification)
card, to pass the entrance gate in De La Salle University. However, the ID card could be used by
other people, which is not the person registered to the card. To deal with this, security guards use
their bare eyes to compare a picture on the screen with the person. In this occasion, this system
8
De La Salle University, Manila Face Recognition System, 2009
could help them to do the work. Also, this system can easily be integrated to a present face
database in the school due to the system only need one image per person in the database.
In this research, some hardware equipments and software tools are needed to do the
experiment. In addition, there are some limitations for the system functions.
In this research, the key equipments used are computers, an Altera NIOS StratixII
Some software programming tools are needed in this research, which are a C/C++
This system does not expect to solve all the issues in face recognition, such as extreme
facial expression, wearing on the face, great age discrepancy, extreme lightening condition,
and without frontal face information. In the system, front-end facial feature extraction is
based on a FPGA hardware implementation, and face recognition is software based, which
9
De La Salle University, Manila Face Recognition System, 2009
There are some researchers have done some reviews in the domain of face recognition
technology. In a review by Andrea F. Abate et al. (2007) [12], the paper studies the recent face
recognition trend in 2D imagery and 3D model based algorithms. In the article, the discussion
includes most important 2D and 3D face model databases, see subsection 2.1.4, issues on face
recognition, see subsection 2.1.1, old and new methods in face recognition, and experimental
results.
Next, in a survey by W. ZHAO et al. (2003) [10], the paper conducts a contemporary survey
of still-based and video-based face recognition research, see subsection 2.1.3. In the paper, some
available commercial face recognition systems are listed. Besides from the general issues [12] in
the face recognition, the psychophysics and neuroscience issues relevant to face recognition are
also discussed in this paper; for example, is face recognition a dedicated process, is face
perception the result of holistic, and ranking of significance of facial features etc. The methods of
key steps, such as face detection and feature extraction, prior to recognition are also discussed.
Next, in a survey by Erik Hjelmas and Boon Kee Low (2000) [11], the paper presents a
comprehensive and critical survey of face detection algorithms, which are the first step in the
10
De La Salle University, Manila Face Recognition System, 2009
face recognition systems. The technical approach and performance of the algorithms are
Next, in a survey by J. Kittler el al. (2005) [16], this paper studies 3D imaging, modeling
registration, 3D based recognition and 3D assisted 2D based recognition, see subsection 2.1.3.3.
Finally, in the subsection 2.2, some researches in face recognition based on hardware
z Illumination issue
reflectance properties of skin, which different race has different complexion, and
automatic face recognition can be seen as a pattern recognition issue [12], the
z Pose issue
Variation of poses, such as head pans and tilts, could influence the result of
11
De La Salle University, Manila Face Recognition System, 2009
z Occlusion issue
Occlusion problem can be seen as expression issue. For some algorithms, such as
influence the accuracy of recognition. One way to deal with this issue is to use
local approach, which divide the face into different parts and find the best match
[12].
z Other issues
There are some other issues, such as the wearing on the face, complex
background, and the aging issue, which could make the recognition more
complicated.
z Low-level analysis
In human vision, some visual features, such as edges, color, and motion are
derived in our inner retina, prior to high-level visual activities in the brain.
12
De La Salle University, Manila Face Recognition System, 2009
Edges
the head outline to identify the face by using shape and position
Gray-levels
The dark color within a face outline can also be used as features.
Color
In face detection, many researched use skin color to detect the face.
There are several skin color models being built by researchers. These
models are in various color formats, such RGB, HSI, YIQ, HSV, YUV,
Motion
13
De La Salle University, Manila Face Recognition System, 2009
as possible face region. Then the region is put into analysis by searching
Generalized measure
z Feature analysis
The results generated from low-level analysis are likely to be ambiguous. For
example, after finding a possible face region by using a skin color model or
Feature searching
nose, and mouth. A pair of eyes is the most commonly used in facial
feature searching. After finding a possible eye pair, then the algorithm
shoulder also implies that a small area on top of a larger area could be a
head.
14
De La Salle University, Manila Face Recognition System, 2009
Constellation analysis
The low edge contrast of features makes the edge detection process
background color. For facial feature extraction, active shape model will
interact with local image features, such as edges and illumination, and
Snakes
15
De La Salle University, Manila Face Recognition System, 2009
Eexternal ,where Einternal, Eexternal are the internal and external energy
functions, respectively.
Deformable templates
E = Ev + Ee + Ep + Ei + Einternal
K1 k 1 k 1 k
E int ernal = ( xe − x c ) 2 + 2 ( p1 + {r + b}) 2 + 2 ( p 2 + {r + b}) 2 + 3 (b − 2r ) 2
2 2 2 2 2 2
where the energy k1, k2, k3 control the course and the deformation of the template.
The contour of PDM is discrete labeled points in a set. These points are
16
De La Salle University, Manila Face Recognition System, 2009
Subspaces of a human face can be extracted from the overall image space.
There are several approaches use subspaces to represent a human face. The
analysis (LDA), and factor analysis (FA). Some examples of eigenfaces, PCA, can
Fig. 2.1.2.2-1 The number below each image indicates the principal
17
De La Salle University, Manila Face Recognition System, 2009
z Neural networks
networks have become popular in recent years. Besides simple MLP network,
z Statistical approaches
rule, are also proposed in face detection and recognition. Each statistical
18
De La Salle University, Manila Face Recognition System, 2009
FLD on eigenspace [Belhumeur et al. 1997; Swets and Weng 1996b; Zhao et
al. 1998]
z SVM
Two-class problem based on SVM [Phillips 1998]
z Evolution pursuit
Enhanced GA learning [Liu and Wechsler 2000a]
z Feature lines
Point-to-line distance based [Li and Lu 1999]
z ICA
ICA-based feature analysis [Bartlett et al. 1998]
z LDA/FLD
LDA/FLD on raw image [Etemad and Chellappa 1997]
z PDBNN
Probabilistic decision based NN [Lin et al. 1997]
z Modular eigenfaces
Eigenfaces and eigenmodules [Pentland et al. 1994]
z Hybrid LFA
Local feature method [Penev and Atick 1996]
z Shape-normalized
Flexible appearance models [Lanitis et al. 1995]
z Component-based
Face region and components [Huang et al. 2003]
19
De La Salle University, Manila Face Recognition System, 2009
facial biometrics. The visual reconstruction techniques are in two categories, such as
active and passive sensing, which passive sensing reconstructs facial appearance
directly from the images or video. After receiving the row facial information, the 3D
face models, such as simple models, biomechanical models, and morphable models,
20
De La Salle University, Manila Face Recognition System, 2009
The ‘*’ points out most used databases. Image variations are indicated by (i) illumination, (p)
pose, (e) expression, (o) occlusion, (i/o) indoor/outdoor conditions and (t) time delay.
21
De La Salle University, Manila Face Recognition System, 2009
There are many researches related to face recognition based on software implementation.
However, only a few face recognitions based on hardware implementation. In recent years,
FPGA-based face recognition becomes popular for researchers due to FPGA’s high performance
and reconfigured ability. In a paper by T. Nakano et al. (2003) [17], coarse region segmentation
based on FPGA is used to detect face regions in an image. The video rate with 64×64 pixels is
architecture is performed on the PC system. The architecture can be seen in Fig. 2.2-1
Next, in a research by Xiitoguaiig Li and Shawlti Areibi (2004) [18], face recognition is
using Artificial Neural Networks (ANN) based on hardware update module (HUM). Both a
processor and HUM are implemented on a FPGA chip. The results only include the FPGA usage
22
De La Salle University, Manila Face Recognition System, 2009
consumed by the system and the time consumed by feed forward calculation, backward
Next, in a research by Y M. Mustafah et al. (2007) [19], a smart camera is developed based
on PicoBlaze microcontroller using FPGA to extract region of interest (ROI). Then the high
resolution face image is sent to client PC to perform the face recognition task. In this research a
HW/SW co-design architecture is implemented. Fig.2.2-2 shows the system architecture and
Fig. 2.2-3 Overall Scene (a), ROI extracted from scene with resolution of 7MP (b), 5 MP(c),
23
De La Salle University, Manila Face Recognition System, 2009
Next, in a research by Mohan A.R. et al. (2008) [20], Principal Component Neural Network
(PCNN) based on FPGA in face recognition system is designed. Result presents real-time video
Next, in a research by I. Sajid, M. M. Ahmed (2008) [21], the paper presents a fixed point
technique with software hardware co-design due to the floating point operations in Eigen values
algorithms are costly and complex in terms of hardware. The system architecture can be seen in
Fig 2.2-4. The result shows that the implementation of householder (HH) algorithm saves power
in the cost of losing precision no more than .008 percent in the diagonal values of computed
matrix.
24
De La Salle University, Manila Face Recognition System, 2009
Next, in a book by Ginhac Dominique et al. (2007) [22], in chapter eight, three different
hardware platforms, such as FPGA, zero instruction set computer (ZISC) chips(Fig), and digital
signal processor (DSP) TMS320C62, dedicated to face recognition are presented. The result
shows the success rate of face tracking and identifying is 92%, 85%, and 98.2% respectively in
FPGA, ZISC, and DSP. Processing speeds are respectively 14, 25, and 4.8 images/s of in image
25
De La Salle University, Manila Face Recognition System, 2009
domain based on FPGA face recognition is presented in this paper. The hardware structure can be
seen in Fig. 2.2-8. The result shows that the identification rate is around 90% and the utilization
of FPGA is 37.84% logic device and 96.875% of block RAMs on Xilinx FPGA XCV1000.
Next, in a research by Nasim Shams et al. (2006) [24], face recognition system based on
Daubechies wavelets algorithm is implemented on a 300K gates Spartan-Il FPGA, as can be seen
in Fig. 2.2-9. The result show that the system processes in high frame per second.
26
De La Salle University, Manila Face Recognition System, 2009
Next, in a research by Gonzalo Carvajal et al. (2007) [25], an analog-VLSI neural network
for face recognition based on subspace methods is proposed in this paper. As can be seen in Fig.
2.2-10, the dimensionality reduction is based on principal components analysis (PCA) and Linear
Discriminant Analysis (LDA), whose coefficients can be either programmed or learned on-chip.
Next, in a research by Rajkiran Gottumukkal and Vijayan K. Asari (2003) [26], a fast
parallel architecture, as can be seen in Fig 2.2-11, based on Composite PCA face recognition is
proposed in this paper. The system works on both feature extraction and classification. The result
shows that the system is able to identify a person from a database of 110 images of 10
27
De La Salle University, Manila Face Recognition System, 2009
recognition system on an FPGA is proposed. As can be seen in Fig. 2.2.12, the system uses
hardware acceleration module and a processor on FPGA. In this research, face detection is in a
acceleration module. The result shows that proposed architecture is 1.7 times faster compared to
the system implemented without HW acceleration and the recognition efficiency achieves 85.3%.
To sum up, only a few face recognition systems have been developed based on hardware
implementation. The works based on hardware studied in this paper, some of the works
[20][21][23][24] implement only with still images input instead of video image from the nature
scene. In addition, the researches are conducted under different input source image. Also, some
28
De La Salle University, Manila Face Recognition System, 2009
In this research, face recognition system is composed of three units, which are input,
process, and output unit. The process unit deals with the image processing, face recognition and
data transmission, as can be seen in Fig. 3-1. A face image, row data received from an image
CMOS sensor, input to the system, which the image processing unit analyzes the input data and
outputs the result, facial feature vectors, to the next stage. Next, the face recognition unit
identifies the person by comparing the input vectors with the facial feature vectors in the face
The process unit is implemented based on Altera StratixII FPGA Develop Kit. The Image COMS
Sensor is connected to the I/O pin, which the image sensor interface is implemented on FPGA.
29
De La Salle University, Manila Face Recognition System, 2009
The major computational tasks are in the image processing unit. The unit includes six
procedures, moving object filter, face detection, frontal face estimation, facial feature
localization, illumination normalization and facial feature extraction, as can be seen in Fig.3.2.
Taking the advantage of video sequences, moving object detection and segmentation are
developed to extract foreground objects from video images. After extracting foreground objects,
complex background noises are eliminated. Therefore, less efforts are needed in the next phase.
For face detection, foreground objects are to be analyzed to find face regions. Once the face
regions are found, face pose estimation is executed to find frontal faces needed in the next
procedure. Next, key region of the faces is extracted and put into illumination normalization.
30
De La Salle University, Manila Face Recognition System, 2009
In Face recognition unit, there are two data storages and an identification unit, as can be
seen in Fig.3-3. The facial feature data will be stored in a face database, and the recent matched
Once the image process unit outputs test face features. The face identification unit first compares
the test image with the data in the LUT, which is to shorten the searching time in the database,
before comparing the data in the database. There is pre-computed threshold to verify the value of
closest matching between test image and the images in database or LUT.
31
De La Salle University, Manila Face Recognition System, 2009
4.1 Development
In this research, the system is implemented by software and hardware co-design. The image
processing unit, NIOS MCU, and other peripheral interfaces are developed using VHDL
hardware language based on FPGA. As for the face recognition, applications are developed using
Altera’s software development IDE environment, which the applications are running in an
Image processing unit includes six procedures, moving object filter, face detection,
frontal face estimation, facial feature localization, illumination normalization, and facial
feature extraction.
For the video sequence image, the foreground objects can be abstracted before
detecting the facial regions. To do so, we can eliminate the background noise and
narrow the range of the processing area. There is variety of object segmentation
advantages and disadvantages of each algorithm [3], which each method deals with
different issues, such as illumination, complex background, blending, and other issues.
32
De La Salle University, Manila Face Recognition System, 2009
In order to deal with the globe illumination changes, A Kalman filter based background
updating algorithm [1] is selected to do the work in this research. The algorithm is
robust to gradual and sharp illumination changes, which the result can be seen in Fig
4.1.1.1-1.
For face detection, two approaches, color-based and feature-based face detection,
are studied in this research. There are many researches using color-based algorithm to
detect the color skin of faces. However, the skin-color models are varied due to nature
conditions [4], such as the lighting condition, the white-balance of cameras, and the
skin color of various race etc. Component-based face detection [5] is developed in this
research thanks to the moving object extraction, which eliminates complex background
in an image. The component-based face detection uses the facial features and its
33
De La Salle University, Manila Face Recognition System, 2009
Once face region is located, the face poses is estimated to search a frontal face.
Besides from learning-based face pose estimation, an asymmetry and geometry based
method [6] is proposed due to human face is composed in certain geometry. As a result,
different face pose geometry model can be built beforehand, as can be seen in Fig
4.1.1.3-1.
34
De La Salle University, Manila Face Recognition System, 2009
After detecting a face, a key region of the face is extracted from the face image.
An extraction method is using the geometry of common facial proportion of eyes and
the high recognition rate in the eigenspace-based face recognition [7]. A result is
35
De La Salle University, Manila Face Recognition System, 2009
Fig. 4.1.1.5-1 A. Original image from the Yale B database B. Image pre-processed by mLBP
modular PCA algorithm has been implemented successfully using FPGA with
mLBP illumination Normalization collaborate to do the job of one image per person
face recognition.
⎛L L ⎞
Eq. 4.1.1.6-1. I ijk (m, n) = I i ⎜ ( j − 1) + m, (k − 1) + n ⎟, ∀i, j , k , m, n
⎝N N ⎠
36
De La Salle University, Manila Face Recognition System, 2009
The size of face image L×L are divided into N2 smaller blocks, so the size of sub-image
see Eq. 4.1.1.6-1. The mean image of all the test sub-images is computed as Eq.
4.1.1.6-2. The weight vector of test image is computed for each of the sub-image using
Every test image from image processing unit will compare with face images in the look
Dijk , see Eq. 4.2.1 -1, represent the distances between the test image’s weight
vector and the weight vectors of the images in the database. The minimum value of Di ,
see Eq. 4.2.1 -2, means the ith face image in the database resembles closest to the test
face image.
1 M'
Eq. 4.2.1 -1 Dijk = ∑ Wtest , jkr − Wijkr , ∀i, j, k
M ' r =1
N N
1
Eq. 4.2.1 -2 Di =
N2
∑ ∑D
j =1 k =1
ijk , ∀i
A pre-defined threshold value is compared with the minimum distance, which if the
minimum distance exceeds the threshold value the test image is rejected as face image.
37
De La Salle University, Manila Face Recognition System, 2009
In order to shorten the searching time from the face database, recent hit of ten test
images will be stored in a look up table (LUT). The LUT stores the distance of hit
One frontal face image for each person is required for the system. The Di and
Row data are received from image COMS sensor in order to reduce the
computational tasks. An interface of a sensor, likely USB, I2C, CAN bus, depends on
VGA interface is developed for the communication between FPGA and a monitor.
In this research, algorithms being selected or created will be put into test before doing
hardware evaluation. In early experiment, software programs are created for each procedure to
38
De La Salle University, Manila Face Recognition System, 2009
The proposed system will be put into test on a PC by means of software programming
language is used to implement the algorithms. In this stage, the accuracy of recognition is
For hardware evaluation, Altera NIOS StratixII FPGA kit [9] (see Fig. 4.2.2-1), a
monitor, and an image CMOS sensor are needed to do the experiment. For hardware
programming language, VHDL language and Altera QuartusII are used to do the hardware
simulation. The conditions of the system evaluation should include real scene experiment,
such as variations of the source image, recognition rate, processing frames per second are
measured.
39
De La Salle University, Manila Face Recognition System, 2009
Chapter V: Summary
In face recognition system, face detection and facial feature extraction are key steps before
into face identification. For video-based face recognition, foreground and background objects
can be separated by detecting moving objects, which eliminating complex background facilitates
tasks of face detection and recognition. However, in face recognition, there are some issues, such
as illumination and pose variation, should be resolved to improve the accuracy. To deal with
these problems, several algorithms are proposed in this research, such as illumination
normalization and pose estimation. In addition, the face recognition in this system requires only
one image per person in face database, which gathering multiple images of various poses or
integrate these algorithms selected in this system. The system will be simulated by software
programming and implemented on hardware using FPGA. Finally, the system will be put into
40
De La Salle University, Manila Face Recognition System, 2009
References
[1] Stefano Messelodi, Carla Maria Modena, Nicola Segata, and Michele Zanin (2005), Kalman
filter based background updating algorithm robust to sharp illumination changes, Lecture
[4] Shinjiro Kawato and Jun Ohya (2000), Automatic Skin-color Distribution Extraction for Face
[5] Kyoung-Mi Lee (2007), Component-based face detection and verification, 2007 Elsevier B.V.
[6] Yuxiao Hu, Longbin Chen, Yi Zhou, Hongjiang Zhang (2004), Estimating Face Pose by
[8] Rajkiran Gottumukkal, Hau T. Ngo, Vijayan K. Asari (2005),Multi-lane architecture for
[9] ALTERA, Nios Development Board Reference Manual, Stratix II Edition, Nov 1, 2009
http://www.altera.com
Recognition: A Literature Survey, ACM Computing Surveys, Vol. 35, No. 4, December 2003,
41
De La Salle University, Manila Face Recognition System, 2009
pp. 399–458.
[11] Erik Hjelmas (2001), Face Detection: A Survey, 2001 by Academic Press
[12] Andrea F. Abate, Michele Nappi, Daniel Riccio, Gabriele Sabatino (2007), 2D and 3D face
[13] Xiaoyang Tan, Songcan Chen, Zhi-Hu Zhou, Fuyan Zhang (2006), Face recognition from a
[15] Elham Bagherian, Rahmita Wirza O.K. Rahmat (2008), Facial feature extraction for face
[17] T. Nakano, T. Morie, and A. Iwata (2003), A Face/Object Recognition System Using FPGA
[18] Xiitoguaiig Li and Shawlti Areibi (2004), A Hardware/Software Go-design Approach for
[20] Mohan A.R. ,N. Sudha and Pramod K. Meher (2008), An Embedded Face Recognition
System on A VLSI Array Architecture and its FPGA Implementation, 2008 IEEE
[21] I. Sajid, M. M. Ahmed, I. Taj, M. Humayun, and F. Hameed (2008), Design of High
Performance FPGA Based Face Recognition System, PIERS Proceedings, Cambridge, USA,
42
De La Salle University, Manila Face Recognition System, 2009
[22] Ginhac Dominique, Yang Fan and Paindavoine Michel (2007), Design, Implementation and
Recognition, Book edited by: Kresimir Delac and Mislav Grgic, ISBN 978-3-902613-03-5,
[23] P. Ravschert, A. Kummert, M. Krips and Y. Klimets (2002), Face Recognition By Means of
[24] Nasim Shams, Iraj Hosseini, Mohammad Sadegh Sadri, Ehsan Azarnasab (2006), LOW
[25] Gonzalo Carvajal, Waldo Valenzuela and Miguel Figueroa (2007), Subspace-Based Face
[26] Rajkiran Gottumukkal, Vijayan K. Asari (2003), System Level Design of Real Time Face
43