Survey On Human Face Recognition Systems

Salam Magdi Khalifa

U00019127
May 14, 2010

Abstract
Human face recognition is experiencing growth in general popu-
larity especially in the field of biometrics and security. Human Face
recognition systems can be used to make cameras intelligent in the
sense that they will be able to use the information they capture and
give meaningful output. The uses for such an application are only lim-
ited by imagination. In this survey we are going to look at different
approaches for developing face recognition systems. And then briefly
try to analyze each approach.

1 Introduction
People interest in developing human face recognition systems goes back to
the 60’s of the last century with the work of Bledsoe [1], where the sys-
tem relied on the geometry of (manually extracted) fiducial points such as
eye/nose/mouth corners and their spatial relationships (angles, length ra-
tios, etc.) . Kanade [2] was the first to develop a fully automatic version
of such a system. This paradigm lasted for around 30 years, until the 80’s
when researchers began experimenting with visual representations, making
use of the appearance or texture of facial images, often as raw 2D inputs to
theirs systems, such as layered neural network systems of O’Toole et al. [3]
, Flemming & Cottrell [4] as well as the associative memory models used by
Kohonen & Lehtio [5]. A debate on features vs. templates is face recognition
was mostly settled by a comparative study by Brunelli & Poggio [6], in which
template based techniques proved significantly superior.
The current state-of-the-art on face recognition is characterized (and to
some extend dominated) by a family of subspace methods originated by Turk
& Pentlands ”Eigenfaces” [7] , which by now has become a de facto standard

1
and a common performance benchmark in the field. For this reason, in this
survey we are going to study this method.
So in this survey I will look at four distinct techniques for the same pur-
pose; face recognition. The first one as I mentioned above is the standard
Eigenface by Turk & Pentland which uses the Euclidian distance as a sim-
ilarity measure. Then I will look at the Hidden Markov Models for Face
Recognition by Nefian & Hayes [8]. Another different approach I will present
is Bayesian face recognition by Moghaddam, Jebara & Pentland [9] which
uses probabilistic measure of similarity. The fourth one will be face recog-
nition using Elastic Bunch Graph Matching by Wiskott et al [10]. which
represent an individual image by a graph.

2 Eigenfaces For Recognition

The standard state-of-the-art method Eigenfaces by Turk and Pentland, this
system basically compares the characteristics of the new individual to those
faces who are known to the system. This is done under the PCA (Principal
Component Analysis) approach [11] based on the information theory ap-
proach that decomposes face images into a small set of characteristic feature
images called ”Eigenfaces”, see Figure 1 .
Eigenfaces are calculated using the known singular value decomposition
(SVD), which will generate the Eigenfaces in addition to the eigenvectors.The
least eigenvectors are then ignored in addition to their corresponding Eigen-
faces. So that we will reduce the dimensionality of the matrix. Now the
system trained with a certain number of individual images, all the Eigen-
faces and eigenvectors were extracted and stored. Then when the new image
is fed to the system (this image is not included in the training set) the sys-
tem will calculate its weight,and the Euclidean distance, if it passes a certain
threshold then it is a known face to the system.

3 Hidden Markov Models for Face Recogni-

tion
The second technique is the Hidden Markov Models for Face Recognition
by Nefian & Hayes. this method is based on the extraction of the 2D-DCT
(Two Dimensional-Discrete Cosine Transform) feature vectors. In this algo-
rithm, first they define the components of a HHM (Hidden Markov Model)
which are N: the number of states in the model, M: the number of differ-
ent observation symbols, A: the state transition probability matrix, B: the

2
 Figure 1: Eigenfaces

observation symbol probability matrix and Π: the initial state distribution.

using a shorthand notation, an HMM is defined as the triplet λ = (A, B, Π).
They first divide the image into set of overlapping blocks with height L and
width W where the amount of overlapping is P. The observation vector con-
sists of a set of 2D-DCT coefficients that are extracted from each block. In
the training phase, each individual is represented by an HMM face model
shown in Figure 2. A set of five different images (grayscale) are used to train
each HMM. Following the block extraction, a set of 39 2D-DCT coefficients
obtained from each block are used to form the observation vectors.

Figure 2: Left to right HMM for face recognition

In the recognition phase a set of test images -not used in the training

3
phase- is used. After extracting the observation vectors in the training phase,
the probability of the observation vector given each HMM face model is
computed. A face image is recognized if the threshold equals to the maximum
probability calculated above.

4 Bayesian Face Recognition

The Bayesian Face Recognition by Moghaddam, Jebara & Pentland, the
different thing about this approach is that it uses probabilistic similarity
measure instead of the traditional Euclidian techniques. Based on the belief
that the image intensity differences are characteristic of typical variations in
appearance of an individual.Two classes of facial image variations are defined:
intrapersonal variations ΩI that is the different facial expressions that belongs
to the same individual, Second class is the extrapersonal variations ΩE that
is the variation between different individuals. So now let I1 be the first facial
image and I2 be the second facial image, then the probabilistic similarity
measure is defined as S(I1 , I2 ) = P (∆ ∈ ΩI ) = P (ΩI |∆).
The first thing done, is gathering training data by computing the in-
tensity differences for a training subset of say N intrapersonal differences.
That is by matching the two views of every individual in the gallery using
the MAP (maximum a pestriori ), which states that two images are deter-
mined to belong to the same individual ifP (ΩI |∆) > P (ΩE |∆) or equiva-
lently S(I1 , I2 ) > 1/2. An alternative measure can be simply defined using
the intrapersonal likelihood alone S 0 = P (ΩI |∆) thus leading to a maxi-
mum likelihood (ML). Also for say M extrapersonal differences (by matching
images of different individuals in the gallery) they compute the intensity dif-
ferences, corresponding to the classes ΩI and ΩE respectively, and performed
a separate PCA on each. By looking at both classes they actually correspond
to a dual Eigenfaces sets (Figure 3). For recognition MAP and ML are used
for the input and then using the probabilistic similarity measure mentioned
above is used.

5 Elastic Bunch Graph Matching

Lastly we will talk about the algorithm by Wiskott et al. Face Recognition
using Elastic Bunch Graph Matching. It is a whole unique system where
individual faces are represented by a rectangular graph; where each node
labeled with a set of complex Gabor wavelet coefficients called a jet. But
only the magnitude of the coefficients is used for matching and recognition.

4
 Figure 3: Dual Eigenfaces set, (a) intrapersonal, (b) extrapersonal

In the beginning they start by defining the Jets and Graphs. A jet, is based
on a wavelet transform, defined as the convolution of the image with a family
of Gabor kernels.A labeled graph G representing a face consists of N nodes
connected by E edges, the nodes are located at facial landmarks called fiducial
points (the pupils, the corners of the mouth, the tip of the nose, the top and
bottom of the ears, etc). The nodes are labeled with jets Jn , the edges are
labeled with 2D distance vector. Note that we refer to a geometrical structure
of a graph unlabeled by jets, as a grid. Due to the rotation in depth some
fiducial points and jets may be occluded. So to be able to do the comparison
of graphs with different poses, they manually define pointers to associate
corresponding nodes in the different graphs. Thus in order to extract image
graphs automatically for new faces, a general representation is needed rather
than models of individual faces. This representation should cover a wide
range of possible variations in the appearance of faces. Obviously that is too
expensive if we cover each feature combination by a separate graph, so they
combine a representative set of M individual model graphs into a stack like
structure, called a Face Bunch Graph (FBG) see Figure .
Each model graph has the same grid structure. The nodes refer to iden-
tical fiducial points. Moreover a set of jets referring to one fiducial point is
called a bunch. After extracting all the models, recognition is then a simple
task which is done by comparing an image graph to all model graphs and
selecting the one with the highest similarity value.The similarity function
used is very simple which is the average over the similarities between pairs

5
 Figure 4: FBG (Face Bunch Graph)

of corresponding jets.

6 Analysis
Pattern recognition in general especially face recognition research aims for
better results more than performance most of the time. Since research did not
reach very satisfying results so the main concerns are about the recognition
rates mainly, most discussion here will be about recognition results.
As mentioned above, Eigenfaces face recognition approach is considered
a de facto by now; most of the current researches compare their recognition
rates results with this. Eigenfaces approach has very good recognition rates
over lightening variations, and acceptable ones over orientation variation.
But the recognition rate then degrades significantly when it come to size
variations.A limitation, in my point of view, in the paper, that the rates
highly depend on the threshold which is specified by the user, and there is
no slandered way of calculating it, but generally it is very good compared with
earlier approaches in the 80s.The choice of P and L in HMM approach can
significantly affect the system recognition rate. A high value of P significantly
increases the rate because it allows the features to be captured in a manner
that is independent of the vertical position. Moreover the choice of L is more

6
delicate; because using small L can bring insufficient discriminant information
to the observation vector while large one increases the probability if cutting
across the features. But they claim that the recognition rate is not very
sensitive to L as long as P is large but overall the system gave acceptable
results with reasonable values of L and P. Looking at the Bayesian approach
mostly they are comparing it with the standard Eigenfaces approach, in
which the Bayesian had much better recognition rates and performance as
well, they also claim that the Bayesian approach is possibly the first instance
of a non-Euclidean similarity measure in face recognition. lastly the face
recognition by Elastic Bunch graph mining, this approach presented a unique
data structure that is called the bunch graph as mentioned before, but the
limitation for this approach that it need manual initialization of the system,
where the user has to define manually pointers to associate corresponding
nodes in the different graphs so then the system will be able to generate the
rest of the graphs. The overall recognition rates were very good on the tested
datasets; also this approach can be used for almost any in-class recognition
because it is not specialized only in human faces.

7 Conclusion
To conclude this survey, we can see here four unique approaches all aiming
to get the highest recognition rates. Also researches are going more towards
using probabilistic measures instead of traditional distance measure, due to
their good recognition rates and good performance. Moreover, the FBG
approach is very good comparing to other approaches, since it only extract
features, whereas the others treat an image as a 2D array. looking at the
future, many researchers are working on the on-line recognition field to make
it robust, efficient and reliable.

References
[1] W. Bledsoe, “The model method in facial recognition,” 1966.

[2] T.Kanade, “Picture processing system by computer complex and recog-

nition of human faces,” 1973.

[3] A. J. O’Toole, R. B. Millward, and J. A. Anderson, “A physical system

approach to recognition memory for spatially transformed faces,” Neural
Networks, vol. 1, no. 3, pp. 179 – 199, 1988.

7
 [4] T.Fleming and G.Cottrell, “Categorization of faces using unsupervised
feature extraction,” Proceedings of the International Neural Network
Conference, vol. 1, pp. 322–325, 1990.

[5] T. Kohonen, E. Oju, , and P. Lehtio, “Storage and processing of infor-

mation in distributed associative memory systems,” Parallel Models of
Associative Memory, pp. 49–81, 1981. G. E. Hinton and J. A. Anderson,
editors, Erlbaum, Hillsdale.

[6] R. Brunelli and T. Poggio, “Face recognition: features versus tem-

plates,” IEEE Transactions on Pattern Analysis and Machine Intelli-
gence, vol. 15, no. 10, pp. 1042–1052, 1993.

[7] M. Turk and A. Pentland, “Eigenfaces for recognition,” Journal of Cog-

nitive Neuroscience, vol. 3, no. 1, pp. 71–86, 1991.

[8] A. V. Nefian, M. H. Hayes, and III, “Hidden markov models for face
recognition,” in Proc. International Conf. on Acoustics, Speech and Sig-
nal Processing, pp. 2721–2724.

[9] B. Moghaddam, C. Nastar, and A. Pentland, “Bayesian face recognition

with deformable image models,” in ICIAP, pp. 26–35, 2001.

[10] L. Wiskott, J.-M. Fellous, N. K. . #252;ger, and C. von der Malsburg,

“Face recognition by elastic bunch graph matching,” IEEE Transactions
on Pattern Analysis and Machine Intelligence, vol. 19, pp. 775–779,
1997.

[11] Wikipedia, “Pca (principal component analysis).”